This forum is closed. Thank you for your contributions.

Remove From My Forums

Working with XDF blocks in a Hadoop/Spark compute context

Question
0

Sign in to vote

I understand that RevoScaleR functions work on data blocks of an XDF file, but what about data locality in a Hadoop/spark context?

How can we run an operation on an XDF file by executing the operation on each node using the local XDF blocks on that node?

Can we do that with RevoPemaR package?

Friday, November 18, 2016 10:32 AM

All replies

0

Sign in to vote
By default XDF files will be created as composite set in HDFS and data locality will be used to determine the nodes used for analysis. For more information, please see help via ?RxXdfData.

The RevoPemaR package is not yet available on Hadoop but you can accomplish many of the same capabilities using rxDataStep and a transform function. See this doc page for more info: https://msdn.microsoft.com/en-us/microsoft-r/scaler-getting-started-4-write-chunking-algorithms
- Edited by rkittler (MSFT) Wednesday, December 7, 2016 5:57 PM
Wednesday, December 7, 2016 5:45 PM