none
Service Fabric Compute Context RRS feed

  • Question

  • Hey I would like to create a compute context for service fabric, what "interfaces" do I need to implement to do this?
    Sunday, October 30, 2016 9:53 PM

All replies

  • We don't currently have an API for creating a new compute context but would be interested in learning more about why you ask.  What would be the use case? 
    Sunday, October 30, 2016 10:59 PM
  • Well there were a few use cases I wanted to explore.

    1) Creating a service fabric compute context allows r clusters to be maintained by service fabric, instead of something complex like Hadoop cluster that request lots of different pieces of technology to actually make the whole thing work.

    2) Create a RxServiceFabricData that can go after Stateful Reliable Collections like Dictionary.

    3) Create a specialized version of RxXdfData that stores Xdf chunks in Reliable Collection

    In general I want to use Data stored in Service Fabric Stateful Services and feed that into all the Rx... functions, especially the machine learning functions.

    I would like to have a 25-50 node service fabric cluster with huge amounts of data (partitioned of course) but then run functions like rxGlm, rxDTree, rxLogit, rxLinMod, etc... but with these functions actually running on each node in the service fabric cluster with the data so only the results have to be moved around the network not the data itself.

    I'm happy to write all the code to make this happen, I just need the "interfaces" into the system to make it possible.

    I'm proficient in R, C, C++, C#, pretty much every language that might be involved in this process.

    Monday, October 31, 2016 1:45 AM
  • I'm looking into ddR

    https://mran.revolutionanalytics.com/package/ddR/

    https://channel9.msdn.com/Events/useR-international-R-User-conference/useR2016/Distributed-Computing-using-parallel-Distributed-R-and-SparkR

    They offer interfaces for creating new backends.

    I would prefer to use the Microsoft stack but it looks like this is not on the roadmap?

    Thanks.

    Thursday, November 3, 2016 3:28 AM