real time number crunching and storage on cloud RRS feed

  • Question

  • I have some hardware devices that send some data that need to be stored on the cloud server and also I need to do some real time processing on them.

    The data they send need to be preserved for months in some custom binary files. These files related to each device can grow in size up to 10GB over time.

    There will client programs (mobile / web) that will be looking at the processed data at real time. 

    My prefered choice of language is C/C++/C#, since there is time sensitive number crunching involved.

    Goal is write scalable application that can have thousands of such devices monitored on the cloud. 

    1) Do I have to upfront write the code for running on the cloud ( undestand Azure / amazon EC2) ? Can I write multi threaded desktop application and later migrate to cloud ?

    2) I have used Message passing interface (MPI) in the past for clusters. Can I still use MPI ?

    3) If I use microsoft azure API can I still host my software on Amazon cloud ?

    4) For mobile devices to talk to the server, I understand that I need to have a webservice running. how can I convert a desktop program writeen in C++ / C# to act as a web service talking to client?

    5) Are there any 3rd part frame works or tools that can help me with my work ?

    Saturday, November 10, 2012 7:59 PM


  • Your use case sounds pretty good for the abilities that Azure offers.  Storage of the binary files can be handled by Azure Storage.

    You will want to make a few decisions upfront... specifically: Azure Compute Services or Azure VM.  Former is the PaaS and latter is the IaaS.  (PaaS is a platform and while offering a few restrictions on the structure/architecture/etc of your application - offers helpful managed services to host/deploy/monitor your application)
    C# is supported for either of the scenarios.  C++ only for the latter.  Incidentally, AWS is also an IaaS offering.

    1) There is usually little to none code that interacts with the Azure management environment - it is mostly config based.  However, if you are using Azure Storage, that code will need to be abstracted away so that it could be migrated to a different storage platform easily

    2) Not sure about MPI. Azure supports a few very scalable Azure-hosted message exchange mechanisms: Azure storage Queues and Service Bus queues.  SignalR can run on top of service bus.

    3) Your lock-in to Azure will come not in form of the compute platform, but in terms of the various services from Azure that you would use: Storage, messaging, cache, etc.  If you build proper abstraction layers when dealing with Azure-specific (or frankly any 3rd party) services, switching them over to different platform/API will be much simpler.

    4) This is a very broad subject and has nothing to do with Azure.  

    5) Once you will be needing a monitoring/auto-scaling solution, feel free to check the link in my sig for help with that.


    Auto-scaling & monitoring service for Windows Azure applications at http://www.paraleap.com

    • Proposed as answer by Johnson - MSFT Wednesday, November 14, 2012 9:38 AM
    • Marked as answer by Johnson - MSFT Friday, November 16, 2012 4:28 AM
    Monday, November 12, 2012 6:25 AM