Microsoft codename “Cloud Numerics” lab (referred to as “Cloud Numerics” in the text that follows) is a numerical and data analytics library for data scientists, quantitative analysts, and others who write C# applications in Visual Studio. It enables these applications to be scaled out, deployed, and run on Windows Azure.
This section includes the following topics:
Microsoft “Cloud Numerics” provides:
A local programming experience has the following benefits:
The following components comprise the “Cloud Numerics” lab:
The table that follows lists the "Cloud Numerics" Mathematical Libraries for .NET.
The table that follows lists the "Cloud Numerics" Distributed Array and Runtime Libraries for .NET.
CSV files
Azure Blobs
Hadoop distributed file system sequences (HDFS)
“Cloud Numerics” has the following primary modes of use:
As outlined in the following figure:
Running locally using the Numerics.Runtime provides the following functionality:
To leverage the "Cloud Numerics" libraries and distributed array data while running locally:
As outlined in following figure:
Running "Cloud Numerics" projects on Windows Azure provides the following functionality:
To leverage the "Cloud Numerics" libraries using either serial or distributed array data while running on Azure:
This section contains the software and platform requirements for using the "Cloud Numerics" libraries for the following modes of use:
You must have one of the following operating systems installed on your local development system to use the "Cloud Numerics" Lab:
These requirements satisfy both:
To run code that uses “Cloud Numerics” libraries on the Windows Azure cloud, ensure that you have the following software installed on your local development machine. Portions of this software, which you will install locally, will be bundled up as part of the package you deploy to your Windows Azure cluster.
http://go.microsoft.com/fwlink/?LinkId=231893
This section contains the additional environment-related requirements for using the "Cloud Numerics" libraries.
The following table lists the additional requirements you need to run your application using the “Cloud Numerics” distributed runtime libraries using the MPI-based local distributed libraries.
http://www.microsoft.com/en-us/server-cloud/forefront/threat-management-gateway-trial.aspx
After you satisfy the software and platform requirements, you can install the Microsoft codename “Cloud Numerics” software. Once you install the software to a local machine running Visual Studio, you can deploy and run the sample application supplied as part of the "Cloud Numerics" Visual Studio project deployment template.
The "Cloud Numerics" installer includes:
Once you are registered for the lab, you can download the software from: http://connect.microsoft.com/
The registration confirmation e-mail contains a hyperlink to the latest “Cloud Numerics” lab distribution media. If you do not yet have a confirmation e-mail, you can get one by registering here.
After you satisfy the software and platform requirements, you can install the “Cloud Numerics” software.
To install the “Cloud Numerics” software:
This section outlines the typical programming model and sequence for writing distributed applications.
You can develop, test, debug, and refine you application and algorithms on a local development system such as a many-processor or multi-core workstation or laptop. This is outlined in the Modes of Use section in the subsection titled Running a Distributed or Serial Application Locally (with Numerics Runtime).
Next, you must prepare your application to be run on Azure.
After you have created and compiled your "Cloud Numerics" project locally, you can switch to Azure simply by selecting a new Start Up Project and by providing your deployment and cluster information using the "Cloud Numerics" Deployment Utility. The "Cloud Numerics" Deployment Utility is detailed in the section titled Run a Distributed Application over an Azure Cluster.
The final step is to get your results data from blob storage, filter it, and visualize it using your favorite visualization tool such as Microsoft Excel.
You will need a Windows Azure subscription ID when you use the "Cloud Numerics" Deployment Utility to set up your cluster's deployment configuration.
You can find instructions on how to get an Azure subscription here: http://www.microsoft.com/windowsazure/offers/.
To get a free Windows Azure platform trial, navigate to http://www.microsoft.com/windowsazure/free-trial/ and sign up for the free trial service.
To get started with “Cloud Numerics” and its associated distribute runtime, you can:
To run your distributed application on a local multi-core machine, complete the steps detailed in this section.
To create a Visual Studio project of type Microsoft Cloud Numerics:
The New Project window displays.
Solution Explorer displays a newly created project. By default, the project is named MSCloudNumericsProject.
A console window displays the sample application’s output.
When you run for the first time in this mode, you might see an access message from Windows Firewall. This occurs in local distributed mode due to interprocess communication between cores on your machine in the form of network calls to localhost. In order for the application to execute, you must grant network access to this application.
In order to run you application locally on more than one processor core, use the following sequence of steps:
The project Properties window displays.
The Debug tab displays.
This section details the sequence for how to:
The following table outlines the steps for both deploying "Cloud Numerics" to a cluster and running your distributed application. This process is detailed in the subsections that follow.
To get your Azure subscription ID:
The main view displays.
To Launch the "Cloud Numerics" Deployment Utility:
Solution Explorer displays a newly created project named MSCloudNumericsProject.
The "Cloud Numerics" Deployment Utility displays.
To configure your Azure service account from the "Cloud Numerics" Deployment Utility:
The main browser pane opens to the Deployment Health view.
The management view for your active Azure subscription displays.
The Add New Management Certificate dialog displays.
You certificate has been uploaded to Azure. You can now continue filling out the fields in the Azure Account tab.
You can add a service by either of the following:
The Cluster in Azure tab displays.
To configure your Azure cluster through the "Cloud Numerics" Deployment Utility:
The SQL Azure Server tab displays.
To manage your SQL Azure Storage Service through the "Cloud Numerics" Deployment Utility:
The Application Code tab displays.
After you fill in the necessary fields in the Azure Account, Cluster in Azure, and SQL Azure Server tabs, you can use the "Cloud Numerics" Deployment Utility to create, configure, and deploy a new "Cloud Numerics" Cluster. If you have previously deployed your cluster, do not attempt to redeploy over an existing cluster.
To create, configure, and deploy your cluster.
The status bar indicates the gradual progress of the configuration step. When it is finished, the Configuration complete” message displays.
The status bar indicates the progress of the deployment. This step includes installing the necessary software on the Azure cluster nodes. When it is finished, the “Deployment complete” message displays.
This procedure lists how to submit a compiled .NET application as a job to be run on the Azure HPC Scheduler. It consists of the following parts:
To submit a job to be run on your Windows Azure cluster:
If you used the default project name, the default directory path for this project executable is in your Visual Studio Projects directory.
C:\Users\<user_name>\Documents\Visual Studio 2010\Projects\MSCloudNumerics\MSCloudNumerics\MSCloudNumericsApp\bin\Debug\
The status bar indicates the progress of the job submit step. When it is finished, the “Job successfully submitted” message displays.
You can also access Job status from either the:
To access job status and application console output from the Azure HPC Scheduler web portal.
“There is a problem with this website’s security certificate”
check that you have entered the correct service name. If the service name you typed is correct, click on the option Continue to this website.
The Windows Azure Sample Application page displays in your browser.
When the page loads, the Windows Azure HPC Scheduler Web Portal Displays.
When the web page loads, you will see the jobs submitted to the cluster scheduler in a table with the following column headings.
Job results display in the Output frame.
You can get your results data by:
(http://msdn.microsoft.com/en-us/library/windowsazure/dd135733.aspx)
You can download the Microsoft codename “Cloud Numerics” library and distributed runtime reference help to your local development machine from the Microsoft Connect portal (http://connect.microsoft.com/). The “Cloud Numerics” library and runtime reference pages are encapsulated in a single compiled HTML archive or .CHM file.
http://connect.microsoft.com/site1267/Downloads/DownloadDetails.aspx?DownloadID=40550
The Microsoft Connect download page appears.
The “Cloud Numerics” library function help file displays.
This section outlines how to convert an existing C# Visual Studio application to use the "Cloud Numerics" distributed runtime by adding the necessary library references, calls to initialize a runtime session, and calls to create distributed data within your project and application.
To convert an existing C# project to use the "Cloud Numerics" Runtime:
The following serial and distributed examples can be used, along with any of those provided in the library function reference help, in place of the sample application provided in your Microsoft codename “Cloud Numerics” Visual Studio C# project. The library function reference help is available by way of the Microsoft Connect portal (http://connect.microsoft.com/).
The following example operates on local (non-distributed) arrays created using the Numerics.Local.NumericDenseArrayFactory method.
using
System;
Microsoft.Numerics;
Microsoft.Numerics.Local;
Microsoft.Numerics.Mathematics;
namespace
BasicMathModExample
{
class
public
static
void
Main()
// Construct input arrays
var values = NumericDenseArrayFactory.CreateFromSystemArray<
double
>
(
new
[]
15.0,
17.0,
24.0,
30.0
});
var divisors = NumericDenseArrayFactory.CreateFromSystemArray<
3.0,
5.0,
7.0,
9.0
// Compute Mod function value and write result
var modulus = BasicMath.Mod(values, divisors);
Console.WriteLine(modulus);
}
The following is a serial signal processing example. The "Cloud Numerics" signal processing library methods can be used with local arrays.
Microsoft.Numerics.Signal;
IIRExample
// Counting of signal
int
N = 100;
// Generation of input singal
var x = Microsoft.Numerics.Local.NumericDenseArrayFactory.UniformGrid<Double>
(N,-2 * Microsoft.Numerics.Mathematics.BasicMath.PI,2 * Microsoft.Numerics.Mathematics.BasicMath.PI);
var inputSignal = Microsoft.Numerics.Mathematics.BasicMath.Sin(x);
// Frequencies
Freq = 0.25;
NoiseFreq = 0.1;
// Generation of noise
var t = Microsoft.Numerics.Local.NumericDenseArrayFactory.UniformGrid<Double>(N, 0, N-1);
var Noise = Microsoft.Numerics.Mathematics.BasicMath.Sin(2 * Microsoft.Numerics.Mathematics.BasicMath.PI * NoiseFreq * t / Freq) / 2;
//Input signal with noise
inputSignal += Noise;
// Creating of low-pass IIR Chebyshev-I filter
filterOrder = 8;
ripple = 0.001;
cutOffFreq = 0.5;
var IIRFilter = IIR.ChebyshevI(filterOrder,FilterKind.LowPass,cutOffFreq,ripple);
//Applying IIR filter to the input noised signal
var outputSignal = Filtering.Filter(IIRFilter, inputSignal);
Console.WriteLine(
"The filtered signal (IIR): {0}"
, outputSignal);
The following example creates local arrays with Numerics.Local.NumericDenseArrayFactory, operates on them with the Kronecker product method, casts the result array to be distributed, and then operates on this distributed array. You can run this example either by using the local distributed runtime or on Windows Azure
using System; using Microsoft.Numerics; using msnl = Microsoft.Numerics.Local; using msnd = Microsoft.Numerics.Distributed; using Microsoft.Numerics.Mathematics; namespace BasicMathExample { class BasicMathExample { public static void Main() { // Demonstrate identity sin(x)*sin(x) + cos(x)*cos(x) = 1, for local real, local complex, and distributed arrays // Initialize distributed runtime NumericsRuntime.Initialize(); long n = 10; double minValue = 0; double maxValue = 5; var values = msnl.NumericDenseArrayFactory.UniformGrid<double>(n, minValue, maxValue); var result = BasicMath.Pow(BasicMath.Sin(values), 2) + BasicMath.Pow(BasicMath.Cos(values), 2); Console.WriteLine("Identity check result: {0}", result); // Show that the identity holds for complex values too // Construct vector of complex numbers with specified phase and magnitude var phase = BasicMath.PI * msnl.NumericDenseArrayFactory.CreateFromSystemArray<double>(new double[] { -0.5, 0.0, 0.5, 1.0 }); var magnitude = msnl.NumericDenseArrayFactory.CreateFromSystemArray<double>(new double[] { 0.25, 0.5, 0.75, 1.0 }); var complexValues = magnitude.ConvertTo<Complex128>() * BasicMath.Cis(phase); // Apply formula var complexResult = BasicMath.Pow(BasicMath.Sin(complexValues), 2) + BasicMath.Pow(BasicMath.Cos(complexValues), 2); Console.WriteLine("Identity check for complex numbers"); Console.WriteLine("Magnitude: {0}", BasicMath.Abs(complexResult)); Console.WriteLine("Real Part: {0}", BasicMath.RealPart(complexResult)); Console.WriteLine("Complex Part: {0}", BasicMath.ImaginaryPart(complexResult)); // Show that the identity holds for large distributed array, // by using distributed function overloads // Use Kronecker product to expand vector into distributed matrix n = 2000; msnd.NumericDenseArray<double> row = msnl.NumericDenseArrayFactory.UniformGrid<double>(n, minValue, maxValue).Reshape(new long[] { 1, n }); msnd.NumericDenseArray<double> column = msnl.NumericDenseArrayFactory.UniformGrid<double>(n, minValue, maxValue).Reshape(new long[] { n, 1 }); var distributedValues = Microsoft.Numerics.LinearAlgebra.Operations.KroneckerProduct(column, row); // Compute distributed sin function and compare to exact result of 1 var distributedResult = BasicMath.Pow(BasicMath.Sin(values), 2) + BasicMath.Pow(BasicMath.Cos(values), 2); var maxError = ArrayMath.Max(BasicMath.Abs(distributedResult - 1)); Console.WriteLine("Maximum error in distributed result: {0}", maxError); // Shut down distributed runtime NumericsRuntime.Shutdown(); } } }
msnl = Microsoft.Numerics.Local;
msnd = Microsoft.Numerics.Distributed;
BasicMathExample
// Demonstrate identity sin(x)*sin(x) + cos(x)*cos(x) = 1, for local real, local complex, and distributed arrays
// Initialize distributed runtime
NumericsRuntime.Initialize();
long
n = 10;
minValue = 0;
maxValue = 5;
var values = msnl.NumericDenseArrayFactory.UniformGrid<
>(n, minValue, maxValue);
var result = BasicMath.Pow(BasicMath.Sin(values), 2) + BasicMath.Pow(BasicMath.Cos(values), 2);
"Identity check result: {0}"
, result);
// Show that the identity holds for complex values too
// Construct vector of complex numbers with specified phase and magnitude
var phase = BasicMath.PI * msnl.NumericDenseArrayFactory.CreateFromSystemArray<
>(
[] { -0.5, 0.0, 0.5, 1.0 });
var magnitude = msnl.NumericDenseArrayFactory.CreateFromSystemArray<
[] { 0.25, 0.5, 0.75, 1.0 });
var complexValues = magnitude.ConvertTo<Complex128>() * BasicMath.Cis(phase);
// Apply formula
var complexResult = BasicMath.Pow(BasicMath.Sin(complexValues), 2) + BasicMath.Pow(BasicMath.Cos(complexValues), 2);
"Identity check for complex numbers"
);
"Magnitude: {0}"
, BasicMath.Abs(complexResult));
"Real Part: {0}"
, BasicMath.RealPart(complexResult));
"Complex Part: {0}"
, BasicMath.ImaginaryPart(complexResult));
// Show that the identity holds for large distributed array,
// by using distributed function overloads
// Use Kronecker product to expand vector into distributed matrix
n = 2000;
msnd.NumericDenseArray<
> row = msnl.NumericDenseArrayFactory.UniformGrid<
>(n, minValue, maxValue).Reshape(
[] { 1, n });
> column = msnl.NumericDenseArrayFactory.UniformGrid<
[] { n, 1 });
var distributedValues = Microsoft.Numerics.LinearAlgebra.Operations.KroneckerProduct(column, row);
// Compute distributed sin function and compare to exact result of 1
var distributedResult = BasicMath.Pow(BasicMath.Sin(values), 2) + BasicMath.Pow(BasicMath.Cos(values), 2);
var maxError = ArrayMath.Max(BasicMath.Abs(distributedResult - 1));
"Maximum error in distributed result: {0}"
, maxError);
// Shut down distributed runtime
NumericsRuntime.Shutdown();