Scope


This article shows how to integrate a bot created with Bot Framework. We will create a UWP App with Cortana for conversion between text and speech while the bot contains the logic and intelligence.

 

What is a Bot?


Bots are programs or logic used for natural human-like interactions with our system. It can be used for providing support for simple issues, for creating digital assistant, automate the responsibilities and number of other applications. The most common use of a bot is to automate the repetitive work we do.

 

Creating the Bot


Bot Backend

First we need to create a bot backend and deploy it on Azure. The bot backend will contain all the logic and AI for the bot. You can use LUIS for natural language processing, Microsoft Cognitive Services or your custom machine learning models. It can scale as much as you want. For this demo, we will just use the prebuilt Bot Application template.

  1. First you need to install the bot application template in Visual Studio. You can download it here.

  2. Create a new Bot Application. You can test it in Bot Framework Emulator locally. Just append the url with /api/messages. For example if your bot application is running on localhost:3797 then the url to write in bot framework will be http://localhost:3979/api/messages
  3. After testing the bot, publish it on azure. Right click on the project in solution explorer and click publish. Then select Microsoft Azure App Service as a publish target. Create a new azure app, resource group and service plan if required. The wizard will guide you through the process of creating and publishing on azure. The app name you choose in the wizard will be used in url.
  4. Now your app will be published to <app name>.azurewebsites.net

Bot Framework

Now go to bot framework website, log in using your Microsoft account and create a new bot.

Enter the messaging endpoint as

<bot url>/api/messages

For example, if your bot url is mybot.azurewebsites.net then set the messaging endpoint to be

mybot.azurewebsites.net/api/messages

Remember the bot handle you set here, we will need it later. After you create the bot you can use add it to skype, your website, slack and number of other services. The easiest way to use the bot in our apps is to use direct link. Just enable the direct link from the available services at your bot’s main page and you are done.

 

Creating the UWP App


Listening to the Speech

Spoken words can be listened easily using Cortana. Use the following code get the recognized text from the speech. You must first set the microphone permission in Package.appxmanifest file.

public async Task<string> SpeechToTextAsync()
 {
         var recognizedText = string.Empty;
 
        //Max time to wait for the speaker to speak
         _recognizer.Timeouts.InitialSilenceTimeout = TimeSpan.FromSeconds(3);
            
        
await _recognizer.CompileConstraintsAsync();          
 
         SpeechRecognitionResult result = await _recognizer.RecognizeAsync();
         if (result.Status == SpeechRecognitionResultStatus.Success)
         {
            recognizedText = result.Text;
         }
         return recognizedText;
}

 

Speaking a Sentence

To speak a given text use the following code,

public async Task<IRandomAccessStream> TextToSpeechAsync(string text)
{
    IRandomAccessStream stream = await _synthesizer.SynthesizeTextToStreamAsync(text);
    return stream;
}

Returned stream can be given to the MediaElement to play the sound. You can change voice properties by modifying the synthesizer object. Let’s use the default voice for the first time.

var stream = await _speech.TextToSpeechAsync(str);
var MediaEl = new MediaElement();
MediaEl.SetSource(stream, "");
MediaEl.Play();

Initializing the Bot

To use the bot first add the directline package from nuget. After that use the following statement to initialize the bot,

try
{
    _botClient = new DirectLineClient(“<<Your direct line key here>>”);
    _conv = _botClient.Conversations.NewConversation();
    IsBotConfigured = true;
}
 
catch (Exception)
{
    IsBotConfigured = false;
}



Sending a Message to the Bot

A message can be sent to the bot using the following code below. The FromProperty distinguishes the participants in the conversation.

public async Task SendMessageAsync(string message)
{
      if (!IsBotConfigured) return null;
      var x = await _botClient.Conversations.PostMessageAsync(_conv.ConversationId, new Message
      {
           FromProperty = "UWPApp",
           Text = message
      });                
}



Receiving a Message from the Bot

 To receive the message, use the code below:

public async Task<string> GetNewMessageAsync()
{
    MessageSet messages;
 
    try
    {
        messages = await _botClient.Conversations.GetMessagesAsync(_conv.ConversationId, _watermark);
    }
 
    catch
    {
        return null;
    }
 
    _watermark = messages.Watermark ?? _watermark;
    if (!messages.Messages.Any()) return null;
 
    var botmsgs = from m in messages.Messages
        where m.FromProperty=="<<Your bot handle>>"
        select m.Text;
 
    string msgStrings = "";
    botmsgs.ToList().ForEach(s=> msgStrings += s + "\n");
    return msgStrings;
}

The watermark is used to keep mark of the messages already received, so the new messages are received even if the new messages were sent during the loss of connection.

 

Adding it All Together


Now we have all pieces of the puzzle, lets add it together.

  1. Initialize the bot on startup
  2. Listen for the spoken word on button press using the SpeechToTextAsync() method.
  3. Send the recognized string to the bot if it’s not empty or null using SendMessageAsync() method.
  4. Get the response using GetNewMessageAsync().
  5. Convert the Speech to sound stream using TextToSpeechAsync.
  6. Play the sound stream using MediaElement.

The total latency varies from 1-5 seconds depending on your network and some other factors.


Conclusion


This article elaborated about how to build the Microsoft Bot application and integrate it with your UWP app to take advantage of Cortana.

See Also

Another important place to find an extensive amount of Cortana Intelligence Suite related articles is the TechNet Wiki itself. The best entry point is Cortana Intelligence Suite Resources on the TechNet Wiki.