Using the Power of Speech in your Windows Phone Apps

Lori LalondeTwitter: @loriblalonde

Blog: geekswithblogs.net/lori-lalonde

About Me

Twitter: @loriblalondeEmail: loriblalonde@gmail.com

Blog: geekswithblogs.net/lorilalondeLinkedIn:

http://ca.linkedin.com/in/lorilalonde

CTTDNUG

Why should I integrate speech in my mobile

app?

Minimum RequirementsWindows Phone 8.1 Development

Windows 8.1 (x64)

Visual Studio 2013 Update 2 or later (Pro, Premium, or Ultimate)OR

Visual Studio 2013 Express for Windows Update 2 or later

*includes Windows Phone 8.1 SDK

*Windows Phone Emulators – require Professional edition of Windows, and a processor that supports Client Hyper-V and Second Level Address

Translation (SLAT)

Speech for Windows Phone

Voice Commands

Speech Recognition

Text-To-Speech

Voice Commands

Simply say something to launch (or navigate into) the app…

… instead of:

tap, tap, scroll, select, dang…go back…

tap, scroll…

oh geez forget it!

Add Capabilities

Add a Voice Command Definition File<?xml version="1.0" encoding="utf-8"?>

<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.0"> <CommandSet xml:lang="en-US"> <CommandPrefix>Contoso Rodeo</CommandPrefix> <Example> play a new game </Example>

<Command Name="PlayGame"> <Example> play a new game </Example> <ListenFor> [and] play [a] new game </ListenFor> <ListenFor> [and] start [a] new game </ListenFor> <Feedback> Starting a new game... </Feedback> <Navigate /> </Command>

... </CommandSet></VoiceCommands>

Phrase Lists

<Command Name="PlayLevel"> <Example> replay level two </Example> <ListenFor> replay level {number} </ListenFor> <Feedback> Going to level {number}... </Feedback> <Navigate /> </Command>

<PhraseList Label="number"> <Item> one </Item> <Item> two </Item> <Item> three </Item> </PhraseList>

Register the VCD using Windows.Media.SpeechRecognition; using Windows.Storage; ... private async Task InitializeVoiceCommands() {

Uri vcdUri = new Uri("ms-appx:///MyVoiceCommands.xml", UriKind.Absolute);

StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(vcdUri);

await VoiceCommandManager.InstallCommandSetsFromStorageFileAsync(file); }

Installs the CommandSet whose specified language matches the device language setting

Handle Navigation protected override void OnActivated(IActivatedEventArgs args) {

if (args.Kind == ActivationKind.VoiceCommand) {

Frame rootFrame = Window.Current.Content as Frame;VoiceCommandActivatedEventArgs vcArgs =

(VoiceCommandActivatedEventArgs)args;

//check for the command name that launched the appstring voiceCommandName = vcArgs.Result.RulePath.FirstOrDefault();

if (voiceCommandName == "ViewEntry") {

rootFrame.Navigate(typeof(ViewDiaryEntry), vcArgs.Result.Text);

} }

}

DEMO!!!

Speech Synthesis (Text-To-Speech)

Enable the app to read text to the user

The text can be:

A simple string

Speech Synthesis Markup Language (SSML) string

SSML file

Speak! Part 1

using Windows.Media.SpeechSynthesis; ...

private SpeechSynthesizer synthesizer;

public async Task SpeakAsync(string textToSpeech) {

SpeechSynthesizer synthesizer = new SpeechSynthesizer();SpeechSynthesisStream synthesisStream =

await synthesizer.SynthesizeTextToStreamAsync(textToSpeech);

... }

Speak! Part 2

public async Task SpeakAsync(string textToSpeech) {

SpeechSynthesizer synthesizer = new SpeechSynthesizer();SpeechSynthesisStream synthesisStream =

await synthesizer.SynthesizeTextToStreamAsync(textToSpeech);

if (synthesisStream != null) {

this.media.AutoPlay = true; this.media.SetSource(synthesisStream, synthesisStream.ContentType); this.media.Play();

} }

<MediaElement x:Name="media“ AutoPlay="False" />Add MediaElement to the view’s XAML

Set the media source to the stream and play

DEMO!!!

Enable the app to react when the user is speaking to it

Speech Recognition

Built-In Speech Prompts/Confirmations

Heard You Say…

Unrecognized input

Did You Say…

Listening…

using Windows.Media.SpeechRecognition; ... private SpeechRecognizer speechRecognizer; public async Task InitializeSpeechRecognizerAsync(string topicHint) {

speechRecognizer = new SpeechRecognizer();

//add web search grammar to the recognizer SpeechRecognitionTopicConstraint topicConstraint = new SpeechRecognitionTopicConstraint(SpeechRecognitionScenario.Dictation, topicHint);

speechRecognizer.Constraints.Add(topicConstraint);

await speechRecognizer.CompileConstraintsAsync(); }

Initialize SpeechRecognizer

A speech recognizer needs at least one constraint, and constraints must be compiled, before speech recognition can be performed.

private SpeechRecognizer speechRecognizer; public async Task SpeakTextAsync() {

SpeechRecognitionResult recognitionResult = await speechRecognizer.RecognizeWithUIAsync();

... }

Using the Built-In UI

private SpeechRecognizer speechRecognizer; public async Task SpeakTextAsync(string audiblePrompt, string exampleText) {

speechRecognizerUI.UIOptions.IsReadBackEnabled = true;speechRecognizerUI.UIOptions.ShowConfirmation = true;

speechRecognizer.UIOptions.AudiblePrompt = audiblePrompt;speechRecognizer.UIOptions.ExampleText = exampleText;

SpeechRecognitionResult recognitionResult = await speechRecognizer.RecognizeWithUIAsync();

... }

Setting UI Options

DEMO!!!

Questions

Additional ResourcesMSDN - Speech Design Guidelines for Windows Phone Apps

MSDN - Quickstart: Speech Recognition (XAML)

MSDN - Responding to speech interactions (XAML)

Channel9 Video: Integrating Your App into the Windows Phone Speech Experience

W3C Speech Recognition Grammar Specification

W3C Speech Synthesis Markup Language

apress.com

30% off Windows Phone 8 Recipes eBook

Discount Code: LORI14Expires: Dec 31, 2014

Lori LalondeTwitter: @loriblalonde

Blog: geekswithblogs.net/lori-lalonde