Puppies or muffins?Easily leverage machine learning in your apps

Sara Robinson@SRobTweets

Bret McGowen@bretmcg

2@SRobTweets @bretmcg

Who are we?

Developer Advocate, Google Cloud PlatformSara Robinson / @SRobTweets

● New York, NY● Swift fan (Taylor and language)● Harry Potter aficionado

Developer Advocate, Google Cloud PlatformBret McGowen / @bretmcg

● New York, NY● U2 fan (band and plane)● Lord of the Rings aficionado

What we’ll cover

01

02

03

04

05

A (very) brief overview of machine learning

Machine learning at Google

Vision API

Speech API

Natural Language API

01 A (very) brief overview of machine learning

5@SRobTweets @bretmcg

Machine Learning, Then

6@SRobTweets @bretmcg

Machine Learning, Now

7@SRobTweets @bretmcg

02 Machine Learning at Google

Combined vision and translation

15@SRobTweets @bretmcg

Image Search

16@SRobTweets @bretmcg

Google Photos

Google Cloud Platform 17

Doesn't seem THAT hard...

Images: WikimediaSource: https://commons.wikimedia.org/wiki/File:Red_Apple.jpg https://en.wikipedia.org/wiki/Orange_(fruit)#/media/File:Orange-Whole-%26-Split.jpg

Google Cloud Platform 19

What if I'm not a machine learning expert?

20@SRobTweets @bretmcg

The Machine Learning Spectrum

TensorFlow Cloud Machine Learning Machine Learning APIs

BYOML skills

(Friendly Machine Lea

rning)

Pre-packaged ML

02 The Cloud Vision API Complex image detection with a simple REST request

03 Making an API request

Making a request{ "requests":[ { "image": { "content": "base64ImageString"

// Alternatively, you can pass a Google Cloud Storage url here }, "features": [ { "type": "LABEL_DETECTION", "maxResults": 10 }, { "type": "FACE_DETECTION", "maxResults": 10 },

// More feature detection types... ] } ]}

Google Cloud Platform 25

Let’s see some JSON responses

26

{ "labelAnnotations" : [ { "mid" : "\/m\/01wydv", "score" : 0.92442685, "description" : "beignet" }, { "mid" : "\/m\/0270h", "score" : 0.90845567, "description" : "dessert" }, { "mid" : "\/m\/033nb2", "score" : 0.74553984, "description" : "profiterole" }, { "mid" : "\/m\/01dk8s", "score" : 0.71415579, "description" : "powdered sugar" } ] }

Label Detection

26

"landmarkAnnotations": [

{

"mid": "/m/0c7ln",

"description": "Navy Pier",

"score": 36,

"boundingPoly": {

"vertices": [

{

"x": 275,

"y": 102

}, //...

]

},

"locations": [

{

"latLng": {

"latitude": 41.888685,

"longitude": -87.601311

}

}

], //...

},

{

"mid": "/m/01_d4",

"description": "Chicago",

"score": 31,

"boundingPoly": {

"vertices": [

{

"x": 1086,

"y": 346

},

{

"x": 1496,

"y": 346

},

{

"x": 1496,

"y": 832

},

{

"x": 1086,

"y": 832

}

]

},

"locations": [

{

"latLng": {

"latitude": 41.866724,

"longitude": -87.60852

}

}

],

"lat": 41.866724,

"long": -87.60852,

"latText": "41.866724",

"longText": "-87.608520"

},

{

"mid": "/m/06_7k",

"description": "Chicago",

"score": 23,

"boundingPoly": {

"vertices": [

{

"x": 1194,

"y": 344

},

{

"x": 1537,

"y": 344

},

{

"x": 1537,

"y": 899

},

{

"x": 1194,

"y": 899

}

]

},

"locations": [

{

"latLng": {

"latitude": 41.889232,

"longitude": -87.62312299999999

}

}

],

"lat": 41.889232,

"long": -87.62312299999999,

"latText": "41.889232",

"longText": "-87.623123"

}

],

Landmark Detection

27

28

..."itemListElement": [ { "@type": "EntitySearchResult", "result": { "@id": "kg:/m/0c7ln", "name": "Navy Pier", "@type": [ "Thing", "Place", "LandmarksOrHistoricalBuildings", "TouristAttraction" ], ...

"detailedDescription": { "articleBody": "Navy Pier is a 3,300-foot-long pier on the Chicago shoreline of Lake Michigan. It is located in the Streeterville neighborhood of the Near North Side community area.", "url": "http://en.wikipedia.org/wiki/Navy_Pier"

...

Knowledge Graph sidebarGET https://kgsearch.googleapis.com/v1/entities:search?ids=%2Fm%2F0b__kbm&key={API_KEY}

29

"textAnnotations": [ { "locale": "en", "description": "U.S. COAST GUARD AUXILIARY\n242039\n", "boundingPoly": { "vertices": [ { "x": 429, "y": 307 }, { "x": 1178, "y": 307 }, { "x": 1178, "y": 770 }, { "x": 429, "y": 770 } ] } }, // ... ]

Text Detection

30

"faceAnnotations" : [

{

"headwearLikelihood" : "VERY_LIKELY",

"surpriseLikelihood" : "VERY_UNLIKELY",

"rollAngle" : 2.8030474,

"angerLikelihood" : "VERY_UNLIKELY",

"landmarks" : [

{

"type" : "LEFT_EYE",

"position" : {

"x" : 221.60617,

"y" : 638.263,

"z" : 0.0017568493

}

},

...

],

"boundingPoly" : {

"vertices" : [

{

"x" : 89,

"y" : 436

},

...

Face Detection

"detectionConfidence" : 0.98838496,

"joyLikelihood" : "VERY_LIKELY",

"panAngle" : -1.0822374,

"sorrowLikelihood" : "VERY_UNLIKELY",

"tiltAngle" : -2.5003448,

"underExposedLikelihood" : "VERY_UNLIKELY",

"blurredLikelihood" : "VERY_UNLIKELY"

31

"faceAnnotations" : [

{

"headwearLikelihood" : "VERY_UNLIKELY",

"surpriseLikelihood" : "VERY_UNLIKELY",

rollAngle" : -4.6490049,

"angerLikelihood" : "VERY_UNLIKELY",

"landmarks" : [

{

"type" : "LEFT_EYE",

"position" : {

"x" : 691.97974,

"y" : 373.11096,

"z" : 0.000037421443

}

},

...

],

"boundingPoly" : {

"vertices" : [

{

"x" : 743,

"y" : 449

},

...

Face Detection

"detectionConfidence" : 0.93568963,

"joyLikelihood" : "VERY_LIKELY",

"panAngle" : 4.150538,

"sorrowLikelihood" : "VERY_UNLIKELY",

"tiltAngle" : -19.377356,

"underExposedLikelihood" : "VERY_UNLIKELY",

"blurredLikelihood" : "VERY_UNLIKELY"

32

"labelAnnotations" : [

{

"mid" : "\/m\/01fklc",

"score" : 0.9337945,

"description" : "pink"

},

{

"mid" : "\/m\/09g5pq",

"score" : 0.83878618,

"description" : "people"

},

{

"mid" : "\/m\/017ftj",

"score" : 0.71847415,

"description" : "sunglasses"

},

{

"mid" : "\/m\/019nj4",

"score" : 0.69381392,

"description" : "smile"

}

]

Putting it all together: face + label + landmark "landmarkAnnotations" : [

{

"boundingPoly" : {

"vertices" : [

{

"x" : 153,

"y" : 64

},

...

]

},

"mid" : "\/m\/0c7zy",

"score" : 0.56636304,

"description" : "Petra",

"locations" : [

{

"latLng" : {

"longitude" : 35.449361,

"latitude" : 30.323975

}

}

]

}

]

33

"safeSearchAnnotation" : {

"spoof" : "VERY_UNLIKELY",

"medical" : "VERY_UNLIKELY",

"adult" : "VERY_UNLIKELY",

"violence" : "VERY_UNLIKELY"

}

But wait...is it appropriate?

04 Live Demo

03 The Speech API Speech to text transcription in over 80 languages

36@SRobTweets @bretmcg

What can I do with the Speech API?● Speech to text transcription in over 80 languages

● Supports streaming and non-streaming recognition

● Filters inappropriate content

37@SRobTweets @bretmcg

Translation Response "responses": [{

"results": [{

"alternatives": [

{

"transcript": "how old is the Brooklyn Bridge",

"confidence": 0.987629

}],

"isFinal": true

}

]

}]

38@SRobTweets @bretmcg

Let’s make a recording!1. Make a recording using SoX, a command line utility for audio files2. Base64 encode the recording3. Build our API request in a JSON file4. Send the JSON request to the Speech API

Bash script for this: bit.ly/speech-request-script

04 Cloud Natural Language API Perform sentiment analysis and entity recognition on text

40@SRobTweets @bretmcg

What can I do with the Natural Language API?Three methods:

1. Analyze entities - The Cubs are an MLB team from Chicago

2. Analyze sentiment - I love Chicago

3. Analyze syntax - Michelle Obama is married to Barack Obama

41@SRobTweets @bretmcg

Analyze Entities

Chicago is

The Wrigley Building, Chicago is

The Union Stockyard, Chicago is

One town that won't let you down

It's my kind of town

-- Frank Sinatra

42@SRobTweets @bretmcg

Analyze Entities

Chicago is

The Wrigley Building, Chicago is

The Union Stockyard, Chicago is

One town that won't let you down

It's my kind of town

-- Frank Sinatra

43@SRobTweets @bretmcg

Analyze Entities

Chicago is

The Wrigley Building, Chicago is

The Union Stockyard, Chicago is

One town that won't let you down

It's my kind of town

-- Frank Sinatra

44@SRobTweets @bretmcg

Analyze Sentiment

Chicago is the best city in the world.

{

“documentSentiment”: {

“polarity”: 1,

“magnitude”: 0.8

}

}

45@SRobTweets @bretmcg

Analyze Syntax

“The Chicago Cubs are an American professional baseball team based in

Chicago, Illinois.”

46@SRobTweets @bretmcg

Analyze Syntax

47@SRobTweets @bretmcg

Let’s see a demo!

Twitter Streaming NL API BigQuery

bit.ly/nl-olympics

48@SRobTweets @bretmcg

APIs we covered● Vision: cloud.google.com/vision

● Speech: cloud.google.com/speech

● Natural Language: cloud.google.com/natural-language

Related APIs:

● Translate: cloud.google.com/translate

● Prediction: cloud.google.com/prediction

● Knowledge Graph API: developers.google.com/knowledge-graph

Thank YouSara Robinson@SRobTweets

Bret McGowen@bretmcg