1 / 28

http://www.flickr.com/photos/joshmichtom/4311110421/

Guardian: A Crowd-Powered Spoken Dialog System for Web APIs

Ting-Hao (Kenneth) HuangCarnegie Mellon University

Walter S. LaseckiUniversity of Michigan

Jeffrey P. BighamCarnegie Mellon University HCOMP

2015

2 / 28

What time is it?

It’s 9:30.

Kenneth’s apartment.

3 / 28

How was the Pirates game last night?

!

Kenneth’s apartment.

4 / 28

How was the Steelers game yesterday?

!

Kenneth’s apartment.

5 / 28

Is the movie Martian still playing in theaters?

!

Kenneth’s apartment.

6 / 28

Use Web APIs to EmpowerDialog Systems

7 / 28

Gap between User & Machine

?

8 / 28

A Crowdsourcing Solution

9 / 28

Two Challenges

term

location

Hi, I’m in San Diego.Any Chinese restaurants here?

Define Parameters

Extract Parameters

10 / 28

How Do Dialog Systems Usually Do?

term

location

Hi, I’m in San Diego.Any Chinese restaurants here?

Define Parameters

Extract Parameters

11 / 28

Bridging this Gap is Expensive

• Define Parameters requires Experts– Experts are expensive.– Most services are not designed for dialog systems.– Unsupervised Slot Induction

• Extract Parameters requires Data– (Which we don’t have.)– Supervised Slot Filling

• Slot Filling / Entity Recognition– No labeled data

• State Tracking– No dialogue data

– Unsupervised Slot Filling

12 / 28

Can the Crowd Do It?

term

location

Hi, I’m in San Diego.Any Chinese restaurants here?

Define Parameters

Extract Parameters

13 / 28

Define Parameters

term

location

Hi, I’m in San Diego.Any Chinese restaurants here?

Define Parameters

Extract Parameters

14 / 28

How machines understand a Web API?

1. Use which parameters ?

2. Ask user what questions to elicit these parameters?

Yelp Search API 2.0 has22 parameters.

15 / 28

Parameter Rating Problem

offset

term

location

sw_latitude

sw_longitude

category_filter

accuracy

deals_filter

radius_filter

...

offset

term

location

sw_latitude

sw_longitude

category_filter

accuracy

deals_filter

radius_filter

...

Pick good parameters for the dialog system.

16 / 28

How about just do a survey?

TaskParameter Name / Desc

17 / 28

Baselines

MAP MRR0

0.10.20.30.40.50.60.70.80.9

1

Not UnnaturalAsk SiriAsk a Friend

Average results of 8 Web APIs’ parameters

Results are not so good...

18 / 28

Match Questions with Parameters

offset

I like Chinese food.

What do you want to eat?? !

I’m in Pittsburgh.

Which city are you in?? !

Dinner.

Is it dinner or lunch?? ! ...

location?

!

term?

!!

?!

?!

?!

?!category_filter

?!

?!

?!

?!

?!

?!

?!

?! ?

!?

! ?!

?!

?!

?!

?!

?!

?!?

!

?!

?!

?!

?!

?!?

!

?!

?!

?!

?!?

!?

!?

!

?!

?!

?!

?!

term

location

sw_latitude

sw_longitude

category_filter

Better Parameter

Yelp API

Question Collection

Parameter Filtering

Qestion-Parameter Matching

19 / 28

Evaluation on Parameter Ranking

MAP MRR0

0.10.20.30.40.50.60.70.80.9

1Question MatchingNot UnnaturalAsk SiriAsk a Friend

Question Matching outperforms all baselines.

Average results of 8 Web APIs’ parameters

20 / 28

Questions Collected Already!

1. Use which parameters ?

2. Ask user what questions to elicit these parameters?

Yelp Search API 2.0 has22 parameters.

21 / 28

Extract Parameters

term

location

Define Parameters

Extract Parameters

Hi, I’m in San Diego.Any Chinese restaurants here?

22 / 28

Dialog ESP Game

Hi, I’m in San Diego.

AnswerAggregate

Location =San Diego

Recr

uite

d Pl

ayer

s

Time Constraint

23 / 28

Guardian: A Crowd-Powered Spoken Dialog System for Web APIs

3

2 Call Web APIHi, I’m in San Diego.Any Chinese restaurants here?

1 Talk and Extract Parameter

Interpret Result to User

Mandarin Wok Restaurant is good ! It’s on 4227 Balboa Ave.

term = Chinese

location = San Diego

YelpSearchAPI 2.0

{ ... "name": "Mandarin Wok Restaurant”,..."address":["4227Balboa Ave”,...], …}

JSON

24 / 28

Engineering Challenges

• Real-time Response ……..…..……… Retainer Model• Converse with User ……………………………….. Chorus • Speech Recognition ………………... Web Speech API• Parameter Extraction ………..…… Dialog ESP-Game• JSON Visualization ………………..….. JSON Visualizer• Response Generation Assistant ………………. jQuery• Workflow Control ………………. Game-like Interface• Dialog Management …………. Finite-state Machine• Crowdsourcing Platform …………. Mechanical Turk

25 / 28

System Evaluation

Web API

TaskFind Chineserestaurants in

Pittsburgh.

Check current weather

by using a zip code.

Find information of “Titanic”.

Valid JSON 9 / 10 9 / 10 6 / 10

Task Completion 10 / 10 9 / 10 10 / 10

Domain Referenced TCR 0.96 0.94 0.88

26 / 28

Guardian: A Hybrid Framework

Annotate Data on the Fly !

27 / 28

What’s next?

• More Automations– Slot Filling / Entity Recognition– Dialog Management– Response Generation

• 1,000+ APIs?• Future of Dialog Systems– What if you can really talk to a machine…– On wearable device?

28 / 28

Thank you!

( SORT OF )

http://www.flickr.com/photos/joshmichtom/4311110421/