Spoken Dialog Systems

Diane J. LitmanProfessor, Computer Science Department

2

Spoken Dialog Systems Systems that interact with users via

speech Provide automated telephone or

microphone access to a back-end Advantages: naturalness, efficiency, eyes

and hands free

user

Speech Recognition

TTS or recording

DB, web,system

Spoken Dialog System

3

Challenges in Spoken Dialog Systems

Automated speech recognition Sphinx, Microsoft Speech, Dragon Naturally Speaking

Natural language understanding Dialog Management

How to keep the conversation going? Best strategy? How to detect errors in communication? How to recover from errors?

Spoken language generation

4

Application areas I have worked on

AT&T Phone-based Information Access Call Centers Social Networking Systems

Pitt (Physics) Tutoring Backup for Port Authority human operators

Other Interests Training, Troublesheeting, PDA’s

5

Speech-based Computer Tutors What are they? Example

Tutor: Well, if an object has non zero constant velocity, is it moving or staying still?

Student: Moving Tutor: Yep. If it’s moving, then its position is changing. So then

what will happen to the packet’s horizontal displacement from the point of its release?

Student: It will change Intersection of two fields:

Spoken Dialog Systems Intelligent Tutoring Systems

6

Intelligent Tutoring Systems

EducationClassroom instruction [most frequent form]Human (one-on-one) tutoring [most effective

form] Computer tutors – Intelligent Tutoring

SystemsNot as good as human tutorsWays to address the performance gap

(Spoken) dialog systems Affective (dialog) systems

7

• Back-end is Why2-Atlas system [VanLehn, Jordan, Rose et al. 2002]• Sphinx2 speech recognition and Cepstral text-to-speech

8

• Back-end is Why2-Atlas system [VanLehn, Jordan, Rose et al. 2002]• Sphinx2 speech recognition and Cepstral text-to-speech

9

• Back-end is Why2-Atlas system [VanLehn, Jordan, Rose et al. 2002]• Sphinx2 speech recognition and Cepstral text-to-speech

10

Current Research Directions Automatic System Optimization

Can a system learn to optimize behavior based on prior data?

Evaluation How can we tell if we are improving a system?

Speech vs keyboard, TTS vs recordings, graphics vs. transcripts Can systems be tested with simulated rather than real users?

Affective Computing How can user emotions be predicted in real-time? How can the system exploit such information?

Prosodic and Linguistic Analysis Respond to both what a user says, and how it is said

Human-Computer ExcerptTutor26: Let me ask my question in a more general way. What

happens to the velocity of a body when there is no force acting on it?Student27: dammit (ASR: it is)Tutor28 : Could you please repeat that?Student29 :same (ASR: i same)Tutor30 : Right. And what happens to the HORIZONTAL velocity of a

body when there is no HORIZONTAL force acting on it?Student31: zero (ASR: the zero)Tutor32 : Well... OK, let's consider how forces in the vertical direction

affect the motion of a body in a horizontal direction in a different scenario <…omitted…>

Student33: oh gravity you already said this (ASR: beats gravity you're exert less)

Tutor34:Fine. Are there any other forces acting on the apple as it falls?Student35: no why are you doing this again (ASR: no y and to it yes)Tutor36:Correct. The only force acting on the apple as it falls to the

earth is the earth's gravitational force. In which direction does gravitational force act?

Student37: downward you computer (ASR: downward you computer)

12

Thank You! Questions? Further Information

http://www.cs.pitt.edu/~litman/itspoke.html