Speech and Language Processing for Educational Applications

Professor Diane Litman

Computer Science Department &

Intelligent Systems Program &

Learning Research and Development Center

2

A few words about me… Currently

– Professor in CS and ISP (director)– Senior Scientist at LRDC– ITSPOKE research group

2 PhD students, your name here?, 3 CS undergrads, 1 postdoc, 1 programmer

– AI Research (speech and NLP, tutoring and education, applied learning, affective computing)

Previously– Member Technical Staff, AT&T Labs Research, NJ– Assistant Professor, CS at Columbia University, NY

More generally...

NLP and the Learning Sciences

More generally...

NLP and the Learning Sciences

Learning Language(reading, writing,

speaking)

Tutors

Scoring

More generally...

NLP and the Learning Sciences

Learning Language(reading, writing,

speaking)

Using Language (to teach everything else)

Tutors

Scoring

ConversationalTutors / Peers

CSCL

More generally...

NLP and the Learning Sciences

Learning Language(reading, writing,

speaking)

Using Language (to teach everything else)

Tutors

Scoring

Readability

Processing Language

ConversationalTutors / Peers

(Michael LipschultzJoanna DrummondHeather Friedberg)

CSCL

DiscourseCoding

LectureRetrieval

Questioning& Answering

NLP for Peer Review (Wenting Xiong)

•An Affect-Adaptive Spoken Dialogue System that Responds Based on User Model and Multiple Affective States

– Detect and adapt to student disengagement– Vary tutor responses based on user model (expertise, gender) to increase

learning and satisfaction

•Improving Learning from Peer Review with NLP and ITS Techniques - Detect important feedback features (i.e. is a solution given, is the review

helpful)- Enhance reviewer, author, and instructor interfaces

•Improving a Natural-Language Tutoring System That Engages Students in Deep Reasoning Dialogues About Physics

- Use of tutor specialization/abstraction- Research “in-vivo” (in a high school!)

Current Research Grants

Prior Dissertations Supervised Machine Learning for Dialogue

– Hua Ai, User Simulation for Spoken Dialog System Development

– Min Chi, Do Micro-Level Tutorial Decisions Matter: Applying Reinforcement Learning to Induce Pedagogical Tutorial Tactics

Discourse Theory for User Interfaces– Mihai Rotaru, Applications of Discourse Structure for Spoken Dialogue

Systems

Cognitive Science for Intelligent Tutoring– Arthur Ward, Reflection and Learning Robustness in a Natural Language

Conceptual Physics Tutoring System

Today: Spoken Tutorial Dialogue Motivation The ITSPOKE Tutorial Dialogue System & Corpora Detecting and Adapting to Student Uncertainty

– Uncertainty Detection – System Adaptation– Impact on Student Meta(Cognition)

» Wizarded and fully-automated experiments

Summing Up

What is Tutoring?

• “A one-on-one dialogue between a teacher and a student for the purpose of helping the student

learn something.”

[Evens and Michael 2006]

• Human Tutoring Excerpt [Thanks to Natalie Person and Lindsay Sears,

Rhodes College]

Intelligent Tutoring Systems

Students who receive one-on-one instruction perform as well as the top two percent of students who receive traditional classroom instruction [Bloom 1984]

Unfortunately, providing every student with a personal human tutor is infeasible– Develop computer tutors instead

Tutorial Dialogue Systems Why is one-on-one tutoring so effective?

“...there is something about discourse and natural language (as opposed to sophisticated pedagogical strategies) that explains the effectiveness of unaccomplished human [tutors].”

[Graesser, Person et al. 2001]

Currently only humans use full-fledged natural language dialogue

Spoken Tutorial Dialogue Systems Most human tutoring involves face-to-face

spoken interaction, while most computer dialogue tutors are text-based

Can the effectiveness of dialogue tutorial systems be further increased by using spoken interactions?

Potential Benefits of Spoken Dialogue: I

Dialogue provides a learning environment that promotes student activity (e.g., self-explanation)– Tutor: The right side pumps blood to the lungs, and the left side pumps blood to

the other parts of the body. Could you explain how that works?

– Student (self-explains): So the septum is a divider so that the blood doesn't get mixed up. So the right side is to the lungs, and the left side is to the body. So the septum is like a wall...

Self-explanation occurs more in speech [Hausmann and Chi 2002], and correlates with learning [Chi et al. 1994]

Potential Benefits of Spoken Dialogue: II

Speech contains prosodic information, providing new sources of information about the student for teacher adaptation [Fox 1993; Tsukahara and Ward 2001; Pon-Barry et al. 2005]

A correct but uncertain student turn– ITSPOKE: How does his velocity compare to that of

his keys?– STUDENT: his velocity is constant

Potential Benefits of Spoken Dialogue: III Spoken conversational environments may foster

social relationships that may enhance learning– AutoTutor [Graesser et al. 2003]

Potential Benefits of Spoken Dialogue: IV

• Some applications inherently involve spoken dialogue– Conversational Skills [Seneff, Johnson]– Reading Tutors [Mostow, Cole]

• Others require hands-free interaction– e.g., NASA training

Outline

Motivation The ITSPOKE System and Corpora Detecting and Adapting to Student Uncertainty

– Uncertainty Detection – System Adaptation– Experimental Evaluation

Summing Up

• Back-end is Why2-Atlas system [VanLehn, Jordan, Rose et al. 2002]• Sphinx2 speech recognition and Cepstral text-to-speech

• Back-end is Why2-Atlas system [VanLehn, Jordan, Rose et al. 2002]• Sphinx2 speech recognition and Cepstral text-to-speech

• Back-end is Why2-Atlas system [VanLehn, Jordan, Rose et al. 2002]• Sphinx2 speech recognition and Cepstral text-to-speech

Three Types of Tutoring Corpora Human Tutoring

– 14 students / 128 dialogues (physics problems)

Computer Tutoring– 72 students / 360 dialogues

Wizard Tutoring– 81 students / 405 dialogues– human performs speech recognition, semantic analysis– computer performs dialogue management

Experimental Procedure

College students without physics– Read a small background document– Took a multiple-choice Pretest – Worked 5 problems (dialogues) with ITSPOKE – Took an isomorphic Posttest

Goal was to optimize Learning Gain– e.g., Posttest – Pretest

Outline

Motivation The ITSPOKE System and Corpora Detecting and Adapting to Student Uncertainty

– Uncertainty Detection – System Adaptation– Experimental Evaluation

Summing Up

Monitoring Student State (motivation)

Tutor26: Let me ask my question in a more general way. What happens to the velocity of a body when there is no force acting on it?

Student27: dammit (ASR: it is)Tutor28 : Could you please repeat that?Student29 : same (ASR: i same)Tutor30 : Right. And what happens to the HORIZONTAL velocity of a body when

there is no HORIZONTAL force acting on it?Student31: zero (ASR: the zero)Tutor32 : Well... OK, let's consider how forces in the vertical direction affect the

motion of a body in a horizontal direction in a different scenario <…omitted…>Student33: oh gravity you already said this (ASR: beats gravity you're exert less)Tutor34: Fine. Are there any other forces acting on the apple as it falls?Student35: no why are you doing this again (ASR: no y and to it yes)Tutor36: Correct. The only force acting on the apple as it falls to the earth is the

earth's gravitational force. In which direction does gravitational force act?Student37: downward you computer (ASR: downward you computer)

Why Uncertainty? Most frequent student state in our dialogue corpora

[Litman and Forbes-Riley 2004]

Focus of other learning sciences, speech and language processing, and psycholinguistic studies [Craig et al. 2004; Liscombe et al. 2005; Pon-Barry et al. 2006; Dijkstra et al. 2006]

.73 Kappa [Forbes-Riley et al. 2008]

Corpus-Based Detection Methodology

Learn detection models from training corpora– Use spoken language processing to automatically extract

features from user turns– Use extracted features (e.g., prosodic, lexical) to predict

uncertainty annotations Evaluate learned models on testing corpora

– Significant reduction of error compared to baselines [Litman and Forbes-Riley 2006; Litman et al. 2007]

Outline

Motivation The ITSPOKE System and Corpora Detecting and Adapting to Student Uncertainty

– Uncertainty Detection – System Adaptation– Experimental Evaluation

Summing Up

System Adaptation: How to Respond?

Theory-based– [VanLehn et al. 2003; Craig et al. 2004]

Corpus-based– [Forbes-Riley and Litman 2005, 2007, 2008, 2010]

Theory-Based Adaptation:Uncertainty as Learning Opportunity

Uncertainty represents one type of learning impasse, and is also associated with cognitive disequilibrium– An impasse motivates a student to take an active role in

constructing a better understanding of the principle. [VanLehn et al. 2003]

– A state of failed expectations causing deliberation aimed at restoring equilibrium. [Craig et al. 2004]

Hypothesis: The system should adapt to uncertainty in the same way it responds to other impasses (e.g., incorrectness)

Corpus-Based Adaptation: How Do Human Tutors Respond?

An empirical method for designing dialogue systems adaptive to student state– extraction of “dialogue bigrams” from annotated

human tutoring corpora

– χ2 analysis to identify dependent bigrams

– generalizable to any domain with corpora labeled for user state and system response

Example Human Tutoring Excerpt

S: So the- when you throw it up the acceleration will stay the same? [Uncertain]

T: Acceleration uh will always be the same because there is- that is being caused by force of gravity which is not

changing. [Restatement, Expansion]

S: mm-k. [Neutral]

T: Acceleration is– it is in- what is the direction uh of this acceleration- acceleration due to gravity?

[Short Answer Question]

S: It’s- the direction- it’s downward. [Certain]

T: Yes, it’s vertically down. [Positive Feedback, Restatement]

Findings Statistically significant dependencies exist

between students’ state of certainty and the responses of an expert human tutor– After uncertain, tutor Bottoms Out and avoids

expansions – After certain, tutor Restates– After any emotion, tutor increases Feedback

Dependencies suggest adaptive strategies for implementation in our computer tutor [Forbes-Riley and Litman 2010]

Outline

Motivation The ITSPOKE System and Corpora Detecting and Adapting to Student Uncertainty

– Uncertainty Detection – System Adaptation– Experimental Evaluation

Summing Up

Adaptation to Student Uncertainty in ITSPOKE

Most systems respond only to (in)correctness

Recall that literature suggests uncertain as well as incorrect student answers signal learning impasses

Experimentally manipulate tutor responses to student uncertainty, over and above correctness, and investigate impact on learning– Platform: Adaptive version(s) of ITSPOKE

Normal (non-adaptive) ITSPOKE

System Initiative Dialogue Format: – Tutor Question – Student Answer – Tutor Response

Tutor Response Types:

– to Corrects (C): positive feedback (e.g. “Fine”)

– to Incorrects (I): negative feedback (e.g. “Well…”) and

» Bottom Out: correct answer with reasoning (easier)

» Subdialogue: questions walk through reasoning (harder)

Our Prior Work: Rank correctness (C, I) + uncertainty (U, nonU) states in terms of impasse severity

State: I+nonU I+U C+U C+nonU

Severity: most less least none

Adaptive ITSPOKE(s)

Our Prior Work: Rank correctness (C, I) + uncertainty (U, nonU) states in terms of impasse severity

State: I+nonU I+U C+U C+nonU

Severity: most less least none

Adaptation Hypothesis:

– ITSPOKE already resolves I impasses (I+nonU, I+U), but it ignores one type of U impasse (C+U)

– Performance improvement if ITSPOKE provides additional content to resolve all impasses

Adaptive ITSPOKE(s)

Simple Adaptation

– Same response for all 3 impasses

– Feedback on only (in)correctness

Complex Adaptation

– Different responses for the 3 impasses

» Based on human responses [Forbes-Riley and Litman 2010]

– Feedback on both uncertainty and (in)correctness

» Based on empathetic computer tutor literature [Wang et al., 2005; Hall et al., 2004; Burleson et al., 2004]

Two Uncertainty Adaptations

Simple Adaptation Example: C+UTUTOR1: By the same reasoning that we used for the car, what’s the

overall net force on the truck equal to?

STUDENT1: The force of the car hitting it?? [C+U]

TUTOR2: Fine. [FEEDBACK] We can derive the net force on the truck by summing the individual forces on it, just like we

did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE]

Same TUTOR2 subdialogue if student was I+U or I+nonU

TUTOR2: That’s exactly right, but you seem unsure, so let’s sum up. [FEEDBACK] The net force on the truck is equal to the impact force on it. We can prove this just like we did for the car. First, we know that gravity and the normal force on the truck must cancel each other, otherwise the truck would not be at rest vertically. Second, we know that the impact force is the only horizontal force exerted on the truck. [LONG BOTTOM OUT]

Complex Adaptation Example: C+U

TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to?

STUDENT1: Zero?? [I+U]

TUTOR2: That’s not correct, but don’t worry. You seem to be aware of your mistake. Let’s resolve it. [FEEDBACK] The net force on the truck is equal to the impact force on it. Let’s walk through this answer step by step. [SHORT BOTTOM OUT] We can derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE]

Complex Adaptation Example: I+U

TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to?

STUDENT1: The force of gravity. [I+nonU]

TUTOR2: I’m sorry, but I see an error in your answer. Let’s fix it. [FEEDBACK] We can derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE]

Complex Adaptation Example: I+nonU

Experiment 1: ITSPOKE-WOZ Wizard of Oz version of ITSPOKE

– Human recognizes speech, annotates correctness and uncertainty – Provides upper-bound language performance

4 Conditions– Simple Adaptation: used same response for all impasses– Complex Adaptation: used different responses for each impasse– Normal Control: used original system (no adaptation) – Random Control: gave Simple Adaptation to random 20% of

correct answers (to control for additional tutoring)

Prediction: Complex Adaptation > Simple Adaptation > Random Control > Normal Control (for increasing learning)

Procedure: reading, pretest, 5 problems, survey, posttest

Results I: Learning

Metric Condition N Mean Diff p

Learning Gain(Posttest – Pretest)

Normal Control 21 .183 < Simple Adaptation .03

Random Control 20 .269 -

Simple Adaptation 20 .307 -

Complex Adaptation 20 .213 -

F(3, 77) = 3.275, p = 0.02

Results I: Learning

Metric Condition N Mean Diff p

Learning Gain(Posttest – Pretest)

Normal Control 21 .183 < Simple Adaptation .03

Random Control 20 .269 -

Simple Adaptation 20 .307 -

Complex Adaptation 20 .213 -

Simple Adaptation yields more student learning than Normal Control (original ITSPOKE)

[Forbes-Riley and Litman 2010]

F(3, 77) = 3.275, p = 0.02

Results I: Learning

Metric Condition N Mean Diff p

Learning Gain(Posttest – Pretest)

Normal Control 21 .183 < Simple Adaptation .03

Random Control 20 .269 -

Simple Adaptation 20 .307 -

Complex Adaptation 20 .213 -

Simple Adaptation yields more student learning than Normal Control (original ITSPOKE)

[Forbes-Riley and Litman 2010]

Similar results for learning efficiency [Forbes-Riley and Litman 2009]

F(3, 77) = 3.275, p = 0.02

Discussion

Predictions versus results:

- Complex Adaptation > Simple Adaptation > Random Control > Normal Control

Why didn’t Complex Adaptation outperform Simple Adaptation?

– Complex Adaptation’s human-based content responses were based on frequency, not effectiveness

– Better data mining methods (e.g. reinforcement learning) needed

Additional Evaluations - Metacognition

Do metacognitive performance measures differ across experimental conditions?– Monitoring Accuracy [Nietfield et al. 2006]

Monitoring Accuracy

Correct Incorrect

NonUncertain CnonU InonU

Uncertain CU IU

• The wizard's annotations for each student are first represented in an array, where each cell represents a mutually exclusive option

• motivated by Feeling of (Another’s) Knowing [Smith and Clark 1993; Brennan and Williams 1995] which is closely related to uncertainty [Dijkstra et al. 2006]

• The array is then used to compute monitoring accuracy

Monitoring Accuracy

Correct Incorrect

NonUncertain CnonU InonU

Uncertain CU IU

)()(

)()(

CUInonUIUCnonU

CUInonUIUCnonUefficientHarmann Co

• Ranges from -1 (no monitoring accuracy) to 1 (perfect monitoring accuracy)

Additional Results I

Metacognitive

Measure

ComplexAdaptation

(20)

Simple Adaptation

(20)

Random Control

(20)

Normal Control

(21)Monitoring Accuracy .58 .62 .62 .52

Simple (and random) increased monitoring accuracy, compared to normal (p < .06 in paired contrasts)

[Litman and Forbes-Riley 2009]

Additional Results II

Metacognitive Measure (n=81) R p

Average Impasse Severity - .56 .00

Monitoring Accuracy .42 .00

Monitoring Accuracy (where higher is better) is positively correlated with learning

[Litman and Forbes-Riley 2009]

Experiment 2: ITSPOKE-AUTO

Sphinx2 speech recognizer– Word Error Rate of 25%

TuTalk semantic analyzer – Correctness Accuracy of 84.7%

Weka uncertainty model– Logistic regression (includes lexical, prosodic, dialogue features)

– Uncertainty Accuracy of 76.8%

Preliminary Results: ITSPOKE-AUTO

Metacognitive Measure WOZ AUTO

R p R p

Monitoring Accuracy .42 .00 .35 .00

Monitoring Accuracy remains correlated with learning under noisy conditions

More modest Local and Global learning differences across experimental conditions

s

Current and Future Work Reduce noise in fully automated system

Incorporation of student disengagement and

user modeling

Crowd sourcing (for acquiring training data)

Remediate metacognition, not just domain content

Outline

Motivation The ITSPOKE System and Corpora Detecting and Adapting to Student Uncertainty

– Uncertainty Detection – System Adaptation– Experimental Evaluation

Summing Up

Summing Up

Spoken dialogue contributes to the success of human tutors By modifying presently available technology, successful

tutorial dialogue systems can also be built Adapting to uncertainty can further improve performance

Similar opportunities and challenges in many educational applications

59

Resources Recommended classes

– Introduction to Natural Language Processing– Foundations of Artificial Intelligence– Machine Learning– Knowledge Representation– Seminar classes

Other resources– ITSPOKE Group Meetings– NLP @ Pitt– Intelligent Systems Program (ISP) Forum– Pittsburgh Science of Learning Center (PSLC)

Thank You!

Questions?

Further Information– http://www.cs.pitt.edu/~litman/itspoke.html