1 Cohesion and Learning in a Tutorial Spoken Dialog System Art Ward Diane Litman.

1

Cohesion and Learning in a Tutorial Spoken Dialog System

Art WardDiane Litman

2

Outline

Tutoring Goals 4 issues in measuring cohesion

Why they’re interesting How we test them

Results

3

Natural Language Dialog Tutoring Human tutors are better than classroom

instruction (Bloom 84) Intelligent Tutoring Systems (ITSs) hope to

replicate this advantage Is Dialog important to learning?

Dialog acts: question answering, explanatory reasoning, deep student answers (Graesser et al. 95, Forbes-Riley et al. 05)

Difficult to automatically tag dialog input, so: Automatically detectable dialog features

Average turn length, etc. (Litman et al. 04) We look at Cohesion

Lexical Co-occurrence between turns

4

Goals and Results Goals

Want to find if cohesion is correlated with learning in our tutoring dialogs.

If it is, may inform ITS design Want to find a computationally tractable measure of

cohesion So can be used in a real-time tutor

Results Do find strong correlations with learning

For low pre-testers For interactive (tutor to student) measures of cohesion

Robust to multiple measures of lexical cohesion

5

4 Issues Why/How identify cohesion in dialogs? Do students of different skill levels respond to

cohesion in the same way? (Is there an aptitude/treatment interaction?)

Is Interactivity Important? What other processing steps help?

6

Issue 1: How identify cohesion in dialogs? Why might cohesion be important in tutoring?

McNamara & Kintsch (96) Students read high & low coherence text

High coherence text was low coherence version altered to:

Use consistent referring expressions Identify anaphora Supply background information

Interaction between pre-test score & response to textual coherence

Low pre-testers learned more from more coherent text High pre-testers learned LESS from more coherent text

7

Measuring Cohesion Measurements from Computational Linguistics

Hearst(94) topic segmentation, text Word-count similarity of spans of text

Olney & Cai (05) topic segmentation, tutorial dialog Several measures, including Hearst’s

Morris & Hirst (91) Lexical Chains Thesaurus entries

Barzilay & Eldihad (97) Automatic Lexical Chains WordNet senses

We develop measures similar to Hearst’s But novel in that:

Applied to dialog rather than text, used to find correlations with learning

8

Issue 1: How identify cohesion in dialogs?

Defining Cohesion Halliday and Hassan (76)

Grammatical vs Lexical Cohesion Lexical Cohesion

Reiteration Exact word repetition Synonym repetition Near Synonym repetition Super-ordinate class General referring noun

Cohesion measured by counting “cohesive ties” Two words joined by a cohesive device (i.e.

reiteration)

9


Defining Cohesion Halliday and Hassan (76)

Grammatical vs Lexical Cohesion Lexical Cohesion

Reiteration Exact word repetition Synonym repetition Near Synonym repetition Super-ordinate class General referring noun

Cohesion measured by counting “cohesive ties” Two words joined by a cohesive device (i.e.

reiteration)

10


How we measure Lexical Cohesion We count cohesive ties between turns

Tokens (with stop words) (token = “word”)

Tokens (stop words removed) (Stops = high frequency, low information

words) Stems (stop words removed)

11

Stems

Stem = non-inflected core of a word Porter Stemmer Allows us to find ties between various

inflected forms of the same word in adjacent turns. “Turns” are tutor and student contributions

to Tutoring Dialogs collected by the ITSPOKE group.

Stem Tokens

packag package, packagespacket packet, packetsspeed speed, speeding, speedsveloc veloc, velocities, velocity, velocitys

horizont horizontal, horizontally displac displace, displaced, displacement, displacements, displacing

find find, findingso sothu thus

12

Applying Cohesion measures to our Corpora: example

Turn Contribution

Student Essay

No. The airplane and the packet have the same horizontal velocity. When the packet is dropped, the only force acting on it is g, and the net force is zero. The packet accelerates vertically down, but does not accelerate horizontally. The packet keeps moving at the same velocity while it is falling as it had when it was on the airplane. There will be displacement because the packet still moves horizontally after it is dropped. The packet will keep moving past the center of the swimming pool because of its horizontal velocity.

ITSPOKE Uh huh. There is more still that your essay should cover. Maybe this will help you remember some of the details need in the explanation. After the packet is released, the only force acting on it is gravitational force, which acts in the vertical direction. What is the magnitude of the acceleration of the packet in the horizontal direction?

Cohesive Ties Matches Count

Token w/stop packet, horizontal, the, it, is, of, only, force, acting, on, there, will, still, after 14

Token, no stop packet, horizontal, only, force, acting, there, will, still, after 9

Stem, no stop packet, horizont, onli, forc, act, acceler, vertic, there, will, still, after 11

13


Turn Contribution

Student Essay







14


Turn Contribution

Student Essay







15


Turn Contribution

Student Essay







16

Issue 2: Is there an aptitude/treatment interaction?

Why there might be: McNamara & Kintsch

How we test it: Mean pre-test split

All students Above-mean pretest students (“high” pre-

testers) Below-mean pretest students (“low” pre-

testers)

17

Issue 3: Is interactivity Important? Why it might be:

Chi et al. (01) Tutor centered, Student centered, Interactive Deep learning through self construction

Not tutor actions alone Litman & Forbes-Riley (05)

Learning correlated with both: student utterances that display reasoning tutor questions that require reasoning

How we test it: Interactive corpus – compare tutor to student turns Tutor–only corpus Student–only corpus

18

Issue 4: What other processing steps help? Tried several on training corpus:

Removing stop words N-turn spans Selecting “substantive” turns TF-IDF normalization Turn-normalized counts

(Raw tie count / # of turns in dialog) Found final options on training corpus:

One turn spans, turn normalization, no TF-IDF, no substantive turn selection

All reported results use these options Tested options on new corpus

19

Where did the corpora come from? ITSPOKE is a speech-enabled version of

Why-2 Atlas (VanLehn et al. 02) Qualitative physics

Tutoring Cycle Student reads instructional materials Takes a pre-test Starts Interactive tutoring cycle

Problem Essay Tutor evaluates essay, engages in dialog Revise essay Repeat

Takes a post-test

20

Tutoring Corpora Transcripts of tutoring sessions

Training corpus (fall 2003): 20 students, 5 problems each 95 dialogs (5 had no dialog) 13 low pre-testers, 7 high pre-testers

Testing corpus (spring 2005): 34 students, 5 problems each 163 dialogs (7 had no dialog) 18 low pre-testers, 16 high pre-testers

21

Results: Aptitude/Treatment

Test: partial correlation of post-test & cohesion count, controlling for pre-test

Cohesion correlated with learning for low pre-test students

Not for high pre-test students

Little difference between types of measurement

Less significant on testing data, “token with stops” level reduced to a trend

Tests

Train: 2003 DataTest: 2005

Data

Students R P-Value RP-

Value

Grouped by Token (with stop words)

All Students 0.380 0.098 0.207 0.239

Low Pretest 0.614 0.026 0.448 0.062

High Pretest 0.509 0.244 0.014 0.958

Grouped by Token (Stop words removed)

All Students 0.431 0.058 0.269 0.124

Low Pretest 0.676 0.011 0.481 0.043

High Pretest 0.606 0.149 0.132 0.627

Grouped by Stem (Stop words removed)

All Students 0.423 0.063 0.261 0.135

Low Pretest 0.685 0.010 0.474 0.047

High Pretest 0.633 0.127 0.121 0.655

22






Slightly less significant on testing data

Tests


Data


Value


All Students 0.380 0.098 0.207 0.239

Low Pretest 0.614 0.026 0.448 0.062

High Pretest 0.509 0.244 0.014 0.958


All Students 0.431 0.058 0.269 0.124

Low Pretest 0.676 0.011 0.481 0.043

High Pretest 0.606 0.149 0.132 0.627


All Students 0.423 0.063 0.261 0.135

Low Pretest 0.685 0.010 0.474 0.047

High Pretest 0.633 0.127 0.121 0.655

23







Tests


Data


Value


All Students 0.380 0.098 0.207 0.239

Low Pretest 0.614 0.026 0.448 0.062

High Pretest 0.509 0.244 0.014 0.958


All Students 0.431 0.058 0.269 0.124

Low Pretest 0.676 0.011 0.481 0.043

High Pretest 0.606 0.149 0.132 0.627


All Students 0.423 0.063 0.261 0.135

Low Pretest 0.685 0.010 0.474 0.047

High Pretest 0.633 0.127 0.121 0.655

24







Tests


Data


Value


All Students 0.380 0.098 0.207 0.239

Low Pretest 0.614 0.026 0.448 0.062

High Pretest 0.509 0.244 0.014 0.958


All Students 0.431 0.058 0.269 0.124

Low Pretest 0.676 0.011 0.481 0.043

High Pretest 0.606 0.149 0.132 0.627


All Students 0.423 0.063 0.261 0.135

Low Pretest 0.685 0.010 0.474 0.047

High Pretest 0.633 0.127 0.121 0.655

25







Tests


Data


Value


All Students 0.380 0.098 0.207 0.239

Low Pretest 0.614 0.026 0.448 0.062

High Pretest 0.509 0.244 0.014 0.958


All Students 0.431 0.058 0.269 0.124

Low Pretest 0.676 0.011 0.481 0.043

High Pretest 0.606 0.149 0.132 0.627


All Students 0.423 0.063 0.261 0.135

Low Pretest 0.685 0.010 0.474 0.047

High Pretest 0.633 0.127 0.121 0.655

26







Tests


Data


Value


All Students 0.380 0.098 0.207 0.239

Low Pretest 0.614 0.026 0.448 0.062

High Pretest 0.509 0.244 0.014 0.958


All Students 0.431 0.058 0.269 0.124

Low Pretest 0.676 0.011 0.481 0.043

High Pretest 0.606 0.149 0.132 0.627


All Students 0.423 0.063 0.261 0.135

Low Pretest 0.685 0.010 0.474 0.047

High Pretest 0.633 0.127 0.121 0.655

27







Tests


Data


Value


All Students 0.380 0.098 0.207 0.239

Low Pretest 0.614 0.026 0.448 0.062

High Pretest 0.509 0.244 0.014 0.958


All Students 0.431 0.058 0.269 0.124

Low Pretest 0.676 0.011 0.481 0.043

High Pretest 0.606 0.149 0.132 0.627


All Students 0.423 0.063 0.261 0.135

Low Pretest 0.685 0.010 0.474 0.047

High Pretest 0.633 0.127 0.121 0.655

28

Results: Aptitude/Treatment (2003 data)

No significant difference between amounts of (turn normalized) cohesion in high and low pre-test groups. Difference in correlation between high and

low pre-testers not due to different amounts of cohesion.

Mean CohesionHigh Pre Low Pre P-Val

Token (with Stop words) 9.978 9.449 0.581Token (Stops removed) 5.375 5.209 0.768Stem (Stops removed) 5.713 5.611 0.867

29

Results: Interactivity (2003)

Cohesion between tutor utterances is not correlated with learning

Tutor Turns OnlyTrain: 2003 Data Test: 2005 Data

Students R P-Value R P-Value

Grouped by Token (with stop words)All Students 0.060 0.800 0.190 0.270Low Pretest 0.121 0.695 0.260 0.290High Pretest 0.531 0.220 0.100 0.700

Grouped by Token (Stop words removed)All Students 0.004 0.987 0.160 0.380Low Pretest 0.114 0.710 0.230 0.350High Pretest 0.351 0.440 0.070 0.780

Grouped by Stem (with stop words)All Students 0.010 0.967 0.160 0.350Low Pretest 0.107 0.727 0.240 0.340High Pretest 0.465 0.293 0.080 0.760

30

Results: Interactivity (2003)

No evidence that cohesion between student productions is correlated with learning (but student utterances are very short with computer tutor)

Student Turns OnlyTrain: 2003 Data Test: 2005 Data

Students R P-Value R P-Value

Grouped by Token (with stop words)All Students 0.242 0.304 0.070 0.695Low Pretest 0.325 0.279 0.121 0.634High Pretest 0.451 0.310 0.076 0.779

Grouped by Token (Stop words removed)All Students 0.111 0.640 0.191 0.280Low Pretest 0.010 0.974 0.248 0.322High Pretest 0.493 0.261 0.214 0.426

Grouped by Stem (with stop words)All Students 0.113 0.637 0.187 0.291Low Pretest 0.009 0.976 0.253 0.311High Pretest 0.501 0.252 0.199 0.460

31

Discussion Both high and low pre-testers successfully

learned from these dialogs Our measure of lexical cohesion seems to

reflect only what the low pre-testers do to learn, not correlated with what high pre-testers do.

McNamara & Kintsch also found a positive correlation for low pre-testers, but a negative correlation for high pre-testers.

32

Discussion Our measures are slightly different:

McNamara & Kintsch: Manipulated coherence in text Reader does not contribute to coherence Coherence is the extent to which semantic relations are spelled

out in the text, rather than inferred by the reader. Low pre-testers probably learned because high coherence text

allowed them to make inferences they couldn’t from the low cohesion text.

Low pre-testers & low coherence: didn’t know the terms High coherence may allow a greater number of successful

inferences for their low pre-testers Our work: Dialog

Student does contribute to cohesion Higher cohesion means using more of same terms Speculation: High cohesion may indicate the number of

successful inferences our low pre-testers already made. High pre-testers already know the terms, so new inferences are

not involved in using them.

33

Summary

We have taken automatically computable measures of cohesion from computational linguistics Applied them to tutorial dialog Found correlations with student

learning

34

Conclusions Simple, automatically computable

measures of lexical cohesion correlate with learning But only for students with low pre-test scores,

even though low and high pre-testers showed similar amounts of cohesion.

Correlation is robust to differences in type of measurement

It’s the cohesion between student and tutor that’s important

35

Future Work Short term:

Cohesion may also be related with learning in high pre-testers, but we’re measuring the wrong kind of cohesion

Work underway to try “sense” level measures Halliday & Hassan’s “synonym” levels of reiteration “Acceleration” & “speeding up”

New issues: Word sense disambiguation (one sense per discourse?)

Or measuring it in the wrong places Try finding cohesion at impasses (VanLehn 03) Try finding change in cohesion over time (Pickering &

Garrod 04) Is it the dialog, or the essay?

Long term: Test by manipulating cohesion in ITSPOKE

36

Thanks

Diane Litman ITSPOKE group

37

Questions?

38

03speech93_sess00_prob55 Low cohesion Tie-count = .419tutor Enter your response to the question at the top right, in the essay window. When you are finished hit the submit

essay

'when', 'at', 'ar', 'hit',

" If the airplane drops the packet when it is directly above the swimming pool, it will not hit the pool. The packet and the plane are moving at the same horizontal velocity, but then the packet is dropped from the plane. The packet's horizontal velocity slowly decreases and the vertical velocity increases. The packet passes the pool

tutor 'ar', 'packet',

You have the correct answer except that you need to go into more depth in your essay. I'll try to help you remember some details to include in the explanation. After the packet is released, what are the forces acting on

student gravity air resistance

tutor 'air',

Almost. Let's consider the kinds of forces that air can exert on an object traveling through it. There are two. a frictional force and a buoyancy force. Sometimes they are so small compared to the gravitational and other forces present that they can be ignored, and sometimes they are so large that they cannot be ignored. For each one, let's discuss when they can be ignored. Let's consider first the frictional force. If you stick your hand out the

student strong wind

tutor 'wind',

You feel the wind blowing your hand backwards, hard. That force is the frictional force of the air that we are talking about. The air exerts a frictional force on your hand. Now if the car moves at about the same speed as you would walk, what would you feel on your hand when you stick it out the window?

student nothing

tutor

Right. So when the relative speed of the object and the air is large, the air exerts a force on the object. When it is small compared to the other forces present, it can be neglected. Thus the speed of the object through the air is one factor. There are others. Suppose you have a large umbrella open above you, and the air is still. You pull

student no

tutor

Well... If you really jerk down hard on the umbrella, it will resist so much that it might turn inside out. You can really feel the force then! But even if you pull down gently, you can feel some force resisting your pull. The resistance you feel is caused by the air resisting the passage of the open umbrella through it. Although the umbrella is not moving fast, it presents a large surface area to the air along the direction of motion. This makes

student 'larg', large

tutor Correct. The larger the surface area presented to the air, the larger the resistive force. So when an object moving through the air presents a small surface area and it is not moving very fast, how big is the force exerted by the air

student 'small', smalltutor Right. Can we ignore the force that the air exerts on a baseball that is dropped from the roof of a one-story student yestutor Fine. Can we ignore the force that the air exerts on a large truck barreling down the freeway?student no

tutor Correct. You seem to understand the frictional force that air can exert on objects. Let's discuss the other force. When a boat floats in water, what is the name of the force that keeps it from sinking? ~~ When a boat floats in

39

student i don't know

tutor The buoyant force of the water on the boat keeps it from sinking. Air can exert a buoyancy force, too. For instance, when a helium-filled balloon rises, the force acting on it is what?

student 'forc', field force

tutor 'forc',

The buoyant force of the air on the balloon lifts it up. An object, such as a piece of stone, sinks in water but a piece of cork floats. However, even stones float in liquid mercury, because mercury is denser than a piece of stone. So whether something sinks or floats depends on the density of the object (taken as a whole including any voids etc in it) RELATIVE to the medium it is in. How dense is air as compared to the objects you come across?

student 'dens', less densetutor 'dens', Correct. Because air is not at all dense, what kinds of objects can float in air?student cloudstutor Could you please repeat that?student cloudstutor I'm sorry, I'm having trouble understanding you. Please try again.student balloons

tutor

Well... Only objects that are less dense than air will float. That is, the buoyant force acting upward is larger than the gravitational force acting downward. Since many objects that move through air (e.g., balls, rocks, pumpkins, cars, etc.) are much denser than air, would you consider the buoyancy force exerted on them by the air to be

student ratetutor Could you please repeat that?student ignore it

tutor 'ignor',

Good. To summarize, we can ignore the forces that air exerts on an object if the object is dense, it is moving slowly relative to the air, and it presents a small surface area in the direction of motion. Air resistance is negligible. Earth's gravity is the only force on the packet after its release. Air resistance is negligible. In which direction does

student downwardtutor Could you please repeat that?student downwardtutor I'm sorry, I'm having trouble understanding you. Please try again.student down

tutor The earth's gravitational force acts vertically downward. I think this is an important point to include. Try to fix it. If you're finished, press the submit button.

essay

'forc', 'vertic', 'earth',

" If the airplane drops the packet when it is directly above the swimming pool, it will not hit the pool. The packet and the plane are moving at the same horizontal velocity, but then the packet is dropped from the plane. We can ignore the forces that air exterts on the packet because it is dense, and presents a small surface area in the direction of motion. The packet's horizontal velocity slowly decreases and the vertical velocity increases. The packet passes the pool because it still has a horizontal velocity when it leaves the plane. It moves further than the

tutor That all looks good. We're done with this problem. For comparison purposes, I'm going to show you one ideal

40

03speech88_sess00_prob58 High Cohesion Tie-count = 2.40

tutor Enter your response to the question at the top right, in the essay window. When you are finished hit the submit button and we'll discuss it.

essay 'at' " The man's keys will be above his face. This will occur because the elevator will move at a faster rate than he will. Since the man is falling at the same rate as the elevator, he is also moving faster than his keys. "

tutor 'kei', 'man' You have answered the problem correctly except that what you have in your essay is not a sufficient explanation. I'll give you some hints about what is needed in the explanation. Before the release, what is the relationship

student 'kei', 'veloc', 'man' the man's velocity is faster than the keys velocity

tutor 'kei', 'veloc', 'man' Considering that the man is holding the keys before the release, what is the relationship between the man's velocity and the keys velocity?

student 'kei', 'veloc', 'man' the man's velocity is larger the the keys velocity

tutor 'kei', 'veloc', 'man' They must be the same. So, back to the original question. Before the release, what is the relationship between the man's velocity and the keys velocity?

student 'thei', 'same' they are the same

tutor Good. What we've just covered is an important point to include in your essay. You should fix your essay now. Press the submit button when you're done.

essay

" Before the man releases the keys, the man's velocity is the same as the keys' velocity. After the man releases the keys, they will be above his face. This will occur because the elevator will move at a faster rate than he will. Since the man is falling at the same rate as the elevator, he is also moving faster than his keys. "

tutor Yeah. There are additional details to include in your essay. I'll remind you of what is needed in the explanation. What force acts on all objects within close proximity to earth?

student 'forc' the force of gravitytutor 'forc' Yep. Besides the Earth's gravitational force, are there any other forces acting on the person, keys, and elevator student 'forc' the normal force

tutor 'forc'

When the elevator is in freefall, there are no other forces acting on it, the man, or the keys. There is no air resistance on the man and the keys because the air in the elevator is moving right along with the man and the keys. There is no air resistance on the elevator because it is in freefall, which is defined to mean that air resistance is negligible. Thus, keep in mind that after the release, the only force on the person, keys, and elevator is the force of gravity. I think this is an important point to include. Please fix your essay now. If you're finished,

41

essay

'move', 'becaus', 'kei', 'releas', 'act', 'there', 'forc', 'ar', 'no', 'other', 'graviti', 'after', 'man', 'elev'

" Before the man releases the keys, the man's velocity is the same as the keys' velocity. After the man releases the keys, the force of gravity acts upon the keys. Since there is no other force acting on the keys they will be above his face after they are released. This will occur because the elevator will move at a faster rate than he will. Since the man is falling at the same rate as the elevator, he is also moving faster than his keys. "

tutor

'kei', 'releas', 'there', 'will', 'ar', 'after', 'upon', 'elev'

Yes. There is one detail we need to discuss. Maybe this will help you remember some of the details need in the explanation. We know that after the release, the keys, the person, and the elevator are in freefall. Is freefall acceleration near earth a constant? Or does it vary depending upon which object is in freefall?

student

'depend', 'freefal', 'object', 'which', 'upon' it depends upon which object is in freefall

tutor 'freefal' Well... It is constant. Freefall acceleration is another name for acceleration due to what?student gravitytutor Good. What is the earth's gravitational acceleration?student nine point eight meters per second squared

tutor Yep. Thus, freefall acceleration is the same for all objects, and the keys, the person, and the elevator are all accelerating downwards with the same acceleration. Try to add something about this to your essay. Try to fix it.

essay

'all', 'freefal', 'object', 'kei', 'acceler', 'ar', 'same', 'elev'

" The elevator, man, and his keys are all in free-fall acceleration. Free-fall acceleration means that only the force of gravity is acting on an object. The force of gravity on Earth is 9.8 meters per second squared. The force of gravity is the only force acting on the elevator, keys, and the man, they are all falling at a rate of 9.8 meters per second squared. Before the man releases the keys, the man's velocity is the same as the keys' velocity. After the man releases the keys, the force of gravity acts seperatly upon the keys. The man will have a larger mass than the keys, so he will fall at a faster velocity than the keys. Since there is no force other than the force of gravity acting on the keys they will be above his face after they are released. This will occur because the elevator will

tutor 'as', 'have', 'so' Good. You need some additional details in your essay. We'll count the problem as done. Although this would need to be rewritten to fit in your essay, so you can compare, here's what I would have said:

42

Cohesion vs Coherence Cohesive Devices

Things that “tie” different parts of a discourse together:

Anaphora, repetition, etc… But still may not make sense:

John hid Bill’s car keys. He likes spinach. (Jurafsky & Martin 00)

Coherence relations Semantic relations between utterances.

Result, Explanation, elaboration, etc. (Hobbs 79)

43

Britton & Gulgoz 91 Original text: Air war in the North, 1965By the fall of 1964, Americans in both Saigon and

Washington had begun to focus on Hanoi as the source of the continuing problem in the south.

Modified text:Air war in North Vietnam, 1965By the beginning of 1965, Americans in both

Saigon and Washington had begun to focus on Hanoi, capital of North Vietnam, as the source of the continuing problems in the south.

Date post:	01-Apr-2015
Category:	Documents
Upload:	muhammad-hubert
View:	221 times
Download:	3 times

1 Cohesion and Learning in a Tutorial Spoken Dialog System Art Ward Diane Litman.

Documents