RelationshipbetweenPhysics UnderstandingandParagraph Coherence

Reva Freedman

November 15, 2012

Outline

• What is AIED?

• Intelligent tutoring systems for physics

• Coherence: an important factor

• Database and experiment

• Results

• Conclusions

What is AIED?

• Artificial Intelligence in Education

• Breakthrough in last decade– Follows transformation in AI (and NLP) in

the previous decade

• “Science of learning”– Not just intuition any more

AIED + NLP: A good match

• Natural Language Processing– NL = what you speak

• Intelligent tutoring systems often use NL for the same reasons people do– More expressive than GUI– More flexible

Teaching/Learning Physics

• Two goals in physics teaching– Conceptual understanding– Problem solving

• Does teaching one help the other?

• Same issue in other sciences

ITSPOKE System

• Intelligent tutoring system for physics– Latest in a series

Andes/Atlas ... APE

(Python/TCP-IP C++/etc/.COM)

• Experiment– Students solve a physics problem– Then write essay about it– Then rewrite essay until tutor is satisfied

Sample Physics Problem

A person running in a straight line at a constant velocity throws a pumpkin straight up. Where does it land and why?

Sample Student Essay

1) While you are carrying the pumpkin, it has the same horizontal velocity as you do.

2) When you throw it up, there is the force of your throw which carries it vertically up, but there is no horizontal force to affect its horizontal velocity.

3) Eventually gravity will overcome the force of your throw and the pumpkin will come back down.

Sample Student Essay

4) During the pumpkin’s entire path up and back down, there is no horizontal force acting on it, so it will maintain the same constant velocity that it had when you were carrying it.

5) So the pumpkin will land on the ground right by your feet.

Coherence

• Coherence used to be hard to get hold of– “Know it when you see it”– But impossible to measure

• Barzilay and Lapata (2005, 2008)– New theory based on the new AI– Coherence is based on the relationship

between the use (e.g., pumpkin) of entities in successive sentences

Importance of Coherence

• Local coherence is the major cause of students’ understanding what they read– McKoon and Ratcliff (1992)

Data Structure: Entity Grid

s = subject o = objectx = other role - = none

Sent. #1 Sent. #2

noun 1 s -

noun 2 s x

noun 3 - o

Algorithm (Barzilay/Lapata)

• Assume human texts are more coherent than permutations of them

• Choose a permutation function

• For every permutation of each essay in the training set, compare original vs. permutation

• Induce coherence function that prefers the original human text most often

A Permutation Function

• Binary discrimination– Compare original against random

permutation– Fastest and easiest– Run multiple times and see who wins

Database

• Data– Experiment | student | problem | essay– No. of “wins” for binary test (0 – 20)– Physics pretest and posttest

• 2219 data points– Small number but big records

Methods

• Quinlan’s C4.5 algorithm– Implementation in Weka’s J48– Numeric fields considered as nominal

values, e.g., four subgroups for posttest values

Results: Null Hypothesis!

• No relationship between essay sequence number and coherence– When students rewrite their essays, they

do not become more coherent

• No relationship between posttest score and coherence– Students with better conceptual

understanding of physics do not write more coherent essays

Current Work

• Using NLP to break down data

• Relationships between linguistic features of the data and– Coherence– Physics knowledge

Wider Interests

• Science of learning without AI– What makes CS students successful?

• Computational linguistics w/o learning– Machine translation of lesser studied

languages

• Programming– Building ITSs

Acknowledgements

• Learning Research and Development Center– University of Pittsburgh

• Diane Litman– Founder and director of ITSPOKE lab

• Kate Forbes-Riley– ITSPOKE researcher

Coherence: Underlying Ideas

• Important entities are more likely to be occur in key syntactic positions such as subject or object

• They are more likely to be introduced in the main clause

• They are more likely to be referred to with pronouns in later mentions

Coherence and Entity Use

• Texts about the same entity will appear more coherent to the reader than texts with multiple topic switches

• Continuity of topic leads to consistent patterns of entity use

Permutation Functions

1) Binary discrimination– Compare original against random

permutation– Fastest – Easiest

2) Insertion– Insert one sentence, given the others– Quadratic in document length

Permutation Functions

3) Ordering– Try all permutations– Exponential in document length

• Implementation– Brown Coherence Toolkit (Elsner, 2008)

Entity Grid: Assumptions

• Noun phrases with the same head noun refer to the same entity– The bank I went to– The bank up the street

• Salience (importance) is measured by number of uses

• Assumptions can be changed