Learning decomposition

WARNING

Goals

• Understand what learning decomposition is– And basic intuition

• See how it was applied to a variety of problems

• Think about how to apply it to your data

Introduction to Reading Tutor

• More free-form than most of the cognitive tutors

• Random interventions

• Kids or tutor can initiate help

• Turn taking

• Never quite sure what student is trying to do

Project LISTEN’s Reading Tutor

What is a practice opportunity?(and are they all equally valuable?)Before story, tutor teaches ‘elephant’Student sees word ‘elephant’ in sentenceStudent clicks for help on itStudent reads it‘Elephant’ occurs twice in the next sentence

• How many practice opportunities?• Did instruction have any benefit?• Did seeing word immediately afterwards help?

Procedure

• Determine (hopefully motivated) learning decompositions

• Find data that reflect learning

• Solve as a non-linear regression model– Fit model to each student

• Interpret model coefficient

Question: Does learner control result in more learning?

• In Reading Tutor, students pick story half the time Tutor picks other half

• Tutor selects stories much faster than student• Suspect motivational benefit from learner control

(willing to tolerate system over a school year)• Is there a cognitive benefit?• Compare learning of words that occur in student-

vs. tutor-selected stories.

Find data that reflect learning

• Students perform many actions• Only want those that indicate “real” learning to

count• Assumptions

– First opportunity each day is purest marker• Albert, Ken, and Joe have all observed difficulties with

closely space trials

– Don’t count stories student has already read

• Need outcome measure– Fuse accuracy, speed, and help performance

Approach

DayAsked for help?

Reading time (Sec)

Prior opportunitiesOutcome

Student selected

Tutor selected

1 Yes 0.4 0 0 3.0

1 No 0.5 1 0

1 No 0.5 2 0

2 No - 3 0 3.0

2 No 0.4 3 1

3 No 0.3 3 2 0.3

Procedure

• Determine (hopefully motivated) learning decompositions

• Find data that reflect learning

• Solve as a non-linear regression model– Fit model to each student

• Interpret model coefficient (B)

Learning curves

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 2 4 6 8 10 12 14 16 18 20

Prior number of trials

Performance (seconds)

Bet

ter

Wor

se

Performance = Ae-bt

Input: number of prior trials (t)

Output: expected performance

What if all trials aren’t equal?

• Normal model = Ae-bt

• Think about student vs. tutor chosen story– t1 = trials where student chose story– t2 = trials where tutor chose story

• Learning decomp model = Ae -b(t1+B*t2)

– B determines relative efficacy of trials of type t1 and t2

Use regression to find relative weight of tutor-selected prior opportunities

DayAsked for help?

Reading time (Sec)

Prior opportunitiesOutcome

Student selected

Tutor selected

1 Yes 0.4 0 0 3.0

1 No 0.5 1 0

1 No 0.5 2 0

2 No - 2 0 3.0

2 No 0.4 2 1

3 No 0.3 2 2 0.3

Fit model to each student’s data

Student B

Chris Smith 0.3

Pat Johnson 1.2

Sam Jackson 0.5

Jessie Stevens 0.9

Reagan Ronald 0.7

Interpret B parameter

• B is scaling parameter– B>1 students benefit from tutor control– B1 no benefit either way– B<1 student control is better

• B 0.8 for tutor-chosen stories (median)– Students learn more from student chosen

stories (not my H0)

• What could be other causes of result?

Which students benefit?

• Top-down approach: – Think of plausible subgroups– See how/if B varies among them

• E.g. 1st grader had 0.98, 2nd graders 0.89, and 3rd graders 0.49– Suggests older kids benefit more (getting

pickier?)

• Many possibilities, want to avoid fishing expedition

Which students benefit? Bottom-up approach

Student B Benefits from tutor control?

Chris Smith 0.3 No

Pat Johnson 1.2 Yes

Sam Jackson 0.5 No

Jessie Stevens 0.9 ?

Reagan Ronald 0.7 No

• Use regression results as training labels for classifier

• Predictors:– Gender– Grade– Test score (grade normed)– Disability status

• Boys benefit from learner control

Other learning decompositions: practice effects

• Open debate if more learning from rereading stories or reading new stories

• Generally believed spaced practice better for long term retention (but not short)

• Results– Reading new material better than rereading

old stories (B = 0.5)– Later practice opportunities on same day are

ineffective (B = 0.2)

Other learning decompositions: impact of instruction

• Reading Tutor has a bunch of random bits of instruction

• Do they do anything?– Solution: model instruction as an encounter

and give it a weight

• Impact of instruction (in progress)– Spelling intervention worth 0.75 exposures– Word ID intervention worth 0.36 exposure– Neither is particularly effective– (but, first analytic approach to find any effect)

Using learning decomposition to model transfer (Xiaonan Zhang)

• How do students represent words?– Naïve model: words are independent– What about “cat” vs “cats”?

• Alternate models:– Word roots (cats, cat CAT)– Rimes (bat, cat AT)

• T1 = # prior times have read word• T2 = # prior times have read root• T3 = # prior times have read rime• Substantial transfer at level of word root

– 55% as good as seeing the word itself

Hopefully

• Understand approach– Think of two types of learning that may have

unequal impact– Divide up trials– Perform curve fitting

• See that it applies to variety of problems

• But…

Concerns

• We say things like “rereading is not as effective as reading different stories”– Is it safe to make causal inference from observational

data?• Wide- vs. Re-reading: troublesome

– What if lower proficiency is true cause?• Massed vs. Distributed practice: ok (?)• Student vs. Tutor control: ok• Interventions: ok

• What about student initiated help?

Interesting view (Jack Mostow)

• Each student has a B parameter

• E.g. Chris Smith has B=0.3 for rereading Chris Smith learns 30% as much from

rereading as wide reading– Impossible for traits of Chris Smith to be a

confound (proficiency, disability, etc.)– But, states could still be a problem

• E.g. Chris only rereads after sleeping poorly

Compare LFA and Learning Decomposition

• Similar:– Use learning curves and performance data– Insight: a model that better predicts student

performance is a better model of student’s mental processes (modulo complexity)

• Different:– Bottom-up vs. top-down– Each manipulates different aspect of

representation

Bottom-up vs. Top-down

• Learning decomposition– Start with theory-driven idea– Estimate effect (if any)– No search

• LFA– Start with variety of factors– Perform search– Might not correspond to higher level construct

• Not necessarily a bad thing

Consider transfer at level of word roots

• Learning decomp:– Student exposure to words of same root is

55% as good as seeing the word• i.e. cats, cats, cat cat, cat• i.e. accepts, accepts, accept accept, accept

• LFA– Cats and cat are same skill (perfect transfer)

• i.e. cats, cats, cat > cat, cat

– Accepts and accept are different skills• i.e. accepts, accepts, accept < accept, accept

Consider transfer at level of word roots

• Learning decomp:– Student exposure to words of same root is

55% as good as seeing the word• i.e. cats, cats, cat = cat, cat• i.e. accepts, accepts, accept = accept, accept

• LFA– Cats and cat are same skill (perfect transfer)

• i.e. cats, cats, cat > cat, cat

– Achieve and achieving are different skills• i.e. accepts, accepts, accept < accept, accept

Consider transfer at level of word roots

• Learning decomp:– Student exposure to words of same root is

55% as good as seeing the word• i.e. cats, cats, cat = cat, cat• i.e. accepts, accepts, accept = accept, accept

• LFA– Cats and cat are same skill (perfect transfer)

• i.e. cats, cats, cat > cat, cat

– Achieve and achieving are different skills• i.e. accepts, accepts, accept < accept, accept

Student learning history

Skill Prior practice opportunities

Skill1 0

Skill1 1

Skill1 2

Skill2 0

Skill3 0

Skill1 3

Skill2 1

Learning factors analysis

Skill Prior practice opportunities

Skill1 0

Skill1 1

Skill1 2

Skill2 0

Skill3 0

Skill1 3

Skill2 1

Did student utilize skill1 here? Is it better to think of it as skill1’?

Learning decomposition

Skill Prior practice opportunities

Skill1 0

Skill1 1

Skill1 2

Skill2 0

Skill3 0

Skill1 3

Skill2 1

Did the student really have 3 prior practice opportunities? 1+1+1 = 3, but

is there a better way of counting?

Wrapup

• Why model individual points

• Scope of learning decomposition

• How learning decomp differs from LFA