An Algorithm for Assessing Student's Reading Levels Mark Brierley Kosuke Adachi Shinshu University.

transcript

An Algorithm for Assessing Student's Reading Levels

Mark Brierley

Kosuke Adachi

Shinshu University

What’s an algorithm?

• a set of rules for solving a problem in a finite number of steps– variant of algorism, (from arabic al Kuwarismi)

Why do we need one?

• Recommending books for students to read

• Making an adaptive reading text

What do we know?

• Student ratings (R)

• Reading times (T)

• Comprehension (C)

• Effective reading speed/Comprehensible WPM (E) = WPM x Comprehension

The data

• Online ER system.– 12,575 reviews by – 937 students on– 634 graded readers

• ERFOPT pre-beta trials– 512 attempts by– 115 students on– 21 texts and questions

The nature of the data

• Stringy– different students read different texts

• Unreliable– opinions, emotions, characters…

(welcome to the real world!)

• Even a fuzzy string vest has a definite shape!

Assumptions

• Each text has a level of difficulty– Harry Potter is easier than Da Vinci Code– Noam Chomsky is more difficult than Bill

Bryson

• Each reader has a reading ability– I’m not very good at reading

What level are the texts?

• Nominal level (N)– How did the students do?– If that was the students’ level, what level were

the texts?– repeat...

P1 Levels

1 2 3 4 5 6 7 8 9 10 11

rating

comp wpm

C5 Levels

1 2 3 4 5 6 7 8 9 10 11

rating

comp wpm

WK Level

1 2 3 4 5 6 7 8 9 10

rating

comp wpm

text code

number of attempts (n)

nominal level

comp. estimate (c )

impress. estimate (i)

time estimate (t)

effective speed estimate (w)

average variation (c t i)

average variation from nominal

words questions

O2 12 8 6.3 9.1 6.0 5.9 1.0 0.9 551 8

O4 13 8 6.5 4.5 6.4 7.1 0.7 1.1 387 9

E1 14 5 6.8 4.9 5.4 6.1 0.6 0.5 500 5

O6 17 10 7.3 12.0 5.6 6.2 2.0 1.7 647 9

O3 21 8 6.7 11.5 6.4 7.7 1.7 1.0 699 9

C5 24 10 6.1 13.5 6.2 5.0 2.6 2.0 664 7

O7 24 11 7.0 10.4 6.0 6.7 1.4 1.9 666 9

C3 26 5 6.2 4.9 6.0 6.5 0.4 0.5 496 9

P2 33 6 6.8 6.0 7.2 7.1 0.4 0.7 529 9

C4 33 7 6.1 5.3 6.4 6.7 0.3 0.5 514 11

C1 39 4 5.6 4.6 6.3 6.0 0.5 0.9 314 10

C2 39 5 6.3 6.1 5.7 6.2 0.2 0.5 518 10

E4 41 3 5.7 4.6 6.4 6.4 0.6 1.4 374 7

E2 43 4 5.6 4.8 4.9 4.7 0.3 0.5 500 5

O1 46 7 6.5 5.6 6.2 6.9 0.3 0.4 629 9

P1 55 6 7.3 4.6 6.1 7.7 0.8 0.6 348 8

average 479 5.5 5.6 5.9 5.4 5.6 0.6 0.8 463.0 7.5

Correlations between tests

Nom. Comp. Imp. Time Eff.Var. (c t i)

Var from nom.

Nominal 100% 65% 81% 9% 16% 76% 64% 74% 23%

Comp. 65% 100% 48% 8% 50% 40% 26% 60% 17%

Imp. 81% 48% 100% -1% -7% 94% 73% 80% 3%

Time 9% 8% -1% 100% 65% -1% 7% -11% 54%

Eff. 16% 50% -7% 65% 100% -13% -15% 12% 57%

0.05 16.00 49%(pink cells: Significant correlations)

Significant correlations within texts

16 textsComp. Imp. Time Eff.

Comp.= 2 0 11

Imp.2 = 1 4

Time0 1 = 16

Eff. 11 4 16 =

Student level correlations

Time Comp. Imp. Eff. EPER E YL

Time 100% 7% 9% 52% 25% 8%

Comp. 7% 100% 53% 53% 39% 33%

Imp. 9% 53% 100% -13% 37% 46%

Effective 52% 53% -13% 100% 33% 7%

sig. 0.05 106 19%

sig. 0.05 64 25%

What about the algorithm?

• What text should we show the student next?

• What level is the student at?

• What level is a new text?

The algorithm

How easy was it?

• 1- Very difficult• 2-• 3-• 4-• 5- Very Easy

• What does this tell us?

• S < Book

• Depends on student...

• S >= Book

Can they read it?

• Logic– If T < Tm ... AND C > Cm – If T < Tm ... OR C > Cm

• Score– (C/Cmax) x (W/T)

Can they read it?

Comprehension

Fast Average Slow

Average

Probably

Maybe not

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

Questions

• What do we trust? – Speed– Comprehension– Effective speed– Student impressions

• Logical or arithmetic

Conclusions

• Publisher level ≠ Performance

• Performance ≠ impression of difficulty

• Publisher level ≈ impression of difficulty

Reading ability multidimensional

• Speed

• Comprehension

• Self-image?

• Reading strategies

• Imagination

• World knowledge– Independent variables?

Strategy

• Keep independent data for each text

• Calculate several levels for each student

• Take an average to estimate their level

Thank you!

• mark2@shinshu-u.ac.jp

An Algorithm for Assessing Student's Reading Levels Mark Brierley Kosuke Adachi Shinshu University.

Documents