An Algorithm for Assessing Student's Reading Levels

Mark Brierley

Kosuke Adachi

Shinshu University

What’s an algorithm?

• a set of rules for solving a problem in a finite number of steps– variant of algorism, (from arabic al Kuwarismi)

Why do we need one?

• Recommending books for students to read

• Making an adaptive reading text

What do we know?

• Student ratings (R)

• Reading times (T)

• Comprehension (C)

• Effective reading speed/Comprehensible WPM (E) = WPM x Comprehension

The data

• Online ER system.– 12,575 reviews by – 937 students on– 634 graded readers

• ERFOPT pre-beta trials– 512 attempts by– 115 students on– 21 texts and questions

The nature of the data

• Stringy– different students read different texts

• Unreliable– opinions, emotions, characters…

(welcome to the real world!)

• Even a fuzzy string vest has a definite shape!

Assumptions

• Each text has a level of difficulty– Harry Potter is easier than Da Vinci Code– Noam Chomsky is more difficult than Bill

Bryson

• Each reader has a reading ability– I’m not very good at reading

What level are the texts?

• Nominal level (N)– How did the students do?– If that was the students’ level, what level were

the texts?– repeat...

P1 Levels

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

11.0

12.0

13.0

14.0

1 2 3 4 5 6 7 8 9 10 11

comp

time

rating

comp wpm

C5 Levels

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

11.0

12.0

13.0

14.0

1 2 3 4 5 6 7 8 9 10 11

comp

time

rating

comp wpm

WK Level

5.0

6.0

7.0

8.0

9.0

10.0

11.0

12.0

13.0

14.0

1 2 3 4 5 6 7 8 9 10

comp

time

rating

comp wpm

text code

number of attempts (n)

nominal level

comp. estimate (c )

impress. estimate (i)

time estimate (t)

effective speed estimate (w)

average variation (c t i)

average variation from nominal

words questions

O2 12 8 6.3 9.1 6.0 5.9 1.0 0.9 551 8

O4 13 8 6.5 4.5 6.4 7.1 0.7 1.1 387 9

E1 14 5 6.8 4.9 5.4 6.1 0.6 0.5 500 5

O6 17 10 7.3 12.0 5.6 6.2 2.0 1.7 647 9

O3 21 8 6.7 11.5 6.4 7.7 1.7 1.0 699 9

C5 24 10 6.1 13.5 6.2 5.0 2.6 2.0 664 7

O7 24 11 7.0 10.4 6.0 6.7 1.4 1.9 666 9

C3 26 5 6.2 4.9 6.0 6.5 0.4 0.5 496 9

P2 33 6 6.8 6.0 7.2 7.1 0.4 0.7 529 9

C4 33 7 6.1 5.3 6.4 6.7 0.3 0.5 514 11

C1 39 4 5.6 4.6 6.3 6.0 0.5 0.9 314 10

C2 39 5 6.3 6.1 5.7 6.2 0.2 0.5 518 10

E4 41 3 5.7 4.6 6.4 6.4 0.6 1.4 374 7

E2 43 4 5.6 4.8 4.9 4.7 0.3 0.5 500 5

O1 46 7 6.5 5.6 6.2 6.9 0.3 0.4 629 9

P1 55 6 7.3 4.6 6.1 7.7 0.8 0.6 348 8

average 479 5.5 5.6 5.9 5.4 5.6 0.6 0.8 463.0 7.5

Correlations between tests

Nom. Comp. Imp. Time Eff.Var. (c t i)

Var from nom.

Ww Qq

Nominal 100% 65% 81% 9% 16% 76% 64% 74% 23%

Comp. 65% 100% 48% 8% 50% 40% 26% 60% 17%

Imp. 81% 48% 100% -1% -7% 94% 73% 80% 3%

Time 9% 8% -1% 100% 65% -1% 7% -11% 54%

Eff. 16% 50% -7% 65% 100% -13% -15% 12% 57%

0.05 16.00 49%(pink cells: Significant correlations)

Significant correlations within texts

16 textsComp. Imp. Time Eff.

Comp.= 2 0 11

Imp.2 = 1 4

Time0 1 = 16

Eff. 11 4 16 =

Student level correlations

  Time Comp. Imp. Eff. EPER E  YL

Time 100% 7% 9% 52% 25% 8%

Comp. 7% 100% 53% 53% 39% 33%

Imp. 9% 53% 100% -13% 37% 46%

Effective 52% 53% -13% 100% 33% 7%

sig. 0.05 106 19%    

sig. 0.05 64 25%

What about the algorithm?

• What text should we show the student next?

• What level is the student at?

• What level is a new text?

The algorithm

How easy was it?

• 1- Very difficult• 2-• 3-• 4-• 5- Very Easy

• What does this tell us?

• S < Book

• Depends on student...

• S >= Book

Can they read it?

• Logic– If T < Tm ... AND C > Cm – If T < Tm ... OR C > Cm

• Score– (C/Cmax) x (W/T)

Can they read it?

Speed

Comprehension

Fast Average Slow

High

Average

Low

Yes

Probably

?

Maybe not

No!

0

1

2

3

4

5

6

7

8

9

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

0

1

2

3

4

5

6

7

8

9

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

0

1

2

3

4

5

6

7

8

9

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

0

1

2

3

4

5

6

7

8

9

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

Questions

• What do we trust? – Speed– Comprehension– Effective speed– Student impressions

• Logical or arithmetic

Conclusions

• Publisher level ≠ Performance

• Performance ≠ impression of difficulty

• Publisher level ≈ impression of difficulty

Reading ability multidimensional

• Speed

• Comprehension

• Self-image?

• Reading strategies

• Imagination

• World knowledge– Independent variables?

Strategy

• Keep independent data for each text

• Calculate several levels for each student

• Take an average to estimate their level

Thank you!

• mark2@shinshu-u.ac.jp