Post on 19-Jan-2016
transcript
Designing Accessible Reading Assessments
Putting Accessibility to the Test:Design of a Multistage Reading Assessment for Accountability
Cara Cahalan LaitusisETS
Designing Accessible Reading Assessments
ETS Contributors
• Linda Cook*
• Kelly Bruce
• Jennifer Dean
• Dan Eignor
• Lois Frankel
• Gena Gourley
• Eric Hansen
• Branden Hart• Teresa King*• Skip Livingston• Pavan Pillarisetti• Kitty Sheehan• Elizabeth Stone*• Klaus Zechner
Designing Accessible Reading Assessments
DARA Goal 4
• Field test a multi-stage component-based reading assessment.– Reduce number of students performing at
“chance level” – Allow students to show what they know– Push instructional to include both
comprehension and reading fluently for students with reading-based LD
Designing Accessible Reading Assessments
DARA Test Design
Designing Accessible Reading Assessments
Data Collection Design
Designing Accessible Reading Assessments
Primary Research Questions• For accountability purposes, is it possible to
combine scores from the two different routes on the component test (i.e., average scores from Test 1 and Test 2)?
• Is the Component test more accessible than the state assessment
– Do RLD students do better on the Component test than the state assessment while students without disabilities (NLD) perform similarly on both assessments?
Designing Accessible Reading Assessments
Other Research Questions• Can we reduce the number of students scoring at
chance level?• Can we use automated scoring technology
(SpeechRater) to score oral reading fluency measure?
• Can we accurately route students based on 7, 14, 21, and 28 items?
• What is the best measure of oral reading fluency? • How do we combine fluency and comprehension
test scores (50/50, 25/75, 75/25)?
Designing Accessible Reading Assessments
Sample
• 8th Grade Students
• 26 Middle Schools
• 294 RLD (final sample=275)
• 194 LP (not include in this presentation)
• 500 Non-Disabled (final sample=486)
Designing Accessible Reading Assessments
Description of Sample by NLD/RLD
• Race, Gender, and cut score impact
Group
SEX RACE
% M % F % A.I. % A % B % H % M % P.I. % W
NLD 1 46.18 53.82 0.00 2.82 2.42 8.47 2.02 3.63 80.65
NLD 2 46.84 53.16 0.42 2.97 4.24 8.05 2.97 3.81 77.54
RLD 1 63.12 36.88 0.00 1.42 4.96 22.70 6.38 2.84 61.70
RLD 2 62.69 37.31 0.75 0.75 5.22 8.96 5.22 4.48 74.63
Designing Accessible Reading Assessments
Routing decision
Best passage
(2)
Passage 1
Passages 1, 2
Passages 1, 2, 3
Full routing
(8 items; Route 1:
<= 3)
(8 items; Route 1:
<= 3)
(16 items; Route 1:
<= 6)
(24 items; Route 1: <= 10)
(32 items; Route 1: <=
13)
Reliability 0.63 0.60 0.74 0.77 0.79
% (N) students assigned to Route 1
40.73% (112)
41.45% (114)
34.55% (95)
43.64% (120)
51.27% (141)
% (N) students assigned to Route 1 on this test
and full routing test
70.21% (99)
72.34% (102)
65.96% (93)
82.98% (117)
100% (141)
Can we accurately route students based on 7, 14, 21, and 28 items?
Designing Accessible Reading Assessments
Designing Accessible Reading Assessments
Designing Accessible Reading Assessments
Test Score Summaries: Route 1
NMean(Std Dev)
Criterion(48 items)
Component Comprehension(42 items)
Component Fluency(obs max=222.75)
Component Total: Scaled(max=48)
RLD (Route 1) 14114.92(3.96)
14119.14(6.16)
14171.44(32.37)
14120.61(5.60)
NLD (Route 1) 24936.34 (7.95)
24933.71(6.12)
249145.49(30.18)
24937.34(6.27)
Designing Accessible Reading Assessments
Test Score Summaries: Route 2
NMean(Std Dev)
Criterion(48 items)
Component Comprehension(42 items)
Component Total:Scaled(max=48)
RLD (Route 2) 13427.21(6.01)
13422.66(7.43)
13425.90(8.49)
NLD (Route 2) 23734.29(8.05)
23730.65(7.41)
23735.03(8.47)
Designing Accessible Reading Assessments
Primary Research Questions• For accountability purposes, is it possible to
combine scores from the two different routes on the component test (i.e., average scores from Test 1 and Test 2)? YES
• Is the Component test more accessible than the state assessment
– Do RLD students do better on the Component test than the state assessment while students without disabilities (NLD) perform similarly on both assessments? YES, for Route 1
Designing Accessible Reading Assessments
Can we reduce the number of students scoring at chance level?
Designing Accessible Reading Assessments
Fluency TestHuman vs. Automated Scoring
ALL
Passage
1
Passage
2
Passage
3
Passage
4
N 547 126 151 148 122
Pearson r 0.68 0.81 0.71 0.60 0.76
Can we use automated scoring technology (SpeechRater) to score oral reading fluency measure?
Designing Accessible Reading Assessments
Future Questions for Study and Policy
Q: What is the best measure of oral reading fluency?
• Corrected words per minute in 1st minute• Words per minute, corrected words per minute, percent correct, rating
Q: How do we combine comprehension and fluency scores
• 25% fluency + 75% comprehension• 50/50, 75/25
Designing Accessible Reading Assessments
Extra Slides
Designing Accessible Reading Assessments
Test Score Correlations: Route 1
CriterionComponent
ComprehensionComponent
FluencyComponent
Total (Scaled)
Criterion 1.00 0.80 0.55 0.83
Component Comprehension
0.30 1.00 0.46 0.97
Component Fluency
0.27 -0.02 1.00 0.67
Component Total (Scaled)
0.38 0.94 0.31 1.00
NLD
RLD
Designing Accessible Reading Assessments
Test Score Correlations: Route 2
CriterionComponent Total
(Scaled)
Criterion
1.00 0.82
Component Total (Scaled)
0.76 1.00
NLD
RLD