Longitudinal Analysis of Effects of Reclassification, Reporting Methods , and Analytical Techniques on Trends in Math Performance of Students with Disabilities. Yi-Chen Wu, Martha Thurlow , & S heryl Lazarus National Center on Educational Outcomes University of Minnesota. - PowerPoint PPT Presentation
Longitudinal Analysis of Effects of Reclassification, Reporting Methods, and Analytical Techniques on Trends in Math Performance of Students with Disabilities Yi-Chen Wu, Martha Thurlow, & Sheryl Lazarus National Center on Educational Outcomes University of Minnesota This paper was developed, in part, with support from the U.S. Department of Education, Office of Special Education Programs grants (#H373X070021and #H326G110002). Opinions expressed herein do not necessarily reflect those of the U.S. Department of Education or Offices within
Transcript
Longitudinal Analysis of Effects of Reclassification, Reporting Methods, and Analytical Techniques on Trends
in Math Performance of Students with Disabilities
Yi-Chen Wu, Martha Thurlow, & Sheryl LazarusNational Center on Educational Outcomes
University of Minnesota
This paper was developed, in part, with support from the U.S. Department of Education, Office of Special Education Programs grants (#H373X070021and #H326G110002). Opinions expressed herein do not necessarily reflect those of the U.S. Department of Education or Offices within it.
NCEO Web site(http://www.cehd.umn.edu/nceo/)
Outline
BackgroundAchievement gapExplanationsYsseldyke and Bielinski (2002) study
QuestionsMethod
Data sourceAnalytical Techniques
ResultsConclusions
Achievement gap
4
Focused on race/ethnicity or poverty. Less attention on achievement gaps between SPED vs. Non-SPED
Research on Achievement Gap (Chudowsky, Chudowsky, & Kober, 2009a; 2009b)Examined gaps for subgroups by proficiency rate &
mean SS, but no comparison between SPED and Non-SPED
Examined the achievement over time for SWD, but not the gap between SWD vs. SWOD over time
Explanations on gap increasing over time between SWD and SWOD
5
SPED drop out of school=high achievement (McMillen & Kaufman, 1997)
Tests given in higher grades are less valid for SWD (Thurlow & Ysseldyke, 1999; Thurlow, Bielinski, Minnema, & Scott, 2002)
Students with lower performance moved in SPED and students with higher performance move out SPED (Ysseldyke and Bielinski, 2002)
Ysseldyke and Bielinski (2002) study Explored the extent to which reclassification impacts the size of the achievement gap between GED and SPED across grades.
to compare the effects of different reporting methods, and to examine the effects of reclassification
They argued that fair comparisons involved using clearly defined and consistent comparison groups, and that special education status complicates the reporting because status changes over time.
Ysseldyke and Bielinski (2002) study They used three methods to analyze trends in performance (cross-sectional, cohort-static and cohort-dynamic), and found that gap trends depended on the method used
examined how the use of scaled scores and effect size could be used for reporting results.
Purpose
The Ysseldyke and Bielinski (2002) study did not use proficiency to examine the reporting results is now more than a decade old was completed prior to the implementation of ESEA
2001. There is a need to take a new look at how achievement gap trends are affected by the method used to calculate them.
Research Questions
Reporting Methods: How does the use of cross-sectional, cohort-static, and cohort-dynamic data analysis methods affect interpretation of trends in the performance of students with disabilities?
Analytical Techniques: How does the score used in the analyses (proficiency level, scaled score, effect size) affect interpretation of trends and achievement gaps?
Reclassification: To what extent do students move in and out of special education each year, and what are the achievement characteristics of those who do and do not move?
Method
Data sourceused math assessment data for grades 3-8 from a
midwestern stateCross-sectional
2005-06 to 2009-10 305,819 records
Cohort2005-06 to 2009-10+ 2004-05 (G3-8)8,231 students with 6-yr records
Method- Methods Used to Measure Gap
Cross-sectional five years of data were used to calculate the average
performance to reduce year-to-year variations that might affect results if data from a single year were selected.
Cohort-staticA cohort across six yearsGroup membership stayed the same across years.
Cohort-dynamicgroup membership was redefined every year
Method- Analytical Techniques
Results—RQ1
How does the use of cross-sectional, cohort-static, and cohort-dynamic data analysis methods affect interpretation of trends in the performance of students with disabilities?Using PF to show the trend over time among the three
methods used to measure gap
Figure 1. Cross-sectional method: Percentage of students above proficiency level on math assessment by SPED and non-SPED
80.2 81.3 79.4 78.9 76.1 76.7
59.154.0
47.4
38.430.0 29.4
0
10
20
30
40
50
60
70
80
90
100
3 4 5 6 7 8
Perc
enta
ge o
f Stu
dent
s abo
ve P
rofic
ienc
y Le
vel
Grade
Non-SPED SPED
Results—Comparing reporting methods
21-->47
Results—Comparing reporting methods
Figure 2: Cohort-static method: Percentage of students above proficiency level on math assessment by SPED and Non-SPED
82.1 84.3 83.2 81.8 81.0 83.3
60.1 61.6 60.556.6 54.7
61.6
0
10
20
30
40
50
60
70
80
90
100
3 4 5 6 7 8
Perc
enta
ge o
f Stu
dent
s abo
ve P
rofic
ienc
y Le
vel
Grade
Non-SPED SPED
22->21
Results—Comparing reporting methods
Figure 3. Cohort-dynamic method: The percentage of students above proficiency level on math assessment by SPED and Non-SPED
82.1 84.6 83.6 81.9 81.6 84.2
60.156.3
52.645.6
34.638.8
0
10
20
30
40
50
60
70
80
90
100
3 4 5 6 7 8
Perc
enta
ge o
f Stu
dent
s abo
ve P
rofic
ienc
y Le
vel
Grade
Non-SPED SPED
22-->45
Results—Comparing reporting methods
Quit differentQuite similarSteady
80.2 81.3 79.4 78.9 76.1 76.7
59.154.0
47.4
38.430.0 29.4
0
10
20
30
40
50
60
70
80
90
100
3 4 5 6 7 8
Perc
enta
ge o
f Stu
dent
s abo
ve P
rofic
ienc
y Le
vel
Grade
Non-SPED SPED
82.1 84.3 83.2 81.8 81.0 83.3
60.1 61.6 60.556.6 54.7
61.6
0
10
20
30
40
50
60
70
80
90
100
3 4 5 6 7 8
Perc
enta
ge o
f Stu
dent
s abo
ve P
rofic
ienc
y Le
vel
Grade
Non-SPED SPED
82.1 84.6 83.6 81.9 81.6 84.2
60.156.3
52.645.6
34.638.8
0
10
20
30
40
50
60
70
80
90
100
3 4 5 6 7 8
Perc
enta
ge o
f Stu
dent
s abo
ve P
rofic
ienc
y Le
vel
Grade
Non-SPED SPED
Cohort-dynamic Cohort-static
Cross-sectional
Results—RQ2
How does the score used in the analyses (proficiency rate, scaled score, effect size) affect interpretation of trends and achievement gaps?
Figure 4. Percent proficient: Achievement gap (difference between non-SPED and SPED) in percent proficient on math assessment
Results—Comparing Analytical Techniques
21.1
27.3
32.0
40.646.2
47.4
22.0
28.4 31.0
36.3
47.0
45.4
22.6 22.725.1 26.2
21.7
0
10
20
30
40
50
60
3 4 5 6 7 8Perc
enta
ge o
f Stu
dent
s abo
ve P
rofic
ienc
y Le
vel
Grade
Cross Sectional
Cohort Dynamic
Cohort Static
Figure 5. Scaled score: Achievement gap (difference between non-SPED and SPED) in mean scaled score on math assessment
Results—Comparing Analytical Techniques
22.7 26.0
28.5
38.8
43.6
40.5
26.8
26.0
35.4
41.8
20.9 22.020.2
23.424.9
21.3
0
5
10
15
20
25
30
35
40
45
50
3 4 5 6 7 8
Scal
e Sc
ore
Grade
Cross Sectional Cohort Dynamic
Cohort Static
Figure 6. Effect size: Achievement gap (difference between non-SPED and SPED) in effect size on math assessment
Results—Comparing Analytical Techniques
-.63 -.60 -.62 -.65-.55-.51
-.68-.74
-.96-1.00 -.99
-.57
-.77 -.76 -.94
-1.14 -1.10-1.2
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
3 4 5 6 7 8
Effec
t Size
Grade
Cohort-Static Cross-SectionalCohort-Dynamic
Results—Comparing analytical techniques
Quit differentQuite similarSteady
Effect sizeScaled Score
Proficiency Level
21.1
27.3
32.0
40.646.2
47.4
22.0
28.4 31.0
36.3
47.0
45.4
22.6 22.725.1 26.2
21.7
0
10
20
30
40
50
60
3 4 5 6 7 8Perc
enta
ge o
f Stu
dent
s abo
ve P
rofic
ienc
y Le
vel
Grade
Cross Sectional
Cohort Dynamic
Cohort Static
22.7 26.0
28.5
38.8
43.6
40.5
26.8
26.0
35.4
41.8
20.9 22.020.2
23.424.9
21.3
0
5
10
15
20
25
30
35
40
45
50
3 4 5 6 7 8
Scal
e Sc
ore
Grade
Cross Sectional Cohort Dynamic
Cohort Static
-.63 -.60 -.62 -.65-.55-.51
-.68-.74
-.96-1.00 -.99
-.57
-.77 -.76 -.94
-1.14 -1.10-1.2
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
3 4 5 6 7 8
Effec
t Size
Grade
Cohort-Static Cross-SectionalCohort-Dynamic
Results—RQ3
To what extent do students move in and out of special education each year, and what are the achievement characteristics of those who do and do not move?
Figure 7. Mean math scaled scores by special education status across years
Note: NS1 = Students who remained in non-special education in both of two consecutive years; NS2 = Students who moved from non-special education to special education in the second of two consecutive years; S1 = Students who remained in special education in both of two consecutive years; S2 = Students who moved from special education to non-special education in the second of two consecutive years.
Results—Reclassification Non-SPED only
Students stayed in non-SPED for six years Non-SPED to SPED
Students moved from non-SPED to SPED only once over six years
SPED to Non-SPED Students moved from SPED to non-SPED only once over six
years Back and forth
Students moved between SPED and non-SPED more than once over six years
SPED only Students stayed in SPED for six years
Figure 8. The effect size between different reclassification groups in math assessment by using non-SPED only group as the reference group
GradeSPED to Non-SPED (N=730) Back and Forth (N=180)Non-SPED to SPED (N=251) SPED only (N=434)
Discussion and Conclusion
Different methods of reporting data present different pictures of the gap between SPED and non-SPED
This study was undertaken to update the work done more than a decade ago by Ysseldyke and Bielinski (2002)Replicated + proficiency levelConfirmed Suggestions
Discussion and Conclusion
SuggestionsThe choice of method affects what the results look like and
the possible interpretation of findings.Tracking individual student performance provides a better
indication of how well schools are educating their students than cross-sectional models where the grade remains the constant but the students change.
Cross-sectional models should not be used when examining trends across grades.
Cohort-static and cohort-dynamic methods enable educators to make comparisons among individual students
Discussion and Conclusion
Specific situation for each reporting method If the goal is to know how well students do yearly without
considering changing students => cross-sectionalhttp://www.schooldigger.com/go/MN/schools/
3243001386/school.aspx If states and districts want to account with precision for the
reclassification of students each year. => cohort-dynamicWhen the goal is to account for individual student
performance over time without regard to the nature of services received=> cohort-static
