1
Children Left Behind in AYP and Non-AYP Schools: Using Student Progress and the Distribution of Student Gains to Validate AYP
Kilchan ChoiMichael Seltzer
Joan Herman Kyo Yamashiro
UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)
2
Research Questions
Are there schools that meet AYP yet still have children who are not making substantial progress? i.e., leaving some children behind?
Are there schools that do not meet AYP yet still enable students to make substantial progress?
Do AYP schools achieve a more equitable distribution of student growth? Are students at all ability levels making progress in AYP schools?
Are there non-AYP schools that are reducing the achievement gap?
3
Sample
Large, Urban District in WA
2,524 students
2 time-point ITBS reading scores (Grade 3 in 2001 & Grade 5 in 2003)
Standard Errors of Measurement (SE) on ITBS reading scores (Bryk, et.al., 1998)
72 schools
• Average # students/school: 35
• Average % qualifying for FRPL: 36.4%
• Average % Minority (African American, Native American, or Latino): 68.6%
4
AYP vs. Non-AYP schools In WA
School AYP decision made based on 4th grade performance on WA Assessment of Student Learning (WASL)
51 schools made AYP; 21 did not make AYP in baseline year (2002), according to WA State Dept of Ed
Our study re-evaluates AYP and non-AYP schools with a new value-added model (an advanced hierarchical Modeling technique)
5
A New Methodology for School Effect / Accountability: Latent Variable Regression
in Hierarchical Model
Additional Questions and Interest using LVR-HM Move beyond school mean growth rates
and examine hidden/underlying process
How equitably is student achievement distributed? (The distribution of student growth: Children Left Behind or No Child Left Behind)
Why is it that student achievement is distributed in a more equitable fashion in some schools than in other schools?
6
Distribution of Student Growth(Relationship between initial status and
rate of change)
Y Y Y t t+1 t t+1 t t+1 Figure 1. Figure 2. Figure 3. High initial low gain High initial medium gain High initial high gain Low initial high gain Low initial medium gain Low initial low gain
7
Why a New Value-Added Model (LVR-HM)?
Gains or Growth might be highly dependent upon a status at certain point of time (i.e., initial status)
Initial status can be a strong and important factor to “valued-added gain or growth”
New value-added gain or growth: Adjusting student intake characteristics PLUS
student initial difference Adjusting school intake characteristics, policies
and practice PLUS school initial difference
Thus, providing value-added gain or growth PLUS revealing the distribution of student achievement
8
Latent Variable RegressionHierarchical Model (LVR-HM)
Level 1: Time series within student
Yti = 0i + 1i Timeti + ti ti ~ N (0, 1)
Estimating initial status and gain for each student i with standard errors
Level 2: Student level
0i = 00 + r0i r0i ~ N (0, 00)
1i = 10 + b(0i - 00) + r1i r1i ~ N (0, 11)
Cov(r0i , r1i ) = 0
Gain for student i is modeled as function of his or her initial status
9
Different Levels of Initial Status
Many ways to define performance subgroups based on initial status
Examined gains for 3 performance subgroups within each school
Defined by initial status
Hi Performers: 15 pts above the school mean initial status
Mean: School mean initial status
Low Performers: 15 pts below the school mean initial status
10
Estimating Expected Gains for Different Levels of Initial Status
We estimate expected (predicted) gain for each of the performance subgroups using LVR-HM
Model-based estimation, not separate group analysis
Point estimate of gain & its 95% confidence interval (statistical inferences)
Possible to estimate expected gains after controlling for factors that lie beyond school’s control (e.g., student SES, school compositional factors)
11
Only 12 of 52 AYP schools have 95% interval above the district avg.1 AYP school’s 95% interval includes 0
Expected mean gain in ITBS reading scores for AYP schools
12
Expected mean gain in ITBS reading scores for non-AYP schools
2 Non-AYP schools have 95% interval above district avg.
13
7 AYP schools’ 95% interval 303 AYP schools’ 95% interval includes 0 (low performers make no gains)
Expected gain for low-performing students (AYP schools)
14Expected gain for low-performing students
(non-AYP schools)
5 Non-AYP schools have gains for low performers >20
15
Expected gain for high-performing students (AYP schools)
9 AYP schools’ 95% interval 303 AYP schools’ 95% interval < 10 (high performers make little or no gains)
16
Expected gain for high-performing students (non-AYP schools)
5 Non-AYP schools’ 95% interval 303 Non-AYP schools’ 95% interval < 10 (high performers make little or no gains)
17
Distribution of Gains Within A School
Type I: Substantial gain across all performance subgroups (e.g., no child left behind – ex: AYP school #8, non-AYP school #26)
Type II: No adequate gain for high performers; substantial gain for low performers (ex: AYP schools #19, non-AYP school #27)
Type III: No adequate gain for low performers; substantial gain for high performers (ex: AYP schools, non-AYP school #6 )
1815pts above the school mean Equal to the school mean 15pts below the school mean
Estimate 95% interval Estimate 95% interval Estimate 95% interval
AYP SchoolType I
Sch. #8Sch. #22 Sch. #25
37.136.337.7
( 30.7, 43.8 )( 30.0, 43.2 )( 30.8, 42.8 )
38.736.936.4
( 35.2, 42.2 )( 31.7, 42.0 )( 32.2, 40.5 )
40.337.436.0
( 35.0, 45.7 )( 31.1, 43.9 )( 31.1, 41.1 )
Type IISch. #19Sch. #63
22.128.2
( 6.9, 35.9 )( 18.2, 39.0 )
33.534.1
( 22.8, 44.1 )( 27.8, 40.3 )
44.840.0
( 30.7, 60.3 )( 30.5, 49.6 )
Type IIISch. #28Sch. #65
41.235.4
( 33.0, 51.2 )( 31.4, 39.5 )
31.231.3
( 27.4, 35.1 )( 28.5, 34.0 )
21.227.1
( 12.1, 28.7 )( 23.7, 40.4 )
Non-AYP school
Type ISch. #26
32.8 ( 24.1, 41.7 ) 32.2 ( 27.6, 36.9 ) 31.6 ( 22.5, 40.4 )
Type IISch. #27
18.7 ( 9.0, 27.2 ) 24.6 ( 17.6, 31.5 ) 30.5 ( 21.5, 40.1 )
Type IIISch. #6
Sch. #38Sch. #64
40.739.237.3
( 31.5, 50.8 )( 30.0, 49.9 )( 31.2, 44.2 )
32.429.930.6
( 26.2, 38.5 )( 25.5, 34.2 )( 26.9, 34.4 )
24.120.524.0
( 13.6, 33.7 )( 9.3, 30.2 )( 17.0, 30.2 )
19
Distribution of student gain for 3 AYP schools
20
Distribution of student gain for 3 non-AYP schools
21
Comparing Features: AYP & the CRESST Approach
AYP CRESST Approach
Data Structure Cross-sectional (follow grade levels, e.g., 4th graders in a school, over time)
Longitudinal (follow individual students over time)
Performance Measure (Outcome)
Proficiency levels (using cut scores)
Individual gains or growth
Subgroup Demographic characteristics Performance-level groups Plus Demographic characteristics
Adjustments / Controls for Student or School Characteristics
No controls or adjustments, just disaggregations – loss of advantages when comparing against other schools
Can adjust for differences between schools and students in the model
Type of Growth Examined
Percent Proficient may mask different underlying growth patterns: Even flexibility given to schools through Safe Harbor option is only for movement around the proficiency cut score
More complete picture of growth PLUS growth distribution
22
Different Growth By Performance Subgroups & Demographic Subgroups
40
50
60
70
80
90
grade7 grade8 grade9 grade10
Grade
Math
Ach.
exp. traj. of change forboys 12 points abovethe mean
exp. traj. of change forgirls 12 points abovethe mean
exp. traj. of change forgirls at the mean
exp. traj. of change forboys at the mean
exp. traj. of change forgirls 12 points belowthe mean
exp. traj. of change forboys 12 points belowthe mean
23
Conclusions
Analyses using our alternative approach:
• More informative picture of growth using individual, longitudinal student gains
• More complete picture of how student growth is distributed within a school
Stimulate discussion among teachers and administrator to identify students in need earlier (Seltzer, Choi & Thum, 2003)
Encourage educators to think about achievement levels rather than (or in addition to) current subgroup categories - may be more productive and actionable