Patrick J. WolfUniversity of Arkansas
Advocates for Parental Choice SymposiumMilwaukee, WIJune 16, 2013
Latest Research Findings on School Vouchers
About Me Long and fruitful relationship with
Catholic education Rigorous training in scientific
research methods “Wolf is a well-known advocate of
vouchers. He bases his advocacy on research…” AERA Newsletter
Department of Education Reform doctoral program
Presentation Plan DC voucher findings after 4+ years Questions/discussion Milwaukee voucher findings after 4
years Questions/discussion Florida and New York City findings Discussion of voucher program
design
Overview of the DC School Choice Incentive Act Program
“3-Sector Strategy” Student Eligibility
Grade K-12, residents of DC Family income ≤ 185% Federal poverty line
Scholarship Up to $7,500 to cover tuition, school fees, and transportation Renewable for up to 5 years
Order of Priority for Receiving a Scholarship Students attending schools in need of improvement (SINI) Students in non-SINI public schools Students already attending private school
Legislative Mandate for Evaluation
Independent, using “… the strongest possible research design for determining the effectiveness of the Program.”
Key Outcomes to be Studied Academic achievement School safety Retention, graduation, college admission “...success… in expanding school choice options”
(satisfaction) Effects on schools in DC
Annual Reports to Congress (Spring 2005-Spring 2010)
Who Was Studied? 2,308 lotteried public school applicants
2,012 students still school-age 4+ years after randomization
Characteristics of participants: Average household income below $18,000 at baseline Only 6 percent have mothers with BA SAT-9 scores: 33rd percentile in reading, 31st in math 44 percent from SINI schools between 2003 and 2005
(highest priority group for scholarships)
Study Design: Randomized Control Trial
Eligible applicants assigned by lottery to be offered (“treatment” group) or not offered (“control” group) a scholarship
Each spring, evaluation team administered: SAT-9 Parent surveys Student surveys (grades 4 and higher) Principal surveys (public and private)
Three comparisons made: Treatment versus control (pure experimental) Scholarship users versus control (adjusts for decliners) Private versus public school in year 4+ (Instrumental
Variables)
Scholarship Use Over 4+ Years
26% of treatment students used scholarship throughout the study
Participation drop-off (net): averaged 22% per year
Top reasons for dropping out of OSP Child got into a charter school (22%) Lack of space (for students transitioning from K-8)
(19%) Moved out of DC (15%) Transportation problems (14%)
Students more likely to drop out of OSP: lower initial test scores, older grade levels, male, with special needs, more siblings
Schools Attended by “Analysis Sample” in 2008-09
Treatment Control0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
55
11.5
18.4
35.3
26.6
53.2
Private Charter Traditional
Enro
llmen
t in
200
8-09
Summary of Key Findings Large increase in high school graduation rates Suggestive evidence of small gains in reading
overall No impacts apparent in math Half of subgroups show reading impacts Parents more satisfied with schools, view them
as safer No impacts on student satisfaction and reports
of safety
High School Graduation Rates, Overall Sample & SINI Subgroup, Parent Reports, 2008-09
Treatment Control Treatment Control0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
82%**
70%79%*†
66%
91%**86%*†
Full Sample SINI 2003-05 Subgroup
*Statistically significant at the 95 percent confidence level.**Statistically significant at the 99 percent confidence level.† = subgroup impact result remained statistically significant after adjustments for multiple comparisons.
IOT
ITTITT
IOT
ITT & IOT Impacts of the OSP on Reading Achievement Overall, by Years After Application
0.25 0.75 1.25 1.75 2.25 2.75 3.25 3.75 4.25 4.75-4-3-2-10123456789
1.35
3.82
5.27*4.754.46
6.88
8.03 7.94
1.03
3.17
4.46*3.9
-2.41
-0.53
0.89-0.13Sc
ale
Scor
e Po
ints
After 1 year After 2 years After three years After at least 4 years
*Statistically significant at the 95 percent confidence interval.
ITT & IOT Impacts of the OSP on Math Achievement Overall, by Years After Application
0.25 0.75 1.25 1.75 2.25 2.75 3.25 3.75 4.25-4
-3
-2
-1
0
1
2
3
4
5
6
3.58
0.29
0.950000000000001
0.850000000000001
5.74
3.534.03
4.41
2.74
0.230.81 0.700000000000
001-0.26
-3.07-2.42
-3.01
Scal
e Sc
ore
Poin
ts
After 1 year After 2 years After 3 years After at least 4 years
Effect Sizes & Months of Schooling for Statistically Significant Reading Subgroup Impacts Based on 3 Analytic Methods
Student Achievement:
Reading
Impact of the Scholarship Offer (ITT)
Impact of Scholarship Use (IOT)
Effect of Private Schooling (IV)
Effect Size Months of Schooling Effect Size Months of
Schooling Effect Size Months of Schooling
Not SINI 2003-05 .16* 3.5 .19* 4.2 .26 5.6
Higher performance .15* 3.9 .18* 4.6 .27 6.7
Female .15* 3.4 .18* 4.0 .37* 8.2
* Statistically significant at the 95 percent confidence interval.
Parent and Student Reports of Safety and an Orderly School Climate, 2008-09
Treatment Control0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
10.00
7.94**7.47
8.52**
Parent Perceptions
ITT
IOT
**Statistically significant at the 99 percent confidence level.
Treatment Control0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
10.00
6.16 6.01
Student Reports
Parent and Student Reports of School Satisfaction, 2008-09
Treatment Control0%
20%
40%
60%
80%
100%
76%**68%
78%** (IOT)
Percentage ofParents Who Gave School
Grade A or B
ITT
Treatment Control0%
20%
40%
60%
80%
100%
69% 72%
Percentage of Students Who Gave School
Grade A or B
**Statistically significant at the 99 percent confidence level.
Highlights of Qualitative Research Focus groups & interviews 2004-
2008 Findings:
Parents more satisfied if two-parent family & extensive search
Experience improved for families over time (stigma, financial policies, welcome)
Safety replaced by academics as key factor
Strict standards or accommodation?
Context DCPS proficiency and DC NAEP scores
increased by over 20 percent 2005-2009
DCPS per-pupil spending of $17,542 Reliable studies indicate graduation is
far more important than achievement: Increases earnings, health, longevity, &
marriage Decreases unemployment and incarceration
Political/Policy Developments Program closed to new applicants in March of
2009 2010 Appropriations Law cut funding, capped
program, mandated shoddy evaluation, added regs
2010 Republicans capture House, new Speaker Boehner lists OSP reauthorization as a priority
Lieberman reauthorization language added to 2011 federal budget as final element of compromise
Obama Administration initially refuses to admit new students to program, then relents
Obama Administration 2014 budget proposes to zero-out program
Reauthorization Changes Increases OSP appropriation to $20
million/year Increases voucher max to 12k for
high school and 8k for K-8 Removes most regulations on
private schools Mandates new rigorous evaluation Good for 5 more years
My 5 Minutes of Famehttp://
www.youtube.com/watch?v=5GTX-278Rpc&feature=player_embedded
Questions on DC OSP?
Milwaukee Parental Choice Program (MPCP) Nation’s first urban school voucher
program Launched in 1990 with 341 students in
7 schools Dramatically expanded in 1998 Served 23,198 students in 106 schools
in 2011-12 In 2006, State of Wisconsin called on
School Choice Demonstration Project to evaluate program over 5 years 23
MPCP Schools and Student Enrollment
1990
-91
1991
-92
1992
-93
1993
-94
1994
-95
1995
-96
1996
-97
1997
-98
1998
-99
1999
-00
2000
-01
2001
-02
2002
-03
2003
-04
2004
-05
2005
-06
2006
-07
2007
-08
2008
-09
2009
-10 -
2,500
5,000
7,500
10,000
12,500
15,000
17,500
20,000
22,500
0102030405060708090100110120130140
Total MPCP Pupil Count Participating Schools
Tota
l MPC
P St
uden
t En
rollm
ent N
umber of Participating Schools
Wisconsin 2005 Act 125 Enacted on March 10, 2006 Modified the main MPCP statute
(119.23):1. Raised enrollment cap to 22,5002. Added school accreditation requirements3. Initiated standardized testing of Choice students in
grades 4, 8, and 104. Mandated that a representative panel take the
Wisconsin Knowledge and Concepts Examinations (WKCE)
5. Required scores be sent to the SCDP, then on to the Legislative Audit Bureau
25
SCDP Longitudinal Evaluation Comprehensive – examining participating
effects on students, parents, and schools, plus systemic and community effects
Multi-method – quantitative and qualitative, with test scores, surveys, focus groups, site visits
Rigorous – careful matching on test and student characteristics for “apples-to-apples” comparisons whenever possible
Longitudinal – five years of data, fall 2006 through fall 2010, with focus on changes from baseline 26
Highlights of Quantitative Results Positive effect of program on attainment Some evidence of reading gains but
none of math gains Lots of school switching Poor performing schools in both sectors
being closed Pressure of competition has increased
MPS achievement slightly Taxpayers save over $50 million/year
Effect of 2006 MPCP Attendance on 2010-11 Educational Attainment
28-6.0
-4.0
-2.0
0.0
2.0
4.0
6.0
8.0
On Time Graduate
Five-Year Graduate
Ever Graduate
Two-Year College Enrollment
Four-Year College Enrollment
Persist in Four-Year College
7.1***
-4.1***
4.0*
-3.0*
6.0*** 6.0**
Effect of 2006 MPCP Attendance on Student Achievement: 2007 to 2010
-.2-.1
0.1
.2
2007 2008 2009 2010 2007 2008 2009 2010
Math Reading
Effe
ct (S
D U
nits
)
YearNOTE: Point estimates and confidence intervals based on results in Table 4, Model 2
Point Estimate and 90% Confidence IntervalEffect of Baseline MPCP Attendance on Student Achievement
29
Program Expansion 2011 Income limit increased to 300% of poverty Permanent income qualification No enrollment cap Students can attend non-Milwaukee private
schools Schools can require top-ups from higher-
income students Testing of all Choice students in grades 3-8 &
10 using the state test Regulations dropped regarding curriculum and
teacher training Program expanded to Racine 30
Program Expansion in 2013 Maximum of 500 students outside
Milwaukee/Racine (1000 beyond 2013) Any school district, limited to 1% of
enrollments Family income < 185% of poverty Maximum amount raised to $7,210
elementary and $7,856 high school in 2014-15
Added individual income tax deduction for private school tuition
Questions/Discussion of MPCP?
Florida Tax Credit Scholarship Program & Evaluation Launched in 2001 in response to court
ruling Served nearly 38,000 low-income
students in over 1000 schools in 2011 State Dept of Education longitudinal
evaluation by David Figlio Latest finding: program boosts reading
achievement for students close to income cut-off
33
New York City Attainment Study Privately-funded partial-tuition
scholarship program expanded in 1997
1900 students entering grades 1-5 randomly assigned
Increased college enrollment rate for African-Americans by 7.1 percentage points if offered and 8.7 if used
No overall effect or effect for Hispanic students
The Question of Policy Design: What Should Arkansas Do?1. Universal or targeted?2. Voucher, tax-credit, or ESA?3. Size of the scholarship?4. What should it cover?5. All top-off, some top-off, no top-off?6. Apply admission criteria, first come, or
lotteries?7. Hold the public schools financially harmless or
$$$ travels with the child?8. Regulations on participating schools?9. Required evaluation, testing, or financial
reporting?10. Programmatic supports for families?
For More InformationElectronic versions of all DC and Milwaukee reports
available at:http://www.uark.edu/ua/der/SCDP.html Patrick J. Wolf, Ph.D.Professor and 21st Century Endowed Chair in School ChoiceDepartment of Education ReformCollege of Education and Health Professions201 Graduate Education BuildingUniversity of ArkansasFayetteville, AR 72701Phone: 479-575-2084FAX: [email protected]