What Was Learned from aSecond Year of Implementation

IES Research Conference

Washington, DCJune 8, 2009

William Corrin, Senior Research AssociateMDRC

Presentation Overview

Project Background Do teachers need more than a year to master new

practices?

- Year 2 Implementation Findings (and Year 1 Comparison)

- Impact findings

- Exploratory findings

o Two-Year Teachers and Replacement Teachers

o Two-Year Teachers: Year 1 vs. Year 2

Summary

Project Background

The ERO Study

The ERO study is an impact evaluation of two supplemental literacy programs targeted at 9th grade students with limited reading skills.

Key research question:

- What are the impacts of the two interventions, together and separately, on ninth-grade students’ reading achievement and reading behaviors?

The Interventions

Two literacy programs targeted at 9th grade students with limited literacy skills (2-5 years below grade level)- Reading Apprenticeship Academic Literacy (RAAL)

- Xtreme Reading (XR)

The programs are supplemental reading classes- Designed as full-year courses that replace a 9th grade

elective class (rather than a core academic class)

The two programs share common goals and instructional principles

Delivery of the Interventions Each high school implemented one of the two programs

for ninth graders for two school years

- 2005-06 (Cohort 1) and 2006-07 (Cohort 2)

One ERO teacher per high school

- Experienced full-time ELA or social studies teacher was trained to teach the ERO reading course

- Responsible for 4 sections of 10-15 students each

Training and technical assistance were provided to teachers by the developers

- Summer training institutes, school-year off-site booster training sessions and on-site coaching

Implementation Data Collection

To answer questions about how well the programs were put into place

Allows us to answer questions about how implementation compared across program years: Do teachers do better with more experience teaching these programs? That is, is implementation stronger in the second year compared to the first?

Allows us to investigate associations between implementation and impacts

Measuring Implementation Fidelity

Two site visits in Year 2 (one per semester; only one site visit in Year 1)

Implementation fidelity was assessed on two dimensions:- Classroom learning environment

- Comprehension instruction

Classroom observation ratings were obtained for 6 constructs common to both programs, and 7 program-specific constructs- Ratings are based on a 3-point scale

Measuring Implementation Fidelity (cont.)

Composite ratings for each of the two dimensions were calculated, and then used to classify sites as:

- “well aligned” (average rating > 2)

- “moderately aligned” (average rating = 1.5 – 1.9)

- “poorly aligned” (average rating < 1.5)

Implementation Findings

Implementation Findings

ERO teachers in Year 2- Of the 34 ERO teachers in Year 2:

o 25 had taught the entire first year of the study (13 RAAL, 12 XR)

o 2 had taught part of the first year (both XR)

o 7 were new to the programs (4 RAAL, 3 XR)

All Year 2 teachers taught the entire year

Implementation Findings (cont.)

Implementation fidelity in Year 2- In terms of classroom learning environment, 1 school

was rated as poorly aligned with program models (vs. 4 schools in Year 1)

- In terms of reading comprehension instruction, 1 school was rated as poorly aligned with program models (vs. 9 schools in Year 1)

- 23 schools were rated as well aligned on both dimensions (vs. 16 schools in Year 1)

- Similar pattern of implementation findings at RAAL and XR schools

Implementation Findings (cont.)

Implementation fidelity rated higher in Year 2- In terms of classroom learning environment, the

average rating at the spring site visit was 2.5 in Year 2 compared to 2.2 in Year 1

- In terms of reading comprehension instruction, the average rating at the spring site visit was 2.3 in Year 2 compared to 1.9

Learning EnvironmentBy Site Visit

Comprehension Instructionby Site Visit

Impact Findings

Random Assignment and Analysis

Random assignment:- School-Level

o 34 schools randomly assigned within 10 districts to RAAL or Xtreme Reading (17 RAAL sites and 17 Xtreme Reading sites)

- Student-Level o In each high school, eligible students were randomly

assigned to either: Enroll in an ERO class (ERO group) or Take one of the regular elective classes (non-ERO

group)

Analysis: Impacts are estimated by comparing the outcomes of students in the ERO and non-ERO group

Impact Findings: Reading Achievement

Reading achievement was measured using the GRADE, a standardized reading assessment.

Overall positive impact on reading comprehension- 0.09 SD for Cohort 1

- 0.08 SD for Cohort 2

No impact on vocabulary- Same result for both cohorts

Impact Findings: Reading Behaviors

Cohort 1- Estimated impacts on three reading behaviors were

not statistically significant (frequency of school-related reading, freq. of non-school-related reading, use of reading strategies)

Cohort 2- Estimated impact on students’ use of the reading

strategies taught by the programs is 0.09 SD and statistically significant

Exploratory Analyses

Implementation Fidelity by Teacher Experience and Year

Implementation ratings by year and by teacher experience:

Implementation Dimension Year

Schools with Two-Year Teachers (N=25)

Schools with Replacement

Teachers (N=9)

Learning Environment

1 2.4 1.7

2 2.5 2.4

Reading Comprehension

Instruction

1 2.0 1.8

2 2.3 2.3

Implementation Fidelity by Teacher Experience and Year

Overall implementation fidelity:

- Of the 25 sites with two-year teachers, 19 had higher overall implementation ratings in the second year compared to the first year

- Of the 9 sites with replacement teachers, 8 sites had higher overall implementation ratings with the replacement teacher compared to the original teacher

Composite Fidelity Scores by Site Visit

Variation in Impacts

Impacts on Student Outcomes by Teacher Experience Teaching ERO: Year 2

Student Outcome

Schools w/ Two-Year Teachers

(N=25)

Schools w/ Replacement

Teachers (N=9)

Reading Comprehension

0.09 SD*

0.06 SD

Use of Reading Strategies

0.10 SD*

0.07 SD

Variation in Impacts (cont.)

Impacts on Student Outcomes by Implementation Year: Schools w/ Two-Year Teachers (N=25) Student Outcome

Year 1

Year 2

Reading Comprehension

0.11 SD*

0.09 SD*

Use of Reading Strategies (diff. betw. yrs. is stat. sig. )

-0.05 SD

0.10 SD*

Summary Supplemental literacy courses for 9th grade students can have a

positive impact on students’ reading comprehension.

These programs can be implemented with reasonable fidelity in a diverse array of high schools, and implementation fidelity can improve over time.

On average, implementation fidelity was stronger in the second year.

There is suggestive, but not convincing, evidence that teachers gained greater mastery of the programs having a second year to teach it.

However, impacts on students reading comprehension remained the same. (Impacts on students’ use of reading strategies were greater.)

Further questions? William Corrin

- william.corrin@mdrc.org

- (212) 340-8840 Marie-Andree Somers

- marie-andree.somers@mdrc.org

- (212) 340-8825 Paul Strasberg

- paul.strasberg@ed.gov

- (202) 219-3400