Behavioral Responses to Teacher Transfer Incentives: Results from a Randomized Experiment
INVALSI Conference on Improving Education through Accountability and Evaluation: Lessons from Around the World
Rome, Italy
October 4, 2012
Steven Glazerman Ali Protik Bing-ru Teh Julie Bruch Neil Seftor
Best teachers may not be working with the students who need them the most
Shift focus from improving productivity of the teacher workforce to composition
Big gaps in knowledge– Weak documentation of the policy problem– Lack of data on teacher transfer behavior– Lack of data on whether skills transfer– Controversy about teacher quality measures
(value added)
Policy Problem
2
Policy Response: Talent Transfer Initiative
$20,000 transfer incentive
Identify highest-performing (HP) teachers– Use value-added analysis, three years of data– Three pools: elementary, MS math, MS language arts– Top 20% are “highest performing”
Identify potential “receiving schools”
Recruit transfer candidates, arrange interviews
Support transfer teachers, issue payments
HP teachers already in potential receiving schools get retention stipend of $10,000
3
1. How do HP teachers respond to a monetary transfer incentive?
2. How do hard-to-staff schools respond to the opportunity to hire a HP teacher?
3. What impact do transfer teachers have in their new settings?– Did their skills transfer, i.e. were they portable?– Was “value added” the right metric?
Research Questions
4
Summary of Findings to Date
Implementation– Filling vacancies was feasible
– Large pool of candidates needed
– Meaningful contrast achieved
Intermediate impacts– Increased experience and credentials slightly
– No significant impact on climate or collegiality
– No change in how students assigned to teachers
– TTI transfers used less & provided more mentoring
Impact on test scores and retention– Will be public in the final report (2013)
5
Study Design
6
Identify potential receiving schools with a vacancy in a targeted grade/subject
Unit of randomization = teacher team– Team types can be:
• Elementary self-contained math and reading• Middle school math• Middle school English/language arts (ELA)
Experimental Design
7
School A School B
Study Design, Illustration
8
Randomization Block
School A School BFocal Teachers
Randomly assign teacher teams (grade within school) to treatment or control
Study Design, Illustration
9
Data
10
Ten Large, Diverse Districts in the Study
11
Cohort 1: seven districts in five states Cohort 2: three districts in two more states
Primary Data Collection: Surveys– Candidates– Receiving school teachers in study grades– Receiving school principals
Secondary Data– District-provided test scores and demographics– School-provided teacher rosters
Data
12
7 districts – Large, diverse– 5 county, 2 city
1,012 transfer candidates– 63 transfers from 51 sending schools
86 receiving schools– 124 teams randomized
15,266 students– Below average prior achievement– 6% white, 48% African American, 72% free lunch
Sample (Cohort 1)
13
Behavioral Response to Incentives: Implementation Findings
14
Low takeup rates, most candidates do not apply
Not too low to fill positions (90% filled)
Hard to predict who transfers
Findings on Response to Incentives
15
Response to Incentives in 7 Districts: Takeup Rates
16
Types of Transfers by Change in School Achievement Ranks Before and After Transfer
17N = 63
Types of Transfers by Change in School Poverty Ranks Before and After Transfer
18N = 63
Who Filled the Vacancies?
Characteristic Treatment Focal
Control Focal
Difference
First Year Teaching (%) 0 21 -21*
Experience, Years in Teaching
13 8 5*
Has Master’s Degree or Doctorate (%)
48 21 27*
Has National Board Certification (%)
23 12 11
Transferred via TTI (%) 95 0 95*
Sample Size 63 41
19
Behavioral Response Within the Receiving Schools:
Intermediate Impacts
20
Survey questions on degree of collaboration, mutual trust, or sharing ideas: no evidence of impact
Differential assignment of students to teachers: mixed evidence of impact
Mentoring and leadership: treatment led to more mentoring provided, less mentoring used
Findings on Impacts on School Dynamics
21
Mentoring Received and Provided to Others
22
Receives Mentoring Mentors Others
Summary of Findings to Date
Implementation– Filling vacancies was feasible
– Large pool of candidates needed
– Meaningful contrast achieved
Intermediate impacts– Increased experience and credentials slightly
– No significant impact on climate or collegiality
– No change in how students assigned to teachers
– TTI transfers used less & provided more mentoring
Impact on test scores and retention– Will be public in the final report (2013)
23
Impacts on test scores and retention
Cost-benefit– Shadow price of raising test scores using CSR– Retention adjusted impacts, extrapolate into future?
Spatial analysis of mobility decisions
Related policies
Future Work
24
Transfer groups of teachers (e.g. through reconstitution)
Additional screening criteria for HP teachers
Bonus conditional on performance in new school
Policy that spans district boundaries (e.g. statewide)
Related Policies
25
THE END(extra slides follow)
26
Districts vary
Elementary and middle school differ
Overall pattern suggests:– Unequal access at middle school level– Less evidence for unequal access at elementary
level
Summary of Prevalence Findings
27
Prevalence of HP Teachers: Do Low-Income Students Have Equal
Access?
28
Prevalence of Highest-PerformingMiddle School Math Teachers*
29
Quintiles Based on Poverty
* Statistically significant
Prevalence of Highest-PerformingMiddle School Language Arts Teachers*
30
Quintiles Based on Poverty
* Statistically significant
Prevalence of Highest-PerformingElementary Teachers
31
Quintiles Based on Poverty
Results for Individual Districts Results, Five Districts at a Time
32
Prevalence of Highest-Performing Middle School Math Teachers (Districts A-E)
33
Quintiles Based on Poverty
Prevalence of Highest-Performing Middle School Math Teachers (Districts F-J)
34
Quintiles Based on Poverty
Overall, but Using Achievement to Divide Schools
35
Prevalence of Highest-PerformingMiddle School Math Teachers*
36
Quintiles Based on Achievement
* Statistically significant
Prevalence of Highest-PerformingMiddle School Language Arts Teachers*
37
Quintiles Based on Achievement
* Statistically significant
Prevalence of Highest-PerformingElementary Teachers*
38
Quintiles Based on Achievement
* Statistically significant
Decompose value added estimate
Components of Estimated Teacher Performance
39
ˆ jt j j jt jtQ X P Total Performance
Persistent Teacher Ability
Returns to Specialization
Transitory Performance
Noise, Luck, Measurement Error
Prevalence of Highest-Performing Middle School ELA Teachers (Districts A-E)
40
Quintiles Based on Poverty
Prevalence of Highest-PerformingMiddle School ELA Teachers (Districts F-J)
41
Quintiles Based on Poverty
Prevalence of Highest-Performing Elementary Teachers (Districts A-E)
42
Quintiles Based on Poverty
Prevalence of Highest-Performing Elementary Teachers (Districts F-J)
43
Quintiles Based on Poverty
School A School B
School pair with matching vacancies in two grades.
Study Design, Crossover Case
44
Randomization Block
School A School B
Study Design, Crossover Case (cont’d.)
45
Teacher Team Dynamics
46
Team-level– Impact estimate has intent-to-treat (ITT) interpretation
Under zero resource allocation effect:
Focal teacher comparison– Impact estimate denotes the direct impact
Nonfocal teacher comparison– Impact estimate denotes the indirect impact
Team and Focal Teacher Analysis
47
Dilution of direct effect
Non-compliers (unfilled vacancies)
Block-defined subgroups– High contrast transfers– High value added transfers– Complier blocks
Interpretation/Analysis Issues
48
Self-Reported Reasons For Not Applying
Factor (selected reasons)Cited as a Reason
Most Important Reason
Happy at old school 91 32Child care or family-related issues 33 15
Commuting issues 53 6Concerns about receiving school neighborhood/safety 36 2
Not confident about effectiveness in new school 29 3
Concerns about being unwelcome, unsupported in new school 43 4
Did not like principals at the receiving schools 6 2
Students in receiving schools too challenging 25 6
Stipend not big enough 25 3
49Percentages, N = 680
How Are Students Assigned to Classrooms?Principal Report (N=57 Treatment, 54 Control)
50
None of the differences are statistically significant.
Who Filled the Vacancies? (Controls)
Final Status of the Vacancy Number Percentage
Positions Filled
New to teaching 9 13.4
New hire (new to district or teaching) 6 9.0
Transfer from another school 13 19.4
Transfer from another grade/subject 18 26.9
Unknown origin/uncertain 6 9.0
Position Lost, Transfer Cancelled, or Layoff Rescinded 7 10.4
Unknown Status 8 11.9
All Vacancies 67 100.0
51
Who Filled the Vacancies? (Treatment)
Final Status of the Vacancy Number Percentage
Filled with TTI Candidate 63 90.0
Filled Outside TTI 4 5.7
Position Lost or Transfer Cancelled 3 4.3
All Vacancies 70 100.0
52
Measure (on a 5-point scale)Treatment
MeanControl
Mean Difference
Current levels
Level of collaboration 3.7 3.6 0.2
Degree of trust and mutual respect 3.9 3.8 0.1
Teaches seek ideas from one another 3.8 3.9 -0.1
Change from Prior Year
Level of collaboration 1.1 0.6 0.5
Degree of trust and mutual respect 1.1 0.6 0.4
Teaches seek ideas from one another 1.0 0.8 0.1
Principal Reports of Collaboration
53
None of the differences are statistically significant.
How Are Students Assigned to Classrooms? Teacher Report (N=169 Treatment, 155 Control)
54
“Compared to my colleagues’ students’ academic ability, my students’ academic ability is…”
None of the differences are statistically significant.
How Are Students Assigned to Classrooms? Teacher Report (N=169 Treatment, 155 Control)
55
“Compared to my colleagues’ students’ behavior, my students’ behavior is…”
None of the differences are statistically significant.