Impact Evaluation MethodsRegression Discontinuity Design and
Difference in Differences
Slides by Paul J. Gertler & Sebastian Martinez
2
Measuring Impact
• Experimental design/randomization
• Quasi-experiments
– Regression Discontinuity
– Double differences (diff in diff)
– Other options
3
Case 4: Regression Discontinuity
• Assignment to treatment is based on a clearly defined index or parameter with a known cutoff for eligibility
• RD is possible when units can be ordered along a quantifiable dimension which is systematically related to the assignment of treatment
• The effect is measured at the discontinuity – estimated impact around the cutoff may not generalize to entire population
4
• Anti-poverty programs targeted to households below a given poverty index
• Pension programs targeted to population above a certain age
• Scholarships targeted to students with high scores on standardized test
• CDD Programs awarded to NGOs that achieve highest scores
Indexes are common in targeting of social programs
5
• Target transfer to poorest households
• Construct poverty index from 1 to 100 with pre-intervention characteristics
• Households with a score <=50 are poor
• Households with a score >50 are non-poor
• Cash transfer to poor households
• Measure outcomes (i.e. consumption) before and after transfer
Example: Effect of Cash Transfer on Consumption
6
6065
7075
80O
utco
me
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Baseline
7
6065
7075
80O
utco
me
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Baseline
Non-Poor
Poor
8
6570
7580
Out
com
e
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Post Intervention
9
6570
7580
Out
com
e
20 30 40 50 60 70 80Score
Regression Discontinuity Design - Post Intervention
Treatment Effect
10
• Oportunidades assigned benefits based on a poverty index
• Where
• Treatment = 1 if score <=750
• Treatment = 0 if score >750
Case 4: Regression Discontinuity
11
Case 4: Regression Discontinuity
Fitt
ed v
alu
es
puntaje estimado en focalizacion276 1294
153.578
379.224
2
Baseline – No treatment
0 1 ( )i i iy Treatment score
12
Estimated Impact on CPC
** Significant at 1% level
Case 4 - Regression DiscontinuityMultivariate Linear Regression
30.58**(5.93)
Fitt
ed v
alu
es
puntaje estimado en focalizacion276 1294
183.647
399.51 Treatment Period
Case 4: Regression Discontinuity
13
Potential Disadvantages of RD• Local average treatment effects – not always
generalizable
• Power: effect is estimated at the discontinuity, so we generally have fewer observations than in a randomized experiment with the same sample size
• Specification can be sensitive to functional form: make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: – Nonlinear Relationships– Interactions
14
Advantages of RD for Evaluation
• RD yields an unbiased estimate of treatment effect at the discontinuity
• Can many times take advantage of a known rule for assigning the benefit that are common in the designs of social policy
– No need to “exclude” a group of eligible households/individuals from treatment
15
Measuring Impact
• Experimental design/randomization
• Quasi-experiments
– Regression Discontinuity
– Double differences (Diff in diff)
– Other options
16
Case 5: Diff in diff
• Compare change in outcomes between treatments and non-treatment
– Impact is the difference in the change in outcomes
• Impact = (Yt1-Yt0) - (Yc1-Yc0)
17
TimeTreatment
Outcome
Treatment Group
Control Group
Average Treatment Effect
18
TimeTreatment
Outcome
Treatment Group
Control Group
Estimated Average Treatment Effect
Average Treatment Effect
19
Diff in Diff
• Fundamental assumption that trends (slopes) are the same in treatments and controls
• Need a minimum of three points in time to verify this and estimate treatment (two pre-intervention)
20
Not Enrolled Enrolled t-statMean ΔCPC 8.26 35.92 10.31
Case 5 - Diff in Diff
Linear Regression Multivariate Linear Regression
Estimated Impact on CPC 27.66** 25.53**(2.68) (2.77)
** Significant at 1% level
Case 5 - Diff in Diff
Case 5: Diff in Diff
21
Case 1 - Before and After
Case 2 - Enrolled/Not
Enrolled
Case 3 - Randomization
Case 4 - Regression
Discontinuity
Case 5 - Diff in Diff
Multivariate Linear
RegressionMultivariate Linear
Regression
Multivariate Linear
Regression
Multivariate Linear
Regression
Multivariate Linear
Regression
Estimated Impact on CPC 34.28** -4.15 29.79** 30.58** 25.53**
(2.11) (4.05) (3.00) (5.93) (2.77)** Significant at 1% level
Impact Evaluation Example – Summary of Results