WHY RANDOMIZE
March 2014
Presented by:Raul Sanchez de la Sierra
Why randomize
?
Because correlation is not causation
Example: Impact of a medicine
Does the medicine increase weight??
Takes the drug
Does not take a drug
Overview
1. Causal impact and counterfactual
2. Comparison groups
3. Randomization
CAUSAL IMPACT
AND
COUNTERFACTUAL
Causal impact of a policy
• Change due to the policy
Causal impact of a policy
Pay for performance Traditional wage
What really happened What would have happened without the policy
Real world Imaginary world
Counterfactual
Counterfactual
Counterfactual: Would would have happened (imaginary)
Problem: Will never observe what would have happened
How to measure the causal impact of a policy?
Causal impact of a policy
Pay for performance Traditional wage
What really happened What would have happened without the policy
Real world Imaginary world
CounterfactualSimilar to Counterfactual
COMPARISON GROUPS
Selecting the comparison group
• Select a group that is exactly like the group of participants in all ways except the policy
Good
Bad
Pratham’s Balsakhi program
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
The intervention Work with Pratham in 124 Municipal Schools
Hire local women (Balsakhis) from the community
Train them to teach basic literacy, numeracy
Identify lowest performing students• Balsakhi teaches them basic competencies
15
Possible comparisons
1. Compare after to before the policy
2. Compare policy group to other group
3. Compare change in policy group to change in other group
4. Other non-experimental methods
5. Randomized Experiment
Method 1: Compare after to before the policy
• Test scores after Balsakhi
• Test scores before Balsakhi
16
17
Method 1: Compare after to before the policy
Before
After Difference
Scores 24 51 +27
18
Method 1: Compare after to before the policy
Improvement could be due to other reasons related to TIME
• Problem with this comparison:
Why this may not be a good strategy
Pay for performance Traditional wage
What really happenedWhat would have happened without the policy
Before
After True counterfactual
Wrong counterfactual
20
Method 2: Compare to other group
Compare test scores of…
Children who got balsakhi
With test scores of…
Children who did not get balsakhi
21
Method 2: Compare to other group
Balsakhi
No Balsak
khi
Difference
Scores 51 56 -5
22
Method 2: Compare to other group
Balsakhi students may be different:
Examples:- Poorer- Worst performing
• Problem with this comparison:
Why this may not be a good strategy
Pay for performance Comparison group
What really happenedWhat happened to the other group
Before
After
Counterfactual
What would have happened without the policy
True counterfactual
Wrong counterfactual
24
Method 3: Difference-in-Differences
Children who got balsakhi
Compare gains in test scores of…
With gains in test scores of…
Children who did not get balsakhi
25
Balsakhi
No Balsak
khi
Difference
After 51 56 -5
Method 3: Difference-in-Differences
26
Balsakhi
No Balsak
khi
Difference
After 51 56 -5
Before 24 37 -13
Method 3: Difference-in-Differences
27
Balsakhi
No Balsak
khi
Difference
After 51 56 -5
Before 24 37 -13
Difference in Differences
+8
Method 3: Difference-in-Differences
28
Balsakhi students may IMPROVE systematically faster for other reasons than the program:
Examples:- They may start from a lower level
Problem with this comparison:Method 3: Difference-in-Differences
Why this may not be a good strategy
Pay for performance Comparison group
What really happenedWhat happened to the other group
Before
After
Counterfactual
What would have happened without the policy
30
Method 4: Regression Analysis
Children who got balsakhi
• Compare test scores at the start and at the end of the program.
• For students of the same gender, age (CONTROL)
With
Children who did not get balsakhi
31
Balsakhi students may IMPROVE systematically faster than other students of their own gender and age for other reasons than the program:
Examples:- They may start from a lower level
Problem with this comparison:
Method 4: Regression Analysis
Why this may not be a good strategy
Pay for performance Comparison group
What really happenedWhat happened to the other group
Before
After
Counterfactual
What would have happened without the policy
33
Fundamental problem:
Cannot know what makes Balsakhi student different before the program
Never fully credible
Method 4: Regression Analysis
Problem with non experiments
• SELECTION BIAS
Non-experimental Methods
• Instrumental Variables
• Regression Discontinuity
35
RANDOMIZATION
Non-random assignment
HQMonthly income, per capita
1000
500
0 Treatment Control
1457
947
Random assignment
Monthly income, per capita
1000
500
0Treatment Control
1257 1242
HQ
1. Baseline
2. RANDOMIZED ASSIGNMENT
3. INTERVENTION
3. STATUS QUO
4. Endline
TARGET POPULATION
How does randomization work?
40
Randomization at work: educationApril 06/07
Jun 05
Aug 05
Tests Tests
Pay for performance Fixed wageInitial Test
How the treated group looks now
One of the two then gets the treatment. Outcomes are then compared between those that did and did not get
treatment
In a randomized trial, two groups are formed through a lottery
(to make sure that they are comparable)
How they would have looked without treatment
This group gets the treatment
This groups does not
Randomization at work: medicineIn a randomized trial, two groups are formed
through a lottery(to make sure that they are comparable)
One of the two then gets the treatment. Outcomes are then compared between those
that did and did not get treatment
CC
The truth If project goes to easy places
If project goes to hard places If lotteries are used
T
T TT
T
TT T
T
T
T
TTC
CC
CCCCCC
C
CC
T
CT
Key advantage of experiments
Members of the groups are statistically identical
any change can be attributed to the program
43
Method 5: Randomized Experiment
• Suppose we evaluated the balsakhi program using a randomized experiment
44
Impact of Balsakhi - Summary
45
Method Impact Estimate(1) Pre-post 26.42*(2) Simple Difference -5.05*(3) Difference-in-Difference
6.82*
(4) Regression 1.92
*: Statistically significant at the 5% level
Impact of Balsakhi - Summary
46
Method Impact Estimate(1) Pre-post 26.42*(2) Simple Difference -5.05*(3) Difference-in-Difference
6.82*
(4) Regression 1.92(5)Randomized Experiment
5.87**: Statistically significant at the 5% level
What is the impact of this program?
Time
Prim
ary
Out
com
e
Impact
Counterfactual
Program starts
Impact: What is it?
Time
Prim
ary
Out
com
e
ImpactCounterfactualProgram starts
Impact: What is it?
Time
Prim
ary
Out
com
e
ImpactCounterfactual
Program starts