Post on 22-Nov-2014
description
transcript
Randomized Social Experiments
Marc GurgandParis School of Economics and J-PAL
OECD - May 19, 2014
Randomized experiments in three sentences
Evaluate the impact of social policies on measurable outcomes
Compare treatment and control goup
Randomize status to make the two groups comparable
Randomized experiments in three sentences
Evaluate the impact of social policies on measurable outcomes
Compare treatment and control goup
Randomize status to make the two groups comparable
Influential experiments
Class size : STAR (US 1985)
Welfare : Self-sufficiency project (Canada, 1994)
Local environment : Moving to Opportunity (US, 1994)
Conditional Cash Transfer : Progressa (Mexico, 1997)
Renewal : Development (Esther Duflo), Europe (Fondsd’experiementation jeunesse, Education Endowment Fundation,Trygfonden/Aarhus, ...)
Methodology in two slides
Counterfactual and causal effects
Objective : measure the causal effect of some intervention (ex.class size reduction) on some outcomes (ex. test scores)
Counterfactual
What would have happened (to a given individual)had he/she not been subject to the intervention
Causal impact defined (and measured) by the difference betweenoutcomes in the obseved state and outcomes in the counterfactualstate
Counterfactual and causal effects
Objective : measure the causal effect of some intervention (ex.class size reduction) on some outcomes (ex. test scores)
Counterfactual
What would have happened (to a given individual)had he/she not been subject to the intervention
Causal impact defined (and measured) by the difference betweenoutcomes in the obseved state and outcomes in the counterfactualstate
Counterfactual and causal effects
Objective : measure the causal effect of some intervention (ex.class size reduction) on some outcomes (ex. test scores)
Counterfactual
What would have happened (to a given individual)had he/she not been subject to the intervention
Causal impact defined (and measured) by the difference betweenoutcomes in the obseved state and outcomes in the counterfactualstate
Counterfactual and causal effects
Objective : measure the causal effect of some intervention (ex.class size reduction) on some outcomes (ex. test scores)
Counterfactual
What would have happened (to a given individual)had he/she not been subject to the intervention
Causal impact defined (and measured) by the difference betweenoutcomes in the obseved state and outcomes in the counterfactualstate
Randomized experiments
Objective : build-up counterfactuals (“same individual”)
Treament and control group randomized
The outcome in one group is a counterfactual for the other groupComparison of the two outcomes measures causal impact
Any systematic difference between the two groups must result formthe intervention
Literature moves away from simple 0/1 experiments towards morecomplex designs that inform policy more precisely
Randomized experiments
Objective : build-up counterfactuals (“same individual”)
Treament and control group randomized
The outcome in one group is a counterfactual for the other groupComparison of the two outcomes measures causal impact
Any systematic difference between the two groups must result formthe intervention
Literature moves away from simple 0/1 experiments towards morecomplex designs that inform policy more precisely
Randomized experiments
Objective : build-up counterfactuals (“same individual”)
Treament and control group randomized
The outcome in one group is a counterfactual for the other groupComparison of the two outcomes measures causal impact
Any systematic difference between the two groups must result formthe intervention
Literature moves away from simple 0/1 experiments towards morecomplex designs that inform policy more precisely
Randomized experiments
Objective : build-up counterfactuals (“same individual”)
Treament and control group randomized
The outcome in one group is a counterfactual for the other groupComparison of the two outcomes measures causal impact
Any systematic difference between the two groups must result formthe intervention
Literature moves away from simple 0/1 experiments towards morecomplex designs that inform policy more precisely
Randomized experiments
Objective : build-up counterfactuals (“same individual”)
Treament and control group randomized
The outcome in one group is a counterfactual for the other groupComparison of the two outcomes measures causal impact
Any systematic difference between the two groups must result formthe intervention
Literature moves away from simple 0/1 experiments towards morecomplex designs that inform policy more precisely
Two examples in labor policy
Example : Counseling the unemployed
Important and costly active policy
Recent trend in subcontracting to private operators
Comparing counseled and not counseled ; or before and afterimplementation of counseling : does not make sense.
Example : Counseling the unemployed
Important and costly active policy
Recent trend in subcontracting to private operators
Comparing counseled and not counseled ; or before and afterimplementation of counseling : does not make sense.
Example : Counseling the unemployed
Important and costly active policy
Recent trend in subcontracting to private operators
Comparing counseled and not counseled ; or before and afterimplementation of counseling : does not make sense.
Example : Counseling the unemployed
Important and costly active policy
Recent trend in subcontracting to private operators
Comparing counseled and not counseled ; or before and afterimplementation of counseling : does not make sense.
Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counselingIntensive private provider counselingstandard PES track
Mesure exit towards employment in the three groups
Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counselingIntensive private provider counselingstandard PES track
Mesure exit towards employment in the three groups
Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counselingIntensive private provider counselingstandard PES track
Mesure exit towards employment in the three groups
Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counseling
Intensive private provider counselingstandard PES track
Mesure exit towards employment in the three groups
Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counselingIntensive private provider counseling
standard PES track
Mesure exit towards employment in the three groups
Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counselingIntensive private provider counselingstandard PES track
Mesure exit towards employment in the three groups
Experiment 1
200 PES offices in France, 44,000 unemployed in the target
Upon registration randomized into :
Intensive public service counselingIntensive private provider counselingstandard PES track
Mesure exit towards employment in the three groups
Intensive Counseling against standard track
After 6 months : entry rate increases from 20% to 30% with PES ;from 20% to 24% with private providers
(source : Behaghel et al. 2014)
Displacement ?
Two theories on overall impact of counseling :
1 Jobs are in given quantity and quickly filed : if one workeraccesses jobs faster, this is detrimental to another worker
2 Vacancies and unemployed workers coexist in largeproportion : counseling reduces frictions, moves the Beveridgecurve and can improve treated outcomes without decreasingthe untreated’s situation
The usual random experiment does not answer this question
Displacement ?
Two theories on overall impact of counseling :
1 Jobs are in given quantity and quickly filed : if one workeraccesses jobs faster, this is detrimental to another worker
2 Vacancies and unemployed workers coexist in largeproportion : counseling reduces frictions, moves the Beveridgecurve and can improve treated outcomes without decreasingthe untreated’s situation
The usual random experiment does not answer this question
Displacement ?
Two theories on overall impact of counseling :
1 Jobs are in given quantity and quickly filed : if one workeraccesses jobs faster, this is detrimental to another worker
2 Vacancies and unemployed workers coexist in largeproportion : counseling reduces frictions, moves the Beveridgecurve and can improve treated outcomes without decreasingthe untreated’s situation
The usual random experiment does not answer this question
Displacement ?
Two theories on overall impact of counseling :
1 Jobs are in given quantity and quickly filed : if one workeraccesses jobs faster, this is detrimental to another worker
2 Vacancies and unemployed workers coexist in largeproportion : counseling reduces frictions, moves the Beveridgecurve and can improve treated outcomes without decreasingthe untreated’s situation
The usual random experiment does not answer this question
Displacement ?
Two theories on overall impact of counseling :
1 Jobs are in given quantity and quickly filed : if one workeraccesses jobs faster, this is detrimental to another worker
2 Vacancies and unemployed workers coexist in largeproportion : counseling reduces frictions, moves the Beveridgecurve and can improve treated outcomes without decreasingthe untreated’s situation
The usual random experiment does not answer this question
Experiment 2
235 PES offices interpreted as local labor markets
Randomize offices : in some no intensive counseling at all
In others, randomize some unemployed that are offeredintensive counseling
11 000 workers followed during 20 months
Experiment 2
235 PES offices interpreted as local labor markets
Randomize offices : in some no intensive counseling at all
In others, randomize some unemployed that are offeredintensive counseling
11 000 workers followed during 20 months
Experiment 2
235 PES offices interpreted as local labor markets
Randomize offices : in some no intensive counseling at all
In others, randomize some unemployed that are offeredintensive counseling
11 000 workers followed during 20 months
Experiment 2
235 PES offices interpreted as local labor markets
Randomize offices : in some no intensive counseling at all
In others, randomize some unemployed that are offeredintensive counseling
11 000 workers followed during 20 months
Experiment 2
235 PES offices interpreted as local labor markets
Randomize offices : in some no intensive counseling at all
In others, randomize some unemployed that are offeredintensive counseling
11 000 workers followed during 20 months
Exit to jobs within 8 months
results
www.povertyactionlab.org 3
Only 35percentof the treatmentgroupenrolled in theintensive counseling program. This low take-up may have been because nearly half of the participants already had some form of employment when they were assigned to receive counseling. Take-up of intensive counseling services was higher among people who were not working, had less advanced degrees, and were receiving government unemployment benefits.
Individuals in the treatment group foundworkmorequickly than those in the comparison group in thesameareas,butthesegainsdisappearedafter twelvemonths.After eight months, job seekers in the treatment group were 2.5 percentage points more likely to have found long-term employment, an effect that was mostly driven by men. Increases in employment were also driven by job seekers who found fixed-term contracts, and there was no increase in permanent employment contracts. However, all employment gains disappeared after 12 months and individuals in the treatment group were no more likely to be employed than those in the comparison group (figure 2).
abdul latif jameel poverty action lab
Providing intensive job counseling created adisplacement effect, lowering the employment rateamongthecomparisongroupindividualswholivedintreatmentareas.In treatment areas, job seekers who were offered counseling were 2.5 percentage points more likely to have found long-term employment after eight months than their peers who did not receive counseling. However, these peers were 2.1 percentage points less likely to have found long-term employment than job seekers in areas where no one received counseling. The net effect of being offered the program is, therefore, small and insignificant (Figure 3, left panel).
The low overall impact of being offered intensivecounseling is partly due to the relatively low take-upof the program. Since only 35 percent of those offered intensive counseling chose to enroll, the direct benefit on those who took up the program was essentially compensated by the displacement effects on the job seekers who were assigned to the treatment group but did not enroll.
Displacement effectswere strongest in placeswherejobseekerseligiblefortheprogramwerecompetingforasmallerpoolofjobs.In treatment areas with extremely competitive job markets, the employment rate among the comparison group individuals was lower by 7.7 points than the employment rate in areas where no treatment was offered at all (figure 3, right panel).
figure 3: employment rates among treated and untreated job seekers
time since counseling offer
perc
ent
wit
h l
on
g-t
erm
em
plo
ymen
t 60%
55%
50%
45%
40%
35%
30%8 months 12 months 20 months
treatment group
comparison group in treated areas
40%
35%
30%
25%
Pure Comparison G
roup
Comparison G
roup in
Treated Areas
Treatment G
roup
Pure Comparison G
roup
Comparison G
roup in
Treated Areas
Treatment G
roup
competitive job marketsall job markets
Signi�cantly di�erent from the pure comparison group
figure 2: rates of long-term employment over time
(source : Crepon et al. 2013)
Lessons on job counseling
Efficient technology (but no long run effects)
Private providers no more efficient (and less cost effective)than PES (similar RCTs in Germany & Sweden)
Evidence in favor of displacement effects : benefit is aroundzero
Strong statements that require RCT to be credible(but additional evidence is needed)
Lessons on job counseling
Efficient technology (but no long run effects)
Private providers no more efficient (and less cost effective)than PES (similar RCTs in Germany & Sweden)
Evidence in favor of displacement effects : benefit is aroundzero
Strong statements that require RCT to be credible(but additional evidence is needed)
Lessons on job counseling
Efficient technology (but no long run effects)
Private providers no more efficient (and less cost effective)than PES (similar RCTs in Germany & Sweden)
Evidence in favor of displacement effects : benefit is aroundzero
Strong statements that require RCT to be credible(but additional evidence is needed)
Lessons on job counseling
Efficient technology (but no long run effects)
Private providers no more efficient (and less cost effective)than PES (similar RCTs in Germany & Sweden)
Evidence in favor of displacement effects : benefit is aroundzero
Strong statements that require RCT to be credible(but additional evidence is needed)
Lessons on job counseling
Efficient technology (but no long run effects)
Private providers no more efficient (and less cost effective)than PES (similar RCTs in Germany & Sweden)
Evidence in favor of displacement effects : benefit is aroundzero
Strong statements that require RCT to be credible(but additional evidence is needed)