Date post: | 14-Dec-2015 |
Category: |
Documents |
Upload: | santiago-copass |
View: | 225 times |
Download: | 3 times |
How to randomize?
povertyactionlab.org
• Quick Review: Why Randomize• Choosing the Sample Frame• Choosing the Unit of Randomization
• Options• How to Choose
• Randomization Methods• Mechanics of randomization • Stratification• Variations on Simple Treatment-Control
Lecture Overview
• Ensure that we compare oranges and oranges• Randomization: randomly select treatment and
comparison group before program starts• If we construct a comparison group after the
program has started Bias– Before and After
• Other things may be going on
– Participants and Non participants (Simple comparison; Diff-in-Diff; Regression)
• Selection Bias
Why randomize?
Selection Bias
Difference of Means
Intrinsic Differences Impact of the program
Observables Non Observables
Selection Bias
Baseline
Randomization
Assignment to Treatment Assignment to Control
Sample frame
Compare T and C
How to Randomize, Part I - 6
Random assignment
2006
Income per person, per month, rupees
1000
500
0Treat Compare
1457 1442
Baseline
Randomization
Assignment to Treatment Assignment to Control
Endline
Compare T and C
Sample frame
Intervention
Compare T and C
• Quick Review: Why Randomize• Choosing the Sample Frame• Choosing the Unit of Randomization
• Options• How to Choose
• Randomization Methods• Mechanics of randomization • Stratification• Variations on Simple Treatment-Control
Lecture Overview
• Geographic area and Population• Where will the project occur?
– Example: Purnea, Ajmer Districts• Who is the target population? What types of
tolas/villages will we be working in? • Choice of area and population is usually not
random– Convenience, eligible population
• How representative do we want to be?
Area and Population
• Quick Review: Why Randomize• Choosing the Sample Frame• Choosing the Unit of Randomization
• Options• How to Choose
• Randomization Methods• Mechanics of randomization • Stratification• Variations on Simple Treatment-Control
Lecture Overview
1. Randomizing at the individual level2. Randomizing at the group level
“Cluster Randomized Trial”
Unit of Randomization: Options
11
Unit of Randomization: Individual?
Unit of Randomization: Individual?
“Groups of individuals”: Cluster Randomized Trial
Unit of Randomization: Clusters?
Unit of Randomization: Class?
Unit of Randomization: Class?
Unit of Randomization: School?
Unit of Randomization: School?
• Quick Review: Why Randomize• Choosing the Sample Frame• Choosing the Unit of Randomization
• Options• How to Choose
• Randomization Methods• Mechanics of randomization • Stratification• Variations on Simple Treatment-Control
Lecture Overview
• Spillovers and Crossovers– How wide is the potential impact?
• How is the intervention administered? – What is the mobilization method? – What is the catchment area of each
“unit of intervention”? • Others
– Logistical constraints– Budget and Sample size requirements
How to Choose the Level
• Remember the counterfactual!• If control group is different from the
counterfactual, results will be biased• Can occur due to
• Spillovers• Crossovers
“Contamination” of the Control
A Deworming Program
Treatment Control
Spillovers
Counterfactual
• Positive spillovers (treatment -> control)– Medicines or vaccines– District gives more resources to control schools
• Negative spillovers (control -> treatment)– Re-infection
• Negative spillovers (treatment -> control)– Less diseases in the area so parents put les efforts in
treating kids • Mother Literacy?• How do we ensure this does not happen?
Spillovers (Indirect “treatment”)
• Control individuals get directly treated – E.g Balsakhis: Control kids get treated (e.g.
transfer to treatment schools)– E.g: Microfinance evaluation in urban areas
• Mother Literacy? • How do we ensure this does not happen?
Crossovers (Direct treatment of control)
• What is the mobilization method?• What is the catchment area of each
“unit of intervention”? • Usually best to keep the randomization unit as
the unit of intervention• Given this, also useful to keep the unit of
randomization as small as possible
Unit and method of Intervention
• Logistics– Large villages as unit of randomization: more
travel, difficult to find volunteers etc.• Budget and sample size
– Individual randomization is cheaper (tomorrow!)– There may not be enough clusters of a certain
type available for the intervention• Villages vs tolas
Logistics, Budget and Sample Size
• Quick Review: Why Randomize• Choosing the Sample Frame• Choosing the Unit of Randomization
• Options• How to Choose
• Randomization Methods• Mechanics of randomization • Stratification• Variations on Simple Treatment-Control
Lecture Overview
• Most programs have limited resources– Vouchers, Farmer Training Programs– Results in more eligible recipients than resources
will allow services for• Programs are often piloted in a smaller scale
before being scaled up • Lottery can be a fair way to select participants
Lottery
• What if you have just enough resources for the number of recipients in your work area?– 500 seats for a training, and 500 applicants
• Increase outreach of activities• May not be possible• Control group may be upset anyway• Phase-in Design
Lottery
• Everyone gets program eventually• Natural approach when expanding program
faces resource constraints• What determines which schools, branches,
etc. will be covered in which year?– Some choices based on need, geography, etc.– Others largely arbitrary
Phase-in: takes advantage of expansion
Phase-in design
Round 1Treatment: 1/3Control: 2/3
Round 2Treatment: 2/3Control: 1/3
Round 3Treatment: 3/3Control: 0 1
1
11
1
1
1
1
1
11
1
1
1
2
2
22
2
2
22
2
2
2
22
2
2
2
3
333
3
33
33
3
33
3
3
3 3
3
Round 1Treatment: 1/3Control: 2/3
Round 2Treatment: 2/3Control: 1/3
Randomized evaluation endsRandomized evaluation ends
Advantages• Everyone gets something eventually
Concerns• Can complicate estimating long-run effects• Natural expansion timeline may be too short • Do expectations of change changes behavior
today?
Phase-in designs
• Groups get treatment in turns• Advantages• Concerns
Rotation design
How to Randomize, Part I - 33
Round 1Treatment: 1/2Control: 1/2
Rotation design
Round 2Treatment from Round 1 Control——————————————————————————
Control from Round 1 Treatment
Round 1Treatment: 1/2Control: 1/2
• Phase-in may not provide enough benefit to late round participants
• Cooperation from control group may be critical
• Consider within-group randomization• E.g., balsakhi program• All participants get some benefit• Concern: increased likelihood of contamination
“Want to survey me? Then treat me”
• Sometimes it’s practically or ethically impossible to randomize program access
• But most programs have less than 100% take-up
• Randomize encouragement to receive treatment
• Encouragement: Something that makes some folks more likely to use program than others
Encouragement design: What to do when you can’t randomize access
Encouragement design
Encourage
Do not encourage
participated
did not participate
Complying
Not complying
compare encouraged to not encouraged
do not compare participants to non-participants
adjust for non-compliance in analysis phase
These must be correlated
Summary
Method Most useful when Advantages Disadvantages
Lottery •Limited resources•Piloting a new program
Easy to understand and implement
Control group may be upset
Phase in •Program expanding over time•Everyone must get treatment
•Easy to understand and explain •Control group not upset
•Anticipation of treatment •Long term effects
Rotation Everyone must get something
No control group upset •Spillovers more likely •More difficult to measure long term
Encouragement •Program has to be done everywhere•Take up low but can easily be increased
Can easily randomize at individual level
•Need very effective incentive•Impact on those who respond to incentive
• Quick Review: Why Randomize• Choosing the Sample Frame• Choosing the Unit of Randomization
• Options• How to Choose
• Randomization Methods• Mechanics of randomization • Stratification• Variations on Simple Treatment-Control
Lecture Overview
• Need sample frame• Pull out of a
hat/bucket
Mechanics of randomization
40
Source: Chris Blattman
• Need sample frame• Pull out of a
hat/bucket• Use random number
generator in spreadsheet program to order observations randomly
• Stata program code
Mechanics of randomization
41
Source: Chris Blattman
• Need sample frame• Pull out of a hat/bucket• Use random number
generator in spreadsheet program to order observations randomly
• Stata program code• What if no existing list?
Mechanics of randomization
43
Source: Chris Blattman
• Quick Review: Why Randomize• Choosing the Sample Frame• Choosing the Unit of Randomization
• Options• How to Choose
• Randomization Methods• Mechanics of randomization • Stratification• Variations on Simple Treatment-Control
Lecture Overview
• Objective: balancing your sample
• Ultimately increases your power
• What is it: – dividing the sample into different subgroups– selecting treatment and control from each subgroup
• What happens if you don’t stratify?
Stratification
45
Stratification
Village Type Status
uyufssg Low lit T
Adjkjdk Low lit C
afiudiufa High Lit T
aduyu Low lit C
afjh High lit T
askfjkfjk Low lit C
fdkvofj High lit T
daisudimv Low lit C
vadlsvkoi Low lit T
Vaofdivofdai Low lit C
caidsudiau High Lit T
Stratification
Village Type Status
uyufssg Low lit T
Adjkjdk Low lit C
afiudiufa Low lit T
aduyu Low lit C
afjh Low lit T
askfjkfjk Low lit C
fdkvofj Low lit T
daisudimv High Lit T
vadlsvkoi High Lit C
Vaofdivofdai High Lit T
caidsudiau High Lit C
• Stratify on variables that could have important impact on outcome variable (bit of a guess)
• Logistics• Stratify on subgroups that you are particularly
interested in (where may think impact of program may be different)
• Stratification more important when small data set• Can get complex to stratify on too many variables• Makes the draw less transparent the more you
stratify
When to stratify
48
• Quick Review: Why Randomize• Choosing the Sample Frame• Choosing the Unit of Randomization
• Options• How to Choose
• Randomization Methods• Mechanics of randomization • Stratification• Variations on Simple Treatment-Control
Lecture Overview
• Sometimes core question is deciding among different possible interventions
• You can randomize these programs• Test different components of treatment in
different combinations• Test whether components serve as substitutes
or compliments• What is most cost-effective combination?
Multiple treatments
How to Randomize, Part I - 50
Treatment 1
Treatment 2
Treatment 3
Multiple treatments
Mother Literacy Research Design
Randomization
Intervention 1:
Mothers Literacy Classes
60 villages
Intervention 2:
Train Mothers to Interact with
Children’s Education
60 Villages
Intervention 3:
Combine Intervention 1 and
2
60 Villages
Comparison group
60 Villages
240 Elligible Villages
In Each State:
• Some schools are assigned full treatment– All kids get pills
• Some schools are assigned partial treatment– 50% are designated to get pills
• Testing subsidies and prices• E.g: Information campaign - how much do you
need to spread information?
Measuring Spillovers: Varying levels of treatment
• Combined
Measuring Spillovers: Double randomization
155 clusters
Treatment schools Control schools
Random Assignment
Treatment individualsControl individuals
Control individuals