34
Evaluation Design: Assignment of Treatment Megha Pradhan Policy and Training Manager, J-PAL South Asia Kathmandu, Nepal 29 March 2017
Evaluation Design:
Assignment of Treatment
Megha Pradhan
Policy and Training Manager, J-PAL South Asia
Kathmandu, Nepal
29 March 2017
What can be randomized?
• Access : We can choose which people will be offered
the program
• Timing of access: We can choose when to provide access to the program
• Encouragement: We can decide which people will be
provided encouragement to participate in the program
Types of Randomization Designs
• Simple lottery
• Phase in
• Encouragement
Types of Randomization Designs
• Simple lottery
• Phase in
• Encouragement
Randomly
sample
from area of
interest
Randomly
assign
to treatment
and control
Simple Lottery
Randomly
sample
from both
treatment and
control
Example: Slum relocation program in Gujarat
Tested the impact of relocating slum residents to suburban housing
on general well being, tenure security and children’s educational
attainment.
Barnhardt, Sharon, Erica Field, and Rohini Pande. “Moving to Opportunity or Isolation? Network Effects of a Randomized Housing Lottery in Urban India.” NBER Working Paper No. 21419, July 2015.
Eligible women who applied
(497 women)
Treatment
(110 women)
Control
(387 women)
Recap: Simple Lottery
Design Most useful
when… Advantages Disadvantages
Basic Lottery
• Program
oversubscribed
• Easy to understand
• Easy to implement
• Can be
implemented in
public
• Control group
may not
cooperate
• Differential attrition
Types of Randomization Designs
• Simple lottery
• Phase in
• Encouragement
Phase-in
• Takes advantage of program expansion (example: the NGO
cannot implement in all villages in the first year)
• Everyone gets program eventually
• Natural approach when expanding program faces resource
constraints
Phase in: Laptop Programme
Phase 0:
No one treated yet
All control
Phase 1:
1/4th treated
3/4ths control
Phase 2:
2/4ths treated
2/4ths control
Phase 3:
3/4ths treated
1/4th control
Phase 4:
All treated (experiment over)
Example: Primary School Deworming in
Kenya
Miguel, Edward, and Michael Kremer. 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities." Econometrica 72(1): 159-217.
School based mass deworming programme tested the impact of providing deworming drugs and worm prevention education on education outcomes
Rationale for Phase in: Logistical and financial constraints
75 Schools in Southern Busia District
Year 1 Group 1
(25 schools)
Group 2
(25 schools)
Group 3
(25 schools)
Year 2 Group 1 Group 2
Group 3
Year 3 Group 1 Group 2
Group 3
Recap: Phase-In
Design Most useful when…
Advantages Disadvantages
Phase-In
• Expanding over
time
• Everyone must
receive
treatment eventually
• Easy to
understand
• Constraint is
easy to explain
• Control group
complies
because they
expect to benefit later
• Anticipation of
treatment may
impact short-run
behavior
• Difficult to measure long-term impact
Types of Randomization Designs
• Simple lottery
• Phase in
• Encouragement
Randomized Encouragement
• Sometimes it’s not possible to randomize program access
(vaccines, savings program, entitlement programs etc.)
• But many programs have less than 100% take-up –
undersubscribed
• Randomize encouragement to receive treatment
• Treatment differential comes from higher program take-
up among the encouraged
What is “encouragement”?
• Something that makes some individuals more likely to use
program than others
– Not in itself a treatment
– E.g., SMS messages, information campaigns, visit from
agent, etc
Encouragement design
Encourage
Do not
encourage
Participated
Did not
participate
compare encouraged to
not encouraged
do not compare
participants to non-
participants
These must be correlated
Example: Household water connections in Tangier,
Morocco
Does an increase in access to water, with no change in water quality, change
household’s value of a private water connection and household’s welfare?
Devoto, Florencia, Esther Duflo, Pascaline Dupas, William Parienté, and Vincent Pons. 2012. "Happiness on Tap: Piped Water Adoption in Urban Morocco." American Economic Journal: Economic Policy 4(4): 68-99.
Randomized Encouragement Design
Amendis Loan
Program:
Both T and C were eligible for
interest free loans
from Amendis to
buy a home
water
connection, paid back over 3, 5, or
7 years.
Treatment Group:
• Door to door awareness campaign
to encourage households to buy a
connection
• Pre approval for the loan from the authorities
• Assistance in preparing paperwork
• Branch officer visited households to
collect down-payment
Comparison Groups:
• No application assistance given
Recap: Encouragement Design
Design Most useful when…
Advantages Disadvantages
Encouragement
• Program has
to be open to
all comers
• When take-
up is low, but
can be easily
improved
with an incentive
• Can randomize
at individual
level even
when the
program is not
administered at that level
• Measures impact of
those who respond
to the incentive
• Need large enough
inducement to
improve take-up
• Encouragement
itself may have
direct effect
Real World Challenges
J-PAL | HOW TO RANDOMIZE 25
Challenge Implication Solution
• Service provider
can’t distinguish
between T & C
• Crossovers
• Change Unit of Randomization
• Control group
finds out, benefits
• Spillovers
• Crossovers
• Attrition
• Change Unit of Randomization
• Resources to
treat all
• No control
group
• Phase in
• Program is an
entitlement
• Can’t
force/deny
program
• Encouragement Design
• Sample size is
small
• Insufficient
power
• Change unit of randomization
• Stratification
Real World Challenges
J-PAL | HOW TO RANDOMIZE 26
Challenge Implication Solution
• Service provider
can’t distinguish
between T & C
• Crossovers
• Change Unit of Randomization
• Control group
finds out, benefits
• Spillovers
• Crossovers
• Attrition
• Change Unit of Randomization
• Resources to
treat all
• No control
group
• Phase in
• Program is an
entitlement
• Can’t
force/deny
program
• Encouragement Design
• Sample size is
small
• Insufficient
power
• Change unit of randomization
• Stratification
When to stratify
1. When we want to achieve balance
2. When we need balance for political/logistical feasibility
3. When we want to increase statistical power
4. When we want to analyze the impact by subgroup
How to stratify
– Dividing the sample into different subgroups
– Do random assignment for each subgroup
– Assign to treatment and control
Stratification by Gender
Stratification by Gender:
Randomize Women separately
Stratification by Gender:
Randomize Women separately
Randomize the men separately
J-PAL | HOW TO RANDOMIZE 32
Stratification by Gender
J-PAL | HOW TO RANDOMIZE 33
Which stratification variables should we
use?
• Stratify on variables that could have important impact
on outcome variable
• Stratify on subgroups that you are particularly interested
in (where may think impact of program may be
different)
• Can get complex to stratify on too many variables
