Date post: | 27-Dec-2015 |
Category: |
Documents |
Upload: | david-rich |
View: | 215 times |
Download: | 0 times |
TRANSLATING RESEARCH INTO ACTION
What is Randomized Evaluation?Why Randomize?
J-PAL South Asia, April 29, 2011
Life Cycle of an Evaluation
< Needs assessment > <~~~~~~~~~~~ Process Evaluation ~~~~~~~~~~~~> <~~~~~~~~~ Impact Evaluation ~~~~~~~~>Other types of Evaluations: Review / Cost-benefit analysis / Cost-effectiveness Analysis
Balsakhi Program
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
• Class sizes are large• Many children in 3rd and 4th standard are not
even at the 1st standard level of competency• Social distance between teacher and some of
the students is large
What was the issue?
• Hire local women (balsakhis) from the community
• Train them to teach remedial competencies– Basic literacy, numeracy
• Identify lowest performing 3rd and 4th standard students– Take these students out of class (2 hours/day)– Balsakhi teaches them basic competencies
Proposed solution
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
Case 2: Remedial Education in IndiaEvaluating the Balsakhi Program
Incorporating random assignment into the program
Pros• Reduced class size• Teaching at appropriate
level• Reduced social distance
• Improved learning for lower-performing students
• Improved learning for higher-performers
Cons• Less qualified• Teacher resentment• Reduced interaction with
higher-performing peers
Possible outcomes
• We want to look at impact• What do we do?
Study design
What is the impact of the balsakhi?
What would have happened without the balsakhi program?
• Look at average change in test scores over the school year for the balsakhi children
Pre-post (Before vs. After)
What was the impact of the balsakhi program?
Before vs. AfterImpact = 26.42 points?
75
50
25
0
02002 2003
26.42 points?
Impact: What is it?
Time
Prim
ary
Out
com
e
Impact
Counterfactual
Intervention
Impact: What is it?
Time
Prim
ary
Out
com
e
ImpactCounterfactual
Intervention
Impact: What is it?
Time
Prim
ary
Out
com
e
ImpactCounterfactual
Intervention
Impact is defined as a comparison between:
1. the outcome some time after the program has been introduced
2. the outcome at that same point in time had the program not been introduced
the ”counterfactual”
14
How to measure impact?
1. Randomized Experiments • Also known as:
– Random Assignment Studies – Randomized Field Trials – Social Experiments– Randomized Controlled Trials (RCTs)– Randomized Controlled Experiments
15
Impact evaluation methods
2. Non- or Quasi-Experimental Methods a. Pre-Postb. Simple Differencec. Differences-in-Differencesd. Multivariate Regressione. Statistical Matchingf. Interrupted Time Seriesg. Instrumental Variablesh. Regression Discontinuity
16
Impact evaluation methods
• There are more sophisticated non-experimental methods to estimate program impacts:– Multivariable Regression– Matching– Instrumental Variables– Regression Discontinuity
• These methods rely on being able to “mimic” the counterfactual under certain assumptions
• Problem: Assumptions are not testable
Other Methods
• We constructed results from the different ways of estimating the impacts using the data from the schools that got a balsakhi
1. Pre – Post (Before vs. After)2. Simple difference3. Difference-in-difference4. Other non-experimental methods5. Randomized Experiment
Methods to estimate impacts
TRANSLATING RESEARCH INTO ACTION
II – What is a randomized experiment?
The Balsakhi Example
povertyactionlab.org
The basics
Start with simple case: • Take a sample of schools• Randomly assign them to either:
Treatment Group – is offered treatment Control Group - not allowed to receive treatment
(during the evaluation period)
20
• Suppose we evaluated the balsakhi program using a randomized experiment
• QUESTION #1: What would this entail? How would we do it?
• QUESTION #2: What would be the advantage of using this method to evaluate the impact of the balsakhi program?
21
6 – Randomized Experiment
Source: www.theoryofchange.org
How to Randomize, Part I - 22
Random assignment in Vadodara
2006
Baseline Test Scores
20
10
0Treat Compare
28 28
Key advantage of experiments
Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment,
any difference that subsequently arises between them can be attributed to the program rather than to other factors.
23
• Pratham had received funding to introduce balsakhis in all schools
• Principals wanted balsakhis in their school• Teachers wanted balsakhis in their class• If denied a balsakhi, they may block data
collection efforts
Constraints of design
Random assignment
Round 1Red: Std. 3Blue: Std. 4
Round 1Red: Std. 3Blue: Std. 4
Rotation design
Round 2Std 3 from Round 1 Std 4 in Round 2——————————————————————————
Std 4 from Round 1 Std 3 in Round 2
Round 1Red: Std. 3Blue: Std. 4
ComparisonRed: TreatmentBlue: ControlStd 3Std 4
Rotation design
The counterfactual:
School Groups
Year 1 (2001-2002) Year 2 (2002-2003)
Grade 3 Grade 4 Grade 3 Grade 4
Group A Balsakhi No Balsakhi No Balsakhi Balsakhi
Group B No Balsakhi Balsakhi Balsakhi No Balsakhi
Key steps in conducting an experiment
1. Design the study carefully
2. Collect baseline data
3. Randomly assign people to treatment or control
4. Verify that assignment looks random
5. Monitor process so that integrity of experiment is not compromised
29
Key steps in conducting an experiment (cont.)
6. Collect follow-up data for both the treatment and control groups
7. Estimate program impacts by comparing mean outcomes of treatment group vs. mean outcomes of control group.
8. Assess whether program impacts are statistically significant and practically significant.
30
Impact of Balsakhi - Summary
Method Impact Estimate
(1) Pre-post 26.42*
(2) Simple Difference -5.05*
(3) Difference-in-Difference 6.82*
(4) Regression 1.92
*: Statistically significant at the 5% level
Impact of Balsakhi - Summary
Method Impact Estimate
(1) Pre-post 26.42*
(2) Simple Difference -5.05*
(3) Difference-in-Difference 6.82*
(4) Regression 1.92
(5)Randomized Experiment 5.87*
*: Statistically significant at the 5% level
Bottom Line: Which method we use matters!
Impact of Balsakhi - Summary
Method Impact Estimate
(1) Pre-post 26.42*
(2) Simple Difference -5.05*
(3) Difference-in-Difference 6.82*
(4) Regression 1.92
(5)Randomized Experiment 5.87*
*: Statistically significant at the 5% level
• This program improved average test score by nearly 10%
• The lowest-performing kids in balsakhi schools improved the most (relative to the control group)
• Highly cost effective: intervention cost less than Rs.100 per child and results far outweigh the cost
• Can be scalable, especially in light of RTE Act: remedial education
Results