Date post: | 24-Dec-2015 |
Category: |
Documents |
Upload: | ross-dawson |
View: | 214 times |
Download: | 1 times |
povertyactionlab.org
Managing Threatsto Evaluations and Data Analysis
Michael KremerHarvard University
2
Outline
• Background: Worms and the Primary School Deworming Project
• Threats to Estimating the Impact of Deworming– Externalities– Compliance
• Deworming Treatment Effects on Health, Nutrition, and School Participation
• Cost Effectiveness• Conclusion
3
Background on Worms
• Worms infect ¼ of people worldwide:• Hookworm 1.3 billion• Roundworm 1.3 billion• Whipworm 900 million• Schistosomiasis 200 million
• Spread through poor hygiene and sanitation
• Prevalent among school-age children in developing countries
4
Impacts of Worms
• Health– listlessness, diarrhea, abdominal pain, anemia,
malnutrition– Light infections often asymptomatic
• Education: – Seen as a key issue in whether deworming should be
a priority in poor countries– Hypothesized to negatively affect cognitive ability and
academic performance– Little previous evidence on impacts of deworming on
school attendance
5
Prevention and Treatment
• Treatment:– Low-cost single-dose oral therapies (50 cents)– Every 6 months– Virtually no side effects.– Girls over 12 typically excluded
• Although there is effective treatment…– Worms often seen as ubiquitous, non-life threatening– Rapid re-infection rates implies treatment must be
regular
• Likely substantial externalities to treatment
6
Research Question
What are the impacts of deworming on health and education?
• A few randomized studies, but overall evidence on impact of deworming on cognition is inconclusive.
• Dickson et al., 2000 conclude that they cannot recommend massive school-based programs.
7
Previous Studies
Previous studies randomize deworming treatment within schools
– Suppose that deworming prevents the transmission of disease, what problems does this create?
– If external benefit of 10% less infection after 6 months, how much underestimate benefit?
– Suppose externalities are local? How can we measure total impact?
8
Externalities: Other contexts
• Programs may create spillover effects on untreated individuals– Physical spillovers
• De-worming program: disease reduction externalities
– Prices• Vermeersch and Kremer (2004): school meals
provision led other schools to reduce school fees
– Learning and imitation effects• Duflo and Saez (2004)
9
Meauring Treatment Effect with Local Externalities
• Form two villages
• Randomly assign $1 in economic development assistance within village
• 25 cent sharing
• What is measured treatment effect?
10
Measuring Externalities
• If we expect externalities that are all within school:– Randomization at the level of the school
allows for estimation of the overall effect of deworming on a school
11
Measuring Externalities
• If we expect externalities across schools or outside of schools– Need to use random variation in density of
treatment
12
Kenyan Deworming Program
• Primary School Deworming Project (PSDP)– 75 primary schools and 30,000 children– Single region of Kenya (Busia)
• Randomize deworming across schools:– Schools stratified by geographic zone and other assistance – Listed alphabetically– Every 3rd school assigned to groups 1,2 & 3
• Group 1: treatment starting in 1998 and 1999• Group 2: treatment starting in 1999• Group 3: treatment starting in 2001
– Balanced Control and Treatment schools
Treatment Group 1998 1999 2000 2001 Group 1 (25 schools) T T T T Group 2 (25 schools) C T T T Group 3 (25 schools) C C C T
13
Treatment Program• Education on worm prevention, hygiene and sanitation.• Treatment delivered by Kenyan MoH officials and ICS officers:
– Deworming drugs at 6-month intervals – All schools with geohelminth prevalence over 50% mass treated with
albendazole.– All schools with schistosomiasis prevalence over 30% mass treated with
praziquantel.– Treatment schools which did not meet these prevalence requirements,
infected pupils were treated individually.
• Not everyone treated in treatment schools– Girls 13 years and older not treated – Consent procedures in treatment schools
• 1998: ICS obtained community consent• 1999: Kenyan MoH required signed parental consent to receive treatment
14
Estimating Impacts of the Program
• Randomization implies program participation not correlated in expectation with individual characteristics or infection status.
• Controls include school average 1996 exam results, average prevalence in zone, indicators for school involvement in other NGO assistance, and grade cohort dummies.
• School-level random effect
15
Table 6: The direct health impact of deworming, January to March 1999, Group 1 schools (1998 Treatment) versus Group 2 schools (1998 Comparison)
Group 1 (Treatme
nt)
Group 2 (Comparis
on)
Group 1 – Group
2 Any moderate-heavy infection, 1998
0.38 - -
Any moderate-heavy infection, 1999 0.27 0.52 -0.25*** (0.06)
Hookworm moderate-heavy infection, 1999
0.06 0.22 -0.16*** (0.03)
Roundworm moderate-heavy infection, 1999
0.09 0.24 -0.15*** (0.04)
Schistosomiasis moderate-heavy infection, 1999
0.08 0.18 -0.10*
(0.06) Whipworm moderate-heavy infection, 1999
0.13 0.17 -0.04 (0.05)
Other Nutritional and Health Outcomes Sick in past week (self-reported), 1999
0.41
0.45
-0.04** (0.02)
Sick often (self-reported), 1999 0.12 0.15 -0.03** (0.01)
Height-for-age Z-score, 1999 (low scores denote undernutrition)
-1.13 -1.22 0.09* (0.05)
Weight-for-age Z-score, 1999 (low scores denote undernutrition)
-1.25 -1.25 -0.00 (0.04)
Hemoglobin concentration (g/L), 1999 124.8 123.2 1.6 (1.4)
Proportion anemic (Hb < 100g/L), 1999 0.02 0.04 -0.02**
(0.01) Worm Prevention Behaviors Clean (observed by field worker), 1999
0.59
0.60
-0.01 (0.02)
Wears shoes (observed by field worker), 1999
0.24 0.26 -0.02 (0.03)
Days contact with fresh water in past week (self-reported), 1999
2.4 2.2 0.2 (0.3)
Health impact after 1st round of treatment:
•Prevalence of worms significantly lower in Group 1
•27% versus 52% moderate-to-heavy infection
•Fewer Group 1 report having been sick
•Group 1 students have better height-for-age
•Worm prevention techniques education failed
16
Health Externalities(Two types: Within-school and between school)
• Localized, within school externalities– Rates of moderate-to-heavy infections substantially
lower in Group 1 pupils who did not receive medical treatment than among comparison students in Group 2 schools
– Incoming pre-school students had only half infection rate
– Moderate-to-heavy infection rates 10% lower for girls>12
17
Deworming treatment effects on health:
0
10
20
30
40
50
60
TreatedStudents
Comparablestudents incomparison
schools
Untreatedstudents intreatmentschools
Comparablestudents incomparison
schools
Moderate-to-Heavy Helminth Infection Rates
18
Cross-school externalities
– Can you look at number of treatment schools nearby?
– What is potential problem?– What need to control for?
19
Yij is health outcome of individual i in village j,
Ti is the indicator variable for individual assignment to the program,
Xij is a vector of village and individual characteristics,
Nit is the total number of individuals at distance d from i
is the number of individuals at distance d from i randomly assigned to the program TitN
ijtiitTitijtititijt euNNXTTY 2211
Empirical Specification
20
Empirical Specification
Treatment externality:
In words:
Change in outcome with each additional treated person living d distance away
Average effect of de-worming on treatment group in year 1:
Average effect of de-worming on treatment group in year 2:
ijtiitTitijtititijt euNNXTTY 2211
d
TitN1
TitN2
21
Results, controlling for density
– Infection rates 26 percentage points lower per 1000 pupils in Group 1 schools within 3 km
– Infection rates 14 percentage points lower per 1000 pupils between 3-6 km away
22
Compliance: Treatment
Some students in treatment schools not treated• 1998: 78% of pupils assigned to receive
treatment received at least some treatment• 1999: around 72%• Absence from school the major cause of non-
compliance• 19% of girls over 13 received treatment in 1998
– Confusion over age– Nurses
Some students in comparison schools treated• 5% received treatment outside of program
23
Compliance: Transfers
24
Potential Comparisons
• What if drop non-compliers from analysis?
• What if analyze based on actual treatment?
• What do you measure if compare based on assigned treatment?
25
Intention to Treat (ITT)
• Y(T) = The average number of days children in a deworming school came to school
• Y(C) = The average number of days children in comparison schools came
• Y(T)-Y(C) = the difference between the two is the “intention to treat” effect of the program.
• This is the difference between the people you intended to treat and those who you did not intend to treat.
26
• Intention to treat: “What happened to the average child who is in a treated school in this population?”
• What does “intention to treat” measure?• Is this the right number to look for?• Remember: In the deworming case many
children in treatment schools were not treated and many children in the control treatment were.
Intention to treat (ITT)
27
When ITT is useful?
May relate more to actual programs For example, we may not be interested in
the medical effect of deworming treatment, but what would happen under an actual deworming program.
If students often miss school and therefore don't get the deworming medicine, the intention to treat estimate may actually be most relevant.
28
What ITT does not measure
• Why might ITT not give good sense of what would happen with real program?
Why might take up be lower?
Higher?
Transfers?
29
What if you wanted to know effect of the medicine on the children who took the medicine?”
30
From ITT to Effect of Treatment of Treated (ITT)
• Initial assignment an instrumental variable for treatment– Correlated with actual treatment, uncorrelated
with other factors affecting outcomes
31
“Treatment on the Treated”
• The effect of the treatment on those who got the treatment:
• Suppose children who got the treatment attended school A days, irrespective of whether they were in a treatment or a control school
• Suppose children who got no treatment attended school B days, again in both kinds of schools
32
“Treatment on the Treated”
Then…• Y(T)=A*Prob[treated|T]+B(1-Prob[treated|T])• Y(C)=A*Prob[treated|C]+B(1-Prob[treated|C])• A-B= (Y(T)-Y(C))
(Prob[treated|T] – Prob[treated|C])• =The “treatment on the treated” effect.
33
How do we calculate this?
• We need to observe the number of people getting treated
Prob[treated|T] – Prob[treated|C]. • This may not always be easy. Why?• We don’t always need the entire sample:
– It may be enough to get it from some representative sub-sample.
• General technique for when compliance is not perfect, for example, in an encouragement design.
34
Impacts on Education:
Treatment associated with dramatic gains in school participation, large externalities
• Baseline absence rate (30%) reduced by ¼ • Total effect: Increase in school participation of
~0.14 years per child• Cost-effectiveness: $3.50/additional year of
schooling
35
Figure 2: School participation rate May 1998 to November 1999 for girls under 13 and all boys, difference between Group 1 and Group 3 (diamonds), and
difference between Group 2 and Group 3 (squares)
36
Estimating the Overall Effect
• Comparison schools: 1.5 percentage point increase in school participation, three Group 2 & 3 pupils for every treated child in Group 1.
• Treatment schools: 7 percentage point increase in school participation for all children, one untreated child for every 2 treated children.
• (.015 * 3) + (.07 * .5) + (.07 * 1) = .15 years• For each child treated, school participation
increased by .15 years.
37
Cost Effectiveness and Welfare Analysis
• Health cost effectiveness
• Educational cost effectiveness
• Human capital investment
• Externalities
38
Health Cost Effectiveness
• One of many worthy health interventions competing for scarce resources—annual government per capita expenditure on health is $5.
• DALY’s
• PCD cost per pupil treated is $0.49
39
Health Cost Effectiveness
• Treatment of schistosomiasis is very cost effective, but only meets the strictest standard when take into account the externalities present.
• $12 per DALY saved• The externality benefits account for 61.1% of the • DALY reduction• Treatment of geohelminths would not meet
standard cost– Effectiveness criteria in the poorest countries based
on health impact alone: $155 per DALY saved.
40
Educational Cost Effectiveness
What is the cost of an additional year of school participation?
• School participation increased by .15 years per treated child
• Cost is $.49 per child in a large-scale program.• Costs only $3.27 per additional year of school
participation.• Much cheaper than other ways of increasing
schooling.
41
Human Capital Investment What is the rate of return to deworming in future earnings?• Labor market benefits alone may far outweigh cost.• Excludes health and nutritional gains, thus understating benefits.• Knight and Sabot (1987) estimate the avg. labor market return to an
additional year of primary schooling in Kenya is 17%.• Years of schooling alone account for approximately 40% of the total
return to education controlling for scores.• If deworming increases participation but has minimal cognitive
gains, the direct return to an additional year of primary education would be 7%.
• Output per worker is $570• Discount rate of 10%• Assume opportunity cost is half adult wage• Deworming increases net present value of wages by more than $20
per treated child at a cost of only $0.49.
42
Externalities
• What subsidy would lead people to fully internalize the externalities present?
• Total net externality gain per child treated is at least $11.70, when cost of treating that child was $0.49.
• Government should heavily subsidize.
• Hard to know the optimal level of a subsidy without an epidemiological model.
43
Deworming program: follow-up
• “Sustainable” financing of deworming?– Health education, commitment ineffective– cost recovery leads to much less take up– Suggests a need for subsidies for de-worming
• Networks and social learning– Children with (randomly) more social links to
early treatment schools are themselves significantly less likely to take deworming drugs
44
References
• Edward Miguel and Michael Kremer (2004) “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities,” Econometrica, 72(1): 159-217.– Kremer and Miguel (2004) “The Illusion of
Sustainability,” mimeo.– Miguel and Kremer (2004) “Networks and
Social Learning about Health,” mimeo.