Impact Evaluation Workshop for Health Sector Reform

Cape Town, South Africa, December 7-11, 2009

Impact Evaluation of Malaria

Prevention and Treatment

Jed Friedman (World Bank)

Evaluating Health

Programs is Different

• Evaluation methods often used in medicine to

determine efficacy of treatment

• What we know less about:

– How to get people to utilize prevention / treatment

services?

– What is the most cost effective mode of

prevention / treatment given behavioral

response?

– What are the socioeconomic effects of health

interventions?

PLASMODIUM

“THE AGENT”

MOSQUITO

“THE VECTOR”

MAN

“THE HOST”

Some Differences

• HIV/AIDS: Largely behaviorally driven

• Malaria: Vector-born disease

…But the behavioral aspects are important

Treat Infected

- Early Diagnosis &

Drug Type (AMT, ACT)

Problems

- Access

- Compliance

- Cost-Effectiveness

- Long-Term Effect

Vector Control

- Prevent Breeding (DDT)

- Prevent Entry (Proofing)

- Prevent Bite (ITN, Spray)

Problems

- Resistance to Insecticides

- Compliance

Protect Everyone

- ITN, LLIN

- Mosquito Proofing

Problems

- Valuation/Usage

- Compliance

Malaria Control

Is Not Principally a Question of

Technical Innovation

“Widespread use of ITNs and state-of-the-art drugs

succeeded in cutting malaria deaths half in 2 countries most

heavily affected by the disease, Rwanda and Kenya.”

Washington Post (01/31/08)

“This is a genuinely historic achievement. This is not

theoretical. We do not have to wait for a vaccine or new

drugs. If we implement today’s technologies aggressively on

a national scale we will have a big impact.”

Richard Feachem, former Director of the Global Fund

“With the resources available, we should be able to eradicate

malaria before I hang up my lab coat.”

Peter Agre, Malaria Research Institute, JHU

Impact of Policy Change on Malaria

Prevalence: South AfricaNational Geographic 07/07

Policy Regime Switches

1. Pesticide Resistance

- DDT Stops: 1996

- Spraying Resumes: 2000

Drug Resistance

- Multidrug Therapy: 2000

Aggravating Exogenous Factors

- Refugee Flow

- Heavy Rains

- Worsening Resistance

In Kwazulu-Natal the combined effect of switching

from SP to the fixed combination of AL and IRS

with DDT was associated with a decrease in cases

of 78% and an increase in cure rate of 87%

(Barnes K, unpublished data; in Yeung et al. 2004).

Case 1: DDT - IndiaQuasi Experiment (Cutler et al. 2007)

• Long-term Effects of Malaria Eradication

• Outcome Measure: Educational Gains 1. Literacy Rate (LR); 2.School Completion Rate (SCR)

• Method: Quasi Experiment using Diff-in-Diff

Sample: 300,000 Households, 1 million Individual Observations (NSS)

- Control: Pre-Eradication Cohorts (C0): 1912-1952

- Treatment: Post-Eradication Cohorts (C1): 1962-1972

- Omitted: Eradication Cohorts: 1953-1961

1947 Pre-Eradication- Population: 334 million

- Cases: 75 million (annual)

- Prevalence: 22% (annual)

- Mortality: 800,000 (annual)

- Mortality: 10% of total deaths

1953 Intervention1st: 1953 NMCP Launched

DDT Spraying

- 2 Rounds per Year

- 125 Malaria Control Units

2nd: 1958 NMEP Launched

Gains12% ↑ in LR, SCR

- Malaria explains

half of these gains

- Income ↓ through

malaria 7-10%

Case 2: ITN – Kenya

Field Experiment (Cohen and Dupas 2008)

• Explore tradeoffs between cost-sharing (CS) & free distribution for ITNs

• Randomize price of ITNs (0 ≤ p <pPrevailingCS) in prenatal clinics in Kenya

• Evaluate impact on pregnant women

1. Demand / Uptake

- Cost-sharing (C/S) does considerably dampen demand.

- Uptake Drops: i) by 75% from 0 price to prevailing CS price; ii) by 20% for ↑10Ksh.

2. Usage

- No evidence that C/S reduces wastage on those who do not use the net.

- Free ITN owner is not less likely to use net than those who paid higher prices.

- Coverage (Uptake + Usage): 63% (Free Net) v. 14% (40Ksh)

3. Need (Health)

- No evidence that C/S induces selection of those who need net more.

- Those paying higher prices appear no sicker (anemia) than control group.

4. Compare Cost Effectiveness (Externality Assumptions)

- Number of child lives saved highest under free distribution.

- Free distribution is more CE when externality threshold is medium level.Source: Cohen and Dupas 2008

1) Demand for ITNs: Monthly

Net Sales by ITN Price

0

20

40

60

80

100

120

140

160

180

200

0 10 20 40

ITN Price (in Ksh)

Avera

ge N

um

ber

of

ITN

s

So

ld/D

istr

ibu

ted

per

Mo

nth

Source: Cohen and Dupas 2008

2) ITN Usage Rates by Price: Share of

“Takers” who Report Using ITN at Home

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 10 20 40

ITN Price (Ksh)

All First Prenatal Visits Only First Pregnancy Only

Source: Cohen and Dupas 2008

Case 3: ITN – UgandaExperimental Evidence on ITNs from

Rubagano and Kimuli Villages (Hoffman1)

Do free goods stick to poor households?

Design

NHH=193 T1: Free Net: 71, T2: Cash Transfer: 72, C: Uncompensated: 50

Findings

- Wealth and endowment effects result in very few HHs selling free net (FN).

- Only 6% of FN would be sold in frictionless market. Accounting for

transaction cost would further reduce this number.

- No significant gender gradient in average compensated valuation of ITNs.

- Man have higher income elasticity of supply for ITNs. Men have higher ATP.

Implications

- Distributing FN to women ►less leakage.

- Marketing among men ► more effective.

Source: Hoffmann, Barrett, and Just (2007)

Case 3: ITN – UgandaExperimental Evidence on ITNs from

Rubagano and Kimuli Villages (Hoffman2)

Psychology, gender, intra-HH allocation of ITNs

Design:

NHH=143 T1: Free Net: 71, T2: Cash Transfer: 72

Findings

- Free nets lead to greater number of children covered, even for HHs with ATP.

- Net retention is higher for free nets (Endowment Effect): NFN > NCT

- Women tend to cover larger proportion of HH with nets.

- Intra-HH allocation of purchased nets (CT) depends on cost-benefit

calculations, with income-earners, net purchasers, and people often suffering

from malaria receiving it.

- Accompanied with a message, in-kind nets (FN) induce allocation to children.

Implications

Beyond price, mode of allocation and communication are important.

Source: Hoffmann (2007)

Overview of the World Bank’s Malaria

Impact Evaluation Program (MIEP)

• Program launched to conduct malaria policy relevant operational impact

evaluation under the Booster Program for Malaria Control

Goals:

• to generate evidence on effective approaches to increase utilization of

malaria prevention and treatment services

• to increase familiarity with IE approaches and build evaluative capacity in

national malaria control programs

•currently conducting evaluation studies in 6 countries assessing

effectiveness of a variety of control strategies

Access to effective treatment

Nigeria MIEP is assessing the involvement of trained community- and

private sector-based agents (CDDs and PMVs) in malaria prevention, and

case management to increase access to prompt diagnosis with RDTs and

treatment with ACTs.

Zambia estimating the gains in access through the introduction of RDTs

and ACTs through village Community Health Workers (CHW)

Access to effective treatment (cont)

Nigeria assessing the introduction of RDTs and ACTs to the workforce of

a large sugar cane plantation in order to estimate productivity costs of

adult malaria infection

India estimating the gains in quality of care through the involvement of

local organizations in supportive supervision and training of CHWs

Integrated vector control

Eritrea determining the cost-effectiveness of continued Indoor Residual

Spraying (IRS) in a low-endemic setting

School based programs

Kenya and Senegal asking whether the introduction of preventive

therapy through schools results in higher attendance and learning as well

as better student health

Overview of the World Bank’s Malaria

Impact Evaluation Program (MIEP) (cont.)

MIEP Portfolio SummaryCountry Project PIs Impact Evaluation Project Value

Eritrea IDF Grant – Capacity building for evidence-based policy making in the health sector

P. Carneiro

J. Keating

Experimental Design (RCT) of impact of indoor residual spraying, and incentives for improving larval habitat management

$485,000

India Leveraging local capacity to assist malaria control efforts

J. Friedman Experimental design of CBO assistance to government malaria prevention and fever case management initiatives

$200 million

Kenya School-based Malaria Prevention

S. Brooker

M. Jukes

Experimental Design (RCT) of teacher training and school-based intermittent preventive treatment

$4 million approved + $10.4 million Pipeline

Nigeria Malaria Control Booster Program

P. Carneiro

E. Velenyi

Experimental Design (RCT) of Community- and Private Sector-based Malaria Control

$180 million + $100 million Additional Financing

Senegal School-based Malaria Prevention

S. Brooker

M. Jukes

Experimental Design (RCT) of teacher training and school-based intermittent preventive treatment

$5 million

Zambia Zambia Access to ACTs Initiative

J. Friedman

E. Velenyi

Quasi Experimental Design (Matching and RCT) of Public Sector Supply Chain Management and Community- and Private Sector-based Malaria Control

$26.85 million

Conclusions

• Impact evaluation studies have already contributed much information for understanding the efficacy and cost-effectiveness of various malaria prevention options

• Recent impact evaluation studies will help policy makers work through certain malaria control decisions such as free ITN distribution vs. cost-recovery

• However many critical questions remain. For example, how do we affect people's behavior to ensure adoption and proper usage of nets? How do we increase the proportion of fever cases seeking treatment at facilities with adequate diagnostic and curative care?

• Impact Evaluation of Malaria Programs (See Handout)– Friedman, Legovini, and Velenyi; Development Dialogue Notes Vol. 1 (2009)

• This week we will discuss methods through which we can answer these questions.

References 1

1. Malaria Control http://www.malariasite.com/malaria/ControlOfMalaria.htm

2. Brown, D. “Malaria Deaths Drop in Rwanda, Kenya.” Washington Post January 31, 2008

3. Finkel, M. “Raging Malaria – Stopping a Global Killer.” National Geographic July 2007

4. Yeung et al. 2004. “Antimalarial Drug Resistance, Artemisinin-Based Combination Therapy, and the

Contribution of Modeling to Elucidating Policy Choices.” Am. J. Trop. Med. Hyg., 71(Suppl 2), 2004, pp.

179–186.

5. Cutler et al. 2007. “Mosquitoes: The Long-term Effects of Malaria Eradication in India.” NBER.

6. Cohen and Dupas. 2008. “Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria

Prevention Experiment.” Draft. January. Presented at the Brookings Institution. January 24, 2008.

7. Hoffmann, Barrett, and Just. 2007. “Do free goods stick to poor households? Experimental evidence on

insecticide treated bednets.” Draft. November. Department of Applied Economics, Cornell University.

8. Hoffmann. 2007. “Psychology, gender, and the intra-household allocation of free and purchased

mosquito nets.” Draft. November. Department of Applied Economics, Cornell University.

9. Clarke et al. 2008. “Health and educational impact of intermittent preventive treatment of malaria in

schoolchildren: a cluster-randomized controlled trial.” Draft. LSHTM.

10. Arrow, K.J. et al., Eds. 2004. “Saving lives, buying time: economics of malaria drugs in an age of

resistance.” Board on Global Health. Washington, D.C.: Institute of Medicine.

11. Laxminarayan, Over, and Smith. 2005. “Will a Global Subsidy of Artemisinin-Based Combination

Treatment (ACT) for Malaria Delay the Emergence of Resistance and Save Lives?” World Bank Policy

Research Working Paper 3670, July.

References 2

12. Gollin and Zimmermann. 2007. “Malaria: Disease Impacts and Long-Run Income Differences.”

Discussion Paper No. 2997. IZA.

13. McCarthy, Wolf, and Wu. 1999. “Malaria and Growth.” World Bank Working Paper. WPS 2303.

14. Gallup and Sachs. 2001. “The economic burden of malaria.” Am. J. Trop. Med. Hyg. 64(1,2)S: 1-11.

15. Acemoglu and Johnson. 2006. “Disease and development: The effect of life expectancy on economic

growth.” NBER Working Paper 12269.

16. Weil. 2007. “Accounting for the effects of health on economic growth.” The Quarterly Journal of

Economics. August. Vol. 122, No. 3, Pages 1265-1306.

17. Bleakley. 2007. “Malaria in the Americas: A retrospective analysis of childhood exposure.” Department

of Economics, University of Chicago.

18. Lucas. 2005. Economic Effects of Malaria Eradication: Evidence from the Malarial Periphery.

Manuscript. Department of Economics, Brown University.

19. Hong. 2007. "A Longitudinal Analysis of the Burden of Malaria on Health and Economic Productivity:

The American Case." University of Chicago.

20. Barreca. 2007. "The Long-Term Economic Impact of In Utero and Postnatal Exposure to Malaria." UC

Davis.

21. Coleman et al. 2004. “A Threshold Analysis of the Cost-Effectiveness of Artemisinin-Based

Combination Therapies in Sub-Saharan Africa.” Am. J. Trop. Med. Hyg., 71(Suppl 2), 2004, pp. 196–

204.

22. Breman et al. 2006. “Conquering Malaria.” Eds. Jamison et al. Disease Control Priorities. Chapter 21.

Impact Evaluation Workshop for Health Sector Reform

Cape Town, South Africa, December 7011, 2009