Date post: | 27-Dec-2015 |
Category: |
Documents |
Upload: | nathan-page |
View: | 218 times |
Download: | 2 times |
Jennifer BryceInstitute for International Programs
The Johns Hopkins University
Institute for International Programs
ASADI VAccra, November 2009
Outline – What have we learned?
1. From the evaluation of the ACCELERATING CHILD
SURVIVAL AND DEVELOPMENT (ACSD) Program?
2. From prospective evaluations of the CATALYTIC
INITIATIVE TO SAVE ONE MIIILION LIVES (CI) to date?
3. The way forward
ACSD, 2002-2005
11 countries in Africa
Support from CIDA and other partners
Implemented through UNICEF
Aim: To reduce mortality among children less than 5 years of age
Strategy: Accelerate coverage with three packages of high-impact interventions, with a special focus on community-based delivery
Mali
ChadNiger
Nigeria
Cameroon
Central AfricanRepublic
Congo - Democratic Republic
Congo
SenegalCape Verde
Gabon
Equatorial Guinea
Sao Tome &Principe
GambiaGuinea Bissau
Guinea
Sierra Leone
Liberia
Burkina Faso
Ghana
TogoBenin
High Impact Package
EPI + Expansion
Accelerated Child Survival Accelerated Child Survival and and DevelopmentDevelopment
CIDA CIDA funded projectfunded project
Côte d’Ivoire
Mauritania
ACSD Program:Intervention packages
EPI+ Vaccinations Vitamin A supplementation ITNs for U5s & pregnant women De-worming
IMCI+ Facility IMCI Community case management
(CCM) of childhood illnesses Diarrhea: oral rehydration therapy
(ORT) Malaria: based on current policy Pneumonia: referral to facility
Promotion of timely initiation of breastfeeding, exclusive breastfeeding to 6 months, timely complementary feeding
Promotion of household consumption of iodized salt
ANC+– Malaria prevention in pregnant
women (IPTp)– Tetanus Toxoid– Iron/folic acid supplementation – Vitamin A post-partum– PMTCT
The retrospective independent evaluation of ACSD
High-impact districts in Benin, Ghana, Mali
Standard indicators
Existing DHS/MICS with oversampling
National comparison areas
Documentation of program implementation & contextual factors
No cost component
Stepwise design
Key: Bars represent districts in the following order: Builsa, Bawku East, Kasena-Nankana, Bolgatanga, Bawku West, Bongo
ACSD Implementation: GHANA
Coverage for EPI+ interventionsbefore and after ACSD, in HIDs
Benin
51
63
10
6
49
60
61
26
Measles
DPT
Vitamin A
ITNs
Ghana Mali
*
*
*
*
*
*
*
*
*
*
*Change was significant at p < 0.05.
Increases in coverage across the board in Ghana and Mali; Benin achieved increases for vitamin A and ITNs.
3
Before ACSD
After ACSD
Key
Coverage for IMCI+ interventionsbefore and after ACSD, in HIDs
Benin Ghana Mali
*
*
*
*
*
*
*
*Change was significant at p < 0.05.
No coverage gains, and some significant losses, in sick child care. Exclusive breastfeeding increased in Ghana, declined in Mali.
Before ACSD
After ACSD
Key
Coverage for ANC+ interventionsbefore and after ACSD, in HIDs
Benin
71
0
44
76
5
64
7
55
74
38
3+ antenatal care visits
IPTp with SP
Tetanus Toxoid
Skilled attendant at delivery
Postnatal vit A
Ghana Mali
*Change was significant at p ≤ 0.05.
*
*
*
*
*
*
*
*
*
*
** Measured level was 28%, but country team reported this was incorrect as IPTp had not been implemented in 2001.
Ghana and Mali improved care for childbearing women; delivery of TT and postnatal vit A benefited from EPI system in Mali.
*
**
Before ACSD
After ACSD
Key
Under-five mortality in the ACSD HIDs
19% (p=0.10)
Under-five mortality in the ACSD HIDs and national comparison areas
Declines in U5M in ACSD focus districts, but not greater than national comparison areas.
141
107
260
145
248
123
86
197
109
172
0
40
80
120
160
200
240
280
Benin Ghana Mali
Un
de
r-fiv
e m
ort
alit
y (p
er 1
00
0 li
ve b
irth
s)
Comparison area Comparison area
Jul 1999-Jun
Jan 2004-Dec 2006
Jul 1998-Dec 2001
Jul 1998-Dec 2001
Jul 2003-Dec 2006
Jan 2004-Jul 2007
141
107
260
145
248
123
86
197
109
172
0
40
80
120
160
200
240
280
Benin Ghana Mali
Un
de
r-fiv
e m
ort
alit
y (p
er 1
00
0 li
ve b
irth
s)
Comparison area Comparison area
Jul 1999-Jun
Jan 2004-Dec 2006
Jul 1998-Dec 2001
Jul 1998-Dec 2001
Jul 2003-Dec 2006
Jan 2004-Jul 2007
No changes in child nutritional status
attributable to ACSD.
Did ACSD implementation contribute to reducing inequities?
Yes, in Mali, where socioeconomic and urban/rural inequities decreased more in the ACSD HIDs than in the comparison area.
Baseline sample sizes too small to support analysis of equity trends in Benin or Ghana.
Socioeconomic inequalities, showing breakdown by wealth quintiles of ANC 3+ coverage in ACSD “high-impact” zones and the comparison area, Mali, 2006-7.
3+ antenatal visits
0%
20%
40%
60%
80%
100%
Poorest 2nd 3rd 4th Richest
Cov
erag
e (%
)
HID (before) Comp (before) HID (after) Comp (after)
Conclusions & implications
1. Intervention coverage CAN be accelerated if there is adequate funding & human resources.
2. Acceleration of mortality declines require:
a) Focus on interventions that have a large and rapid impact on major causes of child death
b) Sufficient time to fully implement approach and for coverage to translate into declines in mortality
c) Reasonable expectations, given level of resources
► Work for closer match between program resources & cause of death
► Be realistic about what can be accomplished
► Level of funding matters
Conclusions & implications
3. Policy barriers prevented key ACSD interventions directed at pneumonia and malaria from being fully implemented.
4. Breakdowns in commodities and gaps in funding stall progress toward impact.
5. More attention and operations research needed on incentives and supports for community-based workers
► Work for policy reform as first step, where needed
► Pay attention to health systems supports such as commodities, supervision, & incentives
Contributors & acknowledgements
Contributors
Jennifer Bryce
Kate Gilroy
Elizabeth Hazel
Gareth Jones
Robert Black
Cesar Victora
AcknowledgementsMinistries of Health, National Statistics Offices, UNICEF country staff, Collaborators in documentation
UNICEF regional and global staff Genevieve Begkoyian, Mark Young, Sam Bickel
Technical consultants Trevor Croft, Macro International
UNICEF leadership For their commitment to learning and change
EVALUATING THE CATALYTIC INITIATIVETO SAVE A MILLION LIVES
Part 2
Independent Evaluation of the MNCH Rapid Scale-Up
Overall objective: Provide “proof of concept” that proven interventions can be scaled up rapidly to reduce newborn and child mortality.
Supported by: BMGF Implementing partners: Governments and
UNICEF, WHO, UNFPA
“Real-time” Mortality Monitoring (RMM)
Overall objective: To monitor changes in under-five mortality in real-time.
Countries: Burkina Faso, Malawi, Mozambique
Countries: Ghana, Malawi, Mali, Mozambique
Two Linked EvaluationsThe Catalytic Initiative
Supported by: CIDA Implementing partner:
Governments and UNICEF
Process of evaluation design
Progress: MalawiIn-country partners: Centre for Social Research and National Statistics Office
Implementation Features of accelerated approach:
Government-paid CHWs trained to deliver CCM for pneumonia, malaria, diarrhea (including zinc)
Strengthening district health management
Implementation status:
In 10 intervention districts, 5-15% of CHWs trained by June 2009
Evaluation (full) Mortality monitored by:
Having CHWs report vital events
Calibrating facility deaths against community deaths
Two rapid survey approaches
Full documentation of program & contextual factors
Quality of care assessments at 1st-level facilities and for CHWs
Costs & equity tracked 12 districts: 6 “accelerated” and 6
routine National platform approach under
discussion
Progress: MozambiqueIn-country partner: Eduardo Mondlane University
Implementation Features of accelerated approach:
Increased access to quality child health care in facilities
Quality of immunization services improved
Long-lasting insecticide-treated nets (ITNs) distributed and used
Vitamin A supplementation
Breastfeeding promotion
BMGF funds used to fill gaps in maternal health
Implementation status:
CI planned to be implemented in 33 districts each year from 2008 to 2012
Evaluation (full)Mortality monitoring by calibrating facility to community deaths.
Stepped-wedge design based on scheduled cohorts for Rapid Scale Up
Documentation of baseline health & nutrition, inputs & contextual factors and coverage for each cohort using national evaluation platform approach
Will support dose (program intensity) – response (coverage & modeled impact) analyses
Progress: Burkina FasoIn-Country Partner: Institut Supérieur des Sciences de la Population (ISSP)
Implementation Features of accelerated approach:
Volunteer community health workers
providing:
• CCM for diarrhea (ORT + zinc) and malaria (ACT) in 9 districts
• CCM for pneumonia in 2 districts
Strengthened district planning and
supervision
Implementation status: Materials ready; cascade training
of all CHWs in 9 districts to be completed before end 2009
Evaluation (no RMM)National platform approach in 9 intervention and 2-3 comparison districts
New “LiST” survey to collect district-level estimates of coverage for proven MNCH interventions
Modeled mortality using LiST
Analysis using pre-/post-intervention with comparison and/or dose response
*If targets fully achieved at adequate service quality.
Lesson 1:Existing plans include high-impact interventions…
…but if feasibility and speed are issues,just 4 or 5 interventions can achieve ≥ 20% reduction in U5M by 2015
Malawi Burkina Faso GhanaNumber of interventions included in national plan
18 13 20
Number to ≥ 20% reduction in U5M4 5 5
Interventions (current and target coverage levels)Pneumonia treatment with antibiotics
(29; 67)
(30; 50)
(33; 60)
Diarrhea treatment with ORS and zinc (55; 85)
(41; 60)
(42; 60)
Malaria prevention with insecticide-treated nets
(23; 69)
(10; 70)
(40; 55)
Malaria treatment with ACTs (27; 69)
(48; 57)
(65; 70)
Vitamin A supplementation (67; 90)
Improved sanitation (18; 70)
Pre-publication results; not for citation or distribution
Lesson 2:Implementation takes time
Implementation status of functional village health clinics with CHW trained in CCM, Malawi CI districts, June 2009(18 months after project start-up)
*1 trained CHW per village health clinic
…especially when policy reform is needed.
In Mali, the MoH scheduled a “forum” to decide on CCM for childhood pneumonia & malaria.
July 2008
November2009
November2008
February2009
March2009
Original date
(cancelled)Planned
(cancelled) Planned
(cancelled)
Forum held; agreed
“YES” on CCM
Months 4 3 1 7 months+ + =
Discussions about how to
implement are still under way
in a 3-year CI project
Lesson 3:“Virgin” comparison areas do not exist
Mozambique
Simultaneous implementation of multiple programsSeparate, uncoordinated, inefficient evaluations, if any
Lesson 4: There are no shortcuts for mortality measurement (at least not yet)
Capturing a 25% difference-in-difference for rates of child mortality in a two-year period requires a survey of ≈ 12,300 households in each group*
Promises of measuring declines in 1 year using survival analysis or other techniques still require these prohibitively large sample sizes, plus detailed info on age of death
CI work on “real-time” mortality monitoring will assess the validity of alternative methods, but in first trials require validation against a gold standard
*based on Malawi; sample sizes will increase as mortality rates decrease, e.g. in Ghana
RMM Options by countryOptions for RMM methods
Data collection at community level Deaths recorded
in facilities vs. com-munity survey
Vital registra-
tion program
New methods for using surveys
Vital events
reporting at Child Health Days
Paid Gov’t health
workers
Unpaid Gov’t health
workers
Lay volun-teers
Ghana Malawi Mali Mozambique
Contributors
In-country partners
Agbessi Amouzou, Abdullah
Baqui, Robert Black, Jennifer
Bryce, Kate Gilroy, Elizabeth
Hazel, Gareth Jones, Marjorie
Opuni, Jeremy Schiefen, Cesar
Victora, Damian Walker
IIP-JHU
Burkina Faso: ISSP, INSP
Ghana: Noguchi Institute, University of Ghana
Malawi: NSO, CSR, Department of Economics, University of Malawi
Mali: CREDOS
Mozambique: Eduardo Mondlane University
THE WAY FORWARD:NATIONAL EVALUATION PLATFORMS (NEPS)
Part 3
What is a national evaluation platform (NEP)?
District-level databases covering the entire country
Containing standard information on: Inputs (partners, programs, budget allocations, infrastructure) Processes/outputs (DHMT plans, ongoing training,
supervision, campaigns, community participation, financing
schemes such as conditional cash transfers) Outcomes (availability of commodities, quality of care
measures, human resources, coverage) Impact (mortality, nutritional status) Contextual factors (demographics, poverty, migration)
Permits national-level evaluations of multiple simultaneous programs
NEPs: A common evaluation framework
Common principles (with IHP+, Countdown, etc.)
Standard indicators Broad acceptance
NEPs: Sound evaluation principles
In-country evaluation counterparts Local expertise, able to provide continuing evaluation research
support to the MOH Continuity of inputs from evaluation team; cross-country
network of investigators
Linked “independence” Investigators not involved in implementation of MNCH activities Regular exchange with in-country implementation team Ongoing activity; not one-off approach
Attribution by approach Documentation of all contributions Comparison of accelerated approach with “routine” approach
What types of questions can an NEP answer?
? Are programs being deployed where need is greatest?
? Is implementation strong enough to have an impact?
? Did programs increase coverage?
? Was coverage associated with impact?
? How equitable are the programs?
? How much did programs cost?
How can the MOH and partnersuse the platform?
To learn from well-performing districts and guide those doing less well
? Which approaches or combinations are contributing to rapid scale-up?
? Are some districts more efficient than others? Why?
? Are changes in epidemiology (e.g., due to IRS) reflected in reallocation of resources in district plans?
Why should you consider a national platform approach (or not) ?
AdvantagesAdapted to current reality of
multiple simultaneous programs/interventions and partners
Flexible design allows for changes in implementation
Can be used to evaluate multiple programs (child survival, HIV, malaria, maternal health, etc.)
Supports country ownership and capacity building
LimitationsObservational design (but no
other alternative may be possible)
Cost, particularly due to large size of surveys (!But cheaper than many standalone surveys!)
Requires transparency and collaboration by multiple programs and agencies
Further details at www.jhsph.edu/iip and
www.cherg.org
Thank you