Measuring the Impact of the Second Urban Primary Health Care
Project in Bangladesh
Manuel Leonard F. Albis, University of the Philippines
Subrata K. Bhadra, National Institute of Population Research and Training
Brian Chin, Asian Development Bank
1
Outline of Presentation
1. Introduction– Background on UPHCP-II– Public-private partnership– Monitoring and evaluation component
2. Methods used to determine project impact– Recalibration of baseline sampling weights– Propensity Score Matching Difference-in-Differences
3. Results and discussions4. Conclusion and recommendations
2
UPHCP-II in Bangladesh
• Second Urban Primary Health Care Project (UPHCP-II) (2005-2012), the follow-up phase to UPHCP, developed by the Government of Bangladesh with support from ADB, SIDA, DFID, UNFPA, and ORBIS International
• Four main outputs:1. Provide PHC services through partnership agreements and
BCCM 2. Strengthen urban PHC infrastructure and environmental health 3. Build capacity and policy support for urban PHC4. Support research into project implementation and operations
research
3Introduction
UPHCP-II in Bangladesh
• Government facilitated 24 partnership agreements, for areas covering 200,000 to 300,000 people, with NGOs
• Comprehensive reproductive health care center (CRHCC) was established in each partnership agreement area, providing full emergency obstetric care, newborn care, and other specialized services
• At least one primary health care center (PHCC) catering to 30,000 to 50,000 people, which provides basic emergency obstetric care
4Introduction
UPHCP-II Health Services
• Essential Services Package Plus (ESP+)1. immunization and growth monitoring of children 2. micronutrient support and malnutrition3. family planning4. prenatal, obstetric and postnatal care with special attention to
prevent eclampsia, STI and HIV/AIDS 5. other reproductive health, and child health
• Systematic case management of pneumonia, diarrhea, tuberculosis, leprosy, malaria, etc.
• Free health services and medicines to the poor; 32% of each major type of service was provided to poor patients
5Introduction
UPHCP-II BCCM Component
• Behavior Change Communication and Marketing (BCCM) component aimed to educate the urban population in order to increase their knowledge, improve their attitude, behavior and practices related to health
• Included comprehensive dissemination of information through posters, stickers, billboards, radio programs, and TV serials
• Capacity-building seminars and trainings were conducted to sustain effective and efficient delivery of health care with a focus on pro-poor and gender-sensitive targeting and monitoring
6Introduction
Contracting Health Services to NGO
• The public-private partnership (PPP) design to deliver health services has been found efficient and effective
• Advantages of this approach (Loevinsohn 2008) include:i. reduced “red tape” and unhelpful politicsii. constructive competition among NGOsiii. less administrative burden to the government
• Effectiveness of such design was documented for several projects in Bolivia, Guatemala, Haiti, India, Madagascar, Senegal and Pakistan (Loevinsohn and Harding 2005)
7Introduction
PPP in UPHCP: Major Findings
• Health services were contracted out to NGOs in Dhaka, Chittagong, Khulna, Barisal, Rajshahi and Sylhet city corporations, and other district municipalities
• Assessment of the UPHCP design (Heard, Nath & Loevinsohn2013) showed that NGO contracted areas were better than local government (Chittagong) contracted areas in terms of:
i. quality of care and higher health care services per capita
ii. wider coverage of the poorest 50% of the population
iii. higher likelihood of health seeking
iv. absorptive capacity to utilize budget
v. more motivation8Introduction
PPP in UPHCP: Major Findings
• The drivers of efficiency in the design of contracting-out health services are attributed to:
i. Proximate competition
ii. Direct procurement from approved suppliers
iii. Simpler decision making
iv. Flexibility in human resources
v. Financial management
vi. Close ties with community
9Introduction
UPHCP-II Monitoring and Evaluation
• UPHCP-II included an M&E component– Baseline and Endline surveys were conducted for each
of the three phases of the project for planned impact evaluation
• A sample of households from project areas (PA) and non-project areas (NPA) were included in the survey following a DHS-type questionnaire
• For UPHCP-II, the baseline and endline surveys were conducted in 2005 and 2012, respectively
10Introduction
UPHCP-II Baseline Survey
• The 2005 UPHCP-II baseline survey was not able to obtain a control sample due to operational problems
• To conduct impact evaluation, the 2006 Urban Health Survey (UHS) was used as the baseline for UPHCP-II; it has a DHS-type questionnaire, making the estimation and comparison of health outcome indicators possible
• The domains between the two surveys are different, thus adjustments were made to establish comparability
11Introduction
Objectives and Significance of the Study
• This study evaluates the impact of UPHCP-II on selected health outcomes using the 2006 UHS as baseline and 2012 UPHCP-II Endline survey
• The effect of the project was estimated through difference-in-differences (DID) with propensity score matching (PSM) between designated project and non-project areas
• The innovation introduced by this paper is the recalibration of sampling weights that allows the use of two unrelated surveys in impact evaluation
12Introduction
METHODOLOGY IN ASSESSING IMPACT OF UPHCP-II
13
Reweighting the 2006 UHS
• The 2006 UHS and 2012 UHPCP-II endline surveys do not have the same domain structure
• To ensure comparability, 2006 UHS sampling design was transformed to mirror the domain structure of the endlinesurvey (recalculating the 2006 UHS survey weights by assuming the sampling procedure was that of the endlinesurvey)
• The rationale is that the sampling weights reflects the sampling design of the survey, thus recalculating the sampling weights restructures the sampling design in order for the two surveys to be comparable
14Methodology
Reweighting the 2006 UHS
15Methodology
Domains in UPHCP-II consist of wards
UHS Design is different from UPHCP-II
Regroup wards in UHS to match the
domain structure of
UPHCP-II
• For illustration, suppose that Domain 1 of UHS contains 8 wards, which belong to the first five Domains of UPHCP-II Endline
Sampling weights of
UHS are recomputed
following UPHCP-II
design
Domain 1Wards 1 & 2
Domain 2Ward 4
Domain 3Wards 5 & 6
Domain 4Ward 7
Domain 5Ward 8
UPHCP-II Design
Ward 2
Ward 4
Ward 6
Ward 7
Ward 8
UHS Design
Ward 1
Ward 5
Ward 3
Ward 2
Ward 4
Ward 6
Ward 7
Ward 8
Revised UHS Design
Ward 1
Ward 5
DROPPED
D1
D2
D3
D4
D5
The Baseline and Endline Surveys
• The 2006 UHS surveyed 14,191 women with eight main domains representing slum and nonslum areas across Dhaka City Corporation, Other City Corporations, and municipalities
• The 2012 UPHCP-II Endline survey has 21,269 women respondents, having a total of 32 PA domains with corresponding 32 NPA domains
• In matching, the sampling design between the two surveys, individuals located inside the specified PA and NPA areas in UPHCP are included in the analysis; 2,405 individuals or 16% of the sample were outside the designated PA and NPAs
16Methodology
Number of Women in PA and NPA
Area2006 2012
PA NPA PA NPADhaka 2,046 1,769 3,314 3,320 Khulna 798 794 711 707 Rajshahi 551 349 746 711 Barisal 398 215 347 341 Sylhet 299 234 363 387 Chittagong 3,434 334 1,092 1,143 Comilla 61 28 375 373 Narayanganj 62 - 358 362 Rangpur - 86 390 383 Municipalities 124 204 2,872 2,974 Total 7,773 4,013 10,568 10,701
17Methodology
DCC PA and NPA Map
• There are 20 survey domains in DCC, 24 in OCC and 16 in municipality areas
• All PAs in DCC have their corresponding NPAs
18Methodology
Propensity Score Matching
• PSM was applied to health outcome indicators by matching using household and individual characteristics; correcting selection bias is the main objective of this approach
• The approach is to perform PSM by survey period between PA and NPA areas, using individual and household characteristics as matching variables
• Treatment effects were then computed between the baseline and endline surveys using difference-in-differences
19Methodology
Estimating Propensity Scores
20Methodology
Nearest Neighbor Matching
21
0 1
Propensity Score
Shortest Distance
Nearest Neighbor to
Individual in the treatment group is matched to an individual in the control if they have almost the same propensity score
Methodology
Impact Evaluation
• PSM variables that were used for matching women are (1) wealth index, (2) age, (3) parity, (4) highest educational attainment, (5) religion, and (6) major geographic grouping (DCC, OCC and Municipalities)
• For matching children, variables used were (1) age of child, (2) gender of child, in addition to the women variables given above
• The treatment group is a sample of individuals from designated PA and not restricted to individuals who have accessed UPHCP-II health facilities
22Methodology
Matched Difference-in-differences (DID)
23Methodology
Matched DID
24
• DID captures the effect of unobservable characteristics that may affect the outcome of the project
• DID assumes that change in health outcomes of counterfactual is the same as that of the control group
• Estimated project impact is the change in health outcomes in the treatment group minus the change in the health outcomes in the control
RESULTS OF IMPACT EVALUATION
25
Child Nutrition
• Of the three indicators of child malnutrition, only stunting was significantly reduced
• Even though wasting and underweight indicators were not reduced significantly, all have negative estimated ATT
Health Indicator N ATT Boot SE
Stunting (height for age)cumulative effect of chronic malnutrition
3,512 -5.5*** 2.46 q
Wasting (weight for height)acute or recent nutritional deficit
3,608 -0.5*** 1.84
Underweight (weight for age)overall indicator of nutritional health
3,736 -2.8*** 2.28
** Significant at 5%
26UPHCP-II Impact
Diarrhea and ARI Prevalence
• No significant impact on diarrhea prevalence and access to health facility, although the signs of the estimated effects are in line with expectations
• ARI and Fever prevalence decreased significantly
* Significant at 10%; ** Significant at 5%; *** Significant at 1%
Health Indicator N ATT Boot SE
Diarrhea prevalence 5,050 -1.4*** 0.89
ARI prevalence 5,050 -5.4*** 1.26 q
Fever prevalence 5,050 -4.2*** 1.94 q
ARI and Fever prevalence 5,050 -5.6*** 1.15 q
27
ANC, PNC and Attended Births
Health Indicator N ATT Boot SE
ANC (at least 1 visit) 3,788 5.2*** 2.03 p
ANC (at least 2 visits) 3,788 5.3*** 2.28 p
ANC (at least 3 visits) 3,788 7.4*** 2.45 p
PNC (mothers who gave birth within 5 years preceding the survey)
3,788 -6.8*** 2.36 q
Attended births at health facility or at home
3,788 3.3*** 2.38
* Significant at 10%; ** Significant at 5%; *** Significant at 1%
• Coverage of antenatal care of at least three visits significantly increased
• Increase in postnatal care coverage in NPA areas from baseline to endlineis higher than the change in PA areas perhaps due to other programs present in NPA
• Attended births increased at a slightly higher rate in PA and NPA areas although not statistically significant
28
Breastfeeding
* Significant at 10%; ** Significant at 5%; *** Significant at 1%
• Proportion of ever breastfed children decreased from 2006 to 2012, also reflecting a significant negative ATT
• Significantly increased complimentary breastfeeding yields an estimated ATT of 3.8 percentage points
Health Indicator N ATT Boot SE
Ever Breastfed 2,057 -2.6** 1.16 q
Breastfed within 1 day of birth 2,057 0.9** 2.66
Complementary breastfeeding 6 months onwards
3,205 3.8** 2.23 p
29
CPR, RTI and AIDS
* Significant at 10%; ** Significant at 5%; *** Significant at 1%
• Modern contraceptive use among women also benefitted from information drive in reducing RTI
• RTI prevalence dropped
• Highly effective in behavioral change and communication on sexually transmitted infection such as AIDS
Health Indicator N ATT Boot SE
CPR (all women) 17,752 3.7*** 1.15 p
CPR (married women) 15,560 3.9*** 1.21 p
RTI Prevalence on Women 18,341 -1.4*** 0.82 q
AIDS Awareness 18,341 2.3*** 0.60 p
AIDS Avoidance 18,341 4.4*** 0.94 p
30
DISCUSSION OF UPHCP-II IMPACT
31
Effectiveness
• Stunting in children improved
• ARI and fever prevalence in children improved
• Antenatal care improved
• Knowledge dissemination changed the behavior of women as seen in the increase of modern contraceptive use and awareness of AIDS/HIV and its avoidance
32Discussion
Efficiency of Contracting Health Services
• UPHCP-II delivered health care services to 10.2 million clients; higher than the project target of 9.4 million
• UPHCP-II efficiently delivered health care services to the poor at reduced or even no cost at all despite utilizing only 80% of allocated funds
• Modality flexible in addressing the evolving needs and concerns of the health workers in terms of their tasks, incentives and supervisory mechanisms, enabled rapid and wide dissemination of health services
33Discussion
Quality of Care
• The project areas in Dhaka and other city corporations have high percentage of trained providers, while around 60% of the staff in municipality project areas received training
• Quality of health services is not highly correlated with lowest cost of health services, high maternal and child health service coverage, quantity of services, or duration of contract
• Quality of care is not uniform across all PA areas; Dhaka and Khulna delivered high quality of care, while municipalities generally had room for improvement
34Discussion
Satisfaction
• Exit client surveys report that around 3 out of 4 patients were satisfied with the health services of UPHCP-II (Midterm Exit Client Report 2010)
• Proximity and good quality of service were main factors
• Municipalities fared lower due to high cost per service, and unavailability of some health services
35Discussion
CONCLUSION AND RECOMMENDATIONS
36
Conclusion
• The project benefitted from the efficiency of contracting-out health services to NGOs—cost effective for:– Urban local bodies (ULBs) as technical and financial support were
provided by government and development partners vs. ULB budgets– PA NGOs’ technical capacity for PHC service delivery strengthened– Urban poor who received quality PHC for no or reduced cost
• Successful in terms of its effectiveness in delivering health services and thus positive impacts on various health indicators
• BCCM was crucial for comprehensive dissemination of health knowledge
37
Conclusion
• Monitoring of project performance through unified record keeping system (HMIS), integrated supervisory instrument surveys (QA), and field visits helped provide timely updates to NGOs
• Experience from UPHCP-II improved design, implementation, management, and delivery of health services for the follow-up phase of the project, Urban Primary Health Care Services Delivery Project (UPHCSDP)
38
Recommendations
1. Health-related– More efforts on malnutrition, breastfeeding, and post-
natal care
– Continuous dissemination of maternal health knowledge through rigorous behavioral change campaigns to improve attended births
– PA areas in municipalities should be strengthened to catch-up with the performance of the PA areas in city corporations
39
Recommendations
2. Monitoring and Evaluation– Enhanced and efficient M&E dashboard mechanism
with outcome, process and input-driven indicators for timely feedback to project implementers
– Survey design should be given greater thought at the onset to avoid changing of design and questionnaire during project implementation
– Standardized and up-to-date data repository of health statistics (HMIS) would help to efficiently measure urban health status
40
Recommendations3. Overall Project Implementation
– Poverty Targeting (transparent selection process for health entitlement cards; scheduled verification; update poverty definition regularly; provide unique beneficiary IDs to monitor health status across PA areas)
– Coordination across various programs minimizes redundancy of benefits and fully optimizes the delivery of health services
– The Urban Health Care Services Coordination Committee could include regular follow-ups and updates on implementation of various programs
41
Thank you.
42
Regions of Common Support
UHS
UPHCP
Baseline
Endline
Children Women
43
Health Status in Baseline and Endline
Health IndicatorPA NPA
2006 2012 2006 2012
Child Mortality
Neonatal Mortality 36.7 27.2 33.2 28.3Infant Mortality (1q0) 51.0 39.0 47.6 39.7Child Mortality (4q1) 15.7 10.4 9.3 10.4U5MR (5q0) 65.9 49.0 56.4 49.6
Child NutritionStunting Prevalence 42.3 40.8 39.3 38.1Wasting Prevalence 16.7 19.6 13.5 20.4Underweight Prevalence 36.5 32.8 30.9 30.7
Diarrhea & ARIDiarrhea Prevalence 6.3 2.2 5.4 3.0ARI and Fever Prevalence 11.6 3.6 10.0 4.4
ANC/PNCANC Coverage 51.8 59.7 52.7 56.2PNC Coverage 33.6 50.8 30.8 48.4
Sexually Transmitted
Infection
Modern Contraceptive Use 43.9 60.8 44.1 58.7AIDS Awareness 91.3 91.9 92.8 92.0RTI Prevalence on women 26.1 3.2 24.4 3.5
44
Reducing Child Mortality
65.9
36.7
51.0
15.7
49.0
27.2
39.0
10.4
U5MR Neonatal Mortality Infant Mortality Child Mortality
Baseline Endline
• The project achieved its target of reducing U5MR by 15%
• All child mortality indicators were reduced in project areas by more than 15% between 2006 and 2012
-25.6% -25.9% -23.5%
-33.8%
Source: UPHCP-II Project Completion Report
(0-5y) (0-1y) (0-4y)(0-27d)
45
42.3
16.7
36.540.8
19.6
32.8
Stunting - cumulative effect of chronic malnutrition
(h/a)
Wasting - acute or recent nutritional deficit
(w/h)
Underweight - overall indicator of nutritional health
(w/a)
Baseline Endline
Reducing Child Malnutrition
• The target of reducing child malnutrition by 10% between baseline and endline was achieved only for underweight children
• Incidence of wasting unexpectedly increased by 17.4% in project areas
Source: UPHCP-II Project Completion Report
-3.5%
17.4%
-10.1%
46
Reducing Diarrhea and ARI Prevalence
• ARI and fever incidence experienced a large reduction from 14.2% in the baseline to 3.7% in the endline
• Childhood diarrhea incidence was reduced from 6.3% in the baseline to 2.2% in the endline
Source: UPHCP-II Project Completion Report
36.7
14.2
6.3
20.7
3.7
2.2
Fever ARI Diarrhea
Baseline Endline
-43.5%
-73.7%
-65.9%
47
43.926.1
60.6 51.860.8
3.2
86.859.7
Modern Contraceptive Use
Reproductive Tract Infection
Knowledge on AIDS Avoidance
ANC Coverage
Baseline Endline
Improving Maternal Welfare
• UPHCP achieved various targets for maternal health in project areas between baseline and endline
Source: UPHCP-II Project Completion Report
38.5%
-87.9%43.2% 15.2%
Target60% endline
Target-20%
Target25% Target
60% endline
48
51.8
33.6
47.5
59.7
50.8
71.9
ANC PNC Attended BirthsBaseline Endline
ANC, PNC and Attended Births
• Pregnant women who had at least three ANC visits 59.7% in the endline is slightly short of the 60% project target
• The proportion of women who received PNC 50.8% in the endline is above project target of 50%
• The proportion of attended births either increased by 51.2%, more than the project target of 10%
Source: UPHCP-II Project Completion Report
15.2%
51.2%51.5%
(at least 3 visits)
49
• Reduction in breastfeeding may be due to emerging economic opportunities in the urban areas, as working women will less likely to breastfeed their children
• Complimentary breastfeeding 6 months onwards increased experienced a 98.5% surpassing the target of 52%
Breastfeeding
Source: UPHCP-II Project Completion Report
98.0
73.1
48.7
92.486.2
96.6
Ever breastfed Breastfed within 1 day of birth Complementary breastfeeding
Baseline Endline
98.5%
17.9%
-5.7%
50
