Impact Evaluation of Performance-based Financing for Health Services in Rwanda
Pay-for-Performance (P4P) for Health Services in RwandaPaulin Basinga Rwanda School of Public Health
Christel VermeerschWorld BankA collaboration between the Rwanda Ministry of Health, CNLS, SPH, INSP Mexico, UC Berkeley and the World Bank1Our teamResearch TeamPaulin Basinga, National University of RwandaPaul Gertler, UC BerkeleyJennifer Sturdy, World Bank and UC BerkeleyChristel Vermeersch, World Bank
Policy Counterpart TeamAgnes Binagwaho, Rwanda MOH and CNLSLouis Rusa, Rwanda MOHClaude Sekabaraga, Rwanda MOHAgnes Soucat, World Bank
The 2005 starting pointProfessionally assisted births: 40%Maternal Mortality: 750 per 100,000 live births Infant Mortality : 86 per 1,000 HIV : 3.1%
3Source: Rwanda 2005: results from the demographic and health survey. 2008. Studies in family planning, 39(2), pp. 147-152. 3Why a pay reform?44P4P for Health in RwandaObjectivesFocus on maternal and child health (MDGs 4 & 5)Increase quantity and quality of health services providedIncrease health worker motivationWhat?Financial incentives to providers tFor more quantityAnd more qualityHow?Contracts between government & health facilitiesWhen?Piloted in 2001-2005, full scale from 2006
55
6
7Evaluating P4P in Rwanda:Evaluation design8Evaluation QuestionsDid P4P improve
the quality and quantity of maternal and child health services?
the health of the population?
9Conceptual framework for qualityWhat they know (Ability/Technology)What They Do(Quality)Production Possibility FrontierProductivity Gap Conditional on AbilityActual Performance10Identifying the impact of P4PWhen we see a change in outcome, how do we know it is caused by P4P? And not by something else
Evaluate the impact = identifying a comparison groupEquivalent to the treated group in all aspectsExcept that they receive the treatment
Gold standard: randomized evaluation11Evaluation DesignPhased roll-out at district level Identified districts without P4P in 2005Group districts into similar pairs based on population density, location & livelihoodsRandomly assign one to treatment and other to controlPhase I: 12 districts, started 2006Phase II: 7 districts, started 2008 Unit of observation is health facility1212Rollout of P4P 2006 200813
13A few challengesThe decentralization surpriseA few new districts had some facilities with P4P must be treatmentExposure time to the treatmentIs it more money? Or more incentives?Other interventions & time trendsCoordination with multiple donors
14DataIndependent data
Facilities levelUtilization (quantity)Structural qualityKnowledge (vignettes)Process quality (patient exit surveys)
Household levelUtilizationProcess qualityHealth outcomes
15SampleOut of 30 districts12 Phase I (treatment)7 Phase II (comparison)165 health facilitiesAll rural health centers located in 19 districts2156 households in catchment areasPower calculations based on expected treatment effect on prenatal care visits, institutional deliveryPanel data: 2006 and 20081616Econometric modelBasic difference-in-differences model specified as a two-way fixed effect cross-sectional time-series regression models.
17
where :Yijt is the outcome of interest for individual i living in facility js catchment area in year t; PBFj,2008 = 1 if facility j was paid by PBF in 2008 and 0 if otherwise; j are facility fixed effects; 2008 =1 if the year is 2008 and 0 if 2006; Xitk are time varying individual characteristics; ijt is a zero mean error term.17Evaluation design challengesOrganizationalManaging expectationsThe John Henry effect in practiceBuilding capacityTime & effort
TechnicalSmall sample sizeReconciling provider and client data
18Baseline, health facilities
1919Baseline, utilization of maternal health services
2020Baseline, women 15-49 with birth in last 24 months
2121Evaluating P4P in Rwanda:Evaluation results22Impact on structure qualityImpact of PBFTime TrendNTreatment 2008 (=1)SD2008 (=1)SDAvailability of vaccines1550.703**(0.355)-0.514***(0.122)Prenatal care service1550.062(0.426)-0.659***(0.100)Delivery service1550.239(0.286)-0.484***(0.083)2323
24
25Impact on quality of prenatal careImpact of PBFTime TrendNTreatment 2008 (=1)SD2008 (=1)SDTotal Prenatal Quality Score (standardized score)36830.157***(0.048)0.090***(0.032)Tetanus vaccine during prenatal visit (=1)28100.054**(0.023)0.032(0.028)Productivity ratio37570.072***(0.017)0.036***(0.012)Treatment * 2008 * Competency > 75% (=1)37390.088***(0.019)NANA2626Impact on quality of prenatal care
2727Impact on use of prenatal careImpact of PBFTime TrendNTreatment 2008 (=1)SD2008 (=1)SDNumber of prenatal visits received 2223-0.028(0.066)0.237***(0.056)Likelihood of first prenatal visit in first trimester 22230.020(0.036)0.163***(0.029)Made 4 or more prenatal care visits 22230.009(0.039)0.121***(0.023)2828Impact on use of maternal servicesImpact of PBFTime TrendNTreatment 2008 (=1)SD2008 (=1)SDInstitutional delivery (=1)21080.077**(0.034)0.134***(0.023)Delivery attended by Qualified Provider (=1)22740.083**(0.035)0.195***(0.028)Use of any modern contraceptive method (=1)3121-0.017(0.023)0.245***(0.033)2929Impact on institutional delivery
3030What our results tell usYou get what you pay for ! Need to get prices rightReturns to effort importantBigger effects in things more in providers controlPatient or community health workers for prenatal care/ImmunizationProvide incentives directly to pregnant women? (conditional cash transfer program).Financial incentive to community health workers Low quality of care : additional training coupled with P4PEvaluation feedback useful
3131DiscussionPrenatal care : entry point!Increase in utilization nationwide due to:MutuelleImihigoHIV servicesSafe motherhood and PCIMEPossible spill over effect to child health
3232Limitations !The original randomized designed was changed due to the political decentralization process: But sample well balanced!Trend analysis with HMIS data ongoingNo measure of all paid and some non paid indicators : HMIS analysisCost effectiveness analysis
3333AcknowledgmentsFunding by:World BankGovernment of Rwanda (PHRD grant)Bank-Netherlands Partnership Program (BNPP)ESRC/DFIDGDNThank you!35Table 1: Output Indicators (Us) and Unit Payments for PBF FormulaOUTPUT INDICATORSAmount paid per unit (US$)
Visit Indicators: Number of
1curative care visits0.18
2first prenatal care visits0.09
3women who completed 4 prenatal care visits0.37
4first time family planning visits (new contraceptive users)1.83
5contraceptive resupply visits0.18
6deliveries in the facility4.59
7child (0 - 59 months) preventive care visits0.18
Content of care indicators: Number of
8women who received tetanus vaccine during prenatal care0.46
9women who received malaria prophylaxis during prenatal care0.46
10at risk pregnancies referred to hospital for delivery1.83
11emergency transfers to hospital for obstetric care4.59
12children who completed vaccinations (child preventive care)0.92
13malnourished children referred for treatment1.83
14other emergency referrals1.83
Table 2: Services (Ss) and Weights (s) Used to Construct the Q for PBF Formula
ServiceWeightShare of weight allocated to structural componentsShare of weight allocated to process componentsMeans of assessment
1General administration0.0521.000.00Direct observation
2Cleanliness0.0281.000.00Direct observation
3Curative care0.1700.230.77Medical record review
4Delivery0.1300.400.60Medical record review
5Prenatal care0.1260.120.88Direct observation
6Family planning0.1140.220.78Medical record review
7Immunization0.0700.400.60Direct observation
8Growth monitoring0.0520.150.85Direct observation
9HIV services0.0901.000.00Direct observation
10Tuberculosis service0.0280.280.72Direct observation
11Laboratory0.0301.000.00Direct observation
12Pharmacy management0.0601.000.00Direct observation
13Financial management0.0501.000.00Direct observation
Total1.000
