Pay for performance initiatives a growing trend, but are they
effective? Josephine Borghi
London School of Hygiene & Tropical Medicine
Ifakara Health Institute
Objective
• Provide an overview of the challenges in measuring the effectiveness of PBF– PBF refers to a diverse set of interventions– Implementation may not match the ideal ‘vision’– Need a better understanding of the health system
impact of PBF– What kinds of evaluation should we be doing?– What is feasible and politically acceptable?
Performance based financing – A range of applications
• Based on who is the recipient of the bonus payment– Donor to government (aid modalities)– Within the public sector (facilities or districts / regions)– Government/donor to non-state provider– Health workers
• Process or outcome indicator/target• Payment method and size• When we say PBF is effective what do we mean?
Implementation – potential constraints in low income
settings• Design constraints
– Complexity versus simplicity – designing a simple system which is effective?
• Roll out constraints– Logistics
• Timely availability of funds• Appropriate management of information• Training of managers and health workers
• Monitoring constraints– Need for external validation - costly– Lack of reliable HMIS data
Tanzanian experience – vision versus reality
Ideal Vision for P4P Actual policy design
HMIS strengthened alongside P4P
HMIS not strengthened – no baseline
Flexibility in way incentives are structured by district
No flexibility – single system applied everywhere
Financial autonomy at facility level
Funds managed at district level
Variable targets based on baseline performance
Absolute targets irrespective of baseline performance
Phased implementation to allow to learn from experience and fine tune policy
National roll out in order to ensure rapid impact
PBF: Impact on the Health System?
•Potential Risks•Understanding the processes of change as well as the impact on health outcomes
Potential risks
• Impact on health worker behaviour– Supplier induced demand – unnecessary service
delivery – Quantity over quality– Opportunity cost – focus on targeted conditions
• Equity effects– Patients: Focus on easy to reach groups– Geographic: areas where health system stronger may
have advantage– Health worker migration
Need for improved evaluation design
Key Components of an Evaluation
Process Evaluation
What is the status of implementation? How does it vary by area?
What facilitates and impedes implementation?
Impact Evaluation
What changes take place that result in the observed impacts?
Economic Evaluation
What are the ‘positive’ effects of PBF on coverage of health services?
Are there any unintended consequences of PBF?
Is PBF a good use of resources?
Need for qualitative assessm
ent to determ
ine causal pathway
What are the health benefits of PBF?
An evaluation framework based on Tanzania
Positive Effects Negative Effects
Re-allocate resources
Improve quality of care / increased patient satisfaction
Increased utilisation of targeted health services
Reduced maternal and newborn mortality
Increased motivation and trust
Coercive strategies to increase utilisation
Mis-reporting performance
Crowding out of non-targeted health services / reduced quality of care
Damage intrinsic motivators
Recommendations for improved performance
Impact Evaluation Design Options
Probability design
Plausibility design
Adequacy design
Randomised Control
Strongest evidence of impact
Political acceptability
Non-randomised Control
Interrupted time series
Need to control for inherent
differences
Need good historical data
Historical or no control
Weakest evidence of impact
Practical challenges and need for innovative thinking
• Political resistance to controls• Randomisation may not be possible– Randomisation of performance contracts
(Rwanda)
• In absence of control or historical data importance of documenting the process
Summary• Need for better understanding of which aspects
of PBF are effective and pre-conditions for their effectiveness
• What is implemented may not match what was intended, need for simple and saleable models
• Some unanswered questions on health system impact
• Need for process evaluations and more impact evaluations within realm of political feasibility