EAST AFRICA PUBLIC HEALTH LABORATORY NETWORKING PROJECT: EVALUATION OF PERFORMANCE-BASED
FINANCING FOR PUBLIC HEALTH LABORATORIES IN RWANDA
D I S C U S S I O N P A P E R
A P R I L 2 0 1 6
Meghan Kumar Joel Lehmann Aniceth Rucogoza Claver Kayobotsi Ashis Das Miriam Schneidman
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
EAST AFRICA PUBLIC HEALTH LABORATORY NETWORKING PROJECT:
Evaluation of Performance-based Financing for Public Health Laboratories in Rwanda
Meghan Kumar Joel Lehmann
Aniceth Rucogoza Claver Kayobotsi
Ashis Das Miriam Schneidman
April 2016
Health, Nutrition and Population (HNP) Discussion Paper This series is produced by the Health, Nutrition, and Population Global Practice of the World Bank. The papers in this series aim to provide a vehicle for publishing preliminary results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. For information regarding the HNP Discussion Paper Series, please contact the Editor, Martin Lutalo at [email protected] or Erika Yanick at [email protected].
© 2016 The International Bank for Reconstruction and Development /The World Bank 1818 H Street, NW Washington, DC 20433 All rights reserved.
ii
Health, Nutrition and Population (HNP) Discussion Paper
East Africa Public Health Laboratory Networking Project: Evaluation of Performance-based Financing
for Public Health Laboratories in Rwanda
Meghan Kumara Joel Lehmannb Aniceth Rucogozac Claver Kayobotsid Ashis Dase Miriam Schneidmanf
a Health Policy and Financing Consultant, Health-E-Net, Ltd., Nairobi, Kenya b Health Sector Research Consultant, Infospective, Ltd., Nairobi, Kenya c Director, Microbiology Unit, National Reference Laboratory, Rwanda Biomedical Center, Kigali, Rwanda
d EAPHLN Project Officer, Rwanda Ministry of Health, Kigali, Rwanda
e Health Specialist, The World Bank, Washington, DC, USA
f Lead Health Specialist, The World Bank, Washington, DC, USA
This paper was prepared with funding from the Health Results Innovation Trust Fund
(HRITF) which aims to support results based financing approaches in the health sector. HRITF is managed by the World Bank and is supported by the Governments of Norway through Norad and the United Kingdom through the Department of International
Development. Abstract: This report summarizes the main findings from the application of performance based incentives linked to progress on a standardized, globally recognized metric-- the Stepwise Laboratory Improvement Process towards Accreditation (SLIPTA) checklist-- under the East Africa Public Health Laboratory Networking Project (EAPHLNP) in Rwanda. The full report is available on the EAPHLNP Web Portal (http://www.eaphln-ecsahc.org/newwebportal). The lab Performance-based Financing (PBF) pilot was introduced in the context of a well-established national PBF program dating back to the early 2000s. The flexible nature of the EAPHLNP and the favorable context in Rwanda provided an ideal backdrop to introduce PBF incentive payments to accelerate progress of five project-supported labs towards accreditation. The evaluation found improved laboratory performance at all project-supported laboratories in Rwanda as measured by the SLIPTA scores. For the first time, laboratories were bringing in PBF revenues, instilling a culture of continuous quality improvements, and focusing management attention on accreditation. PBF appears to have contributed to an accelerated change, with PBF laboratories experiencing an overall greater increase in SLIPTA scores compared to project-supported laboratories in the other countries. No clear patterns were found in terms of improved test volumes or test accuracy, which were not part of the pilot scheme. While it was difficult to disentangle the effects of different interventions, the evaluation found a system-strengthening value to combining investments in modernizing
iii
laboratories, and strengthening human resources with PBF. Relationships between laboratory staff and clinicians improved, with laboratory managers having a greater voice in hospital management and lab staff increasingly valued and respected by clinicians. A spirit of teamwork prevailed at participating sites. Other countries considering PBF mechanisms for public health laboratories need to take into account lessons learned and assess the features which may be relevant to their own contexts. PBF schemes for laboratories need to be viewed as an integral part of a package of interventions that contribute to enhanced performance. Keywords: Public health laboratories, performance-based financing, laboratory improvement process towards accreditation, enhanced accountability and performance. Disclaimer: The findings, interpretations and conclusions expressed in the paper are entirely those of the authors, and do not represent the views of the World Bank, its Executive Directors, or the countries they represent. Correspondence Details: Joel Lehmann, Infospective, Ltd., Nairobi, Kenya, Email: [email protected]
iv
Table of Contents
ABBREVIATIONS AND ACRONYMS ................................................................................. VII
EXECUTIVE SUMMARY ...................................................................................................... VIII
I. INTRODUCTION ................................................................................................................... 11
BACKGROUND ............................................................................................................................ 11 THE EAST AFRICA PUBLIC HEALTH LABORATORY NETWORKING PROJECT ............................. 12 THE RWANDAN CONTEXT .......................................................................................................... 16 RWANDAN EAPHLNP-SUPPORTED LABORATORIES ................................................................. 18
II. METHODOLOGY ................................................................................................................. 20
RESEARCH QUESTIONS .............................................................................................................. 20 CONCEPTUAL FRAMEWORK ....................................................................................................... 20 QUALITATIVE RESEARCH .......................................................................................................... 21 QUANTITATIVE RESEARCH ........................................................................................................ 21 LIMITATIONS .............................................................................................................................. 24
III. PBF AS AN APPROACH TO IMPROVE LABORATORY PERFORMANCE ........... 25
IMPLEMENTATION OF PBF FOR PUBLIC HEALTH LABORATORIES IN RWANDA .......................... 25 PBF IMPLEMENTATION .............................................................................................................. 29 IMPACT OF PBF ON LABORATORY PERFORMANCE AND HOSPITAL PERFORMANCE ................... 32 MOTIVATION, RELATIONSHIPS, AND THE IMPROVEMENT PROCESS ........................................... 36
IV. FINDINGS RELATED TO THE EAPHLN PROJECT ................................................... 42
IMPLEMENTATION OF EAPHLNP INTERVENTIONS ................................................................... 42 EFFECT OF EAPHLNP INTERVENTIONS ON SLIPTA PERFORMANCE ....................................... 44 EFFECT OF EAPHLNP INTERVENTION ON RANGE OF TESTS AVAILABLE .................................. 46 EFFECT OF EAPHLNP INTERVENTIONS ON TEST VOLUMES ...................................................... 47 EFFECT OF EAPHLNP INTERVENTIONS ON TEST ACCURACY ................................................... 49 QUALITATIVE ASSESSMENT OF THE IMPORTANCE OF DIFFERENT INTERVENTION ELEMENTS ... 49
V. DISCUSSION OF FINDINGS AND LESSONS LEARNED FOR THE INTRODUCTION OF LABORATORY PBF IN OTHER COUNTRIES............................. 52
SUMMARY OF FINDINGS ............................................................................................................ 52 VALIDITY OF THE CONCEPTUAL FRAMEWORK........................................................................... 53 V. LESSONS LEARNED FOR OTHER IMPLEMENTERS OF LABORATORY PBF ............................... 54
REFERENCES ............................................................................................................................ 57
APPENDICES ............................................................................................................................. 58
APPENDIX 1: TERMS OF REFERENCE FOR THE LEAD CONSULTANT (ABRIDGED) ...................... 58 APPENDIX 2: FINAL QUALITATIVE SAMPLE .............................................................................. 61 APPENDIX 3: SAMPLE QUALITATIVE DISCUSSION GUIDE (ENGLISH VERSION) ........................ 62 APPENDIX 4: QUANTITATIVE ANALYSIS TABLES ...................................................................... 65
v
ACKNOWLEDGMENTS
The authors would like to thank Kelly Silva for technical backing and critical independent views, Martin Matu for his facilitation with the formal authorization, review of the findings and for sending invaluable datasets, and all the interview participants who are not named for purpose of confidentiality. We also are very grateful for the support extended by Dr. Daniel Ngamije, Manager of the Single Projects Coordination Unit (SPIU) and EAPHLNP Project Coordinator in Rwanda; Emil Ivan, Director of the National Reference Laboratory; Emmanuel Nkuranga, Chief Accountant (SPIU); Gilbert Biraro, Disease Programme Manager (SPIU); Dr. Edouard Ntagwabira, Director of the Laboratory Network Unit (Rwanda Biomedical Center, NRL Division); Dr. Habimana-Mucyo, Director of MDR-TB Unit (Rwanda Biomedical Center, TB Division); Dr. Michel Gasana, TB Program Manager; and all other managerial and technical staff in the Ministry of Health and at the National Reference Laboratory. We also thank the hospital directors and laboratory managers and staff at the EAPHLNP satellite laboratories, as well as other colleagues in Rwanda who helped to make this evaluation possible. The authors wish to acknowledge the helpful contributions from peer reviewers, Gyorgi Fritsche, Senior Health Specialist, World Bank, G.N.V. Ramana, Program Leader, World Bank, and Alaine Nyaruhirira, Senior Technical Adviser for Laboratory Services, Management Sciences for Health. Finally, our thanks go to Dr. Wei Han for her support with the statistical analysis. Without all these contributions, this report would not have been possible. The authors are grateful to the Ministry of Health in Rwanda for its leadership on this initiative and to the World Bank for republishing this report as an HNP Discussion Paper.
vi
ABBREVIATIONS AND ACRONYMS
ASLM African Society for Laboratory Medicine FBC Full Blood Count (laboratory test)
CD4 Test counting the T-cells in blood sample to assess the immune system (laboratory test)
EAPHLNP East Africa Public Health Laboratory Networking Project
ECSA-HC East Central and Southern Africa Health Community EQA External Quality Assessment HFU Health Financing Unit MoH Ministry of Health NRL National Reference Laboratory OECD Organization for Economic Co-operation and Development PBF Performance-based Financing PT Proficiency Testing (laboratory test performance) SLIPTA Stepwise Laboratory Quality Improvement Process Towards Accreditation SLMTA Stepwise Laboratory Management Towards Accreditation SOP Standard Operating Procedure SPIU Special Projects Implementation Unit in the Rwandan MoH TAT Turn-around Time TB Tuberculosis WB World Bank WHO World Health Organization
vii
EXECUTIVE SUMMARY The East Africa Public Health Laboratory Networking Project (EAPHLNP) addresses the historical neglect of public health laboratories and explores strategies for improving the effectiveness of laboratories and the performance of laboratory workers. Laboratory services are fundamental to appropriate diagnosis and treatment of diseases, as well as early detection and management of disease outbreaks. Laboratories have historically been one of the weakest areas in health systems, and laboratory workers are among the neglected cadres of healthcare workers. The Bank-funded EAPHLNP, launched in 2010, aims to establish a network of efficient, high quality, accessible public health laboratories (Burundi, Kenya, Rwanda, Tanzania and Uganda). The project upgraded, supported and networked up to six district hospital laboratories in each country, and one or two referral laboratories that act as national hubs. In addition, the project supports innovations in service delivery and uses a learn-by-doing approach. Each country serves as a center of excellence in a thematic area: Kenya (operational research, disease surveillance); Uganda (lab networking and accreditation); Tanzania (training and capacity building); Rwanda (PBF, ICT MDR-TB); and Burundi (PBF), piloting innovations, and sharing lessons and good practices with the other countries. The East, Central and Southern Africa Health Community (ECSA-HC) facilitates knowledge sharing across countries and shares policy implications with health authorities in member states. Rwanda led the piloting of Performance-Based Financing (PBF) for laboratories. This report reviews and evaluates the application of performance based incentives linked to progress on quality improvements towards laboratory accreditation in Rwanda. The PBF pilot is discussed in the context of a broader assessment of the impact of EAPHLNP-supported interventions with a view to identifying lessons learned which may be of interest to other countries and stakeholders. The lab PBF pilot was introduced in the context of a well-established national PBF program dating back to the early 2000s. The flexible nature of the EAPHLNP and the favorable context in Rwanda provided an ideal backdrop to experimentally introduce a modified version of PBF financing to help accelerate progress of five project-supported labs towards accreditation. While in the past laboratories were evaluated as part of the broader hospital PBF reviews, the main difference introduced was linking incentive payments to progress on a standardized, globally recognized metric---the Stepwise Laboratory Improvement Process towards Accreditation (SLIPTA) checklist. For the first time, laboratories were bringing in PBF revenues, instilling a culture of continuous quality improvements, and helping to focus management attention on accreditation.
In line with the learn-by-doing approach, an appropriate methodology was adopted for a retrospective evaluation, identifying comparator sites with similar characteristics. The performance of the five intervention laboratories in Rwanda was compared both to other district hospital laboratories in the country which operated in a similar regulatory and institutional framework, and to EAPHLNP laboratories which benefitted from a similar package of interventions under the project and non-EAPHLNP sites in the other participating countries. All comparator sites were enrolled in the Stepwise Laboratory Management Training towards Accreditation (SLMTA) program, the gold standard task-based training and mentoring program preparing sites for accreditation based on
viii
international clinical lab standards. A heuristic framework was developed to better understand how the package of interventions (including PBF) might have contributed to performance improvements. Qualitative and quantitative research methods were combined in a mixed-methods design. The qualitative part consisted of in-depth expert interviews with key informants to better understand, contextualize and explain quantitative findings. The quantitative part included analysis of secondary data and collection of structured primary data from intervention and non-intervention sites in Rwanda. While performance was measured and financially incentivized only in terms of the SLIPTA scores, data on service volumes and test accuracy were also collected to ascertain the potential broader impact of project interventions. The main limitations of the study relate to the: limited availability of some data; small sample size for the intervention group; and potential biases from selection of comparator sites and from respondent interviews. Likewise, it remains unclear to what extent the findings from Rwanda can be generalized to other countries which do not have a similar enabling environment. The evaluation found improved laboratory performance at all five project-supported laboratories in Rwanda as measured by the SLIPTA scores. No clear patterns were found in terms of improved test volumes or test accuracy, which were not part of the pilot scheme. PBF appears to have contributed to an accelerated change where it was implemented, with PBF laboratories experiencing an overall greater increase in SLIPTA scores compared to project-supported laboratories in the other countries. In terms of the overall project impact, since the different EAPHLNP interventions were delivered almost homogeneously in Rwanda, it was not possible to isolate the impact of the individual interventions. Both EAPHLNP intervention groups (with and without PBF) out-performed the comparator laboratories. These generally positive trends need to be viewed cautiously given the potential biases noted above. Despite the limitations of the study, the evaluation found a system-strengthening value to introducing a package of interventions, including modernizing laboratories and strengthening human resources, in combination with PBF. While it is difficult to disentangle the effects of these interventions, as shown in other studies, PBF works best when a criticality of initial investments are in place. The evaluation found that motivation of laboratory personnel was positively affected both through the improved work environment, and training and mentoring opportunities as well as through the PBF scheme. Relationships between laboratory staff and clinicians improved over the project period, reflecting a combination of factors, including recognition of enhanced laboratory performance and increased capacity of laboratories to bring in PBF money. Laboratory managers had a greater voice in hospital management and lab staff felt increasingly valued and respected by clinicians. The quarterly PBF assessments were viewed to be time consuming but beneficial in enhancing accountability, maintaining the focus on the quality improvement process towards accreditation, and possibly speeding up the measurement-improvement cycle. A spirit of teamwork prevailed at participating sites, contributing to successful implementation. Other countries considering the application of PBF mechanisms for public health laboratories need to take into account lessons learned and assess the features which may be relevant to their own contexts. There are four broad lessons worth noting:
ix
PBF schemes for laboratories have the potential to instill a results-focused culture, foster accountability, and promote teamwork. As in other PBF schemes, providing incentives to facilities makes laboratory staff and management accountable for the results being incentivized. Likewise, accountability mechanisms are reinforced. The relationship between clinicians and laboratory personnel may improve as laboratories become more performant and clinicians have greater confidence in the results. As laboratories bring in PBF money, they get a seat at the table, and a greater voice in hospital management. PBF schemes also require and engender team spirit which is critical to quality laboratory work.
PBF schemes for laboratories need to be viewed as an integral part of a package of interventions that contribute to enhanced performance. PBF schemes do not operate in isolation and are not a magical bullet. They appear to work best in an environment where minimal investments in modern infrastructure and human resources are in place. PBF may trigger more rapid progress in the quality improvement process towards accreditation in combination with other strategies.
The design of PBF laboratory schemes needs to be tailored to different contexts. In countries with existing PBF programs, it will be easier and less costly to introduce a lab PBF scheme. In settings with no PBF national programs, careful consideration needs to be given to the types of incentives (facility vs individual) to be provided to avoid creating distortions in the system. PBF schemes that remunerate facilities or individual departments are more likely to foster teamwork and minimize risk of perverse effects. Individual incentives while potentially attractive to front line workers may not be easily maintained. In all settings, ensuring sustainability remains a challenge. It is critical for governments to increasingly assume responsibility for mainstreaming the approach. Likewise, linking incentive payments to progress towards accreditation offers new options for fostering sustainability, as accredited laboratories will be more performant and have the capacity to generate revenues.
The judicious selection of indicators for PBF laboratory schemes is critical to success along with careful selection of appropriate measures for measuring performance. Use of SLIPTA composite scores linked to progress towards accreditation was highly innovative, helping to set the facilities on a path towards accreditation. While important, SLIPTA scores were not sufficient to capture overall performance. Future schemes need to consider complementing these scores with other key measures of performance (for example, test volumes, test accuracy, patient satisfaction). Stakeholders considering similar approaches need to increasingly rely on laboratory information management systems for tracking performance, with a view to institutionalizing performance monitoring. Another factor which needs to be considered is the frequency of assessments, to strike the right balance between maintaining focus on results and the additional burden of conducting more frequent audits.
x
I. INTRODUCTION This report presents findings from an application of performance based financing for public health laboratories under the East Africa Public Health Laboratory Networking Project (EAPHLNP), which supports innovations to strengthening laboratory systems. Under this World Bank funded regional project each participating country (Burundi, Kenya, Rwanda, Tanzania and Uganda) serves as a center of excellence in a thematic area, generating knowledge and sharing experiences and lessons with the other countries to help inform public policy and programmatic action. This paper describes Rwanda’s experience with providing performance based incentives to facilities linked to progress on quality improvements towards laboratory accreditation within the context of a broader assessment of the impact of the EAPHLNP-supported laboratory interventions.
The paper is divided into five sections. This section sets the context for the discussion by providing background information on performance based financing, the regional project, and the Rwandan context. The second section describes the methodology used for assessing performance, including the main research questions addressed. The third section describes the application of PBF for public health laboratories in Rwanda and discusses its impact. The next one provides an assessment of the overall interventions supported under the project. The final section summarizes the main findings and identifies lessons learned which may be of interest to other countries and stakeholders.
BACKGROUND
Performance-based financing (PBF) was first introduced in the health sector through a primary health care initiative in Zambia in 1990. Most early work on PBF in health was output-based contracting to expand access to health services (Fritsche et al., 2014). In the past 15 years, PBF initiatives and pilots have been launched in more than 30 countries, primarily in Asia and Africa. They are growing in number and diversity (see Figure 1), applying new methods to different subsections of the health sector.
Figure 1: Rapid expansion of PBF in Africa (Fritsche et al., 2014)
11
PBF is defined as a subset of results-based financing (RBF) that targets health facilities with a conditional fee-for-service mechanism. RBF, more broadly, can be payments for countries, healthcare providers or beneficiaries and can involve fee-for-service, other financing, or even non-monetary rewards (contrary to what the name might suggest). Results-Based Financing (RBF) has been defined as "a cash payment or non-monetary transfer made to a national or sub-national government, manager, provider, payer or consumer of health services after predefined results have been attained and verified. Payment is conditional on measurable actions being undertaken." (www.rbfhealth.org). The following definition of PBF was agreed by the African PBF Community of Practice and posted on its website in August 2010:
Performance-Based Financing is a health systems approach with an orientation on results defined as quantity and quality of service outputs. This approach entails making health facilities autonomous agencies that work for the benefit of health related goals and their staff. Multiple performance frameworks for the regulatory functions, the performance-purchasing agency and community empowerment also characterize it. Performance-Based Financing applies market forces but seeks to correct market failures to attain health gains. PBF at the same time aims at cost-containment and a sustainable mix of revenues from cost-recovery, government and international contributions. PBF is a flexible approach that continuously seeks to improve through empirical research and rigorous impact evaluations, which lead to best practices.
THE EAST AFRICA PUBLIC HEALTH LABORATORY NETWORKING PROJECT In 2010, four East African Community (EAC) member states received US$78.7 million from the World Bank for the EAPHLNP which aimed to establish a network of efficient, high quality, accessible public health laboratories1. The project supports 32 laboratories in the participating countries, the majority located in cross-border areas, to become Centres of Excellence (CoE) and to increase access to laboratory services for poor and vulnerable populations. The strengthened laboratories are expected to provide specialized services to communities in these regions, services that were otherwise available only at the national reference facilities, resulting in faster turnaround times and potential reductions in morbidity and mortality due to improved diagnosis and treatment. In all participating countries, the project upgraded, supported and networked up to six district hospital laboratories, in addition to one or two national referral laboratories that act as national hubs. Role of laboratories in health The EAPHLNP addresses the historical neglect of public health laboratories and explores ways to improve laboratory performance. Laboratory services are fundamental to appropriate diagnosis and treatment of diseases, as well as early detection and management of outbreaks. Laboratories have historically been one of the weakest areas in health systems, and laboratory workers are among the neglected cadres of
1 The project started with Kenya, Tanzania, Uganda and Rwanda. In 2012, Burundi joined the EAPHLNP. Public health laboratories in Rwanda at the district and provincial level do both public health work and provide clinical laboratory services.
12
healthcare workers (Schneidman et al., 2014). Donor funding for health through infectious disease and primary care lenses has resulted in continued neglect of public health laboratories. The 2008 Maputo Declaration on Strengthening of Laboratory Systems has begun to address these issues, but major gaps remain.
Laboratory workers are critical to the performance of laboratories. Training sufficient, well-qualified laboratory staff, distributing them appropriately based on population and need, and ensuring they have career development possibilities that keep them engaged in the system are all challenges facing most countries in the region.
The combination of insufficient numbers, limited training, and skewed distribution of staff working in poorly equipped facilities results in a vicious cycle of demotivation and underperformance. Projects like the EAPHLNP have the potential to raise the profile of laboratory workers and reverse this cycle into what has been termed a virtuous cycle (Figure 2: Schneidman et al., 2014).
Figure 2: Virtuous Cycle of Lab Worker Motivation (Schneidman et al., 2014)
Testing Innovations for Public Health Laboratories
A key strategy of the EAPHLNP was to support and assess innovations to improve the effectiveness of public health laboratories and the performance of laboratory workers. Each participating country agreed to provide regional leadership in a thematic area, generating knowledge and sharing experiences and lessons. The East, Central and Southern Africa Health Community (ECSA-HC) facilitates inter-country learning and response mechanisms, supports south-south collaboration among the member states. Kenya is leading on integrated disease surveillance and response and on operational research; Uganda provides regional support on laboratory networking and accreditation; Tanzania coordinates regional training and capacity building; and Rwanda leads on
13
information and communication technologies. In addition, Rwanda also took the lead on PBF and co-chairs a technical working group together with Burundi to explore the applicability of PBF to public health laboratories, building on its prior successful experience with the national PBF program. In this context, the Ministry of Health of Rwanda played a lead role in promoting and sharing the PBF approach. The Rwandese team decided to use an empirical (action-oriented) research approach to explore the feasibility of applying PBF to public health laboratories.
The flexible nature of the EAPHLNP and the favorable context in Rwanda provided an ideal backdrop to experimentally introduce a modified version of PBF financing to help accelerate the progress of five project-supported laboratories towards readiness for ISO-accreditation. Rwanda already had considerable experience of implementing PBF in the health sector, including quality assessments of hospital departments. It was therefore possible to refine the PBF approach to laboratories by linking incentive payments to progress on the Stepwise Laboratory Improvement Process towards Accreditation (SLIPTA) checklist, at a lower cost than would have been the case elsewhere. Understanding the effectiveness of a PBF application to public health laboratories is of interest to decision-makers in other EAPHLNP-supported countries, as well as to other development partners who are considering various interventions to improve laboratory performance. One of the key innovations supported under the EAPHNLP in all countries has been a regional assessment/review mechanism to guide planning for quality improvement projects (QIPs) and mentorship process to accelerate the laboratory improvement. Figure 3 below details the approach the regional laboratory accreditation and networking TWG has developed to support the countries in the region to develop robust quality management systems in preparation for laboratory accreditation.
Figure 3: EAPHLNP Approach to Quality Improvement Towards Laboratory Accreditation
All five participating countries adopted the WHO-AFRO supported SLIPTA process as an approach to strengthening laboratory quality management systems of its member states. They also adopted the Stepwise Laboratory Management Towards Accreditation (SLMTA) program, the companion ‘gold standard’ task-based training and mentoring program, which involves classroom and laboratory on-site mentoring to build skills of laboratory managers and improve quality with the goal to prepare laboratories for accreditation based on international clinical laboratory standards. The programme is supported by the African Society for Laboratory Medicine (ASLM), US Center for
Peer/SLIPTA assessment
Internal audit/ PBF assessment
SLMTA Trainings
Planning for QIPs
Mentorship and SLMTA
14
Disease Control and Prevention (CDC), and WHO-AFRO, and has been conducted in 36 countries worldwide, including 18 African countries. SLIPTA is a laboratory quality improvement process towards accreditation, which uses a standardized quality checklist with a scoring system in a range of 0 – 100, assessing 12 Quality System Essentials (QSE) of the Laboratory Quality Management System Guidelines. The SLIPTA composite score is derived from 117 audit questions, mostly derived from the 12 QSE of the laboratory management requirements to achieve certification based on the ISO-standard 15189:2012. The SLIPTA scoring system forms an integral part of the SLMTA approach, but it can also be used independently as a management quality improvement and monitoring system. Certificates of recognition are given to laboratories with 1 to 5 stars indicating level of accreditation readiness. Facilities that score 55-64 percent receive one star, 65-74 percent two stars, 75-84 percent receive three stars, 85-94 percent four stars and those with a score of 95 percent or more on the SLIPTA scale obtain 5 stars, indicating readiness to enroll for an ISO 15189/17025 certification scheme (see Figure 4). SLIPTA was launched globally in Kigali in 2009. Further details about the content of the SLIPTA checklist, are provided under section on performance indicators below.
Figure 4: SLIPTA Tiers of Laboratory Quality Management
Source: World Health Organization - AFRO
15
For each SLIPTA assessment, auditors review documents and records, observe the laboratory set-up and specific work processes, and interview laboratory staff according to a prescribed protocol. Twelve areas are assessed in each laboratory, each of which can obtain a certain maximum number of points, adding up to a total of 275 maximum achievable points.
1. Documents and records (maximum score: 28) 2. Management Reviews (14) 3. Organization and Personnel (22) 4. Client Management and Customer Service (10) 5. Equipment (35) 6. Evaluation and Audits (15) 7. Purchasing and Inventory (24) 8. Process Control (32) 9. Information Management (21) 10. Identification of Non Conformities, Corrective and Preventive Actions (19) 11. Occurrence/Incident Management and Process Improvement (12) 12. Facilities and Biosafety (43)
The total score achieved is then normalized, so that 275 constitutes 100 percent2. THE RWANDAN CONTEXT
The context in which the EAPHLNP team, with support from the Single Project Implementation Unit (SPIU) of the Rwandan Ministry of Health (MoH), introduced the PBF pilot, with implementation support from the Health Financing Unit (HFU), played an important role on how the pilot was structured, as well as the way it was able to influence laboratory performance, as discussed below. The Existing PBF Programme in the Rwandan Health System The PBF pilot was introduced in a strong enabling environment which benefits from high level government leadership a strong focus on results and good governance. Health has a prominent position in the Rwandan government’s budget, with total health expenditures representing 10.7 percent of GDP (2012), the highest in the region (World Bank, 2012). The Rwandan health system is known for the mandatory participation of citizens in health insurance schemes, of which the largest is a community-based scheme called ‘mutuelles de santé’. Pilot programs experimenting with supply- and demand side financing initiatives have led to innovative national reforms such as the adoption of a PBF mechanism for health financing at the primary and secondary care level. Another approach that is unique to Rwanda is that donor funds for health - an estimated 47 percent of total health expenditures - are managed centrally by the SPIU in
2 The category names/scores indicated are from the 2015 version of the SLIPTA audit checklist. The full SLIPTA instrument, containing a laboratory profile questionnaire, the audit protocol, and the summary sheet, can be downloaded from the WHO website (http://www.afro.who.int/en/clusters-a-programmes/hss/blood-safety-laboratories-a-health-technology/blt-highlights/3859-who-guide-for-the-stepwise-laboratory-improvement-process-towards-accreditation-in-the-african-region-with-checklist.html)
16
the MoH in a coordinated fashion. This results in a sector-wide approach style of financing, which gives MoH better control and flexibility in the use of donor funds. Rwanda is known globally as one of the pioneers of PBF for healthcare delivery. Encouraging results from three pilot programmes in the early 2000s, which focused on the stimulation of increased uptake of health services, the ‘approche contractuelle’ was adopted as a national policy in 2005 (Rusa et al, 2009). The nationwide scale-up entailed standardization, institutionalization and systematic planning through the development of a standardized set of services, a clear fee structure, and contract templates for different administrative levels. A composite quality score was introduced, which modified performance-based payments to facilities by calculating the amounts disbursed as follows:
Health center PBF earnings = (fees * service quantity) * (% quality score).
According to the findings of a World Bank impact evaluation (Basinga et al., 2011), the PBF approach engendered a results-oriented culture that promoted managerial autonomy and empowered providers to find creative solutions. It also created an environment in which the government gained the confidence to progressively promote the autonomy of facilities and to decentralize the recruitment and dismissal of health professionals to health centers and hospitals. Hospital management allocated PBF funds in part towards paying a bonus to staff, and in part for facility improvements in line with annual budgets. The way this was handled depended on national guidelines and staff levels as well as different hospital management strategies. The revenue that was generated through PBF benefitted hospital laboratories to a variable extent, based on measured performance and on other priorities of the hospital management. The performance of the laboratories was not assessed systematically using a rigorous quality checklist. Laboratories were perceived primarily as a cost rather than a revenue center and given their generally poor and neglected condition, they rarely brought in their own PBF funding. The SLMTA Process in Rwanda The first Rwandan SLMTA cohort of five participating laboratories, including the NRL and four laboratories housed at referral facilities in Kigali and Butare, participated in the programme from 2010 – 2011. The five EAPHLNP-supported satellite laboratories participated in the training programme in 2011-2012. By the time of this assessment in February 2015, a total of 23 laboratories had completed the SLMTA programme in Rwanda, and a total of 101 laboratories had completed the programme, including the final SLIPTA audit, in all the EAPHLNP participating countries. The fact that the EAPHLNP-supported laboratories and a large number of comparator sites participated in the same management training and mentoring programme, with associated standardized measurement of baseline- and exit indicators provided a unique opportunity for this study to compare pre- and post- SLIPTA scores using a quasi-experimental study design as discussed below.
17
RWANDAN EAPHLNP-SUPPORTED LABORATORIES The EAPHLNP supported sites in Rwanda included five district hospital laboratories (Byumba, Kibungo, Gihundwe, Gisenyi, Nyagatare), referred to as ‘satellite labs’ and the national reference laboratory (NRL) in Kigali; see Figure 5 for location of these labs. All the supported hospital laboratories are housed in district hospitals, located 4 – 20 km from the national border, and between 60 and 230 km from the capital Kigali (Figure 5). The hospital catchment population ranged from 150,000 to over 600,000 people. The average number of the catchment population (367,000) was comparable to the average in the other district hospitals that participated in the SLMTA laboratory programme, the comparator sites (378,000).
Figure 5: Location of Rwandan EAPHLN Satellite Labs
Source: Authors
As in other EAPHLNP-supported countries all satellite laboratories in Rwanda received the following basic package of interventions aimed at strengthening both the capacity and efficiency of these facilities:
• Upgrading of laboratories (that is, construction or renovation), and provision of modern laboratory equipment, and ICT hardware.
• Employment or deployment of additional, well-qualified human resources. • Training and capacity building, including enrollment in the Stepwise Laboratory
Management Towards Accreditation (SLMTA) program.
18
• Supportive supervision, and mentorship3, including mentors as expert advisors, coaches and role-models to foster positive change in processes and behavior.
• Facility improvement funds to address gaps identified through SLIPTA audits. • Support for the continuous Stepwise Laboratory Improvement Towards
Accreditation (SLIPTA) quality assurance system.
In Rwanda the EAPHLNP supported laboratories were comparable to other donor-funded laboratories. With some exceptions, the basic equipment installed in comparator laboratories was similar with EAPHLNP-supported laboratories in early 2015 (Table 1). For the more sophisticated equipment such as GeneXpert, there were variations. Almost all laboratories receive donor funding from various sources, primarily the Global Fund, the World Bank, CDC, and Partners in Health. Only two of the comparison facilities (Munini and Ruhango) stated that they received no external funding.
Table 1: Comparison of EAPHLNP Laboratories with Others4
EAPHLNP Labs Other Labs
Facility parameters Catchment population 367,000 378,000 Driving distance from Kigali (mean) 153 km 92 km
Number of beds 246 252 Number of lab rooms 11.8* 7.2 Lab Staff level “A0” (high qualification) 4.4* 1.9
Staff level “A1” (medium qualification) 4.0 5.2
Staff level “A2” (lower qualification) 3.2 4.3
Lab staff, all levels combined 11.5 11.6 Equipment installed (examples) Biochemistry Analyzer, no. of machines 2.4 2.2
Hematology Analyzer, no. of machines 2.0 2.0
ELISA Machine installed 3 / 5 (60%) 3 / 13 (23%) GeneXpert available 5 / 5* (100%) 5 / 13 (38%)
3 The mentorship programme in Rwanda consisted of three experienced and highly qualified laboratory professionals who each spent about three weeks per month on-site in the 5 EAPHLN satellite labs for a duration of almost 2 years from 2012 to 2014. The contact time for each laboratory was an average of 10 days per month.
4 The mean of indicators marked with * are larger for EAPHLNP-supported laboratories with statistical significance at the 95% confidence level.
19
II. METHODOLOGY RESEARCH QUESTIONS
The PBF pilot in Rwanda was reviewed in the context of a broader assessment of the impact of EAPHLNP-supported interventions. The performance of the five EAPHLNP-supported laboratories in Rwanda was compared retrospectively, both to other district hospital laboratories in the country and to EAPHLNP laboratories in the other participating countries. The study aimed to assess the implementation of and additional impact of the PBF incentive payments on top of the EAPHLNP package of interventions (for Terms of Reference, see Appendix 1). Four research questions were identified to help achieve the overall objective:
1. How was the PBF pilot implemented in Rwanda, and what has been the effects
of PBF incentives on the performance of EAPHLNP-supported laboratories and district hospitals in Rwanda?
2. What was the role of lab worker skills and motivation and if and how the relationship between laboratory professionals and clinicians has changed to contribute to the improvements in laboratory performance?
3. What was the impact of EAPHLNP interventions overall on improving laboratory performance? What was the role of the individual interventions, including PBF, in achieving improvements?
4. What are the key lessons learned for other countries interested in implementing a similar program?
CONCEPTUAL FRAMEWORK To facilitate the research process, a simple heuristic framework (Figure 6) was developed. It visualizes the EAPHLNP interventions and likely ways that they might have brought change in the laboratories; in the Discussion section we will briefly examine the validity of this framework in light of the findings.
Figure 6: EAPHLNP Intervention Conceptual Framework
According to this framework, outcome performance indicators beyond the SLIPTA quality/process indicators (as described in the Introduction) include increase in test volumes, improved test timeliness (measured as turn-around time), better test accuracy
20
(measured through external quality assessments (EQA) or proficiency testing (PT)), and patient experience (measured through satisfaction surveys among clients, both health professionals and patients). However, only SLIPTA scores were measured systematically over the project period as performance indicators to determine PBF payments in the Rwandan EAPHLNP model. To answer the research questions, qualitative and quantitative research methods were combined with a mixed-methods design. The qualitative part consisted of in-depth expert interviews with those involved with or affected by the EAPHLNP, and the quantitative part included the compilation and secondary analysis of existing financial, service provision and monitoring data as well as the collection of structured primary data from intervention and non-intervention sites.
QUALITATIVE RESEARCH Generally, qualitative expert interviews involve in-depth discussions with knowledgeable respondents to gather information based on their expertise and experience related to matters under investigation, as well as their assessment and opinions on a process or situation. Responses can help to better understand, contextualize and explain quantitative findings. Conversational-style qualitative interviews are organized with a discussion guide. Discussion guides specific to different respondent types were developed by the research team and revised during an inception meeting in Kigali in early February 20155. In Appendix 2, the completed qualitative sample is described. Qualitative research was conducted in February 2015, and reflects the views and opinions of those interviewed at that time. The research team conducted multiple face-to-face qualitative interviews in two of the five EAPHLNP-supported laboratories and short telephonic discussions with laboratory managers of two additional project sites. The face-to-face interviews included laboratory staff as well as other hospital facility staff and management. Expert interviews were also conducted in Kigali at the National Reference Laboratory, the Single Project Implementation Unit (SPIU), the MoH, and with EAPHLNP mentors and Rwanda project coordinator. Confidentiality was guaranteed to all participants, and they will therefore not be quoted by name or otherwise identified in this report. The interviews were conducted in English or French in a private, quiet atmosphere. With a few exceptions, the interviews were audio-recorded and transcribed for the analysis. Transcripts were analyzed to answer the research questions.
QUANTITATIVE RESEARCH Quantitative research can provide an understanding about the relationship between interventions and the expected outcomes. Statistical analysis can quantify the likelihood that observed differences between groups that received an intervention and those that didn’t are ‘real’ and not due to naturally occurring variations. For the quantitative
5 A sample discussion guide is included in Appendix. The full set of tools can be obtained from evaluators or the Rwandan SPIU upon request.
21
research design, the elements in the conceptual framework were translated into a list of variables: independent variables that might cause a change, confounding variables that might also affect the process, and dependent (outcome) variables that were expected to change as a result of the intervention (see Appendix 4 for further detail). All the analyses were based on a master dataset, aggregating information from several sources6. The main outcome variables that were accessible and used to evaluate laboratory performance are: SLIPTA assessment scores, test volumes (measured by monthly numbers of four routine tests: full blood count, blood glucose test, tuberculosis microscopy, and CD4 count), and test accuracy (measured through PT). Baseline data were taken either from 2011 or from the first quarter of 2012, depending on data availability. Where baseline data were not available (for the test accuracy) to allow assessment of change in performance over the lifetime of the project in the two groups, comparison of end-line performance was done between the EAPHLNP laboratories and the SLMTA-only laboratories in Rwanda.
Figure 7: Number of Laboratories Included in the Analysis by Intervention and Country
Rwanda 17Rwanda
Tanzania 26
Tanzania 5
Uganda 16
Uganda 5Burundi 6
Kenya 18
Kenya 2
SLMTA only EAPHLNwithout PBF
EAPHLNwith PBF
77
186
This evaluation involves the following laboratories: SLMTA-trained laboratories not receiving EAPHLNP support in all five East African countries as comparison (data from CDC/ECSA-HC), EAPHLNP laboratories outside Rwanda (four countries: Burundi, Kenya, Tanzania, Uganda) as intervention group one, and EAPHLNP supported laboratories in Rwanda, all of which received PBF payments, as intervention group two (Figure 7). The Rwandan NRL was excluded from some of the analyses; it was an EAPHLNP supported site but received no PBF payments, and it participated in the SLMTA intervention before the start of the EAPHLNP. Out of the seven interventions listed in the conceptual framework (Figure 1) all the comparator sites in Rwanda received the SLMTA capacity building intervention and functioned in comparable
6 The raw master data file and STATA do-file used to create raw master data file and conduct the analyses are available upon request from evaluators or from the Rwandan SPIU.
22
regulatory and institutional environments and thus provided a similar context, which is a requirement for a quasi-experimental design. It is important to note that some of these comparator facilities also received support from other donors, notably equipment and facility improvements from the Global Fund, but it was not feasible to quantify these in this study.
Table 2: Laboratories Included in the Statistical Analyses
Country SLMTA only EAPHLN Total Rwanda 17/17 6/6 23/23 Tanzania 26/26 5/6 31/32 Uganda 16/16 5/6 21/22 Burundi 0/0 6/6 6/6 Kenya 18/50 2/7 20/57 Total 77/109 24/31 101/140
presents the number of laboratories that have pre- and post-SLMTA intervention scores available as assessed by the SLMTA teams, out of those enrolled in the SLMTA program, by country and by intervention status (i.e. whether labs participated in EAPHLNP intervention or not). No data were available for non-EAPHLNP supported laboratories in Burundi, and it was excluded from some of the analyses; Rwanda, Uganda, and Tanzania had no missing data, whereas approximately two-thirds of the post-SLMTA scores were missing for Kenya, primarily from non-EAPHLNP sites. Standardization of data was done as follows:
• SLIPTA assessment score: For the comparison of pre- and post-SLMTA scores, the CDC assessment data was used. For some of the laboratories, the data file did not contain endline data, but it was available in the records of the Rwandan NRL and appeared consistent. As part of the EAPHLNP, all participating laboratories were also evaluated on the SLIPTA checklist through annual peer assessments over the past four years. The resulting data were analysed separately from pre- and post-SLMTA scores, because the data showed that there was some difference in point levels achieved on assessment by the different organizations and therefore constituted a somewhat different metric7.
• Test volumes: Raw data available on a monthly basis from national HMIS was aggregated to quarters, half-year and annual volumes.
• Disbursement and PBF funds: Raw data was available on a quarterly basis on overall project disbursements to satellite sites and PBF Payments. We averaged quarterly data on an annual basis to address the fact that some quarters had missing data and some disbursements may have delayed from one quarter to another.
7 While there was no formal counter verification in the PBF pilot, there were both national and regional (peer) assessments conducted during the past few years with the results being generally comparable, even though some variation in scores was noted.
23
• Test accuracy: Standardized and comparable raw data was available from One World Accuracy8 in April, July, and November 2014 for chemistry/Immunoassay (BCHE) and hematology; April and July 2014 for HIV lymphocyte immunophenotyping (QASI) and HIV antibiodies (HIVA) in a report named “EQA Performance Overview_2014_RWA”. We use the average achieved percentage acceptable for each of these tests in two groups: SLMTA-only labs and EAPHLN laboratories in Rwanda.
• Confounding variables were extracted from district hospital responses to a semi-structured questionnaire (see Appendix 5), and also included government-defined district equity score used by MoH.
LIMITATIONS The major limitation to the measurement of effect of PBF and the wider EAPHLNP effect were around quantitative data availability, particularly prior to inception of the project. This is not surprising as one of the major improvements driven by SLMTA and EAPHLNP more broadly is an increased emphasis on data collection and quality. As the sample size for the intervention groups was very small, the study therefore doesn’t have a high statistical power. A larger sample size would have enabled a better understanding of the generalizability of the observed differences. A second limitation relates to potential biases in selection of comparator sites and respondent interviews.
The third limitation is that there is no generally accepted standard definition for ‘laboratory performance’. After consultations with stakeholders in Rwanda, performance was defined as increase in test volumes, improved EQA results, shorter TAT, increased client satisfaction, and improved SLIPTA scores as shown in the Conceptual Framework (Figure 1). Possibly, the selection of tests for the assessment of volumes does not fully or validly represent the overall test volume. Other tests or a broader spectrum of tests might have led to different results. The inclusion of performance-related variables such as range of tests available or equipment downtime may have shown differences between EAPHLNP and comparison labs. Due to time and resource constraints and other methodological considerations, they were not included.
The major question that limitations might elicit is whether the impact of PBF in Rwanda is generalizable to other countries, particularly given the fact that the PBF programme is institutionalized for facilities and individuals in the Rwandan health system. The timeline from inception to implementation would likely be longer in other countries where the underlying architecture was not established. Similarly, given the Rwandan decision to measure SLIPTA scores as the PBF-determining indicator, lack of data on other performance areas mean that countries selecting a different means of measuring performance may not be able to predict outcomes based on the findings of this study.
8 One World Accuracy is a Canada-based international collaboration of laboratory EQAs, which Rwanda has joined during the course of the EAPHLN implementation.
24
III. PBF AS AN APPROACH TO IMPROVE LABORATORY PERFORMANCE
IMPLEMENTATION OF PBF FOR PUBLIC HEALTH LABORATORIES IN RWANDA
The objective of the pilot was to determine the applicability of PBF to public health laboratories. The Rwandan HFU, with the support of the EAPHLNP, set out to establish whether – and how - it is feasible to introduce PBF incentive payments linked to progress towards laboratory accreditation to hospital laboratories, and if the approach is effective in improving laboratory performance. This improvement in performance could putatively manifest as an increase in service volume, accountability, efficiency, and a contribution of the laboratories to overall quality of healthcare services – although the only measured performance variable was the SLIPTA composite scores. As a key output of the pilot phase, the team seeks to document lessons and experiences and share them with other countries in the region and with stakeholders active in strengthening public laboratories, as well as those in the broader PBF for health community of practice. These lessons are informing the design of similar PBF programs for the other EAHPLNP countries in the region. With these objectives, the PBF pilot was initiated in Rwanda, with the five EAPHLNP satellite facilities as beneficiaries (excluding the NRL). The laboratory PBF pilot in Rwanda was fully embedded in the existing national PBF program and had to be compliant with the related policy.
Key Design Features and Strategies
Performance Indicators PBF mechanisms are often linked to a combination of quantitative performance indicators, measuring volume of services delivered (e.g. number of patients served with a range of different services), and service quality indicators. In the case of laboratories, the payments could therefore have been linked to indicators such as test volumes, turnaround times, test accuracy as measured by proficiency testing (EQA), and client satisfaction to adhere to classical PBF.
Instead, the Rwandan team took the decision to pioneer the use of PBF for laboratories linked to progress on the SLIPTA scores; for the first time in Rwanda, PBF incentive payments were therefore linked explicitly to progress towards laboratory accreditation.9 The mechanism operated without reference to service volume. Instead, performance was operationalized as a single composite indicator achieved on the SLIPTA audits.
The rationale behind the decision to use the SLIPTA audits score as the performance indicator was that this would reduce the transaction costs. It was decided that paying incentives based on the delivery of individual tests would be tedious and time consuming. The implementation of the stepwise laboratory improvement process towards accreditation was an integral part of the EAPHLNP and working with it removed
9 Since Rwanda had a well-established national PBF program, with a mechanism for evaluating quality of hospital services (including laboratories), the EAPHLN laboratory pilot was innovative to the extent that it used a standardized, regionally recognized quality scorecard that measures progress towards ISO accreditation.
25
the requirement of incurring costs to collect data on individual test volumes and quality, while accelerating the speed by which laboratories progressed towards accreditation. The emphasis on the SLIPTA audits was expected to act as additional impetus for staff motivation, improving their relationship with the hospital management which increasingly viewed the laboratories in a positive light. Data Validation Data collection and data validation were not separated as they would be in a traditional PBF design. Instead, the data collection was done in a single step during the quarterly SLIPTA evaluations that were conducted by a multi-disciplinary “joint team” from the National Reference Laboratory, together with SPIU and the HFU. There was no additional or separate validation process. In line with the SLIPTA philosophy, feedback was then provided to facilities on results, strengths and weaknesses, and the other interventions of the EAPHLNP provided the laboratories with mentorship support and facility grants to execute an improvement plan addressing gaps identified in each round of assessment.
Although initial design was quarterly evaluations using the SLIPTA checklist, human and financial constraints meant that this design feature was altered. In Table 3, the data collection activities as recorded by the SPIU are recorded (total of 11 quarters during 2012 - 2014). In 4 out of the 11 quarters, no score was reported, while in 2 of the 11 quarters, the scores were very close to the scores of the EAPHLNP peer assessment, suggesting that the two exercises might have been combined to leverage synergies. The remaining 5 quarters had unique scores as obtained by the joint SPIU/NRL assessment team. For quarters where no score was obtained, the incentive payments were based on previous scores. As Table 3 shows, the independent assessments were less regular in the later stage of the pilot. This was made official in mid-2013, when the assessment frequency was adjusted to biannual to conform with the wider facility assessments that were ongoing. By leveraging EAHPLNP peer evaluation scores, the joint team carried out three full evaluations; in 2013 and 2014, the team carried out one evaluation per year.
26
Table 3: Source of Indicator for PBF Payment. Derived from a Table Provided by SPIU Indicating PBF Payments and Underlying
Score
Evaluation Quarter10
(start month)
Data collection/verification
2012
January* Combined with EAPHLNP peer evaluation
April Joint team
July Joint team
October Joint team
2013
January Joint team
April No score available
July No score available
October Combined with EAPHLNP peer evaluation 11
2014
January No score available
April Joint team
July No score available Payment Mechanisms and Fund Management At the inception, district hospitals received the conditional PBF payments for laboratory performance on a quarterly basis. The score achieved in the assessment determined the amount they received in the subsequent quarter. Payment amounts were calculated by multiplying the SLIPTA composite score (shown as a percentage) with the pre-determined ceiling amount. No PBF allowance was to be paid if a laboratory scored decreased by more than 3 percentage points on the SLIPTA scale, or the satellite lost a SLIPTA star (as shown in Table 4) by losing points and falling back under the threshold level of the previously achieved SLIPTA score. However, this does not appear to have
10 The month indicated marks the quarter in which the data was collected/validated. The indicator score thus obtained would have influenced the PBF payment for the subsequent quarter.
11 For the quarter Oct-Dec 2013, the SLIPTA/EAPHLN evaluation determined PBF payment within the same quarter, rather than for the subsequent quarter as in other cases. This was due to scheduling restrictions on the ground.
27
been systematically enforced as several slippages in the SLIPTA quarterly scores did not lead to a discontinuation of the incentive payments. The theoretical maximum ceilings for the PBF payments if no deductions had been made in the SLIPTA assessment are shown in Table 4. This is largely consistent across sites, with slight variation in Q3-4 of FY2012-13 at Byumba District Hospital only. The standard amounts shown in Rwanda Francs correspond to USD 15,000 in the financial year (FY) 2012/13; USD 7,500 in FY 2013/14, and USD 5,000 in FY 2014/15. As the project entered the exit phase, the maximum ceilings were progressively decreased. The percentages in the table represent the SLIPTA score achieved by each laboratory, and proportion of the actual payout. Example: in Q4 of the financial year 2011/2012, the ceiling amount for the Nyagatare District Hospital Laboratory was 12.2 M Rwandan Franc, of which 48 percent were paid out.
Table 4: Maximum Amount Available (Million RWF) and Percentage Paid Out by Site
Fiscal year 2011/12 2012/13 2013/14 2014/15
Quarter Q4 Q1 Q2 Q3 Q4 Q2 Q1
Max Paid Max
Paid
Max
Paid
Max
Paid
Max
Paid
Max
Paid
Max
Paid
Nyagatare 12.2 48% 9.3 43% 9.5 77% 9.5 43% 9.6 86% 5.0 72% 3.5 76%
Byumba 12.2 52% 9.3 45% 9.4 65% 9.0 65% 9.0 80% 5.0 83% 3.5 71%
Gisenyi 12.2 25% 9.3 48% 9.5 59% 9.5 80% 9.6 81% 5.0 78% 3.5 75%
Gihundwe 12.2 23% 9.3 55% 9.5 74% 9.5 79% 9.6 85% 5.0 87% 3.5 75%
Kibungo 12.2 28% 9.3 43% 9.5 61% 9.5 73% 9.0 74% 5.0 80% 3.5 79%
The HFU in the Ministry of Health is responsible for invoicing and payment of all PBF funds. All hospital funds that are disbursed as PBF – for laboratory and otherwise - are considered income of the district hospital. As such, the hospital management team allocates all PBF funds. It was not possible for the district hospitals to quantify specific purposes that the laboratory-generated PBF funds were used for, as this money is not earmarked but rather is aggregated with all PBF generated by the facility. This is in line with the national guidelines for PBF: all PBF income contributes towards a pooled hospital fund, the so-called “common basket”; indeed the aggregation of funds reduced the likelihood of project-related distortion of income for any specific clinical area. As one interview participant explained, the national PBF leadership team told staff that money that flows in does not have “a colour,” and it is therefore impossible to identify the source of money that is spent towards a set budget. The new ability of laboratories to contribute toward the common basket, however, enabled the laboratory management to participate in budgetary decision-making (see below). An important discussion related to PBF in Rwanda focuses on the extent to which the facility PBF funds can be used for individual incentives for staff (or bonus payments, also sometimes referred to in Rwanda as individual PBF). Until 2011, this was relatively
28
de-regulated, which resulted in large differences between staff and locations. In some anecdotal cases, the individual bonus payments became larger than the base salary. In 2011, uniform salary schemes were introduced that set rules that were applicable consistently across institutions. The new scheme was designed in such a way that more remote areas that scored lower on a national equity index could be more highly incentivized at the individual level. For all staff, individual performance is assessed quarterly on a scale from 0 to 100. All employees with a score of 85 percent and above get the full bonus (subject to availability of funds), while employees with a score below 70 percent do not get any bonus. For all scores between 70 – 85 percent, individual bonuses are provided in proportional increments. Since individual bonus payments are paid from the common PBF fund, both individual performance and the facility-level performance of the different hospital departments combined determine the amount of individual bonuses: the maximum amount achievable is multiplied by the individual and the joint scores. The importance and proportion of individual payments versus facility-level payments was “subject to hot debate” as one respondent put it, especially during one regional EAPHLNP meeting. The Rwandan national PBF coordination team insisted on de-emphasizing individual incentives in favor of facility-level incentives that reward joint effort, an approach that they see as a necessary condition for sustainability and staff cohesion. With decreasing donor financing and an increasing number of health service providers in the Rwandan public health service provision, additional measures have recently been discussed in the MoH to ensure sustainability of PBF for facilities, a major concern of many respondents. The most recent innovation in the national PBF policy in Rwanda was a result of the method introduced by the EAPHLNP laboratory PBF pilot: the combination of a continuous quality improvement programme towards accreditation with PBF financing. Combining quality assessments with PBF data validation creates a synergy that is being further enhanced by integrating supportive supervision. This new combination of accreditation with PBF financing for health facilities as a whole (rather than for the laboratories only) is still in a very early phase and has not been institutionalized at the national level.
PBF IMPLEMENTATION
Previous to the EAPHLNP, laboratories had been subjected to evaluation of performance as a part of the wider facility PBF program in place in Rwanda. With the inception of the pilot, the laboratories were evaluated by an independent team using an internationally-recognized, standardized tool and process for an independent source of funding (variable over the project lifetime as described in the previous section). As the project PBF has decreased, respondents state that the upgraded EAPHLNP laboratories may be able to bring in income to the hospital through the increased test and patient volumes and thus partly compensate for the decline in project funding. The laboratory PBF was perceived to have substantially influenced the relationship between clinical and laboratory staff – as well as the overall perception of the importance of the laboratory in the work of the hospital.
29
Despite the change in PBF amounts, upon introduction of the EAPHLNP it remained the policy to pool funds obtained through the EAPHLNP PBF with other department- or service-based PBF in the hospital in a so-called ‘common basket’. These overall facility PBF funds were distributed by the hospital management team, a small group typically including the CEO, a finance manager, and a couple of clinical staff representatives. The fact that laboratory PBF is pooled with facility PBF means that the role of the hospital management in determining how to use the overall PBF funds is critical. After the lab PBF was introduced, a laboratory representative was included on the committee that determines the budget and use of funds for the hospital in both EAPHLNP facilities visited. This means that although the “say” of the laboratory management was increased, there was no automatic increase of investment into the laboratory as a result of good performance and increasing laboratory PBF to the hospital. Quarterly disbursements by laboratory are shown in Table 5 on the next page. The use of facility PBF funds is subject to Ministry of Health guidelines which prescribe the proportion used towards individual salaries (60 percent), as well as maximum individual PBF bonus payments for the different levels of qualification in Rwanda. As a result, other than special bonuses for excellent achievements, the impact of the laboratory PBF on individual laboratory staff salaries was expected to be relatively small, especially as these were pooled across the entire hospital. Nonetheless, many individuals mentioned the increase in individual PBF as an important potential effect of the lab PBF (out of 31 mentions of the effect of PBF, 21 mentioned the individual bonus), highlighting the positive expectations and perceptions about individual bonuses. Individual bonus payments are viewed as a powerful motivator, but one that can easily become problematic if not sustained. Given that this was a pilot project, hospital management had more autonomy to decide how PBF was spent, especially since facility improvement grants for the lab were already included as a separate intervention that could be used to address quality issues identified through PBF assessments. No significant relationship was found at the laboratory level between the quality improvement and overall EAPHLNP disbursement (log-transformed), PBF funds (log-transformed) and the proportion of PBF payment with LOS regression. The results must be interpreted with caution due to the limited data availability. This may also be partially attributable to the decreasing ceiling effect – that is, that as performance improved, maximum achievable amounts were decreasing. Thus, the financial impact of improved performance over the project lifetime was not linear: even as performance improved, the amount the facility received may have decreased year-on-year.
“The lab used to generate funds through PBF but now they are generating funds through the
tests they are using [providing]. The income generated by the lab is also for other
departments of the hospital. So it’s a matter of building the whole system. They are not looking
at one department.” - EAPHLN Coordinator
30
Source: SPIU
Table 6: Annual Hospital Budget and Total Facility PBF Funds (laboratory and other; FY 13-14)
Table 5: Quarterly PBF Disbursements by Laboratory in Rwanda
31
Figure 8: Actual Facility and Laboratory PBF as a Percentage of Hospital Budget (FY13-14)
As shown in Figure 8, laboratory PBF represented about 1 percent of hospital budget in the satellite labs (FY13-14 taken as example), and between 24-30 percent of the non-laboratory PBF for each hospital. This partially explains the increased importance of the laboratory in the facility, given its new revenue generating role – achieved through PBF, and increased range of tests available.
IMPACT OF PBF ON LABORATORY PERFORMANCE AND HOSPITAL PERFORMANCE
Since PBF was implemented in all – and only – the EAPHLNP laboratories in Rwanda, the evaluation team could isolate the impact of the PBF intervention on laboratory performance by comparing the Rwandan facilities (except NRL) to those outside of Rwanda. We took two approaches to assessing the impact of PBF on performance:
• Compare the performance according to the peer-assessments for EAPHLNP-only laboratories with those in Rwanda that had the added PBF element
• Compare the difference in performance between PBF/non-PBF labs as compared to the SLMTA-only labs (difference-in-difference), statistically controlling for other influences such as performance variation between countries
Rwandan EAPHLNP+PBF Labs Outperform Peers in Other Countries on SLIPTA Scores
First, we looked at the available annual data on SLIPTA assessment scores (2011 – 2014) of all EAPHLNP labs in the region (25 EAPHLNP-only labs, 5 EAPHLNP+PBF labs12). The 2011 data served as the baseline in the analysis, annual data from 2012 to 2014 data made it possible to explore the change after the intervention on an annual basis.
12 NRL Rwanda and NTRL Uganda were excluded from the analysis. The former participated in the SLMTA exercise before the EAPHLN programme start, and the latter started with a very high baseline score, which resulted in a bias in terms of improvement (ceiling effect).
32
Figure 9: SLIPTA Scores of PBF/Non-PBF Labs
Figure 9 reports the mean SLIPTA score by year for the two different interventions among the EAPHLNP laboratories, excluding RW-NRL13, and shows a significant additional improvement in SLIPTA performance (p=0.01). By the end of the project period, the difference in cumulative improvement between the two groups was 11 percentage points more improvement by the EAPHLNP+PBF laboratories over EAPHLNP only. In the period from 2011 to 2012, the EAPHLNP+PBF labs improved by 44 percentage points, while the EAPHLNP-only labs improved by 9 percentage points, a difference of 35 points. This suggests that the PBF may well contribute to a more rapid improvement than the EAPHLNP intervention by itself, although this analysis does not control for confounding factors, for example the possibility that performance in Rwanda was generally better14. PBF has a Signficant Effect on SLIPTA Performance with a Non-intervention Control Secondly, using a more rigorous approach, we compared the improvement in SLIPTA scores of three groups: the EAPHLNP+PBF labs, the EAPHLNP-only labs and non-EAPHLNP (SLMTA-only) laboratories across all countries. We estimated the effect of intervention on SLIPTA scores using a difference-in-difference approach, which calculated the effect by comparing the average change over time in the assessment score for the laboratories where the intervention was introduced to the average change over time for the laboratories that continued with the status quo15.
13 All the following analyses related to PBF evaluation excluded RW-NRL from the sample, because (a) the RW-NRL did not get the PBF intervention, and (b) the SLMTA pre- and post-assessments were made before the EAPHLN project start.
14 At baseline the laboratories in Rwanda were performing somewhat lower than those in the other countries. The selection criteria (i.e. location in cross border areas; targeting vulnerable groups) were the same. In Uganda and to some extend in Tanzania, the selected labs had better infrastructure at the project outset. Baseline balance is not a pre-requisite for the difference-in-difference analysis we conducted for the evaluation.
15 In technical terms, the estimation controlled for year and lab fixed effects, which means time-invariant unobserved heterogeneity of the labs was taken into account.
33
Both EAPHLNP intervention groups (with and without PBF) out-performed the control laboratories. The EAPHLNP+PBF labs outperformed the SLMTA-only control by 20 points (out of 100), a difference that would likely bring a laboratory to the next star-level of SLIPTA. The EAPHLNP-only labs outperformed the same control group by 8 points. The additional performance gains due to PBF were measured at p=0.11, very close to the 10 percent level that is typically reported in the literature. This is a likely positive impact that is influenced by the very small sample size (for EAPHLNP+PBF, n=5). Figure 10 illustrates the difference-in-differences between Rwanda, Tanzania and Uganda, the countries with complete SLIPTA datasets for non-EAPHLNP laboratories. The EAPHLNP laboratories that received PBF (in Rwanda) outperformed their national peers more (17 point difference) than the EAPHLNP laboratories in other countries outperformed their compatriots (4 and 6 point differences).
Figure 10: Comparison of Improvements in SLIPTA Scores, Rwanda, Tanzania and Uganda
+ 32 p.p.
+ 26 p.p. +24 p.p.
+ 49 p.p.
+ 30 p.p. + 30 p.p.
Rwanda Tanzania Uganda
EAPHLN vs Non-EAPHLN LaboratoriesAverage SLIPTA score improvement pre-and post SLMTA
(without NRL Rwanda and NTRL Uganda)
NON-EAPHLN EAPHLN
22
34
Qualitative Data Support Findings on Performance Improvements
Data from the qualitative interviews helped to explain the trends observed in the quantitative data, as well as broaden the definition of impact. Specific to PBF, the most-mentioned impact was on individual salaries and the financial motivation that drove laboratory staff to work
long hours to reach the targets set16. Respondents believe that the PBF helped to increase the pace of improvement, although perhaps SLMTA and strong supervision / mentorship could have driven similar results. Looking back at the conceptual framework described in Figure 1, the cycle depicted (“measure, analyze, improve, repeat”) may have occurred at a faster pace in the Rwandan EAPHLNP-supported laboratories than elsewhere in East Africa, owing to the fact that the PBF intervention required quarterly (rather than annual) assessments of SLIPTA scores, strengthening accountability and promoting a results culture. In several interviews, particularly with mentors, teamwork was mentioned in tandem with the effects of strong government commitment to progress. Multiple respondents noted strong government leadership as a key factor explaining Rwanda’s success. 17 At the same time, this strong enabling environment was viewed as a potential challenge for other countries interested in applying this approach as the environment may differ in other countries.
Teamwork is in part required to improve performance on the short (quarterly) timelines that are being observed as part of PBF measurements. At the same time, the opportunity for improvement and development, fostered by the efforts of the mentor, can help foster team spirit where there previously was just a collegial atmosphere. The regional nature of this project, including experience sharing and healthy competition through South-South collaboration, may also have helped generate team spirit across country teams.
16 Given time constraints and sensitivity of the information it was not possible to get information on absolute and relative increases in individual salaries.
17 Other factors which contributed to the strong enabling environment in Rwanda included the national PBF program; community health insurance scheme rolled out nationwide; and related health policies which fostered facility autonomy and promoted accountability.
“If there was no PBF probably we would only have one star, or no stars at all. Nowadays the laboratory is four stars accredited. Perhaps five stars one day.”
- EAPHLNP Hospital Doctor
Without teamwork, the PBF will not be maintained. Without teamwork, you
cannot have the PBF. - EAPHLN Laboratory Manager
It’s not one person that can create an impact… It is everywhere, right from the Ministry, the district, the national hospital… Everyone is
involved so there is a factor of team spirit. - Mentor, EAPHLN
35
The role and commitment of the central government and other key stakeholders in the health system to achieving these improvements was viewed as a prerequisite for the PBF to work – there appears to have been significant top-down prioritization of this intervention and emphasis on progress that helped to increase the individual motivation as well.
Qualitatively, respondents noted that the impact on the hospital appeared to be significant as well, a positive spillover effect of the project. Mainly, discussion of those effects centered around views that the increased volume of patients were either attracted to or retained by the district hospitals due to the updated laboratory facilities and capabilities. This was an opportunity that increased finances available to the hospital and its prestige. The impact on the hospital was also shown in the improved relationships between clinical and laboratory staff, further described in the following section. In discussions with project managers, the idea that frequent measurement drove a results-focused culture with greater accountability came up spontaneously in multiple conversations. Even when planned measurements were not conducted, internal mechanisms to reach the desired levels of performance were set by the laboratory teams and reflected in individuals’ performance goals. This culture of accountability was a meaningful change derived from the PBF program that drove ongoing performance improvements and was supported by (4) below. Several respondents, including many at the central level, expressed concern about the sustainability of the EAPHLNP and particularly the PBF component. An EAPHLNP mentor observed a change in morale along with the decreases in PBF that have come already in the lifetime of the project at some sites. In the case of Rwanda, given that the PBF approach is institutionalized in the healthcare system and accountability is strong, the risks inherent in transition from project status are mitigated.
MOTIVATION, RELATIONSHIPS, AND THE IMPROVEMENT PROCESS
According to the conceptual framework shown in Figure 1, the following non-technical areas were hypothesized to play a role in EAPHLNP laboratory performance improvements18: increased skills, a higher level of motivation and improved relationships with non-laboratory staff. These factors were confirmed and explained in more detail during the interviews.
18 Because qualitative interviews were held only with intervention labs given the scope of the evaluation, it is difficult to compare qualitative aspects to any control group.
“It is stressful to be assessed in every quarter….it can’t work if the leadership is not stressing like in
Rwanda. In Rwanda it is top-to-bottom; our president is committed and wants everything to be done even the staff, ministers and others - it is very consistent
and everybody is like that. [Elsewhere] they can introduce the PBF system but if the leadership of the
hospital is not committed to regularly monitor activities I don’t know whether it can work.
- NRL staff member
“People were given incentives and motivation [by the PBF]. They would spend sleepless nights here setting
up policies, making sure that everything is in the right place.”
- EAPHLN Hospital Director
36
The Role of Motivation An increase in motivation – different from a change in attitude and awareness, which also played a role – appeared to have an important intermediary and reinforcing role for the impact of the EAPHLNP interventions, as shown in the conceptual framework. Elements of the EAPHLNP had a direct impact on motivation. The improved infrastructure and equipment resulted in a broader range of tests offered (“the package”) and a safer and more modern work environment, which in itself was immensely motivating. Likewise, the possibility to participate in professional development trainings, which was previously limited to clinical staff, and the support of mentors boosted motivation directly.
The motivation of health workers under the PBF approach hinges partially on the fact that they become stakeholders interested in the performance of their facility, more proactive and more motivated. We were able to identify several other sources of motivation in relation with the EAPHLNP interventions in Rwanda (see Figure 11). The conditional individual bonus payments, even though relatively modest for laboratory workers, also contributed to staff motivation. Staff celebrations after a particularly good performance, along with more visibility and recognition by other hospital staff were in part due to PBF and also helped boost motivation. The most important direct motivational effects of PBF, however, were on one hand through its contribution towards the improved laboratory infrastructure, as well as the effect that the laboratory’s contribution to the PBF ‘common basket’ had on the relationships with the hospital management and other clinical staff. The upgraded, state of the art laboratories were bringing in a substantial amount of money, and this brought them up to equal standing with clinical departments19. In any case, the impact of the project on motivation was unarguable and essential to its success.
In addition to those direct effects from the interventions, staff motivation was also influenced indirectly. These indirect effects on motivation could play a critical role for the sustainability of the intervention. Improved knowledge and skills made laboratory staff more motivated. In these trainings, they acquired expertise that was different from what they learned in school, perhaps – in part - more practically applicable and meaningful. Increased skills made people enjoy their work more. A participant reported that the intervention led not only to compliance with the requirement for documentation, it led to a desire for more documentation.
SLMTA training and the regular evaluation of performance with the SLIPTA tool also had indirect effects on motivation. Again, the PBF intervention had an important role to play, because of the need to assess performance on a quarterly basis to determine payments. According to one expert in Kigali, the regularity of the assessments was
19 The national (EAPHLN independent) PBF mechanism in Rwanda on the district hospital level incorporates 11 indicators related to laboratory service, which focus on the results of internal and external quality controls and existence of staff evaluation reports. They play a relatively minor role in the overall PBF assessments and contribute only marginally to the facility PBF revenue.
"With the SLMTA, we realized that what we called ‘good’ is in fact bad. We needed to change. We
needed more organization in the laboratory, in the activities. Even in our heads. People needed to
change. " - EAPHLN laboratory manager
37
more important than the financial incentives. Another expert pointed out that, in addition to new skills, the SLMTA process led to greater accountability-- if you performed worse than previously, or lower than your peer laboratories, a competitive spirit would be kindled, or – in the case of a decrease in performance – staff might even feel ashamed. Through these processes, and further boosted by the example given by the mentor, a change (or ‘opening’, as one respondent called it) of the mind occurred. Figure 11 illustrates which aspects of the project have a positive effect on motivation – the role of PBF both directly on motivation as well as in improving the timeliness of procurement and the increase in variable budgets to procure supplies. This will be contrasted with the initial conceptual framework (Figure 6) in the discussion section of this report.
38
Figure 11: The Role of Skills, Motivation and Relationships in Improved Laboratory Performance
The Role of Relationships Participants in the interviews confirmed that the EAPHLNP interventions had an effect on the relationship of laboratory professionals with clinical staff and hospital management. Rather than being a precondition to improved performance (as conceived in the
conceptual framework), it appeared that the enhanced relationships are an effect of the improved performance. After they are strengthened, these professional relationships with clinical staff also play an important role for sustainability of performance improvement both in terms of not letting down clients and maintaining the overall facility PBF.
Relationships play a role in several different ways. Within the hospitals, the
EAPHLNP contributed to an increased prestige for laboratories and the laboratory technicians. In the district hospitals, this was initially a result of the new technology and the state-of-the art buildings, combined with the hiring of highly qualified staff members (laboratory staff at the “A0” qualification level were not budgeted for at the district level before the EAPHLNP). The institutionalization of standard operating procedures (SOPs) is something other clinical departments are learning from the laboratories. The increase in prestige led to more confidence for the laboratory professionals, which had a positive effect on individual performance.
"If one department fails to achieve the goal that they are supposed to achieve we all go down – the whole
hospital. The PBF reduces by that factor." – Midwife at EAPHLN Hospital
“When you take a form [to the lab] that is not filled well, they don’t admit it. It should be stamped here
and it should be all filled - be it age, sex, identification of the patient and everything should
be filled. So when you take it some sections not filled, they do not admit.”
– Midwife at EAPHLN Hospital
39
The new modern buildings were also seen and admired by patients from the community, who furthermore tangibly benefited from the improvements in technical capacity through the provision of specialized tests at the local level instead of higher-level facilities. The laboratories also received publicity in newspapers and radios about the accreditation programme. Local access to and awareness of these services meant that patients did not have to travel to Kigali in order to get treated, a savings in both time and money.
As in the other countries in the region, within the MoH the status of laboratories, which had been underappreciated in the health system overall, was upgraded on all levels, including nationally. The accreditation programme for the laboratories has been a
pioneering project for the introduction of general hospital accreditation programs that has become policy in Rwanda.
Several interview participants agreed that the improvement of the laboratory services would not have been possible without a
strong spirit of teamwork, which, according to one respondent, was integral to the health system in Rwanda right from the very top of the central government down to the frontline healthcare workers20. For the EAPHLNP-supported laboratories, this was strongly enhanced by the contribution to the facility’s PBF ‘common basket’ through the incentive payments. For the first time, the laboratory staff felt that they were an integral part of the hospital teams. Laboratory managers had more weight and voice in the hospital management and were given a seat on the hospital boards. Whereas in the past the laboratory was perceived as an inferior provider of routine services to the clinical departments, this appears to have evolved into more of a peer relationship over the lifetime of the project. This also manifests in the fact that laboratory workers sometimes may take clinicians ‘to task’ by demanding comprehensively completed lab forms that meet standards. If not completed correctly, forms are sent back, and this is a sign of improved communication and respect between clinical and laboratory staff, as confirmed by both sides.
The more professional client services offered by laboratories, quicker turnaround times, and more reliable test results resulting from the introduction of automated technologies, such as the GeneXpert, has led to an increase in trust in test results by clinical staff. As a spillover effect, the improved lab services also contributed to a better relationship between clinicians and
20 The introduction of district-level PBF in Rwanda initially came with its own teething problems and required gradual improvement of monitoring and incentive systems. In part, the well-functioning processes and teamwork are a result of this early learning almost a decade ago (cp. Fritsche, Soeters, & Meessen, 2014, p. 168.)
“The laboratory staff felt a little isolated, they didn’t really feel they were part of the hospital team. And
so they would leave their work post from time to time but now they are working properly, and they have
the means to work properly.” – EAPHLN Laboratory Manager
“They tell us that because of laboratory performance, even hospitalization time has been reduced. They
understand that the laboratory plays a crucial role in healthcare in hospital, contrary to what they believed
before the Project.” – EAPHLN programme coordinator
40
patients, which is critical for quality clinical care. As doctors and nurses can offer their clients more predictability, a broader range of testing services and shorter turnaround time, the patients are more satisfied21. The director of one of the district hospitals supported by EAPHLNP stated that their clinical staff has “grown fonder of patients” which gives them reason to keep improving and perform better in the future.
Perhaps most importantly, these different ways in which the relationships and the prestige of the laboratory workers grew had an effect on motivation of laboratory staff. As shown in Figure 11 above, this can (at least theoretically) lead to a feedback loop, in which better relationships lead to increased motivation, thus enhancing performance, which again, increases prestige and further improves relationships. Such positively reinforcing feedback loops are likely to contribute to the sustainability of the benefits of this project. This compliments the work of Schneidman et al. (2014), showing the virtuous cycle that could be built through strengthening laboratory systems as shown in Figure 2.
21 Note that this patient satisfaction is based on clinician perceptions; there is no systematically available satisfaction data covering the pilot period.
“Because of these standards, now they can know after how long their results will be out. They are
happy to know that if they send samples, they will get results within a specific time. In improves the
collaboration between laboratory and clinical staff.” – EAPHLN Laboratory Manager
41
IV. FINDINGS RELATED TO THE EAPHLN PROJECT IMPLEMENTATION OF EAPHLNP INTERVENTIONS
This section turns to an evaluation of the full package of interventions supported under the project, which included: (i) construction of new laboratories; provision of modern laboratory equipment and ICT hardware; (ii) recruitment of additional, well-qualified human resources; (iii) training, capacity building, mentorship, and enrollment in the SLIPTA program; (iv) assisted supervision; (v) facility improvement funds; and (vi) PBF22
There are two reasons that this PBF-focused evaluation looks at the impact of the EAPHLNP on performance. First, many projects have shown that PBF works best when not operating alone; strengthening systems more broadly helps to amplify the effects of PBF, and so it is important to look at the effect of the system strengthening that occurred through EAPHLNP. Second, the timing of the introduction of the various EAPHLNP interventions was not standard between the Rwandan EAPHLNP sites and sites in the other countries (e.g. all satellite facilities in Rwanda had civil works completed a year before the others, and the new facilities had an effect on SLIPTA scores). The PBF intervention began at all sites in Q2 2012 but was not introduced in a way that its effects could directly be isolated from other interventions, and so a consideration of the overall effect of the project interventions is important.
The infrastructure part of the financial support was spent on buildings, equipment and technologies (Figure 12)23. Complete financial data on all the interventions, as defined in the conceptual framework, was not readily accessible during the research period. The management of the different EAPHLNP intervention elements varied. Building and equipment procurement were done at the central (SPIU) level and provided to the facilities. Mentorship and supervision were in-kind contributions, also managed centrally (e.g. selection, management and payment of mentors). Funds were distributed from the SPIU to individual laboratories for human resources (hiring and capacity building), laboratory improvements, and PBF. There was some variation in the total amounts that the EAPHLNP supported laboratories received, mostly due to contributions from other donors (e.g. a similar architectural design was used in Nyagatare, an EAPHLNP site, which was funded by the Global Fund rather than by the World Bank). A rough idea of the amounts attributed to different interventions for one of the laboratories (Gihundwe) is shown in Figure 12.
22 Payments to facilities in the frame of the EAPHLN PBF intervention started in April-June 2012 in Rwanda. In Burundi, a small laboratory component is part of the national PBF mechanism for all district facilities with ca. 20 quality indicators; the EAPHLN “boosted” PBF laboratory mechanism similar to the one described in this document, using the SLIPTA checklist, was formally introduced in 2014 after the pre/post measurements reported in this evaluation. The remaining countries (Uganda, Kenya, and Tanzania) also initiated laboratory PBF pilots.
23 Information and Communications Technology interventions would include computerizing the whole hospital, not just the laboratory, in order to reap the benefits, resulting in a higher cost than originally estimated and thus deemed impractical at this phase.
42
Figure 13: Infrastructure expenditures
(2012–2014) Source: MoH SPIU, Kigali
Figure 12: Expenditures for Different Intervention Elements at one EAPHLN Laboratory
Building$1,750,507
Equipment$ 901,344
ICT$ 766,834 Vehicles $43,245
78,658
173,475
337,147
725,936
PBF
Additional staff
Other Disbursements
Infrastructure
43
EFFECT OF EAPHLNP INTERVENTIONS ON SLIPTA PERFORMANCE
The third research question focused on the impact of the EAPHLNP interventions as a whole on laboratory performance. Since the different elements of the project in Rwanda were delivered to all sites around the same time, it was not possible to isolate the impact of the different interventions as would have been desirable. To understand the impact of EAPHLNP on laboratory performance, it was necessary to operationalize “performance”. As part of the qualitative interviews, participants were asked how they defined a high-performing laboratory. Responses ranged from input factors, such as staff number and training levels, availability and maintenance of equipment to output factors, including client trust (an element of satisfaction), safety, documentation, ISO certification, and timeliness of service. As discussed before, and in convergence with these responses, a range of variables including SLIPTA scores, test volumes, turn-around time (TAT), client satisfaction, and test accuracy were proposed as outcome variables for the quantitative assessment. However, only the SLIPTA scores and laboratory test volume data were readily available and accessible for a statistical intervention/treatment group difference-in-differences comparison. East Central and Southern African Health Community (ECSA-HC) were able to share SLIPTA scores from the annual audits of the EAPHLNP. Figure 14 shows the proportion of laboratories in each country that received two stars or more annually over the project period (2011 – 2014), illustrating a clear improvement in performance. However, these figures alone do not indicate to what extent the improvement is attributable to EAPHLNP, SLMTA, or other contextual influences; support from other donors (e.g. Global Fund and CDC) were mentioned by respondents at some of these laboratories, but the bulk of financing was provided by the WB.
Figure 14: Development of SLIPTA Performance in EAPHLN Labs
0% 0% 0%
17%
83%
17%
71%
83%
29%33%
50%
100% 100%
50%
67%
80%86%
100%
Burundi Kenya Rwanda Tanzania Uganda
Proportion of laboratories with 2 or more stars
Baseline 2012 2013 2014
The annual improvement of SLIPTA scores at the EAPHLNP-supported laboratories in Rwanda is shown in Figure 14, illustrating a rapid increase over the project period. Not surprisingly, the largest increase for all satellite laboratories (excluding NRL, which was enrolled in the SLMTA programme before EAPHLNP started) occurred from 2011-2012, which corresponds to the period the laboratories participated in the SLMTA programme.
44
Figure 15: SLIPTA Scores in EAPHLN Labs Over Project Period
65
52
2328 28 28
7583 85
77
60
76
8682
8778 80
72
99
88 91 88 8590
NRL Byumba Gihundwe Gisenyi Kibungo Nyagatare2011 '12 '13 '14 2011 '12 '13 '142011 '12 '13 '142011 '12 '13 '142011 '12 '13 '142011 '12 '13 '14
An EAPHLNP peer-review team composed of experts from the different participating countries carried out the assessments described above. To assess the impact of the EAPHLNP package of interventions on laboratory performance, however, the independently assessed SLMTA baseline and exit scores were used, as they allow for a comparison with the non-EAPHLNP laboratories in all countries, although they do not capture the full effect of the EAPHLNP24. Baseline and exit scores for the SLMTA interventions were assessed at different points in time for several cohorts of laboratories, as the programme was delivered over time, between 2010 and 2015, at different sites (and is ongoing at others). Because of a possible ceiling effect (i.e. that it is more difficult to make large improvements when starting closer to the maximum possible level), facilities that started with a 75 percent score or higher were excluded from the analysis. At baseline across all East African countries, the EAPHLNP laboratories had a somewhat lower mean assessment score compared with the non-EAPHLNP laboratories (32 vs. 34; statistically not significant at a 90 percent confidence interval), as they were all selected based on their remoteness and proximity to cross border regions. In contrast, the mean exit score of the EAPHLNP laboratories was higher than the non-EAPHLNP ones after the SLMTA intervention. A statistical comparison of the difference of improvements after the SLMTA intervention (using regression analysis25) shows that EAPHLNP laboratories had on average 11 percentage points greater improvement than the non-EAPHLNP laboratories. Without the EAPHLNP interventions, laboratories improved on average by 29 percentage points, while the EAPHLNP laboratories without PBF (i.e. all laboratories outside Rwanda26) improved by an average of 41 percentage points. This difference is statistically significant at a 95 percent confidence interval. It can therefore be concluded that the EAPHLNP interventions as a whole, without the addition of PBF, had a positive effect on laboratory quality as measured by the SLMTA assessors at the
24 The effect is likely even greater than what is observed here as the EAPHLNP interventions continued over several years, whereas SLMTA was a one-year program, and as observed in the previous section, PBF likely augments this positive effect. 25 For technical details of the statistical analysis: see Appendix. 26 NRL Rwanda also didn’t receive PBF, but it was excluded from the analysis since it had participated in the SLMTA
programme before the start of the EAPHLNP interventions.
45
end of their programme, highlighting how targeted interventions such as those provided under the project can address the historical neglect of laboratories.
EFFECT OF EAPHLNP INTERVENTION ON RANGE OF TESTS AVAILABLE
One way to measure the impact of the project on the laboratory was to look at the range or package of possible tests that the laboratory could conduct before and after the introduction of the new equipment and training. The new tests added between 2011 and 2014 are shown in Table 7. Although no tests were discontinued by the laboratories, some tests that had been done primarily at hospital level were decentralized to health centres, either because of improved equipment or introduction of point-of-care tests. This also had an impact on test volumes particularly for some routine tests, as further discussed in the following section.
46
Table 7: New Tests Added to Satellite Lab Capacity
Satellite Lab Sites
GeneXpert (TB)
Viral Load (HIV)
ELISA (HIV) Bacteriology Biochemistry
Byumba X
Gihundwe X X X Copro-, uro-, hemo-, pus, CSF cultures
Bilirubin; uric acid; hepatitis B/C (rapid);
Rh factor; cryptococcal antigen
Gisenyi X X X
Kibungo X
Nyagatare X X X Copro-, uro-, hemo-, pus, CSF cultures
Electrolytes; Bilirubin
TB has been a focus disease for the project. Before the introduction of EAPHLN, all sputum samples from TB patients at high risk of developing multiple drug-resistant TB were supposed to be sent to the NRL for culture. This was a time-consuming process of 8 to 12 weeks, after which satellite laboratories would know whether the patient required treatment with 2nd-line TB drugs. With the implementation of the EAPHLNP, sputum samples are first tested at district hospitals and the results are available in a very short time (2 hours). This means that drug-resistant TB cases are being diagnosed earlier, giving patients access to treatment more quickly and decreasing the spread of the disease. According to the available data from 2014, the 5 satellite laboratories tested 6,482 TB suspects and diagnosed 14 new rifampicin-resistant patients. In 2013, only 67 cases were tested in the same laboratories and no resistant cases were detected.
EFFECT OF EAPHLNP INTERVENTIONS ON TEST VOLUMES
To evaluate changes in test volume as a measure of performance, the research team selected a set of four tests (full blood count – FBC, glucose, TB microscopy, and CD4 count). Laboratory records from the time period from January 2012 to December 2014 at 21 of the SLMTA sites were used for the analysis (including the EAPHLNP-supported sites but not NRL). Monthly test volume data were aggregated to quarterly bins to reduce the effect of short-time variations (“data noise”). Rather than looking at absolute figures, the number of tests in proportion to the hospital catchment population was assessed, taking into account population growth rates as reported by the World Bank. Since information about the catchment population was available for only 17 of the facilities, only those were included in the analysis. Test volumes for the selected tests varied between laboratories, most prominently so for Full Blood Count, as illustrated shown in Figure 16 the year 2012, which is not adjusted for catchment population. We estimated the effect of the intervention on the test volume using a difference-in-difference approach, taking into account the different relative test volumes before the interventions.
47
Figure 16: Variation of Absolute Test Volumes (FBC) in 2012
For the first analysis, the outcome was the mean annual number of tests in proportion to the catchment population. The 2012 calendar year was treated as the baseline. The findings show that there was no significant change in test volumes in 2013. In 2014, the EAPHLNP interventions correlated with a statistically significant decrease in the test volume for CD4 and FBC, whereas it led to a significant increase in the test volume of microscopy at a 95 percent level of significance. Secondly, we used the mean number of tests on a half-year basis as the outcome, and treated the first half-year of 2012 as the baseline, and the first half-year of 2013 to the second half year of 2014 as the post-intervention periods. This showed no clear trend effect of interventions on test volumes of the selected tests. Since testing volumes are dependent on population care-seeking decisions and other external factors, the fact that there are no clear trends cannot be interpreted as an intervention or theory failure. Particularly, routine tests that were already well-established at participating laboratories might not be expected to change. Some routine tests were shifted to lower-level health centres (e.g. CD4 tests), where improved equipment and/or introduction of point of care testing permitted these sites to perform these tests in a more cost effective manner. These factors highlight the difficulties and complexities of capturing performance of laboratories based in district hospitals given their variable workloads. In addition to these selected tests, newer more ‘sophisticated’ tests were also measured (that is, viral load, HIV – ELISA, molecular TB GeneXpert). There was no baseline data for these tests as none of the project supported laboratories had the relevant equipment at the outset of the project and these new equipment were part of the hardware investment. In the first half of 2015, project laboratories conducted more than 24,000 viral load tests; 10,000 ELISA tests; and 6000 GeneXpert tests – a clear increase in test volumes.
48
EFFECT OF EAPHLNP INTERVENTIONS ON TEST ACCURACY
Test accuracy data, as a measure of performance, was available for 2014 (April, July, and November for BCHE435 and Haematology; April and July for QASI432 and HIVA435), so a difference-in-differences or baseline-endline comparison was not possible for this performance indicator. We summarized the mean of observed data points for each test and by EAPHLNP status (Figure 17), and also tested the difference of the means. There was no significant difference between EAPHLNP and non-EAPHLNP laboratories in any of the indicators with regards to BCHE435, Haematology and QASI432. Despite a smaller sample size, non-EAPHLNP had a statistically significant higher accuracy rate of HIVA435. Therefore, the EAPHLNP sites performed comparably to the comparison facilities within Rwanda. In addition, we conducted a regression analysis on the percentage of tests reported as ‘acceptable’ by EAPHLNP status, controlling for a set of confounding variables (that is, the number of lab rooms, the number of staffs at level A0, A1, A2, and the number of equipment relevant to respective tests). No significant difference or otherwise meaningful results were found; this is likely related to the low statistical power of the study due to the very small sample size.
Figure 17: Comparison of EQA Results in 2014
47% 58% 62% 63% 60%40%
97% 89%21%
14% 13% 13%10%
15%
11%
33% 28% 25% 25% 31%45%
Non
-EAP
HLN
EAPH
LN
Non
-EAP
HLN
EAPH
LN
Non
-EAP
HLN
EAPH
LN
Non
-EAP
HLN
EAPH
LN
BCHE435 Hematology QASI432 HIVA435
ACC UNACC NE
QUALITATIVE ASSESSMENT OF THE IMPORTANCE OF DIFFERENT INTERVENTION ELEMENTS
The differential impact of the various EAPHLNP intervention elements could not be assessed through quantitative data analysis. To capture a more subjective view of those involved with – or affected by - the EAPHLNP interventions, they were asked as part of the interviews about how they would prioritize interventions based on impact. The same interviews also captured other, broader aspects of impact of the EAPHLNP interventions that go beyond the numbers.
49
Several participants emphasized the impact of the project on the laboratory staff’s awareness of quality, and an increased priority given to documentation and standardization. Respondents saw the effect of the project – especially capacity building, including mentorship - on attitude and motivation as a prerequisite that led to the observed improvements. Most interview participants agreed that it is hard or impossible to assess elements of the intervention individually or rate them by importance, as they are inseparable, and only lead to the desired effects if implemented jointly. A hypothetical question about how interventions would be prioritized if forced to show that infrastructure/equipment and capacity building (which includes the SLIPTA/SLMTA programme) took the lead, closely followed by the hiring of new staff. There were individual differences between respondents in these matters, so they were aggregated in Figure 18, where darker colour represents higher importance, to illustrate the general tendency on how different participant groups emphasized importance. The importance of mentorship and PBF was emphasized more by central government and program management staff than by hospital-level management or laboratory staff, but it is likely that mentorship – even if not mentioned separately - was perceived to be part of training and capacity building, which was perceived to be invaluable across the board. Hospital management is likely to have emphasized PBF given that the funds were added to a facility common basket.
Figure 18: Qualitative Impression of the Perceived Priority for Different Interventions
LaboratoryStaff
KigaliProgramme Staff
Other HospitalMgmt and Staff
Infrastructure, Equipment
Training, capacity building
Hiring additional staff
PBF payments
Mentorship
Improvement funds
Supervision
“The first thing is the change in ideas. The mindsets. It’s the SLMTA that pushed us to change. Before the SLMTA everything
was good enough for us. But with the SLMTA, we realized that what we called
“good” is in fact bad. We needed to change. We needed more organization in
the laboratory, in the activities. Even in our heads. People needed to change.”
- EAPHLN-supported laboratory manager
50
Role of Mentorship Under the EAPHLNP
• Involvement of the mentor in the project from its inception helped to improve
their ownership and engagement with the work.
• Introduction of the mentor as a collaborator or resource person rather than an auditor to the laboratory and the administration helps reduce resistance and improve utility.
• Contact time is an essential component of mentorship – and more is almost always better. More than one respondent suggested that full-time mentorship could have been helpful. The contact time was increased upon feedback from the mentors after the inception of the project.
• Minimum time to see impact was mentioned between two weeks per quarter and two weeks per month.
• Mentors can also serve as advocates, either increasing the ‘voice’ of the laboratory within the hospital management or at the national level in Kigali.
• This relationship can be maintained even after the formal end of the mentorship period.
In a mentor’s own words: “I just lived with them, doing what they do every day, slowly doing the work as they see what you are doing, demonstrating instead of giving instructions. Most of the time I used to do the things myself - not as their additional employee, but I would do such things so that they can observe and pick up.”
51
V. DISCUSSION OF FINDINGS AND LESSONS LEARNED FOR THE INTRODUCTION OF LABORATORY PBF IN OTHER COUNTRIES
SUMMARY OF FINDINGS
In summary, the previous pages speak to the putative effect of PBF and the EAPHLN as a whole in improving laboratory performance in Rwanda, although due to the project design we are unable to say whether the PBF was the main catalyst increasing the speed of the improvement or the cause of the improvement. The findings are summarized here as a basis for analysis and discussion: Implementation of PBF in Rwanda
The implementation of the laboratory PBF benefited from the well-established PBF programme in the Rwandese healthcare system which focuses on results, good governance and accountability. The frequent assessments were viewed to be beneficial in terms of enhancing accountability and focusing on results even if individual financial incentives for laboratory workers were sometimes modest and the assessments were intense in terms of costs and stress reported by some stakeholders. The PBF scheme also had a positive side-effect of improving the profile of the laboratory within the facility. The laboratory PBF revenues were pooled with other sources of PBF at the facility level to avoid creating distortions. Individual bonus payments appeared to be motivating, with the caveat that if they were to be reduced or removed in the future, an effective strategy would need to be found to mitigate the risk of negative effects on motivation.
Effect of PBF on EAPHLNP-supported laboratories in Rwanda
PBF appears to have contributed to an accelerated change in the laboratories where it was implemented, with PBF laboratories measuring an overall greater increase in SLIPTA performance compared to other EAPHLNP supported laboratories in the other participating countries. The observed magnitude of outperforming national comparison laboratories in SLIPTA scores from baseline to endline between the EAPHLNP-only laboratories in other countries and EAPHLNP+PBF in Rwanda, controlling for other factors, was 12 percentage points, with p=0.11. This was borne out by the qualitative interviews emphasizing both pressures and incentives to conduct quarterly assessments (rather than annual), perhaps speeding up the measurement-improvement cycle. Nevertheless it should be noted that it is difficult to isolate the effect of PBF.
Role of skills, motivations, and relationships on laboratory and hospital performance Motivation was positively affected through direct and indirect processes. Relationships between laboratory staff and clinicians improved over the project period, partially a result of publicized performance improvements and partially as a result of improved turnaround time (TAT) of laboratory results. The contribution to the PBF ‘common basket’ has a strong impact on the relationship of the laboratory manager with the hospital management, increasing the visibility of the laboratory within hospital decision-making and the reputation of laboratory professionals. Improved communication and documentation led to mutual respect within the hospital. A spirit of teamwork prevailed at participating facilities which, combined with strong commitment of the central government, contributed to successful implementation. It is anticipated that the qualitative changes will be sustained, given the contribution of the laboratories to increased hospital revenues. This assumption would require a follow-up study to be confirmed or potentially revised.
52
Impact of the EAPHLNP package of interventions
A clear impact of the EAPHLNP interventions on SLIPTA score improvements was demonstrated both inside Rwanda (19 points at p=0.01) and outside27 (10 points at p=0.01). As a result of the project, the range of sophisticated tests offered at the district level at the intervention sites increased substantially, which saved time for clients in need of those tests, and avoided the need to travel to Kigali. The impact of EAPHLNP on test volumes was measurable for TB microscopy (increased 10 percentage points at p=0.05), showed no impact on FBC, and showed a measurable decrease for CD4 counts, possibly as a result of increased responsiveness of EAPHLNP laboratories to a new policy that prescribes less frequent CD4 counts and increased availability of point-of-care devices in health centres. No measurable difference was shown on results of external quality assessments at the endline (impact was not measured due to a lack of baseline data). Since the different elements of the EAPHLNP were delivered almost homogeneously, it was not possible to isolate the impact of the different interventions. Although these cannot be distinguished, there is clearly a system-strengthening value to introducing a package of interventions including PBF. There also appear to be positive effects on patient retention and public reputation for the district hospitals associated with the laboratory upgrading.
VALIDITY OF THE CONCEPTUAL FRAMEWORK The authors revisited the conceptual framework from Figure 6, assessing its validity in light of the qualitative findings. Respondents were not questioned about the framework directly, but their comments on the process of change in performance informed this assessment. From the outset, the list of EAPHLNP interventions or inputs was known. Performance improvements in the course of the project occurred incrementally over time (rather than in a single period), suggesting that as hypothesized an internal cycle of assessment and improvement was occurring. PBF, with its requirement for frequent measurement, is a means to increase the turnover rate of this cycle and thus increase the speed of improvement. As seen in other PBF schemes, larger improvements are possible at the outset when the baseline performance is lower. The marginal improvements decrease naturally as a laboratory approaches the maximum possible score. When we bring in the findings from Figure 11, it is clear that the proposed way the quantitative SLIPTA-based quality improvement cycle interacts with qualitative improvements is over-simplified. Instead of improvements in motivation and relationships coming subsequent to the quality improvement cycle, it appears that these improvements actually help to drive that cycle (like PBF) and that faster or larger improvements in these areas can similarly speed quality improvements as measured by SLIPTA. Overall, EAPHLNP appears to have had a positive effect on both the qualitative performance indicators as described in the two frameworks (relationships, motivation, and skills) and quantitative measures of performance, particularly SLIPTA scores. Because Rwanda systematically evaluated only SLIPTA scores as a dimension of performance, it is unclear whether improvements in SLIPTA scores are an intermediate step leading to improvements in other quantitatively-measurable areas of performance (e.g. test volumes, EQA, TAT) as
27 Note that missing data were excluded from analyses rather than imputed, which may have introduced bias.
53
hypothesized, whether all areas improve simultaneously, or whether there is little linkage between improvements in the various dimensions of performance.
V. LESSONS LEARNED FOR OTHER IMPLEMENTERS OF LABORATORY PBF
Based on these findings and the nuanced details presented in the previous sections, there are a number of important lessons learned that other countries considering the introduction of a laboratory PBF programme should take note of: PBF instills a results-focused culture and fosters accountability. The findings of
this evaluation suggest that PBF contributes to laboratory performance improvements, as facilities and individuals are held accountable for improvements in the SLIPTA scores. PBF is an effective approach to health systems strengthening, that can be used to strengthen motivation but not promised as a regular individual salary top-up. Facility-focused PBF is effective without raising the same expectations as individual-bonus payments. Interviews showed that individual bonus payments bear the inherent risks of threatening sustainability and leading to a decreased motivation if they are removed. There needs to be strong leadership and decisive communication by public sector managers at the introduction of a PBF scheme, to manage expectations both of the facility overall, the laboratory, and the individuals affected. A difference in perceptions of the role of individual bonus payments was observed at different levels of the health system: at the central, more ‘removed’ level in Kigali the importance of individual payments was given less importance, while in the field it was emphasized.
Linking PBF to accreditation provides further motivation for facility management. PBF is often initiated as a project-based intervention; in the case of the EAPHLNP, significant time and resources were used to educate the facility management about the project’s goals and strategies. While important, another means of achieving buy-in from management outside the laboratory is by linking PBF indicators and outcomes to laboratory accreditation – the EAPHLNP may constitute one of the first attempts at providing financial incentives for progress towards accreditation of laboratories. This is efficient, as it allows the teams to merge the quality assessments with the PBF evaluations. This innovation from the laboratory PBF has triggered discussions around combining facility PBF with hospital accreditation in Rwanda.
Teamwork/motivation of the laboratory teams was significant. There was some disagreement between respondents about whether the ‘team spirit’ that was observed came as a function of the project and particularly the common performance goals or whether this was a prerequisite for successful teams that was particularly strong in Rwanda. Either way, teamwork was observed by many respondents as a significant factor related to PBF programs. Future implementers should consider the possible team benefits that PBF might engender as a qualitative aspect of motivation. Teamwork is a key factor that seems to underlie success in performance improvement.
PBF for laboratories contributes to their perceived importance in health systems. In the course of the study, one of the outstanding changes in the hospital overall was the relationships between management /clinicians and laboratory staff. There was
54
improved recognition of the contributions of laboratory personnel as well as their technical and organizational skills, sometimes elucidated by requests for better documentation from clinicians. Laboratory professionals and their contribution to the health system are often under-appreciated, and this change in perceptions appeared to also be reflected at the MoH level, where increased visibility of laboratories raised their status. This can help the system with issues around appropriate clinical care and differential diagnosis, in addition to improving the motivation of individuals. Revenue generation through PBF or other approaches can raise appreciation of laboratory personnel.
PBF works within a broader enabling environment. The effect of PBF by itself is not
shown in this evaluation as it was implemented as part of the broader EAPHLNP package of interventions in a strong healthcare system. However, in agreement with the views expressed by respondents to this study (who found it hard or impossible to assess elements of the intervention individually or rate them by importance, as they only led to the desired effects if implemented jointly), other research has shown the amplifying effect of system strengthening interventions on performance-related outcomes. PBF is an approach that has to be introduced together with an overall quality improvement process supported by other forms of resource deployment and capacity development. As found in other studies, there is a criticality of initial investments in infrastructure and human resources as well as capacity building for PBF to work. It is noteworthy that even though the resources available for PBF declined over the pilot phase performance continued to improve, underscoring the combined effect of a better working environment, more qualified staff, and enhanced recognition of the laboratory personnel in the participating hospitals.
Selection of performance measures for laboratory PBF is critical, as it drives what
teams focus on, or, in the time-tested phrase: “what gets measured gets done”. The SLIPTA process, while an easily implementable tool, focuses primarily on documentation and process indicators. Using this standardized, internationally recognized performance metric is necessary but needs to be complemented with other metrics (e.g. accuracy of test results, turnaround time and client satisfaction) to provide a more comprehensive picture of performance and to identify weak points in the system from an operational point of view. While the collection of additional indicators will generate administrative costs, this appears important and partly addressable through enhanced laboratory management information systems.
Cycle of PBF provides opportunity to accelerate performance improvements. Frequent assessments drive both individual and institutional accountability; SLIPTA assessments showed consistent improvements in most facilities throughout the project lifetime. Although some respondents shared concerns that a quarterly PBF cycle was too frequent the regularity of the assessments may have helped to maintain the focus on results. Numerous stakeholders noted that PBF may motivate a more rapid improvement in the quality improvement process towards accreditation. Other countries considering PBF laboratory schemes will need to balance the benefits of frequent assessments against the additional costs.
55
Gains in performance may be sustained if institutionalized. Clear gains in performance on SLIPTA scores have been measured during the project period. Use of a standardized global tool (SLIPTA) and peer assessments led by regional experts contribute to institutionalizing the approach and enhancing chances of sustainability. The need for ongoing performance measurement was mentioned by many respondents in the course of this study, particularly referencing concerns about the impact of its termination on staff motivation. Institutionalizing the benefits of laboratory PBF in the national Rwandan healthcare PBF, even if in a modified form, will be important to sustain the gains in performance. The benefits of this common goal and observable reward on the laboratory teams are significant. The new approach of combining performance measurement and rewards with an accreditation program was recognized as a promising innovation by the MoH Health Financing Unit for the national PBF program in Rwanda, and at the time of writing, pilots were underway to test the introduction of that approach for hospitals.
56
REFERENCES Basinga et al., 2011. Effect on Maternal and Child Health Services in Rwanda of Payment to
Primary Health-care Providers for Performance: An Impact Evaluation. The Lancet ; 377 (9775): 1421-1428.
De Walque et al., 2013. Using Provider Performance Incentives to Increase HIV Testing and
Counseling Services in Rwanda. The World Bank, Policy Research Working Paper 6364; February 2013.
Fritsche, György Bèla, Robert Soeters and Bruno Meessen. 2014. Performance-Based
Financing Toolkit. Washington, DC: World Bank. License: Creative Commons Attribution CC BY 3.0
Gertler & Vermeersh. 2012. Using Performance Incentives to Improve Health Outcomes. The
World Bank, Policy Research Working Paper 6100; June 2012. Nzabahimana I, S. Sebasirimu, J. B. Gatabazi, et al. Innovative Strategies for a Successful
SLMTA Country Programme: The Rwanda Story. Afr J Lab Med. 2014;3(2), Art. #217, 6 pages. http://dx.doi. org/10.4102/ajlm.v3i2.217.
Rusa, L., M. Schneidman, G. Fritsche and L Musango. 2009. “Rwanda: Performance Based
Financing in the Public Sector”, in Eichler R (ed.) Performance Incentives for Global Health: Potential and Pitfalls. P.189-214. Brookings Institution Press: Baltimore, MD.
Schneidman, Dacombe & Carter. 2014. Laboratory Professionals in Africa: The Backbone of
Quality Diagnostics. The World Bank, Health, Nutrition & Population Discussion Paper. Washington, DC: World Bank.
World Bank. 2012-14. World Bank Indicators. Washington, DC: World Bank.
57
APPENDICES
APPENDIX 1: TERMS OF REFERENCE FOR THE LEAD CONSULTANT (ABRIDGED)
Objective The overall goal of this consultancy is to evaluate the performance of the project-supported laboratories in Rwanda and to assess the implementation and impact of the Bank-funded PBF incentive payments implemented through the Cellule d’appui à l’approche contractuelle (MOH/CAAC).
Specific Objectives
To this end, the consultancy will involve: 1. Development and application of a methodology for assessing performance of
participating facilities; 2. Analysis of the PBF modalities and assessment of the impact of these payments; 3. Write up of the main findings and conclusions; and 4. Participation in a workshop to share findings from study.
Specific Tasks
In collaboration with the MOH-SPIU, RBC, MOH/CAAC, and the Bank team, the consultant will, jointly with technical experts, carry out the following specific tasks. To this end, the consultant will conduct and synthesize primary research findings along with secondary research and deliver a full report and a workshop presenting results.
1. Develop and apply a methodology for assessing the performance of the participating laboratories over the past five to six years, taking into account the various interventions supported under the project. This will include developing the overall research design, inventory of existing data and reports, finalizing data collection tools, data analysis from qualitative and quantitative studies.
2. Propose and apply a methodology for systematically assessing the effects of the performance based financing payments on the overall performance of the participating laboratories/hospitals. This will include the identification of adequate research participants/ respondents (with the support of SPIU, RBC and CAAS), finalization of qualitative discussion guides, conduct qualitative interviews in English, and coding and analyzing the qualitative interviews.
As a part of tasks (1) and (2), data to assess overall performance of participating laboratories and potential PBF impact will be collected and analyzed.
3. Prepare a report summarizing the main findings, conclusions, and recommendations derived from the analysis, interviews, and focus group discussions, and participate in a disseminating workshop as a facilitator.
4. Coordinate inputs and technical support, fieldwork, and analysis from qualitative and quantitative studies.
For additional details on the activities, please see details on core activities below.
58
Field Work
The SPIU, RBC, and CAAC will assist in the identification of the institutions to be studied and the individuals to be interviewed. The SPIU and CAAC will provide to the consultant the PBF procedure manual and will facilitate access to information on project-funded activities and to facility information systems. The SPIU in collaboration with the CAAC will provide logistic support for the field work.
Timeline
Inception phase 20 Jan 2015 Assessment of Performance and PBF Implementation phase 27 Feb 2015 Analysis and reporting phase 27 Mar 2015 Final report and presentation Apr 2015 TBD
Deliverables for Lead Consultant:
• An inception report • A draft report • Final report • Workshop presentation
Details on Core Activities (Joint Specifications for Lead Consultant and Support Consultant)
Develop and apply a methodology for assessing the performance of the participating laboratories over the past five to six years, taking into account the various interventions supported under the project. This would include an analysis of both pre/post intervention trends as well as a comparison of performance at non-project sites (which will be carefully selected by matching these to intervention facilities based on pre-intervention characteristics). The consultant will need to work in cooperation with the partners noted above to determine appropriate outcome measures to assess “performance,” given data availability/structure, in addition to information on the broader literature on health systems/laboratory performance. A data collection guide has been elaborated for this exercise. Propose and apply a methodology for systematically assessing the effects of the performance based financing payments on the overall performance of the participating laboratories/hospitals. To this end, describe in detail the process of making PBF payments linked to the SLIPTA composite scores, including appropriateness of the SLIPTA indicators (i.e. both the number and type of indicators), payment levels, how the funds were used at each facility, and what has changed as a result. To isolate the PBF effect (vs the other interventions funded under the project), the consultant will propose a research design to isolate the PBF effect which would include but not be limited to the following: (a) facilities where different items and interventions were phased in at different times or were phased out at different times; (b) anything cyclical
59
about the intervention package -- for example, infrastructure investments and equipment (e.g. GeneXpert or other molecular technologies) additions are "lumpy" so it may be easier to see a particular spike in activity after such one-off events, which could possibly be accounted for (and "subtracted from") any remaining overall trend; and (c) positive or negative spillover effects.
. Collect and analyze data to assess overall performance and potential PBF impact. This would include both quantitative and qualitative data. The following type of information would be collected and analyzed, based on data availability:
o Processes and outputs at the lab level (all by disease type): number of specimens received, number of tests performed (and as % of specimens received), time from specimen to test result, multiple similar tests on the same sample, trends in number of patients picking up laboratory results, quality indicators like testing kit stock outs, specimen spoilage and waste.
o Linking data above with patient and/or pharmacy records, if feasible, would provide more options -- time to patient's diagnosis, time to initiation of drug regimen, number of drug regimens per patient, re-visitation rate, multiple testing requests, choice of test/use of test to diagnose (vs other clinical indicators). This would get at issues related to quality of services, efficiency of services, and confidence in results).
o Conducting structured interviews with clinicians, laboratory personnel, and hospital administrators to determine any changes in level of interaction between physicians and laboratory personnel; views/attitudes/practices, such as changes in confidence in lab results (e.g., use of lab tests versus clinical judgment for diagnosis); issues of accountability (within the lab, and in using lab results), how the lab-linked PBF funds are spent, how the various intervention items work in tandem (or not), and what are the prospects for sustainability once the project closes
Prepare a report summarizing the main findings, conclusions, and recommendations derived from the analysis, interviews, and focus group discussions. The report would be 20 to 30 pages and would include annexes with detailed data and other information. The report would outline the results from introducing the PBF approach and provide guidance to other countries/stakeholders interested in introducing this scheme.
60
APPENDIX 2: FINAL QUALITATIVE SAMPLE
Respondent Type Sampling criteria Number of interviewees
Hospital CEO/in-charge Facility containing non-NRL EAPHLN laboratory 1 per site = 2 total
Hospital finance director
Facility containing non-NRL EAPHLN laboratory 1 per site = 2 total
Laboratory staff
One lab in-charge and one lab technologist (first choice:
quality or safety officer) who has been in post for at least 3
years
2 per site + 2 at NRL = 6
Clinical staff
People who see the patients and prescribe the lab tests – one IP and one OP physician
or nurse/clinical officer
2-3 per site = 6 short (10-15 min short
interviews)
SPIU Monitoring and Evaluation for projects 1
NRL Quality management team 1
Non-lab PBF implementer
Current or former head of PBF Division 1
EAPHLN mentors 2
EAPHLN coordinator Claver Kayobotsi 1
TOTAL 15 long / 6 short
61
APPENDIX 3: SAMPLE QUALITATIVE DISCUSSION GUIDE (ENGLISH VERSION)
Guide for SPIU, MOH, BMC and EAPHLN
Introduction (5 min.)
• Please introduce yourself briefly: o What is your role in your organization, and in the EAPHLN project? o How long have you been working in this role? o What are the greatest challenges related to your work with EAPHLN? How do
you overcome them?
EAPHLN interventions (RQ 1 and 2) (15 min.)
• How do you define a high-performing laboratory in the Rwandan context? o Are there any additional aspects? Probes (and get reasoning for each): o Are there any tests a laboratory of this size should be able to conduct, but that
are sometimes lacking in Rwanda? o How about number of tests? o How about turnaround time? o How about external quality assessments? o How about data quality? o How about SOPs? o How about equipment?
Please think about all the different interventions that EAPHLN has offered.
• Which aspects of EAPHLN would you say are the most impactful, and why? Which laboratory(ies) provides a good case study of this/these? Do you think the successes are context-specific?
• I will mention to you a number of different interventions. Please describe for each of them the positive and negative aspects, and what change – if any – it has brought to the laboratories with which you interact, according to your own personal experience.
o Laboratory improvement funds o Hiring of new staff o Facility upgrading – infrastructure, information technology, equipment o Training and capacity building other than SLMTA o SLMTA participation o Supervision o Operations research
62
• Tell us in some more detail how mentorship influences laboratory performance. o How important is contact time of the mentor with the laboratory? o Does more time with the laboratory automatically translate into faster progress –
or what is optimal time use of the mentors?
Performance Based Financing (RQ 3) (15 min.) • Can you tell me about how performance based financing was implemented for
laboratories in Rwanda? o What were the highlights or challenges of the process? o How did you resolve the challenges?
• Can there be scenarios where PBF has a negative effect? Have you observed any of these in practice?
o How about unmet expectations for funding by management?
• In relation to the overall funding, how significant were the laboratory PBF amounts to hospitals?
• How important were the bonus payments that were made directly to laboratory staff? o How have these payments influenced staff motivation and performance?
• Do you think that the laboratory PBF will continue without the funding of EAPHLN?
Why/why not?
• What would be the conditions for a laboratory PBF programme to work without any donor-funding?
I would like us to talk a bit about the way performance is measured for the PBF.
• Are there any of the indicators that you know about that presented challenges? Why? • Can you think of other indicators (outside of SLIPTA assessment) that would be
valuable as an indicator for PBF? Why?
Process of Change (RQ 4) (15 min.)
° When you look at the improved performance that resulted from any of the EAPHLN interventions (beyond PBF), how would you describe the process that led to that change? (in the hospitals, in the laboratories)
o Can you give specific examples? o What role did your organization have in supporting that?
° Have you encountered challenges that may have reduced the impact of EAPHLN
interventions?
63
o Can you give concrete examples?
° How have changes in management influenced the process of laboratory improvement? o Think of both positive and negative effects o How about changes in management structure? o How about changes of individuals? o Please give concrete examples
° How can potential negative effect of changes in management be mitigated?
o Are there examples where that worked?
° Based on everything we discussed today, do you have any additional comments, or question you would like to ask?
THANK THE PARTICIPANT AND END THE INTERVIEW
64
APPENDIX 4: QUANTITATIVE ANALYSIS TABLES
Table A4-1: Intervention Effect on PBF Score
VARIABLES All EA labs
excluding RW-NRL
All EA labs
All labs in RW
All labs in UG
All labs in KE
All EAPHLN labs excluding RW_NRL
y2012 9.190***
(3.440) y2013 27.61***
(3.540) y2014 33.91***
(3.540) y2012XPBF 35.02***
(8.472) y2013XPBF 20.40**
(8.513) y2014XPBF 22.65***
(8.513) post 28.49*** 28.49*** 27.17*** 22.89*** 40.84***
(1.718) (1.726) (2.882) (4.435) (2.987) postXEAPHLN_only 8.225** (3.946) postXEAPHLN_PBF 20.51*** (6.956) postXEAPHLN 10.51*** 18.66*** 11.09 19.16* (3.541) (5.643) (9.090) (9.447) Constant 34.60*** 34.68*** 33.16*** 37.66*** 33.02*** 42.22***
(1.066) (1.066) (1.752) (2.738) (2.004) (2.248)
Observations 200 202 46 42 40 119 R-squared 0.818 0.813 0.895 0.703 0.928 0.691
Number of id 100 101 23 21 20 31 Baseline mean of
control 35.23 35.23 35.74 34.79 34.41 46.20
F-test 2.602 Prob>F 0.110 Standard errors in parentheses
*** p<0.01, ** p<0.05, * p<0.1
Effect of the intervention on PBF assessment score We estimated the effect of intervention on assessment score using a difference-in-difference approach, which calculated the effect by comparing the average change over time in the assessment score for the labs where the intervention was introduced to the average change over time for the labs that continued with the status quo. The estimation controlled for year and lab fixed effects, which means time-invariant unobserved heterogeneity of the labs has been taken into account. Table A4-1 shows all the estimation results related to assessment score.
65
Test Volume
Descriptive statistics
The accessible data contained monthly test volume from January 2012 to December 2014 of 21 labs in Rwanda (an intervention lab (NRL) and a comparison lab (King Faisal Hospital) were missing). We aggregated the monthly volume to quarterly volume since monthly value may pick up more noise rather than reflect the impact of intervention. The main outcome variable is the proportion of volume in relation to the catchment population tested, which is only available for 17 labs (catchment population wasn’t available for the remaining four). The mean of proportion tested –on a year basis as well as half a year basis- is presented in the Table A4-2 by test and by EAPHLN status. It shows that, except for FBC test, EAPHLN labs always had higher proportion tested than that of non-EAPHLN labs.
Regression estimation
We estimated the effect of intervention on the test volume using a difference-in-difference approach. The results are reported in Table A4-3. For column (1)-(4), the outcome is the mean of proportion tested on a year basis, i.e. we treated 2012 as the baseline, and 2013 and 2014 as the post-intervention periods. In 2014, the intervention led to a significant decrease in the test volume of CD4 and FBC, whereas it led a significant increase in the test volume of microscopy. However, there was no significant change in 2013. We also used the mean of proportion tested on a half a year basis as the outcome, and treat the first half year of 2012 as the baseline, the first half year of 2013 to the second half year of 2014 as the post-intervention periods. There was no clear trend of the effect of intervention on test volume.
Table A4-2: Test Volumes (1)
N mean (%CD4)
mean (%FBC)
mean (%glucose)
mean (%microscopy)
2012 (y2012) Non-EAPHLN 12 0.35 0.73 0.30 0.09 EAPHLN 5 0.59 0.66 0.42 0.11
2012 1st half (hy1)
Non-EAPHLN 11 0.38 0.86 0.35 0.12 EAPHLN 5 0.58 0.74 0.41 0.13
2012 2nd half (hy2)
Non-EAPHLN 12 0.35 0.66 0.28 0.08 EAPHLN 5 0.59 0.58 0.43 0.09
2013 (y2013) Non-EAPHLN 12 0.41 0.85 0.30 0.10 EAPHLN 5 0.61 0.59 0.48 0.15
2013 1st half (hy3)
Non-EAPHLN 12 0.39 0.95 0.29 0.10 EAPHLN 5 0.61 0.65 0.45 0.14
2013 2nd half
66
(hy4) Non-EAPHLN 12 0.43 0.75 0.30 0.10
EAPHLN 5 0.62 0.52 0.52 0.16 2014 (y2014) Non-EAPHLN 12 0.43 1.01 0.33 0.11
EAPHLN 5 0.50 0.68 0.53 0.24 2014 1st half
(hy5) Non-EAPHLN 12 0.51 0.95 0.33 0.12
EAPHLN 5 0.57 0.65 0.53 0.25 2014 2nd half
(hy6) Non-EAPHLN 12 0.36 1.08 0.32 0.10
EAPHLN 5 0.42 0.72 0.53 0.22 Note: the statistics are in percentage point, i.e. 0.35 means 0.35%.
Table A4-3: Test Volumes (2)
(1) (2) (3) (4) (5) (6) (7) (8) CD4 FBC glucose microscopy CD4 FBC glucose microscopy
y2013 0.0625 0.115* -0.00769 0.00608
(0.0410) (0.0676) (0.0426) (0.0220) y2014 0.0858** 0.279*** 0.0223 0.0184 (0.0410) (0.0676) (0.0426) (0.0220) y2013XEAPHLN -0.0366 -0.187 0.0687 0.0384 (0.0755) (0.125) (0.0785) (0.0406) y2014XEAPHLN -0.175** -0.256** 0.0852 0.107** (0.0755) (0.125) (0.0785) (0.0406) hy3 0.0215 0.151 -0.0517 -0.0168
(0.0467) (0.107) (0.0461) (0.0253) hy4 0.0651 -0.0505 -0.0368 -0.0169
(0.0467) (0.107) (0.0461) (0.0253)
hy5 0.142*** 0.148 -0.00949 0.00152
(0.0467) (0.107) (0.0461) (0.0253)
hy6 -
0.00897 0.280** -0.0189 -0.0107
(0.0467) (0.107) (0.0461) (0.0253) hy3XEAPHLN 0.00765 -0.236 0.0851 0.0305
(0.0845) (0.194) (0.0835) (0.0457) hy4XEAPHLN -0.0286 -0.163 0.141* 0.0500
(0.0845) (0.194) (0.0835) (0.0457) hy5XEAPHLN -0.146* -0.239 0.127 0.119**
(0.0845) (0.194) (0.0835) (0.0457) hy6XEAPHLN -0.152* -0.298 0.132 0.0997**
(0.0845) (0.194) (0.0835) (0.0457) Constant 0.419*** 0.713*** 0.338*** 0.0974*** 0.433*** 0.789*** 0.363*** 0.120***
(0.0243) (0.0401) (0.0253) (0.0131) (0.0280) (0.0642) (0.0276) (0.0151)
67
Observations 51 51 51 51 84 84 84 84
Number of labs 17 17 17 17 17 17 17 17 Mean at baseline
- Control 0.349 0.734 0.303 0.0923 0.381 0.856 0.349 0.117 - Treatment 0.586 0.661 0.422 0.110 0.579 0.738 0.414 0.131
Standard errors in parentheses; *** p<0.01, ** p<0.05, * p<0.1 Sample: all labs in RW; the total number should be 23 but data only available from 17 labs.
Test Accuracy
Test accuracy data is only available for 2014 (April, July, and November for BCHE435 and Hematology; April and July for QASI432 and HIVA435). We summarized the mean of observed data points for each test (Table A4-4), graphed them by EAPHLN status (Figure A4-1), and also tested the difference of the mean. There was no significant difference between EAPHLN and non-EAPHLN labs in any indicators with regards to BCHE435 and Hematology; non-EAPHLN had significant larger number of analytes with regards to QASI432; with significant smaller number of analytes, non-EAPHLN had significant higher accuracy rate of HIVA435. Besides, we regressed the accuracy of each test on a set of EAPHLN status, controlling for a set of confounding variables (i.e. the number of lab rooms, the number of staffs at level A0, A1, A2, and the number of equipment). See Table A4-5 and A4-6 for the descriptive statistics and the regression results, respectively. No expected differences were found (either they were not significant or did not make sense). This may be caused by the extremely limited sample size.
Table A4-4: Test Accuracy (1)
test ACC UNACC NE Analytes
Non-EAPHLN
BCHE435 46.73 20.67 32.61 1431 Hematology 62.06 13.48 24.50 2241
QASI432 59.62 9.62 30.77 200 HIVA435 97.22 2.78 0.00 109
EAPHLN
BCHE435 58.33 13.87 27.87 1040 Hematology 62.73 12.67 24.60 2620
QASI432 40.00 15.00 45.00 160 HIVA435 89.00 11.00 0.00 170
68
Table A4-5: Test Accuracy (2)
Obs Mean Std. Dev. Min Max BCHE_ACC 22 49.36 25.09 5 98.67 HEMA_ACC 22 61.29 23.62 0 93.33 QASI_ACC 18 54.17 29.70 0 100 HIVA_ACC 22 95.45 8.99 75 100 EAPHLN 22 0.23 0.43 0 1 lab_room 17 8.71 5.37 1 20 staff_a0 17 2.65 3.77 0 15 staff_a1 17 5.00 4.47 0 19 staff_a2 17 3.94 2.59 1 10
equip_BIO 17 2.29 0.69 1 3 equip_HEMA 17 1.88 0.78 0 3 equip_ELISA 17 0.35 0.49 0 1
Table A4-6: Test Accuracy (3)
8 (1) (2) (3) (4)
BCHE_ACC HEMA_ACC QASI_ACC HIVA_ACC EAPHLN 4.216 19.90 -26.07 -10.71
(20.87) (14.74) (19.50) (8.187) lab_room 0.107 -0.980 -0.765 0.899
(2.153) (1.552) (2.299) (0.783) staff_a0 0.991 -1.167 0.795 -0.598
(3.048) (2.183) (2.970) (1.113) staff_a1 0.474 5.622*** 1.414 0.482
(2.409) (1.744) (2.573) (0.887) staff_a2 -2.363 -2.339 1.046 -1.006
(3.241) (2.525) (3.512) (1.344) equip_BIO -2.474
(10.74) equip_HEMA -15.79* 19.96
(7.197) (10.72) equip_ELISA 1.800
(6.922) Constant 60.10* 76.11*** 22.68 91.94***
(30.88) (18.78) (28.01) (6.518)
Observations 17 17 14 17
R-squared 0.104 0.603 0.564 0.314 Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1
69
Figure A4-1
020
4060
8010
0
BCHE435 Hematology QASI432 HIVA435Non-EAPHLN EAPHLN Non-EAPHLN EAPHLN Non-EAPHLN EAPHLN Non-EAPHLN EAPHLN
Test Accuracy 2014
ACC UNACC NE
70
This report summarizes the main findings from the application of performance based incentives linked to progress on a standardized, globally recognized metric-- the Stepwise Laboratory Improvement Process towards Accreditation (SLIPTA) checklist-- under the East Africa Public Health Laboratory Networking Project (EAPHLNP) in Rwanda. The lab Performance-based Financing (PBF) pilot was introduced in the context of a well-established national PBF program dating back to the early 2000s. The flexible nature of the EAPHLNP and the favorable context in Rwanda provided an ideal backdrop to introduce PBF incentive payments to accelerate progress of five project-supported labs towards accreditation. The evaluation found improved laboratory performance at all project-supported laboratories in Rwanda as measured by the SLIPTA scores. For the first time, laboratories were bringing in PBF revenues, instilling a culture of continuous quality improvements, and focusing management attention on accreditation. PBF appears to have contributed to an accelerated change, with PBF laboratories experiencing an overall greater increase in SLIPTA scores compared to project-supported laboratories in the other countries. No clear patterns were found in terms of improved test volumes or test accuracy, which were not part of the pilot scheme. While it was difficult to disentangle the effects of different interventions, the evaluation found a system-strengthening value to combining investments in modernizing laboratories, and strengthening human resources with PBF. Relationships between laboratory staff and clinicians improved, with laboratory managers having a greater voice in hospital management and lab staff increasingly valued and respected by clinicians. A spirit of teamwork prevailed at participating sites.
ABOUT THIS SERIES: This series is produced by the Health, Nutrition, and Population Global Practice of the World Bank. The papers in this series aim to provide a vehicle for publishing preliminary results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For free copies of papers in this series please contact the individual author/s whose name appears on the paper. Enquiries about the series and submissions should be made directly to the Editor Martin Lutalo (mlutalo@ worldbank.org) or HNP Advisory Service ([email protected], tel 202 473-2256). For more information, see also www.worldbank.org/hnppublications.
1818 H Street, NW Washington, DC USA 20433 Telephone: 202 473 1000 Facsimile: 202 477 6391 Internet: www.worldbank.org E-mail: [email protected]
