Date post: | 11-Feb-2017 |
Category: |
Healthcare |
Upload: | rbfhealth |
View: | 448 times |
Download: | 1 times |
Verification of Results
Findings and Recommendations
from a Cross-Case Analysis
Petra Vergeer, Tawab Hashemi, Martin Sabignoso, Olivier Basenya,
Catherine Mugeni, Eubert Vushoma, Chenjerai Sisimayi
• Relatively new function • Verification is the first-order substantiation of results paid for in RBF, including
coverage rates or quantities of patients seen, quality of services provided, and patient satisfaction.
• Counter-verification is the second order substantiation of the above, i.e., it requires that a first order verification has been carried out and is verifying the accuracy of it.
• Donors and government acutely sensitive to potential for
“over-payments” for inflated service reporting
• Avoid appearance of, or actual conflict of interest: contracted party has incentive to over report; separate actor must verify reporting
Verification: Essential element of
implementation
Not to be confused with M&E!!
• Process for ensuring the consistency of routine reporting on the volume (i.e., quantity) of purchased services provided (recount of data)
• Process for confirming with patients the provision of purchased services (patient tracing)
• Direct observation of conditions of service delivery and actual care to assess quality
• Assessment of satisfaction of patients
• RBF mechanisms often include multiple approaches
Different methods used
Objectives of Cross-Case Analysis • Expand knowledge about verification processes and
practices to address the design and implementation needs of RBF projects.
• Add to available knowledge by comparing the characteristics of verification strategies as well as available data on costs (using level of effort as a proxy), savings, and verification results to date in six countries: Afghanistan, Argentina, Burundi, Panama, Rwanda, and the UK.
• Country cases written with a common outline to describe major characteristics of the verification method, the verification results, the use of the verification results, costs, and key lessons and recommendations.
Overview of Verification Structures
and Interactive Quiz
Overview of Verification Structures
Afghanistan, Argentina, Burundi, Rwanda (community RBF)
Afghanistan: Verification structure There are four types of verification activities:
• Quantity of services verified in facilities, conducted
quarterly, ex-ante, by a third party, of 25% of providers;
• Patient tracing, conducted quarterly, ex-ante, by a third
party, of 25% of providers;
• Quality of services assessed by the Provincial Health Office
(PHO) jointly with the NGO
• Counter-verification of quality through health facility
assessment by a third party, sample basis.
There are three types of verification activities:
• Beneficiary enrollment verification, conducted monthly, ex-ante, internally at national level, with electronic data validation all records, no field visits;
• Beneficiary enrollment counter-verification, conducted every two months, ex-post, by a third party, with electronic data validation all records, and a sample is checked to ensure existence of enrollment form;
• Tracer indicator verification, conducted every four months, ex-post, third party, data validation all records, risk-based sample of health facilities (primarily facilities with higher numbers of patients).
Argentina: Verification structure
Burundi: Verification structure There are 4 types of verification activities in Burundi:
• Quantity verification, conducted monthly, ex-ante, jointly by verifiers from the MOH and civil society organizations (public –private partnership), at all sites;
• Technical quality verification, conducted quarterly, ex-ante, internally with civil society engagement, at all sites;
• Patient tracing (including patient satisfaction) as part of the quality score, conducted bi-annually, ex-ante, by a third party, each facility, sample basis; (n.b. for quarters with no patient tracing,
the previous quarter’s score is used in the calculation)
• Quantity, quality and patient tracing counter-verification, conducted quarterly, ex-post, by third party, sample basis.
Rwanda: Verification structure for
community RBF There are four types of verification activities:
• Verification by the health facility of the quantity of referrals
based on information in submitted reports, monthly, ex-
ante, internally, all indicators, all CHWs. The number of
referrals is cross-checked against health center records.
• Quarterly counter-verification, by a sector steering
committee (mostly comprised of health center staff - but
with some community members, considered independent).
Rwanda: Community RBF
verification structure (cont.) • All cooperatives’ reports are assessed each quarter for data
completeness and report submission timeliness, internal.
The evaluation of cooperative management is carried out by
the district hospital. Each quarter, 100% of cooperatives are
evaluated.
• Verification of the demand-side scheme is not systematic
and is integrated into the monitoring of the health center by
the district hospital.
Interactive Quiz
How long does it take to observe improvements in
RBF data due to quantity verification?
1. One (1) year
2. Two (2) years
3. Five (5) years
Findings: Level of agreement between HMIS and
facility data increases over time in Afghanistan
• Structure o Verification of the quantity of services in facilities is conducted quarterly, ex-ante, by a
third party. 25% of providers are sampled each quarter.
• Findings o Error rates in quantity verification declined from 17% to 8% between 2010 and 2013
% a
gre
em
en
t
83 83 83 83
86 87
89
93
89
95 94
91 92
76
78
80
82
84
86
88
90
92
94
96
Figure: Trends in Level of Agreement between HMIS and Facility-Level
Verification Data for the Quantity of Services Delivered
Findings: Error rates in beneficiary enrollment
decline over time in Argentina • Structure
Beneficiary enrollment:
o Verification through electronic data validation of all records (no field visits) monthly, ex-ante,
internally at the national level.
o Counter-verification, every two months, ex-post, third party, electronic data validation all
records, sample checked to ensure existence of enrollment form;
• Findings o Error rates in beneficiary enrollment declined from 20% to less than 1% in 2 years
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
2004 2005 2006 2007 2008 2009 2010 2011 2012
% R
ecord
s R
eje
cte
d
Phase 1 Provinces Phase 2 Provinces
Figure: Counter-verification of
beneficiary enrollment compared to
records submitted by provinces in
Argentina, 2004-2012
Findings: Counter-verification shows <10% errors
in Burundi after 2 years
• Structure: o Quantity verification, conducted monthly, ex-ante, jointly by verifiers from the MOH
and civil society organizations (public –private partnership), at all sites;
o Quantity counter-verification, conducted quarterly, ex-post, by a third party, sample
basis.
• Findings: o Internal verification found that 31% of declarations for health centers were reported
with error and 38% for hospitals. The average error size was 5% for health centers
and 4% for hospitals.
o Difference between verification and third party counter-verification of quantity after 2
years of nationwide PBF implementation were small (<1%) for health centers.
Differences for hospitals were substantially larger (9%) but can be explained in part
by a lack of standardized registers among hospitals.
Findings: Decline in size of indicator over-reporting
in Rwanda
• Structure
o Verification by the health facility of quantity of referrals based on
information in submitted reports is done on a monthly basis, ex-ante, by an
internal verifier. All indicators and all CHWs are verified. The number of
referrals is cross-checked against health center records.
• Findings
o The percentage of service indicators that contained error did not change
dramatically, with 49% of indicator reports inaccurate in Q4 2012 (23%
over-reported and 26% under-reported ). However, the size of the error for
over-reporting declined substantially (from over 140% to around 7%) in two
year’s time.
o This refers to the discrepancies between the performance as self-assessed
by CHWs at cell level and the performance after the verification process is
complete.
After 2 years of implementation, what was the average
percentage of traced patients who could not be found?
1. 20%-30% of all patients traced could not be found
2. 10% of patients traced from health centers and 15% of patients traced from hospitals could not be found
3. <10% of patients traced from health centers and 15% of patients traced from hospitals could not be found
Findings: Percentage of missing patients reduced
over time in Afghanistan
• Structure o Patient tracing is conducted quarterly, ex-ante, by a third party. 25% of providers are
sampled.
• Findings o “Missing patients” reduced from 33% to 7% between 2010 and 2013
% a
gre
em
en
t
67 77
83 86 89 89 91 92 94 95 96 93 93
0
20
40
60
80
100
120
Trends in Level of Agreement between HMIS and Community -Level Verification
Data for the Quantity of Services Delivered
Findings: Most patients traced from health centers
but more difficult in hospitals in Burundi • Findings
o In Q1 2012, 7.4% of patients traced from health
centres (see below graph) and 15.4% of
patients from hospitals were not found.
o More than 98% of those found (both for health
centers and hospitals) confirmed receiving the
services recorded.
o Counter-verification of patient tracing is
performed but with a newly-taken sample of
patients and hence no comparison can be done
between verification and counter-verification.
• Structure
o Patient tracing (including patient satisfaction),
conducted bi-annually, ex-ante, third party, each
facility, sample basis;
o Counter-verification of patient tracing counter-
verification, conducted quarterly, ex-post, third
party, sample basis.
Figure: Health centre patient tracing results 2011-2012
Findings: National study in Rwanda identifies most
patients in community
• Structure:
• A national study, which is to be distinguished from regular patient tracing as in
Burundi and Afghanistan, was conducted by the MoH in 2012 and included,
among other things, patient tracing in the community
• Findings:
• 97% of the patients could be identified in the community.
• Of those found, 97% confirmed having been treated at the facility for the
services for which the CHW referred them.
• In addition, 98% of eligible women confirmed to have received in-kind
incentives.
What are the links between patient confidentiality
and verification?
1. There are no concerns about patient confidentiality in
verification
2. Concerns about protecting patient confidentiality result in
the exclusion of certain indicators from verification (e.g.,
family planning)
3. Electronic checks of records helps to protect patient
confidentiality because patient data are de-identified
4. 2 and 3 are correct
Findings: Indicators excluded because of patient
confidentiality concerns in Burundi
• Findings – Burundi o Only 9 of 22 indicators at the health center and 8 out of 24 for hospitals are verified as
part of patient tracing for confidentiality reasons. Indicators on HIV, tuberculosis, and
family planning are excluded.
o As a result, the existence of “phantom patients” for almost half of the health centre
indicators, and one third of hospital indicators, is never assessed.
What type of indicators have high error rates?
1. Indicators with high patient volume
2. Indicators with complex definitions
3. Indicators with registration difficulties
4. Indicators with a high incentive attached
5. 1 and 2 are correct
6. 1, 2, and 3 are correct
Findings: Indicators with a high rate of
occurrence and with complex definitions
have higher error rates in Burundi
• Findings o Indicators with a high rate of
occurrence have the highest level of
error. These are indicators where
the risk of errors when counting may
be greater.
o Indicators with definitions that are
complicated also have higher errors.
Facilities have greater chances of
counting a service that does not
match the definition and that will
not be validated by the verification
team
o Indicators with high incentives do
not have high error rates
10 Indicators with highest error rates identified during
verification in health centers
(January-August 2012)
Indicator % accuracy of
declared data
n
Consultation (child) 22% 4,085
Consultation (adult) 30% 4,085
Observation day (child) 44% 3,802
Consultation (pregnant woman) 44% 4,062
Small surgery 53% 3,987
Family planning 58% 3,458
Completely vaccinated child 67% 4,065
Anti-tetanus vaccination (TT2-TT5) 68% 4,055
Observation day (adult) 69% 2,896
Prenatal consultation 67% 4,074
Findings: Indicators with a high rate of occurrence
have higher error rates in Rwanda • Findings
o Malnutrition monitoring had the highest error rate in Rwanda which may have been due to
the large number of children involved.
Table: Percentage of inaccurate reports detected by the health centers and by the sector steering committees
in the 4 sectors visited for 8 paid indicators, during Q4 2010, Q4 2011 and Q4 2012
Indicator
Errors detected by the health center (comparison between cell and sector reports)
Errors detected by the sector steering
committee (comparison between sector reports and national db)
% inaccurate indicators Nb. of reports % inaccurate
indicators Nb. of reports
Woman accompanied for delivery 51% 35 14% 35
Woman accompanied for antenatal care 43% 35 14% 35
Patients accompanied for VCT 49% 35 20% 35
Children monitored for nutrition status 59% 34 29% 34
Family planning users referred 23% 35 23% 35
TB-cases followed per month 23% 35 37% 35
TB suspects referred 37% 35 41% 34
Women referred for PMTCT 52% 33 36% 33
Total 42% 277 27% 276
Findings: Higher error rates are associated
with registration difficulties in Argentina • Structure
o Tracer indicator verification is
conducted every four months, ex-post
by a third party. Data validation is
conducted for all records, and a risk-
based sample of health facilities is
selected (primarily facilities with
higher numbers of patients).
• Finding o Higher error rates are associated with:
• Lack of registration tools (tracer
VII)
• Weak adherence to registration
norms (tracer IX)
• More than one source of data
needs to be utilized (tracer IV).
Tracer Percentage of results with error
rates >20% of declared
Percentage of results with error
rates >40% of declared
I Early detection of pregnant
w omen 23% 3%
II Effectiveness of childbirth and
neonatal care 6% 1%
III Effectiveness of prenatal care and
prevention of prematurity 7% 1%
IV Effectiveness of prenatal and
delivery care 19% 5%
V Case assessments in child and
maternal deaths out of all child
and maternal deaths 5% 3%
VI Immunization coverage 10% 1%
VII Sexual and reproductive care 18% 6%
VIII Tracking healthy child up to 1 year 11% 3%
IX Tracking healthy children betw een
1 and 6 years 24% 7%
X Inclusion of the indigenous
population 6% 2%
Table: Error rates >20% and >40% by indicator 2008-2012
identified counter verification
Why is there a large difference between
verification and counter-verification of quality?
1. Time delay between verification and counter-verification
2. Potential conflict of interest between those assessing
quality and those contracted to provide services
3. Objectivity of measuring tool is compromised
4. Sanctions for discrepancies between verification and
counter-verification are not applied
5. All of the above are correct
Findings: Afghanistan quality verification
• National Monitoring Checklist (NMC) is used for quality verification.
o Interviewees understand payment is linked to quality.
o However, they are not necessarily clear about which specific indicators are linked to quality
payment (e.g., clinic infrastructure, facility health information system HMIS data and
essential drugs from the NMC all are part of the indicators making up the payment for
quality.
• NGO and MoPH – SM supervisors fill NMC checklist as part of their routine supervision visit to
health facilities.
o However, PPHOs often do not join the health facility visits which can lead to conflict of
interest.
• BSC is used for verification at the provincial level hospitals.
o Due to delays in the implementation of the BSC, bonus payments to hospitals were also
delayed.
• BSC was also intended as a way of triangulating the NMC results at provincial level.
o Due to delays in the implementation of the BSC, this has not been operationalized.
Findings: Burundi quality verification
• Systematic difference between quality verification and counter-verification (79% of health centers and 84% of hospitals)
• Technical quality was overestimated (by 11% in health centres and by 17% in hospitals).
• Overestimation can be explained by three factors: o Time lag between verification and counter-verification
o Counter-verification team is more rigorous
o Possible conflict of interest as peer hospitals review other hospitals and provincial health teams verify their own health facilities. Sanctions were not applied for discrepancies found during counter-verification
Average
difference
% with
over-
estimation
Average
over-
estimation
% with
under-
estimation
Average
under-
estimation n
Health
centres -11% 79% -20% 21% 24% 101
Hospitals -17% 84% -24% 16% 20% 32
Table. Difference between technical quality assessment performed by the BPS, BDS or peers and counter-verification by HDP in all
hospitals and health centres counter-verified during the 8 counter-verification rounds, 2010-2012
Key Recommendations 1. Consider context to determine whether merging functions
is appropriate (be mindful of conflict of interest)
2. Analyze and use data available from verification and
counter-verification
3. Verification strategies should be dynamic, not static, and
use a risk-based approach
Factors influencing verification: a
conceptual framework Context
Verification Characteristics Impact on accuracy,
cost, sustainability
RBF Characteristics RATIONALE FOR RBF CONTRACT TYPE
USE OF RBF RESULTS
Improving health outcomes/HSS Relational
Payment, improving performance
Financial accountability/Cost control Classic
Transparency, Naming and Shaming
Monthly Annual
Yes
Large
Whole universe Risk-based approach Internal
Verification Results and Their Use
FREQUENCY
ALLOWABLE ERROR MARGIN
SAMPLE SIZE INSTITUTIONAL SETUP ADVANCE WARNING No
Small
Third party
Learning, Error correction Cost recovery, Sanction
PAYMENT FREQUENCY Monthly Annual
POLITICAL ENVIRONMENT GOVERNANCE CULTURE
When and how to change your verification
strategy…examine quantity verification error rates
1
Facility-level patterns
Indicator-level patterns
Generalized
across all
contracted
parties
Localized to specific
facilities (by
geographic area)
Localized to
specific types
of facilities
Indicators with
complex
compliance
criteria
Indicators that are
rewarded more
frequently and/or
have higher patient
volume
Indicators
rewarded at a
higher level
Examine quantity
verification error rates
Activities to
explore
Activities to
explore
Risk-based sampling for verification • Using a risk-based sampling approach likely more cost-
effective
• Sample contracted parties (e.g., facilities) with selection
criteria, such as: o Higher volume (like in Argentina)
o Outliers in performance relative to province- or national- averages (as in the UK)
• Sample indicators with selection criteria such as: o Higher volume (possibly more prone to error as in Burundi and Rwanda example)
o More complex (possibly more prone to error as in Argentina example)
o Higher $ value
• Always ensure a credible threat of verification remains for
all contracted parties
-15
-10
-5
0
5
10
15
HF1 HF2 HF3 HF4 HF5 HF6
Difference Between Declared and Verified 6 Month Totals
Within 5% Difference
Risk-based verification: Zimbabwe model
Green Category:
• Verified on a quarterly
basis
Amber Category
• Verified bi-monthly -
randomly selected 2
months
Red Category
• Verified on a monthly basis
• Also incorporates new
facilities
Difference above 5%
but below or equal to
10% Difference above 10%
• Model based on three risk levels
• Comparison between declared
and verified values for 6-month
totals
Example at District level
Health
Facility
Total
Declared
Total
Verified
%
Difference
Hoyuyu
1 1011 1016 0%
Matedza 344 325 6%
Kawazva 327 417 -28%
Sample Facilities (Mutoko District)
• Districts have a mix of facilities at different risk levels
Total facilities by risk-category
85
63
244
0
50
100
150
200
250
300
Red Amber Green
Num
ber
of F
acili
ties
Category
Areas for further research in verification
• Costs, savings, and cost-effectiveness of
verification and counter-verification
• Application of technology for verification and
potential for cost savings
• Ensuring patient confidentiality is protected
• Patient tracing
• Measuring quality
Conclusions • The conceptual framework and is intended to assist RBF
implementers and policymakers in their deliberations about
the consequences of various verification characteristics on
the accuracy, cost, and sustainability of a chosen approach.
• Verification strategies are not static but should be a
dynamic process. The pathway tool can provide guidance
for how to adapt verification strategies.
• While there is no optimal verification method appropriate to
all settings, the recommendations provided can be useful to
consider in different contexts.
References • Cashin, Cheryl and Lisa Fleisher. Verification of performance in results-based financing:
the case of Afghanistan. World Bank: Washington, DC. Forthcoming.
• Cashin, Cheryl and Petra Vergeer. (2013). Verification in results-based financing: the case
of the United Kingdom. World Bank: Washington, DC.
https://openknowledge.worldbank.org/handle/10986/13567
• Perazzo, Alfredo. Verification of performance in results-based financing: the case of
Panama. World Bank: Washington, DC. Forthcoming.
• Perazzo, Alfredo and Erik Josephson. Verification of performance in results-based
financing: the case of Argentina. World Bank: Washington, DC. Forthcoming.
• Renaud, Adrien. Verification of performance in results- based financing: the case of
Burundi. World Bank: Washington, DC. Forthcoming.
• Renaud, Adrien and Jean-Paul Semasaka. Verification of performance in results-based
financing: the case of the Rwanda community RBF interventions: community PBF and
demand side scheme. World Bank: Washington, DC. Forthcoming.
• Vergeer, Petra, Anna Heard, Erik Josephson, and Lisa Fleisher. Verification in results-
based financing for health: findings and recommendations from a cross-case analysis.
World Bank: Washington, DC. Forthcoming.
Thank You