Contact for this presentation:
Alexander K. Rowe, MD, MPH
Chief, Strategic and Applied Science Unit
Malaria Branch, Division of Parasitic Diseases and Malaria, Center for Global Health
Centers for Disease Control and Prevention
Mailstop A06
1600 Clifton Road
Atlanta, GA 30329
United States
Telephone: 1-404-718-4754
Fax: 1-404-718-4815
Email: [email protected]
Health Care Provider Performance Review
Webinar, December 6, 2018.
Presentation time: 40 minutes
Center for Global Health
Malaria Branch
Effectiveness of strategies to
improve health care provider
performance in low- and
middle-income countries:
a systematic review
Alexander K. Rowe, MD, MPH
Malaria Branch,
Division of Parasitic Disease and Malaria, Center for Global Health,
Centers for Disease Control and Prevention
Health Care Provider Performance Review
Co-investigators
• Samantha Y. Rowe (Malaria Branch, CDC)
• David H. Peters (Johns Hopkins
Bloomberg School of Public Health)
• Kathleen A. Holloway (World Health
Organization, International Institute
of Health Management Research,
and University of Sussex)
• John Chalker (Management Sciences for Health)
• Dennis Ross-Degnan (Harvard Medical School)
Data
abstraction
team
Acknowledgments
• Sushama Acharya
• Charity Akpala
• Dinorah Calles
• Tashana Carty
• Nirali Chakraborty
• Helen Chin
• Adrijana Corluka
• Didi Cross
• Bhavya Doshi
• Onnalee Gomez
• Meg Griffith
• Karen Herman
• Atsumi Hirose
• Simon Lewin
• Banafsheh Siadat
• Sanja Stanojevic
• Laura Steinhardt
• Savitha Subramanian
• Megan Thompson
• Anil Thota
• Ryan Wiegand
• Jeff Willis
• Kindra Willis
• Shannon Wood
• Karen Wosje
• Abera Wouhib
• Alicia Wright
• Chunying Xie
• Special thanks to investigators who responded to queries
• Funding: Bill and Melinda Gates Foundation, CDC, World Bank
• Qing Li
• Connie Liu
• Earl Long
• Jason McKnight
• Eliza McLeod
• Huseyin Naci
• Jan Odgaard-Jensen
• Dawn Osterholt
• Andy Oxman
• Magdalena Paczkowski
• Gabriel Ponce-de-Léon
• Nancy Pulsipher
• Atiq Rahman
• Monica Shah
Background
• Health care providers (HCPs) play essential
roles in delivering health care
• In low- and middle-income countries (LMICs),
however, HCP performance often inadequate
• Estimated 5 million deaths per year due to poor
quality among people using care
• Improving HCP performance is important for
programs and patients they serve, required for
Sustainable Development Goal of achieving
universal health coverage
• Many strategies exist to improve performance, and
summary of evidence would be useful
• Existing reviews have limitations, especially that they
typically focus on only a narrow range of strategies
• Decision-makers, however, ask broader question:
What are most effective ways to improve performance?
• To answer this broader question, one needs to compare
multiple strategies
• Health Care Provider Performance Review (HCPPR):
systematic review designed to help fill this gap by
comparing all strategies
Background
METHODS
Inclusion criteria
• Any quantitative study of effectiveness of any strategy to
improve HCP performance in LMIC, on any health topic,
in any language, published or not
• HCP. Any facility- or community-based health worker,
pharmacists, shopkeepers who sell drugs, private sector
• Literature search
− Included studies from 1960s to May 2016
− 52 electronic databases of published studies (eg, MEDLINE)
− 58 document inventories & websites for unpublished studies
Eligible study designs
• Pre-intervention vs. post-intervention study with
comparison (+/- randomization)
• Post-intervention only study with randomized controls
• Interrupted time series (>3 data points before and after
intervention)
• Determined which individual strategy components were
used (e.g., training + supervision = 2 components)
• 207 components identified
• Created 12 component categories (e.g., training,
supervision, incentives, etc.)
• Defined strategy as unique combination of 12
component categories, for example
– Training only
– Training + supervision
– Training + supervision + incentives
– Etc.
Defining strategy groups
1) Community support: E.g., community health education
2) Patient support: E.g., patient education
3) Strengthening infrastructure: E.g., provision of drugs
4) HCP-directed financial incentives
5) Health system financing and other incentives. E.g., insurance
6) Regulation and governance: E.g., accreditation
7) Group problem solving: E.g., collaborative improvement
8) Supervision: E.g., improving routine supervision, audit with feedback
9) Other management techniques: E.g., HCP self-assessment
10) Training
11) Printed information or job aids for HCPs
12) Information & communication technology (ICT) for HCPs:
E.g., reminders sent to HCP phone
Defining strategies: 12 component categories
Effect size = (FU – BL)intervention – (FU – BL)control
• Effect size in terms of %-point change
• Example formula for outcomes expressed as %:
0
20
40
60
Baseline Follow-up
20%
25% 30%
50%
Control
Change = (50% – 25%) = 25 %-pointsIntervention
Change = (30% – 20%) = 10 %-points
Effect size = 25 – 10
= 15 %-points
% of patients correctly treated
For every 100 patients, 15 treated correctly
Analysis of effect sizes
• Primary method
– Only include strategy vs. control comparisons
(no head-to-head studies)
– If study had >1 primary outcome (thus >1 effect size),
study represented by median of effect sizes (MES)
– Compare MES distributions of various strategies:
weighted medians, IQRs (weight = 1 + ln[no. of HCPs or HFs])
– To reduce bias, effect sizes of outcomes expressed as
percentage from studies of professional HCPs were
adjusted for baseline performance, public HF only, & Asia
Analysis
RESULTS
• 216,477 citations screened
• 2269 reports included in review (all outcomes)
• For HCP practice outcomes (focus of presentation,
e.g., % of patients correctly diagnosed or treated)
– Included 670 reports from 337 studies
– Identified 118 strategies
• Wide range of contexts
– Urban and rural
– Public & private health facilities, community settings
– Numerous health conditions
Literature search
• 64 countries
• 40% from low-income countries
Study sites (337 studies with HCP practice outcomes)
Africa
Americas
Eastern MediterraneanEurope
Southeast Asia
Western Pacific 42%
20%
16%
13%
7%3%
Overall risk of bias
Based on guidance from Cochrane’s EPOC group
Low:
16%
Moderate:
25%
High:
29%
Very high:
30%
Study follow-up time, in months
114
72
3538
18 17
4 6 6 41 1 1
0
20
40
60
80
100
120
No
. o
f stu
die
s
Studies often short, relative to what
most programs would consider
sustained effect
Two-thirds had follow-up times
<10 months
Effectiveness of strategies
to improve HCP practices:
Studies of professional
health workers
Results of outcomes expressed as %
(e.g., % of patients treated correctly)
(generally facility-based health
workers, such as physicians,
nurses, and midwives)
Top photo: TF Org. https://www.tforg.com/how-we-think/sweetspot-blog/2015/02/27/opportunities-
indonesian-healthcare-system-summary-recent-publications/. Accessed November 10, 2018.
General findings
• Mean baseline: 40%
• Among all 101 strategies, median improvement = 12 %-pts
(Typical scenario: 40% BL + 12 %-pt improvement = 52% F/U)
• Most strategies (80%) tested by only 1 or 2 studies
- Generalizability extremely limited
- Presentation focuses on strategies tested by at least 3 studies
• Effect sizes vary widely for most strategies
- Ex. Train only, median effect: 10 %-pts (IQR: 6, 21; range: –20, 61)
Thus, ¼ of effects: <6 %-pts, and ¼ of effects: 21 to 61 %-pts
- Demonstrates difficulty in predicting strategy’s effect
- Underscores importance of monitoring effect of any strategy
Effectiveness of strategies tested by 3+ studies
• Printed information or job aids for HCPs only
• ICT for HCPs as sole strategy (N = 4 studies)
1
1
Median effect
size, %-pts
Information and
communication
technology
(mHealth)
Effectiveness of strategies tested by 3+ studies
• Printed information or job aids for HCPs only
• ICT for HCPs as sole strategy (N = 4 studies)
- Broadened strategy definition (ICT +/- other
strategy components, N = 28 studies)
1
1
Median effect
size, %-pts
8
Goal: analyze larger pool of studies with greater diversity
of context and implementation approaches
Effectiveness of strategies tested by 3+ studies
• Printed information or job aids for HCPs only
• ICT for HCPs as sole strategy (N = 4 studies)
- Broadened strategy definition (ICT +/- other
strategy components, N = 28 studies)
• Training only
• Supervision only
• Training + supervision
1
1
Median effect
size, %-pts
8
10
15
18
28 (12)
Median effect size,
%-pts (broadened
definition)
• Group problem solving only
Effectiveness of strategies tested by 3+ studies
E.g., CQI or collaborative improvement
28 (12)
Median effect size,
%-pts (broadened
definition)
56 (16)
33 (29)
• Group problem solving only
• Group problem solving + training
• Strengthened infrastructure + supervision +
other mgmt techniques + training
Effectiveness of strategies tested by 3+ studies
E.g., Provision of medicines
E.g., HCP group process/meetings
28 (12)
58 (33)
Median effect size,
%-pts (broadened
definition)
56 (16)
33 (29)
• Group problem solving only
• Group problem solving + training
• Strengthened infrastructure + supervision +
other mgmt techniques + training
• Strengthened infrastructure + supervision +
other mgmt techniques + training + financing
Are multi-faceted strategies more effective than simpler ones?
Effectiveness of strategies tested by 3+ studies
Are multi-faceted strategies more effective
than simpler ones?
Number of strategy components
Effe
ct s
ize
(p
erc
en
tage
po
ints
)
Are multi-faceted strategies more effective
than simpler ones?
Number of strategy components
Effe
ct s
ize
(p
erc
en
tage
po
ints
)
Strategies tested by <3 studies (“hot topics”)
• Financial incentives for HCPs only
• Health system financing or other
incentives only (i.e., not financial
incentives for HCPs)
• Regulation/governance only
26 (7)
1 (14)
Median effect size,
%-pts (broadened
definition)
NA (28)
No eligible studies found of
regulation/governance as sole strategy
Context-specific analysis
Stratify effectiveness results:
low versus moderate level of
resources (what works where?)
Low- vs. moderate-resource setting
Strategy (with 3+
comparisons per
stratum and >10 %-pt
difference between
strata)
All settings
(median
MES)
Low
resource*
(median MES)
Moderate
resource**
(median MES)
Group problem
solving only28 12 40
Supervision +
training18 12 25
* Non-hospital settings in low-income countries and rural-only settings
in middle-income countries
** Hospitals in low-income countries and any urban & mixed urban/rural
settings in middle-income countries
Low- vs. moderate-resource setting
Strategy (with 3+
comparisons per
stratum and >10 %-pt
difference between
strata)
All settings
(median
MES)
Low
resource*
(median MES)
Moderate
resource**
(median MES)
Group problem
solving only28 12 40
Supervision +
training18 12 25
Not all strategies have large stratum-specific
differences for “low vs. moderate resource” factor,
and some strategies have large stratum-specific
differences for other contextual factors.
Factors associated with
training effectiveness
(Are some training
approaches more effective
or more efficient?)
Factors associated with training effectiveness
• Interaction between training duration and complexity
of training topic: additional days increase mean effect
but only for multiple health topics
For single-topic
training, no benefit to
having longer courses
Ex: Integrated
Management of
Childhood Illness
(IMCI)
0
10
20
30
40
0 1 2 3 4 5 6 7 8 9 10
Days of training
Effect siz
e (
%-p
oin
ts)
Single topic (p=.04)
Multiple topics (p=.01)
All < 6 days
• Mean effect 6–11 %-pts higher if some or all training is
on-site (compared with all off-site training)
• Time since training: effect of training wanes over time
• Interaction between supervision and time since training:
supervision “protects” against waning effect of training
Waning
effect
0
10
20
30
40
0 5 10 15 20
Months since training
Effe
ct siz
e (
%-p
oin
ts)
No
supervision*
Supervision present
(p=0.8)
* p=<.0001
Factors associated with training effectiveness
Effect of strategies to
improve performance
of lay or community
health workers (CHWs)
Top image. Malaria Consortium. https://www.malariaconsortium.org/blog/recognising-community-
health-workers-this-world-health-day-and-world-health-workers-week-2/. Accessed May 16, 2018
Lower image. World Vision. https://www.worldvision.org/health-news-stories/malaria-burundi-half-
country-sick. Accessed May 16, 2018.
Results of outcomes
expressed as percentage
(e.g., % of patients treated
correctly)
• 18 studies, most with high or very high risk of bias
• 14 strategies, most tested by 1 or 2 studies each
• For training only (N = 4 studies), median effect =
2 %-points
• For strategies that included community support
and training CHWs, effects ranged from 8 to 56
%-points
Improving lay or CHW performance
Evidence-based guidance
on improving HCP
performance in LMICs
1) Effect of any strategy should be monitored so managers
can know how well it works. Monitoring data could be
used to adapt strategies to local conditions and facilitate
learning, with aim of increasing effectiveness.
2) General approach
• Initial strategy (based on research evidence and
knowledge of local context)
• Monitor HCP practices
• Address gaps (which should be expected) by modifying
or abandoning strategy or layering on new one
• Continue to monitor and modify as needed
3) Decision-makers should not assume multi-faceted
strategies are more effective than simpler ones
General guidance on improving HCP practices
Guidance for professional HCPs (i.e., not only CHWs)
1) Printed information or job aids to HCPs as sole strategy
is unlikely to change performance
2) ICT typically has small-to-modest effects
3) Training or supervision generally have moderate effects.
May be more effective to combine training with other
strategies, such as supervision or group problem solving.
• To increase effect of training on multiple health topics,
duration at least 3 days might be beneficial, with additional
days potentially increasing effectiveness
• For training on single health topics, short duration (1–2 days)
seems as effective as longer duration (and less expensive)
4) Group problem solving typically has moderate effects
5) Multifaceted strategies of infrastructure, supervision,
management techniques, and training (+/- financing), and
strategy of group problem solving + training tend to have
large effects
6) Financial incentives for HCPs, & other finance/incentive
strategies typically have modest–moderate effects
7) Effect of regulation/governance alone is unknown; it tends
to have large effects when combined with other components
8) Programs might consider influence of context on strategy
effect. Some (e.g., group problem solving) might be more
effective in moderate-resource areas.
Guidance for professional HCPs (i.e., not only CHWs)
Guidance for improving CHW performance
1) Only training CHWs usually has small effects
2) Strategies that include community support plus training
for CHWs might lead to large improvements, although
evidence is limited
1) Limitations of studies: lack of detail on strategy and
context, lack of standard methods, difficulty in assessing
study precision and strength of implementation, high risk
of bias, and short follow-up time
2) With many statistical tests performed, results represent
hypothesis screening, not true hypothesis testing
3) Overview analysis—i.e., intentionally designed to
identify broad patterns across all studies. Thus, results
do not reflect nuances, e.g., all countries combined.
Future analyses will be more specific.
Limitations
HCPPR website:
www.hcpperformancereview.org
HCPPR website:
www.hcpperformancereview.org
HCPPR website:
www.hcpperformancereview.org
HCPPR website:
www.hcpperformancereview.org
1) Use menus to select studies
2) Click on “Run analysis”
HCPPR website:
www.hcpperformancereview.org
1) Use menus to select studies
2) Click on “Run analysis”
HCPPR website:
www.hcpperformancereview.org
1) Important performance problems exist, but there are
strategies to improve quality of care
2) Research has some important limitations, but results still
useful to inform decision-making
3) Some strategies seem more effective than others
(e.g., training + group problem solving, some multi-faceted
strategies); consider using in appropriate context
4) Might be ways to make training more effective and efficient
5) Avoid ineffective strategies (e.g., only printed info)
6) Important to monitor effectiveness for all strategies
7) High-quality research needed (e.g., on CHWs)
Conclusions
8) HCPPR is largest review of strategies to improve
HCP performance in LMICs
− Programs, donors, and other development partners
consider results when making decisions
− To help disseminate results and encourage more
specific analyses, the database is publicly available
on website
Conclusions
Lancet Global Health, October 2018
(appendices: additional methods and results)
Visit our website:
www.hcpperformancereview.org
Center for Global Health
Malaria Branch
For more information please contact Centers for Disease Control and Prevention
1600 Clifton Road NE, Atlanta, GA 30333
Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348
E-mail: [email protected] Web: http://www.cdc.gov
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Thank you!
Any questions
or comments?
www.hcpperformancereview.org