34
W. Scott Gordon PATH March 2, 2011 Assessing the Performance of Routine Immunization Systems in Bihar, India with a Congruence Model: Findings and Reflections
W. Scott Gordon
PATH
March 2, 2011
Assessing the Performance of Routine Immunization Systems in Bihar, India with a Congruence Model: Findings and Reflections
Overview of Presentation
• Introduction to PATH
• Context of research – changing approaches to immunization in India and Bihar
• Project objectives
• Research design
• Findings
• Discussion and reflections on model and approach
• Conclusions
Page 2
Advancing technologies
Strengthening systems
Encouraging healthy behaviors
Improving the health of people around the world by:
PATH’s mission
Page 3
PATH’s Global Presence
Page 4
PATH’s Areas of Work
Vaccines and
immunization
Emerging
and
epidemic
diseases
Health
technologies
Reproductive
health
Maternal
and child
health and
nutrition
Page 5
Bihar: A Challenge
Uttarakhand
Maharashtra
Karnataka
Lakshadw eep
Goa
KeralaTamil Nadu
Puducherry
Andhra Pradesh
Dadra & Nagar Haveli
Rajasthan
Gujarat
Daman & Diu
Jammu & Kashmir
Uttar Pradesh
Madhya Pradesh
Haryana
Punjab
Delhi
Himachal Pradesh
Chandigarh
Jharkhand
OrissaChhattisgarh
Andaman & Nicobar Islands
MeghalayaBiharAssam
Sikkim
Manipur
TripuraMizoram
Arunachal Pradesh
Nagaland
West Bengal
• State in North India, bordering Nepal
• Population 93 million (9% of total population)
•Poor infrastructure and annual floods add to service delivery challenge
•Poor roads and power supply
•Frequent polio SIAs 15 days / month
Study Context: Bihar India
RAJASTHAN
OR ISSA
GU JAR AT
MAHARASH TRA
MADHYA PR AD ESH
BIH AR
KARNATAKA
UTTAR PRADESH
AND HRA PRADESH
JAM MU & KASHMIR
ASSAM
TAM IL NADU
CHHATTISGARH
PUN JAB
JH AR KH AN D
W EST BENGAL
ARU NACHAL PR .
HAR YANA
KERALA
UTTARANCHAL
HIM AC HAL PRADESH
MANIPUR
MIZOR AM
MEGHALAYANAGAL AND
SIKKIM
GOA
A&N ISL AN DS
D&N HAVEL I
POND ICH ER RY
LAKSHADW EEP
RAJASTHAN
OR ISSA
GU JAR AT
MAHARASH TRA
MADHYA PR AD ESH
BIH AR
KARNATAKA
UTTAR PRADESH
AND HRA PRADESH
JAM MU & KASHMIR
ASSAM
TAM IL NADU
CHHATTISGARH
PUN JAB
JH AR KH AN D
W EST BENGAL
ARU NACHAL PR .
HAR YANA
KERALA
UTTARANCHAL
HIM AC HAL PRADESH
MANIPUR
MIZOR AM
MEGHALAYANAGAL AND
SIKKIM
GOA
A&N ISL AN DS
D&N HAVEL I
POND ICH ER RY
LAKSHADW EEP
Fully Immunized Children in India
DLHS-2 (2002-04) DLHS-3 (2007-08)
Bihar: 24.4 % Bihar: 41.4 %
Source: http://www.mohfw.nic.in/dlhs/dlhs08_release_1.htm
India
DLHS-2Data not available
14 to 40
40 to 55
55 to 75
75 - 85
85 - 93.4
India: 48% India: 54%
Muskan (In English…Smile)
• Enhanced political commitment through change in State administration
• Target children 0-23 months: ~ 4.7 million
• Augmentation of immunization efforts started in 2005 through special immunization drives
• Expansion of financial resources and policy flexibility through National Rural Health Mission
• Formalized as ‘Muskan’ in Oct-2007
Inter-Sectoral
Coordination
Identification
&
Tracking of
beneficiaries
Review of
Microplan
Performance based
incentives for service
providers
Involvement of village
level Mahila
Mandal *Muskan Strategy
Page 8
Routine Immunization System in Bihar
• Sub-district level microplanning• Expansion of workforce through
contracted nurses• Expanded immunization sessions
(Wed and Sat)- Village Health Day• Incorporation of supportive
supervision• Courier–based vaccine delivery
system • Contracted cold chain management• Expanded focus on recording and
reporting (RIMS, HIMS)• Performance based incentives for
nurses and community health workers
Page 9
Project Objectives
• To capture and depict the strengths and weaknesses of the immunization system in Bihar State
• To show the alignment among different system components (formal and informal structures, resources, processes and knowledge and skills) and their impact on the system performance
• To provide the basis for recommendations to improve the immunization system and its performance
Page 10
Congruence & PRISM Models
Alignment Model for Bihar
Formal
Organizational
Determinants
Informal
Organizational
Determinants
Technical
Determinants
Individual and
Behavioral
Determinants
PerformanceAlignment
Inputs
· Resources
· Environment
· History
Page 12
Key Activity and Performance Domains Assessed
• Management and decision making
• Planning (including the development and use of microplans)
• Service provision - measured across three performance domains
• Coverage of services or activities
• Quality of services
• Safety of services (both injection safety and safe disposal)
• Reporting and record keeping
• Support and supervision (including training)
• Logistics and cold chain
• Community mobilization
Page 13
Study Tools and Methods
Observation
Record Review
Participant Surveys
Performance and
Process Assessment
Tool (PPAT)ASHA
AWW
ANM
LHV/MO
MOIC & DIO
Availability of
Resource Assessment
Tool (ARAT)Health Facilities
Immunization
Management
Assessment Tool
(IMAT)Health Facilities
Organizational and
Behavioral
Assessment Tool
(OBAT)All Health Workers
Tools for Immunization Systems Performance Assessment
(TISPA)
Page 14
Study Area and Participants
GAYA
PATNA
JAMUI
KAIMURBANKA
ROHTAS
PURNIASARAN
ARARIA
KATIHAR
SIWAN
MADHUBANI
SUPAUL
NAWADA
CHAMPARANWEST
BHOJPURBUXAR
NALANDA
SITAMARHI
AURANGAABAD
CHAMPARANEAST
BHAGALPUR
VAISHALI
DARBHANGAMUZAFFARPUR
SAMASTIPUR
GOPALGANJ
SAHARSA
BEGUSARAI
MUNGER
KHAGARIA
MADHEPURA
KISHANGANJ
ARWAL LAKHISARAIJEHANABAD SHEIKHPURA
SHEOHAR
District Block
Kishanganj Kishanganj Thakurganj Kochadhaman
Muzaffarpur Aruai Musahiri Saraiya
Patna Barh Bikram Maner
Gaya Barachatti Mohanpur Imamganj
Administrative level
DIO CMO/MO LHV or Male Supervisor
ANM AWW ASHA
District 4Block PHC 12 40 8PHC 8 8Sub-center 16 48AWC 16 48 48Sub-Totals 4 20 40 48 48 96Total Number Participants 256
Health Personnel Interviewed in the Study
Districts and Blocks Sampled for Study
Study Conduct and Analysis
• Field surveys, November 2009- February 2010
• Mixed gender survey teams conducting observation and participant surveys in Hindi
• Triangulation approach to data collection and analysis
• Review of formal policies, guidelines, and broad budget allocations based on key activities and performance domains
• Observation and review of activities and records by field staff
• Assessment of expressed priorities, values, self-efficacy, and capacity
• Study conduct coordinated with external ethnographic study and findings jointly reviewed and validated
Page 16
Findings
• There were consistent and relatively uniform expressions of organizational priorities and values across all of the activity domains. The stated priorities and values did not translate into the actions of the
staff - a substantial “know–do” gap.
The know-do gap also translated into inconsistent performance across most of the activity domains and on many of the technical activities
• There were tensions between the program’s stated goal of decentralization and the manner through which accountability was maintained and planning implemented in the immunization system.All districts showed varying limitations in resources – yet no these
limitations did not show a consistent link to the performance of technical activities.
The formal incentives within the system did not appear to be significantly associated with the performance of many of the technical activities
Page 17
Covera
ge
Quality
Safe
ty
Com
munity
Engagem
ent
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Priority Activities of Health Workers
ANM
LHV
MOIC
Findings – Values and Priorities
• Coverage • 91% “feel committed to improving
the immunization status of the target population
• Quality • 85% “set doable and achievable
targets for performance”• Safety
• 97% “without safe disposal of immunization waste, diseases can occur”
• Community engagement• 83% “use immunization data for
community mobilization and education”
Consistent Expressions of Values and Priorities
Findings – Management & Decision Making
Formal · Formal policies to
decentralize decision
making
· Centralized direction
of microplans
Informal · Varying levels of decision
making in some districts
· Perceived political
interference in decisions
· Some use of evidence-
based decision making
Technical · Few directives on
policies and priorities
provided to block and
PHC levels
· Limited documentation
of decisions at block
levels
· Limited availability of
data at PHC levels
Individual · High perceived
ability to solve
problems based on
data
Performance · Limited awareness
of program
performance at block
level
Inputs· Provision of untied
funds to all levels
(70% to block and
below)
· History of highly-
stratified managerial
culture
Tensions or areas of misalignment
between system components
Findings – Management & Decision Making
District
Fully Vaccinated Children (DLHS 3)
Staff have Low Decision Making Ability Transparency is important
Odds Ratio CI OR CI
Kishanganj 27% 1 (basis of comparison)
Muzaffarpur 57% 4.2 1.2 - 14.7 5.4 1.2 - 24.5
Patna 44% 4.8 1.4 - 17.4 5.4 1.2 - 24.5
Gaya 32% 1.7 0.5 - 5.9 3.8 1.0 - 15.3
Challenges to Decentralization and Improved Coverage
GAYA
PATNA
JAMUI
KAIMURBANKA
ROHTAS
PURNIASARAN
ARARIA
KATIHAR
SIWAN
MADHUBANI
SUPAUL
NAWADA
CHAMPARANWEST
BHOJPURBUXAR
NALANDA
SITAMARHI
AURANGAABAD
CHAMPARANEAST
BHAGALPUR
VAISHALI
DARBHANGAMUZAFFARPUR
SAMASTIPUR
GOPALGANJ
SAHARSA
BEGUSARAI
MUNGER
KHAGARIA
MADHEPURA
KISHANGANJ
ARWAL LAKHISARAIJEHANABAD SHEIKHPURA
SHEOHAR
Coverage 27%
Coverage 32%
Coverage 44%
Coverage 57%
Findings – Planning
Formal · Block-level
microplans
serve as core
planning activity
Informal. · Reported high
priority and positive
value for data
collection and use
· Informal pressure
against local
adaptation
Technical · Unidirectional
flow of data
Individual · Inconsistent staff
participation in
development and
use of microplans
Performance · Strong correlation
between microplan
engagement and
performance of
other duties
Inputs· District funds
allocated for
microplan
formulation
· History of
immunization as
centrally-planned
and directed
program
Tensions or areas of misalignment
between system components
Findings – Planning
Staff Engagement in Microplanning Processes
Microplanning Staff Engagement• ANMs engaged in planning
immunization rosters had 3 times higher likelihood of conducting number of scheduled sessions and 4 times higher likelihood of receiving supervision.
• Highly significant association between supervisor engagement in microplanning activities and supervisory coverage rates
Input area Percent indicating that they engage in specified activities
ASHA & AWW
ANM LHV/MO
Create or update household lists (According to ASHA/AWW)
3% 17% 3%
Identify pregnant women 23%FN 50% 5%
Identify newborn children
19%FN 46% 10%
Identify/note uncooperative households
Not asked –
processes and
frequency
measured
13% 2.5%
Track missing women/children
17% 2.5%
Follow up beneficiaries for full immunization
19% 7.5%
Develop immunization roster/calendar
N/A 38% 58%
Provide inputs into microplan
48%
Page 22
Findings – Service Provision
Formal· Universal
coverage policy
goal· Policies for
expanded outreach
services
Informal · Safety
dominant
priority of staff
· Coverage
least
prioritized
Technical · Limited
availability
and use of
safety
supplies
Individual· Weak link
between
incentives and
individual
behavior
Performance · Notable gaps in
safety procedures
· Significant
variation in levels
of session
coverage
Inputs· Financial incentives to
ASHAs & ANMs based
on coverage
· Increased budgets for
outreach services,
safety supplies and
equipment
Tensions or areas of misalignment
between system components
Findings – Service Provision Priorities & Incentives
• 52% of ASHAs responded that they received cash rewards for meeting their targets
• 54% of ANMs reported receiving cash rewards. • No significant correlation between the receipt of rewards and
• District • Odds of citing maintaining/achieving coverage as a priority for
immunization services• Proportion of scheduled sessions held• Completion of due lists for future sessions,
• Marginally significant relationship between ANMs citing cash rewards and the recording of vaccine administration (OR 2.7 p=.09)
Position Immunization Activity or Performance Area
Coverage Quality Safety Beneficiary’s Awareness
ANM 21% 31% 67% 65%
LHV 25% 42% 65% 48%
MOIC 0% 6% 6% 29%
Immunization Areas Considered a “Priority”
Page 24
Findings – Record Keeping & Reporting
Formal · High attention to
data collection
and reporting
Informal · High agreement
with value of
data collection
and reporting
Technical · Consistent
availability of
personnel but
inconsistent
availability of
equipment
· High demands on
staff for reporting
Individual · Mixed relationship
between self efficacy
and tested abilities
· Training had positive
effect on reporting
accuracy
Performance · Inconsistent and
inaccurate reporting
at session sites and
PHCs
Inputs· State data center
developed
· Training funds
available and utilized
Tensions or areas of misalignment
between system components
Findings – Reporting & Record Keeping
Inconsistent & inaccurate reporting
• 54% of the ANMs correctly recorded vaccines administered during session
• 39% of ANMs noted dropouts for follow up via due lists.
• 18% of block PHCs had 50% or fewer health sub-centers submitting reports
• 76% of block PHCs had less than 65% health sub-centers submitting reports
• Average difference between PHC registers and computerized reports -30%
Cited Use of Data ASHA AWW Clinical
Get appreciated and received incentive
11% 2%
Information on how many vaccinated according to targets
13% 47%
Easy to call beneficiary for Immunization
6% 3%
Benefit of Immunization and preventing
7% 3%
Ensure no child is left out and get information on drop outs
17% 17%
Work improved in immunization program
3% 12%
To know about beneficiaries
13% 1%
Knowledge of next session and immunization due
7% 6%
Other 2% 7%
Don't Know 21% 4%
Reported use of Immunization Data
Findings – Community Mobilization
Formal · ASHAs and
VHSCs
established for
mobilization
activities
Informal · Identified as key
activities and
factor for low
coverage
Technical · Blurred
responsibilities
between
ASHAs and
AWWs
Individual · AWWs showed
greater
engagement than
ASHAs
· Coordination
between CHWs
inconsistent
Performance · All frontline workers
inconsistent in
mobilization
activities
Inputs· Incentives paid for
mothers group
meetings and
coverage
Tensions or areas of misalignment
between system components
Page 27
Findings – Community Mobilization
Session-based activities Percentage of mobilizers
undertaking activities
ASHA & AWWs with household list available/present
31%
ASHA/AWW brought household contact list – based on due list
33%
Shared due list with ANMs 40%
Checked how many women from contact list attended session
25%
ASHA/AWW requested attending women to remind their neighbors to go for vaccination – especially those on list
38%
Question Percentage reporting yes
ASHA AWW
Household list prepared according to the policy guideline
24% 55%
Does the households list mark households having pregnant women
6% 27%
Does the households list mark households having newly born (<32 days)
4% 25%
Does the household list mark which households have children under two year olds
2% 20%
Did you receive a due list from ANM after last immunization session
13% 35%
Activities and responses by Community Health Workers
Page 28
Discussion
• Prevalent “know-do gap”• Prevalence of consistent values and priorities expressed both
through explicit responses to questions and participant concerns
• Gap between values and performance highlights difficulty of linking incentives and structures with culture
• Decision making, planning and management• Engagement with microplanning process has significant
potential to improve engagement
• Microplanning process, through reliance on centrally directed schedules and plans, conflicts with goal of decentralization
• Organizational norms exerting strong pressure and limiting use of available resources
Discussion
• Community engagement
• Sporadic and limited engagement with educating and engaging community
• Misalignment of incentives and community engagement activities
• Disincentive for coordination between community health workers (AWWs and ASHAs)
• Incentives not well aligned with community engagement duties
• Limited engagement of clinical staff with community
• Data collection and reporting
• Significant investment of resources yet conducted as perfunctory process
• Lack of integration of forms, processes, and use of data
Page 30
Assessing the Model and the Approach
• Conceptual challenge of measuring/evaluating alignment across multiple domains
• Challenge of capturing the complexity of immunization system
• Multiple causal pathways between systems misalignment and systems performance
• Consistent and pervasive response bias - dominance of formal organizational values and culture
• Challenge of capturing limited documentation o f formal organizational structures, policies and resources
Page 31
Conclusions
• Bihar continues to make significant progress in improving the performance of its routine immunization program
• Changing the performance of systems as complex as immunization requires the consideration of numerous components, including
• Resources within the system
• Policy and formal structures within the system
• The behavior of the system’s participants
• Informal culture within the systems
• Technical needs, capacity, and interventions
• There is a role for refined tools to examine the coherence and functioning of systems
Page 32
Acknowledgements
• The Bill & Melinda Gates Foundation• The State Health Society Bihar• PATH Study Team:
• Dr. W. Scott Gordon • Dr. Pritu Dhalaria• Dr. Lysander Menezes• Dr. Anwer Aqil• Dr. Dai Hozumi• Dr. Shalini Khare• PATH’s Bhopal and New Delhi staff • PATH’s field investigators
• The health workers in Kishanganj, Muzaffarpur, Patna, and Gaya• The communities that allowed us to observe their services
