Building a Performance Improvement System in a Large
Urban Public Health Department: Linkages and Learning Collaboratives
MLC Open ForumWashington, DC
September 16th, 2010
Dawn Marie Jacobson, MD, MPHDirector, Performance ImprovementLos Angeles County Department of Public Health
Presentation Objectives• Provide an overview of performance
improvement efforts at Los Angeles County Department of Public Health
• Discuss linkages required for efficient analysis and reporting – Data– Standards/Benchmarks– Reports
• Describe the DPH Performance Improvement Learning Collaborative (PILC)
Quality Improvement—LAC DPH
Quality Improvement DivisionOffice of the Medical Director
Organizational Development and Training
Nursing Administration
Health Education Administration
Public Health Investigation
Physician Administration
Oral Health
Quality ImprovementFunctions
1. Performance Improvement
2. Professional Practice
3. Science Review
4. Service Quality
Gunzenhauser JD, Eggena ZP, Fielding JE, Smith KN, Jacobson DM, Bazini-Barakat N. The Quality Improvement Experience in a High-Performing Local Health Department, Los Angeles County. Journal of Public Health Management and Practice, 2010, 16 (1): 39-48
Performance Improvement—LAC DPH
1. Strategic Planning – determine priorities and goals
2. Performance Measurement– data management– reporting
3. Performance Improvement Projects– Modified IHI Method for Improvement– Other tools (RCA, Fishbone diagrams, etc)
Key Elements
1. Strategic Planning: Determine Priorities and Goals
• What are the priority public health issues in Los Angeles County?
• What are the behaviors and outcomes related to these issues that we want for people who live in LA County?
• How can we measure these conditions?
SPA/SPA/ProgramProgram
Strategic PlanStrategic Plan
DPH Strategic Plan
County Strategic PlanCounty Strategic Plan
LAC DPH—Strategic Planning
– Quality Improvement Division• Public Health Measures required Mission and Vision
Statements, Goal Setting, and Evidence-Based Strategies (2004-2007)
– Office of Planning• Department-Level Strategic Plan (2008-2011)
– Division and Program Level Strategic Plans• Office of Planning may provide technical assistance
2. Performance Measurement: Public Health Measures
• The LAC DPH approach based on Mark Friedman’s “Results Accountability”
• 32 operational units identified population health indicators linked to program performance measures to follow over time
• Healthy People 2010 objectives often identified and used as the “Standard” to achieve over time
• Organized by Essential Services of Public Health/NACCHO Standards/Accreditation Domains
Public Health Measures
POPULATION INDICATORS(measures of population-level
health outcomes and behaviors)
PERFORMANCE MEASURES(measures of program
effort and output)AND
Public HealthMeasures
Selecting Indicators and Measures
EffectiveStrategi
es
Strategy 1
Strategy 2
Population
Goals
Goal 1
Population
Indicators
Indicator
Indicator
Performance
Goals
Goal 1
Goal 2
Performance
Measures
Measure 1
Measure 2
Population Health Program Performance
Healthy People 2010/20
20
Federal, State, or
Local Guidelines
Community GuideClinical GuideOther Sources
NACCHOStandards
StrategicPlan
Type (ranked)Research findings: syntheses, systematic reviews, meta-analyses
Research findings: individual studies (quantitative and qualitative)
Performance data such as program evaluation or peer review reports
Demonstrated to be effective in computer modeling, simulations, or exercises
Consensus recommendations of recognized experts either local or national
Anecdotal accounts such as practices of other public health jurisdictions alleged to be effective, clinical narratives, or case reports
Decision-Making in Public Health: Evidence Review
Tier 1 Evidence
Type (unranked)
Philosophical or conceptual bases such as an ethical framework or a professional code of conduct
Regulations, laws, or public policies
Grant requirements
Community preferences
Necessary because of the political climate
Best hunches
Decision-Making in Public Health: Other Rationale
Tier 2 Other Rationale
Public Health Measures:Population Indicators
• Longer life span• Increased quality of life• Increased health equity• Less disease• Less premature death• Healthier choices• Safer environment• Healthier homes
POPULATION-LEVELHEALTH OUTCOMES
& BEHAVIORS
Public Health Measures: Performance Measures
Quantity Quality
Input / Effort
How Much Did We Do?
(#)
How Well Did We Do
It? (%)
Output / Effect
How Much
Change? (#)
Quality of Change?
(%)
1. Who are our clients?2. Which services do we
provide to our clients?3. What evidence-based
strategies will lead to positive change in our clients?
4. How can we measure if our clients are better off?
5. How can we measure if we are delivering services well?
Performance Measures
• Policies Created• People Informed• Partners Engaged• Surveillance Performed• Investigations Completed• Increased Access to Services• Client satisfaction
MEASURES OF PROGRAM
EFFORT & OUTPUT
Population Goal To reduce morbidity and mortality from vaccine-preventable diseases by improving immunization levels
Population IndicatorPercentage of children, ages 19-35 months, who are fully immunized with
one of the series of the Advisory Committee on Immunization Practices (ACIP)recommended vaccines
Effective, Evidence-Based Strategies (selected subset)1. Change provider behavior through systems change—Provider
recall/reminder systems in clinics2. Change provider behavior through education—multi-component
interventions with education3. Increase demand and access to immunizations—reduce out-of-pocket
costsPerformance Goal (NACCHO Standard 9)
Performance MeasurePercent of Immunization Program public and nonprofit clinic partners whoroutinely meet the Standards for Pediatric Immunization Practices for
provider andclient recall/reminder systems
Example: Immunization Program
Public Health Measures:Data Management
• Standardized spreadsheet for reporting data• Labeling System
– Population Indicators= letters• organized by population goals
– Program Performance Measures= numbers• organized by Accreditation domains
• Data Documentation• Standard Documentation
Public Health Measures: Data Measurement Worksheet
– Type of measure (PI or PM)– Measure name and description– How calculated– Target – Data source (Name, govt level, dept, program)– Data collection instrument– Data collection plan– NACCHO Standard (if applicable)
Common Data Sources
Population Indicators
Examples: – Los Angeles County Health Survey– LA FANS– Disease specific surveillance
systems– Vital Records– CA Health Interview Survey– OSHPD (Healthcare Utilization data)– BRFSS– YRBS– National Immunization Survey
Program PerformanceMeasures Examples:
– Casewatch (STD, AIDS)– RASSCLE (lead surveillance)– EHMIS– TRIMS (TB control)– vCMR (outbreak reporting
and investigation)– Syndromic surveillance– Clinic utilization data– Contracts and grants
management– Project-specific databases
Common Standard Sources
Population Indicators
Examples:– Healthy People– State of CA plans – County of LA plans (e.g.
Commission on HIV)– Internal DPH
Program PerformanceMeasuresExamples:
– Healthy People– CDC guidelines– State of CA guidelines or
mandates– Grant-specific guidelines– Professional associations– Internal DPH
Example: Immunization Program
Example: Immunization Program
Public Health Measures: Data Management
• Data collected two times per year• Data analyzed and reported one time per year• Option to update content of Public Health
Measures one time per year• Public Health Measures database in
development
Public Health Measures: Reporting
National Efforts1. CDC Guidelines or
Performance Measures
State Efforts1. State Performance
Measures2. Mandates and
Regulations
County Efforts1. Performance Counts!2. County Progress Report
Department Efforts1. Annual Performance
Report2. PI Project Reports
LAC DPH Annual Performance Report
• Internal report of a subset of Population Indicators and Performance Measures
• Includes:– Department-Level Report Card – Program-Level Performance Snapshots
• In-Person Progress Review with Director and Health Officer
Public Health Report Card
Public Health Report Card
Public Health Measures: Reporting
• Future– Linked to strategic plan objectives– More frequent reports using an automated database– Portfolio of services by NACCHO Standards– Accreditation Preparation
Total
NS1
NS2 NS3 NS4 NS5
NS6 NS7 NS8 NS9
NS10
NS11
PIs 224
PMs 736 51 (7%)
100 (14%)
110 (15%)
34(5%)
22 (3%)
36 (5%)
142 (19%)
92 (13%)
61 (8%)
36 (5%)
87 (12%)
3. Performance Improvement Projects
Since repeated measurement by itself is not enough toimprove public health practice....
What are common processes in our Department? How can we share best practices in common processes?
What support do staff need to use PI methods (e.g. rapid cycle tests) in practice?
How do we spread a successful PI approach throughout the Department?
Performance Improvement Projects
A Learning Collaborative Approach*• Create an internal performance improvement learning
collaborative (PI LC) of a diverse group of DPH units• Teams represent 8 of 32 department Divisions/Programs• Teams learn and work together for a 10 month period• Teams apply common PI methods to improve a priority area
selected by their respective units
*This project is part of the “Building the Evidence for Quality Improvement in Public Health ” grant program funded by the Robert Wood Johnson Foundation.The RAND Corporation is providing training and evaluation support.
PI Method and Tools:The IHI Model for Improvement... Plus
1. Set the Aim• Population health improvement• Customer or service improvement
2. Measure Performance• Population Indicators• Program Performance Measures
3. Map the Process4. Make Changes for Improvement
• Evidence Review and Best Practices• PDSA cycles
5. Apply other Tools (RCA, Fishbone diagrams, etc.)
Set the Aim: PI LC Team Aims• Improve Provider/Contractor Performance
– Office of AIDS Policy and Programs– Children’s Health and Disability Prevention Program– Tobacco Control and Prevention Program– Substance Abuse Prevention and Control– Emergency Preparedness and Response
• Inform and Engage Community Stakeholders– Office of Senior Health– Acute Communicable Disease Control– Office of Women’s Health
Measure Performance: PI LC Project Metrics• Population IndicatorsExamples:
– % of children who qualify for the CHDP program who receive needed follow- up care
– Community incident rates of reptile associated salmonella– Community rates of CVD among women
• Program MeasuresExamples:
– % of CHDP forms with a condition needing referral that have a referral identified
– % of Early Childhood Education providers receiving the photonovela intervention
– # of new callers to hotline per week
Percent of current adult smokers and current youth smokers in Los Angeles County (2001-2006)
14.3% 13.9% 14.3%14.5% 14.4%11.8%
0%
5%
10%
15%
20%
2001 2002 2003 2004 2005 2006Calendar Year
Perc
ent
TCPP-A Percent of adults TCPP-B Percent of youth
HP 2010 Standard(16%)
HP 2010 Standard(12%)
Data not collected where missing
Data Sources: 1) Los Angeles County Health Survey (LACHS), LAC DPH, OHAE and 2) Youth Risk Behavior Survey (YRBS), CDC, NCCDPHP
Number of jurisdictions adopting a legislative-based policy that prohibits smoking in outdoor areas (00-01 to 09-10)
0 1 2 1
5 5
10
6 6
3
0
2
4
6
8
10
12
00-01 01-02 02-03 03-04 04-05 05-06 06-07 07-08 08-09 09-10Fiscal Year
Num
ber
TCPP 5-1 Number of Jurisdictions
DPH and State of CA Standard(4 jurisdictions per 3-year funding cycle)
1st Funding Cycle = 4 3rd Funding Cycle = 152nd Funding Cycle =
Data Source: Internal Records, LAC DPH, TCPP
Map the Process: Examples
Make Changes for Improvement:PDSA and Other Tools
Average score January 2010(scale of 1-4: 4= strongly agree,
1= strongly disagree)
Average score April 2010(scale of 1-4: 4= strongly agree,
1= strongly disagree)
1. Task Force meetings are well facilitated. 3.67 3.8
1. Task Force meetings provide an effective learning forum for my campaign.
3.63 3.7
1. The ideas/strategies discussed at the meetings are helpful to my campaign.
3.75 3.9
1. Ideas/strategies shared by other task force members are helpful to my campaign.
3.71 3.8
1. Information provided by TCPP staff (e.g. billing, announcements, campaign strategies, etc.) are useful.
3.62 3.9
PDSA cycles to improve subcontractor satisfaction with Task Force meetings (complete)
Make Changes for Improvement:PDSA and Other Tools
Average score April 2010(scale of 1-4: 4= very helpful, 1= not helpful)
Community assessment 3.5
Policy campaign strategy 3.8
Coalition building/ broadening 3.6
Policy campaign implementation and policy adoption 3.6
Policy implementation and enforcement 3.4
PDSA cycles to improve subcontractor training for communitycampaigns to pass tobacco policies and ordinances (in progress)
Pareto Chart:Adult smoking rate by SPA
2007
0
5
10
15
20
25
2007
HP 2010 goal
PI LC Evaluation Metrics• Improve project team metrics• Increase staff knowledge and use of QI methods
– % of senior managers reporting they are aware of the 4-step Model for Improvement
– % of senior managers who say they are proficient in selected quality improvement methods and tools
• Disseminate QI methods across the Department– % of DPH Division/Program Directors who report using
rapid-cycle PDSA to improve performance in a priority area each year
– % of staff who report they are encouraged to take risks when implementing QI projects
Based on: 1) Senior Manager Survey of QI culture, QI knowledge and readiness for change; 2) Key Informant Interviews of DPH Executives; 3) Monthly reports from the 8 PI LC project teams; 4) DPH Annual QI Report Card
PI LC Early Lessons Learned• Successes
– Team engagement with learning sessions– Improved understanding of internal processes and links
to key measures with process mapping– Individual team coaching
• Challenges– Competing priorities (e.g., H1N1 response)– Doing rapid small scale cycles is a very new concept– Lack of readily available, validated measures and best
practice tools for team aim areas
Performance Improvement Projects: Future Plans
• Have all operational divisions and units working on at least one PI project each year
• PI to track PI projects• Inclusion of key PI project measures in the
Public Health Measures• Reporting PI project results
– Monthly for selected programs– Yearly summary for all others
Annual TimelineAnnual Timeline
Jan
Feb Mar Apr May June July Aug Sep Oct Nov Dec
Add/Drop//Modify
Public HealthMeasures
DataUpdat
e
Data Analysis and Review
PrepareReports
DataUpdate
Data Analysisand Review
Prepare DPHand CEOPerformanceReports
Progress Reviews with Health Officer
Performance Improvement Training
Performance Improvement Projects
Summary
• A large health department needs to link many sources of data, standards, and reporting processes to build an efficient performance improvement system– This takes time to do properly– Best with department-wide participation – Need to communicate effectively across levels of
government and understand a wide variety of unit demands
Summary
• A learning collaborative approach is essential to explore common processes and small tests of change– Brings PI champions together which generates
enthusiasm– Maximizes learning and sharing– Promotes a culture of openness and transparency– Creates a “centralized” opportunity for technical
assistance and coaching
Questions and Discussion