Date post: | 11-Jan-2016 |
Category: |
Documents |
Upload: | penelope-nash |
View: | 222 times |
Download: | 3 times |
Program Planning, Evaluation and QIOH MY!
PART 1
Kusuma Madamala, PhD, MPHConsultant, NNPHIFebruary 5, 2010
Storyboard Review7 Components
1. Description of situation/problem
2. Articulation of aim statement
3. Description of proposed intervention or solution to address the situation/problem
4. Description of what was done to address the situation/problem
http://nnphi.org/CMSuploads/Storyboard-Guidelines-FINAL-05868.pdf
Storyboard Components (continued)
5. Description of the analysis/evaluation of the actions taken to address the situation/problem
6. Description of subsequent action that took place to standardize the improvement
7. Overarching lessons learned
Storyboard Review
• Wide variation in the storyboards regarding the 7 components
• Space allocation challenges • Some use of PDSA as a template to address 7
components and to test proposed change
Storyboard Review
Description of the situation or problem– Most storyboards had a clearly defined problem– Problem articulated by team– Utilized Quality Improvement tools and methods– As noted by Joly and colleagues in their review of
the MLC(9/18/2009), lack of clarity regarding a QI “project”.
– Process vs outcome focused
Description of the situation or problemProcess focused storyboards
Examples:• Improve community
engagement/representation in coalitions • Implement organizational process
improvement • Implement staff training on a variety of issues • Transfer to electronic systems
Description of the situation/problem: Health Outcome/Intervention focused
storyboards
Examples:• Reduce heart disease• Decrease childhood obesity• Increase nutrition education• Increase immunization rates
Aim statement
• Variation in clearly articulated AIM statement• Goals and objectives (process and outcome) • Measurable objectives
By when, Who, Will Do
What by How much?
Child Health Mini-CollaborativeReduce Incidence of Vaccine Preventable Disease
AIM Statement: To maximize the satisfaction, health, and well-being of the client population in 8 LHDs by assuring that all clients have access to public health services when they want and need it.
By providing timely, efficient, andpatient-centered care, the changes made will benefit LHD staff, clinicians, and practice and reduce costs by eliminating waste in the local system.
Child Health Mini-CollaborativeReduce Incidence of Vaccine Preventable Disease
Goals1) Assist LHDs that provide sick and
well child care to Medicaid children through full implementation of Open Access/Office Efficiency principles in their Child HealthClinics;
2) Identify best practice strategies for implementation ofefficiency and access principles in public health;
3) Develop NCDPH staff competencies in Quality Improvement to more effectively spread the principles statewide.
Objectives1. Meet or exceed established standards for customer service access to services within each service area
2. Meet or exceed customer satisfaction rating of 96% (Excellent/Very Good)
3. Maintain expenses at or below approved budget. Maintain or exceed revenue projections of approved budget
4. Meet established standards for productivity (# persons seen and served) within each service area
5. Maintain/reduce unavoidable employee turnover rate of 11%
Storyboard Components
Varied degree of detail provided in each of the storyboards regarding the components
• Component 3 - description of intervention or solution to address the problem
• Component 4 -matching the actions to the problem
• Component 5 - analysis/evaluation• Component 6 - subsequent action- next steps if
no improvement or how to sustain improvement
Description of intervention or solution to address the problem
• Theory of improvement – varied in what was in it and what it meant– some framed within an “if/then” context– occasional reference to the implemented QI
methods• Reference to what we may already know –
varied use of best or promising practice in public health or other sectors
• References used differed by process vs. outcome nature of the problem
Matching actions to the problem
• Dependent upon clear articulation of the problem – QI tools used to id problem
• Both easy and difficult to tell how the actions were linked to the problem (tools used guided actions)
• Wide range and variety of number of implemented actions
Analysis/Evaluation
Highly dependent upon: • Clear AIM statement (with measurable goals
and objectives) • Description of intervention or proposed
solution to the problem
Generally, description of the problem more detailed than in the analysis and evaluation plans
Analysis/Evaluation: Satisfaction
Problems with solely relying on consumer satisfaction:– Measures have a ceiling effect (generally 4.5-4.8)
on 1-5 scale– Clients report high satisfaction to minimize
dissonance– Dissatisfied rarely reply– Satisfaction says nothing about the impact or
outcome of the program
To better understand storyboard components and the variations among them….
Need to explore some differences between Quality Improvement and Program Planning & Evaluation
Quality Improvement is…
• A set of concepts and methods geared toward improving the ability of a product or services to meet consumer needs.
• Involves an understanding of performance problem in system terms, the will to change, the use of data to implement and track changes, and the sequential building of knowledge from testing through implementation.
Basic tools for QI• Bar Chart• Brainstorming• Cause and effect diagram (i.e. fishbone diagram)• Check sheet• Control chart• Five whys and five hows• Flowchart• Force field analysis• Histogram• Nominal group technique• Pareto chart• Pie chart• Run chart• Scatter diagram• SIPOC+ CM• Solution and effect diagram
Advanced tools of QI• Affinity diagram• Control and influence diagram• Gantt chart• Interrelationship diagram• Matrix diagram• Process decision program chart• Prioritization matrix• Radar chart• SMART matrix• SWOT analysis• Tree diagram
Evaluation is….
• Systematic application of research procedures to assess the conceptualization, design, implementation and utility of intervention programs (Rossi and Freeman,1993)
• Assessing the quality and effect of programs
Program Theory
Current language of explaining the connection between what we do (process and effort) and what difference the program makes (effect, impact/outcome)
Functions of Program Theory
• Guides program staff activities• Guides evaluation plan• Facilitates explaining what went right or
wrong and why program failure occurred• Facilitates communication among
stakeholders
Steps involved in creating a program theory
• Draw upon the scientific literature to identify which theories and interventions are known to be effective in addressing the health problem
• Involve selected stakeholders in developing program theory
• Temporally sequence the causal chain of events• Specify “dose” and “mode of administration” of
the intervention delivered in the program
Steps (continued)
• Construct logic model with or without additional diagrams or drawings of the causal chain of events
• Check the model against – Assumptions held by program staff and
stakeholders– Available resources and commitments– The science behind the final intervention, health
problem and target audience
•Issel, L.M. Health Program Planning and Evaluation – A Practical, Systematic Approach for Community Health. Jones and Bartlett. 2009.•Rossi, P. , Freeman, H. & Lipsey, M. Evaluation: A Systematic Approach, Sixth Edition, 1999. Sage Publications. Adapted with permission o f Sage Publications.
Process Improvement Methodologies
Program Process Evaluation
Philosophy Organizations can be more effective if they use staff expertise to improve services and products
Programs need to be justified in terms of their effect on participants
Purpose Systems analysis and improvement focus on identified problem areas from point of view of customer needs
Evaluators determine whether a program was provided as planned and if it made a difference to the participants (customers)
Approach Team-based approach to identifying and analyzing the problem
Evaluator-driven approach to data collection and analysis
Who does it Staff employees from any or all departments, mid-level managers, top level executives
Evaluators and program manager, with or without participation of employees or stakeholders
Methods Engineering approaches to systems analysis
Scientific research methods
Issel, L.M. Health Program Planning and Evaluation – A Practical, Systematic Approach for Community Health. Jones and Bartlett 2009
References• Deming WE. Out of the crisis. 2nd ed. Boston: MIT Press; 2000.• Gillen, S. McKeever, J. Edwards, K & Thielen, L. Promoting Quality Improvement and Achieving Measurable
Change: The Lead States Initiative. Journal of Public Health Management and Practice 2010, 16(1), 55-60.• Institute for Healthcare Improvement:
http://www.ihi.org/IHI/Topics/Improvement/ImprovementMethods/HowToImprove/ Accessed January 16, 2010.
• Issel, L.M. (2009).Health Program Planning and Evaluation. A Practical, Systematic Approach for Community Health.
• Joly, B. Shaler, M, Booth, M, Conway, A. & Mittal, P. Evaluating the Multi-State Learning Collaborative. Journal of Public Health Management and Practice 2010, 16(1), 61-66.
• Joly, B., Booth, M., Shaler, G. and Conway, A. “What Have We Learned So Far.” 9/18/2009 MLC presentation• Langley GL, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to
Enhancing Organizational Performance.• Lotstein D, Seid M, Ricci K, Leuschner K, Margolis P, Lurie N. Using quality improvement methods to improve
public health emergency preparedness: PREPARE for Pandemic Influenza. Health Affairs. June 15, 2008;27(5):328-339.
• Mark, M. & Pines, E. (1995). Implications of continuous quality improvement for program evaluation and evaluators. Evaluation Practice, 16,131-139.
• Quality portal homepage. 2008; www.thequalityportal.com. Accessed December 10, 2008.• Bialek, Duffy and Moran(2009) The Public Health Quality Improvement Handbook. ASQ Quality Press:
Milwaukee, WI• The Plan-Do-Study-Act (PDSA) cycle was originally developed by Walter A. Shewhart as the Plan-Do-Check-
Act (PDCA) cycle. W. Edwards Deming modified Shewhart's cycle to PDSA, replacing "Check" with "Study." [See Deming WE. The New Economics for Industry, Government, and Education. Cambridge, MA: The MIT Press; 2000.]
• Rossi and Freeman (1993). Evaluation: A systematic approach (6th ed). Thousand Oaks, CA: Sage Publications.
Program Planning, Evaluation, and QI, Oh My!
PART 2
Mary V. Davis, DrPH, MSPHFebruary 5, 2010
PlanPlan: PH Program & QI: PH Program & QI
• Same planning phase BUT more rapid in QI• Targets, goals, theory of change, change
packages (interventions), data needs• QI—Emphasis on root causes that can be
changed and tested in small cycles• PH—Emphasis more contextual, identify the
BIG program that can change everything
NC Mini-Collaborative 2NC Mini-Collaborative 2Target: Workforce is assessed to determine its abilities to deliver
population-based services and a workforce development plan is crafted.
Specific Target (AIM Statement): Improve the retention of public health nurses through a
mentoring program that provides nurturing for individual and professional growth to new public health nurses.
1. Description of Situation1. Description of Situation• High public health nurse turnover rates in local
health departments
• Factors– Pay– Nurses entering public health positions lack training in
epidemiology, health education, case management (job mismatch and dissatisfaction)
• Cost of turnover: 75% of annual salary for position
2. Revised AIM Statement2. Revised AIM Statement
The overall aim of this collaborative is to determine if a PHN mentoring program can be used to improve the competence, retention and satisfaction of new PHN’s (defined as <2 years or in a new PH nursing role).
2a Goals 2a Goals (set up for Evaluation/Study phases)(set up for Evaluation/Study phases)
By October, 2010…• Mentees planning to remain in public health will increase by 25% • Mentees who recommend working in an LHA to a colleagues will
increase by 25%• Mentees and mentor overall job satisfaction will improve by 15%• Understanding of population-based services as well as mentee
knowledge and understanding of their role will increase by 25%• 100% of participating mentors and mentees will complete the 6 month program and all will be satisfied with the process• Greater than 90% of mentees and mentors will be willing to serve
as a mentor in the future
3/4. Description of proposed intervention /3/4. Description of proposed intervention /Actions being planned Actions being planned (DO)(DO)
• Six month pilot mentoring program for new PH nurses NC LHDs– Mentor training/mentee orientation– Twice monthly mentor/mentee contacts – Monthly TA calls to mentors – TA calls for mentees as needed– Reunion meeting @ NCPHA conference– Pre/post measures of new nurse job satisfaction/
intent to stay in job
Change Package DevelopmentChange Package Development
• Nurse mentoring programs designed to job fit– Wayne County NC program written into policy– Georgia District Nurse Mentoring Program– Colorado Leaders Nurse Mentoring Program
• Planning Committee chose to adapt and modify Georgia District Nurse Mentoring Program
Theory of ChangeTheory of Change
Stable PH Nurse capacity to
deliver population
based services
PH Nurse job
satisfactionBetter job
match
PH Nurses trained in
population based services
Nurse mentoring
program for new ph nurses
PH nurse job retention
Intervention vs QIIntervention vs QIClassic PH Intervention• Identified retention
problem• Identify mentoring practices
as solution• Tailor or adopt mentoring
programs to situation• Implement pilot program
with set intervention• Evaluation
QI• Identified retention
problem—check root cause• Identify mentoring practices
as solution• Tailor or adopt mentoring
programs to situation• Implement program with
small cycles tests of change• Evaluation
So How is this QI?So How is this QI?
• Small cycle tests of change by mentors• Mentoring techniques– Problem solving– Meeting times and venues– Improving job fit– Teaching population-based services
• Monthly conference calls with faculty/planning committee
5. Evaluation of Actions Taken5. Evaluation of Actions Taken(QI: Study )(QI: Study )
• Process measures-- number of mentor/mentee meetings, satisfaction with process, challenges and facilitators, mentor best practices
• Outcomes—mentees – Population based knowledge– Job satisfaction and fit– Intention to stay in public health
6. Standardizing Improvements6. Standardizing Improvements(QI: Act)(QI: Act)
• Identify aspects of mentoring program that worked well for most LHDs
• Create mentoring change package• Disseminate statewide
7. Plan Phase Lessons Learned7. Plan Phase Lessons Learned
• Getting to root cause for target with LHDs challenge• Getting right planning committee members
(nurse supervisors, health educators) has been key• Engaging QI experts improved planning for QI:
Aim statements, change package, applications, training, timeline