Program Planning, Evaluation and QIOH MY!

PART 1

Kusuma Madamala, PhD, MPHConsultant, NNPHIFebruary 5, 2010

Storyboard Review7 Components

1. Description of situation/problem

2. Articulation of aim statement

3. Description of proposed intervention or solution to address the situation/problem

4. Description of what was done to address the situation/problem

http://nnphi.org/CMSuploads/Storyboard-Guidelines-FINAL-05868.pdf

Storyboard Components (continued)

5. Description of the analysis/evaluation of the actions taken to address the situation/problem

6. Description of subsequent action that took place to standardize the improvement

7. Overarching lessons learned

Storyboard Review

• Wide variation in the storyboards regarding the 7 components

• Space allocation challenges • Some use of PDSA as a template to address 7

components and to test proposed change

Storyboard Review

Description of the situation or problem– Most storyboards had a clearly defined problem– Problem articulated by team– Utilized Quality Improvement tools and methods– As noted by Joly and colleagues in their review of

the MLC(9/18/2009), lack of clarity regarding a QI “project”.

– Process vs outcome focused

Description of the situation or problemProcess focused storyboards

Examples:• Improve community

engagement/representation in coalitions • Implement organizational process

improvement • Implement staff training on a variety of issues • Transfer to electronic systems

Description of the situation/problem: Health Outcome/Intervention focused

storyboards

Examples:• Reduce heart disease• Decrease childhood obesity• Increase nutrition education• Increase immunization rates

Aim statement

• Variation in clearly articulated AIM statement• Goals and objectives (process and outcome) • Measurable objectives

By when, Who, Will Do

What by How much?

Child Health Mini-CollaborativeReduce Incidence of Vaccine Preventable Disease

AIM Statement: To maximize the satisfaction, health, and well-being of the client population in 8 LHDs by assuring that all clients have access to public health services when they want and need it.

By providing timely, efficient, andpatient-centered care, the changes made will benefit LHD staff, clinicians, and practice and reduce costs by eliminating waste in the local system.

Child Health Mini-CollaborativeReduce Incidence of Vaccine Preventable Disease

Goals1) Assist LHDs that provide sick and

well child care to Medicaid children through full implementation of Open Access/Office Efficiency principles in their Child HealthClinics;

2) Identify best practice strategies for implementation ofefficiency and access principles in public health;

3) Develop NCDPH staff competencies in Quality Improvement to more effectively spread the principles statewide.

Objectives1. Meet or exceed established standards for customer service access to services within each service area

2. Meet or exceed customer satisfaction rating of 96% (Excellent/Very Good)

3. Maintain expenses at or below approved budget. Maintain or exceed revenue projections of approved budget

4. Meet established standards for productivity (# persons seen and served) within each service area

5. Maintain/reduce unavoidable employee turnover rate of 11%

Storyboard Components

Varied degree of detail provided in each of the storyboards regarding the components

• Component 3 - description of intervention or solution to address the problem

• Component 4 -matching the actions to the problem

• Component 5 - analysis/evaluation• Component 6 - subsequent action- next steps if

no improvement or how to sustain improvement

Description of intervention or solution to address the problem

• Theory of improvement – varied in what was in it and what it meant– some framed within an “if/then” context– occasional reference to the implemented QI

methods• Reference to what we may already know –

varied use of best or promising practice in public health or other sectors

• References used differed by process vs. outcome nature of the problem

Matching actions to the problem

• Dependent upon clear articulation of the problem – QI tools used to id problem

• Both easy and difficult to tell how the actions were linked to the problem (tools used guided actions)

• Wide range and variety of number of implemented actions

Analysis/Evaluation

Highly dependent upon: • Clear AIM statement (with measurable goals

and objectives) • Description of intervention or proposed

solution to the problem

Generally, description of the problem more detailed than in the analysis and evaluation plans

Analysis/Evaluation: Satisfaction

Problems with solely relying on consumer satisfaction:– Measures have a ceiling effect (generally 4.5-4.8)

on 1-5 scale– Clients report high satisfaction to minimize

dissonance– Dissatisfied rarely reply– Satisfaction says nothing about the impact or

outcome of the program

To better understand storyboard components and the variations among them….

Need to explore some differences between Quality Improvement and Program Planning & Evaluation

Quality Improvement is…

• A set of concepts and methods geared toward improving the ability of a product or services to meet consumer needs.

• Involves an understanding of performance problem in system terms, the will to change, the use of data to implement and track changes, and the sequential building of knowledge from testing through implementation.

Basic tools for QI• Bar Chart• Brainstorming• Cause and effect diagram (i.e. fishbone diagram)• Check sheet• Control chart• Five whys and five hows• Flowchart• Force field analysis• Histogram• Nominal group technique• Pareto chart• Pie chart• Run chart• Scatter diagram• SIPOC+ CM• Solution and effect diagram

Advanced tools of QI• Affinity diagram• Control and influence diagram• Gantt chart• Interrelationship diagram• Matrix diagram• Process decision program chart• Prioritization matrix• Radar chart• SMART matrix• SWOT analysis• Tree diagram

Evaluation is….

• Systematic application of research procedures to assess the conceptualization, design, implementation and utility of intervention programs (Rossi and Freeman,1993)

• Assessing the quality and effect of programs

Program Theory

Current language of explaining the connection between what we do (process and effort) and what difference the program makes (effect, impact/outcome)

Functions of Program Theory

• Guides program staff activities• Guides evaluation plan• Facilitates explaining what went right or

wrong and why program failure occurred• Facilitates communication among

stakeholders

Steps involved in creating a program theory

• Draw upon the scientific literature to identify which theories and interventions are known to be effective in addressing the health problem

• Involve selected stakeholders in developing program theory

• Temporally sequence the causal chain of events• Specify “dose” and “mode of administration” of

the intervention delivered in the program

Steps (continued)

• Construct logic model with or without additional diagrams or drawings of the causal chain of events

• Check the model against – Assumptions held by program staff and

stakeholders– Available resources and commitments– The science behind the final intervention, health

problem and target audience

•Issel, L.M. Health Program Planning and Evaluation – A Practical, Systematic Approach for Community Health. Jones and Bartlett. 2009.•Rossi, P. , Freeman, H. & Lipsey, M. Evaluation: A Systematic Approach, Sixth Edition, 1999. Sage Publications. Adapted with permission o f Sage Publications.

Process Improvement Methodologies

Program Process Evaluation

Philosophy Organizations can be more effective if they use staff expertise to improve services and products

Programs need to be justified in terms of their effect on participants

Purpose Systems analysis and improvement focus on identified problem areas from point of view of customer needs

Evaluators determine whether a program was provided as planned and if it made a difference to the participants (customers)

Approach Team-based approach to identifying and analyzing the problem

Evaluator-driven approach to data collection and analysis

Who does it Staff employees from any or all departments, mid-level managers, top level executives

Evaluators and program manager, with or without participation of employees or stakeholders

Methods Engineering approaches to systems analysis

Scientific research methods

Issel, L.M. Health Program Planning and Evaluation – A Practical, Systematic Approach for Community Health. Jones and Bartlett 2009

References• Deming WE. Out of the crisis. 2nd ed. Boston: MIT Press; 2000.• Gillen, S. McKeever, J. Edwards, K & Thielen, L. Promoting Quality Improvement and Achieving Measurable

Change: The Lead States Initiative. Journal of Public Health Management and Practice 2010, 16(1), 55-60.• Institute for Healthcare Improvement:

http://www.ihi.org/IHI/Topics/Improvement/ImprovementMethods/HowToImprove/ Accessed January 16, 2010.

• Issel, L.M. (2009).Health Program Planning and Evaluation. A Practical, Systematic Approach for Community Health.

• Joly, B. Shaler, M, Booth, M, Conway, A. & Mittal, P. Evaluating the Multi-State Learning Collaborative. Journal of Public Health Management and Practice 2010, 16(1), 61-66.

• Joly, B., Booth, M., Shaler, G. and Conway, A. “What Have We Learned So Far.” 9/18/2009 MLC presentation• Langley GL, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to

Enhancing Organizational Performance.• Lotstein D, Seid M, Ricci K, Leuschner K, Margolis P, Lurie N. Using quality improvement methods to improve

public health emergency preparedness: PREPARE for Pandemic Influenza. Health Affairs. June 15, 2008;27(5):328-339.

• Mark, M. & Pines, E. (1995). Implications of continuous quality improvement for program evaluation and evaluators. Evaluation Practice, 16,131-139.

• Quality portal homepage. 2008; www.thequalityportal.com. Accessed December 10, 2008.• Bialek, Duffy and Moran(2009) The Public Health Quality Improvement Handbook. ASQ Quality Press:

Milwaukee, WI• The Plan-Do-Study-Act (PDSA) cycle was originally developed by Walter A. Shewhart as the Plan-Do-Check-

Act (PDCA) cycle. W. Edwards Deming modified Shewhart's cycle to PDSA, replacing "Check" with "Study." [See Deming WE. The New Economics for Industry, Government, and Education. Cambridge, MA: The MIT Press; 2000.]

• Rossi and Freeman (1993). Evaluation: A systematic approach (6th ed). Thousand Oaks, CA: Sage Publications.

Program Planning, Evaluation, and QI, Oh My!

PART 2

Mary V. Davis, DrPH, MSPHFebruary 5, 2010

PlanPlan: PH Program & QI: PH Program & QI

• Same planning phase BUT more rapid in QI• Targets, goals, theory of change, change

packages (interventions), data needs• QI—Emphasis on root causes that can be

changed and tested in small cycles• PH—Emphasis more contextual, identify the

BIG program that can change everything

NC Mini-Collaborative 2NC Mini-Collaborative 2Target: Workforce is assessed to determine its abilities to deliver

population-based services and a workforce development plan is crafted.

Specific Target (AIM Statement): Improve the retention of public health nurses through a

mentoring program that provides nurturing for individual and professional growth to new public health nurses.

1. Description of Situation1. Description of Situation• High public health nurse turnover rates in local

health departments

• Factors– Pay– Nurses entering public health positions lack training in

epidemiology, health education, case management (job mismatch and dissatisfaction)

• Cost of turnover: 75% of annual salary for position

2. Revised AIM Statement2. Revised AIM Statement

The overall aim of this collaborative is to determine if a PHN mentoring program can be used to improve the competence, retention and satisfaction of new PHN’s (defined as <2 years or in a new PH nursing role).

2a Goals 2a Goals (set up for Evaluation/Study phases)(set up for Evaluation/Study phases)

By October, 2010…• Mentees planning to remain in public health will increase by 25% • Mentees who recommend working in an LHA to a colleagues will

increase by 25%• Mentees and mentor overall job satisfaction will improve by 15%• Understanding of population-based services as well as mentee

knowledge and understanding of their role will increase by 25%• 100% of participating mentors and mentees will complete the 6 month program and all will be satisfied with the process• Greater than 90% of mentees and mentors will be willing to serve

as a mentor in the future

3/4. Description of proposed intervention /3/4. Description of proposed intervention /Actions being planned Actions being planned (DO)(DO)

• Six month pilot mentoring program for new PH nurses NC LHDs– Mentor training/mentee orientation– Twice monthly mentor/mentee contacts – Monthly TA calls to mentors – TA calls for mentees as needed– Reunion meeting @ NCPHA conference– Pre/post measures of new nurse job satisfaction/

intent to stay in job

Change Package DevelopmentChange Package Development

• Nurse mentoring programs designed to job fit– Wayne County NC program written into policy– Georgia District Nurse Mentoring Program– Colorado Leaders Nurse Mentoring Program

• Planning Committee chose to adapt and modify Georgia District Nurse Mentoring Program

Theory of ChangeTheory of Change

Stable PH Nurse capacity to

deliver population

based services

PH Nurse job

satisfactionBetter job

match

PH Nurses trained in

population based services

Nurse mentoring

program for new ph nurses

PH nurse job retention

Intervention vs QIIntervention vs QIClassic PH Intervention• Identified retention

problem• Identify mentoring practices

as solution• Tailor or adopt mentoring

programs to situation• Implement pilot program

with set intervention• Evaluation

QI• Identified retention

problem—check root cause• Identify mentoring practices

as solution• Tailor or adopt mentoring

programs to situation• Implement program with

small cycles tests of change• Evaluation

So How is this QI?So How is this QI?

• Small cycle tests of change by mentors• Mentoring techniques– Problem solving– Meeting times and venues– Improving job fit– Teaching population-based services

• Monthly conference calls with faculty/planning committee

5. Evaluation of Actions Taken5. Evaluation of Actions Taken(QI: Study )(QI: Study )

• Process measures-- number of mentor/mentee meetings, satisfaction with process, challenges and facilitators, mentor best practices

• Outcomes—mentees – Population based knowledge– Job satisfaction and fit– Intention to stay in public health

6. Standardizing Improvements6. Standardizing Improvements(QI: Act)(QI: Act)

• Identify aspects of mentoring program that worked well for most LHDs

• Create mentoring change package• Disseminate statewide

7. Plan Phase Lessons Learned7. Plan Phase Lessons Learned

• Getting to root cause for target with LHDs challenge• Getting right planning committee members

(nurse supervisors, health educators) has been key• Engaging QI experts improved planning for QI:

Aim statements, change package, applications, training, timeline