The Big PictureThe Role of Evaluation in

Public Health

Luann D’Ambrosio, MEd

Activity

Think of your favorite quote or sage advice that helps guide you through your professional work.

Wise Advice

I hear and I forget.

I see and I believe.

I do and I understand.

—Confucius (551–479 BC)

Standards

Utility

Accuracy

Propriety

Feasibility

CDC Evaluation Framework

• Describe the relationship between performance management, program planning and evaluation

• Describe the steps of program planning and evaluation process

• Identify key stakeholders in an evaluation

• Develop the components and elements of a program logic model

• Create a basic evaluation plan including data collection and analyzing methods

• List three ways to use evaluation data to draw conclusions about a program

• Define ROI, cost benefit ratios and social return on investment.

Objectives

Accreditation

CDC GrantsCDC awards grants for public health performance management systems

•$42.5 million allocated for 94 projects in state, local, tribal, and territories health departments

•Distributed in 49 states, 16 tribes and territories and nine local jurisdictions

•Five-year cooperative agreement awards are being funded by the health care reform law

Benefits of Accreditation

Accountability & credibility

Tool for improvement

Greater collaboration

Recognition & validationBetter

understanding of public health

Team building

Highlights HD strengths

Accreditation+

Phase 1

EvaluationMonitoring

Implementation

BudgetsAction Plans

Strategies

Strategic Issues Goals & Objectives

Mission ValuesVision

SWOCAnalysis &

Environmental Scan

Mandate Analysis

StakeholderAssessment

Plan the

Plan

Readiness Assessment

Phase 2

Phase 3

Phase 4

Strategic Planning

Performance Management (PM)

A systematic process aimed at helping achieve an

organization’s mission and strategic goals by

improving effectiveness, empowering employees,

and streamlining the decision-making process. 

PerformancePerformance BehaviorBehavior ResultsResults= +

They Are Not the Same

Quality assurance Quality improvement

Reactive Proactive

Works on problems after they occur

Works on processes

Regulatory usually by state or federal law

Seeks to improve (culture shift)

Led by management Led by staff

Periodic look-back Continuous

Responds to a mandate or crisis or fixed schedule

Proactively selects a process to improve

Meets a standard (pass/fail) Exceeds expectations

They Are Not the Same

Evaluation Quality improvement

Assess a program at a moment in time

Understand the process that is in place

Static Ongoing

Doesn’t include identification of the source of a problem or potential solutions

Entails finding the root cause of a problem and interventions targeted to address it

Doesn’t measure improvements

Focused on making measurable improvements

Program-focused Customer-focused

A step in the QI process Includes evaluation

Value of Performance Management

• Disciplined approach

• Systemic

• Systematic

• Engaged customers and staff

• Results focused

• Proven method

Culture of QITopic Organization-wide Program/unit

Improvement System focus Specific project focus

QI planning Tied to the strategic plan

Program/unit level

QI goals Strategic plan Individual program/unit level plans

Approaches Baldrige ProgramOrganization QI Council

Lean Six SigmaIndividual QI TeamsRapid Cycle PDCA

Performance Management Frameworks

• Baldrige

• Six Sigma

• Lean

PDSA Cycle

What are we trying to accomplish?What are we trying to accomplish?

How will we know that a change is an improvement?How will we know that a change is an improvement?

What changes can we make that will result in improvement?What changes can we make that will result in improvement?

Memory Jogger

ToolsLogic models and work flow charts

Data and analysis tools

• Customer-supplier relationships

• Client flow, information flow

• Root cause tools: fishbone diagram, Pareto chart

• Force field analysis• Interrelationship digraph

QI PROCESS

to improve policies, programs, & outcomes•Manage changes•Create learning organization

Turning Point Performance Management Collaborative, 2003

PERFORMANCE MANAGEMENT

SYSTEM

PERFORMANCE STANDARDS•Identify relevant standards•Select indicators•Set goals & targets•Communicate expectations

PERFORMANCE MEASUREMENT•Refine indicators & define measures•Develop data systems•Collect data

REPORTING OF PROGRESS•Analyze data•Feed data back to managers, staff, policy makers, constituents•Develop a regular reporting cycle

QI PROCESS

•Use data for decisions

Questions?