1 Integrating Program Integrating Program Planning and Evaluation Using Logic Models B By: Thomas J. Chapel, MA, MBA Chief Evaluation Officer CDC [email protected]404-639-2116 Objectives: Be able to… State CDC Evaluation Framework steps and standards State the central role of program description in evaluation and planning Describe the basic components of a logic model 2 Describe the added value of elaborating logic models State how logic models help guide questions in program planning and evaluation
Transcript
1
Integrating ProgramIntegrating Program Planning and Evaluation Using Logic Models
You Don’t Ever Need a Logic Model, BUT, You Always Need a Program Description
Don’t jump into planning or evaluation without clarity on:
The big “need” your program is to addressThe key target group(s) who need to take actionThe kinds of actions they need to take (your intended outcomes or objectives)
10
outcomes or objectives) Activities needed to meet those outcomes “Causal” relationships between activities and outcomes
6
Logic Models and Program Description
L i M d lLogic ModelsGraphic depictions of the relationship between your program’s activities and its
11
p gintended effects
“Complete” Logic Model
IntermediateShort term Long-term
Activities Inputs Outputs Intermediate Outcomes
Short-term Outcomes
gOutcomes/
Impacts
What the program does…
Who or what will changebecause of the program…
What the program
1212
p gneeds…
Context and Assumptions
External factors that influence getting to outcomes
7
What the program and its staff actually do
Activities Inputs Outputs
Intermediate Effects/
Outcomes
Short-term
Effects/ Outcomes
Long-term Effects/
Outcomes
13
ContextAssumptions
Results of activities:Who/what will change?
IntermediateShort Long term
Activities Inputs Outputs
Intermediate Effects/
Outcomes
Short-term
Effects/ Outcomes
Long-term Effects/
Outcomes
14
ContextAssumptions
8
Finding Activities and Outcomes
Actions/n
GoalsObjectivesActions/Tactics
Activities ST or MT Outcomes
LT Outcomes or Impacts
Pla
Eval
15
p
Process MeasuresProgress Measures
Impl. Measures
Outcome MeasuresImpact Measures
Key Performance IndicatorsSuccess Factors
PM
Goal 3: Disseminate information to guide gpolicy, practice, and
other actions to improve the nation’s
health
9
Finding Activities and OutcomesOffice of Workforce and Career Development
(OWCD) Mission
To improve health outcomes by developing a competent, sustainable and diverse public health workforce
through evidence-based training, career and leadership development,
17
p p ,and strategic workforce planning.
17
Inputs Activities Outcomes
Implicit Logic Model
Conduct trainingDo career leadership
developmentCompetent, sustainable,
diverse workforce
Improved health
outcomesDo Strategic workforce
Evidence Base
18
planning
10
Process Use
When influence/program improvement comes not from findings of an evaluationcomes not from findings of an evaluation, but from insights gleaned during the tasks involved in doing an evaluation
Process Use in Theory, Research, and Practice:New Directions for Evaluation, No. 116, 2008
19
Underlying Logic of CPPW
Activities and
Supports
Changes in
• Policies• Systems• Environments
Changes in
• Physical activity• Nutrition• Tobacco use• Exposure to tobacco smoke
“Nested” Logic Models—Logic Models at Different Levels of the Organization/Effort
Enterprise Level
Division/PartnerLevel
Each logic model is built with reference to the levels above and below…
23
Division/PartnerLevel
Project Level
Integrating Program Planning and Evaluation
General Underlying Logic ofGeneral Underlying Logic of FOA
13
Constructing Logic Models: Identify Activities and Outcomes by….
1. Examining program descriptions,1. Examining program descriptions, MISSIONS, VISIONS, PLANS, etc. and extracting these from the narrative, OR
2. Reverse mapping—Starting with outcomes, ask “how to” in order to generate the activities which produce them, OR
25
p ,3. Forward mapping—Starting with activities,
ask “so what” in order to generate the outcomes that are expected to result
Then…Do Some Sequencing…
Divide the activities into 2 or more l b d th i l i lcolumns based on their logical
sequence. Which activities have to occur before other activities can occur?
26
Do same with the outcomes. Which outcomes have to occur before other outcomes can occur?
14
Infrastructure
Year 1 Years 2-4 Project End
Increased skill-sets of SHD
Implemented strategic plans
Increased diversity of IVP funding and
• Formative •Formative
and resources
•Summative
15
Evaluation
Year 1 Years 2-4 Project End
Established evaluation & monitoring outcomes
Self-evaluation of activities
Achievement of 4 & 5 year goals
• Formative •Formative•Summative•Health Impact
Strategies
Year 1 Years 2-4 Project End
Established MOUs for policy & intervention
Implemented 4 priority topic t t i
Evidence influenced strategies
intervention plans
• Formative
strategies
•Formative•Summative•Health Impact
16
Collaboration
Year 1 Years 2-4 Project End
Increased public awareness of injury &
Enhanced partnerships
Increased political will for IVP issuesviolence
• Formative •Formative
issues
•Summative
Surveillance
Year 1 Years 2-4 Project End
Implemented surveillance standards
Strengthened injury & violence surveillance
Effective use of state injury & violence data
• Formative
surveillance
•Formative
data
•Summative
17
For Planning and Evaluation “Causal” Arrows Can Help
Not a different logic model, but same elements in different formatArrows can go from:
Activities to other activities Which activities feed which other activities?
Activities to outcomes
33
Activities to outcomesWhich activities produce which intended outcomes?
Early effects/outcomes to later onesWhich early outcomes produce which later outcomes
18
Elaborating Your Program Logic
Common TermsMediatorsOutputsInputsModerators
BUT..
35
Not all models need to have (all of) themIf you use them, use them well
Integrating Program Planning and Evaluation
Elaborating Our ProgramElaborating Our Program “Theory”—Mediators
19
Filling in the Blanks….
37 37
Inputs Activities Outcomes
Implicit Logic Model
Conduct trainingDo career leadership
development
Competent, sustainable,
diverse workforce
Improved health
outcomesDo Strategic workforce planning
Evidence Base
38
planning
20
Implicit Logic Model
Inputs Activities Outcomes
Conduct trainingDo career leadership
development
Workforceis
CompetentSustainable
Diverse
Improved health
outcomesDo Strategic
kf
Evidence Base
p
39
workforce planning
ConductWorkforce
is:
Inputs Activities Outcomes
Implicit Logic Model—Mediators
Conduct training
Do career leadership
development
is:
Competent
SustainableImproved
health outcomes
Do Strategic workforce
Evidence Base
Programs are more effective
Continuity in r’ships
and approach
40
Diverseworkforce planning Clients
access and adhere
21
Infrastructure
Year 1 Years 2-4 Project End
Increased skill-sets of SHD
Implemented strategic plans
Increased diversity of IVP funding and
• Formative •Formative
and resources
•Summative
Evaluation
Year 1 Years 2-4 Project End
Established evaluation & monitoring outcomes
Self-evaluation of activities
Achievement of 4 & 5 year goals
• Formative •Formative•Summative•Health Impact
22
Integrating Program Planning and Evaluation
Elaborating Our ProgramElaborating Our Program “Theory”—Outputs, Inputs, and Moderators
Tangible products of activities
Activities Inputs Outputs
Intermediate Effects/
Outcomes
Short-term
Effects/ Outcomes
Long-term Effects/
Outcomes
44
AssumptionsContext/External Factors
23
Upgrading Your Outputs—How Logic Models Help
OutreachOutreach
Behavior
Screening
Train in Self-
45
ChangeImproved Health Outcomes
ID People with Condition
Management
Refer for Medical Treatment
Medical Management
Traditional Outputs—Typical Screening Program
Screening: Pool (#) of screenedScreening: Pool (#) of screened kidsReferrals: (#) referrals to medical treatmentTraining: Pool (#) of families trained
46
Training: Pool (#) of families trained
24
The Plot Thickens
Screening soScreening so good it will lead to…
Behavior Change
ID People with Condition
Training in Self-Management so good it will lead
4747
to…
Referral for Medical Treatment so good it will lead to…
Quality Medical Management
“Upgraded” Outputs: More than Simple Counts
Screening: Pool (#) of screened kids (meeting likely risk profile)Training: Pool (#) of clients trained using culturally-competent curriculum and with appropriate supports Referrals: Pool(#) of referrals to
4848
Referrals: Pool(#) of referrals to (qualified or willing) medical treatment providers
25
Inputs and Moderators
I h f “killIn search of “killer assumptions”
50
26
Resource “platform” for the program
Activities Inputs Outputs
Intermediate Effects/
Outcomes
Short-term
Effects/ Outcomes
Long-term Effects/
Outcomes
51
Context/External Factors
Moderators: Contextual factors th t ill f ilit t
Activities Inputs Outputs
Intermediate Effects/
Outcomes
Short-term
Effects/ Outcomes
Long-term Effects/
Outcomes
that will facilitate or hinder getting our outcomes
52
Context/External Factors
27
Understanding Our Program Logic—How Inputs and Moderators Help
From this
IF THEN
From this…
To this…IF
AND
THEN
Basic CPPW Logic Model—Where Inputs and Moderators Live
Activities and
Supports
Changes in
• Policies• Systems• Environments
Changes in
• Physical activity• Nutrition• Tobacco use• Exposure to tobacco smoke
Reductions in
• Obesity• Smoking rates• Morbidity• Mortalitynp
uts
tobacco smoke• Use of quit lines
MortalityI
Moderators
28
Sample Inputs—Screening Program
FundsTrained staffLegal authority to screenRelationships for med treatment
55
29
Moderators—Four Types
P liti lPoliticalEconomicSocialTechnological
57
Technological
Inputs and Moderators—What Do We Learn
ProspectivelyRedesignWork-arounds
Retrospectively
58
Run toward the killer assumption
30
Note!
Logic Models make the program theory clear,
not true!
59
Integrating Program Planning and Evaluation
Putting Your Logic Model toPutting Your Logic Model to Use
31
Framework forProgram Evaluation
“Program description” stage frames decisions about evaluation focus
61
Step 1. Which Stakeholders Matter Most?
Who is:Affected by the program?Involved in program operations?Intended users of evaluation findings?
Of these, who do we most need to:Enhance credibility?Implement program changes?Advocate for changes?Fund, authorize, expand program?
32
Step 3. Key Domains in Eval FocusImplementation (Process)
Is program in place as intended? p g p
Effectiveness (Outcome)Is program achieving its intended short-, mid, and/or long-term effects/outcomes?
Efficiency yHow much “product” is produced for given level of inputs/resources?
Causal Attribution Is progress on outcomes due to your program?
Utility Questions
Purpose: Toward what end is thePurpose: Toward what end is the evaluation being conducted?User: Who wants the info and what are they interested in? U H ill th th i f ?Use: How will they use the info?
33
(Some) Potential Purposes/Uses
Show accountabilityShow accountabilityTest program implementation“Continuous” program improvementIncrease the knowledge baseOther…Other…
(Some) Potential Purposes/Uses
Show accountabilityShow accountabilityTest program implementation“Continuous” program improvementIncrease the knowledge baseOther…Other…
34
Logic Models and Strategy and Planning
By Clarifying Sequence of Activities y y g qOutcomes, Helps With…
Identifying/refining missionDeveloping consensus on key intended outcomes—the “staked claim”Defining goals and objectives
67
Defining goals and objectives, strategies and actions
Key Take-Aways1. It’s never about the model, it’s about
“understanding your program”2. You need this for “organizational learning”
and not just “accountability”3. Most/best benefits may be process use
benefits4. You know lots about your program even
68
before you draw your model5. A little bit of logic modeling goes a long
way. There’s a trade-off between “accuracy” and “utility”
35
Where Next….
Identify evaluation questionsIdentify evaluation questionsDefine indicators and data sources for questionsAnalyze dataDraw conclusions and resultsTurn results into action
But…
Later Steps Informed by WorkLater Steps Informed by Work of Earliest Steps….
36
Helpful Publications @ www.cdc.gov/eval
71 71
Helpful ResourcesNEW! Intro to Program Evaluation for PH Programs—A Self-Study Guide: http://www.cdc.gov/eval/whatsnew.htmLogic Model Sites
Innovation Network: http://www.innonet.org/Harvard Family Research Project: http://www.gse.harvard.edu/hfrp/University of Wisconsin-Extension: http://www.uwex.edu/ces/lmcourse/CDC/DASH:http://www.cdc.gov/healthyyouth/evaluation/resources.htm#4CDC/STD: http://www.cdc.gov/std/program/progeval/TOC-PGprogeval.htm
TextsKellogg Foundation Logic Model Development Guide:
72
Kellogg Foundation Logic Model Development Guide: www.wkkf.orgW.K. Kellogg Foundation Evaluation Resources: http://www.wkkf.org/programming/overview.aspx?CID=281Rogers et al. Program Theory in Evaluation. New Directions Series: Jossey-Bass, Fall 2000Chen, H. Theory-Driven Evaluations. Sage. 1990
