Date post: | 22-Jan-2018 |
Category: |
Government & Nonprofit |
Upload: | isaac-castillo |
View: | 423 times |
Download: | 0 times |
Program Evaluation and Measurement Outcomes: Can You Prove You’re Making a Difference?
DCPNI – Isaac Castillo - @Isaac_outcomes 1
Isaac D. CastilloDeputy Director
DC Promise Neighborhood Initiative@Isaac_outcomes
December 4, 2015
Why Should You Care About Outcomes?
December 2015 Isaac Castillo - @isaac_outcomes 2
• LAYC domestic violence story– LeapOfReason.org– First Do No Harm…Then Do More Good
• New domestic violence program component designed to teach three things: – Partner violence is not an OK expression of
love– Partner violence is not OK in Latino culture– There are safe ways to get out of violent
relationships
Learning Objectives
• Ability to define success for your organization and its programs/interventions
• Identify ways to measure progress and outcomes (using multiple approaches)
• Understand how to begin creating an organizational culture that embraces information and uses data to make programmatic improvements
December 2015 Isaac Castillo - @isaac_outcomes 3
Who Counts?
• May seem like a simple thing, but even defining who ‘counts’ as one of your participants is a very important decision.
December 2015 Isaac Castillo - @isaac_outcomes 4
Duplicated vs. Unduplicated Counts
• An unduplicated count is one that counts human beings – A person / human being would only be counted ONCE,
regardless of how many activities or sessions they have attended.
– Example: consider this class – what is the unduplicated count of this class?
• A duplicated count is one that counts ‘service slots’ – Not a count of human beings – rather it is a count of how
many service slots are occupied. – Example – if we all came back tomorrow – what would the
duplicated count be? Would the unduplicated count change?
December 2015 Isaac Castillo - @isaac_outcomes 5
Talking and thinking about your counts
• What do you present publically? A duplicated count or an unduplicated count?
• Do you ‘count’ someone who only comes to a single event at your location?
• Do you collect different things for duplicated or unduplicated populations?
December 2015 Isaac Castillo - @isaac_outcomes 6
Who Would You Fund?
• You have $1 million to provide funding to a tutoring program for “at-risk” youth.
• You need to pick one of two programs to fund – but you can only pick one!
• Assume everything else is equal aside from the information provided on the next page. – Same service population
– Same areas of service
– Same tutoring approach
December 2015 Isaac Castillo - @isaac_outcomes 7
Which Program Would You Fund?
December 2015 Isaac Castillo - @isaac_outcomes 8
Program # 1 Program # 2
Served 500 “at-risk” youth Served 50 “at-risk” youth
Provided 2,500 total hours of tutoring
Provided 2,500 total hours of tutoring
Each youth received average of 5 hours of
tutoring
Each youth received average of 50 hours of
tutoring
5 % of youth showed improved math grades on
report cards
90 % of youth showed improved math grades on
report cards
Does this Change Things?
December 2015 Isaac Castillo - @isaac_outcomes 9
Mat
h P
rofi
cie
ncy
Time
Program 1 Comparison Group
Program 1
Mat
h P
rofi
cie
ncy
Program 2 Comparison Group
Program 2
Time
If You Understand Dieting…You Understand Measurement
• What are some things you want to measure if you are dieting (how do you know your diet is ‘working’)?
• Why are you measuring these things?
• How do you measure them (and who measures them)?
• How often (when) do you measure them?
December 2015 Isaac Castillo - @isaac_outcomes 10
Same Concept: Different Words
What do you call this sandwich?
December 2015 Isaac Castillo - @isaac_outcomes 11
Outputs vs Outcomes
• Output measures assess what you do and who you serve
(Did you actually deliver programming and to whom?)
• Served 100 youth during summer camp
• Provided 2,250 hours of tutoring during the academic year
• 9 out of 10 youth attended at least 75 % of available art instruction classes offered
Outcome measures assess changes in knowledge, attitudes, behaviors or conditions for your service population.
• 75 % of youth increased their knowledge of local history during the summer camp
• 50% of youth increased math grades by one grade level during the academic year
• 25% fewer youth reported being involved in bullying over the last year
December 2015 Isaac Castillo - @isaac_outcomes 12
Outputs
• Outputs DO: – Tell you about whether your program was
implemented well. For example, they indicate whether a program:• delivered the intended number of sessions• reached its intended population• resulted in adequate participation levels
• Outputs DO NOT: – Tell you if participants benefited from your program– Serve as indicators of program success or
effectiveness
December 2015 Isaac Castillo - @isaac_outcomes 13
Outcomes
• Outcomes DO:
– Tell you if participants benefited from your program
– Serve as indicators of program success or effectiveness
• Outcomes DO NOT:
– Tell you about whether your program was implemented well (or provide clues about how your program improved participant outcomes)
December 2015 Isaac Castillo - @isaac_outcomes 14
Do You Need To Measure Outcomes for Every Program?
• No.
• You should measure outcomes for your activities that provide a consistent set of programming or services for participants.
• Measuring just outputs is OK for some activities or programmatic approaches.
• Activities or services that are only used once by people likely will NOT produce outcomes.
December 2015 Isaac Castillo - @isaac_outcomes 15
Need to Collect Both Outputs and Outcomes
• Determine if more services lead to more/better outcomes
• Determine if there is a ‘minimum’ amount of service needed to improve outcomes
• Determine if too many services actually lead to decreased or negative outcomes
December 2015 Isaac Castillo - @isaac_outcomes 16
Should You Measure Outcomes For These Activities?
• An organization provides a health fair for community members– No – this is a service that a person would access once. Unlikely
this will lead to any lasting change. – But you should still track outputs – how many attended?
• An organization provides a series of classes lasting 10 weeks that teaches how to lower cholesterol. – Yes – should measure changes in knowledge, attitudes, and
possibly behavior. – Outputs should also be tracked (attendance)
• You could potentially also track: – How many health fair attendees ended up enrolling in and
completing the classes.
December 2015 Isaac Castillo - @isaac_outcomes 17
Confirmatory Outcomes
• Confirmatory outcomes: outcomes you expect will change as a result of program participation– You will hold yourself responsible for confirmatory
outcomes for all (or the majority) of your regular participants.
– You will devote adequate resources and time to achieving confirmatory outcomes .
– You will measure your confirmatory outcomes
December 2015 Isaac Castillo - @isaac_outcomes 18
Exploratory Outcomes
• Exploratory outcomes : outcomes potentially, but not definitely, improved by your program– You think (have a theory) that you may have an effect
on exploratory outcomes.
– But you will not hold yourself responsible for exploratory outcomes.
– You may or may not measure exploratory outcomes.
– If you do not measure your exploratory outcomes, you should have a theory (or prior research) that supports why you think the exploratory outcomes are likely to happen.
December 2015 Isaac Castillo - @isaac_outcomes 19
Confirmatory vs Exploratory Outcomes Example
• A 24-week program designed to teach teens about the dangers of tobacco use and decrease (or prevent) smoking of cigarettes
• Confirmatory outcome: decrease in the self-reported number of cigarettes smoked during the past 7 days
• Exploratory outcome: decrease in the self-reported use of alcohol, marijuana, or other drugs during the past 7 days
The program SHOULD decrease use of cigarettes. And if the program decreases cigarette use, then participants MIGHT decrease use of other substances as well.
December 2015 Isaac Castillo - @isaac_outcomes 20
Some Examples of Outcomes
December 2015 Isaac Castillo - @isaac_outcomes 21
Healthy eating habits
Program Outcome(s)
Sex education for teens
Parent / child communication
skills
Consumption of more fruits and
vegetables
Healthier weight
Decreased yelling / anger during conflicts
Increased knowledge of
dangers of unprotected /
early sex
Decreased risky
sexual activity
Fewer unplanned
teenage pregnancies
Timing of Outcomes
• Best to think about sequence of outcomes – and not focus just on when outcomes may/should occur.
• Early / initial outcomes should lead to more outcomes, which in turn will lead to more outcomes.
• Important question: how far down the chain of outcomes will you measure?
December 2015 Isaac Castillo - @isaac_outcomes 22
Short Term Outcomes
• Expect to be affected in a short period of time
• Closely related to/influenced by the program’s outputs
• Necessary steps
– Ensure program is on track
– Realistic measure of success
December 2015 Isaac Castillo - @isaac_outcomes 23
Intermediate and Long-Term Outcomes
• Intermediate Outcomes– Link between short and longer-term outcomes
– Most will be confirmatory outcomes
• Long-Term Outcomes
– Change after longer period of time
– Ultimate outcomes
– Mix of confirmatory and exploratory outcomes
December 2015 Isaac Castillo - @isaac_outcomes 24
How Long Will Each Type of Outcome Take?
• Will depend on the length of your intervention.
December 2015 Isaac Castillo - @isaac_outcomes 25
Short Term Intermediate Long Term
Six MonthsOne Month Twelve Months
6 Years1 Year 12 Years
Performance Management and Evaluation
December 2015 Isaac Castillo - @isaac_outcomes 26
Performance Management Regular measurement,
Improve outcomes
EvaluationOccasional measurement,
Improve effectiveness
PM and Eval: Speed
Performance Management is rapid and frequent
December 2015 Isaac Castillo - @isaac_outcomes 27
Evaluation is methodical and delayed
PM and Eval: Data Collection and Analysis
December 2015 Isaac Castillo - @isaac_outcomes 28
Performance Management is simple and requires little
expertise
Evaluation can be complex and often requires specialized training
PM and Eval: Completeness
December 2015 Isaac Castillo - @isaac_outcomes 29
Performance Management gives you enough to guess at
whole puzzle
Evaluation lays out as much of the puzzle as possible
Why Bother With All of This?
December 2015 Isaac Castillo - @isaac_outcomes 30
Ultimately, you should be measuring outcomes or effectiveness for a
single reason:
To better serve your clients / population.
Isaac’s Contact Information
31December 2015 @Isaac_outcomes
Isaac D. Castillo
Deputy Director
DC Promise Neighborhood Initiative
On Twitter: @Isaac_outcomes
Email: [email protected]