Program Evaluation: Can You Prove You Are Making a Difference?

transcript

Program Evaluation and Measurement Outcomes: Can You Prove You’re Making a Difference?

DCPNI – Isaac Castillo - @Isaac_outcomes 1

Isaac D. CastilloDeputy Director

DC Promise Neighborhood Initiative@Isaac_outcomes

Isaac.Castillo@dcpni.org

December 4, 2015

Why Should You Care About Outcomes?

December 2015 Isaac Castillo - @isaac_outcomes 2

• LAYC domestic violence story– LeapOfReason.org– First Do No Harm…Then Do More Good

• New domestic violence program component designed to teach three things: – Partner violence is not an OK expression of

love– Partner violence is not OK in Latino culture– There are safe ways to get out of violent

relationships

Learning Objectives

• Ability to define success for your organization and its programs/interventions

• Identify ways to measure progress and outcomes (using multiple approaches)

• Understand how to begin creating an organizational culture that embraces information and uses data to make programmatic improvements

Who Counts?

• May seem like a simple thing, but even defining who ‘counts’ as one of your participants is a very important decision.

Duplicated vs. Unduplicated Counts

• An unduplicated count is one that counts human beings – A person / human being would only be counted ONCE,

regardless of how many activities or sessions they have attended.

– Example: consider this class – what is the unduplicated count of this class?

• A duplicated count is one that counts ‘service slots’ – Not a count of human beings – rather it is a count of how

many service slots are occupied. – Example – if we all came back tomorrow – what would the

duplicated count be? Would the unduplicated count change?

Talking and thinking about your counts

• What do you present publically? A duplicated count or an unduplicated count?

• Do you ‘count’ someone who only comes to a single event at your location?

• Do you collect different things for duplicated or unduplicated populations?

Who Would You Fund?

• You have $1 million to provide funding to a tutoring program for “at-risk” youth.

• You need to pick one of two programs to fund – but you can only pick one!

• Assume everything else is equal aside from the information provided on the next page. – Same service population

– Same areas of service

– Same tutoring approach

Which Program Would You Fund?

Program # 1 Program # 2

Served 500 “at-risk” youth Served 50 “at-risk” youth

Provided 2,500 total hours of tutoring

Each youth received average of 5 hours of

tutoring

Each youth received average of 50 hours of

tutoring

5 % of youth showed improved math grades on

report cards

90 % of youth showed improved math grades on

report cards

Does this Change Things?

Program 1 Comparison Group

Program 1

Program 2 Comparison Group

Program 2

If You Understand Dieting…You Understand Measurement

• What are some things you want to measure if you are dieting (how do you know your diet is ‘working’)?

• Why are you measuring these things?

• How do you measure them (and who measures them)?

• How often (when) do you measure them?

Same Concept: Different Words

What do you call this sandwich?

Outputs vs Outcomes

• Output measures assess what you do and who you serve

(Did you actually deliver programming and to whom?)

• Served 100 youth during summer camp

• Provided 2,250 hours of tutoring during the academic year

• 9 out of 10 youth attended at least 75 % of available art instruction classes offered

Outcome measures assess changes in knowledge, attitudes, behaviors or conditions for your service population.

• 75 % of youth increased their knowledge of local history during the summer camp

• 50% of youth increased math grades by one grade level during the academic year

• 25% fewer youth reported being involved in bullying over the last year

Outputs

• Outputs DO: – Tell you about whether your program was

implemented well. For example, they indicate whether a program:• delivered the intended number of sessions• reached its intended population• resulted in adequate participation levels

• Outputs DO NOT: – Tell you if participants benefited from your program– Serve as indicators of program success or

effectiveness

Outcomes

• Outcomes DO:

– Tell you if participants benefited from your program

– Serve as indicators of program success or effectiveness

• Outcomes DO NOT:

– Tell you about whether your program was implemented well (or provide clues about how your program improved participant outcomes)

Do You Need To Measure Outcomes for Every Program?

• No.

• You should measure outcomes for your activities that provide a consistent set of programming or services for participants.

• Measuring just outputs is OK for some activities or programmatic approaches.

• Activities or services that are only used once by people likely will NOT produce outcomes.

Need to Collect Both Outputs and Outcomes

• Determine if more services lead to more/better outcomes

• Determine if there is a ‘minimum’ amount of service needed to improve outcomes

• Determine if too many services actually lead to decreased or negative outcomes

Should You Measure Outcomes For These Activities?

• An organization provides a health fair for community members– No – this is a service that a person would access once. Unlikely

this will lead to any lasting change. – But you should still track outputs – how many attended?

• An organization provides a series of classes lasting 10 weeks that teaches how to lower cholesterol. – Yes – should measure changes in knowledge, attitudes, and

possibly behavior. – Outputs should also be tracked (attendance)

• You could potentially also track: – How many health fair attendees ended up enrolling in and

completing the classes.

Confirmatory Outcomes

• Confirmatory outcomes: outcomes you expect will change as a result of program participation– You will hold yourself responsible for confirmatory

outcomes for all (or the majority) of your regular participants.

– You will devote adequate resources and time to achieving confirmatory outcomes .

– You will measure your confirmatory outcomes

Exploratory Outcomes

• Exploratory outcomes : outcomes potentially, but not definitely, improved by your program– You think (have a theory) that you may have an effect

on exploratory outcomes.

– But you will not hold yourself responsible for exploratory outcomes.

– You may or may not measure exploratory outcomes.

– If you do not measure your exploratory outcomes, you should have a theory (or prior research) that supports why you think the exploratory outcomes are likely to happen.

Confirmatory vs Exploratory Outcomes Example

• A 24-week program designed to teach teens about the dangers of tobacco use and decrease (or prevent) smoking of cigarettes

• Confirmatory outcome: decrease in the self-reported number of cigarettes smoked during the past 7 days

• Exploratory outcome: decrease in the self-reported use of alcohol, marijuana, or other drugs during the past 7 days

The program SHOULD decrease use of cigarettes. And if the program decreases cigarette use, then participants MIGHT decrease use of other substances as well.

Some Examples of Outcomes

Healthy eating habits

Program Outcome(s)

Sex education for teens

Parent / child communication

skills

Consumption of more fruits and

vegetables

Healthier weight

Decreased yelling / anger during conflicts

Increased knowledge of

dangers of unprotected /

early sex

Decreased risky

sexual activity

Fewer unplanned

teenage pregnancies

Timing of Outcomes

• Best to think about sequence of outcomes – and not focus just on when outcomes may/should occur.

• Early / initial outcomes should lead to more outcomes, which in turn will lead to more outcomes.

• Important question: how far down the chain of outcomes will you measure?

Short Term Outcomes

• Expect to be affected in a short period of time

• Closely related to/influenced by the program’s outputs

• Necessary steps

– Ensure program is on track

– Realistic measure of success

Intermediate and Long-Term Outcomes

• Intermediate Outcomes– Link between short and longer-term outcomes

– Most will be confirmatory outcomes

• Long-Term Outcomes

– Change after longer period of time

– Ultimate outcomes

– Mix of confirmatory and exploratory outcomes

How Long Will Each Type of Outcome Take?

• Will depend on the length of your intervention.

Short Term Intermediate Long Term

Six MonthsOne Month Twelve Months

6 Years1 Year 12 Years

Performance Management and Evaluation

Performance Management Regular measurement,

Improve outcomes

EvaluationOccasional measurement,

Improve effectiveness

PM and Eval: Speed

Performance Management is rapid and frequent

Evaluation is methodical and delayed

PM and Eval: Data Collection and Analysis

Performance Management is simple and requires little

expertise

Evaluation can be complex and often requires specialized training

PM and Eval: Completeness

Performance Management gives you enough to guess at

whole puzzle

Evaluation lays out as much of the puzzle as possible

Why Bother With All of This?

Ultimately, you should be measuring outcomes or effectiveness for a

single reason:

To better serve your clients / population.

Isaac’s Contact Information

31December 2015 @Isaac_outcomes

Isaac D. Castillo

Deputy Director

DC Promise Neighborhood Initiative

On Twitter: @Isaac_outcomes

Email: Isaac.Castillo@dcpni.org

Program Evaluation: Can You Prove You Are Making a Difference?

Government & Nonprofit