How to Focus and Measure Outcomes

Katherine Webb-MartinezCentral Coast and South Region

All Staff Conference April 23, 2008

Workshop Desired Outcomes

An understanding of how basic logic model techniques help define outcomes and focus program evaluation

An understanding of options for data collection methods to measure outcomes

Workshop Agenda

Introductions

Part 1: Logic Model

• Presentation – Logic Model helps with outcomes

• Activity – “Using a logic model to focus outcomes evaluation”

Part 2: Measuring Outcomes

• Presentation - Review of Data Collection Options

• Activity – Methods Café

Close

Part 1: Logic Model as Road Map

Where are you going?

How will you get there?

What will show that you’ve arrived?

“If you don’t know where you are going, how are you gonna’ know when you get there?”

Yogi Berra

University of Wisconsin-Extension, Program Development and Evaluation

HUNGRY

Feel betterFeel betterGet foodGet food Eat food Eat food

Everyday example

University of Wisconsin-Extension, Program Development and Evaluation

Logical chain of connections showing what the program is to accomplish

What we do

Who we reach

INPUTS OUTPUTS OUTCOMES

Program investments

Activities Participation Learning (Short-term)

Action (Medium

-term)

Impacts (Long-term)

What we invest

What results

University of Wisconsin-Extension, Program Development and Evaluation

Outputs vs. outcomes

Example: Number of patients discharged from state mental hospital is an output. Percentage of discharged who are capable of living independently is an outcome

Not how many worms the bird feeds its young,

but how well the fledgling flies (United Way

of America, 1999)

University of Wisconsin-Extension, Program Development and Evaluation

Social-economic-environmental impacts

What do you want to know?

Source: Bennett and Rockwell, 1995, Targeting Outcomes of Programs

Reactions

Learning

Actions

Number and characteristics of people reached; frequency and intensity of contact

Degree of satisfaction with program; level of interest; feelings toward activities, educational methods

Changes in knowledge, attitudes, skills, aspirations

Changes in behaviors and practices

Participation

University of Wisconsin-Extension, Program Development and Evaluation

Needs assessment:

What are the characteristics, needs, priorities of target population?

What are potential barriers/facilitators?

What is most appropriate to do?

Process evaluation:

How is program implemented?

Are activities delivered as intended?

Are participants being reached as intended?

What are participant reactions?

Outcome evaluation:

To what extent are desired changes occurring?

Who is benefiting / not benefiting? How?

What are unintended outcomes?

Impact evaluation:

What are the net effects?

What are final consequences?

To what extent can changes be attributed to the program?

University of Wisconsin-Extension, Program Development and Evaluation

ACTIVITY: PARENT EDUCATION PROGRAM

What do you (and others) want to know about the program?

Staff

Money

Partners

Assess parent ed programs

Design- deliver evidence-based program of 8 sessions

Parents increase knowledge of child dev

Parents better understanding their own parenting style

Parents use effective parenting practices

Improved child-parent relations

Research

INPUTS OUTPUTS OUTCOMES

Facilitate support groups

Parents gain skills in new ways to parent

Parents identify appropriate actions to takeParents

of 3-10 year olds

attend

Reduced stress

Parents gain confidence in their abilities

Strong families

Logic Model Reflection

How does a logic model help focus on what outcomes to measure and when to measure them?

How might you use a logic model in your own work?

Part 2: Measuring Outcomes

How will you answer your questions?

1. Decide on what evidence answers the questions

2. Determine sources of information

3. Choose data collection methods

University of Wisconsin-Extension, Program Development and Evaluation

Identify evidence

How will you know it when you see it?

What are the specific indicators that will be measured?

- Often expressed as # or %

- Outputs and outcomes indicators

- Quantitative or qualitative

University of Wisconsin-Extension, Program Development and Evaluation

Logic model with Indicators for Outputs and Outcomes

Program implemented

Targeted farmers

Farmers learn

Farmers practice new techniques

Farm profitability increases

Number of workshops held

Quality of workshops

Number and percent of farmers attending

Number and percent who increase knowledge

Number and percent who practice new techniques

Number and percent reporting increased profits; amount of increase

Outputs Outcomes

Remember:

“Not everything that counts can be counted.”

9 (Quantity) Happy (Quality) Kids

Quantitative: numbers, breadth, generalizability Qualitative: words, depth, specific

University of Wisconsin-Extension, Program Development and Evaluation

Possible Evaluation questions, indicators

Staff

Money

Partners

Parents increase knowledge of child dev

Parents better understand their own parenting style

Parents use effective parenting practices

Improved child-parent relations

Research Facilitate support groups

Parents gain skills in new ways to parent

Parents identify appropriate actions to take

To what extent is stress reduced?To what extent are relations improved?

To what extent did behaviorschange? For whom? Why? What else happened?

To what extent did knowledge and skills increase? For whom? Why? What else happened?

Who/how many attended/did not attend? Did they attend all sessions?Supports groups? Were they satisfied – why/why not?

How many sessions were held? How effectively?#, quality of support groups?

What amount of $ and time were invested?

Parents of 3-10 year olds

Deliver series of 8 interactive sessions

EVALUATION QUESTIONS

# Staff$ used# partners

# Sessions held

Quality criteria

INDICATORS

#,% attended per session

Certificate of completion

#,% demonstrating increased knowledge/skill

Additional outcomes

#,% demonstrating changes

Types of changes

#,% demonstrating improvements

Types of improvements

Develop parent ed curriculum

Reduced stress

Parents gain confidence in their abilities

Strong families

Determine Sources of Information

Program participants

Existing data• Program records, attendance logs, etc• Pictures, charts, maps, pictorial records

Others/Non-participants• Key informants • Funders • Collaborators• Etc.

University of Wisconsin-Extension, Program Development and Evaluation

Decide on Data Collection Methods

Survey Interview Test Observation Group techniques Portfolio review

Diaries, journals Case study Photography,

video Document review Expert or peer

review

University of Wisconsin-Extension, Program Development and Evaluation

Data collection plan template

Questions Indicators Data collection

Sources Methods Sample Timing

When choosing your methods consider:

Purpose

Participants

Resources Available

Some things to remember…

There is no one right method of collecting data

Each has a purpose, advantages and challenges

The goal is to obtain trustworthy, authentic and credible evidence

Often a mix of methods is preferred

Are the data reliable and valid?

Validity: Are you measuring what you think you’re measuring?

Reliability: If something was measured again using the same instrument, would it produce the same (or near the same) results?

University of Wisconsin-Extension, Program Development and Evaluation

Logic model and reporting

Methods Cafe

What approaches to measure outcomes work well for CE programs and why?

Is there anything you’ve tried that you would not recommend?

Methods Reflection

Given what we have discussed, what might you change or do differently with the methods that work for you?

Are there any other, new methods that might work for you and your clientele?