Evaluation in the Field: Evaluation in the Field:

Putting Concepts into Putting Concepts into ActionAction

Janet Myers, PhD MPHJanet Myers, PhD MPHRichard Vezina, MPHRichard Vezina, MPHCAPS HIV Prevention ConferenceCAPS HIV Prevention ConferenceApril 21, 2006April 21, 2006

Overview

Warm-up Where does Evaluation fit? Approaches to Evaluation Examples Q&A: Evaluating Your Programs

Warm-Up

Who here is …

Evaluator?

Service Provider?

Administrator?

4 Questions:

What are the benefits of evaluating your programs?

What are the challenges to evaluating your programs?

What needs (besides $) do you have in order to plan/conduct evaluation?

What resources do you have/use for evaluation?

Where does Evaluation fit?

Mission

Goals

Objectives

Activities

Outcomes

Impact

Program & Process EvaluationIMPLEMENTATION

Evaluation

Program&

EvaluationPLANNING

Outcome

Mission

Provides the vision

How this work makes a difference in the world

Broadest scope

Goals (Ross & Mico, 1980; McKenzie & Smeltzer 2001)

“A future event toward which a committed endeavor is directed”

Simple & concise

2 basic components

“Who will be affected”

“What will change as a result of the program”

Objectives

Specific steps that contribute to a goal. Often several objectives per goal.

Good objectives are SMART:

SS – specific

MM – measurable

AA – attainable

RR – realistic

TT – time-bound

Good Objectives Show… (McKenzie & Smeltzer

2001)

What will change: Outcome that will be achieved

When will it change: Conditions under which the outcomes will be observed

How much change: Criteria for deciding whether the outcomes has been achieved

Who will change: Target population

Activities

Internal: administrative, etc. External: the services you provide

to clients Based on your goals/objectives

Outcomes Changes that occur in people being served

by your program

Attribution: To the best extent possible, show that change is a result of your program (but note…causality is difficult)

Standards are typically different for evaluation (vs. research)

To assess, you need at least 2 time points (pre- and post-) and/or a comparison group

Impact

The scope of the program’s effects, the duration of its outcomes and the extent of its influence on the broader context (for example, HIV incidence)

Attribution: Can be more challenging to show causality, because looking for more diffuse effects

Usually broad and long-term

Typically not in the scope of program evaluation

Approaches to Evaluation

Why do we evaluate?

To determine if objectives are being met

To improve quality of the program

To decide how to change content

To identify the effects of the program

Process vs. Outcome Evaluation Process

Demographics (Who’s being trained?) Reaction to content (“Smile Sheets”) Service units delivered

Outcome Changes in knowledge/attitudes/beliefs Changes in behavior Impact on patients/clients

Process Evaluation can help us…

Create better learning environment

Improve presentation skills

Show accountability

Reflect the target populations

Track service units

Outcome Evaluation can help us…

Show the program’s effects

Allow for comparisons over time

Provide specific guide points for improving programs

Show accountability

Planning Your Evaluation (1)

Figure out your questions: What will this be used for?

Consider your Resources Staffing Time Materials $$$

Choose Methods Quantitative: Surveys, pre/post tests, etc. Qualitative: Interviews, focus groups, etc.

Planning Your Evaluation (2)

Select Indicator(s) Relevant Measurable Improvable

Instrument/Tool Development Don’t reinvent the wheel!

Analysis: Get answers to your questions

Reporting: Formal & Informal

Examples