Measuring Program Outcomes and Impacts
Ayanava MajumdarExtension Entomologist, ACES
Gulf Coast Research and Extension Center8300 State Hwy 104, Fairhope, AL 36532
Email: [email protected] phone: 251-331-8416
Fax: 251-990-8912
Objectives• Why measure outcomes and impacts?• Discuss timing of evaluations (Logic model)• Discuss 12 major evaluation techniques• How you can make evaluations useful?• Provide critical sources of information
This presentation contains information unique to the presenter and should be taken as suggestions. You may modify content of this presentation to suit your situation.
Few critical sources of information
• Developing a Logic Model (Taylor-Powell & Renner, 2000, UWEX)
• Logic Model Development Guide (W.K. Kellogg Foundation, 1998)
• Kirkpatrick’s Four Levels of Evaluation (1994), Encyclopedia of Education
Conventional measurements (old-school evaluations)
• Participant reaction: usefulness of program
• Teaching & facilitation: suggestions for improvement
• Outcomes: what did you learn today?
• Future programming: what do want to learn more about?
Taylor-Powell & Renner, 2000
…focus on outputs and immediate effects (learning)
…no information about action and conditions
…in effect, this was the push strategy (linear TOT)
Welcome to the Accountability Era!
What gets measured gets done!
If you don’t measure results, you can’t differentiate success from failure.
If you demonstrate results, you can win public support.
Osborne & Gaebler, 1992
Accountability & evolution of concepts
Outputs. The activities, products, and participation generated through the investment ofresources. Goods and services delivered.
Outcomes. Results or changes from the program such as changes in knowledge, awareness,skills, attitudes, opinions, aspirations, behavior, practice, decision-making,policies, social action, condition, or status.
Impact. The social, economic, civic and/or environmental consequences of the program. Impacts tend to be longer-term and so may be equated with goals.
Taylor-Powell & Renner, 2000
Revisiting the LOGIC MODEL
Remember…• It is not a theory, it is not a reality• It is only a MODEL…a framework for visualizing
relationshipsTaylor-Powell & Renner, 2000
Measuring success is complicated!
• Measuring outcomes, by itself, will need resources!• Assumptions and external factors create variations in
outcomes.
Types of evaluationFour basic types:1. Needs assessment2. Process evaluation3. Outcome evaluation4. Impact evaluation
When to conduct? • First thing we should be doing!• Establishes priorities
What questions to ask?• Characteristics of audience• Needs of audience (prioritize)• Where do they find information?• Best learning method• Find barriers to knowledge adoption
Don’t forget your camera, writing instruments, survey print & give time to respond
Types of evaluation contd.Four basic types:1. Needs assessment2. Process evaluation3. Outcome evaluation4. Impact evaluation When to conduct?
• During program implementation• E.g., quality survey, satisfaction survey, future needs survey
What questions to ask?• Were you satisfied with delivery methods?• Was there too much information?• Are you reaching to targeted audience?
Types of evaluation contd.Four basic types:1. Needs assessment2. Process evaluation3. Outcome evaluation4. Impact evaluation
When to conduct? • Measure learning• During on-site programs: workshops, field days, etc.
What questions to ask?• Short-term change: key words in questions “awareness”, “knowledge”, “opinion”, “motivation”• Document who is not benefiting (analyze sample and understand biases)
Part 1
Types of evaluation contd.Four basic types:1. Needs assessment2. Process evaluation3. Outcome evaluation4. Impact evaluation
When to conduct? • Measure behavioral changes• During one-to-one visits, farm visits, telephonic, mail, email…repeat surveys!
What questions to ask?• Medium-term changes: key words in questions “behavior”, “practices”, “decision”, “action”• Are you meeting goals? Unintended outcomes?
Part 2
Types of evaluation contd.Four basic types:1. Needs assessment2. Process evaluation3. Outcome evaluation4. Impact evaluation
When to conduct? • You should have partially achieved this if you did previous steps right.
What questions to ask?• Long-term changes: change in “condition” • Separate real impact from “background noise”• Try to document final consequences: new products, innovations, services, community changes, motivation to act in the absence of program
Determine some INDICATORS
Relatively EASY
Relatively DIFFICULT
Indicators: can be qualitative or quantitative
Evaluation techniques• Survey: collect standardized
information, may be mailed, done on-site, structured interviews (N, P)
• Case study: in-depth examination of particular groups or individuals (O, I)
• Interviews: face-to-face interaction, conversational, one-on-one or small groups (P, I)
• Observation: collecting information via seeing or listening, structured or unstructured (I)
Tayl
or-P
owel
l, 20
02
N = Needs asses., P = Process eval., O = Outcome eval., I = Impact eval.
Evaluation techniques (contd.)• Group assessment: use of
nominal techniques like focus groups, brainstorming, community forum (N)
• Expert/peer review: examination by a review committee, Delphi method (“indicator”, I)
• Portfolio reviews: collection and presentation of materials and samples of work that indicate breadth of program (“indicator”)
• Testimonials: individual statements by people indicating personal reactions, household drop-off (O, I) Ta
ylor
-Pow
ell,
2002
N = Needs asses., P = Process eval., O = Outcome eval., I = Impact eval.
Evaluation techniques (contd.)
• Tests: assess knowledge, skills, performance, e.g., pre-test & post-test (P, O)
• Photos, videos: group or one-on-one interviews ( I)
• Success or problem stories: narrative account by participants about adoption of new practices(“indicator”, N, I)
• Unobtrusive methods: gathering information without making participants aware, e.g., indirect measures, content analysis (N, P, O)
Tayl
or-P
owel
l, 20
02
N = Needs asses., P = Process eval., O = Outcome eval., I = Impact eval.
Four evaluation criteria• Utility:
– Goal: how useful is your program evaluation to you & your audience?
– Know the following information: • State purpose clearly• Consider your audience• Communicate findings & relevance of findings
Boyd, 2002
• Feasibility: – Goal: how practical is your assessment
technique?– Know the following information:
• Keep evaluation practical, nondisruptive• Calculate cost: benefit• Use appropriate evaluation technique/s
Boyd, 2002
Byod’s evaluation criteria contd.
• Appropriateness: – Goal: how appropriate is your program evaluation for
those involved?– Know the following information:
• Respect people and their rights• Use appropriate choice statements• Disclose findings properly
Boyd, 2002
Byod’s evaluation criteria contd.
Byod’s evaluation criteria contd.• Accuracy:
– Goal: how accurate is your program evaluation to you & your audience?
– Know the following information: • Design repeatable surveys• Use appropriate analyses • Draw justifiable conclusions
Boyd, 2002
Final tips on program evaluations• Consult specialist in planning phase• Think backwards in LOGIC model (impact >>
output >> input) & allocate resources• Think about “indicators” of success• If you conduct surveys, allocate time to respond
(don’t rush)• Publicize your programs, use multiple channels
>> create a “pull” system >> more success
Thank you for patient listening!
QUESTIONS FOR AYANAVA?