155 College Street

Room 400

The Banting Institute

University of Toronto

Toronto, Ontario

M5T 3M7

Tel (416) 978-0522

Fax (416) 971-1365

E-mail: hc.unit@utoronto.ca

www.thcu.ca

SMARTRISK Learning Series

August 26, 2008

Robb MacDonaldr.macdonald@thcu.ca

705.321.0410

Introduction to Evaluating Health

Promotion Programs

Overview

• Technology • Phone muting• Posing a question

• Process – step, case, questions• Definitions and Core concepts • THCU’s 10-Steps• Resources

Core concepts

Definitions (1)

• Program – any group of related activities carried out to achieve a specific outcome or result

• Example: To promote sexual health among adolescents = brochures, presentations, sexual health clinics, condom distribution…

Definitions (2)

• Program Evaluation – “The systematic gathering, analysis and reporting of information to assist in decision-making.”

Ontario Ministry of Health, Public Health Branch (1996)

Types of evaluation

• Formative

• Process

• Outcome/Summative

Formative evaluation

• Assesses process of developing a program

• Helps to ensure that programs are developed in accordance with stakeholder needs

• Identifies how/why key decisions were made

Process evaluation

• Assesses the procedures and tasks involved in implementing a program (“what’s happening?”)

• Sometimes known as program tracking or monitoring

Outcome evaluation

• Assesses extent to which program achieved its intended purpose (I.e., did desired change take place?)

• In health promotion, outcome evaluations usually tied to achievement of program objectives

Components of outcome evaluation

• Changes in awareness• Changes in knowledge• Changes in attitudes• Changes in behaviours• Changes in policy• Changes in social/physical environment• Changes in morbidity/mortality rates• Cost effectiveness/cost benefit analysis

Summative evaluation

• Attempts to determine value of program

• Includes measures of short and long term outcomes

• Includes process considerations.

THCU’s Steps in Evaluation Process

Step 1: Clarify your program

Step 2: Engage stakeholders

Step 3: Assess resources

Step 4: Design the evaluation

Step 5: Determine methods of measurement

Step 6: Develop workplan

Step 7: Data collection

Step 8: Data analysis

Step 9: Interpretation and dissemination

Step 10: Take action

Step One

• Clarify your program

Pre-requisites for evaluation

1. Clearly defined goals and objectives: both process and outcome

2. Identified population(s) of interest (aka program participants or recipients)

3. Well defined activities implemented in a prescribed manner

4. Plausible linkages between objectives and activities

5. Clearly specified indicators tied to objectives and activities

6. Resources to conduct evaluation (time, money, person-power, technical expertise, equipment)

Program Goal:

• Statement summarizing ultimate direction or purpose of program (aka purpose, mission)

Examples:

• To foster a school environment that enables students to make healthy choices (positive outcome goal).

• To reduce the incidence of alcohol-related harm in Community X (problem reduction goal).

Elements of a well-written outcome objective

Outcome(what)

Priority Population

(who)

Conditions(when)

Criterion (how much)

A well-written

objective

Examples of outcome objectives in four components

Criterion/How Much

Priority Population/Who

Outcome/What Conditions/When

Trip the number of

children and youth

involved in the development, implementation and evaluation of student nutrition programs

By the end of 2008

Double the number of

schools in the region

with functioning SNPs in one year

Increase by 25% the number of

schools in the region

meeting MCYS nutrition guidelines and public health food safety guidelines

in two years

Increase to 70%

elected officials in the region

aware of the benefits of healthy nutrition for children and youth and the role of the CNN in acting on the issue

by the end of the fiscal

Maintain provincial government

financial support for the CNN

for five years.

Well-defined, logical activities

1. Process/activity objective Example: “To implement peer-led substance abuse prevention programs at all area high schools by September 2004 .”

2. Prescribed, well-understood processes3. Plausible linkages between activities

and outcome objectives

Indicators

• Variable that can be measured in some way (sign that something happened)

• Used as measures to assess extent to which program objectives have been met

Steps in Evaluation Process

Step 1: Clarify your program

Step 2: Engage stakeholders

Step 3: Assess resources

Step 4: Design the evaluation

Step 5: Determine methods of measurement

Step 6: Develop workplan

Step 7: Data collection

Step 8: Data analysis

Step 9: Interpretation and dissemination

Step 10: Take action

This step involves:

• Defining who your stakeholders are• Understanding stakeholder interests

and expectations• Engaging stakeholders• Developing evaluation questions

Levels of stakeholders

Who are your stakeholders?

• Clients• Staff• Managers• Board members• Community partners• Funders

Understanding stakeholder interests

• Consider different types of interests

• stakeholders of the program

• stakeholders of the evaluation

• What do they want to know from the evaluation?

• How do they expect the evaluation to be conducted?

• May need to prioritize stakeholder interests due to budget limitations

Utilization-focused Evaluation

• Focus from beginning on primary intended users and specific intended uses of evaluation results

• Involves stakeholders from Day 1 in decision making re: ‘their’ evaluation

• Best model for stakeholder participation• Produces extremely useful results• Easy for internal evaluators, even without

much experience• Source: Evaluation 101 AOD Group, Dec. 19,

2007

Steps in Evaluation Process

Step 1: Clarify your program

Step 2: Engage stakeholders

Step 3: Assess resources

Step 4: Design the evaluation

Step 5: Determine methods of measurement

Step 6: Develop workplan

Step 7: Data collection

Step 8: Data analysis

Step 9: Interpretation and dissemination

Step 10: Take action

Budget $$$$

Staff availability

special skills of staff

interest in project

interest in learning new skills

Support of partner organizations

Equipment availability photocopier phones computers and software space

Volunteer availability

Time available before you need results

Step 3: Assess Resources

Resources for evaluation

• As a general rule, the World Health Organization (WHO) recommends that at least ten percent of a total program budget should be allocated to evaluation

Steps in Evaluation Process

Step 1: Clarify your program

Step 2: Engage stakeholders

Step 3: Assess resources

Step 4: Design the evaluation

Step 5: Determine methods of measurement

Step 6: Develop workplan

Step 7: Data collection

Step 8: Data analysis

Step 9: Interpretation and dissemination

Step 10: Take action

Step 4: Design Your Evaluation

• Select the type of evaluation to be conducted

• What are your stakeholders’ evaluation questions?

• What is your program’s stage of development?

• What evaluations have already been done?

• What resources do you have available?

• Design the evaluation approach

Step 4: Design Your Evaluation

• What is your program’s stage of development?

• Development

• Implementation

• Up and running

• Sun setting (winding down)

• Completed

• Restarting

Step 4: Design Your Evaluation

• Formative (development or restarting a program)

• Process (during first two years of implementation)

• Summative/Outcome (after program has been operating for a few years)

Step 4: Design Your Evaluation

• Descriptive

• one time assessment; look at relationships

• most commonly used (cost)

• used for all types

• do not establish cause and effect

• case study; pre-testing; community needs; audience analysis

• Analytical

• comparison of groups

• quasi-experimental; true experiments

Steps in Evaluation Process

Step 1: Clarify your program

Step 2: Engage stakeholders

Step 3: Assess resources

Step 4: Design the evaluation

Step 5: Determine methods of measurement

Step 6: Develop workplan

Step 7: Data collection

Step 8: Data analysis

Step 9: Interpretation and dissemination

Step 10: Take action

Quantitative vs. Qualitative Evaluation

• Quantitative – application of numerical (statistical) data collection and analysis methods

• Qualitative – application of more in-depth, open-ended data collection and analysis methods

• Both methods are necessary to fully understand and appreciate the impact of health promotion programs

Your Evaluation Toolbox

• The various data collection methods are like tools. No tool is “better” or “worse” than any other. Each tool has a different purpose.

• Like tools, data collection methods are problematic only when used for the wrong purpose.

Steps in Evaluation Process

Step 1: Clarify your program

Step 2: Engage stakeholders

Step 3: Assess resources

Step 4: Design the evaluation

Step 5: Determine methods of measurement

Step 6: Develop workplan

Step 7: Data collection

Step 8: Data analysis

Step 9: Interpretation and dissemination

Step 10: Take action

Take Action

• Involve your stakeholders in interpreting and taking action on your results

• Revisit your original goals of data collection. • Write a list of recommended actions• Prioritize those changes which are most

important and most feasible• Set up an action plan to implement changes• Implement!

THCU Services and Resources

THCU’s Consultation Service

• Free to those working on Ontario-focused projects.• Scope varies, depending on need:

• short training sessions; • brief, one-time advice; • review your work or product;• hands-on assistance working through our step models; • links to other sources of information and resources.

• Consultation request form http://www.thcu.ca/consultation/request_form.htm

• Sample consultations http://www.thcu.ca/consultation.htm

Upon Request Workshops

• All of our workshops, are available upon request for groups as small as 30 and as large as 50.

• Any coalition or agency can partner with THCU to host a workshop in their community.

• We provide the facilitators at no cost and will work with you to help tailor, organize and promote the event.

• Service request form http://www.thcu.ca/consultation/request_form.htm

• We require at least three months' notice to plan and deliver a workshop.

THCU on Evaluation

• Evaluating Health Promotion Programs Workbook in French and English

• Conducting Survey Research Workbook• Conducting Focus Groups Workbook• Tips for Running Focus Groups with Youth• Continuous Quality Improvement

Slideshow• Evaluation At a Glancehttp://www.thcu.ca/infoandresources/evaluation_resources.htm

Evaluation resources: Not ours, but also good!

• Evaluation checklists website http://www.wmich.edu/evalctr/checklists/

• Re-aim Workgroup to Evaluate and Enhance Reach http://www.re-aim.org/

• Canadian Evaluation Society Unpublished Literature

Bank http://www.evaluationcanada.ca/site.cgi?

s=6&ss=8&_lang=EN

Brought to you by THCU….

• Case Study Series: • Planning, evaluation, health communication

http://www.thcu.ca/infoandresources.htm

• What We’re Reading: • Recommended and summarized resources

http://www.thcu.ca/infoandresources/WhatWe'reReading.htm

• Literature search results:• Completed to support our client consultations

http://www.thcu.ca/infoandresources/LitReviews.htm

• THCU’s Online Learning Community:• Questions generated from our clients, answers generated by

THCU and colleagues http://www.thcu.ca/blogs/lc/

More by THCU

• Guide to French Language Resources http://www.thcu.ca/infoandresources/ressourcesenfrancais.htm

• Changing Behaviours: A Practical Framework http://www.thcu.ca/infoandresources/publications/ChangingBehavioursv4.3.nov30.2005.pdf

• Overview of Sustainability Workbook http://www.thcu.ca/infoandresources/sustainability_resources.ht

m

THCU in collaboration with OHPRS

• Health Promotion 101• This free, online course helps people familiarize themselves with essential health

promotion concepts. http://www.ohprs.ca/hp101/main.htm

• Online Proposal Writing Course• The purpose of this online course is to help both newbies and veterans

prepare a coherent and effective proposal. http://www.thcu.ca/ohcc-thcu-proposal-writing-course/

• Ontario Health Promotion Email Bulletin • Information exchange among Ontario practitioners.• Announcements and events distributed weekly. • Feature articles are distributed every second week. • The bulletins go out every Friday afternoon. www.ohpe.ca

A big thank you to

• Ontario Ministry of Health Promotion• Jodi Thesenvitz, Cathy Duerden, and

Noelle Gadon• Our workshop participants

Disclaimer

The Health Communication Unit and its resources and services are funded by the Ontario Ministry of Health Promotion. The opinions and conclusions expressed in this presentation are those of the author(s) and no official endorsement by the Ministry of Health Promotion is intended or should be inferred.