Gina Baral Abrams, MPH, LSW, CHESAmy Melichar, MEd, CHES

AMERICAN COLLEGE HEALTH ASSOCIATION ANNUAL MEETING

SAN FRANCISCO, CAMAY 28, 2009

Evaluation Essentials for Health Promotion in

Higher Education: A Primer

Learning Objectives

Describe the uses and purposes of program evaluation in health promotion.

List conceptual approaches to evaluation.List the key qualities of measurable evaluation

questions.List qualitative and quantitative data collection

methods.Describe key qualities of an effective evaluation

report.

Evaluation Defined

“Examining the worth of a program, usually measuring it against a set of

predetermined objectives or a standard of acceptability.”

Green & Kreuter, 1999.

Uses and Purposes of Program Evaluation

Determine the rate and level of attainment of program objectives.

Assess the strengths and weaknesses of a program.

Help make decisions. Monitor standards of

performance. Establish quality assurance

and control mechanisms. Determine the generalizability

of an overall program or program elements to other populations.

Meet the demand for public or fiscal accountability.

Improve professional staff skill in program planning, implementation, and evaluation activities.

Promote positive public relations and community awareness.

Fulfill grant or contract requirements.

Contribute to scientific knowledge.

Indentify hypotheses for future study.

Conceptual Approaches to Evaluation

Objectives-based evaluation Determine extent to which objectives are achieved.

Management-oriented evaluation Provide information to aid in making decisions.

Consumer-oriented evaluation Provide information about products to aid decisions about

purchases or adoptions.Expertise-oriented evaluation

Provide professional judgments of quality.Participant-oriented evaluation

Understanding the complexities of programmatic activity, responding to an audience’s requirements for information.

The Evaluator’s Role

Objectivity is key when strategically selecting the evaluator.

Possible evaluators include: External evaluator Internal evaluator Internal evaluator with external consultant

Basic Types of Evaluation

Formative evaluation To provide information for program improvement.

What is working? What needs to be improved? How can it be improved?

Summative evaluation To provide information to make decisions about the

program’s future or adoption. What results occur? With whom? Under what conditions? With what training? At what cost?

Levels of Evaluation in Health Promotion

Process evaluation: to determine feasibility.“Implementation evaluation”“Quality assurance review”

What was delivered and how it was delivered? Did program follow protocol?

Applies non-experimental designs.

Impact evaluation: to provide information to make decisions about the program’s future or adoption.

What were short-term (immediate-24 months) effects of the program on target behaviors and their predisposing, enabling and reinforcing antecedents or on environmental factors?

Applies quasi-experimental and experimental designs.

Outcome evaluation: to assess changes in overall health status and quality of life indicators.

What were long-term changes (1-10 years)?Applies quasi-experimental and experimental designs.

Types of Evaluation Designs

Non-experimental Does not include random assignment or a control group. Asserts little or no control over confounding factors.

Quasi-experimental Includes experimental and control group. Does not include random assignment. Offers some control over confounding factors.

Experimental Experimental and control group. Includes random assignment. Asserts greatest degree over confounding factors. Evidence is most interpretable, defensible, and definitive.

Evaluation Plan: Key Steps

1. Gather background information2. Identify stakeholders and audiences3. Budget for evaluation4. Develop evaluation question(s)5. Determine sources of information required6. Select data collection methods 7. Conduct evaluation (collect, analyze and

interpret data)8. Communicate findings -- provide final report

Step 1: Gather Background Information

Why is the program needed?  General background, research on other programs

and approaches.What are the goals and objectives of the

program? What is the program context?What are the program activities?

Step 1: Case Study Example

Required orientation program for all first-year students.One-hour workshop/discussion.Delivered by residential advisors in small groups.Conducted concurrently in residential setting.Resident advisors trained in advance to lead the workshops.

Program goals: Upon completion of the program, participants will understand the

role of drinking motives in decision-making concerning alcohol. Upon completion of the program, participants will be aware of the

role of social norms in decision-making concerning alcohol. Upon completion of the program, participants will be able to

identify “decision-making” as part of their own drinking behaviors.

Step 2: Identify Stakeholders and Audiences

Who are the stakeholders and audiences? What are their roles?

Stakeholder: anyone who has a stake in the program to be evaluated or the evaluation’s results.

Audiences: groups which have an interest in the evaluation and receive its results.

Consider stakeholder and audience roles: To make policy To make operational decisions To provide input to evaluation To react to For interest only

Step 2: Case Study Example

  To Make Policy

To Make Operational Decisions

To Provide Input to Evaluation

To React

For Interest Only

Health Services X X X    

Student Affairs Office X X X    

Residential Life Office X X X    

Resident Advisors     X X  

Student participants (freshmen)

    X X  

Parents and family of University freshmen

        X

University community         X

Other University students

        X

University faculty and staff

        X

Step 3: Budget for Evaluation

What resources are available to dedicate to the evaluation?

Estimate 7 – 10% of overall project budget for evaluation. Components may include, but are not limited to:

Staff salary with benefits Consultants Travel Communications Printing and duplication Program materials Supplies and equipment Food Incentives

Components may be paid in-kind, with operational dollars, external funding, etc. List all regardless of funding source.

Step 3: Case Study Example

Expenses Description Amount TotalStaff Salary

Health Educator .01 FTE $600Fringe 27.60% $166

Personnel Total $766

ConsultantsEvaluator $150/hour x 30 hours $4,500Statistician $60/hour x 10 hours $600

Focus Group Facilitators $20/hour x 10 groups (20 hours total) x 2 facilitators $800

Outside Consultants Total $5,900

TravelTravel/Meals $500Business Meeting Expenses $150

Travel Total $650

Printing and DuplicationPrinting and Mailing Based on historical average $150

Printing and Duplication Total $150

Program MaterialsSurvey Administration $500

Program Materials Total $500

Supplies and EquipmentMeeting Rooms Based on historical average $100General Supplies Based on historical average $250Computer Supplies $200Telephone Calls $100

Supplies and Equipment Total $650

FoodCatered Meals and Meetings Based on historical average $500

Food Total $500

IncentivesFocus Group Incentives $500Survey Incentives $250

Incentives Total $750

Total Funds Needed to Complete Project 9,866$

Step 4: Develop Evaluation Questions

What questions need to be answered?

Seek input from stakeholders and audiences.Consider process, impact and outcome

evaluation needs.Divergent phase:

Create a laundry list of potentially important questions

Convergent phase: Select most critical questions from laundry list

Step 4: Case Study Example

Process Evaluation Question: To what fidelity was the program implemented?

Does the content of the workshop cover the intended objectives? Are resident advisors sufficiently prepared to lead the

workshops? Do first-year students participate in the workshops in the

intended manner?

Impact Evaluation Question: Did the students who participated in the workshop

achieve the desired results? Do participants understand the role of drinking motives in

decision-making concerning alcohol? Do participants understand the role of social norms in decision-

making concerning alcohol? Do participants identify “decision-making” as part of their own

drinking behaviors?

Step 5: Determine Sources of Information

What information do you need in order to answer the evaluation questions?

Peer review, quality

control, accreditation,

audit, certification, satisfaction

Predisposing, enabling and

reinforcing factors

Protective behavior or

environment

Health Social benefit

Changes in knowledge,

attitudes, beliefs, skills,

resources, social support, policy

Changes in frequency,

distribution, timing of

behavior, or quality of the environment

Changes in mortality, morbidity,

disability or risk factors

Changes in quality of life

Health promotion program

Process Impact Outcome

Adapted from Green & Kreuter, 1999.

Step 6: Select Data Collection Methods

What data will you collect and how will you collect them?

Strategies may include, but are not limited to: Tests and assessments Written questionnaires Policy audit Interviews with key participants Observations of behavior or environmental conditions Self-report logs or diaries Focus groups Community-level indicators of impact Service utilization rates Case studies Document review

Steps 5 & 6: Case Study Example

Determine information required and source/data collection method for each question.

Selected questions appear below. Note that there would be many more data collection methods if all questions were presented.

Research Question Information RequiredInformation Source/

Data Collection Method

To what fidelity was the program implemented?

Does the content of the workshop cover the intended

objectives?

Workshop objectives; workshop components.

Document review: Workshop lesson plan, resources,

handouts, activities.

Did the students who participated in the

workshop achieve the desired results?

Do participants understand the role of drinking motives in decision-making concerning

alcohol?

Change in knowledge of drinking motives.

One group, pre-test post-test design: Web-based self-report

survey of workshop participants.

Step 7: Conduct Evaluation Plan

June 2009

July 2009

August 2009

September 2009

October 2009

November 2009

December 2009

Contract with consultants

Meet with program developers and

stakeholders

Develop data collection methods

Resident advisor training

Program implementation and participant

observation

Administer surveys

Conduct focus groups

Data analysis

Deliver final report

Step 8: Communicate Findings

Components of the final report may include: Front matter (front cover, etc.) Executive summary Program background/evaluation description Present the results Discussion, conclusion, recommendations

Report in ways that encourage follow-through by stakeholders, to increase the likelihood that the evaluation will be used.

Re-visit list of stakeholders and audiences to determine how to communicate findings and to whom.

References

Fitzpatrick, JL, Sanders, JR & Worthen, BR. (2004). Program evaluation: Alternative approaches and practical guidelines, 3rd ed. Boston, MA: Pearson.

Green, LW & Kreuter, MW. (1999). Health promotion planning: An educational and ecological approach, 3rd ed. Mountain View, CA: Mayfield Publishing Company.

McDermott, R., & Sarvela, P. (1999). Health education evaluation and measurement, 2nd ed. The McGraw-Hill Companies.

Modeste, NN & Tamayose, TS. (2004). Dictionary of public health promotion and education terms and concepts, 2nd ed. San Francisco, CA: Jossey-Bass.

WK Kellogg Foundation. (1998). The W.K. Kellogg evaluation handbook. Available at www.wkkf.org.

Windsor, R., Baranowski, T., Clark, N., & Cutter, G., (1994). Evaluation of health promotion, health education, and disease prevention programs, 2nd ed. Mountain View, CA: Mayfield Publishing Company.

Questions?

Gina Baral Abrams, MPH, LSW, CHESPrinceton Universitygbaral@princeton.edu

Amy Melichar, MEd, CHESMarquette Universityamy.melichar@marquette.edu