Overview of Program EvaluationProgram Evaluation Basics Webinar Series

Mary E. Arnold, Ph.D.Associate Professor and Youth Development Specialist

Oregon State University

4-H Professional Development WebinarOctober 11, 2012

Overview of Program Evaluation

Let’s start at the very beginning- a very good place to start

Goals for Today’s Webinar1. Learn a basic history of youth program

evaluation2. Learn reasons for conducting

evaluations3. Learn the different levels of evaluation4. Learn Bennett’s Hierarchy- an well

known Extension evaluation model5. Learn different evaluation purposes6. Learn the steps of the evaluation

process7. Review expectations for evaluation for

OSU Extension8. Review expectations for compliance

with OSU’s Institutional Review Board (IRB)

Arnold, M. E. & Cater, M. (2011). From then to now: Emerging directions for youth program evaluation. Journal of Youth Development, 6(3). On-line: Article NO:110603FA005.

A Brief History of Youth Program Evaluation

• Publications related to youth program evaluation have flourished in the past 10 years; 20 years ago there was very little to be found

• The evaluation field as a whole is only now approaching 40 years

• 1960s saw rise to federally mandated (and funded) evaluations

• Evaluation results were poor, and had little effect on program expansions or reductions

• This gave rise to methodological critiques• Focus on rigor- experimental design to measure

outcomes• The importance of qualitative methods to

understand the “human” side of social programs• And, add in the developmental influences at play

in youth programs

*** Carnegie Council on Youth Development (1989). Turning points: Preparing American youth for the 21st century. Washington, DC: Author.

Youth Development Program Evaluation

• Began in the late 1800s• Focused on success stories – it was the right thing

to do!• 1980s dawn of the age of accountability• Late 1980s youth development programs

emerged as different from intervention programs, with different outcomes*** for youth:

• Intellectually reflective (we might call this critical thinking today)

• Enroute to a life of meaningful work• Good citizens• Caring and ethical• Healthy

• Even so, first efforts were focused on “reach” and participant satisfaction

• Finally, emerged an emphasis on demonstrating impact

Poll # 1: What has been the primary focus of evaluations you have done?

Why Do We Evaluate?• Help others understand the program

(stakeholders)• Understand the need for a program • Improve the program• Improve teaching• Understand the program’s impact• Determine if the program is progressing as

planned• Determine if the program is worth the cost• Meet grant reporting criteria• Meet administrative requirements

Poll # 2: Why do you evaluate your programs?

Tiers of Program Evaluation

Adapted From:Weiss, H. B. & Jacobs, F. H. (1988). Evaluating Family Programs. New York: Aldine de Gruyter.

“Bennett’s Hierarchy”

Bennett, C. (1975). Up the hierarchy. Journal of Extension. PDF available at: http://www.joe.org/joe/1975march/1975-2-a1.pdf

Types of Evaluation• Planning Evaluation (Needs Assessment)

• Process Evaluation• Evaluability Evaluation• Outcome Evaluation• Efficiency Evaluation

Garber, P.A., & Grinnell, R. M. (1994). Evaluation and Quality Improvement. Needham Heights, MA: Allyn & Bacon.

Poll # 3: What types of evaluations have you done?

The Steps of a Program Evaluation

Forming an

Evaluation Team

Developing

Evaluation Questions

Gathering Data

Conducting Data Analysis

Interpreting

Evaluation Results

Sharing Results

Taking Action

The Steps of a Program Evaluation

Forming

an Evaluation Team

Developing

Evaluation

Questions

Gathering Data

Conducting Data

Analysis

Interpreting

Evaluation

Results

Sharing

Results

Taking Action

The Steps of a Program Evaluation

Forming an

Evaluation Team

Developing Evaluation Questions

Gathering Data

Conducting Data Analysis

Interpreting

Evaluation

Results

Sharing

Results

Taking Action

The Steps of a Program Evaluation

Forming an

Evaluation Team

Developing

Evaluation Questions

Gathering Data

Conducting Data Analysis

Interpreting

Evaluation

Results

Sharing

Results

Taking Action

The Steps of a Program Evaluation

Forming an

Evaluation Team

Developing

Evaluation Questions

Gathering Data

Conducting

Data Analysis

Interpreting

Evaluation

Results

Sharing

Results

Taking Action

The Steps of a Program Evaluation

Forming an

Evaluation Team

Developing

Evaluation Questions

Gathering Data

Conducting Data Analysis

Interpreting Evaluation Results

Sharing

ResultsTaking Action

The Steps of a Program Evaluation

Forming an

Evaluation Team

Developing

Evaluation Questions

Gathering Data

Conducting Data Analysis

Interpreting

Evaluation Results

Sharing Results

Taking Action

The Steps of a Program Evaluation

Forming an

Evaluation Team

Developing

Evaluation Questions

Gathering

Data

Conducting Data Analysis

Interpreting

Evaluation Results

Sharing Results

Taking Action

Program Evaluation Expectations

ALL Extension faculty members must:

• Document measurable program outcomes on at lest one priority educational effort annually through an impact statement in SOARS

• This expectation takes into account the faculty member’s length of service, rank and experience in current assignment

• For those who have scholarship in their position description, evaluation efforts often lead to scholarly production (conference presentations, journal articles, etc.)

Poll # 4: What is the primary reason you have conducted program evaluations in the past 3 years?

Institutional Review Board (IRB)AKA… Do I have to do that?

Research is determined by three qualities:1. Systematic inquiry into a phenomena2. That is designed to develop or contribute3. Generalizable knowledge

Human Subjects Are:Living individuals about whom an investigator conducting research obtains:

4. Data through an intervention or interaction with the individual, or5. Identifiable private information

Institutional Review Board (IRB)Okay, I have to do it… now what?

Go to the IRB Website at: http://oregonstate.edu/research/irb/1. Complete the “Does Your Study Require IRB Review? Form2. Complete online ethics training modules3. Complete the steps listing under “Preparing and Initial

Submission” at http://oregonstate.edu/research/irb/preparing-initial-submission

That’s all for now!Join in next month for:

Program Planning for Effective Program Evaluation

Don’t forget to complete an evaluation of today’s webinar at: http://www.surveymonkey.com/s/4HEvaluationwebinar