An Introduction to Evaluation Research

Duane Shell

Research Associate Professor

Nebraska Prevention Center for Alcohol and Drug Abuse

Department of Educational Psychology

Why Evaluate?

Because somebody

says you have to.

[usually a funder]

What is Evaluation?

Something to be “feared”.

A sneaky way for the “powers that be” to get you.

How “the Man” stifles your creativity.

How “they” can justify taking away your money.

What is Evaluation?

Evaluation is a way for YOU to find out if things you are doing are working.

So Why Evaluate?

You Evaluate for Yourself

Not for others.

Really!!

The difference between

Evaluation

and

Research

Evaluation vs Research

Evaluation and Research DO NOT differ in:

Data collection tools

Methodologies

Analysis methods

The purpose of research is to:

Test hypotheses

Inform theory

Or at least:

Describe or draw conclusions

About some phenomenon

The purpose of evaluation is to:

Inform decision making about project success

by

Determining if objectives are achievedand

Activities are done as planned

So What Is Evaluation?

A systematic way

to collect data

to inform decision making

Inform Decision Making

It is not possible to definitively answer the question:

Was the project successful?

The only thing that can be done is to collect evidence to allow a judgment

Collect Data

The foundation of Evaluation is Data driven decision making.

Data are typically collected from multiple sources

Data are typically both quantitative and qualitative

Multiple data are “triangulated” to make judgments and decisions

Systematic Data Collection

Evaluation is based in a systematic approach to collecting data.

Evaluation uses the same methodologies that are used in research.

Standards of methodological rigor are the same as those used in research.

Systematic Data Collection

Evaluation, though must apply methods and rigor within the context of the real world of the project being evaluated.

Where the exacting standards of research design cannot be done,

Evaluation attempts to collect data in the most systematic and open way possible.

Systematic Data Collection

Recognition of the usual failure to achieve the most exacting standards of research is why evaluation decisions are based on:

multiple triangulated data sources

Evaluation Basics

5 Basic Evaluation Questions

1) What will be assessed?

2) What measures/indicators will be used?

3) Who will be evaluated?

4) What data will be collected?

5) How will data be analyzed?

What Will Be Evaluated?

Formative (aka Process) Evaluation:

Done to help improve the project itself.Gather information on how the project worked.Data is collected about activities: What was done.

Summative (aka Outcome) Evaluation:

Done to determine what results were achieved.Data is collected about outcomes (objectives; goals): What happened.

What Measures Will Be Used?

Formative Evaluation:Completion of planned activitiesAdherence to proposed time linesMeeting budget

Summative Evaluation:

Reaching a criterionChange in knowledge, attitude, skill, behavior

Who will be evaluated?

Formative Evaluation:Those responsible for doing activities/delivering services and those participating in activities.

FacultyAgency personnelStudents

Summative Evaluation:Those who were expected to be impacted by activities.

StudentsClients

What data will be collected?

Formative Evaluation:Program recordsObservationsActivity logsSatisfaction surveys

Summative Evaluation:

ObservationsInterviewsTestsSurveys/questionnaires

How will data be analyzed?

1) Qualitative analysis (more for formative)1) Self-reports2) Documentation3) Description4) Case Study

2) Quantitative analysis (more for summative)1) Group comparison2) Group change3) Individual change4) Comparison to population/reference5) Analysis of relationships

An ExampleThe Cosmic Ray Observatory Project (CROP)

Goal: Establish a statewide collaborative network of expert teachers fully capable of continuing the project locally.

Objectives: Teachers will acquire knowledge about cosmic ray physics and skill in high energy

research methods.Teachers will exhibit increased self-efficacy for conducting CROP research and integrating CROP into their teaching.

Activity: High school physics teachers and students will attend a 3-4 week hands-on summer research experience on cosmic ray physics

at UNL

Formative EvaluationWhat activities were evaluated?

The specific components of the Summer Research ExperienceWhat measures were used?

Completion of activitiesParticipant satisfactionParticipant evaluation of goal attainmentParticipant evaluation of activity effectiveness

Who was evaluated?Participants

What data was collected?InterviewsRating scales

How was data analyzed?Content analysis of interview responsesFrequency and descriptive statistical analysis of rating scales.

Examples of Formative Measures Interview Questions

What was the most effective part of the workshop?

Hands-on work with detectors 6

Information from classroom sessions 4

Teacher Comments (by teacher with coded category(s) indicated):

• For me personal was the activities. The actual connecting and wiring and those things. I don’t sit and take lectures very well. That’s just me. [Hands on work with the detectors]

• Um, I think it was the classroom work. There was a good review for those of us that have had physics and it was a good introduction for those that didn’t. [Information from classroom sessions]

Examples of Formative Measures Rating Scales

1. How effective do you think the workshop was in meeting its goals?

1 2 3 4 5

Not Effective Neither Effective Somewhat Effective Very Effective nor ineffective Effective

4. Indicate how USEFUL you think each of the following workshop components was using the following scale.

1 2 3 4 5 6

Very Unuseful Somewhat Somewhat Useful Very

Unuseful Unuseful Useful Useful

a. Classroom/lecture sessions on particle detectors and experimental techniques.

b. Lab work sessions refurbishing and preparing detectors.

Examples of Formative Measures Rating Scales

How effective do you think the workshop was in meeting its goals?

Very Effective Somewhat Neither Effective Not Effective Effective nor Ineffective Effective

(5) (4) (3) (2) (1) M SD 1 3 1 0 0 4.00 .71

Very Unuseful Somewhat Somewhat Useful Very Unuseful Unuseful Useful Useful (1) (2) (3) (4) (5) (6) M SD Component 0 0 0 0 2 3 5.60 .55 Classroom/lecture

sessions on particle detectors and experimental

techniques. 0 0 0 1 3 1 5.00 .71 Lab work sessions

refurbishing and preparing detectors.

Summative EvaluationWhat Outcomes were evaluated?

Teachers’ increase in knowledge about cosmic ray physics and skill in high energy research methodsTeachers’ change self-efficacy for conducting CROP research and integrating CROP into their teaching

What measures were used?Knowledge gainAchieving criteria level of knowledge/skillIncrease in self-efficacy

Who was evaluated?Teachers

What data was collected?Pre- and post-workshop tests of cosmic ray physics and researchPre- and post-workshop self-efficacy ratings

How was data analyzed?Dependent t-tests of pre-post scoresComparing skill scores to criteria

Summative EvaluationKnowledge Test Questions

1. The energy distribution of primary cosmic rays bombarding the earth has been measured by a number of experiments. In the space below, sketch a graph of the number of observed primary cosmic rays vs. cosmic ray energy, and describe the distribution in a sentence or two.

2. Explain how a scintillation counter works, i.e. write down the sequence of events from the passage of a charged particle through a scintillator to the generation of an electric signal in a photomultiplier tube.

3. Describe some characteristic differences between electromagnetic showers and hadronic showers created when particles impinge on a block of matter or a cosmic ray enters the atmosphere. Hint: think in terms of the type of particle which initiates the shower, the type of secondary particles in the shower, the shape of the shower, depth penetration of the shower particles, etc.

Summative EvaluationData Analysis

Table 9

Participants Pre- and Post-Test Mean Scores on Knowledge Tests

Pre-Test Post-Test

df M SD M SD t ES

Teachers 4 5.00 2.69 19.60 1.71 8.67* 6.64

Note. ES = effect size computed by Cohen's d in averaged pre- and post-test SD units. Teachers, n = 5.

*p < .01.

Summative EvaluationSelf-Efficacy Questions

Please rate how confident you are about each of the following from 0 (completely unconfident) to 100 (completely confident).

1. Your ability to set-up and maintain the CROP research equipment at your school.

2. Your ability to conduct CROP research at your school. 3. Your ability to teach students at your school who

haven't attended the Summer Workshop how to conduct CROP research at your school.

4. Your ability to design your own research projects for your students utilizing the CROP research equipment.

5. Your ability to incorporate lessons and activities in high-energy physics into your classes.

6. Your ability to create "hands-on" projects and activities for students in your classes using the CROP research equipment.

Summative EvaluationData Analysis

Table 11Participants Pre- and Post-Test Mean Self-Efficacy Scores

Pre-Test Post-Test df M SD M SD t ES

Conducting 4 41.00 31.58 77.80 16.10 3.06* 1.54CROP Activities

Integrating CROP 4 45.00 31.37 79.25 17.31 3.32*1.41

Into Classes

Utilizing Distance 4 56.67 17.48 70.33 13.35 4.08* .89

Education

Note. ES = effect size computed by Cohen's d in averaged pre- and post-test SD units. Teachers, n = 5.

*p < .01.

Formative Evaluation Example

To obtain student reactions for the development of the campus specific Web based brief intervention versions, student feedback will be obtained. Beta versions will be evaluated by recruiting a panel of students from the each participating campus. These students will complete the intervention and provide verbal and written feedback on their reactions to the program and their suggestions for improvement. Adjustments to the program will be made based on student feedback.

Summative Evaluation Example

Students will complete the web-based brief alcohol intervention (pre-test). Approximately 6-weeks later, they will again complete the web-based brief alcohol intervention (post-test). Change will be determined by comparing post-test scores to pre-test scores using a Repeated Measures Analysis of Variance (ANOVA). Success will be determined by a statistically significant decrease in drinking and driving (Objective 1) and riding with a driver who has been drinking (Objective 2), with an effect size of at least a 10% pre- to post-test decrease for drunk driving and a 6% decrease for riding with a drinking driver.

Planning Evaluation

The Logic Model

A systematic linkage of project goals, objectives, activities, and outcomes.

Steps in Creating a Logic Model

1) Clarify what the goals of the project/ program are.

2) Clarify what objectives the project should achieve.

3) Specify what program activities will occur.

Goal Clarification

High school physics teachers and students will attend a 3-4 week hands-on summer research experience on cosmic ray physics at UNL.

Is this a goal?

Goal Clarification

Establish a statewide collaborative network of expert teachers fully capable of continuing the project locally.

Developing Objectives

Goal: Establish a statewide collaborative network of expert teachers fully capable of continuing the project locally.

Objectives1. Teachers will acquire knowledge about cosmic

ray physics and skill in high energy research methods.

2. Teachers will exhibit increased self-efficacy for conducting CROP research and integrating CROP into their teaching.

CROP Logic ModelGoal: Establish a statewide collaborative

network of expert teachers fully capable of continuing the project locally.

Objectives: Teachers will acquire knowledge about cosmic ray physics and skill in high energy

research methods.

Teachers will exhibit increased self-efficacy for conducting CROP research and integrating CROP into their teaching.

Activity: High school physics teachers and students will attend a 3-4 week hands-on summer research experience on cosmic ray physics at UNL

Evaluating the Logic Model

• Goal – Objective CorrespondenceAre objectives related to the overall goal?

• Goal – Activity CorrespondenceDo anticipated activities adequately implement the goals?

• Activity – Objective CorrespondenceWill program activities result in achieving objectives?

CROP Logic ModelGoal: Establish a statewide collaborative

network of expert teachers fully capable of continuing the project locally.

Objectives: Teachers will acquire knowledge about cosmic ray physics and skill in high energy

research methods.

Teachers will exhibit increased self-efficacy for conducting CROP research and integrating CROP into their teaching.

Activity: High school physics teachers and students will attend a 3-4 week hands-on summer research experience on cosmic ray physics at UNL

An Example

GOAL 1: Increase the availability of attractive student centered social activities located both on and off the NU campus.

Objective 1.1: Increase by 15% from baseline the number of students aware of campus and community entertainment options available to NU students.

Activity: Develop and maintain an interactive web site describing social and entertainment options

for students.

Another Example

GOAL 7: Reduce high-risk alcohol marketing and promotion practices.

Objective 7.3: Reduce by 25% from baseline, the volume of alcohol advertisements in the Daily Nebraskan, The Reader and Ground Zero that mention high-risk marketing and promotion practices.

Activity: Work with the media to encourage at least 3 newspaper articles or television news stories in the Lincoln market each school year concerning high-risk marketing and promotion practices.

Logic Model Example

Goal Objectives Methodology Completion Date 1. Create active, operational campus task forces at the 11 remaining state-funded institutions of higher education serving undergraduate populations.

1.1 Recruit support from upper administration at each institution to commit personnel to task force coordination and participation.

Administrative luncheon presentations to institution chancellors and presidents on statewide initiatives hosted by University of Nebraska President James Milliken; follow-up identifying key contacts

By November, 2005

1.2 Provide technical assistance and training to assist campuses in campus task force recruitment, organization and development.

Drive-in workshop on coalition development; follow-up teleconferences with organizers, internet resources.

By December, 2005

1.3: Provide regular teleconferencing facilitation to allow interaction between task force coordinators at participating campuses.

Monthly telephone conference of campus task force organizers; agenda that allows sharing of issues, problems, needs, and accomplishments

Ongoing through January, 2007

Logic Model Worksheet

Goals Activities Objectives

(Outcome)

Indicators

Measures

Who Evaluated

Data Sources

Data Analysis

Final Thoughts

Funders including

Federal, State, and Foundation

increasingly want more

Summative (Outcome) Evaluation

Especially Summative Evaluation of

Educational Activities

and

Outreach Activities

To be competitive for funding

You need a strong evaluation

That is more than

Did activities get done

and did attendees like it?

Where to get Help

The SSP Core(Well duh-Why else would we be doing this Brown Bag?)

The SSP can assist with:

Developing an evaluation plan and logic model for funding proposals

Conducting evaluations of ongoing projects