Date post: | 15-Dec-2015 |
Category: |
Documents |
Upload: | hana-dawkins |
View: | 213 times |
Download: | 0 times |
An Introduction to Evaluation Research
Duane Shell
Research Associate Professor
Nebraska Prevention Center for Alcohol and Drug Abuse
Department of Educational Psychology
Why Evaluate?
Because somebody
says you have to.
[usually a funder]
What is Evaluation?
Something to be “feared”.
A sneaky way for the “powers that be” to get you.
How “the Man” stifles your creativity.
How “they” can justify taking away your money.
What is Evaluation?
Evaluation is a way for YOU to find out if things you are doing are working.
So Why Evaluate?
You Evaluate for Yourself
Not for others.
Really!!
The difference between
Evaluation
and
Research
Evaluation vs Research
Evaluation and Research DO NOT differ in:
Data collection tools
Methodologies
Analysis methods
The purpose of research is to:
Test hypotheses
Inform theory
Or at least:
Describe or draw conclusions
About some phenomenon
The purpose of evaluation is to:
Inform decision making about project success
by
Determining if objectives are achievedand
Activities are done as planned
So What Is Evaluation?
A systematic way
to collect data
to inform decision making
Inform Decision Making
It is not possible to definitively answer the question:
Was the project successful?
The only thing that can be done is to collect evidence to allow a judgment
Collect Data
The foundation of Evaluation is Data driven decision making.
Data are typically collected from multiple sources
Data are typically both quantitative and qualitative
Multiple data are “triangulated” to make judgments and decisions
Systematic Data Collection
Evaluation is based in a systematic approach to collecting data.
Evaluation uses the same methodologies that are used in research.
Standards of methodological rigor are the same as those used in research.
Systematic Data Collection
Evaluation, though must apply methods and rigor within the context of the real world of the project being evaluated.
Where the exacting standards of research design cannot be done,
Evaluation attempts to collect data in the most systematic and open way possible.
Systematic Data Collection
Recognition of the usual failure to achieve the most exacting standards of research is why evaluation decisions are based on:
multiple triangulated data sources
Evaluation Basics
5 Basic Evaluation Questions
1) What will be assessed?
2) What measures/indicators will be used?
3) Who will be evaluated?
4) What data will be collected?
5) How will data be analyzed?
What Will Be Evaluated?
Formative (aka Process) Evaluation:
Done to help improve the project itself.Gather information on how the project worked.Data is collected about activities: What was done.
Summative (aka Outcome) Evaluation:
Done to determine what results were achieved.Data is collected about outcomes (objectives; goals): What happened.
What Measures Will Be Used?
Formative Evaluation:Completion of planned activitiesAdherence to proposed time linesMeeting budget
Summative Evaluation:
Reaching a criterionChange in knowledge, attitude, skill, behavior
Who will be evaluated?
Formative Evaluation:Those responsible for doing activities/delivering services and those participating in activities.
FacultyAgency personnelStudents
Summative Evaluation:Those who were expected to be impacted by activities.
StudentsClients
What data will be collected?
Formative Evaluation:Program recordsObservationsActivity logsSatisfaction surveys
Summative Evaluation:
ObservationsInterviewsTestsSurveys/questionnaires
How will data be analyzed?
1) Qualitative analysis (more for formative)1) Self-reports2) Documentation3) Description4) Case Study
2) Quantitative analysis (more for summative)1) Group comparison2) Group change3) Individual change4) Comparison to population/reference5) Analysis of relationships
An ExampleThe Cosmic Ray Observatory Project (CROP)
Goal: Establish a statewide collaborative network of expert teachers fully capable of continuing the project locally.
Objectives: Teachers will acquire knowledge about cosmic ray physics and skill in high energy
research methods.Teachers will exhibit increased self-efficacy for conducting CROP research and integrating CROP into their teaching.
Activity: High school physics teachers and students will attend a 3-4 week hands-on summer research experience on cosmic ray physics
at UNL
Formative EvaluationWhat activities were evaluated?
The specific components of the Summer Research ExperienceWhat measures were used?
Completion of activitiesParticipant satisfactionParticipant evaluation of goal attainmentParticipant evaluation of activity effectiveness
Who was evaluated?Participants
What data was collected?InterviewsRating scales
How was data analyzed?Content analysis of interview responsesFrequency and descriptive statistical analysis of rating scales.
Examples of Formative Measures Interview Questions
What was the most effective part of the workshop?
Hands-on work with detectors 6
Information from classroom sessions 4
Teacher Comments (by teacher with coded category(s) indicated):
• For me personal was the activities. The actual connecting and wiring and those things. I don’t sit and take lectures very well. That’s just me. [Hands on work with the detectors]
• Um, I think it was the classroom work. There was a good review for those of us that have had physics and it was a good introduction for those that didn’t. [Information from classroom sessions]
Examples of Formative Measures Rating Scales
1. How effective do you think the workshop was in meeting its goals?
1 2 3 4 5
Not Effective Neither Effective Somewhat Effective Very Effective nor ineffective Effective
4. Indicate how USEFUL you think each of the following workshop components was using the following scale.
1 2 3 4 5 6
Very Unuseful Somewhat Somewhat Useful Very
Unuseful Unuseful Useful Useful
a. Classroom/lecture sessions on particle detectors and experimental techniques.
b. Lab work sessions refurbishing and preparing detectors.
Examples of Formative Measures Rating Scales
How effective do you think the workshop was in meeting its goals?
Very Effective Somewhat Neither Effective Not Effective Effective nor Ineffective Effective
(5) (4) (3) (2) (1) M SD 1 3 1 0 0 4.00 .71
Very Unuseful Somewhat Somewhat Useful Very Unuseful Unuseful Useful Useful (1) (2) (3) (4) (5) (6) M SD Component 0 0 0 0 2 3 5.60 .55 Classroom/lecture
sessions on particle detectors and experimental
techniques. 0 0 0 1 3 1 5.00 .71 Lab work sessions
refurbishing and preparing detectors.
Summative EvaluationWhat Outcomes were evaluated?
Teachers’ increase in knowledge about cosmic ray physics and skill in high energy research methodsTeachers’ change self-efficacy for conducting CROP research and integrating CROP into their teaching
What measures were used?Knowledge gainAchieving criteria level of knowledge/skillIncrease in self-efficacy
Who was evaluated?Teachers
What data was collected?Pre- and post-workshop tests of cosmic ray physics and researchPre- and post-workshop self-efficacy ratings
How was data analyzed?Dependent t-tests of pre-post scoresComparing skill scores to criteria
Summative EvaluationKnowledge Test Questions
1. The energy distribution of primary cosmic rays bombarding the earth has been measured by a number of experiments. In the space below, sketch a graph of the number of observed primary cosmic rays vs. cosmic ray energy, and describe the distribution in a sentence or two.
2. Explain how a scintillation counter works, i.e. write down the sequence of events from the passage of a charged particle through a scintillator to the generation of an electric signal in a photomultiplier tube.
3. Describe some characteristic differences between electromagnetic showers and hadronic showers created when particles impinge on a block of matter or a cosmic ray enters the atmosphere. Hint: think in terms of the type of particle which initiates the shower, the type of secondary particles in the shower, the shape of the shower, depth penetration of the shower particles, etc.
Summative EvaluationData Analysis
Table 9
Participants Pre- and Post-Test Mean Scores on Knowledge Tests
Pre-Test Post-Test
df M SD M SD t ES
Teachers 4 5.00 2.69 19.60 1.71 8.67* 6.64
Note. ES = effect size computed by Cohen's d in averaged pre- and post-test SD units. Teachers, n = 5.
*p < .01.
Summative EvaluationSelf-Efficacy Questions
Please rate how confident you are about each of the following from 0 (completely unconfident) to 100 (completely confident).
1. Your ability to set-up and maintain the CROP research equipment at your school.
2. Your ability to conduct CROP research at your school. 3. Your ability to teach students at your school who
haven't attended the Summer Workshop how to conduct CROP research at your school.
4. Your ability to design your own research projects for your students utilizing the CROP research equipment.
5. Your ability to incorporate lessons and activities in high-energy physics into your classes.
6. Your ability to create "hands-on" projects and activities for students in your classes using the CROP research equipment.
Summative EvaluationData Analysis
Table 11Participants Pre- and Post-Test Mean Self-Efficacy Scores
Pre-Test Post-Test df M SD M SD t ES
Conducting 4 41.00 31.58 77.80 16.10 3.06* 1.54CROP Activities
Integrating CROP 4 45.00 31.37 79.25 17.31 3.32*1.41
Into Classes
Utilizing Distance 4 56.67 17.48 70.33 13.35 4.08* .89
Education
Note. ES = effect size computed by Cohen's d in averaged pre- and post-test SD units. Teachers, n = 5.
*p < .01.
Formative Evaluation Example
To obtain student reactions for the development of the campus specific Web based brief intervention versions, student feedback will be obtained. Beta versions will be evaluated by recruiting a panel of students from the each participating campus. These students will complete the intervention and provide verbal and written feedback on their reactions to the program and their suggestions for improvement. Adjustments to the program will be made based on student feedback.
Summative Evaluation Example
Students will complete the web-based brief alcohol intervention (pre-test). Approximately 6-weeks later, they will again complete the web-based brief alcohol intervention (post-test). Change will be determined by comparing post-test scores to pre-test scores using a Repeated Measures Analysis of Variance (ANOVA). Success will be determined by a statistically significant decrease in drinking and driving (Objective 1) and riding with a driver who has been drinking (Objective 2), with an effect size of at least a 10% pre- to post-test decrease for drunk driving and a 6% decrease for riding with a drinking driver.
Planning Evaluation
The Logic Model
A systematic linkage of project goals, objectives, activities, and outcomes.
Steps in Creating a Logic Model
1) Clarify what the goals of the project/ program are.
2) Clarify what objectives the project should achieve.
3) Specify what program activities will occur.
Goal Clarification
High school physics teachers and students will attend a 3-4 week hands-on summer research experience on cosmic ray physics at UNL.
Is this a goal?
Goal Clarification
Establish a statewide collaborative network of expert teachers fully capable of continuing the project locally.
Developing Objectives
Goal: Establish a statewide collaborative network of expert teachers fully capable of continuing the project locally.
Objectives1. Teachers will acquire knowledge about cosmic
ray physics and skill in high energy research methods.
2. Teachers will exhibit increased self-efficacy for conducting CROP research and integrating CROP into their teaching.
CROP Logic ModelGoal: Establish a statewide collaborative
network of expert teachers fully capable of continuing the project locally.
Objectives: Teachers will acquire knowledge about cosmic ray physics and skill in high energy
research methods.
Teachers will exhibit increased self-efficacy for conducting CROP research and integrating CROP into their teaching.
Activity: High school physics teachers and students will attend a 3-4 week hands-on summer research experience on cosmic ray physics at UNL
Evaluating the Logic Model
• Goal – Objective CorrespondenceAre objectives related to the overall goal?
• Goal – Activity CorrespondenceDo anticipated activities adequately implement the goals?
• Activity – Objective CorrespondenceWill program activities result in achieving objectives?
CROP Logic ModelGoal: Establish a statewide collaborative
network of expert teachers fully capable of continuing the project locally.
Objectives: Teachers will acquire knowledge about cosmic ray physics and skill in high energy
research methods.
Teachers will exhibit increased self-efficacy for conducting CROP research and integrating CROP into their teaching.
Activity: High school physics teachers and students will attend a 3-4 week hands-on summer research experience on cosmic ray physics at UNL
An Example
GOAL 1: Increase the availability of attractive student centered social activities located both on and off the NU campus.
Objective 1.1: Increase by 15% from baseline the number of students aware of campus and community entertainment options available to NU students.
Activity: Develop and maintain an interactive web site describing social and entertainment options
for students.
Another Example
GOAL 7: Reduce high-risk alcohol marketing and promotion practices.
Objective 7.3: Reduce by 25% from baseline, the volume of alcohol advertisements in the Daily Nebraskan, The Reader and Ground Zero that mention high-risk marketing and promotion practices.
Activity: Work with the media to encourage at least 3 newspaper articles or television news stories in the Lincoln market each school year concerning high-risk marketing and promotion practices.
Logic Model Example
Goal Objectives Methodology Completion Date 1. Create active, operational campus task forces at the 11 remaining state-funded institutions of higher education serving undergraduate populations.
1.1 Recruit support from upper administration at each institution to commit personnel to task force coordination and participation.
Administrative luncheon presentations to institution chancellors and presidents on statewide initiatives hosted by University of Nebraska President James Milliken; follow-up identifying key contacts
By November, 2005
1.2 Provide technical assistance and training to assist campuses in campus task force recruitment, organization and development.
Drive-in workshop on coalition development; follow-up teleconferences with organizers, internet resources.
By December, 2005
1.3: Provide regular teleconferencing facilitation to allow interaction between task force coordinators at participating campuses.
Monthly telephone conference of campus task force organizers; agenda that allows sharing of issues, problems, needs, and accomplishments
Ongoing through January, 2007
Logic Model Worksheet
Goals Activities Objectives
(Outcome)
Indicators
Measures
Who Evaluated
Data Sources
Data Analysis
Final Thoughts
Funders including
Federal, State, and Foundation
increasingly want more
Summative (Outcome) Evaluation
Especially Summative Evaluation of
Educational Activities
and
Outreach Activities
To be competitive for funding
You need a strong evaluation
That is more than
Did activities get done
and did attendees like it?
Where to get Help
The SSP Core(Well duh-Why else would we be doing this Brown Bag?)
The SSP can assist with:
Developing an evaluation plan and logic model for funding proposals
Conducting evaluations of ongoing projects