EvaluationEvaluation(a.k.a. Assessment)(a.k.a. Assessment)

Krista S. SchumacherSchumacher Consulting.org

918-284-7276krista@schumacherconsulting.orgwww.schumacherconsulting.org

Prepared for the Oklahoma State Regents for Higher Education

2010 Summer Grant Writing Institute

To Evaluate or to Assess? To Evaluate or to Assess? Technically speaking….

◦Assessment Long-term outcomes, aggregated

judgment

◦Evaluation Short-term outcomes, “unique event”

judgment

krista@schumacherconsulting.org 918-284-7276 2

Why Evaluate? Why Evaluate? How will you know your project is

progressing adequately to achieve objectives?

How will funders know your project was successful? ◦ Increasing emphasis placed on evaluation,

i.e., U.S. Department of Education National Science Foundation Substance Abuse and Mental Health Services

Administration (SAMHSA)

3krista@schumacherconsulting.org 918-284-7276

Why Evaluate?Why Evaluate?Improve the program –

◦ “Balancing the call to prove with the need to improve.” (W.K. Kellogg Foundation

Determine program effectiveness – ◦ Evaluation supports “accountability and quality

control” (Kellogg Foundation)

◦ Significant influence on program’s futureGenerate new knowledge –

◦ Not just research knowledge◦ Determines not just that a program works, but

analyzes how and why it works With whom is the program most successful? Under what circumstances?

4krista@schumacherconsulting.org 918-284-7276

Why Evaluate? Why Evaluate? WHAT WILL BE DONE WITH THE

RESULTS?????

“Evaluation results will be reviewed (quarterly, semi-

annually, annually) by the project advisory board and staff. Results

will be used to make program adjustments as needed.”

5krista@schumacherconsulting.org 918-284-7276

Types of EvaluationTypes of Evaluation

Process evaluation: What processes are used and

how well do they work?

Outcome evaluation: Did the project achieve its stated

objectives?

6krista@schumacherconsulting.org 918-284-7276

Process EvaluationProcess Evaluation◦ What was provided and to whom?

services (modality, type, intensity, duration) recipients (individual demographics and

characteristics) gender, age, race/ethnicity, income level, first-generation status

context (institution, community, classroom) cost (did the project stay within budget?)

• Do processes match the proposed project plan?

◦What types of deviation from the plan occurred?◦What led to the deviations?◦What effect did the deviations have on the project

and evaluation?

7krista@schumacherconsulting.org 918-284-7276

Outcome EvaluationOutcome Evaluation◦ What effect did the program have on

participants? Activities / Objectives Achievement / Attitudes and beliefs

◦ What program/contextual factors were associated with outcomes?

◦ What individual factors were associated with outcomes?

◦ How durable were the effects? What correlations can be drawn between

outcomes and program? How do you know that the program was the cause of the

effect?

krista@schumacherconsulting.org 918-284-7276 8

Who will Evaluate?Who will Evaluate?External evaluators increasingly

required or strongly recommended◦Partners for effective and efficient

programs◦Methodological orientations◦Philosophical orientations◦Experience and qualifications

9krista@schumacherconsulting.org 918-284-7276

How much will it cost?How much will it cost?External evaluations cost

money…period.Standard recommendation:

5% to 10% of total budgetKellogg Foundation; U.S. Dept of Ed.

Check funder limits on evaluationEnsure cost is reasonable but

sufficient

10krista@schumacherconsulting.org 918-284-7276

Two Types of DataTwo Types of DataQuantitative

◦ Numbers based on objectives and activities◦ Types of data needed:

Number of participants (process) Grade point averages (outcome) Retention rates (outcome) Survey data (outcome and process)

Qualitative◦ Interviews◦ Focus groups ◦ Observation

11krista@schumacherconsulting.org 918-284-7276

Methods/InstrumentsMethods/InstrumentsHow are you going to get your

data? ◦ Establish baseline data◦ Pre- and post-assessments (knowledge, skills)

◦ Pre- and post-surveys (attitudinal)◦ Enrollment rosters◦ Meeting minutes◦ Database reports◦ Institutional Research Office (I.R.)

12krista@schumacherconsulting.org 918-284-7276

Data AnalysisData AnalysisQuantitative data

Data analysis programs:◦ SPSS (Statistical Program for the Social Sciences),

Stata, etc...

◦ Descriptive and statistical data: On a scale of 1 to 5, with 1 being “strongly disagree”

and 5 being “strongly agree,” please indicate the extent to which you agree or disagree that pigs can fly.

Descriptive: 150 (or 75%) of respondents agree or strongly agree that pigs can

fly. Statistical (t-test, ANOVA, etc.)

There is a statistically significant difference (p<.05) between the sexes that pigs can fly, with men more likely than women to agree or strongly agree with this statement.

13krista@schumacherconsulting.org 918-284-7276

Data AnalysisData AnalysisQualitative Data

◦ Data analysis programs NVivo , ATLAS.ti, etc…

◦ More than pithy anecdotes “May explain – and provide evidence of – those hard-to-

measure outcomes that cannot be defined quantitatively.” – W.K. Kellogg Foundation

Provides insight into how and why a program is successful

◦ Analyze for themes that support (or don’t) quantitative data

14krista@schumacherconsulting.org 918-284-7276

Two Types of TimeframesTwo Types of TimeframesFormative

◦Ongoing throughout life of grant◦Measures activities and objectives

Summative◦At conclusion of grant funding

NEED BOTH!

15krista@schumacherconsulting.org 918-284-7276

TimelinesTimelinesWhen will evaluation occur?

◦Monthly? ◦Quarterly? ◦Semi-annually? ◦Annually? ◦At the end of each training session? ◦At the end of each cycle?

16krista@schumacherconsulting.org 918-284-7276

Origin of the Evaluation: Origin of the Evaluation: Need and ObjectivesNeed and Objectives

Need: For 2005-06, the fall-to-fall retention rate of first-time degree-seeking students was 55% for the College’s full-time students, compared to national average retention rates of 65% for full-time students at comparable institutions (IPEDS, 2006).

Objective: The fall-to-fall retention rate of full-time undergraduate students will increase by 3% each year from a baseline of 55% to 61% by Sept. 30, 2010.

17krista@schumacherconsulting.org 918-284-7276

Evaluation Data Collection Evaluation Data Collection and Reporting Planand Reporting PlanObjectives

Data collected and timeline

Methods for data collection and timeline

Instruments to be developed and timeline

Reports/outcomes timeline

Increase fall-to-fall retention by 3% per year to 61%

Student enrollment in first fall and second fall within one month of start of second fall

Enrollment entered by gender and race/ethnicity into database within first four weeks of each semester

Enrollment rosters separated by gender and race/ethnicity by Jan. 15, 2009

At mid-point of each semester

18krista@schumacherconsulting.org 918-284-7276

BEWARE THE LAYERED BEWARE THE LAYERED OBJECTIVE!OBJECTIVE!

By the end of year five, five (5) full-time developmental education instructors will conduct 10 workshops on student retention strategies for 200 adjunct instructors.

19krista@schumacherconsulting.org 918-284-7276

Logic ModelsLogic Models

From: University of Wisconsin-Extension, Program Development and Evaluation

http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

A Logic Model is……◦ A depiction of a program showing what the program will do and

what it is to accomplish.

◦ A series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes

◦ The core of program planning and evaluation

20

Inputs Outputs Outcomes

Get food Eat food Feel betterHungry

Situation

krista@schumacherconsulting.org 918-284-7276

Evaluation ResourcesEvaluation Resources W.K. Kellogg Foundation – “Evaluation Toolkit”◦ http://www.wkkf.org/default.aspx?tabid=75&CID=281&NID=61&L

anguageID=0 Newsletter resource – The PEN (Program Evaluation

News)◦ http://www.the-aps.org/education/promote/content/newslttr3.2.pdf

NSF-sponsored program◦ www.evaluatorsinstitute.com American Evaluation Association◦ www.eval.org Western Michigan University, The Evaluation Center◦ http://ec.wmich.edu/evaldir/index.html (directory of evaluators) OSRHE list of evaluators and other resources◦ http://www.okhighered.org/grant%2Dopps/writing.shtml “Evaluation for the Unevaluated” course◦ http://pathwayscourses.samhsa.gov/eval101/eval101_toc.htm

21krista@schumacherconsulting.org 918-284-7276

Evaluation ResourcesEvaluation Resources The Research Methods Knowledge Base

◦ http://www.socialresearchmethods.net/ The What Works Clearinghouse

◦ http://www.w-w-c.org/ The Promising Practices Network

◦ http://www.promisingpractices.net/ The International Campbell Collaboration

◦ http://www.campbellcollaboration.org/ Social Programs That Work

◦ http://www.excelgov.org/ Planning an Effective Program Evaluation short

course◦ http://www.the-aps.org/education/promote/pen.htm

22krista@schumacherconsulting.org 918-284-7276krista@schumacherconsulting.org 918-284-7276