Outline – Part I

1. Definitions

2. Pseudo-evaluation vs. legitimate evaluation

3. Formative vs. summative evaluation

4. Necessary skills

5. Planning an evaluationa. Planning a Formative evaluation

b. Planning a Summative evaluation

6. Cost benefit analysis

7. An experimenting society

8. Words of warning

Definition (1)

Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object.

-- William Trochim (Cornell University)

3

Pseudo-evaluation vs. legitimate evaluation

Pseudo-evaluations Evaluation usually occurs in a political context.

Legitimate evaluations

Be careful not to engage in the first type (Pseudo-evaluation)!

Be careful not to engage in pseudo-evaluations

Doing so may facilitate inappropriate decisions

It will also damage your professional reputation

Pseudo-evaluations – a taxonomy

Here are some procedures to watch out for (E.A. Suchman, 1967):

Eyewash – emphasis on surface appearances

Whitewash – attempts to cover up known failures

Pseudo-evaluations – a taxonomy

Submarine – political use of evaluation to destroy a programme

Posture – ritualistic evaluation to satisfy a funding requirement, without real interest in, or intention to use, its findings

Postponement – using the need for evaluation to delay action

Legitimate Evaluations – Four Criteria

Here are four criteria to help you recognize a legitimate evaluation:

1. Utility

2. Feasibility

3. Propriety

4. Technical adequacy

Four criteria for legitimate evaluations

1. Utility – will someone be able to use it?

As Robson says, “the purpose of an evaluation is not to prove, but to improve.” (2002, p. 209)

Four criteria for legitimate evaluations

2. Feasibility – will you have the resources, time, and co-operation you need? If not, don’t do the evaluation.

Won’t achieve anything useful May damage your professional reputation. Especially an issue in formative evaluation, where

results may be needed for program planning. Remember the engineer’s maxim:

“Good, fast, cheap. Pick any two.”

Four criteria for legitimate evaluations

3. Propriety – only do an evaluation if you can do it fairly and ethically.

No ‘submarines’ Acceptable outcome measures Say ‘no’ if you believe the course of action has

already been decided on, and a decision maker just wants ‘cover.’

Four criteria for legitimate evaluations

4. Technical adequacy – if you are satisfied on the first three issues, carry out the evaluation with technical skill and sensitivity.

How can you tell whether you have the technical skill? What do you have to think about in planning? What are the relevant skills?

We’ll consider these issues below…

What to think about in planning

Reasons for evaluating

Why is the evaluation being done? Who should have access to the information obtained?

What value will results have?

Will action be taken? Will someone not want results published?

What to think about in planning

Interpretation

Is the nature of the evaluation agreed upon by those involved?

Outcome measures

What type of change is good, or bad?

What to think about in planning

Subject

What kind of information do you need?

Evaluators

Who will gather the information?

Who will analyze the data and write the report?

What to think about in planning

Methods

What method is appropriate given the questions?

Can you develop your method in the time allowed?

Is your method acceptable to those involved? (Service providers and consumers.)

What to think about in planning

Time

What time is available? Is this sufficient?

Permissions and control

Necessary permissions obtained? Is participation voluntary? Who decides what goes into the report?

What to think about in planning

Use

Who decides how the evaluation will be used?

Will those involved (providers, consumers) see the report in a modifiable draft version?

Is the form of the report appropriate for the intended audience (style, length, stats)?

An evaluation culture

These ideas are based on Donald Campbell’s (1969) concept of an experimenting society, and Trochim’s related concept of an evaluation culture

To learn more about Trochim’s ideas, see: http://www.socialresearchmethods.net/kb/evalcult.php

An evaluation culture

An evaluation works and improves because the culture is:

Action-oriented Teaching-oriented Diverse, inclusive, participatory, responsive and

fundamentally non-hierarchical. Humble, self-critical

An evaluation culture

An evaluation works and improves because the culture is:

Interdisciplinary Truth-seeking, forward-Looking Ethical, and democratic

Words of warning

Keep it simple

Avoid complex designs and data analysis

Think defensively

Anything that can go wrong, will go wrong. Try to anticipate potential problems and plan how

you will deal with them.

Words of warning

Change will always have sponsors and critics. People’s lives may be radically changed On the basis of your findings. jobs may be on the linecareers may be advanced or sloweda program may be expanded or cut back

Words of warning

There will be many stakeholders – politicians, administrators, deliverers, targets, unions, taxpayers.

It is unlikely that the interests of all these groups will coincide.

Outline – Part II

1. Formative & Summative evaluation defined

2. Elements of a Formative evaluation

3. Elements of a Summative evaluation

4. Evaluation strategiesA.Scientific-Experimental Paradigms

B.Management-oriented systems models

C.Qualitative-Anthropological models

D.Participant-oriented models

5. Necessary Skills

Two Types of Evaluation

Formative evaluation

Helps in the development of a program or service.

Summative evaluation

Assesses the effects and effectiveness of the program

Covers all effects, not just those intended

Formative Evaluation - Elements

Questions about the process being evaluated:

1. Structured conceptualization

2. Logic model

3. Process evaluation

4. Implementation evaluation

Formative Evaluation – elements

1. Structured conceptualization – helps stakeholders define program, targets, and desired outcomes.

Stakeholders – who are they? Outcomes – how do you plan to measure them?

Formative Evaluation – elements 2. A logic model makes explicit the

steps that are expected to produce the desired change. It is often shown as a flow chart or map.

A good logic model may reveal hidden assumptions about how intervention will work.

Formative Evaluation – elements

3. Process evaluation – What alternative procedures are available for delivery of the program?

4. Implementation evaluation – Is program being delivered the way it is supposed to be? Are there unexpected consequences?

Summative Evaluation

Outcome evaluation Did program cause demonstrable effects on

predefined outcome measures?

Impact evaluation Broader – assesses overall effects, intended

and unintended, of a program

Summative Evaluation Cost-benefit analysis

Questions about efficiency Standardizes outcomes in terms of dollar

costs and dollar benefits Important when you have to choose how

to spend limited amounts of money

Cost-Benefit Analysis

To do cost-benefit analysis you need to know (in addition to program cost)

(a) magnitude of benefits a program produces and

(b) that the program produced these benefits.

These things can only be learned through an experimental design.

Cost-Benefit Analysis

Some issues to consider before you do CBA…

1. Opportunity cost

2. Present value of money

3. Fairness

4. Complexity

CBA and Opportunity Cost

CBA expresses values in dollars. This reveals opportunity cost – if you do X with

your money, you cannot do Y with the same money.

Some values are difficult to express in dollars. E.g., what is the value of having mail delivery in rural areas?

How do you express non-market values in dollars?

CBA & Present Value

CBA works with the Present Value (PV) of money.

Future outcomes are uncertain. Inflation alters value of money – e.g., PV of

$1m in 50 years at 5% inflation = $87,000 .

CBA & Present Value

$100 of benefit today is worth more in Present Value than $100 of benefit 5 years from now.

This makes sense, but biases program evaluation away from long-term outcomes

CBA & Fairness

CBA compares benefits and costs without regard to who benefits and who pays costs. Is that fair? Is it unavoidable?

For example, people who live in the city subsidize mail delivery to people who live in the country. Is that fair? CBA doesn’t answer that question.

CBA & Complexity

Most social problems, and many problems in the private sector are complex.

They have many interacting causes, so establishing cause may be difficult.

Any program is likely to make only a small difference.

But it still makes sense to quantify the value of a program, to see if we could spend our money to better effect.

The relevant skills? (Robson, 2002)

Writing a proposal Clarifying purposes of an evaluation Identifying, organizing and working with an

evaluation team Choosing design & data-collection techniques Interviewing Questionnaire construction and use

What are the relevant skills?

Observation Management of complex information systems Data analysis Report-writing Encouraging people to use the findings Sensitivity to political concerns