What is the true impact of evaluation ? 

JM Clinton; R McNeill *& S.K. Appleton-dyer*

Centre for Program EvaluationThe Australasian Evaluation Conference:

Evaluation and Influence

Sydney Sept. 2011

The journey

What is the true impact of evaluation?

The evidnece

The evidence is confusing

Do we add value?

• Changes organisations - Williams

• Develops learning environment - Preskill

• Evaluation willingness and capacity leads to increased program success -Clinton

• Influences individual, interpersonal & the collective-Marks & Henry

• Empowers –Fetterman;

• Evaluation attributes impact on organisations-Appleton

Evaluation research methods

What's our standard?

Our impact

KNOW THY IMPACT!

Click to edit Master title style

This study

The programs & the evaluation

5yr evaluation of a district-wide plan in NZ, aimed at mobilizing a community to prevent the onset of Type II Diabetes.

*10 action areas: Over a 150 different initiatives over a 4 year period

Nelson Marlborough Nutrition & Physical Activity: Collaborate as a community to reverse the tide of obesity.

*5 Action Areas: Over a 100 different initiatives over a 3 year period

EVALUATION METHODS: An adapted form of the CDC Evaluation Framework Standard

dimensions

Multiple forms of data collection :reports, surveys, case studies

interviews, observations and internal data

Key information from 292 initiatives was coded using scoring rubric

A research lens and the evaluation

THIS STUDY: Analysed year by year

and over 4 years. Determine relationships:

Factor analysis, Path models

• Scrubbed the data• Determined the strength & validity of the dimension• Identified outcomes• Looked for causal relationships

MEASURES

ProgramKPI

Adaptation

Degree of implementation

OutcomeProgress

Sustainability

ProcessOrganisational development

Collaboration

Evaluation S

TAN

DA

RD

S

ET

TIN

G

SCORE EXEMPLAR

KPI 1-3 Only 30% achieved. Only ‘easy’ KPIs indicator achieved

4-7 Achieved 40% mix of ‘hard and easy’ indicators

8-10 100% achieved mixture of levels of difficulty

THE MODEL

So what does this mean?

SUSTAINABILITY:

KPI XXX

ADAPTATION --IVE X

DOIX

ODX

COLLABORATION XXX

EVALUATION XXX

PROGRESS:

KPI XXX

ADAPTATION X

DOI XXX

OD XXX

COLLABORATION –IVE

XXX

EVALUATION XXX

The outcome variables

SUSTAINABILITY- WEIGHTING VARIES ACCORDING TO: TIME: DOI, COLLABORATION PROGRAM:OD, COLLABORATION, EVALUATION

PROGRESS- WEIGHTING VARIES ACCORDING TO: TIME:ADAPTATION,OD, DOIPROGRAM: OD, COLLABORATION

EVALUATION IS AN EQUATION

Program Outcome

ImpProcess

Dosage

Intervention

Adaptation

Evaluation influence

CONTEXT

Final word

EVALUATION IS AN INTERVENTION

EVALUATION STANDARD METHODS

EVALUATION ADDS VALUE

EVALUATORS - KNOW THY IMPACT

Thank you

jclinton@unimelb.edu.au