Presentation4 feb 15

Proving Your Impact: monitoring & evaluation techniques

4th February 2015Robin Beveridge and Tony Sacco

• Domestics

• Who’s who

• What’s in store today

• Ground rules

• Networking and learning from each other

Introduction

Structure of the day: AM

9.30 Introductions and warm up

9.45 Key concepts and logic chains

Break

10.45 Capturing information from beneficiaries – what works?

11.30

Lunch

Quantitative evaluation techniques

Structure of the day: PM

14.10

15.00

Workshop feedback and close

Evaluation in practice

Wowing the world

13.00

15:30

‘Telling the story’ using case studies

13.30 Value for money

Getting started

• Over the next 5 minutes…Write down 3 things you hope to get out of today on ‘post-its’

Speak with someone you have never met before today

Introduce yourselves and swap one of your ‘hopes’

Repeat this with two more people

• Then… …post your ‘post-its’ up on the wall.

Setting the ground rules…

Setting the ground rules…

Key Concepts

• Monitoring

• Evaluation

• Outputs

• Outcomes

• Impact

• Aims & Objectives

• Quantitative

• Qualitative

• Beneficiaries

• Stakeholders

• Sample/Sampling

• Extrapolation

• Value for Money

• Additionality

• Social Return

• Gross and Net

Logic chain example 1: This workshop

Caused by

Output

Need Activity

Impact Outcome

Limited ability to demonstrate

impactWorkshops and peer learning

People trainedHigher level of skills and

knowledge

Better demonstration of

impact

Lack of Monitoring and

evaluation knowledge

Rationale

Logic chain example 2: Streetwise

Caused by

Output

Need Activity

Impact Outcome

High level of teenage

pregnancyProviding C-cards

Young people accessing condoms

Less un-protected sex

Lower level of teenage

pregnancy

Poor access to contraceptives

Working in pairs

• Using templates provided, develop a ‘Logic Chain’ for a project you are involved with

• Share that with someone on your table

• Feed back to the workshop:

– What was easy, what was hard?

– Does it make sense?

Break

Capturing information from beneficiaries –What works?

• In groups consider…

–What are the different ways of capturing information from participants / beneficiaries?

– Identify some ‘pros and cons’ of each approach.

Capturing information from beneficiaries –Some thoughts

Questionnaires • Wide coverage• Tick box / narrative approach?• Accessibility• Self-completed or ‘managed’• ‘Survey Monkey’• Before / after / distance travelled• Confidentiality• Response rate

Capturing information from beneficiaries –Some thoughts

Capturing information from beneficiaries –some thoughts

Feedback forms• Immediate feedback from ‘captive

audience’?• More of / less of• Most useful / least useful• Improvements• Can be superficial and ‘too immediate’• Follow up again later?


In-depth interview• Richer content

• Particular insights

• Range of views / perspectives

• Resource intensive


Focus Groups• A group with similar characteristics

(e.g. single parents)• ‘Managing’ the group• Recording the outcomes

Round Table• Different perspectives


Comments / concerns / compliments

• Forms / Box?

• Managing the process

• The feedback loop

Timing issues…

• Captive audience

• Getting buy-in from the outset

• ‘When they’re gone they’re gone!’

• Bias?

• Incentives?

Storage issues…

• Paper based

• Spreadsheets

• Databases

• Bespoke approaches?

Questionnaire Design Considerations

• Questionnaire– Introduction / opening questions– Meaningful order / clear sections– Thank you / return information

• Questions– Numbered– Clear instructions– Routing– Layout

• Testing– Peer review; Cognitive interview; – Focus group; pilot/field test

Don’ts…

• Try to avoid…

…and do’s

• Try to avoid…

What more creative ways can you come up with to gather information from beneficiaries?

Outside the box…

Quantitative techniques

• Why numbers matter

• What numbers do we need?

• How to get useful numbers

• Cross-tabulation

• Sampling and extrapolating

• How to present data

Quantitative techniques

0

10

20

30

40

50

%

Reaction to the word “Spreadsheet”

Men

Women

n.b. fictional data!

Key numbers

The project has achieved significant positive outcomes, with 447 young people supported to become enterprise ready up to December 2011. To date, roughly 1 in 4 of these beneficiaries have gone on to establish a business, and around 30% have moved into education, training or employment.”

“An independent evaluation found that the service costed £145,000 over

three years, and had resulted in significant cost savings to statutory

services: conservatively estimated at £1,211,630.”

• How many beneficiaries?

• Of what sort?

• What happened as a result?

• What did it cost?

• How much did it save?

• Was it good value for money?

CDCF/ESF Requirements

• Filters

• Count

• Pivot Tables

• Cross Tabulations

Getting the numbers: 1) Excel

% of beneficiaries Under 25 26-50 Over 50 Total

Northumberland 0 5 15 20

Tyne & Wear 10 20 10 40

County Durham 15 10 0 25

Tees Valley 0 15 0 15

TOTAL 25 50 25 100

Count of Completed Column Labels

Row Labels Jan Feb Mar Apr May Jun Jul Aug Sep OctGrand Total

f 12 3 3 6 6 3 3 6 9 51

m 3 3 6 9 6 6 3 3 39

Grand Total 12 6 3 9 6 15 9 9 9 12 90

CDCF/ESF Data – Radar Charts

0

1

2

3

4

5

6

7

8

9

10Motivation

Confidence

Forms

CV

Interview

IT

Average

Before

After0

1

2

3

4

5

6

7

8

9

10Motivation

Confidence

Forms

CV

Interview

IT

Adriana

Before

After

Getting the numbers: 2) Other

• Survey Monkey

• Star Online

• Databases:

– Access

– Off the shelf systems

– Bespoke design

Sampling and extrapolation

• Most evaluation is based on a sample

• How representative is your sample?

• How diverse is your ‘population’?

• Evaluation is NOT a science

• Be aware of potential ‘bias’ in the sample

• Use caveats in extrapolating

Sampling example

0

20

40

60

80

100

120

1 2 3 4-6 7-12 12+

Months from completion

Length of time to find work

We found that 100% of respondents hadfound work within 3 months ofcompleting the course. If this is true forall beneficiaries, we will have helped 300people to find work”

Sampling thoughts

Try to get at least 10% of each part of the population:

• By geography

• By gender

• By age

• By time/stage

• Other characteristics…

To address bias in your sample:

• Use a variety of methods

• Ask searching/critical questions

• Triangulate with stakeholder views

• Pay attention to outlier views

Presenting Data – some tips

• Not too much information

• Order bars from high to low

• Use a variety of formats, but not too many

• Keep a consistent colour scheme

• Remember titles, axes and legends

• Keep it as clean as possible

For example

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Ethicity

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

White: British White: Other Other Ethnicity

Ethnicity Other Ethnicity

Mixed: White and Black Caribbean

Mixed: White and Black African

Mixed: White and Asian

Mixed: Other Mixed

Asian or Asian British: Indian

Asian or Asian British: Pakistani

Asian or Asian British: Bangladeshi

Asian or Asian British: Other Asian

Black or Black British: Black Caribbean

Black or Black British: Black African

Black or Black British: Other Black

Chinese or Other Ethnic Group: Chinese

Chinese or Other Ethnic Group: Other

Presenting Data – some tips

Use free tools, eg:

• Google Fusion Tables

• Tableau public

• Plot.ly

• Any others?

That’s quite enough about data

Lunch

Recap

Structure of the day: PM

14.10

15.00

Workshop feedback and close

Evaluation in practice

Wowing the World

13.00

15:30

‘Telling the story’ using case studies

13.30 Value for money

Telling the StoryUsing Case Studies

Spend a couple of minutes discussing with one or two others…

Things to consider when producing an ESF Community Grants Case Study

Case Studies: ESF Approach

About the person:

• Details of the client’s background, education/training and employment history and current situation.

Training/support:

• Details of what activity took place during the project

• What the client feels s/he achieved in regard to soft skills (e.g. more confidence) or hard outcomes (e.g. a qualification, employment, or volunteering).

Case Studies: ESF Approach

Plans for the future

• How does the client hope to progress their employability, training or job searching?

• Have they been referred to another organisation for ongoing support – if so which and what type ?

Quote

• A short quote from the client on how the project has made a difference.

Signature

Telling the StoryUsing Case Studies

• The power of the story

• The power of the actual words…

• … and a picture paints 1,000 of them

The Power of the Story

The Power of the Actual Words

• ‘If I didn’t have this I would have reoffended. I would have ended up getting drunk, fighting with someone and be back in prison.’

• ‘I felt miserable, I had no money and I was on benefits but now I’m independent. I feel good about working for my money.’

…and a picture…

Case Studies: A Quick Checklist

• Before you start

• Setting up the interview

• On the day

• After the event

• Other thoughts

Brandon’s Story

Value for Money

• What is the additional outcome impact?

• What is the cost per outcome?

• What is the return to society?

Gross No ofpeople into employment People into employment

who wouldn’t otherwise

Would have got a job anyway (deadweight)

Eligible people into employment who

wouldn’t otherwise


Inel

igib

le p

eop

le

(lea

kage

)

Additional spend locally (multiplier)

Additional eligible people

into employment who wouldn’t

otherwise

Net/additional impact

Inel

igib

le p

eop

le

(lea

kage

)


Oth

er p

eop

le w

ho

do

n’t

ge

t o

r lo

se jo

bs

(dis

pla

cem

ent)



otherwise

Net additional impact



otherwise

Net additional impact

Work though part 1 of the worksheet

Net additional impact Results

GrossOutcomes 250

Deadweighteffect(thosewho’dhavegotajobanyway) 50

GrossOutcomeslessDeadweight 200

Leakageeffect(ineligiblebeneficiaries) 10

GrossOutcomeslessDeadweightandLeakage 190

Displacementeffect(otherpeoplelosingjobsasaresult) 0

GrossOutcomeslessDeadweight,LeakageandDisplacement 190

Multipliereffect 19

GrossOutcomeslessDeadweight,LeakageandDisplacement,plusMultipliereffect(“NetAdditionalOutcomes”)

209

Cost per outcome

• Simple concept: TOTAL COST

NO. of OUTCOMES

• BUT: Gross or Net Outcomes?

• Total cost or Total Funding?

• See Part 2 of Workbook

Cost per outcome Results

TotalCostperOutcome(Gross) £3,000

ESFperOutcome(Gross) £1,350

TotalCostperNetOutcome £6,000

ESFperNetOutcome £2,700

Social Return

• The financial value to society of the outcomes

• e.g. moving someone off benefit into work:

– Benefits saved

– Taxes contributed

– Local spend

– Reduced risk of costs arising from ill health, crime, anti-social behaviour

Social Return

• Depends on lots of assumptions (rate of pay, length of employment, behaviour of the individual, age, personal circumstances etc)

• Can calculate/estimate on a case-by-case –difficult, lengthy and expensive

• Or use ready-made estimates e.g. Global Value Exchange

Global Value Exchange

• Free to join and use

• Open source estimates of the financial value of a huge range of outcomes

www.globalvaluexchange.org

http://www.globalvaluexchange.org

Social Return Results

Valuetoindividualsofsecuringwork £323,010

Valueofvolunteering £2,600

Valueofincreasedearningspotential(excludingthosewhohavesecuredwork)

£131,040

ValueoffiscalsavingstoDWP £277,020

TotalSocialReturnoftheproject £733,670

Social Return continued…

• Can then divide Total Social Return by Total Cost to get return on investment e.g. if Project Z cost £200,000, SROI is

£733,670 / £200,000 = £3.67

• Could have included other factors, e.g.

– Value of volunteering

– Increased earnings potential

– Improved health

SROI warnings

• Result depends on the factors included

• Also depends on assumptions made

• Similar projects making different assumptions will generate very different SROI scores

• Dealing with time properly is complex

• Doing SROI properly is expensive

• Most projects can demonstrate high positive SROI, so even an high positive result may not unlock funding

Developing an Evaluation Planfor your project

Developing an Evaluation Plan:Some things to think about

• Scope and Purpose

– What?

– Level – e.g. Project, Programme, Organisation

– Formative / Summative

– Audience?


• Timing and Phasing

– When?

– Single phase or multiphase?

• How do they fit together?

– Previous evaluations?

• The same approach or different?

– Key deadlines?

• ‘Doomed to success’


• Collecting Data

– What is required?

• Back to the logic chain

– Where can you get the data

– What have you got already?

– Storage and analysis

– Any barriers to using data?


• Who does it?

– Internal / External?

– Buy-in?

– Objectivity

– Budget


• Collecting views

– Options for beneficiaries?

– Key stakeholders?

– Options for Stakeholders


• Managing the evaluation process

– Who is responsible?

– Sign off?

• Dissemination

– Who and how?

Developing an Evaluation Planfor your project

Wowing the World

• Why are we doing this again?

– Because we have to?

– To improve our services to clients?

– To attract new clients?

– To demonstrate to ourselves and / or others that what we’re doing is effective?

– To show Value for Money?

– To convince people to give us some more?

Wowing the World

• Whatever the reason we need to ‘broadcast it’ in some way

• What approaches have you used that have been particularly effective in broadcasting what you’ve achieved and the impact that it has made?

Wowing the World

The word on the street

• Evaluation Reports

• Summary Documents– Selected highlights

• Newspapers / Newsletters

• Annual Reports



The writing on the wall…

Multi Media

http://www.actionfoundation.org.uk/about_history.html

http://www.actionfoundation.org.uk/about_history.html

Summary and questions

Your Evaluation!

• Any immediate thoughts?

• Evaluation sheet – Free prize draw!

• A more detailed conversation?

• Peer support network?

Date post:	16-Jul-2015
Category:	Government & Nonprofit
Upload:	robin-beveridge
View:	48 times
Download:	0 times

Presentation4 feb 15

Government & Nonprofit