East Lothian Learning Partnership Conference

Demonstrating Impacts and Outcomes – Is it really all that important?

Ann Kivlin & Colin Forbes, CLDS5th December 2011

•

Session Aims

Highlight some of the advantages of being able to demonstrate impacts and outcomes

Explore:WHY we need to do it, WHEN we should do it, WHAT is required in order to do it effectively WHO has responsibility.

Explore and discuss a range of tools and techniques

• Highlight the advantages of being able to demonstrate impacts and outcomes

How confident are you?

Extremely confidentMore than confidentConfidentLess than confidentFind it a challengeNot confident at all

• Hands up how many of you feel confident enough to facilitate a session on this topic with your staff teams?

•

Highlighting the advantages of being able to demonstrate impacts and outcomes

mins

• Settle in and intro 5

• Why, When, What, Who 40

• Building your Capacity 10

• Tools and Techniques 10

• Questions 10

WHY we need to demonstrate impacts and outcomes?WHEN we should be demonstrating impacts and outcomes?WHAT is required in order to do it effectively and demonstrate impacts and outcomes? WHO has the responsibility for demonstrating impacts and outcomes?

Group Task:

In 4 groups spend 5 minutes considering one of the questions and then feedback your groups 3 most important answers

WHY - A list we prepared earlier! Value for moneySpending Public PoundAccountabilityTransparencyMeasure improvementReport progress to stakeholderProof the work makes a difference

WHEN - A list we prepared earlier!

When you feel like it!For every piece of work?Significant pieces of workBest bits!Worst bits?Regularly?When your line manager requests!

WHAT - A list we prepared earlier!

Ability to prove what you are doing is really making a differenceUse a range of effective toolsAbility to measure before and after and monitor performanceAgree what is good evidenceProvide the evidenceConfidence

WHO - A list we prepared earlier!

It’s everybody’s responsibility!Lead person

Building your Capacity in demonstrating Impacts and Outcomes

To increase staff understanding and

confidence in monitoring and evaluating activities through constructing clear outcomes,

indicators and methods which can demonstrate impact of their

engagement.

27 SLIDES!!

Your Thoughts?

•What do you think when you hear the word ‘evaluation’?

What are outcomes and why are they

important?

Discussion

The definition!

Outcomes are the changes or difference you want to make

through your services or activities (outputs)

Why do outcomes matter?

Focus: You need to know what difference you want to make before you can evaluate to see if you have.

Expectations: The people who use your services will know what to expect from you

Motivation: Staff satisfaction is increased when they know they make a difference.

Accountability: Many funders want to fund or purchase outcomes. They want to know what difference their money makes and how it helps target groups have a

better life.

The science bitOutCOMEs are the changes and differences that come out of your activities.

OutPUTs are the activities or services you put on for your users.

Inputs are the resources you put in to do the work: money, people, time, premises, equipment.

input output outcome impact

Outcomes are lazyOutcomes don’t happen by themselves

They need activities, services, people

The output makes the outcome happen

outcomeoutput

RelevanceOutcomes must be relevant to the activity

What you do (activity) must link to your outcome in a logical way

Outcomes are the changes or difference you want to make through your services or activities (outputs)

GOOD words LESS SPECIFIC words

 e.g. improve e.g. support

 

 

 

DEFINITION

Outcome words

decrease improvemaximise minimise

better expandreduce enhance

more lessproduce increase

strengthensustain

- not exhaustive list- about change or difference

These words don’t show change!!

Support Assist

Engage

Encourage

Enable

Help Offering

Seeing

Watch out for verbs (doing words) – they are about activity not changeBUT you could include these words as long as you also describe the change e.g. – ‘INCREASE levels of community participation by OFFERING crèche facilities to local families’.

Expressing the outcomeis changing (e.g. the service user, beneficiary, organisation, community etc.)

Who

What

How it will change (e.g. increase, improve, reduce, etc.)

is changing (e.g. knowledge, skills, environment, feelings etc.)

Examples of outcomesWHO WHAT HOW

• Parents’ confidence is increased

• Young people have reduced, stabilised or eliminated their drug use

• Communities are better represented in decision making

Examples of outcomesWHO WHAT HOW

• Older Asian women have better access to community health and social care services

• There is an increase in the understanding of the recycling procedure within the office staff.

• Young carers are able to make informed choices about the future

The ‘so what’ test• I deliver an Employability Course at Ross High

School - SO WHAT?• I work with a group of 16 S4’s - SO WHAT?• The group has visited colleges and taken part in

indoor climbing sessions - SO WHAT?• The group have learned what courses are

available at local colleges and how to access these courses OUTCOME

• The group have gained confidence and improved communication & team working skills through indoor climbing sessions OUTCOME

All important … but not all outcomes

Weaver’s Triangle

AIM

Outcomes = change Skills

Knowledge Behaviour Feelings

… SO WHAT?

OutputsActivities and Services

the main things you do with your users / clients

Outcomes and impactLochniven Young Carers

Impact (aim) – ‘Our young people are successful learners, confident individuals, effective contributors and responsible citizens’

Outcomes need to be meaningful and realistic steps on the way.

ExampleLochniven Young Carers Project• Short term: Young carers know where to

go for support & are confident to do so • Medium term: Young carers are less

isolated and better able to cope• Longer term: Young carers participation

and attainment at school is improved Impact – Our young people are successful

learners, confident individuals, effective contributors and responsible citizens

The outcome testOutcomes

1. Are about change or difference.

2. Don’t happen by themselves – they are not the activity – they are the result of the activity.

3. Must link logically to the activity.

4. Use words like improve, decrease.

5. Answer the ‘so what?’ question.

6. Should be realistic – achievable in a few years.

7. Should be (largely) in your power to deliver.

8. Should be simple – not too many (3-5)!

Collecting information

• Monitoring what you do

(e.g. numbers, range of activities)

OUTPUTS

AND

• Evaluating whether your activities

are delivering your OUTCOMES

Measuring outcomes

Thinking about each outcome:

What would it look like if we achieved it?

• These are your outcome indicators

• What are the 2-3 most important (and most likely) indicators for each outcome?

• Involve users and stakeholders.

• Your indicator list =

your evaluation plan

Getting it right

1. Measure indicators more than once to show change.

2. What is your baseline (the starting point) for the service and for individuals (from referral info, assessment, evidence of need etc)

3. Don’t collect information on all outcomes all the time.

4. Do you need to ask everyone or can you sample?

5. Follow up some users if you can to find out what happened next.

Getting it right

6. Use one system – but a range of methods.

7. Build into planning – and if possible activities.

8. Test your methods.

9. Be honest and involve clients.

10.Record unexpected outcomes

(e.g. 6 volunteers got a job)

Common sources of info on indicators

Assessment

Observed behaviour changes

Hard evidence

Records

Third party tells you

They tell you

Outcome

Methods for collecting info

Questionnaires Interviews Observation Group structured feedback Wall charts Visual progress tools Mapping Third party feedback (Video) diaries

Outcome Evaluation Plan

Outcome Indicators Methods for collecting information

When to collect info

Who is responsible

Older Asian women have better access to community health and social care services

• Older Asian women know what community health & social care services are available in the local area

• Older Asian women know how to access community health & social care services

• Older Asian women receive more community health & social care services

• One-to-one mapping with the women to find out what services they know about at the start & end of the 1 year project

• Group interviews (focus groups) with the women

• Individual interviews with the women

• Interview a range of health and social care providers to find out if uptake of services has increased

The evaluation pathwayGetting Started - understanding what you want to achieve. Setting and agreeing your outcomes

Collecting information - what are you going to measure and how? Getting systems in place

Analysing and Reporting - making sense of what you’ve collected and reporting on progress and learning.

Learning from your findings - What did you learn? How will you do things differently? Any unexpected results?

Acknowledgement:Evaluation Support Scotland

Tools & Techniques

Wide range of evaluative tools Electronic Project Files VOICE National Standards for Community

Engagement

Presentation adapted by

I n tro d u c in g :

What is VOiCE?

VOiCE is online planning and recording tool designed to assist individuals and organisations to design and deliver effective community engagement.

Why use VOiCE?• It’s simple to use

• Provides evidence on meeting the National Standards for Community Engagement

• You don’t need previous knowledge of the National Standards or experience of community engagement to use VOiCE

Online link to the National Standards for Community Engagement: www.scdc.org.uk/what/national-standards/10-national-standards/

How can VOiCE help you?

• Analyse the need for community engagement

• Plan your engagement • Do it – implement your

plan effectively• Review the outcomes

VOiCE enables you to monitor and record the process

1. Analyse

VOiCEHome

2. Plan

4. Review 3. Do

VOiCE Sitemap

1. Are we doing it?

2. Section checklist

1. What outcomes are we seeking?

2. Are there any barriers?

3. What resources are available?

5. Section checklist

1. Why do we want to engage?

2. What do we want to know?

3. Who are the stakeholders?

4. Section checklist

5. Section checklist

3. Did we succeed?

2. Have we met the Standards?

1. How was the evidence collected?

AnalyseReport

DoReport

PlanReport

ReviewReport

OverallEngagement

Reports

4. What methods will we use?

4. What have we learned?

Features of VOiCE

• Multi user environment

• Secure password protected accounts

• Provides reports

• Strategic analysis

Want support to use VOiCE?

VOiCE training is supported by East Lothian’s Community Engagement Strategy action plan which is available online at:

www.eastlothiancommunityplanning.org.uk/engagement

Key contact: Meriel Deans, Public & Community Involvement Co-ordinator

This is a shared post between East Lothian Community Health Partnership and East Lothian Community Planning Partnership

Email: meriel.deans@nhs.netTel: 0131 536 8025 or 07738 075232

• Further discussion

• 10 minutes

• Highlight the advantages of being able to demonstrate impacts and outcomes

How confident are you?

Extremely confidentMore than confidentConfidentLess than confidentFind it a challengeNot confident at all

• Hands up how many of you now feel confident enough to facilitate a session on this topic with your staff teams?

• Free copy of these slides for every correct answer!