Dr Annalisa AlexanderHead of Outreach

Imperial College London

Workshop Aims:To understand and explore (in an hour):

Evaluation principlesImpact measurement

What… Why… How…

Case studies and sharing best practise

Group work

Please ask questions, share your thoughts and interact!

Definitions….Evaluation

“To Judge or assess the worth of…”Impact

“To have an effect upon….”You can evaluate a scheme in isolation

e.g. Are the participants communicating regularly? What topics are they discussing?

But you can’t easily measure impact without first evaluating the programmee.g. What effect is the programme having on raising aspirations? Are

mentees more likely to achieve higher grades as a result of being part of it?

EvaluationWhat is Evaluation?

Process Evaluation – Is it doing what it should be?Outcome Evaluation – Impact measurement - Has it achieved

what you intended?

Evaluation in a mentoring context:Critical to ensure the scheme is working

Are mentors and mentees interacting appropriately? Do boundaries need to be changed/set? Is the training sufficient? Do the mentors and mentees require more support?

EvaluationCase Study – medical e-mentoring scheme

Piloted in 2003 with a few mentoring pairs (range of ages in three schools)

Feedback from schools suggested Year 10 to Year 13 would be suitable

Evaluation in 2005 showed Year 10s were not engagedScheme then focused in on Year 11+2010 onwards, OFFA required Year 12 interactionSTEM-wide scheme launched 2012 following feedback from

schools and pupilsCurrently 250 mentoring pairs across STEM

www.imperialmentoring.org

EvaluationWhy is evaluation necessary?

Funders often require itExternal bodies (e.g. OFFA) require itInternal rigor should demand it!

Gathering feedback is meaningless unless you are prepared to do something with it….

Evaluation How do we do it?

Keep It SIMPLE!Keep it SHORT!Keep it quantitativeFree text is hard to interpret but can be usefulEncourage participation through prizes or incentivesPre and post course evaluation works well – as long as the

questions are comparable…. Be clear about what you are trying to find out though – a good

evaluation form has a strong focus but beware of bias in your questioning

Evaluation – Group Work The Brief:

You have been running a mentoring scheme for sixth formers at a local school for 18-months to 2 years

The purpose of the scheme was to help them make choices about careers/university

Another school has expressed an interest in the scheme but wants to know what is involved in running it

Create a short evaluation formNo more than one side of A4Remember that you should be able to turn the answers into a

coherent report on the nuts and bolts of the scheme

Impact MeasurementWhat is Impact measurement?

Demonstrating through your evaluation work that your scheme has made a measurable difference

Impact measurement in a mentoring context:Critical to ensure the scheme has done what you intended it to

do The mentors’ and mentees’ expectations have been met The mentors/mentees have grown in confidence and understanding

If the scheme hasn’t achieved what you intended – how can you improve or change it? Feedback is crucial (ideally 360 if possible!)

Impact MeasurementWhy is Impact measurement so important?

Often a requirement of funders/partner organisationsEnsures that your scheme is on track (E.g. OFFA)Allows you to make critical changes along the way if it is not

meeting expectations or hitting targets

How do we measure impact?Keep it simple!Its easy to develop complicated multi-faceted methods but just a

basic demonstration of cause and effect will often suffice

Impact Measurement

Ref: National Council for Voluntary Organisation

Impact MeasurementCase Study – Imperial STEM Enrichment Programme

A series of tailored AS level STEM and personal effectiveness workshops for high achieving Year 12 students

All from disadvantages backgrounds, under represented in Higher Education

First cohort was 96 Year 12 students from across LondonFully funded through Access moneyRigorous evaluation and impact measurement is required

Impact MeasurementQ1: How would you rate the event overall? 95% rated the event Good or Very good

Response Number of responses

Adequate 21 Good 191 Very Good 210 Very Poor 1 Grand Total 423

5%

45%50%

0%Adequate

Good

Very Good

Very Poor

Q2: I now feel more motivated about my studies. 88% feel more motivated about their studies

Response Number of responses

Agree 234 Disagree 2 Not Sure 41 Strongly Agree 139 Strongly Disagree 5 (blank) 2 Grand Total 423

55%

1%10%

33%

1% 0%Agree

Disagree

Not Sure

Strongly Agree

Strongly Disagree

Impact MeasurementQ3: I understand the importance of studying the subject of this event. 92% agree or strongly agree

Response Number of responses

Agree 216 Disagree 4 Not Sure 25 Strongly Agree 173 Strongly Disagree 2 (blank) 3 Grand Total 423

51%

1%6%

41%

0% 1% Agree

Disagree

Not Sure

Strongly Agree

Strongly Disagree

(blank)

Q4: I will consider applying to courses related to the subject of this event at University. 91% believe they will consider applying to courses related to the subject at University.

Response Number of responses

Agree 156 Disagree 4 Not Sure 24 Strongly Agree 230 Strongly Disagree 5 (blank) 4 Grand Total 423

37%

1%6%

54%

1% 1% Agree

Disagree

Not Sure

Strongly Agree

Strongly Disagree

(blank)

Impact MeasurementQ2-5: I am more likely to go on to University. 84% believe they are more likely to go to university following the event

Response Number of responses

Agree 139 Disagree 14 Not Sure 41 Strongly Agree 216 Strongly Disagree 4 (blank) 9

Grand Total 423

33%

3%10%

51%

1% 2% Agree

Disagree

Not Sure

Strongly Agree

Strongly Disagree

(blank)

• The impact of this scheme was therefore: Participants felt more confident in their studies Participants felt more motivated to continue on to Higher

Education Participants felt they were now more likely to apply to study a

course relating to the subject they had received support with

Impact Measurement – Group Work

The Brief: A corporate partner has expressed an interest in your mentoring

scheme and would like to potentially offer three years fundingThey have requested a more detailed report that looks at the

impact your scheme has on pupils’ decisions and choices post A-level

Now revisit your evaluation formWhat do you need to change/add to get a more detailed picture

of impact?What measurements are you interested in?

e.g. how will you measure change in aspiration?

Pitfalls….Over complicated evaluation forms lose effectivenessA danger of investing too much time and resource

You can sink a lot of money into it….!Bias in designing tools - don’t just focus on things you know

will give good feedbackAsking questions that are meaningless or can’t be

interpretedAvoid free text unless you wish to use comments later

Data protection issues – always be sure to state what you intend to do with the information

ReferencesRhodes, J (2002): Gauging the Effectiveness of Youth mentoring. The

National Mentoring PartnershipLee, C (2012): Keeping Impact Measurement Simple. Third Sector (

www.thirdsector.co.uk) National Council for Voluntary Organisations (www.ncvo-vol.org.uk) Parsons et al (2008): Formalised Peer Mentoring Pilot Evaluation.

Canterbury Christ Church University (Research Report DCSF-RR033)Garbarine, S & Holland, J (2009): Quantitative and qualitative methods in

impact evaluation and measuring results. Governance and Social Development Resource Centre

Berk et al (2005): Measuring the effectiveness of faculty mentoring relationships. Academic Medicine, Vol. 80, 1.

Ref : Dilbert.com, Scott Adams