RACHEL Welcome to the BBO ESF webinar on ESF Project Evaluation. The session will take around half an hour, presented by Liz. Liz is a Senior Research Manager at Ecorys and is experienced in designing a range of evaluation methodologies. Today there will be an opportunity to raise questions. We will share these afterwards on the website.

The webinar is being recorded, and will be available to view on our website hopefully later today, along with the slides- and our script. We have technical support on hand as well so please drop us a message if you are not seeing or hearing what you think you should be!

1

RACHEL Before we get going I’d like to remind you about the support we are providing through materials and webinars. Ecorys is responsible for developing a range of written materials for the BBO ESF programme. Hopefully you have seen our ESF manual. This will shortly been combined with the Fund’s Guide to European Funding so that all ESF guidance is in one place. Similarly we have streamlined our Helpline arrangements so our Helpline is no longer operational. If you have a query you should contact your Funding Officer or use the generic Fund BBO email. Our webinars are designed for organisations in stage 2 and early set-up. BLF keen to retain flexibility so we can respond to your needs and interests. So as well as giving us feedback on today’s sessions, please let us know what topics would interest you for future webinars. Today we have 25 participants and 21 organisations represented in this session. We’ll be running 2 polls to get a bit of information back from you, so you can get a sense of context from each other.

2

RACHEL This webinar is intended to provide guidance for projects on how to conduct project-level evaluations.

All BBO projects are expected to carry out a project-level evaluation, and set aside some of their budget for this.

You may be aware that Ecorys are conducting the overall programme evaluation, and may contact individual projects as part of that. This will look at learning around the best ways to support target groups and how the programme has widened access to European funding. I’d like to remind you about the CESI report on what makes a good employment project. This covers the context in which BBO projects are working, designing and delivering your project, supporting participants and working with employers. There is limited good quality evaluation of specific methods, so this is an opportunity to contribute to that bank of knowledge, as well as create hard evidence of the value of your work.

3

Today’s evaluation webinar will cover the reasons why evaluations are important, the two main types of evaluation, the different stages in the evaluation process and how to embed equality and diversity issues in your evaluation.

4

Often, people tend to mix up monitoring with evaluation, so I thought it would be useful to start the webinar by making a distinction between the two. Monitoring involves the routine collection of data linked to your targets. These are the targets agreed between you and the Big Lottery. This is used to measure progress towards these targets. Evaluations capture the strengths and challenges of a project, and explore how and why certain activities worked, whether outcomes were achieved or not, and the reasons why. Developing the right evaluation methodology for any intervention helps to identify whether these interventions work and why, they are crucial for making decisions on whether or not to continue with the activities developed under the project.

5

As mentioned in the previous slide, evaluations capture the strengths and challenges of a project. Questions that evaluations aim to address are set out in this slide. In the next few slides we will cover the two main types of evaluation.

6

Formative evaluations examine the extent to which planned activities are carried out.

7

Clear outcome measures linked to a project’s objectives need to be defined at the start of any evaluation. These measures can then tested through impact assessment surveys. You will need a combination of both evaluation types for your ESF projects.

8

There are 5 key steps that you will need to consider when designing your evaluation. These are also the steps that you need to look out for if you are either recruiting a consultant to evaluate your projects, or need to take into account when putting a specification together for your evaluation.

9

The first step in evaluating any project is to determine its intended outcomes, and the activities you expect to implement to help achieve these outcomes. This is done through a logic model. For illustration purposes, we have set out in the next slide a hypothetical logic model for employment based interventions designed for low-skilled/unemployed adults.

10

Rationale: provides an explanation about the issue that is being addressed by the project. It also argues in favour of implementing the project in the proposed area. Why this project? What issues are you trying to address? The logic model identifies the inputs and potential activities, outputs, results and outcomes of a BBO ESF project. It demonstrates how the project is likely to improve employment outcomes. It is particularly important that your evaluation covers outcomes. Whereas results and outputs are automatically captured as ESF data, the outcomes have to be captured by you. Your results will lead to your outcomes, these may be soft outcomes such as improved confidence, or hard outcomes relating to the number that have moved into employment, and had stayed employed for a particular duration. CESI made the point that actually there is very little evaluation of “what works” in ESF employment projects, underneath all the numbers, so this programme could try to change that.

11

Evaluation frameworks provide the basis for a structured and systematic approach to data collection, analysis and synthesis. The overarching question to keep in mind when developing questions for your evaluation framework is: ‘To what extent have activities delivered under Project A led to the desired outputs, results and outcomes’?

12

Some examples of questions that can be covered in the evaluation framework are set out in this slide. Most questions linked to a formative evaluation will focus on the planning, management and delivery of a project. Summative evaluation questions explore the outcomes and impact of a project. As most projects will involve a large number of delivery partners, it is important that partners are involved in the evaluation process, as active participants in providing feedback on the planning, management and delivery, and also in respect to having the right systems in place to collect information that will feed into the evaluation.

13

Once you have developed your research framework, you then need to consider how you will collect data to address the questions in the framework. What research method will you use to collect your evaluation data? Qualitative and Quantitative? Both methods are required to effectively evaluate your ESF project. For example, some projects establish participants advisory groups to ensure evaluation activities use language, terminology and imagery that is relevant to the target groups they are supporting.

14

Qualitative methods focus on exploring perceptions of beneficiaries, partners and the staff involved in the delivery of the project. Films/pictures to capture and record participants personal stories - sometimes a more visual approach will be needed as an alternative to the more traditional feedback forms – again you need to consider your target groups. Focus groups/workshops : think of the target groups concerned, will they be comfortable in expressing their views in a group setting?

15

Quantitative methods are used to measure changes as a result of the activities delivered. Questions are usually around ‘how many’, ‘how often’. Again, in this case, you need to consider the target groups being supported on your project. If your target groups are likely to have language barriers asking them to complete a questionnaire will not be appropriate.

16

The previous slide discussed the different types of data that can be collected for an evaluation. This slide looks at the levels of data collection. Micro – level e.g. numbers of unemployed people participating in the project; length of time they spend on it; numbers in employment immediately upon completion; number of employers involved Intermediate level – measures progress (including immediate destination as well as economic status in 6 and/or 12 months after completion of the project)

17

Usually in ESF project level evaluations there are three main sources for your evaluation data: project staff, participants and partners.

18

Consider how you will structure your report: How often? And in what areas do you want to focus your report on? Formative evaluation reports could cover: • Planning arrangements - such as setting clear and targeted objectives which are

agreed by all partners; • Procurement arrangements - the effectiveness of the allocation of the funding

across different partners; • Partnership set up arrangements and communication; and • Implementation - such as engagement of participants and delivery of learning

activities. Summative evaluation reports could cover: • the extent to which the project achieved its results, outputs and outcomes • potential for replicating good practice • Lessons learnt – what worked why and what did not work so well.

19

There are specific issues that will need to be explored in relation to equality and diversity in your ESF project evaluations, these are covered in the next two slides, and focus on the following themes: • Responsiveness of the project to the needs of the target groups • Accessibility • Participation • Achievements

20

21

RACHEL Please complete the short feedback form after this webinar, including your choice of further subjects to cover. Get in touch if you have any questions. Reminder that the webinar recording, these slides and our notes, and any questions that have been asked during the webinar, will all appear on our website shortly.

22