September 2017 Wales Ex Post Evaluation Priority Review Paper · The Programme Monitoring Committee...

Ymchwil

Research

September 2017

Wales Ex Post Evaluation

Priority Review Paper

Supplying young people with skills for learning and employment

ESF Convergence Priority 1

ob3research.co.uk

ob3research.co.uk Company Registration Number: 5565984 1

September 2017

Wales Ex Post Evaluation

Priority Review Paper

Supplying young people with skills for

learning and employment

ESF Convergence Priority 1

Browerdd, Heol yr Ysgol

Llanarthne, Sir Gaerfyrddin

SA32 8HJ

01558 822 922

[email protected]

38 Llewelyn Goch

St Fagans, Cardiff

CF5 6HR

029 2089 1724

[email protected]

ob3research.co.uk Company Registration Number: 5565984 2

Author: Nia Bryer

Quality Check: Huw Bryer

Version: FINAL

ESF Convergence Priority 1 Priority Review Paper

3

Contents Page

1. Introduction: Scope of Priority 3

2. Intervention logic and the changing policy context 5

3. Funded projects: ‘fit’ with intervention logic 9

4. Targets and achievements 13

5. Cross cutting themes 16

6. Evaluation evidence: common barriers and success factors 20

7. Evaluation evidence: impact 25

8. Evaluation evidence: other outcomes and unintended consequences

37

9. Availability and quality of evaluation evidence 39

10. Conclusions 45


4

1 Introduction: Scope of Priority

The aim of Priority 1 of the European Social Fund (ESF) Convergence Operational

Programme (Supplying young people with skills for learning and future employment)

was to ‘prepare young people for future employment by raising aspirations and

increasing participation in learning’1.

Priority 1 was focused upon supporting those aged between 11 and 19 years of age

who were considered to be ‘at particular risk of under-achieving or becoming Not in

Employment, Education or Training (NEET) , for future employment by raising their

aspirations, and increasing participation in learning or training.’2 It was explicitly not

made available to fund mainstream educational provision although it was expected

to add value to mainstream provision by ‘using innovative approaches to intervention

which engage young people’3. Most of the funding (70 per cent) within this Priority

was intended to be directed towards young people aged 14 to 19 years old.

Sitting beneath this Priority were two themes, namely:

Theme 1 – Tackling Under Achievement (Closing the gap) which was

intended to focus on improving the educational outcomes of young people at

risk of underachieving. This theme had two clear objectives, namely

combating disaffection and reducing the risk of early school leaving as well as

meeting the specific needs of disadvantaged groups such as those with

special educational needs, young offenders and people from Black and

Minority Ethnic (BME) communities. It was expected that this theme would

account for around 70 per cent of the resources allocated to this Priority

Theme 2 - Raising Skills and Aspirations (Raising the bar) which was

intended to focus on supporting young people to make a successful transition

into employment. This theme had three distinct objectives namely providing

1 Welsh European Funding Office (2009) ‘West Wales and the Valleys Convergence Programme – Operational

Programme For The European Social Fund 2007-2014’ p.115 2 Welsh European Funding Office (2009) ‘West Wales and the Valleys Convergence Programme – Operational

Programme For The European Social Fund 2007-2014’ p.88 3 ESF Convergence Priority 1 Supplying Young People with Skills for Learning and Future Employment Strategic

Framework p.9


5

enhanced information and guidance on employment, vocational training and

higher education to disadvantaged young people; providing access to a

broader and innovative range of learning options to extend and strengthen

their learning experiences and skills; and promoting and supporting the

acquisition of higher levels skills amongst disadvantaged young people. It was

expected that this theme would account for around 30 per cent of the

resources for this priority.


6

2 Intervention logic and the changing policy context

The key underlying rationale for Priority 1 was:

to ensure a skilled and competitive workforce for the future

to equip young people with appropriate life skills for flexible career patterns

that early intervention would lead to higher levels of education attainment and

longer term employment outcomes

that levels of literacy and numeracy in young people remained low

that enabling young people to successfully transition into further or higher

education would lead to a greater choice of learning pathways

that specific groups of young people had particular barriers and needs that

required targeted intervention

that some young people did not make appropriate career choices thereby

reducing their future earning potential

a need for more young people to study science in order to support the

knowledge economy and the needs of employers.

A logic model for this Priority 1 is presented at Figure 1:


7

Figure 1: Logic Model for ESF Convergence Priority 1

Whilst tackling youth engagement and employment had been a common theme

within previous European funding programmes in Wales, the introduction of a

specific priority focused on 11 to 19 year olds represented a major change in

recognising the importance of providing preventative interventions for young people

for the first time.

Its introduction came following the general response received as part of the public

consultation exercise and was highlighted as a ‘welcome addition to the

Programme’4 within the ex-ante evaluation of the 2007-2013 programme. Indeed, the

4 DTZ Ex Ante Evaluation of the 2007-2013 ESF Convergence Programme Final Report (March 2007) p.14


8

ex-ante evaluation specifically noted that there had been an increase in the number

of young people, particularly young men, not in employment, education or training

during 2002-2003 which justified the inclusion of this new Priority within the

programme. Furthermore, the inclusion of this new Priority was supported by the fact

that the attainment of pupils in the final year of compulsory education in the

Convergence region was below the UK and Welsh average.

ESF Convergence Priority 1 was delivered within the context of a fairly stable policy

environment. At the time of its drafting, the Priority was driven by the policy context

set out in One Wales – A Progressive Agenda for the Government of Wales, Wales:

A Vibrant Economy, Wales – A Better County and The Learning Country 2: Vision

into Action. It was also informed by a number of specific educational related policies

such as those set out within Reaching Higher5, Extending Entitlement6, 14-19

Learning Pathways7 and the Skills and Employment Action Plan for Wales 20058.

The Programme Monitoring Committee (PMC) Strategic Framework Summary

Report9 suggests that there had been no major shift in policy relating to young

people since the Strategic Framework was drafted. The original set of proposed

activities therefore remained unchanged although intervention rates were increased

from 54.36% to 65.43%. A review of approved projects under ESF Convergence

Priority 110 suggests that funded provisions were well aligned with the underlying

objectives of the Programme and the wider policy context.

The development and approval of Priority 1 projects has been well documented via

the Annual Implementation Reports (AIR), which spanned 2007 – 2014, and

5 http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/further_and_higher_education/reaching_higher/?lang=en 6 http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/extending_entitlement/?lang=en 7 http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/learning_pathways/?lang=en 8 http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/127035/skills_and_employment_action_pla?lang=en 9 WEFO Programme Monitoring Committee Strategic Framework Summary Report

10 http://gov.wales/funding/eu-funds/previous/searchprojects1/?view=Search+results&lang=en

http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/further_and_higher_education/reaching_higher/?lang=en

http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/further_and_higher_education/reaching_higher/?lang=en

http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/extending_entitlement/?lang=en

http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/extending_entitlement/?lang=en

http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/learning_pathways/?lang=en

http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/learning_pathways/?lang=en

http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/127035/skills_and_employment_action_pla?lang=en

http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/127035/skills_and_employment_action_pla?lang=en

http://gov.wales/funding/eu-funds/previous/searchprojects1/?view=Search+results&lang=en


9

generally whilst pointing to a somewhat slow start for the priority shows that good

progress was made during the second half of the programme. During the first two

years, the level of interest demonstrated in this Priority was good with some 100

Project Idea Forms (PIFs) submitted over the course of 2007 and 200811. The

majority of these PIFs were focused on Theme 1 and on addressing the issues

facing NEETs.

The first Priority 1 projects were approved during 2009 with a total of eight projects

having been approved in that year with 53% of the EU grant available for this Priority

committed by this date. By the close of 2009, 1,345 participants had been assisted

via Priority 1. The 2009 AIR notes that the expectation to deliver more strategic,

integrated and collaborative projects meant that ‘some projects have not progressed

to full approval as quickly as originally envisaged and the start of delivery had, in

some instances, been delayed because of longer than anticipated procurement

procedures’12.

The number of approved projects doubled by 2010 (to 16) and increased further to a

total of 20 projects by 2013, which by 2014, accounted for 100% of the EU grant

available for this Priority.

By the end of 2013, the number of participants engaged stood at 107,926 with

18,930 having gained qualifications. The 2013 AIR reports that some projects had

been incorrectly claiming participants falling outside the eligibility requirements for

the Programme and Priority and that it had been necessary for these projects to

undertake a data cleansing exercise. The 2014 AIR reports that Priority 1 ‘continues

to perform ahead of expectations’ and half of funded projects within the priority had

been completed or closed by 2014. By the end of 2014 121,508 participants had

been assisted and 22,222 gained qualifications.

11

West Wales and the Valleys ESF Convergence Programme 2007-2013 Annual Implementation Reports 2007 and 2008 12

West Wales and the Valleys ESF Convergence Programme 2007-2013 Annual Implementation Report 2009 p.72


10

3 Funded projects: ‘fit’ with intervention logic

In all, 20 projects were approved via Priority 1 of the 2007-2013 European

programme and were awarded funding totalling £129.7m. A total of 13 were funded

via Theme 1 and 7 were funded via Theme 2 of this Priority:

Figure 2: Priority 1 approved projects

Project title Lead sponsor Theme

Regional SEN Transition To Employment Initiative

Caerphilly County Borough Council 1

A KS 3 Intervention Blaenau Gwent County Borough Council 1

Pupils Understanding Problems in their Locality

Cynon Valley Crime Prevention 1

Reach The Heights Routes To The Summit

WAG DCELLS Lifelong Learning and Skills 2

Reach The Heights First Footholds WAG DCELLS Lifelong Learning and Skills 1

Technocamps Swansea University 2

Building The Future Together Rhondda Cynon Taf County Borough Council 1

Gypsy Traveller Learning and Future Employment Project

Pembrokeshire County Council 1

Young BME People Aiming High All Wales Ethnic Minority Association 1

Amlwch Skills 16 plus Hyfforddiant Parys Training 2

Minority Ethnic Learning and Achievement Project


Think Differently Priority 1 ESF Convergence

Fairbridge De Cymru 1

Llwyddon Lleol Creating a Culture of Enterprise amongst YP

Cyngor Gwynedd Council 2

Engage Neath Port Talbot County Borough Council 2

Pathways to Apprenticeships WAG DCELLS Lifelong Learning and Skills 2

Prevent 14 - 19 Bridgend County Borough Council 1

Stem Cymru Engineering Education Scheme Wales 2

Potensial Cyngor Gwynedd Council 1

World Class Traineeships with engagement for Young People


Raising Skills and Aspirations of Young BME People II

YMCA Swansea 1

http://gov.wales/funding/eu-funds/previous/searchprojects/80080?lang=en






























11

Source: Welsh Government http://gov.wales/funding/eu-funds/previous/searchprojects/?view=Search+results&lang=en accessed 12 April 2016

A review of funded projects’ objectives suggests that most of the proposed activities

within this priority were well covered and that interventions attempted to deliver

unique provisions which intended to complement mainstream educational provision.

Several projects were specifically focused upon key target groups identified within

the programme documents, particularly sub-groups such as young offenders, young

people from Black and Minority Ethnic (BME) communities (three projects were

specifically focused on this group) and young people from the traveller community. It

was also the case that two projects were specifically focused on STEM13 subjects. It

appears that an appropriate balance was achieved between projects which were

focused upon supporting young people who were already inactive and those which

were focused on supporting young people still in education (with an objective of

improving school attendance). Other funded projects took a more generic approach

in that they were intended to support more than one area of the indicative activities

proposed within the Operational Document – for instance, Potensial aimed to work

with a large number of participants to ‘engage, re-engage, motivate and inspire ...

through a series of activities which are both innovative and established through three

linked strands’14.

Two of the proposed activities, notably creating linkages to support young people

into higher education and using mentors as positive role models in schools to raise

awareness of the benefits of higher education, do not appear prominently within

project objectives. Indeed, there was no specific result indicator relating to

progression into higher education for this priority, which may have restricted the

development of activities in this area.

Furthermore, it is questionable whether the indicative activity contained within the

Operational Document relating to funding ‘initiatives which promote improved

understanding of sustainable development’ was well covered by funded projects,

given that no single project was specifically focused upon this objective – although

several projects intended to embed this as part of their provision.

13

Science Technology Engineering and Mathematics 14 http://gov.wales/funding/eu-funds/previous/searchprojects/80625?lang=en accessed 12 April 2016

http://gov.wales/funding/eu-funds/previous/searchprojects/?view=Search+results&lang=en

http://gov.wales/funding/eu-funds/previous/searchprojects/?view=Search+results&lang=en



12

The key conclusions made within the Thematic Evaluation15 in respect of this

Priority’s fit and design were that:

Projects funded varied in terms of their aims and objectives, size, complexity,

geographical coverage, intended target groups and project sponsor

Projects were positively utilising ESF funding to test innovative approaches

which were beginning to have a long-term impact upon provision within

mainstream services

Projects included provisions which focused on building capacity, regional

collaboration, young people’s participation and mainstreaming the cross-

cutting themes

Projects were also intended to complement mainstream provision and set out

to increase the capacity of existing services and influencing mainstream

provision to be more responsive to the needs of certain young people

The delivery of more strategic, larger projects was considered to have been

worthwhile and necessary

The involvement of local authorities in the delivery of the Priority was

considered to have been beneficial

The high demand for projects’ activities was evidence that they were filling

gaps in services – particularly in terms of support for young people with

mental health issues, social and emotional wellbeing and self-esteem

Projects were targeting the groups of young people whom were expected to

be supported although it proved difficult to assess whether participants were

receiving a ‘meaningful interaction’ given the relatively limited data which was

available on this.

Furthermore, the Thematic Evaluation of Priority 116 concluded that:

15

GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’ 16

GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’


13

‘Evidence from projects suggests that ESF funding has resulted in many projects

trying innovative approaches; testing successful approaches at scale or extending /

developing existing provision.’

It also went on to argue that:

‘There is evidence to suggest that many projects are effectively complementing

mainstream provision. Some projects are starting to help raise the capacity of

other services and influence mainstream provision to be more responsive to the

needs of certain young people who have previously been overlooked’17.

The Thematic Evaluation found that Priority 1 had been effective in promoting ‘joint

working across organisations with different agendas and in different localities’ and

that ‘projects that built on previous experience of ESF funded interventions were

characterised by more effective partnership arrangements and buy-in’18.

17

GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’ p.55 18

GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’ p.14 and p.18


14

4 Targets and achievements

Figure 3 presents Priority 1 Indicators and Targets and the final outturn achieved:

Figure 3: Priority 1 Indicators and Achievements as at August 2016

Indicator Original

Target

Revised

Target

Achieved

Outputs

Total participants (11-19 year olds) 35,000 35,000 134,031

Female participants 40% 40% 46%

Employers collaborating with education/training

providers

500 500 2,201

Systems developed / / 5

Projects using soft outcome measurement

systems

50% 50% 60%

Projects integrating sustainable development into

awareness raising, education and training

programmes

75% 75% 55%

Results

Participants gaining qualifications (11-19 year

olds)

10,500 10,500 25,143

Participants entering further learning – 16-19 year

olds

21,000 21,000 12,357

Participants gaining other positive outcomes19 –

11-19 year olds

31,500 31,500 35,760

Participants entering employment (16-19 years

old)

/ / 4,617

Source: Welsh European Funding Office (August 2016)

In line with performance across the ESF programme overall, it is worth observing

that the Priority participant output was substantially exceeded (nearly four-fold) with

19

Defined as intermediary outcomes such as completing courses, entering voluntary work and attending a job interview.


15

134,031 participants engaged against a target of 35,000. All other output targets

were also exceeded other than the proportion of projects which integrated

sustainable development into their provision, at just over half doing so. This is clearly

a very impressive performance by projects in engaging such a large number of

participants over the course of delivery – the baseline data20 suggests that 248,500

participants aged between 11 and 19 years old formed the baseline for this Priority,

hence the Priority would have reached and engaged just over half of this overall

population.

In terms of performance against its Results Indicators, the Priority performed

exceptionally well having exceeded its targets of participants gaining qualifications

and participants gaining other positive outcomes. However, the proportion of all

participants assisted who gained a qualification or other positive outcomes formed a

fairly low proportion of the total number of participants supported (i.e. only 19 per

cent of all participants gained a qualification whereas the original indicator suggest

that 30 per cent would do so).

The Priority did not achieve its target for the number of participants entering further

learning having only achieved 59 per cent of its 21,000 target. This is particularly

disappointing given that the Priority assisted a much higher number of participants

than was originally set. No targets were set for the number of participants entering

employment. However it is encouraging to observe that 4,617 did so. If we were to

amalgamate all of the results achieved (and assuming that no participant would have

achieved more than one result each) then 42,117 (or 31 per cent of all participants)

would have either secured a qualification, entered further learning or secured

employment.

As set out at Figure 4 the original budget allocation for Priority 1 stood at just over

£216m, accounting for around 15 per cent of the overall WWV ESF Convergence

Programme funding for 2007-2013. Actual final expenditure, at £197m, was within 91

per cent of the original budget allocation and final EU Grant expenditure, at £105m,

was within 90 per cent of the original budget allocation.

20 Operational Programme Annex K p.4


16

Figure 4: Priority 1 Funding

£ Total Funding EU Grant

Original Budget Allocation 216,336,164 117,807,004

Revised Budget Allocation 157,387,912 106,681,805

Cumulative (Actual) Expenditure 197,372,079 105,149,583

Source: Welsh European Funding Office

When considering levels of investment per output compared with what was

envisaged Figure 5 shows that the actual investment per participant engaged was

four times lower than anticipated, at £1,473 per participant, thereby providing better

value for money – although the original budget, at £6,181 per participant, would

appear high. Similarly, the actual investment per participant gaining qualifications

was much lower than planned whilst the investment per participant gaining other

positive outcomes was marginally lower. However, the actual investment per

participant entering further learning, at £15,973 per participant, was much higher

than expected.

Figure 5: Original and Actual Investment per Outputs

Original

Budget

Allocation

£

Final Actual

Expenditure

£

Investment per participant engaged 6,181 1,473

Investment per participant gaining qualifications 20,603 7,850

Investment per participant entering further learning 10,302 15,973

Investment per participant gaining other positive

outcomes

6,868 5,519

Investment per participant entering employment n/a 42,749

Source: OB3

5 Cross cutting themes


17

This Priority was expected to incorporate the two cross cutting themes of Equal

Opportunities and Environmental Sustainability. An evaluation focused upon the

implementation of the Programmes21 concluded that funded projects had integrated

the Cross Cutting Themes (CCTs) more thoroughly into Project Business Plans at

the application stage than in previous Programmes, but that the extent to which

commitments made had been undertaken on the ground were more mixed.

The CCTs evaluation commissioned by the Welsh European Funding Office (WEFO)

22 (while not being specific to ESF Priority 1) found more generally that:

The lack of incentive was the single most reported barrier for all projects in

achieving their CCTs23

ESF funded projects were more likely to report ‘lack of training for Sponsor’,

‘WEFO change targets’ and ‘Partners not supporting CCT work’ as barriers to

achieving their CCTs24

The CCTs were not considered a ‘top priority’ due to a perception that ESF

participants did not want to engage with CCTs25.

As far as Priority 1 projects were concerned, the cross cutting theme of Equal

Opportunities lay at the heart of most projects’ objectives. The ESF Convergence

Evaluation26 found that across Priority 1, WEFO had funded a number of projects

which had a strong focus upon equal opportunities, particularly targeting vulnerable

groups, including the Regional SEN project, Transition Key Workers under Research

the Heights and the Gypsy Traveller projects. Other funded projects were expected

to approach equal opportunities as part of a mainstream approach e.g. activities

targeting female participation in STEM were commissioned as part of specific STEM

projects.

21 Old Bell 3 (January 2011) ‘European Structural Funds Programme 2007 – 2013 The Effectiveness of Implementation in the 2007 – 2013 Structural Funds Programming Period Final Report’ p.67 22

WEFO Cross-cutting Themes Evaluation. Equality and Sustainability. March 2015. Parker. J et al. 23

Ibid., p.41 24

Ibid., p.42-43 25 Ibid., p.79 26

GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys Final Report’


18

The key conclusions relating to the cross-cutting themes within the ESF

Convergence Thematic Evaluation were that:

Most Priority 1 projects were aware of the Priority specific guidance on the

implementation of the two cross cutting themes

There was evidence of good practice in areas of both equal opportunities and

environmental sustainability but that ‘good practice was not entirely universal

across the projects, and there were also several projects that took more of a

‘tick-box’ approach to one or either of the cross-cutting themes’27

Some project partners were reliant upon existing local initiatives, but offered

little evidence as to how their projects incorporated these into their provision

for young people or added value to their services

In some cases, project sponsors were ineffective in getting partners involved

in implementing the cross-cutting themes

Cross cutting themes’ monitoring indicators were regarded as being

insufficient to capture the benefits for young people

Whilst pockets of good practice in relation to environmental sustainability were

found across the projects (e.g. promotion of green jobs and skills for young

people) there was no evidence that this had led to widespread innovation.

The evaluation drew attention to the feedback from projects and stakeholders

that ‘a small number of vertical projects around young people and

sustainability might be a way to raise the importance of sustainability in

relation to ESF in the future.’28

Overall, the evaluation concluded that ‘the implementation of cross-cutting themes

into Priority 1 was generally thought to have been effective, particularly in relation to

equal opportunities. There were examples of outstanding practice, but this was not

consistent across all projects’29.

A review of final project level evaluations suggests that:

27

GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys Final Report’ p.21 28 GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys Final Report’ p. 22 29 Ibid., p.58


19

Across the board, Priority 1 projects were complying with, and adopting their

own organisations’ policies and practices in relation to equal opportunities and

environmental sustainability. It was frequently reported that these policies and

practices had been embedded into the overall delivery as a matter of course

Overall, greater prominence was given to the cross-cutting theme of equal

opportunities across Priority 1 projects, largely due to the fact that equal

opportunities was a core principle and underlying objective of many Priority 1

projects in that they were specifically targeted at particular groups of young

people who were disadvantaged or not fully participating in educational or

economic opportunities. Indeed, a few projects were specifically focused on

such groups, including gypsy traveller communities, BME communities, young

people with special educational needs, and those either at risk, or were

already, NEET

Despite this, at least two project level evaluations concluded that projects

could have done more in the area of equal opportunities including

encouraging learners to consider a non-traditional route for their gender and

taking a more proactive approach to target and recruit female participants

None of the Priority 1 projects were exclusive focused on environmental

sustainability issues as had been the case for equal opportunities. Despite

this, at least five Priority 1 projects were deemed to have deployed innovative

and appropriate approaches to the cross-cutting theme of environmental

sustainability. These included the creation of an outside learning area for

school pupils to learn about nature and the environment, the provision of work

placement opportunities with employers operating within the environmental

sector and sector specific events covering topics such as renewable energy,

recycling and sustainable developments

At least two evaluations concluded that more could have been achieved by

some projects in relation to environmental sustainability – for instance in the

case of one project, the evaluators concluded that the aim of offering young

people the opportunity to undertake work placements and volunteering in

environmental organisations was not achieved


20

More generally, final project level evaluations suggest that greater awareness

and appreciation of the need to incorporate the cross-cutting themes within

delivery was experienced by projects over time.


21

6 Evaluation evidence: common barriers and success factors

This section explores some of the common success factors and barriers reported

across project level evaluations, drawing upon the findings of the Priority 1 thematic

evaluation as appropriate. It then discusses some of the externalities which impacted

implementation and the extent to which funded projects had considered exit

strategies and longer term sustainability.

Success factors

A review of project level evaluations reveals that Priority 1 projects were considered

to have had very few common success factors in place which accounted for strong

project performance. The following points give a flavour of the key factors which

were deemed to have been in place across projects, although it is worth noting that

no more than two or three examples were reported for each one:

After what was regarded as a generally slow start across most Priority 1

projects, at least two evaluations reported that a greater focus on trying to

achieve funded targets over the final stages of delivery accounted for the fact

that targets were eventually achieved or exceeded. For instance, one such

evaluation report noted that there had been ‘an urgent effort to recruit

participants towards the end’ in order to meet EU funded targets

Several evaluations reflected upon good practice and lessons learnt in terms

of effective participant engagement techniques – at least two projects which

had involved schools had found schools an effective recruitment method.

Others highlighted the importance of engaging parents, families and the home

as part of the recruitment strategy. Interestingly some two evaluations

reported that whilst projects had successfully engaged particular sub-groups

of young people (e.g. those living in poverty, those with Special Educational

Needs (SEN), care leavers) the recruitment of other sub-groups (e.g. ethnic

minorities) had proved to be more challenging. The Thematic Evaluation also

reported that there was evidence of good practice in terms of engaging young

people and assessing their support needs

The on-going tracking of progress being achieved by participants – together

with follow-on support as appropriate, even after their involvement with the


22

project had come to an end, was considered to have been good practice in

enabling projects to secure, capture and thereby report upon the outcomes

achieved by participants. Aligned to this it was also found that reviewing

participant record data had led to a massive upswing in the outputs reported

for one project

Some (around three) evaluations reported that project success could be (in-

part) attributed to the individualised approach adopted to supporting young

people. The key success factors were considered to have been a holistic

approach which offered both emotional and educational support, engaging

young people in the design of the project and offering young people a high

degree of choice in the activities which they engaged with

Several evaluations concluded that the nature of the services and support

offered had been vital components in contributing to the strong performance

of projects against their funded targets, particularly in terms of their ability to

engage individuals, but also in terms of good satisfaction levels amongst

participants. For instance, at least two evaluations reported that the provision

of work experience opportunities had accounted for a significant element of

the projects’ success in meeting its outputs whilst another reported that the

delivery of highly practical activities which were distinct from traditional

classroom activities had helped to recruit and retain participants. In another

case the content of the programme offered was found to have resulted in high

levels of completion, attainment and satisfaction levels

It was not uncommon for evaluation reports to highlight the skills, quality and

attributes of front-line personnel as being key contributors to projects’ strong

performance. Aligned to this it was also argued that success was attributable

to staff’s ability to develop positive relationships with young people and that

trust and friendship had assisted in generating project outcomes achieved.

Common barriers

Some of the most commonly cited factors which were considered to have accounted

for project under-performance (or contributed to delivery difficulties even in those

cases where targets had been achieved) were considered to have been:


23

Practical challenges relating to the delivery of large-scale projects involving

several partners which had manifested itself in issues relating to

communication, the role of the lead partner, reporting and management of

information. It was also reported in at least one case that the performance of

individual partners had varied with the poor-performance of some partners

accounting for the overall weak performance of the project.

Efforts deployed to meet high level EU funded targets were on a few

occasions thought to have detracted from the project’s ability to deliver more

meaningful support to young people. For instance, in one case it was reported

that participants joining the project towards the end (in an effort to meet

participant engaged targets) did not benefit from the range of training and

follow up support typically available

Issues related to reporting and MI systems which included instances of project

level MI databases being put in place later than desirable (thereby possibly

accounting for some under-reporting or inaccurate reporting of outputs

achieved). In other cases, it was reported that there had been an over-focus

upon the capturing and reporting of ESF related outputs, with very little other

MI data collected leading to a very narrow range of outputs and outcomes

being monitored. The Thematic Evaluation also highlighted issues in relation

to the consistency and timeliness of monitoring, including the adoption of

different interpretations of particular indicators and monitoring indicators not

allowing projects to evidence the impact ‘across the breadth of project

activities’30

Difficulties targeting and recruiting particular sub-groups of young people,

including those who are NEET, ethnic minorities, disabled young people and

limited use of engaging young people and their parents. Some of these

difficulties arose from a lack of capacity to recruit from these groups given that

projects were already over-subscribed whereas in other cases projects were

mostly reliant upon participant referrals made from partner organisations

30

GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’ p.58


24

The Thematic Evaluation31 found that projects took longer than expected to

begin full delivery due to three factors namely complexity and compliance with

different procurement procedures, lack of experience amongst project

sponsors of ESF and communication issues between WEFO and project

sponsors as well as between project sponsors and their partners.

Externalities

Very few (around six in all) evaluations cited significant externalities which impacted

upon the performance of the projects, although the three key major factors were

considered to be:

The impact of the economic downturn (cited by at least four evaluation

reports) which was deemed to have reduced opportunities to work within the

labour market – the two-fold impact of this was thought to have been (a) fewer

opportunities to support young people into employment opportunities (thereby

impacting upon project targets) and (b) a natural increase in the numbers of

young people being attracted to training or further learning, which in some

cases was considered to be having a positive impact upon projects’ abilities to

meet their targets. In one case it was reported that a project had responded to

these externalities by promoting volunteering, rather than work, opportunities

instead. In another case, it was reported that participants were more ready to

accept that they required education and training in light of the economic

downturn

The impact of public sector funding cutbacks which were considered to have

had various effects including difficulties securing initial match funding for

projects (given that the original match funding requirements were set prior to

the economic downturn) – in one case it was noted that ‘far less funding was

available than had been anticipated at the outset’ but also in that the cutbacks

would have bearing upon the project’s likelihood of being sustained, post EU

funding

Factors associated with delivery and partner organisations, including the lack

of strategic engagement and support for projects across the wider network

and structural changes across delivery partners.

31

GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’


25

Sustainability and Exit Strategy

At the time of undertaking the evaluations, very few projects had concrete plans in

place to sustain provision post EU funding largely due to the challenges of securing

ongoing funding given public sector financial cutbacks and a lack of funding options.

Indeed, drawing upon eight evaluation reports which specifically covered project

sustainability issues one reported that a project partner had committed to the on-

going funding of its own delivery for a further year whilst another evaluation reported

that a project lead sponsor had made applications for funding to various sources to

continue provision. Other than for these cases it was commonly reported that the

long term sustainability of projects was ‘uncertain’ and in one case the project

structure had already come to an end. In the case of two Priority 1 projects it was

reported that schools were expected to sustain elements of project interventions post

funding and, whilst there was evidence of this happening in at least one case, it was

commonly accepted that pressure upon school budgets meant that interventions

could not be sustained indefinitely or to the scale desired by schools.


26

7 Evaluation evidence: impact

This section considers the findings of project level evaluations on the impact of

Convergence Priority 1 upon the indicators of participants achieving qualifications,

entering further learning, entering employment and achieving other positive

outcomes). The ESF Leavers’ Survey did not cover Priority 1 Convergence

participants and therefore no conclusions can be drawn from this in relation to the

impact of this Priority.

Priority 1 had four impact related targets (relating to qualifications, entering further

learning, entering employment and other positive outcomes) and an analysis of

project level evaluations shows that in terms of the impact upon participants

achieving qualifications:

Nine of the 16 project evaluations concluded that projects had succeeded in

having at least some impact upon the target of participants gaining a

qualification, although in one of these cases (Raising Skills and Aspirations of

Young BME People) the number of participants gaining a qualification was

considered to be very low

The proportion of participants supported at the project level gaining any

qualification varied significantly across Priority 1. For instance, only 3 per cent

of all participants supported via the Reach the Heights First Footholds and 8

per cent of Reach the Heights Routes to the Summit projects achieved a

qualification in contrast to the higher rate of 91 per cent of all participants

achieving qualifications within the Pathways to Apprenticeships project. Two

factors contributing to the lower proportion of participants achieving a

qualification were identified as being (a) the time lag between engaging,

completing and achieving qualifications and (b) the fact that several projects

were focused on changing attitudes towards learning and education amongst

young people as opposed to the delivery of accredited training

Setting a project level qualification target was deemed to be inappropriate for

some projects: in the case of the Technocamps project the original target for

participants achieving qualifications was removed, in light of the fact that the

project was focusing on supporting young people aged 11 to 13 year olds.


27

This issue also become apparent across a few other projects, for instance, in

the case of the Gypsy Traveller Learning project it was anticipated that there

would be more participants in the upper end of the 11 – 19 age range who

would have been able to gain qualifications within the project’s lifetime

(although it was also the case that other factors accounted for weaker

performance - such as unanticipated greater focus upon supporting

participants with basic and soft skills who frequently had to overcome

entrenched difficulties in order to secure a qualification)

It was commonly reported that the full impact of intervention upon participants

achieving a qualification was unlikely to be realised until a specific point of

time in the future, such as at the end of an academic year or when younger

participants would be expected to sit their GSCE examinations which often

went beyond the project timescales.

In terms of the impact upon participants’ entering further learning:

Overall the contribution made by Priority 1 to this target was considered to be

relatively modest, if not weak, across project level evaluations. Indeed, this

finding is supported by the fact that Priority 1 under-achieved against this

target. Several common reasons were identified for this weaker than

anticipated impact: first, it was accepted that a number of Priority 1 projects

were actively engaging participants who were already in education and

therefore better placed to make a difference to their attendance and retention

levels rather than future progression. Second, it was widely reported that it

would take time for funded projects to have bearing upon participants’

progression into further learning, particularly those which focused on the

younger cohorts – for instance the evaluation of the KS3 Intervention Project

noted that it would take ‘several years before young people make choices

about subjects and careers and the impact can be measured empirically.’

Third, a few evaluation reports highlighted the lack of data available to

evidence participants’ progression into further learning, particularly in those

cases where projects had come to an end and where it would take time for

projects to have bearing upon this indicator. For instance, the evaluation


28

report on the Technocamps project32 suggests the need for a longitudinal

study to gather evidence on entry into further learning amongst supported

participants

Where data were available, progression into further learning appears to have

been fairly low – for instance only 106 Reach the Heights Routes to the

Summit participants (less than one per cent of all project participants) were

found to have progressed into further learning although 7 per cent of Reach

the Heights First Footholds participants (1,144) had done so33. Drawing upon

project level evaluations the highest proportion of supported participants who

had entered further learning (at 16 per cent) was found to be the case of

Raising Skills and Aspirations of Young BME People project

It was also commonly the case that projects had under-achieved against their

own targets for participants progressing into further learning (despite targets

having frequently been revised downwards in light of weaker performance).

This was particularly the case for Pathways to Apprenticeship – which was

considered to have fallen well short of its target (and ultimately its overall

objective) of supporting participants to progress into apprenticeship

opportunities given that only a third of participants progressed onto an

apprenticeship. In this context, some two-thirds of learners reported that their

destination on leaving the project was their preferred destination. However in

the case of a third of those who did not progress onto an apprenticeship it was

reported that an apprenticeship had not been available to them whilst a

smaller proportion had applied but were unsuccessful34.

In terms of the Priority’s impact on supporting participants to enter employment:

As might have been expected given that Priority 1 did not have any targets for

supporting participants into employment, project level evaluations found that

this Priority had had a very limited impact upon supporting participants to

enter employment: indeed, this finding was reinforced by the fact that only

32 Wavehill ‘An Independents Evaluation of the Technocamps Project’ p.61 33 Reach the Heights consisted of two projects – First Footholds, funded under Theme 1: Tackling Underachievement (which aimed to improve the educational outcomes of young people at risk of underachieving), and Routes to the Summit, funded under Theme 2: Raising Skills and Aspirations (which aimed to support young people to make a successful transition into employment) 34 BMG Research ‘Final Evaluation of the Pathways to Apprenticeship programme’ p.11


29

three per cent of all participants engaged (or 4,617) were supported into

employment within this Priority

At least eight projects were considered to have had no impact upon

supporting participants into employment, largely due to their focus on working

with participants under 16 (i.e. below the working age). Seven projects were

considered to have had at least some impact – although generally the number

of participants concerned were low e.g. 41 of the 1,054 participants supported

on the Raising Skills and Aspirations of Young BME People project entered

employment

The most commonly cited factors which were considered to have accounted

for the low numbers of participants entering employment were: the lack of

progression data available for participants once their involvement with the

project had come to an end; the fact that projects were supporting participants

aged from 11 who would not be expected to enter the workforce for a number

of years after their involvement with the project; the general state of the

economy and lack of job opportunities

Despite this, there is good evidence available across evaluation reports to

show that projects were impacting upon participants’ attitudes and perception

of work as well as equipping young people with the appropriate skills and

abilities for future work. In a few cases this stemmed from the focus within

projects to provide work placement experiences to participants, including

supported work placement opportunities in some cases, whilst others were

geared to raise aspirations and inspire young people to explore careers within

particular sectors (such as STEM). The findings of one evaluation (Building

the Future Together) sums these issues succinctly: ‘the project is currently

below target in terms of the number of young people entering employment

and there was some evidence that young people being supported lacked

confidence in their abilities which would help them gain employment in the

future. Encouragingly the positive impact upon intermediate outcomes for

employment [e.g. helping young people gain job interviews] suggests that

employment outcomes should improve as the work continues’

It was also the case in one project (World Class Traineeships) that the evaluation

found intervention had a greater impact upon progression within work rather



30

than participants entering work i.e. a third of those working for the same

employer as before had had a promotion since completing their

apprenticeship35.

‘Soft’ or intermediate outcomes for participants

Project level evaluations reported extensive soft outcomes for participants – with

these being considered frequently in terms of the distance travelled by participants.

Indeed, one evaluation36 report noted that the project’s intervention to help a

participant leave their bedroom could be considered a greater achievement than for

another participant to gain a qualification. The most commonly cited soft outcomes

are summarised below:

An improvement to levels of independence and confidence – some

evaluations reported that participants were being able to undertake tasks or

routines for the first time without family support (e.g. travelling by bus)

Behavioural changes such as participants being able to attend events or

volunteering opportunities on time and demonstrating attributes such as

politeness, respectfulness and being better presented for the tasks in question

Improved health and fitness, particularly in terms of reduced anxiety and

participants being better able to cope with their emotions

Increased dealings with other people and a greater circle of friends

An increase in participants’ ability to concentrate on tasks

An increased awareness and understanding of future career opportunities

An improvement to skills, particularly softer skills such as communication,

team working, leadership and planning skills, and basic skills. In one case it

was argued that the project’s impact upon participants’ soft skills was ‘the

most important impact considering the background of the participants’.

A number of project level evaluations reported that project providers had welcomed

WEFO’s decision to introduce a soft outcomes indicator yet acknowledged that the

capturing of this data was often challenging and occasionally inconsistently applied

within projects. In one case, for instance, it was reported that ‘the reporting [of soft

35 Wilson et al ‘Evaluation of Work-Based Learning Programme 2011-14: Apprenticeships’ p.11 36

People and Work Unit (April 2012) ‘External Mid Term External Evaluation of the Business the Future Together (BTFT) European Social Fund (ESF) project. Final Report’ p.16


31

outcomes] has been patchy making it problematic to quantitatively evaluate the

overall impact of the project on soft outcomes’37.

Exploring the Priority’s impact upon other positive outcomes achieved by

participants it was the case that all evaluation reports reviewed found that projects

had made achievements in this respect. Indeed, the proportions of participants

reported as having achieved other positive outcomes across projects were generally

high. The most commonly cited outcomes related to:

Completion of training courses, workshops or events

Adopting a personal development plan

Participating in work experience or voluntary work experiences

Improved participation levels in education and training (including improved

behaviour and attendance levels)

Improved personal skills such as time management, team working,

communication and social skills as well as improved basic and essential skills

(numeracy, literacy, IT)

Increased awareness of future opportunities (be they education or work

related) and changes to participants’ perceptions of future opportunities –

particularly in relation to attitudes towards future careers, most notably around

STEM related study and careers

Much of the evidence presented across evaluation reports on the achievement of

these other positive outcomes were short-term in nature. Indeed, it was notable how

little data were available to assess whether these outcomes were sustained in the

long term and would have therefore contributed to concrete achievements such as

improved educational attainment or improved progression into further and higher

education. The main reason for this was due to the fact that measuring long term

impacts was beyond the scope of the evaluation studies commissioned and the

studies lacked any approaches to track the destinations of supported participants

post intervention.

37

People and Work Unit External Mid Term Evaluation of the Building the Future F Together (BTFT) European Social Fund (ESF) project (April 2012) p.16


32

Deadweight and displacement

Meaningful consideration of the extent to which project intervention had generated

any deadweight (i.e. whether the outcomes achieved would have happened anyway)

was only available in seven of the evaluation reports reviewed, and the findings

suggest a mixed picture.

In two cases levels of deadweight were considered to be high – for instance in the

case of the STEM Cymru project, a high proportion of participants were considered

to be natural scientists who would never have considered a career in anything other

than a STEM related role. In other cases, stronger arguments were presented that

the funding had allowed providers to deliver interventions at a greater scale and

volume than would otherwise had been the case. In the case of one evaluation

study, the relatively high levels of prior qualifications amongst participants raised

questions about the ‘duplication of public funding’38 although it was acknowledged

that this issue had to be considered within the context of the project’s objective being

to help participants gain employment.

Most evaluation reports pointed to there being very little displacement of other

services or existing provision as a result of projects funded via this Priority and the

evaluations generally suggested that other support services had not been reduced

as a result of funded activities. Indeed, a number of evaluation reports provided fairly

robust evidence of how projects had managed to integrate their provision across

existing services such as within schools and colleges or youth and welfare support

providers. A few evaluation reports captured evidence on those activities which

participants would have missed out on as a result of participating in Priority 1

projects – these were mostly classroom based lessons which participants would

have missed as a result of attending project interventions.

Value for money

Figure 6 presents the evidence drawn from five evaluation reports on the value for

money offered by Priority 1 projects in terms of the investment per participant and

outcomes achieved. The data shows that average costs varied significantly – some,

38 Wilson et al (2016) ‘Evaluation of Work-Based Learning Programme 2001-14: Apprenticeships’ p.165


33

such as the Potensial project, reported very high average costs whilst others (such

as Pre-Vent) reported lower average investment per outcome. An exceptionally low

investment per participant was reported for the Technocamps project at £24 per

participant which was a reflection of the one-off, lighter touch involvement of many

participants in group based workshops or events.

Figure 6: Investment per participant and per outcome

£ Investment

per participant

Investment

per participant

entering

further

learning

Investment

per participant

gaining

qualification

Investment per

participant

gaining other

positive

outcomes

Engage 1,719 5,403 3,584 7,258

Building the

Future

Together

1,381 22,573 4,449 3,048

Potensial 3,389 29,759 10,131 8,861

Pre-Vent KS3 1,416 5,252 7,956 4,269

Technocamps 24 n/a n/a n/a

Priority 1

1,473

15,973

7,850

5,519

Source: Project level evaluation reports

In the small number of cases where project level evaluations gave meaningful

consideration to the value for money offered by interventions, a mixed picture was

found – a small number concluded that the value for money offered had been

excellent, others (such as in the case of Amlwch 16+) concluded that the value for

money provided had been good whereas others concluded that better value for

money could have been achieved had the project been delivered somewhat

differently. In those cases where projects were considered to offer good or excellent

value for money it was reported that interventions often ‘added value’, addressed

gaps and enhanced existing capacity (e.g. such as in schools) as well as enabling

provision to be delivered on a greater scale and therefore generate economies of

scale. It is perhaps a reflection of the nature of intervention (i.e. that any impact of


34

interventions would only be achieved in the long term) that so few of the project level

evaluations could offer a more robust calculation of the value for money offered.

A small number of project level evaluations reported differences in the value for

money offered within projects – for instance investment per participant or per

outcomes varied by geographical areas and/or by different partners. These

differences on occasion were considered to have been influenced by the additional

‘rural’ premium associated with the delivery of intervention in a rural county.

It was widely argued across project level evaluations that funded interventions had

often been able to ‘add value’ to existing capacity, particularly across schools and

colleges – thereby offering an acceptable return on investment. Some project

evaluations concluded that interventions had increased the ‘capacity’ of existing

providers (e.g. schools,) to better cater for the needs of specific groups of young

people, particularly challenging young people.

Furthermore, project level evaluations highlighted some common features on how

projects could have offered better value for money. These included:

getting established and underway at a quicker pace

a more consistent performance across all delivery partners

adopting a different implementation model with lower management costs

greater involvement of regional and local stakeholders – particularly to avoid

any duplication in provision.

Very little evidence on the financial return on investment made was offered across

project level evaluations. Indeed, only two calculations were identified within the

review, both of which suggested a very good level of financial return:

STEM Cymru calculated that every £1 spent the Convergence Area resulted

in a financial gain of £3 in benefits for the area, which was considered to offer

‘incredible value for money’

Pathways to Apprenticeship – the evaluation found that for every £1 spent

each learner would generate £3.02 in increased earnings and that £2.12


35

would be generated in increased direct and indirect revenues to public

budgets.

Counterfactual

Very few project level evaluations were found to have attempted some form of

counterfactual impact evaluation and where they had done so, the usefulness of the

data was fairly limited. Overall, there is a lack of robust counterfactual evidence

available for this Priority.

The following counterfactual evidence was presented within project level evaluations:

In the case of the Pathways to Apprenticeships (PtA) project a cost benefit

analysis was undertaken using Propensity Score Matching to analyse the

outcomes for participants against a group of similar learners from the Lifelong

Learning Wales Record (LLWR) database. The analysis revealed that the

total net benefit to participants was £88.316m whilst the total net benefit to

public budgets was £61.906m against a total cost of delivering the project at

£29.26m. It is estimated that PtA would generate £3.02 in increased earnings

for PtA learners for each £ spent in delivering PtA and would generate £2.12

in increased direct and indirect revenues to public budgets for each pound

spent in delivering PtA

In the case of STEM Cymru, applying an overall deadweight of 50% to the

calculation, the project level evaluation reported that the project produced an

economic impact of £4m. The net economic impact was calculated to be

£1.2m having taken into consideration factors such as participants taking up

jobs outside the Convergence area and the depressed state of the labour

market in Wales.

For the World Class Traineeships project, a Propensity Score Matching

analysis was used to match apprenticeship leaver survey respondents with

data from the Annual Population Survey (APS), with a particular focus on the

earnings growth of the two cohorts. The analysis revealed that estimates of

the earnings growth amongst surveyed apprenticeship participants were

higher than those derived for the wider population from the APS. However,

when the differences in the composition of the two groups was taken into



36

consideration (i.e. respondents to the apprenticeship leaver survey were

relatively younger) no significant difference in earnings growth between the

treated and control groups was revealed

The analysis for the Engage project found that Engage eligible areas

significantly outperformed a control group area when considering the number

of Year 11 leavers known not to be in education, training or employment with

a 39 per cent reduction in the number of NEETs between 2009 -2011 within

Engage areas compared with a 22 per cent reduction across non-Engage

areas. The evaluation reports that had Engage areas performed in the same

manner as the control group areas, an additional 83 year 11 leavers would be

NEET

In the case of the Key Stage 3 project, the achievement of young people

eligible for Free School Meals (FSM) was used as a proxy indicator or

measure of the impact of the project upon young people at risk of being

disaffected. The analysis indicated that the performance of pupils eligible for

FSM achieving the expected level in the Core Subject Indicator (CSI) in two

authorities increased more rapidly than the Welsh average between 2010 and

2011 and increased more rapidly than the trend in recent years between 2007

and 2010 within these local authorities.

The experience of undertaking the matching exercise as part of the World Class

Apprenticeship project evaluation reinforces a common view conveyed across

several other project level evaluations i.e. that it is not realistic for participants to

expect benefits to be realised immediately after receiving a Priority 1 type

intervention, with outcomes likely to be longer-term in nature. The World Class

Apprenticeship evaluation noted that ‘it may not be realistic to expect that benefits of

participation in a training programme of 2.25 years duration will be realised within 12

months of its completion’39.

In some cases it is evident that the lack of medium to long term data has hindered

efforts to quantify the counterfactual position, for instance:

39Wilson et al (2016) ‘Evaluation of Work-Based Learning Programme 2001-14: Apprenticeships’ p.162


37

‘Limitations in the quantitative data meant a judgement could not be made on what

would have happened without the interventions’40.

‘A system to track destinations of participants and the sustainability of these

destinations post Potensial intervention would provide further evidence of project

impact. A detailed and centralised record of which participants received support

under which specific interventions, along with hard outcomes and soft, standardised

distance travelled data would be required to compare interventions and establish

which are the most effective.’41

40 People and Work Unit ‘External Mid Term External Evaluation of the Building the Future Together (BTFT) European Social Fund (ESF) project. Final Report’ p.14 41 Wavehill (March 2014) ‘Final Evaluation of the Potensial Project’ p.7


38

8 Evaluation evidence: other outcomes and unintended consequences

This section considers the evidence, principally from project level evaluations of

other outcomes and any unintended consequences of delivery.

Project achievements not reported to WEFO

The key achievements not systematically reported as outputs against this Priority

were mainly:

That supporting young people to stay within education had been an important

objective, even the key objective in some projects like Engage, that perhaps

was not being fully acknowledged when included against the broader

headings of ‘soft outcomes’ or ‘other positive outcomes’ set for this Priority.

This led one evaluation report42 to argue that ‘A participant remaining in

education should represent a key, claimable output’ for future projects

In a similar manner, several project level evaluations highlighted very good

achievements in terms of improved attendance levels, reduced permanent

exclusions and unauthorised absenteeism amongst supported participants

which were not being reflected within the broad category of ‘other positive

outcomes’

There was some evidence to suggest that part-qualifications would not have

been reported as such (but rather captured as other positive outcomes)

Improved partnership working between various organisations and the

resulting improved integration of services and interventions

Some evidence (from at least two project level evaluations) that funded

interventions were helping to reduce levels of discrimination towards the

target groups being supported.

Indirect positive effects

Very little evidence was available to substantiate the indirect positive effects of

Priority 1 funded projects. Indeed, only two project level evaluations provided any

calculation on the savings to public expenditure or increased lifetime earnings for

42 42 Wavehill (March 2014) ‘Final Evaluation of the Potensial Project’ p.7


39

participants: there was no consideration of these issues in other evaluation reports.

These were:

Savings to public expenditure

o In the case of one project (SEN) it was calculated that the net saving to

local authority per participant per year would equate to £12,314

(savings which would be related to transport and staff costs)

Increased lifetime earnings for participants

o Pathways to Apprenticeship – the evaluation found that for every £1

spent each learner would generate £3.02 in increased earnings.

Unanticipated negative effects

The most commonly cited unanticipated negative effect related to the uncertainty

over the long term sustainability of project interventions – particularly given that

several projects had been able to improve the referral systems and pathways for

young people to access other support services. The potential loss of the brokerage

service deployed by some projects to link participants and other support services

was considered a threat which could lead to the discontinuation of support in such

cases.


40

9 Availability and quality of project level evaluation evidence

Introduction

In total, 16 project-level evaluations were available for review against a base of 20

projects which were approved via this Priority. In the case of two projects43, a

combined evaluation was commissioned and published (i.e. there were 15 evaluation

reports). Of the 15 reports, 13 were final and two were mid-term evaluations.

At the time of review four projects did not have project level evaluations, namely

Pupils Understanding Problems in their Locality (Cynon Valley Crime Prevention),

Young BME People Aiming High (All Wales Ethnic Minority Association), Minority

Ethnic Learning and Achievement Project (WAG DCELLS Lifelong Learning and

Skills) and Think Differently (Fairbridge De Cymru). Two of these projects were

relatively small in terms of their funding levels (under £500,000 EU funding) whilst

the other two were awarded up to £4m EU funding each. Overall the evaluation

coverage as a proportion of committed EU funding was 95 per cent. In one case,

WEFO terminated its project agreement with the project sponsor and provision was

transferred to other joint sponsors of the project. It is notable that two of these

projects which are without project-level evaluations were specifically focused upon

BME groups.

16 of the project-level evaluations were prepared in English and one in Welsh. All

were prepared by external independent evaluators – in all nine different teams of

evaluators were commissioned to undertake these studies, with one team of

evaluators involved in four studies in all.

Methodological Detail

Eight of the evaluation reports were considered to have strong methodological

statements. Those categorised as strong typically contained:

A detailed account of the method deployed including details for proposed

sampling strategy (and in very few cases comparative data on samples

43 Reach the Heights Routes to the Summit and Reach the Heights First Footholds


41

achieved), method of recruiting contributors, number of contributors involved

and timescales for conducting fieldwork

A logic model which was used to underpin the investigation

A detailed Evaluation Framework (usually presented in an annex) presenting

the objectives of the study and method of collecting data to assess the extent

to which they were achieved

Various research instruments including topic guides for interviewing staff or

participants or survey tools (usually presented in an annex)

A review of the strengths and weaknesses of the method adopted – one

evaluator adopted this approach across three project level evaluations and

provides a detailed insight into the strengths, challenges and limitations of

various approaches deployed.

Six evaluation reports were categorised as moderate. These typically contained

methodological statements but had some gaps (e.g. in relation to beneficiary

sampling and survey response rates).

Mixed Methods

All but one (14 of 15) of the project-level evaluations were considered to have

deployed a mixed approach (i.e. in terms of primary fieldwork and consideration of

project level data) although there was generally greater use of qualitative data from

participants (rather than quantitative data) across the board, possibly reflecting the

nature of the target group and the difficulty of accessing direct contact data for young

people in order to conduct say, a phone or web survey. In these cases, there

appears to have been a reliance upon project providers to facilitate access for

evaluators to meet with participants. Difficulties or restrictions in sharing participant

contact data has meant that many of the evaluations did not have much control over

the sample of participants interviewed. As a result, interviewed participant samples

have often been purposive and an element of project selection bias is evident.

At least two evaluations specifically noted that in-depth interviews with young people

had worked better than conducting workshops or small group based discussions with


42

participants on the basis that it had been easier to collect feedback from all

contributors and present this in an aggregated manner within the report.

Two common limitations were identified with a qualitative approach to gathering

feedback from participants. In some cases, evaluators noted that it had not been

possible to conduct interviews with participants from across all provisions or

providers and some groups – particularly where the sample numbers had been low.

In other cases, it was acknowledged that some groups or areas were not

represented within the sample e.g. younger participants aged 11-13.

In a positive manner, some studies attempted to present data collected via

qualitative interviews with participants in a quantitative manner e.g. in the case of

Engage, a large number of qualitative interviews (a total of 136) was used in this

manner with data presented in a numerical manner via graphs and tables.

At least four evaluations had involved some element of fieldwork with parents –

although in most of these cases the sample of parents contributing to the research

was very low (32 in the case of Potensial, 10 in the case of Building the Future

Together, 10 in the case of Technocamps and five in the case of Llwyddo’n Lleol).

A slightly greater number of evaluations had undertaken some element of fieldwork

with employers (which involved a mix of qualitative and quantitative research) –

although sample numbers varied from the very low (e.g. in the case of Llwyddo’n

Lleol) to reasonable and robust samples (e.g. in the case of Pathways to

Apprenticeships at 67 and Stem Cymru at 101).

Surprisingly few evaluations reported that they had undertaken observational work –

possible the only project-level evaluation to have done so in an intensive manner

had been in the case of the Technocamps project where some 100 events were

observed.

At least three evaluations reported that they had undertaken longitudinal work with

supported participants (Llwyddo’n Lleol, the Regional SEN Transition to Employment

Initiative and the Gypsy Traveller Learning and Future Employment Project). In the

latter, the approach involved a baseline survey and final survey to provide good data


43

on difference achieved. Timescales were considered to restrict this approach in

many cases e.g. in the case of Technocamps it was reported that a longer term

study would be needed to assess the impact of intervention upon supported young

people.

Project-level evaluations made significant use of project level monitoring data, with a

small number of evaluations relying upon monitoring data as the main quantitative

data source. It was the case that gaps in data or inconsistent data limited the use of

quantitative databases in some cases, particularly around achievement of soft

outcomes (e.g. Building the Future Together).

The extent of triangulation across project-level evaluations varied – in those cases

where primary fieldwork had been undertaken with a cross-section of stakeholders

(including participants, parents, employers, project delivery teams and independent

stakeholders) a general effort was made to reflect upon different viewpoints.

However, it was often the case that specific case study material was considered from

the viewpoint of a single contributor (more often than not being the participant’s view)

rather than from a more rounded view of a parent, employer or provider.

Beneficiary Fieldwork

In all, 13 of the 15 project-level evaluations involved beneficiary fieldwork. Given the

reliance upon qualitative fieldwork it is perhaps not surprising that only seven of

these set out the findings of a quantitative survey of supported participants. Sample

sizes varied both in absolute terms and as a proportion of all participants supported

by the project – and it is noteworthy that some evaluations completed a greater

number of qualitative interviews with participants than did some of the surveys. In

some cases it is difficult to report upon response rates (given the lack of data

available in the report). Project-level evaluations reporting high numbers of surveyed

participants were a survey of 3,197 participants (27% of all project participants) in

the case of Llwyddo’n Lleol and a survey of over 3,000 participants in the case of

STEM Cymru (response rate unknown).

One evaluation (Raising Skills and Aspirations of Young BME People II) had

intended to deploy a web survey of participants but the approach was revised in light


44

of feedback from participants and project staff that a web survey was considered

inappropriate, impersonal and disempowering to participants who had literacy issues

or where were not first language English speakers. The evaluation therefore relied

upon face to face interviews, focus groups and filmed case studies of participants.

Counterfactual Impact Evaluation

Only four project-level evaluation reports attempted to undertake any counterfactual

impact evaluation and overall there is a lack of robust counterfactual evidence

available for this Priority. Each of these project-level evaluations deployed a different

approach:

In the case of Pathways to Apprenticeships the counterfactual impact

evaluation involved undertaking a Propensity Score Matching using the LLWR

database as a source of evidence. The evaluators matched a group of treated

learners with a comparable group of non-treated learners to calculate lifetime

increases in earning as a result of intervention. The approach appears to have

been robust and the report presents financial data on the lifetime benefits of

the qualifications obtained as a result of intervention, as matched against a

non-treated group

In the case of the KS3 Intervention project the evaluators considered the

educational performance and achievements at Key Stage 3 (i.e. proportions

achieving expected levels in core subject indicators) and how this changed

over time between participating and non-participating local authorities. As it

was not possible to identify individual supported pupils (as they were not

‘tagged’ in any way on local authority databases) the evaluators relied upon

proxy data to estimate the impact of the project. These proxy indicators

included the performance of pupils eligible for free-schools meals, rates of

absence, and the proportions of pupils achieving higher levels of performance

in science and mathematics. The limitations of this approach was noted

including the fact that other local authorities had similar ESF funded projects

which was considered to have limited the scope of the counter-factual

analysis

The evaluation of the Potensial project considered the counterfactual position

based upon ‘soft’ evidence gathered from supported participants and delivery


45

staff via qualitative interviews which probed around what would have

happened in the absence of support. Consideration is given to the feedback

from delivery staff and 89 supported participants (although it is not known

what proportion of all supported participants this sample represents).

In the same manner the evaluation of Engage attempts to estimate the

counterfactual position based upon feedback from supported participants but

overall this is quite a light-touch section which reported how the lives of

participants would be different had it not been for the project

It would appear that several of the project-level evaluations, particularly those which

involved face to face interviews with large numbers of participants missed an

opportunity to gather ‘soft’ counterfactual evidence from young people. For instance,

in the case of the Gypsy Traveller Learning and Future Employment Project it could

have been relatively easy to have built in appropriate questions such as ‘what would

have happened without support, or whether changes would have come about

anyway’ during such questioning.


46

10 Conclusions

The evidence base for Priority 1 is comprehensive in that 16 of 20 Convergence

funded projects, equating to 95 per cent of committed EU funding, have evaluation

findings available. However, the lack of CIE data available within project level

evaluations as well as the fact that no data is available via the ESF Leavers Survey

makes it impossible to calculate the net impacts of this Priority in a reliable manner.

The evidence suggests that the original intervention logic for Priority 1 was rational

and that it had been appropriate to introduce a specific priority focused on the

delivery of preventative interventions for 11 to 19 year olds. Priority 1 has been

delivered within the context of a fairly stable policy environment – in contrast funded

projects have had to respond to some significant external changes. The economic

downturn reduced employment opportunities for participants on the one hand but on

the other meant that greater number of young people were being drawn to education

and training opportunities. Moreover, the impact of public sector funding cutbacks

was also a significant external factor although in reality this was considered to have

had a greater bearing upon the sustainability of provision post EU funding.

A total of 20 projects were approved across Convergence Priority 1. These projects

demonstrated a good fit with the Operational Programme and the Strategic

Framework in that the proposed activities were well covered and were focused on

adding value and complementing mainstream educational provision. Projects were

also wide-ranging in terms of the target audiences which they intended to support.

The key delivery issue which impacted negatively on this Priority was the relatively

slow start in approving projects, not least because of the move towards adopting

fewer, more strategic projects but also because of longer than anticipated

procurement procedures and a lack of experience amongst some project sponsors of

ESF funding. Whilst the Priority struggled to meet its profiled output indicators during

the initial few years of delivery, performance improved over the second half of the

Programme.


47

In the event, the Priority participant output was substantially exceeded (by nearly

four-fold) and on a collective basis funded projects exceeded all but one of the

outputs indicators. Performance against Priority results indicators was also very

strong – particularly in terms of the numbers of participants gaining other positive

outcomes and qualifications. However, the proportion of all participants assisted who

gained a qualification or other positive outcome represented a much lower proportion

of all participants supported than envisaged. Furthermore, the Priority fell short of

achieving its targets for participants entering further learning.

One common theme to be raised across project level evaluations related to the fact

that many of the results indicators were unlikely to be realised in the short term given

the age profile of participants supported via this Priority. It was also commonly

argued that projects were better placed to have a direct impact upon pupil

attendance and retention levels within education – significant outputs which were not

necessarily being reported to or captured by WEFO. Aligned to this, the lack of long-

term data available for supported participants to evidence any future progression (be

that in terms of entry to further learning or qualifications obtained) limits the evidence

base available on the overall impact of Priority 1 funded interventions.

The lack of robust evaluation evidence on levels of deadweight and displacement is

notable for Priority 1. The fairly limited evidence suggests that levels of deadweight

have been mixed (some projects reported high levels whilst others reported lower

levels of deadweight). In contrast, levels of displacement appear to have been fairly

low as funded interventions have been integrated and added value to existing

mainstream services.

Overall, Priority 1 achieved better value for money (as measured by the level of

investment per participants engaged and direct results achieved) than planned. At a

project level the fairly limited evidence suggested a mixed picture in terms of the

level of investment per participant engaged and direct results achieved.

In terms of cross-cutting themes, project level evaluations found that Equal

Opportunities was a central consideration in most projects’ objectives in that they

were often specifically targeted at disadvantaged groups of young people.. In terms


48

of environmental sustainability, project level evaluations suggested that whilst a

smaller number of funded projects deployed innovate approaches, more could have

been achieved by some projects in this area of work.

Assessing the net impact of Priority 1 interventions is challenging given that so few

project level evaluations could realistically undertake any meaningful counterfactual

impact evaluation in light of the target audience supported. The lack of any

counterfactual evidence limited evaluators’ ability to test what would have happened

in the absence of specific interventions. Where any CIE assessment has been

attempted, the findings point to a positive impact although the transferability of the

data itself in calculating the more macro level net impact of Priority 1 is limited.

There is evidence of a number of unintended consequences having occurred as a

result of provision funded under Priority 1. Of critical importance has been the impact

of interventions upon levels of educational attendance, reduced unauthorised

absenteeism, reduced permanent exclusions and overall educational retention levels

amongst participants. There is also significant evidence via project-level evaluations

of participants having achieved extensive soft or intermediate outcomes as a result

of participation – these have included increased confidence levels and positive

behavioural changes. Accepting of the fact that it would have been extremely

challenging to have done so, project level evaluations have revealed very little

evidence to substantiate the indirect positive effects of funded projects in terms of

public expenditure savings, increased lifetime earnings for participants and reduced

negative social behaviour. Where available, the data (which should be considered

with caution), however, does suggest a positive impact.

In terms of lessons for the future, these might include:

That effective provisions targeted at young people generally adopt an

individualised, holistic approach which offer both emotional and educational

provisions via advisors who can develop positive relationships with young

people

That work experience opportunities are important and add value to provisions

targeted at young people


49

The varied performance of multiple partners within a single project can impact

the overall performance of a large scale, strategic project

The need to adopt indicators which reflect the direct impact of interventions

aimed at young people within education (e.g. around absenteeism,

attendance and retention rates)

The need to consider improving the capturing and reporting of participant

outputs and results, particularly to reduce any under-reporting or inaccurate

reporting of outputs as well as to enable the identification and analysis of any

longer-term, post-intervention changes and outcomes

The need to continue to explore CIE approaches to measure the difference

made to participants over the medium to long term. In this respect it would be

worth further investigating the possibility of matching WEFO’s central

participant database with data available within the LLWR database.

Date post:	14-Sep-2019
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

September 2017 Wales Ex Post Evaluation Priority Review Paper · The Programme Monitoring Committee...

Documents