Ymchwil
Research
September 2017
Wales Ex Post Evaluation
Priority Review Paper
Supplying young people with skills for learning and employment
ESF Convergence Priority 1
ob3research.co.uk
ob3research.co.uk Company Registration Number: 5565984 1
September 2017
Wales Ex Post Evaluation
Priority Review Paper
Supplying young people with skills for
learning and employment
ESF Convergence Priority 1
Browerdd, Heol yr Ysgol
Llanarthne, Sir Gaerfyrddin
SA32 8HJ
01558 822 922
38 Llewelyn Goch
St Fagans, Cardiff
CF5 6HR
029 2089 1724
ob3research.co.uk Company Registration Number: 5565984 2
Author: Nia Bryer
Quality Check: Huw Bryer
Version: FINAL
ESF Convergence Priority 1 Priority Review Paper
3
Contents Page
1. Introduction: Scope of Priority 3
2. Intervention logic and the changing policy context 5
3. Funded projects: ‘fit’ with intervention logic 9
4. Targets and achievements 13
5. Cross cutting themes 16
6. Evaluation evidence: common barriers and success factors 20
7. Evaluation evidence: impact 25
8. Evaluation evidence: other outcomes and unintended consequences
37
9. Availability and quality of evaluation evidence 39
10. Conclusions 45
ESF Convergence Priority 1 Priority Review Paper
4
1 Introduction: Scope of Priority
The aim of Priority 1 of the European Social Fund (ESF) Convergence Operational
Programme (Supplying young people with skills for learning and future employment)
was to ‘prepare young people for future employment by raising aspirations and
increasing participation in learning’1.
Priority 1 was focused upon supporting those aged between 11 and 19 years of age
who were considered to be ‘at particular risk of under-achieving or becoming Not in
Employment, Education or Training (NEET) , for future employment by raising their
aspirations, and increasing participation in learning or training.’2 It was explicitly not
made available to fund mainstream educational provision although it was expected
to add value to mainstream provision by ‘using innovative approaches to intervention
which engage young people’3. Most of the funding (70 per cent) within this Priority
was intended to be directed towards young people aged 14 to 19 years old.
Sitting beneath this Priority were two themes, namely:
Theme 1 – Tackling Under Achievement (Closing the gap) which was
intended to focus on improving the educational outcomes of young people at
risk of underachieving. This theme had two clear objectives, namely
combating disaffection and reducing the risk of early school leaving as well as
meeting the specific needs of disadvantaged groups such as those with
special educational needs, young offenders and people from Black and
Minority Ethnic (BME) communities. It was expected that this theme would
account for around 70 per cent of the resources allocated to this Priority
Theme 2 - Raising Skills and Aspirations (Raising the bar) which was
intended to focus on supporting young people to make a successful transition
into employment. This theme had three distinct objectives namely providing
1 Welsh European Funding Office (2009) ‘West Wales and the Valleys Convergence Programme – Operational
Programme For The European Social Fund 2007-2014’ p.115 2 Welsh European Funding Office (2009) ‘West Wales and the Valleys Convergence Programme – Operational
Programme For The European Social Fund 2007-2014’ p.88 3 ESF Convergence Priority 1 Supplying Young People with Skills for Learning and Future Employment Strategic
Framework p.9
ESF Convergence Priority 1 Priority Review Paper
5
enhanced information and guidance on employment, vocational training and
higher education to disadvantaged young people; providing access to a
broader and innovative range of learning options to extend and strengthen
their learning experiences and skills; and promoting and supporting the
acquisition of higher levels skills amongst disadvantaged young people. It was
expected that this theme would account for around 30 per cent of the
resources for this priority.
ESF Convergence Priority 1 Priority Review Paper
6
2 Intervention logic and the changing policy context
The key underlying rationale for Priority 1 was:
to ensure a skilled and competitive workforce for the future
to equip young people with appropriate life skills for flexible career patterns
that early intervention would lead to higher levels of education attainment and
longer term employment outcomes
that levels of literacy and numeracy in young people remained low
that enabling young people to successfully transition into further or higher
education would lead to a greater choice of learning pathways
that specific groups of young people had particular barriers and needs that
required targeted intervention
that some young people did not make appropriate career choices thereby
reducing their future earning potential
a need for more young people to study science in order to support the
knowledge economy and the needs of employers.
A logic model for this Priority 1 is presented at Figure 1:
ESF Convergence Priority 1 Priority Review Paper
7
Figure 1: Logic Model for ESF Convergence Priority 1
Whilst tackling youth engagement and employment had been a common theme
within previous European funding programmes in Wales, the introduction of a
specific priority focused on 11 to 19 year olds represented a major change in
recognising the importance of providing preventative interventions for young people
for the first time.
Its introduction came following the general response received as part of the public
consultation exercise and was highlighted as a ‘welcome addition to the
Programme’4 within the ex-ante evaluation of the 2007-2013 programme. Indeed, the
4 DTZ Ex Ante Evaluation of the 2007-2013 ESF Convergence Programme Final Report (March 2007) p.14
ESF Convergence Priority 1 Priority Review Paper
8
ex-ante evaluation specifically noted that there had been an increase in the number
of young people, particularly young men, not in employment, education or training
during 2002-2003 which justified the inclusion of this new Priority within the
programme. Furthermore, the inclusion of this new Priority was supported by the fact
that the attainment of pupils in the final year of compulsory education in the
Convergence region was below the UK and Welsh average.
ESF Convergence Priority 1 was delivered within the context of a fairly stable policy
environment. At the time of its drafting, the Priority was driven by the policy context
set out in One Wales – A Progressive Agenda for the Government of Wales, Wales:
A Vibrant Economy, Wales – A Better County and The Learning Country 2: Vision
into Action. It was also informed by a number of specific educational related policies
such as those set out within Reaching Higher5, Extending Entitlement6, 14-19
Learning Pathways7 and the Skills and Employment Action Plan for Wales 20058.
The Programme Monitoring Committee (PMC) Strategic Framework Summary
Report9 suggests that there had been no major shift in policy relating to young
people since the Strategic Framework was drafted. The original set of proposed
activities therefore remained unchanged although intervention rates were increased
from 54.36% to 65.43%. A review of approved projects under ESF Convergence
Priority 110 suggests that funded provisions were well aligned with the underlying
objectives of the Programme and the wider policy context.
The development and approval of Priority 1 projects has been well documented via
the Annual Implementation Reports (AIR), which spanned 2007 – 2014, and
5 http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/further_and_higher_education/reaching_higher/?lang=en 6 http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/extending_entitlement/?lang=en 7 http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/learning_pathways/?lang=en 8 http://new.wales.gov.uk/topics/educationandskills/policy_strategy_and_planning/127035/skills_and_employment_action_pla?lang=en 9 WEFO Programme Monitoring Committee Strategic Framework Summary Report
10 http://gov.wales/funding/eu-funds/previous/searchprojects1/?view=Search+results&lang=en
ESF Convergence Priority 1 Priority Review Paper
9
generally whilst pointing to a somewhat slow start for the priority shows that good
progress was made during the second half of the programme. During the first two
years, the level of interest demonstrated in this Priority was good with some 100
Project Idea Forms (PIFs) submitted over the course of 2007 and 200811. The
majority of these PIFs were focused on Theme 1 and on addressing the issues
facing NEETs.
The first Priority 1 projects were approved during 2009 with a total of eight projects
having been approved in that year with 53% of the EU grant available for this Priority
committed by this date. By the close of 2009, 1,345 participants had been assisted
via Priority 1. The 2009 AIR notes that the expectation to deliver more strategic,
integrated and collaborative projects meant that ‘some projects have not progressed
to full approval as quickly as originally envisaged and the start of delivery had, in
some instances, been delayed because of longer than anticipated procurement
procedures’12.
The number of approved projects doubled by 2010 (to 16) and increased further to a
total of 20 projects by 2013, which by 2014, accounted for 100% of the EU grant
available for this Priority.
By the end of 2013, the number of participants engaged stood at 107,926 with
18,930 having gained qualifications. The 2013 AIR reports that some projects had
been incorrectly claiming participants falling outside the eligibility requirements for
the Programme and Priority and that it had been necessary for these projects to
undertake a data cleansing exercise. The 2014 AIR reports that Priority 1 ‘continues
to perform ahead of expectations’ and half of funded projects within the priority had
been completed or closed by 2014. By the end of 2014 121,508 participants had
been assisted and 22,222 gained qualifications.
11
West Wales and the Valleys ESF Convergence Programme 2007-2013 Annual Implementation Reports 2007 and 2008 12
West Wales and the Valleys ESF Convergence Programme 2007-2013 Annual Implementation Report 2009 p.72
ESF Convergence Priority 1 Priority Review Paper
10
3 Funded projects: ‘fit’ with intervention logic
In all, 20 projects were approved via Priority 1 of the 2007-2013 European
programme and were awarded funding totalling £129.7m. A total of 13 were funded
via Theme 1 and 7 were funded via Theme 2 of this Priority:
Figure 2: Priority 1 approved projects
Project title Lead sponsor Theme
Regional SEN Transition To Employment Initiative
Caerphilly County Borough Council 1
A KS 3 Intervention Blaenau Gwent County Borough Council 1
Pupils Understanding Problems in their Locality
Cynon Valley Crime Prevention 1
Reach The Heights Routes To The Summit
WAG DCELLS Lifelong Learning and Skills 2
Reach The Heights First Footholds WAG DCELLS Lifelong Learning and Skills 1
Technocamps Swansea University 2
Building The Future Together Rhondda Cynon Taf County Borough Council 1
Gypsy Traveller Learning and Future Employment Project
Pembrokeshire County Council 1
Young BME People Aiming High All Wales Ethnic Minority Association 1
Amlwch Skills 16 plus Hyfforddiant Parys Training 2
Minority Ethnic Learning and Achievement Project
WAG DCELLS Lifelong Learning and Skills 1
Think Differently Priority 1 ESF Convergence
Fairbridge De Cymru 1
Llwyddon Lleol Creating a Culture of Enterprise amongst YP
Cyngor Gwynedd Council 2
Engage Neath Port Talbot County Borough Council 2
Pathways to Apprenticeships WAG DCELLS Lifelong Learning and Skills 2
Prevent 14 - 19 Bridgend County Borough Council 1
Stem Cymru Engineering Education Scheme Wales 2
Potensial Cyngor Gwynedd Council 1
World Class Traineeships with engagement for Young People
WAG DCELLS Lifelong Learning and Skills 1
Raising Skills and Aspirations of Young BME People II
YMCA Swansea 1
ESF Convergence Priority 1 Priority Review Paper
11
Source: Welsh Government http://gov.wales/funding/eu-funds/previous/searchprojects/?view=Search+results&lang=en accessed 12 April 2016
A review of funded projects’ objectives suggests that most of the proposed activities
within this priority were well covered and that interventions attempted to deliver
unique provisions which intended to complement mainstream educational provision.
Several projects were specifically focused upon key target groups identified within
the programme documents, particularly sub-groups such as young offenders, young
people from Black and Minority Ethnic (BME) communities (three projects were
specifically focused on this group) and young people from the traveller community. It
was also the case that two projects were specifically focused on STEM13 subjects. It
appears that an appropriate balance was achieved between projects which were
focused upon supporting young people who were already inactive and those which
were focused on supporting young people still in education (with an objective of
improving school attendance). Other funded projects took a more generic approach
in that they were intended to support more than one area of the indicative activities
proposed within the Operational Document – for instance, Potensial aimed to work
with a large number of participants to ‘engage, re-engage, motivate and inspire ...
through a series of activities which are both innovative and established through three
linked strands’14.
Two of the proposed activities, notably creating linkages to support young people
into higher education and using mentors as positive role models in schools to raise
awareness of the benefits of higher education, do not appear prominently within
project objectives. Indeed, there was no specific result indicator relating to
progression into higher education for this priority, which may have restricted the
development of activities in this area.
Furthermore, it is questionable whether the indicative activity contained within the
Operational Document relating to funding ‘initiatives which promote improved
understanding of sustainable development’ was well covered by funded projects,
given that no single project was specifically focused upon this objective – although
several projects intended to embed this as part of their provision.
13
Science Technology Engineering and Mathematics 14 http://gov.wales/funding/eu-funds/previous/searchprojects/80625?lang=en accessed 12 April 2016
ESF Convergence Priority 1 Priority Review Paper
12
The key conclusions made within the Thematic Evaluation15 in respect of this
Priority’s fit and design were that:
Projects funded varied in terms of their aims and objectives, size, complexity,
geographical coverage, intended target groups and project sponsor
Projects were positively utilising ESF funding to test innovative approaches
which were beginning to have a long-term impact upon provision within
mainstream services
Projects included provisions which focused on building capacity, regional
collaboration, young people’s participation and mainstreaming the cross-
cutting themes
Projects were also intended to complement mainstream provision and set out
to increase the capacity of existing services and influencing mainstream
provision to be more responsive to the needs of certain young people
The delivery of more strategic, larger projects was considered to have been
worthwhile and necessary
The involvement of local authorities in the delivery of the Priority was
considered to have been beneficial
The high demand for projects’ activities was evidence that they were filling
gaps in services – particularly in terms of support for young people with
mental health issues, social and emotional wellbeing and self-esteem
Projects were targeting the groups of young people whom were expected to
be supported although it proved difficult to assess whether participants were
receiving a ‘meaningful interaction’ given the relatively limited data which was
available on this.
Furthermore, the Thematic Evaluation of Priority 116 concluded that:
15
GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’ 16
GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’
ESF Convergence Priority 1 Priority Review Paper
13
‘Evidence from projects suggests that ESF funding has resulted in many projects
trying innovative approaches; testing successful approaches at scale or extending /
developing existing provision.’
It also went on to argue that:
‘There is evidence to suggest that many projects are effectively complementing
mainstream provision. Some projects are starting to help raise the capacity of
other services and influence mainstream provision to be more responsive to the
needs of certain young people who have previously been overlooked’17.
The Thematic Evaluation found that Priority 1 had been effective in promoting ‘joint
working across organisations with different agendas and in different localities’ and
that ‘projects that built on previous experience of ESF funded interventions were
characterised by more effective partnership arrangements and buy-in’18.
17
GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’ p.55 18
GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’ p.14 and p.18
ESF Convergence Priority 1 Priority Review Paper
14
4 Targets and achievements
Figure 3 presents Priority 1 Indicators and Targets and the final outturn achieved:
Figure 3: Priority 1 Indicators and Achievements as at August 2016
Indicator Original
Target
Revised
Target
Achieved
Outputs
Total participants (11-19 year olds) 35,000 35,000 134,031
Female participants 40% 40% 46%
Employers collaborating with education/training
providers
500 500 2,201
Systems developed / / 5
Projects using soft outcome measurement
systems
50% 50% 60%
Projects integrating sustainable development into
awareness raising, education and training
programmes
75% 75% 55%
Results
Participants gaining qualifications (11-19 year
olds)
10,500 10,500 25,143
Participants entering further learning – 16-19 year
olds
21,000 21,000 12,357
Participants gaining other positive outcomes19 –
11-19 year olds
31,500 31,500 35,760
Participants entering employment (16-19 years
old)
/ / 4,617
Source: Welsh European Funding Office (August 2016)
In line with performance across the ESF programme overall, it is worth observing
that the Priority participant output was substantially exceeded (nearly four-fold) with
19
Defined as intermediary outcomes such as completing courses, entering voluntary work and attending a job interview.
ESF Convergence Priority 1 Priority Review Paper
15
134,031 participants engaged against a target of 35,000. All other output targets
were also exceeded other than the proportion of projects which integrated
sustainable development into their provision, at just over half doing so. This is clearly
a very impressive performance by projects in engaging such a large number of
participants over the course of delivery – the baseline data20 suggests that 248,500
participants aged between 11 and 19 years old formed the baseline for this Priority,
hence the Priority would have reached and engaged just over half of this overall
population.
In terms of performance against its Results Indicators, the Priority performed
exceptionally well having exceeded its targets of participants gaining qualifications
and participants gaining other positive outcomes. However, the proportion of all
participants assisted who gained a qualification or other positive outcomes formed a
fairly low proportion of the total number of participants supported (i.e. only 19 per
cent of all participants gained a qualification whereas the original indicator suggest
that 30 per cent would do so).
The Priority did not achieve its target for the number of participants entering further
learning having only achieved 59 per cent of its 21,000 target. This is particularly
disappointing given that the Priority assisted a much higher number of participants
than was originally set. No targets were set for the number of participants entering
employment. However it is encouraging to observe that 4,617 did so. If we were to
amalgamate all of the results achieved (and assuming that no participant would have
achieved more than one result each) then 42,117 (or 31 per cent of all participants)
would have either secured a qualification, entered further learning or secured
employment.
As set out at Figure 4 the original budget allocation for Priority 1 stood at just over
£216m, accounting for around 15 per cent of the overall WWV ESF Convergence
Programme funding for 2007-2013. Actual final expenditure, at £197m, was within 91
per cent of the original budget allocation and final EU Grant expenditure, at £105m,
was within 90 per cent of the original budget allocation.
20 Operational Programme Annex K p.4
ESF Convergence Priority 1 Priority Review Paper
16
Figure 4: Priority 1 Funding
£ Total Funding EU Grant
Original Budget Allocation 216,336,164 117,807,004
Revised Budget Allocation 157,387,912 106,681,805
Cumulative (Actual) Expenditure 197,372,079 105,149,583
Source: Welsh European Funding Office
When considering levels of investment per output compared with what was
envisaged Figure 5 shows that the actual investment per participant engaged was
four times lower than anticipated, at £1,473 per participant, thereby providing better
value for money – although the original budget, at £6,181 per participant, would
appear high. Similarly, the actual investment per participant gaining qualifications
was much lower than planned whilst the investment per participant gaining other
positive outcomes was marginally lower. However, the actual investment per
participant entering further learning, at £15,973 per participant, was much higher
than expected.
Figure 5: Original and Actual Investment per Outputs
Original
Budget
Allocation
£
Final Actual
Expenditure
£
Investment per participant engaged 6,181 1,473
Investment per participant gaining qualifications 20,603 7,850
Investment per participant entering further learning 10,302 15,973
Investment per participant gaining other positive
outcomes
6,868 5,519
Investment per participant entering employment n/a 42,749
Source: OB3
5 Cross cutting themes
ESF Convergence Priority 1 Priority Review Paper
17
This Priority was expected to incorporate the two cross cutting themes of Equal
Opportunities and Environmental Sustainability. An evaluation focused upon the
implementation of the Programmes21 concluded that funded projects had integrated
the Cross Cutting Themes (CCTs) more thoroughly into Project Business Plans at
the application stage than in previous Programmes, but that the extent to which
commitments made had been undertaken on the ground were more mixed.
The CCTs evaluation commissioned by the Welsh European Funding Office (WEFO)
22 (while not being specific to ESF Priority 1) found more generally that:
The lack of incentive was the single most reported barrier for all projects in
achieving their CCTs23
ESF funded projects were more likely to report ‘lack of training for Sponsor’,
‘WEFO change targets’ and ‘Partners not supporting CCT work’ as barriers to
achieving their CCTs24
The CCTs were not considered a ‘top priority’ due to a perception that ESF
participants did not want to engage with CCTs25.
As far as Priority 1 projects were concerned, the cross cutting theme of Equal
Opportunities lay at the heart of most projects’ objectives. The ESF Convergence
Evaluation26 found that across Priority 1, WEFO had funded a number of projects
which had a strong focus upon equal opportunities, particularly targeting vulnerable
groups, including the Regional SEN project, Transition Key Workers under Research
the Heights and the Gypsy Traveller projects. Other funded projects were expected
to approach equal opportunities as part of a mainstream approach e.g. activities
targeting female participation in STEM were commissioned as part of specific STEM
projects.
21 Old Bell 3 (January 2011) ‘European Structural Funds Programme 2007 – 2013 The Effectiveness of Implementation in the 2007 – 2013 Structural Funds Programming Period Final Report’ p.67 22
WEFO Cross-cutting Themes Evaluation. Equality and Sustainability. March 2015. Parker. J et al. 23
Ibid., p.41 24
Ibid., p.42-43 25 Ibid., p.79 26
GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys Final Report’
ESF Convergence Priority 1 Priority Review Paper
18
The key conclusions relating to the cross-cutting themes within the ESF
Convergence Thematic Evaluation were that:
Most Priority 1 projects were aware of the Priority specific guidance on the
implementation of the two cross cutting themes
There was evidence of good practice in areas of both equal opportunities and
environmental sustainability but that ‘good practice was not entirely universal
across the projects, and there were also several projects that took more of a
‘tick-box’ approach to one or either of the cross-cutting themes’27
Some project partners were reliant upon existing local initiatives, but offered
little evidence as to how their projects incorporated these into their provision
for young people or added value to their services
In some cases, project sponsors were ineffective in getting partners involved
in implementing the cross-cutting themes
Cross cutting themes’ monitoring indicators were regarded as being
insufficient to capture the benefits for young people
Whilst pockets of good practice in relation to environmental sustainability were
found across the projects (e.g. promotion of green jobs and skills for young
people) there was no evidence that this had led to widespread innovation.
The evaluation drew attention to the feedback from projects and stakeholders
that ‘a small number of vertical projects around young people and
sustainability might be a way to raise the importance of sustainability in
relation to ESF in the future.’28
Overall, the evaluation concluded that ‘the implementation of cross-cutting themes
into Priority 1 was generally thought to have been effective, particularly in relation to
equal opportunities. There were examples of outstanding practice, but this was not
consistent across all projects’29.
A review of final project level evaluations suggests that:
27
GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys Final Report’ p.21 28 GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys Final Report’ p. 22 29 Ibid., p.58
ESF Convergence Priority 1 Priority Review Paper
19
Across the board, Priority 1 projects were complying with, and adopting their
own organisations’ policies and practices in relation to equal opportunities and
environmental sustainability. It was frequently reported that these policies and
practices had been embedded into the overall delivery as a matter of course
Overall, greater prominence was given to the cross-cutting theme of equal
opportunities across Priority 1 projects, largely due to the fact that equal
opportunities was a core principle and underlying objective of many Priority 1
projects in that they were specifically targeted at particular groups of young
people who were disadvantaged or not fully participating in educational or
economic opportunities. Indeed, a few projects were specifically focused on
such groups, including gypsy traveller communities, BME communities, young
people with special educational needs, and those either at risk, or were
already, NEET
Despite this, at least two project level evaluations concluded that projects
could have done more in the area of equal opportunities including
encouraging learners to consider a non-traditional route for their gender and
taking a more proactive approach to target and recruit female participants
None of the Priority 1 projects were exclusive focused on environmental
sustainability issues as had been the case for equal opportunities. Despite
this, at least five Priority 1 projects were deemed to have deployed innovative
and appropriate approaches to the cross-cutting theme of environmental
sustainability. These included the creation of an outside learning area for
school pupils to learn about nature and the environment, the provision of work
placement opportunities with employers operating within the environmental
sector and sector specific events covering topics such as renewable energy,
recycling and sustainable developments
At least two evaluations concluded that more could have been achieved by
some projects in relation to environmental sustainability – for instance in the
case of one project, the evaluators concluded that the aim of offering young
people the opportunity to undertake work placements and volunteering in
environmental organisations was not achieved
ESF Convergence Priority 1 Priority Review Paper
20
More generally, final project level evaluations suggest that greater awareness
and appreciation of the need to incorporate the cross-cutting themes within
delivery was experienced by projects over time.
ESF Convergence Priority 1 Priority Review Paper
21
6 Evaluation evidence: common barriers and success factors
This section explores some of the common success factors and barriers reported
across project level evaluations, drawing upon the findings of the Priority 1 thematic
evaluation as appropriate. It then discusses some of the externalities which impacted
implementation and the extent to which funded projects had considered exit
strategies and longer term sustainability.
Success factors
A review of project level evaluations reveals that Priority 1 projects were considered
to have had very few common success factors in place which accounted for strong
project performance. The following points give a flavour of the key factors which
were deemed to have been in place across projects, although it is worth noting that
no more than two or three examples were reported for each one:
After what was regarded as a generally slow start across most Priority 1
projects, at least two evaluations reported that a greater focus on trying to
achieve funded targets over the final stages of delivery accounted for the fact
that targets were eventually achieved or exceeded. For instance, one such
evaluation report noted that there had been ‘an urgent effort to recruit
participants towards the end’ in order to meet EU funded targets
Several evaluations reflected upon good practice and lessons learnt in terms
of effective participant engagement techniques – at least two projects which
had involved schools had found schools an effective recruitment method.
Others highlighted the importance of engaging parents, families and the home
as part of the recruitment strategy. Interestingly some two evaluations
reported that whilst projects had successfully engaged particular sub-groups
of young people (e.g. those living in poverty, those with Special Educational
Needs (SEN), care leavers) the recruitment of other sub-groups (e.g. ethnic
minorities) had proved to be more challenging. The Thematic Evaluation also
reported that there was evidence of good practice in terms of engaging young
people and assessing their support needs
The on-going tracking of progress being achieved by participants – together
with follow-on support as appropriate, even after their involvement with the
ESF Convergence Priority 1 Priority Review Paper
22
project had come to an end, was considered to have been good practice in
enabling projects to secure, capture and thereby report upon the outcomes
achieved by participants. Aligned to this it was also found that reviewing
participant record data had led to a massive upswing in the outputs reported
for one project
Some (around three) evaluations reported that project success could be (in-
part) attributed to the individualised approach adopted to supporting young
people. The key success factors were considered to have been a holistic
approach which offered both emotional and educational support, engaging
young people in the design of the project and offering young people a high
degree of choice in the activities which they engaged with
Several evaluations concluded that the nature of the services and support
offered had been vital components in contributing to the strong performance
of projects against their funded targets, particularly in terms of their ability to
engage individuals, but also in terms of good satisfaction levels amongst
participants. For instance, at least two evaluations reported that the provision
of work experience opportunities had accounted for a significant element of
the projects’ success in meeting its outputs whilst another reported that the
delivery of highly practical activities which were distinct from traditional
classroom activities had helped to recruit and retain participants. In another
case the content of the programme offered was found to have resulted in high
levels of completion, attainment and satisfaction levels
It was not uncommon for evaluation reports to highlight the skills, quality and
attributes of front-line personnel as being key contributors to projects’ strong
performance. Aligned to this it was also argued that success was attributable
to staff’s ability to develop positive relationships with young people and that
trust and friendship had assisted in generating project outcomes achieved.
Common barriers
Some of the most commonly cited factors which were considered to have accounted
for project under-performance (or contributed to delivery difficulties even in those
cases where targets had been achieved) were considered to have been:
ESF Convergence Priority 1 Priority Review Paper
23
Practical challenges relating to the delivery of large-scale projects involving
several partners which had manifested itself in issues relating to
communication, the role of the lead partner, reporting and management of
information. It was also reported in at least one case that the performance of
individual partners had varied with the poor-performance of some partners
accounting for the overall weak performance of the project.
Efforts deployed to meet high level EU funded targets were on a few
occasions thought to have detracted from the project’s ability to deliver more
meaningful support to young people. For instance, in one case it was reported
that participants joining the project towards the end (in an effort to meet
participant engaged targets) did not benefit from the range of training and
follow up support typically available
Issues related to reporting and MI systems which included instances of project
level MI databases being put in place later than desirable (thereby possibly
accounting for some under-reporting or inaccurate reporting of outputs
achieved). In other cases, it was reported that there had been an over-focus
upon the capturing and reporting of ESF related outputs, with very little other
MI data collected leading to a very narrow range of outputs and outcomes
being monitored. The Thematic Evaluation also highlighted issues in relation
to the consistency and timeliness of monitoring, including the adoption of
different interpretations of particular indicators and monitoring indicators not
allowing projects to evidence the impact ‘across the breadth of project
activities’30
Difficulties targeting and recruiting particular sub-groups of young people,
including those who are NEET, ethnic minorities, disabled young people and
limited use of engaging young people and their parents. Some of these
difficulties arose from a lack of capacity to recruit from these groups given that
projects were already over-subscribed whereas in other cases projects were
mostly reliant upon participant referrals made from partner organisations
30
GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’ p.58
ESF Convergence Priority 1 Priority Review Paper
24
The Thematic Evaluation31 found that projects took longer than expected to
begin full delivery due to three factors namely complexity and compliance with
different procurement procedures, lack of experience amongst project
sponsors of ESF and communication issues between WEFO and project
sponsors as well as between project sponsors and their partners.
Externalities
Very few (around six in all) evaluations cited significant externalities which impacted
upon the performance of the projects, although the three key major factors were
considered to be:
The impact of the economic downturn (cited by at least four evaluation
reports) which was deemed to have reduced opportunities to work within the
labour market – the two-fold impact of this was thought to have been (a) fewer
opportunities to support young people into employment opportunities (thereby
impacting upon project targets) and (b) a natural increase in the numbers of
young people being attracted to training or further learning, which in some
cases was considered to be having a positive impact upon projects’ abilities to
meet their targets. In one case it was reported that a project had responded to
these externalities by promoting volunteering, rather than work, opportunities
instead. In another case, it was reported that participants were more ready to
accept that they required education and training in light of the economic
downturn
The impact of public sector funding cutbacks which were considered to have
had various effects including difficulties securing initial match funding for
projects (given that the original match funding requirements were set prior to
the economic downturn) – in one case it was noted that ‘far less funding was
available than had been anticipated at the outset’ but also in that the cutbacks
would have bearing upon the project’s likelihood of being sustained, post EU
funding
Factors associated with delivery and partner organisations, including the lack
of strategic engagement and support for projects across the wider network
and structural changes across delivery partners.
31
GHK (January 2012) ‘Thematic Evaluation of ESF Convergence Priority One in West Wales and the Valleys – Final Report’
ESF Convergence Priority 1 Priority Review Paper
25
Sustainability and Exit Strategy
At the time of undertaking the evaluations, very few projects had concrete plans in
place to sustain provision post EU funding largely due to the challenges of securing
ongoing funding given public sector financial cutbacks and a lack of funding options.
Indeed, drawing upon eight evaluation reports which specifically covered project
sustainability issues one reported that a project partner had committed to the on-
going funding of its own delivery for a further year whilst another evaluation reported
that a project lead sponsor had made applications for funding to various sources to
continue provision. Other than for these cases it was commonly reported that the
long term sustainability of projects was ‘uncertain’ and in one case the project
structure had already come to an end. In the case of two Priority 1 projects it was
reported that schools were expected to sustain elements of project interventions post
funding and, whilst there was evidence of this happening in at least one case, it was
commonly accepted that pressure upon school budgets meant that interventions
could not be sustained indefinitely or to the scale desired by schools.
ESF Convergence Priority 1 Priority Review Paper
26
7 Evaluation evidence: impact
This section considers the findings of project level evaluations on the impact of
Convergence Priority 1 upon the indicators of participants achieving qualifications,
entering further learning, entering employment and achieving other positive
outcomes). The ESF Leavers’ Survey did not cover Priority 1 Convergence
participants and therefore no conclusions can be drawn from this in relation to the
impact of this Priority.
Priority 1 had four impact related targets (relating to qualifications, entering further
learning, entering employment and other positive outcomes) and an analysis of
project level evaluations shows that in terms of the impact upon participants
achieving qualifications:
Nine of the 16 project evaluations concluded that projects had succeeded in
having at least some impact upon the target of participants gaining a
qualification, although in one of these cases (Raising Skills and Aspirations of
Young BME People) the number of participants gaining a qualification was
considered to be very low
The proportion of participants supported at the project level gaining any
qualification varied significantly across Priority 1. For instance, only 3 per cent
of all participants supported via the Reach the Heights First Footholds and 8
per cent of Reach the Heights Routes to the Summit projects achieved a
qualification in contrast to the higher rate of 91 per cent of all participants
achieving qualifications within the Pathways to Apprenticeships project. Two
factors contributing to the lower proportion of participants achieving a
qualification were identified as being (a) the time lag between engaging,
completing and achieving qualifications and (b) the fact that several projects
were focused on changing attitudes towards learning and education amongst
young people as opposed to the delivery of accredited training
Setting a project level qualification target was deemed to be inappropriate for
some projects: in the case of the Technocamps project the original target for
participants achieving qualifications was removed, in light of the fact that the
project was focusing on supporting young people aged 11 to 13 year olds.
ESF Convergence Priority 1 Priority Review Paper
27
This issue also become apparent across a few other projects, for instance, in
the case of the Gypsy Traveller Learning project it was anticipated that there
would be more participants in the upper end of the 11 – 19 age range who
would have been able to gain qualifications within the project’s lifetime
(although it was also the case that other factors accounted for weaker
performance - such as unanticipated greater focus upon supporting
participants with basic and soft skills who frequently had to overcome
entrenched difficulties in order to secure a qualification)
It was commonly reported that the full impact of intervention upon participants
achieving a qualification was unlikely to be realised until a specific point of
time in the future, such as at the end of an academic year or when younger
participants would be expected to sit their GSCE examinations which often
went beyond the project timescales.
In terms of the impact upon participants’ entering further learning:
Overall the contribution made by Priority 1 to this target was considered to be
relatively modest, if not weak, across project level evaluations. Indeed, this
finding is supported by the fact that Priority 1 under-achieved against this
target. Several common reasons were identified for this weaker than
anticipated impact: first, it was accepted that a number of Priority 1 projects
were actively engaging participants who were already in education and
therefore better placed to make a difference to their attendance and retention
levels rather than future progression. Second, it was widely reported that it
would take time for funded projects to have bearing upon participants’
progression into further learning, particularly those which focused on the
younger cohorts – for instance the evaluation of the KS3 Intervention Project
noted that it would take ‘several years before young people make choices
about subjects and careers and the impact can be measured empirically.’
Third, a few evaluation reports highlighted the lack of data available to
evidence participants’ progression into further learning, particularly in those
cases where projects had come to an end and where it would take time for
projects to have bearing upon this indicator. For instance, the evaluation
ESF Convergence Priority 1 Priority Review Paper
28
report on the Technocamps project32 suggests the need for a longitudinal
study to gather evidence on entry into further learning amongst supported
participants
Where data were available, progression into further learning appears to have
been fairly low – for instance only 106 Reach the Heights Routes to the
Summit participants (less than one per cent of all project participants) were
found to have progressed into further learning although 7 per cent of Reach
the Heights First Footholds participants (1,144) had done so33. Drawing upon
project level evaluations the highest proportion of supported participants who
had entered further learning (at 16 per cent) was found to be the case of
Raising Skills and Aspirations of Young BME People project
It was also commonly the case that projects had under-achieved against their
own targets for participants progressing into further learning (despite targets
having frequently been revised downwards in light of weaker performance).
This was particularly the case for Pathways to Apprenticeship – which was
considered to have fallen well short of its target (and ultimately its overall
objective) of supporting participants to progress into apprenticeship
opportunities given that only a third of participants progressed onto an
apprenticeship. In this context, some two-thirds of learners reported that their
destination on leaving the project was their preferred destination. However in
the case of a third of those who did not progress onto an apprenticeship it was
reported that an apprenticeship had not been available to them whilst a
smaller proportion had applied but were unsuccessful34.
In terms of the Priority’s impact on supporting participants to enter employment:
As might have been expected given that Priority 1 did not have any targets for
supporting participants into employment, project level evaluations found that
this Priority had had a very limited impact upon supporting participants to
enter employment: indeed, this finding was reinforced by the fact that only
32 Wavehill ‘An Independents Evaluation of the Technocamps Project’ p.61 33 Reach the Heights consisted of two projects – First Footholds, funded under Theme 1: Tackling Underachievement (which aimed to improve the educational outcomes of young people at risk of underachieving), and Routes to the Summit, funded under Theme 2: Raising Skills and Aspirations (which aimed to support young people to make a successful transition into employment) 34 BMG Research ‘Final Evaluation of the Pathways to Apprenticeship programme’ p.11
ESF Convergence Priority 1 Priority Review Paper
29
three per cent of all participants engaged (or 4,617) were supported into
employment within this Priority
At least eight projects were considered to have had no impact upon
supporting participants into employment, largely due to their focus on working
with participants under 16 (i.e. below the working age). Seven projects were
considered to have had at least some impact – although generally the number
of participants concerned were low e.g. 41 of the 1,054 participants supported
on the Raising Skills and Aspirations of Young BME People project entered
employment
The most commonly cited factors which were considered to have accounted
for the low numbers of participants entering employment were: the lack of
progression data available for participants once their involvement with the
project had come to an end; the fact that projects were supporting participants
aged from 11 who would not be expected to enter the workforce for a number
of years after their involvement with the project; the general state of the
economy and lack of job opportunities
Despite this, there is good evidence available across evaluation reports to
show that projects were impacting upon participants’ attitudes and perception
of work as well as equipping young people with the appropriate skills and
abilities for future work. In a few cases this stemmed from the focus within
projects to provide work placement experiences to participants, including
supported work placement opportunities in some cases, whilst others were
geared to raise aspirations and inspire young people to explore careers within
particular sectors (such as STEM). The findings of one evaluation (Building
the Future Together) sums these issues succinctly: ‘the project is currently
below target in terms of the number of young people entering employment
and there was some evidence that young people being supported lacked
confidence in their abilities which would help them gain employment in the
future. Encouragingly the positive impact upon intermediate outcomes for
employment [e.g. helping young people gain job interviews] suggests that
employment outcomes should improve as the work continues’
It was also the case in one project (World Class Traineeships) that the evaluation
found intervention had a greater impact upon progression within work rather
ESF Convergence Priority 1 Priority Review Paper
30
than participants entering work i.e. a third of those working for the same
employer as before had had a promotion since completing their
apprenticeship35.
‘Soft’ or intermediate outcomes for participants
Project level evaluations reported extensive soft outcomes for participants – with
these being considered frequently in terms of the distance travelled by participants.
Indeed, one evaluation36 report noted that the project’s intervention to help a
participant leave their bedroom could be considered a greater achievement than for
another participant to gain a qualification. The most commonly cited soft outcomes
are summarised below:
An improvement to levels of independence and confidence – some
evaluations reported that participants were being able to undertake tasks or
routines for the first time without family support (e.g. travelling by bus)
Behavioural changes such as participants being able to attend events or
volunteering opportunities on time and demonstrating attributes such as
politeness, respectfulness and being better presented for the tasks in question
Improved health and fitness, particularly in terms of reduced anxiety and
participants being better able to cope with their emotions
Increased dealings with other people and a greater circle of friends
An increase in participants’ ability to concentrate on tasks
An increased awareness and understanding of future career opportunities
An improvement to skills, particularly softer skills such as communication,
team working, leadership and planning skills, and basic skills. In one case it
was argued that the project’s impact upon participants’ soft skills was ‘the
most important impact considering the background of the participants’.
A number of project level evaluations reported that project providers had welcomed
WEFO’s decision to introduce a soft outcomes indicator yet acknowledged that the
capturing of this data was often challenging and occasionally inconsistently applied
within projects. In one case, for instance, it was reported that ‘the reporting [of soft
35 Wilson et al ‘Evaluation of Work-Based Learning Programme 2011-14: Apprenticeships’ p.11 36
People and Work Unit (April 2012) ‘External Mid Term External Evaluation of the Business the Future Together (BTFT) European Social Fund (ESF) project. Final Report’ p.16
ESF Convergence Priority 1 Priority Review Paper
31
outcomes] has been patchy making it problematic to quantitatively evaluate the
overall impact of the project on soft outcomes’37.
Exploring the Priority’s impact upon other positive outcomes achieved by
participants it was the case that all evaluation reports reviewed found that projects
had made achievements in this respect. Indeed, the proportions of participants
reported as having achieved other positive outcomes across projects were generally
high. The most commonly cited outcomes related to:
Completion of training courses, workshops or events
Adopting a personal development plan
Participating in work experience or voluntary work experiences
Improved participation levels in education and training (including improved
behaviour and attendance levels)
Improved personal skills such as time management, team working,
communication and social skills as well as improved basic and essential skills
(numeracy, literacy, IT)
Increased awareness of future opportunities (be they education or work
related) and changes to participants’ perceptions of future opportunities –
particularly in relation to attitudes towards future careers, most notably around
STEM related study and careers
Much of the evidence presented across evaluation reports on the achievement of
these other positive outcomes were short-term in nature. Indeed, it was notable how
little data were available to assess whether these outcomes were sustained in the
long term and would have therefore contributed to concrete achievements such as
improved educational attainment or improved progression into further and higher
education. The main reason for this was due to the fact that measuring long term
impacts was beyond the scope of the evaluation studies commissioned and the
studies lacked any approaches to track the destinations of supported participants
post intervention.
37
People and Work Unit External Mid Term Evaluation of the Building the Future F Together (BTFT) European Social Fund (ESF) project (April 2012) p.16
ESF Convergence Priority 1 Priority Review Paper
32
Deadweight and displacement
Meaningful consideration of the extent to which project intervention had generated
any deadweight (i.e. whether the outcomes achieved would have happened anyway)
was only available in seven of the evaluation reports reviewed, and the findings
suggest a mixed picture.
In two cases levels of deadweight were considered to be high – for instance in the
case of the STEM Cymru project, a high proportion of participants were considered
to be natural scientists who would never have considered a career in anything other
than a STEM related role. In other cases, stronger arguments were presented that
the funding had allowed providers to deliver interventions at a greater scale and
volume than would otherwise had been the case. In the case of one evaluation
study, the relatively high levels of prior qualifications amongst participants raised
questions about the ‘duplication of public funding’38 although it was acknowledged
that this issue had to be considered within the context of the project’s objective being
to help participants gain employment.
Most evaluation reports pointed to there being very little displacement of other
services or existing provision as a result of projects funded via this Priority and the
evaluations generally suggested that other support services had not been reduced
as a result of funded activities. Indeed, a number of evaluation reports provided fairly
robust evidence of how projects had managed to integrate their provision across
existing services such as within schools and colleges or youth and welfare support
providers. A few evaluation reports captured evidence on those activities which
participants would have missed out on as a result of participating in Priority 1
projects – these were mostly classroom based lessons which participants would
have missed as a result of attending project interventions.
Value for money
Figure 6 presents the evidence drawn from five evaluation reports on the value for
money offered by Priority 1 projects in terms of the investment per participant and
outcomes achieved. The data shows that average costs varied significantly – some,
38 Wilson et al (2016) ‘Evaluation of Work-Based Learning Programme 2001-14: Apprenticeships’ p.165
ESF Convergence Priority 1 Priority Review Paper
33
such as the Potensial project, reported very high average costs whilst others (such
as Pre-Vent) reported lower average investment per outcome. An exceptionally low
investment per participant was reported for the Technocamps project at £24 per
participant which was a reflection of the one-off, lighter touch involvement of many
participants in group based workshops or events.
Figure 6: Investment per participant and per outcome
£ Investment
per participant
Investment
per participant
entering
further
learning
Investment
per participant
gaining
qualification
Investment per
participant
gaining other
positive
outcomes
Engage 1,719 5,403 3,584 7,258
Building the
Future
Together
1,381 22,573 4,449 3,048
Potensial 3,389 29,759 10,131 8,861
Pre-Vent KS3 1,416 5,252 7,956 4,269
Technocamps 24 n/a n/a n/a
Priority 1
1,473
15,973
7,850
5,519
Source: Project level evaluation reports
In the small number of cases where project level evaluations gave meaningful
consideration to the value for money offered by interventions, a mixed picture was
found – a small number concluded that the value for money offered had been
excellent, others (such as in the case of Amlwch 16+) concluded that the value for
money provided had been good whereas others concluded that better value for
money could have been achieved had the project been delivered somewhat
differently. In those cases where projects were considered to offer good or excellent
value for money it was reported that interventions often ‘added value’, addressed
gaps and enhanced existing capacity (e.g. such as in schools) as well as enabling
provision to be delivered on a greater scale and therefore generate economies of
scale. It is perhaps a reflection of the nature of intervention (i.e. that any impact of
ESF Convergence Priority 1 Priority Review Paper
34
interventions would only be achieved in the long term) that so few of the project level
evaluations could offer a more robust calculation of the value for money offered.
A small number of project level evaluations reported differences in the value for
money offered within projects – for instance investment per participant or per
outcomes varied by geographical areas and/or by different partners. These
differences on occasion were considered to have been influenced by the additional
‘rural’ premium associated with the delivery of intervention in a rural county.
It was widely argued across project level evaluations that funded interventions had
often been able to ‘add value’ to existing capacity, particularly across schools and
colleges – thereby offering an acceptable return on investment. Some project
evaluations concluded that interventions had increased the ‘capacity’ of existing
providers (e.g. schools,) to better cater for the needs of specific groups of young
people, particularly challenging young people.
Furthermore, project level evaluations highlighted some common features on how
projects could have offered better value for money. These included:
getting established and underway at a quicker pace
a more consistent performance across all delivery partners
adopting a different implementation model with lower management costs
greater involvement of regional and local stakeholders – particularly to avoid
any duplication in provision.
Very little evidence on the financial return on investment made was offered across
project level evaluations. Indeed, only two calculations were identified within the
review, both of which suggested a very good level of financial return:
STEM Cymru calculated that every £1 spent the Convergence Area resulted
in a financial gain of £3 in benefits for the area, which was considered to offer
‘incredible value for money’
Pathways to Apprenticeship – the evaluation found that for every £1 spent
each learner would generate £3.02 in increased earnings and that £2.12
ESF Convergence Priority 1 Priority Review Paper
35
would be generated in increased direct and indirect revenues to public
budgets.
Counterfactual
Very few project level evaluations were found to have attempted some form of
counterfactual impact evaluation and where they had done so, the usefulness of the
data was fairly limited. Overall, there is a lack of robust counterfactual evidence
available for this Priority.
The following counterfactual evidence was presented within project level evaluations:
In the case of the Pathways to Apprenticeships (PtA) project a cost benefit
analysis was undertaken using Propensity Score Matching to analyse the
outcomes for participants against a group of similar learners from the Lifelong
Learning Wales Record (LLWR) database. The analysis revealed that the
total net benefit to participants was £88.316m whilst the total net benefit to
public budgets was £61.906m against a total cost of delivering the project at
£29.26m. It is estimated that PtA would generate £3.02 in increased earnings
for PtA learners for each £ spent in delivering PtA and would generate £2.12
in increased direct and indirect revenues to public budgets for each pound
spent in delivering PtA
In the case of STEM Cymru, applying an overall deadweight of 50% to the
calculation, the project level evaluation reported that the project produced an
economic impact of £4m. The net economic impact was calculated to be
£1.2m having taken into consideration factors such as participants taking up
jobs outside the Convergence area and the depressed state of the labour
market in Wales.
For the World Class Traineeships project, a Propensity Score Matching
analysis was used to match apprenticeship leaver survey respondents with
data from the Annual Population Survey (APS), with a particular focus on the
earnings growth of the two cohorts. The analysis revealed that estimates of
the earnings growth amongst surveyed apprenticeship participants were
higher than those derived for the wider population from the APS. However,
when the differences in the composition of the two groups was taken into
ESF Convergence Priority 1 Priority Review Paper
36
consideration (i.e. respondents to the apprenticeship leaver survey were
relatively younger) no significant difference in earnings growth between the
treated and control groups was revealed
The analysis for the Engage project found that Engage eligible areas
significantly outperformed a control group area when considering the number
of Year 11 leavers known not to be in education, training or employment with
a 39 per cent reduction in the number of NEETs between 2009 -2011 within
Engage areas compared with a 22 per cent reduction across non-Engage
areas. The evaluation reports that had Engage areas performed in the same
manner as the control group areas, an additional 83 year 11 leavers would be
NEET
In the case of the Key Stage 3 project, the achievement of young people
eligible for Free School Meals (FSM) was used as a proxy indicator or
measure of the impact of the project upon young people at risk of being
disaffected. The analysis indicated that the performance of pupils eligible for
FSM achieving the expected level in the Core Subject Indicator (CSI) in two
authorities increased more rapidly than the Welsh average between 2010 and
2011 and increased more rapidly than the trend in recent years between 2007
and 2010 within these local authorities.
The experience of undertaking the matching exercise as part of the World Class
Apprenticeship project evaluation reinforces a common view conveyed across
several other project level evaluations i.e. that it is not realistic for participants to
expect benefits to be realised immediately after receiving a Priority 1 type
intervention, with outcomes likely to be longer-term in nature. The World Class
Apprenticeship evaluation noted that ‘it may not be realistic to expect that benefits of
participation in a training programme of 2.25 years duration will be realised within 12
months of its completion’39.
In some cases it is evident that the lack of medium to long term data has hindered
efforts to quantify the counterfactual position, for instance:
39Wilson et al (2016) ‘Evaluation of Work-Based Learning Programme 2001-14: Apprenticeships’ p.162
ESF Convergence Priority 1 Priority Review Paper
37
‘Limitations in the quantitative data meant a judgement could not be made on what
would have happened without the interventions’40.
‘A system to track destinations of participants and the sustainability of these
destinations post Potensial intervention would provide further evidence of project
impact. A detailed and centralised record of which participants received support
under which specific interventions, along with hard outcomes and soft, standardised
distance travelled data would be required to compare interventions and establish
which are the most effective.’41
40 People and Work Unit ‘External Mid Term External Evaluation of the Building the Future Together (BTFT) European Social Fund (ESF) project. Final Report’ p.14 41 Wavehill (March 2014) ‘Final Evaluation of the Potensial Project’ p.7
ESF Convergence Priority 1 Priority Review Paper
38
8 Evaluation evidence: other outcomes and unintended consequences
This section considers the evidence, principally from project level evaluations of
other outcomes and any unintended consequences of delivery.
Project achievements not reported to WEFO
The key achievements not systematically reported as outputs against this Priority
were mainly:
That supporting young people to stay within education had been an important
objective, even the key objective in some projects like Engage, that perhaps
was not being fully acknowledged when included against the broader
headings of ‘soft outcomes’ or ‘other positive outcomes’ set for this Priority.
This led one evaluation report42 to argue that ‘A participant remaining in
education should represent a key, claimable output’ for future projects
In a similar manner, several project level evaluations highlighted very good
achievements in terms of improved attendance levels, reduced permanent
exclusions and unauthorised absenteeism amongst supported participants
which were not being reflected within the broad category of ‘other positive
outcomes’
There was some evidence to suggest that part-qualifications would not have
been reported as such (but rather captured as other positive outcomes)
Improved partnership working between various organisations and the
resulting improved integration of services and interventions
Some evidence (from at least two project level evaluations) that funded
interventions were helping to reduce levels of discrimination towards the
target groups being supported.
Indirect positive effects
Very little evidence was available to substantiate the indirect positive effects of
Priority 1 funded projects. Indeed, only two project level evaluations provided any
calculation on the savings to public expenditure or increased lifetime earnings for
42 42 Wavehill (March 2014) ‘Final Evaluation of the Potensial Project’ p.7
ESF Convergence Priority 1 Priority Review Paper
39
participants: there was no consideration of these issues in other evaluation reports.
These were:
Savings to public expenditure
o In the case of one project (SEN) it was calculated that the net saving to
local authority per participant per year would equate to £12,314
(savings which would be related to transport and staff costs)
Increased lifetime earnings for participants
o Pathways to Apprenticeship – the evaluation found that for every £1
spent each learner would generate £3.02 in increased earnings.
Unanticipated negative effects
The most commonly cited unanticipated negative effect related to the uncertainty
over the long term sustainability of project interventions – particularly given that
several projects had been able to improve the referral systems and pathways for
young people to access other support services. The potential loss of the brokerage
service deployed by some projects to link participants and other support services
was considered a threat which could lead to the discontinuation of support in such
cases.
ESF Convergence Priority 1 Priority Review Paper
40
9 Availability and quality of project level evaluation evidence
Introduction
In total, 16 project-level evaluations were available for review against a base of 20
projects which were approved via this Priority. In the case of two projects43, a
combined evaluation was commissioned and published (i.e. there were 15 evaluation
reports). Of the 15 reports, 13 were final and two were mid-term evaluations.
At the time of review four projects did not have project level evaluations, namely
Pupils Understanding Problems in their Locality (Cynon Valley Crime Prevention),
Young BME People Aiming High (All Wales Ethnic Minority Association), Minority
Ethnic Learning and Achievement Project (WAG DCELLS Lifelong Learning and
Skills) and Think Differently (Fairbridge De Cymru). Two of these projects were
relatively small in terms of their funding levels (under £500,000 EU funding) whilst
the other two were awarded up to £4m EU funding each. Overall the evaluation
coverage as a proportion of committed EU funding was 95 per cent. In one case,
WEFO terminated its project agreement with the project sponsor and provision was
transferred to other joint sponsors of the project. It is notable that two of these
projects which are without project-level evaluations were specifically focused upon
BME groups.
16 of the project-level evaluations were prepared in English and one in Welsh. All
were prepared by external independent evaluators – in all nine different teams of
evaluators were commissioned to undertake these studies, with one team of
evaluators involved in four studies in all.
Methodological Detail
Eight of the evaluation reports were considered to have strong methodological
statements. Those categorised as strong typically contained:
A detailed account of the method deployed including details for proposed
sampling strategy (and in very few cases comparative data on samples
43 Reach the Heights Routes to the Summit and Reach the Heights First Footholds
ESF Convergence Priority 1 Priority Review Paper
41
achieved), method of recruiting contributors, number of contributors involved
and timescales for conducting fieldwork
A logic model which was used to underpin the investigation
A detailed Evaluation Framework (usually presented in an annex) presenting
the objectives of the study and method of collecting data to assess the extent
to which they were achieved
Various research instruments including topic guides for interviewing staff or
participants or survey tools (usually presented in an annex)
A review of the strengths and weaknesses of the method adopted – one
evaluator adopted this approach across three project level evaluations and
provides a detailed insight into the strengths, challenges and limitations of
various approaches deployed.
Six evaluation reports were categorised as moderate. These typically contained
methodological statements but had some gaps (e.g. in relation to beneficiary
sampling and survey response rates).
Mixed Methods
All but one (14 of 15) of the project-level evaluations were considered to have
deployed a mixed approach (i.e. in terms of primary fieldwork and consideration of
project level data) although there was generally greater use of qualitative data from
participants (rather than quantitative data) across the board, possibly reflecting the
nature of the target group and the difficulty of accessing direct contact data for young
people in order to conduct say, a phone or web survey. In these cases, there
appears to have been a reliance upon project providers to facilitate access for
evaluators to meet with participants. Difficulties or restrictions in sharing participant
contact data has meant that many of the evaluations did not have much control over
the sample of participants interviewed. As a result, interviewed participant samples
have often been purposive and an element of project selection bias is evident.
At least two evaluations specifically noted that in-depth interviews with young people
had worked better than conducting workshops or small group based discussions with
ESF Convergence Priority 1 Priority Review Paper
42
participants on the basis that it had been easier to collect feedback from all
contributors and present this in an aggregated manner within the report.
Two common limitations were identified with a qualitative approach to gathering
feedback from participants. In some cases, evaluators noted that it had not been
possible to conduct interviews with participants from across all provisions or
providers and some groups – particularly where the sample numbers had been low.
In other cases, it was acknowledged that some groups or areas were not
represented within the sample e.g. younger participants aged 11-13.
In a positive manner, some studies attempted to present data collected via
qualitative interviews with participants in a quantitative manner e.g. in the case of
Engage, a large number of qualitative interviews (a total of 136) was used in this
manner with data presented in a numerical manner via graphs and tables.
At least four evaluations had involved some element of fieldwork with parents –
although in most of these cases the sample of parents contributing to the research
was very low (32 in the case of Potensial, 10 in the case of Building the Future
Together, 10 in the case of Technocamps and five in the case of Llwyddo’n Lleol).
A slightly greater number of evaluations had undertaken some element of fieldwork
with employers (which involved a mix of qualitative and quantitative research) –
although sample numbers varied from the very low (e.g. in the case of Llwyddo’n
Lleol) to reasonable and robust samples (e.g. in the case of Pathways to
Apprenticeships at 67 and Stem Cymru at 101).
Surprisingly few evaluations reported that they had undertaken observational work –
possible the only project-level evaluation to have done so in an intensive manner
had been in the case of the Technocamps project where some 100 events were
observed.
At least three evaluations reported that they had undertaken longitudinal work with
supported participants (Llwyddo’n Lleol, the Regional SEN Transition to Employment
Initiative and the Gypsy Traveller Learning and Future Employment Project). In the
latter, the approach involved a baseline survey and final survey to provide good data
ESF Convergence Priority 1 Priority Review Paper
43
on difference achieved. Timescales were considered to restrict this approach in
many cases e.g. in the case of Technocamps it was reported that a longer term
study would be needed to assess the impact of intervention upon supported young
people.
Project-level evaluations made significant use of project level monitoring data, with a
small number of evaluations relying upon monitoring data as the main quantitative
data source. It was the case that gaps in data or inconsistent data limited the use of
quantitative databases in some cases, particularly around achievement of soft
outcomes (e.g. Building the Future Together).
The extent of triangulation across project-level evaluations varied – in those cases
where primary fieldwork had been undertaken with a cross-section of stakeholders
(including participants, parents, employers, project delivery teams and independent
stakeholders) a general effort was made to reflect upon different viewpoints.
However, it was often the case that specific case study material was considered from
the viewpoint of a single contributor (more often than not being the participant’s view)
rather than from a more rounded view of a parent, employer or provider.
Beneficiary Fieldwork
In all, 13 of the 15 project-level evaluations involved beneficiary fieldwork. Given the
reliance upon qualitative fieldwork it is perhaps not surprising that only seven of
these set out the findings of a quantitative survey of supported participants. Sample
sizes varied both in absolute terms and as a proportion of all participants supported
by the project – and it is noteworthy that some evaluations completed a greater
number of qualitative interviews with participants than did some of the surveys. In
some cases it is difficult to report upon response rates (given the lack of data
available in the report). Project-level evaluations reporting high numbers of surveyed
participants were a survey of 3,197 participants (27% of all project participants) in
the case of Llwyddo’n Lleol and a survey of over 3,000 participants in the case of
STEM Cymru (response rate unknown).
One evaluation (Raising Skills and Aspirations of Young BME People II) had
intended to deploy a web survey of participants but the approach was revised in light
ESF Convergence Priority 1 Priority Review Paper
44
of feedback from participants and project staff that a web survey was considered
inappropriate, impersonal and disempowering to participants who had literacy issues
or where were not first language English speakers. The evaluation therefore relied
upon face to face interviews, focus groups and filmed case studies of participants.
Counterfactual Impact Evaluation
Only four project-level evaluation reports attempted to undertake any counterfactual
impact evaluation and overall there is a lack of robust counterfactual evidence
available for this Priority. Each of these project-level evaluations deployed a different
approach:
In the case of Pathways to Apprenticeships the counterfactual impact
evaluation involved undertaking a Propensity Score Matching using the LLWR
database as a source of evidence. The evaluators matched a group of treated
learners with a comparable group of non-treated learners to calculate lifetime
increases in earning as a result of intervention. The approach appears to have
been robust and the report presents financial data on the lifetime benefits of
the qualifications obtained as a result of intervention, as matched against a
non-treated group
In the case of the KS3 Intervention project the evaluators considered the
educational performance and achievements at Key Stage 3 (i.e. proportions
achieving expected levels in core subject indicators) and how this changed
over time between participating and non-participating local authorities. As it
was not possible to identify individual supported pupils (as they were not
‘tagged’ in any way on local authority databases) the evaluators relied upon
proxy data to estimate the impact of the project. These proxy indicators
included the performance of pupils eligible for free-schools meals, rates of
absence, and the proportions of pupils achieving higher levels of performance
in science and mathematics. The limitations of this approach was noted
including the fact that other local authorities had similar ESF funded projects
which was considered to have limited the scope of the counter-factual
analysis
The evaluation of the Potensial project considered the counterfactual position
based upon ‘soft’ evidence gathered from supported participants and delivery
ESF Convergence Priority 1 Priority Review Paper
45
staff via qualitative interviews which probed around what would have
happened in the absence of support. Consideration is given to the feedback
from delivery staff and 89 supported participants (although it is not known
what proportion of all supported participants this sample represents).
In the same manner the evaluation of Engage attempts to estimate the
counterfactual position based upon feedback from supported participants but
overall this is quite a light-touch section which reported how the lives of
participants would be different had it not been for the project
It would appear that several of the project-level evaluations, particularly those which
involved face to face interviews with large numbers of participants missed an
opportunity to gather ‘soft’ counterfactual evidence from young people. For instance,
in the case of the Gypsy Traveller Learning and Future Employment Project it could
have been relatively easy to have built in appropriate questions such as ‘what would
have happened without support, or whether changes would have come about
anyway’ during such questioning.
ESF Convergence Priority 1 Priority Review Paper
46
10 Conclusions
The evidence base for Priority 1 is comprehensive in that 16 of 20 Convergence
funded projects, equating to 95 per cent of committed EU funding, have evaluation
findings available. However, the lack of CIE data available within project level
evaluations as well as the fact that no data is available via the ESF Leavers Survey
makes it impossible to calculate the net impacts of this Priority in a reliable manner.
The evidence suggests that the original intervention logic for Priority 1 was rational
and that it had been appropriate to introduce a specific priority focused on the
delivery of preventative interventions for 11 to 19 year olds. Priority 1 has been
delivered within the context of a fairly stable policy environment – in contrast funded
projects have had to respond to some significant external changes. The economic
downturn reduced employment opportunities for participants on the one hand but on
the other meant that greater number of young people were being drawn to education
and training opportunities. Moreover, the impact of public sector funding cutbacks
was also a significant external factor although in reality this was considered to have
had a greater bearing upon the sustainability of provision post EU funding.
A total of 20 projects were approved across Convergence Priority 1. These projects
demonstrated a good fit with the Operational Programme and the Strategic
Framework in that the proposed activities were well covered and were focused on
adding value and complementing mainstream educational provision. Projects were
also wide-ranging in terms of the target audiences which they intended to support.
The key delivery issue which impacted negatively on this Priority was the relatively
slow start in approving projects, not least because of the move towards adopting
fewer, more strategic projects but also because of longer than anticipated
procurement procedures and a lack of experience amongst some project sponsors of
ESF funding. Whilst the Priority struggled to meet its profiled output indicators during
the initial few years of delivery, performance improved over the second half of the
Programme.
ESF Convergence Priority 1 Priority Review Paper
47
In the event, the Priority participant output was substantially exceeded (by nearly
four-fold) and on a collective basis funded projects exceeded all but one of the
outputs indicators. Performance against Priority results indicators was also very
strong – particularly in terms of the numbers of participants gaining other positive
outcomes and qualifications. However, the proportion of all participants assisted who
gained a qualification or other positive outcome represented a much lower proportion
of all participants supported than envisaged. Furthermore, the Priority fell short of
achieving its targets for participants entering further learning.
One common theme to be raised across project level evaluations related to the fact
that many of the results indicators were unlikely to be realised in the short term given
the age profile of participants supported via this Priority. It was also commonly
argued that projects were better placed to have a direct impact upon pupil
attendance and retention levels within education – significant outputs which were not
necessarily being reported to or captured by WEFO. Aligned to this, the lack of long-
term data available for supported participants to evidence any future progression (be
that in terms of entry to further learning or qualifications obtained) limits the evidence
base available on the overall impact of Priority 1 funded interventions.
The lack of robust evaluation evidence on levels of deadweight and displacement is
notable for Priority 1. The fairly limited evidence suggests that levels of deadweight
have been mixed (some projects reported high levels whilst others reported lower
levels of deadweight). In contrast, levels of displacement appear to have been fairly
low as funded interventions have been integrated and added value to existing
mainstream services.
Overall, Priority 1 achieved better value for money (as measured by the level of
investment per participants engaged and direct results achieved) than planned. At a
project level the fairly limited evidence suggested a mixed picture in terms of the
level of investment per participant engaged and direct results achieved.
In terms of cross-cutting themes, project level evaluations found that Equal
Opportunities was a central consideration in most projects’ objectives in that they
were often specifically targeted at disadvantaged groups of young people.. In terms
ESF Convergence Priority 1 Priority Review Paper
48
of environmental sustainability, project level evaluations suggested that whilst a
smaller number of funded projects deployed innovate approaches, more could have
been achieved by some projects in this area of work.
Assessing the net impact of Priority 1 interventions is challenging given that so few
project level evaluations could realistically undertake any meaningful counterfactual
impact evaluation in light of the target audience supported. The lack of any
counterfactual evidence limited evaluators’ ability to test what would have happened
in the absence of specific interventions. Where any CIE assessment has been
attempted, the findings point to a positive impact although the transferability of the
data itself in calculating the more macro level net impact of Priority 1 is limited.
There is evidence of a number of unintended consequences having occurred as a
result of provision funded under Priority 1. Of critical importance has been the impact
of interventions upon levels of educational attendance, reduced unauthorised
absenteeism, reduced permanent exclusions and overall educational retention levels
amongst participants. There is also significant evidence via project-level evaluations
of participants having achieved extensive soft or intermediate outcomes as a result
of participation – these have included increased confidence levels and positive
behavioural changes. Accepting of the fact that it would have been extremely
challenging to have done so, project level evaluations have revealed very little
evidence to substantiate the indirect positive effects of funded projects in terms of
public expenditure savings, increased lifetime earnings for participants and reduced
negative social behaviour. Where available, the data (which should be considered
with caution), however, does suggest a positive impact.
In terms of lessons for the future, these might include:
That effective provisions targeted at young people generally adopt an
individualised, holistic approach which offer both emotional and educational
provisions via advisors who can develop positive relationships with young
people
That work experience opportunities are important and add value to provisions
targeted at young people
ESF Convergence Priority 1 Priority Review Paper
49
The varied performance of multiple partners within a single project can impact
the overall performance of a large scale, strategic project
The need to adopt indicators which reflect the direct impact of interventions
aimed at young people within education (e.g. around absenteeism,
attendance and retention rates)
The need to consider improving the capturing and reporting of participant
outputs and results, particularly to reduce any under-reporting or inaccurate
reporting of outputs as well as to enable the identification and analysis of any
longer-term, post-intervention changes and outcomes
The need to continue to explore CIE approaches to measure the difference
made to participants over the medium to long term. In this respect it would be
worth further investigating the possibility of matching WEFO’s central
participant database with data available within the LLWR database.