Evaluation of the ETF functions: a meta-analysis of recent achievements and impacts
June 2018
Evaluation of ETF functions – Interim Report
2
CONTENTS
Executive Summary .......................................................................................................................................... 3 1. Introduction ....................................................................................................................................... 5 1.1. The ETF’s mandate in a changing context .......................................................................................... 5 1.2. Objectives of the assignment ............................................................................................................. 5 2. Has the ETF responded to the needs of its stakeholders? ................................................................... 7 2.1. The needs of partner countries .......................................................................................................... 7 2.2. Supporting the implementation of EU policies ................................................................................. 10 3. Has the ETF achieved its objectives? ................................................................................................ 11 3.1. Achieving specific objectives in the ETF priority areas ...................................................................... 13 3.2. Implementation of the ETF’s core functions ..................................................................................... 15 4. Have the ETF’s actions led to sustained change in partner countries? .............................................. 18 4.1. Attributing impacts to the ETF’s interventions ................................................................................. 18 4.2. The ETF’s impacts on the policy development in partner countries .................................................. 19 4.3. The influence of contextual factors and the ETF’s impact................................................................. 20 5. To what extent are the ETF’s actions unique compared to those of other actors?............................. 23 5.1. Thematic and geographic expertise ................................................................................................. 23 5.2. Capacity to involve stakeholders and apply participatory approaches .............................................. 24 5.3. Long-term and continuous involvement in partner countries ........................................................... 25 5.4. The European dimension of the ETF’s work ..................................................................................... 25 6. What could the ETF do differently to fully realise its mandate and adapt to its evolving context? .... 26 6.1. Responding to the needs of stakeholders ........................................................................................ 26 6.2. Implementing the objectives and achieving sustained change ......................................................... 27 6.3. Adding value to the actions of other actors ...................................................................................... 28 References ...................................................................................................................................................... 29
LIST OF FIGURES
Figure 1. Relevance of the ETF’s work to the needs of its stakeholders .............................................................. 7 Figure 2. The ETF’s objectives and Strategic Projects (2017-2020)................................................................... 12 Figure 3. Extent of ETF contribution to developments in partner countries: specific objectives compared ....... 13 Figure 4. Stakeholder responses to the question "has the Torino Process been effective in achieving its objectives?" .................................................................................................................................................... 14 Figure 5. The ETF functions ............................................................................................................................. 15 Figure 6. Results and impacts of the ETF (2017-2020) ...................................................................................... 19 Figure 7. Contextual factors and the impact of ETF interventions in partner countries ..................................... 21 Figure 8. Added value of the ETF compared to other relevant actors ............................................................... 23
Evaluation of ETF functions – Interim Report
3
Executive Summary
EU and international policy objectives and commitments, as well as the evolving situation in the ETF
partner countries require the agency to constantly review its actions within its current mandate in order
to maximise its contribution to achieving these policy objectives. This assignment aims to synthesise the
lessons learned from the previous evaluations of the ETF published from 2014 to 2017 to come up with
a single set of conclusions and recommendations for implementing its mandate in the years to come. It
draws on the evidence on the ETF’s performance from internal and external evaluations of the agency.
Based on the results of the meta-analysis, the following key messages can be highlighted:
1. The ETF’s actions of the ETF were relevant to the needs of the partner countries across a
broad range of activities despite dynamic and sometimes unstable political environments,
and the limited human and financial resources it could use in each partner country. The ETF
has remained relevant to the needs of partner countries by collecting information and
conducting in-depth analyses in several key areas of human capital development. The ETF’s
support in the area of VET governance was particularly relevant in partner countries at the
initial stages of policy design aimed at modernising their VET systems (e.g. Albania, Georgia
and Ukraine) and in partner countries undergoing broad institutional reforms such as VET
decentralisation (e.g. Morocco and Tunisia).
2. The ETF’s objectives and actions are highly complementary with EU policies and activities
in the partner countries, as well as with those of other international actors in the field. The
ETF collaborates with international organisations and institutions such as the EBRD, the
OECD, the ILO, and the World Bank, as well as with the development agencies from EU
Member States in programming of assistance, providing country intelligence, sharing
methodological tools, reports and networks, and engaging with stakeholders.
3. The ETF has been particularly useful in supporting the Commission and EU Delegations
during the different phases of the EU external assistance project cycle, including in the
formulation, implementation, monitoring and evaluation of cooperation programmes. The
ETF’s methodological support and knowledge sharing has helped to enhance the quality of
the EU’s interventions. The ETF has the capacity to fit the EU’s instruments into the context
of partner countries, and its long-term presence in partner countries ensures continuity of
national VET reforms initiated under the auspices of Commission interventions.
4. The Torino Process has actively involved an increasing number of the partner countries and
has opened them to a more inclusive VET policy dialogue with a wider group of
stakeholders. All ETF partner countries actively participating in the Torino Process involve
not only the representatives from the ministries of education, but also the ministries of
labour, employer organisations/chambers of commerce and VET providers. Several
countries have made active use of the Torino Process recommendations when considering
which issues to address or how to address specific challenges. Furthermore, EU Delegations
and international donor organisations also make use of the Torino Process reports and
provide very positive feedback on the usefulness of its national, regional and international
events.
5. The ETF has been successful in supporting policy dialogue between the EU and partner
countries. For example, the ETF supported ministerial collaboration at regional level in
Eastern Europe through the Eastern Partnership, and in Central Asia under the Astana
Declaration by providing support to planning and implementation of the Central Asia
Education Platform (CAEP). In the South, it has done so through the Union for the
Evaluation of ETF functions – Interim Report
4
Mediterranean, and in South Eastern Europe and Turkey, through involvement in
monitoring and reporting on the Riga deliverables under the Copenhagen Process.
6. The ETF’s interventions facilitated VET policy developments and had a visible impact on
the reform process in several partner countries. Without the ETF, these developments would
have taken place more slowly, differently or – possibly – not at all. The ETF brought change
to national reform processes in several countries, such as when it was used for international
donor coordination and as an input to EU external assistance (e.g. in Lebanon, Egypt,
Ukraine and Uzbekistan).
7. The ETF’s support to governance has made a considerable contribution to building capacity
of teams of VET reformers in partner countries. Through the common experience of
engaging with the ETF and attending its events, the partner country delegations consisting
of representatives from diverse stakeholder organisations have experienced a strong team
building effect that has increasingly enabled collaboration across institutional boundaries
on VET reform initiatives. In several countries the ETF supported the design and
implementation of innovative coordination mechanisms based on partnerships between key
actors in the VET system (e.g. Morocco, Tunisia and Serbia).
8. The ETF’s contribution has been the strongest in the areas of VET governance and policy-
making, VET provision and quality assurance, and qualifications and qualification systems.
In these areas, the ETF is believed to be the main driver, or to have provided important
support to policy change in the partner countries. It helped partner countries to develop
overarching laws for education, set the foundations for VET quality assurance systems, to
support them in setting up national qualifications systems and prepare for referencing them
with the European Qualifications Framework. The ETF’s governance interventions made a
considerable contribution to building teams of VET reformers in partner countries and has
been particularly valued in relation to stronger coordination and collaboration between
relevant stakeholders and the increased quantity and quality of participation by social
partners in policy-making processes.
9. Previous evaluations emphasised the ETF's unique combination of competences and
approaches which set it apart from other organisations providing support to the
modernisation of human capital development policies. While other international
development organisations conducted similar individual activities, the ETF combined long-
term and continuous involvement in partner countries, thematic and geographical expertise,
a participatory approach empowering a broad range of stakeholders to participate in policy
dialogue and added a European dimension to its reform work. Rather than offering
piecemeal solutions to reforming separate institutions and policies, the ETF helped the
partner countries see the links between the different parts and elements of their education
and labour market systems and to support their development in a consistent and mutually
reinforcing way.
10. The ETF extended its reach by successfully implementing large EU assistance projects such
as the GEMM and FRAME projects. The achievements of the GEMM project included new
methodological tools for mapping and analysing VET governance applied in Southern and
Eastern Mediterranean countries (eight pilot projects within local partnerships based on
local skills analysis, career guidance and counselling in Lebanon, apprenticeship and tracer
study in Jordan and a tracer study of graduates in the labour market in Palestine). The
FRAME project in the Western Balkans developed a ‘roadmap for skills 2020’, reviewed
institutional arrangements, improved regional cooperation and provided support to
ongoing VET reforms in the region.
Evaluation of ETF functions – Interim Report
5
1. Introduction
1.1. The ETF’s mandate in a changing context
Recent trends in human capital development have generated a global consensus among policy-makers,
national institutions, and partner organisations about the importance of combining good education with
quality training and strongly linking skills development with labour market reform1. The UN
Sustainable Development Goals (SDGs) related to “quality education” and “decent work and economic
growth” fall directly within the core expertise of the ETF and represent priorities for its work2. In
addition, vocational education and training (VET) is seen as a transversal policy area contributing to
the overall objectives of the UN 2030 Agenda for Sustainable Development and multiple other goals3.
The ETF’s Founding Regulation gives a mandate to the agency to contribute, in the context of EU
external relations policies, to the improvement of human capital development (HCD) in its partner
countries4. The ETF draws on the EU’s internal human capital development policy approaches, tools
and instruments to support the growth and socio-economic development of its partner countries
according to the priorities set by EU external relations policies.5 The agency is also to follow the
Commission President’s political guidelines6 and other key EU policies such as the Europe 2020 Agenda,
the New Skills Agenda for Europe, and the European Pillar of Social Rights7. The ETF draws on EU
policies by supporting evidence-based and participatory decision making, providing evidence and
advice to the EU institutions and partner countries on human capital investments and institutions
underpinning human capital development.
1.2. Objectives of the assignment
The international and EU policy objectives and commitments as well as the evolving situation in its
partner countries require the ETF to constantly review its actions within its current mandate to
maximise its contribution to achieving those policy objectives. This assignment aims to synthesise the
lessons learned from the previous evaluations of the ETF published since 2014 to come up with a single
1 European Commission (2013), TVET and Skills Development in EU Development Cooperation. Final Report to
DG DEVCO – EuropeAid.
2 Goal 4: Ensure inclusive and quality education for all and promote lifelong learning; and Goal 8: Promote inclusive
and sustainable economic growth, employment and decent work for all. See:
http://www.undp.org/content/undp/en/home/sustainable-development-goals.html.
3 ETF (2016), Single Programming Document 2017-20. Annual Work Programme 2017 and first indications for Work
Programme 2018. GB/16DEC/011.
4 Regulation (EC) No 1339/2008 of the European Parliament and of the Council of 16 December 2008 establishing a
European Training Foundation (recast). OJ L 354/82, 31.12.2008.
5 ETF (2016), Single Programming Document 2017-20. Annual Work Programme 2017 and first indications for Work
Programme 2018. GB/16DEC/011.
6 Available at: http://www.eesc.europa.eu/resources/docs/jean-claude-juncker---political-guidelines.pdf.
7 As well as: the Stronger Global Actor launched by President Juncker in 2014; the Global Approach to Migration
and Mobility (GAMM), and the Copenhagen Process, namely five Medium Term Deliverables (MTDs) agreed
under the Riga 2015 Conclusions between all EU Member States and Candidate Countries.
Evaluation of ETF functions – Interim Report
6
set of conclusions and recommendations for implementing its mandate in the years to come, by focusing
on the following questions:
What have been the key effects of the ETF work in contributing to human capital development
in partner countries in the context of EU external relations policies?
Which methods of implementation and tools should the ETF utilise to further improve its
impact within the objective, scope and functions in its regulation?
How could the ETF optimise its added value in particular to the achievement of EU policy
priorities both for external assistance as well as for the external dimension of internal policies?
This evaluation draws on the evidence on the ETF’s performance from the agency-wide external
evaluations8, the internal project-specific evaluations9, and the internal functional analyses of
performance10.
The subsequent chapters of this report show that the ETF is a unique actor in international human
capital development. Previous evaluations have established that the ETF has been relevant to its
stakeholders, the agency’s actions have been coherent and highly complementary with the EU internal
and external policies and with those of other international actors. The ETF has achieved its objectives
and had positive impacts, has demonstrated its specific added value, and has worked towards greater
sustainability of its results. Our report also presents a selection of conclusions and recommendations
stemming from previous evaluations of the agency’s activities and outlines the scenarios that the ETF
should consider when preparing to respond to the changing needs of its key stakeholders in the EU and
the partner countries in the years to come.
8 Evaluation of the Mid Term Perspective 2010-2013 commissioned by the ETF, June 2015; European Commission
(2016), External Evaluation of the European Training Foundation. Final Report by the EFECTIV Consortium;
European Commission (2018a, upcoming), External evaluation of the four agencies under the remit of DG EMPL –
Cedefop, Eurofound, ETF and OSHA, Final report.
9 ICON (2014a), Evaluation of qualifications activities in ETF partner countries. Final Report; ICON (2014b),
Evaluation of Knowledge Management Innovation in ETF Operations. Final Report; ICON (2015), Evaluation of
Entrepreneurship and Enterprise Skills Development in ETF partner countries. Final Report; ICON (2016),
Evaluation of ETF activities in the Skills and Employment field in Partner Countries Organisation and
Methodology. Final Report; ICON (2017), Evaluation of ETF Activities in improving Continuing Professional
Development of Vocational Teachers and Trainers. Final Report; PPMI and Europe Ltd (2017), Evaluation of ETF
activities in the field of VET governance. Final Report; PPMI (2018), Evaluation of the Torino Process. Interim
Report.
10 Including the 2014 Functional analysis of the ETF in view of performance improvement; the 2015 ETF analysis,
monitoring and reporting on ETF efficiency gains; and the 2016 ETF review of process management framework.
Evaluation of ETF functions – Interim Report
7
2. Has the ETF responded to the needs of its stakeholders?
Key findings
- The ETF has been relevant to the needs of its partner countries across a broad range of
activities.
- The key factors ensuring the relevance of ETF’s actions have been its long-term presence
in and flexibility in addressing the needs of decision makers in partner countries.
- Through constant networking with a wide range of partner country stakeholders and
involving those stakeholders in its activities, the ETF has ensured a high level of trust,
which in return has helped it to constantly access up-to-date information on changing
partner country contexts.
- The agency has coped well and effectively responded to changes in EU policy. A thorough
understanding and analysis of the regional and country contexts made the ETF
particularly well-placed to support EU programming assistance towards partner
countries
- The ETF is coherent with EU policies and its activities in partner countries are highly
complementary with those of other international actors
FIGURE 1. RELEVANCE OF THE ETF’S WORK TO THE NEEDS OF ITS STAKEHOLDERS
2.1. The needs of partner countries
Recent evaluations found that the actions of the agency were relevant to the needs of its partner
countries across a broad range of activities despite dynamic and sometimes unstable political
environments, and despite very limited human and financial resources it could use in each partner
Responding to partner countries needs
- Flexibility and responsiveness
- Thematic expertise and subject knowledge
- Capacity building and policy support
- Stakeholder participation
Responding to EU policy needs
- Responses to changes in EU internal and external policy
- Enhancing the quality of EU interventions through methodological support and knowledge sharing
- Complementarity with the objectives of EU policies and institutions
Relevant ETF interventions
Evaluation of ETF functions – Interim Report
8
country. The ETF has remained relevant to the needs of partner countries by collecting information and
conducting in-depth analyses in many of the key areas of human capital development, which informed
the choice of individualised support provided to its partner countries.
The agency has been deemed highly flexible and responsive to the changing needs of its stakeholders.
The ETF has maintained its long-term presence in and continuous support to partner countries, which
has helped it achieve a high level of trust with policy makers and other key stakeholders in the partner
countries as well as with international and bilateral development actors. The relevance of the ETF’s
interventions has also been ensured through constant networking and the involvement of a broad range
of relevant partner country stakeholders in its activities, which is a key characteristic and success factor
in VET reform implementation11. Finally, ETF’s interventions have not been based on simply replicating
successful interventions found elsewhere but have encouraged policy learning and reflection in partner
countries, which has been valued by partner country stakeholders.
Previous evaluations found that partner countries stakeholders emphasised the relevance of the ETF
in respect of the subject knowledge that it has acquired, being “deeply contextualised in the partner
countries which provides essential input for the preparation of new programmes and projects”12. The
evaluation also emphasised that the agency’s interventions responded to the specific employment,
employability and skills development needs and conditions in each visited partner country13.
Stakeholders also attached a strong relevance to the capacity building and policy support functions
of the agency’s interventions in NQFs and qualifications reform14. Moreover, they considered ETF’s
networking activities and analytical work to be relevant in supporting partner countries in NQF and
qualifications development and linking up with EU policy developments such as the EQF. The
evaluation of the ETF’s activities in entrepreneurial learning and entrepreneurial skills development
found that the Small Business Act (SBA) assessment method was considered highly relevant15.
The ETF’s knowledge sharing interventions and conferences also contributed to build partner
countries’ capacity and networks, as well as helping to keep relevant human capital development
issues on the policy agenda16. The ETF’s actions in VET governance were particularly relevant in the
partner countries at the initial stages of designing new overarching laws aimed at modernizing their
VET system (e.g. Albania, Georgia, Ukraine) or the partner countries undergoing broad institutional
reforms such as VET decentralisation (e.g. Morocco, Tunisia, Ukraine).17 The ETF was able to offer the
11 Caves, K. and Baumann, S. (2018), Getting there from here: A literature review of VET reform implementation.
KOF, Swiss Economic Institute, KOF Working Papers, No. 441.
12 ICON (2016), Evaluation of ETF activities in the Skills and Employment field in Partner Countries Organisation
and Methodology. Final Report submitted to the European Training Foundation, p. 19.
13 Ibid.
14 ICON (2014a), Evaluation of qualifications activities in ETF partner countries. Final Report submitted to the
European Training Foundation.
15 Ibid.
16 ICON (2015), Evaluation of Entrepreneurship and Enterprise Skills Development in ETF partner countries. Final
Report submitted to the European Training Foundation.
17 PPMI and Europe Ltd (2017), Evaluation of ETF activities in the field of VET governance. Final Report.
Evaluation of ETF functions – Interim Report
9
partner countries a holistic approach to developing and linking different VET institutions and policy
domains, which could not be achieved through interventions with a narrow thematic focus.
Social partner organisations from the partner countries positively acknowledged the relevance of the
ETF’s work from their respective perspectives and interests18. The evaluation of the ETF’s activities in
the skills and employment field found that “the relevance of the ETF’s work is mainly due to its unique
knowledge of the context and the specific country’s needs, as well as its operational approach of serving
to strengthen the role of the social partners in the formulation, implementation, monitoring and
evaluation of skills and employment policies, and the VET system in a wider sense”19. Moreover,
specific actions such as the GEMM20 and FRAME21 projects supported cooperation and clarification of
roles of, key stakeholders in partner countries (including government bodies and social partners).
Since 2010, the ETF has supported partner countries in undertaking system wide policy analysis and
monitoring the progress of their VET policies through the Torino Process. This relevance of the Torino
Process is particularly well illustrated by the fact that through the participation in this process many
partner countries have reviewed their human capital development policies in a systematic way for the
first time. The 2016 external evaluation noted that the Torino process “provided a new thoroughness”
to the way in which the ETF identifies the needs of its partner countries22. At least 79% of survey
respondents from the ETF’s stakeholders indicated that the Torino Process was effective and efficient
either to some extent or to a high extent.23 The Torino Process effectively responded to the partner
countries’ need for improving their VET policy analysis capacity and opened the discussions on VET
issues and policies to a wide range of non-governmental stakeholders. However, over time the Torino
Process has grown less relevant as a policy monitoring tool to a small number of partner countries that
have reached the advanced stages of their national VET policy development, implementation and
monitoring.
One area in which the ETF could strengthen the relevance of its interventions concerns the agency’s
country strategy papers. While these papers helpfully summarised the activities of the ETF in a given
partner country, they lacked a strategic and forward planning dimension. For example, they did not
include a description of the expected change in a country or the role of the ETF in supporting that
change. The evaluation of the ETF activities in VET governance pointed out that the country strategy
papers lacked a concrete action plan with measures and activities supporting the desired change24.
Neither did they foresee specific ways for internal cooperation across the ETF’s strategic projects during
the implementation of a given country strategy25. No analysis of country risk factors and plan to manage
the risks were included either. While there is no direct evidence of this affecting the relevance and
18 ICON (2016), Evaluation of ETF activities in the Skills and Employment field in Partner Countries Organisation
and Methodology. Final Report submitted to the European Training Foundation.
19 Ibid., p. 18.
20 Governance for Employability in the Mediterranean region. See:
http://www.etf.europa.eu/web.nsf/pages/GEMM.
21 Frame Skills for the Future. See: http://www.etf.europa.eu/web.nsf/pages/Frame_project.
22 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium.
23 Ibid.
24 PPMI and Europe Ltd (2017), Evaluation of ETF activities in the field of VET governance. Final Report.
Evaluation of ETF functions – Interim Report
10
coherence of ETF actions, the integrated planning of the ETF’s interventions in specific countries could
be improved and have positive effects on the internal work dynamics of the agency as well as the
collaboration with partner country stakeholders.
2.2. Supporting the implementation of EU policies
Evidence shows that the agency has coped well with and effectively responded to changes in EU
policy during the evaluation period. The Torino Process and partner country intelligence in other ETF
strategic projects have allowed the agency to build up a thorough understanding of regional and
country contexts. Therefore, the ETF has been particularly well-placed to support with evidence the
European Commission’s policy programming towards human capital development in partner
countries. The agency’s country-specific expertise has been identified as one of its key strengths, valued
by both the Commission and international organisations26.
The ETF’s methodological support and knowledge sharing has helped to enhance the quality of the
EU’s interventions, such as in the identification and formulation of cooperation programmes and has
supported the work of the EU Delegations27. A recent report from DG DEVCO underlined that the ETF
has “the capacity to fit the EU’s instruments into the context of partner countries”, and that its long-
term presence in partner countries “ensures continuity of national VET reforms initiated under the
auspices of Commission interventions”28. For example, the ETF’s work was considered particularly
relevant to the Mobility Partnerships signed between the Commission and ETF partner countries in the
context of the Global Approach to Migration and Mobility (GAMM)29. This included proposing
activities on the management of migration, including the recognition and portability of qualifications
and the analysis of skill needs for working in domestic and foreign labour markets30.
The ETF’s actions were highly relevant to the objectives of the EU’s external policy and supported the
activities of DG DEVCO and DG NEAR31. The ETF also successfully undertook two projects on behalf
of the Commission services: GEMM (on behalf of DG DEVCO) and FRAME (on behalf of DG NEAR).
In addition, the ETF helped DG DEVCO to elaborate its strategy on VET, placing emphasis on the
involvement of the social partners and other non-governmental actors in the development of national
VET systems. The ETF’s strategic project on VET governance placing VET as a cross-cutting issue at
strategic and operational levels resulted in “particularly fruitful cooperation” between the ETF and the
EU Delegations (such as in Morocco and Tunisia)32. However, the 2016 evaluation of the ETF also found
26 European Commission (2018a, upcoming), External evaluation of the four agencies under the remit of DG EMPL
– Cedefop, Eurofound, ETF and OSHA, Final report.
27 ICON (2016), Evaluation of ETF activities in the Skills and Employment field in Partner Countries Organisation
and Methodology. Final Report submitted to the European Training Foundation.
28 European Commission (2013), TVET and Skills Development in EU Development Cooperation. Final Report
prepared by AETS for DG DEVCO – EuropeAid.
29 Including the Republic of Moldova, Georgia, Armenia, Morocco, Azerbaijan, Tunisia, Jordan and Belarus. See:
https://ec.europa.eu/home-affairs/what-we-do/policies/international-affairs/global-approach-to-migration.
30 ICON (2016), Evaluation of ETF activities in the Skills and Employment field in Partner Countries Organisation
and Methodology. Final Report submitted to the European Training Foundation.
31 PPMI and Europe Ltd (2017), Evaluation of ETF activities in the field of VET governance. Final Report.
32 Ibid.
Evaluation of ETF functions – Interim Report
11
that effective working relationships with EU Delegations were too dependent on informal processes
and lacked a more structured framework for cooperation.
Recent evaluations have also found that the ETF is coherent with EU policies and that its activities in
partner countries are highly complementary with those of other international actors in the field of
human capital development, with which the agency regularly cooperates33. By supporting overall
monitoring of human capital development activities in its partner countries, the ETF has also helped
partner countries “to ensure that the different pieces of the complex ‘jigsaw puzzle’ of development
assistance fits together well”34. Moreover, the ETF has been found to be generally coherent and
complementary to the work of other EU agencies, such as Cedefop and Eurofound.
3. Has the ETF achieved its objectives?
Key findings
- The ETF strongly contributed to policy learning and networking between stakeholders in
partner countries, which informed and enabled: (a) their steps towards improving
governance and modernisation of human capital development policies and (b)
programming of the EU assistance to support those steps.
- The ETF has achieved its specific objectives and priorities. The ETF’s contribution has
been especially strong in its first three thematic objectives: governance, systems and
policy-making; VET provision and quality assurance; and qualifications and
qualifications systems.
- The ETF’s work has been particularly useful in supporting the Commission and EU
Delegations during the different phases of the EU external assistance project cycle, in
strengthening the involvement, coordination and collaboration of diverse partner country
stakeholders on human capital development, providing policy advice and sharing
examples of good practice, improving data collection and analysis.
The 2016 external evaluation led the ETF35 to renew its intervention logic with the aim to clarify the
logical chains between the agency’s general, specific and operational objectives for the period 2017-
201036. The new thematic approach (and the introduction of seven Strategic Projects) adopted by the
agency has also better enabled the ETF to be more aligned with the EU’s aims and objectives37. Figure 2
33 European Commission (2018a - upcoming), Evaluation of the EU Agencies under the remit of DG Employment:
EUROFOUND, CEDEFOP, ETF and EU-OSHA, Annex 5, ETF Update.
34 Ibid., p. 4.
35 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium.
36 ETF (2016) Single Programming Document 2017-2020. Annual Work Programme 2017 and first indications for
Work Programme 2018. GB/16DEC/011.
37 ICON (2016) Evaluation of ETF activities in the skills and employment field in partner countries organisation and
methodology. Final Report.
Evaluation of ETF functions – Interim Report
12
below outlines the ETF’s general, specific and operational objectives, and the agency’s seven Strategic
Projects in 2017-2020.
FIGURE 2. THE ETF’S OBJECTIVES AND STRATEGIC PROJECTS (2017-2020)
Source: adapted from ETF (2017), Annual Work Programme 2018.
The ETF’s general objective is to contribute to human capital development in the context of EU external
relations through improved VET systems and improvements in labour market integration and re-
integration, according to its Founding Regulation38. This is achieved through five specific thematic
objectives which reflect the policy needs of partner countries and draw on relevant EU and international
policy frameworks and methodologies. The ETF aims to support partner countries to improve both
initial and continuing VET in five key policy areas: 1) VET governance systems and policy making; 2)
VET provision and quality; 3) Qualifications and Qualifications systems; 4) Entrepreneurial learning
and enterprise skills; and 5) Employment, skills and employability.
These five specific objectives are implemented through four main operational objectives and priorities.
These outline how the ETF supports partner countries through the four main functions of the agency39:
38 Regulation (EC) No 1339/2008, Art. 1.
39 Ibid., Art. 2.
Evaluation of ETF functions – Interim Report
13
1) Support to EU assistance in the context of EU external policies; 2) Provision of policy analysis and
advice; 3) Support to capacity building; and 4) Communication, knowledge dissemination and
networking.
3.1. Achieving specific objectives in the ETF priority areas
Previous evaluations show that the ETF has generally achieved its five main specific objectives. The
last external evaluation showed that the ETF’s contribution has been especially strong in its three first
specific thematic objectives: governance, systems and policy-making; VET provision and quality
assurance; and qualifications and qualifications systems. The ETF’s contributions in the development
of employment, skills and employability, and in entrepreneurial learning and enterprise skills have
been evaluated in a more heterogeneous way by stakeholders (see Figure 3 below).
FIGURE 3. EXTENT OF ETF CONTRIBUTION TO DEVELOPMENTS IN PARTNER COUNTRIES: SPECIFIC OBJECTIVES COMPARED
Source: European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the EFECTIV Consortium.
Evidence shows that the agency’s work has produced key effects in the area of governance systems and
policy-making. The ETF’s interventions in this area have successfully empowered a critical mass of
stakeholders in partner countries, who have highly utilised the ETF’s products and services and have
improved their capacity40. The Torino Process has been a beneficial framework to improve evidence-
based policy-making and better governance, notably by helping to reinforce coordination and
collaboration between relevant stakeholders and increasing the quantity and quality of participation by
social partners in the policy making process41. Stakeholders have emphasised the effectiveness of the
ETF in achieving its objectives regarding the Torino Process, as demonstrated by the interim results of
the evaluation of the Process (see Figure 4 below).
40 PPMI and Europe Ltd. (2017), Evaluation of ETF activities in the field of VET governance. Final Report.
41 PPMI (2018), ETF Torino Process Evaluation: Interim Report.
5
6
9
5
8
28
30
46
50
59
32
35
30
35
27
6
7
3
4
1
3
1
2
3
25
19
11
4
3
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Employment, skills and employability
Entrepreneurial learning and entreprise skills
Qualifications and qualifications systems
VET provision and quality assurance
Governance systems and policy making
ETF was the main driver of change ETF provided important support
ETF provided some support ETF did not provide any support
No developments in this area Don't know
Evaluation of ETF functions – Interim Report
14
FIGURE 4. STAKEHOLDER RESPONSES TO THE QUESTION "HAS THE TORINO PROCESS BEEN EFFECTIVE IN
ACHIEVING ITS OBJECTIVES?"
Source: European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the EFECTIV Consortium.
The ETF was also found to have successfully contributed to VET provision and quality assurance in
its partner countries, notably by supporting to increase the participation of stakeholders in the VET
process (such as in Central Asia, or Serbia)42, and in improving continuous professional development
(CPD) of VET teachers and trainers43.
In the area of qualification and qualifications systems, evaluations show that the ETF made
particularly strong contributions in relation to strengthening the focus on skills for employability and
the developments of National Qualifications Systems (NQFs) and learning outcomes. Partner countries
also showed a considerable level of progress in qualifications-related reform44. Stakeholders recognised
that the ETF’s contribution has been central in driving the developments of NQFs.
In the area of entrepreneurial learning and enterprise skills, the SBA assessments have been the
cornerstone of the ETF’s work. The 2015 evaluation of the ETF’s work in this area found that the agency’s
work has been considered effective, in particular for bringing together and engage different
stakeholders, and improve the policy dialogue in partner countries45. On the other hand, although the
last external evaluation highlighted that SBA assessments were found to be a “stimulus and motivator
when benchmarking against other countries” by some partner country stakeholders, survey
respondents were generally less positive regarding the ETF’s support in this field compared other
specific objectives.
Similarly, in the area of employment, skills and employability, evaluation results appear to be mixed.
While evidence shows that the ETF had a pivotal role in some countries, respondents to a stakeholder
42 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium.
43 ICON (2017), Evaluation of ETF activities in improving continuous professional development of vocational
teachers and trainers. Final Report.
44 ICON (2014), Evaluation of Qualifications activities in ETF partner countries. Final Report; European
Commission (2016), External Evaluation of the European Training Foundation. Final Report by the EFECTIV
Consortium.
45 ICON (2015), Evaluation of Entrepreneurship and Enterprise Skills Development in ETF partner countries. Final
Report.
14
7
3
7
13
6
56
53
49
67
23
27
42
33
0 10 20 30 40 50 60 70 80 90 100
Partner Countries
EU
ETF staff
ETF GB
don't know
to a poor extent
to some extent
to a high extent
Evaluation of ETF functions – Interim Report
15
survey from partner countries had a heterogeneous feedback regarding the extent to which the support
provided in this area was one of the key drivers of change: 5% thought that the ETF was the main driver
of change in this area, 28% thought that the ETF provided important support, while 32% thought that
the ETF provided some support only (see Figure 3 above)46.
3.2. Implementation of the ETF’s core functions
The ETF’s four operational objectives, determined by the functions of the agency as defined in its
mandate, are tools through which the ETF delivers its support to the partner countries (see Figure 2
above). They comprise support to EU assistance programmes, policy analysis, capacity building, and
communication, knowledge dissemination and networking (see Figure 5 below).
FIGURE 5. THE ETF FUNCTIONS
The ETF’s functions are mostly of a cross-cutting nature and important for achieving all the specific
objectives of the agency. In particular, the provision of policy analysis and policy advice, and the
support to capacity building, are transversal objectives across all ETF projects and policy areas.
Similarly, communication, knowledge dissemination and networking represent a cross-cutting
objective common to all strategic projects which is central to enabling information exchange and policy
learning among ETF stakeholders in partner countries. Support to EU assistance consists of support to
the EU institutions (namely Commission DGs, EEAS and EU Delegations), upon request, for the
different phases of the EU external assistance project cycle: identification, formulation,
operationalisation and implementation, monitoring and evaluation. This function also involves the
provision of input to the bilateral external assistance related policy dialogue and EU reporting processes
on human capital development, inputs to regional policy dialogues, and cross-country support.
Evidence from recent evaluations shows that the ETF’s work has been particularly useful in supporting
EU institutions for the different phases of the EU external assistance project cycle. The ETF is regularly
requested to support identification and formulation, operationalisation and implementation of EU
46 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium.
Support to EU assistance
Provision of policy analysis
and advice
Support to capacity building
Communication, dissemination
and networking
Evaluation of ETF functions – Interim Report
16
external assistance through content and technical advice47. The ETF has also been effective in supporting
and enhancing the involvement, coordination and collaboration of diverse partner country stakeholders
in human capital development policies, providing policy advice and sharing examples of good practice,
better data collection and analysis48. All these achievements notably informed and enabled: (a) partner
countries to take steps towards improving governance and modernisation of their human capital
development policies, and (b) better programming of EU external assistance to support partner
countries in their reforms in the areas of VET and lifelong learning.
A survey of the ETF’s partner country representatives recently indicated that the respondents
overwhelmingly found the ETF’s main functions to be very useful or useful49: 55-56% and 43-44% of
respondents reported that the ETF’s work was very useful or useful, respectively, in the following areas:
collection and provision of information, policy analysis and advice; capacity building for the
development of better governance structures; networks to exchange information and practices of VET
development; and capacity building and support in the field of human capital development. The survey
results showed that there is scope for improvement on activities that contribute to the programming of
EU funds (42% evaluated this as very useful and 55% as useful), and on capacity building for the
development of labour market information systems (46% and 54%, respectively)50.
Recent evaluation evidence also underlined the need to strengthen some of the ETF’s key functions.
In particular, evaluations noted that the ETF should continue strengthening the agency’s
communication and knowledge dissemination efforts to ensure stakeholders fully understand the
ETF’s role and objectives. According to the evaluation of ETF activities in the field of VET governance,
the agency can be misperceived as a donor organisation providing time-limited, project-based
interventions rather than a long-term presence focused on building capacity51. To maximise its impact,
the evaluation recommended the ETF to make its policy learning tools better accessible to more diverse
stakeholders in the partner countries, for example by making its information available in local
languages and to support local partners through training in their use52. The 2015 evaluation of
entrepreneurship and enterprise skills development in partner countries also noted that the ETF’s
products remain difficult to access for stakeholders53. The 2017 evaluation of ETF activities in VET
47 ETF (2017c), European Training Foundation 2017-20. Annual Work Programme 2018. GB/17/DEC/011.
48 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium; European Commission (2018a, upcoming), External evaluation of the four agencies under
the remit of DG EMPL – Cedefop, Eurofound, ETF and OSHA, Final report; PPMI (2018), ETF Torino Process
Evaluation, Interim Report; ETF, Torino Process Stakeholder Survey – Main Results, 2017.
49 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium cited in European Commission (2018a, upcoming), External evaluation of the four agencies
under the remit of DG EMPL – Cedefop, Eurofound, ETF and OSHA, Final report
50 European Commission (2018a, upcoming), External evaluation of the four agencies under the remit of DG EMPL
– Cedefop, Eurofound, ETF and OSHA, Final report
51 PPMI and Europe Ltd (2017), Evaluation of ETF activities in the field of VET governance. Final Report.
52 Ibid.
53 ICON (2015), Evaluation of Entrepreneurship and Enterprise Skills Development in ETF partner countries. Final
Report.
Evaluation of ETF functions – Interim Report
17
governance54 and the 2016 evaluation of ETF activities in the skills and employment field55 emphasised
the need to tailor and simplify its deliverables to facilitate the multiplication of results.
Previous evaluations argued that the success of the ETF’s actions varied according to contextual
conditions in partner countries. The variable capacity of partner countries to absorb ETF interventions
required attention and tailored approaches56, which the ETF was generally able to provide. In the area
of VET governance, the ETF demonstrated a strong capacity for providing differentiated interventions
to its partner countries57.
Some evaluations noted that the ETF’s interventions were not differentiated enough across partner
countries to meet diverse needs. The 2015 evaluation of entrepreneurship and enterprise skills
development in ETF partner countries reported that some of the ETF’s work lacked a broader strategic
focus in partner countries. It seemed that it was more a matter of stakeholders being aware of the
possibility than a strategic deliberate choice by ETF to focus on specific countries58. The 2016 evaluation
of ETF activities in skills and employment stated that countries particularly valued the sharing of
experience with their peers with similar economic, social and labour market contexts, which may call
into question the value of some of the current groupings used59. Moreover, the 2016 external evaluation
recommended that the ETF do more to understand systematically where its interventions are likely to
have most effect and how the nature of the required activities may vary depending upon factors such
as country size and stages of (institutional and policy) development60.
Nevertheless, the evaluations vindicated the ETF’s focus on a limited number of countries in which
a window of opportunity for reforms was open. The ETF successfully took advantage of favourable
contexts and helped catalyse developments that promised positive change and achieved its objectives.
The prioritisation of work within a limited number of partner countries enabled the agency to be flexible
and adjust its interventions to the evolving context of each supported partner country61. The 2016
external evaluation noted that the ETF support was successfully adapted to candidate countries, helping
them to integrate into the EU coordination process for VET (the Copenhagen process)62. Ultimately, one
of the significant effects of the ETF in relation to candidate countries is the role it has played in the
development of evidence-based policies and strategies that are in line with European strategies (such
as the 2020 Vision for Skills through the FRAME initiative), while encouraging national stakeholders to
increase their ownership over the process. Moreover, the coordinated approach taken by the ETF in
54 PPMI (2017), Evaluation of ETF Activities in the Field of VET Governance. Final Report.
55 ICON (2016), Evaluation of ETF Activities in the Skills and Employment field in Partner Countries Organisation
and Methodology.
56 European Commission (2018a – upcoming), Evaluation of the EU Agencies under the remit of DG Employment:
EUROFOUND, CEDEFOP, ETF and EU-OSHA, 2018; ICON (2014a), Evaluation of Qualifications Activities in ETF
Partner Countries.
57 PPMI (2017), Evaluation of ETF activities in the field of VET Governance. Final Report.
58 ICON (2015), Evaluation of Entrepreneurship and Enterprise Skills Development in ETF partner countries. Final
Report. 59 ICON (2016), Evaluation of ETF activities in the Skills and Employment field in Partner Countries Organisation
and Methodology. Final Report.
60 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium.
61 Ibid.
62 Ibid.
Evaluation of ETF functions – Interim Report
18
synergy with the Commission and the EU Delegations helped in achieving important results in the
countries taking part in the Torino process and undergoing VET and skills development reforms63.
4. Have the ETF’s actions led to sustained change in partner countries?
Key findings
- Attributing institutional outcomes and especially societal impacts to the ETF’s
interventions is inherently difficult.
- Nevertheless, the ETF’s interventions effectively facilitated VET policy developments and
had a visible impact in partner countries on reforming VET and supporting the
development of national strategies and reform programmes.
- The ETF’s contribution has been the strongest in the fields of: governance, systems and
policy-making; VET provision and quality assurance; and qualifications and qualification
systems.
- The achievement of long-term impact depended on multiple contextual factors in the
partner countries
4.1. Attributing impacts to the ETF’s interventions
Recent evaluations of the ETF show that attributing institutional outcomes and societal impacts to the
ETF’s interventions was inherently difficult largely because of the intangible nature of ETF activities,
the lack of quantitative data in partner countries, the small scale of the agency’s activities, and the
difficulty to link education and training policy interventions to concrete socio-economic results. Long-
term impacts at policy and socio-economic level were difficult to assess in a direct causal link from the
ETF’s intervention. These are more dependent on a wide range of factors outside the control of the ETF,
such as the stability of the political environment and the economic context.
Nevertheless, in addition to results at system level such as building capacity of national stakeholders,
making labour market analysis tools available, successfully sharing knowledge, information and good
practice64 (see section 3 above), recent evaluations found the ETF to have made impacts in a number of
distinctive areas (see Figure 6 below).
63 European Commission (2013), TVET and Skills Development in EU Development Cooperation. Final Report
prepared by AETS for DG DEVCO – EuropeAid.
64 ICON (2016), Evaluation of ETF activities in the Skills and Employment field in Partner Countries Organisation
and Methodology. Final Report.
Evaluation of ETF functions – Interim Report
19
FIGURE 6. RESULTS AND IMPACTS OF THE ETF (2017-2020)
Source: compiled by PPMI.
4.2. The ETF’s impacts on the policy development in partner countries
Evidence shows that the ETF’s interventions facilitated policy developments and had a visible impact
in several partner countries. Without the ETF, these developments would have taken place more
slowly, differently or not at all. This was supported by evidence from the ETF’s annual activity reports65
and the Torino Process implementation mapping report66. The agency’s 2016 consolidated annual
activity report stated that the ETF brought change to national reform processes, especially in countries
where it was used for international donor coordination and as an input to EU programming (e.g. in
Lebanon, Egypt, Ukraine, Uzbekistan, Morocco, Jordan, Albania, Kosovo, and Montenegro)67.
Evidence shows that countries which participated in the Torino Process benefited from the quality
of the support provided by the ETF, as well as from the synergies between the actions of the ETF, the
EU Delegations and the Commission in support of partner country reforms. This holistic approach to
65 E.g. ETF (2017a), Consolidated Annual Activity Report 2016, GB/17/DEC/005.
66 ETF (2017b), Torino Process 2016 Implementation Mapping–Note.
67 ETF (2017a), Consolidated Annual Activity Report 2016, GB/17/DEC/005.
Evaluation of ETF functions – Interim Report
20
the EU’s external assistance projects, in which the ETF was involved, led to important results in the
partner countries undertaking VET and skills development reforms68. The Torino Process also
contributed to new strategies in several partner countries, such as in Azerbaijan (in the design of the
strategic roadmap for VET and the work of the monitoring system), Albania and Turkey (in the
development of national strategies or in providing evidence for ongoing reforms)69.
As described above, the ETF’s contribution has been considered by stakeholders to be the strongest in
relation to governance, systems and policy-making, VET provision and quality assurance, and
qualifications and qualification systems70. In these areas, the ETF was believed to be the main driver
or to have provided important support to policy change in partner countries71. The 2017 evaluation of
the agency’s activities in VET governance72 showed that the ETF’s governance interventions made a
considerable contribution to building teams of VET reformers in partner countries73. The ETF’s
contribution has been particularly valued in relation to stronger coordination and collaboration
between relevant stakeholders and the increased quantity and quality of participation by social partners
in policy-making processes74.
Stakeholders recognised that the intervention of the ETF helped to improve the participation of a
broad range of stakeholders in debates and in the design and implementation of VET policies: from
11% to 15% believed that ETF was the main driver of change, and from 39% to 44% believed that it
provided important support75. To a lesser extent, from 3% to 8% of stakeholders claimed that the ETF
was the main driver of change in the updating or implementation of laws and regulations establishing
the principles, institutions and processes for good governance in VET. The evaluation of ETF activities
in the field of VET governance found that the ETF’s actions seldom triggered immediate policy changes,
but that these potentially could not be observed yet due to a short time span during which this strategic
project was implemented76.
4.3. The influence of contextual factors and the ETF’s impact
The impact of the ETF’s intervention in partner countries depend on multiple contextual factors that
cannot be directly controlled or influenced by the ETF. Key contextual factors include: the stability of
the economic and political environment; the level of financial resources allocated to the VET sector; the
lack of priorities and the distribution of roles among stakeholders; the lack of trust between
68 European Commission (2013), TVET and Skills Development in EU Development Cooperation. Final Report
prepared by AETS for DG DEVCO – EuropeAid.
69 PPMI (2018), ETF Torino Process Evaluation, Interim Report.
70 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium; PPMI and Europe Ltd. (2017), Evaluation of ETF activities in the field of VET Governance.
Final Report.
71 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium, p. 75
72 PPMI and Europe Ltd. (2017), Evaluation of ETF activities in the field of VET Governance. Final Report.
73 Ibid.
74 Ibid., p. 76.
75 PPMI and Europe Ltd. (2017), Evaluation of ETF activities in the field of VET Governance. Final Report.
76 Ibid.
Evaluation of ETF functions – Interim Report
21
stakeholders; the lack of competences to assume new roles and functions; and the insufficient dialogue
between stakeholders (see Figure 7 below).
FIGURE 7. CONTEXTUAL FACTORS AND THE IMPACT OF ETF INTERVENTIONS IN PARTNER COUNTRIES
Source: compiled by PPMI.
For example, the insecurity and political instability which marked several countries from the Southern
and Eastern Mediterranean (SEMED) region in the last decade were a major obstacle for policy
development and hampered the ETF’s interventions in the fields of VET governance77 as well as in
entrepreneurship and enterprise skills development78. On the other hand, as (youth) unemployment
and poverty were some of the key catalysts for most of the political crises in the region, these topics
gained increasing focus in national policy agendas, and justified the relevance of the ETF’s intervention
in human capital development. Comparatively, a supportive and more stable political context in South
Eastern Europe and Turkey (SEET region) was more favourable to ETF actions and helped to achieve a
greater and more sustainable impact. Unstable political environments and the lack of government
commitment hindered effective implementation of reforms due to the discontinuation of actions and
high turnover of the relevant personnel, which in turn jeopardised the impact of the ETF’s action.
Previous evaluations frequently mentioned the strong influence of contextual factors on the impact of
the ETF activities. However, they often failed to reflect on their implications for the ETF’s work.
Acemoglu and Robinson (2012) help us understand why many ETF partner countries tend to make sub-
optimal public policy choices despite years of technical assistance and development aid, including
support and advice from the ETF79. The fear of losing their monopoly on political and economic power
77 Ibid.
78 ICON (2015), Evaluation of entrepreneurship and enterprise skills development in partner countries. Final
Report.
79 Acemoglu D., and Robinson J.A. (2012), Why Nations Fail: The Origins of Power, Prosperity and Poverty. New
York: Crown Publishers.
Impact of ETFinterventions
Economic and political
environment
Level of final resources
allocated to the VET sector
Lack of priorities in
partner countriesLack of
competences to assume
new roles and functions
Distribution of roles among
stakeholders
Insufficient policy
dialogue
Evaluation of ETF functions – Interim Report
22
often motivates the national elites to resist steps towards making the political and economic institutions
more inclusive and modernising VET systems to make them more responsive to labour market needs.80
The 2018 Commission Communication on EU Enlargement Policy underlines that reforms in the area
of the rule of law, fundamental rights and good governance remain the most pressing issue for the
enlargement countries81. It reveals that corruption persists at all levels despite these countries’ efforts
to bring legal and institutional frameworks in line with the EU acquis and European standards. Weak
state institutions and the lack of capacity of national parliaments to exercise core legislative and
oversight functions are undermining the political dialogue82. The EU uses multiple instruments to
monitor enlargement countries and ensure that they remain on the reform track, such as advisory
missions, systematic peer reviews with the participation of Member State experts, trial monitoring and
other instruments mentioned in the recent Commission’s Communication “A credible enlargement
perspective for and enhanced EU engagement with the Western Balkans”83. However, in most other
ETF partner countries, such instruments do not apply, magnifying the reform challenges that they face.
The ETF has developed and perfected over the years its unique approach to promoting positive
institutional change and to making VET governance institutions more inclusive. The ETF has done
so through its long-term and continuous involvement in partner countries, its participatory approach
involving, providing expertise and empowering less influential stakeholders (including social partners
and civil society organisations) into the public policy dialogue (often for the first time). It also supported
partner countries’ efforts to adapt to the EU acquis and standards, helping to promote the EU as a role
model in human capital development (see section 5 below). The ETF has been particularly successful
where it exploited windows of opportunity opening with the changes in power structures, the initiatives
to reform institutions and by forging alliances with partner countries’ reform-minded elites. Through
its participatory approach and focus on good governance, the ETF has targeted some the fundamental
reasons behind the under-development of human capital in its partner countries.
80 Ibid.
81 European Commission (2018b), Communication from the Commission to the European Parliament, the Council,
the European Economic and Social Committee and the Committee of the Regions. 2018 Communication on EU
Enlargement Policy. Strasbourg, 17.4.2018, COM(2018) 450 final.
82 Ibid.
83 European Commission (2018c), Communication from the Commission to the European Parliament, the Council,
the European Economic and Social Committee and the Committee of the Regions. A credible enlargement
perspective for and enhanced EU engagement with the Western Balkans. Strasbourg, 6.2.2018, COM(2018) 65 final.
Evaluation of ETF functions – Interim Report
23
5. To what extent are the ETF’s actions unique compared to those of other actors?
Key findings
- The added value created by the ETF lies in its unique combination of competences and
approaches to human capital development.
- The ETF’s action has created unique added value due to its long-term and continuous
involvement, thematic and geographical expertise, participatory approach, empowering
a broad range of stakeholders through policy dialogue, and in the European dimension
of its work in the partner countries.
The previous evaluations emphasised the ETF's unique combination of competences and approaches,
notably illustrated by the Torino process. This set the ETF apart from other organisations active in the
field of human capital development. While other international development organisations conducted
similar individual activities, the ETF created most of its added value in four main areas (see Figure 8
below).
FIGURE 8. ADDED VALUE OF THE ETF COMPARED TO OTHER RELEVANT ACTORS
Source: European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the EFECTIV Consortium.
5.1. Thematic and geographic expertise
The thematic and geographic expertise enabled the ETF to create added value through the specific
knowledge of ETF staff in the diverse thematic and geographic areas in which it operates, as well as
through the quality of the data and methodologies used. Only few partner countries have strong
national-level research institutions in human capital development policies. Therefore, the analysis
provided by the ETF is often the only reliable evidence that they can obtain for their evidence-based
policy making. Moreover, as the only EU agency working specifically in the context of the external
relations policies, the ETF has developed a country specific knowledge of its partner countries that no
Thematic and geographic
expertise
Stakeholders' involvement
and participatory approaches
Long-term and
continuous involvement
Coherent European dimension
Evaluation of ETF functions – Interim Report
24
other EU institution or agency possesses84. The ETF has also been able to generate synergies between its
thematic and geographic (country- and region-specific) knowledge, between projects and across
countries based on regional approaches85.
5.2. Capacity to involve stakeholders and apply participatory
approaches
Secondly, the capacity to involve stakeholders and create regional networks was considered as a
strong element of added value of the ETF. The agency’s participatory approach, promoting the
networking between diverse stakeholders such as employers’ organisations and VET institutions,
supporting the ownership and involvement of partner countries, added significant value to its activities
and was particularly distinctive from other international actors in the area86. The Torino Process has
created added value by supporting the participation of a broad range of stakeholders in the policy
dialogue and to foster a policy making culture based on evidence at partner country level87. The ETF’s
action in this framework has allowed to add value to the work of national authorities and other partner
country stakeholders, for example as a promoter and catalyser of national efforts in the design or reform
of national VET strategies, QA, or the development of NQFs88. The ETF’s “perseverance in proposing
and coherently adopting a participatory process in policy making and coherently supporting it with
competent expertise, production and sharing of high quality knowledge” 89 also brought tangible added
value to the agency’s activities compared to that of other national and international actors.
The creation of separate thematic and geographic departments by the agency notably offered potential
to further strengthen regional initiatives in terms of networking and policy learning between
stakeholders90. For instance, the ETF is currently signing cooperation agreements with UNESCO and
the European Bank for Reconstruction and Development (EBRD). Both organisations indicated that they
would like to have a more structured exchange with the ETF’s networks as it gives these international
organisations more credibility and the opportunity to gain endorsement and approval of the work that
they are doing by local actors91.
84 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium; European Commission (2018a - upcoming), Evaluation of the EU Agencies under the remit
of DG Employment: EUROFOUND, CEDEFOP, ETF and EU-OSHA.
85 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium.
86 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium; European Commission (2018a - upcoming), Evaluation of the EU Agencies under the remit
of DG Employment: EUROFOUND, CEDEFOP, ETF and EU-OSHA.
87 PPMI (2018), ETF Torino Process Evaluation, Interim Report.
88 Such as in Morocco. See: ICON (2016), Evaluation of ETF activities in the skills and employment field in partner
countries organisation and methodology. Final Report.
89 Ibid., p. 21.
90 PPMI (2012), External Evaluation of the European Training Foundation. Final Report.
91 European Commission (2018a - upcoming), Evaluation of the EU Agencies under the remit of DG Employment:
EUROFOUND, CEDEFOP, ETF and EU-OSHA, Annex 5, ETF Update.
Evaluation of ETF functions – Interim Report
25
5.3. Long-term and continuous involvement in partner countries
Thirdly, the ETF added value through its long-term and continuous involvement in the partner
countries. Due to its long-term and in-depth cooperation with national actors in the partner countries
where it is engaged, the ETF has a strong knowledge of partner countries’ history of reform progress,
and has been able to provide highly relevant, tailored and timely support. This generated appreciation
and fostered long-term professional relationships in the multiple types of activities that the agency is
engaged in. This also helped generate social capital which was important when introducing innovative
projects.
The ETF demonstrated capacity to establish and maintain good working relationships with a wide
variety of stakeholders in partner countries, which was essential since partners are under no obligation
to cooperate. In turn, this helped build mutual trust which enabled the ETF to operate as a neutral
broker, or a critical friend92 and strengthen the sustainability of the ETF’s actions93. Noticeably, several
evaluations underlined that the ETF was not perceived as a donor organisation, but as “an organisation
that has long-term commitments with the key stakeholders in the field of education and labour market”
94, which makes it truly unique in the field of human capital development.
5.4. The European dimension of the ETF’s work
Finally, the European dimension of the ETF’s work and its high level of complementarity with other
EU institutions and agencies has brought a key added value to the agency’s activities95. Through the
Torino process, the ETF supports partner countries in reviewing their progress in VET reform and
developing necessary policy responses harmonising their policies and governance on VET, lifelong
learning and employment. From the EU’s perspective, as the only decentralized EU agency working on
the social and economic development of the EU neighbourhood countries, the ETF has a specific focus
which gives it a unique added value compared to that of other EU and international organisations.
The ETF is an indispensable knowledge provider in programming and supporting the
implementation of EU assistance to partner countries, and in maintaining a policy dialogue with
them. The agency has been complementary with the work of the Commission’s DGs (including DG
DEVCO and DG NEAR), other EU agencies (including Cedefop and Eurofound) and EU Delegations,
thanks to its specific thematic and geographic expertise, to its unique approaches of engaging with
partner country stakeholders and to the coherence of the its work with that of other relevant national,
European and international actors.
92 European Commission (2016), External Evaluation of the European Training Foundation. Final Report by the
EFECTIV Consortium; ICON (2017), Evaluation of ETF Activities in Improving Continuing Professional
Development of Vocational Teachers and Trainers.
93 ICON (2014a), Evaluation of Qualifications Activities in ETF Partner Countries; PPMI (2018), ETF Torino Process
Evaluation, Interim Report; European Commission (2016), External Evaluation of the European Training
Foundation.
94 ICON (2015), Evaluation of entrepreneurship and enterprise skills development in partner countries. Final
Report.
95 European Commission (2016), External Evaluation of the European Training Foundation.
Evaluation of ETF functions – Interim Report
26
6. What could the ETF do differently to fully realise its mandate and adapt to its evolving context?
The results of this meta-analysis show that the ETF has generally been evaluated very positively since
2014. Evaluations found that the ETF’s work has been relevant to stakeholders in the partner countries
and EU policy stakeholders, has been coherent and complementary with the EU internal and external
policies, has achieved its objectives and had positive impacts, has demonstrated its specific added value,
and has worked towards greater sustainability of its results.
Nevertheless, several recommendations which mainly emerge from recent evaluations can be drawn to
improve further the performance of the agency and allow it to achieve a greater impact in implementing
the EU’s post-2020 policy agenda.
6.1. Responding to the needs of stakeholders
The ETF has been continuously relevant to the needs of its partner countries during the evaluation
period, especially in the areas of VET governance, systems-and policy-making, skills and qualifications.
The Torino Process has been a particularly useful tool to help identify partner countries’ needs and
knowledge gaps and improve the relevance of the agency’s interventions.
The ETF was generally coherent and complementary to other EU policies and to the work of other EU
institutions and agencies, as well as other international actors in the partner countries. Effective working
relationships with the EU Delegations were also crucial for the success of the ETF in partner countries
but were too dependent on informal processes. The introduction of a more structured cooperation
framework between the ETF and EU Delegations would help improve the relevance and effectiveness
of the EU’s human capital development interventions in partner countries in general. Moreover, there
is room to improve the coherence and complementarity of the ETF’s work with that of the Commission
and for increased cooperation with other EU decentralised agencies. Finally, while the ETF’s country
strategic papers helpfully summarised the activities of the ETF in a given partner country, they lacked
a strategic and forward planning dimension. For example, they did not include a description of the
expected change in a country or the role of the ETF in supporting that change.
Recommendations
- The ETF should better evaluate the context in which its interventions will take place in
partner countries (based on their sage of development, preparedness and commitment)
to increase the relevance of its actions.
- Increased communication and coordination between the ETF, the Commission, and EU
Delegations in partner countries would help better align and articulate the ETF’s
objectives at the strategic and detailed partner country levels and would enable ETF to
provide stronger and more relevant policy-oriented advice and evidence to EU
institutions.
- The ETF could make its country strategy papers more strategic by providing a description
of the expected change in a country, explaining the role of the ETF, and elaborating a
concrete action plan as well as provisions for risk management.
Evaluation of ETF functions – Interim Report
27
6.2. Implementing the objectives and achieving sustained change
Among the ETF key functions, capacity building and the provision of policy analysis and advice were
widely valued by stakeholders. Knowledge dissemination and networking were similarly valued,
although the ETF could have ensured more widespread development of networks to help the
sustainability of its interventions. Meanwhile, the provision of expertise with regard to EU project and
programming cycle would benefit from being placed on a more systematic basis so that EU Delegations
are required to tap into ETF expertise96.
There is evidence of variation in the ETF’s effectiveness at the level of specific objectives. Its contribution
to partner country developments has been especially strong in terms of governance, systems and policy-
making, the development of VET provision and quality assurance, and in qualifications and
qualifications systems. Although the ETF’s activities strongly contributed to policy learning and
networking between stakeholders, misperceptions of the ETF’s objectives and difficulties for wider
groups of partner country stakeholders to access its policy learning tools limited its effectiveness.
Nevertheless, evidence shows that the ETF’s interventions facilitated VET policy developments and had
a visible impact in some partner countries. Without the ETF, these developments would have taken
place more slowly, differently or not at all.
The capacity of partner countries to absorb ETF interventions varied and required differentiation.
However, the ETF was not able to satisfy all the needs in all its partner countries. The involvement with
broader groups of stakeholders created the need for even more involvement and assistance. The
growing demand for ETF support in the partner countries, the increasing importance attributed to
human capital development in the EU external policies call into question whether the current level of
ETF funding is adequate for the post-2020 period. Many of the recommendations provided repeatedly
to ETF by previous evaluations require additional human and financial resources to implement.
Recommendations
- The agency should consider whether the more limited contribution that it provided to
developments in partner countries in the areas of labour market systems and skills for
employability as well as in entrepreneurial learning and entrepreneurial skills was due to
circumstances beyond its control or requires stronger action on its part.
- The ETF should take measures to better understand where its interventions are likely to
have most effect and impact at partner country level, and how the nature of its activities
may vary on contextual factors such as general stage of policy development in relevant
fields at the time of selection.
- The ETF should work on identifying supporting conditions for the successful
implementation of its interventions and aim to establish which interventions need to take
place in which sequence, to increase their chance of success.
96 European Commission (2018a - upcoming), Evaluation of the EU Agencies under the remit of DG Employment:
EUROFOUND, CEDEFOP, ETF and EU-OSHA.
Evaluation of ETF functions – Interim Report
28
6.3. Adding value to the actions of other actors
The ETF's combination of competences and approaches set it apart from other organisations in the field
of human capital development. While other organisations might perform some similar functions, the
added value created by the ETF mainly lay in four specific fields: the thematic and geographical
expertise and the synergies it achieves between the two; its capacity to involve stakeholders and create
regional networks; its long-term and continuous involvement in partner countries; the European
dimension of its work and the complementary nature of its activities with respect to the Commission
services and other EU agencies.
Recommendations
- A more structured collaboration framework between the ETF and EU Delegations in the
partner countries would help improve the utilisation of ETF’s expertise and support the
programming of investment in human capital development.
- The regional dimension of ETF’s work could be linked more closely to and support more
explicitly the implementation of the regional cooperation frameworks established by the
Commission and the EEAS.
Evaluation of ETF functions – Interim Report
29
References
Acemoglu D., and Robinson J.A. (2012), Why Nations Fail: The Origins of Power, Prosperity and
Poverty. New York: Crown Publishers.
Caves, K. and Baumann, S. (2018), Getting there from here: A literature review of VET reform
implementation. KOF, Swiss Economic Institute, KOF Working Papers, No. 441.
ETF (2017a), Consolidated Annual Activity Report 2016, GB/17/DEC/005
ETF (2017b), Torino Process 2016 Implementation Mapping–Note.
ETF (2017c), European Training Foundation 2017-20. Annual Work Programme 2018. GB/17/DEC/011.
ETF (2016), Single Programming Document 2017-20. Annual Work Programme 2017 and first
indications for Work Programme 2018. GB/16DEC/011
European Commission (2018a - upcoming), Evaluation of the EU Agencies under the remit of DG
Employment: EUROFOUND, CEDEFOP, ETF and EU-OSHA. Final Report prepared by PPMI and
Ecorys for DG EMPL.
European Commission (2018b), Communication from the Commission to the European Parliament, the
Council, the European Economic and Social Committee and the Committee of the Regions. 2018
Communication on EU Enlargement Policy. COM(2018) 450 final.
European Commission (2018c), Communication from the Commission to the European Parliament, the
Council, the European Economic and Social Committee and the Committee of the Regions. A credible
enlargement perspective for and enhanced EU engagement with the Western Balkans. Strasbourg,
6.2.2018, COM(2018) 65 final.
European Commission (2016), External Evaluation of the European Training Foundation. Final Report
by the EFECTIV Consortium.
European Commission (2013), TVET and Skills Development in EU Development Cooperation. Final
Report prepared by AETS for DG DEVCO – EuropeAid.
ICON (2017), Evaluation of ETF Activities in improving Continuing Professional Development of
Vocational Teachers and Trainers. Final Report.
ICON (2016), Evaluation of ETF activities in the Skills and Employment field in Partner Countries
Organisation and Methodology. Final Report.
ICON (2015), Evaluation of Entrepreneurship and Enterprise Skills Development in ETF partner
countries. Final Report.
ICON (2014a), Evaluation of qualifications activities in ETF partner countries. Final Report.
ICON (2014b), Evaluation of Knowledge Management Innovation in ETF Operations. Final Report
PPMI (2018), Evaluation of the Torino Process. Interim Report.
PPMI and Europe Ltd. (2017), Evaluation of ETF activities in the field of VET governance. Final Report.