Date post: | 14-Feb-2017 |
Category: |
Documents |
Upload: | hoangthuan |
View: | 221 times |
Download: | 2 times |
1
May 2016
Educationand Culture
Final Report
3RD INTERIM EVALUATION OF THE EACEA
2
May 2016
Educationand Culture
Europe Direct is a service to help you find answers
to your questions about the European Union.
Freephone number (*):
00 800 6 7 8 9 10 11
(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).
More information on the European Union is available on the internet (http://europa.eu). Luxembourg: Publications Office of the European Union, 2016 ISBN: 978-92-9492-212-0 doi: 10.2797/203152 European Union, 2016 Reproduction is authorised provided the source is acknowledged.
http://europa.eu.int/citizensrights/signpost/about/index_en.htm#note1#note1http://europa.eu/
3
May 2016
Educationand Culture
3rd Interim Evaluation of the
EACEA
Final Report
Prepared by: PPMI Group
LEGAL NOTICE
Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use which might be made of the following information. The views expressed in this publication are the sole responsibility of the author and do not necessarily reflect the views of the European Commission.
EUROPEAN COMMISSION
Directorate-General for Education and Culture
4
May 2016
Educationand Culture
5
May 2016
Educationand Culture
Table of Contents
List of tables, figures and boxes ................................................................................ 6 List of acronyms .................................................................................................... 8
Introduction ............................................................................................................ 9 1. Background, approach and methodology .......................................................... 10
1.1. Background .................................................................................................. 10 1.1.1. Purpose and scope of the evaluation ......................................................... 10 1.1.2. Evaluation tasks and questions ................................................................. 10
1.2. An overall evaluation approach ....................................................................... 12 2. Overall performance of the Agency, quantitative and qualitative CBA ................... 14
2.1. Overall performance of the EACEA .................................................................... 14 2.1.1. The Agencys performance according to the main indicators .......................... 14 2.1.2. Quantitative benchmarking of the Commissions executive agencies .............. 17 2.1.3. Comparison between the administrative costs of the centralised an decentralised actions of Erasmus+ ......................................................................................... 19
2.2. The retrospective CBA ..................................................................................... 20 2.2.1. Background of the quantitative CBA ........................................................... 20 2.2.2. Actual staffing and costs of the EACEA ....................................................... 22 2.2.3. Cost-effectiveness of the executive agency scenario and actual savings due to
externalisation ....................................................................................................... 24 2.2.4. Workload analysis ................................................................................... 27 2.2.5. Qualitative aspects of the retrospective CBA ............................................... 29
3. Evaluation findings ........................................................................................ 32 3.1. Question 1 ..................................................................................................... 32 3.2. Question 2 ..................................................................................................... 39 3.3. Question 3 ..................................................................................................... 43 3.4. Question 4 ..................................................................................................... 67 3.5. Question 5 ..................................................................................................... 72 3.6. Question 6 ..................................................................................................... 76 3.7. Question 7 ..................................................................................................... 79
4. Conclusions and recommendations .................................................................. 89
6
May 2016
Educationand Culture
List of tables, figures and boxes
List of Tables
Table 1. The relationship between the evaluation tasks and the evaluation questions ...... 12 Table 2. Performance indicators of the EACEA, 2012-2014, EUR million or percentage .... 15 Table 3. Satisfaction with the EACEAs actual performance and the services it provides ... 17 Table 4. Main performance indicators of the Commissions executive agencies in 2014 .... 18 Table 5. EU commitments to the implementation of Erasmus+ in 2014 (EUR) and its
management cost (%) ............................................................................................ 19 Table 6. Budget managed and human resources in the EACEA compared to all executive
agencies in 2013 and 2020 ...................................................................................... 21 Table 7. The administrative budget of the EACEA in 2012-2014, EUR ............................ 24 Table 8. Estimated costs of the in-house (Commission) and the executive agency scenario,
EUR ...................................................................................................................... 26 Table 9. Qualitative items of the CBA, their correspondence to the evaluation questions and
good practices ........................................................................................................ 29 Table 10. Size, number of departments and number of units of the Commissions executive
agencies in 2014 .................................................................................................... 33 Table 11. Distribution of the EACEAs human resources between 2011 and 2014 ............ 35 Table 12. Staff Engagement Index in the EACEA and other Commission services, 2013-2014
............................................................................................................................ 39 Table 13. Evolution in the EACEAs workload indicators in 2009-2014 ........................... 48 Table 14. Satisfaction of applicants and beneficiaries with the overall quality and timeliness
of the EACEAs services ........................................................................................... 51 Table 15. Transparency of the evaluation and selection process ................................... 52 Table 16. Views on e-forms ..................................................................................... 56 Table 17. Satisfaction with the proportionality of administrative requirements and the related
electronic tools ....................................................................................................... 57 Table 18. Satisfaction of applicants and beneficiaries with the timeliness and quality of
support and guidance provided by the EACEA during various stages of the project life-cycle
............................................................................................................................ 62 Table 19. Satisfaction of beneficiaries with the monitoring activities carried out by the EACEA
............................................................................................................................ 66 Table 20. Findings of the previous evaluation in the field of unintended effects compared to
the current evaluation period ................................................................................... 76
List of Figures
Figure 1. Organisational model for the third evaluation of the EACEA ............................ 13 Figure 2. Framework of the Agency's performance ...................................................... 14 Figure 3. Estimated savings of the executive agency scenario in 2012-2014, EUR ........... 16 Figure 4. Number of staff estimated under different delegation scenarios ...................... 22 Figure 5. Evolution of the EACEAs staff during 2012-2014 .......................................... 23 Figure 6. Estimated savings of the executive agency scenario in 2012-2014, EUR ........... 27 Figure 7. Operational budget related to 2012-2014 calls, EUR million ............................ 28 Figure 8. Number of grants related to 2012-2014 calls ................................................ 28 Figure 9. The EACEA's evolution of Time-to-Award ...................................................... 44 Figure 10. The EACEA's evolution of Time-to-Contract ................................................. 45 Figure 11. The EACEA's evolution of TTP .................................................................... 46 Figure 12. Evolution of the EACEAs estimated residual error rate across the programmes
............................................................................................................................ 47 Figure 13. Execution of the operational budget of the EACEA ....................................... 50 Figure 14. Use of e-forms for submission of the proposals at the EACEA in 2009-2014 .... 55
7
May 2016
Educationand Culture
Figure 15. View on the already introduced simplifications ............................................ 58 Figure 16. Potential for future simplifications ............................................................. 59 Figure 17. Information sources on the programmes managed by the EACEA .................. 61 Figure 18. Usefulness of communication channels during the project implementation stage
............................................................................................................................ 63 Figure 19. Visibility of the EU in the programmes managed by the EACEA ..................... 65 Figure 20. Evolution in the number of the EACEA's project monitoring visits .................. 66 Figure 21. Number of officials seconded to the EACEA in the period 2012-2014 .............. 69 Figure 22. Number of coordination meetings that took place in the period 2012-2014 ..... 70 Figure 23. Extent to which the EACEAs staff are aware of the objectives of the Commission
and its parent DGs .................................................................................................. 72 Figure 24. User satisfaction with the overall quality of the Eurydice reports ................... 82 Figure 25. How much do you agree with the following statements? ............................... 83 Figure 26. To what extent do you agree with the following statements on the different
aspects of Eurypedia? ............................................................................................. 84 Figure 27. How often do you consult Eurydice reports? ................................................ 87 Figure 28. For what purposes do you use Eurydice reports? (You can choose more than one
option). ................................................................................................................. 87
List of Boxes
Box 1. Our assumptions for 2012-2013 and 2014 ....................................................... 25 Box 2. Instruments used to promote the Eurydice network products. ............................ 84
8
May 2016
Educationand Culture
List of acronyms
CAs Contracted Agents
CAF Common Assessment Framework
CBA Cost-Benefit Analysis
CHAFEA Consumers, Health and Food Executive Agency
DG Directorate-General
DG BUDG Directorate-General for the Budget
DG CNECT Directorate-General for Communications Networks, Content and Technology
DG COMM Directorate-General for Communication
DG EAC Directorate-General for Education and Culture
DG ECHO Directorate-General for Humanitarian Aid and Civil Protection
DG EMPL Directorate-General for Employment, Social Affairs and Inclusion
DG GROW Directorate-General for Internal Market, Industry, Entrepreneurship and SMEs
DG HOME Directorate-General for Migration and Home Affairs
DG RTD Directorate-General for Research and Innovation
DPO Data Protection Officer
EA Executive Agency
EACEA Education, Audiovisual and Culture Executive Agency
EASME Executive Agency for Small and Medium-sized enterprises
EC European Commission
EDF European Development Fund
ERCEA European Research Council Executive Agency
EU European Union
FTE Full-Time Equivalent
HR Human Resources
HRM Human resource management
H2020 Horizon 2020 programme
INEA Innovation and Networks Executive Agency
IT Information Technology
LLP Lifelong Learning Programme
OLAF European Anti-Fraud Office
PIC Participant Identification Code
PPMI Public Policy and Management Institute
REA Research Executive Agency
SFS Specific Financial Statement
TAs Temporary Agents
ToR Terms of Reference
TTA Time-to-Award
TTC Time-to-Contract
TTP Time-to-Pay
9
May 2016
Educationand Culture
Introduction
This report was prepared as an outcome of the 3rd interim evaluation of the EACEA (specific
contract No. EAC-2015-270, implementing Framework Contract No. EAC/22/2013-4),
carried out by the PPMI Group (Lithuania) in cooperation with the sub-contractor MATRIX
(United Kingdom). The Final Report was produced on the basis of requirements set out in
the Terms of Reference (henceforth - ToR), information gathered and analysed during the
project, as well as the minutes of project meetings with the Steering Group.
In line with requirements set out in the ToR, the Final Report provides the results of the
evaluation, its conclusions and recommendations. It is divided into the following parts:
Part 1: Background, approach and methodology;
Part 2: Overall performance of the Agency, quantitative and qualitative CBA;
Part 3: Evaluation findings;
Part 4: Conclusions and recommendations;
Annex 1: Terms of Reference;
Annex 2: Synthetic description of the changes (the results of Task 1);
Annex 3: Implementation of the actions in response to recommendations (the
results of Task 3);
Annex 4: Evaluation methodology;
Annex 5: List of interviewees;
Annex 6: Survey questionnaires and results;
Annex 7: Fiches on the selected examples of good practices;
Annex 8: Survey data/dataset and metadata;
Annex 9: Presentation summarising the contents of the report.
10
May 2016
Educationand Culture
1. Background, approach and methodology
1.1. Background
1.1.1. Purpose and scope of the evaluation
An external evaluation on the operation of each Agency has to be carried out every three
years and has to include a cost-benefit analysis (henceforth - CBA) according to Article 25
of Council Regulation (EC) No 58/2003 laying down the statute for Executive Agencies. The
CBA performed must assess the costs of coordination and checks, the impact on human
resources, possible savings within the general budgetary framework of the European Union,
efficiency and flexibility in the implementation of outsourced tasks, simplification of the
procedures used, proximity of outsourced activities to final beneficiaries, visibility of the
Community as promoter of the Community programme concerned and the need to maintain
an adequate level of know-how inside the Commission.1
The first two evaluations of the EACEAs operations covered the periods from January 2006
until April 2008 and from May 2008 until 31 December 2011 respectively. In addition, a
detailed CBA analysing four different scenarios for delegating the management of EU
programmes to the Executive Agencies (including the EACEA) was conducted in 2012-13 for
the 2014-2020 Multiannual Financial Framework.
The purpose of this assignment is to provide the third evaluation of the EACEAs
performance in the period January 2012-December 2014. This period includes the closing
phase of the 2007-2013 programmes and the starting phase of the 2014-2020 programmes
managed by the Agency. Although this evaluation assessed the EACEAs performance by the
cut-off date of 31 December 2014, the evaluator also took into account the main
developments that occurred in 2015 in formulating conclusions and recommendations.
The third evaluation of the EACEA is organised into four tasks:
1. providing a synthetic description of the changes during the period 2012-2014;
2. carrying out a retrospective quantitative and qualitative CBA;
3. assessing the implementation by the EACEA of the actions identified in its Action Plan
in response to the Agencys second evaluation;
4. drawing conclusions and providing recommendations for the future on the basis of
the evaluation questions.
Therefore, the evaluation combined a retrospective and prospective analysis:
a retrospective analysis concerned the changes during the period 2012-2014 (Task
1), CBA (Task 2), the implementation of the EACEAs Action Plan (Task 3) and the
evaluation questions of the ToR (Task 4) related to the Agencys past activities.
Where relevant, this analysis compared the situation at the end of the third
evaluation period (the end of 2014) and the situation at the end of the second
evaluation period (the end of 2011);
a prospective analysis provided recommendations for the Agencys performance in
the future based on the findings of the retrospective analysis and other evaluation
methods.
1 Article 3(1) of Council Regulation (EC) No 58/2003.
11
May 2016
Educationand Culture
1.1.2. Evaluation tasks and questions
Task 1 of the evaluation aimed to provide a synthetic description of the changes and
events that occurred during the evaluation period. We structured Task 1 according to the
organisational model presented in section 1.2. In order to synthesise these changes, we
mapped the main contextual and organisational events of the EACEAs operation, focusing
on the third evaluation period (2012-2014). Due to its descriptive nature, Task 1 did not
answer any specific questions, but it provided the necessary information and reference
points for other tasks.
Task 2 of the evaluation aimed to provide a retrospective CBA. According to Article 3(1)
of Council Regulation (EC) No 58/2003, the costs and benefits of the selected delegation
scenario as identified by the original CBA shall be tested again as part of the interim
evaluation and the results of the CBA shall be updated if need be. The CBA includes an
analysis of the quantitative aspects and the qualitative aspects. This task was related
to five of the evaluation questions in the ToR, namely Q.1, Q.2, Q.3, Q.4 and Q.5.
Task 3 consisted of an assessment of the EACEAs response (as part of its Action Plan
and other actions) to the recommendations of the second evaluation. The report made
10 recommendations. According to the ToR of this evaluation, the third evaluation of the
EACEA should give special attention to the following issues highlighted in the previous
evaluation: the implementation of the EACEAs internal mobility policy and its
effectiveness (especially that for A7 job profiles); the effectiveness and efficiency of the
EACEAs internal processes and procedures (especially monitoring, control and
reporting); the sharing of knowledge between the EACEA and policy/programme units of
DG EAC. We assessed the implementation of the Action Plan setting out the actions to
implement the recommendations of the previous evaluation and the execution of other
relevant actions.
The purpose of Task 4 was to draw conclusions and provide recommendations for the
future on the basis of the evaluation questions outlined in the ToR. Unlike other tasks, this
task provided both retrospective and prospective analysis on the basis of the gathered data.
We answered all the evaluation questions based on their operationalisation in terms of
indicators, target groups to be approached and the main data collection and analysis
methods to be applied, which was presented in the Inception Report. Conclusions of this
evaluation were organised according to the organisational model, they informed the
formulation of recommendations addressed to the EACEA and the Commission.
Table 1 outlines the links between the four tasks of the evaluation and the evaluation
questions provided in the ToR. While Task 4 (conclusions and recommendations) addressed
all the evaluation questions, other tasks of this evaluation were related to some specific
questions.
12
May 2016
Educationand Culture
Table 1. The relationship between the evaluation tasks and the evaluation
questions Evaluation
tasks Evaluation question
Task 2 Task 3 Task 4
Q.1: To what extent are the EACEA's organisation and internal structures appropriate to its new profile and to the needs it is intended to meet? What is the potential impact of the new HR policy of seconded officials, intended to maintain closer links between these officials and the parent DGs, on cooperation between the parent DGs and the Agency? To what extent has the EACEA retained and/or improved the competence and efficiency of its staff in relation to the implementation
of current and new tasks?
Task 3 Task 4
Q.2: To what extent is the EACEA operating according to the legal framework establishing it, and to what extent is there an expectations gap with regard to policy responsiveness in this context (flexibility in the implementation of outsourced tasks)?
Task 2 Task 3 Task 4
Q.3: To what extent has the EACEA led to effective management of the programmes and better services to the stakeholders and what concrete improvements can be suggested (suggestions for improvement should include an assessment of the cost-benefit ratio)?
Task 3 Task 4
Q.4: To what extent is the supervision strategy and coordination of the EACEA and the Commission services (including the parent DGs, relevant horizontal services and offices) working satisfactorily? What evidence is there for an expectations gap and how can it be eliminated in the future? Additional aspect of Q.4: To what extent are the EACEAs strategy and planning aligned with its operations and indicators?
Task 3 Task 4
Q.5: To what extent has the Commission, in the presence of the EACEA, been able to maintain an adequate level of know-how in relation to the programmes entrusted to the EACEA, while focusing on its policy role? How was this achieved while avoiding overlaps?
Task 4 Q.6: To what extent have the activities of the EACEA resulted in unintended effects (both desirable and undesirable)?
Task 3 Task 4
Q.7: Evaluate the role, performance and the output of the A7 unit in the EACEA in its provision of support to EAC's policy activities in education and youth as well as to the wider education community (national policy-makers, researchers, general public
etc.).
1.2. An overall evaluation approach The third evaluation of the EACEA was based on the holistic approach to organisational
performance analysis, identifying all important elements of organisational activities and
general relationships among them. In order to organise a diagnostic and prescriptive
analysis of the Agencys activities, we selected the CAF (Common Assessment Framework)
model for developing an organisational model tailored to the third evaluation of the EACEA.
The CAF 2013 model is divided into two main sets of organisational factors: (1) enablers
dealing with the managerial practices of an organisation and (2) various types of
organisational results. We adapted this model to fit the specificity of the executive agencies
and the ToR for the third evaluation of the EACEA. First, we introduced a new set of
organisational factors (variables) related to an external regulatory framework. Second,
we made a few modifications to the enablers of the CAF model by splitting resources from
partnerships of the Agency and by integrating leadership into the factor of people. Third,
social responsibility results were replaced by programme and policy results, which are more
relevant to the ToR of this assignment. Fourth, we divided all organisational results into
intended and unintended results in accordance with the ToR.
As a result, our model for the evaluation of the EACEA includes 12 organisational factors
(variables) grouped into the three sets: (i) regulatory and operational framework, (ii)
13
May 2016
Educationand Culture
enablers and (iii) (intended and unintended) results (see Figure 1 below). The executive
agencies have a clear legal mandate, certain autonomy and are controlled by their parent
DGs of the Commission. The set of organisational factors related to the regulatory and
operational framework of the Agency includes the following items: (1) the mandate and
responsibilities of the Agency, (2) its external governance (autonomy and
control/supervision by the parent DGs), as well as (3) its organisational structure. Figure 1. Organisational model for the third evaluation of the EACEA
Source: adapted from the CAF 2013 model by PPMI.
The so-called enablers determine what the organisation does and how it executes its tasks
to achieve the desired results within the existing regulatory framework and structure.
Organisational factors 4-8 include the following items: (4) strategy and planning, (5) people,
(6) other resources, (7) partnerships and (8) processes of the Agency. Finally, following the
organisational model the agency results are broken down into the following types of
(intended and unintended) results: (9) key performance results, (10) customer-oriented
results, (11) people results and (12) programme and policy results.
The evaluation relied on the evidence gathered and analysed using qualitative and
quantitative methods:
- desk research, including literature review, analysis of the monitoring data and
comparative analysis;
- 35 interviews with EU officials, staff of the EACEA and other executive agencies, as
well as several of the Agencys beneficiaries;
- surveys of the Agencys applicants and beneficiaries with a total response rate of
23.5 %;
- a series of 15 good practice examples allowing for benchmarking of the executive
agencies activities (their fiches are presented in Annex 7 to the report);
- retrospective cost-benefit analysis (involving both quantitative and qualitative
analysis);
- methods of retrospective analysis.
Each of the evaluation methods is described in Annex 4 to the evaluation report. The
collected qualitative and quantitative data were combined in order to answer the evaluation
questions.
14
May 2016
Educationand Culture
2. Overall performance of the Agency, quantitative and qualitative CBA
2.1. Overall performance of the EACEA
2.1.1. The Agencys performance according to the main indicators
In order to evaluate the overall performance of the Agency, we introduced a conceptual
framework which links the main elements of performance (objectives, inputs, processes,
outputs and outcomes) in a linear way. This framework allowed defining the main indicators
against which the Agencys performance can be measured.
Figure 2. Framework of the Agency's performance
Source: PPMI.
First, budget efficiency is defined as the ratio between inputs (staff) and outputs (the
budget managed by the Agency). Our performance criteria measuring budget efficiency are
budget per head (million EUR) and the ratio between the administrative and operational
budget (%). Our evaluation results indicate the ratio of budget per head decreased from
EUR 1.70 million in 2012 to EUR 1.38 million in 2014 (from EUR 1.81 million to EUR 1.47
million if the EACEAs staff in A7 unit, not managing operational appropriations, are
excluded), which means that a single employee of the Agency managed a decreasing volume
of the operational budget during this period. This is related mostly to the decreasing volume
of the operational budget (especially for 2014, which was the first year of the new
programming period). The Communication to the Commission SEC(2013)493 on the
delegation of the management of the 2014-2020 programmes to the executive agencies
provided for an estimated ratio of EUR 1.97 million for 2013, but the calculation of this ratio
did not include the additional CAs financed from the third countries and EDF contributions
(see section 2.2. of the report).
Objectives Inputs Process Outputs Outcomes
Process efficiency
Budget efficiency
Cost-effectiveness
Effectiveness
Value for money
15
May 2016
Educationand Culture
Table 2. Performance indicators of the EACEA, 2012-2014, EUR million or
percentage 2012 2013 2014
Operational budget (including third countries and EDF contributions) (according to executed commitment
appropriations)
A 728 686 602
Administrative budget B 49.33 49.82 46.13
Actual number of staff at the EACEA (at the end of the year) (including CAs financed from third countries and EDF credits)
C 428 435 437
Actual number of staff at the EACEA dealing with Eurydice D 27 27 27
Programme management cost at the EACEA: ratio between
the administrative and operational budget
E=B/A 6.78 % 7.26 % 7.66 %
Budget 'per head' excluding the A7 staff F=A/(C-
D)
1.81 1.68 1.47
Source: PPMI based on the EACEAs administrative data.
Note: during the reference period there were two staff members in A7 who dealt with youth issues.
In terms of administrative efficiency (the ratio between the administrative and
operational budget), the EACEAs cost of programme management increased from about
6.8 % in 2012 to about 7.7 % in 2014 due to the somewhat lower operational budget. This
is related largely to the lower volume of the operational budget in 2013 and 2014. Taking
into account the specificity of the Agency, its operations were moderately efficient during
our reference period. However, there is a need to monitor the indicators of administrative
efficiency closely and to discuss possible actions as to how administrative efficiency could
be enhanced in the future (including a mid-term review of the programmes managed by the
Agency).
Second, process efficiency is the extent to which programme management is simplified
and optimised. The EACEA implemented a number of efficiency-enhancing measures during
the reference period, including further implementation of simplification measures,
optimisation of the business processes, harmonisation of project management requirements
across the programmes, development of IT tools, reorganisation of the organisational
structure (see part 3 of the report and its annex 3). According to the survey results, the
EACEAs applicants and beneficiaries assessed the already introduced simplifications
positively and expressed their appreciation for further simplifications. The development of
more efficient processes at the Agency will continue in 2015 as one of the priorities in its
2015 Work Programme was related to further improving programme management.
Third, cost-effectiveness is the extent to which the Agency achieved its outcomes at a
lower cost compared with in-house management by the Commission. We measured it
according to cost savings relative to the costs of in-house management by the Commission
(million EUR). The CBA exercise found that the actual savings during the 2012-2014 period
were EUR 41.8 million (22 %) compared to the in-house scenario (see section 2.2. below).
This analysis indicates that the executive agency scenario remains more cost-effective than
the in-house scenario due lower staffing costs and a lower number of staff (without
additional staff financed from the third countries and EDF contributions).
16
May 2016
Educationand Culture
Figure 3. Estimated savings of the executive agency scenario in 2012-2014, EUR
Source: PPMI based on the EACEAs data.
Fourth, effectiveness is the extent to which the Agencys objectives were achieved. At the
level of outputs, we assessed it according to the execution of KPIs based on the
administrative data (see question 3 in part 3 of the report below). Our analysis indicated
that despite the transition to the new programming period, the EACEA satisfactorily achieved
almost all of its targets during the evaluation period, with less positive results for the targets
of error rates mostly due to the nature of the programmes delegated to the Agency (small
grants awarded to small organisations with limited administrative and accounting
capacities). The EACEA did not measure any indicators related to the quality of the
evaluation and selection procedures (e.g. redress indicators used by the executive agencies
belonging to the Research and Innovation family of the Commission). The survey data
confirmed that the Agency improved in terms of timely execution of its procedures compared
to the previous evaluation period or in terms of improvement over time during the reference
period (see question 3 in Part 3 of the report below).
At the level of outcomes, we measured effectiveness according to the satisfaction of
applicants and beneficiaries with the quality of services based on our survey data (the
specific question We are satisfied with the overall quality of services provided by the
EACEA). A total of 79.3 % of the Agencys applicants and beneficiaries agreed with this
statement (70.1 % of unsuccessful applicants and 85.6 % of beneficiaries). The respondents
who applied for/participated in Media (85.4 %), Youth in Action (84.3 %) and Lifelong
Learning (82.8 %) programmes were the most satisfied with the overall quality of the
EACEAs services. In contrast, those from the Erasmus+ Charter for Higher Education
(75.5 %), Culture programme (74.7 %) or Bilateral agreements and cooperation initiatives
with industrialised countries (57.1 %; N=14) were least positive about the overall
performance of the EACEA. The average satisfaction score was 2.5 for beneficiaries and 3.4
for unsuccessful applicants with the average of 2.85 (on a scale of 1 to 9, with one being
the most positive rating possible).
The overall satisfaction of the Agencys beneficiaries was quite stable during the evaluation
period, but there was a small increase in the overall satisfaction in 2014. While the
satisfaction score of the 2012 and 2013 beneficiaries was 2.37 and 2.38 respectively, that
of the 2014 beneficiaries was 2.33. While the overall satisfaction level among the
beneficiaries and applicants remained similar compared to the previous evaluation periods,
our survey attracted a higher share of unsuccessful applicants (40 % compared to around
30 %) who tend to be less satisfied with the services of the executive agencies.
The EACEAs applicants and beneficiaries assessed the courtesy, commitment and
competence of the Agencys personnel most positively. For instance, one respondent to the
survey of the Agencys applicants and beneficiaries noted that the quality and
professionalism of the EACEA's staff was always of a very high standard. However, a more
tangible aspect of the Agency performance pertaining to the EACEAs website and
information materials was assessed least positively. The respondents also provided a
favourable opinion on the EACEAs willingness to help them and provide individualised
189 223 896147 449 107
41 774 789
0
100 000 000
200 000 000
In-house scenario Executive agency scenario Savings of EA scenario
17
May 2016
Educationand Culture
attention. Despite a good assessment of the accessibility and clarity of information provided
by the EACEA and its ability to perform the service promptly, accurately and transparently,
some aspects of the service delivery (the information provided by the EACEA is easily
accessible and the EACEAs documents are error free) were perceived in a less positive
way (see Table 3 below). Overall, this points to the possibility of improving information
provision and communications with clients in order to improve the overall satisfaction with
the services the Agency provides.
Table 3. Satisfaction with the EACEAs actual performance and the services it
provides Specific aspects of satisfaction Index
Employees in the EACEA are consistently courteous to you 1.97
The EACEAs employees are committed to doing quality work 2.11
Employees in the EACEA are knowledgeable and competent 2.15
The EACEA strives to provide excellent programme management and high-quality service
2.29
When you have a problem, the EACEA shows a sincere interest in solving it 2.33
Employees in the EACEA are always supportive in the interest of the project 2.34
The EACEA has employees who give you personal attention 2.44
Employees in the EACEA give you prompt service 2.45
Employees in the EACEA closely cooperate with you 2.52
The events (information days, project meetings, information visits, etc.) organised by the EACEA are useful
2.55
The EACEAs communications provide relevant and useful information 2.57
The EACEAs procedures are transparent and objective 2.68
The information provided by the EACEA is easily accessible 2.82
The EACEAs documents are error free 2.89
The EACEAs website and information materials are visually appealing and user-
friendly
3.06
Average 2.48
Source: the survey of the EACEAs applicants and beneficiaries.
Fifth, value for money is the optimal use of human and financial resources to achieve the
intended outcomes. Our analysis indicated that the EACEA provided good value for money
due to a combination of the following:
the Agency effectively executed the delegated tasks and achieved almost all of its
KPIs;
the agency scenario generated actual savings of EUR 41.8 million during the 2012-
2014 period according to our retrospective CBA; and
the overall satisfaction with the quality of services among its applicants and
beneficiaries was quite high (79.3 % (70.1 % of unsuccessful applicants and 85.6 %
of beneficiaries)or the average score of 2.85), the survey respondents provided a
positive assessment of the simplifications introduced and other value-adding
novelties in project management (such as kick-off meetings or support to
dissemination); there was also a small improvement in the satisfaction of the EACEAs
applicants and beneficiaries with the specific aspects of project management
compared to the previous evaluation period.
2.1.2. Quantitative benchmarking of the Commissions executive agencies
Overall, the EACEA faces challenges in reaching the budgetary efficiency ratios achieved by
other executive agencies established by the Commission. Its programme management cost
was 7.66 % and the budget per head stood at EUR 1.38 million (EUR 1.47 million without
the staff dealing with Eurydice) in 2014. The Agencys performance was more comparable
to that of CHAFEA according to the same indicators in the same year (8.97 % and EUR 1.61
18
May 2016
Educationand Culture
million respectively). In contrast, the performance of the executive agencies belonging to
the Research and Innovation family was more efficient. For instance, the REAs cost of
programme management was 3.3 % (2.1 % if the share of costs of the central
administrative and logistical support services was excluded) and its budget per head was
EUR 2.83 million (EUR 3.46 million if these services are excluded) on the same year2.
Table 4. Main performance indicators of the Commissions executive agencies in
2014 The executive agencies of the Research
and Innovation family*
Executive agency EACEA CHAFEA ERCEA REA INEA EASME
Parent DGs (other parent DGs)
DG EAC (ECHO, HOME,
CNECT)
DG SANCO (JUST, AGRI)
DG RTD DG RTD (EAC,
GROW,
CNECT, AGRI, HOME)
DG MOVE (ENER, CNECT,
RTD)
DG GROW (RTD,
ENV, CLIMA, ENER, CNECT,
MARE)
Staff in 2014 (actually filled)
437 48 388 548 (449**)
151 283
Operational budget (million EUR), commitments
602.36 77.18 1 726 1 554 4 621.18 1 291.64
Total administrative
budget (million EUR )
46.13 6.92 36.04 51.13 13.43 24.48
Proposals received in 2014
10 058 193 8 545 11 007 1 104 8 988
Total running projects (at the end of 2014)
3 664 239 4 036 6 925 516 n.a.
Programme
management cost (ratio between the administrative and operational budget)
7.66 % 8.97 % 2.09 % 3.3 %
(2.1 %**)
0.29 % 1.90 %
Budget 'per head' (million EUR )
1.38 (1.47**
*)
1.61 4.45 2.83 (3.46**)
30.69 4.65
No. of proposals 'per head'
23.0 (24.5**
*)
4.0 22.0 20.1 (24.5**)
7.3 31.8
No. of running
projects (at the end of 2014) 'per head'
8.4
(8.9***)
5.0 10.4 12.6
(15.4**)
3.4 n.a.
Source: 2014 Activity Reports of the executive agencies, DG RTD information for the Research and Innovation family. Notes: *INEA and EASME implement other programmes besides H2020; **without central support services; ***excluding the A7 staff.
2 PPMI, Evaluation of the operation of the REA, Final Report, 2016.
19
May 2016
Educationand Culture
This difference between the EACEA and the executive agencies of the Research and
Innovation family was associated mainly with the complexity and diversity of the EACEAs
programme portfolio, the heterogeneity of its applicants and beneficiaries, the quite high
volume of applications and grants, as well as their small average size (as low as
EUR 150,000)3. In terms of number of proposals per head, the EACEA would be among the
most efficient executive agencies, while in terms of running projects per head, the Agency
would be moderately efficient compared to other executive agencies (see Table 4 above on
the main performance indicators of the executive agencies in 2014).
2.1.3. Comparison between the administrative costs of the centralised and
decentralised actions of Erasmus+
We also sought to compare the administrative costs of the centralised actions of Erasmus+
executed by the EACEA and the decentralised actions of this programme carried out by the
National Agencies. The operational grants allocated to the National Agencies in the period
2007-2008 constituted 4.3 % of the decentralised funds. Exact data on national co-financing
and other sources of funding for the operation of the National Agencies were not available,
but our survey results revealed that these sources of funding on average constituted 40-
50 % of the total budget allocated to the LLP management. Therefore, it was estimated that
the total management cost of the LLP decentralised actions in the period 2007-2008
constituted around 7-8 % of the decentralised funds4.
According to the Erasmus+ 2014 report, the management cost of the centralised actions of
Erasmus+ executed by the EACEA was calculated to be 5.94 %5 in 2014 (see Table 5 below).
The EU management cost of the decentralised actions carried out by the National Agencies
was 4.12 %, but in line with the Erasmus+ regulation, the National Authorities provide
additional co-financing for the operations of their National Agencies. Exact data on national
co-financing to the operation of these agencies are not available, but if these agencies
received 40-50 % of the total administrative budget, their total cost would rise to about 7-
8 % of the decentralised funds. Although it is not possible to accurately estimate the total
management cost of the National Agencies, the EACEAs operations in the management of
the Erasmus+ actions in the 2014-2020 period are likely to be as efficient or even more
efficient compared to those of the National Agencies.
Table 5. EU commitments to the implementation of Erasmus+ in 2014 (EUR) and
its management cost (%)
Direct management
mode (EACEA)
Indirect management mode (National
Agencies)
Heading 1 260 506 438 1 396 549 350
Management fees (National Agencies) 64 565 236
Administrative expenditure 25 332 870
International Cooperation (heading 4) 140 702 760 106 174 865
Total commitments: 426 542 068 1 567 289 451
Management cost (% of the total EU
budget commitments)
5.94 % 4.12 %
Source: Annex I (Statistical Annex) to the Erasmus+ Programme Annual Report 2014, p. 5.
3 In sports the average grant size was only somewhat more than EUR 400,000 in 2014. 4 PPMI, the Interim Evaluation of the LLP (2007-2013), the Final Report, 18 February 2011, p. 129-130. 5 This can be compared to the cost of managing all the programmes delegated to the EACEA, which stood at 7.66 % in 2014.
20
May 2016
Educationand Culture
2.2. The retrospective CBA
CBA includes both quantitative and qualitative aspects of the EACEAs performance.
2.2.1. Background of the quantitative CBA
One of the key reasons for establishing executive agencies is their potential to provide a
cost-effective solution for management of the delegated programmes. Therefore, the main
objective of the quantitative retrospective CBA was to ex-post review the validity of the
reasoning and financial calculations leading to the decision to allocate management of the
specific programmes to the EACEA and to provide insights into the EACEAs overall costs
and cost-efficiency between 2012 and 2014.
For the assessment of the quantitative (financial) aspects of the retrospective CBA, we:
i. reviewed the CBA reports and other (ex-ante) working documents (such as legislative
financial statements and specific financial statements) supporting the decision to
allocate various programmes to the Agency;
ii. screened the different scenarios (comparators) foreseen in the ex-ante assessment
(namely the in-house (Commission) and the executive agency scenarios) and the
underlying assumptions, such as the EACEAs staffing estimates and composition of
staff (TAs and CAs), average staff costs and other costs (overheads and programme
support expenditure);
iii. reviewed the estimated figures used in the ex-ante assessments taking into account
the actual data;
iv. verified whether the conclusions of the ex-ante assessments are still valid when
compared to the actual situation and what the overall possible savings are:
do the actual costs (including cost of coordination and monitoring) of the EACEA
correspond to the estimates of the respective CBAs and the legal financial
statements/specific financial statements, which were prepared in view of its
establishment (prolongation of timeframe or extension of tasks)?
is the management and execution of the programme(s) by the EACEA cost-
effective compared to the alternative options (namely the respective in-house
(Commission) scenario)?
has the establishment of the EACEA resulted in savings to the EU budget as
compared to the alternative options?
As with the previous analyses, the retrospective CBA (quantitative aspects) focused on the
comparison of financial aspects and cost-efficiency, while effectiveness indicators (such as
time to select, award and pay and client satisfaction) were analysed alongside costs but not
valued in monetary terms (such analysis is presented under question 3).
The current interim evaluation of the EACEA covers the period from 2012 to 2014, which is
related to several establishment acts and the respective financial statements against which
the actual situation was tested. The most important documents for the current retrospective
CBA are listed below:
legislative financial statement related to Commission Decision 2009/336/EC of 20
April 2009 setting up the Education, Audiovisual and Culture Executive Agency for
the management of Community action in the fields of education, audiovisual and
culture in application of Council Regulation (EC) No 58/2003 (2009/336/EC),
repealing Decision 2005/56/EC, which provides the main basis for the retrospective
CBA for 2012;
financial statement related to Commission Implementing Decision 2012/797/EU of
18 December 2012 amending Commission Decision No 2009/336/EC setting up the
21
May 2016
Educationand Culture
Education, Audiovisual and Culture Executive Agency for the management of
Community action in the fields of education, audiovisual and culture in application of
Council Regulation (EC) No 58/2003, which provides the main basis for the
retrospective CBA for 2013;
specific financial statement related to Commission Implementing Decision
2013/776/EU of 18 December 2013 establishing the Education, Audiovisual and
Culture Executive Agency and repealing Decision 2009/336/EC, which provides the
main basis for the retrospective CBA for 2014.
The current mandate of the EACEA is related to the detailed Cost Benefit Analysis for the
delegation of certain tasks regarding the implementation of the Union Programmes 2014-
2020 to the executive agencies6, which was conducted in 2013 for the 2014-2020
Multiannual Financial Framework. The CBA compared the following four scenarios based on
varying levels of delegation and distribution of programmes between different executive
agencies:
the in-house scenario assuming that the new programmes would be managed by
the Commission while the executive agencies would remain responsible for the
delivery of legacy work (20072013 MFF programmes);
the initial scenario for delegation defined by the Commission;
two alternative scenarios with different options for delegation among executive
agencies (neither of these scenarios concern the EACEA).
It was concluded that alternative scenario 2 was the most efficient in terms of cost savings
and qualitative benefits. It estimated that in order to manage EUR 13,267 million (an
increase of 127 % compared to 2013), the six agencies will need 2,887 FTEs in 2020 (an
increase of 71 %, i.e. additional 1200 FTEs compared to 2013). This compares favourably
to the in-house scenario which would require 3,088 FTEs to manage the same programmes.
The executive agencies will benefit from economies of scale as they become larger.
In order to achieve further efficiency gains, the Commission proposed a few adjustments
stemming from an improved level of productivity and aimed at containing administrative
costs through a 5 % staff reduction. This corrected the ratio of budget per head in the case
of the EACEA from EUR 1.58 million to EUR 1.97 million in 2020, making it similar to the
2013 situation.
Table 6. Budget managed and human resources in the EACEA compared to all
executive agencies in 2013 and 2020
Executive Agency
Budget managed in 2013 (million EUR )
FTEs in
2013
Budget per head
2013 (million EUR )
Budget to be
managed by EA in
2020 (million EUR )
Envisioned
FTEs in 2020
Envisioned
FTEs in 2020
(adjust
ed)
Budget per
head in 2020
(millio
n EUR )
Budget per head in 2020 (million EUR )
(adjusted)
EACEA 847 431 1.97 870 552 442 1.58 1.97
Total 5 846 1 687 3.47 13 267 2 887 2 630 4.60 5.05
Source: Communication to the Commission on the delegation of the management of the 2014-2020 programmes to executive agencies (SEC(2013)493) adapted by PPMI.
6 ICF GHK in association with Technopolis, Cost Benefit Analysis for the delegation of certain tasks
regarding the implementation of Union Programmes 2014-2020 to the Executive Agencies, Final Report, 2013.
22
May 2016
Educationand Culture
These adjusted results of the CBA were used in the Specific Financial Statement (SFS)7. With
regard to the forecasts for the administrative budget, the SFS differs from the CBA in the
following aspects:
As explained above, the number of staff at the EACEA was reduced by 14 % on
average under the executive agency scenario. The in-house scenario was not
adjusted).
The average staff-related costs were increased in the SFS by 13 % on average, while
overheads were reduced by 19 % on average. The average overheads fluctuated
within the range of 10 % over the period.
The costs in the CBA were calculated in constant 2013 prices (i.e. neutralising the
effect of inflation). However, in the SFS these estimations were used as current prices
without any further indexation. In real terms, this constituted another reduction of
the administrative budget. This could have a negative impact in the final years of the
2014-2020 period, as actual average staff costs might rise due to salary indexation,
promotions and seniority, while actual overheads might rise due to inflation.
Figure 4. Number of staff estimated under different delegation scenarios
Sources: CBA, SFS.
2.2.2. Actual staffing and costs of the EACEA
During the 2012-2014 period the overall number of the EACEAs staff increased moderately
from 428 to 437 (see Figure 5). The actual staffing level was higher than the estimations
provided in the relevant financial statements, but this was related to the hiring of additional
CAs (22 CAs in 2012-2013 and 20 CAs in 2014) financed from the third countries and EDF
contributions in order to manage the additional operational budget coming from the
respective sources. Eliminating that, the actual staffing level of the Agency was below the
initial estimations of the respective financial statements based on EU funding sources. In
line with the initial estimations, CAs constituted the majority of the EACEAs staff (about
77 % compared to the initial estimations of 75 %)8.
In 2012-2013, the administrative budget of the EACEA was structured under two titles: Title
I Staff and Title II Buildings, Equipment and Miscellaneous Operating Expenditure. Since
7 Specific financial statement related to Commission Implementing Decision 2013/776/EU of 18 December 2013 establishing the Education, Audiovisual and Culture Executive Agency and repealing Decision 2009/336/EC, which provides the main basis for the retrospective CBA for 2014. 8 A higher share of CAs primarily reflected the hiring of additional CAs financed from the third country and EDF contributions.
523 532537 537
550 557
592
499 499 501 501513 520
552
436 441 441 441 441 441 442
300
350
400
450
500
550
600
650
2014 2015 2016 2017 2018 2019 2020
In-house Executive agency - CBA Executive agency - SFS
23
May 2016
Educationand Culture
2014, the structure of the EACEAs administrative budget has been aligned with the model
agreed by the six executive agencies and now comprises three titles: Title I Staff, Title II
Buildings, Equipment and Miscellaneous Operating Expenditure and Title III Programme
Support Expenditure. In order to provide a meaningful comparison, we used the actually
implemented budget figures for 2012-2014, which are presented according to the 2014
budget structure. Such budget structure also corresponds to the budget estimations in the
respective financial statements indicated above.
Figure 5. Evolution of the EACEAs staff during 2012-2014
Source: PPMI based on the EACEAs data.
Our analysis shows that in 2012-2014 the actual administrative budget implemented by the
EACEA (see Table 7 for more details) amounted to EUR 145 million and was some 3.6 %
lower than the administrative budget estimations in the respective financial statements
(EUR 151 million). It is important to note that the estimations in the respective ex ante CBAs
and financial statements were based on the EU contribution only and the EACEAs
administrative budget during the 2012-2014 period also included third country and EDF
contributions (about 6 % of the EACEAs administrative budget during our reference period)
to manage the additional operational budget coming from the respective sources. Therefore,
in 2012-2014 the EACEAs administrative budget based on the EU contribution was EUR 14.6
million (9.7 %) lower than the administrative budget forecast in the relative financial
statements.
Staff related expenditure and infrastructure and operating expenditure were close to the
estimations of the relevant financial statements. Therefore, the lower actual costs of the
Agency primarily relate to programme support expenditure, which was 22 % lower than
estimated.
The actual average staff costs for CAs were in line with the initial estimations, the average
staff costs for TAs in 2014 were higher than the CBA estimations but in line with the average
costs used in the specific financial statements. The average overhead costs in 2014 were
some 8.5 % lower than the initial estimations in the specific financial statement. However,
these costs are likely to increase once the EACEA moves to new premises.
407 431 436428 435 437406 413 415
0
100
200
300
400
500
2012 2013 2014
Financial statements
Actual
Actual without CAs financed from EEA/EFTA, 3rd country and EDF contributions
24
May 2016
Educationand Culture
Table 7. The administrative budget of the EACEA in 2012-2014, EUR Administrative budget
2012 2013 2014 2012-2014
Title I. Staff
Related Expenditure
Financial
statements
28 431 000 29 963 000 30 925 000 89 319 000
Actual 29 029 693 29 755 699 31 157 442 89 942 834
Title II. Infrastructure and
Operating Expenditure
Financial statements
10 414 000 11 065 000 10 090 000 31 569 000
Actual 11 070 973 11 652 568 9 252 271 31 975 812
Title III. Programme Support
Expenditure
Financial statements
11 288.000 11 994 000 6 527 000 29 809 000
Actual 9 224 580 8 410 756 5 717 725 23 353 061
Total Financial statements
50 133 000 53 022 000 47 542 000 150 697,000
Actual 49 325 246 49 819 023 46 127 438 145 271 707
Difference (Actual - Financial statements)
-807 754 -3 202 977 -1 414 562 -5 425 293
Administrative budget related
to EEA/EFTA, 3rd country and EDF contributions*
3 132 976 3 260 698 2 823 715 9 217 389
Difference (Actual - Administrative budget related to 3rd country and
EDF contributions Financial statements)
-3 940 730 -6 463 675 -4 238 277 -14 642 682
Source: PPMI based on financial statements, final accounts, other EACEA data. Note: Budgeted estimations.
In 2014, the overall administrative budget of the EACEA decreased by 7.4 % compared to
2013, which is primarily related to the costs of independent experts (around EUR 5-6 million annually during the 2012-2014 period). Prior to 2014, these costs were charged to the
administrative budget of the EACEA, whereas they are added to the operational budget of
the respective programmes from 2014 (as in the case of other executive agencies).
2.2.3. Cost-effectiveness of the executive agency scenario and actual savings due
to externalisation
In order to assess whether the conclusions of the ex-ante assessment (the executive
agency scenario saving estimations provided in the financial statements and ex-ante CBAs)
are still valid when compared to the actual situation and what the overall possible savings
are:
1) We draw upon the actual performance of the EACEA (actual execution of the
administrative budget, actual staffing, etc.).
2) In order to ensure comparability and validity of results, we follow the assumptions
laid down in the respective CBAs and financial statements and provide estimations of
the comparable actual in-house (Commission) scenario (comparator), which would
best reflect the actual situation.
3) Based on these estimations, we assess whether the conclusions of the ex-ante
assessment are still valid when compared to the actual situation and what the overall
possible savings are.
25
May 2016
Educationand Culture
In order to deconstruct the actual in-house (Commission) scenario (comparator), we base
our estimations on the assumptions (see Box 1 below) provided for in the financial
statements and the corresponding CBAs. For the 2012-2013 period, in order to ensure
continuity and comparability, we also take account of the practices used in the previous
retrospective CBAs9.
Box 1. Our assumptions for 2012-2013 and 2014 The assumptions for 2012-2013:
Number of staff in the Commission under the in-house scenario equals the number of staff in the EACEA under the executive agency scenario.
Staff under the Commission scenario is composed of permanent officials. The Commission staff costs and overheads correspond to DG BUDG estimations used for
2012-2013: the average cost of establishment plan posts were EUR 127 000 for 2012 and EUR 131 000 for 2013 (including overheads).
The programme support expenditure (Title III) stays the same under the Commission scenario and the executive agency scenario.
The assumptions for 2014:
Number and composition of staff in the Commission and the EACEA under the in-house
scenario correspond to the CBA and the financial statement estimations10. Also, an additional 22 CAs were added to the estimated Commission staff number in 2014 in order to reflect additional staff at the EACEA financed from the contributions of third countries and EDF in order to manage the additional operational budget which was not covered in the CBA/SFS resources calculations.
The Commission staff costs and overheads correspond to DG BUDG estimations used for
2014: establishment plan posts EUR 109 000, external personnel EUR 47 000, overheads EUR 23 000.
The EACEAs average staff costs and overheads under the in-house scenario (for legacy) correspond to the actual average staff costs of the EACEA and overheads in 2014.
The programme support expenditure (Title III) stays the same under the Commission
scenario and the executive agency scenario.
Table 8 below presents the results of analysis of the estimated actual costs of the in-house
(Commission) and the executive agency scenarios.
9 COWI, Interim Evaluation of the EACEA, 2009; Ecorys, Second Interim Evaluation of the Education, Audiovisual and Culture Executive Agency (EACEA), 2013. 10 Two different approaches to deconstruct No. of staff in the actual in-house scenario for 2014 were analysed during the evaluation. The first approach was based on the number of staff indicated in the
SFS for the in-house scenario. Under the second approach, the number of staff of the in-house scenario would be calculated taking into account the actual number of staff in the EACEA and following the assumptions used in the CBA. Although the second approach might lead to more realistic staffing estimations of the in-house scenario, corresponding costs and savings estimations would be incomparable with the estimations provided for in the SFS (the SFS adjusted staffing in the externalisation scenario, however staffing of the in-house scenario was not adjusted. This provided for
a bigger staffing gap between the externalisation and the in-house scenarios than the original CBA. Therefore, it was decided to follow the first approach.
26
May 2016
Educationand Culture
Table 8. Estimated costs of the in-house (Commission) and the executive agency
scenario, EUR
2012 2013 2014 Total 2012-
2014 No. Cost No. Cost No. Cost
In-house scenario
Commission
I. Staff related expenditure
428 54 356 000 435 56 985 000 315.7 27 585 100 138 926 100
Establishment plan posts (TA)
428 54 356 000 435 56 985 000 205.6 22 410 400 133 751 400
External personnel (CA)
0 0 0 110.1 5 174 700 5 174 700
II. Infrastructure and operating expenditure
7 261 100 7 261 100
Total Commission
cost:
428 54 356 000 435 56 985 000 315.7 34 846 200 146 187 200
EACEA
I. Staff related expenditure
- - - - 229.4 14 826 722 14 826 722
Establishment plan posts (TA)
57.3 6 279 666 6 279 666
External
personnel (CA)
172.1 8 547 057 8 547 057
II. Infrastructure and operating expenditure
- - - - 4 856 913 4 856 913
Total EACEA cost:
19 683 635 19 683 635
III. Programme support expenditure
9 224 580 8 410 756 5 717 725 23 353 061
TOTAL COST 428 63 580 580 435 65 395 756 545.1 60 247 560 189 223 896
Externalisation scenario
EACEA
I. Staff related
expenditure
428 29 029 693 435 29 755 699 437 31 157 442 89 942 834
Establishment plan posts (TA)
99 10 411 790 99 10 258 991 101 11 068 869 31 739 651
External personnel (CA)
329 15 970 798 336 16 087 928 336 16 686 875 48 745 600
Interim supportive agents and trainees
2 647 105 3 408 780 3 401 698 9 457 583
II. Infrastructure and operating expenditure
11 070 973 11 652 568 9 252 271 31 975 812
Total EACEA 428 40 100 666 435 41 408 267 40 409 713 121 918 646
Commission
I and II. Staff related expenditure, overheads
4.5 571 500 4.5 589 500 7.7 839 300 2 000 300
27
May 2016
Educationand Culture
2012 2013 2014 Total 2012-
2014 No. Cost No. Cost No. Cost
Establishment plan posts (TA)
4.5 571 500 4.5 589 500 7.7 839 300 2 000 300
External personnel (CA)
0 0
Title II. Infrastructure and operating expenditure
177 100 177 100
Total Commission
cost:
4.5 571 500 4 5 589 500 1 016 400 2 177 400
Title III. Programme support expenditure
9 224 580 8 410 756 5 717 725 23 353 061
TOTAL COST 432.5
49 896 746 439.5
50 408 523 444.7 47 143 838 147 449 107
ESTIMATED SAVINGS
-4.5 13 683 834 -4.5 14 987 233 100 13 103 722 41 774 789
Source: PPMI based on the EACEAs data.
The results of the quantitative retrospective CBA revealed that (see Table 7 and Table 8 for
more details):
The actual costs of the EACEA were lower than the initial estimates in the respective
financial statements, which were prepared in view of its establishment (prolongation
of timeframe or extension of tasks). In 2012-2014 the actual administrative budget
implemented by the EACEA(see section 2.2.2. above) amounted to EUR 136 million
based on the EU contribution and was some 9.7 % lower than the administrative
budget estimations in the respective financial statements (EUR 151 million).
The management and execution of the programme(s) by the EACEA was cost-
effective compared to the in-house (Commission) scenario. In 2012-2014 the
estimated cost savings of the executive agency scenario constituted EUR 41.8 million
(22 %) compared to the in-house scenario. The savings of the executive agency
scenario primarily resulted from a higher share of lower cost external personnel (CAs)
employed within the executive agency and a lower number of staff in 201411.
Figure 6. Estimated savings of the executive agency scenario in 2012-2014, EUR
Source: PPMI based on the EACEAs data.
11 The number of staff of the executive agency scenario estimated in the CBA was reduced by about 13 % in the EACEA, while the in-house scenario was not modified in the specific financial statement.
189 223 896
147 449 107
41 774 789
0
50 000 000
100 000 000
150 000 000
200 000 000
In-house scenario Executive agency scenario Savings of EA scenario
28
May 2016
Educationand Culture
2.2.4. Workload analysis
According to the financial statement estimations, the operational budget allocated by the
EACEA for 2012-2014 calls amounted to approximately EUR 1.8 billion (Figure 7). The
actual operational budget for the same period was EUR 2.3 billion12 and exceeded the
financial statement estimations by 24 %. This primarily resulted from additional third
country and EDF contributions, as well as from extensions of the EACEAs mandate that
were sometimes finalised later than the actual allocation of additional operating budget to
the EACEA13.
Figure 7. Operational budget related to 2012-2014 calls, EUR million
Source: Financial statements, the EACEAs Annual Work Programmes.
The Agencys workload is closely linked with the allocated operational budget, the number
of proposals received and the number of grants concluded. Our analysis showed that the
actual number of grants14 related to the 2012-2014 calls was very close to the initial
estimations provided in the financial statements (see Figure 8 below).
Figure 8. Number of grants related to 2012-2014 calls
Source: Financial statements, Annual Activity Reports of the EACEA.
12 Based on the EACEAs Annual Work Programmes. All funding sources. 13 This primarily refers to 2012 when the actual operational budget exceeded the financial statements estimations by 42 %, as the additional operational appropriations under the programmes that have already been delegated to the EACEA were already allocated to the Agency in 2012. However, the operational budget in the financial statement related to Commission Implementing Decision of 20 December 2012 (C(2012)9475 final) was revised only for 2013. 14 Without Erasmus/Erasmus+ university charters, Erasmus Mundus individual scholarships and Media actions Support to sales agents and Digitalisation of cinemas.
527752
543
1822
751 896 611
2258
0
500
1000
1500
2000
2500
2012 2013 2014 Total
FS estimations Actual (AWP)
4484 45193193
12196
4163 46113439
12213
0
2000
4000
6000
8000
10000
12000
14000
2012 2013 2014 Total
FS estimations Actual
29
May 2016
Educationand Culture
Overall, our analysis of the EACEAs main workload indicators showed that the Agencys
operational efficiency15 reached its peak in 2012-2013, primarily owing to the final years of
the 2007-2013 MFF programmes characterised by the highest operational budgets. In 2014,
the operational budget decreased as a result of transition to the new generation
programmes. This decrease was due primarily to the 2014-2020 Erasmus+ programme.
It is important to note that the operational budget of the 2014-2020 MFF programmes will
gradually increase and it is expected that in 2020 commitment credits will reach a level
similar to 2013 (which was the last year of the 2007-2013 MFF programmes). At the same
time, a decrease in the operational budget also indicates that in the early years of the 2014-
2020 MFF the initial EACEA workload will be lower compared to 2012-2013 (especially in
2015-2016, after a decline in the workload related to the 2007-2013 programmes legacy
management). This could provide an opportunity to improve the qualitative aspects of
programme implementation (such as improving monitoring arrangements both at
programme and project level and information flows between the Agency and the
Commission).
2.2.5. Qualitative aspects of the retrospective CBA
Qualitative aspects of the CBA indicated in the ToR (which reflect the CBA questions provided
in Article 3(1) of Regulation (EC) No 58/2003) were integrated into the overall evaluation
framework (the evaluation questions). This allowed a coherent presentation of the
evaluation results and conclusions in the final evaluation report.
As part of the qualitative CBA exercise we also developed a series of 15 good practice
examples corresponding to the specific CBA items. Each of the selected good practices was
presented in Annex 7 to the report in the form of case fiches and referred to Part 3 of the
report where relevant.
Table 9 below provides information on the links between the qualitative items of the CBA,
the evaluation questions and the titles of the selected good practices.
Table 9. Qualitative items of the CBA, their correspondence to the evaluation
questions and good practices
Qualitative item of the CBA Correspondence
to questions Concrete examples of good
practice
Level of financial and technical expertise in relation to the delegated tasks
Q.1 The Learning and Development Framework of the EACEA
Networks of specific cadre of staff in the REA
Flexibility in response to the needs of stakeholders in the implementation of the delegated tasks
Q.3, Q.2
Simplification of the procedures used (in particular the extent of widening the scope of use of flat-rate financing, electronic submission, remote selection procedures, etc.) and where possible,
impact of simplifications on the efficiency and effectiveness
Q.3 Development of simplified cost options in the programmed delegated to the EACEA The Working Smarter initiative of the EACEA
Proximity of delegated activities to final beneficiaries
Q.3 Organisation of joint events aimed at facilitating exchange of good practices and experience
15 Expressed as average programme expenditure, number of applications and open projects per staff and ratio between the administrative and operational budget.
30
May 2016
Educationand Culture
Qualitative item of the CBA Correspondence
to questions Concrete examples of good
practice
Introduction of the e-complaint form on the EACEA's website
Quality, relevance and timeliness of information provided to applicants and beneficiaries
Q.3 Organisation of kick-off meetings with newly selected beneficiaries
Visibility of the European Union as promoter of the programmes concerned
Q.3 Organisation or participation in promotional and information
events
Level of know-how retained inside the Commission (in particular the accessibility, adequacy, relevance, frequency and timeliness of information
provided to the parent DGs)
Q.5, Q.4 and Q.3
Clear delineation of tasks, clarity of roles, and lack of duplication
Q.1 and Q.4 ICE+ working arrangement
Transparency of the evaluation and selection process
Q.3 Use of IT tools for online briefing of external experts in the EACEA Monitoring quality and consistency of results produced by the external experts and harmonisation of the evaluation results
Exploitation and impact of results achieved through the activities managed by the EACEA
Q.3 and Q.5 The VALOR dissemination platform Common monitoring framework and tools introduced by DG RTD
Reporting, feed-back and cooperating to/with parent DG
Q.4, Q.5 and Q.3 The monitoring framework for EU Aid Volunteers developed by DG ECHO in cooperation with the
Agency Workload measurement process implemented by the REA
Application of the Commission's data protection and transparency rules
Q.4
During the interim period, we also reviewed the operation of the Commissions executive
agencies in the 2012-2014 period based on their Activity Reports and the available
evaluation reports. The purpose of this review was to identify recent changes to the
operation of the Commissions executive agencies and to compare them with the
performance of the EACEA.
Our review indicated that after the last extension of their mandates, the Commissions
executive agencies undertook quite similar changes aimed at increasing their efficiency and
effectiveness. First, the executive agencies introduced simplifications and optimised
their business processes (e.g. simplifications of grant management in the REA and the
ERCEA, simplification of small grants in CHAFEA or streamlining the evaluation process in
INEA), reducing administrative burden for the beneficiaries and/or increasing efficiency at
the agency level.
Second, some of the executive agencies carried out actions aimed at harmonising
management arrangements across different programmes (e.g. INEA harmonised the
administrative and technical management of all three CEF sectorial programmes delegated
it). The introduction of the common rules and IT tools provided by the Common Support
Centre contributed to an increased harmonisation of project management practices in the
Research and Innovation family (with the REA providing central administrative and logistical
31
May 2016
Educationand Culture
support services to the executive agencies, in particular for the evaluation of proposals,
management of the expert evaluators and the validation of beneficiaries' legal data).
Third, the executive agencies continued developing IT tools for application and grant
management. Corporate IT tools of the Research and Innovation family were implemented
in all the executive agencies involved in the implementation of Horizon 2020 and extended
to other executive agencies outside this family (to the EACEA and CHAFEA). Substantial
savings were expected to be generated after the introduction of these tools in CHAFEA.
Fourth, most of the executive agencies reorganised their internal tasks. For instance,
the REA and the ERCEA adapted their organisational structures to the new mandates, while
EASME regrouped financial tasks in one unit and created a centralised procurement cell
providing advice, helping the operational units with public procurement procedures and
dealing with service contracts.
Fifth, some of the executive agencies were engaged in the development of public
databases on supported projects and their results. The REA and the ERCEA provided
data to the CORDA, which is the internal data warehouse for the Commission services and
is also used to inform the EU Member States about their participation in the framework
programmes through the e-CORDA platform. Also, information on all EU-funded research
and innovation projects and their results was disseminated publicly through CORDIS, the
Commissions public repository. CHAFEA developed a public database providing information
about various projects and actions supported by the previous EU Public Health Programme
and the current EU Health Programme. The database is useful for increasing visibility and
demonstrating the results of co-financed projects to both parent DGs and the general public.
Our evaluation findings (see Part 3 of the report below) indicate that all of these changes
also occurred in the EACEA during the 2012-2014 period. Also, the Agency developed a
number of concrete cases of good practice, which are presented in the form of fiches in
Annex 7 to the report. Overall, this evidence proves that the EACEA was results orientated
and engaged in continuous learning in order to create innovation and improve its
business operations during the period 2012-2014.
32
May 2016
Educationand Culture
3. Evaluation findings
3.1. Question 1
Q.1: To what extent are the EACEAs organisation and internal structures appropriate to its new profile and to the needs it is intended to meet? What is the potential impact of the new HR policy on seconded officials, intended to maintain closer links between
these officials and the parent DGs, on cooperation between the parent DGs and the Agency? To what extent has the EACEA retained and/or improved the competence and efficiency of its staff in relation to the implementation of current and new tasks?
This evaluation question addresses the issues of organisational structure and human
resource management.
Organisational structure refers to the way in which an organisation is structured, i.e. the
division of line and staff tasks, formal chains of communication between management and
staff and the way tasks and responsibilities are divided throughout the organisation. Since
executive agencies manage EU programmes on behalf of the Commission and under its close
supervision, close relationships between the Commission and the agencies should be
reflected in the administrative and HR arrangements. Therefore, the organisational structure
of the EACEA should clearly reflect the EU programmes for which the Agency is responsible.
In this evaluation, the key issues related to organisational structure involve the extent to
which it is aligned with the tasks entrusted to the Agency, the extent to which it is clearly
documented, communicated and easy to understand. The size of the organisation and its
adequacy is also an important aspect. It is relevant to note that no shortcomings on these
issues were found in the previous evaluations.
Human resource management (HRM) involves recruitment, management and
development of people who work for an organisation. For HRM analysis, we followed three
sub-criteria of the CAF 2013 model as to how the EACEA:
plans, manages and improves its human resources transparently with regard to
strategy and planning;
identifies, develops and uses competencies of people aligning individual and
organisational goals;
Involves employees by developing open dialogue and empowerment, supporting
their well-being.
These issues are relevant to the evaluation, as a high level of staff turnover, limited staff
mobility and moderate levels of staff satisfaction may have a negative effect on the
organisational performance.
1.1. To what extent is the organisational structure aligned with the tasks entrusted to the
Agency? Is it appropriate to its new mandate and to the needs it is intended to meet?
The overall finding of our analysis is that the organisational structure of the Agency was well
aligned with the tasks delegated to the EACEA and matched well the EU programmes and
actions managed by the Agency. In addition, the appointment of the new Head of
33
May 2016
Educationand Culture
Department16 for Erasmus+ was a positive development that improved the coordination of
programme implementation.
Before our reference period, the organisational structure of the EACEA had been de facto
very flat. Up to 2013, there was a prolonged interim period when the same person acted as
both Director and Head of Department. Before that, there were no departments and all
heads of units were directly responsible to the Director of the Agency. After 2013, a separate
person as a head of department became responsible for seven units dealing with the
Erasmus+ programme and the EU Aid Volunteers programme, while the Director remained
in control of the Creative Europe and Europe for Citizens programme units, as well as units
dealing with the administrative functions. As a result of this reorganisation, the
organisational structure was better adapted to the new programmes delegated to the Agency
and corresponded well to the actual workload of their management. The roles and
responsibilities were clearly allocated within the Agencys organisational structure.
The reorganisation also improved management of this programme and made its
implementation more homogenous at the Agency level. In addition, as a result of this
reorganisation, the Agency was able to speak with one voice in various working groups,
which improved collaboration between the EACEA and the parent DGs. This improved
coordination in the implementation of Erasmus+ and facilitated standardisation of its
implementation modalities (see evaluation question 4).
Despite appointing a new Head of Department for Erasmus+ units, the o