109
EVALUATION OF THE OPERATION OF ERCEA (2012-2015) FINAL REPORT
EVALUATION
OF THE OPERATION OF ERCEA
(2012-2015)
FINAL REPORT
EUROPEAN COMMISSION
Directorate-General for Research and Innovation
Directorate R — Resources
Unit R4 – New Management Modes
E-mail: [email protected]
European Commission
B-1049 Brussels
mailto:[email protected]
EUROPEAN COMMISSION
EVALUATION
OF THE OPERATION OF ERCEA
(2012-2015)
FINAL REPORT
Prepared by: Public Policy and Management Institute
Directorate-General for Research and Innovation
2016
LEGAL NOTICE
Neither the European Commission nor any person acting on behalf of the Commission is
responsible for the use which might be made of the following information.
The views expressed in this publication are the sole responsibility of the author and do not
necessarily reflect the views of the European Commission.
More information on the European Union is available on the Internet (http://europa.eu).
Luxembourg: Publications Office of the European Union, 2016.
PDF ISBN 978-92-79-57958-5 doi:10.2777/75945 KI-01-16-395-EN-N
© European Union, 2016
Reproduction is authorised provided the source is acknowledged.
EUROPE DIRECT is a service to help you find answers
to your questions about the European Union
Freephone number (*):
00 800 6 7 8 9 10 11
(*) Certain mobile telephone operators do not allow access to 00 800 numbers
or these calls may be billed
CONTENTS
List of tables and figures ............................................................................................................................................................. 7
List of abbreviations ........................................................................................................................................................................ 9
Introduction ................................................................................................................................................................................ 11
1. Evaluation objectives and methodology ...................................................................................................... 12
1.1. Objectives and scope of the evaluation .......................................................................................... 12
1.2. Methodology ......................................................................................................................................................... 12
1.2.1. Proposed organisational model for the evaluation of the ERCEA ................. 12
1.2.2. Main evaluation activities and methods............................................................................ 15
2. Background: regulatory framework, ERCEA’s mission and governance ............................ 17
2.1. Framework of EU executive agencies and ERCEA’s role in it ........................................ 17
2.1.1. Overall framework of EU executive agencies: overview of the context
and performance to date ................................................................................................................................. 17
2.1.2. Coherence of management activities in FP7 & Horizon 2020 .......................... 19
2.2. Evolution of the ERCEA in FP7 and Horizon 2020 .................................................................. 22
2.2.1. Governance structure of the ERCEA: interplay between ERC, the
Commission and the Agency ........................................................................................................................ 22
2.2.2. Establishment and operation of the ERCEA under FP7 ......................................... 24
2.2.3. Operation of the ERCEA in Horizon 2020 ........................................................................ 27
2.3. Management structure of the ERCEA ............................................................................................... 28
3. Evaluation of ERCEA’s performance in 2012-2015 ............................................................................. 31
3.1. Effectiveness of ERCEA ................................................................................................................................ 31
3.2. Operational efficiency ................................................................................................................................... 45
3.3. ERCEA’s utility .................................................................................................................................................... 64
3.4. Maintaining an adequate level of know-how inside the Commission ...................... 71
4. Retrospective cost-benefit analysis ................................................................................................................. 73
4.1. Quantitative aspects of the retrospective CBA ......................................................................... 73
4.1.1. Quantitative CBA for the 2012-2013 period ................................................................. 74
4.1.2. Quantitative CBA for the 2014-2015 period ................................................................. 82
4.1.1. Quantitative CBA for the 2014-2015 period ................................................................. 89
4.2. Qualitative aspects of the retrospective CBA ............................................................................. 92
6
4.3. Recommendations how to improve the quality of the CBA............................................. 94
5. Addressing the lessons learned from previous external evaluation of ERCEA (task
4) ..................................................................................................................................................................................................... 96
6. Conclusions and recommendations ................................................................................................................. 99
6.1. Overall conclusions ......................................................................................................................................... 99
6.2. Specific conclusions and recommendations ............................................................................. 102
6.2.1. Task 2.1: effectiveness of the ERCEA ............................................................................. 102
6.2.2. Task 2.2: operational efficiency of the ERCEA ......................................................... 103
6.2.3. Tasks 2.3 and 2.4: ERCEA’s utility and maintenance of an adequate
level of know-how in the Commission ............................................................................................... 105
6.2.4. Task 3: retrospective cost-benefit analysis ................................................................ 106
7
LIST OF TABLES AND FIGURES
List of Tables
Table 1. Relationship between the factors of the organisational model and the ToR questions .............. 14
Table 2. Key evaluation activities ................................................................................................. 16
Table 3. Achievement of Specific Objective 1 ................................................................................. 35
Table 4. Based on your experience, to what extent do you agree with the following statements
concerning the Agency’s actual performance and the services it provides? ........................................ 38
Table 5. Average number of training days attended by staff, 2013-2014 ........................................... 62
Table 6. Staff Engagement Index in ERCEA and other Commission services, 2013-2014 ..................... 62
Table 7. Mechanisms and instruments to ensure adequate coordination and information flow between the
ERCEA and the Commission services ............................................................................................ 68
Table 8. Types of know-how related to programme implementation and their relevance to the
Commission .............................................................................................................................. 71
Table 9. Assumptions used in ex-ante CBA and LFS (year 2013) ...................................................... 75
Table 10. ERCEA’s administrative budget in 2012-2013, EUR .......................................................... 76
Table 11. Staff and infrastructure costs in 2012-2013, EUR ............................................................. 77
Table 12. Administrative expenditure 2012-2013, EUR ................................................................... 77
Table 13. Other management expenditure in 2012-2013, EUR ......................................................... 78
Table 14. Estimated and actual costs of the in-house (Commission) and the executive agency scenarios,
EUR ......................................................................................................................................... 79
Table 15. Average grant size in 2009-2013, EUR million ................................................................ 81
Table 16. Budget managed and human resources in the ERCEA compared to all executive agencies in
2013 and 2020 .......................................................................................................................... 83
Table 17. Assumptions used in ex-ante CBA and LFS ...................................................................... 84
Table 18. ERCEA’s administrative budget 2014-2015, EUR .............................................................. 85
Table 19. Programme support expenditure 2014-2020, EUR ............................................................ 86
Table 20. Estimated and actual costs of the in-house (Commission) and the executive agency scenarios,
EUR ......................................................................................................................................... 87
Table 21. Average grant size in 2014 calls, EUR , million ................................................................ 90
Table 22. Qualitative aspects of the CBA and their correspondence to the evaluation tasks ................. 92
Table 23. Performance indicators of ERCEA, 2012-2014, EUR million or percentage ........................ 100
Table 24. Recommendations made in the second evaluation of ERCEA and their implementation ......... 96
List of Figures
Figure 1. Organisational model for the evaluation of the ERCEA ....................................................... 13
Figure 2. Life cycle of ERCEA’s key activities.................................................................................. 15
Figure 3. Lifeline of the executive agencies ................................................................................... 17
Figure 4. Landscape of management bodies in FP7 (excl. Euratom & JRC) ......................................... 20
Figure 5. Landscape of management bodies in Horizon 2020 (excl. Euratom) ................................... 21
Figure 6. Governance components of ERC and their interactions ...................................................... 24
Figure 7. ERCEA's evolution since its inception: number of calls launched and proposals received ........ 25
Figure 8. ERCEA’s staff from 2008 until 2014 ................................................................................ 26
Figure 9. Total number of grants signed by the ERCEA from 2009-2014 and total commitment credits as
foreseen in the work programmes ................................................................................................ 26
Figure 10. Organisational structure of ERCEA ................................................................................ 30
Figure 11. Change in the overall satisfaction with the Agency in 2011-2014 ...................................... 37
Figure 12. Beneficiaries’ assessment of ERCEA’s performance and service delivery ............................. 39
Figure 13. Satisfaction of beneficiaries with ERCEA’s external communication/responsiveness ............. 42
Figure 14. Independent experts’ assessment of the quality of information/assistance provided by the
ERCEA during the registration process .......................................................................................... 43
Figure 15. Independent experts’ assessment of the quality of communication with the ERCEA during their
work with the Agency ................................................................................................................ 43
file:///C:/Users/v.stanciauskas/Desktop/ERCEA%20deliverable%20Final%20report%20+%20Annex%205.docx%23_Toc445307801
8
Figure 16. Beneficiaries’ awareness of EU funding and the role of the EC/ERC in ERC grants ............... 44
Figure 17. Beneficiaries’ awareness/knowledge about ERC grants/funding......................................... 45
Figure 18. ERCEA's evolution of Time-to-Inform ............................................................................ 46
Figure 19. ERCEA's evolution of Time-to-Sign ................................................................................ 47
Figure 20. ERCEA's evolution of Time-to-Grant .............................................................................. 48
Figure 21. ERCEA's evolution of average Time-to-Pay ..................................................................... 49
Figure 22. ERCEA's evolution of average grant amendment time...................................................... 50
Figure 23. Error rate supporting the declaration of assurance .......................................................... 52
Figure 24. Ratio between ERCEA’s administrative and operational budget ......................................... 52
Figure 25. Satisfaction of beneficiaries with the application process .................................................. 53
Figure 26. Redress ..................................................................................................................... 54
Figure 27. Extent to which experts agree with the related statements on the selection and contracting
process..................................................................................................................................... 54
Figure 28. Extent to which experts agree with the related statements on the management of the experts’
work and communication with the ERCEA ..................................................................................... 55
Figure 29. Extent to which independent experts agree with the related statements on the payments
made by ERCEA ......................................................................................................................... 56
Figure 30. Extent to which independent experts are willing to work with the EARCEA in the future ....... 56
Figure 31. Satisfaction of beneficiaries with the contracting process ................................................. 57
Figure 32. Satisfaction of the beneficiaries with the performance of ERCEA in relation to its KPIs related
to evaluation, selection and contracting process ............................................................................ 57
Figure 33. Satisfaction of beneficiaries with the project implementation phase................................... 58
Figure 34. Satisfaction of beneficiaries with the performance of ERCEA in relation to its KPIs related to
processing grant amendments and payment process ...................................................................... 59
Figure 35. Estimated savings of the executive agency scenario in 2012-2015, EUR ............................ 64
Figure 36. Delimitation of responsibilities between the ERCEA, the ERC Scientific Council and the
European Commission ................................................................................................................ 66
Figure 37. Respondents' awareness of the ERC brand ..................................................................... 70
Figure 38. Evolution of ERCEA’s staff between 2008 and 2013 ......................................................... 76
Figure 39. Planned versus actual total costs under the in-house and externalisation scenarios in 2012-
2013 ........................................................................................................................................ 80
Figure 40. Operational budget 2009-2013, EUR million .................................................................. 81
Figure 41. Number of grants in 2009-2013 ................................................................................... 82
Figure 42. Number of staff estimated under different delegation scenarios ........................................ 83
Figure 43. Estimated savings of the executive agency scenario in 2014-2015, EUR ............................ 89
Figure 44. Operational budget 2014-2015 calls, EUR million ........................................................... 90
Figure 45. Number of grants in 2014-2015 calls ............................................................................ 91
Figure 46. Framework of the Agency's performance ....................................................................... 99
Figure 47. Estimated savings of the executive agency scenario in 2012-2015, EUR .......................... 107
file:///C:/Users/v.stanciauskas/Desktop/ERCEA%20deliverable%20Final%20report%20+%20Annex%205.docx%23_Toc445307831file:///C:/Users/v.stanciauskas/Desktop/ERCEA%20deliverable%20Final%20report%20+%20Annex%205.docx%23_Toc445307831
9
LIST OF ABBREVIATIONS
AAR Annual Activity Report
ABAC Accrual Based Accounting
AD Administrators
AdG ERC Advanced Grants
ANOVA Analysis of variance
AWP Annual Work Programme
CA Contract Agents
CAF Common Assessment Framework
CBA cost-benefit analysis
CEOS Conditions of Employment of Other Servants
CFS Central Financial Service
CHAFEA Consumers, Health and Food Executive Agency
CoG ERC Consolidator Grants
CRaS Common Representative Audit Sample
CRIG Communicating Research Implementation Group
CSA Coordination and Support Actions
CV curriculum vitae
DG BUDG Directorate-General for Budget
DG HR Directorate-General Human Resources and Security
DG RTD Directorate-General for Research & Innovation
DG Directorate-General
DIGIT Directorate-General for Informatics
DIR Directorate
EA Executive Agency
EACEA Education, Audiovisual and Culture Executive Agency
EACI Executive Agency for Competitiveness and Innovation
EAHC Executive Agency for Health and Consumers
EASME Executive Agency for Small and Medium-sized enterprises
EC European Commission
ECA European Court of Auditors
EEA European Economic Area
EFTA European Free Trade Association
EIB European Investment Bank
EIF European Investment Fund
ENV Environment programme under FP7
ERA European Research Area
ERC European Research Council
ERCEA European Research Council Executive Agency
EU European Union
Euratom European Atomic Energy Community
FP Framework Programme
FP7 7th Framework Programme
FTE Full time employees
H2020 Horizon 2020
HR Human Resources
IAC Internal Audit Capability
IAO Internal Audit Office
IAPP Marie Curie Industry-Academia Partnerships and Pathways
IAS Internal Audit Service
IEEA Intelligent Energy Executive Agency
INEA Innovation & Networks Executive Agency
IPR Intellectual property rights
IRSES Marie Curie International Research Staff Exchange Scheme
10
IT Information Technology
ITN Marie Curie Initial Training Network
ITSC Information Technology Steering Committee
JRC Joint Research Centre
KBBE knowledge-based bio-economy
KPI key performance indicator
LFS Legislative Financial Statement
MFF Multiannual financial framework
NCP Nacional Contact Point
NIGHT European Researchers’ Night
NMP Nanosciences, nanotechnologies, materials and new production technologies programme under
FP7.
OIB Office for Infrastructure and Logistics in Brussels
PHEA Public Health Executive Agency
PoC ERC Proof-of-concept Grants
PPMI Public Policy and Management Institute
R&D research and development
REA Research Executive Agency
ScC Scientific Council
SFS Specific Financial Statement
SME small or medium enterprises
SNE seconded national experts
SP Strategic Programme
StC Standing Committee
StG ERC Starting Grants
SyG ERC Synergy Grants
TA temporary agents
TEN-T EA Trans-European Transport Network Executive Agency
ToR Terms of Reference
TTG time to grant
TTI time to inform
TTP time to pay
WG Working Group
11
INTRODUCTION
This final report is submitted for the implementation of specific contract No 30-CE-0717390/00-20 ‘Evaluation
of the operation of ERCEA (2012-2015)’, implementing the framework contract for the provision of services to
the Commission in the field of evaluation of research and innovation programmes and policies (2012/S 144-
240132). The evaluation was carried out by the Public Policy and Management Institute (Lithuania). This
report was produced on the basis of the Terms of Reference, the inception report, the interim report, the draft
final report and comments and suggestions from the Steering Group.
As specified in the Terms of Reference the report is a comprehensive account of all the evaluation tasks.
Following this logic the report is divided into the following parts:
The background part, which describes the Agency’s regulatory framework, mission and evolution during 2012-2015 (covering task 1);
Key findings on ERCEA’s effectiveness, operational efficiency, utility and the Commission’s ability to maintain a sufficient know-how of the programmes delegated to the Agency (covering task 2);
Key findings of our retrospective cost-benefit analysis, covering aspects related to task 3;
Conclusions and recommendations, including the lessons learned in the implementation of FP7 and Horizon 2020 programmes (task 5);
Detailed annexes describing our methodological approach and presenting the interview and survey questionnaires with aggregated data, as well as our assessment of ERCEA’s implementation of the
recommendations made in the previous evaluation (task 4).
12
1. EVALUATION OBJECTIVES AND
METHODOLOGY
1.1. Objectives and scope of the evaluation
The main objective of this evaluation was to assess ERCEA’s implementation of the Ideas programme of the
Seventh Framework programme and the Strengthening frontier research through the activities of the
European Research Council part of the Excellent Science section of Horizon 2020. The evaluation assessed
whether the creation of the Agency has yielded the expected results and what could be done in the future to
further improve the situation.
The scope of the evaluation was to evaluate the effectiveness of the implementation of the Ideas programme
of the Seventh Framework programme and the European Research Council part of the Excellent Science
section of Horizon 2020 by ERCEA. The assignment did not focus on the scientific achievements of these
programmes.
The report covers the activities of the ERCEA from July 2012 to July 2015. The study thus covered two
different legal frameworks and two different programing periods – 18 months of FP7 implementation and 18
months of Horizon 2020 implementation.
According to the Terms of Reference the evaluation was organised into five tasks:
1. provision of a synthetic description of ERCEA’s regulatory framework, mission and governance
structure;
2. assessment of the Agency’s performance in 2012-2015, including its effectiveness, operational
efficiency, utility and maintenance of an adequate level of know-how inside the Commission;
3. cost-benefit analysis;
4. assessment of the implementation by the ERCEA of the actions identified in its action plan in response
to the Agency‘s previous external evaluation;
5. drawing of conclusions and provision of recommendations for the future on the basis of the evaluation
questions.
Below we present our overall approach to the assignment and describe the key assessment evaluation
methods used to cover the tasks described above.
1.2. Methodology
1.2.1. Proposed organisational model for the evaluation of the ERCEA
The external evaluation of the ERCEA was based on the holistic approach to organisational performance
analysis, identifying all the important elements of the Agency’s activities and general relationships among
them. In order to organise a diagnostic and prescriptive analysis of the Agency’s activities, sets of factors
(variables) on the context of an organisation, the enablers (causes) and results of organisational performance
should be developed and linked together in a single framework. After reviewing various organisational
diagnostic tools designed to assess organisational performance and changes, the evaluation team adopted the
CAF (Common Assessment Framework) model for the development of an organisational model tailored to the
evaluation of the ERCEA.
The CAF model is an organisation management tool inspired by the Excellence Model of the European
Foundation for Quality Management. The CAF is based on the assumption that ‘excellent results in
organisational performance, citizens/customers, people and society are achieved through leadership driving
13
strategy and planning, people, partnerships, resources and processes’1. The CAF was developed as a result of
cooperation among the EU Ministers responsible for Public Administration. This tool is promoted within the
European Public Administration Network, which is an informal gathering of the Directors General responsible
for Public Administration in the EU Member States, the European Commission and observer countries. The
most recent version of the CAF 2013 model was selected for the development of our organisational model.
The CAF 2013 model is divided into two main sets of organisational factors: (1) enablers dealing with the
managerial practices of an organisation and (2) various types of organisational results. We adapted this
model to fit the specificity of the executive agencies and the ToR. First, we introduced a new set of
organisational factors (variables) related to an external regulatory framework. Second, we made a few
modifications to the enablers of the CAF model by splitting resources from partnerships of the Agency and
by integrating leadership into the factor of people. Third, social responsibility results were replaced by policy
results, which are more relevant to the ToR of this assignment. Fourth, we divided all organisational results
into intended and unintended results in accordance with the ToR.
As a result, our model for the evaluation of the ERCEA included a number of organisational factors grouped
into the following three sets: (i) regulatory and operational framework, (ii) enablers and (iii) (intended and
unintended) results.
Figure 1. Organisational model for the evaluation of the ERCEA
Source: adapted from the CAF 2013 model by PPMI.
The set of organisational factors related to the regulatory and operational framework of the Agency
include the following items: (1) the mandate and responsibilities of the Agency, (2) its external governance
(autonomy and control/supervision by the parent DGs), as well as (3) its organisational structure. Since the
Agency’s working environment is specific and legally bound by Commission Regulations, external environment
factors such as market forces or industry trends are not relevant influences.
Enablers determine what the organisation does and how it executes its tasks to achieve the desired results
within the existing regulatory framework and structure. These factors include the following items: (4) strategy
and planning, (5) people, (6) other resources, (7) partnerships and (8) processes of the Agency.
Strategy and planning addresses the mission of the organisation and strategic documents targeted at implementing this mission.
The concept of people represents all human resource (HR) management related issues (HR policy, talent management, training and development, etc.), whereas all other resources of the organisation
can be structured into information, knowledge, finances and technology.
Partnerships encompass ERCEA’s relations and communication with its parent DG, other executive agencies and other stakeholders. All information flows which do not correspond to reporting
requirements fall under this category.
1 CAF Resource Centre of the European Institute of Public Administration, CAF 2013: Improving Public Organisations through Self-Assessment,
p. 9.
14
Processes, services and products are defined as follows: (1) core processes, realising the mission and strategy of the institution and thus critical to the delivery of products or services, (2)
management processes, steering the organisation, (3) support processes, delivering the necessary
resources.
Following the CAF 2013 model the Agency results are broken down into the following types of (intended and
unintended) results: (9) key performance results, (10) customer-oriented results, (11) people results and (12)
policy results.
key performance results represent financial and non-financial performance of the Agency, i.e. trends in ERCEA’s key performance and financial indicators (achieved/partly achieved/not
achieved);
customer-related results mirror the perception of the Agency’s work by its clients (applicants and beneficiaries);
people results address several perspectives: a) skills and abilities of ERCEA staff, and b) know-how at the Commission;
policy results are directly linked to the core processes of the Agency and correspond to ERCEA’s contributions to better evidence-based policy-making.
Table 1 specifies the relationship between the factors of the organisational model and the questions of the
ToR, illustrating how the integrated approach was followed during this evaluation. Evidence for assessing the
factors of the organisational model was gathered by applying the evaluation methods indicated in section
1.2.2 and Annex 1 of the report.
Table 1. Relationship between the factors of the organisational model and the ToR questions
Number and title of the
organisational factor
Key aspects to be scrutinised & the related tasks
1: Mandate and responsibilities
Match between the actual operations of ERCEA and its mandate, as well as the
instrument of delegation (task 1, task 2.1)
Delimitation of responsibilities and tasks between ERCEA and the parent DG
(task 2.3, task 2.4)
2: External governance (agency
autonomy and control/supervision by the
parent DGs)
Balancing the autonomy of the Agency with the need for policy coherence and
unified external communication (task 2.1, task 2.3, task 2.4)
The supervision strategy and coordination of ERCEA and the Commission
services (including the parent DG, relevant horizontal services and offices)
(task 2.4)
3: Organisational structure Appropriateness of ERCEA’s organisation and internal structures to its new
profile and the needs it faces (task 1, task 2.2, task 3)
4: Strategy and planning (priorities,
objectives and indicators of the Agency)
Evaluation of the organisational alignment (process of ensuring all aspects of
the organisation are aligned with the realisation of its strategy – operationally
(the organisation’s ability to deliver its ‘mission’) and strategically (the
management of the organisation to achieve its vision) (task 2.1).
5: People (human resources of the
Agency)
The potential impact of the new HR policy of seconded officials, dealing with the
inherent challenges of HR management (staff turnover, mobility), the provision
of adequate coaching and training (task 2.2, task 3)
6: Resources (IT, financial resources,
experts of the Agency)
Effective IT applications, websites, and other collaborative tools; effective and
efficient budget implementation; (task 2.2, task 3)
7: Partnerships (cooperation with the
parent DG, other executive agencies and
stakeholders)
Cooperation between the parent DGs and the Agency; sharing of best practice
with other Agencies; Adequacy of coordination and information flows between
the Agency and the Commission services (task 2.4)
8: Processes, services and products of
the Agency
Operations of the ERCEA & their effectiveness; effective management of the
programmes and better services to the stakeholders; timely provision of
relevant information, adequate, up to date and comprehensive information
(task 2.2, task 3)
9: Key performance results Effective management of the programmes and better services to the
stakeholders (tasks 2, 3)
10: Customer-oriented results Performing the tasks according to the expectations of policy stakeholders and
beneficiaries; Satisfaction of parent DG with the selection process, follow-up
and monitoring, proximity to addressees (task 2.1)
11: People results Competence and efficiency of the Agency staff; level of know-how retained
inside the Commission (task 2.2, task 2.4)
12: Policy results Policy responsiveness, policy coherence and unified external communication;
feeding into the development of better policy, visibility of EU as promoter of the
programmes (task 2.1, task 2.4)
15
The adopted CAF model closely followed key stages of the Agency’s project management life cycle. In the
model a distinction was made between three types of processes, including: 1) core processes, realising the
mandate and responsibilities of the Agency and thus critical to the delivery of products or services; 2)
management processes, steering and controlling the organisation; and 3) support processes, using the
available resources.
Figure 2. Life cycle of ERCEA’s key activities
Source: PPMI based on desk research.
1.2.2. Main evaluation activities and methods
This section presents our detailed approach to the key data collection and analysis methods that were applied
during the project execution. The methods are summarised in the table below and include:
extensive documentary review and desk research;
statistical analysis of survey data;
semi-structured in-depth interviews with stakeholders;
surveys;
cost-benefit analysis.
16
Table 2. Key evaluation activities
Data collection or
analysis method
Activities foreseen
during the inception
phase (month 1 of the
assignment):
finalisation of the
methodology
Activities foreseen during
the first interim phase
(months 2-3 of the
assignment): data
collection and their
preliminary analysis
Activities foreseen during the
second interim & finalisation
phases (months 4-7 of the
assignment): triangulation of
findings & final reporting
Desk research &
documentary review
Preliminary desk research
and documentary review
Desk research and
documentary review (as
required for tasks 2 & 3 of the
assignment)
Desk research and documentary
review (as required for all remaining
tasks of the assignment)
Statistical analysis of
survey data
Data gathering,
assessment of data quality
and gaps
Statistical analysis of the data
gathered (as required for tasks
2 & 3 of the assignment)
Statistical analysis of the data
gathered (as required for all
remaining tasks of the assignment)
Semi-structured in-
depth interviews
Exploratory interviews with
ERCEA’s officials and its
key stakeholders; testing
of interview questionnaires
Execution of interviews
required for completion of
tasks 2 & 3
Execution of any additional
interviews to validate the findings,
conclusions and recommendations
Surveys No activities implemented Survey of ERCEA’s
independent experts
Survey of the project
beneficiaries
Preliminary analysis of survey
results
Analysis of survey results,
triangulation of data and findings
Cost-benefit analysis Collection of relevant data
and assessment of their
quality
Execution of cost-benefit
analysis
Reporting & integration of CBA
results into the report
Final reporting & integration of CBA
results into the report
Both qualitative and quantitative data and information were used to answer the evaluation questions through
the adoption of the mixed method design and triangulation of findings. Annex 1 presents our detailed
approach to key data collection and analysis methods.
17
2. BACKGROUND: REGULATORY FRAMEWORK,
ERCEA’S MISSION AND GOVERNANCE 2.1. Framework of EU executive agencies and ERCEA’s role in it
2.1.1. Overall framework of EU executive agencies: overview of the context and
performance to date
The EU executive agencies were established by the European Commission on a temporary basis, following the
reform of the Commission which took place in 2000. In order for the Commission to refocus its resources on
its core functions, it was intended to externalise some of the Commissions’ management tasks in two ways:
by delegating executive responsibilities either to Community public bodies (devolution) or to national public
bodies (decentralisation). As a consequence of the first approach, the Commission created the executive
agencies whose main objective was to manage EU programmes, and whose existence was usually limited to a
specified time frame.
Council Regulation (EC) No 58/2003 of 19 December 2002 laid out the statute for all the EU executive
agencies, regulating their tasks, structure, operation, budget system, staff, supervision and responsibility.
The Regulation stipulated that in order to achieve their main objective successfully – management of one or
more EU programmes – the executive agencies are entrusted with the following key tasks:
managing some or all of the phases in the lifetime of a project ‘life cycle’, in relation to specific individual projects, in the context of implementing a Community programme and carrying out the
necessary checks to that end, by adopting the relevant decisions using the powers delegated to it
by the Commission;
adopting the instruments of budget implementation for the revenue and expenditure and carrying out all activities required to implement a Community programme on the basis of the power
delegated by the Commission, and in particular activities linked to the awarding of contracts and
grants;
gathering, analysing and transmitting to the Commission all the information needed to guide the implementation of a Community programme.
Between 2004 and 2007, six EAs were established, most of which have been reorganised since then (see
Figure 1). Currently the following agencies are entrusted with the management of various parts of European
programmes:
Education, Audiovisual and Culture Executive Agency (EACEA) which was established in 2006;
Executive Agency for Small and Medium-sized enterprises (EASME), which acted as the Executive Agency for Competitiveness and Innovation (EACI) under FP7. The predecessor agency was
reorganised from the Intelligent Energy EA which was created in FP6;
Consumers, Health and Food Executive Agency (CHAFEA). CHAFEA originally started as the Executive Agency for Public Health (PHEA) under FP6, following which the Agency was re-launched as the EA for
Health and Consumers in 2008;
Innovation & Networks Executive Agency (INEA), which was previously the Trans-European Transport Network EA under FP7;
European Research Council Executive Agency (ERCEA);
Research Executive Agency (REA).
Figure 3. Lifeline of the executive agencies
2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 -
EACEA (Education, Audiovisual and Culture EA)
IEEA (Intelligent Energy EA) EACI (EA for Competitiveness and innovation)
EASME (EA for Small and
Medium-sized Enterprises)
PHEA (Public Health EA) EAHC (EA for Health and Consumers)
CHAFEA (Consumers, Health
and Food EA)
18
2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 -
ERCEA (European Research Council EA)
REA (Research Executive Agency)
TEN-T EA (Trans-European Transport Network EA)
INEA (Innovation & Networks
EA)
Source: König, T. (2015). ‘Funding Frontier Research: Mission Accomplished?’, Journal of Contemporary European Research,
11 (1), pp. 124-135.
According to Council Regulation (EC) No 58/2003, an external evaluation of an executive agency needs to be
carried out every 3 years and submitted to the steering committee of the executive agency, to the European
Parliament, to the Council and to the Court of Auditors. Consequently, a number of evaluation reports were
produced, providing valuable information on the overall performance of the EU executive agencies and largely
confirming the comparative advantages of the executive agencies mentioned below.
Among the advantages are such aspects as high quality of management and service delivery2, effective
processes and procedures3, synergies achieved, substantial improvement in the content of EU
programmes and higher cost-efficiency 4 . However, the available evidence points to a number of
difficulties and structural weaknesses within the executive agencies:
Relatively higher cost-efficiency of the executive agencies sometimes has significant negative side-effects. For instance, although CHAFEA managed to attract highly qualified employees to fulfil its
tasks during the first 3 years of its existence, it was also noticed that relatively low salary levels
result in a higher rate of staff turnover, which in turn caused instability in the agency (higher
programme continuity costs, opportunity costs of staff replacement, loss of know-how in the
organisation etc.)5.
In some cases, the effectiveness of the executive agency’s functioning was reduced by a lack of precision in the division of responsibilities between the Agencies and the Commission.
For example, the external evaluation of the ERCEA revealed that there was initial uncertainty with
respect to the distribution of roles and responsibilities between the Agency and DG RTD6, whereas
the ‘Evaluation of the first 3 years of operation of the Executive Agency for Competitiveness and
Innovation’ found that for many of its activities the Agency was associated with the Commission.
Consequently, some of the Agency’s responsibilities had to be made more specific in order to
ensure its maximal effectiveness and efficiency7.
Inflexible standards and procedures and limitations of the mandate imposed by the Commission were also sometimes identified among the factors negatively affecting the
functioning of the executive agencies. For instance, a recent evaluation of REA found that the
Agency‘s efficiency in its administrative and logistical support services was reduced by the
limitations to its mandate outlined in its Establishment Act and Delegation Act. Moreover,
conflicting scope definitions between the REA Establishment Act and the Delegation Act negatively
affected its services for contracting and paying expert evaluators 8 . Likewise, several of the
standards and procedures inherited from the DGs were identified as burdensome and time-
consuming by the Agency during the 1st external evaluation of EASME.
The lack of appropriate material base/infrastructure was identified as a potential factor negatively affecting the effective performance of the executive agencies. The 2nd evaluation of
2 CSES, Evaluation of the Executive Agency for Competitiveness and Innovation, Final Report, 2011; Deloitte, External Evaluation of the REA,
Final report, June 2013; Directorate General Energy and Transport, Evaluation of the first three years of operation of the Executive Agency for Competitiveness and Innovation, Final report, 2009. 3 Deloitte, External Evaluation of ERCEA, Final report, June 2013; CSES, Evaluation of the Executive Agency for Competitiveness and
Innovation, Final Report, 2011. 4 Deloitte, External Evaluation of ERCEA, Final Report, June 2013; CSES, Evaluation of the Executive Agency for Competitiveness and
Innovation, Final Report, 2011. 5 Directorate General for Health and Consumers, 1st interim evaluation of the Public Health Executive Agency, Final Report, 2010. 6 Deloitte, External Evaluation of ERCEA, Final report, June 2013. 7 Directorate General Energy and Transport, Evaluation of the first three years of operation of the Executive Agency for Competitiveness and
Innovation, Final report, 2009. 8 Deloitte, External Evaluation of the REA, Final report, June 2013.
19
EASME concluded that further IT improvements need to be made throughout the Agency’s systems
to enhance its performance.
Finally, the expansion of the executive agencies was sometimes associated with a decrease in the know-how of relevant DG units and in their capacities to make political decisions. For
example, the 1st evaluation of EASME found that enlargement of the scope of the Agency caused
a decrease in the number of informal contacts which previously provided the necessary feedback
for the DGs and which cannot easily be replaced by paper/document reporting. As a consequence,
the parent DG of the Intelligent Energy Europe Programme faced difficulties in drawing political
conclusions and maintaining a high level of know-how within its units.
Finally, the executive agencies face a number of challenges in the near future. After the delegation of new
EU programmes to the executive agencies that took effect from 2014, they should make a smooth transition
to the programming period of 2014-2020 by adapting their internal structures, filling new positions and
updating procedures for the next generation of programmes, as well as by simultaneously managing the new
programmes and the legacy of their responsibilities. Also, because of the Commission‘s proposals to contain
administrative costs (through a staff reduction of 5%) and further improve productivity, the executive
agencies face the need to achieve further efficiency gains9. These improvements in efficiency can stem
from a number of sources, including simplification measures proposed for the 2014-2020 programmes,
optimising the delivery of some functions (e.g. through the establishment of a Common Support Centre for
Horizon 2020) or continuous innovation and learning in order to strive for organisational excellence.
2.1.2. Coherence of management activities in FP7 & Horizon 2020
Between 2007 and 2013, FP7 was the main EU-level instrument aimed at supporting research and
development and represented a key investment for the realisation of European research and innovation
priorities. FP7 had two major strategic objectives: strengthening Europe’s scientific and technological base and
supporting its international competitiveness and the EU policies, through research cooperation among Member
States and with international partners. At the EU level, the European Commission has contributed to the
structuring of ERA by introducing new, and adapting existing R&D support schemes, including the
Cooperation, Capacities, Ideas and People specific programmes.
Figure 4 shows that under FP7 most Cooperation themes were managed by various DGs, including notably DG
RTD which was in charge of most of the activities in Health, KBBE, ENV and NMP. Space and Security were the
only themes which were partly managed by an executive agency (i.e. REA). The People programme was
predominantly managed by REA, whereas most activities under Capacities were managed by the Commission
(DG RTD) and other institutions. Of direct relevance to this evaluation, the Ideas Specific programme was
managed by the European Research Council and implemented by the ERCEA. By the end of FP7 Ideas was
implemented through 5 funding mechanisms:
Starting grants were designed to support excellent Principal Investigators at the career stage at which they are starting their own independent research team or programme. The maximum amount is
EUR 1.5 million for a period of up to 5 years.
Consolidator grants were designed to support Principal Investigators at the stage at which they are consolidating their own independent research team or programme. Consolidator Grants could reach
up to EUR 2 million for a period of 5 years.
Advanced grants promoted substantial advances in the frontiers of knowledge and encouraged new productive lines of enquiry and new methods and techniques, including unconventional approaches
and investigations at the interface between established disciplines. Principal Investigators for the
prestigious ERC Advanced Grant were required to present a track record of significant research
achievements in the previous 10 years of their scientific careers. The maximum amount of funding
could reach EUR 2.5 million for a period of 5 years.
Proof-of-concept grants provided additional funding to previous ERC frontier research grant holders to establish proofs of concept, or identify a development path and an Intellectual Property Rights (IPR)
strategy for ideas arising from an ERC-funded project. The objective was to provide funds to enable
9 Communication to the Commission on the delegation of the management of the 2014-2020 programmes to executive agencies, Brussels,
SEC(2013) 493/2.
20
Capacities
Research
infrastructures
Research for
the benefit of
SMEs
Regions of
knowledge
Research
potential
Science in
society
Coherent
development
of research
policies
International
cooperation
Cooperation
Health
Food,
Agriculture
and Fisheries,
and
Biotechnology
Energy
Environment
(including
climate
change)
Information
and
communication
technologies
Transport
(including
aeronautics)
Nanosciences,
nanotechnolo
gies,
materials and
new
production
technologies
Socio-
economic
sciences and
the humanities
Security
Space
People
Lifelong
training:
individual
fellowships
Initial training:
ITN
Industry-
academia
actions: IAPP
International
dimension:
IRSES
Specific
actions: NIGHT
FP7
Ideas (ERC)
Investigator-
driven inter-
disciplinary
frontier
research
projects
Starting grants
Advanced grants
Proof-of-concept
grants for existing
ERC grant holders
Synergy grants
Consolidator grants
the ERC-funded ideas to be brought to a pre-demonstration stage where potential opportunities for
exploitation could be identified. The financial contribution could amount to a maximum of
EUR 150,000 over a period of 12 months.
Synergy grants were intended to enable a small group of Principal Investigators and their teams to bring together complementary skills, knowledge, and resources in new ways, in order to jointly
address research problems. The maximum grant could reach up to EUR 15 million for a period up to 6
years.
Figure 4. Landscape of management bodies in FP7 (excl. Euratom & JRC)
Predominantly (solid line) or partly (dashed line) managed by REA DG RTD ERCEA
Other agencies and/or DGs (or several agencies/DGs administering a single programme)
Source: PPMI based on desk research.
In 2014, FP7 ended and was replaced by its successor programme, Horizon 2020. Under the new programme
a number of activities that were previously managed by the Commission services were entrusted to the
executive agencies. ‘Health, demographic change and wellbeing’ was the only societal challenge to be
predominantly managed by the EC (i.e. Directorate E, DG RTD), while some activities under Pillar 2 and 3
were mandated to several institutions (see Figure 5 below). The Research Executive Agency’s mandate was
significantly expanded under Horizon 2020 and included a number of additional programmes that the Agency
was not entrusted with during FP7.
21
In Horizon 2020 the ERCEA retained its mandate as it was under FP7 and funded largely the same types of
grants, including Starting Grants, Consolidator Grants, Advanced Grants, and Proof-of-Concept Grants. The
funding of Synergy Grants was temporarily suspended. Since this mechanism was launched as a pilot in 2012
and 2013, no new calls will be announced until its evaluation is carried out10.
Figure 5. Landscape of management bodies in Horizon 2020 (excl. Euratom)
Predominantly (solid line) or partly (dashed line) managed by REA ERCEA DG RTD
Other agencies/institutions (EASME, INEA, EIB & EIF) Other/several institutions (not incl. REA or ERCEA)
managing different parts of a programme
Source: PPMI based on desk research.
10 ERC Work Programmes 2014 & 2015.
Horizon 2020
Pillar 1:
Excellent Science
Pillar 2: Industrial
leadership
Pillar 3: Societal
Challenges
Future and
Emerging
Technologies
(FET)
Marie
Skłodowska –
Curie actions
Research
Infrastructures
Innovation in
SMEs
Access to Risk
Finance
Leadership and
Enabling and
Industrial
Technologies
Climate action,
environment, resource
efficiency and raw
materials
Smart, green and
integrated transport
Secure, clean and
efficient energy
Food Security,
sustainable agriculture,
forestry, marine,
maritime, inland water
and bio-economy
Health, demographic
change and wellbeing
Secure Societies –
protecting freedom and
security of Europe and
its citizens
Inclusive, innovative and
reflective societies
Spreading
excellence and
widening
participation
Science with
and for Society
Specific
objectives
European
Institute of
Technology
European
Research Council
Consolidator
grants
Starting grants
Advanced grants
Proof-of-concept
grants
Synergy grants
22
2.2. Evolution of the ERCEA in FP7 and Horizon 2020
2.2.1. Governance structure of the ERCEA: interplay between ERC, the Commission and
the Agency
To address the challenge of achieving the ERA, broaden the traditional funding instruments of the Framework
Programme and reach more outstanding researchers, the European Research Council (ERC) was set up by
Commission Decision (2007/134/EC)11. Since its inception in 2007, ERC aims at supporting interdisciplinary
bottom-up investigator-initiated projects across Europe by providing different types of grants. Its mission is to
encourage the highest quality research in Europe through competitive funding and to support investigator-
driven frontier research across all fields, on the basis of scientific excellence. ERC is composed of two main
structures:
Scientific Council;
ERC Executive Agency.
The Scientific Council is responsible for setting up ERC’s scientific strategy, ERC Work Programmes, monitoring
quality and performance, establishing the management of calls for proposals, peer review process and
proposal evaluation, including the identification of evaluation panel members. The Scientific Council is
composed of 21 prominent researchers and a chairperson appointed by EC, following their identification by
two separate independent committees (Identification Committee for the members of the Scientific Council and
Search Committee for the chairperson), for a period of 4 years, renewable once. The Chair of the Scientific
Council is the President of ERC who acts as the formal representative of ERC as well as of its Scientific Council
before the European Commission and other bodies12. There are currently three Vice-Presidents assisting the
President who are at the same time the Vice-Chairs of the Scientific Council.
The members of the Scientific Council also meet in two standing committees (StC) and several working groups
(WGs) that address specific issues such as relations with industry, open access, internationalisation, key
performance indicators and gender balance. The WGs carry out analyses and contribute to ERC’s scientific
strategy through proposals to be adopted by the Scientific Council in plenary on the issues mentioned above.
A new group on widening European participation started in 2014. It aims to raise awareness of ERC schemes
in Central and Eastern European countries and attract scientists from these countries to participate in the
programme.
The ERCEA has been an autonomous executive agency since July 2009 13 and it is accountable to the
European Commission. The Commission appoints members of the Steering Committee of the ERCEA14. In late
2015 the members of the Committee included: Robert-Jan Smits, the Director-General of DG Research and
Innovation, as Chairperson of the Committee; Jack Metthey, Director of DG Research and Innovation, as Vice-
chairperson of the Committee; Henk Post, Director of DG HR, responsible for Organisation and Executive Staff,
and two members of the ERC Scientific Council. The ERC President has observer status. The Steering
Committee is mandated to oversee the Agency’s operations, administrative budget and annual reports and
adopt the Agency’s Annual Work programmes.
Cooperation between the ERCEA and DG RTD was initially fostered via the Operational Committee composed
of representatives from DG Research and Innovation and the Agency and chaired by the Agency Director. The
principles of cooperation are outlined in ‘Operational guidelines between the European Research Council
Executive Agency and the Directorate-General for Research and Innovation’15 and in the Written Agreement
between the European Research Council Executive Agency and DG RTD16 that are currently being revisited. In
11 Commission Decision No 2007/134/EC of 2 February 2007 establishing the European Research Council; Pascual C., Sachini E. 2012, ‘5
years of Excellence in the European Research Area 2007-2011: the case of Greece’, National Documentation Centre. 12 Pascual C., Sachini E. 2012, ‘5 years of Excellence in the European Research Area 2007-2011: the case of Greece’, National Documentation
Centre. 13 ERCEA Annual Activity Report, 2009. 14 ERCEA Annual Activity Report, 2013. 15 ERCEA Annual Activity Report 2012. 16 Written Agreement between the European Research Council Executive Agency and the Directorate-General for Research and Innovation.
Modalities and Procedures of Interaction Ref. Ares(2014)1299711 – 25/04/2014.
23
order to maintain and coordinate the interaction between the parent DG and the ERCEA, the Operational
Committee’s activities were replaced by regular meetings between the agency’s Director and the Director of
RTD Directorate A and other participants from Directorates A, R and J. RTD A is the interface the ERCEA from
a policy perspective; the Director of RTD A is also a member of the Steering Committee whereas RDT R is the
interface for all other aspects. Directorate J is the Common Support Centre (CSC) that was established with
the start of H2020. The Director represents the ERCEA at the Executive Committee of the Common Support
Centre. The Agency uses the services of the CSC for:
legal support that provides harmonised interpretation of the rules governing the implementation of Horizon 2020;
ex-post audit relating to Horizon 2020 management, taking into account the specificities of the ERC grants;
Business Processes, relating to Horizon 2020 management, taking into account the specificities of the ERC evaluation, selection and implementation processes;
information technology, which provides IT tools and support for Horizon 2020 implementation;
H2020 information and data.
The ERC Board is the organisational structure bridging the Scientific Council and ERCEA17 . Through its
coordination meetings, the Board aims to ensure effective liaison between the Scientific Council and the
ERCEA on strategy and operational matters. Membership of the Board consists of the following: the President
and three Vice Presidents of the ERC, and the Director of the ERCEA. Before 2014 the ERC Secretary General
also participated in the Board meetings.
Until 2014 ERC had a Secretary-General who was elected autonomously by the Scientific Council. The election
was strictly based on relevant experience and scientific qualifications18. The main role of the ERC Secretary-
General was to ensure effective communications between the ScC, the ERCEA and the EC. This position was
merged with the role of President in 2014. Currently the role of the president involves chairing and leading
ScC, cooperation with the ERCEA and ERC representation19 and is based in Brussels full-time.
17 EC, 2012. Understanding and Assessing the Impact and Outcomes of the ERC and its Funding Schemes (EURECIA) Final Synthesis Report. 18 König, T. 2015. Funding frontier Research: Mission Accomplished? Journal of Contemporary European Research, Vol 11, Issue 1, pp: 125-
133. 19 Annual Report on ERC activities and achievements in 2013.
24
Source: PPMI based on desk research.
2.2.2. Establishment and operation of the ERCEA under FP7
To ensure the operational autonomy and cost-effective administration of ERC, the ERCEA was established by
Commission Decision No 2008/37/EC20. The ERCEA was set up as the Dedicated Implementation Structure
(DIS) of the European Research Council and is mandated to handle the operational management of ERC and
execute the scientific strategy established by the ERC Scientific Council with respect to the Ideas Specific
Programme.
Ideas was one of the five SPs of the Seventh Framework Programme aimed at supporting research and
development at EU level. The Ideas programme funded frontier research solely on the basis of scientific
excellence criteria and researchers from all disciplines are eligible to apply and conduct studies in any EU
Member State or Associated Country. The allocated budget to Ideas under FP7 was EUR 7.5 billion. While
administering the Ideas SP, the main tasks for the ERCEA included the following21:
management of some or all of the stages in the lifetime of projects in the frame of the Specific Programme Ideas;
execution of the annual ERC work programme, as established by the ERC Scientific Council and adopted by the Commission;
to ensure the management of calls for proposals, peer-reviews and selection of proposals;
grant signature and the subsequent management of grant payments;
20 Commission Decision No. 2008/37/EC of 14 December 2007 setting up the European Research Council Executive Agency. 21 Commission Decision No. 2008/37/EC of 14 December 2007 setting up the European Research Council Executive Agency; Commission
Decision C(2013)9428/F1 of 20/12/2013 on delegating powers to the European Research Council Executive Agency with a view to
performance of tasks linked to the implementation of Union programmes in the field of frontier research comprising, in particular,
implementation of appropriations entered in the general budget of the Union.
Figure 6. Governance components of ERC and their interactions
ERC Scientific Council
-President + 21 other members
-3 Vice-Chairs/Vice-Presidents:
Prof. Sierd Cloetingh
Prof. Mart Saarma
Prof. Núria Sebastian Galles ERC Board:
-ERC President
-ERC Vice-Presidents
-ERCEA Director
Steering Committee
5 members
European Commission
(DG Research and Innovation, DIR A)
ERC Executive Agency
Director: Pablo Amor
ERC President:
Prof. Jean-Pierre
Bourguignon
25
provision of support to the ERC Scientific Council in particular in communication and information activities, collecting, processing and distributing data and monitoring and evaluation of the
programme.
Initially two core funding schemes were launched, the ERC Starting Independent Researcher Grants (‘ERC
Starting Grants’ (StG)) and ERC Advanced Investigator Grants (‘ERC Advanced Grants’ (AdG)). StG were
meant to support the independent careers of excellent researchers, despite their nationality, who were located
in or moving to the Member States and associated countries, who were at the stage of starting or
consolidating their own independent research team or, depending on the field, establishing their independent
research programme. AdG, on the other hand, funded excellent, innovative investigator-initiated research
projects across the Member States and associated countries, directed by leading advanced investigators of
whatever age, who have already established themselves as being independent research leaders in their own
right.
An additional funding initiative was introduced in 2011. The top-up ‘Proof-of-Concept’ (PoC) grant targeting
current ERC grant holders was dedicated to funding the gap between their research and the earliest stage of a
marketable innovation. In 2012 another new initiative was started, the ERC Synergy grant for exceptional
proposals launched by a few small groups of researchers working together. Also, in response to the rapidly
rising number of applications, the Starting Grant scheme was divided into two separate calls, ‘Starters’ and
‘Consolidators’ in 2013. Consolidator Grants (CoG) supports researchers who are at the early stage of their
careers but very often already working with their own group. The starting grants scientists who were in the
process of setting up their own research group.
Over the years since ERCEA’s establishment the numbers of calls has increased as new funding mechanisms
were introduced or existing ones were expanded (see Figure 7 below). The number of applications received
also increased significantly over the years with the exception of 2014 when the new Framework Programme
started. A number of factors contributed to this evolution, including a) the introduction of restrictions to
control the number of applications received in previous years of the programme; as well as b) the suspension
of the Synergy grants.
Figure 7. ERCEA's evolution since its inception: number of calls launched and proposals received
Source: ERC and ERCEA annual activity reports and ERCEA interim report of 2015.
*In the first semester of 2015 calls for second deadlines of the StG2015, CoG2015, AdG2015 and PoC2015 were started.
Over the years of ERCEA’s existence the total number of employees constantly increased as was projected in
the Legislative Financial Statement22. Two categories of employees – temporary agents (TA) and seconded
national experts (SNE) – remained rather stable in terms of staff over the years. The category of contract
agents (CA) which comprises employees recruited from the EPSO CAST database for the administrative,
technical, financial and secretarial jobs has grown over the last few years. The staff numbers were increasing
22 Deloitte, External Evaluation of ERCEA, Final report, June 2013.
2 2
3
4 4 4 4
00,5
11,5
22,5
33,5
44,5
Number of calls launched
4087 4882
6515 7899
10151
8521 7125
0
2.000
4.000
6.000
8.000
10.000
12.000
Proposals received
26
together with the increasing workload as funding increased and the number of proposals received also grew.
It is foreseen that the ERCEA will have 529 employees by 2020.
Figure 8. ERCEA’s staff from 2008 until 2014
Source: ERC and ERCEA annual activity reports and ERCEA interim report of 2015.
The commitment credits were increasing gradually over the course of FP7. Together with the increasing
budget, the total number of funded projects was increasing until 2012 then stabilised in 2013. Following the
drop noted in 2014, the number of funded projects increased considerably in 2015 to over 1 000 grants signed
this year (see Figure 9 below). The number of running grants steadily increased to 4 864 by the end of 2015.
Regarding overall budget implementation, during the Ideas programme the EUR 7.5 billion were fully
implemented in accordance with the indirect centralised management mode. In addition, the administrative
costs were kept low and in 2013 comprised 2.27 % of the operational budget. The application success rates in
FP7 Ideas indicate a very high level of competition for ERC funding. During FP7 on average the success rate
of proposals for starting and consolidator grants was 10.2 % and for advanced grants 13.9 %. The success
rate for synergy grants proposals was the lowest at 2.3 %. Proof-of-concept grants were the least competitive
with approximately 1 in 3 proposals receiving funding.
Figure 9. Total number of grants signed by the ERCEA from 2009-2014 and total commitment
credits as foreseen in the work programmes
Source: ERC and ERCEA annual activity reports and work programmes.
2008 2009 2010 2011 2012 2013 20142015 (Jan-
Jun)
Contract agents 49 162 218 245 275 270 277 286
Temporary agents 1 93 94 97 96 99 99 105
Seconded national experts 11 7 4 8 9 10 12 14
Total 61 262 316 350 380 379 388 405
0
50
100
150
200
250
300
350
400
450
Contract agents Temporary agents Seconded national experts Total
383
581
834 900 875
536 648
0100200300400500600700800900
1000
Number of grant agreements signed
0.8
1.1
1.3
1.6
1.8 1.7 1.7
0
0,2
0,4
0,6
0,8
1
1,2
1,4
1,6
1,8
2
2009 2010 2011 2012 2013 2014 2015
Commitment credits (EUR , billion)
27
In total, 5 995 projects were funded by an ERC grant up to December 201523. Also, in December 2015 the
number of running grants was 3 674 for FP7-funded projects and there were 1 190 running grants for projects
funded under Horizon 2020.
Several evaluations were carried out to review the operation of t ERC and the ERCEA. The review of the
European Research Council’s Structures and Mechanisms conducted in 2009 recognised the successful
initiation and operation of ERC24. But it also stressed that complex staff recruitment practices, the governance
structure and the management of internal procedures could hinder the coherent development and
professionalisation of the Agency.
Two years later the ERC task force recognised once more the significant progress made by ERC in such a short
period of time 25 . Increased operational autonomy and improvement in the administration of scientific,
financial and administrative operations were outlined as the main recommendations by the Task Force. Also,
a quasi-full time role for the President of ERC starting from 2014 and enhanced role for the Director of ERC's
Executive Agency were recommended by the Task Force.
The external evaluation of the ERCEA also reported very positive results on ERCEA’s operation26. It pointed
out that the Agency has achieved its objectives, that it is operating within its legal framework and that the
peer-review system is exemplary and has been copied at national levels. The main areas identified as needing
improvement were cross-unit working and alignment, as well as human resources (particularly career
development, gender balance in management positions and staff retention). Finally, as the organisation was in
the process of rapid growth in terms of budget, it was recommended to prepare for this development by
adapting a HR strategy, organisational, IT structures and management.
2.2.3. Operation of the ERCEA in Horizon 2020
The ERCEA was re-established by decision 2013/779/EU 27 once the Regulation for establishment of the
Horizon 2020 Framework Programme for research and Innovation was adopted28. It was set up for the period
2014–202429 with a budget of over EUR 16 billion to manage its Horizon 2020 programmes and the legacy
grants of FP7. The Agency was mandated to implement the Part I ‘Excellent science’ Framework Programme
for Research and Innovation (2014–2020) of Horizon 2020 with the objective of ‘strengthening frontier
research through the activities of the European Research Council’30.
The Agency was entrusted with similar tasks as under FP7 including project cycle management, annual work
programme execution and providing assistance to the ERC Scientific Council31. In addition, the Agency was
also mandated to perform all operations required to launch contests and award prizes in accordance with
Regulation (EU, Euratom) No 966/2012 as well as conclude public procurement procedures and manage the
ensuing contracts.
ERC under Horizon 2020 maintains the same principles, selection procedures and the main criteria of scientific
excellence for project selection. The types of grants also remain the same as at the end of the FP7. Calls for
23 https://erc.europa.eu/projects-and-results/statistics, this figure takes into account all the grants of the ERC since the first call, for all the
funding schemes including the PoC grants. 24 Review Panel, 2009. Towards a world class Frontier Research Organisation. Review of the European Research Council’s Structures and
Mechanisms. 25 European Research Council, Task Force Final report. 12 July 2011. 26 Deloitte, External Evaluation of the REA, Final report, June 2013. 27 Commission Decision No. 2013/779/EC of 17 December 2013 establishing the European Research Council Executive Agency and repealing
Decision 2008/37/EC. 28 Regulation (EU) No. 1291/2013 OF 11/12/2013 of the European Parliament and of the Council establishing Horizon 2020 – the Framework
Programme for Research and Innovation (2014-2020) and repealing Decision No. 1982/2006/EC. 29 Commission Decision No 2013/779/EC of 17 December 2013 establishing the European Research Council Executive Agency and repealing
Decision 2008/37/EC. 30 Commission Decision C (2013)9428/F1 of 20/12/2013 on delegating powers to the European Research Council Executive Agency with a
view to performance of tasks linked to the implementation of Union programmes in the field of frontier research comprising, in particular,
implementation of appropriations entered in the general budget of the Union. 31 Commission Decision C(2013)9428/F1 of 20/12/2013 on delegating powers to the European Research Council Executive Agency with a view
to performance of tasks linked to the implementation of Union programmes in the field of frontier research comprising, in particular,
implementation of appropriations entered in the general budget of the Union
https://erc.europa.eu/projects-and-results/statistics
28
Synergy grants were not announced in 2014 and 2015 in order to evaluate the results of the pilot phase of
this funding instrument. However, due to the ever-increasing number of proposals and high competition, the
Scientific Council decided to establish some resubmission restrictions for unsuccessful applicants. Those
applicants who were not successful (receiving B or C in their proposal) could only reapply after 1 (if proposal
was graded as B) or 2 (if proposal was graded as C) years respectively. Extension of restrictions on
applications applied to the 2015 calls based on the outcome of the evaluation of the 2014 calls32. In addition,
from the 2015 calls there will not be a pre-defined budget for the three scientific fields. The budget to be
allocated to the different scientific fields will be determined solely by demand33.
Although ERCEA’s mandate was not expanded under Horizon 2020, the budget allocations almost doubled.
The actual yearly amount allocated for the years 2014, 2015 and 2016 was EUR 1.7 billion, while it is
estimated that by 2020 it should increase and reach EUR 2.2 billion. The ERCEA launched four calls for
proposals in 2014 and received over 8 500 applications. A total of 1 025 grants agreements were signed in
2015 under the 2014 calls for proposals34.
In addition, the Agency underwent a number of organisational and procedural changes during the reference
period for this evaluation.
For example, the Grant Management Department was reorganised just before the start of Horizon 2020 to better serve the established workflows35. The original structure was made up of three units,
each responsible for one phase of the grants’ life cycle (preparation, management and audit). This
functional separation was becoming suboptimal with the ever-increasing number of grants to be
processed. The existing staff was then reorganised into four units, with three units dealing with the
complete life cycle of just one type of grant each (StG, CoG, AdG) plus a fourth unit still in charge of
ex-post controls for FP7. PoC and SyG grants were attributed to the CoG unit, as they are very limited
in number.
Another reorganisation took place in 2014 in the Scientific Department. The unit providing support to the evaluation panels was split into three different Units corresponding to the three research areas
that ERC funds, Life Sciences, Physical Sciences and Engineering as well as Social Sciences and
Humanities.
Overall, under Horizon 2020 the Scientific Council remains independent with full authority over scientific
strategy, scientific management, monitoring and quality control, scientific communication and dissemination,
and the EA remains administratively autonomous. Efforts were made to further strengthen the governance of
ERC by strengthening the links between the Scientific Council and the Executive Agency as well as
strengthening the role of Scientists in the Steering Committee of the ERCEA.
2.3. Management structure of the ERCEA
The ERCEA is led by the Director, who is responsible for the day-to-day management of the Agency and its
staff and also for the implementation of ERC's strategy and positions, as defined by the ERC Scientific Council.
The current ERCEA Director, Pablo Amor, took up the post in August 2012 after acting as the Agency‘s
Director ad interim since January 2011.
The Agency is structured into 3 departments, 2 horizontal units and the Chief Accountant’s sector. In more
detail, the following structures and units are overseen by the Director:
Chief Accountant’s Office;
Support to the Scientific Council Unit (A1);
Communication Unit (A2);
Scientific Management Department (B);
Grant Management Department (C);
Resources and Support Department (D).
32 Annual Report on the ERC activities and achievements in 2013. 33 Annual Report on the ERC activities and achievements in 2014. 34 ERCEA Annual Activity Report, 2014. 35 ERCEA Annual Activity Report, 2014.
29
The A1 Unit provides organisational and administrative support as well as advice and analysis to the Scientific
Council36. This unit is responsible for supporting the Scientific Council in establishing the ERC Annual Work
Programmes, ERC Annual Reports, various briefings, data analysis and presentations linked to the Monitoring
& Evaluation Strategy of the Scientific Council. Unit A2 assists the Scientific Council in its communication
strategy towards the scientific community, the public authorities and the public at large, in cooperation with
the Commission.
The Scientific Management Department is composed of 5 Units. Unit B1 administers Process Management and
Review and B2 Call and Project Follow-up Coordination. The other units, B3, B4 and B5 manage the scientific
areas of Life Sciences, Physical Sciences and Engineering and Social Sciences and Humanities, respectively.
The Grant Management Department is responsible for managing the project during the entire project life
cycle, starting with processing the ranking list after the end of the evaluation until final payment and ex-post
control, Within Department C there is Sector C0 responsible for Operational Budget and Reporting, 3 financial
units: Starting Grant, Consolidator Grant, Advanced Grant as well as the Audit and ex-post controls unit37. The
latter also reports directly to the Director.
Under the Resources and Support Department lies the sector for Administrative Budget and three units: ‘IT
and Support Services’, ‘Human Resources’ and ‘Legal Affairs and Internal Control’.
36 ERC Annual Report 2013. 37 ERC Annual Report 2014.
30
Figure 10. Organisational structure of ERCEA
Source: PPMI based on desk research.
C
Grant Management
Department
D
Resources & Support
Department
Director
A1
Support to the
Scientific Council
A2
Communication
CA0
Chief Accountant’s
Office
B
Scientific Management
Department
C0
Operational Budget
and Reporting
C1
Starting Grant
C2
Consolidator Grant
C3
Advanced Grant
B1
Process Management
and Review
B2
Call and Project Follow-
up Coordinator
B3
Life Science
D0
Administrative Budget
D1
IT and Support Services
D3
Legal Affairs and
Internal Control
D2
Human Resources
B4
Physical Sciences and
Engineering
B5
Social Sciences and
Humanities
C4
Audit and ex-post
controls
31
3. EVALUATION OF ERCEA’S PERFORMANCE IN
2012-2015
3.1. Effectiveness of ERCEA
Evaluation of the effectiveness of ERCEA from July 2012 to July 2015 comprised the Agency’s operation
during two separate programming periods: 18 months of the FP7 and 18 months of the Horizon 2020
implementation.
Despite the transition from the 2007–2013 programmes to the 2014–2020 programmes, objectives of the
Agency did not differ significantly during these two periods. According to the new Commission’s Decision
adopted in 2013 (Article 3), under Horizon 2020 the Agency was entrusted with (1) the implementation of the
specific objective ‘strengthening frontier research through the activities of the European Research Council’ of
Part
