EVALUATION OF THE OPERATION OF REA (2012-2015) · E-mail: [email protected] European...

EVALUATION

OF THE OPERATION OF REA

(2012-2015)

Annexes

2

EUROPEAN COMMISSION

Directorate-General for Research and Innovation

Directorate R — Resources

Unit R4 — New Management Modes

E-mail: [email protected]

European Commission

B-1049 Brussels

EUROPEAN COMMISSION

EVALUATION

OF THE OPERATION OF REA

(2012-2015)

Annexes

Prepared by: Public Policy and Management Institute

Directorate-General for Research and Innovation

2016

4

LEGAL NOTICE

Neither the European Commission nor any person acting on behalf of the Commission is

responsible for the use which might be made of the following information.

The views expressed in this publication are the sole responsibility of the author and do not

necessarily reflect the views of the European Commission.

More information on the European Union is available on the Internet (http://europa.eu).

Luxembourg: Publications Office of the European Union, 2016.

PDF ISBN 978-92-79-58797-9 doi: 10.2777/679459 KI-02-16-511-EN-N

© European Union, 2016

Reproduction is authorised provided the source is acknowledged.

EUROPE DIRECT is a service to help you find answers

to your questions about the European Union

Freephone number (*):

00 800 6 7 8 9 10 11

(*) Certain mobile telephone operators do not allow access to 00 800 numbers

or these calls may be billed

5

CONTENTS

List of tables and figures ...................................................................................................................... 6

Annexes ....................................................................................................................................... 8

Annex 1. Detailed approach to key data collection and analysis methods ........................................ 8

Documentary review and desk research ..................................................................................... 8

Interview programme ............................................................................................................ 11

Survey 12

Cost-benefit analysis ............................................................................................................. 14

Annex 2. REA interview questionnaire ...................................................................................... 16

Annex 3. Questionnaires and aggregated data of REA beneficiaries surveys .................................. 30

The questionnaire of the survey of REA grant beneficiaries: FP7 ............................................ 30

The questionnaire of the survey of REA beneficiaries: Horizon 2020 ....................................... 38

Aggregated data of REA beneficiaries surveys (FP7 and Horizon 2020) ................................... 46

Annex 4. Questionnaires of REA independent experts surveys ..................................................... 55

The questionnaire of the survey of REA independent experts: evaluators ............................... 55

The questionnaire of the survey of REA independent experts: monitoring experts ................... 63

6

LIST OF TABLES AND FIGURES

List of Tables

Table 1. Interviews with REA staff and stakeholders conducted 17-21 August, 2-4 September and 16-18

September. ............................................................................................................................... 11

Table 2. Targeting and coverage of the survey programme ............................................................. 12

Table 3. Summary of survey response and completion rates ........................................................... 14

Table 4 Q3 Where did you first hear about the opportunity to become an independent expert of the

Agency/Commission? ................................................................................................................. 71

Table 5 Q4a Please assess the usefulness of the information provided online: ................................... 71

Table 6 Have you sought for information or assistance from the Research Executive Agency (REA) during

the registration process?............................................................................................................. 71

Table 7 Q5a Please assess the quality of information/assistance provided by the REA: ....................... 71

Table 8 Q7 Before you could be contracted as an expert, your identity and bank account details had to

be verified (‘Legal Entity/Bank Account Validation’). This procedure is carried out separately from the

registration process. Please assess the LE/BA validation procedure. ................................................. 72

Table 9 Q9 Were you contacted with regard to your availability to work as an independent expert for a

particular evaluation or project review before you received an expert contract to sign? ...................... 73

Table 10 Q10. Please give your assessment of the following statements. .......................................... 73

Table 11 Q12Experts are required to declare conflicts of interest (CoI), either at the time of signature of

the contract or as soon as a CoI is identified. Please assess the process in relation to CoI. .................. 73

Table 12 Q14 When an expert needs to travel for an assignment, travel costs are reimbursed to and

from the expert's address as indicated in the contract (the ‘Point of Departure’ – POD). An expert who

wishes to travel to and/or from a different address must request prior approval (‘request for change of

point of departure’). Please assess the process in relation to requests to change the point of departure

for assignments involving travel. ................................................................................................. 73

Table 13 Q15 Have you sought information or assistance from the REA during the contracting process? 73

Table 14 Q16 Regarding place of work for the assignment, did you have to carry out your tasks: ........ 74

Table 15 Q16a Please assess the extent to which you are satisfied with the quality of REA evaluation

facilities in Brussels: .................................................................................................................. 74

Table 16 Q17 To what extent do you agree or disagree with the following statements concerning your

work as an expert: ..................................................................................................................... 74

Table 17 Q17 To what extent do you agree or disagree with the following statements concerning your

work as an expert: ..................................................................................................................... 75

Table 18 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? ............ 76

Table 19 Q19a Please assess the quality of information/assistance provided by the REA: .................... 76

Table 20 Q18a Please assess the quality of information/assistance provided by the REA during the

briefing: ................................................................................................................................... 76

Table 21 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? ............ 77

Table 22 Q20 Did you use the [email protected] functional mailbox? .............................. 77

Table 23 Q20a At what stage in the process did you use this service: ............................................... 78

Table 24 Q20b Please assess the quality of information/assistance provided by the REA through the REA-

[email protected] mailbox. ............................................................................................... 78

Table 25 Q21 To what extent do you agree with the following statements concerning payments for your

services? .................................................................................................................................. 79

Table 26 Q23 Would you like to work with the REA as an independent expert again in the future? ....... 79

Table 27 Q23a What are the main reasons for indicating that you are unlikely to work with the REA as an

independent expert again in the future? ....................................................................................... 80

7

List of Figures

Figure 46. Methodology for the surveys ........................................................................................ 13

Figure 47 Q6 To what extent do you agree or disagree with the following statements concerning your

registration process: .................................................................................................................. 72

Figure 48 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? ........... 77

Figure 49 Q20 Did you use the [email protected] functional mailbox? ............................. 77

8

ANNEXES

Annex 1. Detailed approach to key data collection and analysis methods

Documentary review and desk research

Literature review is essential in mapping the academic and analytical field of ideas and studies that can be

related to the context of the external evaluation. It serves as a source of contextual information, additional

evaluation questions, indicators for assessment, and data for comparison with other agencies. The literature

review in this evaluation consisted of:

primary sources of information (documents, empirical studies);

Secondary sources of information (generalisations).

To this date we have reviewed primary and secondary sources that included:

EU-level documents;

Previous evaluations and Audit Reports of REA;

Studies/Action Plans/Reports.

In addition, to conduct a thorough literature review as well as to collect all the necessary data for the cost-

benefit analysis the evaluation team was provided with all the documents outlined in the Tender

Specifications. Also, we extracted important documents from the Agency’s intranet, including monthly

financial reports, bimonthly budget execution reports and Strategy on IT Governance.

A list of reviewed sources is provided below.

EU level documents:

Commission Decision 2008/46/EC of 14 December 2007 setting up the ‘Research Executive Agency’

for the management of certain areas of the specific Community programmes People, Capacities and

Cooperation in the field of research in application of Council Regulation (EC) No 58/2003.

Commission Decision C(2007)2466 of 13 June 2007 on the adoption of the rules to ensure consistent

verification of the existence and legal status of participants, as well as their operational and financial

capacities, in indirect actions supported through the form of a grant under the Seventh Framework

Programme of the European Community for research, technological development and demonstration

activities (2007-2013) and under the Seventh Framework Programme of the European Atomic Energy

Community (Euratom) for nuclear research and training activities (2007-2011).

Commission Decision C(2010)5184 of 30 July 2010 amending Decision C(2008) 3980 final of

31/VII/2008 delegating powers to the Research Executive Agency with a view to performance of tasks

linked to implementation of the specific Community programmes People, Capacities and Cooperation

in the field of research comprising, in particular, implementation of appropriations entered in the

Community budget.

Commission Decision C(2013)9418 of 20 December 2013 on delegating powers to the Research

Executive Agency with a view to performance of tasks linked to the implementation of Union

programmes in the field of frontier research comprising, in particular, implementation of

appropriations entered in the general budget of the Union.

Commission decision C(2014) 7428 of 16 October 2014 approving the amended Annual Work

Programme 2014 of the Research Executive Agency and repealing Decision C(2014) 5003.

Commission decision C(2014)9109 of 2 December 2014 establishing guidelines for the establishment

and operation of executive agencies financed by the general budget of the Union.

Commission decision C(2015) 2297 of 10 April 2015 approving the Annual Work Programme 2015 of

the Research Executive Agency.

Commission Implementing Decision 2013/778/EU of 13 December 2013 establishing the Research

Executive Agency and repealing Decision 2008/46/EC.

Communication to the Commission on the delegation of the management of the 2014-2020

programmes to executive agencies, Brussels, SEC(2013) 493/2.

9

Council Decision 2006/971/EC of 19 December 2006 concerning the Specific Programme ‘Cooperation’

implementing the Seventh Framework Programme of the European Community for research,

technological development and demonstration activities (2007 to 2013).

Council decision of 3 December 2013 establishing the specific programme implementing Horizon 2020

- the Framework Programme for Research and Innovation (2014-2020) and repealing Decisions

2006/971/EC, 2006/972/EC, 2006/973/EC, 2006/974/EC and 2006/975/EC.

Council Regulation (EC) No 58/2003 of 19 December 2002 laying down the statute for executive

agencies to be entrusted with certain tasks in the management of Community programmes.

Decision N° 1982/2006/EC of 18 December 2006 of the European Parliament and the Council

concerning the Seventh Framework Programme of the European Community for research,

technological development and demonstration activities (2007-2013).

Legislative financial statement to the Commission Decision 2008/46/EC of 14 December 2007.

Memorandum of Understanding between the Research Executive Agency and DG Research and

Innovation, DG Education and Culture, DG Enterprise and Industry, DG Communication Networks,

Content and Technology, DG Agriculture and Rural Development: Modalities and Procedures of

Interaction.

Memorandum of understanding of ‘It hosting services for the ABAC system’ between the Research

Executive Agency and the Directorate-General for Informatics (DIGIT).

Regulation (EC) N° 1906/2006 of the European Parliament and the Council of 18 December 2006

laying down the rules for the participation of undertakings, research centres and universities in actions

under the Seventh Framework Programme and for the dissemination of research results (2007-2013).

Regulation (EU) N° 1290/2013 of the European Parliament and the Council of 11 December 2013

laying down the rules for the participation and dissemination in Horizon 2020 - The Framework

Programme for Research and Innovation (2014-2020) and repealing Regulation (EC) No 1906/2006.

Regulation (EU) N° 1291/2013 of 11/12/2013 of the European Parliament and of the Council

establishing Horizon 2020 – The Framework Programme for Research and Innovation (2014-2020)

Regulation (EU, EURATOM) No 966/2012 of the European Parliament and of the Council of 25 October

2012 on the financial rules applicable to the general budget of the Union and repealing Council

Regulation (EC, Euratom) No 1605/2002 [Official Journal L 298, 26.10.2012].

Service Level Agreement between the European Union and the Executive Agencies for the definition of

the administrative and financial terms governing the relations between an Executive Agency and DG

Budget in relation to the implementation of the ABAC System.

Service level agreement Concerning the collaboration between Directorate-General for Human

Resources and Security of the European Commission (DG HR) and the Research Executive Agency

(REA).

Service level agreement Concerning the collaboration of Research Executive Agency (REA) with the

Office for Administration and Payment of Individual Entitlements (PMO).

Specific Financial Statement related to the Decision Establishing the European Research Council

Executive Agency and repealing Decision 2008/37/EC.

Previous evaluations and Audits Reports of REA:

CSES (2011), Evaluation of the Executive Agency for Competitiveness and Innovation, Final Report.

Deloitte (2013), External Evaluation of the ERCEA, Final report, June 2013.

Deloitte (2013), External Evaluation of the REA, Final report, June 2013.

European Court of Auditors (2009), Delegating implementing tasks to executive agencies: a successful

option?, Special Report No 13.

ICF GFK (2013), Cost Benefit Analysis for the delegation of certain tasks regarding the implementation

of Union Programmes 2014-2020 to the Executive Agencie.s

Monthly execution reports of REA Operational and Administrative Budgets 2012, 2013, 2014, 2015.

Monthly Overview report on resources, audit, internal control and key performance indicators (KPIs) in

the REA 2013, 2014, 2015.

REA (2010), Financial circuits in the REA.

REA (2012), REA 2012 Staff Survey Report.

REA (2012), Report on the REA Supervision Campaign 2012: Review of project reports, Brussels.

REA (2013), Report on the assessment of the Internal Control Systems (ICS 15) outcome of 2013

ICAT survey.

REA (2014), REA Interim Report to the parent DGs and the Steering Committee: First Semester 2014.

10

REA (2015), Report on the assessment of the Internal Control Systems (ICS 15) outcome of the 2014

ICAT exercise and proposal of an action plan.

REA Administrative Budget 2012, 2013, 2014, 2015.

REA client satisfaction study and recommendations for stakeholder communication.

REA Risk Assessment Exercise 2012, 2013, 2014.

Technopolis (2007), Cost benefit analysis for the Research Executive Agency, Revised report.

Studies/Action Plans/Reports:

REA Work programmes 2012, 2013, 2014 and 2015.

REA Annual Activity Reports 2012, 2013 and 2014.

Minutes of the REA Steering Committee Meetings.

Minutes of the REA Management Meetings 2012, 2013, 2014, 2015.

REA Information Technology Master Plan 2010, 2011, 2012, 2013, 2014, 2015.

Minutes of the REA IT Management Board Meeting 2012, 2013, 2014.

Minutes of the REA Network of Legal Officers Meetings.

Minutes of the REA Network of Call Coordinators Meetings.

Minutes of the REA Network of Financial Officers Meetings.

Quarterly Reports of Activities of the Network of Project Officers.

Benzecri, J.P. (1973), L'analyse de données, Dunod Edition, Paris.

Bourdieu, P. (1984), Distinction: a Social Critique of the Judgment of Taste, Cambridge, MA: Harvard

University Press.

CAF Resource Centre of the European Institute of Public Administration, CAF 2013: Improving Public

Organisations through Self-Assessment.

DG BUDG (2013), Risk Management: Specific Guidance for Grants (direct management).

DG RTD (2007), Ex-post Audit Strategy of FP6 common to the Research DGs Period 2007-2010.

DG RTD (2009), Overview of the Internal Control Framework in the REA.

DG RTD (2015), H2020 General Model Grant Agreement — Mon.o

DG RTD (2015), H2020 General Model Grant Agreement — Multi.

Di Franco, G. (2001). EDS: esplorare, descrivere e sintetizzare i dati. Milan: Franco Angeli

Di Franco, G., (2006) Corrispondenze multiple e altre tecniche multivariate per variabili categoriali,

Franco Angeli, Milano; Greenacre, M. and Blasius, J., (2006) Multiple Correspondence Analysis and Related Methods] Chapman-Hall, Boca-Raton, FL; Holmes S., (2007), Multivariate Analysis: The

French Way, Probability and Statistics: Essays in Honor of David A. Freedman, Volume 2, 219-233 IMS Lecture Notes - Monograph Series, IMS, Beachwood, OH.

DIGIT (2010), Infrastructure Services Provision Service Catalogue Oracle RDBMS Hosting.

Directorate General Energy and Transport, Evaluation of the first three years of operation of the

Executive Agency for Competitiveness and Innovation, Final report, 2009.

Directorate General for Health and Consumers, 1st interim evaluation of the Public Health Executive

Agency, Final Report, 2010

European Commission (2010), Interim Evaluation of the Seventh Framework Programme, Final report,

Brussels

European Commission (2013), Risk Management in the Commission: Implementation Guide, Brussels

European Commission (2014), ICS 8: Complementary Guidance Exception Reporting

REA (2008), Internal Communication Strategy 2009-2010

REA (2011), Developing an Internal Staff Mobility Policy for the REA: Initial Measures to offer Internal

Mobility Opportunities to REA Staff, Brussels

REA (2011), Document Management in the Research Executive Agency: Internal Procedures

REA (2012), Decision of the Steering Committee of the Research Executive Agency concerning the

terms and conditions for internal investigations in relation to the prevention of fraud, corruption and

illegal activity detrimental to the communities’ interest

REA (2012), IT governance in the REA

REA (2012), Learning and Development Framework 2012

REA (2012), Overview of the REA's financial circuits for the operational expenses

REA (2013), REA Business Continuity Plan

REA (2014), ICT Activities: 2014 - 2016

REA (2015), Overview of the REA's financial circuits for the administrative expenses

11

Interview programme

A total of 29 interviews were carried out in August and September with various REA and EC officials outlined

in the table below. These interviews helped to collect new/internally available data and gain a deeper

understanding of the internal processes of the organisation, its human resources policy, commitment of the

staff to the internal quality procedures, advantages and drawbacks of the governance structure and other

issues. The initial results were also used in developing the survey questionnaires. All interviews were semi-

structured, adapted to the specific target group of the interview programme and were conducted face-to-face

in the premises of the interviewees. A table outlining interviewees is presented below, while the interview

questionnaire can be found in Annex 2.

Table 1. Interviews with REA staff and stakeholders conducted 17-21 August, 2-4 September and

16-18 September.

Person Unit Interview type

1. W. De Meyere REA.C1 Group interview

M. Carneiro

2. S. Fumero REA.C3 Group interview

F. De La Torre Francia

3. E.G. Chira REA.A2 Group interview

J. B. Veyret REA.A2

M. Spulber REA.A2

F. Willekens REA.A4

L. Byrne

4. J. Hemmelskamp REA.B3 Group interview

C. Amting REA.B3

A.Tristan REA.B3

5. E. Tsavalopoulos REA.C1 Individual interview

6. I. Reyes RTD.A6 Individual interview

7. P. Kolar RTD.F3 Individual interview

8. I. Marien-Dusak CNECT.R5 Individual interview

9. J. Methey RTD.A Individual interview

10. T. Telemachou RTD.B5 Group interview

G. Ambroziewicz

11. A. Oram REA.C4 Group interview

E. Pellizzari

12. K. Rosenow REA.B2 Group interview

D. Bennink

O. Pastre

S. Goffin

G. Valcarcel

13. F. Olsson REA.A1+A3 Group interview

F. Marx

K.-G. Barthel

K. Hellevuo

M.-S. Giannonni

B. Mester

14. G. Gascard REA Individual interview

15. W. Beurms REA Individual interview

16. A. Longo DG AGRI Group interview

A. Loncke

17. P. Fernandez-Cañadas DG RTD Individual interview

18. W. Burtscher DG RTD Individual interview

19. B. Pyke DG Home Group interview

G. Willmot

20. A. Luchetti REA, Department A Individual interview

12

21. T. Hallantie REA.A5 Group interview

M. Lange

M. Pettinarolli

22. R. Bultynck REA.C2 Group interview

N. Giacomuzzi

C. Strouzas

23. M. Curavic REA.B1+B4 Group interview

C. Bernot

V. Puzzulo

A. Marino

T. Brefort

P. M. Schmidt

V. Bricola

24. P. van der Zandt REA.B5 Group interview

F. Roffi

G. Suriano

25. M. Tachelet REA Group interview

C. Amting

26. M. Tachelet REA Group interview

W. de Meyere

27. J. Hemmelskamp REA chairs of networks Group interview

O. Pastre

28. V. Canetti REA Staff Committee Individual interview

29. E. Armani DG HR Individual interview

Survey

This data gathering method aimed at collecting first-hand information (in the form of quantitative and

qualitative data) needed to assess the extent to which various organisational factors and specificities were

making a positive/negative impact on the performance of the agency. In total, two interlinked surveys

addressing the following target groups were carried out:

Survey A: an online survey of experts contracted by the REA to carry out evaluation and tasks

in 2014 and 2015.

Survey B was an online survey targeting beneficiaries of programmes managed by the REA

which were financed under the 2011-2014 calls for proposals. Perceptions of respondents to

this survey were particularly relevant for assessing the extent to which the agency made a progress in

improving its customer-oriented results and the degree to which the enablers of the agency were

instrumental to achieving such results.

Table 2. Targeting and coverage of the survey programme

Target groups Types of respondents Coverage

Survey A: REA’s independent experts

Independent experts All expert evaluators and monitors contracted by the REA in 2014 and 2015

Survey B: beneficiaries of programmes managed by REA

FP7 and H2020 beneficiaries All projects managed by the REA from the 2011-2014 calls for proposals

Overall methodology of the survey programme

Our survey programme involved a number of operational activities that ensured an appropriate scoping,

implementation and analysis of the surveys. More specifically, the survey process involved the following steps:

Scoping of the surveys;

Production and programming of the surveys;

Survey dissemination, data collection and management of the related activities (i.e. launching of the

online surveys, monitoring of survey results and undertaking of mitigation actions to ensure active

participation in the survey programme);

Data treatment and analysis by using appropriate analytical instruments aimed at producing analytical

reports and recommendations.

13

Figure 1. Methodology for the surveys

Source: PPMI.

Below we describe each step of the process in greater detail.

Scoping and sampling

Survey A

This survey targeted REA’s independent experts contracted to carry out evaluation and monitoring tasks. The

survey covered all independent experts contracted by the REA in 2014 and 2015.

Survey B

CORDA was the main source of information for surveying FP beneficiaries whose projects were managed by

the REA. The survey covered projects funded under the 2011-2014 calls for proposals.

Production of surveys

Annexes 3 and 4 present the questionnaires for the beneficiary and experts surveys, respectively. The

Steering Group was invited to review, comment on and modify these questionnaires in order to prepare them

for their launch in mid-September.

It is important to note that survey B was designed to avoid any possible duplications with the simplification

survey launched by the European Commission in late September 2015. The evaluation team was given access

to the set of questions to be asked in this survey. Consequently, any information deemed useful for answering

the evaluation questions that could be derived from this survey were utilised by the evaluation team. These

topics/questions were thus not be asked in the survey questionnaire designed for this evaluation.

Data collection

Following the launch of the surveys we monitored their progress and took risk mitigation (contingency) actions

for the satisfactory response rates. In addition, reminders to participate in the survey were sent. Both online

surveys were administered using the Fluid Surveys v.4.0 tool possessed by PPMI.

Data treatment and analysis of survey results

Once the surveys were closed, the collected data were categorised and analysed. To a large degree the initial

steps of this process were ‘automated’ by the pre-defined structure of the database. Detailed scrutiny of

responses was also carried out in order to detect any unforeseen data-collection/recording issues and to spot

any unexpected trends that emerged from the collected data. The table below summarises the response and

completion rates for surveys A and B.

Scoping Production and programming of

surveys

Survey dissemination, data collection and

survey management

Analysis of survey results

14

Table 3. Summary of survey response and completion rates

Title of survey

Number of possible

participants Responses

Response

rate

Completed

questionnaires

Completion

rate

Survey A: REA experts, evaluators 10321 5364 52.0 % 5083 49.2 %

Survey A: REA experts, monitors 574 321 55.9 % 305 53.1 %

Survey B: REA beneficiaries, FP7 3070 873 28.4 % 658 21.4 %

Survey B: REA beneficiaries, H2020 1629 542 33.3 % 480 29.5 %

Our team applied various tools and techniques in order to analyse the survey data, both quantitative and

qualitative. Our detailed approach to processing and examining statistics is presented in the section on

statistical analysis of the survey data.

On the basis of the survey results, we measured the expectations of the Agency’s clients, their perception

concerning the Agency’s performance and assess the gap between these two factors (expectations versus

perceptions). This provided information as to which aspects of organisational performance could be most

strengthened in the future based on their importance and service quality in the eyes of the beneficiaries.

Data protection

The research team fully acknowledged and respected the privacy rights of individuals and is committed to

handling any personal information obtained from participants in the survey programme or from the REA

and/or its parent DGs in accordance with the applicable law and regulations, as well as ensuring protection of

such data. At the beginning of each survey a clear description of its aims was provided to persons and

organisations invited to participate in the survey, so that they could be aware of the purpose of the collection

of data. For analysis and reporting purposes we used only aggregated survey results. Possibilities of disclosing

personal data to third parties were prevented.

Cost-benefit analysis

Cost benefit analysis (CBA) is one of a number of techniques of economic evaluation, a type of evaluation

which provides a comparative assessment of two or more courses of action, in this case the REA as opposed to

other organisational arrangements, in terms of their costs and consequences. Costs capture the resources

(such as staff) which are committed to an activity while the consequences are the results of those activities

which are of value to the funders of the activities concerned. The CBA is summarised by an assessment of:

effectiveness (in particular: the time to select, award and pay; client satisfaction globally):

efficiency (human resources needed at the REA and the Commission, administrative cost including

the cost for coordination and checks; assess any possible duplication):

possible savings (overall).

For this progress evaluation, the methodology and scope of the CBA followed the specific requirements set in

Article 3(1) of the Council Regulation (EC) No 58/2003 of 19 December 2002, in the Commission Decision of 2

December 2014 (C(2014)9109 final) establishing guidelines for the establishment and operation of executive

agencies financed by the general budget of the Union (specifically appendix II ‘Scope and methodology of a

Cost-Benefit-Analysis of the delegation of tasks to an executive agency’) and in the ToR. Following the

requirements of the documents indicated above, the CBA covered both quantitative aspects (which were

addressed in the retrospective CBA) and qualitative aspects (which were integrated into the overall evaluation

framework and evaluation questions).

In many CBAs, costs and consequences are both measured in monetary terms. In the case of the REA, the

costs were monetised but the consequences (qualitative aspects) were measured in terms of the measures of

organisational performance, or effectiveness, set out in the ToR.

When assessing the effectiveness of the REA it is necessary to consider the options against which it is being

compared. The relevant comparison could be an alternative way of organising the same activities. For

example, the previous CBAs compared the costs associated with REA with those which could be expected

under the alternative of carrying out the programmes’ management functions within the Commission.

15

The data inputs for the analysis were obtained from a number of key sources which can be categorised either

as desk research reviewing documents relating to the REA and stakeholder consultation. The desk research

particularly considered the CBA model (background files) from the previous CBA carried out in 2013 and the

necessary quantitative data (CORDA statistics and financial figures). Stakeholder consultation involved

interviews with key officials in the REA and parent DGs, interviews with other relevant stakeholders and focus

groups to test the initial findings. Surveys of beneficiaries were also an important source of evidence on the

agency’s performance.

16

Annex 2. REA interview questionnaire

Overview of REA and its evolution, including changes between the different programming periods

1.1. REA’s governance system

According to the plan, starting from September 2014 the REA was gradually delegated activities: Societal

Challenge 2, Societal Challenge 6, new parts of Societal Challenge 7 and the ‘FET open’ action from the

Excellent Science part of H2020. The transfer of activities from the H2020 Specific Objectives ‘Spreading

Excellence and Widening Participation’ and ‘Science with and for Society’ took place only in January 2015.

Was the handover of the abovementioned activities smooth? Some of the activities were delegated to

the REA before the calls were launched, while others – after the calls. If any, were there any

‘teething’ problems at the beginning of Horizon 2020?

The new mandate triggered changes in the governance of the REA: new parent DGs joined the Steering

Committee under Horizon 2020. A Memorandum of Understanding was signed between the parent DGs in late

2014. What were the key points of discussion and how did the decisions made affect REA’s governance and

management systems?

The EC guidelines for the establishment and operation of executive agencies financed from the Union

budget emphasise the agencies’ autonomy and dependence. If at all, how did the new agreement

between the parent DGs change this balance?

In one of the AARs the possible late adoption of the new mandate of the REA was considered a significant

risk, as it would delay the agency's readiness to take up new tasks. This would impact on the operation of the

agency as the recruitment of additional staff and programme implementation tasks would have to be delayed.

To what extent has this risk materialised in the transitory period between FP7 and Horizon 2020?

According to the EC guidelines for the establishment and operation of executive agencies financed from the

Union budget, the steering committee is made up of five members. In circumstances where the number of

parent DGs exceeds five, the steering committee and its chairmanship may function on the basis of a rotating

representation of the parent DGs. In your opinion, is this arrangement optimal for the REA?

In addition to Steering Committee and other high-level meetings, what mechanisms ensure that the

information prepared by the REA is communicated to and used by relevant units and officials in the EC?

Coordination and monitoring of the REA in the parent DGs: which types of resources were committed to the

activities? What kind of mechanisms, both formal and informal, facilitated the process?

Overall, how would you assess REA’s governance system? Some of the more specific issues to be discussed

include:

Did the REA and its Director effectively support the Steering Committee (& other stakeholders,

including the Programme Committee) by providing effective monitoring of its operations and

performance?

Did the REA and its Director effectively cooperate with other agencies in programmes where more

than one institution is involved in the management activities?

Did the REA and its Director effectively provide direction and coordination to programme

implementation in line with the Commission’s priorities?

Did the Steering Committee effectively coordinate the proper use of REA’s resources?

1.2. Organisational structure

The number of services managed by REA increased in H2020 in comparison with FP7, posing new challenges

for both the internal organisation of the REA and the coordination of its activities. The REA's mission

statement as of 2014 was updated taking into account the new mandate and units have adapted their unit's

mission statements accordingly.

17

The new mandate of the REA was supported by a new organisational structure, which entered into

force on 1 May 2014. In addition to the increased portfolio of programmes, what were other key

reasons for introducing the changes?

A staff redeployment exercise was achieved in April 2014, while internal mobility of staff continued in

2015. To what extent did this process influence the day-to-day operation of the agency?

Were alternative organisational structures considered? If yes, what were their comparative strengths

and weaknesses?

What are the major advantages of the new organisational structure? Would you see any risks and

disadvantages (e.g. compartmentalisation of activities; peak workload in some units may coincide

with low workload in other units – how is this handled in the agency?)


Union budget, the organisation charts of the agencies are reflected in the Commission's organisation charts

as follows: the structure of the agencies' organisation charts is reproduced in full or in part in the

organisation charts of the parent DGs (‘mirror’ chart). Is this arrangement optimal for both the REA and the

EC?

1.3. Cooperation between the EC and REA in preparation of FP7 and Horizon 2020 Annual Work

Programmes

The research and innovation DGs, under the lead of DG RTD, are responsible for the preparation of the

Horizon 2020 work programmes which are adopted by the Commission. The REA contributes to the

preparatory work of the Horizon 2020 work programmes on issues regarding the implementation aspects

(incl. dissemination, use and exploitation of results). Overall, would you assess this process as effective?

Were there any lessons learned?


Union budget, the director must ensure that the agency has recruited or is in the process of recruiting the

staff allocated to it as per the CBA. However, evidence shows that some of the assumptions in the CBA, for

example, overestimated the average grant size. What kind of corrective measures will be implemented to

narrow down the gap with the CBA assumptions?

1.4. Key simplification measures introduced in 2012-2015:

Examples of administrative novelties introduced in programmes managed by the REA under Horizon 2020

include:

Simpler management of projects: a single funding rate for all beneficiaries and all activities in the

same grant; no time-sheets for personnel working full time on EU projects; only one certificate on the

financial statements at the end of the project for beneficiary reaching triggering ceiling; the limitation

of the mandatory ex ante financial capacity check to private coordinators only and in case of funding

in excess of EUR 500 000; indirect costs covered by a single flat-rate applied to the direct costs; etc.

Acceptance of average personnel costs and beneficiaries’ usual accounting practices for direct costs.

‘No negotiation’ approach.

Guidance became centrally provided via National Contact Points

Evaluation criteria and sub-criteria were refined .

New budget flexibility measures were introduced.

To what extent did these simplifications affect REA’s key performance indicators? To what extent did these

simplifications affect REA’s ability to monitor and control the projects, foreseen impact on error rates? Were

there any other simplification measures introduced? In particular, what were the key simplification measures

introduced between late 2012 and 2015?

Client feedback and satisfaction: are there any indications which imply increased satisfaction from the

beneficiaries? If any, did the measures produce any unintended affects (e.g. the time-to-inform period of 5

months in Horizon 2020 may be perceived as a negative development even though the whole time-to-grant

period of max 8 months is an overall improvement compared to FP7)?

18

Evaluation of the simplifications introduced: excluding pre-financing, substantive spending on Horizon 2020

will pick up in 2016 and reach a significant level of 2017, at which point the level of error and the real impact

of simplification can be conclusively assessed. Considering the timing of this evaluation, it will be challenging

to assess the full impact of the simplification measures on some of the KPIs. Do you agree with this

assertion?

2. Effectiveness

2.1. REA’s operation in accordance to its legal framework

The overall EU legal framework governing the operation of executive agencies clearly defines the activities

that can be reserved to the executive agencies and the Commission:

In accordance with Article 6 of Regulation (EC) No 58/2003, the executive agencies may be entrusted

with the management of some or all of the stages in the lifetime of projects; provision of support in

programme implementation; and provision of administrative and logistical support services.

Pursuant to Article 58(7) of Regulation (EU, Euratom) No 966/2012, the following tasks are reserved

to the Commission: tasks involving a large measure of discretion implying political choices; decisions

submitted to comitology; and some other tasks, including inter-service consultations within the

Commission.

In your opinion, have there been any activities entrusted to the REA which pose a risk in terms of crossing

the line between the programme management/REA and policy development/EC sides?

Decision 2008/46/EC and (starting with 2014) Decision 2013/778/EU entrusted the REA with such tasks as

(1) managing some stages of programme implementation and some phases in the lifetime of specific projects

on the basis of the relevant work programmes; (2) adopting the instruments of budget execution for revenue

and expenditure and carrying out all the operations necessary for the management of the programme; (3)

gathering, analysing and passing on to the Commission all the information needed to guide the

implementation of the relevant Community programme parts; and (4) provision of logistical and

administrative support in programme implementation. In your opinion, has the REA carried out these tasks to

the highest extent possible (i.e. was fully coherent with the legal framework) or would you say that activities

in some of these areas could be given more attention?

In 2013/2014, the new Commission Decision establishing the REA and the new Delegation Act came into

effect. To the best of your knowledge, were there any problems related to this transitional period and in

particular to adjusting REA’s operations to the updated legal framework?

The parent DGs must define at the level of the Memorandum of Understanding a supervision strategy aimed

at avoiding gaps or duplication of efforts resulting from crossover between their monitoring and supervision

tasks and the execution tasks of the agency. How is this strategy implemented in practice? In your opinion, is

it effective? Do you see any possibilities to improve this framework?

Officials seconded to the executive agencies in the interests of the service maintain regular contact with their

home DG throughout their secondment. What is their role in ensuring that the REA abides by the policy

guidelines set by the Commission? What formal and informal mechanisms and tools facilitate this process?

How is it ensured that the existing legal framework/formal communication mechanisms do not involve ‘micro-

management’ by parent DGs and undermine the cost-efficiency of the delegation scenario?

Since its creation the REA has undergone continuous fine-tuning of the internal procedures and organisational

structures, streamlining its processes and integrating its activities in FP7 common tools. In your opinion, was

the operation of the REA flexible enough to accommodate the key changes, while at the same time

maintaining concordance with the legal framework establishing the agency? Are you aware of any challenges

related to this transition faced by the agency?

2.2. Extent to which the REA has achieved its objectives

Generally, how would you assess the overall operation of the REA in terms of its:

19

strengths (e.g. generally strong KPI results, particularly in execution on commitment and payment

appropriations, TTG, TTP, etc.);

weaknesses (e.g. staff turnover and its effects on the quality of the services);

opportunities (e.g. further consolidation of the activities following years of change);

threats (e.g. delays in payment appropriations, inherent risks in the SME legacy instrument,

complexity of Security Research activities, high error rate, expert payments; the extremely high

workload periods created by beneficiaries using REA’s central support services)?

When it comes to performance management of the REA (defining and estimating KPIs), is this process

effective? Do you see any challenges/problems in terms of defining KPIs and estimating them? What could be

improved in this regard?

Preparation of FP7 & H2020 AWPs: the research and innovation DGs, under the lead of DG RTD, are

responsible for the preparation of the Horizon 2020 work programmes which are adopted by the Commission.

The REA contributes to the preparatory work of the Horizon 2020 work programmes on issues regarding the

implementation aspects (incl. dissemination, use and exploitation of results). Overall, would you assess this

process as effective? Were there any lessons learned?

Is the REA equally effective in all stages of proposal evaluation/project management (call/proposal stage;

contract negotiations; implementation of the projects/project management; follow-up and audits)? In your

opinion, which stages of those mentioned above could be further improved?

Successfully launched calls and research topics are the key output of the REA. However, in some cases not all

planned research topics are funded due to a variety of reasons, including lack of interest from potential

applicants (i.e. cases where no proposals were received, or no proposal was of sufficient quality). What are

the ways in which the REA aims to ensure the sufficient production of key outputs (calls, proposals received,

grants awarded, and amount of commitment appropriations managed)? Do you see any potential ways for

improvement in this regard?

Overall, what evidence is there to suggest that the Executive Agency has fully achieved its objectives in the

2012-2015 period (both as provided in the legal basis and the annual work programmes)? Considering the

threats and possible weaknesses discussed above, are there any areas of concern? In your opinion, which

areas would be better addressed in case of an alternative scenario (i.e. the Commission-led services)?

2.3. Extent to which the REA led to an improved management of the programme(s), including in

terms of simplification measures, proximity to addressees and visibility of the EU

Examples of alternatives to the current executive agency management model include (1) management of the

programmes by the Commission; (2) mixed agency-Commission management; (3) partial management by

the Commission while outsourcing some activities to the extent legally possible. In your opinion, does the

current model ensure the best management of the programme and highest quality services to the

stakeholders? Would you say that other alternatives could improve the management of the programmes? In

which ways?

Please share your opinion on whether the current model is optimal in terms of (1) addressing resource

constraints (esp. during the periods of high workload); (2) capacity to provide specific skills.

A number of simplification measures (e.g. simpler form of grants, streamlining and harmonisation of funding

rules, etc.) were introduced during FP7. If you are aware of these measures, could you mention those that

seem most effective to you?

Could you also share your views on what type of simplification measures (and in which areas) could be

introduced in the new 2014-2020 financial framework? Please focus on such measures that, in your opinion,

could potentially contribute to the reduction of administrative burden both to the beneficiaries and the agency

as well as enhance the capacity to adapt to periods of high workload.

The REA has an objective to be as responsive as possible to the needs of beneficiaries both quantitatively

(e.g. in terms of TTG and TTP) and qualitatively (in terms of the beneficiaries’ satisfaction with the services

20

provided and overall involvement of the scientific community in the management of the programmes). In

your opinion, is the current extent to which resources and procedures are delegated to maintain contact with

beneficiaries appropriate? What could be changed? What impact does the involvement of institutional

stakeholders (esp. parent DGs) have on this process?

The REA is expected to act in compliance with the Commission’s guidelines on information and visibility of the

programmes, as well as the instruments put in place to ensure the visibility of the Commission as the

promoter of programmes entrusted to the REA. To the best of your knowledge, are the appropriate

funding/instruments in place to ensure the visibility of the Commission as a promoter of the programme?

In your opinion, is the REA brand well known among the beneficiaries? What would you say are the key

perceptions about the REA among beneficiaries and stakeholders? Would you say that these perceptions have

an impact on the effectiveness of the agency? What kind of impact?

2.4. REA as a learning and client-oriented organisation

This aspect of the evaluation will be primarily assessed via beneficiary and staff surveys. Your feedback,

however, may help identify a number of good practices. To this end, please provide outstanding examples of:

Improvements of processes aimed at internal (i.e. REA’s staff) or external users (i.e. beneficiaries)

that have been realised in the past years/months.

Arrangements to ensure a higher level of flexibility when responding to the needs of stakeholders

(applicants and beneficiaries, the Commission, other stakeholders) in the implementation of the

delegated tasks.

Effective learning at the organisation/take-up of lessons learned and recommendations suggested;

Streamlining and harmonisation of funding rules and procedures across different programmes and

units.

Improvements in IT systems and communication with beneficiaries.

Improvements in HR policy.

3. Operational efficiency

3.1. Detailed deconstruction of key steps of the call/application/project management life-cycle

To be discussed:

Call stage: consultations with the Commission, managing calls publications, budgeting…

Application stage: information for applicants, reception of proposals, admissibility and eligibility

check, evaluation, respective indicators (time-to-inform, redress, budget execution/success rates,

etc.)

Conclusion of grant agreements: validation of beneficiaries, financial capacity and cross-checks, grant

finalisation and contracting, respective indicators (TTG)

Follow- up and monitoring: pre-financing, interim and final payments, reporting, amendment of grant

agreements, respective indicators (TTP, TTA)

Ex-post controls and recoveries: audit strategies and plans, ex-post audits, corrections/recoveries

Horizontal question: new H2020 rules and procedures throughout programme/project management

cycle (clarity, stability, suitability, efficiency, etc.)

3.2. Overall efficiency of REA’s activities

General discussion on the achievement of REA’s key performance indicators and the progress made in 2012-

2015. Some of the more specific issues to be addressed during interviews include:

Budget execution of commitment and payment appropriations. Reports show that the

operational commitment and payment appropriations from the EU general budget (excluding

earmarked revenue) were fully executed, while the execution rate for the (non-differentiated)

administrative appropriations reached 97.5 % in 2013.

Time-to-grant (TTG). TTG figures show an overall improvement over 2012-2015, with H2020 calls

significantly outperforming FP7 programmes.; For the first H2020 calls the TTG remained below the

defined target of 8 months (245 days) with an average of 217 days;

21

o Cumulative TTG for FP7 is higher than that for H2020 projects -> was this mainly achieved

due to the simplification measures and maximum durations introduced under Horizon 2020

(Total TTG up to max. 8 months (245 days) including a TTI up to max. 5 months (153

days)); or also additional simplifications/other efforts made by the REA?

o TTG for ITN in 2014 (258 days) was above the target of 245 days – what were the key

reasons behind this outcome?

o In 2012/2013 TTG targets were not reached for COFUND and SME actions, also Security

o Regarding the new limit of 5+3=8 months for TTG, minutes of a Steering Committee

meeting suggest that there has been some negative feedback from stakeholders as

evaluation results were now communicated later (after 5 months) than in the past (after 3-

4 months) and that stakeholders perceived one target of 8 months rather than two

separate targets (for 5 months and for 3 months). Has this aspect been addressed and

communicated to the beneficiaries?

Redress cases. The target for the agency is 0 % of redress cases upheld. Actual data show that less

than 1 % of redress cases were upheld in late year’s FP7. What has been the trend in the number of

complaints received? Is there an increasing trend due to the growing number of proposals or, for

example, introduction of two-stage calls?

Error rate. The target was set at 2 %, however data show that the residual error rate was frequently

not met in Space and Security (error rates of about 3 %), SME programme (5 % residual error rate

and over 10 % non-registration of RTD performer invoices in 2014), but just over 1 % in People.

o What are the inherent risks in the above-mentioned programmes (e.g. subcontracting in

SME instrument, fraudulent behaviour)?

o What are the most recurring errors in FP7 and H2020 programmes (e.g. wrong calculation

of hourly rates, absence of time recording systems and wrong calculation of overhead

rates…)

o The error rate for some instruments increased since 2012 (e.g. for SMEs) – was this the

result of more intensive ex-post controls introduced in later years of FP7?

o According to the 2014 AAR, the high rate of error for the SME actions justifies maintaining

a reservation for this programme while the lower error rate for the People Programme

confirms the REA’s assessment about reasonable assurance that this programme is not

impacted by material errors. As of 2015 the REA will put in place new ex-ante controls

(requesting SMEs to provide evidence that the RTD performers’ invoices have been issued,

registered and paid) to make sure that SME beneficiaries comply with the rules – what have

been the experiences to date?

Time-to-pay (TTP): during 2014 97 % of project-related payments were made on time. Over FP7,

90-95 % of interim and final payments were made on time and around 97-99 % of pre-financing

payments. There was a substantial improvement in 2013 and 2014 – what factors drove this result?

TTP for H2020 first interim payments? Other issues to be discussed:

o For payments to experts, the REA temporarily experienced difficulties in the summer

months due to high workload, staff changes induced by the reorganisations and certain

delays in the recruitment of new staff. Review of the AAR suggests that the same situation

occurred in both 2013 and 2014.

o Some 89.7 % of the administrative payments were made on time (as a rule within 30

days from receipt of payment request), which is an improvement of 7.5 percentage points

compared to 2012.

Average grant size & number of projects per officer. The AARs suggest that for some of the

newly delegated actions, the significantly lower grant size was only partly counterbalanced by a

slightly reduced budget delegated, particularly for Space and Security research where the average

grant size is at around 50 % of what was assumed in the CBA. What measures have been/will be

taken to address this situation in future calls?

Achievement of objectives. The KPI which indicates the share of closed projects that achieved all

or most of their objectives reached about 97-98 % in 2012-2014. The target set by the REA is 90 %.

Has there been discussion on this indicator with respect to its target value?

External evaluations. In 2013 the REA started using external evaluations, thus considerably

reducing the travel and accommodation costs of experts and related carbon emissions. The savings

made on experts' fees and allowances can be estimated to reach up to EUR 25 000-100 000 per call

22

depending on the number of experts attending the briefing. What are the key risks associated with

this move and how has the agency mitigated them?

3.3. Operational budget

Financial management of the operational programmes delegated to the REA is largely decentralised to

operational units who have full control over initiation and authorisation of commitments, payments and

recoveries. The central Finance Unit performs independent financial verification and provides assurance to the

Director on the proper implementation of the operational budget throughout the REA.

In your opinion, what are the main advantages and disadvantages of a decentralised financial

management system? Would it be feasible in principle to adopt a more centralised system? What

would be the costs/disadvantages of such a system?

The lack of payment appropriations was a significant risk in relation to the REA's ability to implement the

programmes smoothly and to reach its targets in terms of TTG and TTP. Various reports indicate that the risk

materialised in 2013 and 2014 as the increased budget had not been fully obtained through the amending

budget procedure and the global transfer.

Which areas/programmes were most affected by the lack of payment appropriations and how did this

affect the agency’s KPIs?

What interventions were made to ensure sufficiency of payment appropriations and minimum delay of

pre-financing, interim and final payments?

To what extent did this lack affect the achievement of the policy objectives for FP7 as well as for

Horizon 2020? What mitigation and control measures can be adopted in in the future to avoid a

similar situation?

3.4. Administrative budget and expenditures

General discussion on the evolution of the administrative budget and its main budget titles during 2012-

2015:

Title I. Staff expenditure comprises the following cost items:

o Remuneration, Allowances and Charges

o Professional Development and Social expenditure

Title II. Infrastructure and operating expenditure includes:

o Building expenditure

o ICT expenditure

o Movable property and Current operating expenditure

Title III. Programme support expenditure:

o Programme management expenditure

o Common support services expenditure

According to the AARs the REA's administrative budget, as a share of the operational budget, remained in the

range of 3 % (just over 2.3 % if central support services are excluded). This figure was substantially lower

than the 6 % ceiling in the FP7 legal base. What are the corresponding figures for 2014-2015 and the

expected future developments, especially considering the expansion of central support services?

Future developments in the agency’s operational costs: the main changes from 2015 to 2016 concern a

decrease in the expenditure related to the evaluation platform, an increase in building-related expenditure, a

further decrease of ex-post audit expenses and an increase in missions. Have the necessary provisions been

sufficient and made on time?

Staff costs (discussion on their evolution for the different types of staff): according to Article 18 of Regulation

(EC) No 58/2003, the staff of the executive agencies consists of:

officials seconded in the interests of the service and engaged by the agency as temporary staff within

the meaning of Article 2(a) of the Conditions of Employment of Other Servants (CEOS) (AT2a) in

positions of responsibility;

temporary staff within the meaning of Article 2(f) of the CEOS (AT2f). Officials seconded at their own

request by an Institution are engaged as AT2f;

contract staff within the meaning of Article 3a of the CEOS (AT3a). The number of contract staff may

not exceed 75 % of the total number of staff employed in an agency;

23

the agency may use seconded national experts (SNEs) and other types of staff under private-law

contracts.

3.5. Internal control systems & audits:

Overview of the REA’s internal and external control systems, particularly:

REA’s Internal Audit Capability (IAC);

the Commission’s Internal Audit Service (IAS);

the European Court of Auditors (ECA).

+ discussion on key mechanisms designed to monitor the functioning of the internal control systems:

bi-annual reports submitted by the Heads of Unit in their capacity of Authorising Officer by Sub-

Delegation (AOSD);

opinion on the state of control of the REA's Internal Control Coordinator (ICC);

outcomes of activities of the ex-post audit function and fraud prevention measures;

independent opinion of the REA's Internal Audit Capability (IAC) on the state of internal controls;

observations and recommendations reported by the Internal Audit Service (IAS);

observations and recommendations reported by the European Court of Auditors (ECA);

observations and recommendations reported by DG BUDG (in the context of the validation of the local

accounting systems by the Commission's Accounting officer).

Multiple reports suggest that the agency’s management has reasonable assurance and that, overall, suitable

controls are in place and working as intended. The risks are being appropriately monitored and mitigated, and

necessary improvements and reinforcements are being implemented. However, the Director, in his capacity

as Authorising Officer (AO) for the administrative budget and Authorising Officer by Delegation (AOD)

expressed reservations for the ABB activities Space and Security (Cooperation Specific Programme) and

Research for the Benefit of SMEs (Capacities Specific Programme).

What have been the key intrinsic risks in Space and Security research and the SME instrument (i.e.

transactions do not reflect normal market conditions; SME bankruptcies; non-recording the RTD performance

invoices in the SMEs accounts)?

To what extent and how have these risks been mitigated (e.g. through additional ex-ante and ex-post

controls)?

Key stakeholders, however, have been concerned about the administrative burden imposed by the FP7

managing services and have requested a re-balancing of trust and control. For this reason the REA and the

Commission services have reduced their level of ex-ante control (whilst respecting the requirements of the

Financial Regulation) and have decided to obtain most of the assurance from ex-post controls.

What are the possible implications and risks of the strategic move from ex-ante to ex-post controls?

What implications does this strategic change, as well as the higher SME participation rate in Horizon 2020,

have for the error rate?

One of the reports states that the target of a maximum residual error rate of 2 % cannot be reasonably met

without a massive increase in the number of audits or in the administrative burden imposed on participants

through widespread ex-ante controls.

Are there any alternative approaches that could improve the residual error rate without excessive controls

(e.g. more extensive use of flat rates, unit costs and lump sums)?

Would additional automated tools aimed at detecting errors/fraud help reduce the agency’s exposure to risk?

A discussion on the changes and simplifications introduced between 2012 and 2015:

The Common Representative Audit Sample (CRaS) was introduced in 2012 across the research family.

In H2020 ex-post audits became in the remit of the Common Support Centre.

In 2014 the REA performed an assessment on the effectiveness of the ICS according to ICS 15 with the

support of the IT tool (iCAT, internal Control Assessment Tool) provided by DG BUDG in order to obtain a

diagnostic of strengths and weaknesses in the REA's internal control framework and to identify further actions

for improvement. What were the results of this assessment?

24

3.6. IT resources

Following the implementation of the IT Master Plan in 2014, completely new IT tools and workflows are used

for H2020. Following this change local IT developments were reduced to a minimum and focused only on

those processes for which no corporate tools were available. More specifically, the REA maintained its local

project support tool PROMIS, which complements the central FP7 grant management. For H2020, the

corporate workflow tool Compass and the Sygma replaced the PROMIS functionalities and Coreflow was used

for FP7 legacy.

How would you assess the appropriateness/functionality of the aforementioned IT tools? What are the

relative advantages/disadvantages of Compass and the Sygma compared to their predecessor?

How would you assess the availability/appropriateness/functionality of the common H2020 dedicated

IT tools

Staff training: the AARs and other documents imply that the REA staff have been intensively trained in the

new IT systems (Sygma, COMPASS) and continued attending training courses about the new rules and

procedures for the new Framework Programme. Has the Agency collected feedback on these training

activities and monitored the progress?

Future plans and developments with IT becoming a common service: in cooperation with the CSC, the REA is

active in the definition of the common procedures and the further development of the corporate IT tools. In

particular the REA will further invest in fine tuning the business process and the related IT tools:

for the centralised contracting and payment of expert evaluators for all H2020 calls and for the

validation of participants;

an automated tool for internal control systems was also being assessed;

Other tools (to be discussed; e.g. in 2015 the focus has been on grant management IT tools).

3.7. Internal and external communication, visibility, proximity to beneficiaries

In cooperation with the parent DGs and the H2020 Common Support Centre, the REA implemented the

dissemination strategy for results of Horizon 2020. What are the main principles of this strategy and can the

evaluation team access the related document?

Reports suggest that during 2012 the REA launched a number of information campaigns aimed at the

beneficiaries and the wider society. Those activities were at least of the following types:

Various conferences, workshops and events promoting the programmes/calls and disseminating the

research results

Information campaigns highlighting key management and control issues, such as the mass information

campaign highlighting actions to correct the errors in the SME instrument.

Which of these activities prove to be most effective and most valued by the beneficiaries? To what extent

could the beneficiaries make impact on the programme/set the agenda for these communication events? How

were the results or even impacts, as well as the quality of these activities evaluated?

Various activities raising staff awareness on the REA’s external communication strategy we implemented

during 2012-2015. What evidence is currently available which can be used to assess their effectiveness?

Internal communication: what key mechanisms and tools ensure the effectiveness of internal

communications? What is the feedback received via the staff survey?

Balancing external communication: the AARs suggest that the REA will revise its strategy for internal

communication and continue to deliver on its external communication strategy, with particular focus on the

communication on project results and success stories.

3.8. HR policy

Discussion on the main HR activities and policies implemented during 2012-2015:

25

finalisation of the internal redeployment exercise linked to the new mandate and to implement the

new REA organisation chart by the beginning of the second quarter;

staff establishment plan for 2014-2020;

staff employed within the REA expressed in FTEs throughout 2013-2015

soordination and implementation of an effective recruitment process to fill vacant posts

staff appraisals for the REA staff (for 2013), including objective setting for 2014;

to adapt the REA Implementing Provisions to the changes in the Staff Regulations and the

Commission's General Implementing Provisions

to work together with DG HR and the other executive agencies to achieve the implementation of the

new system for staff appraisal in executive agencies in good time for the appraisal exercise that is to

be conducted in early 2015;

to support the activities of the REA Staff Committee and moderate the social dialogue between the

Committee and the REA management;

to review and implement the Agency's Learning and Development Framework for 2014 taking into

account, in particular, the training needs arising from the new actions delegated to the REA, the new

H2020 rules and business processes and the new corporate IT tools (in cooperation with the Common

Support Centre in DG RTD);

to continue implementing the action plan developed following the 2012 staff satisfaction survey

(including actions concerning staff mobility, well-being, teleworking and equal opportunities) and

conduct a new survey by end-2014.

In the context of changes to the mandate of the Agency, how did you ensure a clear delineation of tasks and

prevent functions from being duplicated?

How were the changes to the mandate of the REA addressed in the training strategy, as well as in

organisational and individual development plans of the REA staff?

What are the key practices to identify the current competences of people at the individual and organisational

levels in terms of skills and expertise? How are the impacts of training and development programmes on the

workplace and transfer of knowledge to colleagues being monitored?

Discussion on the evolution of staff in the REA (number of staff by category, salary levels, etc.): according to

Article 18 of Regulation (EC) No 58/2003, the staff of the executive agencies consists of:

Officials seconded in the interests of the service and engaged by the agency as temporary staff within

the meaning of Article 2(a) of the Conditions of Employment of Other Servants (CEOS) (AT2a) in

positions of responsibility;

Temporary staff within the meaning of Article 2(f) of the CEOS (AT2f). Officials seconded at their own

request by an Institution are engaged as AT2f;

Contract staff within the meaning of Article 3a of the CEOS (AT3a). The number of contract staff may

not exceed 75 % of the total number of staff employed in an agency.

The agency may use seconded national experts (SNEs) and other types of staff under private-law

contracts.

Average staff costs per category (TA/CA) in 2013-2015 and expected future development.

To what extent does the HR policy support the development of a performance culture in the REA? Can you

provide any examples of concrete schemes, measures to encourage better individual and team results?

Planning of human resources: the agency’s work programme must include a clear presentation of the

operating budget of the agency, as well as the staff figures with an indicative breakdown per programme and,

within each programme, per activity and with an indicative breakdown for the different types of agents.

Based on your experience, have these plans accurately predicted the actual demand for human resources in

the different programmes and activities? How would you assess the adequacy of the staffing level and

workload, possibilities to adapt to changing workload?

High staff turnover is a well-documented structural weakness of the executive agencies which negatively

affects their performance and results in higher programme continuity costs, opportunity costs of staff

replacement and general loss of know-how in the organisation.

Staff turnover in 2012-2015?

26

To what extent and in which areas has staff turnover affect the agency’s performance?

Degree of difficulty to find the right profiles (time spent to recruit specific profiles and number of

candidates for each job advertised)/

What are the main reasons for staff turnover (e.g. relatively low salary level of CAs, other)?

To this end, how can HR policy help mitigate these risks (e.g. specific policies whose goal is to retain

the agency’s top performers)?

Overall, to what extent have the following objectives been reached?

effective recruitment of staff and equal opportunities policy;

maintenance of a constructive social dialogue, especially with the Staff Committee;

staff training;

career development and motivation;

overall staff satisfaction.

4. Utility

4.1. Extent to which the Executive Agency enabled the Commission to focus on its policy-related

tasks

The parent DGs define a supervision strategy aimed at avoiding gaps or duplication of efforts resulting from

crossover between their monitoring and supervision tasks and the execution tasks of the agency. The

supervision strategy and cooperation modalities defined in the memorandum of understanding should be

regularly reviewed and updated if need be to take into account any new developments and ensure that they

are fit for purpose. Specifically, such mechanisms as organisational set-up of the agency and process of

approval of annual work programmes, the AAR, the agency’s budget and accounts by the steering committee

ensure that supervision is implemented.

Overall, does clear delimitation of responsibilities and tasks between the REA and the parent DGs

exist?

Has the creation of the EA enabled the Commission staff to focus better on the institutional tasks as

compared to the alternative options?

Are there any examples of formal and informal activities undertaken by the Agency that could be

classified as policy-making?

The appropriate monitoring of the agency’s activities is primarily achieved by means of participation of the

parent DGs in the steering committee by regular coordination meetings at management level and further

contacts at working level. In the steering committee the representatives of the parent DGs verify whether the

provisions of the memorandum of understanding regarding reporting and supervision of the EA are duly and

timely fulfilled and how the agency has progressed with regard to the operational objectives and performance

targets of the work programme. In addition, the chairman of the agency’s steering committee informs the

Directors-General of the parent DGs about the decisions adopted by the committee and about any relevant

subjects regarding the agency’s work.

To which extent do these mechanism ensure that there is an adequate flow of information and

collaboration between the REA and the parent DGs?

Do the relevant stakeholders agree that the mechanisms and instruments are appropriate in ensuring

adequate coordination and information flow between the REA and the Commission?

Extent to which the stakeholders agree that the information provided by the REA is

o relevant, up to-date and of high-quality;

o provided in a user friendly form, well structured, well communicated and easily accessible;

o delivered and published on time;

What other possible mechanisms exist/are needed to enable the flow of information between the

Agency, and the Commission?

4.2. Extent to which the activities of the Executive Agency resulted in unintended effects (both

desirable and undesirable)

The main unintended effect that was found in the previous evaluation of the REA was the higher than initially

expected staff turnover. In addition, in relation to the start of H2020 and a new mandate for the REA, there is

27

the risk of a drop in performance due to the impact of the change management process. New programmes

were delegated to the REA with a high impact on its organisational structure, including internal redeployment

and transfer of staff from the Commission. At the same time, the new rules and modalities of H2020 will

bring about significant changes in the administration of the programmes. In particular, the roll-out of the new

corporate IT tools poses a risk of reduced efficiency due to the need for bug-fixing and fine-tuning.

Have these changes and novelties produced any undesirable effects?

Is there any evidence of simplification measures being perceived negatively by the beneficiaries?

Is there any evidence of good practice examples of agency’s activities emerging that were identified by

the stakeholders of the REA in the period between 2012 and 2015?

5. Maintaining sufficient know-how in the Commission

5.1. Extent to which the Commission, in the presence of the REA, been able to maintain the

adequate level of know-how in relation to programme(s) entrusted to the agency

Officials seconded by the Commission to the EAs play a particular part in the organisational setting of the

Agencies. They are entrusted with positions of responsibility and are characterised by their twofold statutory

link to both the Commission (as seconded officials) and the agency (as temporary agents). As a result, they

contribute to the transfer of know-how between the Commission and the agency.

To which extent does the seconded staff ensure the flow of know-how between the REA and the

Commission?

To what extent is this process adequate and efficient?

Is the know-how provided by the seconded officials of high quality and up-to date?

What barriers exist in the flow of know-how?

What other mechanisms exist or are needed to foster the flow of know-how between the REA and the

Commission?

Examples of possible improvements that could be implemented in the process.

5.2. Extent to which the monitoring and reporting arrangements in place enabled the Commission

to benefit, in the short and medium term, from the know-how created within the REA

The monitoring of REA’s activities is ensured through participation of the representatives of parent DGs in the

steering committee. They participate in the steering committee meetings and review and scrutinise all the

information provided by the director of the Agency. Their role also involves verifying as far as feasible

whether the information provided by the REA is reliable and sufficient to draw management conclusions at DG

level (e.g. DG’s annual activity reports).

To what extent is the monitoring information provided by the REA to the Commission clear, relevant, of

high quality and timely?

Has the Commission benefited via monitoring and reporting arrangement in short or medium term?

To which extent the know-how helped to serve the needs of policy units of the Commission?

Are there any examples of innovations adopted by the Commission as a results of knowledge sharing

with REA?

5.3. Adequacy of information flow and communication between the REA and the Commission

services (in particular the parent DGs) exist?

The annual planning and reporting cycle form the basis of the monitoring of the agency’s activities, mainly

through the agency’s annual work programme, annual activity report and interim reporting. The content,

format and frequency of reporting is defined in the Memorandum of Understanding between the Commission

and the Agency. Through the agency’s reporting obligations the DGs are always kept informed of the

agency’s work and any management difficulties.

To what extent are the monitoring and reporting arrangements adequate?

What other arrangements (apart from those mentioned above) exist between the REA and the

Commission?

Are quality, clarity, relevancy and timeliness of the information provided to the Commission up to the

highest standards?

28

What checks are put in place for ensuring that the right information is communicated to the right person

at the right time?

5.4. Extent to which closure of the agency would result in losing significant know-how within the

EC in relation to the management of programmes(s) entrusted to the REA

Under certain circumstances the EA could be closed and the management of the programme returned to the

Commission (e.g. the management would prove to be cost-inefficient).

Under which risks/scenarios would the agency be closed? How likely are these risks/scenarios?

Would this result in a loss of know-how within the Commission?

What types of know how would be lost?

6. Management and provision of central support services

Under Horizon 2020 the REA’s mandate was further extended to provide administrative and logistical support

services to all entities involved in Horizon 2020 management; and also expanded to certain programmes for

health/consumer protection, competitiveness and innovation, as well as education, culture and citizenship.

The REA was in charge of the following support services:

call planning

support for call publication

general logistical support for the evaluations

contracting and payment for independent experts (expert evaluators and expert monitors (‘reviewers’,

only for actions managed by the REA))

validation of legal entities (legal validation and SME validation)

preparation of legal entities financial viability assessment

management of the Research Enquiry Service

centralised handling of evaluation review requests

maintenance of the National Contact Points database

What have

Date post:	23-Aug-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

EVALUATION OF THE OPERATION OF REA (2012-2015) · E-mail: [email protected] European...

Documents