STUDY EPRS | European Parliamentary Research Service
Ex-Ante Impact Assessment Unit PE 642.807 – December 2019 EN
Appraising the quality of the
European Commission's
impact assessments
Trends and developments
from 2015 to 2018
EPRS | European Parliamentary Research Service
Appraising the quality of the European
Commission's impact assessments
Trends and developments from 2015 to 2018
This publication provides an assessment of the quality of European Commission ex-ante impact assessments (IAs) published between July 2015 and December 2018, on the basis of their appraisal by the Ex-Ante Impact Assessment (IMPA) Unit of the European Parliamentary Research Service (EPRS). This study, which draws on a review of 132 initial appraisals of Commission IAs, analyses the IAs with respect to quality criteria stemming from the Commission's Better Regulation Guidelines and relevant Parliament resolutions. The review draws on the statistical data collected by the IMPA unit and on a qualitative analysis of the Commission's IAs. The aim is that the findings of the review will feed into the broader ongoing discussion on better regulation in the European Union, and impact assessment in particular, in the context of the Commission's recent stocktaking exercise on its better regulation policy.
AUTHORS
Alexia Maniaki-Griva, Esther Kramer and Vadim Kononenko, Ex-Ante Impact Assessment Unit
The authors would like to thank Elisa Telesca and Giulio Sabbati for their important contribution to the present study.
This paper has been drawn up by the Ex-ante Impact Assessment Unit of the Directorate for Impact Assessment and European Added Value, within the Directorate-General for Parliamentary Research Services (EPRS) of the Secretariat of the European Parliament.
To contact the authors, please email: [email protected]
ADMINISTRATOR RESPONSIBLE
Alexia Maniaki-Griva, Ex-Ante Impact Assessment Unit
To contact the publisher, please e-mail [email protected]
LINGUISTIC VERSIONS
Original: EN
Manuscript completed in December 2019.
DISCLAIMER AND COPYRIGHT
This document is prepared for, and addressed to, the Members and staff of the European Parliament as background material to assist them in their parliamentary work. The content of the document is the sole responsibility of its author(s) and any opinions expressed herein should not be taken to represent an official position of the Parliament.
Reproduction and translation for non-commercial purposes are authorised, provided the source is acknowledged and the European Parliament is given prior notice and sent a copy.
Brussels © European Union, 2019.
PE 642.807 ISBN: 978-92-846-6064-3 DOI: 10.2861/968302 CAT: QA-03-19-911-EN-N
[email protected] http://www.eprs.ep.parl.union.eu (intranet) http://www.europarl.europa.eu/thinktank (internet) http://epthinktank.eu (blog)
Appraising the quality of the European Commission's impact assessments
I
Executive summary
From the start of its mandate in 2014, the Juncker Commission committed strongly to the better regulation principles, stressing the importance of evidence-based and transparent policy-making, to 'deliver better rules for better results'. In May 2015, it published the better regulation agenda1 – an extensive package of new measures to improve and strengthen policymaking – to cover all stages of the policy cycle, including the ex-ante impact assessment (IA) of the potential impacts of EU policy initiatives, which the Commission committed to carry out when the expected economic, environmental or social impacts of EU action are likely to be significant.
In 2018, towards the end of the Juncker presidency, the European Commission took the initiative to take stock of its better regulation activities, including those relating to ex-ante IA. The recent stock-taking exercise2 and, more importantly, the start of a new legislative term with a new European Commission provide the political momentum for enhanced cooperation between the European institutions and exchange of best practices in the realm of ex-ante IA, in line with the commitments they made in this respect in the 2016 Interinstitutional Agreement on Better Law-Making (IIA BLM).3
This study is intended to contribute to this momentum and the ongoing discussion on better regulation in general and on IA in particular. It assesses the quality of the European Commission's IA reports accompanying legislative proposals adopted between July 2015 and December 2018 and appraised by the Ex-ante Impact Assessment (IMPA) unit of the European Parliamentary Research Service (EPRS), with a view to supporting Parliament's capacity for scrutiny of the executive. Concretely, the study draws on a review of 132 initial appraisals of Commission IA reports published during the above-mentioned period. It is based on a scoring exercise of the 10 main dimensions of each IA covered in an initial appraisal, rating each of these dimensions separately (on a range from 1 to 5, corresponding to the perceived quality of 'very poor, poor, satisfactory, good, very good'). It follows a previous review of the first 100 initial appraisals of IAs carried out by IMPA between June 2012 and June 2015.4
The statistical data gathered by IMPA indicate that 66 % of the appraised IAs published between July 2015 and December 2018 were found, on average, to be of overall 'satisfactory' quality, while 5 % were considered to be 'good' and 29 % 'poor'. The overall average quality of the IAs appraised across the 10 dimensions mentioned above slightly exceeded the satisfactory level throughout most of the period under review. These results are in line with the generally perceived positive trend concerning the quality of impact assessment reports and the related consistent efforts of the Commission in the framework of the better regulation agenda. They nevertheless also show that there is still room for improvement. More specifically, this review shows an improvement in the quality of IAs between 2016 and 2017, most likely resulting from the implementation of the Better Regulation Guidelines and their revision in 2017, and a slight decrease in quality between 2017 and 2018. In line with what was already observed in the past,5 this phenomenon could be explained in
1 European Commission communication, Better regulation for better results - An EU agenda, COM(2015) 215, 19.5.2015. 2 European Commission communication, Better regulation – taking stock and sustaining our commitment,
COM(2019) 178, 15 April 2019; more detailed information on this stocktaking exercise can be found in the accompanying staff working document, SWD(2019) 156, 15 April 2019.
3 2016 Interinstitutional Agreement on Better Law-Making, OJ L 123/1, 12.5.2016. 4 See J. Dunne and W. Hiller, European Parliament work in the fields of impact assessment and European added value.
Activity Report for July 2014 – December 2015, Annex One. The first 100 initial appraisals of European Commission Impact Assessments by the European Parliament's Ex-Ante Impact Assessment Unit between June 2012 and June 2015, EPRS, European Parliament, April 2016.
5 For a similar observation, see the review of initial appraisals carried out by the IMPA unit in 2015, see J. Dunne and W. Hiller, p. 42.
EPRS | European Parliamentary Research Service
II
part by the increase in the number of legislative proposals towards the end of the legislative term and the growing time constraints that the author DGs of the Commission were facing when finalising the impact assessment reports.6
An evaluation of the statistical data relating to the 10 main dimensions appraised shows that, as the Commission's Regulatory Scrutiny Board also observes,7 IAs can be strong in one area but weak in another. Whereas an IA's overall score might be 'satisfactory', that does not rule out the fact that some individual aspects of the IA might be of poorer or higher quality. The purpose of this review is to pinpoint the strengths and weaknesses of each dimension, and to identify evolutions and trends over time, with a view to highlighting the areas in need of further improvement. That said, it should be kept in mind that the wide range of EU legislation and, as a result, the divergent requirements and conditions, notably in terms of data availability, research and methods for conducting IAs, are important drivers of an IA's quality.
IMPA's analysis of each of the 10 dimensions shows that, on average, the best scored areas were the problem definition, the provisions for monitoring and evaluation, and the coherence of the proposal with the IA. The weakest scores were given to the presentation of policy options, and the assessment of their relevant direct and indirect impacts, two essential sections of any IA. Based on these findings, the study identifies the need for a more thorough assessment of social and, in particular, environmental impacts, and more consistent analysis of impacts on SMEs and competitiveness. Depending on the initiative, better screening and identification of potentially relevant impacts is necessary, including effects on fundamental rights, and on territorial and gender equality related impacts. This analysis also highlights that a clear and balanced presentation of the policy options is crucial for the transparency, objectivity and accountability of an IA. Moreover, it stresses the importance of a distinct definition of general, specific and operational objectives – not only for the coherent intervention logic of any IA, but also for the effectiveness of the monitoring and ex-post evaluation of an initiative.
While this review found the data and research used in IAs to be generally reliable and solid, it detected in some instances a lack of transparency, owing to inconsistent data or inaccessible sources, as well as a lack of explanations of the methods, assumptions and uncertainties of the IAs. The same is true for the often very general reporting of the results of stakeholder consultations in the IAs, which should describe how stakeholder views fed into the IA and policy choices in a more transparent manner. With respect to the quantification of costs and benefits, this study points to the need for consistency and for balance of qualitative and quantitative analyses, to ensure both the proportionality and accountability of future IAs. The study notes the recent 'back to back' practice, where, owing to time constraints, the Commission has carried out ex-ante and ex-post assessments in one (short) exercise. IMPA invites the Commission to further clarify the methods employed in these cases, with a view to sustaining the distinct objectives, the quality and the accountability of both ex-ante and ex-post assessments in the long term.
Finally, this study sees scope for improvement when it comes to the explanation and substantiation of the respect of the principles of subsidiarity and proportionality in IAs. In this context, it welcomes the recent announcement of the Commission to integrate a new model grid into every IA that was prepared by the Task Force on Subsidiarity, Proportionality and ‘Doing Less More Efficiently’, as a possible measure leading to improvements.
With this study, and in its work more generally, the IMPA unit supports committees' scrutiny role, but also aims to contribute to the shared effort of bringing about further improvements in the quality of law-making in the EU, by promoting the value of IAs as an aid to decision-making and
6 See W. Hiller, European Parliament work in the fields of impact assessment and European added value, activity report
for 2018, p. 21; see also the Regulatory Scrutiny Board Annual Report 2018, p.9. 7 Ibid, p. 12.
Appraising the quality of the European Commission's impact assessments
III
helping to ensure that IAs are of the highest possible quality so as to inform decision-makers effectively. In this regard, the need to strike the right balance between the depth and complexity of research needed to inform policymakers properly on the one hand, and the readability and transparency of IAs on the other, appears to be a common overarching challenge for the entire IA process, and looks likely to continue to be so.
EPRS | European Parliamentary Research Service
IV
Table of contents
1. Introduction __________________________________________________________________ 1
2. Impact assessment in the European Commission and its appraisal in the European Parliament 2
a) Institutional developments and practical aspects of ex-ante impact assessment at the European Commission ______________________________________________________ 2
b) Appraisal of the Commission's impact assessments in the European Parliament _________ 3
3. Main observations and trends ____________________________________________________ 6
Methodology __________________________________________________________________ 6
Problem definition ______________________________________________________________ 7
Policy objectives ________________________________________________________________ 7
Policy options __________________________________________________________________ 8
Assessment of impacts __________________________________________________________ 11
Subsidiarity and proportionality __________________________________________________ 15
Quality of data, research, and analysis _____________________________________________ 16
Stakeholder consultation ________________________________________________________ 19
Monitoring and evaluation ______________________________________________________ 20
Integration of the Regulatory Scrutiny Board's recommendations _______________________ 22
Coherence of the IA with the proposal _____________________________________________ 24
4. Conclusions __________________________________________________________________ 26
References ____________________________________________________________________ 32
Annex – List of the 132 initial appraisals of Commission impact assessment reports published between July 2015 and December 2018 _____________________________________________ 34
Appraising the quality of the European Commission's impact assessments
V
Table of figures
Figure 1 – Average quality of the policy options in the appraised IAs published between July 2015 and December 2018 ______________________________________________________________ 10 Figure 2 – Assessment of the economic, social, and environmental impacts in the appraised IAs published by the Commission between July 2015 and December 2018 _____________________ 12 Figure 3 – Frequency of impacts in the appraised IAs published between July 2015 and December 2018 __________________________________________________________________________ 14 Figure 4 – Average quality of the subsidiarity and proportionality dimension of appraised IAs published between July 2015 and December 2018. _____________________________________ 16 Figure 5 – Integration of RSB comments following positive opinions and positive opinions with reservations in the appraised IAs published between July 2015 and December 2018 __________ 23 Figure 6 – Overall average quality of the appraised IAs published between January 2016 and December 2018 _________________________________________________________________ 26 Figure 7 – Average quality of the 10 essential dimensions of the appraised IAs published between July 2015 and December 2018. _____________________________________________________ 28
Appraising the quality of the European Commission's impact assessments
1
1. Introduction
The aim of this study is to assess the quality of the European Commission's Impact Assessments (IA)8 accompanying legislative proposals adopted between July 2015 and December 2018. It is based on the appraisal of these IAs by the Ex-ante Impact Assessment (IMPA) unit of the European Parliamentary Research Service (EPRS). IMPA initial appraisals analyse the methodological compliance of the IAs with the existing framework on Better Regulation, in order to support Parliament's scrutiny of the Commission's IA work.
The study draws on a review of 132 initial appraisals of Commission IA reports adopted during the above-mentioned period. In addition to pinpointing the strengths and weaknesses of the IAs, it seeks to identify patterns and trends regarding the various dimensions of an IA over time, so as to contribute to the ongoing wider discussion on better regulation in general and ex-ante IA in particular. In this context, the study takes into account the European Commission's 2018 stock-take of its better regulation agenda,9 and refers, wherever relevant, to the results of a previous review of 100 initial appraisals carried out by the IMPA unit between June 2012 and June 2015.10
The study is divided into three parts. The first chapter discusses relevant institutional developments in the field of ex-ante IA in the Commission and in the Parliament during the eighth legislative term (2014-2019), including the criteria and tools used to conduct Commission IAs, and their appraisal in the European Parliament. The second chapter explains the methodology of the study and presents the key observations and main findings of the quality check of the appraised IAs under this review. The final chapter provides conclusions, puts these findings into context, and suggests points for further discussion.
8 Impact assessment is understood here as 'ex-ante analyses of the likely or foreseeable effects of draft EU legislation
or policies proposed for adoption at European Union level', European Parliament Impact Assessment Handbook, annexed in W. Hiller, European Parliament work in the fields of Impact Assessment and European Added Value, Activity Report for 2017, EPRS, European Parliament, May 2018, p.49.
9 European Commission communication, Better regulation – taking stock and sustaining our commitment, COM(2019) 178 (hereafter communication on stocktaking); more detailed information on this stocktaking exercise can be found in the accompanying staff working document, SWD(2019) 156.
10 See J. Dunne and W. Hiller, European Parliament work in the fields of Impact Assessment and European Added Value. Activity Report for July 2014 - December 2015, Annex One, The first 100 initial appraisals of European Commission Impact Assessments by the European Parliament's Ex-Ante Impact Assessment Unit between June 2012 and June 2015, EPRS, European Parliament, April 2016.
EPRS | European Parliamentary Research Service
2
2. Impact assessment in the European Commission and its appraisal in the European Parliament a) Institutional developments and practical aspects of ex-ante impact assessment at the European Commission
From the very start of its mandate in 2014, the Juncker Commission made a strong commitment to better regulation principles, stressing the importance of evidence-based, open and transparent policymaking notably through better impact assessments of draft legislation, to 'deliver better rules for better results'. In May 2015, it published the better regulation agenda – an extensive package of new measures to improve and strengthen the way legislation is prepared, covering all stages of the policy-cycle, from planning, design and adoption of policy measures, to implementation (including enforcement), evaluation and revision.11 This agenda was complemented by the Better Regulation Guidelines and a detailed Toolbox (both revised in 2017)12 providing 'mandatory requirements and obligations for each step in the policy cycle.'13 These reference documents contain integrated and standardised guidance for Commission staff to implement the better regulation principles. As far as the specific approach to ex-ante IA is concerned, there is considerable continuity with the previous guidance on impact assessments;14 the basic premise remains that IA is a tool to support political decision-making, not a substitute for it.
The significant changes brought about by the better regulation agenda included enhanced stakeholder involvement at different stages in the policy cycle and better implementation of the 'evaluate first' principle. Furthermore, the Commission transformed its internal scrutiny body for impact assessments, the Impact Assessment Board, into the more independent Regulatory Scrutiny Board (RSB), extending its scrutiny to major evaluations. According to its rules, the RSB consists of a chair, three high-level Commission officials and three external experts all of whom work full-time for the board.15
In 2016, the Commission, the European Parliament and the Council concluded a new Interinstitutional Agreement on Better Law Making (IIA BLM),16 stressing inter alia the importance of ex-ante IA as a tool for better quality law making – alongside ex-post evaluation and stakeholder consultation. The three institutions also agreed on a common definition of the main purpose and content of an IA and consider IA-related work a shared effort. According to the IIA BLM:
impact assessments should cover the existence, scale and consequences of a problem and the question whether or not Union action is needed. They should map out alternative solutions and, where possible, potential short and long-term costs and benefits, assessing the economic, environmental and social impacts
11 European Commission communication, Better regulation for better results - An EU agenda, COM(2015)0215. 12 European Commission, staff working document, Better Regulation Guidelines, SWD(2017) 350 final. 13 Ibid p. 3. 14 Impact Assessment Guidelines, SEC(2009) 92, 15 January 2009. For a more detailed presentation of the new features
of the Better Regulation Guidelines with regard to ex-ante impact assessment please see, C. Collova, Ex-Ante impact assessment in the Commissions new Better Regulation Guidelines, EPRS, European Parliament, December 2015.
15 The European Parliament had asked in its resolution of 27 November 2014 on the revision of the Commission's impact assessment guidelines and the role of the SME test, P8_TA(2014)0069, that care be taken to ensure that the members of the Commission's Impact Assessment Board were no longer 'subject to political control' and that no proposal could be adopted without a positive opinion from this board on the accompanying IA.
16 2016 Interinstitutional Agreement on Better Law-Making, OJ L 123/1, 12.5.2016 (hereafter the IIA BLM). The 2016 agreement is the second of that kind. The first IIA BLM was concluded in 2003, when the three institutions agreed to improve the quality of EU legislation by means of 'more frequent use of impact assessment'.
Appraising the quality of the European Commission's impact assessments
3
in an integrated and balanced way and using both qualitative and quantitative analyses. The principles of subsidiarity and proportionality should be fully respected, as should fundamental rights.17
Furthermore, the IIA BLM specifies that the Commission 'will carry out impact assessments of its legislative and non-legislative proposals. The initiatives included in the Commission work programme or in the joint declaration will, as a general rule, be accompanied by an impact assessment'.18
At the start of the IA process, the Commission now normally prepares an inception impact assessment (a novelty introduced by the Better Regulation Guidelines). This is an 'enhanced' roadmap open to stakeholder feedback, providing a description of the problem to be tackled, objectives to be met, possible policy options, and a preliminary consideration of relevant impacts and the main features of the consultation strategy. The Commission may outsource the preparation of the ex-ante analysis, or part of it, to external experts. The final report, however, is an official Commission document, published with the legislative proposal to be referred to the Parliament and the Council.
An important aspect of the IA process is the quality control by the RSB. All IAs must be submitted to the board, which checks their quality before the legislative proposal can be submitted to the College of Commissioners for adoption. The RSB issues an opinion (positive, positive with reservations or negative) on the draft IA, including recommendations for improvement. Impact assessment reports require in principle a positive opinion from the board to move forward. If the report receives a negative opinion, it is sent back to the service responsible for revision and must be resubmitted to the board.
In 2018, the European Commission decided to assess the strengths and weaknesses of its better regulation activities with a view to making further improvements. The results of this review were reported in April 2019, in the communication on 'Better regulation – taking stock and sustaining our commitment' which was also informed by a literature review prepared by the Commission's Joint Research Centre.19 The communication draws overall positive conclusions, stating that 'There is a general recognition that progress has been achieved across several dimensions since 2015. There is an equally widespread demand for better regulation to continue as an integral part of the Commission's way of working, with a sustained commitment to achieving further improvements in the future'.20
b) Appraisal of the Commission's impact assessments in the European Parliament
After the conclusion of the first Interinstitutional Agreement on Better Law-Making in 2003, the Parliament had already started carrying out some ex-ante IA work, which was enhanced by the establishment of a dedicated impact assessment service within the Parliament's administration in 2012, (following the Parliament's own report on guaranteeing independent impact assessments, the Niebler Report).21 The 'Directorate for Impact Assessment and European Added Value' has been part of the Parliament's DG EPRS (European Parliamentary Research Service) since 2013. Within the
17 IIA BLM, point 12. 18 Ibid, point 13. 19 G. Listorti, E. Basyte Ferrari, S. Acs, G. Munda, E. Rosenbaum, P. Paruolo, P. Smits (2019). The debate on the EU Better
Regulation Agenda: a literature review, EUR 29691, Publications Office of the European Union, Luxembourg, 2019. 20 European Commission communication, Better regulation – taking stock and sustaining our commitment, COM(2019)
p. 6 (hereafter communication on stocktaking). Another external review of the EUs better regulation activities, conducted by the OECD, confirmed this positive assessment ranking the EU system among the very best when compared to other OECD countries. See 2018 OECD Regulatory Policy Outlook and OECD Report on Better Regulation Practices across the European Union, 2019
21 European Parliament resolution of 8 June 2011 on guaranteeing independent impact assessments (2010/2016(INI)), P7 TA(2011)0259.
EPRS | European Parliamentary Research Service
4
EPRS, the IMPA Unit provides parliamentary committees with support in the area of IA, helping to strengthen the Parliament's role as an effective co-legislator and its capacity for scrutiny of the executive.
At the beginning of the parliamentary legislative process, IMPA routinely assesses the quality of the Commission's IAs accompanying legislative proposals, in light of the criteria set out both in the Commission's own Better Regulation Guidelines and in relevant Parliament resolutions,22 as well as in its own guidelines for IA work – the Impact Assessment Handbook23 (which is based on the Better Regulation Guidelines).
In the light of the Parliament's commitment in the 2016 IIA BLM to 'take full account of the Commission's impact assessments' when considering the latter's legislative proposals,24 IMPA's initial appraisals of the Commission's IAs seek to support the informed and effective consideration of legislative proposals at committee stage by drawing the committees' attention to the content and the methodological strengths and weaknesses of the IAs. Their findings can lead committees to invite the Commission to explain any potential weaknesses or unclear elements of an IA or to request additional or complementary IA work.25
In particular, IMPA appraisals routinely check the main dimensions that any IA is expected to cover in a clear and transparent manner according to the Better Regulation Guidelines, in order to help inform policymaking. Firstly, this includes the seven key questions every IA is required to address:
What is the problem and why is it a problem? Why should the EU act? What should be achieved? What are the various options to achieve the objectives? What are the relevant impacts of the different policy options and who will be affected?26 How do the options compare? How would actual impacts be monitored and evaluated?
Secondly, in addition to the aspects above, IMPA initial appraisals address three additional dimensions important to Parliament's information and scrutiny role. These concern the consistency between the IA and the legislative proposal, i.e. whether a justification or explanation is provided in case of discrepancy between the conclusions of the IA and the provisions of the final legislative proposal. While the Commission is free to decide the content of the final proposal, in line with the fact that the IA is not a substitute for political decisions, by assessing the coherence between the IA and the legislative proposal the initial appraisal can evaluate whether and how the evidence gathered in the IA fed into the proposed legislation. In addition, the appraisals also check, to the
22 The European Parliament has adopted several resolutions in recent years asking for particular attention to be paid to
impact assessment. Most recently, Parliament adopted a resolution on 30 May 2018 on the interpretation and implementation of the Interinstitutional Agreement on Better Law-Making (2016/2018(INI)), reiterating the importance of impact assessment as a tool for better law-making. More specifically, Parliament requested that particular attention be paid to the potential impacts on SMEs, civil society, trade unions and others who do not have the advantage of easy access to the institutions, to pay equal attention to the evaluation of social, health, and environmental impacts, as well as to impacts on the fundamental rights of citizens and on equality between women and men.
23 Conference of Committee Chairs, Impact Assessment Handbook, September 2017, pp. 9-11, annexed to W. Hiller, European Parliament work in the fields of impact assessment and European added value, activity report for 2018, available at http://europarl.europa.eu/EPRS/EPRS_631.723_IA-EAV-Activity_Report_2018.pdf.
24 IIA BLM, point 14. 25 Additional on-request services that can be provided by IMPA include detailed appraisals of Commission IAs,
complementary or substitute impact assessments or impact assessments of Parliament's substantial amendments. 26 According to the Better Regulation Guidelines, the assessment of economic, social and environmental impacts is
mandatory for all IAs. Other relevant impacts depend on the initiative and the policy area, and include for instance impacts on health, gender, administrative burden, national or EU budgets, as well as territorial impacts or impacts on SMEs or on third countries. Better Regulation Guidelines, 2017, p. 25.
Appraising the quality of the European Commission's impact assessments
5
extent possible, whether and how the recommendations expressed in the RSB opinions were addressed in the final IA. In fact, the RSB's quality check concerns the draft IA, not the final version published with the proposal. Draft IAs are not made public and are not available to Parliament, the RSB opinions provide therefore the sole source to deduce, at least in part, the evolution of an IA, and to understand, for example, which provisions of the Better Regulation Guidelines were not adequately implemented despite the RSB's recommendations. Finally, initial appraisals assess the quality and transparency of the data and research undertaken in the IA, as well as the quality of the stakeholder consultations and their use in the analysis, in light of the better regulation criteria.
IMPA initial appraisals are therefore structured around these 10 main dimensions, which also form the core of the present review.
EPRS | European Parliamentary Research Service
6
3. Main observations and trends
Methodology
The analysis in this study draws on the sample of 132 IMPA initial appraisals of Commission IAs published between July 2015 and December 2018.27 The 19 additional IAs accompanying the proposals for the EU's multiannual financial framework (MFF) 2021-2027, which were published by the Commission during the period under review, were also appraised by IMPA. However, as they differed both formally and substantially from the requirements of the Better Regulation Guidelines, they could not be compared to the standard IAs,28 and are therefore not included in the main analysis, in the interests of the coherence of this study.
Table 1 shows the number of IAs appraised by IMPA between July 2015 and December 2018, excluding the MFF IAs. When reading the graphs in this study, account should be taken of the fact that owing to the end of the legislature, the number of IAs appraised by IMPA in the last semester of 2018 was significantly lower than during the previous periods given the low number of corresponding proposals presented by the Commission.
Table 1 – Number of IAs appraised included in this review, July 2015 - December 2018
2015 2016 2016 2017 2017 2018 2018 Total
July-Dec Jan-June July-Dec Jan-June July-Dec Jan-June July-Dec
10 20 22 24 17 37 2 132
Source: IMPA, EPRS.
The present review is based on a scoring exercise of the 10 main dimensions of an IA that are appraised in any initial appraisal, rating each of these dimensions separately (on a range from 1 to 5, corresponding to the perceived quality of 'very poor, poor, satisfactory, good, very good'). Whereas an IA's overall score might be 'satisfactory', that does not rule out the fact that some individual aspects of the IA might be of poorer or higher quality. An IA can in fact be strong in one area and weak in another, as also observed by the RSB.29 The aim of this review is precisely to pinpoint strengths and weaknesses of the IAs and to identify evolutions and trends regarding those 10 elements over time, making notably reference, wherever relevant, to the findings of a previous review conducted in relation to the first 100 initial appraisals carried out by IMPA between June 2012 and June 2015.30
The systematic review of each of the 10 dimensions is presented here in the order of their appearance in IMPA initial appraisals. The review concludes with an evaluation of the overall average quality of the 132 IAs appraised by the IMPA unit. While the overall score is of some importance, the specific structural observations regarding the different parts of an IA may be more helpful to assess quality with a view to future improvement.
27 The timeframe of this study follows the period after the previous review (June 2012-June 2015) up to the end of the
core legislative work of the Commission. 28 See for an analysis of the MFF impact assessments, W. Hiller, European Parliament work in the fields of Impact
Assessment and European Added Value. Activity Report for 2018, EPRS, European Parliament, March 2019, p. 22. 29 See the Regulatory Scrutiny Board Annual Report 2018, p.9. 30 See J. Dunne and W. Hiller, European Parliament work in the fields of Impact Assessment and European Added Value.
Activity Report for July 2014 - December 2015, Annex One. The first 100 initial appraisals of European Commission Impact Assessments by the European Parliament's Ex-Ante Impact Assessment Unit between June 2012 and June 2015, EPRS, European Parliament, April 2016.
Appraising the quality of the European Commission's impact assessments
7
Problem definition
According to the Commission's Better Regulation Guidelines, an IA should start by 'verifying the existence of a problem' in order to give policymakers the information needed to decide whether a policy response may actually be necessary.31 The problem description in the IA report should be 'clear and specific' and include an explanation of the scale, causes and consequences of the problem and identify who is affected.
The evidence-based and coherent definition of the problem to tackle is of particular importance for the quality of any IA, as it prepares the ground and reference framework for the whole analysis. As the RSB indicated in its annual report for 201632 'when the problem is poorly defined, it is hard to define appropriate corresponding objectives and options'. An IA should also assess the likelihood that a problem would persist in the absence of EU policy intervention and develop a baseline scenario, i.e. the current situation, taking into account existing EU and national policies and reflecting possible technological or societal developments. The IA should use the baseline scenario as a benchmark for the comparison of alternative policy options presented in the IA to address the problem, usually leading to a 'preferred option'.
In the sample of 132 IAs under this review, the baseline scenario was found to have been adequately defined in 84 % of cases, which seems to provide a good basis for a pertinent IA. This number appears to be confirmed by the fact that the problem definition was considered of 'satisfactory' quality and above for 85 % of the IAs under this review.33 This appears to be a significant improvement as compared to the years 2012-2015 where the problem definition was found to be entirely 'satisfactory' in only 40 % of the IAs appraised by IMPA during that period'.34 At the same time, however, for 15 % of the IAs the quality of the problem definition was found to be poor, providing either limited evidence or explanations of the nature and scale of the problem, or occasionally, simply defining the problem as the absence of legislation itself.
As had already been observed by IMPA,35 the Commission has made clear overall efforts in recent years to improve the quality of the problem definition in IAs. This dimension is the one that scored on average the best among the 10 dimensions covered under the present analysis, ranking well within the satisfactory range throughout the period under review. In particular, considerable efforts have been made to make a better link between problems and their drivers and to the corresponding objectives and policy options. Considering the essential nature of this first major analytical part for the rest of an IA, the consolidation of the quality of this dimension over recent years is indeed welcome.
Policy objectives
According to the Better Regulation Guidelines, 'the objectives of policy actions should be clearly identified, including the level of policy ambition.'36 Objectives link the analysis of the problem (and
31 Better Regulation Guidelines, 2017, p.18 32 European Commission, Regulatory Scrutiny Board Annual report 2016, p.13. 33 More specifically, the problem definition section was appraised as 'satisfactory' for 33 %, 'good' for 41 %, and 'very
good' for 11 % of the total number of IAs. For a positive example of problem definition in an IA, see S. Vettorazzi, Mutual recognition of goods lawfully marketed in another Member State, Initial appraisal of European Commission Impact Assessment, EPRS, European Parliament, March 2018.
34 J. Dunne and W. Hiller, 2016, op.cit p. 38. 35 See W. Hiller, European Parliament work in the fields of Impact Assessment and European Added Value, Activity Report
for 2017, EPRS, European Parliament, May 2018, p. 20 and W. Hiller, European Parliament work in the fields of Impact Assessment and European Added Value. Activity Report for 2018, EPRS, European Parliament, March 2019, p. 19.
36 Better Regulation Guidelines, 2017, p.21.
EPRS | European Parliamentary Research Service
8
its drivers) with the options for the policy response. Based on the problem definition, an IA should clearly define distinct general and specific objectives. The general objectives are based on the objectives of the EU treaties, whereas the specific objectives identify the goals of the policy initiative more specifically, often relating to several proposed measures to address a problem.
Operational objectives should be developed in relation to the preferred policy option of an IA, thus after the analysis of potential impacts (and not with the general and specific objectives). According to the guidelines (Tool 16 of the Toolbox), they should define clear deliverables for individual policy actions. Specific and especially operational objectives are relevant not only for the ex-ante IA of an initiative, but also in view of future effective monitoring and evaluation: the initiative and its outcome will be measured in relation to them, to evaluate its effectiveness. Consequently, clear objectives and deliverables of the preferred option should be specified in an IA, by pertinent indicators, to allow for effective monitoring and ex-post evaluation. The Better Regulation Guidelines also stipulate that objectives should be S.M.A.R.T. (i.e. specific, measurable, achievable, relevant and time-bound).
Based on the sample of IAs reviewed in this study, the overall average quality of the policy objectives dimension ranked just above the satisfactory threshold, with the analysis showing a rather mixed picture, as many appraisals indicated the need for more consistency and precision of operational objectives. While nearly all of the 132 IAs appraised included general and specific objectives that were clearly linked to the problem(s) to be addressed, about one third of the IAs reviewed (29 %) were found to be lacking operational objectives. In the cases where operational objectives were formulated, the distinction between the latter and the specific objectives was not always clear, the operational objectives regularly being formulated too broadly and not always in terms of concrete deliverables, often repeating the specific objectives. In addition, they were not always option-specific as required by the guidelines. Generally speaking, 70 % of the IAs were found to comply only partially with the guidelines' S.M.A.R.T. criteria. In many appraisals, it was the time-bound aspect and the specificity of the objectives, in particular the operational ones that were deemed most problematic.37
It should be noted, that the requirements of the guidelines and the Toolbox are rather demanding regarding this part of an IA, which, in turn, illustrates its importance for the quality of the ensuing proposed legislation and its monitoring and evaluation. In its recent stocktaking exercise the Commission did not focus on this aspect specifically. It is reasonable however to expect that it will devote greater attention to the need for consistency and precision in the definition of the objectives in the future – given the direct link between the quality of the objectives of an initiative and the effectiveness of its later evaluation –, as it has announced that it intends to pay 'special attention to the inclusion of monitoring and reporting provisions in its future proposals'38 so as to improve the quality of ex-post evaluation.
Policy options
The Commission's Better Regulation Guidelines require that 'due consideration' be given to the widest range of alternative and realistic policy options, so as to achieve the objectives of a policy
37 These observations are in line with the observations made by IMPA in the annual activity reports concerning the
European Parliament's work in the fields of impact assessment and European added value for the years 2016, 2017 and 2018. See W. Hiller, Activity Report for 2016, EPRS, European Parliament, March 2017, p. 17; W. Hiller, Activity report for 2017, EPRS, European Parliament, p. 20; and W. Hiller, Activity report for 2018, EPRS, European Parliament, March 2019, p. 20.
38 Communication on stocktaking, 2019, p. 10.
Appraising the quality of the European Commission's impact assessments
9
initiative. The presentation of the options is indeed 'one of the key functions of an impact assessment process'.39 In the Commission's view, the definition of the options tends to be the part of an IA most frequently criticised, as poor definition of the options can 'significantly undermine the credibility of the whole exercise and its usefulness for political decision-making'.40
The guidelines also stress that 'the IA report will need to show that there is a clear logic between the problems, objectives and policy interventions under consideration' and they explicitly advise avoiding presenting options only to highlight the benefits of the preferred option ('straw man' options).41 They require an IA to present at least two credible alternatives to the baseline, thinking 'outside the box', and to justify why certain options were discarded and not retained for further analysis of their impacts. Lastly, according to the guidelines, the IA report should give a sufficiently detailed description of the options retained for assessment and a particularly strong justification if only one option is retained in addition to the baseline, a situation regarded as exceptional.
More than two thirds of the IAs (79 %) examined during the period under review were indeed found to present at least three alternative policy options in addition to the baseline (with non-regulatory alternatives being increasingly considered in line with the recommendations of the guidelines), while 17 % of all cases included the minimum of two alternatives. There were only five cases (compared to three in the 2015 review) (4 % of the total) where only one alternative was presented, confirming the exceptional nature of such a situation.42 The range of options in the Commission's IAs appears to have slightly improved in comparison to the 2015 review where it was found that approximately 70 % of the cases presented at least three alternatives to the baseline, demonstrating that the Commission has kept up its efforts to comply at least formally with the requirements of the guidelines in this respect.
Not that much improvement can be reported in terms of presentation and description of the options however. This was considered entirely clear and balanced in only 37 % of the cases under the current review, and insufficiently clear and balanced in 47 %. In particular, alternatives to the preferred option were found to have been considered insufficiently in a number of cases under this review.43 The 2015 review found similar results, with the options considered to be presented in a 'less than balanced way' in around 40 % of the cases examined.44 In other instances, it was the presentation of the options that was considered not detailed or clear enough to allow an understanding of how the options would help address the problems identified and meet the policy objectives.
These observations seem to indicate that the way options are presented and described in the IA reports would benefit from further improvements. This is particularly important because an imbalanced presentation of the options, with the preferred option receiving far more attention than the other alternatives, may point to a predetermined choice of action, potentially undermining the
39 Better Regulation Guidelines, 2017, pp. 21-22. 40 Ibid, p. 21. 41 Ibid, p. 21. 42 For example, see M. Tuominen, Multilateral court for the settlement of investment disputes, Initial Appraisal of the
European Commission Impact assessment, EPRS, European Parliament, November 2017. 43 For example, see E. Kramer, Pan-European Personal Pension Product, Initial Appraisal of the European Commission
Impact assessment, EPRS, European Parliament, October 2017. See also J. Marcs, C. Midoes and A. Schout, 'To the Commissioner responsible for better regulation', in M. Demertzi and G. Wolff (eds.), Braver, greener, fairer: Memos to the incoming EU leadership 2019-2024, p. 72.
44 J. Dunne and W. Hiller W., 2016, p.39.
EPRS | European Parliamentary Research Service
10
objectivity and credibility of the analysis45 as the Commission itself recognises in the guidelines. The same holds true when the alternatives presented are of such a nature that only the preferred option appears like a viable course of action. In this regard, an encouraging result is that only about 8 % of the IAs (10 IAs) appraised during the period under review were considered not to present realistic alternatives to the preferred option. This seems to be a positive development compared with the 2015 review where 'in many instances it was felt that the options presented were closer to variants of the preferred option than genuine, viable alternatives'.46 Meanwhile, entirely realistic alternatives were deemed to have been presented in only about half of the cases examined in the present study (49 %) and partially realistic alternatives in 43 %, showing that further efforts are still warranted to ensure the necessary transparency and objectivity in the presentation and selection of the preferred option(s).47
Figure 1 below shows the average quality scores of the options dimension in the IAs appraised during the period under review; the quality level was found to be below 'satisfactory' for several time periods. Overall, the quality of this dimension of the IAs ranked on average among the weakest of the 10 dimensions appraised, falling just below the 'satisfactory' threshold. This finding appears to concur with the information provided in successive RSB annual reports, which show that the design of options remains one of the weakest areas in impact assessments48 in comparison with other components.
Figure 1 – Average quality of the policy options in the appraised IAs published between July 2015 and December 2018
Source: IMPA, EPRS.
45 For example, see V. Kononenko, Adapting the road haulage market to developments in the sector: road transport
operators and access to the market, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, October 2017. The appraisal argues that 'not all the individual options would have been able to address all the specific problems identified by the IA'. Another initial appraisal of an impact assessment regarding the 'Action plan on building a capital markets union, points out an imbalanced presentation of the options, see C. Collova, Action Plan on Building a Capital Markets Union: EU securitisation framework, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, September, 2015.
46 See J. Dunne and W. Hiller, 2016, op.cit. p.39. 47 See W. Hiller, March 2017, p. 17; W. Hiller, May 2018, op. cit., p. 19 and W. Hiller, March 2019, op. cit., p. 20. 48 See RSB Annual Report 2016, p 13; RSB Annual Report 2017, p. 13; and RSB Annual Report 2018, p. 12, op. cit.
Appraising the quality of the European Commission's impact assessments
11
Assessment of impacts
According to the Better Regulation Guidelines, an IA must assess 'all the relevant advantages and disadvantages of the retained policy alternatives or options against the reference of the baseline.'49 Identifying which impacts are relevant, selecting the significant impacts, and finally assessing the most significant impacts is therefore a very important step in any IA. In the framework of the EU's integrated approach to IA, the three main compulsory impacts to be assessed in any IA are the economic, social and environmental impacts.50 Other relevant direct and indirect impacts, depending on the policy area, must also be assessed. These can be, for instance, impacts on fundamental rights, on gender equality, territorial impacts or other impacts that the Commission deems relevant. Potential impacts on SMEs and on competitiveness should be considered and reported systematically in all impact assessment reports.51 At the same time, the appropriate level of analysis should also be taken into account, with a view to keeping the IA proportionate.52
Difficulties in selecting and assessing the most significant impacts in a transparent way were reflected in the overall average quality of this aspect, which showed the weakest scores of the 10 dimensions covered by the present analysis, falling slightly below the satisfactory range. This dimension of IAs was also found to be on average the weakest in the last RSB annual report, being below acceptable level at the final interservice consultation stage.53
The analysis of the IAs appraised under this review shows that the assessment of economic impacts still tends to prevail over social and environmental impacts and to be carried out to a higher standard. Economic impacts were assessed in 98 % of relevant cases, with the analysis being found insufficient in 21 % of those (compared to 14 % in the 2015 review). Social impacts for their part were assessed in 91 % of relevant cases, but the analysis was considered insufficient in 30 % of those (compared to 25 % in the 2015 review). Environmental impacts were assessed in 84 % of relevant cases with further assessment deemed necessary in 31 % of those (as in the 2015 review). This picture appears to have remained relatively stable since 2012, as the 2015 review showed similar overall numbers,54 although the number of cases where the economic and social impacts were considered not to have been fully assessed appears to have increased slightly during the period covered by this review. A number of IMPA appraisals also raised the point that the IA should have given consideration to social and environmental impacts, where this was not the case. This was found to be so for 9 % of IAs with regard to social impacts, and for 16 % of IAs with regard to environmental impacts. In view of the cross-cutting new European Green Deal and stronger focus on environmental and social issues announced in the political guidelines of Commission President Ursula von der Leyen,55 the importance of assessing these impacts at the ex-ante stage can be expected to be even higher in the near future.
49 Better Regulation Guidelines, 2017, p. 23. 50 See also, IIA BLM, point 12. 51 Better Regulation Guidelines, op.cit, p. 30 and IIA BLM, point 12. 52 Better Regulation Guidelines, op.cit, p. 10 and IIA BLM, point 12. 53 RSB, Annual Report 2018, p. 12. 54 Economic impacts were assessed in 95 % of relevant cases; social impacts in 88 % of relevant cases and environmental
impacts in 79 % of relevant cases. 55 European Commission, A Union that strives for more. Political guidelines for the next European Commission 2019-
2024.
EPRS | European Parliamentary Research Service
12
The purple, orange, and blue tabs in Figure 2 below show the breakdown of relevant cases where the three fundamental impacts were addressed ('yes'), were addressed 'partially,' or should have been addressed but were not ('no').56
Figure 2 – Assessment of the economic, social, and environmental impacts in the appraised IAs published by the Commission between July 2015 and December 2018
Source: IMPA, EPRS.
The assessment of other relevant impacts, for instance on SMEs, which is called for in several parliamentary resolutions57 and in the 2016 IIA BLM, and is among the mandatory impacts to be covered according to the guidelines, shows a mixed trend. The impacts on SMEs were found to be addressed (albeit to varying degree) in the vast majority of the IAs (89 % of relevant cases) appraised. This suggests continuity of a positive trend noted in the 2015 review, where a majority of the IAs were found to have taken account of the possible impacts on SMEs. The decreasing number of IAs that do not address SMEs (from 30 % in 2015 to 11 % in the current review) confirms this positive trend. The more stringent requirements of the Better Regulation Guidelines to analyse the effects on SMEs systematically, in particular Tool 19 of the Toolbox, and the instructions on how to conduct an 'SME test' (Tool 22) – also called for repeatedly by Parliament58 – may have contributed to this positive trend.
Notwithstanding these positive observations, there is still some room for improvement in terms of the depth and thoroughness of the SME-related assessment as noted by IMPA in recent years.59 In 41 % of the relevant cases under this review, the potential impacts of the initiative on SMEs were deemed to have only been partially assessed (versus 48 % of relevant cases where they were fully assessed), the analysis often limiting itself to brief and general statements. Furthermore, a proper SME test was carried out only in 28 % of those IAs that had assessed impacts on SMEs. Parliament had already also noted in its 2018 resolution on the interpretation and implementation of the IIA
56 For example, an IA on the protection of workers from exposure to carcinogens and mutagens did not appear to assess
environmental impacts for certain substances (arsenic acid, formaldehyde, and MOCA). According to the IMPA appraisal, this would have merited further justification, given that the proposal deals with carcinogenic chemicals. For details, see S. Vettorazzi, Protection of workers from exposure to carcinogens or mutagens: third proposal, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, June 2018.
57 Resolution on the revision of the Commission's impact assessment guidelines and the role of the SME test (2014/2967(RSP)): Resolution on the interpretation and implementation of the Interinstitutional Agreement on Better Law-Making (2016/2018(INI)) P8_TA(2018)0225, point 16.
58 European Parliament resolution of 27 November 2014 on the revision of the Commission's impact assessment guidelines and the role of the SME test, P8_TA(2014)0069.
59 See W. Hiller, March 2017, op.cit., p.18; W. Hiller, May 2018, p. 19 and W. Hiller, March 2019, op. cit., p. 20.
Appraising the quality of the European Commission's impact assessments
13
BLM that 'SME tests often lack quality and coherent implementation'60 and had called on the Commission 'to consider how the impact on SMEs can be better taken into account'. This call appears therefore to remain valid. In its latest annual report, the RSB indicates that the SME test is a good example of a case where the assessment needs to be relevant and proportionate and take into account various limiting factors such as the availability of enough relevant data and other conditions, such as the diverging definition of 'SMEs' across the EU.61 While the wide range and diversity of measures and policies covered by Commission IAs can indeed justify, on a case by case basis, some flexibility in the depth of the assessment, this should in any event be properly explained in the IA.
As for the impacts on competitiveness, which should be covered in any IA report according to the guidelines and the 2016 IIA BLM,62 as indicated earlier, the present review found that these had been considered in 67 % of the IAs, but assessed insufficiently in 32 % of those, with rather laconic statements on the potential impacts rather than a proper assessment, similarly to what is observed for SMEs.
Several other potentially relevant impacts were found to be covered in only a limited number of Commission IAs. One such example is territorial impacts,63 the limited consideration of which has been recognised in the general literature,64 and appears confirmed by this review, despite the IIA BLM requirement that territorial impacts be taken into account 'whenever possible.' Territorial impacts were found to have been addressed in only 15 of the total of 132 IAs appraised. In response to the recommendations of the Task Force on Subsidiarity,65 the Commission has committed to 'amend its better regulation guidance to highlight the importance of screening and assessing territorial impacts'.66 A more thorough analysis of these impacts could therefore be expected in the future.
Another example of impacts that feature rarely in IAs are gender equality related impacts, the assessment of which has been called for regularly by Parliament67 and which were addressed in only one IA appraised under this review. Given the announcement of a new European gender strategy in the von der Leyen Commission's political guidelines, the gender-related impacts of legislation should receive more systematic attention in the future.68 The same is true for the assessment of impacts on sustainability and innovation, as the recent communication on the European Green Deal
60 European Parliament resolution on the interpretation and implementation of the Interinstitutional Agreement on
Better Law-Making, op. cit, point 16. 61 RSB Annual Report, 2018, p. 34. 62 Better Regulation Guidelines, 2017, p. 30, See also European Parliament, Report on the interpretation and
implementation of the Interinstitutional Agreement on Better Law-Making (2016/2018(INI). 63 Specific guidance has been developed by the Commission to assess territorial impacts. See Better Regulation Toolbox,
Tool 33. 64 See for example B. Taulègne, 'Territorial impact assessment: A key practical experience of territorial scrutiny', in
A. De Feo and B. Laffan (eds.), Scrutiny of EU policies, European University Institute 2017, pp. 88-94. 65 ‘Task force recommendation 5: The Commission should ensure that its impact assessments and evaluations
systematically consider territorial impacts and assess them where they are significant for local and regional authorities', Active Subsidiarity. A new way of working, Report of the Task Force on Subsidiarity, Proportionality and 'Doing less more efficiently', European Commission, 2018, p.16.
66 Commission communication, The principles of subsidiarity and proportionality: Strengthening their role in the EU's policymaking, COM (2018) 703 final, p. 10.
67 European Parliament resolution on the interpretation and implementation of the Interinstitutional Agreement on Better Law-Making, op.cit, paragraph 15.
68 European Commission, A Union that strives for more. Political guidelines for the next European Commission 2019-2024, p. 11.
EPRS | European Parliamentary Research Service
14
states that the Better Regulation Guidelines and supporting tools are to be improved so as to address these impacts more effectively, in line with the 'do no harm' principle.69
Figure 3 below illustrates the frequency with which various impacts feature in the IAs under this review.
Figure 3 – Frequency of impacts in the appraised IAs published between July 2015 and December 2018
Source: IMPA, EPRS.
This review agrees with the need, identified by the Commission in its stocktaking exercise, to strike the right balance between the depth of the analysis of a varying set of impacts on the one hand, and the readability and transparency of the IA on the other.70 It also concurs with the RSB that a 'mandatory assessment of every category of impact would also be unrealistic'71and not proportionate. Nevertheless, the above findings point to the need to improve the screening and identification of potentially relevant impacts. In any event, to ensure a thorough and accountable IA, while keeping in mind the proportionality of the assessment,72 the Commission should duly justify which impacts it analyses and explicitly indicate whether one of the aforementioned compulsory impacts is not relevant or why other potentially relevant impacts are not assessed.73
69 Commission communication, The European Green Deal, COM (2019) 640 final, p. 19 70 In order to properly assess the appropriate level of analysis in relation to the magnitude of the expected impacts, the
communication on stocktaking also highlights the relevance of the quality and availability of data, timing and resources to conduct balanced and comprehensive IAs.
71 RSB Annual Report 2018, p. 34. 72 Ibid, pp. 31-34. 73 For example, the IA regarding the road haulage market, where environmental impacts were not assessed and their
absence not explained in the IA, contrary to the better regulation requirements. For details, see V. Kononenko, Adapting the road haulage market to developments in the sector: road transport operators and access to the market,
Appraising the quality of the European Commission's impact assessments
15
Subsidiarity and proportionality
Subsidiarity and proportionality are key issues that the Commission is required to address in its IAs in a dedicated section.74 Both derive directly from the problem definition and relate to the question of whether the EU should act to address it, and how. According to the Better Regulation Guidelines, if EU action is proposed in areas where the EU does not have exclusive competence, the IA should verify whether this action is compatible with the principle of subsidiarity. In this context, an IA should also consider 'the actual added value from EU action' compared to what could be achieved by Member States.75 This analysis should be complemented, whenever possible, by a consideration of the costs of not acting at Union level (the 'cost of non-Europe') – an element agreed in the 2016 IIA BLM, following Parliament's request. An IA is also expected to address the question of proportionality of the proposed EU action, i.e. whether the intensity and nature of this action does not go beyond what is required to address the problem.
Over the period under review, the average overall quality of this dimension just reached the satisfactory level. While the vast majority of IAs appraised were found to have addressed the necessity of EU action (98 % of all cases) and the expected EU added value of the initiative (64 % of all cases), in many instances the analysis was felt to be limited to general formal statements and rather circular reasoning. This confirms the finding of the communication on stocktaking according to which 'the assessments of subsidiarity presented in impact assessments are frequently rather general, overly legalistic and formalistic'.76 It is notable, that the 'cost of non-Europe', provided for in the IIA BLM and called for by Parliament,77 was generally not analysed in these IAs.78
The situation is similar for the proportionality analysis in the IAs under this review. While the principle appears to be addressed in the absolute majority of cases (86 %), the depth of the analysis seems somewhat limited. In fact, both the proportionality and subsidiarity analysis of an IA should consist not only of stating that both principles were respected, but also of explaining clearly how. This appears even more important as both aspects involve a certain degree of political judgment.79 Although proportionality is listed in the Better Regulation Guidelines as a fourth criterion against which options should be compared alongside efficiency, effectiveness and coherence, options were not compared against this criterion in 32 % of cases.
The Commission has recognised that there is room for improvement in this regard and committed recently to take on board the recommendations of the Task Force on Subsidiarity and Proportionality.80 These include the systematic use of a model grid to assess subsidiarity and proportionality, throughout the whole policy cycle and by all EU institutions. This grid will be included in future IAs and should also be used as part of the explanatory memoranda of legislative
initial appraisal of a Commission impact assessment, EPRS, European Parliament, October, 2017. See also Better Regulation Toolbox, 2017, p. 126.
74 Better Regulation Guidelines, 2017, p. 17. 75 Better Regulation Guidelines, 2017, p.64. 76 Communication on stocktaking, p.10. 77 Interinstitutional Agreement on Better Law-Making, paragraph 12 and European Parliament resolution on the
interpretation and implementation of the Interinstitutional Agreement on Better Law-Making, op.cit, paragraph 24. 78 W. Hiller, 2019, p. 19. 79 In its resolution on the interpretation and implementation of the Interinstitutional Agreement on Better Law-Making,
the Parliament underlined 'the importance of a strengthened and comprehensive assessment and justification regarding compliance with these principles as well as the European added value of the measure proposed', paragraph 41.
80 Communication on stocktaking, 2019 p.10.
EPRS | European Parliamentary Research Service
16
proposals and evaluations.81 A structured checklist of this kind can be expected to facilitate, streamline and improve this part of the Commission's IAs, as well as Parliament's scrutiny of them.
Figure 4 shows the overall trend in the average quality of the subsidiarity and proportionality dimension of IAs throughout the period under review. Although a notable improvement in quality is seen in the July to December 2018 period, consideration should be given to the low number of IAs appraised during that period.
Figure 4 – Average quality of the subsidiarity and proportionality dimension of appraised IAs published between July 2015 and December 2018.
Source: IMPA, EPRS.
Quality of data, research, and analysis
The quality, availability and transparency of data underlying the analysis of the IA are key for the evidence-based analysis of potential impacts. All sources should be robust, credible and reliable, and their origin and accessibility should be clearly indicated to make the analysis accountable.82 The Better Regulations Guidelines state that 'all external material used (studies, reports, scientific findings etc.) should also be systematically referenced'.83 Assumptions underlying the analysis, uncertainties and limitations should be transparently explained,84 as well as methods and, if appropriate, models used. Moreover, the Better Regulation Guidelines stipulate that an IA should be substantiated by 'qualitative and where possible quantitative' evidence and that 'all relevant impacts should be assessed qualitatively and quantitatively whenever possible'.85 The required quantification of the expected costs and benefits makes the expected impacts of an initiative more
81 Commission communication, The principles of subsidiarity and proportionality: Strengthening their role in the EU's
policymaking, COM (2018) 703 final, pp. 6-8 and Annex II to the communication. 82 See Better Regulation Toolbox, 'Tool 4: Evidence-based Better Regulation', p. 17 and p. 19. 83 Ibid, p. 19. 84 Ibid pp. 19 and 20. 85 Better Regulation Guidelines, 2017, pp. 19 and p. 26.
Appraising the quality of the European Commission's impact assessments
17
tangible and comparable according to the RSB.86 Quantification, in turn, depends on the availability of pertinent data, which is why the Commission has recently highlighted that a 'strategic and systematic approach to the collection and management of data' should be part of any IA, with a view to generating these data sources.87
An absolute majority of the appraised IAs (83 %) were rated as 'satisfactory' or above for the dimension related to the quality of data, research, and analysis, and the overall average quality score for this dimension during the period under review fell well within the satisfactory range. Overall, this review found considerable variation of data used and presentation between individual IA reports. In several instances, issues were identified in initial appraisals in relation to the lack of transparency of information, including the unavailability of supporting studies, the lack of coherence and/or transparency regarding the quality of data used, and also regarding the assumptions underlying the analysis or the use of models and methodologies.88 For instance, while in 72 % of all relevant cases, the supporting studies commissioned were publicly available at the time of the publication of the IA, this was not the case in 20 % of relevant cases and only partly so in another 8 %.89 However, without access to the specific research supporting an IA, it is not possible for the co-legislators – or other stakeholders – to check the statements made and conclusions drawn in the IA. In the vast majority of IAs reviewed, data sources were found to be relevant, reliable and publicly accessible, but only 56 % of the IAs were considered to provide an entirely clear explanation of the assumptions and limitations of the analysis, whereas this was found partially clear in 40 % of the cases.
When it comes to quantification of impacts, an aspect traditionally found to be weak in IAs, IMPA has already noted an overall improvement in recent years.90 Quantification was included in 90 % of the IAs under this review. This is a direct consequence of the requirements of the guidelines.91 The introduction in November 2017 of a mandatory quantification template92 featuring the quantified costs and benefits, which is now also annexed to the opinions of the RSB, probably helped to increase awareness and transparency of the quantified effects of an initiative. At the same time, the present review found quantification to be conducted mostly 'partially', in general more often in relation to costs than to benefits93, mainly in the form of a cost-effectiveness analysis. While the OECD has highlighted the relevance of clearly identified (quantified) benefits for facilitating political decision-making,94 the methodological challenges and resource requirements of a proper cost-benefit analysis, including a monetisation of all direct and indirect costs and benefits and a discounting exercise, also need to be taken into account. There are notably considerable differences in data availability, depending on the policy area and on the Member States where data is collected.95
86 RSB Annual Report 2018, p. 35. 87 European Commission, Quantification in Commission Impact Assessments and evaluations, Report of the RSB/SG/
JRC, Working Group, 20 March 2018, p. 4. 88 W. Hiller, 2019, p. 18. 89 In 18 % of the cases reviewed (24 IAs), the IAs appeared to be supported by in-house analysis. 90 W. Hiller, 2019, p. 20. See also European Commission, Quantification in Commission Impact Assessments and
evaluations, Report of the RSB/SG/ JRC Working Group, 20 March 2018, p. 2. The Commission saw significant improvements in particular between 2016 and 2017, including for the assessments of benefits, see RSB Annual Report 2018, pp. 35-36.
91 Better Regulation Guidelines, 2017, pp. 19 and 26. 92 RSB Annual Report 2018, p. 35. 93 For similar observations, see RSB Annual Report, 2018, p. 35, and RSB Annual Report 2017, p. 24. 94 OECD, Better regulation practices across the European Union, 2019, p. 90. 95 RSB Annual Report 2018, p. 36.
EPRS | European Parliamentary Research Service
18
Already in 2011, Parliament called for the focus to not be 'exclusively on cost/benefit-analysis but to take a large number of criteria into account'.96 The measures to improve quantification methods as promoted by the Commission,97 such as the transparent storage of data used in previous IAs and evaluations and the streamlining of quantification methods, can be expected to support further progress, but it should be noted that some 'intangible' benefits cannot be fully quantified (or monetised) and therefore require thorough qualitative ex-ante assessment. Moreover, against the backdrop of the goal of keeping IAs in proportion, quantification should not become a goal per se, to the detriment of proper qualitative assessment, a tendency that had been noted by IMPA in the past, at the risk of reducing the IAs' thoroughness and transparency without necessarily improving its quality.98 Notwithstanding this reasoning, this review agrees with the Commission that there is 'room for improvement' in the way impacts are quantified, in particular given the admission that non-transparent and even 'inappropriate assumptions' were apparently used 'regularly' in IAs when applying modelling.99 This should clearly be avoided, as it could be misleading and reduce the accountability of the IA, as the RSB has also recognised.100 Another point to be noted in this context is the need for consistency of the whole analysis. Both costs and benefits have to be covered by an IA – whether they are assessed quantitatively or qualitatively – and they should be based on coherent assumptions and values in order to be comparable.101 Against the backdrop of the plan announced by the new Commission President, Ursula von der Leyen, to develop and apply a new instrument to deliver on the 'One In, One Out' principle,102 it is likely that quantification will gain further importance in IAs, making consistency and comparability all the more important.
Finally, when it comes to the use of analytical methods and models, the variety in the reviewed IAs is also considerable. Among the models most used in IAs in general are PRIMES, GEM-E3, TREMOVE and GAINS.103 This review suggests that the often complex details of the choice of methods and models, as well as the assumptions and uncertainties underlying the IAs are not always transparent and could be presented more systematically in an accessible manner to increase accountability.104
96 European Parliament, resolution of 8 June 2011 on guaranteeing independent impact assessments, paragraph 15.
Parliament reiterated 'the fact that cost-benefit analyses are only one of many criteria' in the resolution of 30 May 2018 on the interpretation and implementation of the Interinstitutional Agreement on Better Law-Making. This is also reflected in the IIA BLM, asking for balanced IAs 'using both qualitative and quantitative analyses'. For the following quote see communication on stocktaking, p. 12.
97 RSB Annual Report 2017, p. 24. The Commission also set up a joint working group where representatives of the RSB, JRC and SG track and analyse issues related to lack of data and necessary methodological know-how to systematise quantification of both costs and benefits. See European Commission, Quantification in Commission Impact Assessments and evaluations, Report of the RSB/SG/ JRC Working Group, 20 March 2018, pp. 4-5.
98 Communication on stocktaking, p. 12: 'It is important to avoid pushing the efforts to quantify costs and benefits beyond a reasonable limit. Simplification is the objective, not quantification per se'. IMPA has stressed the complementarity of quantitative and qualitative assessments, see J. Dunne and W. Hiller, p. 39.
99 RSB Annual Report 2018, p. 36. 100 RSB Annual Report 2018, p. 36. 101 On the importance of correct valuation techniques see JRC Technical reports, On the use of cost-benefit analysis and
multi-criteria evaluation in ex ante Impact assessment, 2017. 102 Ursula von der Leyen, President of the European Commission, Mission letter to Maroš Šefčovič, 1 December 2019,
pp. 3 and 5. This appears to be a change of position, compared to the Commission's past reservations regarding the 'one in, one out' principle, see Commission communication, Completing the Better Regulation Agenda – Better Solutions for Better Results, COM (2017) 651 final, pp. 10-11, and European Commission, staff working document (SWD(2019)156), Taking stock of the Commission's Better Regulation Agenda, pp. 36-37.
103 For more detail on the models used in Commission IAs, see JRC Technical Reports, Modelling for EU Policy support:
Impact Assessments, 2019, p.19. 104 It has to be noted in this context that, regarding the use of models, there are limits in terms of transparency in cases
where these models are not owned by the Commission
Appraising the quality of the European Commission's impact assessments
19
Only 56 % of the IAs appraised were considered to provide an entirely clear explanation of the assumptions and limitations of the analysis, whereas this was found partially clear in 40 % of the cases. A sensitivity analysis to check the certainty of the analysis was provided in only 16 IAs under this review (12 % of the total).
A very positive step in this context, with a view to fostering transparency and interinstitutional cooperation, is the launch of the Competence Centre on Modelling and of the modelling inventory 'MIDAS' within the Commission's Joint Research Centre in 2017 and, last but not least, EPRS access to an interinstitutional version of MIDAS (Midas II) since 2019.
Stakeholder consultation
Consultation with stakeholders is an important part of the Commission's IA process.105 According to the Better Regulation Guidelines, IA reports must contain a dedicated annex presenting the main features of the consultation activities undertaken for an IA. In the context of the IA process, stakeholder engagement has been governed by a well-defined set of minimum standards (e.g. consulting as widely as possible by means of an open and transparent process at a time when stakeholders' views can still make a difference and in a consistent way).106
More specifically, the Better Regulation Guidelines brought about several changes aimed at improving stakeholder participation, such as feedback opportunities over the entire lifecycle of a policy initiative, from initiation to evaluation, including for draft delegated and implementing acts.107 A web-based common portal (the 'Contribute to law-making' website and the 'Have your say' platform) enables all stakeholders to obtain information about new initiatives and express their views at different stages of the law making process. Furthermore, the guidelines include an obligation to run an open public consultation for a period of 12 weeks on all new proposals and evaluations, and to translate the public consultation questionnaires for initiatives featured in Annex I of the Commission work programme into all official languages (into English, French and German for all other initiatives).
Overall, the average quality of this dimension in the IAs under review was found to be satisfactory. The review shows that the formal minimum standard of the compulsory 12-week open public consultation was upheld in 82 % of cases,108 pursuing the generally positive trend for broader stakeholder involvement that was already noted in the 2015 review.109 Nevertheless, while all the IAs appraised did report on the stakeholder consultation process, they were not always clear in presenting the views expressed by the respondents or how they specifically fed into the formulation of the policy. For example, stakeholder views were not always broken down into categories (with general ambiguous statements such as 'the majority of stakeholders think that'), and their views on the different options examined were not always apparent. This review shows that consultation results were found to be presented in a fully transparent and detailed manner in only 51 % of the
105 The importance of stakeholder consultations when preparing EU legislation is also stated in the Treaty on the
European Union, Article 11 and Protocol 2 on the application of the principles of subsidiarity and proportionality. 106 Better Regulation Guidelines, 2017, 'Guidelines on Stakeholder Consultation', p. 68. 107 The most significant revisions to the relevant tools made in 2017 concern the scope and objectives of stakeholder
consultations, the identification of stakeholders, the consultation activities envisaged, their timing and language regime, and the stakeholder feedback mechanism.
108 In 10 % of cases no open public consultation was conducted and in another 8 %, the open public consultation lasted for less than 12 weeks.
109 J. Dunne and W. Hiller, 2016, p. 41.
EPRS | European Parliamentary Research Service
20
IAs under review, whereas in 41 % of the cases their presentation was deemed only partially transparent and detailed. In 8 % of the cases, the presentation was considered untransparent.110
These observations suggest that there is still some scope for improvement when it comes to transparent and detailed reporting on the use of stakeholder feedback in Commission IAs, confirming the findings of the better regulation stocktaking exercise.111 This is important to prevent 'stakeholder fatigue' and ensure stakeholder responsiveness and increased participation in public consultations, which remains overall limited, raising issues of representativeness and potential regulatory capture. Another shortcoming regularly highlighted in the IMPA appraisals, and which can also discourage stakeholders, concerns the quality of the questionnaires for the consultations.112 It was also often not clear from the appraised IAs whether stakeholders were consulted on a problem in general, or if they had the opportunity to comment on the different policy options. Moreover, to the detriment of transparency and accountability, the questionnaires of consultation activities were not always publicly available after the consultation period. Generally, targeted consultations, which were carried out in 85 % of the IAs under this review, appear to be a necessary complement to public consultations, as it is reasonable to expect the knowledge of the stakeholders and the specificity of the questions (and answers) to be higher in this context.
Based on the above findings, this review welcomes the Commission's commitment to improve its consultation questionnaires and responses to stakeholder contributions, as well as to use a wide range of consultation tools.113 The European Parliament has traditionally supported stakeholder engagement, stressing, inter alia, 'the importance of promoting access to the impact assessments and roadmaps prepared by the Commission, of participating in public and/or stakeholder consultations organised by the Commission or the European Parliament'.114 In the same vein, this review supports the Commission's call for 'proactive involvement' of all levels of governance, including national, regional and local entities, in communicating the importance of public consultations to ensure the quality and accountability of EU lawmaking.115
Monitoring and evaluation
The Better Regulation Guidelines place particular emphasis on the monitoring and evaluation aspect of IAs, stating that 'the full benefits of an EU intervention will only be delivered if the policy is implemented and applied properly'.116 According to the 'evaluate first' principle, policy evaluation should be part of the whole policy cycle, not only ex post, but also ex ante, namely when considering the need for possible future policy initiatives. IAs should provide precise monitoring and evaluation indicators as they are essential to assess later on whether an initiative has achieved the intended results and whether it needs revision. These indicators should be linked to the preferred option of an IA and to the operational objectives of the initiative.
110 W. Hiller, 2019, p. 21. 111 Communication on stocktaking, p.9. The answers to the Commission's public consultation on the stocktaking found
consultations and transparency as the two areas where most progress had been achieved since 2015, while at the same time identifying them as the two areas most in need of improvement in the future.
112 Ibid, p. 9. For similar observations see European Court of Auditors, Special Report No 14: 'Have your say!', Commission's public consultations engage citizens, but fall short of outreach activities, 2019, pp. 4, 31-35.
113 Communication on stocktaking, p. 9. 114 European Parliament resolution of 18 April 2018 on the Annual Reports 2015-2016 on subsidiarity and proportionality. 115 Communication on stocktaking, p. 7, see also European Parliament resolution of 30 May 2018, on the interpretation
and implementation of the Interinstitutional Agreement on Better Law-Making. 116 Better Regulation Guidelines, 2017, p.9.
Appraising the quality of the European Commission's impact assessments
21
The fact that 96 % of the IAs appraised in this review included monitoring and evaluation provisions would suggest that these obligations have generally been respected. Of the ten dimensions covered under the present analysis, this aspect was on average among those that scored better, being largely in the satisfactory range throughout the period under review. As already noted by IMPA in 2018, monitoring and ex-post evaluation requirements were included in IAs more systematically and were generally accompanied by relevant indicators.117 At the same time, however, in 28 % of the IAs appraised, the monitoring indicators were found to be rather broad and only partially relevant in relation to the objectives pursued. The Commission and Parliament share the view that monitoring and evaluation is an important part of each IA.118 The Commission recently reiterated the importance of defining a suitable monitoring framework with clear results and impact indicators that would allow for more systematic collection of relevant, EU-wide (and thus, comparable) data.119 This commitment is certainly a positive sign of efforts to strive for more specific and pertinent monitoring provisions, which would also improve the quality of data available, including for quantification (see chapter above).
Last, but not least, another important development worth noting in this context is the fact that, owing to increasing time constraints towards the end of the 2014-2019 legislative term, in several instances the Commission conducted ex-ante IA and ex-post evaluations of existing legislation in parallel in a single 'back to back' process, not therefore abiding by the normal and more desirable sequence of running the evaluation first to feed into the subsequent IA. While this practice, which was formalised in a specific tool in the revision of the Better Regulation Guidelines in 2017,120 may help address the problem of time pressure, it also raises several questions. Considering that ex-post evaluations should, according to the 'evaluate first' principle, precede ex-ante IAs, it is unclear whether the parallel timing of both exercises can serve their different objectives. Moreover, ex-ante and ex-post assessments differ considerably as regards their methodologies and evidence-base, with ex-ante IA projecting potential impacts in the future, and ex-post evaluation aiming at assessing effects after an initiative has been implemented. The challenges created by these 'misfits', notably in terms of stakeholder consultation, for the quality of both exercises have been already pointed out by scholars and by the IMPA unit.121 They were also raised in the Commission's own internal discussions on this issue.122 Therefore, clarification by the Commission of the cases where 'back to back' is used and of its methods and procedures would be welcome, in particular if this technique were to be used more frequently in the future.
117 W. Hiller, 2019, p. 21. 118 See, paragraph 39, European Parliament resolution of 30 May 2018, on the interpretation and implementation of the
Interinstitutional Agreement on Better Law-Making, op. cit. 119 RSB Annual Report 2018, p. 40. 120 See Better Regulation Toolbox, Tool 52. p. 376. 121 See S. Smismans, 'Policy evaluation in the EU: The Challenges of linking ex ante and ex post appraisal', European
Journal of Risk Regulation, Vol. 1/2015; Symposium on Policy Evaluation in the EU, 2015, pp. 6-26., and W. Hiller, 2019, p. 2.
122 The 'Commission's officials had mixed views on impact assessments and evaluations carried out back-to-back'. See staff working document (SWD(2019) 156), Taking stock of the Commission's Better Regulation Agenda, Accompanying the document Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. Better regulation: taking stock and sustaining our commitment, SWD(2019) 178, footnote 86: 'some thought that in this case the evaluation results might be pre-judged and thus not useful to inform the impact assessment. Others found it a pragmatic and efficient way to comply with the 'evaluate first' principle'.
EPRS | European Parliamentary Research Service
22
Integration of the Regulatory Scrutiny Board's recommendations
An analysis of the follow-up to the comments expressed in its opinions by the RSB, in the final version of the IA, is part of any IMPA initial appraisal. This aspect is considered important by Parliament as it helps trace the evolution of a draft IA and shows, albeit partially as draft IAs are not public, whether the RSB suggestions for revisions of any problematic parts of the draft IA were taken on board sufficiently when finalising the IA report. As IMPA initial appraisals concern the final version of the IA report, they are considered to play a complementary function to the scrutiny work of the RSB by also drawing attention to aspects that are particularly important to Parliament and by further strengthening the Board's messages where necessary.123 In its latest annual report, the RSB referred directly to the IMPA initial appraisals, indicating that it used them to assess its own work.124
As indicated before, the RSB issues either a positive or a negative opinion on the draft IA. In the latter case the draft IA has to be improved and resubmitted for a second (and normally last) opinion, before the legislative proposal can go forward. In the absence of a positive opinion, the Commission must explain publicly, why it adopts a proposal anyway. In the second half of 2016, the RSB introduced an additional category of opinions – 'positive with reservations' – as it considered the choice between a negative or a positive opinion 'too binary'.125 A positive opinion with reservations requires significant revisions of the draft IA, but no re-submission. According to the RSB, following the introduction of this new category, the initial rejection rate decreased, with 43 % of all IAs submitted in 2018 receiving a 'positive with reservations' opinion, on first submission.126 This did not mean, however, that the quality of draft IAs had notably improved. In fact, the RSB admits that to avoid delays, 'the Board sometimes gives a conditional stamp of approval to work that still needs substantial improvement in some important respects, on the understanding that listed shortcomings are appropriately addressed.'127 This, however, is not always the case in the final IA.
Overall, the average quality of this dimension in the IAs appraised ranked just above the satisfactory threshold. The majority of Commission IAs seen by IMPA during the period under review were transparent about the recommendations for improvements made by the RSB and explained how they had addressed them in a specific annex, as required by the Better Regulation Guidelines. However, compliance with the RSB recommendations as regards their substance was not always apparent. A considerable number of the IMPA appraisals (67 %) concluded that the final IA report addressed the comments of the RSB only partially. This trend is particularly noticeable when the two categories of opinions are compared in the light of the response to the RSB comments.
Figure 5 shows that the share of fully integrated comments was higher (39 %) for IAs that had received a positive opinion with reservations, than for those that had received a simple positive opinion (28 %).
123 See C. Radaelli, 'Halfway Through the Better Regulation Strategy of the Juncker Commission: What Does the Evidence
Say?', Journal of Common Market Studies, Volume 56, 2018, Annual Review, pp 85-95. 124 European Commission, Regulatory Scrutiny Board Annual Report 2018, pp. 21-22. 125 RSB, Annual Report 2017, p. 12. 126 RSB Annual Report 2018, p. 10. 127 Ibid.
Appraising the quality of the European Commission's impact assessments
23
Figure 5 – Integration of RSB comments following positive opinions and positive opinions with reservations in the appraised IAs published between July 2015 and December 2018
Source: IMPA, EPRS.
These findings seem to indicate that in the case of a positive opinion 'with reservations', the author DGs were more inclined to work on the RSB comments than when the RSB opinion was positive. However, these figures also show that there is still considerable scope for improvement in terms of taking RSB recommendations on board, in particular in the case of positive opinions with reservations which normally point to significant concerns. In its last annual report, the RSB also expressed concerns about the limited follow-up given to this category of opinions, compared to negative opinions, and indicated that it may have to more stringent in delivering them in the future.128
The partial integration of RSB recommendations seems to suggest that the author DGs either found it difficult to address the RSB comments (in full) or, for other reasons, did not report how the revisions were made. Another explanation for limited modifications of a draft IA following an RSB opinion could be the already mentioned increasing time constraints the IA process faced towards the end of the 2014-2019 legislature, when the time between the last opinion of the RSB and the adoption of the IA and the legislative proposal decreased considerably (on average from 125 calendar days in 2017 to 65 calendar days in 2018). While time constraints are sometimes unavoidable in the political environment and tend to intensify particularly towards the end of a legislature, their potential negative impact on the quality of the final IAs might be worth careful consideration, based on these findings.
During the period under review, the requirement in principle for a positive opinion of the RSB on the IA in order for the initiative to go forward was respected in all but two cases.129
128 RSB Annual Report 2018, p. 43. 129 There were in fact three cases of an IA receiving two negative opinions during the period of this review. They were
the IA regarding the 'free flow of non-personal data', the IA regarding the 'use of energy from renewable sources' and the IA on the 'Sustainable Finance initiative'. In this last case however, the two negative opinions were exceptionally followed by a third positive opinion with reservations (see RSB Annual Report 2018, footnote 2, p. 10). For more detail, see the respective IMPA appraisals: H. Dalli, Free flow of non-personal data in the European Union, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, February 2018; V. Kononenko, Use of energy from renewable sources, initial appraisal of the European Commission impact assessment, EPRS, European Parliament, June 2017 and S. Vettorazzi, An EU framework to facilitate investments in environmentally sustainable economic activities, initial appraisal of the European Commission impact assessment, EPRS, European Parliament, April 2019.
EPRS | European Parliamentary Research Service
24
Overall, it can be concluded that transparent explanations as regards the follow-up of the RSB comments and, in particular, in cases where certain RSB remarks were not taken on board, benefit the quality of an IA. They enhance the information for policymakers and stakeholders about the reasoning and evidence base behind a Commission initiative and increase its transparency. The data gathered for this review confirms that the IAs that were found to have integrated the RSB comments more thoroughly than 'partially', tended to score better overall. IAs that were appraised by IMPA as having very low level of integration of the RSB comments (4 cases out of 132), received a notably lower overall score, and did not reach the 'satisfactory' threshold.
Coherence of the IA with the proposal
The last dimension analysed in any IMPA initial appraisal is the coherence of the IA with the Commission's legislative proposal. The comparison of the IA with the proposal can provide some insight as to whether and how the conclusions of the IA were followed by the political decisionmakers. In IMPA appraisals this is checked by comparing the IA to the proposal in terms of overall coherence between the preferred course of action (or other key findings) of the IA and the proposed policy measures and by comparing monitoring and evaluation provisions presented in the IA with those described in the proposal. Parliament has considered in this respect that 'consistency between the explanatory memorandum and the impact assessment related to the same proposal is necessary'.130
In 71 % of relevant cases reviewed for this study, the proposals were found to be fully coherent with the preferred option (or the findings of the corresponding IAs). The rest (29 %) were 'partially' coherent, meaning that, not all provisions were carried over to the legislative proposal or that some modifications were made.131 The reasons for this partial lack of coherence varied from case to case and concerned mostly minor elements that did not put into question the overall preferred course of action. In some cases, technical elements of the IAs were left out of the proposal or suggested monitoring mechanisms and, in particular, the proposed indicators were not referred to in the proposal itself (in 16 % of cases) or were carried over in the proposal but with a lower level of detail or with some modifications compared with what had been envisaged by the IA (44 % of cases).132 In other cases new technical provisions that had not been discussed at the IA stage, were added in the proposal. On occasion, changes were more substantial than merely a technical redress.133
The main conclusion for this dimension is that major diversions from the findings of the IA, which appear to occur rarely, should be duly justified and clearly explained in the explanatory
130 European Parliament resolution on the interpretation and implementation of the Interinstitutional Agreement on
Better Law-Making, op.cit, paragraph 42. 131 Hiller W. 2019, p.21. 132 For instance, in the proposal on the empowerment of national competition authorities (NCAs), the detailed
monitoring and evaluation approach features in the implementation plan rather than the legislative text. See E. Kramer, Empowerment of national competition authorities, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, July, 2017.
133 For example, the proposal regarding cross-border parcel delivery services appeared to generally follow the recommendations of the IA, but it differed from the IA with regard to treatment of SMEs. See V. Kononenko with G. Caruso, Digital Single Market: Cross-border parcel delivery services, initial appraisal of a European Commission Impact Assessment, EPRS, European Parliament, July 2016.
Appraising the quality of the European Commission's impact assessments
25
memorandum of the legislative proposal,134 as requested by Parliament,135 so that the co-legislators are fully informed about the line of action that has been pursued and the reasoning behind it.136
134 For example, see H. Dalli, Exchange of Information on Third Country Nationals – European Criminal Records
Information System (ECRIS), initial appraisal of a European Commission impact assessment, EPRS, European Parliament, March, 2016. Although the proposal differed from the preferred option in the IA in one of its important elements, the reason for that (urgency caused by the Paris terrorist attack in 2015) was clearly explained in the explanatory memorandum.
135 European Parliament resolution on the interpretation and implementation of the Interinstitutional Agreement on Better Law-Making, op.cit, paragraph 42.
136 The same is true in the absence of an IA for a proposal, where the Commission itself has stressed the importance of ensuring full and transparent information. This was for instance the case in 30 % of the priority initiatives for 2018, which is seen increasingly critically by the Members of European Parliament. See European Commission, decision (SWD(2019) 156), Taking stock of the Commission's Better Regulation Agenda. Accompanying the document Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. Better regulation: taking stock and sustaining our commitment, SWD(2019) 178, pp. 17-18.
EPRS | European Parliamentary Research Service
26
4. Conclusions IMPA's appraisals of the Commission's IAs have become an integral part of the European Parliament's scrutiny of the executive, and its implementation of the IIA BLM. They have been acknowledged both inside and outside the EU institutions. This study notes the findings of the Commission's stocktaking exercise that, while the rationale for better regulation is stronger than in the past and 'a cultural change has been taking hold within the Commission', this change is not yet irreversible.137 In the same vein, the IA 'culture' has also progressed significantly in Parliament, which pays more attention to the presentation of IAs and is increasingly aware of the standards they are supposed to comply with. This has clearly led to increased expectations regarding the quality of the Commission's IAs, and also to more severe criticism of proposals that are presented with a weak IA, or, for that matter, those presented without any IA, in particular when this is not justified convincingly.
Based on the review undertaken by IMPA, it appears that the overall quality of 66 % of the appraised IAs published by the Commission between July 2015 and December 2018 was on average 'satisfactory', while 5 % were considered good and 29 % 'poor'. This concerns all 10 dimensions of an IA scored together. The overall average quality of the IAs appraised slightly exceeded the satisfactory level throughout most of the period under review, as can be seen below in Figure 6. It specifically shows an improvement in quality from 2016 to 2017 and a slight decrease in overall quality from 2017 to 2018, a trend also detected by IMPA for the 2012 to 2015 period. This could be explained, at least in part, by the increase in legislative proposals towards the end of the legislative term and the growing time constraints that the author DGs of the Commission were facing when finalising the impact assessment reports.
Figure 6 – Overall average quality of the appraised IAs published between January 2016 and December 2018138
Source: IMPA, EPRS.
137 Communication on stocktaking, op. cit, pp. 4 and 9-10. 138 This figure shows the data for the three full years under this review.
Appraising the quality of the European Commission's impact assessments
27
Also, similar to the results of the 2015 review, the findings of the present study indicate that the quality of the IAs seems to continue an overall positive trend since 2012.139 This trend reflects the considerable efforts of the Commission in the framework of the better regulation agenda.
In contrast to the observations in IMPA's 2015 review, which saw IAs to be 'improving continuously, although not in in a uniform manner across all Commission DGs', the present review has not found any particular pattern in terms of IA quality differences between the Commission DGs, suggesting that the adoption of the Better Regulation Guidelines has led to a more consistent approach across the services. The availability of relevant, reliable and comparable data sources, the applied research and methodology, and potential time constraints are important drivers of an IA's quality. Variations may also depend on whether an IA is carried out for a completely new or very technical area of legislation, where data scarcity might be particularly challenging.140 In the light of this diversity, the present review acknowledges the observations of the RSB, pointing to the need to adapt IAs to the 'features and needs of each exercise'.141 However, if and where a high(er) degree of flexibility is envisaged, owing to either 'time pressure, lack of data or pre-existing political guidance from the legislator', as the RSB Annual Report 2018 puts it,142 clear and transparent explanations of, and justifications for, any deviation from the better regulation rules are essential. At the same time, there is certainly a limit to the degree of possible simplification of an IA, as it must still be fit for purpose.
While the general quality trend of IAs covered in the present review can be considered positive, the fact that only 5 % of them were rated as 'good' – and, more strikingly, that 29 % of them were found to be poor – indicates considerable scope for improvement. This is all the more true when this finding is taken in conjunction with the fact that a third of the proposals included in the joint declarations on the EU's legislative priorities for 2017 and 2018-19 were not accompanied by IAs, contrary to the Commission's commitment in the Interinstitutional Agreement on Better Law-Making.143
Figure 7 below shows the average quality of the total number of IAs broken down into the 10 essential dimensions. As indicated in the analysis, overall, all 10 policy dimensions reached or exceeded the satisfactory level, except two, the policy objectives and the assessment of relevant impacts, which rated lower. The dimensions that scored on average the best under the present review were the problem definition, followed by monitoring and evaluation, and the coherence of the proposal with the IA. The weakest related to the presentation of policy options and the assessment of their relevant direct and indirect impacts, two rather central sections of any IA.
139 J. Dunne and W. Hiller, 2016, p. 37. 140 See also the RSB Annual Report 2018 for a similar observation, pp. 29-30. 141 RSB Annual Report 2018, p. 17. 142 Ibid p. 29. 143 In its resolution of 30 May 2018 on the interpretation and implementation of the Interinstitutional Agreement on
Better Law-Making, Parliament noted that 'a significant number of Commission proposals were not accompanied by impact assessments and that committees have expressed concern that the quality and level of detail of impact assessments varies from the comprehensive to the rather superficial'. It also pointed out that 'in the first phase of application of the new IIA 20 out of 59 Commission proposals included in the 2017 joint declaration were not accompanied by impact assessments'; noting in this regard that 'while it is in any case foreseen that initiatives which are expected to have significant social, economic or environmental impact should be accompanied by an impact assessment, paragraph 13 of the IIA also states that the initiatives included in the Commission Work Programme or the joint declaration should, as a general rule, be accompanied by an impact assessment'; op.cit, paragraph 22.
EPRS | European Parliamentary Research Service
28
Figure 7 – Average quality of the 10 essential dimensions of the appraised IAs published between July 2015 and December 2018.
Source: IMPA, EPRS.
As indicated at the beginning of this study, despite an overall average score of 'satisfactory', important individual parts of a given IA might have been considered of either poorer or better quality. It is precisely the close look at each of the 10 dimensions individually that enables structural strengths and weaknesses in the different stages of the IA process to be identified, with a view to finding solutions to tackle them, so as to ensure that this tool is of the highest quality to effectively support policymaking.
Nearly all the IAs appraised under this review presented a good definition of the problem and its drivers, generally on the basis of solid sources. This is a significant improvement compared to the 2015 review, which is all the more important, as strong problem definition constitutes the essential basis for the intervention logic of any IA. At the same time, it appeared that the scale of the problem could have sometimes been better substantiated, to illustrate the effects of the problem on the groups of stakeholders most affected. Even though further improvements are still possible, the positive findings regarding this first and essential dimension of the IA should serve as a good starting point when it comes to striving for further effectiveness and transparency in the IA process in the coming years.
As far as the definition of clear and distinct objectives in the reviewed IAs is concerned, the situation is more mixed. While nearly all IAs featured general and specific objectives, they were not always formulated in a precise and distinct manner, and, moreover, many IAs did not present any operational objectives. When the latter were featured, the majority of them did not fulfil all the S.M.A.R.T criteria laid out in the Better Regulation Guidelines. In particular, the timeframe for the achievement of the objectives was often not clear, and the objectives themselves did not always reflect concrete deliverables. This can be expected to affect the monitoring and ex-post evaluation
Appraising the quality of the European Commission's impact assessments
29
of the initiative, as clear and smart objectives are the benchmark. Improvements can perhaps be expected in this area in the light of the stronger focus the Commission has said it intends to put on ex-post evaluation in the future, as the quality of the objectives of an initiative and the effectiveness of its ex-post evaluation go hand in hand.
The important presentation of policy options to address the problem was found to be one of the weakest parts of the IAs under review. On the positive side, the majority of the IAs presented at least three alternatives to the baseline, the minimum requirement of the Better Regulation guidelines being two alternatives. However, while in general the range of options seems to have improved slightly compared to the years before 2015, the same cannot be said with regard to the description of the different options, which was often found, as before, to be not entirely clear and/or balanced, even calling into question the viability of some options. These observations are problematic as they can cast some doubt on the transparency, objectivity and accountability of the IA process.
Another crucial aspect of every IA, the necessary analysis of all relevant direct and indirect impacts of the policy options presented, also scored lowest on average of the 10 dimensions in the IAs under review. Similarly to the 2015 review, of the three mandatory impacts, the analysis of economic impacts and its quality was still found to prevail over social and in particular environmental impacts. Furthermore, the potential impacts on SMEs and competitiveness, which should, where relevant, be considered systematically in all impact assessments according to the Better Regulation Guidelines, were not analysed in sufficient depth in a substantial number of cases. As for other potentially relevant impacts, such as impacts on fundamental rights, territorial and gender equality-related impacts, these were covered in a (very) limited number of the IAs appraised. Therefore, this review concurs with the Commission's acknowledgement in its own stocktaking exercise that 'impacts could be better taken into account',144 while also recognising the need to keep the assessment in proportion, as highlighted in the IIA BLM.145 In any event, improved screening and identification of the potential relevant impacts appears necessary. Furthermore, proper justification regarding the impacts assessed or deemed irrelevant should be provided for the benefit of transparency and accountability of the IA.
The present study also agrees with the conclusion of the Commission's own stocktaking exercise regarding the principles of subsidiarity and proportionality in IAs.146 The present review found that the quality of the assessment of these aspects just reached the satisfactory threshold, frequently noting that the explanation and substantiation of the Commission's claims under this section was limited to general statements and circular reasoning. The integration of a new model grid into every IA to assess subsidiarity and proportionality, recently taken on board by the Commission following the recommendations of the report of the Task Force on Subsidiarity, Proportionality and 'Doing Less More Efficiently', may help to improve this section in the future.147
The important aspect of the quality and transparency of data and research in the IAs under the present review was one of the dimensions that scored better, with appraisals generally acknowledging the use of reliable and solid data sources and research. On a more critical note, however, there were at times inconsistencies in research and the data used and, importantly, in a number of instances, a lack of transparency of data, namely owing to unclear or inaccessible data
144 Communication on stocktaking, 2019. p. 18. 30 % of respondents in the stocktaking exercise replied that in their
opinion the Commission had not considered environmental and social impacts sufficiently. 145 IIA BLM, paragraph 12. 146 Communication on stocktaking, 2019, p.28 147 European Commission, 'Active Subsidiarity, A new way of working', Report of the Task Force on Subsidiarity,
Proportionality and 'Doing Less More Efficiently', July, 2018.
EPRS | European Parliamentary Research Service
30
sources, limiting the accountability of the IA. This also concerned a detected lack of explanation of assumptions and uncertainties underlying some IAs, which is of particular relevance in the context of the quantification of the costs and benefits of policy options. Generally, the Commission has stepped up efforts to provide quantified assessments in recent years, mainly through cost-effectiveness analysis. In this context, this study notes the tension between the resource-intensive investment necessary to undertake pertinent quantification and cost-benefit analyses on the one hand, and the need to keep IAs accessible and proportional on the other. It is worth reiterating here Parliament's call to provide balanced IAs, based on both qualitative and quantitative assessments, not limited to cost-benefit analyses, and to ensure the consistent transparency of both.
Stakeholder consultations were identified in this study as another aspect of mixed quality. Generally, the Commission had clearly made efforts to consult stakeholders broadly, usually giving them 12 weeks to provide their feedback. However, shortcomings were detected in the IAs covered by the present review, regarding both the quality of the questionnaires of some consultations and, more specifically, the often very general reporting of the consultations in the IAs, where the effect of stakeholder views was not transparent. Limited numbers of replies to several consultations also raised questions of representativeness148 This matters, as a poor presentation of how stakeholders' views fed into the IA might discourage them from participating in future consultations, which would work against the future aim of promoting consultations at all governance levels.
Monitoring and evaluation featured steadily among the dimensions that scored better for the IAs in the present review, based on the fact that nearly all of them provided such provisions, with relevant indicators in the majority of cases. In addition, the Commission remains attentive to the importance of the systematic collection of relevant data. This is encouraging, as the pertinence of the provisions in this area affect not only the quality of the ex-post evaluation of an initiative, but also the possibility to carry out quantitative assessments, which depend on the availability of sufficient relevant and reliable data. Furthermore, against the backdrop of the 'evaluate first' principle, the 'back to back' practice, in which owing to time constraints the Commission has carried out ex-ante and ex-post assessments in one (short) exercise, is open to criticism. The Commission should clarify the methods employed in these cases, with a view to preserving the distinct objectives, quality and accountability of both ex-ante and ex-post assessments in the long term.
When it comes to the uptake of RSB recommendations in the final IA, the present review showed broad variation, and overall its findings are mixed. Depending on the individual case, the comments of the RSB on the draft IAs were mostly addressed 'partially' and to varying degrees in the final IA. The IMPA appraisals raised in particular the issue of the IAs' reporting on how the comments of the RSB had been addressed, pointing out the need for explicit and transparent explanations, in particular in the light of the fact that draft IAs are currently not publicly available.149
The last dimension checked for each IA under this review, the coherence between the IA and the legislative proposal, scored well, as in most cases, the legislative proposal appeared to follow the preferred option (or, in the exceptional cases where there was none, the findings of the IA) and the suggested monitoring and evaluation mechanisms. In the event of diversions from the IA, these concerned partial, mostly minor, differences of varying significance, but would not generally alter the decision in principle to take a certain policy measure. Where that is the case, a clear and
148 Communication on stocktaking, 2019, p. 12. 149 In a 2018 judgment, ClientEarth v European Commission, the Court of Justice ruled that the European Commission
cannot refuse access to important documents related to impact assessment reports that are used as a basis for its decision-making process. It is not yet known how this ruling will be implemented by the Commission.
Appraising the quality of the European Commission's impact assessments
31
transparent justification should be provided in the explanatory memorandum of the proposal, to ensure the political decision-makers are fully informed.
IMPA's present review supports the finding of the Commission's stocktaking exercise that there is a need for cooperation between the institutions to make better regulation entirely successful. This view was also expressed in the IIA BLM and the Parliament's own resolutions on better regulation. The start of the new parliamentary term and the new von der Leyen Commission should provide the political momentum necessary for enhanced cooperation, not only between the three EU institutions, but also with the national and regional levels.
Closer interinstitutional cooperation should begin at the very start of the planning stage, with more active exchange of information on upcoming legislative initiatives, as well as on cases where the Commission intends to present an initiative without an IA. The still-to-be-established interinstitutional database on the state of play of legislative files provided for in the IIA BLM, in which the Parliament has suggested including information on the IA process,150 could certainly be useful in this respect.
With the present study, IMPA is making its contribution to the joint effort to make the better regulation activities, and in particular impact assessments, more effective and useful for policymakers. Considering all 10 dimensions of the IAs analysed in the study, the weaknesses of two crucial parts – the presentation of the policy options, and the assessment of their relevant impacts – are certainly notable. The same is true of the findings regarding the proportionality and subsidiarity assessments. Lastly, methodological questions, for instance regarding the quantification of impacts and the use of simplified 'back to back' assessments, should be reconsidered, as they may have an impact on the effectiveness, reliability and usefulness of future IAs.
To conclude, the need to strike a balance between, on the one hand, the depth and complexity of research necessary to inform policymakers properly, and, on the other, the readability and transparency of IAs looks likely to continue to be a common overarching challenge for the entire IA process for the foreseeable future.
150 See paragraph 66 of the European Parliament resolution on the interpretation and implementation of the
Interinstitutional Agreement on Better Law-Making, op.cit.
EPRS | European Parliamentary Research Service
32
REFERENCES
2016 Interinstitutional Agreement on Better Law-Making, OJ L 123/1, 12.5.2016.
2018 OECD Regulatory Policy Outlook.
Active Subsidiarity. A new way of working, Report of the Task Force on Subsidiarity, Proportionality and 'Doing less more efficiently', European Commission, 2018.
Collova C., Ex-Ante impact assessment in the Commission's new Better Regulation Guidelines, EPRS, European Parliament, December 2015.
Collova C., Action Plan on Building a Capital Markets Union: EU securitisation framework, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, September, 2015.
Dalli H., Free flow of non-personal data in the European Union, initial appraisal of the European Commission impact assessment, EPRS, European Parliament, February 2018.
Dunne J. and Hiller W., European Parliament work in the fields of Impact Assessment and European Added Value, Activity Report for July 2014 - December 2015, Annex One, EPRS, European Parliament, April 2016.
European Commission, The European Green Deal, December, 2019.
European Commission, Better regulation – taking stock and sustaining our commitment, 2019.
European Commission, Better regulation for better results - An EU agenda, COM (2015)0215.
European Commission, A Union that strives for more. Political guidelines for the next European Commission 2019-2024.
European Commission, 'Active Subsidiarity, A new way of working', Report of the Task Force on Subsidiarity, Proportionality and 'Doing Less More Efficiently', July, 2018.
European Commission, Completing the Better Regulation Agenda - Better Solutions for Better Results, October, 2017.
European Commission, Quantification in Commission Impact Assessments and evaluations, Report of the RSB/SG/JRC Working Group, 20 March 2018.
European Commission, Regulatory Scrutiny Board Annual Report 2018.
European Commission, Regulatory Scrutiny Board Annual Report, 2017.
European Commission, Regulatory Scrutiny Board Annual report, 2016.
European Commission, staff working document, Better Regulation Guidelines, SWD(2017)350.
European Commission, staff working document (SWD(2019)156), Taking stock of the Commission's Better Regulation Agenda, Accompanying the document Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, Better regulation: taking stock and sustaining our commitment, SWD(2019)178.
European Court of Auditors, Special Report No 14: 'Have your say!' Commission public consultations engage citizens, but fall short of outreach activities, 2019
European Parliament resolution of 30 May 2018 on the interpretation and implementation of the Interinstitutional Agreement on Better Law-Making (2016/2018(INI)).
European Parliament resolution of 18 April 2018 on the Annual Reports 2015-2016 on subsidiarity and proportionality (2017/2010(INI)).
European Parliament, Report on the interpretation and implementation of the Interinstitutional Agreement on Better Law-Making, (2016/2018(INI)).
European Parliament resolution of 27 November 2014 on the revision of the Commission's impact assessment guidelines and the role of the SME test, (2014/2967(RSP)).
Appraising the quality of the European Commission's impact assessments
33
European Parliament resolution of 8 June 2011 on guaranteeing independent impact assessments (2010/2016(INI)).
Hiller W., European Parliament work in the fields of Impact Assessment and European Added Value. Activity Report for 2018, EPRS, European Parliament, March 2019.
Hiller W. European Parliament work in the fields of Impact Assessment and European Added Value. Activity Report for 2016, EPRS, European Parliament, March 2017; Impact Assessment Institute, A year and a half of the Better Regulation Agenda: what happened?, 2017.
Joint Research Centre, The debate on the EU Better Regulation Agenda: a literature review, 2019.
Joint Research Centre, JRC Technical Reports, Modelling for EU Policy support: Impact Assessments, 2019.
Joint Research Centre, JRC Technical reports, On the use of cost-benefit analysis and multi-criteria evaluation in ex ante Impact assessment, 2017.
Kononenko V., EU free trade agreement with Australia and New Zealand, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, February 2018.
Kononenko V., Adapting the road haulage market to developments in the sector: road transport operators and access to the market, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, October 2017.
Kramer E., Pan-European Personal Pension Product, initial appraisal of the European Commission impact assessment, EPRS, European Parliament, October 2017.
Kramer E., Empowerment of national competition authorities, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, July, 2017.
Von der Leyen U, President of the European Commission, Mission letter to Maroš Šefčovič, 1 December 2019.
Listorti G., Basyte Ferrari E., Acs S., Munda G., Rosenbaum E., Paruolo P. and Smits P., The debate on the EU Better Regulation Agenda: a literature review, Publications Office of the European Union, Luxembourg, 2019.
Marcs J., Midoes C. and Schout A., 'To the Commissioner responsible for better regulation', in Demertzis M. and Wolff G. (eds.), Braver, greener, fairer, Memos to the incoming EU leadership 2019-2024, Bruegel, September 2019.
OECD Report on Better Regulation Practices across the European Union, 2019.
Radaelli C., 'Halfway Through the Better Regulation Strategy of the Juncker Commission: What Does the Evidence Say?', Journal of Common Market Studies, 2018, Vol. 56, Annual Review.
Smismans S., 'Policy evaluation in the EU: The Challenges of linking ex ante and ex post appraisal', European Journal of Risk Regulation, Vol. 1/2015, Symposium on Policy Evaluation in the EU, 2015.
Taulègne B., 'Territorial impact assessment: A key practical experience of territorial scrutiny', in De Feo A. and Laffan B., (eds.), Scrutiny of EU policies, European University Institute 2017.
Teasdale A., The Cost of Non-Europe, in: European Parliament, Thinking about the future of Europe 'Ideas Papers' for the European Parliament Administration’s Innovation Day, EPRS 2019.
Tuominen M., Multilateral court for the settlement of investment disputes, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, November 2017.
Vettorazzi S., An EU framework to facilitate investments in environmentally sustainable economic activities, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, April 2019.
Vettorazzi S., Mutual recognition of goods lawfully marketed in another Member State, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, March 2018.
Vettorazzi S. Protection of workers from exposure to carcinogens or mutagens: third proposal, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, June 2018.
Vikolainen V. Revision of the European Electronic Road Toll Service, initial appraisal of a European Commission impact assessment, EPRS, European Parliament, October 2017.
EPRS | European Parliamentary Research Service
34
Annex – List of the 132 initial appraisals of Commission impact assessment reports published between July 2015 and December 2018 An EU framework to facilitate investments in environmentally sustainable economic activities, 12 April 2019.
Establishing a cybersecurity competence centre and a network of national coordination centres, 19 February 2019.
Preventing the dissemination of terrorist content online, 4 February 2019.
European Maritime Single Window environment, 18 December 2018.
Revising the rules on the re-use of public sector information, 5 December 2018.
Enabling sovereign bond-backed securities, 5 December 2018.
Loan servicers and buyers and recovery of collateral, 29 November 2018.
Revising the Visa Information System, 15 November 2018.
Revision of the Fisheries Control System, 25 October 2018.
Implementation and functioning of the '.eu' top level domain name, 12 October 2018.
Promoting fairness and transparency in the online platform environment, 21 September 2018.
Type-approval requirements for the general safety of vehicles, 18 September 2018.
Setting CO2 emission performance standards for new heavy-duty vehicles, 13 September 2018.
Cross-border mobility of companies and use of digital solutions in company law, 12 September 2018.
Setting minimum requirements for water reuse, 6 September 2018.
Streamlining measures for advancing the realisation of the Trans-European Transport Network, 28 August 2018.
Cross-border distribution of collective investment funds, 27 August 2018.
Improving road infrastructure safety management, 13 July 2018.
European production and preservation orders and the appointment of legal representatives for gathering electronic evidence, 13 July 2018.
Security of ID cards and of residence documents issued to EU citizens and their families, 13 July 2018.
EU consumer protection rules, 10 July 2018.
Revision of the Explosives Precursors Regulation, 10 July 2018.
Levelling off the European cross-border payments in euros, 9 July 2018.
Marine litter: single-use plastics and fishing gear, 9 July 2018.
Unfair trading practices in the food supply chain, 5 July 2018.
Minimum loss coverage for non-performing exposures, 3 July 2018.
Protection of workers from exposure to carcinogens or mutagens: third proposal, 27 June 2018.
Appraising the quality of the European Commission's impact assessments
35
Access to financial data by law enforcement authorities, 25 June 2018.
Strengthening EU cooperation on health technology assessment, 18 June 2018.
Fostering cross-border investment – Law applicable to the third-party effects of assignments of claims, 18 June 2018.
Multi-annual plan for western Mediterranean demersal fisheries, 13 June 2018.
European crowdfunding service providers for business, 29 May 2018.
European Labour Authority, 25 May 2018.
A framework for EU covered bonds, 18 May 2018.
Revision of the visa code, 27 April 2018.
Rates of value added tax, 25 April 2018.
A renewed partnership with the countries of Africa, the Caribbean and the Pacific, 17 April 2018.
Mutual recognition of goods lawfully marketed in another Member State, 6 April 2018.
Strengthening the market surveillance of products, 27 March 2018.
Revision of the drinking water directive, 27 March 2018.
Port reception facilities for the delivery of waste from ships, 22 March 2018.
Clean Vehicles Directive, 22 March 2018.
Transparent and predictable working conditions, 22 February 2018.
EU free trade agreement with Australia and New Zealand, 15 February 2018.
Interoperability between EU information systems for security, border and migration management, 14 February 2018.
Free flow of non-personal data in the European Union, 13 February 2018.
Access to the international market for coach and bus services, 1 February 2018.
Review of CO2 emission standards for new cars and vans, 31 January 2018.
Rail passengers' rights and obligations, 12 January 2018.
Common rules for certain types of combined transport of goods, 11 January 2018.
European Market Infrastructure Regulation, Authorisation of central counterparties (CCPs) and recognition of third-country CCPs, 10 January 2018.
EU Cybersecurity Agency and cybersecurity certification, 20 December 2017.
Review of the European supervisory authorities (ESAs) (micro-prudential supervision), 20 December 2017.
Import of cultural goods, 19 December 2017.
European Market Infrastructure Regulation - Regulatory Fitness and Performance (REFIT) proposal, 15 December 2017.
Combating fraud and counterfeiting of non-cash means of payment, 8 December 2017.
Multilateral court for the settlement of investment disputes, 24 November 2017.
EPRS | European Parliamentary Research Service
36
Road transport: Driving times, breaks, rest periods, tachographs and posting of drivers, 21 November 2017.
Safeguarding competition in air transport, 20 November 2017.
Pan-European Personal Pension Product (PEPP), 27 October 2017.
Use of vehicles hired without drivers for carriage of goods by road, 16 October 2017.
Adapting the road haulage market to developments in the sector: road transport operators and access to the market, 16 October 2017.
Revision of the European Electronic Road Toll Service, 12 October 2017.
Single digital gateway, 10 October 2017.
Single Market Information Tool, 5 October 2017.
European business statistics, 27 September 2017.
Monitoring and reporting of CO2 emissions and fuel consumption of new heavy-duty vehicles, 26 September 2017.
Revision of the 'Eurovignette' directive, 26 September 2017.
Work-life balance for parents and carers, 25 September 2017.
Banking reform package, 31 August 2017.
Services in the internal market: Notification procedure for authorisation schemes and requirements related to services, 31 August 2017.
Protection of workers from exposure to carcinogens or mutagens: second proposal (CMD 2), 13 July 2017.
Empowerment of national competition authorities, 13 July 2017.
Use of energy from renewable sources, 26 June 2017.
Controls of cash entering or leaving the European Union, 21 June 2017.
Mutual recognition of freezing and confiscation orders, 20 June 2017.
Coordination of social security systems, 15 June 2017.
Preventive restructuring, second chance and efficient restructuring, insolvency and discharge procedures, 24 May 2017.
Limitations of scope for aviation activities in the EU ETS, 24 May 2017.
Establishing a multi-annual plan for small pelagic stocks in the Adriatic Sea and the fisheries exploiting those stocks, 22 May 2017.
Proportionality test before adoption of new regulation of professions, 12 May 2017.
Initial qualification, periodic training and minimum age of professional drivers of certain road vehicles, 11 May 2017.
The European services e-card, 5 May 2017.
Respect for private life and protection of personal data in electronic communications, 11 April 2017.
(Re-)Designing the internal market for electricity, 11 April 2017.
Appraising the quality of the European Commission's impact assessments
37
Restriction of the use of certain hazardous substances in electrical and electronic equipment, 4 April 2017.
Recovery and resolution of central counterparties, 22 March 2017.
Energy Efficiency, 22 March 2017.
Governance of the Energy Union, 28 February 2017.
Revision of the calculation methodology of dumping, 28 February 2017.
Energy performance of buildings, 28 February 2017.
Control of exports, transfer, brokering, technical assistance and transit of dual-use items, 25 January 2017.
The European Electronic Communications Code and the Body of European Regulators for Electronic Communication (BEREC), 23 January 2017.
Inclusion of greenhouse gas emissions and removals from land use, land use change, and forestry (LULUCF) into the 2030 climate and energy framework, 22 December 2016.
A European Union certification system for aviation security screening equipment, 21 December 2016.
Modernisation of EU copyright rules, 21 December 2016.
Effort sharing: greenhouse gas emission reductions by Member States (2021-2030), 14 December 2016.
Revision of the Instrument contributing to Stability and Peace, 7 December 2016.
European venture capital and social entrepreneurship funds, 7 December 2016.
Establishing a multiannual plan for demersal fisheries in the North Sea, 9 November 2016.
Judicial rules and cooperation in family matters (recast of Brussels IIa Regulation), 7 November 2016.
Prevention of the use of the financial system for the purposes of money laundering or terrorist financing, 3 November 2016.
The New EU Blue Card Directive, 29 September 2016.
Audiovisual Media Services, 23 September 2016.
Geo-blocking in the Digital Single Market, 21 September 2016.
Wholesale Roaming Markets, 12 September 2016.
Cooperation on the enforcement of consumer protection laws, 26 July 2016.
New rules on CE marked fertilising products, 18 July 2016.
Protection of workers from exposure to carcinogens or mutagens, 18 July 2016.
Digital Single Market: Cross-border parcel delivery services, 18 July 2016.
Publishing corporate tax information, Country-by-country reporting for multinational enterprise groups, 7 July 2016.
Fisheries technical measures, 6 June 2016.
Recognition of Professional Qualifications in Inland Navigation, 2 June 2016.
EPRS | European Parliamentary Research Service
38
Revision of the Posting of Workers Directive, 2 June 2016.
The Establishment of an EU Entry/Exit System, 23 May 2016.
Use of radio frequencies in the Union, 21 April 2016.
Intergovernmental agreements in the field of energy, 21 April 2016.
Security of gas supply, 21 April 2016.
Sustainable management of external fishing fleets, 15 April 2016.
Motor vehicles: New approval and market surveillance rules, 15 April 2016.
Aviation strategy - Remotely Piloted Aircraft Systems, 14 April 2016.
Regulation on Mercury - Aligning EU legislation with the Minimata Convention, 14 April 2016.
Accessibility requirements for products and services – The European Accessibility Act, 12 April 2016.
Cross-border portability of online content services in the internal market, 17 March 2016.
Exchange of information on Third Country Nationals – European Criminal Records Information System (ECRIS), 17 March 2016.
Action Plan on Building a Capital Markets Union: Prospectus Regulation, 25 February 2016.
Aviation strategy - European Union Aviation Safety Agency, 25 February 2016.
Contracts for the supply of digital content and for the online and other distance sales of goods, 17 February 2016.
Review of the EU waste management targets: 'Circular Economy Package', 1 February 2016.
Action Plan on Building a Capital Markets Union: EU securitisation framework, 9 November 2015.
Setting a Framework for Energy Efficiency Labelling, 17 September 2015.
ETS: Cost-effective emission reductions and low-carbon investments, 17 September 2015.
Based on a scoring exercise looking at 132 Commission impact assessments (IAs) published between July 2015 and December 2018, and their appraisal by the IMPA unit of EPRS, this study finds that the overall average quality of the IAs was satisfactory, reflecting the considerable efforts of the Commission in the framework of the better regulation agenda.
However, not all parts of the IAs were found to be of equal quality. This analysis shows that the quality of individual aspects of an IA can vary considerably, depending on a variety of factors. On average, in this review, the strongest parts of IAs were found to be problem definition, provisions for monitoring and evaluation, and coherence between the IA and the corresponding legislative proposal. The weakest areas were the presentation of policy options and the assessment of their relevant impacts, two essential sections of any IA. Moreover, this study identifies the need for consistent transparency as an important aspect of the quality and accountability of an IA. This concerns in particular the use of data and stakeholder consultations, as well as the methods and assumptions underlying an IA.
With this study, the IMPA unit of EPRS aims to contribute to the common effort to improve the quality of EU law-making, by promoting the value of IAs in support of political decision-making and helping ensure they are of the highest possible quality so as to inform decision-makers effectively.
This is a publication of the Ex-ante Impact Assessment Unit EPRS | European Parliamentary Research Service
This document is prepared for, and addressed to, the Members and staff of the European Parliament as background material to assist them in their parliamentary work. The content of
the document is the sole responsibility of its author(s) and any opinions expressed herein should not be taken to represent an official position of the Parliament.
PDF ISBN 978-92-846-6064-3 | doi:10.2861/968302 | QA-03-19-911-EN-N
QA
-03-19-911-EN-N
