33
HORIZON 2020 Evaluation Procedure One Stage Topics
HORIZON 2020
Evaluation Procedure One Stage Topics
Content
• Subject of the evaluation: call and schedule
• Types of Action under evaluation
• Evaluation Process: Criteria, Scoring, Important Definitions, Horizontal Issues
• Individual Evaluation
• How to get started IER?
• Drafting of Consensus Report
• How to get started CR?
• Consensus Meeting
• Who can support you?
• Logistics for Brussels
Call
•Climate Services
•Towards a low-carbon Europe
•Nature-based solutions for territorial resilience
•Water
•Raw Materials
•Earth Observation
•Cultural heritage for sustainable growth
•Support to policy and innovation procurement
Greening the Economy (SC5)
Evaluation Schedule
Submission Procedure
Topics Stage Deadline Info to
Applicants Grant
Agreements
One Stage
20 topics and sub-topics (RIA, CSA, ERA-NET,
PCP)
One Stage (full
proposal) 07/03/2017 June 2017 Dec 2017
Two Stage 7 topics and sub-topics
(IA)
First Stage (short
proposal) 07/03/2017 May 2017 -
Second Stage (full proposal)
05/09/2017 Dec 2017 May 2018
Type of action
Research and Innovation Actions
Coordination and Support Actions
Pre-commercial Procurement Actions
ERA-NET Cofund actions
RIA
Action primarily consisting of activities to establish new knowledge and/or explore feasibility of new or improved technology, product, process, service or solution
• May include basic and applied research, technology development and integration, testing and validation on small-scale prototype in laboratory or simulated environment
• Projects may contain closely connected but limited demonstration or pilot activities to show technical feasibility in a near to operational environment
CSA
Actions consisting primarily of accompanying measures such as
• standardisation, dissemination, awareness-raising and communication, networking, coordination or support services, policy dialogues and mutual learning exercises and studies, including design studies for new infrastructure, and
• may also include complementary activities of strategic planning, networking and coordination between programmes in different countries
ERA-NET
Supports public-public partnerships, including joint programming initiatives between Member States, in their preparation, establishment of networking structures, design, implementation and coordination of joint activities as well as EU topping-up of trans-national call for proposals
• The main activity is the implementation of the co-funded joint call for proposals that leads to the funding of transnational research and/or innovation projects. In addition, consortia may implement other joint activities e.g. joint calls without EU co-funding
• May also, depending on the research area and the underlying national programmes and their governing principles, target governmental research organisations. The co-funded call will in these cases be based on in-kind contributions from their institutional funding and the beneficiaries carry out the transnational projects resulting from their call fully or partially themselves
• The in-kind contributions are the resources allocated as direct expenditure in the selected trans-national projects that are not reimbursed by the EU contribution
PCP
Encourage public procurement of research, development and validation of new solutions that can bring significant quality and efficiency improvements in areas of public interest, while opening market opportunities for industry and researchers
• Provides EU co-funding for group of procurers to undertake together one joint PCP procurement, so that there is one joint call for tender, one joint evaluation of offers, and a lead procurer awarding the R&D service contracts in the name and on behalf of the group
• Each procurer contributes its individual financial contribution to the total budget necessary to jointly finance the PCP, enabling the procurers to share the costs of procuring R&D services from a number of providers and comparing together the merits of alternative solutions paths to address the common challenge
• The PCP shall explore alternative solution paths from a number of competing providers to address one concrete procurement need that is identified as a common challenge in the innovation plans of the procurers that requires new R&D
• Cross-border PCP cooperation should better address issues of common European interest, for example where interoperability and coherence of solutions across borders is required
Evaluation
Receipt of proposals
Individual evaluation
Consensus meeting
Finalisation
Experts
Individual Evaluation Reports (remote)
Consensus Report
(central)
EC ranked list
Information sent to applicants
Eligibility check
Allocation of proposals to evaluators
EASME EASME
Panel meeting
Cross reading
Evaluation Summary Report
Panel report
Expert ranked list
Evaluation
Individual Evaluation
Report
Individual Evaluation
Report Individual Evaluation
Report
Draft Consensus
Report
Individual Evaluation
Report
Individual Evaluation
Report
Expert Expert Expert Expert Expert Minimum 3 evaluators
Individual Evaluation Reports (IER)
Draft Consensus report (CR)
Proposal Eligible proposal
Consensus meeting (on-site)
Consensus Report
Consensus meeting
Consensus report (CR)
RAPPORTEUR
SUPPRA-RAPPORTEUR*
The specific schedule for your topic and for the different tasks will be set by your topic
moderator
Please comply with set deadlines
The dates in the contract and in SEP are general for all actors and topics
Evaluation
Evaluation
There are three evaluation criteria
• Excellence
• Impact
• Implementation
• The criteria are adapted to each type of action, as specified in the WP
You give a score of between 0 and 5 to each criterion
• The whole range of scores should be used
• Half-marks can be used
• Scores must pass thresholds if a proposal is to be considered for funding
Thresholds
Excellence 3
Impact 3
Implementation 3
Overall 10
Proposal Scoring
The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.
Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.
Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.
Good. The proposal addresses the criterion well, but a number of shortcomings are present.
Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.
Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.
0
1
2
3
4
5
Criteria - Excellence
Excellence - RIA
• Clarity and pertinence of the objectives;
• Soundness of the concept and credibility of the proposed methodology;
• Extent that the proposed work is beyond the state of the art, and demonstrates innovation potential (e.g. ground-breaking objectives, novel concepts and approaches, new products, services or business and organisational models)
• Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge.
Excellence - CSA
• Clarity and pertinence of the objectives;
• Soundness of the concept and credibility of the proposed methodology;
• Quality of the proposed coordination and/or support measures
Excellence - PCP
• Clarity and pertinence of the objectives;
• Soundness of the concept and credibility of the proposed methodology;
• Progress beyond the state of the art in terms of the degree of innovation needed to satisfy the procurement need
Excellence – ERA-NET
• Clarity and pertinence of the objectives;
• Soundness of the concept and credibility of the proposed methodology;
• Level of ambition in the collaboration and commitment of the participants in the proposed ERA-NET action to pool national resources and coordinate their national/regional research programmes
Co
-desig
n &
Co
-creati
on
Responsible Research and Innovation
Societal actors work together to align research and results with the values, needs and expectations of society.
Public engagement
Iterative/participatory multi-actor dialogues to co-create research and innovation outcomes and policy agendas.
Trans-disciplinarity
Methodologies that integrate scientific disciplines, and non-academic and non-formalized knowledge.
The term 'innovation' is used in the EU policy context and more widely to mean the introduction in the market of new or improved products, services, processes and solutions.
• Innovation potential is the potential of a project to create useful novelties beyond what already existing (RIA)
Evaluation Criteria Important Definitions
Criteria - Impact
Impact - RIA
• The extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic;
• Any substantial impacts not mentioned in the work programme, that would enhance innovation capacity, create new market opportunities, strengthen competitiveness and growth of companies, address issues related to climate change or the environment, or bring other important benefits for society;
• Quality of the proposed measures to:
- Exploit and disseminate the project results (including management of IPR), and to manage research data where relevant.
- Communicate the project activities to different target audiences.
Impact - CSA
• The extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic;
• Quality of the proposed measures to:
- Exploit and disseminate the project results (including management of IPR), and to manage research data where relevant.
- Communicate the project activities to different target audiences.
Criteria - Impact
Impact - PCP
• The extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic;
• Strengthening the competitiveness and growth of companies by developing innovations meeting needs of European and global procurement markets
• Quality of the proposed measures to:
- Exploit and disseminate the project results (including management of IPR), and to manage research data where relevant.
- Communicate the project activities to different target audiences.
• More forward looking procurement approaches reducing fragmentation of demand for innovative solutions
Impact – ERA-NET
• The extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic;
• Achievement of critical mass for the funding of trans-national projects by pooling of national/regional and contribution to establishing and strengthening a durable cooperation between the partners and their national/regional research programmes
• Quality of the proposed measures to:
- Exploit and disseminate the project results (including management of IPR), and to manage research data where relevant.
- Communicate the project activities to different target audiences.
• Innovation capacity is the capacity of the new or improved novelties to create impact beyond their original purpose i.e. the capacity of inspiring or having a knock on effect in another domains / sectors (RIA)
• Intellectual Property is the knowledge / results created by the project. (RIA, CSA, ERA-NET, PCP)
• Intellectual Property Rights are the legal rights granting preferential use. For example, Patents, Copyrights, Trade marks, Design rights, Database rights, Plant Breeders rights, Utility models / petty patents, Confidentiality agreements, Trade secrets, etc. (RIA, CSA, ERA-NET, PCP)
• Intellectual Property Management is the process of monitoring, assessing, protecting, disseminating, exploiting (i.e. making usable elsewhere) the knowledge created. (RIA, CSA, ERA-NET, PCP)
Evaluation Criteria Important Definitions
• Exploitation is the use of the results during and after the project’s implementation. It can be for commercial purposes but also for improving policies, and for tackling economic and societal problems. (RIA, CSA, ERA-NET, PCP)
• Dissemination is making the project outputs available to various stakeholder groups (like research peers, industry and other commercial actors, professional organisations, policymakers) in a targeted way, to enable them to use the results in their own work. (RIA, CSA, ERA-NET, PCP)
• Communication is the process of promoting the action and its results via strategic and targeted measures to a multitude of audiences, including the media and the public, and possibly engaging in a two-way exchange. The aim is to reach out to society as a whole and in particular to some specific audiences while demonstrating how EU funding contributes to tackling societal challenges. (RIA, CSA, ERA-NET, PCP)
Evaluation Criteria Important Definitions
Criteria - Implementation
• Quality and effectiveness of the work plan, including extent to which the resources assigned to work packages are in line with their objectives and deliverables;
• Appropriateness of the management structures and procedures, including risk and innovation management;
• Complementarity of the participants and extent to which the consortium as whole brings together the necessary expertise;
• Appropriateness of the allocation of tasks, ensuring that all participants have a valid role and adequate resources in the project to fulfil that role.
Implementation
• Innovation management is the management of the process to create innovation (from idea to market introduction). (RIA, CSA, ERA-NET, PCP)
Horizontal Issues
Topics flagged for Social Sciences and Humanities (SSH)
• SC5-06-2016-2017
• SC5-15-2016-2017
• SC5-19-2017
• SC5-22-2017
• SC5-30-2017
• SC5-32-2017
• SC5-33-2017
Topics where International Cooperation is required
• SC5-16-2016-2017
Topics where International Cooperation is encouraged
• SC5-02-2016-2017
• SC5-06-2016-2017
• SC5-13-2016-2017
• SC5-15-2016-2017
• SC5-30-2017
• SC5-31-2017
• SC5-32-2017
• SC5-33-2017
Individual Evaluation
• You read the proposal and evaluate it against the evaluation criteria
• Without discussing it with anybody else
• As submitted - not on its potential if certain changes were to be made
• You disregard excess pages in section 1-3 of Part B (RIA – 70 pages, CSA and ERA-NET – 50 pages, PCP – 90 pages)
• Moreover, you do not consider information included in parts originally foreseen for other purposes (e.g. section 4-5 of Part B, Ethics Annexes, etc.)
• In case of doubt please ask your topic moderator
• You look at the substance: Some proposals might be face language difficulties, others are deceptively well written
• You explain shortcomings, but do not make recommendations
Individual Evaluation
If a proposal:
• Is only marginally relevant in terms of its scientific, technological or innovation content relating to the topic addressed, you must reflect this in a lower score for the Excellence criterion
• No matter how excellent the science!
• Does not significantly contribute to the expected impacts as specified in the WP for that call or topic, you must reflect this in a lower score for the Impact criterion
• Would require substantial modifications in terms of implementation (i.e. change of partners, additional work packages, significant budget or resources cut…), you must reflect this in a lower score for the Quality and efficiency of the Implementation criterion
• If cross-cutting issues are explicitly mentioned in the scope of the call or topic, and not properly addressed (or their non-relevance justified), you must reflect this in a lower score for the relevant criterion
Additional Questions
Out-of-Scope
• If you consider a proposal to not be in scope in regards to the topic description please signal it in the appropriate box and provide a justification of why you believe it is out-of-scope.
Operational capacity
• If you identify any partner that seems inadequate or fraudulent partners on basis of the info provided please signal it in the appropriate box. Your individual evaluation should progress normally.
Exceptional funding of third country participants / international organisations
• If you identify any partner that is not eligible for funding please signal it in the appropriate box, as well as your justification of why or why not the partner should be funded.
• Exceptional funding needs to be justified on the basis of: Outstanding competence / expertise; Access to particular geographical environments or research infrastructure or data.
How to get started - IER?
1. Go to "SEP", the online system for proposal evaluation
2. Study the briefing material sent to you by the topic moderator and if being
organised attend the topic web-briefing
3. Screen your proposals - to check potential Conflict of Interest and have
an overview of all proposals
4. Indicate to your topic moderator any potential CoI
5. Evaluate a first proposal and save a draft report in SEP. You can ask your
topic moderator to provide you feedback
How to get started - IER?
6. Read each proposal in detail. Go through each criterion and write your
comments for every aspect mentioned under the criterion. Score!
7. Complete your IER. The quality of your IER is directly proportional to the
quality of the final Consensus Report
8. After completing each IER – check that your scores match your
comments. Use the full scale of scores!
9. Review your facts – when you cite facts, are they truly in the proposal and
will you be able to find them back during the Consensus Meetings?
10.After having completed all IERs – calibrate your IER (have you bveen fair
with every proposal?) and submit the reports by the deadline indicated to
you.
Preparation of draft CR
• The rapporteur is responsible for drafting the CR
• In some cases, the rapporteur is not one of the evaluators
• The quality of the CR is of utmost importance, the aim is to give:
• A clear assessment of the proposal based on its merit, with justification
• Clear feedback on the proposal’s weaknesses and strengths, of an adequate length, and in an appropriate tone
• The goal of this task is to facilitate the work during the consensus meeting as the CR will be a compromise report that balances the views of the different evaluators
How to get started - CR?
1. Study the briefing material sent to you by the topic moderator (even if you
are not an evaluator) and attend the topic web-briefing (if organised)
2. Go to "SEP", the online system for proposal evaluation
3. If not an evaluator screen your proposals - to check potential Conflict of
Interest and indicate to your topic moderator any potential CoI
4. Read all the Individual Evaluation Reports and identify points of
convergence and divergence. You can use the functions "Merge" and
"Initialise" of SEP to help you in your work.
How to get started - CR?
6. Propose a consensus wording for the convergent points highlighting
strengths and weaknesses; and identify the divergences as points for
discussion with pros and cons to guide the consensus meeting
7. If needed, refer to the proposal for checking factual references
8. Save the report in SEP without scores within the set deadline. You can
ask your topic moderator to provide you feedback
9. During the consensus meeting review and update your draft CR to reflect
the outcome of the discussion
10. If you did not evaluate the proposal do not express or impose your own
view
Consensus meeting
• It involves a discussion on the basis of the individual evaluations
• The aim is to find agreement on comments and scores and finalise the CR
• Keep in mind that is normal for individual views to change after arguments are exchanged and that “outlying” opinions need to be explored as they might be as valid as others
• Agree on comments before scoring!
• Moderated by Agency staff
• Manages the evaluation, protects confidentiality and ensures fairness
• Ensures objectivity and accuracy, all voices heard and points discussed
• Helps the group keep to time and reach consensus
Who can support you?
• Your first point of contact is your topic moderator:
• For all clarifications on your assignement
• Backup arrangements: your Panel Moderator will communicate to you any backup arrangements needed during holiday times
• For SEP issues - please contact the IT helpdesk:
• Phone: +32 2 29 92222
• Available on weekdays 9:00 -18:00 CET (Friday 17:00)
• For Expert Payment issues - please contact directly:
Logistics
• Your central evaluation week will take place in the dates indicated by your topic moderator
• Work usually starts at 9h00 and finishes at 17h30
• Please do not arrange any other travel plan without explicit consent of your topic moderator
• SEP – the electronic system for the evaluation of proposals is available and accessible via your ECAS password
• Please make sure you know your ECAS login and password
• Please bring your own laptop/tablet/notebook
• Reduction of paper copies
• A few printers are available in the evaluation building in Brussels
• Please bring your own paper copies of material you need to support your work
