24
HORIZON 2020 Evaluation Procedure Two Stage topics
HORIZON 2020
Evaluation Procedure Two Stage topics
Content
• Subject of the evaluation: call and schedule
• Types of Action under evaluation • Evaluation Process: Criteria, Scoring, Important Definitions, Horizontal
Issues
• Individual Evaluation
• How to get started IER?
• Drafting of Consensus Report
• How to get started CR?
• Consensus
• Who can support you?
Calls
•Nature-based solutions for territorial resilience
•Raw Materials
•Cultural heritage for sustainable growth
Greening the Economy (SC5)
•Circular Economy Industry 2020 in the
Circular Economy (CIRC)
•Sustainable cities through nature-based solutions
Smart and Sustainable Cities
(SCC)
Evaluation Schedule
Submission Procedure
Topics Stage Deadline Info to
Applicants Grant
Agreements
One Stage
20 topics and sub-topics (RIA, CSA, ERA-NET,
PCP)
One Stage (full proposal)
07/03/2017 June 2017 Dec 2017
Two Stage 7 topics and sub-topics
(IA)
First Stage (short
proposal) 07/03/2017 May 2017 -
Second Stage (full proposal)
05/09/2017 Dec 2017 May 2018
Innovation Actions
IA
Action primarily consisting of activities that produce plans and arrangements or designs for new, altered or improved products, processes or services
• May include prototyping, testing, demonstrating, piloting, large-scale product validation and market replication
• Aim to validate the technical and economic viability in a (near) operational environment and/or support the first application/deployment in the market of an innovation that has already been demonstrated but not yet applied/deployed in the market due to market failures/barriers to uptake
• Projects may include limited research and development activities
Evaluation
Receipt of proposals
Individual evaluation
Consensus evaluation
Finalisation
Experts
Individual Evaluation Reports (remote)
Consensus Report
(remote)
Information sent to applicants
Eligibility check
Allocation of proposals to evaluators
EASME EASME
Evaluation
Individual Evaluation
Report
Individual Evaluation
Report Individual Evaluation
Report
Draft Consensus
Report
Individual Evaluation
Report
Individual Evaluation
Report
Expert Expert Expert Expert Expert Minimum 3 evaluators
Individual Evaluation Reports (IER)
Draft Consensus report (CR)
Proposal Eligible proposal
Consensus group
(remote)
Consensus Report
Consensus discussion
Consensus report (CR)
RAPPORTEUR
The specific schedule for your topic and for the different tasks will be set by your topic
moderator
Please comply with set deadlines
The dates in the contract and in SEP are general for all actors and topics
Evaluation
There are two evaluation criteria
• Excellence
• Impact
You give a score of between 0 and 5 to each criterion
• The whole range of scores should be used
• Half-marks can be used
• Scores must pass thresholds if a proposal is to be considered for funding
Thresholds
Excellence 4
Impact 4
Overall Dynamic
The total requested budget of proposals admitted to stage 2 is as close as possible to three
times the available budget and in any case not less than
two and a half times the available budget.
Proposal Scoring
The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.
Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.
Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.
Good. The proposal addresses the criterion well, but a number of shortcomings are present.
Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.
Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.
0
1
2
3
4
5
Excellence Impact
• Clarity and pertinence of the objectives;
• Soundness of the concept and credibility of the proposed methodology;
• Extent that the proposed work is beyond the state of the art, and demonstrates innovation potential (e.g. ground-breaking objectives, novel concepts and approaches, new products, services or business and organisational models)
• Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge.
• Extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic;
Evaluation Criteria
Co
-desig
n &
Co
-creati
on
Responsible Research and Innovation
Societal actors work together to align research and results with the values, needs and expectations of society.
Public engagement
Iterative/participatory multi-actor dialogues to co-create research and innovation outcomes and policy agendas.
Trans-disciplinarity
Methodologies that integrate scientific disciplines, and non-academic and non-formalized knowledge.
The term 'innovation' is used in the EU policy context and more widely to mean the introduction in the market of new or improved products, services, processes and solutions.
• Innovation potential is the potential of a project to create useful novelties beyond what already existing
Evaluation Criteria Important Definitions
Horizontal Issues
Topics flagged for Social Sciences and Humanities (SSH)
• CIRC-01-2016-2017
• SC5-08-2017
• SC5-14-2016-2017
• SC5-21-2016-2017
• SCC-02-2016-2017
Topics flagged for Gender
• CIRC-01-2016-2017
• SCC-02-2016-2017
Topics where International Cooperation is encouraged
• SC5-08-2017
• SC5-14-2016-2017
• SC5-21-2016-2017
• SCC-02-2016-2017
Individual Evaluation
• You read the proposal and evaluate it against the evaluation criteria
• Without discussing it with anybody else
• As submitted - not on its potential if certain changes were to be made
• You disregard excess pages in section 1-3 of Part B (IA – 10 pages)
• In case of doubt please ask your topic moderator
• You look at the substance: Some proposals might be face language difficulties, others are deceptively well written
• You explain shortcomings, but do not make recommendations
Individual Evaluation
Out-of-Scope
• If you consider a proposal to not be in scope in regards to the topic description please signal it in the appropriate box and provide a justification of why you believe it is out-of-scope.
If a proposal:
• is only marginally relevant in terms of its scientific, technological or innovation content relating to the topic addressed, you must reflect this in a lower score for the Excellence criterion
• No matter how excellent the science!
• does not significantly contribute to the expected impacts as specified in the WP for that call or topic, you must reflect this in a lower score for the Impact criterion
How to get started - IER?
1. Go to "SEP", the online system for proposal evaluation
2. Study the briefing material sent to you by the topic moderator and if being
organised attend the topic web-briefing
3. Screen your proposals - to check potential Conflict of Interest and have
an overview of all proposals
4. Indicate to your topic moderator any potential CoI
5. Evaluate a first proposal and save a draft report in SEP. You can ask your
topic moderator to provide you feedback
How to get started - IER?
6. Read each proposal in detail. Go through each criterion and write your
comments for every aspect mentioned under the criterion. Score!
7. Complete your IER. The quality of your IER is directly proportional to the
quality of the final Consensus Report
8. After completing each IER – check that your scores match your
comments. Use the full scale of scores!
9. Review your facts – when you cite facts, are they truly in the proposal and
will you be able to find them back during the Consensus discussions?
10.After having completed all IERs – calibrate yourself (have you been fair
with every proposal?) and submit the reports by the deadline indicated to
you.
Preparation of draft CR
• The rapporteur is responsible for drafting the CR
• In some cases, the rapporteur is not one of the evaluators
• The quality of the CR is of utmost importance, the aim is to give:
• A clear assessment of the proposal based on its merit, with justification
• Clear feedback on the proposal’s weaknesses and strengths, of an adequate length, and in an appropriate tone
• The goal of this task is to facilitate the work during the consensus meeting as the CR will be a compromise report that balances the views of the different evaluators
• The CR should be comprehensive and tackle all the sub-criteria of the evaluation form.
How to get started - CR?
1. Study the briefing material sent to you by the topic moderator (even if you
are not an evaluator) and attend the topic web-briefing (if organised)
2. Go to "SEP", the online system for proposal evaluation
3. If not an evaluator screen your proposals - to check potential Conflict of
Interests and indicate to your topic moderator any potential CoI
4. Read all the Individual Evaluation Reports and identify points of
convergence and divergence. You can use the functions "Merge" and
"Initialise" of SEP to help you in your work.
How to get started - CR?
6. Propose a consensus wording for the convergent points highlighting
strengths and weaknesses; and identify the divergences as points for
discussion or clarification with other experts.
7. If needed, refer to the proposal for checking factual references
8. Save the report in SEP without scores within the set deadline. You can
ask your topic moderator to provide you feedback
9. If you did not evaluate the proposal do not express or impose your own view
Consensus
• It involves a discussion on the basis of the draft consensus report and the individual evaluations
• The aim is to find agreement on comments and scores and finalise the CR
• Keep in mind that is normal for individual views to change after arguments are exchanged and that “outlying” opinions need to be explored as they might be as valid as others
• Use the comment box to discuss when needed (see slides on SEP)
• The rapporteur will suggest a set of scores based on the individual evaluation scores and the draft CR.
• Dot nor forget scores match comments, and not the other way around.
• Moderated by Agency staff
• Manages the evaluation, protects confidentiality and ensures fairness
• Helps the group keep within deadlines and reach consensus
Consensus
• The rapporteur will update the draft CR to reflect the discussion that took place
• Several versions might be needed until the CR is fully approved
• Your moderator might convene a phone discussion to finalise more difficult cases
• The CR is finalised when all experts approve it in the system
Keep in mind that the proposals are evaluated in the context of what is requested in the topic text
Who can support you?
• Your first point of contact is your topic moderator:
• For all clarifications on your assignement
• Backup arrangements: your Panel Moderator will communicate to you any backup arrangements needed during holiday times
• For SEP issues - please contact the IT helpdesk:
• Phone: +32 2 29 92222
• Available on weekdays 9:00 -18:00 CET (Friday 17:00)
• For Expert Payment issues - please contact directly:
