1

Science with and for Society (SwafS) 2019

Evaluation

H2020-SwafS-2019-1

Manual for experts

2

Horizon 2020 – Science with and for Society

Table of Contents Introduction ..................................................................................................................................................................................3

Definitions/abbreviations ....................................................................................................................................................3

1. Science with and for Society call ...............................................................................................................................................4

1.1 General concepts ...............................................................................................................................................................4

1.1.1 SwafS ‘topics’ ..............................................................................................................................................................4 1.1.2 ‘Types of action’ .........................................................................................................................................................4

1.2 Cross-cutting issues ............................................................................................................................................................5

1.2.1 Responsible Research and Innovation (RRI) ...............................................................................................................5 1.2.2 Gender dimension ......................................................................................................................................................5 1.2.3 Open Science ..............................................................................................................................................................6 1.2.4 Social Sciences and Humanities ..................................................................................................................................7 1.2.5 International cooperation ..........................................................................................................................................8

2. Actors in the process .................................................................................................................................................................8

2.1 Guiding principles ...............................................................................................................................................................9 2.2 Conflict of Interest (CoI) .....................................................................................................................................................9 2.3 Confidentiality ..................................................................................................................................................................10 2.4 Implicit gender biases ......................................................................................................................................................11

3. The evaluation procedure .......................................................................................................................................................12

3.1 The Evaluation workflow ..................................................................................................................................................12 3.2 The ‘remote’ phase ..........................................................................................................................................................13

3.2.1 Admissibility and Eligibility Check.............................................................................................................................13 3.2.2 Individual evaluation ................................................................................................................................................13

3.3 The central evaluation (‘central’ phase) ...........................................................................................................................13

3.3.1 Consensus meetings (CM) ........................................................................................................................................13 3.3.2 Panel review and finalisation ...................................................................................................................................14

4. Your IER ...................................................................................................................................................................................15

4.1 Proposals ..........................................................................................................................................................................15 4.2 Evaluation criteria and sub-criteria ..................................................................................................................................15 4.4 Additional elements to assess ..........................................................................................................................................16

4.4.1 Scope ........................................................................................................................................................................16 4.4.2 Operational capacity ................................................................................................................................................17 4.4.3 Exceptional funding of third country participants/international organisations .......................................................17

4.5 Tips for drafting a good IER ..............................................................................................................................................18

5. Other relevant aspects ............................................................................................................................................................19

5.1 Open access and research data ........................................................................................................................................19 5.2 Financial Support to third parties .....................................................................................................................................20 5.3 Additional dissemination obligations ...............................................................................................................................21 5.4 Ethics review ....................................................................................................................................................................21

6. Planning, references and contacts ..........................................................................................................................................22

6.1 Planning............................................................................................................................................................................22 6.2 References........................................................................................................................................................................22 6.3 Contacts ...........................................................................................................................................................................22

3

Introduction This document is focused on the overall evaluation process and the role of expert-evaluators. The

evaluation is performed in two phases: ‘remote’ and ‘central’ evaluation.

Before commencing the remote evaluation, it is important to view/read the full package of briefing

material for experts which consists of a general overview + topic-specific section consisting of 3

presentations + an assessment grid. For each presentation, you will find a video briefing & the

corresponding power point slides

The Work Programme topic description & topic briefing slides & topic assessment grid are the key

references.

A dedicated briefing focusing on the ‘central’ evaluation will take place

in Brussels on 17 June 2019

This manual starts with a general overview of SwafS and the terminology used. It details the role,

responsibilities and tasks of the experts and guiding principles on how you should perform your work.

The evaluation workflow is outlined and the 'remote' and 'central' phases are detailed as well as the

evaluation criteria (and sub-criteria). Finally, other relevant aspects to be considered are presented

before concluding with guidelines on how to draft your reports.

The information provided in this document about the evaluation process (incl. criteria and scoring

principles), does not replace the official Horizon 2020 documents which are referred to in this manual.

Definitions/abbreviations

• CC => Call Coordinator

• CM => Consensus Meeting (i.e. Consensus Group phase)

• CoI => Conflict of Interest

• CR => Consensus report

• ESR => Evaluation Summary Report

• H2020 => Horizon 2020 Framework Programme

• IER => Individual Evaluation Report

• PO => Project Officer

• SEP => Submission and Evaluation of Proposal IT System

• SwafS => Science with and for Society programme

• REA => Research Executive Agency

• VC => Vice-chair, an external expert who helps the Agency/Commission staff in performing

tasks during the evaluation process. Usually he/she has the same role as a PO

• WP => Work Programme

4

1. Science with and for Society call

1.1 General concepts

1.1.1 SwafS ‘topics’

Each SwafS topic is outlined in the Work Programme and follows a standard structure:

1. Topic identifier and title (e.g. Swafs-08-2019: Research innovation needs & skills training in PhD

programmes);

2. Specific Challenge: the problem that needs to be addressed;

3. Scope: the framework within which a solution needs to be found, including an indication of the

funding size;

4. Expected Impact: if the project would achieve its objectives: which impacts should it have?

5. Type of action: (see below, point 1.1.2)

Topics for the SwafS 2019 Call can be found in the Science with and for Society (WP) 2018-2020

1.1.2 ‘Types of action’

In SwafS, there are two ‘types of action’:

Coordination & Support Actions (CSAs) – actions consisting primarily of accompanying measures

(i.e. not focused on research) such as: - standardisation, dissemination, awareness-raising and communication, networking,

coordination and support services, policy dialogues and mutual learning exercises and studies,

including design studies for new infrastructure; and

- may also include complementary activities of strategic planning, networking and coordination

between programmes in different countries.

Research & Innovation Actions (RIAs) – actions primarily consisting of activities to establish new

knowledge and/or explore feasibility of new or improved technology, product, process, service or

solution:

- may include basic and applied research, technology development and integration, testing and

validation on small-scale prototype in laboratory or simulated environment;

- projects may contain closely connected but limited demonstration or pilot activities to show

technical feasibility in a near to operational environment.

CSA in SwafS 2019 => Topics Swafs-05, SwafS-07, SwafS-08, SwafS-09, SwafS-14 and SwafS-16

RIA in SwafS 2019 => Topics Swafs-11, SwafS-12, SwafS-15, SwafS-17 and SwafS-19

For more information about H2020 types of action,

please consult the General Annexes (section D and H) of the Horizon 2020 WP.

Important: Evaluation criteria (and proposal templates) differ for CSA and RIA proposals

5

1.2 Cross-cutting issues

A number of Horizon 2020 cross-cutting issues are integrated in the SwafS WP: RRI (Responsible

Research and Innovation), Gender dimension, Open science, SSH (Social Sciences and Humanities) and

International Co-operation. If a topic is flagged for a cross-cutting issue, it should be considered when

evaluating the relevant sub-criterion. Proposals addressing cross-cutting issues which are not explicitly

mentioned in the scope of the topic can also be evaluated positively.

For more information about how to address cross-cutting priorities, see annex of the topic briefing

slides where FAQs outline how each should be addressed in the evaluation.

1.2.1 Responsible Research and Innovation (RRI)

Responsible Research & Innovation (RRI) is cutting across Horizon 2020, engaging society, integrating

the gender and ethical dimensions, ensuring access to research outcomes and encouraging formal and

informal science education.

RRI implies that societal actors (researchers, citizens, policy makers, business civil society

organisations, etc.) work together during the whole research and innovation process to better align

both the process and its outcomes with the values, needs and expectations of society.

Figure 1: RRI dimensions

Public engagement (PE) is one of the RRI dimensions. It is about co-creating the future with citizens

and civil society organisations and bringing on board the widest possible diversity of actors on matters

of science and technology.

RRI is relevant for all SwafS-2019 topics.

The topic text should provide an indication of why the topic has been "flagged" for RRI and/or

whether certain dimensions of RRI need to be considered. Assess how RRI is taken into account under

the "excellence", sub-criterion 'concept and methodology' in terms of the involvement of societal

actors relevant to the topic in specific activities or in the overall approach in one or more of the five

dimensions of RRI (depending on the topic text).

For other topics, applicants can still address it in their proposal. In these cases, evaluators will deal

with RRI like any other relevant aspects of the proposal.

1.2.2 Gender dimension

A topic is considered gender relevant when it can be expected that its findings affect women and men

differently. Topics flagged for 'gender' indicate under the scope and/or impact section in what way

gender is relevant.

6

The proposal template, under 'concept and methodology', asks applicants to "where relevant,

describe how sex and/or gender analysis is taken into account in the project's content". Applicants

should integrate a gender/sex analysis, and when relevant specific studies, in their proposals.

Gender dimension is relevant for all SwafS-2019 topics.

You should check how sex and/or gender analysis is taken into account and consider this under the

"excellence" criteria.

When the gender dimension is not explicitly integrated into a topic, applicants can still address it in

their proposal. In these cases, evaluators will deal with gender like any other relevant aspects of the

proposal.

1.2.3 Open Science

'Open science' describes the on-going evolution in the modus operandi of doing research and

organising science. These changes in the dynamics of science and research are enabled by digital

technologies and driven by the globalisation of the scientific community. They have an impact on the

way research is produced, accessed and utilised.

A key component of open science is open access, which is the practice of providing on-line access to

scientific information that is free of charge to the reader. This comprises open access to peer-

reviewed publications, which is mandatory for all Horizon 2020 peer-reviewed publications, and open

access to the research data underlying publications for projects under the Open Research Data (ORD)

pilot.

On open access to peer-reviewed publications stemming from H2020 actions, all will need to be

deposited in a suitable repository and be openly accessible within 6 months of publication disciplines

(with the exception of up to 12 months for Social Sciences and Humanities).

On open access to the research data underlying publications, while participation in the ORD pilot is

not evaluated, and regardless of this choice, applicants should address in the impact section of the

proposal how they will manage research data collected and/or generated during the project's

implementation (research data management), including access to research data (see section 5.1 of

this manual for more details).

The European Commission recognizes that some research data cannot be made open and applies the

principle 'as open as possible, as closed as necessary'. It is therefore possible, while partaking in the

ORD, to restrict access to parts of the data or even, if appropriate, opt out of the ORD at any stage by

providing a justification e.g. for intellectual property rights (IPR) concerns, privacy/data protection

concerns, national security concern, if it would run against the main objective of the project or for

other legitimate reasons. Projects in the ORD are expected to provide a data management plan as a

deliverable.

Open access to publications as well as research data management are assessed by evaluators as part

of the 'impact' evaluation criterion in relation to exploitation and dissemination regardless of whether

or not the proposal has been flagged for 'open science'.

Open science however extends beyond open access to an important dimension known as 'citizen

science'. Citizen science covers a range of different levels of participation: from raising public

knowledge about science, encouraging citizens to participate in the scientific process by observing,

gathering and processing data, right up to setting scientific agenda and co-designing and

7

implementing science-related policies. Addressing citizen science could mean citizens ‘doing

science’, for example, through various science education and outreach activities. However, no

definition is provided as these are invariably contested and could close off innovative approaches.

If a topic is flagged for open science, applicants need to go beyond open access and consider 'citizen

science' activities in their proposal. The evaluators will assess how citizen science activities are

reflected in the proposal under the 'impact' criterion.

When a topic is not flagged for open science, applicants will in any case need to address open access

and, can still decide to address citizen science in their proposal if they find it relevant and this will be

assessed like any other relevant aspects of the proposal.

Open science is relevant in particular for:

Swafs-08, Swafs-15, Swafs-16 and SwafS-17

1.2.4 Social Sciences and Humanities

Social Sciences and Humanities (SSH) are needed to tackle many of the complex societal challenges addressed in H2020, and contributions from one or more of these disciplines are frequently necessary for a successful proposal. These contributions are usually part of an interdisciplinary approach, involving either:

collaboration between SSH disciplines and/or

collaboration between SSH disciplines and non-SSH disciplines such as natural sciences, medicine and technology.

SSH disciplines include sociology, psychology, economics, law, political science, public and business

administration, demography, anthropology (except physical anthropology), geography (except

physical geography), peace and conflict studies, human rights, education science, journalism and

communication, cultural studies, religion, linguistics, literature, cultural studies, history, archaeology,

philosophy, ethics, arts and crafts (list adapted from the UNESCO International Standard Classification

of Education, ISCED 2011).

Topics where SSH contributions are required are flagged as SSH-relevant topics which means that

these topics would particularly benefit from a SSH dimension.

Proposals under these topics must take account of the social, economic, political, legal, behavioural,

institutional, historical or cultural dimensions of a given issue, as appropriate for the topic.

In these cases, the evaluators will assess how the SSH contributions are reflected in the proposal

under the 'excellence' criterion notably whether there is an appropriate consideration of an

interdisciplinary approach (i.e. collaboration between SSH disciplines and/or between SSH and non-

SSH disciplines) in relation to the topic description and/or the involvement of SSH partners under the

'implementation' criterion when evaluating the complementarity of participants.

When a topic is not flagged for SSH, applicants can still decide to address it in their proposal if they

find it relevant. In these cases, evaluators will deal with the proposed SSH issues like any other

relevant aspects of the proposal.

SSH is relevant in particular for:

Swafs-17 and Swafs-19

8

1.2.5 International cooperation

The strategic approach to international cooperation consists of a general opening of the WP and

targeted activities across all relevant Horizon 2020 parts.

For some topics, it has been clearly identified that there is a clear interest and benefit in engaging in

international cooperation. In those topics, international cooperation is encouraged and this will be

positively considered during the evaluation. However, the rules for funding third country participants

remain valid (see section 4.4.3).

International cooperation is relevant in particular for

SwafS-12, SwafS-15, SwafS-16 and SwafS-17.

2. Actors in the process

REA Actors:

- Project Officer (PO): lead for topic(s) and some tasks (quality control and moderator) shared

with Vice-chair (external expert);

- Call Coordinator: responsible for the overall evaluation exercise;

- Call Assistant: provides support in terms of the call organisation and logistics.

Independent experts:

- Expert-evaluators: evaluate proposals, submit Individual Evaluation Report (task in SEP =>

Write IER), participate in consensus meetings in Brussels, approve the final version of the CR (task

in SEP => Approve CR) and may participate in the panel review meeting;

- Rapporteurs: do not evaluate proposals, cannot give opinion on proposals, ensure a high

quality drafting of the consensus report (CR), prepare a draft of the CR in advance of consensus

meeting (CM) (task in SEP => Write CR - "save"), write the final CR after CM (task in SEP => Write

CR - "submit" and Approve CR);

- Vice-chairs: do not evaluate proposals, cannot give opinion on proposals, perform the quality

check of IERs and CRs, moderate consensus meetings, participate in panel review meetings, and

may support REA POs in monitoring evaluation progress;

- Observer: does not evaluate proposals, cannot give opinion on proposals, observe the

functioning of the overall process and procedures to ensure a high quality evaluation, advises on

the conduct/fairness of the evaluation and suggest possible improvements (Output => Observer

report).

SEP1 user guide can be found at: https://ec.europa.eu/research/participants/data/support/expert/expert_evaluation_user_manual.pdf

1 SEP (Submission and Evaluation of Proposals) is an online IT tool used to evaluate proposals, where you can write and submit Individual Evaluation Reports (IERs) as well as Consensus Reports (CRs). For technical / IT issues, please contact the SEP helpdesk via e-mail (DIGIT-EFP7-SEP-SUPPORT@ec.europa.eu) or phone +32 2 29 92222, weekdays between 8:00 and 20:00 (Brussels time).

9

2.1 Guiding principles

During the evaluation, experts should respect the following guiding principles:

Independence: you are evaluating in a personal capacity; you represent neither your employer,

nor your country;

Impartiality: you treat all proposals equally and evaluate them impartially on their merits,

irrespectively of their origin or the identity of the applicants;

Objectivity: you evaluate each proposal as submitted; meaning on its own merit, not its potential

if certain changes were to be made;

Accuracy: you make your judgment against the official evaluation criteria and the call or topic the

proposal addresses, and nothing else (you evaluate a proposal, not a call topic);

Consistency: you apply the same standard of judgment to all proposals.

You should perform your tasks and work independently, in a personal capacity, and not on behalf of

any organisation.

a) As expert-evaluator, you must:

i. evaluate each proposal in a confidential and fair way, in accordance with the Horizon

2020 Rules for Participation Regulation No 1290/20132;

ii. perform the work to the best of your abilities, professional skills, knowledge and

applying the highest ethical and moral standards;

iii. follow the instructions and time-schedule given by REA staff;

b) you should not delegate the work to another person or be replaced by another person; c) if a person or entity involved in a proposal approaches you before, or during the evaluation,

you must immediately inform the REA; d) you should not be (or become) involved in any of the actions resulting from the proposal(s)

that you evaluated (at any stage of the procedure).

2.2 Conflict of Interest (CoI)

As expert-evaluator, you must perform your work impartially and take all measures to prevent any situation where the impartial and objective implementation of the work is compromised for reasons involving economic interest, political or national affinity, family or emotional ties or any other shared interest (‘conflict of interests’). The following situations will automatically be considered as conflict of interest:

a) for a proposal you are requested to evaluate:

i. if you are involved in the preparation of the proposal;

ii. if you are a director, trustee or partner or in any way involved in the management of

an applicant (or linked third party or other third party involved in the action);

iii. if you are employed or contracted by one of the applicants (or linked third parties,

named subcontractors or other third parties involved in the action)

b) for a proposal you are requested to evaluate AND for all proposals competing for the same call budget (i.e. same topic in the case of SwafS), if you:

i. were involved in the preparation of any proposal submitted to the same topic;

ii. would benefit if any proposal submitted to the same topic is accepted or rejected;

2 Regulation (EU) No 1290/2013 of 11 December 2013 laying down the rules for the participation and dissemination in “Horizon 2020 - the

Framework Programme for Research and Innovation (2014-2020)” (OJ L 347, 20.12.2013, p. 81).

10

iii. have close family ties (spouse, domestic or non-domestic partner, child, sibling,

parent etc.) or other close personal relationship with a person (including linked third

parties or other third parties) involved in the preparation of any proposal submitted

to the same topic; or with a person who would benefit if such a proposal is accepted

or rejected.

c) for ALL proposals under the call in question, if you:

i. are a member of an advisory group set up by the Commission to advise on the preparation of EU or Euratom Horizon 2020 WPs or WPs in an area related to the call in question;

ii. are a National Contact Point (NCP) or is working for the Enterprise Europe Network (EEN);

iii. are a member of a Programme Committee.

In case you face one of the above mentioned situations,

please contact and inform the CC as soon as possible, in order to take the necessary measures

The following situations may be considered as CoI if the responsible REA staff so decides, in view of the objective circumstances, the available information and the potential risks:

i. you have been employed by one of the applicants (or linked third parties or other third parties involved in the action) in the last three years;

ii. you have been involved in a contract, grant, prize or membership of management structures (e.g. member of management or advisory board etc.) or research collaboration with an applicant, a linked third party or another third party involved in

the action in the last three years; iii. any other situation that could cast doubt on your ability to participate in the

evaluation impartially, or that could reasonably appear to do so in the eyes of an outside third party.

If you breach any of the above mentioned obligations, REA may apply specific measures in particular

terminate the contract (as detailed in the Model Contract for experts).

You will be required to confirm — for each proposal you are evaluating — that there is no CoI, by signing a declaration in the SEP system.

As soon as you become aware of a CoI at any moment during the evaluation, you must immediately

inform the PO & CC via REA-SWAFS@ec.europa.eu and stop working until further instructions,.

IMPORTANT: As soon as the proposals are assigned to you, please check part A of the proposal (includes an abstract of the proposal and details of the partners involved) in order to assess if you have

any potential Conflict of Interest.

2.3 Confidentiality

During the implementation of your contract as expert-evaluator, and for five years after the date of

the last payment, you must keep confidential all data, documents or other material (in any form) that

is disclosed (in writing or orally) and that concerns the work under your contract (‘confidential

information’).

11

Unless otherwise agreed with REA, you may use confidential information only to implement your

work. You must keep your work strictly confidential, and in particular:

a) not disclose (directly or indirectly) any confidential information relating to proposals or

applicants, without prior written approval by the Agency;

b) not discuss proposals with others (including other experts or Commission or REA staff that are not directly involved in the evaluation of the proposal except during evaluation meetings and with prior approval by the responsible REA staff);

c) not disclose: - details on the evaluation process or its outcome, without prior written approval by

the REA staff; - details on your position/advice; - the names of other experts participating in the evaluation.

d) not communicate with applicants (including linked third parties or other third parties involved

in the actions) during the evaluation or afterwards;

e) if REA makes documents or information available electronically for remote work, you are

responsible for ensuring adequate protection and for returning, erasing or destroying all

confidential information after the end of the evaluation.

If you work on REA premises, you:

a) may not remove from the premises any documents, material or information on the

proposal(s) or on the evaluation;

b) are responsible for ensuring adequate protection of electronic documents and information

and for returning, erasing or destroying all confidential information after the end of the

evaluation.

If you use outside sources (for example internet, specialised databases, third party expertise, etc.) for your evaluation, you:

a) must respect the general rules for using such sources; b) may not contact third parties, without prior written approval by REA.

The confidentiality obligations no longer apply if: - REA agrees to release you from the confidentiality obligations; - the confidential information becomes public through other channels; - disclosure of the confidential information is required by law.

If you breach any of these obligations, REA may apply specific measures set (according to the Model Contract for experts).

2.4 Implicit gender biases

Making sure that the allocation of funds is not affected by implicit gender biases is of paramount

importance for ensuring that excellent applications are impartially rewarded through Horizon 2020.

Indeed, significant scientific research shows evidence of implicit gender biases in evaluation

processes. Implicit (or unconscious) biases refer to a cognitive phenomenon that takes place

automatically and without our knowledge when assessing people and situations, influenced by our

background and socio-cultural environment. Categorising people is a trait inherited through human

evolution. We tend to more highly rate a person who belongs to our 'in group' (someone who we

consider to be 'like us'). Implicit biases based on gender stereotypes can affect both men and women,

and influence behaviour and decision making.

12

In order to strengthen the evaluation and guarantee a fair process, it is imperative to mitigate the

effects of implicit gender biases and avoid biased judgements.

Have a look at this video and portals for more info:

Royal Society (UK): Understanding unconscious bias

Gendered Innovations

GenPORT

3. The evaluation procedure

3.1 The Evaluation workflow

The evaluation workflow consists of the following successive steps:

Figure 2: Workflow of the different steps of the evaluation process (blue circles = performed by REA; green circles = involvement of expert-evaluators)

1. Receipt of proposals/admissibility and eligibility check: After the call deadline, REA will check

which submitted proposals have passed all admissibility and eligibility criteria, and will assign

proposals to the expert-evaluators;

2. Individual evaluation (‘remote’ phase): after verifying that you don't have a conflict of interest

(CoI), expert-evaluators should accept to evaluate the proposals they have been allocated in SEP,

and draft and submit an Individual Evaluation Report (IER) – including evaluation comments (for

each sub-criterion) and scores (for each criterion) – for each proposal. This step is done

‘remotely’ (= not in Brussels);

3. Consensus group (‘central’ or ‘consensus meeting’ phase): evaluators are invited to Brussels (up

to 5 days3) to discuss each proposal in a ’consensus meeting’ composed of 3 experts (‘consensus

group‘) and agree on a final version of a Consensus Report (CR), summarising the evaluation

findings (comments and scores for each criterion) of all experts. The CR is drafted and prepared

by a dedicated Rapporteur, who does not evaluate the proposal. A first draft of the CR is

prepared prior to the central week. Each consensus meeting is moderated by a PO or Vice-chair;

4. Panel review (at the end of the ‘central’ phase): after all CRs have been drafted, submitted and

approved in SEP, a panel review will take place for each of the topics to compare the different

3 Exact number of days already communicated by REA SwafS team.

13

CRs in order to ensure consistency and equal treatment of all the proposals. The results of the

review are formulated in a panel report (including notably the panel ranked list);

5. Finalisation: the final ranked lists (one ranked list for each topic) will be based on the outcome of

the panel review, and will be used for subsequent internal procedures in view of the information

to applicants and preparation of Grant Agreements for successful proposals.

More information about the evaluation workflow can be found on

H2020 Grants Manual - Section on Proposal submission and evaluation

(from points III.5 to IV.1, pages 4-17)

3.2 The ‘remote’ phase

3.2.1 Admissibility and Eligibility Check

After the closure of proposal submission, the proposals are checked by REA for admissibility and

eligibility as set in the Annex B and Annex C in the General Annexes of the WP 2018 -2020. Once

those checks are performed, and taking into consideration possible conflicts of interest, REA will

assign the proposals submitted to expert-evaluators.

If you spot any issue in relation to admissibility/eligibility rules please inform the CC.

3.2.2 Individual evaluation

As expert-evaluator you must examine each proposal individually and assess it ensuring that:

you understand the topic under which the proposal was submitted (see Work Programme

topic description);

you read and explore in depth all proposals assigned to you;

you assess on the basis of the evaluation criteria and sub-criteria (see topic briefing slides &

assessment grid);

you provide comments for each sub-criterion;

the scores are consistent with the comments, using the scoring scale appropriately;

the evaluation is made on the basis of the proposal as it was submitted (NOT on its potential if

certain improvements were to be made).

3.3 The central evaluation (‘central’ phase)

The central evaluation in Brussels consists of the consensus meetings (CM) and a panel review

meeting for each topic.

3.3.1 Consensus meetings (CM)

The CM for each individual proposal will take place in Brussels and the discussion involves the expert-

evaluators who carried out the individual evaluations for a given proposal.

The group is guided by a moderator (PO or Vice-chair) and a rapporteur:

the moderator chairs the CM and ensures that each proposal is evaluated fairly according to

the evaluation criteria and helps the experts come to a consensus;

14

the role of the rapporteur is to draft a consolidated version of the CR (one CR per proposal) in

advance of the CM, to propose appropriate wording for the points of consensus reached

during the meeting and once the CM concludes to fine-tune the text of the CR.

As expert-evaluator you are asked to reach an agreement (consensus) on the evaluation of each

proposal (i.e. comments for each of the evaluation sub-criteria and appropriate scores reflecting the

comments).

Details on the logistics and time schedule for CMs

will be sent by email in advance of the central evaluation in Brussels.

3.3.2 Panel review and finalisation

After the consensus phase, once all proposals have been discussed and CRs approved, the REA

organises the panel reviews for each topic (composed of some experts involved in the CMs).

The objective of the panel review meetings is to compare the different CRs for a given topic in order

to ensure consistency and equal treatment of the proposals. During the panel review, the comments

and/or scores of the proposals can be adjusted by the panel.

The final scores and revised comments for each proposal are then captured in an ESR, which is based

on the CR. The results of the panel review are formulated in a ’panel report’ including notably the

’panel ranked list’.

The ESR of an individual proposal is provided to the applicants.

After the finalisation of the evaluation, a ranked list of proposals is produced (’panel ranked list’) and

is the basis for the final decision on funding.

Detailed information about this phase will be provided during

the central evaluation week in Brussels

15

4. Your IER

4.1 Proposals

The proposals you will evaluate consist of 2 parts and follow a standard template according to the

action type, CSA or RIA:

Part A: administrative forms (structured information of the basic administrative data,

declarations of partners, organisations and contact persons, etc.)

Part B: technical annex which is split in 2 sections:

o Sections 1-3: is the core of the proposal (see templates below for details) and is

subject to page limits according to the action type.

Note that excess pages will not be visible to you.

o Sections 4-5: details the members of the consortium, third parties as well as ethics

and security and is not subject to page limits.

Further optional annexes (e.g. supporting documents for ethics issues)

IMPORTANT: Part A of the proposal is the reference for your CoI check.

See CSA & RIA templates for more information. Page limit for sections 1-3:

CSA = 50 pages , RIA = 70 pages

Note that for some SwafS topics the proposal template includes section 4.3 on 'financial support to

third parties' (see section 5 below for more details)

Each expert evaluator will be assigned approximately 6 proposals to evaluate.

4.2 Evaluation criteria and sub-criteria

SwafS proposals are evaluated on the basis of 3 evaluation criteria common for all Horizon 2020:

Excellence, Impact, and (quality and efficiency of the) Implementation.

Each evaluation criterion is further divided into different sub-criteria.

For more information about the evaluation sub-criteria,

please consult the relevant topic briefing slides & assessment grid.

16

4.3 Scoring scale & thresholds

Based on the proposal assessment and comments, each evaluation criterion will receive a score

between 0 and 5 (half-marks can be used). In figure 3 below, the interpretation of each score is

described.

0 — The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.

1 — Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.

2 — Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.

3 — Good. The proposal addresses the criterion well, but a number of shortcomings are present.

4 — Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.

5 — Excellent. The proposal successfully addresses all relevant aspects of the criterion, any shortcomings are minor.

Figure 3: Horizon 2020 scoring scale

In order to be considered for funding, proposals need to pass certain thresholds:

for each individual criteria (excellence, implementation and impact) the threshold is 3/5;

the overall threshold (= the sum of the three individual scores) is 10/15.

A distinction exists between a weakness and a shortcoming which can be interpreted as follows:

Weakness vs Shortcoming

A (significant or inherent) weakness is a major problem/issue, seriously jeopardising the success of

the proposal

A shortcoming is a negative element in the evaluation, but one which will not result in a score

lower than 3.

Both the amount (‘a number’ – score 3; ‘a small number’ – score 4) and nature (‘minor

shortcoming' – score 5) of shortcomings/weaknesses in relation to how well the proposal

addresses the criterion defines the score.

On the positive points: pay attention to the language which should be synonyms of score

justification e.g. (5) excellent=outstanding, exceptional etc.

4.4 Additional elements to assess

In a separate section of the IER and CR, you are asked to give your view on the following additional

aspects.

4.4.1 Scope

Expert-evaluators are asked to assess whether or not the proposal is 'in scope' at least partially. A

proposal is ’in scope’ if its content corresponds - at least in part - to the topic for which it is

submitted.

In case you consider one of the proposals you are evaluating is out of scope,

please contact the CC for confirmation.

17

4.4.2 Operational capacity

Expert-evaluators are asked to evaluate whether each applicant has the necessary basic operational

capacity to carry out their proposed activities. You should evaluate this basic operational capacity

based on information provided in the proposal (part B, section 4), specifically:

The description of the applicant(s);

CVs;

Relevant publications or achievements;

Relevant previous projects or activities;

The description of any significant infrastructure or any major items of technical equipment;

The description of third parties contributing to the work but not represented as project partners

If at the individual evaluation stage, you consider that a partner lacks basic operational capacity,

evaluate the full proposal including parts related to the partner concerned.

However, if at the consensus stage, experts who evaluated the proposal agree that a partner lacks

basic operational capacity, you will make comments and score the proposal without taking into

account this applicant and its associated activity(ies).

This check is primarily designed to identify manifestly inadequate (or even fraudulent) partners.

4.4.3 Exceptional funding of third country participants/international organisations

Horizon 2020 is Open to the World. This means that participants from all over the world, regardless

of their place of establishment or residence, can participate in most of the calls of Horizon 2020.

However some countries are automatically eligible for funding and some are not.

Countries whose entities are automatically eligible for funding:

European Union Member States, including their overseas departments and outermost

regions;

Associated Countries (AC) can participate under the same conditions as legal entities from

Member States: Albania, Armenia, Bosnia and Herzegovina, Faroe Islands, Iceland, Israel,

Moldova, Montenegro, Norway, FYROM, Tunisia, Turkey, Serbia, Ukraine, Georgia,

Switzerland;

Some third Countries which are not a Member State or AC: see the 'Annex A - List of

countries, and applicable rules for funding' for the complete list;

Exceptionally, other countries if:

i. bilateral agreement e.g. EU-US/NIH arrangement, or,

ii. identified in the WP, or

iii. deemed essential for carrying out the action (their participation has clear benefits

for the consortium (e.g. outstanding competence/expertise; access to research

infrastructure; access to particular geographical environments; access to data)).

As an expert evaluator, for any third country not automatically eligible for funding or international organisation, you should give your view on whether or not you consider that they should exceptionally be granted funding.

18

BREXIT: Until the UK leaves the EU, EU law continues to apply to and within the UK, both when it

comes to rights and obligations. This includes the eligibility of UK legal entities to participate and

receive funding in Horizon 2020 actions. Experts should not evaluate proposals with UK participants

any differently than before.

All of the above questions need to be answered in the corresponding sections in the IER in SEP.

4.5 Tips for drafting a good IER

a) Evaluate the proposal ‘as is’ / as it is submitted, not what it could be if certain changes would be

made. Therefore, do not make recommendations (do not suggest additional partners,

additional work packages, resources that need to be cut or added, …), but limit yourself to

evaluating what is written in the proposal and justifying why it is excellent, (very) good, a

shortcoming, or a weakness.

b) Use value judgements that reflect the strengths and weaknesses (e.g. "the proposal addresses

the impact criterion very well"), and do not needlessly copy/paste parts of the proposal text in

the IER. The applicants know what they have written in their proposal, and are mainly

interested in knowing the reasons why you have given a certain score. A useful – not mandatory

- structure to build an argument could be "The proposal aims at doing A and B. However, the

proposal does not address C. Therefore, the proposal addresses criterion X fairly [score 3], very

well [score 4] or in an excellent manner [score 5]". This way, the inclusion of (parts of) the

proposal text in the IER adds value.

c) Use specific examples to make the argument

d) Be precise yet careful and avoid factual mistakes. Instead of writing "This proposal does not

address X", it is advisable to write "This proposal does not significantly [or sufficiently]

addresses X". This approach ensures that human errors during the evaluation process are

proactively tackled and avoids unnecessary cases of evaluation review. Be precise when

providing inputs on positive and negative aspects, providing details for such an evaluation.

Example: “Although the whole value chain is described very well in the proposal, certain

elements of the value chain are not sufficiently covered. For example, it is unclear how a stable

supply of biomass will be generated”. If you give precise information regarding the proposal e.g.

reference to specific work packages, double-check that your statement is correct.

e) Align the score and the evaluation text. The IER texts should be in line with the interpretation of

the scores. For example, if you mostly use 'good' to describe a criterion, you cannot give it a

score of 4 (as a score of 4 is 'very good').

f) Page limits. Any excess pages are clearly watermarked and should be disregarded. Therefore,

when writing the IER, please make sure not to refer to information provided in these excess /

watermarked pages. Please also note that no page limits are applicable to sections 4

(description of partners) and 5 (Ethics and Security) of Part B (this is why in SEP, sections 1-3

and sections 4-5 are visible in separate pdf documents).

g) Evaluate all 3 criteria and all sub-criteria. You are required to evaluate all 3 criteria in the IER

and CR, even if one is (well) below the threshold. Therefore, as each sub-criterion will need to

be covered in the CR, please write at least 2-3 sentences per sub-criterion in your IER (around

100 words). Refer to topic briefing slides to ensure you considered all elements. Keep in mind

that the rapporteur will need to understand your points when preparing the pre-draft of the CR.

h) Qualify AND/OR quantify each proposal's shortcomings. In the scoring table (figure 3), a

qualitative ('(serious) inherent', 'significant', 'minor') or quantitative ('a number', 'a small

number') description of the proposal's shortcomings or weaknesses is described for each score.

19

Please use this terminology to guide you during the evaluation process. e.g. a criterion that has

significant weaknesses cannot receive a score of 4, even if the other sub-criteria are evaluated

as very good or even excellent.

i) Focus on the content first, the score comes last. When drafting an IER, focus on the content of

your (sub)criterion evaluation first, and (only) then on the score. Your score should reflect the

comments well enough to enable an informed reader to arrive at the same score.

After submission, an IER can normally NO longer be changed. If duly justified, REA may allow for

changes by the expert (and re-open the report for change and re-submission).

Your 1st draft IER will be checked by REA PO or VC to ensure that you are applying the guidelines

provided, notably the topic briefing slides/assessment grid.

5. Other relevant aspects

5.1 Open access and research data

Open access can be defined as the practice of providing on-line access to scientific information that

is free of charge to the reader. In the context of R&D, open access typically focuses on access to

'scientific information' or 'research results', which refers to two main categories:

a) peer-reviewed scientific research articles (primarily published in academic journals);

b) research data.

a) Peer-reviewed scientific research articles (primarily published in academic journals)

Under Horizon 2020, each beneficiary must ensure open access (OA) to scientific publications (in

particular scientific peer reviewed articles). This is a contractual obligation (Article 29.2 of the H2020

AGA4).

Under "Measures to maximise impact - a) Dissemination and exploitation of results" applicants

should include and describe the measures to provide open access to peer-reviewed scientific

publications resulting from the project (if any).

In Horizon 2020 the EU supports both self-archiving ('green' OA) and open access publishing ('gold

OA'):

Self-archiving / 'green' OA: beneficiaries must:

i. deposit the final peer-reviewed manuscript in a repository5 of their choice;

ii. provide open access to it with an embargo period of at most 6 months (12 months for

publications in the social sciences and humanities).

Open access publishing / 'gold' OA:

i. researchers may also publish in full OA journals, or in so-called 'hybrid journals' (which

sell subscriptions and offer the option of making individual articles openly accessible).

Note that in case of gold OA the article must also be deposited in a repository.

4 Annotated Model Grant Agreement 5 The project website does not normally qualify as a repository (since it is often terminated at the end of the project).

20

b) Research data

Under "Measures to maximise impact - a) Dissemination and exploitation of results" applicants need

to include, as relevant, information on how they will manage the research data generated and/or

collected during the project, in particular addressing the following issues:

i. What types of data will the project generate/collect?

ii. What standards will be used?

iii. How will this data be exploited and/or shared/made accessible for verification and

re-use? If data cannot be made available, explain why.

iv. How will this data be curated and preserved?

v. How will the costs for data curation and preservation be covered?

Open access to research data is one part of research data management. Please note that

participation in the extended Open Research Data pilot (ORD) is NOT part of the evaluation. In other

words, proposals will not be penalised should they choose to opt out of the ORDP. However,

regardless of this choice, applicants should consider how they will handle data collected during the

project.

More information on the open access requirements of Horizon 2020 and how to fulfil them can be

obtained at our online manual and the FAQ section of the funding and grants portal for ‘open

access’.

Open access is relevant for all SwafS topics.

5.2 Financial Support to third parties

Some SwafS 2019 topics allow for the provision of financial support to third parties in line with the

conditions set out in Part K of the General Annexes of Horizon 2020.

What does financial support to third parties mean?

it's a financial donation to natural persons or legal persons in the form of a 'cascading grant'(

i.e. funding by the beneficiary of recipient(s) that are not party to the Grant Agreement);

should be necessary to achieve the objectives;

budget (amount) should be shown in part A, budget table (column D);

details including award criteria, activities/conditions for participation should be included in

Part B section 4.3* of the proposal based on the template.

Note: this funding is different from third parties involved in the implementation of the action (linked

third parties, subcontractors, contractors, in-kind contributors).

*Financial support to third parties

For detailed specific info on terms and conditions: see General Annex K of the Horizon 2020 Work Programme

published in the reference documents section of the H2020-Work Programme 2018 -2020 General Annexes

Financial support in the form of a grant awarded after a call for proposals

Where this possibility is indicated under the relevant topic in the Work Programme and in the

relevant calls for proposals, proposals which foresee financial support to third parties, shall:

1. clearly detail the objectives and the results to be obtained and

2. contain the following specifications (as a minimum):

21

a) a closed list of activities that qualify for financial support;

b) the definition of persons or categories of persons that may receive financial support;

c) the criteria for awarding financial support;

d) the criteria for calculating the exact amount of the financial support;

e) the maximum amount of financial support per third party, which must not exceed EUR 60 000, unless a higher amount is necessary to achieve the objectives of the action, and the criteria for determining it.

Financial support to third parties is a possibility for

SwafS-05, Swafs-14 and SwafS-15

5.3 Additional dissemination obligations

Several of the SwafS topics are subject to additional dissemination obligations. This means that

consortia must make active efforts to freely share in a timely manner and, as appropriate, the

research strategies, methodologies, and raw and analysed data deriving from their activities

(including any evaluation activities) with the other projects funded by SWAFS (also subject to these

same obligations). These are therefore reciprocal obligations for all projects funded under the topics

concerned.

The topics concerned by additional dissemination obligations are: SwafS-05, SwafS-15, SwafS-17 & SwafS-19

Applicants are expected to demonstrate how they will comply with this, bearing in mind that efforts

are expected to take place organically between projects, with little or no EC intervention, and that

new websites or other types of infrastructure requiring significant time/resources are not specifically

foreseen.

This should be considered when evaluating the dissemination aspects under the corresponding

'impact' sub-criterion.

5.4 Ethics review

For proposals above threshold and considered for funding, an ethics screening and, if necessary, an

ethics assessment is carried out by independent ethics experts. Only proposals that comply with the

ethical principles and legislation may receive funding.

For those proposals in which one or more ethical issues have been identified, the ethics expert will

assess whether the ethics issues are adequately addressed.

The ethics experts will produce an ethics report and give an opinion on the proposal, including:

granting ethics clearance (or not);

recommending the inclusion of ‘ethics requirements’ in the grant agreement; or

recommending a further ethics assessment and/or an ethics check or audit.

According to the Article 34.2 (Activities raising ethical issues) of the H2020 AGA:

Activities raising ethical issues must comply with the 'ethics requirements' set out as

deliverable in Annex I.

22

Before the beginning of an activity raising an ethical issue, each beneficiary must have

obtained:

a) any ethics committee opinion required under national law; and

b) any notification or authorisation for activities raising ethical issues required under

national and/or European law needed for implementing the action tasks in question.

Ethics screening and assessment will be carried out by dedicated Ethics experts.

Therefore, no specific actions are needed from the expert-evaluators.

6. Planning, references and contacts

6.1 Planning

The 2019 SwafS single submission call evaluation timeline is summarised as follows:

Evaluation step What? When?

Receipt of proposals

Call deadline / Proposal submission 2 April 2019

Eligibility and admissibility checks (REA staff) 3 April 2019

Individual evaluation

Web-briefing / Launch of remote evaluation in SEP 25 April 2019

Write / Submit IERs 25 April – 16 May 2019

Preparation of CRs Pre-draft of CRs done by rapporteur 17 – 29 May 2019

QC of CRs (VC/PO & rapporteur) 3 – 7 June 2019

Central week Consensus meeting & panel review meetings in Brussels

17-21 June 2019

Finalisation Inform applicants about evaluation results July 2019 Figure 4: 2019 SwafS Call timeline

See detailed schedule in briefing web-page

6.2 References

H2020-SwafS-2018-2020 Work Programme (read Work Programme topic description)

Topic-briefings (3 presentations). FAQs annexed in Project Officer topic briefing

Topic assessment grid

Proposal template

o CSA

o RIA

SEP manual

6.3 Contacts

Your first contact point for questions related to the topic is the Project Officer lead for the

topic (see contacts in briefing web-page) email: firstname.surname@ec.europa.eu

For any general questions contact Call Coordinator (Niamh Delaney) and Call Assistants

(Roxana Paulovici/Roxana Boar) via: REA-SWAFS@ec.europa.eu