Proposal evaluation in FET
Future and Emerging Technologies Giornata Nazionale di Lancio dei Bandi 2018-20 in Horizon 2020
Roma, 20 Novembre 2017
Nicola Ciulli Head R&D @ Nextworks External expert @ EC, REA, EASME, Eureka, EIT
We are here in the FET map
FET-Open – Novel ideas for radically new technologies FET-Open Challenging Current Thinking
FET-Open Coordination and Support Actions
FET Innovation Launchpad
FET Proactive – Boosting emerging technologies FET Proactive: emerging paradigms and communities
Community building in Neuromorphic Computing Technologies
FET ERA-NET Cofund
FET Proactive – High Performance Computing International Cooperation on HPC
Extreme scale computing technologies, methods and algorithms for key applications and support to the HPC ecosystem
HPC PPP - Co-design of extreme scale HPC systems and applications
FET FLAGSHIPS – Tackling grand interdisciplinary science and technology challenges Preparatory Actions for new FET Flagships
ERA-NET Cofund for FET Flagships
FET Flagship on Quantum Technologies
Quantum Flagship – QuantERA II ERA-NET Cofund
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 2
FET-Open vs FET Proactive
FET-Open Challenging Current Thinking FET Proactive: emerging paradigms and communities
Goals • to lay the foundations for radically new future technologies of any kind from visionary interdisciplinary collaborations that dissolve the traditional boundaries between sciences and disciplines, including the social sciences and humanities
• to explore and consolidate a new technological direction in order to put it firmly on the map as a viable paradigm for future technology
• to foster the interdisciplinary communities that are able to drive this forward, extending from the participating consortia to a wider European pool of expertise
Focus • fully bottom-up, with GKs • specific sub-topics identified
Impact sought
• Scientific and technological contributions to the foundation of a new future technology
• Potential for future social or economic impact or market creation
• Building leading research and innovation capacity across Europe by involvement of key actors (excellent young researchers, ambitious high-tech SMEs or first-time participants)
• Scientific and technological contributions to the foundation and consolidation of a radically new future technology
• Potential for future returns in terms of societal or economic innovation or market creation
• Spreading excellence and building leading innovation capacity across Europe by involvement of key actors that can make a difference in the future, for example excellent young, researchers, ambitious high-tech SMEs or first-time participants to FET under Horizon 2020
• Build-up of a goal oriented interdisciplinary community (within and beyond the consortium)
• Emergence of an innovation ecosystem around a future technology in the theme addressed from outreach to and partnership with high potential actors in research and innovation, and from wider stakeholder/public engagement, with due consideration of aspects such as education, gender differences and long-term societal, ethical and legal implications
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 3
FET-Open vs FET Proactive: criteria
FET-Open Challenging Current Thinking FET Proactive: emerging paradigms and communities
Excellence (Threshold: 4/5, Weight: 60%)
• Clarity of the radical vision of a science-enabled technology and its differentiation from current paradigms.
• Novelty and ambition of the proposed science-to-technology breakthrough that addresses this vision.
• Range of and added value from interdisciplinarity for opening up new areas of research; non-incrementality of the research proposed.
• High-risk, plausibility and flexibility of the research approach.
• Clarity of long-term vision of a science-enabled technology.
• Concreteness and ambition of the proposed science-to-technology breakthrough that addresses this vision.
• Range and added value from interdisciplinarity, novelty and non-incrementality of the research proposed.
• High-risk of the research proposed and plausibility and flexibility of the approach.
Impact (Threshold: 3.5/5, Weight: 20%)
• The extent to which the outputs of the project would contribute to the expected impacts listed in the work programme under this topic.
• Effectiveness of measures and plans to disseminate and use the results (including management of IPR) and to communicate about the project to different target audiences.
• The extent to which the outputs of the project would contribute to the expected impacts mentioned in the work programme under the relevant FET topic.
• Effectiveness of measures and plans to disseminate and use the results (including management of IPR) and to communicate the project to different target audiences.
Implementation (Threshold: 3/5, Weight: 20%)
• Coherence and effectiveness of the research methodology and work plan to achieve project objectives and impacts, including adequate allocation of resources to tasks and partners.
• Role and complementarity of the participants and extent to which the consortium as a whole brings together the necessary expertise.
• Coherence and effectiveness of the work plan to achieve project objectives and impacts, including adequate allocation of resources to tasks and partners.
• Appropriateness of the research and innovation management structures and procedures.
• Role and complementarity of the participants and extent to which the consortium as a whole brings together the necessary expertise.
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 4
Focus on FET-Open
The competition in FET-Open is usually extremely high
It’s not enough to prepare a good proposal; it must be perfect under all aspects
i.e. C.1 Excellence is key, but far from enough
To do so,
it is very important to know what will be evaluated by which expert profile,
according to which specific implementation of the evaluation process
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 5
A glimpse of the evaluation process [1]
Usually the QC is performed on a CR draft
i.e. after the individual assessments have been finalized and a consensus have been achieved
In FET-Open, QC is performed more upstream…
on the IER, i.e. at the root of the assessment
QCs cannot challenge the individual opinions of the experts, but
How they are expressed in the comments
Their pertinence in each subcriterion
Their completeness according to the evaluation subcriteria
Their balance vs. the proposed scores
Even if the expert evaluators might be biased towards some specific aspects, the VC check brings everything back in focus
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 6
crossreading panel
A glimpse of the evaluation process [2]
E1 E2 E3 E4
E1 E2 E3 En
QC
• Comments can change • Scores can
change
• Comments and scores might change, • are endorsed • and proposals
ranked
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 7
Criteria & subcriteria
Excellence Impact Implementation
• Clarity and novelty of long-term
vision, and ambition and
concreteness of the targeted
breakthrough towards that vision.
• Novelty, non-incrementality and
plausibility of the proposed research
for achieving the targeted
breakthrough and its foundational
character.
• Appropriateness of the research
methodology and its suitability to
address high scientific and
technological risks.
• Range and added value from
interdisciplinarity, including
measures for exchange, cross-
fertilisation and synergy.
• Importance of the new technological
outcome with regards to its
transformational impact on
technology and/or society.
• Impact on future European scientific
and industrial leadership, notably
from involvement of new and high
potential actors.
• Quality of methods and measures for
achieving impact beyond the
research world and for establishing
European though leadership, as
perceived by industry and society.
• Soundness of the workplan and
clarity of intermediate targets.
• Relevance of expertise in the
consortium.
• Appropriate allocation and
justification of resources (person-
months, equipment).
Threshold: 4/5, Weight: 60% Threshold: 3.5/5, Weight: 20% Threshold: 3/5, Weight: 20%
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 8
Excellence [1]
Clarity and novelty of long-term vision, and ambition and concreteness of the targeted breakthrough towards that vision.
Is there a clearly defined novel long-term vision?
Is there a clear description of the proposed breakthrough research? Is the research well targeted towards an attainable goal or represents continuation of research along various parallel paths already being followed?
Relevance of the targeted breakthrough for the long-term vision – it should be outlined how the breakthrough would lead to the long-term vision.
Are the objectives well described in the proposal? Are they specific, measurable, achievable, relevant, and time-bound within the duration of the project?
Novelty, non-incrementality and plausibility of the proposed research for achieving the targeted breakthrough and its foundational character.
Is the analysis of the state-of-the-art comprehensive and complete?
Is the proposed research novel? – It should be more than continuation of research being already done or pursued currently.
Is the research proposed ambitious and far-reaching, not just another small incremental step along a path already followed?
Is the research plausible?
Does the research proposed have a potential to start a new line of investigation, which can be followed for years and can lead to completely new technologies
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 9
Excellence [2]
Appropriateness of the research methodology and its suitability to address high scientific and technological risks.
What is the relevance of the chosen methods for the achieving the proposal objectives?
Is the methodology "risk-friendly"? The details of risk analysis and mitigation plan should be addressed under Implementation.
Range and added value from interdisciplinarity, including measures for exchange, cross-fertilisation and synergy.
To what extent the main idea requires involvement of knowledge and methodology from different disciplines?
How "distant" are these disciplines involved? (There are disciplines, which traditionally work together, while others speak different languages and use different methodologies. The expectation is that putting together distant disciplines may bring qualitatively new solutions).
How are these different disciplines intertwined and how the ideas from different disciplines support the scientific breakthrough?
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 10
Impact
Importance of the new technological outcome with regards to its transformational impact on technology and/or society.
Is the long-term technological outcome clearly defined?
Is the transformational impact on technology, on society or on both convincingly argued?
It is not sufficient to discuss only the impact based on peer reviewed publications!
Impact on future European scientific and industrial leadership, notably from involvement of new and high potential actors.
Is the general impact on future European scientific and industrial leadership properly discussed?
Below are some examples of involvement of new and high potential actors.
• Are there researchers at the start of their career, especially principle investigators, who would keep up the research in a long run? or
• Are there SMEs in the consortium with sufficient budget, which would take up the results from the project and would eventually carry them over towards innovation and market realization? or
Quality of methods and measures for achieving impact beyond the research world and for establishing European through leadership, as perceived by industry and society.
Are the "standard" measures (scientific publications, website) properly described and planned?
Does the proposal go sufficiently far beyond the "standard" measures? For example, are there additional measures, based on new social media included?
Is it clearly explained how the dissemination measures would support achieving the expected impact?
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 11
Implementation
Soundness of the workplan and clarity of intermediate targets.
Is there a clear description of the tasks that will be performed, with specific deliverables allowing the verification of the execution of the tasks? Are there well defined work packages with relevant interdependencies among them? Is the time-dependence of the different tasks meaningful and convincing?
Are the intermediate targets (milestones) properly defined and timely? These should be major achievements serving as verification points if the work should continue according to the original plan or alternative paths should be followed.
Relevance of expertise in the consortium.
Is the expertise in the consortium broad enough in order to tackle all proposed tasks and is it of high enough quality?
Is the interdisciplinarity of the ideas in the proposal (if present) properly reflected in the composition of the consortium?
Is the consortium well-balanced, without redundancies?
Appropriate allocation and justification of resources (person-months, equipment).
Do the person-months allocated correspond to the proposed tasks to be performed?
Is the necessary equipment present or properly described and budgeted?
Is the overall budget comprehensive, well-balanced, and convincing?
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 12
Generic golden rules in proposal writing
Avoid using excess wording
It’s either illegal or counterproductive
Improve readability by editing tricks
The reading & discussion time is so short
Write the proposal with the evaluation sub-criteria clear in mind
Avoid writing what’s not needed, and avoid missing what’s needed
Do not overlook any of the subcriteria and template items
Have the proposal proof-read once ready
by someone not participating in the process,
with some experience of the evaluation criteria,
and of the technology involved
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 13
Nicola Ciulli Head of Research & Development External expert @ EC, REA, EASME, Eureka, EIT [email protected]
FET Giornata Nazionale di Lancio dei Bandi 2018-20 H2020 14