Report on the methodology and list of KPIs for the ......In addition, guidelines delivering...

BlueBio D4.1 (Methodology and KPIs for monitoring and evaluation)

H2020-BG-02-2018

Blue Bioeconomy - Public Public Partnership - BlueBio

D4.1

Report on the methodology and list of KPIs for the

monitoring and evaluation of the cofounded projects

(WP 4)

January 2020

BlueBio D4.1 (Methodology and KPIs for monitoring and evaluation)

OUTPUT SUMMARY

Project Information

Project Title: ERA-NET Cofund on Blue Bioeconomy “Unlocking the Potential of Aquatic Bioresources”

Project Acronym: BlueBio

Call Identifier: H2020-BG-02-2018

Contract Number: 817992

Starting Date: 01/12/2018

End Date: 30/11/2023

Web-Site Address: https://bluebioeconomy.eu/

Coordinator: Kristin THORUD (RCN)

E-Mail: [email protected]

Deliverable Title: Report on the methodology and list of KPIs for the monitoring and evaluation of the cofounded projects

Deliverable Number: D 4.1

Work Package: WP 4

WP leader AEI

Nature: Report

Dissemination: Confidential, only for members of the consortium (including the Commission Services)

Editor (s): Dominca Cotet (UEFISCDI), Kristin Thorud (RCN), Evi Afentaki (GSRT), Esther Campollo and Abraham Trujillo (AEI), Bernardo Patti (CNR)

E-Mail(s): [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]

Date of Delivery 22/01/2020 (due delivery date: 31/10/2019)

https://bluebioeconomy.eu/

mailto:[email protected]







BlueBio D4.1 (Methodology and KPIs for monitoring and evaluation) 1

Table of Content

EXECUTIVE SUMMARY.....................................................................................................................................................2

1. INTRODUCTION ......................................................................................................................................................3

2. THE 2018 JOINT CALL UNDER BLUEBIO ................................................................................................................4

3. METHODOLOGY FOR A MONITORING AND EVALUATION FRAMEWORK ...........................................................4

3.1. PROJECT REPORTING AND EVALUATION .................................................................................................................5 3.2. KEY PERFORMANCE INDICATORS (KPIS) ................................................................................................................6


Executive Summary

The ERA-NET Cofund BlueBio responds to the Horizon 2020 (H2020) Societal Challenge 2 2018 Call topic BG-02-2018: Blue Bioeconomy Public-Public Partnership. EU intervention is needed to create the conditions to mobilise investments by aligning national and regional innovation research agendas across different Blue Bioeconomy sectors. This is precisely what BlueBio aims to achieve. BlueBio specifically creates synergies between sectors and leverage adequate investments in the Blue Bioeconomy. Within the ERA-NET Cofund, the first activity conducted has been the BlueBio 2018 Joint Call (JC), co-funded by the European Commission (EC).

The BlueBio 2018 JC has been implemented through the tasks described in WP2, WP3 and WP4 tasks, planned according to the specifications of the ERA-NET Cofund instrument. Whereas WP2 and WP3 focus on the preparation and launch of the call and evaluation of the proposals submitted, WP4 is centred in the monitoring and assessment of the projects funded from the beginning to the final evaluation. In the WP4 “Follow-up and monitoring of projects resulting from the co-funded call”, a specific task was assigned to the monitoring procedures and definition and development of a common set of KPIs. In addition, guidelines delivering recommendations for Human Capacity Building (HBC) activities to be carried out within BlueBio funded projects were developed by task 7.3 (MS23). This last document was intended to pave the way to MS24, a milestone aimed at integrating deliverable D4.1 giving additional inputs in support to the monitoring of training and mobility activities. The current report describes the actions developed, and it is intended to incorporate and consolidate the recommendations established for milestone MS24.

Embed

Step Description

WP2 Preparation and launch of the co-funded call

WP2-T2.1 Call documents

WP2-T2.2 Online submission system

WP2-T2.3 Call launch and Call Secretariat

WP3 Evaluation and proposal selection for the co-funded call

WP3-T3.1 Preparation

WP3-T3.2 Assessment of pre-proposals

WP3-T3.3 International full proposal peer review

WP4 Follow-up and monitoring of projects resulting from the co-funded call

WP4-T4.1 Preparation of monitoring procedures, definition and development of a common set of KPIs

WP4-T4.2 Online monitoring of funded projects through an existing online platform and development of a common reporting procedure. Collection and assessment of results through written follow-up reports.

WP4-T4.3 Organisation of two follow-up seminars

WP4-T4.4 Development of KPIs for assessment of short, mid- and long-term impact evaluation

WP7- T7.3 Development of guidelines on HCB and training activities (MS23)

WP7- T7.3 Identification of common tools for HCB and training activities monitoring and evaluation (MS24)

Table 1: List of the steps in four Work Packages (WP2, WP3 and WP4). Additional work carried out by WP7 is also reported.


1. Introduction

The BlueBio ERA-NET Cofund responds to the Horizon 2020 (H2020) Societal Challenge 2 2018 Call topic BG-02-2018. BlueBio aims at paving the way for a sustainable and competitive Blue Bioeconomy in Europe through targeted research, and development and innovation activities. The goal is to identify new and improve existing ways of bringing bio-based products and services to the market and find new ways of creating value from in the blue bioeconomy. The main objective of BlueBio is to establish a coordinated R&D funding scheme that will strengthen Europe’s position in this complex economic field in a global market.

Aquatic biomass can ensure future food and nutrition security (FNS), supply for commodities and European competitiveness by sustainably cultivating, harvesting and processing resources. BlueBio enabled through EU support, creates cross-sectoral networks and cross-fertilisation between the sectors thereby taking European industries further into the knowledge economy. By addressing the Blue Bioeconomy in this integrated, interdisciplinary, and cross sectoral way, the BlueBio Cofund contributes to the production of safe, nutritious and valuable bio-products and services, while applying the food first principle. BlueBio does not focus on environmental concerns per se, but rather factoring in climate-friendly production and environmental sustainability a prerequisite in all activities.

BlueBio engages funding agencies to implement a co-funded call focusing on Blue Knowledge and technological developments to respond to needs and gaps for R&I to:

• Create critical mass along the value chains from biomass to products, services and markets; • Apply the ‘3R principle’ of Reducing, Reusing and Recycling to achieve a circular economy; • Explore innovative, sustainable and climate-friendly utilisation of aquatic biomass at different

trophic levels, as well as sustainable harvesting, and novel aquaculture production systems targeting a range of markets, products (food, feed, chemistry, nutraceuticals, cosmetics, etc.) in existing or new markets;

• Target new genetic resources and biomolecules using biotechnology to utilise close to 100 % of the available biomass in multistream biorefineries.

In the BlueBio partnership, the EU acts as a policy driver, giving direction to the strategic priorities and EU/MS commitments to deliver on: FOOD2030, Marine Strategy Framework Directive (MSFD), Common Fisheries Policy (CFP), Blue Growth Strategy, the United Nations Sustainable Development Goals (SDGs), COP21 agreements and the International Ocean Governance agenda. BlueBio underpins regional and international policy actions and initiatives. EU funding enables participation of small and/or new countries across the regional basins of Europe; triggering cross-fertilisation, capacity and infrastructure sharing and enhances growth across the whole of the EU, a clear added value.

BlueBio addresses gaps such as:

• developing innovative uses of underutilised and waste material from fisheries and aquaculture to achieve zero waste;

• using biotechnology and ICT to develop smart, efficient, traceable food systems and create synergies between aquaculture and fisheries (genetic assessment and digitalisation);

• unlock the potential of microbiomes to support growth in aquaculture, fisheries, and food processing and biotechnology; apply the latest developments in ICT (IoT, machine learning, big data) to the Blue Bioeconomy;

• creating predictive tools to improve the identificat ion and targeting of biodiversity “hot-spots” in the oceans (omics based technologies);


• exploring synergies with land-based production in areas such as food and feed production and processing, biorefining, bioenergy, biomaterials, chemicals and nutrients and maximise the use of aquatic bioresources in terrestrial value chains;

• improving aquaculture and wild harvesting of stocks by support for the creation of innovative feeds, improved brood stocks, by introducing new species, defining stock baselines, and assessing stocks and by encouraging the adoption of novel production technologies.

These themes and gaps are also echoed in the gaps identified in the SRIAs/Roadmaps of ERA-NETs COFASP, MBT and JPI Oceans. These networks, experiences and strategies will be expanded and aligned with more partners and further developed through BlueBio. Seeking to stimulate new R&I, synergies and collaboration with industries, both off-shore and land-based, will be sought. This includes improving integration of the biotechnology toolbox and knowledge within industrial sectors such as white (industrial), green (agricultural), yellow (environmental) and red (health and medical).

BlueBio contributes to the overall EU objective of building the ERA through enhanced cooperation and coordination of national research programmes.

The BlueBio 2018 JC committed resources from 28 funding partner organisations (FPOs) from 16 countries to implement a joint call for proposals, with EU co-funding on the topic "Blue Bioeconomy Public-Public Partnership". The aim of the BlueBio 2018 JC is to enable transnational, collaborative research, development and innovation projects addressing questions relating to the blue bioeconomy opportunities for society. The FPOs particularly wish to i) promote trans-disciplinary research, ii) encourage proposals combining basic and applied approaches, iii) stimulate mobility of researchers within the consortia, iv) enhance collaborative research and innovation during the project’s lifespan and beyond, and v) promote international cooperation in the blue bioeconomy area.

2. The 2018 Joint Call under BlueBio

On the 18th December 2018, the BlueBio consortium composed by 28 FPOs from 16 countries launched a co-funded transnational and multidisciplinary Call (BlueBio 2018 JC) for RDI proposals addressing the following priority areas:

Exploring new bioresources Exploring improvements in fisheries and aquaculture

Exploring synergies across sectors Exploring Biotechnology and ICT

The BlueBio 2018 JC seeks to maximise the benefit of synergies with other RDI activities taking place within partner countries, and therefore complements the harmonisation of National/Regional Agendas. Work Packages (WPs) WP2 and WP3 will be directly complemented by WP4 (Follow-up and monitoring of projects resulting from the co-funded call), which will cover the co-funded call cycle from kick-off to final evaluation, with the external contribution by WP7 (T7.3) as far as concerns the training and mobility aspects embedded in the BlueBio JC funded projects.

3. Methodology for a Monitoring and Evaluation Framework

A methodology for a monitoring and evaluation framework and indicators are here developed for the BlueBio 2018 Joint Call funded projects. National and Horizon 2020 (H2020) criteria are included in the indicators to avoid duplications and to facilitate the use of the framework at all administrative levels. The partners’ experience of similar processes (UEFISCDI, RCN, GSRT, CNR and AEI), the monitoring and evaluation of the COFASP and ERA-MBT, the information from ERA-LEARN and other


relevant activities have been used in this task. Guidelines for project reporting and monitoring were drafted and approved by the CSC and then communicated to all co-funded projects. Simple reporting templates will be also made available to the cofounded projects.

The work will be completed as follows:

1. Collection and analysis of all relevant information: collection of previous experiences and national criteria from BlueBio partners and collection of H2020 and Public 2 Public (P2P), ERA-Nets, ERA-LEARN criteria and procedures;

2. Definition of project monitoring: selection of indicators and criteria, first draft of procedures and criteria, discussion with task participants, final selection of indicators and criteria;

3. Reporting and evaluation report templates (and approuval by task participants) 4. Selection of the Call Evaluation Panel members (and approuval by task participants) 5. Approval of this Methodology and KPIs and Templates by the CSC: circulation of the draft

Methodology and KPIs and Templates to the Call Steering Committee for comments and approval;

6. Assessment of the funded project via questionnaires

7. Follow-up seminars. Two follow-up seminars (mid-term and final) will be organized to monitor projects´ performance.

The funded projects are intended to be assessed via questionnaires to all parties in funded projects:

1) After the completion of the projects 2) Two years after the end of the projects. Success is measured by: a) Technical results b) Scientific results c) Economic effects d) Transnational benefit c) Surveys, Statistical analysis report, presentation, discussion The follow-up of the projects is comprised of two reporting periods: Mid-Term and Final Progress reporting.

3.1. Project reporting and evaluation

The evaluation of project implementation will be performed through the Mid-Term and Final Progress Reports by the each National Founding Agency of project partners. . The Project Coordinators have to submit a Mid-Term and a Final Progress Report. The evaluation of the progress of the projects funded under the BlueBio 2018 JC will be performed in two steps:

Mid-Term Evaluation (remote evaluation and physical meeting: after mid-term deadline) - mid 2021

Final Evaluation Meeting for 36-month projects: – end 2022 or mid-2023.

The reporting and evaluation procedure for Mid-Term and Final Evaluation is summarized below:

Mid-Term and Final report templates will be designed and implemented in the online submission platform, for project coordinators to submit them at mid-term and final reporting periods (technical and financial information)

Call Steering Committee will be notified as soon as reports are uploaded.


Reports will be reviewed by selected members of the Call Evaluation Panel whose expertise lies in the field of the funded projects (or by external evaluators appointed by the CSC)

Outcomes of this reviews will be summarised and delivered to the CSC by AEI (task leader)

The Call Steering Committee will evaluate the progress of the projects based on the review of the Mid-Term and Final Progress Reports (templates to be provided later), according to the following criteria:

1. Scientific and technological progress; 2. Collaboration, coordination and mobility within the Consortium; 3. Coordination with other international projects funded by the BlueBio 2018 JC, or with

international projects previously funded by any call within COFASP or ERA-MBT; 4. Coverage of the themes and sub-themes of the call; 5. Stakeholder/industry engagement; 6. Transnational added value of the project; 7. Impact of the project (only in the final evaluation); 8. Dissemination of the results (publications, patents, other); 9. Identified problems or specific risks; 10. Recommendations for improvements/amendments of the report; and 11. Any other recommendations /feedback, which could be relevant to the Consortium.

3.2. Key Performance Indicators (KPIs)

Key Performance Indicator (KPI) Definition A Key Performance Indicator is a measurable value that demonstrates how effectively a company is achieving key business objectives. Organizations use KPIs at multiple levels to evaluate their success at reaching targets. High-level KPIs may focus on the overall performance of the business, while low-level KPIs may focus on processes in departments such as sales, marketing, HR, support and others. So what is the definition of KPI? What does KPI mean? What does KPI stand for? Here are a couple other definitions:

Oxford's Dictionary definition of KPI: A quantifiable measure used to evaluate the success of an organization, employee, etc. in meeting objectives for performance.

Macmillan's Dictionary definition of KPI: A way of measuring the effectiveness of an organization and its progress towards achieving its goals.

Why are used KPIs The aim of using KPIs is to measure the projects´ performance against key project objectives.

3.2.1 Logical Framework Approach (LFA) LFA an analytical tool to monitor and evaluate. LFA shows how activities can produce outputs connected to longer-term effects and eventually the realisation of the objectives (the impacts). The objectives and activities. LFA consists of the following steps

• The necessary inputs, in terms of financial and other resources. • The direct outputs of the activities including relevant indicators. • The indirect outcomes of the activities. • The wider societal impacts of the activities

https://en.oxforddictionaries.com/definition/key_performance_indicator

https://www.macmillandictionary.com/dictionary/british/key-performance-indicator


Inputs Activities Outputs Early Impacts (Outcomes)

Impact (Targets)

- National Funding - EC Funding - Own Funds

- Implementation of collaborative research project - Capacity Building; - Cooperation for development of new research themes between researchers involved in the projects;

Technical -New /improved method/methodology/protocols ; Dissemination -Peer-reviewed publications; -Scientific reports; -success stories/good practices guide; -Presentations at meetings/conferences/ brokerage events; -Websites/ Newsletters/ Videos; New cooperation networks

Valorisation of project outputs - to uptake the project outputs by the industry& public sector; - to commercialize the new tools/ technique/ products/services; Strengthens the ERA -New and deepened cooperation and networks among researchers

Scientific - Citations/cooperation with research community Economic: value chain approach - to develop new /improved products and services in industry - to create jobs; - to support the economic growth - Policy; -to facilitate a dialogue between scientists, , industry, policy makers, NGO's and end-users

The Value Chain Concept/Approach

The focus of the collaborative projects within the BlueBio ERA-NET Cofund will be to

generate knowledge for new, integrated value chains from primary production to processing generating innovative products and services within the bioeconomy.


The generation of knowledge about the potential in the Blue Bioeconomy to promote sustainable and economically optimised exploitation and management of captivated/harvested and cultivated aquatic food and non-food biomass including waste

streams will be achieved. Activities will examine new value chains within the circular economy, novel uses, as well as developing and refining exi sting methods and technologies to facilitate a dialogue between scientists, information and service providers, industry, policy makers, NGO's and end-users.

The relationship with the wider bioeconomy

One of the overarching aims of BlueBio is to reduce fragmentation of research and innovation activities, knowledge and skills in the Blue Bioeconomy. Its ability to do so has

already been demonstrated in bringing together the networks, strategies and experiences of JPI Oceans, ERA-MBT and Cofasp. It has further expanded the Consortium by bringing in international partners, regional partners and partners closer to the market. BlueBio will seek a dialogue with other European Industrial networks (e.g. BBI) and align regional activities

with plans, finances and strategies into the project e.g. Baltic, Northern Germany, Western France, Western Sweden, Adriatic-Ionian Region. For additional calls, international aspects will be addressed where relevant. Participation of international partners in the calls will be

welcomed where beneficial and will engage external networks such as AORAC, EURASTIP etc.


Impact Assessment

Activities Outcomes KPIs Source of information

Create, test, upscale and bring to the market new knowledge-intensive products and services derived from aquatic biomass

Added-value goods and services: - increasing the quality and value of products by product innovation and differentiation

- no of functional foods/ ingredients, nutraceuticals, cosmeceuticals, fine chemicals, enzymes and other biomaterials derived from marine organisms; sensors ; -no of biological indicators -no of new jobs created;

-no of SME involvement -TRL (start and the end) level)

Questionnaires

Final Report

Provide consumers and policy makers with the knowledge needed to make informed decisions

- consumer informed about the sustainable products;

- knowledge provided to support policy decisions

-no of industrial partners involved in the projects;

-no of participation in brokerage event;

- no of marine technology transfer

Final Report

Questionnaires

Increase the efficient and sustainable use of by-products

- use of biomass/ underutilized material/ aquatic by-products for human benefit

-no of new product/ services

Questionnaires

Contribute to improving the professional skills and competences of those working and being trained to work within the blue economy

-training on: IT literacy, environmental and regulation issues, soft skills (time management tools, communication, problem solving, entrepreneurship, etc.);

-mobility of personnel involved in the project

-no of trainings;

-no of researchers participating in the training

-no of researchers participating in mobility

Questionnaires

Contribute to policymaking in research, innovation and technology

-support communication between researchers, producers, consumers and policy makers

-no of stakeholders informed/involved

Questionnaires


The BlueBio 2018 JC funded projects will be monitored during their lifetime using the following indicators:

1. Work performed and the results achieved within the project 1.1. Scientific and technological progress 1.2. Collaboration, coordination and mobility 1.3. Impact and knowledge output 1.4. BlueBio 2018 JC topic(s) addressed by the project 1.5. Transnational added value of the project

2. Deliverables 3. Consortium meetings 4. Stakeholder engagement 5. Socio-economic impact statement 6. Knowledge output transfer 7. Publications and other outputs (i.e. patents, licences, policy-oriented briefs, etc.)

In relation to Open Data, as mentioned in the CA documents, the funded projects will be requested to submit metadata on all the resources directly generated by the project, as well as additional information on how this data will be used, if and how the data will be made accessible for verification and re-use, and how the data will be curated and preserved. Metadata on all project resources are required to be submitted as part of the final reporting. In addition, in collaboration with WP7-T7.3, a specific questionnaire was developed, aimed at delivering more detailed information about sections 5 (Stakeholder engagement) and 6 (Socio-economic impact), complementing the proposed KPIs listed below. A draft version of the questionnaire, which is supposed to be submitted to the project managers for its completing using the same timeline requested for the other KPIs (i.e., mid-term and final report), is also attached at the end of the present report (see ANNEX 1). It is expected to be administered independently as it is, even though its integration in the common template for mid-term and final reporting is presently under evaluation.


List of proposed Key Performance Indicators (KPIs) - BLUEBIO FUNDED PROJECTS

Type of KPI Proposed KPIs Source of information

1. Work

performed and the results achieved

within the project

1.1.Scientific and technological progress Mid-term and final report

1.2.Collaboration, coordination and mobility Mid-term and final report

1.3.Number of partners involved in funded project Mid-term and final report

1.4. Number of partners’ types (enterprise, academia, research organizations etc.)

Mid-term and final report

1.5.Experience in working together within the consortium, (e.g. None or working together for the

first time; or yes, part of the consortium; or yes the whole consortium)


1.6. Number of new collaborations outside the consortium during the lifetime of the project, due to project results (If yes, with other COFASP or ERA-MBT project partners?)


1.7.Impact and knowledge output Mid-term and final report

1.8.Preparation of applications to other Calls by the consortium due to Bluebio funded project results


1.9.Development of a method / process / product / service (Time frame for commercialization of results (if applicable)/ how has it progressed throughout

project)


1.10.BlueBio 2018 JC topic(s) addressed by the project


1.11.Transnational added value of the project Mid-term and final report

2.Deliverables

2.1. Number of deliverables and milestones completed on time


3.1.Number of publications/impact factor/citation rate



3. Publications and other outputs (i.e. patents, licences,

policy-oriented briefs, etc.)

3.2.Number of new patents/licensing/ patent applications


4.Consortium meetings 4.1.Participation in conferences and other international scientific events


4.2.Participation in meetings (by at least two partners) ex. consortium meetings, workshops, etc.


5.Stakeholder engagement - HCB

5.1. Researchers exchange (e.g., PhD students, Post-

Docs, etc.): number of people, duration. show exchanges within partner type (e.g. academia to academia) and to a different type (e.g. academia to industry)


6.Socio-Economic Impact

6.1. Number of young researchers included in the teams (Post-docs for 3 years or less)


6.2.Number of new contracts/fellowships for researchers


6.3. Number of master or PhD thesis Mid-term and final report

6.4.Optimization and usage of research infrastructures


6.5.Input to standards (white papers/ regulations) Mid-term and final report

7.Knowledge output

transfer – Raising awareness – policy on

open data

7.1.Optimization of resources – data sharing


7.2.Optimization of resources – material exchange


7.3.Raising awareness in society (Outreach activities)


7.4. Contacts with stakeholders/industry (% of research projects that report real added value by an open science mechanism)



ANNEX 1 - Tool for monitoring and assessing training and mobility and HCB activities (v. 2.0)

Draft questionnaire to be used as Performance Indicator for training and mobility activities embedded in BlueBio funded projects

Are training and mobility issues addressed in your project? Yes No

If yes, please specify:

Training , Mobility , both

How training and mobility issues are addressed in your project?

Please specify (if any):

Training activity: ………………………

Mobility: ………………………

Use of large-scale facilities (e.g. EMBRC)? Yes No

Units of personnel involved (and corresponding man months) by partner type (Public or Private) and by qualification (researchers, technicians, etc.), and exchange between Public and Private partners.

Produce one table for each activity sector involved (e.g., “fisheries”, “aquaculture”, “seafood processing”, “biotech”, etc.)

Qualification

No. of Units Man months

Private sector

(Total)

Public sector (Total)

From Private to Public

From Public to Private

Private sector

(Total)

Public sector (Total)

From Private to Public

From Public to Private

Technicians

Administrative staff

Researchers

Post-docs

PhD Students

Graduate Students

Others

Any certification of the training activity is planned to be delivered?

Yes , No

If yes, specify which kind of certification and duration (if applicable):


Has the training initiative embedded in your project been planned to be uploaded in the MarineTraining.eu portal and/or similar platforms?

Yes , No

If not, are the training initiatives embedded in your project going to be uploaded in the MarineTraining.eu portal and/or similar platforms?

Yes , No

Are the project activities aligned with the principles of Responsible Innovation, creating value for society in an ethical and responsible way?

Yes , No

How Responsible Innovation is embedded in your proposal (in the research and innovation process)?

Please specify:

……………………..

Date post:	17-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Report on the methodology and list of KPIs for the ......In addition, guidelines delivering...

Documents