BONUS impact on policies,
industries and structuring of the
macro-regional research area
Professor Mike Elliott* and Steve Barnard,
Institute of Estuarine & Coastal Studies,
University of Hull,
Hull, HU6 7RX, UK
(* BIREME Review Chair 2006-7; BONUS ERA-NET project evaluation & selection panel 2008;
BONUS+ Evaluation Panel 2012, speaker at BONUS project workshops, etc.)
Ecosystem services
of the Baltic Sea.
Underlying
ecosystem services
important for fish as
food are marked with
yellow arrows and
illustrate linkages
between different
ecosystem
services and benefits
(From SwAM 2013, in
Scharin et al., 2016)
Overarching aims:
The analysis is:
• an impartial and objective overview of the success of the BONUS
programme to date,
• to provide insight into future improvements to the BONUS
programme and similar programmes.
It is not:
• a judgement on the development, management or delivery of any
specific BONUS project.
‘The ambition of the proposers!’
Objectives and approach
BONUS delivery:
1. consider the BONUS programme impact on the 11 BONUS
programme challenges;
2. address the scope of the wider BONUS programme, and
3. review selected recorded performance metrics.
Views of BONUS:
4. assess how stakeholders see the impact(s) of BONUS – in
the context of defined ‘Evaluation Questions’
Phase CallYear
of call
Call details
(hyperlink)
No of
projects
funded
Implementation
period
BONUS+ n/a 2007 tinyurl.com/gle7wlo 16 2009-2011
BONUS 185
Viable
Ecosystems2012 tinyurl.com/j4rglxg 7 2014-2017/18
Innovation 2012 tinyurl.com/hbn7ysg 13 2014-2016/17
Sustainable
Ecosystem
Services
2014 tinyurl.com/zfocjb4 8 2015-2018
Blue Baltic 2015 tinyurl.com/jkg6krr 12 2017-2020
BONUS project calls and themes
A portfolio of BONUS VE, INNO and SES projects descriptions (incl. themes
addressed):
http://www.bonusportal.org/files/4526/2015_12_02_BONUS_projects_portfolio.pdf
http://www.tinyurl.com/gle7wlohttp://www.tinyurl.com/j4rglxghttp://www.tinyurl.com/hbn7ysghttp://www.tinyurl.com/zfocjb4http://www.tinyurl.com/jkg6krrhttp://www.bonusportal.org/files/4526/2015_12_02_BONUS_projects_portfolio.pdf
Delivery against challenges – approach:
• Descriptions of Work and Research Plans used to indicate range of
intended or potential project outputs or products
• Each project/challenge combination assessed on four point scale:
• Presented as ‘heat map’ and interpreted visually
Value judgement regarding project outputs†
Judgement score
Project outputs not likely to be relevant to the challenge, or not likely to contribute to addressing the challenge
0
Project outputs likely to have a low relevance or value in terms of addressing the challenge
1
Project outputs likely to have a moderate relevance or value in terms of addressing the challenge
2
Project outputs likely to have a high relevance or value in terms of addressing the challenge
3
BONUS Challenges
1. Relevant policies & collective
governance
2. Adapting to sustainable living
3. Adapting to climate change
4. Restoring GES
5. Mitigating eutrophication
6. Sustainable use: ecosystem goods and
services
7. Planning use of marine space
8. Fisheries management: ecosystem
stability & stock reproductive capacity
9. Safe maritime traffic
10. Minimising environmental threat (due to
chemicals exposure)
11. Environmental information system
BONUS+ - addressing research themes:
distributionReview original Research Plans (BONUS+) or Descriptions of
Work (BONUS 185) to identify key themes and sub-themes;
assumed that all themes had equal weighting
Theme
BONUS+ projects 1: L
inki
ng
Scie
nce
an
d P
olic
y
2: U
nd
erst
and
ing
Clim
ate
Ch
ange
and
Geo
ph
ysic
al F
orc
ing
3: C
om
bat
ing
Eutr
op
hic
atio
n
4: A
chie
vin
g Su
stai
nab
le F
ish
eri
es
5: P
rote
ctin
g B
iod
iver
sity
6: P
reve
nti
ng
Po
lluti
on
7: I
nte
grat
ing
Eco
syst
em a
nd
Soci
ety
AMBER sub sub sub sub
key
BALTIC-C key key key
key key
HYPER key key key
key
key
BALTGENE key key
key key
key
BALTICGAS key key key
key
RISKGOV key
sub sub sub sub sub
BAZOOCA key key key key key
key
RECOCA key key key
key
INFLOW
key
key
BEAST key sub sub sub sub key
ECOSUPPORT key key key key sub sub key
BALCOFISH
sub sub
sub key
IBAM sub sub sub sub sub sub sub
PROBALT key
sub
key
PREHAB key sub sub sub sub sub sub
BALTICWAY key sub
sub sub sub key
Themes
1. Linking science & policy
2. Understanding climate change and
geophysical forcing
3. Combating eutrophication
4. Achieving sustainable fisheries
5. Protecting biodiversity
6. Preventing pollution
7. Integrating ecosystem & society
BONUS 185 - addressing research
themes: frequenciesBONUS 185 themes
1.1Dynamics of biogeochemical processes
1.2Changing biodiversity
1.3Food web structure & dynamics
1.4Impacts of hazardous substances
2.1Changes in catchment land cover patterns
2.2The role of the coastal systems
2.3Integrated coastal management
2.4Eco-technological approaches
3.1Maritime risk analysis & management
3.2Effects of air and water pollution by shipping
3.3Improving stock assessments, spatial heterogeneity of stocks
3.4Evaluation framework for fisheries management
3.5Sustainable aquaculture in the Baltic Sea
4.1Governance structures, performance & policy instruments
4.2Linking ecosystem goods & services to human lifestyles & well-being
4.3Maritime spatial planning
5.1Integrated monitoring programmes
5.2Innovative measurement techniques
5.3User-driven ICT services
But (and there is always a ‘but’):
- “More is not always better”
• A project centred on just a small number of
themes should not be automatically viewed as
being weak;
• A single project that focuses solely on one priority
area, adding important knowledge on that topic,
can be highly valuable;
• This should lead to the cumulative/progressive
benefits of successive calls.
Bonus challenge:
Statistic BONUS
research call(s) Nu
mb
er
of
pro
ject
s co
nsi
de
red
Re
leva
nt
po
licie
s &
co
llect
ive
gove
rnan
ce
Ad
apti
ng
to s
ust
ain
able
livi
ng
Ad
apti
ng:
clim
ate
ch
ange
Re
sto
rin
g G
ES
Mit
igat
ing
eu
tro
ph
icat
ion
Sust
ain
able
use
: e
cosy
ste
m g
oo
ds
&
serv
ice
s
Pla
nn
ing
use
of
mar
ine
sp
ace
Fish
eri
es
man
age
me
nt:
eco
syst
em
stab
ility
& s
tock
re
pro
du
ctiv
e c
apac
ity
Safe
mar
itim
e t
raff
ic
Min
imis
ing
en
viro
nm
en
tal t
hre
at(d
ue
to c
he
mic
als
exp
osu
re)
Envi
ron
me
nta
l in
form
atio
n s
yste
m
Sequential performance Summary statistics provided represent the maximum score assigned across all projects
in the call
BONUS+ only 16 3 2 3 2 3 3 2 2 2 3 1
Viable Ecosystems only 7 2 1 3 2 3 3 2 3 2 3 1
Innovation only 13 3 3 3 3 3 3 3 3 3 3 3
Sust. Eco. Services only 8 2 1 1 2 3 2 3 3 3 3 0
Cumulative high-level performance Summary statistics provided in format X/Y where X is number of projects assigned a
(maximum) score of 3, and Y is total number of projects considered
I only 16 2/16 0/16 1/16 0/16 1/16 1/16 0/16 0/16 0/16 2/16 0/16
I & II together 23 2/23 0/23 2/23 0/23 3/23 2/23 0/23 1/23 0/23 3/23 0/23
I, II & III together 36 5/36 5/36 8/36 6/36 8/36 3/36 3/36 2/36 4/36 9/36 7/36
I, II,III & IV together 44 5/44 5/44 8/44 6/44 10/44 3/44 5/44 3/44 5/44 11/44 7/44
Key
I refers to the BONUS+ call, II refers to the Viable Ecosystems call, III refers to the Innovation call; and IV refers to the Sustainable Ecosystem Services call
1 At best, from across the full range of projects considered, the proposed project outputs are likely to have only a low relevance or value in terms of addressing the challenge
2 At best, from across the full range of projects considered, the proposed project outputs are likely to have only a moderate relevance or value in terms of addressing the challenge
3 The proposed project outputs are likely to have a high relevance or value in terms of addressing the challenge
x/y At least one project of the cumulative total number of projects under consideration is likely to have a high relevance or value in terms of addressing the challenge (where x=number of projects with high relevance, and y= total number of projects considered)
Learning exercise:
Q. Did the ability to
cover the BONUS
challenges increase
with successive
calls?
A. Yes.
Performance metrics:
Subset of 15 metrics assessed, grouped according to:
• Input by, or value of, the BONUS projects to regulatory management,
and to the development and implementation of policy & process (two
statistics)
• Level of stakeholder engagement across the BONUS projects (three
statistics)
• Prevailing level of research integration between EU states (three
statistics)
• Level of accessible, public dissemination achieved by the projects (four
statistics)
• Academic links developed and promoted by the projects (three statistics)
Stakeholder Responses – ‘from the horse’s mouth!’:
• On-line questionnaire approach but potential for bias;
• Low but useable number of respondents from all countries;
• The participants group were mainly academic-based researchers; those from the
funders group mainly government-funded agencies with roles relating to
management, funding or policy implementation;
• Respondents from the users group were mainly policy-based workers in
government roles;
• All groups showed awareness but funders had higher level of awareness than
users.
Stakeholder views (i)
Responses to statement: Par
tici
pan
ts
Fun
der
s
Use
rs
A: The extent to which science-based thinking and argument is used and incorporated into the policy-making process has increased - B: The perceived level of scientists’ direct involvement in the provision of advice at the policy-making level has improved -
C: The speed of uptake of scientific knowledge into policy-making and management has increased - D: Policy-making and governance have evolved from insular, sectoral processes to ones that are now more integrated and cross-sectoral - E: Cooperation (and coordination) between the funders of academic research and the funders of innovative industries has increased -F: Funding, management and implementation of research has shifted from an insular, predominantly national model to an increasingly transnational and integrative model -G: The level of coordination and integration between the funding organisations for both research and innovation across EU Member States in the Baltic region has increased -H: The incidence of collaborative input from private enterprise (as partners/contributors) into research projects has increased
I: The direct involvement of other stakeholders in scientific research (such as by contributing to research activities) has increased
J: Stakeholders and potential knowledge-users (for example, policy-makers, or innovative industries) have become more involved in defining research agendas
K: The contribution of academia to the development of innovative industries has increased
Stakeholder views (ii)
Responses to statement: Par
tici
pan
ts
Fun
de
rs
Use
rs
L: Since 2009, has the BONUS programme had a positive impact on the joint use of research infrastructure by scientists of different Member States? M: Since 2009, has the BONUS programme had a positive impact on the cost-efficiency of research? N: Since 2009, has the BONUS programme had a positive impact on the development of integrated research, governance and management structures at the European sea-basin scale? O: Since 2009, has the BONUS programme had a positive impact on the development and harmonisation of research management practices in the participating states?
Key to symbols used
Overall opinion of >75% of those respondents that expressed a view is that a positive impact (either slight, moderate, high, or very high) has been seen since 2009 Overall opinion of >50% of those respondents that expressed a view is that a positive impact (either slight, moderate, high, or very high) has been seen since 2009
Overarching conclusions(i): Meeting the challenges
• Despite the variability, the potential for the combined suite of projects commissioned under the
BONUS+ and BONUS 185 calls to successfully address the BONUS challenges is very high and
robust.
(ii): Addressing wider scope of BONUS programme
• BONUS+: coverage of research themes by projects reasonably robust - all themes show good
level of redundancy (each theme by >8 projects);
• BONUS 185: coverage of research themes appeared more variable - some imbalance across the
themes, all themes are addressed, the scope is robust and there is some potential redundancy
for all but one theme (3.4: Evaluation framework for fisheries management)
• Consolidation of delivery across themes been demonstrated through successive calls
(iii): Review of performance metrics• Shortfalls in the data (availability, potential biasing factors) restricted the analysis and needs
refining but there are clear differences between the projects regarding their relative success in
addressing the performance metrics
But (and there is always a ‘but’):
• Is publication the same as
dissemination?
• Do scientists focus on outputs rather
than outcomes?
• Is there a difference in natural and social
science dissemination?
• Do scientists focus on single topics and
policy-makers on multiple topics?
• Are project outputs superseded by
Member State activities?
• Do scientists give policy makers the
‘nice-to-know’ rather than the ‘need-to-
know’?
• Is there a risk of over-promising and
under-delivering?
Who uses what information?
Who? What?
General public News items on TV, radio and newspapers, social
media, Tweets, pub! (2 minutes attention span, 140
char.)
U/g Students Lecture notes, websites, Wikipedia (few pages),
PhD and Early career
scientists
Books, reviews, websites, abstracts, compendia,
Wikipedia, journals (detailed syntheses)
Experienced scientists International journal, peer–reviewed articles (15 pages)
Consultants Consultant reports, government documents (200
pages)
Policy implementers Government and agency reports, commissioned
consultant reports (
And more elephants:
• Does policy development work with
incremental gain rather than major
breakthroughs?
• Would Impact Statements per project (á
la UK REF) be of benefit?
• How to separate project dissemination
from informal dissemination by
researcher networks and hinterlands?
• Are sufficient researchers interested in
using rather than just producing their
information?
• Are natural scientists willing to engage
in societal aspects?
• Do scientists need more training in
engagement?
Recommendations:
• To consider what are ‘stronger’ projects (e.g. most from the Innovation call)
doing to score so well in terms of addressing the challenges.
• There is the need, once when all projects have been completed, for a
retrospective assessment of final delivery against the challenges and
further refinement of statistics collected.
• To ask for a specific and evidence-based narrative by the projects,
otherwise it is difficult to see further gains for policy.
• Project performance measures are valuable but need to check if they
should have equal weighting; the project performance should be gauged
against: the stage (maturity), size and nature of the project, and the
reporting period.
• Pre-engagement with stakeholders would be beneficial/valuable/necessary.
Final comments:
• The analysis vindicates the selection of projects and shows that the
selection of themes was very relevant and delivery against the themes is
achieved, albeit sometimes not balanced and so there is a risk of a theme
not being fully addressed;
• There is significant strength-in-depth provided by the range of projects that
have been, or are planned to be, commissioned;
• Further analysis of projects with multiple outputs would be beneficial;
• Overall, stakeholder views support a strong positive impact on three key
areas:
- the application of science to policy development;- changes in funding patterns both for research and for the development of
innovative industries, and
- the involvement of different actors.
Advertising:
Open Access
book:
With a huge thanks to all at
BONUS Secretariat and to all the
respondents
(those who didn’t respond can’t
complain their views were not
taken into account!!)
mailto:[email protected]:[email protected]
