A concept for inferring « Frontier Research » in research project proposals

A concept for inferring « Frontier Research » in research project

proposals

Dirk HolsteEdgar Schiebel

Thomas ScherngellMarianne Hörlesberger

Ivana RocheDominique BesagniClaire FrançoisPascal Cuxac

ISSI 2011 – Durban, South Africa, July 4-7

ISSI 2011 - A concept for inferring "Frontier Research" in research project proposals - Durban, South Africa, July 4-7

2A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011

- Durban, South Africa, July 4-7

2

Plan of the presentation

• Introducing the concept

• The ERC framework

• Scientometrics & Text mining

• Frontier research & Indicators

• The indicators

• Proposal selection function

• Discussion

A concept for inferring "Frontier Research" in research project proposals - Holste et al. ISSI 2011 - Durban, South Africa, July 4-7

3

Introducing the concept

Concept for inferring attributes of frontier research in peer-reviewed research project proposals under the scheme of the European Research Council (ERC)

Design, implement, test and refine a bibliometric/scientometric model that investigates whether submitted and/or selected grant applications fulfill the requirements of frontier research as this notion

was enounced by the High Level Expert Group of the ERC

Build and compare outcomes of the model with the review decision in order to obtain further insight and to bring a reflection on the influence of frontier research in the peer-review process


4

The ERC framework (1/4)First European funding body to support investigator-driven (frontier) research through open and direct competition

• Main goals:– Scientific excellence as the only scientific selection criteria– Major grants for the truly best and creative researchers and

their ideas, to identify and explore new opportunities and directions in all fields of research

• Scientific domain clusters (panels):– Physical and Engineering Sciences (PE) 10 panels– Life Sciences (LS) 9 panels– Social Sciences and Humanities (SH) 6 panels

• Grant Application schemes:– Starting independent researcher grants (StGs)– Advanced investigator grants (AdGs)

proposals submitted by principal investigator (PI)to every project is allocated a main panel that becomes its

“home discipline”


5

The ERC framework (2/4)

• StGs Up-and-coming investigators: bring means and opportunities for young investigators to develop independent careers

• AdGs Established senior researchers: foster pioneering, far-reaching research endeavors, with high-risk/high-impact potential, break of established disciplinary boundaries or exploration of new productive lines of enquiry, methodology or techniques

• Funded through EU’s Framework Programme FP7

• Annual calls: 1st in 2007


6

0

300

600

900

1200

1500

1800

20072008

20092010

20112012

2013

Mio

. €


• ERC annual budget evolution (2007-2013):

• Rate of selected proposals:– StGs (2009) 10% (244 out of 2,503 submitted proposals)– AdGs (2009) 15% (244 out of 1,584 submitted proposals)

• Selection process:– 25 expert panels– members selected by the Scientific Council– two-step peer-review process


7


The issue of the frontier research for the ERC the ERC’s High Level Expert Group enounced key attributes of frontier research:

Standing at the forefront of creating new knowledge… responsible for fundamental discoveries… achieving occasional revolutionary

breakthroughs

Intrinsically risky

Reducing the barriers between basic and applied research

Pursues questions irrespective of established disciplinary boundaries

The peer-review process shall indentify such activities /researchers out of an immense number of submitted

applications


8

Scientometrics & Text mining

• Identification of 4 key attributes according to the understanding of the ERC High Level Expert Group’s notion of frontier research:

• Novelty• Risk• Applicability• Interdisciplinarity

• Formalisation of the key attributes in terms of indicators build by combining scientometric and text mining approaches:


9

Frontier research & Indicators

Gra

nt A

pplic

ation

sSt

artin

g (S

tGs)

or A

dvan

ced

(AdG

s) g

rant

s

Data related to• Principal investigator

Data related to• Research project

Data related to• Research environment

ERCdatabase

Externaldatabases

Data extraction, text-mining

Risk(independence, originality/ uniqueness)

Novelty (1-timeliness: citedbibliography;2-proximity: nearto emergingresearchareas)

Interdisciplinarity(acrossdisciplinarybarriers)

Pasteuresqueness(potential applicability)

Comparison: Istherecongruencebetweenourresultsand peer-reviewdecision? Are keyattributescorrectlyrepresented?

Scientometricand

content-basedindicators

Peer

-rev

iew

Ste

p2Selectionfunctionmodel: discretechoicemodelbasedon indicators

Peer

-rev

iew

Ste

p1

Corr

espo

nden

ce


10

Timeliness indicator

It is used to infer the « innovative degree » of the proposal through the bibliographic references cited by the PI in his(her) proposal

• Data source: ERC data

• Hypothesis: – the more recent references are, the more likely the

work is at the frontier research

• Calculation:– this indicator focuses on the time elapsed since the

publication of the proposal cited references and Timeliness is obtained from the difference between the year of the proposal submission and the year of publication of each cited reference in the proposal


11

Proximity indicator• It is employed to infer the « innovative degree » of the

proposal through the dynamic change of the scientific landscape corresponding to the proposal’s allocated panel

• Data source: ERC data & bibliographic DB

• Hypothesis:– the closer a proposal is to regions of positive dynamic change,

the more innovative it is

• Calculation:– raw data are obtained from bibliographic DB and ERC panel

description to identify and extract discriminating terminological information

– a clustering step produces a cluster map that groups similar references and represents the publication landscape corresponding to the considered panel

– a diachronic analysis is used to study the evolution of the landscape between two time periods

– clusters are ranked with respect to its innovativeness and the similarity of each proposal to its nearest clusters is calculated


12

Risk indicator

• It is used to infer the « personal risk » of the PI in executing the proposed research

• Data source: ERC data & bibliographic DB

• Hypothesis: – references serve as one informative source for getting a

« bibliometric research profile »– the lower the overlap between two reference profiles,

the more risky is the proposal for the PI

• Calculation:– comparison between the profile of the proposal and, on

one hand, the profile of the past research of the PI and, on the other hand, the profile of the research work in the « home » panel of the proposal


13

Pasteuresqueness indicator

• It is used to infer the applicability of expected results of the proposal by considering evidence for immediate or intended application

• Data source: ERC data & bibliographic DB & patent DB

• Pasteuresqueness… What does it mean?!– this neologism comes from the Pasteur’s Quadrant

which gives a categorization of scientific issues that seek both fundamental understanding and social benefit

• Calculation:– patents granted by PI– information related to industry involvement– self-citations published in journalswith prescribed « applicability » Rele

vanc

e fo

r the

adv

ance

men

t of

know

ledg

e

Pure basic Research

(Bohr)

Use-inspired basic research

(Pasteur)

Pure applied research

(Edison)

Frontier Research

Relevance for immediate applications

Rele

vanc

e fo

r the

adv

ance

men

t of

know

ledg

e

Pure basic Research

(Bohr)

Use-inspired basic research

(Pasteur)

Pure applied research

(Edison)

Frontier Research

Relevance for immediate applications


14

Interdisciplinarity indicator

• It is used to infer self-consistently the presence, in the proposal, of characteristic terminologies which belong to several ERC panels different than the proposal’s « home » panel

• Data source: ERC data

• Hypothesis: – the higher the occurrence of keywords belonging to different

ERC panels, the more interdisciplinary the proposal is considered

– the higher this number of different involved ERC panels, the more interdisciplinary the proposal is considered

• Calculation:– keyword labelling according to its statistical frequency of

occurrence across all panels– assessment of the concentration in the proposal of keywords

labelled as belonging to different ERC panels


15

Proposal selection function (1/2)We use a statistical discrete-choice model (DCM) to estimate the influence of indicators on the selection probability of a research project proposal to be funded

Statistically, we model a dependent variable that takes on the form:

1

0 otherwise

i

proposal is acceptedY

i , j = 1, … , n

The selection probability of a proposal to be accepted, Pr(Yi = 1), depends on the set of indicators summarized in a matrix X such that

Pr( 1) ( )

Pr( 0) 1 ( )

i

i

Y F

Y Fi

i

X

X

where Xi is a set of k observed factors for proposal i , β is the estimated k-by-1 parameter vector reflecting the impact of changes in Xi on the probability Pr(Yi = 1)


16

Proposal selection function (2/2)

The logistic distribution is chosen for model estimation leading to the empirical model given by:

The parameter estimation is based on Maximum-Likelihood procedures.

expPr( 1) ( )

1- exp iY i

ii

XX

X


17

Discussion

The developped concept…

• is ambitious difficult, by employing bibliometric methods, to faithfully represent attributes of frontier research and validly quantify them to evaluate the grant decision

• cannot substitute expertise intends to bring us a bibliometric model in which indicators are expected to have a positive effect on the decision probability for ERC grant applications

• does not rely on performance measures combination of content analysis and scientometric methods using textual information present in the proposals

• can be reused focused on the ERC grant scheme but the concept might be applicable more generally

• results in a methodology allowing the grant agency to monitor, from a bibliometric perspective, the peer-review process and providing a basis for its further refinement


18

Acknowledgements

This work was partially funded by the « Ideas » specific Programme of the EU’s 7th Framework Programme for Research and Technological Development (project reference no. 240765)

Project website: http://www.ait.ac.at/dbf


19

[dirk.holste; edgar.schiebel; thomas.scherngell; marianne.horlesberger]@ait.ac.at[ivana.roche; dominique.besagni; claire.francois; pascal.cuxac]@inist.fr

Thank yououy knahT

ISSI 2011 - A concept for inferring "Frontier Research" in research project proposals - Durban, South Africa, July 4-7

20


21

Timeliness indicator (2/2)

ERCdatabase Research

project proposal

Empirical distribution of

publication dates

Extraction ofpublication date

of references

Calculation of TIMELINESS

indicator

Data pre-processing and text-mining


22

Proximity indicator (2/2)

ERCdatabase

Translate main panels into

database queries


Panel description

Bibliographic (PASCAL) database

query

DB of references

DB of references

Construction of two indexed corpora

(time window T1, T2)

Diachronic cluster analysis

T1

T2

Ranking of clusters by

innovativenessT1,T2

Calc

ulati

on o

f PR

OXI

MIT

Yin

dica

tor

ERCdatabase Data

from proposals

Extraction of terminological

information

Position of proposal in T2

cluster map


23

Risk indicator (2/2)

ERCdatabase

References of PI’spast research

Externaldatabases

References of PI’sresearch proposal

Reference profile in the field of the

submitted proposal

Distance between references (d1)

Distance between references (d2)

Calc

ulati

on o

f RI

SKin

dica

tor

Data pre-processing and parsing


24

Pasteuresqueness indicator (2/2)

ERCdatabase


Externaldatabases

Patents citing PI’s work

Patents submitted

by/granted to PI

Grants, funding, acknowledgments of

collaborations

List of journals publishing

List of self-citations (PI)

Date pertinent to PI’s

curriculum vitae

Journals and

their scopes

Classification of “fundamental” and

“applied” publications

Calc

ulati

on o

f PA

STEU

RESQ

UEN

ESS

indi

cato

r


25

Interdisciplinarity indicator (2/2)

ERCdatabase

Extraction of ERC panel

structure


Calc

ulati

on o

f IN

TERD

ISCI

PLIN

ARIT

Yin

dica

tor

Matching of proposals with panelsand analysis of field

diversity

Extraction of keywords

from proposals

Matching keywords to some “home

discipline”

Analysis of concentration of

“home keywords”in proposals

Date post:	21-Jan-2016
Category:	Documents
Upload:	ginata
View:	43 times
Download:	0 times

A concept for inferring « Frontier Research » in research project proposals

Documents