European Peer Review Guide

8/4/2019 European Peer Review Guide

1/88

Member Organisation Forum

European Peer Review GuideIntegrating Policies and Practices

into Coherent Procedures


2/88

European Science Foundation

The European Science Foundation (ESF) is an

independent, non-governmental organisation, the

members of which are 78 national funding agencies,

research performing agencies, academies and learned

societies from 30 countries. The strength of ESF lies

in the inuential membership and in its ability to bring

together the different domains of European science inorder to meet the challenges of the future.

Since its establishment in 1974, ESF, which has its

headquarters in Strasbourg with ofces in Brussels

and Ostend, has assembled a host of organisations

that span all disciplines of science, to create a

common platform for cross-border cooperation in

Europe.

ESF is dedicated to promoting collaboration in

scientic research, funding of research and science

policy across Europe. Through its activities and

instruments ESF has made major contributions to

science in a global context. The ESF covers the

following scientic domains: Humanities

Life, Earth and Environmental Sciences

Medical Sciences

Physical and Engineering Sciences

Social Sciences

Marine Sciences

Materials Science and Engineering

Nuclear Physics

Polar Sciences

Radio Astronomy

Space Sciences

www.es.org

Cover picture: iStock

Member Organisation Fora

An ESF Member Organisation Forum is an output-

oriented, issue-related venue for the Member

Organisations, involving other organisations as

appropriate, to exchange information and experiences

and develop joint actions in science policy. Typical

subjects areas discussed in the Fora are related to:

Joint strategy development and strategic cooperationwith regard to research issues of a European nature.

Development of best practices and exchange

of practices on science management, to benet

all European organisations and especially newly

established research organisations.

Harmonisation of coordination by MOs of national

programmes and policies in a European context.

Acknowledgements

ESF is grateful to the Forum members and observers,as well as for the special contribution of Cristina

Marras (seconded to the ESF by CNR) and

Farzam Ranjbaran (ESF) for preparing the Guide.

The MO Forum has been chaired by Marc Heppener

(ESF) and coordinated by Laura Marin (ESF).

March 2011ISBN: 978-2-918428-34-3

Printing: Ireg Strasbourg


3/88

Contents

Foreword 3

Part I: Overview o the Peer Review System 5

Chapter 1: Introduction 7

1.1 Key denitions 8

1.2 Applicability 8

1.3 How to use this Guide 8

Chapter 2: Typology o unding instruments 9

2.1 General description of main funding instruments 10

2.2 Variants of funding instruments 10

Chapter 3: Pillars o good practice in peer review 12

3.1 Core principles of peer review 12

3.2 Integrity of the process of peer review 13

3.3 Quality assurance 15

3.4 Governance structure 153.5 Methodology 16

Chapter 4: Peer review methodology 17

4.1 Preparatory phase 17

4.2 Launch of the programme 20

4.3 Processing of applications 21

4.4 Selection and allocation of experts 22

4.5 Reader System 27

4.6 The use of incentives 27

4.7 Expert assessments 27

4.8 Final decision 31

4.9 Communication 34

4.10 Quality assurance 354.11 Variants of funding instruments and their implication for Peer Review 36

4.12 Peer Review of monodisciplinary versus pluridisciplinary research 38

4.13 Programmes explicitly designed for breakthrough research 44

Part II: Guidelines or Specifc Funding Instruments 47

Introduction to Part II 48

Chapter 5: Individual Research Programmes and Career Development Programmes 49

5.1 Purpose and scope 49

5.2 Recommended peer review approaches specic to Individual Research 52

and Career Development proposals

5.3 Processing of applications 535.4 Final selection and funding decisions 56

Chapter 6: Collaborative Research Programmes 57


6.2 Recommended peer review approaches specic to Collaborative Research proposals 58


6.4 Final selection and funding decisions 61

Chapter 7: Programmes or the Creation or Enhancement o Scientifc Networks 62


7.2 Recommended peer review approaches specic to Scientic Network proposals 63




4/88

Chapter 8: Centres o Excellence Programmes 66


8.2 Recommended peer review approaches specic to Centre of Excellence proposals 66



Chapter 9: New Research Inrastructures Programmes 70


9.2 Recommended peer review approaches specic to New Research Infrastructure proposals 719.3 Processing of applications 72


Bibliography 75

Part III: Appendices 77

Appendix 1: Glossary

Appendix 2: ESF Survey Analysis Report on Peer Review Practices 80

Appendix 3: European Code of Conduct for Research Integrity 81

Appendix 4: ESF Member Organisation Forum on Peer Review 82


5/88

EuropeanPeerReviewG

uide

3xcellence in research depends on the quality o the

procedures used to select the proposals or und-ing. ublic and private unding organisations at thetl ttl lls c th chllgo establishing and maintaining the best proceduresto assess quality and potential. Tis is a demand-ing task as each proposal is scientically uniqueand originates rom varying research cultures. Asa result, many dierent systems and criteria arecurrently in use in uropean countries. In orderto address the issue o peer review collectively, thecommon needs have to be specied rst. Te needs

th h t lmt lcs tht both convergent and complementary, whereaercoherent procedures can be conceived, promoted mlmt.

Te eads o the uropean esearch ouncils(EUROHORCs) and the uropean cience oundation(ESF) recognised in their Vision on a Globallyompetitie ERA and their oad ap or ctionsthe need to develop common peer review systemsthat are useable, credible and reliable or all und-ing agencies. o identiy thegood practices o peer

, th gg bs bth gstsinvited the ESF Member rganisation orum onpeer review to compile a eer eview uide to bedisseminated to their members and other interestedstakeholders in urope and beyond. Te orumincluded over 30 uropean research unding andperorming organisations rom 23 countries, withthe partnership o the uropean ommission andth sch cl. T m stb-lished dedicated working groups, ran workshopsand undertook a comprehensive survey on the peerreview systems and practices used by research und-ing and perorming organisations, councils, privateoundations and charities. Te results served to iden-

tiy good practices across urope on the evaluationo grant applications or individual and collabora-t sch jcts.

onsequently, this eer eview uide illustratespractices currently in use across the members o ESFand EUROHORCs, while also refecting the experi-ences o the uropean ommission in its rameworkgmms. It scbs g ctcs by sttg mmm c bsc cls processes commonly accepted at a uropean level.In addition to the quality o the basic procedures,peer reviewers and organisations ace other chal-

lenges such as assessing multidisciplinary proposalsand dening the appropriate level o risk inherent inrontier research. Te management o peer review sls by lg ttl cst ss

yet another challenge, and this is why the uidehas been designed to address the assessment pro-cedures o large scale programmes such as Jointgmmg.

Ts shl s t bchmk tlpeer review processes and to support their harmo-nisation, as well as to promote international peer

review and sharing o resources. It should be con-sidered as a rolling reerence that can be updated ands h cssy.

ESF wishes to acknowledge the key contributionso its Member rganisations to the development oths .

Prfr Mrj Mkrw

hie xecutie

Dr Mr Hppr

Director o cience and trategy Deelopment

Forewordlll


6/88

EuropeanPeerReviewG

uide

4


7/88

Part I

Overview o the Peer ReviewSystem

lll


8/88


9/88

EuropeanPeerReviewG

uide

7

1.

Introductionlll

esearch unding bodies are charged with deliver-

ing public spending programmes in the pursuit oobjectives set at the national level. In the basic inter-ests o good governance, it is incumbent on thesebodies to ensure that their unding decisions areaccountable and target the most deserving researchactivities in accordance with the programme objec-tives, and that the process or doing this deliversl my t th blc. s tht -ing decisions are air and credible, research agenciesuse experts in apeer reiew or expert reiew proc-ess to identiy research proposals or subsequent

g.his uropean Peer eiew Guide draws on

uropean and international good practice in peerreview processes, and seeks to promote a measureo coherence and eectiveness in the orm o apractical reerence document at the uropean level.

While applicable to national settings in uropeand beyond it also aims to engender integrity andmutual trustin the implementation o transnationalresearch programmes. Te content o the uide hasbeen shaped by input rom the representatives o

more than 30 uropean research unding and per-mg gsts h tct th ESFMember rganisation orum on eer eview. Inaddition, a comprehensive survey on peer reviewpractices targeted at the ESF member organisa-tions as well as other key organisations has beenconducted in order to benchmark and identiy goodpractices in peer review. Te analysis and conclu-ss th sy h ls s s c g ths ts cmmts. Tresults o the survey are available as Appendix 2o this document and through the ESF website at:http://www.es.org/activities/mo-ora/peer-review.html.

Te uide presents a minimum set o basic coreprinciples commonly accepted at a uropean level,including those o the EU ramework rogramme.It also presents a series o good practices, identi-ying possible alternatives where appropriate. It isintended to be useul to uropean research und-ing and perorming organisations, councils, privatets chts.

Te uide addresses the peer review processeso grant applications or selected unding instru-ments that comprise the majority o uropeanresearch programmes and initiatives, or example,

Indiidual esearch Programmes, ollaboratie esearch Programmes orNew esearch Inastruc-tures Programmes. I t t th scc sc t ch g stmt, th mybe programmatic or operational variants o theinstruments as practised in dierent countriesacross urope. or example, thematic versus non-thematic, responsive versus non-responsive, andmonodisciplinary versus pluridisciplinary can beconsidered as variants or the dierent undingstmts.

Tis uide is divided into two parts: the com-m cls blg blcks th ctco peer review are set out in art I. More detailedand explicit recommendations applying to particu-l g stmts t II.


10/88

EuropeanPeerReviewG

uide

8

1.1 Key denitions

In order to acilitate the establishment o a commonset o terminologies or the purpose o interpret-ing the content o this eer eview uide, a ewkey denitions are provided in the Appendix :

lssy.

1.2 Applicability

Tis document is aimed at any organisation involved g mg sch, tbly:Publicresearchfundingorganisations;Researchperformingorganisations;Researchcouncils;Privatefoundations;Charities.

he uide has been developed in a uropeancontext, but will be largely relevant beyond thecontinent. Te suggested guidelines are designed topromote common standards that adhere to acceptedgood practices on a voluntary basis. In particular,they aim to support intergovernmental or inter-organisational activities through the identicationand establishment o benchmarks and prevailingapproaches necessary to manage multi-stakeholdergmms.

Te applicability o the uide stops at the level ogranting o the awards. ence, or example, ex-postevaluation o unded research which generally hasstg lc ( xt) hs tb xlctly cl th 1.

. x-st lt, s th ESF Mmb gst m lt g chms sch gmmsctts, tcl th t:aluation in National esearch

unding Agencies: approaches, experiences and case studies, t:htt://.s.g/x.h?ID=tx_scl&=0&l=

lm/b_s/_t/M_A/MM_lt/mm_lt.&t=2332&hsh=333cbb0

1.3 How to use this Guide

In order to make the best use o this document,readers with a general interest in the subject arerecommended to browse through the chapters oart I. Te content o the rst art is structuredaccording to three thematic and easily recognis-able areas: the rst comprises an introduction topeerreviewinageneralsense(Chapter1);atypol-ogyoffundinginstruments(Chapter2);andthe

pillars o good practice in peer review (hapter 3).A second area ocuses on peer review methodol-ogy (hapter , rom ections . to .0) and ath scclly scbs th ts thunding instruments and their implication or peer (cts . t .3).

cience management practitioners with theintention o gathering concrete inormation on good

practices specic to the peer review o particularunding instruments are advised rst to review thechapters o art I, with particular attention givento hapter , and then to consult their programmeo interest in the corresponding chapter in art II.Te chapters o art II are meant to provide inor-mation on the state-o-the-art and benchmarking opeer review practices specic to the selected undingstmts.


11/88

EuropeanPeerReviewG

uide

9

2.

Typologyo unding instruments

lll

haracterising the appropriateness o peer review

practices can be meaningul only when consid-ered in the context o the specic programmes orunding instruments to which they must apply.Tereore, in order to establish common approachesand understanding o the practices o peer review,it is necessary to establish common denitions andmeanings in the context in which they are to beused. Tis context is dened by various undingopportunities with specic objectives that dierentorganisations have developed in order to select com-peting proposals and to allocate merit-based unding

using clearly dened objectives and selection crite-. I ths cmt, ths g tts t s g stmts.

Across uropean countries, all major und-

ing instruments that rely on peer review as theirmain selection tool have been considered or inclu-sion in the uide (see able , below). owever,based on the input received rom the ESF Memberrganisation orum on eer eview and the resultso the ESF urvey on eer eview ractices, the nallist o instruments elaborated in art II o the uideexcludes two o the instruments outlined in thetbl bl, mlyKnowledge ranser Major

Prizes and Awards.Brie descriptions o typical unding instruments

are provided in the next section, while the specicpeer review process or each o them is elaboratedin art II. Many o these unding instruments orprogrammes have dierent variations in terms osc scly chctstcs. T, st sct s t t lbtg thseatures. When these variants have noticeableimplications on the practice o peer review, theyare urther elaborated in hapter , or in the cor-sg chts t II.


12/88

EuropeanPeerReviewG

uide

10

2.1 General descriptiono main unding instruments(s bl b)

2.2 Variants o unding instruments

T m sc bjcts sm th -ing opportunities mentioned in the previous sectionmay be tailored through policy or strategy consid-

erations, giving rise to specic variations. ome oth m ctgs t bfy scbh.

2.2.1 Non-solicited (responsive mode)

versus solicited unding opportunities

egardless o the nature o a unding instrument(sc, bjcts tgt lcts), th tmgand requency o the call can vary rom organisationt gst m gmm t gmm.In this sense, two variants o any typical undinginstrument may be envisaged as: (a) when applicantssubmit their proposals to a call or proposals witha xed duration and specied date or its open-

ing;thesearesolicitedfundingopportunities,alsok s mg m g. (b) Wh thcall or proposals or a given unding l ine is continu-ously open and ideas are submitted in an unsolicitedmanner;thisisknownasresponsivemodefundingin some research councils 4. In terms o the processo peer review and selection o proposals, there aresome dierences between the two modes that wil lb scb ht , ...

2.2.2 Thematic versus non-thematic ocusAnother variant o most typical unding instru-ments can be considered to be the thematic (ortopical) versus non-thematic (open) scope o thecall or proposals. Tematic opportunities can beused or strengthening priority areas o researchthat the unders may identiy through their sci-

2. Btchlgy Blgcl ccs sch cl,SRC sch, It klls Dctt, SRCsch ts. T , ctb 200, . :htt://.bbsc.c.k/b/FES/ls/gts_g .3. Ntl mt sch cl:

htt://.c.c.k/sch/ss/. gg hyscl ccs sch cl (EPSRC):htt://.sc.c.k/g/gts/b/gs/lt.sx

Table 1. List of typical funding instruments

Instrument Description

Individual research

projects

g l ct t sls sbmtt by sgl stgt g stgts th sm tm. Ts sls tyclly cl ly st sl-ct sch gls, k l bgt.

Collaborative research

projects

g l ct t sls cmsg gs lcts hcg

tl/ttl cllbt scc sch jcts.

Career development

opportunities

g l ct t stg c gss schs schlsthgh s, llshs, tmts, ssshs, hs, tc.

Creation o centres or

networks o excellence

g l ct t sls sbmtt by lg g schs tgtg th stblshmt stttl gl cts, tks g s sch.

Knowledge transer and

dissemination grants

g l ct t jcts stg th ts slts m scc tsty th t/blc scts.

Creation or enhancemento scientifc networks

g l ct t mtg tkg schs th m mtgs, ccs, kshs, xchg sts, tc.

Creation or enhancement

o research inrastructure

g l ct t cg lmt, hcmt, mtc / t sch stcts.

Major prizes or awards g l ct t g tstg ctbts sglsch / g schs.


13/88

EuropeanPeerReviewG

uide

11

ence policy or strategic plans. In some programmes,themes o research areas or topics may themselvesbe identied by investigators using peer review andthrough calls or proposals. ome councils use so-called signposting or fagging priority areas intheir responsive mode unding streams. Te impli-

ct thmtc ss -thmtc t cll sls th css st y sgct bt ll b bfy scss ht , ..2 ths .

2.2.3 Monodisciplinary versus

pluridisciplinary ocus

or the purposes o ne-tuning and sharpening theprocess o peer review according to the scope o theproposals, it may be o interest to categorise propos-als into monodisciplinary and pluridisciplinaryh t. sch sls csgly

draw on knowledge and expertise outside o onemain discipline. In some programmes, there are nospecic modalities incorporated to deal with plu-ridisciplinary proposals while other instrumentsmay be designed to specically oster and manageths ks sch.

urrently in the specialised literature there areongoing discussions on the dierent types o plu-ridisciplinary research 5. or the purposes o thisuide the term pluridisciplinary may be used inthe widest sense, i.e., research proposals that clearly

genuinely q xts m b go dierent disciplinary domains. owever, orcompleteness, a brie review o the types o pluridis-ciplinary research as described in the literature isprovided in hapter , ection .2 o this uide 7.In the same section relevant peer review specicitiesand recommendations or the assessment o thesetys sch sls ls scb.

2.2.4 Breakthrough research

Breakthrough research aims at radically changing

the understanding o an existing scientic concept,and could lead to changes o paradigms or to thect gms ls scc. Tlevel o risk associated with the success o these proj-ects is generally higher than mainstream research,i.e., research activities that in general lead to incre-mtl gs th l sks l.

Te survey on peer review practices shows that70% o the respondents do not have instrumentsspecically designed or breakthrough proposals,

. Lttc (2003) Abll (200).

. m, Tms Kl Mtchm (200).. UESCO (), ransdisciplinarity timulating synerg ies,integrating knowledge.

and 20% o the organisations have only one suchct stmt. Whl 33.3 th s-g gsts h t tht thy gllysee breakthrough proposals in their conventionalstmts, 0 thm h stt tht thy sths ty sl ly ly .

xplicit identication and handling o break-through research is generally more complex thanmainstream research. In the context o research sub-jects, priorities and goals, breakthrough researchis characterised not only by exceptional potentialor innovation, and creation o drastically newknowledge, but also by consciously acknowledg-ing and taking the associated risks1. Tis can haveimplications or the process o peer review as briefyscb ht , ct .3 ths .

. cc t (200b),ESF urey Analysiseport on Peer eiew Practices, tcl 3.2., Qst: my g stmts s y gst hhch ct xclsly t bkthgh sls?. cc t (200b),ESF urey Analysiseport on Peer eiew Practices, tcl 3..2, Qst: s y gst s bkthgh sls

th y ctl stmts, i.e. stmts tsclly ct t bkthgh sls?0. yy (200), . .


14/88

EuropeanPeerReviewG

uide

12 Based on a comprehensive review o the existing

practices and the available literature11, ve elementsare identied as key supporting pil lars o good prac-tice in the edice o peer review (see igure ). Tesells ll s tht th ll csss, c-dures and operational steps including decisions are hgh lls qlty, qty blc cct-ability without being excessively rigid, bureaucratic, ct cstly.

Te central pillar consists o a set ocore prin-ciples that are commonly accepted by the relevantorganisations engaged in peer review. Tese are the

. th lst cs t th ths cmt.

ky gg cls tht t b sg

t ch cbl, qtbl ctpeer review. our other pillars that have been identi-edare:safeguardingoftheintegrityoftheprocess;soundmethodology;strongmeansofassuringqual-

ity;andappropriategovernancestructure.

3.1 Core principles o peer review

uiding principles have been dened and used byvarious organisations that deal with peer review.

Although there are strong similarities between di-erent sets o these principles, there are also slightdierences in their scope and ormulations. or thes ths , t s cssy t t sto principles as the guiding ramework, in which sts ch.

Te list o the seven core principles presentedbelow (able 2) are included in the eer eviewramework onditions or the EUs Joint ro-grammes1. It also covers the items identied andelaborated by the ESF Member rganisation orum

. Although identiying core principles as thecentral pillar or good practice in peer review isa necessary step, it will not be su cient withoutensuring other organisational and proceduralingredients necessary or realising good practice.

As mentioned above, our other supporting pillars bfy scb th llg scts.

2. sch A mm tt, gh Ll

Jt gmmg: Voluntary guidelines on amework conditionsor joint programming in research 2010 , Ax, t: htt://gst.cslm..///0/st0/st030.0.

3.

Pillars o good practicein peer review

lll

Figure 1.

Five pillars supporting good practices of peer review with quality

and equity

Process

Integrity

Governance

Structure

Core

PrinciplesQuality

Assurance

Methodology


15/88

EuropeanPeerReviewG

uide

13

3.2 Integrity o the processo peer review

All research institutions (research unding andperorming organisations as well as academiesand universities) have the role and the obligationto promote relevant research and good researchpractice and to ensure the integrity o their con-ct 1.

undamental principles o good research prac-

tice and peer review are indispensable or researchintegrity1415. unding organisations and reviewersshould not discriminate in any way on the basiso gender, age, ethnic, national or social origin,religion or belie, sexual orientation, language,disability, political opinion, social or economicct.

3. mmss (200), Te uropean harter oresearchers.. cc t (200),ostering esearch

Integrity in urope , . -.. mmss (200), Te uropean harter or

esearchers, . .

Integrity o the peer review process should beensured through appropriate resources, policies ctcs, mgmt tts, s llas training and monitoring, such that in essencewe can say what we do and do what we say wedo. o this end, upholding the advertised set oc cls s cst th tgty the process. Dierent organisations have variousmeansofassuringintegrityoftheirpractices;how-, th cmm bsc cls tht mst

be incorporated. lexibility and pragmatic inter-pretations may be exercised only with extreme careand according to the context and without ignoringthe core meaning o these principles or violatingth st. thm, th fxbl ty xcsin the sphere o one principle should not violate orcm t cfct th th cls.

o saeguard integrity it is absolutely essen-tial to avoid discretionary decisions and changes.ective and transparent communication is acrucial element in saeguarding the integrity oy mlt-stkhl systm sch s .Tereore, guidelines on integrity must be ormu-lated and promoted to help all parties implicated

1. Excellence jcts slct g mst mstt hgh qlty th ctxt th tcs ct st t th clls. T xcllc th sls shl b bs ssssmt m by xts. Ts xts, l mmbs xt s shl b slct ccg t cl ct t cs tht bs mg cfcts tst.

2. Impartiality

All sls sbmtt mst b tt qlly. Ty shl b lt thmts, sct th g th tty th lcts.

3. Transparency Dcss mst b bs clly scb ls cs tht blsha priori. Al l lcts mst c qt bck th tcm th lt th sl. All lcts shl h th ght t ly t th cclss th. Aqt cs shl b lc t l th th ght t ly.

4. Appropriateness

or purpose

T lt css shl b t t th t th cll, th sch ss, t th th stmt cmlxty th k.

5. Efciency

and speed

T -t- lt css mst b s s ssbl, cmmst thmtg th qlty th lt, sctg th lgl mk. Tcss s t b ct sml.

6. Confdentiality All sls lt t, tllctl ty th cmts mst btt cc by s gsts l th css. Tshl b gmts th scls th tty th xts.

7. Ethical and integrity

considerations

Ay sl hch cts mtl thcl tgty cls my bxcl t y tm th css.

Table 2. Set of core principles of peer review


16/88

EuropeanPeerReviewG

uide

14

in the peer review process, namely, applicants,reviewers, panels, committee members, hairs,programme oicers and sta. hese principlescl1:Honestyincommunication;Reliabilityinperformingresearch;

Objectivity;Impartialityandindependence;Opennessandaccessibility;Dutyofcare;Fairness in providing references and giving

credit;Responsibilityforthescientistsandresearchersof

th t.

3.2.1 Conficts o interest

Te prevention and management o conficts ointerest (oIs) are the most important ingredients

or ensuring equity and integrity in peer review, andto preserve the credibility o the process and that th ssbl gst. A I ls thbs mss b t ctl, t, cor potential o the trust that the public and theclients must be able to have in proessionals andadministrators who manage or can infuence deci-ss sch g.

A oI is a situation in which nancial or personalconsiderations have the potential to compromiseor bias the proessional judgement and objectivity

o an individual who is in a position to directly orindirectly infuence a decision or an outcome. Inact, oIs are broadly divided into two categories:tgbl, i.e., ths lg cmc cttsandscholarship;andtangible,i.e., those involvingcl ltshs1.

In peer review it is important to set out inadvance in as much detail as possible those condi-tions that are deemed to constituteperceied andreal conicts o interest. It may be appropriate todistinguish conditions that would automatically

disqualiy an expert, and those that are potentialconficts and that must be urther determined orresolved in the light o the specic circumstances.o uphold the credibility o the process, both real c cfcts shl b ss.

ycl sqlyg Is mght lt t:eexpertsaliation;Whetherheorshestandstogainshouldthepro -posalbefunded(ornot);

Personalorfamilyrelationshipwithapplicant;

. cc t (200),ostering esearch

Integrity in urope , . .. lmb sty (2003-200),esponsible onduct o

esearch: onict o Interest.

Researchcooperation/jointpublications/previous

ssy l.In these situations, the reviewers should avoidassessing a proposal with which they have confictso interest. In the case o panel discussions, theseindividuals should not be present when the proposal

qst s bg scss.Whl y t shl b m t h-ing reviewers assessing proposals with which theyhave a potential oI, there may be circumstanceswhere these situations can be resolved or miti-gated without ully excluding the reviewer with acl cfct. xml, h th xtso all parties in a review panel is needed, and pro-vided that the potential oIs o individuals havebeen declared and recorded, it may be decided toallow the reviewer(s) to assess the proposal and/orparticipate in the panel discussion. In this situation

th l(s) th th ttl cfct shlclearly state their own disposition on whether ornot their views are biased and continue their par-tct ly thy clly stt tht st thpotential confict they do not eel biased in anyy.

Te rules or oIs may vary according to thestage o the evaluation, and the role o the expert.or every proposal evaluated, each expert must signa declaration that no oI exists, or must report such ct t th ssbl st mmb. Whl

gcy st mst b lt t l l tms, th shlb stg ms tst xcs th sctto the invited experts and their honesty and objec-tty.

3.2.2 Managing condentiality

ach expert should sign a code o conduct beorethe start o the evaluation process. Te code shouldl bth th th qmt t cl y I(see above), and with the obligation to maintain thectlty th mt h q.

Measures to avoid leaks o condential inor-mation (both deliberate and inadvertent) include:secure systems(password,etc.);watermarks;restricted use o F, GS, etc. when appropri-ate. Te appropriate measures will depend on thestage o the evaluation, and on the sensitivity othe research topics under review. Diering levelso transparency are also important or a good andimpartial peer review. We can broadly identiy threesystms:Double-blind review: the identity o both the

reviewers and o those being reviewed is kept con-dentialfromeachother;

Single-blindreview: the identity o the applicants


17/88

EuropeanPeerReviewG

uide

15

being reviewed is revealed to the reviewers but notice ersa;

Openreview: the identity o both the reviewersand o the applicants being reviewed is revealedt ch th.

According to the peer review survey, single-blinds mtly s css mst g-isationsinmostoftheprogrammes;forexample,

Indiidual esearch Programmes th tty l/mt s s t scls t thapplicantsin80%oftheorganisations;whilein62%

o the organisations the identity o the panel review-s s t scls t th lcts1. , sm c cts s t by th mm-bers o the ESF Member rganisation orum oneer eview, the situation can be very dierent asnational legislations may call or ull transparency

h lg th blc g .

3.2.3 Applicants rights to intervene

It is o utmost importance or a credible peer reviewsystem to provide one or both o the ollowing ea-tures to ensure that the applicants have the rightto understand the basis o the decisions made ontheir proposals and consequently to be able toinfuence the outcome o such decisions in caseswhere these are made based on incorrect or inac-ct mt, fc by ctl s

gg.Right to appeal or redress: this eature allows

the applicants to appeal at the end o the selec-tion process aer the nal decision is made. Tel s mlly m t th g gs-tion or to a dedicated independent oce basedon a known and transparent process o redress.Trough the process o redress the applicants donot infuence the peer review during the selectionprocess, but can object to its outcome. In a generalsense, redress only concerns the evaluation process

or eligibility checks and applicants cannot ques-tion the scientic or technical judgement o thes. Dg th stt thcase where decisions have been made incorrectly,the applicants should be given another chanceth sh th sl.

Right to reply: in contrast with redress, the right toreply is included as part o the peer review proc-ss tsl. It s mlly l t t-stg review systems where a panel o experts makes a

. cc t (200b),ESF urey Analysiseport on Peer eiew Practices, tcl ct .2, bls.3 .3.

selection, prioritisation or ranking o proposalsbased on external reeree assessments. eedbackand intervention rom applicants are not providedt m lbt th tlly sbmtt -posals or to change them in any way. It is onlymeant to allow the applicants to identiy andcomment on possible actual errors or misunder-standings that may have been made by the reereeswhile assessing the proposal. Te external reereesas well as the applicants and the members o thereview panels should be made ully aware o theprocedures and timing related to the right to replystage (more details on this eature can be ound in.. ths ).

3.3 Quality assurance

Another important pillar or ensuring good prac-tice is the adoption o explicit means o assuringquality in all relevant aspects o the process andts.

In order to assure quality o the process and pro-cedures, it is necessary to monitor and measure thequality o the main products and services providedbased on known criteria and indicators. or moni-tg qlty th llg lmts my b s:ta members with an explicit mandate withintheorganisation;

Dedicatedocewithintheorganisation;Dedicatedcommitteesorboardsoutsideofthe

gst.

According to the survey on peer review practices,the quality o the peer review system is otenassured through external ad hocor standing com-mittees (7.7% o respondents), or by a group osta members with an explicit mandate (6.7% orespondents). nly 6.% o the respondents reportedthat there is a dedicated oce with an explicit man-

t ssg qlty th gst1.

3.4 Governance structure

Another supporting pillar or achieving and main-taining good practice in peer review is the presenceo strong governance that is put in place to ensureorganisational and operational coherence and qual-ty. m th ky ts g gc

. cc t (200b),ESF urey

Analysis eport on Peer eiew P ractices, tcl ct 3.2,Qst : Wht ms s y gst s ssgth qlty ts systm? (bl 3.).


18/88

EuropeanPeerReviewG

uide

16

structure are: eectiveness, clarity and simplic-ity. Te governance structure is meant not only toensure that all the relevant players and stakehold-s m lly th ls ssgtasks, their expected contributions and their respon-sibilities but also to ensure that all contributionsare made according to the required standards andwithin the scheduled deadlines. inally, the gov-ernance structure is meant to be able to hold therelevant bodies accountable or any deviations orshtlls.

ome o the main attributes o credible andct gc tl bl:Itct th lt cts, clc-

tion o the scope and levels o their responsibilities(e.g., decision makers, clients such as researchersand the public, other stakeholders such as regionalornationalgovernments);

Denitionofrolesandresponsibilitiesofthekeyactors: programme ocers, management com-mittees, review panels, other decision making orconsulting panels (such as ethical panels or mon-itoring panels or committees), readers, externalobservers,etc.;

Denitionanddisseminationofkeydecisionmak -ingprocessesandapprovalprocesses;

Denitionanddisseminationofproceduresto

eect continuous improvement through appro-priatechangestotheprocess;

Availabilityandeectiveallocationoftherequiredresources (nancial, human resources, technicalrecoursesandinfrastructure,etc.);

Termsofreferenceand,ifpossible,codeofcon-duct or all the participants (terms o appointment,condentiality agreement, declaration o confict tst, tgty c, tc.).

3.5 Methodology

he inal important pillar or achieving goodpractice in peer review is the actual adopted meth-odologies and approaches or conducting peerreview. ince it is under methodology that themain building blocks and common approacheso peer review are described, a dedicated chapter,hapter , is provided to illustrate the dierentsteps and the sequential order o the peer reviewcss gl ss.


19/88

EuropeanPeerReviewG

uide

17In this chapter an overall methodology is suggested

tht s bs th mst cmm chs good practice in use across various organisations andor dierent types o instruments. It breaks downthe overall process into the main sub-processes orblg blcks t th hghst ll s llstt igure 2. Tis is the scheme o the peer review proc-ess across the entire set o instruments covered inths cmt.

I ht lls, ch th m sb-csss

illustrated above will be described separately in theorm o a general model. or particular undinginstruments the models described in this chapterneed to be instantiated and elaborated to suit thespecic needs and characteristics o the required g stmt. Ts s art II where or each instrument a dedicated chap-t s , tlg -t- cycl thth q tls.

Te variants o the typical unding instrumentsdescribed previously in hapter 3 can also impose

specic nuances and variations on the requirementso the peer review process. Tese variations arescb gl ss t th ths ch-ter while urther instrument-specic ne-tuningo the practice based on variations o the types ostmt s scb th csg ch-t t II s t.

4.1 Preparatory phase

In this section a summary o all the elementsrequired or consideration, preparation and elabo-ration beore the launch o a given programme isprovided. Te preparatory phase is marked bya mandate and decision to launch or re-launch aunding instrument and ends when al l technical,organisational and procedural components are inplace and ready or being launched. Te intensity t th ty hs s mstmt t stmt s hth

20. cmlmt ths cht, g cll mlmtt th ctxt ER-ES c b h: htt://ttch.jc.c..//x.cm/sttc/l/l.html

4.

Peer review methodologylll

Figure 2.

High-level process description of a typical peer review system

Preparatory Phase

QualityAssuranceandEvaluation

Com

municationandDissemination

Processing of Applications

Final Decision

Expert Assessment

Selection of Experts

Call for Applications


20/88

EuropeanPeerReviewG

uide

18

or not the programme is responsive or solicited.owever, or a given instrument that recurs peri-odically (e.g., annually) the duration and intensity othe activities in this rst phase are diminished sinceresources, inormation, knowledge and tools wil l bes s lg s mj chgs t cssy.

ths stmts tht lch thrst time or or one-o programmes, or in situa-

tions where major changes are applied to existingunding streams, this phase may be considerablylonger and more involved. ome o the main sub-processes o the preparatory phase are outlined here g 3.

nder each sub-process included in igure 3and described below, the list o items that need tobe considered is also provided. Tese lists are notxhst bt c th mst tycl scts scss t gsts.

4.1.1 Mandate and scopeIn order to establish the programme eciently andchtly, th llg scts t b cllydened by the responsible bodies and communi-ct t ll lt ts:Programmeneeds;Programmeobjectives;Overallprogrammebudget;Potentialstakeholders(beneciaries,clients,deci-sionmakersandotherrelevantparties);

Performancemeasures(ifrequired);Researchclassicationsystem(ifrequired);Typologyoffundinginstrumentorvariants(if

q).

4.1.2 Managerial and technical implementation

nce the mandate and scope o the programme areclearly established and understood, the responsibleorganisation, department(s), or group(s) o sta ischarged with establishing the required technicaland managerial components needed to implementor run the programme. ome o these are listedbl:

Workplansandlogistics;Humanresources;Detailedbudgetfordistributionandindicative

budgetforpeerreview;Timeline;Otherresources(informationsystems,facilities,

databases,etc.);Overalldecisionmakingprocess;Rolesandresponsibilities,delegationofauthority,

cs l sg-s.

4.1.3 Sta and resource allocationg stblsh th mt, sc hghlevels o organisational structure and assignments,responsible departments, groups and units wil l takecharge. ome o the items necessary to keep in mind lst bl:Taskallocation;Budgetallocation;Assignrolesandresponsibilities(contactpointsfor

applicants, check o eligibility, confict o interest,completeness o the application, reviewer assign-ment);

Programmeandscienceocershaveapivotalrolebeore, during and aer the peer review process.

Figure 3. Preparatory phase

4.1.1

Mandate

and scope

4.1.3

Sta and resource

allocation

4.1.4

Peer review

process

4.1.2

Managerial

and technical

implementation

4.1.5

Documentation


21/88

EuropeanPeerReviewG

uide

19

Te responsible sta will thereore need to havea level o education and training in research thatgives them not only credibility but also equipsthem with the basic knowledge and intellectualtools to understand the ield o research andresearchactivity;theseaspectsneedtobecom -

lmt by stg mgl sklls.4.1.4 Peer review process

nce the responsibilities are assigned and the natureo the programme and its objectives are established, t, t--s csshs t b . ths , th llg tms t b cs:Mainstagesoftherequiredpeerreviewprocess:

one-stage submission o ull proposals, versustwo-stage outline proposals (or letters o intent)ll by ll sls. tl sls

normally sied through by a dedicated panel,committee or board. ull proposals normally gothrough a complete peer review either in one orin two or more steps, i.e., either selection throughremote assessments or using remote reviewers plusreviewpanel;

Mainfeaturesoftherequiredpeerreviewmodel:overall decision making process using panels,individual/remote (external) reviewers, othercommittees (or prioritisation, unding, etc.),expert readers, observers, redress or rebuttals,

whether or not re-submissions are accepted andtheirconditionsifany,etc.;

Operationaldetailsandlower-levelrequirementssuch as timelines, workfow, reporting, commu-nication,etc.;

Assessmentprocess:identifyspecicfeaturessuch

as the nature and number o assessors, the sourceo identiying experts, multidisciplinary consid-erations, work load or external experts, and panelmembersincludingrapporteurs,etc.;

Schemesfortheowofinformationanddocu-

mentation, necessary I tools and resources(web pages, online submission orms, guide-lines,etc.);

Processmonitoringandevaluation,including

audits, observers and eedback to relevant spon-sg cmmssg ts clts.

4.1.5 Documentation

All cmts (clg gls, mls reports) must be comprehensive and provide all thenecessary inormation, and at the same time theymust be ecient and as short as possible. ome oth m ts ct cmtt :Availabilityandclarityofallrelevantdocuments

on unding instruments and specic guidelinesandmanualsforapplicants;

Availabilityofallrelevantmanuals,guidelinesortandard perating rocedures or the sta mem-bers responsible or the management o the peerreviewatvariousstages;

Availabilityofalltherelevantdocumentsdening

the process, and the roles/responsibilities o thevarious actors to reviewers, members o the panels cmmtts.

A list o commonly required documents is providedbl:all or Proposals (call text): the call or propos-

ls mlly cmss t m ts: st, thscientiic part which describes the scope andobjectivesoftheprogramme;denesthescien -ticcontext;andoutlinesthescientictopicsand

subtopics to be covered. Te second part o the calltext describes the necessary programmatic aspectso the programme. It clearly describes the peerreview process and its various stages. It denesthe required ormat, length and language o thesls, lsts lgblty ssssmt ct,inorms about the available budgets and eligiblecsts, scbs th tmls m ml-stones throughout the process including variouscmg cmmcts t lcts.

Guidelinesandinstructions to the applicants: these

documents should contain mandatory templates,predened section structure, length per section,list o mandatory annexes and supporting docu-ments, list o optional annexes, list o requiredsgts.

Referencedocumentation: guidelines or applicants,reviewers and panel members, description o thegc stct, tl sct thpeer review process, description o selection anddecision making processes including eligibilityand assessment criteria, code o conduct, redress

and right to reply procedures, proposal and con-sortium agreements i applicable, guidance onpreparation o agreements or dealing with theissues regarding intellectual property and com-mclst.

FrequentlyAskedQuestionsandglossaries.Onlineformsandwebpages.


22/88

EuropeanPeerReviewG

uide

20

4.2 Launch o the programme

nce all the preparatory steps or the launch osolicited unding opportunities or programmes arein place and communicated, the actual opening and

implementation phase can begin. In a general sense,th lmts sh g t b c.or responsive mode programmes, where the callis continuously open, periodic communication andpromotions are still necessary, although some o thests scb bl my t ly.

4.2.1 Dissemination o the programme

In order to reach out to all the eligible applicantsand reviewers and encourage participation, it isessential that the launch o the programme is dis-

smt thgh ll th lcbl ms g tm.

roundwork or the dissemination o the oppor-tunity should have started in the preparatory phaseand be completed in this phase. A continuous dis-semination o the call or proposals should be inplace or responsive mode programmes. In addition,particular attention should be given to targetingthe inormation streams to the appropriate com-munities, or example in the case o collaborative(national or international) research programmes,

thematic or topical programmes, or or break-thgh sch.ome o the main means o disseminating the

tty :Web-basedpromotion;Advertisementinscienticmedia(newspapers,

journalsandmagazines,etc.);Dedicatedmailingliststowhichresearcherscan

ly sbscb.

4.2.2 Opening o the call or proposals

alls should open at the stated date and time anda communication to all relevant parties and stake-holders should be made announcing the launch o

the call. Beore the actual opening o the call orsls th llg tms shl b ly lc:T cs cts by hch g

decisions are to be made must be spelled out in thecalldocumentationasdescribedabove;

Aclearplanofcommunicationofthemaindeci -sions;

Asfaraspossible,dedicatedandsecurewebpages

and databases or online management o all theprocessesandinteractions;

Onlineandclearaccesstoalldocumentation.

4.2.3 Closing o the call

Te closing o the call has to be communicated assoon as possible to all stakeholders (such as the

applicants, reviewers, sta members and otherrelevant parties). Te announced deadline or theclosing o the call has to be clearly stated well inc s t th ty hs mstb sct.

ostponing the deadline or the closure o thecalls should be avoided and be considered only invery exceptional and unpredictable circumstances.In these situations, and especially i the exten-sion can be seen as considerable or the applicants,eorts should be made to allow resubmission o

proposals to all those applicants who had submit-ted their proposals at the time the extension wasannounced and who may wish to take advantage othe additional time given. At any rate, in the caseo extensions, clear statements must be widely dis-seminated describing the reason or and nature oth xts.

Figure 4. Launch of the programme

4.2.1

Dissemination

o the programme

4.2.3

Closing o the call

4.2.2

Opening o the call


23/88

EuropeanPeerReviewG

uide

21

4.3 Processing o applications

I ss m gmms hch th cllor proposals is continuously open, applications css btchs th th tmg

cannot be determined in the same way as or thegeneral case o solicited opportunities. or the latterit is possible to group the subsequent activities oth cssg hs t th llg th sts(g ).

Depending on the size and scope o the pro-grammes, proposals may be solicited in either onestage or in two stages. ence, or one-stage callsthe entire process must be completely described inthe call, whereas or two-stage schemes a rst call isissued through which promising ideas are selected

and retained or a second round o submitting ullproposals based on which nal selection and und-g css ll b m.

T lmy slct s mlly by review panel based on outline proposals, or letterso intent. Tese outline proposals contain a shortdescription o the nature and overall objectives othe research as well as indications on the requiredresources, inrastructures, budgets and the propos-ing team. Te secondary stage is normally doneusing ull proposals through a two-stage peer review

system by remote assessment ollowed by reviewl lbt kg.

4.3.1 Eligibility screening

lgblty scg s glly msttcss, s c t by ssbl mmbso the sta in the unding organisation. owever,in some cases, notably in assessing eligibility inlt t th sctc sc th cll, sctcxt c shl b sght s.

In the case o multidisciplinary or breakthrough(hgh-sk hgh-t) sch, t l l ls bnecessary to involve scientic experts to screen pro-sls ltts tt lgblty.

Any eligibility criteria used to screen proposalsmust be dened and clearly stated in advance o thecall and made available to all as part o the dissemi-nated inormation. ligibility criteria should not beopen to interpretations and must be applied rigor-

ously to all applicants in the same way. ome o theusual eligibility criteria used by unding organisa-ts lst bl:Completenessoftheproposal(inclusionofall

requested inormation, documents, chapters, sec-tions,annexes,formsandsignatures);

Timelinessofthesubmission;Eligibilityoftheapplicantsforreceivinggrants

andforworkinginthehostorganisation;Eligibilityofthescopeoftheresearchproposedinrelationtothecall;

Ethicalconcerns(e.g., applicable national andinternational regulations and directives on saetyand security, embargos, use o animals and humansubjects, controlled inormation, hazardoussch, mtl csts, tc.).

o uphold the principle o impartiality and topromote equal playing ields, eligibility screen-ing should be conducted strictly and consistently.

Applicants who have ailed the eligibility checksshl b m s s s ssbl.

4.3.2 Acknowledgment

During the phase o processing the submittedproposals, the applicants as well as other relevantstakeholders must be inormed o the intermediatesteps. Ideally, the steps below should be considered cl th ll l:Acknowledgment o receipt o the proposals giving

mt th sbsqt sts cmm-nications;

Acknowledgmentoftheeligibilityscreeningas

soon as it is determined. In the case o ineligibleproposals, sucient inormation describing thedecisionmustbecommunicated;

Figure 5. Processing of applications

4.3.1

Eligibility

screening

4.3.3

Resubmissions

4.3.2

Acknowledgment


24/88

EuropeanPeerReviewG

uide

22

Forthesakeoftransparency,itisadvisableto

inorm the applicants o the general statistics onsubmission, e.g., overall numbers received versuslgbl sls, tc.

4.3.3 Resubmissions

In some organisations, particularly or larger pro-grammes, the eligibility checks do not immediatelylead to non-compliance and exclusion o the pro-posals. In these situations, there may be a period oeedback and negotiation between the oce andthe applicants during which the ineligible applica-tions are provided the opportunity to improve theirproposals and to resubmit. Tis practice, i neces-sary, should be handled with great care, openly andlgtly by cmtt xc mmbso the sta in order to avoid personal infuences andinconsistencies. In these cases it is crucial to be ully

tst csstt lyg k cl ct g qtbl t-ties and attention to all applicants consistently andt th sm g.

In most cases, however, the eligibility checksare nal and determining, without the possibil-ity o resubmission in the current call. or thesesituations, it is also necessary to be clear on thepossibilities and means o resubmitting improvedproposals in the next round o the call or propos-ls.

4.4 Selection and allocationo experts

ne o the most important and challenging phases th css s t cllct th qnumber o willing and available experts who wouldagree to conduct the task o expert assessments bothas individual/remote reviewers and/or members ols cmmtts s scb bl.

Te activities to be undertaken or typical pro-grammes are grouped under the ollowing oursts (g ).

4.4.1 Identication o the types o experts

needed

Depending on the nature o the programme and theadopted peer review model, dierent types o experts lts my b q. xm-ple, there are instruments or which peer review isconducted by remote experts only. owever, orthe majority o the instruments both remote andpanel review are used. Tereore it is rst necessaryto consider the types o experts needed. vidently,

this process should start in the preparatory phase,bt b mlmt g ths hs.

Means o identifcation o expert reviewers

unding organisations oten have a database oreviewers which is structured based on a given andoen multi-level research classication system (tax-onomy o research proles). As discussed below, withthe advent o increasingly more advanced inorma-tion management systems and tools, the originalneed or conventional multi-level classication sys-tems may be reconsidered now. urrently, however,most o the existing operational systems across di-erent science management organisations seem to relyon some kind o hierarchical structuring o researchls tms scls sb-scls.

Te peer review survey shows that 90% o theorganisations use a multi-level research classication

system or the structuring o their research prolesand proposals. Te results o the survey point to astrong tendency to rely on internal sources or thet ths clssct systms: xm-l, 0 sts ly th gstssta;39%ontheirscienticcouncil,while28.6%of

the organisations use the system oered by the OECrascati Manual1. T t cllct thgh thESF urvey suggests that the current classicationsystems in place may not be ully compatible. omove towards more comparable and thereore more

widely accepted common peer review practices, it iscrucial that thepeer reiewers are assigned scientic/expert proles that can be interpreted clearly andwithout ambiguity across dierent organisations th tbss.

urthermore, detailed analysis o the survey datasuggests that those organisations that have indi-ct sg th OECrascati Manuals th bss th clssct systm h by lg lsbeen more satised with the eectiveness o theirclassicationsystem;thisisincontrasttothosethat

s tlly clssct systms.Tereore, the use o commonly accepted systemssuch as the OECrascati anualor o any otherclassiication system that allows a unique map-ping o the research proles rom one system intoanotherwithoutambiguityshouldbeencouraged;

2. cc t (200b),ESF urey Analysiseport on Peer eiew Practices, tcl ct 3., Qst: Ds y gst s sch clssct systm th gg y sls? (bl 3.) Qst2: Wht s th sc ths clssct? (g 3.). ls Ax B t th t:esearch lassication ystem: A

preliminary map o existing uropean app roaches.22. cc t (200b),ESF urey Analysiseport on Peer eiew Practices, ct 3., 3.., bl 3.3.


25/88

EuropeanPeerReviewG

uide

23

this will help to create the needed ingredients orcross-reerencing and thereore comparable interac-ts cllbts t th ll.

unding organisations normally use their con-ventional research classication system in order

t mtch th ls th q xts t thsctc sc th sls . Tsmay be reerred to as discipline matching whenselecting reviewers and it relies on updated, accurate cmtbl sch clssct systms.

In contrast to this standard method and ena-bl by th t m tmt madvanced inormation management systems, manyorganisations are considering the use o match-ing okeywords between proposals and reviewersproles. Tis means searching or reviewers in data-

bases using electronic matching (text mining) okeywords or key phrases stemming rom the pro-posals to the keywords attached to the proles othe reviewers within their dedicated database. Tismy b t s ky mtchg.

Te two aorementioned methods have strengthsin addressing the selection o reviewers in dierentways. or example, discipline matching may notbe as eective in identiying specialised reviewerssuch as those needed or multi-, inter-, cross- andtrans-disciplinary (C) proposals, whereas key-

word matching will generally be more adequate innding reviewers with particular research expertise.n the other hand, as described in ection .2, itmy b tgs t mt scly -spectives when dealing with peer review o Csls. c, t my b qt tgs tuse the two schemes in conjunction and comple-mtg th.

Experts who take part in the peer review process

In a general sense there are two main groups oxts h tk t th css:Externalorindividualremotereviewerswho assess

the proposals on their own and separately rom

th mmbs h my lk t th sm s-ls. Ts s t scss th slsth y th ssssmts sgk cl ct scs.

Membersofreviewpanels who will collectively dis-

cuss and evaluate groups o proposals. Te mainunction o the panel is to evaluate and consoli-date external assessments by experts on a groupo competing proposals and to rank or priori-tise them based on clear and stated criteria andmts. T l s ctbts normally needed within the last phase o the peer s scb ths , i.e., h ldecisions are made. owever, it is possible that -stg systm, ssssmts sls by l.

It is important not to mix the two unctions men-tioned above and to keep the two groups separate asmch s ssbl, i.e., t h t lsg mt ssssmts m ths h llparticipate in ranking, prioritisation or consolida-tion meetings in order to make sense o the multiplessssmts ch sl.

our distinct ormats can be used or setting th mt l s s llstt g . T slts bt m th ESF sy

on peer review indicate that across all organisationsthat have responded and considering all undinginstruments, the ormat o choice or constitutingremote and panel membership is option A illustratedin igure 7;thesecondchoicehasbeenidentieds t B.

he nature and scope o the unding instru-ment will determine the required nature o the peer

23. xct s th mmsss lt systm (-). , th l , th xts cctk t -th scss th sl cc,

csss t cmmts scs. It s thscsss t, t th l , hch s ss tth l stg.

Figure 6. Selection and allocation of experts

4.4.2

Number

o experts

required

4.4.3

Criteria

or selection

o experts

4.4.1

Identifcation

o the types

o experts

needed

4.4.4

Allocation

o experts

to proposals


26/88

EuropeanPeerReviewG

uide

24

review bodies, although clearly a two-stage peerreview comprising external assessments ollowedby a review panel deliberation is considered optimaland should be used as much as possible. or smallergmms th sctc sc lghtmodels can be used and thereore a one-stage review

my b s ct

.

2. t II ths s m scc mt thst.

Recommendations

Invite non-European experts (it is also important

to involve experts from emerging regions), both

to ensure availability of scientic expertise and

perspectives, and also to decrease the chances

of conflicts of interest.

Provide concise and clear instructions andguidance to the identied reviewers and panel

members; this should cover all aspects of

their involvement, including their tasks and

contributions, requirements on conflicts of interest,

condentiality, language prociency, etc.

Provide as much advance notice to reviewers

as possible, in order to increase chances of

availability.

Use dedicated and reliable information

management systems including a reviewer

database. The use of a common European

database that would include its own quality

assurance and possibly cer tication system would

clearly help in promoting good practice.

rom the survey a need or a common uropeaneviewer Database (also known as ollege)emerges, which could better meet the growingdemands or highly qualiied and experienceds s th lblty.

Tis is particularly evident or cross-border col-

laborations and mobility o scientists across urope.uch a common database would have clear advan-tages and strengths by creating an opportunity tourther develop the common methodologies, proc-esses, criteria and requirements o peer review, andor the selection and assignment o reviewers acrossdierent nations. Moreover, through availability othis potential shared resource, common approachesin dening and managing conficts o interest couldbe promoted and practised more extensively andcssttly.

As a result o the ESF peer review survey, severalsch gsts h ct th l lg-ness to contribute to constituting such a databaseproviding high-quality reviewers (63.3%) and thento requently use the common database (6.7%)27.

2. cc t (200b),ESF urey Analysiseport on Peer eiew Practices, tcl ct 3., 3..2,Qst 3: m y gsts sct, s th cmm tbs? (g 3.).2. tly th mmss mts tbs xts t mst th th mk gmm.

Whl ths s ts m y s, th tbs c b mlbl t th blc g bs.

2. cc t (200b),ESF ureyAnalysis eport on Peer eiew Practices, tcl ct 3.,Qsts 3 3, bls 3. 3..

Figure 7. Types of reviewers

Remote

(External)Panel

Remote

=

Panel

Remote

Panel

Remote Panel

A. The panel members are entirely different

from the individual/remote reviewers

B.All panel members are also remote reviewers

C. The panel is entirely made up of some

of the remote reviewers

D. The panel is made up of some of the remote

reviewers as well as some additional experts


27/88

EuropeanPeerReviewG

uide

25

owever, some concerns have also been expressed inth sy by th mmbs th ESF Mmbrganisation orum in relation to the cost andms mtg sch systm.

4.4.2 Number o experts required

Te minimum number o reerees and possibly panelmmbs ssg sl ll thormat o peer review, number and size o the pro-posals expected, scientic scope o the programme, th sz th gts qst.

he goal should be to ensure availability odiverse viewpoints, scientic perspectives and schol-arly thinking. Tis is particularly important whenpreliminary assessments are to be generated or asbsqt l stg tst kg.

I gl, th m shl b t t lstthree expert assessments beore a nal decision is

m.or the review panel stage that may ollow

remote assessments, it is recommended to assignrapporteurs rom the panel to each proposal. orlarger programmes, three rapporteurs are essen-tl hl smll gmms ( tms sz,sc, g), t my b sct.

4.4.3 Criteria or the selection o experts

It is important to identiy the right individualswith the responsibility o selecting and inviting the

experts. Tese persons should stay in contact withth s m th bgg t th thprocess. Tey will treat all proposals and al l review-ers in the same way and provide the same support mt t ll.

As mentioned in the previous section, as an ele-ment o good practice in peer review, a sae distanceshould be maintained between panel membershipand individual/remote reviewers. Te choice os s slly th ssblty -gramme ocers and through their own searches or

suggestions rom others such as the review panelsor other advisory committees and boards, andapplicants suggestions o names either or possiblecls xcls.

Te goal should be to attract qualied reviewerswith all the necessary attributes in proportion withthe scope o the task. When required, selection ointernationally recognised and leading scientists schs hs t b cg shl bgiven a high priority or certain programmes, but

2. tls th cmm ctcs css s g

stmts s t II ths cct (200b),ESF urey Analysis eport on Peer eiew

Practices, ht .

this may not be easible (or even necessary) or allpeer review assignments across al l unding instru-ments. Tereore, it is extremely important to paysm ttt t th tst t g th g required expertise and levels o eminence and trackc th s stbl th tsk t h.

election criteria or identication o individ-ual/remote reviewers and panel members have tobe dened and communicated to the responsibleindividuals. Tere are a number o possible eaturest k m h slctg s, sm hch :ctc xcllc, ms thgh ctb-tionsandtrackrecords;

Coverageofthescopeandobjectivesofthecall;Formembershipofpanels(especiallyforchairing

them), it is necessary to include active researcherswho are well established and who have broader

disciplinaryperspectives;Appropriatelevelsofexpertiseinrelationtothe

nature o the task such that authoritative judgmentsandcommentscanbeexpectedwithoutexcess;

Leveloffamiliarity/prociencyofthelanguage

used. Tis requirement applies substantially di-erently rom discipline to discipline and accordingt th cssy lls msty th lggused;

Asolidrecordofpublications:bibliometricindi -ces are increasingly used or assessing publication

track records. are should be taken when applyingthesequantitativemeasures;thesemustbeusedas

complementary inormation and not as sole deter-mining actors in valuing publication track records.

An authoritative and elaborate set o recommenda-tions on the usage o bibliometric in peer review lt s mstl t by th ch Acmy ccs;

Previousparticipationinotherresearchandaca-demicadjudicationcommittees;

Diversity(genderbalance,scholarlythinking,

background,geography,turnover);Independence:externaltothefundingbody;Conictofinterest:reviewersshouldnotbefrom

the same institution as the applicant(s). or verylarge institutions this requirement may be relaxedtosomeextent;reviewersshouldnothavebeen

a research supervisor or graduate student o theapplicant during a period o at least 0 years pre-cedingtheapplication;havecollaboratedwith

the applicant or any o the co-applicants within

2. Isttt c, Acm s ccs,Du bon usage de labibliomtrie pour laluation indiiduelle des chercheurs, Jy20 htt://.cm-sccs./ctlts/lls.htm.


28/88


29/88

EuropeanPeerReviewG

uide

27

4.5 Reader System

I t cm sm th ht bl-ity and inconsistency o the conventional approacheso peer review the so-called eader ystem1 hasbeen proposed as an alternative method. A poten-tial problem with the conventional methods isthe measurement error due to the idiosyncraticresponses when large numbers o dierent assessorseach evaluate only a single or very ew proposals.In the proposed reader system approach, a smallnumber o expert readers are chosen or eachsub-discipline. Te same readers review all theproposals in their remit. Tey will then prioritiseor rank all the proposals they have read. owever,the results o the survey on peer review practicesshow that the reader system procedure is only rarelyapplied, at least or the three most common und-

ing instruments:Indiidual esearch Programmes,areer Deelopment Programmes andInternationalollaboratie esearch Programmes.

4.6 The use o incentivesarticipating in peer review and evaluation exer-cises in varying capacities is now considered as anecessary ingredient o the activities o scientistsand researchers throughout their careers. Tosewho publish and who submit research proposals

create demands or peer review. Tey must there-ore be prepared to contribute their share o peer t mt th lls sl-g-isation required or the selection o the best scienceto receive public unds through peer review andlt.

Items listed below are pertinent to the use octs:eaforementionedself-organisationexpected

th systm s stss, hsbecauseofincreaseddemands;

Someorganisationspaytheirreviewers(bothexternal and panel) to conduct assessments whileothersdonot;

Althoughmonetaryincentivestendtoincreasethechances o acceptance by the targeted reviewers,it is not clear whether or not it will increase thequalityofassessments;

Itisrecommendedtousemonetaryincentivesonly

whenreallynecessary;

3. Jysgh, Msch B (200).32. cc t (200b),ESF urey

Analysis eport on Peer eiew Practices , tcl ct .2,Qst 02: D y c ccg t th systmh gsg th ths stmt? (bl .).

Itisrecommendedtoconsiderothertypesofincen-tives either to the reviewers directly or to theirinstitutes. ome organisations pay the institutesoftheirreviewersforeveryreviewcompleted;

Incentivesshouldhaveamotivationalimpactas

thy mt t b tk cklgmtand appreciation. Tey should not contributeto creating additional adverse side-eects andexpectations such as a race to pay more or bet-terreviewers;compromiseofqualityforquantity;

giving rise to an exaggerated commercial value orpeer reviewing which is inherently an intellectualand scientic endeavour regarded as normal pro-ssl ctbts ch l.

4.7 Expert assessments

nce the experts have been selected, invited and con-rmed as reviewers, and proposals are assigned tothm, th ctl css ssssmt ll bg.

Tere are substantial dierences between theroles o the individual/remote reviewers and thepanel members when conducting their assessment lt tsks (g ).

4.7.1 Brieng

Beore the tasks o both individual/remote review-ers and panel members begin, it is essential that

their assignments are clearly described and com-mct. Ts s mlly thgh bgsessions (possibly using video or teleconerences),orientation sessions, emails and documentationincluding manuals, protocols, recommendations stcts.

Te inormation provided should, as a mini-mum, cover the scope and objectives o the cal l, theprocess o peer review, evaluation criteria and thetimeline to be ollowed. ther relevant inormationthat could be communicated to the reviewers may

contain explicit instructions and guidance on theuse o bibliometric indices, and on providing equalplaying elds through promotion o gender balanceand recognition o individual non-standard careerths ( ..3).

Dg mt lts tl th ssss-ments are submitted, the channel or inormationexchange should be kept open to respond to ques-ts tht my s.

4.7.2 Evaluation criteria

At this stage it is assumed that a clear set o evalu-ation criteria specic to the unding instrumentat hand has been determined and included in the


30/88

EuropeanPeerReviewG

uide

28

promotional material and in the call or proposals.Tese criteria must be sharp, clear and concise. Teyshould be ormulated such that the key aspects othe proposals can be measured in relation to themain scope and objectives o the programme. Te

assessment criteria should not attempt to be exhaus-tive and include criteria that will not be stronglyrelevant and determining in the decision makingcss th g stmt.

Te criteria must be clearly draed and easilyapplicable. All attempts must be made to minimiseroom or diverging interpretations o the criteriaand or ambiguity. valuation criteria in the mostgeneral sense may be grouped into our categories asdescribed below. It should be noted that, dependingon the unding instrument and the variants under

consideration, dierent combinations o these maings ct my b lcbl.

I. Relevance and expected impacts (driven by

programme policy, strategy, mandates, etc.)

Relevanceoftheproposedworktothescopeof

thecall;Broaderimpact(scientic,knowledgecreation,

socio-economic,etc.);Incrementalversustransformativegains;Associatedrisks;

Requestedresources: bgt: lthgh t my b tbl smorganisations to actually scrutinise the overallmts qst by th ss, t s mappropriate to avoid this and instead to assessthe appropriateness o the cost items mentionedbelow that can be used as a measure o conrm-g th qst bgt,

st t, ccss t stct, qmt csmbls,

33. t II t hs l l m tl ct ch stmt.

tl,networkinganddissemination;

Ethicalissues:compliancewithstandardnorms

and ethical practices when dealing with saetyand security, use o animals and human subjects,

environment,embargosandsanctions;Genderbalance:someorganisationspayspecic

ttt t mt g blc th thtl gmms.

II. Scientifc quality

Scientific/intellectualmeritsoftheproposed

research:clear,convincingandcompelling;oroughness:denitionoftheproblemandpro-posedsolutions,reviewofstateoftheart;

Noveltyandoriginality:

ctl, potential or the creation o new knowledge,

xctg s chs, s l tchlgs/mthlgs, innovative application o existing methodolo-

gs/tchlgs s, potential or the creation o new undamental

qsts cts sch, easibility: scientic, technological, access to

inrastructure, recruitment, project timeline,management plan and deliverables, associated

sks, tss th sch mths, -stcts, qmt lk.

III. Applicant

Academicqualicationsandachievementsinrela-tiontotheirstageofcareer;

Researchexperienceandlevelofindependence;Demonstratedexpertiseoftheapplicant(s)in

similarprojects;Applicantsscientificnetworksandabilityto

successully disseminate research ndings, i.e.,knowledgetransferactivities;

Appropriatenessoftheteamofapplicantsinterms

4.7.2

Evaluation

criteria

4.7.3

Scoring

4.7.1

Briefng

4.7.4

Right to reply

Figure 8. Expert assessments


31/88

EuropeanPeerReviewG

uide

29

o availability and complementarities o all the rel-evantexpertiseandsynergies;

Publicationtrackrecord.Itissuggestedtorequire

the applicants to report only on a selected numbero their most relevant and important articles ( to10maximum)insteadofprovidinglonglists;

Bibliometricindices:Asmentionedin4.4.3,forthe use o bibliometric indices, reviewers shouldb xlctly s t ly ths th c only as a complementary tool and not as a soledetermining actor without taking into considera-tion a variety o other actors that can infuencepublication patterns and scientic standing o thelct (s tt 2 g 2).

Whenassessingscientificstandingandquali-cation o the applicants, conscious attentionshl b t l c ths c-cumstances caused by career interruptions and

changes, e.g. due to amily reasons or inter-sectoraland non-academic mobility such as working orsty ( tt 30).

IV. Research environment

Availabilityandaccessibilityofpersonnel,facili-tiesandinfrastructures;

Suitabilityoftheenvironmenttoconductthepro-posedresearch;

Availabilityofothernecessaryresources;Mobilityandcareerdevelopmentaspects.

4.7.3 Scoring

In order to synthesise and compare assessmentso proposals under evaluation, it can be very ben-ecial to assign a scoring scheme to each o theadopted criteria. Most evaluation criteria used orassessment come with a set o multiple choices orthe reviewer to select rom. Tese are normallycomparative statements that carry a numeric oralphabetic score. Te resolution o the scoring sys-tm l ct my y ccg t

th tcl ccmstcs th cll ssss-ment criteria but, generally speaking, a scale o ouror ve statements with determining scores or pointsmaybeused.Forexample:A.forExcellent;B.forVeryGood;C.forGood;andD.forPoor.Itshould

be noted that adopting an odd number o choices ct my l t mlctly ct bssts th ml.

Dierent weighting actors may be applied tothe dierent criteria with a diering degree oimportance. owever, it is advisable to keep such systm s sml s ssbl. It s ls cmm tclclt th g l l th scs t a single overall score or the purpose o comparison

and ranking. A threshold could be set as a cut-oline or the overall scores or or the scores on a givencriterion in order to determine undable versus non-bl sls.

Te relative position o the cut-o line on theull spectrum o scores will have to be determinedby the unding organisation in charge o the pro-gramme and based on the size o the availablebudget. xperts are asked to provide a score orch ct, sbsttt by tt cmmts.Te comments should justiy and be in line withthe given score. eviewers comments should bechecked to ensure usability, legibility and tone olgg b thy s th sts.

Tere are dierent sets or scoring the mainassessment criteria described above that can beadopted, each with slight advantages and disadvan-tgs. I bl 3 xml -t scg

systm s .or example, when measuring the scientic qual-

ity o a proposal, the ollowing denitions can bes:Poor: T ct s ss qtm, th ss ht ksss.Fair: Whl th sl bly sss thct, th sgct ksss.Good: T sl sss th ct ll,lthgh mmts l b cssy.Very Good: T sl sss th ct

y ll, lthgh ct mmts stllssbl.Excellent: T sl sccsslly sss lllt scts th ct qst. Ayshtcmgs m.

vidently, dierent organisations may use otherschemes based on their particular requirements andxstg ctcs. Accg t th scc t th g schms th cll, t my ls bdecided to assign diering weights to some or all th ct.

Budget

When assessing the requested budget or typicalprogrammes the ollowing scoring scheme may bes: ( A): ghly t3 ( B): At2 ( ): Mglly t ( D): It.

3. mmss (200),ules or submission oproposals, and the related ealuation, selection and award procedures, tcl ct 3., . .


32/88


33/88

EuropeanPeerReviewG

uide

31

4.8 Final decision

Te nal stage o a generic peer review systemtyclly cssts th sts tht scb this section and illustrated in igure 9. or specicg stmts, sm th blg blckssuggested here may not apply to all unding instru-ments as urther elaborated in art II. In a general

sense this last stage consists o the prioritisationor ranking o proposals which leads to the naldecisions on the unding o selected applicationss bfy tl bl.

4.8.1 Constitution o the review panel

It is assumed that some o the preliminary work tyg th mmbsh th lstarts at the preparatory stage. At this stage, thepanel needs to be ully constituted with a sucientnumber o experts required to cover the depth

and breadth o the expertise needed. In some pro-grammes, the panel may be created per call andaccording to the disciplines concerned, and in someother cases the panel may be a standing or a dedi-ct cmmtt.

nce the panel has been assembled, the ollow-g t tms shl b cs:I. Terms o reerence, or terms o participation

or the panel members

Conflictofinterestandconfidentialityagree-mts.

II. Mandate o the panel members (some o the

items below may not apply to all instruments)

Reviewandappraisalofexternal(remote)assess-ments;

Prioritisation(e.g., or responsive mode) and/orrankingofproposals;

Recommendationsonfundingthreshold;Recommendationsontheappropriatenessofthe

requested resources, equipment and inrastruc-ture;

Preparationofthepre-meetingassessmentreports

andevaluationsummaries;Preparationoftheconsensusreportssummarising

thedecisionsandfeedbacktoapplicants;Approvaloftheminutesofmeetings.

4.8.2 Prioritisation or ranking meeting

Te ranking or prioritisation meetings are the mostcs sts bth th -stg

th t-stg slct schms.Normally, while the review panel is being con-stituted preparatory work or the scheduling andconvening o the meeting should start. or one-stage selection schemes the panel will make thenal selection o the proposals based on their ownexpert assessments o the competing proposals. ortwo-stage schemes, the panel relies on expert assess-ment by individual/remote reviewers who may ormy t b t th l. T ls in these situations responsible or arriving at con-sensus decisions on the competitive merits o theproposals using external assessments and possiblythe replies rom the applicants to the remote/indi-

4.8.1

Constitution

o the review panel

4.8.3

Funding

decisions

4.8.4

Inorming

the applicants and

other stakeholders

4.8.2

Prioritisation

or ranking

meeting

4.8.5

Possible redress

or appeals

4.8.6

Grant negotiations

and wrap-up

Figure 9. Final decision


34/88

EuropeanPeerReviewG

uide

32

vidual assessments. Te unding decisions shouldnormally ollow and be according to the ranking tst sggst by th ls.

ome o the aspects to be considered are listedbl:

I. Eective planning and running o the meeting

Sucientlylongadvancenoticeandschedules;ProvisionofreliableITtoolsandresourcessuch

that panel members can access the proposals,remote assessments and applicants replies onlineand ideally be able to provide their pre-meetingcomments and evaluations also online. In this way,supporting documentation or the meeting can begeneratedveryeciently;

Agendaforthemeetingallocatingenoughtime

forthedierentpartsofthemeeting;Provisionofallbackgroundandsupportingdocu-

mts t s cmm t s lctc lswherever possible and not to print les i not reallynecessary;

Descriptionoftherequireddeliverablesfromthemeetingandfromthemembers;

Ensurethemeetingisofsucientlengthsuchthatth l s bl t scss ll sls th thq lls ttt s bl t cctth kg/tst th cs-susreport(s);

Ensureanexperiencedscienceocerandpossi-

bly an administrator are assigned and available to sctt st t th mtg.

II. Assigning an authoritative Chair or the panel

ItisveryimportantthattheChairsunderstand

clearlywhatisexpectedofthemeeting;BriengnotesparticularlypreparedfortheChairs

with clear instructions, rules o procedure and listo deliverables need to be communicated to thehs c.

III. Assigning rapporteurs and/or designatedreviewers to all proposals

Normallytwoorthreerapporteursorreviewers

shouldbedesignatedforeachproposal;Theprofileoftherapporteursshouldcollec -

tively cover all disciplinary perspectives that areneeded;

Ensureuniformityandconsistencyofattentionto

all proposals (number o rapporteurs, coverage othescope,etc.);

Normally(especiallyfortwo-stagepeerreview

systems) the members o the panel are not askedt ssss th ssg sls sg th smcriteriausedbytheindividual/remotereviewers;

th thy sk t s mk sso the proposal in relation to the remote assess-ments and, i available, to the applicants repliestotheremoteassessments;

Avoidassigninganexcessivelyhighnumberof

sls t ch mmb. T t lmt

cl y sbsttlly t t th szo the programme and the length o the propos-ls.

IV. Running the meeting (rules o procedure)

Declaringandrecordingconictsofinterest,how

thy lt th y mj bjcts tdecisionsofthepanel;

Ensureactive,all-inclusiveandrichparticipa -tion;

Ensureclearunderstandingonthemodeofcol-lective decision making and approval by the panel:

to decide between unanimous agreement versusconsensusdrivenbymajority;howtodealwith

majorobjectionstonaldecisions;theweight

and priority o the views o rapporteurs on theirproposal versus the views o the other memberso the panel, versus the potential intervention otheChairs;

Inthecaseofhavingmorethanoneortworap-porteurs, it is advisable to assign a lead rapporteurwith the mandate o starting the discussion on agiven proposal by rst providing a brie summary

o the work proposed ollowed by their appraisal otheremoteassessmentsandtheapplicantsreply;

Conducttheprioritisationorrankinginatleast

two rounds. During the rst round, divide theproposals into three priority bins o high (to beunded), medium (may be unded) and low (notto be unded). In consecutive second or possiblythird rounds, the relative position o the propos-als in and across the three groups will be urtherrened and a nal prioritised or rank-ordered listwillbedetermined;

epanelshouldapprovethenalrank-orderedlst.

V. Consensus reports 37

Consensusreportsarepreparedbytherapporteurs

and approved by the panel. Tese reports containstatements on behal o the panel that can beorwarded to the applicant describing the nal

3. Dsgmt s tgl t sctc scss scc ls thgh lctc ctt lg.T, lthgh th css chg csss mg

s c smtms s mbl tsk, t shl bll cssttly ssttly ccc th thg tms c th lbtg g.


35/88

EuropeanPeerReviewG

uide

33

decisions. onsensus reports should not replacethe minutes o the meeting but rather be attachedto the nal approved minutes. onsensus reportsshould strongly refect the relative position o theproposalsontherank-orderedorprioritisedlist;

ecommentsprovidedbytherapporteursshouldbe o high scientic quality, be objective and toth t. Ty shl b sct th ldecision o the panel on the proposal, especially ithat decision is not in line with the overall viewsoftheremoteassessors;

Asfaraspossible,ensurethattheconsensus

ts tt b th mt-ingisadjourned;

eminutesofthemeetingarepreparedaerthe

meeting by the assigned science ocer/adminis-trator and must be approved by the panel beorebg ls. T mts ll ls cl th

nal prioritised or rank-ordered list, as well as theconsensus statements and intermediate changes,cfcts tsts, tc.

4.8.3 Funding decisions

Normally the nal unding decision is made orthe unding organisation by a dedicated commit-t b bs th cmmts threview panel and their suggested rank-ordered orts lst.

It is recommended that the rank-ordered or

prioritised lists are consistently and thoroughlyrespected when unding decisions are being made. Ithe body which makes the nal decision on undingis to be given the right to change the order o propos-ls th k lsts, st th cmmts th l, cl ct jstctsor such changes should be described in advance c s th css st thmsls.

Most unding organisations negotiate theamount o the requested grants with the appli-cts, hl sm gsts th gts

s qst tht y chgs.

Recommendations

The rank-ordered or prioritised list must be

consistently and thoroughly respected when

funding decisions are made.

The feedback from the review panel on the

appropriateness of the requested budgets should

be used if funding negotiations are to be included.

4.8.4 Inorming the applicants and other

stakeholders

Applicants should be inormed o the outcome oth l b g ccss t th cs-ss ts th sl s s s ssbl.

Whether or not the ranking position o a pro-

posal is given to the applicants diers across undingorganisations;this

Date post:	07-Apr-2018
Category:	Documents
Upload:	expunerimedicale
View:	215 times
Download:	0 times

European Peer Review Guide

Documents