External Panelmeeting

Evaluation of SRC Medicine Panel Reorganization

January 8th, 2009

Program9.00 Welcome and introduction

Mission of the external panel Objectives of the meetingDesignation of a panel chair

9.15 Information about the Swedish Research Council Medicine and background to the evaluation

10.15 Coffee Break

10.30 Project planParameters to analyzeHow has the new organization been received by the scientific coummunity?Who should be approached - and how?Peer review evaluation and development in other organizationsThe Research Bill and peer review

12.30 Lunch

13.30 Practical and logistical details 13.45 Drafting of an interim report14.30 Action plan and conclusion

Milestones/time linesImmediate action plan

15.00 Any other business and remaining questions15.15 Meeting concludes

External panel mission

• Appointed by SRC Medicine on September 9th, 2008Dr. ir. Janna O. de Boer, ZonMw : Netherlands Organisation for Health Research and DevelopmentProfessor Rolf Reed, University of Bergen, NorgeProfessor Ulf Pettersson, Uppsala University

• To evaluate the new review panel organization• To present a final report in May 2009 to the SRC medicine• The report should contain the conclusions of the expert

panel and recommendations to the SRC on it’s review panel organization

• The appointment includes an evaluation of how the new review panels are organized, composed and functioning. The evaluation also includes, as far as possible, an evaluation on the effects of the reorganization for the peer reveiw at SRC Medicine.

• The mission does not include an evaluation of the SRC budget process

Evaluation, continued

The evaluation will be based both on information from the SRC data base and information gained through questionnaires, interviews and an open web consultation

The project is coordinated by the Deputy Secretary General

The panel assisted by a project group from the SRCDr Tove Andersson, Research Officer

Ms Maria Karlberg, Administrative Assistant

Ms Marlene Truedsson Information Officer for Medicine

There will be an interim report to the SRC Medicine for their meeting on February 3rd.

The content of the interim report will be decided at today’s meeting

Objectives of today’s meeting

• Background/information• Decided content of evaluation/parameters to

analyze and how• Drafting of interim report• Set mile stones/time lines

Information about Swedish Medical research, the Swedish Research Council and background to the evaluation

SWEDISH RESEARCH COUNCIL

The Swedish Research Council is a government agency

funding basic research of the highest scientific quality in

all disciplines. The Swedish Research Council has a

national responsibility to support and develop basic

research and promote research innovation and research

communication. The goal is for Sweden to be a leading

nation in scientific research.

RESEARCH FUNDING

• allocates support for Swedish basic research, with an emphasis on attaining the highest quality and bringing about development and renewal. The Swedish Research Council is the largest state body providing funds for basic research in Sweden.

RESEARCH POLICY

• advises the Government in matters relating to research policy. The Swedish Research Council is engaged in strategic issues concerning research and research funding in a national and international perspective.

RESEARCH INFORMATION

• to increase the contacts and dialogue between researchers and the general public in close cooperation with universities and university colleges, organisations, and other players.

• Board• Director General and Deputy Director General• Scientific Councils and Committees in five fields of operations• Secretary Generals • Office for management • Advisory bodies in special fields• Review organisation

ORGANISATION

SWEDISH RESEARCH COUNCIL

The Swedish Research Council is a government agency

funding basic research of the highest scientific quality in

all disciplines. The Swedish Research Council has a

national responsibility to support and develop basic

research and promote research innovation and research

communication. The goal is for Sweden to be a leading

nation in scientific research.

R&D within the higher education sector in Sweden 2007

Medical sciences31%

Engineering23%

Natural sciences19%

Social sciences14%

Humanities7%

Agricultural sciences5%

Not categorised1%

Funding of medical research

• The direct government R&D funding differs between the scientific areas

• For medicine it is 46 percent and includes funding that goes directly to the university hospitals (”ALF-funds”)

• The rest of the funding comes from several other sources, making the funding picture complex for the medical area

Share of direct government R&D funding and external funding per subject area

Data from Statistics Sweden

0

10

20

30

40

50

60

70

80

90

100

Human

ities

Social

scien

ces

Med

icine

Natur

al sc

ience

s

Engine

ering

Agricu

ltura

l scie

nces

%

EU

Foreign non-profit organisations

Foreign companies

Swedish companies

Swedish non-profit organisations

University funds

Research foundations

County councils and municipalities

Research councils and governmentagencies

Direct government funding

Bibliometric analyses of the productivity and citation levels

• Overview of productivitiy – trends• Indicates weak and strong fields as well as areas• Citation statistics quality indicator for aggreated

data based on large numer of publications• Journals classified according to subject fields• 255 subject fields defined, 63 medical; 2 macro

fields: biomedicine and clinical medicine• Covering 6200 journals in medicine• Defined by Thompson Reuters in the Science

Citation Index database• 91% of publications from medical university

institutions including univ hospitals appear in medical journals

Citations of Swedish publications according to subject (as determined by journals/Thompson data base)

1980 1990 2000

0,6

0,8

1,0

1,2

1,4

1,6

1,8

2,0

2,2

Fäl

tnor

mal

iser

at m

edel

värd

e

Medicin

Biomedicin

Klin. medicin

1980 1990 2000

Naturvetenskap

Biologi

Kemi

Geovet.

Fysik

Teknik

1980 1990 2000

Ingenjörsvet.

ICT

Materialvetenskap

Citations, Nordic countries and the Netherlands – competitiveness

Source: Vetenskapsrådets rapportserie 13:2006

0,8

1,0

1,2

Medicin

0,8

1,2

1,6 Naturvetenskap

Fäl

tnor

mal

iser

ad m

edel

cite

ring

0,4

0,8

1,2

0,4

0,8

1,2 Samhällsvetenskap

1985 1990 1995 2000 2005

0,6

1,0

1,4

1,8Skogs- och jordbruksvet, veterinärmed.

0,8

1,2

1,6

2,0

1985 1990 1995 2000 2005

Teknik

DanmarkNorgeSverigeFinlandHolland

0,8

1,0

1,2

Medicin

0,8

1,2

1,6 Naturvetenskap

Fäl

tnor

mal

iser

ad m

edel

cite

ring

0,4

0,8

1,2

0,4

0,8

1,2 Samhällsvetenskap

1985 1990 1995 2000 2005

0,6

1,0

1,4

1,8Skogs- och jordbruksvet, veterinärmed.

0,8

1,2

1,6

2,0

1985 1990 1995 2000 2005

Teknik

DanmarkNorgeSverigeFinlandHolland

1985 1990 1995 2000 2005

Netherlands

Fäl

tnor

mal

iser

ad m

edel

cite

ring

Medicin

1990 1995 2000 2005

0,7

0,8

0,9

1,0

1,1

1,2

1,3

1,4

Sverige

DanmarkNederländerna

Schweiz

USAGreat Britain. 1,09Canada 1,06Finland 1,04

Swedish publications in medical journals – an international comparison

Australia 0,95Belgium 0,95Norway 0,95

Number of publications per year for the 15 largest countries, Switzerland, Denmark, Finland, Norway and Sweden.

Biomedicine

0

10 000

20 000

30 000

40 000

50 000

1980 1985 1990 1995 2000 2005

USA

EU15

J apanUK

0

3 000

6 000

9 000

1980 1985 1990 1995 2000 2005

Germany

France

Italy

Canada

China

Spain

South Korea

0

1000

2000

3000

1980 1985 1990 1995 2000 2005

Netherlands

N

umbe

r of

pub

licat

ions

per

yea

r

Australia

Turkey

Switzerland

Denmark

Norway

Sweden

Finland

Number of publications per year for the 15 largest countries, Switzerland, Denmark, Finland, Norway and Sweden cont’d

Clinical medicine

0

20 000

40 000

60 000

80 000

100 000

1980 1985 1990 1995 2000 2005

USA

EU15

UK

JapanGermany

0

5 000

10 000

1980 1985 1990 1995 2000 2005

France

Italy

Canada

Netherlands

Australia

Spain

0

1000

2000

3000

4000

5000

6000

1980 1985 1990 1995 2000 2005

China

N

um

ber

of

public

ati

ons

per

year

SwedenSouth Korea

Denmark

Turkey

Switzerland

NorwayFinland

Biomedicine

1985 1990 1995 2000 2005

0.8

0.9

1.0

1.1

1.2

1.3

1.4

1.5

1.6

USA

UK

Switzerland

NetherlandsGermanySingapore

1985 1990 1995 2000 2005

0.6

0.7

0.8

0.9

1.0

1.1Denmark

NorwaySweden

Finland

Clinical medicine

1985 1990 1995 2000 2005

0.8

0.9

1.0

1.1

1.2

1.3

1.4

1.5

USA

UK

SwitzerlandNetherlands

Belgium

1985 1990 1995 2000 2005

0.8

0.9

1.0

1.1

1.2

DenmarkSweden

NorwayFinland

Field normalised citation rate 1982 to 2006 for a selection of countries (3-year moving averages). Note the varying scale on the y-axis

Research policy

•Inquiries•Research Bills – the 4 year cycle

SCM: In preparation for the Research Bill 2008

• A long-term research strategy and increased state funding are needed if Sweden is to remain a leading research nation in the future.

• Long-term support of free, research-initiated research is important and should be provided on a broad front.

• All external state resources for health research in Sweden should be handled by a Scientific Council for Health Research in the Swedish Research Council.

• a clear, predictable career structure to attract the most suitable young people to academic research

• Prompt measures are needed to improve the scope for conducting clinical research and non-commercial clinical trials in Sweden.

• strengthening infrastructure for medical research to make it accessible for the very best research.

• Researchers of both sexes must be given the same opportunities of bring their full intellectual capacity forward in their work and to enable Sweden to attain more competitive, productive research.

• Sweden must, more unequivocally than to date, assume its international responsibility for research on major global issues, especially global health.

Clinical research – a great challengeClinical research – the link between the lab and the patient.

Quality of Swedish clinical research risks to fall back (currently #8 in the world), ”good breadth, but lack of cutting edge”

The need to train a new generation of clinical investigators and to boost translational research

The Government has commissioned an inquiry (”World Class”) regarding clinical research led by Prof. O. Stendahl. An action plan was presented in February 2008. A final report will be presented in March 2009.

Action plan clinical researchIncreased health focus and better implementation of research results

• Replace the current SCM with a Scientific council for health research

• Create a national program for implementation and innovation

• The resources should be allocated to the best quality research. The ”ALF-funds” must be followed up and evaluated.

• Develop leadership at the university hospitals• Create national research schools• Create new junior and senior positions for research• Increase support for clinical infrastructure

Recent governmental inquiries with impact on the Research Bill

Forskningsfinansiering – kvalitet och relevans

SOU 2008:30

Karriär för kvalitetSOU 2007:98

Resurser för kvalitetSOU 2007:81

More resources to the Universities needed: Allocation of resources based on activity and quality

Suggested new research council organization: Priority to applied research – present needs of society and industry

Career system: tenure track

Priority not given to basic research. Increased political demands of research outcome and what research should be performed.

”Ett lyft för forskning och innovation”

Regeringens forskningspolitiska proposition 2008/09:50

Research Bill 2008A boost to research and innovation

New resources for research and develpment 2009-2012

2009-2012

Direkta anslag till lärosäten 1 550 Strategiska satsningar: 1 800 varav medicin och hälsa 585 teknik 610 klimat och miljö 535 humsam 30 övrigt 40 Forskningsfinansiärer 675 Infrastruktur 150 Industriforskningsinstitut 200 Innovationspaket 150 Forskningsmoms 300 ESS 150 Regeringens disposition 25 Totalt 5 000

Källa: Faktablad Utbildningsdepartementet

Miljoner kronor

Direct funds to universitiesStrategic areas

Medicin and healthTechnologiesClimate and environm.Humanities and social sci.other

MSEK

Funding organizationsInfrastructure

Industrial research institutesInnovation package

Resarch taxesEuropean Spallation Source

Governement’s own disposalTotal

Vetenskapsrådets organisationOrganization of Swedish Research Council

• Utökat ansvarsområde Increased responsibility– Strategiska prioriteringar av grundforskning Strategic prioritization of basic research– Samordna satsningar mellan ämnesråden coordination of calls between councils

• Styrelsen The Board– Ny roll, utökat mandat och inflytande, ökade krav New role, increased mandate and

influence, increased demands– Ändrad sammansättning Change of composition (inte företräda olika forskningsområden, bred erfarenhet från

samhälle/industri, forskningsbakgrund) not represent research areas, broad experience

• Nya medel kommer till styrelsen ej fördelat på ämnesråd New funds to the board, not to the councils

• Ämnesråd – betoning av forskarrepresentation The councils ,emphasis on scientist representation(en ny roll, ämnesstrategisk) new role, strategic within medical research

• ÄR-M ombildas till Ämnesrådet för medicin och hälsa SRC medicine to become Scientific Council for Medicine and Health

• KFI blir ämnesråd Infrastructure committee to become a council• U kvar som kommitté Educational science remains, as a committee• Expertgrupp för oredlighet inrättas vid CEPN Expert panel for misconduct in

research will be established

Ämnesråd för medicin och hälsaScientific council for medicine and health• Utökat uppdrag – hela medicinska området och

vårdområdet Increased assignment for the whole medical and care area

• Koppla forskning till hälso och sjukvårdens och industrins behov Link research to the needs of the health system and industry

• Ämnesrådets sammansättning ändras, elektorsförfarande (företrädare för industri och sjukvård) The composition should change, include representatives from industry and health care. NB this is already the case

• Ingen tidsangivelse Not communicated when this should start

Ämnesrådet för medicin Swedish Research Council Medicine

• Allmänt om forskningsråden: ”beviljningsgraden bör ökas och projektstöden bör vara tillräckligt stora för att täcka alla kostnader i ett projekt”

• Research councils, in general: the approval rate should increase and the project support be large enough to cover all expenses of a project

• Ansvar för utvärdering av ALF-medel tillsammans med SBU• Evaluate resources allocated by County Councils, toghether with the

Swedish Council on Technology Assessment in Health Care • Utlysa, bedöma, rekommendera strategiska områden 2010-2012• Announce, evaluate and recommend funding for strategic areas 2010-2012

• Psykiatri (25 milj) och vårdvetenskap (20 milj) 2009 egen utlysning• Psychiatry (25 MESK) and Care sciences (20 MSEK) 2009 in separate call• ”VR uppmanas särskilt att uppmärksamma behovet av infrastruktur inom

området” SRC should especially consider needs in infrastructure in the areas

• Kliniska forskarskolor Clinical Doctoral Programmes• Strategiska områden Strategic areas

Universitet och högskolorUniversities and Higher Education Institutions• Ett anslag till forskning och utbildning på forskarnivå• One grant for research and education at doctoral level• Kraftigt ökade basresurser – 2,865 miljarder • Increased resources - 2,865 billion SEK• Tilldelas efter konkurrens• Awarded in competition

– Dels via strategiska forskningsområden – 1,315 Strategic areas– Dels mha indikatorer – 1,55 Using indicators/metrics

• Externa medel• Bibliometri

• Ska bli strategiska och göra egna prioriteringar Universities will be strategic and make their own priorities,more independent

• Ökad ansvar för lokal infrastruktur (exv motsv VR dyr och medeldyr utrustning)

• Increased responsibility for local infrastructure• Postdok-anställningar: 2 år (+ 1år) Post doc positions 2 yr• Full kostnadstäckning Full coverage of costs• Lärarundantaget kvar Teacher´s/scientist’s right to patent • ”genom ökade anslag till UoH respektive strategiska forskningssatsningar kommer

trycket på forskningsråd… att minska när det gäller reguljära ansökningar”

Research and Innovation Bill, 2009–2012 Strategic investments

SEK 1.8 billion (of the SEK 5 billion) will be allocated to what is planned to be a permanent, annual increase in appropriations to research in a number of strategically important areas.

Guiding criteria: 1. Research that can help to find solutions to important global

problems and challenges, 2. Areas in which Sweden is already conducting world-class

research. 3. Areas where companies in Sweden are already conducting

their own research and development,

Research and Innovation Bill, 2009–2012 Strategic investments

Medicine gets about 1/3 of the investments into strategic areas, i.e. SEK 585 million

• Molecular bioscience 190• Stem cells and regenerative medicine 65• Diabetes 70• Neuroscience 70• Epidemiology 25• Cancer 70• Psychiatry 25• Health care research 70

Research and Innovation Bill, 2009–2012

Other important proposals for medical research:

• The Scientific Council for Medicine will be transformed to a Scientific Council for Medicine and Health. This includes larger responsibilities for the whole health area.

• Follow up and evaluation of the research performed with ALF-funds

• Funding of research schools in clinical medicine

Scientific Council for Medicine

Scientific Council for Medicine - Strategy 2009-2012

Vision

The Scientific Council for Medicine within the Swedish Research Council is an internationally highly respected financing organization for medical basic research. The research supported is of highest scientific quality and of great impact for human health both at a national and an international level.

Scientific Council for Medicine - Goals

1. Increased support to the research of highest scientific quality2. Recruit and provide for the coming generation of researchers 3. Collaborate and coordinate resources

4. Promote a major increase in governmental funding of Swedish medical research

5. Promote the same possibilities and rights in medical research regardless of gender or ethnic origin

6. Strengthen public awareness about medical research and its results.

Scientific council for medicineGrants

•Responsive mode•Bottom-up-funding•Few programs

• Project grants

• Positions (junior and senior scientist)

Initiatives

Researcher-initated

Funding principles

•Responsive mode•Bottom-up-funding•Few programs

Grants for research projects by research area

REVIEW ORGANISATION

• Members of the review groups are highly qualified Swedish and foreign researchers.

• The scientific councils’ evaluation panels review applications, rank them on the basis of scientific quality and the competence of the applicants, and propose funding.

• Decisions are taken by the scientific council.

Scientific Council for Medicine

Council AdvisoryPanel (Rådsberedn)

Scientific Council for Medicine Chair: Ann-Marie Begler Secretary General: Håkan BilligDeputy Secretary General : Karin Forsberg NilssonCoordinator: Teresa Karlsson

Evaluation panels

Committee for Research Infrastructure

Number of project grant applications (including junior researcher postitions) (new and

continuation)

0

250

500

750

1000

1250

1500

k200

0

k200

1

k200

2

k200

3

k200

4

k200

5

k200

6

k200

7

k200

8

k200

9

Nu

mb

er

(1127)

(936)(1029)

(922)

Applications k2000-k2006 and running projects (granterade )

0

200

400

600

800

1000

1200

1400

k200

0

k200

1

k200

2

k200

3

k200

4

k200

5

k200

6

k200

7

k200

8

k200

9

An

tal

Nya

Omprövning

garanterade

New

Competetive

renewals

Granted

Project grants

0

5

10

15

20

25

30

35

40

45

2001 2002 2003 2004 2005 2006 2007 2008 2009

r

%

Samtliga

Nya

Omprvning

0

200

400

600

800

1000

1200

1400

0-5 6-10 10-15 16-20 21-25 26-30 31-35 >35

r e fte r disputation

An

tal a

ns

knin

gar

Kvinnor

Mn

Ratio of female scientists applying for project grants

Amount of male and female applicants by career age

0

5

10

15

20

25

30

35

40

45

50

2001 2002 2003 2004 2005 2006 2007 2008 2009

r

%

Kv innor

Mn

Sucess rate in approvals

0

5

10

15

20

25

30

35

40

2001 2002 2003 2004 2005 2006 2007 2008 2009

r

%

Kvinnor

Mn

0

10

20

30

40

50

60

70

80

90

2001 2002 2003 2004 2005 2006 2007 2008 2009

r

%

Kvinnor (nya)

Mn (nya)

Kvinnor (omprvning)

Mn (omprvning)

New and competitive renewals; applications from men and women

Approval rate the first five years after doctoral degree awarded

Women, new appl.Men, new applWomen, competitive renewalsMen, competitive renewals

WomenMen

Medelprojektbidrag (tkr)

583

663

200

250

300

350

400

450

500

550

600

650

700

k199

7

k199

8

k199

9

k200

0

k200

1

k200

2

k200

3

k200

4

k200

5

k200

6

k200

7

k200

8

k200

9

Med

elan

slag

(tk

r)

Project grants average grant size tKr

Average grant size

0

100000

200000

300000

400000

500000

600000

700000

800000

2001 2002 2003 2004 2005 2006 2007 2008 2009

r

Kro

no

r

Kvinnor

Mn

SEKWomenMen

The Swedish Research Council’s gender-equality strategy (p61)

General objectives and guidelines• Attaining an even gender distribution within the Council’s

own review organisation.• Attaining an even gender distribution in the allocation of

research support.• Active efforts to promote gender equality in the research

community.• Analyze outcome after evaluation panel meeting, motivate

priority list

Women in the evaluation panels(>40% since 2000)

Junior researcher position applicatons

0

50

100

150

200

250

300

350

400

k200

0

k200

1

k200

2

k200

3

k200

4

k200

5

k200

6

k200

7

k200

8

k200

9

Budgetår

An

tal

Ansökningar

Beviljade, prelApplied

Approved

Junior researcher positions - awarded

New junior researcher positions - Medicine 2000-2009

0

10

20

30

40

50

60

k2000 k2001 k2002 k2003 k2004 k2005 k2006 k2007 k2008

Ant

al

tot beviljat

kv beviljat

man beviljat

total

women

men

Development of the peer review process

SCM continuously works to develop the peer review system

• Transparency: – web based information– dialogue with the scientific community

• biannual road-shows)• regular meetings with the Deans• Information meetings for junior scientists

– SCM Magazine– SCM newsletter– web based nomination of evaluators since 2008 open to all (now open)

• Budget process– Working group of the SCM– SCM strategy meeting focused around budget process

• Rating system– Developed feedback to applicants

• Mean values of individual criteria scores (3 or 4)• Extended information on wpv mean/max/median score values of the panel

– New criteria since 2008 (result of the SCM working group for quality assessment)

– Scoring system presently investigated• Instructions to reviewers

– Improve written instructions– One day information meeting– Half day panel chair information

SCM Evaluation panels(before 2001 MRC)

• major changes 1992 and 2005• 1992: After 30 years of unchanged evaluation

panels – due to increased application load and change of

research profiles

• 2004-2005: SCM working group to design a model to adopt to disease-oriented review panel organization

New organization of evaluation panels from 2006

• To assure continued high quality assessment in a flexible organization where the number of applications directs number of panels and the allocation of grants to the disease studied is more visible

• Meet with stricter requirements for handling conflicts of interest

• Possibility of a new budget process

• Evaluation after three years, which is this year!

Evaluation panels

• 19 panels (before 13)• 5 reviewers per application out of which 1 is the

evaluator (same)• Every application is reviewed in an evaluation

panel

Recruitment of panel members from Sweden and the other Nordic countries • Panel members appointed by the Scientific

Council for medicine (same)• 2008: 39% new reviewers in the 17 panels (38%

2007)• NEW 2008: Web nomination of panel members• Appointment on 3 year-basis with a maximum 6

year term (Swedish scientists “step out” if they apply for project grants and can then come back the next year)

• Appoint Nordic experts as regular members of Swedish evaluation committees.

• Nordic experts may serve a 2x3 year term• Reviewers from Denmark, Finland, Iceland and

Norway constitute 20%

Evaluation panels 2008A1 Musculo-skeletal diseases, Oral health and Maxillofacial diseasesA2 Musculo-skeletal diseases, Anaesthesiology and Radiology

B1 Endocrinology/Metabolic diseases and Gynaecology, Reproduction/perinatalB2 Endocrinology/Metabolic diseases including Gastrointestinal diseases

C1 Infections and Global healthC2 Infections, Respiratory tract diseases and Allergy including DermatologyC3 Infections

D1 Nervous system diseases and PsychiatryD2 Nervous system diseases and PsychiatryD3 Nervous system diseases including Sensory Organs

E1 Cardiovascular and Urogenital diseases, Transplantation and diseases of haematogenous organs

E2 Cardiovascular and Urogenital diseases, Transplantation

F1 Basic disease mechanisms: Molecular, cellular and biochemical aspectsF2 Basic disease mechanisms: Molecular, cellular and biochemical aspectsF3 Basic disease mechanisms: Molecular, cellular and biochemical aspects

G1 Public Health and Care-Sciences/NursingG2 Public Health and Care-Sciences/Nursing

Collaboration grantsInfrastructure2 Post doc panels (spring and fall call)

How are the applications distributed over the panels 2008?

A Rörelseorganens sjukdomar, Oral hälsa, Käkens sjukdomar, Anestesiologi ,Radiologi

B Endokrinologi, Metabola sjukdomar, Gynekologi, Reproduktion/perinatal

C Infektion, luftvägarnas sjd, Allergi, Global hälsa, Hudsjukdomar

D Nervsystemets sjd, psykiatri, Sinnesorganen

E Hjärt-kärlsjukdomar, Blodsjukdomar, Urogenitala sjukdomar, Transplantation

F Grundläggande sjukdomsmekanismer; molekylära, cellulära och biokemiska aspekter

G Folkhälso- och vårdvetenskap

Disease studied: 61%Basic mechanisms of disease: 24%Care sciences and public health : 15%

Applications 2008

A9%

C13%

D16%

E11%

F24%

G14%

B13%

A

B

C

D

E

F

G

Number of project grant applications per panel 2006-2008

Antal ansökningar om projektbidrag per beredningsgrupp 2006-2008

0

10

20

30

40

50

60

70

80

A1 A2 B1 B2 C1 C2 C3 D1 D2 D3 E1 E2 F1 F2 F3 G1 G2

beredningsgrupp

an

tal

an

sökn

ing

ar

Projekt 2006Projekt 2007Projekt 2008

SCM decision for Evaluation panels 2009

• No major change awaiting the expert panel evaluation

• Project grant applications: A, B, C1-2, D1-2, E1-2, F1-3, G1-2

• Junior researcher position applications separate panel (as post doc applications)

• Contiune triage procedure, next year also for junior researcher positions

• Increase number of experts in Collaboration grant panel

• Add one expert to the infrastrucure panel (large database expert)

The review process

Council Advisory Paneland

Scientific Council

April June-Aug Aug-Sept Nov

Applicationssubmitted

Distributionof applications

Reviewing Evaluation panels

Nov 3Discuss and weigh the preliminary proposals of the evaluation panels

Nov 4Final decision

Dec

Communicationof decisions

May-June

”Triage” process of applications 2008• All applications are assessed by five panel

members• All applications will recieve the same quality

assessment by individual panel members.• Only applications with a reasonable chane to be

considered for funding will be discussed further at the panel meetings

• The margin to ”sort out” applications will be considerable

• These applicants will receive wpv and information about wpv in the panel, but not an individual written assessment

• The “triage” will allow more time for discussions • The “triage” will be evaluated after the application

process this year

Triage, how will it be done?1. All assessments, including preliminary evaluations, are reported for all

applications belonging to the evaluations panel in VR-Review no later than two week sbefore the panel meeting.

2. The administrstion provides a list with the joint rating of the applications to the panel chair who is responsible for conveying the information to the panel members (avilable in VR Review).

3. During a joint telephone meeting, the panel members agree on which applications should not be subject to further discussions at the regular meeting. The margin for sorting out applications shouldbe considerable The meeting should take place no later than a week before the regular panel meeting. All panel members must be present, otherwise the meeting is not valid.

4. At the panel’s regular meeting, the triaged applications are normally not subject for further discussions. It is however possible for a panel member to, at any time and without specific grounds, bring a triaged application back into the panel for discussion.

5. After the panel meeting, the preliminary evaluations for the triaged applications will not be finalized as evaluations. An applicant of a triaged project grant application will only receive the weighted point value and other potential information that is distributed to the applicants, but will not receive an individual written evaluation. A standard cover letter explaing the triage principle will be enclosed.

Project plan

Parameters to analyzeQualitative data• How has the new organization been received by the scientific

coummunity? • Experiences gained from questionnaires 2007 and 2008• Who should be approached - and how?

applicants, reviwers, panel chairs, the SCM, the secretariatquestionnaires, interview, web consultation

Quantitative data• Numbers of applications per reviewer, per panel• Approval rates with regard to scientfic area, gender, career age, etc

Peer review evaluation and development in other organizations

The Research Bill 2008 and peer review• Peer review as an instrument for measuring quality of scientific proposals• Demands on a Scientific Council for Medicine and Health. Can this be met

by the current organization? If not, what changes are needed?

45. Did you participate in an Evaluation Panel at the Swedish Reseach CouncilMedicine in 2005 or earlier, i.e. before the reorganization of Evaluation Panels?1. Yes 2. No  

2007 2008

no

yes

46 Has your workload changed due to the reorganization of the grant review process? (2007)

1 Increased a lot  (1) 2.0 % 2 Increased a little (2) 4.0 %3 Unchanged  (4) 8.0 %4 Decreased a little  (3) 6.0 %5 Decreased a lot  (3) 6.0% 0 I did not participate in an Evaluation Panel before the reorganization  (37) 74.0 %  Total: (50)No. of missing replies: 186

2007 2008

47 How well do you think the new grant review process is working out in terms of categorizing the applications?1 Very well  (7) 13.5 %2   (32) 61.5 %3   (7) 13.5 %4   (6) 11.5 %5 Very badly  (0) 0.0  % Total: (52) No. of missing replies: 184

1 Very well (16) 23.5 %2 (28) 41.2%3 Neither well nor badly (12) 17.6 %4 (3) 4.4 %5 Very badly (1) 1.5 %0 No opinion (8) 11.8 % Total: (68) No. of missing replies: 178

2007 2008

42 This year (2008), a ”triage” process was tested. Only applications with a reasonable chance to be considered for funding were discussed further at the evaluation panel meeting. Did this change your workload? 1 Increased a lot  (2) 2.9 % 2   (4) 5.9 % 3 Unchanged  (20) 29.4 %4   (26) 38.2 % 5 Decreased a lot  (4) 5.9 %0 No opinion  (12) 17.6 %   Total: (68)

43. What is your overall opinion of the triage procedure? (2008)1 Very good  (32) 47.1 %2   (17) 25.0 %3 Neither good nor bad  (6) 8.8 %4   (2) 2.9 %5 Very bad  (3) 4.4 % 0 No opinion  (8) 11.8 %   Total: (68) No. of missing replies: 178

44 What is your overall opinion of this year’s peer-review process? (2008) NB Replies for reviewers at all councils and committees 1 Very good  (86) 35.0 %2   (125) 50.8 % 3 Neither good nor bad  (32) 13.0 %4   (3) 1.2 %5 Very poor  (0) 0.0 %0 No opinion  (0) 0.0 %  Total: (246) No. of missing replies: 0

A. Approval rate comparison for women and men applicants

41,3

44,843,1

37,3

33,9

30,4

34,8 34,5

31

34,6

40,4

35,7

28,5 29,4

25,7 26,5

30,7

25,3

43,846,8 47

41,2

36,233,1

38,936,9

34,7

0

5

10

15

20

25

30

35

40

45

50

2001 2002 2003 2004 2005 2006 2007 2008 2009

year

%

Approval rate (%) Approval rate for women (%) Approval rate for men (%)

Figure 1a. Approval rate for women and men including all project grant applications, both competitive renewal and new applications (%).

Figure 1b. Approval rate for women and men for applicationsfor competitive renewal (%).

65,7

78,375,8 76,4 74,7 74,3

82,1 80,577,8

81,3 80,983,3

79,576

81,584 82,7

74,6

0

10

20

30

40

50

60

70

80

90

2001 2002 2003 2004 2005 2006 2007 2008 2009

year

%Approval rate for competitive renewal for women (%)

Approval rate for competitive renewal for men (%)

Figure 1c. Approval rate for women and men for new applications (%).

20,1

26,324,3

16,3

11,1

14,9 15,2 16,1

12,4

15,9

24,927,3

20,5

13,614,8

17,7 17,215

0

5

10

15

20

25

30

2001 2002 2003 2004 2005 2006 2007 2008 2009

Approval rate for new applications for women (%)

Approval rate for new applications for men (%)

B. Application statistics per evaluation panel over time

Figure 2a. Number of applications reviewed per evaluation panel before panel reorganisation (2001-2006) and after panel reorganisation (2007-2009).

Number of project grant applications per evaluation panel

0

20

40

60

80

100

120

140

160

180

2001 2002 2003 2004 2005 2006 2007 2008 2009

Cellbiology1

Cellbiology2

Cellbiology3

Public health and nursing

Clinical science 1

Clinical science 2

Clinical science 3

Clinical science 4

Medical chemistry

Microbiology and immunology 1

Microbiology and immunology 2

Odontology

Phychiatrics

System physiology and pharmacology

M-A1

M-A2

M-B1

M-B2

M-C1

M-C2

M-C3

M-D1

M-D2

M-D3

M-E1

M-E2

M-F1

M-F2

M-F3

M-G1

M-G2

Figure 2b. Approval rate per evaluation panel before panel reorganisation (2001-2006) and after panel reorganisation (2007-2009) (%).

Approval rate per evaluation panel

0

10

20

30

40

50

60

70

2001 2002 2003 2004 2005 2006 2007 2008 2009

Cellbiology1

Cellbiology2

Cellbiology3

Public health and nursing

Clinical science 1

Clinical science 2

Clinical science 3

Clinical science 4

Medical chemistry

Microbiology and immunology 1

Microbiology and immunology 2

Odontology

Phychiatrics

System physiology and pharmacology

M-A1

M-A2

M-B1

M-B2

M-C1

M-C2

M-C3

M-D1

M-D2

M-D3

M-E1

M-E2

M-F1

M-F2

M-F3

M-G1

M-G2

C. Statistics for subject areas (defined by the applicants)

Table 3a. Number of applications within specific subject areas (3-year intervals).

Number of applications within specific subject areas

0

50

100

150

200

250

300

350

400

450

Anasth

esiol

ogy,

inte

nsive

car

e an

d tra

uma

Bioch

emica

l stru

cture

and

met

abolis

m

Cance

r

Cell a

nd m

olecu

lar b

iolo

gy

Coagu

latio

n an

d th

rom

bosis

Derm

atol

ogy

Develo

pmen

tal b

iolog

y

Diabe

tes

Diges

tive

syst

em a

nd kid

ney

Drugs

and

dru

g ad

dicti

on

Endoc

rinolog

y

Enviro

nmen

tal m

edici

ne a

nd to

xicolo

gy

Gen

etics

Heart

and

blood

vess

els

Gen

eral

medic

in

Micr

obio

logy,

imm

unol

ogy a

nd in

fecti

ous

Mus

culo

skel

etal

syst

em

Nervo

us sys

tem

Nursin

g

Odo

ntol

ogy

Pharm

acy

Prena

tal a

nd p

erin

atal

dise

ase

Phsyc

iatri

c dise

ase

Public

hea

lth

Radio

logy

and

imag

ing

tech

nique

s

Repro

duct

ive sy

stem

Respir

ator

y sys

tem

Senso

ry o

rgans

Subject area (defined by applicant)

Nu

mb

er

of

ap

plic

ati

on

s

2001-20032004-20062007-2009

Figure 3b. Distribution of applications in specific subject areas (3-year intervals)

Distribution of applications in subject areas (%)

0,00

2,00

4,00

6,00

8,00

10,00

12,00

14,00

16,00

18,00

Anasth

esiol

ogy,

inte

nsive

car

e an

d tra

uma

Bioch

emica

l stru

cture

and

met

abolis

m

Cance

r

Cell a

nd m

olecu

lar b

iolo

gy

Coagu

latio

n an

d th

rom

bosis

Derm

atol

ogy

Develo

pmen

tal b

iolog

y

Diabe

tes

Diges

tive

syst

em a

nd kid

ney

Drugs

and

dru

g ad

dicti

on

Endoc

rinolog

y

Enviro

nmen

tal m

edici

ne a

nd to

xicolo

gy

Gen

etics

Heart

and

blood

vess

els

Gen

eral

medic

in

Micr

obio

logy,

imm

unol

ogy a

nd in

fecti

ous

Mus

culo

skel

etal

syst

em

Nervo

us sys

tem

Nursin

g

Odo

ntol

ogy

Pharm

acy

Prena

tal a

nd p

erin

atal

dise

ase

Phsyc

iatri

c dise

ase

Public

hea

lth

Radio

logy

and

imag

ing

tech

nique

s

Repro

duct

ive sy

stem

Respir

ator

y sys

tem

Senso

ry o

rgans

%

2001-20032004-20062007-2009

Figure 3c. The approval rate of applications within each subject area (%).

Approval rate within subject areas

0

10

20

30

40

50

60

70

Anasth

esiol

ogy,

inte

nsive

car

e an

d tra

uma

Bioch

emica

l stru

cture

and

met

abolis

m

Cance

r

Cell a

nd m

olecu

lar b

iolo

gy

Coagu

latio

n an

d th

rom

bosis

Derm

atol

ogy

Develo

pmen

tal b

iolog

y

Diabe

tes

Diges

tive

syst

em a

nd kid

ney

Drugs

and

dru

g ad

dicti

on

Endoc

rinolog

y

Enviro

nmen

tal m

edici

ne a

nd to

xicolo

gy

Gen

etics

Heart

and

blood

vess

els

Gen

eral

medic

in

Micr

obio

logy,

imm

unol

ogy a

nd in

fecti

ous

Mus

culo

skel

etal

syst

em

Nervo

us sys

tem

Nursin

g

Odo

ntol

ogy

Pharm

acy

Prena

tal a

nd p

erin

atal

dise

ase

Phsyc

iatri

c dise

ase

Public

hea

lth

Radio

logy

and

imag

ing

tech

nique

s

Repro

duct

ive sy

stem

Respir

ator

y sys

tem

Senso

ry o

rgans

% 2001-20032004-20062007-2009

D. Statistics based on applicant age

Figure 4a. Number of applications received from different age categories.

Applicant age distribution (years after PhD)

0

100

200

300

400

500

600

700

800

0-5 6-10 11-15 16-20 21-25 26-30 31-35 >35

Years after PhD

Nu

mb

er o

f ap

plic

ants

2001-2003

2004-2006

2007-2009

Figure 4b. Approval rate for different age categories.

Approval rate for age categories

0

10

20

30

40

50

60

70

80

0-5 6-10 11-15 16-20 21-25 26-30 31-35 >35

Years after PhD

Ap

pro

va

l ra

te %

2001-2003

2004-2006

2007-2009

Milestones/time lines

January Start questionnaires/interviews

Web based consultation opens

February 3rd SCM meeting Interim report (presentation of project outline)

February (mid) Panel phone conference

February (end) Questionnaries concluded, web consultation closes

March Meeting with representatives of panels etc

April Analysis of data, drafting of report

May (early) Panel conference to finalize report

May 18-19th SCM strategy meeting presentation of final report

Immediate action plan:

Reviewing and scoring (p42)

Project research grant• Project 1-7• Feasiblity 1-7• Project management 1-7• Results (only for competitive renewals) 1-7

Each criterion shall be evaluated and scored independently.

ProjectThis dimension relates to the overall design of the project, its

research questions and hypotheses, and any influence it may exert on scientific knowledge. Is the project original or groundbreaking? Does it challenge prevalent opinions or practice? Does the project include an innovative hypothesis or seek to remove key barriers to further progress in the research field concerned? Does the project entail new ideas, approaches or interpretations? If the aims of the project are attained, will it substantially enrich knowledge in the field concerned or our understanding of health and ill-health?

Score: 1–7

7 = Outstanding, innovative project of highest scientific importance

1=Project with no scientific relevance and/or of poor scientific quality

Feasibility• This aspect includes the materials, methods, resources, finances and

time schedule of the project. Is the proposed procedure optimal in terms of scope for implementing the project, when it comes to general design, choice of methods, equipment, work inputs and time? Are the total resources realistic in relation to the scale of the project described? Does the methodological know-how exist to implement the project in the manner describes? Are the materials, patients and methods well adapted to the hypothesis or research question concerned? Are the numbers of measurements, tests, animals, patients or experimental subjects optimal for tackling the research issue addressed? Whether the materials and methods are appropriate for the purpose should be regarded as more important than the uniqueness of the method as such or the material. If unique material is not required or if well-established methods are fully adequate, this is not a disadvantage. The key aspect is whether the research question is well addressed in terms of methodology, so that the project yields reliable answers with minimal resource consumption. What are the strengths and weaknesses of the route chosen? Are there any alternative routes to the destination in question if the one selected does not lead to success? Are the separate subprojects well founded and properly integrated with the overall objectives of the project?

7 = Materials/patients and methods, resources, infrastructure and scientific environment are all optimal and aims are well integrated

1= The project is unrealistic

Project management• This aspect includes the skills and qualifications of the applicant and other people

involved, and also the research environment and collaboration. The project manager's skills weigh very heavily in the assessment. Does the applicant have the independence, experience and contact network that are required for the project to be implemented in the manner, and during the period, specified in the application? Do the applicant's previous publications show a distinct line of research? Independence is most clearly evident when applicants have publications to show in which they are the senior authors; when they have acted as supervisors (especially principal supervisors) to PhD students up to their disputations (public thesis defences); and when they have supervised doctoral students. For researchers still in the early stages of their careers, signs of independence also include, for example, publications as first author, preferably without participation of their own supervisors, and the existence of PhD or postdoctoral students of their own. Having been awarded grants of their own as principal applicants following peer review may also indicate independence. In applications from researchers who belong to teams or groups, it should be clearly apparent how the project in question is distinguished from the other projects under way in the team or group. It is common for young researchers to liberate themselves and develop research lines of their own by hiving off from larger projects. For the assessment of independence, it is essential for both junior and senior researchers to describe the boundaries between these projects clearly in their applications.Do the team members working on the project complement one another in their skills in an optimal way? Are the roles and responsibilities of all the project staff clearly described? Is there a critical mass of researchers in the environment concerned? An account of these aspects under a separate heading, ‘Project management', in the research plan facilitates a favourable assessment

Score: 1–77 = Established/Independent PI with expertise in the research

area, performing world-leading research1= PI lacking independence, or independent PI without

knowledge in the research area

Results (only for competitive renewals)

• The concern here is how far the project has, during the grant period, significantly contributed to an increase in knowledge in the field of medical research. Scientific publications in international journals are the key measure of the findings and importance of a project, and the assessment emphasises quality rather than quantity. Assessment focuses primarily on publications relevant to the project concerned. Other indicators of results and their importance may, for example, be amendments to national or international guidelines for medical diagnostics or treatment.

Score: 1–77= Top results including key authorship of several published

articles of highest scientific quality1= No published articles or other results of scientific relevance

Types of grants in Medicine 2008 (p5)

• Project Research GrantsThis form of grant is intended to cover the performance of a defined research task. The maximumgrant period is five years, but up to three years is the norm. The grant may comprise funding fortechnical staff, salaries for other staff, equipment up to a maximum cost of SEK 170 000, materials,publication costs and travel. Funding for his/her own salary cannot be included.In the subject area ofMedicine, a person can be the principal applicant for only one approved Project Research Grant in anygiven period.

• Grant for Junior Research Position in general fieldsJunior Research Positions are obtainable in all the fields supported by the Scientific Council forMedicine. The grant period is four years. A person applying for a Junior Research Position Grantautomatically applies for a Project Research Grant and a Starting/Establishment Grant. The amount ofthe Project Research Grant for successful applicants for Junior Research Positions in 2008 is SEK 325000 a year. The Starting/Establishment Grant, which is paid during the first year, totals SEK 500 000.

• Grant for Junior Research Position in Care SciencesThis Junior Research Position Grant relates to research in the priority field of Care Sciences. This fieldcomprises theory, methods and techniques for the study of problems and measures in healthcare andmedical contexts related to human health, quality of life and capacity for activity, and also thehealthcare and social care of people in various situations and settings. A Project Research Grant and aStarting/Establishment Grant is included as above.

• Grant for Junior Research Position in a Clinical Environment, 50%This Junior Research Position Grant relates to research carried out by people who have obtained PhDsand are working in clinical settings, such as doctors, nurses, physiotherapists, occupational therapistsand dentists. The position must be combined with half-time employment in a clinical environment. AProject Research Grant and a Starting/Establishment Grant is included as above.

• Collaboration Grant for translational and interdisciplinary researchThe purpose of this grant is to expand and strengthen mutual collaboration among researchers invarious fields, including clinical research. Three to five researchers (one coordinator and two to fourparticipating researchers) apply for the grant. At least one of the applicants must be a ‘youngresearcher’ (who obtained his or her PhD not more than six years ago). The majority of the applicantsshould come from the same HEI. One or more of the participants may come from an HEI abroad. Themajority of the applicants must have current research funding obtained in national or internationalcompetition.

Effekt 2008-2012 av periodisering av anslag 2008 (olika bidragsformer)

0,00

50,00

100,00

150,00

200,00

250,00

300,00

2008 2009 2010 2011 2012

Mil

j kr

Projekt - en huvudman

Samverkandbidrag

startbidrag (forskare + foass)

Forskartjänster (hel +halv)

postdoc + repatriering

Foasstjänster

Internationalisering (gäst+postdoc+konf)

Dyrbarutr

Blå toner = projekt,

Gröna toner = tjänster,

gul = international-isering,

röd =medeldyr-utrustning

Evaluation Policy 2008, Medicine

• The number of projects financed by the Scientific Council’s own budget should not increase

• The average funding level should increase

• The grant period should be 3 (to 5) years

• Funds should be allocated to individuals using the principle of one project per principal investigator