Italian Research Evaluation Exercise “VQR 2004‐2010”
and the
University of Milano‐BicoccaBonaria Biancu
University of Milano‐Bicocca
University of Milano‐Bicocca
• Young
(1998),
dynamic
and
multidisciplinary, offering Bachelors, Masters, PhDs
• 16 Departments
• +32,000 students• +900 scholars
and researchers
• 23rd in the Times
Higher
Education
World Ranking
of
universities
under 50 years
old
ICT Area ‐
University of Milano‐Bicocca 2
http://www.unimib.it/english
My
appointment:ICT Area ‐
Research
Information Systems
Office
CAVEAT
• Data published
on ANVUR website on July 31st, 2013
but…
• Still
unstable
/ missing
• Slides based on ANVUR final report (07.31.2013 version)
• Focus on rating, not ranking
• Slides based on ANVUR final report (07.31.2013 version)
• Focus on rating, not ranking
ICT Area ‐
University of Milano‐Bicocca 3
VQR 2004‐2010
• Evaluation of research quality of Italianuniversities and departments (and otherresearch institutions)
• Starting date: November 7th, 2011
• Ending date: July 31st, 2012– June 2nd, 2013
• Publication of VQR results report: July 16th,2013
ICT Area ‐ University of Milano‐Bicocca 4
VQR 2004‐2010 Data Requests• Departments (pre‐ and post‐ University Reform Law and Bicocca
bylaws changes)• ‘Soggetti valutati’ i.e. academic staff in post on the date of the
census (11.7.2011)• 3 scientific products for every researcher (2 / 1 depending on the
scientific age or leaves; 6 for researchers with a joint appointmentboth with Bicocca and a research institution)
• Researchers/academics recruited or advanced• PhD, Specialization, Post‐doc students• Research and financial data:
– Income from competitive grants and contracts with private firms– University expenditure for research projects– Patents, spin‐offs, consortia– Other 3rd stream activities• Self‐evaluation Report
ICT Area ‐ University of Milano‐Bicocca 5
Subject Area indicatorsand weights
w = mean of 4 indices (research
products and costs)
Subject Area wj (%)1 Mathematics 4,00%2 Physics 5,67%3 Chemistry 7,80%4 Earth Science 2,10%5 Biology 10,75%6 Medicine 19,47%9 Computer Engineering 10,73%
10 Literature 7,66%11a Philosophy, Pedagogy 4,94%11b Psychology 1,97%12 Law 5,07%13 Economics 5,12%14 Sociology 2,72%
7, 8a, 8b
Agriculture; Civil Engineering, Architecture ‐12%
Tot.: 0,88
ICT Area ‐ University of Milano‐Bicocca6
• IRAS1: Publications and patents: 0.5• IRAS2: Income from competitive
grants: 0.1• IRAS3: Publications and patents of
researchers recruited/advanced: 0.1• IRAS4: Internationalization: 0.1• IRAS5: PhD/Spec/Post‐doc students:
0.1• IRAS6: Expenditure of the university
for research projects: 0.05• IRAS7: Improvement on VTR 2001‐
2003: 0.05
Internal procedure steps• Working group of technical and administrativestaff– ICT, Research, HR, Accounting&Budget,representative for Departments
• University and Departments’ Faculty Committeesto evaluate research products– Avoid submitting the same publication more thanonce
• All steps of publication selection and submissionmanaged via an internal CRIS software (SURplus)
ICT Area ‐ University of Milano‐Bicocca 7
Strong collaboration between Techical‐Administrative Staff and Scholars‐University Officials
Internal procedure steps
• Working group of technical and administrativestaff– ICT, Research, HR, Account&Financial, deptsrepresentative
• University and departments’ Committees to evaluate research products– Avoid presenting same publication twice
• All steps of publications selection and submissionmanaged via an internal CRIS software (SURplus)
Focus on publications
ICT Area ‐ University of Milano‐Bicocca
Strong collaboration between Techical‐Administrative Staff and Scholars‐University Officials
Strong collaboration between Techical‐Administrative Staff and Scholars‐University Officials 8
Selection and submission of publications
The Scholar
selects, using an
internal procedure
(CRIS module),3 to 20
publications
previously
registered in the
institutional
repository
The Rector + the
University Committee
verify that VQR
criteria are met, no
duplicates have been
selected
(following
negotiation) and give
the OK to the
submission
February to June 2012
ICT Area ‐
University of Milano‐Bicocca9
The Department
Committees verify
that every scholar has
completed the
procedure, VQR
criteria are met and
no duplicates have
been selected
(following
negotiation); finally
check the overall
quality of publications
Publication selection
ICT Area ‐ University of Milano‐Bicocca
10
+ setting thepriority i.e.ranking 3 up to20 publications
Additional information onselected publications
ICT Area ‐ University of Milano‐Bicocca 11
Additional information on selected publications
ICT Area ‐
University of Milano‐Bicocca 12
Here InCites would have
helped! ‐> Having
normalized field citation
score ( / expected), IF
quartile, university
publications ranking
within Subect Areas etc.
Subject Categories
ISI/SCOPUS were used by
ANVUR to assign
publications to the correct
Panel and to evaluate
their impact factor and
citation score in context
Hopefully for the next VQRs we will save
time and perform even better…
PDF upload or send request to the publisher
ICT Area ‐
University of Milano‐Bicocca 13
University/Dept Committees ViewChecking completeness and correctness
ICT Area ‐ University of Milano‐Bicocca 14
University/Dept Committee View Confirming or rejecting the proposed ranking
ICT Area ‐
University of Milano‐Bicocca 15
Package of publications XML record
i.e. NO manualintervention by
academics other than inSURplus…
ICT Area ‐ University of Milano‐Bicocca16
+ one ormore PDF
files…
…
Sent to ANVUR FTP server
… Hence major control over the procedure by the ICT staff
Overall FTE / publications
• 916 academic staff
• 21 departments– 16 after bylaws changes
• 2,309 products (over 2,339 expected)– Small number of low‐
or non‐productive staff
ICT Area ‐
University of Milano‐Bicocca 17
Area# prodotti attesi (n)
% sul totale di Area
# prodotti M % su attesi# prodotti conferiti
# Prodotti P% su
conferiti
1 Mathematics 236 2,71 12 5,08 224 0 0,002 Physics 166 2,75 2 1,20 164 0 0,003 Chemistry 117 1,49 0 0 117 0 0,004 Earth Science 75 2,58 0 0 75 0 0,005 Biology 227 1,73 0 0 227 0 0,006 Medicine 292 1,06 1 0,34 291 4 1,37
missing Subject Areas 7 and 8a/8b (less than 10 products)9 Computer
Engin.33 0,24 2 6,06 31 0 0,00
10 Literature 55 0,39 0 0 55 0 0,0011.a Philosophy,
Pedagogy193 2,09 0 0 193 0 0,00
11.b Psychology 184 5,40 0 0 184 0 0,0012 Law 260 2,07 7 2,69 253 1 0,4013 Economics 360 2,94 6 1,67 354 0 0,0014 Sociology 130 3,09 0 0 130 0 0,00
TOTALE 2328 1,52 30 1,29 2298 5 0,22
Number of publications per Subject Area and % of missing / not suitable
ICT Area ‐ University of Milano‐Bicocca18
Area# prodotti
attesi (n)
% sul totale
di Area
# prodotti M % su attesi# prodotti
conferiti
# Prodotti P% su
conferiti
1 Mathematics 236 2,71 12 5,08 224 0 0,002 Physics 166 2,75 2 1,20 164 0 0,003 Chemistry 117 1,49 0 0 117 0 0,004 Earth Science 75 2,58 0 0 75 0 0,005 Biology 227 1,73 0 0 227 0 0,006 Medicine 292 1,06 1 0,34 291 4 1,37
missing Subject Areas 7 and 8a/8b (less than 10 products)9 Computer
Engin.
33 0,24 2 6,06 31 0 0,00
10 Literature 55 0,39 0 0 55 0 0,0011.a Philosophy,
Pedagogy
193 2,09 0 0 193 0 0,00
11.b Psychology 184 5,40 0 0 184 0 0,0012 Law 260 2,07 7 2,69 253 1 0,4013 Economics 360 2,94 6 1,67 354 0 0,0014 Sociology 130 3,09 0 0 130 0 0,00
TOTALE 2328 1,52 30 1,29 2298 5 0,22
Number of publications per Subject Area and % of missing / not suitable
ICT Area ‐
University of Milano‐Bicocca19
«…
The percentage of missing products over expected ones (1,32%) is lower than the overall universities mean (4,68%), as is
the percentage of non evaluable products over the submitted ones (0,22% against an overall mean of 0,89%), thus denoting a particular care in selecting and submitting products»
ANVUR Final Report – Univ. of Milano Bicocca, page 4th
Quality of research publicationsScore of the publications
Area I Quartile1 0,7 Q22 0,82 Q23 0,89 Q14 0,63 Q25 0,76 Q16 0,65 Q19 0,53 Q4
10 0,75 Q111a 0,69 Q111b 0,66 Q112 0,54 Q213 0,4 Q114 0,6 Q1
7, 8a, 8b < 10 publications ‐
Subject Areas(size‐
independent):1st Quartile: 82nd Quartile: 44th Quartile: 1
ICT Area ‐ University of Milano‐Bicocca20
Quality of research publicationsExcellent publications
Area X Quartile1 1,12 Q22 1,12 Q23 1,17 Q14 1,57 Q15 1,12 Q26 1,34 Q29 0,84 Q4
10 1,63 Q111a 1 Q211b 1,3 Q112 0,92 Q213 1,02 Q214 1,84 Q1
7, 8a, 8b < 10 publications ‐
ICT Area ‐ University of Milano‐Bicocca21
Ratio ≥ 1for all but two Subject Areas
Quality of research publications Ratio of quantity and quality and
IRAS 1
Area R Quartile IRAS1 IRAS1*1001 1,17 Q2 0,031687403 3,162 1,06 Q2 0,029066606 2,93 1,12 Q1 0,016720134 1,674 1,11 Q2 0,028623661 2,865 1,24 Q1 0,021431242 2,146 1,39 Q1 0,014727496 1,479 0,73 Q4 0,001779182 0,18
10 1,14 Q1 0,004477044 0,4511a 1,19 Q1 0,02477823 2,4811b 1,2 Q1 0,064709183 6,4812 1,09 Q2 0,022529619 2,2513 1,24 Q1 0,036191855 3,6114 1,35 Q1 0,041398247 4,14
7, 8a, 8b < 10 publications ‐ ‐ ‐
Subject Areas(size‐
independent):1st Quartile: 82nd Quartile: 44th Quartile: 1
ICT Area ‐
University of Milano‐Bicocca22
ANVUR ranking(size‐dependent):
1st in the segment: 11st Quartile: 6
2nd‐3rd Quartile: 6
IRAS 3
• Mean score for recruited and advanced researchers within the Subject Area; mean score within the Subject Area /
BicoccaArea # Soggetti assunti o
promossi nel settennioCategoria Voto medio assunti o
promossi nel settennio /
Voto medio assunti o
promossi nel
settennio Italia (Rmobil.
Italia)
Posizione nel
segmento
dimensionale
Voto medio assunti o
promossi nel settennio /
Voto medio Area‐Bicocca
(Rmobil. Area)
1 47 Medie 1,14 5°
su 22 1,48
2 25 Medie 1,08 14°
su 20 1,16
3 18 Medie 1,07 9°
su 22 1,16
4 16 Medie 1,26 4°
su 17 1,51
5 50 Medie 1,11 6°
su 20 1,30
6 57 Piccole 0,99 11°
su 17 1,33
8 1 Piccole ‐ ‐ ‐
9 8 Piccole 0,58 14°
su 16 0,66
10 15 Piccole 1,07 6°
su 18 1,16
11 98 Grandi 1,05 2°
su 12 1,14
12 62 Medie 1,00 23°
su 30 1,15
13 87 Grandi 1,11 11°
su 36 1,41
14 31 Grandi 1,26 1°
su 15 1,38
Totale 515
Ratio ≥
1
for all but one
Subject Areas
ICT Area ‐
University of Milano‐Bicocca23
IRAS 1‐7 IRFS 1 and IRFS 2
Università Prodotti attesi
% Prodotti attesi sul totale
IRFS1 x100
16 aree
IRFS2 x100
16 aree
IRFS1 x100 14 aree
IRFS2 x100 14 aree
Milano Bicocca 2.339 1,51995 1,95428 1,94394 1,94902 1,93740
Università
% Prodotti attesi sul
totale delle Università
IRAS1 x w x 100
IRAS2 x w x 100
IRAS3 x w x 100
IRAS4 x w x 100
IRAS5 x w x 100
IRAS6 x w x 100
IRAS7 x w x 100
Milano Bicocca 1,51995 1,71124 1,83654 2,03698 3,04352 1,65572 2,45750 2,37015
IRAS 1‐7
IRFS 1 and IRFS 2
Actually Bicocca governmental budget is underestimated
Here lies the importance of evaluation!
ICT Area ‐ University of Milano‐Bicocca24
This talk is in loving
memory ofLuisella Sironi,THEBIGBOSS
of VQR Working Group
ICT Area ‐ University of Milano‐Bicocca 25
Thank you!
Bonaria BiancuICT Area – University of Milano‐Bicocca
Via Cozzi, 53 – Ed. U520125 Milan (Italy)
Website: http://www.unimib.itEmail: [email protected] phone: +39.0264486236Mobile phone: +39.3386940012