University evaluation systems: The experience of the University of Palermo

Antonella PlaiaRector’s Delegate for the Evaluation

Palermo April 10-11 2014Antonella Plaia

ENQA Standards and Guidelines for Quality Assurance in the European Higher Education Area

“Institutions should have a policy and associated procedures for the assurance of the quality and standards of their programmes and awards. They should also commit themselves explicitly to the development of a culture which recognises the importance of quality, and quality assurance, in their work. To achieve this, institutions should develop and implement a strategy for the continuous enhancement of quality.The strategy, policy and procedures should have a formal status and be publicly available.Formal policies and procedures provide a framework within which higher education institutions can develop and monitor the effectiveness of their quality assurance systems.Institutions should ensure that they collect, analyse and use relevant information for the effective management of their programmes of study and other activities.”

Palermo April 10-11 2014Antonella Plaia

Evaluation in UNIPA:1990-2000Campus Project • Experimental project (without funding). • First experience (for UNIPA) of Self-Evaluation Report• Deming “Plan – Do – Check – Act”:  iterative four-step management method

used for the control and continuous improvement of processes and products• Peer Review

The Ministry of Education, using European Funds, funded 7 short degrees (diploma).

• Not only financial reporting• The first experiments of performance evaluation: tracking the careers of

students. • Since 2000 UNIPA has a delegate to evaluation activities, UNIPA declares

its Mission, the Charter of Services is issued.

Palermo April 10-11 2014Antonella Plaia

CampusOne Experience

CampusOne national project , a project funded by the government with the use of UMTS funds (Law 388 - 31.12.2000) and managed by the CRUI (Conference of Italian University Rectors ) .

The aim of the national project is to be able to provide the universities - in the three years 2001-2004 – with models and tools with which to implement, measure and improve the process of reform introduced by the DM 509/99 (the so called 3+2 reform of universities – bachelor + master).

The goal is to transfer to the new three-year degrees the standards applied to university degrees financed by Campus project.

Evaluation in UNIPA:2001-2009

Palermo April 10-11 2014Antonella Plaia

CampusOne Experience

UNIPA participates with 5 programs, and, during the second year of the project, 9 new programs entered (but without funding), in order to experiment the main elements provided by the project: the teaching management , the stage activities and the evaluation.

The objective was achieved through the following specific actions :• Educational Management (Educational tutors, 14 educational managers are now

present in UNIPA)• Certification of language and computer skills (start-up UNIPA Language Center)• Stage and internships (start-up of Liaison Office and Stage and Internship Office)• Evaluation (development of a culture of teaching evaluation)• Relationship with the territory (involvement of stakeholders)• Strengthening of ICT (Information and Communication Technology) services (start-up

of UNIPA Museum network)

Evaluation in UNIPA:2001-2009

Palermo April 10-11 2014Antonella Plaia

EUA experience

In 2008 UNIPA participates to the EUA’s Institutional Evaluation Programme (IEP).IEP provides external institutional evaluation services to higher education institutions in

the context of their own aims and profiles since 1994.The evaluation begins with a self-evaluation process and a self-evaluation report

conducted by the institution, followed by two site visits by an evaluation team, and is concluded with a final report by the evaluation team highlighting good practices identified and providing recommendations for improvement.

The focus of an IEP evaluation is the institution as a whole and not the individual study programmes or units. Recommendations and insights are provided on the institutions’ structures, processes, policies and culture, to enable them to perform the full range of their activities in line with their strategic plans and objectives, and build the capacity to address change processes.

Evaluation in UNIPA:2001-2009

Palermo April 10-11 2014Antonella Plaia

EUA experience

The internal quality processes for which clear and defined responsibilities exist andwhich appear to be more widely shared within the Institution are: 1. Teaching evaluation on the basis of student assessment (since 1999)2. Research activity evaluation (since 2006)3. Evaluation of the objectives of technical-administrative staff (since 2004)

UNIPA Actions after EUA report • 82 departments: too many, now 20 • 12 faculties: too many, now 5 schools• Set out a Strategic Plan followed by a structured implementation plan with “mile-

stones” and a monitoring procedure.• Students are now involved in many commissions and groups.• Measures have been taken to develop the graduates’ entrepreneurial skills (ARCA)

Evaluation in UNIPA:2001-2009

Palermo April 10-11 2014Antonella Plaia

Since 2010 a competitive fund is distributed by Italian University Ministry to reward both teaching and research quality. The incidence of this competitive fund on the total amount of fund distributed increased since then.

Teaching evaluation

Teaching quality is evaluated according to 2 performance indicators, one that considers the amount of “regular-active” students and another based on the amount of credits earned by students.

Since 2010 UNIPA started to monitor these indicators at different levels:• University• Faculty• Single Program

The results of this monitoring process are shown to heads of faculties and programs every year, usually after the Ministry has assigned the competitive fund.

Evaluation in UNIPA:2010-

Palermo April 10-11 2014Antonella Plaia

Palermo April 10-11 2014Antonella Plaia

Palermo April 10-11 2014Antonella Plaia

Research assesment (since 2006)

Research database where each professor/postdoc/phd student uploads his/her research products, research projects, …14 research assessment commitees that evaluate, each year, UNIPA research.

The results of the evaluation process allow to join competitions for research funds, to join phd board, to become postdoc tutor.

Since 2010 a competitive fund is distributed by Italian Ministry of Education to reward both teaching and research quality. The incidence of this competitive fund on the total amount of fund distributed increased since then.

Palermo April 10-11 2014Antonella Plaia

Results of the ANVUR VQR 2004-2010

VQR (Evaluation of Research Quality) is the Italian research assessment exercise that ANVUR (the National Agency for the Evaluation of the University and Research system) carried out on behalf of the Italian Ministry for Education.

We use VQR results to see how each department performed.We have now aligned internal research assessment to VQR criteria.

Palermo April 10-11 2014Antonella Plaia

Average grade of expected research products with respect to average grade of the (research) area: Ri,j

Antonella Plaia

Palermo April 10-11 2014Antonella Plaia

Ene

rgia

, Ing

egne

ria d

ell'I

nfor

maz

ione

e M

odel

li M

atem

atic

i (D

EIM

)

Mat

emat

ica

e In

form

atic

a

Fisi

ca e

Chi

mic

a

Fisi

ca e

Chi

mic

a

Sci

enze

e T

ecno

logi

e B

iolo

gich

e, C

him

iche

e F

arm

aceu

tiche

Sci

enze

del

la T

erra

e d

el M

are

(DIS

TEM

)

Bio

med

icin

a S

perim

enta

le e

Neu

rosc

ienz

e C

linic

he (B

ioN

eC)

Bio

pato

logi

a e

Bio

tecn

olog

ie M

edic

he e

For

ensi

( D

IBIM

EF)

Sci

enze

Agr

arie

e F

ores

tali

Sci

enze

del

la T

erra

e d

el M

are

(DIS

TEM

)

Sci

enze

e T

ecno

logi

e B

iolo

gich

e, C

him

iche

e F

arm

aceu

tiche

Sci

enze

Giu

ridic

he, d

ella

Soc

ietà

e d

ello

Spo

rt

Sci

enze

per

la P

rom

ozio

ne d

ella

Sal

ute

e M

ater

no In

fant

ile

Bio

med

icin

a S

perim

enta

le e

Neu

rosc

ienz

e C

linic

he (B

ioN

eC)

Bio

med

ico

di M

edic

ina

Inte

rna

e S

peci

alis

tica

(DIB

IMIS

)

Bio

pato

logi

a e

Bio

tecn

olog

ie M

edic

he e

For

ensi

( D

IBIM

EF)

Dis

cipl

ine

Chi

rurg

iche

, Onc

olog

iche

e S

tom

atol

ogic

he (D

i.Chi

r.On.

S.)

Sci

enze

Giu

ridic

he, d

ella

Soc

ietà

e d

ello

Spo

rt

Sci

enze

per

la P

rom

ozio

ne d

ella

Sal

ute

e M

ater

no In

fant

ile

Sci

enze

Agr

arie

e F

ores

tali

Arc

hite

ttura

(DA

RC

H)

Inge

gner

ia C

ivile

, Am

bien

tale

, Aer

ospa

zial

e, d

ei M

ater

iali

(DIC

AM

)

Ene

rgia

, Ing

egne

ria d

ell'I

nfor

maz

ione

e M

odel

li M

atem

atic

i (D

EIM

)

Inge

gner

ia C

him

ica,

Ges

tiona

le, I

nfor

mat

ica,

Mec

cani

ca

Inge

gner

ia C

ivile

, Am

bien

tale

, Aer

ospa

zial

e, d

ei M

ater

iali

(DIC

AM

)

Ben

i Cul

tura

li - S

tudi

Cul

tura

li

Sci

enze

Um

anis

tiche

Stu

di E

urop

ei e

del

l'Int

egra

zion

e In

tern

azio

nale

. Diri

tti, E

cono

mia

, M

anag

emen

t, Ben

i Cul

tura

li - S

tudi

Cul

tura

li

Psi

colo

gia

Sci

enze

Giu

ridic

he, d

ella

Soc

ietà

e d

ello

Spo

rt

Sci

enze

Um

anis

tiche

Sci

enze

Eco

nom

iche

, Azi

enda

li e

Sta

tistic

he (S

EA

S)

Sci

enze

Giu

ridic

he, d

ella

Soc

ietà

e d

ello

Spo

rtS

tudi

Eur

opei

e d

ell'I

nteg

razi

one

Inte

rnaz

iona

le. D

iritti

, Eco

nom

ia,

Man

agem

ent,

Sci

enze

Eco

nom

iche

, Azi

enda

li e

Sta

tistic

he (S

EA

S)

Sci

enze

Giu

ridic

he, d

ella

Soc

ietà

e d

ello

Spo

rtS

tudi

Eur

opei

e d

ell'I

nteg

razi

one

Inte

rnaz

iona

le. D

iritti

, Eco

nom

ia,

Man

agem

ent, B

eni C

ultu

rali

- Stu

di C

ultu

rali

Sci

enze

Giu

ridic

he, d

ella

Soc

ietà

e d

ello

Spo

rtS

tudi

Eur

opei

e d

ell'I

nteg

razi

one

Inte

rnaz

iona

le. D

iritti

, Eco

nom

ia,

Man

agem

ent,

1 2 3 4 5 6 7 8 9 10 11 12 13 14

-0.5

0

0.5

1

1.5

2

R

nd

DARCH

BC - SC

BioNeC

DIBIM

IS

DIB

IMEF

Di.Chir

.On.S

DEIM

Fisica

e Chim

ica

DICGIM

DICAM MeI

Psicolo

gia SAFSEAS

SGSS SU

DISTEM

STBCF

SPSeMI

D.E.M

.S.

7.576

4.901

6.449

4.1924.02

3.203

4.235

5.61

3.5683.848

4.47100000000001

2.644

3.697

6.47

4.213

8.727

5.15899999999999

2.666

7.481

4.10599999999999

2.773

1.902

6.612 6.501

3.26

5.032

3.4963.28

7.5597.896

5.229 5.241

2.242

2.966

6.005

1.817

6.7626.384

3.524

8.88200000000001

3.269

2.138

% Prodotti attesi sul totale struttura IRFD x 100

Antonella Plaia

Conclusions“Institutional self-knowledge is the starting point for effective quality assurance. It is important that institutions have the means of collecting and analysing

information about their own activities. Without this they will not know what is working well and what needs attention,

or the results of innovatory practices.” ENQA Standards and Guidelines.

UNIPA tries to follow these recommendations since early nineties.

UNIPA tries to remain aligned with all the changes in the evaluation activities implemented by the Ministry and ANVUR, namely: indicators for the distribution of competitive fund, indicators for University Strategic plans, VQR, …

The main problem is sometime the rate (or the speed, actually really high) with which the Ministry changes the evaluation indicators.

Palermo April 10-11 2014Antonella Plaia