of 21
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
1/21
Evaluation systems and evaluationpractice: towards a framework for
research evaluation
Maria Nedeva, MIoIR
the University of Manchester
Warsaw, December 3, 2010
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
2/21
Outline of presentation
Science/innovation systems and evaluation
systems
Evaluation framework
Evaluation practice
Broad conditions for evaluation
What to watch for
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
3/21
Science/innovation systems
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
4/21
Science/innovation systems
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
5/21
Research spaces and fields
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
6/21
Research fields and policy spaces
Research space is a policy space
Relationships between regulators, funders
and performers of research
Funders can be layered e.g. Government
organisations, executive agencies etc.
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
7/21
National Evaluation System:definition
Research evaluation systems are organised sets ofprocedures for assessing the merit of researchundertaken in publicly funded organisation that are
implemented on a regular basis, usually by state orstate-delegated agencies...[NESs vary] in terms oftheir frequency, formalisation, standardisation and
transparency.R. Whitley, 2007
(The Changing Governance of the Sciences)
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
8/21
National Evaluation System
Permeates research spaces and fields
Institutionalised: rules, responsibilities,
procedures, consequences Repeated
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
9/21
National Evaluation System:organisation
Who commissions
Who organises
Who carries out
Who applies the results
Example the RAE in the UK
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
10/21
Organisation of science andevaluation systems
Answers depend on the organisation of theresearch space (national)
As a rule of the thumb:
Political organisations commission (Ministries, GovntDepartments)
Intermediaries (agencies) organise
Multiple actors carry out (contractors, panels, experts, in-
house specialists) Political organisations and agencies apply
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
11/21
Organisation of science andevaluation systems
NESs are conceived, developed and
governed in the Research Space
They can target social phenomena inresearch spaces and research fields
Generally evaluations are about performance
management: of funders, researchorganisations, research teams, researchers
and knowledge
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
12/21
Evaluation as management practice
Information
Judgement Action
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
13/21
Information
About what? Object
Types of information
How collected? Methodologies
Sources
Who collects? In-house
Outsourced
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
14/21
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
15/21
Action
Funding decisions
Regulation
Change
Closure
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
16/21
Evaluation practice
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
17/21
Distortions
Indicator based
Peer review based
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
18/21
Evaluation methodology(ies)
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
19/21
Broad conditions of evaluation
Fit between the overall and localisedevaluation practices
Time-lines
Sharing of practice Experience
Comparison
Relying on others
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
20/21
Pitfalls
Over- evaluation
Distortions of
the science system knowledge
management
Assumptions
8/12/2019 20110114 Dr Maria Nedeva - Evaluation Systems and Evaluation Practice
21/21
Thanks for listening! Any questions?
