Case study narrativesDiscussion session – Leonie van Drooge

2 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives

Societal relevance (SEP 2009-2015):

Self-evaluation report:• “Group leaders collaborate with more than 31 pharmaceutical,

robotics and technical companies” “prof X founded a company” “5 patents were filed”…..“Several group leaders appeared in prominent television (63 times) and radio programs (75 times)”…..

Evaluation report:• “In general societal impact is guaranteed since health and

disease are central issues in any society; and, in particular, subfield Y is highly relevant given increase in the number of elderly people in Western societies.”

3 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives

Case studies: why is it an issue?

• Case studies have been introduced in a number of ex post evaluation schemes (REF, SEP)

• However, the use seems problematic • Stefan de Jong, Jorrit Smit & Leonie van Drooge (2015):

“read it with scepticism, because I already knew….something is being imposed on us which did not originate from us. Without any motivation.”“And the international reviewer has no idea what we mean by it [knowledge utilization], absolutely no idea.”

• Gabrielle Samuel & Gemma Derrick (2015):“different types of conceptualizations impact evaluators may bring to formal assessment frameworks before embarking on assessment”

4 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives

Instructions for case studies or impacts

• REF (UK ex post): a clear explanation of the process or means….

• SEP (NL ex post): how the unit achieved this….• Broader Impacts (US ex ante): describe the potential of the

proposed activity to benefit society and contribute to the achievement of specific, desired societal outcomes

5 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives

SEP

• The committee assesses the quality, scale and relevance of contributions targeting specific economic, social or cultural target groups, of advisory reports for policy, of contributions to public debates, and so on. The point is to assess contributions in areas that the research unit has itself designated as target areas.

• The point of the narrative is to describe the most convincing examples of relevance, impact or added value to society achieved by means of the scientific work of the research unit.

• The indicators given in the table are only examples. The research unit may select these or other indicators compatible with its own profile and mission, within clearly defined boundaries.

6 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives

What are the pitfalls (or challenges)?

• Researchers and groups (evaluated) are advised and supported, however, evaluators are not.

• Narritives that show no relation between research, actions and impact (“and then…. and then…. and then”)

• Narratives showing too much intention, leading to overly positivistic or deductive narratives

7 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives

What is required – what is our responisibility now

• Do we have a responsibility and are we needed?• Case study should provide insight• Context is important – learn from ex ante situations• We should come up with something helpful, not complicated• We know a range of schemes (different countries, ex post &

ex ante)

8 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives

What do you need?

• ?

9 | Leonie van Drooge / QMM2015 / October 1st&2nd, 2015 / Discussion session Case study narratives

Thank you

Leonie van DroogeRathenau Instituut

l.vandrooge@rathenau.nl