Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation

transcript

Ingeborg MeijerCentre for Science and Technology Studies, Leiden University

QMM Workshop, Brunel University, London1-2 October 2015

Outline

• Societal quality: Ensuring use of of research • DHF and the end user panel• Evaluation in practice - triangulation• Methods

– Observation of peer review– Questionnaire– Interview

• Results• Conclusion

Scientific & societal quality

• Economic returns

• Cultural returns

• Social returns

• Scientific returns

Scientific interaction

sProfessional interactions

Private interaction

Public interaction

Science and society

Evaluation scienceMonitoringInvolving End

ProgrammingSocietal Demand driven

• The process of value creation is by transferring knowledge from a research institute into society (private or public parties, or general public), and includes (demand-driven, user-inspired) research programming, and interaction with potential users during the research.

Dutch Heart Foundation objectives

Dutch Heart Foundation (DHF)• Charity Fund > 50% of budget to scientific research • Mission: 3 times less….

– Less people get a cardiovascular disease;– Less people suffer from cardiovascular disease;– Less people die from cardiovascular disease.

• Change in objective:– Closer connected to donators,– Faster translation from results science to patients– More participation from stakeholders

• Goal: to experiment with processes in the core of the DHF > evaluation of research proposals.• End User panel (EP) & criteria to evaluate

societal aspects

Societal quality operationalised5 Criteria• Relevance of the health

problem• Contribution of the research to

solving the health problem• To the next step in the

research (or development) process.

• Focus on activities towards (eventual) application of results in healthcare– CV of applicant– Objectives, strategy and

actions• Participation of stakeholders

• Relation between criteria

Relevance

Activity towards actors

Societal quality

Participation

Phase 2 Practicing in reality• Separate calls

– Young investigator personal grant– Focused topic: women and cardiovascular disease– Focused topic: congenital heart disease in children

• End user panel evaluation in parallel to scientific evaluation panel

• Different types of information sharing• No formal input of evaluation of societal aspects

on decision by the board of DHF (despite formal advice)

• Evaluation of the whole process by CWTS

Overview of calls and EP input

Societal aspects in evaluation process

Personal Women Congenital

Information in call text + + + -Intermediate advice to scientists - + - +Final advice to scientists + + + -Advice to ISC as referent - + - -Advice to ISC in person - - + -Interaction with EP on same day as ISC + + NAInteraction with EP on another day as ISC - - + NA

Research questions• Is the EP able to play a role in evaluation of

research proposals, and what kind?• Is there a difference in focus between ISC and EP?• How do scientists value the feedback from

external stakeholders?• Are scientists aware of societal relevance or

quality aspects of their work?• What do they think about the aim and set up of

the DHF?

Evaluation in practice: triangulation

Questionnaire Interview

EP Societal

Interaction

Observation

Observations

Observations of peer review

• Aim: comparing distribution of arguments between ISC and EP

• 2x ISC and 1x EP • Session was chaired• Only arguments

were scored, not final judgement

Observations – Scores• Use of observation score form• Scoring arguments

– Positive– Negative– Person– Before or after presentation– Content of remarks

• Counting occurrence of arguments• Limitation: Validation of scores by independent

other; field knowledge

Result observation Peer review – congenital EP vs ISC

EP most used“Next step” MA3 > positive “Activity” MA4 > negativeOften in combination with: “Feasibility” WA4 positive“Solution” WA2 negativeNegative goes downAP and CL arguments used

ISC most used“Feasibility” WA4 both in positive and negative contextAll others: very limited

Result observation Peer review –ISC congenital & women

Comparison two calls:Feasibility is dominant argument in discussionSecondly, the chosen approach (WA2) and workplan (WA5) are discussed most and it relates to how realistic plan isNote: ISC has not been asked to discuss societal aspects, and contribution of EP in person (congenital) has little effect.

Preliminary conclusion

• EP and ISC use different arguments, where the EP uses more scientific argumemts than the ISC societal arguments.

• Suggests that EP has a broader vision compared to ISC which has little attention for applicant or context related aspects.

• EP peer review could have added value

Questionnaire

Questionnaire• Aimed at scientists involved in session with End

user panel (n=20)• Goal: Establishing perceptions of “societal” and

opinion on capabilities of End User Panel• Three main questions, worked out through

statements to agree upon (4-point Likert scale)– Vision of scientists on the concept of societal quality– Vision of scientists on the role of the End Users and their

feedback– Weighing the contribution of the EP and societal criteria on the

outcome of the evaluation

Q: What is societal quality? Yes, agree No, don’t agree

Process of knowledge valorisation leads to societal benefit.

I think that societal benefit is mainly related to communication with users.

Societal benefit has social, economic and cultural aspects

I think that valorisation has mainly to do with economic innovation (some doubt)

Societal quality of research relates to value creation through the interaction of the research group with the societal environment

I think that societal quality of research is independent of the scientific quality and interaction with fellow scientists.

Societal quality is related to focused activities towards diverse users.

Paying attention to societal quality in an early stage of a research project increases the chance that results are indeed transferred to the right actor much quicker.

It is important that scientists take into account the context of users, and adjust the way they transfer knowledge to that user.

Q: Contribution of EP to assessmentStatement All

(median 2)

personal EP

The judgement of my scientific peers on my proposal is sufficient for me.

2,5 3 2 (2)

I understand that when a proposal is scientifically ‘state of the art’, it may be rejected based on insufficient societal quality.

2,32 1,75 2,63 (3)

I understand that only proposal that have sufficient societal quality proceed to scientific evaluation process.

2,21 1,75 2,5 (2)

I think that scientific and societal quality should count equally

2,18 1,5 2,38 (2)

I find all this attention for societal quality or benefit unnecessary and too much

1,95 2,75 2,25 (2)

Preliminary conclusion• Scientists and EP largely agree on what societal quality is. • Young researchers have less interest in application of

knowledge, don’t want to pay attention to it in an early stage, and think it has mainly to do with communication.

• Societal quality cannot be leading in deciding what proposals to fund (scientists and EP)

• Method wise, also look at distribution of answers since n=low

Interviews

Interviews – semi-structured• N=20• 3 topics:

– Preparing for the interaction with EP– The actual interaction with the EP– Feedback from EP

• Interview usually 30 minutes, after questionnaire• All comments collected, not counted, broad

picture

Interviews – snapshot of results

PresentationProfessionals in EPRole in decision

PositiveNext step

CommunicationCareer/grant

Feedback vagueNot transparentPersonal grants

Mixed method message

Preliminary conclusion• The contribution, assessment, and feedback of the

EP has added value, but in general scientists do not hear anything new. It is a different emphasis.

• Scientsist think that End users have a different view, which is true in some aspects but not in others.

• Interaction EP with ISC could be valuable• Young scientists: different approach necessary

because of funding pressure and scientific careers (MM)

• Procedure needs to be transparent• Feedback from EP more explicit

Thank you for your attention!

Want to know more.....contact me at

i.meijer@cwts.leidenuniv.nl