2012

KCE REPORTS

STAKEPROC

174C

EHOLDESSES

ER INV

VOLVEMMENT INN KCE WWORKINNG

www.kce.fgoov.be

Belgian He

Executive B

ealth Care K

Board

Knowledge CThe Bof DeKCE health PresCEOInsurPresFoodpresiPresSecuGeneMediRepr Repr Repr Interm ProfephysProfenurseHosp Socia Hous

Centre Belgian Health Cecember 2002 un

is in charge of h insurance

ident O - National Institrance (vice presidident of the Fed

d Chain Safetyident) ident of the Fed

urity (vice presideneral Administratoricines and Health resentatives of the

resentatives of the

resentatives of the

mutualistic Agenc

essional Organisicians

essional Organises pital Federations

al Partners

se of Representat

are Knowledge Cder the supervisconducting stu

tute for Health andent) deral Public Servy and Environm

deral Public Sernt) r of the Federal Products

e Minister of Publi

e Minister of Socia

e Council of Minis

cy

sations - represe

sations - represe

tives

Centre (KCE) is asion of the Ministdies that suppo

AcPi

nd Disability Jo

vice Health, ment (vice

Di

rvice Social Fr

Agency for Xa

ic Health BeM

al Affairs ORi

sters JeDaMPaXa

entatives of MJe

entatives of MMJoJeRiPaLi

an organization ter of Public Hea

ort the political d

tual Members ierre Gillet o De Cock

irk Cuypers

rank Van Massen

avier De Cuyper

ernard Lange arco Schetgen liver de Stexhe i De Ridder ean-Noël Godin aniel Devos ichiel Callens atrick Verertbruggavier Brenez arc Moens

ean-Pierre Baeyenichel Foulon yriam Hubinon

ohan Pauwels ean-Claude Praetita Thys aul Palsterman eve Wierinck

of public interesalth and Social Adecision making

Substit Benoî

Chris

hove Jan B

Greet

Franç Annic Karel Lambe Frédé Bart O Frank

gen Yolan Geert

ns Rolan

Rita C Ludo

Olivie Katrie Pierre Leo N Celien

st, created on theAffairs. g on health care

tute Members

ît Collin

Decoster

ertels

Musch

çois Perl k Poncé Vermeyen ert Stamatakis

éric Lernoux Ooghe k De Smet de Husden Messiaen

nd Lemye Cuypers Meyers r Thonon

en Kesteloot e Smiets Neels n Van Moerkerke

e 24th

e and

Control

Manageme

Contact

ent

Gove

ChiefAssisMana

Belgi

DoorbBouleB-100Belgi

T +32

F +32

info@

http://

ernment commissi

f Executive Officestant Chief Executagers Program Ma

an Health Care K

building (10th Floevard du Jardin B00 Brussels um

2 [0]2 287 33 88

2 [0]2 287 33 85

@kce.fgov.be

/www.kce.fgov.be

ioner

r tive Officer anagement

Knowledge Centre

or) otanique, 55

e

Yv

RaJeChKr

(KCE).

ves Roger

af Mertens ean-Pierre Closonhristian Léonard ristel De Gauquierr

 

2012

KCE REPORTS METHOD

STAKEPROC

JULIEN PIERA

174C

EHOLDESSES

ART, CHRISTIAN

ER INV

LEONARD, PATR

VOLVEM

RICE CHALON, F

MENT IN

FRANÇOIS DAUE

N KCE W

E, RAF MERTENS

WORKIN

S

NG

www.kce.fgoov.be

COLOPHOTitle :

Authors :

Acknowledgem

External Valida

Conflict of intere

Layout :

Disclaimer :

Publication date

Domain:

MeSH:

NLM Classificat

Language:

Format:

ON

ents:

tors :

est :

e:

tion:

Stake

JulieHealt

Laure

Bert Sant

None

Ine V

The comnece

Subsa corepo

Fina

Onlyalso

Marc

Meth

Stake

Q 18

Engli

Adob

eholder Involveme

n Piérart (KCE), Cthcare Projects), R

ence Kohn (KCE)

Boer (College voé Publique - UCL

e Declared

Verhulst, Sophie V

external expertments were disc

essarily agree wi

sequently, a (finansensus or a vo

ort and did not ne

lly, this report ha

y the KCE is respunder the full re

ch 28th 2012

hods

eholder involveme

80.55

ish

be® PDF™ (A4)

ent in KCE workin

Christian Léonard Raf Mertens (KCE

)

oor Zorgverzeker)

Vaes

ts were consultcussed during mth its content.

al) version was soting process beecessarily all thr

as been approve

ponsible for erroesponsibility of th

ent ; Organization

ng processes

(KCE), Patrice X E)

ringen, Netherland

ted about a (pmeetings. They

submitted to theetween the validaee agree with its

ed by common as

ors or omissionshe KCE

nal Objectives

Chalon (KCE), F

ds), François Mé

preliminary) versdid not co-auth

e validators. The ators. The validas content.

ssent by the Exe

that could pers

rançois Daue (Op

élard (ULg), Alain

sion of the sciehor the scientific

validation of theators did not co-

ecutive Board.

ist. The policy re

ptifa - Optimum fo

n Deccache (Ecol

entific report. Tc report and did

e report results f-author the scien

ecommendations

r

le de

Their d not

from ntific

s are

Legal depot:

Copyright:

How to refer to

this document ?

D/20

KCE http:/

Pieraproce174C

This

12/10273/11

reports are publis//kce.fgov.be/cont

art J, Leonard Cesses. Method. B

C. D/2012/10.273/

document is avai

shed under a “by/tent/about-copyrig

, Chalon P, DauBrussels: Belgian/11

lable on the webs

/nc/nd” Creative Cghts-for-kce-repor

ue F, Mertens R Health Care Kn

site of the Belgian

Commons Licencets.

. Stakeholder Invnowlegde Centre

Health Care Kno

e

volvement in KC(KCE). 2012. KC

owledge Centre

CE working CE reports

 

KCE Reports 17

FORE

74C

EWORD

In 20KCEwhichcommclose

On thfield,agenprojestudi

We anecetakencomealternelderthe fi

In thikey eKCEpracttechnOthe

But methdefinwe cstudy

Jean

Assis

S

009 the KCE com’s scientific excellh progress couldmunication with the involvement in th

he one hand there by involving the

nda. On the otheects as well couldes.

are now over two essary emphasis n, including a newes to stakeholdenative medicine, orly, to name but aield.

is report we want elements of stake’s specific duties tical “process noniques. The first

ers will follow in th

the essence of hodological handbnes the KCE’s tascannot disregard wy provides a majo

n-Pierre CLOSON

stant Managing D

Stakeholder Involv

mmissioned an exence and credibil

d still be made. he target groups che KCE study pro

e was the concernstakeholders earlr hand it appeare

d contribute to a

years further dowon these issues.

w editorial approar involvement, won burnout amonga few, each had a

to place this appeholder involvemeand possibilities.tes” on the concof these processe coming years.

an effective invbooks. What is bsk on the basis of what all those othr impetus for this

irector

vement

xternal audit of itsity were generallyFor example, th

could be further imocesses.

n to have the subjly in the projects, ed that a good inbetter impact of t

wn the line. The neIn the field of c

ach to the studieswe haven’t been g general practitioa sizeable compo

proach within a moent and reflect on The document a

crete terms and notes deals with

volvement of thebasically involved

social relevance her players in the approach.

s impact on healty recognised, the he interviewed stmproved, and they

jects under study and in particular

nteraction with ththe messages an

ew management communication, a s and a thorough

sitting around doners and on the nent of intensive

ore scientific and n how these can aims to be the refconditions governh qualitative meth

e interested parti is a vision and and added valuehealth system ha

thcare policy in thaudit pointed outtakeholders werey also clearly expr

better attuned toin the drafting of

he target groups nd recommendatio

plan of 2010 of cohost of initiative

revamping of thedoing nothing, eit

future need for reinteraction with th

conceptual framebest be structure

ference frameworning specific stakhods, and is curre

ies is much mo organisational c

e. And when bringave to say. We the

he country. Whilet a number of field

e of the opinion ressed a call for m

the complaints inthe specific resein the final phas

ons formulated in

ourse also placeds have already b

e website. But whther. The studiesesidential care forhe groups involve

ework. We look ated and geared tork for a series of keholder involvemently being prepa

ore than a seriesculture that expreging this into pracerefore hope that

Raf MERTENS

Managing Directo

i

e the ds in that

more

n the arch

se of n the

d the been en it s on r the ed in

t the o the very

ment ared.

s of essly ctice, t this

or

ii

SUMMMARY

SStakeholder Involv

Sths

Inpsdis

Thasdin“wla

BIncthebere

Wrescs

vement

Stakeholder involvhat are potentiallysince the KCE was

n the vast majorityprojects, care prostudy subject madistinction betwees not always easy

The involvement however, has beea comprehensive stakeholders in odetailed methodonvolvement into pwhen” and “why”atter point will be

BASIC PRIn the second chconceptual and emhe specialist liteexperiences from be meaningful aexperiences of oesearch.

We first outline thesearch field. Nex

stakeholder involvcompletion of a studies:

vement, i.e. the ay or actually affecs set up.

y of the KCE projeoviders and polictter on a day-to-n their contributio

y.

of other groupsn much less systereflection on wheour processes,

ological handbookpractice. This repo”, without going inthe subject of a s

INCIPLESapter we look at

mpirical standpoinerature in the fother sectors whnd relevant. In

other agencies a

he origin of stakext, we examine th

vement in greater project and diss

active involvemencted by our studi

ects, and in the vacy-makers who h-day basis will b

on as an expert ve

than the providematic. Hitherto wen and why best neither have weks on how bestort focuses primanto depth on the eries of methodol

t stakeholder invnt. In the former wield, and in the

here stakeholder so doing we d

and players, but

holder involvemehe various aspects

depth, from the isemination of the

KCE Reports 1

nt of groups or pes, has been fos

arious phases of tave dealings wit

be consulted. Yetersus as a stakeh

ders or policy-mawe have not carrie

to involve the vae developed spet to put this kin

arily on the questimatter of “how”.

ogical process no

olvement from bwe based ourselvee latter on publinvolvement provraw not only on

also on sociolo

ent in the public hs of and approachnitial phase throue results. This r

174C

eople stered

those h the t, the

holder

akers, ed out arious ecific, nd of ion of . This otes.

oth a es on ished

ved to n the ogical

health hes to ugh to report

KCE Reports 17

• The possibgenuine “partners in

• How to ide

• The imporphases of a

• The challen

• The skillsinvolvemen

FROM THRESEARIn this methodostakeholders inopportunities anbased themselvdown during foresearchers tomoderator.

Among other ttraining, and aneeds of each one that was nwas that the KCinvolvement rerespect.

74C

ble objectives, fro“democratisation”the study who ac

ntify and describe

rtance of open a project;

nges and risks pre

s and principlent.

HE ANGLECHERS

ological research pn past and currend success factorves here on their ur discussion foraok part and the

things, the KCE n adapted way oproject. Patients

not always easy CE ought not to loquired time and

om mere informa, whereby the

ctively contribute t

e stakeholders;

communication

esented by contro

s needed for

E OF THE K

project we have aent KCE projectsrs according to thown experience,

a in October and e meetings were

experts pointed of involving stakes were named as

to approach. Onose its independe

could only occu

S

tion gathering thrstakeholders

to the results;

and transparenc

oversial subjects;

successful stak

KCE

also looked at the s, and at the obe KCE researcheand their view waNovember 2011. chaired by an

to the need for eholders accordins an important groe oft expressed nce, and that stakr in a climate of

Stakeholder Involv

rough to become

cy in all

keholder

place of bstacles, rs. They as noted All KCE external

suitable g to the oup, but concern keholder f mutual

HIKOpeit

Wins

1

2

3

vement

HOW TO PNVOLVEMKCE? On the basis of thepresents a numbeensuring a more ets projects.

We identified five mnvolvement couldsuccessive phases

. Make the subis useful to faThis can be patients and cmatter on a da

2. Define the rito address thseen by the cwe must try towill help us values, and sgo into certaparticipative fothe project amunderstandingimplications.

3. Ensure the sinsofar as is approach. If nuseful for obta

PUT STAKEMENT INTO

e above-mentioneer of approaches teffective involvem

major strategic obd be useful. Bys of a research pr

bject tangible. Bemiliarise oneself wdone, for exam

care providers whaily basis.

ght scope and re really importan

care providers, pao expose the undto understand th

sometimes also thain study questioormulation of thesmong the end usg of the regula

study structure apossible, that the

necessary, add infaining a broad vie

EHOLDER O PRACTIC

ed insights, the lasthat could increas

ment of various re

bjectives in respecy and large, theroject:

efore commencinwith the subject aple, by asking f

ho are confronted

research questiot problems and c

atients, the publicderlying problems he more deep-sehe reasons for a ons in greater dse questions can sers. It is also imatory context, in

and methods are e main stakeholdformation sources

ew.

CE AT THE

st chapter of the rse the KCE’s impaelevant stakeholde

ct of which stakehey correspond to

g a research projand to make it tanfor the contributio

with the study su

ons. If our studiesconcerns, as thesc and/or policy-ma

or controversieseated motivationspossible resistan

depth. A carefulalso create suppo

mportant to have ncluding its fina

acceptable. Seeders accept the Ks or data that mig

iii

report act by ers in

holder o the

ect, it gible. on of ubject

s aim se are akers, . This s and nce to l and ort for a full

ancial

e to it, KCE’s ght be

iv

4. Arrive at good view to be ableclear and erealistic, anleast in the

5. Find effecto obtain thof the repo

These objectivevalues in our objectivity, mod

In practice, theinvolvement. Thto the above-minterlocutors. Ccorresponding the results we e

The report endsinto formal KCEinto the KCE’s r

clear-cut resultsof contrasting sta

e to draft your owexplicit terms. Evend therefore also e long term.

ctive communicahat the target gro

orts.

es can only be acontacts with t

desty and sincere

ere is no single she question has mentioned object

Consequently, we to the different s

expect it to yield.

s with a list of tecE process notes, research methods

s and realistic aandpoints and thewn conclusions aery recommendatifeasible – perhap

ation channels aoups themselves b

achieved if we bethe stakeholderscuriosity.

standard way of inumerous dimenstives, the nature will have to deve

situations of stak

chniques that neewhich, ultimately

s.

S

answers. Try to oe main argumentand recommendaion should in any ps not immediatel

and spokespersobecome active ad

ear in mind a nu: respect, transp

mplementing staksions that vary ac

of the project elop an array of mkeholder involvem

d to be further dey, should be inco

Stakeholder Involv

obtain a ts, so as ations in case be y, but at

ons. Try dvocates

umber of parency,

keholder ccording and the

methods, ment and

eveloped rporated

vement KCE Reports 1174C

KCE Reports 17

SCIETABLE O

74

NTIFIC ROF CONTEN

REPORTNTS

LI1.1.1.1.2.2.

2.

2.

2.

2.

S

T ST OF ABBREVI

RATIONA1. POLICY C2. SCIENTIF3. STRUCTU

KEY PRIN1. THE RISE

2.1.1. W2.1.2. T2.1.3. T2.1.4. W2.1.5. C2.1.6. W

2. STAKEHO2.2.1. E2.2.2. B2.2.3. O

3. COMMUN2.3.1. A2.3.2. A

4. STAKEHO2.4.1. R2.4.2. R2.4.3. C

5. DIALOGU

Stakeholder Involv

IATIONS .............ALE OF STAKEHOCHALLENGES ANFIC CHALLENGESURE OF THIS STUNCIPLES OF STAE OF STAKEHOLDWho are stakeholdThe circle of stakeThe patient and theWhy stakeholder inCriteria of effectiveWhat are the key cOLDER IDENTIFICExplore the world :Be prepared beforeOrganize your stakNICATION AT AN Analyse the risks aAnticipate time-conOLDER CONSULTRefer to standardsRefine your procesChoose your methE IN CONTROVE

vement

............................OLDER INVOLVEND OBJECTIVESS AND OBJECTIVUDY REPORT ....

AKEHOLDER INVDER THINKING ..ders? ...................holders ...............e citizen ..............nvolvement? .......e stakeholder invocomponents of staCATION AND AN: from unfolding coe the journey ......keholder identificaEARLY STAGE ..

and benefits linkednsuming communTATION ALL ALOs ...........................ss ........................od .......................

ERSIES ...............

............................EMENT AT KCE ..............................VES .................................................

VOLVEMENT ...................................................................................................................................................

olvement ..............akeholder involvemALYSIS ...............omplexity to orde............................

ation and analysis............................d to communicatio

nication activities ..ONG THE PROJE

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................ment? .............................................ring complexity ...............................

s .......................................................on ....................................................CT DEVELOPME................................................................................................................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................ENT PROCESS ...................................................................................................................

1

...... 7

...... 8

...... 8

...... 8

...... 8

...... 9

...... 9

.... 11

.... 12

.... 13

.... 14

.... 15

.... 15

.... 16

.... 17

.... 20

.... 20

.... 25

.... 25

.... 27

.... 27

.... 27

.... 28

.... 29

.... 34

2

2.

2.

2.

2.

2.3.3.3.3.

3.

S

2.5.1. D2.5.2. D2.5.3. D

6. MANAGEM2.6.1. A2.6.2. U

7. STAKEHO2.7.1. C2.7.2. C

8. REGULAR2.8.1. P2.8.2. D2.8.3. A2.8.4. U2.8.5. G

9. SKILLS TO2.9.1. B2.9.2. B2.9.3. B

10. GUIDING CONSULT

1. RATIONA2. METHOD3. RESULTS

3.3.1. L3.3.2. R3.3.3. P3.3.4. T

4. CONCLUS

Stakeholder Involv

Dialogue beyond eDialogue to producDialogue beyond cMENT OF STAKE

Acknowledge comUse forms and foruOLDER INVOLVEConsider added-vaConsider the way tR FEED-BACK TOPlan your communDisplay relevant inAdapt your feed-baUse electronic medGive regular feed-bO MANAGE THE

Be trustworthy andBe involved ...........Be expert ..............

PRINCIPLES ......TATION ROUND LE AND OBJECT............................

S ...........................Lessons learned ...Requests ..............Possible avenues .Treats ...................SION ....................

vement

effectiveness .......ce knowledge syncontroversies .......EHOLDERS’ COMment and respondums .....................MENT IN PROJEalues to involve stto deal with controO STAKEHOLDERnication ................formation ............ack to the level of dia and meetingsback ....................PROCESS OF S

d respectful ..............................................................................................OF KCE EXPERT

TIVES ......................................................................................................................................................................................................................

............................nthesis ..............................................

MMENTS ..............d ......................................................

ECT MONITORINGtakeholders in prooversy..................RS ...............................................................................

f stakeholder invo........................................................TAKEHOLDER IN................................................................................................................TS .......................................................................................................................................................................................................................................................

............................

............................

............................

............................

............................

............................G .........................oject monitoring ...................................................................................................................lvement ......................................................................NVOLVEMENT ...............................................................................................................................................................................................................................................................................................................................................................................

KCE Reports

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

............................

s 174

.... 34

.... 36

.... 37

.... 37

.... 37

.... 39

.... 39

.... 39

.... 42

.... 43

.... 43

.... 43

.... 43

.... 43

.... 43

.... 44

.... 44

.... 45

.... 46

.... 46

.... 47

.... 47

.... 47

.... 48

.... 48

.... 49

.... 50

.... 51

.... 52

KCE Reports 17

74

4.4.

4.4.4.4.

5.

6.

S

STAKEHO1. RATIONA

4.1.1. Im4.1.2. F

2. THE FIVE3. THERE AR4. FROM ST5. STAKEHO

4.5.1. P4.5.2. C4.5.3. R4.5.4. FAPPENDIXEXPERTSREFEREN

Stakeholder Involv

OLDER INVOLVELE : ENHANCINGmpact of the KCE Five strategic obje KEY VALUES FORE MANY DIFFE

TRATEGY TO COOLDER INVOLVEPrerequisites ........Choice of the mostRequired competeFurther elaborationX: APPROACHE

S ...........................NCES ...................

vement

EMENT IN KCE PG THE RELEVAN

reports: levers anctives to overcomOR FRUITFUL STRENT WAYS TO NCRETE ACTIONMENT IN PRACT............................t appropriate meth

encies ..................n of methods that S ILLUSTRATED........................................................

ROCESSES .......CE AND IMPACTnd barriers ...........

me the impact barrTAKEHOLDER ININVOLVE STAKE

NS .......................TICE .................................................hod(s) ..............................................were considered

D DURING THE C........................................................

............................T OF KCE REPOR............................riers and operate tNTERACTION .....EHOLDERS ....................................................................................................................................................as eligible for KC

CONSULTATION ........................................................

............................RTS .................................................the levers ................................................................................................................................................................................................................

CE .........................ROUND OF KCE........................................................

3

.... 53

.... 53

.... 53

.... 54

.... 55

.... 55

.... 56

.... 59

.... 59

.... 59

.... 59

.... 59 E .... 60 .... 70

4

LIST OF F

FIGURES FiFiFiFiFiweFiFiFiFiFiFiFiFiFiFiFiFi

S

gure 1: Stakeholdgure 2: Bridging bgure 3: Key compgure 4: Primary idgure 5: Actors-neebsites links ........gure 6 : Actors-negure 7: Timeline ogure 8 : Example gure 9: Core elemgure 10 : Health Cgure 11: The NICgure 12: The IQWgure 13: The VIKCgure 14: The Cocgure 15: Comparagure 16: HAS progure 17: The ena

Stakeholder Involv

der involvement inbetween decision-ponents of stakehodentification of staetworks diagram, c............................etwork diagram, reof a KCE project ..of table of interpr

ments of a partnerCanada’s Public I

CE key stages of cWIG production proC key stages of g

ck Burning & Honiative chart for par

ocedure followed iblers/resistors spe

vement

n research-use the-making and reseaolder involvement

akeholders in legaconnections betwe............................epresentation of B............................retive frames of acrship process .......nvolvement Conti

clinical developmeocess of reports ..uideline developmngh decision tree

rticipatory methodn formal consensectrum ................

eoretical models ..arch domains ......t ...........................

al texts ..................een osteopathic u............................

Belgian osteopath............................ctors involved ..................................nuum ..................

ent ....................................................

ment used in KCE to identify relevas..........................us method .......................................

............................

............................

............................

............................unions in Belgium ............................

hic unions and sch........................................................................................................................................................................

E project on Prostant participatory m....................................................................................

KCE Reports

............................

............................

............................

............................through their ............................

hools in 2010 ...............................................................................................................................................................................ate cancer ...........ethods ....................................................................................................

s 174

.. 10

.. 11

.. 16

.. 18

.. 19

.. 22

.. 23

.. 24

.. 26

.. 28

.. 31

.. 32

.. 33

.. 33

.. 34

.. 42

.. 44

KCE Reports 17

LIST

74

T OF TABLES TaTaTaTaTapaTaTaTaTaTa

S

able 1 : EBM agenable 2: Six questioable 3 : Consultatable 4 : Dimensionable 5: The potentarticipatory researable 6: Rationale fable 7: Levers andable 8. Dimensionable 9. Timing of table 10: Impact en

Stakeholder Involv

ncies’ stakeholderons to identify releion approaches ans for analyzing ptial added value orch process ..........for stakeholder ind barriers with resns of stakeholder ithe different interanhancing strategie

vement

r involvement’ tasevant consultationnd application in K

public policies ......of participatory res............................volvement ...........

spect to the impacinvolvement ........actions with stakees ........................

sks and group typen methods with staKCE projects ...................................

search approache........................................................

ct of KCE reports .............................holders ............................................

es ........................akeholder involve........................................................

es across the three........................................................................................................................................................................

............................ment ...........................................................................e phases of the ........................................................................................................................................................................

5

.. 29

.. 30

.. 31

.. 35

.. 40

.. 45

.. 53

.. 56

.. 57

.. 58

6

LIST OF B

BOXES BoBoBoBoBoBoBoBoBoBoBo

S

ox 1: Stakeholder ox 2: Evolution of ox 3: an epistemoox 4: Diving in maox 5: Information dox 6: The ADAPTox 7: Acceptabilityox 8 : The Deliberox 9: DACEHTA rox 10: Some guidox 11: List of ques

Stakeholder Involv

r engagement .......knowledge-to-act

ological shift from agma, the cartogradisclosure ............E assessment of y as the most comrative Process – Arecommendationsing principles to sstions for the cons

vement

............................tion thinking and smatter of facts to aphy of controvers............................acceptability and

mplex dimension oA Snapshot ..........s to create the bassupport the use ofsultation round of

............................semantics ............matter of concernsies .................................................applicability of re

of the analysis ..................................

sis for a good andf research in practKCE experts .......

............................

............................ns ................................................................................commendations ......................................................... rewarding coopetice ..................................................

KCE Reports

............................

............................

............................

............................

............................

............................

............................

............................eration ..........................................................................

s 174

.... 9

.. 10

.. 11

.. 16

.. 26

.. 28

.. 35

.. 36

.. 45

.. 46

.. 47

KCE Reports 17

LIST OF A

74

ABBREVIAATIONS

S

ABBREVIATION

ADAPTE AGREE

AHRQ

CCOHTA

CIHR

DACEHTA

EBM EUnetHTA

HAS

HTA IQWiG

KBF

NCCHPP

NICE

(P&)CP VIKC

Stakeholder Involv

N DEFINITI

http://wwwAppraisahttp://wwwAgency fohttp://wwwCanadianhttp://wwwCanadianhttp://wwwDanish Chttp://wwwEvidenceEuropeanhttp://wwwHaute Auhttp://wwwHealth TeInstitut German King Bauhttp://wwwNational http://wwwNational http://www(Patient &Verenigin

vement

ION

w.adapte.org l of Guidelines Rew.agreetrust.org/or Healthcare Resw.ahrq.gov/ n Coordinating Ofw.cadth.ca/ n Institutes of Heaw.cihr-irsc.gc.ca/

Centre for Health Tw.sst.dk/English/D

e-Based Medicinen Network for Heaw.eunethta.eu/

utorité de Santé / w.has-sante.fr/ echnology Assessfür Qualität uInstitute for Qualit

udouin Foundationw.kbs-frb.be/ Collaborating Cenw.ncchpp.ca/ Institute for Healtw.nice.org.uk/ &) Citizen-Patientng van Integrale K

esearch & Evaluat

search and Qualit

ffice for Health Te

alth Research

Technology AssesDACEHTA.aspx

alth Technology A

French National A

sment und Wirtschaftlicty and Efficiency in

ntre for Healthy P

h and Clinical Exc

Kankercentra

tion

ty

chnology Assessm

ssment

Assessment

Authority for Healt

chkeit im Gesn Health Care htt

ublic Policy

cellence

ment

th

sundheitswessen tps://www.iqwig.de

7

/ e/

8

1. RATIINVO

Although stakeprocesses sincwillingness of diversified way‘when’ to involvsocietal pertineonly linked to tthe policy and p

1.1. Policy cThe ‘raison dfundamentally pbe seen as ‘emfinite resourceshealthcare neechallenge requunambiguous reObviously, for ain several othealso have to anthropology. perceptions, exindependence a

1.2. ScientiCombining a truof scientific rigoas the achievestudy processeThis must, withoWhen addressinvolve the diff

IONALE OFOLVEMENT

eholder involvemece its early daysthe organization . The primary aim

ve stakeholders, wence and the imphe scientific qualpractical challenge

challenges andd’être’ of scientifpolitical and pragm

mployed’ by societys of the healthcareds of the populauires that we foresults and proposaddressing this brer disciplines than

mobilize methodWe must find ad

xperiences and opand objectivity.

fic challenges ue openness to por and objectivity

ement of valid reses, including qualiout any doubt, being this challengferent groups of

F STAKEHT AT KCEent has been pas, this methodoloto address it in

m of this work is twith as an ultimatepact of the KCE rity of the researces are met.

d objectives fic advice institumatic. In this sensy to help and findre system for meation. Taking up

rmulate relevant se realistic recommroader perspectivn medicine and hds specific to sodequate ways to pinions of stakeho

and objectivesarticipative appro

y poses serious ssults by means oitative research a

e part of the core bge, i.e. learning tstakeholders, we

S

HOLDER

art of the KCE rogical study refle

a more systemato reflect about ‘we objective to increreports. The lattech, but also to the

utions such as se, KCE collaborad the best ways toeeting the virtually this fundamentaresearch questiomendations.

ve, we need comphealth economics;ociology, law, an

deal with and vaolders, without lo

s oaches with the sacientific challengeof systematic and

and sociological mbusiness of KCE. to master valid

e can tap into a

Stakeholder Involv

research ects the atic and

why’ and ease the er is not e degree

KCE is ators can o use the y infinite al policy ons, find

petences ; we will nd even alue the sing our

afeguard es, such d robust

methods. ways to body of

kse

1InWinInKpfothinFineas

vement

knowledge availasimilar institutionsexperiences from o

1.3. Structuren chapter 2, we eWe address thenvolvement but fron chapter 3, we pKCE researchers principles could orour group discusheir perception ofnvolving stakeholdFinally, chapter 4 nvolvement at Kendeavour fruitfuappendix, illustratestudy processes.

ble in the scients in other counour own work.

of this study reexplore the key pre ‘who’, ‘why’,’ om a theoretical apresent how we themselves, in t

r could not mean ssions, the experf the threats and tders in their day tsynthesizes the s

KCE, including thl. A number of e how we involved

tific literature andtries, but also g

eport rinciples of the sta

how’ and ‘wheand empirical poininvolved our ownthe reflection aboin practice for th

rts have shared the barriers but alo day work. strategic objectivehe key values nf succinct technid stakeholders at

KCE Reports

d the know-how get inspiration in

akeholder involveen’ questions ofnt of view. n stakeholders, i.eout what the idene KCE studies. Dtheir experienceslso the opportunit

es of the stakehoneeded to makeical sheets, givet different stages i

s 174

from n the

ment. f this

e. the ntified During s and ties of

lder’s e this en in in our

KCE Reports 17

2. KEY INVO

2.1. The risStakeholder thevolved1: • from stak

stakeholdeprocess-or

• from a budynamic inwell as influ

A set of similarconceived to dpossibilities to iRational-linearmakers rarelycommunicatio• Supply-sid

of research• Demand-s

research u(rational-lin

These models emphasize linkaInteractive mrelationshipsresearch: multicontext of its us

a See the Sb See the IN

74

PRINCIPLOLVEMENTse of stakeholdeinking grew up i

keholders vieweders as individuariented view; uffered dependennterdependency uential stakeholder theoretical modeescribe whether ncrease the resear models focusey used researcon between researde research initih by adapted publside research iusers about thenear model with fewere abandonedages between resodels, recently between the ke

ple actors, multipse. These variable

SUPPORT program NVOLVE program d

ES OF STAT er thinking n the economic

d as subjects als in a networ

ncy on stakeholdbetween a firm

ers. els has emerged and how researcarch impact2. d on disseminatio

ch it was becrchers and policy atives were deveication formats annitiatives were ir needs and p

eedback loop)b. in favour of mor

searchers and decdeveloped, put

ey variables thaple exchanges, naes are no longer c

description in Lavis

description4.

S

AKEHOLD

c area in the 193

to be manark-based, relatio

ders towards a and its surround

in the knowledgch is used and to

on of research. Icause of problemakers:

eloped to support nd effective packa

developed to qrovide tailored r

re interactive modcision makers5.

a focus on cat shape policy ature of the reseaconsidered in isola

s, 20093

Stakeholder Involv

ER

30s and

aged to nal and

bridging dings as

ge area, explore

f policy-ems of

the use aginga. question research

dels that

complex use of

arch and ation but

inraaeth

B

Inassininsda

c

vement

n interaction. Andationalc. These in

and rational modeexperience in the he postmodern cr

Box 1: Stakehold

n 1984, Freemaacademic attentiostakeholder. The subsequent ones nfluences. Thesenfluence and howstakeholder engadiffusion of these a good practice ha

Recently, ano

to establish nprocess by process (Nut

d the ways researnteractive modelsels, incorporate the

research use proritique (and the re

der engagement

an pushed a ston and provided

core idea that is that manageme

e theories aim tow firms respond tagement. The Woideas by publishinandbook in 20117.

other set of initiativenew kinds of relatiodeveloping intermeley 2008, p246)2.

rch is used are ns, in comparison te role of researchocess, and ask thal possibility to em

takeholder theory boundaries of underlies Freem

ent choice is a fuo understand theto those influenceorld Bank Group ng a good practice.

es between supply onship between resediary broker orga

no more considereto the traditional users’ knowledgee question of powmpower partners)

y to the forefrowhat could defi

man’s theory andnction of stakeho types of stakehes with what is chas contributed te manual in 1998

and demand have bsearchers and the nizations into the

9

ed as linear e and wer in .

nt of ine a d the

olders’ holder called to the

86 and

begun policy policy

10

Figure 1: Stake

Figure inspired byCanada 2000, p.

eholder involvem

y “Health Canada’s128 and “Spectrum

ment in research

s public involvementof stakeholder enga

S

-use theoretical

t continuum » in Heagement” in IFC 20

Stakeholder Involv

models

alth 11, p.3 7

B

Bthc

LRvT“

R

TdKrere“

STmitthRsin

vement

Box 2: Evolution

Best and colleaguhe knowledge-to-cited in Nutley, 20

Linear models (19Research is dissevarious settings. WThe predominantknowledge trans

Relationship mod

The key processedevelop within neKnowledge produelationships and elationships is theknowledge exch

System models (The way knowledgmost important fact needs not only the priorities, cuResearch use is system. The langntegration”, “tran

of knowledge-to

ues provide a fram-action thinking i10, p.135)9:

960s to mid-1990eminated as resulWhether it gets ust language of sfer” and “dissem

dels (mid-1990s t

es for improving reetworks of collaboucts are defined improving the inte key. The langua

hange”.

more recently) ge is embedded wctor in improving rto be embedded iultures and cont

thus a dynamicuage of research

nslation” and “mo

o-action thinking

mework that descrn three phases (

0s) ts are handed ovsed is a function oresearch use f

mination”.

to present)

esearch used areorating research and utilized in

terpersonal commage of research u

within organizatioresearch use. Forn relationships butexts of organizc process withinh use for these mobilization”.

KCE Reports

g and semantics

ribes the generati(Best et al frame

ver to others for uof effective packafor these mode

e the relationshipsproducers and uthe context of t

munication within use for these mod

ons and systems r knowledge to be ut also interwovenzations and sysn a complex adamodels is “knowl

s 174

ion of ework

use in aging. els is

s that users. these these

dels is

is the used

n with tems. aptive ledge

KCE Reports 17

Box 3: an episconcerns

The knowledgemodels towardepistemologicaresearch knowResearch-policysystems thinkinbecome blurred

• In the tradprovide dstraightforw(Nutley 200

• In interactarrive as udecisions, its use thr2008, p.25

EXAMPLE: Th(DACEHTA) DACEHTA pubnew scientific monly with elemeand ethics, but

HTA may be coand the researcfocus of an HTdocumented ba

74

stemological shif

e-to-action thinkids more interal shift which req

wledge as somey relations becom

ng, the boundariesd”9.

ditional linear andiscrete, unambwardly be transfe08, p.123)2.

tive models, we runcomplicated “fabut may be trans

rough ongoing di1)2

he Danish Centr

blished in 2008 amethods and apprents of HTA, i.e. talso with the step

onsidered as bridch domain. In ordA must be based

asis for decisions

ft from matter of

ing has historicactive models inquires us to ret

ething separate me at least two-was between these

nd rational modebiguous, factuaerred and applie

recognise that “reacts” to be weighslated and reconsalogue with rese

re for Health Te

a new handbook roaches to HTA10.technology, patienps in the HTA proc

ging between theer to fulfil such a

d on the need of tabout the use of h

S

facts to matter o

ally evolved fromn association wthink our concefrom policy kno

ay and in some vetwo realms of kno

els research was l accounts thed to policy or

esearch evidence hed up in makinstructed in the proearch producers”.

echnology Asse

presenting upda This handbook d

nt, organisation, ecess.

e decision-makingpurpose, the prob

the decision makehealth technology

Stakeholder Involv

of

m linear with “an ption of

owledge. ersion of owledge

seen to at can practice

may not ng policy ocess of (Nutley

essment

ated and deals not economy

g domain blems in ers for a

y.

F

Tomdb2

2In

T

Ecin

d

e

vement

Figure 2: Bridgin

The DACEHTA modone-way flow represmaking domain. It sedecision-makersd bubetween the policy-m2008, p15)10

2.1.1. Who aren the economic ar

“any group orachievement o

The International F“those who minfluence its “stake” that eproject or inve

EUnetHTA recentlcharacteristics ofnterested in and

See the “two (Freeman 19

g between decis

del is inspired by thesentation, where theeparates two commut allows some spacmaking paradigm an

e stakeholders?rea, Freeman defr individual who cof the firm’s objecFinance Corporati

may have interestoutcome, either ach of these diffeestment will vary” ly referred to thesf stakeholders : d/or potentially infl

communities” thesi84, p25)12 cited in (

sion-making and

e rational-linear mode research domain ta

munities, one of resece for interactions innd the HTA paradig

fined stakeholderscan affect or who ctives”e. ion’s definition adts in a project anpositively or neg

erent individuals o(IFC 2011, p.10-1

se two definitionsthey are affec

luent.

s of Caplan11. Andriof 2002, p30)1

research domai

del of research use argets the decision-

earchers and the othn the overlapping zogm. From (Kristense

s as is affected by the

ds nd/or the ability togatively. […] Theor groups has in a11)7. to underline the cted by the pr

1.

11

ns

in a -her of one en

e

o e a

three roject,

12

When comparilists, the most c• industry • policy mak• patient/use• health care• health relat• academia• commercia

2.1.2. The cDepending on wis more or less

2.1.2.1. WidFollowing the G(IQWiG) has cinstitutions, or the preliminarypreliminary repinterim results13

restricted. Prinstitutes, and c

2.1.2.2. CloIn contrast, thapproach in wstakeholders. Srepresentativeshealth professpractices. Thesentire productirepresentativesincluded in thes

ng several EBMcommonly cited st

ers and/or instituter representativese professionals ted media

al enterprises

circle of stakeholwhich stakeholderopen to these sev

de-open consultaGerman Institute chosen an apprcommercial ente

y report plans, oports and thereby

3. The circle of perivate persons, companies, may s

osed monitoring ghe French Natiohich all actors ha

Stakeholders may s. An equilibriumionals, types of

se stakeholders aion process of s, health insuranse groups ; they m

M or health-relatetakeholders appea

tions

lders r involvement metven types of stake

ation for Quality and E

roach in which aerprises may submon amendments y comment on thersons entitled toas well as pr

submit comments

groups nal Authority forave to candidatebe health profess

m is searched beopinions, types

are involved in direcommendation

nce companies may be consulted

S

ed agencies’ stakar to be :

thod is chosen, theholders.

Efficiency in Heaall interested pmit written commto report plans

he methods or p submit commentrofessional asso.

r Health (HAS) e or to be designsionals and patienetween types of s of practice, plfferent groups dus. But public aand industries on a more ad hoc

Stakeholder Involv

keholder

he circle

lth Care persons, ments on

and on ublished ts is not ciations,

has an nated by nt/users’ involved aces of

uring the agencies'

are not c basis.

EWda

Sba

EEF

•

•

•

•

•

•

f

vement

EXAMPLE in the When identifying sdefinition as in thea consultative role

Stakeholders are gby, or have an intactions or aims of

EUnetHTA has chEuropean level, anFollowing groups a

Policy makers a

Policy makers a

Patient organiz

Healthcare pro

Industry

Health related

Definition cite

research area : Estakeholders, the e economic area be:

groups or organizterest in and mayan organization, p

hosen to focus on nd on generic ratare targeted:

at national/region

at hospitals/statut

zations

fessional organisa

media

ed by Nielsen 20091

EUnetHTA definiEUnetHTA Collab

but restrains stake

zations which potey in a consultativeproject or policy d

umbrella organizher than disease

al level

tory health insuran

ations

14.

KCE Reports

ition of stakeholdboration used a seholder involveme

entially will be affee role influence odirectionsf.

zations operating aspecific organiza

nce/HMOs

s 174

ders similar ent to

ected on the

at the ations.

KCE Reports 17

2.1.3. The p

2.1.3.1. A pThere exist macorresponding consider that wapplication: (1) the involvementhe involvemenhealthcare instinvolvement of the P&CP in ththese 5 domainhis or her opiniothe first domainpublic health. There exist difranging from mapproach, bothare needed. Yeand execute sTronto17.

2.1.3.2. A tr‘car

Tronto suggesthuman activity and repair our world includes oseek to interwp.143)17. It h‘capacitating’ bA liberty to exphope18-20. This individual relati

74

patient and the c

patchwork of posany forms of patie

to different objewe can divide th

the involvement nt of the patient int of the patienttitutions (hospitalCP in the financie determination o

ns, the P&CP has on, to co-decide on, the research ac

fferent models tomere information

h generic and speet, each of these should be chara

ransversal qualitre’ ts that on the m“that includes ev‘world’ so that w

our bodies, ourseweave in a comhas been suggeecause it could ‘r

press an opinion, condition of ‘car

ionship between

itizen

ssibilities ent and citizen-patectives and domis kind of involveof the P&CP in thn its relationship t in its relationshl, mutuality’s, prng options to ado

of health prioritiesto be informed, to

or to decide. We lictivities and more

o involve the cin up to actual decific technical aprocedures that w

acterized by the

ty of the involvem

ost general levelverything that we we can live in it aelves, and our envmplex, life-sustain

sted that this preveal’ a certain lan anxiety, a wisre’ seems to be the patient and

S

tient (P&CP) invomains of decisioement into 5 domhe research activwith her practitio

hip with the heaivate insurance) opt (5) the involves and choices. Ino give informationimit the present seprecisely the res

tizens and the pdecision15, 16. Fnd human compwe have to conce

‘care’ defined b

ment: the practic

, caring be viewdo to maintain, c

as well as possibvironment, all of wning web.”(Trontopractice of care liberty for the ben

sh, an expectationquite trivial in ththe practitioner.

Stakeholder Involv

olvement n. We mains of vities, (2) oner, (3) alth and

(4) the ement of each of

n, to give ection to

search in

patients, or each etencies eptualize by Joan

ce of

ed as a continue, ble. That which we o, 2009,

can be neficiary. n and an he inter-But it is

min

2NinifAfogpawW(cininddimaFhsfop‘i

2Wththe

vement

much less straightn public health.

2.1.3.3. InvolvNevertheless, wenvolvement, effecf this concept wasA first example of or normal low-risgroup of gynaecoplain-language veall socioeconomicwith the input fromWe also find exNational Institute

cited example, thenform the researnstitution is able developed a systdetails can be foumpressive list ofanxiety25 or the AlzFurthermore, the helpful to confirmsomewhat unexpeor every patientpermanent presennvasion’ of their l

2.1.3.4. EvaluaWhereas publicatihe pertinence of he patient involvevaluation remain

tforward to concre

vement of the pae can find sevectively characterizes not explicitly ‘namsuch type of patie

sk childbirth, develogists and midwrsion has been de categories of wo

m women of differexamples of patie

for Health and Cle information follorch institution andto provide better

tematic procedureund on the NICEf guidances for zheimer disease26

information origi research hypoth

ectedly, peritoneat. It appeared nce of medical eqives by their disea

ation of the patieons attest the inflthe research28-31,

vement32, 33. Theto be defined.

etize in other setti

tient in the ‘reseeral examples oed by the practicemed’ or ‘used’. ent involvement iseloped by KCE iives21. On the baerived, in order toomen. This exercent education levent involvement ilinical Excellence,ows a ‘reciprocal’ d, given this infoinformation to th

e to involve patiE website 23, whipatients (e.g. on6). inating from the heses. This is hoal dialysis was no

that some of tquipment in theirase27.

ent and citizen-pluence of the way, there is also a actual format a

ings, such as rese

earch activities’ of successful pe of ‘care’ indeed,

s the practice guidn collaboration w

asis of this guidelio be understandabcise has been reaels22. in the work of , UK). As in the ab pathway: the pa

ormation, the resehe patients. NICEents and citizensch is also offerin

n caesarian sect

patients can alsow NICE learnedot the preferred othem experiencer home as a com

patient involvemy to involve patienneed for evaluati

and modalities o

13

earch

atient even

deline with a ine, a ble by alized

NICE bove-tients earch E has s; full ng an tion24,

so be that,

option e this mplete

ent nts for ion of f this

14

2.1.4. Why sReasons to invnumerous, anddescribed as us• provide leg• promote EB• prevent dis• improve ac• enhance re• reduce dupStakeholder inv• Transparen• Accountab• Participatio• Objectivity • Evidence-b• Patient-orie• Scientific wThe ‘objectivity of a richer picpractices, in a s

EXAMPLES ofEBM and healtEUnetHTA: becsense that theimprove the helegitimacy hasrelation to the transparency o

stakeholder invovolve stakeholded vary betweenseful to : gitimacy or credibiBM practices and sagreement or concceptance of resuelevance of reseaplication. volvement is seenncy ility

on

based outcome ented outcome work.

and scientific wocture and the oso-called second-d

f declared reasoth-related agenccause HTA agen

ey are (often) puealthcare systems to be considescientific quality

of the production

olvement? rs in a research

n agencies. Stak

lity to the agency EBM culture ; nflicts ; lts and increase imrch ;

n as serving the pr

ork mentioned herbjective renderindegree heuristic o

ons to use stakies ncies resemble public, tax-finance

m by their activitieered. Legitimacy y of HTA productprocesses14.

S

h production prockeholder involve

;

mpact ;

rinciples of:

re point to the acg of all viewpoi

objectivity.

keholder involvem

political institutiond institutions seees, the question must be unders

ts and the fairne

Stakeholder Involv

cess are ment is

cquisition nts and

ment in

s in the eking to of their

stood in ess and

Ca

DcT

Aha

C

Fw

Stap

Nasb

IQto

Hpre

vement

CCOHTA: stakehoan enhanced awa

DACEHTA: stakehcourse by prevenTo improve the po

AHRQ: stakeholdehelps avoid prioritand reduces poten

CIHR: there are tw

First, the creationwidespread adopt

Second, all of the axpayers’ dollarspractice/action (ac

NICE: comments assurance and pstakeholders offebalance the scienc

QWIG: stakeholransparency wh

objective and evid

HAS: stakeholder principles which aepresentatives), t

older consultationareness and cult

holder involvemennting the occurrossible acceptanc

er involvement pizing topics that hntial duplication.

wo primary reason

of new knowledtion or have an im

interested partiess invested in heaccountability, Tet

from stakeholdeer-review procesr specialist expece of the guideline

lder involvemenhich is one of thience-based, patie

involvement meare : participatiotransparency and

s are important ature of best pract

nt is to ensure therence of disagrece and applicatio

rovides credibilithave no relevanc

ns.

ge often does nompact on patients

s want to see the alth research by troe 2007, p.1)34.

ers are “a vitalsses” (NICE 2007ertise and a prae methodology.

nt corresponds is agency’s princent-oriented and s

thodologies are gon (of professionad independency.

KCE Reports

as it helped in cretices.

e most efficient peement and confon of the HTA resu

ty to AHRQ resece to real-world is

ot, by itself, lead s’ health.

benefits reaped bmoving research

part of the qu7, p.80)35. In this actical perspectiv

to its principlciples36 : indepenscientific.

grounded in the als and patients/u

s 174

eating

roject flicts. ults.

earch, sues,

to its

by the h into

uality way,

ve to

e of ndent,

same users’

KCE Reports 17

2.1.5. Criteri

2.1.5.1. InvoDACEHTA undinvolved from thfeel co-responacceptance of tcooperative behhigh work ethics

2.1.5.2. InvoIQWIG, NICE involvement protasks to do andthe amount of w

2.1.5.3. InvoThe CIHR poinoccurs in the inresulting in mucomplexity, andresearch result2007, p.1)34. Siresearch-use is“dissemination”“translation” or

2.1.5.4. InvoAs influence anattention to staprocess and itabout the purpoof the outcome.

74

ria of effective st

olved at an earlyderlines that it ishe beginning in thnsible for the pthe different occuhaviour is requireds among the proje

olved in transpaand HAS have

ocess to clarify cd planning. Stake

work needed (the

olved in mutual lnts out the imponteraction betweeutual learning. Td level of engagts and on the nence stakeholderss no more descri” but “knowledg“mobilization”.

olved with satisfnd impact are eluakeholders’ satists outcome. Satiose and power of.

takeholder involv

y stage of the prs important that he project. It helpsproject and creapational backgroud in stakeholder iect participants.

rency e developed speconfidentiality issueholders have to number of meetin

learning ortance of the knn the knowledge

These interactionsgement, dependineeds of the partics are involved in thibed in terms likege exchange”,

faction usive concepts, Nfaction with the isfaction relates f stakeholders, th

S

vement

roject all relevant bod

s to make the parates understandiunds. Implicit is thnvolvement with t

ecific guidelines ues and publicatio

be informed earngs) and tasks to

nowledge exchanuser and the res

s may vary in ing on the naturecular stakeholder he research-use pe “knowledge tran“knowledge inte

NICE recommendsstakeholder invoto realistic expee usability and re

Stakeholder Involv

dies are rticipants ing and hat good trust and

on the on rules, rly about do.

nge that earcher, ntensity, e of the r (Tetroe process, nsfer” or gration”,

s to pay olvement ectations elevance

2Sd••••••••

vement

2.1.6. What areStakeholder involvdeveloped in furth Stakeholder id Communicatio Stakeholder c Dialogue in ca Management Stakeholder in Regular feed- Skills to mana

e the key compovement encompaer sections. dentification and aon at an early stagconsultation all aloase of controversiof stakeholders’ cnvolvement in pro-back to stakeholdage the process o

onents of stakehasses a range of

analysis ge ong the project deies comments oject monitoring ders f stakeholder invo

holder involvemef activities that w

evelopment proces

olvement

15

ent? will be

ss

16

Figure 3: Key c

Figure inspired by

components of s

y Key component o

stakeholder invo

of stakeholders enga

S

olvement

agement in IFC 200

Stakeholder Involv

07, p.127.

2Assinoa12

Tino

B

“vNsuoMg

S

vement

2.2. StakeholdAlso called envistakeholder identstakeholder involvnterested or infloutcome of your analysis has two a

. to clarify who 2. to assess whi

identify which This task is compnterconnections bof stakeholders to

“Collective sitconcerned, th

Box 4: Diving in m

“The cartographyvisualize issues, dNetwork Theory tsocio-technical deunfortunately, not of controversies hMACOSPOL (Magathers eight Euro

Source : Venturini 20

der identificatioronmental scantification and avement. To negleuential stakeholdproject. As men

aims (Kristensen 2are the stakeholdch stakeholders a function to assignplex as one neebefore being able

involve. tuations are alwayhe more intricate th

magma, the carto

y of controversiesdeveloped by Brunto train studentsebate. Today it a much documenas also become tpping COntrover

opean universities

010, p137

on and analysisn when it is qunalysis is the

ect or to miss ideders may impactioned by DACE2007, chapter 1)10

ders; are the most impon them to in the cods to explore theto order it and to

ys intricate and thhey can get.”(Ven

ography of contr

s is a set of techno Latour as a dids in the investiga

is a full reseanted one. […] Recthe object of the rsies on Sciences and research cen

KCE Reports

s uickly performedfirst crucial ste

entification of affect negatively theHTA, this stakeh

0:

ortant to include, aontext of the projee social complexproduce one’s ow

he more actors arenturini 2009, p.6)37

roversies

niques to exploredactic version of Aation of contemparch method, thocently, the cartogEU funded conso

e for POLitics), wnters.”

s 174

, the ep of ected, e end holder

and to ect. xity of wn list

e 7

e and Actor-porary ough, raphy

ortium which

KCE Reports 17

2.2.1. Explo

compl“Social cadescription

2.2.1.1. Obs“Following statementsconsider cStatementsbeings, tecand so on”

Starting by readon websites, inopportunity to them. Figure 4mental health sa careful scrutinPerplexity is reconnections. Incondition for thapproach. But,quickly ordered

2.2.1.2. DesNotes and mapvisibility to diffeto: 1. representa2. influence : 3. relevance

original per

74

re the world : frolexity rtographers shou

ns at once”.(Ventu

servation : from the webs of

s, social cartogconnections that s are always part chnical objects, na” (Venturini 2010, pding texts about t

n press, in legal tidentify potential

4 illustrates in a services for childrny of the successiequired in such en this way, perplhe second-order h quick increase in notes and map

scription : from aps have to rendeerent stakeholder

ativeness : how maactors on influent: even if stake

rspectives and qu

om unfolding com

uld work out theurini 2010, p.2)38

texts to actors relations surrou

graphers are inespread beyond t

t of larger networkatural organisms, p.2)38. the issue under sexts or in scientifl stakeholders an

graphical way hren and adolescenive legal text on thexercise to take ilexity becomes aheuristic objectiviin complexity of

ps.

actors to texts er the complexity rs we need to be

any actors subscrtial positions deseholders are mino

uestion what is giv

S

mplexity to orde

eir observations

unding controverevitably brought the textual univeks comprising hummetaphysical enti

scrutiny (in grey litfic literature) offernd connections bhow the stakehonts can be identifhe subject. into account a lota research tool, ty that characterif connections ha

legible. To give e proportional ac

ribe to their viewperve special attentoritary, they mig

ven for granted.

Stakeholder Involv

ering

and

rsial t to erse. man ities

terature, rs a first between

olders in fied from

t of new and the izes this

as to be

different ccording

oint. tion ght offer

vement 17

18

Figure 4: Prim

Working docume

ary identification

ent from KCE Repor

n of stakeholders

rts 170 “Organisatio

S

s in legal texts

on of child and adole

Stakeholder Involv

escent mental health

vement

h care: study of the literature and an innternational overview

KCE Reports

w” 39

s 174

KCE Reports 17

Figure 5: Acto

Mapping of the in“[Ostheopathy an

74

rs-networks diag

ntensity (or paucity) nd chiropractic: state

gram, connection

of the connections e of affairs in Belgiu

S

ns between oste

between osteopathum]” 40.

Stakeholder Involv

eopathic unions i

hic unions in Belgium

vement

in Belgium throu

m through their web

ugh their website

bsites links. Working

es links

g document from KC

CE reports 148

19

20

2.2.2. Be preA social cartogtrying to identp.7)38: 1. you shall lis2. you shall o3. you shall

methodolog4. you shall a5. you shall s6. you shall a7. you shall

flexible.

2.2.3. OrganUsing the basEuropean MACstakeholder ide1. The glossa2. The docum3. The analys4. The review5. The tree of6. The scale o7. The diagra8. The chrono9. The table o10. The forum

epared before thgrapher has to fify and analyze

sten to actors’ voiobserve from as m

not restrain yougy;

adjust your descripimplify complexity

attribute to each acprovide descrip

nize your stakehsic framework of COSPOL project, sentification and anary of non-controvmentation repositosis of scientific litew of media and puf disagreement of controversies

am of actors-netwoology of dispute of interpretive fram(see the negotiat

he journey follow all these stakeholders’ po

ices more than to many viewpoints as

ur observation t

ptions and observy respectfully; ctor a visibility protions that are a

holder identificaticontroversy ma

several tasks mayalysis, including t

versial elements ory erature blic opinions

orks

mes ion and partnersh

S

recommendationsositions (Venturin

your own presums possible; to any single th

vations recursively

oportional to its weadapted, redunda

ion and analysispping, developedy be required for he production of:

hips chapter)

Stakeholder Involv

s before ni 2010,

mptions;

heory or

y;

eight; ant and

s d in the relevant

2Inopreid

E

FH

vement

2.2.3.1. The gln HTA, GCP or organizations of spictures or diagramesponsible of thdentified, these ac

EXAMPLE : Sourc

Figure used in KCE Hulpmiddelen Komp

lossary of non-cHSR projects, ne

services or practicms to underline their construction ctors may be invo

ces of oxygen for

report 156C “Homepas 2004, Zuurstofa

controversial elemew technologies ces, may be adeqheir various compor implementatio

olved in the study.

home therapy

e oxygen therapy”, ppparatuur – ISBN 9

KCE Reports

ments and devices, comquately describedponents and the aon. After having

page ii41. Source: 90-70918-38-2

s 174

mplex d with actors been

KCE Reports 17

2.2.3.2. TheFrom observatcomplexity of simplifications, documents, i.e.Publication of tkeeping regardconsidered caexperts and sta

2.2.3.3. The“Scientometricsthe analysis oresearch centeideas through literature may identify pools o

2.2.3.4. TheSince articles ano longer neceto quickly scan See the figure the KCE study “

2.2.3.5. The“Cartograpconnected

74

e documentationtion to descriptiointerrelationship.field notes, in

all traces shouldthis material musing information d

arefully to guaraakeholders (see ne

e analysis of scies can reveal the nof co-authorship, rs, journals…) thrlexicographic anbe investigated r ‘schools’ of scie

e review of mediand press releasesessary. Useful too

debates and/or co6 with connectio“Osteopathy and

e tree of disagrephers should not r

and structured i

n repository on, analysis is To assure the

nterview recordin be kept and offer

st be discussed inisclosure and/or

antee trustworthyext chapter).

entific literature networks of scien

the relative autrough citation ananalysis”(Venturini not only to collentists and stakeho

ia and public opis are digitally pub

ols such as internontroversies to idns between osteochiropractic : stat

ement renounce to tracein discourses. A

S

needed to simpe reversibility ongs, raw data, red to public examn the project teampublication rules y relationships b

: bibliometric antific collaboration thority actors (scalysis, and the diff

2010, p.14)38. Sect evidence but olders.

inions blished, hand-seaet crawlers may entify stakeholderopathy websites, e of affairs in Belg

e how arguments position taken o

Stakeholder Involv

plify the of these

archive mination. m. Gate-must be between

nalysis through

cientists, fusion of Scientific

also to

rching is be used rs. used in

gium.”40

are on a

2

TpththdSinc

2

DSuin

vement

specific issueissues. This r(Venturini 201

2.2.3.6. The sc“Cartographerinvestigation, in the scale of

This scale has tpotentially involve he opportunity toheir European udisputes that ariseSee the internanterventions for Gcan it be prevente

2.2.3.7. The di“[…] every aevery networsingle actor. Win an explosioconstellation o(Venturini 201

During a researchSome stakeholderumbrella of organnteresting ways o

e limits the positiramification can b10, p16)38.

cale of controvers are free to but they must bef disputes where ito be chosen to

in the study. Inteo follow connectioumbrella organize at that level of diational confrontatGP’s burnout in “d and how can it b

iagram of actorsctor can be dec

rk can be conneWhat used to be aon of conflicting aof agents can soli10, p.16)38. process, it’s not rs become closer nizations may expf representation o

ions that could bbe represented in

ersies choose the gra

able to situate thit belongs” (Ventu limit the numbeernational compa

ons between natiozation; and to discussion. tion of philosopBurn-out of genebe cured ?”42

s-networks : mapomposed into a

ected tightly enoua single actor can gents and what uidify into a unique

rare that stakeholthan before, othe

plode. Evolutive of these magmatic

be taken on othen numerous ways

anularity of theiheir object of studyurini 2010, p.16)38.er of stakeholde

arisons in projectsonal stakeholdersdescribe coalition

phies underlyingeral practitioners :

pping burnout network and […

ugh to become asuddenly dissolve

used to be a loosee source of action.

lders’ positions evers become distangraphs and map

c movements.

21

er s”

ir y .

ers to s offer s and ns or

g the : how

…] a e e .”

volve. nt. An s are

22

Figure 6 : Acto

Working docume

2.2.3.8. The“Obviously, thecontroversies rsudden cascadmost events ar(Venturini 2010

ors-network diag

ent from KCE Repor

e chronology of de evolution of cremain dormant fe of quarrels. Thisre packed in shor0, p.17)38. But the

gram, representa

rts 148 “[Ostheopath

dispute controversies is for years or decas makes timelinesrt periods separa

ese timelines offe

S

tion of Belgian o

hy and chiropractic:

not uniform: somades and then bus difficult to draw bated by long empr renewed possib

Stakeholder Involv

osteopathic unio

: state of affairs in B

metimes urst in a because ty gaps”

bilities to

dKa

vement

ns and schools

Belgium]”40.

distinguish ‘permaKCE appears in tha flow of interactio

in 2010

anent’ stakeholdehe timeline, it’s inons that may have

ers from intermittteresting to positi

e a long history.

KCE Reports

tent ones. And, ion its intervention

s 174

when n into

KCE Reports 17

Figure 7: Time

Working docume

74

eline of a KCE pro

ent from KCE Repor

oject

rts 165 “[Burnout am

S

mong general practit

Stakeholder Involv

tioners: prevention a

vement

and management]”442

23

24

2.2.3.9. The“To handle thsimplified interinterpretive fra

Figure 8 : Exam

From Moret-Hartm

e table of interpre growing comprpretation grids” ames should r

mple of table of i

man 2007, p.32043.

retive frames plexity of social

(Venturini 2010epresent all th

interpretive fram

S

life, all actors , p.18)38. The t

hese differing p

mes of actors invo

Stakeholder Involv

develop table of point of

vinac

olved

vement

views, showing wnterpretative framand the solution, corresponding wor

where they diverme describes for in

and how is theirrldview43.

rge and where tnstance how actorr underlying syste

KCE Reports

hey may overlaprs express the proem of values, wi

s 174

p. An oblem ith its

KCE Reports 17

2.2.3.10. TheStakeholder idobserving andinformation forarenas where rand the KCE aalso provide a sSee the use of in Belgium (cha

EXAMPLE : DAStakeholders adegree, have ingiven HTA. To during the stake

• Who is th

• Who are

• Who has

• Who pay

• Who is inconven

• Who has

The answers comprising botmanager who involve.

Two specific tyrecommends th

• stakeholdteam (we

74

e forum dentification and d describing”. Sr this environmenelationships betwre lively and vividspace to perform tDebategraph© to

apter Dialogue in c

ACEHTA stakehoare defined by Dnfluence on or aridentify stakeholdeholder analysis (

he initiator?

the users of the r

s to accept the res

ys for the work and

affected: Who bnienced by the res

s the knowledge a

to these questioth individuals anassesses which

ypes of stakehohe use of specific

ders with obviouse call them “oppos

analysis may bStakeholder meent scanning may

ween stakeholdersdly performed. Cothese public deba

o perform public dcase of controvers

older identificatioDACEHTA as gro

e influenced by tders, the following(see below) (Kriste

results?

sults?

d the results?

benefits/profits/hasults?

nd resources, or c

ons will producend institutions. It h stakeholders a

lders are identifimethods to involv

s opposed interesed stakeholders”

S

become more thetings used to y transform into s or between stakeontroversy-websiteates. debate around ostsies)

on oups which, to a he possible chang questions can bensen 2007, p.24

as drawbacks/risk

contributes?

e a list of stakeis primarily the

re the most rele

ed by DACEHTAve them (see belo

ests to the actua);

Stakeholder Involv

an “just gather

political eholders es might

teopathy

marked ges in a

be asked )10 :

ks or is

eholders e project evant to

A which w).

l project

•

2

2LcincliresalicnBse9wsme

vement

stakeholderthe decisioimplementinthem “end s

2.3. Commun2.3.1. Analyse Like identification challenge driven bnteractive model conceived as circnear models, buteached by puttin

sociology of innoactors: informationnk up the netw

conception, sharenetwork. But, while transpstakeholder involvelements in collab9). And these are when sensitive infstakeholders. Litemutual respect aestablished, can n

rs who – at the coon or planning ng the results andstakeholders”).

ication at an eathe risks and beof stakeholders,

by principles of tof stakeholder invulation of messagt as opportunities ng go-betweens ovation means en, things, money,ork actors, they ed knowledge is

parency and accvement, mutual borative processehighly vulnerable formation is disclo

erature on particiand trust takes never be taken for

ompletion of the prprocesses and

d recommendation

arly stage enefits linked to

communication wtransparency andvolvement, commges only, as is thto strengthen a n'into circulation'. verything that ci people and theirhelp to rebind

s a go-between

countability are twrespect and trus

es (Cargo 2008, pto information dis

osed (un-)intentiopatory research time and patiegranted.

roject – will be clowho can assi

ns of the HTA (w

communication with stakeholders

d accountability. Imunication is no lohe case within ranetwork. This couBy go-betweens

irculates betweenr skills44. Go-betwand bind. In suthat strengthens

wo core elemenst are two other p.336)45 (see FIGsclosure problemsonally by experts stresses that bunce and, even

25

ose to st in

we call

s is a n the onger tional

uld be s, the n the

weens uch a s the

nts of core

GURE s, e.g. or by

uilding once

26

Figure 9: Core

From Cargo 2008

e elements of a p

8, p.32945.

artnership proce

S

ess

Stakeholder Involv

STos

2SossthacB

“inm(aaspt

S

2InsInpdInc

vement

Secure your comTo resolve problemother EBM or heasolutions, impleme

2.3.1.1. PublicScientists, industryopportunity to substages in the projestakeholders havehese institutions’ acknowledged anchapter ManagemBox 5: Informatio

Disclosure is a fonterested and afmanner that is un(and ongoing) steactivities, from cand resolution stakeholders […project, its impacthem”

Source: IFC 2007, p

2.3.1.2. Confidn the HAS commsigned by all stanformation disclopublication. Particdisagree with the fn its Guide to thconsultation proce

mmunication ms of perceived blth-related agenciented before publi

c communicationy, professional sobmit comments oects. In the IQWI

e to give their conswebsites. As a

d answered as fment of stakeholdeon disclosure

ormal-sounding teffected parties. Cnderstandable to ep in the process onsultation and

of grievances…] have accurate

cts, and any othe

p.277.

dential communimunication strateakeholders involvosure is prohibitecipants have the final version of thehe methods of teess where eviden

betrayal linked to ies are using two ication of the repo

n ocieties, doctors, on IQWiG's or NIG and NICE comsent to publication

counterpart, eacfully and as factu

ers’ comments).

rm for making infCommunicating syour stakeholderof stakeholder einformed partici

s, will be moe and timely iner aspects that m

ication egy, confidentialityved in guideline ed, both before right to mention

e report46. echnology appraisnce submitted in

KCE Reports

information disclotypes of very diff

ort.

and patients havCE’s work at diff

mmunication straten of their commench comment muually as possible

formation accessibsuch information rs is an importanengagement. All oipation to negotiore constructiv

nformation aboumay have an effe

y clauses have tdevelopment grand after the r

n in the report if

sal, NICE describ confidence is in

s 174

osure, ferent

ve the ferent egies, nts on st be (see

ble to in a

nt first other iation ve if ut the ect on

to be oups. report f they

bes a nvited

KCE Reports 17

from manufactuappraised. Undaccept unpublexample, if thconfidence’) or(‘academic in c

2.3.2. AnticipA 2004 evaluindicated that sof all the larger are re-deployedtheir job descractivities may rsuch as web-coPersonal telepinformation dismoods of interlrespectful andactivities. How these activities“opposed stakewith others, asfocused (see vproject monitori

«These daphone) to various expstakeholde(KCE expetranslated f

74

urers and sponsoder exceptional clished evidence he information isr if its use might onfidence’)” (NICE

pate time-consuation of the sta

stakeholding activorganizations inv

d to cope with theiptions (Ilott 2004require much moonsultation procesphone contacts, closure are not ocutors, developm

d maybe trustwoto deal with it a

s, are two queseholders”, their ins this group is evarious options ining). ays, it takes me

pass all the nexplanation. It almoers] » ert during preparfrom French).

rs of the technolocircumstances, it

under agreemens commercially st adversely affectE 2008, p.22)47.

uming communicakeholder involvevities are now intevolved in this proce workload, and s4, p.3)48. This mere time, even out

sses or stakeholdepersonal mail

unfrequent. Manament of convincinorthy relationshi

and how much timtions that shouldvolvement may b

expected to be vn chapter 2.7. Sta

about one hour ecessary diplomaost seems like we

ration phase of c

S

ogy or technologieis stated that NInt of confidentia

sensitive (‘commet future publicatio

cation activitiesement process iegrated in businescess. New or existtakeholding is inceans that commutside dedicated mer meetings.

exchanges, ofagement of feelinng arguments, bups are time-con

me has to be devd be anticipatedbe more problemavery labour-intensakeholder involve

a day (by e-maiatic steps, this e ‘frighten’ them

consultation proce

Stakeholder Involv

es being ICE “will ality; for ercial in on rights

n NICE ss plans ting staff cluded in unication moments

ff-record ngs and

uilding of nsuming voted to

d. As to atic than sive and ement in

il or with [the

ess,

2

2SdinbtoeFoIninainpth•

•••

vement

2.4. Stakeholddevelopm

2.4.1. Refer to Stakeholder consdecision-making anformation level (between stakeholdo do more than gevolve towards a dFrameworks for aoffer standards to n the case of a nnstance, is usedaddressed adequnternally and extep.2)49. This meanhat the process of The guideline

relevant profe The patients’ The target use The guideline

der consultatioment process

standards sultation is situaactivities, below (see FIGURE 10)ders and experts

give or receive infdeeper engagemeassessing the quaassess the stakeh

new guideline ded to check wheuately and wheternally valid, and s that stakeholdef their involvemene development gessional groups; views and prefereers of the guidelin has been piloted

n all along the

ated at mid-levelthe partnership

). At this consultamay occur as st

formation. Sometient of stakeholderality of clinical prholder involvemen

evelopment, the Ather the potentiather the recommare feasible for pers have been cont may meet thesegroup includes in

ences have been ne are clearly defin among target use

project

l on a continuulevel but above

ation level, discutakeholders are inimes, discussionsrs in the project. ractice guidelinesnt process. AGREE instrumenal biases have mendations are

practice (AGREE 2orrectly consultede requirements : ndividuals from a

sought; ned; ers.

27

m of e the ussion nvited s may

s now

nt, for been both

2003, d and

ll the

28

Figure 10 : Hea

From Health Can

In the case of to assess accein existing guidkey stakeholde• clinical kno• personal ex• policy and • methodolog

appraisal, • information• implementa• and manag

alth Canada’s Pu

nada 2000, p.128

guideline adaptaeptability and appldelines. This task rs. Necessary ski

owledge in the topxperiential expertadministrative exgical expertise

n retrieval expertisation expertise, gerial and facilitati

ublic Involvemen

ation, the ADAPTlicability of the rehas to be performlls include (ADAP

pic area, ise (patient viewspertise, in guideline de

se,

ion skills.

S

nt Continuum

TE instrument is ucommendations amed by a panel i

PTE 2007, p.6)50 :

and preferences)

evelopment and

Stakeholder Involv

used e.g. as found ncluding

),

critical

Br

Anq

•

•

•

•

•

•

•

2Eogrefo

vement

Box 6: The ADAPrecommendation

Assessing whethenot is done by discquestions:

Does the popwhich the (acceptable)?

Does the intcontext of use

Are the interv(applicable)?

Is the necesscontext of use

Are there anyand/or resourthe implemen

Is the recommsetting where

Does the benmake it worth

2.4.2. Refine yEBM agencies pioown methodologicguidelines develoeport publication,orms along the pr

PTE assessment ns

er a recommendacussing each reco

pulation describedrecommendation

?

tervention meet e (acceptable)?

vention and/or eq

sary expertise (ke (applicable)?

y constraints, orgarces in the healthtation of the recom

mendation compait is to be used (a

efit to be gained fimplementing (ac

your process oneering in stakehcal handbooks thapment (see abov, stakeholder conroject developmen

of acceptability

ation is acceptablommendation in th

d for eligibility mais targeted i

patient views an

uipment available

knowledge and s

anisational barrier care setting of ummendation (app

atible with the culacceptable and ap

from implementingcceptable)?

holder involvemenat are quite similave). From projectnsultation occurs nt process.

KCE Reports

and applicability

e and/or applicabhe light of the follo

atch the populatin the local s

nd preferences in

e in the context o

skills) available in

rs, legislation, pouse that would imlicable)?

ture and values ipplicable)?

g this recommend

nt have producedar to standards ust scoping meetinin a limited numb

s 174

y of

ble or owing

on to etting

n the

of use

n the

licies, mpede

in the

dation

d their sed in gs to

ber of

KCE Reports 17

2.4.3. ChoosThe King Baudstep by step pconsultation. Faccount when c1. Objectives2. Topic: The3. Participan4. Time: Amo5. Budget: AvThen, the objeimpose to answneeded. Thesebe asked usingrefer to six quParticipatory Mbelow:

74

se your method ouin Foundation’sprocedure to ideirst of all, the fochoosing your mes: Reasons for inve nature and scopnts: Who is affecteount of time availavailability of resouective of the conwer some questioe questions aboutg a decision tree uestions synthesiethods Toolkit (K

s Participatory Meentify the relevanllowing five elemthod (KBF 2005,

volvement and expe of the issue ed, interested or cable urces nsultation and thons before evaluat nature and obje(see Figure 14). zed in The King

KBF 2005, p.13)51

S

ethods Toolkit hant method to cho

ments must be tap.12-15)51: pected outcomes

can contribute to s

he nature of the ating the time andctives of the projThese decisions

g Baudouin Foun and detailed in t

Stakeholder Involv

as built a oose for ken into

solutions

subject d budget ject may globally

ndation’s the table

Tty

G

Pg

In

Dg

R

Pg

g

h

vement

Table 1 : EBM agypes

Group types

Project scoping group

nventory groupg

Development group

Rating group46 ;

Peer-review grouph

See the VIKC See the NICE

encies’ stakehol

Size

Small, with physical presence

Small, with physical presence

Small, with physical presence

Medium, with physical presence and web-consultation Very large, with web-consultation (“hearings”)

C methodology in FIE and IQWIG flow c

lder involvement

Timespan

At the beginning

At the beginning

All along the project development process

In case of disensus

Before finalization, once or several times, during 4 to 8 weeks.

IGURE 13 charts in FIGURES 1

t’ tasks and grou

Task

To help framingproject, scopingresearch questiochoosing the appropriate methodology To identify a limset of questionsdevelop in the project To develop a project or guidefrom the selectioa topic to the identification of specific questioTo formalize thedegree of agreement needto select proposguidelines, etc. To comment scoreport plan, preliminary reposet of guidelines

11 and 12

29

up

g the g the on,

mited s to

line on of

ns e

ded sals,

ope,

ort or s

30

Table 2: Six qustakeholder in

Motivation

Targeted outp

Knowledge

Maturity

Complexity

Controversy

From: KBF 2005,

2.4.3.1. TheWhat is your mo• “The purpo

their own kdirectly con

• “The purpknowledgedecision-m

What is your ta• “The purpo

to enable explicit) or(mapping

• “The purpoon an issue

uestions to identvolvement

Democratiz

ut Mapping ou

A lot of cocommon kn

Most peopsubject or ttheir opinio

The topic iscomplex no

The topic controversi

, p.13 51

e objective of theotivation for undeose of using the mknowledge to creancern them” (dempose of using te, values and idemaking” (advising)

rgeted output ? ose is to generate a group to discl

r test alternative out diversity).

ose is to enable ae” (reaching cons

tify relevant cons

zation or advising

ut diversity or reac

ommon knowledgenowledge ?

le have already the subject is newns ?

s highly complex or technical ?

is highly contal ?

e consultation rtaking this stakeh

method is to enabate options for ta

mocratization). the method is eas that are rele).

a spectrum of oplose information strategies in a

a group to reach asensus).

S

sultation method

?

ching consensus

e exists or there

formed opinions w; people are still f

or technical or n

troversial or no

holder consultatioble participants tockling (policy) iss

to reveal stakeevant to the pro

ptions and informa(making tacit knpermissive envir

a single informed

Stakeholder Involv

ds with

?

is little

on the forming

not very

ot very

on? o employ sues that

eholders’ ocess of

ation and owledge

ronment”

decision

2T•

•

•

•

2AisTDK

vement

2.4.3.2. The naThink about the to “To what ex

knowledge of “To what exte

legislation on emergent that

“Is the subjecinformation is

“Is the issuepolarised, suc

2.4.3.3. The apAn extended set os available in thToolkit 51 and in thDecision Making 8

KCE projects and

ature of the subjpic or nature of thxtent does the f the subject?” (knent has the socie

the subject ? Dot norms have not ct highly complexrequired?” (comp

e highly controvch that consensus

pproaches of consultation aphe King Baudouihe Health Canada8. Some of these may offer good ex

ject he subject:

society alreadynowledge) ety already develoo strong views exbecome establish

x, such that a greplexity) versial and has s is difficult to reac

pproaches with stain Foundation’s a Policy Toolkit forapproaches havexamples.

KCE Reports

y possess a ge

oped opinions or xist or is the issu

hed?” (maturity) reat deal of (tech

the debate bech?” (controversy

akeholder involveParticipatory Mer Public Involvemee already been us

s 174

eneral

even ue so

hnical)

come y)

ement thods ent in sed in

KCE Reports 17

Table 3 : Cons

APPROACHES

Participatory Monitoring and Scenario BuildiConsensus conDeliberative poSearch confereStudy circles Study groups Think tanks Charrette Constituent assDelphi process

Retreats Round tables Advisory commplanning cell Interactive wwwOnline discussi

Televoting or wIssue conferencNominal group Workshops

i Ongoing p

managem

74

sultation approac

S

AssessmEvaluation (PAMng Exercise

nference lling

ence

sembly

mittee, board/coun

w/e-conferencing on groups/list ser

websurvey ces process

project : 2011-01 G

ment of localised pro

ches and applica

KCE PRO

ment, E)

Child Men Breast canGP burnou GP burnou

cil or All reports

rvers Alternative

osteopathProstate c GP burnou

CP A national clinic

ostate cancer

S

ation in KCE proj

OJECTS

ntal Health39

ncer screening ut42

ut42 s

e Medicines : y40

cancer screeningi

ut42

cal practice guidelin

Stakeholder Involv

jects

ne on the

F

F

vement

Figure 11: The NI

From NICE 2009, p.

ICE key stages o

1052

of clinical development

31

32

Stakeholder orgguideline at vabrings his/her increases the empirical evideof stakeholderNCCs to assimThe flow chartreports and itspreliminary repsubmitted commassessed with Institute will hosubmitted in wdebate are invitIn the VIKC mquestions (“knecomprehensiveassessed. Recquestions, selecIn the KCE Orecommendatiopotentially probby means of a cancer patients

j Ongoing p

managem

ganizations can carious stages duown sectional i

potential for connce for many comrelations for NICilate. t below describes two hearing moports plans or repments during thesregard to their r

old a scientific debwriting. This debated personally53. ethodology, the ielpunten analyse”e list with needscommendations wcted by web-cons

Ongoing Project ons to be scrutinblematic recomme

websurvey to pras.

project : 2011-01 G

ment of localised pro

contribute to and uring its developinterests. The m

nflict. This, combmmon interventionE and to the volu

es the productionoments inviting sports and amendse hearing momenrelevance to the bate on unclear aate is not public.

inventory group s”). This inventorys and sticking p

will have to answsultation.

2011-01 GCPj, ized by KCE exp

endations was duractitioners involve

CP A national clinic

ostate cancer

S

comment on thepment. Each stakmultitude of stake

ined with the pans, adds to the coume of comments

n process of thestakeholders to cdments to it. All cnts will be considereport. If necess

aspects of the co The participants

selects only contry group then propoints which sher a limited set o

to limit the numperts, an inventoring one month pred in the care for

cal practice guidelin

Stakeholder Involv

e clinical keholder eholders aucity of mplexity s for the

e IQWIG comment correctly ered and sary, the omments s of the

roversial duces a ould be of these

mber of ry of 19 resented prostate

ne on the

F

“F

vement

Figure 12: The IQ

Flow chart for repor

QWIG production

rts”, available on the

n process of repo

e IQWIG website 54

KCE Reports

orts

s 174

KCE Reports 17

Figure 13: TheKCE project on

Flow chart based

74

e VIKC key stagen Prostate cance

d on VIKC methodol

s of guideline deer

logy55.

S

evelopment used

Stakeholder Involv

d in

Fr

F

vement

Figure 14: The Corelevant participa

From KBF 2007, p.3

ock Burning & Hatory methods

3215

Honingh decisionn tree to identify

33

34

Figure 15: Com

From KBF 2005,

mparative chart f

p.27 51

for participatory

S

methods

Stakeholder Involv

2

Sinnepre•

•

2AimweimAcsa

vement

2.5. Dialogue “Social cartocomplicated, knots.” (Ventu

Stakeholder consnvolvement. It henetwork, to nurtuexpectations. A sprocess will be chesearch area: In the busine

characterized In the resea

experts, decisan issue and problem (NCC

2.5.1. DialogueAs stakeholders mplementation of with the effectiveeffects, their effecmplementation (coAmongst these dcomplex one. Mscrutinized but alabout acceptability

in controversieographers knowsubtle and ever-c

urini 2009, vol1, psultation is only elps to establish ure a dialogue stakeholder dialoharacterized diffe

ess area, a dialog by negotiation (IFrch area, a dialosion makers and discuss solutions

CHPP 2010, p.14-

e beyond effectivare actors in

f a policy 57, dialogness of public pcts on equity, anost, feasibility, andimensions of pu

Moreover, acceptalso may be “pery may evolve duri

es w that issues

changing to be s.11)37 one preliminarytrust and credibiprocess and to

ogue organized erently in the bus

gue that follows tFC 2007, p.63-68ogue that can beother actors, aim

s they judge suita-15)56.

veness nvolved with thgue with stakeho

policies but also nd mainly with isd acceptability) (Tblic policies, accability is a dimrformed”, i.e. judng stakeholder di

KCE Reports

are always toosliced like Gordian

y step in stakehility, to strengthe

foster more reaafter the consul

siness area and i

he consultation st)7. e established bet

ms to critically exaable for addressin

he objectives alders will not onlywith their uninte

ssues related to Table 4). ceptability is the

mension that magments and posalogue.

s 174

o n

holder n the alistic tation n the

tep is

tween amine ng the

and/or y deal ended

their

most y be

sitions

KCE Reports 17

Table 4 : Dime

Effects

Implementation

From NCCHPP, p

74

ensions for analy

Effectiveness

Unintended effects

Equity

n Cost

Feasibility

Acceptability

p.456

yzing public polic

s What effectstudy have

What are tthe policy?

What are groups?

What are tpolicy?

Is the policy

y Do the relethe policy a

S

cies

ts does the policon the targeted p

he unintended ef

the effects on

he financial cost

y technically feasib

evant stakeholdes acceptable?

Stakeholder Involv

cy under roblem?

ffects of

different

s of the

ble?

ers view

B

F

Sb

Feke

E

•

•

•

•

•

•

•

F

vement

Box 7: Acceptabi

First, it involves su

Secondly, it is inflbeing studied, amo

Finally, the accepexternal to it: theknowledge, belieetc.).

Examples of Key

Which actorsconsidered?

Is the problemissue that me

Is it on the dis

What are stakthis problem?

What type of issue?

What do theyunintended efdegree of coe

What do theyimplementatio

From NCHPP 2010,

ility as the most

ubjective elements

luenced by all theong other things.

ptability of a pole position of eacefs, values and in

y Questions: Acc

s are or will be

m the policy aimsrits intervention?

scussion agenda?

keholders’ reactio

intervention do st

y think of the proffects, its effects oercion it involves?

y think of the coon of this policy?

p.9-1056

complex dimens

s (the judgment of

e other dimensio

licy also dependch actor is deternterests (politica

ceptability

affected by the

s to address con

?

ons to the idea of

takeholders propo

oposed policy? Oon equity, its costs

onditions surround

sion of the analy

f actors).

ns of the public p

s on factors tharmined by that al, economic, sym

e public policy

sidered to be a s

intervening to ad

ose for addressin

Of its effectiveness, its feasibility? O

ding the adoption

35

ysis

policy

at are ctor’s

mbolic,

being

social

dress

g this

ss, its Of the

n and

36

2.5.2. Dialog

“Knowledgresearch wsystematic knowledge

Recent developthese judgmentfor political dep.10)56. Withinprocesses areother means (litdeliberative pro15)56: • Combining• Contextual• GeneratingBox 8 : The De

“A typical delismall group oprocesses havemaker communthe public, pabackgrounds, ina specific issuethe issue is usubefore the seParticipants areask further quesmall sub-grothemselves, witnot a consensudeliberative prodays.”

gue to produce kge synthesis is a swith information froc and transparente” (Bos 2007, p.8)5

pment in knowledts because “thesecision makers thn the context e set up to enrichterature, consulta

ocesses can fulfil

different forms ofizing data drawn

g new knowledge eliberative Proce

iberative processof participants (soe been used to nities but are alsoatients and othernterests and valuee and to debate ually provided in aession or throue often given the

estions before begoups, participantth the goal of dev

us, to inform decisocess exercise c

knowledge synthstrategy for combinom policymakers t way in order to 58. dge synthesis atte perceptions oftehan objective evi

of knowledge h and contextualiation, comments m

at least three role

f “evidence” from the literature

ess – A Snapshot

s exercise involveomewhere betwebring together th

o relied upon to brrs. Group membes, meet face-to-fpotential options

advance, either thugh presentations opportunity to ch

ginning deliberatiots then debateveloping some forsion-making. Depcan range from o

S

hesis ning information fand practitioners promote the use

tach much importen carry more impdence” (NCCHPsynthesis, delib

ize the data collemanagement, etc.es (NCCHPP 201

e on the subject

t

es engaging a reen 10 and 40)he scientific and ring together members, who bring face to weigh evid. Information perhrough assigned rs by relevant hallenge the expe

ons. In a large groe the issues armal recommenda

pending on its struone day to three

Stakeholder Involv

from in a e of

tance to portance P 2010, berative ected by ). These

10, p.14-

relatively . These decision

mbers of diverse

dence on rtinent to readings experts. erts and oup or in amongst ations, if ucture, a e or four

F

E

DreBeforofiafe

Fm

vement

From CHRSF 20095

EXAMPLE : The d

During one dayecommendations

Belgium.. These experts and wereoreign experienceounds with GPs inal workshop wiafter which the staeasibility.

From KCE Reports management] “41

59

deliberative proc

y, stakeholders to prevent and

preliminary recome based on literaes, qualitative inteand stakeholdersth stakeholders,

akeholders assess

165 “[Burnout amon

cess in KCE proj

have deliberated treat burnout mmendations weature review, a serviews with GPss and expert coneach recommendsed its level of pri

ng general practition

KCE Reports

ect

ed about prelimamong the GP

ere proposed by study of nationa, a Delphi study insultations. Durindation was discuiority, acceptability

ners: prevention and

s 174

minary P’s in

KCE l and n two g the ssed, y and

d

KCE Reports 17

2.5.3. DialogBesides observspaces to perfoDebategraph©)arrangedk in co

EXAMPLE : DeThe Debategrastakeholders inmonths debateaccess to profregulations. Pa

From KCE ReporBelgium]40

k As the co

in (Ventur

74

gue beyond conving and represeorm them. Places)- where disputes-production activit

ebategraph© useaph© was used tn osteopathic fielde about issues fession, patient rticipants were tra

rts 148 “[Ostheopat

ontroversy-websites rini 2010, p.18)38.

troversies enting public des –it may be digits are collectivelyties.

ed in KCE projeco perform a debds. This was intelike specificity osafety, competen

ained to use this to

thy and chiropractic:

in the cartography

S

bates, we may tal ones (see exay elaborated and

ct ate on the web bended to produceof osteopathic mnce profile and ool.

: state of affairs in

y of controversies d

Stakeholder Involv

imagine ample of d maybe

between e a two-

medicine, financial

described

2

2AcqawTc•

•

•

IQcInec•••••

vement

2.6. Managem“A good prosatisfaction thoutcome is les

2.6.1. AcknowlAs stated in thcomments, commquality-assurance are addressed apwhen responding tThe following key comments from sta Each commen

factually as po If changes are

must be made Responses a

whole GuideliQWIG has also comments during n these two guidelements in the considered: transparency publication ru accessibility to response time good records

ment of stakehoocess can enhahat their complaiss than optimal” (

ledge comment e NICE protoco

ments received froand peer-review

ppropriately (NICEto comments on tpoints should be akeholders: nt must be acknoossible. e made to the gue clear in the resp

and changes musne Development developed spec

“Hearing” momenelines and in themanagement of

of the process les o comment procee and feed-back

olders’ commeance outcomes ints have been h(IFC 2007, p.68)7.

and respond ol to manage gom stakeholders processes, and it

E 2007, p.123)35. he draft scope antaken into accou

owledged and ans

uideline as a resuponse. st be made with Group before pubcific guidelines onts13 (see examplee IFC good pract

stakeholders’ co

ss

nts and give people

heard, even if the

guidance consultare a vital part ot is important thatNICE principles

nd later. nt when respond

swered as fully an

lt of the comment

the agreement oblication. on the submissioe). ice handbook, seomments have t

37

e e

tation of the t they apply

ing to

nd as

t, this

of the

on of

everal to be

38

EXAMPLE : ThIt is mentioned

• the person(publication o

• all authors potential cosummarized

• the deadline

• the proceduin a separareport plan.meeting minunpublishedpublished.

From IQWIG 200

he IQWIG managin IQWIG guidelin

s) submitting comon the IQWIG web

of comments shonflicts of intered in the documenta

e is 4 weeks after

re regarding the sate document pu. Comments fulfinutes of the sciend documents attac

0753

ement of stakehnes on the submis

mments explicitlybsite;

hould complete aest. The connecation of comment

publication;

submission of comublished simultanilling the formal ntific debate, will ched to comments

S

olders’ commenssion of comment

y give(s) their con

a form for discloctions disclosed s and published.;

mments will be deneously with the

criteria, as wellbe published. Pr

s as evidence will

Stakeholder Involv

nts s that:

nsent to

osure of will be

escribed revised

as the reviously l also be

Er

F

vement

EXAMPLE : Exreceived on the N

From NICE 2007, p.

xamples of resNICE guideline fo

8135

sponses to staor schizophrenia

KCE Reports

akeholder comma

s 174

ments

KCE Reports 17

2.6.2. Use foWeb-based proor IQWIG procea standardizedp.80)35, all comin a Word file, responses fromfollowing inform• Organisatio• Chapter/se

consultatiothe identific

• Commentsunchanged

• ResponsesIn addition, literutilization highlithrough forumssuch a way, thhas to assesscollected in muback to the paplanned in thescientific debatecomments.

74

orms and forumsocedures of stakeedures) propose fd way. As specifments received bwhich is sent to

m the Guideline Dmation. on – name of orgaection – dependinon. This column cacation of commens – comments d. s – blank column frature on the efficights the importans aimed to facilite ADAPTE metho

s acceptability anltiple existing guid

anel to be discuse IQWIG processe may be held to

s eholder commentsforms to collect afied in the NICE

by NICE are enterthe National Collevelopment Grou

anisation that subg on the documean be sorted by th

nts by section. received from

for the developerscacy of interactionnce of face-to-factate interpretationodology proposesnd applicability odelines; and a forssed. Discussion s after “hearing”

discuss any uncl

S

s management (aassessment informE guideline (NICEred into a commenlaborating Centre

up. The table cont

mitted the commeent that has been he developers to

stakeholders,

s to complete. n approaches in rce contact and intn of research ress a form when thof the recommenrum where resultsabout commentsmoments where ear aspects of the

Stakeholder Involv

as NICE mation in E 2009, nts table

e to elicit tains the

ents. sent for facilitate

entered

research teraction sults5. In he panel ndations s are fed s is also

an oral e written

2Wsp123AtoothcC

2

AcupreresToc

l

vement

2.7. StakeholdWe may talk abostakeholders are projectl :

. shaping the p2. implementing 3. and interpretinAt this level of invo define stakeholdof this report) becheoretical modecorresponds to lContinuum (see F

2.7.1. Considemonitorin

At such a level ofcompared to partiused “as an umbrphilosophy of inclesearch processesearch) those w

stakeholders of theThe added value oof a research projcritical review of th

Phases desc

der involvemenout stakeholder in

involved through

urpose and scopethe research and

ng and applying thvolvement, the “brder involvement in

comes a lively nols, this is the level 5 in the igure 10).

r added-values ng f stakeholder invocipatory researchrella term for a susivity and of rec (rather than incwho are intendee research” (Cargof stakeholder invject could be sumhe PR literature. (C

ribed in Cargo 2008

nt in project monvolvement in prohout the entire p

e of the research, d considering conthe research outcoridging dynamic in the economic ation in the researhighest level oHealth Canada’s

to involve stake

olvement, a reseah. “Participatory reschool of approaccognizing the valcluding them ond to be the ben

go 2008, p.326)45. volvement throughmmarized in tableCargo 2008 ,p.33

8, p.33745.

onitoring oject monitoring

process of a rese

textual factors, omes. nterdependency” rea (in the first chrch area. As figurof involvement ws Public Involve

holders in projec

arch project has esearch” (PR) maches that share aue of engaging ily as subjects oneficiaries, users,

hout the entire pro 5 that is based 8-339)45

39

when earch

used hapter red in which ement

ct

to be ay be

a core n the

of the , and

ocess on a

40

Table 5: The p

Level of stakeh

Shaping the sresearch

Research impl

otential added v

holder involveme

scope and purp

lementation and

value of participa

ent Po

pose of the •

••

context Co•••

• Ca•••••••

S

atory research ap

otential added va

Enhanced relehealth systemResearch is reInitiation of owresearch

ontextual advantaResearch is lesEnhanced credLinkage of stubuilding Ethical agreem

apacity, empowerMore targeted Strengthened sIncreased capaAcquisition of sEconomic deveAcquisition of mDevelopment o

Stakeholder Involv

pproaches acros

alue

evance and import

sponsive to the cownership, empow

age: ss disruptive to imdibility for other acdy participants w

ments negotiated w

rment and ownersand efficient plansense of ownershacity of non acadespecialized researelopment through management and of decision-making

vement

s the three phas

tance of research

ommunity of interewerment, and ca

mplementing contectivities due to par

with needed health

with academic par

ship: ning and problemip through active

emic partners to drch knowledge, skemployment oppleadership skills

g skills

ses of the particip

h questions to the

est apacity building t

exts rticipation in PR ph care resources

rtners address co

m solving participation in re

do PR kills, and experienportunities and loc

patory research

e organization, co

through active p

projects by treating resea

ncerns of the com

esearch activities

nce al resource utiliza

KCE Reports

process

ommunity, or publ

participation in th

arch as awarenes

mmunity of interes

ation

s 174

ic

he

ss

t

KCE Reports 17

Level of stakeh

Interpretation research outco

From Cargo 2008

74

holder involveme

and applicatiomes

8, p.338-33945

ent Po

on of the Ca••

•

•• In••• Pa•

Su•

•

•

S

otential added va

apacity, empowerTimely feedbacEnhanced captranslation Enhanced undact Increased capaEnhanced med

strumental use ofEnhanced cultuCreation of invImproved form

articipation: Potential for hdevelopment

ustaining the partnMore effective capacity Augmented intcatalyzed by paImproved linka

Stakeholder Involv

alue

rment, and ownersck of research res

pacity, empowerm

erstanding of hea

acity for health prodia and education

f scientific knowledural and contextuaentories, training ulation of policy re

higher interventio

nership and reseaapplications for f

tersectoral mobiliarticipation in PRges among comm

vement

ship: sults to non acade

ment, and ownersh

alth problems, the

omotion al capabilities

dge: al relevance of demanuals, and hanecommendations

on participation r

arch products: funding and lever

zation of leaders

munity-, state-, and

emic partners and hip from participa

ir root causes, an

eveloped interventndbooks to informand policy chang

ates when end

raging of resource

s, volunteers, age

d federal-level ag

the community oating in research

nd solutions can g

tions, program plam practice

es

users are involv

es due to establis

encies, institutions

encies

f interest dissemination an

galvanize people t

anning, and action

ved in interventio

shed credibility an

s, and businesse

41

nd

to

n

on

nd

es

42

2.7.2. Consi“Being atteveryone th

Stakeholder invEBM agencies’involvement mthroughout thechapter), involvcomments (SeeYet, the stakehEBM agencies • either by li

everyone’s• or by exte

around maof concern

Agencies pionereveal matters consequences,way they deal wof controversy, highest levels o(see examples)

m A distinct

objectivity

ider the way to dtentive to all vihe same status” (volvement in pro processes scruti

may be performede entire processvement is usuallye the HAS figure).holder involvemehave chosen to dmiting their missio

s agreement; nding their missio

atters of concern a. eering in stakehoof concern (stak …) beyond mattwith these matterthe core philoso

of involvement m).

ion made by Bruno

y and the second-de

deal with controviewpoints does (Venturini 2010, poject monitoring hnized for this repd at an early st

s of project devy limited to cons nt level may varyeal with controveron to identify the

on to reveal the and by choosing t

older involvement keholder intereststers of fact (literas of concern mayphy of inclusion tay be weakened

o Latour between

egree objectivity (Ve

S

versy not mean gran.4)38.

has never been fport. Although staktage of the proj

velopment (see tsultation, i.e. che

y depending on rsym matters of fact th

full range of oppto deal with these

enlarge their mis, anticipation of ature evidences). y differ extremely.that is at the heaby exclusion pro

the positivistic ‘firsenturini 2010, p.3)38

Stakeholder Involv

nting

found in keholder ect and the first cks and

the way

hat meet

positions e matters

ssion to possible But the

In case art of the ocedures

st-degree’ 8

F

F

vement

Figure 16: HAS p

From HAS 2010, p.4

procedure followe

446.

ed in formal con

KCE Reports

sensus method

s 174

KCE Reports 17

2.8. RegulaA follow-througkeep accountaon “communicamore constructiabout the projeeffect on them.

2.8.1. Plan yDetermine at an• what inform• to which st• by what me• and how fre

2.8.2. DisplaInternational strecommend to the research ar2008, p.37)60 :• the purpos• the nature • how these • what policie• what the E

questions.

2.8.3. Adaptinvolv

Information reinvolvement: • for intere

involvemeinvolvemen

74

ar feed-back to h principle has to

ability at its higheation at an earlyive if stakeholdersect, its impacts, a

your communican early stage of thmation needs to beakeholders, ethod, equently.

ay relevant informandards in stakebrief stakeholders

rea, this relevant i

e and scope of thof the research qquestions are cures and services aEBM agency can

t your feed-backvement elevancy also d

ested stakeholdent process as ant (IFC 2007, p.88

stakeholders o be applied in stst level. As alread

y stage”, stakehos […] have accuraand any other as

ation he project (IFC 20e reported,

rmation holder engagemes with relevant infinformation should

e stakeholder invuestions; rrently managed w

are already in placn and wants to d

k to the level of s

depends on the

ders, informationa whole has to be8)7;

S

takeholder involvedy stressed in theolder involvementate and timely infospects that may

07, p.88)7:

ent in the economformation. Translad address (Accou

olvement;

within the EBM agce; do about these r

stakeholder

e level of stak

n on the stake displayed to en

Stakeholder Involv

ement to e chapter t will be ormation have an

mic area ated into untAbility

gency;

research

keholder

keholder ncourage

•

•

•

2EfeowE2acSbs

2Rppa

n

o

vement

for consulteresponses tochapter Mana

for main staksufficient levresponsibility;

for end stakprocess is neneed for the themn.

2.8.4. Use elecElectronic mediaeedback informatof stakeholder. EUwebsite, to give sEUnetHTA and e2009, p.88)14. NICas a task to sendcurrent publicationStakeholder meebriefings are othestakeholders (Acco

2.8.5. Give regRegularity of feedproduction proceproviding clear andalso corresponds

Following th

stakeholders decision or presults and re

This was onnew program

ed stakeholderso their comment agement of stakehkeholders being el of details is

keholders, informeeded to make suresults, so that t

ctronic media ana are widely usion on the project

UnetHTA has put takeholders targe

establish a platfoCE and IQWIG had to all interestedns and new commetings, one-to-oer means recommountAbility 2008,

gular feed-backd-back is fundamesses (see figurd ongoing commus to stakeholde

he definition of D

who – at the complanning processesecommendations (Ke result of the CC

m COMPUS61.

s, appropriate ais needed to susholders’ commentrepresented in th

s required to su

mation about the ure that they undhey expect them

nd meetings ed to collect cot development proa Stakeholder Op

eted information arm for virtual coave both an inford parties informa

missions advertisedone meetings, mended to give cp.41)60.

ental in IQWIG, res). To maintaunications is a cleers’ needso. As

DACEHTA, “end smpletion of the projes and who can asKristensen 2007, p.2OHTA stakeholders

acknowledgment stain involvementts); e project monitorustain feeling o

project developerstand that thereand will make u

omments and to ocess to large numpen Forum page about developmenommunication (Nirmation service hation on new prod. follow-up telep

consistent feedba

NICE and HAS rain transparencyearly stated objects underlined in

stakeholders” are ect – will be close sist in implementin

25)10. s consultations abo

43

and t (see

ing, a f co-

pment e is a

use of

give mbers on its nts in elsen aving

ojects,

phone ack to

report y by tive. It the

those to the

ng the

out its

44

EUNETHTA stabe the only wa(Nielsen 2009, In participatory activity, the chasustainability design of parpartnership willaction and instit

2.9. Skills toHow to play aPositive and guaranteed, neenablers are foa status quo, faand many othertowards stakeh• from res

multistakeh• to enablers

involvemenAttitudes, behavneeded to overc

akeholder involvey to “prove” that p.86)14. research, regular

allenges of which (Cargo 2008, p.3

rticipatory effortsl continue and thtutionalized” (Carg

o manage the a significant role proactive attitudeeither by stakeh

ound both sides, dailure to recognizr reasons (Griffitholder involvement

sistors who dholder working s who have identnt (see FIGURE 1viours and rationacome the barrier o

ement policy, “trano interests were

r feedback is consare to maintain t

334)45. “[…] consids can enhance hat research resugo 2008, p.337)45

process of stakin a stakeholde

es in stakeholdeholders, nor by due to vested intee the potential ofs 2007, p.20)62. Tt is very large :

don’t even cre

ified a clear role f8).

ales may evolve wof inability.

S

ansparent processe inappropriately

sidered as a mainrust and respect dering sustainabilthe likelihood tlts will be transla

5.

keholder involvr involvement prer involvement experts. Resisto

erests, interest in f stakeholder invoThe spectrum of a

eate opportunit

for themselves in

with experience. S

Stakeholder Involv

ses may favored”

ntenance to foster ity in the that the ated into

vement ocess ? are not ors and keeping

olvement attitudes

ies for

terms of

Skills are

F

F

2Treesbusa2Doep

vement

Figure 17: The en

From Griffiths 2008,

2.9.1. Be trustwTrust and mutuaesearch (see figu

elements have to stakeholder involvbeginning of the understand the cosidestepped withoand respect, which2007, p.335)45. DACEHTA stressoutcome require ethics among theproject manageme

nablers/resistors

p.2062

worthy and respl respect are th

ure in chapter Combe established anvement. This can

study. “Taking tontext, the peopleout compromisingh, if violated, can

es that construcgood cooperative

e project participent skills are requ

s spectrum

ectful e core elements

mmunication at annd maintained thron be time-consumthe time to becoe, their culture, an the fragile foundhamper methodo

ctive meetings ane behaviour withpants (Kristensenired (see BOX 9).

KCE Reports

s of any participn early stage)45. Toughout the proce

ming, especially aome familiar withnd priorities canndations of mutualological quality” (C

nd, ultimately, a h trust and high 2007, p.27)10. .

s 174

patory These ess of at the h and not be trust

Cargo

good work

Good

KCE Reports 17

Box 9: DACEHand rewarding

It is important in

• Problem fopolicy ques

• Clarificatio

• informationevidence?

• Establishm

• Planning o

• Planning oresults to b

• Planning ohow do we

• Strategy re

From Kristensen

74

HTA recommendag cooperation

nitially to agree on

ormulation and destion) and what do

n of the alternativ

n search strateg

ment of time sched

f the work phase –

of the completion –be presented?

of the implementae do it?

egarding follow-up

2007, p.2710

ations to create t

n:

elineation – what o we want to study

es to be studied

gy – what is av

dule/meeting plan

– who does what

– how, in which fo

ation – if changes

p and problem solv

S

the basis for a g

is the main quesy?

vailable and how

for the project pe

and when?

orm and to whom

s are to take plac

ving of derived qu

Stakeholder Involv

ood

tion (the

w is its

eriod

m are the

ce, then,

uestions

2Wr(sT

A

In

CLo

F

Teqin•••

TEm•••

vement

2.9.2. Be involvWhy should any rationales for stasee Table).

Table 6: Rational

Altruism

nvestment

Compulsion Lost opportunity

From Griffiths 2007,

These rationalesexpert/researcher questions, issuesnvolvement2: the degree of the politicizati the establish

policy is a chaTo answer these qEBM agencies pimeans : clearly stated publication of research (col

EUnetHTA Pr

ved potential stakeho

akeholder involve

e for stakeholde

we get involvedto do irrespectivwe get involveda return on our we get involvedwe get involvedgreat that we competitors aremaintain our po

p.962

s may be fogroups. But we

s and challeng

independence reon of research is ment of differentallenge. questions, solve toneering in stake

rules in standardsgood practices orlaboration) dedicaroject WP6).

older wish to bement are multiple

er involvement

d because we belive of cost d because we perinvestment because we hav

d because the pocannot afford

e doing it, thus weosition

und in a sime have to take ges when talkin

searchers can sean issue, t relationships b

these issues and eholder involvem

s, guides, protocor lessons learned;ated to stakehold

come involved ?e and probably m

ieve it is the right

rceive that there w

e been told we haotential benefits a

not to, or thate must do the sam

milar manner winto account sp

ng about resea

cure is a question

between research

win these challement are using se

ols or handbooks; ; der involvement (

45

? The mixed

thing

will be

ave to are so t our me to

within pecific archer

n,

h and

nges, everal

(as in

46

2.9.3. Be exTo support stathis report, specomponents ofbelow : • to recogniz

be weighed• to take into

and perple• to overcom

(traditore=t• to accept p

(in chapterprocess);

• to review dialogue (in

• to hear, acand querie

• to maintainStakeholde

• to developexperts’ intstakeholde

These skills can

xpert keholder involvemecific skills havef stakeholder inv

ze that research ed up in making poo account a lot of xity (see chapter me inherent probtraditere) (in chapphysical presencer Stakeholder con

one’s own judgn chapter Dialogucknowledge and ms (in chapter Mann a good power er involvement in p an interesting sterest) at its high

er). n be improved by

ment requires moe been overviewvolvement. These

evidence may not olicy decisions (senew connections Stakeholder identblems of betraya

pter Communicatioe of stakeholderssultation all along

ments and positue in controversiesmanage stakeholdnagement of stake

balance, even inproject monitoringstudy to keep staest level (in chap

experience or tra

S

ore expertise. Thrwed for each of e skills are sum

arrive as simple “ee introductory cha

by maintaining optification and analal in translation on at an early stags without any screg the project deve

tions during staks); ders' comments, eholder's commenn controversy (in g); akeholders’ interepter Regular feed

aining.

Stakeholder Involv

roughout the key

mmarized

“facts” to apter); penness lysis); process

ge); een filter elopment

keholder

critiques nts);

chapter

est (and -back to

2HqspthoBp

•

•

•

•

•

•

•

•

F

vement

2.10. guiding pHow can researcquestion regardisummarized a ranprinciples, that cohe backbone of thof stakeholder invoBox 10: Some gupractice

Research mus

Ownership is

The need for e

Conduct a con

Ensure credib

Provide leade

Give adequate

Develop integ

From Nutley 2008, p

rinciples ch inform public ng evidence-bange of guiding prntribute to the likehe first part of thisolvement.

uiding principles

st be translated

key

enthusiasts

ntextual analysis

bility

ership

e support

gration

p.3122

services ? To sed policy, Nurinciples (see BOelihood of evidens report presentin

to support the u

KCE Reports

answer this prautley and al.

OX 10). These guce-based practice

ng the key compo

use of research in

s 174

actical have

uiding e, are nents

n

KCE Reports 17

3. CONEXPE

3.1. RationaBefore embarkprocedures, theprevailing opiniothe opportunityintegrate stakespecifically, thefears, and askIncidentally, theinvolvement, sitheir daily woinvolvement poThe ultimate oapproach in mainto the own potential internmeasures or ad

3.2. MethodThe format choallow for eachexchange betwviewpoints andallow every sinwe opted for a The meetings tNovember 2011In preparation owas drawn umanagement a

74

SULTATIOERTS ale and objectiing into new and e management ofons and attitudes y to contribute ieholder involveme researchers wered to share theire very exercise itsnce the KCE expork by whatevelicy as an organis

objective of this atters of stakeholdcollective knowle

nal barriers and dapt the stakehold

d osen for this conh participant to eween participants,

opinions, and cgle KCE researchseries of discuss

took place in the1. of the discussion up, after severand project team. (

ON ROUND

ves more formalised

f KCE wanted to of the KCE reseaideas and sugge

ment in the resre also polled abr own positive anself was a tangiblperts will be the fir decision to c

sation. consultation rounder involvement medge and experi

threats and byder involvement p

sultation round wexpress him or so as to bring uplarify them in a dher to optimally psion groups with e last half of Octo

groups, a list of qal discussion se(see BOX 11).

S

D OF KCE

stakeholder invoget a better insigarchers, and to ofestions on how search processesout their resistannd negative expee instance of stakrst ones to be affchange our stak

nd is to make tmore effective, byence, by identify

y taking the approcedures accord

was discussion grherself, and for

p in the discussiondialectic way. In participate and comax 8 participanober and the firs

questions to be dessions with th

Stakeholder Involv

olvement ht in the ffer them

to best s. More ces and

eriences. keholder fected in keholder

he KCE y tapping ying the propriate dingly.

roups to a lively

n shared order to

ontribute, nts each. st half of

ealt with he KCE

B

Q

•

•

•

•

•

Q

•

•

•

•

•

vement

Box 11: List of qu

Question 1 – WHO

What are theAuthorities, PAcademia?

What is yostakeholders?

What do yostakeholders?

Do you have ones and why

Do you have relations and

Question 2 - HOW

Many differenis your knowle

Which onesdisadvantages

These methoparticularly pr

Who must inexternalization

Is training or s

uestions for the

O?

e different types Payers, Private In

ur knowledge o?

ou think about ?

fears about the y?

any suggestions tprocesses connec

W?

nt types of methodedge of and exper

s seem more s?

ods require somroblematic to you?

nvest in these mn?

support necessary

consultation rou

of stakeholdersndustry, Patients-

of / experience

the involvemen

involvement of s

to improve your ccted with these?

ds are mentioned rience with these

useful to you

me specific skills? Why?

methods? Internal

y? Your suggestio

und of KCE expe

s : Providers , P-Citizens + Media

with these diff

nt of these diff

some of them? W

capacity to manag

in the document. different methods

u? Advantages

s. Which ones

specialization a

ons?

47

rts

Public a and

ferent

ferent

Which

ge the

What s?

and

seem

and/or

48

• Question 3

• The reporGCP - andyour opiniosubjects? Isome of the

• Why? No Timing? Hu

Question 4 – D

• Different d(see chaptdimensions

Question 5 – O

The participantreport in preparThe moderator knowledge of thof observer andsessions were tAt the beginninwere briefly remexplained and cwas guaranteeresults would cindividual particAfter each sesobserver. The min depth listeniobserver. No fu

3 – WHY?

rts produced by Kd a large spectruon about the staIs it relevant for aem?

added value? Tuman resources a

IMENSIONS OF I

dimensions of invter 4). What ares?

THER SUGGEST

ts were asked to ration to the discu

of the discussionhe KCE and the ‘ed note taker was gtape recorded, anng of the sessionminded to the paconsent was aske

ed, in the sense contain no names cipants. ssion, there was moderator drafteding to the record

ull transcripts were

KCE cover three um of subjects inakeholder’s involall types of subje

Too dangerous? available ... Any su

INVOLVEMENT

volvement are dee your opinions/s

TIONS AND REM

read a draft verussion. n was an externaenvironment’ in wgiven to the KCE nd had a duration n, the objectives articipants, the rued about the tape that the reportinor other recognis

a short debriefind detailed minutesdings, complemene made.

S

fields - HTA, HSside these fieldsvement for the

ects or must be li

Risk for indepenuggestions?

escribed in the dosuggestions abou

ARKS.

rsion of chapter 4

al consultant with which it operates.

knowledge manaof 2 hours. of the discussio

ules of the exercirecording. Confidg and publicationsable details perta

ng of the moderas of each session,nted by the note

Stakeholder Involv

SR, and . What’s different mited to

ndency?

ocument ut these

4 of this

a prior The role ger , All

on group se were dentiality n of the aining to

ator and through s of the

Fli

3TethmmtrMW7pthin

3MTninilApAdMTewpre

vement

Finally, the four sesting all key mess

3.3. Results The groups discuexpress them is sohe key points amessages, but onmessages are simranslations of the Messages have Weaknesses, Opp7, 9, 14, 15, 16, 1person inside this he group. Other mn the four groups

3.3.1. LessonsMessage 1. StakeThe four groups enot new for KCEnvolvement. Sevelustrate methods

A grid has been practical way, thiAppendix providediscussions. Message 2. BoardThe KCE board except for a direcway to automaticaphase of the projecommendations

ession minutes wesages coming from

ussions resulted ometimes differenare the same. Enly selected exammilar. The provide

original messagebeen grouped in

portunities and Tre19 were expressegroup, but withou

messages were ex(1, 2, 6, 8, 10, 12

s learned eholder involvemxpressed the sam

E. KCE has alreaeral examples weor report some leelaborated to p

s grid will be ues grids build o

d's involvement has representativ

ct representation ally get some formects. Yet, the intis not without stir

ere synthesised im the discussion

in several messnt from group to gExamples are g

mples are includeed examples are es (expressed in Dn 4 categories weats) approach ined only by one grut contradiction byxpressed in 2 to 3, 13, 18).

ment is a long stame opinion: stakehady much experiere provided duri

essons learned. present the diffesed in the forthcn the examples

is not optimal ves of the main of the patients.

m of stakeholder iterference of the rring some disquie

KCE Reports

n a final overall regroups.

sages. The way roup, but concept

given to illustrated if the content oas truthful as pos

Dutch or French). with SWOT (Stren mind. Messagesroup and often byy the other membe3 groups (5, 11, 1

anding practice holders’ involvemence of stakehoing the discussio

rent approaches coming process

provided during

stakeholders grIt is perceived asnvolvement at theboard with the p

etude.

s 174

eport,

they tually, e the of the ssible

ength, s 3, 4, y one ers of 7), or

ent is olders’ ons to

in a note.

g the

roups, s one e end policy

KCE Reports 17

“The mostwhere all opportunityabout their “The steprecommen“How to iaccepted. Sbecome ridthe recomm

Message 3: InvSome collabordecision making

“The Commof directorsbeginning competenc

Message 4: To“To be und“Be undersinvolvemen

Message 59: DThe experienceas not experts regularly percei

“The probleexperts. Thbetween eThe stakeh“The objecdifferent. experiencegenerated

74

t important groupthe decision m

y to comment or tor implementation.”p between the dations is not easimprove this? It Some recommenddiculous. Sometimmendations.” volvement of comators stressed thg munities and Regs. They should be

of the projeccies.” o be understood derstood by the othstood by is a misnt. This is importaDistinction betwee with so-called etend also to be

ived as rather uncem is to make a he best way shou

experts and stakeholders are often tctives of expert mIt is important

e, knowledge ... by these meeting

p of stakeholdersmakers are prese

o reject recomme”

conclusion of sy – This is a diffic

is complex; somdations are so mu

mes we are disapp

mmunities he importance to

gions are not repre acknowledged acts. They have

by her ones is cruciassing dimension i

ant.” een stakeholdersexpert meetings astakeholders to a

comfortable or disdistinction betwe

uld be to organizeholders. But it isthe experts for someetings and stake

to define exacand to think abo

gs.”

S

s is the KCE boent. They have endations and to th

a study and cult issue.” me reports are uch reduced that tpointed ourselves

o involve other le

resented in the boand involved from

more and m

al ...” in the dimensions

s and experts at KCE is that moa varying degreesturbing. een stakeholders ze separate meetis difficult in practme topics.” eholder meetings ctly the objectivout the expectati

Stakeholder Involv

oard the

hink

the

not they s by

evels of

oard the

more

s of

ore often . This is

and ings tice.

are ves: ions

3MTfrsth

MTsmcs

MTo

vement

3.3.2. RequestMessage 6. IndepThe concern aborequently expresshould at least unhem, but not to su

“Independenc“We must be the stakeholdstakeholders have a scienti“We must be game. It is ntake their opin“We should restakeholders.“How is it posbe perceived

Message 7: ClearThe KCE researcstakeholders woumechanism of oneclearly earmark ascientific findings p

“It is probablybetween whacoming from opinions anddistinction bet

Message 8 : TrainThe researchers feof stakeholder invo

“For smaller gskills: how to

ts pendence is a prout the risk of sed, in a varietynderstand clearlyubjugate ourselvece must be a prior

cautious that recders. The recomm

don’t agree. Thific background.” very clear from thecessary to annonions as such.” einforce this perce”

ssible to stay indeas independent?”r distinction to inchers are excludild be taken overe sort or another

any input from stproper. y necessary to maat’s coming from

other meetingsd contributions tween scientific asning is necessareel they lack approlvement without groups, it would bmanage the mee

iority losing one’s indy of different way that we are pres to their opinions

rity.” commendations amendations must he recommendati

he beginning aboounce very clearl

eption of independ

ependent? And ho” nclude in the reping by all meansr in the conclusior. Hence, they strakeholders and d

ake a clear distinthe scientific lite

s or inputs. Themust be clearlyspect and meetingry ropriate training toadequate trainingbe useful to be tr

eting, how to give

dependence was ays. The stakehoepared to interacts.

re not oriented bynot change if the

ions must always

out the rules of thely that we will no

dence amongst a

ow is it possible to

port s that the opinioons by a ‘democress the importandistinguish it from

nction in the reporrature and what’se different partsy identified. Thegs is important.”

o engage in new fg. trained for specifithe floor, to boos

49

very olders t with

y e

ys

e ot

ll

o

ns of cratic’ nce to m the

rt ’s s, e

forms

c st

50

a debate (.important pimportant, the group a“To apply t

This perceived actually conducdeal with thecharacterising t

“Involvemeemotions, would be ifirst need to

Finally, accordthemselves wou

“The stakefor all the need to tra

Message 9: EvInsofar as the increase the imevaluate its effe

“Evaluationexpected im“The targeimpact. Thcompared

Message 10. A “It is very to have a This databa “Not an exthe contact

...) Sometimes wepersons. It is miso as to avoid th

and that the patienthe right method foneed of training

cting interviews, e interpersonal, this type of interacent generates emto respect the othimportant. As of yo acquire new comding to the KCuld benefit from so

eholders don’t havprojects. If we w

ain them.” valuation of impaprimary reason t

mpact of the KCE wect. n of the impact mpact is not there

et of the stakehohis way of imprwith other method

A knowledge basdifficult to identifytool, a database ase must be up-toxhaustive databats...”

e don’t dare whenissing and would hat some individunts for instance do

for the right topic, not only pertains meetings, etc…,

relational and ctions.

motions. A traininther ones and to yet, we are not yempetences.”

CE researchers, ome sort of trainin

ve the scientific bawant to involve th

act is necessaryto invest in stakework, the researc

is important: wee.” lder’s involvemenroving the impads. Is it the best ae of stakeholder

fy the stakeholderto identify the a

o-date.” se but just the ex

S

n the participants be useful.” “Thi

ual experts dominon’t dare to speaktraining is neededto the technical but also to the a

psychological

ng to manage thconduct the procet ready for this.

also the stakeng ackground necesshem at all levels,

eholder involvemechers express the

e should stop if

nt is to increase act will have to pproach?”

rs would be usefrs. It would be us

associations by to

xperiences we ha

Stakeholder Involv

are is is nate k.” d.” skills for ability to aspects

hese cess We

eholders

sary we

ent is to need to

the

the be

ul seful opic.

ave,

So

3MAsthfo

Me

Ts

vement

Such a database often solicited duri

3.3.3. PossibleMessage 11: MotAlthough the resestakeholders, the he whole span ofocus on one or tw

“Why to involformulate an stakeholder’s with the quest“The ultimate of the KCE.” “Do we want the case we m

Message 12: Invoeach project

“It is necessaBut we will nintensity. It wi“Involvement project for tinvolvement toperception of “Involvement sure to answetype of subjec

The researchers asteps of a study.

“The involvemthe research, the post-publi

could also help toing the same perio

e avenues tivations of Stakeearchers identify a

discussions alsof possible objectiv

wo goals. lve? Is it more to

advice? What involvement? Do

tions or the methotarget of the man

to increase the amust find new toololvement is alwa

ary to involve the not involve them ill be different andis not monolithic.

the recommendao define the reseaindependence wilwill be different fr

er to the needs. It ct, type of researcacknowledge that

ment of stakeholdat the end to co

ication phase.”

o ”avoid that the od.”

eholders' involvea range of differe

o reveal that theyves and motivatio

o install a democrdo we want to

o we want that eodology?” nagement is to in

appropriation by thls.” ays important bu

stakeholders in ein each step, n

d specific from stu The involvement

ations is quite darch questions. Till be variable.“ from project to pro

depends of the ach, type of questio

the needs may v

ders is important aommunicate the c

KCE Reports

same persons ar

ement ent reasons to eny seldom acknowons, but rather te

ratic process or too reach with thiseverybody agrees

ncrease the impac

he citizens? If it is

ut must be adapt

every single studyor with the samedy to study.” t at the end of thedifferent from an

The results and the

oject. We must beavailable evidenceon ...” vary along the diff

at the beginning oconclusions and in

s 174

re too

ngage ledge

end to

o s s

ct

s

ed to

y. e

e n e

e e,

ferent

of n

KCE Reports 17

Message 13: partnership Involvement is levels of intensi

“ ‘Discuss wwe should “The scienpragmatic: all the staanalyse theaspects cowant to imp“We must implementamust comethem.” “This is anexample oliterature gcountries atrue partnedecided bereforms.”

Message 14: KWhen stressingmean that theyand independeviewpoints only

“If these kerejected. Tmutual resp

Message 15: S“It would beThis steerin

74

Involvement is

perceived to beity, up to the levelwith’ is the minimanalyse if we go ftific character of oit must be implem

akeholders aroune reaction on the oming from the flplement new guidshow them that wation. We have toe from them. We

n internal decisioof a study in twogives only a few care bad. Second pers to define the reefore. Only this wa

Key values are crg the importance oy expect to be rence. They are wi

y if this respect is rey values are no

This respect of valupect of key values

Steering committe useful to set up ing committee wo

s possible unt

e both important l of co-creation an

mum level, and thefurther.” our work is prioritmented. We accepnd the table ... w

floor: this would loor into our reco

delines this is relevwe are ready to tho plan time to do th

must show that w

on of the manageo parts. First paconclusions and thpart: the stakehol

reform.” “This musay, they are true p

rucial of our key values,espected in their lling to respect threciprocal. t respected the sues is also imports.” tee a steering commi

ould gather at the

S

il implementatio

and possible at nd partnership en, project by proj

ty, but we have topt to be influencedwhy not to see allow to include n

ommendations. If vant and importanhink with them abhis, but the questiwe are ready to h

ement. We haveart of the study: he results from olders are involvedst be transparent partners to define

the researchers pown values of ob

he stakeholders a

stakeholders musttant for KCE: this

ittee for each projdifferent stages.

Stakeholder Involv

on and

different

ject,

o be d by and new f we nt.” bout ions help

e an the

ther d as and

e the

primarily bjectivity and their

t be is a

ject. We

MSdEs

3MTsrep

MAisinreo

vement

could use thisvalidate the m

Message 16: InvoSome collaboratodecision making aEuropean Medicinsuggested.

“We should in"This group cinvolvement knowledge ofcould be paid

3.3.4. Treats Message 17: TimThere is a widesprstakeholders will cesearchers ask to

projects. “The problempossible to usat the end of t“The choices we must chanecessary and

Message 18: PatiAlthough patients s very little, if nvolvement is peeasons. There

objectivity, of feas“I think that thvery difficult tthem. It is an

s group to identifymethodology, the rolvement of EMAors stressed the above the Belgia

nes Agency) ; and

nvolve people fromcould be used fois important. T

f the methodologthrough attendan

ing is an obstacread concern thatcost a substantialo acknowledge th

m with these methse these methodsthe project“. are often dictate

ange this way od then request theients' involvemenare invariably seeany, experience

erceived as particare issues of ribility…

he patients are theto involve the patiimportant challen

y the societal trenrecommendationsA and citizens' pa

importance to aan federal structud a fixed panel o

m European Medicfor all projects wThey can acquirgies and the way nce fees."

le t adding more thol amount of extra

his extra workload

hods is the time ts if we don’t have

d by the time thaof working. We me necessary budgnt is the most difen as a central sta

with this groupcularly difficult to representativity,

e most important stients and to workge to imply them

nds of the societys...” anels also involve leveures (EU commisf five to ten citize

cines Agency.” where the citizensre a fairly good

of working. They

rough interactionsa time. Converselyd in the planning o

they take. It is noenough time, e.g

at is available, bumust plan what’s

get.” fficult akeholder group, p at KCE, and implement for seof perceived lac

stakeholders. It isk scientifically withefficiently.”

51

y,

els of ssion, ens is

s’ d y

s with y, the of the

ot g.

ut ’s

there their

everal ck of

s h

52

“The mostwith the passociationinterests... “There arefor instanctheir own a“Patients aTheir opinio“A distincassociationorganisatioknowledge“The identiare not alinfluenced

Furthermore, thwell perceived.

“A distinctcitizens whchallenge i

Message 19: AIn one of the dsensitive data, and collaboratioopenness from

“Many studcancer surpatients wamany requthey are 5 “We considexpectationbarriers. W

t difficult ones arepatients themselvn – There are “

e no such probleme, even if it is nec

agenda.” are only experts inon will not be use

ction must be ns and the gons. The speciale than the general ification’s phase lways patient asby industry.”

he need to disting

tion must be maho are not ill. Theis to identify peop

Are we ready to giscussion groupse.g. on hospital

on from patients KCE in return.

dies publish resulrvival rates. But ant to know which

uests of this type :good hospitals... der that it is notns of the patient

What is our respon

re the patients. Twves or to speak

problems of a

ms with other stacessary to filter a

n their own treatmeful for each reseamade between eneral patient lized associationsones.“ of the patients is

ssociations and t

guish between pa

de between patieey tend to see thile having the cap

give all informati, the issue was rquality indicatorsor citizens, they

lts on the quality these results are

h hospitals are the:’My father has a where must I go?t our role to makts are legitimate.se?”

S

wo options: to spk with the patieavailability, perso

akeholders – induand even if they h

ment. There is a barch question.”

specialized patassociation cup

s have much m

s also difficult. Ththey are sometim

atients and the cit

ents and tax-payings differently.” “acity to discuss.”on to the patientraised of public acs. If we expect opy may well expec

of hospitals, e.ge anonymous. Me best ones. We h

cancer: you say ?’ “ ke rankings. But . We have no le

Stakeholder Involv

peak ents’ onal

ustry have

bias.

tient pola

more

here mes

tizens is

yers/ “The

ts ccess to penness

ct similar

. 5y Many have that

the egal

3TwmAgcTin

vement

3.4. ConclusioThe four discussiwere quite prolifimotivation, frankneAll relevant aspecgroups have beechapter 4. The actual bodynvolvement of the

on ion groups organc and generatedess and dynamismcts of the messaen integrated into

y of experience e stakeholders is g

nized in October d a real added m of the participanges that emergeo the KCE polic

originating fromgiven in appendix

KCE Reports

and November value, thanks to

nts. d from the discu

cy, that is detaile

m past projects .

s 174

2011 o the

ussion ed in

with

KCE Reports 17

4. STAKPROC

4.1. RATIOKCE re

The ultimate oevidence-basedof advice andpractitioners, redegree to whiliterature reviemodeling is notAn external audthe healthcare be more involvdissemination. different stagesconsequently, tFrom a scientielements of thbased upon stawere applied, a

4.1.1. ImpacIn a general mreports, a seriecan be identifieoptimal impact.

74

KEHOLDERCESSES

ONALE : enhanceports objective of KCEd healthcare and d guidelines, p

espectively. The imich this objectiveews and state-ot sufficient. dit, performed in 2policies63 showed

ved, from the veryBeyond scientific

s of the projects, ihe potential impafic point of view

he study results,akeholder input, inand their limitation

ct of the KCE repmanner, over andes of levers potened. On the other h

R INVOLVE

cing the releva

E is to contributehealthcare policy

primarily aimed mpact of the KCEe is reached. Cof-the-art data a

2009 to evaluate d the explicit demy onset of the stuc excellence, stais a means to enhct of the KCE rec, it is important , conclusions anncluding a descris and risks.

ports: levers andd above the scientially increasing

hand, there are als

S

EMENT IN

ance and impac

e in an effective y, through the format policy make

E can be measureClearly, performinanalyses and e

the impact of the mand from stakeho

udies up to the pkeholder involvemhance the relevanommendations. to indicate clearl

nd recommendatiption of the meth

d barriers entific value of tthe impact of KC

so potential barrie

Stakeholder Involv

KCE

ct of

way to mulation ers and ed by the ng good conomic

KCE on olders to phase of ment, at nce and,

ly which ons are ods that

he KCE CE work ers to an

Tr

L

TcSRfeAhORcPMb

vement

Table 7: Levers areports

Levers

The real questionsconcerns are dealtSuggested solutioRecommendationseasible A priori impartialityheard Ownership by stakRecognised scientcomprehensive apPublic support Message (can be)by credible actors

and barriers with

s and relevant t with ns are acceptables are operational,

y: all groups are

keholders tific excellence;

pproach

adopted/relayed

respect to the im

Barriers

e

Mismatch betparadigms of those of stakeViolation of ininterests Real-life cont(financial, orgare ignored KCE perceiveoffending, disPrevailing cohidden agendComplexity isPerception ofjudged

mpact of KCE

tween values andf researchers vs. eholders ndividual/group

text and constrainganisational, other

ed as taking srespectful stancentroversies, tensioda’s are ignored s (partly) disregardf being accused o

53

nts r…)

e ons,

ded or

54

4.1.2. Five sand op

As a researchstrategies to othese actions n 1. Make your sKCE researchemethods, but mof the actual hbefore embarkiget acquainted this specific heand feel? Whatone’s own eyehave had a first 2. Get your scoEvidently, relevall! Our studiesas perceived bymakers. This retensions, contrvalues… Addrecan also be an among end-recEqually importaregulatory cont(‘cui prodest’), c

strategic objectivperate the leverer or research ivercome these beed the involveme

subject tangible ers are supposedmore often than nhealthcare subjecng into a researcwith the subject althcare approact are the troubles s, to have heardt-hand experience

ope and researcvancy is an absos should address y the providers, tequires to get beoversies or conflessing these queimportant step in ipients. ant at this stageext with its financcompeting interes

ves to overcomers nstitution, we ca

barriers or to opeent of stakeholde

d to be specialistot, they have onlct on which they

ch project on a givand to make it soh organized in prwith it on the fie

d from the mouthe, is worth several

h questions righlute prerequisite the really importahe patients, the peyond first impresicts, to understan

estions in a carefthe creation of ac

e, is to get a fucial implications. Wsts, barriers, comp

S

e the impact barr

an mobilize a nuerate the levers. rs, one way or an

ts in matters of ry limited prior kny are appointed. ven subject, it is uomehow tangibleractice? How doeld? … To have se of a patient, or l days of reading w

ht for getting any imant issues and copublic and/or the ssions, to identifynd deep motivatioful, participatory cceptance and ow

ull understandingWhere are the inplexities …?

Stakeholder Involv

riers

mber of Most of other.

research owledge Hence,

useful to . How is

es it look een with even to

work!

mpact at oncerns, decision

y hidden ons and manner,

wnership

g of the centives

3PwashthWigwsa

4TthcWscaaththdojurewaBosa

vement

3. Gain acceptanPeople are rarely wwhen they come oanalyses will not should be aware ohow do they see hey see as a solWhen we ignore sgnored ourselves we should adopt oshould not evade and take a look at

4. Reach clear resThis is all about that will be formconsequence fromWe very often prstakeholder groupcourse also our awareness of oppand discuss them.hat you have listehem, can certadisrespectfully rejonly cause anger ust pertain to thecommendation,

words, if people aas a ground for furBesides being acopinion), as discushould also be reaat least in the long

ce of your reseawilling to readily a

out of a black boxso easily yield v

of the conceptual the problems andution; what are th

strong voices ‘out when we come w

other paradigms our duty to engagother sources of

sults and acceptthe acceptability mulated, and, n

m the preceding poroduce answers

ps would have exp‘raison d’être’.

posing viewpoints. Showing respecened to them in aainly help to ecting other viewand hostility towa

he mere fact theven if they differ

accept our role in rther discussion.ceptable (even ifussed above, whalistic in terms of g term.

arch setup and maccept conclusionx. Likewise, ‘esotevery convincing aframework used bd in what terms dheir underlying vathere’, what else with our solution?and jeopardize Ege into a dialogue‘evidence’

table recommendand feasibility of

normally, this shoints. that are not ma

pected, let alone pEven so, we shs and their mostt for other viewpo

a honest way, eveget respect in wpoints without pards KCE. The ahat KCE formular from those of ththe system, we h

f only acceptablehatever recommefeasibility – mayb

KCE Reports

methods ns or recommendaeric’ or little underanswers. Researby the target audido they spell out alues and worldvican we expect tha This is not to say

EBM; it means thae with other viewp

dations the recommenda

hould be the na

tching what impopreferred, but thishould keep an t important argumoints, by demonstren if you did not f

return. Conveproper argument

acceptability couldated conclusionse stakeholder. In

have at least this

e as having a diffendation we formbe not immediately

s 174

ations stood

rchers ence: what

iews? an be y that at we points

ations atural

ortant s is of acute

ments rating follow

ersely, s will

d thus s and

other asset

ferent mulate y, but

KCE Reports 17

5. Get effectiveThe very best actively dissemreport has gunderstanding, certain, the prevrequire addition

4.2. The fivThe rationale benhance the evthey are also healthcare syst 1. Respect To involve thossimply a matterIt is based on input to offer antaking an omnidisrespect. When showing to expect recipand for the legit 2. TranspareInvolving stakemean, nor requmade perfectly the system demdistance, so aevidence. Morwhichever stakeach of the acto

74

e communicationproof of impact i

minate and get to wgone throughout

acceptance andvious steps shoul

nal efforts of trans

e key Values fobehind each of tventual impact of the translation o

tem and the role K

se who will be mor of respect, and, the premises th

nd that KCE has siscient or paterna

respect towards procal respect in timacy of our rese

ency holders, taking thuires that we takclear from the on

mands that, at a ces to reach valid reover, there sheholder being invors is playing.

n channels and ris when the key work with our rep the long wa

d eventually adopd greatly facilitatelation, disseminat

or fruitful stakethe above-cited othe work of KCE

of a number of KCE is playing in t

ost affected by thin principle, shouat these key stasomething to learalistic stance, cou

one’s interlocutorreturn for our ex

earch.

heir views and opike over these vienset. The specificertain stage in theconclusions in t

hould be a mutvolved in the cour

S

relays target actors theorts. This means y from awaren

ption and ownerse true adoption, bution and advocacy

eholder interactobjectives is not , but, by the samvalues underpinnthis system.

e outcome of ouruld be beyond disakeholders have vrn from them. Conuld be seen as a

r, one is evidentlyxpertise in EBM m

inions seriously, dewpoints. This shc role KCE has toe research, we takthe view of all atual understandirse of a study, of

Stakeholder Involv

mselves that this

ness to ship. For ut it may y.

tion only to

e token, ning our

r work is cussion. valuable nversely, form of

y entitled methods

does not hould be o play in ke some available ng with the role

Tp

3TreLintaao

4Eontoinbis

5Cdeacweo

vement

This goes along process, and abou

3. Objectivity Taking people’s oender them truthf

Likewise, the selnfluenced by pre-akes some judgmand weight he orobjective grounds

4. Modesty Even if KCE is objectivity, it cannonot even to have ao differ in opinionterlocutor. Recobehind deviant opssues that are rea

5. Curiosity Curiosity is a valudirectly aimed at extend to the psyand policies undeconsequence of thwhen venturing epidemiological, eonto shifting groun

with a clear cut the study progre

opinions and viewfully, i.e. without slection of the st-existing convictio

ment, though, to atr she deserves, whenever possib

and should be ot pretend to haveanswered the righon, while refraininognition and apprpinions, can only ally at stake.

uable character trprocuring healthc

ychology, sociologer study. This fohe respectful and

out of the baeconomical certitunds!

communication aess.

ws creates the oubjective interprettakeholders to inons or preferencettribute to each stabut this judgmele.

keeping up its e the definitive an

ht questions. The ng from any formreciation of the vlead to a deeper

rait for a scientistcare policy advicegy and politics beorm of opennessmodest attitude aastion of EBM udes, one better

bout the involve

obligation to somtation or filtering. nvolve should noes of the researchakeholder the attent should be bu

scientific rigournswers to all queschallenge is to be

m of disdain for values and worldr understanding o

. But, when sciene, this curiosity sehind the technols should be a naadvocated above.

with its statisr be prepared to

55

ement

ehow

ot be her. It ention ilt on

r and stions, e able one’s dview of the

nce is hould ogies atural Yet, stical, step

56

4.3. There aWhen decidingprofessional intto mind is a meis just one of thand for each ocould be approprecise set of systematic wayeach stakeholdchoose the mos

are many differg to involve a terest lobby groupeeting with five to he many ways onbjective there mig

opriate and effectcircumstances an

y, one can distinguder interaction shst suitable approa

rent ways to ingroup of key

p of the doctors, tten persons at thee could seek to inght be a number ive. The actual cnd on the expectuish a number of hould be situatedach.

S

volve stakeholstakeholders, e

the first approache KCE premises. nteract with stakeof different meth

choice will dependted outcomes. In dimensions alon

d in order to be

Stakeholder Involv

ders e.g. the h coming Yet, this

eholders, ods that d on the

a more ng which

able to

T

D

Win

R

Cs

LinD

MF

InnLc

C

vement

Table 8. Dimensio

Dimensions

Who to nvolve?

Representation

Circle of stakeholders

Level of nvolvement Desired output

Motivation Format

nformation need Level of controversy

Confidentiality

ons of stakehold

Possible choice

Patients; Genergeneral); Providers; Potecommercial actoDecision makersDirect contact w(representative aActive exclusion(reference) grouselected) Obtain informatioObtain engagemFactual informcontroversies) ↔solutions Advising ↔ DemFace to face ↔ Use of web toolsStakeholders hastake ↔ StakehoSubject is highlyDepends also othe stakeholdersEverything can bcannot be said in

der involvement

es/situations

ral population; M

ntial future proviors, Foreign expers; Administration; with ‘the field’ ↔associations) n of certain groupup ↔ Open,

on from ↔ Listenment ↔ Be partnermation ↔ De↔ Consensus ↔

mocratisation Forum ↔ Survey

s, of social media…ave poor prior knoolders are well inf

y controversial ↔ on the objective os be openly discussn public

KCE Reports

Media (specialise

ders; Industry; Orts in the domain Insurers; Academ

↔ Via represent

s ↔ Closed; seleunselected (i.e.

n to ↔ Discuss wirs escription (e.g. ↔ Co-constructio

y… … wledge of the issuformed Subject is consenof the interaction

sed ↔ Some elem

s 174

d or

Other

mia ation

ected self-

th ↔

of on of

ue at

nsual with

ments

KCE Reports 17

4.4. From sAs apparent froto involve stakeeach of the strmethods) shouare more arbitexperience (seeFurthermore, eaat specific mom Table 9. Timing

1. Make subje2. Get scope a

questions r3. Gain accep

methods 4. Reach reali

5. Get effectivtion channe

74

strategy to concom the first part ofeholders in the strategic objectivesuld be selected. Strary and will poe Table 10, next pach of the objectiv

ments during the p

g of the different

ect tangible and research right ptance of

stic answers

ve communica-els

crete actions f this report, thereudy work of advis, the most approSome choices arossibly have to bpage). ves requires inter

project (Table 9)

t interactions wit

At the very onsBefore finalisi(projectfiche-ficEarly in the reto be repeated Is an ongoidifferent types the project; pparticularly cruFrom pre-finalbut facilitated b

S

e is a wide variety sory bodies like Kopriate method (ore straightforwardbe revised with

ractions with stake

th stakeholders

set of the project.ng the study p

che projet) alisation phase, ponce or twice ng concern, reof interaction thro

pre-finalisation phcial isation phase on

by earlier interactio

Stakeholder Involv

of ways KCE. For or mix of d, others

growing

eholders

protocol

possibly

equiring oughout hase is

nwards, ons

vement 57

58

Table 10: Impa

Stakeholder in

Search of medOn-site visitsmanufacturer, Individual inteIndividual inteproposal, engaFocus groupsDiscussion fodiscussion of inventory grouDelphi methodScenario BuildConsensus coDeliberative poOnline discussSurvey (Web, tUse of social mWorkshop K= To be applied

A number of ostudy processetanks, Charrette

act enhancing st

nvolvement appro

dical and lay press (to healthcpatient…), rviews with ‘typi

erviews with keyaged stakeholde

orum (meeting f contentious poups; rating groupds ding Exercise onference olling sion groups/list stelephone, papermedia

d/developed by KCE

other participatoryes : Search confere, Constituent as

rategies

oaches

ss care institution

ical’ stakeholdery informants (a

er, policy watche

with 8-20 stakoints; project scps.

servers r self-administer

E √

y methods may brences, Study circ

ssembly, Retreats

S

or surgery,

rs author of study r)

keholders), for coping groups;

red, ...)

√ = Other eligible me

be less relevant cles, Study groups, Round tables, A

Stakeholder Involv

1.Make subject tangible

K

K

+

K

+

ethod; could be subc

for KCE ps, Think Advisory

cOp

vement

2.Get scope and research

questions right

+

+

K

K

+

K

√ √ √ √ √ K √ +

contracted; + = M

committee, Board/Online discussionprocess (see KBF

3.Gain

acceptance omethods

+

+

K

+ + +

+ Method could be of h

/council or plannin groups/list serve200551 and Healt

of 4.Reach realistic answers

+

+

K

x

K

√ √ √ √ + + + K

help or contribute

ng cell, Interactiveers, Issue confereth Canada 20008)

KCE Reports

5.Get effectcommunica

n channels

+

+

+ +

e www/e-conferenences, Nominal g).

s 174

ive atios

ncing, group

KCE Reports 17

4.5. Stakeh4.5.1. PrereqAs a prerequisnumber of unavSome of themdimensions liste• Who are th

Given the drepresentaindustry cobe built upneeded in groups, of

• What are ton the stra

• What are th• SWOT ana

o What ao What

involveo What o

The requirementhe type of studwill need otherdebated subjec

4.5.2. ChoicDepending on specific strateglisted in table 1one might deciare perceived tdominance by l

74

holder involvemquisites ite for every stakvoidable question

m correspond to ed in table 8: he key stakeholddifficulty of finding

atives of patient ontacts, key persop. Even so, the order to get the pcitizens, of the hethe significant istegic objective thahe real goals of thalysis (Dimensionsare the strengths

risks are assoement and with a opportunities are nts and expectatidy and subject. Er types of contac

ct.

ce of the most apthe answers to

gic objective, one 10 which one(s) isde not to engageto outweigh the plobby groups. The

ment in Practice

keholder involvems to be answeredchoices that ha

ers to involve ? (Dg the appropriate

organizations, ons in the administhelp of external

perceptions and oealthcare workersssues for the staat is pursued) he involvement ? s 9 and 10) and weaknesses

ociated with thepoor involvementassociated with aons might be qui

E.g. a guideline incts and inputs th

ppropriate methothe above-mentiowill have to cho

s (are) the most ae into stakeholderotential advantage choice will also

S

e

ment exercise, the by the KCE reseave to be made

Dimensions1, 2 anindividuals, a dataof professional tration… should gsubcontractors mpinions of specific‘on the floor’… keholders ? (Will

(Dimensions 4, 5

of the involvemene involvement, wt

a complete involvete different depen

n a non-controverhan a HTA on a

od(s) oned questions,

oose among the mappropriate. Alterr requirement, if tes, e.g. for fear ohave to take into

Stakeholder Involv

re are a earchers. e in the

nd 3) abase of groups,

gradually might be c patient

depend

and 6)

nt ? with no

ement nding on sial field fiercely

and the methods rnatively, the risks of undue account

ma

4SaporekdTidthFs

4

Ad2eTaduIninwaed

vement

more practical issand timing conside

4.5.3. RequiredSome of these cattention should bpatients or with soother parties with esearcher. He or

keeping the righdynamics… Thorough debriefindentifying the “dohe gradual buildinFor other compesubcontractors wh

4.5.4. Further eeligible f

A number of the during the interna2011 (see Chapteelaborated into deThey are briefly advantages and mdescribed explicitlyuse in other contexnsofar as KCE nvolvement methwith much certaintand institutions caexperience, this cduring the coming

ues like the avaierations.

d competenciescompetencies shbe paid to the pometimes exigent,vested interests she may benefit

ht distance, in

ngs after completo’s and don’t’s” ong-up of an experietencies, we co

ho have a specific

elaboration of mfor KCE

methods listed ial expert consulter 3) will in the tailed, operationadescribed in app

merits, as perceivy target their applxts is outside the has still a limiteods, key successty at this stage, aannot readily be tchapter will have few years.

lability of resourc

ould be developsychological asp, suspicious or evmay entail psychofrom training in cothe managemen

tion of a researchof specific approacience and knowleould make use experience in the

methods that wer

in table 10 that tation organizedcoming months

al KCE Process Nopendix, with a foved by KCE reseication to stakehoscope of this repo

ed experience wis factors and riskand the experienctransposed eitherto be updated a

ces and competen

ped internally. Spects. Interactions

ven hostile provideological distress ionflict management of complex g

h project should hches and contribudge base at the K

of the servicee field.

re considered as

have been mentin October-Noveand years be fu

otes. ocus on their sparchers. The me

older involvementort. ith formal stakehks cannot be idence from other cour. Hence, with groand further elabo

59

ncies,

pecial s with ers or in the ent, in group

elp in ute to

KCE. es of

s

ioned ember urther

pecific thods ; their

holder ntified ntries owing orated

60

5. APPEILLUCONEXPE

APPROACHinstitutions

Example(s)

Objective

Timing Who to involve

Selection

Level of involvement Motivation Desired outpu

ENDIX: APSTRATED SULTATIOERTS

1 – On-site vis

KCE Report2005 64 KCE ReportPrevention o65 Make the suclinical and At the onset

e? Providers/hetechnology (under studyguideline. Not necessarather via pecentres andInformation

Avoiding a pt Good grasp

technology, approach, inparticularitie

PPROACHEDURING T

ON ROUND

sits to healthca

ts 20 [Molecular D

ts 46 [Home Moniof Sudden Infant D

ubject tangible (wipractical aspects)t of the project. ealthcare institutio(HTA) or providing

y, or caring for the

arily via representersonal contacts; quality-consciousof the researcher

purely theoretical p of the concrete (c

service or diagnoncluding the regules (if debatable).

S

ES THE D OF KCE

are providers o

Diagnostics in Belg

itoring of Infants inDeath Syndrome]

ith the focus on th).

on currently usingg the service (HS condition covered

tative organizationlook for large-volus providers. rs.

stance. clinical) practicalit

ostic or therapeutiatory/financial

Stakeholder Involv

r

gium],

n , 2006

he

the R) d by the

ns; ume

ties of a c

Afo

DPinn

PmD

FLc

C

L

vement

Appropriate ormat

Duration Prior nformation need

Preparatory material Deliverable

Follow-up Level of controversy

Confidentiality

Limits

Non-interferingprocesses, alteproviders and (any case, inforpatient. Ideally, all reseparticipate. In pediting of the d½ to 1 day. Stakeholders nResearchers cwith the technoregulatory framNone, or a set preparatory reaBrief descriptiono other specifof the final studtechnology or sobservation noIn principle notIn theory, out overy fact that thcreate specific bring out potenvisits. Hence, tthe function in All observationinformation shoAs representatof the researchresults, and doreport.

g observation of thernated/followed b(if needed and fearmed consent is to

earchers involved practice, at least tdiscussion and con

need little prior infoould benefit from ology, medical con

mework. of remaining quesading work. on of the meeting fic deliverable; andy report, describiservice, may beneotes of the researct applicable. of scope for this mhe researches beperceptions amo

ntially controversiathe nature of the vthe project should

ns, and, a fortiori, aould remain strictltivity is not aimed hers cannot be coo, as such, not app

KCE Reports

he actual clinical by discussion withasible) with patieno be obtained from

in the study shouthose involved in tnclusions.

ormation. prior acquaintancndition, jargon,

stions after

(who, when , whe introductory chaping the problem, efit from the persochers.

method; in practicelong to the KCE mng the providers, al aspects during visit, its objectivesd be made very clall patient-relatedy confidential. for, the observatinsidered to be forpear in the study

s 174

h the nts. In m the

uld the

ce

ere), pter

onal

e, the may and the

s and lear.

ons rmal

KCE Reports 17

Risks of the method

How to evaluate?

74

Perception bbased uponpersonal exthese potenother actorsAbsence of know what tsuggestions

bias and preconce a very limited, buperience. This cotial preconception

s, preferentially wisignals that “rese

they are talking abs.

S

eptions in the reseut potentially stronuld require to expns and test them wth opposing views

earchers do not rebout” or similar

Stakeholder Involv

earcher, ng press with s.

eally

As

E

O

TW

S

LinM

D

vement

APPROACH 2 stakeholders

Example(s)

Objective

Timing Who to involve?

Selection

Level of nvolvement Motivation

Desired output

– Individual in

KCE Reports 4Prevention of S65 Understand the‘in the field’. At the onset ofProviders/healtechnology (HTunder study, oguideline; Patients confro‘Normal’ providrepresentativesPatients could social media, spartners speciaInformation of

Avoiding a purmiss importantpatient and clinquestions, but recommendatioRicher view of under study Gopracticalities oftherapeutic apRelevant resea

nterviews with ‘

46 [Home MonitorSudden Infant Dea

e subject also in t

f the project. thcare institution cTA) or providing thr caring for the co

onted with the issuders and patients,s; be recruited via p

sickness funds, oralised in participathe researchers.

rely theoretical stat aspects linked tonician when decidalso in view of forons at the end of the different dimeood grasp of the cf a technology, seproach; arch questions.

typical’

ring of Infants in ath Syndrome], 20

he way it is perce

currently using thehe service (HSR) ondition covered b

ue under study. not their political

patient organisatior with the help of otory work.

ance; make sure no the perception oding on the researrmulating relevantthe project.

ensions of the proconcrete (clinical) ervice or diagnosti

61

006

eived

e

by the

ons, other

not to of the rch t

oblem

ic or

62

Appropriate format

Duration

Prior informatineed

Preparatory material

Deliverable

Follow-up

Level of controversy Confidentiality

Limits

Risks of the method

In depth intePreferentialcollaboratorlocation of cby telephonApproximateby telephon

ion StakeholderResearcherwith the clinSet of half-oIntervieweelines of the Tape/video-consent) anis not requirA synthesis mandatory)publication oIn principle

y All observatconfidentialAs represenobservationto be formalthe study reindividual peInformants mdesirable faPerception bbased upon

erviews. ly face-to-face (1,rs or subcontractinchoice of the stakee, after having fixely ¾ to 1H per ine). rs need little prior rs could benefit fronical/medical aspeopen or open quess should be pre-ininterview. -recording of the fnd notes (full transred); synthesis anof the interview (

. Active informatioof the report low.

tions and recordin. ntativity is not strics of the researchel results, and do, a

eport as qualitativeersons’ views. may retain informcts or opinions).bias and preconce

n a limited number

S

, or max 2 KCE ng researchers), aeholder; second c

xed an appointmennterview (20 to 30

information. om prior acquaintaects. stions. nformed of the ge

focus groups (aftescription of the recd analysis for validation, not

on of participants

ngs should remain

ctly aimed for, theers cannot be conas such, only appe observations of

ation (e.g. socially

eptions in the reser of contacts. Tea

Stakeholder Involv

at the choice : nt.. min. if

ance

neral

er cordings

of the

n strictly

e nsidered pear in

y less

earcher, m

He

vement

How to evaluate?

discussions shscientific objecNo new issuesprojects; main duly identified.

hould address thectivity. s appear at the latpoints of debate a.

KCE Reports

ese challenges to

er stages of the and controversy w

s 174

were

KCE Reports 17

APPROACH(author of stuwatcher)

Example(s)

Objective

Timing

Who to involve

Selection

Level of involvement

74

3 – Individualudy proposal, e

KCE Reportfor diabetesKCE Reportprocesses oKCE ReportPrevention oUnderstand aspects (regVery early, bin the pre-finconclusions

e? The initiatorkey-personsProviders/hetechnology (under studyguideline; Foreign expPatients; paIndustry Providers anorganizationinterests; buInformation

l interviews witengaged stakeh

ts 27 [Quality ands 2], 2006 66 ts 133C Optimisatof the Special Solits 46 [Home Moniof Sudden Infant D the subject also igulatory, financialbefore final choicenal phase, to test s and recommendars of the research s in the administraealthcare institutio(HTA) or providing

y, or caring for the

perts in the domainatient organization

nd patients : via rens; look for balancut limit to 6 à 8 in tof the researcher

S

h key informanholder, policy

organization of th

tion of the operatidarity Fund 67 itoring of Infants inDeath Syndrome]in its more ‘politica, other interests). e of research queacceptability of ations topic, key policy m

ation; ons currently using the service (HS condition covered

n; ns (if applicable).

epresentative ce of viewpoints atotal. rs.

Stakeholder Involv

nts

he care

onal

n , 200665 al’ stions;

makers,

g the R) d by the

and

M

D

Afo

D

Pinn

Pm

D

F

Lc

vement

Motivation

Desired output

Appropriate ormat

Duration

Prior nformation need

Preparatory material

Deliverable

Follow-up

Level of controversy

Avoiding ‘to midiscussions onIdentification oGood grasp of regulatory and potential controRelevant reseaand/or recommIn depth interviPreferentially facollaborators), stakeholder; sevideoconferenc1H to 1.30H petelephone). Stakeholders ndiscussion of cbe sent > 1 weResearchers cwith the regulaexpressed viewSet of half-opeInterviewees slines of the inteTape/video-recconsent) and nis not required)A synthesis of mandatory). Acthe publication Can be high. Eobjectives and made very clea

ss the point’ befon the recommendaf the key questionthe ‘political’ stakfinancial aspects

oversies betweenarch questions; ac

mendations. iews. ace-to-face (1, or at the location of

econd choice : by cing, after having er interview (30 to

need little prior infoconclusions / recoeek in advance. ould benefit from

atory framework, fiwpoints (medical aen or open questiohould be pre-inforerview. cording of the focunotes (full transcrip); synthesis and athe interview (for ctive information oof the report

Even so, the naturthe function in the

ar.

re the final ations. ns to be answeredkes, including the under debate an interest groups. cceptable conclus

max 2 KCE choice of the telephone or fixed an appointm

o 45 min. if by

ormation, except mmendations : te

prior acquaintancinancial implicatioand lay press). ons. rmed of the gener

us groups (after ption of the record

analysis validation, not

of all participants o

re of the visit, its e project should b

63

d.

d the

sions

ment..

if for ext to

ce ons,

ral

dings

of

be

64

Confidentiality

Limits

Risks of the method

How to evaluate?

y All observatconfidential.As represenobservationto be formalthe study reindividual pePerception bbased upondiscussions scientific obNo new issuprojects; maduly identifie

tions and recordin. ntativity is not strics of the researchel results, and do, a

eport as qualitativeersons’ views.. bias and preconce a limited numbershould address t

bjectivity. ues appear at the ain points of debaed..

S

ngs should remain

ctly aimed for, the ers cannot be conas such, only appe observations of

eptions in the reser of contacts. Teamhese challenges t

later stages of thete and controvers

Stakeholder Involv

n strictly

nsidered pear in

earcher, m to

e sy were

A

EO

T

W

S

LinM

D

A

DPn

vement

APPROACH 4

Example(s) Objective

Timing

Who to involve?

Selection

Level of nvolvement Motivation

Desired output

Appropriate form

Duration Prior informationneed

Focus groups

Understandaspects (regorder to formproject. To be plannTo be conduPatients; Gegeneral); Providers; POther commDecision maAcademia Providers aorganizationinterests Listen to

Avoiding ‘todiscussionsIdentificatioInventory ofand / or conregarding th

mat Several meeincludes onBetween 2.

n Several optobjective of

the subject also gulatory, financialmulate the resear

ned at the beginniucted at the approeneral population;

Potential future promercial actors akers; Administrat

nd patients : via rns; look for balanc

o miss the point’ bes on the recommen of the key questf knowledge, percnsensus elementshe topics exploredetings (6-12 peoply one type of stakhours and 3 hourion possible, depethe focus groups

KCE Reports

in its more ‘politic, other interests) ich question(s) of

ng of the project opriate time. ; Media (specialis

oviders; Industry;

tion; Insurers;

epresentative ce of viewpoints a

efore the final ndations. tions to be answe

ceptions, controves of the participantd by the focus grople), each focus grkeholder . rs ending on the .

s 174

cal’ in the

ed or

and

ered. ersy ts ups. roup

KCE Reports 17

Preparatory material

Deliverable

Follow-up

Level of controversy

Confidentiality

Limits

Risks of the method How to evalua

74

A script irelated toincludingTape/vidconsent) recordingA synthemandatothe publicFocus grpeople wopinions to minimtopic of d

y All obserstrictly co Learningbias (avalimited toof debateorganisaParticipa

te? No new iprojects; were duly

n which the topicso a general and og an indication of teo-recording of th and notes (full trags is not required)

esis of focus groupory). Active informacation of the reporoups methodologwith previously ide

in separated homize the level of codiscussion. rvations and recoronfidential. gs will be limited bailability, interest to the perceptions oe is strongly relatetional aspects can

ants do not share t

ssues appear at tmain points of dey identified..

S

s to be addressedr particular questithe timing. he focus groups (aanscription of the ); synthesis and ap (for validation, nation of all particip

ort. gy recommend to ntified heterogene

mogenous groups ontroversy on the t

rdings should rem

by a de facto seleco participate, ....),of the participantsed to the animatorn be very limiting.the real informatio

the later stages ofebate and controv

Stakeholder Involv

d, are on,

after

analysis. ot pants of

invite eous in order targeted

main

ction , are s; quality r skills,

on

f the versy

A

E

O

T

W

S

Lin

M

D

AfoD

vement

APPROACH 5

Example(s)

Objective

Timing

Who to involve?

Selection

Level of nvolvement

Motivation

Desired output

Appropriate ormat

Duration

Survey (Web s

Study 2011-1the supportiveSelect and prproject. EvalurecommendatTo be plannedTo be conducPatients; Gengeneral); Providers; Pocommercial aDecision makNormal providrepresentativeviewpoints anPatients couldsocial media, other partnersObtain informListen to Avoiding ‘to mdiscussions oIdentification oGet quantitatigroups of stakDepending on

Must be adaprespondents (

survey, telepho

0 (GCP) Clinical pe treatment for caioritize the resear

uate the acceptabitions d at the beginningcted at the appropneral population; M

otential future provctors, Foreign exp

kers; Administratioders and patients,es organizations; nd interests; but limd be recruited via sickness funds, o

s specialised in pamation

miss the point’ befon the recommendof the key questiove data, possibly keholders n the project : We

pted regarding ava(think about holida

one survey ...)

practice guidelinencer patients

rch question(s) of ility of proposed

g of the project priate time. Media (specialised

viders; Industry; Operts in the domaon; Insurers; Acad, not their: politicalook for balance o

mit to 6 à 8 in totapatient organisat

or with the help of articipatory work.

fore the final dations. ons to be answereto compare differ

b, Phone, paper,

ailability of potentiays)

65

e for

the

d or

Other in

demia al via of al ions, f

ed. rent

....

ial

66

Prior informatineed Preparatory material Deliverable

Follow-up

Level of controversy ConfidentialityLimits

Risks of the method How to evalua

ion Participantcomplete tQuestionnacall centre,Quantitativtext,...) A synthesiscomment, participantsLow

y All questioGathered dbias (availaselected m...). Low participarticipants

te? No new issprojects; mduly identif

ts must receive dihe survey by advaaire, directive, tec, ..)

ve data (raw data,

s of results (for innot mandatory). As of the publicatio

nnaires should redata will be limitedability, interest to

media format (acce

ipation rate, bias is sues appear at the

main points of debfied..

S

rective on how to ance.. chnical media (we

tables, structured

formation and furtActive informationon of the report.

main strictly confid by a de facto separticipate, ....), a

ess to, acceptabili

in representativity

e later stages of thate and controver

Stakeholder Involv

bsite,

d

ther of all

idential. election and the ity of

y of the

he rsy were

As

E

O

T

W

S

LinM

D

Afo

DPinn

vement

APPROACH 6 stakeholders)

Example(s)

Objective

Timing

Who to involve?

Selection

Level of nvolvement Motivation

Desired output

Appropriate ormat

Duration Prior nformation need

Discussion for

KCE Reports in Belgium: ProStudy 2011-10the supportive Confront draft rresults or recomstakeholders, shopefully moreDepending on the start, duringRepresentativeindustry,…. For patients, prrepresentative Consulting and

Make sure thatstakeholder grounderstood. Identification ofreconsidered, rFace to face mCan be in the K2 hours Participants recdiscussed cleashould be limite

rum (meeting w

167 [Residential cojections 2011 – 2 (GCP) Clinical prtreatment for canresearch questionmmendations to thso as to obtain a me acceptable end pthe points to be dg, or shortly befores of patients, pro

roviders and induorganisations.

d discussion

t the viewpoints ofoups have been id

f elements in the reformulated, bett

meeting of max 15 KCE premises.

ceived document arly identified. Reqed to less than 1

KCE Reports

with 8-20

care for older pers2025] 68 ractice guideline fcer patients

ns or preliminary he criticism of relemore robust and product. discussed, shortly re the end of the s

oviders, policy mak

stry: through

f the principal dentified and

study that need toter argued. to 20 persons.

with points to be quired reading timhour.

s 174

sons

for

evant

after study. ker,

o be

e

KCE Reports 17

Preparatory material Deliverable

Follow-up

Level of controversy ConfidentialityLimits

Risks of the method

How to evaluate?

74

Questions/s

Minutes (at acceptable –minutes; notActive informof the reportCan be very

y In principle nInformation necessarily Researcherindependenat the begingame shoulParticipantsof the officiaFollow-up ofpublication ocould possib

statements that wi

least 2 persons) a– tape recording (t to be transcribedmation of all partict. y high. A strong m

not applicable. obtained are ‘officreflecting the full s could feel to be cy of KCE may nening of the meetind be clear to all p could provide pe

al viewpoint they af reactions of stakof report and reflebly have been avo

S

ll be discussed.

and – if feasible a(for verification of d or kept) cipants of the pub

moderator is advisa

cial viewpoints’, nimage of an issueput under pressu

eed to be clearly sng, i.e. the rules oarticipants. rsonal viewpoint i

are excepted to trakeholders after ection on how proboided.

Stakeholder Involv

nd

blication

able.

ot e. ure; stressed

of the

instead ansmit.

blems

A

E

O

T

W

SLinM

DAfoDPinnPmDF

vement

APPROACH 7

Example(s)

Objective

Timing

Who to involve?

Selection Level of nvolvement Motivation

Desired output Appropriate ormat

Duration Prior nformation need Preparatory material Deliverable Follow-up

: Delphi Metho

KCE Reports 1practitioners: pKCE Reports 1Primary Care pReach a consecontent of recoDepending on the start, duringPractitioners, pindustry “Experts” in theConsulting and

Make the resulimplementableConsensus Web, e-mail or

Several roundsNot necessary

Questionnaire.the aim of the mSemi-quantitatActive informatof the report

od

165 [Burnout amoprevention and ma125 Impact of Acaphysicians 69 ensus about reseaommendations …the points to be dg, or shortly befor

patients, providers

e field d discussion

lts and recommene, get effective com

r postal survey

s over several we

Briefing about thmethod tive data, prioritiestion of all participa

ng general anagement], 2011ademic Detailing o

arch questions,

discussed, shortly re the end of the ss, policy maker,

ndations realistic ammunication chan

eks

he expectancies a

s lists ants of the publica

67

42 on

after study

and nnels

and

ation

68

Level of controversy Confidentiality

Limits

Risks of the method

How to evaluate?

Probably higprocess

y Responses anonymousThe results questions asInstrumentaArtificial conNumber andpart of the re

gh at baseline, an

of the participants depend on the chsked.

alisation by some nsensus d/or content of theeport

S

nd low at the end o

s have to remain

hosen experts, and

participants

e critics concernin

Stakeholder Involv

of the

d of the

ng this

A

E

OT

W

S

LinMD

Afo

DPinn

Pm

vement

APPROACH 8

Example(s)

Objective Timing

Who to involve?

Selection

Level of nvolvement Motivation Desired output

Appropriate ormat

Duration Prior nformation need

Preparatory material

: Workshop

KCE Reports 1practitioners: pStudy 2010-25health servicesBelgium (seconCo-creation of During the rese(pre)-final phasStakeholder grthe most knowCombination oinvitations High: stakeholdpartners. Gain maximumRelevant and athe potential toMax 40 persongroups of max table configuranumber of facilAt least 3 ½ to Series of informquestions or prparticipants willeast 1 week bSee above.

165 [Burnout amoprevention and ma5 (HSR) The organs for children and nd phase, ongoincertain elements earch process, pose roups who will be

wledgeable expertsf ‘official’ represen

ders are deeply e

m relevance, acceacceptable recomo effectively boost ns. Most of the wo6 to 7 persons. R

ation /meeting roolitators for sub-gro4 hours; preferen

mation cards or shroposals on whichll be solicited shouefore the meeting

KCE Reports

ong general anagement], 2011nisation of mentaladolescents in g) of the study

ossibly rather in th

most affected, ans ntatives and perso

ngaged or real

ptability, impact. mendations that hchange.

ork to be done in sRequires appropriam(s), flip charts, oups, … ntially whole day. heets describing th input from the uld be circulated a

g.

s 174

42

he

nd/or

onal

have

small ate

the

at

KCE Reports 17

Deliverable

Follow-up

Level of controversy ConfidentialityLimits

Risks of the method

How to evaluate?

74

Written mateminutes (miplenary discrecording neAllow re-reaparticipants editing) Can be relaadvisable.

y In principle Product will word; to be (explicit disc‘Politics’ invexpectationsfeel to be pushould be mEnd productapplied by ta

erial from ‘rapportn. 2 persons) and

cussions. No full treeded. ading of final resul

for validation (bu

tively high. A stro

not applicable. not be ‘scientific’ clearly identified aclaimer…) vading the study; gs among stakehout under pressure

made clear to all pt is effectively adoarget group.

S

teurs’ of the subgd tape recording oranscript of tape

lt of workshop by t avoid extensive

ng moderator is

in the strict senseas such in the fina

generation of falselders; researchers; the rules of the garticipants. opted, disseminat

Stakeholder Involv

roups; of

re-

e of the al report

e s could game

ed or

vement 69

70

6. REFE1. Andriof J

Unfoldingengagem

2. Nutley S,inform pu

3. Lavis J, Oevidence2009;7(S

4. INVOLVEhealth an2012-01-

5. Ginsburgin knowle

6. Riddle MConsultaInternatiofrom: http://ww$FILE/Pu

7. SequeiraHandbooWashingfrom: http://wwngageme

8. CorporatCommunInvolvem662-2924asc/alt_foeng.pdf

ERENCES , Waddock S. Unf

g stakeholder thinment. Sheffield: Gr, Walter I, Davies ublic services. BrisOxman A, Lewin Se-informed health Suppl 1):S1. E | INVOLVE Supnd social care rese-11]. Available frog L, Lewis S, Zackedge translation. I. Doing Better Bu

ation and Disclosuonal Finance Corp

ww.ifc.org/ifcext/enublicConsultation.a D, Warner M. Staok for Companies ton: International

ww.ifc.org/ifcext/enent_full/$file/ifc_stte Consultation Senications Branch.

ment in Decision M43-X Available froormats/pacrb-dga

folding stakeholdenking. Theory, respreenleaf; 2002. p.H. Using evidencstol: The Policy PS, Fretheim A. SUPolicymaking. He

pporting public invoearch [Eastleigh: m: http://www.invokheim L, Casebeemplementation S

usiness through Efure: A good practicporation; 1998. (0

nviro.nsf/Attachmepdf akeholder EngageDoing Business iFinance Corporat

nviro.nsf/attachmetakeholderengageecretariat, Health Health Canada P

Making. Ottawa: Hom: http://www.hc-apcr/pdf/public-con

S

er engagement. Inponsibility and 19-42.

ce. How research ress; 2008.

UPPORT Tools forealth Res Policy S

olvement in NHS,INVOLVE;2011 [co.org.uk/

er A. Revisiting intcience. 2007;2:34ffective Public ce Manual. Washi0-8213-4342-4) A

entsByTitle/p_pub

ement: A Good Pn Emerging Marktion; 2011. Availa

entsbytitle/p_stakeement.pdf Policy and olicy Toolkit for Pealth Canada; 20-sc.gc.ca/ahc-nsult/2000decision

Stakeholder Involv

n:

can

r Syst.

, public cited

teraction 4.

ington: Available

bconsult/

ractice ets. able

eholdere

ublic 00. 0-

n-

9

1

1

1

1

1

1

1

1

vement

9. Nutley S, MEuropean cEvidence&P

0. Kristensen FHandbook. AssessmenAvailable frohttp://www.sfinal.pdf

1. Caplan N. TAmerican B

2. Freeman R.Boston: Pitm

3. SubmissionIQWIG;2011from: https:/

4. Nielsen C, Kstakeholderthe EuropeaEUnetHTA. 91.

5. Leys M, Reypolitique deinitiatives inBaudouin; 2frb.be/uploaFRB/05%29ications/FR_

6. Becher G, Vprenante devoix. BruxelAvailable froFRB/05%29ications/Pat

7. Tronto JC. ULa Découve

orton S, Jung T, Bountries: diverse

Policy. 2010;6(2):1FB, Sigmund H. HCopenhagen: Dant, National Board om: sst.dk/publ/Publ20

The two-communitehavioral Scientis. Strategic managman; 1984. of comments | Co1 [updated 2011-1//www.iqwig.de/inKristensen F, Bistrs and developing an network for HeInt J Technol Ass

yntens S, Gobert s soins de santé: ternationales et b

2007. 978-2-87212adedFiles/KBS-9_Pictures,_docum_Participatie3.pdfVandenbroeck P, We la politique des sles: Fondation Room: http://www.kb9_Pictures,_documtients_partie_prenUn monde vulnéraerte; 2009.

Boaz A. Evidenceapproaches and c131-44. Health Technologynish Centre for Heof Health; 2008.

008/MTV/Metode/

ties theory and knst. 1979;22(3):459gement: a stakeho

ommenting proce11-23; cited 2012-dex.507.en.htmlrup M, Cecchetti Aa policy for stakealth Technology A

sess Health Care.

M. La participatiorevue de la littéra

belges. Bruxelles: 2-526-5 Available

ments_and_externfWouters A. Les psoins de santé: faoi Baudouin; 2008bs-frb.be/uploadedments_and_externnante%281%29.pdable: pour une po

KCE Reports

e and policy in six common challeng

y Assessment ealth Technology (978-87-7676-64

/HTA_Handbook_

nowledge utilizatio9-70. older approach.

edure (Hearing) [K-01-13]. Available

A, Turk E. Involvineholder involvemeAssessment, . 2009;25(Suppl 2

on des patients daature et aperçu deFondation Roi

e from: http://www

nal_sites/09%29_

patients, partie ire entendre d’aut

8. 978-2-87212-54dFiles/KBS-nal_sites/09%29_df litique du care. Pa

s 174

es.

9-8)

_net_

on.

Koeln: e

ng ent in

2):84-

ans la es

.kbs-

_Publ

tres 43-2

_Publ

aris:

KCE Reports 17

18. Léonard paradigméconomiq

19. Léonard responsaMangez innovatiocrise ? Rl'Associa2010. p.

20. Léonard responsa2010;30(

21. Mamboupratique (GCP). B(KCE); 20from: http://kceeline_low

22. Albertijn Jonckheeen langagfemmes ede santé based Adhttp://kceconseils-inte%CC

23. Patient aExcellenchttp://wwchome.js

24. NICE. Ca132. 201http://ww

74

C. La responsabime pour refonder lque/Ethics and EcC. Un nouveau p

abilisation capacitE, Nyssens M, Re

ons économiques Regards interdiscip

tion d’Economie S113-31. C. Fixer les priori

abilisation "capaci(322):622-4. rg F, Gailly J, Wepour l’accouchem

Bruxelles: Centre f010. KCE Reports

e.fgov.be/sites/defw_risk_birth_0.pdfM, Baeke A-M, Jaer P. Accouchemege courant à intégenceintes. Bruxel(KCE); 2011. KC

dvices and Recome.fgov.be/fr/publica-scientifiques-trad

C%81gr and public home [Nce;2011 [cited 20

ww.nice.org.uk/getsp aesarean section:1. Understanding

ww.nice.org.uk/nice

lisation capacitan'Etat-providence?conomics. 2012;Aaradigme pour lesante. In: Degavereman P, editors. Tet sociales en Eu

plinaires - Actes dSociale: Presse U

tés en soins de satante" des citoyen

ei-Hong Z. Recomment à bas risque.fédéral d’expertises 139B (D/2010/1

fault/files/page_dof anssens S, Bielenent normal : consegrer aux documenles: Centre fédéra

CE CLEAR (Commmmendations) 1B ation/report/accouuits-en-langage-c

National Institute f12-01-11]. Availabtinvolved/patientsa

Information abou NICE guidance Aemedia/live/13620

S

nte: un nouveau ? Éthique et Accepted for publics assurances socre F, Desmette D, Transformations europe : quelles sordu Colloque de Universitaires de L

anté: pour une ns. Prescrire.

mandation de bon Good Clinical Prae des soins de sa0.273/63) Availa

ocuments/kce_13

n F, Mambourg F, eils scientifiques tnts d’information dal d'expertise des

mon Language EvAvailable from:

uchement-normal-courant-a%CC%8

for Health and Clible from: andpublic/patienta

ut NICE clinical guAvailable from: 0/57166/57166.pd

Stakeholder Involv

cation. iales: la

et rties de

Louvain;

nne actice nté ble

9c_guid

traduits des soins idence-

-0-

nical

andpubli

uideline

df

2

2

2

2

2

3

3

3

3

vement

25. NICE. Treatadults: InforUnderstandfrom: http://w

26. NICE. DoneAlzheimer's guidance 21534-5) Avahttp://www.n

27. Thomas V. involvementles soins deFondation R

28. Caldon LJMConsumers Health ServStudies. 201

29. Milne RG. Revaluation oand SMS reof Consume

30. Ward LJ, Rhsocial care eJournal of C

31. Barr J, Ogdwhat patientinvolvementof Consume

32. Avard D, Stin health geJournal of C

33. Thomas P, Whealth serviinform future2010;34(5):

ting generalised armation about NICing NICE guidancwww.nice.org.uk/nepezil, galantamin

disease: Informa17. 2011. Understilable from: nice.org.uk/nicem"We were the grot at NICE. In: Pro

e santé et la politiqRoi Baudouin. M, Marshall-Cork H

as researchers –vice Research. Int10;34(5):547-50.

Reducing non-atteof the effectiveneseminders at a Scoer Studies. 2010;3hodes CA. Embededucation – a univ

Consumer Studiesen K, Rooney K. Vt partnership in clt of educationally er Studies. 2010;3anton Jean M, Grnomics: the reality

Consumer StudiesWilson C, Jones Pce users and caree policy. Internatio525-31.

anxiety disorder anCE clinical guidelince 197 (978-1-849nicemedia/live/13

ne, rivastigmine anation about NICE ttanding NICE guid

edia/live/13419/5oup's conscience"-oceedings of 'Patieque de santé'; 201

H, Speed G, Reed– innovative experernational Journa

endance at speciass and cost of patittish health board

34(5):570-80. dding consumer cversity office's pe

s. 2010;34(5):596-Viewpoint: let's teinical practice canengaged patients

34(5):610-2. régoire G, Page My behind the polic

s. 2010;34(5):508-P. Strengthening ters in Wales: a foconal Journal of Co

nd panic disorder ne 113. 2011. 936-463-8) Availa314/52602/52602nd memantine for technology apparadance (978-1-849

3624/53624.pdf - patient and publents, partenaires 11; Bruxelles:

d MWR, Collins KAriences in UK Natial of Consumer

alist clinics: an ient-focussed boo

d. International Jou

culture in health anrspective. Interna-602.

each medical studn be, with the s. International Jo

M. Public involvemcies. International -24. the voice of mentcus group study toonsumer Studies.

71

in

able 2.pdf

aisal 936-

ic dans

A. ional

oking urnal

nd tional

ents

urnal

ment

al o

72

34. Tetroe J.Research

35. NICE. Thhttp://ww7.pdf

36. IQWiG | cited 201procedur

37. Venturiniactor-net2010;19(

38. Venturinidigital mebefore pr

39. MommerN, AnnemadolesceinternatioKCE = Fefédéral dKnowledgAvailablehttp://kceldren_me

40. De GendMélard FBelgium]Health CAvailableand-chiro

41. Van MeeBuyse B,AssessmCentre (KAvailabletherapy

Knowledge Tranh: A Primer. Focuhe guidelines man

ww.nice.org.uk/me

Basic principles [K2-01-12]. Availab

res.926.en.html i T. Diving in magtwork theory. Pub(3):258-73. i T. Building on faethods. Public Unrint(Dec.5). rency G, Van Denmans L, Schoentjeent mental health conal overview. Heederaal Kennisce'expertise des soige Centre; 2011.

e from: e.fgov.be/sites/defental_health.pdf dt T, Desomer A, GF, et al. [Ostheopa

. Health Services are Knowledge C

e from: http://kce.fopractic-state-of-aerhaeghe A, Anne, Benoit K, et al. H

ment (HTA). BrussKCE); 2011. KCE e from: http://kce.f

slation at the Cans Technical brief.

nual. London: NICdia/FA1/59/Guide

Koeln: IQWIG;201ble from: https://ww

ma: How to explolic Understanding

ults: how to reprenderstanding of Sc

n Heede K, Verhaees E, et al. Organcare : study of thealth Services Res

entrum voor de geins de santé = BeKCE Reports 170

fault/files/page_do

Goossens M, Hanathy and chiroprac

Research (HSR)Centre (KCE); 201fgov.be/publicationaffairs-in-belgiummans L, Haentjen

Home Oxygen Theels: Belgian HealtReports 156C (D

fgov.be/publication

S

nadian Institutes o2007;18.

CE; 2007. AvailabelinesManualChap

11 [updated 2010-ww.iqwig.de/meth

ore controversies wg of Science.

sent controversiecience. 2010;onlin

eghe N, Swartenbisation of child an

e literature and ansearch (HSR). Bruzondheidsdzorg =lgian Health Care

0C (D/2011/10.27

ocuments/KCE_1

nquet G, Léonard ctic: state of affairs. Brussels: Belgia0. KCE Reports 1n/report/ostheopa

ns P, San Miguel Lerapy. Health Tecth Care Knowledg

D/2011/10.273/25)n/report/home-oxy

Stakeholder Involv

of Health

le from: pters200

-11-24; hods-

with

s with ne

broekx nd n ussels: = Centre e 3/78)

70c_Chi

C, s in

an 48

athy-

L, chnology ge ygen-

4

4

4

4

4

4

4

455

vement

42. Jonckheer PDe Troyer Mand manageBelgian Hea165 Availabamong-gene

43. Moret-Hartmassessmentthe drug meAssessmen

44. Amblard H, traduction. Lorganisation

45. Cargo M, Mresearch: st2008;29:325

46. HAS. Practi2010. Availsante.fr/port06/guideline531.pdf

47. NICE. Guide2008. (1-84http://www.ne2008.pdf

48. Ilott I, MeadInvolvementthe experienmidwifery an

49. AGREE. AG50. ADAPTE. M51. Elliott J, Hee

Methods ToFoundation;

P, Stordeur S, LebM, et al. [Burnout aement]. Health Sealth Care Knowledble from: http://kceeral-practitioners-

man M, van der Wt and ill-structured

ebeverine. Internat in Health Care. 2Bernoux P, Herre

Les nouvelles appns. Paris: Seuil; 19

Mercer SL. The valtrenghtening its pr5-50. ce guidelines “Folable from: http://wtail/upload/docs/ae_by_formal_cons

e to the methods 4629-741-9) Avainice.org.uk/media

d J, Roberts J, Hamt in NICE Guidelinnce of the allied hnd nursing. Octob

GREE Instrument Manual for Guidelinesterbeek S, Luke

oolkit: a practitione; 2005. 90-5130-5

beer G, Roland Mamong general prervices Research dge Centre (KCE)e.fgov.be/publicati-prevention-and-m

Wilt GJ, Grin J. Head problems: a casational Journal of T2007;23(3):316–2eros G, Livian Y-Fproches sociologiq996. lue and challengeractice. Annu Rev

rmal consensus” www.has-application/pdf/201sensus_quick_me

of technology applable from:

a/B52/A7/TAMetho

mmond R. Enhanne Development: ealth professions

ber 2004. training manual. 2ne Adaptation v1.ensmeyer CJ, Sloer’s manual. Bruss506-0 Available fro

KCE Reports

M, De Schampheleractitioners: preve(HSR). Brussels:

); 2011. KCE Repon/report/burnout

management alth technology e study concerninTechnology 23. F. Une sociologie dques des

es of participatory v Public Health.

method. Paris: HA

11-ethodology_guide_

praisal. London: N

odsGuideUpdated

ncing Stakeholder Learning lessons , health visiting,

2003. .0. 2007.

ocum N. Participatsels: King Baudouom: http://www.kb

s 174

eire J, ention

orts t-

ng

de la

AS;

_110

NICE;

dJun

from

tory uin

bs-

KCE Reports 17

frb.be/upFRB/File

52. NICE. Hostakehold(1-84629http://ww_are_dev

53. IQWIG. Greports: shttps://wwsion_of_c

54. IQWiG | 09-23; cithttps://ww

55. VIKC. Drevaluerefrom: http://wwe/Draaibo%20aang

56. Morestin Synthesiz2010. Avhttp://ww

57. AlexandeRole RelaChronic Iprint(Nov

58. Bos V, VZonMw; 2

59. CHSRF. achieve mServices Available

74

ploadedFiles/KBS-s/EN/PUB_1540_ow NICE clinical gders, the public an9-850-4) Availableww.nice.org.uk/meveloped_4th_edn_Guideline on the ssample. Koeln: IQww.iqwig.de/downcomments_on_prProduct preparatted 2012-01-13]. Aww.iqwig.de/produraaiboek richtlijnenn van richtlijnen. U

ww.oncoline.nl/uplooek%20openbaregepast%20voor%F, Gauvin F-P, Hzing Knowledge available from:

ww.ncchpp.ca/docser JA, Hearld LR, ationships and PaIllness. Health Sev. 18).

Van Kammen J. Kn2007. Public engagememeaningful publicResearch Found

e from:

-_Participatoty_tooguidelines are devnd the NHS. londoe from: dia/62F/36/How__FIANL_LR.pdf submission of com

QWIG; 2010. Avainload/Sample_Gureliminary_reportstion [Koeln: IQWIGAvailable from: uct-preparation.92n. Onwikkelen, imUtrecht: VIKC; No

oaded/docs/Algeme%20versie%202%20drukwerk_moi.

Hogue M-C, Benoiabout Public Polic

s/MethodPP_EN.Mittler JN, Harvey

atient Activation arvices Research.

nowledge synthes

ent (part II): how dc engagement? Otation; 2009 Feb.

S

olkit_New_edition.veloped: an overvon: NICE; 2009 Ja

NICE_clinical_gu

mments on prelimiilable from: uidelines_on_the_s.pdf G;2011 [updated 2

29.en.html mplementeren en ovember 2009. Av

meenOncolineEnP%20november%2pdf t F. Method for

cies. Montréal: NC

pdf y J. Patient–Physmong Individuals 2011;Epub ahead

sis : a guide. The

do deliberative prottawa: Canadian HInsight and action

Stakeholder Involv

pdf iew for anuary

idelines

inary

_submis

2011-

vailable

Pallialin02009

CCHPP;

ician with

d of

Hague:

ocesses Health n 49

6

6

6

6

6

6

vement

http://www.c49_e.pdf

60. AccountAbilLondon: Acchttp://www.a202010%20

61. CCOHTA. SConsultationHealth Techhttp://64.26.pdf

62. Griffiths J, MBackgroundpreventing NWHO; 2008http://www.winvolvement

63. E.P P, Vijfvithe impact oZoetermeerhttp://www.k55_hsr_imp

64. Hulstaert F,L, Vernelen Technologyknowledge CAvailable frodiagnostics-

65. Eyssen M, KMonitoring oSyndrome]. Belgian HeaKCE reportshttp://kce.fgprevention-o

chsrf.ca/Migrated/

lity. AA1000 StakecountAbility; 2008accountability.org/0PRINT.pdf Summary Report on Sessions. Ottawhnology Assessme.163.205/media/p

Maggs H, George d paper prepared fNoncommunicable8. Available from: who.int/dietphysict.pdf inkel D, Vennekenof the Belgian Hear: 2010. Availablekce.fgov.be/sites/dpact_study%20kce Huybrechts M, VK, et al. [Molecul

y Assessment (HTCentre (KCE); 20om: http://kce.fgov-in-belgium Kohn L, Lambert Mof Infants in PreveHealth Technolog

alth Care Knowleds 46 Available fromov.be/publication/of-sudden-infant-d

/PDF/insightAction

eholder Engagem8. Available from:/images/content/5

of 2004 COMPUSwa: Canadian Cooent (CCOHTA); 2df/compus_stake

E. Stakeholder Infor the WHO/WEFe Diseases in the

calactivity/griffiths-

ns A, van Hoesel alth Care Knowlede from: default/files/page_e.pdf

Van Den Bruel A, Car Diagnostics in

TA). Brussels: Belg05 22/11/2005. Kv.be/publication/re

M-L, Van Den Steention of Sudden gy Assessment (Hdge Centre (KCE)m: /report/home-mondeath-syndrome

n/Insight_and_Ac

ment Standard.

5/4/542/AA1000S

S Stakeholder ordinating Office fo004. Available froholder_report_ap

nvolvement: F Joint Event on Workplace. Gene

-stakeholder-

P, Daue F. Studydge Centre.

_documents/2008

Cleemput I, BonneBelgium]. Health gian Health Care CE reports 20 eport/molecular-

een D. [Home Infant Death HTA). Brussels: ); 2006 21/12/200

nitoring-of-infants-

73

ction_

ES%

or om: ril05.

eva:

y into

8-

eux

6.

-in-

74

66. Mathieu al. [QualiClinical PCentre (Khttp://kcecare-for-d

67. GuillaumL, et al. OSolidarityHealth Creports 1http://kceprocesse

68. Van den Sande SProjectioBrusselsreports 1http://kcepersons-

69. BorgermaFallon C,physicianHealth Creports 1http://kceon-prima

C, Nobels F, Peetty and organizatio

Practice (GCP). BKCE); 2006 12/05e.fgov.be/publicatidiabetes-2

me P, Moldenaers Optimisation of they Fund. Health Seare Knowledge C33C (D/2010/10.2

e.fgov.be/publicaties-of-the-special-sBosch K, Willemé, et al. [Residentians 2011 – 2025 ].: Belgian Health C67 Available from

e.fgov.be/publicatiin-belgium-projecans L, Dubois C, , et al. Impact of ans. Health Serviceare Knowledge C25C (D/2010/10.2

e.fgov.be/publicatiary-care-physician

ters G, Van Royeon of the care for russels: Belgian H

5/2006. KCE reporion/report/quality-

I, Bulté S, Debruye operational procervices Research Centre (KCE); 201273/46) Availableion/report/optimisasolidarity-fund é P, Geerts J, Breal care for older p. Health Services Care Knowledge C

m: ion/report/residenctions-2011-%E2%Rieppi S, Vanhae

academic detailinges Research (HSR

Centre (KCE); 201273/16) Availableion/report/impact-

ns

S

n P, Dirven K, Wediabetes 2]. Good

Health Care Knowrts 27 Available frand-organization-

yne H, Devriese Scesses of the Spe(HSR). Brussels: 0 28/07/2010. KC

e from: ation-of-the-opera

eda J, Peeters S, Versons in BelgiumResearch (HSR).

Centre (KCE); 201

tial-care-for-older%80%93-2025 eren S, Geukens Ng on primary care R). Brussels: Belg0 29/03/2010. KC

e from: of-academic-deta

Stakeholder Involv

ens J, et d

wledge om: -of-the-

S, Kohn ecial Belgian

CE

ational-

Van de m :

11. KCE

r-

N,

gian CE

ailing-

vement

KCE Reportss 174