Deciding how to decide
Multi-Criteria Decision Analysisis (probably) the future for
both Health Technology Assessment
andShared Clinical Decision Making
Jack DowieProfessor Emeritus of Health Impact Analysis
London School of Hygiene and Tropical [email protected]
Abstract• I introduce JUDEMAKIA, a map of the world of Judgement and
Decision Making, in which we can locate various Belief Technologies, Preference Technologies and Decision Technologies in the framework provided Ken Hammond’s Cognitive Continuum of changing Analysis to Intuition balances.
• I locate (Single-Criterion) Decision Analysis – SCDA - and particular implementations of Multi-Criteria Decision Analysis – MCDA (Analytic Hierarchy Process, Annalisa) on this map
• I introduce some applications of MCDA in both the health service context (Health Technology Assessment - HTA) and clinical context (Shared Clinical Decision Making - SCDM)
• I suggest that Decision Resource-Effectiveness Analysis – DREA - is the appropriate way to tackle the meta decision of how to decide to decide – i.e. determine which Decision Technology should be used for a specific decision in a specific context, given the relative weights assigned to the high level criteria of rigour and practicality
• DREA is best implemented via MCDA
Some of the slides in the handout may not appear in the presentation but they have been included for later follow-up if desired.
Taxonomy of tasks and technologies• To produce beliefs*, elicit beliefs and evaluate belief
claims we need a Belief* Technology [*Knowledge/Evidence]
• To establish what preferences are held and elicit preference judgements we need a
Preference* Technology [*Value]• To make a decision we need a
Decision TechnologyDTs need inputs from BTs and PTs
and BTs and PTs need inputs from DTsand the transfer process requires ITs and CTs
• To transfer information we need anInformation Technology
• To communicate we need aCommunication Technology
Examples• Belief or Knowledge Technologies
judgement, interview, panel, cohort study, RCT• Preference or Values Technologies
judgement, interview, Visual Analog Scale, Standard Gamble, Time Trade-Off, DCE
• Decision Technologiesjudgement, coin toss, meeting, pro/con checklist, F&F decision analysis, Comp Stoch Decision Model
• Information / Communication Technologiesbody language, ppt presentation, Report with tables and graphs, Youtube video, twitter
A very inefficient BT- Gary Larson
A very simple PT– Randy Glasbergen
A moderately analytic DT
3 Risk factors
Black
Male
Baseball capbackwards
4 Mitigating factors
White
Groceries
Humming Motown
Over 40
3 Risk factors
White
Male
Wrong neighbourhood
4 Mitigating factors
Loafers
Fed Ex envelope
Whistling Sondheim
Polo shirt
– Garry Trudeau
A fairly inappropriate ICT– Nick D Kim
2005 2006
Underlying framework• Cognitive Continuum Theory of Ken Hammond
Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice (Oxford UP, 1996)
• 2 dimensions: • COGNITIVE dimension (how we think about a JDM task)
which runs from highly Intuitively to highly Analytically• TASK STRUCTURE dimension (how a JDM task presents itself to us)
which runs from Very well-structured to Very ill-structured
• Any instantiation of a BT, PT, DT or I/CT can be• characterised by its Analysis-to-Intuition ratio / balance• ‘quality’ assessed by the high-level criteria of Coherence and
Correspondence [not today]• Location of a JDM on the Task Structure dimension does
induce (?) (should induce?) matching process on Cognitive dimension
Highest demands and requirements
Lowest demands and requirements
Control,ability to
manipulatevariables
Timeand
resources(??)
Analyticalskillsand
capacities
Interventional-ObservationalBorder Conflicts
Qualitative-QuantitativeBorder Conflicts
MODELLING
ARGUMENTATION
INTUITION
INTERVENTION
OBSERVATION
INTERVENTION
‘INSTINCT’
‘Taking Into Account and Bearing In Mind’• ‘Taking things into account’• ‘Giving considerations due weight’• ‘Establishing the right balance’• ‘Keeping things in proportion’• ‘Taking a measured view’• ‘Bringing everything into the equation’• ‘Figuring it out’• ‘Seeking a degree of consensus’• ‘Gauging the impact’• ‘Making sure things add up’• ‘Summing up…’
• note how TIABIM DT, a basically qualitative discourse, is given a quantitative flavour
Key rule• The products of BTs in B-land (beliefs =knowledge,
evidence, judgments, opinions) and of PTs in P-land (preferences=values, importance weights) can only be input into a DT in D-land at the same latitude as that DT
• E.g. an effect size from an RCT created at the A-I ratio (level of explicitness and precision) of Randomistan can only become an input into TIABIMIA by being moved to the A-I ratio characteristic of Conferalot (Don’t be deceived by appearances or claims)
• E.g. an intuitive probability judgement from Judgia can only become an input into Analysia by being moved to the A-I ratio of Modelia
• Analysing up/down = Dumbing up/down ??
HEALTHIMPACTASSESSMENT
E-BGUIDELINES
Evidencebase
Patient’spreferences
Note the position of the apostrophe
HTAI
HTAI
TIABIM – the verdicts• TIABIM is fine, no problems, flaws• TIABIM is not perfect, it does produce lots of
bad/poor decisions, but this is because• the wrong people with the wrong values dominate –
bring in the true/right ‘stakeholders’ and it will be fine• we have the right people with the right values, but they
lack knowledge/information/evidence – supply them with better k/i/e and it will be fine
• we have the right people but TIABIM processes need improvement
• we have the right people but many currently lack the relevant TIABIM skills - build their capacity in these and it will be fine (classics education?)
BELIEFSYNTHESIS
PROBABILITIES
VALUESYNTHESIS
PREFERENCES?
HTAII
SCDA
HTAII
SCDA
TreatEff0.70
85.00; P = 0.13
TreatNotEff0.30
0.00; P = 0.05
TestPos0.90
X remains15.00; P = 0.02
TestNeg0.10
X0.20
TreatWorthless85.00; P = 0.08
TestPos0.10
NoTreatment100.00; P = 0.72
TestNeg0.90
NotX0.80
Test and Treat if Positive
89.81
TreatEff0.70
85.00
TreatNotEff0.30
0.00
X0.20
NotX0.80
85.00
Treat without testing
79.90
X0.20
15.00
NotX0.80
100.00
Neither Test nor Treat
83.00
Choice in possible case of XdisuTreat=1 5priorprobX=.2pTestNegg iv enNotX=.9PTestNegg iv enX=1-pTestPosgiven XpTestPosgiven No tX=1 -p TestNeg givenNotXpTestPosgiven X=.9pTreatEffective=.7uFH=100uX=1 5
Test and Treat if Positive : 89.81
Archetypal decis ion 2An otherwise fully healthy patient presents with symptoms and s igns which lead you to suspect either X or Z. Z is of no clinical s ignificance and the patient will fully recover very quickly with an hour's res t. We can therefore define Z as NotX.X has serious consequences if not treated quickly and successfully.A tes t exists for X but it is not perfect either in terms of sensitivity or specificity. We will assume it has no underiable aspects (disutility) as a process.A treatment exists for X but it is not certainly effective. Moreover it has some undesirable aspects (disutility) as a process. But if treated effectively the patient will return immediately to full health (FH)Evaluate the 3 options.
George Bernard Shaw The Doctor’s DilemmaSir Colenso Ridgeon has developed Opsonin, ‘works’ (?) but only when a test is done and result correctly interpretedLouis Dubedat : a dissolute artist and ‘bounder’ (needs opsonin)Blenkinsop : an outstanding GP who has dedicated his life to the poor … and so is poor (needs opsonin)BB : a very successful consultant who has dedicated his life to the rich and so is rich and doesn’t believe in testingJennifer Dubedat : the beautiful wife who Ridgeon falls in love with
George Bernard Shaw The Doctor’s DilemmaSir Colenso Ridgeon has developed Opsonin, ‘works’ (?) but ontest is done and result correctly interpretedLouis Dubedat : a dissolute artist and ‘bounder’ (needs opsoninBlenkinsop : an outstanding GP who has dedicated his life to thand so is poor (needs opsonin)BB : a very successful consultant who has dedicated his life to so is rich and doesn’t believe in testingJennifer Dubedat : the beautiful wife who Ridgeon falls in love w
The problem of the Single Criterion• Most common single criteria are Life
Expectancy/survival/mortality; ‘natural outcomes’; individual biomarkers; generalised ‘utility’…
• The QALY is a multi-attribute measure, bringing together Life Years and HRQOL (the Quality Adjustment)
• And the HRQOL measure is often based on a multi-dimensional instrument (e.g. Euroqol has 5)
• But it is still used as a single criterion
• Many other considerations are not ‘QALY-able’ –not incorporable into a ‘super QALY’
Patient-centred care• Requires patient’s preferences• If we use a population derived QALY tariff (set of weights
over the 5 dimensions of health) we are applying those of mean or median patient
• But it is not just that the QA is not for this patient… there are other considerations in their mind…• ‘Post-process’ Considerations (e.g. bother of follow-ups)• ‘In-Process’ Considerations (e.g. experience of surgery)• ‘Pre-Process’ Considerations (e.g. anxiety before surgery)
• Hazen’s GALY (Goal Achieved Life Years) is a recent attempt to add ‘extrinsic’ considerations to QALY (e.g. desire to live to see child’s wedding)
• Brazier, Dixon, Ratcliffe (2009) try to rescue SCDA• But it can’t deal with multiple considerations and certainly
not in a specific decision support role.
Note the position of the apostrophe!
Conclusion re SCDA• In context of either the clinical consultation OR an HTA the
necessary condition is that the stakeholder/s are able to indicate the relative importance of multiple considerations in real time with minimal complexity
• Requiring them to make numerous pair-wise comparisons or process alternative vignettes are simply not practical, despite the impressive normative arguments for them as rigorous elicitation methods
• Visual analog scales (or their equivalent) for separateattributes are the only serious contender
• SCDA is simply not a viable basis for a Clinical Decision Support System or a Health Technology Assessment
MCDA - variations• MCDA covers a multitude of different techniques• Mainly differ in the way they address issues like:
• the way criteria are assessed• the application and computation of weights• the algorithm used to derive the overall ranking• the model to describe individual preferences (compensatory
versus non-compensatory criteria, linear versus non-linear preferences)
• the uncertainty embedded in the data• the ability of stakeholders to participate in the process.
• Two emerge from my ‘MCDA’ quest in both contexts• Analytic Hierarchy Process• Annalisa
Baltussen and Niessen• The more analytical approaches that have been developed
over the past decades offer little guidance to policy makers.• Concentrate on single criteria such as effectiveness, cost-
effectiveness, burden/impact, equity• If address more than one criterion don’t advise on how to
integrate or judge the relative importance of each - policy makers need to make choices taking into account such multiple criteria simultaneously.
• Moreover, often do not cover all criteria that are relevant to policy makers (particularly ‘soft’ ones)
• In other disciplines MCDA (transport, agriculture, natural resource management) is routinely used in similar problems
TIABIMIA
ANALYSIA
van Til et al. 2008 p461
• “As was shown by Hummel et al. patients value different aspects of treatment compared with health professionals.”
[and compared with other patients] • “The AHP model used in this study could be a
way to include the personal aims, wishes and demands of each patient… different trade-offs regarding criteria importance could lead to a different preference for treatment.”
But…
• “The time requirements to complete the model [7 hours] were considered a disadvantage,
• and the AHP was thought to be bothersome for use in day-to-day decision making. This reservation might be a result of the large size of the current model. Simpler models might be possible, although care must be taken to maintain important details.”
Annalisa
Annalisa• An “intermediate” Decision Technology• Sits on boundary between Tiabimia and Analysia
- a ‘boundary object’ in itself• Makes possible optimal practical balancing
• intuition and analysis• rigour and relevance• complexity and practicality
• One screen picture of whole decision with interactivity
• Equally useful in HTA and CDM – important if link is ever to be established
Reprise…Step 1
TIABIMIA to ANALYSIAto increase analysis to intuition ratio
Step 2SCDA to MCDA
to increase coverage of attributes / outcomes / considerations not able to be synthesised into Single Criterion
Step 3AHP to Annalisa
to increase practicality, communicability…
Contexts• Clinical • Public• Public-clinical regulatory, e.g. NICE, PBAC, …• NICE currently struggling to deal with its responsibility
to ‘take into account’ multiple considerations in addition to clinical and cost-effectiveness
• Increasing emphasis on rigour of the Cost-effectiveness but with a QALY definition of effectiveness that leaves ‘other considerations’ to be processed with almost no rigour… e.g. terminality!
• Why not MCDA?
NICE attributes/considerations to be TIA• This exemplar set is derived from Guide to Technology Appraisal and other
NICE documents, including appeals (e.g. that on Bortezomib). Clearly it would be NICE’s task to come up with an appropriate set (as well as organize/outsource ratings and weightings) for an MCDA.• pClEff= probability that the NT is clinically effective relative to the
Comparator• pCostE20k= probability that the NT is cost effective relative to the
Comparator at a WTP below £20,000 per QALY• Acceptability/Appropriateness/Preferences [of patients and professionals]• Terminality= End of Life Use• Orph/NoAlt/Rescue= NT is 'orphan drug' OR has no alternative besides Best
Supportive Care OR is used in a 'Rule of Rescue' situation• OtherEq= Other Equity considerations• DHpriorities= clinical priority area as designated by Secretary of State for
Health and Welsh Assembly Government • HSFeasability/Impact= Health Service Feasibility/Impact• Innovatoriness• WiderSocietalConsiderations
DREA• HTA in the form of Cost Effective Analysis (CEA)
addresses the adoption decision: what is the best intervention to undertake, given this set of options, their Effectiveness, and a Willingness to Pay (WTP) Cost threshold per incremental unit of effectiveness?
• When undertaken stochastically and extended to VOIA (Value of Information Analysis) it simultaneously addresses the research decision: what is the best uncertain parameter to find out more about and how?
• In contrast, DREA addresses the decision decision - what is the best way to choose between intervention (and research) possibilities, given the alternative decision processes available and a decision resource threshold per unit of decision effectiveness?
• We can think of it as Value of Analysis Analysis
Practical usefulness(multi-attribute index based on key high-level criteria
of (a) social/individual fit and (b) time/resource constraints )
'Scientific'rigour
(multi-attribute indexbased on
key high-levelcriteria of
(a) logical coherence(b) empirical
correspondence)
0 1000
100 MinimumUsefulnessThreshold
Annalisa
AHP
CDA
CBDM
CJ
Choosing a D(S)T• The meta-decision faced by all practitioners and patients
and public policy makers ishow should we decide?
• There is no such thing as a ‘good decision process’ (or DT) in the abstract, only one that performs well on specified attributes with specified weights
• So MCDA can provide the framework within which to evaluate Decision Technologies (for a specified task in a specific context)
• DREA suggests that sometimes (often? usually?) Annalisa will be the optimal DT given the Willingness to Decision Resource
Why do economists ignore opportunity cost?• CEA/HTA1 (like most analytical techniques) continues to be
developed in pursuit of ever-greater theoretical/conceptual normativity and empirical/statistical rigour.
• It is taken for granted (e.g. Drummond, Schwartz, Jönsson et al 2008) that this continuous raising of the ‘best practice’ level by scientific standards is a good thing and it is occurring with remarkably little interest in, or attention to, either the ability and willingness of decision owners to resource best/good practice decision processes of this kind, or the complexity of decision owner’s maximand.
• Rex Brown has been one of the few people exploring the reasons for this growing gap and for the consequent failure of these advanced techniques to spread as widely as the professionals involved have urged and expected.
• His primary explanation lies in ‘analyst mismotivation / misalignment’, in the form of analysts’ pursuit of professional advancement/standing, mental comfort and financial well-being -which interact of course…
Descriptive
How close should ‘prescriptive’ be to ‘normative’ – and how far from
‘descriptive’?
Prescriptive?
Prescriptive?
Prescriptive?
Prescriptive?
Normative
Descriptive
Prescriptive?
Prescriptive?
Prescriptive?
Prescriptive?
Prescriptive?
Prescriptive?
Normative
WTDRX
WTDRY
WTDRZ
DE
DR
Predominantly Intuitive DT
Predominantly Analytical DT
rigour oughtis• an affliction in which the setting of unrealistically or
impractically rigorous requirements on some criterion or criteria leads to rejection of realistic / practical means of improvement … and hence maintenance of status quo
• often self-inflicted and cultivated as a defence of existing interests / competencies / processes
• but sometimes (as in many judgment and decision making situations) simply a reflection of misguided belief that 'scientific rigour' does not have to be incrementally traded-off with 'practical usefulness'.
•http://cafeannalisa.org.uk
References
• http://knol.google.com for my knols(key ‘dowie’ into search box)
Brown (2006) ‘Making decision research useful – not just rewarding’ Judgment and Decision Research 1 (2) 162-173
Brazier, Dixon and Ratcliffe (2009) ‘The role of patient preferences in Cost-Effectiveness Analysis’ Pharmacoeconomics 27 (9) 705-712
Drummond, Schwartz, Jönsson et al (2008) ‘Key principles for the improved conduct of Health Technology Assessments for resource allocation decisions’ International Journal of Technology Assessment in Health Care 24 (3) 244-258
Hazen (2007) ‘Adding Extrinsic Goals to the Quality-Adjusted Life Year Model’ Decision Analysis 4 (1) 3-16