1
References References 1. Baltussen R, Niessen L. Priority setting of health interventions: the need for multi- criteria decision analysis. Cost Eff Resour Alloc 2006;4:14. 2. INAHTA - International Network of Agencies for Health Technology Assessment. HTA Resources. 2010. http://www.inahta.org/HTA/ . (Accessed 8 Mar 2010). 3. Velasco GM, Gerhardus A, Rottingen JA, Busse R. Developing Health Technology Assessment to address health care system needs. Health Policy 2010;94(3):196-202. 4. Johri M, Lehoux P. The great escape? Prospects for regulating access to technology through health technology assessment. Int J Technol Assess Health Care 2003;19(1):179-93. 5. Lehoux P, Williams-Jones B. Mapping the integration of social and ethical issues in health technology assessment. Int J Technol Assess Health Care 2007;23(1):9-16. 6. Battista RN. Expanding the scientific basis of health technology assessment: a research agenda for the next decade. Int J Technol Assess Health Care 2006;22(3):275-80. 7. Baltussen R, Stolk E, Chisholm D, Aikins M. Towards a multi-criteria approach for priority setting: an application to Ghana. Health Econ 2006;15(7):689-96. 8. Baltussen R, ten Asbroek AH, Koolman X, Shrestha N, Bhattarai P, Niessen LW. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Health Policy Plan 2007;22(3):178-85. 9. Baltussen R, Youngkong S, Paolucci F, Niessen L. Multi-criteria decision analysis to prioritize health interventions: Capitalizing on first experiences. Health Policy 2010; 10. Nobre FF, Trotta LT, Gomes LF. Multi-criteria decision making--an approach to setting priorities in health care. Stat Med 1999;18(23):3345-54. 11. Peacock S, Mitton C, Bate A, McCoy B, Donaldson C. Overcoming barriers to priority setting using interdisciplinary methods. Health Policy 2009;92(2-3):124-32. 12. Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Evidence and Value: Impact on DEcisionMaking - the EVIDEM framework and potential applications. BMC Health Serv Res 2008;8(1):270. 13. Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C. Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decisionmaking framework to growth hormone for Turner syndrome patients. Cost Eff Resour Alloc 2010;8(1):4. BRIDGING MULTICRITERIA DECISION ANALYSIS (MCDA) WITH HEALTH TECHNOLOGY ASSESSMENT (HTA) FOR POLICY AND CLINICAL DECISIONMAKING: CASE STUDIES IN CANADA AND SOUTH AFRICA Mireille M Goetghebeur PhD, 1,2 Cheri Deal MD PhD, 2,3 Jacqui Miot PhD, 4 Michele Tony MSc, 1* Monika Wagner PhD, 1 Hanane Khoury PhD, 1 Donna Rindress PhD, 1 Tina Papastavros PharmD, 5 Paul Oh MD 6 1 BioMedCom Consultants inc, Dorval, Quebec, Canada; 2 Centre Hospitalier Universitaire Ste-Justine, Montréal, Québec, Canada; 3 University of Montreal, Montréal, Québec, Canada; 4 University of Pretoria, Pretoria, South Africa; 5 Ontario Workplace Safety Insurance Board, Toronto, Ontario, Canada; 6 Toronto Rehabilitation Institute, Toronto, Ontario, Canada Acknowledgments Acknowledgments We wish to acknowledge the contributions of the members of: the Drug Advisory Committee of the WSIB, the Clinical Policy Unit of Discovery Health, and the Expert panel (Mary Edwards, Turner Syndrome Society of Canada; Jack Holland, MD, McMaster University; Philip Jacobs, PhD, University of Alberta; Sheila Kelton RN, British Columbia Children’s Hospital; Farid Mahmud, MD, University of Toronto; Shayne Taback, MD, University of Manitoba; and Guy Van Vliet, MD, Ste Justine University Hospital Center).The expert panel study was funded by an unrestricted research grant for Pfizer Canada. For the WSIB and Discovery Health studies, internal sources of support were provided by the WSIB, Discovery Health and BioMedCom. *Michèle Tony received a MSc grant from BioMedCom for the WSIB Study. Background Background • Healthcare decisionmaking is a complex process that requires consideration of a wide range of scientific and contextual criteria and inherently involves value judgments. 1 • In assisting healthcare decisionmaking, Health Technology Assessment (HTA) is evolving to address, in addition to clinical and economic evidence, social, organizational, ethical and legal dimensions of health technology. 2-6 • Multicriteria decision analysis (MCDA) structures complex decision problems into a comprehensive set of criteria which allows decisionmakers to systematically and explicitly consider multiple dimensions, 1,7-10 while clarifying their fundamental objectives and values. 11 • Bridging HTA with MCDA, the EVIDEM framework includes a comprehensive set of decision criteria with tools and processes: i) to synthesize evidence for each decision criterion (HTA module), and ii) to clarify the perspectives of decisionmakers and allow for systematic and explicit consideration of each decision criterion (MCDA module). 12,13 Methods Methods • Decisionmakers field-tested the framework by selecting case-studies relevant to their settings including: a) tramadol for chronic non-cancer pain for a drug coverage advisory committee of a public Canadian health plan (Ontario Workplace Safety Insurance Board [WSIB]); b) liquid-based cytology for cervical cancer screening for a coverage advisory committee of a private South African health plan (Discovery Health); c) growth hormone for Turner syndrome for Canadian pediatric endocrinologists and other healthcare stakeholders (Expert panel). 13 • For each case study, evidence was systematically extracted and synthesized for each decision criterion of the framework (15 scientific decision criteria - MCDA matrix- and 6 contextual decision criteria - qualitative tool); the HTA report thus produced was posted on a web information system. • To test the framework during workshops, decisionmakers first assigned weights to each criterion of the MCDA matrix, a step designed to quantitatively capture individual perspectives. Then, using synthesized evidence, they appraised the selected intervention by assigning scores to each scientific criterion and by designating a type of impact on the appraisal (negative, neutral, positive) to each contextual decision criterion • MCDA value estimates were obtained using a linear model combining normalized weights and scores • Feedback on the instruments and process was collected from decisionmakers via survey and discussion. Objective Objective • To field-test the EVIDEM framework and explore its utility for both clinical and health policy decisionmaking. • In all three settings, a majority of decisionmakers felt that most decision criteria of the framework should be considered systematically, independently of the intervention under scrutiny. • For scientific criteria, greatest variations across settings were observed for “Size of the population affected by the disease” and “Clinical guidelines. • For contextual criteria, largest discrepancies across settings were observed for “Population priority and access - Fairness”, “Stakeholder pressures” and “Political/historical context”. • Importance of criteria varied to a great extent among individual decisionmakers (SDs) and across settings. • In all three settings, “Improvement in efficacy/effectiveness” and “Relevance and validity of evidence” were among the criteria considered most important. • “Clinical guidelines” was among the least important criterion in all 3 case- studies. 1 Which decision criteria should be considered systematically when appraising healthcare interventions? Case study Expert Panel Canada 2009 Private Health Plan South Africa 2009 Public Health Plan Canada 2010 Number of committee/panel members (N=9) (N=9) (N=10) Scientific criteria (MCDA matrix) Disease impact Disease severity 67% 89% 80% Size of population affected by disease 11% 67% 100% Context of intervention Clinical guidelines 13% 89% 100% Comparative interventions limitations (unmet needs) 89% 78% 100% Outcomes of intervention Improvement of efficacy/effectiveness 89% 89% 100% Improvement of safety & tolerability 100% 78% 100% Improvement of patient reported outcomes 67% 44% 100% Type of benefit Public health interest 56% 78% 70% Type of medical service 44% 89% 100% Economics Budget impact on health plan (cost of intervention) 44% 100% 100% Cost-effectiveness of intervention 67% 89% 100% Impact on other spending (e.g. hospitalization, disability) 44% 78% 70% Quality of evidence Completeness and consistency of reporting evidence (meeting scientific reporting standards and consistency with sources) 89% 89% 100% Relevance and validity of evidence (relevant to decision makers & meeting scientific standards) 100% 89% 100% Contextual criteria (qualitative tool) Ethical principles Goals of healthcare - utility 67% NA 100% Opportunity cost - efficiency 78% 78% 100% Population priority & access - fairness 78% 22% 88% Other criteria System capacity and appropriate use of intervention 63% 89% 100% Stakeholder pressures 11% 33% 67% Political/Historical context 22% 11% 67% 2 Weighting the importance of scientific decision criteria, independent of intervention (MCDA matrix) 3 Appraisal of intervention: Scoring scientific decision criteria (MCDA matrix) • Mean scores were considered to be a good representation of the perceived performance of the intervention. • In all settings, greatest inter-rater agreement on scores (smaller SDs) was observed for “Improvement of efficacy/effectiveness”, “Impact on other spending” and “Completeness and consistency of reporting evidence”. • Greatest inter-rater variability of scores (greater SDs) was observed for “Clinical guidelines”, “Cost-effectiveness” and “Comparative interventions limitations”. 4 Appraisal of intervention: Assigning impacts of contextual decision criteria (Contextual tool) • Information synthesised for each contextual criterion supported decisionmakers in their reflection on ethical and context-specific considerations • The Contextual tool allowed explicit capture of the perspectives of decisionmakers on contextual factors 5 Benefits and challenges of the process • A majority of decisionmakers reported that the framework improved: • Consideration of all elements of decision (all 3 settings) • Transparency of the decision (public and private health plans studies) • Comprehension of the decision among stakeholders (private health plan and expert panel studies) • Challenges were pointed out: • Understanding the language and meaning of certain decision criteria • Perception that distilling evidence to populate the framework is more difficult than current process • Current lack of reference points to compare/rank MCDA value estimates of interventions Conclusions Conclusions • There is growing interest in MCDA to promote more explicit and systematic approaches in clinical and policy decisionmaking • The proposed approach allows to capture variations in perspectives across individuals and settings, highlighted in this study by striking differences on the importance of criteria between a panel of experts with a clinical focus and standing advisory committees of health plans • By structuring HTA on decision criteria rather than on data produced, MCDA based approaches represent a paradigm shift for appraisal of healthcare interventions • Further research and testing of MCDA based frameworks such as EVIDEM is essential to collect feedback from users and tackle challenges in implementing these innovative approaches. Results Results Impact reported by a majority of respondents Expert panel Canada Growth hormone Public health plan Canada Tramadol Ethical principles Goals of healthcare - utility positive positive Opportunity cost - efficiency negative negative Population priority & access - fairness none none Other criteria System capacity and appropriate use of intervention positive positive Stakeholder pressures none none Political/Historical context none negative Table 1: Percentage of decisionmakers reporting that criteria should be considered systematically* *Data obtained from survey questionnaire administered during workshops Figure 1: Mean relative weights (SD) assigned to scientific criteria by decisionmakers Figure 2: Mean scores (SD) assigned to scientific criteria by decisionmakers Table 2: Qualitative impacts assigned to contextual criteria by decisionmakers • Linear combination of normalized weights and scores resulted in committee specific MCDA value estimates of 0.43 (0.36 min – 0.50 max) for liquid based cytology, 0.41 (0.26 min – 0.54 max) for growth hormone and 0.44 (0.36 min – 0.61 max) for tramadol, on a scale of 0 (no value) to 1 (maximum value). 4.1 4.2 4.1 4.2 4.4 3.9 4.2 3.8 4.2 4.7 4.7 4.1 4.7 4.4 1 2 3 4 5 Low H igh 4.0 4.1 3.7 3.1 4.6 4.0 3.6 3.2 3.7 4.0 4.0 3.2 3.9 4.3 1 2 3 4 5 Low High 4.1 3.1 3.1 3.8 4.8 4.4 4.5 4.5 3.9 3.5 4.3 3.8 4.4 4.5 1 2 3 4 5 Low H igh D1 Disease severity D2 Size of population affected by disease C1 Clinical guidelines C2 Comparative interventions limitations I1 Improvement of efficacy/effectiveness I2 Improvement of safety & tolerability I3 Improvement of patient reported outcomes T1 Public health interest T2 Type of medical service E1 Budget impact on health plan E2 Cost-effectiveness of intervention E3 Impact on other spending Q2 Completeness and consistency of reporting evidence Q3 Relevance and validity of evidence Expert Panel Canada Private Health Plan South Africa Public Health Plan Canada Expert Panel Canada Growth hormone Private Health Plan South Africa Liquid based cytology Public Health Plan Canada Tramadol 2.0 1.5 2.1 2.0 2.1 0.6 1.6 0.1 0.6 0.6 0.6 0.9 2.0 1.8 0 1 2 3 Low H igh 1.9 2.6 1.3 1.2 1.0 1.0 0.9 1.1 1.1 1.2 1.2 1.8 1.2 1.1 0 1 2 3 Low H igh 2.2 1.2 2.0 1.1 1.1 1.0 1.3 2.0 2.0 0.7 0.3 1.1 1.0 2.5 0 1 2 3 Low High D1 Disease severity D2 Size of population affected by disease C1 Clinical guidelines C2 Comparative interventions limitations I1 Improvement of efficacy/effectiveness I2 Improvement of safety & tolerability I3 Improvement of patient reported outcomes T1 Public health interest T2 Type of medical service E1 Budget impact on health plan E2 Cost-effectiveness of intervention E3 Impact on other spending Q2 Completeness and consistency of reporting evidence Q3 Relevance and validity of evidence
