60
1 Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved. DEVELOPING PROMs TO HAVE MORE IMPACT
1Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
DEVELOPING PROMs TO HAVE MORE IMPACT
2Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Introduction & Session overview
Topic
Developing PROMs to have more impact
Panelists
1. Konrad Pesudovs, PhD - Foundation Chair of Optometry and Vision Science at Flinders University, Adelaide, Australia
2. Matthias Rose, MD, PhD - Head of the Psychosomatic Medicine Centre at Medical Clinic of the Charité, Universitätsmedizin Berlin, Germany
3. James Willig, MD, MSPH - Associate Professor at the Department of Medicine, Division of Infectious Diseases at University of Alabama at Birmingham, Alabama, USA
4. Helen Burstin, MD, MPH, FACP - Chief Scientific Officer of The National Quality Forum, Washington DC, USA
Agenda▪ Introduction (5 minutes)
▪ Panel presentations (~10-12 minutes each)
▪ Discussion, Q&A (35 minutes)
▪ Summary and closing (2 minutes)
3Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
INTRODUCTION
4Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Psychometrics concerns itself with turning qualitative measures into quantitative data
1, 2, 3, 4, …
“I am in pain”
Intelligence Instrument
Ability to ambulate
5Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Psychometrics concerns itself with turning qualitative measures into quantitative data
1, 2, 3, 4, …
“I am in pain”
Intelligence
Ability to ambulate
Construct, domain, trait, patient-reported
outcome PROM Score
Instrument
6Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Psychometrics concerns itself with turning qualitative measures into quantitative data
1, 2, 3, 4, …
“I am in pain”
Intelligence
Ability to ambulate
Construct, domain, trait, patient-reported
outcome PROM Score
Instrument
PRO-PM - PRO-based Performance Measure
7Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Important to differentiate between ordinal and interval scales, health outcomes are mostly measured on ordinal scales
Data Scale
Ordinal
Interval
Ratio
Attributes can be rank-ordered (sorted), but distances between attributes have
no meaning
Distance between attributes have meaning (averages can be computed),
but ratio’s have no meaning
A meaningful absolute zero exists, a meaningful ratio can therefore be
calculated
Examples
Level of education, IQ, ability to ambulate,
health outcomes, …
Temperature (oC), date
Weight, mass, length, temperature (K)
What we want to have are interval
scales
What we (might) have
is ordinalscales
*Source: Matthew Hankins, Senior lecturer in Public Health at University of Southampton, 2011 presentation
“There is no rational basis for adding up a set of ordinal Likert scores, unless they have been shown to scale”*
8Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Item response theory is the field of work concerning the translation of ordinal health outcomes scores to interval scores
Data Scale
Ordinal
Interval
Ratio
Attributes can be rank-ordered (sorted), but distances between attributes have
no meaning
Distance between attributes have meaning (averages can be computed),
but ratio’s have no meaning
A meaningful absolute zero exists, a meaningful ratio can therefore be
calculated
Examples
Level of education, IQ, ability to ambulate,
health outcomes, …
Temperature (oC), date
Weight, mass, length, temperature (K)
Item Response
Theory
9Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Determining a questionnaire’s psychometric properties is an important step in its development
…which results in the ”anchoring” of questions to a scale
Rasch analysis is one of many possible Item Response Theory models…
Q1Q2Q3Q4Q5Q6Q7Q8Q9
Q10Q11Q12Q13
Q13
Q4
Q3
Q1
Q9
Q5
Q12
Q6
Q7
Q8
Q10
Q11
High ‘difficulty’
Low ‘difficulty’
Q2
10Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
PANEL PRESENTATIONS
11Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Developing PROMs to have more impact
Konrad Pesudovs, PhD
Item Banking: A Generational Change in Patient-Reported Outcome Measurement. Optometry and Vision Science, Vol. 87, No. 4, April 2010
12Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
PROM Measurement
13Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Item Banking
14Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Computer adaptive testing
15Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
The Eye-tem Bank Project
• To develop the most comprehensive and technologically advanced patient-reported outcome measurement system for eye disease- “the Eye-tem Bank”
• To develop, validate, and implement an item banking and CAT system to assess ophthalmic quality of life (QOL) across all eye diseases
Neuro-ophthalmic
Eye-tem Bank modules
Corneal diseases
Refractive error
Strabismus and Amblyopia
Uveitis spectrum of diseases
Ocular inflammation other than uveitis
Ocular-surface and Lacrimal
Diabetic retinopathy
Glaucoma
.Age-related macular degeneration
Retinal detachment
Other vitreo-retinal
Cataract and Corneal opacities
Eye-tem Bank QOL domains
Activity limitation
Mobility
.Ocular comfort symptoms
Visual symptoms
General symptoms
Emotional well-being
Health concerns
Convenience
Social participation and well-being
Economic (work and finance)
16Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
The Eye-tem Bank
17Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Developing PROMs to have more impact
Matthias Rose, MD, PhD
Developing PROMs to have
more impact“Harmonisation”
Matthias Rose
Department of Psychosomatic Medicine
Center for Internal Medicine and Dermatology
Charité Universitätsmedizin Berlin, Germany
Department of Quantiative Health Sciences
University of Massachusetts, USA
ICHOM
London 2016
What is wrong withtoday’s instruments ?
Konrad
Pesudovs
talk!
What is Wrong with Today‘s Measurement?
not comparablesame constructs are often measured with different
instruments without being comparable
restricted measurement rangeclinical instruments do not provide information
for the general population
not precisemeasurement precision does not meet required
standards for individual clinical decision making
CAT
adapted from Ware, Medical Care 1995
Reported health transition l l l
SIP = Sickness Impact Profile (1976)
HIE = Health Insurance Experiment surveys (1979)
NHP = Nottingham Health Profile (1980)
QLI = Quality of Life Index (1981)
COOP = Dartmouth Function Charts (1987)
QLQ-C30 = EORTC
MOS FWBP = MOS Functioning and Well-Being Profile (1992)
MOS SF-36 = MOS 36-Item Short-Form Health Survey (1992)
QWB = Quality of Well-Being Scale (1973)
EUROQOL = European Quality of Life Index (1990)
HUI = Health Utility Index (1996)
SF-6D= SF-36 Utility Index (Brazier, 2002)
Psychometric Utility Related
Concepts and Characteristics SIP HIE NHP QLI COOP QLQ MOS
FWBP
MOS
SF-36
QWB EURO
-QOL
HUI SF-6D
CONCEPTS
Physical functioning l l l l l l l l l l l l
Social functioning l l l l l l l l l l l
Role functioning l l l l l l l l l l l
Psychological distress l l l l l l l l l l l
Health perceptions (general) l l l l l l
Pain (bodily) l l l l l l l l l
Energy/fatigue l l l l l l l
Psychological well-being l l l l
Sleep l l l
Cognitive functioning l l l l
Quality of life l l l
Not Comparable
Financial Impact l
not comparablesame constructs are often measured with different
instruments without being comparable
restricted measurement rangeclinical instruments do not provide information
for the general population
not precisemeasurement precision does not meet required
standards for individual clinical decision making
not anchoredscore values have no intuitive meaning
CAT
What is Wrong with Today‘s Measurement?
Anchoring and Standardization
Rankine
49
2
558
672
0
1859
Reaumur
0
30
80
-218
1731
0
Celsius
37
100
-273
Fahrenhei
t
32
≈100
212
-459
Kelvin
27
3
310
373
0
184817421724
water
freezes
water
boils
body
temperature
absolute
zero
e.g. temperature
Can IRT item banks also
solve those problems ?
Item Bank - Depression
Q Q
Q
Q QQQQ Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Questionaire A
Questionnaire B
11 established
instruments
feeling
depressed feeling
guilty
lost of
interestsuicidal
ideation
feeling
down
happy
relaxed
difficulties
concentrate
Theta Q
highlow30 40 50 60 70 80 90
Representative Samples Patients with Depression
7 Clinical Sites / 12 Health
Centers
total n > 33,000
Wahl et al. J Clin Epi 2014
45
Standardized Metric
BDI
5
20
1961
60
WHO-5
50
30
70
Depression
50
HADS
10
8
3
1983 1998 2013
Cut-Off
Depression
Mean
Representative
Sample55
60
65
70
Theta Q
10
15
25 75
40
20
5
12
Beck Depression
Inventory
Well-Being
Index
Hospital Anxiety
Depression Scale
36
Standardized Metric
1714
Temperature
37
1965 1992 1742
Cut-Off
Fever
Mean
Representative
Sample37.5
38
38.5
39
Celsius Q
39.5
Mercury InfraredQuartz
Comparability
Depression
Upper
limit
HbA1c
8
655
65
75
Depressions CAT
10
Week 1 Week 3 Week 6
%
Week 1 Week 6
HPCL Methode (IFCC)
95%CI
95%CI
Theta Q
Standard tool (HADS)
4
8
12
HADS
9
17
25
BDI
Cut off
Depression
Fliege et al. QLR 2005, Rose et al. Psychosomatic Med 2012, Devine et al. JCE 2015
Fisher et al. 2014
Are patient-reported
health parameters
disease specific ?
Phenotype of Depressivity
Hasselmann et al. Multiple Sclerosis Journal 2016
Health Parameters
g-globulin
creatinine
hemoglobin
fatigue
plasmacytomakidney failure
σύμπτωμα (symptoma) =
it falls together with something
Nephrology Hematology
disease specific measures =
compilation of health parameters
?
but is not unique to it
For which constructs do
harmonized measurements
make sense ?
Suitable Constructs
Fatigue
Pain
Depression
Social role functioning
Physical functioning
Anxiety
Cognitive functioning
Sleep functioning
main health constructs
interval scales, e.g.:
Shortness of breath
Asthma control Questionnaire
composite scores,
preference instruments,
treatment satisfaction, e.g.:
Anti-Clot Treatment Scale
Alzheimer‘s Disease Care
Giver Performance
QuestionnaireAscites Impact Measure
Comprehensive Assessment
of Satisfaction with Care
Diabetes Foot Ulcer Scale
Conclusion
Conclusion
Item banks provide a common metric for existing
tools measuring the same constructII
I Item banks allow to move away from an instrument
to a construct defined measurement system
Item banks make sense for key health indicators,
and will allow to treat subjective health assessments
similar to other clinical markers
III
37Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Developing PROMs to have more impact
James Willig, MD, MSPH
PROMs in Clinical Care Settings
James H. Willig, MD, MSPH
University of Alabama at Birmingham
1917 Clinic Implementation
One person monitoring completion real time
1917 Palliative Care Clinic
Symptoms
Pain
Anxiety
QOL
Depression
PRO Implementation
Selection of instruments
Sequence
Internal logic
Frequency
Alerts
Improved Pain
Notification Triggered • Clinical: SI, IPV (starts clinic response protocol)• Study enrollment
Settings Sessions
Social Services1 2,610
Palliative1 1,208
Signs & Symptoms1 52,249
Viral Hepatitis1 1,213
Supportive Care2 1,376
Breast Health2 106
iEngage3 & BA2C3 471
Pediatric Neurosurgery 65
(1) 1917 Clinic; (2) TKC; (3) Research(2) As of 3/31/2016
47Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
Developing PROMs to have more impact
Helen Burstin, MD, MPH, FACP
Developing PROMs to Have More Impact
Helen Burstin, MD, MPH, FACPChief Scientific Officer, NQF
ICHOM 2016 Annual Conference
May 2016
What is NQF?
49
NQF is an independent, nonprofit, membership organization that brings together all stakeholders
working to improve health and healthcare through quality measurement.
U.S. Policy Environment: From Volume to Value
50
52
Patient Focused Episodes
• Functional Status• Quality of Life• Shared decision-making• Clinical outcomes & PROs• Costs
Population at risk
Acute Phase
Post-Acute/
Rehab
Secondary Prevention
53
Selection and Use of PROs in Measurement
Guiding principles for PRO selection: psychometric soundness, person-centered, meaningful, actionable, implementable
Challenges to use PROs for accountability and performance improvement:
▫ Frequently used in research, but not in clinical use
▫ Aggregation of patient-reported information to measure provider performance challenging
▫ Proprietary tools
The Pathway from PROs to PRO-PMs
54
PRO
• Identify the quality performance issue (include broad input)
• Identify outcomes meaningful to target population and amenable to change
• Determine whether (PRO) is the best way to assess the outcome of interest
PROM
• Identify existing PROMs for measuring the outcome in the target population
• Select PROM suitable for use in performance measurement (e.g., reliable, valid, feasible)
• Use the PROM in real world with the intended target population and setting
PRO-PM
• Specify the outcome performance measure (e.g., average change, percentage improved or meeting a benchmark)
• Test PRO-PM for reliability, validity, and threats to validity (e.g., measure exclusions; missing data; poor response rate; risk adjustment; discrimination of performance; equivalence of results across PROMs)
PRO (concept)
PROM(instrument, tool,
single-item measure)
PRO-PM(PRO-based
performance measure)
Symptom: Depression
The Pathway from PROs to PRO-PMs
55
Percentage of depressed patients with remission or progress toward remission in PHQ-9 scores at 6 months and at 12 months
PHQ-9 © standardized tool to assess depression
NQF Measure Incubator: Getting to Quality Measures that Matter
56
Topic Developer Data Funding
eMeasuresOutcome Measures
Patient-reported Outcome Measures
Cost/Efficiency/ Value Measures
Improved Patient Care and Outcomes
NovelMeasures
58Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
DISCUSSION AND Q&A
59Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
A warm thank you to our panelists and to all of you for your participation!
60Copyright © 2015 by the International Consortium for Health Outcomes Measurement. All rights reserved.
