A Comprehensive Approach to the Measurement of Health Outcomes
Ron D. Hays, Ph.DUCLA Division of General Internal Medicine &
Health Services Research
K30 Module 2November 16, 2010 (9:00-10:30 am)
1st floor Conference Room 1357, UCLA
Patient-Reported Outcomes Measurement Information System (PROMIS)
• A nine-year $70 million commitment of NIH to improve and standardize measurement of patient-reported outcomes (PROs)– Self-reported health
• An answer to the PRO “Tower of Babel”
The Tower of Babel (Brueghel, 1563)
3
PROMIS-1 Network: 2004-2009
UNC –Chapel Hill ●
● Duke University*
● Stanford
●● University of
Pittsburgh
● University of Washington
Northwestern ♥ ●
NIH
♥Coordinating Center
StoneyBrook
Psycho-metricTesting
Item Bank(IRT-calibrated items)
0.0
0.5
1.0
1.5
2.0
2.5
-3 -2 -1 0 1 2 3
Theta
Info
rmat
ion
0.0
0.2
0.4
0.6
0.8
1.0
-3 -2 -1 0 1 2 3
Theta
Prob
abili
ty o
f R
espo
nse
Short FormInstruments
CAT
Literature Review
Item Pool
Patient Focus Groups
Expert Input and Consensus
Existing Items
Questionnaireadministered to largerepresentative sample
SecondaryData Analysis
CognitiveTestingTranslationExpert
Review
Newly Written
Items
Literature review
Focus groups
Archival data
analysis Expert review/
consensus
Binning and
winnowing
The Life Story of a PROMIS Item
Literacy level
analysis
Expert item revision
Cognitive interviews
Translation review
Large-scale testing
Validation studies
Calibration decisions
Intellectual property
Short formCAT
Statistical analysis
Domain Framework
Physical Functioning Item Bank
Item1
Item2
Item3
Item4
Item5
Item6
Item7
Item8
Item9
Itemn
50
• Are you able to get in and out of bed?• Are you able to stand without losing your balance for 1 minute?• Are you able to walk from one room to another?• Are you able to walk a block on flat ground?• Are you able to run or jog for two miles?• Are you able to run five miles?
Low High
Person Fatigue Score
Interpretation
Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Likely“I get tiredwhen I run
a marathon”
Unlikely“I get tiredwhen I getout of a chair” Item Location
Q Q QQ Q Q Q Q Q
Q Q Q Q Q Q Q Q QQ Q Q
Q Q QQ Q QQ Q Q
Q Q
Q Q
Q Q
Low High
PRO Bank Person Score
Interpretation Aids
30 40 50 60 70
M = 50, SD = 10
T = (z * 10) + 50
Low High
Example of high fatigue
Fatigue Score=60
30 40 50 60 70
This patient’s fatigue score is 60, significantly worse than average (50). People who score 60 on fatigue tend to answer questions as follows:
…”I have been too tired to climb one flight of stairs: VERY MUCH
…”I have had enough energy to go out with my family: A LITTLE BIT
Low High
Example of low fatigue
Fatigue Score=40
30 40 50 60 70
This patient’s fatigue score is 40, significantly better than average (50). People who
score 40 on fatigue tend to answer questions as follows:
…”I have been too tired to climb one flight of stairs: SOMEWHAT…”I have had enough energy to go out with my family: VERY MUCH
Computerized Adaptive Testing (CAT)• Select questions based on a person’s response
to previously administered questions.
• Iteratively estimate a person’s standing on a domain (e.g., fatigue, depressive symptoms)
• Administer most informative items
• Desired level of precision can be obtained using the minimal possible number of questions.
Beginning of CAT
01
23
45
Item 15
T-ScoreIn
form
atio
n
20 30 40 50 60 70 80
|
Max at T-Score=56
20 40 60 80
T-Score
Pos
terio
r Dis
tribu
tion
20 30 40 50 60 70 80
T-Score = 50 SE = 10
Items Administered
T-S
core
0 1 2 3 4 5 6 7 8
20
30
40
50
60
70
80 Items: 15
T-Score: 50 SEM: 10
Best Item-I felt depressed
0.0
0.2
0.4
0.6
0.8
1.0
Item 15
T-Score
Pro
babi
lity
20 30 40 50 60 70 80
12345
20 40 60 80
T-Score
Pos
terio
r Dis
tribu
tion
20 30 40 50 60 70 80
I felt depressed1. Never2. Rarely3. Sometimes4. Often5. Always
T-Score = 52 SE = 4
Items Administered
T-S
core
0 1 2 3 4 5 6 7 8
20
30
40
50
60
70
80 Items: 15
T-Score: 52 SEM: 4 01
23
45
Item 10
T-ScoreIn
form
atio
n
20 30 40 50 60 70 80
|
Max at T-Score=57
Next Best Item-I felt like a failure
0.0
0.2
0.4
0.6
0.8
1.0
Item 10
T-Score
Pro
babi
lity
20 30 40 50 60 70 80
12345
I felt like a failure1. Never2. Rarely3. Sometimes4. Often5. Always
T-Score = 53 SE = 3
Items Administered
T-S
core
0 1 2 3 4 5 6 7 8
20
30
40
50
60
70
80 Items: 15,10
T-Score: 53 SEM: 3 01
23
45
Item 1
T-ScoreIn
form
atio
n
20 30 40 50 60 70 80
|
Max at T-Score=59
Next Best Item-I felt worthless
0.0
0.2
0.4
0.6
0.8
1.0
Item 1
T-Score
Pro
babi
lity
20 30 40 50 60 70 80
12345
20 40 60 80
T-Score
Pos
terio
r Dis
tribu
tion
20 30 40 50 60 70 80
I felt worthless1. Never2. Rarely3. Sometimes4. Often5. Always
T-Score = 55 SE = 2
Items Administered
T-S
core
0 1 2 3 4 5 6 7 8
20
30
40
50
60
70
80 Items: 15,10,1
T-Score: 55 SEM: 2 01
23
45
Item 3
T-ScoreIn
form
atio
n
20 30 40 50 60 70 80
|
Max at T-Score=58
Next Best Item-I felt helpless
0.0
0.2
0.4
0.6
0.8
1.0
Item 3
T-Score
Pro
babi
lity
20 30 40 50 60 70 80
12345
20 40 60 80
T-Score
Pos
terio
r Dis
tribu
tion
20 30 40 50 60 70 80
I felt helpless1. Never2. Rarely3. Sometimes4. Often5. Always
T-Score = 55
SE = 2
Items Administered
T-S
core
0 1 2 3 4 5 6 7 8
20
30
40
50
60
70
80 Items: 15,10,1,3,21,2,5
T-Score: 55 SEM: 2
20 40 60 80
T-Score
Pos
terio
r Dis
tribu
tion
20 30 40 50 60 70 80
CAT assessments can achieve higher precision than fixed forms
Rose et al, J Clin Epidemiol 2007 (accepted)
SE = 3.2rel = 0.90
SE = 2.2rel = 0.95
0
0.1
0.2
0.3
0.4
0.5
0.6
0 10 20 30 40 50 60 70 80
SF-36 items
CAT 10 itemsFull Item Bank
measurement precision (standard error)
normed theta values
HAQ items
SF-12 items
representativesample
rheumatoid arthritis patients
US-Representative Sample
4
5
3
2
1
10 20 30 40 50 60 70
Self-Reported
Health
Social Health
Mental Health
Physical Health
Physical Health
Function
Symptoms
Behavior
Interference
Quality
Intensity
Physical Function
Upper Extremity
GI Symptoms
Pain
Fatigue
Asthma Impact
Sexual Function
Sleep Function
Physical Activity
Mobility
Sleep Disturbance
Sleep-related Impairment
Satisfaction
Interest
LubricationVaginal
DiscomfortErectile FunctionOrgasm
Interfering Factors
Therapeutic Aids
Sexual Activities
Anal Discomfort
Self-Reported
Health
Social Health
Mental Health
Physical Health
Mental Health
Behavior
Cognition
Affect
Anxiety
Depression
AngerExperience of
Stress
Substance Abuse
Applied Cognition –
General Concerns
Self-efficacy
Applied Cognition –
Abilities
Positive
Negative
Illness Impact - Neg
Illness Impact - Pos
Subjective Well-being
Alcohol Use
Self-Reported
Health
Social Health
Mental Health
Physical Health
Social Health
Function
Relationships
Ability to Participate
Family Belongingness
Social Isolation
Quality Social Support
Peer Relationships
Satisfaction w Participation
Social RolesDiscretionary
Activit.
Self-Reported
Health
Social Health
Mental Health
Physical Health
Domains# Items
Adult Bank
# Items Adult ShortForms
# Items Peds Bank
# Items Peds Short
FormEmotional Distress –
Anger29 8 6
Emotional Distress – Anxiety
29 4, 6, 7, 8 15 8
Emotional Distress – Depression
28 4, 6, 8a, 8b 14 8
Fatigue 95 4, 6, 7, 8 23 10
Pain – Behavior 39 7
Pain – Interference 41 4, 6a, 6b, 8 13 8
Physical Function 124 4, 6, 8, 10, 20
-- Mobility 23 8
-- Upper Extremity 29 8
Asthma Impact 17 8
PROMIS Domains in AC, 2010
Domains# Items
Adult Bank
# Items Adult ShortForms
# Items Peds Bank
# Items Peds Short
FormsSatisfaction with
Discretionary Social Activities
12 7
Satisfaction with Social Roles
14 4, 6, 7, 8
Peer Relationships 15 8
Sleep Disturbance 27 4, 6, 8a, 8b
Sleep-Related Impairment
16 8
Global Health 10
PROMIS Domains in AC, 2010
Domains PROMIS-29 PROMIS-43 PROMIS-57Emotional Distress –
Anxiety4 6 8
Emotional Distress – Depression
4 6 8
Fatigue 4 6 8
Pain – Interference 4 6 8
Pain – Intensity 1 1 1
Satisfaction with Social Role
4 6 8
Sleep Disturbance 4 6 8
Physical Function 4 6 8
2010 PROMIS Profile Instruments
PROMIS Profiles
Anxiety
Depression
Fatigue
Pain InterferenceSleep DisturbancePhysical Function
Social Role
46
8
Mental
Physical
Social
Thank You
www.nihpromis.org
Reliability and SEM
• For z-scores (mean = 0 and SD = 1):– Reliability = 1 – SEM2
= 0.91 (when SEM = 0.30)= 0.90 (when SEM = 0.32)
• With 0.90 reliability– 95% Confidence Interval
• z-score: - 0.62 0.62• T-score = (z-score * 10) + 50
• T-score: 44 56
35