Jonathan H Chen MD, PhD Muthuraman Alagappan MD
Mary K Goldstein MD, MS Steven M Asch MD, MPH Russ B Altman MD, PhD
General Medical Disciplines, Department of Medicine,
Stanford University
ClickWell Care, Hospitalist Group
Undesirable Variability Patients receive only ~55% of recommended care1
E.g., ICU care in last 6 months varies >5x by state2
Practice Uncertainty
RCTrials cost >$15M to answer individual questions
~11% of guideline recs have high quality evidence3
1. McGlynn E, Asch S, Adams J, et al. Quality of health care delivered to adults in the United States. N. Engl. J. Med. 2003 2. http://www.darthmouthatlas.org 3. Tricoci P, Allen JM, Kramer JM, et al. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA 2009
2
Meaningful Use
E.g., Order sets and alerts
Features of Success1
Computer-based
Clinical workflow
Time and location of decision-making
Recommendations not assessments
Open Challenges
Top-down distribution constraints
Mine clinical data from the bottom-up?2
3
1. Kawamoto K, Houlihan C a, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems. BMJ 2005
2. Sittig DF, Wright A, Osheroff J a, et al. Grand challenges in clinical decision support. J. Biomed. Inform. 2008
Recommender Systems
Collaborative Filtering
Automated Decision Support
Distributing crowd sourced info
Predict real clinical practices?
Collaborative Filter
Doctor Collective Patient
Medical Record
Linden G, Smith B, York J. Amazon.com recommendations: item-to-item collaborative filtering. Smith B, ed. IEEE Internet Comput. 2003
4
5 years Inpatient Data (2009-2013)
>74,000 Patients >11,000 Unique Orders >16,000,000 Order Instances
>7,800 Medications >36,000,000 Lab Results
>1,600 Labs >1,100,000 Diagnosis Codes
>1,100 Imaging >2,200,000 Team Assignments
>1,000 Nursing / Procedures
Commerce Customer (Basket) Purchases
Medicine Patient (Encounter) Orders 5
Item Date Category Description
12/25/2010 12:07 ECG ECG 12-LEAD
12/25/2010 12:13 Imaging XR CHEST 1V 12/25/2010 12:13 Lab BLOOD GASES, ARTERIAL
12/25/2010 12:13 Med (Intravenous) Sodium Chloride (Intravenous) 12/25/2010 12:13 Microbiology BLOOD CULTURES
12/25/2010 12:13 PoCare Test POC ISTAT TROPONIN I
12/25/2010 12:13 Lab Result ALBUMIN, SER/PLAS(ALB) (Low)
12/25/2010 12:13 Lab Result WBC(WBC) (High)
12/25/2010 12:13 Lab Result SODIUM, SER/PLAS (Low) 12/25/2010 12:13 Lab Result UREA NITROGEN, SER/PLAS (High)
12/25/2010 12:13 Lab Result NEUT, ABS (High)
12/25/2010 12:13 Lab Result WBC, URINE (Abnormal)
12/25/2010 12:13 Lab Result BACTERIA(UBAC) (Abnormal)
12/25/2010 12:13 Lab Result PROCALCITONIN(PROCTL) (High)
12/25/2010 12:28 PoCare Test POC ISTAT,VENOUS BLOOD GASES AND LACTATE (CG4)
12/25/2010 12:28 Med (Intravenous) Methylprednisolone (Intravenous)
12/25/2010 12:35 Med (Inhalation) Levalbuterol (Inhalation)
12/25/2010 12:54 Med (Intravenous) Levofloxacin (Intravenous)
12/25/2010 12:55 Med (Intravenous) Piperacillin/Tazobactam (Intravenous)
12/25/2010 12:55 Med (Intravenous) Vancomycin (Intravenous)
12/25/2010 13:36 PROBLEM_LIST Pneumonia, organism unspecified
12/25/2010 13:36 ADMIT_DX Shortness of breath
12/25/2010 13:36 Admission ADMIT/PLACE PATIENT
12/25/2010 15:03 ECHO ECHO - TRANSTHORACIC ECHO +DOPPLER
12/25/2010 15:04 Microbiology URINE CULTURE 12/25/2010 20:37 Imaging US VEIN LOWER EXTREMITY BILAT R/O DVT
12/25/2010 20:37 Isolation DROPLET ISOLATION
12/25/2010 20:37 Lab LACTIC ACID
12/25/2010 20:37 Lab RESP DFA PANEL-REFLEX PCR
12/25/2010 20:37 Med (Inhalation) Albuterol/Ipratropium (Inhalation)
12/25/2010 20:37 Med (Intravenous) Pantoprazole (Intravenous)
12/25/2010 20:37 Med (Oral) Acyclovir (Oral) 12/25/2010 20:37 Med (Oral) Gabapentin (Oral)
12/25/2010 20:37 Med (Subcutaneous) Heparin (Subcutaneous)
12/25/2010 20:37 Microbiology FUNGAL CULTURE
12/25/2010 20:37 Microbiology RESPIRATORY CULTURE + Gram Stain
12/25/2010 20:37 Respiratory Care RESP - OXYGEN ADMINISTRATION
12/25/2010 22:29 Med (Intravenous) DEXMEDETOMIDINE IV INFUSION
12/25/2010 22:30 Respiratory Care RESP - NON INVASIVE POS. PRESS VENT (BIPAP/CPAP)
12/25/2010 23:58 Lab HEPARIN ACTIVITY LEVEL
12/25/2010 23:58 Med (Intravenous) Heparin (Intravenous)
B+ B- Total
A+ NABt NA-NABt NA
A- NB-NABt N-NA-NB+NABt N-NA
Total NB N-NB N
6
Episode Mining Apriori Algorithm (Co)-Occurrence Counts
NA = Patients with event A
NABt = Patients with event A followed by event B, within time t
N = Total number of patients
Item Description Prevalence PPV OR P
POC Troponin I 16% 71% 14.4 2E-148
ECG 12-Lead 52% 93% 12.7 8E-80
Nitroglycerin (Sublingual) 1% 9% 11.2 2E-25
Consult Cardiology 5% 28% 9.6 8E-64
D – Dimer 1% 10% 9.4 5E-24
Aspirin (Oral) 24% 68% 7.2 2E-85
CK, MB 16% 51% 6.1 2E-68
Troponin I 24% 62% 5.6 3E-67
Clopidogrel (Oral) 6% 21% 4.7 2E-27
Cardiac Catheterization 3% 10% 4.4 7E-14
Heparin Activity Level 6% 19% 3.8 2E-20
Lipid Panel 9% 24% 3.7 5E-24
NT – proBNP 10% 26% 3.4 6E-23
Nitroglycerin (Topical) 2% 6% 3.2 8E-07
Dobutamine Stress Echo 1% 3% 3.0 6E-04
… … … … …
Top orders following an admission diagnosis of “Chest Pain” Co-occurrences within 24 hours, ranked by Odds Ratio (OR) 7
Item Date Category Description
12/25/2010 12:07 ECG ECG 12-LEAD
12/25/2010 12:13 Imaging XR CHEST 1V 12/25/2010 12:13 Lab BLOOD GASES, ARTERIAL
12/25/2010 12:13 Med (Intravenous) Sodium Chloride (Intravenous) 12/25/2010 12:13 Microbiology BLOOD CULTURES
12/25/2010 12:13 PoCare Test POC ISTAT TROPONIN I
12/25/2010 12:28 PoCare Test POC ISTAT,VENOUS BLOOD GASES AND LACTATE (CG4)
12/25/2010 0:00 Lab Result ALBUMIN, SER/PLAS(ALB) (Low)
12/25/2010 0:00 Lab Result WBC(WBC) (High) 12/25/2010 0:00 Lab Result SODIUM, SER/PLAS (Low)
12/25/2010 0:00 Lab Result UREA NITROGEN, SER/PLAS (High)
12/25/2010 0:00 Lab Result NEUT, ABS (High)
12/25/2010 0:00 Lab Result WBC, URINE (Abnormal)
12/25/2010 0:00 Lab Result BACTERIA(UBAC) (Abnormal)
12/25/2010 0:00 Lab Result PROCALCITONIN(PROCTL) (High)
12/25/2010 12:28 Med (Intravenous) Methylprednisolone (Intravenous)
12/25/2010 12:35 Med (Inhalation) Levalbuterol (Inhalation)
12/25/2010 12:54 Med (Intravenous) Levofloxacin (Intravenous)
12/25/2010 12:55 Med (Intravenous) Piperacillin/Tazobactam (Intravenous)
12/25/2010 12:55 Med (Intravenous) Vancomycin (Intravenous)
12/25/2010 13:36 PROBLEM_LIST Pneumonia, organism unspecified
12/25/2010 13:36 ADMIT_DX Shortness of breath
12/25/2010 13:36 Admission ADMIT/PLACE PATIENT
12/25/2010 15:03 ECHO ECHO - TRANSTHORACIC ECHO +DOPPLER
12/25/2010 15:04 Microbiology URINE CULTURE 12/25/2010 20:37 Imaging US VEIN LOWER EXTREMITY BILAT R/O DVT
12/25/2010 20:37 Isolation DROPLET ISOLATION
12/25/2010 20:37 Lab LACTIC ACID
12/25/2010 20:37 Lab RESP DFA PANEL-REFLEX PCR
12/25/2010 20:37 Med (Inhalation) Albuterol/Ipratropium (Inhalation)
12/25/2010 20:37 Med (Intravenous) Pantoprazole (Intravenous)
12/25/2010 20:37 Med (Oral) Acyclovir (Oral) 12/25/2010 20:37 Med (Oral) Gabapentin (Oral)
12/25/2010 20:37 Med (Subcutaneous) Heparin (Subcutaneous)
12/25/2010 20:37 Microbiology FUNGAL CULTURE
12/25/2010 20:37 Microbiology RESPIRATORY CULTURE + Gram Stain
12/25/2010 20:37 Respiratory Care RESP - OXYGEN ADMINISTRATION
12/25/2010 22:29 Med (Intravenous) DEXMEDETOMIDINE IV INFUSION
12/25/2010 22:30 Respiratory Care RESP - NON INVASIVE POS. PRESS VENT (BIPAP/CPAP)
12/25/2010 23:58 Lab HEPARIN ACTIVITY LEVEL
12/25/2010 23:58 Med (Intravenous) Heparin (Intravenous)
Query
Set
Valid
ation S
et
Separate validation set Identify real order set use Use initial data as query Predict subsequent orders Compare against next
actual 24 hour orders
Recommender
Test
Set
8
Rank Category Description 1 Lab NT - PROBNP
2 Med (Inhalation) Albuterol/Ipratropium (Inhalation)
3 Med (Intravenous) Furosemide (Intravenous)
4 Med (Oral) Aspirin (Oral) 5 Diet DIET NPO 6 Lab CK, MB (MASS)
7 ECHO ECHO - TRANSTHORACIC ECHO +DOPPLER
8 Med (Subcutaneous) Heparin (Subcutaneous)
9 Med (Subcutaneous) Enoxaparin (Subcutaneous)
10 Med (Oral) Prednisone (Oral)
Precision Recall (Sensitivity) Weighted Recall
12.0%
7.8%
8.2%
27.7%
21.3%
15.9%
P-value (Chi2)
Relative Risk (RR)
Odds Ratio (OR)
Pos Pred Value (PPV)
Prevalence
Order Sets
18.9%
19.6%
18.4%
53.2%
42.9%
35.1%
27.3%
28.5%
28.6%
20.9%
9.7%
20.3%
𝑇𝑃𝑖𝑘𝑖=1
𝑇𝑃𝑖𝑘𝑖=1 + 𝐹𝑃𝑖
𝑘𝑖=1
𝑇𝑃𝑖𝑘𝑖=1
𝑇𝑃𝑖𝑘𝑖=1 + 𝐹𝑁𝑖
𝑘𝑖=1
1
𝑁𝑖𝑇𝑃𝑖
𝑘𝑖=1
1
𝑁𝑖𝑇𝑃𝑖
𝑘𝑖=1 +
1
𝑁𝑖𝐹𝑁𝑖
𝑘𝑖=1
9
“…invariably, simple models and a lot of data trump more elaborate models based on less data.”
“…use available large-scale data rather than hoping for annotated data that isn’t available.”
10
2009 Top Items
Overlap at Rank Rank
Fraction Overlap 2012 Top Items
Levofloxacin (IV) 0 1 0.00 Azithromycin (IV)
Blood Culture (2x Aerobic) 1 2 0.50 Levofloxacin (IV)
Blood Culture ((An)Aerobic) 1 3 0.33 Vancomycin (IV)
Respiratory Virus Panel 1 4 0.25 Piperacillin-Tazobactam (IV)
Respiratory Isolation 1 5 0.20 Ceftriaxone (IV)
Oseltamivir (Oral) 1 6 0.17 Azithromycin (Oral)
Vancomycin (IV) 2 7 0.29 Albuterol-Ipratropium (Inh)
Respiratory Culture 2 8 0.25 Sodium Chloride (Inhalation)
Albuterol-Ipratropium (Inh) 4 9 0.44 Blood Culture (2x Aerobic)
CBC w/ Diff 5 10 0.50 Blood Culture ((An)Aerobic)
Influenza A PCR 5 11 0.45 Ipratropium (Inhalation)
… … … … … 11
Predicting 2013 Admission Orders with Different Training Data Samples
41%
46%
51%
56%
Re
call
2013 (12 Months)
2012 Base
2009 (12 Months)
21%
23%
25%
27%
0 10 20 30 40 50
Pre
ciso
in
Training Duration (Months)
2013 (12 Months)
2012 Base
2009 (12 Months)
12
“Prediction is difficult, especially about the future.”
Evolving clinical knowledge and practices
Recent data more useful than simply more data
Predicting 2013 Admission Orders with Decaying Weight to Historical Data
50.5%
51.0%
51.5%
52.0%
52.5%
Re
call
Non-Decaying Model 12 MonthsDecaying Models 48 MonthsNon-Decaying Model 48 Months
26.0%
26.5%
27.0%
0 50 100 150 200
Pre
cisi
on
Decay Window (Months)
Non-Decaying Model 12 MonthsDecaying Models 48 MonthsNon-Decaying Model 48 Months
13
Ni: Item count during i-th interval z: Total number of intervals w: Decay window
𝑁 = 𝑁𝑖 1 −1
𝑤
𝑧−𝑖𝑧
𝑖=1
Decaying estimate for N
“Half-life” ~ 4 months
NIH Big Data 2 Knowledge K01ES026837 (2015-2020) Mentored Career Development Award in Biomedical Big Data Science Russ B. Altman, Mary K. Goldstein, Steven M. Asch Nigam Shah, Lester Mackey, Mike Baiocchi (Vinnie Liu: Writing sample)
Veteran Affairs, Advanced Fellowship in Medical Informatics Doug Owens and Ruth C. Cronkite
Stanford Translational Research and Applied Medicine Grant Dean Felsher and Joanna E. Liliental
Stanford Learning Healthcare Systems Innovation Fund Stanford Clinical and Translational Science Award (CTSA) to Spectrum (UL1
TR001085), NIH National Center for Advancing Translational Sciences (NCATS)
Prof. Russ B Altman and Research Group NIH/National Institute of General Medical Sciences PharmGKB resource:
R24GM61374, LM05652, GM102365
STRIDE Data Group ▪ Tanya Podchiyska, Lee Ann Yasukawa, Gomathi Krishnan, Susan Weber ▪ National Center for Research Resources and the National Center for Advancing
Translational Sciences, National Institutes of Health: UL1 RR025744
Speaker discloses no conflicts of interests Content does not necessarily represent the views of the NIH, VA, or Stanford Health Care
Students Muthu Alagappan Gustavo Chavez
14
Property Death ICU Life Support Discharge / Length of Stay Readmission Evaluation
Period 30 days 2 days 4 days 1 week 2 days 4 days 1 week 2 week 30 days
Outcome Rate
2.2% 1.0% 1.6% 2.3% 22.2% 58.6% 79.2% 90.1% 8.3%
ROC AUC (c-stat) 0.91 0.87 0.83 0.84 0.69 0.75 0.80 0.82 0.68
ROC AUC 95% CI
0.88- 0.93
0.80-0.95
0.76-0.90
0.79- 0.89
0.66-0.72
0.72-0.77
0.78-0.83
0.79-0.85
0.64- 0.73
Related Prediction
Models
APACHE, MPM, SAPS
CURB-65, PSI, SCAP, REA-ICU
Tu et al 1995, ISS, NISS, CSI
Amarasingham et al 2010,
LACE, CMS
ROC AUC for
Related 0.75-0.90 0.69-0.81
0.69 (Only Tu et al reports as ROC AUC)
0.56-0.72
15
Based upon separate validation patients’ first 24 hours of
clinical items, score predicted outcomes by estimated PPV
Frequency Conditional Baseline Ratio Rank Description Confidence Prevalence Lift P
1 Comfort Care Measures 0.11 0.02 5.18 3E-89
2 Lactic Acid (High) 0.46 0.11 4.15 3E-296
3 Calcium Chloride Infusion 0.06 0.01 3.94 9E-32
4 Norepinephrine IV Infusion 0.14 0.04 3.69 4E-71
5 Consult Palliative Care 0.15 0.04 3.63 3E-71
6 Lung Protective Ventilation 0.07 0.02 3.61 7E-35
7 pH Venous (Low) 0.22 0.06 3.50 9E-103
8 Osmolality, Serum (High) 0.06 0.02 3.43 3E-27
9 INR (High Panic) 0.05 0.02 3.30 3E-21
10 Cont Renal Replacement 0.10 0.03 3.24 4E-40
16
Inverted query results for top “recommendations” for items preceding patient death ranked by freqRatio(B|A)week
Chemo (V58.11)
Pneumonia (486)
Joint Pain (719.4)
0%
5%
10%
15%
20%
25%
30%
0 0.2 0.4 0.6 0.8 1
Ave
rag
e W
eig
hte
d R
eca
ll
Rank Biased Overlap (RBO)
R = 0.63
R² = 0.39
P<10-5
Order Prediction
Accuracy vs. Practice
Stability
240 261
178 214
228 532
187 245
458 311
191 283
307 365
196 202
662 442
388 938
351 845
239 267
257
0.33 0.42 0.44 0.44 0.47 0.48 0.48 0.51 0.51 0.53 0.54 0.54 0.54 0.55 0.57 0.57 0.57 0.58 0.59 0.59 0.59 0.60 0.66 0.68 0.68
0 250 500 750 1000
Symptoms involving digestive system (787)Back disorders (724)
Cardiac dysrhythmias (427)Abdominal pain, unspecified site (789.00)
Osteoarthrosis, unspecified (715.9)Abdomen and pelvis symptoms (789)
Osteoarthrosis, lower leg (715.36)Chest pain, unspecified (786.50)
Abdominal pain (789.0)Chest pain (786.5)
Gastrointestinal hemorrhage (578)Fever, unspecified (780.60)
Fever and temperature dysregulation (780.6)Osteoarthrosis, localized (715.3)
Altered mental status (780.97)Other general symptoms (780.9)
Osteoarthrosis (715)Dyspnea (786.0)
Specific procedure complications (996)General symptoms (780)
Shortness of breath (786.05)Respiratory and chest symptoms (786)
Encounter for antineoplastic chemotherapy (V58.11)Encounter for procedures and aftercare (V58)
Encounter for chemotherapy/immunotherapy (V58.1)
[RBO] Number of Admissions in 2012 Admission Diagnosis (ICD9)
Top 25 Admission Diagnoses (2012) with Rank Biased Overlap (RBO) of Related Orders (2009 vs. 2012)
17
Use Rate Size Description
35.1% 51 Anesthesia - Post-Anesthesia Care (Inpatient) 27.2% 161 Medicine - General Admit 23.5% 51 General - Pre-Admission / Pre-Operative
18.4% 17 Insulin - Subcutaneous
15.7% 28 General - Transfusion
9.3% 13 General - Discharge
7.6% 150 Surgery - General Admit 6.9% 9 Emergency - Admit 6.1% 224 Intensive Care - General Admit 5.9% 147 Orthopedics - Total Joint Replacement
… … …
The most commonly used order sets
revolve around non-specific, provider-
focused care processes
18
Metric (First 24 hours of each hospitalization) Mean
Std Dev Median
Interquartile Range
A: Order Sets Used 3.0 (1.4) 3 (2, 4)
B: Orders Entered (including non-Order Set) 32.7 (15.7) 30 (22, 41)
C: Orders Entered from Order Sets 13.3 (7.9) 12 (8, 18)
D: Orders Available from Used Order Sets 129.0 (47.5) 130 (102, 153)
E: Order Set Precision = (C / D) 11% (7.2%) 9.5% (6.3%, 13.8%)
F: Order Set Recall = (C / B) 43% (20%) 42% (28%, 58%)
Metric 2009 2010 2011 2012 2013
Patients 13,493 18,459 19,070 19,327 19,523
Age (mean) 58.4 58.1 58.6 58.5 58.6
Age (std dev) 18.5 18.8 18.7 18.7 18.6
Female 52% 52% 52% 51% 51%
White 60% 63% 62% 60% 58%
Hispanic/Latino 12% 13% 13% 13% 14%
Asian 11% 11% 11% 12% 12%
Black 5% 5% 5% 5% 5%
Providers 1,709 1,892 1,917 1,798 1,821
New Providers … 41% 33% 29% 34%
Departing Providers 34% 32% 33% 33% …
Item Description Prevalence Sodium Chloride (IV) 90%
CBC w/ Diff 79% Metabolic Panel, Basic 77% Potassium Chloride (IV) 68%
Glucose (IV) 60% Docusate (Oral) 58%
Prothrombin Time (PT/INR) 58% Physical Therapy Eval 55%
Metabolic Panel, Comp. 55% Diet NPO 53%
CBC 53% Magnesium, Serum 52%
ECG 12-Lead 52% Occupational Therapy Eval 50% Partial Thromb. Time (PTT) 47%
Xray Chest 1 View 45% … …
20 Top orders by baseline prevalence across all hospitalized patients
Item Description Prevalence PPV OR P ECG 12-LEAD 52% 93% 12.7 8.E-80
CBC w/ Diff 79% 90% 2.5 5.E-10 Metabolic Panel, Comp. 55% 76% 2.6 2.E-19
POC Troponin I 19% 75% 14.4 5.E-147 Prothrombin Time (PT/INR) 58% 70% 1.7 3.E-07
Aspirin (Oral) 24% 68% 7.2 2.E-85 Metabolic Panel, Basic 77% 65% 0.5 8.E-09
Troponin I 24% 62% 5.6 3.E-67 Partial Thromb. Time (PTT) 47% 53% 1.3 1.E-02
Xray Chest 1 View 45% 52% 1.3 5.E-03 CK, MB 16% 51% 6.1 2.E-68
Xray Chest 2 View 29% 49% 2.4 5.E-18 Sodium Chloride (IV) 90% 44% 0.1 7.E-127 Magnesium, Serum 52% 40% 0.6 1.E-07
… … … … …
21
Top orders following an admission diagnosis of “Chest Pain” Co-occurrences within 24 hours, by Positive Predictive Value (PPV)
Cognitive Overload
75 RCTs + 11 systematic reviews published daily1
“The complexity of modern medicine exceeds the
capacity of the unaided expert mind.” – David Eddy
1. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: How will we ever keep up? PLoS Med. 2010
22