Reducing Provider Cognitive Workload in CPOE Use: Optimizing Order Sets
Yiye ZhangRema Padman, PhD
James E. Levin*, MD, PhDThe H. John Heinz III College
Carnegie Mellon University, Pittsburgh, PA, [email protected]; [email protected]
MedInfo2013, Copenhagen, Denmark
*Dr. James E. Levin passed away on February 11, 2013. We are greatly indebted to his vision, contributions and support that made this study possible.
Introduction• Significant healthcare delivery challenges in the U.S. and worldwide
– Cost, quality, safety, efficiency, satisfaction– 1999 landmark Institute of Medicine report indicated that 44,000
to 98,000 Americans die each year from medical errors1 – Medication errors are a major component of these errors2
• Potential of healthcare information technology (HIT)– Traditional paper prescription prone to errors due to poor
legibility and miscommunication during patient transfers – Computerized provider order entry (CPOE), a core feature of the
electronic health record (EHR) system, has been recommended to mitigate errors in inpatient orders
1. Institute of Medicine. (1999). To Err is Human: Building a Safer Health System. Retrieved March 28, 2004, from http://www.iom.edu/2. Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients. JAMA 2001;285(16):2114–20.
Computerized Provider Order Entry (CPOE) • CPOE systems are software applications designed to enhance
patient safety by allowing clinicians to enter inpatient orders electronically; Used by one third of US hospitals1
• CPOE systems have been shown to improve patient care through better order legibility, reduced rule violations, improved clinician compliance with best practices, and advanced clinical decision support features2
• Within CPOE, order sets allows clinicians to place multiple, relevant orders for each patient with fewer mouse clicks, thus the creation of order sets is an important prerequisite to successful CPOE implementation and use
1. HIMSS Analytics: Healthcare IT Data, Research, and Analysis. http://www.himssanalytics.org/hc_providers/emr_adoption.asp.2. Potts AL, Barr FE, Gregory DF, Wright L, Patel NR. Computerized physician order entry and medication errors in a pediatric critical care unit. Pediatrics. 2004
Jan;113(1 Pt 1):59-63.
• Collection of individual orders commonly entered as an aggregate for a specific clinical purpose or procedure
• Typically developed by clinical experts in a generic format
• Support clinicians in high risk situations by serving as expert-recommended guidelines, reducing prescribing time by eliminating unnecessary duplication of work, and increasing clinician compliance with the current best practices1
Order Sets
1. Payne TH, Hoey PJ, Nichol P, Lovis C. Preparation and use of pre-constructed orders, order sets, and order menus in a computerized provider order entry system. J Am Med Inform Assoc. 2003 Jul-Aug;10(4):322-9.
Challenges with Order Set Usage • Large Variability in Order Set Usage1 • Difficult to maintain order set content and combinations up-to-date
with current best practices• Lack of involvement in order set development by physicians who
are familiar with both the guidelines as well as the actual practice • Providers switch to ‘a la carte’ orders instead of ordering from
order set, potentially resulting in unsafe and inefficient ordering process
• Poorly designed order sets contribute negatively to treatment quality by exposing users to excessive mouse clicks (physical cost) and cognitive workload (cognitive cost)
1. Zhang Y, Levin JE, Padman R. Data-driven Order Set Generation and Evaluation in the Pediatric Environment. AMIA Symposium Proc. 2012.
Physical and Cognitive Costs• “poor usability--such as poorly designed
screens, hard-to-navigate files, conflicting warning messages, and need for excessive keystrokes or mouse clicks--adversely affects clinical efficiency and data quality” - a recent report from Agency for Healthcare Research and Quality (AHRQ)1
• There is a need to design features of CPOE according to “human factor best practices.” 2,3
1: Schumacher RM, Lowry SZ. NIST Guide to the Process Approach for Improving the Usability of Electronic Health Records. 2010.2. Wright P, Lickorish A, Milroy R. Remembering While Mousing: The Cognitive Costs of Mouse Clicks. SIGCHI Bulletin. 1994.3. Horsky J, Kaufman DR, Oppenheim ML, Patel VL. A framework for analyzing the cognitive complexity of computer-assisted clinical ordering. J Biomed Inform 2003;36(1–2):4–22.
Number of order sets
Physical/cognitive cost
One order set All a la carte
Optimal number of order sets
Research Question• Can the development of order sets be automated using
historical ordering data to learn new order sets that are evidence-based, up-to-date with current best practices, and incur least physical/cognitive costs?
Study Setting• Children’s Hospital of Pittsburgh (CHP) of UPMC, a HIMSS
level 7 pediatric facility• Since October 2002, all inpatient orders at CHP have been
entered directly into the CHP eRecord (Cerner Millenium™)• Over 12,000 pediatric patients admitted each year• Over 10 million order actions in total• On average, a patient at CHP is hospitalized for 5.5 days, and
during that time, 36 unique individuals create 871 order actions
• ~ 2000 departmental, in-house order sets
Sample Appendectomy Orders
PatientID Order Name Order time since admission (hours) Order Set Name Defaults
4092 NPO -1.08Admission Orders
Appendicitis, Complicated
ON
4092 Up Ad Lib -1.08Admission Orders
Appendicitis, Complicated
ON
4092 Vital Signs 4.85 Post Anesthesia Care Orders - Pediatric ON
4092 fentanyl 4.85 Post Anesthesia Care Orders - Pediatric OFF
4092 BUpivacaine 4.87 N/A N/A
A la Carte being utilized
Order set being utilized
Distribution of Orders: Appendectomy MinorSolid blue: a la carte, dotted yellow: order set
Optimization and Clustering ModelsMinimize Cognitive Click Cost (CCC)Subject to1) ‘Default option’ choice constraints2) Cluster formation constraints3) Time interval constraints
Approach: • Order set development from order items
Eliciting Cognitive Costs• CCC with expert estimate (CCCE): Expert input
• CCC based on survey result (CCCS): Survey of 15 subjects including physicians and nurses
• Each survey contains 6 questions with sub-questions, asking subjects to estimate the time it takes them to perform tasks while placing orders with large, mid-size, and small order sets
Approach: Order Set Development• Determine optimal time interval and number of order
sets within the time interval that minimize MCC/CCC
• Cluster orders using bisecting K-means clustering within each time interval
• Map new order set assignment back to historical treatment data to evaluate goodness of clustering using MCC/CCC and coverage rate
Patient Time of order placement Order Order set Default setting CCC
Patient 1 1.4 A O1 ON ?
Patient 1 1.4 B O2 OFF ?
…. … … … … …
Patient 1 10.0 N O3 ON ?
Ex. Fixed patient, time, and order
ON if more than 80% patients use; OFF otherwise
Select De-select
Order Set/ A la Carte 1.2 --
Default ON 0.2 1.5
Default OFF 0.5 0.1
Order Set 1
Order Set 2
Such that CCC can be lowered !Order A
Order B Order COrder E
Order D
Time interval 1, 2,…., n
Results: Significant Reduction in CCC and Increase in Coverage Rate
CCCE per patient(actual mouse clicks)
CCCS per patient(actual mouse clicks)
Current New % change Current New % change
AppendectomyMinor 145.7 111.1
(77)23.4%
** 229.6 116.1(83)
49.4%***
Appendectomy Moderate 208.4 143.3
(110)31.2%
*** 289.7 162.7(103)
43.8%***
***: p-value less than 0.01, **: p-value less than 0.05, *: p-value less than 0.1
Closer Look: Appendectomy Minor - CCCE
Time Interval Training Set Test Set
T5: 0 to 2 174 patients, 153 items 23 patients, 72 items
Number of orders
Number of order
sets
Average Coverage Rate per
OS
CCC per Patient
% Reduction
in CCC per Patient
Average Coverage Rate per
OS
CCC per Patient
% Reduction
in CCC per Patient
Current 68 12 0.34 10.2
25.0%
0.29 19.1
31.9 %
New 66 20 0.75 7.6 0.47 13.0
• 12 order sets used per patient on average in training set• 6 order sets used per patient on average in test set
Sample Case: Under Current Order Set
Item Order Set (size) Default
Admit to Admission Orders General Pediatric Medical Order Set (63) ON
Height Admission Orders General Pediatric Medical Order Set ON
Weight Admission Orders General Pediatric Medical Order Set ON
Notify MD For Oxygen Saturations Admission Orders General Pediatric Medical Order Set OFF
Notify MD For TPR Admission Orders General Pediatric Medical Order Set OFF
Regular (4 yrs & >) Diet Admission Orders General Pediatric Medical Order Set OFF
Up Ad Lib Admission Orders General Pediatric Medical Order Set OFF
Vital Signs Admission Orders General Pediatric Medical Order Set OFF
Subsequent Oxygen Therapy Oxygen Therapy (2) ON
Initial Oxygen Therapy Oxygen Therapy OFF
Subsequent Pulse Oximetry Continuous Pulse Oximetry Continuous (2) ON
Initial Pulse Oximetry Continuous Pulse Oximetry Continuous OFF
CCCE = 20.3, CCCS = 76, number of actual mouse clicks (MC) = 15
Sample Case: Under New Order Sets
Item MCC (size) CCCE (size) CCCS (size) Default
Admit to C1 (3) C1 (2) C1 (5) ON
Height C2 (3) C2 (3) C2 (4) OFF
Weight C2 C2 C2 OFF
Notify MD For Oxygen Saturations a la carte C3 (3) C3 (6) ON
Notify MD For TPR a la carte C2 C2 OFF
Regular (4 yrs & >) Diet a la carte C3 C3 OFF
Up Ad Lib a la carte C4 (2) C3 OFF
Vital Signs C1 C1 C1 ON
Subsequent Oxygen Therapy C3 (3) C5 (2) C1 OFF
Initial Oxygen Therapy C3 C5 C1 OFF
Subsequent Pulse Oximetry Continuous C4 (3) C6 (2) C4 (3) ON
Initial Pulse Oximetry Continuous C4 C6 C4 ON
CCCE = 12.9 (36.4% drop ), CCCS = 23.1 (69.6% drop), MC = 13 (15.4% drop)
• Order Set development based on data-driven approaches is promising
• Can be generalized for not only CHP order sets but also for order sets in other settings with different workflows
Conclusions
Limitations and Challenges
• Large variations in ordering patterns• Influence on usage by the current order sets• Rare combinations of orders need to be
addressed separately in a data-driven approach• Constant CCC weights assumption• Incorporation of new scientific evidence
Future WorkDevelop new approaches and extend/test current methods on
other diagnoses and in other settings1
• Implemented an order set development platform and tested on pneumonia patients
• Incorporate alternate methods using heuristic optimization
Evaluation by physicians on the usability and clinical validity of newly created order sets
• Currently looking for interested institutions to partner on the clinical evaluation studies
1: Zhang Y, Padman R, Levin JE. Data-driven Order Set Development Using Tabu Search. Heinz College Working Paper, Carnegie Mellon University, Pittsburgh, PA, May 2013.
Relevant Publications• Zhang Y, Padman R, Levin JE. Clustering Methods for Data-driven Order Set
Development in the Pediatric Environment. INFORMS 2012 DM-HI Workshop Proc., October 2012
• Zhang Y, Levin JE, Padman R. Data-driven Order Set Generation and Evaluation in the Pediatric Environment. AMIA Symposium Proc., November 2012.
• Zhang Y, Levin JE, Padman R. Toward Order Set Optimization Using Click Cost Criteria in the Pediatric Environment. HICSS-46 Proc., January 2013.
• Zhang Y, Padman R, Levin JE. Data-driven Order Set Development in the
Pediatric Environment: Toward Safer and More Efficient Patient Care. Heinz College Working Paper, Carnegie Mellon University, Pittsburgh, PA, December 2012.
THANK YOU!
QUESTIONS?