I Test Results Summary for 2014 Edition EHR Certification 16-2423-R-0024-PRI Vl.O, July 8, 2016 lNFOIGARD Part 3: NVLAP-Accredited Testing Laboratory Information
Report Number: 16-2423-R-0024 Vl.2
Test Date(s): June 23, 2016
Location of Testing: lnfoGard and Vendor Site
3.1 NVLAP-Accredited Testing Laboratory Information
ATLName:
Accreditation Number:
Address:
Website:
Email:
Phone:
A TL Contact:
lnfoGard Laboratories, Inc.
NVLAP Lab Code 100432-0
709 Fiero Lane Suite 25
San Luis Obispo, CA 93401
www.infogard.com
ehr@infogard .com
(805) 783-0810
Milton Padilla
For more information on scope of accreditation, please reference http://ts.nist.gov/standards/scopes/1004320.htm
Part 3 of this test results summary is approved for public release by the following Accred ited Test ing Laboratory Representative:
EHR Test Body Manager Function/Title
Ii 12
3.2 Test Information
3.2.1 Additional Software Relied Upon for Certification
Additional Software Applicable Criteria Functionality provided by Additional Software
Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter]
No alteration (customization) to the test data was necessary
3.2.4 Standards
3.2.4.1 Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted
Criterion # Standard Successfully Tested
(a)(8)(ii)(A)(2)
§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain
(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(e)(1) View, download and transmit data to a third party utilizing the Edge Protocol IG version 1.1
(f)(3) Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)
(f)(7) Ambulatory setting only – transmission to public health agencies – syndromic surveillance - Create Data Elements
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)
No optional functionality tested
Test Results Summary for 2014 Edition EHR Certification 16-2423-R-0024-PRI V1.0, July 8, 2016
*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD)
*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9). **Required for every EHR product
Test Results Summary for 2014 Edition EHR Certification 16-2423-R-0024-PRI V1.0, July 8, 2016
10275 West Higgins Road, Suite 250 Rosemont, IL 60018 Tel: 844 240 9554 picis.com
June 23, 2016 Milton Padilla InfoGard, a UL Company 709 Fiero Ln, Suite 25 San Luis Obispo, CA 93401 Dear Mr. Padilla, Per 170.314(g)(3) criteria, this letter serves as attestation of our User-Centered Design practices. Picis PulseCheck follows Solution Architectural Design documents (specifically sections 4.0 – 4.4) for the development of each version, including ED PulseCheck version 5.6. The Solution Architectural Design documents are part of the Picis overall Quality Management System which is designed to meet the requirements of the international standard ISO 13485 Medical Devices – Quality Management Systems – Requirements for Regulatory Purposes Standards as well as the FDA Quality System Regulation 21 CFR Part 820. Additionally, Picis maps our procedures to IEC 62366-1 Application of Usability Engineering to Medical Devices. The procedures were followed for planning, development, testing, and release of each application version and capability for which Picis sought certification, including:
170.314(a)(1) Computerized provider order entry 170.314(a)(2) Drug-drug, drug-allergy interaction checks 170.314(a)(6) Medication list 170.314(a)(7) Medication allergy list 170.314(b)(3) Electronic prescribing
Please reference the attached Solution Architectural Design documents for further details and please do not hesitate to contact me with any outstanding questions. Thank you,
Merrily King Turnbull, MBA Director, Product Management Emergency Department Solutions
Page 1 of 56
EHR Usability Test Report of ED PulseCheck Version 5.4 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
Optum ED PulseCheck Version 5.4
Date of Usability Test 12/03/2013 – 12/12/2013 Date of Report 01/27/2014 Report Prepared By John Romadka, Senior User Experience Researcher, Optum
PERFORMANCE DATA ....................................................................................................................................................... 5 Physicians .................................................................................................................................................................. 6
Major Findings ......................................................................................................................................................................... 7 Areas for Improvement ............................................................................................................................................................ 7
RNs ............................................................................................................................................................................ 8 Major Findings ......................................................................................................................................................................... 8 Areas for Improvement ............................................................................................................................................................ 9
PROCEDURES ................................................................................................................................................................ 14 TEST LOCATION ............................................................................................................................................................. 14 TEST ENVIRONMENT ...................................................................................................................................................... 14 TEST FORMS AND TOOLS ............................................................................................................................................... 15 PARTICIPANT INSTRUCTIONS .......................................................................................................................................... 15 USABILITY METRICS ....................................................................................................................................................... 16
DATA SCORING ............................................................................................................................................................. 17
DATA ANALYSIS AND REPORTING .................................................................................................................................... 18 Physicians ................................................................................................................................................................ 19
Discussion of the Findings ..................................................................................................................................................... 20 Major Findings ....................................................................................................................................................................... 23
RNs .......................................................................................................................................................................... 25 Discussion of the Findings ..................................................................................................................................................... 25 Major Findings ....................................................................................................................................................................... 27
Combined (Physician and RN) Areas for Improvement .......................................................................................... 28
APPENDIX 2: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ................................................................. 32 APPENDIX 3: EXAMPLE MODERATOR’S GUIDE FOR PHYSICIANS AND RNS ........................................................................ 33 APPENDIX 4: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................................................................ 34 APPENDIX 5: TASKS ....................................................................................................................................................... 35
Physicians ................................................................................................................................................................ 35 Task 0 – Login ....................................................................................................................................................................... 35 Task 1 - Enter a laboratory, radiology, and imaging orders ................................................................................................... 36 Task 2 - Enter medication orders .......................................................................................................................................... 37
Page 3 of 56
Task 3 – Cancel or hold a medication order .......................................................................................................................... 38 Task 4 – Delete or cancel a medication order ....................................................................................................................... 39 Task 5 – Update a medication order...................................................................................................................................... 40 Task 6 –Update a radiology order ......................................................................................................................................... 41 Task 7 – Approve a medication order with a drug-allergy reaction ........................................................................................ 42 Task 8 – Approve a medication order with a drug-drug interaction ....................................................................................... 43 Task 17 - Configure pharmacy for e-prescription .................................................................................................................. 44 Task 18 - Change pharmacy for e-prescription ..................................................................................................................... 45
RNs .......................................................................................................................................................................... 46 Task 0 – Login ....................................................................................................................................................................... 46 Task 10 - Enter active medications ....................................................................................................................................... 47 Task 11 - Update a medication currently being taken............................................................................................................ 48 Task 12 - Inactivate an active medication .............................................................................................................................. 49 Task 13 – Find active and inactive allergies .......................................................................................................................... 50 Task 14 - Enter active allergies ............................................................................................................................................. 51 Task 15 - Update the effects of an allergy ............................................................................................................................. 52 Task 16 - Inactivate an allergy ............................................................................................................................................... 53
Executive Summary A summative usability test of ED PulseCheck version 5.4 was conducted 12/03/2013 – 12/12/2013 by the Optum User Experience Center of Excellence. The test was conducted remotely using a web-based testing tool, UserZoom, and WebEx. The purpose of this test was to provide evidence of usability in the EHR Under Test (EHRUT) and identify areas for incremental usability improvement.
Ten physicians and 13 RNs matching the target demographic criteria participated in the usability test, using the EHRUT in simulated but representative tasks.
During the 60-minute one-on-one usability sessions, each participant was greeted by the administrator and asked to review and sign a non-disclosure/consent form (see Appendix 2). Participants were instructed that they could withdraw at any time. All participants had prior experience with the EHRUT.
The administrator introduced the test and instructed participants to complete a series of tasks (presented randomly one at a time) using the EHRUT. During the testing, the testing tool (UserZoom) recorded user performance data electronically. A Data logger also recorded user performance. The administrator did not give the participant assistance in how to complete the task (and if it became necessary, that task was marked as failed).
7 Approve a medication order with a drug-allergy reaction
8 Approve a medication order with a drug-drug interaction
§170.314(b)(3) Electronic prescribing
17 Configure pharmacy for e-prescription
18 Change pharmacy for e-prescription
Page 5 of 56
RNs performed the following tasks:
Area of Certification Tasks
§170.314(a)(6) Medication list
10 Enter active medications
11 Update a medication currently being taken
12 Inactivate an active medication
§170.314(a)(7) Medication allergy list
13 Find a list of active and inactive allergies
14 Enter active allergies
15 Update the effects of an allergy
16 Inactivate an allergy
Each participant’s interactions with the EHRUT were recorded for subsequent analysis. The following types of data were collected:
• Number of tasks successfully completed within the allotted time without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant’s verbalizations
• Participant’s satisfaction ratings of the tasks and system
All participant data was de-identified to avoid linking the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire. Physicians and RNs were compensated $250 and $150 respectively for their time.
Performance Data Various recommended metrics, in accordance with the examples set forth in the NISTIR Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT.
Page 6 of 56
Physicians
Measure N Task
Success Path
Deviation Task Time Errors Task
Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
1. Enter laboratory and radiology/imaging orders 10 70%
(±48%) 1.7 99 (±54) 179% 40%
(±70%) 3.9 (±74%)
2. Enter medication orders 10 50% (±53%) 2.1 180
(±50) 279% 70% (±95%)
3.8 (±63%)
3. Cancel or hold a medication order 10 90%
(±32%) 0.9 44 (±22) 98% 10%
(±32%) 4.6 (±84%)
4. Delete or cancel a medication order 10 80%
(±42%) 1.5 46 (±19) 133% 20%
(±42%) 4.7 (±67%)
5. Update a medication order 10 90%
(±32%) 1.4 98 (±40) 121% 10%
(±32%) 4.3 (±67%)
6. Update a radiology order 10 90% (±32%) 1.8 78
(±29) 144% 10% (±32%)
4.0 (±105%)
7. Approve a medication order with a drug-allergy reaction
10 80% (±42%) 3.0 75
(±30) 123% 20% (±42%)
4.9 (±32%)
8. Approve a medication order with a drug-drug interaction
10 80% (±42%) 2.3 76
(±21) 148% 20% (±42%)
4.5 (±53%)
17. Configure pharmacy for e-prescription 10 50%
(±53%) 2.4 167 (±56) 207% 50%
(±53%) 3.4 (±97%)
18. Change pharmacy for e-prescription 10 50%
(±53%) 1.9 184 (±142) 196% 80%
(±103%) 3.7 (±82%)
The System Usability Scale was used to score participants’ subjective satisfaction based on their overall experience with the EHRUT. The overall score was 86 (±10).1
In addition to the performance data, we recorded the following qualitative observations.
1 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.
Page 7 of 56
Major Findings Major findings include:
• Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness
• Participants indicated an above-average level of product satisfaction
• The EHRUT has multiple navigation paths that participants can use to complete tasks; participants used a variety of the methods available (in some cases, this resulted in slower task times)
• Significant efficiencies when participants used Quicklists and filter/find as you type (FAYT
As well as issues with:
• E-prescription search
• Data loss when users entered free text
• Incomplete record entries/updates (e.g., without schedules)
• Multiple action buttons too close to each other
• Dropdown lists (number of items and inconsistencies in position)
Areas for Improvement Recommendations for improvement include the following:
• Improve flexibility in search entries and results presentation for e-Prescription searches
• Alert the user when necessary data (e.g., medication schedule) are skipped to avoid incomplete entries/updates
• Help users find the correct action (e.g., visually distinguishing the button required to complete the action)
• Recognize free text data entries (e.g., medical abbreviations) to avoid data loss
• Consistent dropdown list display (location and length)
Page 8 of 56
RNs
Measure N Task
Success Path
Deviation Task Time Errors Task
Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
10. Enter active medications 13 60% (±52%) 2.3 172
(±29) 235% 80% (±104%)
4.6 (±70%)
11. Update a medication currently being taken 13 69%
(±48%) 1.8 120 (±45) 187% 54%
(±88%) 3.9
(±126%)
12. Inactivate an active medication 13 85%
(±38%) 2.0 114 (±30) 165% 15%
(±38%) 4.4
(±96%)
13. Find a list of active and inactive allergies 13 46%
(±52%) 1.4 59 (±25) 198% 62%
(±51%) 4.3
(±125%)
14. Enter active allergies 13 92% (±28%) 1.4 113
(±33) 145% 31% (±85%)
4.5 (±78%)
15. Update the effects of an allergy 13 77%
(±44%) 1.4 82 (±57) 133% 31%
(±48%) 4.4
(±119%)
16. Inactivate an allergy 13 69% (±48%) 1.2 98
(±32) 128% 23% (±44%)
4.7 (±48%)
The System Usability Scale was used to score participants’ subjective satisfaction with the system based on their overall experience with the EHRUT. The overall score was 87(±17).2
In addition to the performance data, we recorded the following qualitative observations.
Major Findings Major findings include:
• Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness
• Participants’ indicated an above-average level of product satisfaction
• The EHRUT has multiple navigation paths that participants can use to complete tasks; participants used a variety of the methods available (in some cases, this resulted in slower task times)
• Significant efficiencies when participants used Quicklists and filter/find as you type (FAYT)
As well as issues with:
• Difficulty finding “Show inactive” allergies
• Multiple action buttons too close to each other
2 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.
Page 9 of 56
• Difficulty finding how to update the effects of an allergy
• Consistent dropdown list display (location and length)
Areas for Improvement Recommendations for improvement include the following:
• Make “Show inactive” more visually identifiable
• Help users find the correct action buttons
• Don’t rely on a hyperlinked line with no label or text for navigation
• Consistent dropdown list (location and length)
Introduction The EHRUT tested for this study was ED PulseCheck version 5.4. Designed to present medical information to healthcare providers in an emergency department, the EHRUT consists of a tracking board, patient charts, medication ordering system, lab ordering system, and e-prescription system. The usability testing represented realistic tasks.
The purpose of this test was to provide evidence of usability in the EHRUT and identify areas for incremental usability improvement. To this end, measures of effectiveness, efficiency, and user satisfaction (such as task completion, time on task, and task efficiency) were captured during the usability testing.
Method
Participants A total of 23 participants, consisting of 10 physicians and 13 RNs, were tested on the EHRUT. All participants were given the same orientation instructions. Physicians and RNs were compensated $250 and $150 respectively for their time.
Participants were recruited by Leede Research. Participants had no direct connection to the development of or organization producing the EHRUT. Participants were not from the testing or supplier organization. All participants currently use the EHRUT in a professional workplace and were recruited based on job title, gender, and age.
The following table summarizes the characteristics of the physicians who participated in the usability test, including demographics, professional experience, computing experience and use of assistive technology. Participant names were replaced with Participant IDs so that data cannot be tied back to identities.
Page 10 of 56
Part ID
Part ID Gender Age Education Occupation Professional
Experience Computer
Experience Product
Experience Assistive
Technologies
1 P4 Male 30-39 Professional degree (e.g. MD)
Physician 7-9 years 20-29 years
1-3 years No
2 P11 Male 50-59 Professional degree (e.g. MD)
Physician 20-29 years 7-9 years 4-6 years Yes, Dragon
3 P13 Male 30-39 Professional degree (e.g. MD)
Physician 7-9 years 10-19 years
7-9 years No
4 P18 Male 50-59 Professional degree (e.g. MD)
Physician 10-19 years 20-29 years
4-6 years No
5 P25 Female 50-59 Professional degree (e.g. MD)
Physician 20-29 years 7-9 years 4-6 years No
6 P26 Male 60-69 Professional degree (e.g. MD)
Physician 30-39 years 20-29 years
1-3 years No
7 P14 Male 60-69 Professional degree (e.g. MD)
Physician 20-29 years 7-9 years Less than 1 year
No
8 P29 Male 30-39 Professional degree (e.g. MD)
Physician 7-9 years 10-19 years
4-6 years No
9 P28 Male 30-39 Professional degree (e.g. MD)
Physician 7-9 years 4-6 years 4-6 years Yes, Dragon
10 P23 Male 30-39 Professional degree (e.g. MD)
Physician 4-6 years 7-9 years 4-6 years No
Fourteen physicians were originally recruited, including two pilots.
• Three sessions were eliminated from results because of an inconsistency between the script and the test environment in task 2
• One participant was found not to meet the requirements • There were two no-shows
Two additional physicians were recruited to provide test results for 10 physicians.
Page 11 of 56
Participants were scheduled for 60-minute sessions. A minimum of 60 minutes was reserved between each session for debrief by the administrator and data logger and to reset systems to proper test conditions. A spreadsheet was used to track the participant schedule.
RNs
The following table summarizes the characteristics of the RNs who participated in the usability test, including demographics, professional experience, computing experience, and use of assistive technology. Participant names were replaced with Participant IDs so that data cannot be tied back to identities.
Part ID
Part ID Gender Age Education Occupation Professional
Experience Computer
Experience Product
Experience Assistive
Technologies
1 P2 Female 30-39 Associate’s degree
Registered Nurse
10-19 years
10-19 years 7-9 years No
2 P3 Female 40-49 Doctoral degree
Nurse Practitioner
10-19 years
10-19 years 1-3 years No
3 P5 Male 50-59 Bachelor’s degree
Registered Nurse
7-9 years 10-19 years 7-9 years No
4 P6 Male 40-49 Bachelor’s degree
Registered Nurse
10-19 years
10-19 years 1-3 years No
5 P8 Female 30-39 Bachelor’s degree
Registered Nurse
1-3 years 7-9 years 1-3 years No
6 P9 Female 30-39 Associate’s degree
Registered Nurse
4-6 years 1-3 years Less than 1 year
Yes, Dragon
7 P12 Female 40-49 Bachelor’s degree
Registered Nurse
20-29 years
20-29 years 7-9 years No
8 P19 Male 60-69 Associate’s degree
Registered Nurse
20-29 years
4-6 years 4-6 years No
9 P20 Female 40-49 Bachelor’s degree
Registered Nurse
10-19 years
10-19 years 1-3 years No
10 P21 Male 40-49 Bachelor’s degree
Registered Nurse
4-6 years 10-19 years 4-6 years Yes, Dragon
11 P22 Female 40-49 Bachelor’s degree
Registered Nurse
20-29 years
20-29 years 4-6 years No
12 P15 Female 18-29 Bachelor’s degree
Registered Nurse
1-3 years 4-6 years 4-6 years No
13 P27 Male 40-49 Bachelor’s degree
Registered Nurse
10-19 years
10-19 years 4-6 years No
Page 12 of 56
A total of 14 RNs were recruited for the test. One participant failed to show for the study. Test results for 13 RNs are included in this report.
Participants were scheduled for 60-minute sessions. A minimum of 60 minutes was reserved between each session for debrief by the administrator and data logger and to reset systems to proper test conditions. A spreadsheet was used to track the participant schedule.
Study Design
Overall, the objective of this test was to measure how well the application performed with regard to effectiveness, efficiency, and user satisfaction; to identify areas for improvement; and to recommend improvements. Additionally, the data from this test will serve as a baseline for future tests for comparison with updated versions of the EHR and/or other EHRs. In short, this testing serves as a means to benchmark current usability and to identify areas where improvements can be made.
All participants were given the same orientation instructions and accessed the EHRUT using the same remote testing environment. Two task sets were defined—one for physicians and one for RNs. All participants used the same test data. The EHRUT was evaluated using the following measures:
• Number of tasks successfully completed within the allotted time without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant’s verbalizations (comments)
• Participant’s satisfaction ratings of the tasks and system
Additional information about the various measures can be found in the Usability Metrics section of this document.
Tasks Tasks were selected based on how frequently they are used, how critical they are, and how likely they are to be troublesome for users. Task details were defined to be realistic and representative of the kinds of activities users might perform with the EHRUT.
7 Approve a medication order with a drug-allergy reaction
8 Approve a medication order with a drug-drug interaction
§170.314(b)(3) Electronic prescribing
17 Configure pharmacy for e-prescription
18 Change pharmacy for e-prescription
RNs
RNs performed the following tasks:
Area of Certification Tasks
§170.314(a)(6) Medication list
10 Enter active medications
11 Update a medication currently being taken
12 Inactivate an active medication
§170.314(a)(7) Medication allergy list
13 Find a list of active and inactive allergies
14 Enter active allergies
15 Update the effects of an allergy
16 Inactivate an allergy
Page 14 of 56
Procedures Upon arrival, participants were greeted and their identity was verified and matched with the participant schedule. Participants were then assigned a participant ID.3 Each participant digitally signed a non-disclosure and informed consent form. See Appendix 2: Non-Disclosure Agreement and Informed Consent Form for an example of the form.
The test was conducted by two staff members, the administrator, and the data logger. The individuals conducting the test were usability practitioners with over 15 years of experience and educational backgrounds in Human Factors.
The administrator moderated the session, delivered initial instructions, and took notes on participant comments, path deviations, and errors. The data logger also took notes on task success, path deviations, errors, and comments. The testing tool (UserZoom) led participants through the tasks. UserZoom recorded task times, post-task rating data, post-test satisfaction data, and demographic data.
Participants were instructed to perform the tasks (see the Participant Instructions below):
• As quickly as possible making as few errors and deviations as possible
• Without assistance
• Without commenting until they had completed each task (if possible)
UserZoom provided participants with instructions before and during each task. Tasks were presented to each participant in a randomized order to avoid learning bias. Task timing began once the participant clicked the “Start Task” button in UserZoom. Task timing ended once the participant clicked the “Completed Task” button in UserZoom. Scoring is discussed in the Data Scoring section.
Following the session, UserZoom displayed the post-test survey, including a System Usability Scale (SUS) questionnaire and demographic questions. See Appendix 4: System Usability Scale Questionnaire for additional information on SUS questionnaires. The administrator told participants to expect to receive compensation in 2-3 weeks.
Test Location Tests were conducted remotely, with each individual at their own location. Only the participant, administrator, and data logger were present.
Test Environment The EHRUT typically is used in an emergency department. Tests were conducted remotely, using the administrator’s computer to run WebEx, Windows Remote Desktop Connection (RDC), and TechSmith Camtasia (for recording). RDC was used to connect remotely to the EHRUT test environment via a Virtual Private Network (VPN) using a broadband cable connection with a minimum throughput of 10Mbps download and 3Mbps upload. Participants used their own PCs during studies. They were given control of the administrator’s computer to access the application during the session, but used their mouse, keyboard, and display. (Note, many participants indicated they were not using the computer setup they normally use with the EHRUT.) Participants PC information was not collected.
3 All participant data has been de-identified and will be kept confidential.
Page 15 of 56
Additionally, participants were told to maximize the WebEx sharing window to full screen and not to change any default system settings (such as control of font size). They also were told to close any applications other than the WebEx session.
The administrator’s computer is an HP EliteBook 8570w laptop with a Core i7 Processor, 180GB SSD hard drive, 16GB RAM, running the Windows 7 Basic Theme. The display resolution was set to 1440x900. This computer ran WebEx screen sharing, the VPN broadband connection, the RDC to the EHR test environment, and Camtasia.
The EHR test environment on the RDC used Internet Explorer 8.0 (zoom level of 125%). The RDC displayed at a resolution of 1920x1080 in full screen mode with 24-bit True Color and was set for high-speed broadband (2-10Mbps) with persistent bitmap caching.
Test Forms and Tools During the usability test, UserZoom presented various documents, including:
1. Informed Consent
Participants were emailed the Informed Consent prior to session so they could review it. At the beginning of session, they were asked to indicate in UserZoom if they agreed or disagreed with the terms.
2. Test instructions
3. Post-task Questionnaire
4. Post-test Questionnaire
5. Participant Demographics Questionnaire
Examples of these documents can be found in Appendix 2: Non-Disclosure Agreement and Informed Consent Form, Appendix 3: Example Moderator’s Guide, and Appendix 4: System Usability Scale Questionnaire. The Moderator’s Guide was designed to enable the data logger to easily capture data.
The script for participant orientation follows:
Participant Instructions Thank you for joining us today; we really appreciate your time. I’m John Romadka; I work at
Optum, makers of ED PulseCheck. Jamie Peffley (or Scott Stewart) is also on the on line, and they’ll be helping take notes.
Before we begin, I’d like to take a moment to describe the study we’re conducting.
We’re collecting feedback on a software product called ED PulseCheck.
I know someone already asked you this, but I just wanted to verify that you currently use ED PulseCheck to work with patient records.
Has not used ED PulseCheck Has used ED PulseCheck
We’re going to ask you to perform a series of tasks. You should know that the tasks were created to be independent of each other and some may sound redundant. You will be asked to complete these tasks on your own, trying to do them as quickly as possible with the fewest possible errors or
Page 16 of 56
deviations. Please do not do anything more than the task asks.
You are not being tested on your ability to complete the tasks. We’re evaluating how intuitive the product is to use, so just do your best to complete them as you typically would. We didn’t have anything to do with designing the product, and your comments are entirely anonymous, so please feel free speak candidly. However, please save your comments until the end of a task, or the end of the session as a whole, when we can then discuss them freely.
We’re going to be using a study tool that will lead you through the tasks. We’d ask that you please read the task information out loud, so we can follow along.
I’m going to share my screen with you in the WebEx. You’ll be using Ed PulseCheck on my computer, but I’ll give you control of my computer’s mouse and keyboard for the duration of the study. Because we can’t always control the bandwidth between our two locations, you may notice a tiny lag or latency when you’re in control of the mouse and keyboard. However, we don’t anticipate this will affect your ability to use ED PulseCheck.
Before I do that, please quit or close any other applications that are currently running on your computer (email and Instant Messaging), with the exception of your browser and WebEx. Also, please set any mobile devices to silent during the study.
Participants were then presented with tasks one at a time, randomly assigned by UserZoom. See Tasks for a complete list of tasks.
Usability Metrics According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency, and user satisfaction were captured during the usability testing.
The goals of the test were to assess:
1. Effectiveness of the EHRUT (ED PulseCheck) by measuring participant success rates and errors
2. Efficiency of the EHRUT by measuring the average task time and path deviations
3. Satisfaction with the EHRUT by measuring ease-of-use ratings
Page 17 of 56
Data Scoring The following table details how the testing team scored the tasks, evaluated the errors, and analyzed the time data.4
Measures Rationale and Scoring
Effectiveness:
Task Success
A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted.
The total number of successes were calculated for each task, and then the results were divided by the total number of times that task was attempted. The results are provided as a percentage.
The total Task times were recorded for successes. Observed task times were divided by the optimal performance time for each task to provide a measure of optimal efficiency. The results are provided as a percentage mean.
Optimal task performance time was benchmarked when constructing tasks by using Keystroke Level Modeling (KLM) to measure the clicks, keystrokes, and task time to obtain an expected time based on the optimal path. (KLM was used rather than expert time because the product includes many paths to accomplish the tasks, and experts frequently didn’t follow optimal paths.) Additional time, based on test environment measurements, was added to the KLM results to account for the following:
1. User’s need to review information while interacting with the user interface (medical professionals double-check their work routinely to avoid error)
2. System response times
See Appendix 7: Data Scoring Detail for additional information regarding how the KLM was calculated.
Effectiveness:
Task Failures
If the participant abandoned the task, did not reach the correct answer, or performed it incorrectly, the task was counted as a “Failure.”
The total number of errors was calculated for each task, and the results were divided by the total number of times the task was attempted. This was also expressed as the mean number of failed tasks per participant.
Efficiency:
Task Deviations
The participant’s path (i.e., steps) through the application was recorded. Deviations occurred if the participant went to a wrong screen, clicked an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. Each participant’s path was compared to the optimal path. The number of steps in the observed path was divided by the number of optimal steps to provide a ratio of path deviation.
4 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman. Also see www.measuringusability.com.
• Task Recognition: failure to recognize task or completion of a different task
• Data Entry: failure to enter complete data or entry of incorrect data
Efficiency:
Task Time
Each task was timed from when the participant clicked the “Start Task” button until the participant clicked the “Task Complete” button. Only completion times for successfully completed tasks were included in the average task time analysis. Average times per task and variance measures (standard deviation and standard error) were calculated for each task.
Satisfaction:
Task Rating
Participant’s subjective impression of the ease-of-use of the application was measured by administering a post-task question and a post-session questionnaire.
After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants.5 Common convention is that average ratings for systems judged easy to use should be 3.3 or above.
To measure participants’ confidence in and satisfaction with the EHRUT overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system frequently”, “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.” See Appendix 4: System Usability Scale Questionnaire for a complete list of SUS questions.6
Results
Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the preceding Usability Metrics section. We excluded data for participants who failed to follow session and task instructions.
We conducted four sessions with physicians and discovered an issue with one task. The original script asked the participant to administer the medication nitro “as needed,” but there was no “as needed” option in the application test environment. We removed “as needed” from the script to avoid confusion. Sessions impacted by this were eliminated from the results. We recruited additional physicians so that 10 physician results could be included in the analysis.
The usability testing results for the EHRUT are detailed below. The results are presented in the context of the objectives and goals outlined in Study Design. The report includes actionable results that, if corrected, should yield a material, positive impact on user performance.
The following table presents data for physicians.
5 See Tedesco and Tullis (2006) for a comparison of post-task ratings for usability tests. Tedesco, D. & Tullis, T. (2006) A comparison of methods for eliciting post-task subjective ratings in usability testing. Usability Professionals association Conference, June 12 – 16, Broomfield, CO. 6 The SUS survey yields a single number that represents a composite measure of the overall perceived usability of the system. SUS scores have a range of 0 to 100 and the score is a relative benchmark that is used against other iterations of the system.
Page 19 of 56
Physicians
Measure N Task
Success Path
Deviation Task Time Errors Task
Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
1. Enter laboratory and radiology/imaging orders 10 70%
(±48%) 1.7 99 (±54) 179% 40%
(±70%) 3.9 (±74%)
2. Enter medication orders 10 50% (±53%) 2.1 180
(±50) 279% 70% (±95%)
3.8 (±63%)
3. Cancel or hold a medication order 10 90%
(±32%) 0.9 44 (±22) 98% 10%
(±32%) 4.6 (±84%)
4. Delete or cancel a medication order 10 80%
(±42%) 1.5 46 (±19) 133% 20%
(±42%) 4.7 (±67%)
5. Update a medication order 10 90% (±32%) 1.4 98
(±40) 121% 10% (±32%)
4.3 (±67%)
6. Update a radiology order 10 90% (±32%) 1.8 78
(±29) 144% 10% (±32%)
4.0 (±105%)
7. Approve a medication order with a drug-allergy reaction 10 80%
(±42%) 3.0 75 (±30) 123% 20%
(±42%) 4.9 (±32%)
8. Approve a medication order with a drug-drug interaction 10 80%
(±42%) 2.3 76 (±21) 148% 20%
(±42%) 4.5 (±53%)
17. Configure pharmacy for e-prescription 10 50%
(±53%) 2.4 167 (±56) 207% 50%
(±53%) 3.4 (±97%)
18. Change pharmacy for e-prescription 10 50%
(±53%) 1.9 184 (±142) 196% 80%
(±103%) 3.7 (±82%)
The SUS scored participants’ subjective satisfaction with the system based on their overall experience with the EHRUT. The overall score was 86 (±10). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.7
7 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).
Page 20 of 56
Discussion of the Findings
Findings for each major area’s research question follow.
§170.314(a) (1) Computerized provider order entry
Can a user order medication, laboratory, and radiology / imaging orders? (Task 1-2)
• Participants were more successful entering lab/radiation/imaging orders (70%) than entering a medication order (50%). Entering medication orders accounted for some of the highest incidences of data entry errors (70%), primarily because participants frequently didn’t use the clinical pathways or chose the incorrect clinical pathway (e.g., Data set 1). These two tasks also accounted for some of the lowest task satisfaction ratings.
Can a user change an order? (Task 3-6)
• Overall, participants had the most success completing these three tasks, the lowest number of errors, and some of the highest task satisfaction ratings.
Can a user order a medication with a drug-allergy interaction? (Task 7)
• While the task success was comparatively high (80%), participants exhibited more path deviations than with any other task. Compared to the expected path, participants needed three times more clicks to complete the task, mainly because they were unfamiliar with the schedule and route dropdown lists and had to search more. (The applications they use in their ERs have customized dropdown lists.) However, this task had the highest task satisfaction.
• 30% of participants entered medical notations (e.g., PO or one time stat) for the medication schedule, were unaware that the application didn’t accept the entry, and submitted the order without a schedule.
Can a user order a medication with a drug-drug interaction? (Task 8)
• The task success (80%) and satisfaction (4.5) was comparatively high, and errors (20%) were relatively low.
§170.314(b) (3) Electronic prescribing
Can a user issue e-prescriptions? (Task 17-18)
• These two tasks represented the lowest task success ratios (50%) and had comparatively high error rates. These tasks have the lowest task satisfaction scores primarily because search results were very difficult to obtain, and the results display was inconsistent. Participants entered correct data, but if they included apostrophes or capital letters, no results appeared. Also, sometimes results appeared “below the fold” of the pop-up window, and participants didn’t realize that they needed to scroll down.
Page 21 of 56
Effectiveness
Task 1
• 20% of participants entered the incorrect Gas & Carbon panel.
Task 2
• 20% of participants didn’t enter the schedule for one (or both) of the medications.
• 30% of participants entered medical notations (e.g., PO or one time stat) for the medication schedule, were unaware the application didn’t accept the entry, and submitted the order without a schedule.
Tasks 3-6
• These four tasks represented the highest task success ratios.
o Task 3: 90%
o Task 4: 80%
o Task 5: 90%
o Task 6: 90%
Task 17-18
• 50% of participants failed to complete these two tasks. This was because: (1) search did not accept entries that included capitalization or apostrophes and (2) results occasionally were “below the fold” of the pop-up window where participants didn’t notice to would have needed to scroll down.
Half of the participants who successfully completed the task made multiple attempts entering search criteria.
Efficiency
In general, participants who were the most efficient understood that they could type the first letter or two into dropdown lists to narrow their results. The user interface does not indicate that this is possible.
The greatest variability in the actual task time vs. expected task time was due to the following:
• Use of Search instead of Quicklists for medications
• Unfamiliarity with dropdown list items and length
• Limited use of navigational shortcuts (e.g., GO button)
Task 1
• Average task time was 79% longer than expected because participants either didn’t use the predefined clinical pathway (Abdominal and chest pain) or selected the “Data Set 1” or “Data Set 2” clinical pathway. Many opted to search for the orders. This was likely a result of the participants’ unfamiliarity with the test environment data and customizations.
Page 22 of 56
Task 2
• 70% of participants searched for medications, rather than using Quicklists. As a result, the average task time was 179% longer than expected, higher than any other task, and they had more than twice the path deviations than the optimal path. Not using Groups accounted for a 70% error rate, primarily because the participants then had to enter the medication schedules manually. Fifty per cent of participants either didn’t enter a medication schedule or entered a medical abbreviation (e.g., PO or one time stat) which the EHRUT did not accept. The EHRUT did not alert the participants, who thought they had submitted the medication order correctly.
Task 3-6
• These four tasks represented the lowest average error rates. Participants were generally able to successfully complete the task at, or very close to, the expected task time.
o Task 3:
Errors: 10% (Only an average of 10% of participants made errors)
Task Time: -2% (Participants, on average, performed this task about the same, but slightly faster than, expected)
o Task 4:
Errors: 20%
Task Time: 33% longer than expected, primarily because some participants used the “cancel” link rather than “delete.” This added an additional step (and click)
o Task 5:
Errors: 10%
Task Time: 21% longer than expected as a result of inefficiencies typing the search entry for penicillin. Some participants either misspelled or didn’t type enough letters using “filter/find as you type” to easily find the item in the dropdown list
o Task 6:
Errors: 10%
Task Time: 44% longer than expected. When performing a search for “wrist,” some participants either misspelled or didn’t type enough letters using “filter/find as you type” to easily find the item in the dropdown list
Task 7
While participants’ task times were only slightly longer (23%) than expected, there were three times the path deviations than was optimal. This was a direct result of inefficiencies typing the search entry for “Vicodin ES.” Most participants either misspelled or didn’t type enough letters using “filter/find as you type” to easily find the item in the dropdown list
Page 23 of 56
Task 8
• The average task time was 48% longer than expected, because most participants used “Search” rather than “Groups” to enter the medication. This required them to manually enter the dose, unit, and route.
Task 17-18
• Because participants had difficulty finding the Search link, which was below the fold, the task times for task 17 and 18 were higher than any other task. Participants took 107% (for task 17) and 96% (for task 18) longer than the expected task times. Successful participants attempted the search several times by editing the search parameters until they achieved a result.
• This was the first time any participants had used the e-prescription search, so they weren’t familiar with its location. These task times should improve with greater familiarity and improved flexibility in search entries.
Satisfaction
Task 17-18
• These two tasks represented the lowest satisfaction score, which is likely due to participants’ frustration with search results not being returned as expected.
Major Findings
1. Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness
2. Participants consistently reported above average levels of product satisfaction. Many reported that of all the similar applications they’ve used, this was “the most intuitive and easy to use.” Other comments were:
"Initially, I was skeptical about (ED PulseCheck) but a month after using it I found I was faster than (hand-written notes and dictated orders).”
“I’ve grown fond of the product even though I “came to the dance” reluctantly.”
3. Participants often used mixed methods to navigate. Participants who used the GO button would occasionally also use the top menu. Using the GO button is more efficient.
4. Multiple and often less efficient navigational paths are used to accomplish tasks, increasing users’ cognitive load.
5. Only 50% of participants successfully completed Tasks 17 and 18 (relating to e-prescription search) because the search results were not provided or were not visible. Those participants that were unsuccessful made multiple attempts to change the search entry before they gave up.
E-prescription search did not accept search entries that included capitalization or apostrophes. For example, an entry of “Walgreen’s”, which is correct, did not yield results, while an entry of “walgreens” did yield results. Results sometimes displayed “below the fold” of the pop-up window. It wasn’t evident to participants that they needed to scroll down to view them.
Participants were very encouraged with the prospect of being able to enter e-prescriptions.
Page 24 of 56
6. Data was lost and record updates were incomplete. This occurred when participants entered medical notations (e.g., PO or one time stat) for the medication routes or schedule, were unaware the application didn’t accept it, and submitted the order without the entry.
7. Because action buttons are too close to each other, participants accidently clicked (or hovered over and nearly clicked) the Print Triage button rather than Enter button. The proximity and similarity of various buttons (e.g., Print Triage, Enter/Task, Enter/Med, Repeat) conflicts with the Enter button.
8. Dropdown lists displayed an inconsistent number of items depending on where they were on the page. For example, if the dropdown list for Quicklist medication schedules was towards the bottom of the page, it displayed 9-10 items, whereas, if the dropdown was near the middle, it displayed approximately 25 items.
9. Regarding medication orders, one participant commented, "I checked 'bulk' (medication for one of the tasks). It's the same mistake I make at work. Sometimes those differentiations don't mean anything to a physician. I just delete the medication, go back, and search again."
10. In general, task times and path deviations that were higher than anticipated may have been because participants normally have localized customizations (e.g., predefined pathways, drug lists, and schedules) but those familiar customizations weren’t available in the test environment. (They are specific to each ER.) Using an EHR with the customizations participants are familiar with would likely improve task times and reduce path deviations to more closely match the expected task times and optimal paths.
One participant said “90-95% of the meds I use are on my Quicklist and 70-75% of the time they're on my summary page. It's the same with orders. It's a wonderful shortcut."
11. One participant suggested a feature request: “I want to be able to draw on a picture [to show] big lacerations on a body.”
12. Three participants suggested that customized order set personalization (by a user) would be very useful and more efficient.
Page 25 of 56
RNs
Measure N Task
Success Path Deviation Task Time Errors Task
Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
10. Enter active medications 13 60% (±52%) 2.3 172
(±29) 235% 80% (±104%)
4.6 (±70%)
11. Update a medication currently being taken 13 69%
(±48%) 1.8 120 (±45) 187% 54%
(±88%) 3.9
(±126%)
12. Inactivate an active medication 13 85%
(±38%) 2.0 114 (±30) 165% 15%
(±38%) 4.4
(±96%)
13. Find a list of active and inactive allergies 13 46%
(±52%) 1.4 59 (±25) 198% 62%
(±51%) 4.3
(±125%)
14. Enter active allergies 13 92% (±28%) 1.4 113
(±33) 145% 31% (±85%)
4.5 (±78%)
15. Update the effects of an allergy 13 77%
(±44%) 1.4 82 (±57) 133% 31%
(±48%) 4.4
(±119%)
16. Inactivate an allergy 13 69% (±48%) 1.2 98
(±32) 128% 23% (±44%)
4.7 (±48%)
The SUS scored participants’ subjective satisfaction with the system based on their overall experience with the EHRUT. The overall score was 87 (±17). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.8
Discussion of the Findings
Findings for each major area’s research questions follow.
§170.314(a)(6) Medication list
Can a user enter active medications? (Task 10)
• This task resulted in the highest error rate (80%) and over twice the number of path deviations than the optimal path, primarily due to incorrect medication being updated. Task satisfaction was the second highest (4.6), likely due to the ease with which the EHRUT allows participants to efficiently and effectively enter active medications.
8 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).
Page 26 of 56
Can a user update active medications? (Task 11-12)
• 69% of participants successfully completed Task 11 and did so in nearly the same as the expected task time. However, this task had the lowest task satisfaction (3.9), primarily because participants had difficulty finding the control needed to update the medication schedule. This is a line with no text that is a hyperlink; its functionality is not obvious.
• 85% of participants successfully completed Task 12 and had the fewest errors (15%). However, there were two times more path deviations (clicks) than the optimal path, primarily because participants used search rather than the Quicklist to find the medication.
§170.314(a)(7) Medication allergy list
Can a user find a list of active and inactive allergies? (Task 13)
• 54% could not complete the task, primarily because participants couldn’t find the “Show Inactive” link to display the list of active and inactive allergies. This looks like a column heading rather than a link or button, and the instructional test is not close to the control.
Can a user enter active allergies? (Task 14)
• 92% of participants successfully completed this task.
Can a user update active allergies? (Task 15-16)
• 77% of participants successfully completed Task 15.
• 69% successfully completed Task 16. However, it took 28% longer than expected, primarily because several participants used Search to find the medication rather than the Quicklist.
Effectiveness
Task 10
• Low task success (60%) was primarily due to data entry errors (80%). Participants:
o Added notes to the wrong medication
o Selected an extra medication
o Recorded a schedule in the notes field
o Did not enter a schedule or entered it incorrectly.
Task 11
• Errors were made by 54% of participants which included:
o Choosing a medication unit "mcg" rather than "mg"
o Clicked on "Make Inactive'
o Clicked Print Triage Report rather than Enter. This resulted in the entries disappearing. The EHRUT did not alert the user.
Page 27 of 56
Task 13
• 54% of participants couldn't find the "Show Inactive" link and gave up without completing the task.
Task 14
• 92% of participants successfully completed this task.
Efficiency
• In general, the most efficient participants understood they could type the first letter or two into dropdown lists to narrow their results. The user interface does not indicate that this functionality is available.
• The greatest variability in the actual task time vs. expected task time was due to:
o Use of Search instead of Quicklists for medications
o Unfamiliarity with the dropdown lists items and length
o Limited use of navigational shortcuts (e.g., GO button)
• Longer than expected task times for Tasks 10, 11, 12, and 16 were primarily due to participants’ use of Search rather than the Quicklist. Participants didn't notice that the medications they needed to enter were in the Quicklist.
o In task 10, some participants had to navigate extremely long lists in the schedule dropdown. If the participant didn't "type ahead" to narrow the list, they had to scan as many as 36 choices.
o In task 11, several participants had difficulty finding the hyperlink line, causing longer than expected task times. One participant couldn’t figure out how to modify the current medication, so they inactivated it and created a new entry.
• 46% of participants weren't familiar with or couldn't find the "Show Inactive" link in Task 13. Those that were successful, but unfamiliar with it, eventually found it after reading the instructional note below the table.
• The higher number of path deviations (compared to optimal) for Tasks 10, 12, and 16 was primarily due to participants’ use of Search rather than the Quicklist. Most participants didn't notice that the medications they needed to enter were in the Quicklist.
Satisfaction
• Task 11: The lower satisfaction rating was due to participants’ difficulty finding the control, which was a hyperlinked line with no text.
Major Findings 1. Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness.
2. Participants expressed above average level of product satisfaction.
3. Significant efficiencies when participants used Quicklists and filter/find as you type (FAYT).
Page 28 of 56
4. The EHRUT has multiple navigation paths that participants can use to complete tasks; participants used a variety of the methods available (in some cases, this resulted in slower task times). For example, some participants clicked the patient radio button (which is unnecessary) before they used the GO button.
5. Participants often used mixed methods to navigate. Participants who used the GO button would occasionally also use the top menu. Using the GO button is more efficient.
6. Participants had difficulty finding “Show inactive” allergies.
7. Because action buttons are too close to each other, participants accidently clicked (or hovered over and nearly clicked) the Print Triage button rather than Enter button. The proximity and similarity of the Print Triage button conflicts with the Enter button.
8. Participants had difficulty identifying the control (hyperlinked line) for editing an allergy. Participants frequently clicked on the drug name link, expecting to be able to edit. This displayed a drug information help window.
9. Dropdown lists displayed an inconsistent number of items depending on where they were on the page. For example, if the dropdown list for Quicklist medication schedules was towards the bottom of the page, it displayed 9-10 items, whereas, if the dropdown was near the middle, it displayed approximately 25 items.
10. Pharmacy search parameters were too restrictive (case-sensitive and not permitting punctuation). Results sometimes required scrolling to view.
11. Users are not made aware when the system deletes unacceptable entries, resulting in data loss.
12. Users are unaware of “filter/find as you type” that would increase their efficiency.
Combined (Physician and RN) Areas for Improvement
1. Continue to emphasize to users that the GO button will improve their efficiency.9
2. Make “Show inactive” more visually identifiable.
3. Help users distinguish and click on the desired action button; consider the proximity and order of importance.
4. Don’t rely on hyperlinked lines for navigation; use text.
5. Provide consistent behavior regarding how many items a dropdown list will display regardless of where the list is clicked on in the page.
6. Provide more flexibility in the e-prescription search entry; allow for punctuation and capital letters. Consistently present the results where users can see them without needing to scroll.
7. Allow entry of medical notations (e.g., PO or one time stat) for the medication routes or schedules to avoid data loss, or alert users before loss of data.
8. Provide hints for “filter/find as you type” functionality.
9 Cognitive modeling indicates that users will save an estimated .5/sec. by using the GO button vs. the top-level menu. When .5/sec. is extrapolated over the number of times a single user interacts with patient clinical documents using the GO button, over the course of a year, the efficiency improvement can be significant.
Page 29 of 56
Appendices The following appendices include supplemental data for this usability test report. Following is a list of the appendices provided:
1. Participant demographics 2. Non-Disclosure Agreement (NDA) and Informed Consent Form 3. Example Moderator’s Guide 4. System Usability Scale Questionnaire 5. Tasks 6. Demographics Questionnaire
Page 30 of 56
Appendix 1: Participant Demographics Following is a high-level overview of the participants in this study.
Physicians
The following table summarizes the characteristics of the physicians that participated in the usability test, including demographics, professional experience, computing experience and use of assistive technology. Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.
Part ID
Part. ID Gender Age Education Occupation Professional
Experience Computer Experience
Product Experience
Assistive Technologies
1 P4 Male 30-39 Professional degree (e.g. MD)
Physician 7-9 years 20-29 years
1-3 years No
2 P11 Male 50-59 Professional degree (e.g. MD)
Physician 20-29 years 7-9 years 4-6 years Yes, Dragon
3 P13 Male 30-39 Professional degree (e.g. MD)
Physician 7-9 years 10-19 years
7-9 years No
4 P18 Male 50-59 Professional degree (e.g. MD)
Physician 10-19 years 20-29 years
4-6 years No
5 P25 Female 50-59 Professional degree (e.g. MD)
Physician 20-29 years 7-9 years 4-6 years No
6 P26 Male 60-69 Professional degree (e.g. MD)
Physician 30-39 years 20-29 years
1-3 years No
7 P14 Male 60-69 Professional degree (e.g. MD)
Physician 20-29 years 7-9 years Less than 1 year
No
8 P29 Male 30-39 Professional degree (e.g. MD)
Physician 7-9 years 10-19 years
4-6 years No
9 P28 Male 30-39 Professional degree (e.g. MD)
Physician 7-9 years 4-6 years 4-6 years Yes, Dragon
Page 31 of 56
10 P23 Male 30-39 Professional degree (e.g. MD)
Physician 4-6 years 7-9 years 4-6 years No
RNs
The following table summarizes the characteristics of the RNs that participated in the usability test, including demographics, professional experience, computing experience and use of assistive technology. Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.
Part ID
Part. ID
Gender Age Education Occupation Professional Experience
Computer Experience
Product Experience
Assistive Technologies
1 P2 Female 30-39 Associate’s degree
Registered Nurse 10-19 years 10-19 years 7-9 years No
2 P3 Female 40-49 Doctoral degree
Nurse Practitioner 10-19 years 10-19 years 1-3 years No
3 P5 Male 50-59 Bachelor’s degree
Registered Nurse 7-9 years 10-19 years 7-9 years No
4 P6 Male 40-49 Bachelor’s degree
Registered Nurse 10-19 years 10-19 years 1-3 years No
5 P8 Female 30-39 Bachelor’s degree
Registered Nurse 1-3 years 7-9 years 1-3 years No
6 P9 Female 30-39 Associate’s degree
Registered Nurse 4-6 years 1-3 years Less than 1
year Yes, Dragon
7 P12 Female 40-49 Bachelor’s degree
Registered Nurse 20-29 years 20-29 years 7-9 years No
8 P19 Male 60-69 Associate’s degree
Registered Nurse 20-29 years 4-6 years 4-6 years No
9 P20 Female 40-49 Bachelor’s degree
Registered Nurse 10-19 years 10-19 years 1-3 years No
10 P21 Male 40-49 Bachelor’s degree
Registered Nurse 4-6 years 10-19 years 4-6 years Yes, Dragon
11 P22 Female 40-49 Bachelor’s degree
Registered Nurse 20-29 years 20-29 years 4-6 years No
12 P15 Female 18-29 Bachelor’s degree
Registered Nurse 1-3 years 4-6 years 4-6 years No
13 P27 Male 40-49 Bachelor’s degree
Registered Nurse 10-19 years 10-19 years 4-6 years No
Page 32 of 56
Appendix 2: Non-Disclosure Agreement and Informed Consent Form Understanding Your Participation I agree to participate in this study conducted by Optum.
I understand Optum will record my participation for later review. By signing/acknowledging this form, I give Optum permission to use the data gathered and my comments for the purposes of evaluating the product in the study and showing the results of these evaluations, combined with other participants, to the product team. Optum will not include my name or any identifying characteristics in the results.
Any information I acquire about the product in the study is confidential and proprietary and is being disclosed only so that I can participate in the evaluation. By signing this form, I agree not to talk about this product to anyone. I may only tell others that I helped to evaluate the software. I understand that if I disclose my health information in my comments, I have disclosed it voluntarily, and Optum may use it anonymously for research and evaluation purposes only.
I understand that I can stop participating in this study at any time.
I agree to raise any concerns or areas of discomfort with the study moderator.
Name (print): __________________________________________________
Appendix 3: Example Moderator’s Guide for Physicians and RNs
Thank you for joining us today; we really appreciate your time. I’m John Romadka; I work at Optum, makers of ED PulseCheck. Jamie Peffley (or Scott Stewart) is also on the on line, and she’ll be helping take notes.
Before we begin, I’d like to take a moment to describe the study we’re conducting.
We’re collecting feedback on a software product called ED PulseCheck.
I know someone already asked you this but I just wanted to verify that you currently use ED PulseCheck to work with patient records.
Has not used ED PulseCheck Has used ED PulseCheck
We’re going to ask you to perform a series of tasks. You should know that the tasks were created to be independent of each other and some may sound redundant. You will be asked to complete these tasks on your own, trying to do them as quickly as possible with the fewest possible errors or deviations. Please do not do anything more than the tasks asks.
You are not being tested on your ability to complete the tasks. We’re evaluating how intuitive the product is to use, so just do your best to complete them as you typically would. We didn’t have anything to do with designing the product, and your comments are entirely anonymous, so please feel free speak candidly. However, please save your comments until the end of a task, or the end of the session as a whole, when we can then discuss them freely.
We’re going to be using a study tool that will lead you through the tasks. We’d ask that you please read this task information out loud, so we can follow along.
I’m going to share my screen with you in the WebEx. You’ll be using Ed PulseCheck on my computer, but I’ll give you control of my computers mouse and keyboard for the duration of the study. Because we can’t always control the bandwidth between our two locations, you may notice a tiny lag or latency when you’re in control of the mouse and keyboard. However, we don’t anticipate this will affect your ability to use ED PulseCheck.
Before I do that, please quit or close any other applications that are currently running on your computer (email and Instant Messaging), with the exception of your browser and WebEx running. Also, please set any mobile devices to silent during the study.
Page 34 of 56
Appendix 4: System Usability Scale Questionnaire In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, at http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
16 Brooke, J.: SUS: A “quick and dirty” usability scale. In: Jordan, P. W., Thomas, B., Weerdmeester, B. A., McClelland (eds.) Usability Evaluation in Industry pp. 189--194. Taylor & Francis, London, UK (1996). SUS is copyrighted to Digital Equipment Corporation, 1986.
Lewis, J R & Sauro, J. (2009) "The Factor Structure Of The System Usability Scale." in Proceedings of the Human Computer Interaction International Conference (HCII 2009), San Diego CA, USA
There are multiple correct paths to successfully complete the tasks; however, the “optimal path” is indicated in bold.
Task 0 – Login Role: Physician
Starting URL: http://localhost/ibex23.mpex
For the purpose of this study, you are Chris Huffman, a medical professional working in the emergency department of PulseCheck Hospital. The emergency department uses ED PulseCheck to triage incoming patients. Your ED PulseCheck information is below.
Username: chuffman
Password: 123
Log in to ED PulseCheck and view the tracking board.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
No time tracking required.
No completion required.
Page 36 of 56
Task 1 - Enter a laboratory, radiology, and imaging orders
Robert Michael O’Brien was brought to the emergency department for abdominal and chest pain. A patient record has already been created for Robert in ED PulseCheck and has been assigned to you.
Enter the following lab, radiology, and imaging orders:
Complete blood count (CBC)
Hemoglobin A1c/hemoglobin total
Gas & carbon monoxide panel
CT abdomen & pelvis w/ contrast material
Myocardial image PET perfus single study rest/stress
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Orders page. a. Directly from the Tracking Board page
i. Select the patient > hover over All on the menu bar > click Orders ii. Select the patient > click Order on the menu bar
iii. Select the patient > click Go > Orders on the patient row b. Via the My Patients page (selecting the patient first is not necessary)
i. Click All on the menu bar > My Patients > select patient if necessary > Orders ii. Click My Patients on the menu bar > select patient if necessary > Orders
2. Enter orders. i. Select orders from Abdominal and chest pain clinical pathway ii. Search for individual order by name
Successful Completion: User clicks Enter, Enter/Task, Enter/Med, or Repeat after entering the orders.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 37 of 56
Task 2 - Enter medication orders
Robert Michael O’Brien was brought to the emergency department for abdominal and chest pain. A patient record has already been created for Robert in ED PulseCheck and assigned to you.
Task: Enter the following medication orders:
Digoxin 0.25 mg tablet by mouth once daily
Morphine 2 mg IV push x1 for pain , every 4 hours as needed
Nitro-bid oral 2.5 mg oral, as needed
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the Medication Services tab
2. Enter orders.
a. Click Group>Abdominal & chest pain; select medications > Enter
b. Search for individual medications by name > Enter
Successful Completion Criteria: User clicks Enter after the orders are entered.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 38 of 56
Task 3 – Cancel or hold a medication order
Robert Michael O’Brien was brought to the emergency department for abdominal and chest pain. A patient record was created for Robert in ED PulseCheck and assigned to you. You ordered Aspir-81 for Robert earlier but he is refusing the medication.
Task: Update Robert’s medication orders to reflect the fact that he is refusing the medication. Enter a note stating, “Patient refused.”
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the Medication Services tab
2. Cancel medication
a. Locate medication > click Cancel link > check Medication refused > clicks checkbox “Patent refused” >click Cancel button
b. Locate medication > click Hold link > check Medication refused > enter note >click Hold button
Successful Completion Criteria: User clicks Cancel or Hold button after entering a note.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 39 of 56
Task 4 – Delete or cancel a medication order Role: Physician w/ experience entering medication orders
Monica Webster was brought to the emergency department for a laceration to her left hand. It has come to your attention that you accidently ordered Norcuron 10 mg IVP for her instead of your other patient.
Task: Update Monica’s medication orders to reflect the fact that Norcuron 10 mg IVP is not needed.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the Medication Services tab
2. Delete medication
i. Locate medication > click Delete link
ii. Locate medication > click Cancel link > click Cancel button
Successful Completion: User clicks Delete link or Cancel button.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 40 of 56
Task 5 – Update a medication order Role: Physician w/ experience entering medication orders
Javier Knight was brought to the emergency department for a sore throat. It has come to your attention that Penicillin V Potassium 125mg/5ml oral was inadvertently ordered for Javier instead of Penicillin G Potassium 1 million units, IM.
Task: Update Javier’s medication orders to reflect the fact that Penicillin G Potassium 1 million units, IM should be given.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page: i. Select the patient > hover over All on the menu bar > click Med SVC ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page: i. Select the patient > hover over All on the menu bar > click Orders > click the Medication
Services tab ii. Select the patient > click Orders on the menu bar > click the Medication Services tab iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the
Medication Services tab v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the
Medication Services tab 2. Cancel, delete, or hold medication.
a. Cancel: Locate medication > click Cancel > check Medication refused > enter note b. Delete: Locate medication > click Delete c. Hold: Locate medication > click Hold > check Medication refused > enter note
3. Enter order. a. Search for medication (Penicillin) by name.
Successful Completion: User clicks Enter, Enter/Task, Enter/Med, or Repeat after entering medication order with correct dosage. Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 41 of 56
Task 6 –Update a radiology order Role: Physician w/ experience entering lab and imaging orders
John Smith was brought to the emergency department for an injury to his right wrist. A patient record was created for John in ED PulseCheck and assigned to you. John needs a 2 view wrist x-ray but a 2 view chest x-ray was accidently ordered for him.
Task: Update John’s record to reflect the fact that he needs a 2 view wrist x-ray rather than chest x-rays.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Navigate to the My Tasks page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Orders
ii. Select the patient > click Order on the menu bar
iii. Select the patient > click Go > Orders on the patient row
iv. Click the Orders icon
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > Orders
ii. Click My Patients on the menu bar > select patient if necessary > Orders
2. Cancel the chest x-ray.
i. Click Cancel
3. Click Orders
4. Order the wrist x-ray.
i. Search by SVC Name > type wrist > select wrist or wrist (2 views) from list
Successful Completion: Depends on order in which user completes tasks. User can click Enter after entering the wrist x-ray or click Cancel after selecting the chest x-ray.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 42 of 56
Task 7 – Approve a medication order with a drug-allergy reaction Role: Physician w/ experience entering medication orders
John Smith was brought to the emergency department for an injury to his right wrist. A patient record has already been created for John in ED PulseCheck and assigned to you. John is allergic to Tylenol but has been treated with Vicodin in the past and has responded well with no side effects.
Task: Enter an order for one tablet of Vicodin ES for John.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the Medication Services tab
2. From the Medication Services page, search for medication by name.
3. In Allergy Reaction alert, check Patient tolerated same medication(s) in the past.
4. Click Enter.
Successful Completion: Interaction alert has been acknowledged and user clicks Enter.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 43 of 56
Task 8 – Approve a medication order with a drug-drug interaction Role: Physician w/ experience entering medication orders
Zackary A Pearson was brought to the emergency department for a fever. A patient record has already been created for Zackary in ED PulseCheck and assigned to you. Zackary is currently taking HCTZ. You believe the benefits of taking lithium carbonate outweigh the risks associated with drug interactions.
Task: Order two lithium carbonate 150mg capsules for Zackary.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the Medication Services tab
2. Enter orders.
i. Use Groups to enter medications
ii. Search for individual medications by name
iii. In Severe Interaction alert for HCTZ and lithium carbonate, check Benefits outweigh risks.
iv. Click Enter
Successful Completion: User clicks Enter after the orders are entered. Completed
Task 17 - Configure pharmacy for e-prescription Roles: Physician
Janet Kikura was brought to the emergency department for a severe migraine headache. A patient record has already been created for Janet in ED PulseCheck and assigned to you. You have learned that you can use the newest version of ED PulseCheck to send an electronic prescription to Janet’s pharmacy so it will be available for her to pick up on the way home.
Task: Use ED PulseCheck to write a prescription for Ibuprofen 800mg, 1 tablet taken orally every 8 hours as needed, no refills, 20 tablets to be dispensed. The closest pharmacy to Janet’s home is Walgreen’s, located at 930 Elk Grove Town Ctr. Elk Grove Village, IL.
**Formulary information is not currently available. You can ignore any message regarding formulary information.**
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Prescriptions page.
a. To access the Prescriptions page from the Tracking Board page: i. Select the patient > hover over All on the menu bar > click Rx ii. Select the patient > click Rx on the menu bar
iii. Select the patient > click Go > Rx on the patient row b. To access the Rx page from the My Patients page (selecting the patient first is not necessary):
i. Select patient (if necessary) > hover over All on the menu bar > click Rx ii. Select patient (if necessary) > click Rx on the menu bar iii. Select patient (if necessary) > click Prescriptions in the right panel
c. To access the Rx page from the Disposition page: i. Scroll to bottom > click Rx button
2. Add medication. a. Find medication
i. Click Search All ii. Type medication name iii. Select medication from list
b. Enter dose, schedule, and dispense instructions. 3. Send to pharmacy.
i. Click Pharmacy ii. From the Pharmacy popup, enter pharmacy name iii. Select pharmacy from list iv. Click OK v. Click Enter
Successful Completion: User clicks Enter after selecting medication and pharmacy. Completed
Yes No/Assisted Overall this task was
1 2 3 4 5
Page 45 of 56
Task 18 - Change pharmacy for e-prescription Role: Physician
Zackary A Pearson was brought to the emergency department for a fever. A patient record has already been created for Zackary in ED PulseCheck and assigned to you. You have learned that you can use the newest version of ED PulseCheck to send an electronic prescription to Zackary’s pharmacy so it will be available for him to pick up on the way home.
Task: Zackary usually fills prescriptions at Osco Drug, located on 6200 W. Higgins Avenue Chicago IL, which is close to his home. He is planning on staying with a family member when he is discharged and would like to fill his prescription at another pharmacy instead. Use ED PulseCheck to write a prescription for Amoxicillin (SUSPENSION, RECONSTITUTED, ORAL (ML) : 125 mg/5 mL ) ; 5mls every 6 hours ; dispense 1 bottle and send the prescription to Costco Pharmacy located at 505 W. Army Trail Road Bloomingdale IL.
**Formulary information is not currently available. You can ignore any message regarding formulary information.**
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Prescriptions page.
a. To access the Prescriptions page from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Rx
ii. Select the patient > click Rx on the menu bar
iii. Select the patient > click Go > Rx on the patient row
b. To access the Rx page from the My Patients page (selecting the patient first is not necessary):
i. Select patient (if necessary) > hover over All on the menu bar > click Rx
ii. Select patient (if necessary) > click Rx on the menu bar
iii. Select patient (if necessary) > click Prescriptions in the right panel
c. To access the Rx page from the Disposition page:
i. Scroll to bottom > click Rx button
2. Send to pharmacy.
i. Click Pharmacy
ii. From the Pharmacy popup, enter pharmacy name
iii. Select pharmacy from list
iv. Click OK
v. Click Enter
Successful Completion: User clicks Enter after selecting medication and pharmacy.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 46 of 56
RNs
There are multiple correct paths to successfully complete the tasks; however the “optimal path” is indicated in bold.
For the purpose of this study, you are Chris Huffman, a medical professional working in the emergency department of PulseCheck Hospital. The emergency department uses ED PulseCheck to triage incoming patients. Your ED PulseCheck information is below.
Username: chuffman
Password: 123
Log in to ED PulseCheck and view the tracking board.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
No time tracking required.
No completion required.
Page 47 of 56
Task 10 - Enter active medications
Susan McCoy was brought to the emergency department for stomach pain that was previously diagnosed as gastro esophageal reflux disease (GERD). A patient record has already been created for Susan in ED PulseCheck and assigned to you. Susan neglected to mention that she takes several over-the-counter medications regularly for GERD.
Task: Update Susan’s record to reflect the medications she takes.
Pepcid AC 10mg chewables, 2 daily
TUMS E-X 300mg chewables, 4 daily
Zegerid OTC 20mg capsule, 1 daily
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Current Medications page. a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Current Meds ii. Select the patient > click Current Meds on the menu bar
iii. Select the patient > click Go > Current Meds on the patient row b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Current Meds ii. Click My Patients on the menu bar > select patient if necessary > click Current Meds
c. Via the patient’s Charting page: i. Click the patient’s name > click Current Medications (only valid if current medications have not been
previously entered) d. Via the Allergies page:
i. Select the patient > hover over All on the menu bar > click Allergies > click Medications tab ii. Select the patient > click Allergies on the menu bar > click Medications tab iii. Select the patient > click Go > Allergies on the patient row > click Medications tab iv. Click All on the menu bar > My Patients > select patient if necessary > click Allergies > click
Medications tab v. Click My Patients on the menu bar > select patient if necessary > click Allergies > click Medications
tab vi. Click the patient’s name > click Allergies (only valid if current medications have not been previously
entered) > click Medications tab 2. Enter medications.
a. Use the QuickList to add the medications > click Enter Successful Completion Criteria: User clicks Enter or Enter/Allergies after entering three medications. Completed
Yes No/Assisted Overall this task was
1 2 3 4 5
Page 48 of 56
Task 11 - Update a medication currently being taken Role: Nurse
Zackary A Pearson was brought to the emergency department for fever. A patient record has already been created for Zackary in ED PulseCheck and assigned to you. Zachary said he took digoxin but neglected to say how often he took it.
Task: Update Zachary’s record to reflect the fact that he takes digoxin, orally, 0.25 mg once a day (in the morning).
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Current Medications page. a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Current Meds ii. Select the patient > click Current Meds on the menu bar
iii. Select the patient > click Go > Current Meds on the patient row b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Current Meds ii. Click My Patients on the menu bar > select patient if necessary > click Current Meds
c. Via the patient’s Charting page: i. Click the patient’s name > click Current Medications (only valid if current medications have not
been previously entered) d. Via the Allergies page:
i. Select the patient > hover over All on the menu bar > click Allergies > click Medications tab ii. Select the patient > click Allergies on the menu bar > click Medications tab iii. Select the patient > click Go > Allergies on the patient row > click Medications tab iv. Click All on the menu bar > My Patients > select patient if necessary > click Allergies > click
Medications tab v. Click My Patients on the menu bar > select patient if necessary > click Allergies > click
Medications tab vi. Click the patient’s name > click Allergies (only valid if current medications have not been
i. Click on the hyperlinked line to the right of the medication name ii. In the Schedule field, select once a day iii. Click Update
Successful Completion: User clicks Update after entering dosage. Completed
Yes No/Assisted Overall this task was
1 2 3 4 5
Page 49 of 56
Task 12 - Inactivate an active medication Role: Nurse
Zackary A Pearson was brought to the emergency department for a fever. A patient record has already been created for Zackary in ED PulseCheck and assigned to you. Earlier, Zackary said he took Zegerid OTC once a day. A family member has arrived at the hospital with his medications and you learn that Zackary is taking Tums EX, not Zegerid OTC.
Task: Update Zackary’s record to reflect the fact he is taking Tums EX 300 mg calcium (750 mg), 1 tab oral, once a day, not Zegerid OTC 20 mg-1.1 gram 1 tab oral once a day.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Current Medications page.
a. Directly from the Tracking Board page: i. Select the patient > hover over All on the menu bar > click Current Meds ii. Select the patient > click Current Meds on the menu bar
iii. Select the patient > click Go > Current Meds on the patient row b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Current Meds ii. Click My Patients on the menu bar > select patient if necessary > click Current Meds
c. Via the patient’s Charting page: i. Click the patient’s name > click Current Medications (only valid if current medications have not
been previously entered) d. Via the Allergies page:
i. Select the patient > hover over All on the menu bar > click Allergies > click Medications tab ii. Select the patient > click Allergies on the menu bar > click Medications tab iii. Select the patient > click Go > Allergies on the patient row > click Medications tab iv. Click All on the menu bar > My Patients > select patient if necessary > click Allergies > click
Medications tab v. Click My Patients on the menu bar > select patient if necessary > click Allergies > click
Medications tab vi. Click the patient’s name > click Allergies (only valid if current medications have not been
Task 13 – Find active and inactive allergies Role: Nurse
Zackary A Pearson was brought to the emergency department for a fever. A patient record has already been created for Zackary in ED PulseCheck and assigned to you. Zackary and his family have provided conflicting lists of allergies for Zackary. You want to read through a list of allergies that have been mentioned so you can verify which ones affect him.
Task: Display a complete list of Zackary’s active and inactive allergies that you would use to verify each allergy with him and his family.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Known Allergies page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Allergies
ii. Select the patient > click Allergies on the menu bar
iii. Select the patient > click Go > Allergies on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Allergies
ii. Click My Patients on the menu bar > select patient if necessary > click Allergies
c. Via the patient’s Charting page:
i. Click the patient’s name > click Known Allergies
2. Display active and inactive allergies.
i. Click Show Inactive to include inactive allergies in the list of allergies
Successful Completion: User clicks Show Inactive.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 51 of 56
Task 14 - Enter active allergies Role: Nurse
Susan McCoy was brought to the emergency department for gastro esophageal reflux disease (GERD). A patient record has already been created for Susan in ED PulseCheck and assigned to you. Susan says she is allergic to several medications.
Allergies: Aspirin
Bactrim
Cipro
Nausea
Hives
Difficulty breathing
Mild
Moderate
Severe
Task: Update Susan’s record to reflect her allergies.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Known Allergies page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Allergies
ii. Select the patient > click Allergies on the menu bar
iii. Select the patient > click Go > Allergies on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Allergies
ii. Click My Patients on the menu bar > select patient if necessary > click Allergies
c. Via the patient’s Charting page:
i. Click the patient’s name > click Known Allergies
2. Add allergies.
i. From the Known Allergies page, type medication names in the Quick List tab > Click checkbox, enter reaction, severity and source > click Enter
ii. From the Known Allergies page, type medication names in the Search tab > enter reaction, severity and source > click Enter
Successful Completion: User clicks Enter.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 52 of 56
Task 15 - Update the effects of an allergy Role: Nurse
Richard Bird was brought to the emergency department with a broken arm. A patient record has already been created for Richard in ED PulseCheck and assigned to you. Richard said he was allergic to penicillin but the severity and nature of his allergy was not noted. A family member mentioned that Richard had difficulty breathing due to a swollen throat when he’d been given penicillin in the past.
Task: Update Richard’s record to reflect the fact that he is severely allergic to penicillin.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Known Allergies page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Allergies
ii. Select the patient > click Allergies on the menu bar
iii. Select the patient > click Go > Allergies on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Allergies
ii. Click My Patients on the menu bar > select patient if necessary > click Allergies
c. Via the patient’s Charting page:
i. Click the patient’s name > click Known Allergies
2. Update the allergy.
i. Click on the hyperlinked line to the right of the allergy name
ii. Enter reaction, severity and source
iii. Click Update
Successful Completion: User clicks Update after entering reaction details have been added to allergy.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 53 of 56
Task 16 - Inactivate an allergy Role: Nurse
John Smith was brought to the emergency department for right wrist pain. A patient record has already been created for John in ED PulseCheck and assigned to you. Earlier, a family member said John was allergic to Tylenol. After talking with John, you learn that he is allergic to Lortab ASA , not Tylenol.
Task: Update John’s record to reflect the fact that he is allergic (mild hives) to Lortab ASA, not Tylenol.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Known Allergies page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Allergies
ii. Select the patient > click Allergies on the menu bar
iii. Select the patient > click Go > Allergies on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Allergies
ii. Click My Patients on the menu bar > select patient if necessary > click Allergies
c. Via the patient’s Charting page:
i. Click the patient’s name > click Known Allergies
2. Inactivate allergy.
i. Click on Make inactive to the right of Tylenol
3. Add allergy.
i. From the Known Allergies page, type medication names in the Quick List tab > enter reaction, severity and source > click Enter
ii. From the Known Allergies page, type medication names in the Search tab > enter reaction, severity and source > click Enter
Successful Completion: User clicks Enter after entering reaction details. Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 54 of 56
Appendix 6: Demographics Questionnaire 1. Gender
O Female O Male
2. Age
O 18-29 years O 30-39 years O 40-49 years
O 50-59 years
O 60-69 years O 70 or more years
3. What is the highest degree or level of school you have completed?
O High School, GED, or equivalent O Some college, no degree O Technical certification O Associate degree O Bachelor degree O Master degree O Professional degree O Doctorate degree O Other: (please specify)
_____________________________
4. What title best describes your current occupation?
O Emergency Medical Technician
O Health Information System Administrator
O Nurse Practitioner O Office Administrator O Pharmacist O Physician O Physician’s Assistant
O Psychologist, Counselor, or Social Worker
O Registered Nurse O Other: (please specify)
_____________________________
Page 55 of 56
5. How long have you held your current title?
O Less than 1 year O 1-3 years
O 4-6 years
O 7-9 years O 10-19 years O 20-29 years O 30-39 years O 40-49 years O 50 or more years
6. How long have you used a computer in a professional workplace?
O Less than 1 year O 1-3 years
O 4-6 years
O 7-9 years O 10-19 years O 20-29 years O 30-39 years O 40-49 years O 50 or more years
7. How long have you used ED PulseCheck?
O Less than 1 year O 1-3 years
O 4-6 years
O 7-9 years O 10 or more years
8. Do you use assistive technology; such as an enhanced visual display, physically adapted pointing device, or text-to-speech engine; in the workplace?
O Yes O No
If yes, please describe __________________________________
Page 56 of 56
Appendix 7: Data Scoring Details
As was described in the Data Scoring – Effectiveness, Task Success section, optimal task performance time was benchmarked when constructing tasks by using Keystroke Level Modeling (KLM) to measure the clicks, keystrokes, and task time to obtain an expected time based on the optimal path. (KLM was used rather than expert time because the product includes many paths to accomplish tasks, and experts frequently didn’t follow optimal paths.)
To quantify the additional time added to the KLM calculations, three expert performances of three tasks with varying complexity were reviewed (Physician tasks: 2, 4, 18; RN tasks: 10, 13, 16) based on the rules described below. An average time for each rule was calculated and then consistently applied throughout the KLM calculations.
The rules below detail how this additional time, based on test environment measurements, was added to the KLM results to account for the following:
1. User’s need to review information while interacting with the user interface (medical professionals double-check their work routinely to avoid error):
• 5 seconds: Review data entry before next step
• 3 seconds: Look at information displayed (e.g., table data)
• 3 seconds: Find patient in list of Tracking Board
2. System response times:
• 7 seconds: (Tasks 17-18) After Search button is clicked until the search window appears
• 4 seconds: For next page to appear after clicking Enter button
Or
2 seconds: (If there was no Enter button) System responds and displays the next page
Test Results Summary for 2014 Edition EHR Certification 16-2423-R-0024-PRI V1.0, July 8, 2016
