(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(f)(3)
Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)
(a)(4)(iii) Plot and display growth charts
(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)
Common MU Data Set (15) Express Procedures according to the standard specified
at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)
Test Results Summary for 2014 Edition EHR Certification15-3281-R-0016-PRA V1.1, February 24, 2016
(a)(16) Inpt. only (f)(3) (a)(17) Inpt. only (f)(4) Inpt. only
(a)(18) (f)(5) Optional & Amb. only
(b)(5) (g)(3) (b)(6) Inpt. only (g)(4) (b)(7) (h)(1)
(b)(2) (f)(7) Amb. only
(b)(3) (g)(1) (b)(4) (g)(2)
*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)**Indicates the version number for the Test Procedure (TP)***Indicates the version number for the Test Data (TD)
(b)(8) (h)(2) (b)(9) (h)(3) (c)(1) (c)(2)
Test Results Summary for 2014 Edition EHR Certification15-3281-R-0016-PRA V1.1, February 24, 2016
Ambulatory Inpatient No CQMs tested*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov (navigation: 2014 Clinical Quality Measures)
Ambulatory CQMs
Inpatient CQMs
Test Results Summary for 2014 Edition EHR Certification15-3281-R-0016-PRA V1.1, February 24, 2016
Quality Management System** C Privacy and Security D*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (b)(3), (b)(4)**Required for every EHR product
(a)(7) (a)(15) (b)(5)
Attestation Forms (as applicable) Appendix Safety-Enhanced Design* B
Test Results Summary for 2014 Edition EHR Certification15-3281-R-0016-PRA V1.1, February 24, 2016
PERFORMANCE DATA ....................................................................................................................................................... 5 Providers .................................................................................................................................................................... 6
Major Findings ......................................................................................................................................................................... 7 Areas for Improvement ............................................................................................................................................................ 7
Clinical Support Staff ................................................................................................................................................. 8 Major Findings ......................................................................................................................................................................... 9 Areas for Improvement ............................................................................................................................................................ 9
STUDY DESIGN .............................................................................................................................................................. 13 TASKS ........................................................................................................................................................................... 14
Providers .................................................................................................................................................................. 14 Clinical Support Staff ............................................................................................................................................... 15
PROCEDURES ................................................................................................................................................................ 15 TEST LOCATION ............................................................................................................................................................. 16 TEST ENVIRONMENT ...................................................................................................................................................... 16 TEST FORMS AND TOOLS ............................................................................................................................................... 16 PARTICIPANT INSTRUCTIONS .......................................................................................................................................... 17 USABILITY METRICS ....................................................................................................................................................... 17
DATA SCORING ............................................................................................................................................................. 18
DATA ANALYSIS AND REPORTING .................................................................................................................................... 19 Providers .................................................................................................................................................................. 20
Discussion of the Findings ..................................................................................................................................................... 21 Major Findings ....................................................................................................................................................................... 23
Clinical Support Staff ............................................................................................................................................... 25 Discussion of the Findings ..................................................................................................................................................... 26 Major Findings ....................................................................................................................................................................... 28 Areas for Improvement .......................................................................................................................................................... 29
Combined (Provider and Clinical Support Staff) Areas for Improvement ................................................................ 30
APPENDIX 2: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ................................................................. 33 APPENDIX 3: EXAMPLE MODERATOR’S GUIDE FOR PROVIDERS AND CLINICAL SUPPORT STAFF ........................................ 34 APPENDIX 4: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................................................................ 35 APPENDIX 5: TASKS ....................................................................................................................................................... 36
Providers .................................................................................................................................................................. 36 Task 1 – Enter laboratory and radiology/imaging orders ....................................................................................................... 36
Page 3 of 61
Task 2 – Enter medication orders .......................................................................................................................................... 38 Task 3 – Discontinue a medication order .............................................................................................................................. 40 Task 4 – Update a medication order...................................................................................................................................... 41 Task 5 – Update a radiology order ........................................................................................................................................ 42 Task 6 – Approve a medication order with a drug-allergy reaction ........................................................................................ 43 Task 7 – Approve a medication order with a drug-drug interaction ....................................................................................... 44 Task 16 – Select a pharmacy for e-prescription .................................................................................................................... 45 Task 19 – Access reference clinical information .................................................................................................................... 46 Task 20 – Activate a Patient Care Management alert ........................................................................................................... 47
Clinical Support Staff ............................................................................................................................................... 48 Task 8 – Find a list of active and inactive medications .......................................................................................................... 48 Task 9 – Enter active medications ......................................................................................................................................... 49 Task 10 – Update a medication currently being taken ........................................................................................................... 50 Task 11 – Inactivate an active medication ............................................................................................................................. 51 Task 12 – Find a list of active and inactive allergies .............................................................................................................. 52 Task 13 – Enter active allergies ............................................................................................................................................ 53 Task 14 – Update the effects of an allergy ............................................................................................................................ 54 Task 15 – Inactivate an allergy .............................................................................................................................................. 55 Task 17 – Reconcile a patient medication list ........................................................................................................................ 56 Task 18 – Trigger a Patient Care Management alert ............................................................................................................. 59
Executive Summary A summative usability test of Optum PM and Physician EMR version 8.1 was conducted 6/11/2014 – 6/25/2014 by the Optum User Experience Center of Excellence. The test was conducted remotely using a web-based testing tool, UserZoom, and WebEx. The purpose of this test was to provide evidence of usability in the EHR Under Test (EHRUT) and identify areas for incremental usability improvement.
Eight providers and 12 clinical support staff matching the target demographic criteria participated in the usability test, using the EHR in simulated but representative tasks.
During the 60-minute one-on-one usability sessions, each participant was greeted by the administrator and asked to review and sign a non-disclosure/consent form (see Appendix 2). Participants were instructed that they could withdraw at any time. All participants had prior experience with the EHR.
The administrator introduced the test and instructed participants to complete a series of tasks (presented randomly one at a time) using the EHR. The testing tool (UserZoom) recorded user performance data electronically. A data logger also recorded user performance. The administrator did not give the participant assistance in how to complete the task (and if it became necessary, that task was marked as failed).
6 Approve a medication order with a drug-allergy reaction
7 Approve a medication order with a drug-drug interaction
§170.314(a)(6) Medication list
8 Find a list of active and inactive home medications
9 Enter active medications
10 Update a medication currently being taken
11 Inactivate an active medication
§170.314(a)(7) Medication allergy list
12 Find a list of active and inactive allergies
13 Enter active allergies
14 Update the effects of an allergy
Page 5 of 61
Area of Certification Tasks
Prov
ider
s
CSS
15 Inactivate an allergy
§170.314(b)(3) Electronic prescribing 16 Configure pharmacy for e-prescription
§170.314(b)(4) Clinical information reconciliation
17 Reconcile a patient medication list
18 Trigger a Patient Care Management alert
19 Access reference clinical information
§170.314(a)(8) Clinical decision support
20 Activate a Patient Care Management alert
Each participant’s interactions with the EHR were recorded for subsequent analysis. The following types of data were collected:
• Number of tasks successfully completed within the allotted time, without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant’s verbalizations
• Participant’s satisfaction ratings of the tasks and system
All participant data was de-identified to avoid linking the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire. Providers and Clinical Support Staff were compensated $250 and $150 respectively for their time.
Performance Data Various recommended metrics, in accordance with the examples set forth in the NISTIR Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHR. The following is a summary of the performance and rating data collected on the EHR.
Page 6 of 61
Providers
Measure N Task
Success Path
Deviation Task Time Errors Task
Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
1. Enter laboratory and radiology/imaging orders 8 75%
(±46%) 1.0 168 (±75) 86% 25%
(±46%) 4.5
(±53%)
2. Enter medication orders 8 100%
(±0%) 1.1 165 (±67) 75% 0%
(±0%) 4.8
(±46%)
3. Discontinue a medication order 8 100%
(±0%) 3.4 103 (±39) 113% 0%
(±0%) 3.8
(±71%)
4. Update a medication order 8 88%
(±35%) 1.6 95 (±51) 100% 13%
(±35%) 4.3
(±71%)
5. Update a radiology order 8 88%
(±35%) 1.2 85 (±23) 89% 13%
(±35%) 4.3
(±46%)
6. Approve a medication order with a drug-allergy reaction
8 88% (±35%) 1.1 123
(±79) 87% 13% (±35%)
4.5 (±53%)
7. Approve a medication order with a drug-drug interaction
8 88% (±35%) 1.0 79
(±27) 92% 13% (±35%)
4.3 (±139%)
16. Configure pharmacy for e-prescription
8 100% (±0%) 1.0 124
(±35) 97% 0% (±0%)
4.6 (±52%)
19. Access reference clinical information 8 63%
(±52%) 1.3 78 (±97) 84% 38%
(±52%) 3.8
(±116%)
20. Activate a Patient Care Management alert 8 38%
(±52%) 1.1 119 (±81) 91% 63%
(±52%) 1.9
(±99%)
The System Usability Scale was used to score participants’ subjective satisfaction based on their overall experience with the EHR. The overall score was 86 (±15).1
1 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.
Page 7 of 61
In addition to the performance data, we recorded the following qualitative observations.
Major Findings Major findings include:
• Orders and updates did not automatically refresh
• Difficulty distinguishing between similarly named items (e.g., Radiologic examination, wrist, 2-view vs. Radiologic examination, chest, single view, frontal) of favorite orders, caused errors
• The terminology used for the referenced clinical information link (i.e., Diagnosis info) didn’t match participants’ expectations
• The application displayed erratic behavior which prevented one participant from completing two tasks (e.g., black background and disappearing buttons)
Additional issues:
• A missing confirmation message after participants attempted to save a Patient Care Management alert
• The number of links and organization in the Administrative section prevented some participants from easily finding the necessary Patient Care Management form
• Inconsistency in the style, placement, and behavior of controls (i.e., action buttons, filters, dropdowns, and links) throughout the application
• Data entry screens that do not indicate which fields are required
• The alignment of form elements (e.g., radio buttons) with the associated text labels
Areas for Improvement Recommendations for improvement include the following:
• Display orders and updates automatically
• Help user distinguish between similarly labeled items in the favorite orders list
• Ensure terminology used for referenced clinical information aligns with users’ expectations
• Eliminate erratic application behavior that prevents users from performing their tasks
• Ensure that controls have consistent style, placement, and behavior
• Consistently provide confirmation messages
• Consider more effective ways to organize and display items in the Administrative section
• Indicate the required fields on data entry screens
• Align form elements with the associated text labels
Page 8 of 61
Clinical Support Staff
Measure N Task
Success Path
Deviation Task Time Errors Task
Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
8. Find a list of active and inactive medication 12 75%
(±45%) 1.4 65 (±37) 121% 25%
(±45%) 4.5
(±67%)
9. Enter active medications 12 58% (±51%) 2.3 75
(±25) 146% 58% (±90%)
4.6 (±90%)
10. Update a medication currently being taken 12 100%
(±0%) 2.0 101 (±43) 86% 0%
(±0%) 4.5
(±100%)
11. Inactivate an active medication 12 100%
(±0%) 1.1 42 (±11) 70% 0%
(±0%) 4.8
(±39%)
12. Find a list of active/inactive allergies 12 50%
(±52%) 1.3 60 (±24) 102% 50%
(±52%) 4.5
(±67%)
13. Enter active allergies 12 83% (±39%) 1.1 105
(±23) 210% 52% (±145%)
4.5 (±100%)
14. Update the effects of an allergy 12 83%
(±39%) 1.3 74 (±49) 78% 17%
(±39%) 4.3
106%)
15. Inactivate an allergy 12 92% (±29%) 1.6 95
(±48) 164% 17% (±58%)
4.3 (±89%)
17. Reconcile a patient medication list 11 18%
(±40%) 1.6 258 (±42) 404% 100%
(±63%) 3.0
(±89%)
18. Trigger a Patient Care Management alert 12 67%
(±47%) 1.0 98 (±32) 57% 33%
(±46%) 3.9
(±156%)
The System Usability Scale was used to score participants’ subjective satisfaction with the system based on their overall experience with the EHR. The overall score was 78(±13).2
2 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.
Page 9 of 61
In addition to the performance data, we recorded the following qualitative observations.
Major Findings Major findings include:
• Orders and updates did not automatically refresh
• Allergy reaction data entry was lost
• Low completion rate for reconciliation of a patient medication list, even when participants had the help documentation
• Multipage data entry screens did not indicate where the participants were in the CCDA Reconciliation process contributed to low task success
• Multiple competing action buttons (e.g., Reconcile, Import, Next) within the CCDA Reconciliation page contributed to low task success
• No assistance or suggestions for misspelled medication search queries
• Inconsistency in the style, placement, and behavior of controls (i.e., action buttons, filters, dropdowns, and links) throughout the application
• Vital signs “Out of Range” page-level alerts were not consistent or accurate
Additional issues:
• Disabled controls (i.e., Active checkbox for allergy status) caused significant inefficiencies for participants
• Inconsistency in the style, placement, and behavior of controls (i.e., action buttons, filters, dropdowns, and links) throughout the application
Areas for Improvement Recommendations for improvement include the following:
• Display orders and updates automatically
• Eliminate data loss of allergy reactions
• Lead users through multipage data entry screens: provide visual progress indicator; clearly identify primary action buttons; enable/disable secondary action buttons.
• Provide a confirmation message and a consistent control to close the window when participants complete the patient medication reconciliation
• Consider using suggestive search or autocomplete for search queries
• Consistently display “Out of Range” alerts
• Don't use a disabled control to display read-only information
• Ensure that controls have consistent style, placement, and behavior
Page 10 of 61
Introduction The EHR tested was Optum PM and Physician EMR version 8.1, which is an electronic health record (EHR) system offering comprehensive clinical documentation, workflow, and Meaningful Use compliance reporting.
The purpose of this test was to provide evidence of usability in the EHR and identify areas for incremental usability improvement. To this end, measures of effectiveness, efficiency, and user satisfaction (such as task completion, time on task, and task efficiency) were captured during the usability testing.
Method
Participants A total of 20 participants, consisting of 8 providers and 12 clinical support staff, were tested on the EHR. All participants were given the same orientation instructions. Providers and clinical support staff were compensated $250 and $150 respectively for their time.
Participants were recruited by Leede Research. Participants had no direct connection to the development of or organization producing the EHR. Participants were not from the testing or supplier organization. All participants currently use the EHR in a professional workplace and were recruited based on job title, gender, and age.
Page 11 of 61
Providers
The following table summarizes the characteristics of the providers who participated in the usability test, including demographics, professional experience, computing experience, and use of assistive technology. Participant names were replaced with Participant IDs so that data cannot be tied back to identities.
Part ID
Part ID Gender Age Education Occupation Professional
Experience Computer
Experience Product
Experience Assistive
Technologies
1 P4 Female 30-39 Doctoral degree Physician 4-6 years 10-19 years 1-3 years No
2 P20 Female 50-59 Professional degree (e.g. MD)
Physician 10-19 years 20-29 years 10 or more years
7 P27 Male 60-69 Doctoral degree Physician 20-29 years 4-6 years 4-6 years No
8 P28 Male 50-59 Professional degree (e.g. MD)
Physician 20-29 years 4-6 years 4-6 years No
Twelve providers were originally recruited, including two pilots.
• Three were no-shows • One was discontinued due to technical problems at the participants location
Participants were scheduled for 60-minute sessions. A minimum of 60 minutes was reserved between each session for debrief by the administrator and data logger and to reset systems to proper test conditions. A spreadsheet was used to track the participant schedule.
Page 12 of 61
Clinical Support Staff
The following table summarizes the characteristics of the Clinical Support Staff who participated in the usability test, including demographics, professional experience, computing experience, and use of assistive technology. Participant names were replaced with Participant IDs so that data cannot be tied back to identities.
Part ID
Part ID Gender Age Education Occupation Professional
Experience Computer
Experience Product
Experience Assistive
Technologies
1 P6 Female 18-29 Some college, no degree
Medical Assistant
7-9 years 7-9 years 1-3 years No
2 P7 Female 40-49 Technical certification
Licensed Practical Nurse
20-29 years 10-19 years 1-3 years No
3 P8 Female 30-39 Technical certification
Licensed Vocational Nurse
10-19 years 10-19 years 1-3 years No
4 P9 Female 18-29 Technical certification
Medical Assistant
7-9 years 10-19 years 4-6 years No
5 P10 Female 40-49 Some college, no degree
Medical Assistant
10-19 years 20-29 years 10 or more years
No
6 P11 Female 30-39 Technical certification
Medical Assistant
7-9 years 10-19 years 1-3 years No
7 P12 Female 30-39 Some college, no degree
Medical Assistant
10-19 years 7-9 years 1-3 years No
8 P13 Female 50-59 Master’s degree
Registered Nurse
30-39 years 7-9 years 4-6 years No
9 P17 Female 50-59 High School, GED, or equivalent
Office Administrator
10-19 years 20-29 years 7-9 years No
10 P18 Female 40-49 Associate’s degree
Physician’s Assistant
4-6 years 4-6 years 1-3 years No
11 P19 Male 40-49 Bachelor’s degree
Medical Assistant
7-9 years 1-3 years 1-3 years No
12 P14 Female 50-59 Professional degree
Licensed Practical Nurse
30-39 years 20-29 years 1-3 years No
Page 13 of 61
A total of 14 clinical support staff was recruited for the test. Two participants failed to show for the study. Test results for 12 clinical support staff are included in this report.
Participants were scheduled for 60-minute sessions. A minimum of 60 minutes was reserved between each session for debrief by the administrator and data logger and to reset systems to proper test conditions. A spreadsheet was used to track the participant schedule.
Study Design
Overall, the objective of this test was to measure how well the application performed with regard to effectiveness, efficiency, and user satisfaction; to identify areas for improvement; and to recommend improvements. Additionally, the data from this test will serve as a baseline for future tests and for comparison with updated versions of the EHR and/or other EHRs. In short, this testing serves as a means to benchmark current usability and to identify areas where improvements can be made.
All participants were given the same orientation instructions and remote testing environment to access the EHR. Two task sets were defined—one for providers and one for clinical support staff. All participants used the same test data. The EHR was evaluated using the following measures:
• Number of tasks successfully completed within the allotted time, without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant’s verbalizations (comments)
• Participant’s satisfaction ratings of the tasks and system
Additional information about the various measures can be found in the Usability Metrics section of this document.
Page 14 of 61
Tasks Tasks were selected based on how frequently they are used, how critical they are, and how likely they are to be troublesome for users. Task details were defined to be realistic and representative of the kinds of activities users might perform with the EHR.
6 Approve a medication order with a drug-allergy reaction
7 Approve a medication order with a drug-drug interaction
§170.314(b)(3) Electronic prescribing
16 Configure pharmacy for e-prescription
§170.314(b)(4) Clinical information reconciliation
19 Access reference clinical information
§170.314(a)(8) Clinical decision support
20 Activate a Patient Care Management alert
Page 15 of 61
Clinical Support Staff
Clinical Support Staff performed the following tasks:
Area of Certification Tasks
§170.314(a)(6) Medication list
8 Find a list of active and inactive home medications
9 Enter active medications
10 Update a medication currently being taken
11 Inactivate an active medication
§170.314(a)(7) Medication allergy list
12 Find a list of active and inactive allergies
13 Enter active allergies
14 Update the effects of an allergy
15 Inactivate an allergy
§170.314(b)(4) Clinical information reconciliation
17 Reconcile a patient medication list
18 Trigger a Patient Care Management alert
Procedures Upon arrival, participants were greeted and their identity was verified with the participant schedule. Participants were then assigned a participant ID.3 Each participant digitally signed a non-disclosure and informed consent form. See Appendix 2: Non-Disclosure Agreement and Informed Consent Form for an example of the form.
The test was conducted by two staff members, the administrator, and the data logger. The individuals conducting the test were usability practitioners with over 15 years of experience and educational backgrounds in Human Factors.
The administrator moderated the session, delivered initial instructions, and took notes on participant comments, path deviations, and errors. The data logger also took notes on task success, path deviations, errors, and comments. The testing tool (UserZoom) led participants through the tasks. UserZoom recorded task times, post-task rating data, post-test satisfaction data, and demographic data.
Participants were instructed to perform the tasks (see the Participant Instructions below):
• As quickly as possible making as few errors and deviations as possible
3 The participants’ data has been de-identified to be kept confidential.
Page 16 of 61
• Without assistance
• Without commenting until they had completed each task (if possible)
UserZoom provided participants with instructions before and during each task. Tasks were presented to each participant in a randomized order to avoid learning bias. Task timing began once the participant clicked the “Start Task” button in UserZoom. Task timing ended once the participant clicked the “Completed Task” button in UserZoom. Scoring is discussed in the Data Scoring section.
Following the session, UserZoom displayed the post-test survey, including a System Usability Scale (SUS) questionnaire and demographic questions. See Appendix 4: System Usability Scale Questionnaire for additional information on SUS questionnaires. The administrator told participants to expect to receive compensation in 2-3 weeks.
Test Location Tests were conducted remotely, with each individual at their own location. Only the participant, administrator, and data logger were present.
Test Environment The EHR typically is used in a clinical setting. Tests were conducted remotely, using the administrator’s computer to run WebEx, and TechSmith Camtasia (for recording). Participants used their own PCs during studies. They were given control of the administrator’s computer to access the application during the session, but used their own mouse, keyboard, and display.
Additionally, participants were also were asked to close any applications other than the WebEx session.
The administrator’s computer is an HP EliteBook 8570w laptop with a Core i7 Processor, 180GB SSD hard drive, 16GB RAM, running the Windows 7 Basic Theme. The display resolution was set to 1440 x 900. This computer ran WebEx screen sharing, Camtasia, and Internet Explorer 8.0 at 118% zoom level.
Test Forms and Tools During the usability test, UserZoom presented various documents, including:
1. Informed Consent
Participants were emailed the Informed Consent prior to session so they could review it. At thebeginning of session, they were asked to indicate in UserZoom if they agreed or disagreed with theterms.
2. Test instructions
3. Post-task Questionnaire
4. Post-test Questionnaire
5. Participant Demographics Questionnaire
Examples of these documents can be found in Appendix 2: Non-Disclosure Agreement and Informed Consent Form, Appendix 3: Example Moderator’s Guide, and Appendix 4: System Usability Scale Questionnaire. The Moderator’s Guide was designed to be easy for the data logger.
The script for participant orientation follows:
The participant's PC information and connection type information were not collected.
Page 17 of 61
Participant Instructions Thank you for joining us today; we really appreciate your time. I’m John Romadka; I work at Optum, makers of Optum PM and Physician EMR. A research assistant is also on the on line, and they’ll be helping take notes.
Before we begin, I’d like to take a moment to describe the study we’re conducting.
We’re collecting feedback on a software product called Optum PM and Physician EMR.
I know someone already asked you this, but I just wanted to verify that you currently use Optum PM and Physician EMR to work with patient records.
We’re going to ask you to perform a series of tasks. You should know that the tasks were created to be independent of each other and some may sound redundant. You will be asked to complete these tasks on your own, trying to do them as quickly as possible with the fewest possible errors or deviations. Please do not do anything more than the task asks.
You are not being tested on your ability to complete the tasks. We’re evaluating how intuitive the product is to use, so just do your best to complete them as you typically would. We didn’t have anything to do with designing the product, and your comments are entirely anonymous, so please feel free speak candidly. However, please save your comments until the end of a task, or the end of the session as a whole, when we can then discuss them freely.
We’re going to be using a study tool that will lead you through the tasks. We’d ask that you please read the task information aloud, so we can follow along.
I’m going to share my screen with you in the WebEx. You’ll be using Optum PM and Physician EMR on my computer, but I’ll give you control of my computer’s mouse and keyboard for the duration of the study. Because we can’t always control the bandwidth between our two locations, you may notice a tiny lag or latency when you’re in control of the mouse and keyboard. However, we don’t anticipate this will affect your ability to use Optum PM and Physician EMR.
Before I do that, please quit or close any other applications that are currently running on your computer (email and Instant Messaging), with the exception of your browser and WebEx. Also, please set any mobile devices to silent during the study.
Participants were then presented with tasks one at a time, randomly assigned by UserZoom. See Tasks for a complete list of tasks.
Usability Metrics According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency, and user satisfaction were captured during the usability testing.
The goals of the test were to assess:
1. Effectiveness of the EHR (Optum PM and Physician EMR) by measuring participant success rates and errors
2. Efficiency of the EHR by measuring the average task time and path deviations
Page 18 of 61
3. Satisfaction with the EHR by measuring ease-of-use ratings
Data Scoring The following table details how the testing team scored the tasks, evaluated the errors, and analyzed the time data.4
Measures Rationale and Scoring
Effectiveness:
Task Success
A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted.
The total number of successes were calculated for each task, and then the results were divided by the total number of times that task was attempted. The results are provided as a percentage.
Effectiveness:
Task Failures
If the participant abandoned the task, did not reach the correct answer, or performed it incorrectly, the task was counted as a “Failure.” No task times were taken for errors5.
The total number of errors was calculated for each task, and the results were divided by the total number of times the task was attempted. This was also expressed as the mean number of failed tasks per participant.
Efficiency:
Task Deviations
The participant’s path (i.e., steps) through the application was recorded. Deviations occurred if the participant went to a wrong screen, clicked an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. Each participant’s path was compared to the optimal path. The number of steps in the observed path was divided by the number of optimal steps to provide a ratio of path deviation.
Deviations were defined as:
• Task Recognition: failure to recognize task or completion of a different task
• Data Entry: failure to enter complete data or entry of incorrect data
• Task Completion: failure to take final steps required to complete the task
Efficiency:
Task Time
Each task was timed from when the participant clicked the “Start Task” button until the participant clicked the “Task Complete” button. Only completion times for successfully completed tasks were included in the “average task time analysis.” Average times per task and variance measures (standard deviation and standard error) were calculated for each task.
Optimal task performance time was benchmarked by expert performance when constructing the tasks. Target task times were operationally defined by taking
4 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman. Also, see www.measuringusability.com. 5 Errors were operationally defined by the UX team prior to testing.
multiple measures of optimal performance and multiplying by a factor (1.25) that allows a time buffer because the participants are using the EHR without their normal customizations. Thus, if expert optimal performance on a task was 100 seconds, then allotted task time performance would be 125 seconds. This ratio was aggregated across all the tasks and reported with mean and variance scores.
Satisfaction:
Task Rating
Participant’s subjective impression of the ease-of-use of the application was measured by administering a post-task question and a post-session questionnaire.
After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants.6 Common convention is that average ratings for systems judged easy to use should be 3.3 or above.
To measure participants’ confidence in and satisfaction with the EHR overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system frequently”, “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.” See Appendix 4: System Usability Scale Questionnaire for a complete list of SUS questions.7
Results
Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the preceding Usability Metrics section. We excluded data for one participant on task 17, because they failed to print out the necessary help documentation required to complete the task.
The usability testing results for the EHR are detailed below. The results are presented in the context of the objectives and goals outlined in Study Design. The report includes actionable results that, if corrected, should yield a material, positive impact on user performance.
The following table presents data for providers.
6 See Tedesco and Tullis (2006) for a comparison of post-task ratings for usability tests. Tedesco, D. & Tullis, T. (2006) A comparison of methods for eliciting post-task subjective ratings in usability testing. Usability Professionals association Conference, June 12 – 16, Broomfield, CO. 7 The SUS survey yields a single number that represents a composite measure of the overall perceived usability of the system. SUS scores have a range of 0 to 100 and the score is a relative benchmark that is used against other iterations of the system.
Page 20 of 61
Providers
Measure N Task
Success Path
Deviation Task Time Errors Task
Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
1. Enter laboratory and radiology/imaging orders 8 75%
(±46%) 1.00 168 (±75) 86% 25%
(±46%) 4.5
(±53%)
2. Enter medication orders 8 100%
(±0%) 1.07 165 (±67) 75% 0%
(±0%) 4.8
(±46%)
3. Discontinue a medication order 8 100%
(±0%) 3.38 103 (±39) 113% 0%
(±0%) 3.8
(±71%)
4. Update a medication order 8 88%
(±35%) 1.63 95 (±51) 100% 13%
(±35%) 4.3
(±71%)
5. Update a radiology order 8 88%
(±35%) 1.20 85 (±23) 89% 13%
(±35%) 4.3
(±46%)
6. Approve a medication order with a drug-allergy reaction
8 88% (±35%) 1.08 123
(±79) 87% 13% (±35%)
4.5 (±53%)
7. Approve a medication order with a drug-drug interaction
8 88% (±35%) .99 79
(±27) 92% 13% (±35%)
4.3 (±139%)
16. Configure pharmacy for e-prescription
8 100% (±0%) .98 124
(±35) 97% 0% (±0%)
4.6 (±52%)
19. Access reference clinical information 8 63%
(±52%) 1.29 78 (±97) 84% 38%
(±52%) 3.8
(±116%)
20. Activate a Patient Care Management alert 8 38%
(±52%) 1.09 119 (±81) 91% 63%
(±52%) 1.9
(±99%)
The SUS scored participants’ subjective satisfaction with the system based on their overall experience with the EHR. The overall score was 86 (±15). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average. 8
8 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).
Page 21 of 61
Discussion of the Findings
Findings for each major area’s research question based on the findings.
§170.314(a) (1) Computerized provider order entry
Can users order medication, laboratory, and radiology / imaging orders? (Task 1-2)
• Task 1 (Entering lab/radiation/imaging orders) success rate was 75%. Two errors occurred: one (of two) orders wasn’t added, and a wrong order was added.
• Task 2 (Entering a medication order) success rate was 100%.
Can a user change an order? (Task 3-5)
• Task 3 (Discontinue medication order) success rate was 100%. However, 75% of participants took the additional step to add penicillin as an allergy after discontinuing it, causing more than three times the path deviations (3.4) and a task time 13% longer than optimal. This isn’t an error because it’s a common practice.
Can users order a medication with a drug-allergy interaction? (Task 6)
• 37% of participants had to select repeatedly an override reason because the dropdown disappeared when their mouse briefly moved off it.
Can a user order a medication with a drug-drug interaction? (Task 7)
• One participant couldn’t complete the task because the application behaved erratically. It displayed black backgrounds, missing buttons and an unprompted session timeout notice.
• 37% of participants had to select repeatedly an override reason because the dropdown disappeared when their mouse briefly moved off it.
§170.314(b) (3) Electronic prescribing
Can users issue e-prescriptions? (Task 16)
• The task success rate was 100% (highest), path deviations were 1.0 (lowest), and zero errors.
§170.314(b) (4) Clinical information reconciliation
Can users access reference clinical information? (Task 19) • The task success rate was 63%, error rate was 38%, and the task satisfaction rating was 3.8. This
was primarily because the terminology used for the referenced clinical information link (i.e., Diagnosis info) didn’t match participants’ expectations.
§170.314(a) (8) Clinical decision support
Can a user activate a patient management alert? (Task 20) • The task success rate was 38%, the error rate was 63% (highest), and the satisfaction rating was
1.9 (lowest), because they couldn’t find and activate a Patient Care Management alert.
Page 22 of 61
• This is not a frequently performed task, and participants either couldn’t recall where to find it or gave up without attempting it.
• Participants who were successful (37%), repeatedly clicked on the Save button because there was no confirmation it was saved.
Effectiveness
Findings based on success, failure, and path deviation data.
Task 1
• 25% of participants either missed one (of two) order, or entered the wrong order.
Task 2
• Success rate was 100%.
Tasks 3-5
• Task 3: 75% of participants took the additional step to add penicillin as an allergy after discontinuing it, causing three times the path deviations.
Task 6
• 37% of participants had to select repeatedly an override reason because the dropdown disappeared when their mouse briefly moved off it.
Task 16
• Success rate was 100% (highest), path deviations were 1.0 (lowest).
Task 19
• The task success rate was 38%. Participants couldn’t find and activate a Patient Care Management alert.
Efficiency
Findings based on task time and deviation data.
Participants performed all tasks, except task 3, more efficiently than the optimal time.
Task 3
• The task time deviations were 13% longer than optimal. This is primarily because 75% of participants took the additional step to add penicillin as an allergy after discontinuing it.
Satisfaction
Findings based on task ratings and SUS results data.
Participants average task ratings were above the conventional average score (3.3), with the exception of task 20.
Task 20
• Task satisfaction was 1.9 (lowest). Participants couldn’t find and activate a Patient Care
Page 23 of 61
Management alert. This is not a frequently performed task. They couldn’t recall where to find the alert, or they gave up without attempting the task.
Major Findings Findings are based on quantitative data, verbal reports of the participants, and observations from the administrators and research assistants.
Major findings include:
• Orders and updates did not automatically refresh
• Allergy reaction data entry was lost
• Low completion rate for reconciliation of a patient medication list, even when participants had the help documentation
• Multipage data entry screens did not indicate where the participants were in the CCDA Reconciliation process contributed to low task success
• Multiple competing action buttons (e.g., Reconcile, Import, Next) within the CCDA Reconciliation page contributed to low task success
• No assistance or suggestions for misspelled medication search queries
• Inconsistency in the style, placement, and behavior of controls (i.e., action buttons, filters, dropdowns, and links) throughout the application
• Vital signs “Out of Range” page-level alerts were not consistent or accurate
• As well as issues with: • Disabled controls (i.e., Active checkbox for allergy status) caused significant inefficiencies for
participants
• Inconsistency in the style, placement, and behavior of controls (i.e., action buttons, filters, dropdowns, and links) throughout the application
Verbal reports include:
• Users are currently unable to see patients past medical history.
• Two participants reported there were substantial lags in performance. o “Every time I click on a tab, there will be an 8-10 second delay. I’m very disappointed with
the performance in any other browser version other than IE8. Even though they claim they support IE10, it’s simply no usable.”
o “Anytime you log in and move from the progress notes from one tab to the other, it takes 8-10 seconds. For a physician who is seeing 20-25 patients a day and has to wait 8-10 seconds after every click adds up to a lot of wasted time.”
• The upgrade last December caused a major disruption to one participant practice. o “I was the first on the West coast to complete the Meaningful Use using the iPad, and it was
working perfectly before the upgrade in December, but now it’s not working and has totally disrupted my practice. It makes me very angry that the response from support is that the application doesn’t support the iPad, because it was working perfectly for the last four years. I was promised that on March 8, it was going to be fixed, and then it was going to be another month after that it was going to be fixed. Then it was supposed to come out in the next
Page 24 of 61
release in June, and now I’ve heard that it’s not even going to be available until August.” o “Currently, you cannot import previous labs, x-rays, and issues into the notes (other than just
for that day). I have to hire a person to manually enter this data.” • Miscellaneous verbalizations
o “Kudos. The software is robust enough to allow us to meet the Meaningful Use 2. I don’t know any other practice in our area that is using other software has met the Meaningful Use 2 criteria.”
o “When you put in the (patient) medical history, anytime there’s a chief complaint, it should be bold to allow us to easily scan and spot it in the text.”
o “When you enter the assessment of the diagnosis, it would be really nice if there’s an ICD-9 code next to the diagnosis, rather than just the code itself.”
o “Database for medications and doctors isn’t updated frequently enough. For example, we have a physician that has been out of his position for a year, but he still shows up in the database (in his old position). The medications database has the same issues. Medications that have been approved a couple months ago don’t show up in the database.”
Page 25 of 61
Clinical Support Staff
Part ID
Part ID Gender Age Education Occupation Professional
Experience Computer
Experience Product
Experience Assistive
Technologies
1 P6 Female 18-29 Some college, no degree
Medical Assistant
7-9 years 7-9 years 1-3 years No
2 P7 Female 40-49 Technical certification
Licensed Practical Nurse
20-29 years 10-19 years 1-3 years No
3 P8 Female 30-39 Technical certification
Licensed Vocational Nurse
10-19 years 10-19 years 1-3 years No
4 P9 Female 18-29 Technical certification
Medical Assistant
7-9 years 10-19 years 4-6 years No
5 P10 Female 40-49 Some college, no degree
Medical Assistant
10-19 years 20-29 years 10 or more years
No
6 P11 Female 30-39 Technical certification
Medical Assistant
7-9 years 10-19 years 1-3 years No
7 P12 Female 30-39 Some college, no degree
Medical Assistant
10-19 years 7-9 years 1-3 years No
8 P13 Female 50-59 Master’s degree
Registered Nurse
30-39 years 7-9 years 4-6 years No
9 P17 Female 50-59 High School, GED, or equivalent
Office Administrator
10-19 years 20-29 years 7-9 years No
10 P18 Female 40-49 Associate’s degree
Physician’s Assistant
4-6 years 4-6 years 1-3 years No
11 P19 Male 40-49 Bachelor’s degree
Medical Assistant
7-9 years 1-3 years 1-3 years No
12 P14 Female 50-59 Professional degree
Licensed Practical Nurse
30-39 years 20-29 years 1-3 years No
The SUS scored participants’ subjective satisfaction with the system based on their overall experience with the EHR. The overall score was 78 (±13). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.9
9 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).
Page 26 of 61
Discussion of the Findings
Findings for each major area’s research question based on the findings.
§170.314(a) (6) Medication list
Can a user find a list of active and inactive medications? (Task 8)
• 25% of participants didn’t successfully complete this task because they couldn’t find “All” or believed that all the medications were already displayed.
Can a user enter active medications? (Task 9)
• The success rate was 58%. This is primarily due to data entry errors with the medication dose or frequency. The path deviations were 2.3 (highest) because 42% of the participants searched for medications rather than using the Favorites list.
• One participant misspelled Pepcid (twice as "Pepsid"). When the search didn't provide results, the participant created a new unscreened medication.
Can a user update active medications? (Task 10-11)
• The success rate for both tasks was 100% (highest), and zero errors (lowest). • Task 10 (Update medication) path deviations were 2.0, primarily because 50% of the participants
didn’t know how to complete the task efficiently. • Task 11 (Inactivate medication) satisfaction score was 4.8 (highest), and path deviations were 1.1
(lowest).
§170.314(a) (7) Medication allergy list
Can a user find a list of active and inactive allergies? (Task 12)
• 50% of participants didn’t complete this task. This was primarily because they didn’t click on the “Show Inactive” checkbox.
Can a user enter active allergies? (Task 13)
• 83% of participants successfully completed this task. However, the task was 110% longer than expected, because participants searched for each allergy, rather than using Favorites. The error rate was 50%. This was primarily due to one participant who wasn’t aware that entering end dates would inactivate the allergies.
Can a user update active allergies? (Task 14-15)
• Task 14 (Update allergy effects) success rate was 83%.
• Task 15 (Inactivate an allergy) success rate was 92%. However, the path deviations were 1.6, and the task time was 64% greater than expected. This was primarily because 58% of the participants attempted to inactivate the allergy by clicking on the “Active” (read-only) checkbox.
§170.314(b) (4) Clinical information reconciliation
Can a user reconcile clinical information? (Task 17) • This was a new feature, and required multipage data entry. Participants were given a printed copy of
the Help documentation specific to this task prior to the study session. One participant didn’t print it
Page 27 of 61
out before their session, and their data was not included for this task. • The task success rate was 18% (lowest). The error rate was 100%. The satisfaction rating was 3.0
(lowest), and the task time was 304% greater than expected. Task times were significantly longer than expected because the optimal time was based on experts who weren’t performing the task for the first time.
• This was because participants primarily struggled with the path navigation, button order, and importing the correct data.
Can users trigger a Patient Care Management alert? (Task 18) • 67% of participants successfully completed this task. Those who didn’t complete it stated that they
weren’t familiar with how to perform it.
Effectiveness
Findings based on success, failure, and path deviation data.
Task 8
• 25% of participants didn’t successfully complete this task because they couldn’t find “All.”
Task 9
• The success rate was 58%, and there were 2.3 more path deviations than optimal. This was primarily due to data entry errors with the medication dose or frequency.
Task 10
• The success rate was 100% (highest), with zero errors (lowest).
Task 11
• The success rate was 100% (highest), with zero errors (lowest). Path deviations were 2.0, primarily because 50% of the participants didn’t know how to complete the task efficiently.
Task 12
• 50% of participants didn’t complete this task. This was primarily because they didn’t click on the “Show Inactive” checkbox.
Task 15
• The success rate was 92%. However, the path deviations were higher than optimal (1.6), and the task time was 64% longer than expected. This was primarily because 58% of the participants attempted to inactivate the allergy by clicking on the “Active” (read-only) checkbox.
Task 17
• The task success rate was 18% (lowest), primarily because participants struggled with path navigation, button order, and importing the correct data.
Task 18
• 67% of participants successfully completed this task. Those who didn’t complete it stated that they
Page 28 of 61
weren’t familiar with how to perform it.
Efficiency
Findings based on task time and deviation data.
Task 9
• The task time was 46% longer than expected, primarily because participants searched for the medication, rather than using Favorites.
Task 13
• The task time was 110% longer than expected, primarily because participants searched for the allergies, rather than using Favorites.
Task 15
• The task time was 64% longer than expected. This was primarily because 58% of the participants attempted to inactivate the allergy by clicking on the “Active” (read-only) checkbox.
Task 17
• The task time was 304% longer than optimal. The optimal time was based on experts who weren’t performing the task for the first time.
Satisfaction
Findings based on task ratings and SUS results data.
Participants average ratings for systems were above the conventional average score (3.3), with the exception of task 17.
Task 17
• The task satisfaction was 3.0 (lowest). Participants were performing the task for the first time, and struggled with path navigation, button order, and importing the correct data.
Major Findings • Orders and updates did not automatically refresh
• Allergy reaction data entry was lost
• Low completion rate for reconciliation of a patient medication list, even when participants had the help documentation
• Multipage data entry screens did not indicate where the participants were in the CCDA Reconciliation process contributed to low task success
• Multiple competing action buttons (e.g., Reconcile, Import, Next) within the CCDA Reconciliation page contributed to low task success
• No assistance or suggestions for misspelled medication search queries
• Inconsistency in the style, placement, and behavior of controls (i.e., action buttons, filters, dropdowns, and links) throughout the application
Page 29 of 61
• Vital signs “Out of Range” page-level alerts were not consistent or accurate
Additional issues:
• Disabled controls (i.e., Active checkbox for allergy status) caused significant inefficiencies for participants
• Inconsistency in the style, placement, and behavior of controls (i.e., action buttons, filters, dropdowns, and links) throughout the application
Areas for Improvement Recommendations for improvement include the following:
• Display orders and updates automatically
• Eliminate data loss of allergy reactions
• Lead users through multipage data entry screens: provide visual progress indicator; clearly identify primary action buttons; enable/disable secondary action buttons.
• Provide a confirmation message and a consistent control to close the window when participants complete the patient medication reconciliation
• Consider using suggestive search or autocomplete for search queries
• Consistently display “Out of Range” alerts
• Don't use a disabled control to display read-only information
• Ensure that controls have consistent style, placement, and behavior
Page 30 of 61
Combined (Provider and Clinical Support Staff) Areas for Improvement
• Display orders and updates automatically
• Help user distinguish between similarly labeled items in the favorite orders list
• Ensure terminology used for referenced clinical information aligns with users’ expectations
• Eliminate erratic application behavior that prevents users from performing their tasks
• Ensure that controls have consistent style, placement, and behavior
• Consistently provide confirmation messages
• Consider more effective ways to organize and display items in the Administrative section
• Indicate the required fields on data entry screens
• Align form elements with the associated text labels
• Eliminate data loss of allergy reactions
• Lead users through multipage data entry screens: provide visual progress indicator; clearly identify primary action buttons; enable/disable secondary action buttons.
• Provide a confirmation message and a consistent control to close the window when participants complete the patient medication reconciliation
• Consider using suggestive search or autocomplete for search queries
• Consistently display “Out of Range” alerts
• Don't use a disabled control to display read-only information
Appendices The following appendices include supplemental data for this usability test report. Following is a list of the appendices provided:
1. Participant demographics 2. Non-Disclosure Agreement (NDA) and Informed Consent Form 3. Example Moderator’s Guide 4. System Usability Scale Questionnaire 5. Tasks 6. Demographics Questionnaire
Page 31 of 61
Appendix 1: Participant Demographics Following is a high-level overview of the participants in this study.
Providers
The following table summarizes the characteristics of the providers that participated in the usability test, including demographics, professional experience, computing experience and use of assistive technology. Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.
Part ID
Part ID Gender Age Education Occupation Professional
Experience Computer
Experience Product
Experience Assistive
Technologies
1 P4 Female 30-39 Doctoral degree Physician 4-6 years 10-19 years 1-3 years No
2 P20 Female 50-59 Professional degree (e.g. MD)
Physician 10-19 years 20-29 years 10 or more years
7 P27 Male 60-69 Doctoral degree Physician 20-29 years 4-6 years 4-6 years No
8 P28 Male 50-59 Professional degree (e.g. MD)
Physician 20-29 years 4-6 years 4-6 years No
Page 32 of 61
Clinical Support Staff
The following table summarizes the characteristics of the Clinical Support Staff that participated in the usability test, including demographics, professional experience, computing experience and use of assistive technology. Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.
Part ID
Part ID Gender Age Education Occupation Professional
Experience Computer
Experience Product
Experience Assistive
Technologies
1 P6 Female 18-29 Some college, no degree
Medical Assistant
7-9 years 7-9 years 1-3 years No
2 P7 Female 40-49 Technical certification
Licensed Practical Nurse
20-29 years 10-19 years 1-3 years No
3 P8 Female 30-39 Technical certification
Licensed Vocational Nurse
10-19 years 10-19 years 1-3 years No
4 P9 Female 18-29 Technical certification
Medical Assistant
7-9 years 10-19 years 4-6 years No
5 P10 Female 40-49 Some college, no degree
Medical Assistant
10-19 years 20-29 years 10 or more years
No
6 P11 Female 30-39 Technical certification
Medical Assistant
7-9 years 10-19 years 1-3 years No
7 P12 Female 30-39 Some college, no degree
Medical Assistant
10-19 years 7-9 years 1-3 years No
8 P13 Female 50-59 Master’s degree
Registered Nurse
30-39 years 7-9 years 4-6 years No
9 P17 Female 50-59 High School, GED, or equivalent
Office Administrator
10-19 years 20-29 years 7-9 years No
10 P18 Female 40-49 Associate’s degree
Physician’s Assistant
4-6 years 4-6 years 1-3 years No
11 P19 Male 40-49 Bachelor’s degree
Medical Assistant
7-9 years 1-3 years 1-3 years No
12 P14 Female 50-59 Professional degree
Licensed Practical Nurse
30-39 years 20-29 years 1-3 years No
Page 33 of 61
Appendix 2: Non-Disclosure Agreement and Informed Consent Form Understanding Your Participation
I agree to participate in this study conducted by Optum.
I understand Optum will record my participation for later review. By signing/acknowledging this form, I give Optum permission to use the data gathered and my comments for the purposes of evaluating the product in the study and showing the results of these evaluations, combined with other participants, to the product team. Optum will not include my name or any identifying characteristics in the results.
Any information I acquire about the product in the study is confidential and proprietary and is being disclosed only so that I can participate in the evaluation. By signing this form, I agree not to talk about this product to anyone. I may only tell others that I helped to evaluate the software. I understand that if I disclose my health information in my comments, I have disclosed it voluntarily, and Optum may use it anonymously for research and evaluation purposes only.
I understand that I can stop participating in this study at any time.
I agree to raise any concerns or areas of discomfort with the study moderator.
Name (print): __________________________________________________
Appendix 3: Example Moderator’s Guide for Providers and Clinical Support Staff
Thank you for joining us today; we really appreciate your time. I’m John Romadka; I work at Optum, and we’re gathering information about a new version of Optum PM & Physician EMR. [Research Assistant’s name] is also on the on the line, and will be helping take notes.
Before we begin, I’d like to take a moment to describe the study we’re conducting.
We’re going to ask you to perform a series of tasks in Optum PM & Physician EMR. You should know that the tasks were created to be independent of each other and some may sound redundant. You will be asked to complete these tasks on your own, trying to do them as quickly as possible with the fewest possible errors or deviations. We ask that you please do not do anything more than asked in the task. For this study, all prescriptions will be printed rather electronically sent to the pharmacy.
You are not being tested on your ability to complete the tasks. We’re actually evaluating how intuitive the product is to use, so just do your best to complete them as you typically would. I’d ask that you please save your detailed comments until the end of a task or the end of the session when we can discuss freely.
I’d like to mention that we didn’t have anything to do with designing the product, and your comments are entirely anonymous, so please feel free speak candidly. We’ll be using a tool that will lead you through the study tasks and some survey questions. When you’re presented with each scenario and task, I’d ask that you please read this information aloud to help me follow along.
In a moment, I’m going to share my screen with you in the WebEx. You’ll be using Optum PM & Physician EMR on my computer, but I’ll give you control of my computers mouse and keyboard for the duration of the study.
I just should mention that because we can’t control the bandwidth between our two locations, you may notice just a tiny lag in the application, but it shouldn’t affect your ability to complete the tasks.
Before I do that, please quit or close any other applications other than your browser (including email and Instant Messaging) that are currently running on your computer.
Before we get started, do you have any questions?
I’ll make you the presenter and then ask you to share your desktop.
Page 35 of 61
Appendix 4: System Usability Scale Questionnaire In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, at http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
16 Brooke, J.: SUS: A “quick and dirty” usability scale. In: Jordan, P. W., Thomas, B., Weerdmeester, B. A., McClelland (eds.) Usability Evaluation in Industry pp. 189--194. Taylor & Francis, London, UK (1996). SUS is copyrighted to Digital Equipment Corporation, 1986.
Lewis, J R & Sauro, J. (2009) "The Factor Structure of the System Usability Scale." in Proceedings of the Human Computer Interaction International Conference (HCII 2009), San Diego CA, USA
There are multiple correct paths to complete the tasks. The “optimal path” is indicated in bold.
Task 1 – Enter laboratory and radiology/imaging orders Robert O’Brien came to the clinic for a scheduled appointment with you complaining of shortness of breath and a persistent cough.
Enter the following lab and radiology orders:
• CBC W auto differential panel in Blood • Chest x-ray (Radiologic examination, chest; single view, frontal)
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 195 seconds
Key Data Entries:
Medication Order 1 • Medication: Erythromycin Base Oral Tablet 500 mg • Dose amount: 1 • Unit: tab or tablet • Frequency: once a day • Duration: 14 days • Fill quantity: 14
Medication Order 2 • Medication: Ibuprofen oral tablet 800 mg • Dose amount: 1 • Unit: tab or tablet • Frequency: every 8 hours (as needed) • Duration: 5 days
Correct Paths (Optimal in Bold)
1. Open Robert Michael O’Brien’s medical record a. Navigate to the Clinical Today screen
i. Directly from the Clinical Today page select the patient > navigate to new tab with patient record
ii. Close clinical alert window OR
b. Search for patient (last name, first name) in search field in upper left hand corner i. Select patient in search results window ii. Select Medical Record in Health History Pain on left iii. Select Order button on quick task tool bar iv. Select current encounter
2. Enter orders. a. Enter lab order and then “Save and New” to add Radiology order
Page 37 of 61
i. Click on the Orders button on the Quick Tasks tool bar ii. Ensure Order Type is set to Lab
iii. Select radio button for CBC W auto differential panel in Blood iv. Search for diagnosis
1. Cough or shortness of breath are acceptable v. Select the appropriate diagnosis from search results
vi. Click Save and New vii. Set Order Type to Diagnostic Imaging
viii. Search for test ix. Select Radiologic examination, chest; single view, frontal
1. (Optimal time used: Chest) x. Search for diagnosis
1. Shortness of breath xi. Click Add Test xii. Click Save
OR b. Enter each test individually using “Save” button
i. Click on the Orders button on the Quick Tasks tool bar ii. Ensure Order Type is set to Lab iii. Search for CBC W auto differential panel in Blood iv. Click Add Test v. Click Save. vi. Click on the Orders button on the Quick Tasks tool bar vii. Ensure Order Type is set to Diagnostic Imaging viii. Search for chest x-ray (Radiologic examination, chest; single view, frontal) ix. Click Add Test x. Click Save.
Successful Completion
User clicks save after entering orders.
Completed: ___Yes ____No
Page 38 of 61
Task 2 – Enter medication orders Jessica Gilbert came to the clinic for a scheduled appointment with you (Dr. Wilson), complaining of ear pain.
You feel that the benefits of the drugs outweigh the risk of all drug interactions. Enter the following medication orders for the patient:
• Erythromycin Base Oral Tablet 500 mg, once daily by mouth, for 14 days • Ibuprofen oral tablet 800 mg, 1 tablet, every eight hours as needed, by mouth for 5 days
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 219 seconds
Key Data Entries:
Medication Order 1 • Medication: Erythromycin Base Oral Tablet 500 mg • Dose amount: 1 • Unit: tab or tablet • Frequency: once a day • Duration: 14 days • Fill quantity: 14
Medication Order 2 • Medication: Ibuprofen oral tablet 800 mg • Dose amount: 1 • Unit: tab or tablet • Frequency: every 8 hours (as needed) • Duration: 5 days
Correct Paths (Optimal in Bold)
1. Open Jessica Gilbert’s medical record a. Navigate to the Clinical Today screen. b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alerts window
OR d. Search for patient (last name, first name) in search field in upper left hand corner
i. Select patient in search results window ii. Select Medical Record in Health History Pain on left iii. Select Order button on quick task tool bar iv. Select current encounter
2. Enter Medication Order a. Enter both orders together
i. Click on Rx button on Quick Task Toolbar ii. Search for Erythromycin
iii. Select Erythromycin Base Oral Tablet 500 mg from the favorites iv. Select an interaction screening override reason v. Fill in dose (1), duration (14 days), fill quantity (14), dose unit (tablet)
Page 39 of 61
vi. Click Add Rx vii. Search for Ibuprofen
viii. Select Ibuprofen oral tablet 800 mg ix. Select an interaction screening override reason x. Fill in dose, duration, fill quantity, dose unit and fill unit
xi. Click Add Rx xii. Click Complete
OR b. Enter each order separately
i. Click on Rx button on Quick Task Toolbar ii. Search for Erythromycin iii. Select Erythromycin Base Oral Tablet 500 mg iv. Fill in dose, duration, fill quantity, dose unit and fill unit v. Click Add Rx vi. Click Complete vii. Enter the provider pin viii. Click on Rx button on Quick Task Toolbar ix. Search for Ibuprofen x. Select Ibuprofen oral tablet 800 mg xi. Fill in dose, duration, fill quantity, dose unit and fill unit xii. Click Add Rx xiii. Click Complete
Successful Completion Criteria
Clicks complete after adding both prescriptions.
Completed: ___Yes ____No
Page 40 of 61
Task 3 – Discontinue a medication order Jennifer Soto came to the clinic for a scheduled appointment with you (Dr. Wilson), complaining of hives. A patient record has already been created for Jennifer and assigned to you. She had been prescribed penicillin two days ago for an ear infection.
Discontinue the penicillin prescription because of a probable allergic reaction.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 91 seconds
Correct Paths (Optimal in Bold)
1. Open Jennifer Soto medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alerts screen
OR a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left
2. Discontinue penicillin medication order a. Click on Medications HHP (Health History Pain) b. Check box next to penicillin order c. In change status drop down menu, choose “Discontinued”
OR a. Click on Medication Name b. Click on Medications HHP (Health History Pain) c. Click on medication name d. In medication window, change status in status drop-down menu to “Discontinued” e. Click “Save”
Successful Completion Criteria
Clicks Save after discontinuing penicillin.
Completed: ___Yes ____No
Page 41 of 61
Task 4 – Update a medication order Javier Knight came to the clinic for a scheduled appointment with you (Dr. Wilson), is still complaining of continuing back pain two weeks after you prescribed cyclobenzaprine HCL ER Oral capsule Extended Release 15 mg, 1 capsule daily.
Increase and update Javier’s medication order of cyclobenzaprine to 2 capsules daily.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 95 seconds
Correct Paths (Optimal in Bold)
1. Open Javier Knight’s medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Click on patient name
Close clinical alerts OR
a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left
2. Update cyclobenzaprine medication order a. Click on Medications HHP (Health History Pain) b. Click on “actions” button icon on cyclobenzaprine line item and select “Re-order” c. Prescription page opens, Change Dose to 2 d. Click “Add Rx” e. Click “Complete” f. Rx window appears to print g. Click print (however, not necessary for participants in task) h. Close Rx window
Successful Completion Criteria
User clicks Complete Completed: ___Yes ____No
Page 42 of 61
Task 5 – Update a radiology order John Smith came to the clinic for a scheduled appointment with you (Dr. Wilson). He is complaining of an injury to his right wrist. John needs a 2-view wrist x-ray, but a 2-view chest x-ray was accidentally ordered for him.
Update John’s record to reflect the fact that he needs a 2-view wrist x-ray, rather than chest x-rays.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 95 seconds
Correct Paths (Optimal in Bold)
1. Open John Smith’s medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close the clinical alert window
OR a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left d. Select Order button on quick task tool bar e. Select current encounter
2. Update order a. Click on Order/referrals HHP b. Click on order number c. Remove test (red x icon) (Or Cancel Order button) d. Click OK to remove order e. Search for new test, searching by wrist f. Select Radiologic exam, wrist 2-view g. Add Test h. Save
Successful Completion Criteria
Clicks Save
Completed: ___Yes ____No
Page 43 of 61
Task 6 – Approve a medication order with a drug-allergy reaction John Smith came to the clinic for a scheduled appointment with you (Dr. Wilson), complaining of an injury to his right wrist. John is allergic to Tylenol but has been treated with Vicodin in the past and has responded well with no side effects. Enter an order for John of Vicodin ES Oral Tablet 7.5 – 300 mg (CIII), one tablet, once a day, for five days. When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 141 seconds
Key Data Entries:
• Medication Order • Override reason (any of the following are acceptable):
o Med previously tolerated o Benefit outweighs risk o Patient has taken med without complication
• Medication: Vicodin ES Oral Tablet 7.5 – 300 mg (CIII) or the generic version: Hydrocodone-Acetaminophen Oral Tablet 7.5-300 MG (CIII)
o Dose amount: 1 o Unit: tab or tablet o Frequency: once a day o Duration: 5 days
Correct Paths (Optimal in Bold)
1. Open John Smith’s medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record which is now showing Chart Summary in HCC c. Close clinical alert OR a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left d. Select Order button on quick task tool bar e. Select current encounter
2. Enter Medication Order and override alert a. Click on Rx button on Quick Task Toolbar b. Select Vicodin ES Oral Tablet 7.5 – 300 mg (CIII) or Hydrocodone-Acetaminophen
Oral Tablet 7.5-300 MG (CIII) c. In interaction screening window, select override reason (Med previously tolerated
or Benefit outweighs risk or Patient has taken med without complication), and click “Accept”
d. Click Add Rx e. Click Complete
Successful Completion Criteria
Clicks Complete after adding the prescription. Completed: ___Yes ____No
Page 44 of 61
Task 7 – Approve a medication order with a drug-drug interaction Zackary A Pearson came to the clinic for a scheduled appointment with you. He has a recent history of atrial fibrillation and is currently taking a daily low dose of aspirin. You believe the benefits of taking Warfarin outweigh the risks associated with drug interactions.
Order Zackary one Warfarin Sodium Oral Tablet 10 Mg, once daily, for 30 days.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 86 seconds
Key Data Entries:
Medication Order • Override reason: Benefit outweighs risk • Medication: Warfarin Sodium Oral Tablet 10 Mg • Dose amount: 1 • Unit: tab or tablet • Frequency: once a day • Duration: 30 days
Correct Paths (Optimal in Bold)
1. Open Zackary A Pearson’s medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record OR
a. Search for patient (last name, first name) in search field in upper left hand corner c. Select patient in search results window d. Select Medical Record in Health History Pain on left e. Select Order button on quick task tool bar f. Select current encounter
2. Enter Medication Order and override alert a. Click on Rx button on Quick Task Toolbar b. Click the OK button c. Search for Warfarin d. Select one Warfarin Sodium Oral Tablet 10 Mg e. In interaction screening window, select override reason (benefit outweighs risk), and
click “Accept” f. Fill in dose amount (1), unit (tablet) frequency (once a day), duration (30) g. Click Add Rx h. Click Complete
Successful Completion Criteria
User clicks Complete after the orders are entered.
Task 16 – Select a pharmacy for e-prescription Janet Kikura came to the clinic for a scheduled appointment complaining of a severe migraine headache. You want to send an electronic prescription to Janet’s pharmacy so it will be available for her to pick up on her way home.
Using Optum PM & Physician EMR, write a prescription for Ibuprofen 800mg, 1 tablet, taken by mouth, every 8 hours as needed, for 10 days, no refills, 30 tablets to be dispensed. The closest pharmacy to Janet’s home is Walgreen’s, located on McKee Rd., in Madison, WI.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 128 seconds
Key Data Entries:
Medication Order • Medication: Ibuprofen Oral Tablet 800 MG • Dose amount: 1 • Unit: tab or tablet • Frequency: every 8 hours (as needed) • Duration: 10 days • Refills: No
Correct Paths (Optimal in Bold)
1. Open Zachary Pearson’s medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alert window d. Click on Rx icon e. Click the OK button
2. Add medication a. Select medication (Ibuprofen Oral Tablet 800 MG) from favorites b. Enter dose (1) tab, every 8 hours as needed, dispense 20 tablets c. Select pharmacy d. Click on search e. Select Walgreens f. Action selected as Send (prompt the participant to assume it is selected, and ask what
they’d do next.) g. Click Add Rx button h. Click complete button
Successful Completion Criteria
User clicks Complete after selecting medication and pharmacy. Completed: ___Yes ____No
Page 46 of 61
Task 19 – Access reference clinical information Susan Sanson came to your clinic today and has been diagnosed her with an ear infection. You have heard that Optum PM and Physician EMR now provides you a NEW way to access referenced clinical information for a specific diagnosis. Task: Find the linked clinical reference information in Susan Sanson’s problem list pertaining to her ear infection.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 93 seconds
Correct Paths (Optimal in Bold)
1. Problem list a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinic alert d. Select Problem List from the HCC e. Hover over action arrow at the far right of ear ache problem f. Click on Diagnosis Info g. Opens a window that is linked to Medline
OR 2. Patient Education
a. Launch the patient medical record b. Click Education button c. Search for problem (ideal criteria)
i. Search term: “ear infection” ii. Age: ALL (Note: if this isn’t changed to All, the search won’t return results)
d. Any document in search results under ENT category is acceptable.
Successful Completion Criteria
Clicks on Diagnosis Info link OR Patient Education, ENT
Completed: ___Yes ____No
Page 47 of 61
Task 20 – Activate a Patient Care Management alert You would like activate a Patient Care Management alert for Comprehensive diabetes care.
Activate a Patient Care Management Alert for comprehensive diabetes care for yourself.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 131 seconds
Correct Paths (Optimal in Bold)
1. Click on Admin HHP 2. Click on Clinical tab 3. Under Sys Admin column> Care Management Activation 4. Check box next to Comprehensive Diabetes care 5. Clicks save button
Successful Completion Criteria
Clicks Save button
Completed: ___Yes ____No
Page 48 of 61
Clinical Support Staff
There are multiple correct paths to complete the tasks. The “optimal path” is indicated in bold.
Task 8 – Find a list of active and inactive medications Amy Anderson came to the clinic for a scheduled appointment. You want to review a complete list of medications in her record so you can verify which ones she is taking.
Display a complete list of Amy’s active and inactive medications so that you can verify each medication with her.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 54 seconds
Correct Paths (Optimal in Bold)
1. Open Amy Anderson medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close the clinical alerts window
OR a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left
2. View Active and Inactive Medications a. Click on Medications HHP (Health History Pain) b. Click “All” next to the “View” filter at the top c. Should see Ibuprophen inactive
Successful Completion Criteria
Active and inactive (Ibuprophen) medications are visible.
Completed: ___Yes ____No
Page 49 of 61
Task 9 – Enter active medications Susan McCoy came to the clinic for a scheduled appointment regarding stomach pain. She was previously diagnosed with gastro esophageal reflux disease (GERD). Susan neglected to mention that she takes an over-the-counter medication regularly for GERD. Update Susan’s record to reflect the medication she takes Pepcid AC 10mg chewables, 2 daily. When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 51 seconds
Key Data Entries:
• Medication: Pepcid AC 10mg chewables or Famotidine Oral Tablet Chewable 10 MG • Dose amount: 2 • Unit: tab or tablet
Correct Paths (Optimal in Bold)
1. Open Susan McCoy’s record - a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alert window
OR a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left
2. Enter Medication a. Enter both medications together
i. Click on Medication HHP ii. Click on ‘+ Patient Med’ at top of screen
iii. Search for Pepcid iv. Select Pepcid AC 10mg chewables or Famotidine Oral Tablet Chewable 10 MG v. Fill in dose (2) unit (tab), frequency (twice a day) (OPTIONAL)
vi. Click ‘Save” vii. Pepcid shows in mediation list
OR b. Enter each order separately
i. Click on Medication HHP ii. Click on ‘+ Patient Med’ at top of screen iii. Search for Pepcid iv. Select Pepcid AC 10mg chewables v. Fill in dose, dose unit, frequency vi. Click ‘Save’ vii. Click on ‘+ Patient Med’ at top of screen viii. Click ‘Save’
Successful Completion Criteria
User clicks save Completed: ___Yes ____No
Page 50 of 61
Task 10 – Update a medication currently being taken Susan McCoy came to the clinic for a scheduled appointment regarding stomach pain. In a past visit, she said she took Metformin 500 mg, but neglected to say how often she took it. She mentioned today that she is taking one tablet of Metformin twice daily.
Update Susan McCoy’s record to reflect the fact she takes Metformin HCL Oral tablet 500 mg, one tablet, twice daily.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 118 seconds
Key Data Entries:
• Medication: Metformin HCL Oral tablet 500 m • Dose: 1 • Unit: tablet • Frequency: Twice a day
Correct Paths (Optimal in Bold)
1. Open Susan McCoy’s record - a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alert window
OR a. Search for patient (last name, first name) in search field in upper left hand corner a. Select patient in search results window b. Select Medical Record in Health History Pain on left
2. Update Medication a. Click on Medication HHP b. Click on Medication Name: Metformin HCL Oral tablet 500 mg c. Update Frequency to “Twice a Day” d. (OPTIONAL) unit 1, tab e. Click “Save”
Successful Completion Criteria
User clicks save after updating frequency.
Completed: ___Yes ____No
Page 51 of 61
Task 11 – Inactivate an active medication Heather Jones came to the clinic for a scheduled appointment regarding stomach pain. At a previous appointment, Heather said she took Zegerid OTC once a day. She now says she is no longer taking Zegerid.
Update Heather’s record to reflect the fact that she is no longer taking Zegerid.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 60 seconds
Correct Paths (Optimal in Bold)
1. Open Heather Jones’ medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alert window
OR a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left
2. Inactivate medication order a. Check box next to medication b. Click on Medications HHP (Health History Pain) c. Check box next to Zegerid d. In change status drop down menu, choose “Inactive”
OR a. Click on Medication Name b. Click on Medications HHP (Health History Pain) c. Click on medication name Zegerid d. In medication window, change status in status drop-down menu to “Inactive” e. Click “Save”
Successful Completion Criteria
User clicks save after inactivating a medication.
Completed: ___Yes ____No
Page 52 of 61
Task 12 – Find a list of active and inactive allergies Cara Banks came to the clinic for a scheduled appointment regarding a fever.
Display a complete list of Cara’s active and inactive allergies so that you can verify each allergy with her and her family.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 59 seconds
Correct Paths (Optimal in Bold)
1. Open Cara Banks medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alerts box
OR a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left
2. View allergies a. Click on Allergies HHP b. Check “Show Inactive Allergies” checkbox at top to display both active and inactive
allergies c. Septra is displayed as inactive
Successful Completion Criteria
User checks ‘Show Inactive Allergies’ box.
Completed: ___Yes ____No
Page 53 of 61
Task 13 – Enter active allergies Susan McCoy came to the clinic for a scheduled appointment, regarding stomach pain. Susan says she is allergic to several medications.
Update Susan’s record to reflect her allergies and reactions.
1. Open Susan McCoy’s medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alert window
OR a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left
2. Add allergies a. Click on Allergies HHP b. Click ‘+ Add Allergy” c. Select allergy from favorites, reaction: nausea, and alert (aspirin) d. Click ‘Save and New’ to add more allergies e. Select allergy from favorites, reaction: hives, and alert (Bactrim) f. Click ‘Save and New’ to add more allergies g. Select allergy from favorites, reaction, abdominal pain, and alert (Cipro) h. Click ‘Save and New’ to add more allergies i. After all allergies are entered, click ‘save’
OR j. Search for medication by clicking magnifying glass k. Select aspirin or adult aspirin l. Select reaction (nausea) and click outside of list, alert status (no alert) (no click) m. Click ‘Save and New’ to add more allergies
Successful Completion Criteria
User clicks save. Completed: ___Yes ____No
Page 54 of 61
Task 14 – Update the effects of an allergy Javier Knight came to the clinic for a scheduled appointment complaining of continuing back pain. He previously reported a penicillin allergy with a reaction of nausea. He now reports that the last time he took penicillin he experienced severe swelling of his throat.
Update Javier’s record to reflect the fact his reaction to penicillin is anaphylaxis.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 95 seconds
Key Data Entries:
• Allergy (Reaction) 1: penicillin (anaphylaxis)
Correct Paths (Optimal in Bold)
1. Open Javier Knight’s medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinic alert
OR a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left
2. Update allergy a. Click on Allergies HHP b. Click on penicillin allergy c. Change allergy reaction to anaphylaxis and alert status to Pop-up d. Click ‘save’ e. Reaction shows as anaphylaxis
Successful Completion Criteria
User clicks save after entering reaction details have been added to allergy.
Completed: ___Yes ____No
Page 55 of 61
Task 15 – Inactivate an allergy Zachary Pearson came to the clinic for a scheduled appointment complaining of heartburn. His record shows that he is allergic to valium. After reviewing allergies with Zachary, he tells you that he is not allergic to valium.
Update Zachary’s record to reflect the fact he is not allergic to valium.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 58 seconds
Correct Paths (Optimal in Bold)
1. Open Zachary Pearson’s medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alert window
OR a. Search for patient (last name, first name) in search field in upper left hand corner b. Select patient in search results window c. Select Medical Record in Health History Pain on left
2. Update allergy a. Click on Allergies HHP b. Click on Valium allergy c. Enter ‘reported end date’ as today d. Click ‘save’
Successful Completion Criteria
User enters reported end date and clicks save
Completed: ___Yes ____No
Page 56 of 61
Task 17 – Reconcile a patient medication list Last week, Holly Preston was admitted to the hospital for shortness of breath, and during her stay, she was given Proventil. Upon discharge, she was then given a prescription for Proventil and was scheduled for a follow-up appointment with her Primary Care Provider (PCP). The hospital sent a Continuity of Care Document (CCD) via direct mail to her primary care provider within your office.
Holly is now in your office for her scheduled follow-up appointment with her PCP, and you’ve heard that the Optum PM and Physician EMR application now provides a way to import medical information available in a CCD into a patient’s medical record.
Use the Help documentation that you were provided to import the CCD that includes the medication (Proventil) she was prescribed, and merge it with Holly Preston’s active medication list. (Note: The CCD has already been uploaded into Holly’s medical record and is now available to be imported.)
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 64 seconds
Correct Paths (Optimal in Bold)
1. Open Holly Preston’s medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alert window d. Documents in HHC e. Click on Plus icon to expand the documents folder f. Click on CDA import g. Click in checkbox for Medications (import into med records) h. Create new encounter i. Select type to Visit (or Historical Import) j. Click Next k. Click C_CDA Medication checkbox for Proventil l. Select Reconcile m. Click Import button n. Click Next (shows summary)
Successful Completion Criteria
Click next (after clicking Import button)
Completed: ___Yes ____No
Page 57 of 61
Help Documentation
The information below is the actual Help documentation from the Optum PM and Physician EMR application. It will be provided to the participants prior to the study session, and they can refer to it during the task.
Page 58 of 61
Page 59 of 61
Task 18 – Trigger a Patient Care Management alert Susan Sanson came to the clinic for a scheduled visit. She has been experiencing an earache for the past two days. You obtain her temperature and it is 103.1 Fahrenheit, and blood pressure is 143 over 80.
Record Susan Sanson’s temperature and blood pressure in her record. Identify and note which of the vital signs are out of normal range.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Optimal time: 178
Key Data Entries:
• Temperature: 103.1 • Blood Pressure: 143 over 80
Correct Paths (Optimal in Bold)
1. Open Holly Preston’s medical record a. Navigate to the Clinical Today screen b. Directly from the Clinical Today page select the patient > navigate to new tab with
patient record c. Close clinical alert window d. Click on vital signs HHP (on the vertical toolbar on the left hand side) e. Click Record Vitals f. Click ok to create the encounter g. Enter temperature
Successful Completion Criteria
Enters vital signs and correctly answers validation question.
Validation question: Which of the vital signs are out of normal range?
Answers:
1. Blood pressure only (UI indicates only this in alert text) 2. Blood pressure and Temperature (Correct Answer) 3. Temperature only 4. None of the vital signs
Completed: ___Yes ____No
Page 60 of 61
Appendix 6: Demographics Questionnaire 1. Gender
O Female O Male
2. Age
O 18-29 years O 30-39 years O 40-49 years
O 50-59 years
O 60-69 years O 70 or more years
3. What is the highest degree or level of school you have completed?
O High School, GED, or equivalent O Some college, no degree O Technical certification O Associate degree O Bachelor degree O Master degree O Professional degree O Doctorate degree O Other: (please specify)
_____________________________
4. What title best describes your current occupation?
O Emergency Medical Technician
O Health Information System Administrator
O Nurse Practitioner O Office Administrator O Pharmacist O Physician O Physician’s Assistant
O Psychologist, Counselor, or Social Worker
O Registered Nurse O Other: (please specify)
_____________________________
5. How long have you held your current title?
O Less than 1 year
Page 61 of 61
O 1-3 years
O 4-6 years
O 7-9 years O 10-19 years O 20-29 years O 30-39 years O 40-49 years O 50 or more years
6. How long have you used a computer in a professional workplace?
O Less than 1 year O 1-3 years
O 4-6 years
O 7-9 years O 10-19 years O 20-29 years O 30-39 years O 40-49 years O 50 or more years
7. How long have you used Optum PM and Physician EMR?
O Less than 1 year O 1-3 years
O 4-6 years
O 7-9 years O 10 or more years
8. Do you use assistive technology; such as an enhanced visual display, physically adapted pointing device, or text-to-speech engine; in the workplace?
O Yes O No
If yes, please describe __________________________________
Harris CareTracker UCD ISO Implementation
1 ISO STANDARDS
The ISO standards are methods of User Centered Design (UCD) that CareTracker uses to
incorporate user feedback during the development process.
The UCD process starts at the ‘Understand User Needs’ (pictured to the right).
As features move through development, CareTracker follows the ISO standards below.
Additionally, CareTracker submitted the ‘Summative Usability Test’ to conclude the UCD ‘Test and
Evaluate’ portion during Meaningful Use Stage II Certification.
2 MEANINGFUL USE
The following measures were developed using the standards mentioned in this document.
5.4 Use existing knowledge to develop design solutions
5.5 Specify system and use
5.6 Develop prototypes
5.7 Develop user training
5.8 Develop user support
6 Evaluate designs against requirements
6.1 Specify and validate context of evaluation
6.2 Evaluate early prototypes in order to define the requirements for the system
6.3 Evaluate prototypes in order to improve the design
6.4 Evaluate the system to check that the stakeholder and organizational requirements have been met
6.5 Evaluate the system in order to check that the required practice has been followed
6.6 Evaluate the system in use in order to ensure that it continues to meet organizational and user needs
7 Introduce and operate the system
7.1 Management of change
7.2 Determine impact on organization and stakeholders
7.3 Customization and local design
7.4 Deliver user training
7.5 Support users in planned activities
7.6 Ensure conformance to workplace ergonomic legislation
4 ISO 9241-210
CareTracker adheres to all six principles of ISO 9241-210 to engage customers during product development.
Design based on an explicit understanding of users, tasks, environments
Users are involved throughout the design and development
Design is driven and refined by user-centered evaluation
The Process Is Iterative
Design addresses the whole user experience
Design includes multi-disciplinary skills and perspectives
1 Provide an easy way for customer to submit features or ideas
1.1 Idea submission form created Form entry to submit ideas
1.2 Internal meeting to review idea submissions The idea is reviewed by the product teams, categorize and reply to customers
1.3 Post ideas for customer to view and vote Allow customers to review other ideas submitted by end users and vocalize which ideas would be most helpful
2 Identify product requirements on an idea Analysis and document functional product requirements for the new feature based on end user scenarios
2.1 Engage customer Ensure an explicit understanding of the users who this feature will benefit, tasks and workflows.
2.2 Prototype The customer and product teams work together to solidify the feature
2.3 Review prototype with customer The customer and product team’s work together to solidify the feature.
2.4 Allow time for revisions This is an iterative process.
3 Develop feature or idea Development begins and is published to a beta environment.
3.1 Publish to beta Customer access to beta is given by the product team. Those customers who have access to beta have the opportunity to review and evaluate the feature in context to their actual workflows (a holistic reviewing processing).
3.2 Engage customer The customer and product team’s work together to solidify the feature.
3.3 Customer rated features To measure success; customer ratings are required. Once the customer provides a ‘5-star’ review… the feature will be eligible to move into the next version of the software.
Additionally; the diagram below illustrates the iterative process to solicit customer feedback, gain explicit understanding of requirements and design a solution that embodies a holistic positive end user experience.
Test Results Summary for 2014 Edition EHR Certification15-3281-R-0016-PRA V1.1, February 24, 2016
