Customized Common Industry Format Template for Electronic ... · 1. Order medications, review and...

athenaClinicals® 17.11 Customized Common Industry Format Template for Electronic Health Record Usability Testing

A User-Centered Design (UCD) Activity

© 2017 athenahealth, Inc. Rev 1 - Page 2 of 210

Copyright Notice Copyright © 2017 athenahealth, Inc. All rights reserved.


Contents Executive Summary .............................................................................................................................................................. 7

Data Summary .......................................................................................................................................................................... 8 Major Findings and Areas for Improvement..........................................................................................................................19

Introduction ......................................................................................................................................................................... 24 Method .................................................................................................................................................................................. 25

Participants ..............................................................................................................................................................................25 Study Design ...........................................................................................................................................................................33 Tasks .......................................................................................................................................................................................33 Procedures ..............................................................................................................................................................................34 Test Environment ....................................................................................................................................................................34 Test Forms and Tools.............................................................................................................................................................35 Test Location ...........................................................................................................................................................................35 Participant Instructions ...........................................................................................................................................................35

Participant Instructions – Clinical ....................................................................................................................................35 Participant Instructions – Configuration ..........................................................................................................................37 Participant Instructions – Registration ............................................................................................................................39

Usability Metrics ......................................................................................................................................................................41 Data Scoring ...........................................................................................................................................................................41

Results: § 170.315(a)(1) Computerized Provider Order Entry (CPOE) – Medications.................................................. 45 Task Mapping ............................................................................................................................................................................45 Task Participant and Instructions ...............................................................................................................................................45 Data Analysis and Reporting ......................................................................................................................................................46 Discussion of the Findings ..........................................................................................................................................................46

Results: § 170.315(a)(2) Computerized Provider Order Entry (CPOE) – Laboratory.................................................... 49 Task Mapping ..........................................................................................................................................................................49 Task Participant and Instructions ..........................................................................................................................................49 Data Analysis and Reporting .................................................................................................................................................50 Discussion of the Findings .....................................................................................................................................................50

Results: § 170.315(a)(3) Computerized Provider Order Entry (CPOE) – Diagnostic Imaging ..................................... 52 Task Mapping ..........................................................................................................................................................................52 Task Participant and Instructions ..........................................................................................................................................52 Data Analysis and Reporting .................................................................................................................................................53 Discussion of the Findings .....................................................................................................................................................53

Results: § 170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks – Clinical ......................................................... 55 Task Mapping ............................................................................................................................................................................55 Task Participants and Instructions ..............................................................................................................................................55 Data Analysis and Reporting ......................................................................................................................................................55 Discussion of the Findings ..........................................................................................................................................................56

Results: § 170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks – Configuration .............................................. 59 Task Mapping ..........................................................................................................................................................................59 Task Participant and Instructions ..........................................................................................................................................59 Data Analysis and Reporting – Configuration .......................................................................................................................59 Discussion of the Findings .....................................................................................................................................................60

Results: § 170.315(a)(5) Demographics - Clinical ............................................................................................................ 62 Task Mapping ..........................................................................................................................................................................62 Task Participant and Instructions ..........................................................................................................................................62 Data Analysis and Reporting .................................................................................................................................................63 Discussion of the Findings .....................................................................................................................................................63

Results: § 170.315(a)(5) Demographics – Registration ................................................................................................... 66 Task Mapping ............................................................................................................................................................................66 Task Participant and Instructions ...............................................................................................................................................66


Data Analysis and Reporting ......................................................................................................................................................68 Discussion of the Findings ..........................................................................................................................................................69

Results: § 170.315(a)(6) Problem List ............................................................................................................................... 71 Task Mapping ..........................................................................................................................................................................71 Task Participant and Instructions ..........................................................................................................................................71 Data Analysis and Reporting .................................................................................................................................................71 Discussion of the Findings .....................................................................................................................................................72

Results: § 170.315(a)(7) Medication List ........................................................................................................................... 74 Task Mapping ..........................................................................................................................................................................74 Task Participant and Instructions ..........................................................................................................................................74 Data Analysis and Reporting .................................................................................................................................................74 Discussion of the Findings .....................................................................................................................................................75

Results: § 170.315(a)(8) Medication Allergy List.............................................................................................................. 77 Task Mapping ..........................................................................................................................................................................77 Task Participant and Instructions ..........................................................................................................................................77 Data Analysis and Reporting .................................................................................................................................................77 Discussion of the Findings .....................................................................................................................................................78

Results: § 170.315(a)(9) Clinical Decision Support – Clinical ........................................................................................ 81 Task Mapping ............................................................................................................................................................................81 Task Participant and Instructions ...............................................................................................................................................82 Data Analysis and Reporting ......................................................................................................................................................83 Discussion of the Findings ..........................................................................................................................................................86

Results: § 170.315(a)(9) Clinical Decision Support – Configuration.............................................................................. 90 Task Mapping ..........................................................................................................................................................................90 Task Participant and Instructions ..........................................................................................................................................91 Data Analysis and Reporting .................................................................................................................................................91 Discussion of the Findings .....................................................................................................................................................92

Results: § 170.315(a)(14) Implantable Device List ........................................................................................................... 94 Task Mapping ............................................................................................................................................................................94 Task Participants and Instructions ..............................................................................................................................................95 Data Analysis and Reporting ......................................................................................................................................................95 Discussion of the Findings ..........................................................................................................................................................96

Results: § 170.315(b)(2) Clinical Information Reconciliation and Incorporation (CIRI) ............................................... 99 Task Mapping ............................................................................................................................................................................99 Task Participants and Instructions ..............................................................................................................................................99 Data Analysis and Reporting – Clinical ..................................................................................................................................... 100 Discussion of the Findings ....................................................................................................................................................... 101

Results: § 170.315(b)(3) Electronic Prescribing............................................................................................................. 105 Task Mapping ......................................................................................................................................................................... 105 Task Participant and Instructions ............................................................................................................................................ 105 Data Analysis and Reporting ................................................................................................................................................... 107 Discussion of the Findings ....................................................................................................................................................... 108

Results: Critical Use Risks for Patient Safety and NISTIR 7804-1 ............................................................................... 112 Task Mapping ......................................................................................................................................................................... 112 Task Participant and Instructions ............................................................................................................................................ 112 Data Analysis and Reporting ................................................................................................................................................... 112 Discussion of the Findings ....................................................................................................................................................... 113

System Satisfaction .......................................................................................................................................................... 116 About System Usability Scale (SUS) Scores ..................................................................................................................... 116 Clinical System Satisfaction Results .................................................................................................................................. 116 Configuration System Satisfaction Results ........................................................................................................................ 116 Registration System Satisfaction Results .......................................................................................................................... 116

Appendices ........................................................................................................................................................................ 117


Appendix 1: Sample Recruiting Screener .......................................................................................................................... 118 Appendix 2: Participant Demographics for athenahealth’s EHR ...................................................................................... 119 Appendix 3: NDA and Informed Consent Form ................................................................................................................. 120 Appendix 4: Example Moderator Guides ........................................................................................................................... 121

Moderator Guide Specific to Inpatient Prescribers ..................................................................................................... 121 Moderator Guide Specific to Ambulatory Prescribers ................................................................................................ 138 Moderator Guide Specific to Inpatient Nurses ............................................................................................................ 154 Moderator Guide Specific to Ambulatory Clinical Staff............................................................................................... 162 Moderator Guide Specific to Configuration Test ......................................................................................................... 171 Moderator Guide Specific to Inpatient Registration Test ............................................................................................ 175 Moderator Guide Specific to Ambulatory Registration Test ....................................................................................... 179

Appendix 5: Example Task Data Sheets ........................................................................................................................... 183 Task Data Sheets Specific to Inpatient Prescribers ................................................................................................... 183 Task Data Sheets Specific to Ambulatory Prescribers ............................................................................................... 190 Task Data Sheets Specific to Inpatient Nurses .......................................................................................................... 198 Task Data Sheets Specific to Ambulatory Clinical Staff ............................................................................................. 201 Task Data Sheets Specific to Configuration Tests ..................................................................................................... 204 Task Data Sheets Specific to Registration – Inpatient ............................................................................................... 205 Task Data Sheets Specific to Registration– Ambulatory............................................................................................ 207

Appendix 6: End of Task Rating Scale Questionnaire ...................................................................................................... 209 Appendix 7: System Usability Scale Questionnaire .......................................................................................................... 210


EHR Usability Test Report of athenaClinicals 17.11

Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports athenaClinicals 17.11

Date of Usability Test: April 19, 2017 – May 24, 2017 (Clinical, Configuration, and Registration Tests) and July 18, 2017 – August 17, 2017 (Clinical and Registration Tests)

Date of Report: September 8, 2017 Report Prepared By: User-View, Inc.

Jennifer Mauney, Erika Orrick, Jen Moore, and Janey Barnes, PhD Human Factors Specialists 919.697.5329 [email protected] 1109 Holland Ridge Drive Raleigh, NC 27603


Executive Summary A usability test of athenaClinicals 17.11 was conducted remotely from April 19, 2017 through May 24, 2017 and July 18, 2017 through August 17, 2017 by User-View, Inc. The purpose of this test was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the usability test, 66 healthcare providers, 15 registration specialists, and 18 configuration specialists matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on 19 scenarios typically conducted on an EHR:

1. Order medications, review and act upon drug-allergy interaction alert, access source attribute for

alert, change medication order, order additional medication, review and act upon drug-drug interaction alert, access source attribute for alert, order a specific lab.

2. Access medication, lab, and imaging orders entered by prescriber. 3. Order a specific lab, order specific diagnostic image, change a lab and change an image order. 4. Reconcile specific clinical information (medications, problems, allergies) based on the information

provided, add new and review medications and make updates based on the information provided. 5. Create a Continuity of Care Document with the reconciled information incorporated. 6. Access medication allergy list, add new medication allergy, update existing medication allergy,

review current medication allergies. 7. Access medication lists, review current and previous medications, update existing medication, add

new medication. 8. Access problem list, update existing problem, add new problem, review problem list. 9. Add, edit, and remove implantable device data. Access and obtain detailed information about

devices. 10. Work with Electronic Prescribe: prescribe a liquid medication, review system restriction on cc’s. 11. Review linked referential CDS, access source attributes for alerts. 12. Review and act on evidence-based CDS. 13. Review and act on evidence-based CDS, access source attributes for alerts. 14. Work with Electronic Prescribe: order a prescription, review system restriction on decimals, cancel a

prescription, refill prescription, and change prescription. 15. Work with Electronic Prescribe: review fill status and review medication history. 16. Access and document the death of a patient, update death date and death. 17. Configure user preferences for Drug-Drug/Drug-Allergy alerts. 18. Record demographics for a patient, update demographics for a patient. 19. Subscribe to and configure system for CDS recommendations.

During the 60-minute and 30-minute one-on-one remote clinical usability tests, the 30-minute one-on-one remote configuration usability tests, and the 30-minute one-on-one remote registration usability tests, each participant was greeted by the administrator and asked to review and sign an informed consent/release form (included in Appendix 3: NDA and Informed Consent Form); they were instructed that they could withdraw at any time. Participants had prior experience with the EHR. Training material was distributed to prospective participants prior to the test. The administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the EHRUT. During the testing, the administrator timed each task and, along with the data logger, if used, recorded user performance data on paper and electronically. The administrator did not give the participant assistance in how to complete the task. Participant screens and audio were recorded for subsequent analysis. The following types of data were collected for each participant:

• Number of tasks successfully completed within the allotted time without assistance • Time to complete the tasks


• Number and types of errors • Path deviations • Participant’s verbalizations • Participant’s satisfaction ratings of the system

All participant data was de-identified – no correspondence could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire and thanked for their time.

Data Summary Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT (see Table 1).

Table 1. Summary of the performance and rating data.

Measure Task

N Task Success

Path Deviation Task Time Errors

Scenario Ratings 5=Easy

# (%) Deviations (Observed / Optimal)

Mean (SD) Sec

Deviations (Observed /

Optimal) (%) Mean (SD)

CPOE – Medications Task 1A.1: Access CPOE medications

16 100%

16/14

327* (76)

327* /72

0%**

4.4* (0.8)

Task 1A.2: Order new medication [Ampicillin] and link to the diagnosis

16 100% 0%**

Task 1D: Change medication [Levofloxacin (Levaquin)]

14^ of 16

100% 11/9 0%**

Task 1E: Prescribe new medication (Prednisone and link to the diagnosis

15^ 100% 16/14 0%**

Task 1RNA/C: Access pending medication orders

15^ 100% 4/4 70* (37)

70* /14

0%** 4.6* (0.6)



Measure Task

N Task Success




Mean (SD) Sec



CPOE – Laboratory Task 2A.1: Access CPOE labs 17 100%

4/4 103* (41)

103* /29

0%**

4.5* (0.6)

Task 2A.2: Order a lab (CBC w/auto diff) and link to a diagnosis

17 100% 0%**

Task 2D: Change a lab order 17 100% 4/4 0%**

1RNB/A: Access pending lab orders 16 100% 2/2

70* (37)

70* /14

0%** 4.6* (0.6)

CPOE – Diagnostic Imaging Task 2B.1: Access CPOE diagnostic imaging orders

17 100%

5/4 103* (41)

103* /29

0%**

4.5* (0.6)

Task 2B.2: Order diagnostic imaging (X-ray, chest 2-view)

16^ 100% 0%**

Task 2C: Change imaging order 17 100% 3/3 0%**

Task 1RNC/B: Access pending imaging orders

16 100% 1/1 70* (37)

70* /14

0%** 4.6* (0.6)

Drug-Drug, Drug-Allergy Interaction Drug-Drug, Drug-Allergy Interaction – Clinical

Task 1B: Review and act upon Drug-Allergy interaction alert for Ampicillin and Penicillin

16 94% 0/0

327* (76)

327* /72

6%**

4.4* (0.8) Task 1F: Review and

act upon Drug-Drug interaction alert for Levofloxacin and Warfarin

13^ of 16 77% 0/0 23%**



Measure Task

N Task Success




Mean (SD) Sec



Drug-Drug, Drug-Allergy Interaction – Configuration Task 10A: Update settings for interaction warnings.

16 88%<> 7/6

91* (37)

91* /21

13%<**

4.0* (1.3) Task 10B: Update

settings for interaction warnings.

16 94%> 6/6 6%**

Demographics Demographics – Clinical Task 9A.1: Access patient preliminary cause and date of death.

16 100%

23@

/18 208* (59)

208* /56

0%**

4.3* (0.7)

Task 9A.2: Record patient preliminary cause and date of death.

16 100% 0%**

Task 9B: Change patient preliminary cause and date of death.

16 100% 17@

/13 0%**

Demographics – Registration

Task 11A: Access and record demographics for a patient:

A.1: Sex

15

93%

17~ /23

362* (249)

362* /101

7%**

4.3* (0.8)

A.2: DOB 100% 0%** A.3: Language 93% 7%** A.4: Race 93% 7%** A.5: Ethnicity 93% 7%** A.6: Gender identity

73% 27%**

A.7: Sexual Orientation

73% 27%**



Measure Task

N Task Success




Mean (SD) Sec



Task 11B: Change demographics for a patient:

B.1: Sex

15

100%

23~ /24

215* (94)

215* /53

0%**

3.8* (1.3)

B.2: DOB 100% 0%** B.3: Language 100% 0%** B.4: Race (1) 100% 0%** B.5: Race (2) 73% 27%** B.6: Ethnicity 100% 0%** B.7: Gender Identity 93% 7%** B.8: Sexual Orientation

100% 0%**

Problem List Task 4I: Access problem/diagnosis list

20 100% 1/1

58* (41)

58* /20

0%**

4.5* (0.9)

Task 4J: Change problem (Childhood Asthma)

20 95% 4/4 5%**

Task 4K: Record problem (Acute Laryngitis)

20 100% 5/5 0%**

Task 4L: Review current and previous problems

20 100 1/1 0%**



Measure Task

N Task Success




Mean (SD) Sec



Medication List Task 4E: Access medication list 20 100% 2/2

114* (76)

114* /35

0%**

4.3* (0.9)

Task 4F: Review current and previous medications

20 100% 1/1 0%**

Task 4G: Change medication (Azithromycin)

20 100% 6/5 0%**

Task 4H: Record medication [Aldactone (spironolactone)]

20 95% 10@/7 5%**

Medication Allergy List Task 4A: Access medication allergy list

20 100% 2/2

112* (52)

112* /43

0%**

4.5* (0.7)

Task 4B: Record new medication allergy (Iodine)

20 100% 10/10 0%**

Task 4C: Change medication allergy (Sulfa)

20 90% 8/7 10%**

Task 4D: Review current and previous medication allergies

20 100% 1/1 0%**



Measure Task

N Task Success




Mean (SD) Sec



Clinical Decision Support Clinical Decision Support – Clinical Task 7.2A: Order Ritalin 10mg based on CDS alert (CDS: Medications)

14^ Performance after initial exposure to icon.

100% 9/6

170* (70)

170* /30

0%**

3.8* (1.0)

Task 7.2B: Record completion of pediatric diabetes recommendation based on CDS alert (CDS Combo: Demographics and Diagnosis)

17 100% 3/3 0%**

Task 7.3A: Record completion of breast cancer screening based on CDS alert (CDS: Demographics)

17

Performance after initial exposure to icon.#

88% 8/8

193* (52)

193* /58

12%**

4.4* (0.7)

Task 7.3B: Record completion of foot exam based on CDS alert (CDS: Problems)

17 100% 3/3 0%**

Task 7.3D: Record results of LDL test based on CDS alert (CDS: Lab Tests)

17 100% 5/5 0%**

Task 7.3E: Document screening plan for hypertension based on CDS alert (CDS: Vital Signs)

17 100% 3/3 0%**

Task 7.3F: Verify there is no CDS alert for a shingles vaccine (CDS: Med Allergy)

16^ 100% 3/3 0%**



Measure Task

N Task Success




Mean (SD) Sec



Task 7.1A: Access linked referential decision support for CDS demographics and diagnosis alert (children with diabetes)

20

Initial Exposure for some participants who had not performed this type of task during their careers.

65%> 2/1

130* (80)

130/ 22

35%**

4.1* (1.1)

Task 7.1C: Access linked referential decision support for CDS problems alert (asthma)

20 100%> 1/1 0%**

Task 7.1E: Access linked referential decision support for CDS medications alert (ADHD medication)

20 100% 1/1 0%**

Task 1C: Access source attributes for the CDS alert content (Drug x Allergy Interaction)

15^ 80% 0/0 327* (76)

327* /72

20%** 4.4* (0.8)

Task 7.1D: Access source attributes for InfoButton content

20 Performance after initial exposure.

100% 0/0 130* (80)

130* /22

0%** 4.1* (1.1)

Task 7.3C: Access source attributes for CDS alert content


88% 1/1 193* (52)

193* /58

12%** 4.4* (0.7)



Measure Task

N Task Success




Mean (SD) Sec



Clinical Decision Support – Configuration

Task 12C: Activate CDS Intervention: Alcohol Use Assessment Guideline

18

Performance after initial exposure.

89%θ 5/4

103* (54)

103* /27

11%**

4.3* (0.9)

Task 12D: Adjust Age of CDS Intervention: Alcohol Use Assessment Guideline

18 94% 3/3 6%**

Task 12E: Adjust Queried Year of CDS Intervention: Alcohol Use Assessment Guideline

18 94% 3/3 6%**

Implantable Device List Task 5B: Record and parse implanted device ID on patient chart

24


100% 8/7

235* (130)

235* /48

0%**

3.8* (1.1)

Task 5C: Change implanted device ID on patient chart

23 91% 8/6 9%**

Task 5D: View implantable device UDI and description information

24 100% 1/1 0%**

Task 5E: View implantable device details

24 100% 0/0 0%**

Task 5F: View all active implantable devices

24 100% 1/1 0%**

Task 5G: View all implantable devices

23 100% 0/0 0%**



Measure Task

N Task Success




Mean (SD) Sec



Clinical Information Reconciliation Integration (CIRI) Task 3A: Identify reconciliation is needed

15 93% 4/3

396* (208)

396* /73

7%**

3.4* (1.2)

Task 3B: Reconcile medication allergies from a recent clinic visit

15 80%> 6/6 20%**

Task 3C: Reconcile medications from a recent clinic visit

15 87%> 12/10 13%**

Task 3D: Reconcile problems from a recent clinic visit

15 93%> 9/8 7%**

Task 3E: Complete the reconciliation 15 93% 3/3 7%**

Task 3F: Create a Continuity of Care Document and submit electronically

22 82% 13/6 82* (87)

82* /25

18%** 3.6* (1.3)

Electronic Prescription

Task 6A: Create new ePrescribed liquid medication [Zithromax (azithromycin)]

16 100% 23/20 120* (60)

120* /34

0%** 4.6* (0.5)



Measure Task

N Task Success




Mean (SD) Sec



Task 8A: ePrescribe half tablet of medication (Metformin)

16$ 94% 25/23

168* (115)

168* /50

6%**

3.6* (1.0) Task 8B: Cancel

ePrescribed medication (Albuterol)

10^$ of 16 70% 12/7 30%**

Task 8C: Review and approve eRefill medication request (Ondansetron)

16$ 88%< 10/6

228* (110)

228* /30

13%<**

3.7* (1.0)

Task 8D.1,2,3: Review and respond to pharmacy change request

16+$ 94% 9/9 6%**

Task 8F: Request and review fill status (Cefdinir)

17$ 100% 6/4 281* (96)

281* /30

0%** 2.9* (1.5) Task 8G: Request

and review medication history

17$ 94% 1/0 6%**

Critical Use Risks for Patient Safety and NISTIR 7804-1

Scenario 1: Select John Wilson (prescribers only)

16 94%

N/A N/A N/A

6%**

N/A

Scenario 2: Select John Wilson (prescribers only)

17 100% 0%**

Scenario 1RN: Select Jack Wilson (nurses/clinical staff only)

16 94% 6%**

Scenario 3: Select Perry Benson 15 93% 7%**

Scenario 4: Select Perry Benson 20 100% 0%**

Scenario 5: Select Tracy Grace 24 100% 0%**



Measure Task

N Task Success




Mean (SD) Sec



Scenario 6: Select Thomas Douglas (prescribers only)

16 100% 0%**

Scenario 7.1: Select Jackson Wallace; this task had the same but inverted name on the patient list

20 90% 10%**


17 88% 12%**

Scenario 7.3: Select Leslie Rose (prescribers only)

17 100% 0%**

Scenario 8: Select Sidney Cassidy (prescribers only)

16 100% 0%**

Scenario 8: Select Teague Rodgersson (prescribers only)

17 100% 0%**

Scenario 9: Select James Scott (inpatient only)

16 100% 0%**

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. ^ The data was not included for one or more participants due to an artifact or testing or an operational difficulty. Refer to the appropriate results chapter for detailed information. < Rounding error. $ Only ambulatory prescribers attempted these tasks. + Eighteen participants attempted to perform one of three subtasks related to changing a prescription requested by the pharmacy that received the order. ~ Low click count is due to some participants not completing all required fields. # A few participants did not complete Scenario 7.2, and thus Scenario 7.3 was their first CDS task to complete. @ Path deviations that were observed in testing but not included in the optimal click count included adding additional information to the EHR specified in the task scenario that many clinicians found relevant (e.g., adding that a diagnosis was a primary diagnosis). > Some tasks were performed two or more times in testing because these tasks were new to most participants or are performed very infrequently (e.g., configuration tasks). This was because the MU3 Safety Enhanced Design (SED) testing goal is to find true use error issues that persist over time, rather than "discoverability" or "learnability" issues. If higher pass rates, lower click counts, and/or lower task times were seen during additional testing of similar tasks, this is indicative that initial tests more likely showed discoverability issues. A higher pass


rate the second time does not mean there are no other issues present, just that the initial lower pass rate is influenced by the participants' unfamiliarity with the part of the product being tested. θ Initial exposure data included many participants who had not performed this adjustment during their careers and had no familiarity with these tasks. The initial exposure for activating a CDS intervention: BP Control (140/90) Guideline was 44% Task Success and 56%** Errors, 5/4 Path Deviation, 147* (51) Mean (SD) Task Time, 147*/25 Deviations Task Time, and 3.7* (1.9) Mean (SD) Task Ratings. Included in the initial exposure was the task to adjust the age CDS Intervention: BP Control (140/90) Guideline. The results were 83% Task Success and 17%** Errors, 2/4 Path Deviation, 147* (51) Mean (SD) Task Time, 147*/25 Deviations Task Time, and 3.7* (1.9) Mean (SD) Task Ratings.

The SUS is a reliable and valid measure of system satisfaction. Sauro (http://www.measuringusability.com/sus.php accessed 3/14/2013) reports that the average SUS score from 500 studies across various products e.g., websites, cell phones, enterprise systems and across different industries is a 68. A SUS score above a 68 is considered above average and anything below 68 is below average. The results from the System Usability Scale scored the subjective satisfaction with the system based on performance with these tasks to be: Clinical System Satisfaction Results:

Forty-one inpatient and ambulatory prescribers and 25 inpatient and ambulatory nurses / clinical staff completed the SUS questionnaire at the end of their session. However, due to an operational difficulty, one prescriber’s data was not usable. The system scored an average of 70 (SD = 21) based on 40 prescribers’ and 25 nurses’ responses.1

Configuration System Satisfaction Results:

Eighteen configuration participants completed the SUS questionnaire at the end of their session. The system scored an average of 78 (SD=14) based on the responses of 18 configuration participants.

Registration System Satisfaction Results: Fifteen registration participants completed the SUS questionnaire at the end of their session. The system scored an average of 83 (SD=17) based on the responses of 15 registration participants.

In addition to the performance data, the following qualitative observations were made:

Major Findings and Areas for Improvement § 170.315(a)(1) Computerized Provider Order Entry (CPOE) – Medications

Task performance for all of the subtasks were at or above the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below. Areas for improvement related to effectiveness and efficiency include:

• Continue to review cognitive workflow and associated interactions for improved interactions for effectiveness and efficiency.

§ 170.315(a)(2) Computerized Provider Order Entry (CPOE) – Laboratory

Task performance for all of the subtasks were at or above the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below.

Areas for improvement related to effectiveness and efficiency include:

• Continue to review cognitive workflow and associated interactions for improved effectiveness and efficiency.

§ 170.315(a)(3) Computerized Provider Order Entry (CPOE) – Diagnostic Imaging


1 See System Satisfaction chapter for more information.




• Keep the same interactions consistent across the EHR. § 170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks – Clinical

Performance of both subtasks – reviewing and acting upon the Drug-Allergy Interaction Alert for Ampicillin and Penicillin and reviewing and acting on the Drug-Drug Interaction Alert displayed for Prednisone and Levofloxacin – fell below the 95% success rate. Task failures were associated with alert fatigue, as described above. Alert fatigue, validated during usability testing, highlights an industry-wide risk concern. Many of the Meaningful Use 3 Safety-Enhanced Design criteria involve alerts and messages. Depending on the scenario, both critical and noncritical alerts appeared. Opportunities to improve the effectiveness and efficiency were identified and are described below.

The athenahealth team is well aware of the industry’s known issue with alert fatigue. The team currently addresses alert fatigue by allowing sites to configure the levels of alerts and by allowing individual prescribers to configure what level(s) of alerts are presented. There are alerts that cannot be ignored because of patient safety. The team has calibrated the system so as to provide safeguards related to these critical alerts. In addition, the athenahealth team would like to call for and be involved with the development and sharing of industry best practices, guidelines, and templates regarding safety-enhanced design that impacts patient safety associated with alerts and alert fatigue. Areas for improvement related to effectiveness and efficiency were also identified related to the visual layout of the content provided in the alerts and regarding the action buttons in the alerts:

• Explore alternatives to optimize for scanning. • Continue to review cognitive workflow, associated interactions, and the content of pop up alerts

for improved effectiveness and efficiency. § 170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks – Configuration

Performance on all tasks fell below the 95% success criterion. Task failures were associated with unfamiliarity with the screen flow, menu labeling confusion, and location of the action button below the fold. Opportunities to improve effectiveness and efficiency were identified and are described below.



§ 170.315(a)(5) Demographics – Clinical Task performance for all of the subtasks were at or above the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below.


• Consider adding a label for the time of death field.

§ 170.315(a)(5) Demographics – Registration Performance of the sub-tasks was at or above the 95 % success criterion for: DOB, change Sex, change DOB, change Language, change Race (1), change Ethnicity, and change Sexual Orientation. Performance of the sub-tasks was below the 95% success criterion for Sex, Language, Race, Ethnicity, Gender Identity, Sexual Orientation, change Race (2), and change Gender Identity. Opportunities to improve the effectiveness and efficiency were identified and are described below.


Areas for improvement related to effectiveness and efficiency include: • Review icons for saliency. • Continue to review the cognitive workflow and associated interactions to improve effectiveness and efficiency.

§ 170.315(a)(6) Problem List

Task performance for all subtasks associated with accessing, adding, and changing a problem were at or above the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below. Areas for improvement related to effectiveness and efficiency include:


§ 170.315(a)(7) Medication List

Task performance for all subtasks associated with reviewing, adding, and changing a patient’s active and historical medications were at or above the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below.



• Consider providing an easy way to convert a free-form medication sig to a structured sig. • Allow the Find function to search all items in the chart regardless of source.

§ 170.315(a)(8) Medication Allergy List

Performance of the subtask changing the medication allergy fell below the 95% success criterion. Performance for all other subtasks was at or above 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below.


• Consider supporting an Undo function. • Continue to review cognitive workflow and associated interactions, including error messages, for

improved effectiveness and efficiency. § 170.315(a)(9) Clinical Decision Support – Clinical

Performance for the following were at or above the 95% success criterion: • Recording completion of CDS intervention that happened on today’s date. • Prescribing a medication to fulfill the CDS Medication criteria. • Verifying there was no CDS alert for a vaccine the patient is allergic to, to fulfill the CDS Medication

Allergy criteria • Accessing linked referential decision support (InfoButton) after initial exposure. • Accessing source attributes for drug-drug and drug-allergy alerts after initial exposure. • Accessing source attributes for InfoButton content after initial exposure.

Performance for the following were below the 95% success criterion:

• Recording completion of a CDS intervention that happened on a previous date. • Accessing linked referential decision support (InfoButton) on initial exposure. • Accessing source attributes for CDS alert content after initial exposure.

Opportunities to improve the effectiveness and efficiency were identified and are described below. However, alert fatigue remains an industry-wide risk concern and must be balanced with the amount of the information that is displayed.


The athenahealth team is well aware of the industry’s known issue with alert fatigue. The team currently addresses alert fatigue by allowing sites to configure the levels of alerts and allowing individual prescribers to configure what level(s) of alerts are presented. There are alerts that cannot be ignored because of patient safety. The team has calibrated the system so as to provide safeguards related to these critical alerts. In addition, the athenahealth team would like to call for and be involved with the development and sharing of industry best practices, guidelines, templates regarding safety-enhanced design that impacts patient safety associated with alerts and alert fatigue. Minimal training and familiarity will aid end users to be able to locate and launch the CDS alert, InfoButton and Source Attribute features. Areas for improvement related to effectiveness and efficiency were also identified related to the visual layout of the content provided in the alerts and regarding the action buttons in the alerts:

• Continue to review the cognitive workflow and associated interactions to improve effectiveness and efficiency.

• Review icons for saliency. • Review similar workflows across products for consistency. • Ensure that activities during the normal clinical workflow are incorporated and reflected in the

system’s CDS.

§ 170.315(a)(9) Clinical Decision Support – Configuration Performance in the all tasks fell below the 95% success criterion. As described above, many task failures were associated with selecting the incorrect measure as a result of using filters that did not include the correct measure. Opportunities to improve effectiveness and efficiency were identified and are described below.


● Consider allowing users to search the menu pertaining to CDS guidelines. ● Consider improving filtering. ● Continue to review the cognitive workflow and associated interactions to improve effectiveness

and efficiency. § 170.315(a)(14) Implantable Devices List

Task performance for all but one subtask was at or above the 95% success criterion. Task performance for the subtask to add an implantable device on first exposure fell below the 95% success criterion. Errors associated with accessing, changing, and adding a new device as well as finding the device identifier information and determining the number of active devices and total devices are described above. Opportunities to improve the effectiveness and efficiency were identified and are described below. Areas for improvement related to effectiveness and efficiency include:


• Consider using more familiar terminology. § 170.315(b)(2) Clinical Information Reconciliation and Incorporation (CIRI)

Performance of all subtasks associated with Clinical Information Reconciliation and Incorporation was below the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below.



• Consider using more familiar terminology. • Continue to review the cognitive workflow and associated interactions, including the use of

warnings, to improve effectiveness and efficiency. • Review panel designs to optimize effectiveness and efficiency.

§ 170.315(b)(3) Electronic Prescribing

Performance for prescribing a new medication (Scenario 6) and requesting and reviewing fill status, as well as the verification tasks was at or above 95% success criterion. Performance for all other electronic prescription subtasks was below the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below. Areas for improvement related to effectiveness and efficiency include:


• Consider using more familiar terminology. NISTIR 7804-1 Identification of Information Critical Risk

Performance of selecting the correct patient fell below the 95% success criterion for some tasks. Opportunities to improve the effectiveness are described below. The athenahealth team is well aware of the industry’s known issue with patient identification. The team currently addresses patient identification by consistently displaying information critical to patient identification in a reserved area on the patient banner. The information is displayed in the same location regardless of scrolling or other navigational mechanisms to move within the screen/window. Critical information for patient identification includes:

• The patient’s name is shown first with the first (given) name, middle name and last (family) name capitalized, followed by any modifier e.g., Taylor John LEE, III.

• Preferred names follow the patient’s name and are displayed in quotations e.g., John Jamison “Jack”.

• The date of birth is shown using the following format e.g., 04-20-1991. • The age is shown using the following format e.g., 25yo. • The gender is provided as “M” or “F”. • The Medical Record number is also provided in the patient banner.

The athenahealth team is actively investigating solutions to indicate if there are similar names on a list that include a risk to cause a misidentification of patient data. In addition, the athenahealth team would like to call for and be involved with the development and sharing of industry best practices, guidelines, and templates regarding safety-enhanced design that impact patient identification.


Introduction The EHRUT tested for this study was a pre-release instance of athenaClinicals 17.11, an EHR and clinical cycle management solution designed to help deliver increased revenue, decreased cost, and increased clinical control to healthcare providers of a variety of specialties in ambulatory and inpatient settings. The athenaClinicals service offers a unique approach to clinical cycle management by combining a web-based electronic health record (EHR) with clinical content and a unique service offering that helps manage practices’ clinical documentation. The athenaClinicals service includes a web-based EHR application that addresses the core clinical workflows of a practice or facility, including: clinical facesheet, encounter documentation, order entry, results viewing, patient call tracking, clinical reminder tracking, and workflow task management. Providers and their staff use athenaClinicals to facilitate and document patient care. The usability testing attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction, such as the number of tasks successfully completed within the allotted time without assistance, time to complete the tasks, number and types of errors, path deviations, participant’s verbalizations, and participant’s satisfaction ratings of the system were captured during the usability testing.


Method

Participants A total of 99 participants were tested on the EHRUT. Participants in the test were inpatient clinical personnel (N=19), ambulatory clinical personnel (N=47), registration personnel (N=15), and configuration specialists (N=18). Participants were recruited by the athenahealth EHR team and were not compensated for their time. In addition, participants had no direct connection to the development of or organization producing the EHRUT. Participants were not from the testing or supplier organization. Participants were given the opportunity to have a similar orientation and level of training as the actual end users would have received. For the test purposes, end-user characteristics were identified and translated into a recruiting screener used to solicit potential participants; an example of a screener is provided in Appendix 1: Sample Recruiting Screener. Recruited participants had a mix of backgrounds and demographic characteristics conforming to the recruitment screener. The following is a table (Table 2) of participants by characteristics, including demographics, professional experience, computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.

Table 2. Participant demographics.

ID

Gender

Age (years)

Education

Role

Professional Experience (months)

Computer Experience (months)

Product Experience (months)

Assistive Technology Needs

INP-P1 Male 40-49

Doctorate

degree

Attending

Physician 240 324 24 No

INP-P2 Male 50-59

Doctorate

degree

Family

Practice


INP-P3 Female 30-39

Bachelor's

degree CNO/NP 12 216 3 No

INP-P4 Male 40-49

Doctorate

degree Physician 144 360 14 No

INP-P5 Female 60-69

Doctorate


INP-P6 Female 30-39

Doctorate


INP-RN1 Female 30-39

Bachelor's

degree RN/CNO 180 240 5 No


Bachelor's

degree

Director of

Nursing 36 180 2 No


Bachelor's

degree RN 360 240 18 No



ID

Gender

Age (years)

Education

Role






Bachelor's

degree

RN,

Informatics,

OR, Floater 120 300 3 No


Bachelor's


INP-RN6 Male 30-39

Associate



Associate

degree

Director of

Respiratory 192 192 9 No


Master's

degree

Nurse

Executive/

RN 408 336 36 No


Master's

degree

Informatics

Nurse and

Infection

Control 216 240 4 No


Bachelor's

degree

Director of

Nursing 156 240 12 No


Associate

degree

Labor &

Delivery

Nurse/

Med Surg/

ER Nurse 48 120 12 No


Associate

degree

Respiratory

Therapist 180 300 18 No


Bachelor's

degree

RN, Nurse

Manager 48 240 11 No

AMB-P1 Female 40-49

Master's

degree

Family Nurse

Practitioner 36 360 2 No

AMB-P2 Male 60-69

Doctorate


AMB-P3 Male 60-69

Doctorate

degree

Allergist/

Immunolo-

gist 384 360 2.5 No



ID

Gender

Age (years)

Education

Role





AMB-P4 Female 60-69

Doctorate


AMB-P5 Female 40-49

Doctorate


AMB-P6 Female 40-49

Doctorate


AMB-P7 Male 40-49

Doctorate


AMB-P8 Male 60-69

Doctorate


AMB-P9 Male 50-59

Doctorate

degree

Professor/

MD 360 480 6 No

AMB-P10 Female 40-49

Doctorate

degree MD 216 456 72 No


Doctorate


AMB-P12 Male 50-59

Doctorate

Degree OB/GYN 276 324 2 No

AMB-P13 Male 40-49

Doctorate



Master's

degree NP 72 60 48 No

AMB-P15 Male 50-59

Doctorate


AMB-P16 Male 60-69

Doctorate

degree

Internal

Medicine



Master's

degree

Nurse

Practitioner 12 300 12 No


Doctorate

degree

Family


AMB-P19 Male 30-39

Doctorate




ID

Gender

Age (years)

Education

Role





AMB_P20 Female 40-49

Doctorate


AMB-P21 Male 60-69

Doctorate



Doctorate


AMB-P23 Male 30-39

Doctorate



Doctorate



Doctorate

degree

Physician/

CMIO 192 336 18 No


Doctorate

degree

Primary

Care


AMB-P27 Male 50-59

Doctorate

degree

Urgent Care

Physician/

CEO 312 360 60 No


Doctorate


AMB-P29 Male 40-49

Doctorate

degree

Physician/

CIO 192 300 2 No

AMB-P30 Male 30-39

Doctorate

degree

Outpatient

Family

Practice



Doctorate


AMB-P32 Male 40-49

Doctorate

degree

Family

Medicine


AMB-P33 Male 50-59

Doctorate




ID

Gender

Age (years)

Education

Role






Doctorate

degree

Outpatient

Physician/

Palliative

Care 324 420 60 No


Doctorate

degree

Family


AMB-CS1 Female 50-59

Associate

degree

CMA/MU

Admin 72 408 2 No

AMB-CS2 Male 30-39

Associate

degree

RN, Quality

Risk Mgmt,

Nurse

Informatics 156 240 2 No


Bachelor's



Bachelor's

degree LPN 228 300 108 No


Trade/tech

nical/vocati

onal

training

Training and

Implementa-

tion Analyst

/ CMA /

Phleboto-

mist 180 300 56 No


Some

college

credit, no

degree

Medical

Assistant 120 240 78 No


Some

college

credit, no

degree

RN/Clinical

Office

Supervisor 420 396 30 No


Associate




ID

Gender

Age (years)

Education

Role






High school

graduate,

diploma or

the

equivalent

(for

example:

GED)

Medical


AMB-

CS10 Female 40-49

Associate

degree

Medical


AMB-

CS11 Female 30-39

Associate

degree

Medical


AMB-

CS12 Female 30-39

Trade/tech

nical/vocati

onal

training

Medical


CI1 Female 30-39

Master's

degree

Physician

Initiative

Coordinator 72 240 12 No

CI2 Female 40-49

Bachelor's

degree Director of IT 120 240 48 No

CI3 Male 50-59

Bachelor's

degree IT Director 372 372 96 No

CI4 Female 30-39

Bachelor's

degree

Practice


CO1 Male 50-59

Some

college

credit, no

degree

Manager of

Clinical


CO2 Female 30-39

Bachelor's

degree

Electronic

Health

Records


CO3 Male 30-39

Master's

degree

Strategy

Analyst 36 132 36 No



ID

Gender

Age (years)

Education

Role





CO4 Female 40-49

Bachelor's

degree

Practice

Administra-

tor 96 240 48 No

CO5 Female 40-49

Bachelor's

degree

Practice

Administra-

tor 240 420 156 No

CO6 Female 50-59

Bachelor's

degree

Clinical

Operations


CO7 Female 30-39

Bachelor's

degree

Office


CO8 Female 50-59

Associate

degree

Office


CO9 Female 20-29

Master's

degree

Operations

Specialist 60 180 24 No

CO10 Female 30-39

Some

college

credit, no

degree COO 36 240 30 No

CO11 Female 40-49

Bachelor's

degree

Office


CO12 Female 30-39

Bachelor's

degree

Manager of

Ambulatory


CO13 Female 20-29

Associate

degree

EHR

Manager 12 48 36 No

CO14 Female 30-39

Master's

degree FNP 84 360 60 No

R201 Female 30-39 Associate degree

Patient Services Rep 12 180 12 No


Office Manager/ Medical Assistant 168 168 96 No


MA and Clerk 66 156 36 No



ID

Gender

Age (years)

Education

Role





R204 Female 50-59

High school graduate, diploma or the equivalent (for example: GED)


R205 Female 40-49

High school graduate, diploma or the equivalent (for example: GED)



Site Coordinator 180 180 48 No

R207 Male 30-39

Trade/technical/vocational training

Medical Assistant 96 240 96 No

R208 Female 30-39 Master's degree

Patient Services Associate 21 384 21 No

R209 Female 30-39 Associate degree RRT 180 240 18 No

R210 Female 30-39

Some college credit, no degree


R211 Female 40-49 Master's degree IS Director 48 240 12 No

R212 Female 30-39 Bachelor's degree

LPN/IT Coordinator 60 120 24 No

R213 Female 30-39

Trade/technical/vocational training

Office Manager/ Medical Assistant 48 300 24 No

R214 Female 30-39 Master's degree

Business Office Director 24 240 18 No

RO11 Female 50-59

Bachelor's

degree

Billing


One hundred and forty participants (matching the demographics in the section on Participants) were recruited and 99 participated in the usability test. Forty-one participants failed to show for the study.


Participants were scheduled for 60-minute sessions and/or 30-minute sessions. Time was scheduled in between each session for debrief by the administrator and data logger and to reset systems to proper test conditions. A spreadsheet was used to keep track of the participant schedule, and each participant’s demographic characteristics were collected and recorded during the session or through an online survey.

Study Design Overall, the objective of this test was to uncover areas where the application performed well – that is, effectively, efficiently, and with satisfaction – and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted with one EHR. Each participant used the system over WebEx in his/her location, and was provided with the same instructions. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant:

• Number of tasks successfully completed within the allotted time without assistance • Time to complete the tasks • Number and types of errors • Path deviations • Participant’s verbalizations (comments) • Participant’s satisfaction ratings of the system

Additional information about the various measures can be found in the section on Usability Metrics.

Tasks Tasks were designed based on the 2015 Edition Health IT Certification criteria and were based on the criticality of function and frequency of use. As part of the task construction, tasks were informed by athenahealth’s risk management program and by NISTIR 7804-1. Tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR. As part of the task construction, tasks were prioritized in accordance with the risk associated with use errors. NISTIR 7804; Form for Expert Review Items 1A through 1H was used to create and prioritize tasks based on design areas related to known use errors. Tasks with an “*” in the list below were identified to be highest priority related to risks based on this exercise.

1. Order medications, review and act upon drug-allergy interaction alert, access source attribute for alert, change medication order, order additional medication, review and act upon drug-drug interaction alert, access source attribute for alert, order a specific lab. *

2. Access medication, lab, and imaging orders entered by prescriber. 3. Order a specific lab, order specific diagnostic image, change a lab and change an image order. * 4. Reconcile specific clinical information (medications, problems, medication allergies) based on the

information provided, add new and review medications and make updates based on the information provided. *

5. Create a Continuity of Care Document with the reconciled information incorporated. 6. Access medication allergy list, add new medication allergy, update existing medication allergy,

review current medication allergies. * 7. Access medication lists, review current and previous medications, update existing medication, add

new medication. * 8. Access problem list, update existing problem, add new problem, review problem list. * 9. Add, edit, and remove implantable device data. Access and obtain detailed information about

devices. 10. Work with Electronic Prescribe: prescribe a liquid medication, review system restriction on cc’s. * 11. Review linked referential CDS, access source attributes for alerts.


12. Review and act on evidence-based CDS. 13. Review and act on evidence-based CDS, access source attributes for alerts. 14. Work with Electronic Prescribe: order a prescription, review system restriction on decimals, cancel a

prescription, refill prescription, and change prescription. * 15. Work with Electronic Prescribe: review fill status and review medication history. 16. Access and document the death of a patient, update death date and death. 17. Configure user preferences for Drug-Drug/Drug-Allergy alerts. 18. Record demographics for a patient, update demographics for a patient. 19. Subscribe to and configure system for CDS recommendations.

Procedures Upon joining the remote session, participants were greeted; and their identity was verified and matched with a name on the participant schedule. Participants were then assigned a participant ID. Each participant reviewed and gave verbal consent to an informed consent and release form (See Appendix 3: NDA and Informed Consent Form). The recording captures the audio of the participant giving consent. To ensure that the 60-minute clinical tests ran smoothly, two staff members participated in the test, the usability administrator and the data logger. The 60-minute clinical tests, the configuration tests, and the registration tests were run with one staff member serving as both the usability administrator and data logger. The usability testing staff conducting the tests were experienced usability practitioners with over seventeen years in the field for each of the administrator and data logger of the clinical tasks, and over thirteen years in the field for the configuration / registration administrator / data logger. All the administrators and data loggers hold advanced degrees in Human Factors / Applied Cognition or Human Factors Engineering. The administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and took notes on participant comments. For the 60-minute clinical tests, a second person served as the data logger and took notes on task success, path deviations, number and type of errors, task times and comments. Participants were instructed to perform the tasks (see specific instructions below):

• As quickly as possible making as few errors and deviations as possible. • Without assistance, administrators were allowed to give immaterial guidance and clarification on

tasks, but not instructions on use. • Without using a think aloud technique.

For each task, the participants were given a written copy of the task. Task timing began once the administrator finished reading the question. The task time was stopped once the participant indicated they had successfully completed the task. Scoring is discussed below in the section on Data Scoring. An end-of-task rating was administered for each scenario. Following the session, the administrator gave the participant the post-test questionnaire (e.g., the System Usability Scale, see Appendix 7: System Usability Scale Questionnaire) and thanked each individual for their participation. Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded into a spreadsheet.

Test Environment The EHRUT would typically be used in an ambulatory setting, although it is also used in inpatient settings. Usability testing was conducted using a remote testing procedure.


For the remote testing sessions, system data was not collected from participants. The participants used a keyboard and mouse when interacting with the EHRUT. The remote sessions were conducted via WebEx. Participants were instructed to call into an audio conference and login to a WebEx meeting. Control of the moderator’s computer was passed to the participant and sessions were moderated using the materials and methods as would be used in face-to-face sessions. All sessions were audio and video recorded. For all of the clinical sessions, the study was run from a Lenovo W550s; screen resolution for the study was set at 1600 x 900; participants used their own machines with various screen sizes. Screen size for Lenovo W550s, is 15.5", OS: Windows 7 Professional. All participants interacted using a keyboard and a mouse when interacting with the application. For all of the configuration and registration sessions, the study was run from Mac mini (mid 2011); screen resolution for the study was set at 1600 x 900; for remote sessions, participants used their own machines with various screen sizes. Screen size for Mac mini is 23.6", OS: Macintosh El Capitan. All participants interacted using a keyboard and a mouse when interacting with the application.

Test Forms and Tools

During the usability test, various documents and instruments were used, including: 1. Non-Disclosure Agreement (NDA) and Informed Consent Form 2. Moderator Guides 3. Task Data Sheets 4. End of Task Rating Scale Questionnaire 5. System Usability Scale Questionnaire Examples of these documents can be found in Appendices 3-7 respectively. The Moderator’s Guide was devised so as to be able to capture required data. The participant’s interaction with EHRUT was captured and recorded digitally using WebEx Recording.

Test Location Clinical sessions conducted April 28 – May 24, 2017 and July 18, 2017 – August 17, 2017 were remote sessions. For these remote testing sessions, the moderator was at her personal office; the data logger documented the data from her personal office; and each participant was at his/her location. Configuration and registration sessions conducted April 19 – May 10, 2017 and July 18, 2017 – August 11, 2017 were remote sessions. For these remote testing sessions, the moderator/data logger was at her personal office and each participant was at his/her location.

Participant Instructions

Participant Instructions – Clinical The administrator read the following instructions aloud to each clinical participant (also see the full moderator’s guide in Appendix 4: Example Moderator Guides.)

Thank you for participating in this study. Our session today will last 60/30 minutes. During that time, you will be performing tasks with athenahealth’s EHR in order to evaluate the extent to which the application is effective/not effective and efficient/not efficient in terms of Meaningful Use certification criteria.


The first thing I would like to do is to show you this consent form to participate. Do you have any questions? I am going to start the recording now. Basically, because parts of this product are not public yet, we are asking you not to discuss what you see today with anyone. We are recording the audio and screenshots of our session today. This recording is only to make sure we capture and analyze the data accurately. It will not be used for any other purpose. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. OK, so I will have it on the record that you provided consent to be in this session, is that correct? This is what you agreed to on the consent form – I show it here so research teams that might go back and view this recording are under the same expectations. Because this is for Meaningful Use, there are some background questions we need to ask.

• Gender • Age • Education (highest degree) • Occupation/Role • Professional Experience • Computer Experience • athenahealth EHR Product Experience • Assistive Technology Needs (yes-what, no) • Reviewed Training (yes/no)

I am going to read this introduction to you because I want to be sure that I don’t miss anything. We did not have any involvement in this software’s creation. We are from an independent consulting company. This is a great chance for you to give the athena team feedback about the application we are going to look at today, so please be honest with your opinions. I will ask you to complete a few activities using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. Some activities might seem simple to you. Other activities might seem difficult. And there will be some activities that you will not able to complete. I am telling you this because I want you to remember that we are not testing you. You are helping us test the usability of the application. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Please stick to the task instructions provided to you. We are aggregating results so we need all participants to complete the same tasks. When you are doing these activities, I am not going to interact or talk to you. I and <<note taker>> will be taking notes about what you are doing. Because I am not going to be talking with you while you do the activities, I want you to make it clear to me when you are done with an activity by saying "I'm done." There are a number of reasons you might be done. (1) Done because you completed the activity. You know you completed it. So you say I’m Done.


(2) Done because you have tried, you know you have not completed the activity, but you are not going to try anything else. So you say I’m Done. (3) Done because you feel like if you asked a question you could finish the activity. The system you are using is likely set different from what you use in your practice. So you might say I have a such and such button on my system. I don’t see on here. And I will tell you the next step and we will go from there. Sometimes I may not answer you. I’m not trying to be difficult; it’s just that I want to see how you would interact with the product on your own. If you get stuck, see if you can figure out what to do. As I mentioned, you can stop the task at any time, but we’d like you to give it your best shot before giving up. And if you get stuck or decide to give up, it means athena will be getting feedback that the EHR needs more work done to make it easier to use. Remember, we are evaluating the product, not you. Do you have any questions or concerns before we begin? During the session, I will be reading a scenario to you. Any information that you need to complete your work is on the slide, we call it a task sheet. Let me show it to you now. You can click on it anytime to see what is needed. Or you can also ask me to repeat the task. To return to the application, click on this icon. After each task, you will rate how easy or difficult it was to perform. You will find that under this icon. During the session, I will be reading a scenario to you. Any information that you need to complete your work is on the slide, we call it a task sheet. Let me show it to you now. You can click on it anytime to see what is needed. Or you can also ask me to repeat the task. To return to the application, click on this icon. After each task, you will rate how easy or difficult it was to perform. You will find that under this icon.

Following the procedural instructions, the administrator read the task and the following instructions:

Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start.

Participants were given various numbers of tasks to complete. Tasks are listed in the moderator’s guide in Appendix 4: Example Moderator Guides. Participants could refer to a Task sheet to complete their tasks. The task sheets are listed in Appendix 5: Example Task Data Sheets.

Participant Instructions – Configuration The administrator read the following instructions aloud to each configuration participant (also see the full moderator’s guide in Moderator Guide Specific to Configuration Test.)

Thank you for participating in this study. Our session today will last 30 minutes. Just so we are all on the same page, we have you here today to help us evaluate the usability of the athenahealth EHR in order to evaluate the extent to which the application is effective/not effective and efficient/not efficient in terms of Meaningful Use certification criteria.


The first thing I would like to do is show you this consent form to participate. Do you have any questions? I am going to start the recording now.

Basically, because parts of this product are not public yet, we are asking you not to discuss what you see today with anyone. We are recording the audio and screenshots of our session today. This recording is only to make sure we capture and analyze the data accurately. It will not be used for any other purpose. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. OK, so I will have it on the record that you provided consent to be in this session, is that correct? Because this is for Meaningful Use, there are some background questions we need to ask.

• Gender • Age • Education (highest degree) • Occupation/Role • Professional Experience • Computer Experience • EHR Product Experience • Assistive Technology Needs (yes-what, no) • Reviewed Training

I am going to read this introduction to you because I want to be sure that I don’t miss anything. I did not have any involvement in this software’s creation. I am from an independent consulting company. This is a great chance for you to give the athena team feedback about the application we are going to look at today, so please be honest with your opinions.

I will ask you to complete a few activities using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. Some activities might seem simple to you. Other activities might seem difficult. And there will be some activities that you will not able to complete. I am telling you this because I want you to remember that we are not testing you. You are helping us test the usability of the application. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Please stick to the task instructions provided to you. We are aggregating results so we need all participants to complete the same tasks. When you are doing these activities, I am not going to interact or talk to you. Because I am not going to be talking with you while you do the activities, I want you to make it clear to me when you are done with an activity by saying, "I'm done." There are a number of reasons you might be done. (1) Done because you completed the activity. You know you completed it. So you say I’m Done. (2) Done because you have tried, you know you have not completed the activity, but you are not going to try anything else. So you say I’m Done.


(3) Done because you feel like if you asked a question you could finish the activity. The system you are using is likely set different from what you use in your practice. So you might say I have a such and such button on my system. I don’t see on here. And I will tell you the next step and we will go from there. Sometimes I may not answer you. I’m not trying to be difficult; it’s just that I want to see how you would interact with the product on your own. If you get stuck, see if you can figure out what to do. As I mentioned, you can stop the task at any time, but we’d like you to give it your best shot before giving up. And if you get stuck or decide to give up, it means athena will be getting feedback that the EHR needs more work done to make it easier to use. Remember, we are evaluating the product, not you. Do you have any questions or concerns before we begin? During the session, I will be reading a scenario to you. Any information you need to complete your work is on the task sheet on the right. You can click on it anytime to see what is needed. Or you can also ask me to repeat the task. To return to the application, click on this icon. After each task, you will rate how easy or difficult it was to perform. You will find that under this icon. Ok, I will pass you control of the mouse and keyboard. You can click that you will take control.


Do you have any questions about what I am asking you to do? Remember to tell me when you are done. Go ahead and start.

Participants were given two guidelines to configure. Tasks are listed in the moderator’s guide in Appendix 4: Example Moderator Guides. Participants could refer to a Task sheet to complete their tasks. The task sheets are listed in Appendix 5: Example Task Data Sheets.

Participant Instructions – Registration The administrator read the following instructions aloud to each registration participant (also see the full moderator’s guide in Moderator Guide Specific to Registration Test.)

Thank you for participating in this study. Our session today will last 30 minutes. Just so we are all on the same page, we have you here today to help us evaluate the usability of the athenahealth EHR in order to evaluate the extent to which the application is effective/not effective and efficient/not efficient in terms of Meaningful Use certification criteria.

The first thing I would like to do is show you this consent form to participate. Do you have any questions? I am going to start the recording now.

Basically, because parts of this product are not public yet, we are asking you not to discuss what you see today with anyone. We are recording the audio and screenshots of our session today. This recording is only to make sure we capture and analyze the data accurately. It will not be used for any other purpose. All of the information that you


provide will be kept confidential and your name will not be associated with your comments at any time. OK, so I will have it on the record that you provided consent to be in this session, is that correct? Because this is for Meaningful Use, there are some background questions we need to ask.

• Gender • Age • Education (highest degree) • Occupation/Role • Professional Experience • Computer Experience • EHR Product Experience • Assistive Technology Needs (yes-what, no)

I am going to read this introduction to you because I want to be sure that I don’t miss anything. I did not have any involvement in this software’s creation. I am from an independent consulting company. This is a great chance for you to give the athena team feedback about the application we are going to look at today, so please be honest with your opinions.

I will ask you to complete a few activities using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. Some activities might seem simple to you. Other activities might seem difficult. And there will be some activities that you will not able to complete. I am telling you this because I want you to remember that we are not testing you. You are helping us test the usability of the application. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Please stick to the task instructions provided to you. We are aggregating results so we need all participants to complete the same tasks. When you are doing these activities, I am not going to interact or talk to you. Because I am not going to be talking with you while you do the activities, I want you to make it clear to me when you are done with an activity by saying, "I'm done." There are a number of reasons you might be done. (1) Done because you completed the activity. You know you completed it. So you say I’m Done. (2) Done because you have tried, you know you have not completed the activity, but you are not going to try anything else. So you say I’m Done. (3) Done because you feel like if you asked a question you could finish the activity. The system you are using is likely set different from what you use in your practice. So you might say I have a such and such button on my system. I don’t see on here. And I will tell you the next step and we will go from there. Sometimes I may not answer you. I’m not trying to be difficult; it’s just that I want to see how you would interact with the product on your own. If you get stuck, see if you can figure out what to do. As I mentioned, you can stop the task at any time, but we’d like you to give it your best shot before giving up. And if you get stuck or decide to give up, it means athena will be getting feedback that the EHR


needs more work done to make it easier to use. Remember, we are evaluating the product, not you. Do you have any questions or concerns before we begin? During the session, I will be reading a scenario to you. Any information you need to complete your work is on the task sheet on the right. You can click on it anytime to see what is needed. Or you can also ask me to repeat the task. To return to the application, click on this icon. After each task, you will rate how easy or difficult it was to perform. You will find that under this icon. Ok, I will pass you control of the mouse and keyboard. You can click that you will take control.


Do you have any questions about what I am asking you to do? Remember to tell me when you are done. Go ahead and start.

Participants were given two tasks to complete. Tasks are listed in the moderator’s guide in Appendix 4: Example Moderator Guides. Participants could refer to a Task sheet to complete their tasks. The task sheets are listed in Appendix 5: Example Task Data Sheets.

Usability Metrics According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess: • Effectiveness (safety-enhanced design) of the athenaClinicals EHR by measuring participant success

rates (pass, fail) and usage errors, including: o Identification and explanation of the root cause of any potential patient safety risks resulting from

usage errors in usability task performance. o Identification of potential mitigations for identified usage errors in usability task performance.

• Efficiency of the athenaClinicals EHR by measuring participant task time and task click path notes, including suggestion of potential solutions for identified inefficiencies observed in usability task performance.

• Satisfaction with the athenaClinicals EHR by administering a post-task/scenario ease-of-use questionnaire and the System Usability Scale (SUS).

Data Scoring The following table (see Table 3) details how tasks were scored, errors evaluated, and the time data analyzed.


Table 3. Details of how observed data were scored.

Measures Rationale and Scoring Effectiveness:

Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. A task was counted as a “Success with navigation assistance” if the participant received administrator help with navigation to the screen directly related to the certification criteria. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage. The standard deviation for each percent success is a required calculation by the ONC. It is not appropriate to calculate a mean or standard deviation for a percentage in the way we are reporting task success. Therefore, the standard deviation for each task success percentage was calculated by representing each “Pass” value as 1 and each “Fail” value as 0, then applying the formula for standard deviation with the values. Note, a zero standard deviation represents a 100% pass or fail rate. Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, was recorded. If the defined threshold time for inactivity (30 seconds) passed, the moderator would stop the participant.



Measures Rationale and Scoring Effectiveness:

Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, received assistance from the moderator associated with completing the task directly related to the certification criteria or reached the end of the allotted time before successful completion, the task was counted as a “Failure.” No task times were taken for errors. The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors.

The standard deviation for each percent failure is a required calculation by the ONC. It is not appropriate to calculate a mean or standard deviation for a percentage in the way we are reporting task success. Therefore, the standard deviation for each task success percentage was calculated by representing each “Fail” value as 1 and each “Pass” value as 0, then applying the formula for standard deviation with the values. Note, a zero standard deviation represents a 100% pass or fail rate. On a qualitative level, an enumeration of errors and error types was collected.

Efficiency:

Task Deviations

The participant’s path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation. Optimal paths (i.e., procedural steps) were recorded when constructing tasks.

Efficiency: Task Time

Each task was timed from when the administrator said, “Begin” until the participant said, “Done.” If he or she failed to say, “Done,” the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation) were also calculated.



Measures Rationale and Scoring Satisfaction: Task Rating

Participant’s subjective impression of the ease-of-use of the application was measured by administering both a post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. To measure participants’ confidence in and likeability of the athenaClinicals EHR overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system frequently,” “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.” See full System Usability Score questionnaire in Appendix 7: System Usability Scale Questionnaire.


Results: § 170.315(a)(1) Computerized Provider Order Entry (CPOE) – Medications

Task Mapping Table 4 maps the Computerized Provider Order Entry (CPOE) – Medications criteria to usability test tasks to aid verification that the report will contain all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 4. CPOE – Medications criteria mapped to usability test tasks. § 170.315(a)(1) Computerized Provider Order Entry – Medications (i) Enable a user to record, change, and access medication orders. (ii) Optional. Include a “reason for order” field. To successfully complete the clinical task, participants were required to complete each of the following subtasks. Only tasks related to § 170.315 (a)(1) CPOE – medications will be addressed in this section. Scenario 1:

Task A: Access and order new medication [Ampicillin] and link to the diagnosis Task D: Change medication [Levofloxacin (Levaquin)] Task E: Prescribe new medication [Prednisone] and link to the diagnosis

Scenario 1RN: Task A (Inpatient) / Task C (Ambulatory): Access pending medication orders

Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, ambulatory prescribers attempted the access, change, and record task. Inpatient and ambulatory nurses attempted the access task. Inpatient and ambulatory data were combined based on the fact that neither the task nor the user characteristics differ significantly based on these settings. Participants were given the following instruction:

Scenario 1 (Prescribers only) You are Pat Rogers providing care for a new patient. The patient is a 58-year old male whose chief complaint is respiratory distress. He has a diagnosis of pneumonia. Task A: Please order Ampicillin 500mg for the patient’s pneumonia. If any alerts/warnings come up during the session, please let me know what the alert or warning is telling you and how you would typically handle the alert in your practice but please wait and let me tell you how to handle the alert for the situation. Task D and E: You decide to order Levofloxacin (Levaquin)500mg for the patient’s pneumonia instead. In addition, to address the patient’s wheezing, you decide to order Prednisone 20mg tablet. Please order prednisone for the patient’s pneumonia. If you are presented with an alert, please tell the moderator. Scenario 1RN (Nurses / Clinical Staff only) You are Alex Jones providing care for a patient. The patient is a 60-year old male whose chief complaint is respiratory distress. He has a diagnosis of pneumonia. Task A, B, and C (Inpatient): You are looking at the patient’s chart, and you first want to see if there are any orders placed since you were last in the chart. Please acknowledge the following new orders for the patient:

• Levofloxacin IV • CBC w/ auto diff • X-ray, chest, 2-view

Task C (Ambulatory): If you wanted to see if there were any medication orders placed for this patient, where would you find that information?


Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Ambulatory prescribers and inpatient and ambulatory nurses performed the CPOE scenarios. Only the activities associated with the CPOE – Medications criteria are reported in this chapter. Sixteen participants attempted Scenario 1 and sixteen participants attempted Scenario 1RN. Task performance was not differentiated by setting (inpatient, ambulatory). Therefore, the data were not separated by user role for the setting. The usability testing results for the EHRUT are detailed below (see Table 5). The table lists out the usability test results for each subtask associated CPOE – Medications.

Table 5. CPOE – Medications usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)

Errors Scenario Ratings

(5=Easy) # Percent

(%) Deviations

(Observed / Optimal)

Mean (SD)

Deviations (Observed / Optimal)

Percent (%) **

Mean (SD)

Task 1A.1: Access CPOE medications 16 100%

16/14

327* (76)

327* /72

0%**

4.4* (0.8)

Task 1A.2: Order new medication (Ampicillin) and link to the diagnosis

16 100% 0%**

Task 1D: Change medication [Levofloxacin (Levaquin)]

14^

of 16 100% 11/9 0%**

Task 1E: Prescribe new medication (Prednisone) and link to the diagnosis

15^ 100% 16/14 0%**

Task 1RNA/C: Access pending medication orders

15^ 100% 4/4 70* (37)

70* /14 0%** 4.6*

(0.6) * Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. ^ The data was not included for one or more participants due to an artifact or testing or an operational difficulty. For Change Medication (Levofloxacin) task and for Prescribe New Medication (Prednisone) and Link to the Diagnosis task, one of sixteen participant’s data was not included due to operational difficulties with the system. For Change Medication (Levofloxacin) task, one of sixteen participant’s data was not included due to an artifact of testing. For the Access Pending Medication Orders task, the data was not included for one of sixteen participants because s/he did not understand the task.

Discussion of the Findings The following sections discuss the results organized around an error analysis, test performance and error rates. The error analysis includes identification of use errors and user interface design issues as well as classification of severity based on the consequence of the error. Use errors and user interface design issues that resulted in subtask failures, that are known industry risk issues, and errors and issues related to aspects of the user interface that are configured per customer site are considered more severe compared to noncritical system usability issues related to efficiency. As such the discussion of more serious errors and issues is provided in the Error Analysis section and the associated mitigation strategy is provided in the Areas for Improvement section.


Based on our definition of effectiveness metrics, performance, use errors and issues stemming from task failures are also discussed in the Effectiveness section as effectiveness was measured with task success and failure. Noncritical system usability issues related to efficiency are discussed in the Efficiency section. Associated recommendations are provided in the Areas for Improvement section. Satisfaction was assessed at the scenario level and at the system level. Both are discussed in the Satisfaction section.

ERROR ANALYSIS

One critical use error was identified or observed as part of ordering or changing medication orders. One participant did not understand how to close the details of an open medication order. S/he had ordered other medications, but couldn’t see them with the open medication order details present. The same participant had a close call when s/he added two prednisone orders, but couldn’t initially see them with the medication order open. Once the medication order was closed, the participant saw the two orders of prednisone and deleted one order. A noncritical use error was seen when two participants thought the diagnosis had been removed. It had scrolled up off the screen. Both participants added a different, but similar diagnosis because the system did not allow the diagnosis to be added a second time. The participants were able to successfully complete the task. One participants had a close call when s/he couldn’t initially find the diagnosis when it scrolled up off the page. The participant was able to successfully complete the task. One participant encountered operational difficulties when trying to change the medication order (Levofloxacin) task and when trying to prescribe a new medication (Prednisone). The operational difficulties were due to a system issue. The data was not included in the analyses. Another participant encountered operational difficulties when trying to complete the acknowledge medication (Ampicillin) task. The participant was able to complete the task. An artifact of testing occurred when one participant fully ordered a new medication, but potentially used a clinical judgement for the specified duration of the medication. That data was not included in the analyses. Multiple participants misunderstood the directions and defaulted to common clinical practices when performing the ordering new medication and changing the medication tasks. The participants completed the tasks and after verifying they had used their clinical judgements, the tasks were scored as task successes. One participant did not understand the directions and initially did not compete the acknowledge the medication task. S/he stopped in the middle of the task and did not complete it. After the task ended, the moderator discussed the task with the participant. S/he understood the task and completed it correctly. This data was not included in the analyses. EFFECTIVENESS Task performance for all of the subtasks were at or above the 95% success criterion. Errors associated with ordering and changing medication orders are described above.


EFFICIENCY

Noncritical usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of this feature. Some participants specified they would like, when they choose a medication and dose (e.g., Lipitor 40 mg tablet), that the known information would prepopulate the SIG. For example, “tablet” would prepopulate in Take 1 tablet daily, and the user would only need to choose the “1” and the “daily”.

SATISFACTION Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean for the providers’ tasks were 4.4 with a standard deviation of 0.8. The mean for the nurses’ task was 4.6 with a standard deviation of 0.6.

MAJOR FINDINGS


AREAS FOR IMPROVEMENT




Results: § 170.315(a)(2) Computerized Provider Order Entry (CPOE) – Laboratory

Task Mapping Table 6 maps the Computerized Provider Order Entry (CPOE) – Laboratory criteria to usability test tasks to aid verification that the report will contain all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 6. CPOE – Laboratory criteria mapped to usability test tasks.

§ 170.315(a)(2) Computerized Provider Order Entry – Laboratory (i) Enable a user to record, change, and access lab orders. (ii) Optional. Include a “reason for order” field. To successfully complete the task, participants were required to complete each of the following subtasks. Only tasks related to § 170.315 (a)(2) CPOE – laboratory will be addressed in this section. Scenario 2:

Task A: Access and order a lab (CBC w/auto diff) and link to a diagnosis Task D: Change a lab order

Scenario 1RN: Task B (Inpatient) / Task A (Ambulatory): Access pending lab orders

Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, both inpatient and ambulatory prescribers attempted the access, change, and record task. Inpatient and ambulatory nurses attempted the access task. Inpatient and ambulatory data were combined based on the fact that neither the task nor the user characteristics differ significantly based on these settings. Participants were given the following instruction:

Scenario 2 (Prescribers only) (Same scenario and patient as Scenario 1) Task A (Inpatient): You had ordered labs for Mr. Wilson yesterday and they returned with an elevated white blood count. You want to order a repeat CBC w/ auto diff to continue to monitor the patient’s labs. Order a CBC w/ auto diff, once, for the patient’s pneumonia. Task A (Ambulatory): Please order a CBC w/ auto diff for the patient’s pneumonia. Task D: (Inpatient / Ambulatory) In addition, you also decide you would like the lab you just ordered to be performed STAT. Please update the lab order. Once you complete these orders, you may sign for the orders. Scenario 1RN (Nurses / Clinical Staff only) You are Alex Jones providing care for a patient. The patient is a 60-year old male whose chief complaint is respiratory distress. He has a diagnosis of pneumonia. Task A, B, and C (Inpatient): You are looking at the patient’s chart, and you first want to see if there are any orders placed since you were last in the chart. Please acknowledge the following new orders for the patient: • Levofloxacin IV • CBC w/ auto diff • X-ray, chest, 2-view Task A and B (Ambulatory): You are looking at the patient’s chart, and you first want to see if there are any orders placed since you were last in the chart. Please tell us any pending orders for the patient.


Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient and ambulatory prescribers and inpatient and ambulatory nurses performed the CPOE scenarios. Only the activities associated with the CPOE – Laboratory criteria are reported in this chapter. Seventeen participants attempted Scenario 2 and sixteen participants attempted Scenario 1RN. Task performance was not differentiated by setting (inpatient, ambulatory). Therefore, the data was not separated by user role for the setting. The usability testing results for the EHRUT are detailed below (see Table 7). The table lists out the subtasks associated with CPOE – Laboratory.

Table 7. CPOE – Laboratory usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%) Deviations


Mean (SD)


Percent (%) **

Mean (SD)

Task 2A.1: Access lab orders

17 100%

4/4 103* (41)

103* /29

0%**

4.5* (0.6)

Task 2A.2: Order a lab (CBC w/auto diff) and link to a diagnosis

17 100% 0%**

Task 2D: Change a lab order

17 100% 4/4 0%**

1RNB/A: Access pending lab orders

16 100% 2/2 70* (37)

70* /14

0%** 4.6*

(0.6) * Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures.

Discussion of the Findings The following sections discuss the results organized around an error analysis, test performance and error rates. The error analysis includes identification of use errors and user interface design issues as well as classification of severity based on the consequence of the error. Use errors and user interface design issues that resulted in subtask failures, that are known industry risk issues, and errors and issues related to aspects of the user interface that are configured per customer site are considered more severe compared to noncritical system usability issues related to efficiency. As such the discussion of more serious errors and issues is provided in the Error Analysis section and the associated mitigation strategy is provided in the Areas for Improvement section. Based on our definition of effectiveness metrics, performance, use errors and issues stemming from task failures are also discussed in the Effectiveness section as effectiveness was measured with task success and failure. Noncritical system usability issues related to efficiency are discussed in the Efficiency section. Associated recommendations are provided in the Areas for Improvement section. Satisfaction was assessed at the scenario level and at the system level. Both are discussed in the Satisfaction section.


ERROR ANALYSIS

One critical use error was identified or observed as part of ordering or changing laboratory test orders. When trying to order a laboratory test, one participant could not find the diagnosis to attach the order to. The participant had scrolled down the page which resulted in the diagnosis being scrolled up off of the page. The participant was unaware that s/he was not viewing the information at the top of the page. One participant encountered operational difficulties when trying to complete the acknowledge laboratories (CBC w/ auto diff) task. The participant was able to complete the task.

One participants misunderstood the directions and defaulted to common clinical practices when performing the record a laboratory task. The participant completed the task and the task was scored as a task success. EFFECTIVENESS Task performance for all of the subtasks were at or above the 95% success criterion. Errors associated with ordering and changing laboratory tests are described above. EFFICIENCY

No major noncritical usability issues that did not result in use errors were identified for improvement related to the efficiency of this feature.

SATISFACTION

Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean for the providers’ tasks were 4.5 with a standard deviation of 0.6. The mean for the nurses’ task was 4.6 with a standard deviation of 0.6. MAJOR FINDINGS






Results: § 170.315(a)(3) Computerized Provider Order Entry (CPOE) – Diagnostic Imaging

Task Mapping Table 8 maps the Computerized Provider Order Entry (CPOE) – Diagnostic Imaging criteria to usability test tasks to aid verification that the report will contain all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 8. CPOE – Diagnostic Imaging criteria mapped to usability test tasks. § 170.315(a)(3) Computerized Provider Order Entry – Diagnostic Imaging (i) Enable a user to record, change, and access lab orders. (ii) Optional. Include a “reason for order” field. To successfully complete the task, participants were required to complete each of the following subtasks. Scenario 2:

Task B: Access and order diagnostic imaging (XR Chest, 2-view) Task C: Change imaging order

Scenario 1RN: Task C (Inpatient) / Task B (Ambulatory): Access pending imaging orders

Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, both inpatient and ambulatory prescribers attempted the access, change, and record task. Inpatient and ambulatory nurses attempted the access task. Inpatient and ambulatory data were combined based on the fact that neither the task nor the user characteristics differ significantly based on these settings. Participants were given the following instruction:

Scenario 2 (Prescribers only) (Same scenario and patient as Scenario 1) Task B (Inpatient): In addition, you want to order a two-view chest X-ray for your patient’s pneumonia. Task B (Ambulatory): In addition, you want to order a two-view chest X-ray for your patient’s pneumonia. You will want this order to go today. You want an alarm if the patient has not had the X-ray within 2 weeks. Task C (Inpatient): Now you’re wondering about the Chest X-ray you ordered for Mr. Wilson’s pneumonia diagnosis. You realize that you really want it to be performed “now”. Please change its status from routine to Now. Task C (Ambulatory): Now you’re wondering about the Chest X-ray you ordered for Mr. Wilson’s pneumonia diagnosis. You realize that you really want an alarm if the patient has not had the X-ray within 1 week. Change the alarm to 1 week. Scenario 1RN (Nurses / Clinical Staff only) You are Alex Jones providing care for a patient. The patient is a 60-year old male whose chief complaint is respiratory distress. He has a diagnosis of pneumonia. Task A, B, and C (Inpatient): You are looking at the patient’s chart, and you first want to see if there are any orders placed since you were last in the chart. Please acknowledge the following new orders for the patient: • Levofloxacin IV • CBC w/ auto diff • X-ray, chest, 2-view Task A and B (Ambulatory): You are looking at the patient’s chart, and you first want to see if there are any orders placed since you were last in the chart. Please tell us any pending orders for the patient.


Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient and ambulatory prescribers and inpatient and ambulatory nurses performed the CPOE scenarios. Only the activities associated with the CPOE - Diagnostic Imaging criteria are reported in this chapter. Seventeen participants attempted Scenario 2 and sixteen participants attempted Scenario 1RN. Task performance was not differentiated by setting (inpatient, ambulatory). Therefore, the data were not separated by user role for the setting. The usability testing results for the EHRUT are detailed below (see Table 9). The table lists out the subtasks associated with CPOE – Diagnostic Imaging.

Table 9. CPOE – Diagnostic Imaging usability test results. Measure

Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%) Deviations


Mean (SD)


Percent (%) **

Mean (SD)

Task 2B.1: Access CPOE diagnostic imaging orders

17 100%

5/4 103* (41)

103* /29

0%**

4.5* (0.6)

Task 2B.2: Order diagnostic imaging (X-ray, chest 2-view)

16^ 100% 0%**

Task 2C: Change imaging order

17 100% 3/3 0%**

Task 1RNC/B: Access pending imaging orders

16 100% 1/1 70* (37)

70* /14

0%** 4.6*

(0.6)

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. ^ The data was not included for one of seventeen participants who attempted the task because s/he did not understand the task.

Discussion of the Findings The following sections discuss the results organized around an error analysis, test performance and error rates. The error analysis includes identification of use errors and user interface design issues as well as classification of severity based on the consequence of the error. Use errors and user interface design issues that resulted in subtask failures, that are known industry risk issues, and errors and issues related to aspects of the user interface that are configured per customer site are considered more severe compared to noncritical system usability issues related to efficiency. As such the discussion of more serious errors and issues is provided in the Error Analysis section and the associated mitigation strategy is provided in the Areas for Improvement section. Based on our definition of effectiveness metrics, performance, use errors and issues stemming from task failures are also discussed in the Effectiveness section as effectiveness was measured with task success and failure. Noncritical system usability issues related to efficiency are discussed in the Efficiency section. Associated recommendations are provided in the Areas for Improvement section.


Satisfaction was assessed at the scenario level and at the system level. Both are discussed in the Satisfaction section.

ERROR ANALYSIS

No critical or noncritical use errors were identified or observed as part of ordering or changing diagnostic imaging orders. When recording the image, one participant had a close call because s/he almost clicked the “X” in the upper right corner to close the details. Clicking “X” deletes the order. One participant encountered operational difficulties when trying to complete the acknowledge imaging (Chest X-ray, 2 view) task. The participant was able to complete the task. One participant encountered an artifact of testing when trying to complete the record an image (Chest X-ray, 2 view) task. The participant was having difficulty scrolling, which is suspected to be due to an issue with the conferencing software. The participant was able to complete the task. One participant encountered an artifact of testing when trying to complete the record an imaging order task. The participant was unable to fully complete the task because the test system was not configured similar to his/her current hospital’s system and s/he did not fully understand the task. The participant was able to talk through how the task would be correctly completed on his/her hospital’s system. This data was not included in the analyses. EFFECTIVENESS

Task performance for all of the subtasks were at or above the 95% success criterion. No errors were identified or observed as part of ordering or changing diagnostic imaging orders. EFFICIENCY


SATISFACTION Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean for the providers’ tasks were 4.5 with a standard deviation of 0.6. The mean for the nurses’ task was 4.6 with a standard deviation of 0.6. MAJOR FINDINGS





• Keep the same interactions consistent across the EHR.


Results: § 170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks – Clinical

Task Mapping Table 10 maps the Drug-Drug, Drug-Allergy Interaction Checks – Clinical criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 10. Drug-Drug, Drug-Allergy Interaction Checks – Clinical criteria mapped to usability test tasks. § 170.315 (a)(4) Drug-Drug, Drug-Allergy Interaction Checks for CPOE. (i) Interventions. Before a medication order is completed and acted upon during computerized provider order entry (CPOE), interventions must automatically indicate to a user drug-drug and drug-allergy contraindications based on a patient’s medication list and medication allergy list. (ii) Adjustments. (A) Enable the severity level of interventions provided for drug-drug interaction checks to be adjusted. (B) Limit the ability to adjust severity levels to an identified set of users or available as a system administrative function. To successfully complete the clinical task, participants were required to detect and acknowledge automatically generated alerts as they completed each of the following tasks. Only clinical tasks related to § 170.315 (a)(4) Drug-Drug, Drug-Allergy Interaction Checks will be discussed in this chapter. Scenario 1:

Task B: Review and act upon Drug-Allergy interaction alert for Ampicillin and Penicillin Task F: Review and act upon Drug-Drug interaction alert for Levofloxacin and Warfarin

Task Participants and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, ambulatory prescribers attempted this task. Participants were not given instructions related to Drug-Drug or Drug-Allergy alerts. However, they were given the following instruction:

Scenario 1 You are Pat Rogers providing care for a new patient. The patient is a 58-year old male whose chief complaint is respiratory distress. He has a diagnosis of pneumonia. Please order Ampicillin 500mg for the patient’s pneumonia. If any alerts/warnings come up during the session, please let me know what the alert or warning is telling you and how you would typically handle the alert in your practice but please wait and let me tell you how to handle the alert for the situation. Task B: (Did they review the Drug x Allergy alert?) You decide to order Levofloxacin (Levaquin)500mg for the patient’s pneumonia instead. In addition, to address the patient’s wheezing, you decide to order Prednisone 20mg tablet. Please order prednisone for the patient’s pneumonia. If you are presented with an alert, please tell the moderator. Task F: (Did they review the Drug x Drug alert?)

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Ambulatory prescribers reviewed and acted upon Drug-Drug and Drug-Allergy alerts as they


were presented in the context of the tasks. Only the review and act-upon activities associated with the drug-drug and drug-allergy criterion are reported in this chapter. Sixteen participants attempted the tasks. The usability testing results for the EHRUT are detailed below (see Table 11). The table lists out the subtasks associated Drug-Drug and Drug-Allergy Interaction Checks – Clinical criteria.

Table 11. Drug-Drug, Drug-Allergy Interaction Checks – Clinical usability test results.

Measure

Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent (%) Deviations


Mean (SD)


Percent (%) **

Mean (SD)

Task 1B: Review and act upon Drug-Allergy interaction alert for Ampicillin and Penicillin

16 94% 0/0

327*

(76)

327*

/72

6%**

4.4*

(0.8) Task 1F: Review and act upon Drug-Drug interaction alert for Levofloxacin and Warfarin

13^

of 16 77% 0/0 23%**

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. ^ The data was not included for one participant due to an operational difficulty with the system. The data was not included for 2 participants due to artifacts of testing; one participant was not reminded to specify if s/he saw an alert and one participant did not have Warfarin added to the patient’s medication list, so the Drug-Drug alert did not display.



ERROR ANALYSIS

Alert fatigue is a known issue across the industry and was validated during usability testing. Alert fatigue is a serious issue that plagues providers. Potential risks related to alert fatigue include desensitization to alerts and/or overreliance on alerts. Throughout the usability test session, participants interacted with a number of warnings. In some cases, the participant carefully reviewed the alert. In most cases, however, the participant routinely dismissed the alert. Two types of drug interaction alerts are incorporated into this system: alerts that do not pop up and interrupt the participant’s workflow and alerts that do pop up and interrupt the participant’s workflow. For the tested tasks, many participants clicked through and did not review the initial alerts that did not stop the provider’s workflow. At least one provider specified s/he would expect the serious Drug-Drug interaction alert to be a pop up alert and interrupt the workflow. Even with a pop up alert, one provider did not review in detail the Drug-Allergy alert. While missing a Drug Interaction alert can be a critical use error, in these cases we categorized these as a noncritical use errors with the root cause of alert fatigue. The approach used in this application addresses alert fatigue: clinicians know the information is available for review; visual indicators are provided based on the criticality of the alert; and customer sites and providers can configure the alert conditions. Many of the Meaningful Use 3 Safety-Enhanced Design criteria involve alerts and messages. As such, much of the alerting experienced during the usability test session was an artifact of testing. Configuration was purposely not set to control the severity of alerts and the level of alerts presented to the participants. Depending on the scenario, both critical and noncritical alerts appeared. Participants expressed frustration regarding over-alerting in the usability test session and in their daily practice.

EFFECTIVENESS

Performance of both subtasks – reviewing and acting upon the Drug-Allergy Interaction Alert for Ampicillin and Penicillin and reviewing and acting on the Drug-Drug Interaction Alert displayed for Warfarin and Levofloxacin – fell below the 95% success rate. Task failures were associated with alert fatigue, as described above. EFFICIENCY Noncritical usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of this feature.

• Alerts were not always optimized for scanning, which resulted in decreased efficiency. • UI error messaging around required fields can overshadow the drug-drug, drug-allergy interaction

alerts in some cases, as these required field alerts are brighter and closer to the central field of vision than the interaction alerts.

SATISFACTION Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean for Scenario 1, which contained reviewing and acting upon both Drug-Allergy and Drug-Drug interaction alerts, was 4.4 with a standard deviation of 0.8.


MAJOR FINDINGS

Performance of both subtasks – reviewing and acting upon the Drug-Allergy Interaction Alert for Ampicillin and Penicillin and reviewing and acting on the Drug-Drug Interaction Alert displayed for Warfarin and Levofloxacin – fell below the 95% success rate. Task failures were associated with alert fatigue, as described above. Alert fatigue, validated during usability testing, highlights an industry-wide risk concern. Many of the Meaningful Use 3 Safety-Enhanced Design criteria involve alerts and messages. Depending on the scenario, both critical and noncritical alerts appeared. Opportunities to improve the effectiveness and efficiency were identified and are described below.


The athenahealth team is well aware of the industry’s known issue with alert fatigue. The team currently addresses alert fatigue by allowing sites to configure the levels of alerts and by allowing individual prescribers to configure what level(s) of alerts are presented. There are alerts that cannot be ignored because of patient safety. The team has calibrated the system so as to provide safeguards related to these critical alerts. In addition, the athenahealth team would like to call for and be involved with the development and sharing of industry best practices, guidelines, and templates regarding safety-enhanced design that impacts patient safety associated with alerts and alert fatigue. Areas for improvement related to effectiveness and efficiency were also identified related to the visual layout of the content provided in the alerts:

• Explore alternatives to optimize for scanning. • Continue to review cognitive workflow, associated interactions, and the content of pop up alerts

for improved effectiveness and efficiency.


Results: § 170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks – Configuration

Task Mapping Table 12 maps Drug-Drug, Drug-Allergy Interaction Checks – Configuration criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 12. Drug-Drug, Drug-Allergy Interaction Checks – Configuration criteria mapped to usability test tasks. § 170.314 (a)(4) Drug-Drug, Drug-Allergy Interaction Checks for CPOE. (i) Interventions. Before a medication order is completed and acted upon during computerized provider order entry (CPOE), interventions must automatically indicate to a user drug-drug and drug-allergy contraindications based on a patient’s medication list and medication allergy list. (ii) Adjustments. (A) Enable the severity level of interventions provided for drug-drug interaction checks to be adjusted. (B) Limit the ability to adjust severity levels to an identified set of users or available as a system administrative function. Note: This system can only limit the ability to adjust severity levels at the individual level by assigning the individual a prescriber or administrator role. Thus, the criterion was not included in the usability test. To successfully complete the configuration task, participants were required to complete each of the following subtasks. Only activities related to § 170.314 (a)(2) Drug-Drug, Drug-Allergy Interaction Checks – Configuration will be discussed in this chapter. Scenario 10:

Task A: Update settings for interaction warnings Task B: Update settings for interaction warnings

Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, inpatient and ambulatory prescribers attempted this task. Inpatient and ambulatory prescriber data were combined based on the fact that neither the task nor the user characteristics differ based on these user roles. Participants were sent training and asked to complete it at least 24 hours before their session. Only some of the participants completed the training. Those who did not complete the training had to rely on their experience with design patterns and interactions already in place elsewhere in the application. Participants were given the following instruction:

Scenario 10: Task A: You are being notified too often for drug interactions. Change the Drug interaction warning levels from displaying all warnings to only serious and severe warnings. Task B: You are being notified too often for drug interactions. Set the system settings so that the prescriber is warned when there is are only severe drug interaction warnings.

Data Analysis and Reporting – Configuration The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient and ambulatory prescribers updated their configuration settings to limit their drug-


drug and drug-allergy interaction warnings based on severity level. Only the activities associated with configuring the drug-drug, drug-allergy interaction checks criteria are reported in this chapter Sixteen participants attempted the scenario. The usability testing results for the EHRUT are detailed below (see Table 13). Task performance was not differentiated by setting (inpatient, ambulatory). Therefore, the data were not separated by user role. The table lists out the subtasks associated Drug-Drug and Drug-Allergy Interaction Checks – Configuration criteria.

Table 13. Drug-Drug, Drug-Allergy Interaction Checks – Configuration usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)

Errors Task Ratings

(5=Easy) # Percent

(%)


Optimal)

Mean (SD)


Percent (%) **

Mean (SD)

Task 10A: Update settings for interaction warnings

16 88%<> 7/6 91* (37)

91* /21

13%**< 4.0* (1.3) Task 10B: Update

settings for interaction warnings

16 94%> 6/6 6%**

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. < Rounding error. > Some tasks were performed two or more times in testing because these tasks were new to most participants or are performed very infrequently (e.g., configuration tasks). This was because the MU3 Safety Enhanced Design (SED) testing goal is to find true use error issues that persist over time, rather than "discoverability" or "learnability" issues. If higher pass rates, lower click counts, and/or lower task times were seen during additional testing of similar tasks, this is indicative that initial tests more likely showed discoverability issues. A higher pass rate the second time does not mean there are no other issues present, just that the initial lower pass rate is influenced by the participants' unfamiliarity with the part of the product being tested.

Discussion of the Findings The following sections discuss the results organized around a risk analysis of use, test performance and error rates. The risk analysis of use includes identification of use errors and user interface design issues as well as classification of severity based on the consequence of the error. Use errors and user interface design issues that resulted in subtask failures, that are known industry risk issues, and errors and issues related to aspects of the user interface that are configured per customer site are considered more severe compared to noncritical system usability issues related to efficiency. As such the discussion of more serious errors and issues is provided in the Risk Analysis section and the associate mitigation strategy is provided in the Areas for Improvement section. Based on our definition of effectiveness metrics, performance, use errors and issues stemming from task failures are also discussed in the Effectiveness section as effectiveness was measured with task success and failure. Noncritical system usability issues related to efficiency are discussed in the Efficiency section. Associated recommendations are provided in the Areas for Improvement section. Satisfaction was assessed at the scenario level and is discussed in the Satisfaction section.


ERROR ANALYSIS One critical use error was identified or observed for enabling the severity level of interventions provided for drug interaction checks to be adjusted. Some participants had difficulty navigating to complete Task 10A. All participants were able to successfully complete the second subtask. The subtask failure is attributed to the lack of familiarity with this feature and menu labeling confusion. Another critical use error was identified or observed when participants adjusted the severity level of the drug interaction, but did not complete the adjustment due to a control interaction. All participants were able to successfully complete the second subtask. The subtask failure is attributed to the lack of familiarity with this feature and page design. One participant did not choose the correct severity level for the task. The subtask failure is attributed to an artifact of testing. The participant either misunderstood the task or had a short-term memory lapse. After the test, the participant was able to correctly complete the task. EFFECTIVENESS Performance in all tasks fell below the 95% success criterion. As described above, some task failures were associated with lack of familiarity with the screen flow, menu labeling confusion, and location of the action button below the fold. Errors associated with the configuration of drug-drug and drug-allergy interaction checks are described above. EFFICIENCY Noncritical system usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of this feature. Some participants initially selected a different menu option before selecting the correct menu option. This action does not prevent the user from completing the task, but represents additional, unnecessary actions. In addition, no feedback is given that the severity level for the drug interaction has been changed. SATISFACTION Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean was 4.0 with a standard deviation of 1.3. MAJOR FINDINGS Performance on all tasks fell below the 95% success criterion. As described above, task failures were associated with unfamiliarity with the screen flow, menu labeling confusion, and location of the action button below the fold. Opportunities to improve effectiveness and efficiency were identified and are described below. AREAS FOR IMPROVEMENT Areas for improvement related to effectiveness and efficiency include:



Results: § 170.315(a)(5) Demographics - Clinical

Task Mapping Table 14 maps the clinical Demographics criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 14. Demographics – Clinical criteria mapped to usability test tasks. § 170.315(a)(5) Demographics (i) Enable a user to record, change, and access patient demographic data including race, ethnicity, preferred language, sex, sexual orientation, gender identity, and date of birth. (A) Race and ethnicity. (1) Enable each one of a patient’s races to be recorded in accordance with, at a minimum, the standard specified in § 170.207(f)(2) and whether a patient declines to specify race. (2) Enable each one of a patient’s ethnicities to be recorded in accordance with, at a minimum, the standard specified in § 170.207(f)(2) and whether a patient declines to specify ethnicity. (3) Aggregate each one of the patient’s races and ethnicities recorded in accordance with paragraphs (a)(5)(i)(A)(1) and (2) of this section to the categories in the standard specified in § 170.207(f)(1). (B) Preferred language. Enable preferred language to be recorded in accordance with the standard specified in § 170.207(g)(2) and whether a patient declines to specify a preferred language. (C) Sex. Enable sex to be recorded in accordance with the standard specified in § 170.207(n)(1). (D) Sexual orientation. Enable sexual orientation to be recorded in accordance with the standard specified in § 170.207(o)(1) and whether a patient declines to specify sexual orientation. (E) Gender identity. Enable gender identity to be recorded in accordance with the standard specified in § 170.207(o)(2) and whether a patient declines to specify gender identity. (ii) Inpatient setting only. Enable a user to record, change, and access the preliminary cause of death and date of death in the event of mortality. To successfully complete the task, participants were required to complete each of the following subtasks. Only clinical tasks related to § 170.315(a)(5) Demographics will be addressed in this section. Scenario 9: Demographics

Task A: Access and record preliminary cause of death and date of death in the event of mortality. Task B: Change preliminary cause of death and date of death in the event of mortality.

Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, only inpatient prescribers and nurses attempted this task. Prescriber and nurse data were combined based on the fact that neither the task nor the user characteristics differ based on these user roles. Participants were sent training and asked to complete it at least 24 hours before their session. Only some of the participants completed the training. Those who did not complete the training had to rely on their experience with design patterns and interactions already in place elsewhere in the application.

Participants were given the following instruction:

Scenario 9: Task A: Meaningful Use requires that we get your feedback on how the system supports documenting and editing the cause and date of death. You will need to discharge the patient and document the cause of death as acute heart failure and document the date of death as today’s date.

Task B: Now change the information you just entered for James Scott. The diagnosis was not acute heart failure, please change the cause of death as cardiac arrest and document the date of death as yesterday’s date.


Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient prescribers and nurses performed the clinical sections of the Demographics scenario. Only the activities associated with the clinical sections of the Demographics criteria are reported in this chapter. Sixteen participants attempted the scenario to access and record a patient’s cause and date of death. The same participants attempted the scenario to change a patient’s cause and date of death. Task performance was not differentiated by provider type (prescriber, nurse). Therefore, the data was not separated by user role. The usability testing results for the EHRUT are detailed below (see Table 15). The table lists out the usability test results for each clinical subtask to evaluate the Demographics certification criteria.

Table 15. Demographics – Clinical usability test results.

Measure Tasks

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%)


Optimal)

Mean (SD)


Optimal)

Percent (%) **

Mean (SD)

Task 9A.1: Access patient preliminary cause and date of death.

16 100%

23@

/18 208* (59)

208* /56

0%**

4.3* (0.7)

Task 9A.2: Record patient preliminary cause and date of death.

16 100% 0%**

Task 9B: Change patient preliminary cause and date of death.

16 100% 17@

/13 0%**

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the relevant tasks. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. @ Path deviations that were observed in testing but not included in the optimal click count included adding additional information to the EHR specified in the task scenario that many clinicians found relevant (e.g., adding that a diagnosis was a primary diagnosis).

Discussion of the Findings The following sections discuss the results organized around an error analysis, test performance and error rates. The error analysis includes identification of use errors and user interface design issues as well as classification of severity based on the consequence of the error. Use errors and user interface design issues that resulted in subtask failures, that are known industry risk issues, and errors and issues related to aspects of the user interface that are configured per customer site are considered more severe compared to noncritical system usability issues related to efficiency. As such the discussion of more serious errors and issues is provided in the Error Analysis section and the associated mitigation strategy is provided in the Areas for Improvement section. Based on our definition of effectiveness metrics, performance, use errors and issues stemming from task failures are also discussed in the Effectiveness section as effectiveness was measured with task success and failure.


Noncritical system usability issues related to efficiency are discussed in the Efficiency section. Associated recommendations are provided in the Areas for Improvement section. Satisfaction was assessed at the scenario level and at the system level. Both are discussed in the Satisfaction section.

ERROR ANALYSIS

A noncritical use error involved the time of death field. Participants were unsure what was to be entered into this field. Multiple participants skipped over this field, completed the date field, and then went back and entered the time of death. One participant tried putting the date in the time field. The root cause was lack of familiarity and lack of labels or indicators to suggest what was to be entered into this field. One noncritical use error involved confusion between the diagnosis onset date and the date of death. Multiple participants seemed to have confusion between the two dates. Possibly confounding this error may have been an operational difficulty seen for a number of participants where the death date was not required; it should be required. All participants completed the date field and all but one completed the time field. The one participant who did not complete the time field had never done this task before and did not know time was a required field. In addition, one participant changed the diagnosis date but struggled with changing the date of death. That participant, who was a nurse, was run through the provider workflow and not the nurse workflow. The participant was run through the provider workflow due to an operational difficulty. The nurse workflow for this task was not working appropriately; the time of death could not be entered into the system. The participant was able to complete the tasks. Two participants had difficulty navigating to complete the task of accessing the cause and date of death. Both participants had never performed the task before. Each of the participants successfully demonstrated cause and date of death. One participant had initially struggled to remember to complete the last step, clicking an action button, for the task of adding a cause and date of death. The participant successfully completed the cause and date of death. One participant experienced an operational difficulty on changing the cause of death task. The system would not allow the diagnosis to be changed. The participant talked through how the task should be completed. The participant correctly explained how to complete the task and the task was scored as a pass.

EFFECTIVENESS Task performance for all of the subtasks were at or above the 95% success criterion. Errors associated with ordering and changing the date and cause of death tests are described above.

EFFICIENCY

No major noncritical system usability issues that did not result in use errors were identified for improvement related to the efficiency of this feature.

SATISFACTION Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean was 4.3 with a standard deviation of 0.7.


MAJOR FINDINGS Task performance for all of the subtasks were at or above the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below.



• Consider adding a label for the time of death field.


Results: § 170.315(a)(5) Demographics – Registration

Task Mapping Table 16 maps the registration Demographics criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 16. Demographics – Registration criteria mapped to usability test tasks. § 170.315(a)(5) Demographics (i) Enable a user to record, change, and access patient demographic data including race, ethnicity, preferred language, sex, sexual orientation, gender identity, and date of birth. (A) Race and ethnicity. (1) Enable each one of a patient’s races to be recorded in accordance with, at a minimum, the standard specified in § 170.207(f)(2) and whether a patient declines to specify race. (2) Enable each one of a patient’s ethnicities to be recorded in accordance with, at a minimum, the standard specified in § 170.207(f)(2) and whether a patient declines to specify ethnicity. (3) Aggregate each one of the patient’s races and ethnicities recorded in accordance with paragraphs (a)(5)(i)(A)(1) and (2) of this section to the categories in the standard specified in § 170.207(f)(1). (B) Preferred language. Enable preferred language to be recorded in accordance with the standard specified in § 170.207(g)(2) and whether a patient declines to specify a preferred language. (C) Sex. Enable sex to be recorded in accordance with the standard specified in § 170.207(n)(1). (D) Sexual orientation. Enable sexual orientation to be recorded in accordance with the standard specified in § 170.207(o)(1) and whether a patient declines to specify sexual orientation. (E) Gender identity. Enable gender identity to be recorded in accordance with the standard specified in § 170.207(o)(2) and whether a patient declines to specify gender identity. (ii) Inpatient setting only. Enable a user to record, change, and access the preliminary cause of death and date of death in the event of mortality. To successfully complete the task, participants were required to complete each of the following subtasks. Only clinical tasks related to § 170.315(a)(5) Demographics will be addressed in this section. Scenario 11: Demographics

Task A: Access and record demographics for a patient including race, ethnicity, preferred language, sex, sexual orientation, gender identity, and date of birth. Task B: Change demographics for a patient including race, ethnicity, preferred language, sex, sexual orientation, gender identity, and date of birth.

Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, only registration specialists attempted this task. Inpatient and ambulatory registration specialists’ data was combined based on the fact that neither the task nor the user characteristics differ based on these user roles. Participants were given the following instruction:

Scenario 11 Task A (Inpatient): You need to start a record for a new patient and you need to document some demographic items. Using the system, document this new information: • Last Name: Kelly • First Name: Chris • Sex: Female • Date of Birth: 11/ 7/ 1980 • Address: 123 Main Street Schenectady, NY 12345 • Home phone: 555-555-5555 • Contact preference: Mail


• Primary Care Provider: None • Language: English • Race: Declined to provide • Ethnicity: Declined to provide • Sexual Orientation: Declined to provide • Gender Identity: Declined to provide • Level of care: Inpatient • Department: Local Community Hospital • Check-in: Today • Admission source: 2- Clinic or Physician’s Office • Admission Type: 3 - Elective • Attending provider: Pat Rogers • Admitting provider: Charles Brown • Insurance: Self Pay • Patient’s relationship to guarantor: Self Task A (Ambulatory): You need to start a record for a new patient and you need to document some demographic items. Using the system, document this new information:

• Last Name: Kelly • First Name: Chris • Date of Birth: 11/ 7/ 1980 • Sex: Female • Mobile Phone: None • Email: None • Usual Provider: George Cohen • Race: Declined to provide • Ethnicity: Declined to provide • Language: English • Sexual Orientation: Declined to provide • Gender Identity: Declined to provide • Consent to Call: No Task B (Inpatient): The patient states he was feeling poorly and completed his paperwork as fast as he could. He would like to complete the paperwork more thoroughly and accurately. He returns the paperwork. Please change the following information in the patient’s file: • Sex: Male • Date of Birth: 1/17/ 1980 • Preferred Language: Spanish • Race: Black or African American; European (2 races) • Ethnicity: Dominican • Sexual Orientation: Don’t know • Gender Identity: Genderqueer Task B (Ambulatory): The patient states he was feeling poorly and completed his paperwork as fast as he could. He would like to complete the paperwork more thoroughly and accurately. He returns the paperwork. Please change the following information in the patient’s file: • Date of Birth: 1/17/ 1980 • Sex: Male • Race: Black or African American; European • Ethnicity: Dominican • Preferred Language: Spanish • Sexual Orientation: Don’t know • Gender Identity: Genderqueer, neither exclusively male nor female


Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient and ambulatory registration specialists performed the access, record and change demographics scenario. Only the activities associated with the Demographics criteria, not including the inpatient mortality demographics, are reported in this chapter. Fifteen registration specialists attempted the scenario to access and record patient demographic data. The same registration specialists attempted the scenario to change patient demographic data. Task performance was not differentiated by setting (inpatient, ambulatory). Therefore, the data was not separated by user role for the setting. The usability testing results for the EHRUT are detailed below (see Table 17). The table lists out the usability test results for each registration subtask to evaluate the Demographics certification criteria.

Table 17. Demographics – Registration usability test results. Measure Tasks

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%)


Optimal)

Mean (SD)


Optimal)

Percent (%) **

Mean (SD)

Task 11A: Access and record demographics for a patient:

A.1: Sex

15

93%

17~

/23 362* (249)

362* /101

7%**

4.3* (0.8)

A.2: DOB 100% 0%** A.3: Language 93% 7%** A.4: Race 93% 7%** A.5: Ethnicity 93% 7%** A.6: Gender identity 73% 27%** A.7: Sexual Orientation

73% 27%**

Task 11B: Change demographics for a patient:

B.1: Sex

15

100%

23~

/24 215* (94)

215* /53

0%**

3.8* (1.3)

B.2: DOB 100% 0%** B.3: Language 100% 0%** B.4: Race (1) 100% 0%** B.5: Race (2) 73% 27%** B.6: Ethnicity 100% 0%** B.7: Gender Identity 93% 7%** B.8: Sexual Orientation 100% 0%**

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the relevant tasks. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. < Rounding error. ~ Some participants did not complete all of the required fields. > Some tasks were performed two or more times in testing because these tasks were new to most participants or are performed very infrequently (e.g., configuration tasks). This was because the MU3 Safety Enhanced Design (SED) testing goal is to find true use error issues that persist over time, rather than "discoverability" or "learnability" issues. If higher pass rates, lower click counts, and/or lower task times were seen during additional testing of similar tasks, this is indicative that initial tests more likely showed discoverability issues. A higher pass


rate the second time does not mean there are no other issues present, just that the initial lower pass rate is influenced by the participants' unfamiliarity with the part of the product being tested.


ERROR ANALYSIS

Critical use errors were associated with recording and changing patient demographic data. One use error was due to registration incompleteness and/or failure to save. Participants failed to complete a field due to unfamiliarity, oversight, or training to only complete required fields when registering a patient. A few participants did not complete Gender Identity or Sexual Orientation due to unfamiliarity or training. One participant believed Gender Identity and Gender were synonymous. Another critical use error identified or observed when recording and changing patient demographic data was that participants failed to save the information they entered. The root cause was oversight. One participant said they were trying to do the task exactly as described and there were not instructions to save. Most participants who did not save, said they would have saved if doing this in the real world. Many of these were artifacts of testing. A critical use error was associated with changing patient demographic data. Several participants could not figure out how to change Language, Race, or Ethnicity. The root cause for this error was unclear interactions and a nonsalient error message.

Noncritical use errors were observed with recording patient demographic data. One participant said the font was small.

EFFECTIVENESS Performance of the sub-tasks was at or above the 95 % success criterion for: DOB, change Sex, change DOB, change Language, change Race (1), change Ethnicity, and change Sexual Orientation. Performance of the sub-tasks was below the 95% success criterion for Sex, Language, Race, Ethnicity, Gender Identity, Sexual Orientation, change Race (2), and change Gender Identity. The contributing factors were unfamiliarity, oversight, interaction design confusion, or training to only complete required fields when registering a patient.


EFFICIENCY

Noncritical system usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of this feature. As mentioned above, one participant said the font was small. SATISFACTION Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. For the first task to record, the mean was 4.3 with a standard deviation of 0.8 and to change demographics for a patient the mean was 3.8 with a standard deviation of 1.3. MAJOR FINDINGS Performance of the sub-tasks was at or above the 95 % success criterion for: DOB, change Sex, change DOB, change Language, change Race (1), change Ethnicity, and change Sexual Orientation. Performance of the sub-tasks was below the 95% success criterion for Sex, Language, Race, Ethnicity, Gender Identity, Sexual Orientation, change Race (2), and change Gender Identity. Opportunities to improve the effectiveness and efficiency were identified and are described below.



• Review icons for saliency. • Continue to review the cognitive workflow and associated interactions to improve effectiveness

and efficiency.


Results: § 170.315(a)(6) Problem List

Task Mapping Table 18 maps the Problem List criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 18. Problem List criteria mapped to usability test tasks. § 170.315 (a)(6) Problem List. Enable a user to record, change, and access a patient's active problem list: (i) Ambulatory setting only. Over multiple encounters in accordance with, at a minimum, the version of the standard specified in § 170.207(a)(4). (ii) Inpatient setting. For the duration of an entire hospitalization in accordance with, at a minimum, the version of the standard specified in § 170.207(a)(4). To successfully complete this task, participants were required to complete each of the following subtasks. Only subtasks associated with § 170.315 (a)(6) Problem List are discussed in this chapter. Scenario 4

Task I: Access problem list Task J: Change problem (Childhood Asthma) Task K: Record problem (Acute Laryngitis) Task L: Review current and previous problems

Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, both inpatient and ambulatory prescribers and nurses / clinical staff attempted this task. Inpatient and ambulatory prescriber and nurse / clinical staff data were combined, where appropriate, based on the fact that neither the task nor the user characteristics differ based on these user roles. Participants were given the following instruction:

Scenario 4 Your next patient is a 55-year old male with a history of rheumatoid arthritis whose chief complaint was sinus pressure. He has a diagnosis of sinus infection. You want to go over his chart with him. You start by reviewing his allergies. Please name one of the medications the patient is allergic to? Task I (Inpatient): Please locate the patient’s diagnoses list. Task I (Ambulatory): Please locate the patient’s problem list. Task J, K: While reviewing the patient’s problem list, you notice an old diagnosis of childhood asthma is still listed as a current problem. Please update the system so that the childhood asthma is no longer listed. Also add that the patient has acute laryngitis. Task L: When was the patient’s diabetes mellitus originally diagnosed?

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient and ambulatory prescribers and inpatient and ambulatory nurses / clinical staff performed the Problem List scenario. Only the activities associated with the Problem List criteria are reported in this chapter. Twenty participants attempted the scenario to access the problem list and add and update a problem. Task performance was not differentiated by provider type (prescriber, nurse) or setting (inpatient, ambulatory).


Therefore, the data was not separated by user role. The usability testing results for the EHRUT are detailed below (see Table 19). The table lists out the usability test results for each subtask associated with the Problem List criteria.

Table 19. Problem List usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%) Deviations


Mean (SD)


Percent (%) **

Mean (SD)

Task 4I: Access problem/diagnoses list

20 100% 1/1

58* (41)

58* /20

0%**

4.5* (0.9)

Task 4J: Change problem (Childhood Asthma)

20 95% 4/4 5%**

Task 4K: Record problem (Acute Laryngitis)

20 100% 5/5 0%**

Task 4L: Review current and previous problems

20 100% 1/1 0%**

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures.



ERROR ANALYSIS One critical use error was identified or observed as part of the change a problem task. One participant clicked the X icon in the corner to start the removal process, but then clicked “Delete (added in error)”. S/he saw the Delete checkbox become available after clicking the X icon and did not read the entire label. S/he just assumed it was needed to check it to complete the removal from the active problems list since it was now available. The root cause of this error was a misunderstanding of the usage of the checkbox. EFFECTIVENESS

Task performance for all subtasks associated with accessing, adding, and changing a problem were at or above the 95% success criterion. Errors associated with accessing, ordering, and changing problems are described above.

EFFICIENCY

Noncritical usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of this feature. One participant tried to specifically add childhood asthma to the patient’s historical problems list. S/he had not realized that simply removing it from the active problems list would automatically move it over. Many participants did not understand how to close data panels by clicking on the title. They would instead just leave them open once they had completed acting on the item within the panel.

SATISFACTION

Satisfaction levels were rated at the system level and at a scenario level. Refer the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included the subtasks to access the problem list and add and change a problem, on a 1 (very difficult) to 5 (very easy) scale. The mean was 4.5 with a standard deviation of 0.9. MAJOR FINDINGS

Task performance for all subtasks associated with accessing, adding, and changing a problem were at or above the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below.

AREAS FOR IMPROVEMENT Areas for improvement related to effectiveness and efficiency include:



Results: § 170.315(a)(7) Medication List

Task Mapping Table 20 maps the Medication List criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 20. Medication List criteria mapped to usability test tasks. § 170.315 (a)(7) Medication list. Enable a user to record, change, and access a patient's active medication list as well as medication history: (i) Ambulatory setting. Over multiple encounters. (ii) Inpatient setting. For the duration of an entire hospitalization. To successfully complete this task, participants were required complete each of the following subtasks. Only subtasks associated with § 170.315 (a)(7) Medication list are discussed in this chapter. Scenario 4

Task E: Access medication list Task F: Review current and previous medications Task G: Change medication (Azithromycin) Task H: Record medication [Aldactone (spironolactone)]


Scenario 4 Your next patient is a 55-year old male with a history of rheumatoid arthritis whose chief complaint was sinus pressure. He has a diagnosis of sinus infection. Task E: (Inpatient) You continue reviewing Mr. Benson’s chart, please locate his active home medications. Task E: (Ambulatory) You continue reviewing Mr. Benson’s chart, please locate his current medications Task F: You see that Mr. Benson is currently taking Cardizem but you wonder if he has ever taken Lipitor. Please look to see if he may have taken Lipitor in the past. Task G, H: (Inpatient) Mr. Benson tells you his PCP discontinued his Azithromycin today. Please discontinue the Azithromycin from the patient’s home medication list. Also, the patient tells you that his Endocrinologist has put him on Aldactone (spironolactone) 25mg daily for high blood pressure a month ago. If Aldactone is not in the chart, add that to the patient’s home medication list. Task G, H: (Ambulatory) Discontinue the Azithromycin, because he completed the course. Also, the patient tells you that his Endocrinologist has put him on Aldactone (spironolactone) 25mg daily for high blood pressure a month ago. If Aldactone is not in the chart, add that to the patient’s medication list.

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient and ambulatory prescribers and inpatient and ambulatory nurses / clinical staff performed the Medication List scenario. Only the activities associated with the Medication List criteria are reported in this chapter.


Twenty participants attempted the scenario to access the medication list and add and update a medication. Task performance was not differentiated by provider type (prescriber, nurse) or setting (inpatient, ambulatory). Therefore, the data was not separated by user role. The usability testing results for the EHRUT are detailed below (see Table 21). The table lists out the usability test results for each subtask associated with the Medication List criteria.

Table 21. Medication List usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%) Deviations


Mean (SD)


Percent (%) **

Mean (SD)

Task 4: Access medication list 20 100% 2/2

114* (76)

114* /35

0%**

4.3* (0.9)

Task 4F: Review current and previous medications

20 100% 1/1 0%**

Task 4G: Change medication (Azithromycin)

20 100% 6/5 0%**

Task 4H: Record medication [Aldactone (spironolactone)]

20 95% 10@/7 5%**

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. @ Path deviations that were observed in testing but not included in the optimal click count included adding additional information to the EHR specified in the task scenario that many clinicians found relevant (e.g., adding that a diagnosis was a primary diagnosis).



ERROR ANALYSIS One critical use error was identified or observed as part of this usability task. One participant failed to add the medication to the chart. After filling out all of the needed information, s/he clicked off the dialog instead of pressing the Add button. This interaction pattern works in many other workflows, including several of the update workflows, but not in the initial add workflow. This inconsistency can confuse users and result in lost data.

EFFECTIVENESS

Task performance for all subtasks associated with reviewing, adding, and changing a patient’s active and historical medications were at or above the 95% success criterion. Errors associated with accessing, ordering, and changing medications are described above.

EFFICIENCY

Noncritical usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of the medication list feature. Some participants initially added the medication to the patient’s chart without a discrete sig. When prompted, they were able to do it correctly, but only by completely removing their first entry and adding a new version of the medication. Some inpatient participants initially had trouble finding the link to the Home Medications. This is due to the fact that this link is at the very bottom of the screen, away from the rest of the medication list information. Once participants found this link, they were able to find it again during the rest of the tasks. One participant tried to use the Find function to look for the Lipitor and was unsuccessful even though the medication is in the patient’s chart. S/he was able to find the Lipitor by navigating to the patient’s Home Medication list. Many participants did not understand how to close data panels by clicking on the title. They would instead just leave them open once they had completed acting on the item within the panel.

SATISFACTION

Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean was 4.3 with a standard deviation of 0.9 for this scenario.

MAJOR FINDINGS

Task performance for all subtasks associated with reviewing, adding, and changing a patient’s active and historical medications were at or above the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below. AREAS FOR IMPROVEMENT



• Consider providing an easy way to convert a free-form medication sig to a structured sig. • Allow the Find function to search all items in the chart regardless of source.


Results: § 170.315(a)(8) Medication Allergy List

Task Mapping Table 22 maps the Medication Allergy List criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 22. Medication Allergy List criteria mapped to usability test tasks. § 170.315 (a)(8) Medication Allergy List. Enable a user to record, change, and access a patient's active medication allergy list as well as medication allergy history: (i) Ambulatory setting. Over multiple encounters. (ii) Inpatient setting. For the duration of an entire hospitalization. To successfully complete the task, participants were required to complete each of the following subtasks. Only tasks related to § 170.315 (a)(8) Medication Allergy List will be addressed in this section. Scenario 4

Task A: Access medication allergy list Task B: Record new medication allergy (Iodine) Task C: Change medication allergy (Sulfa) Task D: Review current and previous medication allergies


Scenario 4 (Inpatient/Outpatient) Your next patient is a 55-year old male with a history of rheumatoid arthritis whose chief complaint was sinus pressure. He has a diagnosis of sinus infection. You want to go over his chart with him. You start by reviewing his allergies. Task A: Please name one of the medications the patient is allergic to. Task B, C: Mr. Benson’s wife arrives as you are reviewing his allergies. She tells you he gets a severe rash to iodine, which he always forgets to mention. He confirms the allergy. Add this allergy to the patient’s current allergy list. In addition, Mr. Benson’s wife reminds him that he started getting a mild headache, besides a rash, due to his sulfa allergy, and he agrees. Add a reaction of “headache” with mild severity to his sulfa allergy. Task D: When was the patient’s first reaction to penicillin?

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient and ambulatory prescribers and inpatient and ambulatory nurses / clinical staff performed the Medication Allergy List scenario. Only the activities associated with the Medication Allergy List criteria are reported in this chapter. Twenty participants attempted the scenario to access the medication allergy list and add and update a medication allergy. Task performance was not differentiated by provider type (prescriber, nurse) or setting (inpatient, ambulatory). Therefore, the data was not separated by user role. The usability testing results for the


EHRUT are detailed below (see Table 23). The table lists out the usability test results for each subtask in the Medication Allergy List task.

Table 23. Medication Allergy List usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%) Deviations


Mean (SD)


Optimal)

Percent (%) **

Mean (SD)

Task 4A: Access medication allergy list

20 100% 2/2

112* (52)

112* /43

0%**

4.5* (0.7)

Task 4B: Record new medication allergy (Iodine)

20 100% 10/10 0%**

Task 4C: Change medication allergy (Sulfa)

20 90% 8/7 10%**

Task 4D: Review current and previous medication allergies

20 100% 1/1 0%**




ERROR ANALYSIS A critical use error was identified or observed when some participants did not initially see the Add Reaction button. These participants instead changed the existing reaction already recorded. These participants struggled to complete the task, and two of these participants were unable to complete the task. The root cause was unfamiliarity and the Add Reaction button is not in the same relative location as other add buttons within the application.

Two participants tried to add (as new) an existing allergy and received an error message that was not salient as it looked the same as normal instructional text. Neither of the participants noticed it. Their first indication that there was an issue was once they had created the allergy and added the reaction but the Add button did not become active. One participant figured out the issue when rereading the task; the other struggled to complete the task. Both were able to complete the task once they realized the issue. Some participants were looking for an Update button to save their work when they were done updating the medication allergy. One participant clicked on the X icon to close the window and ended up in the delete workflow. S/he was able to recover, but did not understand how s/he had done so. Other participants tried to click the X icon to open items. None of these instances resulted in lost data during testing, but could potentially do so during normal workflow. EFFECTIVENESS Performance of one subtask, changing medication allergy, fell below the 95% success rate. This was due to the participants’ changing the existing reaction already recorded instead of adding an additional reaction when updating a current medication allergy. Errors associated with accessing, ordering, and changing problems are described above. EFFICIENCY Noncritical usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of this feature. Most participants had issues with scrolling when adding a rash to an allergy. Adding “rash” requires scrolling past the initial list presented when the dropdown is opened, and doing an “infinite” scroll when the user reaches the end of the list currently presented using the arrow keys. The dropdown forces the user to return to the mouse and click on the down arrow or use the scroll wheel to reach the next part of the list. This adds additional clicks. The athenahealth team has already updated the system to mitigate this issue. Some users attempted to use type-ahead when adding the reaction of “headache,” but none were successful in using the keyboard to make “headache” come up. All resorted to using the mouse to navigate the dropdown list. Many participants did not understand how to close data panels by clicking on the title. They would instead just leave them open once they had completed acting on the item within the panel.

SATISFACTION

Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean was 4.5 with a standard deviation of 0.7.


MAJOR FINDINGS

Performance of the subtask changing the medication allergy fell below the 95% success criterion. Performance for all other subtasks was at or above 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below.



• Consider supporting an Undo function. • Continue to review cognitive workflow and associated interactions, including error messages, for

improved effectiveness and efficiency.


Results: § 170.315(a)(9) Clinical Decision Support – Clinical

Task Mapping Table 24 maps the Clinical Decision Support – Evidence-Based Decision Support Interventions – clinical criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. In addition, the Linked Referential CDS and CDS – Source Attributes criteria are provided. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 24. Clinical Decision Support – Clinical criteria mapped to usability test tasks.

§ 170.315(a)(9) Clinical Decision Support (CDS) (i) CDS intervention interaction. Interventions provided to a user must occur when a user is interacting with technology. (ii) CDS configuration. (A) Enable interventions and reference resources specified in paragraphs (a)(9)(iii) and (iv) of this section to be configured by a limited set of identified users (e.g., system administrator) based on a user’s role. (B) Enable interventions: (1) Based on the following data: (i) Problem list; (ii) Medication list; (iii)Medication allergy list; (iv)At least one demographic specified in paragraph (a)(5)(i) of this section; (v) Laboratory tests; and (vi) Vital signs. (2) When a patient’s medications, medication allergies, and problems are incorporated from a transition of care/referral summary received and pursuant to paragraph (b)(2)(iii)(D) of this section. (iii)Evidence-based decision support interventions. Enable a limited set of identified users to select (i.e., activate) electronic CDS interventions (in addition to drug-drug and drug-allergy contraindication checking) based on each one and at least one combination of the data referenced in paragraphs (a)(9)(ii)(B)(1)(i) through (vi) of this section. (iv)Linked referential CDS. (A) Identify for a user diagnostic and therapeutic reference information in accordance at least one of the following standards and implementation specifications: (1) The standard and implementation specifications specified in § 170.204(b)(3). (2) The standard and implementation specifications specified in § 170.204(b)(4). (B) For paragraph (a)(9)(iv)(A) of this section, technology must be able to identify for a user diagnostic or therapeutic reference information based on each one and at least one combination of the data referenced in paragraphs (a)(9)(ii)(B)(1)(i), (ii), and (iv) of this section. (v) Source attributes. Enable a user to review the attributes as indicated for all CDS resources: (A) For evidence-based decision support interventions under paragraph (a)(9)(iii) of this section: (1) Bibliographic citation of the intervention (clinical research/guideline); (2) Developer of the intervention (translation from clinical research/guideline); (3) Funding source of the intervention development technical implementation; and (4) Release and, if applicable, revision date(s) of the intervention or reference source. (B) For linked referential CDS in paragraph (a)(9)(iv) of this section and drug-drug, drug-allergy interaction checks in paragraph (a)(4) of this section, the developer of the intervention, and where clinically indicated, the bibliographic citation of the intervention (clinical research/guideline). To successfully complete the clinical task, participants were required to detect and acknowledge automatically generated Clinical Decision Support alerts as they completed each of the following tasks. Only tasks related to § 170.315 (a)(9) Clinical Decision Support – Clinical criteria will be discussed in this chapter.


Table 24. Clinical Decision Support – Clinical criteria mapped to usability test tasks.

Scenario 1 Task C: Access source attributes for the alert content (Drug x Allergy)

Scenario 7.1:

Task A: Access linked referential decision support for CDS demographics and diagnosis alert (children with diabetes) Task B: Access source attributes for the InfoButton content Task C: Access linked referential decision support for CDS problems alert (asthma) Task D: Access source attributes for the InfoButton content Task E: Access linked referential decision support for CDS medications alert (ADHD medication)

Scenario 7.2: Task A: Order Ritalin 10mg based on CDS alert (CDS: Medications) Task B: Record completion of pediatric diabetes recommendation based on CDS alert (CDS Combo: Demographics and Diagnosis)

Scenario 7.3: Task A: Record completion of breast cancer screening based on CDS alert (CDS: Demographics) Task A1: Access source attributes for the CDS alert content (CDS: Problems) Task B: Record completion of a foot exam based on CDS alert (CDS: Problems) Task C: Access source attributes for the CDS alert content (CDS: Lab Tests) Task D: Record results of LDL test based on CDS alert (CDS: Lab Tests) Task E: Document screening plan for Hypertension based on CDS alert (CDS: Vital Signs) Task F: Verify there is no CDS alert regarding shingles vaccine (CDS: Med Allergy)

Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, inpatient and ambulatory prescribers completed all tasks, and inpatient and ambulatory nurses / clinical staff attempted the tasks in Scenario 7.1. Prescriber and nurse data were combined for Scenario 7.1 based on the fact that neither the task nor the user characteristics differ based on these settings or roles. Inpatient and ambulatory prescriber data were combined for Scenarios 7.2 and 7.3 based on the fact that neither the task nor the user characteristics differ significantly based on these settings. Participants were not given instructions related to Clinical Decision Support alerts, which were displayed when participants attempting the following instruction:

Scenario 1 You are Pat Rogers providing care for a new patient. The patient is a 58-year old male whose chief complaint is respiratory distress. He has a diagnosis of pneumonia. Please order Ampicillin 500mg for the patient’s pneumonia. If any alerts/warnings come up during the session, please let me know what the alert or warning is telling you and how you would typically handle the alert in your practice but please wait and let me tell you how to handle the alert for the situation. Task C: What is the source of this alert? (Drug- Allergy interaction between Ampicillin and existing severe Penicillin allergy).

Scenario 7.1 Your next patient is 8-year-old Jackson Wallace. You want to see if the system has provided any meaningful use recommendations for this patient in their chart. Task A: The EHR provides you the ability to get additional information about Pediatric Diabetes from an outside source. Please go to where you would find this additional information. Task B: Who is the developer of this information? Task C: You would like to get additional information about children with asthma. Find this information. Task D: Who is the developer of this information? Task E: You would like to get additional information about ADHD medication. Find this information. Scenario 7.2


Your patient is still Jackson Wallace. Please go back to the meaningful use recommendations for this patient. Tasks A and B: Using the recommendations, complete the ADHD medication recommendation and order Ritalin 10mg. Also indicate the Pediatric Diabetes recommendation was completed. Scenario 7.3 Your next patient is a 60-year-old female. Please use the system provided meaningful use recommendations to indicate the items have been completed. Task A: Your patient says she had her Breast Cancer screening on March 10. Please specify that. Task A1: Who is the developer of the foot exam intervention? Task B: (Complete the foot exam today.) Task C: Who is the developer of the LDL recommendation? Task D: Go ahead and do what the LDL Recommendation tells you to do. The information you need for the LDL is: Today’s date and a result of 80. Task E: (Complete the High Blood Pressure Care Plan today) Task F: Your patient has not had a shingles vaccine; however, the patient has an allergy to neomycin which is an ingredient in the shingles vaccine. Please verify the patient has an allergy to neomycin and verify there is NO recommendation advising the patient get a shingles vaccine.

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Prescribers and nurses interacted with Clinical Decision Support alerts as they were presented in the context of the tasks. Only the review and act-upon activities associated with the Clinical Decision Support – Clinical criterion are reported in this chapter.

CDS Intervention Interaction and Evidence-Based Decision Support Interventions

Seventeen inpatient and ambulatory prescriber participants received specific clinical decision support during the scenarios. Task performance for Scenario 7.1 was not differentiated by provider type (prescriber, nurse) or setting (inpatient, ambulatory). Therefore, the data was not separated by user role or setting. Task performance for Scenarios 1, 7.2, and 7.3 were not differentiated by setting (inpatient, ambulatory). Therefore, the data was not separated by user role for the setting. The usability testing results for the EHRUT are detailed below (see Table 25). The table lists out the subtasks associated with Clinical Decision Support – Clinical criteria.


Table 25. Clinical Decision Support – Clinical usability test results. Measure Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%)


Optimal)

Mean (SD)


Percent (%) **

Mean (SD)

Task 7.2A: Order Ritalin 10mg based on CDS alert (CDS: Medications)

14^

Performance after initial exposure to icon.

100% 9/6

170* (70)

170* /30

0%**

3.8* (1.0)

Task 7.2B: Record completion of pediatric diabetes recommendation based on CDS alert (CDS Combo: Demographics and Diagnosis)

17 100% 3/3 0%**

Task 7.3A: Record completion of breast cancer screening based on CDS alert (CDS: Demographics)

17

Performance after initial exposure to icon.#

88% 8/8

193* (52)

193* /58

12%**

4.4* (0.7)

Task 7.3B: Record completion of foot exam based on CDS alert (CDS: Problems)

17 100% 3/3 0%**

Task 7.3D: Record results of LDL test based on CDS alert (CDS: Lab Tests)

17 100% 5/5 0%**

Task 7.3E: Document screening plan for hypertension based on CDS alert (CDS: Vital Signs)

17 100% 3/3 0%**

Task 7.3F: Verify there is no CDS alert for a shingles vaccine (CDS: Med Allergy)

16^ 100% 3/3 0%**

* Mean (SD) Task Time is the time to complete the entire Task. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. # A few participants did not complete Scenario 7.2, and thus Scenario 7.3 was their first CDS task to complete.

Linked Referential CDS

Twenty participants attempted the scenario to locate linked referential information for a problem, linked referential information for medications, and linked referential information for a combined demographic and diagnosis. Task performance was not differentiated by provider type (prescriber, nurse) or setting (inpatient, ambulatory). Therefore, the data was not separated by user role or setting. The usability testing results for the


EHRUT are detailed below. Table 26 below provides the usability test results associated with accessing linked referential decision support.

Table 26. Accessing Linked Referential Decision Support usability test results. Measure

Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%) Deviations


Mean (SD)


Optimal)

Percent (%) **

Mean (SD)

Task 7.1A: Access linked referential decision support for CDS demographics and diagnosis alert (children with diabetes)

20

Initial Exposure for some participants who had not performed this type of task during their careers.

65%> 2/1

130* (80)

130* /22

35%**

4.1* (1.1)

Task 7.1C: Access linked referential decision support for CDS problems alert (asthma)

20 100%> 1/1 0%**

Task 7.1E: Access linked referential decision support for CDS medications alert (ADHD medication)

20 100% 1/1 0%**

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. > Some tasks were performed two or more times in testing because these tasks were new to most participants or are performed very infrequently (e.g., configuration tasks). This was because the MU3 Safety Enhanced Design (SED) testing goal is to find true use error issues that persist over time, rather than "discoverability" or "learnability" issues. If higher pass rates, lower click counts, and/or lower task times were seen during additional testing of similar tasks, this is indicative that initial tests more likely showed discoverability issues. A higher pass rate the second time does not mean there are no other issues present, just that the initial lower pass rate is influenced by the participants' unfamiliarity with the part of the product being tested.

Access Source Attributes

Sixteen ambulatory prescriber participants attempted the task to access source attributes for drug-allergy interaction checks. Twenty participants from all user groups (i.e., inpatient, ambulatory, provider, nurse / clinical staff) attempted the tasks to access source attributes for InfoButton content. Seventeen inpatient and ambulatory prescriber participants attempted the tasks to access source attributes for CDS alert content. Task performance was not differentiated by provider type (prescriber, nurse) for accessing source attributes for InfoButton content. Task performance was not differentiated by setting (inpatient, ambulatory) for accessing source attributes for Infobutton content or for CDS alert content. Therefore, the data was not separated by user role, where appropriate, or setting. Table 27 below provides the usability test results associated with accessing source attributes.


Table 27. Accessing Source Attributes usability test results. Measure

Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%) Deviations


Mean (SD)


Percent (%) **

Mean (SD)

Task 1C: Access source attributes for alert content (Drug x Allergy Interaction)

15^ 80% 0/0 327* (76)

327* /72

20%** 4.4* (0.8)

Task 7.1D: Access source attributes for InfoButton content


100% 0/0 130* (80)

130* /22

0%** 4.1* (1.1)

Task 7.3C: Access source attributes for CDS alert content


88% 1/1 193* (52)

193* /58

12%** 4.4* (0.7)

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. ^ The data was not included for one participant due to an artifact or testing. The participant did not understand the question despite repeated attempts to explain it.


ERROR ANALYSIS Alert fatigue is a known issue across the industry and was validated during usability testing. Alert fatigue is a serious issue that plagues providers. Potential risks related to alert fatigue include desensitization to alerts and/or overreliance on alerts. Throughout the usability test session, participants interacted with a number of clinical decision support alerts. In some cases, the participant carefully reviewed the alert. In other cases, the participant routinely dismissed the alert. Many of the Meaningful Use 3 Safety-Enhanced Design criteria involve alerts and messages that interrupt the workflow. As such, some of the alerting experienced during the usability test session was an artifact of testing. Configuration purposely was not set to control the severity of alerts and the level of alerts


presented to the participants. Depending on the scenario, both critical and noncritical alerts appeared. Participants expressed frustration regarding over-alerting in the usability test session and in their daily practice. CDS Intervention Interaction and Evidence-Based Decision Support Interventions Critical use errors existed for clinical decision support intervention recommendations that did not automatically interrupt the participant’s workflow. Some participants had difficulty finding where these recommendations existed in the user interface and had to be directed to the correct area of the screen, though once the association was learned, participants were generally able to find their way back to the correct location on subsequent tasks. A critical use error was found when recording completion of CDS alerts that were performed on a previous day. For many of the alert items, the system will automatically fill in the date when the completion checkbox is clicked. If the user then clicks anywhere on the screen, the system assumes they are done with that item and records it as done with the current date. In this case, however, there is no discernible way to update the item to correct the date. Another critical use error was found when prescribing Ritalin to complete the CDS recommendation for Medications. When using the provided shortcut to add the medication to the Orders list, there was no feedback to the participant, nor was the participant given an easy way to move to that screen to finish acting on the alert. Additionally, in some cases, Ritalin was already on the patient’s medication list, but the participant was not clearly warned until they went to sign the order. Linked Referential CDS Noncritical use error issues were found around the use of the InfoButton. Many users did not recognize the epocrates “e” icon as a clickable button within the context of the CDS alerts. Most of these participants tried clicking on an informational link to get additional information instead. When asked about this behavior, participants either said they did not recognize the “e” as an actionable button at all, or recognized it as the epocrates logo but did not associate epocrates with anything other than medication and so did not recognize it as clickable in this context. After completing the initial exposure, however, all participants were successful in using the “e” button to find linked referential information. Access Source Attributes Generally, lower success percentages associated with accessing source attributes were influenced by participants’ not understanding why the task was required since it is not a task they would do.

EFFECTIVENESS Performance for recording completion of clinical decision support alerts was at or above the 95% success criterion for documentation of items performed on today’s date (Tasks 7.2B, 7.3B, 7.3D, and 7.3E). Performance on prescribing a medication to fulfill the CDS Medication criteria (Task 7.2A) was at or above the 95% success criterion. Performance on verifying there was no CDS alert for a vaccine the patient is allergic to, to fulfill the CDS Medication Allergy criteria (Task 7.3F) was at or above the 95% success criterion. Performance of recording completion of clinical decision support alerts was below the 95% acceptance criterion for documentation of items performed on a previous date Task 7.3A). As discussed above, contributing factors included the automatic addition of today’s date followed by an accidental click outside the CDS item, which automatically saved, with no way to edit the saved item to correct the date. Performance of identifying the InfoButton was above the 95% acceptance criterion once users had an initial exposure to the InfoButton. For issues during the initial exposure, the root cause was that many participants did not initially recognize the epocrates logo as a button or they did not initially recognize


the epocrates logo as a button in the context of CDS since they stated that they thought it was only a “medication thing.” Performance of identifying the source attributes of linked referential content found through the InfoButton, after initial exposure, was at or above the 95% acceptance criterion. Performance of identifying the drug-allergy alert source attributes and identifying CDS intervention source attributes was below the 95% acceptance criterion after first exposure to the feature. The contributing factor was the participants’ not understanding why the task was required since it is not a task they would do. Some participants indicated that some of the information they were asked to find, specifically the source of the alert information, was not at all useful in their daily work. This was reflected in the success (Pass) scores because these participants had no idea where to start when attempting the task and some did not see the need to try it or try too hard to complete it.

EFFICIENCY

Noncritical system usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of this feature. Several of the inpatient participants also use the ambulatory product and were initially confused when completing the CDS Intervention task for Medications since the shortcut that exists in the ambulatory product is missing in the inpatient product. These participants were able to complete the task, but all mentioned the difference in the optimal path between the two systems. Many participants mentioned having to fulfill quality reporting requirements around CDS since the CDS alerts found under the Quality section did not always take into account activities done during normal clinical workflow.

SATISFACTION

Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated each scenario on a 1 (very difficult) to 5 (very easy) scale. The subtasks to access source attributes for Drug-Drug and Drug-Allergy Interaction Alerts were completed as part of Scenario 1. The mean satisfaction rating was 4.4 with a standard deviation of 0.8. Scenario 7.1 contained subtasks to access linked referential decision support and access source attributes for alert content. The mean satisfaction rating was 4.1 with a standard deviation of 1.1. Scenario 7.2 contained clinical decision support interaction subtasks. The mean satisfaction rating was 3.8 with a standard deviation of 1.0. Scenario 7.3 contained clinical decision support interaction and accessing source attributes subtasks. The mean satisfaction rating was 4.4 with a standard deviation of 0.7.

MAJOR FINDINGS

Performance for the following were at or above the 95% success criterion:

• Recording completion of CDS intervention that happened on today’s date. • Prescribing a medication to fulfill the CDS Medication criteria. • Verifying there was no CDS alert for a vaccine the patient is allergic to, to fulfill the CDS Medication

Allergy criteria. • Accessing linked referential decision support (InfoButton) after initial exposure. • Accessing source attributes for InfoButton content after initial exposure.

Performance for the following were below the 95% success criterion:

• Recording completion of a CDS intervention that happened on a previous date. • Accessing linked referential decision support (InfoButton) on initial exposure. • Accessing source attributes for drug-allergy alerts. • Accessing source attributes for CDS alert content after initial exposure.


Opportunities to improve the effectiveness and efficiency were identified and are described below. However, alert fatigue remains an industry-wide risk concern and must be balanced with the amount of the information that is displayed. AREAS FOR IMPROVEMENT

The athenahealth team is well aware of the industry’s known issue with alert fatigue. The team currently addresses alert fatigue by allowing sites to configure the levels of alerts and allowing individual prescribers to configure what level(s) of alerts are presented. There are alerts that cannot be ignored because of patient safety. The team has calibrated the system so as to provide safeguards related to these critical alerts. In addition, the athenahealth team would like to call for and be involved with the development and sharing of industry best practices, guidelines, templates regarding safety-enhanced design that impacts patient safety associated with alerts and alert fatigue. Minimal training and familiarity will aid end users to be able to locate and launch the CDS alert, InfoButton and Source Attribute features. Areas for improvement related to effectiveness and efficiency were also identified related to the visual layout of the content provided in the alerts and regarding the action buttons in the alerts:


• Review icons for saliency. • Review similar workflows across products for consistency. • Ensure that activities during the normal clinical workflow are incorporated and reflected in the

system’s CDS.


Results: § 170.315(a)(9) Clinical Decision Support – Configuration

Task Mapping Table 28 maps Clinical Decision Support – Configuration criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 28. Clinical Decision Support – Configuration criteria mapped to usability test tasks. § 170.315(a)(9) Clinical Decision Support (CDS) (i) CDS intervention interaction. Interventions provided to a user must occur when a user is interacting with technology. (ii) CDS configuration.

(A) Enable interventions and reference resources specified in paragraphs (a)(9)(iii) and (iv) of this section to be configured by a limited set of identified users (e.g., system administrator) based on a user’s role. (B) Enable interventions:

(1) Based on the following data: (i) Problem list; (ii) Medication list; (iii)Medication allergy list; (iv)At least one demographic specified in paragraph (a)(5)(i) of this section; (v) Laboratory tests; and (vi)Vital signs.

(2) When a patient’s medications, medication allergies, and problems are incorporated from a transition of care/referral summary received and pursuant to paragraph (b)(2)(iii)(D) of this section.

(iii)Evidence-based decision support interventions. Enable a limited set of identified users to select (i.e., activate) electronic CDS interventions (in addition to drug-drug and drug-allergy contraindication checking) based on each one and at least one combination of the data referenced in paragraphs (a)(9)(ii)(B)(1)(i) through (vi) of this section. (iv)Linked referential CDS.

(A) Identify for a user diagnostic and therapeutic reference information in accordance at least one of the following standards and implementation specifications:

(1) The standard and implementation specifications specified in § 170.204(b)(3). <InfoButton> (2) The standard and implementation specifications specified in § 170.204(b)(4). <InfoButton>

(B) For paragraph (a)(9)(iv)(A) of this section, technology must be able to identify for a user diagnostic or therapeutic reference information based on each one and at least one combination of the data referenced in paragraphs (a)(9)(ii)(B)(1)(i), (ii), and (iv) of this section.

(v) Source attributes. Enable a user to review the attributes as indicated for all CDS resources: (A) For evidence-based decision support interventions under paragraph (a)(9)(iii) of this section:

(1) Bibliographic citation of the intervention (clinical research/guideline); (2) Developer of the intervention (translation from clinical research/guideline); (3) Funding source of the intervention development technical implementation; and (4) Release and, if applicable, revision date(s) of the intervention or reference source.

(B) For linked referential CDS in paragraph (a)(9)(iv) of this section and drug-drug, drug-allergy interaction checks in paragraph (a)(4) of this section, the developer of the intervention, and where clinically indicated, the bibliographic citation of the intervention (clinical research/guideline).

To successfully complete this Task, participants were required to complete each of the following subtasks. Only tasks related to § 170.315 (a)(9) Clinical Decision Support – Configuration criteria will be discussed in this chapter. Scenario 12:

Task A: Activate CDS Intervention: BP Control (140/90) Guideline Task B: Adjust Age of CDS Intervention: BP Control (140/90) Guideline Task C: Activate CDS Intervention: Alcohol Use Assessment Guideline Task D: Adjust Age of CDS Intervention: Alcohol Use Assessment Guideline Task E: Adjust Queried Year of CDS Intervention: Alcohol Use Assessment Guideline


Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, configuration specialists attempted this task. Inpatient and ambulatory configuration specialists’ data was combined based on the fact that neither the task nor the user characteristics differ based on these user roles. Participants were given the following instruction:

Scenario 12 Tasks A and B: The BP Control (140/90) guideline is a measure that should identify patients 18 years and older whose most recent blood pressure is less than 140/90 within the past 12 months. You would like to use this measure in your practice. In addition, your patients tend to be as young as 16 years old. You would like to adjust the age range to 16-999 years old for this guideline. Please activate the BP Control guideline and adjust the age range. Tasks C, D, and E: The Alcohol Use Assessment guideline is a measure that identifies patients 18 years and older whose alcohol use was recorded during an encounter in the last 24 months. You would like to use this measure in your practice. In addition, your patients tend to be as young as 16 years old. You would like to adjust the age range to 16-999 years old for this guideline and you would like this guideline to be queried yearly rather than every 2 years. Please activate the Alcohol Use Assessment guideline, and adjust both the age range and the queried time period.

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient and ambulatory configuration specialists performed the CDS configuration scenario. Only the activities associated with the Clinical Decision Support – Configuration criteria are reported in this chapter.

Eighteen participants attempted Scenario 12. Task performance was not differentiated by setting (inpatient, ambulatory). Therefore, the data was not separated by user role for the setting. The usability testing results for the EHRUT are detailed below (see Table 29). The table lists out the subtasks associated Clinical Decision Support – Configuration criteria. Task 12 A/B was considered an initial exposure since not all participants had performed the required adjustments in their career.

Table 29. Clinical Decision Support – Configuration usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)

Errors Task Ratings

(5=Easy) # Percent

(%) Deviations


Mean (SD)


Percent (%) **

Mean (SD)

Task 12C: Activate CDS Intervention: Alcohol Use Assessment Guideline

18


89%θ 5/4

103* (54)

103* /27

11%**

4.3* (0.9)

Task 12D: Adjust Age of CDS Intervention: Alcohol Use Assessment Guideline

18 94% 3/3 6%**

Task 12E: Adjust Queried Year of CDS Intervention: Alcohol Use Assessment Guideline

18 94% 3/3 6%**



θ Initial exposure data included many participants who had not performed this adjustment during their careers and had no familiarity with these tasks. The initial exposure for activating a CDS intervention: BP Control (140/90) Guideline was 44% Task Success and 56%** Errors, 5/4 Path Deviation, 147* (51) Mean (SD) Task Time, 147*/25 Deviations Task Time, and 3.7* (1.9) Mean (SD) Task Ratings. Included in the initial exposure was the task to adjust the age CDS Intervention: BP Control (140/90) Guideline. The results were 83% Task Success and 17%** Errors, 2/4 Path Deviation, 147* (51) Mean (SD) Task Time, 147*/25 Deviations Task Time, and 3.7* (1.9) Mean (SD) Task Ratings.

Discussion of the Findings The following sections discuss the results organized around a risk analysis of use, test performance and error rates. The risk analysis of use includes identification of use errors and user interface design issues as well as classification of severity based on the consequence of the error. Use errors and user interface design issues that resulted in subtask failures, that are known industry risk issues, and errors and issues related to aspects of the user interface that are configured per customer site are considered more severe compared to noncritical system usability issues related to efficiency. As such the discussion of more serious errors and issues is provided in the Risk Analysis section and the associate mitigation strategy is provided in the Areas for Improvement section. Based on our definition of effectiveness metrics, performance, use errors and issues stemming from task failures are also discussed in the Effectiveness section as effectiveness was measured with task success and failure. Noncritical system usability issues related to efficiency are discussed in the Efficiency section. Associated recommendations are provided in the Areas for Improvement section. Satisfaction was assessed at the scenario level and is discussed in the Satisfaction section.

ERROR ANALYSIS Critical use errors were identified for Clinical Decision Support – Configuration tasks. A few participants were unable to find “Guidelines”. The root cause was lack of familiarity with the navigation since it is a task that is not performed often or had never been performed. Some participants selected the incorrect measure with a similar name in the initialization task; however, all of these participants were able to correctly edit the incorrect measure. All but one of the participants who selected the incorrect measure used filters that filtered out the correct measure and the one participant searched by the measure word spelled out as opposed to the acronym. Only two of these participants selected the incorrect measure for the second guideline that was to be configured and both used filters that filtered out the correct measure.

EFFECTIVENESS Performance in the all tasks fell below the 95% success criterion. As described above, many task failures were associated with selecting the incorrect measure as a result of using filters that did not include the correct measure. EFFICIENCY



SATISFACTION

Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. For the initial exposure task of subscribing providers to a CDS alerts for a medication allergy, the mean satisfaction was 3.7 with a standard deviation of 1.9. For the second task of subscribing providers to a CDS combination, the mean satisfaction was 4.3 with a standard deviation of 0.9. MAJOR FINDINGS Performance in the all tasks fell below the 95% success criterion. As described above, many task failures were associated with selecting the incorrect measure as a result of using filters that did not include the correct measure. Opportunities to improve effectiveness and efficiency were identified and are described below.



● Consider allowing users to search the menu pertaining to CDS guidelines. ● Consider improving filtering. ● Continue to review the cognitive workflow and associated interactions to improve effectiveness

and efficiency.


Results: § 170.315(a)(14) Implantable Device List

Task Mapping Table 30 maps the Implantable Device List criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 30. Implantable Device List criteria mapped to usability test tasks. 70.315(a)(14) Implantable Device List

(i) Record Unique Device Identifiers associated with a patient’s Implantable Devices. (ii) Parse the following identifiers from a Unique Device Identifier:

(A) Device Identifier; (B) The following identifiers that compose the Production Identifier:

(1) The lot or batch within which a device was manufactured; (2) The serial number of a specific device; (3) The expiration date of a specific device; (4) The date a specific device was manufactured; and (5) For an HCT/P regulated as a device, the distinct identification code required by 21 CFR § 1271.290(c).

(iii) Obtain and associate with each Unique Device Identifier: (A) A description of the implantable device referenced by at least one of the following:

(1) The “GMDN PT Name” attribute associated with the Device Identifier in the Global Unique Device Identification Database.

(2) The “SNOMED CT® Description” mapped to the attribute referenced in paragraph (a)(14)(iii)(A)(1) of this section.

(B) The following Global Unique Device Identification Database attributes: (1) “Brand Name”; (2) “Version or Model”; Page 3 of 9 (3) “Company Name”; (4) “What MRI safety information does the labeling contain?”; and (5) “Device required to be labeled as containing natural rubber latex or dry natural rubber (21 CFR 801.437).”

(iv) Display to a user an implantable device list consisting of: (A) The active Unique Device Identifiers recorded for the patient; (B) For each active Unique Device Identifier recorded for a patient, the description of the implantable

device specified by paragraph (a)(14)(iii)(A) of this section; and (C) A method to access all Unique Device Identifiers recorded for a patient.

(v) For each Unique Device Identifier recorded for a patient, enable a user to access: (A) The Unique Device Identifier; (B) The description of the implantable device specified by paragraph (a)(14)(iii)(A) of this section; (C) The identifiers associated with the Unique Device Identifier, as specified by paragraph (a)(14)(ii) of this

section; and (D) The attributes associated with the Unique Device Identifier, as specified by paragraph (a)(14)(iii)(B) of

this section. (vi) Enable a user to change the status of a Unique Device Identifier recorded for a patient. To successfully complete this Task, participants were required to complete each of the following subtasks. Only subtasks associated with § 170.315(a)(14) Implantable Device List are discussed in this chapter. Scenario 5

Task B: Record and parse implanted device ID on patient chart Task C: Change implanted device ID on patient chart Task D: Access and view implantable device UDI and associated description and identifier information Task E: View implantable device description and attributes Task G: View all active implantable devices Task H: View all implantable devices


Task Participants and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, ambulatory prescribers and both inpatient and ambulatory nurses / clinical staff attempted this task. Prescriber and nurse data were combined based on the fact that neither the task nor the user characteristics differ based on these settings or roles. Several of the tasks were new features in the application. Participants were sent training and asked to complete it at least 24 hours before their session. Only some of the participants completed the training. Those who did not complete the training had to rely on their experience with design patterns and interactions already in place elsewhere in the application. Participants were given the following instruction:

Scenario 5 Your next patient is 64 years old. She tells you she had surgery on her leg. Task B: She hands you a document with some information about an orthopedic bone pin that was implanted at the same time as a bone screw. Please add this bone pin to her chart. Task C: The patient says that her polyester suture was taken out on June 1, 2017. Please update her chart. Task D: If you needed to find the Device Identifier information for the cardiopulmonary tubing connector because there was a recall on these devices, where would you find this? Task E: (Split between participants)

• What is the model number of the device? • What is the expiration date of the device? • Where would you find a description, the GMDN PT name, of this device?

Task F: How many Active Implanted Devices does your patient currently have in her record? Task G: How many Total Implanted Devices are recorded for your patient, this includes all devices, not just active ones?

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Ambulatory prescribers and inpatient and ambulatory nurses / clinical staff performed the Implantable Devices scenario. Only the activities associated with the Implantable Devices criteria are reported in this chapter. Twenty-four participants attempted all the tasks in the scenario except for the change implanted device ID task and view all implanted device IDs task. Only twenty-three participants attempted these two tasks. One participant did not attempt these two tasks due to lack of time. Task performance was not differentiated by provider type (prescriber, nurse) or setting (inpatient, ambulatory). Therefore, the data were not separated by user role. The usability testing results for the EHRUT are detailed below (see Table 31). The table lists out the usability test results for each subtask associated with the Implantable Device List criteria.

Table 31. Implantable Device List usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Mean (SD)

Deviations


Mean (SD)


Mean (SD)

Mean (SD)

Task 5B: Record and parse implanted device ID on patient chart

24


100% 8/7 235* (130)

235* /48

0%** 3.8* (1.1)


Table 31. Implantable Device List usability test results. Measure

Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Mean (SD)

Deviations


Mean (SD)


Mean (SD)

Mean (SD)

Task 5C: Change implanted device ID on patient chart

23 91% 8/6 9%**

Task 5D: View implantable device UDI and description information

24 100% 1/1 0%**

Task 5E: View implantable device details

24 100% 0/0 0%**

Task 5F: View all active implantable devices

24 100% 1/1 0%**

Task 5G: View all implantable devices 23 100% 0/0 0%**



ERROR ANALYSIS Many participants had difficulty initially navigating to record an implantable device, and critical use errors were identified or observed. Several participants were unsure where to add implantable devices. Participants tried navigating to Problems and Medications. Many participants who successfully navigated to the correct tab then tried to add the implanted device as part of the patient’s Surgical History, either


as new item or by adding the provided alphanumeric as a text note to a previously recorded surgery. The root cause of these errors includes unfamiliarity with the task and location in the application. In the workflow, participants were presented with the initials “UDI” by a text field and did not immediately make the connection between the alphanumeric they were given and the text field. Some participants typed the description of the device into this text field, and others selected a custom button from the dialog and tried the workflow again. After entering the implanted device ID into the text field, participants were required to look up the implanted device ID so that the associated information could be added to the patient’s chart. One participant was confused with the labeled control for this purpose. The root cause of these errors includes unfamiliarity with the initials “UDI” and unclear labeling. Many participants struggled to add the device details, likely because the associated button was off the screen and required scrolling. Some participants clicked out of the panel to save the added device and had to restart the task from the beginning. Similarly, many participants struggled to save their changes when asked to inactivate a device. Participants scrolled down to find an Update button where none existed. The root cause of this is inconsistent add/edit workflows in the application. Two participants changed the implant date rather than adding a removal date. Once the implant date was changed, there is no way to see what date was previously recorded. One participant realized s/he put the removal date in the implant date field and had to remember the original implant date. One participant did not realize s/he put the removal date in the implant date field. When asked if the device was active, s/he said no when the device still showed in the system to be active. The root cause of these errors is the workflow should facilitate the necessary steps to display the appropriate text field and should help the user recover from date entering errors. Another participant added the removal information to the surgical history of the patient rather than to the specific implanted device. This led to the device still displaying as active when in another part of the system the device was noted as removed. The root cause of this is unfamiliarity with the task and location in the application. One participant initially misread the expiration date for the requested Implanted Device. The participant recovered and determined the correct date. The date information is displayed differently than the date information elsewhere in the system. Most participants could not figure out how to close the Implantable Device panel after inactivating or reviewing a device. A critical use error occurred when many of these participants clicked the X icon and accidentally invoked the delete/removal workflow. Most participants recovered from this error during testing, though some did not understand how they did so. One participant experienced an artifact of testing when s/he was presented with the “Active” button. The participant said s/he doesn’t need to use it in their practice and didn’t think it would be required in the session. Another participant experienced an artifact of testing due to delays in the system when scrolling. That participant was able to correctly complete the task. EFFECTIVENESS

Task performance for all but one subtask was at or above the 95% success criterion. Task performance for the subtask to change an implantable device fell below the 95% success criterion. The contributing factors were not finding features due to lack of familiarity with location of the features, unfamiliar navigation, and terminology. Other contributing factors also included inconsistency in design and workflow. EFFICIENCY

Noncritical system usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of this feature. Task times and number of clicks varied considerably during changing and adding an implanted device and looking for the number of active devices. Several interactions led to this:


• Several participants initially missed the Active button when adding the device and had to scroll back up to find what was preventing them from adding the device to the patient’s record.

• A number of fields associated with a specific device were displayed on the screen. When asked to locate and identify information in specific fields, participants had difficulty finding some fields. The root cause was the use of unfamiliar terminology for some of the fields.

SATISFACTION Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean was 3.8 with a standard deviation of 1.1. MAJOR FINDINGS Task performance for all but one subtask was at or above the 95% success criterion. Task performance for the subtask to change an implantable device fell below the 95% success criterion. Errors associated with accessing, changing, and adding a new device as well as finding the device identifier information and determining the number of active devices and total devices are described above. Opportunities to improve the effectiveness and efficiency were identified and are described below. AREAS FOR IMPROVEMENT Areas for improvement related to effectiveness and efficiency include:


• Consider using more familiar terminology.


Results: § 170.315(b)(2) Clinical Information Reconciliation and Incorporation (CIRI)

Task Mapping Table 32 shows the Clinical Information Reconciliation and Incorporation (CIRI) criteria to usability test tasks to aid verification that the report will contain all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 32. CIRI criteria mapped to usability test tasks. § 170.315(b)(2) Clinical Information Reconciliation and Incorporation (i) General requirements. Paragraphs (b)(2)(ii) and (iii) of this section must be completed based on the receipt of a transition of care/referral summary formatted in accordance with the standards adopted in § 170.205(a)(3) and § 170.205(a)(4) using the Continuity of Care Document, Referral Note, and (inpatient setting only) Discharge Summary document templates. (ii) Correct patient. Upon receipt of a transition of care/referral summary formatted according to the standards adopted § 170.205(a)(3) and § 170.205(a)(4), technology must be able to demonstrate that the transition of care/referral summary received can be properly matched to the correct patient. (iii) Reconciliation. Enable a user to reconcile the data that represent a patient's active medication list, medication allergy list, and problem list as follows. For each list type:

(A) Simultaneously display (i.e., in a single view) the data from at least two sources in a manner that allows a user to view the data and their attributes, which must include, at a minimum, the source and last modification date. (B) Enable a user to create a single reconciled list of each of the following: medications; medication allergies; and problems. (C) Enable a user to review and validate the accuracy of a final set of data. (D) Upon a user's confirmation, automatically update the list, and incorporate the following data expressed according to the specified standard(s):

(1) Medications. At a minimum, the version of the standard specified in § 170.207(d)(3); (2) Medication allergies. At a minimum, the version of the standard specified in § 170.207(d)(3); and (3) Problems. At a minimum, the version of the standard specified in § 170.207(a)(4).

(iv) System verification. Based on the data reconciled and incorporated, the technology must be able to create a file formatted according to the standard specified in § 170.205(a)(4) using the Continuity of Care Document template. To successfully complete this Task, participants were required to complete each of the following subtasks. Only tasks related to § 170.315 (b)(2) will be addressed in this section. Scenario 3:

Task 3A: Identify reconciliation is needed Task 3B: Reconcile medication allergies from a recent clinic visit Task 3C: Reconcile medications from a recent clinic visit Task 3D: Reconcile problems from a recent clinic visit Task 3E: Complete the reconciliation Task 3F: Create a Continuity of Care Document and submit electronically

Task Participants and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, ambulatory prescribers and nurse / clinical staff attempted this task. Prescriber and nurse data were combined based on the fact that neither the task nor the user characteristics differ based on these user roles. Participants were sent training on the continuity of care document and asked to complete it at least 24 hours before their session.


Only some of the participants completed the training. Those who did not complete the training had to rely on their experience with design patterns and interactions already in place elsewhere in the application Most of the participants were unfamiliar with the CIRI features. To ensure that participants had initial exposure with the reconciliation feature prior to performing the task, the moderator provided guided training with a section of one of the three areas to be reconciled. For example, the moderator would give the participant guided training on the reconciliation feature using a few of the items in the patient medication allergy reconciliation view. Participants were given the following instructions:

Scenario 3 There is a feature in the EHR that electronically reconciles clinical information from an external source. Today we are going to reconcile Allergies, Medications, and Problems from the external source named, James Getwell with information in this EHR. For example, let’s say the patient came to the clinic and Community Health folks sent you clinical information for your team to reconcile with the information in your EHR. Task A: Please see if your patient has items that need to be reconciled. Task B, C, & D: We are going to have you reconcile the patient’s allergies (Task B), medications (Task C) and problems (Task D). For this task, I am going to ask you to do things that you would not do if you were using your clinical judgment. So you can refer to the task sheet for the items that should or should not be included in the EHR. I am going to walk you through part of the patient’s Medication Allergy reconciliation. In the task list, it specifies to add Peanut to the patient’s chart. To do that, you would click the radio button from the imported data on Peanut. Or if you changed your mind and didn’t want that item added to the patient’s chart, you could just click on the Patient Chart side and it would not be included in the patient’s chart. If you preferred to have the item on the patient’s chart and not from the outside source, you would just click the radio button for the item in the patient’s chart. If any medications are in the patient’s chart, but are not listed in the Continuity of Care Document (CCD), choose the medication from the patient’s chart. Task E: Please complete the reconciliation. Task F: In this scenario, the reconciliation from the outside source for this patient has been completed. You now want to create and export a Continuity of Care Document (CCD) for this patient that will include the information you just brought into the chart. Please export the CCD now and submit by electronic file.

Data Analysis and Reporting – Clinical The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Ambulatory prescribers and clinical staff performed the CIRI scenario. Only the activities associated with clinical reconciliation criteria are reported in this chapter. Fifteen participants attempted the clinical reconciliation tasks and twenty-two participants attempted the task to create a continuity of care document. Task performance was not differentiated by provider type (prescriber, nurse). Therefore, the data was not separated by user role. The usability testing results for the EHRUT are detailed below (see Table 33). The table lists out the subtasks associated with clinical reconciliation criteria.


Table 33. Clinical Information Reconciliation Incorporation usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%) Deviations


Mean (SD)


Percent (%) **

Mean (SD)

Task 3A: Identify reconciliation needed

15 93% 4/3

396* (208)

396* /73

7%**

3.4* (1.2)

Task 3B: Reconcile medication allergies from a recent clinic visit

15 80%> 6/6 20%**

Task 3C: Reconcile medications from a recent clinic visit

15 87%> 12/10 13%**

Task 3D: Reconcile problems from a recent clinic visit

15 93%> 9/8 7%**

Task 3E: Complete the reconciliation 15 93% 3/3 7%**

Task 3F: Create a Continuity of Care Document and submit electronically

22 82% 13/6 82* (87)

82* /25

18%** 3.6* (1.3)

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. > Some tasks were performed two or more times in testing because these tasks were new to most participants or are performed very infrequently (e.g., configuration tasks). This was because the MU3 Safety Enhanced Design (SED) testing goal is to find true use error issues that persist over time, rather than "discoverability" or "learnability" issues. If higher pass rates, lower click counts, and/or lower task times were seen during additional testing of similar tasks, this is indicative that initial tests more likely showed discoverability issues. A higher pass rate the second time does not mean there are no other issues present, just that the initial lower pass rate is influenced by the participants' unfamiliarity with the part of the product being tested.



ERROR ANALYSIS Some participants had difficulty reconciling information from an outside source due to mechanical confusion and others due to cognitive confusion. Critical errors were observed during three “phases” of the reconciliation scenario: initial simultaneous display of information, reconciliation actions (i.e., accept information into the EHR), and review/validation of final reconciled data. Errors were also observed during the creation of a new Continuity of Care Document with the reconciled data. Identification of information to be reconciled: To complete the task, participants first had to navigate to the location of the reconciliation. Some were unfamiliar with using clinical reconciliation and were initially unable to navigate to the reconciliation area, though most of these participants eventually found the correct link. Lack of familiarity with the reconciliation feature is a root cause of this initial failure to find the reconciliation feature. As participants attempted to identify the source of medication allergies, medications, or problems, some participants misidentified the source. During the simultaneous display of two sources of information (e.g., medication allergies recorded in the EHR and medication allergies from an external source), labeled groupings were used to distinguish items from different sources. The coding was generally effective for most participants, though some participants were not familiar with the term “CCD.” The root cause of the error is confusion associated with the source coding cues and lack of familiarity with the reconciliation feature.

Reconciliation actions: Some participants were unable to determine what to do with reconciliation items that had no matching value, especially items that were already in the chart with no matching value in the CCD. They would ignore those items on the first pass through the reconciliation and then attempt to update the chart, expressing confusion when they were unable to do so. The root cause is that these participants did not realize they had to explicitly reselect items that were already in the chart that had no match in the CCD. A few participants expressed confusion at the conflicts and matches that were or were not listed. Participants would sometimes choose the wrong or a duplicate item when presented with Items that a human might deem a match but the system did not, such as Penicillin and Penicillin G. Participants also expressed confusion as to why some paired items were deemed a conflict rather than a match. One participant suggested that additional information, such as ICD codes, could have been provided to help make the reason for the conflict clearer. Some participants missed items that needed to be reconciled due to the small amount of screen real estate devoted to the actual reconciliation process. The current layout requires a large amount of scrolling to see everything on the screen and participants scrolled past items without realizing it. These participants struggled initially to understand why they could not save their work, then scrolled up and down through the reconciliation screens looking for the unreconciled items. A few participants were not able to find the next group of items to reconcile because of the control interaction. Some errors were associated with an artifact of testing. A few participants initially selected the incorrect item to be reconciled. This was due to the participant’s misunderstanding of the task or a short-term memory lapse. Review and validation of final reconciled data: Some participants did not realize the system was asking them to reconcile two different strengths of the same medication (Cardizem) onto the patient’s chart at any time during the reconciliation or when presented with the confirmation dialog. All of these participants saw the duplicate medication when the task directed them to the patient’s medication lists. In addition, no participants noticed that a penicillin-based medication, which the patient was allergic to, was reconciled to the chart. Clinicians may still want


to reconcile this, but need to be aware it is contra-indicated. The root cause of this issue is a lack of salient warnings. Many participants were unsure if the reconciled information had been saved to the chart, citing a lack of clear feedback. Creation of a Continuity of Care Document: Many participants had difficulty creating the Continuity of Care Document. Most participants were able to navigate to where the CCD creation option is located, but once there, most of these participants had difficulty finding the creation option. Once they found it, most participants were able to complete the task, however a small number had difficulty finding the appropriate control to create the Continuity of Care Document. The root cause of these issues is unfamiliarity with the CCD creation workflow, the discoverability of the CCD option, and an unintuitive workflow on the Continuity of Care Document panel page. Several participants were not sure if they had completed the task after pressing the Save button. Some were unsure what the feedback they received was telling them and others expressed that they were unsure where to send the document to or where the document they had just created had gone. A few participants clicked additional buttons, possibly trying to submit it a second time since they were not sure they successfully completed the task the first time. The root cause of these issues is the lack of salient feedback. EFFECTIVENESS Performance of all subtasks associated with Clinical Information Reconciliation and Incorporation was below the 95% success criterion. The contributing factors were confusion associated with the source coding cues, terminology, size of the reconciliation workspace, confusion associated with items that would be or would not be imported into a patient’s chart, and lack of familiarity with the feature. EFFICIENCY

Noncritical usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of this feature. Interactions with the controls resulted in inefficiencies for some participants.

SATISFACTION

Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean ease-of-use rating for Scenario 3.1, the clinical information reconciliation incorporation, was 3.4 with a standard deviation of 1.2. The mean ease-of-use rating for Scenario 3.2, the creation of a Continuity of Care Document, was 3.6 with a standard deviation of 1.3. MAJOR FINDINGS

Performance of all subtasks associated with Clinical Information Reconciliation and Incorporation was below the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below.





• Continue to review the cognitive workflow and associated interactions, including the use of warnings, to improve effectiveness and efficiency.

• Review panel designs to optimize effectiveness and efficiency.


Results: § 170.315(b)(3) Electronic Prescribing

Task Mapping Table 34 maps the Electronic Prescribing criteria to usability test tasks to aid verification that the report contains all required test scenarios for this EHR capability submitted for testing. Purple font is used within the certification criteria and within the steps for successful task completion to aid verification that the usability test tasks address the details of the specified criteria.

Table 34. Electronic Prescribing criteria mapped to usability test tasks. § 170.315(b)(3) Electronic Prescribing (i) Enable a user to perform all of the following prescription-related electronic transactions in accordance with the standard specified in § 170.205(b)(2) and, at a minimum, the version of the standard specified in § 170.207(d)(3) as follows: (A) Create new prescriptions (NEWRX). (B) Change prescriptions (RXCHG, CHGRES). (C) Cancel prescriptions (CANRX, CANRES). (D) Refill prescriptions (REFREQ, REFRES). (E) Receive fill status notifications (RXFILL). (F) Request and receive medication history information (RXHREQ, RXHRES). (ii) For each transaction listed in paragraph (b)(3)(i) of this section, the technology must be able to receive and transmit the reason for the prescription using the diagnosis elements in DRU Segment. (iii) Optional. For each transaction listed in paragraph (b)(3)(i) of this section, the technology must be able to receive and transmit the reason for the prescription using the indication elements in the SIG Segment. (iv) Limit a user’s ability to prescribe all oral liquid medications in only metric standard units of mL (i.e., not cc). (v) Always insert leading zeroes before the decimal point for amounts less than one and must not allow trailing zeroes after a decimal point when a user prescribes medications. To successfully complete the task, participants were required to complete each of the following Subtasks. Scenario 6:

Task A: Create new ePrescribed liquid medication [Zithromax (azithromycin)] Task A.1: Verify only metric standard units of mL

Scenario 8: Task A: ePrescribe half tablet of medication (Metformin) Task A.1: Verify leading zeroes before the decimal point for amounts less than one and does not allow trailing zeroes after a decimal point Task B: Cancel ePrescribed medication (Albuterol) Task C: Review and approve eRefill medication request (Ondansetron) Task D.1,2,3: Review and respond to pharmacy change request Task E: Skipped Task F: Request and review fill status (Cefdinir) Task G: Review medication history

Note: Participants confirmed that they do not manually request medication history information. Participants’ systems pull this data automatically. Thus, manually requesting medication history information was not tested.

Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, both inpatient and ambulatory prescribers attempted some of these tasks. Inpatient and ambulatory data were combined based on the fact that neither the task nor the user characteristics differ significantly based on these settings.


Some of the electronic prescribing features will not be included in the inpatient system, thus these features were only tested with tasks using the ambulatory system. Several of the tasks were new features in the application. Participants were sent training and asked to complete it at least 24 hours before their session. Only some of the participants completed the training. Those who did not complete the training had to rely on their experience with design patterns and interactions already in place elsewhere in the application. Participants were given the following instruction:

Scenario 6: Your next patient is a 10-year-old who has come in with strep throat. Task A (Inpatient): He is being discharged and needs to be prescribed with an antibiotic for Acute Tonsillitis. Please prescribe Zithromax (azithromycin) 200mg/ 5ml daily for 5 days and send it the NYC pharmacy. Please sign the order once you are done. Task A (Ambulatory): He needs to be prescribed with an antibiotic for Acute Tonsillitis. Please prescribe Zithromax (azithromycin) 200mg/ 5ml daily for 5 days and send it to the NYC pharmacy. Please sign the order once you are done. Task A.1: Does the system allow you to order in CCs? Scenario 8.1 Task A: Your next patient is a 67-year-old female whose has a diagnosis of type 2 diabetes mellitus. Please prescribe half of a tablet of 1000 mg Metformin daily for 30 days to her local NYC Pharmacy. Please type “.50” for the Dose. We ask you to type “.50” to see what the system will do to meet the MU3 regulations. Task A.1: Did the dose on this screen change to 0.5? Task B: The same patient tells you that she has already received a new prescription for her inhaler from another prescriber. You had also prescribed that inhaler. Please cancel your prescription to the pharmacy for the Albuterol inhaler for this patient. Scenario 8.2 Task C: You are reviewing requests from the pharmacy and see that the pharmacy has sent an electronic refill request for this same patient (Sidney Cassidy). You saw the patient in the clinic 3 days ago for vomiting. At the time, you prescribed symptomatic Ondansetron. Now that pharmacy has sent an electronic refill request for another 3 days for the patient. Approve that request. Task D.1: After receiving your prescription for Protonix 40mg Dispensed as Written, that same pharmacy sent a change request asking for authorization to change the drug to pantoprazole 40mg because Protonix is not covered by the patient’s insurance plan. Approve the pharmacy’s request to change the prescription. Task D.2: The pharmacist sent a change request to suggest other therapeutic alternatives that the patient may try to reduce their unpleasant side effects. The pharmacy recommends one Lipitor 20 mg oral tablet daily but upon review of the patient’s symptoms and cholesterol results, you prefer Zocor 20mg daily with three refills. Respond to the pharmacy’s request by letting them know you want to change the prescription to Zocor. Task D.3: After receiving the prescription for Breo Ellipta, the pharmacy sends a change request asking for prior authorization to supply this drug since other similar drugs on the market are much cheaper. Respond to their request by authorizing the Breo Ellipta prescriptions. Scenario 8.3 Task F: Your next patient is Teague Rodgersson. The pharmacy has sent a refill request for Mr. Rodgersson to renew his prescription for Cefdinir. You suspect that Mr. Rodgersson is requesting a refill too soon. You want to determine when this prescription was last filled since he recently started taking that medication. Please tell me the last fill date for Cefdinir. Task G: For the next task, your patient, Mr. Rodgersson, has come to the clinic stating that he thinks that one of his medications is causing very unpleasant side effects. You did not prescribe his medications, but you want to review his


medication history. Please locate the patient’s medication history and see when the fill information was last downloaded to his chart.

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient and ambulatory prescribers performed some or all of the electronic prescription tasks. Only the activities associated with the electronic prescription criteria are reported in this chapter. Sixteen participants attempted Scenario 6, which was to electronically prescribe a new prescription consisting of a liquid medication and verify only metric standard units may be utilized for a prescription. Sixteen participants attempted Scenario 8, tasks A through D. Task A consisted of electronically prescribing a medication and verifying the system would insert leading zeros and remove trailing zeros for amounts less than one. Tasks B through D consisted of canceling, refilling, and changing an electronic prescription. Seventeen participants attempted Scenario 8, tasks F and G, which included reviewing the fill status and medication history. The participants described varying levels of experience with electronic prescriptions. Some participants described it had been some time since they last performed electronic prescribing tasks in their current EHR. Others were unfamiliar with prescribing on the EHR system. Most of the participants had familiarity with the basic prescribing feature. However, many tasks were new and unfamiliar to the participants. Due to testing time constraints, for Task D, participants were asked to complete one of three tasks. The participant’s results were then reported together since the interface elements were identical or nearly identical and the error’s committed were similar across them. Task performance, when appropriate, was not differentiated by setting (inpatient, ambulatory). Therefore, that data was not separated by user role for the setting. The usability testing results for the EHRUT are detailed below (see Table 35). The table lists out the usability test results for each subtask in the electronic prescription tasks.

Table 35. Electronic Prescription usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%) Deviations


Mean (SD)


Percent (%) **

Mean (SD)

Task 6A: Create new ePrescribed liquid medication [Zithromax (azithromycin)]

16 100% 23/20 120* (60)

120* /34

0%** 4.6* (0.5)

Task 8A: ePrescribe half tablet of medication (Metformin)

16$ 94% 25/23

168* (115)

168* /50

6%**

3.6* (1.0) Task 8B: Cancel

ePrescribed medication (Albuterol)

10^$ of 16 70% 12/7 30%**


Table 35. Electronic Prescription usability test results.

Measure Task

N Task Success

Path Deviation

Task Time (Seconds)


(5=Easy) # Percent

(%) Deviations


Mean (SD)


Percent (%) **

Mean (SD)

Task 8C: Review and approve eRefill medication request (Ondansetron)

16$ 88%< 10/6

228* (110)

228* /30

13%<**

3.7* (1.0) Task 8D.1,2,3: Review

and respond to pharmacy change request

16+$ 94% 9/9 6%**

Task 8F: Request and review fill status (Cefdinir)

17$ 100% 6/4 281* (96)

281* (30)

0%** 2.9* (1.5) Task 8G: Request

and review medication history

17$ 94% 1/0 6%**

* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is the time to complete the entire scenario. Scenario Ratings are Mean (SD) ease-of-use rating for the entire scenario. ** Reporting the % Task Failures. $ Only ambulatory prescribers attempted these tasks. ^ The data was not included for six participants due to an operational difficulty. The Cancel option was not available for these participants and thus they could not complete the task. < Rounding error. + Sixteen participants attempted to perform one of three subtasks related to changing a prescription requested by the pharmacy that received the order. As part of the usability test, sixteen participants verified information as part of software validation to indicate the system would limit the user’s ability to enter oral liquid medications in ml. Sixteen participants verified information as part of software validation to indicate that the system would insert leading zeros and remove trailing zeroes (see Table 36).

Table 36. Electronic Prescription software validation test results.

Measure Task

N Task Success

Errors

# Percent (%)

Mean (SD)

Task 6A.1: Limit user’s ability to enter oral liquid medications in mL (do not allow ‘cc’ units) 16* 100% 0%

Task 8A.1: Insert leading zero and remove trailing zeroes for amounts less than 1 16* 100% 0%

* A validation test is a test of system functionality and is intended to observe how the application functions or reacts to a user’s input. This differs from a usability test which evaluates how the interface supports a user’s task.

Discussion of the Findings The following sections discuss the results organized around an error analysis, test performance and error rates. The error analysis includes identification of use errors and user interface design issues as well as classification


of severity based on the consequence of the error. Use errors and user interface design issues that resulted in subtask failures, that are known industry risk issues, and errors and issues related to aspects of the user interface that are configured per customer site are considered more severe compared to noncritical system usability issues related to efficiency. As such the discussion of more serious errors and issues is provided in the Error Analysis section and the associated mitigation strategy is provided in the Areas for Improvement section. Based on our definition of effectiveness metrics, performance, use errors and issues stemming from task failures are also discussed in the Effectiveness section as effectiveness was measured with task success and failure. Noncritical system usability issues related to efficiency are discussed in the Efficiency section. Associated recommendations are provided in the Areas for Improvement section. Satisfaction was assessed at the scenario level and at the system level. Both are discussed in the Satisfaction section.

ERROR ANALYSIS A critical use error was identified or observed as part of the electronic prescribing scenario. Some participants had difficulty or were unable to navigate to the locations to correspond with the pharmacy about a prescription and/or to review specific types of prescriptions the pharmacy acknowledged or provided feedback about. The root cause was lack of familiarity with the task and expecting the task to be completed or in another area of the menu or in a different area of the EHR. When trying to complete the tasks, a critical use error was identified or observed when some participants could not complete the task due to confusion associated with information organization. The root cause was lack of familiarity with the task and categorization of information layout. Related to this critical use error is another critical use error in which a few participants did a search on the prescription or added it to the current Assessment and Plan, and those actions produced multiple versions of the prescription. The root cause of this error was lack of familiarity with the task and unclear labeling. In addition, participants were not sure they had completed the task when they were done. The root cause was lack of familiarity with the task and unrecognized and/or not understood feedback. One critical use error was identified or observed in the create a new electronic prescription task. One participant did not complete the task because s/he did not complete a required field. The root cause was lack of familiarity and lack of salient feedback. In the cancel an electronic prescription task, a critical use error was identified or observed when two participants thought they had to reorder the prescription before they could cancel it. Another critical use error was identified or observed when one participant did not complete the task because s/he was not sure which option to choose to cancel the prescription. The root causes were lack of familiarity with the task and unclear options. One critical use error was identified or observed in the change an electronic prescription task. One participant did not complete the task because s/he did not locate where to change the name of the medication being prescribed. The root cause was lack of familiarity with the task. A non-critical use error was found when one participant could not find the appropriate pharmacy in the list of pharmacies. The list of pharmacies does not function similarly to other lists. The root cause is inconsistency in design. Two participants who were unfamiliar with prescribing on the inpatient athenahealth EHR had difficulty navigating to the prescribing page for a patient who was to be discharged. Both participants completed the task. In addition, one participant, unfamiliar with cancelling prescriptions, had difficulty finding the link that displayed the action to cancel. The participant completed the task.


One participant, after completing the cancel the medication to the pharmacy task, was confused why the medication still displayed as an active medication in the patient’s medication list. A few participants misunderstood the directions and defaulted to common clinical practices when performing the electronic prescription tasks. All but one participant completed the tasks and after verifying they had used their clinical judgements, the tasks were scored as task successes. The one participant realized the dose should have been doubled and completed that subtask, but then forgot to change the SIG from twice a day to once a day. Thus, the patient was overmedicated. This was scored as a failure but likely due to an artifact of testing. One participant did not understand the directions and initially deleted the prescription instead of cancelling the prescription. Due to this artifact or testing, this data was not included in the analyses. One participant experienced an artifact of testing when trying to complete the Prior Authorization task. The participant initially expected to need a prior authorization number to complete the task. The participant was able to correctly complete the task. Another participant experienced an artifact of testing when s/he tried to complete the status fill task in a way that should have worked, but due to limitations of the test system and task, did not. In addition, a third participant experienced an artifact of testing when s/he tried to complete the medication history task. The participant did not understand the task after having it explained multiple times, and thus did not compete the task. One participant experienced an operational difficulty on prescribing a new medication task. The participant was initially unable to order the medication, but after the system was restarted, the participant was able to correctly complete the task. Six participants experienced an operational difficulty on the cancel a prescription task when the system did not display the option to cancel the prescription to the pharmacy. The participants data were not included in the analyses. EFFECTIVENESS Performance for prescribing a new medication (Scenario 6) and requesting and reviewing fill status, as well as the verification tasks were at or above 95% success criterion. Performance for all other electronic prescription subtasks was below the 95% success criterion. The contributing factors were lack of familiarity, unrecognized and/or not understood feedback, unclear labeling, and inconsistent design, as described above. EFFICIENCY

Noncritical system usability issues that did not result in use errors were identified and provide opportunities for improvement related to the efficiency of this feature. If the name of the medication includes the dose or form, aspects of that information (e.g., “tablet”) should autofill the SIG. Quantity / Days should autofill if the SIG and Days / Quantity are filled. Some participants tried to search by ICD-10 codes. When the ICD-10 codes did not display, the participants had to search by the diagnoses’ names. SATISFACTION

Satisfaction levels were rated at the system level and at a scenario level. Refer to the chapter System Satisfaction for the system level Satisfaction ratings. for the system level Satisfaction ratings. Participants rated the scenario, which included these tasks, on a 1 (very difficult) to 5 (very easy) scale. The mean for Scenario 6 was 4.6 with a standard deviation of 0.5. The mean for Scenario 8.1, which contained Tasks A and B was 3.6 with a standard deviation of 1.0. The mean for Scenario 8.2, which contained Tasks C and D was 3.7 with a standard deviation of 1.0. The mean for Scenario 8.3, which contained Tasks F and G, was 2.9 with a standard deviation of 1.5.


MAJOR FINDINGS

Performance for prescribing a new medication (Scenario 6) and requesting and reviewing fill status, as well as the verification tasks was at or above 95% success criterion. Performance for all other electronic prescription subtasks was below the 95% success criterion. Opportunities to improve the effectiveness and efficiency were identified and are described below.

AREAS FOR IMPROVEMENT Areas for improvement related to effectiveness and efficiency include:




Results: Critical Use Risks for Patient Safety and NISTIR 7804-1

Task Mapping The ONC 2015 Edition Certification final rule states that the test scenarios participants used during the summative usability testing be submitted as part of the test results report. The ONC 2015 Edition Companion Guide for Safety-Enhanced Design includes clarification that in accordance with NISTIR 7804 Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records (EUP), it is recommended that the test scenarios be based upon an analysis of critical use risks for patient safety, which can be mitigated or eliminated by improvements to the user interface design. In addition, the Companion Guide for Safety-Enhanced Design explains that NIST recently developed an additional recommended resource for test scenarios: NISTIR 7804-1: Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically Based Use Cases for Validating Safety-Enhanced Usability and Guidelines for Standardization to inform summative usability test task development. As part of the task construction, tasks were informed by athenahealth’s risk management program and informed by NISTIR 7804-1. Findings associated with critical use risks identified via athenahealth’s risk management program are reported within the results chapters that align with each certification criteria. Findings associated with critical use risks that align with NISTIR 7804-1 Consistency of information and Integrity of Information are reported within the results chapters that align with these areas. Findings associated with critical risks that align with NISTIR 7804-1 Identification of Information spanned multiple scenarios and are thus reported in this chapter. As part of constructing scenarios to assess Identification of Information, the usability test team used NISTIR 7804-1 to inform usability test task creation. Examples include using mock-patient names where commonly used first names were also possible last names (e.g., William John, John William) were included in patient lists.

Task Participant and Instructions Based on user characteristics, typical workflow, and tasks performed as part of their daily work, both inpatient and outpatient prescribers and nurses attempted multiple tasks that called for the prescriber and for the nurse to select a specific mock-patient from a patient list. Participants were given the patient names verbally and via a data sheet that was available to the participant throughout the usability test tasks. Inpatient and outpatient prescriber and nurse data were combined based on the fact that neither the task nor the user characteristics differ based on these user roles.

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Inpatient and outpatient prescribers and nurses selected specific patients from a patient list within the context of each usability test tasks. Task performance was not differentiated by provider type (prescriber, nurse) or EHR type (inpatient, outpatient). Therefore, the data was not separated by user role. The usability testing results for the EHRUT are detailed below (see Table 37) which lists out the tasks of selecting a specific mock patient from a patient list. These tasks are associated with the NISTIR 7804-01 Identification of Information critical risk criteria.


Table 37. Identification of Information Critical Risk results. Measure

Task

N Task Success

Errors

# (%) (%)

Identification of Information Critical Risk Tasks

Scenario 1: Select John Wilson (prescribers only) 16 94% 6%**

Scenario 2: Select John Wilson (prescribers only) 17 100% 0%** Scenario 1RN: Select Jack Wilson (nurses/clinical staff only)

16 94% 6%**

Scenario 3: Select Perry Benson 15 93% 7%** Scenario 4: Select Perry Benson 20 100% 0%** Scenario 5: Select Tracy Grace 24 100% 0%** Scenario 6: Select Thomas Douglas (prescribers only) 16 100% 0%** Scenario 7.1: Select Jackson Wallace; this task had the same but inverted name on the patient list

20 90% 10%**


17 88% 12%**

Scenario 7.3: Select Leslie Rose (prescribers only) 17 100% 0%** Scenario 8: Select Sidney Cassidy (prescribers only) 16 100% 0%** Scenario 8: Select Teague Rodgersson (prescribers only)

17 100% 0%**

Scenario 9: Select James Scott (inpatient only) 16 100% 0%**



ERROR ANALYSIS Identification of Information is a known issue across the industry and was validated during usability testing. As described in NISTIR 7804-1 users must know; Am I in the right place and doing the right thing:

• For patient. • For patient lists. • For records. • For medication/order.

Potential consequences included:

• Missed, omitted, delayed care. • Care or billing activity conducted on the wrong patient. • Having to pull up every patient chart.

These consequences are propagated when pass/sharing information and results in fragmented information in the intended patient’s record. Throughout the usability test session, participants selected multiple patients from multiple patient lists. In many cases the participant selected the correct patient. In other cases, the participant selected the wrong patient. There were instances of the participant selecting the wrong patient based on flipping the first and last names (e.g., Jackson Wallace and Wallace Jackson), as well as names that sounded similar (e.g., John Wilson and Jack Wilson). Other participants choose the previous patient they had been working on rather than the next patient’s chart. Some participants completed some of the subtasks and did not realize the wrong patient was selected even though they repeatedly looked at a patient fact sheet with the patient’s name. The root cause is a lack of knowing that similar patient name can be found in the system. Two participants had close calls and chose the wrong patient’s record, but after trying to complete a task, realized they were not in the correct patient’s record and then pulled that correct patient’s record. Another critical use risk for identification of information was spontaneously seen in one of the testing sessions. A provider was performing a task in a patient’s chart and was interrupted by the practice nurse for a medical emergency. The participant returned to the testing session to complete the task after about two minutes and quickly completed the task using their clinical knowledge and not the provided testing directions. Thus, the participant was able to complete the task, but not as originally directed. The root cause to this difference was the interruption which pulled his/her attention away from the testing task. EFFECTIVENESS Performance of selecting the correct patient fell below the 95% success criterion for some tasks. The contributing factor is a lack of knowing that a similar patient name can be found in the system. EFFICIENCY


SATISFACTION

Satisfaction levels were rated at the system level and at a scenario level. Ratings are discussed in the chapters associated with 2015 Edition Certification criteria and are not repeated in this section. Refer to the chapter System Satisfaction for the system level Satisfaction ratings.


MAJOR FINDINGS

Performance of selecting the correct patient fell below the 95% success criterion for some tasks. Opportunities to improve the effectiveness are described below.


The athenahealth team is well aware of the industry’s known issue with patient identification. The team currently addresses patient identification by consistently displaying information critical to patient identification in a reserved area on the patient banner. The information is displayed in the same location regardless of scrolling or other navigational mechanisms to move within the screen/window. Critical information for patient identification includes:

• The patient’s name is shown first with the first (given) name, middle name and last (family) name capitalized, followed by any modifier e.g., Taylor John LEE, III.

• Preferred names follow the patient’s name and are displayed in quotations e.g., John Jamison “Jack”.

• The date of birth is shown using the following format e.g., 04-20-1991. • The age is shown using the following format e.g., 25yo. • The gender is provided as “M” or “F”. • The Medical Record number is also provided in the patient banner.

The athenahealth team is actively investigating solutions to indicate if there are similar names on a list that include a risk to cause a misidentification of patient data. In addition, the athenahealth team would like to call for and be involved with the development and sharing of industry best practices, guidelines, and templates regarding safety-enhanced design that impact patient identification.


System Satisfaction

About System Usability Scale (SUS) Scores Participants completed the System Usability Scale (SUS) questionnaire at the end of their session. The SUS survey yields a single number that represents a composite measure of the overall perceived usability of the system. SUS scores have a range of 0 to 100 and the score is a relative benchmark that is used against other iterations of the system. The SUS is a reliable and valid measure of system satisfaction. Sauro (http://www.measuringusability.com/sus.php accessed 3/14/2013) reports that the average SUS score from 500 studies across various products e.g., websites, cell phones, enterprise systems and across different industries is a 68. A SUS score above a 68 is considered above average and anything below 68 is below average. Additional research has shown that while the benchmark of 68 stills hold true, SUS scores for domain software such as EHR products are influenced by users’ experience levels.2 Unlike consumer software, domain software requires an inherent level of complexity. The need to allow users to eventually be efficient on the software requires functionality be less obvious on first inspection and can require training or users to become comfortable. This means new users to the software can go through an initial period of dissatisfaction while they become acquainted with the product, especially when companies do not provide adequate support and training. Once the users become more proficient, SUS scores will begin to conform to more standard cross-industry trends.

Clinical System Satisfaction Results Forty-one inpatient and ambulatory prescribers and 25 inpatient and ambulatory nurses / clinical staff completed the SUS questionnaire at the end of their session. However, due to an operational difficulty, one prescriber’s data was not usable. The system scored an average of 70 (SD = 21) based on 40 prescribers’ and 25 nurses’ responses.

Configuration System Satisfaction Results Eighteen configuration participants completed the SUS questionnaire at the end of their session. The system scored an average of 78 (SD=14) based on the responses of 18 configuration participants.

Registration System Satisfaction Results Fifteen registration participants completed the SUS questionnaire at the end of their session. The system scored an average of 83 (SD=17) based on the responses of 15 registration participants.

2 McLellan, Sam, Andrew Muddimer, and S. Camille Peres. 2012. "The Effect of Experience on Rating Systems." Journal of Usability Studies 7

(2): 56-67. Accessed June 6, 2017. http://uxpajournal.org/wp-content/uploads/pdf/JUS_McLellan_February_2012.pdf.


Appendices The following appendices include supplemental data for this usability test report. Following is a list of the appendices provided:

1. Sample Recruiting Screener 2. Participant Demographics 3. Non-Disclosure Agreement (NDA) and Informed Consent Form 4. Example Moderator’s Guides 5. Task Data Sheets 6. End of Task Rating Scale Questionnaire 7. System Usability Scale Questionnaire


Appendix 1: Sample Recruiting Screener

The purpose of a screener is to ensure that the participants selected represent the target user population as closely as possible. Recruiters were told the clinical participants needed to meet the following criteria: - Had to work with patients - Had to be familiar with the athenahealth EHR

Recruiters were told the configuration and registration participants needed to meet the following criteria: - Had to be familiar with the athenahealth EHR In addition, the following screener (survey) was used for the participants.

Pre-Session Survey

1. What is your name?

2. What is your gender? [Female, Male]

3. What is your age bracket? [Teens, 20-29, 30-39, 40-49, 50-59, 60-69, 70+]

4. What is your highest educational degree earned? [HS, Some college, Technical degree, AS, BS, MS,

MD, Other]

5. What is your occupation or role title?

6. How many months/years of professional experience do you have in this role?

7. How many months/years of computer experience do you have (all computer experience)?

8. How many months/years have you worked with the athenahealth EHR?

9. Do you have any assistive technology needs (e.g., screen reader, etc.)?


Appendix 2: Participant Demographics for athenahealth’s EHR

The following is a high-level overview of the participants in this study.

Gender Count Men 26 Women 73 Total (participants) 99

Age Range Count 20-29 6 30-39 37 40-49 22 50-59 23 60-69 11 Total (participants) 99

Education Range Count High school graduate 3 Some college credit, no degree 5 Trade/technical/vocational training 4 Associate degree 16 Bachelor’s degree 22 Master’s degree 12 Doctorate degree 37 Total (participants) 99

Occupation/Role Count RN/LPN/MA 25 Prescribers 41 Configuration Specialists 18 Registration Specialists 15 Total (participants) 99

Years of Experience Mean Range Professional Experience 14.8 1 – 41 Computer Experience 23.6 4 – 49 EHR Experience 3.2 0 – 13

Assistive Technology Needs Count Yes 0 No 99 Total (participants) 99


Appendix 3: NDA and Informed Consent Form I voluntarily agree to participate in an evaluation being conducted by athenahealth, Inc., (“Athena”), through its authorized agent, User-View, Inc. This evaluation will test the usability of the athenaClinicals product (“Product”). The purpose of this study is to gather feedback about the effectiveness and efficiency of the Product. By participating in this study, I agree and consent to the following: I authorize Athena to keep, preserve, use in any manner and dispose of the findings from this evaluation, including my feedback and suggestions expressed. I relinquish any rights to my feedback and suggestions about the Product. Athena will not associate my name or organization name with the results of this evaluation. I give my permission for Athena and its authorized agent User-View, Inc. to make video and audio records and to take photos of me during this evaluation. I understand that these recordings and photos can be used only for the purpose of evaluating the Product and showing the results of the evaluations and not for any other purpose. I understand that my participation is completely voluntary and I may withdraw my consent and discontinue my participation at any time without penalty. _______________________ _________________________ _______ Name (Please Print) Signature Date


Appendix 4: Example Moderator Guides

Moderator Guide Specific to Inpatient Prescribers Hello. Welcome, my name is _____________ and I’m a moderator, also we have ______________ online with us, s/he is the data logger. Can you see the welcome screen on your screen? Be sure to enlarge the shared screen on your screen. If you haven’t already, you can click the enlarge icon in the upper right corner and you can click the diagonal arrows here Thank you for participating in this study. Our session today will last 60 minutes. During that time, you will be performing tasks with AthenaHealth’s EHR in order to evaluate the extent to which the application is effective/not effective and efficient/not efficient in terms of Meaningful Use certification criteria. The first thing I would like to do is to show you this consent form to participate. Do you have any questions? I am going to start the recording now. Basically, because parts of this product are not public yet, we are asking you not to discuss what you see today with anyone. We are recording the audio and screenshots of our session today. This recording is only to make sure we capture and analyze the data accurately. It will not be used for any other purpose. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. OK, so I will have it on the record that you provided consent to be in this session, is that correct? This is what you agreed to on the consent form – I show it here so research teams that might go back and view this recording are under the same expectations. Thank you for filling out the Meaningful Use Demographics survey. Because this is for Meaningful Use, there are some background questions we need to ask.

Gender

Age 20s, 30s, 40s, 50s, 60s, 70s, older Education (highest degree) No high school degree High school graduate, diploma Some college credit, no degree Trade/technical/vocational Associate degree Bachelor’s degree Master’s Degree Doctorate Degree

Occupation/Role

Professional Experience (yrs)

Computer Experience (yrs)

Athena’s EHR Product Experience (yrs)

Assistive Technology Needs (yes-what, no)

Reviewed Training (yes/no) I am going to read this introduction to you because I want to be sure that I don’t miss anything. We did not have any involvement in this software’s creation. We are from an independent consulting company. This is a great chance for you to give the Athena team feedback about the application we are going to look at today, so please be honest with your opinions.


I will ask you to complete a few activities using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. Some activities might seem simple to you. Other activities might seem difficult. And there will be some activities that you will not able to complete. I am telling you this because I want you to remember that we are not testing you. You are helping us test the usability of the application. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Please stick to the task instructions provided to you. We are aggregating results so we need all participants to complete the same tasks. When you are doing these activities, I am not going to interact or talk to you. I and <<note taker>> will be taking notes about what you are doing. Because I am not going to be talking with you while you do the activities, I want you to make it clear to me when you are done with an activity by saying "I'm done." There are a number of reasons you might be done. (1) Done because you completed the activity. You know you completed it. So you say I’m Done. (2) Done because you have tried, you know you have not completed the activity, but you are not going to try anything else. So you say I’m Done. (3) Done because you feel like if you asked a question you could finish the activity. The system you are using is likely set different from what you use in your practice. So you might say I have a such and such button on my system. I don’t see on here. And I will tell you the next step and we will go from there. Sometimes I may not answer you. I’m not trying to be difficult; it’s just that I want to see how you would interact with the product on your own. If you get stuck, see if you can figure out what to do. As I mentioned, you can stop the task at any time, but we’d like you to give it your best shot before giving up. And if you get stuck or decide to give up, it means Athena will be getting feedback that the EHR needs more work done to make it easier to use. Remember, we are evaluating the product, not you. Do you have any questions or concerns before we begin? During the session, I will be reading a scenario to you. Any information that you need to complete your work is on the slide, we call it a task sheet. Let me show it to you now. You can click on it anytime to see what is needed. Or you can also ask me to repeat the task. To return to the application, click on this icon. After each task, you will rate how easy or difficult it was to perform. You will find that under this icon. Ok, I will pass you control of the mouse and keyboard. You can click that you will take control. Scenario 1 – CPOE Meds, DxD

I will read this to you now, please do not start until I tell you to.

Task A: You are Pat Rogers providing care for a new patient. The patient is a 58-year old male whose chief complaint is respiratory distress. He has a diagnosis of pneumonia. Please order Ampicillin 500mg IV for the patient’s pneumonia. If any alerts/warnings come up during the session, please let me know what the alert or warning is telling you and how you would typically handle the alert in your practice but please wait and let me tell you how to handle the alert for the situation. Your patient’s name is: John Wilson First name: John; Last name: Wilson Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Ampicillin 500mg IV, 500mg 3x/day


Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Did they review the DxA alert) (Amoxicillin and Penicillin Allergy) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task C: (DxA CDS reference) What is the source of this alert? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: (Change CPOE Medications) You decide to order Levofloxacin (Levaquin)500mg IV for the patient’s pneumonia instead. In addition, to address the patient’s wheezing, you decide to order Advair. Please order Advair for the patient’s pneumonia Levofloxacin 500mg IV, 1x/day Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (Record CPOE Medications) Advair (Fluticasone/Salmeterol) 250mcg/50mcg, 1 puff 2x/day Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task F: (Did they review the DxD alert) (Warfarin and Levofloxacin) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task G: Completion of task “You know that the co-administration of prednisone and levofloxacin may increase the risk of tendon rupture. However, you decide this isn’t a significant risk for this patient as he is bedridden, so you decide to keep this order. And you are finished ordering for this patient and may sign for the orders.”

Go ahead and sign the orders Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD )

Stop Time:_______________________________ <<Go to Scenario Rating 1>>


Please go to the Scenario Questionnaire – please rate the tasks based on how you felt performing the tasks – don’t rate it for how you think others would feel or how you would feel if you had more training on the task.

Scenario 2 – CPOE Labs & Imaging

Scenario 2 – Slide 4 Task A: (Access & Record CPOE Labs) You had ordered labs for Mr. Wilson yesterday and they returned with an elevated white blood count. You want to order a repeat CBC w/ auto diff to continue to monitor the patient’s labs. Order a CBC w/ auto diff, once, for the patient’s pneumonia. In addition, you want to order a two-view chest X-ray for your patient’s pneumonia. You do not need to complete any other fields. Save this order, but do not sign your orders yet. Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Access & Record CPOE Imaging) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task C: (Change CPOE Imaging) Now you’re wondering about the Chest X-ray you ordered for Mr. Wilson’s pneumonia diagnosis. You realize that you really want it to be performed “now”. Please change its status from routine to Now. In addition, you also decide you would like the lab you just ordered to be performed STAT. Please update the lab order.

Once you complete these orders, you may sign for the orders

Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: (Change CPOE Labs) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 2 >> Please go to the Scenario Questionnaire

Scenario 4 – Med List, Problem List, Med Allergy List

Scenario 4 – Slide 5 Go to PATIENT LIST


Task A: Your next patient is a 55-year old male with a history of rheumatoid arthritis whose chief complaint was sinus pressure. He has a diagnosis of sinus infection. You want to go over his chart with him. You start by reviewing his allergies. Please name one of the medications the patient is allergic to? Your patient’s name is: Perry Benson First name: Perry; Last name: Benson Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Record Medication Allergy List) Mr. Benson’s wife arrives as you are reviewing his allergies. She tells you he gets a severe rash to iodine, which he always forgets to mention. He confirms the allergy. Add this allergy to the patient’s current allergy list. In addition, Mr. Benson’s wife reminds him that he started getting a mild headache, besides a rash, due to his sulfa allergy, and he agrees. Add a reaction of “headache” with mild severity to his sulfa allergy. Iodine, Reaction: rash, Severity: Severe, No onset date Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (Change Medication Allergy List) Sulfa Allergy: Additional Reaction: headache, Severity: mild, Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: When was the patient’s first reaction to Penicillin? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.1>> Start Time:_______________________________ Task E: You continue reviewing Mr. Benson’s chart, please locate his active home medications. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD )


Task F: You see that Mr. Benson is currently taking Cardizem but you wonder if he has ever taken Lipitor. Please look to see if he may have taken Lipitor in the past. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task G: (Change Medication List) Mr. Benson tells you his PCP discontinued his Azithromycin today. Please discontinue the Azithromycin from the patient’s home medication list. Also, the patient tells you that his Endocrinologist has put him on Aldactone (spironolactone) 25mg daily for high blood pressure a month ago. If Aldactone is not in the chart, add that to the patient’s home medication list. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task H: (Record Medication List) Aldactone 25mg Tablet, Take 1 tablet daily, Started: one month ago Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.2>> Start Time:_______________________________ Task I: Please locate the patient’s diagnoses list. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task J: (Change Problem List) While reviewing the patient’s problem list, you notice an old diagnosis of childhood asthma was pulled in as an active problem. Please update the system so that the childhood asthma is no longer listed as an active problem. Also add that the patient has acute laryngitis today. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task K: (Record Problem List) Acute Laryngitis Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task L: When was the patient’s diabetes mellitus originally diagnosed? Clicks / Steps: ____________________________


Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.3>>

Scenario 5 – Implantable Devices Scenario 5 – Slide 6 Task A: Your next patient is 64 years old. She tells you she had surgery on her leg and hands you a document with some information about the bone screw that was used. Please add the bone screw alphanumeric to her chart. Your patient’s name is: Tracy Grace First name: Tracy; Last name: Grace Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: She hands you a document with some information about an orthopedic bone pin that was implanted at the same time as a bone screw. Please add this bone pin to her chart. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: The patient says that her polyester suture was taken out on July 1, 2017. Please update her chart. Date: July 1, 2017 Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: If you needed to find the Device Identifier information for the cardiopulmonary tubing connector because there was a recall on these devices, where would you find this? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (Split between participants)


Clicks / Steps: ____________________________ Pass / Fail


Use Error (UE) / Close Call (CC) / Operational Difficulties (OD )

Task F: How many Active Implanted Devices does the patient currently have in her record? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task G: How many total Implanted Devices are recorded for the patient, this includes all devices not just active ones? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 5>>

Scenario 6 – Order ePrescribe/cc

Scenario 6 – Slide 7 Task A: Your next patient is a 10-year-old who has come in with strep throat. He is being discharged and needs to be prescribed with an antibiotic for Acute Tonsillitis. Please prescribe Zithromax (azithromycin) 200mg/ 5ml daily for 5 days and send it the NYC pharmacy. Please sign the order once you are done. Your patient’s name is: Thomas Douglas First name: Thomas; Last name: Douglas Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Zithromax 200 mg / 5 mL, Take 5mL everyday for 5 days; Qty: 25 mL; Pharmacy: NYC Pharmacy 10.6mu Does the system allow you to order in CCs? Y / N

Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 6>>

Scenario 7.1 – InfoButton

Scenario 7.1 – Slide 8 Task: (CDS: Find CDS Icon)


Your next patient is 8-year-old Jackson Wallace. You want to see if the system has provided any meaningful use recommendations for this patient in their chart. Your patient’s name is: Jackson Wallace First name: Jackson; Last name: Wallace Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task A: (CDS: InfoButton, Demographics) The EHR provides you the ability to get additional information about Pediatric Diabetes from an outside source. Please go to where you would find this additional information. Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task B: Who is the developer of this information? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (CDS: InfoButton, Problem List) You would like to get additional information about children with asthma. Find this information. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: Who is the developer of this information? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (CDS: InfoButton, Medication List) You would like to get additional information about ADHD medication. Find this information. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________


<<Go to Scenario Rating 7.1>>

Scenario 7.2 – CDS: Child Scenario 7.2 – Slide 9 Go to PATIENT LIST Task: (CDS: Find CDS Icon) Did not Complete Infobutton: Your next patient is an 8-year old. You want to see if the system has provided any meaningful use recommendations for this patient in their chart. Your patient’s name is: Jackson Wallace First name: Jackson; Last name: Wallace Completed Infobutton: Your patient is still Jackson Wallace. Please go back to the meaningful use recommendations for this patient. Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task A: (CDS: Med List) Using the recommendations, complete the ADHD medication recommendation and order Ritalin 10mg. Also indicate the Pediatric Diabetes recommendation was completed. Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Ritalin 10 mg tablet Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (CDS: Combo: Demographic and Problem; Pediatric Diabetes) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 7.2>>

Scenario 7.3 – CDS - Adult Scenario 7.3 – Slide 10


Task A1: (CDS: Demographics: Breast Cancer) Your next patient is a 60-year-old female. Please use the system provided meaningful use recommendations to indicate the items have been completed. Your patient’s name is: Leslie Rose First name: Leslie; Last name: Rose Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Your patient says she had her Breast Cancer screening on March 10. Please specify that. Start Time:_______________________________ Completed the Breast Cancer Screening on March 10, 2017 Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task A2: (CDS: Reference 1) Who is the developer of the foot exam intervention?

Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (CDS: Problem List) Completed the Foot exam today Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task C: (CDS: Reference 2) Who is the developer of the LDL recommendation? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: (CDS: Lab Test) Go ahead and do what the LDL Recommendation tells you to do. The information you need for the LDL is: Today’s date and a result of 80. Enter Results of the LDL-C: 80 mg/dl for today Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (CDS: Vital Signs)


Completed the High Blood Pressure Care Plan today Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task F: (CDS: Medication Allergy: Shingles vaccine and neomycin allergy) Your patient has not had a shingles vaccine, however, the patient has an allergy to neomycin which is an ingredient in the shingles vaccine. Please verify the patient has an allergy to neomycin and verify there is NO recommendation advising the patient get a shingles vaccine. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 7.3>>

Scenario 8 – ePrescribe

Scenario 8 – Slide 11 Task A: (Validation Task for “0.5”) Your next patient is a 67-year-old female who came into the hospital with chest pain. It turned out to be musculoskeletal, but you did find she has elevated A1C and high blood sugar. You’ve diagnosed diabetes and you want to discharge her and prescribe Metformin. Please prescribe half of a tablet of 1000 mg Metformin daily for 30 days to her local pharmacy. Please type “.50” for the Dose. We ask you to type “.50” to see what the system will do to meet the MU3 regulations. Your patient’s name is: Sidney Cassidy First name: Sidney; Last name: Cassidy Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Metformin1000mg .5 tablets daily for 30 days, 3 refills; Pharmacy: NYC Pharmacy 10.6mu Did the dose on this screen change to 0.5? Y / N Please complete the order and sign for the order. Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Cancel Prescription, Part A) The patient tells you that she has already received a new prescription for her inhaler from another prescriber. You had also prescribed that inhaler. Please cancel your prescription to the pharmacy for the Albuterol inhaler for this patient. Is this type of task one you currently do as part of your role, where you would want to cancel a prescription you had sent to the pharmacy? Y / N Is this something you would or would not like to do? Would / Would Not


If “Y” or “Would”: If you had to complete this task, show me the first few steps you would do to try and complete it. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (Refill Prescription, Part A & B) You are reviewing requests from the pharmacy and see that the pharmacy has sent an electronic refill request for this same patient (Sidney Cassidy). You saw the patient in the clinic 3 days ago for vomiting. At the time, you prescribed symptomatic Ondansetron. Now that pharmacy has sent an electronic refill request for another 3 days for the patient. Approve that request. Is this type of task one you currently do as part of your role, where you would want to cancel a prescription you had sent to the pharmacy? Y / N Is this something you would or would not like to do? Would / Would Not If “Y” or “Would”: If you had to complete this task, show me the first few steps you would do to try and complete it. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Slide 13 for all three Change tasks Task D1: (Change Prescription: approve Generic) After receiving your prescription for Protonix 40mg Dispensed as Written, that same pharmacy sent a change request asking for authorization to change the drug to pantoprazole 40mg because Protonix is not covered by the patient’s insurance plan. Approve the pharmacy’s request to change the prescription. pantoprazole 40mg, one tablet daily Is this type of task one you currently do as part of your role, where you would want to cancel a prescription you had sent to the pharmacy? Y / N Is this something you would or would not like to do? Would / Would Not If “Y” or “Would”: If you had to complete this task, show me the first few steps you would do to try and complete it. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 8.1INP>> Slide 13 for all three Change tasks Task D2: (Change Prescription: Approve with edits) The pharmacist sent a change request to suggest other therapeutic alternatives that the patient may try to reduce their unpleasant side effects. The pharmacy recommends one Lipitor 20 mg oral tablet daily but upon review of the patient’s symptoms and cholesterol results, you prefer Zocor 20mg – daily with three refills. Respond to the pharmacy’s request by letting them know you want to change the prescription to Zocor Zocor 20mg tablet, Take 1 table daily, 30 tablets with 3 refills Is this type of task one you currently do as part of your role, where you would want to cancel a prescription you had sent to the pharmacy? Y / N Is this something you would or would not like to do? Would / Would Not


If “Y” or “Would”: If you had to complete this task, show me the first few steps you would do to try and complete it. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 8.1INP>> Slide 13 for all three Change tasks Task D3: (Change Prescription: Prior Authorization) After receiving the prescription for Breo Ellipta, the pharmacy sends a change request asking for prior authorization to supply this drug since other similar drugs on the market are much cheaper. Respond to their request by authorizing the Breo Ellipta prescriptions. Is this type of task one you currently do as part of your role, where you would want to cancel a prescription you had sent to the pharmacy? Y / N Is this something you would or would not like to do? Would / Would Not If “Y” or “Would”: If you had to complete this task, show me the first few steps you would do to try and complete it. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 8.1INP>> Slide 16 Task F: (Fill Status) Your next patient is Teague Rodgersson. The pharmacy has sent a refill request for Mr. Rodgersson to renew his prescription for Cefdinir. You suspect that Mr. Rodgersson is requesting a refill too soon. You want to determine when this prescription was last filled since he recently started taking that medication. Please tell me the last fill date for Cefdinir. Your patient’s name is: Teague Rodgersson First name: Teague; Last name: Rodgersson Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD )


Task G: (Medication History) For the next task, your patient, Mr. Rodgersson, has come to the clinic stating that he thinks that one of his medications is causing very unpleasant side effects. You did not prescribe his medications, but you want to review his medication history. Please locate the patient’s medication history and see when the fill information was last downloaded to his chart. Do you ever manually download a patient’s medication history? Y / N Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 8.2>>

Scenario 8 – Slide 11 Task F: (Fill Status) Your next patient is Teague Rodgersson. The pharmacy has sent a refill request for Mr. Rodgersson to renew his prescription for Cefdinir. You suspect that Mr. Rodgersson is requesting a refill too soon. You want to determine when this prescription was last filled since he recently started taking that medication. Please check to see the last fill date for Cefdinir. Your patient’s name is: Teague Rodgersson First name: Teague; Last name: Rodgersson Is this type of task one you currently do as part of your role? Y / N Is this something you would or would not like to do? Would / Would Not

If “Y” or “Would”: If you had to complete this task, show me the first few steps you would do to try and complete it. Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task G: (Medication History) For the next task, your patient, Mr. Rodgersson, has come to the clinic stating that he thinks that one of his medications is causing very unpleasant side effects. You did not prescribe his medications, but you want to review his medication history. Please locate the patient’s medication history and see when fill information was last downloaded to his chart. Is this type of task one you currently do as part of your role? Y / N Is this something you would or would not like to do? Would / Would Not

If “Y” or “Would”: If you had to complete this task, show me the first few steps you would do to try and complete it. Clicks / Steps: ____________________________ Pass / Fail


Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 8.2INP>>

Scenario 9 – Death Demographics Scenario 9 – Slide 13 Task A: Meaningful Use requires that we get your feedback on how the system supports documenting and editing the cause and date of death. You will need to discharge the patient and document the cause of death as acute heart failure and document the date of death as today’s date. We are going to use the patient named James Scott First name: James; Last name: Scott Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: Now change the information you just entered for James Scott. The diagnosis was not acute heart failure, please change the cause of death as cardiac arrest and document the date of death as yesterday’s date. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Stop Time:_______________________________ <<Go to Scenario Rating 9>>

Scenario 10 – Provider Configuration

Scenario 10 – Slide 14 Task A: You are being notified too often for drug interactions. Change the Drug interaction warning levels from displaying all warnings to only serious and severe warnings. Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Clicks / Steps: ____________________________ Pass / Fail


Use Error (UE) / Close Call (CC) / Operational Difficulties (OD )

Task B: Please go back to the patient list. You are being notified too often for drug interactions. Set the system settings so that the prescriber is warned when there is are only severe drug interaction warnings. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Stop Time:_______________________________ <<Go to Scenario Rating 10>>

OK, that is the last task I have for you. Now we will go through a brief questionnaire about your experience today. <<Administer the SUS>> Note taker, do you have any questions? If time permits: We would like to collect any items, questions or concerns you have about the current athenaClinicals® system you have installed. Is there anything you would like to tell us?


Moderator Guide Specific to Ambulatory Prescribers Hello. Welcome, my name is _____________ and I’m a moderator, also we have ______________ online with us, s/he is the data logger. Can you see the welcome screen on your screen? Be sure to enlarge the shared screen on your screen. If you haven’t already, you can click the enlarge icon in the upper right corner and you can click the diagonal arrows here Thank you for participating in this study. Our session today will last 60 minutes. During that time, you will be performing tasks with AthenaHealth’s EHR in order to evaluate the extent to which the application is effective/not effective and efficient/not efficient in terms of Meaningful Use certification criteria. The first thing I would like to do is to show you this consent form to participate. Do you have any questions? I am going to start the recording now. Basically, because parts of this product are not public yet, we are asking you not to discuss what you see today with anyone. We are recording the audio and screenshots of our session today. This recording is only to make sure we capture and analyze the data accurately. It will not be used for any other purpose. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. OK, so I will have it on the record that you provided consent to be in this session, is that correct? This is what you agreed to on the consent form – I show it here so research teams that might go back and view this recording are under the same expectations. Thank you for filling out the Meaningful Use Demographics survey. Because this is for Meaningful Use, there are some background questions we need to ask.

Gender


Occupation/Role





Reviewed Training (yes/no) I am going to read this introduction to you because I want to be sure that I don’t miss anything. We did not have any involvement in this software’s creation. We are from an independent consulting company. This is a great chance for you to give the Athena team feedback about the application we are going to look at today, so please be honest with your opinions. I will ask you to complete a few activities using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. Some activities might seem simple to you. Other activities might seem difficult. And there will be some activities that


you will not able to complete. I am telling you this because I want you to remember that we are not testing you. You are helping us test the usability of the application. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Please stick to the task instructions provided to you. We are aggregating results so we need all participants to complete the same tasks. When you are doing these activities, I am not going to interact or talk to you. I and <<note taker>> will be taking notes about what you are doing. Because I am not going to be talking with you while you do the activities, I want you to make it clear to me when you are done with an activity by saying "I'm done." There are a number of reasons you might be done. (1) Done because you completed the activity. You know you completed it. So you say I’m Done. (2) Done because you have tried, you know you have not completed the activity, but you are not going to try anything else. So you say I’m Done. (3) Done because you feel like if you asked a question you could finish the activity. The system you are using is likely set different from what you use in your practice. So you might say I have a such and such button on my system. I don’t see on here. And I will tell you the next step and we will go from there. Sometimes I may not answer you. I’m not trying to be difficult; it’s just that I want to see how you would interact with the product on your own. If you get stuck, see if you can figure out what to do. As I mentioned, you can stop the task at any time, but we’d like you to give it your best shot before giving up. And if you get stuck or decide to give up, it means Athena will be getting feedback that the EHR needs more work done to make it easier to use. Remember, we are evaluating the product, not you. Do you have any questions or concerns before we begin? During the session, I will be reading a scenario to you. Any information that you need to complete your work is on the slide, we call it a task sheet. Let me show it to you now. You can click on it anytime to see what is needed. Or you can also ask me to repeat the task. To return to the application, click on this icon. After each task, you will rate how easy or difficult it was to perform. You will find that under this icon. Ok, I will pass you control of the mouse and keyboard. You can click that you will take control. Scenario 1 – CPOE Meds, DxD


Task A: You are Pat Rogers providing care for a new patient. The patient is a 58-year old male whose chief complaint is respiratory distress. He has a diagnosis of pneumonia. Please order Ampicillin 500mg for the patient’s pneumonia. If any alerts/warnings come up during the session, please let me know what the alert or warning is telling you and how you would typically handle the alert in your practice but please wait and let me tell you how to handle the alert for the situation. Your patient’s name is: John Wilson First name: John; Last name: Wilson Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Ampicillin 500mg capsule, every 6 hours for 10 days, Quantity: 40 Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________


Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Did they review the DxA alert) (Amoxicillin and Penicillin Allergy) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task C: (DxA CDS reference) What is the source of this alert? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: (Change CPOE Medications) You decide to order Levofloxacin (Levaquin)500mg for the patient’s pneumonia instead. In addition, to address the patient’s wheezing, you decide to order Prednisone 20mg tablet. Please order prednisone for the patient’s pneumonia Levofloxacin 500mg tablet, 1x/day for 14 days, Quantity: 14 Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (Record CPOE Medications) Prednisone Oral 20mg tablet daily for 7 days, Quantity: 7 Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task F: (Did they review the DxD alert) (Warfarin and Levofloxacin) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task G: Completion of task “You know that the co-administration of prednisone and levofloxacin may increase the risk of tendon rupture. However, you decide this isn’t a significant risk for this patient as he is bedridden, so you decide to keep this order. And you are finished ordering for this patient and may sign for the orders.”

Go ahead and sign the orders Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD )

Stop Time:_______________________________ <<Go to Scenario Rating 1>>



Scenario 2 – CPOE Labs & Imaging Scenario 2 – Slide 4 Task A: (Access & Record CPOE Labs) Please order a CBC w/ auto diff for the patient’s pneumonia. In addition, you want to order a two-view chest X-ray for your patient’s pneumonia. You will want this order to go today. You want an alarm if the patient has not had the X-ray within 2 weeks. You do not need to worry about the facility or the ask-on-order questions. Your MA will complete the details. Save these order, but do not sign your orders yet. Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Access & Record CPOE Imaging) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task C: (Change CPOE Imaging) Now you’re wondering about the Chest X-ray you ordered for Mr. Wilson’s pneumonia diagnosis. You realize that you really want an alarm if the patient has not had the X-ray within 1 week. Change the alarm to 1 week. In addition, you also decide you would like the lab you just ordered to be performed STAT. Please update the lab order. And once you complete these orders, you may sign for the orders

Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: (Change CPOE Labs) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 2 >> Please go to the Scenario Questionnaire

Scenario 3 – CIRI Scenario 3 – Slide 5 Task A: There is a feature in the EHR that electronically reconciles clinical information from an external source. Today we are going to reconcile Allergies, Medications, and Problems from the external source named, James Getwell with information in this EHR. For example, let’s say the patient came to the clinic and Community Health folks sent you clinical information for your team to reconcile with the information in your EHR.


Have you used this feature in your practice? Yes / No If yes: what have you used it for? Meds / Med Allergies / Problems

Please see if your patient has items that need to be reconciled. Your patient’s name is: Perry Benson First name: Perry; Last name: Benson Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Example with Med Allergy, Reconcile Allergies) We are going to have you reconcile the patient’s allergies, medications and problems. For this task, I am going to ask you to do things that you may not do if you were using your clinical judgment. So you can refer to the task sheet for the items that should or should not be included in the EHR. I am going to walk you through part of the patient’s Medication Allergy reconciliation. In the task list it specifies to add Peanut to the patient’s chart. To do that, you would click the radio button from the imported data on Peanut. Or if you changed your mind and didn’t want that item added to the patient’s chart, you could just click on the Patient Chart side and it would not be included in the patient’s chart. If you preferred to have the item on the patient’s chart and not from the outside source, you would just click the radio button for the item in the patient’s chart. If any medications are in the patient’s chart, but are not listed in the Continuity of Care (CCD), choose the medication from the patient’s chart. Now let’s have you reconcile the patient’s Medication Allergies. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (Reconcile Medications, Cardizem mg difference) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: (Reconcile Problems) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task E: (Complete Reconciliation): Please complete the reconciliation. Based on this list, please tell me which allergies are being brought into the EHR. Clicks / Steps: ____________________________


Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Stop Time:_______________________________ <<Go to Scenario Rating 3.1 >> Task F: In this scenario, the reconciliation from the outside source for this patient has been completed. You now want to create and export a Continuity of Care Document (CCD) for this patient that will include the information you just brought into the chart. Please export the CCD now and submit by electronic file. Start Time:_______________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Stop Time:_______________________________ <<Go to Scenario Rating 3.2 >> And please rate only on the easy or difficulty of creating a continuity of care document.


Scenario 4 – Slide 6 Go to PATIENT LIST Task A: If did not complete CIRI: Your next patient is a 55-year old male with a history of rheumatoid arthritis whose chief complaint was sinus pressure. He has a diagnosis of sinus infection. You want to go over his chart with him. You start by reviewing his allergies. Please name one of the medications the patient is allergic to? Your patient’s name is: Perry Benson First name: Perry; Last name: Benson If completed CIRI: Please go to the schedule, that is where we need to start this task. You next want to go over Perry Benson’s chart with him. You start by reviewing his allergies. Please name one of the medications the patient is allergic to? Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Record Medication Allergy List) Mr. Benson’s wife arrives as you are reviewing his allergies. She tells you he gets a severe rash to iodine, which he always forgets to mention. He confirms the allergy. Add this allergy to the patient’s current allergy list. In addition, Mr. Benson’s wife reminds him that he started getting a mild headache, besides a rash, due to his sulfa allergy, and he agrees. Add a reaction of “headache” with mild severity to his sulfa allergy. Iodine, Reaction: rash, Severity: Severe, No onset date Clicks / Steps: ____________________________


Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (Change Medication Allergy List) Sulfa Allergy: Additional Reaction: headache, Severity: mild, Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: When was the patient’s first reaction to Penicillin? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.1>> Start Time:_______________________________ Task E: You continue reviewing Mr. Benson’s chart, please locate his current medications. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task F: You see that Mr. Benson is currently taking Cardizem but you wonder if he has ever taken Lipitor. Please look to see if he may have taken Lipitor in the past. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task G: (Change Medication List) Discontinue the Azithromycin, because he completed the course. Also, the patient tells you that his Endocrinologist has put him on Aldactone (spironolactone) 25mg daily for high blood pressure a month ago. If Aldactone is not in the chart, add that to the patient’s medication list. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task H: (Record Medication List) Aldactone 25mg Tablet, Take 1 tablet daily, Started: one month ago Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.2>> Start Time:_______________________________


Task I: Please locate the patient’s problem list. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task J: (Change Problem List) While reviewing the patient’s problem list, you notice an old diagnosis of childhood asthma is still listed as a current problem. Please update the system so that the childhood asthma is no longer listed. Also add that the patient has acute laryngitis. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task K: (Record Problem List) Acute Laryngitis Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task L: When was the patient’s diabetes mellitus originally diagnosed? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.3>>

Scenario 5 – Implantable Devices

Scenario 5 – Slide 7 Task A: Your next patient is 64 years old. She tells you she had surgery on her leg and hands you a document with some information about the bone screw that was used. Please add the bone screw alphanumeric to her chart. Your patient’s name is: Tracy Grace First name: Tracy; Last name: Grace Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: She hands you a document with some information about an orthopedic bone pin that was implanted at the same time as a bone screw. Please add this bone pin to her chart. Clicks / Steps: ____________________________


Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: The patient says that her polyester suture was taken out on July 1, 2017. Please update her chart. Date: July 1, 2017 Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: If you needed to find the Device Identifier information for the cardiopulmonary tubing connector because there was a recall on these devices, where would you find this? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (Split between participants)


Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task F: How many Active Implanted Devices does the patient currently have in her record? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task G: How many total Implanted Devices are recorded for the patient, this includes all devices not just active ones? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 5>>

Scenario 6 – Order ePrescribe/cc

Scenario 6 – Slide 8 Task A: Your next patient is a 10-year-old who has come in with strep throat and needs to be prescribed with an antibiotic for Acute Tonsillitis. Please prescribe Zithromax (azithromycin) 200mg/ 5ml daily for 5 days and send it to the NYC pharmacy. Please sign the order once you are done. Your patient’s name is: Thomas Douglas First name: Thomas; Last name: Douglas Do you have any questions about what I am asking you to do?


Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Zithromax 200 mg / 5 mL, Take 5mL everyday for 5 days; Qty: 25 mL; Pharmacy: NYC Pharmacy 10.6mu Does the system allow you to order in CCs? Y / N

Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 6>>


Scenario 7.1 – Slide 9 Task: (CDS: Find CDS Icon) Your next patient is 8-year old Jackson Wallace. You want to see if the system has provided any meaningful use recommendations for this patient in their chart. Your patient’s name is: Jackson Wallace First name: Jackson; Last name: Wallace Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task A: (CDS: InfoButton, Demographics) The EHR provides you the ability to get additional information about Pediatric Diabetes from an outside source. Please go to where you would find this additional information. Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task B: Who is the developer of this information? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U)


Task C: (CDS: InfoButton, Problem List) You would like to get additional information about children with asthma. Find this information. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: Who is the developer of this information? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (CDS: InfoButton, Medication List) You would like to get additional information about ADHD medication. Find this information. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 7.1>>

Scenario 7.2 – CDS: Child Scenario 7.2 – Slide 10 Go to PATIENT LIST Task: (CDS: Find CDS Icon) Did not Complete Infobutton: Your next patient is an 8-year old. You want to see if the system has provided any meaningful use recommendations for this patient in their chart. Your patient’s name is: Jackson Wallace First name: Jackson; Last name: Wallace Completed Infobutton: Your patient is still Jackson Wallace. Please go back to the meaningful use recommendations for this patient. Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task A: (CDS: Med List) Using the recommendations, complete the ADHD medication recommendation and order Ritalin 10mg. Also indicate the Pediatric Diabetes recommendation was completed. Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start.


Ritalin 10 mg tablet Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (CDS: Combo: Demographic and Problem; Pediatric Diabetes) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 7.2>>

Scenario 7.3 – CDS - Adult Scenario 7.3 – Slide 11 Task A1: (CDS: Demographics: Breast Cancer) Your next patient is a 60-year-old female. Please use the system provided meaningful use recommendations to indicate the items have been completed. Your patient’s name is: Leslie Rose First name: Leslie; Last name: Rose Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Your patient says she had her Breast Cancer screening on March 10. Please specify that. Start Time:_______________________________ Completed the Breast Cancer Screening on March 10, 2017 Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task A2: (CDS: Reference 1) Who is the developer of the foot exam intervention?

Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (CDS: Problem List) Completed the Foot exam today Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD )


Task C: (CDS: Reference 2) Who is the developer of the LDL recommendation? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: (CDS: Lab Test) Go ahead and do what the LDL Recommendation tells you to do. The information you need for the LDL is: Today’s date and a result of 80. Enter Results of the LDL-C: 80 mg/dl for today Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (CDS: Vital Signs) Completed the High Blood Pressure Care Plan today Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task F: (CDS: Medication Allergy: Shingles vaccine and neomycin allergy) Your patient has not had a shingles vaccine, however, the patient has an allergy to neomycin which is an ingredient in the shingles vaccine. Please verify the patient has an allergy to neomycin and verify there is NO recommendation advising the patient get a shingles vaccine. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 7.3>>

Scenario 8 – ePrescribe

Scenario 8 – Slide 12 Task A: (Validation Task for “0.5”) Your next patient is a 67-year-old female whose has a diagnosis of type 2 diabetes mellitus. Please prescribe half of a tablet of 1000 mg Metformin daily for 30 days to her local NYC Pharmacy. Please type “.50” for the Dose. We ask you to type “.50” to see what the system will do to meet the MU3 regulations. Your patient’s name is: Sidney Cassidy First name: Sidney; Last name: Cassidy Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Metformin1000mg .5 tablets daily for 30 days, 3 refills; Pharmacy: NYC Pharmacy 10.6mu Did the dose on this screen change to 0.5? Y / N Please complete the order and sign for the order.


Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Cancel Prescription, Part A) The patient tells you that she has already received a new prescription for her inhaler from another prescriber. You had also prescribed that inhaler. Please cancel your prescription to the pharmacy for the Albuterol inhaler for this patient. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (Refill Prescription, Part A & B) You are reviewing requests from the pharmacy and see that the pharmacy has sent an electronic refill request for this same patient (Sidney Cassidy). You saw the patient in the clinic 3 days ago for vomiting. At the time, you prescribed symptomatic Ondansetron. Now that pharmacy has sent an electronic refill request for another 3 days for the patient. Approve that request. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Slide 13 for all three Change tasks Task D1: (Change Prescription: approve Generic) After receiving your prescription for Protonix 40mg Dispensed as Written, that same pharmacy sent a change request asking for authorization to change the drug to pantoprazole 40mg because Protonix is not covered by the patient’s insurance plan. Approve the pharmacy’s request to change the prescription. pantoprazole 40mg, one tablet daily Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 8.1>> Slide 13 for all three Change tasks Task D2: (Change Prescription: Approve with edits) The pharmacist sent a change request to suggest other therapeutic alternatives that the patient may try to reduce their unpleasant side effects. The pharmacy recommends one Lipitor 20 mg oral tablet daily but upon review of the patient’s symptoms and cholesterol results, you prefer Zocor 20mg – daily with three refills. Respond to the pharmacy’s request by letting them know you want to change the prescription to Zocor Zocor 20mg tablet, Take 1 table daily, 30 tablets with 3 refills Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 8.1>> Slide 13 for all three Change tasks Task D3: (Change Prescription: Prior Authorization)


After receiving the prescription for Breo Ellipta, the pharmacy sends a change request asking for prior authorization to supply this drug since other similar drugs on the market are much cheaper. Respond to their request by authorizing the Breo Ellipta prescriptions. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 8.1>> Slide 16 Task F: (Fill Status) Your next patient is Teague Rodgersson. The pharmacy has sent a refill request for Mr. Rodgersson to renew his prescription for Cefdinir. You suspect that Mr. Rodgersson is requesting a refill too soon. You want to determine when this prescription was last filled since he recently started taking that medication. Please tell me the last fill date for Cefdinir. Your patient’s name is: Teague Rodgersson First name: Teague; Last name: Rodgersson Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task G: (Medication History) For the next task, your patient, Mr. Rodgersson, has come to the clinic stating that he thinks that one of his medications is causing very unpleasant side effects. You did not prescribe his medications, but you want to review his medication history. Please locate the patient’s medication history and see when the fill information was last downloaded to his chart. Do you ever manually download a patient’s medication history? Y / N Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 8.2>>

Scenario 10 – Provider Configuration Scenario 10 – Slide 15 Task A: You are being notified too often for drug interactions. Change the Drug interaction warning levels from displaying all warnings to only serious and severe warnings. Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________


Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: Please go back to the patient list. You are being notified too often for drug interactions. Set the system settings so that the prescriber is warned when there is are only severe drug interaction warnings. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Stop Time:_______________________________ <<Go to Scenario Rating 10>>



Moderator Guide Specific to Inpatient Nurses Hello. Welcome, my name is _____________ and I’m a moderator, also we have ______________ online with us, s/he is the data logger. Can you see the welcome screen on your screen? Be sure to enlarge the shared screen on your screen. If you haven’t already, you can click the enlarge icon in the upper right corner and you can click the diagonal arrows here Thank you for participating in this study. Our session today will last 60 minutes. During that time, you will be performing tasks with AthenaHealth’s EHR in order to evaluate the extent to which the application is effective/not effective and efficient/not efficient in terms of Meaningful Use certification criteria. The first thing I would like to do is to show you this consent form to participate. Do you have any questions? I am going to start the recording now. Basically, because parts of this product are not public yet, we are asking you not to discuss what you see today with anyone. We are recording the audio and screenshots of our session today. This recording is only to make sure we capture and analyze the data accurately. It will not be used for any other purpose. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. OK, so I will have it on the record that you provided consent to be in this session, is that correct? This is what you agreed to on the consent form – I show it here so research teams that might go back and view this recording are under the same expectations. Thank you for filling out the Meaningful Use Demographics survey. Because this is for Meaningful Use, there are some background questions we need to ask.

Gender


Occupation/Role





Reviewed Training (yes/no) I am going to read this introduction to you because I want to be sure that I don’t miss anything. We did not have any involvement in this software’s creation. We are from an independent consulting company. This is a great chance for you to give the Athena team feedback about the application we are going to look at today, so please be honest with your opinions. I will ask you to complete a few activities using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. Some


activities might seem simple to you. Other activities might seem difficult. And there will be some activities that you will not able to complete. I am telling you this because I want you to remember that we are not testing you. You are helping us test the usability of the application. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Please stick to the task instructions provided to you. We are aggregating results so we need all participants to complete the same tasks. When you are doing these activities, I am not going to interact or talk to you. I and <<note taker>> will be taking notes about what you are doing. Because I am not going to be talking with you while you do the activities, I want you to make it clear to me when you are done with an activity by saying "I'm done." There are a number of reasons you might be done. (1) Done because you completed the activity. You know you completed it. So you say I’m Done. (2) Done because you have tried, you know you have not completed the activity, but you are not going to try anything else. So you say I’m Done. (3) Done because you feel like if you asked a question you could finish the activity. The system you are using is likely set different from what you use in your practice. So you might say I have a such and such button on my system. I don’t see on here. And I will tell you the next step and we will go from there. Sometimes I may not answer you. I’m not trying to be difficult; it’s just that I want to see how you would interact with the product on your own. If you get stuck, see if you can figure out what to do. As I mentioned, you can stop the task at any time, but we’d like you to give it your best shot before giving up. And if you get stuck or decide to give up, it means Athena will be getting feedback that the EHR needs more work done to make it easier to use. Remember, we are evaluating the product, not you. Do you have any questions or concerns before we begin? During the session, I will be reading a scenario to you. Any information that you need to complete your work is on the slide, we call it a task sheet. Let me show it to you now. You can click on it anytime to see what is needed. Or you can also ask me to repeat the task. To return to the application, click on this icon. After each task, you will rate how easy or difficult it was to perform. You will find that under this icon. Ok, I will pass you control of the mouse and keyboard. You can click that you will take control. Scenario 1RN – CPOEs


You are Alex Jones providing care for a patient. The patient is a 60-year old male whose chief complaint is respiratory distress. He has a diagnosis of pneumonia. You are looking at the patient’s chart, and you first want to see if there are any orders placed since you were last in the chart. Please acknowledge the following new orders for the patient:

• Levofloxacin IV • CBC w/ auto diff • X-ray, chest, 2-view

Your patient’s name is: Jack Wilson First name: Jack; Last name: Wilson Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________


Task A: Acknowledge Medication Orders Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: Acknowledge Laboratory Orders Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task C: Acknowledge Imaging Orders Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD )

Stop Time:_______________________________ <<Go to Scenario Rating 1RN>>



Scenario 4 – Slide 4 Task A: Your next patient is a 55-year old male with a history of rheumatoid arthritis whose chief complaint was sinus pressure. He has a diagnosis of sinus infection. You want to go over his chart with him. You start by reviewing his allergies. Please name one of the medications the patient is allergic to? Your patient’s name is: Perry Benson First name: Perry; Last name: Benson Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Record Medication Allergy List) Mr. Benson’s wife arrives as you are reviewing his allergies. She tells you he gets a severe rash to iodine, which he always forgets to mention. He confirms the allergy. Add this allergy to the patient’s current allergy list. In addition, Mr. Benson’s wife reminds him that he started getting a mild headache, besides a rash, due to his sulfa allergy, and he agrees. Add a reaction of “headache” with mild severity to his sulfa allergy. Iodine, Reaction: rash, Severity: Severe, No onset date


Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (Change Medication Allergy List) Sulfa Allergy: Additional Reaction: headache, Severity: mild, Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: When was the patient’s first reaction to Penicillin? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.1>> Start Time:_______________________________ Task E: You continue reviewing Mr. Benson’s chart, please locate his active home medications. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task F: You see that Mr. Benson is currently taking Cardizem but you wonder if he has ever taken Lipitor. Please look to see if he may have taken Lipitor in the past. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task G: (Change Medication List) Mr. Benson tells you his PCP discontinued his Azithromycin today. Please discontinue the Azithromycin from the patient’s home medication list. Also, the patient tells you that his Endocrinologist has put him on Aldactone (spironolactone) 25mg daily for high blood pressure a month ago. If Aldactone is not in the chart, add that to the patient’s home medication list. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task H: (Record Medication List) Aldactone 25mg Tablet, Take 1 tablet daily, Started: one month ago Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________



Start Time:_______________________________ Task I: Please locate the patient’s diagnoses list. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task J: (Change Problem List) While reviewing the patient’s problem list, you notice an old diagnosis of childhood asthma was pulled in as an active problem. Please update the system so that the childhood asthma is no longer listed as an active problem. Also add that the patient has acute laryngitis today. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task K: (Record Problem List) Acute Laryngitis Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task L: When was the patient’s diabetes mellitus originally diagnosed? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.3>>


Scenario 5 – Slide 5 Task A: Your next patient is 60 years old. She tells you she had surgery on her leg and hands you a document with some information about the bone screw that was used. Please add the bone screw alphanumeric to her chart. To do that, you will want to click inside the alphanumeric. Your patient’s name is: Tracy Grace First name: Tracy; Last name: Grace Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________


Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: She hands you a document with some information about an orthopedic bone pin that was implanted at the same time as a bone screw. Please add this bone pin to her chart. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: The patient says that her polyester suture was taken out on July 1, 2017. Please update her chart. Date: July 1, 2017 Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: If you needed to find the Device Identifier information for the cardiopulmonary tubing connector because there was a recall on these devices, where would you find this? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (Split between participants)




Scenario 7.1 – Slide 8 Task: (CDS: Find CDS Icon)


Your next patient is 8-year-old Jackson Wallace. You want to see if the system has provided any meaningful use recommendations for this patient in their chart. Your patient’s name is: Jackson Wallace First name: Jackson; Last name: Wallace Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task A: (CDS: InfoButton, Demographics) The EHR provides you the ability to get additional information about Pediatric Diabetes from an outside source. Please go to where you would find this additional information. Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task B: Who is the developer of this information? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (CDS: InfoButton, Problem List) You would like to get additional information about children with asthma. Find this information. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: Who is the developer of this information? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (CDS: InfoButton, Medication List) You would like to get additional information about ADHD medication. Find this information. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________



Scenario 9 – Death Demographics

Scenario 9 – Slide 13 Task A: Meaningful Use requires that we get your feedback on how the system supports documenting and editing the cause and date of death. You will need to discharge the patient and document the cause of death as acute heart failure and document the date of death as today’s date. We are going to use the patient named James Scott First name: James; Last name: Scott Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: Now change the information you just entered for James Scott. The diagnosis was not acute heart failure, please change the cause of death as cardiac arrest and document the date of death as yesterday’s date. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Stop Time:_______________________________ <<Go to Scenario Rating 9>>



Moderator Guide Specific to Ambulatory Clinical Staff Hello. Welcome, my name is _____________ and I’m a moderator, also we have ______________ online with us, s/he is the data logger. Can you see the welcome screen on your screen? Be sure to enlarge the shared screen on your screen. If you haven’t already, you can click the enlarge icon in the upper right corner and you can click the diagonal arrows here Thank you for participating in this study. Our session today will last 60 minutes. During that time, you will be performing tasks with AthenaHealth’s EHR in order to evaluate the extent to which the application is effective/not effective and efficient/not efficient in terms of Meaningful Use certification criteria. The first thing I would like to do is to show you this consent form to participate. Do you have any questions? I am going to start the recording now. Basically, because parts of this product are not public yet, we are asking you not to discuss what you see today with anyone. We are recording the audio and screenshots of our session today. This recording is only to make sure we capture and analyze the data accurately. It will not be used for any other purpose. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. OK, so I will have it on the record that you provided consent to be in this session, is that correct? This is what you agreed to on the consent form – I show it here so research teams that might go back and view this recording are under the same expectations. Thank you for filling out the Meaningful Use Demographics survey. Because this is for Meaningful Use, there are some background questions we need to ask.

Gender


Occupation/Role





Reviewed Training (yes/no) I am going to read this introduction to you because I want to be sure that I don’t miss anything. We did not have any involvement in this software’s creation. We are from an independent consulting company. This is a great chance for you to give the Athena team feedback about the application we are going to look at today, so please be honest with your opinions. I will ask you to complete a few activities using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. Some activities might seem simple to you. Other activities might seem difficult. And there will be some activities that


you will not able to complete. I am telling you this because I want you to remember that we are not testing you. You are helping us test the usability of the application. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Please stick to the task instructions provided to you. We are aggregating results so we need all participants to complete the same tasks. When you are doing these activities, I am not going to interact or talk to you. I and <<note taker>> will be taking notes about what you are doing. Because I am not going to be talking with you while you do the activities, I want you to make it clear to me when you are done with an activity by saying "I'm done." There are a number of reasons you might be done. (1) Done because you completed the activity. You know you completed it. So you say I’m Done. (2) Done because you have tried, you know you have not completed the activity, but you are not going to try anything else. So you say I’m Done. (3) Done because you feel like if you asked a question you could finish the activity. The system you are using is likely set different from what you use in your practice. So you might say I have a such and such button on my system. I don’t see on here. And I will tell you the next step and we will go from there. Sometimes I may not answer you. I’m not trying to be difficult; it’s just that I want to see how you would interact with the product on your own. If you get stuck, see if you can figure out what to do. As I mentioned, you can stop the task at any time, but we’d like you to give it your best shot before giving up. And if you get stuck or decide to give up, it means Athena will be getting feedback that the EHR needs more work done to make it easier to use. Remember, we are evaluating the product, not you. Do you have any questions or concerns before we begin? During the session, I will be reading a scenario to you. Any information that you need to complete your work is on the slide, we call it a task sheet. Let me show it to you now. You can click on it anytime to see what is needed. Or you can also ask me to repeat the task. To return to the application, click on this icon. After each task, you will rate how easy or difficult it was to perform. You will find that under this icon. Ok, I will pass you control of the mouse and keyboard. You can click that you will take control. Scenario 1RN – CPOEs


You are Alex Jones providing care for a patient. The patient is a 60-year old male whose chief complaint is respiratory distress. He has a diagnosis of pneumonia. You are looking at the patient’s chart, and you first want to see if there are any orders placed since you were last in the chart. Please tell us any pending orders for the patient. Your patient’s name is: Jack Wilson First name: Jack; Last name: Wilson Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Task A: Pending Laboratory Orders Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________


Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: Pending Imaging Orders Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task C: Pending Medication Orders

Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD )

Stop Time:_______________________________ <<Go to Scenario Rating 1RN>>


Scenario 3 – CIRI

Scenario 3 – Slide 4 Task A: There is a feature in the EHR that electronically reconciles clinical information from an external source. Today we are going to reconcile Allergies, Medications, and Problems from the external source named, James Getwell with information in this EHR. For example, let’s say the patient came to the clinic and Community Health folks sent you clinical information for your team to reconcile with the information in your EHR. Have you used this feature in your practice? Yes / No If yes: what have you used it for? Meds / Med Allergies / Problems

Please see if your patient has items that need to be reconciled. Your patient’s name is: Perry Benson First name: Perry; Last name: Benson Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Example with Med Allergy, Reconcile Allergies) We are going to have you reconcile the patient’s allergies, medications and problems. For this task, I am going to ask you to do things that you may not do if you were using your clinical judgment. So you can refer to the task sheet for the items that should or should not be included in the EHR.


I am going to walk you through part of the patient’s Medication Allergy reconciliation. In the task list it specifies to add Peanut to the patient’s chart. To do that, you would click the radio button from the imported data on Peanut. Or if you changed your mind and didn’t want that item added to the patient’s chart, you could just click on the Patient Chart side and it would not be included in the patient’s chart. If you preferred to have the item on the patient’s chart and not from the outside source, you would just click the radio button for the item in the patient’s chart. If any medications are in the patient’s chart, but are not listed in the Continuity of Care (CCD), choose the medication from the patient’s chart. Now let’s have you reconcile the patient’s Medication Allergies. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (Reconcile Medications, Cardizem mg difference) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: (Reconcile Problems) Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task E: (Complete Reconciliation): Please complete the reconciliation. Based on this list, please tell me which allergies are being brought into the EHR. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Stop Time:_______________________________ <<Go to Scenario Rating 3.1 >> Task F: In this scenario, the reconciliation from the outside source for this patient has been completed. You now want to create and export a Continuity of Care Document (CCD) for this patient that will include the information you just brought into the chart. Please export the CCD now and submit by electronic file. Start Time:_______________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Stop Time:_______________________________ <<Go to Scenario Rating 3.2 >> And please rate only on the easy or difficulty of creating a continuity of care document.


Scenario 4 – Slide 5 Go to PATIENT LIST


Task A: If did not complete CIRI: Your next patient is a 55-year old male with a history of rheumatoid arthritis whose chief complaint was sinus pressure. He has a diagnosis of sinus infection. You want to go over his chart with him. You start by reviewing his allergies. Please name one of the medications the patient is allergic to? Your patient’s name is: Perry Benson First name: Perry; Last name: Benson If completed CIRI: Please go to the schedule, that is where we need to start this task. You next want to go over Perry Benson’s chart with him. You start by reviewing his allergies. Please name one of the medications the patient is allergic to? Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: (Record Medication Allergy List) Mr. Benson’s wife arrives as you are reviewing his allergies. She tells you he gets a severe rash to iodine, which he always forgets to mention. He confirms the allergy. Add this allergy to the patient’s current allergy list. In addition, Mr. Benson’s wife reminds him that he started getting a mild headache, besides a rash, due to his sulfa allergy, and he agrees. Add a reaction of “headache” with mild severity to his sulfa allergy. Iodine, Reaction: rash, Severity: Severe, No onset date Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (Change Medication Allergy List) Sulfa Allergy: Additional Reaction: headache, Severity: mild, Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: When was the patient’s first reaction to Penicillin? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.1>> Start Time:_______________________________ Task E: You continue reviewing Mr. Benson’s chart, please locate his current medications. Clicks / Steps: ____________________________


Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task F: You see that Mr. Benson is currently taking Cardizem but you wonder if he has ever taken Lipitor. Please look to see if he may have taken Lipitor in the past. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task G: (Change Medication List) Discontinue the Azithromycin, because he completed the course. Also, the patient tells you that his Endocrinologist has put him on Aldactone (spironolactone) 25mg daily for high blood pressure a month ago. If Aldactone is not in the chart, add that to the patient’s medication list. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task H: (Record Medication List) Aldactone 25mg Tablet, Take 1 tablet daily, Started: one month ago Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.2>> Start Time:_______________________________ Task I: Please locate the patient’s problem list. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task J: (Change Problem List) While reviewing the patient’s problem list, you notice an old diagnosis of childhood asthma is still listed as a current problem. Please update the system so that the childhood asthma is no longer listed. Also add that the patient has acute laryngitis. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task K: (Record Problem List) Acute Laryngitis Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task L: When was the patient’s diabetes mellitus originally diagnosed?


Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 4.3>>


Scenario 5 – Slide 6 Task A: Your next patient is 64 years old. She tells you she had surgery on her leg and hands you a document with some information about the bone screw that was used. Please add the bone screw alphanumeric to her chart. Your patient’s name is: Tracy Grace First name: Tracy; Last name: Grace Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task B: She hands you a document with some information about an orthopedic bone pin that was implanted at the same time as a bone screw. Please add this bone pin to her chart. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: The patient says that her polyester suture was taken out on July 1, 2017. Please update her chart. Date: July 1, 2017 Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: If you needed to find the Device Identifier information for the cardiopulmonary tubing connector because there was a recall on these devices, where would you find this? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (Split between participants)





Scenario 7.1 – Slide 6 Task A: (CDS: Find CDS Icon) Your next patient is 8-year old Jackson Wallace. You want to see if the system has provided any meaningful use recommendations for this patient in their chart. Your patient’s name is: Jackson Wallace First name: Jackson; Last name: Wallace Do you have any questions about what I am asking you to do? Remember to tell us when you are done. Go ahead and start. Start Time:_______________________________ Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task A: (CDS: InfoButton, Demographics) The EHR provides you the ability to get additional information about Pediatric Diabetes from an outside source. Please go to where you would find this additional information. Correct Patient Y / N ___________________________ Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U)


Task B: Who is the developer of this information? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) / Unfamiliar Navigation U) Task C: (CDS: InfoButton, Problem List) You would like to get additional information about children with asthma. Find this information. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task D: Who is the developer of this information? Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Task E: (CDS: InfoButton, Medication List) You would like to get additional information about ADHD medication. Find this information. Clicks / Steps: ____________________________ Pass / Fail Use Error (UE) / Close Call (CC) / Operational Difficulties (OD ) Stop Time:_______________________________ <<Go to Scenario Rating 7.1>>



Moderator Guide Specific to Configuration Test This Moderator Guide was used for both Ambulatory and Inpatient Configuration sessions.

Administrator __________________ Data Logger ____________________ Date __________________________ Participant # ____________________ Location __Remote______________

Orientation (5 minutes) Hello. Welcome, my name is _____________ and I’m a moderator. Can you see the welcome screen on your screen? Be sure to enlarge the shared screen on your screen. If you haven’t already, you can click the enlarge icon in the upper right corner and you can click the diagonal arrows here. Thank you for participating in this study. Our session today will last 30 minutes. During that time, you will be performing tasks with AthenaHealth’s EHR in order to evaluate the extent to which the application is effective/not effective and efficient/not efficient in terms of Meaningful Use certification criteria. The first thing I would like to do is to show you this consent form to participate. Do you have any questions? I am going to start the recording now. Basically, because parts of this product are not public yet, we are asking you not to discuss what you see today with anyone. We are recording the audio and screenshots of our session today. This recording is only to make sure we capture and analyze the data accurately. It will not be used for any other purpose. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. OK, so I will have it on the record that you provided consent to be in this session, is that correct? Because this is for Meaningful Use, there are some background questions we need to ask. If any make you feel uncomfortable, just let me know.

Gender What best describes your Age 20s, 30s, 40s, 50s, 60s, 70s, older Education (highest degree) No high school degree High school graduate, diploma Some college credit, no degree Trade/technical/vocational Associate degree Bachelor’s degree Master’s Degree Doctorate Degree Occupation/Role Professional Experience (months/yrs) Computer Experience (months/yrs) All computer experience


EHR Product Experience (months/yrs) Assistive Technology Needs (yes-what, no) Reviewed Training (yes, no)

I am going to read this introduction to you because I want to be sure that I don’t miss anything. I did not have any involvement in this software’s creation. I am from an independent consulting company. Companies hire my company to conduct activities with people like you that use their products, services, and websites. This is a great chance for you to give the athenahealth team feedback about the application we are going to look at today, so please be honest with your opinions. I will ask you to complete a few activities using this system and answer some questions. We are interested in how easy or how difficult this system is to use, what in it would be useful to you, and how we could improve it. Some activities might seem simple to you. Other activities might seem difficult. And there will be some activities that you will not able to complete. I am telling you this because I want you to remember that we are not testing you. You are helping us test the application. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Do not do anything more than asked. When you are doing these activities, I am not going to interact or talk to you while you are completing the activity. I will be taking notes about what you are doing. Because I am not going to be talking with you while you do the activities, I want you to make it clear to me when you are done with an activity by saying "I'm done." There are a number of reasons you might be done. (1) Done because you completed the activity. You know you completed it. So you say I’m Done. (2) Done because you have tried, you know you have not completed the activity, but you are not going to try anything else. So you say I’m Done. (3) Done because you feel like if you asked a question you could finish the activity. The system you are using may be set different from what you use in your practice. So you might say I have a such and such button on my system. I don’t see on here. And I will tell you the next step and we will go from there. Do you have any questions or concerns before we begin? During the session, I will be reading a scenario to you. Any information that you need to complete your work is on the task sheet on the right. You can look at it anytime to see what is needed. Or you can also ask me to repeat the task. To return to the application, click on this icon. After each task, you will rate how easy or difficult it was to perform. You will find that under this icon. Ok, I will pass you control of the mouse and keyboard. You can click that you will take control.

Configuration Scenario 1 I will read this to you now, please do not start until I tell you to.


Task A and B: “The BP Control (140/90) guideline is a measure that should identify patients 18 years and older whose most recent blood pressure is less than 140/90 within the past 12 months. You would like to use this measure in your practice. In addition, your patients tend to be as young as 16 years old. You would like to adjust the age range to 16-999 years old for this guideline. Please activate the BP Control guideline and adjust the age range.” Do you have any questions about what I am asking you to do? Go ahead and get started. Remember to tell us when you are done.” Start Time:_______________________________ Stop Time:_______________________________ Clicks / Steps: ____________________________ Use Error (UE) / Close Call (CC) / Operational Difficulties (OD) Pass/Fail Please go to the Task Questionnaire – please rate the tasks based on how you feel right now performing the task – don’t rate it for how you think others would feel or how you would feel if you had more training on the task.

Configuration Scenario 2 I will read this to you now, please do not start until I tell you to. Task C, D, and E: “The Alcohol Use Assessment guideline is a measure that identifies patients 18 years and older whose alcohol use was recorded during an encounter in the last 24 months. You would like to use this measure in your practice. In addition, your patients tend to be as young as 16 years old. You would like to adjust the age range to 16-999 years old for this guideline and you would like this guideline to be queried yearly rather than every 2 years. Please activate the Alcohol Use Assessment guideline, and adjust both the age range and the queried time period.” Do you have any questions about what I am asking you to do? Go ahead and get started. Remember to tell us when you are done.” Start Time:_______________________________ Stop Time:_______________________________ Clicks / Steps: ____________________________ Use Error (UE) / Close Call (CC) / Operational Difficulties (OD) Pass/Fail Please go to the Task Questionnaire – please rate the tasks based on how you feel right now performing the task – don’t rate it for how you think others would feel or how you would feel if you had more training on the task.

OK, that is the last task I have for you. Now we will go through a brief questionnaire about your experience today.


<<Administer the SUS>> If time permits: We would like to collect any items, questions or concerns you have about the current AthenaHealth EHR you have installed. Is there anything you would like to tell us?


Moderator Guide Specific to Inpatient Registration Test This Moderator Guide was used for Inpatient Registration sessions.


Orientation (5 minutes) Hello. Welcome, my name is _____________ and I’m a moderator. Can you see the welcome screen on your screen? Be sure to enlarge the shared screen on your screen. If you haven’t already, you can click the enlarge icon in the upper right corner and you can click the diagonal arrows. Thank you for participating in this study. Our session today will last 30 minutes. During that time, you will be performing tasks with AthenaHealth’s EHR in order to evaluate the extent to which the application is effective/not effective and efficient/not efficient in terms of Meaningful Use certification criteria. The first thing I would like to do is to show you this consent form to participate. Do you have any questions? I am going to start the recording now. Basically, because parts of this product are not public yet, we are asking you not to discuss what you see today with anyone. We are recording the audio and screenshots of our session today. This recording is only to make sure we capture and analyze the data accurately. It will not be used for any other purpose. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. OK, so I will have it on the record that you provided consent to be in this session, is that correct? Because this is for Meaningful Use, there are some background questions we need to ask. If any make you feel uncomfortable, just let me know.



EHR Product Experience (months/yrs) Assistive Technology Needs (yes-what, no)


Registration Scenario 1 I will read this to you now, please do not start until I tell you to.


Task A and B: “You need to start a record for a new patient and you need to document some demographic items. Using the system, document this new information that you see on your task sheet.”

• Sex: Female • Date of Birth: 11/ 7/ 1980 • Address: 123 Main Street Schenectady, NY 12345 • Home phone: 555-555-5588 • Contact preference: Mail • Primary Care Provider: None • Language: English • Race: Declined to provide • Ethnicity: Declined to provide • Sexual Orientation: Declined to provide • Gender Identity: Declined to provide • Level of care: Inpatient • Department: Local Community Hospital • Check-in: Today • Admission source: 2- Clinic or Physician’s Office • Admission Type: 3 - Elective • Attending provider: Pat Rogers • Admitting provider: Charles Brown • Insurance: Self Pay • Patient’s relationship to guarantor: Self

Your patient’s name is Chris Kelly. First name: Chris, Last name: Kelly. Do you have any questions about what I am asking you to do? Go ahead and get started. Remember to tell us when you are done.” Start Time:_______________________________ Stop Time:_______________________________ Clicks / Steps: ____________________________ Use Error (UE) / Close Call (CC) / Operational Difficulties (OD) Pass/Fail Please go to the Task Questionnaire – please rate the tasks based on how you feel right now performing the task – don’t rate it for how you think others would feel or how you would feel if you had more training on the task.



Task C, D, and E: “The patient states he was feeling poorly and completed his paperwork as fast as he could. He would like to complete the paperwork more thoroughly and accurately. He returns the paperwork. Please change the following information in the patient’s file to what you see on your task sheet:

• Date of Birth • Race • Ethnicity • Preferred Language • Sex • Sexual Orientation • Gender Identity”

Do you have any questions about what I am asking you to do? Go ahead and get started. Remember to tell us when you are done.” Start Time:_______________________________ Stop Time:_______________________________ Clicks / Steps: ____________________________ Use Error (UE) / Close Call (CC) / Operational Difficulties (OD) Pass/Fail Please go to the Task Questionnaire – please rate the tasks based on how you feel right now performing the task – don’t rate it for how you think others would feel or how you would feel if you had more training on the task.

OK, that is the last task I have for you. Now we will go through a brief questionnaire about your experience today. <<Administer the SUS>> If time permits: We would like to collect any items, questions or concerns you have about the current AthenaHealth EHR you have installed. Is there anything you would like to tell us?


Moderator Guide Specific to Ambulatory Registration Test This Moderator Guide was used for Ambulatory Registration sessions.


Orientation (5 minutes) Hello. Welcome, my name is _____________ and I’m a moderator. Can you see the welcome screen on your screen? Be sure to enlarge the shared screen on your screen. If you haven’t already, you can click the enlarge icon in the upper right corner and you can click the diagonal arrows. Thank you for participating in this study. Our session today will last 30 minutes. During that time, you will be performing tasks with AthenaHealth’s EHR in order to evaluate the extent to which the application is effective/not effective and efficient/not efficient in terms of Meaningful Use certification criteria. The first thing I would like to do is to show you this consent form to participate. Do you have any questions? I am going to start the recording now. Basically, because parts of this product are not public yet, we are asking you not to discuss what you see today with anyone. We are recording the audio and screenshots of our session today. This recording is only to make sure we capture and analyze the data accurately. It will not be used for any other purpose. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. OK, so I will have it on the record that you provided consent to be in this session, is that correct? Because this is for Meaningful Use, there are some background questions we need to ask. If any make you feel uncomfortable, just let me know.



EHR Product Experience (months/yrs) Assistive Technology Needs (yes-what, no)


Registration Scenario 1 I will read this to you now, please do not start until I tell you to.


Task A and B: “You need to start a record for a new patient and you need to document some demographic items. Using the system, document this new information that you see on your task sheet.”

• Date of Birth: 11/ 7/ 1980 • Race: Declined to provide • Ethnicity: Declined to provide • Preferred Language: English • Sex: Female • Sexual Orientation: Declined to provide • Gender Identity: Declined to provide

Your patient’s name is Chris Kelly. First name: Chris, Last name: Kelly. Do you have any questions about what I am asking you to do? Go ahead and get started. Remember to tell us when you are done.” Start Time:_______________________________ Stop Time:_______________________________ Clicks / Steps: ____________________________ Use Error (UE) / Close Call (CC) / Operational Difficulties (OD) Pass/Fail Please go to the Task Questionnaire – please rate the tasks based on how you feel right now performing the task – don’t rate it for how you think others would feel or how you would feel if you had more training on the task.



Task C, D, and E: “The patient states he was feeling poorly and completed his paperwork as fast as he could. He would like to complete the paperwork more thoroughly and accurately. He returns the paperwork. Please change the following information in the patient’s file to what you see on your task sheet:

• Date of Birth: 1/17/ 1980 • Race: Black or African American; Dominican • Ethnicity: Dominican • Preferred Language: Spanish • Sex: Male • Sexual Orientation: Don’t know • Gender Identity: Genderqueer, neither exclusively male nor female

Do you have any questions about what I am asking you to do? Go ahead and get started. Remember to tell us when you are done.” Start Time:_______________________________ Stop Time:_______________________________ Clicks / Steps: ____________________________ Use Error (UE) / Close Call (CC) / Operational Difficulties (OD) Pass/Fail Please go to the Task Questionnaire – please rate the tasks based on how you feel right now performing the task – don’t rate it for how you think others would feel or how you would feel if you had more training on the task.

OK, that is the last task I have for you. Now we will go through a brief questionnaire about your experience today. <<Administer the SUS>> If time permits: We would like to collect any items, questions or concerns you have about the current AthenaHealth EHR you have installed. Is there anything you would like to tell us?


Appendix 5: Example Task Data Sheets

Task Data Sheets Specific to Inpatient Prescribers








Task Data Sheets Specific to Ambulatory Prescribers









Task Data Sheets Specific to Inpatient Nurses




Task Data Sheets Specific to Ambulatory Clinical Staff




Task Data Sheets Specific to Configuration Tests This Task Sheet was used for both Ambulatory and Inpatient Configuration sessions.


Task Data Sheets Specific to Registration – Inpatient



Task Data Sheets Specific to Registration– Ambulatory



Appendix 6: End of Task Rating Scale Questionnaire At the end of each group of related tasks, participants were asked to rate that set of tasks on a scale ranging from Very Difficult to Very Easy. These textual descriptions were then converted to numeric ratings on a 1 to 5 scale to produce numeric a mean and standard deviation for each rating. Participants rated only based on the textual rating scale.

Very Difficult

Difficult Neither Difficult nor Easy

Easy Very Easy

(1) (2) (3) (4) (5)

1. Overall, the tasks in this scenario were__________.


Appendix 7: System Usability Scale Questionnaire In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems usability” known as the System Usability Scale or SUS. Lewis and Sauro (2009) and others have elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).

Strongly Agree

Strongly Agree

1 2 3 4 5

2. I think that I would like to use this system frequently

3. I found the system unnecessarily complex

4. I thought the system was easy to use

5. I think that I would need the support of a technical person to be able to use this system

6. I found the various functions in this system were well integrated

7. I thought there was too much inconsistency in this system

8. I would imagine that most people would learn to use this system very quickly

9. I found the system very cumbersome to use

10. I felt very confident using the system

11. I needed to learn a lot of things before I could get going with this system

http://www.usabilitynet.org/trump/documents/Suschapt.doc

Date post:	10-Jun-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Customized Common Industry Format Template for Electronic ... · 1. Order medications, review and...

Documents