Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 1
Part 1: Product and Developer Information1.1 Certified Product Information
1.2 Developer/Vendor Information
Part 2: ONC-Authorized Certification Body Information2.1 ONC-Authorized Certification Body Information
ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification
Product Name: Aprima PRM 2016Product Version: 16.0.16Domain: Ambulatory
Developer/Vendor Contact: Cornell Hardman
Richardson, TX 75081Website: Aprima.com
Test Type: Complete
Developer/Vendor Name: Aprima Medical Software, IncAddress: 1010 E Arapaho Rd, Suite 100
ONC-ACB Name: InfoGard Laboratories, Inc.Address: 709 Fiero Lane Suite 25
Email: [email protected]: 214-466-8000
ONC-ACB Contact: Adam Hardcastle
This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative:
Adam Hardcastle EHR Certification Body ManagerONC-ACB Authorized Representative Function/Title
San Luis Obispo, CA 93401Website: www.infogard.comEmail: [email protected]: (805) 783-0810
Signature and Date3/6/2017
Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 2
2.2 Gap Certification
(a)(1) (a)(19) (d)(6) (h)(1)(a)(6) (a)(20) (d)(8) (h)(2)(a)(7) (b)(5)* (d)(9) (h)(3)(a)(17) (d)(1) (f)(1)(a)(18) (d)(5) (f)(7)**
*Gap certification allowed for Inpatient setting only**Gap certification allowed for Ambulatory setting only
2.3 Inherited CertificationThe following identifies criterion or criteria certified via inherited certification
(a)(14)
(c)(3)
(f)(1)
(a)(2)
(a)(15)
(d)(1)
(f)(2)
The following identifies criterion or criteria certified via gap certification§170.314
No gap certification
§170.314
(a)(3)
(a)(16) Inpt. only
(d)(2)
(f)(3)
(a)(4)
(a)(17) Inpt. only
(d)(3)
(f)(4) Inpt. only
(a)(1)
(a)(7) (b)(3) (d)(6) (a)(8) (b)(4) (d)(7)
(a)(5) (b)(1) (d)(4) (a)(6) (b)(2) (d)(5)
(e)(2) Amb. only
(g)(4)
(a)(9) (b)(5) (d)(8)
(g)(1) (a)(10)
(b)(6) Inpt. only (d)(9) Optional
(g)(2)
(b)(8) (a)(13)
(c)(2)
(e)(3) Amb. only
No inherited certification
(f)(5) Amb. only
(f)(6) Amb. only
(f)(7) Amb. Only
(h)(1) (h)(2) (h)(3)
(a)(11) (b)(7) (e)(1)
(g)(3) (a)(12)
(c)(1) (b)(9)
(a)(18) (a)(19) (a)(20)
Test Results Summary for 2014 Edition EHR Certification
17-2999-R-0009-PRA Vl.O, March 6, 2017
Part 3: NVLAP-Accredited Testing Laboratory Information
Report Number: Test Date(s):
17-2999-R-0009 Vl.O
2/17/2017
3.1 NVLAP-Accredited Testing Laboratory Information ATL Name:
Accreditation Number:
Address:
Website:
Email:
Phone:
ATL Contact:
lnfoGard Laboratories, Inc.
NVLAP Lab Code 100432-0
709 Fiero Lane Suite 25
San Luis Obispo, CA 93401
www.infogard.com
(805) 783-0810
Milton Padilla
For more information on scope of accreditation, please reference
http://ts.nist.gov/Sta nda rds/scopes/1004320. htm
lNFOIGARD
Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory
Representative:
Milton Padilla EHR Test Body Manager
Function/Title
3.2 Test Information
3.2.1 Additional Software Relied Upon for Certification
Microsoft Excel (a)14, (c)l-3 Report generation
D No additional software required
©2017 lnfoGard. May be reproduced only in its original entirety, without revision 3
Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 4
3.2.2 Test Tools
Version
No test tools required
3.2.3 Test Data
3.2.4 Standards3.2.4.1 Multiple Standards Permitted
HL7 v2 Laboratory Restults Intervace (LRI) Validation ToolHL7 v2 Syndromic Surveillance Reporting Validation ToolTransport Testing ToolDirect Certificate Discovery Tool
Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter ] No alteration (customization) to the test data was necessary
Test ToolCypressePrescribing Validation ToolHL7 CDA Cancer Registry Reporting Validation ToolHL7 v2 Electronic Laboratory Reporting (ELR) Validation ToolHL7 v2 Immunization Information System (IIS) Reporting Valdiation Tool
Edge Testing Tool
(a)(13)
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
§170.207(j)HL7 Version 3 Standard: Clinical Genomics; Pedigree
(a)(15)(i)
§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain
§170.204(b)(2)HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide
The following identifies the standard(s) that has been successfully tested where more than one standard is permitted
Criterion # Standard Successfully Tested
(a)(8)(ii)(A)(2)
§170.204(b)(1)HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain
§170.204(b)(2)HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide
Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 5
None of the criteria and corresponding standards listed above are applicable
3.2.4.2 Newer Versions of Standards
No newer version of a minimum standard was tested
(b)(7)(i)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
(e)(1)(i) Annex A of the FIPS Publication 140-2List encryption and hashing algorithms
(a)(16)(ii) §170.210(g) Network Time Protocol Version 3 (RFC 1305)
§170. 210(g)Network Time Protocol Version 4 (RFC 5905)
(b)(2)(i)(A)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
(b)(8)(i)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
Common MU Data Set (15)
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
§170.207(b)(2)The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4)
The following identifies the newer version of a minimum standard(s) that has been successfully tested
Newer Version Applicable Criteria
(e)(1)(ii)(A)(2) §170.210(g) Network Time Protocol Version 3 (RFC 1305)
§170. 210(g)Network Time Protocol Version 4 (RFC 5905)
(e)(3)(ii) Annex A of the FIPS Publication 140-2List encryption and hashing algorithms
Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 6
3.2.5 Optional Functionality
No optional functionality tested
(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
Criterion # Optional Functionality Successfully Tested(a)(4)(iii) Plot and display growth charts
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)
(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(f)(3)
Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)
(e)(1) View, download and transmit data to a third party using the standard specified at §170.202(d) (Edge Protocol IG version 1.1)
(f)(7) Ambulatory setting only – transmission to public health agencies – syndromic surveillance - Create Data Elements
Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 7
3.2.6 2014 Edition Certification Criteria* Successfully Tested
TP** TD*** TP** TD***
Criteria #Version
Criteria #Version
(a)(4) (d)(3) (a)(5) (d)(4) (a)(6) (d)(5)
(a)(1) (c)(3) (a)(2) (d)(1) (a)(3) (d)(2)
(a)(10) (d)(9) Optional
(a)(11) (e)(1) (a)(12) (e)(2) Amb. only
(a)(7) (d)(6) (a)(8) (d)(7) (a)(9) (d)(8)
(a)(16) Inpt. only (f)(3) (a)(17) Inpt. only (f)(4) Inpt. only
(b)(1)
(f)(5) Optional & Amb. only
(b)(2)
(a)(13) (e)(3) Amb. only
(a)(14) (f)(1) (a)(15) (f)(2)
(a)(18)
(a)(20) (a)(19)
(b)(3)
(f)(6) Optional & Amb. only
(b)(4) (b)(5)
(g)(1)
(b)(6) Inpt. only
(g)(2)
(b)(8) (b)(9)
(f)(7) Amb. only
(h)(1) (h)(2) (h)(3)
***Indicates the version number for the Test Data (TD)
(b)(7)
(g)(3)
(c)(1)
(g)(4)
(c)(2)
**Indicates the version number for the Test Procedure (TP)
1.2 1.2
*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)
Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 8
3.2.7 2014 Clinical Quality Measures*Type of Clinical Quality Measures Successfully Tested:
CMS ID Version CMS ID Version CMS ID Version CMS ID Version2 90 136 155
22 117 137 15650 122 138 15752 123 139 15856 124 140 15961 125 141 16062 126 142 16164 127 143 16365 128 144 16466 129 145 16568 130 146 16669 131 147 16774 132 148 16975 133 149 17777 134 153 17982 135 154 182
CMS ID Version CMS ID Version CMS ID Version CMS ID Version9 71 107 172
26 72 108 17830 73 109 18531 91 110 18832 100 111 19053 102 11355 104 11460 105 171
Ambulatory Inpatient No CQMs tested*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov (navigation: 2014 Clinical Quality Measures)
Ambulatory CQMs
Inpatient CQMs
Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 9
3.2.8 Automated Numerator Recording and Measure Calculation3.2.8.1 Automated Numerator Recording
Automated Numerator Recording was not tested
3.2.8.2 Automated Measure Calculation
Automated Measure Calculation was not tested
3.2.9 Attestation
(a)(3) (a)(12) (a)(19) (b)(8)(a)(4) (a)(13) (a)(20) (b)(9)
(e)(2)(b)(2)
Quality Management System** B Privacy and Security C*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9)**Required for every EHR product
(e)(1)(a)(6) (a)(15) (b)(3)
(e)(1)
(e)(3)(a)(9)
(a)(7) (a)(16) (b)(4)
(a)(3) (a)(12) (a)(19) (b)(8)(a)(4) (a)(13) (a)(20) (b)(9)
(a)(7) (a)(16) (b)(4)
Automated Numerator Recording Successfully Tested(a)(1) (b)(6)
(a)(5) (a)(14)
Safety-Enhanced Design* A
(a)(5) (a)(14) (b)(2)
Attestation Forms (as applicable) Appendix
(a)(11) (a)(18)
(a)(17) (b)(5)
(a)(9) (a)(17) (b)(5)
(e)(2)(e)(3)
(a)(6) (a)(15) (b)(3)
Automated Numerator Recording Successfully Tested(a)(1) (a)(11) (a)(18) (b)(6)
Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 10
Appendix A: Safety Enhanced Design
EHR USABILITY TEST REPORT
OF
APRIMA 2014
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
Aprima 2014
Date of Usability Test: 05/21/13-05/23/13
Date of Report: 05/27/13
Report Prepared By: Aprima Medical Software Project Management
Cornell Hardman
(214) 466-8107
Aprima Medical Software
3330 Keller Springs Road
Suite 201
Carrollton, TX 75006
Table of Contents
EXECUTIVE SUMMARY ................................................................................................................. 3
Major findings ............................................................................................................................... 5
Areas for improvement ................................................................................................................. 5
INTRODUCTION ............................................................................................................................. 7
METHOD ......................................................................................................................................... 7
PARTICIPANTS ........................................................................................................................... 7
STUDY DESIGN .......................................................................................................................... 9
TASKS ....................................................................................................................................... 10
PROCEDURES .......................................................................................................................... 10
TEST LOCATION ...................................................................................................................... 11
TEST ENVIRONMENT .............................................................................................................. 12
TEST FORMS AND TOOLS ...................................................................................................... 12
PARTICIPANT INSTRUCTIONS ............................................................................................... 13
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 2 of 48
USABILITY METRICS ............................................................................................................... 14
DATA SCORING ........................................................................................................................ 14
RESULTS ...................................................................................................................................... 17
DATA ANALYSIS AND REPORTING ........................................................................................ 17
DISCUSSION OF THE FINDINGS ............................................................................................ 18
EFFECTIVENESS ..................................................................................................................... 18
EFFICIENCY .............................................................................................................................. 19
SATISFACTION ......................................................................................................................... 19
MAJOR FINDINGS .................................................................................................................... 20
AREAS FOR IMPROVEMENT .................................................................................................. 20
APPENDICES ................................................................................................................................ 22
Appendix 1: SAMPLE RECRUITING SCREENER .................................................................... 23
Appendix 2: PARTICIPANT DEMOGRAPHICS ........................................................................ 26
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ........... 27
Appendix 4: EXAMPLE MODERATOR'S GUIDE ...................................................................... 29
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................. 46
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM ................................. 47
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 3 of 48
EXECUTIVE SUMMARY
A usability test of Aprima 2014 ambulatory EHR was conducted on the dates of
May 21-23 of 2013 in The Aprima Medical Software corporate office by Aprima
Medical Software Project Management team. The purpose of this test was to
test and validate the usability of the current user interface, and provide
evidence of usability in the EHR Under Test (EHRUT). During the usability
test, 2 healthcare providers and 4 other knowledgeable clinic staff matching the
target demographic criteria served as participants and used the EHRUT in
simulated, but representative tasks. This study collected performance data on
13 tasks including 6 tasks specifically designed to test clinical decision support
functionality that are typically conducted on an EHR:
Task 1 – Create an Order
Task 2 – Utilize and existing Health Maintenance Rule
Task 3 – Creation of a Procedure Task Message
Task 4 – Approve a Result
Task 5 – Drug-Drug Interaction
Task 6 – Verify import in Allergy list and Medication List
During the 2-hour one-on-one usability test, each participant was greeted by
the administrator and asked to review and sign an informed consent/release
form (included in Appendix 3); they were instructed that they could withdraw at
any time. Participants had a mix of experience with the EHR. The
administrator introduced the test, and instructed participants to complete a
series of tasks (given one at a time) using the EHRUT. During the testing, the
administrator timed the test and, along with the data logger(s) recorded user
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 4 of 48
performance data on paper and electronically. The administrator did not give
the participant assistance in how to complete the task.
Participant screens were recorded for subsequent analysis.
The following types of data were collected for each participant:
Number of tasks successfully completed within the allotted time without
assistance
Time to complete the tasks
Number and types of errors
Path deviations
Participant's verbalizations
Participant's satisfaction ratings of the system
All participant data was de-identified - no correspondence could be made from
the identity of the participant to the data collected. Following the conclusion of
the testing, participants were asked to complete a post-test questionnaire and
were compensated with $300 for their time. Various recommended metrics, in
accordance with the examples set forth in the NIST Guide to the Processes
Approach for Improving the Usability of Electronic Health Records, were used
to evaluate the usability of the EHRUT. Following is a summary of the
performance and rating data collected on the EHRUT.
Task
Measure N Task
Success
Path
Deviation
Task Time Errors Task
Rating
5=Easy
# Mean
(SD)
Deviations
(Observed
/ Optimal)
Mean
(SD)
Min/Max
Optimal
Mean
(SD)
Mean
(SD)
1. Task 1 6 100%
(0.00)
2.8 (59/42) 152
(58)
1:04/3:44
0:38
0.3 (0.7) 3.5 (1.1)
2. Task 2 6 83%
(0.37)
1.7 (34/24) 42
(32)
0:10/1:42
0:11
0.3 (0.7) 4.0 (1.0)
3. Task 3 6 83%
(0.37)
2.0 (60/48) 118
(40)
1:03/2:54
0:36
1.0 (0.6) 3.8 (0.7)
4. Task 4 6 100%
(0.00)
1.7 (58/48) 78
(41)
0:28/2:36
0:28
0.5 (0.5) 4.5 (0.5)
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 5 of 48
5. Task 5 6 100%
(0.00)
0.3 (62/60) 46
(14)
0:28/1:17
0:30
0.0 (0.0) 4.7 (0.5)
6. Task 6 6 100%
(0.00)
0.3 (38/36) 33
(13)
0:22/1:00
0:22
0.2 (0.4) 4.3 (0.5)
The results from the System Usability Scale scored the subjective satisfaction
with the system based on performance with these tasks to be: 83
In addition to the performance data, the following qualitative observations were
made:
Major findings
The link to sources in lead to a screen with 2 buttons. This is the way
the vendor implemented the function. Participants would like to go
directly to the source info rather than having an extra click.
Task 2 was an alternate way for a clinical user to get CDS information.
It is not a workflow that is familiar to the clinical staff and while it does
give the results needed, it is not a common enough workflow to use in
the future.
Customization of the product and unfamiliarity with the interface
slowed many of the users down. The majority of the path deviations
and errors we primarily due to the limited exposure of the participants.
Repetition on subsequent tasks showed rapid improvement (Path
deviations decreased by 1.1 on second attempt and decreased by 2.5
by the fourth attempt)
Creation of the procedure task message starts with the participant
needing to find the new button. This caused users to wander around
the screen trying to find the button. It is not where users expect it to be
and caused all users to take longer than expected for the task.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 6 of 48
Subsequent repetition showed immediate improvement to within
expected limits.
Approval of results caused some user confusion. There are many ways
to approve the results, but the UI does not specifically lead users to a
specific path. This caused wide variation in user completion times.
Areas for improvement
The vendor should remove the extra click to show source information
for CDS.
The process of approving a message needs to be looked at to help
lead unfamiliar users to the correct actions more quickly. The users
were able to successfully complete the task, but users found it difficult
and it impacted confidence.
The New button for creating all workflow messages is difficult for users
to discover unaided, and should be evaluated to improve ease of use.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 7 of 48
INTRODUCTION
The EHRUT tested for this study was Aprima 2014 ambulatory EHR. Designed
to present medical information to healthcare providers in ambulatory medical
offices, the EHRUT consists of a user interface presented on a computer
display that stores and retrieves data from a database server through an
application server. The usability testing attempted to represent realistic
exercises and conditions. The purpose of this study was to test and validate
the usability of the current user interface, and provide evidence of usability in
the EHR Under Test (EHRUT). To this end, measures of effectiveness,
efficiency and user satisfaction, such as ease of access, number of mouse
clicks, time on task, time to repeat task, time to first entry and number of errors,
were captured during the usability testing.
METHOD
PARTICIPANTS
A total of 6 participants were tested on the EHRUT. Participants in the test
were physicians, nurses and other knowledgeable office staff. Participants
were recruited by Health Care Strategies and were compensated $300 for their
time. In addition, participants had no direct connection to the development of
or organization producing the EHRUT(s). Participants were not from the
testing or supplier organization. Participants were given the opportunity to
have an abbreviated orientation and a minimal level of training compared to the
amount the actual end users would have received. For the test purposes, end-
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 8 of 48
user characteristics were identified and translated into a recruitment screener
used to solicit potential participants; an example of a screener is provided in
Appendix 1.
Recruited participants had a mix of backgrounds and demographic
characteristics conforming to the recruitment screener. The following is a table
of participants by characteristics, including demographics, professional
experience, computing experience and user needs for assistive technology.
Participant names were replaced with Participant IDs so that an individual's
data cannot be tied back to individual identities.
Part
ID
Gender Age Education Occupation
/Role
Professional
Experience
Computer
Exp.
Product
Exp.
Assistive
Technology
Needs
1 142101 F 25 College Degree
Medical
Assistant 5 9 0/4
2 142102 F 42 College Degree
Medical
Assistant 3 13 0/1
3 142201 F 65 MD/PhD/DO Physician 38 4 0/0
4 142202 F 39 College Degree
Medical
Assistant 16 16 0/3
5 142301 F 31
Masters
Degree
Physician
Assistant 9 19 0/6
6 142302 F 30 College Degree
Medical
Assistant 8 13 0/7
9 participants (matching the demographics in the section on Participants) were
recruited and 6 participated in the usability test. 3 participants failed to show
for the study.
Participants were scheduled for 2 hour session with at least 1 hour in between
each session for debrief by the administrator(s) and data logger(s), and to reset
systems to proper test conditions. A spreadsheet was used to keep track of
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 9 of 48
the participant schedule, and included each participant's demographic
characteristics as provided by the recruiting firm.
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application
performed well - that is, effectively, efficiently, and with satisfaction - and areas
where the application failed to meet the needs of the participants. The data
from this test may serve as a baseline for future tests with an updated version
of the same EHR and/or comparison with other EHRs provided the same tasks
are used. In short, this testing serves as both a means to record or benchmark
current usability, but also to identify areas where improvements must be made.
During the usability test, participants interacted with a single EHR. Each
participant used the system in the same location, and was provided with the
same instructions. The system was evaluated for effectiveness, efficiency and
satisfaction as defined by measures collected and analyzed for each
participant:
Number of tasks successfully completed within the allotted time without
assistance
Time to complete the tasks
Number and types of errors
Path deviations
Participant's verbalizations (comments)
Participant's satisfaction ratings of the system
Additional information about the various measures can be found in
Section 3.9 on Usability Metrics.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 10 of 48
TASKS
A number of tasks were constructed that would be realistic and representative
of the kinds of activities a user might do with this EHR, including:
Task 1 – Create an Order
Task 2 – Utilize and existing Health Maintenance Rule
Task 3 – Creation of a Procedure Task Message
Task 4 – Approve a Result
Task 5 – Drug-Drug Interaction
Task 6 – Verify import in Allergy list and Medication List
Tasks were selected based on their frequency of use, criticality of function, and
those that may be most troublesome for users. Tasks should always be
constructed in light of the study objectives.
PROCEDURES
Upon arrival, participants were greeted; their identity was verified and matched
with a name on the participant schedule. Participants were then assigned a
participant ID. Each participant reviewed and signed an informed consent and
release form (See Appendix 3). A representative from the test team witnessed
the participant's signature. To ensure that the test ran smoothly, two staff
members participated in this test, the usability administrator and the data
logger. The usability testing staff conducting the test was experienced usability
practitioners with 3 years of usability experience at GE Healthcare and
additional experience at other facilities. The administrator moderated the
session including administering instructions and tasks. The administrator also
monitored task times, obtained post-task rating data, and ensured notes on
participant comments were recorded. A second person served as the data
logger and took notes on task success, path deviations, number and type of
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 11 of 48
errors, and comments. Participants were instructed to perform the tasks (see
specific instructions below):
As quickly as possible making as few errors and deviations as
possible.
Without assistance; administrators were allowed to give immaterial
guidance and clarification on tasks, but not instructions on use.
Without using a think aloud technique.
For each task, the participants were given a written copy of the task. Task
timing began once the administrator finished reading the question. The task
time was stopped once the participant indicated they had successfully
completed the task. Scoring is discussed below in Section 3.9. Following the
session, the administrator gave the participant the post-test questionnaire (e.g.,
the System Usability Scale, see Appendix 5), compensated him or her for their
time, and thanked each individual for their participation.
Participants' demographic information, task success rate, time on task, errors,
deviations, verbal responses, and post-test questionnaire were recorded into a
spreadsheet.
Participants were thanked for their time and compensated. Participants signed
a receipt and acknowledgement form (See Appendix 6) indicating that they had
received the compensation.
TEST LOCATION
The test facility included a waiting area and a quiet testing room with a table,
computer for the participant, and recording computer for the administrator.
Only the participant and administrator and the data logger were in the test
room. There were no observers. To ensure that the environment was
comfortable for users, noise levels were kept to a minimum with the ambient
temperature within a normal range. All of the safety instruction and evacuation
procedures were valid, in place, and visible to the participants.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 12 of 48
TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or facility. In this
instance, the testing was conducted in the Aprima Medical Software corporate
headquarters in the main training room. For testing, the computer used a
Fujitsu T902 running Microsoft Windows 8. The participants used whatever
input method he or she was most comfortable with from among mouse,
keyboard, touchpad, stylus or voice input when interacting with the EHRUT.
The EHRUT used a 12.7-inch diagonal display running 1280x720 pixels at 32
bit color. The application was set up by the administrator according to the
vendor's documentation. The application itself was running on a Dell 620
server running Windows Server 2008 using a test database on a LAN
connection. Technically, the system performance (i.e., response time) was
representative to what actual users would experience in a field implementation.
Additionally, participants were instructed not to change any of the default
system settings (such as control of font size).
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used,
including:
1. Informed Consent
2. Moderator's Guide
3. Post-test Questionnaire
4. Incentive Receipt and Acknowledgment Form
Examples of these documents can be found in Appendices 3-6 respectively.
The Moderator's Guide was devised so as to be able to capture required data.
The participant's interaction with the EHRUT was captured and recorded
digitally with screen capture software running on the test machine.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 13 of 48
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each participant:
Thank you for participating in this study. Your input is very important.
Our session today will last about 30 minutes. During that time you will
use an instance of an electronic health record. I will ask you to
complete a few tasks using this system and answer some questions.
You should complete the tasks as quickly as possible making as few
errors as possible. Please try to complete the tasks on your own
following the instructions very closely. Please note that we are not
testing you we are testing the system, therefore if you have difficulty all
this means is that something needs to be improved in the system. I will
be here in case you need specific help, but I am not able to instruct you
or provide help in how to use the application. Overall, we are interested
in how easy (or how difficult) this system is to use, what in it would be
useful to you, and how we could improve it. I did not have any
involvement in its creation, so please be honest with your opinions.
There are a variety of tools that record screens and transmit those
recordings across a local area network for remote observations. All of
the information that you provide will be kept confidential and your name
will not be associated with your comments at any time. Should you feel
it necessary you are able to withdraw at any time during the testing.
Following the procedural instructions, participants were shown the EHR and as
their first task, were given time (10 minutes) to explore the system and make
comments. Once this task was complete, the administrator gave the following
instructions:
For each task, I will read the description to you and say "Begin." At that
point, please perform the task and say "Done" once you believe you
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 14 of 48
have successfully completed the task. I would like to request that you
not talk aloud or verbalize while you are doing the tasks. I will ask you
your impressions about the task once you are done.
Participants were then given 13 tasks to complete. Tasks are listed in the
moderator's guide in the Appendix.
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the
Usability of Electronic Health Records, EHRs should support a process that
provides a high level of usability for all users. The goal is for users to interact
with the system effectively, efficiently, and with an acceptable level of
satisfaction. To this end, metrics for effectiveness, efficiency and user
satisfaction were captured during the usability testing. The goals of the test
were to assess:
1. Effectiveness of EHRUT by measuring participant success rates and errors
2. Efficiency of EHRUT by measuring the average task time and path
deviations
3. Satisfaction with EHRUT by measuring ease of use ratings
DATA SCORING
The following table (Table 1) details how tasks were scored, errors evaluated,
and the time data analyzed.
Measures Rational and Scoring
Effectiveness:
Task Success
A task was counted as a "Success" if the participant was able to
achieve the correct outcome, without assistance, within the time
allotted on a per task basis.
The total number of successes were calculated for each task and
then divided by the total number of times that task was attempted.
The results are provided as a percentage.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 15 of 48
Task times were recorded for successes. Observed task times
divided by the optimal time for each task is a measure of optimal
efficiency.
Optimal task performance time, as benchmarked by expert
performance under realistic conditions, is recorded when
constructing tasks. Target task times used for task times in the
Moderator's Guide must be operationally defined by taking multiple
measures of optimal performance and multiplying by some factor
that allows some time buffer because the participants are
presumably not trained to expert performance. Scoring used a
factor of 2. Thus, if expert, optimal performance on a task was 80
seconds then allotted task time performance was 80 * 2 = 160
seconds. This ratio should be aggregated across tasks and
reported with mean and variance scores.
Effectiveness:
Task Failures
If the participant abandoned the task, did not reach the correct
answer or performed it incorrectly, or reached the end of the
allotted time before successful completion, the task was counted
as a "Failure." No task times were taken for errors.
The total number of errors was calculated for each task and then
divided by the total number of times that task was attempted. Not
all deviations would be counted as errors. This should also be
expressed as the mean number of failed tasks per participant.
On a qualitative level, an enumeration of errors and error types
were collected.
Efficiency:
Task Deviations
The participant's path (i.e., steps) through the application was
recorded. Deviations occur if the participant, for example, went to
a wrong screen, clicked on an incorrect menu item, followed an
incorrect link, or interacted incorrectly with an on-screen control.
This path was compared to the optimal path. The number of steps
in the observed path is divided by the number of optimal steps in
order to provide a ratio of path deviation.
It is strongly recommended that task deviations be reported.
Optimal paths (i.e., procedural steps) should be recorded when
constructing tasks.
Efficiency:
Task Time
Each task was timed from when the administrator said "Begin" until
the participant said, "Done." If he or she failed to say "Done," the
time was stopped when the participant stopped performing the
task. Only task times for tasks that were successfully completed
were included in the average task time analysis. Average time per
task was calculated for each task. Variance measures (standard
deviation and standard error) were also calculated.
Satisfaction: Participant's subjective impression of the ease of use of the
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 16 of 48
Task Rating application was measured by administering both a simple post-
task question as well as a post-session questionnaire. After each
task, the participant was asked to rate "Overall, this task was:" on
a scale of 1 (Very Difficult) to 5 (Very Easy). These data are
averaged across participants.
Common convention is that average ratings for systems judged
easy to use should be 3.3 or above.
To measure participants' confidence in and likeability of the
EHRUT overall, the testing team administered the System
Usability Scale (SUS) post-test questionnaire. Questions included,
"I think I would like to use this system frequently," "I thought the
system was easy to use," and "I would imagine that most people
would learn to use this system very quickly." See full System
Usability Score questionnaire in Appendix 5.
Table 1. Details of how observed data were scored.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 17 of 48
RESULTS
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods
specified in the Usability Metrics section above. Participants who failed to
follow session and task instructions had their data excluded from the analyses.
In this test cycle, there were no data excluded.
The usability testing results for the EHRUT are detailed below. The results
should be seen in light of the objectives and goals outlined in Section 3.2 Study
Design. The data should yield actionable results that, if corrected, yield
material, positive impact on user performance.
Task
Measure N Task
Success
Path
Deviation
Task Time Errors Task
Rating
5=Easy
# Mean
(SD)
Deviations
(Observed
/ Optimal)
Mean
(SD)
Min/Max
Optimal
Mean
(SD)
Mean
(SD)
1. Task 1 6 100%
(0.00)
2.8 (59/42) 152
(58)
1:04/3:44
0:38
0.3 (0.7) 3.5 (1.1)
2. Task 2 6 83%
(0.37)
1.7 (34/24) 42
(32)
0:10/1:42
0:11
0.3 (0.7) 4.0 (1.0)
3. Task 3 6 83%
(0.37)
2.0 (60/48) 118
(40)
1:03/2:54
0:36
1.0 (0.6) 3.8 (0.7)
4. Task 4 6 100%
(0.00)
1.7 (58/48) 78
(41)
0:28/2:36
0:28
0.5 (0.5) 4.5 (0.5)
5. Task 5 6 100%
(0.00)
0.3 (62/60) 46
(14)
0:28/1:17
0:30
0.0 (0.0) 4.7 (0.5)
6. Task 6 6 100%
(0.00)
0.3 (38/36) 33
(13)
0:22/1:00
0:22
0.2 (0.4) 4.3 (0.5)
The results from the SUS (System Usability Scale) scored the subjective
satisfaction with the system based on performance with these tasks to be: 83.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 18 of 48
Broadly interpreted, scores under 60 represent systems with poor usability;
scores over 80 would be considered above average
DISCUSSION OF THE FINDINGS
Customization of the product and unfamiliarity with the interface slowed many
of the users down. The majority of the path deviations and errors we primarily
due to the limited exposure of the participants. Repetition on subsequent tasks
showed rapid improvement (Path deviations decreased by 1.1 on second
attempt and decreased by 2.5 by the fourth attempt)
Creation of the procedure task message starts with the participant needing to
find the new button. This caused users to wander around the screen trying to
find the button. It is not where users expect it to be and caused all users to
take longer than expected for the task. Subsequent repetition showed
immediate improvement to within expected limits.
Approval of results caused some user confusion. There are many ways to
approve the results, but the UI does not specifically lead users to a specific
path. This caused wide variation in user completion times.
EFFECTIVENESS
The users were able to do what was expected with the product. Many times the
users went slightly off the optimal the path but did not lose their way. Users
with a small amount of time on the system improved dramatically. All users
rated the product either agree or strongly agree with regard to “I thought the
system was easy to use”. All users also rated “I found the system
unnecessarily complex” either disagree or strongly disagree.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 19 of 48
EFFICIENCY
The number of path deviations on some tasks was above 2.0 this is large
number. For most tasks the number was at or below 1.0. Users were in almost
all cases able to quickly come back to the optimal path and complete the task.
This is true for the tasks in which the path deviations where the highest as well.
Variance on the timings show that the untrained users are not efficient using
Aprima. User learning is rapid; by task 5 the users were able to be more
efficient with task times being one half or less compared to the first 4 tasks.
SATISFACTION
User satisfaction was quite high. Overall user rating was 83.
Question Agree or
Strongly Agree
Disagree or
Strongly Disagree
I think that I would like to use
this system frequently 83% 0%
I found the system unnecessarily
complex 0% 100%
I thought the system was easy to
use 100% 0%
I think that I would need the
support of a technical person to
be able to use this system
17% 67%
I found the various functions in
this system were well integrated 83% 0%
I thought there was too much
inconsistency in this system 0% 100%
I would imagine that most people
would learn to use this system
very quickly
100% 0%
I found the system very
cumbersome to use 33% 67%
I felt very confident using the
system 100% 0%
I needed to learn a lot of things
before I could get going with this
system
0% 100%
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 20 of 48
“I found the system cumbersome to use” seems to be a polarizing question. In
future user testing this will have to be explored more, as we did not find this to
be reflected in the user comments and was not discovered until the data
analysis phase.
MAJOR FINDINGS
The link to sources in lead to a screen with 2 buttons. This is the way
the vendor implemented the function. Participants would like to go
directly to the source info rather than having an extra click.
Task 2 was an alternate way for a clinical user to get CDS information.
It is not a workflow that is familiar to the clinical staff and while it does
give the results needed, it is not a common enough workflow to use in
the future.
Customization of the product and unfamiliarity with the interface
slowed many of the users down. The majority of the path deviations
and errors we primarily due to the limited exposure of the participants.
Repetition on subsequent tasks showed rapid improvement (Path
deviations decreased by 1.1 on second attempt and decreased by 2.5
by the fourth attempt)
Creation of the procedure task message starts with the participant
needing to find the new button. This caused users to wander around
the screen trying to find the button. It is not where users expect it to be
and caused all users to take longer than expected for the task.
Subsequent repetition showed immediate improvement to within
expected limits.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 21 of 48
Approval of results caused some user confusion. There are many ways
to approve the results, but the UI does not specifically lead users to a
specific path. This caused wide variation in user completion times.
AREAS FOR IMPROVEMENT
The vendor should remove the extra click to show source information
for CDS.
The process of approving a message needs to be looked at to help
lead unfamiliar users to the correct actions more quickly. The users
were able to successfully complete the task, but users found it difficult
and it impacted confidence.
The New button for creating all workflow messages is difficult for users
to discover unaided, and should be evaluated to improve ease of use.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 22 of 48
APPENDICES
The following appendices include supplemental data for this usability test
report. Following is a list of the appendices provided:
1: Sample Recruiting screener
2: Participant demographics
3: Non-Disclosure Agreement (NDA) and Informed Consent Form
4: Example Moderator's Guide
5: System Usability Scale Questionnaire
6: Incentive receipt and acknowledgment form It is important to note that these
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 23 of 48
Appendix 1: SAMPLE RECRUITING SCREENER
The purpose of a screener to ensure that the participants selected represent
the target user population as closely as possible. (Portions of this sample
screener are taken from www.usability.gov/templates/index.html#Usability and
adapted for use.)
Recruiting Script for Recruiting Firm
Hello, my name is _______________, calling from [Insert name of recruiting
firm]. We are recruiting individuals to participate in a usability study for an
electronic health record. We would like to ask you a few questions to see if
you qualify and if would like to participate. This should only take a few minutes
of your time. This is strictly for research purposes. If you are interested and
qualify for the study, you will be paid to participate. Can I ask you a few
questions?
Customize this by dropping or adding questions so that it reflects your EHR's
primary audience
1. [If not obvious] Are you male or female? [Recruit a mix of participants]
2. Have you participated in a focus group or usability test in the past xx
months? [If yes, Terminate]
3. Do you, or does anyone in your home, work in marketing research, usability
research, web design [If yes, Terminate]
4. Do you, or does anyone in your home, have a commercial or research
interest in an electronic health record software or consulting company? [If yes,
Terminate]
5. Which of the following best describes your age? [23 to 39; 40 to 59; 60 and
older] [Recruit Mix]
6. Which of the following best describes your race or ethnic group? [e.g.,
Caucasian, Asian, Black/African-American, Latino/a or Hispanic, etc.]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 24 of 48
7. Do you require any assistive technologies to use a computer? [if so, please
describe] Professional Demographics Customize this to reflect your EHR's
primary audience
8. What is your current position and title? (Must be healthcare provider)
□ RN: Specialty _______________________
□ Physician: Specialty _______________________
□ Resident: Specialty _______________________
□ Medical Assistant _______________________
□ Administrative Staff
□ Other [Terminate]
9. How long have you held this position?
10. Describe your work location (or affiliation) and environment? (Recruit
according to the intended users of the application. e.g., private practice, health
system, government clinic, etc.)
11. Which of the following describes your highest level of education? [e.g.,
high school graduate/GED, some college, college graduate (RN, BSN),
postgraduate (MD/PhD), other (explain)
12. Besides reading email, what professional activities do you do on the
computer? [e.g., access EHR, research; reading news; shopping/banking;
digital pictures; programming/word processing, etc.] [If no computer use at all,
Terminate]
13. About how many hours per week do you spend on the computer? [Recruit
according to the demographics of the intended users, e.g., 0 to 10, 11 to 25,
26+ hours per week]
14. What computer platform do you usually use? [e.g., Mac, Windows, etc.]
15. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]
16. In the last month, how often have you used an electronic health record?
17. How many years have you used an electronic health record?
18. How many EHRs do you use or are you familiar with?
19. How does your work environment handle patient records? [Recruit
according to the demographics of the intended users]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 25 of 48
□ On paper
□ Some paper, some electronic
□ All electronic
Contact Information If the person matches your qualifications, ask
Those are all the questions I have for you. Your background matches the
people we're looking for. For your participation, you will be paid $300. Would
you be able to participate on [date, time]? [If so collect contact information]
May I get your contact information?
■ Name of participant:
■ Address:
■ City, State, Zip:
■ Daytime phone number:
■ Evening phone number:
■ Alternate [cell] phone number:
■ Email address:
Before your session starts, we will ask you to sign a release form allowing us to
record your session. The record will only be used internally for further study if
needed. Will you consent to be digitally recorded? This study will take place at
3330 Keller Springs Road, Suite 201, Carrollton, TX 75006. I will confirm your
appointment a couple of days before your session and provide you with
directions to our office. What time is the best time to reach you?
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 26 of 48
Appendix 2: PARTICIPANT DEMOGRAPHICS
Following is a high-level overview of the participants in this study.
Gender
Men: 0
Women: 6
Total (participants): 6
Occupation/Role
RN/BSN: 0
Physician: 2
Admin Staff: 4
Total (participants): 6
Years of Experience
Facility Use of EHR
All paper: 1
Some paper, some electronic: 5
All electronic: 0
Total (participants): 6
The full break down is included in the report above.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 27 of 48
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM
Non-Disclosure Agreement
THIS AGREEMENT is entered into as of _________________, 2013, between
________________ ("the Participant") and the testing organization Aprima Medical Software
located at 3330 Keller Springs Rd. Suite 201, Carrollton, TX 75006.
The Participant acknowledges his or her voluntary participation in today's usability study may
bring the Participant into possession of Confidential Information. The term "Confidential
Information" means all technical and commercial information of a proprietary or confidential
nature which is disclosed by Test Company, or otherwise acquired by the Participant, in the
course of today's study.
By way of illustration, but not limitation, Confidential Information includes trade secrets,
processes, formulae, data, know-how, products, designs, drawings, computer aided design files
and other computer files, computer software, ideas, improvements, inventions, training methods
and materials, marketing techniques, plans, strategies, budgets, financial information, or
forecasts.
Any information the Participant acquires relating to this product during this study is confidential
and proprietary to Test Company and is being disclosed solely for the purposes of the
Participant's participation in today's usability study. By signing this form the Participant
acknowledges that s/he will receive monetary compensation for feedback and will not disclose
this confidential information obtained today to anyone else or any other organizations.
Participant's printed name: ___________________________________________________
Signature: __________________________________________________________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 28 of 48
Informed Consent
Test Company would like to thank you for participating in this study. The purpose of this study is
to evaluate an electronic health records system. If you decide to participate, you will be asked to
perform several tasks using the prototype and give your feedback. The study will last about 30
minutes. At the conclusion of the test, you will be compensated for your time.
Agreement
I understand and agree that as a voluntary participant in the present study conducted by Test
Company I am free to withdraw consent or discontinue participation at any time. I understand
and agree to participate in the study conducted and videotaped by the Test Company.
I understand and consent to the use and release of the videotape by Test Company. I
understand that the information and videotape is for research purposes only and that my name
and image will not be used for any purpose other than research. I relinquish any rights to the
videotape and understand the videotape may be copied and used by Test Company without
further permission.
I understand and agree that the purpose of this study is to make software applications more
useful and usable in the future.
I understand and agree that the data collected from this study may be shared with outside of Test
Company and Test Company's client. I understand and agree that data confidentiality is assured,
because only de- identified data - i.e., identification numbers not names - will be used in analysis
and reporting of the results.
I agree to immediately raise any concerns or areas of discomfort with the study administrator. I
understand that I can leave at any time.
Please check one of the following:
□ YES, I have read the above statement and agree to be a participant.
□ NO, I choose not to participate in this study.
Signature: _________________________________ Date: _______________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 29 of 48
Appendix 4: EXAMPLE MODERATOR'S GUIDE
Only four tasks are presented here for illustration.
EHRUT Usability Test
Moderator's Guide
Administrator _____________________
Data Logger _____________________
Date _________________ Time _____________________
Participant # _________
Location _____________________
Prior to testing
■ Confirm schedule with Participants
■ Ensure EHRUT lab environment is running properly
■ Ensure Consent forms are complete for all participants
■ Ensure lab and data recording equipment is running properly
Prior to each participant:
■ Reset application
■ Restore application to base state using script restoredb.bat
■ Start session recordings with GTM Recording
Prior to each task:
■ Reset application to starting point for next task
After each participant:
■ End session recordings with tool
■ Fill out all paper work and acknowledgement form.
After all testing
■ Scan all forms and shred originals
■ Back up all video and data files
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 30 of 48
Orientation (5 minutes)
Thank you for participating in this study. Our session today will last 2 hours today. During that
time you will take a look at an electronic health record system.
I will ask you to complete a few tasks using this system and answer some questions. We are
interested in how easy (or how difficult) this system is to use, what in it would be useful to you,
and how we could improve it. You will be asked to complete these tasks on your own trying to do
them as quickly as possible with the fewest possible errors or deviations. Do not do anything
more than asked. If you get lost or have difficulty I cannot answer help you with anything to do
with the system itself. Please save your detailed comments until the end of a task or the end of
the session as a whole when we can discuss freely.
I did not have any involvement in its creation, so please be honest with your opinions.
The product you will be using today is describe the state of the application, i.e., production
version, early prototype, etc. Some of the data may not make sense as it is placeholder data.
We are recording the audio and screenshots of our session today. All of the information that you
provide will be kept confidential and your name will not be associated with your comments at any
time.
Do you have any questions or concerns?
Preliminary Questions (5 minutes)
What is your job title / appointment?
How long have you been working in this role?
What are some of your main responsibilities?
Tell me about your experience with electronic health records.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 31 of 48
Preview: First Impressions (6 Minutes)
This is the application you will be working with. Have you heard of it? Yes ___ No ___
If so, tell me what you know about it.
Show test participant the EHRUT.
Please don't click on anything just yet.
What do you notice?
What are you able to do here?
Please be specific.
Notes / Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 32 of 48
Task 1: Create an Order – Ed Wood (5 Minutes)
Take the participant to the starting point for the task. - FNC
The patient you are examining 5’ 10” 285 lbs. male needs a differential diagnosis of 034.0
Streptococcal Sore Throat and needs a 87880 rapid strep to confirm diagnosis. Enter Vitals, Dx
and SP into the system
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, enter 5 feet 10 inches and 285 lbs 2. go to the DX tab and Enter the diagnosis 034.0 Streptococcal Sore Throat 3. Note Alert for Obesity and Hypertension in CDS slider 4. Still inside FNC, go to the SP tab 5. Enter procedure code 87880 Strep A Assay 6. Still inside FNC, click the Close button to close the note, save the note as incomplete 7. Click okay.
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 33 of 48
Task 2: Utilize an existing Clinical Decision Support Alert – Norman Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
You run are in consultation with a patient and his parent who is 71 years old. You want to quickly
see if there are any Clinical Support Alerts for the parent. Go to the Appointment to review CDS
Alerts and look at the source information for the alert.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
Open Patient Norman Wood
Open Update Appointment from Envelope
Open Clinical Decision Support Tab
Note Fall Risk Rule on Slider
Click on the Hyperlink for Fall Risk
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 34 of 48
Task 3: Creation of a Procedure Task Message to the Nurses - Ed Wood (5 Minutes)
Take the participant to the starting point for the task. - FNC
A patient you saw earlier that requested an order for a test at a different lab wants the lab found it was not covered by his insurance. He wants it to be performed in your office. You need to order a rapid strep test (87880) for the patient, as he is on his way back into the office.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open the scheduler 2. Find the appointment for Ed Wood 3. Click the Blue Folder (Open/Create note) 4. Click New->Procedure Task Message
a. Enter the Nurses b. Click Okay
5. Click to close the note a. Click Okay
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 35 of 48
Task 4: Approve a Lab result (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
You have a lab result request message in your inbox to process. You need to review and approve the request.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
Approval of Lab Results
1. Locate the Lab Result Approval message in the Desktop Message pane 2. Click on the Time and Date Hyperlink or Plus (+) sign next to the message 3. Click on Process Message 4. From the Modify Message window 5. After review of the results, click the Approved All button
a. Your user (provider) name will auto populate along with the current date 6. Click Complete
a. This will simultaneously set the approved lab results and complete the message
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 36 of 48
Task 5: Document Drug Allergy in History – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
The patient you are examining reports to you that she has been taking no meds because she has
an anaphylactic reaction to Penicillins. Please document the allergy in History
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Drug Allergy
6. In the Allergen column, select Penicillins.
7. In the Reaction column, select anaphylaxis.
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 37 of 48
Task 6: Drug Allergy Alert in Rx Interaction – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
Need to prescribe Keflex to Mary Woods chart 1 tablet per day by mouth for 10 days
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Rx Tab
6. In the Drug field, enter Keflex
7. Click the find control
8. Select Keflex 500 mg
9. Pop up: Receive Drug Screening Warning: ALLERGY ALERT: The use of Keflex cross-sensitivity reaction based on a reported history of allergy to PENICILLINS. Do you want to continue? Click Yes
Comments:
Observed Errors and Verbalizations:
Comments:
Rating: Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 38 of 48
Task 7: Drug Drug Interaction – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
An agitated patient wants you to prescribe Prozac immediately. 20mg 1 time per day by mouth.
Please prescribe the medication.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Find Mary Wood that has an appointment scheduled.
2. Click the Open/Create Note icon
3. FNC
4. Click Rx Tab
5. In the Drug field, enter Prozac
6. Click the find control
7. Select Prozac 20mg tablet
8. Pop up: Receive Drug Screening Warning: Duplicate Therapy: The use of Prozac is a duplicate therapy with Zoloft 50 mg. Do you want to continue?
a. Click No
9. Print -> Save and print Button
a. Print Dialog
10. Click ok to save and exit the note
Comments:
Observed Errors and Verbalizations:
Comments:
Rating: Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 39 of 48
Task 8: Medication List in OPS – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
You are about to walk into an exam room for a med check on a patient. Go to One Page
Summary and review the patient’s drug list.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Go to Patient Demographics 2. Click Patient 3. Click One Page Summary 4. Click hyperlink in index for Current Medication 5. Verify the data is displayed correctly
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 40 of 48
Task 9: Medication List in Hx Tab with Filter – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
Previously very ill patient with a very large medication history is asking about her current medications. You want to compare the active/inactive medications with the maintenance medication list (which are two different filters in the system Go to Hx Tab and change the filter to show only Active Medications and them maintenance medications.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, go to the HX tab 2. Click on the Medication History 3. Review the items on the list 4. Click the search icon to change the filter 5. Change the filter to maintenance 6. Review the Maintenance Medications
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 41 of 48
Task 10: Rx Refill Complete (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
You have a few minutes to complete your drug refill request messages
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Messages
2. Locate message
3. Click on patient’s blue envelope.
4. Select Process Rx Refill
5. Medication History screen
6. Select the drug by clicking the Refill button.
7. Sig Writer screen
a. Verify information in sig
8. Click Ok
a. Return to the Modify Message screen
9. Click Complete.
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 42 of 48
Task 11: import Allergies from external list - Norman Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
The patient you are seeing has a C-CDA that needs to be checked against your record. Start with
reconciling the Allergy list.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. Go to attachments – Allergies - CCD 3. Click the import allergies icon 4. Sulfasalazine -> Add 5. Penicillin V -> Merge/Add 6. Carbamazepine -> ignore/remove 7. Click Reconcile 8. Go to Hx
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 43 of 48
Task 12: Import Medications from external list – Norman Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
The patient you are seeing has a C-CDA that needs to be checked against your record. Start with
reconciling the Medication list.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. View the medication CCD 3. Click the import medications icon 4. Simvastatin -> Add 5. Lorazepam -> Add 6. Lisinopril -> Add/Merge 7. Click reconcile 8. Go to patient Hx – Medications
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 44 of 48
Task 13: Verify import in Allergy list and Medication List – Norman Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
Please validate that the import of Allergy list and Medication list import has the expected
information
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed
Comments:
Task Time: _____________ Seconds
Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Note the CDS rule for the combination of the new allergy and medication
6. Click Drug Allergy
a. Verify the information is correct from the import
7. Click on the Medication History
a. Verify the information is correct from the import
Comments:
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5)
Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 45 of 48
Final Questions (10 Minutes)
What was your overall impression of this system?
What aspects of the system did you like most?
What aspects of the system did you like least?
Were there any features that you were surprised to see?
What features did you expect to encounter but did not see? That is, is there anything that is
missing in this application?
Compare this system to other systems you have used.
Would you recommend this system to your colleagues?
Administer the SUS
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 46 of 48
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
Strongly
disagree
Strongly
Agree
1. I think that I would like to use this system
frequently
1 2 3 4 5
2. I found the system unnecessarily complex
1 2 3 4 5
3. I thought the system was easy to use
1 2 3 4 5
4. I think that I would need the support of a
technical person to be able to use this system
1 2 3 4 5
5. I found the various functions in this system
were well integrated
1 2 3 4 5
6. I thought there was too much inconsistency in
this system
1 2 3 4 5
7. I would imagine that most people would learn to
use this system very quickly
1 2 3 4 5
8. I found the system very cumbersome to use
1 2 3 4 5
9. I felt very confident using the system
1 2 3 4 5
10. I needed to learn a lot of things before I could
get going with this system
1 2 3 4 5
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 47 of 48
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM
Acknowledgement of Receipt I hereby acknowledge receipt of $ _____ for my participation in a
research study run by Aprima Medical Software.
Printed Name: _____________________________________________________________
Address: __________________________________________________________________
__________________________________________________________________________
Signature: _______________________________________________ Date: ____________
Usability Researcher: ________________________________________________________
Signature of Usability Researcher: ____________________________ Date: ______________
Witness: ___________________________________________________________________
Witness Signature: ________________________________________ Date: ______________
EHR USABILITY TEST REPORT
OF
APRIMA 2014
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
Aprima 2014 Date of Usability Test: 05/21/13-05/23/13 Date of Report: 05/27/13 Report Prepared By: Aprima Medical Software Project Management Cornell Hardman (214) 466-8107 Aprima Medical Software 3330 Keller Springs Road Suite 201 Carrollton, TX 75006
Table of Contents EXECUTIVE SUMMARY ................................................................................................................. 3
Major findings ............................................................................................................................... 5
Areas for improvement ................................................................................................................. 5
INTRODUCTION ............................................................................................................................. 7
METHOD ......................................................................................................................................... 7
PARTICIPANTS ........................................................................................................................... 7
STUDY DESIGN .......................................................................................................................... 9
TASKS ....................................................................................................................................... 10
PROCEDURES .......................................................................................................................... 10
TEST LOCATION ...................................................................................................................... 11
TEST ENVIRONMENT .............................................................................................................. 11
TEST FORMS AND TOOLS ...................................................................................................... 12
PARTICIPANT INSTRUCTIONS ............................................................................................... 13
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 2 of 48
USABILITY METRICS ............................................................................................................... 14
DATA SCORING ........................................................................................................................ 14
RESULTS ...................................................................................................................................... 17
DATA ANALYSIS AND REPORTING ........................................................................................ 17
DISCUSSION OF THE FINDINGS ............................................................................................ 18
EFFECTIVENESS ..................................................................................................................... 18
EFFICIENCY .............................................................................................................................. 18
SATISFACTION ......................................................................................................................... 19
MAJOR FINDINGS .................................................................................................................... 19
AREAS FOR IMPROVEMENT .................................................................................................. 19
APPENDICES ................................................................................................................................ 22
Appendix 1: SAMPLE RECRUITING SCREENER .................................................................... 23
Appendix 2: PARTICIPANT DEMOGRAPHICS ........................................................................ 26
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ........... 27
Appendix 4: EXAMPLE MODERATOR'S GUIDE ...................................................................... 29
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................. 46
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM ................................. 47
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 3 of 48
EXECUTIVE SUMMARY
A usability test of Aprima 2014 ambulatory EHR was conducted on the dates of
May 21-23 of 2013 in The Aprima Medical Software corporate office by Aprima
Medical Software Project Management team. The purpose of this test was to
test and validate the usability of the current user interface, and provide
evidence of usability in the EHR Under Test (EHRUT). During the usability
test, 2 healthcare providers and 4 other knowledgeable clinic staff matching the
target demographic criteria served as participants and used the EHRUT in
simulated, but representative tasks. This study collected performance data on
13 tasks including 3 tasks specifically designed to test Computerized Physician
Order Entry (CPOE) that are typically conducted on an EHR:
Task 1 – Create an Order
Task 2 – Creation of a Procedure Task Message
Task 3 – Approve a Result
During the 2-hour one-on-one usability test, each participant was greeted by
the administrator and asked to review and sign an informed consent/release
form (included in Appendix 3); they were instructed that they could withdraw at
any time. Participants had a mix of experience with the EHR. The
administrator introduced the test, and instructed participants to complete a
series of tasks (given one at a time) using the EHRUT. During the testing, the
administrator timed the test and, along with the data logger(s) recorded user
performance data on paper and electronically. The administrator did not give
the participant assistance in how to complete the task.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 4 of 48
Participant screens were recorded for subsequent analysis.
The following types of data were collected for each participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations Participant's satisfaction ratings of the system
All participant data was de-identified - no correspondence could be made from
the identity of the participant to the data collected. Following the conclusion of
the testing, participants were asked to complete a post-test questionnaire and
were compensated with $300 for their time. Various recommended metrics, in
accordance with the examples set forth in the NIST Guide to the Processes
Approach for Improving the Usability of Electronic Health Records, were used
to evaluate the usability of the EHRUT. Following is a summary of the
performance and rating data collected on the EHRUT.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations (Observed / Optimal)
Mean(SD)
Min/Max Optimal
Mean (SD)
Mean (SD)
1. Task 1 6 100% (0.00)
2.8 (59/42) 152 (58)
1:04/3:44 0:38
0.3 (0.7) 3.5 (1.1)
2. Task 2 6 83% (0.37)
2.0 (60/48) 118 (40)
1:03/2:54 0:36
1.0 (0.6) 3.8 (0.7)
3. Task 3 6 100% (0.00)
1.7 (58/48) 78 (41)
0:28/2:36 0:28
0.5 (0.5) 4.5 (0.5)
The results from the System Usability Scale scored the subjective satisfaction
with the system based on performance with these tasks to be: 83
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 5 of 48
In addition to the performance data, the following qualitative observations were
made:
Major findings
The information presented and the way in which it was presented was
well received by the test users. Even when unfamiliar with the
interface participants were able to complete the task. Only one failure
on the three sets of tasks occurred across all users.
Customization of the product and unfamiliarity with the interface
slowed many of the users down. The majority of the path deviations
and errors we primarily due to the limited exposure of the participants.
Repetition on subsequent tasks showed rapid improvement (Path
deviations decreased by 1.1 on second attempt and decreased by 2.5
by the fourth attempt)
Creation of the procedure task message starts with the participant
needing to find the new button. This caused users to wander around
the screen trying to find the button. It is not where users expect it to be
and caused all users to take longer than expected for the task.
Subsequent repetition showed immediate improvement to within
expected limits.
Approval of results caused some user confusion. There are many ways
to approve the results, but the UI does not specifically lead users to a
specific path. This caused wide variation in user completion times.
Areas for improvement
The process of approving a message needs to be looked at to help
lead unfamiliar users to the correct actions more quickly. The users
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 6 of 48
were able to successfully complete the task, but users found it difficult
and it impacted confidence.
The New button for creating all workflow messages is difficult for users
to discover unaided, and should be evaluated to improve ease of use.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 7 of 48
INTRODUCTION
The EHRUT tested for this study was Aprima 2014 ambulatory EHR. Designed
to present medical information to healthcare providers in ambulatory medical
offices, the EHRUT consists of a user interface presented on a computer
display that stores and retrieves data from a database server through an
application server. The usability testing attempted to represent realistic
exercises and conditions. The purpose of this study was to test and validate
the usability of the current user interface, and provide evidence of usability in
the EHR Under Test (EHRUT). To this end, measures of effectiveness,
efficiency and user satisfaction, such as ease of access, number of mouse
clicks, time on task, time to repeat task, time to first entry and number of errors,
were captured during the usability testing.
METHOD
PARTICIPANTS
A total of 6 participants were tested on the EHRUT. Participants in the test
were physicians, nurses and other knowledgeable office staff. Participants
were recruited by Health Care Strategies and were compensated $300 for their
time. In addition, participants had no direct connection to the development of
or organization producing the EHRUT(s). Participants were not from the
testing or supplier organization. Participants were given the opportunity to
have an abbreviated orientation and a minimal level of training compared to the
amount the actual end users would have received. For the test purposes, end-
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 8 of 48
user characteristics were identified and translated into a recruitment screener
used to solicit potential participants; an example of a screener is provided in
Appendix 1.
Recruited participants had a mix of backgrounds and demographic
characteristics conforming to the recruitment screener. The following is a table
of participants by characteristics, including demographics, professional
experience, computing experience and user needs for assistive technology.
Participant names were replaced with Participant IDs so that an individual's
data cannot be tied back to individual identities.
Part ID
Gender Age Education Occupation /Role
Professional Experience
Computer Exp.
Product Exp.
Assistive Technology
Needs
1 142101 F 25 College Degree
Medical
Assistant 5 9 0/4
2 142102 F 42 College Degree
Medical
Assistant 3 13 0/1 3 142201 F 65 MD/PhD/DO Physician 38 4 0/0
4 142202 F 39 College Degree
Medical
Assistant 16 16 0/3
5 142301 F 31
Masters
Degree
Physician
Assistant 9 19 0/6
6 142302 F 30 College Degree
Medical
Assistant 8 13 0/7
9 participants (matching the demographics in the section on Participants) were
recruited and 6 participated in the usability test. 3 participants failed to show
for the study.
Participants were scheduled for 2 hour session with at least 1 hour in between
each session for debrief by the administrator(s) and data logger(s), and to reset
systems to proper test conditions. A spreadsheet was used to keep track of
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 9 of 48
the participant schedule, and included each participant's demographic
characteristics as provided by the recruiting firm.
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application
performed well - that is, effectively, efficiently, and with satisfaction - and areas
where the application failed to meet the needs of the participants. The data
from this test may serve as a baseline for future tests with an updated version
of the same EHR and/or comparison with other EHRs provided the same tasks
are used. In short, this testing serves as both a means to record or benchmark
current usability, but also to identify areas where improvements must be made.
During the usability test, participants interacted with a single EHR. Each
participant used the system in the same location, and was provided with the
same instructions. The system was evaluated for effectiveness, efficiency and
satisfaction as defined by measures collected and analyzed for each
participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations (comments) Participant's satisfaction ratings of the system
Additional information about the various measures can be found in
Section 3.9 on Usability Metrics.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 10 of 48
TASKS
A number of tasks were constructed that would be realistic and representative
of the kinds of activities a user might do with this EHR, including:
Task 1 – Create an Order
Task 2 – Creation of a Procedure Task Message
Task 3 – Approve a Result
Tasks were selected based on their frequency of use, criticality of function, and
those that may be most troublesome for users. Tasks should always be
constructed in light of the study objectives.
PROCEDURES
Upon arrival, participants were greeted; their identity was verified and matched
with a name on the participant schedule. Participants were then assigned a
participant ID. Each participant reviewed and signed an informed consent and
release form (See Appendix 3). A representative from the test team witnessed
the participant's signature. To ensure that the test ran smoothly, two staff
members participated in this test, the usability administrator and the data
logger. The usability testing staff conducting the test was experienced usability
practitioners with 3 years of usability experience at GE Healthcare and
additional experience at other facilities. The administrator moderated the
session including administering instructions and tasks. The administrator also
monitored task times, obtained post-task rating data, and ensured notes on
participant comments were recorded. A second person served as the data
logger and took notes on task success, path deviations, number and type of
errors, and comments. Participants were instructed to perform the tasks (see
specific instructions below):
As quickly as possible making as few errors and deviations as possible.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 11 of 48
Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use.
Without using a think aloud technique.
For each task, the participants were given a written copy of the task. Task
timing began once the administrator finished reading the question. The task
time was stopped once the participant indicated they had successfully
completed the task. Scoring is discussed below in Section 3.9. Following the
session, the administrator gave the participant the post-test questionnaire (e.g.,
the System Usability Scale, see Appendix 5), compensated him or her for their
time, and thanked each individual for their participation.
Participants' demographic information, task success rate, time on task, errors,
deviations, verbal responses, and post-test questionnaire were recorded into a
spreadsheet.
Participants were thanked for their time and compensated. Participants signed
a receipt and acknowledgement form (See Appendix 6) indicating that they had
received the compensation.
TEST LOCATION
The test facility included a waiting area and a quiet testing room with a table,
computer for the participant, and recording computer for the administrator.
Only the participant and administrator and the data logger were in the test
room. There were no observers. To ensure that the environment was
comfortable for users, noise levels were kept to a minimum with the ambient
temperature within a normal range. All of the safety instruction and evacuation
procedures were valid, in place, and visible to the participants.
TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or facility. In this
instance, the testing was conducted in the Aprima Medical Software corporate
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 12 of 48
headquarters in the main training room. For testing, the computer used a
Fujitsu T902 running Microsoft Windows 8. The participants used whatever
input method he or she was most comfortable with from among mouse,
keyboard, touchpad, stylus or voice input when interacting with the EHRUT.
The EHRUT used a 12.7-inch diagonal display running 1280x720 pixels at 32
bit color. The application was set up by the administrator according to the
vendor's documentation. The application itself was running on a Dell 620
server running Windows Server 2008 using a test database on a LAN
connection. Technically, the system performance (i.e., response time) was
representative to what actual users would experience in a field implementation.
Additionally, participants were instructed not to change any of the default
system settings (such as control of font size).
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used,
including:
1. Informed Consent
2. Moderator's Guide
3. Post-test Questionnaire
4. Incentive Receipt and Acknowledgment Form
Examples of these documents can be found in Appendices 3-6 respectively.
The Moderator's Guide was devised so as to be able to capture required data.
The participant's interaction with the EHRUT was captured and recorded
digitally with screen capture software running on the test machine.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 13 of 48
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each participant:
Thank you for participating in this study. Your input is very important.
Our session today will last about 30 minutes. During that time you will
use an instance of an electronic health record. I will ask you to
complete a few tasks using this system and answer some questions.
You should complete the tasks as quickly as possible making as few
errors as possible. Please try to complete the tasks on your own
following the instructions very closely. Please note that we are not
testing you we are testing the system, therefore if you have difficulty all
this means is that something needs to be improved in the system. I will
be here in case you need specific help, but I am not able to instruct you
or provide help in how to use the application. Overall, we are interested
in how easy (or how difficult) this system is to use, what in it would be
useful to you, and how we could improve it. I did not have any
involvement in its creation, so please be honest with your opinions.
There are a variety of tools that record screens and transmit those
recordings across a local area network for remote observations. All of
the information that you provide will be kept confidential and your name
will not be associated with your comments at any time. Should you feel
it necessary you are able to withdraw at any time during the testing.
Following the procedural instructions, participants were shown the EHR and as
their first task, were given time (10 minutes) to explore the system and make
comments. Once this task was complete, the administrator gave the following
instructions:
For each task, I will read the description to you and say "Begin." At that
point, please perform the task and say "Done" once you believe you
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 14 of 48
have successfully completed the task. I would like to request that you
not talk aloud or verbalize while you are doing the tasks. I will ask you
your impressions about the task once you are done.
Participants were then given 13 tasks to complete. Tasks are listed in the
moderator's guide in the Appendix.
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the
Usability of Electronic Health Records, EHRs should support a process that
provides a high level of usability for all users. The goal is for users to interact
with the system effectively, efficiently, and with an acceptable level of
satisfaction. To this end, metrics for effectiveness, efficiency and user
satisfaction were captured during the usability testing. The goals of the test
were to assess:
1. Effectiveness of EHRUT by measuring participant success rates and errors
2. Efficiency of EHRUT by measuring the average task time and path
deviations
3. Satisfaction with EHRUT by measuring ease of use ratings
DATA SCORING
The following table (Table 1) details how tasks were scored, errors evaluated,
and the time data analyzed.
Measures Rational and Scoring
Effectiveness: Task Success
A task was counted as a "Success" if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 15 of 48
Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator's Guide must be operationally defined by taking multiple measures of optimal performance and multiplying by some factor that allows some time buffer because the participants are presumably not trained to expert performance. Scoring used a factor of 2. Thus, if expert, optimal performance on a task was 80 seconds then allotted task time performance was 80 * 2 = 160 seconds. This ratio should be aggregated across tasks and reported with mean and variance scores.
Effectiveness: Task Failures
If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a "Failure." No task times were taken for errors. The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. This should also be expressed as the mean number of failed tasks per participant. On a qualitative level, an enumeration of errors and error types were collected.
Efficiency: Task Deviations
The participant's path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps in order to provide a ratio of path deviation. It is strongly recommended that task deviations be reported. Optimal paths (i.e., procedural steps) should be recorded when constructing tasks.
Efficiency: Task Time
Each task was timed from when the administrator said "Begin" until the participant said, "Done." If he or she failed to say "Done," the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.
Satisfaction: Participant's subjective impression of the ease of use of the
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 16 of 48
Task Rating application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate "Overall, this task was:" on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants' confidence in and likeability of the EHRUT overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, "I think I would like to use this system frequently," "I thought the system was easy to use," and "I would imagine that most people would learn to use this system very quickly." See full System Usability Score questionnaire in Appendix 5.
Table 1. Details of how observed data were scored.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 17 of 48
RESULTS
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods
specified in the Usability Metrics section above. Participants who failed to
follow session and task instructions had their data excluded from the analyses.
In this test cycle, there were no data excluded.
The usability testing results for the EHRUT are detailed below. The results
should be seen in light of the objectives and goals outlined in Section 3.2 Study
Design. The data should yield actionable results that, if corrected, yield
material, positive impact on user performance.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations (Observed / Optimal)
Mean(SD)
Min/Max Optimal
Mean (SD)
Mean (SD)
1. Task 1 6 100% (0.00)
2.8 (59/42) 152 (58)
1:04/3:44 0:38
0.3 (0.7) 3.5 (1.1)
2. Task 2 6 83% (0.37)
2.0 (60/48) 118 (40)
1:03/2:54 0:36
1.0 (0.6) 3.8 (0.7)
3. Task 3 6 100% (0.00)
1.7 (58/48) 78 (41)
0:28/2:36 0:28
0.5 (0.5) 4.5 (0.5)
The results from the SUS (System Usability Scale) scored the subjective
satisfaction with the system based on performance with these tasks to be: 83.
Broadly interpreted, scores under 60 represent systems with poor usability;
scores over 80 would be considered above average
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 18 of 48
DISCUSSION OF THE FINDINGS
The information presented and the way in which it was presented was well
received by the test users. Even when unfamiliar with the interface participants
were able to complete the task. Only one failure on the three tasks occurred.
Customization of the product and unfamiliarity with the interface slowed many
of the users down. The majority of path deviations and errors we due to the
limited exposure of the participants to the system. Repetition on subsequent
tasks showed rapid improvement (Path deviations decreased by 1.1 on second
attempt and decreased by 2.5 by the fourth attempt to 0.3)
EFFECTIVENESS
Users with a small amount of time on the system improved dramatically. All
users rated the product either agree or strongly agree with regard to “I thought
the system was easy to use”. All users also rated “I found the system
unnecessarily complex” either disagree or strongly disagree.
EFFICIENCY
The number of path deviations on some tasks was above 2.0 this is large
number. For most tasks the number was at or below 1.0. Users were in almost
all cases able to quickly come back to the optimal path and complete the task.
This is true for the tasks in which the path deviations where the highest as well.
Variance on the timings show that the untrained users are not efficient using
Aprima. User learning is rapid; by task 5 the users were able to be more
efficient with task times being one half or less compared to the first 4 tasks.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 19 of 48
SATISFACTION
User satisfaction was quite high. Overall user rating was 83.
Question Agree or Strongly Agree
Disagree or Strongly Disagree
I think that I would like to use this system frequently
83% 0%
I found the system unnecessarily complex
0% 100%
I thought the system was easy to use
100% 0%
I think that I would need the support of a technical person to be able to use this system
17% 67%
I found the various functions in this system were well integrated
83% 0%
I thought there was too much inconsistency in this system
0% 100%
I would imagine that most people would learn to use this system very quickly
100% 0%
I found the system very cumbersome to use
33% 67%
I felt very confident using the system
100% 0%
I needed to learn a lot of things before I could get going with this system
0% 100%
“I found the system cumbersome to use” seems to be a polarizing question. In
future user testing this will have to be explored more, as we did not find this to
be reflected in the user comments and was not discovered until the data
analysis phase.
MAJOR FINDINGS
The information presented and the way in which it was presented was
well received by the test users. Even when unfamiliar with the
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 20 of 48
interface participants were able to complete the task. Only one failure
on the three tasks occurred.
Customization of the product and unfamiliarity with the interface
slowed many of the users down. The majority of the path deviations
and errors we primarily due to the limited exposure of the participants.
Repetition on subsequent tasks showed rapid improvement (Path
deviations decreased by 1.1 on second attempt and decreased by 2.5
by the fourth attempt)
Creation of the procedure task message starts with the participant
needing to find the new button. This caused users to wander around
the screen trying to find the button. It is not where users expect it to be
and caused all users to take longer than expected for the task.
Subsequent repetition showed immediate improvement to within
expected limits.
Approval of results caused some user confusion. There are many ways
to approve the results, but the UI does not specifically lead users to a
specific path. This caused wide variation in user completion times.
AREAS FOR IMPROVEMENT
The process of approving a message needs to be looked at to help
lead unfamiliar users to the correct actions more quickly. The users
were able to successfully complete the task, but users found it difficult
and it impacted confidence.
The New button for creating all workflow messages is difficult for users
to discover unaided, and should be evaluated to improve ease of use.
When a user goes off the correct path, he will often take a very long
time and many more errors before finding the correct path again.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 21 of 48
Aprima has many ways to do the same operation. This leads to
confusion for users. Aprima should limit access to feature models to
simplify the user paradigm.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 22 of 48
APPENDICES
The following appendices include supplemental data for this usability test
report. Following is a list of the appendices provided:
1: Sample Recruiting screener
2: Participant demographics
3: Non-Disclosure Agreement (NDA) and Informed Consent Form
4: Example Moderator's Guide
5: System Usability Scale Questionnaire
6: Incentive receipt and acknowledgment form It is important to note that these
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 23 of 48
Appendix 1: SAMPLE RECRUITING SCREENER
The purpose of a screener to ensure that the participants selected represent
the target user population as closely as possible. (Portions of this sample
screener are taken from www.usability.gov/templates/index.html#Usability and
adapted for use.)
Recruiting Script for Recruiting Firm
Hello, my name is _______________, calling from [Insert name of recruiting
firm]. We are recruiting individuals to participate in a usability study for an
electronic health record. We would like to ask you a few questions to see if
you qualify and if would like to participate. This should only take a few minutes
of your time. This is strictly for research purposes. If you are interested and
qualify for the study, you will be paid to participate. Can I ask you a few
questions?
Customize this by dropping or adding questions so that it reflects your EHR's
primary audience
1. [If not obvious] Are you male or female? [Recruit a mix of participants]
2. Have you participated in a focus group or usability test in the past xx months? [If yes, Terminate]
3. Do you, or does anyone in your home, work in marketing research, usability research, web design [If yes, Terminate]
4. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record software or consulting company? [If yes, Terminate]
5. Which of the following best describes your age? [23 to 39; 40 to 59; 60 and older] [Recruit Mix]
6. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-American, Latino/a or Hispanic, etc.]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 24 of 48
7. Do you require any assistive technologies to use a computer? [if so, please describe] Professional Demographics Customize this to reflect your EHR's primary audience
8. What is your current position and title? (Must be healthcare provider)
□ RN: Specialty _______________________
□ Physician: Specialty _______________________
□ Resident: Specialty _______________________
□ Medical Assistant _______________________
□ Administrative Staff
□ Other [Terminate]
9. How long have you held this position?
10. Describe your work location (or affiliation) and environment? (Recruit according to the intended users of the application. e.g., private practice, health system, government clinic, etc.)
11. Which of the following describes your highest level of education? [e.g., high school graduate/GED, some college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)
12. Besides reading email, what professional activities do you do on the computer? [e.g., access EHR, research; reading news; shopping/banking; digital pictures; programming/word processing, etc.] [If no computer use at all, Terminate]
13. About how many hours per week do you spend on the computer? [Recruit according to the demographics of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]
14. What computer platform do you usually use? [e.g., Mac, Windows, etc.]
15. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]
16. In the last month, how often have you used an electronic health record?
17. How many years have you used an electronic health record?
18. How many EHRs do you use or are you familiar with?
19. How does your work environment handle patient records? [Recruit according to the demographics of the intended users]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 25 of 48
□ On paper
□ Some paper, some electronic
□ All electronic
Contact Information If the person matches your qualifications, ask
Those are all the questions I have for you. Your background matches the
people we're looking for. For your participation, you will be paid $300. Would
you be able to participate on [date, time]? [If so collect contact information]
May I get your contact information?
■ Name of participant:
■ Address:
■ City, State, Zip:
■ Daytime phone number:
■ Evening phone number:
■ Alternate [cell] phone number:
■ Email address:
Before your session starts, we will ask you to sign a release form allowing us to
record your session. The record will only be used internally for further study if
needed. Will you consent to be digitally recorded? This study will take place at
3330 Keller Springs Road, Suite 201, Carrollton, TX 75006. I will confirm your
appointment a couple of days before your session and provide you with
directions to our office. What time is the best time to reach you?
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 26 of 48
Appendix 2: PARTICIPANT DEMOGRAPHICS
Following is a high-level overview of the participants in this study.
Gender
Men: 0 Women: 6 Total (participants): 6
Occupation/Role
RN/BSN: 0 Physician: 2 Admin Staff: 4 Total (participants): 6
Years of Experience
Facility Use of EHR All paper: 1 Some paper, some electronic: 5 All electronic: 0 Total (participants): 6
The full break down is included in the report above.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 27 of 48
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM
Non-Disclosure Agreement THIS AGREEMENT is entered into as of _________________, 2013, between ________________ ("the Participant") and the testing organization Aprima Medical Software located at 3330 Keller Springs Rd. Suite 201, Carrollton, TX 75006. The Participant acknowledges his or her voluntary participation in today's usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by Test Company, or otherwise acquired by the Participant, in the course of today's study. By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts. Any information the Participant acquires relating to this product during this study is confidential and proprietary to Test Company and is being disclosed solely for the purposes of the Participant's participation in today's usability study. By signing this form the Participant acknowledges that s/he will receive monetary compensation for feedback and will not disclose this confidential information obtained today to anyone else or any other organizations. Participant's printed name: ___________________________________________________ Signature: __________________________________________________________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 28 of 48
Informed Consent Test Company would like to thank you for participating in this study. The purpose of this study is to evaluate an electronic health records system. If you decide to participate, you will be asked to perform several tasks using the prototype and give your feedback. The study will last about 30 minutes. At the conclusion of the test, you will be compensated for your time. Agreement I understand and agree that as a voluntary participant in the present study conducted by Test Company I am free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study conducted and videotaped by the Test Company. I understand and consent to the use and release of the videotape by Test Company. I understand that the information and videotape is for research purposes only and that my name and image will not be used for any purpose other than research. I relinquish any rights to the videotape and understand the videotape may be copied and used by Test Company without further permission. I understand and agree that the purpose of this study is to make software applications more useful and usable in the future. I understand and agree that the data collected from this study may be shared with outside of Test Company and Test Company's client. I understand and agree that data confidentiality is assured, because only de- identified data - i.e., identification numbers not names - will be used in analysis and reporting of the results. I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can leave at any time. Please check one of the following: □ YES, I have read the above statement and agree to be a participant. □ NO, I choose not to participate in this study. Signature: _________________________________ Date: _______________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 29 of 48
Appendix 4: EXAMPLE MODERATOR'S GUIDE
Only four tasks are presented here for illustration. EHRUT Usability Test Moderator's Guide Administrator _____________________
Data Logger _____________________
Date _________________ Time _____________________
Participant # _________
Location _____________________
Prior to testing
■ Confirm schedule with Participants ■ Ensure EHRUT lab environment is running properly ■ Ensure Consent forms are complete for all participants ■ Ensure lab and data recording equipment is running properly
Prior to each participant: ■ Reset application ■ Restore application to base state using script restoredb.bat ■ Start session recordings with GTM Recording
Prior to each task: ■ Reset application to starting point for next task
After each participant: ■ End session recordings with tool ■ Fill out all paper work and acknowledgement form.
After all testing ■ Scan all forms and shred originals ■ Back up all video and data files
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 30 of 48
Orientation (5 minutes) Thank you for participating in this study. Our session today will last 2 hours today. During that time you will take a look at an electronic health record system. I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with anything to do with the system itself. Please save your detailed comments until the end of a task or the end of the session as a whole when we can discuss freely. I did not have any involvement in its creation, so please be honest with your opinions. The product you will be using today is describe the state of the application, i.e., production version, early prototype, etc. Some of the data may not make sense as it is placeholder data. We are recording the audio and screenshots of our session today. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Do you have any questions or concerns? Preliminary Questions (5 minutes) What is your job title / appointment? How long have you been working in this role? What are some of your main responsibilities? Tell me about your experience with electronic health records.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 31 of 48
Preview: First Impressions (6 Minutes) This is the application you will be working with. Have you heard of it? Yes ___ No ___ If so, tell me what you know about it. Show test participant the EHRUT. Please don't click on anything just yet. What do you notice? What are you able to do here? Please be specific. Notes / Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 32 of 48
Task 1: Create an Order – Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC The patient you are examining 5’ 10” 285 lbs. male needs a differential diagnosis of 034.0 Streptococcal Sore Throat and needs a 87880 rapid strep to confirm diagnosis. Enter Vitals, Dx and SP into the system Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, enter 5 feet 10 inches and 285 lbs 2. go to the DX tab and Enter the diagnosis 034.0 Streptococcal Sore Throat 3. Note Alert for Obesity and Hypertension in CDS slider 4. Still inside FNC, go to the SP tab 5. Enter procedure code 87880 Strep A Assay 6. Still inside FNC, click the Close button to close the note, save the note as incomplete 7. Click okay.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 33 of 48
Task 2: Utilize an existing Clinical Decision Support Alert – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You run are in consultation with a patient and his parent who is 71 years old. You want to quickly see if there are any Clinical Support Alerts for the parent. Go to the Appointment to review CDS Alerts and look at the source information for the alert. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
Open Patient Norman Wood
Open Update Appointment from Envelope
Open Clinical Decision Support Tab
Note Fall Risk Rule on Slider
Click on the Hyperlink for Fall Risk Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 34 of 48
Task 3: Creation of a Procedure Task Message to the Nurses - Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC
A patient you saw earlier that requested an order for a test at a different lab wants the lab found it was not covered by his insurance. He wants it to be performed in your office. You need to order a rapid strep test (87880) for the patient, as he is on his way back into the office.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open the scheduler 2. Find the appointment for Ed Wood 3. Click the Blue Folder (Open/Create note) 4. Click New->Procedure Task Message
a. Enter the Nurses b. Click Okay
5. Click to close the note a. Click Okay
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 35 of 48
Task 4: Approve a Lab result (5 Minutes) Take the participant to the starting point for the task. - DESKTOP
You have a lab result request message in your inbox to process. You need to review and approve the request.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations Approval of Lab Results
1. Locate the Lab Result Approval message in the Desktop Message pane 2. Click on the Time and Date Hyperlink or Plus (+) sign next to the message 3. Click on Process Message 4. From the Modify Message window 5. After review of the results, click the Approved All button
a. Your user (provider) name will auto populate along with the current date 6. Click Complete
a. This will simultaneously set the approved lab results and complete the message Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 36 of 48
Task 5: Document Drug Allergy in History – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are examining reports to you that she has been taking no meds because she has an anaphylactic reaction to Penicillins. Please document the allergy in History Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Drug Allergy
6. In the Allergen column, select Penicillins.
7. In the Reaction column, select anaphylaxis.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 37 of 48
Task 6: Drug Allergy Alert in Rx Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Need to prescribe Keflex to Mary Woods chart 1 tablet per day by mouth for 10 days Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Rx Tab
6. In the Drug field, enter Keflex
7. Click the find control
8. Select Keflex 500 mg
9. Pop up: Receive Drug Screening Warning: ALLERGY ALERT: The use of Keflex cross-sensitivity reaction based on a reported history of allergy to PENICILLINS. Do you want to continue? Click Yes
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 38 of 48
Task 7: Drug Drug Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP An agitated patient wants you to prescribe Prozac immediately. 20mg 1 time per day by mouth. Please prescribe the medication. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Find Mary Wood that has an appointment scheduled.
2. Click the Open/Create Note icon
3. FNC
4. Click Rx Tab
5. In the Drug field, enter Prozac
6. Click the find control
7. Select Prozac 20mg tablet
8. Pop up: Receive Drug Screening Warning: Duplicate Therapy: The use of Prozac is a duplicate therapy with Zoloft 50 mg. Do you want to continue?
a. Click No
9. Print -> Save and print Button
a. Print Dialog
10. Click ok to save and exit the note
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 39 of 48
Task 8: Medication List in OPS – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You are about to walk into an exam room for a med check on a patient. Go to One Page Summary and review the patient’s drug list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Go to Patient Demographics 2. Click Patient 3. Click One Page Summary 4. Click hyperlink in index for Current Medication 5. Verify the data is displayed correctly
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 40 of 48
Task 9: Medication List in Hx Tab with Filter – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
Previously very ill patient with a very large medication history is asking about her current medications. You want to compare the active/inactive medications with the maintenance medication list (which are two different filters in the system Go to Hx Tab and change the filter to show only Active Medications and them maintenance medications.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, go to the HX tab 2. Click on the Medication History 3. Review the items on the list 4. Click the search icon to change the filter 5. Change the filter to maintenance 6. Review the Maintenance Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 41 of 48
Task 10: Rx Refill Complete (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You have a few minutes to complete your drug refill request messages Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Messages
2. Locate message
3. Click on patient’s blue envelope.
4. Select Process Rx Refill
5. Medication History screen
6. Select the drug by clicking the Refill button.
7. Sig Writer screen
a. Verify information in sig
8. Click Ok
a. Return to the Modify Message screen
9. Click Complete.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 42 of 48
Task 11: import Allergies from external list - Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Allergy list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. Go to attachments – Allergies - CCD 3. Click the import allergies icon 4. Sulfasalazine -> Add 5. Penicillin V -> Merge/Add 6. Carbamazepine -> ignore/remove 7. Click Reconcile 8. Go to Hx
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 43 of 48
Task 12: Import Medications from external list – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Medication list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. View the medication CCD 3. Click the import medications icon 4. Simvastatin -> Add 5. Lorazepam -> Add 6. Lisinopril -> Add/Merge 7. Click reconcile 8. Go to patient Hx – Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 44 of 48
Task 13: Verify import in Allergy list and Medication List – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Please validate that the import of Allergy list and Medication list import has the expected information Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Note the CDS rule for the combination of the new allergy and medication
6. Click Drug Allergy
a. Verify the information is correct from the import
7. Click on the Medication History
a. Verify the information is correct from the import
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 45 of 48
Final Questions (10 Minutes) What was your overall impression of this system? What aspects of the system did you like most? What aspects of the system did you like least? Were there any features that you were surprised to see? What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? Compare this system to other systems you have used. Would you recommend this system to your colleagues? Administer the SUS
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 46 of 48
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
Strongly
disagree Strongly
Agree
1. I think that I would like to use this system frequently
1 2 3 4 5
2. I found the system unnecessarily complex
1 2 3 4 5
3. I thought the system was easy to use
1 2 3 4 5
4. I think that I would need the support of a technical person to be able to use this system
1 2 3 4 5
5. I found the various functions in this system were well integrated
1 2 3 4 5
6. I thought there was too much inconsistency in this system
1 2 3 4 5
7. I would imagine that most people would learn to use this system very quickly
1 2 3 4 5
8. I found the system very cumbersome to use
1 2 3 4 5
9. I felt very confident using the system
1 2 3 4 5
10. I needed to learn a lot of things before I could get going with this system
1 2 3 4 5
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 47 of 48
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM
Acknowledgement of Receipt I hereby acknowledge receipt of $ _____ for my participation in a research study run by Aprima Medical Software. Printed Name: _____________________________________________________________ Address: __________________________________________________________________ __________________________________________________________________________ Signature: _______________________________________________ Date: ____________ Usability Researcher: ________________________________________________________ Signature of Usability Researcher: ____________________________ Date: ______________ Witness: ___________________________________________________________________ Witness Signature: ________________________________________ Date: ______________
EHR USABILITY TEST REPORT
OF
APRIMA 2014
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
Aprima 2014 Date of Usability Test: 05/21/13-05/23/13 Date of Report: 05/27/13 Report Prepared By: Aprima Medical Software Project Management Cornell Hardman (214) 466-8107 Aprima Medical Software 3330 Keller Springs Road Suite 201 Carrollton, TX 75006
Table of Contents EXECUTIVE SUMMARY ................................................................................................................. 3
Major findings ............................................................................................................................... 5
Areas for improvement ................................................................................................................. 5
INTRODUCTION ............................................................................................................................. 6
METHOD ......................................................................................................................................... 6
PARTICIPANTS ........................................................................................................................... 6
STUDY DESIGN .......................................................................................................................... 8
TASKS ......................................................................................................................................... 9
PROCEDURES ............................................................................................................................ 9
TEST LOCATION ...................................................................................................................... 10
TEST ENVIRONMENT .............................................................................................................. 10
TEST FORMS AND TOOLS ...................................................................................................... 11
PARTICIPANT INSTRUCTIONS ............................................................................................... 12
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 2 of 46
USABILITY METRICS ............................................................................................................... 13
DATA SCORING ........................................................................................................................ 13
RESULTS ...................................................................................................................................... 16
DATA ANALYSIS AND REPORTING ........................................................................................ 16
DISCUSSION OF THE FINDINGS ............................................................................................ 17
EFFECTIVENESS ..................................................................................................................... 17
EFFICIENCY .............................................................................................................................. 17
SATISFACTION ......................................................................................................................... 18
MAJOR FINDINGS .................................................................................................................... 19
AREAS FOR IMPROVEMENT .................................................................................................. 19
APPENDICES ................................................................................................................................ 20
Appendix 1: SAMPLE RECRUITING SCREENER .................................................................... 21
Appendix 2: PARTICIPANT DEMOGRAPHICS ........................................................................ 24
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ........... 25
Appendix 4: EXAMPLE MODERATOR'S GUIDE ...................................................................... 27
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................. 44
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM ................................. 45
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 3 of 46
EXECUTIVE SUMMARY
A usability test of Aprima 2014 ambulatory EHR was conducted on the dates of
May 21-23 of 2013 in The Aprima Medical Software corporate office by Aprima
Medical Software Project Management team. The purpose of this test was to
test and validate the usability of the current user interface, and provide
evidence of usability in the EHR Under Test (EHRUT). During the usability
test, 2 healthcare providers and 4 other knowledgeable clinic staff matching the
target demographic criteria served as participants and used the EHRUT in
simulated, but representative tasks. This study collected performance data on
13 tasks including 3 tasks specifically designed to test Drug Allergy
functionality that are typically conducted on an EHR:
Task 1 – Document Drug Allergy in History
Task 2 – Drug Allergy Alert in Rx Interaction
Task 3 – Verify import in Allergy list and Medication List
During the 2-hour one-on-one usability test, each participant was greeted by
the administrator and asked to review and sign an informed consent/release
form (included in Appendix 3); they were instructed that they could withdraw at
any time. Participants had a mix of experience with the EHR. The
administrator introduced the test, and instructed participants to complete a
series of tasks (given one at a time) using the EHRUT. During the testing, the
administrator timed the test and, along with the data logger(s) recorded user
performance data on paper and electronically. The administrator did not give
the participant assistance in how to complete the task.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 4 of 46
Participant screens were recorded for subsequent analysis.
The following types of data were collected for each participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations Participant's satisfaction ratings of the system
All participant data was de-identified - no correspondence could be made from
the identity of the participant to the data collected. Following the conclusion of
the testing, participants were asked to complete a post-test questionnaire and
were compensated with $300 for their time. Various recommended metrics, in
accordance with the examples set forth in the NIST Guide to the Processes
Approach for Improving the Usability of Electronic Health Records, were used
to evaluate the usability of the EHRUT. Following is a summary of the
performance and rating data collected on the EHRUT.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations (Observed / Optimal)
Mean(SD)
Min/Max Optimal
Mean (SD)
Mean (SD)
1. Task 1 6 100% (0.00)
1.0 (48/42) 75 (38)
0:34/2:13 0:22
0.2 (0.4)
4.5 (0.8)
2. Task 2 6 100% (0.00)
0.5 (57/54) 50 (25)
0:16/1:33 0:17
0.3 (0.5)
4.8 (0.4)
3. Task 3 6 100% (0.00)
0.3 (38/36) 33 (13)
0:22/1:00 0:22
0.2 (0.4)
4.3 (0.5)
The results from the System Usability Scale scored the subjective satisfaction
with the system based on performance with these tasks to be: 83
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 5 of 46
In addition to the performance data, the following qualitative observations were
made:
Major findings
The information presented and the way in which it was presented was
well received by the test users.
The Drug Allergy Item in history was more difficult for users to find the
time because it was below the visible part of the history items list. This
caused unfamiliar users to slow down the first time through the
workflow, decreasing average time to complete by a 33% the second
time through and another 33% the third time.
All users were able to complete all the related tasks.
No major rework is required for the Medication Allergy list functionality
Areas for improvement
None
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 6 of 46
INTRODUCTION
The EHRUT tested for this study was Aprima 2014 ambulatory EHR. Designed
to present medical information to healthcare providers in ambulatory medical
offices, the EHRUT consists of a user interface presented on a computer
display that stores and retrieves data from a database server through an
application server. The usability testing attempted to represent realistic
exercises and conditions. The purpose of this study was to test and validate
the usability of the current user interface, and provide evidence of usability in
the EHR Under Test (EHRUT). To this end, measures of effectiveness,
efficiency and user satisfaction, such as ease of access, number of mouse
clicks, time on task, time to repeat task, time to first entry and number of errors,
were captured during the usability testing.
METHOD
PARTICIPANTS
A total of 6 participants were tested on the EHRUT. Participants in the test
were physicians, nurses and other knowledgeable office staff. Participants
were recruited by Health Care Strategies and were compensated $300 for their
time. In addition, participants had no direct connection to the development of
or organization producing the EHRUT(s). Participants were not from the
testing or supplier organization. Participants were given the opportunity to
have an abbreviated orientation and a minimal level of training compared to the
amount the actual end users would have received. For the test purposes, end-
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 7 of 46
user characteristics were identified and translated into a recruitment screener
used to solicit potential participants; an example of a screener is provided in
Appendix 1.
Recruited participants had a mix of backgrounds and demographic
characteristics conforming to the recruitment screener. The following is a table
of participants by characteristics, including demographics, professional
experience, computing experience and user needs for assistive technology.
Participant names were replaced with Participant IDs so that an individual's
data cannot be tied back to individual identities.
Part ID
Gender Age Education Occupation /Role
Professional Experience
Computer Exp.
Product Exp.
Assistive Technology
Needs
1 142101 F 25 College Degree
Medical
Assistant 5 9 0/4
2 142102 F 42 College Degree
Medical
Assistant 3 13 0/1 3 142201 F 65 MD/PhD/DO Physician 38 4 0/0
4 142202 F 39 College Degree
Medical
Assistant 16 16 0/3
5 142301 F 31
Masters
Degree
Physician
Assistant 9 19 0/6
6 142302 F 30 College Degree
Medical
Assistant 8 13 0/7
9 participants (matching the demographics in the section on Participants) were
recruited and 6 participated in the usability test. 3 participants failed to show
for the study.
Participants were scheduled for 2 hour session with at least 1 hour in between
each session for debrief by the administrator(s) and data logger(s), and to reset
systems to proper test conditions. A spreadsheet was used to keep track of
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 8 of 46
the participant schedule, and included each participant's demographic
characteristics as provided by the recruiting firm.
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application
performed well - that is, effectively, efficiently, and with satisfaction - and areas
where the application failed to meet the needs of the participants. The data
from this test may serve as a baseline for future tests with an updated version
of the same EHR and/or comparison with other EHRs provided the same tasks
are used. In short, this testing serves as both a means to record or benchmark
current usability, but also to identify areas where improvements must be made.
During the usability test, participants interacted with a single EHR. Each
participant used the system in the same location, and was provided with the
same instructions. The system was evaluated for effectiveness, efficiency and
satisfaction as defined by measures collected and analyzed for each
participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations (comments) Participant's satisfaction ratings of the system
Additional information about the various measures can be found in
Section 3.9 on Usability Metrics.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 9 of 46
TASKS
A number of tasks were constructed that would be realistic and representative
of the kinds of activities a user might do with this EHR, including:
Task 1 – Document Drug Allergy in History
Task 2 – Drug Allergy Alert in Rx Interaction
Task 3 – Verify import in Allergy list and Medication List
Tasks were selected based on their frequency of use, criticality of function, and
those that may be most troublesome for users. Tasks should always be
constructed in light of the study objectives.
PROCEDURES
Upon arrival, participants were greeted; their identity was verified and matched
with a name on the participant schedule. Participants were then assigned a
participant ID. Each participant reviewed and signed an informed consent and
release form (See Appendix 3). A representative from the test team witnessed
the participant's signature. To ensure that the test ran smoothly, two staff
members participated in this test, the usability administrator and the data
logger. The usability testing staff conducting the test was experienced usability
practitioners with 3 years of usability experience at GE Healthcare and
additional experience at other facilities. The administrator moderated the
session including administering instructions and tasks. The administrator also
monitored task times, obtained post-task rating data, and ensured notes on
participant comments were recorded. A second person served as the data
logger and took notes on task success, path deviations, number and type of
errors, and comments. Participants were instructed to perform the tasks (see
specific instructions below):
As quickly as possible making as few errors and deviations as possible.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 10 of 46
Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use.
Without using a think aloud technique.
For each task, the participants were given a written copy of the task. Task
timing began once the administrator finished reading the question. The task
time was stopped once the participant indicated they had successfully
completed the task. Scoring is discussed below in Section 3.9. Following the
session, the administrator gave the participant the post-test questionnaire (e.g.,
the System Usability Scale, see Appendix 5), compensated him or her for their
time, and thanked each individual for their participation.
Participants' demographic information, task success rate, time on task, errors,
deviations, verbal responses, and post-test questionnaire were recorded into a
spreadsheet.
Participants were thanked for their time and compensated. Participants signed
a receipt and acknowledgement form (See Appendix 6) indicating that they had
received the compensation.
TEST LOCATION
The test facility included a waiting area and a quiet testing room with a table,
computer for the participant, and recording computer for the administrator.
Only the participant and administrator and the data logger were in the test
room. There were no observers. To ensure that the environment was
comfortable for users, noise levels were kept to a minimum with the ambient
temperature within a normal range. All of the safety instruction and evacuation
procedures were valid, in place, and visible to the participants.
TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or facility. In this
instance, the testing was conducted in the Aprima Medical Software corporate
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 11 of 46
headquarters in the main training room. For testing, the computer used a
Fujitsu T902 running Microsoft Windows 8. The participants used whatever
input method he or she was most comfortable with from among mouse,
keyboard, touchpad, stylus or voice input when interacting with the EHRUT.
The EHRUT used a 12.7-inch diagonal display running 1280x720 pixels at 32
bit color. The application was set up by the administrator according to the
vendor's documentation. The application itself was running on a Dell 620
server running Windows Server 2008 using a test database on a LAN
connection. Technically, the system performance (i.e., response time) was
representative to what actual users would experience in a field implementation.
Additionally, participants were instructed not to change any of the default
system settings (such as control of font size).
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used,
including:
1. Informed Consent
2. Moderator's Guide
3. Post-test Questionnaire
4. Incentive Receipt and Acknowledgment Form
Examples of these documents can be found in Appendices 3-6 respectively.
The Moderator's Guide was devised so as to be able to capture required data.
The participant's interaction with the EHRUT was captured and recorded
digitally with screen capture software running on the test machine.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 12 of 46
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each participant:
Thank you for participating in this study. Your input is very important.
Our session today will last about 30 minutes. During that time you will
use an instance of an electronic health record. I will ask you to
complete a few tasks using this system and answer some questions.
You should complete the tasks as quickly as possible making as few
errors as possible. Please try to complete the tasks on your own
following the instructions very closely. Please note that we are not
testing you we are testing the system, therefore if you have difficulty all
this means is that something needs to be improved in the system. I will
be here in case you need specific help, but I am not able to instruct you
or provide help in how to use the application. Overall, we are interested
in how easy (or how difficult) this system is to use, what in it would be
useful to you, and how we could improve it. I did not have any
involvement in its creation, so please be honest with your opinions.
There are a variety of tools that record screens and transmit those
recordings across a local area network for remote observations. All of
the information that you provide will be kept confidential and your name
will not be associated with your comments at any time. Should you feel
it necessary you are able to withdraw at any time during the testing.
Following the procedural instructions, participants were shown the EHR and as
their first task, were given time (10 minutes) to explore the system and make
comments. Once this task was complete, the administrator gave the following
instructions:
For each task, I will read the description to you and say "Begin." At that
point, please perform the task and say "Done" once you believe you
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 13 of 46
have successfully completed the task. I would like to request that you
not talk aloud or verbalize while you are doing the tasks. I will ask you
your impressions about the task once you are done.
Participants were then given 13 tasks to complete. Tasks are listed in the
moderator's guide in the Appendix.
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the
Usability of Electronic Health Records, EHRs should support a process that
provides a high level of usability for all users. The goal is for users to interact
with the system effectively, efficiently, and with an acceptable level of
satisfaction. To this end, metrics for effectiveness, efficiency and user
satisfaction were captured during the usability testing. The goals of the test
were to assess:
1. Effectiveness of EHRUT by measuring participant success rates and errors
2. Efficiency of EHRUT by measuring the average task time and path
deviations
3. Satisfaction with EHRUT by measuring ease of use ratings
DATA SCORING
The following table (Table 1) details how tasks were scored, errors evaluated,
and the time data analyzed.
Measures Rational and Scoring
Effectiveness: Task Success
A task was counted as a "Success" if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 14 of 46
Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator's Guide must be operationally defined by taking multiple measures of optimal performance and multiplying by some factor that allows some time buffer because the participants are presumably not trained to expert performance. Scoring used a factor of 2. Thus, if expert, optimal performance on a task was 80 seconds then allotted task time performance was 80 * 2 = 160 seconds. This ratio should be aggregated across tasks and reported with mean and variance scores.
Effectiveness: Task Failures
If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a "Failure." No task times were taken for errors. The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. This should also be expressed as the mean number of failed tasks per participant. On a qualitative level, an enumeration of errors and error types were collected.
Efficiency: Task Deviations
The participant's path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps in order to provide a ratio of path deviation. It is strongly recommended that task deviations be reported. Optimal paths (i.e., procedural steps) should be recorded when constructing tasks.
Efficiency: Task Time
Each task was timed from when the administrator said "Begin" until the participant said, "Done." If he or she failed to say "Done," the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.
Satisfaction: Participant's subjective impression of the ease of use of the
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 15 of 46
Task Rating application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate "Overall, this task was:" on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants' confidence in and likeability of the EHRUT overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, "I think I would like to use this system frequently," "I thought the system was easy to use," and "I would imagine that most people would learn to use this system very quickly." See full System Usability Score questionnaire in Appendix 5.
Table 1. Details of how observed data were scored.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 16 of 46
RESULTS
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods
specified in the Usability Metrics section above. Participants who failed to
follow session and task instructions had their data excluded from the analyses.
In this test cycle, there were no data excluded.
The usability testing results for the EHRUT are detailed below. The results
should be seen in light of the objectives and goals outlined in Section 3.2 Study
Design. The data should yield actionable results that, if corrected, yield
material, positive impact on user performance.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations(Observed / Optimal)
Mean (SD)
Min/Max Optimal
Mean (SD)
Mean (SD)
1. Task 1 6 100% (0.00)
1.0 (48/42)
75 (38) 0:34/2:13 0:22
0.2 (0.4)
4.5 (0.8)
2. Task 2 6 100% (0.00)
0.5 (57/54)
50 (25) 0:16/1:33 0:17
0.3 (0.5)
4.8 (0.4)
3. Task 3 6 100% (0.00)
0.3 (38/36)
33 (13) 0:22/1:00 0:22
0.2 (0.4)
4.3 (0.5)
The results from the SUS (System Usability Scale) scored the subjective
satisfaction with the system based on performance with these tasks to be: 83.
Broadly interpreted, scores under 60 represent systems with poor usability;
scores over 80 would be considered above average
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 17 of 46
DISCUSSION OF THE FINDINGS
The information presented and the way in which it was presented was well
received by the test users.
All users were able to complete all the related tasks.
No major rework is required for the Medication Allergy list functionality
EFFECTIVENESS
The users were able to do what was expected with the product. Many times the
users went slightly off the optimal the path but did not lose their way. Users
with a small amount of time on the system improved dramatically. All users
rated the product either agree or strongly agree with regard to “I thought the
system was easy to use”. All users also rated “I found the system
unnecessarily complex” either disagree or strongly disagree.
EFFICIENCY
The Drug Allergy Item in history was more difficult for users to find the time
because it was below the visible part of the history items list. This caused
unfamiliar users to slow down the first time through the workflow, decreasing
average time to complete by a 33% the second time through and another 33%
the third time.
The number of path deviations on some tasks was above 2.0 this is large
number. For most tasks the number was at or below 1.0. Users were in almost
all cases able to quickly come back to the optimal path and complete the task.
This is true for the tasks in which the path deviations where the highest as well.
Variance on the timings show that the untrained users are not efficient using
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 18 of 46
Aprima. User learning is rapid; by task 5 the users were able to be more
efficient with task times being one half or less compared to the first 4 tasks.
SATISFACTION
User satisfaction was quite high. Overall user rating was 83.
Question Agree or Strongly Agree
Disagree or Strongly Disagree
I think that I would like to use this system frequently
83% 0%
I found the system unnecessarily complex
0% 100%
I thought the system was easy to use
100% 0%
I think that I would need the support of a technical person to be able to use this system
17% 67%
I found the various functions in this system were well integrated
83% 0%
I thought there was too much inconsistency in this system
0% 100%
I would imagine that most people would learn to use this system very quickly
100% 0%
I found the system very cumbersome to use
33% 67%
I felt very confident using the system
100% 0%
I needed to learn a lot of things before I could get going with this system
0% 100%
“I found the system cumbersome to use” seems to be a polarizing question. In
future user testing this will have to be explored more, as we did not find this to
be reflected in the user comments and was not discovered until the data
analysis phase.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 19 of 46
MAJOR FINDINGS
The information presented and the way in which it was presented was
well received by the test users.
The Drug Allergy Item in history was more difficult for users to find the
time because it was below the visible part of the history items list. This
caused unfamiliar users to slow down the first time through the
workflow, decreasing average time to complete by a 33% the second
time through and another 33% the third time.
All users were able to complete all the related tasks.
No major rework is required for the Medication Allergy list functionality
AREAS FOR IMPROVEMENT
None
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 20 of 46
APPENDICES
The following appendices include supplemental data for this usability test
report. Following is a list of the appendices provided:
1: Sample Recruiting screener
2: Participant demographics
3: Non-Disclosure Agreement (NDA) and Informed Consent Form
4: Example Moderator's Guide
5: System Usability Scale Questionnaire
6: Incentive receipt and acknowledgment form It is important to note that these
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 21 of 46
Appendix 1: SAMPLE RECRUITING SCREENER
The purpose of a screener to ensure that the participants selected represent
the target user population as closely as possible. (Portions of this sample
screener are taken from www.usability.gov/templates/index.html#Usability and
adapted for use.)
Recruiting Script for Recruiting Firm
Hello, my name is _______________, calling from [Insert name of recruiting
firm]. We are recruiting individuals to participate in a usability study for an
electronic health record. We would like to ask you a few questions to see if
you qualify and if would like to participate. This should only take a few minutes
of your time. This is strictly for research purposes. If you are interested and
qualify for the study, you will be paid to participate. Can I ask you a few
questions?
Customize this by dropping or adding questions so that it reflects your EHR's
primary audience
1. [If not obvious] Are you male or female? [Recruit a mix of participants]
2. Have you participated in a focus group or usability test in the past xx months? [If yes, Terminate]
3. Do you, or does anyone in your home, work in marketing research, usability research, web design [If yes, Terminate]
4. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record software or consulting company? [If yes, Terminate]
5. Which of the following best describes your age? [23 to 39; 40 to 59; 60 and older] [Recruit Mix]
6. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-American, Latino/a or Hispanic, etc.]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 22 of 46
7. Do you require any assistive technologies to use a computer? [if so, please describe] Professional Demographics Customize this to reflect your EHR's primary audience
8. What is your current position and title? (Must be healthcare provider)
□ RN: Specialty _______________________
□ Physician: Specialty _______________________
□ Resident: Specialty _______________________
□ Medical Assistant _______________________
□ Administrative Staff
□ Other [Terminate]
9. How long have you held this position?
10. Describe your work location (or affiliation) and environment? (Recruit according to the intended users of the application. e.g., private practice, health system, government clinic, etc.)
11. Which of the following describes your highest level of education? [e.g., high school graduate/GED, some college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)
12. Besides reading email, what professional activities do you do on the computer? [e.g., access EHR, research; reading news; shopping/banking; digital pictures; programming/word processing, etc.] [If no computer use at all, Terminate]
13. About how many hours per week do you spend on the computer? [Recruit according to the demographics of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]
14. What computer platform do you usually use? [e.g., Mac, Windows, etc.]
15. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]
16. In the last month, how often have you used an electronic health record?
17. How many years have you used an electronic health record?
18. How many EHRs do you use or are you familiar with?
19. How does your work environment handle patient records? [Recruit according to the demographics of the intended users]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 23 of 46
□ On paper
□ Some paper, some electronic
□ All electronic
Contact Information If the person matches your qualifications, ask
Those are all the questions I have for you. Your background matches the
people we're looking for. For your participation, you will be paid $300. Would
you be able to participate on [date, time]? [If so collect contact information]
May I get your contact information?
■ Name of participant:
■ Address:
■ City, State, Zip:
■ Daytime phone number:
■ Evening phone number:
■ Alternate [cell] phone number:
■ Email address:
Before your session starts, we will ask you to sign a release form allowing us to
record your session. The record will only be used internally for further study if
needed. Will you consent to be digitally recorded? This study will take place at
3330 Keller Springs Road, Suite 201, Carrollton, TX 75006. I will confirm your
appointment a couple of days before your session and provide you with
directions to our office. What time is the best time to reach you?
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 24 of 46
Appendix 2: PARTICIPANT DEMOGRAPHICS
Following is a high-level overview of the participants in this study.
Gender
Men: 0 Women: 6 Total (participants): 6
Occupation/Role
RN/BSN: 0 Physician: 2 Admin Staff: 4 Total (participants): 6
Years of Experience
Facility Use of EHR All paper: 1 Some paper, some electronic: 5 All electronic: 0 Total (participants): 6
The full break down is included in the report above.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 25 of 46
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM
Non-Disclosure Agreement THIS AGREEMENT is entered into as of _________________, 2013, between ________________ ("the Participant") and the testing organization Aprima Medical Software located at 3330 Keller Springs Rd. Suite 201, Carrollton, TX 75006. The Participant acknowledges his or her voluntary participation in today's usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by Test Company, or otherwise acquired by the Participant, in the course of today's study. By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts. Any information the Participant acquires relating to this product during this study is confidential and proprietary to Test Company and is being disclosed solely for the purposes of the Participant's participation in today's usability study. By signing this form the Participant acknowledges that s/he will receive monetary compensation for feedback and will not disclose this confidential information obtained today to anyone else or any other organizations. Participant's printed name: ___________________________________________________ Signature: __________________________________________________________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 26 of 46
Informed Consent Test Company would like to thank you for participating in this study. The purpose of this study is to evaluate an electronic health records system. If you decide to participate, you will be asked to perform several tasks using the prototype and give your feedback. The study will last about 30 minutes. At the conclusion of the test, you will be compensated for your time. Agreement I understand and agree that as a voluntary participant in the present study conducted by Test Company I am free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study conducted and videotaped by the Test Company. I understand and consent to the use and release of the videotape by Test Company. I understand that the information and videotape is for research purposes only and that my name and image will not be used for any purpose other than research. I relinquish any rights to the videotape and understand the videotape may be copied and used by Test Company without further permission. I understand and agree that the purpose of this study is to make software applications more useful and usable in the future. I understand and agree that the data collected from this study may be shared with outside of Test Company and Test Company's client. I understand and agree that data confidentiality is assured, because only de- identified data - i.e., identification numbers not names - will be used in analysis and reporting of the results. I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can leave at any time. Please check one of the following: □ YES, I have read the above statement and agree to be a participant. □ NO, I choose not to participate in this study. Signature: _________________________________ Date: _______________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 27 of 46
Appendix 4: EXAMPLE MODERATOR'S GUIDE
Only four tasks are presented here for illustration. EHRUT Usability Test Moderator's Guide Administrator _____________________
Data Logger _____________________
Date _________________ Time _____________________
Participant # _________
Location _____________________
Prior to testing
■ Confirm schedule with Participants ■ Ensure EHRUT lab environment is running properly ■ Ensure Consent forms are complete for all participants ■ Ensure lab and data recording equipment is running properly
Prior to each participant: ■ Reset application ■ Restore application to base state using script restoredb.bat ■ Start session recordings with GTM Recording
Prior to each task: ■ Reset application to starting point for next task
After each participant: ■ End session recordings with tool ■ Fill out all paper work and acknowledgement form.
After all testing ■ Scan all forms and shred originals ■ Back up all video and data files
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 28 of 46
Orientation (5 minutes) Thank you for participating in this study. Our session today will last 2 hours today. During that time you will take a look at an electronic health record system. I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with anything to do with the system itself. Please save your detailed comments until the end of a task or the end of the session as a whole when we can discuss freely. I did not have any involvement in its creation, so please be honest with your opinions. The product you will be using today is describe the state of the application, i.e., production version, early prototype, etc. Some of the data may not make sense as it is placeholder data. We are recording the audio and screenshots of our session today. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Do you have any questions or concerns? Preliminary Questions (5 minutes) What is your job title / appointment? How long have you been working in this role? What are some of your main responsibilities? Tell me about your experience with electronic health records.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 29 of 46
Preview: First Impressions (6 Minutes) This is the application you will be working with. Have you heard of it? Yes ___ No ___ If so, tell me what you know about it. Show test participant the EHRUT. Please don't click on anything just yet. What do you notice? What are you able to do here? Please be specific. Notes / Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 30 of 46
Task 1: Create an Order – Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC The patient you are examining 5’ 10” 285 lbs. male needs a differential diagnosis of 034.0 Streptococcal Sore Throat and needs a 87880 rapid strep to confirm diagnosis. Enter Vitals, Dx and SP into the system Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, enter 5 feet 10 inches and 285 lbs 2. go to the DX tab and Enter the diagnosis 034.0 Streptococcal Sore Throat 3. Note Alert for Obesity and Hypertension in CDS slider 4. Still inside FNC, go to the SP tab 5. Enter procedure code 87880 Strep A Assay 6. Still inside FNC, click the Close button to close the note, save the note as incomplete 7. Click okay.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 31 of 46
Task 2: Utilize an existing Clinical Decision Support Alert – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You run are in consultation with a patient and his parent who is 71 years old. You want to quickly see if there are any Clinical Support Alerts for the parent. Go to the Appointment to review CDS Alerts and look at the source information for the alert. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
Open Patient Norman Wood
Open Update Appointment from Envelope
Open Clinical Decision Support Tab
Note Fall Risk Rule on Slider
Click on the Hyperlink for Fall Risk Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 32 of 46
Task 3: Creation of a Procedure Task Message to the Nurses - Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC
A patient you saw earlier that requested an order for a test at a different lab wants the lab found it was not covered by his insurance. He wants it to be performed in your office. You need to order a rapid strep test (87880) for the patient, as he is on his way back into the office.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open the scheduler 2. Find the appointment for Ed Wood 3. Click the Blue Folder (Open/Create note) 4. Click New->Procedure Task Message
a. Enter the Nurses b. Click Okay
5. Click to close the note a. Click Okay
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 33 of 46
Task 4: Approve a Lab result (5 Minutes) Take the participant to the starting point for the task. - DESKTOP
You have a lab result request message in your inbox to process. You need to review and approve the request.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations Approval of Lab Results
1. Locate the Lab Result Approval message in the Desktop Message pane 2. Click on the Time and Date Hyperlink or Plus (+) sign next to the message 3. Click on Process Message 4. From the Modify Message window 5. After review of the results, click the Approved All button
a. Your user (provider) name will auto populate along with the current date 6. Click Complete
a. This will simultaneously set the approved lab results and complete the message Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 34 of 46
Task 5: Document Drug Allergy in History – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are examining reports to you that she has been taking no meds because she has an anaphylactic reaction to Penicillins. Please document the allergy in History Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Drug Allergy
6. In the Allergen column, select Penicillins.
7. In the Reaction column, select anaphylaxis.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 35 of 46
Task 6: Drug Allergy Alert in Rx Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Need to prescribe Keflex to Mary Woods chart 1 tablet per day by mouth for 10 days Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Rx Tab
6. In the Drug field, enter Keflex
7. Click the find control
8. Select Keflex 500 mg
9. Pop up: Receive Drug Screening Warning: ALLERGY ALERT: The use of Keflex cross-sensitivity reaction based on a reported history of allergy to PENICILLINS. Do you want to continue? Click Yes
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 36 of 46
Task 7: Drug Drug Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP An agitated patient wants you to prescribe Prozac immediately. 20mg 1 time per day by mouth. Please prescribe the medication. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Find Mary Wood that has an appointment scheduled.
2. Click the Open/Create Note icon
3. FNC
4. Click Rx Tab
5. In the Drug field, enter Prozac
6. Click the find control
7. Select Prozac 20mg tablet
8. Pop up: Receive Drug Screening Warning: Duplicate Therapy: The use of Prozac is a duplicate therapy with Zoloft 50 mg. Do you want to continue?
a. Click No
9. Print -> Save and print Button
a. Print Dialog
10. Click ok to save and exit the note
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 37 of 46
Task 8: Medication List in OPS – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You are about to walk into an exam room for a med check on a patient. Go to One Page Summary and review the patient’s drug list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Go to Patient Demographics 2. Click Patient 3. Click One Page Summary 4. Click hyperlink in index for Current Medication 5. Verify the data is displayed correctly
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 38 of 46
Task 9: Medication List in Hx Tab with Filter – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
Previously very ill patient with a very large medication history is asking about her current medications. You want to compare the active/inactive medications with the maintenance medication list (which are two different filters in the system Go to Hx Tab and change the filter to show only Active Medications and them maintenance medications.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, go to the HX tab 2. Click on the Medication History 3. Review the items on the list 4. Click the search icon to change the filter 5. Change the filter to maintenance 6. Review the Maintenance Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 39 of 46
Task 10: Rx Refill Complete (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You have a few minutes to complete your drug refill request messages Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Messages
2. Locate message
3. Click on patient’s blue envelope.
4. Select Process Rx Refill
5. Medication History screen
6. Select the drug by clicking the Refill button.
7. Sig Writer screen
a. Verify information in sig
8. Click Ok
a. Return to the Modify Message screen
9. Click Complete.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 40 of 46
Task 11: import Allergies from external list - Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Allergy list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. Go to attachments – Allergies - CCD 3. Click the import allergies icon 4. Sulfasalazine -> Add 5. Penicillin V -> Merge/Add 6. Carbamazepine -> ignore/remove 7. Click Reconcile 8. Go to Hx
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 41 of 46
Task 12: Import Medications from external list – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Medication list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. View the medication CCD 3. Click the import medications icon 4. Simvastatin -> Add 5. Lorazepam -> Add 6. Lisinopril -> Add/Merge 7. Click reconcile 8. Go to patient Hx – Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 42 of 46
Task 13: Verify import in Allergy list and Medication List – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Please validate that the import of Allergy list and Medication list import has the expected information Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Note the CDS rule for the combination of the new allergy and medication
6. Click Drug Allergy
a. Verify the information is correct from the import
7. Click on the Medication History
a. Verify the information is correct from the import
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 43 of 46
Final Questions (10 Minutes) What was your overall impression of this system? What aspects of the system did you like most? What aspects of the system did you like least? Were there any features that you were surprised to see? What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? Compare this system to other systems you have used. Would you recommend this system to your colleagues? Administer the SUS
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 44 of 46
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
Strongly
disagree Strongly
Agree
1. I think that I would like to use this system frequently
1 2 3 4 5
2. I found the system unnecessarily complex
1 2 3 4 5
3. I thought the system was easy to use
1 2 3 4 5
4. I think that I would need the support of a technical person to be able to use this system
1 2 3 4 5
5. I found the various functions in this system were well integrated
1 2 3 4 5
6. I thought there was too much inconsistency in this system
1 2 3 4 5
7. I would imagine that most people would learn to use this system very quickly
1 2 3 4 5
8. I found the system very cumbersome to use
1 2 3 4 5
9. I felt very confident using the system
1 2 3 4 5
10. I needed to learn a lot of things before I could get going with this system
1 2 3 4 5
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 45 of 46
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM
Acknowledgement of Receipt I hereby acknowledge receipt of $ _____ for my participation in a research study run by Aprima Medical Software. Printed Name: _____________________________________________________________ Address: __________________________________________________________________ __________________________________________________________________________ Signature: _______________________________________________ Date: ____________ Usability Researcher: ________________________________________________________ Signature of Usability Researcher: ____________________________ Date: ______________ Witness: ___________________________________________________________________ Witness Signature: ________________________________________ Date: ______________
EHR USABILITY TEST REPORT
OF
APRIMA 2014
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
Aprima 2014 Date of Usability Test: 05/21/13-05/23/13 Date of Report: 05/27/13 Report Prepared By: Aprima Medical Software Project Management Cornell Hardman (214) 466-8107 Aprima Medical Software 3330 Keller Springs Road Suite 201 Carrollton, TX 75006
Table of Contents EXECUTIVE SUMMARY ................................................................................................................. 3
Major findings ............................................................................................................................... 5
Areas for improvement ................................................................................................................. 5
INTRODUCTION ............................................................................................................................. 6
METHOD ......................................................................................................................................... 6
PARTICIPANTS ........................................................................................................................... 6
STUDY DESIGN .......................................................................................................................... 8
TASKS ......................................................................................................................................... 9
PROCEDURES ............................................................................................................................ 9
TEST LOCATION ...................................................................................................................... 10
TEST ENVIRONMENT .............................................................................................................. 10
TEST FORMS AND TOOLS ...................................................................................................... 11
PARTICIPANT INSTRUCTIONS ............................................................................................... 12
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 2 of 46
USABILITY METRICS ............................................................................................................... 13
DATA SCORING ........................................................................................................................ 13
RESULTS ...................................................................................................................................... 16
DATA ANALYSIS AND REPORTING ........................................................................................ 16
DISCUSSION OF THE FINDINGS ............................................................................................ 17
EFFECTIVENESS ..................................................................................................................... 17
EFFICIENCY .............................................................................................................................. 17
SATISFACTION ......................................................................................................................... 18
MAJOR FINDINGS .................................................................................................................... 18
AREAS FOR IMPROVEMENT .................................................................................................. 18
APPENDICES ................................................................................................................................ 20
Appendix 1: SAMPLE RECRUITING SCREENER .................................................................... 21
Appendix 2: PARTICIPANT DEMOGRAPHICS ........................................................................ 24
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ........... 25
Appendix 4: EXAMPLE MODERATOR'S GUIDE ...................................................................... 27
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................. 44
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM ................................. 45
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 3 of 46
EXECUTIVE SUMMARY
A usability test of Aprima 2014 ambulatory EHR was conducted on the dates of
May 21-23 of 2013 in The Aprima Medical Software corporate office by Aprima
Medical Software Project Management team. The purpose of this test was to
test and validate the usability of the current user interface, and provide
evidence of usability in the EHR Under Test (EHRUT). During the usability
test, 2 healthcare providers and 4 other knowledgeable clinic staff matching the
target demographic criteria served as participants and used the EHRUT in
simulated, but representative tasks. This study collected performance data on
13 tasks including 3 tasks specifically designed to test Drug-Drug and Drug
Allergy Interaction that are typically conducted on an EHR:
Task 1 – Document Drug Allergy in Medication History
Task 2 – Drug-Drug interaction in Rx Tab
Task 3 – Complete a Drug refill request
During the 2-hour one-on-one usability test, each participant was greeted by
the administrator and asked to review and sign an informed consent/release
form (included in Appendix 3); they were instructed that they could withdraw at
any time. Participants had a mix of experience with the EHR. The
administrator introduced the test, and instructed participants to complete a
series of tasks (given one at a time) using the EHRUT. During the testing, the
administrator timed the test and, along with the data logger(s) recorded user
performance data on paper and electronically. The administrator did not give
the participant assistance in how to complete the task.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 4 of 46
Participant screens were recorded for subsequent analysis.
The following types of data were collected for each participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations Participant's satisfaction ratings of the system
All participant data was de-identified - no correspondence could be made from
the identity of the participant to the data collected. Following the conclusion of
the testing, participants were asked to complete a post-test questionnaire and
were compensated with $300 for their time. Various recommended metrics, in
accordance with the examples set forth in the NIST Guide to the Processes
Approach for Improving the Usability of Electronic Health Records, were used
to evaluate the usability of the EHRUT. Following is a summary of the
performance and rating data collected on the EHRUT.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations (Observed / Optimal)
Mean(SD)
Deviations (Observed /Optimal)
Mean (SD)
Mean (SD)
1. Task 1 6 100% (0.00)
0.5 (57/54) 50 (25)
0.3 (0.5) 4.8 (0.4)
2. Task 2 6 100% (0.0)
0.7 (58/54) 61 (22)
0.0 (0.0) 4.2 (0.9)
3. Task 3 6 100% (0.0)
0.3 (62/60) 46 (14)
0.0 (0.0) 4.7 (0.5)
The results from the System Usability Scale scored the subjective satisfaction
with the system based on performance with these tasks to be: 83
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 5 of 46
In addition to the performance data, the following qualitative observations were
made:
Major findings
The information presented and the way in which it was presented was
well received by the test users.
Drug refill request was not as efficient as anticipated. Users spent time
looking at the form while attempting to figure out what medicine was
being refilled
Users after the long pause consistently did either the optimal path or a
slight variation that took an extra 8-15 seconds, but did not cause any
errors.
No major rework is required for the Drug-Drug interaction and Drug-
Allergy interaction modules.
Areas for improvement
The Refill screen needs to be optimized to direct users to the correct
actions more quickly.
When a user goes off the correct path, he will often take a very long
time and many more errors before finding the correct path again.
Aprima has many ways to do the same operation. This leads to
confusion for users. Aprima should limit access to feature models to
simplify the user paradigm.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 6 of 46
INTRODUCTION
The EHRUT tested for this study was Aprima 2014 ambulatory EHR. Designed
to present medical information to healthcare providers in ambulatory medical
offices, the EHRUT consists of a user interface presented on a computer
display that stores and retrieves data from a database server through an
application server. The usability testing attempted to represent realistic
exercises and conditions. The purpose of this study was to test and validate
the usability of the current user interface, and provide evidence of usability in
the EHR Under Test (EHRUT). To this end, measures of effectiveness,
efficiency and user satisfaction, such as ease of access, number of mouse
clicks, time on task, time to repeat task, time to first entry and number of errors,
were captured during the usability testing.
METHOD
PARTICIPANTS
A total of 6 participants were tested on the EHRUT. Participants in the test
were physicians, nurses and other knowledgeable office staff. Participants
were recruited by Health Care Strategies and were compensated $300 for their
time. In addition, participants had no direct connection to the development of
or organization producing the EHRUT(s). Participants were not from the
testing or supplier organization. Participants were given the opportunity to
have an abbreviated orientation and a minimal level of training compared to the
amount the actual end users would have received. For the test purposes, end-
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 7 of 46
user characteristics were identified and translated into a recruitment screener
used to solicit potential participants; an example of a screener is provided in
Appendix 1.
Recruited participants had a mix of backgrounds and demographic
characteristics conforming to the recruitment screener. The following is a table
of participants by characteristics, including demographics, professional
experience, computing experience and user needs for assistive technology.
Participant names were replaced with Participant IDs so that an individual's
data cannot be tied back to individual identities.
Part ID
Gender Age Education Occupation /Role
Professional Experience
Computer Exp.
Product Exp.
Assistive Technology
Needs
1 142101 F 25 College Degree
Medical
Assistant 5 9 0/4
2 142102 F 42 College Degree
Medical
Assistant 3 13 0/1 3 142201 F 65 MD/PhD/DO Physician 38 4 0/0
4 142202 F 39 College Degree
Medical
Assistant 16 16 0/3
5 142301 F 31
Masters
Degree
Physician
Assistant 9 19 0/6
6 142302 F 30 College Degree
Medical
Assistant 8 13 0/7
9 participants (matching the demographics in the section on Participants) were
recruited and 6 participated in the usability test. 3 participants failed to show
for the study.
Participants were scheduled for 2 hour session with at least 1 hour in between
each session for debrief by the administrator(s) and data logger(s), and to reset
systems to proper test conditions. A spreadsheet was used to keep track of
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 8 of 46
the participant schedule, and included each participant's demographic
characteristics as provided by the recruiting firm.
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application
performed well - that is, effectively, efficiently, and with satisfaction - and areas
where the application failed to meet the needs of the participants. The data
from this test may serve as a baseline for future tests with an updated version
of the same EHR and/or comparison with other EHRs provided the same tasks
are used. In short, this testing serves as both a means to record or benchmark
current usability, but also to identify areas where improvements must be made.
During the usability test, participants interacted with a single EHR. Each
participant used the system in the same location, and was provided with the
same instructions. The system was evaluated for effectiveness, efficiency and
satisfaction as defined by measures collected and analyzed for each
participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations (comments) Participant's satisfaction ratings of the system
Additional information about the various measures can be found in
Section 3.9 on Usability Metrics.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 9 of 46
TASKS
A number of tasks were constructed that would be realistic and representative
of the kinds of activities a user might do with this EHR, including:
Task 1 – Document Drug Allergy in Medication History
Task 2 – Drug-Drug interaction in Rx Tab
Task 3 – Complete a Drug refill request
Tasks were selected based on their frequency of use, criticality of function, and
those that may be most troublesome for users. Tasks should always be
constructed in light of the study objectives.
PROCEDURES
Upon arrival, participants were greeted; their identity was verified and matched
with a name on the participant schedule. Participants were then assigned a
participant ID. Each participant reviewed and signed an informed consent and
release form (See Appendix 3). A representative from the test team witnessed
the participant's signature. To ensure that the test ran smoothly, two staff
members participated in this test, the usability administrator and the data
logger. The usability testing staff conducting the test was experienced usability
practitioners with 3 years of usability experience at GE Healthcare and
additional experience at other facilities. The administrator moderated the
session including administering instructions and tasks. The administrator also
monitored task times, obtained post-task rating data, and ensured notes on
participant comments were recorded. A second person served as the data
logger and took notes on task success, path deviations, number and type of
errors, and comments. Participants were instructed to perform the tasks (see
specific instructions below):
As quickly as possible making as few errors and deviations as possible.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 10 of 46
Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use.
Without using a think aloud technique.
For each task, the participants were given a written copy of the task. Task
timing began once the administrator finished reading the question. The task
time was stopped once the participant indicated they had successfully
completed the task. Scoring is discussed below in Section 3.9. Following the
session, the administrator gave the participant the post-test questionnaire (e.g.,
the System Usability Scale, see Appendix 5), compensated him or her for their
time, and thanked each individual for their participation.
Participants' demographic information, task success rate, time on task, errors,
deviations, verbal responses, and post-test questionnaire were recorded into a
spreadsheet.
Participants were thanked for their time and compensated. Participants signed
a receipt and acknowledgement form (See Appendix 6) indicating that they had
received the compensation.
TEST LOCATION
The test facility included a waiting area and a quiet testing room with a table,
computer for the participant, and recording computer for the administrator.
Only the participant and administrator and the data logger were in the test
room. There were no observers. To ensure that the environment was
comfortable for users, noise levels were kept to a minimum with the ambient
temperature within a normal range. All of the safety instruction and evacuation
procedures were valid, in place, and visible to the participants.
TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or facility. In this
instance, the testing was conducted in the Aprima Medical Software corporate
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 11 of 46
headquarters in the main training room. For testing, the computer used a
Fujitsu T902 running Microsoft Windows 8. The participants used whatever
input method he or she was most comfortable with from among mouse,
keyboard, touchpad, stylus or voice input when interacting with the EHRUT.
The EHRUT used a 12.7-inch diagonal display running 1280x720 pixels at 32
bit color. The application was set up by the administrator according to the
vendor's documentation. The application itself was running on a Dell 620
server running Windows Server 2008 using a test database on a LAN
connection. Technically, the system performance (i.e., response time) was
representative to what actual users would experience in a field implementation.
Additionally, participants were instructed not to change any of the default
system settings (such as control of font size).
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used,
including:
1. Informed Consent
2. Moderator's Guide
3. Post-test Questionnaire
4. Incentive Receipt and Acknowledgment Form
Examples of these documents can be found in Appendices 3-6 respectively.
The Moderator's Guide was devised so as to be able to capture required data.
The participant's interaction with the EHRUT was captured and recorded
digitally with screen capture software running on the test machine.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 12 of 46
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each participant:
Thank you for participating in this study. Your input is very important.
Our session today will last about 30 minutes. During that time you will
use an instance of an electronic health record. I will ask you to
complete a few tasks using this system and answer some questions.
You should complete the tasks as quickly as possible making as few
errors as possible. Please try to complete the tasks on your own
following the instructions very closely. Please note that we are not
testing you we are testing the system, therefore if you have difficulty all
this means is that something needs to be improved in the system. I will
be here in case you need specific help, but I am not able to instruct you
or provide help in how to use the application. Overall, we are interested
in how easy (or how difficult) this system is to use, what in it would be
useful to you, and how we could improve it. I did not have any
involvement in its creation, so please be honest with your opinions.
There are a variety of tools that record screens and transmit those
recordings across a local area network for remote observations. All of
the information that you provide will be kept confidential and your name
will not be associated with your comments at any time. Should you feel
it necessary you are able to withdraw at any time during the testing.
Following the procedural instructions, participants were shown the EHR and as
their first task, were given time (10 minutes) to explore the system and make
comments. Once this task was complete, the administrator gave the following
instructions:
For each task, I will read the description to you and say "Begin." At that
point, please perform the task and say "Done" once you believe you
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 13 of 46
have successfully completed the task. I would like to request that you
not talk aloud or verbalize while you are doing the tasks. I will ask you
your impressions about the task once you are done.
Participants were then given 13 tasks to complete. Tasks are listed in the
moderator's guide in the Appendix.
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the
Usability of Electronic Health Records, EHRs should support a process that
provides a high level of usability for all users. The goal is for users to interact
with the system effectively, efficiently, and with an acceptable level of
satisfaction. To this end, metrics for effectiveness, efficiency and user
satisfaction were captured during the usability testing. The goals of the test
were to assess:
1. Effectiveness of EHRUT by measuring participant success rates and errors
2. Efficiency of EHRUT by measuring the average task time and path
deviations
3. Satisfaction with EHRUT by measuring ease of use ratings
DATA SCORING
The following table (Table 1) details how tasks were scored, errors evaluated,
and the time data analyzed.
Measures Rational and Scoring
Effectiveness: Task Success
A task was counted as a "Success" if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 14 of 46
Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator's Guide must be operationally defined by taking multiple measures of optimal performance and multiplying by some factor that allows some time buffer because the participants are presumably not trained to expert performance. Scoring used a factor of 2. Thus, if expert, optimal performance on a task was 80 seconds then allotted task time performance was 80 * 2 = 160 seconds. This ratio should be aggregated across tasks and reported with mean and variance scores.
Effectiveness: Task Failures
If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a "Failure." No task times were taken for errors. The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. This should also be expressed as the mean number of failed tasks per participant. On a qualitative level, an enumeration of errors and error types were collected.
Efficiency: Task Deviations
The participant's path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps in order to provide a ratio of path deviation. It is strongly recommended that task deviations be reported. Optimal paths (i.e., procedural steps) should be recorded when constructing tasks.
Efficiency: Task Time
Each task was timed from when the administrator said "Begin" until the participant said, "Done." If he or she failed to say "Done," the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.
Satisfaction: Participant's subjective impression of the ease of use of the
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 15 of 46
Task Rating application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate "Overall, this task was:" on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants' confidence in and likeability of the EHRUT overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, "I think I would like to use this system frequently," "I thought the system was easy to use," and "I would imagine that most people would learn to use this system very quickly." See full System Usability Score questionnaire in Appendix 5.
Table 1. Details of how observed data were scored.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 16 of 46
RESULTS
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods
specified in the Usability Metrics section above. Participants who failed to
follow session and task instructions had their data excluded from the analyses.
In this test cycle, there were no data excluded.
The usability testing results for the EHRUT are detailed below. The results
should be seen in light of the objectives and goals outlined in Section 3.2 Study
Design. The data should yield actionable results that, if corrected, yield
material, positive impact on user performance.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations(Observed / Optimal)
Mean (SD)
Deviations(Observed/Optimal)
Mean (SD)
Mean (SD)
1. Task 1 6 100% (0.00)
0.5 (57/54)
50 (25) 0.3 (0.5)
4.8 (0.4)
2. Task 2 6 100% (0.0)
0.7 (58/54)
61 (22) 0.0 (0.0)
4.2 (0.9)
3. Task 3 6 100% (0.0)
0.3 (62/60)
46 (14) 0.0 (0.0)
4.7 (0.5)
The results from the SUS (System Usability Scale) scored the subjective
satisfaction with the system based on performance with these tasks to be: 83.
Broadly interpreted, scores under 60 represent systems with poor usability;
scores over 80 would be considered above average
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 17 of 46
DISCUSSION OF THE FINDINGS
The information presented and the way in which it was presented was well
received by the test users.
Drug refill request was not as efficient as anticipated. Users spent time looking
at the form while attempting to figure out what medicine was being refilled
Users after the long pause consistently did either the optimal path or a slight
variation that took an extra 8-15 seconds, but did not cause any errors.
No major rework is required for the Drug-Drug interaction and Drug-Allergy
interaction modules.
EFFECTIVENESS
The users were able to do what was expected with the product. Many times the
users went slightly off the optimal the path but did not lose their way. Users
with a small amount of time on the system improved dramatically. All users
rated the product either agree or strongly agree with regard to “I thought the
system was easy to use”. All users also rated “I found the system
unnecessarily complex” either disagree or strongly disagree.
EFFICIENCY
The number of path deviations on some tasks was above 2.0 this is large
number. For most tasks the number was at or below 1.0. Users were in almost
all cases able to quickly come back to the optimal path and complete the task.
This is true for the tasks in which the path deviations where the highest as well.
Variance on the timings show that the untrained users are not efficient using
Aprima. User learning is rapid; by task 5 the users were able to be more
efficient with task times being one half or less compared to the first 4 tasks.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 18 of 46
SATISFACTION
User satisfaction was quite high. Overall user rating was 83.
Question Agree or Strongly Agree
Disagree or Strongly Disagree
I think that I would like to use this system frequently
83% 0%
I found the system unnecessarily complex
0% 100%
I thought the system was easy to use
100% 0%
I think that I would need the support of a technical person to be able to use this system
17% 67%
I found the various functions in this system were well integrated
83% 0%
I thought there was too much inconsistency in this system
0% 100%
I would imagine that most people would learn to use this system very quickly
100% 0%
I found the system very cumbersome to use
33% 67%
I felt very confident using the system
100% 0%
I needed to learn a lot of things before I could get going with this system
0% 100%
“I found the system cumbersome to use” seems to be a polarizing question. In
future user testing this will have to be explored more, as we did not find this to
be reflected in the user comments and was not discovered until the data
analysis phase.
MAJOR FINDINGS
The information presented and the way in which it was presented was
well received by the test users.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 19 of 46
Drug refill request was not as efficient as anticipated. Users spent time
looking at the form while attempting to figure out what medicine was
being refilled
Users after the long pause consistently did either the optimal path or a
slight variation that took an extra 8-15 seconds, but did not cause any
errors.
No major rework is required for the Drug-Drug interaction and Drug-
Allergy interaction modules.
AREAS FOR IMPROVEMENT
The Refill screen needs to be optimized to direct users to the correct
actions more quickly.
When a user goes off the correct path, he will often take a very long
time and many more errors before finding the correct path again.
Aprima has many ways to do the same operation. This leads to
confusion for users. Aprima should limit access to feature models to
simplify the user paradigm.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 20 of 46
APPENDICES
The following appendices include supplemental data for this usability test
report. Following is a list of the appendices provided:
1: Sample Recruiting screener
2: Participant demographics
3: Non-Disclosure Agreement (NDA) and Informed Consent Form
4: Example Moderator's Guide
5: System Usability Scale Questionnaire
6: Incentive receipt and acknowledgment form It is important to note that these
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 21 of 46
Appendix 1: SAMPLE RECRUITING SCREENER
The purpose of a screener to ensure that the participants selected represent
the target user population as closely as possible. (Portions of this sample
screener are taken from www.usability.gov/templates/index.html#Usability and
adapted for use.)
Recruiting Script for Recruiting Firm
Hello, my name is _______________, calling from [Insert name of recruiting
firm]. We are recruiting individuals to participate in a usability study for an
electronic health record. We would like to ask you a few questions to see if
you qualify and if would like to participate. This should only take a few minutes
of your time. This is strictly for research purposes. If you are interested and
qualify for the study, you will be paid to participate. Can I ask you a few
questions?
Customize this by dropping or adding questions so that it reflects your EHR's
primary audience
1. [If not obvious] Are you male or female? [Recruit a mix of participants]
2. Have you participated in a focus group or usability test in the past xx months? [If yes, Terminate]
3. Do you, or does anyone in your home, work in marketing research, usability research, web design [If yes, Terminate]
4. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record software or consulting company? [If yes, Terminate]
5. Which of the following best describes your age? [23 to 39; 40 to 59; 60 and older] [Recruit Mix]
6. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-American, Latino/a or Hispanic, etc.]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 22 of 46
7. Do you require any assistive technologies to use a computer? [if so, please describe] Professional Demographics Customize this to reflect your EHR's primary audience
8. What is your current position and title? (Must be healthcare provider)
□ RN: Specialty _______________________
□ Physician: Specialty _______________________
□ Resident: Specialty _______________________
□ Medical Assistant _______________________
□ Administrative Staff
□ Other [Terminate]
9. How long have you held this position?
10. Describe your work location (or affiliation) and environment? (Recruit according to the intended users of the application. e.g., private practice, health system, government clinic, etc.)
11. Which of the following describes your highest level of education? [e.g., high school graduate/GED, some college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)
12. Besides reading email, what professional activities do you do on the computer? [e.g., access EHR, research; reading news; shopping/banking; digital pictures; programming/word processing, etc.] [If no computer use at all, Terminate]
13. About how many hours per week do you spend on the computer? [Recruit according to the demographics of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]
14. What computer platform do you usually use? [e.g., Mac, Windows, etc.]
15. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]
16. In the last month, how often have you used an electronic health record?
17. How many years have you used an electronic health record?
18. How many EHRs do you use or are you familiar with?
19. How does your work environment handle patient records? [Recruit according to the demographics of the intended users]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 23 of 46
□ On paper
□ Some paper, some electronic
□ All electronic
Contact Information If the person matches your qualifications, ask
Those are all the questions I have for you. Your background matches the
people we're looking for. For your participation, you will be paid $300. Would
you be able to participate on [date, time]? [If so collect contact information]
May I get your contact information?
■ Name of participant:
■ Address:
■ City, State, Zip:
■ Daytime phone number:
■ Evening phone number:
■ Alternate [cell] phone number:
■ Email address:
Before your session starts, we will ask you to sign a release form allowing us to
record your session. The record will only be used internally for further study if
needed. Will you consent to be digitally recorded? This study will take place at
3330 Keller Springs Road, Suite 201, Carrollton, TX 75006. I will confirm your
appointment a couple of days before your session and provide you with
directions to our office. What time is the best time to reach you?
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 24 of 46
Appendix 2: PARTICIPANT DEMOGRAPHICS
Following is a high-level overview of the participants in this study.
Gender
Men: 0 Women: 6 Total (participants): 6
Occupation/Role
RN/BSN: 0 Physician: 2 Admin Staff: 4 Total (participants): 6
Years of Experience
Facility Use of EHR All paper: 1 Some paper, some electronic: 5 All electronic: 0 Total (participants): 6
The full break down is included in the report above.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 25 of 46
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM
Non-Disclosure Agreement THIS AGREEMENT is entered into as of _________________, 2013, between ________________ ("the Participant") and the testing organization Aprima Medical Software located at 3330 Keller Springs Rd. Suite 201, Carrollton, TX 75006. The Participant acknowledges his or her voluntary participation in today's usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by Test Company, or otherwise acquired by the Participant, in the course of today's study. By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts. Any information the Participant acquires relating to this product during this study is confidential and proprietary to Test Company and is being disclosed solely for the purposes of the Participant's participation in today's usability study. By signing this form the Participant acknowledges that s/he will receive monetary compensation for feedback and will not disclose this confidential information obtained today to anyone else or any other organizations. Participant's printed name: ___________________________________________________ Signature: __________________________________________________________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 26 of 46
Informed Consent Test Company would like to thank you for participating in this study. The purpose of this study is to evaluate an electronic health records system. If you decide to participate, you will be asked to perform several tasks using the prototype and give your feedback. The study will last about 30 minutes. At the conclusion of the test, you will be compensated for your time. Agreement I understand and agree that as a voluntary participant in the present study conducted by Test Company I am free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study conducted and videotaped by the Test Company. I understand and consent to the use and release of the videotape by Test Company. I understand that the information and videotape is for research purposes only and that my name and image will not be used for any purpose other than research. I relinquish any rights to the videotape and understand the videotape may be copied and used by Test Company without further permission. I understand and agree that the purpose of this study is to make software applications more useful and usable in the future. I understand and agree that the data collected from this study may be shared with outside of Test Company and Test Company's client. I understand and agree that data confidentiality is assured, because only de- identified data - i.e., identification numbers not names - will be used in analysis and reporting of the results. I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can leave at any time. Please check one of the following: □ YES, I have read the above statement and agree to be a participant. □ NO, I choose not to participate in this study. Signature: _________________________________ Date: _______________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 27 of 46
Appendix 4: EXAMPLE MODERATOR'S GUIDE
Only four tasks are presented here for illustration. EHRUT Usability Test Moderator's Guide Administrator _____________________
Data Logger _____________________
Date _________________ Time _____________________
Participant # _________
Location _____________________
Prior to testing
■ Confirm schedule with Participants ■ Ensure EHRUT lab environment is running properly ■ Ensure Consent forms are complete for all participants ■ Ensure lab and data recording equipment is running properly
Prior to each participant: ■ Reset application ■ Restore application to base state using script restoredb.bat ■ Start session recordings with GTM Recording
Prior to each task: ■ Reset application to starting point for next task
After each participant: ■ End session recordings with tool ■ Fill out all paper work and acknowledgement form.
After all testing ■ Scan all forms and shred originals ■ Back up all video and data files
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 28 of 46
Orientation (5 minutes) Thank you for participating in this study. Our session today will last 2 hours today. During that time you will take a look at an electronic health record system. I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with anything to do with the system itself. Please save your detailed comments until the end of a task or the end of the session as a whole when we can discuss freely. I did not have any involvement in its creation, so please be honest with your opinions. The product you will be using today is describe the state of the application, i.e., production version, early prototype, etc. Some of the data may not make sense as it is placeholder data. We are recording the audio and screenshots of our session today. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Do you have any questions or concerns? Preliminary Questions (5 minutes) What is your job title / appointment? How long have you been working in this role? What are some of your main responsibilities? Tell me about your experience with electronic health records.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 29 of 46
Preview: First Impressions (6 Minutes) This is the application you will be working with. Have you heard of it? Yes ___ No ___ If so, tell me what you know about it. Show test participant the EHRUT. Please don't click on anything just yet. What do you notice? What are you able to do here? Please be specific. Notes / Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 30 of 46
Task 1: Create an Order – Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC The patient you are examining 5’ 10” 285 lbs. male needs a differential diagnosis of 034.0 Streptococcal Sore Throat and needs a 87880 rapid strep to confirm diagnosis. Enter Vitals, Dx and SP into the system Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, enter 5 feet 10 inches and 285 lbs 2. go to the DX tab and Enter the diagnosis 034.0 Streptococcal Sore Throat 3. Note Alert for Obesity and Hypertension in CDS slider 4. Still inside FNC, go to the SP tab 5. Enter procedure code 87880 Strep A Assay 6. Still inside FNC, click the Close button to close the note, save the note as incomplete 7. Click okay.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 31 of 46
Task 2: Utilize an existing Clinical Decision Support Alert – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You run are in consultation with a patient and his parent who is 71 years old. You want to quickly see if there are any Clinical Support Alerts for the parent. Go to the Appointment to review CDS Alerts and look at the source information for the alert. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
Open Patient Norman Wood
Open Update Appointment from Envelope
Open Clinical Decision Support Tab
Note Fall Risk Rule on Slider
Click on the Hyperlink for Fall Risk Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 32 of 46
Task 3: Creation of a Procedure Task Message to the Nurses - Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC
A patient you saw earlier that requested an order for a test at a different lab wants the lab found it was not covered by his insurance. He wants it to be performed in your office. You need to order a rapid strep test (87880) for the patient, as he is on his way back into the office.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open the scheduler 2. Find the appointment for Ed Wood 3. Click the Blue Folder (Open/Create note) 4. Click New->Procedure Task Message
a. Enter the Nurses b. Click Okay
5. Click to close the note a. Click Okay
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 33 of 46
Task 4: Approve a Lab result (5 Minutes) Take the participant to the starting point for the task. - DESKTOP
You have a lab result request message in your inbox to process. You need to review and approve the request.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations Approval of Lab Results
1. Locate the Lab Result Approval message in the Desktop Message pane 2. Click on the Time and Date Hyperlink or Plus (+) sign next to the message 3. Click on Process Message 4. From the Modify Message window 5. After review of the results, click the Approved All button
a. Your user (provider) name will auto populate along with the current date 6. Click Complete
a. This will simultaneously set the approved lab results and complete the message Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 34 of 46
Task 5: Document Drug Allergy in History – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are examining reports to you that she has been taking no meds because she has an anaphylactic reaction to Penicillins. Please document the allergy in History Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Drug Allergy
6. In the Allergen column, select Penicillins.
7. In the Reaction column, select anaphylaxis.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 35 of 46
Task 6: Drug Allergy Alert in Rx Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Need to prescribe Keflex to Mary Woods chart 1 tablet per day by mouth for 10 days Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Rx Tab
6. In the Drug field, enter Keflex
7. Click the find control
8. Select Keflex 500 mg
9. Pop up: Receive Drug Screening Warning: ALLERGY ALERT: The use of Keflex cross-sensitivity reaction based on a reported history of allergy to PENICILLINS. Do you want to continue? Click Yes
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 36 of 46
Task 7: Drug Drug Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP An agitated patient wants you to prescribe Prozac immediately. 20mg 1 time per day by mouth. Please prescribe the medication. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Find Mary Wood that has an appointment scheduled.
2. Click the Open/Create Note icon
3. FNC
4. Click Rx Tab
5. In the Drug field, enter Prozac
6. Click the find control
7. Select Prozac 20mg tablet
8. Pop up: Receive Drug Screening Warning: Duplicate Therapy: The use of Prozac is a duplicate therapy with Zoloft 50 mg. Do you want to continue?
a. Click No
9. Print -> Save and print Button
a. Print Dialog
10. Click ok to save and exit the note
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 37 of 46
Task 8: Medication List in OPS – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You are about to walk into an exam room for a med check on a patient. Go to One Page Summary and review the patient’s drug list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Go to Patient Demographics 2. Click Patient 3. Click One Page Summary 4. Click hyperlink in index for Current Medication 5. Verify the data is displayed correctly
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 38 of 46
Task 9: Medication List in Hx Tab with Filter – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
Previously very ill patient with a very large medication history is asking about her current medications. You want to compare the active/inactive medications with the maintenance medication list (which are two different filters in the system Go to Hx Tab and change the filter to show only Active Medications and them maintenance medications.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, go to the HX tab 2. Click on the Medication History 3. Review the items on the list 4. Click the search icon to change the filter 5. Change the filter to maintenance 6. Review the Maintenance Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 39 of 46
Task 10: Rx Refill Complete (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You have a few minutes to complete your drug refill request messages Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Messages
2. Locate message
3. Click on patient’s blue envelope.
4. Select Process Rx Refill
5. Medication History screen
6. Select the drug by clicking the Refill button.
7. Sig Writer screen
a. Verify information in sig
8. Click Ok
a. Return to the Modify Message screen
9. Click Complete.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 40 of 46
Task 11: import Allergies from external list - Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Allergy list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. Go to attachments – Allergies - CCD 3. Click the import allergies icon 4. Sulfasalazine -> Add 5. Penicillin V -> Merge/Add 6. Carbamazepine -> ignore/remove 7. Click Reconcile 8. Go to Hx
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 41 of 46
Task 12: Import Medications from external list – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Medication list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. View the medication CCD 3. Click the import medications icon 4. Simvastatin -> Add 5. Lorazepam -> Add 6. Lisinopril -> Add/Merge 7. Click reconcile 8. Go to patient Hx – Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 42 of 46
Task 13: Verify import in Allergy list and Medication List – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Please validate that the import of Allergy list and Medication list import has the expected information Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Note the CDS rule for the combination of the new allergy and medication
6. Click Drug Allergy
a. Verify the information is correct from the import
7. Click on the Medication History
a. Verify the information is correct from the import
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 43 of 46
Final Questions (10 Minutes) What was your overall impression of this system? What aspects of the system did you like most? What aspects of the system did you like least? Were there any features that you were surprised to see? What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? Compare this system to other systems you have used. Would you recommend this system to your colleagues? Administer the SUS
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 44 of 46
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
Strongly
disagree Strongly
Agree
1. I think that I would like to use this system frequently
1 2 3 4 5
2. I found the system unnecessarily complex
1 2 3 4 5
3. I thought the system was easy to use
1 2 3 4 5
4. I think that I would need the support of a technical person to be able to use this system
1 2 3 4 5
5. I found the various functions in this system were well integrated
1 2 3 4 5
6. I thought there was too much inconsistency in this system
1 2 3 4 5
7. I would imagine that most people would learn to use this system very quickly
1 2 3 4 5
8. I found the system very cumbersome to use
1 2 3 4 5
9. I felt very confident using the system
1 2 3 4 5
10. I needed to learn a lot of things before I could get going with this system
1 2 3 4 5
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 45 of 46
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM
Acknowledgement of Receipt I hereby acknowledge receipt of $ _____ for my participation in a research study run by Aprima Medical Software. Printed Name: _____________________________________________________________ Address: __________________________________________________________________ __________________________________________________________________________ Signature: _______________________________________________ Date: ____________ Usability Researcher: ________________________________________________________ Signature of Usability Researcher: ____________________________ Date: ______________ Witness: ___________________________________________________________________ Witness Signature: ________________________________________ Date: ______________
EHR USABILITY TEST REPORT
OF
APRIMA 2014
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
Aprima 2014 Date of Usability Test: 05/21/13-05/23/13 Date of Report: 05/27/13 Report Prepared By: Aprima Medical Software Project Management Cornell Hardman (214) 466-8107 Aprima Medical Software 3330 Keller Springs Road Suite 201 Carrollton, TX 75006
Table of Contents EXECUTIVE SUMMARY ................................................................................................................. 3
Major findings ............................................................................................................................... 5
Areas for improvement ................................................................................................................. 5
INTRODUCTION ............................................................................................................................. 6
METHOD ......................................................................................................................................... 6
PARTICIPANTS ........................................................................................................................... 6
STUDY DESIGN .......................................................................................................................... 8
TASKS ......................................................................................................................................... 9
PROCEDURES ............................................................................................................................ 9
TEST LOCATION ...................................................................................................................... 10
TEST ENVIRONMENT .............................................................................................................. 11
TEST FORMS AND TOOLS ...................................................................................................... 11
PARTICIPANT INSTRUCTIONS ............................................................................................... 12
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 2 of 46
USABILITY METRICS ............................................................................................................... 13
DATA SCORING ........................................................................................................................ 13
RESULTS ...................................................................................................................................... 16
DATA ANALYSIS AND REPORTING ........................................................................................ 16
DISCUSSION OF THE FINDINGS ............................................................................................ 17
EFFECTIVENESS ..................................................................................................................... 17
EFFICIENCY .............................................................................................................................. 17
SATISFACTION ......................................................................................................................... 17
MAJOR FINDINGS .................................................................................................................... 18
AREAS FOR IMPROVEMENT .................................................................................................. 18
APPENDICES ................................................................................................................................ 20
Appendix 1: SAMPLE RECRUITING SCREENER .................................................................... 21
Appendix 2: PARTICIPANT DEMOGRAPHICS ........................................................................ 24
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ........... 25
Appendix 4: EXAMPLE MODERATOR'S GUIDE ...................................................................... 27
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................. 44
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM ................................. 45
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 3 of 46
EXECUTIVE SUMMARY
A usability test of Aprima 2014 ambulatory EHR was conducted on the dates of
May 21-23 of 2013 in The Aprima Medical Software corporate office by Aprima
Medical Software Project Management team. The purpose of this test was to
test and validate the usability of the current user interface, and provide
evidence of usability in the EHR Under Test (EHRUT). During the usability
test, 2 healthcare providers and 4 other knowledgeable clinic staff matching the
target demographic criteria served as participants and used the EHRUT in
simulated, but representative tasks. This study collected performance data on
13 tasks including 4 tasks specifically designed to test ePrescribing
functionality that are typically conducted on an EHR:
Task 1 – Drug Allergy Alert in Rx Interaction
Task 2 – Drug-Drug Interaction
Task 3 – Complete a Drug refill request
Task 4 – Rx Refill Message Completion
During the 2-hour one-on-one usability test, each participant was greeted by
the administrator and asked to review and sign an informed consent/release
form (included in Appendix 3); they were instructed that they could withdraw at
any time. Participants had a mix of experience with the EHR. The
administrator introduced the test, and instructed participants to complete a
series of tasks (given one at a time) using the EHRUT. During the testing, the
administrator timed the test and, along with the data logger(s) recorded user
performance data on paper and electronically. The administrator did not give
the participant assistance in how to complete the task.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 4 of 46
Participant screens were recorded for subsequent analysis.
The following types of data were collected for each participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations Participant's satisfaction ratings of the system
All participant data was de-identified - no correspondence could be made from
the identity of the participant to the data collected. Following the conclusion of
the testing, participants were asked to complete a post-test questionnaire and
were compensated with $300 for their time. Various recommended metrics, in
accordance with the examples set forth in the NIST Guide to the Processes
Approach for Improving the Usability of Electronic Health Records, were used
to evaluate the usability of the EHRUT. Following is a summary of the
performance and rating data collected on the EHRUT.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations (Observed / Optimal)
Mean(SD)
Min/Max Optimal
Mean (SD)
Mean (SD)
1. Task 1 6 100% (0.00)
0.5 (57/54) 50 (25)
0:16/1:33 0:17
0.3 (0.5) 4.8 (0.4)
2. Task 2 6 100% (0.00)
0.3 (62/60) 46 (14)
0:28/1:17 0:30
0.0 (0.0) 4.7 (0.5)
3. Task 3 6 100% (0.00)
0.5 (39/36) 41 (8)
0:28/0:56 0:28
0.2 (0.4) 4.5 (0.8)
4. Task 4 6 100% (0.0)
0.7 (58/54) 61 (22)
0:30/1:35 0:27
0.0 (0.0) 4.2 (0.9)
The results from the System Usability Scale scored the subjective satisfaction
with the system based on performance with these tasks to be: 83
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 5 of 46
In addition to the performance data, the following qualitative observations were
made:
Major findings
The information presented and the way in which it was presented was
well received by the test users.
Initial verbal input from users was that the screen for specifying sig
details was visually complex and a bit overwhelming. On subsequent
use, it was verbally stated that the implementation was efficient, but
not visually pleasing
Drug refill request was not as efficient as anticipated. Users spent time
looking at the form while attempting to figure out what medicine was
being refilled. After the long pause, participants consistently did either
the optimal path or a slight variation that took an extra 8-15 seconds,
but did not cause any errors.
Areas for improvement
The Refill screen needs to be optimized to direct users to the correct
actions more quickly.
When a user goes off the correct path, he will often take a very long
time and many more errors before finding the correct path again.
Aprima has many ways to do the same operation. This leads to
confusion for users. Aprima should limit access to feature models to
simplify the user paradigm.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 6 of 46
INTRODUCTION
The EHRUT tested for this study was Aprima 2014 ambulatory EHR. Designed
to present medical information to healthcare providers in ambulatory medical
offices, the EHRUT consists of a user interface presented on a computer
display that stores and retrieves data from a database server through an
application server. The usability testing attempted to represent realistic
exercises and conditions. The purpose of this study was to test and validate
the usability of the current user interface, and provide evidence of usability in
the EHR Under Test (EHRUT). To this end, measures of effectiveness,
efficiency and user satisfaction, such as ease of access, number of mouse
clicks, time on task, time to repeat task, time to first entry and number of errors,
were captured during the usability testing.
METHOD
PARTICIPANTS
A total of 6 participants were tested on the EHRUT. Participants in the test
were physicians, nurses and other knowledgeable office staff. Participants
were recruited by Health Care Strategies and were compensated $300 for their
time. In addition, participants had no direct connection to the development of
or organization producing the EHRUT(s). Participants were not from the
testing or supplier organization. Participants were given the opportunity to
have an abbreviated orientation and a minimal level of training compared to the
amount the actual end users would have received. For the test purposes, end-
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 7 of 46
user characteristics were identified and translated into a recruitment screener
used to solicit potential participants; an example of a screener is provided in
Appendix 1.
Recruited participants had a mix of backgrounds and demographic
characteristics conforming to the recruitment screener. The following is a table
of participants by characteristics, including demographics, professional
experience, computing experience and user needs for assistive technology.
Participant names were replaced with Participant IDs so that an individual's
data cannot be tied back to individual identities.
Part ID
Gender Age Education Occupation /Role
Professional Experience
Computer Exp.
Product Exp.
Assistive Technology
Needs
1 142101 F 25 College Degree
Medical
Assistant 5 9 0/4
2 142102 F 42 College Degree
Medical
Assistant 3 13 0/1 3 142201 F 65 MD/PhD/DO Physician 38 4 0/0
4 142202 F 39 College Degree
Medical
Assistant 16 16 0/3
5 142301 F 31
Masters
Degree
Physician
Assistant 9 19 0/6
6 142302 F 30 College Degree
Medical
Assistant 8 13 0/7
9 participants (matching the demographics in the section on Participants) were
recruited and 6 participated in the usability test. 3 participants failed to show
for the study.
Participants were scheduled for 2 hour session with at least 1 hour in between
each session for debrief by the administrator(s) and data logger(s), and to reset
systems to proper test conditions. A spreadsheet was used to keep track of
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 8 of 46
the participant schedule, and included each participant's demographic
characteristics as provided by the recruiting firm.
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application
performed well - that is, effectively, efficiently, and with satisfaction - and areas
where the application failed to meet the needs of the participants. The data
from this test may serve as a baseline for future tests with an updated version
of the same EHR and/or comparison with other EHRs provided the same tasks
are used. In short, this testing serves as both a means to record or benchmark
current usability, but also to identify areas where improvements must be made.
During the usability test, participants interacted with a single EHR. Each
participant used the system in the same location, and was provided with the
same instructions. The system was evaluated for effectiveness, efficiency and
satisfaction as defined by measures collected and analyzed for each
participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations (comments) Participant's satisfaction ratings of the system
Additional information about the various measures can be found in
Section 3.9 on Usability Metrics.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 9 of 46
TASKS
A number of tasks were constructed that would be realistic and representative
of the kinds of activities a user might do with this EHR, including:
Task 1 – Drug Allergy Alert in Rx Interaction
Task 2 – Drug-Drug Interaction
Task 3 – Complete a Drug refill request
Task 4 – Rx Refill Message Completion
Tasks were selected based on their frequency of use, criticality of function, and
those that may be most troublesome for users. Tasks should always be
constructed in light of the study objectives.
PROCEDURES
Upon arrival, participants were greeted; their identity was verified and matched
with a name on the participant schedule. Participants were then assigned a
participant ID. Each participant reviewed and signed an informed consent and
release form (See Appendix 3). A representative from the test team witnessed
the participant's signature. To ensure that the test ran smoothly, two staff
members participated in this test, the usability administrator and the data
logger. The usability testing staff conducting the test was experienced usability
practitioners with 3 years of usability experience at GE Healthcare and
additional experience at other facilities. The administrator moderated the
session including administering instructions and tasks. The administrator also
monitored task times, obtained post-task rating data, and ensured notes on
participant comments were recorded. A second person served as the data
logger and took notes on task success, path deviations, number and type of
errors, and comments. Participants were instructed to perform the tasks (see
specific instructions below):
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 10 of 46
As quickly as possible making as few errors and deviations as possible.
Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use.
Without using a think aloud technique.
For each task, the participants were given a written copy of the task. Task
timing began once the administrator finished reading the question. The task
time was stopped once the participant indicated they had successfully
completed the task. Scoring is discussed below in Section 3.9. Following the
session, the administrator gave the participant the post-test questionnaire (e.g.,
the System Usability Scale, see Appendix 5), compensated him or her for their
time, and thanked each individual for their participation.
Participants' demographic information, task success rate, time on task, errors,
deviations, verbal responses, and post-test questionnaire were recorded into a
spreadsheet.
Participants were thanked for their time and compensated. Participants signed
a receipt and acknowledgement form (See Appendix 6) indicating that they had
received the compensation.
TEST LOCATION
The test facility included a waiting area and a quiet testing room with a table,
computer for the participant, and recording computer for the administrator.
Only the participant and administrator and the data logger were in the test
room. There were no observers. To ensure that the environment was
comfortable for users, noise levels were kept to a minimum with the ambient
temperature within a normal range. All of the safety instruction and evacuation
procedures were valid, in place, and visible to the participants.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 11 of 46
TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or facility. In this
instance, the testing was conducted in the Aprima Medical Software corporate
headquarters in the main training room. For testing, the computer used a
Fujitsu T902 running Microsoft Windows 8. The participants used whatever
input method he or she was most comfortable with from among mouse,
keyboard, touchpad, stylus or voice input when interacting with the EHRUT.
The EHRUT used a 12.7-inch diagonal display running 1280x720 pixels at 32
bit color. The application was set up by the administrator according to the
vendor's documentation. The application itself was running on a Dell 620
server running Windows Server 2008 using a test database on a LAN
connection. Technically, the system performance (i.e., response time) was
representative to what actual users would experience in a field implementation.
Additionally, participants were instructed not to change any of the default
system settings (such as control of font size).
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used,
including:
1. Informed Consent
2. Moderator's Guide
3. Post-test Questionnaire
4. Incentive Receipt and Acknowledgment Form
Examples of these documents can be found in Appendices 3-6 respectively.
The Moderator's Guide was devised so as to be able to capture required data.
The participant's interaction with the EHRUT was captured and recorded
digitally with screen capture software running on the test machine.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 12 of 46
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each participant:
Thank you for participating in this study. Your input is very important.
Our session today will last about 30 minutes. During that time you will
use an instance of an electronic health record. I will ask you to
complete a few tasks using this system and answer some questions.
You should complete the tasks as quickly as possible making as few
errors as possible. Please try to complete the tasks on your own
following the instructions very closely. Please note that we are not
testing you we are testing the system, therefore if you have difficulty all
this means is that something needs to be improved in the system. I will
be here in case you need specific help, but I am not able to instruct you
or provide help in how to use the application. Overall, we are interested
in how easy (or how difficult) this system is to use, what in it would be
useful to you, and how we could improve it. I did not have any
involvement in its creation, so please be honest with your opinions.
There are a variety of tools that record screens and transmit those
recordings across a local area network for remote observations. All of
the information that you provide will be kept confidential and your name
will not be associated with your comments at any time. Should you feel
it necessary you are able to withdraw at any time during the testing.
Following the procedural instructions, participants were shown the EHR and as
their first task, were given time (10 minutes) to explore the system and make
comments. Once this task was complete, the administrator gave the following
instructions:
For each task, I will read the description to you and say "Begin." At that
point, please perform the task and say "Done" once you believe you
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 13 of 46
have successfully completed the task. I would like to request that you
not talk aloud or verbalize while you are doing the tasks. I will ask you
your impressions about the task once you are done.
Participants were then given 13 tasks to complete. Tasks are listed in the
moderator's guide in the Appendix.
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the
Usability of Electronic Health Records, EHRs should support a process that
provides a high level of usability for all users. The goal is for users to interact
with the system effectively, efficiently, and with an acceptable level of
satisfaction. To this end, metrics for effectiveness, efficiency and user
satisfaction were captured during the usability testing. The goals of the test
were to assess:
1. Effectiveness of EHRUT by measuring participant success rates and errors
2. Efficiency of EHRUT by measuring the average task time and path
deviations
3. Satisfaction with EHRUT by measuring ease of use ratings
DATA SCORING
The following table (Table 1) details how tasks were scored, errors evaluated,
and the time data analyzed.
Measures Rational and Scoring
Effectiveness: Task Success
A task was counted as a "Success" if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 14 of 46
Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator's Guide must be operationally defined by taking multiple measures of optimal performance and multiplying by some factor that allows some time buffer because the participants are presumably not trained to expert performance. Scoring used a factor of 2. Thus, if expert, optimal performance on a task was 80 seconds then allotted task time performance was 80 * 2 = 160 seconds. This ratio should be aggregated across tasks and reported with mean and variance scores.
Effectiveness: Task Failures
If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a "Failure." No task times were taken for errors. The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. This should also be expressed as the mean number of failed tasks per participant. On a qualitative level, an enumeration of errors and error types were collected.
Efficiency: Task Deviations
The participant's path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps in order to provide a ratio of path deviation. It is strongly recommended that task deviations be reported. Optimal paths (i.e., procedural steps) should be recorded when constructing tasks.
Efficiency: Task Time
Each task was timed from when the administrator said "Begin" until the participant said, "Done." If he or she failed to say "Done," the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.
Satisfaction: Participant's subjective impression of the ease of use of the
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 15 of 46
Task Rating application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate "Overall, this task was:" on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants' confidence in and likeability of the EHRUT overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, "I think I would like to use this system frequently," "I thought the system was easy to use," and "I would imagine that most people would learn to use this system very quickly." See full System Usability Score questionnaire in Appendix 5.
Table 1. Details of how observed data were scored.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 16 of 46
RESULTS
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods
specified in the Usability Metrics section above. Participants who failed to
follow session and task instructions had their data excluded from the analyses.
In this test cycle, there were no data excluded.
The usability testing results for the EHRUT are detailed below. The results
should be seen in light of the objectives and goals outlined in Section 3.2 Study
Design. The data should yield actionable results that, if corrected, yield
material, positive impact on user performance.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations (Observed / Optimal)
Mean(SD)
Min/Max Optimal
Mean (SD)
Mean (SD)
1. Task 1 6 100% (0.00)
0.5 (57/54) 50 (25)
0:16/1:33 0:17
0.3 (0.5) 4.8 (0.4)
2. Task 2 6 100% (0.00)
0.3 (62/60) 46 (14)
0:28/1:17 0:30
0.0 (0.0) 4.7 (0.5)
3. Task 3 6 100% (0.00)
0.5 (39/36) 41 (8)
0:28/0:56 0:28
0.2 (0.4) 4.5 (0.8)
4. Task 4 6 100% (0.0)
0.7 (58/54) 61 (22)
0:30/1:35 0:27
0.0 (0.0) 4.2 (0.9)
The results from the SUS (System Usability Scale) scored the subjective
satisfaction with the system based on performance with these tasks to be: 83.
Broadly interpreted, scores under 60 represent systems with poor usability;
scores over 80 would be considered above average
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 17 of 46
DISCUSSION OF THE FINDINGS
The information presented and the way in which it was presented was well
received by the test users.
Drug refill request was not as efficient as anticipated. Users spent time looking
at the form while attempting to figure out what medicine was being refilled
Users after the long pause consistently did either the optimal path or a slight
variation that took an extra 8-15 seconds, but did not cause any errors.
EFFECTIVENESS
The users were able to do what was expected with the product. Many times the
users went slightly off the optimal the path but did not lose their way. Users
with a small amount of time on the system improved dramatically. All users
rated the product either agree or strongly agree with regard to “I thought the
system was easy to use”. All users also rated “I found the system
unnecessarily complex” either disagree or strongly disagree.
EFFICIENCY
The number of path deviations on some tasks was above 2.0 this is large
number. For most tasks the number was at or below 1.0. Users were in almost
all cases able to quickly come back to the optimal path and complete the task.
This is true for the tasks in which the path deviations where the highest as well.
Variance on the timings show that the untrained users are not efficient using
Aprima. User learning is rapid; by task 5 the users were able to be more
efficient with task times being one half or less compared to the first 4 tasks.
SATISFACTION
User satisfaction was quite high. Overall user rating was 83.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 18 of 46
Question Agree or Strongly Agree
Disagree or Strongly Disagree
I think that I would like to use this system frequently
83% 0%
I found the system unnecessarily complex
0% 100%
I thought the system was easy to use
100% 0%
I think that I would need the support of a technical person to be able to use this system
17% 67%
I found the various functions in this system were well integrated
83% 0%
I thought there was too much inconsistency in this system
0% 100%
I would imagine that most people would learn to use this system very quickly
100% 0%
I found the system very cumbersome to use
33% 67%
I felt very confident using the system
100% 0%
I needed to learn a lot of things before I could get going with this system
0% 100%
“I found the system cumbersome to use” seems to be a polarizing question. In
future user testing this will have to be explored more, as we did not find this to
be reflected in the user comments and was not discovered until the data
analysis phase.
MAJOR FINDINGS
The information presented and the way in which it was presented was
well received by the test users.
Initial verbal input from users was that the screen for specifying sig
details was visually complex and a bit overwhelming. On subsequent
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 19 of 46
use, it was verbally stated that the implementation was efficient, but
not visually pleasing
Drug refill request was not as efficient as anticipated. Users spent time
looking at the form while attempting to figure out what medicine was
being refilled. After the long pause, participants consistently did either
the optimal path or a slight variation that took an extra 8-15 seconds,
but did not cause any errors.
AREAS FOR IMPROVEMENT
The Refill screen needs to be optimized to direct users to the correct
actions more quickly.
When a user goes off the correct path, he will often take a very long
time and many more errors before finding the correct path again.
Aprima has many ways to do the same operation. This leads to
confusion for users. Aprima should limit access to feature models to
simplify the user paradigm.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 20 of 46
APPENDICES
The following appendices include supplemental data for this usability test
report. Following is a list of the appendices provided:
1: Sample Recruiting screener
2: Participant demographics
3: Non-Disclosure Agreement (NDA) and Informed Consent Form
4: Example Moderator's Guide
5: System Usability Scale Questionnaire
6: Incentive receipt and acknowledgment form It is important to note that these
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 21 of 46
Appendix 1: SAMPLE RECRUITING SCREENER
The purpose of a screener to ensure that the participants selected represent
the target user population as closely as possible. (Portions of this sample
screener are taken from www.usability.gov/templates/index.html#Usability and
adapted for use.)
Recruiting Script for Recruiting Firm
Hello, my name is _______________, calling from [Insert name of recruiting
firm]. We are recruiting individuals to participate in a usability study for an
electronic health record. We would like to ask you a few questions to see if
you qualify and if would like to participate. This should only take a few minutes
of your time. This is strictly for research purposes. If you are interested and
qualify for the study, you will be paid to participate. Can I ask you a few
questions?
Customize this by dropping or adding questions so that it reflects your EHR's
primary audience
1. [If not obvious] Are you male or female? [Recruit a mix of participants]
2. Have you participated in a focus group or usability test in the past xx months? [If yes, Terminate]
3. Do you, or does anyone in your home, work in marketing research, usability research, web design [If yes, Terminate]
4. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record software or consulting company? [If yes, Terminate]
5. Which of the following best describes your age? [23 to 39; 40 to 59; 60 and older] [Recruit Mix]
6. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-American, Latino/a or Hispanic, etc.]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 22 of 46
7. Do you require any assistive technologies to use a computer? [if so, please describe] Professional Demographics Customize this to reflect your EHR's primary audience
8. What is your current position and title? (Must be healthcare provider)
□ RN: Specialty _______________________
□ Physician: Specialty _______________________
□ Resident: Specialty _______________________
□ Medical Assistant _______________________
□ Administrative Staff
□ Other [Terminate]
9. How long have you held this position?
10. Describe your work location (or affiliation) and environment? (Recruit according to the intended users of the application. e.g., private practice, health system, government clinic, etc.)
11. Which of the following describes your highest level of education? [e.g., high school graduate/GED, some college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)
12. Besides reading email, what professional activities do you do on the computer? [e.g., access EHR, research; reading news; shopping/banking; digital pictures; programming/word processing, etc.] [If no computer use at all, Terminate]
13. About how many hours per week do you spend on the computer? [Recruit according to the demographics of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]
14. What computer platform do you usually use? [e.g., Mac, Windows, etc.]
15. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]
16. In the last month, how often have you used an electronic health record?
17. How many years have you used an electronic health record?
18. How many EHRs do you use or are you familiar with?
19. How does your work environment handle patient records? [Recruit according to the demographics of the intended users]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 23 of 46
□ On paper
□ Some paper, some electronic
□ All electronic
Contact Information If the person matches your qualifications, ask
Those are all the questions I have for you. Your background matches the
people we're looking for. For your participation, you will be paid $300. Would
you be able to participate on [date, time]? [If so collect contact information]
May I get your contact information?
■ Name of participant:
■ Address:
■ City, State, Zip:
■ Daytime phone number:
■ Evening phone number:
■ Alternate [cell] phone number:
■ Email address:
Before your session starts, we will ask you to sign a release form allowing us to
record your session. The record will only be used internally for further study if
needed. Will you consent to be digitally recorded? This study will take place at
3330 Keller Springs Road, Suite 201, Carrollton, TX 75006. I will confirm your
appointment a couple of days before your session and provide you with
directions to our office. What time is the best time to reach you?
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 24 of 46
Appendix 2: PARTICIPANT DEMOGRAPHICS
Following is a high-level overview of the participants in this study.
Gender
Men: 0 Women: 6 Total (participants): 6
Occupation/Role
RN/BSN: 0 Physician: 2 Admin Staff: 4 Total (participants): 6
Years of Experience
Facility Use of EHR All paper: 1 Some paper, some electronic: 5 All electronic: 0 Total (participants): 6
The full break down is included in the report above.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 25 of 46
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM
Non-Disclosure Agreement THIS AGREEMENT is entered into as of _________________, 2013, between ________________ ("the Participant") and the testing organization Aprima Medical Software located at 3330 Keller Springs Rd. Suite 201, Carrollton, TX 75006. The Participant acknowledges his or her voluntary participation in today's usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by Test Company, or otherwise acquired by the Participant, in the course of today's study. By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts. Any information the Participant acquires relating to this product during this study is confidential and proprietary to Test Company and is being disclosed solely for the purposes of the Participant's participation in today's usability study. By signing this form the Participant acknowledges that s/he will receive monetary compensation for feedback and will not disclose this confidential information obtained today to anyone else or any other organizations. Participant's printed name: ___________________________________________________ Signature: __________________________________________________________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 26 of 46
Informed Consent Test Company would like to thank you for participating in this study. The purpose of this study is to evaluate an electronic health records system. If you decide to participate, you will be asked to perform several tasks using the prototype and give your feedback. The study will last about 30 minutes. At the conclusion of the test, you will be compensated for your time. Agreement I understand and agree that as a voluntary participant in the present study conducted by Test Company I am free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study conducted and videotaped by the Test Company. I understand and consent to the use and release of the videotape by Test Company. I understand that the information and videotape is for research purposes only and that my name and image will not be used for any purpose other than research. I relinquish any rights to the videotape and understand the videotape may be copied and used by Test Company without further permission. I understand and agree that the purpose of this study is to make software applications more useful and usable in the future. I understand and agree that the data collected from this study may be shared with outside of Test Company and Test Company's client. I understand and agree that data confidentiality is assured, because only de- identified data - i.e., identification numbers not names - will be used in analysis and reporting of the results. I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can leave at any time. Please check one of the following: □ YES, I have read the above statement and agree to be a participant. □ NO, I choose not to participate in this study. Signature: _________________________________ Date: _______________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 27 of 46
Appendix 4: EXAMPLE MODERATOR'S GUIDE
Only four tasks are presented here for illustration. EHRUT Usability Test Moderator's Guide Administrator _____________________
Data Logger _____________________
Date _________________ Time _____________________
Participant # _________
Location _____________________
Prior to testing
■ Confirm schedule with Participants ■ Ensure EHRUT lab environment is running properly ■ Ensure Consent forms are complete for all participants ■ Ensure lab and data recording equipment is running properly
Prior to each participant: ■ Reset application ■ Restore application to base state using script restoredb.bat ■ Start session recordings with GTM Recording
Prior to each task: ■ Reset application to starting point for next task
After each participant: ■ End session recordings with tool ■ Fill out all paper work and acknowledgement form.
After all testing ■ Scan all forms and shred originals ■ Back up all video and data files
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 28 of 46
Orientation (5 minutes) Thank you for participating in this study. Our session today will last 2 hours today. During that time you will take a look at an electronic health record system. I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with anything to do with the system itself. Please save your detailed comments until the end of a task or the end of the session as a whole when we can discuss freely. I did not have any involvement in its creation, so please be honest with your opinions. The product you will be using today is describe the state of the application, i.e., production version, early prototype, etc. Some of the data may not make sense as it is placeholder data. We are recording the audio and screenshots of our session today. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Do you have any questions or concerns? Preliminary Questions (5 minutes) What is your job title / appointment? How long have you been working in this role? What are some of your main responsibilities? Tell me about your experience with electronic health records.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 29 of 46
Preview: First Impressions (6 Minutes) This is the application you will be working with. Have you heard of it? Yes ___ No ___ If so, tell me what you know about it. Show test participant the EHRUT. Please don't click on anything just yet. What do you notice? What are you able to do here? Please be specific. Notes / Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 30 of 46
Task 1: Create an Order – Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC The patient you are examining 5’ 10” 285 lbs. male needs a differential diagnosis of 034.0 Streptococcal Sore Throat and needs a 87880 rapid strep to confirm diagnosis. Enter Vitals, Dx and SP into the system Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, enter 5 feet 10 inches and 285 lbs 2. go to the DX tab and Enter the diagnosis 034.0 Streptococcal Sore Throat 3. Note Alert for Obesity and Hypertension in CDS slider 4. Still inside FNC, go to the SP tab 5. Enter procedure code 87880 Strep A Assay 6. Still inside FNC, click the Close button to close the note, save the note as incomplete 7. Click okay.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 31 of 46
Task 2: Utilize an existing Clinical Decision Support Alert – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You run are in consultation with a patient and his parent who is 71 years old. You want to quickly see if there are any Clinical Support Alerts for the parent. Go to the Appointment to review CDS Alerts and look at the source information for the alert. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
Open Patient Norman Wood
Open Update Appointment from Envelope
Open Clinical Decision Support Tab
Note Fall Risk Rule on Slider
Click on the Hyperlink for Fall Risk Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 32 of 46
Task 3: Creation of a Procedure Task Message to the Nurses - Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC
A patient you saw earlier that requested an order for a test at a different lab wants the lab found it was not covered by his insurance. He wants it to be performed in your office. You need to order a rapid strep test (87880) for the patient, as he is on his way back into the office.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open the scheduler 2. Find the appointment for Ed Wood 3. Click the Blue Folder (Open/Create note) 4. Click New->Procedure Task Message
a. Enter the Nurses b. Click Okay
5. Click to close the note a. Click Okay
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 33 of 46
Task 4: Approve a Lab result (5 Minutes) Take the participant to the starting point for the task. - DESKTOP
You have a lab result request message in your inbox to process. You need to review and approve the request.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations Approval of Lab Results
1. Locate the Lab Result Approval message in the Desktop Message pane 2. Click on the Time and Date Hyperlink or Plus (+) sign next to the message 3. Click on Process Message 4. From the Modify Message window 5. After review of the results, click the Approved All button
a. Your user (provider) name will auto populate along with the current date 6. Click Complete
a. This will simultaneously set the approved lab results and complete the message Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 34 of 46
Task 5: Document Drug Allergy in History – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are examining reports to you that she has been taking no meds because she has an anaphylactic reaction to Penicillins. Please document the allergy in History Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Drug Allergy
6. In the Allergen column, select Penicillins.
7. In the Reaction column, select anaphylaxis.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 35 of 46
Task 6: Drug Allergy Alert in Rx Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Need to prescribe Keflex to Mary Woods chart 1 tablet per day by mouth for 10 days Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Rx Tab
6. In the Drug field, enter Keflex
7. Click the find control
8. Select Keflex 500 mg
9. Pop up: Receive Drug Screening Warning: ALLERGY ALERT: The use of Keflex cross-sensitivity reaction based on a reported history of allergy to PENICILLINS. Do you want to continue? Click Yes
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 36 of 46
Task 7: Drug Drug Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP An agitated patient wants you to prescribe Prozac immediately. 20mg 1 time per day by mouth. Please prescribe the medication. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Find Mary Wood that has an appointment scheduled.
2. Click the Open/Create Note icon
3. FNC
4. Click Rx Tab
5. In the Drug field, enter Prozac
6. Click the find control
7. Select Prozac 20mg tablet
8. Pop up: Receive Drug Screening Warning: Duplicate Therapy: The use of Prozac is a duplicate therapy with Zoloft 50 mg. Do you want to continue?
a. Click No
9. Print -> Save and print Button
a. Print Dialog
10. Click ok to save and exit the note
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 37 of 46
Task 8: Medication List in OPS – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You are about to walk into an exam room for a med check on a patient. Go to One Page Summary and review the patient’s drug list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Go to Patient Demographics 2. Click Patient 3. Click One Page Summary 4. Click hyperlink in index for Current Medication 5. Verify the data is displayed correctly
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 38 of 46
Task 9: Medication List in Hx Tab with Filter – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
Previously very ill patient with a very large medication history is asking about her current medications. You want to compare the active/inactive medications with the maintenance medication list (which are two different filters in the system Go to Hx Tab and change the filter to show only Active Medications and them maintenance medications.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, go to the HX tab 2. Click on the Medication History 3. Review the items on the list 4. Click the search icon to change the filter 5. Change the filter to maintenance 6. Review the Maintenance Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 39 of 46
Task 10: Rx Refill Complete (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You have a few minutes to complete your drug refill request messages Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Messages
2. Locate message
3. Click on patient’s blue envelope.
4. Select Process Rx Refill
5. Medication History screen
6. Select the drug by clicking the Refill button.
7. Sig Writer screen
a. Verify information in sig
8. Click Ok
a. Return to the Modify Message screen
9. Click Complete.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 40 of 46
Task 11: import Allergies from external list - Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Allergy list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. Go to attachments – Allergies - CCD 3. Click the import allergies icon 4. Sulfasalazine -> Add 5. Penicillin V -> Merge/Add 6. Carbamazepine -> ignore/remove 7. Click Reconcile 8. Go to Hx
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 41 of 46
Task 12: Import Medications from external list – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Medication list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. View the medication CCD 3. Click the import medications icon 4. Simvastatin -> Add 5. Lorazepam -> Add 6. Lisinopril -> Add/Merge 7. Click reconcile 8. Go to patient Hx – Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 42 of 46
Task 13: Verify import in Allergy list and Medication List – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Please validate that the import of Allergy list and Medication list import has the expected information Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Note the CDS rule for the combination of the new allergy and medication
6. Click Drug Allergy
a. Verify the information is correct from the import
7. Click on the Medication History
a. Verify the information is correct from the import
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 43 of 46
Final Questions (10 Minutes) What was your overall impression of this system? What aspects of the system did you like most? What aspects of the system did you like least? Were there any features that you were surprised to see? What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? Compare this system to other systems you have used. Would you recommend this system to your colleagues? Administer the SUS
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 44 of 46
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
Strongly
disagree Strongly
Agree
1. I think that I would like to use this system frequently
1 2 3 4 5
2. I found the system unnecessarily complex
1 2 3 4 5
3. I thought the system was easy to use
1 2 3 4 5
4. I think that I would need the support of a technical person to be able to use this system
1 2 3 4 5
5. I found the various functions in this system were well integrated
1 2 3 4 5
6. I thought there was too much inconsistency in this system
1 2 3 4 5
7. I would imagine that most people would learn to use this system very quickly
1 2 3 4 5
8. I found the system very cumbersome to use
1 2 3 4 5
9. I felt very confident using the system
1 2 3 4 5
10. I needed to learn a lot of things before I could get going with this system
1 2 3 4 5
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 45 of 46
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM
Acknowledgement of Receipt I hereby acknowledge receipt of $ _____ for my participation in a research study run by Aprima Medical Software. Printed Name: _____________________________________________________________ Address: __________________________________________________________________ __________________________________________________________________________ Signature: _______________________________________________ Date: ____________ Usability Researcher: ________________________________________________________ Signature of Usability Researcher: ____________________________ Date: ______________ Witness: ___________________________________________________________________ Witness Signature: ________________________________________ Date: ______________
EHR USABILITY TEST REPORT
OF
APRIMA 2014
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
Aprima 2014 Date of Usability Test: 05/21/13-05/23/13 Date of Report: 05/27/13 Report Prepared By: Aprima Medical Software Project Management Cornell Hardman (214) 466-8107 Aprima Medical Software 3330 Keller Springs Road Suite 201 Carrollton, TX 75006
Table of Contents EXECUTIVE SUMMARY ................................................................................................................. 3
Major findings ............................................................................................................................... 5
Areas for improvement ................................................................................................................. 5
INTRODUCTION ............................................................................................................................. 6
METHOD ......................................................................................................................................... 6
PARTICIPANTS ........................................................................................................................... 6
STUDY DESIGN .......................................................................................................................... 8
TASKS ......................................................................................................................................... 9
PROCEDURES ............................................................................................................................ 9
TEST LOCATION ...................................................................................................................... 10
TEST ENVIRONMENT .............................................................................................................. 11
TEST FORMS AND TOOLS ...................................................................................................... 11
PARTICIPANT INSTRUCTIONS ............................................................................................... 12
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 2 of 46
USABILITY METRICS ............................................................................................................... 13
DATA SCORING ........................................................................................................................ 13
RESULTS ...................................................................................................................................... 16
DATA ANALYSIS AND REPORTING ........................................................................................ 16
DISCUSSION OF THE FINDINGS ............................................................................................ 17
EFFECTIVENESS ..................................................................................................................... 17
EFFICIENCY .............................................................................................................................. 17
SATISFACTION ......................................................................................................................... 17
MAJOR FINDINGS .................................................................................................................... 18
AREAS FOR IMPROVEMENT .................................................................................................. 18
APPENDICES ................................................................................................................................ 20
Appendix 1: SAMPLE RECRUITING SCREENER .................................................................... 21
Appendix 2: PARTICIPANT DEMOGRAPHICS ........................................................................ 24
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ........... 25
Appendix 4: EXAMPLE MODERATOR'S GUIDE ...................................................................... 27
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................. 44
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM ................................. 45
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 3 of 46
EXECUTIVE SUMMARY
A usability test of Aprima 2014 ambulatory EHR was conducted on the dates of
May 21-23 of 2013 in The Aprima Medical Software corporate office by Aprima
Medical Software Project Management team. The purpose of this test was to
test and validate the usability of the current user interface, and provide
evidence of usability in the EHR Under Test (EHRUT). During the usability
test, 2 healthcare providers and 4 other knowledgeable clinic staff matching the
target demographic criteria served as participants and used the EHRUT in
simulated, but representative tasks. This study collected performance data on
13 tasks including 5 tasks specifically designed to test Medication list
functionality that are typically conducted on an EHR:
Task 1 – Drug Allergy Alert
Task 2 – Drug-Drug interaction in Rx Tab
Task 3 – Medication List in One Page Summary
Task 4 – Medication List in Hx Tab with a Filter
Task 5 – Verify import in Allergy list and Medication List
During the 2-hour one-on-one usability test, each participant was greeted by
the administrator and asked to review and sign an informed consent/release
form (included in Appendix 3); they were instructed that they could withdraw at
any time. Participants had a mix of experience with the EHR. The
administrator introduced the test, and instructed participants to complete a
series of tasks (given one at a time) using the EHRUT. During the testing, the
administrator timed the test and, along with the data logger(s) recorded user
performance data on paper and electronically. The administrator did not give
the participant assistance in how to complete the task.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 4 of 46
Participant screens were recorded for subsequent analysis.
The following types of data were collected for each participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations Participant's satisfaction ratings of the system
All participant data was de-identified - no correspondence could be made from
the identity of the participant to the data collected. Following the conclusion of
the testing, participants were asked to complete a post-test questionnaire and
were compensated with $300 for their time. Various recommended metrics, in
accordance with the examples set forth in the NIST Guide to the Processes
Approach for Improving the Usability of Electronic Health Records, were used
to evaluate the usability of the EHRUT. Following is a summary of the
performance and rating data collected on the EHRUT.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations (Observed / Optimal)
Mean(SD)
Min/Max Optimal
Mean (SD)
Mean (SD)
1. Task 1 6 100% (0.00)
0.5 (57/54) 50 (25)
0:16/1:33 0:17
0.3 (0.5) 4.8 (0.4)
2. Task 2 6 100% (0.00)
0.3 (62/60) 46 (14)
0:28/1:17 0:30
0.0 (0.0) 4.7 (0.5)
3. Task 3 6 100% (0.00)
0.3 (32/30) 20 (8)
0:11/0:31 0:11
0.0 (0.0) 5.0 (0.0)
4. Task 4 6 100% (0.00)
0.5 (39/36) 41 (8)
0:28/0:56 0:28
0.2 (0.4) 4.5 (0.8)
5. Task 5 6 100% (0.00)
0.3 (38/36) 33 (13)
0:22/1:00 0:22
0.2 (0.4) 4.3 (0.5)
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 5 of 46
The results from the System Usability Scale scored the subjective satisfaction
with the system based on performance with these tasks to be: 83
In addition to the performance data, the following qualitative observations were
made:
Major findings
The information presented and the way in which it was presented was
well received by the test users. Easy of use averaged above 4.6
across all Medication list related tasks.
The scenario for using a filter was not realistic for the users, so they
had some difficultly understanding the reason for the task. This
uncertainty caused some users to slow down and verbalize their
concern. Some of the variation in timings between users was
attributed to the quality of the scenario.
The task to change the filter showed a path that was unexpected.
Users modified the filter itself, rather than use a different filter. This
was considered a success, as the information the users were expected
was displayed on screen, and the task was repeatable without
degradation in usability or overall system performance.
Areas for improvement
Recommend limited access to modify filters
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 6 of 46
INTRODUCTION
The EHRUT tested for this study was Aprima 2014 ambulatory EHR. Designed
to present medical information to healthcare providers in ambulatory medical
offices, the EHRUT consists of a user interface presented on a computer
display that stores and retrieves data from a database server through an
application server. The usability testing attempted to represent realistic
exercises and conditions. The purpose of this study was to test and validate
the usability of the current user interface, and provide evidence of usability in
the EHR Under Test (EHRUT). To this end, measures of effectiveness,
efficiency and user satisfaction, such as ease of access, number of mouse
clicks, time on task, time to repeat task, time to first entry and number of errors,
were captured during the usability testing.
METHOD
PARTICIPANTS
A total of 6 participants were tested on the EHRUT. Participants in the test
were physicians, nurses and other knowledgeable office staff. Participants
were recruited by Health Care Strategies and were compensated $300 for their
time. In addition, participants had no direct connection to the development of
or organization producing the EHRUT(s). Participants were not from the
testing or supplier organization. Participants were given the opportunity to
have an abbreviated orientation and a minimal level of training compared to the
amount the actual end users would have received. For the test purposes, end-
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 7 of 46
user characteristics were identified and translated into a recruitment screener
used to solicit potential participants; an example of a screener is provided in
Appendix 1.
Recruited participants had a mix of backgrounds and demographic
characteristics conforming to the recruitment screener. The following is a table
of participants by characteristics, including demographics, professional
experience, computing experience and user needs for assistive technology.
Participant names were replaced with Participant IDs so that an individual's
data cannot be tied back to individual identities.
Part ID
Gender Age Education Occupation /Role
Professional Experience
Computer Exp.
Product Exp.
Assistive Technology
Needs
1 142101 F 25 College Degree
Medical
Assistant 5 9 0/4
2 142102 F 42 College Degree
Medical
Assistant 3 13 0/1 3 142201 F 65 MD/PhD/DO Physician 38 4 0/0
4 142202 F 39 College Degree
Medical
Assistant 16 16 0/3
5 142301 F 31
Masters
Degree
Physician
Assistant 9 19 0/6
6 142302 F 30 College Degree
Medical
Assistant 8 13 0/7
9 participants (matching the demographics in the section on Participants) were
recruited and 6 participated in the usability test. 3 participants failed to show
for the study.
Participants were scheduled for 2 hour session with at least 1 hour in between
each session for debrief by the administrator(s) and data logger(s), and to reset
systems to proper test conditions. A spreadsheet was used to keep track of
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 8 of 46
the participant schedule, and included each participant's demographic
characteristics as provided by the recruiting firm.
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application
performed well - that is, effectively, efficiently, and with satisfaction - and areas
where the application failed to meet the needs of the participants. The data
from this test may serve as a baseline for future tests with an updated version
of the same EHR and/or comparison with other EHRs provided the same tasks
are used. In short, this testing serves as both a means to record or benchmark
current usability, but also to identify areas where improvements must be made.
During the usability test, participants interacted with a single EHR. Each
participant used the system in the same location, and was provided with the
same instructions. The system was evaluated for effectiveness, efficiency and
satisfaction as defined by measures collected and analyzed for each
participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations (comments) Participant's satisfaction ratings of the system
Additional information about the various measures can be found in
Section 3.9 on Usability Metrics.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 9 of 46
TASKS
A number of tasks were constructed that would be realistic and representative
of the kinds of activities a user might do with this EHR, including:
Task 1 – Drug Allergy Alert
Task 2 – Drug-Drug interaction in Rx Tab
Task 3 – Medication List in One Page Summary
Task 4 – Medication List in Hx Tab with a Filter
Task 5 – Verify import in Allergy list and Medication List
Tasks were selected based on their frequency of use, criticality of function, and
those that may be most troublesome for users. Tasks should always be
constructed in light of the study objectives.
PROCEDURES
Upon arrival, participants were greeted; their identity was verified and matched
with a name on the participant schedule. Participants were then assigned a
participant ID. Each participant reviewed and signed an informed consent and
release form (See Appendix 3). A representative from the test team witnessed
the participant's signature. To ensure that the test ran smoothly, two staff
members participated in this test, the usability administrator and the data
logger. The usability testing staff conducting the test was experienced usability
practitioners with 3 years of usability experience at GE Healthcare and
additional experience at other facilities. The administrator moderated the
session including administering instructions and tasks. The administrator also
monitored task times, obtained post-task rating data, and ensured notes on
participant comments were recorded. A second person served as the data
logger and took notes on task success, path deviations, number and type of
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 10 of 46
errors, and comments. Participants were instructed to perform the tasks (see
specific instructions below):
As quickly as possible making as few errors and deviations as possible.
Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use.
Without using a think aloud technique.
For each task, the participants were given a written copy of the task. Task
timing began once the administrator finished reading the question. The task
time was stopped once the participant indicated they had successfully
completed the task. Scoring is discussed below in Section 3.9. Following the
session, the administrator gave the participant the post-test questionnaire (e.g.,
the System Usability Scale, see Appendix 5), compensated him or her for their
time, and thanked each individual for their participation.
Participants' demographic information, task success rate, time on task, errors,
deviations, verbal responses, and post-test questionnaire were recorded into a
spreadsheet.
Participants were thanked for their time and compensated. Participants signed
a receipt and acknowledgement form (See Appendix 6) indicating that they had
received the compensation.
TEST LOCATION
The test facility included a waiting area and a quiet testing room with a table,
computer for the participant, and recording computer for the administrator.
Only the participant and administrator and the data logger were in the test
room. There were no observers. To ensure that the environment was
comfortable for users, noise levels were kept to a minimum with the ambient
temperature within a normal range. All of the safety instruction and evacuation
procedures were valid, in place, and visible to the participants.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 11 of 46
TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or facility. In this
instance, the testing was conducted in the Aprima Medical Software corporate
headquarters in the main training room. For testing, the computer used a
Fujitsu T902 running Microsoft Windows 8. The participants used whatever
input method he or she was most comfortable with from among mouse,
keyboard, touchpad, stylus or voice input when interacting with the EHRUT.
The EHRUT used a 12.7-inch diagonal display running 1280x720 pixels at 32
bit color. The application was set up by the administrator according to the
vendor's documentation. The application itself was running on a Dell 620
server running Windows Server 2008 using a test database on a LAN
connection. Technically, the system performance (i.e., response time) was
representative to what actual users would experience in a field implementation.
Additionally, participants were instructed not to change any of the default
system settings (such as control of font size).
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used,
including:
1. Informed Consent
2. Moderator's Guide
3. Post-test Questionnaire
4. Incentive Receipt and Acknowledgment Form
Examples of these documents can be found in Appendices 3-6 respectively.
The Moderator's Guide was devised so as to be able to capture required data.
The participant's interaction with the EHRUT was captured and recorded
digitally with screen capture software running on the test machine.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 12 of 46
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each participant:
Thank you for participating in this study. Your input is very important.
Our session today will last about 30 minutes. During that time you will
use an instance of an electronic health record. I will ask you to
complete a few tasks using this system and answer some questions.
You should complete the tasks as quickly as possible making as few
errors as possible. Please try to complete the tasks on your own
following the instructions very closely. Please note that we are not
testing you we are testing the system, therefore if you have difficulty all
this means is that something needs to be improved in the system. I will
be here in case you need specific help, but I am not able to instruct you
or provide help in how to use the application. Overall, we are interested
in how easy (or how difficult) this system is to use, what in it would be
useful to you, and how we could improve it. I did not have any
involvement in its creation, so please be honest with your opinions.
There are a variety of tools that record screens and transmit those
recordings across a local area network for remote observations. All of
the information that you provide will be kept confidential and your name
will not be associated with your comments at any time. Should you feel
it necessary you are able to withdraw at any time during the testing.
Following the procedural instructions, participants were shown the EHR and as
their first task, were given time (10 minutes) to explore the system and make
comments. Once this task was complete, the administrator gave the following
instructions:
For each task, I will read the description to you and say "Begin." At that
point, please perform the task and say "Done" once you believe you
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 13 of 46
have successfully completed the task. I would like to request that you
not talk aloud or verbalize while you are doing the tasks. I will ask you
your impressions about the task once you are done.
Participants were then given 13 tasks to complete. Tasks are listed in the
moderator's guide in the Appendix.
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the
Usability of Electronic Health Records, EHRs should support a process that
provides a high level of usability for all users. The goal is for users to interact
with the system effectively, efficiently, and with an acceptable level of
satisfaction. To this end, metrics for effectiveness, efficiency and user
satisfaction were captured during the usability testing. The goals of the test
were to assess:
1. Effectiveness of EHRUT by measuring participant success rates and errors
2. Efficiency of EHRUT by measuring the average task time and path
deviations
3. Satisfaction with EHRUT by measuring ease of use ratings
DATA SCORING
The following table (Table 1) details how tasks were scored, errors evaluated,
and the time data analyzed.
Measures Rational and Scoring
Effectiveness: Task Success
A task was counted as a "Success" if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 14 of 46
Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator's Guide must be operationally defined by taking multiple measures of optimal performance and multiplying by some factor that allows some time buffer because the participants are presumably not trained to expert performance. Scoring used a factor of 2. Thus, if expert, optimal performance on a task was 80 seconds then allotted task time performance was 80 * 2 = 160 seconds. This ratio should be aggregated across tasks and reported with mean and variance scores.
Effectiveness: Task Failures
If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a "Failure." No task times were taken for errors. The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. This should also be expressed as the mean number of failed tasks per participant. On a qualitative level, an enumeration of errors and error types were collected.
Efficiency: Task Deviations
The participant's path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps in order to provide a ratio of path deviation. It is strongly recommended that task deviations be reported. Optimal paths (i.e., procedural steps) should be recorded when constructing tasks.
Efficiency: Task Time
Each task was timed from when the administrator said "Begin" until the participant said, "Done." If he or she failed to say "Done," the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.
Satisfaction: Participant's subjective impression of the ease of use of the
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 15 of 46
Task Rating application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate "Overall, this task was:" on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants' confidence in and likeability of the EHRUT overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, "I think I would like to use this system frequently," "I thought the system was easy to use," and "I would imagine that most people would learn to use this system very quickly." See full System Usability Score questionnaire in Appendix 5.
Table 1. Details of how observed data were scored.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 16 of 46
RESULTS
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods
specified in the Usability Metrics section above. Participants who failed to
follow session and task instructions had their data excluded from the analyses.
In this test cycle, there were no data excluded.
The usability testing results for the EHRUT are detailed below. The results
should be seen in light of the objectives and goals outlined in Section 3.2 Study
Design. The data should yield actionable results that, if corrected, yield
material, positive impact on user performance.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations (Observed / Optimal)
Mean(SD)
Min/Max Optimal
Mean (SD)
Mean (SD)
1. Task 1 6 100% (0.00)
0.5 (57/54) 50 (25)
0:16/1:33 0:17
0.3 (0.5) 4.8 (0.4)
2. Task 2 6 100% (0.00)
0.3 (62/60) 46 (14)
0:28/1:17 0:30
0.0 (0.0) 4.7 (0.5)
3. Task 3 6 100% (0.00)
0.3 (32/30) 20 (8)
0:11/0:31 0:11
0.0 (0.0) 5.0 (0.0)
4. Task 4 6 100% (0.00)
0.5 (39/36) 41 (8)
0:28/0:56 0:28
0.2 (0.4) 4.5 (0.8)
5. Task 5 6 100% (0.00)
0.3 (38/36) 33 (13)
0:22/1:00 0:22
0.2 (0.4) 4.3 (0.5)
The results from the SUS (System Usability Scale) scored the subjective
satisfaction with the system based on performance with these tasks to be: 83.
Broadly interpreted, scores under 60 represent systems with poor usability;
scores over 80 would be considered above average
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 17 of 46
DISCUSSION OF THE FINDINGS
The information presented and the way in which it was presented was well
received by the test users. Easy of use averaged above 4.6 across all
Medication list related tasks.
The scenario for using a filter was not realistic for the users, so they had some
difficultly understanding the reason for the task. This uncertainty caused some
users to slow down and verbalize their concern. Some of the variation in
timings between users was attributed to the quality of the scenario.
EFFECTIVENESS
The users were able to do what was expected with the product. Many times the
users went slightly off the optimal the path but did not lose their way. Users
with a small amount of time on the system improved dramatically. All users
rated the product either agree or strongly agree with regard to “I thought the
system was easy to use”. All users also rated “I found the system
unnecessarily complex” either disagree or strongly disagree.
EFFICIENCY
The number of path deviations on some tasks was above 2.0 this is large
number. For most tasks the number was at or below 1.0. Users were in almost
all cases able to quickly come back to the optimal path and complete the task.
This is true for the tasks in which the path deviations where the highest as well.
Variance on the timings show that the untrained users are not efficient using
Aprima. User learning is rapid; by task 5 the users were able to be more
efficient with task times being one half or less compared to the first 4 tasks.
SATISFACTION
User satisfaction was quite high. Overall user rating was 83.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 18 of 46
Question Agree or Strongly Agree
Disagree or Strongly Disagree
I think that I would like to use this system frequently
83% 0%
I found the system unnecessarily complex
0% 100%
I thought the system was easy to use
100% 0%
I think that I would need the support of a technical person to be able to use this system
17% 67%
I found the various functions in this system were well integrated
83% 0%
I thought there was too much inconsistency in this system
0% 100%
I would imagine that most people would learn to use this system very quickly
100% 0%
I found the system very cumbersome to use
33% 67%
I felt very confident using the system
100% 0%
I needed to learn a lot of things before I could get going with this system
0% 100%
“I found the system cumbersome to use” seems to be a polarizing question. In
future user testing this will have to be explored more, as we did not find this to
be reflected in the user comments and was not discovered until the data
analysis phase.
MAJOR FINDINGS
The information presented and the way in which it was presented was
well received by the test users. Easy of use averaged above 4.6
across all Medication list related tasks.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 19 of 46
The scenario for using a filter was not realistic for the users, so they
had some difficultly understanding the reason for the task. This
uncertainty caused some users to slow down and verbalize their
concern. Some of the variation in timings between users was
attributed to the quality of the scenario.
The task to change the filter showed a path that was unexpected.
Users modified the filter itself, rather than use a different filter. This
was considered a success, as the information the users were expected
was displayed on screen, and the task was repeatable without
degradation in usability or overall system performance.
AREAS FOR IMPROVEMENT
Recommend limited access to modify filters
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 20 of 46
APPENDICES
The following appendices include supplemental data for this usability test
report. Following is a list of the appendices provided:
1: Sample Recruiting screener
2: Participant demographics
3: Non-Disclosure Agreement (NDA) and Informed Consent Form
4: Example Moderator's Guide
5: System Usability Scale Questionnaire
6: Incentive receipt and acknowledgment form It is important to note that these
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 21 of 46
Appendix 1: SAMPLE RECRUITING SCREENER
The purpose of a screener to ensure that the participants selected represent
the target user population as closely as possible. (Portions of this sample
screener are taken from www.usability.gov/templates/index.html#Usability and
adapted for use.)
Recruiting Script for Recruiting Firm
Hello, my name is _______________, calling from [Insert name of recruiting
firm]. We are recruiting individuals to participate in a usability study for an
electronic health record. We would like to ask you a few questions to see if
you qualify and if would like to participate. This should only take a few minutes
of your time. This is strictly for research purposes. If you are interested and
qualify for the study, you will be paid to participate. Can I ask you a few
questions?
Customize this by dropping or adding questions so that it reflects your EHR's
primary audience
1. [If not obvious] Are you male or female? [Recruit a mix of participants]
2. Have you participated in a focus group or usability test in the past xx months? [If yes, Terminate]
3. Do you, or does anyone in your home, work in marketing research, usability research, web design [If yes, Terminate]
4. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record software or consulting company? [If yes, Terminate]
5. Which of the following best describes your age? [23 to 39; 40 to 59; 60 and older] [Recruit Mix]
6. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-American, Latino/a or Hispanic, etc.]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 22 of 46
7. Do you require any assistive technologies to use a computer? [if so, please describe] Professional Demographics Customize this to reflect your EHR's primary audience
8. What is your current position and title? (Must be healthcare provider)
□ RN: Specialty _______________________
□ Physician: Specialty _______________________
□ Resident: Specialty _______________________
□ Medical Assistant _______________________
□ Administrative Staff
□ Other [Terminate]
9. How long have you held this position?
10. Describe your work location (or affiliation) and environment? (Recruit according to the intended users of the application. e.g., private practice, health system, government clinic, etc.)
11. Which of the following describes your highest level of education? [e.g., high school graduate/GED, some college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)
12. Besides reading email, what professional activities do you do on the computer? [e.g., access EHR, research; reading news; shopping/banking; digital pictures; programming/word processing, etc.] [If no computer use at all, Terminate]
13. About how many hours per week do you spend on the computer? [Recruit according to the demographics of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]
14. What computer platform do you usually use? [e.g., Mac, Windows, etc.]
15. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]
16. In the last month, how often have you used an electronic health record?
17. How many years have you used an electronic health record?
18. How many EHRs do you use or are you familiar with?
19. How does your work environment handle patient records? [Recruit according to the demographics of the intended users]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 23 of 46
□ On paper
□ Some paper, some electronic
□ All electronic
Contact Information If the person matches your qualifications, ask
Those are all the questions I have for you. Your background matches the
people we're looking for. For your participation, you will be paid $300. Would
you be able to participate on [date, time]? [If so collect contact information]
May I get your contact information?
■ Name of participant:
■ Address:
■ City, State, Zip:
■ Daytime phone number:
■ Evening phone number:
■ Alternate [cell] phone number:
■ Email address:
Before your session starts, we will ask you to sign a release form allowing us to
record your session. The record will only be used internally for further study if
needed. Will you consent to be digitally recorded? This study will take place at
3330 Keller Springs Road, Suite 201, Carrollton, TX 75006. I will confirm your
appointment a couple of days before your session and provide you with
directions to our office. What time is the best time to reach you?
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 24 of 46
Appendix 2: PARTICIPANT DEMOGRAPHICS
Following is a high-level overview of the participants in this study.
Gender
Men: 0 Women: 6 Total (participants): 6
Occupation/Role
RN/BSN: 0 Physician: 2 Admin Staff: 4 Total (participants): 6
Years of Experience
Facility Use of EHR All paper: 1 Some paper, some electronic: 5 All electronic: 0 Total (participants): 6
The full break down is included in the report above.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 25 of 46
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM
Non-Disclosure Agreement THIS AGREEMENT is entered into as of _________________, 2013, between ________________ ("the Participant") and the testing organization Aprima Medical Software located at 3330 Keller Springs Rd. Suite 201, Carrollton, TX 75006. The Participant acknowledges his or her voluntary participation in today's usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by Test Company, or otherwise acquired by the Participant, in the course of today's study. By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts. Any information the Participant acquires relating to this product during this study is confidential and proprietary to Test Company and is being disclosed solely for the purposes of the Participant's participation in today's usability study. By signing this form the Participant acknowledges that s/he will receive monetary compensation for feedback and will not disclose this confidential information obtained today to anyone else or any other organizations. Participant's printed name: ___________________________________________________ Signature: __________________________________________________________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 26 of 46
Informed Consent Test Company would like to thank you for participating in this study. The purpose of this study is to evaluate an electronic health records system. If you decide to participate, you will be asked to perform several tasks using the prototype and give your feedback. The study will last about 30 minutes. At the conclusion of the test, you will be compensated for your time. Agreement I understand and agree that as a voluntary participant in the present study conducted by Test Company I am free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study conducted and videotaped by the Test Company. I understand and consent to the use and release of the videotape by Test Company. I understand that the information and videotape is for research purposes only and that my name and image will not be used for any purpose other than research. I relinquish any rights to the videotape and understand the videotape may be copied and used by Test Company without further permission. I understand and agree that the purpose of this study is to make software applications more useful and usable in the future. I understand and agree that the data collected from this study may be shared with outside of Test Company and Test Company's client. I understand and agree that data confidentiality is assured, because only de- identified data - i.e., identification numbers not names - will be used in analysis and reporting of the results. I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can leave at any time. Please check one of the following: □ YES, I have read the above statement and agree to be a participant. □ NO, I choose not to participate in this study. Signature: _________________________________ Date: _______________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 27 of 46
Appendix 4: EXAMPLE MODERATOR'S GUIDE
Only four tasks are presented here for illustration. EHRUT Usability Test Moderator's Guide Administrator _____________________
Data Logger _____________________
Date _________________ Time _____________________
Participant # _________
Location _____________________
Prior to testing
■ Confirm schedule with Participants ■ Ensure EHRUT lab environment is running properly ■ Ensure Consent forms are complete for all participants ■ Ensure lab and data recording equipment is running properly
Prior to each participant: ■ Reset application ■ Restore application to base state using script restoredb.bat ■ Start session recordings with GTM Recording
Prior to each task: ■ Reset application to starting point for next task
After each participant: ■ End session recordings with tool ■ Fill out all paper work and acknowledgement form.
After all testing ■ Scan all forms and shred originals ■ Back up all video and data files
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 28 of 46
Orientation (5 minutes) Thank you for participating in this study. Our session today will last 2 hours today. During that time you will take a look at an electronic health record system. I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with anything to do with the system itself. Please save your detailed comments until the end of a task or the end of the session as a whole when we can discuss freely. I did not have any involvement in its creation, so please be honest with your opinions. The product you will be using today is describe the state of the application, i.e., production version, early prototype, etc. Some of the data may not make sense as it is placeholder data. We are recording the audio and screenshots of our session today. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Do you have any questions or concerns? Preliminary Questions (5 minutes) What is your job title / appointment? How long have you been working in this role? What are some of your main responsibilities? Tell me about your experience with electronic health records.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 29 of 46
Preview: First Impressions (6 Minutes) This is the application you will be working with. Have you heard of it? Yes ___ No ___ If so, tell me what you know about it. Show test participant the EHRUT. Please don't click on anything just yet. What do you notice? What are you able to do here? Please be specific. Notes / Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 30 of 46
Task 1: Create an Order – Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC The patient you are examining 5’ 10” 285 lbs. male needs a differential diagnosis of 034.0 Streptococcal Sore Throat and needs a 87880 rapid strep to confirm diagnosis. Enter Vitals, Dx and SP into the system Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, enter 5 feet 10 inches and 285 lbs 2. go to the DX tab and Enter the diagnosis 034.0 Streptococcal Sore Throat 3. Note Alert for Obesity and Hypertension in CDS slider 4. Still inside FNC, go to the SP tab 5. Enter procedure code 87880 Strep A Assay 6. Still inside FNC, click the Close button to close the note, save the note as incomplete 7. Click okay.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 31 of 46
Task 2: Utilize an existing Clinical Decision Support Alert – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You run are in consultation with a patient and his parent who is 71 years old. You want to quickly see if there are any Clinical Support Alerts for the parent. Go to the Appointment to review CDS Alerts and look at the source information for the alert. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
Open Patient Norman Wood
Open Update Appointment from Envelope
Open Clinical Decision Support Tab
Note Fall Risk Rule on Slider
Click on the Hyperlink for Fall Risk Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 32 of 46
Task 3: Creation of a Procedure Task Message to the Nurses - Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC
A patient you saw earlier that requested an order for a test at a different lab wants the lab found it was not covered by his insurance. He wants it to be performed in your office. You need to order a rapid strep test (87880) for the patient, as he is on his way back into the office.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open the scheduler 2. Find the appointment for Ed Wood 3. Click the Blue Folder (Open/Create note) 4. Click New->Procedure Task Message
a. Enter the Nurses b. Click Okay
5. Click to close the note a. Click Okay
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 33 of 46
Task 4: Approve a Lab result (5 Minutes) Take the participant to the starting point for the task. - DESKTOP
You have a lab result request message in your inbox to process. You need to review and approve the request.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations Approval of Lab Results
1. Locate the Lab Result Approval message in the Desktop Message pane 2. Click on the Time and Date Hyperlink or Plus (+) sign next to the message 3. Click on Process Message 4. From the Modify Message window 5. After review of the results, click the Approved All button
a. Your user (provider) name will auto populate along with the current date 6. Click Complete
a. This will simultaneously set the approved lab results and complete the message Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 34 of 46
Task 5: Document Drug Allergy in History – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are examining reports to you that she has been taking no meds because she has an anaphylactic reaction to Penicillins. Please document the allergy in History Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Drug Allergy
6. In the Allergen column, select Penicillins.
7. In the Reaction column, select anaphylaxis.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 35 of 46
Task 6: Drug Allergy Alert in Rx Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Need to prescribe Keflex to Mary Woods chart 1 tablet per day by mouth for 10 days Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Rx Tab
6. In the Drug field, enter Keflex
7. Click the find control
8. Select Keflex 500 mg
9. Pop up: Receive Drug Screening Warning: ALLERGY ALERT: The use of Keflex cross-sensitivity reaction based on a reported history of allergy to PENICILLINS. Do you want to continue? Click Yes
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 36 of 46
Task 7: Drug Drug Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP An agitated patient wants you to prescribe Prozac immediately. 20mg 1 time per day by mouth. Please prescribe the medication. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Find Mary Wood that has an appointment scheduled.
2. Click the Open/Create Note icon
3. FNC
4. Click Rx Tab
5. In the Drug field, enter Prozac
6. Click the find control
7. Select Prozac 20mg tablet
8. Pop up: Receive Drug Screening Warning: Duplicate Therapy: The use of Prozac is a duplicate therapy with Zoloft 50 mg. Do you want to continue?
a. Click No
9. Print -> Save and print Button
a. Print Dialog
10. Click ok to save and exit the note
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 37 of 46
Task 8: Medication List in OPS – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You are about to walk into an exam room for a med check on a patient. Go to One Page Summary and review the patient’s drug list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Go to Patient Demographics 2. Click Patient 3. Click One Page Summary 4. Click hyperlink in index for Current Medication 5. Verify the data is displayed correctly
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 38 of 46
Task 9: Medication List in Hx Tab with Filter – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
Previously very ill patient with a very large medication history is asking about her current medications. You want to compare the active/inactive medications with the maintenance medication list (which are two different filters in the system Go to Hx Tab and change the filter to show only Active Medications and them maintenance medications.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, go to the HX tab 2. Click on the Medication History 3. Review the items on the list 4. Click the search icon to change the filter 5. Change the filter to maintenance 6. Review the Maintenance Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 39 of 46
Task 10: Rx Refill Complete (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You have a few minutes to complete your drug refill request messages Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Messages
2. Locate message
3. Click on patient’s blue envelope.
4. Select Process Rx Refill
5. Medication History screen
6. Select the drug by clicking the Refill button.
7. Sig Writer screen
a. Verify information in sig
8. Click Ok
a. Return to the Modify Message screen
9. Click Complete.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 40 of 46
Task 11: import Allergies from external list - Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Allergy list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. Go to attachments – Allergies - CCD 3. Click the import allergies icon 4. Sulfasalazine -> Add 5. Penicillin V -> Merge/Add 6. Carbamazepine -> ignore/remove 7. Click Reconcile 8. Go to Hx
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 41 of 46
Task 12: Import Medications from external list – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Medication list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. View the medication CCD 3. Click the import medications icon 4. Simvastatin -> Add 5. Lorazepam -> Add 6. Lisinopril -> Add/Merge 7. Click reconcile 8. Go to patient Hx – Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 42 of 46
Task 13: Verify import in Allergy list and Medication List – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Please validate that the import of Allergy list and Medication list import has the expected information Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Note the CDS rule for the combination of the new allergy and medication
6. Click Drug Allergy
a. Verify the information is correct from the import
7. Click on the Medication History
a. Verify the information is correct from the import
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 43 of 46
Final Questions (10 Minutes) What was your overall impression of this system? What aspects of the system did you like most? What aspects of the system did you like least? Were there any features that you were surprised to see? What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? Compare this system to other systems you have used. Would you recommend this system to your colleagues? Administer the SUS
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 44 of 46
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
Strongly
disagree Strongly
Agree
1. I think that I would like to use this system frequently
1 2 3 4 5
2. I found the system unnecessarily complex
1 2 3 4 5
3. I thought the system was easy to use
1 2 3 4 5
4. I think that I would need the support of a technical person to be able to use this system
1 2 3 4 5
5. I found the various functions in this system were well integrated
1 2 3 4 5
6. I thought there was too much inconsistency in this system
1 2 3 4 5
7. I would imagine that most people would learn to use this system very quickly
1 2 3 4 5
8. I found the system very cumbersome to use
1 2 3 4 5
9. I felt very confident using the system
1 2 3 4 5
10. I needed to learn a lot of things before I could get going with this system
1 2 3 4 5
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 45 of 46
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM
Acknowledgement of Receipt I hereby acknowledge receipt of $ _____ for my participation in a research study run by Aprima Medical Software. Printed Name: _____________________________________________________________ Address: __________________________________________________________________ __________________________________________________________________________ Signature: _______________________________________________ Date: ____________ Usability Researcher: ________________________________________________________ Signature of Usability Researcher: ____________________________ Date: ______________ Witness: ___________________________________________________________________ Witness Signature: ________________________________________ Date: ______________
EHR USABILITY TEST REPORT
OF
APRIMA 2014
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
Aprima 2014 Date of Usability Test: 05/21/13-05/23/13 Date of Report: 05/27/13 Report Prepared By: Aprima Medical Software Project Management Cornell Hardman (214) 466-8107 Aprima Medical Software 3330 Keller Springs Road Suite 201 Carrollton, TX 75006
Table of Contents EXECUTIVE SUMMARY ................................................................................................................. 3
Major findings ............................................................................................................................... 5
Areas for improvement ................................................................................................................. 5
INTRODUCTION ............................................................................................................................. 6
METHOD ......................................................................................................................................... 6
PARTICIPANTS ........................................................................................................................... 6
STUDY DESIGN .......................................................................................................................... 8
TASKS ......................................................................................................................................... 9
PROCEDURES ............................................................................................................................ 9
TEST LOCATION ...................................................................................................................... 10
TEST ENVIRONMENT .............................................................................................................. 10
TEST FORMS AND TOOLS ...................................................................................................... 11
PARTICIPANT INSTRUCTIONS ............................................................................................... 12
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 2 of 46
USABILITY METRICS ............................................................................................................... 13
DATA SCORING ........................................................................................................................ 13
RESULTS ...................................................................................................................................... 16
DATA ANALYSIS AND REPORTING ........................................................................................ 16
DISCUSSION OF THE FINDINGS ............................................................................................ 17
EFFECTIVENESS ..................................................................................................................... 17
EFFICIENCY .............................................................................................................................. 17
SATISFACTION ......................................................................................................................... 18
MAJOR FINDINGS .................................................................................................................... 19
AREAS FOR IMPROVEMENT .................................................................................................. 19
APPENDICES ................................................................................................................................ 20
Appendix 1: SAMPLE RECRUITING SCREENER .................................................................... 21
Appendix 2: PARTICIPANT DEMOGRAPHICS ........................................................................ 24
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ........... 25
Appendix 4: EXAMPLE MODERATOR'S GUIDE ...................................................................... 27
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................. 44
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM ................................. 45
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 3 of 46
EXECUTIVE SUMMARY
A usability test of Aprima 2014 ambulatory EHR was conducted on the dates of
May 21-23 of 2013 in The Aprima Medical Software corporate office by Aprima
Medical Software Project Management team. The purpose of this test was to
test and validate the usability of the current user interface, and provide
evidence of usability in the EHR Under Test (EHRUT). During the usability
test, 2 healthcare providers and 4 other knowledgeable clinic staff matching the
target demographic criteria served as participants and used the EHRUT in
simulated, but representative tasks. This study collected performance data on
13 tasks including 3 tasks specifically designed to test external information
reconciliation features that are typically conducted on an EHR:
Task 1 – Allergy Reconciliation
Task 2 – Medication Reconciliation
Task 3 – Verify import in Allergy list and Medication List
During the 2-hour one-on-one usability test, each participant was greeted by
the administrator and asked to review and sign an informed consent/release
form (included in Appendix 3); they were instructed that they could withdraw at
any time. Participants had a mix of experience with the EHR. The
administrator introduced the test, and instructed participants to complete a
series of tasks (given one at a time) using the EHRUT. During the testing, the
administrator timed the test and, along with the data logger(s) recorded user
performance data on paper and electronically. The administrator did not give
the participant assistance in how to complete the task.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 4 of 46
Participant screens were recorded for subsequent analysis.
The following types of data were collected for each participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations Participant's satisfaction ratings of the system
All participant data was de-identified - no correspondence could be made from
the identity of the participant to the data collected. Following the conclusion of
the testing, participants were asked to complete a post-test questionnaire and
were compensated with $300 for their time. Various recommended metrics, in
accordance with the examples set forth in the NIST Guide to the Processes
Approach for Improving the Usability of Electronic Health Records, were used
to evaluate the usability of the EHRUT. Following is a summary of the
performance and rating data collected on the EHRUT.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations (Observed / Optimal)
Mean(SD)
Min/Max Optimal
Mean (SD)
Mean (SD)
1. Task 1 6 17% (0.37)
2.3 (62/48) 129 (0)
2:09/2:09 0:11
1.7 (0.5) 3.3 (0.5)
2. Task 2 6 100% (0.00)
0.8 (53/48) 24 (7)
0:11/0:34 0:11
0.3 (0.5) 3.8 (1.5)
3. Task 3 6 100% (0.00)
0.3 (38/36) 33 (13)
0:22/1:00 0:22
0.2 (0.4) 4.3 (0.5)
The results from the System Usability Scale scored the subjective satisfaction
with the system based on performance with these tasks to be: 83
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 5 of 46
In addition to the performance data, the following qualitative observations were
made:
Major findings
Users had a difficult time using the import screen. Only 1 user was
successful the first time. The functions are not in the context area of
the window and are often missed.
Once shown how to use the screen, users were able to successfully
and efficiently utilize the functionality. With verbalizations like “Now that
I know how to use it, that was easy”
Some users found the lack of feedback on the import disconcerting.
Significant consideration needs to be given before this screen is
reworked and ready for production
There are several ways to move data from each list, but the different
ways are not supportive of each other leading to confusion.
Areas for improvement
The design of the functionality needs to be reworked in a way that will
allow users to more easily access and successfully complete the task.
The import process needs to give feedback to the user that the
information has been migrated
When a user goes off the correct path, he will often take a very long
time and many more errors before finding the correct path again.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 6 of 46
INTRODUCTION
The EHRUT tested for this study was Aprima 2014 ambulatory EHR. Designed
to present medical information to healthcare providers in ambulatory medical
offices, the EHRUT consists of a user interface presented on a computer
display that stores and retrieves data from a database server through an
application server. The usability testing attempted to represent realistic
exercises and conditions. The purpose of this study was to test and validate
the usability of the current user interface, and provide evidence of usability in
the EHR Under Test (EHRUT). To this end, measures of effectiveness,
efficiency and user satisfaction, such as ease of access, number of mouse
clicks, time on task, time to repeat task, time to first entry and number of errors,
were captured during the usability testing.
METHOD
PARTICIPANTS
A total of 6 participants were tested on the EHRUT. Participants in the test
were physicians, nurses and other knowledgeable office staff. Participants
were recruited by Health Care Strategies and were compensated $300 for their
time. In addition, participants had no direct connection to the development of
or organization producing the EHRUT(s). Participants were not from the
testing or supplier organization. Participants were given the opportunity to
have an abbreviated orientation and a minimal level of training compared to the
amount the actual end users would have received. For the test purposes, end-
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 7 of 46
user characteristics were identified and translated into a recruitment screener
used to solicit potential participants; an example of a screener is provided in
Appendix 1.
Recruited participants had a mix of backgrounds and demographic
characteristics conforming to the recruitment screener. The following is a table
of participants by characteristics, including demographics, professional
experience, computing experience and user needs for assistive technology.
Participant names were replaced with Participant IDs so that an individual's
data cannot be tied back to individual identities.
Part ID
Gender Age Education Occupation /Role
Professional Experience
Computer Exp.
Product Exp.
Assistive Technology
Needs
1 142101 F 25 College Degree
Medical
Assistant 5 9 0/4
2 142102 F 42 College Degree
Medical
Assistant 3 13 0/1 3 142201 F 65 MD/PhD/DO Physician 38 4 0/0
4 142202 F 39 College Degree
Medical
Assistant 16 16 0/3
5 142301 F 31
Masters
Degree
Physician
Assistant 9 19 0/6
6 142302 F 30 College Degree
Medical
Assistant 8 13 0/7
9 participants (matching the demographics in the section on Participants) were
recruited and 6 participated in the usability test. 3 participants failed to show
for the study.
Participants were scheduled for 2 hour session with at least 1 hour in between
each session for debrief by the administrator(s) and data logger(s), and to reset
systems to proper test conditions. A spreadsheet was used to keep track of
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 8 of 46
the participant schedule, and included each participant's demographic
characteristics as provided by the recruiting firm.
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application
performed well - that is, effectively, efficiently, and with satisfaction - and areas
where the application failed to meet the needs of the participants. The data
from this test may serve as a baseline for future tests with an updated version
of the same EHR and/or comparison with other EHRs provided the same tasks
are used. In short, this testing serves as both a means to record or benchmark
current usability, but also to identify areas where improvements must be made.
During the usability test, participants interacted with a single EHR. Each
participant used the system in the same location, and was provided with the
same instructions. The system was evaluated for effectiveness, efficiency and
satisfaction as defined by measures collected and analyzed for each
participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks Number and types of errors Path deviations Participant's verbalizations (comments) Participant's satisfaction ratings of the system
Additional information about the various measures can be found in
Section 3.9 on Usability Metrics.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 9 of 46
TASKS
A number of tasks were constructed that would be realistic and representative
of the kinds of activities a user might do with this EHR, including:
Task 1 – Allergy Reconciliation
Task 2 – Medication Reconciliation
Task 3 – Verify import in Allergy list and Medication List
Tasks were selected based on their frequency of use, criticality of function, and
those that may be most troublesome for users. Tasks should always be
constructed in light of the study objectives.
PROCEDURES
Upon arrival, participants were greeted; their identity was verified and matched
with a name on the participant schedule. Participants were then assigned a
participant ID. Each participant reviewed and signed an informed consent and
release form (See Appendix 3). A representative from the test team witnessed
the participant's signature. To ensure that the test ran smoothly, two staff
members participated in this test, the usability administrator and the data
logger. The usability testing staff conducting the test was experienced usability
practitioners with 3 years of usability experience at GE Healthcare and
additional experience at other facilities. The administrator moderated the
session including administering instructions and tasks. The administrator also
monitored task times, obtained post-task rating data, and ensured notes on
participant comments were recorded. A second person served as the data
logger and took notes on task success, path deviations, number and type of
errors, and comments. Participants were instructed to perform the tasks (see
specific instructions below):
As quickly as possible making as few errors and deviations as possible.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 10 of 46
Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use.
Without using a think aloud technique.
For each task, the participants were given a written copy of the task. Task
timing began once the administrator finished reading the question. The task
time was stopped once the participant indicated they had successfully
completed the task. Scoring is discussed below in Section 3.9. Following the
session, the administrator gave the participant the post-test questionnaire (e.g.,
the System Usability Scale, see Appendix 5), compensated him or her for their
time, and thanked each individual for their participation.
Participants' demographic information, task success rate, time on task, errors,
deviations, verbal responses, and post-test questionnaire were recorded into a
spreadsheet.
Participants were thanked for their time and compensated. Participants signed
a receipt and acknowledgement form (See Appendix 6) indicating that they had
received the compensation.
TEST LOCATION
The test facility included a waiting area and a quiet testing room with a table,
computer for the participant, and recording computer for the administrator.
Only the participant and administrator and the data logger were in the test
room. There were no observers. To ensure that the environment was
comfortable for users, noise levels were kept to a minimum with the ambient
temperature within a normal range. All of the safety instruction and evacuation
procedures were valid, in place, and visible to the participants.
TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or facility. In this
instance, the testing was conducted in the Aprima Medical Software corporate
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 11 of 46
headquarters in the main training room. For testing, the computer used a
Fujitsu T902 running Microsoft Windows 8. The participants used whatever
input method he or she was most comfortable with from among mouse,
keyboard, touchpad, stylus or voice input when interacting with the EHRUT.
The EHRUT used a 12.7-inch diagonal display running 1280x720 pixels at 32
bit color. The application was set up by the administrator according to the
vendor's documentation. The application itself was running on a Dell 620
server running Windows Server 2008 using a test database on a LAN
connection. Technically, the system performance (i.e., response time) was
representative to what actual users would experience in a field implementation.
Additionally, participants were instructed not to change any of the default
system settings (such as control of font size).
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used,
including:
1. Informed Consent
2. Moderator's Guide
3. Post-test Questionnaire
4. Incentive Receipt and Acknowledgment Form
Examples of these documents can be found in Appendices 3-6 respectively.
The Moderator's Guide was devised so as to be able to capture required data.
The participant's interaction with the EHRUT was captured and recorded
digitally with screen capture software running on the test machine.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 12 of 46
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each participant:
Thank you for participating in this study. Your input is very important.
Our session today will last about 30 minutes. During that time you will
use an instance of an electronic health record. I will ask you to
complete a few tasks using this system and answer some questions.
You should complete the tasks as quickly as possible making as few
errors as possible. Please try to complete the tasks on your own
following the instructions very closely. Please note that we are not
testing you we are testing the system, therefore if you have difficulty all
this means is that something needs to be improved in the system. I will
be here in case you need specific help, but I am not able to instruct you
or provide help in how to use the application. Overall, we are interested
in how easy (or how difficult) this system is to use, what in it would be
useful to you, and how we could improve it. I did not have any
involvement in its creation, so please be honest with your opinions.
There are a variety of tools that record screens and transmit those
recordings across a local area network for remote observations. All of
the information that you provide will be kept confidential and your name
will not be associated with your comments at any time. Should you feel
it necessary you are able to withdraw at any time during the testing.
Following the procedural instructions, participants were shown the EHR and as
their first task, were given time (10 minutes) to explore the system and make
comments. Once this task was complete, the administrator gave the following
instructions:
For each task, I will read the description to you and say "Begin." At that
point, please perform the task and say "Done" once you believe you
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 13 of 46
have successfully completed the task. I would like to request that you
not talk aloud or verbalize while you are doing the tasks. I will ask you
your impressions about the task once you are done.
Participants were then given 13 tasks to complete. Tasks are listed in the
moderator's guide in the Appendix.
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the
Usability of Electronic Health Records, EHRs should support a process that
provides a high level of usability for all users. The goal is for users to interact
with the system effectively, efficiently, and with an acceptable level of
satisfaction. To this end, metrics for effectiveness, efficiency and user
satisfaction were captured during the usability testing. The goals of the test
were to assess:
1. Effectiveness of EHRUT by measuring participant success rates and errors
2. Efficiency of EHRUT by measuring the average task time and path
deviations
3. Satisfaction with EHRUT by measuring ease of use ratings
DATA SCORING
The following table (Table 1) details how tasks were scored, errors evaluated,
and the time data analyzed.
Measures Rational and Scoring
Effectiveness: Task Success
A task was counted as a "Success" if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 14 of 46
Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator's Guide must be operationally defined by taking multiple measures of optimal performance and multiplying by some factor that allows some time buffer because the participants are presumably not trained to expert performance. Scoring used a factor of 2. Thus, if expert, optimal performance on a task was 80 seconds then allotted task time performance was 80 * 2 = 160 seconds. This ratio should be aggregated across tasks and reported with mean and variance scores.
Effectiveness: Task Failures
If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a "Failure." No task times were taken for errors. The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. This should also be expressed as the mean number of failed tasks per participant. On a qualitative level, an enumeration of errors and error types were collected.
Efficiency: Task Deviations
The participant's path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps in order to provide a ratio of path deviation. It is strongly recommended that task deviations be reported. Optimal paths (i.e., procedural steps) should be recorded when constructing tasks.
Efficiency: Task Time
Each task was timed from when the administrator said "Begin" until the participant said, "Done." If he or she failed to say "Done," the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.
Satisfaction: Participant's subjective impression of the ease of use of the
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 15 of 46
Task Rating application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate "Overall, this task was:" on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants' confidence in and likeability of the EHRUT overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, "I think I would like to use this system frequently," "I thought the system was easy to use," and "I would imagine that most people would learn to use this system very quickly." See full System Usability Score questionnaire in Appendix 5.
Table 1. Details of how observed data were scored.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 16 of 46
RESULTS
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods
specified in the Usability Metrics section above. Participants who failed to
follow session and task instructions had their data excluded from the analyses.
In this test cycle, there were no data excluded.
The usability testing results for the EHRUT are detailed below. The results
should be seen in light of the objectives and goals outlined in Section 3.2 Study
Design. The data should yield actionable results that, if corrected, yield
material, positive impact on user performance.
Task
Measure N Task Success
Path Deviation
Task Time Errors Task Rating 5=Easy
# Mean (SD)
Deviations(Observed / Optimal)
Mean (SD)
Min/Max Optimal
Mean (SD)
Mean (SD)
1. Task 1 6 17% (0.37)
2.3 (62/48)
129 (0) 2:09/2:09 0:11
1.7 (0.5)
3.3 (0.5)
2. Task 2 6 100% (0.00)
0.8 (53/48)
24 (7) 0:11/0:34 0:11
0.3 (0.5)
3.8 (1.5)
3. Task 3 6 100% (0.00)
0.3 (38/36)
33 (13) 0:22/1:00 0:22
0.2 (0.4)
4.3 (0.5)
The results from the SUS (System Usability Scale) scored the subjective
satisfaction with the system based on performance with these tasks to be: 83.
Broadly interpreted, scores under 60 represent systems with poor usability;
scores over 80 would be considered above average
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 17 of 46
DISCUSSION OF THE FINDINGS
Users had a difficult time using the import screen. Only 1 user was successful
the first time. The functions are not in the context area of the window and are
often missed.
Once shown how to use the screen, users were able to successfully and
efficiently utilize the functionality. With verbalizations like “Now that I know how
to use it, that was easy”
Some users found the lack of feedback on the import disconcerting.
Significant consideration needs to be given before this screen is reworked and
ready for production
There are several ways to move data from each list, but the different ways are
not supportive of each other leading to confusion.
EFFECTIVENESS
The users were not able to do what was expected with this set of tasks. Many
times the users went slightly off the optimal and then completely lost their way.
Users with a small amount of time on the system improved dramatically. All
users rated the product either agree or strongly agree with regard to “I thought
the system was easy to use”. All users also rated “I found the system
unnecessarily complex” either disagree or strongly disagree.
EFFICIENCY
The number of path deviations on some tasks was above 2.0 this is large
number. For most tasks the number was at or below 1.0. Users were in almost
all cases able to quickly come back to the optimal path and complete the task.
This is true for the tasks in which the path deviations where the highest as well.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 18 of 46
Variance on the timings show that the untrained users are not efficient using
Aprima. User learning is rapid; by task 5 the users were able to be more
efficient with task times being one half or less compared to the first 4 tasks.
SATISFACTION
User satisfaction was quite high. Overall user rating was 83.
Question Agree or Strongly Agree
Disagree or Strongly Disagree
I think that I would like to use this system frequently
83% 0%
I found the system unnecessarily complex
0% 100%
I thought the system was easy to use
100% 0%
I think that I would need the support of a technical person to be able to use this system
17% 67%
I found the various functions in this system were well integrated
83% 0%
I thought there was too much inconsistency in this system
0% 100%
I would imagine that most people would learn to use this system very quickly
100% 0%
I found the system very cumbersome to use
33% 67%
I felt very confident using the system
100% 0%
I needed to learn a lot of things before I could get going with this system
0% 100%
“I found the system cumbersome to use” seems to be a polarizing question. In
future user testing this will have to be explored more, as we did not find this to
be reflected in the user comments and was not discovered until the data
analysis phase.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 19 of 46
MAJOR FINDINGS
Users had a difficult time using the import screen. Only 1 user was
successful the first time. The functions are not in the context area of
the window and are often missed.
Once shown how to use the screen, users were able to successfully
and efficiently utilize the functionality. With verbalizations like “Now that
I know how to use it, that was easy”
Some users found the lack of feedback on the import disconcerting.
Significant consideration needs to be given before this screen is
reworked and ready for production
There are several ways to move data from each list, but the different
ways are not supportive of each other leading to confusion.
AREAS FOR IMPROVEMENT
The design of the functionality needs to be reworked in a way that will
allow users to more easily access and successfully complete the task.
The import process needs to give feedback to the user that the
information has been migrated
When a user goes off the correct path, he will often take a very long
time and many more errors before finding the correct path again.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 20 of 46
APPENDICES
The following appendices include supplemental data for this usability test
report. Following is a list of the appendices provided:
1: Sample Recruiting screener
2: Participant demographics
3: Non-Disclosure Agreement (NDA) and Informed Consent Form
4: Example Moderator's Guide
5: System Usability Scale Questionnaire
6: Incentive receipt and acknowledgment form It is important to note that these
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 21 of 46
Appendix 1: SAMPLE RECRUITING SCREENER
The purpose of a screener to ensure that the participants selected represent
the target user population as closely as possible. (Portions of this sample
screener are taken from www.usability.gov/templates/index.html#Usability and
adapted for use.)
Recruiting Script for Recruiting Firm
Hello, my name is _______________, calling from [Insert name of recruiting
firm]. We are recruiting individuals to participate in a usability study for an
electronic health record. We would like to ask you a few questions to see if
you qualify and if would like to participate. This should only take a few minutes
of your time. This is strictly for research purposes. If you are interested and
qualify for the study, you will be paid to participate. Can I ask you a few
questions?
Customize this by dropping or adding questions so that it reflects your EHR's
primary audience
1. [If not obvious] Are you male or female? [Recruit a mix of participants]
2. Have you participated in a focus group or usability test in the past xx months? [If yes, Terminate]
3. Do you, or does anyone in your home, work in marketing research, usability research, web design [If yes, Terminate]
4. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record software or consulting company? [If yes, Terminate]
5. Which of the following best describes your age? [23 to 39; 40 to 59; 60 and older] [Recruit Mix]
6. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-American, Latino/a or Hispanic, etc.]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 22 of 46
7. Do you require any assistive technologies to use a computer? [if so, please describe] Professional Demographics Customize this to reflect your EHR's primary audience
8. What is your current position and title? (Must be healthcare provider)
□ RN: Specialty _______________________
□ Physician: Specialty _______________________
□ Resident: Specialty _______________________
□ Medical Assistant _______________________
□ Administrative Staff
□ Other [Terminate]
9. How long have you held this position?
10. Describe your work location (or affiliation) and environment? (Recruit according to the intended users of the application. e.g., private practice, health system, government clinic, etc.)
11. Which of the following describes your highest level of education? [e.g., high school graduate/GED, some college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)
12. Besides reading email, what professional activities do you do on the computer? [e.g., access EHR, research; reading news; shopping/banking; digital pictures; programming/word processing, etc.] [If no computer use at all, Terminate]
13. About how many hours per week do you spend on the computer? [Recruit according to the demographics of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]
14. What computer platform do you usually use? [e.g., Mac, Windows, etc.]
15. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]
16. In the last month, how often have you used an electronic health record?
17. How many years have you used an electronic health record?
18. How many EHRs do you use or are you familiar with?
19. How does your work environment handle patient records? [Recruit according to the demographics of the intended users]
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 23 of 46
□ On paper
□ Some paper, some electronic
□ All electronic
Contact Information If the person matches your qualifications, ask
Those are all the questions I have for you. Your background matches the
people we're looking for. For your participation, you will be paid $300. Would
you be able to participate on [date, time]? [If so collect contact information]
May I get your contact information?
■ Name of participant:
■ Address:
■ City, State, Zip:
■ Daytime phone number:
■ Evening phone number:
■ Alternate [cell] phone number:
■ Email address:
Before your session starts, we will ask you to sign a release form allowing us to
record your session. The record will only be used internally for further study if
needed. Will you consent to be digitally recorded? This study will take place at
3330 Keller Springs Road, Suite 201, Carrollton, TX 75006. I will confirm your
appointment a couple of days before your session and provide you with
directions to our office. What time is the best time to reach you?
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 24 of 46
Appendix 2: PARTICIPANT DEMOGRAPHICS
Following is a high-level overview of the participants in this study.
Gender
Men: 0 Women: 6 Total (participants): 6
Occupation/Role
RN/BSN: 0 Physician: 2 Admin Staff: 4 Total (participants): 6
Years of Experience
Facility Use of EHR All paper: 1 Some paper, some electronic: 5 All electronic: 0 Total (participants): 6
The full break down is included in the report above.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 25 of 46
Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM
Non-Disclosure Agreement THIS AGREEMENT is entered into as of _________________, 2013, between ________________ ("the Participant") and the testing organization Aprima Medical Software located at 3330 Keller Springs Rd. Suite 201, Carrollton, TX 75006. The Participant acknowledges his or her voluntary participation in today's usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by Test Company, or otherwise acquired by the Participant, in the course of today's study. By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts. Any information the Participant acquires relating to this product during this study is confidential and proprietary to Test Company and is being disclosed solely for the purposes of the Participant's participation in today's usability study. By signing this form the Participant acknowledges that s/he will receive monetary compensation for feedback and will not disclose this confidential information obtained today to anyone else or any other organizations. Participant's printed name: ___________________________________________________ Signature: __________________________________________________________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 26 of 46
Informed Consent Test Company would like to thank you for participating in this study. The purpose of this study is to evaluate an electronic health records system. If you decide to participate, you will be asked to perform several tasks using the prototype and give your feedback. The study will last about 30 minutes. At the conclusion of the test, you will be compensated for your time. Agreement I understand and agree that as a voluntary participant in the present study conducted by Test Company I am free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study conducted and videotaped by the Test Company. I understand and consent to the use and release of the videotape by Test Company. I understand that the information and videotape is for research purposes only and that my name and image will not be used for any purpose other than research. I relinquish any rights to the videotape and understand the videotape may be copied and used by Test Company without further permission. I understand and agree that the purpose of this study is to make software applications more useful and usable in the future. I understand and agree that the data collected from this study may be shared with outside of Test Company and Test Company's client. I understand and agree that data confidentiality is assured, because only de- identified data - i.e., identification numbers not names - will be used in analysis and reporting of the results. I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can leave at any time. Please check one of the following: □ YES, I have read the above statement and agree to be a participant. □ NO, I choose not to participate in this study. Signature: _________________________________ Date: _______________________
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 27 of 46
Appendix 4: EXAMPLE MODERATOR'S GUIDE
Only four tasks are presented here for illustration. EHRUT Usability Test Moderator's Guide Administrator _____________________
Data Logger _____________________
Date _________________ Time _____________________
Participant # _________
Location _____________________
Prior to testing
■ Confirm schedule with Participants ■ Ensure EHRUT lab environment is running properly ■ Ensure Consent forms are complete for all participants ■ Ensure lab and data recording equipment is running properly
Prior to each participant: ■ Reset application ■ Restore application to base state using script restoredb.bat ■ Start session recordings with GTM Recording
Prior to each task: ■ Reset application to starting point for next task
After each participant: ■ End session recordings with tool ■ Fill out all paper work and acknowledgement form.
After all testing ■ Scan all forms and shred originals ■ Back up all video and data files
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 28 of 46
Orientation (5 minutes) Thank you for participating in this study. Our session today will last 2 hours today. During that time you will take a look at an electronic health record system. I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with anything to do with the system itself. Please save your detailed comments until the end of a task or the end of the session as a whole when we can discuss freely. I did not have any involvement in its creation, so please be honest with your opinions. The product you will be using today is describe the state of the application, i.e., production version, early prototype, etc. Some of the data may not make sense as it is placeholder data. We are recording the audio and screenshots of our session today. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Do you have any questions or concerns? Preliminary Questions (5 minutes) What is your job title / appointment? How long have you been working in this role? What are some of your main responsibilities? Tell me about your experience with electronic health records.
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 29 of 46
Preview: First Impressions (6 Minutes) This is the application you will be working with. Have you heard of it? Yes ___ No ___ If so, tell me what you know about it. Show test participant the EHRUT. Please don't click on anything just yet. What do you notice? What are you able to do here? Please be specific. Notes / Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 30 of 46
Task 1: Create an Order – Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC The patient you are examining 5’ 10” 285 lbs. male needs a differential diagnosis of 034.0 Streptococcal Sore Throat and needs a 87880 rapid strep to confirm diagnosis. Enter Vitals, Dx and SP into the system Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, enter 5 feet 10 inches and 285 lbs 2. go to the DX tab and Enter the diagnosis 034.0 Streptococcal Sore Throat 3. Note Alert for Obesity and Hypertension in CDS slider 4. Still inside FNC, go to the SP tab 5. Enter procedure code 87880 Strep A Assay 6. Still inside FNC, click the Close button to close the note, save the note as incomplete 7. Click okay.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 31 of 46
Task 2: Utilize an existing Clinical Decision Support Alert – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You run are in consultation with a patient and his parent who is 71 years old. You want to quickly see if there are any Clinical Support Alerts for the parent. Go to the Appointment to review CDS Alerts and look at the source information for the alert. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
Open Patient Norman Wood
Open Update Appointment from Envelope
Open Clinical Decision Support Tab
Note Fall Risk Rule on Slider
Click on the Hyperlink for Fall Risk Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 32 of 46
Task 3: Creation of a Procedure Task Message to the Nurses - Ed Wood (5 Minutes) Take the participant to the starting point for the task. - FNC
A patient you saw earlier that requested an order for a test at a different lab wants the lab found it was not covered by his insurance. He wants it to be performed in your office. You need to order a rapid strep test (87880) for the patient, as he is on his way back into the office.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open the scheduler 2. Find the appointment for Ed Wood 3. Click the Blue Folder (Open/Create note) 4. Click New->Procedure Task Message
a. Enter the Nurses b. Click Okay
5. Click to close the note a. Click Okay
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 33 of 46
Task 4: Approve a Lab result (5 Minutes) Take the participant to the starting point for the task. - DESKTOP
You have a lab result request message in your inbox to process. You need to review and approve the request.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations Approval of Lab Results
1. Locate the Lab Result Approval message in the Desktop Message pane 2. Click on the Time and Date Hyperlink or Plus (+) sign next to the message 3. Click on Process Message 4. From the Modify Message window 5. After review of the results, click the Approved All button
a. Your user (provider) name will auto populate along with the current date 6. Click Complete
a. This will simultaneously set the approved lab results and complete the message Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 34 of 46
Task 5: Document Drug Allergy in History – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are examining reports to you that she has been taking no meds because she has an anaphylactic reaction to Penicillins. Please document the allergy in History Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Drug Allergy
6. In the Allergen column, select Penicillins.
7. In the Reaction column, select anaphylaxis.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 35 of 46
Task 6: Drug Allergy Alert in Rx Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Need to prescribe Keflex to Mary Woods chart 1 tablet per day by mouth for 10 days Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Click Rx Tab
6. In the Drug field, enter Keflex
7. Click the find control
8. Select Keflex 500 mg
9. Pop up: Receive Drug Screening Warning: ALLERGY ALERT: The use of Keflex cross-sensitivity reaction based on a reported history of allergy to PENICILLINS. Do you want to continue? Click Yes
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 36 of 46
Task 7: Drug Drug Interaction – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP An agitated patient wants you to prescribe Prozac immediately. 20mg 1 time per day by mouth. Please prescribe the medication. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Find Mary Wood that has an appointment scheduled.
2. Click the Open/Create Note icon
3. FNC
4. Click Rx Tab
5. In the Drug field, enter Prozac
6. Click the find control
7. Select Prozac 20mg tablet
8. Pop up: Receive Drug Screening Warning: Duplicate Therapy: The use of Prozac is a duplicate therapy with Zoloft 50 mg. Do you want to continue?
a. Click No
9. Print -> Save and print Button
a. Print Dialog
10. Click ok to save and exit the note
Comments: Observed Errors and Verbalizations: Comments: Rating: Overall, this task was: ___________ Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 37 of 46
Task 8: Medication List in OPS – Mary Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You are about to walk into an exam room for a med check on a patient. Go to One Page Summary and review the patient’s drug list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Go to Patient Demographics 2. Click Patient 3. Click One Page Summary 4. Click hyperlink in index for Current Medication 5. Verify the data is displayed correctly
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 38 of 46
Task 9: Medication List in Hx Tab with Filter – Mary Wood (5 Minutes)
Take the participant to the starting point for the task. - DESKTOP
Previously very ill patient with a very large medication history is asking about her current medications. You want to compare the active/inactive medications with the maintenance medication list (which are two different filters in the system Go to Hx Tab and change the filter to show only Active Medications and them maintenance medications.
Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Inside FNC, go to the HX tab 2. Click on the Medication History 3. Review the items on the list 4. Click the search icon to change the filter 5. Change the filter to maintenance 6. Review the Maintenance Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 39 of 46
Task 10: Rx Refill Complete (5 Minutes) Take the participant to the starting point for the task. - DESKTOP You have a few minutes to complete your drug refill request messages Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Messages
2. Locate message
3. Click on patient’s blue envelope.
4. Select Process Rx Refill
5. Medication History screen
6. Select the drug by clicking the Refill button.
7. Sig Writer screen
a. Verify information in sig
8. Click Ok
a. Return to the Modify Message screen
9. Click Complete.
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 40 of 46
Task 11: import Allergies from external list - Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Allergy list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. Go to attachments – Allergies - CCD 3. Click the import allergies icon 4. Sulfasalazine -> Add 5. Penicillin V -> Merge/Add 6. Carbamazepine -> ignore/remove 7. Click Reconcile 8. Go to Hx
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 41 of 46
Task 12: Import Medications from external list – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP The patient you are seeing has a C-CDA that needs to be checked against your record. Start with reconciling the Medication list. Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Open attachments 2. View the medication CCD 3. Click the import medications icon 4. Simvastatin -> Add 5. Lorazepam -> Add 6. Lisinopril -> Add/Merge 7. Click reconcile 8. Go to patient Hx – Medications
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 42 of 46
Task 13: Verify import in Allergy list and Medication List – Norman Wood (5 Minutes) Take the participant to the starting point for the task. - DESKTOP Please validate that the import of Allergy list and Medication list import has the expected information Success: (Yes / No) □ Easily completed □ Completed with difficulty or help □ Not completed Comments: Task Time: _____________ Seconds Optimal Path: □ Correct □ Minor Deviations / Cycles □ Major Deviations
1. Start at PRM desktop calendar
2. Find Mary Wood that has an appointment scheduled.
3. Click the Open/Create Note icon
4. FNC
5. Note the CDS rule for the combination of the new allergy and medication
6. Click Drug Allergy
a. Verify the information is correct from the import
7. Click on the Medication History
a. Verify the information is correct from the import
Comments: Observed Errors and Verbalizations: Comments: Rating:
Overall, this task was: ___________
Show participant written scale: "Very Easy " (1) to "Very Difficult" (5) Administrator / Note taker Comments:
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 43 of 46
Final Questions (10 Minutes) What was your overall impression of this system? What aspects of the system did you like most? What aspects of the system did you like least? Were there any features that you were surprised to see? What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? Compare this system to other systems you have used. Would you recommend this system to your colleagues? Administer the SUS
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 44 of 46
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
Strongly
disagree Strongly
Agree
1. I think that I would like to use this system frequently
1 2 3 4 5
2. I found the system unnecessarily complex
1 2 3 4 5
3. I thought the system was easy to use
1 2 3 4 5
4. I think that I would need the support of a technical person to be able to use this system
1 2 3 4 5
5. I found the various functions in this system were well integrated
1 2 3 4 5
6. I thought there was too much inconsistency in this system
1 2 3 4 5
7. I would imagine that most people would learn to use this system very quickly
1 2 3 4 5
8. I found the system very cumbersome to use
1 2 3 4 5
9. I felt very confident using the system
1 2 3 4 5
10. I needed to learn a lot of things before I could get going with this system
1 2 3 4 5
Company Confidential © 2012-2013 Aprima Medical Software 5/27/2013
Page 45 of 46
Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM
Acknowledgement of Receipt I hereby acknowledge receipt of $ _____ for my participation in a research study run by Aprima Medical Software. Printed Name: _____________________________________________________________ Address: __________________________________________________________________ __________________________________________________________________________ Signature: _______________________________________________ Date: ____________ Usability Researcher: ________________________________________________________ Signature of Usability Researcher: ____________________________ Date: ______________ Witness: ___________________________________________________________________ Witness Signature: ________________________________________ Date: ______________
Aprima PRM Usability Standards © 2015 Aprima Medical Software, Inc. All rights reserved. Confidential and Proprietary. Aprima is a registered trademark of Aprima Medical Software. All other trademarks are the property of their respective holders. Reference Number: 8022.02
Ref: 8022.02 3
Contents Preamble .................................................................................................... 5
Definition and Standards .................................................................................. 5
Definitions .............................................................................................. 5
International Standards ............................................................................... 5
History of the Software Development ................................................................... 5
Previous Generations .................................................................................. 6
Existing User Base ...................................................................................... 6
Definition of User Groups and Qualifications ........................................................... 7
Clinical User Groups ................................................................................... 7
Non-Clinical User Groups ............................................................................. 7
Definition of User-Centered Development Process .................................................... 7
Definition of User Requirements ......................................................................... 9
Easy to Implement, Train, and Support ............................................................. 9
Improve the Practice’s Overall Practice Efficiency .............................................. 10
User Context and Environment .......................................................................... 10
Interaction with Patients ............................................................................ 10
Data Collection ........................................................................................ 11
Data Management ..................................................................................... 11
Consistency and Usability ................................................................................ 11
Global Screen Layout ................................................................................. 11
Templates and Fonts ................................................................................. 12
Color Schemes ......................................................................................... 12
Quality Management ...................................................................................... 12
System Analysis ........................................................................................ 12
Development ........................................................................................... 12
Testing .................................................................................................. 13
Regression Testing ................................................................................ 13
Performance Testing ............................................................................. 13
Manual and Automation Testing ................................................................ 13
User Acceptance Testing ........................................................................ 14
Documentation and Training ........................................................................ 14
Online Help ........................................................................................ 14
User’s Guides and Other Documents ........................................................... 14
4 Ref: 8022.02
Training ............................................................................................ 15
Customization Potential and Processes ................................................................ 15
General Adaptation to User Specialties ............................................................ 15
Language and Terminologies ........................................................................ 16
Clinical Pathways ..................................................................................... 16
Workflow Management ............................................................................... 16
Test Management .......................................................................................... 16
Post-Market Tests ..................................................................................... 17
Conclusion .................................................................................................. 17
Ref: 8022.02 5
Preamble Aprima Medical Software, Inc. is dedicated to easing the transition for physicians moving from either a paper-based environment, or another electronic health record (EHR), to one where information technology makes it easier to practice medicine and run the business of a practice. Since 1998, Aprima has helped thousands of doctors improve quality of care, patient satisfaction, the bottom line, and their own quality of life.
Our EHR, Practice Management, and Revenue Cycle Management solutions are designed to complement and simplify practice workflow. We serve physicians nationwide, in primary care and more than 30 specialties, from solo practitioners to large, multi-site clinics. Clients benefit from our vast experience in healthcare IT, the long-term vision of our leadership, our advanced technology that physicians use to optimize their practice, and our U.S.-based client support.
The 2015 version of the Aprima Patient Relationship Manager (Aprima PRM) system was developed based on over 20 years of experience with software applications for ambulatory clinics. The system was developed together with clinical specialists, software engineering experts, and IT professionals.
Definition and Standards
Definitions
Usability of software is defined as the ease of use and learnability. Usability includes methods for analyzing user needs and the benchmarking of “state of the art” health IT applications.
International Standards
Aprima PRM was developed using some concepts and practices from international guidelines and standards (ISO), but Aprima has chosen to implement other best practices not yet codified in international standards. In general, Aprima is transitioning from a waterfall software development methodology to an agile development methodology. Some aspects of each development methodology have had an impact on the Aprima PRM product.
History of the Software Development The software development of Aprima PRM 2015 is based on a long history of software engineering in the area of health IT. Aprima Medical Software, previously iMedica, started developing health IT software in 1998.
6 Ref: 8022.02
Previous Generations
The first PRM generation was developed by the iMedica company. In 2011, the Aprima PRM application was developed following guidelines for usability of health IT applications. In 2013, the relevant components have been redesigned according to the ONC Meaningful Use requirements.
Existing User Base
The existing Aprima users play an important role in the improvement and update processes of the Aprima PRM software. The Aprima product team holds regular user feedback sessions via web session meetings and in-person workshops at Aprima User Conferences to gather critical user input into the enhancement, modification, and quality improvements for the Aprima PRM product. The results of these sessions and review processes have been factored into the customization and redesign processes of the Aprima PRM system.
Even the most brilliant physician can’t go it alone, and neither can we. Aprima and its staff are members of the following organizations that help us stay fully abreast of all the latest healthcare-related developments and news, from market trends to technological breakthroughs to regulatory requirements:
• American Academy of Family Physicians
• MGMA
• CHIT
• HIMS Electronic Health Record Association
• HIMSS
• Software advice
• Physicians Alliance Inc.
• Professional Association of Healthcare Office Management
Ref: 8022.02 7
Aprima is also very responsive to new government regulations and standards for electronic health record (EHR) systems. Aprima also participates in a number of quality and safety organizations to ensure that Aprima PRM meets the highest standard of quality, performance, and usability.
Definition of User Groups and Qualifications The Aprima PRM system is typically used by these specific user groups:
• Physicians and other providers of ambulatory medical services (clinical user groups)
• Ambulatory clinic support personnel
• Administrators and members of the practice management
• Revenue cycle managers and billers
Clinical User Groups
The clinical user groups typically include physicians, nurses, medical assistants, and lab specialists. They use the system when patients are with them in an exam or treatment area. The clinical user groups typically enter all medical data, such as medical history, general exam, vital signs, HPI, and chief complaint. These users follow clinical pathways which they can implement together with the Aprima product specialists within the customization process.
Non-Clinical User Groups
All other team members in an ambulatory clinical environment who are not defined as clinical users are classified as non-clinical users. They enter billing codes, generate reports and letters, run analyses, manage patient flow through the practice, schedule appointments, and perform any other administrative tasks in the practice. These personnel are typically members of the ambulatory clinic support personnel, administrators, and members of the practice management team and/or an outsourced revenue cycle management team.
Definition of User-Centered Development Process Aprima has incorporated a unique user-centered design process that includes an iterative approach to development. Although the full process is comprised of internal processes adapted from a long history of successful design. This process applies to each of the EHR Certification criteria below:
• §170.314(a)(1) Computerized provider order entry
• §170.314(a)(2) Drug-drug, drug-allergy interaction checks
• §170.314(a)(6) Medication list
• §170.314(a)(7) Medication allergy list
8 Ref: 8022.02
• §170.314(a)(8) Clinical decision support
• §170.314(b)(3) Electronic prescribing
• §170.314(b)(4) Clinical information reconciliation
Aprima’s design is based upon an understanding of users and their tasks and environments. We involve our users throughout an iterative design and development process to ensure that the design is driven and refined by user-centered evaluation. Our design team includes multidisciplinary skills and perspectives to ensure that our design addresses the whole user experience.
The design for new functionality, which includes any new feature or enhancement to existing features, is based on user enhancement requests. User enhancement requests entered into Salesforce Ideas are voted on by other users, which determines a ranking of requests by the user population as a whole. Product management reviews of these requests and their rankings to incorporate the enhancements into the product design. Enhancement requests from direct user contact and other user input are also reviewed for incorporation..
The definitions of users groups and their characteristics are included in the projects defined for development. Also included is knowledge of the user environments in which the functionality will be used (i.e. Mobile, Remote, Thick Client).
User involvement in the design and development includes a number of web-sharing meetings to discuss and review future and current product design. Aprima incorporate the feedback from the schedule conferences by capturing the requests/feedback as work items to be added to the product backlog for evaluation. Meeting notes and feedback are documented and provided to all participants, and screenshots are captured and mocked. These meetings include:
• Weekly Doctor Advisory Design Meeting
• Monthly Physician/Customer Design Meeting
• Weekly Revenue Cycle Management Design Meeting
• Monthly Practice Management Design Meeting
• Internal Brown-Bag meetings
• Quarterly Management Roadmap Review
• Yearly User Conference
Other sources of user input may include, but is not limited to, the following:
• Aprima Sales and Marketing provide feedback from customer demonstrations on features and functions they would like to see.
• Requests for proposals contain detailed lists of functionality requested by perspective customers.
• Product Analysts review various federal and state regulations to determine additional functionality.
Ref: 8022.02 9
From this input, the design team prioritizes and ranks identified work items to determine the high priority items that most effect the design. User priorities are identified in Salesforce Ideas, and then reviewed and evaluated by the product owners. Once work items are identified from these priorities, then work items are created in Team Foundation Services, where they are ranked and tracked. Work items are then addressed based on priority, resource availability, and impact/risk on current design.
Each update to the product is developed with an iterative process. The key steps within each version are:
• Define the users, context, and expected tasks
• Develop prototypes (storyboards and drawings)
• Incorporate feedback from stakeholders and users
• Develop functionality
• Review and incorporate feedback
• Release user testing (release candidate) product version to limited user group
• Incorporate feedback from the release candidate users
• Release the General Release product version
The develop process for any project must keep in mind general user concerns. This includes such things as typical use of the system is eight or more hours a day, that users are concerned about the number of clicks needed to complete any task, and that users need to be able to complete tasks as quickly and efficiently as possible.
The design team for all development projects is cross-functional. Design teams include members from Support, Training, Sales, Marketing, Quality Assurance, Engineering, and customers (including Doctor Advisors), as needed for each specific project. Team members provide input throughout the process on both the product and related materials.
Product design includes user documentation and training to ensure that users are aware of and educated on product functionality.
Definition of User Requirements The user requirements depend on the tasks the user groups have to perform within the software. Therefore, the development of the user interface and the test processes generally address these user design concepts.
Easy to Implement, Train, and Support
The introduction of a PRM system means significant change for the entire clinical and non-clinical team. The easy to implement, train, and support requirements allow the Aprima PRM system to be installed and used by all of the clinic’s personnel in a short amount of time. The
10 Ref: 8022.02
involvement of our user base in designing the product facilitates the rapid deployment of the system within clinical settings.
Any PM/EHR system is a substantial investment by the practice, and to become productive with it is an important goal of the development process.
Improve the Practice’s Overall Practice Efficiency
Aprima’s goal is to save the practice time, improve their patient care, and get them paid promptly. Our goal is to make the overall practice more efficient. This efficiency also meets the new Meaningful Use standards.
The following components are identified for an efficient and easy to use system:
• Clinical pathways with structured exam templates dedicated to the encounter type of the patient
• Intelligent pick lists which represent the most common data codes and descriptions for the corresponding chart area
• Chart monitor functionality which reminds the user to enter important data which has been forgotten
The Aprima PRM system is highly configurable and customizable. It is easy to configure for almost any type of practice environment ranging from home-health to typical ambulatory care practices.
The user experience in any of these environments guides the development of new functionality, product improvements, and third-party product adoption.
User Context and Environment A great deal of attention is paid to the user experience. Reducing the number of clicks, speeding the capture of data through automation, and improving the patient experience are all goals of the Aprima PRM.
Interaction with Patients
The Aprima PRM takes into account the patient experience as related to Aprima by the clinics themselves. This ranges from weekly physician calls, to user group meetings, to the Aprima release candidate (RC) program. Aprima has developed a patient portal to ensure the patient has access to their data in real time. In addition, this gives the provider an ideal means to communicate securely direct with their patients.
Ref: 8022.02 11
Data Collection
The Aprima PRM system supports a large number of ever increasing third-party devices and interfaces. There are over 8,000 different interfaces to the Aprima PRM including all of the major labs, many medical devices, and third-party clinical registries.
All of these interfaces reduce the amount of data entry the clinicians have to perform while increasing the overall quality of the data.
Aprima’s customers have often referred device manufacturers as well as other interface prospects to Aprima. Aprima has turned a number of these into direct released products.
Aprima is also compliant with the numerous Meaningful Use requirements for data collection. These include the secure messaging and medical record exchange standards.
Aprima also provides a mobile application which supports many of the functions of the desktop system. Clinicians are a very mobile customer base, and their feedback into the features required of a mobile app have been a key part of the developmental process.
Data Management
The Aprima PRM has to be able to report consistently clinical and financial information. The system provides a multitude of reports to meet these needs. Customer input was a key part of developing these reports.
The Aprima PRM also provides numerous security models to these reports and other functionality within the system. The enables the customer to manage what data is used and by whom.
As mentioned before, Aprima supports numerous government standards for reporting and data exchange. Our users provide critical feedback into how data is managed.
Consistency and Usability The visual display, colors, and the dialogs for filling templates and forms have a high priority. Aprima PRM 2015 was developed strictly according to the principals of the ISO standard.
Global Screen Layout
The global screen layout is based on the experience with previous EHR generations and corresponding with user tests for data entry efficiency and data display quality. Aprima PRM ships a default color and screen layout, but through the configurability options, custom forms and reports and dynamic procedure note technology, Aprima offers many options for the customer.
12 Ref: 8022.02
There are many pick lists in the Aprima PRM product implemented through a “find control”. These lists make is much easier for the user to choose the appropriate information and speed the data entry process. These pick lists were defined through numerous customer interactions, especially with our physicians review board.
Templates and Fonts
The templates have been designed for standard screen resolution of 1280 x 800 minimum. This combination allows a template design with easy to use buttons for touch screens and mouse clicks.
Color Schemes
Aprima PRM permits custom selection of color schemes. This permits compliance with ISO standards as well as user preferences. Many aspects of Aprima PRM permit the choice of colors ranging from the appointment schedule to the GUI experience.
Quality Management The entire processes of system analysis, development, testing, and documentation follow well known industry quality management principles including regression, performance, and automated testing.
System Analysis
Development uses several methods for system analysis. The primary mechanism today is the use of gap analysis and scenario development. During the customer feedback sessions, product quality gaps are identified and scenarios/quality assurance tests are generated to reproduce the problem. Development then addresses the issue, and the tests are rerun to verify the problem has been addressed.
This process is documented using Microsoft’s TFS system. Error test cases are automatically generated from scenarios and bugs.
Development
Engineering is transitioning from a waterfall methodology to the widely accepted agile methodologies. A number of engineering development efforts now use scenarios and sprints to build new functionality and address defects in existing product releases.
All of these scenarios are documented (functional definitions) in the Microsoft’s TFS system. Engineering status’ of percentage complete is then tracked by the development team lead (SCRUM Master), the QA manager, Release Manager and Engineering senior management.
Ref: 8022.02 13
Releases are tied to a particular set of sprints. These sprints define what changes have been implemented. This change management is reported through TFS analysis reports.
Testing
Aprima Quality Assurance (QA) team creates test plans for each sprint from development. Test suites are created along with test cases for each suite from a provided product backlog user story. Daily results are provided for hotfixes, patches, and major releases.
Work items created from the user base are controlled through QA to ensure that the user’s workflows are being tested as the user workflow. These workflows are transformed into additional test cases for manual/regression testing.
The manual team uses a test management software to track, create, and test all new product backlog items. When a manual team member deems a test case necessary for automation, the test case is marked to be included in the automation testing.
The automation team uses an automation testing software to create and run scripts for automation testing. These scripts run against a daily build of the Aprima PRM software to monitor any changes in the code.
Prior to any release, QA presents to the release management team all testing results, known issues, and outstanding work items.
Regression Testing
Regression testing runs on a daily basis. Both the manual and automation teams run test cases on a daily basis. All results are published to engineering.
Performance Testing
Performance testing occurs in automation testing on a daily basis. On-demand performance testing is accomplished for each hotfix prior to release.
Manual and Automation Testing
• The Aprima QA manual team creates test suites and test cases for each iteration/sprint from development’s backlog to record test steps and results. All bug fixes are tested by the manual team.
• The Aprima QA automation team creates automation scripts using an automation testing software. These scripts are ran against a daily build of our code to verify any changes that may occur.
• All test results for hotfixes, patches and major releases are presented daily and prior to giving to release management.
14 Ref: 8022.02
User Acceptance Testing
User acceptance testing is completed by Aprima, and the results are documented using the NISTIR-7742 template.
Documentation and Training
The Aprima system provides a large number of documents, online help screens, various training videos, webinars, and hands on on-on-one training.
Online Help
The online help includes conceptual information for the various functions, window-specific information, and workflow information for common business tasks.
Users can access the online help by selecting Help → Contents, or by pressing the F1 key. Accessing the help through the menu opens the help file to the splash window with the help table of contents displayed. Pressing F1 opens the help file to the window-specific information for the active window.
Updates and corrections to the online help are pushed to customer database monthly.
User’s Guides and Other Documents
The basic user’s guide set covers the functionality of the entire application, and the tasks that various user roles perform with the application. The user’s guide set includes:
• Administrative User’s Guide
• EHR User’s Guide
• Practice Management User’s Guide
• General User’s Guide
• Reports Guide
In addition to the comprehensive user’s guide set, Aprima also maintains installation and upgrade guides, various technical documents, and quick guides for specific features (such as Direct messaging) and purposes (such as Meaningful Use). Support alerts and weekly product updates are used to distribute timely information to users.
All user’s guides and other documents are available to clients through the Aprima Customer Portal.
Ref: 8022.02 15
Training
Training is offered in a variety of formats. Hosted web-session and quick reference video training is available through the Aprima Academy on the Aprima.com website.
• Pre-Implementation Training: Prior to implementing Aprima PRM for a new customer, the customer's users are able to attend a number of web-base training sessions. All users are encouraged to attend, and may attend any of these sessions as often as they like. On-site, in-person training is conducted immediately before or during the implementation of the application.
• Hosted Web Sessions: Free hosted web-based training sessions are offered on a routine basis, and cover specific topics or areas of the application, including Meaningful Use, CQM, and product features required to meet the various measures.
• Upgrade Training: Free hosted web-based training sessions are offered to customers who upgrade to a new version of Aprima PRM. The upgrade training is specifically designed to cover product changes and enhancements.
• Quick-Reference Videos: The quick reference videos cover many topics for different types of users. There are videos on the different versions of the Aprima PRM, Meaningful Use, and support tips.
• Private Web Session and On-Site Training: Customers may request and purchase private web sessions and on-site training to review any topics that they desire.
Customization Potential and Processes In the health care market, customization of the user experience is essential. While Aprima has adopted certain standards for the product, there are many options to customer and tailor the product to the practice, provider group, and individual user.
It is often said of the Aprima PRM that there is more than one way to accomplish any task. The product is very flexible in response to the diverse customer needs of the medical community.
General Adaptation to User Specialties
APRIMA PRM offers specialty and subspecialty templates and enhancements. This makes the product useable by the majority of practice specialties and institutions. Aprima PRM also offers many features which are specific to a particular specialty, such as home care.
The areas of customization include:
• Codes and terminologies
• Clinical pathways with specialized dynamic procedure notes
• Clinical decision support systems
• Pick lists with frequently used documentation components
16 Ref: 8022.02
• Reviews and reports
Language and Terminologies
The system can be adapted to the local clinical language and terminology. The user can chose from SNOMED compatible codes and descriptions and modify the descriptions according to the user context.
Aprima also supports multi-lingual patient management. Multi language patient education and information documents are supported.
Clinical Pathways
Clinical pathways are essential for the realization of “evidence based medicine” which support the clinical decision process. The user can define and preset exam and therapy routines following clinical guidelines. The customized exam forms support the usability of the system according to well know AMA protocols.
Workflow Management
In addition to the clinical pathways, the users can manage the organizational workflows within the Aprima PRM. Using the messaging capabilities in the system, the users in the clinic can coordinate all relevant tasks such as:
• Practice billing and collections
• Scheduling and appointment management
• Patient management
• Prescription management
• Physician order management
• Patient education
• Support of the patient portal
Test Management Aprima runs a set of rigorous test in our Quality Assurance organization. The QA team develops tests based upon engineering specifications or scenarios. Many of these tests are automated and run as regressions test for each release.
Other tests which cannot be automated are performed by QA team members for each hotfix, patch, and major release of the system.
Ref: 8022.02 17
Post-Market Tests
Aprima runs a release candidate (RC) program for every minor and major release. This program engages real physician practices in using the newest Aprima PRM product prior to general release. The feedback from these RC programs provide invaluable feedback to engineering for the correction to defects, the enhancement of the product, and the development of new workflows.
Aprima utilizes two substantial feedback systems: Salesforce.com customer management software and Microsoft’s TFS to manage and correct bugs. These systems are routinely analyzed for defect status, bug resolution statistics, and root cause analysis.
Conclusion The usability of the Aprima PRM 2015 version is based on standardized system analysis, development, test and improvement processes, some of which align with International Standards (ISO), but some of which are new and innovative development methods and practices. Aprima strives to adopt industry best practices while pursuing unique and important innovations.
The feedback loops for the Aprima PRM a numerous; beginning with the definition and design phases, through the RC program, and ultimately in the product release customer bug reporting and feedback process.
The Aprima PRM development team is always striving to improve the product functionality and quality. User Centered Design is a key component of these goals.
Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 11
Appendix B: Quality Management System
Test Results Summary for 2014 Edition EHR Certification17-2999-R-0009-PRA V1.0, March 6, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 12
Appendix C: Privacy and Security