Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter]
No alteration (customization) to the test data was necessary
3.2.4 Standards
3.2.4.1 Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted
Criterion # Standard Successfully Tested
(a)(8)(ii)(A)(2)
§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain
(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(f)(3) Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)
No optional functionality tested
Test Results Summary for 2014 Edition EHR Certification 14-2710-R-0092-PRA V1.2 May 4, 2017
*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD)
An inaccurate description of the summative usability testing measures used for Effectiveness, Efficiency, and Satisfaction was provided in the "Results" section of the report. The information provided in the table of results data did not match the results as described for measures of Effectiveness, Efficiency, and Satisfaction.
4.1 DATA ANALYSIS AND REPORTING 16 4.2 DISCUSSION OF THE FINDINGS 17
4.3 PARTICIPANT DEMOGRAPHICS 18
Page 2 of 18
Mar 2014
EXECUTIVE SUMMARY
A usability test of PowerSoftMD Certified, MU2, Ambulatory Complete EHR was
conducted between the dates of Mach 31st through November 16th in Des
Peres, MO, by Data Tec, Inc. The purpose of this test was to test and
validate the usability of the current user interface, and provide evidence
of usability in the EHR Under Test (EHRUT).
During the usability test, 3 healthcare providers, 1 medical assistant
and 1 office worker matching the target demographic criteria served as participants
and used the EHRUT in simulated, but representative tasks.
This study collected performance data on 9 tasks typically conducted
on an EHR:
• First Impressions • Review Computerized Orders • Create New Medication Orders • Create New Laboratory Orders • Create New Radiology Orders • Mark Orders Complete • Electronic Prescriptions • Build Medication List • Build Allergy List • Clinical Information Reconciliation • Clinical Decision Support Rule • Drug-Drug, Drug-Allergy Interactions
During the 120 minute one-on-one usability test, each
participant was greeted by the administrator and asked to review and
sign an informed consent/release form (included in Appendix 3); they
were instructed that they could withdraw at any time. Participants between no experience to several years prior experience with the EHR. The administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the EHRUT. During the testing, the administrator timed the test and, along with the data logger(s) recorded user performance data on paper and electronically. The administrator did not give the participant assistance in how to complete the task.
Page 3 of 18
Mar 2014
Participant screens, head shots and audio were recorded for subsequent
analysis.
The following types of data were collected for each participant:
• Number of tasks successfully completed within the allotted time without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant’s verbalizations
• Participant’s satisfaction ratings of the system
All participant data was de-identified – no correspondence could be
made from the identity of the participant to the data collected. Following
the conclusion of the testing, participants were asked to complete a post-
test questionnaire and were compensated with $50 per hour compensation] for their time. Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT.
Page 4 of 18
Mar 2014
Chart of Task Attempts, Successes and Failures: Task # of
attempts
Task Success - Mean
Path Deviation, n
Task Time – Mean, in seconds
Task Time – mean of Deviations, in Seconds
Errors, n
Task Rating, (mean) 1=Very Easy
1. First Impressions
5 1 0 36 N/A N/A N/A
2. Review Computerized Orders
5 1 0 50 0 0 1.2
3. Create New Medication Orders
5 1 3 156 48 3 1.4
4. Create New Laboratory Orders
5 1 1 96 4 1 1.4
5. Create New Radiology Orders
5 1 1 78 0 0 1.4
6. Mark Orders Complete
5 1
1 54 12 1 1.2
7. Electronic Prescriptions
5 0.8 4 184 94 5 1.6
8. Build Medication List
5 1 1 54 12 1 1.2
9. Build Allergy List
5 1 0 32 0 0 1
10. Clinical Information Reconciliation
3 3 0 200 86.67 1 1.33
11. Clinical Decision Support Rule
2 2 2 1200 1200 2 4
12. Drug-Drug Drug-Allergy Interaction
2 2 0 600 600 1 1
Page 5 of 18
Mar 2014
The results from the System Usability Scale scored the subjective
satisfaction with the system based on performance with these tasks to
be a mean of: 1.52.5
In addition to the performance data, the following qualitative observations
were made:
- Major findings
Overall, the subjects found the software easy to use once they acclimated to it. Most responded that once they knew or were shown how to do a task, they did not have to be shown again. They
found the flashing arrow and prompts very helpful. Some subjects were confused by the difference
between medications and prescriptions in e-scripts, but once they understood the meaning of the
terms, were able to find the correct information.
- Areas for improvement
Several subjects had difficulty remembering to click the “Select to Move to Current Meds” button once they had select the correct med, suggesting removal of this in a future update, if possible. In general, some subjects were uncomfortable with the slight interface change between PowerSoft and the e-scripts window, although they quickly adapted.
Subjects took a long time with the Clinical Decision Support Rule, and
appeared to get hung up on the many options available.
5 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; cores over 80 would be considered above average.
Page 6 of 18
Mar 2014
INTRODUCTION
The EHRUT tested for this study was Software engineering — Software product Quality Requirements and Evaluation(SQuaRE) — Common Industry Format (CIF) for usability test reports (ISO/IEC 25062:2006(E)) Designed to present medical information to healthcare
providers in Ambulatory settings, the EHRUT consists of
timed, task-oriented trials. The usability testing attempted to represent
realistic exercises and conditions.
The purpose of this study was to test and validate the usability of the
current user interface, and provide evidence of usability in the EHR
Under Test (EHRUT). . To this end, measures of effectiveness,
efficiency and user satisfaction, such as the time taken for each task, and the ease of completion were captured during the usability testing.
METHOD
PARTICIPANTS
A total of 5 participants were tested on the EHRUT(s). Participants in
the test was a medical assistant, a former office manager, 2 current medical office managers and a medical doctor. Participants were recruited
by Data Tec, Inc. and were compensated $50 per hour (typically 1 hour) for their time. In addition, participants had no direct connection to the development of or organization producing the EHRUT(s). Participants were not from the testing or supplier organization. Participants were not given the opportunity to have the same orientation and level of training as the actual end users
would have received. They were seeing several of these features for the
first time. For the test purposes, end-user characteristics were identified
and translated into a recruitment screener used to solicit potential
participants; an example of a screener is provided in Appendix 1.
Page 7 of 18
Mar 2014
Recruited participants had a mix of backgrounds and demographic
characteristics conforming to the recruitment screener. The following is a
table of participants by characteristics, including demographics,
professional experience, computing experience and user needs for
assistive technology. Participant names were replaced with Participant
IDs so that an individual’s data cannot be tied back to individual
identities.
5 participants (matching the demographics in the section on Participants) were recruited and 5 participated in the usability test. Z e r o participants failed to show for the study.
Participants were scheduled for 120 minute sessions with
1 minute in between each session for debrief by the administrator(s) and
data logger(s), and to reset systems to proper test conditions. A
spreadsheet was used to keep track of the participant schedule, and
included each participant’s demographic characteristics as
provided by the recruiting firm.
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the
application performed well – that is, effectively, efficiently, and with
satisfaction – and areas where the application failed to meet the needs of
Page 8 of 18
Mar 2014
the participants. The data from this test may serve as a baseline for
future tests with an updated version of the same EHR and/or comparison
with other EHRs provided the same tasks are used. In short, this testing
serves as both a means to record or benchmark current usability, but
also to identify areas where improvements must be made.
During the usability test, participants interacted with 01 EHR. Each
participant used the system in the same location, and was provided with
the same instructions. The system was evaluated for effectiveness,
efficiency and satisfaction as defined by measures collected and analyzed
for each participant:
• Number of tasks successfully completed within the allotted time
without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant’s verbalizations (comments)
• Participant’s satisfaction ratings of the system
Additional information about the various measures can be found in
Section 3.9 on Usability Metrics. TASKS
A number of tasks were constructed that would be realistic and
representative of the kinds of activities a user might do with this EHR,
including:
• First Impressions • Review Computerized Orders • Create New Medication Orders • Create New Laboratory Orders • Create New Radiology Orders • Mark Orders Complete • Electronic Prescriptions • Build Medication List
Page 9 of 18
Mar 2014
• Build Allergy List • Clinical Information Reconciliation • Clinical Decision Support Rule • Drug-Drug, Drug-Allergy Interactions
Tasks were selected based on their frequency of use, criticality of
function, and those that may be most troublesome for users.6 Tasks
should always be constructed in light of the study objectives.
PROCEDURES Upon arrival, participants were greeted; their identity was verified and matched with a name on the participant schedule. Participants were then
assigned a participant ID. 7 Each participant reviewed and signed an
informed consent and release form (See Appendix 3). A representative
from the test team witnessed the participant’s signature.
To ensure that the test ran smoothly, two staff members participated in
this test, the usability administrator and the data logger. The usability
testing staff conducting the test was experienced usability practitioners
with Peter Peter Goodall, a bachelor of political science from the University of Missouri, with 7 years experience in the medical software field.
The administrator moderated the session including administering
instructions and tasks. The administrator also monitored task times,
obtained post-task rating data, and took notes on participant comments.
A second person served as the data logger and took notes on task
success, path deviations, number and type of errors, and comments.
Participants were instructed to perform the tasks (see specific
instructions below):
• As quickly as possible making as few errors and deviations as
possible.
6 Constructing appropriate tasks is of critical importance to the validity of a usability test. These are the actual functions, but most tasks
contain larger and more fleshed out context that aligns with the sample data sets available in the tested EHR. Please consult usability references for guidance on how to construct appropriate tasks.
7 All participant data must be de-identified and kept confidential.
Page 10 of 18
Mar 2014
• Without assistance; administrators were allowed to give
immaterial guidance and clarification on tasks, but not
instructions on use.
• Without using a think aloud technique.
For each task, the participants were given a written copy of the task.
Task timing began once the administrator finished reading the question.
The task time was stopped once the participant indicated they had
successfully completed the task. Scoring is discussed below in Section
3.9.
Following the session, the administrator gave the participant the post-test
questionnaire (e.g., the System Usability Scale, see Appendix 5),
compensated them for their time, and thanked each individual for their
participation.
Participants' demographic information, task success rate, time on task,
errors, deviations, verbal responses, and post-test questionnaire were
recorded into a spreadsheet.
Participants were thanked for their time and compensated. Participants
signed a receipt and acknowledgement form (See Appendix 6) indicating
that they had received the compensation. TEST LOCATION
The test facility included a waiting area and a quiet testing room with a
table, computer for the participant, and recording computer for the
administrator. Only the participant and administrator were in the test
room. All observers and the data logger worked from a separate room
where they could see the participant’s screen and face shot, and listen to
Page 11 of 18
Mar 2014
the audio of the session. To ensure that the environment was
comfortable for users, noise levels were kept to a minimum with the
ambient temperature within a normal range. All of the safety instruction
and evacuation procedures were valid, in place, and visible to the
participants.
TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or facility.
In this instance, the testing was conducted in an office building. For
testing, the computer used an Intel-based computer running Windows 8.
The participants used a mouse and keyboard
when interacting with the EHRUT.
The test subject used a 17” monitor at 1366x768 resolution with 16 million colors. The application was set up by Data Tec, Inc. according to the vendor’s documentation describing the system set-up and preparation. The application itself was running on a Windows 8 Computer using a test database on a LAN connection. Technically, the system performance (i.e., response time) was representative to what actual users would experience in a field implementation. Additionally, participants were instructed not to change any of the default system settings (such as control of font size).
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used,
including:
1. Informed Consent
2. Moderator’s Guide
Page 12 of 18
Mar 2014
3. Post-test Questionnaire
4. Incentive Receipt and Acknowledgment Form
Examples of these documents can be found in Appendices 3-6
respectively. The Moderator’s Guide was devised so as to be able to
capture required data.
The participant’s interaction with the EHRUT was captured and recorded
digitally with screen capture software running on the test machine. A
web camera recorded each participant’s facial expressions synced
with the screen capture, and verbal comments were recorded
with a microphone. 8 The test session were electronically transmitted to a
nearby observation room where the data logger observed the test
session.
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each
participant (also see the full moderator’s guide in Appendix [B4]):
Thank you for participating in this study. Your input is very important. Our session today will last about 120 minutes. During that time you will use an instance of an electronic health record. I will ask you to complete a few tasks using this system and answer some questions. You should complete the tasks as quickly as possible making as few errors as possible. Please try to complete the tasks on your own following the instructions very closely. Please note that we are not testing you we are testing the system, therefore if you have difficulty all this means is that something needs to be improved in the system. I will be here in case you need specific help, but I am not able to instruct you or provide help in how to use the application.
Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. Please be honest with your opinions. All of the information
8 There are a variety of tools that record screens and transmit those recordings across a local area network for remote observations.
Page 13 of 18
Mar 2014
that you provide will be kept confidential and your name will not be associated with your comments at any time. Should you feel it necessary you are able to withdraw at any time during the testing.
Following the procedural instructions, participants were shown the EHR
and as their first task, were given time (10 minutes) to explore the
system and make comments. Once this task was complete, the
administrator gave the following instructions:
For each task, I will read the description to you and say “Begin.” At that point, please perform the task and say “Done” once you believe you have successfully completed the task. I would like to request that you not talk aloud or verbalize while you are doing the tasks. 9 I will ask you your impressions about the task once you are done.
Participants were then given 9 tasks to complete. Tasks are listed in
the moderator’s guide in Appendix [B4].
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the
Usability of Electronic Health Records, EHRs should support a process
that provides a high level of usability for all users. The goal is for users to
interact with the system effectively, efficiently, and with an acceptable
level of satisfaction. To this end, metrics for effectiveness, efficiency and
user satisfaction were captured during the usability testing.
The goals of the test were to assess:
1. Effectiveness of the test by measuring participant success rates and errors
2. Efficiency of the test by measuring the average task time and path deviations
9 Participants should not use a think-aloud protocol during the testing. Excessive verbalization or attempts to converse with the moderator during task performance should be strongly discouraged. Participants will naturally provide commentary, but they should do so, ideally, after the testing. Some verbal commentary may be acceptable between tasks, but again should be minimized by the moderator.
Page 14 of 18
Mar 2014
3. Satisfaction with the test by measuring ease of use ratings DATA SCORING
The following table (Table [x]) details how tasks were scored, errors
evaluated, and the time data analyzed. 10
Measures Rationale and Scoring Effectiveness:
Task Success A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis.
The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.
Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency.
Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator’s Guide must be operationally defined by taking multiple measures of optimal performance.
Effectiveness:
Task Failures If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as an “Failures.” No task times were taken for errors.
The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors.11 This should also be expressed as the mean number of failed tasks per participant.
On a qualitative level, an enumeration of errors and error types should be collected.
Efficiency:
Task Deviations The participant’s path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation.
10 An excellent resource is Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman. Also see 11 Errors have to be operationally defined by the test team prior to testing.
Page 15 of 18
Mar 2014
It is strongly recommended that task deviations be reported. Optimal paths (i.e., procedural steps) should be recorded when constructing tasks.
Efficiency:
Task Time Each task was timed from when the administrator said “Begin” until the participant said, “Done.” If he or she failed to say “Done,” the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.
Satisfaction:
Task Rating Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. 12
Common convention is that average ratings for systems judged easy to use should be 3.3 or above.
To measure participants’ confidence in and likeability of the test overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system frequently,” “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.” See full System Usability Score questionnaire in Appendix 5.13
Details of how observed data were scored.
Page 16 of 18
Mar 2014
RESULTS
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods
specified in the Usability Metrics section above. Participants who failed to
follow session and task instructions had their data excluded from the
analyses
12 See Tedesco and Tullis (2006) for a comparison of post-task ratings for usability tests. Tedesco, D. & Tullis, T. (2006) A comparison of methods for eliciting post-task subjective ratings in usability testing. Usability Professionals association Conference, June 12 – 16, Broomfield, CO. 13 The SUS survey yields a single number that represents a composite measure of the overall perceived usability of the system. SUS scores have a range of 0 to 100 and the score is a relative benchmark that is used against other iterations of the system.
The usability testing results for the EHRUT are detailed below14. The
results should be seen in light of the objectives and goals outlined in
Section 3.2 Study Design. The data should yield actionable
results that, if corrected, yield material, positive impact on user
performance.
The results from the SUS (System Usability Scale) scored the subjective
satisfaction with the system based on performance with these tasks to
be a mean of : 2.94.15
DISCUSSION OF THE FINDINGS
14 Note that this table is an example. You will need to adapt it to report the actual data collected. 15 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).
Mar 2014
EFFECTIVENESS It appeared that the subjects only pressed the buttons they needed to, and mostly ignored anything they didn’t use, although it is reasonable to think that less buttons would lead to less hesitation. EFFICIENCY
Most users found they had several options from each of the screens and they could easily jump to any screen they wanted to from any other screen. Again, if they did not need a particular button or option, most would simply ignore it. If they did need it, they were happy it was there. Overall, they found it very efficient. SATISFACTION
The subjects were very impressed with the software and reported finding it very easy-to-use. Most would recommend it to others and those that don’t use this EMR in an office setting would be happy to give it a try. Moreover, they would be happy to receive additional training and learn more about the software, if they knew nothing about it in the first place. MAJOR FINDINGS
Overall, the subjects found the software easy to use once they acclimated to it. Most responded that once they knew or were shown how to do a task, they did not have to be shown again. They found the flashing arrow and prompts very helpful. Some subjects were confused by the difference between medications and prescriptions in e-scripts, but once they understood the meaning of the terms, were able to find the correct information.
AREAS FOR IMPROVEMENT
Several subjects had difficulty remembering to click the “Select to Move to Current Meds” button once they had select the correct med, suggesting removal of this in a future update, if possible. In general, some subjects were uncomfortable with the slight interface change between PowerSoft and the e-scipts window, although they quickly adapted.
Mar 2014
PARTICIPANT DEMOGRAPHICS The report should contain a breakdown of the key participant demographics. A representative list is
shown below. Following is a high-level overview of the participants in this study.
Gender Men [2 ] Women [3 ] Total (participants) [5 ]
Years of Experience Years experience [0 ] Facility Use of EHR All paper [0 ] Some paper, some [5 ] electronic All electronic [0 ] Total (participants) [5 ]
1. EXECUTIVE SUMMARY A usability test of NewCrop Core 13.05 was conducted on 07/25/2013 by online meeting via GoToMeeting by Jennifer Harvey. The purpose of this test was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the usability test, one healthcare provider and one nurse matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks.
This study collected performance data on 18 tasks typically conducted on NewCrop Core 13.05:
CPOE (314.a.1)
Record Medication Order Change Medication Order Access Medication Order Record Laboratory Order Change Laboratory Order Access Laboratory Order Record Radiology/Imaging Order Change Radiology/Imaging Order Access Radiology/Imaging Order
2. METHOD During the 60 minute usability test, participants were greeted by the administrator and asked to review and sign an informed consent/release form (included in Appendix 2); they were instructed that they could withdraw at any time. Participants had prior experience with NewCrop Core 13.05 through their EHR application. Testing was completed in a Pre-Production environment, using a direct log in to NewCrop Core 13.05. The administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time). During the testing, the administrator recorded user performance data on paper and electronically. The administrator did not give the participant assistance in how to complete the task. �
The following types of data were collected for each participant:
• Number of tasks successfully completed
• Number and types of errors
• Path deviations
• Participant’s verbalizations
• Participant’s satisfaction ratings of the system �
Following the conclusion of the testing, the practice was asked to complete a post- test questionnaire. Following is a summary of the performance and rating data collected on the EHRUT.
2.1 PARTICIPANTS Participants for the test consisted of the following subjects
• 1 Doctor of Internal Medicine with 5 years of experience with EHR software. • 1 Licensed Practical Nurse with 5 years of experience with EHR Software.
5
2.2 TASKS
Task Ratings 1= Very Easy 2= Easy 3= Neither Easy, Nor Difficult 4= Difficult 5= Very Difficult
Physician Results:
Task Path Deviation Number of errors
Type of Error Task Rating
Record Medication Order
None 0 0 1-Very Easy
Change Medication Order
None 0 0 1-Very Easy
Access Medication Order
None 0 0 1-Very Easy
Record Laboratory Order
None 0 0 1-very easy
Change Laboratory Order
Clicked “Validate order” prior to modifying the order
0 0 1-very easy
Access Laboratory Order
None 0 0 1-very easy
Record Radiology/Imaging Order
None 0 0 1-very easy
Change Radiology/Imaging Order
None 0 0 1-very easy
Access Radiology/Imaging Order
None 0 0 1-very easy
Access Formulary Checking
None 0 0 1-very easy
Create Drug-Drug Intervention
None 0 0 1-very easy
Create Drug-Allergy Intervention
None 0 0 1-very easy
6
Adjust Severity of Drug-Drug Intervention
None 0 0 1-very easy
Adjust Severity of Drug-Allergy Intervention
None 0 0 1-very easy
Record Medication Allergy List
None 0 0 1-very easy
Change Medication Allergy List
None 0 0 1-very easy
Access Medication Allergy List
None 0 0 1-very easy
Create Electronic Prescription
None 0 0 1-very easy
Compose Clinical Message
None 0 0 1-Very Easy
View Unread Clinical Message
No deviations, but test subject stated that the layout was cluttered and appeared to be one solid link rather than individual links for Compose, Inbox, and Sent, prompting him to give a rating of 2-Easy
0 0 2-Easy
View Sent Message
No deviations, but test subject stated that the layout was cluttered and appeared to be one solid link rather than individual links for Compose, Inbox, and Sent, prompting him to give a rating of 2-Easy
0 0 2-Easy
7
Nurse Results:
Task Path Deviation Number of errors
Type of Error Task Rating
Record Medication Order
None 0 0 1-Very Easy
Change Medication Order
None 0 0 1-Very Easy
Access Medication Order
None 0 0 1-Very Easy
Record Laboratory Order
None 0 0 1-very easy
Change Laboratory Order
None 0 0 1-very easy
Access Laboratory Order
None 0 0 1-very easy
Record Radiology/Imaging Order
None 0 0 1-very easy
Change Radiology/Imaging Order
None 0 0 1-very easy
Access Radiology/Imaging Order
None 0 0 1-very easy
Access Formulary Checking
None 0 0 1-very easy
Create Drug-Drug Intervention
None 0 0 1-very easy
Create Drug-Allergy Intervention
None 0 0 1-very easy
Adjust Severity of Drug-Drug Intervention
None 0 0 1-very easy
8
Adjust Severity of Drug-Allergy Intervention
None 0 0 1-very easy
Record Medication Allergy List
None 0 0 1-very easy
Change Medication Allergy List
None 0 0 1-very easy
Access Medication Allergy List
None 0 0 1-very easy
Create Electronic Prescription
None 0 0 1-very easy
Compose Clinical Message
None 0 0 1-Very Easy
View Unread Clinical Message
No deviations, but test subject stated that the layout was cluttered and appeared to be one solid link rather than individual links for Compose, Inbox, and Sent, prompting him to give a rating of 2-Easy
0 0 2-Easy
View Sent Message
No deviations, but test subject stated that the layout was cluttered and appeared to be one solid link rather than individual links for Compose, Inbox, and Sent, prompting him to give a rating of 2-Easy
0 0 2-Easy
2.3 USABILITY METRICS 1= Very Easy 2= Easy 3= Neither Easy, Nor Difficult 4= Difficult 5= Very Difficult
3. RESULTS
3.1 DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the Task Ratings specified above.
9
3.2 DISCUSSION OF FINDINGS
SATISFACTION
The test subjects were satisfied with the workflow of commonly used features such as writing and transmitting new medications, discontinuing medications, formulary checking, drug-drug interaction checking, drug-allergy interaction checking. They were also satisfied with new features such as clinical messaging and lab ordering, though feedback was given regarding areas for improvement in clinical messaging.
MAJOR FINDINGS
Overall, the test subjects navigated the system with ease. There was a slight deviation in changing the lab order the first time where the test doctor clicked “validate” before adding a new lab and validating the order. This was the only actual error in 18 tasks tested.
AREAS FOR IMPROVEMENT
In the course of testing, the subjects indicated that they would like to see a confirmation for drug-drug interaction checking that involved actually clicking an acknowledgement of the interaction, rather than simply viewing it on the screen. The subjects also indicated that the separate links for Compose, Inbox, and Sent messages in the clinical messaging portion of the system are too close together and give the impression of being one link. This led to some confusion in the messaging portion of the test, but no errors or deviations were recorded due to this confusion.
In the imaging and laboratory portion of the test, the physician recommended that the drop down box for lab location selection be renamed lab/imaging facility because it could cause confusion if taken literally, as imaging orders aren’t sent to a “lab.” It did not cause errors or deviations with either subject.
4. APPENDICES The following appendices include supplemental data for this usability test report. Following is a list of the appendices provided:
4.1 Appendix 1: PARTICIPANT DEMOGRAPHICS Following is a high-level overview of the participants in this study.
Gender
Men [1] Women [1] Total (participants) [2]
Occupation/Role
LPN [1] Physician [1] Admin Staff [0] Total (participants) [2]
Years of Experience
10
Facility Use of EHR [5]�All paper [8] Some paper, some electronic [3]
All electronic [2] Total (participants) [2]
Detailed Participant Overview
The intended users, of NewCrop Core 13.05, include Physicians, Mid-level Prescribers, Nurses, and Clinical Staff. For this case study, one Physician and one Nurse performed all tests.
4.2 Appendix 2: INFORMED CONSENT FORM
11
12
4.3 Appendix 3: FINAL QUESTIONS What was your overall impression of this system? Overall impression is very good, nice interface and easy to understand and intuitive
What aspects of the system did you like most? eRx we have done and works great, appreciate adding interface for lab ordering
What aspects of the system did you like least? messaging, once a message is sent the message should leave the screen
Were there any features that you were surprised to see? yes, surprised to see the lab feature with a Medication company
What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? not sure as I have not seen everything but the question is if this is solely lab ordering or what I'd like to see is also interface of receiving lab as structured data
Compare this system to other systems you have used. Would you recommend this system to your colleagues? Only using MacPractice that does not really have this feature yet
4.4 Appendix 4: SYSTEM USABILITY SCALE QUESTIONNAIRE Please answer the following questions with a response of strongly agree, agree, neither agree nor disagree, disagree, or strongly disagree
1. I think that I would like to use this system frequently. strongly agree
2. I found the system unnecessarily complex. strongly disagree
3. I thought the system was easy to use. strongly agree
4. I think that I would need the support of a technical person to be able to use this system. strongly disagree
5. I found the various functions in this system were well integrated. strongly agree
6. I thought there was too much inconsistency in this system. strongly disagree
7. I would imagine that most people would learn to use this system very quickly. strongly agree
8. I found the system very cumbersome to use. strongly disagree
9. I felt very confident using the system. strongly agree
10. I needed to learn a lot of things before I could get going with this system. strongly disagree
Page 7 Supplement -
Test Subjects Demographics Chart
PartID
Gender
Age Education
ProfessionalExperience
ComputerExperience
ProductExperience
AssistiveTechnologyNeeds
1 Subject 1
female
Notcollected
Notcollected
20 yearsmedical officemanager
Notcollected
7 years Notcollected
2 Subject 2
female
Notcollected
Notcollected
7 yearsmedicalassistant
Notcollected
none Notcollected
3 Subject 3
female
Notcollected
Notcollected
6 yearsmedical officemanager
Notcollected
6 years Notcollected
4 Subject 4
male Notcollected
Notcollected
39 years as aphysician
Notcollected
5 years Notcollected
5 Subject 5
male Notcollected
Notcollected
12 years as amedical officemanager
Notcollected
Notcollected
Page 14 Supplement -
Addendum To DATA SCORING
Qualitative Evaluation of Errors:
Task 1) First Impressions - errors not applicable
Task 2) Review Computerized Orders - zero errors total
Task 3) Create New Medication Orders - 3 errors totalError 1) Subject 2 tried to click on existing orders instead of new orders as the“create new order” button is below the list of existing ordersError 2) Subject 3 did not know to press “complete meds” button once theyhad selected medsError 3) Subject 4 added to the meds list, but also transmitted the medicationto the pharmacy as if it was a prescription
Task 4) Create new lab orders - 1 error totalError 1) Subject 3 did not hit the “save & exit” button after composing the order
Task 5) Create New Radiology Orders - zero errors total
Task 6) Mark Orders Complete - 1 error totalError 1) Subject 4 marked multiple orders complete instead of just 1
Task 7) Electronic Prescriptions - 5 errors totalError 1) Subject 1 ignored the warnings for contraindicationError 2) Subject 2 made a spelling error, could not find the drug and could notfigure out how to search againError 3) Subject 3 got to the review page and thought she was finished, butshe still needed to send the prescriptionsError 4) Subject 4 decided to send the drug, even with the contraindicationwarningError 5) Subject 5 had trouble viewing the screen, had to wait for each screen
Task 8) Build medication List - 1 error totalError 1) Subject 2 added an order instead of medication list, then started overand left that order pending
Task 9) Build Allergy List - zero error total
Test Results Summary for 2014 Edition EHR Certification 14-2710-R-0092-PRA V1.2 May 4, 2017
170.314(d)(2) Auditable events and tamper-resistance
Not Applicable (did not test to this criteria)
Audit Log:
Cannot be disabled by any user.
Audit Log can be disabled.
The EHR enforces that the audit log is enabled by default when initially configured
Audit Log Status Indicator:
Cannot be disabled by any user.
Audit Log Status can be disabled
The EHR enforces a default audit log status. Identify the default setting (enabled or disabled):
There is no Audit Log Status Indicator because the Audit Log cannot be disabled.
Encryption Status Indicator (encryption of health information locally on end user device):
Cannot be disabled by any user.
Encryption Status Indicator can be disabled
The EHR enforces a default encryption status. Identify the default setting (enabled or disabled):
There is no Encryption Status Indicator because the EHR does not allow health information to be stored locally on end user devices.
Identify the submitted documentation that describes the inability of the EHR to allow users to disable the audit logs, the audit log status, and/or the encryption status: PowerSoftMD does not allow the Audit Logs to be disabled. Therefore, there is no Audit Log Status indication because it is always active. PowerSoftMD does not ever give users access to Audit Log Files or Data Bases; only the ability for authorized user to view reports. PowerSoftMD Certified is designed to not store any electronic health information on workstations or end user devices. Since, even when in use, no electronic health information is stored on the device, there is not any electronic health information on the workstations or end user device to encrypt. You can also see AuditLogsAndEncryptionAffidavit.pdf
Privacy and Security Attestation Form-EHR-36-V04
InfoGard Laboratories, Inc. Page 2
Identify the submitted documentation that describes the method(s) by which the EHR protects 1) recording of actions related to electronic health information, 2) recording of audit log status, and 3) recording of encryption status from being changed, overwritten, or deleted by the EHR technology: Actual logs are never presented for users, when requested, only copies of logs are presented. Logs are not stored in user editable format. No options to view, change, overwrite, or delete log files exist in the PowerSoftMD Certified software. Also refer to our AuditLogsAndEncryptionAffidavit.pdf.
Identify the submitted documentation that describes the method(s) by which the EHR technology detects whether the audit log has been altered: The Audit Log is not accessible by users. Refer to our AuditLogsAndEncryptionAffidavit.pdf.
170.314(d)(7) End-user device encryption
Storing electronic health information locally on end-user devices (i.e. temp files, cookies, or other types of cache approaches).
Not Applicable (did not test to this criteria)
The EHR does not allow health information to be stored locally on end-user devices.
Identify the submitted documentation that describes the functionality used to prevent health information from being stored locally: There are no functions in the PowerSoftMD software for storing patient health information on end user devices. Also refer to our AuditLogsAndEncryptionAffidavit.pdf.
The EHR does allow health information to be stored locally on end user devices.
Identify the FIPS 140-2 approved algorithm used for encryption:
Identify the submitted documentation that describes how health information is encrypted when stored locally on end-user devices:
The EHR enforces default configuration settings that either enforces the encryption of locally stored health information or prevents health information from being stored locally.
Identify the default setting: The default, and only setting, is for health information to not be stored on any end user devices Also refer to our AuditLogsAndEncryptionAffidavit.pdf.
170.314(d)(8) Integrity
Not Applicable (did not test to this criteria)
Identify the hashing algorithm used for integrity (SHA-1 or higher): SHA1
Privacy and Security Attestation Form-EHR-36-V04
InfoGard Laboratories, Inc. Page 3
170.314(e)(1) View, Download, and Transmit to 3rd Party
Not Applicable (did not test to this criteria)
Identify the FIPS 140-2 approved algorithm used for encryption:
AES256
Identify the FIPS 140-2 approved algorithm used for hashing:
SHA1
170.314(e)(3) Secure Messaging
Not Applicable (did not test to this criteria)
Identify the FIPS 140-2 approved algorithm used for encryption:
AES256
Identify the FIPS 140-2 approved algorithm used for hashing:
SHA1
Statement of Compliance
I, the undersigned, attest that the statements in this document are accurate.
Vendor Signature by an Authorized Representative
Date Dec. 5, 2014
Test Results Summary for 2014 Edition EHR Certification 14-2710-R-0092-PRA V1.2 May 4, 2017
