EHR Usability Test Report of AllMeds Specialty EHR Version 11
Report based on ISO/IEC 25062:2006 Common Industry Formatfor Usability Test Reports
AllMeds Specialty EHR Version 11
Date of Usability Test:
Date of Report: December 6th, 2017
Report Prepared By: Stephen Nichols
Table of Contents
1 EXECUTIVE SUMMARY
2 INTRODUCTION
3 METHOD 3.1 PARTICIPANTS
3.2 STUDY DESIGN
3.3 TASKS
3.4 PROCEDURE
3.5 TEST LOCATION
3.6 TEST ENVIRONMENT
3.7 TEST FORMS AND TOOLS
3.8 PARTICIPANT INSTRUCTIONS
3.9 USABILITY METRICS
4 RESULTS 4.1 DATA ANALYSIS AND REPORTING
4.2 DISCUSSION OF THE FINDINGS
5 APPENDICES 5.1 APPENDIX 1: PARTICIPANT DEMOGRAPHICS AND INFORMED CONSENT FORM
5.2 APPENDIX 2: EXAMPLE MODERATORS GUIDE
5.3 APPENDIX 3: SYSTEM USABILITY SCALE QUESTIONAIRE
1. EXECUTIVE SUMMARY
A usability test of AllMeds Specialty EHR version 11 was conducted on 10/15/2017 in Oak Ridge, TN by AllMeds
QA and Informatic team. The purpose of this test was to test and validate the usability of the current user
interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the usability test, 10
healthcare professionals matching the target demographic criteria served as participants and used the EHRUT
in simulated, but representative tasks.
This study collected performance data on 7 measures typically conducted on an EHR:
• 170.315 (a)(2) Computerized provider order entry — laboratory
• 170.315 (a)(3) Computerized provider order entry — diagnostic
• 170.315 (a)(5) Demographics
• 170.315 (a)(6) Problem list
• 170.315 (a)(9) Clinical decision support
• 170.315 (a)(14) Implantable device list
• 170.315 (b)(2) Clinical information reconciliation and incorporation
The EHRUT uses the DrFirst “Rcopia” application for all medications and allergy measures in the EHRUT. The
current Rcopia usability study for these measures can be found on the Certified Health IT Product List website
https://chpl.healthit.gov and includes the following:
• 170.315 (a)(1) COPE Medications
• 170.315 (a)(4) Drug-drug, Drug-allergy Interaction Checks
• 170.315 (a)(7) Medication List
• 170.315 (a)(6) Medication Allergy list
• 170.315 (b)(3) E-Prescribing
During the 60 minutes one-on-one usability test, each participant was greeted by the administrator and
asked to review and sign an informed consent/release form, they were instructed that they could
withdraw at any time. Participants had prior experience with the EHR. The administrator introduced
the test, and instruction participants to complete a series of tasks (given one at a time) using the EHRUT.
During the testing, the administrator timed the test and recorded user performance on paper.
The following types of data were collected for each participant:
• Number of tasks successfully completed within the allotted time without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant's verbalizations: Few suggestions
• Participant's satisfaction ratings of the system
All participant data was de-identified — no correspondence could be made from the identity of the
participant to the data collected. Following the conclusion of the testing, participants were asked to
complete a post-test questionnaire and were compensated with $0 for their time. Various
recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes
Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability
of the EHRUT.
The results of the study indicated that the EHRUT was satisfactory with regards to effectiveness and efficiency and that the participants were very satisfied with the system. In addition to the performance data, the following qualitative observations were made:
• Major Findings o The look and feel of the user interface generally provides an intuitive experience for
most users. o Least experienced users have the most difficulty in navigating the interface and
workflows. o Most of the errors or inability to complete tasks was due to a non-software technical
issue that did not push the current version of the application and databases to all servers.
• Areas for Improvement o Provide more context sensitive help and mouse over information on major interface
objects. o Continue to review application for interface consistency issues.
2. INTRODUCTION The EHRUT tested for this study was AllMeds Specialty EHR version 11. Designed to present medical
information to healthcare providers in ambulatory specialty practices. The usability testing attempted
to represent realistic exercises and conditions.
The purpose of this study was to test and validate the usability of the current user interface, and
provide evidence of usability in the EHRUT. To this end, measures of effectiveness, efficiency and user
satisfaction, such as ease of use and time taken, were captured during the usability testing.
3. METHOD
3.1 PARTICIPANTS
A total of 10 participants were tested on the EHRUT. Participants in the test were EHR Trainers.
Participants were not compensated for their time. In addition, participants had no direct connection
to the development of the EHRUT. Participants were given the opportunity to have the same
orientation and level of training as the actual end users would have received. For the test purposes,
end-user characteristics were identified and used to select test participants.
Recruited participants had a mix of backgrounds and demographic characteristics conforming to the
recruitment screener. The following is a table of participants by characteristics, including
demographics, professional experience, computing experience and user needs for assistive
technology. Participant names were replaced with Participant IDs so that an individual's data cannot
be tied back to individual identities.
All experience values are in months.
Part ID Gender Age Education Occupation
or Role Computer Experience
Professional Experience
Product Experience
Assistive Technology Needs
User01 Female 50-59 Associate Degree
Clinical Assistant
345 396 181 None
User02 Male 40-49 Master's Degree
MD 60 396 24 None
User03 Female 30-39 Bachelor's Degree
Physician Assistant
300 320 95 None
User04 Female 40-49 Associate degree
RN 180 240 72 None
User05 Female 50-59 Bachelor's Degree
RN 280 340 180 None
User06 Female 30-39 Bachelor's Degree
Physician Assistant
168 240 36 None
User07 Female 50-59 High School Graduate
Clinical Assistant
240 480 66 None
User08 Male 60-69 Master's Degree
MD 480 600 100 None
User09 Male 20-29 High School Graduate
Clinical Assistant
72 240 5 None
User10 Female 40-49 No High School Diploma
Clinical Assistant
420 96 5 None
10 participants (matching the demographics in the section on Participants) were recruited and 10
participated in the usability test. None of the participants failed to show for the study. Participants
were scheduled for 60 minutes sessions with 15 minutes in between each session for debrief by the
administrator and to reset systems to proper test conditions. A spreadsheet was used to keep track of
the participant schedule, and included each participant's demographic characteristics.
3.2 STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application performed well that is,
effectively, efficiently, and with satisfaction and areas where the application failed to meet the needs
of the participants. The data from this test may serve as a baseline for future tests with an updated
version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In
short, this testing serves as both a means to record or benchmark current usability, but also to
identify areas where improvements must be made.
During the usability test, participants interacted with EHR. Each participant used the system in the
same location, and was provided with the same instructions. The system was evaluated for
effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each
participant:
• Number of tasks successfully completed within the allotted time without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant's verbalizations (comments)
• Participant's satisfaction ratings of the system
Additional information about the various measures can be found in Section 3.9 on Usability Metrics.
3.3 TASKS
A number of tasks were constructed that would be realistic and representative of the kinds of activities
a user might do with this EHR, including:
• 170.315(a)(2) CPOE Labs
• 170.315(a)(3) CPOE Imaging
• 170.315(a)(5) Demographics
• 170.315(a)(6) Problem List
• 170.315(a)(9) Clinical Decision Support
• 170.315(a)(14) Implantable Devices
• 170.315(b)(2) Clinical Information Reconciliation
Tasks were selected based on their frequency of use, criticality of function, and those that were
required for 2015 Edition certification.
3.4 PROCEDURES
Upon arrival, participants were greeted; their identity was verified and matched with a name on
the participant schedule. Participants were then assigned a participant ID.
To ensure that the test ran smoothly, two staff members participated in this test, the usability
administrator and the data logger. The usability testing staff conducting the test was
experienced usability practitioners.
The Administrator moderated the session including administering instructions and tasks. The
administrator also monitored task times, obtained post-task rating data, and took notes on the
participant comments. A second person served as the data logger and took notes on task
success, path deviations, number and type of errors, and comments.
Participants were instructed to perform the tasks (see specific instructions below):
• As quickly as possible making as few errors and deviations as possible.
• Without assistance; administrators were allowed to give immaterial guidance and
clarification on tasks, but not instructions on use.
• Without using a think aloud technique.
For each task, the participants were given a written copy of the task. Task timing began once
the administrator finished reading the question. The task time was stopped once the
participant indicated they had successfully completed the task. Scoring is discussed below in
Section 3.9.
Following the session, the administrator gave the participant the post-test questionnaire (e.g.,
the System Usability Scale, see Appendix 4), thanked each individual for their participation.
Participants' demographic information, task success rate, time on task, errors, deviations, verbal
responses, and post-test questionnaire were recorded into a spreadsheet.
Participants were thanked for their time.
3.5 TEST LOCATION
The test facility included a quiet testing room with a table and computers for participants. Only the
participants, data logger, and administrator were in the room. To ensure the environment was
comfortable for the users, noise levels were kept to a minimum with the ambient temperature within
a normal range. All of the safety instruction and evacuation procedures were valid and visible to the
participants.
3.6 TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or facility. In this instance, the testing
was conducted in office location. For testing, the computers used were notebooks running Windows
7.0 or Windows 10. The participants used a mouse and keyboard when interacting with the EHRUT.
The EHRUT used the monitor screen with a minimum resolution of 1024 X 768, with a minimum
screen display size of 15 inches and 32 bit color settings. The application was set up by the test
laboratory according to the vendor's documentation describing the system set-up and preparation.
The application itself was running on remote desktop virtual machine using a test database on a LAN
connection. Technically, the system performance of 3-6 seconds was representative to what actual
users would experience in a field implementation. Additionally, participants were instructed not to
change any of the default system settings (such as control of font size).
3.7 TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used, including:
1. Informed Consent
2. Moderator's Guide
3. Post-test Questionnaire
The participant's interaction with the EHRUT was observed the manually noted.
3.8 PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to the each:
Thank you for participating in this study. Your input is very important. Our session today will
last about 60 minutes. During that time you will use an instance of an electronic health
record. I will ask you to complete a few tasks using this system and answer some questions.
You should complete the tasks as quickly as possible making as few errors as possible. Please
try to complete the tasks on your own following the instructions very closely. Please note
that we are not testing you we are testing the system, therefore if you have difficulty all this
means is that something needs to be improved in the system. I will be here in case you need
specific help, but I am not able to instruct you or provide help in how to use the application.
Overall, we are interested in how easy (or how difficult) this system is to use, what in it would
be useful to you, and how we could improve it. I did not have any involvement in its creation,
so please be honest with your opinions. All of the information that you provide will be kept
confidential and your name will not be associated with your comments at any time. Should
you feel it necessary you are able to withdraw at any time during the testing.
Following the procedural instructions, participants were shown the EHR and as their first task,
were given time 60 minutes to explore the system and make comments. Once this task was
complete, the administrator gave the following instructions:
For each task, I will read the description to you and say "Begin." At that point, please perform
the task and say "Done" once you believe you have successfully completed the task. I would
like to request that you not talk aloud or verbalize while you are doing the tasks. I will ask you
your impressions about the task once you are done.
Participants were then given 7 tasks to complete.
3.9 USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic
Health Records, EHRs should support a process that provides a high level of usability for all users.
The goal is for users to interact with the system effectively, efficiently, and with an acceptable
level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were
captured during the usability testing.
The goals of the test were to assess:
1. Effectiveness of AllMeds Specialty EHR version 11 by measuring participant success rates
2. Efficiency of AllMeds Specialty EHR version 11 by measuring the average task time and
path deviations
3. Satisfaction with AllMeds Specialty EHR version 11 by measuring ease of use ratings
4. RESULTS
4.1 DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods specified in the
Usability Metrics section above. Participants who failed to follow session and task instructions
had their data excluded from the analyses. there are data exclusions. The usability testing results
for the EHRUT are detailed below (see Table 4.2. The results should be seen in light of the
objectives and goals outlined in Section 3.2 Study Design. The data should yield actionable
results that, if corrected, yield material, positive impact on user performance.
Summary of Performance and Rating Data collected on EHRUT – Table 4.1
Summary of Performance and Rating Data collected on EHRUT – Table 4.1
Task/Measure N
Task Success
Path Deviation Task Time (seconds)
Errors Task
Rating 5-Easy
# Mean (SD)
Observed/Optimal Mean (SD)
Deviations Observed/Optimal
Mean (SD)
Mean (SD)
Add, Change, Access a Lab Order
a.2 100% 29/29 192 0 0 4.07
Record, Change, Access a Diagnostic Order
a.3 100% 24/24 236 0 0 3.87
Add, Change, Access a Demographic Data
a.5 100% 10/10 322 0 .2 3.90
Create, Change, and Delete Problem
a.6 70% 7/7 170 0 .9 2.50
Create, Trigger Alert, Delete Existing CDS
a.9 100% 29/29 382 0 0 3.40
Lookup, Add, Edit/Deactivate, Run Report on Implantable Devices
a.14 90% 22/22 346 0 2.5 3.20
Assign, Send, Reconcile CCD/Clinical Information
b.2 100% 800 0 0 4.1
The results from the SUS (System Usability Scale) scored the subjective satisfaction with the
system based on performance with these tasks to be: 3.57. Broadly interpreted, scores under 3
represent systems with poor usability; scores over 4 would be considered above average.
4.2 DISCUSSION OF THE FINDINGS
EFFECTIVENESS
o The users were able to perform the tasks in an effective manner. o Most of the errors or inability to complete tasks was due to a non-software technical issue that
did not push the current version of the application and databases to all servers.
EFFICIENCY
o Users were generally able to perform tasks in an efficient manner.
SATISFACTION
o Most users were satisfied with ease of use in performing the tasks.
MAJOR FINDINGS
o The look and feel of the user interface generally provides an intuitive experience for most users.
o Least experienced users have the most difficulty in navigating the interface and workflows.
AREAS FOR IMPROVEMENT
o Provide more context sensitive help and mouse over information on major interface objects. o Continue to review application for interface consistency issues.
Name of Senior
Company
Representative:
Lucy Stephenson, MSN, RN Title of
Senior Company
Representative:
VP Customer Relations/Informatics
Signature of
Senior Company
Representative:
Date Signed:
12/11/2017
5. APPENDICES
The following appendices include supplemental data for this usability test report. Following is a list of
the appendices provided:
1. Participant demographics and Informed Consent Form
2. Example Moderator’s Guide
3. System Usability Scale Questionnaire
5.1 APPENDIX 1: PARTICIPANT DEMOGRAPHICS AND INFORMED CONSENT FORM
Following is a high-level overview of the participants in this study.
Gender
Men 3
Women 7
Total (participants) 10
Occupation/Role
RN/BSN 2
Physician 2
Physician Assistant 2
Clinical Assistant 4
Years of Experience
Mean Years experience 6.4
Total (participants) 10
Sample Form AllMeds EHR Usability Study
User Consent and Demographics Form
AllMeds would like to thank you for your participation in this study. The purpose of the study is to evaluate the usability of the AllMeds Specialty EHR. Your participation in this study will include performing specific tasks within the EHR; and completing a short survey following the study. The study should take approximately 60 minutes. The information collected by AllMeds during the study is for research purposes only.
NOTE: Participant Names will NOT be used in any reports. All information will be referenced to the Tester ID ONLY.
By signing below, I agree to participate in the study.
Name Tester ID Participant Name and Tester ID
Assigned EHR Role
Professional Experience (yrs/months)
Decline
AllMeds EHR Experience (yrs/months)
Decline
Computer Experience (yss/months)
Decline
Highest Level of Education
GED HS AA BA BS MA DR Decline
Sex Male Female Decline
Age Range
20-29 30-39 40-49 50-59 60-Older Decline
Date of Study
November, 2017
Location
AllMeds Inc., Oak Ridge, TN
Signature: _________________________________ Printed Name: ______________________________ Date: _____________
5.2 APPENDIX 2: EXAMPLE MODERATOR GUIDE
Moderator's Guide
Administrator: ______________________________
Data Logger: ______________________________
DateTime: _________________________
Participant #: _________________________
Location: _________________________
Prior to testing
• Confirm schedule with Participants
• Ensure EHRUT lab environment is running properly
• Ensure lab and data recording equipment is running properly
Prior to each participant:
• Reset application
Prior to each task:
• Reset application to starting point for next task
After each participant:
• Gather all forms and notes
After all testing
• Return all equipment to IT.
• Verify all forms and notes have been collected
• Return testing room to pre-test condition
5.3 APPENDIX 3: SYSTEM USABILITY SCALE QUESTIONAIRE
AllMeds EHR Usability Survey
1. The process under test was easy to complete.
a. Strongly Disagree
b. Disagree
c. Neither Agree or Disagree
d. Agree
e. Strongly Agree
2. The functionality of the process under test is essential to my workflow.
a. Strongly Disagree
b. Disagree
c. Neither Agree or Disagree
d. Agree
e. Strongly Agree
3. Would you find this functionality useful on a mobile device?
a. Yes
b. No
4. How would you rate your overall satisfaction with your AllMeds EHR experience?
a. Very Dissatisfied
b. Somewhat Dissatisfied
c. Somewhat Satisfied
d. Satisfied
e. Very Satisfied
5. Are there any changes or improvements you would like to see?
6. Additional Comments: (continue on back if necessary)
EHRUsabilityTestReportofRcopia ProductVersion:V3ReportbasedonISO/IEC25062:2006CommonIndustryFormatforUsabilityTestReports
DateofUsabilityTest:June12-14,2017DateofReport:June20,2017
ReportPreparedBy:
TheUsabilityPeople,LLC
4000LegatoRoad,Suite1100Fairfax,VA22033
www.TheUsabilityPeople.com
TABLEOFCONTENTSExecutiveSummary.................................................................................................................................................3Introduction................................................................................................................................................................4Method..........................................................................................................................................................................5Participants............................................................................................................................................................5StudyDesign..........................................................................................................................................................7Tasks.........................................................................................................................................................................8TestLocation.........................................................................................................................................................9TestEnvironment.............................................................................................................................................10TestFormsandTools.....................................................................................................................................11ParticipantInstructions.................................................................................................................................12Procedure.............................................................................................................................................................13UsabilityMetrics...............................................................................................................................................14DataScoring...................................................................................................................................................15
Results........................................................................................................................................................................17DataAnalysisandReporting.......................................................................................................................17EffectivenessandEfficiency....................................................................................................................17Satisfaction.....................................................................................................................................................18
Detailsforeachcriteriaevaluated............................................................................................................21DiscussionofFindings.........................................................................................................................................26
Effectiveness..................................................................................................................................................26Efficiency.........................................................................................................................................................26Satisfaction.....................................................................................................................................................27SummaryofMajorFindings....................................................................................................................28AreasforImprovement.............................................................................................................................28
Appendices...............................................................................................................................................................29AppendixA:RecruitingScreener...............................................................................................................29AppendixB:InformedConsentForm......................................................................................................31AppendixC:ParticipantTasks....................................................................................................................32AppendixD:SystemUsabilityScaleQuestionnaire...........................................................................34AppendixE:ComputerSystemUsabilityQuestionnaire.................................................................35
ExecutiveSummary
On June 12th through June 14th 2017, The Usability People, LLC conducted a
summative usability test of the DrFirst.com Rcopia V3 system. The test was conducted in the
Fairfax, VA office of The Usability People over remote tele-conferencing sessions using Go To
Meeting.Thepurposewastotestandvalidatetheusabilityofthecurrentuserinterfaceand
provideevidenceofusabilityofRcopiaastheEHRUnderTest(EHRUT).Ten(10)
healthcareprovidersmatchingthetargetdemographiccriteriaparticipatedintheusability
testusingtheEHRUTinsimulated,butrepresentativetasks.
Thestudyfocusedonmeasuringtheeffectivenessof,efficiencyof,andsatisfaction
withRcopiaamongasampleofparticipantsrepresentingpotentialusersofthesystem.
Performancedatawascollectedonnine(9)taskstypicallyconductedonanEHR.Tasks
createdwerebaseduponthecriteriaspecifiedwithinthetestprocedurestructurefor
evaluatingconformanceofElectronicHealthRecord(EHR)technologytothecertification
criteriadefinedincertificationcriteriaidentifiedin45CFRPart170SubpartCofthe
HealthInformationTechnology:2015EditionHealthInformationTechnology(HealthIT)
CertificationCriteria.
ResultsofthestudyindicatedthattheRcopiasystemwasquitesatisfactorywith
regardstoeffectivenessandefficiencyandthattheparticipantswereverysatisfiedwith
thesystem.
Introduction
TheElectronicHealthRecordSystemUnderTest(EHRUT)testedforthisstudy,
Rcopia,wasspecificallydesignedtopresentmedicalinformationtotheintendedusers
(includingphysicians;nursesandotherhealthcarepractitioners)ondesktopcomputersin
standardmedicalcaresettings.Thisstudytestedandvalidatedtheusabilityofthecurrent
userinterfaceandprovidesevidenceoftheusabilityofRcopiawithrepresentative
exercisesandinrealisticuserconditions.Tothisend,measuresofeffectivenessand
efficiency,suchastimeontask,numberoferrorsmade,andcompletionrateswere
capturedduringusabilitytesting.Satisfactionwasassessedandusercommentscollected
usingtwoindustry-standardquestionnaires.
Method
Participants
Ten(10)individuals(5menand5women)participatedintheEHRUT(RcopiaV3).
ParticipantswererecruitedfromcontactsobtainedfromDrFirst.comretailcustomersand
fromadatabaseofparticipantsmaintainedbyTheUsabilityPeople,LLC.Thosewho
respondedtotheinvitationtotakepartinthestudyweredirectedtoanonline
questionnairethatservedastheparticipantscreener.(Thescreeningquestionnaireis
providedasAppendixA.)Participantsmeetingthecriteriaforparticipationinthestudy
werecontactedandscheduledviaemail,telephoneandusinganonlineschedulingsystem.
ParticipantsintheusabilitytestofRcopiahadavarietyofhealthcarebackgrounds
anddemographiccharacteristics.Amajorityoftheparticipantsdidnothaveanydirect
experienceortrainingusingtheversionoftheRcopiaSystem.
Table1presentsparticipantcharacteristics,includingdemographics,professional
experience,computingexperience,andnumberofpreviousEHR'sused.Noneofthe
participantswerefromthevendororganization(DrFirst.com)thatproducedandsupplied
theevaluatedsystemnordidanyparticipanthaveanydirectconnectiontothetesting
organization(TheUsabilityPeople,LLC).Ascompensationfortheirparticipationall
individualsreceivedagiftcard.
Table1.ParticipantCharacteristics
PartID Gender Age Education Role/Title
ProfessionalExperience
(yrs)
EHRExperience
(yrs)
ExperiencewithRcopia(yrs)
AssistiveTechNeeds
p1
Male
40to49
Tradetechnicalvocationaltraining
Administrative
Director
17
10
0
None
p2
Male
50to59 Doctoratedegree(e.g.,MD,DNP,DMD,PhD)
MD 29 10 10
None
p3
Male40to49 Master's
degreeEHRInstructor 4 11 0
None
p4Male 30to39 Master's
degreeNurse 1 2 0
None
p5
Female
20to20 Tradetechnicalvocationaltraining
MedicalAssistant 5 0 3
None
p6
Male
40to49 Doctoratedegree(e.g.,MD,DNP,DMD,PhD)
MD 25 7 0
None
p7Female 30to30 Bachelor's
degreeNurse 9 7 0
None
p8
Female
20to29 Tradetechnicalvocationaltraining
CertifiedMedicalAssistant
5 4 0
None
p9Female 30to30 Bachelor's
degreeRN 11 11 0
None
p10
Female
20to29 Tradetechnicalvocationaltraining
LPN 3 3 3
None
SummaryofParticipantCharacteristics:GenderMale 5Female 5Total 10Occupation/RolePhysician 2Nurse 4MedicalAssistant 2EHRAdmin 2
EducationDoctor 2Masters 2Bachelors 3Trade/Vocational 3YearsofExperiencewithRcopiaNone 83years 15ormoreyears 1
StudyDesign
TheoverallobjectiveofthisusabilitytestwastouncoverareaswheretheRcopia
systemperformedwell–thatis,effectively,efficiently,andwithsatisfaction–andareas
wherethesystemfailedtoservetheneedsofusers.Datafromthistestmaybeusedasa
baselineforfuturetestsofupdatedversionsofRcopiaand/orforcomparingRcopiawith
otherEHRspresentingthesametasks.Inshort,thistestingservesasbothameansto
recordorbenchmarkcurrentusabilityandtoidentifyareaswhereimprovementsmustbe
made.
ParticipantshadarangeofexperiencewithEHRsingeneral,andmanyhadlittle
directexperienceand/ortrainingwiththeRcopiasystem.Participantscompletedthetest
ofRcopiausabilityduringindividual45-minuteGoToMeetingsessions.Duringthetest,
eachparticipantinteractedwithvariouscomponentsoftheRcopiasystem.Each
participantwasprovidedwiththesameinstructions.
Rcopiawasevaluatedforeffectiveness,efficiencyandsatisfactionasdefinedbythe
followingmeasurescollectedandanalyzedforeachparticipant:
• Numberoftaskssuccessfullycompletedwithoutassistance
• Timetocompletethetasks
• Numberanddescriptionoferrors
• Pathdeviations
• Participant’sverbalizations(comments)
• Participant’ssatisfactionratingsofthesystem
Tasks
TheUsabilityPeople,constructedatotalofnine(9)tasksinclosecollaborationwith
theDrFirst.comteam,toberealisticandrepresentativeoftheactivitiesausermight
engagewiththeRcopiasysteminactualmedicalsettings.Thenine(9)taskswerecreated
baseduponthecriteriaspecifiedwithinthetestprocedurestructureforevaluating
conformanceofElectronicHealthRecord(EHR)technologytothecertificationcriteriaas
definedin45CFRPart170SubpartCoftheHealthInformationTechnology:Standards,
ImplementationSpecifications,andCertificationCriteriaforElectronicHealthRecord
Technology.
Thetasksfocusedonthefollowingissues:
• Section170.315(a)(1)Computerizedproviderorderentry–medications
• Section170.315(a)(4)Drug-drug,drug-allergyinteractionchecks
• Section170.315(a)(6)Problemlist
• Section170.315(a)(7)Medicationlist
• Section170.315(a)(8)Medicationallergylist
• Section170.315(b)(3)Electronicprescribing
AcopyofthetaskspresentedtoparticipantsintheusabilitytestofRcopiacanbefoundin
AppendixC.
TestLocation
AllparticipantsweretestedontheRcopiasystemduringremoteconferencing
sessionsusingGoToMeeting.Eachparticipantwasrequestedinadvancetosecureaquiet
roomwithminimaldistractionsandadesktoporlaptopcomputerthatcouldconnecttothe
InternetwithaGoToMeetingsession.Althoughthetypeofcomputer,operatingsystem
anddisplayresolutionoftheremoteparticipantsystemwasunknown,thesystemthatwas
usedbythetestadministratorandcontrolledbytheremoteparticipantwasanHPAll-in-
oneDesktoprunningtheWindows10operatingsystemataresolutionof1600x900.
DuringagivenGoToMeetingsession,onlythetestadministratorandparticipant
communicatedwithoneanother.
TheGoToMeetingusabilitytestsessionwasconductedbyatestadministratorfrom
thetestingorganization(TheUsabilityPeople,LLC)workingfromasmallconferenceroom
atTheUsabilityPeople’sFairfax,VAlocation.Seatedneartheadministrator,adatalogger
fromthetestingorganizationtookdetailednotesoneachsession,includingusercomments
andsatisfactionratingsfollowingeachtask.Duringasessionboththetestadministrator
andthedataloggercouldseeonlytheparticipant’sscreenandheartheparticipant’s
comments,questions,andresponses.
TestEnvironment
WhiletheEHRUTtypicallywouldbeusedinahealthcareoffice,inpatientor
ambulatorycenterfacility,testingoftheRcopiasystemwasconductedviaremote
connectionduringindividualGo To Meetingsessions.EachparticipantcalledintoaGoTo
Meetingsessionandwasconnectedbythetestadministratortotheapplication.
TheRcopiaapplicationitselfranonaweb-basedbrowserplatformonaLAN
connectionusingasampledatabasethatwassetupspecificallyforthetest.Participants
usedamouseandkeyboardwheninteractingwiththeEHRUTandweregivenremote
controloftheadministrator’sworkstationtoperformthetasks.
TestFormsandTools
Aspartoftheusabilitytest,severaldocumentsandinstrumentswereused.Examples
ofthedocumentsusedduringtheusabilitytest,includinganinformedconsentform,the
tasks,andpost-testquestionnaires,canbefoundinAppendicesBtoE,respectively.
Participants’interactionwithRcopiawascapturedandrecordeddigitallyusingthe
Moraescreencapturesoftwarerunningonthetestadministrator’sworkstation.Verbal
responseswererecordedthrougheitherthemicrophoneintegratedintotheparticipant’s
computerorthroughatelephoneconnection.Thisinformationwaselectronically
transmittedtotheadministratorandtothedataloggerduringeachtestsession.
ParticipantInstructions
Theadministratorreadthefollowinginstructionsaloudtoeachparticipant:
Thankyouforparticipatinginthisstudy.Yourinputisveryimportant.Oursessiontodaywilllastabout45minutes.Duringthattimeyouwilluseaninstanceofanelectronichealthrecord.Iwillaskyoutocompleteafewtasksusingthissystemandanswersomequestions.
Pleasenotethatwearenottestingyou;wearetestingthesystem.Thereforeifyouhaveanydifficultythismaymeanthatsomethingneedstobeimprovedinthesystem.Iwillbehereincaseyouneedspecifichelp,butIamnotabletoinstructyouorprovidehelpinhowtousetheapplication.
Overall,weareinterestedinhoweasy(orhowdifficult)thissystemistouse,whatinitwouldbeusefultoyou,andhowwecouldimproveit.Ididnothaveanyinvolvementinitscreation,sopleasebehonestwithyouropinions.Alloftheinformationthatyouprovidewillbekeptconfidentialandyournamewillnotbeassociatedwithyourcommentsatanytime.Shouldyoufeelitnecessaryyouareabletowithdrawatanytimeduringthetesting.
Participantswerethengivennine(9)taskstocomplete.
Procedure
Uponconnectiontotheonlinemeetingtool(GoToMeeting),eachparticipantwas
greeted,hisorheridentityverified,andmatchedtoanameontheparticipantschedule.
ParticipantnameswerereplacedwithparticipantIDssothatagivenindividual’sdata
cannotbelinkedtohis/heridentity.Priortobeginningtesting,eachparticipantreviewed
andsignedaninformedconsentform(SeeAppendixB)andemailedittotheorganization
(The Usability People, LLC) conductingthetest.
TwostaffmembersoftheUsabilityPeople,ausabilitytestadministratorandadata
logger,administeredthetest.Theadministratormoderatedthesessionbyprovidingboth
verbalandwritteninstructionsfortheoverallusabilitytestandforeachofthetasks
comprisingthetest.Theadministratoralsomonitoredtasksuccess,pathdeviations,
numberanddescriptionoferrors,andaudio-recordedparticipantverbalcomments.The
dataloggerloggedtasktimes,obtainedpost-taskratingdata,andtooknotesonparticipant
commentsandadministratorfeedback.
Foreachofthenine(9)tasks,participantswerepresentedwritteninstructionsto
theircomputers.Followingtheadministrator’sinstructions,eachparticipantperformed
eachtaskbyfirstreadingthetaskoutloudthenstatinginhisorherownwordshisorher
interpretationofthetaskrequirements.Whentheparticipant’sinterpretationmatchedthe
actualgoalofthetask,theadministratorinstructedtheparticipanttobeginandtasktiming
began.Tasktimewasstoppedandrecordedwhenthetestadministratorobservedontheir
workstationthattheparticipanthadsuccessfullycompletedthetask.Ifaparticipantfailed
tocompleteataskbeforetheexpectedamountoftimeforeachtask,thattaskwasmarked
as“TimedOut.”Aftereachtask,thetestadministratoraskedtheparticipant,“Onascale
from1to5,where1is‘VeryDifficultand5is‘VeryEasy,’howsatisfiedwereyouwiththe
easeofuseforthistask?”Thissameprocedurewasconductedforeachofthenine(9)
tasks.
Followingcompletionofthenine(9)EHRtasks,theadministratorelectronically
presentedtotheparticipanttwopost-testquestionnaires(SystemUsabilityScale(SUS),see
AppendixDandComputerSystemUsabilityQuestionnaire(CSUQ),seeAppendixE).After
theparticipantcompletedbothquestionnaires,theadministratorthankedeachparticipant
forhisorhertimeandallowedtheparticipanttomakeanycommentsonoraskany
questionsaboutthesystemand/orthetaskspresented.Foreachsession,theparticipant’s
schedule,demographicinformation,tasksuccessrate,timeontask,errors,deviations,
verbalresponses,andpost-testquestionnaireweredigitallyrecorded.Thesystemwas
thenresettopropertestconditionsforthenextparticipant.
UsabilityMetrics
AccordingtotheNISTGuidetotheProcessesApproachforImprovingtheUsabilityof
ElectronicHealthRecords(NISTIR7741,November,2010)EHRsshouldsupportaprocess
thatprovidesahighlevelofusabilityforallusers.Thegoalisforuserstointeractwiththe
systemeffectively,efficiently,andwithanacceptablelevelofsatisfaction.Tothisend,
metricsforeffectiveness,efficiencyandusersatisfactionwerecapturedduringthe
usabilitytesting.Thegoalsofthetestweretoassess:
• EffectivenessofRcopiabymeasuringparticipantsuccessratesanderrors.
• EfficiencyofRcopiabymeasuringtheaveragetasktimeandpathdeviations.
• SatisfactionwithRcopiabymeasuringease-of-useratings.
DataScoringTable2detailshowtaskswerescored,errorsevaluated,andthetimedataanalyzed:Table2.ScoringProtocolsforEffectiveness,Efficiency,andSatisfaction
Measures RationaleandScoring
Effectiveness:• TaskSuccess
Ataskwascountedas“Success”iftheparticipantwasabletoachievethecorrectoutcome,withoutassistance,withinthetimeallottedonapertaskbasis.The totalnumberofSuccesseswascalculated foreach taskand thendividedbythetotalnumberoftimesthattaskwasattempted.Resultsareprovidedasapercentage.
Effectiveness:• TaskFailures
Iftheparticipantabandonedthetask,didnotreachthecorrectansweror performed it incorrectly, or reached the end of the allotted timebeforesuccessfulcompletion, thetaskwascountedas “Fail.” Notasktimesweretakenforfailedattempts.Thetotalnumberoferrorswascalculatedforeachtaskanddividedbythe total number of times that task was attempted. Results arepresentedastheaverageerrorrate.Note:Notalldeviationsarecountedaserrors
Effectiveness:• Prompted
Successes
Becausesometasksaredependentuponthesuccessfulcompletionofprevious tasks, participants may receive a limited number of“prompts” to helpprepare the systemdata for thepre-requisites forsubsequenttasks.Whenaparticipantwasabletocompletethedataentryonataskwith3orfewerprompts,thetaskwascountedasan“Assisted”competition.NotasktimeswererecordedforAssistedcompletions.
Efficiency:• Task
Deviations
The participant’s path (i.e., steps) through the application wasrecorded.Deviationsoccurifforexample,theparticipantnavigatedtoan incorrect screen, clicked on an incorrectmenu item, followed anincorrectlink,orinteractedincorrectlywithanon-screencontrol.
Efficiency:• TaskTime
Each task was timed from the administrator’s prompt “Begin” untilsaid, “Done.” If the participant failed to say, “Done,” timing stoppedwhentheparticipantstoppedperformingthetask.Only task times for tasks that were successfully completed wereincludedintheaveragetasktimeanalysis.Averagetimepertaskwascalculatedforeachtask.
Satisfaction:• EaseofUse
ratings• System
Satisfaction
Participant’ssubjectiveimpressionoftheeaseofuseoftheapplicationwas measured by administering both a single post-task question aswellastwopost-sessionquestionnaires.Aftereachtask,theparticipantdeterminedonascaleof1to5theirsubjective satisfactionwith performance on the task. These data areaveragedacrossparticipants.To measure participants’ confidence in and likeability of the EHRoverall, the testing team administered electronic versions of theSystem Usability Scale (SUS) and the Computer System UsabilityQuestionnaire(CSUQ).SeetheSUSquestionnaireasAppendixD.,andtheCSUQasAppendixE.
Results
DataAnalysisandReporting
TheresultsoftheusabilitytestoftheRcopiasystemwereanalyzedaccordingtothe
methodsdescribedintheUsabilityMetricssectionaboveandaredetailedbelow.Notethat
theresultsshouldbeevaluatedrelativetothestudyobjectivesandgoals,asoutlinedinthe
studydesignsectionabove.Thedatashouldyieldactionableresultsthat,ifcorrected,yield
material,positiveimpactonuserperformance.
EffectivenessandEfficiency
Table3presentsasummaryofoveralltaskperformanceshowingtask,averagetime
ontask,taskcompletionrates,pathdeviationsandtasksatisfaction:
Table3.UsabilityTestResults
TaskMeanTaskTime
CompletionRate(%)
Mean#Path
Deviations
MeanTaskSatisfaction
Task1MedicationAllergyList-Add
1:09 SD=0:21 100% 0.20 SD=0.44 4.90 SD=0.32
Task2Drug-AllergyInteractionCheck
1:29 SD=0:57 100% 0.30 SD=0.46 4.60 SD=0.66
Task3ElectronicPrescribing 2:47 SD=0:53 100% 0.50 SD=1.02 4.60 SD=0:66Task4ElectronicPrescribing--Cancel/Change
2:25 SD=0:41 100% 0.20 SD=0.40 4.70 SD=0.46
Task5ElectricPrescribing--Renew
0:33 SD=0:16 100% 0.00 SD=0.0 5.00 SD=0.0
Task6ProblemListCDSIntervention
1:19 SD=0:32 90% 0.70 SD=1.8 4.50 SD=0.67
Task7StopaMedication 0:47 SD=0:25 100% 0.00 SD=0.60 4.80 SD=0.40Task8MedicationOrder--Drug-DrugInteraction
3:31 SD=0:54 90% 0.60 SD=0.80 4.80 SD=0:54
Task9MedicationAllergyList–Remove
0:38 SD=0:17 100% 0.00 SD=0.0 5.00 SD=0.0
AsTable3shows,relativetooptimalperformancestandardsasdefinedby
DrFirst.comandTheUsabilityPeople,participantperformanceintheRcopiaEHRusability
testwasquitesatisfactory.Theoverallaveragetaskcompetitionratewasninety-eight(98)
percent.
Satisfaction
IndividualTaskSatisfactionParticipantsverballyindicatedtheirsatisfactionwiththeeasyofuseforeachtaskusinga
scaleof“1”(“VeryDifficult”)to“5,”(“VeryEasy”).AsFigure1showsindividualtask
satisfactionrangedfromalowof4.5outof5onTask6(ProblemListIntervention)toahigh
of5outof5onTask5(ElectronicPrescribing-Renew)andTask9(RemoveanAlergy)
Figure1.SatisfactionRatingsofIndividualTasks
0.00 1.00 2.00 3.00 4.00 5.00
Task1MedicationAllergyList-Add
Task2Drug-AllergyInteractionCheck
Task3ElectronicPrescribing
Task4ElectronicPrescribing--Cancel/Change
Task5ElectricPrescribing--Renew
Task6ProblemListCDSIntervention
Task7StopaMedication
Task8MedicationOrder--Drug-Drug
Task9MedicationAllergyList-Remove
MeanSatisfaction
SystemUsabilityScale
TheSystemUsabilityScale(SUS)isasimple,10-itemLikert-typeattitudescale
providingaglobalsubjectiveassessmentofusabilityfromtheuser’sperspective(John
BrookeatDigitalEquipmentCompanydevelopedtheSUSin1986).TheSUSscaleisscored
from0to100;scoresunder60representsystemswithlessthanoptimalusability,scores
over80areconsideredbetterthanaverage.SeeAppendixDforacopyoftheSUS.
ThemeantotalSUSscorefortheRcopiaEHRwaseighty-four(84)andrangedfrom
alowofsixty(60)andahighofonehundred(100).Overall,participant-usersratedtheir
satisfactionwiththeRcopiaEHRsystemtobewithintheveryhigh-rangeofausableand
satisfyingEHR.
ComputerSystemUsabilityQuestionnaire
UsingtheComputerSystemUsabilityQuestionnaire(CSUQ;Lewis,J.R.(1995).(See:
IBMComputerUsabilitySatisfactionQuestionnaires:PsychometricEvaluationand
InstructionsforUse.InternationalJournalofHuman-ComputerInteraction,7:1,57-78).),
participantsratedeachof19itemsonascalefrom1to7,witharatingof7beingmostin
agreementwiththepositively-wordeditem.Responsesforeachitemweresummedand
averagedtofourscales–InterfaceQuality,InformationQuality,SystemUsefulness-andan
overallscale.SeeAppendixEforacopyoftheCSUQ.
Figure2displaysCUSQratingsforeachofthefourscales.Ingeneral,participantsin
theRcopiastudyratedsystemusabilitytobeveryhigh.OnInterfaceQualitytheaverage
scorefortheparticipantswas5.77;onInformationQualitytheaveragescore6.0;on
SystemUsefulnesstheaveragescorewas6.20;andtheoverallaverageCUSQscorewas
6.05.
2.50 3.50 4.50 5.50 6.50
Overall Score
System Usefulness
Information Quality
Interface Quality
User Rating (Out of 7)
Figure 2: Computer System Usability Questionnaire
DetailsforeachcriteriaevaluatedSection170.315(a)(7)MedicationlistSection170.315(a)(1)Computerizedproviderorderentry–medicationsParticipantNumber
Task7StopaMedication
TaskTime Outcome #PathDeviationsTask
Satisfaction
p1 0:29 Success 0 5p2 0:41 Success 0 5p3 0:57 Success 0 5p4 1:02 Success 0 4p5 0:48 Success 0 5p6 0:43 Success 0 5p7 1:50 Success 2 4p8 0:51 Success 0 5p9 0:14 Success 0 5p10 0:22 Success 0 5
ExpectedTimeonTask 0:30
AverageTimeonTask 0:47 SD=0:25
AverageTaskSatisfaction 4.80 SD=0.4
Average#PathDeviations 0.00 SD=0.6
PercentSuccess 100%
ParticipantNumber
Task8MedicationOrder--Drug-DrugInteraction
TaskTime Outcome #PathDeviationsTask
Satisfaction
p1 6:11 Timeout 0 5p2 3:06 Success 0 5p3 4:04 Success 2 5p4 3:11 Success 0 5p5 3:23 Success 1 5p6 5:03 Success 0 4p7 4:52 Success 1 4p8 3:06 Success 0 5p9 2:49 Success 2 5p10 2:08 Success 0 5
ExpectedTimeonTask 3:00
AverageTimeonTask 3:31 SD=0:54
AverageTaskSatisfaction 4.80 SD=0.4
Average#PathDeviations 0.60 SD=0.8
PercentSuccess 90%
Section170.315(a)(4)Drug-drug,drug-allergyinteractionchecksParticipantNumber
Task2Drug-AllergyInteractionCheck
TaskTime Outcome #PathDeviationsTask
Satisfaction
p1 2:53 Success 0 5p2 1:06 Success 0 5p3 1:57 Success 1 5p4 0:59 Success 0 4p5 1:23 Success 1 5p6 3:31 Success 0 4p7 0:43 Success 0 5p8 1:16 Success 0 3p9 0:31 Success 1 5p10 0:34 Success 0 5
ExpectedTimeonTask 1:00
AverageTimeonTask 1:29 SD=0:57
AverageTaskSatisfaction 4.60 SD=0.66
Average#PathDeviations 0.30 SD=0.46
PercentSuccess 100%
ParticipantNumber
Task8MedicationOrder--Drug-DrugInteraction
TaskTime Outcome #PathDeviationsTask
Satisfaction
p1 6:11 Timeout 0 5p2 3:06 Success 0 5p3 4:04 Success 2 5p4 3:11 Success 0 5p5 3:23 Success 1 5p6 5:03 Success 0 4p7 4:52 Success 1 4p8 3:06 Success 0 5p9 2:49 Success 2 5p10 2:08 Success 0 5
ExpectedTimeonTask 3:00
AverageTimeonTask 3:31 SD=0:54
AverageTaskSatisfaction 4.80 SD=0.4
Average#PathDeviations 0.60 SD=0.8
PercentSuccess 90%
Section170.315(a)(6)ProblemlistParticipantNumber
Task6ProblemListCDSIntervention
TaskTime Outcome #PathDeviationsTask
Satisfaction
p1 1:02 Success 0 5p2 1:08 Success 0 4p3 2:23 Success 0 5p4 1:25 Success 1 4p5 2:01 Success 0 4p6 1:21 Success 0 5p7 0:40 Success 0 5p8 3:47 Fail 6 3p9 1:17 Success 0 5p10 0:35 Success 0 5
ExpectedTimeonTask 2:00
AverageTimeonTask 1:19 SD=0:32
AverageTaskSatisfaction 4.50 SD=0.67
Average#PathDeviations 0.70 SD=1.8
PercentSuccess 90%
Section170.315(a)(8)MedicationallergylistParticipantNumber
Task1MedicationAllergyList–Add
TaskTime Outcome #PathDeviationsTask
Satisfaction
p1 0:46 Success 0 5p2 1:06 Success 0 5p3 1:59 Success 1 5p4 1:16 Success 1 4p5 1:23 Success 0 5p6 1:13 Success 0 5p7 0:53 Success 0 5p8 1:09 Success 0 5p9 0:39 Success 0 5p10 1:06 Success 0 5
ExpectedTimeonTask 1:30
AverageTimeonTask 1:09 SD=0:21
AverageTaskSatisfaction 4.90 SD=0.32
Average#PathDeviations 0.20 SD=0.44
PercentSuccess 100%
ParticipantNumber
Task9MedicationAllergyList-Remove
TaskTime Outcome #PathDeviationsTask
Satisfaction
p1 1:10 Success 0 5p2 0:27 Success 0 5p3 0:25 Success 0 5p4 0:41 Success 0 5p5 0:38 Success 0 5p6 1:11 Success 0 5p7 0:34 Success 0 5p8 0:43 Success 0 5p9 0:19 Success 0 5p10 0:21 Success 0 5
ExpectedTimeonTask 0:30
AverageTimeonTask 0:38 SD=0:17
AverageTaskSatisfaction 5.0 SD=0
Average#PathDeviations 0.00 SD=0
PercentSuccess 100%
Section170.315(b)(3)ElectronicprescribingParticipantNumber
Task3ElectronicPrescribing
TaskTime Outcome #PathDeviationsTask
Satisfaction
p1 2:18 Success 0 5p2 2:48 Success 2 5p3 2:28 Success 0 5p4 2:52 Success 0 4p5 2:18 Success 0 5p6 1:58 Success 0 5p7 3:14 Success 0 5p8 4:08 Success 0 3p9 4:25 Success 3 4p10 1:23 Success 0 5
ExpectedTimeonTask 3:00
AverageTimeonTask 2:47 SD=0:53
AverageTaskSatisfaction 4.60 SD=0:66
Average#PathDeviations 0.50 SD=1.02
PercentSuccess 100%
ParticipantNumber
Task4ElectronicPrescribing--Cancel/Change
TaskTime Outcome #PathDeviationsTask
Satisfaction
p1 3:10 Success 0 5p2 1:59 Success 0 5p3 2:08 Success 0 5p4 2:09 Success 0 4p5 1:24 Success 0 5p6 2:46 Success 0 5p7 3:22 Success 0 4p8 3:29 Success 1 4p9 2:17 Success 0 5p10 1:35 Success 1 5
ExpectedTimeonTask 3:00
AverageTimeonTask 2:25 SD=0:41
AverageTaskSatisfaction 4.70 SD=0.46
Average#PathDeviations 0.20 SD=0.4
PercentSuccess 100%
ParticipantNumber
Task5ElectricPrescribing—Renew
TaskTime Outcome #PathDeviationsTask
Satisfaction
p1 0:21 Success 0 5p2 0:32 Success 0 5p3 0:49 Success 0 5p4 0:23 Success 0 5p5 1:04 Success 0 5p6 0:49 Success 0 5p7 0:22 Success 0 5p8 0:47 Success 0 5p9 0:21 Success 0 5p10 0:09 Success 0 5
ExpectedTimeonTask 0:30
AverageTimeonTask 0:33 SD=0:16
AverageTaskSatisfaction 5.00 SD=0
Average#PathDeviations 0.00 SD=0
PercentSuccess 100%
DiscussionofFindings
IngeneraltheparticipantsperformedverywellandfeltsatisfiedwiththeRcopia
system.Afewoftheparticipantsstruggledinitiallywithsomeportionsofthetasksbutin
generalmostwereabletosuccessfullycompletealargemajorityofthetasks.Participants
weremostlyabletoperformalltaskssuccessfullyontheirownwithnotrainingor
documentation.Theparticipantaverageperformanceratewasveryhigh,aswerethe
overallparticipantsatisfactionrates.TheRcopiasystemappearstobehighlyusable.
Effectiveness
Ofthenine(9)taskspresented,alargemajorityofthetasksweresuccessfully
completedbyalloftheparticipants.Twotaskshadonlyoneparticipantfailtosuccessfully
completethattaskintheallottedtime.Thesetaskswere:Task6(ProblemList
Intervention),andTask8(MedicationOrder--Drug-DrugInteraction).Overallof
participants,themeansuccessfultaskcompetitionratewasveryhighwithanoverall
averagerateofninety-eight(98)percentindicatingthatingeneraltheparticipantshad
littleornodifficultycompletingthetasks.
PriorexperiencewithEHRsystemsandwithRcopiawasrelatedtosuccessfultask
performance;participantswithmorepriorEHRexperienceweremorelikelyto
successfullycompletetasksthanthoselesspriorexperience.
Efficiency
Participantswhosuccessfullycompletedtasksgenerallycompletedthosetasks
withinanacceptabletime.Sometaskswerecompletemorequicklythanthecalculated
optimaltime,whileseveraltaskstookonlyslightlylongerthanexpected.Thetasksthat
tookthelongestrequiredtheparticipantstonavigatetoaparticularunfamiliarportionofa
page,interactwithaworkflow,locateandselectspecificactions.Thosetaskscouldbe
performedmorequicklywithaminorupdatetotheinformationarchitecture.
Someparticipantsmadeerrorswhenattemptingtonavigatetowardsolvingtheir
assignedtasks.Theseerrorsmaybeassociatedwiththoseparticipantsnotbeingfamiliar
andnotunderstandingthepresentedinformationarchitectureoftheRcopiaEHRsystem.
Asnotedabove,priorexperiencewithEHRsystemswasrelatedtosuccessfultask
completion.Similarly,experienceandpracticewiththegivensystemmayhavepositive
effectswithregardtouserefficiency.
Satisfaction
ParticipantswereverysatisfiedwiththeRcopiaEHRsystem;ratingsontheSUS
(mean=84)andtheCSUQ(Overallscore=6.05)demonstratedaveryhighlevelof
satisfactionwiththesystem.
OntheCSUQ,participantsrankedthescale“(SystemUsefulness)”highestofthe
threescales,suggestingthatthesystemprovidedaneffectiveandefficientuserinterface.
Individualtasksatisfactionratingswererelatedtoindividualuserperformance.Those
participantswhowereabletosuccessfullycompletetaskswerealsomorelikelytorank
thosetasksassatisfying,whilethoseparticipantswhodidpoorlyorwerenotableto
completeataskrankedthosetasksaslesssatisfying.Overallhowever,thehighparticipant
satisfactionwithRcopiaEHRwasexpectedgiventhehighperformancedata.
SummaryofMajorFindings
ThisevaluationdemonstratedthattheRcopiasystemisahighlyusablesystemwith
arelativelyshortlearningcurve.SomeparticipantshadneverusedtheRcopiasystem
beforethestudyandexperiencedverylittleinitialdifficultyunderstandingthenavigation
andinformationarchitecture.
AreasforImprovement
Thefollowingisalistofpotentialareasforimprovement.Makingtheseandother
minorenhancementswillimprovetheoveralluserexperienceoftheRcopiasystemand
increasetheeffectiveness,efficiency,andsatisfactionforbothexperiencedandnovice
Rcopiausers.
• IndicationofRequiredFields
o Themostfrequenterrorwascausedwhenparticipantstriedtosubmita
formwhenrequiredfieldswerenotcompleted.Thisismostlikely
becausethesystemdidnotprovideanindicationofwhichfieldsare
required.Addingthissimplefeaturewouldlikelyeliminatemanyofthese
errorsobserved.
• ProblemListNavigation
o Acommonissuethatsomeparticipantshadwiththesystemwasthat
theyhaddifficultyfindingthe“ManageProblems”feature.Mostother
majorfeatureshadlinksthatareprominentlyatthetopofthescreen.
Perhapsaddingthistothemainnavigationwouldhelpelevatethisissue.
Appendices
AppendixA:RecruitingScreener1.Areyoumaleorfemale? 2.Haveyouparticipatedinafocusgrouporusabilitytestinthepast6months?3.Doyou,ordoesanyoneinyourhomeworkinmarketingresearch,usabilityresearch,and/orwebdesign?4.Doyou,ordoesanyoneinyourhome,haveacommercialorresearchinterestinanelectronichealthrecordsoftwareorconsultingcompany?5.Whichofthefollowingbestdescribesyourage?
20-29,30-39,40-49,50-59,60-69,70-79,80-89,90-99,100andolder.6.Whichofthefollowingbestdescribesyoureducationlevel?
• Nohighschooldiploma• Highschoolgraduate,diplomaortheequivalent• Somecollegecredit,nodegree• Tradetechnicalvocationaltraining• Associatedegree• Bachelor’sdegree• Master’sdegree• Doctoratedegree(e.g.,MD,DNP,DMD,PhD)
7.Doyourequireanyassistivetechnologiestouseacomputer?8.Pleasedescribeyourmedicalornursingcredentials9.Whatisyourcurrentjobtitle?10.Howlonghaveyouheldthisposition?(numberofyears):11.Whattypeoffacilitydoyouworkinandwhatisyourrolethere?12.Howaremedicalrecordshandledatyour(main)workplace?
_____AllPaper _____SomePaper/SomeElectronic ___AllElectronic13.HowmanyEHRsdoyouuseorhaveyouworkedwith?14.Howmanyyearshaveyouusedanelectronichealthrecord?
15.HowmanyyearshaveyouusedtheDrFirstRcopiasystem?16.Abouthowmanyhoursperweekdoyouspendusingacomputer?17.Whatcomputerplatform(s)doyouusuallyuse?18.Inthelastmonth,abouthowoftenhaveyouusedanelectronichealthrecord?
_____Didnotuselastmonth ___Everyday _____Afewtimesaweek.
AppendixB:InformedConsentFormTheUsabilityPeoplewouldliketothankyouforparticipatinginthisstudy.Thepurposeofthisstudy is toevaluateanelectronichealthrecordssystem. Ifyoudecide toparticipate,youwillbeaskedtoperformseveraltasksusingtheprototypeandgiveyourfeedback.Thestudywilllastabout45minutes.AgreementIunderstandandagreethatasavoluntaryparticipant inthepresentstudyconductedbyThe Usability People. I am free towithdraw consent or discontinue participation at anytime. I understandandagree toparticipate in the study conducted and recordedbyTheUsabilityPeople.I understandand consent to theuse and releaseof the video recordingbyTheUsabilityPeople.Iunderstandthattheinformationandvideoisforresearchpurposesonlyandthatmynameandimagewillnotbeusedforanypurposeotherthanresearch.Irelinquishanyrights to the video and understand the video recordingmay be copied and used by TheUsabilityPeoplewithoutfurtherpermission.I understand and agree that the purpose of this study is tomake software applicationsmoreusefulandusableinthefuture.I understandandagree that thedata collected from this studymaybe sharedoutsideofTheUsabilityPeople. Iunderstandandagreethatdataconfidentiality isassured,becauseonlyde-identifieddata– i.e., identificationnumbersnotnames–willbeused inanalysisandreportingoftheresults.I agree to immediately raise any concerns or areas of discomfort with the studyadministrator.IunderstandthatIcanleaveatanytime.Pleasecheckoneofthefollowing:____YES,Ihavereadtheabovestatementandagreetobeaparticipant.____NO,Ichoosenottoparticipateinthisstudy.Signature:_____________________________________Date_____________________
AppendixC:ParticipantTasksTodayyouareOneProvider,MD,andatreatingphysicianatBeavertonFamilyMedicine.You’llbeinteractingwiththeDrFirstRcopiasystemanddocumentingtheinformationabouttoday’sencounterwithyourpatient.TASK1:MedicationAllergyList-Addanallergy.YourpatientRachelTaylorhascomeinfortheirappointment.Theyareanexistingpatientandtheirinformationisalreadyloadingintothesystem.Examinetheirmedicalrecord.Asyoubeginyourconsultationthepatienttellsyouthattheyrememberthataftertakingadrugthatcontainedsulfaawhilebacktheyhadexperiencedsevereurticaria(hives).
YoudecidetoaddanallergyforSulfadixineorSulfa(SulfonamideAntibiotics)totheirrecord.EnterandsavethisinformationintotheEHR.
TASK2:Drug-AllergyInteractionCheck.
Yourpatientsaidthatthemainreasontheycameintoyourofficeisthattheystartedtofeelpoorlyaftereatingasaladbowlfromalocalfast-foodMexicanrestaurantchain.
Youthinkthattheymayhavecontractedane-Coliinfection.
YoudecidetoprescribeKeflex(cephalexin)capsule250MG.However,onaddingKeflexyounoticeadrug-allergyinteractionalertsoyoudecidetostopthisprescription.
EntertheinformationaboveintotheEHRandidentify/viewthedrug-allergyalert.
TASK3:ElectronicPrescribing.
Youstillneedtoprescribeamedicationtowhichthepatientisnotallergic.YoudecidetoorderDoxycyclineinstead.
Pleaseprescribe:
• DoxycyclineHyclate20MGtabletstobetakenonetablettwiceadayaftermealsfor5days.
• SendtheRXtothechosenpharmacyusingthefollowingSIGpassword:userrx
Task4.ElectronicPrescribingCancel/Change
Oops,youjustrealizedthatyouprescribedandsentelectronicallytothepharmacytheantibioticaboveat20mg,butyoureallywantedtoprescribethe100mgDoxycyclinetablet.
• Navigatetoareportoftheprescriptionsthatweresentelectronicallyandcancelthe20mgtablet.• PrescribeDoxycyclineHyclate100MGtabletstobetakenonetablettwiceadayaftermeals
for5days.
Task5.ElectronicPrescribing–renew
Youpatienthasalsocomplainedofoccasionalbreathingproblems.YounoticethattheyhaveanexistingprescriptionforProventilHFA(albuterolsulfate).
• Renew/RefilltheRX.
TASK6:ProblemListCDSIntervention.
Yourassistanthasmeasuredthepatient’sbloodpressureandrecordedthefollowingvalues:150/90.Consideringtheexaminationandotherfindings,youhavediagnosedyourpatientwiththefollowing:
401.9-HypertensionNotOtherwiseSpecified(ICD-9)
EnterthisinformationintotheEHR.
Task7:StopamedicationUponexaminationofthepatient’scurrentmedicationlist,younoticethatthebloodpressuredrugLisinoprilislistedasanactivemedication.YouhaveplanstoenteranewprescriptionforAccupriltotreattheirhighbloodpressure,butfirstyouneedtostopthepatientfromtakingLisinoprilandindicatethisstopintheirrecord.
• EnteramedicationstopforLisinopril.
TASK8:MedicationOrder,ViewDrug-DrugInteractions,MedicationListInterventions.Inordertotreatyourpatient’selevatedbloodpressure,youdecidetoaddAccupriltotheirhypertensiondiagnosis.
Enteraprescriptionfor:
• AccuprilOralTablet10MGtotakeonetablettwiceadayfor30days.
Yourpatientalsocomplainsofjointpain,soyoudecidetoprescribeNaproxenTablet(generic)250mg.
Enteraprescriptionfor:
• Naproxen250MGOralTablettotakeonetablet,onceadayasneededfor30days.
Duringprescribing,youwillbenotifiedaboutadrug-druginteraction.Viewtheassociatedwarning.Thepatientclaimsthattheynolongerhaveanallergicreactiontoaspirinproducts.TASK9:MedicationAllergyList-Removeanallergy.
Duringtheconsultation,yourpatienthastoldyouthattheynolongerhaveanallergytoAspirin.TheaspirinallergywasrecordedasAddedStrengthHeadacheRelief(aspirin-acetaminophen-caffeine).Youdecidetoremovethisallergyfromthepatientrecord.
• RemovetheAspirinallergyfromthepatientrecordintheEHR.
AppendixD:SystemUsabilityScaleQuestionnaire
AppendixE:ComputerSystemUsabilityQuestionnairePleaseprovideyourimpressionoftheusabilityofthesystembyansweringeachofthequestionsbelow:1.Overall,IamsatisfiedwithhoweasyitistousethissystemStrongly 1234567 NA StronglyDisagree Agree2.ItwassimpletousethissystemStrongly 1234567 NA StronglyDisagree Agree3.IcaneffectivelycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree4.IamabletocompletemyworkquicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree5.IamabletoefficientlycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree6.IfeelcomfortableusingthissystemStrongly 1234567 NA StronglyDisagree Agree7.ItwaseasytolearntousethissystemStrongly 1234567 NA StronglyDisagree Agree8.IbelieveIbecameproductivequicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree9.ThesystemgiveserrormessagesthatclearlytellmehowtofixproblemsStrongly 1234567 NA StronglyDisagree Agree10.WheneverImakeamistakeusingthesystem,IrecovereasilyandquicklyStrongly 1234567 NA StronglyDisagree Agree
11.Theinformation(suchasonlinehelp,on-screenmessages,andotherdocumentation)providedwiththissystemisclearStrongly1 234567 NA StronglyDisagree Agree12.ItiseasytofindtheinformationIneededStrongly 1234567 NA StronglyDisagree Agree13.TheinformationprovidedforthesystemiseasytounderstandStrongly 1234567 NA StronglyDisagree Agree14.TheinformationiseffectiveinhelpingmecompletethetasksandscenariosStrongly 1234567 NA StronglyDisagree Agree15.TheorganizationofinformationonthesystemscreensisclearStrongly 1234567 NA StronglyDisagree Agree16.TheinterfaceofthissystemispleasantStrongly 1234567 NA StronglyDisagree Agree17.IlikeusingtheinterfaceofthissystemStrongly 1234567 NA StronglyDisagree Agree18.ThissystemhasallthefunctionsandcapabilitiesIexpectittohaveStrongly 1234567 NA StronglyDisagree Agree19.Overall,IamsatisfiedwiththissystemStrongly 1234567 NA StronglyDisagree Agree