Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 1
Part 1: Product and Developer Information1.1 Certified Product Information
1.2 Developer/Vendor InformationMeta Healthcare IT Solutions
Part 2: ONC-Authorized Certification Body Information2.1 ONC-Authorized Certification Body Information
4/10/2017
Test Type: Modular
Developer/Vendor Name:Address: 401 Franklin Ave. Suite 106
Garden City, NY 11530
ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification
Product Name: MetaCare Enterprise EHRProduct Version: 6.4Domain: Inpatient
Developer/Vendor Contact: Matthew Adams
ONC-ACB Name: InfoGard Laboratories, Inc.Address: 709 Fiero Lane Suite 25
Website: www.metacaresolutions.comEmail: [email protected]: (800) 768-1920
ONC-ACB Contact: Adam Hardcastle
This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative:
Adam Hardcastle EHR Certification Body ManagerONC-ACB Authorized Representative Function/Title
San Luis Obispo, CA 93401Website: www.infogard.comEmail: [email protected]: (805) 783-0810
Signature and Date
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 2
2.2 Gap Certification
(a)(1) (a)(19) (d)(6) (h)(1)(a)(6) (a)(20) (d)(8) (h)(2)(a)(7) (b)(5)* (d)(9) (h)(3)(a)(17) (d)(1) (f)(1)(a)(18) (d)(5) (f)(7)**
*Gap certification allowed for Inpatient setting only**Gap certification allowed for Ambulatory setting only
2.3 Inherited CertificationThe following identifies criterion or criteria certified via inherited certification
(a)(1) (a)(16) Inpt. only (c)(2) (f)(2) (a)(2) (a)(17) Inpt. only (c)(3) (f)(3)
The following identifies criterion or criteria certified via gap certification§170.314
No gap certification
§170.314
(d)(3) (a)(6) (b)(1) (d)(4)
(f)(6) Amb. only (a)(7) (b)(2) (d)(5)
(a)(3) (a)(18) (d)(1) (f)(4) Inpt. only
(a)(4) (a)(19) (d)(2) (f)(5) Amb. only
(a)(5) (a)(20)
(a)(10) (b)(5) (d)(8) (g)(2) (a)(11) (b)(6) Inpt. only (d)(9) Optional (g)(3)
(a)(8) (b)(3) (d)(6) (f)(7) Amb. Only
(a)(9) (b)(4) (d)(7) (g)(1)
(a)(14) (b)(9) (e)(3) Amb. only (h)(2) (a)(15) (c)(1) (f)(1) (h)(3)
(a)(12) (b)(7) (e)(1) (g)(4) (a)(13) (b)(8) (e)(2) Amb. only (h)(1)
No inherited certification
Test Results Summary for 2014 Edition EHR Certification
16-2592-R-0017-PRI Vl.O, April 10, 2017
Part 3: NVLAP-Accredited Testing Laboratory Information
Report Number: Test Date(s):
16-2592-R-0017 Vl.O
N/A
3.1 NVLAP-Accredited Testing Laboratory Information ATLName:
Accreditation Number:
Address:
Website:
Email:
Phone:
ATL Contact:
lnfoGard Laboratories, Inc.
NVLAP Lab Code 100432-0
709 Fiero Lane Suite 25
San Luis Obispo, CA 93401
www.infogard.com
ehr@infogard .com
(805) 783-0810
Milton Padilla
For more information on scope of accreditation, please reference
http://ts.nist.gov/Standards/scopes/1004320.htm
lNFOIGARD
Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory
Representative:
Mark Shin EHR Test Body Approved Signatory
ATL Authorized Representative Function/Title
~ 4/10/2017 Signature and Date
3.2 Test Information
3.2.1 Additional Software Relied Upon for Certification
Additional Software Applicable Criteria Functionality provided by
Additional Software
Secure Trans (b)1,(b)2,(e)1 Secure transmission
MetacareRx (Meta Info System) (a)16 eMAR
First Data Bank (a)lO Drug formulary checking
D No additional software required
©2017 lnfoGard. May be reproduced only in its original entirety, without revision 3
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 4
3.2.2 Test Tools
Version
No test tools required
3.2.3 Test Data
3.2.4 Standards3.2.4.1 Multiple Standards Permitted
HL7 v2 Laboratory Restults Intervace (LRI) Validation ToolHL7 v2 Syndromic Surveillance Reporting Validation ToolTransport Testing ToolDirect Certificate Discovery Tool
Edge Testing Tool
Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter ]
Test ToolCypressePrescribing Validation ToolHL7 CDA Cancer Registry Reporting Validation ToolHL7 v2 Electronic Laboratory Reporting (ELR) Validation ToolHL7 v2 Immunization Information System (IIS) Reporting Valdiation Tool
(a)(13)
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
§170.207(j)HL7 Version 3 Standard: Clinical Genomics; Pedigree
No alteration (customization) to the test data was necessary
The following identifies the standard(s) that has been successfully tested where more than one standard is permitted
Criterion # Standard Successfully Tested
(a)(8)(ii)(A)(2)
§170.204(b)(1)HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain
§170.204(b)(2)HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 5
None of the criteria and corresponding standards listed above are applicable
3.2.4.2 Newer Versions of Standards
No newer version of a minimum standard was tested
(a)(15)(i)
§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain
§170.204(b)(2)HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide
(b)(7)(i)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
(b)(8)(i)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
(a)(16)(ii) §170.210(g) Network Time Protocol Version 3 (RFC 1305)
§170. 210(g)Network Time Protocol Version 4 (RFC 5905)
(b)(2)(i)(A)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
(e)(3)(ii) Annex A of the FIPS Publication 140-2
Common MU Data Set (15)
§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
§170.207(b)(2)The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4)
(e)(1)(i) Annex A of the FIPS Publication 140-2
(e)(1)(ii)(A)(2) §170.210(g) Network Time Protocol Version 3 (RFC 1305)
§170. 210(g)Network Time Protocol Version 4 (RFC 5905)
The following identifies the newer version of a minimum standard(s) that has been successfully tested
Newer Version Applicable Criteria
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 6
3.2.5 Optional Functionality
No optional functionality tested
(a)(4)(iii) Plot and display growth charts
(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)
Criterion # Optional Functionality Successfully Tested
(f)(3)
Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)
(f)(7) Ambulatory setting only – transmission to public health agencies – syndromic surveillance - Create Data Elements
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)
(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(e)(1) View, download and transmit data to a third party using the standard specified at §170.202(d) (Edge Protocol IG version 1.1)
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 7
3.2.6 2014 Edition Certification Criteria* Successfully Tested
TP** TD*** TP** TD***
(a)(1) (c)(3) (a)(2) (d)(1) (a)(3) (d)(2)
Criteria #Version
Criteria #Version
(a)(7) (d)(6) (a)(8) (d)(7) (a)(9) (d)(8)
(a)(4) (d)(3) (a)(5) (d)(4) (a)(6) (d)(5)
(a)(13) (e)(3) Amb. only
(a)(14) (f)(1) (a)(15) (f)(2)
(a)(10) (d)(9) Optional
(a)(11) (e)(1) (a)(12) (e)(2) Amb. only
(a)(19) (a)(20) (f)(6) Optional &
Amb. only (b)(1)
(a)(16) Inpt. only (f)(3) (a)(17) Inpt. only (f)(4) Inpt. only
(a)(18) (f)(5) Optional & Amb. only
(b)(5) (g)(3) (b)(6) Inpt. only (g)(4) (b)(7) (h)(1)
(b)(2) (f)(7) Amb. only
(b)(3) (g)(1) (b)(4) (g)(2)
*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)**Indicates the version number for the Test Procedure (TP)***Indicates the version number for the Test Data (TD)
(b)(8) (h)(2) (b)(9) (h)(3) (c)(1) (c)(2)
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 8
3.2.7 2014 Clinical Quality Measures*Type of Clinical Quality Measures Successfully Tested:
CMS ID Version CMS ID Version CMS ID Version CMS ID Version2 90 136 155
22 117 137 15650 122 138 15752 123 139 15856 124 140 15961 125 141 16062 126 142 16164 127 143 16365 128 144 16466 129 145 16568 130 146 16669 131 147 16774 132 148 16975 133 149 17777 134 153 17982 135 154 182
CMS ID Version CMS ID Version CMS ID Version CMS ID Version9 71 107 172
26 72 108 17830 73 109 18531 91 110 18832 100 111 19053 102 11355 104 11460 105 171
Ambulatory Inpatient No CQMs tested*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov (navigation: 2014 Clinical Quality Measures)
Ambulatory CQMs
Inpatient CQMs
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 9
3.2.8 Automated Numerator Recording and Measure Calculation3.2.8.1 Automated Numerator Recording
Automated Numerator Recording was not tested
3.2.8.2 Automated Measure Calculation
Automated Measure Calculation was not tested
3.2.9 Attestation
(a)(3) (a)(12) (a)(19) (b)(8)(a)(4) (a)(13) (a)(20) (b)(9)
Automated Numerator Recording Successfully Tested(a)(1) (a)(11) (a)(18) (b)(6)
(a)(7) (a)(16) (b)(4) (e)(3)(a)(9) (a)(17) (b)(5)
(a)(5) (a)(14) (b)(2) (e)(1)(a)(6) (a)(15) (b)(3) (e)(2)
(a)(4) (a)(13) (a)(20) (b)(9)(a)(5) (a)(14) (b)(2) (e)(1)
Automated Numerator Recording Successfully Tested(a)(1) (a)(11) (a)(18) (b)(6)(a)(3) (a)(12) (a)(19) (b)(8)
Attestation Forms (as applicable) Appendix
(a)(6) (a)(15) (b)(3) (e)(2)(a)(7) (a)(16) (b)(4) (e)(3)
Safety-Enhanced Design* A Quality Management System** B Privacy and Security C
(a)(9) (a)(17) (b)(5)
*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9)**Required for every EHR product
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 10
Appendix A: Safety Enhanced Design
User-Centered Design Process
Meta Healthcare IT Solutions has employed the User Centered Design approach outlined below
to enhance the usability of our inpatient EHR application, Metacare Enterprise, and to meet the
Safety-Enhanced Design 170.314(g)(3) requirement for Meaningful Use Stage 2 EHR
Certification. This outline is derived from the NIST Guide to the Processes Approach for
Improving the Usability of Electronic Health Records (NIST
7741: http://www.nist.gov/customcf/get_pdf.cfm?pub_id=907313) and has be modified to fit in to
Meta Healthcare IT Solution’s quality management system design processes.
1. Documentation of User Needs and Requirements
Prior to development of new modules within the EHR product, Meta conducts a requirements
analysis which includes any client needs, regulatory requirements, or EHR certification
requirements. This requirements analysis includes interviews with client end-users to ensure
user needs are incorporated into the requirements if they did not originate from users
themselves. The requirements are then documented, along with suggestions on how the EHR
application should fulfill the requirement, using appropriate methodologies (user stories,
diagrams, etc.)
2. Workflow Development
Once requirements analysis has been conducted, the requirements are given to the
development team to review. Using the requirements analysis, the development team conducts
a workflow analysis as a basis for the user interface design. This analysis includes consultation
with Meta’s clinical analysts and client managers as well as consultation with client end-users
themselves. Baseline performance criteria are also documented for the requirements. The
results of the analysis are documented and then used during the development of the database
design and information architecture.
3. Development of Low-Fidelity Prototypes
After workflow development and information architecture design has been completed, the
development team creates a prototype for review by Meta’s clinical team prior to development.
The prototypes can be in the form of wireframes, screenshots of early development, or even
demonstrations of a basic working prototype application module, given to or conducted in front
of client end-users for their review. The development team uses feedback from this end-user
review to develop higher-level prototypes.
4. Development of High-Fidelity Prototypes
A full, executable prototype is then developed using the end-user feedback from earlier
prototypes. This prototype is subjected to unit testing by Meta’s software quality assurance
team. Any bugs in the software are reported to the developers. Once the prototype as passed
quality assurance testing, the module subjected to full system and regression testing as part of
the EHR. The application is then tested by client end-users who conduct testing of the
application and send feedback to the development test for further iterations of application
development.
5. Formal Usability Testing
Once the module is incorporated into the EHR, Meta conducts formal lab-based summative
usability testing on the EHR software module in question to obtain measures of users’
effectiveness, efficiency, and satisfaction using the application, the results of which are used for
further iterations of application development.
EHRUsabilityTestReportofMetaHealthcareITSolutions ProductVersion:MetaCareEnterpriseEHRv6.3.2ReportbasedonISO/IEC25062:2006CommonIndustryFormatforUsabilityTestReports
DateofUsabilityTest:May202014DateofReport:May23,2014
ReportPreparedBy:
TheUsabilityPeople,LLC
4000LegatoRoad,Suite1100Fairfax,VA22033
TABLEOFCONTENTSExecutiveSummary.................................................................................................................................................3Introduction................................................................................................................................................................4Method...........................................................................................................................................................................5Participants.............................................................................................................................................................5StudyDesign...........................................................................................................................................................7Tasks..........................................................................................................................................................................8TestLocation..........................................................................................................................................................8TestEnvironment................................................................................................................................................9TestFormsandTools......................................................................................................................................10ParticipantInstructions.................................................................................................................................10Procedure.............................................................................................................................................................11UsabilityMetrics................................................................................................................................................12DataScoring....................................................................................................................................................13
Results........................................................................................................................................................................15DataAnalysisandReporting........................................................................................................................15EffectivenessandEfficiency....................................................................................................................15Satisfaction......................................................................................................................................................16
DiscussionofFindings....................................................................................................................................19Effectiveness...................................................................................................................................................19Efficiency..........................................................................................................................................................19Satisfaction......................................................................................................................................................20SummaryofMajorFindings....................................................................................................................21RiskAnalysis..................................................................................................................................................22AreasforImprovement.............................................................................................................................23
Appendices...............................................................................................................................................................24Appendix1:RecruitingScreener...............................................................................................................24Appendix2:AdditionalParticipantInformation................................................................................25Appendix3:InformedConsentForm.......................................................................................................26Appendix4:ParticipantTasks.....................................................................................................................27Appendix5:SystemUsabilityScaleQuestionnaire............................................................................31Appendix6:ComputerSystemUsabilityQuestionnaire..................................................................32Appendix7:Detailedtaskperformanceforeachparticipant........................................................34
ExecutiveSummary
On May 20, 2014 The Usability People, LLC conducted a usability test of Meta
Healthcare IT Solutions’ MetaCare Enterprise EHR. The test was conducted in the Fairfax, VA
office of The Usability People over remote tele-conferencing sessions using Go To Meeting.The
purposewastotestandvalidatetheusabilityofthecurrentuserinterfaceandprovide
evidenceofusabilityofMetaCareEnterpriseEHRastheEHRUnderTest(EHRUT).Six(6)
healthcareprovidersmatchingthetargetdemographiccriteriaparticipatedintheusability
testusingtheEHRUTinsimulated,butrepresentativetasks.
Thestudyfocusedonmeasuringtheeffectivenessof,efficiencyof,andsatisfaction
withMetaCareEnterpriseEHRamongasampleofparticipantsrepresentingpotentialusers
ofthesystem.Performancedatawascollectedontwelve(12)taskstypicallyconductedon
anEHR.Taskscreatedwerebaseduponthecriteriaspecifiedwithinthetestprocedure
structureforevaluatingconformanceofElectronicHealthRecord(EHR)technologytothe
certificationcriteriadefinedin45CFRPart170SubpartCoftheHealthInformation
Technology:Standards,ImplementationSpecifications,andCertificationCriteriafor
ElectronicHealthRecordTechnology,2014Edition;RevisionstothePermanent
CertificationProgramforHealthInformationTechnology,FinalRuleaspublishedinthe
FederalRegisteronSeptember4,2012.
ResultsofthestudyindicatedthattheMetaCareEnterpriseEHRsystemwas
satisfactorywithregardstoeffectivenessandefficiencyandparticipantswereminimally
satisfiedwiththesystem.
Introduction
TheElectronicHealthRecordSystemUnderTest(EHRUT)testedforthisstudy,
MetaCareEnterpriseEHR,wasspecificallydesignedtopresentmedicalinformationto
physicians,nursesandotherhealthcarepractitionersondesktopcomputersinstandard
medicalcaresettings.Thisstudytestedandvalidatedtheusabilityofthecurrentuser
interfaceandprovidesevidenceoftheusabilityofMetaCareEnterpriseEHRwith
representativeexercisesandinrealisticuserconditions.Tothisend,measuresof
effectivenessandefficiency,suchastimeontask,numberoferrorsmade,andcompletion
rateswerecapturedduringusabilitytesting.Satisfactionwasassessedandusercomments
collectedusingtwoindustry-standardquestionnaires.
Method
Participants
Sixindividuals(3menand3women)participatedintheEHRUT(MetaCare
EnterpriseEHR).Participantswererecruitedfromadatabaseofpotentialparticipants
maintainedbyTheUsabilityPeople,LLC.Thecontactscontainedwithinthisdatabasewere
generatedviapotentialparticipantsresponsestopostingsinpopularInternetandsocial
mediasites,andalinkatthebottomofTheUsabilityPeoplewebsite.Thosewho
respondedtotheinvitationtotakepartinthestudyweredirectedtoanonline
questionnairethatservedastheparticipantscreener.(Thescreeningquestionnaireis
providedasAppendix1.)Participantsmeetingthecriteriaforparticipationinthestudy
werecontactedandscheduledviaemail.
ParticipantsintheusabilitytestofMetaCareEnterpriseEHRhadavarietyof
healthcarebackgroundsanddemographiccharacteristics.Participantsdidnothaveany
directexperienceortrainingusingtheMetaCareEnterpriseEHREHRsystem.
Table1presentsparticipantcharacteristics,includingdemographics,professional
experience,computingexperience,andnumberofpreviousEHR'sused.Noneofthe
participantswerefromthevendororganization(MetaHealthcareITSolutions)that
producedandsuppliedtheevaluatedsystemnordidanyparticipanthaveanydirect
connectiontothetestingorganization(TheUsabilityPeople,LLC).Ascompensationfor
theirparticipationallindividualsreceivedagiftcard.(Additionalparticipantbackground
characteristicsarepresentedinAppendix2.)
Table1.ParticipantCharacteristics
PartID Gender Age Education Role/Title Workfacility
ProfessionalExperience
(yrs)
#EHRsworkedwith
EHRExperience
(yrs)
Hrs/WkwithEHRs?
AssistiveTechNeeds
p1
Female 40to59 RTTRadiationTherapist
Freestandingclinic.50%RTT/50%Manager 18 2 4 20 None
p2
Male 23to39 BSHA,R.T.(R)(T)
AdministrativeDirectorofOncologyServices
University(Academic)setting,
overseeday-to-day
operatingproceduresofanOncologyServiceLine. 12 2 3 30
None
P3
Male 23to39 MD Nephrologist
Publicuniversityhospital.
Manageandtreatpatientswithrenaldisease. 7 3 7 50 None
p4
Female 23to39 MSN,CNL,RN
ClinicalInformaticsOfficer
WorkflowoptimizationwithinEHRs
andregulatory/ARRA/Meaningfu
lUseguidance. 16 6 16 50 None
p5
Male 40to59
CompletedmultipleEHR
coursesEHRSystemsDirector
DirectorofEHRatmajor
hospitalnetworkinChicago 13 6 12 45 None
p6
Female 40to59BSN,OncologyCertifiedNurse
OncologyNurse
Manager
UniversityofTexasHealth
ScienceCenteratTyler
outpatientclinic 23 2 8 45 None
StudyDesign
TheoverallobjectiveofthisusabilitytestwastouncoverareaswheretheMetaCare
EnterpriseEHRapplicationperformedwell–thatis,effectively,efficiently,andwith
satisfaction–andareaswheretheapplicationfailedtoservetheneedsofusers.Datafrom
thistestmaybeusedasabaselineforfuturetestsofupdatedversionsofMetaCare
EnterpriseEHRand/orforcomparingMetaCareEnterpriseEHRwithotherEHRspresenting
thesametasks.Inshort,thistestingservesasbothameanstorecordorbenchmark
currentusabilityandtoidentifyareaswhereimprovementsmustbemade.
ParticipantshadawiderangeofexperiencewithEHRsandgeneral,andnonehad
anydirectexperienceand/ortrainingwiththeMetaCareEnterpriseEHRsystem.
ParticipantscompletedthetestofMetaCareEnterpriseEHRusabilityduringindividual60-
minuteGoToMeetingsessions.Duringthetest,eachparticipantinteractedwithvarious
componentsoftheMetaCareEnterpriseEHR.Eachparticipantwasprovidedwiththesame
instructions.
MetaCareEnterpriseEHRwasevaluatedforeffectiveness,efficiencyandsatisfaction
asdefinedbythefollowingmeasurescollectedandanalyzedforeachparticipant:
• Numberoftaskssuccessfullycompletedwithoutassistance
• Timetocompletethetasks
• Numberanddescriptionoferrors
• Pathdeviations
• Participant’sverbalizations(comments)
• Participant’ssatisfactionratingsofthesystem
Tasks
ArepresentativeofMetaHealthcareITsolutionscreatedatotaloftwelve(12)tasks
toberealisticandrepresentativeoftheactivitiesausermightengagewithMetaCare
EnterpriseEHRinactualmedicalsettings.Thetwelve(12)taskswerecreatedbasedupon
thecriteriaspecifiedwithinthetestprocedurestructureforevaluatingconformanceof
ElectronicHealthRecord(EHR)technologytothecertificationcriteriaasdefinedin45CFR
Part170SubpartCoftheHealthInformationTechnology:Standards,Implementation
Specifications,andCertificationCriteriaforElectronicHealthRecordTechnology.
Thetasksfocusedonthefollowingissues:
1. ComputerizedProviderOrderEntrySystem(CPOE)(§170.314(a)(1));
2. Drug-drug,drug-allergyinteractionchecks(§170.314(a)(2));
3. Medicationlist(§170.314(a)(6));
4. Medicationallergylist(§170.314(a)(7));
5. Clinicaldecisionsupport(§170.314(a)(8));
6. Electronicmedicationadministrationrecord(§170.314(a)(16));
7. Electronicprescribing(§170.314(b)(3));
8. Clinicalinformationreconciliation(§170.314(b)(4)).
AcopyofthetaskspresentedtoparticipantsintheusabilitytestofMetaCareEnterprise
EHRcanbefoundinAppendix4.
TestLocation
AllparticipantsweretestedontheMetaCareEnterpriseEHRsystemduringremote
conferencingsessionsusingGoToMeeting.Eachparticipantwasrequestedinadvanceto
secureaquietroomwithminimaldistractionsandacomputerthatcouldconnecttothe
InternetwithaGoToMeetingsession.Thetypeofcomputer,operatingsystemanddisplay
resolutionoftheparticipantsystemwasunknown,butthesystemthatwasusedbythetest
administratorandtheremoteparticipantwasanHPAll-in-oneDesktoprunningthe
Windows8operatingsystemataresolutionof1280x768.DuringagivenGoToMeeting
session,onlythetestadministratorandthatparticipantcommunicatedwithoneanother.
TheGoToMeetingusabilitytestsessionwasconductedbyatestadministratorfrom
thetestingorganization(TheUsabilityPeople,LLC)workingfromasmallconferenceroom
atTheUsabilityPeople’sFairfax,VAlocation.Seatedneartheadministrator,adatalogger
fromthetestingorganizationtookdetailednotesoneachsession,includingusercomments
andsatisfactionratingsfollowingeachtask.ArepresentativefromMetaHealthcareIT
Solutionsloggedintotheremotesessiontoobservethesessionsandprovidetechnical
assistancerunningMetaCareEnterpriseEHR.Duringasessionthetestadministrator,the
datalogger,andtheMetaHealthcareSolutionsrepresentativecouldseeonlythe
participant’sscreenandheartheparticipant’scomments,questions,andresponses.
TestEnvironment
WhiletheEHRUTtypicallywouldbeusedinahealthcareofficeorambulatory
surgerycenterfacility,testingoftheMetaCareEnterpriseEHRsystemwasconductedvia
remoteconnectionduringindividualGo To Meetingsessions.Eachparticipantcalledintoa
GoToMeetingsessionandwasconnectedbythetestadministratortotheapplication.
TheMetaCareEnterpriseEHRapplicationitselfranonabrowserplatformonaLAN
connectionusingasampledatabasesetupspecificallyforthetest.Participantsuseda
mouseandkeyboardwheninteractingwiththeEHRUTandweregivenremotecontrolof
theadministrator’sworkstationtoperformthetasks.
TestFormsandTools
Aspartoftheusabilitytest,severaldocumentsandinstrumentswereused.Examples
ofthedocumentsusedduringtheusabilitytest,includinganinformedconsentform,the
tasks,andpost-testquestionnaires,canbefoundinAppendices3-6,respectively.
Participants’interactionwiththeMetaCareEnterpriseEHRwascapturedandrecorded
digitallyusingMoraescreencapturesoftwarerunningonthetestadministrator’s
workstation.Verbalresponseswererecordedthrougheitherthemicrophoneintegrated
intotheparticipant’scomputerorthroughatelephoneconnection.Thisinformationwas
electronicallytransmittedtotheadministratorandtothedataloggerduringeachtest
session.
ParticipantInstructions
Theadministratorreadthefollowinginstructionsaloudtoeachparticipant:
Thankyouforparticipatinginthisstudy.Yourinputisveryimportant.Oursessiontodaywilllastabout60minutes.Duringthattimeyouwilluseaninstanceofanelectronichealthrecord.Iwillaskyoutocompleteafewtasksusingthissystemandanswersomequestions.
Pleasenotethatwearenottestingyou;wearetestingthesystem.Thereforeifyouhaveanydifficultythismaymeanthatsomethingneedstobeimprovedinthesystem.Iwillbehereincaseyouneedspecifichelp,butIamnotabletoinstructyouorprovidehelpinhowtousetheapplication.
Overall,weareinterestedinhoweasy(orhowdifficult)thissystemistouse,whatinitwouldbeusefultoyou,andhowwecouldimproveit.Ididnothaveanyinvolvementinitscreation,sopleasebehonestwithyouropinions.Alloftheinformationthatyouprovidewillbekeptconfidentialandyournamewillnotbeassociatedwithyourcommentsatanytime.Shouldyoufeelitnecessaryyouareabletowithdrawatanytimeduringthetesting.
Participantswerethengiventwelve(12)taskstocomplete.
Procedure
Uponconnectiontotheonlinemeetingtool(GoToMeeting),eachparticipantwas
greeted,hisorheridentityverified,andmatchedtoanameontheparticipantschedule.
ParticipantnameswerereplacedwithparticipantIDssothatagivenindividual’sdata
cannotbelinkedtohis/heridentity.Priortobeginningtesting,eachparticipantreviewed
andsignedaninformedconsentform(SeeAppendix3)andemailedittotheorganization
(The Usability People, LLC) conductingthetest.
TwostaffmembersoftheUsabilityPeople,ausabilitytestadministratorandadata
logger,administeredthetest.Theadministratormoderatedthesessionbyprovidingboth
verbalandwritteninstructionsfortheoverallusabilitytestandforeachofthetasks
comprisingthetest.Theadministratoralsomonitoredtasksuccess,pathdeviations,
numberanddescriptionoferrors,andaudio-recordedparticipantverbalcomments.The
dataloggertasktimes,obtainedpost-taskratingdata,andtooknotesonparticipant
commentsandadministratorfeedback.
Foreachofthetwelve(12)tasks,participantswerepresentedwritteninstructions
totheircomputers.Followingtheadministrator’sinstructions,eachparticipantperformed
eachtaskbyfirstreadingthetaskoutloudthenstatinginhisorherownwordshisorher
interpretationofthetaskrequirements.Whentheparticipant’sinterpretationmatchedthe
actualgoalofthetask,theadministratorinstructedtheparticipanttobeginandtasktiming
began.Tasktimewasstoppedandrecordedwhenthetestadministratorobservedonhis
workstationthattheparticipanthadsuccessfullycompletedthetask.Ifaparticipantfailed
tocompleteataskbeforetheexpectedamountoftimeforeachtask,thattaskwasmarked
as“TimedOut.”Aftereachtask,thetestadministratoraskedtheparticipant,“Onascale
from1to5,where1is‘NotSatisfied’and5is‘Satisfied,’howsatisfiedwereyouwiththis
task?”Thissameprocedurewasconductedforeachofthetwelve(12)tasks.
Followingcompletionofthetwelve(12)EHRtasks,theadministratorelectronically
presentedtotheparticipanttwopost-testquestionnaires(SystemUsabilityScale(SUS),see
Appendix5andComputerSystemUsabilityQuestionnaire(CSUQ),seeAppendix6).After
theparticipantcompletedbothquestionnaires,theadministratorthankedeachparticipant
forhisorhertimeandallowedtheparticipanttomakeanycommentsonoraskany
questionsaboutthesystemandtaskspresented.Foreachsession,theparticipant’s
schedule,demographicinformation,tasksuccessrate,timeontask,errors,deviations,
verbalresponses,andpost-testquestionnairewererecorded.Thesystemwasthenresetto
propertestconditionsforthenextparticipant.
UsabilityMetrics
AccordingtotheNISTGuidetotheProcessesApproachforImprovingtheUsabilityof
ElectronicHealthRecords(NISTIR7741,November,2010)EHRsshouldsupportaprocess
thatprovidesahighlevelofusabilityforallusers.Thegoalisforuserstointeractwiththe
systemeffectively,efficiently,andwithanacceptablelevelofsatisfaction.Tothisend,
metricsforeffectiveness,efficiencyandusersatisfactionwerecapturedduringthe
usabilitytesting.Thegoalsofthetestweretoassess:
• EffectivenessofMetaCareEnterpriseEHRbymeasuringparticipantsuccessratesanderrors
• EfficiencyofMetaCareEnterpriseEHRbymeasuringtheaveragetasktimeandpath
deviations.
• SatisfactionwithMetaCareEnterpriseEHRbymeasuringeaseofuseratings.
DataScoringTable2detailshowtaskswerescored,errorsevaluated,andthetimedataanalyzed:Table2.ScoringProtocolsforEffectiveness,Efficiency,andSatisfaction
Measures RationaleandScoring
Effectiveness:• TaskSuccess
Ataskwascountedas“Success”iftheparticipantwasabletoachievethecorrectoutcome,withoutassistance,withinthetimeallottedonapertaskbasis.The total number of Sucesseswas calculated for each task and thendividedbythetotalnumberoftimesthattaskwasattempted.Resultsareprovidedasapercentage.
Effectiveness:• TaskFailures
Iftheparticipantabandonedthetask,didnotreachthecorrectansweror performed it incorrectly, or reached the end of the allotted timebeforesuccessfulcompletion, thetaskwascountedas “Fail.” Notasktimesweretakenforfailedattempts.Thetotalnumberoferrorswascalculatedforeachtaskanddividedbythe total number of times that task was attempted. Results arepresentedastheaverageerrorrate.Note:Notalldeviationsarecountedaserrors
Effectiveness:• Prompted
Successes
Becausesometasksaredependentuponthesuccessfulcompletionofprevious tasks, participants may receive a limited number of“prompts” to help prepare the systemdata for the pre-requisites forsubsequenttasks.Whenaparticipantwasabletocompletethedataentryonataskwith3orfewerprompts,thetaskwascountedasan“Assisted”competition.NotasktimeswererecordedforAssistedcompletions.
Efficiency:• Task
Deviations
The participant’s path (i.e., steps) through the application wasrecorded.Deviationsoccurifforexample,theparticipantnavigatedtoan incorrect screen, clicked on an incorrect menu item, followed anincorrectlink,orinteractedincorrectlywithanon-screencontrol.
Efficiency:• TaskTime
Each task was timed from the administrator’s prompt “Begin” untilsaid, “Done.” If the participant failed to say, “Done,” timing stoppedwhentheparticipantstoppedperformingthetask.Only task times for tasks that were successfully completed wereincludedintheaveragetasktimeanalysis.Averagetimepertaskwascalculatedforeachtask.
Satisfaction:• EaseofUse
ratings• System
Satisfaction
Participant’ssubjectiveimpressionoftheeaseofuseoftheapplicationwas measured by administering both a single post-task question aswellastwopost-sessionquestionnaires.Aftereachtask,theparticipantdeterminedona“scaleof1(being“NotSatisfied”) to 5 (being “Very Satisfied”) “ their subjective satisfactionwith performance on the task. These data are averaged acrossparticipants.Tomeasureparticipants’confidenceinandlikeabilityoftheMetaCareEnterpriseEHREHR overall, the testing team administered electronicversionsoftheSystemUsabilityScale(SUS)andtheComputerSystemUsability Questionnaire (CSUQ). See the SUS questionnaire asAppendix5.,andtheCSUQasAppendix6.
Results
DataAnalysisandReporting
TheresultsoftheusabilitytestoftheMetaCareEnterpriseEHRwereanalyzed
accordingtothemethodsdescribedintheUsabilityMetricssectionaboveandaredetailed
below.Notethattheresultsshouldbeevaluatedrelativethestudyobjectivesandgoalsas
outlinedintheDesignsectionabove.Thedatashouldyieldactionableresultsthat,if
corrected,yieldmaterial,positiveimpactonuserperformance.
EffectivenessandEfficiency
Table3presentsasummaryoftaskperformanceshowingtask,averagetimeon
task,expectedtasktimes,taskcompletionrates,pathdeviationsandtasksatisfaction:
Table3.UsabilityTestResults
TaskMeanTaskTime
ExpectedTaskTime
CompletionRate(%)
Mean#Path
DeviationsMeanTaskSatisfaction
Task1(SelectandActivateCDS) 3:59 3:00 33% 2.17 2.67Task2(ModifyDrug-DrugInteractionSeverity) 3:16 3:00 33% 1.67 2.67Task3(PlaceMed,Lab&RadiologyOrders) 5:57 5:00 33% 3.33 2.67Task4(ModifyPatientMedicationList) 1:54 3:00 67% 1.00 3.67Task5(ChangeMedicationOrders) 2:15 3:00 83% 1.17 3.83Task6(ModifyPatientAllergyList/ViewCDS) 0:00 1:30 0% 3.83 1.67Task7(UpdateProblemList/ViewCDS) 1:47 3:00 33% 1.60 2.20Task8(UpdateMedicationList/ViewCDS) 2:51 3:00 60% 2.00 2.80
Task9(UpdateVitalsInformation) 1:17 3:00 60% 2.00 2.75Task10(ReviewMedAdmin&DocAdmin) 0:00 4:00 0% 2.20 1.20Task11(ReconcileClinicalInformation) 3:45 5:00 60% 3.40 3.40Task12(SubmitElectronicPrescription) 3:41 4:00 40% 2.20 2.00
AsTable3shows,relativetooptimalperformancestandardsasdefinedbyMeta
HealthcareITSolutions,participantsintheMetaCareEnterpriseEHRusabilitytest
performancewasweakerthanwouldbeexpected.
Satisfaction
IndividualTaskSatisfactionParticipantsverballyindicatedtheirsatisfactionwitheachtaskusingascaleof“1”(“Not
Satisfied”)to“5,”(“VerySatisfied”).AsFigure1showsindividualtasksatisfactionranged
fromalowof1.2outof5onTask10(ReviewMedAdmin&DocAdmin)toahighof3.8on
Task5(ChangeMedicationOrders).
Figure1.SatisfactionRatingsofIndividualTasks
T a s k N u m b e r
SystemUsabilityScale
TheSystemUsabilityScale(SUS)isasimple,10-itemLikert-typeattitudescale
providingaglobalsubjectiveassessmentofusabilityfromtheuser’sperspective(John
BrookeatDigitalEquipmentCompanydevelopedtheSUSin1986).TheSUSscaleisscored
from0to100;scoresunder60representsystemswithlessthanoptimalusability,scores
over80areconsideredbetterthanaverage.SeeAppendix5foracopyoftheSUS.
ThemeantotalSUSscorefortheMetaCareEnterpriseEHRwas29andrangedfrom
alowof10andahighof48.Overall,participant-usersratedtheirsatisfactionwiththe
MetaCareEnterpriseEHRsystemtobelessthanoptimal.
ComputerSystemUsabilityQuestionnaire
UsingtheComputerSystemUsabilityQuestionnaire(CSUQ;Lewis,J.R.(1995).(See:
IBMComputerUsabilitySatisfactionQuestionnaires:PsychometricEvaluationand
InstructionsforUse.InternationalJournalofHuman-ComputerInteraction,7:1,57-78).),
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
1 2 3 4 5 6 7 8 9 10 11 12
2.67 2.67 2.67
3.67 3.83
1.67
2.20
2.80 2.75
1.20
3.40
2.00
MeanTaskSatisfaction
participantsratedeachof19itemsonascalefrom1to7,witharatingof7beingmostin
agreementwiththepositively-wordeditem.Responsesforeachitemweresummedand
averagedtofourscales–InterfaceQuality,InformationQuality,SystemUsefulness-andan
overallscale.SeeAppendix6foracopyoftheCSUQ.
Figure2displaysCUSQratingsforeachofthefourscales.Ingeneral,participantsin
theMetaCareEnterpriseEHRstudyratedsystemusabilitytobelessthanwhatwouldbe
expected,withaveragescoresforeachofthethreesubscalesandtheoverallscalefalling
belowthemidpointoneach7-pointscale.OnInterfaceQualitytheaveragescoreforthe
participantswas3.27;onInformationQualitytheaveragescore2.40;onSystemUsefulness
theaveragescorewas2.80;andtheoverallaverageCUSQscorewas2.71.
2.50 3.50 4.50 5.50 6.50
Overall Score
System Usefulness
Information Quality
Interface Quality
User Rating (Out of 7)
Figure 2: Computer System Usability Questionnaire
DiscussionofFindings
Severaloftheparticipantsstruggledwithtasksandwereunabletosuccessfully
completeanumberofthem.Severaloftheparticipantswerenotabletocompletesome
tasksgiventhetimeconstraintsofasummativetest.Someparticipants,however,were
mostlyabletoperformtaskssuccessfullywithsomeminorassistance.
IngeneraltheparticipantsfeltsomewhatdissatisfiedwiththeMetaCareEnterprise
EHRandtheparticipantperformancerateseemedaboutthesameastheparticipant
satisfactionratingswouldsuggest.Whenlookingtheratingsandperformanceacross
participants,itisclearthatbothtrainingandexperienceoftheuserisdirectlytiedtothe
satisfaction,effectivenessandefficiencyoftheMetaCareEnterpriseEHR.
Effectiveness
OftheTwelve(12)taskspresented,thehighesttaskcompletionrateforanytask
was83percent.Twotasks,Task6(ModifyPatientAllergyList/ViewCDS)andTask10
(ReviewMedAdmin&DocAdmin)werenotsuccessfullycompletedbyanyparticipant.
Overallparticipants,themeansuccessfultaskcompetitionratewaslessthatexpected(42
percent)indicatingthatmanyparticipantshaddifficultycompletingthetasks.
PriorexperiencewithEHRsystemswaspositivelyrelatedtosuccessfultask
performance;participantswithpriorEHRexperienceweremorelikelytosuccessfully
completetasksthanthosewithoutpriorexperience..
Efficiency
Participantswhosuccessfullycompletedtasksgenerallycompletedthosetasks
withinanacceptabletime.Sometaskswerecompletemorequicklythanthecalculated
optimaltime,whileseveraltaskstooklongerthanexpected.Thetasksthattookthe
longestrequiredtheparticipantstonavigatetoaparticularpage,interactwithacomplex
workflow,locateandselectspecificactions.Thosetaskscouldbeperformedmorequickly
withanupdatetotheinformationarchitecture,anincreaseintheamountofembedded
assistance,andperhapsanenhancedvisualindicationofprimaryorsecondaryactions.
Someparticipantsmadeanumberoferrorswhenattemptingtonavigatetoward
solvingtheirassignedtasks.Manyoftheseerrorsmaybeassociatedwiththose
participantsnotbeingfamiliarandnotunderstandingthepresentedinformation
architectureoftheMetaCareEnterpriseEHRsystem.Asnotedabove,priorexperiencewith
EHRsystemswasrelatedtosuccessfultaskcompletion.Similarly,experienceandpractice
withthegivensystemmayhavepositiveeffectswithregardtouserefficiency.
Satisfaction
ParticipantswereminimallysatisfiedwiththeMetaCareEnterpriseEHRsystem;
ratingsontheSUS(mean=29)andtheCSUQ(Overallscore=(2.71)demonstratedaless
thanexpectedlevelofsatisfactionwiththesystem.
OntheCSUQ,participantsrankedthescale“(InterfaceQuality)”highestofthethree
scales,suggestingthatthesystemwasperhapswelllikedvisually.Individualtask
satisfactionratingswererelatedtoindividualuserperformance.Thoseparticipantswho
wereabletosuccessfullycompletetaskswerealsomorelikelytorankthosetasksas
satisfying,whilethoseparticipantswhodidpoorlyorwerenotabletocompleteatask
rankedthosetasksasunsatisfying..Overallhowever,participantsatisfactionwith
MetaCareEnterpriseEHRwasaboutwhatwasexpectedgiventheperformancedata
SummaryofMajorFindings
ThisevaluationdemonstratedthattheMetaCareEnterpriseEHRsystemisafairly
usablesystemwitharelativelyshortlearningcurve.Participantshadneverusedthe
MetaCareEnterpriseEHRsystembeforethestudyandexperienceddifficulty
understandingthenavigationandinformationarchitecture.Theresultssuggestthatwith
minorchangestheirperformanceandsatisfactionwouldlikelyimprove.
RiskAnalysisThefollowingtablepresentsaprioritizedlistoftasksprioritizedbytheriskoferroras
observedduringthetesting.
Table5.RiskAnalysis
T A S K PercentComplete
RiskStatus
Task1(SelectandActivateCDS) 33% ModerateTask2(ModifyDrug-DrugInteractionSeverity) 33% ModerateTask3(PlaceMed,Lab&RadiologyOrders) 33% ModerateTask4(ModifyPatientMedicationList) 67% LowTask5(ChangeMedicationOrders) 83% NoneTask6(ModifyPatientAllergyList/ViewCDS) 0% HighTask7(UpdateProblemList/ViewCDS) 33% ModerateTask8(UpdateMedicationList/ViewCDS) 60% LowTask9(UpdateVitalsInformation) 60% LowTask10(ReviewMedAdmin&DocAdmin) 0% HighTask11(ReconcileClinicalInformation) 60% LowTask12(SubmitElectronicPrescription) 40% Moderate
AreasforImprovement
Thefollowingisalistofpotentialareasforimprovement.Makingtheseandother
minorenhancementswillimprovetheoveralluserexperienceoftheMetaCareEnterprise
EHRsystemandincreasetheeffectiveness,efficiency,andsatisfactionforbothexperienced
andnoviceMetaCareEnterpriseEHRusers.
• ExcessiverelianceontheLeft-sidenavigationstructure.
o Themajorityoftheinformationarchitectureispresentedasatreeview
intheleftsidenavigation.Betterorganizationandplacementof
additionalsub-navigationelementswithinthecontentareawillhelp
usersfindwhattheyarelookingfor.
• Indicationofexternallinks.
o Manyparticipantsclickedonseverallinksthatprovidedinformation
aboutspecificdrugs.Givingtheusersanindicationthatthisisan
“informationallink”andnotaplacetoperformanactionwouldgreatly
reducethenumberoferrorsmade.
• Inconsistentlocationofcontrols.
o Theplacementofthesameorsimilarcontrolsand/orbuttonsshouldbe
consistentacrosstheapplication.
• Primaryactionsneedtobemademoresalient.
o Participantsoftenhaddifficultlylocatinga“save”button.Providinga
consistent,salientbuttonwillprovideuserswithaeasierpathtotask
completion.
Appendices
Appendix1:RecruitingScreener1.Areyoumaleorfemale? 2.Haveyouparticipatedinafocusgrouporusabilitytestinthepast6months?3.Doyou,ordoesanyoneinyourhomeworkinmarketingresearch,usabilityresearch,and/orwebdesign?4.Doyou,ordoesanyoneinyourhome,haveacommercialorresearchinterestinanelectronichealthrecordsoftwareorconsultingcompany?5.Whichofthefollowingbestdescribesyourage?
_____23to39; _____40to59; _____60to74; _____75orolder.6.Doyourequireanyassistivetechnologiestouseacomputer?7.Pleaselistyourmedicalornursingcredentials8.Howlonghaveyouheldthisposition?(no.ofmonths):9.Whattypeoffacilitydoyouworkinandwhatisyourrolethere?10.Howaremedicalrecordshandledatyour(main)workplace?
_____AllPaper _____SomePaper/SomeElectronic ___AllElectronic11.HowmanyEHRsdoyouuseorhaveyouworkedwith?12.Howmanyyearshaveyouusedanelectronichealthrecord?13.Abouthowmanyhoursperweekdoyouspendusingacomputer?14.Whatcomputerplatform(s)doyouusuallyuse?15.Inthelastmonth,abouthowoftenhaveyouusedanelectronichealthrecord?
_____Didnotuselastmonth ___Everyday _____Afewtimesaweek.
Appendix2:AdditionalParticipantInformation
ParticipantNumber
Doyou,ordoesanyoneinyourhome,
workinmarketingresearch,usability
research,webdesignorElectronic
HealthRecordsoftware?
Haveyouparticipatedinafocusgroupor
usabilitytestinthepast3months?
Doyou,ordoesanyoneinyourhome,haveacommercialor
researchinterestinanelectronic
healthrecordsoftwareorconsultingcompany?
Abouthowmanyhoursperweekdoyouspendusinga
computer?
Whatcomputerplatform(s)doyou
usuallyuse?
Howaremedicalrecordshandledatyour
facility
Lastmonth,howoftenusedEHR?
p1No No No 20 PC AllElectronic
Everyday
p2No No No 30
PCandApple AllElectronic
Everyday
p3
No No No 50 PC
SomePaper/SomeElectronic
Everyday
p4
No No No 50 PC
SomePaper/SomeElectronic
Afewtimesperweek
p5
No No No 45PC,Apple,
Unix
SomePaper/SomeElectronic
Everyday
p6No No No 45 Other AllElectronic
Everyday
Appendix3:InformedConsentFormMetaHealthcare ITSolutionswould like to thankyou forparticipating in this study.Thepurposeof this study is toevaluateanelectronichealth records system. If youdecide toparticipate,youwillbeaskedtoperformseveral tasksusingtheprototypeandgiveyourfeedback.Thestudywilllastabout60minutes.AgreementIunderstandandagreethatasavoluntaryparticipant inthepresentstudyconductedbyMetaHealthcareITSolutions.Iamfreetowithdrawconsentordiscontinueparticipationatanytime.IunderstandandagreetoparticipateinthestudyconductedandrecordedbytheMetaHealthcareITSolutions.IunderstandandconsenttotheuseandreleaseofthevideorecordingbyMetaHealthcareITSolutions.Iunderstandthattheinformationandvideotapeisforresearchpurposesonlyand that my name and image will not be used for any purpose other than research. IrelinquishanyrightstothevideoandunderstandthevideorecordingmaybecopiedandusedbyMetaHealthcareITSolutionswithoutfurtherpermission.I understand and agree that the purpose of this study is tomake software applicationsmoreusefulandusableinthefuture.IunderstandandagreethatthedatacollectedfromthisstudymaybesharedwithoutsideofMetaHealthcareITSolutions.Iunderstandandagreethatdataconfidentialityisassured,becauseonlyde-identifieddata– i.e., identificationnumbersnotnames–willbeused inanalysisandreportingoftheresults.I agree to immediately raise any concerns or areas of discomfort with the studyadministrator.IunderstandthatIcanleaveatanytime.Pleasecheckoneofthefollowing:____YES,Ihavereadtheabovestatementandagreetobeaparticipant.____NO,Ichoosenottoparticipateinthisstudy.Signature:_____________________________________Date_____________________
Appendix4:ParticipantTasksTask1:SelectandActivateClinicalDecisionSupportAsasystemadministrator,you’vebeentaskedtoensurethatallrelevantClinicalDecisionSupportIntelliOrderGuidelinesareactiveintheapplication.ThelistofClinicalDecisionSupportIntelliOrderGuidelinesinquestionareasfollows:
• SuspectedApendicitis• AceInhibitorGuidelines• MedicationAllergySulfonamides• AdultCommunityAcquiredPneumonia• LowKandDigoxin• AsthmaTempMesalamine
UseMetacareEnterprisetoensurealloftheCDSguidelinesintheapplicationareactive.Task2:ModifyDrug-DrugInteractionSeverityLevelAsasystemadministrator,you’vebeentaskedtoensurethelevel2drug-druginteractionbetweenpotassiumsparingdiureticsandpotassiumpreparationsisactive.UseMetacareEnterprisetoensurethePOTASSIUMSPARINGDIURETICS/POTASSIUMPREPSdrug-druginteractionisactiveinthesystem.Task3:PlaceMedication,Laboratory,andRadiologyOrdersAfterdailyroundsforJaneSmith,youidentifythefollowingordersneedtobeplacedforthepatient:
• BasicMetabolicPanel(One-timeorder)• ChestX-Ray(One-timeorder)• Potassium10MEQOrallyTwiceDaily• Acetaminophenw/Codeine30/300mgOrallyEverySixHours
Duringorderentry,youreceiveadrug-druginteractionalertbetweenthePotassiumandthepatient’scurrentorderforThiazide.Additionally,youreceiveadrug-allergyalertbetweentheAcetaminophenw/Codeineandthepatient’scodeineallergy.YoudecidetoproceedwithorderentryofthePotassiumbutdecidenottoordertheAcetaminophenw/Codeine.Uponsave,youarepromptedtore-enteryourpasswordforpositiveidentification.UseMetacareEnterprisetoplacetheordersforJaneSmith.
Task4:ModifythePatient’sMedicationListAfterreviewingthedrug-druginteractionbetweenthenewlyplacedPotassiumorderandthecurrentTriamtereneorderonJaneSmith’sprofile,youdecidetodiscontinuetheTriamterene50mgDailyorderforJaneSmith.Upondiscontinuation,youarepromptedtore-enteryourpasswordforpositiveidentification.UseMetacareEnterprisetodiscontinuetheThiazideorderonJaneSmith’smedicationlist.Task5:ChangeMedicationOrdersAfterreviewingJaneSmith’spotassiumlabvalues,youidentifythatthePotassiumregimenneedstobeadjusted.YoureviewJaneSmith’sprofileandidentifythePotassium10MEQTwiceDailyordershouldbechangedtothefollowing:
• Potassium10MEQOrallyOnceDailyBeforesaving,youoverrideanyclinicalalerts.Uponsave,youarepromptedtore-enteryourpasswordforpositiveidentification.UseMetacareEnterprisetochangethePotassiumorderforJaneSmithTask6:ModifyPatient’sAllergyListandViewRelatedClinicalDecisionSupportAfterinterviewingJaneSmith,youareinformedthatsheisnolongerallergictoCodeine,butisactuallyallergictoSulfa.Youneedtoupdatethepatient’sallergylisttoreflectthenewinformation.AfterrecordingtheSulfaallergy,aclinicaldecisionsupportalerttriggerswarningofanexistingallergytoBactrimDS.UseMetacareEnterprisetoupdatethepatient’sallergylistwiththenewallergyinformationandreviewtheclinicaldecisionsupportinformationTask7:UpdatePatient’sProblemListandViewRelatedClinicalDecisionSupportJohnFrankisadmittedtothefacility.Adiagnosisofbacterialpneumoniaisidentified.AfterrecordingthepneumoniadiagnosisintheEHRapplication,thesystemrecognizesthepatientasanadultandtriggersaclinicaldecisionsupportalertforadultcommunityacquiredpneumonia.UseMetacareEnterprisetoupdatethepatient’sdiagnosisinformationwith‘bacterialpneumonia’andreviewtheclinicaldecisionsupportinformation.
Task8:UpdateMedicationListandViewRelatedClinicalDecisionSupportJohnFrankalsohasahistoryofcongestiveheartfailure.You’veidentifiedDigoxin0.25mgOnceDailyasahistoricalmedicationwhichmustbecontinuedwhileinthehospital.Youentertheorderonthepatient’smedicationlistandarealertedwithaclinicaldecisionsupportwarningofalowpotassiumlevelforthepatient.UseMetacareEnterprisetoupdatethepatient’smedicationlistwith‘Digoxin0.25mgOrallyOnceDaily’andreviewtheclinicaldecisionsupportinformation.Task9:UpdateVitalsInformationYouareanurseonapatientcarelocationandhavejusttakenthetemperatureofSamJohnson,apatientwithadiagnosisof‘chronicobstructiveasthmawithstatusasthmaticus.’Thepatientisrunningatemperatureof102degreesF.Afterrecordingthepatient’stemperature,aclinicaldecisionsupportalerttriggerssuggestingtheuseofmesalamineforasthmapatientsrunningatemperature.UseMetacareEnterprisetoupdatethepatient’stemperatureandreviewtheclinicaldecisionsupportinformation.Task10:Reviewthe‘5RightsofMedicationAdministration’andDocumentAdministrationofaMedicationAftertakinganddocumentingSamJohnson’stemperature,youreviewSamJohnson’smedicationrecordtoseeifhehasanordertouseincaseofafever.YouidentifyaTylenol500mgOralTabletscheduledasneededEvery4Hours.AfterpullingaTylenol500mgtabletfromtheautomateddispensingmachine,youreviewthepatient’selectronicmedicationadministrationrecordtoreviewthemedication5rightsinformationpriortoadministration.Youneedtoverifythefollowinginformationinthemedicationadministrationrecord:
• RightPatient• RightMedication• RightDose• RightRoute• RightTime
Afterverifyingthe‘5rights’information,youdetermineitissafetoadministertheTylenol500mgtothepatient.Afteradministration,youdocumenttheadministrationelectronicallyinthemedicationadministrationrecord.UsingMetacareEnterprise,verifythe‘5rights’informationanddocumenttheadministrationoftheTylenol500mgtabletforSamJohnson.
Task11:ReconcileaPatient’sClinicalInformationSummaryofcareinformationforLarryDonaldhasbeenreceivedfromCommunityHospitalanduploadedintotheMetacareEnterpriseEHR.Thefollowinginformationwasrecognized:Problems:
• CongestiveHeartFailure• DiabetesMellitus
Medications:• LantusInsulin15uSQatBedtime
MedicationAllergies:• Penicillin• Codeine
UseMetacareEnterprisetoviewandcomparetheuploadedmedicalrecordswiththeexistingpatientrecordstocreateasinglereconciledlistofmedications,medicationallergies,andproblems.Overrideanyclinicalalertswhichmayoccur.Task12:SubmitanElectronicPrescriptionSuzieBakerisbeingdischargedfromthefacilityonLipitor10mgOrallyOnceDailyfortheongoingtreatmentofherhighbloodpressure.Thepatientwasreceivingthemedicationwhileinthefacility,butneedsaprescriptiontransmittedtoCommunityPharmacyforpickupafterdischarge.UseMetacareEnterprisetocontinueSuzieBaker’sLipitor10mgathomeandsubmitaprescriptionelectronicallytoCommunityPharmacy.
Appendix5:SystemUsabilityScaleQuestionnaire
Appendix6:ComputerSystemUsabilityQuestionnairePleaseprovideyourimpressionoftheusabilityofthesystembyansweringeachofthequestionsbelow:1.Overall,IamsatisfiedwithhoweasyitistousethissystemStrongly 1234567 NA StronglyDisagree Agree2.ItwassimpletousethissystemStrongly 1234567 NA StronglyDisagree Agree3.IcaneffectivelycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree4.IamabletocompletemyworkquicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree5.IamabletoefficientlycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree6.IfeelcomfortableusingthissystemStrongly 1234567 NA StronglyDisagree Agree7.ItwaseasytolearntousethissystemStrongly 1234567 NA StronglyDisagree Agree8.IbelieveIbecameproductivequicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree9.ThesystemgiveserrormessagesthatclearlytellmehowtofixproblemsStrongly 1234567 NA StronglyDisagree Agree10.WheneverImakeamistakeusingthesystem,IrecovereasilyandquicklyStrongly 1234567 NA StronglyDisagree Agree
11.Theinformation(suchasonlinehelp,on-screenmessages,andotherdocumentation)providedwiththissystemisclearStrongly1 234567 NA StronglyDisagree Agree12.ItiseasytofindtheinformationIneededStrongly 1234567 NA StronglyDisagree Agree13.TheinformationprovidedforthesystemiseasytounderstandStrongly 1234567 NA StronglyDisagree Agree14.TheinformationiseffectiveinhelpingmecompletethetasksandscenariosStrongly 1234567 NA StronglyDisagree Agree15.TheorganizationofinformationonthesystemscreensisclearStrongly 1234567 NA StronglyDisagree Agree16.TheinterfaceofthissystemispleasantStrongly 1234567 NA StronglyDisagree Agree17.IlikeusingtheinterfaceofthissystemStrongly 1234567 NA StronglyDisagree Agree18.ThissystemhasallthefunctionsandcapabilitiesIexpectittohaveStrongly 1234567 NA StronglyDisagree Agree19.Overall,IamsatisfiedwiththissystemStrongly 1234567 NA StronglyDisagree Agree
Appendix7:Detailedtaskperformanceforeachparticipant.ParticipantNumber
Task1(SelectandActivateCDS)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 5:24 Success 1 2p2 2:35 Success 2 5p3 3:04 TimeOut 1 1p4 3:07 TimeOut 1 3p5 3:12 TimeOut 6 1p6 3:47 TimeOut 2 4
Expectedtime 3:00
AverageTimeonTask 3:59
AverageTaskSatisfaction 2.67
Average#PathDeviations 2.17
PercentSuccess 33%
ParticipantNumber
Task2(ModifyDrug-DrugInteractionSeverity)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 3:39 TimeOut 1 1p2 3:03 TimeOut 2 1p3 3:34 Success 1 2p4 3:18 TimeOut 3 3p5 2:59 Success 1 5p6 3:27 Assisted 2 4
ExpectedTime 3:00
AverageTimeonTask 3:16
AverageTaskSatisfaction 2.67
Average#PathDeviations 1.67
PercentSuccess 33%
ParticipantNumber
Task3(PlaceMed,Lab&RadiologyOrders)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 6:01 TimeOut 4 2p2 5:00 Success 4 3p3 6:54 Success 1 2p4 6:30 Assisted 3 2p5 5:50 Assisted 5 4p6 5:32 TimeOut 3 3
ExpectedTime 5:00
AverageTimeonTask 5:57
AverageTaskSatisfaction 2.67
Average#PathDeviations 3.33
PercentSuccess 33%
ParticipantNumber
Task4(ModifyPatientMedicationList)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 3:11 TimeOut 0 2p2 3:08 TimeOut 2 3p3 1:11 Success 0 4p4 2:07 Success 2 5p5 2:32 Success 2 5p6 1:48 Success 0 3
ExpectedTime 3:00
AverageTimeonTask 1:54
AverageTaskSatisfaction 3.67
Average#PathDeviations 1.00
PercentSuccess 67%
ParticipantNumber
Task5(ChangeMedicationOrders)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 3:19 Success 1 3p2 2:00 Success 0 4p3 2:38 Success 1 4p4 1:51 Success 1 4p5 3:04 TimeOut 3 3p6 1:27 Success 1 5
ExpectedTime 3:00
AverageTimeonTask 2:15
AverageTaskSatisfaction 3.83
Average#PathDeviations 1.17
PercentSuccess 83%
ParticipantNumber
Task6(ModifyPatientAllergyList/ViewCDS)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 4:27 TimeOut 2 1p2 3:15 TimeOut 6 4p3 3:02 TimeOut 6 1p4 3:07 TimeOut 0 2p5 3:07 Fail 6 1p6 3:03 TimeOut 3 1
ExpectedTime 1:30
AverageTimeonTask 0:00
AverageTaskSatisfaction 1.67
Average#PathDeviations 3.83
PercentSuccess 0%
ParticipantNumber
Task7(UpdateProblemList/ViewCDS)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 4:06 TimeOut 2 1p2 1:27 Success 0 5p3 2:07 Success 0 3
p4 SystemIssue
p5 3:09 TimeOut 4 1p6 3:02 TimeOut 2 1
ExpectedTime 3:00
AverageTimeonTask 1:47
AverageTaskSatisfaction 2.20
Average#PathDeviations 1.60
PercentSuccess 33%
ParticipantNumber
Task8(UpdateMedicationList/ViewCDS)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 4:02 TimeOut 0 1p2 3:10 Success 1 5p3 1:56 Success 0 4
p4 SystemIssue
p5 3:40 Fail 6 1p6 3:27 Success 3 3
ExpectedTime 3:00
AverageTimeonTask 2:51
AverageTaskSatisfaction 2.8
Average#PathDeviations 2.00
PercentSuccess 60%
ParticipantNumber
Task9(UpdateVitalsInformation)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 1:25 Success 0 3p2 3:05 TimeOut 5 1p3 3:16 TimeOut 5 1
p4 SystemIssue
p5 0:43 Success 0 5p6 1:43 Success 0 4
ExpectedTime 3:00
AverageTimeonTask 1:17
AverageTaskSatisfaction 2.8
Average#PathDeviations 2.00
PercentSuccess 60%
ParticipantNumber
Task10(ReviewMedAdmin&DocAdmin)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 4:03 TimeOut 2 1p2 4:07 TimeOut 2 2p3 4:21 TimeOut 1 1
p4 SystemIssue
p5 4:00 TimeOut 5 1p6 4:15 TimeOut 1 1
ExpectedTime 4:00
AverageTimeonTask 0:00
AverageTaskSatisfaction 1.2
Average#PathDeviations 2.20
PercentSuccess 0%
ParticipantNumber
Task11(ReconcileClinicalInformation)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 3:07 Fail 6 1p2 3:17 Success 2 5p3 4:15 Success 2 3
p4 SystemIssue
p5 3:54 Fail 6 3p6 3:43 Success 1 5
ExpectedTime 5:00
AverageTimeonTask 3:45
AverageTaskSatisfaction 3.4
Average#PathDeviations 3.40
PercentSuccess 60%
ParticipantNumber
Task12(SubmitElectronicPrescription)
TaskTime Outcome#Path
DeviationsTask
Satisfaction
p1 3:02 TimeOut 2 1p2 4:05 TimeOut 2 1p3 2:43 Success 0 3
p4 SystemIssue
p5 4:01 TimeOut 5 1p6 4:40 Success 2 4
ExpectedTime 4:00
AverageTimeonTask 3:41
AverageTaskSatisfaction 2.00
Average#PathDeviations 2.20
PercentSuccess 40%
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 11
Appendix B: Quality Management System
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 12
Appendix C: Privacy and Security
Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017
©2017 InfoGard. May be reproduced only in its original entirety, without revision 13
Test Results Summary Document History Version
V1.0 Initial release April 10, 2017
END OF DOCUMENT
Description of Change Date