ONC HIT Certification Program Test Results Summary for...

transcript

Test Results Summary for 2014 Edition EHR Certification16-2592-R-0017-PRI V1.0, April 10, 2017

Part 1: Product and Developer Information1.1 Certified Product Information

1.2 Developer/Vendor InformationMeta Healthcare IT Solutions

Part 2: ONC-Authorized Certification Body Information2.1 ONC-Authorized Certification Body Information

4/10/2017

Test Type: Modular

Developer/Vendor Name:Address: 401 Franklin Ave. Suite 106

Garden City, NY 11530

ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification

Product Name: MetaCare Enterprise EHRProduct Version: 6.4Domain: Inpatient

Developer/Vendor Contact: Matthew Adams

ONC-ACB Name: InfoGard Laboratories, Inc.Address: 709 Fiero Lane Suite 25

Website: www.metacaresolutions.comEmail: madams@metacaresolutions.comPhone: (800) 768-1920

ONC-ACB Contact: Adam Hardcastle

This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative:

Adam Hardcastle EHR Certification Body ManagerONC-ACB Authorized Representative Function/Title

San Luis Obispo, CA 93401Website: www.infogard.comEmail: ehr@infogard.comPhone: (805) 783-0810

Signature and Date

2.2 Gap Certification

(a)(1) (a)(19) (d)(6) (h)(1)(a)(6) (a)(20) (d)(8) (h)(2)(a)(7) (b)(5)* (d)(9) (h)(3)(a)(17) (d)(1) (f)(1)(a)(18) (d)(5) (f)(7)**

*Gap certification allowed for Inpatient setting only**Gap certification allowed for Ambulatory setting only

2.3 Inherited CertificationThe following identifies criterion or criteria certified via inherited certification

(a)(1) (a)(16) Inpt. only (c)(2) (f)(2) (a)(2) (a)(17) Inpt. only (c)(3) (f)(3)

The following identifies criterion or criteria certified via gap certification§170.314

No gap certification

§170.314

(d)(3) (a)(6) (b)(1) (d)(4)

(f)(6) Amb. only (a)(7) (b)(2) (d)(5)

(a)(3) (a)(18) (d)(1) (f)(4) Inpt. only

(a)(4) (a)(19) (d)(2) (f)(5) Amb. only

(a)(5) (a)(20)

(a)(10) (b)(5) (d)(8) (g)(2) (a)(11) (b)(6) Inpt. only (d)(9) Optional (g)(3)

(a)(8) (b)(3) (d)(6) (f)(7) Amb. Only

(a)(9) (b)(4) (d)(7) (g)(1)

(a)(14) (b)(9) (e)(3) Amb. only (h)(2) (a)(15) (c)(1) (f)(1) (h)(3)

(a)(12) (b)(7) (e)(1) (g)(4) (a)(13) (b)(8) (e)(2) Amb. only (h)(1)

No inherited certification

Test Results Summary for 2014 Edition EHR Certification

16-2592-R-0017-PRI Vl.O, April 10, 2017

Part 3: NVLAP-Accredited Testing Laboratory Information

Report Number: Test Date(s):

16-2592-R-0017 Vl.O

3.1 NVLAP-Accredited Testing Laboratory Information ATLName:

Accreditation Number:

Address:

Website:

Email:

Phone:

ATL Contact:

lnfoGard Laboratories, Inc.

NVLAP Lab Code 100432-0

709 Fiero Lane Suite 25

San Luis Obispo, CA 93401

www.infogard.com

ehr@infogard .com

(805) 783-0810

Milton Padilla

For more information on scope of accreditation, please reference

http://ts.nist.gov/Standards/scopes/1004320.htm

lNFOIGARD

Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory

Representative:

Mark Shin EHR Test Body Approved Signatory

ATL Authorized Representative Function/Title

~ 4/10/2017 Signature and Date

3.2 Test Information

3.2.1 Additional Software Relied Upon for Certification

Additional Software Applicable Criteria Functionality provided by

Additional Software

Secure Trans (b)1,(b)2,(e)1 Secure transmission

MetacareRx (Meta Info System) (a)16 eMAR

First Data Bank (a)lO Drug formulary checking

D No additional software required

3.2.2 Test Tools

Version

No test tools required

3.2.3 Test Data

3.2.4 Standards3.2.4.1 Multiple Standards Permitted

HL7 v2 Laboratory Restults Intervace (LRI) Validation ToolHL7 v2 Syndromic Surveillance Reporting Validation ToolTransport Testing ToolDirect Certificate Discovery Tool

Edge Testing Tool

Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter ]

Test ToolCypressePrescribing Validation ToolHL7 CDA Cancer Registry Reporting Validation ToolHL7 v2 Electronic Laboratory Reporting (ELR) Validation ToolHL7 v2 Immunization Information System (IIS) Reporting Valdiation Tool

(a)(13)

§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release

§170.207(j)HL7 Version 3 Standard: Clinical Genomics; Pedigree

No alteration (customization) to the test data was necessary

The following identifies the standard(s) that has been successfully tested where more than one standard is permitted

Criterion # Standard Successfully Tested

(a)(8)(ii)(A)(2)

§170.204(b)(1)HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2)HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

None of the criteria and corresponding standards listed above are applicable

3.2.4.2 Newer Versions of Standards

No newer version of a minimum standard was tested

(a)(15)(i)

§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2)HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

(b)(7)(i)

§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions

(b)(8)(i)

(a)(16)(ii) §170.210(g) Network Time Protocol Version 3 (RFC 1305)

§170. 210(g)Network Time Protocol Version 4 (RFC 5905)

(b)(2)(i)(A)

(e)(3)(ii) Annex A of the FIPS Publication 140-2

Common MU Data Set (15)

§170.207(b)(2)The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4)

(e)(1)(i) Annex A of the FIPS Publication 140-2

(e)(1)(ii)(A)(2) §170.210(g) Network Time Protocol Version 3 (RFC 1305)

§170. 210(g)Network Time Protocol Version 4 (RFC 5905)

The following identifies the newer version of a minimum standard(s) that has been successfully tested

Newer Version Applicable Criteria

3.2.5 Optional Functionality

No optional functionality tested

(a)(4)(iii) Plot and display growth charts

(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)

Criterion # Optional Functionality Successfully Tested

(f)(3)

Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)

(f)(7) Ambulatory setting only – transmission to public health agencies – syndromic surveillance - Create Data Elements

Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)

(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)

(e)(1) View, download and transmit data to a third party using the standard specified at §170.202(d) (Edge Protocol IG version 1.1)

Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)

3.2.6 2014 Edition Certification Criteria* Successfully Tested

TP** TD*** TP** TD***

(a)(1) (c)(3) (a)(2) (d)(1) (a)(3) (d)(2)

Criteria #Version

(a)(7) (d)(6) (a)(8) (d)(7) (a)(9) (d)(8)

(a)(4) (d)(3) (a)(5) (d)(4) (a)(6) (d)(5)

(a)(13) (e)(3) Amb. only

(a)(14) (f)(1) (a)(15) (f)(2)

(a)(10) (d)(9) Optional

(a)(11) (e)(1) (a)(12) (e)(2) Amb. only

(a)(19) (a)(20) (f)(6) Optional &

Amb. only (b)(1)

(a)(16) Inpt. only (f)(3) (a)(17) Inpt. only (f)(4) Inpt. only

(a)(18) (f)(5) Optional & Amb. only

(b)(5) (g)(3) (b)(6) Inpt. only (g)(4) (b)(7) (h)(1)

(b)(2) (f)(7) Amb. only

(b)(3) (g)(1) (b)(4) (g)(2)

*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)**Indicates the version number for the Test Procedure (TP)***Indicates the version number for the Test Data (TD)

(b)(8) (h)(2) (b)(9) (h)(3) (c)(1) (c)(2)

3.2.7 2014 Clinical Quality Measures*Type of Clinical Quality Measures Successfully Tested:

CMS ID Version CMS ID Version CMS ID Version CMS ID Version2 90 136 155

22 117 137 15650 122 138 15752 123 139 15856 124 140 15961 125 141 16062 126 142 16164 127 143 16365 128 144 16466 129 145 16568 130 146 16669 131 147 16774 132 148 16975 133 149 17777 134 153 17982 135 154 182

CMS ID Version CMS ID Version CMS ID Version CMS ID Version9 71 107 172

26 72 108 17830 73 109 18531 91 110 18832 100 111 19053 102 11355 104 11460 105 171

Ambulatory Inpatient No CQMs tested*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov (navigation: 2014 Clinical Quality Measures)

Ambulatory CQMs

Inpatient CQMs

3.2.8 Automated Numerator Recording and Measure Calculation3.2.8.1 Automated Numerator Recording

Automated Numerator Recording was not tested

3.2.8.2 Automated Measure Calculation

Automated Measure Calculation was not tested

3.2.9 Attestation

(a)(3) (a)(12) (a)(19) (b)(8)(a)(4) (a)(13) (a)(20) (b)(9)

Automated Numerator Recording Successfully Tested(a)(1) (a)(11) (a)(18) (b)(6)

(a)(7) (a)(16) (b)(4) (e)(3)(a)(9) (a)(17) (b)(5)

(a)(5) (a)(14) (b)(2) (e)(1)(a)(6) (a)(15) (b)(3) (e)(2)

(a)(4) (a)(13) (a)(20) (b)(9)(a)(5) (a)(14) (b)(2) (e)(1)

Automated Numerator Recording Successfully Tested(a)(1) (a)(11) (a)(18) (b)(6)(a)(3) (a)(12) (a)(19) (b)(8)

Attestation Forms (as applicable) Appendix

(a)(6) (a)(15) (b)(3) (e)(2)(a)(7) (a)(16) (b)(4) (e)(3)

Safety-Enhanced Design* A Quality Management System** B Privacy and Security C

(a)(9) (a)(17) (b)(5)

*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9)**Required for every EHR product

Appendix A: Safety Enhanced Design

User-Centered Design Process

Meta Healthcare IT Solutions has employed the User Centered Design approach outlined below

to enhance the usability of our inpatient EHR application, Metacare Enterprise, and to meet the

Safety-Enhanced Design 170.314(g)(3) requirement for Meaningful Use Stage 2 EHR

Certification. This outline is derived from the NIST Guide to the Processes Approach for

Improving the Usability of Electronic Health Records (NIST

7741: http://www.nist.gov/customcf/get_pdf.cfm?pub_id=907313) and has be modified to fit in to

Meta Healthcare IT Solution’s quality management system design processes.

1. Documentation of User Needs and Requirements

Prior to development of new modules within the EHR product, Meta conducts a requirements

analysis which includes any client needs, regulatory requirements, or EHR certification

requirements. This requirements analysis includes interviews with client end-users to ensure

user needs are incorporated into the requirements if they did not originate from users

themselves. The requirements are then documented, along with suggestions on how the EHR

application should fulfill the requirement, using appropriate methodologies (user stories,

diagrams, etc.)

2. Workflow Development

Once requirements analysis has been conducted, the requirements are given to the

development team to review. Using the requirements analysis, the development team conducts

a workflow analysis as a basis for the user interface design. This analysis includes consultation

with Meta’s clinical analysts and client managers as well as consultation with client end-users

themselves. Baseline performance criteria are also documented for the requirements. The

results of the analysis are documented and then used during the development of the database

design and information architecture.

3. Development of Low-Fidelity Prototypes

After workflow development and information architecture design has been completed, the

development team creates a prototype for review by Meta’s clinical team prior to development.

The prototypes can be in the form of wireframes, screenshots of early development, or even

demonstrations of a basic working prototype application module, given to or conducted in front

of client end-users for their review. The development team uses feedback from this end-user

review to develop higher-level prototypes.

4. Development of High-Fidelity Prototypes

A full, executable prototype is then developed using the end-user feedback from earlier

prototypes. This prototype is subjected to unit testing by Meta’s software quality assurance

team. Any bugs in the software are reported to the developers. Once the prototype as passed

quality assurance testing, the module subjected to full system and regression testing as part of

the EHR. The application is then tested by client end-users who conduct testing of the

application and send feedback to the development test for further iterations of application

development.

5. Formal Usability Testing

Once the module is incorporated into the EHR, Meta conducts formal lab-based summative

usability testing on the EHR software module in question to obtain measures of users’

effectiveness, efficiency, and satisfaction using the application, the results of which are used for

further iterations of application development.

EHRUsabilityTestReportofMetaHealthcareITSolutions ProductVersion:MetaCareEnterpriseEHRv6.3.2ReportbasedonISO/IEC25062:2006CommonIndustryFormatforUsabilityTestReports

DateofUsabilityTest:May202014DateofReport:May23,2014

ReportPreparedBy:

TheUsabilityPeople,LLC

4000LegatoRoad,Suite1100Fairfax,VA22033

TABLEOFCONTENTSExecutiveSummary.................................................................................................................................................3Introduction................................................................................................................................................................4Method...........................................................................................................................................................................5Participants.............................................................................................................................................................5StudyDesign...........................................................................................................................................................7Tasks..........................................................................................................................................................................8TestLocation..........................................................................................................................................................8TestEnvironment................................................................................................................................................9TestFormsandTools......................................................................................................................................10ParticipantInstructions.................................................................................................................................10Procedure.............................................................................................................................................................11UsabilityMetrics................................................................................................................................................12DataScoring....................................................................................................................................................13

Results........................................................................................................................................................................15DataAnalysisandReporting........................................................................................................................15EffectivenessandEfficiency....................................................................................................................15Satisfaction......................................................................................................................................................16

DiscussionofFindings....................................................................................................................................19Effectiveness...................................................................................................................................................19Efficiency..........................................................................................................................................................19Satisfaction......................................................................................................................................................20SummaryofMajorFindings....................................................................................................................21RiskAnalysis..................................................................................................................................................22AreasforImprovement.............................................................................................................................23

Appendices...............................................................................................................................................................24Appendix1:RecruitingScreener...............................................................................................................24Appendix2:AdditionalParticipantInformation................................................................................25Appendix3:InformedConsentForm.......................................................................................................26Appendix4:ParticipantTasks.....................................................................................................................27Appendix5:SystemUsabilityScaleQuestionnaire............................................................................31Appendix6:ComputerSystemUsabilityQuestionnaire..................................................................32Appendix7:Detailedtaskperformanceforeachparticipant........................................................34

ExecutiveSummary

On May 20, 2014 The Usability People, LLC conducted a usability test of Meta

Healthcare IT Solutions’ MetaCare Enterprise EHR. The test was conducted in the Fairfax, VA

office of The Usability People over remote tele-conferencing sessions using Go To Meeting.The

purposewastotestandvalidatetheusabilityofthecurrentuserinterfaceandprovide

evidenceofusabilityofMetaCareEnterpriseEHRastheEHRUnderTest(EHRUT).Six(6)

healthcareprovidersmatchingthetargetdemographiccriteriaparticipatedintheusability

testusingtheEHRUTinsimulated,butrepresentativetasks.

Thestudyfocusedonmeasuringtheeffectivenessof,efficiencyof,andsatisfaction

withMetaCareEnterpriseEHRamongasampleofparticipantsrepresentingpotentialusers

ofthesystem.Performancedatawascollectedontwelve(12)taskstypicallyconductedon

anEHR.Taskscreatedwerebaseduponthecriteriaspecifiedwithinthetestprocedure

structureforevaluatingconformanceofElectronicHealthRecord(EHR)technologytothe

certificationcriteriadefinedin45CFRPart170SubpartCoftheHealthInformation

Technology:Standards,ImplementationSpecifications,andCertificationCriteriafor

ElectronicHealthRecordTechnology,2014Edition;RevisionstothePermanent

CertificationProgramforHealthInformationTechnology,FinalRuleaspublishedinthe

FederalRegisteronSeptember4,2012.

ResultsofthestudyindicatedthattheMetaCareEnterpriseEHRsystemwas

satisfactorywithregardstoeffectivenessandefficiencyandparticipantswereminimally

satisfiedwiththesystem.

Introduction

TheElectronicHealthRecordSystemUnderTest(EHRUT)testedforthisstudy,

MetaCareEnterpriseEHR,wasspecificallydesignedtopresentmedicalinformationto

physicians,nursesandotherhealthcarepractitionersondesktopcomputersinstandard

medicalcaresettings.Thisstudytestedandvalidatedtheusabilityofthecurrentuser

interfaceandprovidesevidenceoftheusabilityofMetaCareEnterpriseEHRwith

representativeexercisesandinrealisticuserconditions.Tothisend,measuresof

effectivenessandefficiency,suchastimeontask,numberoferrorsmade,andcompletion

rateswerecapturedduringusabilitytesting.Satisfactionwasassessedandusercomments

collectedusingtwoindustry-standardquestionnaires.

Method

Participants

Sixindividuals(3menand3women)participatedintheEHRUT(MetaCare

EnterpriseEHR).Participantswererecruitedfromadatabaseofpotentialparticipants

maintainedbyTheUsabilityPeople,LLC.Thecontactscontainedwithinthisdatabasewere

generatedviapotentialparticipantsresponsestopostingsinpopularInternetandsocial

mediasites,andalinkatthebottomofTheUsabilityPeoplewebsite.Thosewho

respondedtotheinvitationtotakepartinthestudyweredirectedtoanonline

questionnairethatservedastheparticipantscreener.(Thescreeningquestionnaireis

providedasAppendix1.)Participantsmeetingthecriteriaforparticipationinthestudy

werecontactedandscheduledviaemail.

ParticipantsintheusabilitytestofMetaCareEnterpriseEHRhadavarietyof

healthcarebackgroundsanddemographiccharacteristics.Participantsdidnothaveany

directexperienceortrainingusingtheMetaCareEnterpriseEHREHRsystem.

Table1presentsparticipantcharacteristics,includingdemographics,professional

experience,computingexperience,andnumberofpreviousEHR'sused.Noneofthe

participantswerefromthevendororganization(MetaHealthcareITSolutions)that

producedandsuppliedtheevaluatedsystemnordidanyparticipanthaveanydirect

connectiontothetestingorganization(TheUsabilityPeople,LLC).Ascompensationfor

theirparticipationallindividualsreceivedagiftcard.(Additionalparticipantbackground

characteristicsarepresentedinAppendix2.)

Table1.ParticipantCharacteristics

PartID Gender Age Education Role/Title Workfacility

ProfessionalExperience

#EHRsworkedwith

EHRExperience

Hrs/WkwithEHRs?

AssistiveTechNeeds

Female 40to59 RTTRadiationTherapist

Freestandingclinic.50%RTT/50%Manager 18 2 4 20 None

Male 23to39 BSHA,R.T.(R)(T)

AdministrativeDirectorofOncologyServices

University(Academic)setting,

overseeday-to-day

operatingproceduresofanOncologyServiceLine. 12 2 3 30

Male 23to39 MD Nephrologist

Publicuniversityhospital.

Manageandtreatpatientswithrenaldisease. 7 3 7 50 None

Female 23to39 MSN,CNL,RN

ClinicalInformaticsOfficer

WorkflowoptimizationwithinEHRs

andregulatory/ARRA/Meaningfu

lUseguidance. 16 6 16 50 None

Male 40to59

CompletedmultipleEHR

coursesEHRSystemsDirector

DirectorofEHRatmajor

hospitalnetworkinChicago 13 6 12 45 None

Female 40to59BSN,OncologyCertifiedNurse

OncologyNurse

Manager

UniversityofTexasHealth

ScienceCenteratTyler

outpatientclinic 23 2 8 45 None

StudyDesign

TheoverallobjectiveofthisusabilitytestwastouncoverareaswheretheMetaCare

EnterpriseEHRapplicationperformedwell–thatis,effectively,efficiently,andwith

satisfaction–andareaswheretheapplicationfailedtoservetheneedsofusers.Datafrom

thistestmaybeusedasabaselineforfuturetestsofupdatedversionsofMetaCare

EnterpriseEHRand/orforcomparingMetaCareEnterpriseEHRwithotherEHRspresenting

thesametasks.Inshort,thistestingservesasbothameanstorecordorbenchmark

currentusabilityandtoidentifyareaswhereimprovementsmustbemade.

ParticipantshadawiderangeofexperiencewithEHRsandgeneral,andnonehad

anydirectexperienceand/ortrainingwiththeMetaCareEnterpriseEHRsystem.

ParticipantscompletedthetestofMetaCareEnterpriseEHRusabilityduringindividual60-

minuteGoToMeetingsessions.Duringthetest,eachparticipantinteractedwithvarious

componentsoftheMetaCareEnterpriseEHR.Eachparticipantwasprovidedwiththesame

instructions.

MetaCareEnterpriseEHRwasevaluatedforeffectiveness,efficiencyandsatisfaction

asdefinedbythefollowingmeasurescollectedandanalyzedforeachparticipant:

• Numberoftaskssuccessfullycompletedwithoutassistance

• Timetocompletethetasks

• Numberanddescriptionoferrors

• Pathdeviations

• Participant’sverbalizations(comments)

• Participant’ssatisfactionratingsofthesystem

ArepresentativeofMetaHealthcareITsolutionscreatedatotaloftwelve(12)tasks

toberealisticandrepresentativeoftheactivitiesausermightengagewithMetaCare

EnterpriseEHRinactualmedicalsettings.Thetwelve(12)taskswerecreatedbasedupon

thecriteriaspecifiedwithinthetestprocedurestructureforevaluatingconformanceof

ElectronicHealthRecord(EHR)technologytothecertificationcriteriaasdefinedin45CFR

Part170SubpartCoftheHealthInformationTechnology:Standards,Implementation

Specifications,andCertificationCriteriaforElectronicHealthRecordTechnology.

Thetasksfocusedonthefollowingissues:

1. ComputerizedProviderOrderEntrySystem(CPOE)(§170.314(a)(1));

2. Drug-drug,drug-allergyinteractionchecks(§170.314(a)(2));

3. Medicationlist(§170.314(a)(6));

4. Medicationallergylist(§170.314(a)(7));

5. Clinicaldecisionsupport(§170.314(a)(8));

6. Electronicmedicationadministrationrecord(§170.314(a)(16));

7. Electronicprescribing(§170.314(b)(3));

8. Clinicalinformationreconciliation(§170.314(b)(4)).

AcopyofthetaskspresentedtoparticipantsintheusabilitytestofMetaCareEnterprise

EHRcanbefoundinAppendix4.

TestLocation

AllparticipantsweretestedontheMetaCareEnterpriseEHRsystemduringremote

conferencingsessionsusingGoToMeeting.Eachparticipantwasrequestedinadvanceto

secureaquietroomwithminimaldistractionsandacomputerthatcouldconnecttothe

InternetwithaGoToMeetingsession.Thetypeofcomputer,operatingsystemanddisplay

resolutionoftheparticipantsystemwasunknown,butthesystemthatwasusedbythetest

administratorandtheremoteparticipantwasanHPAll-in-oneDesktoprunningthe

Windows8operatingsystemataresolutionof1280x768.DuringagivenGoToMeeting

session,onlythetestadministratorandthatparticipantcommunicatedwithoneanother.

TheGoToMeetingusabilitytestsessionwasconductedbyatestadministratorfrom

thetestingorganization(TheUsabilityPeople,LLC)workingfromasmallconferenceroom

atTheUsabilityPeople’sFairfax,VAlocation.Seatedneartheadministrator,adatalogger

fromthetestingorganizationtookdetailednotesoneachsession,includingusercomments

andsatisfactionratingsfollowingeachtask.ArepresentativefromMetaHealthcareIT

Solutionsloggedintotheremotesessiontoobservethesessionsandprovidetechnical

assistancerunningMetaCareEnterpriseEHR.Duringasessionthetestadministrator,the

datalogger,andtheMetaHealthcareSolutionsrepresentativecouldseeonlythe

participant’sscreenandheartheparticipant’scomments,questions,andresponses.

TestEnvironment

WhiletheEHRUTtypicallywouldbeusedinahealthcareofficeorambulatory

surgerycenterfacility,testingoftheMetaCareEnterpriseEHRsystemwasconductedvia

remoteconnectionduringindividualGo To Meetingsessions.Eachparticipantcalledintoa

GoToMeetingsessionandwasconnectedbythetestadministratortotheapplication.

TheMetaCareEnterpriseEHRapplicationitselfranonabrowserplatformonaLAN

connectionusingasampledatabasesetupspecificallyforthetest.Participantsuseda

mouseandkeyboardwheninteractingwiththeEHRUTandweregivenremotecontrolof

theadministrator’sworkstationtoperformthetasks.

TestFormsandTools

Aspartoftheusabilitytest,severaldocumentsandinstrumentswereused.Examples

ofthedocumentsusedduringtheusabilitytest,includinganinformedconsentform,the

tasks,andpost-testquestionnaires,canbefoundinAppendices3-6,respectively.

Participants’interactionwiththeMetaCareEnterpriseEHRwascapturedandrecorded

digitallyusingMoraescreencapturesoftwarerunningonthetestadministrator’s

workstation.Verbalresponseswererecordedthrougheitherthemicrophoneintegrated

intotheparticipant’scomputerorthroughatelephoneconnection.Thisinformationwas

electronicallytransmittedtotheadministratorandtothedataloggerduringeachtest

session.

ParticipantInstructions

Theadministratorreadthefollowinginstructionsaloudtoeachparticipant:

Thankyouforparticipatinginthisstudy.Yourinputisveryimportant.Oursessiontodaywilllastabout60minutes.Duringthattimeyouwilluseaninstanceofanelectronichealthrecord.Iwillaskyoutocompleteafewtasksusingthissystemandanswersomequestions.

Pleasenotethatwearenottestingyou;wearetestingthesystem.Thereforeifyouhaveanydifficultythismaymeanthatsomethingneedstobeimprovedinthesystem.Iwillbehereincaseyouneedspecifichelp,butIamnotabletoinstructyouorprovidehelpinhowtousetheapplication.

Overall,weareinterestedinhoweasy(orhowdifficult)thissystemistouse,whatinitwouldbeusefultoyou,andhowwecouldimproveit.Ididnothaveanyinvolvementinitscreation,sopleasebehonestwithyouropinions.Alloftheinformationthatyouprovidewillbekeptconfidentialandyournamewillnotbeassociatedwithyourcommentsatanytime.Shouldyoufeelitnecessaryyouareabletowithdrawatanytimeduringthetesting.

Participantswerethengiventwelve(12)taskstocomplete.

Procedure

Uponconnectiontotheonlinemeetingtool(GoToMeeting),eachparticipantwas

greeted,hisorheridentityverified,andmatchedtoanameontheparticipantschedule.

ParticipantnameswerereplacedwithparticipantIDssothatagivenindividual’sdata

cannotbelinkedtohis/heridentity.Priortobeginningtesting,eachparticipantreviewed

andsignedaninformedconsentform(SeeAppendix3)andemailedittotheorganization

(The Usability People, LLC) conductingthetest.

TwostaffmembersoftheUsabilityPeople,ausabilitytestadministratorandadata

logger,administeredthetest.Theadministratormoderatedthesessionbyprovidingboth

verbalandwritteninstructionsfortheoverallusabilitytestandforeachofthetasks

comprisingthetest.Theadministratoralsomonitoredtasksuccess,pathdeviations,

numberanddescriptionoferrors,andaudio-recordedparticipantverbalcomments.The

dataloggertasktimes,obtainedpost-taskratingdata,andtooknotesonparticipant

commentsandadministratorfeedback.

Foreachofthetwelve(12)tasks,participantswerepresentedwritteninstructions

totheircomputers.Followingtheadministrator’sinstructions,eachparticipantperformed

eachtaskbyfirstreadingthetaskoutloudthenstatinginhisorherownwordshisorher

interpretationofthetaskrequirements.Whentheparticipant’sinterpretationmatchedthe

actualgoalofthetask,theadministratorinstructedtheparticipanttobeginandtasktiming

began.Tasktimewasstoppedandrecordedwhenthetestadministratorobservedonhis

workstationthattheparticipanthadsuccessfullycompletedthetask.Ifaparticipantfailed

tocompleteataskbeforetheexpectedamountoftimeforeachtask,thattaskwasmarked

as“TimedOut.”Aftereachtask,thetestadministratoraskedtheparticipant,“Onascale

from1to5,where1is‘NotSatisfied’and5is‘Satisfied,’howsatisfiedwereyouwiththis

task?”Thissameprocedurewasconductedforeachofthetwelve(12)tasks.

Followingcompletionofthetwelve(12)EHRtasks,theadministratorelectronically

presentedtotheparticipanttwopost-testquestionnaires(SystemUsabilityScale(SUS),see

Appendix5andComputerSystemUsabilityQuestionnaire(CSUQ),seeAppendix6).After

theparticipantcompletedbothquestionnaires,theadministratorthankedeachparticipant

forhisorhertimeandallowedtheparticipanttomakeanycommentsonoraskany

questionsaboutthesystemandtaskspresented.Foreachsession,theparticipant’s

schedule,demographicinformation,tasksuccessrate,timeontask,errors,deviations,

verbalresponses,andpost-testquestionnairewererecorded.Thesystemwasthenresetto

propertestconditionsforthenextparticipant.

UsabilityMetrics

AccordingtotheNISTGuidetotheProcessesApproachforImprovingtheUsabilityof

ElectronicHealthRecords(NISTIR7741,November,2010)EHRsshouldsupportaprocess

thatprovidesahighlevelofusabilityforallusers.Thegoalisforuserstointeractwiththe

systemeffectively,efficiently,andwithanacceptablelevelofsatisfaction.Tothisend,

metricsforeffectiveness,efficiencyandusersatisfactionwerecapturedduringthe

usabilitytesting.Thegoalsofthetestweretoassess:

• EffectivenessofMetaCareEnterpriseEHRbymeasuringparticipantsuccessratesanderrors

• EfficiencyofMetaCareEnterpriseEHRbymeasuringtheaveragetasktimeandpath

deviations.

• SatisfactionwithMetaCareEnterpriseEHRbymeasuringeaseofuseratings.

DataScoringTable2detailshowtaskswerescored,errorsevaluated,andthetimedataanalyzed:Table2.ScoringProtocolsforEffectiveness,Efficiency,andSatisfaction

Measures RationaleandScoring

Effectiveness:• TaskSuccess

Ataskwascountedas“Success”iftheparticipantwasabletoachievethecorrectoutcome,withoutassistance,withinthetimeallottedonapertaskbasis.The total number of Sucesseswas calculated for each task and thendividedbythetotalnumberoftimesthattaskwasattempted.Resultsareprovidedasapercentage.

Effectiveness:• TaskFailures

Iftheparticipantabandonedthetask,didnotreachthecorrectansweror performed it incorrectly, or reached the end of the allotted timebeforesuccessfulcompletion, thetaskwascountedas “Fail.” Notasktimesweretakenforfailedattempts.Thetotalnumberoferrorswascalculatedforeachtaskanddividedbythe total number of times that task was attempted. Results arepresentedastheaverageerrorrate.Note:Notalldeviationsarecountedaserrors

Effectiveness:• Prompted

Successes

Becausesometasksaredependentuponthesuccessfulcompletionofprevious tasks, participants may receive a limited number of“prompts” to help prepare the systemdata for the pre-requisites forsubsequenttasks.Whenaparticipantwasabletocompletethedataentryonataskwith3orfewerprompts,thetaskwascountedasan“Assisted”competition.NotasktimeswererecordedforAssistedcompletions.

Efficiency:• Task

Deviations

The participant’s path (i.e., steps) through the application wasrecorded.Deviationsoccurifforexample,theparticipantnavigatedtoan incorrect screen, clicked on an incorrect menu item, followed anincorrectlink,orinteractedincorrectlywithanon-screencontrol.

Efficiency:• TaskTime

Each task was timed from the administrator’s prompt “Begin” untilsaid, “Done.” If the participant failed to say, “Done,” timing stoppedwhentheparticipantstoppedperformingthetask.Only task times for tasks that were successfully completed wereincludedintheaveragetasktimeanalysis.Averagetimepertaskwascalculatedforeachtask.

Satisfaction:• EaseofUse

ratings• System

Satisfaction

Participant’ssubjectiveimpressionoftheeaseofuseoftheapplicationwas measured by administering both a single post-task question aswellastwopost-sessionquestionnaires.Aftereachtask,theparticipantdeterminedona“scaleof1(being“NotSatisfied”) to 5 (being “Very Satisfied”) “ their subjective satisfactionwith performance on the task. These data are averaged acrossparticipants.Tomeasureparticipants’confidenceinandlikeabilityoftheMetaCareEnterpriseEHREHR overall, the testing team administered electronicversionsoftheSystemUsabilityScale(SUS)andtheComputerSystemUsability Questionnaire (CSUQ). See the SUS questionnaire asAppendix5.,andtheCSUQasAppendix6.

Results

DataAnalysisandReporting

TheresultsoftheusabilitytestoftheMetaCareEnterpriseEHRwereanalyzed

accordingtothemethodsdescribedintheUsabilityMetricssectionaboveandaredetailed

below.Notethattheresultsshouldbeevaluatedrelativethestudyobjectivesandgoalsas

outlinedintheDesignsectionabove.Thedatashouldyieldactionableresultsthat,if

corrected,yieldmaterial,positiveimpactonuserperformance.

EffectivenessandEfficiency

Table3presentsasummaryoftaskperformanceshowingtask,averagetimeon

task,expectedtasktimes,taskcompletionrates,pathdeviationsandtasksatisfaction:

Table3.UsabilityTestResults

TaskMeanTaskTime

ExpectedTaskTime

CompletionRate(%)

Mean#Path

DeviationsMeanTaskSatisfaction

Task1(SelectandActivateCDS) 3:59 3:00 33% 2.17 2.67Task2(ModifyDrug-DrugInteractionSeverity) 3:16 3:00 33% 1.67 2.67Task3(PlaceMed,Lab&RadiologyOrders) 5:57 5:00 33% 3.33 2.67Task4(ModifyPatientMedicationList) 1:54 3:00 67% 1.00 3.67Task5(ChangeMedicationOrders) 2:15 3:00 83% 1.17 3.83Task6(ModifyPatientAllergyList/ViewCDS) 0:00 1:30 0% 3.83 1.67Task7(UpdateProblemList/ViewCDS) 1:47 3:00 33% 1.60 2.20Task8(UpdateMedicationList/ViewCDS) 2:51 3:00 60% 2.00 2.80

Task9(UpdateVitalsInformation) 1:17 3:00 60% 2.00 2.75Task10(ReviewMedAdmin&DocAdmin) 0:00 4:00 0% 2.20 1.20Task11(ReconcileClinicalInformation) 3:45 5:00 60% 3.40 3.40Task12(SubmitElectronicPrescription) 3:41 4:00 40% 2.20 2.00

AsTable3shows,relativetooptimalperformancestandardsasdefinedbyMeta

HealthcareITSolutions,participantsintheMetaCareEnterpriseEHRusabilitytest

performancewasweakerthanwouldbeexpected.

Satisfaction

IndividualTaskSatisfactionParticipantsverballyindicatedtheirsatisfactionwitheachtaskusingascaleof“1”(“Not

Satisfied”)to“5,”(“VerySatisfied”).AsFigure1showsindividualtasksatisfactionranged

fromalowof1.2outof5onTask10(ReviewMedAdmin&DocAdmin)toahighof3.8on

Task5(ChangeMedicationOrders).

Figure1.SatisfactionRatingsofIndividualTasks

T a s k N u m b e r

SystemUsabilityScale

TheSystemUsabilityScale(SUS)isasimple,10-itemLikert-typeattitudescale

providingaglobalsubjectiveassessmentofusabilityfromtheuser’sperspective(John

BrookeatDigitalEquipmentCompanydevelopedtheSUSin1986).TheSUSscaleisscored

from0to100;scoresunder60representsystemswithlessthanoptimalusability,scores

over80areconsideredbetterthanaverage.SeeAppendix5foracopyoftheSUS.

ThemeantotalSUSscorefortheMetaCareEnterpriseEHRwas29andrangedfrom

alowof10andahighof48.Overall,participant-usersratedtheirsatisfactionwiththe

MetaCareEnterpriseEHRsystemtobelessthanoptimal.

ComputerSystemUsabilityQuestionnaire

UsingtheComputerSystemUsabilityQuestionnaire(CSUQ;Lewis,J.R.(1995).(See:

IBMComputerUsabilitySatisfactionQuestionnaires:PsychometricEvaluationand

InstructionsforUse.InternationalJournalofHuman-ComputerInteraction,7:1,57-78).),

1 2 3 4 5 6 7 8 9 10 11 12

2.67 2.67 2.67

3.67 3.83

2.80 2.75

MeanTaskSatisfaction

participantsratedeachof19itemsonascalefrom1to7,witharatingof7beingmostin

agreementwiththepositively-wordeditem.Responsesforeachitemweresummedand

averagedtofourscales–InterfaceQuality,InformationQuality,SystemUsefulness-andan

overallscale.SeeAppendix6foracopyoftheCSUQ.

Figure2displaysCUSQratingsforeachofthefourscales.Ingeneral,participantsin

theMetaCareEnterpriseEHRstudyratedsystemusabilitytobelessthanwhatwouldbe

expected,withaveragescoresforeachofthethreesubscalesandtheoverallscalefalling

belowthemidpointoneach7-pointscale.OnInterfaceQualitytheaveragescoreforthe

participantswas3.27;onInformationQualitytheaveragescore2.40;onSystemUsefulness

theaveragescorewas2.80;andtheoverallaverageCUSQscorewas2.71.

2.50 3.50 4.50 5.50 6.50

Overall Score

System Usefulness

Information Quality

Interface Quality

User Rating (Out of 7)

Figure 2: Computer System Usability Questionnaire

DiscussionofFindings

Severaloftheparticipantsstruggledwithtasksandwereunabletosuccessfully

completeanumberofthem.Severaloftheparticipantswerenotabletocompletesome

tasksgiventhetimeconstraintsofasummativetest.Someparticipants,however,were

mostlyabletoperformtaskssuccessfullywithsomeminorassistance.

IngeneraltheparticipantsfeltsomewhatdissatisfiedwiththeMetaCareEnterprise

EHRandtheparticipantperformancerateseemedaboutthesameastheparticipant

satisfactionratingswouldsuggest.Whenlookingtheratingsandperformanceacross

participants,itisclearthatbothtrainingandexperienceoftheuserisdirectlytiedtothe

satisfaction,effectivenessandefficiencyoftheMetaCareEnterpriseEHR.

Effectiveness

OftheTwelve(12)taskspresented,thehighesttaskcompletionrateforanytask

was83percent.Twotasks,Task6(ModifyPatientAllergyList/ViewCDS)andTask10

(ReviewMedAdmin&DocAdmin)werenotsuccessfullycompletedbyanyparticipant.

Overallparticipants,themeansuccessfultaskcompetitionratewaslessthatexpected(42

percent)indicatingthatmanyparticipantshaddifficultycompletingthetasks.

PriorexperiencewithEHRsystemswaspositivelyrelatedtosuccessfultask

performance;participantswithpriorEHRexperienceweremorelikelytosuccessfully

completetasksthanthosewithoutpriorexperience..

Efficiency

Participantswhosuccessfullycompletedtasksgenerallycompletedthosetasks

withinanacceptabletime.Sometaskswerecompletemorequicklythanthecalculated

optimaltime,whileseveraltaskstooklongerthanexpected.Thetasksthattookthe

longestrequiredtheparticipantstonavigatetoaparticularpage,interactwithacomplex

workflow,locateandselectspecificactions.Thosetaskscouldbeperformedmorequickly

withanupdatetotheinformationarchitecture,anincreaseintheamountofembedded

assistance,andperhapsanenhancedvisualindicationofprimaryorsecondaryactions.

Someparticipantsmadeanumberoferrorswhenattemptingtonavigatetoward

solvingtheirassignedtasks.Manyoftheseerrorsmaybeassociatedwiththose

participantsnotbeingfamiliarandnotunderstandingthepresentedinformation

architectureoftheMetaCareEnterpriseEHRsystem.Asnotedabove,priorexperiencewith

EHRsystemswasrelatedtosuccessfultaskcompletion.Similarly,experienceandpractice

withthegivensystemmayhavepositiveeffectswithregardtouserefficiency.

Satisfaction

ParticipantswereminimallysatisfiedwiththeMetaCareEnterpriseEHRsystem;

ratingsontheSUS(mean=29)andtheCSUQ(Overallscore=(2.71)demonstratedaless

thanexpectedlevelofsatisfactionwiththesystem.

OntheCSUQ,participantsrankedthescale“(InterfaceQuality)”highestofthethree

scales,suggestingthatthesystemwasperhapswelllikedvisually.Individualtask

satisfactionratingswererelatedtoindividualuserperformance.Thoseparticipantswho

wereabletosuccessfullycompletetaskswerealsomorelikelytorankthosetasksas

satisfying,whilethoseparticipantswhodidpoorlyorwerenotabletocompleteatask

rankedthosetasksasunsatisfying..Overallhowever,participantsatisfactionwith

MetaCareEnterpriseEHRwasaboutwhatwasexpectedgiventheperformancedata

SummaryofMajorFindings

ThisevaluationdemonstratedthattheMetaCareEnterpriseEHRsystemisafairly

usablesystemwitharelativelyshortlearningcurve.Participantshadneverusedthe

MetaCareEnterpriseEHRsystembeforethestudyandexperienceddifficulty

understandingthenavigationandinformationarchitecture.Theresultssuggestthatwith

minorchangestheirperformanceandsatisfactionwouldlikelyimprove.

RiskAnalysisThefollowingtablepresentsaprioritizedlistoftasksprioritizedbytheriskoferroras

observedduringthetesting.

Table5.RiskAnalysis

T A S K PercentComplete

RiskStatus

Task1(SelectandActivateCDS) 33% ModerateTask2(ModifyDrug-DrugInteractionSeverity) 33% ModerateTask3(PlaceMed,Lab&RadiologyOrders) 33% ModerateTask4(ModifyPatientMedicationList) 67% LowTask5(ChangeMedicationOrders) 83% NoneTask6(ModifyPatientAllergyList/ViewCDS) 0% HighTask7(UpdateProblemList/ViewCDS) 33% ModerateTask8(UpdateMedicationList/ViewCDS) 60% LowTask9(UpdateVitalsInformation) 60% LowTask10(ReviewMedAdmin&DocAdmin) 0% HighTask11(ReconcileClinicalInformation) 60% LowTask12(SubmitElectronicPrescription) 40% Moderate

AreasforImprovement

Thefollowingisalistofpotentialareasforimprovement.Makingtheseandother

minorenhancementswillimprovetheoveralluserexperienceoftheMetaCareEnterprise

EHRsystemandincreasetheeffectiveness,efficiency,andsatisfactionforbothexperienced

andnoviceMetaCareEnterpriseEHRusers.

• ExcessiverelianceontheLeft-sidenavigationstructure.

o Themajorityoftheinformationarchitectureispresentedasatreeview

intheleftsidenavigation.Betterorganizationandplacementof

additionalsub-navigationelementswithinthecontentareawillhelp

usersfindwhattheyarelookingfor.

• Indicationofexternallinks.

o Manyparticipantsclickedonseverallinksthatprovidedinformation

aboutspecificdrugs.Givingtheusersanindicationthatthisisan

“informationallink”andnotaplacetoperformanactionwouldgreatly

reducethenumberoferrorsmade.

• Inconsistentlocationofcontrols.

o Theplacementofthesameorsimilarcontrolsand/orbuttonsshouldbe

consistentacrosstheapplication.

• Primaryactionsneedtobemademoresalient.

o Participantsoftenhaddifficultlylocatinga“save”button.Providinga

consistent,salientbuttonwillprovideuserswithaeasierpathtotask

completion.

Appendices

Appendix1:RecruitingScreener1.Areyoumaleorfemale? 2.Haveyouparticipatedinafocusgrouporusabilitytestinthepast6months?3.Doyou,ordoesanyoneinyourhomeworkinmarketingresearch,usabilityresearch,and/orwebdesign?4.Doyou,ordoesanyoneinyourhome,haveacommercialorresearchinterestinanelectronichealthrecordsoftwareorconsultingcompany?5.Whichofthefollowingbestdescribesyourage?

_____23to39; _____40to59; _____60to74; _____75orolder.6.Doyourequireanyassistivetechnologiestouseacomputer?7.Pleaselistyourmedicalornursingcredentials8.Howlonghaveyouheldthisposition?(no.ofmonths):9.Whattypeoffacilitydoyouworkinandwhatisyourrolethere?10.Howaremedicalrecordshandledatyour(main)workplace?

_____AllPaper _____SomePaper/SomeElectronic ___AllElectronic11.HowmanyEHRsdoyouuseorhaveyouworkedwith?12.Howmanyyearshaveyouusedanelectronichealthrecord?13.Abouthowmanyhoursperweekdoyouspendusingacomputer?14.Whatcomputerplatform(s)doyouusuallyuse?15.Inthelastmonth,abouthowoftenhaveyouusedanelectronichealthrecord?

_____Didnotuselastmonth ___Everyday _____Afewtimesaweek.

Appendix2:AdditionalParticipantInformation

ParticipantNumber

Doyou,ordoesanyoneinyourhome,

workinmarketingresearch,usability

research,webdesignorElectronic

HealthRecordsoftware?

Haveyouparticipatedinafocusgroupor

usabilitytestinthepast3months?

Doyou,ordoesanyoneinyourhome,haveacommercialor

researchinterestinanelectronic

healthrecordsoftwareorconsultingcompany?

Abouthowmanyhoursperweekdoyouspendusinga

computer?

Whatcomputerplatform(s)doyou

usuallyuse?

Howaremedicalrecordshandledatyour

facility

Lastmonth,howoftenusedEHR?

p1No No No 20 PC AllElectronic

Everyday

p2No No No 30

PCandApple AllElectronic

Everyday

No No No 50 PC

SomePaper/SomeElectronic

Everyday

No No No 50 PC

Afewtimesperweek

No No No 45PC,Apple,

Everyday

p6No No No 45 Other AllElectronic

Everyday

Appendix3:InformedConsentFormMetaHealthcare ITSolutionswould like to thankyou forparticipating in this study.Thepurposeof this study is toevaluateanelectronichealth records system. If youdecide toparticipate,youwillbeaskedtoperformseveral tasksusingtheprototypeandgiveyourfeedback.Thestudywilllastabout60minutes.AgreementIunderstandandagreethatasavoluntaryparticipant inthepresentstudyconductedbyMetaHealthcareITSolutions.Iamfreetowithdrawconsentordiscontinueparticipationatanytime.IunderstandandagreetoparticipateinthestudyconductedandrecordedbytheMetaHealthcareITSolutions.IunderstandandconsenttotheuseandreleaseofthevideorecordingbyMetaHealthcareITSolutions.Iunderstandthattheinformationandvideotapeisforresearchpurposesonlyand that my name and image will not be used for any purpose other than research. IrelinquishanyrightstothevideoandunderstandthevideorecordingmaybecopiedandusedbyMetaHealthcareITSolutionswithoutfurtherpermission.I understand and agree that the purpose of this study is tomake software applicationsmoreusefulandusableinthefuture.IunderstandandagreethatthedatacollectedfromthisstudymaybesharedwithoutsideofMetaHealthcareITSolutions.Iunderstandandagreethatdataconfidentialityisassured,becauseonlyde-identifieddata– i.e., identificationnumbersnotnames–willbeused inanalysisandreportingoftheresults.I agree to immediately raise any concerns or areas of discomfort with the studyadministrator.IunderstandthatIcanleaveatanytime.Pleasecheckoneofthefollowing:____YES,Ihavereadtheabovestatementandagreetobeaparticipant.____NO,Ichoosenottoparticipateinthisstudy.Signature:_____________________________________Date_____________________

Appendix4:ParticipantTasksTask1:SelectandActivateClinicalDecisionSupportAsasystemadministrator,you’vebeentaskedtoensurethatallrelevantClinicalDecisionSupportIntelliOrderGuidelinesareactiveintheapplication.ThelistofClinicalDecisionSupportIntelliOrderGuidelinesinquestionareasfollows:

• SuspectedApendicitis• AceInhibitorGuidelines• MedicationAllergySulfonamides• AdultCommunityAcquiredPneumonia• LowKandDigoxin• AsthmaTempMesalamine

UseMetacareEnterprisetoensurealloftheCDSguidelinesintheapplicationareactive.Task2:ModifyDrug-DrugInteractionSeverityLevelAsasystemadministrator,you’vebeentaskedtoensurethelevel2drug-druginteractionbetweenpotassiumsparingdiureticsandpotassiumpreparationsisactive.UseMetacareEnterprisetoensurethePOTASSIUMSPARINGDIURETICS/POTASSIUMPREPSdrug-druginteractionisactiveinthesystem.Task3:PlaceMedication,Laboratory,andRadiologyOrdersAfterdailyroundsforJaneSmith,youidentifythefollowingordersneedtobeplacedforthepatient:

• BasicMetabolicPanel(One-timeorder)• ChestX-Ray(One-timeorder)• Potassium10MEQOrallyTwiceDaily• Acetaminophenw/Codeine30/300mgOrallyEverySixHours

Duringorderentry,youreceiveadrug-druginteractionalertbetweenthePotassiumandthepatient’scurrentorderforThiazide.Additionally,youreceiveadrug-allergyalertbetweentheAcetaminophenw/Codeineandthepatient’scodeineallergy.YoudecidetoproceedwithorderentryofthePotassiumbutdecidenottoordertheAcetaminophenw/Codeine.Uponsave,youarepromptedtore-enteryourpasswordforpositiveidentification.UseMetacareEnterprisetoplacetheordersforJaneSmith.

Task4:ModifythePatient’sMedicationListAfterreviewingthedrug-druginteractionbetweenthenewlyplacedPotassiumorderandthecurrentTriamtereneorderonJaneSmith’sprofile,youdecidetodiscontinuetheTriamterene50mgDailyorderforJaneSmith.Upondiscontinuation,youarepromptedtore-enteryourpasswordforpositiveidentification.UseMetacareEnterprisetodiscontinuetheThiazideorderonJaneSmith’smedicationlist.Task5:ChangeMedicationOrdersAfterreviewingJaneSmith’spotassiumlabvalues,youidentifythatthePotassiumregimenneedstobeadjusted.YoureviewJaneSmith’sprofileandidentifythePotassium10MEQTwiceDailyordershouldbechangedtothefollowing:

• Potassium10MEQOrallyOnceDailyBeforesaving,youoverrideanyclinicalalerts.Uponsave,youarepromptedtore-enteryourpasswordforpositiveidentification.UseMetacareEnterprisetochangethePotassiumorderforJaneSmithTask6:ModifyPatient’sAllergyListandViewRelatedClinicalDecisionSupportAfterinterviewingJaneSmith,youareinformedthatsheisnolongerallergictoCodeine,butisactuallyallergictoSulfa.Youneedtoupdatethepatient’sallergylisttoreflectthenewinformation.AfterrecordingtheSulfaallergy,aclinicaldecisionsupportalerttriggerswarningofanexistingallergytoBactrimDS.UseMetacareEnterprisetoupdatethepatient’sallergylistwiththenewallergyinformationandreviewtheclinicaldecisionsupportinformationTask7:UpdatePatient’sProblemListandViewRelatedClinicalDecisionSupportJohnFrankisadmittedtothefacility.Adiagnosisofbacterialpneumoniaisidentified.AfterrecordingthepneumoniadiagnosisintheEHRapplication,thesystemrecognizesthepatientasanadultandtriggersaclinicaldecisionsupportalertforadultcommunityacquiredpneumonia.UseMetacareEnterprisetoupdatethepatient’sdiagnosisinformationwith‘bacterialpneumonia’andreviewtheclinicaldecisionsupportinformation.

Task8:UpdateMedicationListandViewRelatedClinicalDecisionSupportJohnFrankalsohasahistoryofcongestiveheartfailure.You’veidentifiedDigoxin0.25mgOnceDailyasahistoricalmedicationwhichmustbecontinuedwhileinthehospital.Youentertheorderonthepatient’smedicationlistandarealertedwithaclinicaldecisionsupportwarningofalowpotassiumlevelforthepatient.UseMetacareEnterprisetoupdatethepatient’smedicationlistwith‘Digoxin0.25mgOrallyOnceDaily’andreviewtheclinicaldecisionsupportinformation.Task9:UpdateVitalsInformationYouareanurseonapatientcarelocationandhavejusttakenthetemperatureofSamJohnson,apatientwithadiagnosisof‘chronicobstructiveasthmawithstatusasthmaticus.’Thepatientisrunningatemperatureof102degreesF.Afterrecordingthepatient’stemperature,aclinicaldecisionsupportalerttriggerssuggestingtheuseofmesalamineforasthmapatientsrunningatemperature.UseMetacareEnterprisetoupdatethepatient’stemperatureandreviewtheclinicaldecisionsupportinformation.Task10:Reviewthe‘5RightsofMedicationAdministration’andDocumentAdministrationofaMedicationAftertakinganddocumentingSamJohnson’stemperature,youreviewSamJohnson’smedicationrecordtoseeifhehasanordertouseincaseofafever.YouidentifyaTylenol500mgOralTabletscheduledasneededEvery4Hours.AfterpullingaTylenol500mgtabletfromtheautomateddispensingmachine,youreviewthepatient’selectronicmedicationadministrationrecordtoreviewthemedication5rightsinformationpriortoadministration.Youneedtoverifythefollowinginformationinthemedicationadministrationrecord:

• RightPatient• RightMedication• RightDose• RightRoute• RightTime

Afterverifyingthe‘5rights’information,youdetermineitissafetoadministertheTylenol500mgtothepatient.Afteradministration,youdocumenttheadministrationelectronicallyinthemedicationadministrationrecord.UsingMetacareEnterprise,verifythe‘5rights’informationanddocumenttheadministrationoftheTylenol500mgtabletforSamJohnson.

Task11:ReconcileaPatient’sClinicalInformationSummaryofcareinformationforLarryDonaldhasbeenreceivedfromCommunityHospitalanduploadedintotheMetacareEnterpriseEHR.Thefollowinginformationwasrecognized:Problems:

• CongestiveHeartFailure• DiabetesMellitus

Medications:• LantusInsulin15uSQatBedtime

MedicationAllergies:• Penicillin• Codeine

UseMetacareEnterprisetoviewandcomparetheuploadedmedicalrecordswiththeexistingpatientrecordstocreateasinglereconciledlistofmedications,medicationallergies,andproblems.Overrideanyclinicalalertswhichmayoccur.Task12:SubmitanElectronicPrescriptionSuzieBakerisbeingdischargedfromthefacilityonLipitor10mgOrallyOnceDailyfortheongoingtreatmentofherhighbloodpressure.Thepatientwasreceivingthemedicationwhileinthefacility,butneedsaprescriptiontransmittedtoCommunityPharmacyforpickupafterdischarge.UseMetacareEnterprisetocontinueSuzieBaker’sLipitor10mgathomeandsubmitaprescriptionelectronicallytoCommunityPharmacy.

Appendix5:SystemUsabilityScaleQuestionnaire

Appendix6:ComputerSystemUsabilityQuestionnairePleaseprovideyourimpressionoftheusabilityofthesystembyansweringeachofthequestionsbelow:1.Overall,IamsatisfiedwithhoweasyitistousethissystemStrongly 1234567 NA StronglyDisagree Agree2.ItwassimpletousethissystemStrongly 1234567 NA StronglyDisagree Agree3.IcaneffectivelycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree4.IamabletocompletemyworkquicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree5.IamabletoefficientlycompletemyworkusingthissystemStrongly 1234567 NA StronglyDisagree Agree6.IfeelcomfortableusingthissystemStrongly 1234567 NA StronglyDisagree Agree7.ItwaseasytolearntousethissystemStrongly 1234567 NA StronglyDisagree Agree8.IbelieveIbecameproductivequicklyusingthissystemStrongly 1234567 NA StronglyDisagree Agree9.ThesystemgiveserrormessagesthatclearlytellmehowtofixproblemsStrongly 1234567 NA StronglyDisagree Agree10.WheneverImakeamistakeusingthesystem,IrecovereasilyandquicklyStrongly 1234567 NA StronglyDisagree Agree

11.Theinformation(suchasonlinehelp,on-screenmessages,andotherdocumentation)providedwiththissystemisclearStrongly1 234567 NA StronglyDisagree Agree12.ItiseasytofindtheinformationIneededStrongly 1234567 NA StronglyDisagree Agree13.TheinformationprovidedforthesystemiseasytounderstandStrongly 1234567 NA StronglyDisagree Agree14.TheinformationiseffectiveinhelpingmecompletethetasksandscenariosStrongly 1234567 NA StronglyDisagree Agree15.TheorganizationofinformationonthesystemscreensisclearStrongly 1234567 NA StronglyDisagree Agree16.TheinterfaceofthissystemispleasantStrongly 1234567 NA StronglyDisagree Agree17.IlikeusingtheinterfaceofthissystemStrongly 1234567 NA StronglyDisagree Agree18.ThissystemhasallthefunctionsandcapabilitiesIexpectittohaveStrongly 1234567 NA StronglyDisagree Agree19.Overall,IamsatisfiedwiththissystemStrongly 1234567 NA StronglyDisagree Agree

Appendix7:Detailedtaskperformanceforeachparticipant.ParticipantNumber

Task1(SelectandActivateCDS)

TaskTime Outcome#Path

DeviationsTask

Satisfaction

p1 5:24 Success 1 2p2 2:35 Success 2 5p3 3:04 TimeOut 1 1p4 3:07 TimeOut 1 3p5 3:12 TimeOut 6 1p6 3:47 TimeOut 2 4

Expectedtime 3:00

AverageTimeonTask 3:59

AverageTaskSatisfaction 2.67

Average#PathDeviations 2.17

PercentSuccess 33%

ParticipantNumber

Task2(ModifyDrug-DrugInteractionSeverity)

DeviationsTask

Satisfaction

p1 3:39 TimeOut 1 1p2 3:03 TimeOut 2 1p3 3:34 Success 1 2p4 3:18 TimeOut 3 3p5 2:59 Success 1 5p6 3:27 Assisted 2 4

ExpectedTime 3:00

PercentSuccess 33%

ParticipantNumber

Task3(PlaceMed,Lab&RadiologyOrders)

DeviationsTask

Satisfaction

p1 6:01 TimeOut 4 2p2 5:00 Success 4 3p3 6:54 Success 1 2p4 6:30 Assisted 3 2p5 5:50 Assisted 5 4p6 5:32 TimeOut 3 3

ExpectedTime 5:00

PercentSuccess 33%

ParticipantNumber

Task4(ModifyPatientMedicationList)

DeviationsTask

Satisfaction

p1 3:11 TimeOut 0 2p2 3:08 TimeOut 2 3p3 1:11 Success 0 4p4 2:07 Success 2 5p5 2:32 Success 2 5p6 1:48 Success 0 3

ExpectedTime 3:00

PercentSuccess 67%

ParticipantNumber

Task5(ChangeMedicationOrders)

DeviationsTask

Satisfaction

p1 3:19 Success 1 3p2 2:00 Success 0 4p3 2:38 Success 1 4p4 1:51 Success 1 4p5 3:04 TimeOut 3 3p6 1:27 Success 1 5

ExpectedTime 3:00

PercentSuccess 83%

ParticipantNumber

Task6(ModifyPatientAllergyList/ViewCDS)

DeviationsTask

Satisfaction

p1 4:27 TimeOut 2 1p2 3:15 TimeOut 6 4p3 3:02 TimeOut 6 1p4 3:07 TimeOut 0 2p5 3:07 Fail 6 1p6 3:03 TimeOut 3 1

ExpectedTime 1:30

PercentSuccess 0%

ParticipantNumber

Task7(UpdateProblemList/ViewCDS)

DeviationsTask

Satisfaction

p1 4:06 TimeOut 2 1p2 1:27 Success 0 5p3 2:07 Success 0 3

p4 SystemIssue

p5 3:09 TimeOut 4 1p6 3:02 TimeOut 2 1

ExpectedTime 3:00

PercentSuccess 33%

ParticipantNumber

Task8(UpdateMedicationList/ViewCDS)

DeviationsTask

Satisfaction

p1 4:02 TimeOut 0 1p2 3:10 Success 1 5p3 1:56 Success 0 4

p4 SystemIssue

p5 3:40 Fail 6 1p6 3:27 Success 3 3

ExpectedTime 3:00

PercentSuccess 60%

ParticipantNumber

Task9(UpdateVitalsInformation)

DeviationsTask

Satisfaction

p1 1:25 Success 0 3p2 3:05 TimeOut 5 1p3 3:16 TimeOut 5 1

p4 SystemIssue

p5 0:43 Success 0 5p6 1:43 Success 0 4

ExpectedTime 3:00

PercentSuccess 60%

ParticipantNumber

Task10(ReviewMedAdmin&DocAdmin)

DeviationsTask

Satisfaction

p1 4:03 TimeOut 2 1p2 4:07 TimeOut 2 2p3 4:21 TimeOut 1 1

p4 SystemIssue

p5 4:00 TimeOut 5 1p6 4:15 TimeOut 1 1

ExpectedTime 4:00

PercentSuccess 0%

ParticipantNumber

Task11(ReconcileClinicalInformation)

DeviationsTask

Satisfaction

p1 3:07 Fail 6 1p2 3:17 Success 2 5p3 4:15 Success 2 3

p4 SystemIssue

p5 3:54 Fail 6 3p6 3:43 Success 1 5

ExpectedTime 5:00

PercentSuccess 60%

ParticipantNumber

Task12(SubmitElectronicPrescription)

DeviationsTask

Satisfaction

p1 3:02 TimeOut 2 1p2 4:05 TimeOut 2 1p3 2:43 Success 0 3

p4 SystemIssue

p5 4:01 TimeOut 5 1p6 4:40 Success 2 4

ExpectedTime 4:00

PercentSuccess 40%

Appendix B: Quality Management System

Appendix C: Privacy and Security

Test Results Summary Document History Version

V1.0 Initial release April 10, 2017

END OF DOCUMENT

Description of Change Date

ONC HIT Certification Program Test Results Summary for...

Documents