Evaluation of electronic medical records

- a clinical task perspective

Presentation of thesis

Norwegian University ofScience and Technology, Trondheim

Hallvard Lærum

Error ?

The Kvalis Project

• Quality assurance of electronic medical records in hospitals

• Funded by The Research Council of Norway

• Interdisciplinary, three PhD fellows– Sociology: Gro Underland– Informatics: Gunnar Ellingsen– Medicine: Hallvard Lærum

Evaluation of electronic medical records

Descriptive aspectsWhat is the level of functionality of current EMR systems?Where are they found?

Explorative aspectsHow may the effects of various EMR systems on physicians’ clinical practice in various hospitals be evaluated?

InvestigationsHow does the method perform in various contexts?Reliability and validity studies

The 3rd Scandinavian University Course in medical informatics

Building a researchnetwork

Building a researchnetwork

Keeping up a websiteKeeping up a website

Arranging monthly seminarsArranging monthly seminars

Arranging meetingsArranging meetings

Establishing contact withEMR vendors Establishing contact withEMR vendors

Department of Medical Informatics & Clinical Epidemiology, Oregon Health and Science University

Establishing contact withresearch centersinternationally

Establishing contact withresearch centersinternationally

Establishing contact with hospitalsEstablishing contact with hospitals

Descriptive aspects of the thesis

EMR systems are very common in Norwegianhospitals (80% of hospital beds as of 2002)

The systems are essentially document-based, covering the narrative parts of the medical records, lab data and radiology reports.

NoNo systematic evaluations systematic evaluations of Norwegian EMRs of Norwegian EMRs published by 1999published by 1999

Explorative aspects of the thesis 1

Considerations for development of an evaluation method

• Applicable to various EMR systems and hospitals– Not tied to a single system or hospital

• Carried out using limited resources– Fast results

• Used in operational reviews, incorporating organizational issues

– Laboratory simulations less feasible

• Involve measurement close to the site of impact– Involving how clinical work is done

The first task inventory

• Direct observations, 40 h– 20 h St.Olavs Hospital, 20h RiTø (Gunnar E.)– 7h transcribed verbatim– Basis for hierarchical task analysis– Noticed that the PCs were not much in use

• The first task inventory: 104 tasks– Too large for practical use– Task wording rather technical– Covert tasks not included (e.g. cognitive tasks)

D e finep rob lem

G e t m e d ica lkn o w le d ge

G e t p a tie n td a ta

C re a te n ewin fo rm a tion

In fo rmP a tie n t

N e g o tia tew ith p a tie n t

E xecu tein te rve n tion

D o cum e nta c tiv ity

D ia gn o s tic s an d Tre a tm e nt

G e t e x is t ingp lan

G e t in foa b ou t reso urces

N e g otia tew ith p a tie n t

In fo rmP a tie n t

D o cum e ntp lan

P la nn ing

G e ne ra llo g is tics

C o m m u n ica tion

A d m in is tra tive

R e se a rch

Q u a lityC o n tro l

T e a ch ing

P a tien t's re la tionto soc ie ty

O th er

O th er

Explorative aspects 2

Video observations – Flow of information

OntoLog Courtesy of Jon Heggland, IDI NTNU

• Information seeking behaviour– Looking for incidents where needed

information was not found

• Tagging of video recordings

• Stimulated recall* not sucessful– Too time-consuming (and boring!)

for an average clinician– Recruitment problems

• Prohibitively resource intensive– 9 patients, 4.5 h video recordings,

two physicians

• Noticed that PCs were not much in use

* Kushniruk & Patel 1995

Simplification of the task list

• Task list to be used in questionnaires and interview guides

• Considerations for the process– Emphasis on information-related tasks essential for patient

care– Wording adapted to clinicians’ way of speaking– Covert tasks were included– Tasks suitable for most specialties were retained– Tasks representing negligible work were deleted– Gorman’s five information needs were taken into account– Tasks should be supported by functionality found in current

EMR systems or in those expected in the near future

• Multiple iterations by work group (2 physicians, 2 informaticians)

!

A list of relevant tasks for the clinician

Norwegian English translation

1. Få oversikt over pasientens problemstilling Review the patient's problems

2. Lete frem enkeltopplysninger fra pasientjournalen Seek out specific information from patient records

3. Følge resultatene av en bestemt prøve eller us. over tid Follow the results of a test or investigation over time

4. Slå opp svar på nye prøver eller undersøkelser Obtain the results from new tests or investigations

5. Føre daglige og/eller forefallende journalnotater Enter daily notes

6. Få tak i oppl. om prosedyre for utredning eller behandling Obtain information on investigation or treatment procedures

7. Få svar på spørsmål om generell med.-faglig kunnskap Answer questions concerning general medical knowledge

8. Få ut samledata for en gruppe pasienter Produce data reviews for specific patient groups

9. Rekvirere klin-kjem. lab.analyser Order clinical biochemical laboratory analyses

10. Slå opp svar på klin.-kjem. lab.analyser Obtain the results from clinical biochemical laboratory analyses

11. Rekvirere rtg.us., UL eller CT Order X-ray, ultrasound or CT investigations

12. Slå opp svar på rtg., UL eller CT Obtain the results from X-ray, ultrasound, or CT investigations

13. Rekvirere. andre supplerende undersøkelser Order other supplementary investigations

14. Slå opp svar på andre supplerende undersøkelser Obtain the results from other supplemental investigations

15. Henvise pas. til annen avdeling eller spesialist Refer the patient to other departments or specialists

16. Ordinere behandling direkte (med., op. eller annen) Order treatment directly (e.g. medicines, operations etc.)

17. Skrive resept Write prescriptions

18. Skrive sykmelding Complete sick-leave forms

19. Samle inn pasientopplysninger til ulike legeerklæringer Collect patient data for various medical declarations

20. Gi skriftlig individuell info til pasienten Give written individual information to patients

21. Gi skriftlig generell medisinsk-faglig info til pasienten Give written general information to patients about the illness

22. Samle inn opplysninger til epikrise Collect patient information for discharge reports

23. Kontrollere og signere ferdig skrevne diktater Check and sign typed dictations

The first Questionnaire was compiled

• Translated sections (two-way translation validation involving native english speaking translator)– End User Computing Satisfaction scale (Doll & Torkzadeh)– Global user satisfaction questions (SGUS, Anderson & Aydin)– Computer literacy (Brown & Coney)

• New sections– Frequency of PC /EMR use for the 23 clinical tasks– Basic availability of computers

Pilot study (n=22)

Investigations in this thesis

• National survey, 2001

• Local survey in Arendal, 2002

• Validation and reliability studies, 2003

The national survey Feb 2001

• Survey– 314 physicians in 32 hospital units in 19

hospitals– 227 (72.3%) responded after one reminder

(135 respondents were reminded)

• Telephone interviews with key IT personnel in each hospital

– What clinical tasks could be performed using the EMR system?

– A set of minimal functionality requirements, incl. requirements for integration of external systems

The section covering PC/EMR useMissing

responses!

Implemented functions

DIPS Infomedix

1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge8. Produce data reviews for specific patient groups9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv.15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients22. Collect patient information for discharge reports23. Check and sign typed dictations

DocuLive

0 % 50 % 100 % 50 % 100 %50 % 100 %

General

Medical knowledge

Aggregated dataSupplementary investigations

Actions

Information to patient

Communication and verification

Percent of respondents offered minimal functionality for the given task

• There is a considerable difference in implemented functions between systems

Use and implementation of EMR systems

DIPS Infomedix

1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations

DocuLive

0 % 50 % 100 % 50 % 100 %50 % 100 %

General

Medical knowledge

Aggregated dataSupplementary investigations

Actions

Patient info

Communication and flow of information

Percent of respondents offered functionality/ using it

Use both EMR and other software

Use EMR Type of program not stated

Implementation

A lot of implemented functionality appears not to be used

Cut-off: Respondents answering ”half of the

time” or better are users

Use and implementation of EMR systems

DIPS Infomedix

1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes (i.e. progress notes)6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations

DocuLive

0 % 50 % 100 % 50 % 100 %50 % 100 %

General

Medical knowledge

Aggregated dataSupplementary investigations

Actions

Patient info

Communication and flow of information

Percent of respondents offered functionality/ using it

Use both EMR and other software

Use EMR Type of program not stated

Implementation

Some physicians enter daily notes themselves

Use and implementation of EMR systems

DIPS Infomedix

1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations

DocuLive

0 % 50 % 100 % 50 % 100 %50 % 100 %

General

Medical knowledge

Aggregated dataSupplementary investigations

Actions

Patient info

Communication and flow of information

Percent of respondents offered functionality/ using it

Use both EMR and other software

Use EMR Type of program not stated

Implementation

Obtaining lab data (and other results) is popular, order entry is not

Use and implementation of EMR systems

DIPS Infomedix

1. Review the patient's problems2. Seek out specific information from patient records3. Follow the results of a test or investigation over time4. Obtain the results from new tests or investigations5. Enter daily notes 6. Obtain info on investigation or treatment procedures7. Answer questions concerning general medical knowledge8. Produce data reviews for specific patient groups 9. Order clinical biochemical laboratory analyses10. Obtain the results from clin. biochemical lab. analyses 11. Order X-ray, ultrasound or CT investigations 12. Obtain the results from X-ray, ultrasound or CT inv. 13. Order other supplementary investigations 14. Obtain the results from other supplementary inv. 15. Refer the patient to other departments or specialists16. Order treatment directly (medical, surgery. or other)17. Write prescriptions 18. Complete sick-leave forms19. Collect patient info for various medical declarations 20. Give written specific information to patients21. Give written general information to patients 22. Collect patient information for discharge reports23. Check and sign typed dictations

DocuLive

0 % 50 % 100 % 50 % 100 %50 % 100 %

General

Medical knowledge

Aggregated dataSupplementary investigations

Actions

Patient info

Communication and flow of information

Percent of respondents offered functionality/ using it

Use both EMR and other software

Use EMR Type of program not stated

Implementation

Big surprise: Nobody is using the EMR system to write prescriptions or complete sick-leave forms

Why this lack of EMR use? – 1Answers provided by the questionnaire is limited

I wish the EMR couldbe more like WordStar

Where’s the PCwhen I need it?Lack of available computers?

Con: Some PCs were available: 93% a computer in their office, 97% had a computer available in other rooms used for clinical work.

Pro: Clinicians’ work is not stationary. No hospitals have yet implemented mobile computing, and no hospitals can afford a PC in every room (personal communications)

Low computer literacy?Con: The physicians had at least a basic knowledge of computers (average score 72.2 ±1.6, 92% owned a computer)

Pro: Specific EMR system training may be needed

Why the lack of EMR use? - 2

I know my pre-scriptions by heart!

I’d like to find allthe information

in one place

Follow-up studies in each hospital are necessary

Low usability of the EMR system as a whole?Prescriptions: Navigating to the prescription module, finding the correct medication, typing dosage, administration route and package size and printing on the correct type of paper may represent more work than doing it by hand.Sick-leave forms: Slow system response times (12-15 sec)

The EMR is not completePro: Until all relevant clinical information is found in the EMR, it will serve a secondary role.Pro: The functionality of current EMR systems is limited

Resistance to new work roles?Pro: The usage patterns found in the national study are conform to traditional work roles.Pro: Work role issues were the most prevalent theme in the answers to the open-ended questions in the validity study (Paper 4).

Typing is a secretary’s job!

Order entry is anurse’s job!

Local study of EMR system in Aust-Agder Hospital, Arendal(Paper 2 and 3)

• The paper-based medical record is scanned and obliterated!

• Questionnaire rev.2 (versions for medical secretaries and nurses developed separately)

– 70 of 80 physicians (88%)– 79 of 85 medical secretaries (93%)– 172 of 235 nurses (73%)

• Interviews– 8-12 representatives of each profession,

0.5-2 h interviews

The EMR in Aust-Agder Hospital

2nd revision of the questionnaire

Physician’s use of the regular EMR at Aust-Agder Hospital

Much higher frequency of EMR use than in other hospitals having the same system

The physicians reported a less frequent use of the scanned document images

Physicians: Change in ease of performing the clinical tasks

Most tasks related to information retrieval are reportedlyeasier using the system, but 33% of internists found task 1 and 2 more difficult

User satisfaction of physicians

The physicians are relatively satisfied with the regular EMR, but not with the use of the scanned document images

Validation and reliability studies3rd revision of questionnaire

• Test-retest reliability study– 37 of 96 physicians (39%) from three hospitals having different EMR

systems completed the questionnaire two times. ”Test” questionnaire: 55.2% (52 of 96), ”Retest” questionnaire. 71% (37/52).

• Content validity study – interviews– 10 physicians, 1 h interviews– Relevance of tasks, estimation of accuracy of answers to task-oriented

questions, themes in open-ended questions.

• Criterion validity using data from local and national studies– Criteria

• General information retrieval vs. tasks involving information retrieval• Overall work performance vs.task performance • Task performance vs. user satisfaction

– Majority of tasks covered, median correlation coefficient 0.445-0.513

Test-retest reliability: Weighted kappa was generally high

Problematic tasks were related to functionality not available locally, and problems in discering the EMR from other software

Themes appearing in the interviews

• Work role issues (8/10 physicians) – resistance to doing ”clerikal” tasks– Tasks 10 and 19 – The third method would be the "ask-the-nurse" method. This is convenient, though,

then I may do other things. [In the future] It could be that it will be so easy to do it, that I could do it myself…if it's really easy, a completely negligible task. But if it takes some time..if I have to wait or something..then I feel that it should be a medical secretary's task, at least in a hospital. (respiratory diseases)

• Various wording problems (7/10 physicians)– Tasks 16, 4 and 21– I don't understand what you mean with "directly"…write orders on the [order entry

form], request or order an operation…one other [example] is requesting treatment by physiotherapist (orthopedy)

• Questions regarding use of non-existent functionality (7/10 physicians)– Task 3– Some questions are difficult to answer, as we can't log on [to the EMR system] and

find results from X-ray investigations (plastic surgery)

• Distinguishing EMR from other software or media (6/10 physicians)– Task 4– Is [the separate lab system] regarded as a part of [the EMR system]? (neurology)

Considerations of self-reported use

• Sources of error in self-reported use– Telescoping and other memory-related effects– Implicit expectations of the questionnaire– Willingness to respond– Strategic responses– Other

– There seem to be more reasons for the user to report too high frequency of use than too low.

• Use of the EMR system may not necessarily benefit the patient– EMR system used the wrong way– Unintended effects of using the EMR system– Logical errors in the EMR system

Application of the task list and the questionnaire

• The questionnaire may be used for screening, focusing the evaluation effort and providing a basis for further exploration– Separating reported use from non-use

• The combination of EMR use and task performance may identifiy problematic aspects of the EMR system.

• The questionnaire should always be combined with a qualitative study to investigate the ”why” (and validate the findings)

Thank you!

Extra information

National study: Inclusion of hospitals

• Inclusion of hospitals– The EMR should at least contain medical narratives (admission reports, progress

notes, discharge reports), directly or indirectly updated by and electronically available to the physicians. (minimal functional requirement for task 1)

National study: Lessons learned in this survey:

• Missing responses– ”Use EMR/Other program” questions frequently not

answered (median 10%, IQR 5-15%)

• Two hospital units (8 physicians) had to be excluded post hoc– 1: Minimal requirements for EMR implementation not met– 2: Two EMR systems in use simultaneously

• Error discovered one year after publication– Task not supported after all in one hospital unit (11

physicians)– Lab results not integrated in the EMR in this unit, available

in stand-alone system only. – Correction sent to BMJ (paper 1) Nov 2002.

Local study: User satisfaction of physicians, nurses and medical secretaries

The medical secretaries are most satisfied, the physicians least satisfied with the system

Local study: Use of hospital information system for individual tasks

Local study: Results of interviews

• All professions found that the patient data were more accessible when stored electronically.

• Physicians (internists) found searching in scanned multiple documents time-consuming and difficult

• The medical secretaries found that generation, handling, fetching and delivery of paper documents and logistics of paper-based patient records had diminished dramatically.

• Nurses were still using pen and paper documenting their activities.

Changes in rev.3 of the questionnaire

• Added ”escape” choices in task-related questions– Task not relevant for me– Task not possible using the local EMR system

Validation: Interviews 1: Accuracy

Discordant interpretationfor 1/10 in task 6, ”Obtain information on investigation or treatment procedures”

validation: Interviews 2: Relevance of tasks

Task 8, ”Produce data reviews for specific patient groups” was not found relevant by a majority of the physicians.(not part of one’s job/ infrequently performed)