Why and How to Improve

DIAGNOSIS DECISION MAKING

www.health.ebsco.com

2

In one of the classic papers in our field, Dr. Georges Bordage asked a very simple question: “Why did I miss the diagnosis?”1 The answer is equally simple. “I just didn’t think of it.” Over the past decade, we have learned a great deal about why this happens. In short, it is System 1’s fault. According to the dual process paradigm that describes how doctors think, if we believe we recognize what’s going on, we assign the diagnosis automatically and subconsciously, using our “intuition.” This is System 1. It works remarkably well, is extremely fast and efficient, and best of all, it’s almost always correct. The problem we have in medicine is that “almost always” isn’t good enough. The mistakes that invariably arise using intuitive thought are the diagnostic errors that lead to inappropriate medical costs, injury, and harm. Cognitive diagnostic errors include:

• Being in the wrong context• Being influenced by someone else’s diagnosis presented to us• Accepting the first diagnosis that explains all the facts (premature closure, satisficing)

System 2 is the counterpart to System 1 and represents the deliberate, conscious consideration of all the diagnostic possibilities. Conscious consideration is the antidote to many of the shortcomings of System 1. The deliberate consideration of alternatives may bring something to mind that just wasn’t considered. The question is: How can we overcome the “I just didn’t think of it” problem that underlies much of diagnostic error.

FORWARD BY:Mark Graber, MD, is a Senior Scientist at RTI’s Health Care Quality and Outcomes Program, and Professor Emeritus of Medicine at the State University of New York at Stony Brook. Dr. Graber has an extensive background in biomedical and health services research, with over 70 peer-reviewed publications. He is a national leader in the field of patient safety and originated, with Ilene Corina of New York, Patient Safety Awareness Week in 2003, an event now recognized internationally.

Dr. Graber has been a pioneer and national leader of efforts to address diagnostic errors in medicine. He founded and chaired the Diagnostic Error in Medicine conference series and has several landmark publications on this topic. Currently, Dr. Graber directs an AHRQ ACTION study focused on interventions to reduce diagnostic errors in ambulatory care settings using checklists.

FORWARD

Why and How to Improve DIAGNOSIS DECISION MAKING

3

WHY IS THERE A NEED TO IMPROVE DIAGNOSIS DECISION MAKING?

Diagnosis is the first and most important decision made about the patient. It determines all subsequent

treatment and determines the course of each patient encounter. How well this decision is made, therefore, is

one of the most significant determinants of healthcare quality and efficiency.

The following are some of the areas where the speed and accuracy of diagnosis have a key impact:

• Referrals from primary care to specialists - Research shows that 30-50% of referrals from primary care to

specialists are inappropriate, leading to delays in diagnosis, patient dissatisfaction and lengthy waits at

specialist clinics.

• Test ordering - Surveys and anecdotal evidence place the level of unnecessary and defensive test

ordering at 40%. This is extremely costly and subjects patients to unnecessary clinical risk through invasive

procedures and radiation exposure.

• Medical malpractice - Misdiagnosis accounts for 30-40% of all malpractice claims and about 2/3 of all

claims in primary care. Additionally, diagnostic errors are frequently the leading or second leading cause

of malpractice claims in the United States, accounting for twice as many alleged and settled claims as

medication errors.

• Patient satisfaction - Patient satisfaction accounts for about 30% of Medicare payments. As a result, many

hospitals are investing in typical customer service initiatives used for years in other industries. However,

in many cases these are viewed as gimmicks by patients and will not make up for poor quality of care. A

survey of patients’ concerns showed that their top concern when visiting their primary care physician is

diagnosis, and in hospitals it is their 2nd most important concern.

• Employee skills - Healthcare is a knowledge-intensive industry, and a key issue underpinning an institution’s

success is the clinical skills of all its clinicians. One way of boosting skills across the board is to provide tools

that improve clinical skills. Although diagnosis is traditionally seen as the preserve of the physicians, it is

the nurses who are caring for the patient most of the time and improving their diagnosis skills can lead to

better patient safety and quality of care.

• Errors related to missed or delayed diagnoses are a frequent cause of patient injury, and as such, are an

underlying cause of patient safety-related events. Diagnosis errors, which are almost always preventable,

happen frequently and can cause significant harm.

The primary cause of diagnosis errors is related to how physicians think.

Why and How to Improve DIAGNOSIS DECISION MAKING

4

The deliberate consideration of alternatives may bring a diagnosis to mind that wasn’t considered.

DIAGNOSIS ERRORS AND SOLUTIONS

There is substantial literature showing there are many reasons for delays and errors in diagnosis. Some causes

are system related, such as test results being misplaced or not received by the physician, and therefore not

acted on or communicated to the patient. However, the primary cause of diagnosis errors is related to how

physicians think and the process of working up a patient’s diagnosis. In a 2008 paper in “Medical Teacher,”

Robert Trowbridge from Maine Medical Center sets out 12 tips for teaching avoidance of diagnostic error. 2

Premature Closure and Biases

The more common causes of diagnosis errors are due to how a doctor thinks. There is a large body of work

describing the many biases, that we, as human beings and not just clinicians, are prone to. The research lists

over a 100 different biases but the main types that cause the errors in diagnosis are the ‘availability’ ones.

In a time-constrained industry, this is to be expected. In one landmark paper, “Diagnostic error in internal

medicine,”3 a classic cause is “premature closure” where the clinician decides on a diagnosis very quickly,

but then fails to consider other reasonable possibilities until it is too late. In any analysis of cases where the

diagnosis was delayed or missed, premature closure has been the most common contributing bias.

Cognitive De-biasing

One of the proposed solutions to this cognitive problem is termed “cognitive de-biasing” and involves clinicians

being made aware of these issues as part of their medical training. This solution will help, but in order to be

sustainable, it needs to be accompanied by the routine use of tools to help at the point of care.

Differential Diagnosis

Another solution commonly proposed is actually very old and is the routine construction of a comprehensive,

differential diagnosis. Olga Kostopoulou has carried out a number studies looking at the predictors of

diagnostic accuracy, including “Missing celiac disease in family medicine: the importance of hypothesis

generation” 4 and “Diagnosis of difficult cases in primary care.” 5

In the research, Kostopoulou found the most significant factor in the process is having a good differential

diagnosis that includes what turns out to be the correct diagnosis.

As a growing amount of research is published about misdiagnosis and its causes in various clinical settings, one

of the most interesting points to emerge is the consistent conclusion that clinicians should have broadened

Why and How to Improve DIAGNOSIS DECISION MAKING

5

their differential or done one in the first place. Examples of this are “Diagnostic Errors in Primary Care: Lessons

Learned” 6 by John Ely et al. and “Types and origins of diagnostic errors in primary care settings” 7 by Hardeep

Singh et al.

Although the construction and use of a comprehensive differential diagnosis has been taught for over 100

years, it is not used routinely in medicine. One of the main reasons for this is the time needed to construct

one. Due to a lack of time in the emergency department or primary care, for example, many clinicians rely on

their memory to construct a differential. However, with a universe of diagnoses within primary care of 200-300,

compared to a total universe of about 12,000 diseases, it is obvious that, on occasion, a clinician will simply not

think of a diagnosis either because they did not remember it or never knew it in the first place.

If there is diagnostic doubt, the clinician typically consults with colleagues, reads textbooks, or researches

online to investigate further. With medical textbooks and online reference resources, it is very difficult for

physicians to search for something when they don’t know what to look for. A search for “toxic shock,” for

example, will provide huge amounts of information. But, if you are unsure and just know that the patient

has ankle pain, ankle edema, diarrhea and fever, the traditional reference resources are not very helpful in

connecting and making sense of all of the signs and symptoms.

In these more unusual or complex clinical presentations, diagnostic decision aids can be particularly helpful, as

they are designed to produce a list of likely diagnoses for a given set of signs and symptoms. Their job is to get

the clinician thinking about a disease that he had not thought about previously. Instead of taking several hours,

days or even years to suggest the right diagnosis using the traditional methods, the diagnosis decision aids

work in seconds. These tools buy the time that the clinician needs to think.

If there is diagnostic doubt, clinicians typically consult with colleagues, read textbooks, or research online to investigate further.

Patient Engagement

There is an increasing body of evidence and growing movement to engage patients in their care and treat

them as partners in the care team rather than as passive recipients. This is particularly relevant to diagnosis as

the patient is clearly the expert on their symptoms and their story, and 80% of the time, the information they

provide reveals a diagnosis. The National Patient Safety Foundation’s Lucien Leape Institute has published a

report on patient engagement. 8

In order to help patients make sense of their symptoms, they should be encouraged to use properly-designed

‘symptom checkers’ to help them research their diagnoses and contribute to a differential diagnosis jointly

produced by them and their doctor. Patients could even be encouraged to use a symptom checker before a

Why and How to Improve DIAGNOSIS DECISION MAKING

6

consultation. Their findings can form part of the medical note and a good basis for the start of the consultation.

The Current State of Diagnostic Tools

With the nature of the diagnostic process, technology can be useful to help support the clinician. Initial

attempts in the 1960s were focused on the improvement in diagnosis of one specific problem, such as

abdominal pain. Although these showed that clinicians did a better job when using them, the tools were time-

consuming and proved to be impractical for use in a busy clinical setting, so they were never adopted. Another

factor was the intended users were specialists, who had less need.

The 1970s and 1980s brought the first general diagnostic. These tools were also not widely adopted, primarily

due to the time taken to use them. These systems were highly developed but were limited by the technology

available at the time. The tools were “rules-based systems,” which means that each symptom was associated

with a particular disease with an assigned probability. These systems worked satisfactorily on a small scale but

became difficult to manage on a large scale as each symptom or diagnosis needed to be kept current. The rigid

nature of a rules-based system also means users could only enter a feature that was in the system’s database. A

by-product of this problem was the difficulty of integrating these systems into electronic medical records.

In 2012, new diagnosis decision support systems were developed using statistical natural language processing

engines that are applied to a database of disease presentations or illness scripts, rather than a rules-based

model. These new systems are considerably more advanced and employ the following:

Statistical Natural Language Processing

Statistical natural language processing software understands the meaning and concepts within natural

language. As opposed to natural language processing, which tries to make sense of every word, statistical

natural language processing works by understanding key concepts. This ability is similar to a human being who

can generally understand what a conversation is about by just hearing a small amount of it rather than having

to study every word. Higher success rates occur with applications using statistical natural language processing

rather than natural language processing. The most important part of the statistical natural language processing

application is the database and how the system is trained.

Diagnostic decision support systems comb through massive amounts of medical knowledge and return a set of likely diagnoses for a given set of signs and symptoms. This allows the

clinician to think about a disease that he had not thought about previously.

Why and How to Improve DIAGNOSIS DECISION MAKING

7

High Quality of Knowledge

Diagnosis decision support systems leverage what computers do best – quickly comb through massive

amounts of medical knowledge and return a short list of possible diagnoses to hand over to the clinician. The

key to the performance of these systems is not the quantity of knowledge that is indexed but its nature and

quality.

Robust Algorithms

Algorithms improve the quality of the initial query and filter the raw output. For example, a synonym file

employs a database of similar medical terms. When a query of fever is entered into the system, the diagnosis

decision support system also searches for pyrexia and high temperature. The system also allows for common

medical abbreviations such as SOB (shortness of breath) to be understood. Other algorithms filter the raw

output so that the results are relevant for the age, gender and region of the patient.

Assisting with the Differential

Today’s diagnosis decision support systems are designed to support System 2 thinking by extracting clinical

features and producing a short list of possible diagnoses that could be the cause of the combination of signs

and symptoms. The clinician can use this list to form a differential diagnosis and the relative likelihoods of each

diagnosis.

Mobilizing Knowledge

Traditional reference resources are only useful when you know what you are looking for and have a specific

question. Modern diagnosis decision support systems start from the premise that you don’t know what you

are looking for but have a patient with a certain set of clinical features. By providing a list of diagnoses that

could be the cause of those clinical features, the diagnosis decision support system provides the clinician with

specific questions to ask and mobilizes knowledge around each diagnosis.

EMR / HIT Integration

By utilizing free text, modern diagnosis decision support systems easily integrate with other systems and avoid

the need for encoded data. When integrated into an EMR, the diagnosis decision support systems use the

clinical features that have already been entered by the clinician, either by free text or a structured template,

to provide a list of possible diagnoses. Selected diagnoses can then be saved back into the EMR to serve as a

record of the differential diagnosis considered for the patient.

Return on Investment

With the significance and downstream impact of the diagnosis decision, there are many areas where the

impact of diagnosis decision making systems can be seen and measured. This information helps to build a

return on investment case. Some examples include the following.

Avoiding Litigation - The simplest and starkest impact of diagnosis support is simply an avoided lawsuit. The

saving of just one diagnostic malpractice case is likely to pay for the use of the tool for at least a decade.

Why and How to Improve DIAGNOSIS DECISION MAKING

8

Appropriate Referrals - Within an Accountable Care Organization, getting referrals to be appropriate can have a

key impact on utilization of specialists or secondary care and profitability. About 50% of referrals from primary

care to specialists are for diagnosis reasons, therefore the use of diagnostic decision support tools can be

effective in improving referral appropriateness, especially when referral management centers have not been

shown to be effective. The Kings Fund published a comprehensive paper on this topic in 2010. In addition,

a study examining the impact on referrals showed a significant benefit with 29% of General Practitioners no

longer feeling the need to make a referral after using Isabel.

Appropriate Test Ordering - With thousands of tests now available, it is nearly impossible for clinicians to

remember which tests to order and when they should be used. Over 40% of tests are ordered just for defensive

medicine. A diagnosis decision support system encourages clinicians to take a good history and consider

carefully which tests are necessary to rule in or out the diagnoses they suspect rather than ordering a barrage

of tests to hopefully reveal what the diagnosis could be.

Improved workforce skills - Since healthcare is a knowledge-intensive industry, the skills of an institutions’

workforce have a direct bearing on how well important decisions, such as diagnosis, are made. Logistically, it

is practically impossible to sustainably increase the overall sum of knowledge-based skills through additional

training. The provision of a diagnosis decision support system is an alternative and practical way of improving

skills. One study by Rosalind Franklin Medical School showed how the use of a diagnosis decision support

system led to a 22% increase in diagnostic accuracy among their year four-medical students.

Another study by the same team compared one group of students using diagnosis decision support with

another group consulting with the senior colleague, in this case their resident, and found that the group using

diagnosis decision support became more accurate with their diagnoses and appropriately more confident

while the group who consulted with their resident became marginally less accurate and inappropriately more

confident.

Shorter Length of Stay - Some diagnosis decision support systems offer tables that enable an institution to

make assumptions about the proportion of patients where there might be diagnostic doubt and the additional

bed days or outpatient visits needed if the diagnosis or referral was not made in the optimal time.

Improved Patient Satisfaction - Many hospitals are investing in customer service techniques from other

industries to boost their patient satisfaction. However, many patients view these as gimmicks and not a tool

to improve poor care. Since one of a patient’s key concerns when visiting their doctor is getting the right

The impact of diagnosis decision support systems can be measured by looking at decreases in litigation, increases in appropriate referrals, and improvements in workforce skills.

Why and How to Improve DIAGNOSIS DECISION MAKING

9

diagnosis, using a diagnostic decision support system with patients has been shown to significantly increase

satisfaction, as the patient feels they are being listened to and are reassured that a thorough assessment is

being done.

IMPLEMENTATION OF A DIAGNOSIS DECISION SUPPORT SYSTEM

Diagnosis decision support systems have come of age and are practical for use in busy clinical settings. The

only remaining hurdle is widespread clinical adoption. Here are some guidelines for implementing diagnosis

decision support systems within clinical settings.

Establish Leadership Support - The administration and clinical leadership must be strongly supportive of

diagnosis decision support tools and encourage use in the hospital. Leadership should establish clear goals

such as improvement in diagnosis quality or reduction in inappropriate referrals. The widespread adoption and

regular use of diagnosis tools should always be the means and not the end.

Secure Stakeholder Buy In - Get buy in from key stakeholders on the need to improve diagnoses. Examples of

key stakeholders include risk management, education, nursing, mid-level clinical staff, utilization management,

accountable care champions, patient satisfaction, and patient safety and quality.

The administration and clinical leadership must be strongly supportive of diagnosis decision support tools and encourage use in the hospital.

Find a Champion - Identify a senior clinical champion to drive adoption and awareness. This person needs to

be passionate about improving the quality of care and patient safety and not just have diagnosis as one of

another 20 projects. Ideally this person also needs to be a practicing clinician or perceived by his peers to be “in

the trenches.”

Communicate Early - Before “go live,” key clinicians should be fully aware that the system is coming so that they

are expecting it and, ideally, looking forward to it.

Ensure Ease-of-Use - Access needs to be made very easy with prominent links to the system from the systems

that clinicians use every day. It’s no good if it takes six clicks to find the diagnosis tool. It should be prominent in

the EMR and be easily accessible from desktops and mobile devices.

Plan a Launch Kick Off Event - Have a well-planned launch event to kick start awareness and show that your

institution is serious about improving diagnoses. Senior clinicians should be in attendance and be visibly and

vocally supportive. It doesn’t help with widespread adoption if senior clinicians take the attitude that using

diagnosis decision support systems is beneath them.

Why and How to Improve DIAGNOSIS DECISION MAKING

10

Focus on the Right Audience - The initial focus should be on the right clinical groups such as family practice,

hospitalists, internists, pediatrics, residents and nurses. The specialists will generally be less receptive since they

will not see the need.

Keep it Top of Mind - Awareness campaigns should be continual with use of the tool built into or preferably

mandated into routine events such as daily rounds, daily reports and other educational forums. Clinical

champions should inquire if residents have used the tool and remind them to do so. Build triggers into the EMR

that, for example, ask for a list of all patients who have been there for two days or more with no diagnosis and

then run them through the diagnosis decision support tool. Another obvious trigger could be patients re-

attending for the same complaint.

Promote Successes - Clinical champions should be on the look out for early wins and then publicize them

widely to show the value of diagnosis decision support tools.

Measure the Improvement - Try to measure the changes and improvements. This task can be a challenge

because diagnosis is often not measured. However, a picture of success can be drawn by examining changes in

referrals, test ordering, patient satisfaction scores, and readmissions. You can also survey clinicians and use the

results to show improvement.

Clinical champions should be on the look out for early wins and then publicize them widely to show the value of diagnosis decision support tools.

CONCLUSION

In the era of changing reimbursement, accountable care organizations, bundled payments and pay-for-

outcome, it is critical to determine the patient’s diagnosis as soon as possible and get them on the appropriate

treatment plan or guideline. Minimizing readmissions, unnecessary testing and the costs associated with them

is crucial, and these improvements start with getting the diagnosis correct as soon as possible. Today, in 15-

30% of the cases, there is error or delay in reaching a diagnosis. The future of improved care quality depends on

getting the first step of the care continuum, diagnosis, correct as quickly and efficiently as possible.

Although huge sums of money have been invested in EMR systems including CPOE and other patient safety

related technology, using technology to assist in mitigating diagnosis error has all too often been put to one

side as “too difficult to fix.”

The excuse that diagnosis is “too difficult to fix” is now just that: an excuse.

11

Isabel was started in 1999 after the founder’s daughter, Isabel, suffered a near fatal misdiagnosis. She was three years old and had chicken pox. Her local family doctor and emergency department doctors all missed a secondary infection which turned out to be necrotising fasciitis. Isabel spent three weeks in intensive care and four weeks in a high dependency unit. She survived and is a healthy young woman.

Isabel Healthcare develops and markets the world-leading diagnosis decision support system called Isabel. The Isabel system was originally started as a charity in 2000 but converted to a for-profit business in 2004 to ensure that it had sufficient funding to continue developing its unique system. The founding charity remains one of the largest shareholders in the company.

Isabel has been a proven diagnosis decision support system used by clinicians around the world. Over 30 articles have appeared in peer-reviewed articles covering various aspects of the system. The system was selected by the American Medical Association as the diagnosis tool for its portal. More recently, the British Medical Journal (BMJ) endorsed Isabel as a new joint product was launched incorporating the BMJ’s Best Practice tool.

Today, many high-profile health systems, family practices and individual physicians use Isabel to help improve the quality of care they provide.

Isabel uses a database of over 10,000 diagnoses of which 6,000 are diseases and 4,000 are drugs. This database has been manually built and populated over nearly two decades with knowledge about how each disease presents from a multitude of sources.

About Isabel Healthcare

Why and How to Improve DIAGNOSIS DECISION MAKING

12

References

1. Bordage G. Why did I miss the diagnosis? Some cognitive explanations and educational implications. Acad Med. 1999 Oct;74 (10 Suppl):S138-43. http://www.ncbi.nlm.nih.gov/pubmed/10536619

2. “12 Tips for teaching avoidance of diagnostic error,” Trowbridge, Robert. http://informahealthcare.com/doi/abs/ 10.1080/01421590801965137

3. “Diagnostic error in internal medicine.” Graber ML, Franklin N, Gordon R., Arch Intern Med. 2005 Jul 11; 165(13):1493-9. http:// www.ncbi.nlm.nih.gov/pubmed/16009864

4. “Missing celiac disease in family medicine: the importance of hypothesis generation.” Kostopoulou O, Devereaux-Walsh C, Delaney BC., Med Decis Making. 2009 May-Jun;29(3):282-90. Epub 2009 Mar 6. http://www.ncbi.nlm.nih.gov/pubmed/ 19270107

5. “Diagnosis of difficult cases in primary care.” Kostopoulou O., J Health Serv Res Policy. 2010 Jan;15 Suppl 1:71-4., http:// www.ncbi.nlm.nih.gov/pubmed/20075135

6. “Diagnostic Errors in Primary Care: Lessons Learned”, John W. Ely, MD, Lauris C. Kaldjian, MD, PhD and Donna M. D’Alessandro, MD, http://www.jabfm.org/content/25/1/87.full

7. “Types and origins of diagnostic errors in primary care settings”, Hardeep Singh, MD, MPH; Traber Davis Giardina, MA, MSW; Ashley N. D. Meyer, PhD; Samuel N. Forjuoh, MD, MPH, DrPH; Michael D. Reis, MD; Eric J. Thomas, MD, MPH, http:// jamanetwork.com/journals/jamainternalmedicine/fullarticle/1656540

8. ”SAFETY IS PERSONAL Partnering with Patients and Families for the Safest Care”, The National Patient Safety Foundation’s Lucian Leape Institute, https://c.ymcdn.com/sites/npsf.site-ym.com/resource/resmgr/LLI/Safety_Is_Personal.pdf