Teaching Tips for Diagnostic Studies

Teaching Tips for Diagnostic Studies

Dr. Annette Plüddemann

Nuffield Department of Primary Care Health SciencesCentre for Evidence-Based Medicine

www.oxford.dec.nihr.ac.uk/horizon-scanning

Typically someone with abnormal symptoms consults aphysician, who will obtain a history of their illness and examine them for signs of diseases.

The physician formulates a hypothesis of likely diagnoses and may or may not order further tests to clarify the diagnosis

Diagnosis

• 2/3 legal claims against GPs in UK

• 40,000-80,000 US hospital deaths

from misdiagnosis per year

• Adverse events, negligence cases,

serious disability more likely to be

related to misdiagnosis than drug

errors

• clinical monitoring (such as failure to act upon test results or monitor patients appropriately) – identified as a problem in 31% of preventable deaths

• diagnosis (such as problems with physical examination or failure to seek a specialist opinion) – identified as a problem in 30% of preventable deaths

• drugs or fluid management – identified as a problem in 21% of preventable deaths

Wolf JA, Moreau J, Akilov O, Patton T, English JC, Ho J, Ferris LK.Diagnostic Inaccuracy of Smartphone Applications for Melanoma Detection.JAMA Dermatol. 2013 Jan 16:1-4.

40 yr old woman

non smoker

2 children (10 and 16 yrs)

pre-menopausal

aunt developed breast cancer at 63 yrs

what evidence is needed to inform decision?

1. proportion of women like her who have breast cancer (i.e.

prior to having a mammogram)

2. accuracy of mammograms for detecting cancer

Should she have a mammogram?

Rod Jackson, University of Auckland

What evidence is needed to inform decision?




40 year-old women:

~ 0.1% will have breast cancer

only ~ 1% of 40 year old women with

a positive mammogram have

breast cancer

test accuracy

= 9 (if positive)

0.1

1

5

10

50

95

%

100%

0%

proportion of population

being tested who have condition

40 year-old women:


✖

Fagan nomogram

0.1

1

5

10

50

95

%

100%

0%



40 year-old women:


✖

Fagan nomogram

test accuracy

= 9 (if positive)

✖

100

10

1.5

.05

.005

∞

0

0.1

1

5

10

50

95

%

100%

0%

proportion of population being tested who have

condition

40 year-old women:


✖

test accuracy

= 9 (if positive)

✖

100

10

1.5

.05

.005

∞

0

5

10

1

0.1

50

%

90

0%

100%

✖

only ~ 1% of 40 year old women with a positive mammogram have breast

cancer

proportion of those who test positive /

negative who have the condition

Prevalence/pre-test probability

Likelihood ratio

Post-test probability

0.1

1

5

10

50

95

%

100%

0%40 year-old women:


✖

test accuracy

= 0.1 (if negative)

✖

100

10

1.5

.05

.005

∞

00.01

5

10

50

%

90

0%

100%

1

✖

Only ~ 0.01% of 40 year old women with a

negative mammogram have breast

cancer

Diagnostic strategies and what

tests are used for

How do clinicians make diagnoses?

• Aim: identify types and frequency of diagnostic strategies used in primary care

– 6 GPs collected and recorded strategies used on 300 patients.

(Diagnostic strategies used in primary care. Heneghan, et al,. BMJ 2009. 20;338:b9462009)

• Patient history…examination…differential

diagnosis…final diagnosis

Refinement of the

diagnostic causes

•Restricted Rule Outs

•Stepwise refinement

•Probabilistic reasoning

•Pattern recognition fit

•Clinical Prediction Rule

•Spot diagnoses

•Self-labelling

•Presenting complaint

•Pattern recognition

Initiation of the

diagnosis

Defining the final

diagnosis

•Known Diagnosis

•Further tests ordered

•Test of treatment

•Test of time

•No label

(Heneghan et al, BMJ 2009)

Stage Strategies used

Diagnostic stages & strategies

What are tests used for?

• Increase certainty about presence/absence of disease

• Disease severity

• Monitor clinical course

• Assess prognosis – risk/stage within diagnosis

• Plan treatment e.g., location

• Stall for time!

Bossuyt et al BMJ 2006;332:1089–92

• Replacement – new replaces old

– E.g. CT colonography for barium enema

• Triage – new determines need for old

– E.g. B-natriuretic peptide for echocardiography

• Add-on – new combined with old

– E.g. ECG and myocardial perfusion scan

Roles of new tests

• You are a GP with a busy practice and see a lot of

children with influenza-like symptoms during the winter

• You find it difficult to differentiate influenza from other

RTIs and you’ve heard about a diagnostic test for

influenza called ‘near patient testing’ that can be done in

your office

• You decide to assess the evidence to determine how

accurate this test is to help determine if the test will be

useful in your practice

Clinical scenario

• Patient/ProblemHow would I describe a group of patients similar to mine?

• Index testWhich test am I considering?

• Comparator… or …Reference StandardWhat is the best reference standard to diagnose the target condition?

• Outcome….or….Target conditionWhich condition do I want to rule in or rule out?

Defining the clinical question: PICO or PIRT

• Patient/ProblemChildren (<12 years) presenting with cough and fever thought to be

more than a common cold

• Index testNear patient influenza test

• Comparator… or …Reference StandardLaboratory influenza test

• Outcome….or….Target conditionInfluenza - test positive or negative

Defining the clinical question: PICO or PIRT

Harnden et al. Near patient testing for influenza in children in

primary care: comparison with laboratory test. BMJ 2003;326:480

Critical appraisal of a diagnostic

accuracy study

• Validity of a diagnostic study

• Interpret the results

Diagnostic tests: What you need to knowDiagnostic tests: What you need to know

Series of patients

Index test

Reference standard

Compare the results of the index test with the reference standard,

blinded

Diagnostic Accuracy Studies

Are the results valid?

What are the results?

Will they help me look

after my patients?

•Appropriate spectrum of patients?

•Does everyone get the reference standard?

•Is there an independent, blind or objective

comparison with the reference standard?

Appraising diagnostic studies: 3 easy steps

The Ugly 5….

Biases in Diagnostic Accuracy Studies…

1. Appropriate spectrum of patients?

Ideally, test should be performed on a group of

patients in whom it will be applied in the real

world clinical setting

Spectrum bias:

study uses only highly selected

patients…….perhaps those in

whom you would really suspect

have the diagnosis

Case-control vs consecutive

2. Do all patients have the reference standard?

Ideally all patients get the reference standard test

Verification bias:

only some patients get the reference

standard…..probably the ones in whom

you really suspect have the disease

Series of patients

Index test


blinded

Partial Reference Bias

Ref. Std. A

Series of patients

Index test

Ref. Std. A

Blinded cross-classification

Differential Reference Bias

Ref. Std. B

Series of patients

Index test

Reference standard….. includes parts of Index test


Incorporation Bias

Ideally, the reference standard is independent,

blind and objective

3. Independent, blind or objective comparisonwith the reference standard?

Observer bias:

test is very subjective, or

done by person who knows

something about the

patient or samples

Series of patients

Index test

Reference standard

Unblinded cross-classification

Observer Bias

Lijmer, J. G. et al. JAMA 1999;282:1061-1066

Effect of biases on results

Diagnostic Study Example

1. Spectrum

3. Reference standard

4. Blinding

2. Index test

Teaching tips….

40 yr old woman

non smoker

2 children (10 and 16 yrs)

pre-menopausal

aunt developed breast cancer at 63 yrs

what evidence is needed to inform decision?




Should she have a mammogram?

Start with a clinical scenario

Get tips from other teachers! (Thank you Rod Jackson!)

Rod Jackson, University of Auckland

0.1

1

5

10

50

95

%

100%

0%



40 year-old women:


✖

Fagan nomogram

Introduce the nomogram early –how does evidence help make clinical

decisions

Try something new!

• Validity of a diagnostic study

• Interpret the results

Diagnostic tests: What you need to knowDiagnostic tests: What you need to know

Create a relaxed atmosphere;

Humour

Diagnostic Accuracy Studies

Series of patients

Index test

Reference standard


blinded

Series of patients

Index test

Ref. Std. A


Ref. Std. B

In pictures

The Ugly 5….

Biases in Diagnostic Accuracy Studies…

Think about how to help

people remember

Case-control vs consecutive

Use analogies that are not

medical

Lijmer, J. G. et al. JAMA 1999;282:1061-1066

Effect of biases on results

Show some evidence, but don’t make it too complex

Diagnostic Study Example

Interactive; Use an easy

example!…

If you want to use something which shows potential bias, don’t use a

complex test

The Numbers




after my patients?


•Does everyone get the reference standard?

•Is there an independent, blind or objective

comparison with the gold standard?

Appraising diagnostic tests

•Sensitivity, specificity

•Likelihood ratios

•Positive and Negative Predictive Values

Influenza Study Example

Sensitivity and Specificity

Disease

Test

+ -

+

-

True

positives

False

negatives

True

negatives

False

positives

The 2 by 2 table

Disease

Test

+ -

+

-

Sensitivity = a / a + c

Proportion of people

WITH the disease who

have a positive test result.

a

True

positives

c

False

negatives

The 2 by 2 table: Sensitivity

90

10

Sensitivity = 90/100

So, a test with 90%

sensitivity….means that

the test identifies 90 out

of 100 people WITH the

disease

Disease

Test

+ -

+

-

b

False

positives

d

True

negatives

Specificity = d / b + d


WITHOUT the disease

who have a negative test

result.

The 2 by 2 table: Specificity

75

25

Specificity = 75/100

So, a test with 75%

specificity will be

NEGATIVE in 75 out of

100 people WITHOUT

the disease

The Influenza Example

Disease: Lab Test

Test: Rapid Test

+ -

+

-

27 3

34 93

30

127

1579661

Sensitivity = 27/61 = 0.44 (44%) Specificity = 93/96 = 0.97 (97%)

There were 96 children who did not have influenza… the rapid test was negative in 93 of them

There were 61 children who had influenza…the rapid test was positive in 27 of them

Ruling In and Ruling Out

High Sensitivity

High Specificity

A good test to help in Ruling Out disease

A good test to help in Ruling In disease

High sensitivity means there are very few false negatives – so if the test comes back negative it’s highly unlikely the person has the disease

High specificity means there are very few false positives – so if the test comes back positive it’s highly likely the person has the disease

Disease

Test

+ -

+

-

a

True

positives

c

False

negatives

b

False

positives

d

True

negatives

Specificity = d/b+dSensitivity = a/a+c

Disease: Influenza

Test: Rapid Test

+ -

+

-

27 3

34 93

Sensitivity = 44% Specificity = 97%

SnNOUT

SpPIN

Predictive Values

Disease

Test

+ -

+

-

a

True

positives

c

False

negatives

Positive and Negative Predictive Value

b

False

positives

d

True

negatives

PPV = Proportion of

people with a positive test

who have the disease.

NPV = Proportion of

people with a negative test

who do not have the

disease.

PPV = a / a + b

NPV = d / c + d


Disease: Lab Test

Test: Rapid Test

+ -

+

-

27 3

34 93

30

127

1579661

PPV = 27/30 = 90%

NPV = 93/127 = 73%

Your father went to his doctor and was told that his test for a disease was positive. He is really worried, and comes to ask you for help!

Predictive Value: Natural Frequencies

After doing some reading, you find that for men of his age:

The prevalence of the disease is 30%

The test has sensitivity of 50% and specificity of 90%

“Tell me what’s the chance I have this disease?”

• 100% Likely

• 50% Maybe

• 0% Unlikely

Disease has a prevalence of 30%.

The test has sensitivity of 50% and specificity

of 90%.

Predictive Value

2:001:591:581:571:561:551:541:531:521:511:501:491:481:471:461:451:441:431:421:411:401:391:381:371:361:351:341:331:321:311:301:291:281:271:261:251:241:231:221:211:201:191:181:171:161:151:141:131:121:111:101:091:081:071:061:051:041:031:021:011:000:590:580:570:560:550:540:530:520:510:500:490:480:470:460:450:440:430:420:410:400:390:380:370:360:350:340:330:320:310:300:290:280:270:260:250:240:230:220:210:200:190:180:170:160:150:140:130:120:110:100:090:080:070:060:050:040:030:020:01End


The test has sensitivity of 50% and specificity of 90%.

Given a positive test, what is the probability your dad has the disease

Natural Frequencies

30

70

15

7

100

22 people test positive………

of whom 15 have the disease

So, chance of disease is

15/22 = 68%

Disease +ve

Disease -ve

Testing +ve

Sensitivity = 50%

False positive rate = 10%

Prevalence of 30%, Sensitivity of 50%, Specificity of 90%

4

96

2

9.6

100

11.6 people test positive………



2/11.6 = 17%

Disease +ve

Disease -ve

Testing +ve

Sensitivity = 50%



Positive and Negative Predictive Value

•PPV and NPV are not intrinsic to the test – they also depend on

the prevalence!

•NPV and PPV should only be used if the ratio of the number

of patients in the disease group and the number of patients

in the healthy control group is equivalent to the prevalence

of the diseases in the studied population

•Use Likelihood Ratio - does not depend on prevalence

NOTE

Teaching tips….

Use examples from the news, blogs, things that people come across –relevant to everyone;

Suspense…

Find a simple paper with different measures and the actual numbers

Disease

Test

+ -

+

-

Sensitivity = a / a + c


WITH the disease who

have a positive test result.

a

True

positives

c

False

negatives

The 2 by 2 table: Sensitivity

90

10

Sensitivity = 90/100

So, a test with 90%

sensitivity….means that

the test identifies 90 out

of 100 people WITH the

disease

Explain the concepts in words. Don’t focus on formulas – some like

them (so provide them), but it is more important to understand what the

measures are.


Disease: Lab Test

Test: Rapid Test

+ -

+

-

27 3

34 93

30

127

1579661

Sensitivity = 27/61 = 0.44 (44%) Specificity = 93/96 = 0.97 (97%)

There were 96 children who did not have influenza… the rapid test was negative in 93 of them

There were 61 children who had influenza…the rapid test was positive in 27 of them

Use numbers from a paper; simple

language; It’s more important to

understand what it all means than to know

how to calculate

Ruling In and Ruling Out

High Sensitivity

High Specificity

A good test to help in Ruling Out disease

A good test to help in Ruling In disease

High sensitivity means there are very few false negatives – so if the test comes back negative it’s highly unlikely the person has the disease

High specificity means there are very few false positives – so if the test comes back positive it’s highly likely the person has the disease

Disease

Test

+ -

+

-

a

True

positives

c

False

negatives

b

False

positives

d

True

negatives

Specificity = d/b+dSensitivity = a/a+c

Disease: Influenza

Test: Rapid Test

+ -

+

-

27 3

34 93

Sensitivity = 44% Specificity = 97%

SnNOUT

SpPIN

Acronyms help some…but confuse

others

For beginners this may be a step too

far…

Touch on it…then park it and move

on…

Your father went to his doctor and was told that his test for a disease was positive. He is really worried, and comes to ask you for help!

Predictive Value: Natural Frequencies

After doing some reading, you find that for men of his age:

The prevalence of the disease is 30%

The test has sensitivity of 50% and specificity of 90%

“Tell me what’s the chance I have this disease?”

A simple, common scenario everyone

can relate to

• 100% Likely

• 50% Maybe

• 0% Unlikely


The test has sensitivity of 50% and specificity

of 90%.

Predictive Value

Have a go… interactive… safe

environment

2:001:591:581:571:561:551:541:531:521:511:501:491:481:471:461:451:441:431:421:411:401:391:381:371:361:351:341:331:321:311:301:291:281:271:261:251:241:231:221:211:201:191:181:171:161:151:141:131:121:111:101:091:081:071:061:051:041:031:021:011:000:590:580:570:560:550:540:530:520:510:500:490:480:470:460:450:440:430:420:410:400:390:380:370:360:350:340:330:320:310:300:290:280:270:260:250:240:230:220:210:200:190:180:170:160:150:140:130:120:110:100:090:080:070:060:050:040:030:020:01End



Given a positive test, what is the probability your dad has the disease

Natural Frequencies

Set a time and stick to it!

30

70

15

7

100




15/22 = 68%

Disease +ve

Disease -ve

Testing +ve

Sensitivity = 50%



Simple numbers; reinforces

sensitivity and specificity;

No formulas!

4

96

2

9.6

100

11.6 people test positive………



2/11.6 = 17%

Disease +ve

Disease -ve

Testing +ve

Sensitivity = 50%



Change the prevalence, keep other

numbers the same… learning by doing;Good transition to

likelihood ratios

Likelihood Ratios

Likelihood ratios

LR =Probability of clinical finding in patients with disease

Probability of same finding in patients without disease

Positive likelihood ratio (LR+)

How much more likely is a positive test to be found in a person with the disease than in a person without it?

Likelihood ratios

LR+ = sens/(1-spec)

LR- = (1-sens)/(spec)

Negative likelihood ratio (LR-)

How much more likely is a negative test to be found in a person without the disease than in a person with it?

LR>10 = strong

positive test

result

LR<0.1 = strong

negative test

result

LR=1

No diagnostic

value

What do likelihood ratios mean?

Diagnosis of Appendicitis

McBurney’s point If palpation of the left lower quadrant

of a person's abdomen results in more

pain in the right lower quadrant

Rovsing’s sign

Abdominal pain resulting from

passively extending the thigh of a

patient or asking the patient to actively

flex his thigh at the hip

Psoas sign

Ashdown’s sign

Pain when driving over speed bumps

McGee: Evidence based Physical Diagnosis (Saunders Elsevier)

For Example

(LR+ = 3.4)

(LR- = 0.4)

Speed bump test (Ashdown’s sign): LR+ = 1.4LR- = 0.1

Post test ~20%

?Appendicitis:

McBurney tenderness LR+ = 3.4

Pre test 5%

Fagan nomogram

%

%

Post-test odds = Pre-test odds x Likelihood ratio

Post-test odds for disease after onetest become pre-test odds for next

test etc.

Speed bump test LR- = 0.1

Post test ~0.5%

Teaching tips….

Positive likelihood ratio (LR+)

How much more likely is a positive test to be found in a person with the disease than in a person without it?

Likelihood ratios

LR+ = sens/(1-spec)

LR- = (1-sens)/(spec)

Negative likelihood ratio (LR-)

How much more likely is a negative test to be found in a person without the disease than in a person with it?

Calculation in terms of sensitivity/ specificity is simpler and more useful than formula from the 2x2 table

LR>10 = strong

positive test

result

LR<0.1 = strong

negative test

result

LR=1

No diagnostic

value

What do likelihood ratios mean?

Knowing what LRs mean is more

important than how to calculate

Diagnosis of Appendicitis

McBurney’s point If palpation of the left lower quadrant

of a person's abdomen results in more

pain in the right lower quadrant

Rovsing’s sign

Abdominal pain resulting from

passively extending the thigh of a

patient or asking the patient to actively

flex his thigh at the hip

Psoas sign

Ashdown’s sign

Pain when driving over speed bumps

Simple example…

McGee: Evidence based Physical Diagnosis (Saunders Elsevier)

For Example

(LR+ = 3.4)

(LR- = 0.4)

Speed bump test (Ashdown’s sign): LR+ = 1.4LR- = 0.1

Putting numbers on the scale makes it

clearer

Post test ~20%

?Appendicitis:

McBurney tenderness LR+ = 3.4

Pre test 5%

Fagan nomogram

%

%

Post-test odds = Pre-test odds x Likelihood ratio

Post-test odds for disease after onetest become pre-test odds for next

test etc.

Speed bump test LR- = 0.1

Post test ~0.5%

Bring it back to the Nomogram

What about the news story…?

Dementia Prevalence:1.3% of the entire UK population7% of the UK population over 65

Sensitivity: 90%Specificity: 90%

2:001:591:581:571:561:551:541:531:521:511:501:491:481:471:461:451:441:431:421:411:401:391:381:371:361:351:341:331:321:311:301:291:281:271:261:251:241:231:221:211:201:191:181:171:161:151:141:131:121:111:101:091:081:071:061:051:041:031:021:011:000:590:580:570:560:550:540:530:520:510:500:490:480:470:460:450:440:430:420:410:400:390:380:370:360:350:340:330:320:310:300:290:280:270:260:250:240:230:220:210:200:190:180:170:160:150:140:130:120:110:100:090:080:070:060:050:040:030:020:01End

Dementia has a prevalence of 1%.


Given a positive test, what is the probability the person has “preclinical” Alzheimer’s?

Natural Frequencies

1

99

0.9

9.9

100


of whom 1 has the disease

So, chance of disease is 1/11

= 9%

Disease +ve

Disease -ve

Testing +ve

Sensitivity = 90%



7

93

6

9

100



So, chance of disease is 6/15

= 40%

Disease +ve

Disease -ve

Testing +ve

Sensitivity = 90%


Prevalence of 7%, Sensitivity of 90%, Specificity of 90%Over 65 years:

Pre test 1%

Fagan nomogram

%

%

Alzheimer’s test LR+ = 9Post test ~10%

www.xkcd.com




after my patients?


•Does everyone get the gold standard?

•Is there an independent, blind or

objective comparison with the gold

standard?

Appraising diagnostic tests

•Sensitivity, specificity

•Likelihood ratios

•Positive and Negative Predictive Values

•Can I do the test in my setting?

•Do results apply to the mix of patients I see?

•Will the result change my management?

•Costs to patient/health service?

• Reproducibility of the test and interpretation in my setting

• Do results apply to the mix of patients I see?

• Will the results change my management?

• Impact on outcomes that are important to patients?

• Where does the test fit into the diagnostic strategy?

• Costs to patient/health service?

Will the test apply in my setting?




after my patients?

What is the ONE thing I need to remember from today?

Don’t believe everything you are told,

Ask for the Evidence!

Teaching tips….

Bring it back to the news story, reinforce…

• Reproducibility of the test and interpretation in my setting

• Do results apply to the mix of patients I see?

• Will the results change my management?

• Impact on outcomes that are important to patients?

• Where does the test fit into the diagnostic strategy?

• Costs to patient/health service?

Will the test apply in my setting?

There is more to diagnostics than

accuracy!




after my patients?

What is the ONE thing I need to remember from today?

Don’t believe everything you are told,

Ask for the Evidence!

Take home message!

The Diagnostic Process. John Balla. Cambridge Univ. Press

Diagnostic Tests Toolkit. Thompson & Van den Bruel. Wiley-Blackwell.

Evidence base of Clinical Diagnosis. Knottnerus & Buntinx. Wiley-Blackwell

Evidence based Physical Diagnosis. Steven McGee. Saunders

Evidence-based Diagnosis.Newman & Kohn. Cambridge Univ. Press

Useful books on diagnostics

• Bossuyt. Additional patient outcomes and pathways in evaluations of testing.

Med Decis Making 2009

• Heneghan et al. Diagnostic strategies used in primary care. BMJ 2009

• Ferrante di Ruffano. Assessing the value of diagnostic tests: a framework for

designing and evaluating trials. BMJ 2012

• Mallett et al. Interpreting diagnostic accuracy studies for patient care. BMJ 2012

• Bossuyt et al. STARD initiative. Ann Int Med 2003

• Lord et al. Using priniciples of RCT design to guide test evaluation. Med Decis

Making 2009

• Rutjes et al. Evidence of bias and variation in diagnostic accuracy studies.

CMAJ 2006

• Lijmer et al. Proposals for phased evaluation of medical tests. Med Decis

Making 2009

• Whiting et al. QUADAS-2: revised tool for quality assessment of diagnostic

accuracy studies. Ann Int Med 2011

• Halligan S, Altman DG, Mallett S. Disadvantages of using the area under the

receiver operating characteristic curve to assess imaging tests: A discussion and

proposal for an alternative approach. Eur Radiol. 2015

Useful journal articles on diagnostics

Date post:	19-Feb-2022
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Teaching Tips for Diagnostic Studies

Documents