Appraising A Diagnostic Test

Clinical Epidemiology and Evidence-based Medicine Unit

FKUI-RSCM

What is diagnosis ?Increase certainty about presence/absence of diseaseDisease severityMonitor clinical courseAssess prognosis – risk/stage within diagnosisPlan treatment e.g., location Stalling for time!

Knottnerus, BMJ 2002

Key Concept Pre-test Probability– The probability of the target

condition being present before the results of a diagnostic test are available.

Post-test Probability– The probability of the target

condition being present after the results of a diagnostic test are available.

Key ConceptPre-test Probability– The probability of the target condition

being present before the results of a diagnostic test are available.

Post-test Probability– The probability of the target condition

being present after the results of a diagnostic test are available.

Basic Principles (1)Ideal diagnostic tests – right answers:(+) results in everyone with the

disease and( - ) results in everyone elseUsual clinical practice:–The test be studied in the same

way it would be used in the clinical setting

Observational study, and consists of:–Predictor variable (test result)–Outcome variable (presence /

absence of the disease)

Basic Principles (2)Sensitivity, specificityPrevalence, prior probability, predictive valuesLikelihood ratiosDichotomous scale, cutoff points (continuous scale)Positive (true and false), negative (true & false)ROC (receiver operator characteristic) curve

General structure : 2 X 2 table

Target disorderPositive(disease)

Target disorderNegative (normal)

PredictorTest

positive

True positiveTPa

False positiveFPb

PredictorTest

negative

False negative

FNc

True negativeTNd

Disease(+)

Disease(-) Total

Test (+) True posa

False posb

a+b

Test (-)False neg

cTrue neg

d c+d

Total a+c b+d a+b+c+d a+ca+b+c+dPrevalence Pretest probability

Sensitivity

The proportion of people who truly have a designated disorder who are so identified by the test.Sensitive tests have few false negatives. When a test with a high Sensitivity is Negative, it effectively rules out the diagnosis of disease. SnNout

Specificity

The proportion of people who are truly free of a designated disorder who are so identified by the test. Specific tests have few false positivesWhen a test is highly specific, a positive result can rule in the diagnosis. SpPin

Disease(+)

Disease(-) Totals

Test (+) a b a+b

Test (-) c d c+d

Totals a+c b+d a+b+c+d a/a+c d/b+d

Probability of positive test result in patients with the disease

Probability of negative test result

in patients without the disease

Sensitivity Specificity

SnNOut SpPIn

SnNOutThe sensitivity of dyspnea on exertion for the diagnosis of CHF is 100% (41/(41+0)), and the specificity 17% (35/(183+35)). If DOE, it is very unlikely that they have CHF (0 out of 41 patients with CHF did not have this symptom)."SnNOut", which is taken from the phrase: "Sensitive test when Negative rules Out disease".

SpPinConversely, a very specific test, when positive, rules in disease. "SpPIn"! 

The sensitivity of gallop for CHF is only 24% (10/41), but the specificity is 99% (215/218).  Thus, if a patient has a gallop murmur, they probably have CHF (10 out of 13).

Iron deficiency anemiaTotals

Present Absent

Diagnostic

test result (Serum ferritin)

(+)<65

mmol/L731

a270b

1001 a+b

(-)>65

mmol/L78c

1500d

1578 c+d

Totals 809 a+c

1770 b+d

2579

a+b+c+d

Sensitivity=a/a+c=90%Specificity =d/b+d=85%

Pos predictive value=a/a+b=73%Neg predictive value=d/c+d=95%

LR + = sn/(1-sp)=90/15=6

Prevalence=(a+c)/(a+b+c+d)= 32%

Posttest odd =Pretest odd xLikelihood Ratio

PredictorOutcome

Odds = ratio of two probabilities Odds = p/1-p Probability = odds/1+odds

Likelihood ratio (+):Prob (+) result in people with the

diseaseProb (+) result in people w/out the

disease

Pretest Odds X LR = Posttest Odds

Key Concept Likelihood Ratio– Relative likelihood that a given test

would be expected in a patient with (as opposed to one without) a disorder of interest.

probability of the test result in pts without disease

LR=probability of a test result in pts with disease

Likelihood ratios (LR) General Rules of Thumb

LR > 10 or < 0.1 produce large changes in pre-test probabilityLR of 5 to 10 or 0.1 to 0.2 produce moderate changesLR of 1 to 2 or 0.5 to 1 produce small changes in pre-test probability

Test

CA B

pretest probability

0 .10 .20 .30 .40 .50 .60 .70 .80 .90 1

do not test

do nottreat

do not test

get on with treatment

Likelihood ratio

posttest probability

Test

+ = Sn/(1-Sp)(1-Sn)/Sp= -

PreTest odds x LR

pretest probability

Serum ferritin (mmol/L)

Iron def positive Iron def negative Likelih

ood ratio

Diagnostic impactNo % No %

Very positive <15 474 59(474/809) 20 1.1

(20/1770) 52 Rule inSpPin

Moderately positive 15-34 175 22

(175/809) 79 4.5(79/1770) 4.8 Intermed

High

Neutral 35-64 82 10(82/809) 171 10

(171/1770) 1 Indeter mine

Moderately negative 65-94 30 3.7

(30/809) 168 9.5(168/1770) 0.39 Intermed

low

Extremely negative >95 48 5.9

(48/809) 1332 75(1332/1770) 0.08 Rule out

SnNout

809 100(809/809) 1770 100

(1770/1770)

The usefulness of five levels of a diagnostic test result

Pretest probability

Likelihood ratio

Posttest probability

T4 value Hypothyroid

Euthyroid

5 or less 18 15.1 – 7.0 7 177.1 – 9.0 4 369 or more 3 39Totals 32 93

T4 value Hypothyroid

Euthyroid

≤ 5 18 1

> 5 14 92

Totals 32 93

T4 value Hypothyroid

Euthyroid

≤ 7 25 18> 7 7 75

Totals 32 93

T4 value Hypothyroid

Euthyroid

≤ 9 29 54

> 9 3 39

Totals 32 93

Cutoff point

Sens Spec

5 0.56 0.997 0.78 0.819 0.91 0.42

T4 level in suspected hypo-thyroidism in children

For tests / predictors with continuous values result , cutoff points should be determine to choose the best value to use in distinguishing those with and without the target disorder

Cutoff point

Sens Spec

5 0.56 0.997 0.78 0.819 0.91 0.42

Cutoff point

SensTP

1-SpecFP

5 0.56 0.017 0.78 0.199 0.91 0.58

Accuracy of the testThe accuracy of the test depends on how well the test separates the group being tested into those with and without the disease in questionAccuracy is measured by the area under the ROC curve. An area of 1 represents a perfect test; an area of 0.5 represents a worthless test (AUC)

• 0.90-1.00 = excellent (A)

• 0.80-0.90 = good (B)

• 0.70-0.80 = fair (C) • 0.60-0.70 = poor (D) • 0.50-0.60 = fail (F)

An ROC curve demonstrates several things:

It shows the tradeoff between sensitivity and specificity

• any increase in sensitivity will be accompanied by a decrease in specificity

The closer the curve follows the left-hand border and then the top border of the ROC space, the more accurate the test. The closer the curve comes to the 45-degree diagonal of the ROC space, the less accurate the test. The slope of the tangent line at a cutoff point gives the likelihood ratio (LR) for that value of the test.

Appraising DxTestIs the evidence valid? (V)

• Was there an independent, blinded comparison with a gold standard?

• Was the test evaluated in an appropriate spectrum of patients?

• Was the reference standard applied regardless of the test result?

• Was the test validated in a second, independent group of patients?

Can I trust the accuracy data?

RAMMbo

Recruitment: Was an appropriate spectrum of patients included? (Spectrum Bias)Maintainence: All patients subjected to a Gold Standard? (Verification Bias)Measurements: Was there an independent, blind or objective comparison with a Gold standard? (Observer Bias; Differential

Reference Bias)

Guyatt. JAMA, 1993

Critical AppraisalIs this valid test important? (I)

• Distinguish between patients with and those without the disease

• Two by two tables• Sensitivity and Specificity

–SnNOut–SpPIn

• ROC curves• Likelihood Ratios

Critical Appraisal

Can I apply this test to my patient (A)

• Similarity to our patient• Is it available• Is it affordable• Is it accurate• Is it precise

Critical AppraisalCan I apply this test to my patient?

• Can I generate a sensible pre-test probability–Personal experience–Practice database–Assume prevalence in the study

Critical AppraisalDiagnosis– Can I apply this test to a specific patient

• Will the post-test probability affect management

– Movement above treatment threshold– Patient willing to undergo testing

Thank You