7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
1/43
Critical Appraisal of
Articles About
Diagnostic Tests
Rani Gereige, M.D., MPHAssociate Professor
University of South Florida
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
2/43
Learning Objectives Test validity What is it? and how to ensure it?
Putting diagnostic tests to the test: Critical appraisal ofstudies about a diagnostic test
Be familiar with the concepts of
Treatment and Test thresholds Pre-test and post-test probabilities and odds
Likelihood Ratios (LRs)
Evaluate a VALID evidence using The 2x2 table
Calculations and interpretation of Sensitivity, Specificity, PPV,NPV, pre and post test probabilities and odds, LR(+) and LR(-)
Apply the evidence
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
3/43
QuestionsA patient presents to us with a chief
complaint
Why do we order tests?
What tests to order? Based on what?What do we hope to achieve as we get the
result of the test?
What if there are multiple tests that arerelated to this complaint?
What if we are considering 6 or 7 possiblediagnoses that might explain this chiefcomplaint?
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
4/43
Hurricane Fever
What ifthere is a new test that claims to
accurately pinpoint hurricanes location of the
attack??What would you want to know about this test?
Anything you want to know about the hurricane itself?
What about if the hurricane is right at your doorstep?
What if this new test can only be done at anelevation of 15000 ft above sea level and you
live in Florida?
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
5/43
Diagnostic Test
Validity Definition
The validity of a diagnostic test is the
extent to which the resultsrepresent an unbiased estimate of
the underlying truth
(it is testing what it supposed to test)
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
6/43
Validity versus Relevance
Validity Is it true?
Was the study appropriatelyrandomized, blinded?
Did they use appropriate goldstandard?
Was the population similarenough to your own?(generalizable)
Was follow-up adequate?
Recommended first by thetext
Relevance Does it matter to my
patients/population?
Is the test feasible for my patients?
Will it affect something thatMATTERS to patients? (Morbidity,mortality, quality of life)
Will it change my practice?
Most clinicians look atrelevance first
Easy and quick to find(abstract)
If does not matter do not botherwith validity testing
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
7/43
How to Use an Article about a
Diagnostic Test??Three main questions:
Is this evidence about the accuracy of a
diagnostic test valid? Does this (valid) evidence demonstrate an
important ability of this test to accuratelydistinguish patients who do or do not have a
specific disorder? i.e. Magnitude Can I apply this valid, important diagnostic test
to a specific patient?
If answer isNOStop
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
8/43
I- Is this evidence about the accuracy
of a diagnostic test VALID?
Measurement
Representative
Ascertainment
Reproducibility
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
9/43
I- Is this evidence about the accuracy
of a diagnostic test VALID? Measurement: Was there an independent, blind
comparison with a reference Gold Standard? Patients should have undergone both the test and the
reference (gold standard) Confirmation by surgicalfindings, biopsy, autopsy, long term follow-up
Investigator should not know the result of goldstandard
Limitations:
Gold standard is not always possible (psych disorders) Bias cannot always be avoided
Some gold standards are for research only
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
10/43
I- Is this evidence about the accuracy
of a diagnostic test VALID? Representative: Was the diagnostic test
evaluated in an appropriate spectrum of patients(like those in who we would use it in practice)?Was the spectrum of disease appropriate?
Florid cases vs asymptomatic volunteers
Disease severity: Mild, severe, early or late cases, bothtreated and untreated
Apply the test to patients with different disorder commonlyconfused with the target disorder
Was the spectrum of patients appropriate? Similar ages, sexes, co-morbidities
Enough patient numbers?
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
11/43
I- Is this evidence about the accuracy
of a diagnostic test VALID?Ascertainment: Was the reference
standard ascertained regardless of the
diagnostic test result? Important to confirm or refute the test on ALL
patients, not just those with abnormal test
May use different method of gold standard toavoid harming patients (Surgery vs close
follow-up)
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
12/43
I- Is this evidence about the accuracy
of a diagnostic test VALID? Reproducibility: Was the test (or cluster of
tests) validated in a second, independent
group of patients?Diagnostic tests are predictors, not explainers
of the diagnosis
Same results of the test in a separateindependent group will reassure us on the
accuracy
Not always done initially
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
13/43
Eligible Patients
n =
Excluded Patients
Reasons n =
Index test
n =
Positive
n =
Negative
n =
Inconclusive
Missing
n =
Missing
n =
Missing
n =
Reference Standard Reference StandardReference Standard
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
14/43
How to Use an Article about a
Diagnostic Test??Three main questions:
Is this evidence about the accuracy of a
diagnostic test valid? Does this (valid) evidence demonstrate an
important ability of this test to accuratelydistinguish patients who do or do not have a
specific disorder? i.e. Magnitude Can I apply this valid, important diagnostic test
to a specific patient?
If answer isNOStop
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
15/43
Probability versus Odds
Odds of disease does not equal
probability of disease
Example: 10 patients 3 have anemia and7 do not
Probability of having anemia is 3/10 = 30% =
Prevalence = Pretest probabilityOdds of having anemia are 3:7 or 0.43
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
16/43
Probabilities and Odds
Can be calculated back and forth Odds of a:b will give probability of a/(a+b)
Probability of X% will give Odds of X/(100-X)
You start withPre-test probability (Prevalence)
Pre-test Odds
Post-test Odds
Post-test probability
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
17/43
Pre-test Probability and Odds
Pre-test probability = Prevalence = The
proportion of people with the target
disorder in the population at risk at aspecific time
Pre-test Odds = The odds that the patient
has the target disorder before the test iscarried = Pre-test Probability/[1- Pre-test
Probability]
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
18/43
Post-test Probability and
Odds Post-test probability = The proportion of
patients with that particular test result who
have the target disorder = Post-test odds/[1+ Post-test Odds]
Post-test Odds = The odds that the patient
has the target disorder after the test iscarried out
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
19/43
Sources of Pre-test Probability Clinical experience
Biased by frequency of diagnosis, last patient, fear ofmissing a diagnosis
Easily and quickly accessed
Use with caution
Not for young careers Regional/ national prevalence statistics on
disorder frequency (as good as accuracy ofdiagnosis)
Local/ regional/ national practice databases
Calculated from the study being appraised
Track research report of a study devoted to
pretest probability documentation (Best - if welldone)
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
20/43
Inputs Into Pre-Test Probability
The frequency of the disease in
populations or groups of individuals similar
to a particular patient Risk factor exposure of the individual
Pattern of symptoms presented by the
patient
Use of a prior screening test
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
21/43
Test and Treatment Thresholds
Test Threshold probabilities below which a
clinician would dismiss a diagnosis and order
no further tests
Treatment Threshold probabilities above
which a clinician would consider the diagnosis
confirmed, and would stop testing
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
22/43
Test accuracy
Ability of a test to change our minds from
what we thought before the test (pre-test
probability) to what we think afterwards
(post-test probability)
Diagnostic tests that produce big changes
from pre-test to post-test probability, are
important and likely to useful in practice
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
23/43
Clinical Implications
When the probability of the target disorder
falls between the test and treatment
thresholds, further testing is mandated
Once test and treatment thresholds are
determined, the post-test probabilities
have direct treatment implications
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
24/43
Evaluating the Evidence Diagnostic Test
Construct the 2x2 table
Sensitivity = a /a+c = P(+/D)= P(TP amongdiseased)=PID
Specificity = d /b+d = P(-/D)= P(TN among non diseased)
PPV = a /a+b = P(D/+)= P(TP among all Positives)
NPV= d /c+d = P(D/-)= P(TN among all Negatives)
SnNout = Result of a test with high Sensitivity, whenNegative, rules out the diagnosis
SpPin = Result of a test with high Specificity, when
Positive, rules in the diagnosis
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
25/43
More Definitions
Sensitivity = PID = Positive in Disease
Specificity = NIH = Negative in Health
Discriminant ability of a test =
(sensitivity+specificity)/2
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
26/43
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
27/43
Likelihood Ratios
Indicate by how much a given diagnostictest result will raise or lower the pretest
probability of the target disorderLR = 1 means the post-test probability is
exactly the same as the pre-test probability
LR > 1 increases the probability that the target
disorder is presentLR < 1 decreases the probability that the
target disorder is present
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
28/43
Guide to the Significance of LRs
LR > 10 or < 0.1 generate large and often conclusive changes frompre-test to post-test probability
LR = 5 - 10 or 0.1 - 0.2 generate moderate shifts pre-test to post-test
LR = 2 5 or 0.5 0.2 generate small, but sometimes importantchanges in probability
LR = 1 2 or 0.5 1 are rarely important shifts
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
29/43
How to Use LRs?? Bayes
Theorem Used to calculate Post-test probability
Nomogram
(Textbook card 3A) Table
LR calculator
PDAMed Calc
LR calculator: http://araw.mede.uic.edu/cgi-alansz/testcalc.pl
Pre-test probability (Prevalence) - X
Pre-test Odds: Y = X/(100-X)
Post-test Odds: Z = Y x LR
Post-test probability = Z/(Z+1)
LR
http://araw.mede.uic.edu/cgi-alansz/testcalc.plhttp://araw.mede.uic.edu/cgi-alansz/testcalc.plhttp://araw.mede.uic.edu/cgi-alansz/testcalc.plhttp://araw.mede.uic.edu/cgi-alansz/testcalc.pl7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
30/43
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
31/43
LR Nomogram & Table
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
32/43
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
33/43
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
34/43
The Ferritin Systematic
Review DiseasePresent
Disease
Absent
Totals
Ferri t in
65 mmo l/L
78
(c)
1500
(d)
1578
(c+d)
Totals 809
(a+c)
1770
(b+d)
2579
(a+b+c+d)
Sensitivity = 731/809 = 90%
Specificity = 1500/1770 = 85%
PPV = 731/1001 = 73%
NPV = 1500/1578 = 95%
LR(+) = Sens/(1-Spec) = 90%/15% = 6
LR(-) = (1-Sens)/Spec = 10%/85% =
0.12
Pre-test Probability = Prevalence =
(a+c)/(a+b+c+d) = 809/2579 = 31%
What do you think of these LRs??
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
35/43
Calculating Post-Test
Probability Pre-test Odds =Prevalence/(1-Prevalence) =
31%/69% = 0.45
Post-test odds=pre-test oddsx LR
For (+) = 0.45x6=2.7
For (-) = 0.45x0.12=0.054
Post-test probability= Post-
test odds/(post-test odds+1)
For (+) = 2.7/3.7=72.99%
For (-) = 0.054/1.054=5.12%
(+)
(-)
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
36/43
What Conclusions Can You Draw
About Ferritin? High sensitivity and specificity
LR(+) of 6 means that using Ferritin level allowsto increase the probability of iron deficiency
anemia from 31% (or 50%) to 73% (or 86%) forlevels < 65 mmol/L
LR(-) of 0.12 means that using Ferritin levelallows to decrease the probability of iron
deficiency anemia from 31% (or 50%) to 5% (or11%) for levels > 65 mmol/L
This is known as sensitivity analysis
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
37/43
Multilevel Likelihood Ratios
Provide superior information to sensitivity and Specificity
Allow detailed and specific clinical conclusions
809 1770
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
38/43
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
39/43
How to Use an Article about a
Diagnostic Test??Three main questions:
Is this evidence about the accuracy of adiagnostic test valid?
Does this (valid) evidence demonstrate animportant ability of this test to accuratelydistinguish patients who do or do not have aspecific disorder? i.e. Magnitude
Can I apply this valid, important diagnostic testto a specific patient?
If answer isNOStop
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
40/43
Applying the Valid Evidence to
YOUR PatientThree Questions Q1. Is the diagnostic test available,
affordable, accurate and precise in
our setting?Requires knowledge of your clinical
capacity
Do you have the test? Personnel?
Cost? Insurance coverage?
What is your patient population? And status
of disease presentation?
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
41/43
Q2. Can we generate a clinically sensibleestimate of our patients pre-test probability? From personal experience (H&P, memory),
prevalence statistics (texts, government), practicedatabases (Hospitals, med records), primarystudies (validity testing)
Are the study patients similar to our own?
Is it unlikely that the disease possibilities or
probabilities have changed since this evidence wasgathered? (e.g. study from 1996 on AIDSdiagnosis)
Applying the Valid Evidence to YOUR
PatientThree Questions
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
42/43
Q3. Will the resulting post-test probabilitiesaffect our management and help our patient? Could it move us across a test-treatment threshold?
Depends on the disease and the relative risks of treatment vsno treatment
May move us from one test to another
Post-test odds after one test become the pre-test odds for thenext
Would our patient be a willing partner in carrying itout? (This is critical)
Would the consequences of the test help our patientreach his or her goals in all this?
Applying the Valid Evidence to YOUR
PatientThree Questions
7/30/2019 Critical Appraisal of Articles About Diagnostic Tests
43/43
Look at things CRITICALLY