STATISTICAL EVALUATION STATISTICAL EVALUATION OF DIAGNOSTIC TESTSOF DIAGNOSTIC TESTS
Describing the performance of a new diagnostic testDescribing the performance of a new diagnostic test
Physicians are often faced with the task of evaluation
the merit of a new diagnostic test. An adequate critical
appraisal of a new test requires a working knowledge
of the properties of diagnostic tests and the
mathematical relationships between them.
The gold standard test: Assessing a new diagnostic
test begins with the identification of a group of
patients known to have the disorder of interest, using
an accepted reference test known as the gold
standard.
Limitations:
1) The gold standard is often the most risky,
technically difficult, expensive, or impractical of
available diagnostic options.
2) For some conditions, no gold standard is
available.
The basic idea of diagnostic test interpretation is to
calculate the probability a patient has a disease under
consideration given a certain test result. A 2 by 2 table
can be used for this purpose. Be sure to label the table
with the test results on the left side and the disease
status on top as shown here:
Test Disease
Present Absent
Positive True Positive False Positive
Negative False Negative True Negative
The sensitivity of a diagnostic test is the
probability that a diseased individual will have a
positive test result. Sensitivity is the true positive
rate (TPR) of the test.
Sensitivity = P(T+|D+)=TPR
= TP / (TP+FN)
diseased all
test positive withdiseased
The specificity of a diagnostic test is the
probability that a disease-free individual will have a
negative test result. Specificity is the true negative
rate (TNR) of the test.
Specificity=P(T-|D-) = TNR
=TN / (TN + FP).
free-disease all
test negative withfree-disease
False-positive rate: The likelihood that a nondiseased patient has an abnormal test result.
FPR = P(T+|D-)=
= FP / (FP+TN)
free-diseased all
test positive withfree-disease
False-negative rate: The likelihood that a diseased patient has a normal test result.
FNR = P(T-|D+)=
= FN / (FN+TP)
diseased all
test negative withdiseased
Pretest Probability is the estimated likelihood of
disease before the test is done.
It is the same thing as prior probability and is often
estimated. If a defined population of patients is being
evaluated, the pretest probability is equal to the
prevalence of disease in the population. It is the
proportion of total patients who have the disease.
P(D+) = (TP+FN) / (TP+FP+TN+FN)
Sensitivity and specificity describe how well the test
discriminates between patients with and without
disease. They address a different question than we
want answered when evaluating a patient, however.
What we usually want to know is: given a certain test
result, what is the probability of disease? This is the
predictive value of the test.
Predictive value of a positive test is the proportion
of patients with positive tests who have disease.
PVP=P(D+|T+) = TP / (TP+FP)
This is the same thing as posttest probability of
disease given a positive test. It measures how well
the test rules in disease.
Predictive value of a negative test is the
proportion of patients with negative tests who do not
have disease. In probability notation:
PVN = P(D-|T-) = TN / (TN+FN)
It measures how well the test rules out disease. This
is posttest probability of non-disease given a
negative test.
Evaluating a 2 by 2 table is simple if you are methodical in your approach.
Test Disease
Present Absent
Positive TP FP Total positive
Negative FN TN Total negative
Total with disease
Total with- out disease
Grand total
Bayes’ Rule MethodBayes’ Rule MethodBayes’ rule is a mathematical formula that may be used as an alternative to the back calculation method for obtaining unknown conditional probabilities such as PVP or PVN from known conditional probabilities such as sensitivity and specificity.
FPRDpTPRDp
TPRDpTDPPVP
))(1()(
)()(
FNRDpTNRDp
TNRDpTDPPVN
))(1()(
)()(
)()()()(
)()()(
ABPAPABPAP
ABPAPBAP
General form of Bayes’ rule is
Using Bayes’ rule, PVP and PVN are defined as
Example The following table summarizes results of a study to evaluate the dexamethasone suppression test (DST) as a diagnostic test for major depression. The study compared results on the DST to those obtained using the gold standard procedure (routine psychiatric assessment and structured interview) in 368 psychiatric patients.
1. What is the prevalence of major depression in the study group?
2. For the DST, determine
a-Sensitivity and specificity
b-False positive rate (FPR) and false negative rate (FNR)
c-Predictive value positive (PVP) and predictive value negative (PVN)
DST
Result
Depression Total
+ -
+ 84 5 89
- 131 148 279
Total 215 153 368
Sensitivity = P(T+|D+)=TPR=TP/(TP+FN)=84/215=0.391
Specificity=P(T-|D-)=TNR=TN / (TN + FP)=148/153=0.967
FPR = P(T+|D-)=FP/(FP+TN)=5/153=0.033
FNR = P(T-|D+)=FN/(FN+TP)=131/215=0.609
PVN = P(D-|T-) = TN / (TN+FN)=148/279=0.53
PVP=P(D+|T+) = TP / (TP+FP)=84/89=0.944
P(D+) =215/368 =0.584
FNR=1-Sensitivity=1-0.391=0.609
FPR=1-Specificity=1-0.967=0.033
ROC (Receiver Operating Characteristic ) CURVEROC (Receiver Operating Characteristic ) CURVE
The ROC Curve is a graphic representation of the
relationship between sensitivity and specificity for a
diagnostic test. It provides a simple tool for applying the
predictive value method to the choice of a positivity
criterion.
ROC Curve is constructed by plottting the true positive rate
(sensitivity) against the false positive rate (1-specificty) for
several choices of the positivity criterion.
ROC Curve
Diagonal segments are produced by ties.
1 - Specificity
1,00,75,50,250,00
Sensitivity
1,00
,75
,50
,25
0,00
Disease-free Diseased
Test negative Test positive
TP
FP
FN
TN
2 1
1
11
ix
z
2
22
ix
z
Positivity criterion
xi
Example: One of the parameters which are evaluated for the diagnosis of CHD, is the value of “HDL/Total Cholesterol”. Consider a population consisting of 67 patients with CHD, 93 patients without CHD. The result of HDL/Total Cholesterol values of these two groups of patients are as follows.
CHD+
Hdl/Total Cholestrol
CHD-
Hdl/Total Cholestrol
0,29
0,26
0,39
0,16
.
.
.
0,25
0,36
0,30
0,20
.
.
.
To construct the ROC Curve, we should find sensitivity and specificity for each cut off point. We have two alternatives to find these characteristics.
• Cross tables
• Normal Curve
Descriptive Statistics
HDL/Total Cholestrol
,2926 ,066 ,16 ,52,2301 ,048 ,06 ,34
GROUPCHD-CHD+
Mean SD Min Max
If HDL/Total Cholestrol is less than or equal to 0,26, we classify this group into diseased.
64 15 79
68,8% 22,4% 49,4%
29 52 81
31,2% 77,6% 50,6%
93 67 160
100,0% 100,0% 100,0%
Count
Count
Count
0,26>
0,26<=
RATIO
Total
- +
CHD
Total
SensitivitySpecificity
Best cutoff point
Cutoff TPR FPR0,000 0,000 0,0000,093 0,015 0,0000,129 0,030 0,0000,142 0,045 0,0000,156 0,060 0,0000,158 0,075 0,0000,162 0,075 0,0110,168 0,104 0,0110,171 0,119 0,0110,173 0,119 0,0220,175 0,119 0,032
. . .
. . .
. . .0.26 0.78 0.31
. . .
. . .0,393 1,000 0,9350,402 1,000 0,9460,407 1,000 0,9570,420 1,000 0,9680,446 1,000 0,9780,493 1,000 0,9891,000 1,000 1,000
92 59 151
98,9% 88% 94%
1 8 9
1,1% 12% 5,6%
93 67 160
100% 100% 100%
0,171<
0,171>=
RATIO
Total
- +
CHD
Total
Let cutoff=0,171
ROC Curve
1 - Seçicilik
1,0,9,8,7,6,5,4,3,2,10,0
Se
ns
itiv
ity
1,0
,9
,8
,7
,6
,5
,4
,3
,2
,1
0,0
1-Specificity
Cutoff=0.26
TPR=0.78
FPR=0.31
TNR=0.69
FNR=0.22
Area Under the Curve
Test Result Variable(s): ORAN
,778 ,036 ,000 ,708 ,849Area Std. Error
aAsymptotic
Sig.b
Lower Bound Upper Bound
Asymptotic 95% ConfidenceInterval
The test result variable(s): ORAN has at least one tie between thepositive actual state group and the negative actual state group. Statisticsmay be biased.
Under the nonparametric assumptiona.
Null hypothesis: true area = 0.5b.
If Patients with CHD and without CHD are normally distributed, we can easily find sensitivity and specificity from the area under these normal curves. Sensitivity and specificity are calculated for each different cotoff points
CHD+ CHD-
Cutoff=0,28
TP
FP
FN
TN
0,23 0,29
ZCHD+=(0.28-0.23)/0.048=1.04
TPR=0.5+0.3508=0.8508
FNR=1-TPR=0.1492
If we take cut off point=0.28, the characteristics of test are:
ZCHD-=(0.28-0,29)/0.066=-0.15
TNR=0.5+0.0596=0.5596
FPR=1-TNR=0.4404
Cutoff=0,28
0,23 0,29
CHD+ CHD-
Cutoff TPR FNR TNR FPR0,10 0,00 1,00 1,00 0,000,15 0,05 0,95 0,98 0,020,20 0,27 0,73 0,91 0,090,25 0,66 0,34 0,73 0,270,28 0,85 0,15 0,56 0,440,30 0,93 0,07 0,44 0,560,35 0,99 0,01 0,18 0,820,45 1,00 0,00 0,00 1,00
0,000,100,200,300,400,500,600,700,800,901,00
0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1
FPR
TP
R