Brandy M. Ringham,1 Todd A. Alonzo,2 John T. Brinton,1 Aarti Munjal,1 Keith E. Muller,3 Deborah H. Glueck1
1Department of Biostatistics and Informatics, University of Colorado Denver
2Department of Preventive Medicine, University of Southern California 3Department of Health Outcomes and Policy, University of Florida
Reducing Decision Errors in the Paired Comparison of the Diagnostic Accuracy of
Continuous Screening Tests
The project described was supported by Award Number NCI 1R03CA136048-01A1 from the National Cancer Institute, and by Award Number NIDCR 3R01DE020832-01A1S1 from the National Institute of Dental and Craniofacial Research. The content is solely the responsibility of the authors, and does not necessarily represent the official views of the National Cancer Institute, the National Institute of Dental and Craniofacial Research nor the National Institutes of Health.
Acknowledgements
2
Outline
3
• Case Study • Cancer Screening Trial Design • Cancer Screening Analysis
• Bias Correction Algorithm • Evaluation Studies • Oral Cancer Screening Demonstration
Science
Statistics
Oral Cancer Screening Case Study
4
No visible lesion Dark region confirmed to be carcinoma in situ
VISIBLE LIGHT AUTOFLUORESCENCE
Confirmed disease?
Paired Cancer Screening Trial
5
Screen positive on either test?
Gold standard test
Follow-up
No observed disease
Observed disease
Yes
No
No Yes
Signs and symptoms?
Yes
No
Participants screened by both tests
Confirmed disease?
Paired Cancer Screening Trial
6
Screen positive on either test?
Gold standard test
Follow-up
No observed disease
Observed disease
Yes
No
No Yes
Signs and symptoms?
Yes
No
Participants screened by both tests
Screening Test 1
Screening Test 2
Confirmed disease?
Paired Cancer Screening Trial
7
Screen positive on either test?
Gold standard test
Follow-up
No observed disease
Observed disease
Yes
No
No Yes
Signs and symptoms?
Yes
No
Participants screened by both tests
Screening Test 1
Screening Test 2
Confirmed disease?
Paired Cancer Screening Trial
8
Screen positive on either test?
Gold standard test
Follow-up
No observed disease
Observed disease
Yes
No
No Yes
Signs and symptoms?
Yes
No
Participants screened by both tests
Screening Test 1
Screening Test 2
Confirmed disease?
Paired Cancer Screening Trial
9
Screen positive on either test?
Gold standard test
Follow-up
No observed disease
Observed disease
Yes
No
No Yes
Signs and symptoms?
Yes
No
Participants screened by both tests
Screening Test 1
Screening Test 2
Confirmed disease?
Paired Cancer Screening Trial
10
Screen positive on either test?
Gold standard test
Follow-up
No observed disease
Observed disease
Yes
No
No Yes
Signs and symptoms?
Yes
No
Participants screened by both tests
Screening Test 1
Screening Test 2
11
True Non-Cases
True Cases
Screening Test 2 Score
Scre
enin
g Te
st 1
Sco
re
Omniscient Viewpoint
Hypothetical Cancer Screening Data
1
2
3
4
12
Observed Non-Cases
Observed Screen Positive Cases
Screening Test 2 Score
Scre
enin
g Te
st 1
Sco
re
Observed Interval Cases
Study Investigator’s Viewpoint
Hypothetical Cancer Screening Data
1
2
3
4
13 Screening Test 2 Score
Scre
enin
g Te
st 1
Sco
re
Observed Non-Cases
Observed Screen Positive Cases
Observed Interval Cases
Missed Cases
Omniscient Viewpoint
Hypothetical Cancer Screening Data
1
2
3
4
True Screening Test 1
Comparing Two Continuous Screening Tests
14
Sens
itivi
ty
1 - Specificity
Screening Test 2
(Glueck et al., 2009)
True
Observed
Corrected
Standard Screening Test 1
Comparing Two Continuous Screening Tests
15
Sens
itivi
ty
1 - Specificity
Screening Test 2
(Glueck et al., 2009)
True
Observed
Corrected
Screening Test 1
Comparing Two Continuous Screening Tests
16
Sens
itivi
ty
1 - Specificity
Screening Test 2
(Glueck et al., 2009)
True
Observed
Corrected
Standard
Corrected Screening Test 1
Comparing Two Continuous Screening Tests
17
Sens
itivi
ty
1 - Specificity
Screening Test 2
(Glueck et al., 2009)
True
Observed
Corrected
1. Find the maximum likelihood estimates of the parameters of the bivariate Gaussian distribution of test scores for the cases.
2. Use the maximum likelihood estimates and the sampling fractions in each partition to calculate weighted estimates.
Bias Correction Algorithm
18
0.02
0.04
0.06
0.08
0.1
0.12
0.14
1. Find the maximum likelihood estimates of the parameters of the bivariate Gaussian distribution of test scores for the cases.
Bias Correction Algorithm
19
(Nath, 1971) Screening Test 2 Score
Scre
enin
g Te
st 1
Sco
re
0.02
0.04
0.06
0.08
0.1
0.12
0.14
1. Find the maximum likelihood estimates of the parameters of the bivariate Gaussian distribution of test scores for the cases.
Bias Correction Algorithm
20
Screening Test 2 Score
Scre
enin
g Te
st 1
Sco
re
(Nath, 1971)
0.02
0.04
0.06
0.08
0.1
0.12
0.14
1. Find the maximum likelihood estimates of the parameters of the bivariate Gaussian distribution of test scores for the cases.
Bias Correction Algorithm
21
1
2
3
4 Screening Test 2 Score
Scre
enin
g Te
st 1
Sco
re
(Nath, 1971)
2. Use the maximum likelihood estimates and the sampling fractions in each partition to calculate weighted estimates.
Bias Correction Algorithm
22
0.02
0.04
0.06
0.08
0.1
0.12
0.14
2. Use the maximum likelihood estimates and the sampling fractions in each partition to calculate weighted estimates.
Bias Correction Algorithm
23
1
2
3
4 Screening Test 2 Score
Scre
enin
g Te
st 1
Sco
re
0.02
0.04
0.06
0.08
0.1
0.12
0.14
2. Use the maximum likelihood estimates and the sampling fractions in each partition to calculate weighted estimates.
Bias Correction Algorithm
24
1
2
3
4 Screening Test 2 Score
Scre
enin
g Te
st 1
Sco
re
Evaluation of the Method
25 25
Sens
itivi
ty
1 - Specificity
25
Evaluation of the Method
26 26
Type I Error Sens
itivi
ty
1 - Specificity
26
Evaluation of the Method
27 27
Sens
itivi
ty
1 - Specificity
27
Evaluation of the Method
28 28
Wrong Rejection Fraction
Correct Rejection Fraction
Sens
itivi
ty
1 - Specificity
Power
• Study investigators should conduct a simulation of their study using both the standard analysis and the bias correction method.
• Study investigators should choose the analysis plan that has a nominal Type I error rate and the highest power for the correct decision.
Recommendation
29
Oral Cancer Screening Demonstration
30
No visible lesion Dark region confirmed to be carcinoma in situ
VISIBLE LIGHT AUTOFLUORESCENCE
0.0
0.5
1.0
0.0 0.5 1.0
Oral Cancer Screening Analysis
31
Sens
itivi
ty
1 - Specificity
Standard
Δ = 0.06 (p = 0.005)
Visible Light Autofluorescence
Decision Errors Simulation
32
Type I Error Wrong Rejection
Standard Corrected Standard Corrected
0.0
0.5
1.0
0.0 0.5 1.0
Oral Cancer Screening Analysis
33
Standard
Δ = 0.06 (p = 0.005)
1 - Specificity
Sens
itivi
ty
Visible Light Autofluorescence
0.0
0.5
1.0
0.0 0.5 1.0
Oral Cancer Screening Analysis
34 0.
00.
51.
0
0.0 0.5 1.0
Standard Corrected
Δ = 0.06 (p = 0.005)
Δ = -0.06 (p = 0.001)
1 - Specificity
Sens
itivi
ty
Visible Light Autofluorescence
0.0
0.5
1.0
0.0 0.5 1.0
Oral Cancer Screening Analysis
35 0.
00.
51.
0
0.0 0.5 1.0
True Standard Corrected
Δ = 0.06 (p = 0.005)
Δ = -0.06 (p = 0.001)
Δ = -0.06 (p = 0.001)
1 - Specificity
Sens
itivi
ty
Visible Light Autofluorescence
References
36
Glueck, D. H., Lamb, M. M., O'Donnell, C. I., Ringham, B. M., Brinton, J. T., Muller, K. E., Lewin, J. M., et al. (2009). Bias in trials comparing paired continuous tests can cause researchers to choose the wrong screening modality. BMC Medical Research Methodology, 9, 4. doi:10.1186/1471-2288-9-4Kish, L. (1995). Survey Sampling. Wiley-Interscience.
Kish, L. (1995). Survey Sampling. Wiley-Interscience.
Lewin, J. M., D'Orsi, C. J., Hendrick, R. E., Moss, L. J., Isaacs, P. K., Karellas, A., & Cutter, G. R. (2002). Clinical comparison of full-field digital mammography and screen-film mammography for detection of breast cancer. AJR. American Journal of Roentgenology, 179(3), 671-677.
Nath, G. B. (1971). Estimation in Truncated Bivariate Normal Distributions. Journal of the Royal Statistical Society. Series C (Applied Statistics), 20(3), 313-319. doi:10.2307/2346762
Obuchowski, N. A., & McClish, D. K. (1997). Sample size determination for diagnostic accuracy studies involving binormal ROC curve indices. Statistics in Medicine, 16(13), 1529-1542.
Ross, S. (2009). First Course in Probability, A (8th ed.). Prentice Hall.
Zhou, X.-H., McClish, D. K., & Obuchowski, N. A. (2002). Statistical Methods in Diagnostic Medicine (1st ed.). Wiley-Interscience.
