Assessment of Radiologist Performance in theDetection of Lung Nodules:
Dependence on the Definition of “Truth”1
Samuel G. Armato III, PhD, Rachael Y. Roberts, MD, Masha Kocherginsky, PhD, Denise R. Aberle, MDElla A. Kazerooni, MD, MS, Heber MacMahon, MD, Edwin J.R. van Beek, MD, PhD, David Yankelevitz, MD
Geoffrey McLennan, MD, PhD, Michael F. McNitt-Gray, PhD, Charles R. Meyer, PhD, Anthony P. Reeves, PhDPhilip Caligiuri, MD, Leslie E. Quint, MD, Baskaran Sundaram, MD, Barbara Y. Croft, PhD, Laurence P. Clarke, PhD
Rationale and Objectives. Studies that evaluate the lung nodule detection performance of radiologists or computerizedmethods depend on an initial inventory of the nodules within the thoracic images (the “truth”). The purpose of this studywas to analyze (1) variability in the “truth” defined by different combinations of experienced thoracic radiologists and (2)variability in the performance of other experienced thoracic radiologists based on these definitions of “truth” in the con-text of lung nodule detection in computed tomographic (CT) scans.
Materials and Methods. Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independentlymarked lesions they considered to be nodules �3 mm in maximum diameter. Panel “truth” sets of nodules were then de-rived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detectionperformance of the other radiologists was evaluated based on these panel “truth” sets.
Results. The number of “true” nodules in the different panel “truth” sets ranged from 15 to 89 (mean 49.8 � 25.6). Themean radiologist nodule detection sensitivities across radiologists and panel “truth” sets for different panel “truth” condi-tions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case.
Conclusions. Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The defini-tion of “truth” on which lung nodule detection studies are based must be carefully considered, because even experienced tho-racic radiologists may not perform well when measured against the “truth” established by other experienced thoracic radiologists.
alth Studies (M.K.), University of Chicago, 5841 S. Maryland Ave, MC 2026,ornia-Los Angeles, Los Angeles, CA (D.R.A., M.F.M.-G.); Department of Radiol-artment of Radiology (E.J.R.V.B.) and Departments of Medicine, Radiology, and
al College (D.Y.) and the Department of Electrical and Computer Engineeringtional Cancer Institute, Rockville, MD (B.Y.C., L.P.C.). Received February 28,
A091085, U01CA091090, U01CA091099, U01CA091100, and U01CA091103.
Acad Radiol 2009; 16:28–38
1 From the Departments of Radiology (S.G.A., R.Y.R., H.M., P.C.) and HeChicago, IL 60637; Department of Radiological Sciences, University of Califogy, University of Michigan, Ann Arbor, MI (E.A.K., C.R.M., L.E.Q., B.S.); DepBiomedical Engineering (G.M.), University of Iowa, Iowa City, IA; Weill Medic(A.P.R.), Cornell University, Ithaca, NY; and the Cancer Imaging Program, Na2008; accepted May 19, 2008. Supported in part by USPHS Grants U01CAddress correspondence to: S.G.A. e-mail: [email protected]
Academic Radiology, Vol 16, No 1, January 2009 RADIOLOGIST PERFORMANCE IN CT NODULE DETECTION
Studies that evaluate the lung nodule detection performanceof computer-aided diagnostic (CAD) methods or of differentgroups of radiologists fundamentally depend on an initialinventory of the nodules in the images. This assessment of“truth” is usually provided by a panel of experienced tho-racic radiologists who review the images used in the studyto identify lesions that are defined as targets of the study(1–3). Change the “truth,” however, and the performance ofthe CAD method or radiologist under evaluation necessarilychanges (4,5). The “truth” for a specific study is affected bya number of factors, including the composition of the expertpanel (6), the defined targets of the study, the instructionsprovided to panel members, and the manner in which indi-vidual panel members interpret the defined study targets andinstructions.
Lung nodules as a study target are especially subjec-tive. The term nodule refers to abnormalities that span awide spectrum, which is itself a subset of a broader spec-trum of lesions that can be described as “focal abnormali-ties” (7). Varying interpretations of these spectra by dif-ferent radiologists lead to variability in radiologists’ iden-tification of lung nodules (8). Compound variability in thedefinition of “nodule” with subjective qualifying at-tributes, such as minimum size, radiographic solidity, oractionability, and the potential for discordant interpreta-tion is further magnified. The determination that a noduleis present at a specific location is almost always based onimage features alone as interpreted by a radiologist, with-out independent objective verification, given the inherentlimitations of obtaining lung tissue or postmortem data inhumans. According to Dodd et al (9), “dependence onexpert opinion derived from the very same images usedfor the assessment of the imaging system or algorithmleads to an additional source of uncertainty that is notpresent when an independent source of ‘ground truth’ isavailable.” These investigators suggest that some form ofresampling of the expert panel may be useful to under-stand this additional uncertainty (9).
To create a publicly available database of annotatedthoracic computed tomographic (CT) scans as a referencestandard for the medical imaging research community, theLung Image Database Consortium (LIDC) developed atwo-phase process for the interpretation of CT scans bysuch an expert panel. Specifically, a panel of four experi-enced thoracic radiologists, one from each of four differ-ent institutions, reviews the CT scans under two separateand distinct conditions (10). According to the LIDC pro-cess, the initial “blinded read phase” requires radiologists
to independently mark nodules and other lesions they
identify in a thoracic CT scan using a computer interface.During the subsequent “unblinded read phase,” theblinded read results of all radiologists are revealed toeach of the radiologists, who then independently reviewtheir marks along with the anonymous marks of their col-leagues; each radiologist may choose to alter or deleteany of his or her own marks during the unblinded read ormay place additional marks in the CT scan. This two-phase approach was developed to identify as completelyas possible all lesions interpreted as nodules in a CT scanwithout requiring forced consensus. The blinded and un-blinded read phases are intended to comprise a single,comprehensive process for establishing a robust “truth”for lung nodules in CT scans. The “truth” created throughthis process will be associated with the LIDC database forall who use it to train and test CAD methodologies or toconduct any other studies that evaluate nodule detectionperformance.
The LIDC two-phase process, however, deliberatelydeviates from the more commonly used approaches toestablish “truth.” Most studies incorporate a single-readpanel approach, whereas others include a limited secondround to arbitrate discordant findings. The findings ofthese expert panels may be combined and permuted (in alogical and scientifically sound manner) to obtain a seriesof non-unique “truth” sets against which the performanceof the system under consideration may vary substantially(11–13).
The present study circumvents the more robust LIDCprocess and simulates the single-read panel paradigm byinvestigating the “truth” sets that may be constructedfrom the blinded reads for CT scans in the LIDC data-base. Then, rather than evaluate a CAD system, the per-formance of other LIDC radiologists is evaluated againstthese blinded-read–only “truth” sets.
This study poses several fundamental questions. Whena group of two or three experienced thoracic radiologistsforms a consensus panel to establish “truth” for a nodule-detection study, how would the results of the study differif two or three other thoracic radiologists of equivalentexperience were employed to establish the “truth?” Howwould the radiologists in the first panel fare against the“truth” established by the second panel? The purpose ofthis study was to analyze: 1) variability in the “truth” de-fined by different combinations of experienced thoracicradiologists from the expert panel and 2) variability in theperformance of other experienced thoracic radiologists inthe identification of lung nodules in CT scans based on
these different definitions of “truth.”
29
ARMATO ET AL Academic Radiology, Vol 16, No 1, January 2009
MATERIALS AND METHODS
Patient Image DataA total of 25 thoracic helical CT scans were collected
from a single LIDC site in accordance with the previouslypublished inclusion criteria (7,14). Appropriate local insti-tutional review board approval was obtained for the re-search use of scans that had been acquired in accordancewith established clinical or ongoing research imaging pro-tocols. Each CT scan had been acquired from a differentpatient (10 females, 15 males; age 40–75 years, median59) on Aquilion (Toshiba Medical Systems, Tochigi,Japan) (n � 17), Sensation 16 (Siemens) (n � 5), orEmotion 6 (Siemens, Forchheim, Germany) (n � 3) CTscanners. The tube peak potential energies used for scanacquisition were as follows: 120 kV (n � 5), 130 kV(n � 3), and 135 kV (n � 17). Tube current ranged from45 to 499 mA (mean 228.9). The slice thickness and re-construction interval were equal for each scan at 2.0 mm(n � 4) and 3.0 mm (n � 21). The in-plane resolution ofthe 512 � 512 pixel sections ranged from 0.537 to 0.946mm (mean 0.682). A “standard/nonenhancing” convolu-tion kernel was used for image reconstruction. The major-ity of the CT scans (n � 15) had been performed usingintravenous contrast material.
Image EvaluationMonitors with clinically acceptable specifications were
used at each site, and each monitor was calibrated with aVeriLUM Color Dual Mode Pod (IMAGE Smiths, Kens-ington, MD). Ambient lighting was set to simulate theclinical reading environment. Each CT scan was initiallypresented at a standard brightness/contrast setting withoutmagnification, but the radiologists were allowed to adjustbrightness, contrast, and magnification as appropriate toenable the most complete interpretation of the scan.
The scans identified at a single LIDC site were anony-mized to remove all protected health information withinthe Digital Imaging and Communications in Medicineheaders of the images in accordance with Health Insur-ance Portability and Accountability Act guidelines (15)and electronically transferred to each of the four otherLIDC sites to initiate the blinded read process (10). OneLIDC radiologist at each of the four sites independentlyevaluated each scan for the presence of lesions in threedifferent categories: (1) nodules with greatest in-planedimension �3 mm but �30 mm, regardless of presumedhistology (“nodule �3 mm”); (2) nodules �3 mm that
are not clearly benign (ie, diffusely calcified) (“nodule
30
�3 mm”); and (3) other intraparenchymal lesions �3 mm(“non-nodule �3 mm”) (eg, scars, areas of consolidation;lesions �30 mm in diameter), which were noted for thesake of completeness (10). Through discussions and train-ing, the radiologists were familiar with the subtleties ofeach lesion category before the study. The radiologistsindicated the location of lesions through the placement ofcategory-specific marks on the images using an interactivecomputer interface. The spatial positions of lesions ineach category as defined by each radiologist were re-corded in an XML file for later analysis. This interfaceincluded measurement tools to help the radiologists deter-mine whether a lesion’s dimension exceeded the 3-mmthreshold. These cases subsequently proceeded to the un-blinded read phase of the LIDC process. Because the pur-pose of this study was to simulate a single-read panelapproach to establishing “truth,” the final post-unblindedread results were not included in this study.
An LIDC site may have more than a single “LIDCradiologist” to handle the workload generated by theLIDC database, which will eventually contain nearly 1000CT scans with lung nodules. Each LIDC radiologist is athoracic radiologist, and each was trained by the site’sprimary LIDC radiologist to become familiar with thedetails of the LIDC process. Accordingly, reads per-formed for the LIDC database are considered on an insti-tutional basis, and, for the purpose of this study, “Radiol-ogist A” will refer to the LIDC radiologist or radiologistsat one specific LIDC site. All 25 scans were evaluated bya single LIDC radiologist at each of two sites (Radiolo-gist C and Radiologist D), whereas at the other two sites,the scans were distributed between two LIDC radiologists(at one site, one radiologist evaluated 21 scans and theother evaluated four scans [Radiologist A]); at the othersite, one radiologist evaluated 14 scans and the otherevaluated 11 scans [Radiologist B]). The radiologistswere aware that they were reviewing the CT scans to pro-vide an assessment of “truth” for lung nodule studies.
The image evaluation process (the blinded reads) ef-fectively yielded four independent sets of nodule “truth”data. To determine the physical correspondence of marksfrom different radiologists, all radiologist marks were vi-sually reviewed and inventoried by a single LIDC princi-pal investigator. Using the computer interface and theXML files created during the blinded reads, the marks ofall four radiologists were displayed simultaneously at theappropriate spatial locations within the images. Throughdifferences in color and shape, the displayed marks iden-
tified the institution of the radiologist who placed the
Academic Radiology, Vol 16, No 1, January 2009 RADIOLOGIST PERFORMANCE IN CT NODULE DETECTION
mark and the lesion category indicated by that radiologist.Marks considered to represent the same physical lesionwithin the scan were grouped together by visual inspec-tion of all marks followed by a subjective determinationof the three-dimensional contiguity of the lesions thosemarks were intended to represent. It should be noted thatthe same lesion could have been assigned to different le-sion categories (ie, “nodule �3 mm,” “nodule �3 mm,”or “non-nodule �3 mm”) by different radiologists or notannotated at all by a subset of radiologists. This groupingof marks defined the inventory of lesions that providedthe basis for all subsequent analyses; this comprehensiveinventory process identified which lesions were annotatedby which radiologists and the lesion category to whichthe lesion was assigned by each radiologist.
Evaluation of Radiologist Performance Based onthe “Truth” of the Other Radiologists
With four independent and equally valid assessments of“truth,” analysis was confined to the “nodule �3 mm” le-sion category. The nodule identification performance of eachLIDC radiologist was evaluated in the context of a panel“truth” set derived from the “truth” sets of different combi-nations of two or three other LIDC radiologists. A true posi-tive was recorded for the radiologist being evaluated if thatradiologist did not annotate as a “nodule �3 mm” a lesionthat was included in the panel “truth” set, a false positivewas recorded if that radiologist had annotated as a “nodule�3 mm” a lesion that was not included in the panel “truth”set, and a false negative was recorded if that radiologist didnot annotate a lesion that was included in the panel “truth”set (or annotated it as one of the other two lesion categories)(ie, “nodule �3 mm” or “non-nodule �3 mm”) or did notannotate the lesion at all.
The performance of each of the four radiologists wasevaluated against panel “truth” sets formed by the three pos-sible pairwise combinations of the three other radiologists
Table 1The Number of Nodules Identified by Each Radiologist
No. of Nodules Mean � SD
Radiologist A 63 49.8 � 20.2Radiologist B 62Radiologist C 20Radiologist D 54
SD, standard deviation.
through (1) a logical OR (ie, union) of the “truth” sets of the
pair of radiologists and (2) a logical AND (ie, intersection)of the “truth” sets of the pair of radiologists. Each logicalOR panel “truth” set included lesions annotated as a “nodule�3 mm” by at least one radiologist of the pair, whereaseach logical AND panel “truth” set included lesions anno-tated as a “nodule �3 mm” by both radiologists of the pair.
The performance of each radiologist was also evaluatedagainst the panel “truth” sets that consisted of (1) a logicalOR combination of the “truth” sets of the other three radiol-ogists, (2) a majority combination of the other three radiolo-gists’ “truth” sets, and (3) a logical AND combination of the“truth” sets of the other three radiologists. Each logical ORpanel “truth” set included lesions annotated as a “nodule �3mm” by at least one of the three radiologists, each majoritypanel “truth” set included lesions annotated as a “nodule �3mm” by at least two of the three radiologists, and each logi-cal AND panel “truth” set included lesions annotated as a“nodule �3 mm” by all three radiologists.
RESULTS
Number of NodulesA total of 91 lesions were identified as “nodule �3
mm” by at least one of the four radiologists. The numberof nodules identified by each of the four radiologists isshown in Table 1. Radiologist C defined the fewest le-sions as nodules (n � 20), and Radiologist A defined themost lesions as nodules (n � 63). For the nodules thatwere identified by each radiologist, Figure 1 presents thenumbers of those nodules that were identified by that ra-
Figure 1. The number of lesions that were identified as a“nodule �3 mm” by each radiologist solely, by each radiologistand one other radiologist, by each radiologist and two otherradiologists, and by each radiologist and the three other radiol-ogists (ie, lesions that all four radiologists identified as nod-ules). The sum of the four bars for each radiologist corre-sponds to the data in Table 1.
diologist alone, by the radiologist and one other radiolo-
31
ARMATO ET AL Academic Radiology, Vol 16, No 1, January 2009
gist, by the radiologist and two other radiologists, and bythe radiologist and all three other radiologists. The com-plexities of the varied combinations of radiologists thatidentified each of the 91 nodules can be appreciated fromthe Venn diagram in Figure 2.
Variability in “Truth” SetsTwenty-four panel “truth” sets were created in total:
the logical OR and the logical AND sets for the six possi-ble pairwise combinations of the four radiologists (n �12) (Fig 3) and the logical OR, the majority, and the logi-cal AND sets for the four possible combinations of threeof the four radiologists (n � 12) (Fig 4). The number of“true” nodules in these “truth” sets spanned a wide range(Table 2), with the smallest number of nodules included
Figure 2. Venn diagram of the different combinations of radiolo-gists that identified the 91 lesions that were defined as “nodule�3 mm” by at least one radiologist.
Figure 3. The number of lesions identified as “nodule �3 mm”in the panel “truth” sets created from pairwise combinations ofthe four radiologists’ individual reads combined through a logicalOR operation and combined through a logical AND operation.
in the panel “truth” set derived from the logical AND
32
combination of Radiologists B, C, and D (n � 15) andthe largest number of nodules included in the panel“truth” set derived from the logical OR combination ofRadiologists A, B, and D (n � 89). The mean number of“true” nodules across all panel “truth” sets was 49.8 �25.6.
Figure 5a shows a lesion that was identified as a “nod-ule �3 mm” by one radiologist but not by another radiol-ogist, so that this lesion was considered a “true” nodulefor the logical OR combination of these two specific radi-ologists but not for their logical AND combination. Fig-ure 5b shows a lesion that was identified as a “nodule �3
Figure 4. The number of lesions identified as “nodule �3 mm”in the panel “truth” sets created from triplet combinations of thefour radiologists’ individual reads combined through a logical ORoperation, a majority approach, and a logical AND operation.
Table 2The Number of Nodules Contained in the Panel “Truth” SetsObtained from Different Combinations of Radiologists underDifferent Conditions
mm” by both of these radiologists; this lesion was consid-
D co
Academic Radiology, Vol 16, No 1, January 2009 RADIOLOGIST PERFORMANCE IN CT NODULE DETECTION
ered a “true” nodule for both the logical OR and the logi-cal AND combinations of the two radiologists. Figure 5cshows a lesion that was identified as a “nodule �3 mm”by one radiologist but not by two others, so that this le-sion was considered a “true” nodule for the logical ORcombination of these three specific radiologists but not fortheir majority or logical AND combinations. Figure 5dshows a lesion that was identified as a “nodule �3 mm”by two of these radiologists but not by the third; this le-sion was considered a “true” nodule for both the logicalOR and the majority combinations of the three radiologistsbut not for their logical AND combination. Figure 5e showsa lesion that was identified as a “nodule �3 mm” by all
Figure 5. Examples of lesions that were identified as (a) a “noduthe logical OR combination of these two specific radiologists); (b) aboth the logical OR and the logical AND combinations); (c) a “nodufor the logical OR combination of these three specific radiologists)third (a “true” nodule for both the logical OR and the majority com(a “true” nodule for the logical OR, the majority, and the logical AN
three of these radiologists; this lesion was considered a
“true” nodule for the logical OR, the majority, and the logi-cal AND combinations of the three radiologists.
Table 2 presents the mean number of nodules acrossall radiologist combinations for each panel “truth” condi-tion. The trends observed in these means are consistentwith expectations. First, the mean number of nodules ob-tained for all logical OR combinations of individual radi-ologist “truth” sets exceeds the mean number of nodulesobtained for all logical AND combinations of individualradiologist “truth” sets because AND (the intersection ofthe sets of nodules identified by each radiologist) is morerestrictive than OR (the union of these sets). Second, themean number of nodules obtained for all majority combi-
mm” by one radiologist but not by another (a “true” nodule fordule �3 mm” by both of these radiologists (a “true” nodule for3 mm” by one radiologist but not by two others (a “true” nodule“nodule �3 mm” by two of these radiologists but not by the
ions); and (e) a “nodule �3 mm” by all three of these radiologistsmbinations).
le �3“no
le �; (d) abinat
nations of individual radiologist “truth” sets (for the com-
33
ARMATO ET AL Academic Radiology, Vol 16, No 1, January 2009
binations of three radiologists) is between the mean numberof nodules obtained for all logical OR combinations and themean number of nodules obtained for all logical AND com-binations because the majority is more strict than an OR butless strict than an AND. Third, the mean number of nodulesobtained for all logical OR combinations of three radiolo-gists exceeds the mean number of nodules obtained for alllogical OR combinations of two radiologists because an ORamong three is more inclusive than an OR between two.Fourth, the mean number of nodules obtained for all logicalAND combinations of three radiologists is less than themean number of nodules obtained for all logical AND com-binations of two radiologists because an AND among threeis less inclusive than an AND between two. Fifth, the meannumber of nodules obtained for all majority combinations ofindividual radiologists (for the combinations of three radiolo-gists) exceeds the mean number of nodules obtained for alllogical AND combinations of two radiologists because amajority among three is more inclusive than an AND be-tween two specific radiologists.
Radiologist PerformanceWhen the “nodules �3 mm” identified by the individ-
ual radiologists were compared against the different panel“truth” sets (for panel combinations that did not includethat specific radiologist), a wide range of nodule detectionsensitivities and false-positive rates resulted (Table 3).The mean sensitivities ranged from 51.0% for radiologistperformance compared against the logical OR combina-
Table 3Radiologist Nodule-detection Sensitivities and False-positive R“Truth” Condition
Panel “Truth” Condition
Sensitivity
Minimum Maximum
Radiologist pairs(logical OR) 20.2% 75.4%
Radiologist pairs(logical AND) 37.5% 100.0%
Radiologist triplets(logical OR) 20.2% 64.1%
Radiologist triplets(majority) 29.8% 81.4%
Radiologist triplets(logical AND) 45.5% 100.0%
SD, standard deviation.
tion of three radiologists to 83.2% for radiologist perfor-
34
mance compared against the logical AND combination ofthree radiologists. The mean false-positive rates rangedfrom 0.33 false positives per case for radiologist perfor-mance compared against the logical OR combination ofthree radiologists to 1.39 false positives per case for radi-
Figure 6. The means and ranges (across radiologist combinations)of (a) radiologist nodule detection sensitivities and (b) radiologistfalse-positive rates based on the different panel “truth” sets.
across Radiologists and Panel “Truth” Sets for Each Panel
False-positive Rate(False Positives/Case)
an � SD Minimum Maximum Mean � SD
� 19.9% 0.08 0.84 0.48 � 0.31
� 24.1% 0.16 1.88 1.19 � 0.70
� 20.7% 0.08 0.56 0.33 � 0.25
� 25.1% 0.12 1.12 0.79 � 0.47
� 25.4% 0.20 1.92 1.39 � 0.81
ates
Me
54.8
76.7
51.0
67.4
83.2
ologist performance compared against the logical AND
erest
Academic Radiology, Vol 16, No 1, January 2009 RADIOLOGIST PERFORMANCE IN CT NODULE DETECTION
combination of three radiologists. Both the average sensi-tivities and the average false-positive rates increased asthe panel “truth” set became more restrictive (Fig 6).
The nodule detection sensitivities of individual radiolo-gists are shown in Table 4 for the pairwise “truth” setsand in Table 5 for the triplet “truth” sets. Consistent withthe trends observed for the aggregate sensitivities in Table3, the sensitivities of individual radiologists increased asthe panel “truth” sets became more restrictive. Each radi-ologist tended to be fairly consistent in terms of sensitiv-ity across “truth” sets from different radiologist pairs fora given panel condition (ie, logical OR or logical AND),with no coefficient of variation exceeding 0.10 (Table 4).
The combination of radiologist performance and vari-
Table 4Individual Radiologist Nodule-detection Sensitivities (in percenthrough a Logical OR/AND
A/B A/C A/D
Radiologist A — — — 6Radiologist B — 62.1/94.1 64.0/78.6Radiologist C 20.2/39.0 — 24.0/38.1Radiologist D 58.3/80.5 65.2/94.1 — 6
CV, coefficient of variation.
Table 5Individual Radiologist Nodule-detection SensTriplet Panel “Truth” Sets Combined through
A/B/C A/
Radiologist A —Radiologist B —Radiologist C — 20.2/2Radiologist D 57.5/81.0/93.8
Table 6The Lesion Categories Assigned by the Four Radiologists to th
Radiologist A Radiologist
Fig 5(a) Nodule �3 mm Nodule �3 mFig 5(b) Nodule ≥3 mm Nodule ≥3Fig 5(c) Nodule ≥3 mmFig 5(d) Nodule ≥3 mm Nodule ≥3Fig 5(e) Nodule ≥3 mm Nodule ≥3
The “nodule �3 mm” category, which is the only category of int
able panel “truth” sets may be appreciated by referring to
Table 6, which presents the lesion categories assigned bythe four radiologists to the five lesions shown in Figure5a–e. The “nodule �3 mm” category, which is the onlycategory of interest for this study, is shown in bold. Eachlesion could have three other possible category assign-ments: “nodule �3 mm,” “non-nodule �3 mm,” or nocategory at all. Of immediate note is that none of thesefive lesions was assigned to the same category by all fourradiologists. Furthermore, the inclusion of any of the fivelesions in a specific panel “truth” set depends on how thatpanel is constructed (pair vs. triplet, the specific radiolo-gists included, and the combination rule [OR, majority, orAND]). The lesion in Figure 5a was selected to demon-strate a lesion identified as a “nodule �3 mm” by one
) Based on Different Pairwise Panel “Truth” Sets Combined
ies (in percentages) Based on Differentgical OR/Majority/AND
adiologist Triplet
A/C/D B/C/D
— 64.1/81.4/100.062.3/77.3/93.8 —
5.5 — —— —
e Lesions Shown in Figure 5
Radiologist C Radiologist D
Nodule ≥3 mmNodule ≥3 mm
Non-nodule �3 mmNodule ≥3 mm
Nodule ≥3 mm Nodule �3 mm
for this study, is shown in bold.
tages
R
B/C
3.6/1——
3.6/9
itivita Lo
R
B/D
——9.8/4—
e Fiv
B
mmm
mmmm
radiologist but not by another radiologist (ie, a “true”
35
ARMATO ET AL Academic Radiology, Vol 16, No 1, January 2009
nodule for the logical OR combination of two radiolo-gists). From Table 6, the previous statement holds for alogical OR combination of Radiologists B and C or Radi-ologists A and C or Radiologists C and D; however, thelesion in Figure 5a would not be included in the logicalOR panel “truth” set of Radiologists A and B or Radiolo-gists A and D or Radiologists B and D. Considering thelogical OR panel “truth” set of Radiologists B and C forFigure 5a, it is interesting to note from Table 6 that thislesion would be recorded as a false negative for both Ra-diologist A and Radiologist D, but for different reasons:Radiologist A identified the lesion but considered it to beless than 3 mm in diameter and hence not a study target,whereas Radiologist D either did not observe the lesion atall or considered the structure to be beyond the scope ofthe three defined lesion categories (eg, normal anatomy).Similar observations regarding differences in interpreta-tion or “missed” lesions may be made from Table 6 forthe other four lesions in Figure 5b–e.
DISCUSSION
Several limitations are inherent in this study. First,the task of identifying nodules in the context of estab-lishing “truth” for research studies differs from theidentification task in the clinical setting, and the radiol-ogists were asked to identify lesions without the benefitof accompanying clinical data. Second, pathologic in-formation was not available for any of the lesions.Third, to define the study targets, radiologists wereforced to make binary decisions as to the presence ofappropriately sized nodules in the CT scans. Interest-ingly, these limitations are shared with many publishednodule detection studies.
The design of our study presents the potential for aninteresting bias. Although radiologists were instructed toreview the CT scans from the perspective of identifying“truth,” these same findings were used to evaluate the“performance” of each radiologist against the findings (ie,the “truth”) of the other radiologists. The alternative studydesign would have included an initial session in whichradiologists would be instructed to evaluate the scans for“truth,” and then, after sufficient time had elapsed, a sec-ond session would have been conducted in which the ra-diologists would be instructed to review the same scansfor the presence of lung nodules in a more routine man-ner and without the added burden of establishing “truth.”
Either scenario, however, differs from the reality of clini-
36
cal practice, a fact that underlies any observer study con-ducted in a research setting; the psychology of the radiol-ogist is necessarily altered. It is difficult to know whetherthe radiologists might have interpreted the scans differ-ently with the knowledge that they were establishing“truth” rather than the thought that their findings were tobe compared against an already existing “truth.” It couldbe argued that the process of establishing “truth” wouldcause the radiologists to be more vigilant, especially inthe absence of clinical information that could mitigate theneed to report a specific lesion as a potential nodule.Conversely, under the alternative study design, the radiol-ogists could demonstrate greater attention to the task withthe knowledge that their performance would be comparedagainst some reference. Despite these potential differ-ences, the approach adopted in the present study reflects aconsistent psychology that existed for both the “truthing”task and the performance evaluation task.
Ninety-one lesions were identified as “nodule �3 mm”by at least one of the four radiologists. This finding doesnot imply that the 25 CT scans contained only 91 nod-ules; had a fifth radiologist been involved, additional le-sions might have been defined as “nodule �3 mm.” Suchpostulation further supports the conclusions that may bedrawn from this study regarding the variability of “truth”assessments.
The assignment of a lesion to a specific category in thecontext of the panel “truth” set or the evaluated radiolo-gist required three subjective steps: (1) identification of alesion (is the observed structure an abnormality or normalanatomy?); (2) determination of lesion size (is the longestdimension of the lesion �3 mm but �30 mm?); and (3)evaluation of lesion features (does the lesion represent a“nodule”?). The multiple levels of inherently subjectiveinterpretation required on the part of the radiologists helpexplain the observed variability in this study, and suchvariability, based on equally subjective aspects of imageinterpretation, is certainly present in clinical practice. Alesion included as a “nodule �3 mm” in any particularpanel “truth” set but not identified as such by the evalu-ated radiologist could represent, in the context of that“truth” set, a search error or a decision-making error ac-cording to the categories of Kundel et al (16), although afull accounting of this distinction may not be extractedfrom the data collected.
A 3-mm size threshold separated the “nodules �3mm” that were of interest in this study from the “nodules�3 mm” that were not of interest. The measurement of
lesion size, even with the use of electronic measurement
Academic Radiology, Vol 16, No 1, January 2009 RADIOLOGIST PERFORMANCE IN CT NODULE DETECTION
tools, is a highly variable task both in clinical practiceand in observer evaluation studies (17–19). From the per-spective of establishing “truth,” a lesion marked as a“nodule �3 mm” in the “truth” set became a target thatmust be identified by the “system” (ie, a CAD method or,in this study, the radiologist being evaluated), whereas alesion marked as a “nodule �3 mm” was not to be identi-fied by the system. From the perspective of performanceassessment, a lesion marked by the evaluated radiologistas a “nodule �3 mm” was considered a true positive ifthe lesion was marked as a “nodule �3 mm” in the“truth” set, a lesion marked by the radiologist as a “nod-ule �3 mm” (or as a “non-nodule �3 mm” or notmarked at all) was considered a false negative if the le-sion was marked as a “nodule �3 mm” in the “truth” set,and a lesion marked by the radiologist as a “nodule �3mm” was considered a false positive if the lesion was notmarked as a “nodule �3 mm” by the requisite number oftruth panel radiologists regardless of whether the othertruth panel radiologists marked the lesion as a “nodule�3 mm” or as a “non-nodule �3 mm” or whether theother truth panel radiologists provided no mark at all. Agreater degree of variability would be expected in the“truth” sets for nodules with diameter near 3 mm and,accordingly, in the radiologists’ nodule detection perfor-mance for these nodules.
The imposition of such a size threshold is consistentwith the design of most reported CAD system evaluationstudies. CAD systems are typically developed to identifynodules above some minimum size (as determined by theinvestigators), and at some point in the algorithm the sys-tem must determine whether each nodule candidate satis-fies that size threshold. Similarly, when establishing“truth,” three binary decisions must be made: (1) whethera lesion is present at a specific location, (2) whether thelesion is a “nodule,” and (3) whether that nodule satisfiesthe size threshold. These are the same decisions that wererequired of the radiologists in this study; analogous deci-sions are required of radiologists in clinical practice on adaily basis. Accordingly, a “false negative” or “false posi-tive” could result when the opinion of the evaluated radi-ologist differed from that of the “truth” panel with regardto the size or the “nodularity” of an abnormality that wasrecognized by both the evaluated radiologist and thepanel.
Although the number of nodules identified by differentradiologists is similar (especially for Radiologists A, B,and D) (see Fig 1), the specific nodules identified by the
individual radiologists are quite distinct. For example,
whereas Radiologists A and B identified 63 and 62 nod-ules, respectively, the number of nodules contained withinthe logical OR combination of Radiologists A and B (n �84) is more than twice the number of nodules containedwithin their logical AND combination (n � 41). Thisfinding means that only about two thirds of the nodulesidentified by either Radiologist A or Radiologist B werealso identified by the other.
Radiologist nodule detection sensitivity increased asthe panel “truth” set criterion became more strict (eg,from OR to AND) for two main reasons. First, the num-ber of “actual nodules” based on the more restrictive“truth” set decreased, and so the denominator of the ex-pression for sensitivity decreased. Second, the nodulescontained within the more strict “truth” set likely repre-sent more obvious nodules that present a greater likeli-hood of radiologist agreement.
Similarly, the number of false positives increased asthe panel “truth” set criterion became more strict. Be-cause the more strict “truth” set contained fewer “ac-tual nodules,” fewer of the lesions marked by the radi-ologist under evaluation were considered “true” nod-ules. The remaining marked lesions, therefore, wereconsidered false positives. Note that because this studywas one of detection rather than classification, the con-cept of a “true negative” did not exist, and the estima-tion of several quantities frequently used in rater agree-ment studies (eg, specificity and the � statistic) was notpossible.
The findings presented here challenge the certitudeinherently associated with the expert-observer– defined“truth” that provides the basis for many medical imageanalysis studies across a diversity of imaging modali-ties. Similar considerations exist for nodule segmenta-tion studies, which depend on variable definitions of“truth” based on the nodule outlines of different radiol-ogists (17,20). For many tasks, radiologist interpreta-tion is the closest approximation to “truth” that may beattained; the limitations of that approximation, how-ever, must be recognized and appreciated by investiga-tors. The two-phase approach to the definition of“truth” for nodule detection studies developed by theLIDC was intended to reduce the variability inherentamong radiologists (8). The results of the present studycould have important implications for the clinical inter-pretation of CT scans in the context of lung noduledetection in which a single reader is responsible forwhat is clearly a difficult detection (and subsequent
classification) task— even double and triple readings
37
ARMATO ET AL Academic Radiology, Vol 16, No 1, January 2009
have limitations and variability that must be understoodand should be taken into account when comparing theperformance of CAD systems against radiologist“truth.”
REFERENCES
1. Wormanns D, Ludwig K, Beyer F, et al. Detection of pulmonary nodulesat multirow-detector CT: effectiveness of double reading to improvesensitivity at standard-dose and low-dose chest CT. Eur Radiol 2005;15:14–22.
2. Leader JK, Warfel TE, Fuhrman CR, et al. Pulmonary nodule detectionwith low-dose CT of the lung: agreement among radiologists. Am JRoentgenol 2005; 185:973–978.
3. Novak CL, Qian J, Fan L, et al. Inter-observer variations on interpreta-tion of multi-slice CT lung cancer screening studies, and the implica-tions for computer-aided diagnosis. SPIE Proc 2002; 4684:68–79.
4. Ochs R, Kim HJ, Angel E, et al. Forming a reference standard fromLIDC data: impact of reader agreement on reported CAD performance.SPIE Proc 2007; 6514
5. Paquerault S, Petrick N, Myers KJ, et al. Impact of a computer-aideddetection (CAD) system on reader performance: assessment based ona truthing panel compared to the true gold standard. Radiology 2007;245:546–547.
6. Petrick N, Gallas BD, Samuelson FW, et al. Influence of panel size andexpert skill on truth panel performance when combining expert ratings.SPIE Proc 2005; 5749:49–57.
7. Armato SG III, McLennan G, McNitt-Gray MF, et al. Lung Image Data-base Consortium: developing a resource for the medical imaging re-search community. Radiology 2004; 232:739–748.
8. Armato SG III, McNitt-Gray MF, Reeves AP, et al. The Lung Image Da-tabase Consortium (LIDC): an evaluation of radiologist variability in the
identification of lung nodules on CT scans. Acad Radiol 2007;14:1409–1421.
38
9. Dodd LE, Wagner RF, Armato SG III, et al. Assessment methodologiesand statistical issues for computer-aided diagnosis of lung nodules incomputed tomography: contemporary research topics relevant to theLung Image Database Consortium. Acad Radiol 2004; 11:462–475.
10. McNitt-Gray MF, Armato SG III, Meyer CR, et al. The Lung Image Da-tabase Consortium (LIDC) data collection process for nodule detectionand annotation. Acad Radiol 2007; 14:1464–1474.
11. Miller DP, O’Shaughnessy KF, Wood SA, et al. Gold standards and ex-pert panels: a pulmonary nodule case study with challenges and solu-tions. SPIE Proc 2004; 5372:173–184.
12. Revesz G, Kundel HL, Bonitatibus M. The effect of verification on theassessment of imaging techniques. Invest Radiol 1983; 18:194–198.
13. Jiang Y. A Monte Carlo simulation method to understand expert-panelconsensus truth and double readings. Medical Image Perception Con-ference XII. 2007. The University of Iowa, Iowa City, IA, 2007.
14. Clarke LP, Croft BY, Staab E, et al. National Cancer Institute initiative:lung image database resource for imaging research. Acad Radiol 2001;8:447–450.
15. Department of Health and Human Services. Standards for privacy ofindividually identifiable health information: final rules. Fed Register2002; 67:53182–53272.
16. Kundel HL, Nodine CF, Carmody D. Visual scanning, pattern recogni-tion and decision-making in pulmonary nodule detection. Invest Radiol1978; 13:175–181.
17. Meyer CR, Johnson TD, McLennan G, et al. Evaluation of lung MDCTnodule annotation across radiologists and methods. Acad Radiol 2006;13:1254–1265.
18. Reeves AP, Biancardi AM, Apanasovich TV, et al. The Lung ImageDatabase Consortium (LIDC): a comparison of different size metricsfor pulmonary nodule measurements. Acad Radiol 2007; 14:1475–1485.
19. Schwartz LH, Ginsberg MS, DeCorato D, et al. Evaluation of tumormeasurements in oncology: Use of film-based and electronic tech-niques. J Clin Oncol 2000; 18:2179–2184.
20. Ross JC, Miller JV, Turner WD, et al. An analysis of early studies re-
leased by the Lung Imaging Database Consortium (LIDC). Acad Radiol2007; 14:1382–1388.
