lournal of Critical Care

VOL 13, NO 4 DECEMBER 1998

ORIGINAL INVESTIGATIONS

Adjudicating Ventilator-Associated Pneumonia in a Randomized Trial of Critically Ill Patients

Deborah Cook, Stephen Walter, Andreas Freitag, Gordon Guyatt, Hugh Devitt, Maureen Meade, Lauren Griffith, Alicia Sarabia, Hugh Fuller, MarkTurner, and Kevin Gough

Purpose: The purpose of this study was to evaluate an adjudication strategy for diagnosing ventilator- associated pneumonia (VAP) in a randomized trial. Materials and Methods: In a double-blind trial of su- cralfate versus ranitidine, one of four pairs of adjudica- tors examined each case of clinically suspected VAP Nurse and physician notes and all relevant laboratory data were allocated to each adjudication pair in groups of five patients. Each reader in the pair decided whether the patient had VAP; differences were resolved by consensus discussion. Results: The overall unadjusted study odds ratio for VAP was 0.82 (P = .21) representing a trend toward less pneumonia with sucralfate compared with raniti- dine. The odds ratio adjusted for adjudication pair was

A DJUDICATION COMMITTEES in random- ized clinical trials in pulmonary and critical

care medicine are used for many reasons, including ratification or refutation of eligibility, examination of cointerventions and protocol violations, and evaluation of outcomes. Outcomes most likely to be examined by adjudication committees are mor- bid events (eg, ventilator-associated pneumonia or the acute respiratory distress syndrome), outcomes requiring attribution (eg, cause-specific mortality), and appraisal of patient dysfunction (eg, signs and symptoms). A review of the reliability of causes of death in four major medical journals documented myriad approaches, and an absence of methodo- logical details in 22% of publications.’ Estimates from autopsy research suggest that erroneous causes of death by clinical criteria are common.2s3 Out- comes requiring less judgement, such as physiologi- cal variables, laboratory measures, and all-cause

Journal ofCritical Care, Vol 13, No 4 (December), 1998: pp 159-163

0.85 (P = .27). The proportion of charts adjudicated as VAP positive among pairs ranged from 50% to 92%; crude agreement between readers in each pair varied from 50% to 82%. When adjudicators disagreed, the final consensus was split evenly between the two adjudicators’ initial opinions in two pairs; in the other two pairs, the final decision reflected one dominant initial opinion. Personnel time to adjudicate all pa- tients with a suspicion of VAP was 74 days. Conclusions: Though adjudication of outcomes such as VAP is time-consuming, consistent decision- making requires strict criteria, training, and calibra- tion. Patients should be assigned to adjudication teams through random allocation. Copyright o 1998 by W.B. Saunders Company

mortality, are less likely to undergo such adjudica- tion.

Careful adjudication can reduce random error and, in unblinded studies, bias. For example, in the Cardiac Arrhythmia Pilot Study, classification of

From the Departments of Medicine and Clinical Epidemiol-

ogy & Biostatistics, MeMaster Lmiversity, Hamilton, Ontario:

Departments ofAnesthesia and Medicine, University of Toronto, Toronto, Ontario; and Department of Medicine, Univetsity of

British Columbia, Vancouver, British Columbia, Canada. Received July I, 1998. Accepted August IO, 1998. Address correspondence to Deborah J. Cook, MD, Depart-

ment of Medicine, St. Joseph S Hospital, 50 Charlton Ave East, Hamilton, Ontario, Canada L8N 4A6.

This trial was funded by the Medical Research Council of Canada and Hoechst Marion Roussel Inc. Dr Cook is a Career

Scientist of the Ontario Ministry of Health; Dr Walter is a National Health Scientist, National Health Research and Devel-

opment Program, Health Canada. Copyright 0 1998 by WB. Saunders Company 0883-9441/98/1304-0001$8.00/O

159

160 COOK ET AL

the deaths or cardiac arrests after myocardial infarction was in disagreement with the two- member Events Committee review for 23% of the patients.4 Frequent disagreement led to time- consuming deliberation by the entire committee to make the final classification. Realizing that catego- rizing deaths as sudden or nonsudden was not equivalent to classifying deaths as arrhythmic or nonarrhythmic led to development of specific crite- ria, which were used in the full-scale Cardiac Arrhythmia Suppression Trial. Amiodarone ap- pears to decrease arrhythmic death when the cause of death is determined by a committee; however, controversy continues5 about amiodarone having no effect on all-cause mortality in the CAMIAT6 and EMIAT’ trials.

Decisions that trialists face in establishing an adjudication process include the size and composi- tion of the committee, training requirements, whether to use adjudication teams, what outcomes will be adjudicated, how cases will be allocated to adjudicators, to what extent the review will be blinded and independent, how the magnitude of agreement will be reported, how consensus will be achieved, and how study results vary with and without adjudication of events. Despite the crucial role of adjudication committees in the conduct of randomized clinical trials, few studies describe their methods. An exception in the critical care literature is the impact of the Clinical Evaluation Committee of the INTERSEPT Study, which re- ported its role in the trial conduct, as well as different trial results using strict and more liberal inclusion criteria.8 This article describes the pro- cess and outcomes of adjudication of ventilator- associated pneumonia (VAP) in a randomized trial in critically ill patients.9

MATERIALS AND METHODS

In a multicenter randomized trial, 1,200 critically il l patients received either oral sucralfate 1 g every 6 hours and placebo intravenous ranitidine, or ranitidine 50 mg every 8 hours and placebo sucralfate. Patients, caregivers, research personnel, outcome adjudicators, and analysts were blind to group alloca- tion. Physicians identified patients with a clinical suspicion of VAP using the modified Center for Disease Control criteria’O: new radiographic infiltrate persistent for at least 48 hours, plus at least 2 of: temperature >38S”C or <35.O”C; leukocyte count of >lO X 109/L or <3 X 109iL; purulent sputum or change in character of sputum; or isolation of pathogenic bacteria from an endotracheal aspirate. Designated study radiologists masked to treatment allocation evaluated chest radiographs. Study person- nel forwarded to the methods center all case report forms; nurse

and physicians notes; intensive care unit flow sheets; laboratory, radiographic, procedure notes; and other relevant data collected from 48 hours before the clinical suspicion of VAP to 48 hours after the pneumonia resolved. Owing to the well accepted inaccuracies of the diagnosis of VAP and the absence of a reference standard,‘1x12 we adjudicated all suspicious cases of pneumonia, but not the charts of patients in whom the bedside clinicians had no suspicion of VAP.

Because of the large amount of time required for adjudication, we chose to construct four adjudication teams of two practicing internists, one in each pair being certified intensivists. These four pairs of adjudicators attended two training sessions during which they examined 10 charts of patients with clinical suspi- cion of VAP and agreed on the documentation required to make a final decision. During the trial, adjudicators received sets of five charts of patients with a clinical suspicion of VAP. When they returned their charts, they received the next consecutive set of five cases. Each reader in the pair examined documents indepen- dently, blinded to treatment group. Each reader completed a Clinical Pulmonary Infection Score composed of clinical signs and symp- toms (pneumonia defined as a score of at least 7)13 and made a summary judgement as to whether each patient had VAP Pairs of adjudicators resolved disagreement through discussion and achieved consensus on all patients. After initial training, adjudication teams received no further monitoring or feedback.

Analysis We report both crude agreement and kappa statistics within

each pair of adjudicators. We found variable rates of VAP among adjudication pairs. To examine whether some pairs received records of more severely il l patients with a higher probability of pneumonia, we recorded the mean Clinical Pulmonary Infection Score for patients adjudicated by each pair. We also examined the proportion of patients in each treatment group adjudicated by each team.

We conducted log-linear models to examine three-factor interactions, testing whether the relationship between adjudi- cated positive pneumonia cases and treatment group (ranitidine or sucralfate) differed by adjudication pair. Having found no important three-factor interaction, we examined whether the overall study odds ratio changed when we adjusted for adjudica- tion pair. We also examined what might have happened had we used only four individual adjudicators, rather than paired adjudicators with consensus in the event of disagreement. We calculated odds ratios and 95% confidence intervals for the 16 possible combinations of four adjudicators, obtained by choos- ing one number of each of the four original pairs.

RESULTS

According to the final consensus adjudication of pneumonia by each pair of adjudicators, 114 of 596 (19.1%) of patients in the ranitidine group devel- oped VAP compared with 98 of 604 (16.2%) in the sucralfate group (odds ratio 0.82, 95% confidence interval 0.61 to 1.10). Full results of this study are reported elsewhere,9 including significantly less gastrointestinal bleeding in patients receiving rani- tidine but similar mortality between the two groups. Table 1 shows the proportion of charts adjudicated

VENTILATOR-ASSOCIATED PNEUMONIA ADJUDICATION 161

Table 1. Adjudication of Patients Wiih Suspected Ventilator-Associated Pneumonia

Pair Adjudicated Positive (%I*

Agreement Raw Chance- Adjusted for Agreement Corrected

CPIS (%H (%H Agreementl Agreement With Consensus (%)I1

1 (n = 60) 50.0 66.7 49.2 33.3

2 (n = 24) 62.5 50.0 64.5 -5.9

3 (n = 99) 91.9 81.8 94.0 5.1

4 (n = 89) 69.7 78.7 73.1 53.6

This table displays the four pairs of readers and the number of charts that each pair reviewed.

*The proportion of charts adjudicated as positive for pneumonia. tThe proportion of charts adjudicated as positive, adjusted for the Clinical Pulmonary Infection Score (CPIS).‘3

*The raw agreement between readers in a pair.

80.0

91.7 50.0

50.0

§The kappa (chance-corrected agreement) between the pair of adjudicators. liThe proportion of consensus classifications agreeing with the original assessment of the first reader in each pair.

as positive (ie, the patient was considered to have VAP) by each adjudication pair. This ranged from 50% to 92% and was highest for pair 3. The mean Clinical Pulmonary Infection Score for patients reviewed by pairs 1 through 4 was 8.1,8.0, 8.8, and 8.4, respectively. When adjudication results were adjusted for the Clinical Pulmonary Infection Score, the proportion of cases adjudicated as positive changed (Table 1).

The crude agreement between readers in each pair varied from 50% to 82% and again was highest for pair 3. In contrast, the chance-corrected agree- ment ranged from -0.06 to 0.54. When adjudica- tors disagreed, the final consensus was split evenly between the two adjudicators’ initial opinions in pairs 3 and 4 (50% for both pairs); in the other two pairs, the consensus decision reflected the initial opinion of the dominant adjudicator in the pair in 80% and 92% of the decisions.

There were no three-factor interactions demon- strated during log linear modelling. The proportion of patients reviewed who were allocated to the sucralfate arm was 46.7%, 50.0%, 41.4%, and 5 1.7%, for pairs 1 through 4 inclusive. The overall unadjusted study odds ratio was 0.82 (P = .21), and the study odds ratio adjusted for adjudication pair was 0.85 (P = .27).

The point estimate of the odds ratios varied somewhat across the possible combinations of adjudicators from 0.80 to 0.95, although the confi- dence intervals around each estimate were overlap- ping (Table 2). These estimates were in all cases consistent with a nonsignificant trend toward a lower rate of VAP in the sucralfate group.

A total of 74 working days were needed to adjudicate all 282 charts, including personnel time

of the clinical centers, methods center, and adjudi- cators (Table 3).

DISCUSSION

Ensuring the validity and reproducibility of outcome measurement in randomized trials is cru- cial. Diagnosing ventilator-associated pneumonia in critically ill patients is dificult due to the nonspecific nature of the clinical and radiographic features,14 the modest reproducibility of histology as a reference standard,15 and the lack of agreement on a reference standard in either clinical practice or

Table 2. Study Odds Ratios for All Possible

Combinations of Adjudicators

Combination of Study Odds Ratio* Adjudicators* Sucralfatet Ranitidinet (95% CII

AF, KG, DC, GG 95/604 113/596 0.80 (0.59, 1.08)

AF, KG, DC, MM 101/604 1141596 0.85 (0.63, 1.14) AF, KG, HF, GG 90/604 104/596 0.83 (0.61, 1.13) AF, KG, HF, MM 96/604 IOU596 0.88 (0.65, 1.20) AF, MT, DC, GG 99/604 11 l/596 0.86 (0.64, 1.15) AF, MT, DC, MM 105/604 1121596 0.91 (0.68, 1.221 AF, MT, HF, GG 94/604 102/596 0.89 (0.66, 1.21) AF, MT, HF, MM 100/604 103/596 0.95 (0.70. 1.28) AS, KG, DC, GG 971604 1151596 0.80 (0.59, 1.08) AS, KG, DC, MM 103/604 116/596 0.85 (0.63, 1.14) AS, KG, HF. GG 92/604 106/596 0.83 (0.66, 1.20) AS, KG, HF. MM 98/604 107/596 0.89 (0.66, 1.20) AS, MT, DC, GG 101/604 113/596 0.86 (0.64, 1.15)

AS, MT, DC, MM 1071604 114/596 0.91 (0.68,1.22) AS, MT, HF. GG 96/604 104/596 0.89 (0.66, 1.21) AS, MT, HF, MM 102/604 105/596 0.95 (0.70, 1.28)

*All possible combinations of adjudicators are shown, includ- ing one member from each of the four adjudication pairs (letters refer to the initials of the adjudicators).

tThe proportion of patients adjudicated with pneumonia in the sucralfate and ranitidine groups using the pre-concensus opinion of each of the four adjudicators from the correspond-

ing row. *The corresponding study odds ratio.

162

Table 3. Personnel Costs of Pneumonia Adjudication

1 2 Activity Adjudicator Adjudicators

File preparation (min)” 20 20 File review (min)t IO 10 Copy & mail (min)t 15 20 Adjudication review (min)S 15 30 Consensus meeting (min)* 0 10 Data entry (min)t 10 20 Subtotal (min/file) 70 110 Subtotal (h/file) 1.2 1.8 Total (h/282 files) 329 517 Total (d/282 files) 47 74

*Clinical Center (research nurse preparation of materials). tMethods Center (study coordinator activities).

*Adjudication Committee.

To estimate the person-hours involved in the adjudication of 282 cases of pneumonia, we considered the file preparation

time of the research nurse at each site to be 20 minutes. Into

the methods center personnel costs, we incorporated file review, copying, mailing, and data entry. We estimated adjudi-

cator time to reach each chart and added consensus discus- sion for each file read by two adjudicators.

research. Therefore, adjudication of each suspi- cious event was necessary in this trial.

In summary, we found that adjudication teams had different thresholds for diagnosing VAP, lead- ing to different proportions of patients in whom they identified pneumonia, as indicated in Table 1. Second, only 1 of the 4 pairs was able to achieve conventionally acceptable levels of chance-cor- rected agreement, though the level of agreement in some pairs may be low because of the high proportion of cases adjudicated as positive. Third, in 2 pairs, consensus decision-making in the face of disagreement generally followed the opinion of 1 dominant member of the pair.

Given the differing distributions of VAP across pairs and the uneven distribution of patients in the 2 treatment groups, different standards of adjudica- tion could have biased the overall results of the trial. We therefore conducted an analysis in which we adjusted for adjudication pair and found results similar to the unadjusted analysis.

It is difficult to estimate the extent to which random or systematic error among the adjudicators might have attenuated our ability to detect a difference in VAP rates between the two treatment arms, if one truly exists. The best estimate of treatment effect in this trial was a clinically impor- tant but not statistically significantly lower VAP rate in patients receiving sucralfate; it is uncertain

COOK ET AL

whether the adjudication process would have a more substantial effect in a trial with a larger treatment effect. If instead of constructing teams of two adjudicators, we had relied on a single adjudi- cation for each patient (Table 2), the results of this study could have been different than those ob- served. These considerations support a more rigor- ous process of adjudication that might have taken more than the estimated 74 working day investment of research personnel and faculty time used for this study. In a previous trial, we calculated that approxi- mately 90 minutes of person time per patient was required for activities directly relating to the adjudi- cation of lung cancer status, or 6 months of full-time work for all 685 patients. l6

Adjudication of study eligibility, protocol adher- ence, and outcomes all can be subject to random and systematic error. For many outcomes in clinical intensive care unit trials for which no reference standard exists, adjudication procedures are used to enhance internal validity. The importance of mea- suring somewhat subjective outcomes in clinical research notwithstanding, potential bias inherent in their assessment needs to be weighed against potential bias introduced by an adjudication pro- cess itself.

Accordingly, we recommend the following strat- egies for ensuring rigorous adjudication of clinical outcomes without a gold standard definition. First, investigators should train adjudicators using a standardized, reliable process. Formal monitoring of adjudicator agreement throughout the trial may be useful; if agreement is less than acceptable (eg, kappa scores CO.40 or a similar conventional threshold), adjudicators may require additional training. Second, investigators should monitor the extent to which a single adjudicator dominates “consensus” decision-making; if this occurs, explo- ration of why this is the case may be illuminating. Third, if independent adjudication teams are cre- ated, investigators should ensure that readers have similar thresholds for deciding when an outcome event is present versus not present, not just at the beginning of the study, but throughout it. Monitor- ing strategies could include keeping track of the proportion of positive events in each group and having a random selection of charts periodically adjudicated by more than one team. When inter- team disagreement occurs, the adjudication teams should meet to discuss the reasons. Finally, despite

VENTILATOR-ASSOCIATED PNEUMONIA ADJUDICATION

these attempts to ensure more consistency across adjudication teams, some differences in criteria and their interpretation may persist. To minimize the chance that these differences bias the estimate of treatment effect, patient charts should be block randomized to adjudication teams. Educational videotapes may also be useful, which have been used to train, certify, and remediate clinicians involved in multicenter stroke studies to increase intra-rater and inter-rater reliability. l7

Not only the risk-benefit ratio but also the cost-benefit ratio of using adjudication committees to assess outcomes in clinical trials is important to consider. The financial costs of using an adjudica- tion process may be nontrivial, depending on the complexity and number of outcomes. The time- consuming nature of adjudication has implications

163

not only for study protocols but their grant budgets. The cost of adjudication committees may theoreti- cally divert research funds away from study infra- structure, data acquisition, or analysis. The adjudi- cation guidelines we suggest make the adjudication process more resource-intensive. More empirical research on the cost-benefit ratio of adjudicating major morbid events in critical care clinical re- search is warranted.

ACKNOWLEDGMENTS

We are grateful to members of the Canadian Critical Care Trials Group for their support of this study, particularly Drs. Mark Heule, John Marshall, Ben Guslits, Roman Jaeschke, and Jeff Lang. We thank the research nurses who collected the data, and Ms. Peggy Austin and Susan Troyan for their help co- ordinating the adjudication process.

REFERENCES

1. Start RD, Bury JP, Strachan AG, et al: Evaluating the reliability of cause of death in published clinical research. BMJ 314:271, 1997

2. B&ton M: Diagnostic errors discovered at autopsy. Acta Med Stand 196:203-210, 1974

3. B&ton M: The role of autopsy findings in multicenter treatment evaluations as exemplified by aspirin trials. Qua1 Assur Health Care 5:299-302, 1993

4. Greene I-IL, Richardson DW, Barker AH, et al: Classifica- tion of deaths as myocardial infarction as arrhythmic or nonarrhythmic (The Cardiac Arrhythmia Pilot Study). Am J Cardiol63:1-6, 1989

5. Gottlieb SS: Dead is dead-artificial definitions are no substitute. Lancet 349:662-663, 1997

6. Cairns JA, Connolly SJ, Roberts R, et al: Randomized trial of outcome after myocardial infarction in patients with frequent or repetitive ventricular premature depolarizations: CAMIAT. Lancet 349:6 682, 1997

7. Julian DG, Camm AJ, Frangin G, et al: Randomized trial of effect of amiodarone on mortality in patients with left- ventricular dysfunction after recent myocardial infarction: EMIAT. Lancet 349:667-674, 1997

8. Sprung CL, Finch RG, Thijs LG: International sepsis trial (INTERSEPT): Role and impact of the Clinical Evaluation Committee. Crit Care Med 23:1441-1447, 1996

9. Cook DJ, Guyatt GH, Marshall JC, et al: Sucralfate versus ranitidine for stress ulcer prophylaxis: A randomized double- blind trial. N Engl J Med 338:791-797, 1998

10. Centers for Disease Control: CDC definitions for nosoco- mial infections. Am Rev Respir Dis 139:1058-1059, 1989

11. Niederman MS, Torres A, Summer W: Invasive diagnos- tic testing is not needed routinely to manage suspected ventilator- associated pneumonia. Am J Respir Crit Care Med 150:5 574, 1994

12. Chastre J, Fagon JY: Invasive diagnostic testing should be routinely used to manage ventilated patients with suspected pneumonia. Am J Respir Crit Care Med 150:570-574, 1994

13. Pugin J, Auckenthaler R, Mili N, et al: Diagnosis of ventilated associated pneumonia by bacteriologic analysis of bronchoscopic and nonbronchoscopic bronchoalveolar lavage fluid.AmRevRespirDis 143:1121-1129, 1991

14. Andrews CP, Coalson JJ, Smith JD, et al: Diagnosis of nosocomial bacterial pneumonia in acute diffuse lung injury. Chest 80:254-258, 1981

15. Kirtland SH, Corley RS, Winterbauer SC, et al: diagnosis of ventilator-associated pneumonia: A comparison of histologic, microbiologic clinical criteria. Chest 112:445-457, 1997

16. Walter SD, Cook DJ, Guyatt GH, et al: Outcome assessment for clinical trials: How many adjudicators do we need? Controlled Clin Trials 18:27-42, 1997

17. Albanese MA, Clarke WR, Adams HP, et al: Ensuring reliability of outcome measures in multicenter clinical trials of treatments for acute ischemic stroke. Stroke 25: 1746-175 I, 1994