1
Ann Versporten,
Ingrid Morales, Carl Suetens
IPH, wednesday seminar: May 7, 2003
Scientific Institute of Public Health
Data validation study of the National surveillance of
nosocomial infections in intensive care units
2
Overview• Background: overview national surveillance ICU• Reasons for validation• Validation study
– Aims– Methods– Results
• Pneumonia• Bacteraemia
– Discussion– Conclusions– Recommendations
3
Background: National surveillance ICU
• 1996: Start National Surveillance of Hospital Infections (NSIH) : intensive care component (Pneumonia & Bacteraemia) – HELICS-based protocol (Hospitals in Europe
link for Infection Control through Surveillance)– patient-based surveillance: 1 file by patient, +
infection file if ICU-acquired PN or BAC– Nosocomial: infection acquired during hospital
stay (admitted >48h in ICU)
4
Background: National surveillance ICU
• Objective: to follow-up nosocomial-infection rates
• Risk-adjusted infection rates are used as external benchmarks for comparison purposes
5
Methods: Data collection for ICU surveillance
1. Data at admission
2. Day-by-day e.g. central venous catheter, mechanical
ventilation, antibiotic use
3. Infection data e.g. diagnostic criteria of PN, origin of BSI
4. Data at discharge
6
Reasons for validation
• Assessment of the validity of the findings
• Need to evaluate the accuracy of infection data reported to the NSIH program
7
Validation study
8
Main aim
• Validate reported ICU-surveillance data (ICU protocol: PN & Bac) against a reference gold standard
• Evaluate the accuracy of all data reported to the surveillance
• Evaluate the credibility of the surveillance
9
Specific aims
• Exhaustivity (completeness) denominator
• Sensitivity: probability of reporting a true PN & Bac to the ICU-surveillance
• Specificity: probability of reporting a PN & Bac as negative to the ICU-surveillance if the disease is truly absent
• Positive predictive value
• Negative predictive value
11
Methods - 1
• Sampling of hospitals: Systematic sampling of 45 hospitals on the
base of a list of hospital-trimesters
(ICU participation period 01/01/1997 – 31/12/1999)
• Replacement: later period accepted
• Informed consent, voluntary participation
• Retrospective chart review methodology
12
Methods - 2: Research program
Sampling of patient files: All reported PN+ & Bac+ (from surv.) All records with a positive hemoculture reported
on a laboratorium list (for all admitted patients on ICU) (estimation false-neg Bac)
A 20% random sample of the negative files (estimation of false-neg PN)
Estimation of exhaustivity of denominator on the base of administrative lists of ICU-admissions
13
Methods - 3• Calculation Se, Sp and Predictive values
“gold standard” = research team
• Trained data collectors (IPH) Application protocol definitions
validation: uniform & standardised evaluation = blind discrepant infections: reviewed by other
colleague
• Confidential & anonymous treatment of patient data
14
Methods - 4
• National results
• No individual hospital results, only discussion at end validation proccess Quality of dataQuestions
15
Results - 1
• 563 investigated patient files in analysis: pts staying >24h in ICU (23 hospitals)
• Infections reported by hospitals to surveillance: 147 Pneumonia 49 Bacteraemia
• Type of ICU: 91% polyvalent• Size of ICU: mean 10 beds • Length of stay: median = 4,7 days
16
Results - 2
• Exhaustivity of denominators:– For all patients staying >24h in ICU
72,8%
– For all patients staying >48h in ICU81,2%
17
Results - 3: Pneumonia
(106/133)*147=117.2(24/430)*1843=102.9
Validation+ - Total
Surv. + 117 30 147- 103 1740 1843
Total 220 1770 1990
Validation+ - Totaal
Surv. + 106 27 133- 24 406 430
Total 130 433 563
All PN inf.file &/or bdb Freq %
1 147 7.392 1843 92.61
Total 1990 100
Results of validation study for PN (inf. file &/or dbd) Results from surveillance for PN (inf. file &/or dbd)
Results applied on total sample(proportional balancing to files not been validated
18
Results - 4: Bacteraemia
Validation+ - Total
Surv. + 32 17 49- 22 1919 1941
Total 54 1936 1990
Validation+ - Total
Surv. + 32 12 44- 22 497 519
Total 54 509 563
All Bac inf.file &/or bdb Freq %
1 49 2.462 1941 97.54
Total 1990 100
Results of validation study for Bac (inf. file &/or dbd) Results from surveillance for Bac (inf. file &/or dbd)
Results applied on total sample(proportional balancing to files not been validated
19
Results: SE & SP
Se % (95% CI) Sp % (95% CI)
Pneumonia
Infection file 32,7 (25,2-41,2) 98,5 (97,4-99,2)
Inf.file &/or dbd 53,2 (43,5-62,7) 98,5 (97,4-99,0)
Bacteraemia
Infection file 48,1 (29,2-67,6) 99,3 (98,5-99,7)
Inf.file &/or dbd 59,3 (39,0-76,9) 99,1 (98,2-99,6)
20
Results: predictive values
PPV (%)(CI) NPV (%)(CI)
Pneumonia
Infection file 78,6 (65,6-87,9) 85,9 (83,5-87,9)
Inf.file &/or dbd 79,6 (68,3-87,8) 88,9 (86,8-90,8)
Bacteraemia
Infection file 65,0 (40,9-83,7) 97,3 (96,0-98,2)
Inf.file &/or dbd 65,3 (43,6-82,4) 97,3 (96,0-98,2)
21
Discussion - 1
• Exhaustivity denominator: improvement possible – risk of bias, e.g. if only high risk patients included
• Pneumonia: low Se., good Sp.
• Bacteraemia: low Se., good Sp.
22
Discussion – 2
• Possible reasons for lack of sensitivity– 30% of the results originate from 1997 (start
surv. NI in ICU). – 50% of the collected data correspond with the 3
first surveillance trimesters that hospitals participated to our ICU surveillance. = Explanation of lack of accuracy in the
interpretation of the protocol ?
23
Who are those missed patients ??
Why are there so many false negative Pneumonias ?
24
Characteristics false negative PN
Pneumonia N % mort.
mean length of
stay (days)
n ventilation
daysmean
SAPS II
median PN Risk Score
% with micro-org.
% pts with >=1 (other)
missing value
mean stay post infection
(days)
True + PN 93 24.7 20.0 7.1 42.6 47 84 8.6 13.9
False - PN 23 30.4 13.6 8.9 35.1 41 70 21.7 9.3
True - PN 1626 6.7 5.6 1.7 31.3 19 - 15.7 -
28
Factors influencing the Se. & Sp. of the infection data
• Who collects data ?
• Who decides whether a PN should be reported or not ?
• Criteria of bloodculture?
• Adherence to protocol definitions
• Degree of workload (ratio pat.-staff)
• Size of hospital
• …
29
Conclusions
• Exhaustivity varies for each hospital, but remains satisfactory in general
• Bac more accurately reported than PN (Se)
• Seldomly infections reported which were not a nosocomial infection (Sp)
• Absence of a gold standard ! (problem for diagnostic of PN)
30
Conclusions (next)
• Establishing Se & Sp only possible at the end of validation studyPreliminary conclusions:
Sensitivity rather low (identification of a NI through surveillance)Specificity is high (% files truly classified as non-NI)
Low Se has also been reported by the CDC: “The data collectors detected over 2,5 times as many PN, ..” (Emori, Edwards, et al. 1998)
31
Recommendations
• Training of professionals in charge of surveillance (Ehrenkranz, Shultz, et al. 1995)
case definitions (e.g. PN-diagnostic: use of micro-biologic reports & AB-administration)
surveillance-methods
• Simplification of protocol
• Development of electronic surveillance
32
Recommendations (next)
• Validation on continuous basis Training on the field Optimalisation contacts IPH / hospitals