Data Mining in VAERS to Enhance Vaccine Safety Monitoring at the FDA

Robert Ball, MD, MPH, ScM

Dale Burwen MD, MPH

M. Miles Braun, MD, MPH

Division of Epidemiology

Office of Biostatistics and Epidemiology

DIMACS, October 18, 2002

What is the Vaccine Adverse Event Reporting System (VAERS)?

– National system for surveillance of adverse events after vaccination initiated by National Childhood Vaccine Injury Act 1986 and established 1990

– Jointly managed by FDA and CDC

– Reports received from health professionals, vaccine manufacturers, and the public

Post-licensure Safety Monitoring

• How we do it– VAERS

• Potentially rapid detection of signal of new safety concern

• Rarely allows determination of causality

– Enhanced surveillance• Obtain standardized information on reports

– Controlled studies of hypothesized causal relationships raised in surveillance

– Communicate results

Uses of VAERS

• Detecting unrecognized adverse events

• Monitoring known reactions

• Identifying possible risk factors

• Vaccine lot surveillance

Limitations of VAERS

• Reported diagnoses are not verified• Lack of consistent diagnostic criteria• Wide range in data quality• Underreporting• Inadequate denominator data• No unvaccinated control group• Usually not possible to assess whether a

vaccine caused the reported adverse event

Analysis of VAERS Data

• Describe characteristics and look for patterns to detect “signals” of adverse events plausibly linked to a vaccine

• Signals detected through analysis of VAERS data almost always require confirmation through a controlled study

Fundamental Problem in Assessing Spontaneous Reports

• VAERS ~10-15K reports / year

• AERS ~20K reports / year (CBER)

• How can a sensitive system to detect potential product problems not be overloaded and overwhelmed by information to which we have to respond?

“Data Mining”• Identify events reported more commonly for one

product than others – Proportional Reporting Ratios (PRR)– Empirical Bayesian Geometric Mean (EBGM) – Don’t account for medical knowledge or biases in reporting

• EBGM algorithm implemented by Lincoln Technologies and PPD Informatics – VAERS Data Mining Environment (VDME)

• PRR algorithm implemented in standard packages (e.g. SAS, STATA) on an ad hoc basis

Proportional Reporting Ratio• Compares the adverse event profile of one vaccine to

other vaccines

Number of reports with

Adverse Other Adverse Total Event Y Events

Vaccine X a b (a+b) Other vaccines c d (c+d)

PRR = [a/(a+b)] / [c/(c+d)]

Proportional Reporting Ratio

• Compares the adverse event profile of one vaccine to other vaccines

• Evans has proposed using PRR 2, n 3, and chi square 4 as criteria for selecting pairs for further evaluation

Background: Empirical Bayesian Data Mining

• Similar to PRR in comparing one vaccine to others

• Calculates observed and expected frequencies– Observed: # of reported events/vaccine– Expected: Based on overall frequency of the event

for all vaccines, and the total # of reports of the vaccine of interest

• Identifies cells with very small expected counts – accounts for the instability of the small number

Empirical Bayesian Data Mining

• Ranks vaccine-event combinations by Empirical Bayesian Geometric Mean (EBGM)

• Dumouchel has proposed EBGM 2 as criterion to select pairs for further evaluation

• Multi-item Gamma Poisson Shrinkage (MGPS) algorithm detects multi-way combinations – V=vaccine; S=symptom

• V S

• V S S

• V S S S

Rotavirus Vaccine-Intussusception

• Clinical Trials Signal• Wild type RV & intussusception study• FDA - licensure• CDC - recommendations for use• Post-marketing Surveillance (VAERS)• Background rates• Population-based incidence rates• Withdrawal

Rotavirus Vaccine and Intussusception:Signal Emergence

Vaccine Profiles

Anthrax Vaccine: 3-Dimensional Assessment (V-S-S)

Effect of Stratification on EBGM:Anthrax Vaccine and Selected COSTARTS

0

2

4

6

8

10

12

14

16

18

Crude Age Stratified Age-SexStratified

AmnesiaDerm contactHypogonad maleNodule SubcutPrev reactSkin dryTinnitis

EBGM

Selection of “Item Sets” for Empirical Bayesian Data Mining

• The choice of “Item Sets” influences the Multi-item Gamma Poisson Shrinkage (MGPS) algorithm

• Currently all combinations (e.g. 2D v-v, s-s, v-s where v=vaccine; s=symptom)

• If input is restricted to only v-s combinations the magnitude of the EBGM and rank for pairs with small numbers are affected

• Appropriate selection of Item Sets needs systematic evaluation

Effect of Item Set Selection on EBGM

Challenges• What is the best method?

– Bayesian vs. PRR vs. other?– What are criteria for making this decision?

• How should each method be applied and interpreted?– What level of PRR/EBGM?– How should statistics be interpreted?

Challenges• Should data mining methods be used for automated

screening or as analytic tools?– Importance of stratification suggests need for intermediate level

epi/stat sophistication in users– Users need training to properly interpret results

• Computing resources– Substantial effort required for data preparation– Software needs user-friendly features to enhance end-user

control over:• Defining data subsets of interest• Stratification• Combining adverse event terms• Selecting item sets prior to data mining

Challenges• Usual method of monitoring for signals:

• Physician review of individual reports as they arrive• Physician review of serious reports• Committee review of serious reports at weekly meeting• Physician review of monthly numerical summaries of selected

vaccines• Periodic vaccine or disease-specific surveillance summaries

• Where does data mining best fit in this process?• How can data mining results be best communicated to

decision makers, health care providers, and the public?

Next Steps and Future Challenges• Continue using PRR and Empirical Bayesian

methods in routine practice

• Systematic comparison of methods

• Simulation study in collaboration with CDC

• Large size of AERS database, especially with 2 way and 3 way interactions– Is simpler better? e.g. PRR with chi-square

• Drug dictionary in AERS

Summary

• Automated summary of a large amount of data

• Potential for improving usual methods of monitoring for signals– Other methods should also be considered

• Further understanding and experience is needed

Acknowledgments

• FDA– Manette Niu, Phil Perucci, other CBER staff, Ana

Szarfman and other CDER staff

• CDC– Henry Rolka, Vitali Poole, Penina Haber, John

Iskander, and other CDC staff

• Others– Lincoln Technologies, Inc. – PPD Informatics– William DuMouchel

Selected References

• Dumouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. American Statistician 1999;53:177-190.

• Evans SW, Waller PC, Davis S. Use of proportional reporting ratios for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf 2001;10:483-486.

• Niu MT, Erwin DE, Braun MM. Data mining in the US Vaccine Adverse Event Reporting System (VAERS): early detection of intussusception and other events after rotavirus vaccination. Vaccine 2001;19:4627-4634.