An agency of the European Union

Presented by: Name SurnamePosition/organisational descriptor

Quantitative methods of signal detectionFrançois MAIGNEN

2

Learning objectives•Current methods: Disproportionality analysis•Improvement of the current methods: Bayesian methods and (Bayesian) log-linear models•Comparison of the methods (limitations, stats vs clinical)•Star wars – a new hope? Longitudinal health records: better? Different and not devoid of biases•Future directions

3

Measures of disproportionate reportingMost of the methods routinely used in pharmacovigilance (spontaneous reporting systems) databases are based on measures of disproportionate reporting (i.e. ROR, PRR, BCPNN, MGPS, etc …).Basically: “Observed vs Expected” analysis in a given database i.e. % of reports involving a given reaction for a given medicine compared to the % of reports involving this reaction on the whole database

4

A spontaneous reporting system database

SRS Drug 1 Drug 2 Drug 3 Drug 4 Drug 5 Drug 6 Drug 7 … Drug N

Event 1 n11 n12 n13 n14 n15 n16 n17 … n1N

Event 2 n21 n22 n23 n24 n25 n26 n27 … n2N

Event 3 n31 n32 … … … … … … n3N

Event 4 n41 n42 … … … … … … n4N

Event 5 n51 n52 … … … … … … n5N

Event 6 n61 n62 … … … … … … n6N

… … … … … … … … … …

Event P nP1 nP2 nP3 nP4 nP5 nP6 nP7 … nPN

5

Drug 1 All other medicinal products

Total

Event 1 a c

All other reaction terms

b d

Total N = a + b + c + d

c + d

a + c

a + b

Proportional Reporting Ratio

6

Proportional Reporting RatioPRR = a/(a+b) / c/(c+d) WHAT DOES THAT MEAN IN PRACTICAL TERMS?a/(a+b) = Proportion of reports involving a specific adverse event among all the reports involving DRUG Ac/(c+d) = Proportion of reports involving THE SAME adverse event among all the reports of your database but DRUG A

7

Proportional Reporting RatioIf the rate of reporting of AE for drug 1 is similar to the rate of reporting of this AE for all the other products of the database, the PRR will be equal to 1 (same proportion of reports involving the reaction for drug A than for the other drugs) …

BUT … If the reaction is proportionately MORE reported with drug A than for the other products, the PRR will be increased (typically > 1). DIS-PROPORTIONALITY of reporting

8

Disproportionality analysis (example). CNS drug for which the total No of reports is 400, of these 20 reports of diarrhoea. All other products in the database (1 million reports excluding reports involving drug A), of these 50,000 reports of diarrhoea.

PRR = [20/400] / [50,000/1,000,000] = 1 (no SDR)

9

Disproportionality analysis (example)CNS drug for which the total No of reports is 400, of these 40 reports of drowsiness. All other products in the database (1 million reports excluding reports involving drug A), of these 25,000 reports of diarrhoea.

PRR = [40/400] / [25,000/1,000,000] = 4 (presence of a SDR)

10

Improvements of these methods•Considering possible confounding factors: stratification and log-linear models (ROR – see work from E. Van Puijenbroek)•Trying to circumvent low expected values or low case counts: Bayesian models (A. Bate & W. DuMouchel)•Other regression methods: LASSO and Bayesian logistic regressions (N. Noren, D. Madigan)•Clinical relevance not always clear or demonstrated•Some methods can be computationally demanding

11

Bayesian methodsBCPNN and MGPS rely on the same principle of conjugate prior distributions:•These methods will shrink the value of the measure of disproportionality using a Bayesian approach (prior based on existing dataset)•BCPNN: cell counts ~ Binomial dist., conjugate prior = beta•MGPS: cell counts ~ Poisson, conjugate prior = Gamma (mixture of Gammas).

12

Bayesian methodsAssume binomial y=7 successes, 20 trials.Non informative prior = Beta (2,4)

13

Thresholds - ARBITRARYAll these methods provide a ranking …Thresholds = arbitraryTrade-off between•Reviewing too many drug-event pairs(loss of operational benefit)•Missing some signals

No ADR ADR

14

Comparison of the methodsMethodological difficultiesNo gold standard / no standardised reference methodImprecision of what constitutes a signalRetrospective vs prospective evaluationImportance of clinical judgement. The added value of clinical evaluation is currently unknown (if any).

15

Comparison of methods

1

2

3

4

7

11

6

910

12

5

8

1

2

3

4

7

11

6

910

12

5

8

4

2

3

1

7

11

6

910

12

5

8

Meth.1

Meth.2

Meth.3

• Threshold 1: Meth. 2 = 5 true signals, meth. 1/3 = 4.• Threshold 1+2: Meth. 2=Meth.3• First 5 signals: Meth. 1 ≠ Meth. 2 = 3.

ADR No ADR

16

Comparison of methods (PRR vs BCPNN)

17

Performances of these methodsOperational benefit (screening of large databases)Anecdotal evidence (in opposition to structured) of signals discovered thanks to the quantitative methods (recent examples incl. D:A:D and MI)Time benefit in some cases (Hochberg & EV study)NND ~ 7/15 (depending whether the study is retrospective or prospective)Idea: Quant. Methods + DMEs/TMEs

18

New approaches to signal detectionDeviation of Obs. vs Expect. distr. from a fitted distribution (Jim)Modelling of the hazard function of the time to onset (DSRU / François) hazard # mechanismUse of longitudinal databases (record linkage and electronic health records – OMOP / Noren / Callreus) ~ incidence rate ratio• Same patients different time windows (A. Bate)• Hospital records of different patients (T. Callreus)

19

Merci Andrew

20

21

Interpretation / LimitationsPatients prescribed a PPI are associated withacute pancreatitis in the month after thePrescription (ICdiff positive)But graph shows that these patients are generallikely to have acute pancreatitis around the timeof PPI prescribing (In agreement with confounding by indication)

22

Record linkage

23

24

Interpretation / limitationsBias associated with the hospitalisation (confirmed by later events which occurred remotely after the administration of the vaccine)Spurious / unexplained associationsRely heavily on temporal association (Post hoc ergo propter hoc)

25

Longitudinal health recordsPowerful to detect some associationsEven if reporting artefacts do not influence this method, not devoid of other biases (selection, protopathic, misclassification, etc …)Much more complicated to implement (very large datasets, confidentiality aspects, linkage of records and interoperability of databases, etc …)

26

Future directions (?)•Performances in a prospective setting, added value of clinical judgement needs to be quantified•Duplicate detection•Other approaches to signal detection (use all the information included in the reports, PK/PD properties of the substances)•Surveillance networks (incl. spontaneous reporting)

27

Key elements•PRR is a measure of disproportionality of reporting in a specific database (observed vs expected value computed on the whole database)•A SDR indicates a possible signal which needs to be medically confirmed•All other methods proceed from the same logic•Data mining on EHR will probably improve the detection of signals (organised method of collection) but is not free of biases and is resources demanding

28

BibliographyGuideline on the use of statistical signal detection methods in EVDAS

http://www.ema.europa.eu/pdfs/human/phvwp/10646406enfin.pdf

Bate & Evans quantitaitve signal detection using ADR reporting. PDS 2009.

Hauben & Bate. Decision support methods for the detection of adverse events in post-marketing data. Drug Discovery Today. April 2009.

Evaluation of Signal Detection Methods for use in Prospective Post Licensure Medical Product Safety Surveillance [FDA Sentinel Initiative Safety Signal Identification Contract] – Report

http://www.fda.gov/Safety/FDAsSentinelInitiative/ucm149343.htm