Introduction to Systematic Review and Meta-Analysis

Brennan Spiegel, MD, MSHS

VA Greater Los Angeles Healthcare SystemDavid Geffen School of Medicine at UCLA

UCLA School of Public HealthCURE Digestive Diseases Research Center

UCLA/VA Center for Outcomes Research and Education (CORE)

Objectives• Define and discuss “systematic review”

– Contrast with “narrative review”

– Describe the 4 components of appropriate question

– Define steps for a successful search strategy

– Review construction of evidence tables

• Define and discuss “meta-analysis”– Describe calculations of summary estimates

– Review how to evaluate for heterogeneity

– Define fixed versus random effects models

– Describe “funnel plots” for publication bias

Purposes of Systematic Review and Meta-Analysis

• Combine data from multiple studies to arrive at summary conclusion

• Calculate summary estimate of effect size– May overcome Type II error

• Test for and explain heterogeneity

• Test for publication bias

• Inform decision models

Some Basic Premises

• All meta-analyses must begin with a systematic review

• Knowledge and application of statistical models cannot overcome inadequacies in qualitative systematic review

• Qualitative approach is primary – quantitative approach is secondary

Decision Analysis and Systematic Review

If decision analysis is the engine for making decisions under conditions of uncertainty, then systematic review provides the fuel to run the engine.

The Nature of Meta-Analysis

“Meta-analysis should not be used exclusively to arrive at an average or ‘typical’ value for effect size. It is not simply a statistical method but rather a multicomponent approach for making sense of information.”

• Diana Petitti, in Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis, Oxford U Press 2000

Feature Narrative Review Systematic Review

Question Broad in Scope Focused

Sources and Search

Not usually specified, potentially biased

Comprehensive sources and explicit search strategy

Selection Not usually specified, potentially biased

Criterion-based selection, uniformly applied

Appraisal Variable Rigorous critical appraisal

Synthesis Often a qualitative summary

May include quantitative summary (meta-analysis)

Systematic versus Narrative Review

Adapted from Mulrow C, Cook D: Systematic Reviews; ACP Press 1998

Steps to Systematic Review

Step 1 Define focused question

Step 2 Define inclusion / exclusion criteria

Step 3 Develop search strategy

Step 4 Identify databases to search

Step 5 Run search and abstract data

Step 6 Compile data into evidence tables

Step 6 Pool data

Step 7 Interpret data

Four Elements of a Systematic Review Question

1. Type of person involved

2. Type of exposure experienced• Risk factor• Prognostic factor• Intervention• Diagnostic test

3. Type of control with which the exposure is being compared

4. Outcomes to be addressed

Adapted from Mulrow C, Cook D: Systematic Reviews; ACP Press 1998

Example of Inadequate Question

Does smoking cause lung cancer?

Exposure Outcome

Better Question

What is the relative risk of…

lung cancer…

in cigarette smokers…

compared to non cigarette smokers?

Control

Exposure and Type of Person

Outcome

Inadequate Question

Better

Do SSRI improve health related quality of life in patients with depression compared with Elavil?

Are SSRIs, like Prozac, effective for depression?

Feels Better

Does not Feel Better

Feels Better

Does not Feel Better

Chance Nodes

Decision Node

Depression

Developing Inclusion / Exclusion Criteria

• Think of each study as a patient in an RCT

– Must carefully specify inclusion and exclusion criteria to include in the study

• Criteria should mirror carefully formulated question

• Criteria should strike a balance in scope – avoid being too narrow or too broad

• Make sure you target clinically relevant outcomes

• Consider limiting to RCTs if possible

Considerations for Inclusion / Exclusion Criteria

• Definition of target disease/condition

• Stage or severity of condition

• Patient sub-groups (age, sex, symptoms)

• Population or setting (community, hospital)

• Intensity, timing, or duration of exposure

• Method of delivery (e.g. group therapy or individual therapy, oral or IV, etc)

• Type of outcome (survival, HRQOL, adverse events)

• Study design (experimental vs. observational; randomized vs. unrandomized)

Search Strategy Principles

• Balance sensitivity with specificity

– Highly sensitive search strategy may yield untenable number of titles by casting the net too widely

– Highly specific search may yield too few titles and miss key articles by failing to cast a wide enough net

• Said another way:

“The overall goal of any search strategy is to identify all of the relevant material and nothing else.”

• Diana Petitti, in Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis, Oxford U Press 2000

Components of Search Strategy

• Select target databases– US National Library of Medicine (MEDLINE)– EMBASE– “Fugitive” or “gray” literature – Cochrane Database of Systematic Review

• Determine language restrictions

• Establish time horizon for search

• Operationalize targeted material with MeSH terms, text words (tw), and publication types (pt)

• Operationalize excluded material and set after “NOT” operator

Example of Defining the Search Strategy

Group Search Terms Significance of Grouping

1 RANDOMIZED-CONTROLLED-TRIAL OR CONTROLLED-CLINICAL-TRIAL OR RANDOMIZED-CONTROLLED-TRIALS OR RANDOM-ALLOCATION OR DOUBLE-BLIND-METHOD OR SINGLE-BLIND-METHOD OR CLINICAL-TRIAL OR CLINICAL-TRIALS OR (CLIN* NEAR TRIAL*) OR ((SINGL* OR DOUBL* OR TREBL* OR TRIPL*) NEAR (BLIND* OR MASK*))

Filter for Randomized

Controlled Trials

2 (ROFECOXIB OR CELECOXIB OR VALDECOXIB OR ETORICOXIB OR COXIB OR COX-2 OR CYCLOOXYGENASE-2) OR ((NAPROXEN OR DICLOFENAC OR IBUPROFEN OR KETOROLAC OR MELOXICAM OR INDOMETHACIN OR KETOPROFEN OR NABUMETONE OR ETODOLAC OR PIROXICAM OR SULINDAC OR ASPIRIN OR ASA OR SALSALATE OR NSAID) AND (LANSOPRAZOLE OR OMEPRAZOLE OR ESOMEPRAZOLE OR RABEPRAZOLE OR PANTOPRAZOLE OR PROTON PUMP INHIBITOR*))

Targeted Content Keywords

3 (TG=ANIMAL OR LETTER [pt] OR EDITORIAL [pt] OR REVIEW [pt] OR NEWS [pt] OR CANCER OR CARCINOMA OR MALIGNANCY OR NEOPLASM)

Excluded Study Types and Content

Spiegel et al. Am J Med 20061 AND 2 NOT 3

Another Example

Spiegel et al. Alim Pharm Ther 2007

Example Search Strategy

Spiegel et al. Arch Int Med 2001

Example Flow Diagram

Spiegel et al. Arch Int Med 2001

Other Best Practices for Systematic Review

• Identify titles, abstract, and manuscripts in 3 separate steps

• Two reviewers search in tandem– Test set for training

– Target high inter-rater reliability (>0.7)

• Develop standardized abstraction form for manuscript review

• Transfer data onto evidence tables

Example of Data Abstraction Using Evidence Tables

Spiegel et al. Am J Med 2006

Another Example

Spiegel et al. Arch Int Med 2001

Evaluating Study Quality

Quality Indicator Points Assessed

Was study described as “randomized?”

If yes, score +1If no, score 0

If study randomized, was there concealed allocation?

If yes, score +1If no, score -1

Was study described as “double-blind?”

If yes, score +1If no, score 0

If study blinded, was it appropriate?

If yes, score +1If no, score -1

Was there a description of withdrawals and dropouts?

If yes, score+1If no, score 0

Jadad et al. Control Clin Trials 1996

Abstracting Data: 2x2 Table

Exposed Unexposed

Event

No Event

NE NU

NE - nE NU - nU

nUnE

= nE NERiskExposed = nE NERiskUnexposed

Abstracting Data: 2x2 Table

Exposed Unexposed

Event

No Event

C D

BA

OR = AD / BC

Before you Combine Data

• Look at the studies you’ve collected. Ask yourself, are they qualitatively similar in terms of 4 key characteristics:

– Patient population

– Exposure

– Comparision group

– Outcome

Before you Combine Data

• Test for statistical evidence of heterogeneity

– Cochrane’s Q statistic

– I2 statistic

• Measure degree of between-study variance

– Wider the variance, higher the heterogeneity

• Tests to see if you are combining “apples” and “oranges”

Cochrane’s Q Statistic

• Tests the sum of the weighted difference between the summary effect measure and the measure of effect from each study

• Compared against 2 distribution with k-1 degrees of freedom, where k=N of studies

• Null hypothesis is that studies are homogeneous

• Test has low sensitivity for detecting heterogeneity, especially when small N of studies – most use p<0.1 for significance

Visual Evidence of Heterogeneity

Juni et al. Lancet 2004

I2 Statistic

• Improves upon Q statistics because less conditional on sample size of studies

• Describes the percentage of total variation across studies that is due to heterogeneity rather than chance.

• I2 calcuation based on Q as follows:

I2 = 100% x (Q-df) / QHiggins et al. BMJ 2003;327

Interpreting I2 Statistic

Range of 0-100%

0-25% = “Low” Heterogeneity

26-50% = “Moderate” Heterogeneity

>50% = “High” Heterogeneity

Higgins et al. BMJ 2003;327

What if there is Heterogeneity?

• More important to explain heterogeneity than to force a summary estimate

• Some turn to “random effects model” (more soon – not a good solution for heterogeneity)

• Can explain heterogeneity through various mechanisms:

– Perform sensitivity analyses stratified by key study characteristics

– Perform meta-regression if sample size permits

Example of Sub-Group Analyses

Watson et al. Curr Med Res Opin 2004

Fixed vs. Random Effects Models

• Two types of statistical procedures to combine data from multiple studies:

– Fixed effects models

• Mantel-Haenszel Method

• Peto Method

– Random effects models

• DerSimonian & Laird Method

Fixed Effects Models

• Inference is conditional on the studies actually done – i.e. the studies at hand

• Assumes there are no other studies outside of the group being evaluated

• Focuses on “within study variance,” which assumes a fixed effect in each study with a variance around the study

– Weight of each study is thus driven by sample size

Random Effects Models

• Inference is based on the assumption that studies in analysis are random sample of larger hypothetical population of studies

• Assumes there are other studies outside of the group being evaluated

• Focuses on both “within study variance” and “between study variance”

– Heterogeneity driven by 2 factors: random variation of each study around fixed effect, and random variation of each study compared to other studies

Within Study Variance

Between Study Variance

More on Fixed vs. Random Models

• Fixed effects model answers question:

“Did the treatment produce benefit on average in the studies at hand?”

• Random effect model answer question:

“Will the treatment produce benefit on average?”

More on Fixed vs. Random Models

• Random effects model usually more conservative than fixed effects model

– Random effects usually has narrower confidence intervals

• When between-study variance is large, within study variance becomes relatively less important, and large and small studies tend to be weighted equally

• Fixed effect is special case of random effect in which between-study variance is zero

• If there is no heterogeneity, then fixed and random effects models yield similar results

Random Effects Model as Solution for Heterogeneity

“The use of the random-effects model is not a defensible solution to the problem of heterogeneity… When there is lack of homogeneity, calculating a summary estimate of effect size is of dubious value… Random effects models should not be used to ‘adjust for’ or ‘explain away’ heterogeneity. The main focus should be on trying to understand sources of heterogeneity.”

- Diana Petitti

Weighted Mean OR = wi*ORi /

W

n

i=1

Where

W = wi

n

i=1

wi = 1 / variancei ORi = ai di/ bi ci

Mantel-Haenszel Method

Coxibs vs. NSAIDS: Dyspepsia Forest Plot

Spiegel et al. Am J Med 2006

Running Meta-Analysis in STATA

Spreadsheet set-up:

Study N_Group_A N_Group_B n_Event_Group_A n_Event_Group_B

Jones 10 10 5 5

James 20 18 3 8

Johnson 100 95 25 40

Marshall 300 280 59 88

Gen n_No_Event_Group_A=N_Group_A-n_Event_Group_A

Gen n_No_Event_Group_B=N_Group_B-n_Event_Group_B

Metan n_Event_Group_A n_No_Event_Group_A n_Event_Group_B n_No_Event_Group_B, rr fixed xlab (.8,1,2) texts(5) label(namevar=study)

Publication Bias

• Editors and journal readers like big, positive studies

• Small, negative studies are inherently less exciting or publishable

• When small negative studies are suppressed, there is an artificially inflated effect

Symmetric Funnel Plot

Effect Size

Sample Size

Asymmetric Funnel Plot

Effect Size

Sample Size

Asymmetric Funnel Plot

Effect Size

Sample Size

Begg's funnel plot with pseudo 95% confidence limits

lo

go

r_6

mo

s.e. of: logor_6mo0 .5 1 1.5

-2

0

2

4

Study Size (SE)

Stu

dy E

ffec

t (L

og O

dds)

Smaller Studies

Larger Studies

Larger Effect

Smaller Effect

Question

Does every probability estimate mandate a full

systematic review and/or meta-analysis?

Answer NO!

Considerations for Determining Rigor of Probability Development

• A priori hypotheses based on literature

• Physical location of variable in tree

• Impact of variable in sensitivity analysis

• Editor pet-peeves and targeted journal for submission