How to Use Systematic How to Use Systematic ReviewsReviews

Primary Care ConferencePrimary Care Conference

June 27, 2007June 27, 2007

David Feldstein, MD David Feldstein, MD

ObjectivesObjectives

Systematically analyze validity of a Systematically analyze validity of a meta-analysismeta-analysis

Discuss results of a meta-analysisDiscuss results of a meta-analysis Apply evidence from a meta-analysis to Apply evidence from a meta-analysis to

patient care decisions patient care decisions

Systematic Review vs. Systematic Review vs. Meta-AnalysisMeta-Analysis

Systematic ReviewSystematic Review– Critical evaluation of research to answer a Critical evaluation of research to answer a

focused question where an attempt is focused question where an attempt is made to made to evaluate all available researchevaluate all available research

– Also known as an overviewAlso known as an overview Meta-analysisMeta-analysis

– Quantitative strategy to combine the Quantitative strategy to combine the resultsresults of multiple studies into a single of multiple studies into a single pooled estimatepooled estimate

CaseCase

A 55 y.o. male with a history of an MI in 1995 A 55 y.o. male with a history of an MI in 1995 presents to clinic for an annual physical. presents to clinic for an annual physical.

He has been feeling well, but is concerned He has been feeling well, but is concerned about the risk of future MI’s and death.about the risk of future MI’s and death.

He is on ASA, statin, ACE and B-blockerHe is on ASA, statin, ACE and B-blocker His brother who also has CAD told him that His brother who also has CAD told him that

he should take vitamin E he should take vitamin E Should I recommend vitamin E?Should I recommend vitamin E?

QuestionQuestion

In patients with CAD does vitamin E In patients with CAD does vitamin E supplementation decrease the risk of supplementation decrease the risk of death?death?

SearchSearch

PubMed PubMed – Clinical QueriesClinical Queries

» Find Systematic ReviewsFind Systematic Reviews» ““Vitamin E and cardiovascular disease”Vitamin E and cardiovascular disease”

ArticlesArticles

Eidelman RS. Hollar D. Hebert PR. Lamas Eidelman RS. Hollar D. Hebert PR. Lamas GA. Hennekens CH. Randomized trials of GA. Hennekens CH. Randomized trials of vitamin E in the treatment and prevention of vitamin E in the treatment and prevention of cardiovascular disease. cardiovascular disease. Archives of Internal Archives of Internal Medicine. 164(14):1552-6, 2004 Jul 26.Medicine. 164(14):1552-6, 2004 Jul 26.

Shekelle PG. Morton SC. Jungvig LK. Udani J. Shekelle PG. Morton SC. Jungvig LK. Udani J. Spar M. Tu W. J Suttorp M. Coulter I. Spar M. Tu W. J Suttorp M. Coulter I. Newberry SJ. Hardy M. Effect of supplemental Newberry SJ. Hardy M. Effect of supplemental vitamin E for the prevention and treatment of vitamin E for the prevention and treatment of cardiovascular disease. cardiovascular disease. Journal of General Journal of General Internal Medicine. 19(4):380-9, 2004 Apr.Internal Medicine. 19(4):380-9, 2004 Apr.

Steps in Critical Appraisal Steps in Critical Appraisal

Is the study valid?Is the study valid? What are the results?What are the results? Can I apply the results to my patient?Can I apply the results to my patient?

Critical Appraisal Worksheet - Critical Appraisal Worksheet - ValidityValidity

1.1. Did the overview explicitly address a Did the overview explicitly address a sensible clinical question?sensible clinical question?

2.2. Was the search for relevant studies detailed Was the search for relevant studies detailed and exhaustive?and exhaustive?

3.3. Were the primary studies of high Were the primary studies of high methodologic quality?methodologic quality?

4.4. Were assessments of studies reproducible?Were assessments of studies reproducible?

Guyatt G, Rennie D. Users' guides to the medical literature : a manual for evidence-based clinical practice. Chicago, IL: AMA Press; 2002.

Critical AppraisalCritical Appraisal

First read the abstractFirst read the abstract Each article has sections that are Each article has sections that are

labeledlabeled Look at the labeled areas to answer the Look at the labeled areas to answer the

corresponding validity questions for corresponding validity questions for your assigned articleyour assigned article

Feel free to look at the other article if Feel free to look at the other article if time permitstime permits

ValidityValidity

Did the overview address a sensible Did the overview address a sensible clinical question?clinical question?

ValidityValidity

Basic Principles - Sensible questionBasic Principles - Sensible question– Is it reasonable to expect similar results Is it reasonable to expect similar results

across trials?across trials?– Consider:Consider:

» PopulationPopulation» InterventionIntervention» OutcomesOutcomes

– Take into account inclusion and exclusion Take into account inclusion and exclusion criteriacriteria

ValidityValidity

Was the search for relevant studies Was the search for relevant studies detailed and exhaustive?detailed and exhaustive?

ValidityValidity

Basic Principles – SearchBasic Principles – Search– Sensitive search to find all published dataSensitive search to find all published data

» Multiple databasesMultiple databases

– Avoid language biasAvoid language bias» Non-English speaking authors more likely to Non-English speaking authors more likely to

publish positive studies in English journalspublish positive studies in English journals

– Avoid publication biasAvoid publication bias» Larger studies or positive results more often Larger studies or positive results more often

publishedpublished» Need to search for unpublished dataNeed to search for unpublished data

FP

ValidityValidity

Exhaustive search:Exhaustive search:– MEDLINE, CINAHLMEDLINE, CINAHL– Cochrane controlled trials registry Cochrane controlled trials registry

(CENTRAL)(CENTRAL)– Foreign language literature (EMBASE and Foreign language literature (EMBASE and

LILACS) LILACS) – References cited in primary sourcesReferences cited in primary sources– Conference proceedingsConference proceedings– Unpublished data from experts in field, Unpublished data from experts in field,

authors, drug manufacturersauthors, drug manufacturers

ValidityValidity

Were the primary studies of high Were the primary studies of high methodologic quality?methodologic quality?

Jadad ScoreJadad Score

Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996

ValidityValidity

Basic Principles - Quality of individual Basic Principles - Quality of individual studiesstudies– Should look at criteria similar to User’s Should look at criteria similar to User’s

Guides for individual studiesGuides for individual studies– Often use predefined scoring systems, but Often use predefined scoring systems, but

is not necessaryis not necessary– Lower quality trials overestimate treatment Lower quality trials overestimate treatment

benefitbenefit» Allocation ConcealmentAllocation Concealment» BlindingBlinding

Jadad Scores (Shekelle)Jadad Scores (Shekelle)

TrialTrial Jadad ScoreJadad Score

CHAOSCHAOS 33

GISSIGISSI 33

HaegerHaeger 00

HOPEHOPE 33

SPACESPACE 44

ValidityValidity

Were assessments of studies Were assessments of studies reproducible?reproducible?

ValidityValidity

Basic Principles – Assessment Basic Principles – Assessment reproduciblereproducible– 2 people independently 2 people independently

»Evaluate studies for inclusion Evaluate studies for inclusion »Abstract dataAbstract data»Assess validity Assess validity

– To prevent bias or random errorTo prevent bias or random error– Check agreement between reviewersCheck agreement between reviewers

Validity ConclusionsValidity Conclusions

UselessUseless PerfectPerfect

0 1 2 3 4 5 6 7 8 9 10

X

X = JGIM

= Archives

ResultsResults

Where the results similar from study to Where the results similar from study to study?study?– Heterogeneity Heterogeneity

HeterogeneityHeterogeneity

Are differences between studies due to chance Are differences between studies due to chance alonealone

Check forest plot Check forest plot Often statistically checked using xOften statistically checked using x2 2 testtest

– not sensitivenot sensitive– p<0.1 implies heterogeneityp<0.1 implies heterogeneity

II2 2 test newer method for evaluatingtest newer method for evaluating Authors should provide reasons for Authors should provide reasons for

heterogeneity a prioriheterogeneity a priori

Favors Treatment Favors Control1

Favors Treatment Favors Control1

Forest PlotsForest Plots

Forest PlotsForest Plots

Forest PlotsForest Plots

Forest PlotsForest Plots

Forest PlotsForest Plots

Results (JGIM)Results (JGIM)

What is the magnitude of the treatment What is the magnitude of the treatment effect?effect?– Mortality Vit E v. placebo (secondary Mortality Vit E v. placebo (secondary

prevention)prevention)– Look at the top of Figure 2 pg 384Look at the top of Figure 2 pg 384

Results (JGIM)Results (JGIM)

Relative Risk for death with Vit E = 0.96Relative Risk for death with Vit E = 0.96

Results (JGIM)Results (JGIM)

How precise is the treatment effect?How precise is the treatment effect?»RR death = 0.96 RR death = 0.96 »95% confidence interval = 0.84-1.1095% confidence interval = 0.84-1.10

Applying ResultsApplying Results

Is our patient so different from those in the Is our patient so different from those in the study that its results can not apply?study that its results can not apply?– No reason to think that our patient would not have No reason to think that our patient would not have

similar resultssimilar results

Is the intervention feasible in our setting?Is the intervention feasible in our setting?– CertainlyCertainly

What are our patients values and What are our patients values and preferences?preferences?– Prevention of MI and DeathPrevention of MI and Death

Patient ConclusionPatient Conclusion

What would you do with this patient?What would you do with this patient? Well done meta-analysis of reasonable Well done meta-analysis of reasonable

quality individual trials quality individual trials No evidence of decrease in deathNo evidence of decrease in death Fairly tight confidence intervalsFairly tight confidence intervals I would not recommend vitamin E I would not recommend vitamin E

Objectives/ Wrap-upObjectives/ Wrap-up

Systematically analyze validity of a Systematically analyze validity of a meta-analysismeta-analysis

Discuss results of a meta-analysisDiscuss results of a meta-analysis Apply evidence from a meta-analysis to Apply evidence from a meta-analysis to

patient care decisionspatient care decisions

Egger M, Ebrahim S, Smith GD. Where now for meta-analysis? Int J Epidemiol. 2002;31(1):1-5.

Egger M, Ebrahim S, Smith GD. Where now for meta-analysis? Int J Epidemiol. 2002;31(1):1-5.