Forum: Meta-analyses
Meta-Analyses: Considerations for Probiotics & Prebiotics Studies
Daniel J. Tancredi, PhD Associate Professor in Residence
Department of Pediatrics UC Davis School of Medicine
Forum: Meta-analyses
Objectives
Provide an overview of systematic reviews & meta-analyses
Discuss with you key considerations & controversies for sponsors of probiotic/prebiotic studies
Forum: Meta-analyses
Key sources of material for talk
Julie Glanville, Sarah King, Francisco Guarner, Colin Hill, Mary Ellen Sanders. Nutrition Journal (2015) 14:16
Forum: Meta-analyses
Key sources of material for talk
Systematic Reviews in Health Care: Meta-Analysis in Context, 2nd Edition
Matthias Egger (Editor), George Davey-Smith (Editor), Douglas Altman (Editor)
March 2001, BMJ Books
Forum: Meta-analyses
Definitions:
“Systematic Review: A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review.
Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies.”
Source: http://community.cochrane.org/glossary/
Forum: Meta-analyses
Systematic Reviews: Scientific Virtues
Protocol-driven
Transparent (understandable)
Reproducible
Objective
Rigorous (aims to reduce biases and imprecision)
Forum: Meta-analyses
Narrative vs. Systematic Reviews Feature Narrative Systematic
Question Often broad Focused
Sources & search
Not usually specified, potentially
biased
Comprehensive sources, explicit search strategy
Selection ” Criterion-based,
uniformly applied
Appraisal Variable Rigorous
Synthesis Often qualitative Quantitative
Inferences Sometimes
evidence-based Usually evidence-
based
Source: Cook DJ et al (1998) “Synthesis of best evidence for clinical
decisions” Ch. 1 of “Systematic Reviews: Synthesis of Best Evidence
for Health Care Decisions” (ed. Mulrow C & Cook D). ACP, Philadelphia.
Forum: Meta-analyses
When to do a systematic review
Beginning and end of every RCT!? – Clarke et al. Reports of clinical trials should begin
and end with up-to-date systematic reviews of other relevant evidence: a status report. J R Soc Med. 2007 Apr; 100(4): 187–190.
CONSORT 2010 recommends (does not require) post-RCT SR:
– as an internal validity check (item 20)
– to situate results in context (item 22) • CONSORT 2010 Explanation and Elaboration:
updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869
Meta-analysis of RCTs of probiotics for management of infant colic (crying time at 21 days post-intervention)
Valerie Sung et al. BMJ 2014;348:bmj.g2107 ©2014 by British Medical Journal Publishing Group
Previous
Example: Before/After Meta-analysis in RCT
Updated
Forum: Meta-analyses
When to do a systematic review (cont.)
According to EFSA “food/feed safety” guidance, a systematic review
may be useful when
– research question could be answered by an envisionable primary research study design
and practical and worthwhile when – SR on available data could improve precision of
estimated values for parameters of interest
Source: Application of SR methodology to food and feed safety
assessments…EFSA Guidance for those carrying out systematic reviews.
EFSA Journal 2010; 8(6):1637
Forum: Meta-analyses
…also “from” EFSA Q. What are the requirements for meaningful SR results for health benefits of foods or constituents?
A. SR’s (with or without meta-analyses) can be useful if:
1. They ask a clear and specific question
2. They are carried out rigorously – so as to minimise bias and random error
3. Reported well enough to allow assessment of the level of bias in the underlying evidence & in the review process
--Lee Hooper, U. East Anglia In presentation (http://www.efsa.europa.eu/en/events/documents/131120-p07.pdf ) at 2013 EFSA “Technical meeting on the reporting of human studies submitted for the scientific substantiation of health claims” http://www.efsa.europa.eu/en/supporting/doc/569e.pdf)
Forum: Meta-analyses
Don’t forget: RCTs are still primary
• Sponsors must strive for high quality RCTs
• Rationale: The assessment of individual RCTs is the key stage in weighing the “totality of evidence” in favor of a claim.
Sources: Appendix H (Study Quality) of EFSA's "Opinion of the Panel on dietetic products, nutrition and allergies (NDA) on a request from the Commission related to scientific and technical guidance for the preparation and presentation of the application for authorisation of a health claim" and
Table 13a of “Health Canada‘s Guidance Document for Preparing a Submission for Food Health Claims (March, 2009)”
Probiotics for the prevention of Clostridium difficile‐associated diarrhea in adults and children
Cochrane Database of Systematic Reviews 31 MAY 2013 DOI: 10.1002/14651858.CD006095.pub3 http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD006095.pub3/full#CD006095-fig-0003
Another Example
Forum: Meta-analyses
How to do a SR: Seven Steps
1. Frame question (& preparing protocol)
2. Systematically locate relevant studies
3. Study selection
4. Data extraction (or assembly for IPDMA)
5. Assess quality of studies
6. Synthesize evidence (via meta-analyses)
7. Interpret findings Source: Glanville J, King S, Guarner F, Hill C, Sanders ME. A review of the systematic review process and its applicability for evaluating evidence for health claims on probiotic foods in the EU. Nutrition Journal (2015) 14:16
Forum: Meta-analyses
Register and publish protocol early!
Pre-specify research question, eligibility criteria & methods to be used for all steps in protocol
Register SR at PROSPERO http://www.crd.york.ac.uk/PROSPERO/
Publish protocol
Forum: Meta-analyses
Frame question
Explicit specification of PICOS
– Population, including settings/locations
– Intervention(s), including vehicles/matrices
– Comparison
– Outcome(s), including timing & how measured
– Study types (designs & methodological quality)
Forum: Meta-analyses
Systematically locate relevant studies
Comprehensive (published & unpublished)
Explicit description of
– Sources (trial registries, MEDLINE, EMBASE, Cochrane Controlled Trials Register)
– Queries (search terms & logic operators)
– Methods for tracing citation linkages
– [handsearching key journals & personal communication with experts]
Forum: Meta-analyses
Study Selection
2+ observers check eligibility
Develop strategy to resolve disagreements
2-stages:
– Titles & Abstracts
– Full study
Keep track of inclusion/exclusion decisions for each stage (for PRISMA Flow Chart)
From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(6): e1000097. doi:10.1371/journal.pmed1000097
Forum: Meta-analyses
Data Extraction/Assembly
Develop, pilot test & refine a standard data extraction form
2+ observers
[Replicate published results]
Frank D’Amico, PhD Professor of Mathematics Duquesne University
Forum: Meta-analyses
Assess Quality of Studies
GRADE approach (quality is degree of confidence that effect size estimate is close to true parameter)
For RCTs, assess
random sequence generation/concealment,
Blinding,
Incomplete outcome data,
Selective reporting and other biases
[validation of intervention/comparator composition]
See http://www.gradeworkinggroup.org/index.htm for guidelines from the Grading of Recommendations Assessment, Development and Evaluation Working Group Also see: http://handbook.cochrane.org/chapter_8/table_8_5_a_the_cochrane_collaborations_tool_for_assessing.htm
Probiotics for the prevention of Clostridium difficile‐associated diarrhea in adults and children
Cochrane Database of Systematic Reviews 31 MAY 2013 DOI: 10.1002/14651858.CD006095.pub3 http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD006095.pub3/full#CD006095-fig-0002
Forum: Meta-analyses
Synthesize Evidence
Tabulate results (study-specific)
Examine forest plots
Assess heterogeneity
– Statistical distinction: sample v. population
Consider MA of all trials or subgroups
Perform sensitivity analysis
Examine funnel plots (publication bias)
Make available list of excluded studies
Sackett DL, Glasziou P, Chalmers I. Meta-analysis may reduce imprecision, but
it can’t reduce bias. Unpublished commentary commissioned by the New
England Journal of Medicine, 1997. (see SR in Health Care, p. xiv)
Forum: Meta-analyses
Funnel Plots Example
http://handbook.cochrane.org/chapter_10/figure_10
_4_a_hypothetical_funnel_plots.htm
Symmetrical plot in
the absence of
reporting bias
Asymmetrical plot
in the presence of
reporting bias
Forum: Meta-analyses
Meta-Analysis Meta-analysis may reduce imprecision, but it
can’t reduce biases
Heuristically, a pooled effect size is estimated as a weighted averages of sample effect sizes, resulting in a more precise estimate (with a smaller uncertainty interval)
Assess explainable/unexplainable heterogeneity in effect sizes, including subgroups
Sensitivity analyses to assess robustness
EFSA requires rationale & justification for MA Sackett DL, Glasziou P, Chalmers I. Meta-analysis may reduce imprecision, but
it can’t reduce bias. Unpublished commentary commissioned by the New
England Journal of Medicine, 1997. (see SR in Health Care, p. xiv)
Figure Example from Jack Cuzick’s. ‘Forest plots and the interpretation of subgroups.’ The Lancet 2005 365,Issue 9467
Copyright © 2005 Elsevier Ltd
Meta-analysis of RCTs of probiotics for management of infant colic
Outcome is crying time (min/day) at 21 days post-intervention)
Valerie Sung et al. BMJ 2014;348:bmj.g2107 ©2014 by British Medical Journal Publishing Group
Boxplots of Infant Crying Time in 3 RCTs for Managing Colic By Study Arm and Day of Follow-up
Placebo
L. reuteri DSM 17938
Forum: Meta-analyses
Present & Interpret Findings
Follow PRISMA statement (www.prisma-statement.org) for reporting
(Indeed, follow PRISMA after conceiving SR, to make sure you’re accounting for everything you’ll need!)
Explain departures from PRISMA (per EFSA)
Forum: Meta-analyses
Heterogeneity of causal effects…
Forum: Meta-analyses
Source: J R Soc Med. 2009 May 1; 102(5): 186–194
Copyright © 2009, The Royal Society of Medicine
Three classes of evidence for causal claims in Howick et al’s “The evolution of evidence hierarchies: what can Bradford Hill’s ‘guidelines for causation’ contribute?”
Forum: Meta-analyses
RCTs can provide direct (probabilistic) evidence of causality
Randomization could yield sufficient evidence to outweigh plausible sources of confounding, even without knowing mechanism.
Intervention Black box Outcome
Control Black box Outcome
Forum: Meta-analyses
Howick et al’s Two Levels of Mechanistic Evidence
1. Internal substudies of causal links in Black box
2. Purported Mechanism of Action demonstrated in external studies
Source: J R Soc Med. 2009 May 1; 102(5): 186
– 194
© 2009, The Royal Society of Medicine
Challenge to Combinability: Black Boxes Depend on Study PICOS
Intervention Black
box(A) Outcome
Control Black
box(A) Outcome
Study A
Intervention Black
box(B) Outcome
Control Black
box(B) Outcome
Study B
The blank “wheel” of the pragmatic–explanatory continuum indicator summary (PRECIS) tool. “E” represents the
“explanatory” end of the pragmatic–explanatory continuum.
Kevin E. Thorpe et al. CMAJ 2009;180:E47-E57
©2009 by Canadian Medical Association
Figure A: PRECIS summary of four unrelated RCTs, for illustration
Kevin E. Thorpe et al. CMAJ
2009;180:E47-E57
©2009 by Canadian Medical Association
NB: Potential
use for modified
PRECIS tools
for SR has
begun (see
PMID:
21474282).
Possible distribution of mechanisms among probiotics
Hill, C. et al. (2014) The International Scientific Association for Probiotics and Prebiotics consensus
statement on the scope and appropriate use of the term probiotic
Nat. Rev. Gastroenterol. Hepatol. doi:10.1038/nrgastro.2014.66
Forum: Meta-analyses
Some rationales for combining strains
Rationale: Common structural or secreted product
Example: All L. bulgaricus & S. thermophilus
Rationale: Common MoA known to be n.&s. for effect (e.g., production of a specific bacteriocin or range of bacteriocins known to be active against a specific pathogen, or induction of immune mechanisms needed for the effect)
Example: L. salivarius strains A & B
Other rationales: common identity, taxonomy, physiological effect (in humans)
Glanville et al. Nutrition Journal (2015) 14:16
Forum: Meta-analyses
Remember
High quality RCTs are key
Well-done SR’s can provide
– more precise estimates of causal effects (narrower uncertainty intervals) and
– assess (some) heterogeneity of treatment effects
Pooling different studies requires scientific basis for the “common black box” assumption