Date post: | 02-Oct-2014 |
Category: |
Documents |
Upload: | mphilramesh |
View: | 127 times |
Download: | 1 times |
, , ,
Publication Bias Funnel Plots Examples References
Publication Bias and Funnel Plots
Julia Braun
University of Zurich
Zurich, March 31, 2009
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Outline
1 Publication Bias
2 Funnel Plots
3 Examples
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Definition
Publication Bias (Dickersin, 1990):
Publication bias occurs when the publication of research resultsdepends on their nature and direction.
Other kinds of bias: language bias
availability bias
cost bias
familiarity bias
outcome bias
...
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Sources of publication bias
Explorative studies:
Experimental: submission of similar manuscripts with positiveand negative findings, comparison of recommendations topublish
Cohort: pursuit of all studies accepted by a local ethicscommittee
Main factors:
Failure to publish due to negative or null findings (mostly onthe side of the researchers, not editors/journals)
Selective reporting of outcomes (e.g. due to changes inresearch plan)
Selective citation of positive results
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Impact of publication bias
Comparison of the results that are reported and the best guess(informed by the results of publication bias analyses).
Three levels:
minimal: essentially similar estimates of the effect size
modest: effect size changes substantially, but key finding(e.g. treatment is or is not effective) remains the same
severe: basic conclusion of the analysis changes
=⇒ danger of false conclusions and patient harm
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Solutions?
Registration and consequent tracking of clinical trials⇒ Mandatory? One general registry?
Encouraging the publication of all studies,e.g. Journal of Negative Results in Biomedicine⇒ Acceptance? Practicability?
Required:
Every meta-analysis should include an analysis of publication bias,and this analysis should be reported as a standard part of theresults.
=⇒ Techniques for the assessment of publication bias
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Funnel Plot
−1.5 −1.0 −0.5 0.0 0.5 1.0 1.5
0.6
0.4
0.2
0.0
log OR
Sta
ndar
d er
ror
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
● ●●
●
●
●
●
●
●
●
●
●●
−1.0 −0.5 0.0 0.5 1.0
02
46
810
1214
log OR
Inve
rse
stan
dard
err
or
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●●●
●
●
●
●
●
●
●
●
● ●
Fundamental assumption: In the absence of bias, a plot fromstudies which estimate the same intervention effect will besymmetrical and bear some resemblance to a funnel(German translation: funnel = Kamin, Schacht, Trichter).
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Definition
Funnel Plot:
A funnel plot is a scatter plot of the treatment effects estimatedfrom individual studies against a measure of study size.
Properties:
The smaller the study size, the wider the spread of thetreatment effects and vice versa
If there is no bias: form of a funnel, symmetrical
If there is bias: asymmetrical
Explorative tool, no quantitative information on the amountor the source of the bias
No empirical investigations on funnel plots for continuousoutcomes so far
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Composition of a funnel plot
Decisions:
Which direction (horizontal vs. vertical, ascendingvs. descending order)?
Which measure for the treatment effect (log OR, log RR,RD)?
Which measure for the study size?
Inclusion of summary estimate?
Which confidence intervals (pointwise vs. lines around thesummary estimate)?
Mirror option?
Choice of these elements can have a substantial influence on thevisual impression!
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Measures for the treatment effect
Risk difference:
Easy interpretation
Possibly increased heterogeneitycan cause additional asymmetry
−0.4 −0.2 0.0 0.2 0.4
0.20
0.15
0.10
0.05
0.00
Risk difference
Sta
ndar
d er
ror
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
⇒ not recommended
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Measures for the treatment effect
Log risk ratio:
Easy interpretation
Possible heterogeneity if eventrate is high
Conclusions may depend onoutcome definition
−3 −2 −1 0 1 2 3
1.5
1.0
0.5
0.0
Log risk ratio
Sta
ndar
d er
ror
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
⇒ not recommended if event rate is high
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Measures for the treatment effect
Log odds ratio:
Interpretation more complicated
Shape of the plot not dependenton outcome definition
−3 −2 −1 0 1 2 3
1.5
1.0
0.5
0.0
Log odds ratio
Sta
ndar
d er
ror
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
⇒ recommended in most cases
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Measures for the treatment effect
−3 −2 −1 0 1 2 3
1.5
1.0
0.5
0.0
Log odds ratio
Sta
ndar
d er
ror
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
−3 −2 −1 0 1 2 3
1.5
1.0
0.5
0.0
Log risk ratio
Sta
ndar
d er
ror
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
−0.4 −0.2 0.0 0.2 0.4
0.20
0.15
0.10
0.05
0.00
Risk difference
Sta
ndar
d er
ror
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Measures for the study size
Study size/logarithmic study size:
−2 −1 0 1
050
0010
000
1500
0
Log odds ratio
Stu
dy s
ize
● ● ● ●
●●●
●● ● ●●
●● ● ●
●●
●●
●●
●●
●● ●● ● ●● ●●●
●
●
●
● ●● ● ●● ●● ● ●● ●●● ● ●●●
●
●●●
●
●
● ●●
●
●● ● ●●
−2 −1 0 13
45
67
89
10Log odds ratio
Log
stud
y si
ze
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
Expected shape of the plot in the case of no bias can not bederived⇒ not recommended!
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Measures for the study size
Standard error:
Characteristic funnel shape
Straight confidence lines
Emphasis on smaller studies(bias more likely)
Axis often inverted
−3 −2 −1 0 1 2 3
1.5
1.0
0.5
0.0
Log odds ratio
Sta
ndar
d er
ror
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
⇒ recommended in most cases
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Measures for the study size
Inverse standard error:
No funnel shape
Curved confidence lines
Emphasis on larger studies(small studies compressed at thebottom)
Probably useful for thecomparison of meta-analyses ofsmall and large studies
−3 −2 −1 0 1 2 3
05
1015
20
Log odds ratio
Inve
rse
of s
tand
ard
erro
r
●
●●
●
●
●●
●
●
●●●
●
●●
●
●●
●
●
●
●
●
●
●
● ●●
●
●
●●
●●
●
●
●
●
●
● ● ●
●
●●● ●●
●●●●
●
●
●
●
●
●●
●
●
●●●
●
●●
●
●
●
⇒ recommended for special tasks
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Measures for the study size
Variance:
No funnel shape
Curved confidence lines
Emphasis on smaller studies(bias more likely)
Axis often inverted
−3 −2 −1 0 1 2 3
2.5
2.0
1.5
1.0
0.5
0.0
Log odds ratio
Var
ianc
e
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
⇒ not recommended
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Measures for the study size
Inverse variance:
No funnel shape
Curved confidence lines
Emphasis on larger studies(small studies compressed at thebottom)
Probably useful for thecomparison of meta-analyses ofsmall and large studies
−3 −2 −1 0 1 2 3
010
020
030
040
0
Log odds ratio
Inve
rse
of v
aria
nce
● ● ● ●
●
●●●
●● ●●
●●● ●
●●●
●
●
●●
●●● ●● ● ●● ●●●
●
●
●
● ●● ● ●● ●● ● ●● ●●● ● ●●●●
●●●
●
●
● ●●
●
●●● ●●
⇒ sometimes useful, but inverse standard error should be preferred
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Measures for the study size
−3 −2 −1 0 1 2 3
1.5
1.0
0.5
0.0
Log odds ratio
Sta
ndar
d er
ror
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
−3 −2 −1 0 1 2 3
2.5
2.0
1.5
1.0
0.5
0.0
Log odds ratioV
aria
nce
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
−2 −1 0 1
050
0010
000
1500
0
Log odds ratio
Stu
dy s
ize
● ● ● ●●
●●●● ● ●● ●●● ●
●●●●
●●●●●● ●● ● ●● ●●●
●
●
●
● ●● ● ●● ●● ● ●● ●●● ● ●●●
●●●●
●
●
● ●●
●
●● ● ●●
−3 −2 −1 0 1 2 3
05
1015
20
Log odds ratio
Inve
rse
of s
tand
ard
erro
r
●● ●
●
●
●●●
●
●●●
●
●●
●
●●
●
●
●
●
●
●●
● ●●
●
●
●●●●
●
●
●
●
●
● ● ●●
●● ● ●●●●● ● ●●
●
●
●
●●
●
●
● ●●
●
●●
●●
●
−3 −2 −1 0 1 2 3
010
020
030
040
0
Log odds ratio
Inve
rse
of v
aria
nce
● ● ● ●
●
●●●●● ●●
●●● ●
●●●
●
●
●●
●●● ●● ● ●● ●●●
●
●
●
● ●● ● ●● ●● ● ●●●●● ● ●●●●
●●●
●
●
● ●●
●
●● ●●●
−2 −1 0 13
45
67
89
10
Log odds ratio
Log
stud
y si
ze●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Confidence intervals
Alternative to confidence lines: Pointwise confidence intervals
●
●
●
●
●
●
●
● ●
●
−2.0 −1.5 −1.0 −0.5
01
23
4
Log odds ratio
Inve
rse
stan
dard
err
or
●
●
●
●
●
●
●
● ●
●
−3 −2 −1 0
01
23
4
Log odds ratio
Inve
rse
stan
dard
err
or
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Mirror option
●
●
●
●
●
●
●
● ●
●
−4 −3 −2 −1 0
01
23
4
Log odds ratio
Inve
rse
stan
dard
err
or
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
−4 −3 −2 −1 0 1 2
01
23
4
Log odds ratio
Inve
rse
stan
dard
err
or
●
●
●
●
●
●
●
●●
●
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Use of funnel plots
Problems:
No clear recommendations for non-binary outcomes
Primary visual tool, no quantification of bias
Asymmetry not necessarily due to bias (e.g. heterogeneity inthe underlying effects can also lead to asymmetry)
No information on the type of bias
Interpretation difficult if only few studies in meta-analysis
Funnel plots and tools for meta-analysis in R:packages meta and rmeta
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Example 1: Environmental tobacco smoke
Meta-analysis by Hackshaw et al. (1997):
Effect of environmental tobacco smoke on risk of lung cancer
Comparison of spouses of smokers and non-smokers
37 studies
Controversy: Affected by publication bias?
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Funnel plot of Hackshaw et al. (1997)
−1.0 −0.5 0.0 0.5 1.0 1.5
0.6
0.4
0.2
0.0
Log odds ratio
Sta
ndar
d er
ror
●●
●●●
●●● ●●●
●
●●●
●●●● ●● ●
●
●●
●●●
●●●
●
●●
●
●
●
Smaller studies tend to show greater adverse effects⇒ asymmetry
Not clear if real bias or chance
Visual inspection not enough
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Example 2: Antibacterial catheter coating
Meta-analysis by Veenstra et al. (1998):
Effectiveness of a special coating on venuous catheters
Two endpoints: Bacterial colonisation of the catheter andbloodstream infection
16 controlled trials
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Funnel plot of Veenstra et al. (1998)
Bacterial colonisation:
−4 −3 −2 −1 0 1 2
1.5
1.0
0.5
0.0
Log odds ratio
Sta
ndar
d er
ror
●
●
●
●
●
●
●
●
●
●
● ●
●
⇒ hints for publication bias
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Funnel plot of Veenstra et al. (1998)
Bloodstream infection:
−3 −2 −1 0 1 2
1.5
1.0
0.5
0.0
Log odds ratio
Sta
ndar
d er
ror
●
●
● ●
●
●
●
●
●
●
●
●
⇒ hints for publication bias
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
Summary
Visual tool for investigation of (publication) bias inmeta-analysis
Scatter plots of treatment effect versus study size
For binary outcomes: odds ratio and standard errors bestchoice
In the case of bias: asymmetry
Other causes of asymmetry possible
Interpretation difficult if number of studies small
Julia Braun University of Zurich
Publication Bias and Funnel Plots
, , ,
Publication Bias Funnel Plots Examples References
References
Dickersin, K. (1990). The existence of publication bias and risk factors for itsoccurrence, Journal of the American Medical Association263(10): 1385–1395.
Hackshaw, A., Law, M. and Wald, N. (1997). The accumulated evidence onlung cancer and environmental tobacco smoke, British Medical Journal315: 980–988.
Rothstein, H., Sutton, A. and Borenstein, M. (eds) (2005). Publication Bias inMeta-Analysis, Wiley.
Veenstra, D., Saint, S., Saha, S., Lumley, T. and Sullivan, S. (1998). Efficacyof antiseptic impregnated central venous catheters in preventing nosocomialinfections: a meta analysis, Journal of the American Medical Association281: 261–267.
Julia Braun University of Zurich
Publication Bias and Funnel Plots