Date post: | 04-Jan-2016 |
Category: |
Documents |
Upload: | conan-ashley |
View: | 86 times |
Download: | 2 times |
Systematic Reviews and Meta-Analysis
Methodologies for a new era summer school
School of Applied Social Studies, University College Cork
20 June 2011
Dr Paul Montgomery
Aims
1) Discuss the advantages and main features of systematic reviews
2) Introduce basic principles of meta-analysis
Course feedback
The Problem
Millions of articles published in thousands of journals each year
Practitioners and researchers are busy
Subjective summaries may misrepresent research
Reviews
Systematic Reviews Aim to answer specific questions, reduce
uncertainty, identify outstanding questions
Common methods include narrative synthesis, meta-analysis (meta-regression)
Traditional ‘journalistic’ reviews Aim to persuade, draw attention to a
topic, synthesise information, etc. Narrative synthesis most common
Systematic Review:
“the application of scientific strategies that limit bias to the systematic assembly, critical appraisal, and synthesis of all relevant studies on a specific topic."
Cook DJ, Sakett DL, Spitzer WO. Methodological guidelines for systematic reviews of randomized contro trials in health care from the Potsdam Consultation
on Meta-Analysis. J. Clin. Epidemiol. 1995;48:167-71
Systematic Reviews
Clear Question Define the population, problem,
intervention, alternative interventions, and outcomes
Replicable Method Search strategy Inclusion criteria Analytical strategy
Transparent Process
Advantages
Explicit methods limit bias in identifying and rejecting studies
Information can be understood quicklyReduced delay between discoveries
and implementation Results can be formally comparedHeterogeneity can be identified and
new hypotheses generatedQuantitative reviews increase precision
Producers
CochraneCampbellEPPIDARENICE Interested practitioners/ academics
Cochrane Review ProcessRegister titles and check for overlapProtocols developed and peer reviewedSearches performed widely on all main
databases, grey literature searches, personal contacts
Abstracts reviewed by two authorsData collected and trial quality assessedData synthesis and analysisWrite-up Reviewed by Cochrane/ Campbell editors,
then peer reviewed
Systematic reviews
Key components:1. Ask a good question2. Identify studies3. Extract data4. Synthesise data5. Interpreting the results
Who Should Review?
“Experts, who have been steeped in a subject for years and know what the answer ‘ought’ to be, are less able to produce an objective review of the literature in their subject than non-experts. This would be of little consequence if experts' opinions could be relied on to be congruent with the results of independent systematic reviews, but they cannot.”
(Trisha Greenhalgh)
PICO Mad-libs
ForP ____________ doesI ____________ compared toC ____________ improve/reduceO ____________ ?
Highly Sensitive Search
Electronic SearchesDatabases/IndexesAdditional Electronic Searches
Hand SearchesPersonal Contacts
Electronic databases:
BA
Medline
PsycInfo
Medline covers 23% of the core 505 ‘psychiatric’
journals, plus most of the major biomedical
journals. Biological Abstracts covers
48%, plus lots of life sciences stuff.
Embase covers 67% plus lots of
European journals that Medline
misses.
Embase
PsycInfo covers 73% and has a
psychological focus
Searching PsycLit and Embase will cover 92%
of the core 505 ‘psychiatric’ journals.
Electronic Searches
Sensitivity vs. SpecificityEven if 2 terms and 3 databases return almost all literature on a subject, the goal of a systematic review is to find everything.
Electronic Searches
Specific AuthorsReverse CitationAgencies / Non-ProfitsFunding BodiesAcademic Groups / Research CentersGoogle
Additional Searches
Previous ReviewsBibliographies of Related ArticlesHand Search Journals (that aren’t
indexed)Conference Reports
(many are electronically published)
Personal Communication
Call or Email AuthorsAttend ConferencesWrite to:
Agencies / Non-ProfitsProviders / Manufacturers /
DistributorsFunding BodiesAcademic Groups / Research Centers
Questions to Ask
Which programs will be studied?Compared to what?What study designs are acceptable?What must a study measure?How must it be measured?Must researchers be blind at
allocation, during the trial, etc?How will dropouts be handled?What about missing data?
Inclusion and Exclusion
Types of studiesTypes of studies Types of participantsTypes of participants Types of comparisonsTypes of comparisons
Specify:Specify:Types of outcomesTypes of outcomes
Multiplicity (time, comparisons, measures, Multiplicity (time, comparisons, measures, statistics)statistics)
Transparency
Be clear about all definitions, searches, inclusion and exclusion criteria, etc.
Report ongoing trialsList excluded studies, particularly if:
The trials contain valuable information Exclusion was a close call You discovered something about a trial
Evaluating a Review
Even if a review is Even if a review is ‘systematic’ it may ‘systematic’ it may
not be well-not be well-conducted. How do conducted. How do
we tell the we tell the difference?difference?
Validity
1. Did the review address a clearly focussed question?
2. Were the right sort of studies selected?
3. Was the search strategy explicit and comprehensive?
4. Did the reviewers assess the quality of the identified studies?
Importance:
1. Were the results similar from study to study?
2. What is the overall result of the review?
3. How precise are the results?
Potential Sources of Bias
Describe aspects of study design that might have influenced the magnitude or direction of results
Use of rating scales with fixed cut-offs potentially misleading
Consider external validity
Juni P, Witschi A, Bloch R, Egger M. The hazards of
scoring the quality of clinical trials for meta-
analysis. JAMA 1999; 282: 1054-1060
Tower of Babel
Studies that find a treatment effect are more likely to be published in English-language journals.
Opposing studies may be published in non-English-language journals.
Gregoire G, Derderan F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel Bias? J.Clin.Epidemiol.
1995;48:159-163
Publication Bias
“the tendency of investigators, reviewers and editors to differentially submit or accept manuscripts for publication on the direction or strength of the study findings.”
Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should unpublished data be included in meta-analyses? Current convictions and controversies.
JAMA 1993; 269: 2749-2753
Unpublished data
ControversialUnpublished data may not be a full or
representative sample (Cook 1993)Publication is no guarantee of scientific quality
(Oxman 1991)Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should
unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269: 2749-2753
Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA et al. Agreement among reviewers of review articles. J.Clin.Epidemiol. 1991;44:91-98.
Meta-analyses:
“A systematic review that employs statistical methods to combine and summarise the results of several studies.”
Cook DJ, Sakett DL, Spitzer WO. Methodological guidelines for systematic reviews of randomized contro trials in health care from the Potsdam Consultation
on Meta-Analysis. J. Clin. Epidemiol. 1995;48:167-71
Summarising trials
Reviews
Systematic reviews
Meta-analyses
Meta-analyses
Mathematically combine the results of different studies
For dichotomous or continuous outcomes
From analytical (treatment) or observational (aetiology, diagnosis, prognosis) studies
‘Weighted’ by study size (usually 1/se2) and/or quality
Benefits of meta-analysis:
1. To increase statistical power for primary end points and for subgroups.
2. To improve estimates of effect size.3. To resolve uncertainty when reports disagree4. To answer questions not posed at the start of
individual trials.
Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of randomized controlled trials. N.Engl.J.Med. 1987;316:450-455
Outcome Measures
Continuous / Dichotomous (/ Ordinal)
Objective / Subjective
Meta-analysis
Some outcomes are measured on scales – e.g. depression or continuously e.g. sleep minutes
Continuous outcomes can be calculated using the scale on which they were measured (WMD)
If changes in depression are measured on different scales it is still possible to combined them but on a standardised scale
Meta-analysis
Alternatively we might be interested in binary data - two mutually exclusive states
Dead/alive; hospitalised/not hospitalisedThese data will be measured in a
different way to continuous (scale) dataReported as ‘event rates’
Meta-analysis
Central Tendency: Mean (Cohen’s d, Hedges’s g) Odds Ratio / Relative Risk / Rate Ratio
Variance (Confidence Interval)Clinical Significance (NNT/NNH)Heterogeneity (I2, Q, Chi2)
Dichotomous Outcomes
Odds are calculated by dividing the number of events by non-events (ie clients experiencing the event divided by clients not experiencing an event)
Risk/Rate is more widely reported in reviews as it tends to be easier to communicate
Weighting
Some studies contribute more weight to the ‘average’ result than do others
The more precise the effect estimate, the more weight is given
Wide variation is sometimes associated with small studies
Weighting
Clinical trials are rarely conducted according to identical protocols
Severity of the problem, intensity of the intervention, duration, setting of trial, age may account for differences in response
Apples and oranges?Sources of Heterogeneity:
Study participants Comparisons Intervention design Delivery Duration of follow-up Outcome measures Methods
Heterogeneity
Estimates from individual trials vary more than can be explained by the play of chance alone
N.B. Meta-analysis should NOT overlook important material differences in subgroup response
Heterogeneity – approaches
Qualitative v. quantitative Qualitative – reconsider pooling Does it makes sense to average
effects from the studies? Fixed v. random effects
Subgroup Analysis
If together there is excessive variation, when analysed separately there is a uniform response to treatment in each subgroup
Hypothesis generating
Sensitivity analysis:
Sensitivity analyses investigate how the conclusions of a review change when one or more of the decisions or assumptions are altered.
Testing for heterogeneity
Look at plots of resultsFormal tests of homogeneity
I2
Q Chi2
Assess qualitative differences in study design or implementation
.
Weeks Study name Comparison Outcome Statistics for each study Sample size Hedges's g and 95% CI
Hedges's Lower Upper Relative g limit limit Media Comp weight
4 Hassan 1992 Wait List Combined 4.30 2.55 6.05 10 8 0.68
5 Bickel 2007 Wait List Combined 0.81 -0.33 1.96 8 5 1.40
4 Milne 1998 Wait List (TaU) Combined -0.33 -1.35 0.70 7 6 1.67
8 Rosen 1976 Wait List Combined -0.73 -1.72 0.27 16 6 1.74
1 Klein 2001 Wait List Combined 0.66 -0.18 1.51 10 12 2.20
8 Abramowitz 2009 Wait List Combined 0.39 -0.45 1.22 11 10 2.23
8 Lidren 1994 Wait List Combined 0.91 0.09 1.74 12 12 2.29
8 Richards 2006 Wait List Combined 0.63 -0.14 1.41 23 9 2.46
12 Fletcher 2005 Wait List HADS - Anxiety 0.10 -0.65 0.86 11 15 2.56
1 Heading 2001 Wait List Combined 0.17 -0.58 0.92 13 13 2.57
13 Kiely 2002 Wait List (TaU) Combined 0.85 0.12 1.58 16 14 2.66
8 Lewis 1978 Monitoring Combined 0.58 -0.13 1.29 38 10 2.77
4 Jones 2002 Wait List (TaU) Combined 0.58 -0.06 1.21 19 20 3.15
8 Grime 2004 Wait List HADS - Anxiety 0.41 -0.22 1.04 16 23 3.16
10 Carlbring 2001 Wait List Combined 0.81 0.18 1.44 21 20 3.18
8 Sorby 1991 No Int (Plus TaU) Combined 0.62 -0.00 1.24 25 17 3.22
6 Smith 1997 Attention Combined 0.33 -0.29 0.94 30 15 3.25
10 Titov 2009 Wait List Combined 0.98 0.37 1.59 24 21 3.27
11 Arpin-Cribbie 2007 No Int Combined 0.81 0.24 1.38 29 22 3.50
10 Berger 2009 Wait List Combined 0.75 0.18 1.31 31 21 3.55
10 Carlbring 2006 Wait List Combined 1.19 0.64 1.74 30 30 3.63
9 Carlbring 2007 Wait List Combined 1.01 0.47 1.55 29 29 3.70
14 Hazen 1996 Wait List Combined 0.43 -0.10 0.97 27 27 3.75
13 Zetterqvist 2003 Wait List Combined 0.39 -0.07 0.84 37 45 4.28
12 Van Boeijen 2005 Treatment as Usual Combined -0.13 -0.59 0.32 53 28 4.29
10 Titov 2008b Wait List Combined 0.79 0.34 1.24 41 40 4.32
10 Titov 2008c Wait List Combined 0.45 0.02 0.87 61 34 4.54
10 Titov 2008a Wait List Combined 0.83 0.42 1.23 50 49 4.64
13 Mead 2005 Wait List HADS 0.18 -0.20 0.56 50 53 4.83
12 Rapee 2007 Wait List Combined 0.38 0.00 0.76 56 52 4.87
9 Proudfoot 2004 Treatment as Usual BAI 0.38 0.10 0.66 99 98 5.65
0.55 0.40 0.70 903 764
-2.00 -1.00 0.00 1.00 2.00
Favours No-Treatment Favours Self-Help
Anxiety (self-rated) at Post-Treatment compared to No-TreatmentAnxiety (Self-Rated Symptoms) at Post-Treatment
Group byTime
Study name Time (m) Risk ratio and 95% CI Statistics for each study
Relative Risk Lower Upper weight ratio limit limit
12-23 Dalby (2000) 14 1.20 0.35 0.01 8.41 Institutionalisation
12-23 Hogan (2001) 12 2.14 2.05 0.19 22.17 Institutionalisation
12-23 Newbury (2001) 12 3.21 0.94 0.13 6.55 Institutionalisation
12-23 Hall (1992) 12 7.41 0.38 0.11 1.37 Institutionalisation
12-23 Hebert (2001) 12 8.06 1.04 0.30 3.53 Institutionalisation
12-23 Kono (2004) 18 10.83 0.64 0.22 1.83 Institutionalisation
12-23 Yamada (2003) 18 17.26 1.30 0.56 3.01 Institutionalisation
12-23 Bernabei (1998) 12 21.52 0.67 0.32 1.43 Institutionalisation
12-23 Gill (2002) 12 28.37 0.72 0.38 1.39 Institutionalisation
12-23 0.78 0.55 1.10
24-35 Hall (1992) 24 42.30 0.16 0.04 0.67 Institutionalisation
24-35 Sorenson (1988) 30 57.70 1.02 0.81 1.28 Institutionalisation
24-35 0.46 0.08 2.80
36+ Hall (1992) 36 9.53 0.16 0.04 0.67 Institutionalisation
36+ Van Rossum (1993) 36 12.00 1.38 0.44 4.30 Institutionalisation
36+ Pathy (1992a) 36 12.86 1.29 0.46 3.66 Institutionalisation
36+ Byles (2004) 36 14.97 2.84 1.25 6.43 Institutionalisation
36+ Stuck (1995) 36 15.52 0.42 0.20 0.90 Institutionalisation
36+ Pathy (1992b) 36 16.48 0.56 0.29 1.09 Institutionalisation
36+ Stuck (2000) 36 18.64 1.51 0.99 2.30 Institutionalisation
36+ 0.90 0.49 1.67
0.1 0.2 0.5 1 2 5 10
Favours Home Visits Favours Controls
Institutionalisation (RR<1 favours home visits)
-2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0
0.0
0.5
1.0
1.5
2.0
Sta
nd
ard
Err
or
Log risk ratio
Funnel Plot of Standard Error by Log risk ratioMortality: Trim and Fill (missing studies shown, 1 trimmed)
-4 -3 -2 -1 0 1 2 3 4
0.0
0.2
0.4
0.6
0.8
1.0
Stan
dard
Err
or
Hedges's g
Funnel Plot of Standard Error by Hedges's gAnxiety (Self-Rated Symptoms) at Post-Treatment
Regression of Ave Age on Log risk ratio
Ave Age
Lo
g r
isk
rat
io
67.16 69.01 70.86 72.70 74.55 76.40 78.25 80.10 81.94 83.79 85.64
2.00
1.60
1.20
0.80
0.40
0.00
-0.40
-0.80
-1.20
-1.60
-2.00
Mortality by age: Meta-regression
Regression of Number of Contacts on Hedges's g
Number of Contacts
He
dg
es
's g
-1.35 1.47 4.29 7.11 9.93 12.75 15.57 18.39 21.21 24.03 26.85
2.00
1.72
1.44
1.16
0.88
0.60
0.32
0.04
-0.24
-0.52
-0.80
Anxiety (Self-Rated Symptoms) at Post-Treatment: Number of contacts with researchers and clinicians
Point Estimate SE P Q df P
Slope 0.03 <0.01 <0.001 Model 12.93 1 <0.001
Intercept 0.27 0.08 <0.01 Residual 30.34 27 0.30Total 43.27 28 0.03
Limitations
Junk-In, Junk-OutThe results of large trials
sometimes differChance Events: Aggregation and
Disaggregation
Conclusion
Systematic reviews seek to reduce bias and improve the reliability and accuracy of the conclusions.
Meta-analysis is a powerful research tool, but it should be conducted only in the context of a systematic review, and it has important limitations.