Post on 15-Jan-2015
description
transcript
Tools for Assessing Risk of Bias
Doug AltmanCentre for Statistics in Medicine
University of Oxford
Acknowledgement
Cochrane Collaboration Bias Methods Group
Systematic review
A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria to answer a specific research question
It uses explicit, systematic methods that are selected with a view to minimising bias, thus providing reliable findings from which conclusions can be drawn and decisions made
3
Key characteristics of a systematic review
A clearly stated set of objectives with an explicit, reproducible methodology
A systematic search that attempts to identify all studies that would meet the eligibility criteria
An assessment of the validity of the findings of the included studies, such as through the assessment of risk of bias
Systematic presentation and synthesis of the characteristics and findings of the included studies
[Liberati et al, BMJ 2007]4
Taking account of study quality
Methodological quality of the primary studies may affect their results
The influence of study quality should routinely be examined– How?
Quality is difficult to define– design, conduct and analysis of a trial? – clinical relevance?– quality of reporting?– all of these?
Focus on Randomised Controlled Trials (RCTs) 5
Quality scales and checklists
Many exist (>40)– vary hugely in complexity and scope
Many combine trial quality with reporting quality and other aspects (e.g. sample size)
Very popular: Jadad et al 1996– Simple score from 0 to 5– Serious flaws: combines methodology and reporting
Comparison of high vs low quality trials depends – Results depend strongly on which scale is used
[Jüni et al, JAMA 1999]
6
The Cochrane Collaboration aims to help people make well-informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions (and diagnostic test accuracy)
7
Cochrane library
8
Rethinking the approach to assessing study quality
2005: Cochrane Collaboration initiative
Rationale: quality scores are not helpful
Focus should be on aspects with potential for bias, such as – concealment of treatment allocation– blinding of outcome assessment– handling of patient attrition
9
Cochrane Risk of Bias (RoB) tool
The Risk of Bias (RoB) tool was developed over several years as a way to address concerns about the existing method of assessing study “quality” in Cochrane reviews
Based on empirical evidence where possible
It is intended to lead to uniform approach across reviews from all Cochrane review groups
10
Developing a Risk of bias tool
In May 2005, 16 statisticians, epidemiologists, and review authors attended a three day meeting to develop the new tool
Identified items that were truly potential biases rather than sources of heterogeneity or imprecision
Seven areas were discussed leading to a proposed set of criteria for assessing protection from bias – as adequate, inadequate, or unclear
11
The Risk of Bias Tool
12
http://www.cochrane-handbook.org/
13
BMJ 2011;343:d5928
Risk of biasGeneral principle of the RoB tool Avoidance of bias
Assessment of risk of specific types of bias
Selection bias
Performance bias
Detection bias
Attrition bias
Risk of bias summary14
15
Participants
Treatment A Comparator B
OutcomeAssessment
OutcomeAssessment
Detection bias
Selection bias
Attrition bias
Bias
Performance bias
16
Population
Treatment A Comparator B
OutcomeAssessment
OutcomeAssessment
Detection bias
Selection bias
Attrition bias
Bias
Performance bias
Randomisation
Blinding of participants and
personnel
Blinding of outcome assessors
Domains of the Cochrane “Risk of Bias” tool
Review authors’ judgement: Was the allocation sequence adequately
generated? Was allocation adequately concealed? Was knowledge of the allocated intervention
adequately prevented during the study?– Participants– Trial personnel
Were incomplete outcome data adequately addressed?
Are reports of the study free of suggestion of selective outcome reporting?
Was the study apparently free of other problems that could put it at a high risk of bias?
17
Domains of the Cochrane “Risk of Bias” tool
Review authors’ judgement: Was the allocation sequence adequately
generated? Was allocation adequately concealed? Was knowledge of the allocated intervention
adequately prevented during the study?– Participants– Trial personnel
Were incomplete outcome data adequately addressed?
Are reports of the study free of suggestion of selective outcome reporting?
Was the study apparently free of other problems that could put it at a high risk of bias?
18
Separate assessment for each outcome
The “Risk of Bias tool” (RoB)General principles
19
2 steps What was reported
Extracted text: what was reported in the published report / protocol/ contact with authors
Commentary
Judgment relating to the risk of bias Low risk of bias High risk of bias Unclear (judgment is impossible)
The “Risk of Bias tool” (RoB)General principles. What was reported?
Sequence generation. Low risk
Quote: “patients were randomly allocated”.Comment: Probably done, since earlier reports from the same investigators clearly describe use of random sequences (Cartwright 1980).
Blinding of participants and personnel (performance bias)
Low risk
Quote: “double blind, double dummy”; “High and low dose tablets or capsules were indistinguishable in all aspects of their outward appearance. For each drug an identically matched placebo was available”.Comment: Probably done
Blinding of outcome assessment (detection bias) (Mortality)
Low risk
Quote: “Obtained from medical records”Comment: review authors do not believe this will introduce bias.
etc
20
The “Risk of Bias tool” (RoB)General principles. Judgment
High risk of bias Bias of sufficient magnitude to have a notable impact on the results
Unclear risk of bias Insufficient details reported Appropriate reporting, but the risk of bias is unknown
21
Attrition bias: Considerations
How much data is missing from each group?
Why is data missing in each group?
How were data analysed? Handling of incomplete outcome data
22
Attrition biasLow risk of bias• No missing outcome data
• Reasons for missing data not related to outcome
• Missing data balanced across groups, with similar reasons
• Missing data not enough to have a clinically relevant impact on the intervention effect estimate
• Missing data have been imputed using appropriate methods
23
Attrition biasHigh risk of bias• Reason for missing data related to outcome, with either imbalance in numbers or reasons
• Missing data enough to induce clinically relevant bias in intervention effect estimate
• ‘As-treated’ analysis with substantial departure of the intervention received from that assigned at randomization
• Inappropriate use of imputation24
The “Risk of Bias tool” (RoB)General principles
Reviewers specifically trained
Independent duplicate assessment with consensus
Decisions need to be pre-specified in the protocol Classification of outcomes (subjective / objective) Blinding: successful blinding procedure Missing data Other risk of bias
Contact authors for missing information25
Selective outcome reporting
Concern is that there is a tendency to selectively report outcomes that are statistically significant– Empirical evidence [Kirkham et al, BMJ 2010]
Challenges in detecting such behaviour
Ideally need study protocol May be internal evidence from publications or
trials register Usually have to assess this item as “Unclear”
26
Risk of bias summary – each study
27
Risk of bias summary – all studies
28
Summary assessment of the risk of bias across trials
ClassificationLow risk of bias Most information is from trials at low risk of biasUnclear risk of bias Most information is from trials at low or unclear risk
of biasHigh risk of bias The proportion of information from trials at high
risk of bias is sufficient to affect the interpretation of results
29
Risk of bias for non-randomised studiesNonrandomised studies of interventions: Many issues are the same Main concern is selection bias – how were treatment
chosen? High risk of non-comparable groups
Some tools exist – e.g. “Newcastle-Ottawa” Cochrane Collaboration is planning to develop a RoB
tool
Other observational studies – e.g. prognosis, aetiology: Parallel developments are needed
30
Conclusions Assessing the risk of bias is an essential step for an appropriate interpretation of systematic reviews and meta-analysis
Cochrane Risk of Bias tool includes 7 items to be evaluated for RCTs Need for transparency Training and use of the handbook recommendations
Similar approach should be adopted for other study types
31
Closing comments
Good systematic reviews are a major source of reliable information on the benefits and harms of health care interventions
A key aspect is assessment of risk of bias of the primary studies
Cochrane Risk of Bias tool includes 7 items to be evaluated for RCTs– A similar approach is needed for other study types
Should take account of risk of bias in the analysis and interpretation
32
33