Comparing apples and oranges? A Bayesian meta-regression of effect estimates from
non-randomized studies and randomized controlled trials
Lakhbir Sandhu, George Tomlinson, Erin D. Kennedy, Alice C. Wei, Nancy N. Baxter, David R. Urbach
Department of Surgery University of Toronto
Disclosures
• No financial disclosures or conflicts of interest
• Personal biases
– General Surgery Resident – PhD Candidate enrolled in the Clinical Epidemiology
Doctoral Program at the Institute of Health Policy, Management & Evaluation
– Montreal Canadians fan
Hierarchy of study design...
• Evidence from RCTs regarded as less biased than non-randomized studies (NRS)
NRS dominant study design in surgery...
• RCT study design is underrepresented in surgery – 3.4% of the articles in 20031
– 56% involved a comparison of analgesics, anesthetics, antibiotics or adjuvant cancer treatment
• Regulatory mechanisms do not encourage RCTs in surgery2
• Significant apprehension towards randomization by surgeons3
1Wente et al. Dig Surgery 2003, 2 McLeod World J of Surgery 1999, 3 McCulloch et al. British J of Surgery 2005
Study characteristics that lead to biased effect estimates have been investigated for RCTs
• Generation of randomization sequence • Concealment of allocation • Blinding
Schulz (1995), Moher (2003), Kjaergard (2000), Juni (2000), Pildal (2007), Wood (2008),
Nuësch (2009), Boutron (2010), Hrobjartsson (2012), Savovic (2012).
Comparisons of NRS and RCTs to date...
• Period effects • Clinical heterogeneity
Similar results Important Differences
Concato ei al. 2000 Benson and Hartz 2000
Britton et al. 1998 Shikata et al. 2006
Kunz, Vist, and Oxman 2007
Objective
To compare effect estimates from NRS with those from RCTs at low risk of bias.
Equipoise existed early on...
Is laparoscopy acceptable for the treatment of colon cancer?
Laparoscopy was recommended in the context of a research setting only1
Martel and Boushey, Surgical Clinics of North America (2
Methods
Medline & EMBASE search
(1980-2010)
Piloted & standardized
data extraction form
Abstracted multiple
outcomes
Meta-epidemiological (meta-regression)
analysis
Outcomes
• Two subjective outcomes – Post-operative complications – Length of stay
• Two objective outcomes
– 30-day peri-operative mortality – Number of lymph nodes harvested
Gold standard for analyses
• Cochrane Risk of Bias Tool – Low risk of bias
• Random sequence generation • Allocation concealment • Incomplete outcome data • Selective outcome reporting • Other bias*
– Blinding domains non-contributory
* Converted patients analyzed in LAP group
Results 7832 abstracts
7628
204 Duplicates
Excluded 6486
Rectal Ca 343
LAP vs OC 323
Included 192
Foreign Language
40
Meta-analyses/ Systematic Reviews
50
Molecular Studies
43
Case Series 476
4 4 4
9
2 3
4
7
5
7
10
4 5
11
14 15
16
18
0 0 1 1
2 2 1
2
0
4
1
6
4
2
6 5
3
5
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Comparative Lap vs OC Studies Published 1993-2010
NRS RCTs
Included Studies
Low-risk of bias RCTs
• Across four outcomes, four RCTs consistently identified at low risk of bias – Nelson 2004, Guillou 2005, Veldkamp 2005,
Hewett 2008 • Multinational studies • Publicly funded • Large sample sizes
Post-Operative Complications
Frequentist Meta-regression Results
Length of Stay
Mortality
Number of Lymph Nodes Harvested
Sensitivity Analysis
Post-operative complications
• ROR for design predictor variable remains statistically significant.
• Remainder of results (with other outcomes remained robust)
Discussion
• Three main findings: 1) Among subjective outcomes, NRS had more extreme
estimates of benefit for laparoscopy than Strong RCTs
2) Among subjective outcomes, effect estimates from Typical RCTs were similar to those from NRS.
3) There was significant between-study heterogeneity across all four outcomes, and
• NRS were more heterogeneous than Typical or Strong RCTs.
NRS vs RCTs in Surgery
• NRS evaluating arthroplasty and internal fixation for hip fracture were compared with the results of RCTs. – 13 NRS and 12 RCTs – RR mortality with arthroplasty as compared with internal
fixation in NRS was 40% larger than the estimate in RCTs • 1.44 in NRS (95% CI 1.13,1.85) • 1.04 in RCTs (95% CI, 0.84,1.29)
• Magnitude of bias in our study is similar to the bias detected by
Bhandari et al., but the direction of bias is not. Bhandari et al. Archives of Orthopaedic and Trauma Surgery 2004
Strengths & Limitations
• Strengths – Incorporated a consideration of study quality – Sensitivity analyses addressing period effects and
patient case-mix • Limitations
– reliance on reported study methods – baseline event rate – external validity
Implications
• Evidence synthesis in surgery
• Additional studies (in other subject areas) required to confirm findings
• Further investigation required to empirically identify characteristics of NRS associated with bias
