1

Indirect Comparisons: heroism or heresy

Neil HawkinsWith acknowledgements to Sarah DeWilde

2

Outline

• Brief introduction to indirect comparisons• Two practical examples• Heroism or Heresy?

3

A Taxonomy of Comparisons

A

B

A C

B C

A B

A C

B C

A B

Direct Comparison

Naive or Unadjusted Indirect Comparison: Absolute effect estimates from individual trial arms.

‘Adjusted’ indirect comparison: Difference between relative treatment

Network Meta-Analysis:Adjusted Indirect comparison extended to more complex networks of trial evidence

4

Adjusted Indirect Comparisons

Based on an assumption of transitivity

• HRAB = HRAC/HRBC

• Log Transformation: LN HRAB = LN HRAC – LN HRBC

5

Assumptions

• Transitivity on the chosen scale:A-B = (A-C)-(B-C)• Requires exchangeability of relative treatment effects:

– Between Subjects Within trials (randomisation)– Between trials including the same comparators

(pairwise meta-analysis)– Between trials comparing different comparators– Ultimately, between different treatment comparisons

6

Indirect comparisons: heroism or heresy?

7

Stuart Pocock: Heresy

“… their statistical methods are so complex, “an extension of multivariable Bayesian hierarchical random effects models for mixed multiple treatment comparisons” that many are mystified by whether the conclusions make sense.” from Safety of drug-eluting stents: demystifying network meta-analysis

www.thelancet.com Vol 370 December 22/29, 2007

8

Lu & Ades: Heroism

“. . . to ignore indirect evidence either makes the unwarranted claim that it is irrelevant, or breaks the established precept of systematic review that synthesis should embrace all available evidence”

. Stat Med 2004;23:3105–24.

9

Cochrane: Not Sure

“Indirect comparisons are not randomized comparisons, and cannot be interpreted as such. They are essentially observational findings across trials, and may suffer the biases of observational studies, for example due to confounding. “

Reference: Cochrane Handbook 2.4.6.

10

Ref: NEJM 359;12

11

Could the (-ve) results of PRoFESS have been predicted?

AS

P

AS

P +

ER

DP

CLO

P

CAPRIE

ESPRIT

ESPS2

12

Network Graph (Stroke Endpoint)

ASP

CLOP ASP + ER DP

0.92(0.8:1.07) 0.79(0.67:0.92)

13

ASP + ERDP vs. CLOP

• Indirect Comparison: ESPS2, ESPRIT & CAPRIE Trials– Odds Ratio 0.85 ( 0.66:1.06 )

• Direct Comparison: PRoFESS Trial– Odds Ratio 1.02 (0.93 to 1.11)

• What happened?

Indirect Comparison ASP + ERDP vs CLOP Odds Ratio 0.85 (0.69:1.06)

14

Kent & Thaler (2008)“the results of the PRoFESS trial show us once again that

the compelling logic of the transitive property, so reliable in mathematics, has little authority in the often illogical world of clinical trials”

“Although the inconsistency among trial results should make us examine the trials for differences in design or populations that might support explanatory hand-waving, it is also reasonable to conclude from these comparisons that efficacy should not be the sole, or perhaps even the major, determinant of treatment decisions for antiplatelettherapy after stroke.”

15

Extended Trial Network

AS

P +

CLO

P

AS

P +

ER

DP

AS

P h

igh

dose

AS

P lo

w d

ose

AS

P m

ed d

ose

CLO

P

ATC2002

CAPRIE

CHARISMA

ESPRIT

ESPS2

MATCH

16

Extended Network Graph

ASP med dose

ASP + CLOP

CLOP

ASP + ER DP

ASP high dose

ASP low dose

0.83(0.66:1.05)

1.08(0.93:1.25)0.84(0.64:1.08)

1.06(0.95:1.18)

1.08(0.94:1.25)

1.59(1.39:1.83)

1.31(1.08:1.6)

1.5(1.34:1.68)

17

ASP + ERDP vs CLOP

• Extended Network Meta-Analysis: – Odds Ratio 1.11 ( 0.87 to1.4)

• Direct Comparison: PRoFESS Trial– Odds Ratio 1.02 (0.93 to 1.11)

• Extended Network Meta-Analysis + PRoFESS– Odds Ratio 1.03 ( 0.94 to 1.13 )

18

Kent and Thaler (2008)“In the era of comparative effectiveness, when

multiple agents are pitted against one another, randomized trials often cannot be understood in isolation. Rather, they need to be interpreted in the context of a sometimes complex network of other similar or relevant evidence.

The reduction of such complex networks to treatment recommendations is not always straightforward, since different paths within the network may give inconsistent results, and the network may be incoherent”

19Ref: Statist. Med. 2007; 26:1237–1254

20

NMA of Anti-TNFs in RA

Adalimunab

0.0 0.5 1.0 1.5 2.0 2.5

Log Odds Ratio of ACR50

Anakinra

Etanercept

Infliximab

MTX

21Ref: Statist. Med. 2007; 26:1237–1254

22

Adjusted for Disease Duration

0.0 0.5 1.0 1.5 2.0 2.5

Log Odds Ratio of ACR50

Anakinra

Etanercept

Infliximab

Adalimunab

MTX

23

Final thought: exchangeability is implicit in clinical decision-making

RCT: A vs Placebo: Exchangeability

RCT: B vs Placebo:

Future Patient:

24

Heroism, Heresy, or Pragmatism

– Formal methods of indirect comparisons are invaluable for analysing more complex networks

– Assumptions required for indirect comparisons are related to the assumptions of generalisability

– Identifying and seeking to explain the heterogeneity and incoherence identified in networks of trial evidence should improve our understanding of comparative effectiveness