HEALTH ECONOMIC

MODELLING IN THE

DIAGNOSTICS DEVELOPMENT

PROCESS

Peter Hall

Why Develop Biomarkers?

•Affect 60-70% of all critical clinical decisions

•Essential for stratified/personalised/precision medicine :

diagnosis, prognostication, monitoring, response to therapy……….

•Better biomarkers may lead to:

- improvements in patient outcomes

- improved use of NHS resources

A gap in biomarker translation

“biomarker X has potential…but further research is needed.”

0

2

4

6

8

10

12

Num

ber

of

cle

are

d o

r app

roved F

DA

IV

Ds

Year

In Vitro Diagnostic Device Regulatory Approvals

Diagnostics research challenges

Large sample size requirements

Lack of funding

Lack of commercial incentive

Methodological challenges

Protocol design challenges

Multiple technologies

Rapidly evolving market

Poorly defined regulatory standards

NICE Diagnostics Advisory Panel

Accepts alternative evidence availability

Decision modelling plays an important role

NIHR Diagnostic Evidence Co-operatives

Facilitate the generation of high-quality

evidence on in vitro diagnostic tests

Diagnostic Evidence Cooperatives

Stage 5:

Disseminate

Outputs

Stage 4: Model /

Evaluate Clinical Utility

Stage 3: Qualify

Clinical Validity

Stage 2: Qualify

Analytical Validity

Patient

Benefit

Economic

Value

-*Strategic decision making points

Stage 1: Selection, Modelling

& Prioritisation

IVD Developers

-Industry

-Academia

-NHS

* * *

Methodological expertise and resources

Pathway & Economic Modelling, Evidence Synthesis, Clinical

Informatics

*

Clinical expertise and resources

Research sites, Biobank samples & data, databases, registries

Stakeholders

-Patient & Public

-Commissioners

-NHS/NIHR

-Clinical Networks

-Trade

associations

Pertinent Methods (from Health Economists)

Early decision analysis for efficient research design

Clinical Pathway modelling for decision impact

Joined up modelling across validation stages

Decision modelling for tests

% True positives treated [correctly]

% False positives treated [incorrectly]

% True negatives not treated [correctly]

% False negatives not treated [incorrectly]

Early modelling for tests

% True positives treated [correctly]

% False positives treated [incorrectly]

% True negatives not treated [correctly]

% False negatives not treated [incorrectly]

QALYs

COSTs

QALYs

COSTs

QALYs

COSTs

QALYs

COSTs

Mean cost per QALY

Hospital

admission

Rehabilitation/

recovery

Late side-

effects

Chronic

disease

relapse

Death

Hospital

discharge Acute

disease

diagnosed Disease

response

Treatment

initiation

Clinical care pathway

Quality of Life Costs Survival

Hospital

admission

Rehabilitation/

recovery

Late side-

effects

Chronic

disease

relapse

Death

Hospital

discharge Acute

disease

diagnosed Disease

response

Treatment

initiation

Early

diagnosis

Risk

stratification

Treatment

benefit

prediction Monitoring

Where are tests useful?

Example – the OPTIMA trial

Modelling to inform a clinical trial design

Adjuvant chemotherapy for early breast cancer

Surgical resection aims to cure, but residual risk of

relapse.

Chemotherapy reduces risk of relapse, but risks side

effects, treatment mortality, and costs.

Decision historically based on predicted risk.

Effects of chemotherapy on breast

cancer survival

EBCCTG Lancet 2005 365: 1687

Effect of nodal status on survival Long-term results from NSABP B-06 (1039 patients)

Fisher Cancer 2001 31(S8) 1679-1687

Genomic profiling

Parker 2009 J Clin Oncol 27:1160

• PAM 50 genomic subtypes in early breast cancer

Oncotype DX® 21-Gene Recurrence Score (RS) Assay

PROLIFERATION

Ki-67

STK15

Survivin

Cyclin B1

MYBL2

ESTROGEN

ER

PR

Bcl2

SCUBE2

INVASION

Stromelysin 3

Cathepsin L2

HER2

GRB7

HER2

BAG1 GSTM1

REFERENCE

Beta-actin

GAPDH

RPLPO

GUS

TFRC

CD68

16 Cancer and 5 Reference Genes From 3 Studies

Category RS (0 -100) Low risk RS <18

Int risk RS 18 - 30

High risk RS ≥ 31

Paik et al. N Engl J Med. 2004;351:2817

RS = + 0.47 x HER2 Group Score

- 0.34 x ER Group Score

+ 1.04 x Proliferation Group Score

+ 0.10 x Invasion Group Score

+ 0.05 x CD68

- 0.08 x GSTM1

- 0.07 x BAG1

Oncotype DX NICE approval

Clinically low risk patients with ER +ve early breast cancer

Challenges

No prospective clinical trial data

Lack of UK clinical utility data

Lack of knowledge regarding UK clinical pathway

Rapidly evolving technologies

Many new competitors since appraisal

Many barriers to their evaluation and adoption

Multi-parameter signatures in high risk early breast cancer

Predict benefit from chemotherapy?

Oncotype DX

SWOG 8814 Trial

= chemotherapy

= no chemotherapy

Low risk

High risk

Clinical utility

Clinical validity

Reimbursement

decision (NICE/NHS)

Cost-effectiveness

Diagnostics development process

Analytical validity

Evidence synthesis Decision Model

Laboratory

Clinical

datasets

Clinical

studies

VoI analysis

Conduct a RCT?

(Standard care vs. test)

No chemotherapy

Chemotherapy

Chemotherapy

Test low risk

Test high risk

Model

Test - low

Model

Test - high

Model

Control

0.0

0.2

0.4

0.6

0.8

1.0

0 10 20 30 40 50

Qua

lity

-adju

sted

Surv

iva

l

Follow-up (years)

0.0

0.2

0.4

0.6

0.8

1.0

0 10 20 30 40 50

Qua

lity

-adju

sted

Surv

iva

l

Follow-up (years)

0.0

0.2

0.4

0.6

0.8

1.0

0 10 20 30 40 50

Qua

lity

-adju

sted

Surv

iva

l

Follow-up (years)

Markov model

Model specification

NHS and PSS perspective

Chemo effect (hazard ratio) from SWOG 8814 RCT

Chemo and tox costs from BNF/eMIT/NHS ref costs

Costs of recurrence – Karnon et al.

Utilities – UK RCTs (EQ-5D) Campbell et al.

RCT framework

Analytical validity

Evidence synthesis

Randomised feasibility

study

Economic Model

Clinical validity

VoI analysis

Economic Model Full RCT

Research design

VoI analysis

Research design

Testing logistics

Patient acceptability

Clinician acceptability

Test selection

STOP/GO STOP/GO

OPTIMA prelim

low score

high score R

1

1

Group 1

Group 2

trea

tment

ass

igne

d

by r

isk

Oncotype DX

Cent

ral elig

ibili

ty

conf

irm

ation

chemo.

chemo.

blinded to

randomisation

• 302 patients recruited

• Established acceptability of randomisation to patients & clinicians.

• Gain intelligence on the clinical pathway

• Evaluate the logistics and budget impact of testing

Candidate tests

Test Technology Parameters Location

Oncotype DX RT-PCR 21 genes /RNA Central (USA)

MammaPrint array 70 genes /RNA Central (NL)

BluePrint array 80 genes /RNA Central (USA)

PAM50/Prosigna RT-PCR 55 genes /RNA Central (USA)

BCI RT-PCR 7 genes /RNA Central (USA)

Randox BCA array 23 genes /RNA Regional?

Mammostrat IHC 7 proteins Regional?

IHC4 IHC 4 proteins Local? (QA)

IHC4 AQUA IHC 4 proteins Central

MammaTyper RT-PCR Genes/RNA Local

NPI plus IHC 10 proteins ?

MULTI-parameter assays evaluated in

OPTIMA prelim

PAM50

16 (+5) gene RT-PCR

performed by GHI

50 gene - nCounter

performed at OICR

70 (/80) gene array

performed by Agendia

4-gene fluorescent IHC

performed on TMA by Genoptix

IHC4 4-gene IHC

performed on TMA at OICR

Risk score low/ (int)/ high

Risk category low/ high

Subtyping Luminal A/B

Her2 Enriched, Basal

Risk score low/ int/ high

Risk score low/ int/ high

Risk score low/ int/ high

Subtyping Luminal A/B

Her2 Enriched, Basal

Subtyping Luminal A/B (int/ hi)

Her2 Enriched, Basal

4-gene RT-PCR

performed by Stratifyer MammaTyper

Interim modelling

Test results on 302 patients in OPTIMAprelim

(Oncotype DX, Prosigna ROR, Prosigna Subtype,

Mammaprint, MammaTyper, IHC4)

UK data on % high versus low-risk

Cost and QoL for chemo versus no chemo

comparison

Real world data on costs of cancer recurrence

The Doctor

Case notes

Chemotherapy

Radiotherapy

Surgery

MDT Pathology

Radiology

Blood results

Microbiology

Annotations

Letters

Appointments

Research / Audit

CWTs

PPM – clinical database for hospital care

- across Yorkshire network (catchment ~3 million) ICD10, OPCS, PLICS

Interim modelling – test comparison

No long term survival outcomes for tests other than

Oncotype DX

BUT ?? partially estimable from:

Risk prediction tool (Adjuvant! Online)

Modelled impact of test discordance

Modelling predictive effect

Parameterise predictive effect of Oncotype DX

Assign prior for predictive effect of Alternative Test

Weighted sample from these two distributions

posterior for predictive effect of Alternative test

% discordance

𝑙𝑜𝑔𝐻𝑅 = 𝛼 + 𝛽𝑅𝑆

Deliberative selection process

IHC-4

Poor analytical reproducibility at local role out

MammaTyper & IHC4-Aqua

Very limited clinical validity data

Value of Information Analysis

0 1000 2000 3000 4000

Oncotype DX (cut-off 25)

MammaPrint

Prosigna (ROR_P cut-off 61)

Prosigna (ROR_P cut-off 41)

Prosigna sub (lum A= low risk)

Population EVPPI for 5-year RFS (QALYs)

EVPPI for 5 year RFS parameters for RCT 2-way comparison vs standard care

Optimal cut-point

RCT framework

Analytical validity

Evidence synthesis

Randomised feasibility

study

Economic Model

Clinical validity

VoI analysis

Economic Model Full RCT

Research design

VoI analysis

Research design

Testing logistics

Patient acceptability

Clinician acceptability

Test selection

STOP/GO STOP/GO

Lessons learned

Close collaboration between industry, academia,

clinical community and funders is essential (and is a

lot of effort!)

Results of PSA easily misinterpreted

Improvements?

Modelling impact of test discordance

Quantitatively modelling analytical validity

Diagnostics development process

Evidence synthesis

Clinical utility (RCT)

Clinical validity

Reimbursement decision

(NICE/NHS)

Cost-effectiveness

Decision

Model

Analytical validity Laboratory

Clinical

datasets

Clinical

studies

Evidence synthesis Decision

Model

VoI Analysis

Evidence synthesis Decision

Model

VoI Analysis

Sources of Measurement Uncertainty

UM

Inter-dependency between measurement

uncertainty and clinical validity

Measurement uncertainty (UM), within-individual variation (CVwi), within-group variation (CVG) and differences between populations or groups (DG)

Summary

Diagnostics evaluation requires intelligent research design

Rapidly evolving multiple competitor technologies

Early modelling challenging due to data paucity and time limitation

Embedding economic modelling in interim analyses and stop-go decisions is feasible and should be encouraged

In this example – good buy-in from funders, some clinicians and manufacturers

National DEC methodology group

Value of early modelling – there was general agreement across the group regarding the high utility of early/interim economic modelling (i.e. conducted before large-scale phase II/III trials) to inform and optimise research pathways. There is a need to encourage greater utilisation of early modelling and ensure appropriate dissemination of such studies to increase awareness and understanding.

Importance of perspective/ budget impact analysis – current economic analyses conducted from a “NICE decision-maker” perspective may be failing to provide the type of data required by local commissioners/ decision makers. Future research should place greater emphasis on conducting analyses from alternative perspectives with a more detailed breakdown of costs and specific budget impacts.

Incorporating test performance measures – current economic evaluations of diagnostic technologies ignore the potential impact of test performance measures (such as analytical and pre-analytical validity) on the accuracy and utility of a test. There is a need to develop methodology to determine how such test performance measures should be incorporated in to economic evaluations.

Parameter elicitation – often elicitation of model parameter values from clinicians and experts is required in diagnostic evaluations due to paucity of data. There is a need to develop robust elicitation methods to inform diagnostic analyses.

Email: p.s.hall@ed.ac.uk

The Diagnostic Evidence Co-operative Leeds is funded by the National Institute for Health Research and is

a partnership between the Leeds Teaching Hospitals Trust and the Universities of Alberta, Edinburgh,

Leeds, Oxford, Southampton and University College London.

NIHR Leeds DEC: Peter Selby, Michael Messenger, et al.

Prof Chris McCabe, Prof Claire Hulme & AUHE Leeds

Rob Stein, Janet Dunn & the OPTIMA trial management group

Acknowledgements

This project was funded by the National Institute for Health Research Health Technology Assessment

programme (project number 10/34/01) and will be published in full in the Health Technology Assessment

Journal. Further information available at: http://www.nets.nihr.ac.uk/projects/hta/103401.

Research at the Ontario Institute for Cancer Research is funded by the Government of Ontario.

Agendia Inc., NanoString Technologies, Stratifyer/BioNTech Diagnostics, and Genoptix Medical

Laboratories supported testing by provision of reagents and test results (as appropriate) at no financial

cost to the current study.