Illustration of the evaluation of risk prediction models in randomized trialsExamples from women’s health studies

Parvin Tajik, MDPhD candidateDepartment of Clinical Epidemiology & BiostatisticsDepartment of Obstetrics & GynecologyAcademic Medical Center, University of Amsterdam, the Netherlands

FHCRC 2014 Risk Prediction SymposiumJune 11, 2014

Clinical Problem I

Pre-eclampsia

fullPIERS model

Lancet, 2011

Development Method

• Patients: • 2000 women admitted in hospital for pre-eclapmsia

(260 event)

• Outcome: • Maternal mortality or other serious complications of

pre-eclampsia

• Logistic regression model with stepwise backward elimination

Final model

Logit P(D) = 2.68 – (0.054 × gestational age at eligibility) + (1.23 × chest pain or dyspnoea) – (0.027 × creatinine) + (0.21 × platelets) + (0.00004 × platelets2) + (0.01 × AST) – (0.000003 × AST2) + (0.00025 × creatinine × platelet) – (0.00007 × platelets × AST) – (0.0026 × platelets × SpO2)

Performance of full-PIERS model

Reported good risk discrimination and calibration

Online calculator

HYPITAT trial (2005-2008)

• PP Women at 36-41 wks of pregnancy with mild pre-eclampsia (n=750)

• I I Early Induction of labor (LI)

• C C Expectant monitoring (EM)

• O O Composite measure of adverse maternal outcomes

HYPTAT Results

(relative risk 0.71, 95% CI 0.59–0.86, p<0·0001)

ManagementManagement Adverse maternal Adverse maternal outcomesoutcomes

TotalTotal

Labor induction 117 (31%) 377Expectant monitoring 166 (44 %) 379

Modeling

Logit P(D=1|T,Y) = β0 + β1T + β2Y + β3TY

•D = 1 Adverse maternal outcome•Y = fullPIERS score•T = Treatment

• 1 Labor induction • 0 Expectant monitoring

FullPIERS for guiding labor induction

P for interaction: 0.93

fullPIERS score

Clinical Problem II

Preterm birth

Cervical pessary• Medical device inserted to vagina• to provide structural support to cervix

ProTWIN trial (2009-2012)

• P Women with multiple pregnancy (twin or triplet) between 12 & 20 weeks pregnancy

• I Cervical Pessary (n = 403)• C Control (n = 410)

• O Primary: Composite Adverse perinatal outcome

ProTWIN Results

(relative risk 0.98, 95% CI 0.69–1.39)

ManagementManagement Composite adverse Composite adverse perinatal outcomeperinatal outcome

TotalTotal

Pessary 53 (13%) 401No pessary 55 (14 %) 407

Pre-specified subgroup analysis

Cervical length (<38 mm vs >= 38 mm)

Pre-specified subgroup analysis

Trial Conclusion: Clinicians should consider a cervical pessary in women with a multiple pregnancy and a short

cervical length.

Cervical length Pessary group

Control group

RR (95%CI)

CxL < 38 mm 12% 29% 0.42 (0.19-0.91)CxL >= 38 mm 13% 10% 1.26 (0.74-2.15)

(P for interaction 0.01)

Other Markers

1. Obstetric history (parity) • Nulliparous• Parous with no previous preterm birth• Parous with at least one previous preterm birth

2. Chorionicity• Monochorionic• Dichorionic

3. Number of fetuses• Twin• Triplet

One marker at a time analysis

Other Potential Treatment Selection Factors

% Poor Outcome Odds Ratio (95% CI)

Odds Ratio (95% CI)

Int. P-value

Pessary Control

Cervical length

< 38 mm 11.54 29.09 0.32 (0.13-0.79) 0.010

≥ 38mm 12.85 10.13 1.31 (0.75-2.30)

Chorionicity

Monochorionic 13.79 26.00 0.46 (0.21-0.97) 0.015

Dichorionic 13.06 9.51 1.43 (0.86-2.37) Obstetric history

Nulliparous 13.12 18.30 0.67 (0.40-1.13) 0.212

Parous with no previous preterm birth 9.93 8.28 1.22 (0.56-2.66)

Parous with at least one previous preterm birth

31.03 3.85 11.25 (1.31-96.4) 0.012

Number of foetuses

Twin 12.50 13.32 0.98 (0.61-1.41) 0.301

Triplet 44.44 22.22 2.8 (0.36-21.73)

Modeling

Logit P(D=1|T,Y) = β0 + β1T + Σ βiYi + Σ βjTYj

•D = 1 composite poor perinatal outcome•Y = Markers•T = Treatment

• 1 pessary• 0 control

- Internal validation by bootstrapping

Multi-marker modelPredictor OR (95% CI) Beta*

P-value

Intercept

-2.08

<0.001 Main terms Pessary 1.13 (0.57-2.24) 0.12 0.426

Cervical length <38 mm 2.20 (1.09-4.46) 0.79 <0.001

Monochorionic 2.44 (1.33-4.47) 0.89 <0.001

Parous with no previous preterm birth 0.53 (0.27-1.06) -0.63 0.031

Parous with at least one previous preterm birth 0.34 (0.04- 2.63) -1.09 0.165

Triplet 1.49 (0.28- 8.05) 0.40 0.010

Interaction terms

Pessary × Cervical length <38 mm 0.52 (0.19-1.42) -0.65 0.058

Pessary × Monochorionic 0.41 (0.16-1.05) -0.89 0.009

Pessary × Parous with no previous preterm birth 1.52 (0.58-3.98) 0.42 0.312

Pessary × Parous with at least one previous preterm birth 7.24 (0.78-67.65) 1.98 0.020

* Shrunken with an average shrinkage factor of 0.76c-stat : 0,71 (95%CI: 0,66-0,77); optimism-corrected c-stat: 0,69 (95%CI: 0,63-0,74)

How can the model be used in practice?

Predicted benefit from pessary

050

100

150

200

250

Stu

dy

Pa

rtic

ipa

nts

, %

* ** *** ** ** ** **** * ** ** **** *** **** ***** * *** ** *** **** * ** *** ** **** * ***** ***** ** ** ** ** ***** ****

-0.2 -0.1 0.0 0.1 0.2

Predicted Difference (Control-Pessary) in Poor Perinatal Outcome

Favors Control Favors Pessary

Calibration of the predicted benefit

-30 -20 -10 0 10 20 30 40

-30

-20

-10

01

02

03

04

0

Expected Treatment Effect

Ob

serv

ed

Tre

atm

en

t Effe

ct

Model performance

-30 -20 -10 0 10 20 30 40

-30

-20

-10

01

02

03

04

0

Expected Treatment Effect

Ob

serv

ed

Tre

atm

en

t Effe

ct

Conclusion

• Common assumption for application of risk prediction models for treatment selection:“Being at higher risk of outcome implies a

larger benefit from treatment” • Not necessarily true

• Developing models using trial data and modeling the interaction between markers and treatment might be a more optimal strategy

Open Research Questions

• Optimal modeling strategy?

• Optimal algorithm for variable selection?

• Optimal method for optimism correction?

Thanks!Any Questions?

Multimarker vs. CxL only

Multimarker + Multimarker -

Short cervix 174 9

Long cervix 120 505

Two examples