1

Variable Selection for Tailoring Treatment

L. Gunter, J. Zhu & S.A. Murphy

ASA, Nov 11, 2008

2

Outline

• Motivation

• Need for Variable Selection

• Characteristics of a Tailoring Variable

• A New Technique for Finding Tailoring Variables

• Comparisons

• Discussion

3

Motivating ExampleSTAR*D "Sequenced Treatment to Relieve Depression"

Preference Treatment Intermediate Preference Treatment Intermediate Treatment Two Outcome Three Outcome Four

Follow-up Follow-up

CIT + BUS Remission L2-Tx +THY Remission

Augment R Augment RTCP

CIT + BUP L2-Tx +LI

CIT Non- Non- Rremission remission

BUP MIRTMIRT + VEN

Switch R Switch RVEN

SER NTP

30+ baseline variables, 10+ variables at each treatment level, both categorical and continuous

4

Simple Example

Nefazodone - CBASP Trial

Randomization

Nefazodone

Nefazodone + Cognitive Behavioral Analysis System of Psychotherapy (CBASP)

50+ baseline covariates, both categorical and continuous

5

Simple Example

Nefazodone - CBASP Trial

Which variables in X are important for tailoring the treatment?

Xpatient’s medical history, severity of depression, current symptoms, etc.

A Nefazodone OR Nefazodone + CBASP

R depression symptoms post treatment

6

Optimization

• We want to select the treatment that “optimizes” R

• The optimal choice of treatment may depend on X

],|[maxarg aAXREa

7

Optimization

• The optimal treatment(s) is given by

• The value of d is

8

Need for Variable Selection

• In clinical trials many pretreatment variables are collected to improve understanding and inform future treatment

• Yet in clinical practice, only the most informative variables for tailoring treatment can be collected.

• A combination of theory, clinical experience and statistical variable selection methods can be used to determine which variables are important.

9

Current Statistical Variable Selection Methods

• Current statistical variable selection methods focus on finding good predictors of the response

• Also need variables to help determine which treatment is best for which types of patients, e.g. tailoring variables

• Experts typically have knowledge on which variables are good predictors, but intuition about tailoring variables is often lacking

10

What is a Tailoring Variable?

• Tailoring variables help us determine which treatment is best

• Tailoring variables qualitatively interact with the treatment; different values of the tailoring variable result in different best treatments.

No Interaction Non-qualitative Interaction Qualitative interaction

0.0 0.4 0.8

0.0

0.4

0.8

X1

R

A=1

A=0

0.0 0.4 0.8

0.0

0.4

0.8

X2

R

A=1

A=0

0.0 0.4 0.8

0.0

0.4

0.8

X3R

A=0

A=1

11

Qualitative Interactions

• Qualitative interactions have been discussed by many within stat literature (e.g. Byar & Corle,1977; Peto, 1982; Shuster & Van Eys, 1983; Gail & Simon, 1985; Yusuf et al., 1991; Senn, 2001; Lagakos, 2001)

• Many express skepticism concerning validity of qualitative interactions when found in studies

• Our approach for finding qualitative interactions should be robust to finding spurious results

12

Qualitative Interactions

• We focus on two important factors– The magnitude of the interaction between the

variable and the treatment indicator– The proportion of patients for whom the best choice

of treatment changes given knowledge of the variable

big interaction small interaction big interaction

big proportion big proportion small proportion

0.0 0.4 0.8

0.0

0.4

0.8

X4

R

A=0

A=1

0.0 0.4 0.8

0.0

0.4

0.8

X5

R

A=0

A=1

0.0 0.4 0.8

0.0

0.4

0.8

X6R

A=0

A=1

13

Ranking Score S

• Ranking Score:

where

• S estimates the quantity described by Parmigiani (2002) as the value of information.

ˆ ˆ ˆmax | , | , *j j j j ja

S E E R X x A a E R X x A a

0.0 0.4 0.80

.00

.6

Xj

R

A=0

A=1=a*

]|[ˆmaxarg* aAREaa

14

Ranking Score S

• Higher S scores correspond to higher evidence of a qualitative interaction between X and A

• We use this ranking in a variable selection algorithm to select important tailoring variables.– Avoid over-fitting in due to large

number of X variables– Consider variables jointly

],|[ˆ AXRE

15

Variable Selection Algorithm

1. Select important predictors of R from (X, X*A) using Lasso

-- Select tuning parameter using BIC

2. Select all X*A variables with nonzero S.-- Use predictors from 1. to form linear

regression estimator of to form S.],|[ˆ AXRE

(using linear models)

16

Lasso• Lasso on (X, A, XA) (Tibshirani, 1996)

– Lasso minimization criterion:

where Zi is the vector of predictors for patient i, λ is a penalty parameter

– Coefficient for A not penalized

– Value of λ chosen by Bayesian Information Criterion (BIC) (Zou, Hastie & Tibshirani, 2007)

n

iii ZR

11

2minarg)(ˆ

17

Variable Selection Algorithm

3. Rank order (X, X*A) variables selected in steps 1 & 2 using a weighted Lasso

-- Weight is 1 if variable is not an interaction

-- Otherwise weight for kth interaction is

-- is a small positive number.

-- Produces a combined ranking of the selected (X, X*A) variables (say p variables).

18

Variable Selection Algorithm

4. Choose between variable subsets using a criterion that trades off maximal value of information and complexity.

-- The ordering of the p variables creates p subsets of variables. Estimate the value of information for each of the p subsets

-- Select the subset, k with largest

pkVVk ,...,1,ˆˆ0

19

Simulations

• Data simulated under wide variety of realistic decision making scenarios (with and without qualitative interactions)– Used X from the CBASP study, generated new A and

R

• Compared:– New method: S with variable selection algorithm – Standard method: BIC Lasso on (X, A, XA)

• 1000 simulated data sets: recorded percentage of time each variable’s interaction with treatment was selected for each method

20

Simulation Results

Generative ModelAve # of Spurious

Interactions Selected over BIC LASSO

Ave % increase in Value over

BIC LASSO*

No Interactions 0.5 -0.03

Non-qualitative

Interactions Only 0.1 0.00

Qualitative Interaction

Only 1.1 0.23

Both Qualitative and

Non-qualitative Interactions 0.2 0.39

* Over the total possible increase; 1000 data sets each of size 440

21

Simulation Results

• Pros: when the model contained qualitative interactions, the new method gave significant increases in expected response over BIC-Lasso

• Cons: the new method resulted in a slight increase in the number of spurious interactions over BIC-Lasso

22

Nefazodone - CBASP Trial

Aim of the Nefazodone CBASP trial – to compare efficacy of three alternate treatments for major depressive disorder (MDD):1. Nefazodone, 2. Cognitive behavioral-analysis system of

psychotherapy (CBASP) 3. Nefazodone + CBASP

Which variables might help tailor the depression treatment to each patient?

23

Nefazodone - CBASP Trial

• For our analysis we used data from 440 patients with

X 61 baseline variables

A Nefazodone vs. Nefazodone + CBASP

RHamilton’s Rating Scale for Depression score, post treatment

24

Method Application and Confidence Measures

• When applying new method to real data it is desirable to have a measure of reliability and to control family-wise error rate

• We used bootstrap sampling to assess reliability– On each of 1000 bootstrap samples:

1. Run variable selection method

2. Record the interaction variables selected

– Calculate selection percentages over bootstrap samples

25

Error Rate Thresholds

• To help control family-wise error rate, compute the following inclusion thresholds for selection percentages:

1. Repeat 100 timesa. Permute interactions to remove effects from the data

i. Run method on 1000 bootstrap samples of permuted dataii. Calculate selection percentages over bootstrap samples

b. Record largest selection percentage over the p interactions

2. Threshold: (1-α)th percentile over 100 max selection percentages

• Select all interactions with selection percentage greater than threshold

26

Error Rate Thresholds

• When tested in simulations using new method, error rate threshold effectively controlled family-wise error rate

• This augmentation of bootstrap sampling and thresholding was also tested on BIC Lasso and effectively controlled family-wise error rate in simulations

27

Nefazodone - CBASP Trial

0 20 40 60

02

04

0

BIC Lasso

variable number

% o

f tim

e c

ho

sen

0 20 40 600

20

40

New Method

variable number

% o

f tim

e c

ho

sen

OCD

OCD

ALC

ALC

28

Interacion Plot

Alcohol Dependence

Fitte

d R

25

30

35

0 1

Txt=Combo

Txt=Nef

Interaction Plot

29

Interacion Plot

Obsessive Compulsive Disorder

Fitte

d R

10

15

20

25

30

0 1

Txt=Combo

Txt=Nef

Interaction Plot

30

Discussion• This method provides a list of potential

tailoring variables while reducing the number of false leads.

• Replication is required to confirm the usefulness of a tailoring variable.

• Our long term goal is to generalize this method so that it can be used with data from Sequential, Multiple Assignment, Randomized Trials as illustrated by STAR*D.

31

• Email Susan Murphy at samurphy@umich.edu for more information!

• This seminar can be found at http://www.stat.lsa.umich.edu/~samurphy/seminars/ASA11.11.08.ppt

• Support: NIDA P50 DA10075, NIMH R01 MH080015 and NSF DMS 0505432

• Thanks for technical and data support go to– A. John Rush, MD, Betty Jo Hay Chair in Mental Health at the

University of Texas Southwestern Medical Center, Dallas– Martin Keller and the investigators who conducted the trial `A

Comparison of Nefazodone, the Cognitive Behavioral-analysis System of Psychotherapy, and Their Combination for Treatment of Chronic Depression’

32

Interacion Plot

Alcohol Dependence

Fitte

d R

25

30

35

0 1

Txt=Combo

Txt=Nef

Interaction Plot

33

Interacion Plot

Obsessive Compulsive Disorder

Fitte

d R

10

15

20

25

30

0 1

Txt=Combo

Txt=Nef

Interaction Plot

34

Lasso Weighting Scheme

• Lasso minimization criterion equivalent to:

so smaller wj means greater importance

• Weights where

– vj = 1 for predictive variables

– vj = for prescriptive variables

n

i

p

jjjiii wZRw

1 1

2minarg)(ˆ

pk kjj vvw 1

))((max kkj SS

35

AGV Criterion

• For a subset of k variables, X{k} the Average Gain in Value ( AGV) criterion is

where

• The criterion selects the subset of variables with the maximum proportion of increase in E[R] per variable

{ }

{ *}

ˆ ˆmax [ | , ] [ | *] *ˆ ˆmax [ | , ] [ | *]

ka

k

ma

E R X A a E R A a mAGV

kE R X A a E R A a

{ }* arg max max [ | , ]kak

m E R X A a

36

Simulation Results (S-score)

× Qualitative Interaction Spurious Interaction

× Qualitative Interaction Non-qualitative Interaction Spurious Interaction

0 20 40 60

02

06

0

New Method

variable number

% o

f tim

e c

ho

sen

0 20 40 60

02

06

0

BIC Lasso

variable number

% o

f tim

e c

ho

sen

0 20 40 60

01

02

0

New Method

variable number

% o

f tim

e c

ho

sen

0 20 40 60

01

02

0

BIC Lasso

variable number

% o

f tim

e c

ho

sen