Dynamic Treatment Regimes:Challenges in Data Analysis

S.A. MurphySurvey Research Center

January, 2009

2

Outline

• What are Dynamic Treatment Regimes?

• Myopic Decision Making

• Constructing Regimes

• Q-Learning

• Example using CATIE

3

Dynamic Treatment Regimes operationalize multi-stage decision making.

These are individually tailored sequences of interventions, with intervention type and dosage adapted to the individual.

•Generalization from a one-time decision to a sequence of decisions concerning interventions

•Operationalize clinical practice.

Each decision corresponds to a stage of intervention

4

Dynamic Treatment Regime “Jobs First” Welfare Program

• At each stage of intervention– Use individual characteristics (assets, income, age,

health, employment), characteristics of the environment (domestic violence, incapacitated family member, # children, living arrangements…),

– To select actions/interventions such as child care, job search skills training, amount of cash benefit, medical assistance, education,

– In order to maximize long term rewards (maximize employment/independence over longer term).

5

6

Why use a Dynamic Treatment Regime?

– High heterogeneity in response to any one intervention

• What works for one person may not work for another

• What works now for a person may not work later

– Improvement often marred by relapse• Remitted or few current symptoms is not the same

as cured.

– Co-occurring disorders/adherence problems are common

7

Outline

• What are Dynamic Treatment Regimes?

• Myopic Decision Making

• Constructing Regimes

• Q-Learning

• Example using CATIE

8

Myopic Decision Making

• In myopic decision making, decision makers use regimes that seek to maximize immediate rewards.

Problems:– Ignore longer term consequences of present actions.– Ignore the range of feasible future actions/interventions– Ignore the fact that immediate responses to present actions

may yield information that pinpoints best future actions

(A dynamic treatment regime tells us how to use the observations to choose the actions/interventions.)

9

Treatment of Schizophrenia

• Myopic action: Offer patients a treatment that reduces schizophrenia symptoms for as many people as possible.

• The result: Some patients are not helped and/or experience abnormal movements of the voluntary muscles (TDs). The class of subsequent medications is greatly reduced.

• The mistake: We should have taken into account the variety of treatments available to those for whom the first treatment is ineffective.

• The message: Use an initial medication that may not have as large a success rate but that will be less likely to cause TDs.

10

Treatment of Opioid Dependence• Myopic action: Choose an intensive multi-component

treatment (methadone + counseling + behavioral contingencies) that immediately reduces opioid use for as many people as possible.

• The result: Behavioral contingencies are burdensome/expensive to implement and many people may not need the contingencies to improve.

• The mistake: We should allow the patient to exhibit poor adherence prior to implementing the behavioral contingencies.

• The message: Use an initial treatment that may not have as large an immediate success rate but carefully monitor patient adherence to ascertain if behavioral contingencies are required.

11

Outline

• What are Dynamic Treatment Regimes?

• Myopic Decision Making

• Constructing Regimes

• Q-Learning

• Example using CATIE

12

Action ActionObservations Observations Reward

Stage 1 Stage 2 Stage 1 Stage 2

Basic Idea for Constructing a Regime: Move Backwards Through Stages.

(Pretend you are “All-Knowing”)

13

2 Stages for each individual

Observations available at jth stage

Action at jth stage

14

2 Stages

History available at each stage

Primary Outcome/Reward:

15

A dynamic treatment regime is the sequence of decision rules:

A simple decision rule is: given weights β, switch treatment at stage j if

otherwise maintain on current treatment; Sj is a vector summary of the history, Hj.

16

Goal:

Use data to construct decision rules that input information in the history at each stage and output a recommended decision; these decision rules should lead to a maximal mean Y.

In the future we employ the actions recommended by

the decision rules:

17

Example of Decision Rules

Treatment of depression. Goal is to achieve and maintain remission.

Provide Citalopram for up to 12 weeks gradually increasing dose as required.

If either the maximum dose has been provided for two weeks, or 12 weeks have occurred, yet there is no remission, then

if there has been a 50% improvement in symptoms, augment with Mirtazapine.

else switch treatment to Bupropion.

Else (remission is achieved) maintain on Citalopram and provide web-based disease management.

18

Idealized Data for Constructing the Dynamic Treatment Regime:

Data from sequential, multiple assignment, randomized trials in which at each stage subjects are randomized among alternative options.

That is, Aj is a randomized action with known randomization probability.

Binary actions with P[Aj=1]=P[Aj=-1]=.5

19

Outline

• What are Dynamic Treatment Regimes?

• Myopic Decision Making

• Constructing Regimes

• Q-Learning

• Example using CATIE

20

Regression-based methods for constructing decision rules

•Q-Learning (Watkins, 1989) (a popular method from computer science)

•A-Learning or optimal nested structural mean model (Murphy, 2003; Robins, 2004)

•The first method is an inefficient version of the second method when each stages’ covariates include the prior stages’ covariates and the actions are centered to have conditional mean zero.

21

Action ActionObservations Observations Reward

Stage 1 Stage 2 Stage 1 Stage 2

Basic Idea for Constructing a Regime: Move Backwards Through Stages.

(Pretend you are “All-Knowing”)

22

(k=2)

Dynamic Programming

23

Approximate for S', S vector summaries of the history,

A Simple Version of Q-Learning –binary actions

• Stage 2 regression: Use least squares with outcome, Y, and covariates to obtain

• Set

• Stage 1 regression: Use least squares with outcome, and covariates to obtain

24

Approximate for S', S vector summaries of the history,

A Simple Version of Q-Learning –binary actions

Stage j decision rule:

Select treatment = 1 if

Otherwise select treatment = -1

25

Outline

• What are Dynamic Treatment Regimes?

• Myopic Decision Making

• Constructing Regimes

• Q-Learning

• Example using CATIE

26

Clinical Antipsychotic Trials of Intervention Effectiveness

(Schizophrenia)

• Multi-stage trial of 18 months duration• Relaxed entry criteria • A large number of sites representing a broad

array of clinical settings (state mental health, academic, Veterans’ Affairs, HMOs, managed care)

• Approximately 1500 patients

27

CATIE Randomizations (simplified)

Phase 1

Randomized Treatments OLAN QUET RISP ZIPR PERP Phase 2

Treatment preference Efficacy TolerabilityRandomized Treatments CLOZ OLAN QUET RISP OLAN QUET RISP ZIPR

Phase 3Treatments selected many options by preference

28

Preliminary Analyses

• Reward: Time to Treatment Dropout• Phase 1 analysis:

– Controls: TD, recent exacerbation, site– Tailoring variable: pretreatment PANSS

• Phase 2 analysis: – Controls: TD, recent exacerbation, site– Tailoring variables: “treatment preference,” phase 1

treatment, end of phase 1 PANSS

Constructing Dynamic Treatment Regimes using CATIE

29

30

31

Preliminary Analyses

Myopic versus Non-myopic Analyses

• Reward: Integrated Quality of Life (QoL)• Phase 1 analysis:

– Controls: TD, recent exacerbation, site– Tailoring variable: pretreatment QoL

• Phase 2 analysis: – Controls: TD, recent exacerbation, site– Tailoring variables: “treatment preference,” phase 1

treatment, end of phase 1 QoL

32

33

34

Challenges

• It is extremely challenging to provide measures of confidence that possess “good frequentist properties.”

• Clinical Decision Support Systems– We need to be able construct dynamic treatment regimes

that recommend a group of treatment actions when there is no evidence that a particular treatment action is best.

• Even in this randomized trial setting, the most straightforward analyses are subject to confounding bias. Some methods to avoid confounding bias are available.

35

Acknowledgements: This presentation is based on work with many individuals including Eric Laber, Dan Lizotte, John Rush, Scott Stoup, Joelle Pineau, Daniel Almirall and Bibhas Chakraborty,.

Email address: samurphy@umich.edu

Slides with notes at:

http://www.stat.lsa.umich.edu/~samurphy/

Click on seminars > health science seminars

36

Causal Inference Challenges Behavioral/Social/Medical Sciences

• Incomplete mechanistic models – Unknown causes

• Use data on individuals to combat the dearth of mechanistic models.– Drawback: non-causal “associations” occur due

to the unknown causes of the observations.

37

Unknown UnknownCauses Causes

Observations Treatment Observations Treatment RewardStage 1 Stage 2

Stage 1 Stage 2

Unknown, Unobserved Causes

(Incomplete Mechanistic Models)

38

Maturity/Unknown DecisionCauses to join "Adult"

Society

+

-

Binge Drinking Counseling on - Binge Drinking Sanctions Functionality

Yes Health Yes/No + counselingConsequences Time 2 Yes/No Time 3 Yes/No

Unknown, Unobserved Causes (Incomplete Mechanistic Models)

39

Unknown High SATCauses Scores

+

+

Observations Student + Student Treatment Gradesis a superior admitted to Time 2athlete University

Unknown, Unobserved Causes (Incomplete Mechanistic Models)

40

• The problem: Even when treatments are randomized, non-causal associations occur in the data.

• The solution: Statistical methods should appropriately “average” over the non-causal associations between treatment and reward.

Unknown, Unobserved Causes (Incomplete Mechanistic Models)

41

Unknown MaturityCauses of Student

+ -

Binge Drinking Treatment Frequent Drinking Treatment FunctionalityTime 1 Binge Drinking Time 2

Time 2

Unknown, Unobserved Causes (Incomplete Mechanistic Models)

42

Unknown, Unobserved Causes

• Problem: We recruit students via flyers posted in dormitories. Associations between observations and rewards are highly likely to be (due to the unknown causes) non-representative.

• Solution: Sample a representative group of college students.

43

Summary of Solutions To Causal Problems

• If possible randomize treatments (e.g. actions).

• Develop methods that avoid being influenced by non-causal associations yet help you construct the policy.

• Subjects in your data should be representative of population of subjects.