1 Possible Roles for Reinforcement Learning in Clinical Research S.A. Murphy November 14, 2007.

1

Possible Roles for Reinforcement Learning in

Clinical Research

S.A. Murphy

November 14, 2007

2

OutlineGoal: Improving Clinical Decision Support

Systems Using Data

– Clinical Decision Support Systems– Critical Decisions– Types of Data– Challenges

• Incomplete, primitive, mechanistic models

• Measures of Confidence

– Clinical Trials

3

4

5

QuestionsPatient Evaluation Screen with MSE

6

7

Outline


• Incomplete mechanistic models


– Clinical Trials

8

Critical Decisions

• Which treatments should be offered first?

• How long should we wait for these treatments to work?

• How long should we wait before offering a transition to a maintenance stage?

• Which treatments should be offered next?

9

Critical Decisions

• All of these questions relate to the formulation of a policy.

• Actions include medications, behavioral therapies, delivery mechanisms, monitoring

• Observations include biological measures, family history, severity, side effects, functionality, symptoms

• Rewards include functionality, side effects, symptoms

10

Outline




– Clinical Trials

11

Types of Data

• Large Observational Data Sets– Actions are not manipulated by scientist

• Clinical Trial Data– Actions are manipulated by scientist

• Bench research on cells/animals/humans

12

Clinical Trial Data Sets

• Experimental trial data collected for research purposes– Scientists decide proactively which data to collect and

how to collect this data

– Use scientific knowledge to enhance the quality of the proxies for observation, reward

– Actions are manipulated (randomized) by scientist

– Short Horizon (less than 5)

– Hundreds of subjects.

13

Observational Data Sets

• Observational data collected for research purposes– Scientists decide proactively which data to

collect and how to collect this data– Use scientific knowledge to enhance the quality

of the proxies for observation, action, reward– Actions are not manipulated by scientist– Moderate Horizon– Hundreds to thousands of subjects.

14

Observational Data Sets

• Clinical databases or registries– (an example in the US would be the VA registries)– Data was not collected for research purposes– Use gross proxies to define observation, action,

reward– Moderate to Long Horizon– Thousands to Millions of subjects

15

Outline




– Clinical Trials

16

Availability of Mechanistic Models

• In many areas of RL, scientists can use mechanistic theory, e.g., physical laws, to model or simulate the interrelationships between observations and how the actions might impact the observations.

• Scientists know many (the most important) of the causes of the observations and know a model for how the observations relate to one another.

17

Incomplete Mechanistic Models in Medical Sciences

• Scientists who want to use data on individuals to construct policies must confront the fact that non-causal “associations” occur due to the unknown causes of the observations.

18

Unknown Unknown Causes Causes

Observations Action Observations Action RewardTime 1 Time 2

Time 2 Time 3

Conceptual Structure in the Medical Sciences (observational data)

19

Unknown, Unobserved Causes

(Incomplete Mechanistic Models)

Maturity/Unknown DecisionCauses to join "Adult"

Society

+

+

Binge Drinking Treatment Binge Drinking Counseling Functionality Time 1 Time 2

Time 2 Time 3

20

Unknown, Unobserved Causes (Incomplete Mechanistic Models)

• Problem: Non-causal associations between treatment (here counseling) and rewards are likely.

• Solutions:– Collect clinical trial data in which treatments are

randomized. This breaks the non-causal associations yet permits causal associations.

– Prior to applying methods to observational data proactively brainstorm with domain experts to ascertain and measure the main determinants of treatment selection. Then take advantage of causal inference methods designed to minimize assumptions on the data

21

Maturity/

Unknown DecisionCauses to join "Adult"

Society

"+"

Observations Treatment Binge Drinking Counseling FunctionalityTime 1 Time 2

Time 2 Time 3


(Incomplete Mechanistic Models)

22

Maturity/

Unknown DecisionCauses to join "Adult"

Society

+ -

Binge Drinking Counseling on - Binge Drinking Sanctions FunctionalityYes Health Yes/No + counseling

Consequences Time 2 Yes/No Time 3 Yes/No


23

• The problem: Even when treatments are randomized, non-causal associations occur in the data.

• Solutions:– Recognize that parts of the Q-function/transition probabilities can

not be informed by domain expertise as these parts reflect non-causal associations

– Or use methods for constructing policies that “average” over the non-causal associations between action and reward.

• I think that the importance of this second causal inference problem depends on the kind of data and how you use it.


24

Measures of Confidence

– Measures of confidence are essential• Noisy data

• Need to know when any one of a subset of actions will yield the best rewards –that is, when there is no or little evidence otherwise.

• It is important to minimize the number of observations that must be collected in the clinical setting

25


• We would like measures of confidence for the following:– To compare the value of two estimated policies

(both estimated using the training data).– To assess if there is sufficient evidence that a

particular observation (e.g. output of a biological test) should be part of the policy.

– To assess if there is sufficient evidence that a subset of the actions lead to better rewards for a given observation than the remaining actions.

26


• I must both learn the policy and provide an evaluation of the policy using one data set.

• The data set is small

27


• Traditional methods for constructing measures of conference require differentiability (if frequentist properties are desired).

• Q-functions are constructed via non-differentiable operations (e.g. maximization).

• The value of a policy is a non-differentiable function of the policy.

28

Outline


• Causal ::: Unknown, unobserved causes


– Clinical Trials

29

Clinical Trials• Data from the --short horizon– clinical trials make

excellent test beds for combinations of supervised/unsupervised and reinforcement learning methods.– Developing methods for variable selection in decision

making (in addition to variable selection for prediction)

– Model selection when goal is learning good policies.

– Confidence intervals for the difference in value between two policies.

– Feature Construction

30

ExTENd

• Ongoing study at U. Pennsylvania (D. Oslin)

• Goal is to learn how best to help alcohol dependent individuals reduce alcohol consumption.

31

Oslin ExTENd

Late Trigger forNonresponse

8 wks Response

TDM + Naltrexone

CBIRandom

assignment:

CBI +Naltrexone

Nonresponse

Early Trigger for Nonresponse

Randomassignment:

Randomassignment:

Randomassignment:

Naltrexone

8 wks Response

Randomassignment:

CBI +Naltrexone

CBI

TDM + Naltrexone

Naltrexone

Nonresponse

32

Adaptive Treatment for ADHD

• Ongoing study at the State U. of NY at Buffalo (B. Pelham)

• Goal is to learn how best to help children with ADHD improve functioning at home and school.

33

ADHD Study

B. Begin low dosemedication

8 weeks

Assess-Adequate response?

B1. Continue, reassess monthly; randomize if deteriorate

B2. Increase dose of medication with monthly changes

as neededRandom

assignment:B3. Add behavioral

treatment; medication dose remains stable but intensity

of bemod may increase with adaptive modifications

based on impairment

No

A. Begin low-intensity behavior modification

8 weeks

Assess-Adequate response?

A1. Continue, reassess monthly;randomize if deteriorate

A2. Add medication;bemod remains stable butmedication dose may vary

Randomassignment:

A3. Increase intensity of bemod with adaptive modifi-

cations based on impairment

Yes

No

Randomassignment:

34

Studies under review

• H. Jones study of drug-addicted pregnant women (goal is to reduce cocaine/heroin use during pregnancy and thereby improve neonatal outcomes)

• J. Sacks study of parolees with substance abuse disorders (goal is reduce recidivism and substance use)

35

Jones’ Study for Drug-Addicted Pregnant Women

rRBT

2 wks Response

rRBT

tRBTRandom

assignment:

rRBT

Nonresponse

tRBT

Randomassignment:

Randomassignment:

Randomassignment:

aRBT

2 wks Response

Randomassignment:

eRBT

tRBT

tRBT

rRBT

Nonresponse

36

Sack’s Study of Adaptive Transitional Case Management

Standard Services

Standard TCM

Randomassignment:

Randomassignment:

4 wks Response

Standard TCM

Augmented TCM

Standard TCM

Nonresponse

37

Discussion

• Methods for online updating the policy as data accumulates.

• Methods for producing composite rewards.– High quality elicitation of functionality

• Human-Computer interface

• Improving tactics

38

This seminar can be found at:

http://www.stat.lsa.umich.edu/~samurphy/

seminars/UAlberta07.ppt

Email me with questions or if you would like a copy:

[email protected]

39


• Problem: We recruit students via flyers posted in dormitories. Associations between observations and rewards are highly likely to be (due to the unknown causes) non-representative.

• Solution: Sample a representative group of college students.

40

STAR*D

• This trial is over and the data is being analyzed (PI: J. Rush).

• One goal of the trial is construct good treatment sequences for patients suffering from treatment resistant depression.

www.star-d.org

41

Date post:	21-Dec-2015
Category:	Documents
View:	215 times
Download:	2 times

1 Possible Roles for Reinforcement Learning in Clinical Research S.A. Murphy November 14, 2007.

Documents