1 Eloise E. Kaizar The Ohio State University Combining Information From Randomized and Observational...

11

Eloise E. KaizarThe Ohio State University

Combining Information From Combining Information From Randomized and Observational Randomized and Observational

Data: A Simulation StudyData: A Simulation Study

June 5, 2008

Joel Greenhouse

Howard SeltmanCarnegie Mellon University

22

OutlineOutline

Motivating ExampleMotivating Example– Association between suicidality and Association between suicidality and

antidepressant use in pediatric antidepressant use in pediatric populationpopulation

Trying to answer the right questionTrying to answer the right question Exploiting strengths of different dataExploiting strengths of different data Simulation StudySimulation Study

33

Pediatric Antidepressant UsePediatric Antidepressant Use

Problem: Antidepressant use may Problem: Antidepressant use may cause suicide for some cause suicide for some children/adolescentschildren/adolescents

Goal: Estimate the average Goal: Estimate the average treatment effect for use in regulatory treatment effect for use in regulatory decision makingdecision making

44

Randomized Controlled Randomized Controlled TrialsTrials

Hammad, et al. (2006) Archives of General Hammad, et al. (2006) Archives of General PsychiatryPsychiatry

55

The Right QuestionThe Right Question

Study population average treatment Study population average treatment effecteffect

Population average treatment effectPopulation average treatment effect

66

HeterogeneityHeterogeneity

Variation due to differences in Variation due to differences in population (“True”)population (“True”)

Variation due to differences in study Variation due to differences in study design (“Artifactual”)design (“Artifactual”)

77

Evidence for Weak External Evidence for Weak External ValidityValidity

Administrative dataAdministrative data– Show no significant association between Show no significant association between

antidepressant use and suicidal actions antidepressant use and suicidal actions (Valuck et al. 2004, Jick et al. 2004) (Valuck et al. 2004, Jick et al. 2004)

Epidemiological dataEpidemiological data– Suggest inverse relationship between Suggest inverse relationship between

antidepressant use and completed suicideantidepressant use and completed suicide Geographically (Gibbons et al. 2006, Isacsson Geographically (Gibbons et al. 2006, Isacsson

2000, Ludwig and Marcotte 2005)2000, Ludwig and Marcotte 2005) Temporally (Gibbons et al. 2007, Olfson, et al Temporally (Gibbons et al. 2007, Olfson, et al

1998)1998)

88

Assessing External ValidityAssessing External Validity

Compare the RCT patients with a Compare the RCT patients with a nationally representative probability nationally representative probability sample of adolescentssample of adolescents– Youth Risk Behavior Survey (YRBS)Youth Risk Behavior Survey (YRBS)– Representative of adolescents attending Representative of adolescents attending

school (aged 12-18)school (aged 12-18)– Basic demographic informationBasic demographic information– Self-report depressionSelf-report depression– Self-report suicidalitySelf-report suicidality

99

Match RCTs and YRBSMatch RCTs and YRBS

Consider only MDD RCTs of ages 12-18Consider only MDD RCTs of ages 12-18 Consider only YRBS respondents reporting Consider only YRBS respondents reporting

depressiondepression

YRBSYRBS RCTsRCTs

Average AgeAverage Age 16.14 (0.04)16.14 (0.04) 14.76 (2.99)14.76 (2.99)

% Female% Female 62.6 (1.8)62.6 (1.8) 63.8 (1.4)63.8 (1.4)

% White% White 54.1 (3.5)54.1 (3.5) 80.1 (1.2)80.1 (1.2)

Poststratify YRBS to match RCTsPoststratify YRBS to match RCTs

1010

Compare OutcomesCompare Outcomes

8-week suicidality8-week suicidality– RCTs 3.6%RCTs 3.6%– YRBS 7.1%YRBS 7.1%

Suicide attemptSuicide attempt– RCTs 5.4% (lifetime)RCTs 5.4% (lifetime)– YRBS 19.9% (12-month)YRBS 19.9% (12-month)

1111

Randomized Controlled Randomized Controlled TrialsTrials

Hammad, et al. (2006) Archives of General Hammad, et al. (2006) Archives of General PsychiatryPsychiatry

1212

Generalizing RCT DataGeneralizing RCT Data

Low RiskLow Risk

High RiskHigh Risk

Reduce the size of Reduce the size of the excluded the excluded populationpopulation– Practical Clinical Practical Clinical

TrialTrial Estimate the effect Estimate the effect

size in the size in the excluded excluded populationpopulation

1313

Current Approaches to Current Approaches to Estimating Average Effect SizeEstimating Average Effect Size

Use meta-analysis to combine RCT Use meta-analysis to combine RCT datadata– Assume effect is not systematically Assume effect is not systematically

heterogeneous by exclusion criteriaheterogeneous by exclusion criteria Use multi-level meta-analysis to Use multi-level meta-analysis to

combine RCT and observational datacombine RCT and observational data– Partial exchangeabilityPartial exchangeability– Assumes the mean is of interestAssumes the mean is of interest

Include bias parametersInclude bias parameters

1414

Proposed ApproachesProposed Approaches

Confidence Profile Method [Eddy, et al., 1988, Confidence Profile Method [Eddy, et al., 1988, 1989]1989]– Model the biases in observational and RCT dataModel the biases in observational and RCT data

Response Surface Approach [Rubin, 1990, Response Surface Approach [Rubin, 1990, 1991]1991]– Create a response surface that incorporates design Create a response surface that incorporates design

variablesvariables– Extrapolate to the ideal designExtrapolate to the ideal design

Cross Design Synthesis [US GAO, 1992]Cross Design Synthesis [US GAO, 1992]– Stratify data based on design variablesStratify data based on design variables– Extrapolate to empty cellsExtrapolate to empty cells

1515

Usefulness of EvidenceUsefulness of Evidence

External Validity

Inte

rnal

Val

idity

RCT

Obs.

IdealS

tron

ger

Wea

ker

StrongerWeaker

1616

FrameworkFrameworkSelf-Selection VariableSelf-Selection Variable

RandomizedRandomized(Strong Internal (Strong Internal

Validity)Validity)

Self-SelectedSelf-Selected(Weak Internal (Weak Internal

Validity)Validity)GeneralizabiliGeneralizabili

ty Variablety Variable

Eligible for Eligible for RandomizatioRandomizatio

nn

Ineligible for Ineligible for RandomizatioRandomizatio

nn

1717

FrameworkFrameworkSelf-Selection VariableSelf-Selection Variable


Validity)Validity)





nn


nn

1818

Self-Selection VariableSelf-Selection Variable


Validity)Validity)





nn


nn

FrameworkFramework

RandomizedExperiments

ObservationalStudies

1919



Validity)Validity)





nn


nn

Linear Bias ModelLinear Bias Model

2020

Simulation StudySimulation Study Goal:Goal:

– Use simulation study to investigate effectiveness Use simulation study to investigate effectiveness of different methods for combining information of different methods for combining information from diverse sources in a realistic settingfrom diverse sources in a realistic setting

Key characteristics:Key characteristics:– 24 high-quality experiments with complete 24 high-quality experiments with complete

compliance and uniform randomization eligibilitycompliance and uniform randomization eligibility 200 subjects, individual data unavailable200 subjects, individual data unavailable

– 1 high-quality observational study with no 1 high-quality observational study with no generalizability biasgeneralizability bias 25,000 subjects, individual data available25,000 subjects, individual data available

2121

Simulation Study:Simulation Study:ImplementationImplementation

Generate 1000 data setsGenerate 1000 data sets

Fit models using Bayesian approachFit models using Bayesian approach Compare on MSE, bias and coverageCompare on MSE, bias and coverage

ScenarioScenario 00 11 22 33 44

Effect SizeEffect Size 0.80.8 0.70.7 0.70.7 0.80.8 0.70.7

Generalizability Generalizability BiasBias

00 0.40.4 0.40.4 00 0.40.4

Self-Selection BiasSelf-Selection Bias 00 0.40.4 00 0.40.455

-0.4-0.4

2222

0.0 0.5 1.0 1.5 2.0

Scenario 0 Estimate of Population Effect

RSMIncorrect Coefficients

RSMCorrect Coefficients

Three Level PoolingIG Prior

Random Effects PoolingIG Prior

Fixed Effects Pooling

CDSLinear Formulation

TruePopulation

Effect

0.011

0.011

0.013

0.016

0.016

0.015

MSE

-0.002

-0.002

0.004

0.020

0.020

-0.005

Average Bias

34.8 %

33.6 %

100.0 %

93.6 %

10.8 %

72.5 %

Coverage

2323

0.0 0.5 1.0 1.5 2.0








TruePopulation

Effect

0.055

0.011

0.162

0.175

0.175

0.014

MSE

0.211

0.002

0.388

0.402

0.401

0.000

Average Bias

5.3 %

38.5 %

95.4 %

9.3 %

0.0 %

73.3 %

Coverage

2424

0.0 0.5 1.0 1.5 2.0








TruePopulation

Effect

0.011

0.011

0.045

0.175

0.175

0.014

MSE

0.000

0.001

0.169

0.401

0.401

0.001

Average Bias

35.7 %

36.0 %

100.0 %

10.5 %

0.1 %

73.4 %

Coverage

2525

0.0 0.5 1.0 1.5 2.0








TruePopulation

Effect

0.020

0.011

0.083

0.016

0.016

0.016

MSE

0.092

0.000

0.258

0.026

0.025

-0.025

Average Bias

27.7 %

42.1 %

100.0 %

93.8 %

9.6 %

72.5 %

Coverage

2626

0.0 0.5 1.0 1.5 2.0








TruePopulation

Effect

0.058

0.012

0.021

0.176

0.176

0.017

MSE

-0.215

-0.002

-0.008

0.400

0.400

0.036

Average Bias

6.4 %

37.2 %

100.0 %

11.0 %

0.0 %

68.7 %

Coverage

2727



Validity)Validity)





nn


nn

Linear Bias ModelLinear Bias Model

2828

SummarySummary

Even RCTs that have no heterogeneity may Even RCTs that have no heterogeneity may not be estimating the effect of interest.not be estimating the effect of interest.

Observational data may be used to assess Observational data may be used to assess the extent of the generalizability problemthe extent of the generalizability problem

The Cross Design Synthesis approach can The Cross Design Synthesis approach can potentially be effective for estimating potentially be effective for estimating average effect sizeaverage effect size

Still at the beginning of this workStill at the beginning of this work– More fair comparisons More fair comparisons – Extend to real settingsExtend to real settings

Date post:	27-Mar-2015
Category:	Documents
Upload:	benjamin-reeves
View:	216 times
Download:	2 times

1 Eloise E. Kaizar The Ohio State University Combining Information From Randomized and Observational...

Documents