Getting SMART about Adaptive Interventions: Applications in...

Getting SMART about Adaptive Interventions:Applications in Autism and Implementation Science

Daniel Almirall and FriendsSurvey Research Center

Institute for Social ResearchUniversity of Michigan

24 February 2019

University of MinnesotaInstitute for Translational Research

In Children’s Mental Health

UMN ITR Adaptive Interventions February 2019 1 / 119

CollaboratorsMethods/Statistics Collaborators

Inbal Nahum-Shani, Mich

Susan A. Murphy, Harvard

Linda M. Collins, Penn State

Daniel F. McCaffrey, ETS

Behavioral Science Collaborators

Connie Kasari, UCLA

Amy Kilbourne, Mich

Megan Patrick, UMN/ITR

Meredith G-S, UMN/ITR

Current Statistics Students

Nick Seewald, PhD

Brook Luers, PhD

Tim NeCamp, PhD

Olivia Hackworth, PhD

Madison Stoms, Ugrad

Sophia Luo, Ugrad

Previous Statistics Students

Xi Lu, Penn State, Google

Robert Yuen, Liberty Mutual

Kelly Hall, Princeton

Josh Kim, Applied Grad School


Outline (Don’t worry, I don’t really have 100+ slides!)

Adaptive Interventions

Multi-phase Optimization STrategy (MOST) Framework

Sequential Multiple Assignment Randomized Trials (SMART)

Case Studies

I SMART: Minimally-verbal Children with AutismI Pilot SMART: Social Engagement in Autism-inclusive SchoolsI Clustered SMART: Adaptive Implementation of CBT in Schools

Seemingly-Restricted SMART Designs


Outline (Don’t worry, I don’t really have 100+ slides!)

Adaptive Interventions

Multi-phase Optimization STrategy (MOST) Framework


Case Studies

I SMART: Minimally-verbal Children with AutismI Pilot SMART: Social Engagement in Autism-inclusive SchoolsI Clustered SMART: Adaptive Implementation of CBT in Schools

Seemingly-Restricted SMART Designs


Intervention often entails a sequential, individualized approachwhereby treatment is adapted and re-adapted over time...

...Adaptive Interventions provide a guide for intervening this way


Adaptive Intervention (Dynamic Treatment Regimen)

In words

A pre-specified sequence of decision-rules used to guide whether, how, orwhen—and based on which measures—to alter an intervention (e.g.,

monitoring schedule, treatment type, duration, frequency or amount) atcritical decision points during the course of education/care.

In symbols

Given knowledge about a student’s changing status over time{S1 = s1, a1, S2(a1) = s2, a2, . . . ,ST (aT−1) = sT , aT}

a T -stage adaptive intervention is{d1(s1)→ a1, d2(s1, a1, s2)→ a2, . . . , dT (aT−1, sT )→ aT}.


Adaptive Intervention (Dynamic Treatment Regimen)

In words

A pre-specified sequence of decision-rules used to guide whether, how, orwhen—and based on which measures—to alter an intervention (e.g.,

monitoring schedule, treatment type, duration, frequency or amount) atcritical decision points during the course of education/care.

In symbols

Given knowledge about a student’s changing status over time{S1 = s1, a1, S2(a1) = s2, a2, . . . ,ST (aT−1) = sT , aT}

a T -stage adaptive intervention is{d1(s1)→ a1, d2(s1, a1, s2)→ a2, . . . , dT (aT−1, sT )→ aT}.


An Example Adaptive Intervention


Example of an Adaptive Intervention in AutismSome Background First...

≥50% of children with autism who received interventions beginningat age 2 remained non-verbal at age 9

Failure to develop spoken language by age 5 = poor prognosis

Evidence Base:I Discrete Trials Training,I Joint Attention, Symbolic Play, Engagement & Regulation (JASPER)I Enhanced Milieu Teaching (EMT)

Promising: Augmentative, Alternative Communication (AAC) devices

But AAC’s are costly & not all children need it.I Research is limited. Mostly single-subject studies. No rigorous trials.

Motivation for an adaptive intervention involving AAC’s in contextof JASPER-EMT among older, minimally-verbal children with

autism.







But AAC’s are costly & not all children need it.

I Research is limited. Mostly single-subject studies. No rigorous trials.


autism.







But AAC’s are costly & not all children need it.I Research is limited. Mostly single-subject studies. No rigorous trials.


autism.


Example of an Adaptive Intervention in AutismFor minimally verbal children with autism spectrum disorder

Stage One JASP+EMT for 12 weeks;Stage Two At the end of week 12, determine early sign of response:

I IF slow responder: Augment JASP+EMT with AAC for 12 weeks;I ELSE IF responder: Maintain JASP+EMT for 12 weeks.


Example of an Adaptive Intervention in AutismFor minimally verbal children with autism spectrum disorder

Stage One JASP+EMT for 12 weeks;Stage Two At the end of week 12, determine early sign of response:

I IF slow responder: Augment JASP+EMT with AAC for 12 weeks;I ELSE IF responder: Maintain JASP+EMT for 12 weeks.


How was response/slow-response defined?

Percent change from baseline to week 12 was calculated for 7variables:

7 variables: socially communicative utterances (SCU), percent SCU,mean length utterance, total word roots, words per minute, totalcomments, unique word combinations

Fast Responder: if ≥25% change on 7 measures;

Slower Responder: otherwise (this includes kids with no improvement,which is rare)


Three Directions for Your Adaptive Interventions Research

1. Need to prepare or address feasibility/acceptability?

2. Learn how best to build/optimize an adaptive intervention?

3. Ready to evaluate an adaptive intervention?


Multi-phase Optimization STrategy (MOST) FrameworkCollins (2018); Collins & Kugler (2018); Almirall, et al. (2018)



1. Prepare? Feasibility/acceptability?⇒ e.g., pilot study, observational study, pilot SMART



2. Build/optimize?⇒ e.g., Enhanced non-responder trial, SMARTs, MRT



3. Evaluate?e.g., standard RCT, non-inferiority trial


Example of a Confirmatory RCT of an AdaptiveIntervention

MOST Framework (Collins et al 2011, Ann Beh Med)

e.g., Pilot SMART,Obs Studies

e.g., SMART

e.g., Standard RCTnot our focus


Typical Scientific Questions when Building/Optimizing anAdaptive Intervention.

Is it better to provide AAC from the start?

Who benefits from initial AAC versus who benefits from delayed AAC?

For slow responders, what is the effect of providing the AAC vsintensifying JASP (not providing AAC)?

Why begin treatment with JASP? Why not a high-quality course ofcore-principles of discrete trials training?

Should we train parents following an initial successful course oftreatment? If so, which families benefit most from this?


Insufficient Empirical Evidence or Theories to Address suchQuestions

In the past we have relied on

Expert opinion

Clinical expertise

Piecing together an adaptive intevrention using results from separateRCTs




What is a Sequential Multiple Assignment RandomizedTrial (SMART)?

A type of multi-stage, randomized trial design—a factorial design.

At each stage, subjects randomized to a set of feasible/ethicaltreatment options.

Treatment options at latter stages may be restricted by response toearlier treatments.

SMARTs were developed explicitly for the purpose of building ahigh-quality Adaptive Intervention.

SMARTs are not (typically) used for confirming the effectiveness ofan AI (evaluation).


What is a Sequential Multiple Assignment RandomizedTrial (SMART)?

A type of multi-stage, randomized trial design—a factorial design.

At each stage, subjects randomized to a set of feasible/ethicaltreatment options.

Treatment options at latter stages may be restricted by response toearlier treatments.

SMARTs were developed explicitly for the purpose of building ahigh-quality Adaptive Intervention.

SMARTs are not (typically) used for confirming the effectiveness ofan AI (evaluation).


Prototypical SMART

SMART Case Study:

Characterizing Cognition in Non-verbal Individuals with ASD (CCNIA)


Example of a (first-ever) SMART in Autism ResearchPI: Kasari (UCLA)

The population of interest:

Children with autism spectrum disorder

Age: 5-8

Minimally verbal: <20 spontaneous words in a 20-min. language test

History of treatment: ≥2 years of prior intervention

Functioning: ≥2 year-old on non-verbal intelligence tests


Example of a SMART in Autism Research (N = 61)PI: Kasari (UCLA)


SMARTs permit scientists to answer many interestingquestions for building a high-quality adaptive intervention.


A Common Primary Aim in a SMART is theComparison of “Embedded” Adaptive Interventions

Over the past 3 years, a good portion of my time has been devotedto developing methods to do this with a repeated-measuresoutcome arisinh from a SMART.


Three AIs “Embedded” in this Example Autism SMART

(JASP,JASP+)

(JASP,AAC)

(AAC,AAC+)


Three AIs “Embedded” in this Example Autism SMART

(JASP,JASP+)

(JASP,AAC)

(AAC,AAC+)


A Common Primary Aim in a SMART is theComparison of these Embedded Adaptive Interventions

Methodological challenges include

Modeling Considerations: The intermixing of repeated measures andsequential randomizations required new modeling considerations

Effect Interpretation: Traditional comparison of slopes may no longerbe appropriate; need for a clinically relevant, easy-to-understandsummary measure of the mean trajectories

Statistical Efficiency: Permit domain scientists to take advantage ofthe within-person correlation in the outcome in a way they areaccustomed to

Random Effects: Psuedo-likelihood with a rather non-standard form


Example of a SMART in Autism Research (N = 61)PI: Kasari (UCLA)


Longitudinal Outcomes in this Example SMART in Autism

Outcomes collected at baseline, and weeks 12, 24 and 36

Primary outcome (verbal): from 20-minute “natural language sample”:

Total spontaneous communicative utterances (TSCU)

Secondary outcome (non-linguistic):

Initiating joint attention (IJA; e.g., pointing; JASP mechanism)


Results

Adaptive (a) TSCU (b) IJAIntervention AUC 95% CI AUC 95%CI

(AAC,AAC+) 51.7 [43, 60] 9.5 [7.2,11.8](JASP,AAC) 36.0 [28, 44] 7.2 [5.6,8.8]

(JASP,JASP+) 33.1 [25, 42] 6.6 [5,8.2]No diff null p < 0.01 p < 0.05


You may be wondering about sample size/power?

Prototypical SMART

Sample Size Formulae with Repeated-measures AnalysesFor comparing 2 embedded AIs that begin with different treatments in a prototypicalSMART; Hypothesis test is that there is no difference in means at the end of the study

N ≥ 4(z1−α/2+z1−β)2

δ2 × (2− r)×(1− ρ2

)where

r = response rate after stage 1 treatment

ρ = within-person correlation in outcome

r = ρ = δ = 1/2, α = .05, need N = 142 for 80% power.Same question using a standard 2-arm trial requires N = 96.


Sample Size Formulae with Repeated-measures AnalysesFor comparing 2 embedded AIs that begin with different treatments in a prototypicalSMART; Hypothesis test is that there is no difference in means at the end of the study

N ≥ 4(z1−α/2+z1−β)2

δ2 × (2− r)×(1− ρ2

)where


ρ = within-person correlation in outcome

r = ρ = δ = 1/2, α = .05, need N = 142 for 80% power.Same question using a standard 2-arm trial requires N = 96.


References

Lu, Nahum-Shani, Kasari, Lynch, Oslin, Pelham, Fabiano, Almirall. (2016).Comparing DTRs using repeated-measures outcomes: modelingconsiderations in SMART studies, Statistics in Medicine.

Almirall, DiStefano, Chang, Shire, Lu, Nahum-Shani, Kasari, C. (2016).Adaptive interventions and longitudinal outcomes in minimally verbalchildren with ASD: Role of speech-generating devices, Journal of ClinicalChild and Adolescent Psychology.

Seewald, Nahum-Shani, McKay, Almirall (about to be submitted). Samplesize considerations for comparing DTRs in a sequentially-randomized trialwith a continuous longitudinal outcome. about to be submitted

Luers, Qian, Nahum-Shani, Almirall (in progress). LongitudinalMixed-effects Models to compare Dynamic Treatment Regimens inSequentially-randomized Trials to be submitted in April

NeCamp, Kilbourne, Almirall (2017). Cluster-level adaptive interventionsand sequential, multiple assignment, randomized trials: Estimation andsample size considerations, Statistical Methods in Medical Research.


Two More SMART Case Studies


SMART Case Study:

Getting SMART about Social and Academic Engagement in ElementaryAged Children with ASD


Social & Academic Engagement in School-Children w/ASDBackground

Targeting students with ASD, ages 5-12, ≥ 70 IQ

In inclusive schools/classrooms in the LAUSD

Improve social skills, behavioral, and academic engagement outcomes

Early on, social skills interventions rarely studied in the child’s naturalenvironment (befuddles me!); so had no “generalization”

Others that were in the child’s natural environment, such as 1:1pairing with an aide at the school, turned out to have iatrogeniceffects!

Social & Academic Engagement in School-Children w/ASDSet-up and Questions

Evidence Base: Remaking Recess (school-level), Classroom Supports(class-level), Parent-assisted social skills intervention (individual-levelat home), Peer-mediated social skills intervention (individual-level)

But no empirically derived guidelines to help guide interventiondecision making of children with ASD in schools.

Combined Peer + Parent is also promising, but this cannot beprovided to all children due to cost and not all children need it

Begin individual level interventions with Peer or with Parent?

Academic & Social Engagement in School-Children w/ASDPIs: Kasari; Co-I: Almirall; IES-funded Pilot SMART

This is a 2-arm trial. But my mad scientist friend had some other scientificquestions, leading to a different design...

Social & Academic Engagement in School-Children w/ASDSet-up and Questions Motivating the SMART

Evidence Base: Remaking Recess (school-level), ClassroomSupports (class-level), Parent-assisted social skills intervention(individual-level at home), Peer-mediated social skills intervention(individual-level)

But no empirically derived guidelines to help guide interventiondecision making of children with ASD in schools.

Begin individual level interventions with Peer or with Parent?

Do all classrooms require Classroom Supports?

Synergistic effect between CS and individual-level interv.?

Academic & Social Engagement in School-Children w/ASDPIs: Kasari; Co-I: Almirall; IES-funded Pilot SMART

Primary and Secondary Aims of this Pilot SMART

Primary Aim: Address feasibility and acceptability concerns related tothe embedded adaptive interventions

I identifying children as early vs. slower responders by theparaprofessionals in the context of RR,

I transitioning children to Parent or Peer at wk8,

I providing augmented Peer+Parent to slower responders

I not providing augmented treatment to early responders at wk20

I satisfaction with txt sequences by children, parents, teachers,paraprofessionals & school champions

I teacher-rated measures of progress during CS for deciding Parent vsPeer

Secondary Aim: To obtain preliminary data to support a full-scaleSMART.


References

Almirall, Kasari, McCaffrey, Nahum-Shani. (2018). Developing OptimizedAdaptive Interventions in Education, Journal of Research on EducationalEffectiveness.

Almirall, Nahum-Shani, Wang, Kasari. (2018). Experimental Designs forResearch on Adaptive Interventions: Singly- and Sequentially-RandomizedTrials, Optimization of Multicomponent Behavioral Biobehavioral andBiomedical Interventions using MOST. L. Collins, K. Kugler (Editors).

Almirall, Compton, Gunlicks-Stoessel, Duan, Murphy (2012). Designing aPilot Sequential Multiple Assignment Randomized Trial for Developing anAdaptive Treatment Strategy.” Statistics in Medicine

Kim, H. and Almirall (2016). A sample size calculator for SMART pilotstudies, SIAM Undergraduate Research Journal, Vol. 9.

I Web-applet: https://methodologycenter.shinyapps.io/PilotShiny/


Clustered SMART Case Study:

Adaptive Implementation for School-based CBT


Adaptive Implementation Intervention in Mental HealthPI: Kilbourne

Sample Size Formulae for Cluster-Randomized SMARTsFor comparing 2 embedded AIs that begin with different treatments in a prototypicalSMART

N ≥ 4(z1−α/2+z1−β)2

m∗δ2 × (2− r)× (1 + (m − 1)ρ)

where


m = avg number of SPs at each school

ρ = inter-school correlation in outcome

r = δ = 1/2, ρ = .03,m = 20, α = .05, need N = 100 for 80%power.Same question using standard 2-arm cluster trial requires N = 66.


Sample Size Formulae for Cluster-Randomized SMARTsFor comparing 2 embedded AIs that begin with different treatments in a prototypicalSMART

N ≥ 4(z1−α/2+z1−β)2

m∗δ2 × (2− r)× (1 + (m − 1)ρ)

where


m = avg number of SPs at each school

ρ = inter-school correlation in outcome

r = δ = 1/2, ρ = .03,m = 20, α = .05, need N = 100 for 80%power.Same question using standard 2-arm cluster trial requires N = 66.


References

Kilbourne, Smith ... Almirall (2018). Adaptive School-based Implementationof CBT (ASIC): clustered-SMART for building an optimized adaptiveimplementation intervention to improve uptake of mental healthinterventions in schools., Implementation Science.

NeCamp, Kilbourne, Almirall (2017). Cluster-level adaptive interventionsand sequential, multiple assignment, randomized trials: Estimation andsample size considerations, Statistical Methods in Medical Research.

Kilbourne, Almirall, et al. (2014). Adaptive Implementation of EffectivePrograms Trial (ADEPT): cluster randomized SMART trial comparing astandard versus enhanced implementation strategy to improve outcomes of amood disorders program, Implementation Science.


Myths or Misconceptions about Adaptive Interventions andSMARTs


Myths or Misconceptions about Adaptive Interventions

Tailoring variables cannot differ based on previous intervention

An adaptive intervention must recommend a single interventioncomponent at each decision point

Adaptive interventions seek to replace clinical judgement

Adaptive interventions are only relevant in treatment settings

Adaptive interventions are non-standard because they involverandomization


Interventions for Minimally Verbal Children with AutismPIs: Kasari(UCLA), Almirall(Mich), Kaiser(Vanderbilt), Smith(Rochester), Lord(Cornell)

Myths or Misconceptions about SMART Studies

All research on adaptive interventions requires a SMART

SMARTs are an alternative to the RCT

SMARTs require prohibitively large sample sizes

All SMARTs require Multiple-Comparisons Adjustments

All SMARTs must include an embedded tailoring variable

All aspects of an embedded adaptive intervention must be randomized

SMARTs are a form of adaptive research design

SMARTs never include a control group

SMARTs require a “business as usual” control group

SMARTs require multiple consents

SMARTs are susceptible to high levels of study drop-out


Interventions for Minimally Verbal Children with AutismPIs: Kasari(UCLA), Almirall(Mich), Kaiser(Vanderbilt), Smith(Rochester), Lord(Cornell)

Myths or Misconceptions about SMART Studies

All research on adaptive interventions requires a SMART

SMARTs are an alternative to the RCT

SMARTs require prohibitively large sample sizes

All SMARTs require Multiple-Comparisons Adjustments

All SMARTs must include an embedded tailoring variable

All aspects of an embedded adaptive intervention must be randomized

SMARTs are a form of adaptive research design

SMARTs never include a control group

SMARTs require a “business as usual” control group

SMARTs require multiple consents

SMARTs are susceptible to high levels of study drop-out


Multi-Tiered Systems of SupportPIs: Greg Roberts and Nathan Clemens (UT Austin)


Let’s call this ”Academic Adaptive Intervention.”


This study investigates the role of the self-regulation component (should itbe provided in stage 1, in stage 2, or at all?) in the context of theAcademic Adaptive Intervention.


I call this a “Seemingly-Restricted SMART”. Here, a 2x2 SMART design.

What did you learn today?

You learned about Adaptive Interventions (in our literature,statisticians call these Dynamic Treatment Regimens)

You learned about some study designs for piloting and optimizingAdaptive Interventions

You learned about 5 SMART case studies (1 pilot, 3 standardSMART, 1 clustered SMART)

You learned about the sample size formulae for the repeated-measuresand clustered outcomes analyses

You learned about common misconceptions about AdaptiveInterventions and SMART

Thank you! [email protected], http://www-personal.umich.edu/∼dalmiral/

Apply for next year’s IES training! http://d3lab-isr.com/training


Date post:	16-Oct-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

Getting SMART about Adaptive Interventions: Applications in...

Documents