How do treatments work and who for? Efficacy and mechanisms evaluation in
complex interventions and personalised medicine
Dr Richard Emsley
Centre for Biostatistics, Institute of Population Health,
The University of Manchester
www.medicine.manchester.ac.uk/staff/RichardEmsley
Presentation at Symposium on Causal Mediation Analysis, Gent, Belgium
Tuesday 29th January 2013
Manchester, England
Research Programme: Efficacy and Mechanisms Evaluation
Joint work with Graham Dunn, Hanhua Liu, Andrew Pickles, Sabine Landau and Ian White.
Funded by MRC Methodology Research Programme grants:
• Estimation of causal effects of complex interventions in longitudinal studies with intermediate variables (2009-2012)
Richard Emsley (MRC Fellow), Graham Dunn.
• MRC Early Career Centenary Award (2012-13)
• Designs and analysis for the evaluation and validation of social and psychological markers in randomised trials of complex interventions in mental health (2010-12)
Graham Dunn (PI), Richard Emsley, Linda Davies, Jonathan Green, Andrew Pickles, Chris Roberts, Ian White & Frank Windmeijer with Hanhua Liu.
• Developing methods for understanding mechanism in complex interventions (2013-15)
Sabine Landau (PI), Richard Emsley, Andrew Pickles, Graham Dunn, Ian White, Paul Clarke
Randomised trials: efficacy and mechanisms evaluation in mental health
Recent and Ongoing Randomised Trials (Manchester and elsewhere)
• MIDAS; Cognitive behaviour therapy (CBT) for dual diagnosis of substance abuse and psychosis (n=327)
• PRP; CBT for psychosis relapse prevention (n=296)
• SoCRATES; CBT/supportive listening for relapse in early psychosis (n=309)
• FINE; interventions for Chronic Fatigue Syndrome/ME (n=296)
• CuTLASS; Antipsychotics in schizophrenia (n=345)
• ASSIST; group therapy for self-harm in adolescents (n=370)
• PACT; parent treatment for autism in infants (n=152)
• ADAPT; CBT and antidepressant medication in adolescent depression (n=208)
• SMaRT; CBT for depression in cancer patients (n=200)
• ODIN; cognitive intervention for depression in primary care (n=427)
• CiSH; self-harm/attempted suicide in emergency departments (n=100)
Contents
1. How do treatments work?
Complex interventions in mental health
Distinguishing different types of mediators and questions
Mediation methods: principal stratification and instrumental variables
2. Who do they work for?
Personalised medicine: a causal inference perspective
Current methods
Future methods: the role of mediation analysis
3. Conclusions
MRC Complex Intervention Guidelines 2008
• What makes a complex intervention? – MRC Guidelines 2008
“Number of interacting components within the experimental and control interventions
Number and difficulty of behaviours required by those delivering or receiving the intervention
Number of groups or organisational levels targeted by the intervention
Number and variability of outcome
Degree of flexibility or tailoring of the intervention permitted.”
• Questions for complex intervention trials:
Does it work?
How does it work? What are the active ingredients?
What components of the intervention are responsible for efficacy?
Can it be made to work more effectively with particular patients?
MRC Guidelines on Complex Interventions, 2008; Craig et al, BMJ (2008).
Psychotherapy as complex interventions
• They often have a large collection of multivariate outcomes, usually scales of measurement (e.g. measuring symptoms, quality of life, side effects, therapeutic alliance, …), and interacting components delivered by different groups of people (therapists, support workers, doctors, etc.).
• Mediation for explanation
• Mediation by design
• Cognitive model for positive symptoms in psychosis (Garety et al. (2000); Fowler et al. (1995)).
• Rather than compress all the treatment information into a binary variable we measure the various underlying process measures of the therapy and test the underlying psychological theory.
• Do people jump to conclusions? (PRP trial) Does psychotherapy reduce jumping to conclusions which improves positive symptoms in psychosis?
• What is the concomitant substance abuse? (MIDAS trial) Does psychotherapy reduce cannabis use, which in turn leads to
improvements in psychotic symptoms?
• What are the participant’s beliefs? (FINE trial)
Does psychotherapy change attributions (beliefs), which, in turn, lead to better outcome? How much of the treatment effect is explained by changes in attributions?
Intermediate variables in psychological trials: “True” Mediators/process variables
Rand.
Process Measure
Outcome
Do we need a treatment effect to perform mediation analysis?
• Do people jump to conclusions? (PRP trial) Does psychotherapy reduce jumping to conclusions which improves positive symptoms in psychosis? No overall ITT effect on relapse (Garety et al., 2008) But no effect of intervention on jumping to conclusions Theory fine, need new interventions…
• What is the concomitant substance abuse? (MIDAS trial) Does psychotherapy reduce cannabis use, which in turn leads to
improvements in psychotic symptoms?
No overall ITT effect on outcomes (Barrowclough et al., 2010)
But there was an ITT effect on amount of substance used
No link between amount of cannabis and psychotic symptoms (Barrowclough et al., 2012)
Intermediate variables in psychological trials: “Nuisance” mediators
• Variables measured post-randomisation that we may wish to rule out having a mediated effect - essentially we want to estimate the residual direct effects and find a small indirect effect.
• Use of concomitant medication (SMaRT; PROSPECT trials)
Does psychotherapy improve compliance with medication which, in turn, leads to better outcome? What is the direct effect of psychotherapy?
a b
c’
Random allocation
to CBT
Antidepressant Use
Depression Score
• What makes these variables ‘mediators’?
We are interested in all three pathways in the diagram, and the effect decomposition:
• Requirements for mediation:
1. Aim is to estimate the size of the indirect effect, and
2. The mediator is measured in both arms.
Intermediate variables in psychological trials: “True” or “Nuisance” Mediators
a b
c'
Random allocation to CBT
Antidepressant Use
Depression Score
Suitable methods for ‘true’ and ‘incidental’ mediators
• Controlled direct effects are a natural question for ‘nuisance’ mediators:
What is the direct effect of randomisation on outcome if everyone in the population has the value M=m?
Interaction between X and M on Y might be important.
• For ‘true’ mediators, depends on question/subject matter:
Confounding; non-linearity; interaction; etc.
• Some other approaches for causal mediation analysis in software;
Stata: paramed, mediation (Imai et al.), gformula (Daniel et al.)
SAS: gformula, mediation.
Lange et al. (2012) and Vansteelandt et al. (2012).
Intermediate variables in psychological trials: Post-randomisation effect modifiers
• Aspects involved in process of therapy that might explain differential treatment effects/effect heterogeneity.
• Compliance with allocated treatment
Does the participant turn up for any therapy?
How many sessions does she attend?
• Quality of the therapeutic relationship
What is the strength of the therapeutic alliance?
• Fidelity of therapy How close is the therapy to that described in the treatment
manual? Is it a cognitive-behavioural intervention, for example, or merely emotional support?
• Why do we say these aren’t true or nuisance mediators?
Generally interested in some other causal question, such as how do they account for heterogeneity? Are they effect modifiers?
Intermediate variables in psychological trials: Post-randomisation effect modifiers
Random allocation
Sessions
Outcomes
U
U – unmeasured confounders
a b
c’=0 error
error
• It is plausible that these may only be measured in the therapy arm of a randomised trial. For example, if the control arm has some form of treatment as usual which doesn’t contain an active ‘therapy’ on which they can be measured.
Process variables
Treatment group
Patient engagement
in therapy
OUTCOME
Therapeutic alliance
Therapeutic empathy
Control group (TAU)
OUTCOME
Number of sessions
Treatment effects on outcome
• Consider a randomised controlled trial with two arms: treatment (X=1) versus control (X=0) and a continuous outcome Y
• Prior to randomisation to one of two competing treatment arms we can envisage two potential outcomes for each participant in the trial:
the outcome after receiving treatment, YX=1=Y1
the outcome after receiving the control, YX=0=Y0
• For a given individual, the effect of treatment is the difference:
ITE(Y)=Y1 –Y0
• The average treatment effect ATE(Y) is:
E[ITE(Y)] = E[Y1 –Y0]=E[Y|X=1] – E[Y|X=0]
• Quality of therapeutic relationship in psychotherapy
What is the strength of the therapeutic alliance? Is there an empathic relationship?
• Does the individual treatment effect, Y1 –Y0, increase in magnitude with increasing alliance?
• More realistically, in a population of clients given psychotherapy, is the average treatment effect (the average of the above differences) related to (correlated with) the strength of the therapeutic alliance?
Example: Does the therapeutic alliance influence the treatment effect?
• Ignore the control group (if there is one) and anyone else who has not received treatment.
• Ask if there is a correlation between alliance score and outcome.
• Infer that this correlation (if found) tells us something reliable about the relationship between the strength of the therapeutic alliance and the effect of therapy.
• The problem arises from ambiguity of ‘treatment outcome’. They are looking at the correlation between alliance score and Y1.
• A client with a good prognosis is likely to be the one who is capable of developing a strong therapeutic alliance. That is, Y0 and alliance are likely to be correlated. The relationship between Y1 (i.e. outcome) and alliance is therefore confounded.
What question do people usually attempt to answer?
Why not predict treatment-outcome?
• Given an additive treatment effect, the outcome of treatment is:
Y1 =Y0 + ITE(Y)
(i.e. their treatment-free outcome plus the effect of treatment)
• Now let's introduce a baseline marker, C.
• Correlate C with treatment outcome Y1:
Corr(C,Y1)=Corr(C,Y0 + ITE(Y))
• A correlation can arise from two sources:
Y0 is correlated with C (prognosis), or
ITE(Y) is correlated with C (prediction)
• If C is prognostic then you can get a correlation between Y1 and C even when the ITE(Y) is ZERO for everyone in the study.
Prognostic variables (risk factors)
Outcome
Prognostic variable
Treated
Untreated
Treatment effect
Predictive variable
Outcome
Predictive variable
Treated Treatment effect depends on
predictive variable
Untreated
An alternative approach for post-randomisation effect-modifiers
• When the mediator (e.g. therapeutic alliance or treatment fidelity) is not observed in the control arm, we can also use principal stratification to estimate a principal stratum direct effect
PSDE = E[Y1 –Y0 |M(1)=m]
• Work by Booil Jo and Elizabeth Stuart, including use of propensity scores, to predict M(1) when X=0.
• Use a finite mixture model in Mplus or can be fitted with interaction between M(1) and X with gllamm in Stata.
Principal strata – therapeutic alliance example
Treatment group
Control group
High alliance class
Low alliance class
?? ??
Treatment group
Control group
High alliance class
Low alliance class
High alliance class
Low alliance class
Class specific
ITT effects
Principal strata – therapeutic alliance example
25
Example: SoCRATES - results
Estimated ITT effect on 18 month PANSS scores (high is worse)
Low alliance High alliance
Missing data ignorable (MAR) +7.50 (8.18) -15.46 (4.60)
Missing data ignorable (MAR) 0 (*) -12.73 (4.75)
Missing data latently ignorable (LI) +6.49 (7.26) -16.97 (5.95)
Missing data latently ignorable (LI) 0 (**) -13.50 (5.31)
* Zero ITT constraint in low alliance group (exclusion restriction)
** Compound exclusion restriction i.e. no ITT effect on PANSS or probability of missing value
Emsley, Dunn and White, Stats Methods in Medial Research (2010).
Example - PRP Trial: Was treatment as intended?
• We consider patient engagement in therapy procedures as a potential treatment-effect moderator post randomisation effect modifier.
• Determined by careful examination of recordings of therapy
sessions using the Cognitive Therapy for Psychosis Adherence Scale (CTPAS) and Cognitive Therapy Scales (CTS).
• CTPAS/CTS classification for observed data in the CBT arm:
No dose (21) Medium dose (39) Full dose (42)
• Dose not defined (missing) in TAU arm.
• A few dose assessments missing in CBT arm.
Dunn et al. (2012). The effective elements of CBT for psychosis. Psychological Medicine.
PRP Trial: PANSS outcome at 12m & 24m
PANSS 0m PANSS 12m PANSS 24m
No dose 63.0 56.4 52.3 Medium dose 66.2 60.1 58.9 Full dose 63.5 56.0 56.3 Controls 65.0 58.5 58.5 A high PANSS score implies a worse symptom outcome. This is still not very promising – high dose worse than no
dose! * Note that only about 50% of the No dose group provide outcome
data.
Dunn et al. (2012). The effective elements of CBT for psychosis. Psychological Medicine.
PRP Trial: Principal stratification
Defined as before in terms of potential response to randomisation • Statum 1: a group of participants who receive little or no therapy
whatever their treatment allocation.
• Stratum 2: a group of participants who would receive no therapy if allocated to the control condition but a medium dose of CBT if allocated to the treatment group.
• Stratum 3: a group who would receive no therapy if allocated to the control condition but a full dose of CBT if allocated to the treatment group.
• Membership of one of these three classes (the Principal Strata) is directly observable in the CBT arm but remains latent (hidden) under TAU.
• Principal stratum membership is independent of treatment allocation.
• Potentially, we can stratify by stratum membership and evaluate the ITT effects of treatment allocation within these strata.
Dunn et al. (2012). The effective elements of CBT for psychosis. Psychological Medicine.
PRP Trial: marginal ITT estimates over 12m and 24m (Mplus v6.1) Joint analysis of PANSS 12m & 24m to get common ITT estimates.
No Medium Full
Estimate 0* +5.2 (3.2) -12.7 (4.1)
Estimate 0* 0* -12.2 (4.8)
* Exclusion restriction (constraint)
With bootstrap standard errors
This assumes a Missing at Random missing data mechanism.
Dunn et al. (2012). The effective elements of CBT for psychosis. Psychological Medicine.
Solutions to unmeasured confounding in mediation analysis
• We’ve proposed three solutions to analyse mediation allowing for unmeasured confounding:
1. Measure and adjust for potential confounders (sounds obvious, not always done);
2. Instrumental variables;
3. Principal stratification.
Explained in detail in:
• Emsley RA & Dunn G. (2012) Evaluation of potential mediators in randomized trials of complex interventions (psychotherapies). In: Causal Inference: Statistical perspectives and applications. Eds: Berzuini C, Dawid P & Bernardinelli, L. Wiley.
• Emsley RA, Dunn G & White IR. (2010). Modelling mediation and moderation of treatment effects in randomised controlled trials of complex interventions. Statistical Methods in Medical Research, 19(3), 237-270.
• An instrumental variable is:
1. (Strongly) predictive of the mediating variable;
2. Has no direct effect on the outcome, except through the intermediate;
3. Does not share common causes with the outcome.
• Randomisation, where available, often satisfies this criteria when accounting for departures from randomised treatment.
• Long history: Sewell-Wright, econometrics, “Correlation and Causality” by David Kenny (1979).
Instrumental variables
Mediation diagram with instrumental variables
Mediator
Outcomes
error
error
U
Covariates
Instruments
Random allocation
Multiple instruments
• When we are trying to estimate the direct effect of randomisation we need alternative instruments.
• Likewise, if we have more than one endogenous variable (multiple mediators), then we need multiple instruments.
• For IV model identification, we always need to have as many instruments as we have endogenous variables.
i.e. if considering two mediators in the model then we need at least two instrumental variables.
• If we consider this at the design stage of the complex intervention trial, we can measure variables that MIGHT meet these requirements.
Where do we find instruments in trials?
• Possibilities for instrumental variables:
1. Augmented designs (“innocuous vaccine”).
2. Joint analysis of several similar trials.
3. Use of a multi-centre trial.
4. Measure baseline variables which predict variation in the mediators but not the outcome.
5. Use of gene as an instrument (Mendelian randomisation).
6. Randomize to multiple treatments.
Interactions with baseline covariates and randomisation?
• In the analysis, a lot of these reduce to using randomisation by baseline interactions as instruments.
Gennetian LA, Morris PA, Bos JM, Bloom HS. Constructing instrumental variables from experimental data to explore how treatments produce effects. In Bloom HS, ed. Learning more from social experiments: evolving analytic approaches (1st edn). Russell Sage Foundation, New York, 2005: 75–114.
Dunn and Bentall (2007), Ten Have et al (2007), Emsley et al (2010), Small (2012).
• Key assumption: the vital component of all our models is randomization which ensures that treatment-free outcome is independent of treatment allocation (i.e. X⊥Y0) and therefore,
given baseline covariates, C:
E(Y0|C, X)=E(Y0|C)
Equivalence with other approaches: mediated moderation
C
X*C
X
Y
M
Interaction X*C is mediated by one other variable M Consistent with complete mediation of X*C by M
Figure 12.3 from Klein – Principles and Practice of Structural Equation Modelling
Equivalence with other approaches: causal inference
• Fischer-Lapp and Goetghebeur (1997;1999) describe an algorithm for estimating the parameters of a structural mean model using g-estimation.
• Goetghebeur and Vansteelandt (2005) show how g-estimation consistently estimates mediator effects (compliance) in the presence of hidden confounding and random errors in the mediators.
• In the case of linear models with continuous mediators and outcomes the two methods are equivalent (Dunn and Bentall (2007), Emsley et al. (2010)).
• Ten Have, Joffe and colleagues (2007, 2010) propose a rank-preserving model for binary mediators estimated by g-estimation.
Also equivalent to instrumental variables (Emsley and Dunn 2012).
Instrumental variables: estimation
• An instrumental variable regression in Stata:
ivregress 2sls Y X C (M = X*C), first
• This is a two-stage least-squares procedure which simultaneously estimates:
1. E[M|X,C,X*C]
2. E[Y|X,C, E[M|X,C,X*C] ].
• Other estimation procedures:
FIML
LIML
Jackknife 2SLS
Generalised method of moments…
Mediation with instrumental variables
• Our estimate of c’ is the controlled direct effect, which because of no interaction between randomisation and mediator, is equal to the natural direct effect.
• We try and obtain an unbiased estimate of b, allowing for unmeasured confounding, which we could then use to calculate natural indirect effects
Some issues around which a to use.
• The identifying assumptions that we require are:
1. The effect of X on M is moderated by C;
2. The effect of M on Y is neither moderated by X nor C (moderation by other covariates is still possible).
3. The direct effect of X on Y is not moderated by C (moderation by other covariates is still possible).
Contents
1. How do treatments work?
Complex interventions in mental health
Distinguishing different types of mediators and questions
Mediation methods: principal stratification and instrumental variables
2. Who do they work for?
Personalised medicine: a causal inference perspective
Current methods
Future methods: the role of mediation analysis
3. Conclusions
Personalised medicine and treatment effect heterogeneity
• Treatment effect heterogeneity, whereby a given treatment will be more efficacious for some patients than for others (variation in Y1 –Y0), is the underlying foundation of personalised medicine.
Stratified/predictive/targeted medicine
Genomic medicine
Pharmacogenomics
• If a treatment is effective, we are interested in knowing who is it (most) effective for, in advance of treatment allocation/decisions to treat?
• We need access to pre-treatment characteristics that predict treatment-effect heterogeneity
Not just predict outcome
Why not predict treatment-outcome?
• Similar to a case-control study searching for differential predictors
• Same problem as we saw previously, we are trying to predict only Y1.
Outcome Responder
Outcome Non-responder
Treatment is the same
Prognostic markers
Prognostic markers
Biomarker stratified design
1. Stratify patients according to marker status, and randomize to treatments (T and C) within each marker stratum.
2. Two parallel randomized clinical trials are conducted to compare the treatments within each marker stratum.
3. We assess the predictive value of the marker by formally testing whether the treatment effect is the same in each of the marker strata; that is, we assess the marker–treatment interaction
• These tells us whether the intention-to-treat effect differs in each strata
Treatment effect moderation;
Subgroup analysis.
• Says nothing about WHY there might be differences…
Are the mechanisms different in subgroups of people?
• We need more information to make progress
Genetic and phenotypic markers
Clinical history
Past environmental exposures, lifestyle, etc.
• Advantage of genetic markers is that they are essentially randomised and, in particular, (in a conventional RCT) independent of treatment allocation. And, of course, they are not influenced by treatment.
• Can we use markers (prognostic and predictive markers, e.g. biological or biomarkers, social and psychological markers) as this extra information?
How we do this depends on the assumptions we make about relationships between markers and outcomes.
The BS-EME trial design
Random allocation
Mediator
Outcomes
U
Prognostic biomarker
(risk factor)
Predictive biomarker
(moderator)
U – unmeasured confounders Using the treatment by marker
interaction as an instrument
The BS-EME trial design
X
M
Y
U
C
X*C
Using the treatment by marker interaction as an instrument
The BS-EME trial design
• Taking the biomarker stratified design as described previously we supplement the baseline information by:
measuring all previously-validated prognostic markers
baseline covariates (demographic information; clinical and treatment history; co-morbidity; social, psychological and cultural variables; etc.) thought to have prognostic value.
baseline measurement of the putative mediator.
baseline value for the final outcome measurement.
• The rationale for all of these measurements is
(a) to allow for as much confounding of the effects of the mediator on final outcome as is feasible,
(b) to assess sensitivity of the results to assumptions concerning residual hidden confounding and, perhaps more importantly,
(c) increase the precision of the estimates of the important causal parameters.
BS-EME example: psoriasis
Treatment (Methotrexate)
Drug immunogenicity
Outcome (PASI)
U
Prognostic biomarkers (risk factor)
Genetic marker
BS-EME example: cancer
Treatment
Tumour size
Outcome (survival)
U
Prognostic biomarkers (risk factor)
Genetic marker
Contents
1. How do treatments work?
Complex interventions in mental health
Distinguishing different types of mediators and questions
Mediation methods: principal stratification and instrumental variables
2. Who do they work for?
Personalised medicine: a causal inference perspective
Current methods
Future methods: the role of mediation analysis
3. Conclusions
1. Process measure only available in the treated group (alliance, dose, fidelity) – undefined in controls
2. Measurement error (number of sessions is only a proxy for ‘dose’, scales may only have modest reliability)
3. Subjective measures, possibly leading to…
4. Hidden confounding between mediator and outcome
5. Can’t directly intervene on mediator (beliefs)
6. Treatment is not the same for all patients – treatment irrelevance assumption
7. Clustering by therapists – both therapist effects and interference
8. Missing data – mediators and outcomes.
Solution: better designs?
Some technical challenges to answering causal questions in psychological trials
“Innocuous vaccine” design; Follmann (2006)
Infection rate
Infection rate
HIV vaccine group
Control group
Immune response to vaccine
Immune response to
rabies
Immune response to
rabies
Immune response
Time
Departures from randomised treatment
Mediator
Outcomes
U
Random allocation
Treatment received
Conclusions on stratified medicine
1. Personalised (stratified) medicine and treatment-effect mechanisms evaluation are inextricably linked;
2. Stratification without corresponding mechanisms evaluation lacks credibility;
3. In the almost certain presence of mediator-outcome confounding, mechanisms evaluation is dependent on stratification for its validity;
4. Both stratification and treatment-effect mediation can be evaluated using a biomarker stratified trial design together with detailed baseline measurement of all known prognostic biomarkers and other prognostic covariates;
5. Direct and indirect (mediated) effects should be estimated through the use of instrumental variable methods together with adjustments for all known prognostic biomarkers (confounders) – the latter adjustments contributing to increased precision (as in a conventional analysis of treatment effects) rather than bias reduction.
UK-Causal Inference Meeting https://sites.google.com/site/ukcausalinferencemeeting/
The KEY question for this conference…
• How many citations can we get Baron and Kenny upto?
• As of very early this morning on Web of Science:
Kaplan-Meier curves (1958) – 37,046
Cox Proportion Hazards model (1972) - 26,963
Bland-Altman plot (1986) – 21,141
Baron and Kenny (1986) – 19,745
• Thank you for your attention.
Some references
• Dunn G, Emsley RA, Liu H & Landau S. (2013). Trial designs fully integrating biomarker information for the evaluation of treatment-effect mechanisms in personalised (stratified) medicine. Under review.
• VanderWeele TJ, Emsley RA. (2013). Discussion of “Experimental designs for identifying causal mechanisms”. JRSS-A, 176(1), pp46.
• Emsley RA, Liu H, Dunn G, Valeri L, VanderWeele TJ. (2012). PARAMED: A command to perform causal mediation analysis using parametric models. In preparation for The Stata Journal.
• Emsley RA & Dunn G. (2012) Evaluation of potential mediators in randomized trials of complex interventions (psychotherapies). In: Causal Inference: Statistical perspectives and applications. Eds: Berzuini C, Dawid P & Bernardinelli, L. Wiley.
• Emsley RA., Green J., Dunn G. (2011). Designing trials of complex interventions for efficacy and mechanisms evaluation. Trials, 12(Suppl1), A143.
• Emsley RA, Dunn G & White IR. (2010). Modelling mediation and moderation of treatment effects in randomised controlled trials of complex interventions. Statistical Methods in Medical Research, 19(3), 237-270.
