Musttheevaluationofcomplexinterventionsbe
complex?
IsoldeBirdthistleandSianFloydWithDanielKwaro,MaryamShahmanesh,DanielCarter,Abdhalah Ziraba,
FrancesCowan,KathyBaisley,JudithGlynnandcolleagues
ISSTDR2019
LearningfromtheimpactevaluationofDREAMS
The fascination of what’s difficultHas dried the sap out of my veins, and rent
Spontaneous joy and natural content Out of my heart.
WBYeats,1916
1. Whatisacomplexintervention?WhatmakesDREAMSone?
2. Howdoyouevaluateacomplexintervention?Withoutrandomisation,canwebenefitfromemulatingatrial?
3. WhatarethelessonslearnedfromevaluationofDREAMSsofar?
Areflectionin3parts
TodrivedownHIVincidence,increasingconsensusthat:oMultiplestrategiesandmultiplesectorsareneeded(nosingleinterventionislikelytoworkalone)
o Thestrategiesmaydifferbycontext&ageo Similartrendinadolescenthealth&development…
Increasingadvocacyfor:o Combinationpackageso Coordinatedresponses
Complexinterventions
Thechangingprogrammelandscape
What makesaninterventioncomplex?
“Complexityresidesin…
…thenumberofinteractingcomponents,thenumberanddifficultyofbehavioursrequiredbythosedeliveringorreceivingtheintervention,thenumberofgroupsororganisational levelstargetedbytheinterventions,thenumberandvariabilityofoutcomes,andthedegreeofflexibilityortailoringoftheinterventionpermitted.”
MarkPetticrew,“Whenarecomplexinterventions‘complex’?Whenaresimple interventions ‘simple’?”EuropeanJofPublicHealth,2011
“Today,weareannouncingthatPEPFARisnowinvestingnearlyhalfabilliondollarstosupportanAIDS-freefutureforadolescentgirlsandyoungwomen.”
- USNationalSecurityAdvisorSusanERice,26Sept2015
“DREAMSisaboutmultiple
solutionssurroundingoneproblem:
newHIVinfectionsamong
adolescentgirlsandyoung
women.”
PEPFAR.GOV
DREAMStargets4relatedgroups…
ThroughaCorePackagewithmanyinteractingcomponents…withmanyinteractingcomponents
Heterogeneityinreal-worldimplementation- across65+districtsin15countries
15 DREAMS
Heterogeneityinreal-worldimplementation- across65+districtsin15countries
DifferentinterventionsindifferentcontextsDifferentmodelsofimplementation
Differentstrategiesfor”layering”(mentors,safespaces,‘passports’,uniqueIDs…)
Chimbindi &Birdthistle:‘TranslatingDREAMSintopractice:Earlylessonsfromimplementationinsixsettings’
PLOSOne2018
15 DREAMS
“Complexityresidesin……thenumberofinteractingcomponents,thenumberanddifficultyofbehavioursrequiredbythosedeliveringorreceivingtheintervention,thenumberofgroupsororganisationallevelstargetedbytheinterventions,thenumberandvariabilityofoutcomes,andthedegreeofflexibilityortailoringoftheintervention permitted.”
MarkPetticrew,“Whenarecomplexinterventions‘complex’?Whenaresimple interventions ‘simple’?”EuropeanJofPublicHealth,2011
Inalltheseways,andmore,DREAMSisacomplex
intervention
Howdoweevaluatesuchacomplexintervention?
(whilepreserving clarityandutilityandspontaneous joy…?)
Itdependsonourperspective&question
AcomplexquestionHowandwhetherthecomponentsworkindividually andtogether?Consideringtheirsynergies,phasechanges,feedbackloops,interactionsbetweenoutcomes,andtheprocessbywhichthecomponentsbringaboutchange.
AsimplequestionIsthewholepackageassociatedwithimprovedhealth?
MarkPetticrew,EuropeanJofPublicHealth,2011
Itdependsonourperspective&question
AcomplexquestionHowandwhetherthecomponentsworkindividually andtogether?Consideringtheirsynergies,phasechanges,feedbackloops,interactionsbetweenoutcomes,andtheprocessbywhichthecomponentsbringaboutchange.
AsimplequestionIsthewholepackageassociatedwithimprovedhealth?
MarkPetticrew,EuropeanJofPublicHealth,2011
Westartedhere,because…EvidenceoftheindividualcomponentsisalreadyknownTheknowledgegapiswhethertheyhaveanimpactwhendeliveredtogetherasapackage.
Howtoanswera‘simplequestion’aboutimpact,given…
Norandomisation
Whynot?Timing,feasibility,ethics
• Anurgencytobeginroll-outofDREAMS• DREAMSwouldtargetthemostvulnerableadolescent
girlsandyoungwomeninprioritydistricts• Equipoise:theinterventionwasexpectedtobebeneficial
(hardtojustifycontrolsorlargeexpenseofatrial)
Populationlevelchange
Individuallevelchange
Processevaluation
• Largecommunity-wide opencohorts
• Analysed asC/Sbefore,during,after
• Nested,closedcohortsofAGYW
• Analysedlongitudinally
• In-depthqualitativeresearch
Withoutrandomisation(balance),canweestimatecausaleffects?
Anobservationaldesign– usingpopulation-basedlongitudinaldatawithindemographicsurveillancesites
“TheC-Word…”
“…‘causal’[must]stopbeingconsideredtheC-wordthatinvestigatorsandeditorsavoid.Onlybypreciselydefiningthecausaleffectofinterestwillwehaveachanceofestimating itaccurately.”
MiguelHernan,“TheC-Word:Themorewediscussit,thelessdirtyitsounds”AJPH2018
Answeringcausalquestionsusingobservationaldatabyemulatingatarget(hypothetical)trial
1. Specifyingthetargettrial2. Emulatingthetargettrial3. Triangulating
MiguelHernan, AJPH2018
Applyingcausalinferenceapproaches
Thoughnotrandomised,theprinciples&techniquesofarandomisedtrialcanhelp
Populationlevelchange
Individuallevelchange
Processevaluation
• Largecommunity-wide opencohorts
• Analysed asC/Sbefore,during,after
• Nested,closedcohortsofAGYW
• Analysedlongitudinally
• In-depthqualitativeresearch
Applyingcausalinferenceapproaches
1.Specifyingthetargettrial(thehypotheticalexperiment)
Classificationof‘treatment’groups
WhoisaDREAMSbeneficiary?apriorimeasures
• InvitedtoparticipateinDREAMSversusnot[analogoustoITT]
• Invitedandreceivedmin3corepackagecategoriesversus 0-2
Mustbeupdatedovertime(withnewbeneficiaries)…
Categoricalmeasure1. Never/None2. 2017only3. 2018only4. 2017&2018
Binarymeasureo AnyDREAMSby2018:
Yes/No
62
29
2
914
5
22
48
1
91
38
1014
24
10
48
1014
0
86
48
2428
48
22
53
6
1713
83
46 45 4449
38
64
9
2723
0
10
20
30
40
50
60
70
80
90
100
HIVTesting&Counselling Socialassetbuilding Condom promotion andprovision Socialprotection SchoolbasedHIVprevention
CorePackage Categories
Never invited (N=166)
Invited in 2017 only (N=21)
Invited in 2018 only (N=109)
Invited both in 2017 & 2018 (N=281)
Expand contraceptive mix
Post-violence care
PrEP Parenting/ Caregiver programs
Community mobilisationand norms changes
ParticipationinDREAMScorepackageinterventionsin2018:18-22/24AGYWinNairobi
2.Emulatingthetargettrial
‘Randomassignment’oftreatmentgroupsAimtoachieve(near)balanceonbaselinecovariatesà Adjustforall(measured)confoundingfactors,e.g.,
throughpropensityscore*adjustment
Generateoverallcausaleffectsà Predictoutcomeforfullsampleifallversus ifnone
gotDREAMS*Propensity score=probability ofreceivingtheintervention basedonconfounder values(useful ifmanyco-variates,esp/forrareoutcomes;comparedwithadjustmentforeachindividual confounding variable)
Applyingcounterfactualreasoningforcausalinference
EarlyimpactsexpectedonKnowledgeofHIVStatus
Whatwouldbethedifference intheproportionofAGYWwhoknowtheirstatusifeverybodygotDREAMScomparedtoifnobodygotDREAMS?
Framingincausallanguagehelpsclearlyarticulatethequestionandidentifyasuitableestimand
ExamplesfromDREAMSevaluation(beforeendline dataareavailable)
DAGforNairobi
65.161.1
70.968.1
63.5
75.5
42.3
26.5
61.1
92.9 92.8 93.5
80.276.8
85.1
45.9
35.4
58.3
0
10
20
30
40
50
60
70
80
90
100
All:15-22 15-17 18-22 All:13-22 13-17 18-22 All:13-22 13-17 18-22
Nairobi[by2017] Siaya[by2018] uMkhanyakude [by2017]
NoAGYWareaDREAMSbeneficiaryAllAGYWareaDREAMSbeneficiary
Predictedproportions whoKnowtheirHIVStatus ifnoneversusallbenefitedfromDREAMS(in3DREAMSsettings)
Theabsolutedifference• Nairobi:27.7%increase[95%CI:22.8%,32.6%]• Gem:12.1%increase[95%CI7.7-19.6]• KwaZulu Natal:effectmodificationbyage
– 13-17Yearolds:8.95%[95%CI4.8%,14.4%]– 18-22Yearolds:-2.8%[95%CI-11.1%,5.7%]
• ImportanceofCONTEXT– Verydifferenteffectsacrosssite;agegroup
• ImportanceofMECHANISM(how?)– differenteffectsdependingontargetingand
implementation – needprocesseval data!)
Backtothecausalquestion…WhatwouldbethedifferenceintheproportionofAGYW
whoknowtheirstatusifeverybodygotDREAMScomparedtoifnobodygotDREAMS?
Processevaluationofcomplexinterventions:MedicalResearchCouncilguidanceBMJ 2015;350doi:https://doi.org/10.1136/bmj.h1258
Notenoughtoknowwhetheraninterventioniseffective,orevenbyhowmuch.
Importanttounderstandhowandwhy,andforwhom,especiallyinthe‘real-world’,undernon-trialconditions,ifwewantlessonsforreplication.
Forcomplexinterventions,‘outcomeevaluation’maynotbeenough
“Effectsizesdonotprovidepolicymakerswithinformationonhowaninterventionmightbereplicatedintheirspecificcontext,orwhethertrialoutcomeswillbereproduced.”
Processevaluation- Anessentialpartofdesigningandtestingcomplexinterventions- Guidedby3keythemes…
HighKnowledgeofHIVStatusinNairobiAreflectionofhowtheinterventionwasdeliveredandreceivedinthiscontext?
Insightsfromprocessevaluation
DeliveryofHIVtestingthrough DREAMSinKenya…• HIVtestingwasofferedattimeof
enrolment intoDREAMS,toallAGYW,regardlessofage,circumstance,orperceivedrisk
• AllDREAMSinterventions arecoordinatedbyoneIP,sothisapproachisconsistentandsystematic
• IPswereexperienced inHIVtestingprior toDREAMS
• Testingmadeavailableincommunity-basedsettings,including DREAMSsafespaces,home-based testing,andreferralstofacilities
Insightsfromprocessevaluation
HIVtestingthrough DREAMSwaspositivelyreceivedbyAGYWinKenya,e.g.,fortheconfidentiality…
“Initially there were these people who were afraid of going to the hospital, but right now you find that the HTS person comes to the safe space at least you can have the courage. Because for them they will come and test you and leave, they won’t talk about your results to other people. They will just tell you personally.”
- FGDwithoutofschoolAGYW
Musttheevaluationofcomplexinterventionsbe
complex?
Ananswer:“Whenitishelpfultoseeandanalysethemassuch”
MarkPetticrew,EuropeanJofPublicHealth,2011
(And itusuallyis)
Someclarityinthefaceofcomplexity…?
Ifthecomplexinterventionisnotrandomised,don’tnecessarilyshyawayfromcausality
Aimforcausalinference,whilewaryoftheassumptionsthatcaneasilybeviolatedunlesswe:
• Knowtheintervention,howitisimplemented(by/withwhom)andhowitchangesindifferentcontextsandovertime
• Consider,measureandaccountforconfounders• Triangulate:Usemultiple,complementaryapproachestoanswer
thequestion(withdifferentadvantagesanddisadvantages)
”Gesamtkunstwerk”(German: [gəˈzamtˌkʊnstvɛʁk],translatedas"synthesisofthe
arts",whendifferentformsarecombinedintoasingleunifiedwhole)
Anelegantaspirationforevaluationsofcomplexinterventions
With thanks to colleagues & partners
IsoldeBirdthistlehasnoconflict ofinterestandnothingtodisclose
