Forskningsdesignsogevidens

BobbyZachariaeProfessor,dr.med.,cand.psych.AarhusUniversitetshospitalog

AarhusUniversitet

Aimsofthepresentation

• Evaluatingtheavailableevidence– challengesandavailablesolutions– Problemsinevery-daythinking– Aimsofscientificmethod– Typesofevidence– Threatstovalidity–Whenonestudyisnotenough– Aspectstoconsiderwhenevaluating“effectiveness”

Zachariae

UnitforPsychooncologyandHealthPsychology - EPoS

• PsychooncologyResearchUnitestablishedin2000attheDept.OfOncology,AUHbasedonagrantfromtheDanishCancerSociety

• EPoS establishedin2011incollaborationbetweenAUH,Dept.OfOncology,BSS,AU,andDeptofPsychologyandBehavioural Science

• Currentstaff:17(1professor,2assoc.prof.1assistprof,1seniorresearcher,4post-docs,7PhD’s,1adm.)+8-10researchassistants.

Mybackground

• Researchareas– Psychoneuroimmunology– Painresearch– Psycho-socialcancerresearch– Patient-healthprofessionalinteractions– Healthpsychology– Psychosocialinterventions– Hypnosis,guidedimagery,mindfulness-basedintervention,expressivewriting

– Internet-deliveredinterventions– Efficacyofcomplementaryandalternativetreatments

4Zachariae

Mybackground• Researchmethodologies– Experimentalstudies(psychophysiology)– RandomizedControlledTrials(RCT)– Pragmatictrials–Mixed(qualitativeandquantitative)methods– Cross-sectionalandcohortstudies– Psychometrics– Systematicreviewsandmeta-analysis

5Zachariae

• Intellectualadvisetofuturegenerations:

• “Askyourselfonlywhatarethefactsandwhatisthetruththatthefactsbearout.Neverletyourselfbedivertedeitherbywhatyouwishtobelieve,orbywhatyouthinkwouldhavebeneficentsocialeffectsifitwerebelieved.Butlookonly,andsolely,atwhatarethefacts.”

Zachariae6

BertrandRussell,1872-1970

Whyscientificmethod?

Personalexperiencevs

Scientificevidence

Zachariae

”Manpreferstobelievewhathepreferstobetrue”

FrancisBacon,1561-1626

Automaticvsmanualprocessing“Fast”vs“slow”thinking

Zachariae

Manualmode

Dorsolateral prefrontal cortexVentromedial prefrontal cortex

Automaticmode

Shivetal.2005;Kahneman,2011

System2SlowEffortfulLogicalInfrequentCalculatingConscious

System1FastAutomaticEmotionalFrequentStereotypicSubconscious

Evolutionhasequippedhumans withtwo types

ofthinking

Zachariae

ProblemsinEveryday- (fast)Thinking

• Thepracticalvalueofeveryday(”fast”)thinkingisobtainedatthecostofbiasandprecision,e.g.,:– Wetendtoseepatterns,evenwheretherearenone– Weseecausalrelationships,evenwheretherearenone– Wetendtofocusonandrememberpositiveevidence– Wetendtooverestimateevidenceconfirmingourposition– Ourjudgmentsareinfluencedbythejudgmentsofour

surroundings(conformity)– Wetendtobelievethatpositiveandnegativetraits,

respectively,areassociated(clustering)– Wetendtooverestimatetheprobabilityofdramaticevents

Problemsinfastthinking• Thepracticalvalueofeveryday(”fast”)thinkingis

obtainedatthecostofbiasandprecision,e.g.,– Wehaveatendencytoseepatterns,alsowhenthereareno

patterns:• E.g.incompletelyrandomsequences:122212221221112112211.• E.g.Whenrollingdice:Ifwehavenotobtaineda”six”inmanyrolls,wetendtobelievethattheprobabilityincreases(althoughthechanceremains1/6evenafter100rolls)

– Wetendtoperceivecausalrelationships,alsowhentherearenone:• E.g.,wetendtoimaginecausalitybetweenXandY,ifYtakesplaceafterX(anecessarybutnotsufficientconditionforcausality)

Zachariae

Theechochamberproblem

• Closedideologyechochamber• Appliestopoliticsaswellasotherdomains

Zachariae

QualitativeMethods

QuantitativeMethods

Researchcommunity

ClinicalPractice

Thesolutionisano-brainer– butdifficulttopractice

Zachariae

”Welovetopredictthings– andwearen’tverygoodatit”

Nate Silver(1978-)

Sept.30

Thedomainsofscience• Meta-physics:Philosophy,epistemology:deductionand

reasoning

• Theoriesofscience:Meta-theoriesaboutmethod

• Theoreticalscience:Collecting,condensing,discussing,andinterpretingexistingtheoreticalandempiricalresearchresults

• Empiricalscience:Measuringphenomenaandtestinghypotheses

• Observinganddescribing

• Predicting

• Determiningcauses

• Explaining

Zachariae

Theaimofscientificmethod

• Generalaim:Togeneratemeasurableandtestabledata,graduallyaddingtotheaccumulationofhumanknowledge– Toproducereliable knowledge

– Toproducevalid knowledge

– Aboutcausal relationships

– Byaddressingpossiblesourcesoferror

Zachariae

Thefalsificationprinciple

Zachariae

KarlPopper:AScientificHypothesis MustBe"Falsifiable".• Wesupportahypothesisbyfalsifyingthenull-hypothesis• Ageneralapproach:Wedonot“prove”hypotheses– butmaximizeour

attemptstofalsifystatementsaboutobservations,associations,causality,andmechanisms

Typesofevidence

Zachariae

Evidencehierarchy

Zachariae18

Anecdote

Expertopinion

Casereports

Crosssectionalstudy

Case-controlstudy

Cohortstudy

Experimentaldesign(RCT)

Primaryresearch

Clinicalexperience

Challenge:themodelfavors internal validity

Typesofvalidity• Validity

– Internalvalidity(causality,excludingalternativeexplanations,sourcesoferror)

– Externalvalidity(generalizability)

– Ecologicalvalidity(pragmaticvalidity)

• Thethreetypesofvaliditysupplementeachother• Aredifficulttoobtainwithonesinglemethod• Internalvalidityaprerequisiteforexternaland

ecologicalvalidity• Reliabilityaprerequisiteforvalidity– butnotthe

reverse

Zachariae

Mistakingreliabilityforvalidity

Zachariae20

Researchquestions• Internalvalidity– Doesitwork(statisticalsignificance,superiority)?– Howwelldoesitwork(practicalsignificance)?– Doesitworkaswellassomethingelse(non-inferiority)?– Howdoesitwork(mechanisms,specific,non-specific)?

• Externalvalidity– Forwhomdoesitwork?– Forwhomdoesitnotwork?

• Ecologicalvalidity– Doesitworkintheclinicalpracticalcontext?– Clinicianandpatientadherence

Zachariae

Exampleofdesignmaximizinginternalvalidity:Randomizedcontrolledtrial

MBCTforpersistentpaininwomentreatedforbreastcancer

• 16-20%ofwomentreatedforbreastcancerexperiencepainafter5-9years

• Limitedpharmacologicaltreatmentefficacy

• Painisamultidimensionalphenomenonconsistingofsensory,cognitive,andaffectivefactors

• Mindfulness-basedtherapyteacheswaysofrelatingtobodilysensationsandemotionaldiscomfortwithhigherdegreeofacceptanceandopenness

• Mindfulness-BasedCognitiveTherapymaybeeffectiveforcancer-relatedpain

Zachariae

MBCTforpersistentpaininwomentreatedforbreastcancer

Zachariae

MBCTforpersistentpaininwomentreatedforbreastcancer

• Mediators oftheeffect ofMBCTonpain intensity:– Mindfulnessnon-reactivity facet;Pain catastrophizing

Zachariae

Johannsen,O’Toole,O’Connor,Jensen&Zachariae(underreview)

MBCTforpersistentpaininwomentreatedforbreastcancer

Zachariae

Cost-effectiveness ofMBCTforpersistent pain inwomen treated forbreast cancer

Johannsen,Sørensen,O’Connor,O’Toole,Zachariae(inpreparation)

ExampleofdesignPragmatictrialmaximizingbothinternal,external,andecological

validity

Zachariae

Evaluatingcomplementaryandalternativetreatments

• LabexperimentsandRCTsofenergyhealingyieldnegativeresults

• Complementaryandalternativemedicinesandtreatments(CAMs)areassociatedwithincreasedsymptomsofdepressioninbreastcancerpatients

• ActiveacupunctureisnotmoreeffectivethanplaceboacupunctureinadoubleblindedRCT

Zachariae

Zachariaeetal.2005

Pedersenetal.2013

Vaseetal.2013

Effectivenessofenergyhealingonqualityoflifeincolorectalcancerpatients

• CAMpractitionerscriticizevalidityoftraditionalresearchmethodologies,e.g.,RCTs

• Criticismsinclude:– Maynotbegeneralizedtothegeneralpopulation– Peoplemayhavestrongtreatmentpreferences– Standardizedoutcomemeasuresmaynotcoverpatients’individualconcerns

– Patientsmayprefersomepractitionerstoothers– Standardizationoftreatmentcontextmaycancelouteffects

Zachariae

Effectivenessofenergyhealingonqualityoflifeincolorectalcancerpatients

• Studydesignedtomaximizeinternal,external,andecologicalvalidity

• Colorectalcancerpatientsrandomizedto:– ARandomization

• Healingorcontrol

– BSelf-selection• Healingorcontrol

• Patients– Selectedtheirhealerfromalist– Treatmenttookplaceinhealer’sclinic– CompletedstandardizedQoLmeasures– Prioritizedpreferredoutcome– CompletedmeasureofattitudetowardsCAM

Zachariae

Effectivenessofenergyhealingonqualityoflifeincolorectalcancerpatients

• Nooveralleffectsonanyoutcomes• SmalleffectonQoLinsubgroup:Patientsinself-selectedhealinggroupwho

hadratedQoLasimportant,andwhohadapositiveattitudetowardsCAM

Zachariae

ExampleofdesignPragmaticevaluationofdailypractice

Zachariae

Pragmaticevaluation• Fagligt Selskab forPsykologer i Palliationog Onkologi• 11psychologiststreating92patientsorcaregivers• Psychologistquestionnaires:psychotherapeuticmodelsandtoolsused

• Patientquestionnairespre- andpostconsulation:– MYCaW (MeasureYourselfConcernsandWell-being):Primaryandsecondaryconcernandgeneralwell-being

– WorkingAllianceInventory• Nof1statistics:ReliableChangeIndex(RCI)

– Determineswhetherachangeisbeyondastatisticalerror

Zachariae

Pedersenetal.(unpublished)

PragmaticevaluationSelvvalgtproblemområdeogalmentvelbefindende

Sign.(RCI)Forbedring Ingenændring

Sign.(RCI)Forværring Datamangler

N(%) N(%) N(%) N(%)Primærtproblemområde

22(23,9) 64(69,6) 0(0,0) 6(6,5)

Sekundærtproblemområde

20(21,7) 43(46,7) 0(0,0) 29(31,5)

Almentvelbefindende

28(30,4) 60(65,2) 1(1,1) 3(3,3)

Zachariae

Statisticallysignificantpredictorsofsign.improvement:• Positiveexpectancies:Thesessionwillimprovemy

understandingofmyreactionsandemotions• Perceivedworkingalliance• Highereducationallevel

Pedersenetal.(unpublished)

Interpretingresults

Whatisaneffectsize?

• Study1:MeanscoreandstandarddeviationofHamiltonDepressionRatingScale(range:0-49):

• Intervention:16.5(13.0)• Control:20.5(14.0)• Cohen’sd=

• Study2:MeanscoreandstandarddeviationofBeck’sDepressionInventory(range:0-63)

• Intervention:17.5(7.0)• Control:21.5(8.0)• Cohen’sd=

Zachariae

Astandardizedeffect,e.g.,standardizedmeandifference,enablingcomparisonsacrossmeasuresandstudies

Cohen’s d=(Mean1– Mean2)/SD(pooled)

Whichinterventionismosteffective?

0.530.29

*)Todetectthedifferenceind betweenstudy1and2requiresasampleof610ineachgroup

BDIMCID(17%)2 =3.7=SD:0.49HDRSMCID(0,5SD)1

1)NICE,2004; 2)Buttonetal.2015

Significanceandprecision

Zachariae

Botheffectsizesarestatisticallysignificantlydifferentfrom”0”,arenotdifferentfromeachother

Onlytheeffectsizeofstudy1issignificantlydifferentfrom”0”.Thetwoeffectsizesarenotdifferentfromeachother

Botheffectsizesarestatisticallysignificantlydifferentfrom”0”,andtwoeffectsizesaresign.differentfromeachother(p=0.003)

Significance:p-values<0.05;Precision:95%Confidenceinterval

MCID

Challenges:Non-replicationpublicationbias“cherrypicking”

Onestudyisnotenough!

Interpretingnon-replicatedresults

Zachariae

”Replicationcrisis”

Zachariae

Collaboration,OpenScience(2015-08-28)."Estimatingthereproducibilityofpsychologicalscience".Science.349

Zachariae

IstreatmentXbetterthancontrol?(Note:Asmallervalueisbetter)

Evidencehierarchy

Zachariae42

Anecdote

Expertopinion

Casereports

Crosssectionalstudy

Case-controlstudy

Cohortstudy

Experimentaldesign(RCT)

Narrativesystematicreview

Quantitativesystematicreview

Primaryresearch

Clinicalexperience

Secondaryresearch

Challenge:themodelfavors internal validity

Systematicreviewsandmeta-analysis

Hyp

Method

Result

Hyp

Method

Result

Hyp

Method

Result

Hyp

Method

Result

Singlestudies

Hyp.

MethodResult

Systematicreviewandmeta-analysis

Zachariae

Zachariae

Resultsofmeta-analysis:

• Cantest(falsify)hypotheses• Testreproducibility(wereinitialresultsrandomorreliable?)

• Controlforrandomerror(variation)betweenstudies

• Testsystematicvariationbetweenstudies• Generalizeresults

Systematicreviewsandmeta-analysis

Zachariae

Numberofpublishedmeta-analysesperyear-PubMED1990-2013

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012

Zachariae

Zachariae

Historyofmeta-analysis• Pearson(1904)averagedcorrelationsbetweenmortalityand

inoculationfortyphoidfever.• Firstmedical“meta-analysis”onplaceboeffects(Beecher,1955)• Eysenck (1952)arguedthatpsychotherapywasineffective• Glassstandardizedandaveragedtreatment-controldifferencesfrom

375studies,namingit”meta-analysis”(Smith&Glass,1977)• ”Anexerciseinmega-silliness”(Eysenck,1978)• SimilarmethodsdevelopedbyRosenthalandRubin(1978)• Today(1977-2014):PubMed:60460”hits”- PsychINFO:13833”hits”• CochraneCollaboration(1993):Medicine• CampbellCollaboration(1999):Socialsciences• Handbookofresearchsynthesis(Cooper&Hedges,1994)

Zachariae

Narrativevs Systematic review:Amatteroflife or death

• From1972-81,7studiesinvestigatedtheeffectofsteroid-injektionsonprematuredelivery(associatedwithincreasedinfantmortality)

• Twostudiesshowedaweakpositiveeffect– theremainingstudieswerenon-significant

• Thetreatmentwasabandoned• Alater1989meta-analysisoftheoriginaldata• revealedasignificantpositiveeffectoninfantmortality(OR:0.50)• TheCochraneCollaborationlogoshowsdatafromthe1989meta-

analysiswww.cochrane.org

Riskofbias

Zachariae

Studyquality• Validity:”Theapproximatetruthofaninferenceorclaimabouta

relationship”

• Internalvalidity– Threats:allalternativemechanismsthatcouldexplainresults,e.g.,

”placebo”,group-differencesatbaseline,unevendropout

• Externalvalidity– Areresultsgeneralizabletootherintendedparticipantsandcontexts?

• Constructvalidity– Dotheoperationalcharacteristicsofinterventionandmeasures

adequatelyrepresentintendedabstractcategories?

• Statisticalconclusionvalidity– Thevalidityofthestatisticalinferencesregardingthestrengthofthe

relationship.Threatsincludeinsufficientstatisticalpower,regressiontowardsthemean,incorrectassumptionsabouttheunderlyingvariance

Zachariae

Shadish,Cook&Campbell,2002

Qualityassessment

• Allstudiesshouldbesubjectedapre-definedqualityassessment

• Alreadydevelopedormodifiedexistingchecklist

• Anewlydevelopedchecklist

E.g.,Jadad checklist(Jadad,1996)

Zachariae

Qualityassessment

Possiblescore:0-15Meanscore=11.3(SD=2.4;range:8-15)Inter-rateragreement:89.9%of225individualqualityratings

Masking conditions Poweranalysis Manipulationcheck

Zachariae

Zachariae&O’Toole,2015

Zachariae

ESofpublishedvsunpublished

Lipsey &Wilson,1993

Zachariae

Examplesofpublicationbias

• MedicaljournalsfromChinaalmostneverpublishnegativeresults(e.g.Panetal.2005)

• Only5%ofarticlesinjournalsfocusingonAlternativeandComplementaryMedicinepresentnegativeresults(Schmidtetal.2001)

• StudiesoriginatingfromEuropehavemorepositiveresultsthanstudiesfromtheUS(Soodetal.2007)

Publication bias

Zachariae

• CONCLUSION:• Theefficacyof

psychologicalinterventionsfordepressionhasbeenoverestimatedinthepublishedliterature

• Justasithasbeenforpharmacotherapy.

• Bothareefficaciousbutnottotheextentthatthepublishedliteraturewouldsuggest.

Preregistering

• ClinicalTrials.gov• Aims:– Increasetransparency

– Reduce fishingexpedition bias

– Presentationofpost-hochypotheses asapriory

– Enableassessment ofpublication bias

56Zachariae

Zachariae

Publication biasassessmentResearcher bias against submitting negative resultsPublisher bias against publishing negative results

Cooper,H.,DeNeve,K.&Charlton,K.(1997).

Begg,C.B.&Berlin,J.A.(1988).

Preregistering

• PROSPERO– CentreforReviewsandDissemination

58Zachariae

Examples

Internet-deliveredCBTforinsomnia• Annualprevalenceofinsomnia:10-20%• 6%withachronictrajectory• Pharmacologicaltreatmentisnon-curativeandlong-termuseis

associatedwithdependence,tolerance,side-effects,andincreasedmortality

• CognitiveBehavioralTherapyforInsomnia(CBT-I)isrecommendedasfirstchoice– basedonevidencefromsystematicreviewsandmeta-analyses

• LimitedavailabilityofCBT-I(trainedtherapists,geography,financialreasons)

• OnepossibilityisInternet-deliveredCBT-I(eCBT-I)• IseCBT-Ieffectiveandareeffectscomparabletoface-to-face

deliveredCBT-I?

Zachariae

Vidensråd forforebyggelse(2015)Søvnogsundhed;AmericanAcademyofSleep Medicine

Internet-deliveredCBT-I

Zachariae

Statistically significant effects found forprimary outcomes:Insomnia severity:Hedges’sg=1.09,p<0.001Sleep efficiency:Hedges’sg=0.58,p<0.001

Internet-deliveredCBT-I

Zachariae

Comparing withface-to-facedelivery

Nostatisticallysignificantdifferencesbetweeninternet-deliveredandface-to-face-deliveredCBT-I.Needfornon-inferioritytrialsdirectlycomparingeCBT-IandFtF

Psychologicalinterventionfordistressininformalcancercaregivers

• Informalcancercaregivers(ICCs)reportincreasedlevelsofpsychologicalandphysicalmorbidityandhighermortality

• PsychologicalinterventionssuchasCognitiveBehavioralTherapies(CBTs)havebeenshownefficaciousfordistress(anxietyanddepression)

• Aim:toevaluatetheavailableevidencefortheefficacyofCBTsfordistressandphysicalsymptomsamongICCs

Zachariae

Psychologicalinterventionfordistressininformalcancercaregivers

• Results– 36uniquetrials– 27RCTs,9– TotalN=4746ICCs– Smallsign.effectforalltrialscombined(g=0.08,p=0.014)

– NoeffectinRCT´s(g=0.04,p=0.200)

Zachariae

O’Toole,Zachariae,Penna,Mennin,Applebaum,2016

Criticismsofmeta-analysis

Criticisms ofmeta-analysis

• Mixesapplesandoranges

• Garbagein– garbageout

• File-drawerproblem

• Reductionism(onenumber)

Zachariae

d =2,7(95%CI:1,7– 3,7)

”Exerciseinmega-silliness”(H.Eysenck,1978)”Statisticalalchemy”(Feinstein,1995)

Zachariae

SystematicreviewsTransparency Riskofbias Effectestimation

Non-systematicreview

Low High None

Narrativesystematicreview

Medium-high Medium Qualitative“Votecounting”

Quantitativesystematicreview

(meta-analysis)

High Low MagnitudeDirectionPrecisionSub-groupcomparisons

Zachariae

Lessons learned

Zachariae

”Onemustseekthetruthwhereitis,notwhereonewouldlikeittobe”

Abbé deFaria,1746-1819

Lessonslearned• Whenevaluatingevidence– Onestudyisnotenough(replicability)– Avoidcherrypicking– focusonthecombinedevidenceofallavailableevidence

– Lessemphasisonp-valuesofindividualstudies– moreemphasisonmagnitude(effectsize)

– Moreemphasisonpracticalsignificance(e.g.,MCID)• Ahighlystatisticallysignificanteffectcouldbeofairrelevantmagnitude

• Anon-statisticallysignificanteffectcouldpotentiallybeclinicallyrelevant

Zachariae

Lessonslearned• Consider– StatisticalpowerandriskofType-2error– Theprecisionoftheeffect(theconfidenceinterval)– Homogeneityoftheexistingevidence– Studyquality– potentialbiasandthreatstovalidity– Publicationbias– thetendencytounderreportnull-findings

– Cost-effectiveness– relativetotreatmentasusual• Establishingevidence– Isacomplexcumulativeprocess

Zachariae

Futuretasks• Thefuturewillbringpublicandpolicy-baseddemandsforevaluationofclinicalefficacyandcost-effectiveness

• Psychologistsareadvisedto:– Worktoestablishaproactive,evidence-basedprofessional

culture– Nottotaketheeffectivenessofpsychologicalapproachesas

self-evidentbuttofocusonthebestavailableevidence– Acceptwhenpsychologicalapproachesarenoteffective,be

transparentaboutit,andworktoimprovethesituation– Promoteresearch-basedpracticeandpractice-relevant

research,establishcollaborationbetweenresearchersandclinicians,andconductresearch-basedevaluationincollaborativenetworks

Zachariae

Zachariae

Examples