RoB 2.0:Arevisedtooltoassessriskofbiasinrandomizedtrials

MatthewPageUniversityofBristol,UK

WithspecialthankstoJulianHiggins,Jelena Savović,Asbjørn Hróbjartsson,IsabelleBoutron,BarneyReeves,RoyElbers,JonathanSterne

Overview

• ReminderoftheCochraneriskofbiastoolforrandomizedtrials• Theneedforanewtool• Developmentofthenewtool• Keyinnovationstothetool• Someexcerptsfromthetool• Someunresolvedissues

3

BMJ 2011; 343: d5928

4

Foam dressings for venous leg ulcers

CurrentCochranetoolforriskofbiasinrandomizedtrials

• Sixsourcesofbias(withoptional‘Other’)

• Foreachsource,• Freetexttodescribewhathappened• Judgement:Lowrisk/Unclearrisk/Highriskofbias

• Somesourcesofbiascanberepeatedfordifferentendpoints

CurrentCochranetoolforriskofbiasinrandomizedtrials

• CochraneRoB toolisverywidelyused(Jørgensen 2016)• 100outof100Cochranereviewsfrom2014(100%)• 31outof81non-Cochranereview(38%)

• >2700citationsfromnon-Cochranesources

• Thescientificdebateonriskofbiashascontinued

• Evaluationstudiesofthetool• Userexperience:surveyandfocusgroups(Savovic 2014)• Inter-agreementstudies(e.g.Hartling2009&2013)• Actualuseinreviewsandpublishedcomments(Jørgensen2016)

Someissuesraisedwithexistingtool

• Usedsimplistically

• Usedinconsistently (domainsaddedorremoved)

• Modestagreement rates

• Only5-10%oftrialsinCochranereviewsarescoredasLowriskofbias

• overuseof“unclearrisk”?

• RoB judgementsaredifficult forsomedomains,particularlyincompleteoutcomedataandselectivereporting

• Challengeswithunblindedtrials

• Notwellsuitedtocross-overtrialsorcluster-randomizedtrials

• Notwellsetuptoassessoverallriskofbias

Funding

• Therevisedtoolforrandomizedtrials(RoB2.0)wassupportedbytheUKMedicalResearchCouncil NetworkofHubsforTrialsMethodologyResearch(MR/L004933/1- N61)

RoB2.0:developmentchronology

• RevisionoftheRoBtoolstartedinMay2015• 1st DevelopmentmeetingheldinBristolinAugust2015• 1st ‘workingdraft’ofthetoolcompletedJanuary2016• PilotingphaseFeb– March2016• Revised‘workingdraft’• 2nd DevelopmentmeetingheldinBristolon21-22April2016• Developmentoffurtherguidanceandpiloting• ReleasedforSeoulColloquium

RoB2.0:contributors

• Coregroup:• JulianHiggins,Jelena Savović,MatthewPage,AsbjørnHróbjartsson,Isabelle

Boutron,BarneyReeves,RoyElbers,JonathanSterne• WorkingGroupmembers:

• DougAltman,NatalieBlencowe,MikeCampbell,ChristopherCates,RachelChurchill,MarkCorbett,NickyCullum,FrancoisCurtin,AmyDrahota,SandraEldridge,JonathanEmberson,BrunoGiraudeau,JeremyGrimshaw,ShareaIjaz,SallyHopewell,AsbjørnHróbjartsson,PeterJüni,JamieKirkham,TobyLasserson,TianjingLi,StephenSenn,SashaShepperd,IanShrier,NandiSiegfried,LesleyStewart,PennyWhiting

• And:HenningKeinke Andersen,MikeClarke,JonDeeks,GeraldineMacDonald,RichardMorris,MonaNasser,Nishith Patel,JaniRuotsalainen,HolgerSchünemann, JayneTierney

Keyinnovations

• Result-focussed assessments• Fixed(inclusive)biasdomains,notmodifiable• “Signallingquestions”tofacilitateriskofbiasjudgements• Newresponseoptionsforriskofbias,without‘Unclear’option• Formaloverall riskofbiasjudgement

• Somerethinkingoftheassessment:• Importantdistinctionbetweeneffectsofinterest• Selectivereportingfocusesonreportedresult

RoB1.0 RoB2.0

Randomsequencegeneration(selectionbias) Biasarisingfromtherandomization

processAllocationconcealment(selectionbias)

Blindingofparticipantsandpersonnel(performancebias)

Biasduetodeviationsfromintendedinterventions

Incompleteoutcomedata(attritionbias) Biasduetomissingoutcomedata

Blindingofoutcomeassessment(detectionbias) Biasinmeasurementoftheoutcome

Selectivereporting(reportingbias) Biasinselectionofthereportedresult

Otherbias N/A

N/A Overallbias

Fundingandvestedintereststobeaddressed,butnotwithinthispartofthewiderframeworkWorkinggroupledbyAsbjørn Hróbjartsson and

IsabelleBoutron

Signallingquestionsandjudgements

• Signallingquestionsareintroducedtomakethetooleasier(andmoretransparent)• ‘Yes’,‘Probablyyes’,‘Probablyno’,‘No’,‘Noinformation’

• Riskofbiasjudgementsfollowfromanswerstosignallingquestions(canbeover-ridden)• ‘Lowriskofbias’,‘Someconcerns’,‘Highriskofbias’

• Achangeintheinterpretationofthejudgements,sothata‘Highriskofbias’judgementinonedomainputsthewholestudyathighriskofbias

• Overallriskofbiasjudgementcanthenbecompletedautomatically(canbeover-ridden)

Overallriskofbiasjudgement

Lowriskofbias Thestudyisjudgedtobeatlowriskofbias foralldomainsforthisresult.

Someconcerns Thestudyisjudgedtobeatsomeconcernsinatleastonedomainforthisresult.

Highriskofbias Thestudyisjudgedtobeathighriskofbias inatleastonedomainforthisresult.ORThestudyisjudgedtohavesomeconcernsformultipledomains inawaythatsubstantiallylowersconfidenceintheresult.

riskofbias.info

Someexcerptsfromthetool

Examplealgorithm

4.1Wereoutcomeassessorsawareoftheinterventionreceivedby

studyparticipants?

4.2Wastheassessmentoftheoutcomelikelyto

beinfluencedbyknowledgeofintervention

received?

Highrisk

Someconcerns

Lowrisk

Lowrisk

Y/PY/NI

N/PN

Y/PY

N/PN

NI

Biasarisingfromtherandomizationprocess

Biasarisingfromtherandomizationprocess

• Currenttoolincludestwoseparatedomains:• sequencegeneration• allocationconcealment(bothunder“selectionbias”)

• Botharerelatedtorandomization/allocationofparticipatesintotreatmentarms

• Failuretoimplementeitherprocessadequatelycreatesopportunitiesforeithertheenrolmentintothestudyortheallocationofenrolledparticipantsintogroupstobeinfluencedbyprognosticfactors

• Theendresultisthesame– unbalanced(biased)distributionofpatientsbetweengroups(notafaircomparison,confounding)

Ø ItmakessensetocombineSGandACintoasingledomain

Biasarisingfromtherandomizationprocess

• EvaluationstudiesoftheuseoftheRoB toolinCochraneshowthatreviewersoftenconsiderbaselineimbalanceas“Otherbias”

• Butthisisrelatedtothesuccessofrandomization

Ø Itmakessensetoincludebaselineimbalanceinthesamebiasdomain

• Indicatorsthatrandomizationwasnotperformedadequately:• unusuallylargedifferencesbetweeninterventiongroupsizes;• asubstantialexcessinstatisticallysignificantdifferencesinbaselinecharacteristics;

• asubstantialexcessinclinicallyimportantdifferencesinbaselinecharacteristics

Biasarisingfromtherandomizationprocess

1.1Wastheallocationsequencerandom?1.2Wastheallocationsequenceconcealeduntil

participantswererecruitedandassignedtointerventions?

1.3Weretherebaselineimbalancesthatsuggestaproblemwiththerandomizationprocess?

Randomizationmethods

Additionalevidenceofproblems

Biasduetodeviationsfromintendedinterventions

Theeffectofinterest

• Thecurrenttoolhasverylittletosayaboutsituationsinwhichblindingisnotfeasible• (otherthantoclassifyasnotblindhencehighriskofbias)

• Issuesofperformancebiasverydifferentfordifferenteffectsofinterest,yetpoorlyaddressedincurrentRoB tool

Theeffectofinterest

• Thecurrenttoolhasverylittletosayaboutsituationsinwhichblindingisnotfeasible• (otherthantoclassifyasnotblindhencehighriskofbias)

• Issuesofperformancebiasverydifferentfordifferenteffectsofinterest,yetpoorlyaddressedincurrentRoB tool

• effectofassignmenttointervention• e.g.doesreferraltophysicaltherapyincreasepost-operativemobility?(thequestionofinteresttoahospitalmanageraboutwhethertointroduceareferralprogramme)

• effectofstartingandadheringtointervention• e.g.doesattendingaphysicaltherapyprogramincreasepost-operativemobility?(thequestionofinteresttoanindividualaboutwhethertoattendphysicaltherapy)

Theeffectofinterest

• Wheninterestedineffectofassignment tointervention• Deviationsfromintendedinterventionarenotimportantprovidingthesedeviationsreflectusualpractice

• e.g.itisusualpracticeforsomereferredpatientstonotattendphysicaltherapy,ortocompleteonlysomesessions

• thisdifferstobehaviourthatreflectsexpectationsofadifferencebetweeninterventionandcomparator

• Wheninterestedineffectstartingandadheringtointervention• Deviationssuchaspooradherence,poorimplementationandco-interventionsmayleadtoriskofbias

• Wethereforehavedifferenttoolsforthesetwoeffectsofinterest

Biasduetodeviationsfromintendedinterventions

Effectofassignment tointervention2.1.Wereparticipantsawareoftheirassignedinterventionduring

thetrial?2.2.Werecarersandtrialpersonnelawareofparticipants'assigned

interventionduringthetrial?2.3.IfY/PY/NIto2.1or2.2:Weretheredeviationsfromthe

intendedinterventionbeyondwhatwouldbeexpectedinusualpractice?

2.4.IfY/PYto2.3:Werethesedeviationsfromintendedinterventionunbalancedbetweengroupsand likelytohaveaffectedtheoutcome?

2.5Wereanyparticipantsanalysedinagroupdifferentfromtheonetowhichtheywereassigned?

2.6IfY/PY/NIto2.5: Wastherepotentialforasubstantialimpact(ontheestimatedeffectofintervention)ofanalysingparticipantsinthewronggroup?

Blinding

Deviationsreflectusual

practice?

Firstprincipleof

ITT

Biasduetodeviationsfromintendedinterventions

Effectofstartingandadheringtointervention2.1.Wereparticipantsawareoftheirassignedintervention

duringthetrial?2.2.Werecarersandtrialpersonnelawareofparticipants'

assignedinterventionduringthetrial?2.3.IfY/PY/NIto2.1or2.2:Wereimportantco-interventions

balancedacrossinterventiongroups?2.4.Wastheinterventionimplementedsuccessfully?2.5.Didstudyparticipantsadheretotheassignedintervention

regimen?2.6.IfN/PN/NIto2.3,2.4or2.5:Wasanappropriateanalysis

usedtoestimatetheeffectofstartingandadheringtotheintervention?

Blinding

Specificdeviations

Overcomebyanalysis?

Biasduetomissingoutcomedata

Missingoutcomedata

• Whencompleteoutcomedataforallparticipantsisnotavailableforyourreview• attrition- losstofollowup,withdrawals,othermissingdata• exclusions– someavailabledatanotincludedinreport

• Considerations• howmuchdataismissingfromeachgroup?(includenumbersinyourdescription)

• whyisitmissing?• howwerethedataanalysed?

Source:CochraneTraininghttp://training.cochrane.org/resource/assessing-risk-bias-included-studies

Biasduetomissingoutcomedata

3.1.Wereoutcomedataavailableforall,ornearlyall,participantsrandomized?

3.2.IfN/PN/NIto3.1:Aretheproportionsofmissingoutcomedataandreasonsformissingoutcomedatasimilaracrossinterventiongroups?

3.3.IfN/PN/NIto3.1:Isthereevidencethatresultswererobusttothepresenceofmissingoutcomedata?

Anymissingdata?

Amountandreasons?

Resultsrobust?

Biasinmeasurementoftheoutcome

Biasinmeasurementoftheoutcome

• Systematicdifferencesbetweengroupsinhowoutcomesareassessed

• Someoutcomesaremorepronetobiasthanothers• Patient-reportedoutcome(e.g.pain,qualityoflife)• Observer-reportedinvolvingjudgement(e.g.clinicalexamination)

• Observer-reportednotinvolvingjudgement(e.g.all-causemortality)

Biasinmeasurementoftheoutcome

4.1.Wereoutcomeassessorsawareoftheinterventionreceivedbystudyparticipants?

4.2.IfY/PY/NIto4.1:Wastheassessmentoftheoutcomelikelytobeinfluencedbyknowledgeofinterventionreceived?

Blinding?

Assessmentinfluenced?

Biasinselectionofthereportedresult

Selectivereporting

• Currenttooltakesabroadapproachtoselectivereporting• Anyevidenceofitinthetrialreports?

38

Results

• Selectivenon-reportingbiasestheresultofthemeta-analysiswhichcannotincludethetrialthatomittedtheoutcome;itdoesnotbiasthetrialresult

• Thisissimilartopublicationbias(non-reportingofastudy)

WeincludeonlyselectionofthereportedresultintheRoB 2.0tool

...andconsiderselectivenon-reporting inotherways

Biasinselectionofthereportedresult

Trialresultisbiasedbecauseithasbeenselectedonthebasisoftheresultsfrommultiple:• Outcomemeasurements

• Scales• Definitionsof/criteriaforanevent• Timepoints

• Analyses• Unadjustedvsadjustedmodels• Differentsetsofcovariatesinadjustedmodels• Finalvaluesvschangefrombaselinevsanalysisofcovariance• Continuousscaleconvertedtocategoricaldatawithdifferentcut-

points

Biasinselectionofthereportedresult

Arethereportedoutcomedatalikelytohavebeenselected,onthebasisoftheresults,from...

5.1....multipleoutcomemeasurements(e.g.scales,definitions,timepoints)withintheoutcomedomain?

5.2...multipleanalysesofthedata?

Selectiveoutcomereporting

Selectiveanalysisreporting

Piloting

• RoB 2.0hasundergonemultiplephasesofpiloting• informeddevelopmentandrefinement• moreisalwayswelcome

• Formalstudiesofinter-rateragreementnotyetperformed

• Fullguidanceavailableatriskofbias.info• initialdraft,subjecttominorrefinements

Someunresolvedissues

• Howmanyresultstoassessperstudy?• Howtointegrateintodatacollectionprocess?• Howtopresentassessmentsinareview?

• Implementation• RoB 2.0willneedcarefulconsiderationtomaketheprocessefficientformultipleoutcomes

• DiscussionsinitiatedwithRevMan andCovidence teamatSeoulColloquium

Concludingremarks

• WebelieveRoB 2.0offersconsiderableadvantagesovertheexistingtool

• Onceprogrammedintosoftware,weexpectthetoolwillbeeasytouseandintegrateintotheinterpretationofresults

• WeareextremelygratefultoallthosewhohavecontributedtothedevelopmentofRoB 2.0

• RoB 2.0isavailableatriskofbias.info