Questions for proponents of theory-drivenevaluation

1

InAppreciation• Focusonuseful/’good’theoryofchange• Learningisaprocessnotjustaproduct• Empiricalbasisforatheoryofchange• Arichervocabularyaroundtheoriesofchange

– Capabilities,opportunities,motivation– Unpackingtheblack-boxofinterventions

• Ascienceofcausation– Assumptions,risks,mechanisms– Contributionanalysis

• Richerdescriptionofhowprogramscanwork• Knowledgetranslationdescription• Highlightsthecomplexityoftheintervention• Stepstowardsprecisionintheorizing

2

GuidingQuestions

AStoryfromtheWorldHealth

Organization

EvaluationofDancingwithParkinson’s

Summary

Pushingtheory-drivenevaluations:

Somequestions

Somequestions• Howdoesonetestatheoryofchange?

• Howdoesonetestalinkofthetheoryofchange?

• Howdoesonetestkeyassumptions?

• Whatworksforwhom,underwhatcontexts?

• Howdoesonetestiftheassumptionswillholdacrosscontexts?

• Representationchallenges;• Fromrepresentationtotesting• Understandingdiversity

4

ASTORYFROMTHEWORLDHEALTHORGANIZATION:OfMiceandElephant

5

LESSONS

The importanceof understandingthenatureofconnections Issuesofpower

Theattribution/contributionproblem

Theinequityproblem

6

ANEXAMPLEOFAREALISTEVALUATION:EVALUATIONOFDANCINGWITHPARKINSON’S

7

• PDisthesecondmostcommonneurodegenerativedisorderafterAlzheimer’sdisease

• Causesavarietyofphysicalsymptoms:rigidity,tremor,slownessofmovement,impairedbalance,aswellas“freezing”ortheinabilitytoinitiatemovement

• Othernon-motorsymptomsinclude:depression,anxiety,cognitiveimpairment,problemsswallowing,chewing,andspeaking,maskedfacialexpressions,fatigue,andsleepdisturbances

• Thesesymptomsinvariablyleadtoincreasedsocialisolationandlackofengagementwiththeworld

Parkinson’sdisease(PD)

8

• Danceofferssimilarbenefitsasothermovementforms:focusonbalance,strength,andflexibility

• Butartisticelementsofdanceaddvaluethatothertypesofexercisedonot

• ForExample:– Music:rhythmicandemotionalstimulihelpinitiatemovementmoreeasilyforpeoplewithPD

– Imagery:assistsinfosteringmovementandself-expression– Social:danceislearned,practiced,andenjoyedwithothers

– Cognitive:newmovementsequences,improvisationexercisesengageandstimulatethemind

WhyDancewithPD?

9

IndividualswhodancedregularlybeforesymptomsofParkinson’sdisease

Dancingataloweredcapacityremindsthemofhowtheyhavedeteriorated

Theyavoidattendingclassesandthereforedon’tgethealthbenefitsofparticipation

IndividualswhohaveneverdancedregularlybeforeDWP

Whiledancingtheyexperiencethefeelingoffreedomfromtheirsymptomsandrelieffromstressanddepression

Theyfeelbetter,haveanincentivetoattendclassesregularly,andthereforegainhealthbenefits

Individualswholivealone

Socialconnectionwithotherparticipants,teachersandvolunteers,physicalcontact,andcreativeandemotionalexpression

Deepenedsocialconnections,enhancedartisticexpression,improvedqualityoflife

Individualswithcare-partnerorcaregiver

1+hourthatpersonwithPDisengagedindancinginasafeenvironmentwithotherswithPD

Enhancedcare-partner/caregiverreliefandsupport

Context Mechanism Outcome

10

Data:Focusonshorttermvariations,linkagesbetweenDWPandoutcomes,understandingofotherdrivers,temporal

context,betweenandwithin-individualvariations1.ReversedPre- andPost-surveydesign

3. NewClientBaselineSurvey+QualitativeLongitudinalResearch

15WeeksJune22-October4

EndofSpringClasses SummerBreak StartofAutumnClassesLastweekofJune Julytomid-Sept Mid-September

2.DailyJournals

StartofAutumnClassesFrommid-September Followup

11

• AnExampleofaSingleIndividual:– DailyEntries:Howareyoudoingtoday?– Responds“okay”91%ofthetime,“good”5%ofthetime,and“bad”4%ofthejournaldays– Ratesthedayas“okay”evenondayswithsomethingtolookforwardto(e.g.,dinnerwith

friends,concerts,birthdays)– Notably,3ofthefive“good”dayswereDWPclassdays

12

Jun-24DWP Jun-25 Jul-10 Jul-22DWP Jul-29DWP Aug-8 Aug-11 Aug-27 Sep-9DWP Sep-26

GOOD“Busy day! Attended my nephew's graduation from law school, had an important business meeting in the PM, and attended a jazz concert in the evening.”

OK“Had lunch with my best friend. Always cheers my

spirit.”

GOOD“Good news from the oncologist - my PSA continues to be low.”

OK“Looking forward to dinner with 3 of my

cousins and my brother.”

OK“My brother took me out to dinner on my birthday. Felt fatigued but otherwise ok.”

OK“I went to afternoon concert at

Koerner Hall with friends. Felt tired at the end.”

OK“Another full day at a

conference at Harbourfront. Felt

tremendous fatigue around 4pm.”

GOOD“Had blood work done in the morning at Princess Margaret Hospital in preparation for my appointment next week with the radiation oncologist.”

OK“Long day - picked up by a friend who drove us to another friend who was

having a party at her place in Caledon. Long, long day. Very fatigued.”

OK“DWP begins today – looking forward.”

13

14

SOMEQUESTIONSFORPROPONENTSOFTHEORY-DRIVENEVALUATION

15

16

1.Whatisagoodenoughtheoryofchange?

• “Whatarecharacteristicsofausefulimplementabletheory?Howdoesthetheoryofacomplexinterventiondifferfromatheoryofasimpleintervention?”

• “…invirtuallyeverycase,thetheoriesofchangethatdrovetheinitiativesandtheirevaluationswerenotreallyadequatetosubstantiatetheconnectionsbetweenactivitiesandoutcomes.”

– Auspos andKubisch,2004

17

• ““Isittimetoshiftthebalanceinwhatwedefineasqualityfromanexclusivefocusonempiricalmethod(theextenttowhichauthorshaveadheredtotheacceptedrulesofcontrolledtrials)toonethatembracestheory(theextenttowhichatheoreticalmechanismwasexplicitlydefinedandtested)?”

• Theneedforcasestudieson“whatareusefultheoriesofchange?Whatmakesagoodtheoryofchange?

18

2) TheExpectationsProblem:TheAnticipatedTrajectoryofImpacts

• “Fundamentaltojudgingperformanceishavingclearexpectationsofananticipatedtrajectoryofimpactofprogrammes (Woolcock,2009).Itisnotentirelyclearhowtheknowledgeofsuchatrajectoryofimpactsisarrivedat,especiallyforcomplexinterventions.”

• Howdoesonedevelopaclearunderstandingoftheanticipatedtrajectoryofimpacts?

19

20

• “Evenacursoryreadingofsocialtheory,forexample,wouldsuggestthatinfactthemostlikelyshapeofsuchprojects’functionalformisaJ-curve(thatis,thingsgetworsebeforethey–hopefully,maybe– getbetter)orastepfunction(thatis,longperiodsofstasisfollowedbyasuddenrupturebroughtonby,say,anelectionorthereachingofa‘tippingpoint’intheadoptionofanewfertilizertechnology,inwhichprevailingnormsand/oruptakebyaninfluentiallocalleaderrapidlyleadsotherstodolikewise).”

21

• “Inshort,thedevelopmentprofessionstronglypreferstosellknown,universalsolutionswithhigh,immediateandreadilymeasurableimpactratherthanwrestlewithambiguous,context-specificproblemsthatmaynothave(atleastexante)aknownorevenknowablesolution.“

22

3) TheKnowledgeTranslationProblem

• Howdoesaprogrambuildknowledgeofwhatworksforwhom?

• “Whatprogrammemechanismsareneededforwhatcontexts?Theevaluationexplorationinourviewshouldnotberestrictedtotheimpactevaluationquestion(whatworksforwhom?)butalsoshouldbefocusedontheknowledgetranslationquestion:whatshouldworkforwhom?”

23

• Howissuchknowledgeofheterogeneousmechanismsimplementedinprograms?

• Howdoweincorporateknowledgeofsuchheterogeneityintoourtheoriesofchange?

24

4)TheMethodsProblem:DevelopinganEcologyofEvidence

• Aproblemthatalsocomesthroughreasonablyclearlyintherecentliteratureonevaluationmethodsistherecognitionthat‘learningisnotthemonopolyofasinglemethod’(Rodrik,2009).

• Thelessonincreasinglyisonthetypesofevidencethatareusefulgiventheheterogeneousnatureoftherealworld/complexsystems.

– ForexampleRodrik (2008:5)makesthisimportantpoint:“The‘hardevidence’fromtherandomizedevaluationhastobesupplementedwithlotsofsoftevidencebeforeitbecomesusable.”

25

• Forthemostpartthetheory-drivenevaluationfieldhasnotfullyengagedwithmethodsthatcanhelpexplicatetheprogrammetheorymorefully,norhasitclearlydescribedhowmethodscanbeusedtotestlinkagesintheprogrammetheory.

26

Someexamplesofmethods• NetworkAnalysis Howdoestheprogrammeoperationalize networksof

interventions?

• EventStructureAnalysis Whatistheimpactoftheeventstructure(forexample,theinteractionsbetweenpartneringorganisations)onoutcomesovertime?

• ConceptMapping Howdodifferentgroupsofstakeholdersconceptualisetheintervention?Dotheseconceptualisations changeovertime?

• SystemDynamics Whataretheanticipatedtrajectoriesoftheimpactsoftheoverallpolicyintervention?Howwouldchangingkeycomponentsofthepolicyinterventionaltertheexpectedtrajectoryofoutcomes?

27

• ObservationalStudies(Propensityscoringtechniques) Intheabsenceofrandomizeddesigns,whatalternativeanalyticaltechniquescanhelpassesscausaleffectsofinterventions?Whatisthehiddenbiasinobservationalstudies?

• MethodsofDevelopmentalTrajectoriesDoesthesameinterventionhaveverydifferentimpactsovertimefordifferentgroupsofindividuals(orcommunities)?

• Respondent-DrivenSampling Howdoestheprogramme(andevaluation)reachindividualswhoarehardtoreachandnotfoundinastandardsamplingframe?

• Adaptive/SequentialDesigns Howcanadesignhelpaninterventionadapt/evolvefromlargecomplexitywithmanycomponentstoaninterventionwithafew‘effective’components?

• RealistSynthesis Howdoesevidencesynthesishelpidentifythemechanismsbywhichaninterventionworks?

28

29

Summary

Theneedforcasestudies

Utilityfromtheuserperspective

Generativecausation

Improvedlanguage HeterogeneousImpacts:ConnectAssumptionsto

Contexts

TestableAssumptions

Prioritization

MethodologicalImplications