How to get started with evaluation that can be used

Sonal Zaveri, Secretary, Community of Evaluators, South Asia

Presentation at APEA, November 2016

Agenda

Introductions A.  Find out where your ‘are at’ in evaluation B.  Concepts & language

C.  Users, uses & readiness - exercise D.  Utilization-focused evaluation: our practice

A. Where are you at in evaluation?

1.Whenyouhearthewordevalua&on:a)Whatposi&veexperiences/ideascometomind?YELLOWLARGEb)Whatdifficultexperiences/ideascometomind?ORANGE

Instruc(onsforCards:oneideapercard;usedarkmarkers;useshortphrases;usecardcoloursasmarkedfora,b,c,andd

A. Where are you at in evaluation?

2.Thinkbackaboutasuccessfulevalua&onexperience:c)Canyouexplainwhatcharacteris)csmadeitsuccessful?YELLOWSMALLd)Whatwerethekey ,youexperienced?GREEN

Instruc(onsforCards:oneideapercard;usedarkmarkers;useshortphrases;usecardcoloursasmarkedfora,b,c,andd

A. Where are you at in evaluation?

Haveyouheardabout‘uJlizaJon-focusedevaluaJon’?

Ifyouhave,whatdoyouknowaboutit?

B. Concepts & language

Readiness: •  Senior management buy-in •  Allocation of staff and budget to cover the

implementation of the evaluation •  Stated interest by staff to learn about evaluation •  Staff and management time commitment •  Internal coordination •  Willingness to learn, share, document and report

B. Concepts & language

•  Input, Activity, Output, Outcome, Impact

B. Concepts & language

OutcomeMapping

OutcomeMappingconcepts

hOp://www.hedon.info/OutcomeMapping

B. Concepts & language

B. Concepts & language

Expect-to-see

outcomes

Like-to-seeoutcomes

Love-to-seeoutcomes

B. Concepts & language

•  Input, Output, Outcome, Impact

•  Simple, complicated, complex

Simple,complicated,complex

Glouberman,S.&Zimmerman,B.2002.Complicatedandcomplexsystems:Whatwouldsuccessfulreformofmedicarelooklike?CommissionontheFutureofHealthcareinCanada.DiscussionPaperNo.8hOp://c.ymcdn.com/sites/www.plexusinsJtute.org/resource/collecJon/6528ED29-9907-4BC7-8D00-8DC907679FED/ComplicatedAndComplexSystems-ZimmermanReport_Medicare_reform.pdf

B. Principles & languageTRADITIONALEVALUATIONS DEVELOPMENTALEVALUATIONSRenderdefiniJvejudgmentsofsuccessorfailure. Providefeedback,generatelearnings,supportchanges

indirecJon.

Measuresuccessagainstpredeterminedgoals. Developnewmeasuresandmonitoringmechanismsasgoalsemergeandevolve.

PosiJontheevaluatoroutsidetoassureindependenceandobjecJvity.

PosiJonevaluaJonasinternal,teamfuncJonintegratedintoacJonandongoinginterpreJveprocesses.

DesigntheevaluaJonbasedonlinearcause-and-effectlogicmodels.

DesigntheevaluaJontocapturesystemdynamics,interdependencies,modelsandemergentinterconnecJons.

AimtoproducegeneralizablefindingsacrossJmeandspace.

Aimtoproducecontext-specificunderstandingsthatinformongoinginnovaJon.

AccountabilityfocusedonanddirectedtoexternalauthoriJes,stakeholdersandfunders.

Accountabilitycentredontheinnovators’deepsenseoffundamentalvaluesandcommitment.

Accountabilitytocontrolandlocateresponsibility. Learningtorespondtolackofcontrolandstayintouchwithwhat’sunfoldingandtherebyrespondstrategically.

Evaluatordeterminesthedesignbasedontheevaluator’sperspecJveaboutwhatisimportant.TheevaluatorcontrolstheevaluaJon.

EvaluatorcollaborateswiththoseengagedinthechangeefforttodesignanevaluaJonprocessthatmatchesphilosophicallywithanorganizaJon’sprinciplesandobjecJves.

EvaluaJonresultsinopinionofsuccessorfailure,whichcreatesanxietyinthoseevaluated.

EvaluaJonsupportsongoinglearning.

Source:Gamble,Jamie.2008.Adevelopmentalevalua(onprimer.Montreal:TheJ.W.McConnellFamilyFoundaJon.P.62

B. Concepts & language

•  Input, Output, Outcome, Impact

•  Simple, complicated, complex

•  Contribution; Attribution

•  Theory or Trajectory of Change

•  Logic frameworks; Outcome mapping

B. Concepts & language

•  Input, Output, Outcome, Impact

•  Simple, complicated, complex

•  Contribution; Attribution

•  Theory or Trajectory of Change

•  Logic frameworks; Outcome mapping

Theoryofchange

Theoryofchange

Theoryofchange

Source:Guidedexample:ProjectSuperwoman.2003www.theoryofchange.org

B. Concepts & language

•  Input, Output, Outcome, Impact

•  Simple, complicated, complex

•  Contribution; Attribution

•  Theory or Trajectory of Change

•  Logic frameworks Vs Outcome mapping

Results-orientedlogicalframework

C. Evaluation users, uses & readiness

InstrucJons1.  Selectaproject2.  RevisitReadiness3.  ListpotenJalevalua&onowners/users4.  Listpossibleevalua&onuses/purposes5.  Draisomekeyevalua&onques&ons

UJlizaJonfocusedevaluaJonis…

Users

EvaluaJonfocusUses

Atfirst,UFEfeelslikeamaze….

But,an‘overview’makesitclear….

BasicU-FEroles

Evaluator

Facilitatestheprocess,organizesmeeJngs,getsthePIUinvolvedindesigningtheevaluaJon,collectsandanalyzesdata,writesreports,facilitatesuseoffindings.

IntendedusersParJcipateacJvelyintheevaluaJonprocess,definepurposeandusesoftheevaluaJon,helpcollectandinterpretdata,usefindingsaccordingtodefineduses.

Interestedaudience

Mayreadandcommentonreportorprocess;indirectbeneficiariesofthework

AdaptedJenniePhillips,2014

C. Evaluation users, uses & readiness

USERS-definiJonembeddedinpowerdynamics•  Nameandrole(notposiJon)•  Empowered,commiOed•  Available•  Interestedinlearning

ChoosingtheUSER:Tips

USERs•  PeoplewhowilluseandapplythefindingsoftheevaluaJon.PaOon(2008)suggeststhefollowingprofiler:1.  Interested2.  Knowledgeable3.  Openminded4.  Representsanimportantinterestgroup5.  Hascredibility6.  Teachable7.  AvailableforongoinginteracJonthroughout

theevaluaJonprocess.

C. Evaluation users, uses & readiness

USES

Usesorpurposes:•  Programimprovementanddesign•  Accountabilityandoutputs•  Outcomes,knowledgegenerated,measurableresults•  Methods&pracJces•  Engagementapproach•  Financialviability,cost-benefit• 

Taking the time to reflect on the purpose, �reveals the opportunity to learn

Why this evaluation, what for?

FocusingtheEvaluaJon

•  WhataretheKeyEvaluaJonQuesJons(KEQ)thattheUSERSareinterestedin?

•  WhataretheintendedUSES?HowwilltheUSERSUSEthefindingsfromtheevaluaJon?

Users

EvaluaJonfocusUses

C. Evaluation users, uses & readiness

USES KEQs

Toverifyprogramoutputs

ToinformorganizaJonalstrategy

TurningpurposesintoKeyEvalua&onQues&ons(focusingtheevaluaJon):•  Avoidyes/noquesJons(“Towhatextent…”)•  Makethemspecificenoughtoguideyourinquiry•  Broadenoughthattheycannotbebrokendown•  Data(quanJtaJve,qualitaJve)canbebroughttobear•  Leadtoauseful,credibleevaluaJon•  Therearen’ttoomany(2-4)

hOp://evaluaJoninpracJce.files.wordpress.com/2010/05/revsz_ufe_steps4-6.pdf

C. Evaluation users, uses & readiness

USES KEQs Data,Evidence

Toverifyprogramoutputs

TowhatextentwerenewlinkagesbuiltbetweencommuniJesandservicesproviders?

Whatwasthetotalgrossincomegeneratedaieroneseason?

ToinformorganizaJonalstrategy

Towhatextentwastheexpertpanelengagedintrouble-shooJng?

C. Evaluation users, uses & readiness

USES KEQs Data,Evidence Tools

Toverifyprogramoutputs

TowhatextentwerenewlinkagesbuiltbetweencommuniJesandservicesproviders?

Numberandintensityoflinkagesatstartand12monthslater.

Base-linesurveyquesJons4and6.InterviewquesJona,f.VideoobservaJons.

Whatwasthetotalgrossincomegeneratedaieroneseason?

ToinformorganizaJonalstrategy

Towhatextentwastheexpertpanelengagedintrouble-shooJng?

C. Example of uses (research network)

1)Tovalidateproject-nameoutcomesforfunder;2)Toprovideevidenceof,anddocumentoutcomes/relevanceforotherfunders;3)ToInformtheinternalproject’stransiJon(leadership,skills,funding);4)Tochartoutcomepathways;5)TochartcommunicaJonstrategyoutcomes;and,6)ToinformorganizaJonalsustainability.

C. Example of KEQs (research network)

1)Towhatextentdidproject-nameinfluencepolicyatnaJonal,regionalandinternaJonallevels[outputs/outcomes]?

2)Howdoesproject-nameuseresearchtochallengedominantinternaJonalpolicyandpracJce?

3)Towhatextentdidproject-namebuildcapacity(generaJonanduJlizaJonoflocalknowledge)among:i)individualresearchers;ii)universiJes;andiii)policymakers&regulators?

4)Towhatextentdidproject-name’snetworkingmodaliJessupporttheiroutcomes(policyandcapacitybuilding)?a)project-namenetwork(peernetwork)versusb)CPRconference(peerandmentoringreviewprocess–individuals)

/Cont’d.

C. Example of KEQs (research network)

5.a)Whatapproachdidproject-nameusetoposiJonitselftoinfluencepolicyindifferentcountriesandthroughwhatprocessesandsystems(tellthestory)[outcomes/process]?

5.b)HoweffecJvehasproject-namebeeninunderstandingthelocalcontextortakingupopportuniJestoinfluencelocaldiscourse?

6.a)WhatwouldbethefundingmodaliJesthatbestfitproject-name’svalues?

6.b)Whataretheconsequences,incenJves,disincenJvesforproject-nameasanindependentpublicinterestresearchenJtyinpursuingahybridfundingmodel?

•  Nazdeek: hOp://nazdeek.org/•  ICAAD: hOp://www.icaadglobal.org/•  Pajhra: hOp://www.pajhra.org•  ISIF: hOp://isif.asia/•  IDRC: hOp://www.idrc.ca/EN/Pages/default.aspx

•  Assamhashighestmaternalmortalityrates•  ExisJngfaciliJesareseverelyunderequippedandunderstaffed•  MostwomenbelongtotheAdivasicommunityandhavehighrates

ofmaternalandinfantmortality.•  MostteagardenworkersareisolatedandhaveliOleaccesstolegal

andadvocacyservices

• Haveyougotbuy-inandcommitmentfromprogramleadersanddonortodoUFE?DoyouhavetheresourcesandJmetodoUFE?

ProjectReadiness

Seniormanagement

commiOedJmeandhumanresources

• DoestheevaluatorhavetheskillstodoUFE?Canhe/shebeafacilitatorandnotbejudgmental?Doeshe/shehavetheJme?Iswillingtolearn?

EvaluatorReadiness

‘internal’evaluator

willingtolearnandreceivementoring

Toinformthegovernmentbeforebudgetsession;to

disclosetogovt.aboutmappinghealthrightsviolaJons;Fieldstaffafraidtodisclosetheir

acJvism

• Arethereanybarriers–people,resources,poliJcalorculturalenvironmentinwhichevaluaJonisbeingconducted?

• ArethereanydeadlinesthatUFEmustmanage?

SituaJonalAssessment

• WHOisinterestedintheevaluaJonandwillusethefindings:Bespecific.Engagetherightperson

IdenJfyPrimaryUsers

•  IdenJfytheUSE–isitforprogramimprovement,tomakemajordecisions,togenerateknowledgeortounderstandtheprocess

IdenJficaJonofUSES

Ini&alUsersandUses FinalUsersandUses1)LocalacJvistsandactors(alsoNazdeek,PAJHRA–tocollectdatatoadvocateandliJgategovt.

ProjectCoordinator,PAJHRA

2)Communitymembersandlocalleaders,tocreateawarenessforaccesstomaternalhealthservices

USE:TolearnaboutimplementaJontoimprovetheprogram,sothatPajhraandNazdeekarebeOerinformedtosourcethenextroundoffunding.

3)LocalGovernmentandCourtsforpolicylevelchoicestoimprovehealthservices

• HighpriorityquesJonsareaddressed• Intendeduseofanswersisclear

Focusingthe

EvaluaJon

• SelecJonofmethodsdependsonKEQanddataneededforUSE.Oienmentorsstepinforcoachinganddesignsupport.

EvaluaJonDesign

• Beforedataarecollected,asimulaJonusing‘dummy’datatorefineKEQandverifythatdatawillleadtousablefindings.

SimulaJonofUSE

Validate&adjustoveralldesignbeforeit’stoolate!

• PIUarealwaysinformedandinvolved

• PIUsareinvolvedwhichincreasestheirunderstandingoffindingsandaddstosenseofownershipandcommitmenttouJlizaJon

• Usehastobefacilitated,doesnotjusthappennaturally• ThisstepiscentraltoUFE;allocateJmeandresources

DataCollecJon

DataAnalysis

FacilitaJonofUSE

FocusGroupDiscussion

Datareview

Profile–educaJon,caste/class,occupaJon,locaJon,community

InterviewsofHighAchievers

QuesJonnaireWearecollecJngthedata

WeareabletoanswertheKEQ

Primaryusersareusingthedata.

D. Utilization-focused evaluation

UJlizaJon-focusedevaluaJonemphasizesUSEbyprimaryUSERS;theapproachemphasizeslearning

UFEallowsforprojectstobecomethedriversoftheirownevaluaJon;thisensuresownershipoverprocess

andfindingsUFEisadecision-makingframeworkforenhancingthe

uJlityandactualuseofevaluaJons.(PaOon,2008a:slide9)

D. Premises of UFE

•  NoevaluaJonshouldgoforwardunlessandunJlthereareprimaryintendeduserswhowillusetheinformaJonthatcanbeproduced.

•  Primaryintendedusersareinvolvedintheprocess.•  EvaluaJonispartofini&alprogramdesign.Theprimary

intendeduserswantinformaJontohelpansweraquesJonorquesJons.

•  Evaluator’sroleistohelpintendedusersclarifytheirpurposeandobjec&ves.

•  Makeimplica&onsforuseapartofeverydecisionthroughouttheevaluaJon–itisthedrivingforceoftheevaluaJonprocess.(PaOon,2008a:slide20)

• Evaluatorsmakethemselvestheprimarydecisionmakers.• IdenJfyvague,passiveaudiencesasusers.• TargeJngorganiza&onsasusers.• Focusingondecisionsinsteadofdecisionmakers.• Assumingtheevalua&on’sfunderistheprimarystakeholder.

UFEtrapsortemptaJons(1of2)

• Wai&ngun&lthefindingsareintoidenJfyintendedusersandintendeduses.• TakingastanceofstandingabovethemessinessofpeopleandpoliJcs.• Beingco-optedbypowerfulstakeholders.• IdenJfyingprimaryintendedusersbutnotinvolvingthemmeaningfully.

(PaOon,2008,adaptedfromp.90-Ch.3)

UFEtrapsortemptaJons(2of2)

1.  Project/networkreadinessassessment.2.  Evaluatorreadinessandcapabilityassessment.3.  IdenJficaJonofprimaryintendedusers.4.  SituaJonalanalysis.5.  IdenJficaJonofprimaryintendeduses.6.  FocusingonevaluaJon.7.  EvaluaJondesign.8.  SimulaJonofuse.9.  DatacollecJon.10.  Dataanalysis.11.  Facilitateuse.12.  Meta-evaluaJon.

UFEin12steps

D. Steps of UFE

LEGEND

evaluaJonandcommunicaJoninpracJce.net

SonalZaverisonalzaveri@gmail.com

UFEmodulesandwebinars

Recommended reference website

http://betterevaluation.org/

Recommended reference website

http://betterevaluation.org/

THANKYOU!HOPEYOUWILLUSEUFEINYOUR

WORK!