THINKINGEVALUATIVELYIN
PEACEBUILDINGDESIGN,
IMPLEMENTATIONANDMONITORING:
ThreeReflectingonPeacePractice(RPP)andDoNoHarm(DNH)-infusedoptionstostrengthentheeffectivenessofpeacebuildingstrategiesandprograms
DevelopedbyAnitaErnstorfer,IsabellaJean,andPeterWoodrow
ofCDACollaborativeLearningProjects,withinputsfromDianaChigasofTuftUniversity
November2016
Thisreportwasmadepossibleby
theCarnegieCorporationofNewYork
ResourceGuide
ThisresourcehasbeendevelopedbyCDACollaborativeLearningProjectsinpartnershipwiththePeacebuildingEvaluationConsortium(PEC)andwithfundingfromtheCarnegieCorporationofNewYork.
Particular thanks go to Cynthia ClappWincek (Alliance for Peacebuilding), LeslieWingender(Mercy Corps), Mark M. Rogers (independent consultant), and Kiely Barnard-Webster (CDACollaborativeLearningProjects)fortheirreview,andhelpfulsuggestions.
ThePeacebuildingEvaluationConsortium(PEC)
The Peacebuilding Evaluation Consortium (PEC) is a project of Alliance for Peacebuilding (AfP) inpartnership with CDA Collaborative Learning Projects, Mercy Corps and Search for Common Ground(SFCG).TheprojectisfundedbytheCarnegieCorporationofNewYork(CCNY)andisfield-wideefforttoaddresstheuniquechallengestomeasuringandlearningfrompeacebuildingprograms.ThePECconvenesdonors, scholars, policymakers, local and international practitioners, and evaluation experts in anunprecedentedopendialogue,exchange,andjointlearning.Itseekstoaddresstherootcausesofweakevaluationpracticesanddisincentivesforbetter learningbyfosteringfield-widechangethroughthreestrategic and reinforcing initiatives: 1) DevelopingMethodological Rigor; 2) Improving the Culture ofEvaluationandSharedLearning;and3)FosteringtheUseofEvidencetoInformPeacebuildingPolicy.
CDACollaborativeLearningProjects
CDA (www.cdacollaborative.org) is a US based non-profit organization committed to improving theeffectiveness of thosewhowork internationally to provide humanitarian assistance, engage in peacepractice,supportsustainabledevelopment,andconductcorporateoperations inasocially responsiblemanner.Ourdonorsandpartners supportCDAbecausewecombine rigorousanalysiswithpragmaticfield-levelworkanddeliverpracticaltoolstofieldstaffandpolicymakersalike.
CDA is known as a leader in peacebuilding effectiveness work. We help peace practitioners, andorganizationsimprovetherelevanceandaccountabilityofprogrammingthroughbettertoolsforconflictanalysis,programstrategy,design,andmonitoringandevaluation.CDAhasalsocontributedtoinfluentialpolicy guidance, suchas theOECD/DACguidanceonevaluating conflict preventionandpeacebuildingactivities.
TABLEOFCONTENTS
1.Introduction................................................................................................................................................11.1WhatisthisGuide—andwhatisitnot?.............................................................................................................11.2WhoshouldusetheGuide?................................................................................................................................11.3OriginsoftheGuide............................................................................................................................................21.4WhyistheOptionsGuideneeded–whyevaluativethinkinginpeacebuilding.................................................31.5WhenisthisGuideuseful?.................................................................................................................................6
2.EmergingCriteriaandProgrammingConceptsinPeacebuilding............................................................72.1BackgroundRPPandDNH...................................................................................................................................72.2KeylessonsfromCDA’sReflectingonPeacePracticeProgram(RPP)................................................................82.3KeylessonsfromCDA’sDoNoHarmProgram(DNH).......................................................................................14
3.Whatistherightoptionformyteam?Whatarethebasicsweneedtohaveinplacetobenefitfrom
anyoftheseoptions?....................................................................................................................................16
4.ProgramQualityAssessment(PQA)–criteriaforassessingpeacebuildingprogramquality............194.1Overview:PurposeofaProgramQualityAssessment......................................................................................194.2KeyelementsofaprogramqualityassessmentbasedonRPPcriteria............................................................19
5.EvaluabilityAssessment(EA)–areyoureadyforanevaluation?.........................................................275.1Overview–PurposeofanEvaluabilityAssessment..........................................................................................275.2KeyelementsofanevaluabilityassessmentbasedonRPPcriteria:.................................................................28
6.StrategyandProgramReflectionExercise...............................................................................................356.1Overview–PurposeofaStrategy/ProgramReflectionExercise......................................................................356.2KeystepsoftheStrategyandProgramReflectionExercisebasedonRPPcriteria..........................................35
7.AdaptivePeacebuildingProgrammingthroughEffectiveFeedbackLoops..........................................41
8.SummaryOverviewofOptionscoveredinthisGuide............................................................................43
9.Recommendationsforpractice................................................................................................................46
10.Bibliography.............................................................................................................................................50
11.Annexes:ToolsandHand-outs...............................................................................................................53Annex1:FiveBuildingBlockstowardsPeace.........................................................................................................53Annex2.1:RPPMatrix............................................................................................................................................54Annex2.2:RPPMatrix‘Plus’...................................................................................................................................55Annex3:RPPProgramPlanningChart...................................................................................................................56Annex4:DoNoHarmActionPatterns(ResourceTransferPatterns).....................................................................57Annex5:ToolforPeacebuildingQualityAssessment............................................................................................23Annex6:ChecklistforEvaluabilityAssessmentofPeacebuildingPrograms.........................................................29Annex7:CDACollaborativeLearningProcessMethodologyandSteps.................................................................33
GLOSSARY
AfP - AllianceforPeacebuilding
CDA - CDACollaborativeLearningProjects
DNH - DoNoHarm
DM&E - Design,Monitoring,andEvaluation
EA - EvaluabilityAssessment
FCAS - FragileandConflictAffectedStates
M&E - MonitoringandEvaluation
OECD - OrganizationforEconomicCooperationandDevelopment
PEC - PeacebuildingEvaluationConsortium
PQA - ProgramQualityAssessment
pwl - peacewritlittle
PWL - PeaceWritLarge
RPP - ReflectingonPeacePracticeProgram(CDA)
TOC - TheoryofChange
UN - UnitedNations
USIP - UnitedStatesInstituteofPeace
1
1.Introduction
1.1WhatisthisGuide—andwhatisitnot?
TheGuideputsforwardthreeoptions:(i)ProgramQualityAssessments, (ii)EvaluabilityAssessments,and(iii)StrategyandProgramReflectionExercisesusingfindingsandlessonsfromCDA’sReflectingonPeacePractice(RPP)andDoNoHarm(DNH)Programsascriteriaforeffectiveandrelevantpeacebuildingengagement.
ThisisnotaGuideonhowtoevaluatepeacebuildingefforts…itisabouthowtoapplyEvaluativeThinkingtopeacebuildingstrategiesandprogramming.
This resource responds to theneed forclearerevaluative thinkingandpractice duringpeacebuildingstrategyandprogramdesign,andimplementation,andstrongermonitoringmechanismstoimprovetheevaluability of peacebuilding initiatives. It providesconcrete guidance for practitioners on how toimplement different ‘evaluative’ options – short offormalevaluations.
ThisGuide does not provide guidance on planning orconducting an evaluation. Nor does it provide acomprehensive overview of other evaluationapproachesavailabletoaddressthequestionsraisedintheprocessesoutlinedbytheGuide.
TheGuide can be used during peacebuilding strategyandprogramdesign,duringprogramplanningandset-up,duringimplementation,andduringthepreparationfor a formal evaluation. It is also relevant forengagements that are ending, instead of orcomplementary to other assessment processes or aformalevaluation.
1.2WhoshouldusetheGuide?
ThisGuideisintendedforusebydifferentaudiences:
§ Peacebuildingpractitionersdesigningandimplementingprograms;
ThisGuideprovidestoolsandguidanceforintegratinggreaterEVALUATIVETHINKINGintodesign,implementationandmonitoringofPEACEBUILDINGinitiatives,toenhanceprogramquality,ensurethattheymaximizetheirpotentialforimpactandarepreparedforaformalevaluationwhenitisnecessary.
Evaluative Thinking | ongoing process ofquestioning,reflecting,learning,andmodifying.
“Critical thinking applied in the context ofevaluation, motivated by an attitude ofinquisitiveness and a belief in the value ofevidence, that involves identifying assumptions,posing thoughtful questions, pursuing deeperunderstandingthroughreflectionandperspectivetaking,andinformingdecisionsinpreparationforaction.”aea365.org/blog/tag/evaluative-thinking
Peacebuilding|thisGuideusesthefollowingtwointerpretationsof‘Peacebuilding’:
(1) “Direct work that intentionally focuses onaddressing the factors driving and mitigatingconflict”;and
(2)“Effortstocoordinateacomprehensive,multi-leveled, multi-sectoral strategy, includingdevelopment, humanitarian assistance,governance, security, justice and other sectorsthat may not use the term peacebuilding todescribethemselves.”
(Schirch2013,Glossary)
2
§ Peacebuildingmanagerswhosuperviseothersonprogramdesign,implementationandMonitoringandEvaluation(M&E),andwhoareinvolvedinstrategicprogrammingandfundingdecisions;
§ M&Eprofessionalsoperatinginconflict-affectedcontexts,andworkingonpeacebuildinginitiatives;§ Commissionersormanagersofevaluations;§ Evaluatorswillingtoexperimentwithalternativestoformalevaluationprocesses.
1.3OriginsoftheGuide
ProgramQuality Assessments (PQAs), Evaluability Assessments (EAs) and strategy evaluation1 alreadyexistinevaluationpractice,andhavebeenusedwidelyinnon-peacebuildingfields.Aspartofitsbroaderefforts to enhance program relevance and effectiveness within the peacebuilding field, CDA hasdevelopedaspecificapproachtoStrategyandProgramReflectionforpeacebuildingengagements,andadaptedPQAsandEAstothespecificneedsofpeacepractitioners.Forthispurpose,lessonsfromCDA’sReflectingonPeacePracticeandDoNoHarmProgramshavebeen integratedandusedascriteria foreffectivepeacebuildingwork.2 Sections4,5and6willdescribePQAs,EAs,andStrategyandProgramReflectionExercisesinmoredetail.
TheseprocessesweredevelopedfollowingrecommendationsofastudybyCheyanneScharbatke-Churchin2011,inwhichsheexploredhow“ReflectingonPeacePractice”(RPP)conceptsandtoolswerebeingusedinevaluation.3ShefoundthatRPPconceptswerebeingusedinmanypeacebuildingprograms,oftenimplicitly, as de-facto ‘standards’ for understanding peacebuilding results, as a frame of inquiry forevaluations,asameanstoassessrelevance,aswellastostrengthenprogramdesignduringthedesignandimplementationphase.4ThepartsofRPPthatseemtobeusedthemostweretheRPPmatrix,keydrivingfactorsofconflict,thebuildingblocksforpeace/criteriaofeffectiveness,linkages,determiningadd-uptoPeaceWritLarge,andsystemsapproachestoconflictanalysis.TheseapproacheswillbeexplainedinmoredetailinSections3-6.
The Guide will use lessons and findings from RPP as criteria for
assessing peacebuilding program quality, with the goal to worktowardsgreaterimpact.ThisalsoincludesinsightsfromCDA’sworkon systems approaches to conflict analysis, peacebuilding programdesign,andM&E.Inaddition,toalesscomprehensiveextentbutinacomplementaryfashion,insightsandlessonsfromCDA’sDoNoHarm(DNH)programandongoingworkonconflict-sensitivitywillalsobeusedasprinciplesforeffectiveprogramming.5
OnemightlegitimatelyaskwhywechosetouseRPPandDNHcriteriaandtools,ratherthanothernormsforpeacebuildingprogrammingandconflictsensitivity.Onthepeacebuildingside,wefindthattherearefew,ifany,widelyacceptednormsforprogramdesignandeffectivenessbeyondbroadgeneralities.AsScharbatke-Churchfound,manypolicymakersandpractitionerswerealreadysuccessfullyusingRPPasa
1See(PatriziandPatton2011),forexample,onstrategyevaluation2See(Reimann,ChigasandWoodrow2012)3Scharbatke-Church,TheUseofReflectingonPeacePractice(RPP)inPeacebuildingEvaluation.Review&Recommendations4TherecommendationfromScharbatke-Church’sreporttoCDAwasto(i)buildonexistingworkinRPPtosupportandstrengthenthequalityofpeacebuildinginterventiondesigninawaythatsetsthestageformoreeffectiveevaluation,and(ii)forCDAtouseitslessonsfromRPPtocontributetotheunderstandingofhowtoapplysomeoftheOECD-DACevaluationcriteriaforconflictpreventionandpeacebuildingwork.5Itisoften–wrongly–assumedthatpeacebuildingworkisautomaticallyconflict-sensitive.Lessonsfrompracticeshowthatthisisnotnecessarilythecaseandthatpeacebuildingprogrammingneedstoapplythesamerigorofconflict-sensitivityassessmentas,forexample,developmentandhumanitarianprogramming.Atthesametime,thereisalsoasignificantlevelofconfusionbetween‘conflict-sensitiveprogramming’and‘peacebuildingprogramming’.Ausefulresourceinthisregardis(WoodrowandChigas2009)
Moreinformationaboutpeacebuildingevaluationandwhichevaluationoptions:
® dmeforpeace.org/introduction-to-the-field-Guide
® dmeforpeace.org/evaluation-planning
*ThePECOnlineFieldGuidetoPeacebuildingEvaluationonDM&E
3
setofreasonablenorms—sowecouldbuildonandenhancethatapplication.And,whilethereareothermodels for conflict sensitivity (such as PCIA – Peace and Conflict Impact Assessment), they do notcontradictDNH.
Section2ofthisGuidewillintroducethemainapproachesandtoolsofbothRPPandDNHasafoundationtounderstandhowRPPandDNHapproachesandlessonshavebeenintegratedintothethreeoptionspresentedinthisresource.
TheGuidebringstogetherCDA’sexperienceworkingwithvariousprogramteamsandprogrampartnersontheapplicationofRPP,andDNHtostrengthenpeacebuildingstrategyandprogrammingwithaviewto enhance relevance, effectiveness, and conflict-sensitive programming. TheGuide alsobuilds on findings from two RPP and DNH infused Program Quality Assessments and Evaluability Assessmentsconductedin2013and2014withfourpartnerorganizations.6Inaddition,thecollectiveexperienceofthePeacebuildingEvaluationConsortiumhasbeenleveragedtoproducethisGuide.
1.4WhyistheOptionsGuideneeded–whyevaluativethinkinginpeacebuilding
PeacebuildingandEvaluation–wheredowestand?
Thepeacebuildingfieldhasbeenstrugglingtoagreeonandadheretouniversallyrecognizedprinciplesand standards for quality and accountability (e.g., for conflict analysis, theories of change, clearlyarticulated goals that are relevant from a peacebuilding perspective). In other development orhumanitariansectorstherearestandardssuchastheSPHEREstandardsinthehumanitariansectorortheINEEminimumstandards for education in emergencies. Likewise, theevaluation communityoperateswith clear standards and principles.7On the other hand, definitions ofwhat constitutes relevant andeffectivepeacebuildingvariesgreatlyacrossorganizations.
Atthesametime,overthepasttenyears,thefieldofpeacebuildingevaluationhassignificantlymatured,anddonorshaveincreasedtheirpressuretoshowconcreteresults.ArangeofGuidelines,frameworksandtoolkitshavebeendevelopedbypeacebuildingandevaluationorganizationsandpractitioners.TheOECDDevelopmentAssistanceCommittee(DAC)Guidelinesontheevaluationofconflictpreventionandpeacebuildingactivities(OECD/DAC20128)arenowonesetofstandardsinthefield.CDAprovidedkeycontributionstotheOECD/DACGuidelines,basedonfindingsfrombothitsReflectingonPeacePractice(RPP)aswellasDoNoHarm(DNH)collaborativelearningprograms.
Increasingly, development and peacebuilding organizations are making a conscious effort toinstitutionalizepeacebuildingprogramdesignandrelatedmonitoringandevaluation(M&E)approachesintheirwork.Thisincludesprominentinternationalnon-governmentalorganizationssuchasSearchforCommonGround,9MercyCorps,CARE,Saferworld,andCatholicReliefServices.Multilateralorganizations
6ThosewereaPQAfacilitatedwithInternationalAlertintheSouthCaucasus(facilitator:IsabellaJean/CDA),aPQAfacilitatedwithInterpeaceinMali(facilitator:AnitaErnstorfer/CDA),anEAfacilitatedwithNorwegianChurchAidinAfghanistan(facilitator:MarkRogers/independentconsultant),andanEAfacilitatedwithWorldVisioninSriLanka(facilitator:CordulaReimann/CoreConsulting&Training).7See,forexample,“AmericanEvaluationAssociationGuidingPrinciplesForEvaluators,”AmericanEvaluationAssociation,http://www.eval.org/p/cm/ld/fid=518InadditiontotheOECD/DACevaluationcriteriaforallfields(relevance,effectiveness,efficiency,impact,sustainability,andcoherence),(OECD/DAC2012)recommendsthreeoptionalcriteriatobeusedforpeacebuilding,specifically:linkages,coverageandconsistencywithvalues/conflictsensitivity.9See“DM&EforPeacePortal,”SearchForCommonGround(SFCG),http://www.dmeforpeace.org/
4
havealsomadeeffortstobemorerigorousaboutstrategydesignandM&E,suchastheUNPeacebuildingFund.
Despiteallthepositivedevelopmentsmentionedabove,thepeacebuildingfieldhasbeenstrugglingtoapplyevaluationsasasystematicprofessionalpractice.In2011,areportbytheAllianceforPeacebuilding(AfP)notedthat“[...]thepeacebuildingfieldseemstohavereachedafrustratinglylongplateauintheuse,understanding,andapplicationofevaluation.Asaresult,mostpeacebuildingfundersandimplementersexpressdissatisfactionat thecurrent stateofevaluation.”10Progressandpractical learninghavebeenslow.Thereisstillnowidelyacceptedmethodologicalagreementabouthowbesttoconductevaluationsincomplexandconflict-affectedcontexts(Paffenholz2011).11
Four challenges are worth highlighting in this regard:
1.Thefirstchallengeishowtomeasuretheeffectsofmicro-levelinterventionsonmacro-levelconflict
dynamics.Thisrefersnotonlytothechallengesofattribution,butalsotothehighlycomplexnon-linearprocessesofsocialchangewhichcannotbecapturedbylinearcause-effectlogic(e.g.,asmanifestedinsomeofthestandardlogicalframeworksusedinM&Esystems).Forthatreason,systemsapproachestopeacebuilding and peacebuilding evaluation are capturing increased attention, but how to use thempracticallyandsystematicallyfromamonitoringandevaluationperspectiveisyetunderexplored.
2.Thesecondchallengeisthatmanypeacebuildinginitiativesarenotbasedonclearlydefinedstrategy
and design principles or criteria, with either limited or no conflict analysis, unarticulated theories ofchange,or ‘fuzzy’peacebuildinggoals.Thismakesevaluationextremelychallengingandhighlights theneedtoapplyanevaluativeapproachfromtheinitialstagesofprogramdesign,throughimplementation,andM&E–notsimplyattheendofaproject.
3.Thethirdchallengeisthatthereisnotyetawell-established‘cultureof evaluation’within the peacebuilding field,which often results invery limited M&E systems and capacities. Many peacebuildingpractitionersdon’thaveexpertisewithM&E,andmanyM&Eexpertsdon’t have practical experience with peacebuilding. Manypeacebuildershaveresistedasystematicapplicationofrigorousandprofessionalresultsmeasurementtoolsandframeworkstotheirwork.Many peacebuilding practitioners claim that it is too difficult tomeasureimpact,giventhecomplexityoftheconflictfactorsatstake,highlydynamicandquicklychangingenvironments,andthelong-termnature of conflict transformation and peace efforts.12 Severalorganizationshavemadeprogress in strengtheningmore integratedDesign,M&E and Learning capacities; however, often capacities areconcentratedinheadquartersandarenotreplicatedatthesamelevelincountryandfieldoffices.Atthesametime,thecombinedpressuresof scarce funding, the requirement to demonstrate results and the10Kawano-Chiu,StartingontheSamePage:ALessonsReportfromthePeacebuildingEvaluationProject,811Oneofthemostrecentcontributionsinthisregardis(Andersen,Bull,andKennedy-Chouane2014).Seealso(CorlazzoliandWhite2011)12Thisisoneoffive‘mythsandmisconceptions’identifiedbytheAfPreport(Kawano-Chiu2011,9andfollowing.)Theotherfourare:“Staffincountryofficesmustbetrainedsocialscientists”;“Theprimarypurposeofevaluationsistohighlightflawsandfaultsandassesswhenaprogramisa‘success’or‘failure’”;“Theexpectationisthatnearlyallprojectswillbe‘successful”;and“Countervailingforcesagainstgoodevaluationpracticesaretooentrenchedtochange”.
Moreinformationaboutstandardsinevaluation:
® dmeforpeace.org/introduction-to-evaluationunder“ArethereQualityStandardsforEvaluations?”
Resourcesonevaluatingpeacebuildingprogramminginaparticularsector:
® dmeforpeace.org/evaluation,under“IamInterestedinEvaluatingPeacebuildingProgramminginaParticularSector.ArethereSpecialConsiderationsandTools?”
5
need to establish peacebuilding as a legitimate field have led to a tendency among peacebuildingprogramsto‘over-claim’results(achallengethatisnotuniquetothepeacebuildingfield).13
4.Commitmenttoactuponassessmentandevaluationresults,andestablishingacultureoflearningisafourthchallenge.Thisisclearlynotuniquetothepeacebuildingfield.Iftheobjectives,unit(s)ofanalysis,andpurposesofevaluationsarenotclearlyarticulated,itdecreasesthelikelihoodthatevaluationswillleadtoachangeinpracticeandcontributetotheuptakeoflearninginanorganization.Often,localstaffarelefttotheirowndeviceswithlittlecontinuingexternalsupporttoimplementtherecommendationsof evaluation reports.14 Most literature on (peacebuilding) evaluation is clear about the fact thatevaluation should never be “[...] an end in itself. It should be a mechanism that contributes toaccountability and learning at a variety of levels: project team, office, organization or peacebuildingfield.”15 Inpractice,however, thedonoraccountabilityaspectofevaluationshas receivedmuchmoreattentionthanthelearningandprogramqualityimprovementaspects.Frequently,evaluationprocessesarenotdesignedinawaythatsupportsaconsciousprocessoflearningandadaptation.
Whydoweneeddifferentevaluativeoptionstostrengthenpeacebuildingstrategy,designandimplementation?
In timeswhenthere is increasedrecognitionof theneedformoreadaptiveandflexibleprogrammingapproaches in highly complex and conflict-affected contexts (by donors and policymakers aswell asprogram partners and local stakeholders), the application of different evaluative options becomesincreasinglyrelevant.Theycanprovideafoundationfornimblerandmoreadaptabledecision-making,coursecorrectiononprogrammingdirections,anddifferentoptionsforengagement.
CDAhasfoundseveralissuesinthepeacebuildingfieldthattheuseoftheseapproachesaddresses:
§ Manypeacebuildingprogramsarenotreadyforformalevaluations—eitherbecausetheywerenotdesignedwith evaluation criteria inmind or because they are not prepared in otherways.Whiledonorsoftenimposearequirementthatprogramsperforman“evaluation”,itisnotalwaysclearwhatstandardtheyareapplyingandwhatthepurposeofsuchanevaluationwouldbe—norareadequatefunds provided for a full evaluation.Many donors (and implementing organizations)will bewell-served by other evaluative processes that meet more modest objectives for program qualityimprovement.
§ Conduciveenvironment:Insomesituations,theconditionsofconflict,violenceandinsecurityarenotconducive to robust evaluation per the highest accepted standards of the American EvaluationAssociationorotherinternationalbodies,suchastheOECD/DAC.Datamaynotbeavailable.Accesstocertainareasmaybelimited.Posingcertaintypesofkeyquestionsmayexacerbateconflictorputprogramstaffindanger.
§ Inmost cases, program teams also have significantM&E capacity development needswhich areusuallynotmetandcannotbemetbyevaluationsalone.
Basedontheabove,thepeacebuildingandevaluationcommunityhavebeenexploringarangeofnewapproaches to evaluation.16 This guidance builds on these developments to focus on guidance for
13Scharbatke-Church,PeacebuildingEvaluation:NotYetAllItCouldBe,BerghofHandbookforConflictTransformation.SectionII,476
14ActionAsia,ExaminingRPPasAToolforEvaluation-TheActionAsiaExperience,16
15Scharbatke-Church,PeacebuildingEvaluation:NotYetAllItCouldBe,BerghofHandbookforConflictTransformation.SectionII,471
16Theseinclude“mostsignificantchange”technique(DaviesandDart2005),developmentalevaluation(QuinnPatton2011),andoutcomemapping.
6
programsthatseektoreviewandstrengthenprogramqualityandstrategy, thatarenotyetreadyforevaluation,orwhereformalevaluationmaynotbeappropriateordesired.
1.5WhenisthisGuideuseful?
TheGuideisusefulforbothstrategyandprogramdesign:itcanbeusedtoreviewandassessmacro-levelpeacebuilding strategies, involving various projects and programs within one organization or acrossagencies,aswellasattheindividualprojectandprogramlevel.
Therefore,thisGuideisintendedtoberelevantforafairlybroadrangeofinitiativespursuingeitheroftheabove,includingshorter-term,medium-term,aswellaslonger-terminitiatives.
TheStagesofConflictFigure(CDA2016)impliesafairlysmoothsetofstagesthatmoveinexorablytowards“stablepeace.”Therealityisquitedifferent,asprocessesstartandstop,returntoearlierphases(suchasviolence),makesomeprogressandstall,andsoforth.
Peacebuildingprogramsmaybeorientedtowardsanyofthesestagesormayaccompanyapeaceprocessthroughseveralphases.
The threeoptionsoffered in thisGuideare intended tohelppractitionersanddecisionmakers locatethemselvesinarangeofpeaceeffortsandachievecontinuousimprovementineffectivenesstowardsthelargergoalofstablepeace—whatRPPcalls“PeaceWritLarge.”Therefore,theGuideisexpectedtobehelpfulinarangeofdifferentconflict-affectedcontextsandnotlimitedtoaspecific‘conflictphase’.
…theGuideisexpectedtobehelpful
inarangeofdifferentconflict-
affectedcontextsandnotlimitedtoa
specific‘conflictphase’.
Figure:StagesofConflict(CDACollaborativeLearningProjects2016)
ManifestConflict
Pre-ConflictArguments,disagreements,culturalandstructuralviolence
Post-ConflictPoliticalsettlement,peaceagreements
LatentConflictAggressivepositioning,targetedattacks,
culturalandstructuralviolence
PoliticalInstability,frequentphysicalviolence,'civilwar'
StablePeace
7
2.EmergingCriteriaandProgrammingConceptsinPeacebuilding
AsnoteveryreaderofthisresourcewillbefamiliarwiththecoreconceptsofRPPandDNH,thissectionprovidesabrief introductiontothemainconceptsandtools–thefoundationofwhyandhowweareusingRPPandDNHascriteriaforPQAs,EAs,andstrategyandprogramreflections.Foramorein-depthoverviewofeachoftheseapproaches,pleaserefertothebibliography.
2.1BackgroundRPPandDNH
ReflectingonPeacePractice
Launched in1999,CDA’sReflectingonPeacePracticeProgram (RPP)offeredpractical answers to thecorequestionsaboutrelevanceandeffectivenessinthepeacebuildingfield.RelevanceandEffectivenessarealsotwoofthekeyOECD/DACcriteriaforpeacebuildingevaluation,anddefinedasfollows:
CDAbuildsonthisdefinitionofeffectivenessbyintroducingadistinctionbetweenprogrameffectivenessandpeaceeffectiveness:
CDA’sworkonpeacebuilding effectivenessbegan in 1999with the launchof theReflectingonPeacePracticeProgram(RPP).RPPposedasimple,albeitcomplexquestion:Whatworks-andwhatdoesn’t
work – in peacebuilding? RPPworkedwith hundreds of agencies and individuals, and conducted 26
Relevance|assessestheextenttowhichtheobjectivesandactivitiesoftheintervention(s)respondtotheneedsofthepeacebuildingprocess,i.e.whethertheyaddressthekeydrivingfactorsofconflictrevealedthroughaconflictanalysis.Relevancelinkstheoutcomesoftheconflictanalysiswiththeintervention’sobjectives,althoughtherelevanceoftheinterventionmightchangeovertimeascircumstanceschange.Understandingrelevancemayalsoinvolveanassessmentoftheextenttowhichaninterventiontiesinwithoverallstrategiesandpolicyframeworksofthecountryorexternalpartners.Differentconflictgroupsoractorsmayhavedifferentperspectivesontherelevanceofaninterventionanditsresults.(OECD/DAC,2012,56)
Effectiveness|isusedtoevaluatewhetheraninterventionhasmetitsintendedobjectiveswithrespecttoitsimmediatepeacebuildingenvironment,orislikelytodoso.Thekeytoevaluatingeffectiveness–andthusthelinkagebetweenoutputs,outcomesandimpacts–isfindingouttowhatdegreetheenvisagedresultshavebeenachievedandnotingchangesthattheinterventionhasinitiatedortowhichithascontributed.[…]Itisimportanttodrawadistinctionbetweentwokindsofresults.Oneis“programmeeffectiveness”,i.e.towhatextenttheprogrammeachieveditsstatedobjective.Theotheris–iftheprogrammemetitsobjectivesorgoal–theimmediateorsecondaryoutcomesastheyrelatetopeacebuildingandconflictdynamicsidentifiedintheanalysis.(OECD/DAC2012,57)
ProgramEffectiveness|focusesonassessingwhetheraspecificprogramisachievingitsintendedgoalsinaneffectivemanner.Thiskindofevaluationaskswhethertheprogramisfulfillingitsgoalsandissuccessfulonitsownterms.
PeaceEffectiveness|askswhether,inmeetingspecificgoals,theprogrammakesacontributiontoPeaceWritLargeandhasapositiveeffectbyreducingkeydrivingfactorsofconflict.Thisrequiresassessingchangesintheoverallenvironmentthatmayormaynotresultdirectlyfromtheprogram.InmostinstancesthisrequiresidentifyingthecontributionofthespecificprogramtoPWL,ratherthanseekingclearattributionofimpactsfromdiscretepeaceinitiatives.ImpactsatthelevelofPWLtypicallycannotbeachievedbysingleactivitiesandprojects,butratherarecumulative,resultingfrommanydifferenteffortshappeningsimultaneously,especiallywhentheseeffortsaredeliberatelydesignedtocomplementoneanother.Strategiclinkagesamongeffortsinasinglecontextarethereforecritical.(CDACollaborativeLearningProjects2013,28)
8
peacebuildingcasestudiesthroughouttheworldtogleanlessonsapplicableacrossconflictcontextsanddevelopuser-friendlytoolkits.TheresultinglessonsarepresentedinConfrontingWar:CriticalLessonsforPeacePractitioners(2003).Between2007and2009,CDAundertook16casestudiesthatinvestigatedthecumulativeimpactsofpeacebuildingprograms inEurope,Africa,MiddleEast,AsiaandLatinAmerica.Findingsfromthesehavebeenpublishedinissuepapers,selectpublications.17
Buildingonthiscumulativeimpactwork,CDAhasdevelopedspecificapproachestosystemsthinkingandpeacebuilding,includingsystemicconflictanalysis,systemsmapping,andtheidentificationofleverage
pointsforchangeasanothermeansofexpandingthepeacebuildingeffectivenessfield.Theexperienceand lessons gained through the years of RPP’s operation are the foundation of CDA’s currentPeacebuildingEffectivenesspracticearea,whichcontinuestopromotelearninginthisfield,boththroughadvisoryservicesandthroughongoingcollaborativelearningefforts.18
DoNoHarm
CDA’sworkonconflictsensitivitybeganin1993withthelaunchoftheLocalCapacitiesforPeaceProject,whichcametobeknownastheDoNoHarmProgram.Overtheyears,DoNoHarmhasinvolvedhundredsofaidagencies,andmorethan1000aidpractitionersfromallovertheworldinitscollaborativelearningprocesses.TheresultinglessonsarepresentedinDoNoHarm:HowAidCanSupportPeace–OrWar,aswell asmany other publications andGuides available on the CDAwebsite. Today, “DoNo Harm” isrelevantinpracticeasaprincipleandasatool(theDNHFramework),andisusedbymanypractitionerstodescribetheirworkonconflict-sensitivity.
ThepurposeofthefollowingsectionistointroducebasicRPPandDNHapproachesandtools,whichareused as criteria for program quality assessments, evaluability assessments, as well as strategy andprogramreflectionexercisesashighlightedinthisGuide.
2.2KeylessonsfromCDA’sReflectingonPeacePracticeProgram(RPP)
CDAhasdistilledthefollowingkeylessonsthroughthevariousphasesofpracticallearningfromRPP:
I. PeacebuildingprogramsshouldbeaccountabletoPeaceWritLarge;II. Conflictanalysisiscrucial.Goodconflictanalysisshould:
a. identifyKeyFactorsandKeyActorsvis-à-vispeace/conflict,b. identifytheRelationshipsandDynamicsamongthem,andc. clarifypointsofpossibleintervention.
III. AnalysismustbelinkedtoprogrammingandtoPeaceWritLargethroughastrongtheoryofchange;
IV. ProgramsmustreachtheSocio-PoliticalLevelinordertoaffectPeaceWritLarge;V. “MorePeople”workmustengage“KeyPeople”andviceversa;VI. Itisimportanttoengagethehard-to-reach;VII. ItispossibletoassesstheimpactofprogramsonPeaceWritLargeiftheyarebasedonconflict
analysis,strongtheoriesofchange,androbustprogramgoals.
17Formoreinformation,pleasevisit:http://cdacollaborative.org/18FormoreinformationaboutCDA’sworkand/orRPP,pleasevisit:http://cdacollaborative.org/
9
KeyLessonI:PeacebuildingprogramsshouldbeaccountabletoPeaceWritLarge
PeaceWritLarge(PWL)isconcernedwiththe“biggerpicture”ofaconflict.This“biggerpicture”referstotheoverallsocio-politicalconditionsinagivencontext.Itcaninvolvenationallevelconflictdynamics(orin some contexts, sub-national or regional dynamics). Being accountable to PeaceWrit Largemeansensuringthatinitiativesaddresskeydriversofconflictandmakeacontributiontothe'biggerpicture'.Thisrequiresanexplicitstrategyforinfluencingthosedrivers,andawaytomonitorandevaluateeffectsbeyondthelifeoftheproject.Itdoesnotmeanthatallprogramsshouldbeexpectedtoproduceconcretechangesatthelargersocietallevel.Infact,manyprogramsaresuccessfulatsmallerscaleinterventions,suchasoperatingatthecommunitylevel,orwithsmallgroupsofpeople,thuscontributingto‘peacewritlittle’. The impact of these interventionswill not be directly observable at a societal level. However,CDA/RPPhasfoundthatmanypractitionersassumethattheirprograms,becausetheyhavesolidgoals,willsomehowleadtoorsupportPeaceWritLarge.Thisisnotalwaysthecase.
AssessingcontributiontoPeaceWritLargeisdifficult,asmostpeacebuildingprogramsarediscreteeffortsaimedataffectingone(oftensmall)pieceofthepuzzle,andnooneprojectcandoeverything.RPPhasfound that certain elements of program strategy and logicmake it more likely that peacebuilding
projects/programswillhaveaninfluenceonPeaceWritLarge.Thosearefurtheroutlinedbelow–andarealsoatthecoreofthePQA,EA,andStrategyandProgramReflectionExercisedescribedinthisGuide.
KeyLessonII:ConflictAnalysisisCrucial
Basedonareviewofdozensofpeacebuildinginitiatives,thereisstrongevidenceabouttherelationshipbetweenconflictanalysisandstrategicandrelevantprogramminginconflict-affectedcontexts.
Goodconflictanalysisshould:
§ identifyKeyFactorsandKeyActorsvis-à-vispeace/conflict,§ identifytheRelationshipsandDynamicsamongthem,and§ clarifypointsofpossibleintervention.
Key driving factors of conflict are factors without which the conflict would not exist or would besignificantly different. Key actors are people or groups that can significantly influence the conflictdynamics.
Foramoredetailed introductiononhowtoconductconflictanalysis,pleaserefertotheCDATrainingManualsandtheConflictAnalysisFrameworkdevelopedbytheGlobalPartnershipforthePreventionofArmed Conflict in collaboration with CDA Collaborative Learning Projects and Norwegian Church Aid(2016)(seeSection8,Resources).
“Analysisisnotoptional;itisessentialandobligatoryforpeacework”
RPPconsultationparticipant
10
KeyLessonIII:AnalysismustbelinkedtoprogrammingandtoPeaceWritLargethroughastrongtheoryofchange
ExamplesofTheoriesofChange19
Projectlevel(individuallevelchange)20 Portfolio/Sectorlevel(socio-politicallevelchange)
If[activity]childreninthisschoolaregivenindividualtreatmentfortraumarecovery,
then[change]theywilldevelopincreasedabilitytocontroltheiremotionsandnotactoutagainstothers,especiallythosewhoaredifferentfromthem;
because[rationale]theactivitieswillhavehelpedthembegintohealfromthepsychologicalwoundsofwarandreducetheiroverallfearandsenseofvulnerabilityatschool.
Ifwewantedtomovethisengagementtopotentiallyshowresultstowardssocio-politicalchange:
[Note:undertheseconditions,ifweintroduceinter-groupskills(negotiation,mediation,problem-solving)tochildrenofdifferentreligiousgroupstogether,thentheywillbeabletolearnthemandusethemtoresolvedisputesatschool,includingthosethatmayarisebetweenreligiousgroups.21]
If[activity]westrengthenthecapacitiesofselectlocalandnationallevelgovernmentinstitutionsinviolencepreventionandcoexistence;
then[change]interactionswithinthegovernmentandbetweenstateandcivilsocietywillbemoreconstructiveandinclusive,
because[rationale]localandnationalgovernmentinstitutionswillbebetterequippedtodealwithtensionsmoreconstructivelyandengageinforwardlooking,preventiveapproacheswithingovernmentandinstate-societyrelations.
[Note:thissectortheoryofchangeisverymacro-levelandneedstobeaccompaniedbymoreconcreteandmeasurabletheoriesofchangeattheprogramandprojectlevel.Thiswouldhelpfurtherembedthetheoryinspecificoperationalcontexts.]
19Formorebackgroundonthedifferentlevelsatwhichtheoriesofchangecanbeuseful(activity,project,program,portfolio/sector,countryleveletc.)pleasesee(WoodrowandOatley2013)20Babbittetal,TheoriesandIndicatorsofChange:ConceptsandPrimersforConflictManagementandMitigation,921Ibid.
ATheoryofChange|isanexplanationofhowandwhyanactionisbelievedtobringaboutitsplannedobjectives,i.e.thechangesithopestocreatethroughitsactivities,therebyrevealingunderlyingassumptions.Acleartheoryofchangehelpstoarticulatethelogicalflowfromthestartingpoint(analysis)tothegoaloftheinitiativetothebroaderchangetheorganizationplanstoachieve.
Apracticalformulaforarticulatingatheoryofchangeisthefollowing:
Ifx[activity],
theny[expectedchange],
becausez[rationale-whydoyouthinkthischangewillhappen?]
11
Theoriesofchangeoperateatdifferentlevels.Theycanrelateto micro-level changes (e.g. project level), usually associatedwith specific activities. They can describe how an overallprogramapproachandgoalwillbeachieved(e.g.howvariousjustice and human rights initiatives achieve progress in thatsector), as well as how achieving the goal will contribute toPeace Writ Large (e.g. how different sector activities acrosshuman rights, rule of law, and dialogue promotionwill worktowardsreconciliation– if that iswhathadbeen identifiedasthevisionforPeaceWritLarge).Theoriesofchangeneedtobegrounded in the particular context, and should be specific
enoughtobetestable.
KeyLessonIV:ProgramsmustreachtheSocio-PoliticalLevelinordertoaffectPeaceWritLarge
For peacebuilding programs to be effective, they must linkchangeattheindividual/personalleveltochangeatthesocio-
politicallevel.Theindividual/personallevelincludesattitudes,perceptions, behaviors, skills and interpersonal relations. Thesocio-political level includes relations among social groups,public opinion, social norms, societal institutions, and deeperelements embedded in social and economic structures andculture.
Individual/PersonalChange
Healing/recoveryPerceptionsAttitudesSkillsKnowledge
BehaviorIndividualrelationships
Socio-PoliticalChange
Groupbehavior/relationshipsPublicopinionSocialnorms
Institutionalchange
Structural+culturalchange
RPP found that programming which focuses on change at the individual/personal level but nevertranslatesthisintoactionorresultsatthesocio-politicallevelhasnodiscernibleeffectonpeace.Inmanycases, it is also important to link change at the socio-political level back to individual/personal levelchange—especiallyifthechangesaretobemeaningfulandsustainable.
While the desired changes do not necessarily need to be observable at the national level,programs/projects should, at the level atwhich they are operating, affect the creationof institutions(formalorinformal),resultinlocally-drivenpeaceagendasandaction,orleadtocollectiveattitudesand
Morein-depthreadingontheoriesofchange:
® dmeforpeace.org/evaluation-planning® WoodrowandOatley2013® Babbit,ChigasandWilkinson2013
Cartoonist:SidneyHarris
12
behaviors that reflect improvements in the key driving factors of conflict or peace at that level (e.g.,increased security or perceptions of security, improved group attitudes or relations, resistance toviolence,etc.).22
KeyLessonV:“MorePeople”workmustengage“KeyPeople”andviceversa
Effectiveprogramsalsolinkworkwith“morepeople”—i.e.,peopleatmanylevelsofsocietyandinmanysectors—to“keypeople,”peopleorgroups thathave thepoweror influence todecide fororagainstprogresstowardspeace.
MorePeople KeyPeople
Peaceneedssupportandparticipationofthegeneralpopulation.
Peacecannotbeachievedwithoutinvolvementofcertainpeoplewithmajorinfluenceonthesituation.
Work that influences “more people” or“keypeople”butdoesnotconnector linkto efforts to affect the other has limitedimpact.
InsightsIVandVcombinedpromptedCDAto develop a very practical tool used toassessprogramstrategies:theRPPMatrix(SeeAnnexes2.1and2.2)
KeyLessonVI:Itisimportanttoengagethe‘hardtoreach’
Manyprogramsoperateoncertainbiases.They…
§ engagewiththe“easytoreach”(thosewhowanttoworkwithus),§ workwiththoseseenasnon-political,willingtocooperate,lesscommittedtoviolence,and§ focusondoing“good”vs.stopping“bad”(e.g.,participatorycommunitydevelopment,inter-ethnic
dialogue,etc.)anddonotdealwithdynamicsandpeoplethatpromoteorperpetuateviolentconflictand/orfragility.
Therefore,itiscriticalforpeacebuilderstoaskthemselvesthefollowingquestions:
§ Whatgroupsaresystematicallyleftoutofpeacebuildingefforts?§ Whoisthepeacebuildingcommunityavoiding?§ Whichgroupsmighthaveanegativeeffectonpeaceefforts,andcouldunderminefragilegains?§ Who(ifanyone!)hasaccesstothosegroups?
22Forfurtherreadingreferto(Ernstorfer,ChigasandVaughan-Lee2015)
Healing/recovery
Perceptions
Attitudes
Skills
Group behavior/
relationships
Public opinion
Social norms
MORE PEOPLE KEY PEOPLE
VISION: A desired futureSocietal change/Peace Writ Large
CURRENT SITUATION:
Conflict AnalysisKey Driving Factors of Conflict and “Key People” or
Actor Analysis
INDIVIDUAL/
PERSONAL
CHANGE
SOCIO-
POLITICAL
CHANGE
Institutional
change
Structural
change
Behavior
Individual
relationships
Programme Activities
Program Theory:
How do the activities
lead to the goal?
Socio-political
goal
Theory of Change:
How does the goal contribute
to Peace Writ Large?
What is the
gap between
the current
situation and
the desired
future? à
“peace
needs” and/or
strategic
space.
What needs to change
and how?
Programme Activities
13
The ‘hard to reach question’ will not beexamined to a great level of detail in thisGuide. However, it is often one criticalquestion in program relevance (next toothers).23
KeyLessonVII:ItispossibletoassesstheimpactofprogramsonPeaceWritLarge
For many years, the peacebuildingcommunity has been avoiding amore rigidapproachtomonitoringandevaluation(seeSection2).However, it ispossibletoassesstheimpactofprogramsonPeaceWritLargeIFprograms:
§ arebasedongoodconflictanalysis–toensurerelevance,§ haveawell-articulatedtheoryofchange,and§ haverobustandwell-definedgoals.
Wehavealreadytoucheduponconflictanalysisandtheoriesofchange.Robustprogramgoalsneedtobeset in termsofdesiredchanges.Manyprogramssetgoals forpersonalchange, includingattitudes,perceptions,personalbehaviors,skills,andrelationsamongindividuals—andchangeatthislevelisoftennecessary,thoughrarelysufficient.
However, programs that formulate goals as desired changes at the socio-political level, and/or havestrategiesinplacethatworkwithotherprogramsandorganizationsoperatingatthatlevel(iftheydon’tdoitthemselves),aremorelikelytohaveimpactsonPeaceWritLarge.
BuildingBlocksforPeace(CriteriaofEffectiveness)
Basedonextensivecaseanalysisandpractitionerreflection,theRPPprocessidentifiedfiveintermediateBuildingBlocksforPeaceorCriteriaofEffectivenesstosupportprogresstowardsPeaceWritLarge.
Thesecanbeusedtoassess,acrossabroadrangeofcontextsandprogrammingapproaches,whetheraprogramismakingameaningfulcontributiontoPeaceWritLarge.
The effort results in the creation or reform of political institutions to handle grievances insituationswheresuchgrievancesdo,genuinely,drivetheconflict.
The effort contributes to amomentum for peaceby causingparticipants and communities todeveloptheirownpeaceinitiativesinrelationtocriticalelementsofcontextanalysis.
Theeffortpromptspeopletoresistviolenceandprovocationstoviolencemorefrequently.
Theeffortresultsinanincreaseinpeople’ssecurityandintheirsenseofsecurity.
Theeffortresultsinmeaningfulimprovementininter-grouprelations.
TheseBuildingBlockscanbestbethoughtofasintermediate-levelbenchmarksorindicatorsofsuccessapplicable to thebroad rangeofpeaceworkbeingdone.TheBuildingBlockscanbeused inprogram23Forfurtherreadingpleasesee(CDACollaborativeLearningProjects2012)
1
2
3
4
5
It’salmostenoughtomakeanevaluatornostalgicforfuzzygoals.Cartoonist:MarkM.Rogers
14
planningtoensurethatspecificprogramgoalsarelinkedtothelargerandlong-termgoalof“PeaceWritLarge.” Theycanbeusedduringprogram implementation to reflectoneffectivenessandGuidemid-coursechanges,andasabasisforevaluationaftertheprogramhasbeencompleted.
Foramorein-depthintroductiontoRPP’slessonsandinsights,pleaserefertotheresourceslistedinthebibliography(Section10).
2.3KeylessonsfromCDA’sDoNoHarmProgram(DNH)
A conflict-sensitive approach minimizes the negative and maximizes the positive impacts of anyinterventionsonpeaceandconflictdynamics.Manyorganizationsworkonconflict-sensitivity,andusethe DNH approach developed by CDA – both as a tool, a framework, and a ‘standard’ for conflict-sensitivity.
ThecollaborativelearningprocessthatCDA’sDoNoHarmprogramwentthroughledtothefollowingsixmainconcludinglessons:
1. Whenaninterventionofanykindentersacontext,itbecomespartofthatcontext;2. AllcontextsarecharacterizedbybothDividersandConnectors;3. AllinterventionswillinteractwithbothDividersandConnectors,makingthembetterorworse;4. InterventionsinteractwithDividersandConnectorsthroughtheirorganizationalActionsandthe
Behaviorofstaff;5. TheDetailsofaninterventionarethesourceofitsimpacts;6. TherearealwaysOptions(e.g.forprogramre-designordoingthingsdifferently.)
TheselessonsresultedinthecreationoftheDNHframework:
CONTEXTOFCONFLICT
Options Dividers Interventions Connectors Options
Redesign
Who?What?Where?How?When?Why?
Redesign
ForfurtherintroductiontotheDNHframeworkpleaserefertotheDNHresourceslistedinSection10.
Actionsand
Behaviors
HeadquartersMandateFunding
15
ThefollowingSections4,5,and6willprovidedetailedguidanceandastep-by-stepapproachtoProgramQualityAssessments,EvaluabilityAssessmentsandStrategyandProgramReflectionExercisesusingRPPandDNHascriteria.
Principles of participation, partner and local stakeholder engagement throughout thisdocument
Program Quality Assessments, Evaluability Assessments, and Strategy and ProgramReflectionExercisesprovideexcellentopportunitiesforprogramteams toworkwith theirprogrampartners,localstakeholdersanddonorstojointlydevelopavisionforstrengthenedstrategiesandprogramsinaparticipatoryfashion.
Eachteamwillneedtodeterminewhattypeandwhatlevelofpartnerengagementseemsmostappropriateanduseful.Ingeneral,alltheseprocessesencourageamaximumlevelofprogram partner and local stakeholder engagement. No program quality assessment orEvaluabilityAssessmentcanbecrediblydonewithoutawell-thought-outprocessofprogrampartnerengagement.Ingeneral,StrategyorProgramReflectionExercisesshouldalsoinvolvepartnerstoacertainextent.However,StrategyandProgramReflectionExercisesmightbethe option most frequently used to support internal team alignment around programstrategy–andnotalwaysinvolveprogramandotherlocalpartners.
ThequestionofhowprogrampartnersandlocalstakeholdersareengagedinEAsandPQAswill need to be determined case by case. In some instances, the facilitation of a jointworkshopwithallrelevantteams,partners,anddonors isfeasibleandadvisable. Inotherinstances, separate processes of consultations and bi-lateral interviews are moreappropriate,alsoregardingoverallpoliticalandculturalsensitivities.Inanycase,especiallyforEAs,acertainamountofindependenceindatacollectionwithlocalpartnersisessentialforthecredibilityandrobustnessoftheprocess.
16
3.Whatistherightoptionformyteam?Whatarethebasicsweneedtohaveinplacetobenefitfromanyoftheseoptions?
AllthreeoptionsputforwardinthisGuidearelearningtoolswithdifferentfoci,differentpurposes,anddifferentdegreesofrigor(seeSection7forasummaryoverviewofkeyfeaturesofPQAs,EAsandStrategyandProgramReflectionExercisesvis-à-visformalevaluations).Allthreeoptionsreviewprogramdesignandstrategy,inordertomaximizethepotentialimpactofpeacebuildinginitiativesonconflictsystems.However,theseoptionsarenotdesignedasevaluationstoassesstheactualchangesin-country.
Beforedecidingwhichevaluativeoptiontochoose,afewkeyquestionswillneedtobeanswered:
� Whereareyouinthelifeoftheprogram?
� Whatpromptedtheneedtoundertakeoneoftheseexercises?
� Whatisthepurposeandexpectedresultoftheprocess?Howwillyouusewhatyoulearnthroughthisprocess?
� Whatisthenatureoftheexercise(whichoptionsandforwhatpurpose)andrelateddemandsonteam,staffandprogrampartners:areexistingcapacitiesandresourcessufficient?
� Whatarethemainlinesofinquiry?Isdataavailabletoanswerthekeyquestions?Howmuchdatacollectionisneeded?Dataavailabilitymightalsoinfluencethechoiceofanevaluativeprocess.
� TeamCohesion:IsSeniorManagementonboard?Dothedonorsneedtobeinvolved?Isthereacommon vision about the purpose/timing/requirement amongst staff and/or partners andeveryonewhoneedstobeinvolved?Howwillpartnersbeincluded?
� Whowillleadandfacilitatetheexercise?
� Whatismoreappropriate–aninternal,externalorblendedprocess?Sometimesalessformalprocess ismoreconducivetostrengtheninganinitiative. Itcanbefacilitatedinternallyorwithexternalsupport.
Moreresourcesonwhetheryourprogramisreadyforanevaluation:
® dmeforpeace.org/evaluation-planningunder"Istheprogramorinitiativereadyforevaluation?”
17
ChoosingtheRightEvaluativeOption
Thebelowtablemighthelpwithdecisionmakingbasedondifferenttypesofpurposesandintentions:
Ifyourgoalisto… …youshould…
…conductanindependentassessmentofyourprogramstrategyreviewingconflictanalysis,goals,theoriesofchange,andprogramstrategy…
→ …chooseaprogramqualityassessment.
…assesshowreadyyourprogramisforanevaluation… → …chooseanevaluabilityassessment.• …assesstheperformanceoftheinterventions,review
outputs,outcomes,andresults,andassessimplementationpractices…
→ …conductanevaluation.
• ...facilitateaninternal,fairlyinformalprocessofreflectionandimproveyourteam’s(andpossiblyyourpartner’s)understandingofwhatmakesapeacebuildinginitiativerelevantandeffective…
→ …conductastrategy/programreflectionexercise–orpossiblyaprogramqualityassessment(thelatterappliesthepeacebuildingcriteriamoresystematically).
…obtainanindependentassessmentofyourprogramtoshowaccountability24toyourdonorand/orprogrampartners…
→ …conductanevaluation.
…obtainaninternalassessmentofyourworktoshowaccountabilitytoadonorand/orprogrampartners
…conductaninternalevaluationorself-evaluation.
…analyzedataavailability,andunderstandtheconducivenessofthecontextforyourprogram’seffectiveness…
→ …conductanevaluabilityassessment.
…developacommonunderstandingwithintheprojectteam(andpossiblyamongstpartners)aboutthecontext,overallgoals,theoriesofchange,andprogramstrategy,aswellasstrengthenskills…
→ …initiateastrategy/programreflectionexercise.
…strengthenthecapacityofyourteamandpartnersinprogramstrategydevelopmentandimplementationbyassessingyourprogrambasedonRPPcriteria…
→ …planforaprogramqualityassessment.
…totrainyourstaffand/orpartnersinRPPand/orDNHtoolsandapproaches…
→ …notconductanyoftheseprocesses,butdevelopa/RPP/DNHtrainingprogramforstaffandpartners.
…tointroducebasicM&Eframeworksandapproaches… → …notconductanyoftheseprocesses,butdevelopanM&Etrainingprogramforstaffandpartners.
…analyzewhetheryourinitiativemightneedtobeadaptediftherearemajorchangestothecontext…(e.g.anelection,orsigningofapeaceagreement)
→ …consideranyoftheseoptions.Butastrategyandprogramreflectionexercisewill,inmostcases,bethemostappropriate.
24Notallformalevaluationsfordonorsonlyservethepurposeofaccountability.Somedonorsalsoconductlearningevaluations.
18
ConditionsforEngagementinRPPinfusedEvaluabilityAssessments,ProgramQualityAssessmentsandStrategy/ProgramReflectionExercise
Regardlessofwhichmodalityischosen,thefollowingconditionsshouldbeinplacebeforeconductinganyoftheseprocesses:
Condition Explanation
Willingnesstochallengeassumptions
RPPtoolsandlessons-learnedoccasionallyprovidechallengingmessagesaboutwhatmakesforeffectivepeacebuilding.Inparticular,theprocessinvolvescarefulconsiderationofthetheoriesofchangeunderpinningtheprogram–i.e.theveryassumptionswehaveabouthowtheprogramwillcontributetochange.Theprogramteamandprogrampartnersneedtobeopenandreflective,reflexiveandwillingtochallengetheassumptionsuponwhichprogrammingisbased.
Abilitytoadjustprogram
Adjustmentstoprogramdesignand/orimplementationarelikelytoresultfromeitherprocess.Thereneedstobeawillingnesstoadaptprogrammingonthepartofallrelevantstakeholders,includingdonors.Thismaybeparticularlychallengingiftheprogramisbeingimplementedbyaconsortiumwheredifferentagenciesareimplementingdifferentpartsoftheprogram.
OpennesstoRPPtoolsandmethods
SometeamsandprogramwillhavehadpriorexposuretoRPPtoolsandapproaches,somewon’t.Whileitisnotapre-requisitethatprogramteamsand/orprogrampartnershavebeenexposedtooralreadyknowRPP,moretimewillneedtobeallocatedifteamsarenewtothemethodologies.Inanycase,theprogramteamwillneedtobeopentousingthesetoolsandmethods.
Realisticsenseofcapacitydevelopment
TeamsandSeniorManagersneedtohavearealisticsenseofhowmuchcapacitydevelopmentispossiblethroughastrategy/programreflectionexerciseoraprogramqualityassessment.ForteamsandprogrampartnerswithnopriorexposuretoRPPtools,thebasicconceptscanbeconveyedthrougheitherofthose.However,thisdoesnotreplaceamoreformaltrainingand/orcapacitydevelopmentprocessforstaff.EvaluabilityAssessmentsusuallydonotincludeafocusoncapacitydevelopment.
HavingtheseconditionsinplacewillmaximizethebenefitofanyofthethreeoptionsinthisGuideandfacilitateuptakeofthefindingsandfollow-up.
19
4.ProgramQualityAssessment(PQA)–criteriaforassessingpeacebuildingprogramquality
4.1Overview:PurposeofaProgramQualityAssessment
ThepurposesofaPQAareto:
§ reviewandlearnaboutthequalityofaprogram’sdesign,§ strengthen/adaptprogramdesign throughapplicationofa clear setofprofessional standardsand
criteria,§ maximizepotentialforprogramtocontributetoPeaceWritLarge,§ laythefoundationforadaptiveprogrammingandactionplanningwithprogramteamsandprogram
partners,and§ supportstrategicandlong-termlearning.
A PQA examines how the program is functioning in practice, but only to the extent necessary tounderstandifandhowaprogramismeetingthegivencriteria.Particularly,thisincludesanassessmentof the quality of the program strategy, logic and theory of change, and how conducive these are toachievingtheenvisionedoutputs,results,andoutcomesoftheprogram.Thus,itexaminesifitislikelytoachieveitsgoals,aswellasachieveacontributiontoPeaceWritLarge.Itdoesnotassessresults,outputs,andoutcomesindetail.
It is similar toa formativeormid-termevaluation,butdiffers in that itdoesnot involvea systematicevaluationprocessnorapplyevaluationstandards,andhasastrongfocusoncapacitybuilding.PQAsdonot substituteanevaluation.However, theremightbe circumstances (e.g. anextremelydynamicandquickly changing context) under which a PQA might be the appropriate level of rigor, and a formalevaluationmightnotbepossible.
4.2KeyelementsofaprogramqualityassessmentbasedonRPPcriteria
ThePQAinvolvesareviewofkeyprogramdimensionsandassessmentagainstRPP-basedcriteria:
1. Performanceofaconflictanalysiscontributingtoprogramrelevance;2. Clearandappropriateprogramgoals;3. Well-formulatedandplausibletheory(ies)ofchangeatdifferentlevels(macro–meso-micro);4. Programstrategyandlogic,includingincorporationofanM&Esystemaspartoftheprogram
design;255. Inclusionofconflict-sensitivityindesignandimplementation.
Thefollowingoverviewtablesprovidekeyassessmentcriteria,andexamplesofcommongapsandweaknessesfoundinmanyprograms.
Annex5willprovideapracticalPQAtoolthatteamscanusetodocumenttheoutcomesofprogramqualityassessmentofeachoftheseprogramdimensions.
25WhereasinanEvaluabilityAssessmenttheM&Esystemismuchmoreinthefocusintermsofdatacollectionmechanisms.
20
PQATable1:ConflictAnalysisbasedonRPPcriteria
RPPCriteria Commongaps/weaknessesorcautionsconcerninganalysis
1 Theanalysisidentifieskeydrivingfactorsandkeyactorsforconflict/peace.26
§ Analysisistoocomprehensive:toomanyfactorswithnoprioritiesidentified.
§ Analyzesentirecontext,butdoesnotfocusonconflictdeterminants;everythingisseenasrelevanttopeacebuilding.
§ Factors/issuesareidentified,butnotdynamicsamongthemorwhicharemore/lessimportant(priorities).
§ Analysisisimplicit,andthusnotsharedamongteamandprogrampartners.
2 Theanalysisconsiderswhatneedstobestopped(andwhowillresist)andwhatforcespromotepeaceinthiscontext.
§ Analysisfocusesonpositivefactorsthatmightbestrengthened,butdoesnotconsidercountervailingnegativeforces.
§ Analysisdoesnotanalyzewhatfactorsconnectpeopleorpromotepeaceinthiscontext.
3 Theanalysisisupdatedandtestedregularly/periodically.
§ Analysisisperformedonceatbeginningofprogram,butnotupdated.§ Conclusionsaboutdriversanddynamicsofconflictarenotutilizedto
strengthenprogram.
4 Analysisofsimilarprogrameffortsincludinganylessonsfromtheirresults(programeffortsincludethosefromthepastorongoing).
§ Analysisdoesnotidentifyresults/lessons§ Theprogramteamhasnoknowledgeofwhathasbeentriedbefore(or
resultingeffects).§ Programsrepeatfailedapproaches.§ Programs/projectsduplicateeffortsofotherswithoutaddedvalue.
5 Theprogramstrategybuildsontheanalysis:identifiespeacebuildingneedsorpointsofleverageforchange.
§ Programgoalsanddesigndonotaddressfactorsidentified.§ Analysisdoesnotenableprogramdesignerstoidentifywhattodoto
changeconflictdynamic.
6 Thescopeoftheanalysisisappropriate(nottoobroadornarrow);andmitigatesbiastowardsagency’sexpertiseorgeneralbeliefsaboutconflict.
§ Analysisisperformedtojustifyfavoredprogramapproach(methodology,focus,constituency).
§ Analysisisbasedonbeliefsabouthowtobringaboutpeacegenerally(andnotcontextualized).
§ Analysishasomittedorexcludedsignificantperspectives.§ Analysisexistsatonelevel,butdoesnotconsiderotherlevels(e.g.,at
locallevel,missingwiderdynamics;national/regionalanalysiswithoutlocalparticularities;nationalanalysiswithoutinternational/regionaldimensions).
7 Theanalysisprocesshasbeenconflictsensitive—consideredpotentialharmitmightcause
§ Teamcompositionexposesteammembers,partners,orintervieweestodanger.
§ Teamcomposition,behaviorperceivedasbiased.§ Analysisprocessdeepenspolarization.§ Causesofconflictarecontestedamongkeyparties—analysisprocessis
fraught.
26Keydrivingfactorsareelements/dynamicswithoutwhichtheconflictwouldnotexistorwouldbesignificantlydifferent.Keyactorsarepeopleorgroupsthatcansignificantlyinfluencetheconflictdynamics.
21
PQATable2:RPPcriteriaforprogramgoals
RPPCriteria Commongaps/weaknessesorcautionsingoals
1 Thegoaladdresses,directlyorindirectly,keydriversofconflictorpeace.
§ Thegoalisnotappropriateforconflictcontext;othergoalsmaybemoreappropriate.
§ Thegoal(s)addressessymptomsorconsequencesofconflictbutnotdrivers,oritaddressesfactorsofsecondaryimportance.
2 Thegoalisstatedasadesiredchange. § Intendedorexpectedchangesfromtheeffortarenotclear.
§ Goalsarestatedasactivities,outputsortasks.
3 Thegoalisspecificandrealisticforthetimeframe—neithertoobroad(along-termvision)nortoonarrow(attheactivitylevel).
§ Goalsarevague,grandioseand“overclaim”.§ Goalsareexpressedatavisionlevel.§ Goalsstatedareprocesses(ratherthanthe
outcomeofprocesses)oraseriesofactivities.§ Programteamisunabletoarticulateclear
benchmarks,indicatorsorothersignsthatwouldhelpthemknowifprogressisbeingmade.
4 Thegoalisstatedasadesiredchangeinthesocio-politicalrealm.Ifnot,thereisanexplicitlonger-termstrategyforeffectingsocio-politicallevelchange,ortheprogrammakeslinkagestotheactivitiesofotheragenciesinthesocio-politicalrealm.
§ Theprogram/projectseekschangeattheindividual-personallevelonly(attitudes,skills,etc.),andunrealistically“hopes”orassumesthatchangesatthesocio-politicalwillcomeabout.
§ Theprogramgoalattheindividual-personallevelisappropriate,butlinkagestootherprogramsorstrategiesforfollow-onworktomovetothesocio-politicalleveldonotexist.
§ Theprogramassumes(withoutcontext-basedevidenceorconflictanalysis)thatalotofworkatthemicro(community)levelwillsomehow“addup”tosignificantchangesathigherlevels(PeaceWritLarge).
§ Changesdesiredarenotsustainable,bigenoughinscaleorfastenoughinthiscontext.
5 Thechangescontributetooneofthefollowingbuildingblocksforpeace:
1. Politicalinstitutionsthataddresskeydriversofconflictarecreatedorreformed.
2. Locallydrivenpeaceinitiative/saddress(indirectlyordirectly)keydriversofconflict/peace.
3. Peopleincreasinglyresistviolenceand/orprovocationstoviolence.
4. Peoplegainincreasedsecurityand/orasenseofsecurity.
5. Inter-grouprelationsimprovesignificantly(e.g.,groupattitudes,publicopinion,socialnorms,publicbehavior).
§ Goalsaimatindividualattitude,skillorbehavioralchangeonly.
§ Teamsand/orpartnersdonotagreeonthetypeofchangetheyarepursuing.
§ Programgoalsrepresentmeaningfulchange,butitisnotclearhowthechangemightbesustained.
22
PQATable3:RPPcriteriaregardingoveralltheoryofchange
RPPCriteria Commongaps/weaknessesorcautionsconcerningoveralltheoryofchange
1 TheOverallTheoryofChangeisexplicit,withclearandunderstandableconceptualizationofpathwaytochange.
§ Theoveralltheory(ies)ofchangeisimplicit,unstated.§ Itisunclearhowtheprogram,ifsuccessful,willaffect
keydriversofconflictorpeace.§ Theoryofchangeisbasedonfalseassumptionsabout
howchangecomesaboutinthiscontext.
2 TheOverallTheoryofChangeprovidesaplausibleexplanationofhowachievingthegoalwillaffectkeydriversofconflictorpeace.
§ Programsseekchangesthatarereasonableinthemselves,butwillultimatelyfailtoachievesustainablepeace(e.g.,passingalawwillnotaffectconflictdriversbecauseagreementonprinciplesandenforcementmechanismsarenotbeingworkedon).
3 TheoverallTheoryofChangeisgroundedinanunderstandingofhowchangehappensinthiscontext
§ Waysinwhichchangeprocessesaredifferentfromcontexttocontextarenotexaminedwhenprogramideasorapproachesfromonecontextareappliedinanother.
Insightfrompractice.NoneoftheprogramsexaminedundertheCDAfacilitatedEAsandPQAshadbeenplannedaccordingtoRPPcriteria.OneofthePQAshappenedshortlyafterprogramdesign,sosomeoftheRPPtoolsandapproacheswereintroducedaspartofaprogramre-designthathappenedduringthePQA(questionremainedwhetheraProgramReflectionExercisewouldhavebeenmoreappropriateforthatpurposeinsteadofaPQA).Asfortheothers,someoftheRPPelementswerealignedwithotherapproachesused,or freshly introducedaspartof theAssessmentFramework. Incases inwhichRPPprinciplesarenew,additionaltimeandarelatedbudgetneedstobebuiltintofamiliarizeparticipantswiththeconceptsand/orplantimeforrelatedcapacitydevelopment.
Insightfrompractice.InoneoftheCDAfacilitatedPQAs,theCDAfacilitatorworkedwiththeprogramteamandpartnerstoturntheprogramgoal,whichwasstatedatthemacro-level/long-termvisionlevel,intoamorerealistic,andachievablegoalfora2-yeartimeframe.Thishelpedclarifyprogramactivitiesandbenchmarksfor theprogram,whileadoptingthemacro-levelarticulationof theobjectiveastheoverall and long-termvisionof the initiative (so the team felt theywere going in the rightdirectionbroadlyspeaking,butnowwithaclearerunderstandingofactualprogramachievements).
23
PQATable4:RPP-basedcriteriaforprogramstrategyandlogic
RPP-BasedCriteria Commongaps/weaknessesinprogramstrategyandlogic1 Theactivities“addup”to
thegoal—thereisanexplicit,rationalandplausiblelinkbetweencomponentsoftheprogram(input,output,outcome,impact)thatisvalidinthiscontext.
§ Programtheory/logicisunclearornotexplicit.§ Programlogicisweak/illogicalorbasedonfalseassumptionabout
howchangecomesabout.§ Therearegapsorleapsoflogicinthepathwaytothegoal.§ Assumptionsabouthowonechangewillleadtoanother(howthe
programactivitieswill“addup”)havenotbeenexploredorarticulated.
§ Teammembersandprogrampartnersproceedonverydifferentassumptionsaboutprogramgoals,objectivesandchangeprocesses.
§ Programsfailtoaccountforkeyrequirements(e.g.,willingness&availabilityofparticipants).
§ Assumptionshavenotbeenchallengedorthoughtthrough.
2 Theprogrammakeslinkagesbetweenactivities/changesattheindividual/personallevelandthesocio-politicallevel.
§ Programactivitiesandchangesareexclusivelyattheindividual/personallevel(attitudes,skills,relationships),withnostrategytotranslatethesechangestosocio-politicalchange(eitherthroughfollow-upactivitiesorprograms,orlinkageswithotherefforts).
§ Linkagesoreffectsfromtheindividual/personaltothesocio-politicallevelbasedonlyon“hopes”orassumptions.
3 Theprogrammakeslinkagesbetween“morepeople”and“keypeople”.
§ Programfocusesonthe“easytoreach”withnostrategyforreachingbeyondtoaffectthe“hardtoreach”orconstituenciesignored.
§ Programhasnotincorporatedstrategiesforaffecting“keypeople”(ifworkingwith“morepeople”)or“morepeople”(ifworkingwith“keypeople”).
§ Programworksbothwith“morepeople”and“keypeople”andassumeslinkagethatmaynotoccur,i.e.areunrealisticinthiscontext.
§ Programbelievesitisworkingwithkeypeoplewhenitisnot(e.g.,assumesgovernmentofficialsare“key”whentheyhavelittleinfluenceontheconflict;assumespeoplekeytoimplementationoftheprogramortothemissionoftheagencyare“key”toconflict;assumesvictimsofconflictarekey).
§ Analysisdefinesentiregroupsofpeople(e.g.,youth15-25)askeybutdoesnotexaminewhetheritwillreachthoselikelytoperpetrateviolence.
4 Thescaleandleveloftheoutputsarereasonableinrelationtotheintendedimpactsandthesizeoftheissueinthiscontext.
§ Thereismismatchbetweenscaleofgoal(e.g.,toleranceorreconciliation)andscaleandlevelofoutput(e.g.,numberofparticipants,communities,etc.)
§ Theprogramisnot“big”enough—doesnothaveenoughscaletohavemeaningfulinfluence—andthereisnostrategy(eitherwithinorbeyondtheprogramorinconjunctionwithotherefforts)forachievingmeaningfulscale.
24
RPP-BasedCriteria Commongaps/weaknessesinprogramstrategyandlogic5 Theprogramdesignhas
accountedforfactorsthatcouldimpedesuccess,includingwayssocial&politicalsystemsmightresistchangesthattheprogramistryingtoworktowards.
§ Programhasnotaccountedforhowthesocialandpoliticalsystemswillpushbackagainstchangeefforts.
§ Programcan/doesachievemeaningfulinfluence,butitisnotclearhowchangescan/willbesustained—how,whyandbywhom?
6 Theeffortrelatestootherinitiativesinthesameorrelatedareasofwork,intermsofnecessarycomplementarities,linkagesorduplicationofeffort.
§ Programduplicatesothereffortsunnecessarilyorfailstoidentifyandlinktocomplementaryeffortsinthesamedomain.
7 ThereisanM&Eplanorfeedbackmechanismthatwillprovidetimely,accurateandusefulinformationaboutprogresstowarddesiredchangesandaboutassumptionsunderlyingthetheoryofchange.
§ Thereisnoprocessinplaceformonitoringandtestingtheprogramlogicandensuringtheprogramisnotcreatingunintendednegativeeffects.
§ Indicatorsareunrealistic.
RPPcriteriaonprogramstrategycombinedwithotherspecializedandsector-specificstandards
InadditiontotheapplicationofRPPcriteriaassessthequalityofprogramstrategy,specializedand/orsector-specificstandardsforthespecifictypeofpeacebuildinginterventionsalsoneedtobeconsideredaspartofaPQA-wheretheseexistorcanbeinferred.
Forinstance,ifthecentralprogrammethodologyinvolvesmulti-stakeholderdialogue,therearenumerous studies and scholarly articles that posit best practices and norms that can bereferenced when assessing a dialogue program. Likewise, programs that focus on electoralviolencepreventionnowcanbuildonvariousstandardsandtoolboxes.RPPcriteriaaregeneraltoalltypesofpeacebuildingprogrammingandfocusontheeffectivenessandrelevanceoftheprogrammingmorebroadly.
25
PQATable5:Inclusionofconflict-sensitivityintoprogramdesignandimplementation
DoNoHarmCriteria Commongaps/weaknessesinprogramstrategyandlogic1 Theprogramisconflict
sensitive—itconsiderspotentialunintendednegativeimpactsitmightcause.
§ Programhasunintendednegativeeffects.§ Programdesignand/orimplementationhasnotexamined
commoncausesofunintendednegativeeffects(thepotentialnegativeimpactsofchoicesaboutprogrampartners,contractors,suppliers,location,distributionofbenefits,timingofprogrammingetc.).
2 Theinitiativeisconsciousoftheimpactofitsactionpatterns/resourcetransferpatternsinfiveareas:1. Theft2. MarketEffects3. DistributionEffects4. SubstitutionEffects5. LegitimizationEffects[SeeAnnex4,DNHActionPatterns]
§ Goodsormoneyintendedfordistributionofpaymentisstolenorusedbyactorsintheconflictand/ortosupportongoingviolence.
§ Localmarketsaredestroyed,localpeoplearepricedoutoftheirownmarkets.
§ Unevendistributionofgoodsandservicesalongconflictlines.§ Unintentionalweakeningofthestate’sabilitytorespondand
manageconflictsanddisaster,anditsowndevelopment.§ Inadvertentlylegitimizingagovernment,institution,orleader.
3 TheinitiativeconsidersunintendednegativeimpactsoftheprograminitsM&Esystem
§ M&Esystemsarenotdesignedtocapturethedetailsofhowaninterventioninteractswiththeconflictcontext.
§ Therearefeworganizationalincentivestocaptureunintendedimpactsandtoactuponthosefindings.
Insightfrompractice.Manyprogramsarefamiliarwithriskassessmentaspartofprogramming–andmanypeacebuildingprogramsassumethattheyareautomaticallyconflictsensitive.TheCDA-ledPQAshelpedtounpackthedistinctionsbetweenconflictsensitivityandpeacebuildingprogramming,astheyprovidedaspaceforconversationsaboutriskmanagement(anentrypointfordiscussionsaboutconflictsensitivity).Theyalsohelpedclarify therequirements forconflict-sensitiveprogrammingandrelatedM&E.
Insight frompractice.TheCDA-ledPQAsandEAsrevealedagreaterneedfor capacitydevelopmentthanwasinitiallyanticipated.OnlyafewindividualswithinthoseteamsandpartnershadpriorexposuretotheRPPandDNHapproachesandtools,andonlyafewhadin-depthM&Eand/orpeacebuildingskills.Allprocessesendedup includingsignificantportionsof capacitydevelopment,which, insomecases,compromisedthedepthoftheactualassessment,asparticipantswerelearningthetoolsatthesametime as engaging on the assessments. Some prior training would have been useful, as would havefactoringinmoretimefortheassessmentsgiventheextensivecapacity-developmentneeds.
26
KeypreparationstepsforaProgramQualityAssessmentbasedonRPPcriteria
Step1:ClarifyoverallpurposeofthePQAwiththeteam.Stressthelearningaspect,thefactthatPQAsarenotevaluations.Clarifywhatwillhappenwiththeresultsoftheprocessandhowthefindingswillbeusedinternally,withpartners,andpossiblywithdonors.IdentifycapacitydevelopmentneedsonRPPandDNHtoolsandapproaches,aswellasM&Eofpeacebuildingskillsaspartoftheprocess.Determinehowmuch is realistic tobedone in termsofcapacity-developmentaspartof thePQA-andwhatneeds tohappenseparately.
Step2:GetSeniorManagementonboardwiththeprocessand/ortoparticipate(atleastinpartsofit).Re-confirmavailabilityofbudgetfortheprocess(stafftime,venue,facilitator).
Step3:DevelopTORforexternal facilitatorandrecruitsomeonewiththerightskillset–e.g.,amixofM&E,peacebuilding,facilitation,andcapacitydevelopmentskills.
Step4: Identifyparticipants in thePQA:programteam,programpartners,donors.Partneranddonorparticipation in a PQA is critical, the type of engagement can vary based on context and needs (e.g.workshopwitheveryone,orworkshopwithsomeandbi-lateralinterviewsandfocusgroupswithothers).Clarify expectationswith the team: an open mind set, active participation, willing to constructivelychallengethemselvesandcolleagues,abilitytoengagewiththeresultsandengageinfollow-up.
Step 5: PQA Facilitator: Review relevant documents, program monitoring data, and conduct selectinterviewswithprogrampartnersandotherlocalstakeholders.
Step6:JointlydeterminewiththefacilitatorwhetherthereareanyparticularelementsofthePQAthatshouldreceiveaparticularfocus.ClarifycapacitydevelopmentexpectationsonRPPandDNHandhowtheywill be addressed through PQA – or a follow-up process. Consult SeniorManagement on thesequestions.
Step 7: Logistics:Make sure sufficient time is being set aside for the PQA and that Sr.Managementreleasesstafffromotherobligationsduringtheexercise.Arrangeforaconducivevenue–ifintheoffice,ensurepeoplecommittoparticipating(asopposedtobeingpulledbacktotheirofficestodootherworkorrespondtoemails)
Step8:ConductPQAthroughafacilitatedworkshopwithallrelevantparticipants,includingadditionalbi-lateralinterviewsasneeded.
Step9:SummarizefindingsfromthePQAincludingafilled-inassessmenttemplate(seeAnnex5),withclear recommendationsonhowto improveprogramquality.Developstrategyonhowto include intorevisedprogramdocumentationandplan,andhowtouseasareferenceforfutureexercisesofthiskind(e.g.updatedanalysis,reformulatedgoalsortheoriesofchange,RPPMatrixetc.).
Step10:Developafollow-upactionplanonhowtousethefindingsfromthePQAsinanongoingprocessof learning, reflection, and program improvement. Determine engagement of partners and donors.Determinefuturecapacitydevelopmentneedsandplans.
27
5.EvaluabilityAssessment(EA)–areyoureadyforanevaluation?
5.1Overview–PurposeofanEvaluabilityAssessment
Thepurposesofanevaluabilityassessmentareto:
§ determinewhetheraprogramcanbemeaningfullyevaluated(isit‘evaluable’?),§ determinetheobstaclestoaneffectiveandusefulevaluation,andhowtostrengthentheprogramto
increaseitsevaluability,§ reviewthecoherenceandlogicoftheprogram,§ clarifydataavailability(qualityandquantityofdataavailable),§ analyzetheextent towhichprogramteamsandprogrampartnersare likely tousetheevaluation
findings,§ determinecapacityandorganizationalsystems’needsinrelationtodatacollectionandanalysis;§ laythefoundationforadaptiveprogrammingandactionplanningwithprogramteamsandpartners,
withaviewtowardstheimpendingevaluation,§ informthedesignoftheimpendingevaluation,andto§ supportstrategicandlong-termlearning.
EAsmaybeparticularlysuitedforlargeandcomplexprograms,whereitwillbehelpfultoclarifywhatwillbeevaluatedandhow.ItallowsanexternalorinternalEAfacilitatortoobservetheprograminaction,engagekey stakeholdersand todemonstrate thevalueofevaluativeprocessesandevaluation to keydecision-makersandprogramstakeholders.
Just like PQAs, EAs do not take the place of an evaluation. However, they are useful precursors toevaluations,especially fordeterminingwhetheranevaluation isworthwhileand feasible,andatwhatstageofprogramimplementation.Evaluabilityassessmentsaskabouttheplausibilityofresultsinorderto assess the plausibility of the theories and assumptions underpinning a program, but donot try toevaluatetheactualresultsachieved.AnEAalsoexamineswhethertheprogramisbeingimplementedasdesigned,i.e.ifthetheoryandprogramimplementationalignandwhetherthedatamonitoringsystemsareappropriateandfunctioning.
Therefore,anEAshouldcoverthefollowingthreeelements,asillustratedinthefollowingdiagram:27
Thethreeelementsarebestunderstoodas interwovenandinterrelated.
Theresultsofanevaluabilityassessmentfallalongaspectrumsuchasthefollowing:28§ Fullyevaluable;§ Mostlyevaluable:canimprove;§ Limitedevaluability:needssubstantialimprovement;and§ Notevaluable.
27Reimann,Chigas,andWoodrow,AnAlternativetoFormalEvaluationofPeacebuilding:ProgramQualityAssessment.Thethree-elementapproachissimilartothatfoundin(UNIFEM2009)28See(InternationalLaborOrganization2011)
Evaluability Assessment
Data Availability
Conduciveness of Context
Program Design
28
Clearly,ifaprogramorprojectwasfoundtobe“notevaluable,”ornotworthyofanevaluationfromapeacebuildingperspective(seeSection5.2),afullevaluationwouldnotbewarranted.Ifaprogramhasbeenfoundtobemostlyorfullyevaluable,theproductofanEAmightbeaprogramevaluationproposalorevaluationplan.Ontheotherhand,iftheresultsindicatedthattheefforthadlimitedevaluabilityorwas mostly evaluable, it might make sense to postpone an evaluation and concentrate instead onstrengthening theprogram inways the EAprocess has suggested are needed in order to increase itsreadinessforevaluation.29
Annex 6 provides a check-list for Evaluability Assessments of Peacebuilding Programs based on RPPcriteria.
5.2KeyelementsofanevaluabilityassessmentbasedonRPPcriteria:
TheEAinvolvesareviewofkeyprogramdimensionsandassessmentagainstthefollowingcriteria:
1. Strengthofprogramdesign;2. Availabilityofdataandinformation;3. Conducivenessofthecontext.
Thebelowtablepointsoutkeyprogrammingdimensionsthatdeterminewhetheraninitiativeis‘readyforanevaluation’and‘worthyofanevaluation’–fromapeacebuildingperspective.
ThiscanbeusedinconjunctionwithAnnex6,whichprovidesa‘trafficlight’checklistinordertodeterminethelevelofevaluabilityofanintervention.
29Reimann,ChigasandWoodrow,EvaluabilityAssessmentsinPeacebuildingProgramming,5
29
EATable1:ReadyforevaluationANDworthyofevaluation(fromapeacebuildingperspective)?
EvaluabilityRubrics
ProgramDimensionsProgramDesign DataAvailability ContextualConduciveness
Readyforevaluation
Cleardefinitionofintendedchangebasedonconflictanalysis Capacitytoprovidedata Adequatesecurityand
access
Thoroughstakeholderidentification Repeatableanddurablebaselinemeasures Availabilityofstakeholders
Plausibletheoriesofchange SMARToutcomeindicators Sufficientresources
Goals&objectivesfitneeds(relevance/appropriateness)
Monitoringsystemlinkedtodecision-makingoncoursecorrectionsorprogramsteering(adaptiveprogramming)
Conflictsensitivity
Participantbuy-in
Worthyofevaluation(fromapeacebuildingperspective)
Important&significantinitiative(i.e.effectonPWL–basedonconflictanalysis-orimportancetothefieldofpeacebuilding)
Necessarydataisobtainableatreasonablecosts
Anticipatedutilityoffindingsisworththeassociatedcosts
BasedonworkMarkRogersdidforCDAin2014,and(Reimann,ChigasandWoodrow2012b)
30
Thefollowingtablesprovidekeycriteria,andexamplesofcommongapsandweaknessesfoundinmanyprograms.
EATable2:ProgramDesign–EvaluabilityAssessment(Criteria1)
Evaluabilitycriteriaforpeacebuilding
programming
Commongaps/
Weaknessesinprogramstrategyandlogic
1 ConflictAnalysis
§ Isconductedatthebeginningofdesignandthroughoutprogramimplementationthroughtheintegrationoftheperspectivesofmultiplestakeholders
§ Identifieskeydriversofconflict,andtherelationshipsanddynamicsbetweenstakeholdersandkeyconflictdrivers.
§ Iswelldocumented,updatedthroughoutprogramimplementation,andaccessibletotheteam
§ Isusedasafoundationforprogrammingdecisions(analysis–strategy/programlink).
§ Noconflictanalysisisconductedasthebasisforprogramming.
§ Onlypartialanalysisisconducted.§ Analysisisnotupdated.§ Othertypesofanalysismistakenas‘conflictanalysis’.§ Analysisistoogeneral(contextanalysis),narrow(e.g.
onlysectorspecific),ortoocomprehensive(e.g.notprovidinginformationonlocallevelconflictdrivers.)
§ Analysisisbiasedanddoesnotincludeamultitudeofperspectivesfromdifferentpartsofsociety.
§ Analysisisnotdocumentedand/oraccessibletoallrelevantteammembersandprogrampartners.
§ Analysisisnotlinkedtostrategy.
2 ProgramGoals
§ Clearlydescribethesocio-politicalchangethatisexpected/desired.
§ Areformulatedinaclearandmeasurableway,arespecificandachievablewithinthedescribedtimeframe
§ Goalsaretoobroad/general,statedatthemacro/visionandnotprogramlevel.
§ Goalsareunclear.§ Goalsaretooambitious.§ Goalsaredefinedasactivities,notasdesiredchange.§ Goalsarenotappropriateforthe(conflict)context.
3 Linkagesbetweenactivities,programgoals,andtheoverallobjective/vision
§ Noexplicitlinksexistoronlyimplicitlinksexistconnectingactivities,programgoalsandtheoverallvision/PeaceWritLarge.
§ Only‘hopelines’exist(linksbetweenactivitiesandoutcomesatvariouslevelsthatarebasedonassumptionsonly).
4 Theoriesofchange(TOCs)
§ Areexplicitlyarticulated§ Areclearandrealistic
§ TOCsareimplicitandnotarticulated.§ TOCsarenotsharedwithintheteamand/orwith
partners.§ TOCSarenotrealisticforthescopeoftheprogram.§ TOCsareunclear/fuzzy/don’texplainhowthebroader
changewillbeachieved.
Insightfrompractice.Manyprogramsfocusonthe‘lowhangingfruit’–theyworkwithpartnerswhowanttoengage(notthe‘hardtoreach’),andonprogrammaticareasfamiliartotheorganization.Itisequallyimportanttoworkonwhatneedstobe‘stopped’–keyconflictdriversanddynamicsidentifiedintheconflictanalysis thatnurturetheconflictonanongoingbasis–ongoingsocialexclusion,inter-ethnictensions,mistrustetc.Hence,itiscriticaltobaseprogramdesignandtheoriesofchangeontheoutcomesoftheconflictanalysistoensurerelevance.
31
EATable3:DataAvailability–EvaluabilityAssessment(Criteria2)
Evaluabilitycriteriaforpeacebuildingprograms Commonevaluabilitygapsandweaknesses
1 Theoriesofchange(TOCs)
§ Areexplicitlyarticulated§ Areclearandrealistic
§ TOCsareimplicitandnotarticulated.§ TOCsarenotsharedwithintheteamand/or
partners.§ TOCsarenotrealisticforthescopeoftheprogram.§ TOCsareunclear/fuzzy/don’texplainhowthe
broaderchangewillbeachieved.§ “Theoriesofchange”indesigndonotmatch
“theoriesinuse”–anEAservesthepurposeofrevealingwhethertherearegapsbetweenwhatwasdesignedandwhatisbeingimplemented.
2 Baselines
§ Arecompleted,basedonconflictanalysis§ Areadaptedtothecontext,e.g.recognizing
volatilityinconflictcontexts
§ Baselinesarenotbasedonfindingsfromconflictanalysis.
§ Baselinesareincomplete.§ Baseline’snotdefinedinawaythatwouldallowfor
monitoringofchangesinthebroaderconflictcontext.
3 ProgramMonitoringSystem
§ Isestablishedwithclearresponsibilitieswithinandacrossprogramteams
§ Issetuptoinformchangesinprogrammingandsupportflexibleprogramming
§ Measuresbroaderchangesinconflictcontext§ Monitorsunintendedimpactsoftheprogram.
§ MonitoringSystemsareconsideredonlyafunctionofprogrammanagementanddonoraccountability–notasakeyelementofadaptiveandflexibleprogramminginconflictcontexts.
§ Insightsfrommonitoringdonotinformdecision-makingandprogramchanges.
§ Aggregatedanalysisofmonitoringdataflawedornon-existent.Trendsanalysisnotcommunicatedtokeydecision-makersintimelyandactionableway.
§ Monitoringsystemonlymonitorsdirectprogramresults,andnotlargerimpactsonconflictcontext.
§ Monitoringsystemdoesnotmeasureunintended(negative)impacts(conflictsensitivity).
4 Indicators§ Areclear,realisticandmeasurable§ Arequalitativeandquantitative§ MeetsSMART30criteria,especiallyatthe
outcomelevel§ Expressthebroaderchangeintheconflict
contextthattheprogramintendstoachieve§ Aresetuptomeasureconflictsensitivity
considerations
§ IndicatorsarenotdesignedbasedonSMARTcriteria(especiallyattheoutcomelevel).
§ Indicatorsareunrealistic,un-measurableorunclear.§ Noindicatorsexist.§ Onlyquantitativeindicators.§ Arenotdesignedtocapturethebroaderchangeon
theconflictcontext.§ Donottakeintoaccountmeasurementof
unintended(negative)impacts(conflictsensitivity).
5 Accesstostakeholders§ Forprogramdataverification§ Forongoingfeed-backfromprogram
participantsandpartners
§ Noaccesstoprogrampartnersandotherlocalstakeholders.
§ Onlyaccesstothe‘easytoreach’.§ Nomechanismsinplacetocreativelymanage
remoteimplementationscenarios.
30SMART=specific,measurable,achievable/attainable,result-oriented,andtime-bound.FormorebackgroundonSMARTindicatorsseehttp://dmeforpeace.org/sites/default/files/3.9%20Indicators.pdf
32
Evaluabilitycriteriaforpeacebuildingprograms Commonevaluabilitygapsandweaknesses
§ Toassessbroaderchangesintheconflictcontextandpossibleunintended(negative)impacts
§ Tojointlyagreeonprogramchangesandadaption
§ Noongoingfeed-backmechanismsinplacewithpartnersandstakeholders.
EATable4:Conducivenessofthecontext–EvaluabilityAssessment(Criteria3)
Evaluabilitycriteriaforpeacebuildingprograms Commonevaluabilitygapsandweaknesses
1 Generalconditionsarefavorable(weather,security,availabilityofstakeholders,currentpoliticaleventsetc.)
§ Limitedaccessb/cofweatherorsecurityrisks.§ Currentpoliticalclimateand/oreventsimpede
access.§ Restrictedaccesstoprogrampartnersand/or
localstakeholders
2 Financialresourcesavailabletoconductevaluation,includinglogistics
§ Evaluationplannedasan‘afterthought’andnotasakeyelementoftheprogram,leadingtolimitedresourcesbeingavailable.
§ Limitedfinancialresourcesimpacttherangeofstakeholdersbeingconsulted(e.g.inremotelocations).
3 Availablestaffcapacitiestoparticipatein,shape,andfollow-uponanevaluation
§ Limitedstaffawarenessaboutkeyelementsneededtomakepeacebuildingprogramsevaluable(conflictanalysis,DNHprinciples,basiscriteriaofstrategydesignandprogramlogic,theoriesofchangeetc.).
§ Nocapacitytofollow-uponimplementationofevaluationfindings.
4 Commitment/Internallydrivenprocess § BothEAsandevaluationsoftendonor-driven,ordrivenbySr.Managementwithoutbuy-infromstaff.
§ Partnersnotinvolved.§ UnderstandingofwhatanEAorevaluationis,
andhowthefindingswillbeusedvaryandimpactthesuccessoftheEAorevaluation.
5 Evaluationprocessdesignedinaconflict-sensitivemanner
§ Conflict-sensitive/DNHprinciplesarenotappliedtothedesignoftheEAortheevaluationprocess,e.g.inrelationtotiming,whoisinvolved/selectionofinterviewees,locationofEAand/orevaluationetc.
33
KeypreparationstepsforanEvaluabilityAssessmentforpeacebuildingprograms
Step1:ClarifyoverallpurposeoftheEAwiththeprogramteamandmanagement.Clarifywhatwillhappenwiththeresultsoftheprocess andhowthe findingswillbeused internally,withpartners, and possibly with donors. Identify capacitydevelopment needs on RPP and DNH tools, on M&E ofpeacebuilding, and approaches as part of the process.DeterminehowmuchcapacitydevelopmentcanbedoneaspartoftheEA–andwhatneedstobeachievedinaseparateprocess.
Step2:ClarifythetypeofevaluationthattheEAprecedes.
Step3:GetSeniorManagementonboardwiththeprocessand/ortoparticipate(atleastinpartsofit).Re-confirmavailabilityofbudgetfortheprocess(stafftime,venue,facilitator).
Step3:DevelopTORforexternalfacilitatorandrecruitsomeonewiththerightskillset–i.e.,mixofM&E,peacebuilding,facilitation,andcapacitydevelopmentskills.
Step4:IdentifyparticipantsintheEA:team,partners,donors.Programpartnersandlocalpartnersneedtobeengagedwithacertainlevelofindependence.EngageprogramstaffandpartnersinarticulatingEAquestionstobuildownershipandcapacity.Clarifyexpectationswiththeteam:anopenmindset,activeparticipation,willing toconstructivelychallenge themselvesandcolleagues,ability toengagewith theresultsandengageinfollow-up.
Step5:EAFacilitator:Reviewrelevantdocuments,programM&Esystem,availableprogramdata.
Step6:JointlydeterminewiththefacilitatorwhetherthereareanyelementsoftheEAthatshouldreceivea particular focus. Clarify capacity development expectations on RPP and DNH and how theywill beaddressedthroughEA–orafollow-upprocess.ConsultSeniorManagementonthesequestions.
Step7:Logistics:MakesuresufficienttimeisbeingsetasidefortheEAandthatSr.Managementreleasesstafffromotherobligationsduringtheexercise.Arrangeforaconducivevenue–ifintheoffice,ensurepeople commit toparticipating (asopposed tobeingpulledback to their offices todootherworkorrespondtoemails).
Step8:ConductEAthroughatwo-stepprocess:i)individualinterviewsconductedindependently(e.g.,aswouldbedoneforanevaluation),and(ii) facilitatedworkshopwithall relevantparticipants, includingteam,partnersanddonorstoreviewthethreekeyareasofanEA(programdesign,dataavailability,andconducivenessofcontext).
Step9:SummarizefindingsfromEAincludingafilled-inEAcheck-list(seeAnnex6)
Step10:DevelopstrategyandimplementationplanonhowtousefindingsfromEA:Istheprogramreadyforaformalevaluation?Forwhattypeofevaluationandwhen?Ifnot,whichprogramelementsneedtobe strengthened for the program to become ready? Determine engagement of partners and donors.Determinefuturecapacitydevelopmentneedsandplans.
Moreresourcesonethicalandconflict-sensitivityquestionsinrelationtoformalevaluations:
® dmeforpeace.org/evaluationunder"Whatareethicalandconflict-sensitivityissuesinevaluation,andhowdoImanagethem?”
34
Insightfrompractice.Clarifyingthepurposeoftheassessmentprocessandhowtheresultswillbeusediscritical.Often,programteamsandprogrampartners arehesitant todiscussdifficulties inprogramstrategyandimplementation–andthereissignificantfearwithinteamsaboutmakingmistakes.Thisisa challenge especially in organizations that do not have an explicit culture of learning or adaptivepractice–orothertypesofstaffincentivesthatarecontrarytoalearningculture.Theremightbefearof negative repercussions, or negative reactions from donors if challenges are discussed openly.Ensuring that teams (including local partners) understand that the focusof theseassessments is onlearningandnotonaccountabilityiscritical.SeniorManagementbuy-intocommunicatetheprocessofhowfindingswillbeused,includingrelatedfollow-upprocesseswithbothprogramstaffandprogrampartners, iscritical inthisregard.Alloftheseoptionsrequirestrongsupportfromtheteamandlocalpartners,andshouldnotbeconductedifthereismajoruncertaintyaboutthepurposeand/ornatureoftheexercise.
35
6.StrategyandProgramReflectionExercise
6.1Overview–PurposeofaStrategy/ProgramReflectionExercise
ComparedtoaProgramQualityAssessmentorEvaluabilityAssessment,aStrategyorProgramReflectionExerciseisoftenamoreinformalandinternalprocessthatcanbeconductedatprettymuchanymomentofprogramdesignandimplementation.
ThepurposesofaStrategyandProgramReflectionExerciseareto:
§ improvespecificelementsofprogramstrategy,design,andimplementation,
§ maximizepotentialforprogramtocontributetoPeaceWritLarge,andto
§ contributetoacommonunderstandingwithintheteam,andpossiblyprogrampartners,aboutkeyelementsofeffectiveandrelevantpeacebuildingprogramming.
6.2KeystepsoftheStrategyandProgramReflectionExercisebasedonRPPcriteria
ReflectionExerciseStepsPossibleimplicationsofthisstepduring
Strategyand/orProgramDesign
Possibleimplicationsofthisstep
duringImplementation
Step1:Reviewconflictanalysis Domoreanalysisifconclusionisthatconflictanalysisisnotgoodenough.Possiblyadaptprogramdesignifcontexthaschangedrecently.
Updateconflictanalysisandpossiblymodifystrategyandactivitiesifcontexthaschanged.
Step2:Reviewprogramgoal Possiblymodifyprogramgoal(andrelatedactivities).
Reviewandpossiblymodifygoal-activityalignment(istheprogrammakingprogresstowardsthegoal?).Possiblegoalre-designed(e.g.inchangingcontext)?
Step3:Identifyprogramactivities,intendedchanges,andtheoriesofchange
Jointarticulationofkeycomponentsoftheprogram(activities,theoriesofchange).Validateassumptionsandjointapproach.
Takestockofpossibledifferencesbetweendesignandimplementation:aretheredifferencesandwhy,doestheteamhavethesameunderstanding?
Step4:Plottheprogramgoal,activities,andchangesontotheRPPmatrix
DesignprogramstrategyusingtheRPPmatrix.
ReviewprogramstrategyandpossiblyupdateitusingtheRPPmatrix.
Step5:Assesstheprogram’stheoriesofchange
Developrealistictheoriesofchange;revealunderlyingassumptionswithintheteamandwithpartners.
Review&possiblymodifytheoriesofchange
Step6:Exploreprogramlogic Defineandtestdesignlogic. Reviewandpossiblyadaptprogramlogic.
Step7:Assessconflict-sensitivedesignand/orimplementationoftheengagement
Designconflictsensitivitymechanismsfortheengagement.
Reviewwhetherexistingconflictsensitivitymechanismsareworkingandappropriate,orwhetheradaptionisrequired.
Step8:Reflect&recommendchangesinprogramdesignand/orimplementation
Agreeondesignchangesandrelatedresponsibilities,involvingteam,partners,anddonors.
Agreeonchangesrequiredandrelatedresponsibilities,involvingteam,partners,anddonors.
36
Noteveryprogramneedstoexamineallthesesteps.Determinewhetherallstepsneedtobeperformed,orwhichonesshouldbeselected.Thebelowsectionillustrateskeyquestionsthatprogramteamsandpartnerscanusetoreflectoneachoftheabovesteps.
ReflectionExercise–Step1:Reviewtheconflictanalysis
Programdesignandperiodicreviewshouldbebasedonanup-to-dateconflictanalysis.Thisstepisaimedatensuringthe“relevance”oftheprogram—thatis,whetheritisworkingontherightissuewiththerightpeopleattherighttimeusinganappropriatemethodology.
§ Istheavailableanalysiscurrent(withinafewmonths)? Hasthesituationchangedsignificantlysincepreviousanalyses? Is theavailableanalysis in factageneral ‘context’analysis rather thanaconflictanalysis?
§ Doestheanalysisidentifythekeydrivingfactorsofconflict(bothissuesandpeople)?[Asareminder:Keydrivingfactorsarefactorswithoutwhichtheconflictwouldnotexistorwouldbesignificantlydifferent.Keyactorsarepeopleorgroupsthatcansignificantlyinfluencetheconflictdynamics.]
§ Doestheanalysisexploreordepicttherelationshipsamongfactorsand/oramongfactorsandactors?
§ Doestheanalysisidentifyactualorpotentialfactorsforpeace?Whataretheforcesinthesituationthatcanbebuiltupontopromotemovementtowardspeace?Whatconnectspeopleacrossconflictlines?Whoexercisesleadershipforpeaceandhow?
§ Whatneedstochange?Whoorwhatneedstochangetotransformanegativeanddestructivedynamicintoamoreconstructiveone?Doestheanalysisidentifywhatmustabsolutelybestoppedorreducedbeforepeaceeffortshaveachancetomakeanimpact?
§ Doestheanalysisconsiderpastandongoingeffortsatpeace,andwhatcanormustbelearnedfromtheirperceivedeffectiveness—orlackthereof?
§ Doestheanalysissuggestpossiblepointsofleveragetocreatechangeinconflictdynamics?Giventhenatureofyourorganization,whatdoyouseeaspointsofleverage,andwhy?Toeffectchange,willyoureffortsalonebesufficient,ordotheyneedtobesupportedandcomplementedbyotherefforts?Ifso,bywhom?
§ If theprogram isalreadybeing implemented:Hasanupdatedconflictanalysisbeenperformed?Doesthebasicapproachorprogramfocusneedtochangeasaresult?
WhoseAnalysis?Ajointprocessofconflictanalysisandrelatedreflectionwiththeteamandprogrampartnershelpstoestablishacommonunderstandingaboutthekeydriversofconflict–andwheretheremightbedifferenceswithintheteamandwithpartnersonwhatthespecificconflictdynamicsare,andhow that difference in understanding might impact program design and implementation. Often,differencesinvisionaboutthelargerchangeaprogramintendstoachieve(workingtowardsPeaceWritLarge)isbasedonalackofsystematic,andsharedanalysis–eventhoughpeopleoftenassume‘theyareonthesamepage’,whentheyactuallyhaveadifferentunderstandingof‘theproblem’.Agreeingontheunitofanalysisisequallyimportant:doteamandpartnersneedabroadmacro-levelviewofnationalconflictdynamicsoristheanalysisofasub-regionmoreimportant?Wouldasystemicconflictanalysishelptore-assessandre-confirmleveragepointsforchangeandprogrammaticentrypoints?
.
37
ReflectionExercise–Step2:Reviewtheprogramgoal
Programgoalsshouldbearticulatedasintendedchangesatanappropriatelevelofambition!
§ Istheprogramgoal,asstated,‘robust’?Isitchange-oriented,realistic,timeconscious?
§ Istheprogramgoaltoogeneral(atthelong-termvisionlevel)ortooambitious(over-claiming)?Oristheprogramgoaltoospecific—thatis,isitmorelikeanactivity?
§ Howdoestheprogramgoalrelatetothedynamicsofconflictandpeaceasanalyzed? If there isnoapparentlinktothekeydriversofconflictandpeace,theprogrammaybeworthwhile,butwillnothaveapeacebuildingimpact.
§ Iftheprogramgoalenvisionschangeatthelocallevelorofanon-criticalcomponentoftheconflictandpeacedynamics(peacewritlittle),howmightitcreatelinkagestowiderpeaceatthesocietallevel(PeaceWritLarge)intheparticularcontext?
§ Iftheprogramgoalisstatedas‘acontributionto[anelementofpeace]’,isitclearhowitwillmakesuchacontributionandhowtoobserveormeasureit?
§ Doestheprogramgoalseekchangesatthesocio-politicallevel?Peaceeffortsthatfocusonlyonchangeattheindividual-personallevelanddonotlinkthoseeffortstochangeatthesocio-politicallevelwillhavenodiscernibleimpactonpeace.
§ How does the program goal relate to any of the RPP Building Blocks for Peace (‘Criteria ofEffectiveness’)?
§ Ifaprogram isalreadybeing implemented: Is theprogrammakingreasonableprogress towardsthegoal?Whatkindoffeedback(monitoringdata)indicatessuchprogress?Isthegoalstillappropriate—orhasthesituationchangedsignificantly,requiringredesign?
ReflectionExercise-Step3:Identifytheprogramactivities,intendedchangesandtheoriesofchange
ThisstepusestheRPPprogramplanningchartshowingactivities,expectedchanges,theoryofchange,andotherassumptions(seeAnnex3).
§ Inthetoprowofthechart,enterthegoal/objectiveoftheprogram/projectandtheassociatedoveralltheoryofchangeatthislevel.
§ Identifyfiveorsixkeyactivitiesinthisproject/program?Entertheminthefirstcolumnoftheprogramplanningchart.
§ Intheothercolumns,identifytheactualorexpectedchangefromeachindividualactivity,aswellasthetheoryofchangeandotherassumptionsassociatedwiththeactivity.
ReflectionExercise-Step4:UsetheRPPMatrixtoassessprogramstrategy:Plottheprogramgoal,activities,andchangesontotheRPPmatrix(seeAnnexes2.1and2.2)
§ Startwith locatingthegoalontheRPPmatrix. Is itattheIndividual/Personal levelofchange,ortheSocio-Politicallevel?Isitmoreintherealmof‘MorePeople’or‘KeyPeople’?
§ Plot the program activities and their intended changes onto the matrix. Use different visualidentifications for ‘activities’ and ‘changes’. (Note: for a program already being implemented,
TheMovingGoalPost.Oftentherearedifferentviewswithinaprogramteam(includinginternationalstaff,localstaff,partners,localstakeholders)onwhattheprogramgoalis–andhowbesttoachieveit.Adeliberateprocessof jointreflectionandgoalarticulationhelpstoputdifferentunderstandingsonthetableandtodevelopconsensusonarealisticandachievablegoalwithintheavailablescopeofwork.Inputandguidancefromthosewithanin-depthunderstandingofthelocalcontextiscritical.
38
completed activities and actual results can be plotted, as well as further planned activities andexpectedresults.)
ReflectionExercise-Step5:Analyzetheprogram’stheoriesofchange
Analyzing theprogramtheoryofchangewillhelp toseehowtheconflictanalysis isconnected to theprogramgoal,andultimately,toPeaceWritLarge.Itwillalsohelptheteam,partners,anddonorsseewhethertheinitiativeisonitswaytocontributingtoPeaceWritLargebeyondthelifeofprogram,andwhetherthereareanyassumptionsmadethatwouldneedtobeaddressedintheprogramstrategy.
§ Arethetheoriesofchangeappropriateandrealisticinthecontext?Willchangecomeaboutinthewaysenvisionedasaresultoftheplannedactivities?
§ How would successful achievement of the program goal make a significant contribution to therealizationofPeaceWritLarge?Whatisthetheoryofchangeatthislevel?
§ If the program is already being implemented: Have the activities completed so far resulted in theexpected changes? Are there any unexpected positive or negative outcomes? Are the theories ofchangeprovingviableinthecontext—orisrethinkingindicated?
ReflectionExercise-Step6:Exploretheprogramlogic
This sectionuses theRPPprogramplanningchart,and theRPPMatrix together (Annexes3and4). Inadditiontoreviewingthetheoryofchangeassociatedwithindividualactivitiesinthepreviousstep,itisimportanttomakesurethattheoverallprogramwilladduptotheintendedgoal.
§ Examinethelogicbetweentheactivitiesandthegoal.Wouldachievementoftheactivitiesleadtothegoal?Isanythingmissing?
§ Are there unexamined assumptions underlying the links between the different activities, such aswillingness,availability,externaleventsetc.?
§ Whatkindsofobstaclesmighttheprojectencounterinitsimplementation?Who/whatmightgetintheway?
§ Are“hopelines”revealedontheMatrixorinthelogicpresentedintheFour-ColumnChart?(“Hopelines”are leaps in logicorgapsbetweenactivitiesanddesiredresults—depictedontheMatrixbydottedlines.)Howmighthopelinesbeconvertedtodesiredchanges?
§ If the program is already being implemented: Is the program on track to achieve itsgoal/objective? Have new gaps in program logic or other obstacles appeared during
ActivityDisharmony.Staffandprogrampartnersoftenworktowardanagreed-upongoalinsurprisinglydifferent ways. This compromises program quality and causes friction within a team, and possiblyconfusionwithpartnersandwithdonors.Often,theunderlyingreasonisadifferentunderstandingoftheories of change. Articulating the different assumptions team members, partners, other localstakeholders,anddonorsmakeonhowtoachievetheenvisagedchangehelpsteamsandallpartnersinvolvedmoveinasimilardirection.
TheBusinessasUsualTrap.Manyprogramsareexcellentatimplementingcertainstrategies.Butoftentheworkhastakenona“cookiecutter”pattern,wherebydifferentproblemsarealltackledviasimilarstrategiesandactivities(e.g.dialogueeffortsasa‘recipe’foreverything),ratherthanbeingaddressedbytailoredinterventionsspecifictothe‘problem’orthecontext-specificconflictdriver.Reviewingtheunderlying analysis and related key drivers of conflict, as well as using the RPP Matrix to revealassumptions,linkages(andthelackthereof)inprogramlogichelpstorevealhowtheprogramcanbemademorerelevanttotheconflictcontext.Inclusionoflocalpartnersinthisprocessiscritical.
39
implementation, requiringadjustments in the futureplannedactivitiesoranewapproach?
ReflectionExercise-Step7:Assessingconflict-sensitivedesignand/orimplementationoftheinitiative.Istheinitiativeawareoftheactualorpotentialunintendednegativeimpactsitmightcause?
§ Hastheprogramdesignexaminedcommoncausesofunintendednegativeeffects,suchase.g.thechoiceofprogrampartners,contractors,suppliers,locationoftheengagement,distributionofbenefits,timingoftheprogramming?
§ Does the initiative consider conflict-sensitivity considerations (mainly possible unintended negativeimpacts)initsM&Esystem?
§ Dostaffandpartnershaveskillsinconflict-sensitiveprogramimplementationand/orbeentrainedinDoNoHarmapproaches?
ReflectionExercise-Step8:Reflect and recommend changes in programdesign and/orimplementation
The fundamental purposeof this reflection exercise is to strengthenprogramdesignor to encouragechanges indirectionor implementation.
§ Whatinsightshaveyougainedregardingthisprogram?Whatchallengeshavebeenraised?
§ Basedonthepreviousstepsandassociatedreflections,howmightthisprogramoritscontinuationbestrengthenedoritsconceptsfurtherelaborated?
§ Doesthisprogramneedtolinkmoreactivelywithotheragencies,withotherefforts?Whichonesandwhy?
§ Reflectonotherdimensionsofthisprogram,consideringthecontextandwhatyouknowaboutthepeaceeffortsofotheractors/agencies:
§ Is thisprogram ‘big’ enough,does it haveenough ‘scale’ tobeable tohave somemeaningfulinfluence?Why?
§ Isthisprogrammovingattherightpace,nottoofastandnottooslow?Why?
§ If thisprogramachievesmeaningful influenceand impact, can thisbe sustained?Whyandbywhom?
§ Ifyouareproposingchangestotheprogramdesignorimplementationprocess,areyouconfidentthattheredesignedprogramwillindeedbestrongerormoreeffective?Why?
§ Willitbenecessarytoseekapprovalforprogramdesignchangesfromothersintheorganizationorfromaprimarydonor?Whatwillbethebeststrategyforgainingdonorapprovalforchanges?
§ Finally,assessthisReflectionExerciseitselffromaprocessperspective.Whatwashelpfulornotsohelpful?Whatsuggestionswouldyoumakeforimprovingtheprocess?
The peacebuilding and conflict-sensitivity confusion. Often, there is an assumption that allpeacebuildingprogrammingthatfocusesonaddressingkeydriversofconflictdirectlyisautomaticallyconflictsensitive.Thisisnotthecase.Inanytypeofprogramming,beitdevelopment,humanitarian,orpeacebuilding,thedetailsofaninterventionmatterfromaconflictsensitivityperspective.Theactionsand behaviors within programs, as well as programs themselves can have significant unintendednegativeconsequencesandimpactsontheconflictcontext.It’simportanttoconsidercriticalprogramdetailssuchaschoiceofpartners,location,timing,orchoiceswithinprocurementandhumanresources.
40
KeypreparationstepsforaStrategy/ProgramReflectionExercisebasedonRPPcriteria
Step1:Clarifyoverallpurposeofthereflectionexercisewiththeteam.Stressthelearningaspectofastrategy/programreflectionexercise–theyarenotassessmentsorevaluations.Clarifywhatwillhappenwiththeresultsoftheprocessandhowthefindingswillbeusedinternally,withpartners,andpossiblywithdonors. IdentifycapacitydevelopmentneedsonRPPandDNHtoolsandapproaches,aswellasM&Einpeacebuildingtools,aspartoftheprocess.DeterminehowmuchcapacitydevelopmentcanbedoneaspartoftheStrategy/ProgramReflectionexercise.
Step2:GetSeniorManagementonboardwiththeprocessand/ortoparticipate(atleastinpartsofit).Re-confirmavailablebudget–costsforstafftime,venue(possibly).
Step3:Nominatefacilitatorandclarifyhis/herrole&TOR(inmostcasesaninternalfacilitator,butcanbeexternal).ClarifyinformationneedsofthefacilitatorbeforetheReflectionExercise(documents,bi-lateralconversationsetc.)
Step 4: Identify participants in the Strategy/Program Reflection exercise: team, partners, donors.Programreflectionexercisesmightbebestconductedwiththeteaminternally iftherearesignificantgapsinunderstandingoftheoverallprogramwithintheteam,differenttheoriesofchangeetc.Inclusionof localpartnersand stakeholders,on theotherhand, is critical for anuancedunderstandingof thecontext,overalladviceonfeasibilityoftheprogram,anddevelopmentofajointvisionbetweenprogramstaffandprogrampartners. Itneeds tobedetermined ina context-specificwaywhat levelofdonorengagementisdesired–e.g.,participationofdonorsinpartsoftheprocessaftertheteamhashadanopportunitytodiscussinternallyandwiththeirlocalpartners.Clarifyexpectationswiththeteam:anopenmindset,activeparticipation,willingtochallengethemselvesandcolleaguesconstructively,abletoengagewiththeresultsandengageinfollow-up.
Step5:ReflectionExerciseFacilitatorandprogramteam:Reviewrelevantprogramdocumentation.
Step6:Determine(jointlybetweenfacilitator,teamandpartners)whichelementsoftheprogrammightrequirespecificattentionduringtheStrategy/ProgramReflectionExercise.ClarifycapacitydevelopmentexpectationsonRPPandDNHandhowtheywillbeaddressedthroughthestrategy/programreflectionprocess–orafollow-upprocess.ConsultSr.Managementonthisquestion.
Step7:Arrangeforconducivelogistics:MakesuresufficienttimeisbeingsetasidefortheexerciseandthatSr.Managementreleasesstafffromotherobligationsduringtheexercise.Arrangeforaconducivevenue–ifintheoffice,ensurepeoplecommittoparticipating(asopposedtobeingpulledbacktotheirofficestodootherworkorrespondtoemails)
Step 8: Conduct Reflection Exercise through a facilitated workshop with all relevant participants,includingadditionalbi-lateralinterviewsasneeded.
Step9:SummarizefindingsfromtheReflectionExerciseinashort,action-orientedformat.Capturekeyoutcomesfromtheworkshoptofeedintorevisedprogramdocumentationandasareferenceforfutureexercisesofthiskind(e.g.updatedanalysis,reformulatedgoalsortheoriesofchange,RPPMatrixetc.).
Step 10: Develop a follow-up action plan on how to use thefindingsfromtheReflectionExercise.Inanongoingprocessoflearning, reflection, and program improvement. Determineengagementofpartnersanddonors.Determinefuturecapacitydevelopmentneedsandplans.
Moreinformationonpreparationstepsforformalevaluation,bymeansofcomparison:
® dmeforpeace.org/evaluation-planning
41
7.AdaptivePeacebuildingProgrammingthroughEffectiveFeedback
Loops
Acrosstheinternationalaidsector,thereisagrowinginterestinevaluativethinkingandfeedbackloops.Thereisalsoanincreasingcommitmenttoengagefrontlinestaff,localpartners,communitymembersandprogramparticipantsas“users”ofdata,informationandlessons.Asreflectivepractitioners,ifwetrulystriveforrapidfeedbackandlearningcycles,weneedskillstobeabletothinkincriticalwaysaboutwhatwe do, and observe and learn in the course of designing and implementing programs and initiatives.Listeningbroadlyandintentionally,andsolicitinglocalanalysisandfeedbackareallcriticalrequirementsfor organizations supporting local peacebuilding efforts. An effective feedback process goes beyondcollecting participant satisfaction data using pre-determined questions. Well-designed and context-appropriatefeedbackprocessesprovideanopportunitytoregularlyengagekeyprogramconstituentsinevaluativeconversationsabouttheassumptionsthatunderpininterventions,toquestionthevalidityoftheseassumptionsandprogrammaticchoicesandtooffersuggestionsforprogramqualityimprovementduringimplementationandevaluationphases.
CDA’sresearchoneffectivefeedbackloopsandfactorsthatenablefeedbackutilization31hasproducedcase studies and evidence-based practical guidance to assist donors and operational agencies in thehumanitarian anddevelopment fields. CDAhas identified several problems that apply equally to thepeacebuildingfield:
§ Therearefrequentlymissedopportunitiesforengaginglocalpeople,notonlythroughlisteningto their perspectives, but also by engaging them in analysis of feedback and identification ofprogramchanges.
§ Staff and local partners often need enhanced individual skills in active listening, appreciativeinquiry,anddataanalysis.
§ Skillsandprocessesarenotenough;institutionalstructures,decisionmakingthatincorporatesfeedback,managementsupportandincentivesarenecessary.
§ Increaseduseoftechnologycansupportsustainedandreal-timefeedback loops,oftenasoneelementofanM&Esystem.Buttechnologydoesnotaddressliteracylevels,culturallyappropriatemonitoring/feedbackprocesses,language,andwhoisengagedandwhoisleftout(bygender,age,elitesvs.marginal/vulnerablestatus,etc.)
In2016-2017,CDAiscollaboratingwithpeacebuildingorganizationstodocumenteffectivepracticeswithfeedback and how it supports adaptive programming through improved monitoring and evaluationpractice.
Feedback mechanisms can serve as a powerful means to support evaluative processes such as
Evaluability and ProgramQuality Assessments. For peacebuilding programs, feedback loops broadlyservetwomainpurposes:programqualityimprovementandparticipatorycontextmonitoring.
Programqualityimprovement:Solicitedandunsolicitedfeedbackcanbegatheredtoinformreal-timeprogram improvements and adaptation. To this end, feedback is sought on the quality andappropriatenessofprograminterventions,staffperformanceandprogramresults.Feedbackisgatheredaspartofroutinemonitoringprocessesorbyestablishingadditional,accessibleandconfidentialfeedback
31“EffectivelyUtilizingFeedback,”CDACollaborativeLearningProjects,http://cdacollaborative.org/cdaproject/effectively-utilizing-feedback/.
42
channels.Suchpracticeshavebecomecommonplaceinmanyhumanitarianoperations,particularlytheuseofcomplaintsresponseandgrievancemechanisms.32Challengesremaininactiveconflictareaswhererestrictedaccessandsecurityconcernsprohibitregularcommunicationchannels.
Inadditiontofeedbackonprogramquality,localviewscanbesoughtonbroader,strategicareastoinformstrategyreviewforprograms,sectorsorcountryspecificpolicies.Bothprogram-levelandstrategy-levelfeedbackloopsrequireanintentionalandpurposefulapproachtofeedbackcollection,acknowledgement,analysisandmakingsenseofdata,opinionsandperceptions.ProgramQualityAssessmentscanbenefitfromthisadditionaldatacollectionmethod.PQAfacilitatorscanexamineaccumulatedfeedbackgatheredonprogramqualityanduseformalandinformalfeedbackchannelstogatheradditionalperceptionsfromawiderrangeofstakeholders,toincludeinformedobservers.
Participatorycontextmonitoring.Organizationsseekingtoidentifyunintendedandunanticipatedeffectsof their programs need to establish feedback channels and practices that reach beyond their targetprogramparticipants.Thisisparticularlytruewhenover-relianceonindicator-basedmethodologiescanresultinoversightofdynamicchangesinthecontextandaprogram’seffectsonthelocalcontext.Well-functioningfeedbackmechanismscancaptureandrespondtoreal-timeinformationaboutunintendedimpactsofprogramsoninter-groupandintra-grouprelationsandavoiddoingharm.
Afunctioningfeedbackloopgoesbeyondfeedbackcollectionandanalysis.Itrequiresaresponseand/orcorrectiveaction,inotherwords--theclosingoftheloop.33Ultimately,forfeedbackloopstobeeffective,thefeedbackneedstobeutilizedindecision-makingandevaluativeprocesses.
32See(CDACollaborativeLearningProjects2011)33See(Bonino,JeanandKnox-Clarke2014)and(Bonino,JeanandKnox-Clarke2014b)
43
8.SummaryOverviewofOptionscoveredinthisGuide
TheGuide provides a selectmenuof three options for reviewing and strengthening of peacebuildinginterventions,namely:
1. ProgramQualityAssessments;2. EvaluabilityAssessments;3. StrategyandProgramReflectionExercises.
Thetableonthenexttwopagesprovidesanoverviewoftheobjectives,characteristics,purposesandbenefitsofeachoption.
44
ComparisonofcharacteristicsoftheStrategyandProgramReflectionExercise,PQA,andEAvis-à-visEvaluation
StrategyandProgramReflectionExercise ProgramQualityAssessment(PQA) EvaluabilityAssessment(EA) Evaluation(notcoveredinthisGuide)
Purpose(s)
§ Improvespecificelementsofprogram
strategyordesign
§ Maximizepotentialforprogramto
contributetoPeaceWritLarge
§ Contributetoacommonunderstanding
withintheteam(andpossiblywith
partners)aboutkeyelementsof
effectiveandrelevantpeacebuilding
programming.
§ Learnaboutqualityofprogram
design
§ Adapt/improveprogramdesign
andimplementationthrough
applicationofaclearsetof
professionalstandardsand
criteria
§ Maximizepotentialfor
programtocontributetoPeace
WritLarge.
§ Assesswhetheraprogramisreadyforaformal
evaluation
§ Identifyareasforimprovementin(specifically)
datacollection,programlogic,andsupport
evaluationplanning.
§ Identifyandassessworthof
results/outcomesofprogram
§ Learnaboutqualityandvalueof
program,includingareasandoptions
forimprovement(ifformative
evaluation)
§ Fulfillobligationsofaccountability(to
donors,toparticipants,communities,
ororganizations,etc.)
Timing
§ Duringprogram/designphase
§ Atkeymomentsduring
implementation.
§ Mid-term
§ Limiteduseatthebeginningof
program,butcanbeusedto
validatetheoriesofchangeand
programstrategywithteams
andprogrampartners
§ Possiblyusefulatendofa
projecttodrawlessonsfor
futureengagement.
§ Beforeaformalevaluation-ideallyonceitis
knownwhattypeofevaluationwillbe
conducted
§ Mid-termreviewstagetoidentifyareasfor
improvementbeforeconductinganevaluation.
§ Atendofprojectorprogram
(summative)or
§ Mid-term(formative,developmental)
or
§ Throughout(developmental)
AssessmentCriteria
§ Notan‘assessment’
§ RPPandDoNoHarmconceptsand
toolsrelatedtoimpactsonPeaceWrit
Largeandconflict-sensitivedesign&
implementation.
§ Qualityanduseofconflict
analysisinprogramming
§ Articulationofprogramgoals
§ Theory/iesofchange
§ Programstrategy&logic
§ StrengthofM&Esystem–
relevancefromapeacebuilding
perspective
§ Applicationofconflict
sensitivity.
§ Qualityofprogramdesign
§ Conducivenessofcontext
§ Dataavailability.
§ OECDDACevaluationcriteria(impact,
relevance,sustainability,efficiency,
andeffectiveness)
§ Contextually-relevantstandardsof
achievementsetbyprogram/project
DataNeeds
§ Reliesonknowledgeandexperienceof
participants
§ Generalunderstandingoftheoverall
(conflict)contextisimportant.
§ Reliesonprogram’smonitoring
data,documentreviewand
someinterviewswithprogram
teamandpartners.
§ Publiclyavailablerelevantdata
(e.g.,violencestatistics,or
externalconflictanalyses).
§ Reviewstheavailabilityofdata,including
baselines(basedonconflictanalysis)
§ ReviewsstrengthofM&Esystems,relevanceof
collecteddata,accesstostakeholderstocollect
data.
§ Checklistsandsimplepresence/absencetests.
§ Testing,smallscalesamplingwithoutregardto
generalizability
§ Standardsfordata
collection/methodologyarefollowed
§ Triangulationofevidencesought.
§ Mixedmethodswherefeasible.
45
StrategyandProgramReflectionExercise ProgramQualityAssessment(PQA) EvaluabilityAssessment(EA) Evaluation(notcoveredinthisGuide)
Whoconducts
§ Inmostcasesinternalfacilitator
§ Externalfacilitatorrecommendedifno
internalfacilitationskillsavailable.
§ Inmostcases,external
facilitatorwithsubstantive
knowledgeofprogramarea,
programassessment
experience,andfacilitation
skills.
§ Ifinternalfacilitator(e.g.team
memberofprogrampartner)is
chosen(self-assessment),
personneedsacertainlevelof
independenceinadditionto
therightskillset.
§ IdeallyexternalfacilitatorinclosecoordinationwithprogramteamsandSeniormanagement.
§ Internalfacilitationpossible.
§ Externalevaluatorwithevaluation
expertiseandcredentialsand
understandingoftheprogramarea.
§ Self-evaluationusingsameskills,
standardsandtechniquesasother
evaluationsalsopossibledependingon
purpose.
§ Blendedexternal/internalevaluators;
Programteam;Seniormanagement;
Programparticipants;Subjectmatter
experts;Programpartners;Donors;
Hostgovernmentstakeholders;
Spoilers;Otherstakeholders.
WhoParticipates
intheprocess
Programteam;SeniorManagement;Possibly
programpartners;Possiblyanexternal
facilitator;Donor(s).
Facilitator(externalorinternal);
Programteam;SeniorManagement;
ProgramPartners;Donor(s).
Programteam;Externalorinternalfacilitator;Senior
Management;ProgramPartners,Donor(s).Programteam;SeniorManagement;
ProgramPartners;Donor(s);Blended
external/internalevaluators;Senior
management;Subjectmatterexperts;Host
governmentstakeholders;Other
stakeholders.
LevelofCapacitybuilding
KnowledgeaboutbasicRPPandDNH
conceptsrequired,andoftenbuiltinto
processtosomeextent
Capacitybuildingisanintegral
element—thoseimplementing
findingsofqualityassessmentsare
supportedonhowtoapplythe
assessment’sfindings.Criticalto
determinehowmuchcapacitydev.is
neededbytheteamandprogram
partnersaspartofPQA.Some
capacitybuildingonapproachesand
toolsforM&Eofpeacebuilding
mightalsoberequired.
CapacitybuildingandknowledgetransferonRPPand
DNHapproachesnotarequiredelement,butare
oftennecessarytoapplytheassessmentcriteriaina
processwiththeteamandprogrampartners.Some
capacitybuildingonapproachesandtoolsforM&Eof
peacebuildingmightalsoberequired.
§ Capacitybuildingisnotanelementin
mostcases.
§ Learningisanimportantcomponentof
evaluations,ideallycontributingtothe
capacitiesofstaff.Inmanycases,there
isnotsufficienttimebuiltinto
evaluationstofulfilthispurpose.§ Capacitybuildingcomponentmore
pronouncedininternalevaluations.
Resourcesrequired
Limited:resourcestocoverstafftime,venue
andpartnerparticipation,and,wherethere
isexternalfacilitator,feesforhis/her
services.
Medium:resourcestocoverstaff
time+interviews,facilitator,venueMediumastheseareusuallyshortduration:
resourcestocoverstafftime,andfacilitator,
resourcesforfacilitation.
§ MediumtoHigh:resourcesforevaluator/evaluationteamtodesign
andimplementrigorousdatacollection
andanalysisprocess.
§ Highforlargeexternalevaluation.
§ Mediumforblendedteam
§ Limited-mediumforself-evaluation
dependingontheevaluationquestions
anddatacollectionmethodology.
Benefitofthismodality
versusothers
Moreinformalprocessthatcanbe
conductedmorefrequently,islessexpensive
whenexternalfacilitationisomitted.
Strengthensthecapacityofteams
andprogrampartnerstoimprove
programqualityonanongoingbasis.
Iffindingsareusedandaddressed,EAhasthe
potentialtosignificantlyimproveafutureevaluation.Independentevaluationandexternal
assessment;credibilityvis-a-visdonors
9.Recommendationsforpractice
EventhoughdecisionstoconductRPP infusedStrategy/ProgramReflectionExercises,ProgramQualityand/orEvaluabilityAssessmentcanbetakenad-hoc,itisgenerallyadvisabletoplanforsuchprocessesaspartofanagreementofprogrammanagementstepsaspartoftheoverallplanningandprogrammingcycle.Thefollowingrecommendationsarefocusedondesigningprocessesfortheoptionsdealtwithinthisresource:
RECOMMENDATION1:Beclearabout thepurposeand timingof theRPP-infusedStrategy/ProgramReflection Exercise, EA or PQA as well as how the process fits into a larger plan for programimprovement,includingstaffandpartnercapacitydevelopment.
Beforeanyengagementisplanned,therightprocessneedstobedeterminedandexpectationswithintheteamandwithprogrampartnersneedstobeclear(seeTable3).
Someteamsmightbereadyorhaveaclearneedforassessingtheirprogramsagainstclearpeacebuildingprogramqualitycriteria(PQA),othersmightbenefitfromamoreinformalStrategy/ProgramReflectionExercise,whileothersneedtogetreadyforanevaluationandbenefitmostfromanEA.Whateverthefinalchoiceis,teamsneedtobecomfortablewiththerequirementsandconditionsforeachoptionasoutlinedinthisresource.
OnekeyconsiderationishowreadyteamsandprogrampartnersaretojumpintoanyoftheseoptionsrightawaywithoutpriorexposuretoRPPandDNHtools.SometeamsmightbeabletodothisandpickuponsomeofthetoolsastheygothroughaReflectionExercise,PQAorEA.Otherteamsmightbebestadvisedtoconductsometraining,alsojointlywithlocalpartners,oratleastexposurebeforeanyofthemodalitiesarechosen.Itisalsocriticaltoreflectonhoweachofthemodalitieswillwork,basedonthetoolsusedduringprogramdesign–e.g.ifnoconflictanalysiswasdone,whataretheoptionsfore.g.aPQA? If different typesofprogram strategy toolswereused,howdo theyalignwith theRPP-infusedprocesses?
RECOMMENDATION 2: Evaluative processes need to be integrated into organizational policies andprocesses,suchastheplanningandprogrammingcycles,andadequatefinancialresourcesneedtobeallocated.
Evaluative processes need to become part of regular organizational routine. For this to happen,organizationsneedseniormanagementdecisionsandadequatepriority setting.Theyneed to identifyexisting institutional capacities for the facilitation of evaluative processes and reach out for externalsupportwhennecessary.Documentingthelessonsanditerativeprogramadaptationstepsareusefulfromaninstitutionallearningandmemoryperspectiveandcanbearichsourceofdataforexternalevaluationswhenthesetakeplace.
Integratingevaluativeprocessesfromthebeginningisalsocriticalfromafinancialperspective:ifanyoftheoutlinedoptionsinthisresourcecameinonlyasan‘afterthought’whenbudgetshadalreadybeenallocatedandspent,possibilitiesforanyoftheseoptionsbecomemuchmorelimited.Clearallocationofadedicatedbudget (stafftime,costsforexternalfacilitator,meetingvenue, logisticsandtravelwithincountryetc.)isarequirement.
47
Thebelowfigure(CDA,2016)demonstratesonescenarioforsequencingthedifferentevaluativeoptions:this isonewayaprogramcouldintegrateeachoftheoptionsintoitsprogramreflection,learningandimprovementcycle.
RECOMMENDATION3:Seniormanagementbuy-in,partneranddonorengagementarecritical.
ItiscriticaltogetSeniorManagementanddonorbuy-intosupporttheintegrationofevaluativetoolsintopeacebuildingprograms.Anyevaluativeprocessisof limitedvalueifperceivedonlyasatechnicalexercisedrivenbytheimplementationteam.Follow-uptoanyoftheevaluativeprocessesneedstobeensured and must be supported by Senior Management, including difficult decisions about coursecorrection and strategy review. Some donors are already leading on the promotion of evaluativeapproaches,butthereisaneedformoreworkwithothers.Thiscouldincludecloseengagementbetweendonorsregardingrelevantevaluativeapproaches,aswellascloserengagementofdonorsregardingtheintegrationofOECD/DACcriteriaintopoliciesandprogramguidance.
AshighlightedthroughouttheGuide,Partnerinvolvementiscriticalinallthreeprocesses,mostpronouncedforPQAsandEAs.Thelevelofdetail(howmany,whattypesofpartners)needtobedeterminedindividually.
Insight from practice. A strong program management team committed to the reflection and/orassessmentprocess,aswellasbuy-infromSr.Management,anddonorsprovedcriticalduringtheCDA-led,RPP/DNHinfusedPQAsandEAs.Partnerswereinvolvedthroughouttheprocess,andthebenefitsoftheprocessaswellasfollow-upengagementwereclarifiedfromthebeginning.InbothPQAsandoneoftheEAs,theassessmentwasonlythebeginningofalongerprocessofprogramqualityenhancement,andthefindingsandresultsfromthePQAsandEAswereusedasinitialbenchmarksforongoingprogramqualityimprovement.
InthreeoftheEAsandPQAs,donorseitherparticipatedintheassessments,orwereinformedaboutthe outcomes. It was clearly an advantage to get the donor on board and increase mutualunderstanding,jointlearning,andsupportforfutureprogrammingdecisions.
ConflictAnalysis
(inthebeginingand throughouttheprogrammingcycle)
Phase:ProgramDesign
Includeevaluativethinkingfromthebegining; Possiblestrategy/programreflectionexerciseearlyon
Phase:ProgramImplementation
Strategy/ProgramReflectionExercise(PQA)
Phase:Mid-term Review
Strategy/ProgramReflectionExercise(PQA)
Evaluability Assessment
Phase: Evaluation
Isformalevaluationrequired&appropriate?
48
RECOMMENDATION4:DeterminetherightamountofdatacollectionandanalysisforbothEAandPQA.
Akeyquestion inboth EAandPQAprocesses is howmuchdata (in addition toprogram reports andinformation, baselines and other data collected by the program itself) needs to be collected for theEA/PQAassessments.
Fornewerprograms,datacollectionislessapplicable;instead,thefocusisonreviewingprogramdesignandM&Esystems.ForprogramsthatundertakeanEAorPQAlaterintheprogramcycle,thequestionarisesregardingwhattypeofdatathefacilitator/evaluatorshouldcollect–orfocusmoreonthequestionofwhattypeofdataisavailablewithoutcollectinganyofitdirectly.Datacollectionshouldbetargetedandlimited,asnoneoftheseprocessesareactualevaluationsand,especiallyinthecaseoftheEA,shouldnotbecomeone.
Datacollectioncouldbeusefulinseveralareas:
§ Conflictanalysis:Thiscanbecometricky,especiallyincasesinwhichaconflictanalysishadnotbeendoneorupdated.Itmightbenecessarytoconductatleastavalidationexercisewithlocalpartnersonthekeyconflictdriversinordertodeterminethepeacebuildingrelevanceoftheprogram.
§ Understanding theoryof change: Inmany cases, theories of change arenot explicitly articulated.SomeadditionaldatacollectionmightbeneededtoarticulatethetheoryofchangeduringtheEAand/orPQA.
§ Conflicting reports or beliefs about facts related to the program:Where there are conflictingaccountsoffactsrelatedtotheprogram(e.g.,programactivities,outputs,outcomes,theories,etc.),datacollectionmaybeneededforvalidationandclarification.
Sharing of information and data: A big shortfall for peacebuilding evaluation and gathering data forfeedbackmechanismsinfragileandconflictaffectedstatesisthatdataandevaluationfindingsarenotsharedwithinandacrossdifferentorganizationsworkingtowardssimilargoals.Sharinginformationanddata, particularly regarding conflict analysis, has the potential to decrease costs for all organizationsinvolved, increase quality of analyses due to thewide range of data included, aswell as provide thefoundationforjointplanningandimplementation.
Insight from practice. The CDA-facilitated EAs and PQAs included a mix between facilitated multi-stakeholderworkshops (withprogram teams,partners, anddonors)and individual interviews.Whilegroupreflectionbringsoutadditionalelements,andiskeytothere-designofprogramelements,aswellas for capacity-development purposes, individual interviews are critical from a data collectionperspective.Decisionswillhavetobemadeforeachcasespecifically.IndependenceofthePQAorEAfacilitatoriscriticalwhencollectingdata,asistriangulationofinformationindependentoftheamountofdatacollectionchosen.
49
RECOMMENDATION5: Facilitators need tobehighly competent in bothpeacebuildingpractice andevaluativemethods.Itisimportanttochoosetherightfacilitator/facilitationteam.
Thefacilitatorsofsuchevaluativeprocesseswearmany“hats."FacilitatingaStrategy/ProgramReflectionExercise,PQAorEArequiresagreatamountofadaptabilityonthepartofthefacilitators,whoneedtorespondflexiblytoarisingneedsthroughouttherespectiveprocesses,whileatthesametimestayingtruetothePQA/EAbenchmarks.
Facilitators of such processes needmultiple skills to be effective (independent of whether they areexternal or internal): skills in facilitation, coaching, training, peacebuilding and conflict sensitivityexpertise, and a strongM&Ebackground. In addition, for the specific CDAprocesses, knowledge andfluency in the application of RPP and DNH tools was critical. Given the sensitive contexts in whichpeacebuildinghappens,facilitatorsalsoneedtobewellinformedaboutthecontextandself-aware.
Arethereparticularbenefitstointernalversusexternalfacilitation?‘Outsiders’areoftenappreciatedasthey come with a fresh and unbiased perspective, and often bring additional and more in-depthknowledgeonRPPandDNH,evaluativethinking,M&Eskills,and/orfacilitationskills.Atthesametime,engagingexternalfacilitatorsobviouslyhasbudgetimplications.Consideringlocalfacilitators,e.g.fromwithin thenetworkof local stakeholdersofanorganization is anothergoodpossibility.Consequently,buildingregularself-evaluativecomponentsintoprogramsispotentiallyagoodalternative–facilitatedbyoutsidersonlywhenneeded.
Moreresourcesondatacollectionanddataanalysisinpeacebuildingevaluation:
® dmeforpeace.org/evaluation,under,"HowdoIcollectandanalyzedata?”
Insight from practice. In the two EAs facilitated by CDA, an international facilitator worked in a teamapproachwithalocalfacilitator.Whilethisarrangementisinvaluablefromtheperspectiveofbuildingonandworkingwithahigh levelof localcontextknowledgeandsharing facilitation, theset-upneeds tobecarefully planned, including the following questions: profile of the local facilitator, language skills,perceptionsofthelocalfacilitatorwithdifferentpartnergroupsbasedonhis/herbackground,rolewithinthe team, division of labor between international and local facilitator. In any case, both local andinternationalfacilitatorsneedtocollectivelycombinetheskillsetsoutlinedinthischapter.
50
10.Bibliography
ActionAsia.ExaminingRPPasAToolforEvaluation-TheActionAsiaExperience.ResultsfromtheActionAsiaPractitionersForumFebruary8-9,2011.Colombo:ActionAsia.2011.
AmericanEvaluationAssociation.“AmericanEvaluationAssociationGuidingPrinciplesforEvaluators.”AccessedJanuary5,2017.http://www.eval.org/p/cm/ld/fid=51.
Andersen,OleWinckler,BeateBull,andMeganKennedy-Chouane,eds.EvaluationMethodologiesforAidin
Conflict.NewYork,NY:Routledge,2014.
Anderson,MaryB.DoNoHarm:HowAidCanSupportPeace–OrWar.Boulder,CO:LynneRiennerPublishers,1999.
——— OptionsforAidinConflict:LessonsfromFieldExperience.Cambridge,MA:CDACollaborativeLearningProjects,2000.
Anderson,MaryB.,andLaraOlson.ConfrontingWar;CriticalLessonsforPeacePractitioners.Cambridge,MA:CDACollaborativeLearningProjects,2003.
Anderson,MaryB.,andMarshallWallace.OptingOutofWar:Strategiestopreventviolentconflict.Boulder,CO:LynneRiennerPublishers,2013.
Babbitt,Eileen,DianaChigas,andRobertWilkinson,withAMEXInternational.TheoriesandIndicatorsofChange:ConceptsandPrimersforConflictManagementandMitigation.UnitedStatesAgencyforInternationalDevelopment(USAID),2013.
Barnard-Webster,Kiely,NicoleGoddard,andIsabellaJean.“MonitoringandEvaluationofConflictSensitivity:PracticalConsiderations.”ConflictSensitivity:TakingittotheNextLevel,swisspeaceWorkingPaperSeriesno.2(2016):88-95.
BetterEvaluation.AccessedJanuary5,2017.http://betterevaluation.org/.
Blum,Andrew.ImprovingPeacebuildingEvaluation:AWhole-of-FieldApproach.UnitedStatesInstituteofPeace(USIP)SpecialReportno.280(2011).
Bonino,Francesca,withIsabellaJean,andPaulKnoxClarke.ClosingtheLoop-EffectiveFeedbackMechanismsin
HumanitarianContexts.ALNAP-CDAPractitionerGuidance.London:ALNAP/ODI,2014.
——— HumanitarianFeedbackMechanisms:Research,EvidenceandGuidance.ALNAPStudy.London:ALNAP/ODI,2014b.
CDACollaborativeLearningProjects.ReflectingonPeacePractice:ProgramReflectionExercise.Cambridge,MA:CDA,2014.
——— ReflectingonPeacePracticeParticipantTrainingManual.Cambridge,MA:CDA,2013.
——— StrategiesforDealingwiththeHardtoReach.Cambridge,MA:CDA,2012.
51
——— FeedbackMechanismsinInternationalAssistanceOrganizations.Cambridge,MA:CDA,2011.
——— EffectivelyUtilizingFeedback.AccessedJanuary5,2017.http://cdacollaborative.org/cdaproject/effectively-utilizing-feedback/.
Chigas,Diana,andNicoleGoddard.“MonitoringandEvaluationofConflictSensitivity–MethodologicalConsiderations.”EvaluationConnections(August2016):10-11.
Chigas,Diana,andPeterWoodrow.“SystemsThinkinginPeacebuildingEvaluations:ApplicationsinGhana,Guinea-BissauandKosovo,”inEvaluationMethodologiesforAidinConflict.EditedbyOleWincklerAndersen,BeateBull,andMeganKennedy-Chouane.NewYork,NY:Routledge,2013.
Corlazzoli,Vanessa,andJonathanWhite.“BacktoBasics.Acompilationofbestpracticesindesign,monitoringandevaluationinfragileandconflict-affectedenvironments.”AConflict,Crime,andViolenceResults
Initiative(CCVRI)product.London:UKDepartmentforInternationalDevelopment(DFID),2013.
Davies,Rick.PlanningEvaluabilityAssessments.ASynthesisoftheLiteraturewithRecommendations.WorkingPaperno.40.London:UKDepartmentforInternationalDevelopment(DFID),2013.
Davies,Rick,andJessDart.The‘MostSignificantChange’(MSC)Technique:AGuidetoItsUse.2005.
Duggan,Colleen,andKennethBush."TheEthicalTippingPointsofEvaluatorsinConflictZones,"inAmerican
JournalofEvaluation35(2014):1-22.
Ernstorfer,Anita,DianaChigas,andHannahVaughan-Lee.“FromlittletoLarge:WhendoesPeacebuildingAddUp?”inJournalofPeacebuildingandDevelopment10,no.1(2015):72-77.
Ernstorfer,Anita,andIsabellaJean.“AlternativeApproachestoEvaluatingPeacebuildingInitiatives.”Paperpresentedatthe11thEuropeanEvaluationSocietyBiennialConference,Dublin,October1-3,2014.
GlobalPartnershipforthePreventionofArmedConflict(GPPAC)incollaborationwithCDACollaborativeLearningProjectsandtheNorwegianChurchAid.ConflictAnalysisFramework:FieldGuidelinesand
Procedure.TheHague:GPPAC,2015.
Goldwyn,Rachel,andDianaChigas.Monitoringandevaluatingconflictsensitivity.Methodologicalchallengesand
practicalsolutions.AConflict,Crime,andViolenceResultsInitiative(CCVRI)product.London:UKDepartmentforInternationalDevelopment(DFID),2013.
Harris,Sidney.What'sSoFunnyAboutScience?fromAmericanScientist.LosAltos,CA:W.Kaufmann,1997.
InternationalLaborOrganization.“GuidanceNote11:UsingtheEvaluabilityAssessmentTool,”partoftheI-evalResourceKit.Geneva:InternationalLaborOrganization—EvaluationUnit,2011.
Kawano-Chiu,Melanie.StartingontheSamePage:ALessonsReportfromthePeacebuildingEvaluationProject.Washington,D.C.:AllianceforPeacebuilding,2011.
OECD/DAC.EvaluatingPeacebuildingActivitiesinSettingsofConflictandFragility:ImprovingLearningforResults.Paris:OECD,2012.
——— GuidanceonEvaluatingConflictPreventionandPeacebuildingActivities.WorkingDraftforApplication.Paris:OECD,2008
Paffenholz,Thania.“PeacebuildingEvaluation:AssessingtheRelevanceandEffectivenessofPeacebuildingInitiatives:LessonsLearnedfromTestingNewApproachesandMethodologies.”PaperpresentedattheAnnualConventionoftheInternationalStudiesAssociation,Montreal,March16-19,2011.
Patrizi,PatriciaA.,andMichaelQuinnPatton,eds.“EvaluatingStrategy:NewDirectionsforEvaluation.”Jossey-BassQuarterlyReportSeries128(2011).
52
PeacebuildingEvaluationConsortium.“PeacebuildingEvaluationOnlineFieldGuide.”AccessedJanuary5,2017.http://dmeforpeace.org/online-field-Guide.
QuinnPatton,Michael.DevelopmentalEvaluation:ApplyingComplexityConceptstoEnhanceInnovationandUse.NewYork,NY:GuilfordPress,2011.
Reimann,Cordula,DianaChigas,andPeterWoodrow.AnAlternativetoFormalEvaluationofPeacebuilding:
ProgramQualityAssessment.Cambridge,MA:CDACollaborativeLearningProjects,2012.
——— EvaluabilityAssessmentsinPeacebuildingProgramming.Cambridge,MA:CDACollaborativeLearningProjects,2012b.
Rogers,MarkM.EvaluatingImpactinConflictPreventionandPeacebuildingPrograms.Cambridge,MA:CDACollaborativeLearningProjects,2012.
——— EvaluatingRelevanceinConflictPreventionandPeacebuildingPrograms.Cambridge,MA:CDACollaborativeLearningProjects,2012b.
——— cartooninNewRoutes13,no.3(2008):8.
Scharbatke-Church,Cheyanne.PeacebuildingEvaluation:NotYetAllItCouldBe,BerghofHandbookforConflictTransformation.SectionII.Berlin:BerghofFoundation,2011.
——— TheUseofReflectingonPeacePractice(RPP)inPeacebuildingEvaluation.Review&Recommendations.FinalReport.Cambridge,MA:CDACollaborativeLearningProjects,2011b.
——— “PeacebuildingEvaluation:FromInfancytoTeenager.”NewRoutes13,no.3(2008):3-6.
Scharbatke-Church,Cheyanne,andMarkRogers.DesigningforResults:IntegratingMonitoringandEvaluationin
ConflictTransformationPrograms.Washington,D.C.:SearchforCommonGround,2006.
Schirch,Lisa.ConflictAssessmentandPeacebuildingPlanning:TowardAParticipatoryApproachtoHuman
Security.Boulder,CO:KumarianPress,2013.
SearchForCommonGround(SFCG).“Design,MonitoringandEvaluation(DM&E)forPeacePortal.”AccessedJanuary5,2017.http://www.dmeforpeace.org/.
UnitedNationsDevelopmentFundforWomen(UNIFEM)EvaluationUnit.“GuidanceNoteonCarryingOutanEvaluabilityAssessment.”NewYork,NY:UNIFEM,2009.
Wallace,Marshall.FromPrincipletoPractice:AUser’sGuidetoDoNoHarm.Cambridge,MA:CDACollaborativeLearningProjects,2015.
Woodrow,PeterandDianaChigas.“ConnectingtheDots:EvaluatingWhetherandHowProgramsAddressConflictSystems,”inTheNon-LinearityofPeaceProcesses:TheoryandPracticeofSystemicConflict
Transformation.EditedbyHansJ.Giessmann,DanielaKörppen,andNorbertRopers.Opladen&FarmingtonHills:BarbaraBudrichPublishers,2011.
——— ADistinctionwithaDifference:ConflictSensitivityandPeacebuilding.Cambridge,MA:CDACollaborativeLearningProjects,2009.
Woodrow,Peter,withNickOatley.“PracticalApproachestoTheoriesofChangeinConflict,SecurityandJusticePrograms.PartI:Whattheyare,differenttypes,howtodevelopandusethem.”AConflict,Crime,and
ViolenceResultsInitiative(CCVRI)product.London:UKDepartmentforInternationalDevelopment(DFID),2013.
53
11.Annexes:ToolsandHand-outs
Annex1:FiveBuildingBlockstowardsPeace
Fromanalysisofthecasesandpractitionerreflectionontheirownexperiences,theRPPprocessidentifiedfiveintermediateBuildingBlocksthatcansupportprogresstowardsPeaceWritLarge.Thesecanbeusedtoassess,acrossabroadrangeofcontextsandprogrammingapproaches,whetheraprogramismakingameaningfulcontributiontoPeaceWritLarge.TheBuildingBlockscanbeusedinprogramplanningtoensurethatspecificprogramgoalsare linkedtothe largerand long-termgoalof“PeaceWritLarge.” Theycanbeusedduringprogram implementation to reflect on effectiveness and Guide mid-course changes, and as a basis forevaluationaftertheprogramhasbeencompleted.
Theeffortresultsinthecreationorreformofpoliticalinstitutionstohandlegrievancesinsituationswheresuchgrievancesdo,genuinely,drivetheconflict. Asignificantcontributiontopeace is thedevelopmentorsupportforinstitutionsormechanismsthataddressthespecificinequalities,injusticesand other grievances that cause and fuel a conflict. This approach underlines the importance ofmovingbeyondimpactsattheindividualorpersonal(attitudinal,materialoremotional)leveltothesocio-politicallevel.Thisideamustbeappliedinconjunctionwithacontextanalysisidentifyingwhatthe conflict isNOTabout andwhatneeds tobe stopped. To reformorbuild institutions that areunrelatedtotheactualdriversofaspecificconflictwouldbelesseffective.
Theeffortcontributestoamomentumforpeacebycausingparticipantsandcommunitiestodeveloptheirownpeace initiatives in relation tocriticalelementsof contextanalysis. Suchanalysis,andresultingprograms,shouldaddresswhatneedstobestopped,howtoreinforceareaswherepeopleinteractinpositiveways,andtheregionalandinternationaldimensionsoftheconflict.Thisapproachstressestheimportanceof“ownership”andsustainabilityofactionandeffortstobringaboutpeace,aswellascreatingmomentumforpeace,involvingmorepeople.
Theeffortpromptspeopleincreasinglytoresistviolenceandprovocationstoviolence.OnewayofaddressingandincludingKeyPeoplewhopromoteandcontinuetensions(e.g.,warlords,spoilers)istohelpMorePeopledeveloptheabilitytoresistthemanipulationandprovocationsofthesenegativekeypeople. Inmost circumstances,one important aspectofPeaceWrit Large is a significant andsustainedreductioninviolence.ThisBuildingBlockisasteppingstonetothatlong-termgoal.
Theeffortresults inan increase inpeople’ssecurityand intheirsenseofsecurity. Thisapproachreflects positive changes, both at the socio-political level (in people’s public lives) and at theindividual/personallevel,aspeoplegainasenseofsecurity,animportantelementofPWL.Securityandpeople’sperceptionsofitcontainmanydifferentaspects,whichmustbeidentifiedandattainedbasedonthelocalcontext.
The effort results inmeaningful improvement in inter-group relations, reflected in, for example,changesingroupattitudes,publicopinion,socialnorms,orpublicbehaviors.ImprovedrelationshipsbetweenconflictinggroupsconstituteanimportantBuildingBlockforpeace—oftenapreliminarysteptowardsotherinitiatives. Itentailstransformingpolarized(andpolarizing)attitudes,behaviorsandinteractionstomoretolerantandcooperativeones,aspartofaddressingunderlyinggrievancesandbuildingthewillingnessandabilitytoresolveconflictsandsustainpeace.
TheseBuildingBlockscanbestbethoughtofasintermediate-levelbenchmarksofsuccessapplicabletothebroadrangeofpeaceworkbeingdone.
1
2
3
4
5
55
Annex2.2:RPPMatrix‘Plus’
MorePeople KeyPeople
Individual/PersonalLevel
Healing/recovery
Perceptions
Attitudes
Skills
Behavior
IndividualRelationships
Socio-PoliticalLevel
Groupbehavior/relationships
Publicopinion
Socialnorms
Institutionalchange
Structuralchange
Programactivities
Programactivities
Socio-politicalgoal
CurrentSituation:ConflictAnalysis
KeyDrivingFactorsofConflictand“KeyPeople”orActorAnalysis
Whatisthegapbetweenthe
currentsituationandthedesiredfuture?->“peaceneeds”and/orstrategicspace.
Whatneedstochange
andhow?
ProgramTheory:Howdotheactivitiesleadto
thegoal?
TheoryofChange:HowdoesthegoalcontributetoPeaceWritLarge?
Vision:AdesiredfutureSocietalchange/PeaceWritLarge
56
Annex3:RPPProgramPlanningChart
IdentificationofActivities,ChangesandTheoriesofChangeandAssumptions
Program Goal(determinetimeframe!): ProgramGoal–TheoryofChange:Ifwedoxxx[activities]Thenweachieveyyy[thetypeofchangeyouexpectfromtheactivity/ies]Becausezzz[Rationaleforwhythischangewillbeachieved:whydoes‘then’happen?]
Proposed ProgramActivity Expected Changes, due totheActivity/ies Whatassumptionsdoyoumake?1.
2.
3.
4.
57
Annex4:DoNoHarmActionPatterns(ResourceTransferPatterns)
TheImpactsofanorganization’sACTIONS
Howanorganizationtransfersresourcesintoacontextmatters.Thewaysthesetransfershaveimpactsinfivespheres,andtheseimpactscanbepositiveornegative.Ifanorganizationisstrategic,understandsthecontextandmakescontext-appropriateprogrammingchoices,theycanhavepositiveimpactsinthesefivespheres.But,organizationscanalso,throughlackofattention,orprogramplanningnotlinkedtocontextanalysis,havenegativeimpactsinthesefivespheres.
Incompleteanalysisand/orinappropriateprogramming Strategicandcontext-appropriateprogramming
1. Theft/PreventionGoodsormoneyintendedfordistributionorpaymentmaybestolen,andusedbyfightersorusedtopayforongoingfighting.
Theftcanbeprevented,money,time,andresourcesaresavedandusedtobenefitcommunities.
2. MarketEffects
Adverseimpactsonpricesofgoodsandservices.Pricinglocalpeopleoutoftheirownmarkets,Goodsthatareavailablelocallybroughtinfromoutsideandgivenawayforfreecandroppricesiflocalgoodsandforcefarmersandsellersoutoftheirjobs.
Balancingandstabilizingmarkets.Ensuringpeoplecancontinuetoaffordlocalgoodsandservices.
3. DistributionEffects
Unevendistributionalongconflictlinescanexacerbatetensions/divisions,unfairlybenefitonesideofaconflictoveranother.
Understandinglocaldefinitionsoffairdistributioncanhelptodeterminebeneficiaryselectionwithoutexacerbatingtensions.Fairdoesnotalwaysequal“even”distribution.
4. SubstitutionEffects
Freeingupgovernmentresourcestocontinuefighting.Weakeningthestate’sabilitytorespondandmanagedisasters,,conflictsanditsowndevelopment.
Strategic,short-term,negotiatedsubstitution.Involvegovernmentinprogramdesignsotheyunderstandandareheldaccountablefortheirroleintheprogram(incl.transferofresponsibilitiesandtimeframes)
5. LegitimizationEffects
Inadvertentlylegitimizingagovernment,institutionorleaderbyinvolvingthemintheaidprocess.
Strategicallylegitimizingagovernment,leaderorinstitutionwithaneyetochangingorimprovinglocalperceptionsoftheirabilitytomanagedevelopmentordisasterresponse.MustunderstandWHYandHOWtheywillbelegitimized.
Annex5:ToolforPeacebuildingQualityAssessment34
The following tool is an illustrative framework for organizing and documenting a peacebuilding program qualityassessment.
Intheleftcolumnarequalitycriteriaforaprogramdesign,basedonthekeyfindings,conceptsandtoolsofRPP.Theframeworkprovidesaseriesofquestionsdividedintoseveralbroadcategories:conflictanalysis,programgoals/outcomes,programstrategyandlogic,andunintendedimpact.
Thesecondcolumnasksforaratingoftheprogramdesign,rangingfrom1to3:
1. Programdesignmeetsnoneofcriteria;2. Program design meets some of criteria, but there are significant gaps and weaknesses in several
categories;3. Programdesignmeetscriteria,withnosignificantgaps,althoughtheremayberoomforimprovement.
Thethirdcolumnasksforevidenceonthebasisofwhichtheassessmentteamorassessorhasarrivedattherating.Itshouldreflectstrengthsandopportunities—thatis,whereaprogrammayfulfillqualitycriteriabutmightbestrengthenedfurther,aswellasgapsandchallenges—thoseaspectsofaprogramthatdonotmeetthecriteriaorcriteriaofeffectivepeacebuildingprograms.
Thefourthcolumnprovides forrecommendations for improvingthequalityof theprogramdesign. Thiscanincludewhatanagencycandotosustainorimprovetheprogramdesign,whatitshouldkeepinmindormonitorwhile implementing theprogram, and suggestions on things that canbedone to improve the quality of thepeacebuildingprogram.Herethequestionofcapacitybuildingforresponsiblestaffmightalsoarise.
ThisframeworktoolmaybestbeunderstoodasaframeworkforanRPP-inspiredlearningprocessforpeacebuildingorganizations.Itisnottobeunderstoodascomprehensiveandexhaustive,butaspreliminaryreflectionandfoodforthought.
34BasedonReimann,AnAlternativetoFormalEvaluationofPeacebuilding:ProgramQualityAssessment,withadaption
1. ConflictAnalysis
Criteria/QuestionsbasedonRPP Rating Evidence Recommendations/WaystoImprove
Doestheanalysisidentifykeydrivingfactorsandkeyactors?
Doestheanalysisconsiderwhatneedstobestoppedandwhatforcespromotepeace?
Istheanalysisupdatedandtested?
Hastheanalysisidentifiedandexaminedpastorongoingsimilareffortsandanylessonsfromtheirresults?
Doestheanalysisidentifypeacebuildingneedsorpointsofleverage?
Isthescopeoftheanalysisisappropriate(nottoobroadornarrow,mitigatesbiastowardsagency’sexpertiseorgeneralbeliefsaboutconflict)?
Wastheanalysisprocessconflict-sensitive?
2. ProgramGoals
Criteria/QuestionsbasedonRPP Rating Evidence Recommendations/WaystoImprove
Doesthegoaladdress,directlyorindirectly,keydriversofconflictorpeace?
Isthegoalstatedasadesiredchange?
Isthegoalspecificandrealisticforthetimeframe—neithertoobroad(long-termvision)nortoonarrow(activitiesoroutputs)?
Isthegoalstatedasadesiredchangeatthesocio-politicallevel,orisachievingitpartofalonger-termstrategyforeffectingchangeatthesocio-politicallevel?Doestheprogrammakelinkagestoeffortsofotheragenciesatthesocio-politicallevel?
Dothechangescontributetooneormoreofthefollowingintermediatebuildingblocksofpeace:
1. CreationorreformofpoliticalinstitutionsthataddressKDF?
2. Locally-drivenpeaceinitiativesaddressingKDF?3. Increasingresistancetoviolenceor
provocationstoviolence?4. Increasedsecurityorsenseofsecurity?5. Meaningfulimprovementininter-group
relations?
3. OverallTheoryofChange
Criteria/QuestionsbasedonRPP Rating Evidence Recommendations/WaystoImprove
IstheOverallTheoryofChangeexplicit,withclearandunderstandableconceptualizationofpathwaytochange?
DoestheOverallTheoryofChangeprovideaplausibleexplanationofhowachievingtheprogramgoalwillaffectkeydriversofconflictorpeace?
IstheOverallTheoryofChangegroundedinanunderstandingofhowchangehappensintheparticularconflictandcontextinwhichtheprogramisbeingimplemented?
4. ProgramStrategyandLogic
Criteria/QuestionsbasedonRPP Rating Evidence Recommendations/WaystoImprove
Doactivities“addup”tothegoal?Arethereexplicit,rationalandplausiblelinksbetweencomponentsoftheprogramthatarevalidinthecontext?
Doestheprogrammakelinkagesbetweenactivitiesandchangesattheindividual-personallevelandatthesocio-politicallevel?
Doestheprogrammakelinkagesbetween“morepeople”and“keypeople”?
Arethescaleandleveloftheoutputsreasonableinrelationtotheintendedimpactsandthesizeoftheissueinthiscontext?
Hasthedesignaccountedforfactorsthatcouldimpedesuccess(includingwayssocialandpoliticalsystemsmightresistchanges)?
Doestheeffortrelatetoorlinkwithotherinitiativesinthesameorrelatedareas?
WilltheM&Eplanorfeedbackmechanismprovidetimely,accurateandusefulinformationaboutprogresstowarddesiredchangesandaboutassumptionsunderlyingtheoryofchange?
5. Conflict-Sensitivity/DoNoHarm
Criteria/QuestionsbasedonDNH Rating Evidence Recommendations/WaystoImprove
Istheprogramconflict-sensitive?Doesitconsiderpotentialunintendednegativeimpactsitmightcause?
Istheinitiativeconsciousoftheimpactofitsactionpatterns/resourcetransfersinthefollowingareas:1. Theft;2. MarketEffects;3. DistributionEffects;4. SubstitutionEffects;5. LegitimizationEffects.
Doestheinitiativeconsiderunintended(negative)impactsoftheprograminitsM&Esystem?
Annex6:ChecklistforEvaluabilityAssessmentofPeacebuildingPrograms35
Thefollowingcontinuumsprovideaframeworkforassessingtheevaluabilityofapeacebuildingprogramand illustratewhatmakes a peacebuilding program or project less ormore evaluable. RPP tools andfindingshavebeenintegratedintothequestionsregardingprogramdesign.
Theredcolumnontheleftindicateslowevaluabilitywhilethegreencolumnontherighthighlightshighevaluability.Thedottedlinesbetweenthered,yellowandgreencolumnsindicatethatthedivisionisnotclear-cutorfixed.
Theteamortheevaluabilityassessorcanticktherespectiveboxestoindicatewheretheprogramstandsintermsofevaluability.Ifmostofthetickedboxesareintheleft,redcolumn,aprogramisnotreadyforevaluation,andthechecklistindicateswherechangesintheprogramdesignwouldneedtobemade.Theevaluabilityassessorcouldalsorecommendwheretochangeandimprovetheprogramdesignandhencemaketheprogrammoreevaluable.Thefieldstaffandprogramdesignersinthefieldcouldlikewiseusethechecklistasamonitoringtoolthathelpsthemtoworkontheaspectsoflowevaluability.
35Thischeck-listisbasedon(Reimann,Chigas,andWoodrow2012b)andhasbeenadaptedbyCDA
ProgramDesign(includesRPPtoolsandfindings)
Lowevaluabilityinpeacebuilding
Highevaluabilityinpeacebuilding
Noconflictanalysisatall
Partialanalysis
“Goodconflictanalysis”
prioritizingKDF,stakeholders
Implicitanalysis/Informalanalysis
Notupdated
Nodocumentedanalysis Contextanalysis
Analysisnotupdated
Notlinkedtostrategy
Biasedandnarrow
Toocomprehensive
Uncleargoalsandobjective
Goalsandobjectivetoogeneralandbroad
Well-definedandrobustgoalsand
objectives
Goalsdefinedasactivities
Goalsandobjectivetooambitious Goalsareformulatedas
desiredchangesatthesocio-political
level
Nolinksbetweenactivities,goalsandoverallobjective
Indirectundunclearlinksbetweenactivities,goalsandobjective
Explicitlinksbetweenactivities,goalsandobjective
ofPWL
“Hopelines”(linksbetweenactivitiesandoutcomesatvariouslevelsthatarebasedonweakassumptions)
ImplicitToC ImplicitandrealisticToC ExplicitToC
UnclearToC ExplicitbutinappropriateToC(i.e.ToCcontradict/sknowledgeofpeace-buildingpracticeorisnotsuitedto
thecontext.)
ClearandrealisticToC
DataAvailability
Lowevaluabilityinpeacebuilding
Highevaluabilityinpeacebuilding
ImplicitToC ImplicitandrealisticToC ExplicitToC
UnclearToC
ExplicitbutinappropriateToC(i.e.ToCcontradict/sknowledgeofpeace-
buildingpracticeorisnotsuitedtothecontext.)
ClearandrealisticToC
Nobaseline Condensedbaselinewithdata-gatheringisfocusedonafewkey
indicatorsforselectedgoals
Completebaseline,basedonconflictanalysis
Nobaselinebutamore
comprehensivemonitoringatthebeginning
Nomonitoringsystem
Monitoringsysteminplacebutnotused
Monitoringsysteminplacetogatherandsystematizeallnecessaryinformation
Insightsfrommonitoringarenottranslatedintoprogramchanges
Monitoringsystemusedtoinformprogramchanges
Monitoringsystemmonitorsimpactsonconflictcontextaswellasunintended(negative)
impacts
Noindicators Indicatorsexist,butunrealistic,un-
measurableorunclear IndicatorsareSMART
Indicatorsaredesignedtocapturechangesinconflict
contextandpossibleunintended(negative)impacts
Noaccesstostakeholders
Difficultandlimitedaccesstostakeholders Accesstostakeholders
Ongoingfeed-backprocesswithpartnersinplace
ConducivenessoftheContext
Lowevaluabilityinpeacebuilding
Highevaluabilityinpeacebuilding
Conditions(weather,security,availability…)
notfavorable
Someconditionsquestionable,butgenerally
workable
Noapparentimpedimentstoaccessorsecurity
Nofinancialresourcesavailabletoconduct
evaluation
Financialresourcesavailablebutlimited
Fullfinancialresourcesavailable
Nointernalcommitment/Internally
drivenprocess
Internallydrivenbutonlybytheheadsofagencies
CommitmentAvailable-
Internallydrivenandlocallyownedprocess
Noownershipofprocess
Notdonor-drivenprocessbutnolocalownershipof
process
Internalcapacitiesavailabletoengageandfollow-up
Donor-drivenprocess
DoNoHarmprinciplesnotappliedtotheevaluationprocess
DNHprinciplesonlypartiallyapplied
DNHapplied:Timing,locationofevaluationandselectionofinterviewpartnersareconflict-
sensitive
DNHprinciplesappliedoncebutnotupdated