8/13/2019 24pub_br217

1/60

~~~.. .--.~~~~~~~~~~~~~~.~... ,..... .WORLD BANK TECHNICAL PAPER NO. 334\WrP33L0

Work n progress 9for public discussion

Performance MonitoringIndicators Handbook

AGRICULTURE4 - 3g1,', I IUSTMUENT

EDUCATIONENVIRONMENT

FMANCIAIECTOR- -x.t - : @HOUSINGNDUR_ANEVELOPMENT

_.USTRYND INING} - _ t ~~~~~~~~~~~~~OIllND AS

POPULAnON,EALTH.* a _ i: AND UrRMON? POVERTYREDUCTION

_ SECTOREVELOPMENT-1 ',Wiiun MANAGEMENT

TECHNICALSSISTANCETELECOMMUNICATIONS

TRANSPORT- ANDWASTEWATER

Roberto IosseLeigh Ellen Sontheimer

8/13/2019 24pub_br217

2/60

RECENT WORLD BANK TECHNICAL PAPERSNo. 256 Campbell, Designand Operation f Smallholderrrigation n SouthAsiaNo. 258 De Geyndt, Managing he Qualityof HealthCare n DevelopingCountriesNo. 259 Chaudry, Reid, and Malik, editors, Civil ServiceReform n LatinAmericaand the Caribbean: roceedingsf a ConferenceNo. 260 Humphrey, PaymentSystems:Principles,Practice, ndImprovementsNo. 261 Lynch, Provisionor Childrenwith SpecialEducational eeds n the Asia RegionNo. 262 Lee and Bobadilla,HealthStatisticsfor heAmericasNo. 263 LeMoigne, Subramanian, Xie,and Giltner, editors, A Guide o theFormulation f WaterResources trategyNo. 264 Miller and Jones, Organicand Compost-Based rowingMediaforTreeSeedlingNurseriesNo. 265 Viswanath, BuildingPartnershipsforPovertyReduction: he Participatory rojectPlanningApproach f the Women'sEnterpriseManagementTrainingOutreachProgramWEMTOP)No. 266 Hill and Bender, Developinghe RegulatoryEnvironmentforCompetitive griculturalMarketsNo. 267 Valdes and Schaeffer,Surveillance f AgriculturalPricesand Trade: Handbookfor he DominicanRepublicNo. 268 Valdes and Schaeffer,Surveillance f AgriculturalPrices nd Trade: HandbookforColombiaNo. 269 Scheierling, OvercomingAgriculturalPollutionof Water:TheChallenge f IntegratingAgriculturalandEnvironmentalPolicies n the EuropeanUnionNo. 270 Banerjee,Rehabilitation f DegradedForests n AsiaNo. 271 Ahmed, Technological evelopment nd PollutionAbatement:A Study ofHow Enterprises re FindingAlternatives oChlorofluorocarbonsNo. 272 Greaney and Kellaghan, Equity Issues n PublicExaminationsn DevelopingCountriesNo. 273 Grimshaw and Helfer, editors, Vetiver GrassforSoil and WaterConservation, andRehabilitation, nd EmbankmentStabilization: Collection f Papers nd NewslettersCompiled y the VetiverNetworkNo. 274 Govindaraj, Murray, and Chellaraj,HealthExpenditures n LatinAmericaNo. 275 Heggie, Management nd Financingof Roads:An Agenda or ReformNo. 276 Johnson, Quality Review Schemesor Auditors:TheirPotentialforSub-Saharan fricaNo. 277 Convery,Applying EnvironmentalEconomicsn AfricaNo. 278 Wijetillekeand Karunaratne, Air QualityManagement: onsiderationsforDevelopingCountriesNo. 279 Anderson and Ahzned, The CaseforSolarEnergy nvestmentsNo. 280 Rowat, Malik, and Dakolias,JudicialReform n Latin America nd the Caribbean:roceedings f a WorldBankConferenceNo. 281 Shen and Contreras-Hermosilla, Environmental nd Economicssues n Forestry: electedCaseStudies n AsiaNo. 282 Kimand Benton,Cost-BenefitAnalysis of theOnchocerciasis ontrolProgram OCP)No. 283 Jacobsen,Scobie and Duncan, Statutory Interventionn AgriculturalMarketing: NewZealandPerspectiveNo. 284 Valdesand Schaeffer in collaboration with Roldos and Chiara, Surveillance f AgriculturalPriceandTradePolicies:HandbookforUruguayNo. 285 Brehm and Castro, TheMarketforWaterRights in Chile:MajorIssuesNo. 286 Tavoulareasand Charpentier, CleanCoalTechnologiesforevelopingCountriesNo. 287 Gillham, Bell,Arin, Matthews, Rumeur, and Hearn, CottonProduction rospectsforhe Next DecadeNo. 288 Biggs,Shaw, and Srivastiva, Technological apabilitiesnd Learning n AfricanEnterprisesNo. 289 Dinar, Seidl, Olem, Jorden, Duda, and Johnson, Restoring nd Protecting he World'sLakesand ReservoirsNo. 290 Weijenberg,Dagg, Kampen Kalunda, Mailu, Ketema, Navarro, and Abdi Noor,StrengtheningNationalAgricultualResearch ystems in Easternand CentralAfrica:A Frameworkfor ctionNo. 291 Vald6s and Schaefferin collaboration with Errazuriz and Francisco,Surveillance f AgriculturalPriceand TradePolicies: HandbookforChileNo. 292 Gorriz,Subramanian, and Simas,IrrigationManagementTransfer nMexico:Process ndProgressNo. 293 Preker and Feachem,MarketMechanisms nd the HealthSector n CentralandEasternEuropeNo. 294 Valdes and Schaeffer in collaborationwith Sturzenegger and Bebczuk,Surveillance f AgriculturalPriceand TradePolicies: HandbookforArgentina

(List continues on the inside back cover)

8/13/2019 24pub_br217

3/60

8/13/2019 24pub_br217

4/60

Copyright 1996The International Bankfor Reconstructionand Development/THE WORLDBANK1818H Street, N.W.Washington, D.C. 20433,U.S.A.All rights reservedManufactured in the United States of AmericaFirst printing September 1996

Technical Papers are published to communicate the results of the Bank's work to the developmentcommunity with the least possible delay. The typescript of this paper therefore has not been prepared inaccordance with the procedures appropriate to formal printed texts, and the World Bank accepts noresponsibility for errors. Some sources cited in this paper may be informal documents that are not readilyavailable.The findings, interpretations, and conclusions expressed in this paper are entirely those of theauthor(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations, orto members of its Board of Executive Directors or the countries they represent. The World Bank does notguarantee the accuracy of the data included in this publication and accepts no responsibility whatsoeverfor any consequence of their use. The boundaries, colors, denominations, and other information shown onany map in this volume do not imply on the part of the World Bank Group any judgment on the legalstatus of any territory or the endorsement or acceptance of such boundaries.The material in this publication is copyrighted. Requests for permission to reproduce portions of itshould be sent to the Office of the Publisher at the address shown in the copyright notice above. TheWorld Bank encourages dissemination of its work and will normally give permission promptly and, whenthe reproduction is for noncommercial purposes, without asking a fee. Permission to copy portions forclassroomuse is granted through the Copyright Clearance Center, Inc., Suite 910, 222 Rosewood Drive,Danvers, Massachusetts 01923, U.S.A.The complete backlist of publications from the World Bank is shown in the annual Index of Publications,which contains an alphabetical title list (with full ordering information) and indexes of subjects, authors,and countries and regions. The latest edition is available free of charge from the Distribution Unit, Officeof the Publisher, The World Bank, 1818 H Street, N.W., Washington, D.C. 20433, U.S.A., or from Publica-tions, The World Bank, 66, avenue d'Iena, 75116 Paris, France.ISSN: 0253-7494

Both authors work in the Bank's Operations Policy Group in the Operations Policy Department. Robert Mosse isoperations adviser; Leigh Ellen Sontheimer is a consultant.Library of Congress Cataloging-in-Publication DataMosse, Roberto, 1944-Performance monitoring indicators handbook / Roberto Mosse andLeigh Ellen Sontheimer.p. cm. - (World Bank technicalpaper ; no. 334)Includes bibliographical references.ISBN 0-8213-3731-91. World Bank. 2. Economic development projects-Evaluation.1. Sontheimer, Leigh Ellen, 1964- . II. World Bank. III. Title.IV. Series.HG3881.5.W57M685 1996338.9'0068'6-dc2O 96-27346CIP

8/13/2019 24pub_br217

5/60

8/13/2019 24pub_br217

6/60

Appraisal ndnegotiations 21Implementationndsupervision 21Evaluation 21Performancendicatorsndprojectmanagement 21Field-levelmplementers 22Implementationnitmanagers 22Borrowerfficials ndBank taff 23Someelatedssues 23Good racticen monitoringndevaluation 23Data ollectionnd management 24Institutionalrrangementsndcapacity uilding 25Feedbackrommonitoringndevaluationfforts nd nterpretingindicators 25Relationo oancovenants 26Working t crossurposes 26Last ord 27Notes 27SECTORAL NOTES ON INDICATORS 28Whatopics o he notesover? 28Howwere henotes eveloped? 29Howare ndicatorselectedrom henotes? 29Where re he notes vailable? 29How o learnmore 29EXAMPLES OF INDICATORS 3 1Table Hondurasasic ducationroject(SAR 3791-HO, arch , 1995) 32Table IndonesiauralElectrificationroject(SAR 2920-IND,ebruary , 1995) 33Table Indonesia:econdgricultural esearchanagementroject(SAR 3933-IND,pril21, 1995) 35Table LithuaniaiauliaiEnvironmentroject(SAR 4981LT, ovember, 1995) 37Table PeruRuralRoads ehabilitationndMaintenanceroject(SAR 4939-PE,ovember, 1995) 38Table ThePhilippinesomen's ealth ndSafeMotherhoodroject(SAR 3566-PH,anuary 7, 1995) 40Table Venezuelagriculturalxtensionroject(SAR 3591VE,March7, 1995) 42Table Chad:Structural djustmentredit(President'seport -6785-CD) 43Table Morocco:inancial arkets evelopmentoan(President'seport -6633-MOR) 45

iv

8/13/2019 24pub_br217

7/60

FOREWORDAs part of ongoingefforts o improve he qualityand impactof itswork, the WorldBank s placing new emphasis n theuse of performancemonitoring ndicators.Theseindicators,which are basedon a logical framework f projectobjectives nd end-means elationships, elp generatemore houghtful,ogicallyconstructed rojectdesigns.And because heyserveas benchmarks gainstwhich to measure rojectprogress oward development bjectives, hey result n moremeaningful rojectmoni-toring and evaluation.Over the post wo years Bankstaff havedevelopednoteson suggestedperformancemonitoring ndicators or each of the main sectorsn which he Bank sactive.Thesenotesoffer a framework or useby taskmanagers, orrowers,and pro-ject mplementation nits n analyzing he relationship etweenobjectives nd moni-torable outcomes nd impacts.Theyalsooffer a menuof possible ndicators.This handbook,which introduces nd supplementshe sectornotes, s divided ntothreesections. he irst sectionexplainswhymenus f indicatorswere developed;provides he background n the ogical frameworkand typology of indicators;describes ow indicatorsare developedand applied n projectdesign,supervision,and evaluation;and discussesmportant ssues elated o the meaningful seof indi-cators.The secondsection escribeshe sectornoteson indicatorsand their useandexplainshow to get copies. The hird sectionprovidesexamples f performancendi-catorsdeveloped or Bank-financedrojectsand showshow the ndicatorsweredeveloped n thebasis of each project's evelopmentbjectives.

Myrna AlexanderDirector,OperationsPolicyMay 1996

V

8/13/2019 24pub_br217

8/60

ACKNOWLE DGME NTSThis andbookwas edited by PatriciaRogers f the World Bank'sOperationsPolicyDepartment ndMeto de Coquereaumontnd PaulHoltzof AmericanWriting Corpo-ration. LaurelMorais and ChristianPerez AmericanWriting Corporation)aid out thetext. Theauthorsare grateful o Gerald Britan U.S.Agency or International evelop-ment),MosesThompsonTeams echnologies,nc.),UlrichThumm OperationsEvalua-tion Department,World Bank),and DeborahWetzel (Developmentconomics,WorldBank) or their valuablecontributions nd comments.Theauthorsalso hank he following Bank askmanagers or preparingobjectivesand indicatorsmatrixes:Maria Dalupan,HennieDeboeck,DelyGapasin,JoseLuisIrigoyen,RamaLakshminarayanan,avidNielson,Arun Sanghvi,Anna Sant'Anna,StanleyScheyer,Sari Soderstrom, nd ChristopherWalker.

vi

8/13/2019 24pub_br217

9/60

PERFORMANCE MONITORINGINDICATORSWhatareperformanceonitoringndicators?Performancendicatorsare measures f project mpacts,outcomes, utputs, ndinputs hat are monitoredduring project mplementationo assess rogress owardprojectobjectives.Theyare also used ater o evaluate project'ssuccess.ndicatorsorganize nformation n a way that clarifies he relationships etween project'simpacts,outcomes, utputs, nd inputsand help to identify problems long the waythat can mpede he achievement f projectobjectives.Whyareperformanceonitoringndicatorsmportant?In recent earsseveral mportant tudies-mostnotably he PortfolioManagementTaskForceReportand reviewsby the OperationsEvaluationDepartment OED)-have ound hat he monitoringand evaluationof Bank-financedrojectsdid not focusadequatelyon the factors hat are critical or positivedevelopmentmpact.Both hetask orce and OED concluded hat performancendicators houldbe integratedwiththe monitoringand evaluationprocedures sedby the Bankand itsborrowers.ThePortfolioManagement askForceReport alsoknownas the WapenhansReport) Porifolloanalyzed he factors hat affect he developmentmpactof Bankoperations.As part Managementaskof this analysis he task orce ocusedon how evaluationmethodologies,ncluding he arce indingscalculationof economic ate of returnand the project ating methodology, ere usedduring projectappraisaland supervisiono enhance he quality of Bank-supportedprojects.The ask orce ound hat project atingswere not providing mplementationunits,borrowers,and Bank askmanagerswith adequate eedbackabout progresstoward developmentmpact or several easons:* Toomuchemphasiswas placed on the mechanicsphysical nd financial)of pro-ject implementation.* The isksand actors hat most nfluenceprojectoutcomeswere poorly dentified.* Objectivecriteria, ransparency, nd-since the methodology ependedon thejudgmentof individuals-consistency cross nitswere acking.* Ratingsended o be overlyoptimistic.

1

8/13/2019 24pub_br217

10/60

Without ppropriateeedback,one f thepartiesoncernedithproject utcomescouldmake ppropriate,nformedecisionsboutwhether ndhow o adjust rojectdesign r mplementationrrangementso better chieve project'sntendedobjectives.related roblems hat heobjectiveshemselvesreoften otwell houghtoutor clearly rticulated,loudinghedevelopmentf appropriateerformanceoni-toring ndicatorsndmakingmonitoringndevaluationvenmore ifficult.Further,he ask orcepointed ut hat heBank'sraditionalmethod f appraisaland evaluationf developmentmpact-thecalculationf economicateof return rnetpresentalue-cannot e appliedo all projects nd hat orsome rojects,meaningfulost-benefitnalysiss not easible. venor projectsorwhichnetpresentvalue r economicateofreturns calculated, ankpractices to calculatet onlyafew imes-during ppraisal,uringa midtermeview, pon estructuringfnecessary,ndat the imeof the mplementationompletioneport. eitherhecal-culations or hecritical ariableshataffecthem remonitored xplicitlyhroughoutimplementation.oreover,he eportpointed ut, n many nstancesosts ndbene-fitscouldbe moreeasilydentified ndmeasured ithperformancendicators.Theask orce oncludedhat heBank's rojectatingmethodologyndsupervision

reportingystemhould e adaptedo includeheuseof project erformanceonitor-ing ndicatorserivedroma project'sevelopmentbjectivesnd mplementationlan.OED tudies TwoOED tudieseviewingheBank'secord n hemonitoringndevaluationfprojectseinrorcedhe indings f the ask orceeport. In fiscal1994 a study ssess-ingtwenty ears f Bankprojectsound hatBank uidelinesnddirectivesn moni-toringandevaluationadnot beenollowed dequately,ither uringprojectappraisalwhenmonitoringndevaluationreplanned) r during mplementation.Buthe study lso oundevidencehat hesituation aschanging.A follow-uptudy f monitoringndevaluationlansn a sample f fiscal1995projectsuggestshat hequalityof such lanning as mproved.he mprovements

evidentn the isinguseof performance onitoringndicators-thehare f projectswith at east omendicatorsoserom72 percentn iscal1993 to 77percentn fis-cal 1995.Nevertheless,heexpandingseof ndicators asnotbeenmatchedyarrangementsor datacollectionr monitoringndevaluationapacity-buildingeffortsn eitherhe Bank rborrowing ountries. elativelyew projects14 percentof thesampleeviewed y OEDn iscal1995)achieve verall oodpracticen com-prehensiveesign r useof monitoringndevaluation.nd performance onitoringindicators,lthoughmorewidelyused, reweakn structurendusually onot ol-low a logical rameworkr a typology, nd heres notalwaysollow-throughndatacollection.NextSteps In esponseo theseoncerns,ankmanagementasmade ffortsofosterheuseofActionPlan performancendicators.ntheNextSteps ctionPlan,whichwasdesignedo imple-ment he ecommendationsf theWapenhansask orce,managementalledor ncor-porating erformanceonitoringndicatorsn theprojectating ystemsed or projectmonitoringthroughorm 90 and heAnnualReportn Portfolioerformance),o

2

8/13/2019 24pub_br217

11/60

bettermonitorprogressowarda project's evelopment bjectives.Management lsorecognized hat he Bankneeded o developsector-specificndicatorso help borrow-ersand Bankstaffdefineprojectobjectivesmorenarrowlyand to derive ogicalmea-sures f projectoutcomes nd impact n order o measure chievementf projectobjectives. heBankalso needed o support he useof indicators n the revised rojectrating system. hereforehe actionplan called or the sectordepartments ithin heCentralVice Presidencieso developsetsof sector-specificndicators hatare most ele-vant o projectdesignand monitoring-the sectornoteson indicatorsdiscussedn thesecondpart of thishandbook.Staffwould henbe required o ensurehat key sector-specificproject mpact ndicatorswere dentified n projectappraisaldocumentationand that progresswas monitored gainst hesebenchmarks.What are theadvantages f performance onitoringndicators?Performance onitoring nvolves eriodicallymeasuring project's rogressowardexplicitshort- nd long-term bjectives nd giving eedbackon the resultso decision-makerswho can use he nformationn variousways o improveperformancebox 1).

Box1. Usesf performancendicatorsSTPATEGIctANNING.For any program or activity, From developmentproject to a sales plan, incorpo-ratingperformance easurementnto he design orces reater onsiderationf hecriticalassump-tions hat underliehatprogram'selationshipsnd causal aths.'Thus erformancendicatorselpclarify the objectivesand logic of the program.PERFORMANCECCOUNTING.erformancendicators can help nform esource llocation ecisionsf theyare used o direct resourceso the most successful ctivities and thereby promote he most efficient useof resources.FORECASTINGNDEARLYARNINGURINGROGRAMMPLEMENTATION.easuring progressagainst indicatorsmaypoint oward uture erformance,rovidingeedbackhatcanbe used or planning,dentifyingareasneedingmprovement,nd suggestinghatcan bedone.MEASURINGROGRAMESULTS.ood performance ndicatorsmeasurewhat a program has achieved ela-tive o tsobjectives, ot ustwhat t hascompleted;hus heypromote ccountability.PROGRAMARKETINGNDPJBUCELAnONS.erformancendicatorscan beused o demonstrateprogramresultso satisfy nexternal udience. erformanceatacan beused o communicatehevalueof aprogramor project o elected fficials nd he public.BENCHMARKING.erformancendicators an generateata againstwhich o measure therprojects rprograms. hey lsoprovide way o improve rograms ylearning romsuccess,dentifying oodperformers, and leaming from their experience o improve he performanceof others.QUALITYANAGEMENT.erformance ndicatorscan be used o measure ustomer(beneficiary)atisfac-tion, and thereby assess hether and how the program is improving their lives.1. Thomas . Cook, erryVanSant,eslie tewort, ndJamieAdrian,PerformanceMeosurement:essonsearnedfor Developmentanagement,'WorldDevelopment3181:1303-1519951.

3

8/13/2019 24pub_br217

12/60

For hepurposesf theBank nd tsclientshemost ignificantenefitsf performanceindicatorsccruen project esignstrategiclanning),rojectupervisionndmonitor-ing(forecastingesults),ndproject valuationmeasuringesultsndqualitymanage-ment). he eedednformationnddatacollection gfortsecomevident s projectobjectivesre ormulated.na performanceonitoringystem,ndicatorserve s oolsformeasuringhe lowofchange.Baselinesre hevaluesf performancendicatorstthebeginningf theplanning eriod;argetsre hevalues t theend.Thebenefitsf indicatorsomeromheirmeasurabilitynd rom heirdirectderiva-tion romproject bjectives,hich regroundedn sector, conomic,isk,andbenefi-ciaryanalysis.ndicatorspecificallyinka project'snputs ndactivities ithquantifiedmeasuresfexpectedutputsnd mpact. eforeelectingndicators,heborrowerrprojectmplementationnitand heBankmust onsider hichmeasuresfperformanceill tell hemwhetherndhowa project'sroposedbjectivesrebeingachievedndwhowill benefit-therebyontributingo a morepreciseefinitionf theobjectives.orrowersnd heBankmust lsoaskwhetherhenecessaryataareavail-ableanddecidewhatusers hould o in responseothe ndicator utcomes.With ndicators, onitoringndevaluations more ompellingecauset isobjec-tive,notbased npersonaludgmentsr puredescription.urthermore,ndicatorsholda project's ltimateoalsclearlyn view hroughoutmplementation.fdesignedandused orrectly,ndicators eethespecificnformationeeds ndscope fauthorityf allthe partiesoncerned ith mplementation-fieldtaff,mplementationunit,borrower,askmanager,ndBankmanagement.hemplementationnitand tssubunitsremostnterested-andn a better ositiono respondo-indicatorsofinputs,isk actors, ndoutputs.heBank nd tsborrowersre mostnterestednindicatorsf output, utcome,nddevelopmentmpact. hushe ndicatorselpallparties ocus n heareas f greatestoncernothem.Howare performance onitoringndicatorsdeveloped?Performancendicators ust e based n heunique bjectivesf ndividual rojects.Butanysetof performancendicatorshould lsobe based n anunderlyingogicalframeworkhat inks roject bjectivesithprojectomponentsnd heir espectiveinputs, ctivities,ndoutputs tdifferentmplementationtages.The rameworksobjective-driven,ince nyaction nder project hould eaimed t achievingtsobjectives.iven heproject'sevelopmentbjectives,hebestmixof outputsoachievehese bjectivesndcomponentshatwill yieldhese utputs rederived.general chematicf thepointofviewofthe ogical rameworks shownn box2.Understandingow o derive erformancendicatorsroma project'sbjectivesndcomponentsequiresome nderstandingf theconceptf the ogicalramework.

Theogicl The ogical rameworks a methodologyorconceptualizingrojects ndananalyticfrmework tool hathas hepowerocommunicatecomplex roject learly ndunderstand-ably on a single heet f paper. t sa participatory lanningoolwhose owerdepends n howwell t incorporateshe ullrange fviewsof intended eneficiaries

4

8/13/2019 24pub_br217

13/60

Box . Point f iewof heogiclrameworkDEvELoWaqSECUnE

Describeshe proiect'sealoutcome-the mpac hat heproject's utputs ill hae on thebeneficiary,nstitution,r systemn terms f changed ehavior r improved erwformrc.Thedevelopmentbjective efinesheprojects uccess.

PROJECT UuwsDefinewhat heprojectcan6e helddirectlyaccountableor producing-theprojeds deliv-erables hegoodsand servicest will produce. ypically,utputs re ndepwdnt, synergis-tic, and ntegrated.

Clustersf ac6ivifieshat rdefine owvheproducts nd servicesWfille deli-YOOhchnicalossistance,hysicalnfrastructure,nd the lilke).

Source:Team Technologies,nc.

and otherswho havea stake n the projectdesign. t is best used o help projectdesigners nd stakeholders:e Setproper objectives* Define ndicatorsof successIdentifykey activityclusters projectcomponents)* Definecritical assumptionsn which the project s based* Identifymeansof verifyingprojectaccomplishments* Define esourcesequired or implementation.

As an up-frontplannerhe logical framework an be used o helpdesign ools orproject mplementationnd evaluation.Knowingts advantages nd limitationshelpsin assessinghevalue of the ogicalframeworkmethodology t variouspoints n theprojectcycle (box 3).Antecedentso the ogical frameworkThe ogicalframework an mprove he dentification, reparation,and performanceappraisal process y clarifying a project'sdesignand making t transparento theborrower, he ender,and beneficiaries ndotherstakeholders. he rameworkassumeshat projectsare instrumentsf changeand hat hey are selectedrom

5

8/13/2019 24pub_br217

14/60

Ibx3. AdvantagesndihitationsfusingheogkalrameworkethodADVANTAGES LMTA11ONS* Ensureshat undamentalueshonsreasked * Maygive ise o rigidity n project dministra-andweaknessesreanalyzedn order o provide tionwhenobjectivesndextemalactors pecifieddecisionmakersith better nd moreelevant at he outsetre overemphasized.hisconbeinformation. avoided y regularprojecteviews t which he

keyelementsanbe reevaluatedndadjusted.* Guides ystematicnd ogicalanalysis f theinterrelatedlementshatconstitute . As a general nalytic ool, s policyneutral nwell-designedroject. questionsf income istribution, mploymentopportunities,ccesso resources,ocal* Improveslanning y highlightinginkages participation,ostand easibility f strategiesndtechnology,ndeffects n theenvironment. between roject lementsndextemcalactors.* Facilitatesommon nderstandingndbetter . Isonly oneof severalools o be usedduringcommunicationetween ecisionmakers,anagers, roject reparation,mplementation,ndandotherpartiesnvolvedn the project. evaluafion.t doesnot replace eneficiaryanalysis;imeplanning; conomic,inancial, nd* Usedalongvith systematic onitoring, nsures cost-benefitnalysis; nvironmentalmpactcontinuityf approachwhenoriginalprojectstaff assessment;r similar ools.are replaced.* May acilitate ommunicationetweengovemmentsnddonoragenciesnce t hasbeenadoptedby more nstitutions.. Makes t easier o undertakeectoral tudies ndcomparative tudiesn generol,f usedwidely.

amongalternatives s he most ost-effective ay of achievinga desiredoutcome.tbrings ogetheras itsantecedentseveralprojectmanagementerspectives:* Results-orientedanagement. rojects eginwith a set of objectivesather hancomponents, nd successs measured y the degreeo which developmentbjec-

tiveshave beenmet.Managements heldaccountableor results.* Basicscientificmethod.Projects reexperiments ndertakenn an uncertainworld.From hispoint of view a project s a structured rocessor earningaboutwhatproducesesults. hisperspective ssumeshat projects re learningsystems.* Systemsnalysis.A project, ikea system,s notdefineduntilwe havedefined helargersystem f which t is a part.Projectsakeplacenot n a vacuum ut na rela-tionship with an external environmentof organizations, institutions,and other projects.* Contract aw. Every ontract project)has hesamebasic eatures:. A setof deliverablesoutputs). Circumstanceshatconstituteorcemajeure assumptions). A meetingof the mindsaboutwhat thedeliverableswill produce

(developmentbjective).* Cause nd effect.Thecoreconcept nderlyinghe ogical rameworks causeandeffect.The betterhecauseand effect inkages etween bjectives,hebetter hepro-jectdesign.By definition, ach projecthas his f-then r causeand effect ogicembeddedn it. If theprojectproduces ertainoutcomes ndercertain onditions,

6

8/13/2019 24pub_br217

15/60

then t canbe expectedo achieveertain ther utcomes.orexample,f theprojectsuppliesarmers ith mprovedeed ndputs credit ystemn place, ndassumingtheresadequateainfall,henproductionill increase.heogical rameworkorcesproject lannerso makehis ause ndeffectogicexplicit, ut t does otguaranteea gooddesign. he alidity f thecause ndeffectogicdependsn hequality ndexperiencef thedesigneam.Importancef clarifying ssumptionsAssumptionsre riskandenablingactors-externalonditionshatare outsidehedirect ontrol f theproject figure1).Achieving bjectivesandepend n whetherassumptionsold rue.Assumptionsremade bout hedegree f uncertainty(degree f risk)between ifferentevels f objectives.he ower heuncertaintyhatcertain ssumptionsill hold rue, hestrongerheproject esign. ailing ssumptionscanderaila project s oftenas poorly xecutedutputs.If cause ndeffect elationshipsre hecoreconcept f goodproject esign, ec-essary nd sufficientonditionsre hecorollary. he ause nd effect elationshipsbetweenevelsn he project's ierarchy f objectivesinputso outputs, utputso out-comesnd mpact,mpacto relevance)escribehenecessaryonditionsor achiev-ing developmentbjectivesfordefinition f terms eepages12-13). Thisstheinternalogicof theproject.Also mportant re hesufficientonditionst each evel or achievinghenexthigher evel conditionsn which henextobjective anbe met). he ufficientondi-Figure. Assumptions/risksboutxternalactorsustlso emadexpliet

Program jectve Assumptionscaegory/Couny RisksAssisloncelrabgy Risks

Development Assumptions/objective Risks

/ Otput s tAssumptions/

Components t Assumplios/\~RipksIF | >AND \

Note:This s a simplified epresentationf the ogicalframeworkshownater n box 4), highlightingheimportance f assumptionsnd risks.The wocentrolcolumnsblankherelwould normallycontainperformancendicators nd monitoring ndsupervision easuresor verifyingeach ndicator.Source:TeamTechnologies,nc.

7

8/13/2019 24pub_br217

16/60

tionsbetweenevelsn hehierarchy f objectivesre heassumptions.hiss theexternalogicof the project. heobjectivesierarchynecessaryonditions)lusheassumptionssufficientonditions)ogether iveonea much learerdeaof thepro-ject's esign. hiss what he ogical rameworkraphicallyepresents.Hierarchyf objectivesndthe ink o performancendicatorsUsers f the ogical rameworkave omeo agreeona commonetof termsout-comes,esults,argets,ndicators,utputs,oals, chievements,evelopmentbjec-tives). heseerms reessentiallyaysof referringo or classifyingbjectives.Theogical rameworkssumeshat hereareseveralevels fobjectivesn a pro-ject(ahierarchyf objectives).houghhere s no logical imit o thenumber f ev-els,most roject eamsind t difficulto managemorehan our evels ffectively.helogical rameworkypically pecifiesour.TheWorldBank sesndicators redicatedon a frameworkhatspecifieshree evels f objectives:nputsor project ctivities,outputsf project ctivities, ndoutcomesnd mpacts. or heWorldBank'spurposes,nputs ndoutputsorrespondtrictlyo projectmanagement,hileoutcomesnd mpacts orrespondo theproject's evelopmentbjectives.

TheCountry ssistancetrategyeflectsheagreementetweenheBankand tsborrowers n a country'sverall evelopmentoals nd heanticipated eans-projects, conomicndsector ork, echnicalssistance-forheBankandotherfinanciers)o helpachievehem.Projects ust howhow heirobjectivesrerelevantto the realizationf overall ountry evelopmentoals.Forany project setof performancendicatorshould e designed ithin he ogi-cal framework.he evelopmentf performancendicatorseginswith heproject'sobjectivesnd eflectshe associatedierarchyf activities nd heiroutputs ndintendedutcomesor eachproject omponent.heactivitiesonductednd heresults chievedt owerevels f objectivesre nputsoward he achievementfhigher-levelroject bjectives,t the nstitutional,ectoral, rogram, r countryevel.Thedefinitionf indicatorst eachevel hushinges n heproject'sltimatebjec-tive,whichcanbe modestforexample,o reducehe ncidencef preventableis-easewithina givenpopulationy ncreasingmmunizations)r more mbitiousnscale to educe hild mortality).Organizinghehierarchy f objectivesThe ogical rameworkresentschematicallyhehierarchy f project bjectives,heperformancendicatorsor measuringheachievementf eachobjective,hemeansof verifying achndicator, nd heassumptionsriskandenablingactors)riticaloachievinghenextobjectiveevel.Box4 illustratessampleogicalrameworkor anutrition ndchilddevelopmentrojectn Uganda.

The elationshipsmong roject bjectives-andheneedor performancenforma-tion-can bealsobe clarified y graphicallyepictingheoverall rogramogicandperformancexpectationsn an objectivesree figure ).The ree eginswith heover-arching evelopmentbjectivesf a projectconsistentith heobjectivesf theCoun-try Assistancetrategy),istshe ower-levelutputshrough hichhese bjectivesre

8

8/13/2019 24pub_br217

17/60

Box4. Logkairomeworkor heUgandautritionndChildDevelopmenrsctNARRA IuMRY PEWORMANC DICATORS MONTOING AND RSUbSIO4N ASSmwPT AmD mS6IMPACT (impactorelevance)* Improvedealth, urition, * Reducedrevalencef protein * Noainalanthropomeiric * Oiler national rogramsn heathandcogniivestatus fyoung energymalnutrition-measuredy survey (Immunizoaion,rimaryhealhcore),children underweightreschoolhildren- food production,rimaryeducaionare

from25% o 13% y 2003 sustained;ynergy iththen programswillachievehigherprojed mpoctReduced icronuirienteficiency * DemogrophicndhealthproblemsIDM revalenceeduced surveyby 50% nendemicreas, nemiain childreneducedo 33%,wormloodreducedo50%,andxeophthalmiareducedo2%by 2003)* Netenrollmentngrode 1 * Schoa nrollmentecordsincreasedo 80%by 2003OUTCOMES (outcomeo mpoct)* Improvedhildcare, utrition, . Number f children eachedy . Annual rojec report * UNICEF aintainsnterestnandhealth ervices services partnership ithprojed* Improvednoome eneraion * Number Fwomeneached * Annualprojed eport . Governmenigns hildbill into wcopocityorwomen by services * Partnershipithother onorsL sustainedOUTPUTS (outputooutcome)* Establishedommunity-based * Number f children ndwomen * Projedconitoringurvey, * Decentralizoaionolicymplementedchildcare rogram who participatedn program householdurveys* Number f health ay * Project onitoringoutreachffortsounched reports* Number f ECD aycore enters * Projedc onitoringeports

established* Establishedicronutrient * Number F itaminA capsules, . Projecmonitaringeports * Distridgovernmentsontinueupportprogram iron ablets istributed. Percentge f salt odized * Surveyomple f swtsold* Establishedomen'sncome * Number f womenwhoreceived * Projedmonitaringeportscomponent trainingn skills or ivelihood/childcarenterpriseACTIVITIES INPUTS (octivityooutput)* Community-6asedrowth * btal $42million,ncluding . Adiviieswill 6everifiedhrough * Sufficientnterestromcommunitiesa monitoringrograms operationsnd maintenancend regular rojedmonitoringmonitoringnd evaluation* Heoah-dayutreochrograms* Matching rants r oommunityECD enter* Massmedia ampaigns* Gronts or women'sivelihood * Mothers pplynewskillsandcapocity uilding knowledge* VitaminA capsules,ron abletdistributed* Iodizing f salta Dewormingompoign

TaskManager:Marib Garcia.

9

8/13/2019 24pub_br217

18/60

Figure.Partialblectlvesroeor hypothetkalhiMealthroFgmOBJECTIVE Reduce hildmortality

Reodingown nsversIMPACTS Reduced Improved Improvedamily Expanded the uestionshy ndincidenceof nutrition health ractices availoble how?whoat oweneed o do to achievpreventable care thisesult?disease

OUTCOMESncreasedpolio Increasedther Increasedseofimmunizations immunizations oralrehydration

ReadingcrossnswersOUTPUTS Expanded Increased Improved ublic the eshonwhoatse?school availability f awareness =at dditionalhingsimmunizations vaccines doweneedodotoachievehe extI= = =aobjecthve?ACTIVITIESncrase Improve Expandsupplies training facilities

Readingpanswershequestionsawhato?-what is theINPUTS Provide Enforce Provide significoncef ourfunds regulations vaccines accomphdshments?Note hat this s a partial objectivesree.Only he elementsn bold are expandedn the igure o show hevariousactivities nd the ntended utcomes, utputs, nd nputs eeded o achieve he projectobjective.The ome reatment an be applied to all elementsn the igure o completehe objectivesree.Source: dopted romGeraldM. Britan,Measuring ProgramPerformanceor FederalAgencies: ssuesand Opfions or Performancendicators' Washington, .C.: U.S.GeneralAccountingOffice).achieved,and showshe specific nputsequired o undertake rojectacfivities.Usingan objectivesree o represent project's oals helps erify he ogic of projectdesignand confirm hat he right ndicatorswere defined o measure erformance.An objectivesree should ncludeall the ower-evel esultshat are necessary on-ditions or achievinghigher-level bjectives. hat ncludes xternal actorsandassumptionsbout conditionsn the wider projectenvironmenthat do not actually allunder he controlof the project-the external isk and sensitivityactors seepages7and 14).6These spectsmustbe monitored hroughouthe projectalongwithimpacts,outcomes, utputs,and inputs,and testedduring mplementationo ensurethat he assumptionsemainvalid.In theexample f the partialobjectivesreeshown n figure 2, eachof the ower-levelresultss dependent ot only on thefactorsand assumptionshown n the figurebutalsoon others hatare not shown.These ouldbe the responsibilityf the projectshownin the diagram,with thedetails uppressederebecause f space imitations. ut heycouldalso bethe responsibility f anotherprojector projects r of the government,making hema part of the assumptionsnd external isk actorsunderlyinghe project.

10

8/13/2019 24pub_br217

19/60

Forexample,expandedchoolmmunizationsepends, sshown, n increasingsupplies, improvingraining nd expandingacilities all,ccordingo the igure,responsibilitiesf theproject.t alsodependsn increasedvailability fvaccinesand improvedublicawareness, hich ould e partof theprojectthoughhe iguredoesnotshowhat) r partof another roject nd hereforeartof theproject'sassumptionsndexternalisk actors.A tool o conceptualize,esign, ndappraise rojects,uchas an objectivesreeor similaranalyticalool-and the performancendicators erivedrom t-should notbe considerednchangeable.It should e continuallyeevaluateduringeachphase f project reparation,mplementation,ndevaluation.f the esults nalyzedduringprojectmplementationoint o a need o changehemeans f achieving ro-jectobjectives,hemixof inputsactivitieso be undertaken)nddefinition f outputsshould e adjusted ndnew ndicators erivedo measurehenewlydefinedargets.Thegeneral tepshatgo nto ormulatingroject bjectivesndconstructinghelogical tructuref a project re mportant spects f the dentificationndprepara-tionstages f theBank's roject ycle box5).8

There redifferentways o measureerformanceor anygiven ariable objective, Typesfperformanimpact, utcome, utput,nput).A systemf indicators nchoredn he ogical rame- indicatorsworkapproachs modeledn igure3. While igure3 shows comprehensiveystemof indicatorsn order o provide completerame f reference,heelementsf thesystemhatare notcoveredn detailby thishandbook reshownn dotted ines.These lementsre ncludedn OED's valuationsf project erformance.ResultsndicatorsResultsndicators easureroject esultselativeo project bjectives. esultsremeasuredt the evel eterminedy a project's bjectives. ememberhat ollowingthe ogical rameworkpproach,heyshould edefined tarting ith he mpact ndoutcomendicatorsandworkingbackwardso the nput ndicators).

INPUTNDICATORS.nputndicators easurehequantityand ometimeshequality) fresourcesrovidedorproject ctivities. ependingn heproject,hese an nclude:* Funding-counterpartunds,Bank oan unds, ofinancing,rants* Guarantees* Human esources-numberf person-yearsor membersf the mplementationnit,consultants,nd echnical dvisers* Training* Equipment,aterials,ndsupplies, r recurrentosts f thesetems-forexample,

textbooks,yringes,accines,lassroomacilities.OUTPUTNDICATORS.utput ndicatorsmeasurehe quantity and ometimeshe qual-ity)of thegoods r servicesreated r providedhroughheuseof inputs.Dependingon heproject,hese an nclude uch lementss:

11

8/13/2019 24pub_br217

20/60

Box . Processesithin he ogicalrameworkSTEP PROBLEMND BENEFICIARYNALYSISThis nalysis Htemptso answer everal uestions:What s he problem t stake, ndwho s nvolvedin it?What are the needs, xpectations,ntentions,ndmotivationsf thevariousstakeholders,isag-gregoted y gender nd other elevantocialcriteria?Does heproblemequire xtemaldevelopmentassistance,r could t beresolved ome therway2Thissteps normally artof economic ndsectorworkor of the projectdentificationnd preparationxercise henhere s noprior economic ndsector ork, and t contributeso he ormulation f the CountryAssistancetrategy.STEP2 OBJECTIVESNALYSIS: ONSIDERATIONF ALTERNATIVESND PERFORMANCEXPECTATIONSThisstepdentifies eedsr problems nd transformshemntosolutions-specific,uantified bjec-tives.Eachobjectivemaybebrokendown nto subobjectivesnd outputsobe achieved t differentstages f the project. here realmost lwaysalternative aysof meeting projectsobjectives,ndthemethod hosenwill affect heconfigurationf outputs. hus onsiderationf alternativesnd selec-tion of themostappropriates partof thisstep.ConsiderationFaltemativesntails conomic,iskandsensitivity,ost-effectiveness,nvironmentalndsocial nalyses, ndbeneficiaryonsultation.Based n theobjectivesnalysis,hemainproject lements-comprisinghewholeogicalchain,including ector nd project bjectives,lanned utputs, ecessaryctivities,nd their nputs-canbegraphically epresentedn the ogical ramework r an objectivesree (seeigures1 and 2).

Each lement-inputs,ctivities.,utputs, utcomes,nd mpacts, nd the iskand enablingactorsthataffect hern-is measuroble.his swhere ndicatorsomen. Indicatorshatdefine nd measureeachof these lementsre dentified uringheobjectivesnalysis ndprojectplanning tage. or hisreasont is essentialo formulatendexpress roject bjectivesndoutputsn a way hatallowsmea-surementf bothshort-termnd onger-termesults. efinitionf anoutput,outcome, r mpactobjec-tiveshouldnclude:* Target roup for whom)* Quantity r evel howmuch)* Quality in terms fdesign tandardsnd environmentalr other mpact)* Time when t should e accomplished)* Locationwheret should appen)STEP3 FINALIZATIONF PROJECTESIGN ND INDICATORSIn thisstepplanners arefully xamineheproject o ensurehatall tselementsre ogically elated.Planners lsoassesshe ntegrity f indicatorsnd realism f targetsat thisstage, aking ntoaccountall project ssumptionsnd baselineata, and inalizeheir plans.

* Clientsvaccinated by a healthproject)* Farmers isited an extension roject)* Miles of roadsbuilt (a highwayproject)* Electricity enerationand transmissionacilitiesnstalled arural electrificationproject)* Pollution ontrolmeasuresnstalledor incentivesr regulations nforced a pollu-tioncontrolor air or waterquality mprovementroject).OUTCOMENDMPACTNDICATORS.utcomeand impact indicators measure he quan-tity and qualityof theresults chievedhrough heprovisionof projectgoodsand ser-vices.Dependingon the kindof project, hesecan nclude:

12

8/13/2019 24pub_br217

21/60

igure. Asystemf ndicatorsRISKNDICATORS DIRECTNDICATORS EFFICACYNDICATORS

RelevanceRiskor enabling factors -- - --------- SustainabilityRiskor enabling factors Outcomesand impacts EffectivenessRiskor enabling factors Outputs Efficiency

Inputs

------ Explicitevelopmentf these inds f indicatorss not equiredorWorldBankprojects.* Reducedncidence f disease through accinations)* Improvedarmingpractices throughextension isits)* Increased ehicleuseor trafficcounts through oad constructionr improvement)* Increasedural supplyand consumption f electricity through xpansionof elec-tricity network)* Reducedmortalityor lower healthcosts throughmprovedamily healthpracticesor improvednutrition,or cleanerair and water)

RELEVANCENDICATORS.Someprojectshave ntendedmpacts n higher-order bjec-tives hat are not capturedby direct outcomendicators uchas the onesdescribedabove.For nstance, omeprojects ave nationalor sectoralobjectives, nd or themimpactmustbe measured t those evels.Projectsmay alsohave unintended-oftennegative-impacts.Theseoo can be capturedby evaluation tudies.Relevancendi-catorsmeasurerendsn the wider policy problemshat projectmpactsare expectedto influence.f appropriate o theproject, hesendicators houldbe used.Dependingon the project, hesemay nclude:* Improved ationalhealthas measured y healthndicators throughmprovedhealthcare, healthsystem erformance)* Increasedarmprofitsand reducedood costs throughmprovedarmingpractices)* Reducedransportationostsand expandedeconomic evelopmentthrough oadconstruction r improvement)* Improved conomicgrowthand enhanced onsumer ell-being throughexpandedelectrification, ollution ontrols,and othernew technology).

A system f indicatorsdrawn from he logical frameworkmeasuresherelevanceof project esults s well as outcomes nd impacts see igure3). However, ucheffects an be verydifficult o attribute o individualproject esults.For he Bank'smonitoringpurposes,herefore,measurementf results topswith project mpact el-ative to projectobjectives.Thatsaid, as mentioned bove, relevancendicatorsareappropriate or objectives hat are sectoralor national n scope.

13

8/13/2019 24pub_br217

22/60

Risk ndicatorsRisk ndicatorsmeasurehe statusof the exogenousactors dentifiedas criticalthrough he riskand sensitivity nalysis riskand enabling actors)performedas partof a project'seconomicanalysis.These rethe factors hat are determinedo bethemost ikely to havea direct nfluence n the outcomeof variousaspects f the project(forexample,economic rices or power orcompetitive alariesor projectstaff)-theassumptionshat are madeabout conditions xternal o theproject.A project'sobjec-tivescan only be achieved f the ogical means-endelationship f the projectelementss secureand the external isk actorsare favorable.Efficacy ndicatorsEfficacy ndicatorsshow how well the resultsat one levelof project mplementationhave been translated nto resultsat the next evel: the efficiencyof inputs, effective-nessof project outputs,and sustainabilityof project mpact. They measurea pro-ject's efficacy in achieving its objectives, ather han its results.The ogicalframeworkapproachsometimes ses hesemeasuresn addition to direct indica-tors of results.For he Bank'smonitoringpurposes,he measurementf results suallystopswithimpact: Bankstaffare not expected o measure ustainability, ffectiveness,refficiencyas part of projectsupervision r Implementation ompletionReports utrather o stick o direct results ndicators.However, fficacy ndicators an be rele-vant ndicatorsof resultsf efficiency of an institution,or example) s a projectobjec-tive.Moreover,OED uses hesemeasuresn its evaluations f projectperformance. tis for these easonshat hesedescriptions f indicatorsare includedhere.

EFFICIENCYNDICATORS.fficiency ndicatorsusually epresent he ratio of inputsneededper unit of output produced-for example,physical nputs,dollars, orlabor required per unit of output. Accountability ndicators which are the centralfocusof muchproject and financial auditing) can be considereda subsetof effi-ciency indicators.They measure he extent o which resources re available forand appropriatelyapplied to the activities or which they were targeted.EFFECTIVENESSNDICATORS.ffectivenessndicatorsusually epresent he ratio ofoutputs or the resources sed o produce he outputs)per unit of project outcomeor impact, or the degree o which outputsaffect outcomes nd impacts.Forexample:

* Numberof vaccinations dministeredor heir cost)per unitdecline n morbidityrate (illnessprevented) r per unit decline n mortality ate* Numberof farmers isitedper measured hange n farm practices number f farm-ers adoptingnewpractices), r numberof farmersadoptingnew practicesper unitincrease n agriculturalproductivity* Miles of read built per unit ncreasen vehicleusage,or new road usageper unitdecreasen traffic congestion.

14

8/13/2019 24pub_br217

23/60

SUSTAINABILITYNDICATORS.Sustainabilityndicatorsepresenthe persistencef pro-jectbenefitsver ime,particularlyfterproject unding nds. hey ouldnclude,orexample:* Diseasencidencerends fterexternalundingor a vaccinationroject nds* Persistencef changedarming ractices fterextensionisitsare completed* Maintenancenduseof roads fterhighway onstructionnds* Persistencef institutionsprograms,rganizations,elationships,ndso on)cre-ated odeliver rojectbenefits.Informationn performances measuredy mpact, utcome, utput,nput,efficacy, Ways f measuringand isk ndicatorsanbe expressedndgatheredn differentways.Thechoice f performanceindicator ndmeans f collection epend n dataavailability,imeconstraints,ndcost-benefitonsiderationsswell as he elationshipsetweenhevariables.DirectmeasuresDirectmeasuresorrespondreciselyo resultst anyperformanceevel.For nstance,quantitiesf goods eliveredr counts f clients erved redirectmeasuresf output,instancesf changen beneficiaryehavior rea directmeasuref project utcome,anda decreasen infantmortalitys a directmeasuref projectmpact. given ari-ablecouldpossiblyerve s an ndicator f resultst anyof variousevels input, ut-put,outcome,r mpact)ependingn heproject bjectives.IndirectmeasuresIndirectmeasuresorrespondess reciselyhandirectmeasuresotheperformancesought. hey re often sedwhere irectmeasuresre oo difficult,nconvenient,rcostlyo obtain.ndirectmeasuresrebased n a known elationshipetweenheperformanceariable nd he measurehoseno expresst-for example, singlowerarmgate rices s an ndirectndicator f increasedgriculturalroductivity,using ecliningreightor taxi ariffsas an ndirectmeasuref decreasingrafficcon-gestion, r using educed umbersf consumeromplaintss an ndirectndicator fimproved ustomerrocessing.Early ointers:ntermediatend eadingndicatorsAt timesnformationn ikelyprojectesultss needed eforeinalperformanceataare available. t otherimest is mportanto gaugewhether projects on rackeven houghinal esults avenotyetbeen chieved.nbothcasesntermediaterleadingndicatorsanprovidean earlyassessmentf performancefigure ).Intermediatendicators easurentermediateesultsr nterveningtepsowardproject bjectives.hey suallymeasurehanges ssociatedith heultimatempactsought ut or whichnformationanbe obtainedarlier.Forexample,ertilizer ur-chases ouldbe used sa preliminaryndicator f changedarming ractices,rincreasedutritionalnowledges an indicator f mproved ating ractices.wopointsmust ekept n mind,however.irst,ntermediatendicatorsften epresent

15

8/13/2019 24pub_br217

24/60

Figure. Early ointers

DevelopmentbjectiveSustainableimberproduction

-4 CL~ ~ ~~C

0 3 5 25 yearsI I

Lifeof project Actual mpactSource:TeamTechnologies,nc.preliminary inks n a causal hain, so heir usefulnessependson thevalidity of thehypothesishat inks hosemeasureso final performanceesults.Second, ntermedi-ate indicators ometimesepresentesultsrom nitialor selected rogramsites,sotheir usefulnessependson theextent o which they prove o be representative.In addition, t is sometimesossibleo identify eading ndicatorsor ndexes f indica-tors)hat are clearly inkedwith longer-termesults.While similar o intermediatendica-tors n concept,eading ndicators enerally ave multiple pplications nd a statisticallyvalid recordof reliability for example,heU.S. ndexof leadingeconomicndicators).Quantitative nd qualitativemeasuresFormany Bank-financedrojects, ndicators f impacts, utcomes,utputs, nd inputsare easilyquantified,hat s, measured y definednumerical alues.These re typi-cally the basisor calculations f economic ateof returnor net present alueduringappraisal.During mplementationhe monitoring nd evaluation ystem f the projector entitybeing inancedcanprovide hesedata by recording, or example,he numberof studentsmatriculated nd he rate of repetition,he numberof farmers isitedbyextensionworkers,or thequantities f fuel consumed, lectricity old,or technicalcapacity ost.Dataon kilometersf road maintained, ollscollected, olumeof traffic,and so oncan easilybe collected. n fact, there s a dangerof collecting oo muchinformation.n some nstanceshecostof collectingdata on projectoutcomess high-for example,a new systemmay have o be developed o measure ducational ttain-ment. t is importanto determine ow precise nd timely nformation eeds o be,andwho will collect t and at whatcost,before nformation-gatheringffortsare nitiated.

16

8/13/2019 24pub_br217

25/60

A project'soutcomes nd impactsmay not manifesthemselves s directly mea-surablenumeric nformation,however.These rojects' effectsmay be felt more ntermsof the attitudesof beneficiaries.For example,do parentsnow have moreinput into their children'seducation? s teachermorale higher?A purely narrativedescriptionof theseeffectsmay be insufficient o measureesults,however. t maybe necessaryo deviseways of measuringoutcomes nd impactsn quantitativeterms,convertingqualitativedescriptions nto quantitative nformation.This conver-sion can be achievedusing survey echniques uchas beneficiaryassessments,rapid rural appraisals,or focusgroup interviewswith structured uestions. heinformationgained through hese echniques an be used o calculatenominal mea-sures, ank orderingsof categories,and frequencycounts.Ratioand interval scalescan also be used.The potential or quantifying nformationon project esultsshouldnot be underestimated.MeasurementcopeMeasurement cope efers o the use of samplepopulations.Performancendica-tors sometimesmeasure esults irectly for an entire arget population (of ndividu-als or organizations) hroughadministrative ecords,observations, r censussurveys.Often, however,he scopeof measurements limited o a sampleof tar-gets or sites.Thisapproach raisesan additional technical ssue:how reliably canoverall project performancebe statistically nferred on the basis of this sample?Sometimes erformance s measured n only one project setting,or in a very few,as casestudies nsteadof statisticalsampling.While case studies an provide use-ful informationon how projects work (or why they do not work as expected)andhow they can be improved,care mustbe taken (evenmore than in the use of sam-ples) not to assurnehat results rom one site necessarily epresentproject perfor-manceoverall. Accordingly, suchcase studiesare usuallyconducted n the contextof speciol studies seebelow) rather han as a replacementor broaderperformancedata.Special studiesSometimes project's routine monitoringand evaluationdata do not provide suffi-cient information. f an unexpectedproblem arises,additional in-depthanalysisthrough specialstudiescan guide the way toward solving t---and avoiding t in thefuture.Specialstudiesare formativeevaluationsof the fundamentals f problemsand their origins, and in that way differ from monitoring ndicators,which are partof an early warning system.For nstance,project managersmightneed o learnmoreabout thecausal inksamong project outputs,outcomes, nd impacts,espe-cially when indicators eveal hat the broader purposesof a project are not beingachievedeven hough ts planned outputsare being delivered. n thiscase some-thing clearly is wrong: the project ogic may be faulty,assumptions bout risk andenabling factorsmay be invalid, or some necessarynput may be lacking. Specialstudiesoften provide mportant eedback or project redesignand higher-level ol-icy debates.

17

8/13/2019 24pub_br217

26/60

General rinciples or Threegeneralprinciplesshouldguide the selection f performancendicators:ndica-selectingndicators tors mustbe meaningful nd relevant,a reliable systemor collecting herequisitedata mustbe developed n a timely ashion,and the borrower'snstitutional apacityfor usinga monitoringandevaluation ystem-and itswillingnesso do so-must betaken nto account.These eneral principlesmply he following considerations:* Relevance. he ndicators electedmustbe relevant o the basic sectoral evelop-

mentobjectives f the projectand, if possible, o overallcountryobjectives.* Selectivity. he ndicators hosen or monitoringpurposes houldbe fewand mean-ingful. t isrecommendedhat he Bankmonitorno more hana dozen ndicators,at leasthalf of them mpact ndicatorshat explicitlymeasure roject mpactagainsteach major developmentbjective.* Practicality f indicators,borrowerownership,and data collection. f performanceindicatorsare to meaningfullyeflecta project'sobjectives, hey shouldbe selectedjointly by the borrowerand the Bankduring participatoryprojectpreparation,andthe data hey measure houldbe useful o bothprojectand country.Thedatarequired o compile he key ndicatorsmustbe easilyavailable; f collecting hedata will requirea specialeffort, his needshouldbe identifiedearly n the projectcycleand included n the projectdesign.Datacollectioneffortscan thenbe pro-grammeddirectly nto project mplementation r contractedo institutionshat dosurveywork at the sectoral, egional,or national evel.* Intermediate nd leading ndicators.n the absence f moredefinite mpact ndica-tors, early pointersof developmentmpactmay be usedduring project mplementa-tion to indicateprogress owardachievingprojectobjectives. n manycasesoutcome ndicators ogetherwith indicatorsof risk actorscan serveas suitableintermediate r leading ndicatorsof impact.

Quantitative nd qualitative ndicators. o heextentpossible, erformancendica-torsshouldallow for quantitativemeasurementf developmentmpact.For someprojectobjectives for instance, apacitybuilding) t may be necessaryo developqualitativendicators o measure uccess, hich shouldstill allow credible and dis-passionatemonitoring. Fordetailson converting ualitative ndicators o quantita-tive measures f the qualitative spectsbeing nvestigated, ee pages 16--17.)

Problemswith defining ndicatorsA previous ectiondescribed he different ypesof indicatorsused o measureevelsofperformance.However,t is sometimesifficult o distinguish etween project'sout-putsand outcomes,or example,or betweenoutcomes nd impacts.When definingindicators, t is important o thinkof the typologyof indicatorsas a continuummirror-ing the ogical means-endelationship f the project: nputs o variousactivities,whichyield outputs, ll of whichcontribute o outcomes nd impacts.A particularmeasure'slogical relation o the project'sstrategic bjectivewill define he ype of indicator hatit represents.Consideran agriculturalextension roject.Two possible ndicatorsare the numberof farmers rainedand the numberof farmersadopting he recommendedechniques.

18

8/13/2019 24pub_br217

27/60

Thesecondndicators obviouslyn outcome,utwhatabouthe irst? rainings anoutput f theproject, ut t is alsoan inputnto mprovinggriculturalractices.hisexampleemonstratesheneed othinkof indicatorsn erms f a continuum,ithinputseadinghrough ctivitieso certain utputshat eado desired utcomesndimpacts.t s notalways asyo distinguishetween utputs ndoutcomesrimpacts, hereasnputs re usually traightforward.onsider projecthathopesoimprove irl'semploymentrospectsy improvingheiraccesso formalandvoca-tionaleducation.n outputndicator ouldbe henumber f girlsgraduatingromsecondaryndvocationalchools,n outcomendicatorwouldbe henumber f girlsemployedromproject chools, ndan impactndicatorwouldbe heirearningsela-tive o earlierevels r relativeo averageohort arnings.Problems ithmeasuringmpactImpactndicators re hemost ifficulto measurendcollect,mainly ecause f lagsbetweenrojectmplementationnd mpact r,putanother ay,betweenhe imeofimpact nd he ime t is easibleo collect ata elatingo mpact.But hemonitoringofprojectmpact uringmplementations oneofthemainmotivationsorusing er-formance onitoringndicators. singeadingndicatorsnd ntermediatendicatorsas proxiesor mpacts a way otacklehemeasurementroblem. eneficiaryssess-ments,apid uralappraisals,nd ocus roup nterviewsre useful aysofcollect-ing qualitativempact ata.Before ndafterhouseholdndcommunityurveys reexcellentools or collectingomprehensivempact ata,but he imeandcostrequiredmakehemmpracticalor regular rojectmonitoring.hey remoreoftenused uringprojectdentification,ntermediateheckpointssuch s midtermeviews),implementationompletion,ndex post valuation.Whenselectingndicators uringproject reparationndappraisal,heborrower Indicatorenchmark(with heBank's ssistance,s needed)hould sebaselineataandcomparative and nternationaldata romotherprogramso set argetsor the ndicatorst will monitor-thats, he comparatorsminimumalueshat t expectso aim or.Some f the ndicatormenusssued ytheBank'sCentral icePresidenciesrovideomparatoralues san indicationf whatis highand owfora givenvariable.Bank taffshouldonsulthe elevant ectordepartmentsf theCentral icePresidenciesn heselectionf ndicatorsor a pro-jectandshould eekheiradvice n targets.Howdo performance onitoringndicatorsffectheBank's ork?Indicatorslayan ntegralole n heBank'swork, romeconomicndsector ork,throughheentire roject ycle, oex post valuation.Indicatorsfsector erformanceanbe bothderivedromandusedn theanalysis Economicnddoneas partof economicndsector orkandusedater o inform roject evelop- sector orkment, rojectmpactmonitoring,ndsectormonitoring.esidesroviding frame-work or evaluatingector erformance,ector-specificerformancendicatorsan

19

8/13/2019 24pub_br217

28/60

helppolicymakers,askmanagers,ndBank taff ationalize country ssistanceand ending trategy,elect roject trategies,ndcreate frameworkor monitoringproject ctivitiesseebelow).Sectoralndicators,btainedrom ector orkor fromgenerallyvailable ector ata,areespeciallyseful uringprojectdentificationndappraisalo clarifyproject bjectivesnd ater o assessmpact.Projectycle A schematicepresentationf howperformance onitoringndicatorsit into he pro-jectcycle s shownn igure5.

Projectdentification,reparation,ndpreappraisalProject lanningtarts y defining bjectiveshat eflect project'sesired evel-opmentmpact;hese bjectives aybehierarchicallyrranged ut hey ave o bedefinedn terms f themeasurableargets nwhichperformancendicators ill bebased. he roposedbjectivesuide heselectionf a first-roundetof indicatorsur-ingprojectdentificationnddesign.As partof theproblem, eneficiary,ndobjectivesnalyses,everal uestionsshould ediscussedith heborrower ndother takeholders:hatarewe rying oachieve? owdo we measure hatwe are achieving? hat ypes f indicatorsrmeasuresill we need o develop?What arget alues hould e use?Theprocess f selecting erformancendicators elps orrowers ndBank taffdefine roject bjectives ore learly, etmeasurableoals,andensurehata pro-ject's ctivitiesead ogicallyo the ealization f itsobjectives. uring roject repa-ration t is mportanto develop clear deaof thebaseline alues f impactindicators nd he plannedarget aluesor thedevelopmentmpact f theproject, swell as he meansor measuringrogressthe ogical hainof indicators).This ctivity elps heborrower nd heBank ssesshe ealism f project bjec-tives nddetermineataavailability ndcollection ethodsor measuringndicators.(The extsection iscussesssues ssociated ithdefinition f objectives nddatameasurement.)oncentratingn how o quantify ndmeasurempact t thisstagehelpsheborrower nd heBank mprovehedesign ndconsistencyf project om-ponentso achievehedesiredesults s cost-effectivelys possible.hiseffortmayFigure. Performancendicatorsndheprojectycle

PROJECTDENTIFIEATION,REPARATION,REAPPRAISALPerformaselinetudiesAnalyzeectoraloatIdenlifyisk nd nablingactors

PREPARATION/ IMPLEMENTATION/ SUPERVISION/ COMPLETIONAPPRAISSAL SUPERVISION OOMPLETION ONWARDSIdentifyisk,nput, Usendicators Evaluateutputnd Evoluateutcomendoutput,utcome,nd lo monitor outcomendicators impactndicatorsfimpactndicalors implemenitlion projectuccess

20

8/13/2019 24pub_br217

29/60

also point to the need o developmonitoring apacity before mplementinghe-projector to buildmonitoringndevaluationapacity evelopmentnto heproject.Appraisal ndnegotiationsAsproject lannings inalized, roject bjectivesre ranslatedntospecificrojectcomponents.ndicatorsor measuringrogressoward achobjective nd or monitor-ing heprovisionf projectnputs nd hestate f riskandenablingactorsdentifiedntheeconomicnd isksensitivitynalysesre also inalized uringappraisal.Themplementationlandevelopedy theborrower ndappraisedy heBankmustncludeheperformanceonitoringndicators-inputs,utputs,utcome,ndimpactndicators-for ach spect f projectmplementation.uring ppraisalhebor-rower nd heBank gree n no morehanabout half-dozenf themostmportantinput ndoutputndicatorsndabout hesame umber f outcomend mpactndica-tors-keyperformancendicatorshat heBankwill monitor. hesendicators-asagreed erformanceenchmarks,otcovenantedargets-are etout n heprojectappraisal ocumentationnd n theBank'segalagreementith heentity esponsiblefor projectmplementationsee ankProcedure0.00,Annex and elatedOperational emorandums).ncases herearget alues recritical o theattainmentof project bjectives,heborrower nd he Bankmayalsowant o makehemcovenantf the oanagreementsee age26).ImplementationndsupervisionDuring rojectmplementationheborrower ndprojectmplementationnitmonitorthe ndicators eveloped uringproject esign,ollowinghe schedulef the mple-mentationlan.During egular upervision issionsnd hemidtermeviewheBankmonitorshekey ndicators efined ndagreedwith heborrower t appraisal. heseindicatorsorm hebasis or theBank'smeasuresf mplementationerformanceIPratings) nd mpact developmentbjective, r DO ratings) n Form 90. If the ndi-cators riginally evelopedor he project ecomenappropriateecause xternalconditionshange r project esigns restructured,he Bank ndborrower evelopnew ndicators o reflect he changedcircumstances.EvaluationAfter mplementationscompleted,heachievementf project bjectivess assessedusingmeasurablendicators-theutcomend mpactndicatorsevelopeduring ro-jectdesign-that ompareheproject'sctualmpact ith ts ntendedmpact. he seofindicatorsemovesubjectivityromevaluations,iving hem more bjectiveasis.The ogicof project esign overnshe ogicof monitoring:he mplementationnit Performancemonitorsnputs ndoutputsn greater etail handoes heborrowing overnmentr indicators ndthe Bank.mplementationanagersreprimarily oncerned ith he actics f pro- projectmanagemengram mplementation,hileborrowers ndBank taff re more oncerned ith hestrategicmplicationsf projectmplementationndalternativesor realizing rojectobjectives.

21

8/13/2019 24pub_br217

30/60

Tomeethedifferentnformationeeds f thevarious takeholders,erformanceindicatorssually eed o be hierarchicallytructured.his tructureanbeaccom-plishedhrough estedetworksf partially verlappingerformancenformationys-temsn which ower-levelerformancendicatorsresummarized,r selectivelyskimmed,s a basis or higher-levelnformation.he ollowing ectionsescribehedifferent erformancendicators nd nformationeeds f thevarious ctors esponsi-ble or projectmplementation.Thisnformations summarizedn box 6.)Field-level mplementersImplementationanagersre primarily oncerned ith hemethodssed o mple-ment particular roject r program omponent. ithin heir elatively arrow reasof responsibility,heyneedo know hatnecessarynputs human,inancial, ndphys-icalresources)re available nd hatplanned utputsgoods ndservices)re beingproduced. hey lsoneed o know hat esourcesre beingusedegitimately,hatfinancial ccountabilitys ensured, nd hat nputs rebeingefficientlyransformedintooutputs.mplementationanagersn the ieldalsoneedat east omenformationon program utcomesnd mpacts;hat s, theyneed o know heextento whichpro-gramoutputs re achievingheir ntendedesults o hat heycan ine-tunerojectimplementationndmotivate erformance.ImplementationnitmanagersThemanagersf an implementationnit orof whateverrganizationalnit s responsi-ble oroverall rojectmanagement)re usuallyesponsibleor mplementingn entire

Box. Projectanagementnd erformancenformationeedsIMPLEMENTERSN THEFIELD EED* Inputndicatorsa Outputndicators* [Efficiencyndicators]e Riskndicators* Some utcomend mpactndicatorsTHE MPLEMENTATIONNITNEEDS* Summarynput nd utputndicators,ncludingite-comparativendicatorssappropriate* Outcomendicators,ncludingite-comparativendicatorssappropriate* [Effectivenessndicators]* Riskndicatorsa ImpactndicatorsTHEBORROWERND THEBANKNEED* Summarynputndicators* Summaryutputndicatorsa Riskndicators* Key utcome,mpact,and elevance]ndicatorse [Sustainabilityndicators]Note: Indicatorsnbracketsre ot requiredart fBankonitoringr projectupervision.

22

8/13/2019 24pub_br217

31/60

program r a majorproject omponent. his esponsibilitynvolves rangeof activities,oftenacross number f sites,aimedat achieving ome trategic bjective.Becausethesemanagers avea primary esponsibilityor project mplementation,heyshouldplay a supportingole n definingstrategic bjectives,hoosing mong hemor articulat-ing he underlying olicies romwhich heseobjectives eriveduringprojectpreparation.Implementationnit managers re concerned ith managing heir programbetter,selecting nd ine-tuning rojectactivitieso improveoutcomes, nd enhancing rojectimpact.Thus hey needsummarynformation n project nputsand outputs,particularlycomparativenformation cross ites,so that hey can dentifyemergingproblems nddirect managerial ttention nd resourceso solving hem.More mportant, hey needcomparativenformation n projectoutcomes.mplementation anagers lso needatleastsome nformation n project mpact o validate he significance f their projectstrategy.Borrower fficialsand BankstaffRepresentativesf the borrowerand Bankstaffare primarilyconcerned ith strategicmanagement.heyhaveparticipated n articulatinghe policyassumptionshatunderliethe projectand he strategic bjectivesheproject s ntended o achieve.Moreover,they must ontinue o push hese bjectivesn thecontext f a dynamicexternalenviron-ment.Thushey needsummarynformation n projectnputsand outputso be sure hatany emergingmplementationr outcome roblems re being dentifiedand addressedat appropriateprojectmanagementevelsand to be able to answerquestions bout heproject.To managestrategically, owever, orrowers nd Bankstaffmostneedcompar-ative nformation n projectoutcomes nd impacts nd risk actors.When mpact smeasuredn relation o thecostof project nputs, his nformation an also eeddirectlyinto a performance-basedudgetsystem. naddition, n the context f the CountryPort-folio Performance eview r CountryAssistance trategy pdate, he borrowerandBankmanagers eed nformation n the national rends o whichprojects re expectedto contribute.Some elatedssuesSeveralactors hatare part of good practice n monitoring nd evaluation re ntegralto the effective seof projectperformance ata, n particulardata collection nd man-agement, he nstitutional rrangementsor managing nformation, nd the useof feed-back rom monitoring ndevaluation.n addition, users eedo be aware of thedifference etweenoan covenants nd indicators nd potential itfalls n the definitionof indicators. or performancemonitoringndicatorso work, a managementtructureand incentiveshatvalue esultsmustbe in place.Performancendicators re a tool:ontheir own hey can do nothing,but n the properenvironmenthey nspireaction.It is critical hat managers-Bank,borrower,and projectmanagers-regard perfor- Goodpracticenmancemeasurements an integralpart of their nstitution'smissionand strategic monitoringndplan.10 Often hey do not; nstead, hey see t as an adjunct o the plan, n the same evaluation

23

8/13/2019 24pub_br217

32/60

way hat heymayseeevaluationsa requiremento besatisfied fter hemoreimportant orkof projectmplementations done. lTobe effectivemonitoringndevaluationmust eaddresseduringproject esign.Once rojectplannersavecarefully efinedhestrategic bjectivesndselectedogicalndicators,heirnextconsiderationhould e he equirementsor datacollectionndmanagement,nstitu-tionalarrangementsndcapacity uilding, nd heuseof feedbackrom hedata.DatacollectionndmanagementThenature f a variable etermineshe ogical ource f dataabout t. Indicatorsfinputs ndof some utputs f project ctivitiesre derived rimarilyromprojectrecords;owever,ndicators f some utputs, utcomes,nd mpactsmay equirehatdatabecollectedsing urveys rspecial tudies,ncludinghosehatuseparticipa-torymethods. heret is possible,t is almost lways etter o piggybackegular ur-veys ntoexisting ationally r internationallyupportedurveyssuch s heLivingStandards easurementtudy)han o create datacollectionacility.Special tud-iesmaybe managed y theproject nitdirectly r subcontractedo a private ntity.Any proposalo collect ata oran indicatorequires discussionf:* Objectivesf thestudy r survey* Sourcesf data* Choicesnd proposed ethod f collection* Likely eliability f thedata.

Collectionf somendicators,articularlyutcomend mpactndicatorssuch smorbidity ndmortalityates, ducationalchievement,rcropproduction) aydepend n heexistencendqualityof national ensusr surveyystems.eforechoosinguch rojectndicators,heborrower nd heBankmust onfirmhat henecessaryystemsre n placeand eliable nd hat heavailable ataarestatisti-callyvalid or thepopulation r areacovered y he project. he omplexityf statis-ticsand heproblems f attributingausalitymean hat n many asest is betterouseservice elivery ndbeneficiaryesponsesproxymeasureshan o attemptomeasurempact irectly.Manyoutputndicators rederivedrom ecords eptby he participatinggen-cies,oftenat projectieldsites.For his eason,or purposesf projectmonitoringandevaluationesignincludingndicator election),roject lannershould xaminethe mplementer'secord-keepingnd eporting rocedureso assesstscapacityogenerate ata.The ssentialoints re hatdatashould ecollected ndused loseo thesourceand hatdatacollectione cost-effectivend eliable.t s mportantot o createseparatemeasurementureaucracyithina project tructure. aving uch bureau-cratichomeor dataproductionsnot usuallyost-effectivend presentshe isk hatthoseesponsibleor producinghedatamayhaveittlecontact ith hoseesponsi-ble or usingt. Thushedatausers o not ullyunderstandhats behind henum-bers heyare given o use, nd hedatacollectorsave ittleappreciationorthe

24

8/13/2019 24pub_br217

33/60

issueshat stakeholders nd seniorproject managersace, for whichperformancedatawould be useful. ustas performancemeasurementhouldbe fully integratedwith projectdesign, nformation enerated houldbe integratedwith the project'smanagement tructure. s with any otherprojectcomponent,he benefitsof usingper-formancedata mustat leastequal he costof collecting hem.A projectcan providethe best nformation t the leastcostby usingexistingdata, sampling echniques,rapid appraisal methods, nd othercreativecollectionmethods.A few otherconsiderations bout data collection houldguide he designof indica-tors. Aboveall, thedata shouldmeasure esults, ot ust processes. heperformancemeasured y thedata should ocuson what the project s accomplishing, specially ntermsof its mpacton people.Thepoint s not only to know what projectsare doing,but to know whether hey are doing any good.Selectivitys desirable.Performance nalysisshouldbe limited o the few areasthat are directly relevant o the project'sstrategic bjectives, s definedby theborrowerand Bank.Finally,performancemeasurementystems houldusedata hat are not construedas threatening y thosewho are reporting t. Simply elling projectmanagers-whetherborrowers,Bankstaff, or thoseworking or them-to report data on their pro-jectswithoutactively nvolving hem n the performancemeasurementrocess,withoutexplaininghow and by whom hedata are going to be used,and withoutassuringthem hat he datawill not be used o judge thempersonallys likely o be viewedasthreatening.Relyingon sucha compliancemodeof performancemeasurementslikely o backfire,not only minimizingmanagerownershipbut also ikely producingbureaucratic esistance nd possibly ata corruption.Institutional rrangements ndcapacitybuildingDatacollectionarrangements ave mplicationsor the nstitutional rrangementsbetween he implementationnit, he borrower, nd the Bank.The ationale or indica-torsdemands hat he ndicatordata be of value o the borrower, ot just o the Bank.A project'soverallmonitoringand evaluation esignshouldbuild on the reportingarrangements lreadyusedby the mplementation nit and he borrower,while furtherdeveloping he technical kills heyneed o plan nformation eeds,designdata collec-tion, execute tudies nd surveys, nalyzedata, and report esultsn a format hat srelevanto projectmanagement.f necessary,heBankcan finance echnicalassistance nd equipment equired o designand implementmonitoring nd evalua-tion systems nd o meet rainingneeds.Feedbackrommonitoringandevaluationeffortsand interpretingndicatorsCare shouldbe taken o time data collectionactivitiesso hat nformationwill beavailablewhen t is needed, orexampleat the time of the Bank'sportfolioormidterm eviews.Projectsor which operatingperformance tandards re specified san objective,or for which decentralized rocessesall for local capacity o plan andmanagework programs nd budgets,will need specialconsideration f how indica-tor findingsare used o inform decisions.

25

8/13/2019 24pub_br217

34/60

Project ndicators rebest nterpretedhrough omparison.deally, mpacts evalu-atedby comparingbaseline ata with projectoutcomes,o providean ideaof what theindicatorswouldbe if the projecthad notbeencarriedout. Butbecause uchanalysisssometimesot practical,analysis f indicators anuseotherkindsof comparisons:actual esultselative o targets, eforeand afteranalysis, imeseries more llustrativethan simplebeforeand aftercomparisons),r comparisons f controlgroups.Trends r fluctuations re common n the values hat are used o measure utputsoroutcomes nd impacts. f this s the case, ime seriesor controlgroupsshouldbeused.Timeseriesdatacapture rendsor fluctuations,nd controlgroupshelp verifyattributionof causality.For example,he controlgroupchosen s criticalfor compar-ing the ncreasen averageeducationalattainment r incomeamong a project'sben-eficiarieswith the increasen a nonproject rea: thecontrolgroupmustbe identicalto the projectsample n all respects ther han he presence f the project.Finally,anyinterpretation f indicatorsmust onsiderhe relationship etweenexogenousactors

and the indicatorshey may nfluence. his s one reasonwhy the monitoringof riskfactors s so important.

Relationo loan Indicators re factsabout project mplementationesults, otactions.Thus tandardcovenants practiceby the Bankand borrowerss to consider erformanceargets s indicativeand not egallybinding.Theunderstandings that he agreed argets re usedprimarilyto gaugeprogressn project mplementationnd n realizingdevelopmentbjectives.However,n instanceshereattainment f certain ndicators r targetssconsideredessentialo the attainment f a project's bjectives forexample,n the caseof certainfinancial ndicators f a profit-makingntitysupported y a project), hese ndicators(target alues) houldbe ncorporatedn the egaldocumentsotonly asperformancebenchmarksutalso as a loan covenantseepage 21).Workingat cross Projects ometimesack well-defined bjectives ecause he borrowerand the Bank

purposes cannotagreeon what a project'spurposes re or shouldbe. Forexample, t is some-timesunclearwhethera project'sultimate bjectives to transferesourcesr to trulyalleviatepoverty.A project'spurposes nd strategic bjectivesmustbe agreedonearly,and performancendicatorsprovide he clearest uide for management ctionwhenthey ogically eflectclearprogrampriorities. n practice,however, rogramsmay encompassmultipleand evenconflictingobjectives.While it is difficult o portraymultiple trategic urposeswithin a singleobjectivesreeor logicalframework,t maysometimese possibleo defineparallelobjectives rees eflectinghesemultiplepur-poses.The performance atacould thenbe usefuln makingdecisions hatresult ntradeoffs mong competing bjectives, speciallyf suchobjectives an be prioritizedor weighted.2Anotherdifficulty s thatclearlydefinedobjectives ringa measure f accountabil-ity that maymakesomeofficials uncomfortable.n somecases mbiguous rojectobjectivesmay, n fact, be politicallydesirable.Delineatingusefulperformancendi-catorsmay provedifficult f this s thecase;and in the absence f willingnesso com-mit o clearlydefinedobjectives, heBankshoulddissociatetself rom he project.

26

8/13/2019 24pub_br217

35/60

In theendperformance onitoringndicators nd he eedbackheyprovide reonly Lastwordas goodas heunderlying nalysiseconomicnd inancial nalysis,conomicndsector ork,social ndenvironmentalssessment)upportingheproject esign, ndthedata o beassessedver ime.The ogical rameworkpproacho projectormu-lation s onlya structureor project esign ndevaluation,ot he ull extent f projectdesign r evaluation.oneof the oolsdescribedn hishandbookan replacesound conomic,inancial, ocial,environmental,nd iskandsensitivitynalysis rcomprehensiveonitoring ndevaluation.ogether,heanalysis,he ogical rame-work,and he ndicatorsorma systemor continuousnalysis nda holisticapproacho project esign,monitoring, ndevaluation.Notes

1. Effective mplementation:Key to Development Impact,September 1992.2. An Overview of Monitoring and Evaluation in the World Bank, OED Report 13247, June 30, 1994,

and Monitoring and Evaluation Plans n ProjectsApproved in Fiscal Year 1995, OED draft report, October6, 1995.

3. The logical framework approach and indicator typology described in this section draw extensivelyfrom material prepared by R. Moses Thompson,Team Technologies nc.; Gerald M. Britan, 'MeasuringProgram Performance or FederalAgencies: Issuesand Options for Performance ndicators,' prepared forthe U.S. General Accounting Office, 1991; and 'Guidelines for Result-Based lanning, Management andMonitoring, preliminary version, from the Swedish International DevelopmentAgency, January 1995.

4. Several formal techniques or collaborative project planning and managementare variations on theoverall logical framework concept describedhere: Logical frameworks, ZOPP,and TeamUP, or example.Appendix 1 of the World Bonk Porticipation Sourcebook (February 1996) describes ZOPP objectives-oriented project planning), TeamUP, nd other methods and tools that support participatory development,and provides additional references.

5. The relationship among project objectives may also involve aggregation across spoce, with results ndifferent project locations added to yield regional objectives and achievements.Hierarchically linked andspatially aggregated objectives often exist in tandem. For instance, a notional child health project whoseoverall objective is to reduce child mortality may have a subobjective of reducing the ncidence ofpreventable disease by educating and inoculating a certain number of individuals nationwide-that is, thesumof the individuals reached by individual clinics throughout the notion.

6. See also Handbook on EconomicAnalysis of InvestmentOperations (Washington, D.C.:World Bank,1996), Chapter 10, Risk and SensitivityAnalysis.7. Various formal logical framework techniques use tools such as problem and objective trees, situation

and SWOT (strengths-weaknesses-opportunities-threats)nalyses, and project planning matrixes to graphi-cally represent he problem, beneficiary, and objectives analysis.

8. Even f a borrower is not explicitly aware of or does not follow the logical framework approach indesigning a project, the Bank can follow the approach to appraise the project.

9. This discussion draws heavily on a report of the EastAsia Working Group on Project PerformanceMonitoring Indicators prepared for consideration by the EastAsia Regional Management Group. The work-ing group included staff from the EastAsia Region, the Asia TechnicalDepartment, HumanResourcesSer-vices, OED, and Operations Policy.

10. For additional discussion,seeMonitoring and Evaluotion Plans n StaffAppraisol Reports ssued nFiscal Year 1995: A Follow-up o OED's Report 'An Overview of Monitoring and Evaluation in the WorldBank,' OED Report 15222, December 29, 1995.11. ThomasJ. Cook, Jerry VanSant, Leslie Stewart, and JamieAdrian, Performance Measurement:Lessons earned or DevelopmentManagement, World Development23(8): 1303-15 (1995).

12. Briton, p. 20.

27

8/13/2019 24pub_br217

36/60

SECTORAL NOTES ON INDICATORSTo help Bankstaffand borrowers electand useperformancendicators, he Bank sdeveloping ectoralnoteshat discusshe useof indicatorsn relation o the majorobjectives r categories f problems ormallyaddressedn Bank-financedrojects.Eachnote ncludesmenusof possiblendicators.What opics o thenotes over?Eighteen ectoralnoteswill eventually e prepared.Theywill covereach sector nwhich the Bankworksand areasof emphasishat crosseconomicand socialsectors,suchasenvironmental oncerns, overty eduction,publicsectormanagement,ndtechnical ssistance. f the sectorperformancendicatornotesplanned,seventeenhave been ssued:

Agriculture Poverty eductionEconomic djustment PowerEducation Private ectordevelopmentEnvironment Technical ssistanceFinancial ector TelecommunicationsHousing TransportIndustryand mining UrbandevelopmentOil and gas Water and wastewaterPopulation, ealth,and nutrition

Theonly note hat remainso be issued s public sectormanagementexpected nJune 1996). All of the notesexceptone are in their irst edition, hat is, they will berevised s the Bankand its clientsgain experiencewith theuse of indicators. Theeducationnote s in its second dition and is consideredo be an exampleof bestpractice.)Furtherwork on thesenoteswill: Standardize hem o make hemmoreconsistent ith the methodologicalframeworkdescribed arlier* Incorporatemorebestpractices nd lessonsearned* Incorporatenternational omparator ata at the globaland regional evels o facil-itate benchmarking mongcountries, rovinces, egions,and so on, to appreciatewhat is high or low in a given instance.

28

8/13/2019 24pub_br217

37/60

Howwere henotes eveloped?As part of the Next StepsAction Plan, heCentralVice Presidencies,ectordepart-ments, nd Development conomics ice Presidency ere asked o developsector-specificperformancemonitoring ndicators. he OperationsPolicyDepartment OPR)coordinated he efforts o develop he ndicatornotes, acilitating heexchange fideasamong sectordepartments nd reviewing he product.After severaldepartments ad produceddrafts,OPR ormeda quality review groupcomprisingseniorstaff romthe Regions,he CentralVice Presidencies,nd OED.The groupreviewedall of the notesand developeda framework or consistencyn approach.From hese nitial efforts, he Bankdeveloped he framework or performancendi-cators hat has been applied n mostof thesenotes.Most of the sectornotes:* Followa typology of indicatorsbasedon a logical frameworkapproach o projectdesign (indicators f project nputs,outputs,outcomes,mpacts, iskand enablingfactors,efficiency, ffectiveness,nd relevance); nd Providean executive ummary, menuof recommendedey ndicators,and real

examples f indicatorsused n Bank-financedrojects.Howare indicatorselectedrom henotes?Thenotesare referencematerials o specific sectors nd to overarchingconcernssuchas poverty eduction,macroeconomic djustment, nd environmentalssues.Theyare meant o guide taskmanagersand borrowers n applying performancemonitoring ndicators.They are not, however, ntended o replace he udgmentorknowledgeof task managers r borrowers.The notesprovidemenusof indicatorsthat are neitherexhaustive or mandatory-indicatorsare project-specific nd mustbe customized y the borrowerand taskmanager o project,sector,and countrycircumstances. inceevery projecthas ts own uniqueobjectives, ask managersand borrowers mustdevelop ndicators hat correspond o theseobjectives,and notrestrict hemselveso the menus rovided n the notes.Conversely, are shouldbetaken notto order the entire menu; ather, t shouldbe treatedas an indicative istfrom whichto choose he mostappropriate selections.Whereare thenotes vailable?The notesare issued y theoriginatingCentralVicePresidencieso all SeniorOpera-tionsAdvisers, irectors,projectadvisers, nd sectordivisionchiefs.Staffmembersshouldhave received opiesof the notespertaining o the sectorshat heir departmentcovers r havebeen notified hat he notesare available.Additionalcopiesare avail-able from he originatingdepartments. henoteswill also be available n electronicform through he Bank'sEnterprise etwork contacthe relevantaskmanager istedbelowto confirma note'selectronic vailability).Thedepartments,askmonagersorthe notes, nd personso contact or a copy of the noteare

29

8/13/2019 24pub_br217

38/60

Sector Department Task manager Contactpersonand extensionAgriculture AGR Cornelis e Haan JoyceSaboya 38959Economic djustment DEC DeborahWetzel DeborahWetzel 31698Education* HDD Sverrir igurdsson Joe-Shin ang 81418Environment ENV JohnDixon JohnDixon 38594Financial ector FSD DavidScott/ HediaArbi 34663MonikaQueisserHousing nd urbandevelopment TWVU Patricia nnez LauraO'Connor 37009Industryndmining IEN FelixRemy Elisa orre 80323Oil and gas IEN VVilliam orter KyranO'Sullivan 32722Population,ealth,andnutrition HDD TomMerrick VivionOctran 33639Povertyeduction PSP SoniyoCarvalho SoniyaCarvalho 35705Power IEN Jean-Pierreharpenlier KyranO'Sullivan 32722Private ector evelopment PSD SyedMahmood ShirleyWallace 38 1 1Public ectormanagement PSP MichaelL.O Steven.s MichaelStevens 37493Technicalssistance OPR NimrodRaphoeli NimrodRaphaeli 84015Telecommunications IEN RogatiKoyani Rogati oyani 34515Transport TWU ColinGannon Colin Gannon 85784Waterand wastewater TWU Guillermo epes Rose oole 33749* Second ditionnote ssued.How o earnmoreBesides evelopingheirsecond-editionotes, heCentral icePresidenciesre dis-seminatinghe irstedition otes nd advising egional taff n theuseofperformance onitoringndicators.heCentralVicePresidenciesill continueosponsor orkshopsn their ndicator otes nd, when equested,dvise roject taffon allaspects f theuseof performance onitoringndicators.n addition,he Bank'sLearningndLeadershipenter s providingraining n performancendicatorsspart of its egular ourses n project reparation,ppraisal, nd supervision.

30

8/13/2019 24pub_br217

39/60

EXAMPLES OF INDICATORSThe ollowing examples f key performancendicatorswere developed or severalBank-financedrojects.These rojects epresent urrentbest practice n the useofperformancemonitoring ndicators;mostwere highlighted n OED's iscal 1995follow-up eview of monitoringand evaluation Monitoringand EvaluationPlans nStaffAppraisal Reportsssuedn FiscalYear95, Report15222). Foreach project,amatrixpresentshe project'sobjectives,with indicatorsof inputs,outputs, isk factors,outcomes, nd impacts.The matrixes o not ist all the detailed ndicators isted n theborrower'sProject mplementation lan; hey only show he key ndicators hat heBankwill monitoras the basis or projectsupervision nd evaluation.Thesematrixeswere devisedby the project's askmanagerswith OPR's uidance.The nformation etout in thesematrixeswas alsogiven n the StaffAppraisal Reportfor each of the projects, lthoughnot n this ormat. n the future he most mportantprojectoutcomeand impact ndicators, uchas thosepresentedn thesematrixes,shouldbe listed n everyprojectappraisaldocument nd monitoredusingForm590.

31

8/13/2019 24pub_br217

40/60

Table. Summaryfobjectivesnd eyperformancendicators,HondurasasicducationrojectStaff ppraisaleport3791HO,March, 1995)RISKSNDCRITICALINPUTS OUTPUTS A