Date post: | 08-Apr-2018 |
Category: |
Documents |
Upload: | robert661989 |
View: | 215 times |
Download: | 0 times |
of 37
8/7/2019 Impact Eval and Project Cycle
1/37
Impact Evaluation andImpact Evaluation and
the Project Cyclethe Project CycleArianna LegoviniArianna Legovini
PREM WEEK May 2, 2006PREM WEEK May 2, 2006This presentation is based on work by theThis presentation is based on work by the
Thematic Group on Impact EvaluationThematic Group on Impact Evaluation
8/7/2019 Impact Eval and Project Cycle
2/37
Objective of the presentationObjective of the presentation
Walk you through what it takes to do anWalk you through what it takes to do an
impact evaluation for your project fromimpact evaluation for your project from
Identification to ICRIdentification to ICR
Persuade you that impact evaluation willPersuade you that impact evaluation will
add value to your projectadd value to your project
8/7/2019 Impact Eval and Project Cycle
3/37
We will talk aboutWe will talk about
Objective of the evaluationObjective of the evaluation
General PrinciplesGeneral Principles
Evaluation activitiesEvaluation activities the core issues forthe core issues forevaluation design and implementation,evaluation design and implementation,
andand
Housekeeping activitiesHousekeeping activitiesprocedural,procedural,administrative and financial managementadministrative and financial management
issuesissues
8/7/2019 Impact Eval and Project Cycle
4/37
Why do an impact evaluation ofWhy do an impact evaluation of
your project?your project?Provide a sound basis for policy developmentProvide a sound basis for policy development
Measure the impact of the project on intended andMeasure the impact of the project on intended and
unintended outcomes for:unintended outcomes for: Making budget decisions and reallocating resources (fiscalMaking budget decisions and reallocating resources (fiscal
accountability)accountability)
Scaling up (provisos apply)Scaling up (provisos apply)
Measure the relative effectiveness of alternativesMeasure the relative effectiveness of alternatives(modes of delivery, packages, pricing schemes) to:(modes of delivery, packages, pricing schemes) to: Modify project features overtime (managing by results)Modify project features overtime (managing by results)
Inform future project designsInform future project designs
8/7/2019 Impact Eval and Project Cycle
5/37
Some general principlesSome general principles
Government ownershipGovernment ownershipwhat matters iswhat matters is
institutional buyinstitutional buy--inin
Relevance and applicabilityRelevance and applicabilityasking theasking theright questionsright questions
Flexibility and adaptabilityFlexibility and adaptability
Horizon mattersHorizon matters
8/7/2019 Impact Eval and Project Cycle
6/37
Country ownershipCountry ownership Ensure Government involvement at all stages toEnsure Government involvement at all stages to
build institutional capacity and a culture ofbuild institutional capacity and a culture of
managingmanaging--byby--results.results.
Agree on a dissemination plan to maximize use ofAgree on a dissemination plan to maximize use ofresults for policy development.results for policy development.
Identify entry points in project and policy cyclesIdentify entry points in project and policy cycles
midpoint and closing, for project;midpoint and closing, for project;
sector reporting, CGs, MTEF, budget, for policysector reporting, CGs, MTEF, budget, for policy
Use partnerships with local academics to buildUse partnerships with local academics to build
local capacity for impact evaluation.local capacity for impact evaluation.
Example, Kenya educationExample, Kenya education
8/7/2019 Impact Eval and Project Cycle
7/37
Relevance and ApplicabilityRelevance and Applicability
For an evaluation to be relevant, it must beFor an evaluation to be relevant, it must bedesigned to respond to the policy questions thatdesigned to respond to the policy questions thatare of importance to the clients.are of importance to the clients.
Clarifying early what it is that the client wants toClarifying early what it is that the client wants tolearn and designing the evaluation to that endlearn and designing the evaluation to that endwill go some way to ensure that thewill go some way to ensure that therecommendations of the evaluation will feed intorecommendations of the evaluation will feed into
policy making.policy making. Example, devolution and decentralization inExample, devolution and decentralization in
ZambiaZambia
8/7/2019 Impact Eval and Project Cycle
8/37
Flexibility and adaptabilityFlexibility and adaptability
The evaluation must be tailored to the specificThe evaluation must be tailored to the specificproject and adapted to the specific institutionalproject and adapted to the specific institutionalcontext.context.
The project design must be flexible to secure ourThe project design must be flexible to secure our
ability to learn in a structured manner, feedability to learn in a structured manner, feedevaluation results back into the project andevaluation results back into the project andchange the project midchange the project mid--course to improvecourse to improveproject end results.project end results.
Example, Ethiopia energyExample, Ethiopia energy
This is an important point: In the past projectsThis is an important point: In the past projectshave been penalized for affecting midhave been penalized for affecting mid--coursecoursechanges in project design. Now we want tochanges in project design. Now we want to
make change part of the project design.make change part of the project design.
8/7/2019 Impact Eval and Project Cycle
9/37
Horizon mattersHorizon matters The time it takes to achieve results is an importantThe time it takes to achieve results is an important
consideration for timing the evaluation. Conversely, theconsideration for timing the evaluation. Conversely, thetiming of the evaluation will determine what outcomestiming of the evaluation will determine what outcomesshould be focused on.should be focused on. Early evaluations should focus on outcomes that are quick toEarly evaluations should focus on outcomes that are quick to
show changeshow change For longFor long--term outcomes, evaluations may need to span beyondterm outcomes, evaluations may need to span beyond
project cycle. Example, Madagascar early childhoodproject cycle. Example, Madagascar early childhooddevelopmentdevelopment
Think through how things are expected to change overThink through how things are expected to change over
time and focus on what is within the time horizon for thetime and focus on what is within the time horizon for theevaluationevaluation
Do not confuse the importance of an outcome with the timeDo not confuse the importance of an outcome with the timeit takes for it to changeit takes for it to changesome important outcomes aresome important outcomes are
obtained instantaneously !obtained instantaneously !
8/7/2019 Impact Eval and Project Cycle
10/37
Identification to PCNIdentification to PCN
8/7/2019 Impact Eval and Project Cycle
11/37
Get an Early StartGet an Early StartHow do you get started?How do you get started?
Get help and access to resources: contactGet help and access to resources: contactperson in your region or sector responsible forperson in your region or sector responsible forimpact evaluation and/or Thematic Group onimpact evaluation and/or Thematic Group on
Impact Evaluation and HDN CEO
Impact Evaluation and HDN CEO
Define the timing for the various steps of theDefine the timing for the various steps of theevaluation to ensure you have enough lead timeevaluation to ensure you have enough lead timefor preparatory activities (e.g. baseline goes tofor preparatory activities (e.g. baseline goes tothe field before program activities start)the field before program activities start)
The evaluation will require support from policyThe evaluation will require support from policy--makers: start building and maintainingmakers: start building and maintainingconstituents, dialogue with relevant part ofconstituents, dialogue with relevant part ofgovernment, build a broad base of support,government, build a broad base of support,include stakeholdersinclude stakeholders
8/7/2019 Impact Eval and Project Cycle
12/37
Build the TeamBuild the Team Select impact evaluation team and defineSelect impact evaluation team and define
responsibilities of:responsibilities of: program managers (government),program managers (government),
project team, and other donors,project team, and other donors,
lead researcher (impact evaluation specialist),lead researcher (impact evaluation specialist),
local research team, andlocal research team, and data collection agency or firmdata collection agency or firm
Selection of lead researcher is critical for ensuringSelection of lead researcher is critical for ensuringquality of product, and so is the capacity of the dataquality of product, and so is the capacity of the data
collection agencycollection agency
Partner with local researchers and researchPartner with local researchers and researchinstitutes to build local research capacityinstitutes to build local research capacity
8/7/2019 Impact Eval and Project Cycle
13/37
Shift ParadigmShift Paradigm
From a project design based on we know whats bestFrom a project design based on we know whats best
To project design based on the notion that we can learnTo project design based on the notion that we can learnwhats best in this context, and adapt to new knowledge aswhats best in this context, and adapt to new knowledge asneededneeded
Work iteratively:Work iteratively: Discuss what the team knows and what it needs to learnDiscuss what the team knows and what it needs to learnthethe
questions for the evaluationquestions for the evaluationto deliver on project objectivesto deliver on project objectives
Discuss translating this into a feasible project designDiscuss translating this into a feasible project design
Figure out what questions can feasibly be addressedFigure out what questions can feasibly be addressed
Housekeeping: Include these first thoughts in a paragraph in theHousekeeping: Include these first thoughts in a paragraph in thePCNPCN
8/7/2019 Impact Eval and Project Cycle
14/37
Example: Incorporate learning in anExample: Incorporate learning in an
energy access in Ethiopiaenergy access in Ethiopia One component of the project will distribute CompactOne component of the project will distribute CompactFluorescent Light bulbs (CFLs) at subsidized prices to increaseFluorescent Light bulbs (CFLs) at subsidized prices to increaseenergy efficiency and reduce energy costs for the poor.energy efficiency and reduce energy costs for the poor.
The team wants to know how to deliver the greatest number ofThe team wants to know how to deliver the greatest number of
CFLs under budget constraints. What is the optimal value forCFLs under budget constraints. What is the optimal value forthe subsidy value? What is the best delivery mechanism?the subsidy value? What is the best delivery mechanism?
During the first six months of CFL distribution, the electricDuring the first six months of CFL distribution, the electriccompany will experiment with alterative subsidy values (high,company will experiment with alterative subsidy values (high,medium, low) and distribution mechanisms (local market,medium, low) and distribution mechanisms (local market,
company distribution) in different localities. Localities will becompany distribution) in different localities. Localities will berandomly selected to one or the other treatment.randomly selected to one or the other treatment.
At the end of the first six months, the company with evaluateAt the end of the first six months, the company with evaluatewhich model was most costwhich model was most cost--efficient (use of CFLs per dollarefficient (use of CFLs per dollarspent), and implement that model for the rest of the project.spent), and implement that model for the rest of the project.
8/7/2019 Impact Eval and Project Cycle
15/37
Preparation through appraisalPreparation through appraisal
8/7/2019 Impact Eval and Project Cycle
16/37
Define project developmentDefine project development
objectives and results frameworkobjectives and results framework This activityThis activity
clarifies the results chain for the project,clarifies the results chain for the project,
identifies the outcomes of interest and the indicatorsidentifies the outcomes of interest and the indicators
best suited to measure changes in those outcomes,best suited to measure changes in those outcomes,andand
the expected time horizon for changes in thosethe expected time horizon for changes in those
outcomes.outcomes.
This will provide the lead researcher with theThis will provide the lead researcher with theproject specific variables that must be includedproject specific variables that must be included
in the survey questionnaire and a notion ofin the survey questionnaire and a notion of
timing for scheduling data collection.timing for scheduling data collection.
8/7/2019 Impact Eval and Project Cycle
17/37
Work out project design featuresWork out project design features
that will affect evaluation designthat will affect evaluation design Target population and rules of selectionTarget population and rules of selection
This provides the evaluator with the universe for theThis provides the evaluator with the universe for the
treatment and comparison sampletreatment and comparison sample
Roll out planRoll out plan This provide the evaluation with a framework forThis provide the evaluation with a framework for
timing data collection and, possibly, an opportunity totiming data collection and, possibly, an opportunity to
define a comparison groupdefine a comparison group
The impact evaluator will work iteratively with theThe impact evaluator will work iteratively with theteam to agree on selection and roll out thatteam to agree on selection and roll out that
allows for rigorous evaluation (F&A)allows for rigorous evaluation (F&A)
8/7/2019 Impact Eval and Project Cycle
18/37
Narrow down the questions for theNarrow down the questions for the
evaluationevaluation
Questions aimed at measuring the impact ofQuestions aimed at measuring the impact of
the project on a set of outcomes, andthe project on a set of outcomes, and
Questions aimed at measuring the relativeQuestions aimed at measuring the relative
effectiveness of different features of theeffectiveness of different features of the
projectproject
8/7/2019 Impact Eval and Project Cycle
19/37
What is your hypothesis? (Results framework)What is your hypothesis? (Results framework) By expanding water supply, the use of clean water will increase,By expanding water supply, the use of clean water will increase,
water borne disease decline, and health status improvewater borne disease decline, and health status improve
What is the evaluation question?What is the evaluation question? Does improved water supply result in better health outcomes?Does improved water supply result in better health outcomes?
How can do you test the hypothesis?How can do you test the hypothesis? The government might randomly assign areas for expansion in waterThe government might randomly assign areas for expansion in water
supply during the first and second phase of the programsupply during the first and second phase of the program
What will you measure?What will you measure?
Measure the change in health outcomes in phase I areas relative toMeasure the change in health outcomes in phase I areas relative tothe change in outcomes in phase II areas. Outcomes will include usethe change in outcomes in phase II areas. Outcomes will include useof safe water (Sof safe water (S--T), incidence of diarrhea (S/MT), incidence of diarrhea (S/M--T), and health statusT), and health status(L(L--T, depending on when phase II occurs). Add other outcomes.T, depending on when phase II occurs). Add other outcomes.
What will you do with the results?What will you do with the results? If the hypothesis proves true go to phase II; if false, modify policy.If the hypothesis proves true go to phase II; if false, modify policy.
Questions aimed at measuring the impact ofQuestions aimed at measuring the impact of
the projectthe project are relatively straightforwardare relatively straightforward
8/7/2019 Impact Eval and Project Cycle
20/37
Questions aimed at measuring the relativeQuestions aimed at measuring the relative
effectiveness of different project featureseffectiveness of different project features
require identifying the tough design choices on therequire identifying the tough design choices on thetabletable
What is the issue?What is the issue?
What is the best package of products or services?What is the best package of products or services? Where do you start from?Where do you start from?
What package is the government delivering now?What package is the government delivering now?
Which changes do you or the government think couldWhich changes do you or the government think couldbe made to improve effectiveness?be made to improve effectiveness?
8/7/2019 Impact Eval and Project Cycle
21/37
How do you test it?How do you test it? The government might agree to provide a package to aThe government might agree to provide a package to a
randomly selected group of households and anotherrandomly selected group of households and anotherpackage to another group of households to see how thepackage to another group of households to see how thetwo package performtwo package perform
What will you measure?What will you measure?
The average change in relevant outcomes forThe average change in relevant outcomes forhouseholds receiving one package versus the same forhouseholds receiving one package versus the same forhouseholds receiving the other packagehouseholds receiving the other package
What will you do with the results?What will you do with the results? The package that is most effective in deliveringThe package that is most effective in delivering
desirable outcomes becomes the one adopted by thedesirable outcomes becomes the one adopted by theproject from the evaluation onwardsproject from the evaluation onwards
8/7/2019 Impact Eval and Project Cycle
22/37
Application, features that should beApplication, features that should be
tested early ontested early on
Early testing of project features (say 6 months toEarly testing of project features (say 6 months to1 year) can provide the team with the1 year) can provide the team with theinformation needed to adjust the project early oninformation needed to adjust the project early on
in the direction most likely to deliver success.in the direction most likely to deliver success. Features might include:Features might include:
alternative modes of delivery (e.g. schoolalternative modes of delivery (e.g. school--based vs.based vs.vertical delivery),vertical delivery),
alternative packages of outputs (e.g. awarenessalternative packages of outputs (e.g. awarenesscampaigns vs. legal services), orcampaigns vs. legal services), or
different pricing schemes (e.g. alternative subsidydifferent pricing schemes (e.g. alternative subsidylevels).levels).
8/7/2019 Impact Eval and Project Cycle
23/37
8/7/2019 Impact Eval and Project Cycle
24/37
Develop identification strategyDevelop identification strategy((to identify the impact of the project separately from changes due toto identify the impact of the project separately from changes due to
other causesother causes ))
One the questions are defined, the leadOne the questions are defined, the leadresearcher selects one or more comparisonresearcher selects one or more comparisongroups against which to measure results in thegroups against which to measure results in the
treatment group.treatment group. The rigor with which the comparison group isThe rigor with which the comparison group is
selected will determine the reliability of theselected will determine the reliability of theimpact estimates.impact estimates.
Rigor?Rigor? MoreMore--same observables and unobservablessame observables and unobservables
(experimental),(experimental),
LessLess--same observables (nonsame observables (non--experimental)experimental)
8/7/2019 Impact Eval and Project Cycle
25/37
Explore Existing DataExplore Existing Data
Explore what data exists that might be relevantExplore what data exists that might be relevantfor use in the evaluation.for use in the evaluation. Discuss with the agencies of the national statisticalDiscuss with the agencies of the national statistical
system and universities to identify existing datasystem and universities to identify existing data
sources and future data collection plans.sources and future data collection plans. Record data periodicity, quality, variablesRecord data periodicity, quality, variables
covered and sampling frame and sample size,covered and sampling frame and sample size,forfor CensusesCensuses
Surveys (household, firms, facility, etc)Surveys (household, firms, facility, etc)
Administrative dataAdministrative data
Data from the project monitoring systemData from the project monitoring system
8/7/2019 Impact Eval and Project Cycle
26/37
New DataNew Data Start identifying additional data collection needsStart identifying additional data collection needs..
Data for impact evaluation must be representative of treatment andData for impact evaluation must be representative of treatment andcomparison groupcomparison group
Questionnaires must include outcomes of interest (consumption,Questionnaires must include outcomes of interest (consumption,income, assets etc), questions about the program in question andincome, assets etc), questions about the program in question andquestions about other programsquestions about other programs
The data might be at household, community, firm, facility, or farmThe data might be at household, community, firm, facility, or farmlevels and might be combined with specialty data such as thoselevels and might be combined with specialty data such as thosefrom water or land quality tests.from water or land quality tests.
Investigate synergies with other projects to combine dataInvestigate synergies with other projects to combine datacollection efforts and/or explore existing data collectioncollection efforts and/or explore existing data collectionefforts on which the new data collection could piggy backefforts on which the new data collection could piggy back
Develop a data strategy for the impact evaluation including:Develop a data strategy for the impact evaluation including: The timing for data collectionThe timing for data collection
The variables neededThe variables needed
The sampleThe sample
Plans to integrate data from other sourcesPlans to integrate data from other sources
8/7/2019 Impact Eval and Project Cycle
27/37
Prepare for collecting dataPrepare for collecting data
Identify data collection agencyIdentify data collection agency Lead researcher will work with the dataLead researcher will work with the data
collection agency to design sample, and traincollection agency to design sample, and trainenumeratorsenumerators
Lead researcher will prepare surveyLead researcher will prepare surveyquestionnaire or questionnaire module asquestionnaire or questionnaire module asneededneeded
PrePre--testing survey instrument may take place attesting survey instrument may take place atthis stage to finalize instrumentsthis stage to finalize instruments
If financed with outside funds, baseline can nowIf financed with outside funds, baseline can nowgo to the field. If financed by project funds,go to the field. If financed by project funds,baseline will go to the field just afterbaseline will go to the field just aftereffectiveness but before implementation startseffectiveness but before implementation starts
8/7/2019 Impact Eval and Project Cycle
28/37
Develop a Financial PlanDevelop a Financial Plan
Costs:Costs: Lead researcher and researcher team,Lead researcher and researcher team,
Data collection,Data collection,
Supervision andSupervision and
DisseminationDissemination
Finances:Finances: BB,BB,
Trust fund,Trust fund,
Research grants,Research grants, Project funds, orProject funds, or
Other donor fundsOther donor funds
8/7/2019 Impact Eval and Project Cycle
29/37
HousekeepingHousekeeping Initiate an IE activity. The IE code in SAP is a way ofInitiate an IE activity. The IE code in SAP is a way of
formalizing evaluation activities. The IE code recognizesformalizing evaluation activities. The IE code recognizesthe evaluation as a separate AAA product.the evaluation as a separate AAA product. Prepare concept notePrepare concept note
Identify peer reviewersIdentify peer reviewers impact evaluation and sector specialistimpact evaluation and sector specialist
Carry out review processCarry out review process
Appraisal documentsAppraisal documents Include in the project description plans to modify project overtimeInclude in the project description plans to modify project overtime
to incorporate resultsto incorporate results
Work the impact evaluation into the M&E section of the PAD andWork the impact evaluation into the M&E section of the PAD and
Annex 3Annex 3
Include the impact evaluation in the QualityInclude the impact evaluation in the QualityEnhancement Review (TTL).Enhancement Review (TTL).
8/7/2019 Impact Eval and Project Cycle
30/37
Negotiations to CompletionNegotiations to Completion
8/7/2019 Impact Eval and Project Cycle
31/37
Ensure timely implementationEnsure timely implementation
Ensure timely procurement of evaluationEnsure timely procurement of evaluation
services especially contracting the dataservices especially contracting the data
collection, andcollection, and
Supervise timely implementation of theSupervise timely implementation of the
evaluation includingevaluation including
Data collectionData collection
Data analysisData analysis
Dissemination and feedbackDissemination and feedback
8/7/2019 Impact Eval and Project Cycle
32/37
Data collection agency/firmData collection agency/firm
Data collection agency or firm must haveData collection agency or firm must have
technical knowledge and sufficienttechnical knowledge and sufficient
logistical capacity relative to the scale oflogistical capacity relative to the scale of
data collection requireddata collection required
The same agency or firm should beThe same agency or firm should be
expected to do baseline and follow up dataexpected to do baseline and follow up data
collectioncollection
8/7/2019 Impact Eval and Project Cycle
33/37
Baseline data collection and analysisBaseline data collection and analysis
Baseline data collection should be carriedBaseline data collection should be carriedout before program implementationout before program implementationbegins; optimally even before program isbegins; optimally even before program isannouncedannounced
Analysis of baseline data will provideAnalysis of baseline data will provideprogram management with additionalprogram management with additionalinformation that might help finalizeinformation that might help finalizeprogram designprogram design
8/7/2019 Impact Eval and Project Cycle
34/37
FollowFollow--up data collection andup data collection and
analysisanalysis
The timing of followThe timing of follow--up data collectionup data collectionmust reflect the learning strategy adoptedmust reflect the learning strategy adopted
Early data collection will help modifyingEarly data collection will help modifying
programs mid course to maximize longerprograms mid course to maximize longer--term effectivenessterm effectiveness
Later data collection will confirmLater data collection will confirm
achievement of longerachievement of longer--term outcomes andterm outcomes andjustify continued flows of fiscal resourcesjustify continued flows of fiscal resourcesinto the programinto the program
8/7/2019 Impact Eval and Project Cycle
35/37
DisseminationDissemination
Implement plan for dissemination of evaluation resultsImplement plan for dissemination of evaluation resultsensuring that the timing is aligned with governmentsensuring that the timing is aligned with governments
decision making cycle.decision making cycle.
Ensure that results are used to inform projectEnsure that results are used to inform project
management and that available entry points aremanagement and that available entry points areexploited to provide additional feedback to theexploited to provide additional feedback to the
governmentgovernment
Ensure that wider dissemination takes place only afterEnsure that wider dissemination takes place only after
the client has had a chance to preview and discuss thethe client has had a chance to preview and discuss the
resultsresults
Nurture collaboration with government and localNurture collaboration with government and local
researchers throughout the processresearchers throughout the process
8/7/2019 Impact Eval and Project Cycle
36/37
HousekeepingHousekeeping
Involve local project implementation unitInvolve local project implementation unit
and PIU person responsible for monitoringand PIU person responsible for monitoring
and evaluationand evaluation
Put in place arrangements to procure thePut in place arrangements to procure the
impact evaluation work and fund it on timeimpact evaluation work and fund it on time
Use early results to inform midUse early results to inform mid--termterm
reviewreview
Use later results to inform the ICR, CASUse later results to inform the ICR, CAS
and future operationsand future operations
8/7/2019 Impact Eval and Project Cycle
37/37
Concluding remarksConcluding remarks
Making evaluation work for you requires aMaking evaluation work for you requires achange in the culture of project design andchange in the culture of project design andimplementation, one that maximizes the use ofimplementation, one that maximizes the use of
learning to change course when necessary andlearning to change course when necessary andimprove the chances for successimprove the chances for success
Impact evaluation more than a tool is anImpact evaluation more than a tool is anorganizing analytical framework for doing thisorganizing analytical framework for doing this
Thank youThank you