Date post: | 26-Dec-2015 |
Category: |
Documents |
Upload: | jasmine-erika-lawson |
View: | 217 times |
Download: | 0 times |
Impact Evaluation Impact Evaluation for for International International Development:Development:
Why we need it and why it is Why we need it and why it is hardhardIrene Irene GuijtGuijt
March 25, 2013CDI Conference on Impact Evaluation
Copy right statementCopy right statement
On the ideas asserted in this power point On the ideas asserted in this power point Intellectual Property Rights apply.Intellectual Property Rights apply.
Appropriate citation:Appropriate citation: Guijt, I. (2013).Guijt, I. (2013). Impact Evaluation Impact Evaluation for for
International Development: Why we need it and International Development: Why we need it and why it is hardwhy it is hard. Keynote speech for CDI conference . Keynote speech for CDI conference on Impact Evaluation, March 25 and 26, 2013. on Impact Evaluation, March 25 and 26, 2013.
Key messagesKey messages
Crucial and growing area of evaluation expertiseCrucial and growing area of evaluation expertise Much contestation – definitions, methods, Much contestation – definitions, methods,
standards, utility standards, utility Utility of IEs remain to be provenUtility of IEs remain to be proven
Impact EvaluationImpact Evaluation
1.1. What is it?What is it?
2.2. Why do it? When is it appropriate?Why do it? When is it appropriate?
3.3. Who’s involved?Who’s involved?
4.4. How?How?• Options for key tasksOptions for key tasks• QualityQuality• UseUse
Controversial? All of the above!Controversial? All of the above!
small ‘e’, big ‘E’small ‘e’, big ‘E’
‘‘Results and targets’ Results and targets’
vs vs
‘‘(Why) do or don’t things work?’(Why) do or don’t things work?’
Seeds of interestSeeds of interest• Aid ‘failure’Aid ‘failure’• Financial crisisFinancial crisis• Shifting relationships (economic & players)Shifting relationships (economic & players)• Others?Others?
When to investWhen to invest
Testing innovation for scaling upTesting innovation for scaling up Risky context, investment utilityRisky context, investment utility Accountability for high investmentsAccountability for high investments Downward accountability (?)Downward accountability (?)
Policy (and practice) direction for the future:Policy (and practice) direction for the future:• Will the past be a good predictor of the future?Will the past be a good predictor of the future?• When what works, doesn’t work?When what works, doesn’t work?• And when what doesn’t work, works?And when what doesn’t work, works?
Learning vs accountabilityLearning vs accountability
Questions (that) matterQuestions (that) matter
Which questions matter?Which questions matter?• Whose questions are heardWhose questions are heard• Which theories or strategies do we need to questionWhich theories or strategies do we need to question• Who casts a voteWho casts a vote
3ie 3ie enduring questions, eg are school vouchers enduring questions, eg are school vouchers the solution, what works in reducing FGM, market the solution, what works in reducing FGM, market access for the poor, etcaccess for the poor, etc
MethodsLab MethodsLab rolling out an innovation within a rolling out an innovation within a flagship sector (25% of budget)flagship sector (25% of budget)
Defining ImpactDefining Impact
the positive and negative, intended and the positive and negative, intended and unintended, direct and indirect, primary and unintended, direct and indirect, primary and secondary effects produced by an intervention secondary effects produced by an intervention (OECD)(OECD)
I = Y2 – Y1 = (I) evaluation as a rigorous estimation of the difference between indicators of interest with (Y1) and without the intervention (Y2) (Howard White, 3ie)
People and MoneyPeople and Money
““An IE industry has developed which believes it is doing good workAn IE industry has developed which believes it is doing good work .”.”(Morton et al 2012)(Morton et al 2012)
Demand for IEDemand for IE• Development banksDevelopment banks• Bilateral aidBilateral aid• Multi-donor programsMulti-donor programs• NGOsNGOs
Supply for IE expertiseSupply for IE expertise• JPAL network (2003 - Abdul Latif Jameel Poverty Action Lab )JPAL network (2003 - Abdul Latif Jameel Poverty Action Lab )
““Conducting Rigorous Impact EvaluationsConducting Rigorous Impact Evaluations: J-PAL researchers conduct : J-PAL researchers conduct randomized evaluations to test and improve the effectiveness of programs randomized evaluations to test and improve the effectiveness of programs
and policies aimed at reducing poverty.”and policies aimed at reducing poverty.”• 3ie (2009)3ie (2009)• university departments, esp. economistsuniversity departments, esp. economists• CLEAR Network (World Bank funded)CLEAR Network (World Bank funded)• consultantsconsultants
The case of 3ieThe case of 3ie
20042004 Origin with Closing the Evaluation Gap Origin with Closing the Evaluation Gap Initiative (Centre for Global Development), Initiative (Centre for Global Development), Hewlett Foundation, BMGFHewlett Foundation, BMGF
20062006 paper ‘When Will We Ever Learn’, the paper ‘When Will We Ever Learn’, the manifesto of the Impact Evaluation movement manifesto of the Impact Evaluation movement
2008/092008/09 3ie established, $55.14 million (2010-3ie established, $55.14 million (2010-2013) Supply and demand 2013) Supply and demand • Quality standards for rigorous evaluationsQuality standards for rigorous evaluations• Review process for designs/studiesReview process for designs/studies• Identifying priority topicsIdentifying priority topics• Provide grants for IE designsProvide grants for IE designs
3IE Review Conclusions 3IE Review Conclusions (Morton et al (Morton et al 2012)2012)
Membership disappointingly lowMembership disappointingly low• 10 bilateral and one multilateral donor agencies; 6developing country 10 bilateral and one multilateral donor agencies; 6developing country
governments/agencies; 2 philanthropic foundations; 3INGOsgovernments/agencies; 2 philanthropic foundations; 3INGOs
Policymakers, implementers and donors: IE low Policymakers, implementers and donors: IE low priority; sizeable minority of practitioners not of priority; sizeable minority of practitioners not of experimental IE experimental IE
What is considered ‘good’ is funded, what is funded What is considered ‘good’ is funded, what is funded shapes the face of IEshapes the face of IE• 139 specialists, wide review net, esp US universities focused on 139 specialists, wide review net, esp US universities focused on
experimental IEexperimental IE
Journal of Development Effectiveness since 2009, Journal of Development Effectiveness since 2009, respected, used (4000 times in 2010) respected, used (4000 times in 2010)
To date 9 SRs and 9 IEs finalised but website includes To date 9 SRs and 9 IEs finalised but website includes many more studiesmany more studies
Others emerging (partial listing!)Others emerging (partial listing!)
INGOsINGOs• ICCO, World Vision, Oxfam, Plan International, CARE, Hivos, Save ICCO, World Vision, Oxfam, Plan International, CARE, Hivos, Save
the Children, Veco, Freedom from Hungerthe Children, Veco, Freedom from Hunger
MultilateralsMultilaterals• RCT studies undertaken for many UN agenciesRCT studies undertaken for many UN agencies• IFAD developing own approach (PIALA)IFAD developing own approach (PIALA)
BilateralsBilaterals• GiZ, AfD, AusAid, DfID, USAID, DGIS….GiZ, AfD, AusAid, DfID, USAID, DGIS….
Philanthropic FoundationsPhilanthropic Foundations• BMGF, Hewlet, Packard, Rockefeller….BMGF, Hewlet, Packard, Rockefeller….
It has to be ‘rigorous’It has to be ‘rigorous’
Rigour = value judgementRigour = value judgement• USAID: ‘USAID: ‘attributable to a defined intervention; im- pact
evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual’
• PIALA: data, sensemaking, utility, low resource effortsPIALA: data, sensemaking, utility, low resource efforts
Five standards in evaluation – which one wins? Five standards in evaluation – which one wins? • Utility or accuracyUtility or accuracy• Relevance or objectivityRelevance or objectivity
Hard in practiceHard in practice• Eg 3IE’s own standards, the 2012 review was very critical of IEs Eg 3IE’s own standards, the 2012 review was very critical of IEs
funded through 3IEfunded through 3IE
How does one do IE?How does one do IE?
Gold standard of whatGold standard of what• Statistics or rigour of thoughtStatistics or rigour of thought
Experimentalism vs ‘other’Experimentalism vs ‘other’• A method or naïve experimentalist beliefA method or naïve experimentalist belief
Attribution Attribution • Attribution as one form of contributionAttribution as one form of contribution
Counterfactual Counterfactual • rigour or philosophically dead?rigour or philosophically dead?
Emerging alternativesEmerging alternatives
Realist evaluation framingRealist evaluation framing Contribution analysisContribution analysis People’s narratives and surveys for attitudes / People’s narratives and surveys for attitudes /
behaviour shiftsbehaviour shifts Qualitative Comparative AnalysisQualitative Comparative Analysis Participatory impact assessment and learning Participatory impact assessment and learning
approachapproach PADEV …PADEV …
Participatory impact Participatory impact assessment and learning assessment and learning
approach (PIALA)approach (PIALA)
IFAD committed to 30 IEsIFAD committed to 30 IEs Needs an approach that works for low Needs an approach that works for low
resource programs, must show progress resource programs, must show progress towards 80 million, weak baselinestowards 80 million, weak baselines
PIALA = statistically sampled in nested PIALA = statistically sampled in nested hierarchy, PRA and survey data collection plus hierarchy, PRA and survey data collection plus project data, collective sensemakingproject data, collective sensemaking
GirlHub (DfID and Nike GirlHub (DfID and Nike Foundation)Foundation)
Impact of ‘starting a national conversation to Impact of ‘starting a national conversation to revalue girls’revalue girls’
Focus – attitude and behaviour change as a result Focus – attitude and behaviour change as a result of social media workof social media work
Control group hard – radio, magazineControl group hard – radio, magazine Purposively, stratified sample: detailed survey Purposively, stratified sample: detailed survey
and self-signified stories of change, before and and self-signified stories of change, before and after, supplementary context analysisafter, supplementary context analysis
Beyond only Beyond only data data collection: IE collection: IE is about more is about more than just than just describing describing ‘impact’ well‘impact’ well
betterevaluation.orgbetterevaluation.org
UseUse
Uptake? Uptake? • Evidence of evidence use is weakEvidence of evidence use is weak
Assumptions about uptakeAssumptions about uptake• Naïve politics Naïve politics • Many, statistically rigorous, published IEs will lead to useMany, statistically rigorous, published IEs will lead to use• IDRC 90+ variables that influence uptakeIDRC 90+ variables that influence uptake
Have to deal with the boundary between research Have to deal with the boundary between research and policy uptakeand policy uptake
Uptake requires fostering interest = we need to Uptake requires fostering interest = we need to understand the psychology of use, as well as the understand the psychology of use, as well as the politics of usepolitics of use
Assumptions in 3IEs Theory of Assumptions in 3IEs Theory of ChangeChange
overriding need is for more rigorous evidenceoverriding need is for more rigorous evidence quality will automatically be a result of more IEs quality will automatically be a result of more IEs
of a certain typeof a certain type research community will generate enough policy research community will generate enough policy
relevant proposals and it will only be necessary to relevant proposals and it will only be necessary to select the most policy relevantselect the most policy relevant
policy influence requires major effort from onset policy influence requires major effort from onset of researchof research
Future of IEFuture of IE
AudienceAudience• Convince policymakers that rigorous IE important for policy Convince policymakers that rigorous IE important for policy
processprocess
Feasibility and rigourFeasibility and rigour• Fit to context = (quasi) experimental studies difficult (conditions, Fit to context = (quasi) experimental studies difficult (conditions,
questions, capacities)questions, capacities)
Relevance for policyRelevance for policy• Inverse relationship? most policy relevant programmes least Inverse relationship? most policy relevant programmes least
amenable to experimental IE techniquesamenable to experimental IE techniques