1
Impact Evaluation in the Global Environment Facility:
Looking back for Our Common Future
Cairo, April 2009
David M. [email protected]
Lee Alexander [email protected]
2
Global Environment Facility (GEF): The 10 second Overview
• Established in1991• Funds projects undertaken by developing and
transition countries to achieve environmental benefits
• Funding areas: Climate change; biodiversity; international waters; ozone; persistent organic pollutants and land degradation
• Over 10US$ billion in project financing since 1991• GEF Structure: Council (Governance & Strategy);
Secretariat (Project approvals & Monitoring); Evaluation Office (Independent Evaluation)
• Projects implemented by MDBs and UN agencies
3
Impact Evaluation (IE) in GEF: Overview• IE launched in 2005: Demand from GEF Council
(donors and recipients) • Set against background of complexity of the
environment and complexity of projects• Designs employed so far:
– Straightforward before/after analysis with comparison between projects (Ozone)
– With/without + before/after analysis through quasi-experimental design (Biodiversity)
– Theory-of-Change (TOC) (Biodiversity and Ozone)– Combination of before / after and TOC (Ozone)
• “mixed design / methods” approach where possible should be pursued:
5
Ozone Depleting Substances Impact Evaluation in Countries with
Economies in Transition
Building and Implementing Mixed-Design and Method Evaluation
6
Ozone Depleting Substances IE in Countries with Economies in Transition (CEITs):
• GEF began ODS activities in CEITs in 1992 to phase-out CFCs and other chemicals that contribute to Ozone damage (The “Ozone Hole”)
– Supports Montreal Protocol compliance– 26 projects (18 countries) and approximately US$380m of financing.
• Projects aimed at stopping consumption and production of Ozone Depleting Chemicals and technologies
• Key stakeholders: Governments, Implementing Agencies and private sector • Main issues for the evaluation:
– Impact of the GEF on production and consumption of ODS – Impact on legal/illegal trade ODS– Impact on Stockpiling of ODS – Sustainability and catalytic effects– Attribution / contribution issue
• Scope: Portfolio impact – why and how? Project and data synergy
7
ODS IE: Tailoring Design to Context • Data availability assessment:
– Quantitative data on production and consumption exists for CEITs early 1990s – to present.
– Significant quant-qual literature on ODS phase-out, many evaluations but no impact evaluations
– Academic and technical literature on ozone science and environmental / human health impacts
• Availability of scientific modeling of ozone hole recovery• And also on human and animal health impacts (skin cancers)
• Initial Evaluation Design assessment– ODS portfolio would provide an opportunities for mix design and
method approaches:• Quasi-experimental with a TOC – responding to what worked
and why …
8
ODS IE: Tailoring Design to Context
• But, desk review found no viable control / counter-factual countries exist– All other countries developed and developing have taken actions
to stop ODS consumption / production• Only exceptions in 2008: Vatican, San Marino and Timor
• Adjusted design to reflect observed data and sample limitations:– Before & after measures of changes:
• By chemical • Time • Internal comparison between GEF Bank and UN countries• External matched comparison between GEF and Multi-lateral Fund
countries• + Business as usual forecast: what would have happened without
the MP and the GEF project?– TOC: development of TOC based on review of project documents and
available terminal evaluations
9
Output: National commitment to eliminate use (production & consumption)
Output: Coordinated Regional Policies
Output: Technology producers and distributers supported to convert
Output: Users supported to adopted sustainable technology
Outcome: Phased-out supply and demand for ODS technology
Assumption Availability of new technologies
Intermediate: ODS substances phased-out
Intermediate: ODS technology phased-out
Impact: ODS tons eliminated
Impact Evaluation ODS Theory of Change
Assumption: Enterprises are reached (large, medium and small)
Impact: Ozone layer improved
Assumption: New Technologies are affordable
Strategy: To protect human health and the environment by assisting countries to phase-out ODS consumption and production, while enabling alternative technologies and practices
Assumption: Enterprises investing are sustainable
Assumption: Effective monitoring and reporting on ODS
Assumption: Recovery and recycling schemes or stockpiling meet temporary demand for ODS
Threat Reduction: Reversion to ODS technologies
Threat Reduction: Illegal trade in ODS
10
Methodology• TOC provides input into ODS IE evaluation matrix: sub-questions /
data sources / methods and indictors• Pre-fieldwork meta-evaluation of existing terminal evaluations
– Including means-ends linkage assessment (e.g., economic and trade instruments – and ends – market transformation)
• Statistical methods:– Descriptive and Regression-based analyses
• Taking into account differences in time, population, GDP etc• Fieldwork (qualitative method) used to confirm and / or dis-confirm
TOC, with particular focus on ‘assumptions / risks and threats’– 4 detailed country case studies– 8 ‘lite’ country case studies: Drawing on parallel UNDP – UNEP terminal
evaluation of GEF-ODS projects – Semi-structured interviews with Government Ozone Units (20%) Private
Sector (60%), Customs and Excise (20%)– Questionnaire survey to validate key issues arising from qualitative data
collection– Triangulation between and within countries
11
Forthcoming findings
• Draft report released mid-May– International ODS impact evaluation workshop in
Tashkent: June 2009
• Now over to David to finish with a delicious ROTI
Review of Outcomes to Impacts (ROtI)
Impact Evaluation Framework
Impact
ReducedThreats to
GEB
Enhancedstatus of
GEBOutcomeOutputs State/
conditionState/
condition
Assumption Assumption Assumption
Threats Based AnalysisOutcomes-Impacts TOC AnalysisProject Logframe Analysis
Assess direct effects of the project
Assess how these effects are leading to impacts
Assess whether impacts have actually occurred
+ ++ +++
Project stakeholders create their own theory of change
How the ROtI addresses a key methodological dilemma in impact evaluation:
representation versus reliability
Reliability:Data Quality
Impact Evaluations($100K)
1
Field-based ROtIs
($10-20K) 5-10
Desk–based Rotis based on Terminal
Evaluations($400)
250
Represent-ation:
Number of projects
per $100,000
Steps in field ROtI
Results from Field Review
• Pilot in Seychelles showed that ROtI based on field review can produce detailed qualitative assessment (with some quantitative support) of achievement of outcomes and of progress from this stage towards achievement of impacts.
• Key questions could all be assessed : did assumptions hold, were intended impact drivers active and were intermediate stages achieved.
The role of the ROtI
• The desk ROtI enables a rapid scaling up of proxy information on impacts
• The field ROtI provides moderate scaling up possibilities, with higher data quality
• The Impact Evaluation provides detailed and reliable data on a few projects
• The three levels together can provide a much richer knowledge base on impacts than is currently available
19
Thank you