Using Impact Evaluation to Improve Development Projects
Arianna Legovini World Bank
USG Evaluation Forum
December 2, 2015
What does DIME do?
2
Transform development policy
1. Run experiments to inform decisions
2. Build agencies’ capacity to do this systematically
3. Draw lessons and share them face-to-face to global audiences
ideas
1. Improve evidence-base at design stage
2. Experiment with causal mechanisms that can change results during implementation
3
We evaluate the impact of packages of interventions (“what”) and experiment with mechanisms (“why/how”)
to make them work better:
IE evaluation questions in IEs with DIME Involvement
(at a minimum) IE increases project disbursements & closes implementation gap
statistically significant at 1% level
5
40.8%***
-54.2%***
40.0%***
15.6%
-45.3%**
49.1%***
Rate of Disbursement(Up to Mid-Term)
Gap (Up to Mid-Term)
Rate of Disbursement- Adjusted by SectorAverage (Up to Mid-
Term)
Rate of Disbursement(overall)
Gap (overall) Rate of Disbursement- Adjusted by Sector
Average (overall)
What is i2i impact evaluation to development impact?
A global platform to
generate and share IE knowledge and
transform the path of development policy
(especially in under-researched areas)
6
What makes i2i different from other IE initiatives?
1. Bottom-up approach – Empower government agencies to solve their own problems/find their own solution
– Open platform for expert advice and support
– Equalize opportunities in selection for technical and financial support
2. High research quality and policy relevance – methods for causal analysis
– data quality
– attention to implementation
– analysis and peer review
– collaboration and policy dialogue to ensure policy actions
3. Generate evidence throughout policy lifecycle – policy design
– mid-course adjustments
– policy decisions on scale up/down.
7
We value equally the generation of knowledge and its application to policy decisions
i2i pillars and thematic areas
8
Shared Prosperity
• Finance, Private
Sector and Jobs • Agriculture • Infrastructure
Governance
• Public Sector Governance
• Justice • Local Development
Climate Change
• Energy • Environment • Natural Resource
Management • Agriculture • Transport
Fragility & Conflict
• Reintegration • Governance • Gender-based
Violence • Urban crime &
violence
Gender
• Human Capital • Economic
Opportunities • Voice/Agency
5
1 2 3
6
Cross Cutting Themes
Human Development
• Education • Health • Social Protection
4
What type of technical support does i2i provide?
• Build capacity through workshops and learning by doing
• Match researchers and operations
• Full implementation support
• In-house research
• Coordination across thematic areas in close collaboration with operational departments
– Knowledge priorities
– Operationalize results
DIME i2i Workshops
No. teams
Impact Evaluation 4 Peace Mar 2014 18
Innovations for Agriculture Apr 2014 8
Impact Evaluation for Legal Empowerment May 2014 11
Innovations for Agriculture June 2014 23
Energy & Environment Oct. 2014 18
Governance Jan. 2015 27
Trade & Competitiveness & Finance May 2015 22
Transport and ICT June 2015 20
Number of participating teams per region
DIME i2i workshops: 128 teams in FY15-16
Putting in perspective: The Bank approves around 400
projects a year
AFR 41%
LAC 31%
SAR 12%
ECA 6%
MENA 5%
EAP 5%
Workshops increase participants’ knowledge
61
65
69
60
72
73
79
70
AGRICULTURE ENERGY & ENVIRONMENT T&C AND F&M TRANSPORT & ICT
PRE/POST TEST
Pre post
56%
79%
89%
63%
77% 80%
FCS Agriculture Energy & Environment Governance T&C and F&M Transport & ICT
workshop projects that submitted an EoI
note: projects with ongoing IE are excluded
Impact of Workshop on (External Double Blind) Technical Ratings of IE proposals
14
2.25
2.04 2.18
2.55
2.14 2.04 2.01
1.76
2.08 2.13
1.85 1.91
FCS Agriculture Energy & Environment Governance T&C and F&M Transport & ICT
Technical ratings of proposals (0-3 scale)
Attended workshop Did not attend workshop
Applications received across all windows
In total 229 applications received:
• Ongoing IEs: 49
• New IEs: 180
FCS 33
Agriculture 26
Energy & Environment 27
Governance 28
T&C and F&M 35
Transport & ICT 31
A THEORY OF CHANGE IMPROVING DEVELOPMENT PROJECTS
16
Clear roadmap
17
Clearer roadmap
What
To whom When
Where
Better capacity
18
Better capacity
Research team
Full time field coordinator
Training
More and better data
dime results indicators
88%
74%
75%
36%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Client participated in IE workshop(s)
Baseline results discussed with client
IE results discussed with client
Training provided for data analysis
Capacity Building
19
dime results indicators
84%
85%
48%
50%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
High-quality baseline survey
High-quality follow-up survey(s)
Improved administrative data
Availability of Data as a Public Good
Quality of Data
20
More effort into implementation
21
Motivation to perform
Goal: this is what we are
trying to achieve
Control: this is how we want
to do it
…and BTW somebody
measures (and cares)
dime results indicators
36%
23%
30%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Baseline informed policydesign/implementation
Adopted delivery or behavioral mechanismbased on IE results
IE results were used to motivate scale-up/scale-down of policy at national level
Quality of Policy Decisions
23
In short…
• Put development in the hands of the implementing agencies
• Challenge them to improve project design and experiment with new ideas
• Provide them with the technical resources to generate great data and evidence
• Help them make smart decisions
Thank You! facebook.com/ieKnow
#impacteval
blogs.worldbank.org/impactevaluations
microdata.worldbank.org/index.php/catalog/impact_evaluation
http://dime.worldbank.org WEB