EVALUATION FOR ACCOUNTABILITY: Measuring Development Effectiveness of NGOs
Keynote Address ECSP Conference ‘Challenges for Philanthropy: Scale, Governance & Impact’
Rotterdam, 22 September 2011
Prof. dr. Ruerd RubenDirector Policy & Operations Evaluation (IOB)Ministry of Foreign Affairs, The Netherlands
Policy & Operations Evaluation Department (IOB)
Policy & Operations Evaluation (1)
Policy & Operations Evaluation Department (IOB)
Mandate:
• Assessing policy effectiveness
• Promoting learning from experiences
• Improving evidence-based policy-making
Accountability to Parliament, Public and Partners
(autonomous programming & independent reporting)
Policy & Operations Evaluation (2)
Policy & Operations Evaluation Department (IOB)
Activities
• Impact studies on main development priorities
• Analysis of the coherence in foreign policy
• Thematic studies on key success factors
Creating Transparency & Trust
Key Messages
Policy & Operations Evaluation Department (IOB)
1. Development effectiveness has to show ‘value for money’
2. Showing effectiveness may enhance public trust (necessary, but not sufficient condition)
3. Transparency starts with autonomy & objectivity
4. Evaluability is key responsibility of donors and recipients
5. No learning without insights in quantitative impact
Development Effectiveness
Policy & Operations Evaluation Department (IOB)
• Delivery of meaningful results
• Net impact over time
• Value for Money
• Attribution/Contribution
• Additionality
Evaluating Civil Society
Policy & Operations Evaluation Department (IOB)
Large share of Dutch aid:
5% 20 % of ODA
(+ 10% ODA through
Delegated budgets)
Three components: Direct Poverty Reduction (MDGs)
Capacity Development (5C)Lobby & Advocacy
0
5
10
15
20
25
From Input to Impact
Outreach = # of participants
Throughput = activities within the system
Policy & Operations Evaluation Department (IOB)
Common Pitfalls
Policy & Operations Evaluation Department (IOB)
1. Strong selection bias
2. Spatial overlaps: externalities & free riding
3. Absence of baselines data
4. Scarce controls
5. Long result chains
6. Limited additionality
Credible evidence
Policy & Operations Evaluation Department (IOB)
Impact measurement: comparing:
• before/after (time)
• with/without (counterfactual)
Everything else is additional
(e.g. story harvesting, outcome mapping, SROI, etc)
(See: NONIE Guidelines ; 3ie)
Time
Goal
without
with
T=0 T=1
Difference in Difference (DiD)
Policy & Operations Evaluation Department (IOB)
Intervention group
Control group Difference across groups
Baseline I0 Co I0 – C0
Follow-up I1 C1 I1 – C1
Difference across time
I1 – I0 C1 – C0 (I1 – C1) – (I0 – C0) = (I1 – I0) – (C1 – C0)
Propensity Score Matching (PSM)
Policy & Operations Evaluation Department (IOB)0
.51
1.5
22.
5D
ensi
ty
0 .2 .4 .6 .8 1Propensity Score
ControlTreated
Before Matching
0.5
11.
52
Den
sity
0 .2 .4 .6 .8Propensity Score
ControlTreated
After Matching
Evaluation Criteria
Policy & Operations Evaluation Department (IOB)
1. Effectiveness
2. Efficiency
3. Relevance
4. Coherence
5. Sustainability
See: OECD-DAC Evaluation Guidelines
Methods & Approaches
Policy & Operations Evaluation Department (IOB)
Document review
Policy reconstruction
Robust Impact studies (DiD)
Experiments (Real-time evaluation)
Systematic Reviews (Campbell protocol)
Thematic studies (Key success factors)
Worrisome Evidence
Policy & Operations Evaluation Department (IOB)
‘NGO support is highly appreciated, but no evidence can be generated to demonstrate its impact’
‘Local capacities of NGO partners have grown substantially, but it remains unclear whether and how donor support has contributed to this’
‘Access to services and markets for the poor have improved, mainly due to better socio-economic performance of the country’
Declining trust
Policy & Operations Evaluation Department (IOB)
Reputation & Trust
Policy & Operations Evaluation Department (IOB)
“Trust me”
Tell me” “Show me”
“Show me”
“Trust me”
“Tell me”
Insight in development effectiveness is a necessary (but not sufficient) condition for maintaining public support
Policy & Operations Evaluation Department (IOB)
Evaluation & Accountability
Guaranteeing Evaluability
• Intervention theory
• SMART indicators
• Random sampling
• Valid baselines
• Comparison/control groups
Organizing Evaluations
Policy & Operations Evaluation Department (IOB)
Independency
Quality control systems
Peer review
Horizontal control
Certification (CBF, NEN)
Learning from Evaluations
Policy & Operations Evaluation Department (IOB)
Single, double & triple loop learning
Defining innovation spaces
Real-time experiments
Methodological Challenges
Policy & Operations Evaluation Department (IOB)
Overcoming poverty thresholds
Focus on Complementarities
Search for Policy coherence
Measuring Additionality
Institutional Strength (proxies)
Spillovers (catalyst effects)
Externalities
Institutional Challenges
Policy & Operations Evaluation Department (IOB)
Organizing Independent NGO Evaluation Units
Upfront Focus on Evaluability
Training of local (Southern) partners
Web-based open data systems (see: www.ngo-database.nl)
Openness on Failures & Successes
Comparing NGO performance (ranking)
Thanks for your Attention…
Policy & Operations Evaluation Department (IOB)