| Date post: | 31-Dec-2015 |
| Category: |
Documents |
| Upload: | charleen-richard |
| View: | 214 times |
| Download: | 0 times |
3D impact analysisA new tool to approach impact evaluations
April 23, 2015
CDI is a joint initiative between:
1
and and
For more information: www.ids.ac/cdi or email: [email protected]
Rob D. van den BergVisiting Fellow, IDS
2
Overview
• What is impact?• What is evidence?• What is causality?• What is attribution/contribution?• Time• Space• Scale
3
Impact
• Impact is an ordinary word in the English language– “the effective action of one thing or person on
another; the effect of such action; influence; impression”
• Its meaning cannot be scientifically claimed• Demand for impact evidence can refer to a
wide variety of effects, influences and impressions
4
Evidence
• Evidence is an ordinary word in the English language– “the quality or condition of being evident;
clearness; evidentness”• Its meaning cannot be scientifically claimed• Demand for impact evidence can refer to a
wide variety of qualities or conditions
5
Causality
• The word “cause” is an ordinary word in the English language– “A person or thing that gives rise to an action,
phenomenon, or condition”• Its meaning cannot be scientifically claimed• Demand for evidence of cause and effect can
refer to a wide variety of actions, phenomena and conditions
6
Attribution / Contribution
• Both words are ordinary words in the English language, with great variety in meaning– Attribution: in copyright law, requiring an author
to be credited; in marketing, assigning a value to a marketing activity based on desired outcome; journalism, practice of attributing information to its source
– Contribution: donation, sharing, payment, publication, a song by Mica Paris
7
Impact Evaluation
• Focus of Impact Evaluations:– Impact = evidence of causality between an
intervention and the desired effect by establishing a counterfactual through controlled experimentation, which attributes part of the effect to the intervention
• This partially meets the demand for impact evidence in politics, the media and society
• So what to do with other demands?
8
Meeting impact demand
• Broaden the concepts of impact and causality• Broaden the range of scientific methods and
tools• Develop a framework for understanding
demand for impact evidence• Incorporate issues of time, space and scale• This is urgent, given the adoption of the
Sustainable Development Goals in September 2015
9
Understanding causality
• Schaffer (2013) proposes two kinds of causality: “difference” and “production”– Difference: with/without (counterfactual) analysis – Production: A “produces” B (natural systems, physics &
technology)• Concepts that include causality:
– Catalytic roles (the change agent speeds up change but is not involved in the change itself)
– Dynamic and chaotic systems (feedback loops, iterative processes, Fibonacci sequences)
• Terry Pratchett: “hardly anything important has a single cause”
Issues of time, space and scale
• Some changes can be observed immediately – others take decades– Short-term results: vaccinations, technology transfer, new livelihood
approaches etc.– Either short- or long-term: market transformations, societal change– Long-term results: health trends, ecosystem services, ozone layer
• Some changes are local, other regional, national or even global• Some changes concern one actor, intervention or institution,
others involve multiple actors or institutions, and multiple sectors
• Sustainable development involves longer time horizons, overlapping locations and many scales
Matrix of evaluable impact
• Impact can be evaluated at different moments in time: ex ante, in real time and ex post– These can be refined: ex ante tends to be done once, but real time
and ex post have many possibilities– Different processes tend to have different time horizons
• Geographical space runs from local to global– These can also be refined: the boundaries of societies, economies
and natural systems are different from each other and may overlap• Scales of involvement can go from one actor to a multiplicity,
from one market to a full economic system, from one governance level (or sector) to many – Actors, markets and governance may not fully overlap
12
Appraisal
InceptionImplementation
End-of-project
2 years ex-post
5-8 years ex-post
13
Local Regional National Global
One
Multi-actor
Multi-sector
14
Matrix dimensions space and time
Ex ante Inception Mid-term End of project
Ex post < 2 years
Ex post 5-8 years
Local
National
Regional
Global
Ecosystem(overlap with other rows)
Ex ante evaluation and impact assessment
Experimentation
Mixed methods / theory of change
approachesMonitoring and data analysis (including “big data”)
15
Matrix dimensions space and scale
One inter-vention
Multiple inter-ventions
Enabling environ-ment
Market change
Market transform-ation
Climate change
Local
National
Regional
Global
Ecosystem(overlap with other rows)
RCTs
Data analysis
Monitoring
Mixed methods / theory of change
Double evaluand evaluations
16
Matrix dimensions scale and time
16
One inter-vention
Multiple inter-ventions
Enabling environ-ment
Market change
Market transform-ation
Climate change
Ex-ante
Inception
Real-time
End-of-projectEx-post
RCTs and quasi-experimental
Data analysis
Monitoring
Mixed methods / theory of change
17
Counterfactual analysis
One inter-vention
Multiple inter-ventions
Enabling environ-ment
Market change
Market transform-ation
Climate change
Local
National
Regional
Global
Ecosystem(overlap with other rows)
RCTs
Modelling of data and
experimentation (quasi- and natural)
Quasi-experimental and QCA
Social Network analysis
Delphi
18
Production causality
One inter-vention
Multiple inter-ventions
Enabling environ-ment
Market change
Market transform-ation
Climate change
Local
National
Regional
Global
Ecosystem(overlap with other rows)
Inspection, validation before/after data
Verification of data, trend analysis
Systems evaluation
