Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, CairoSlide 2 University of Applied Sciences What is Impact Evaluation? OECD/DAC (2002): “positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended” emphasises on ‘produced by’: -measures impact with clear causation (causal attribution) -considers the counterfactual, i.e. the question “What difference did this program make?” “What would have happened without the intervention?” Rigorous Impact Evaluation (RIE): Distinction against more “usual evaluations” by adding “rigorous” focus on clear causation use of adequate methods (to meet methodological shortcomings) most important point: selection of the evaluation design to consider the counterfactual
13
Prof. (FH) Dr. Alexandra Caspari [email protected]frankfurt.de Rigorous Impact Rigorous Impact Evaluation Evaluation What It Is About and What It Is About and How It Can Be Done In How It Can Be Done In Practice Practice Alexandra Caspari, Frankfurt/Main Germany Alexandra Caspari, Frankfurt/Main Germany Conference Conference » » Perspectives on Impact Evaluation: Perspectives on Impact Evaluation: Approaches to Assessing Development Approaches to Assessing Development Effectiveness Effectiveness « « 31 31 st st March – 2 March – 2 nd nd April 2009, Cairo April 2009, Cairo
Rigorous Impact EvaluationRigorous Impact EvaluationWhat It Is About and What It Is About and How It Can Be Done In PracticeHow It Can Be Done In PracticeAlexandra Caspari, Frankfurt/Main GermanyAlexandra Caspari, Frankfurt/Main Germany
Conference Conference »»Perspectives on Impact Evaluation:Perspectives on Impact Evaluation:Approaches to Assessing Development EffectivenessApproaches to Assessing Development Effectiveness««3131stst March – 2 March – 2ndnd April 2009, Cairo April 2009, Cairo
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 2University of Applied Sciences
Historical Review – The Evaluation Gap
MDGs (2000), ‘Paris Declaration on Aid Effectiveness’ (2005), and ‘Agenda for Action’ (Accra, 2008): Increasing attention to Impact Evaluations
Lack of knowledge about effectiveness of projects and programs 2006: Report “When will we ever learn?”
of the CGD ‘Evaluation Gap Working Group’ gap in quantity and quality of impact evaluations:- too few impact evaluations are being carried out and- those conducted often unable to properly assess impact
because of methodological shortcomings Recommendation: ‘Collective Action’
International Initiatives (NONIE, 3IE, …)
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 3University of Applied Sciences
What is Impact Evaluation?
OECD/DAC (2002): “positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended” emphasises on ‘produced by’:- measures impact with clear causation (causal attribution)- considers the counterfactual, i.e. the question
“What difference did this program make?” “What would have happened without the intervention?”
Rigorous Impact Evaluation (RIE): Distinction against more “usual evaluations” by adding “rigorous”
focus on clear causation use of adequate methods (to meet methodological shortcomings) most important point:
selection of the evaluation design to consider the counterfactual
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 4University of Applied Sciences
The Counterfactual
Causal effect: An actual effect δi caused by a treatment T (a program) is the difference between the outcome Yi1 under a treatment T (T=1), i.e. program participant, minus the alternative outcome Yi0 that would have happened without the treatment T (T=0), i.e. non-participant:
Impact is not directly observable:- one can observe any given individual either
as a treated person (participant) or untreated person (non-participant) but not both states
- if individual i is participating in a program (T=1), then the outcome Yi0 is unobservable
- this unobservable outcome Yi0 is called counterfactual Analyzing the difference between the observed outcome and the
unobserved potential outcome by choosing the best evaluation design
01)0,()1,( iiiiiii YYTXYTXY
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 5University of Applied Sciences
Considering the Counterfactual
often used non-experimental designs:
●: observation, P: participants (treated), t: time (first, second observation), X: project intervention
one-group pre-test post-test design (a)
time t2t1
P
measuredimpact
impa
ct in
dica
tor measured impact =
the counterfactual is not considered! with non-experimental designs
causal attribution is not possible!
1,2, tPtP YY
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 6University of Applied Sciences
Considering the Counterfactual
necessary: experimental or quasi-experimental designs adequate comparison group (‘with-and without comparison’)
- random assignment of individuals to treatment (P) and control group (C) groups differ solely due to chance
- treatment and conditions are known/checkableField experiments: - take place in real-world settings- anyhow treatment and control groups are assigned at random
Quasi-Experiments:- no random assignment- has a source of randomization that is “as if” randomly assigned- control group is often reconstructed ex-post
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 7University of Applied Sciences
Considering the Counterfactual
●: observation, P: participants (treaded), C: control group (non-treated), D: difference, t: time (first, second observation), X: project intervention
over- estimated impact
one-group pre-test post-test design (a)
time t2t1
P
measuredimpact
impa
ct in
dica
tor
static group comparison (4)
time
impa
ct in
dica
tor
t2t1
C
Pmeasuredimpact= Dt2
(single difference)
2,2, tCtT YY
timeim
pact
indi
cato
rt2t1
C
P
Dt1
pre-test post-test control group design (1)/(2)
(double difference)
measuredimpact = Dt2 – Dt1
Dt2
)()( 1,1,2,2, tCtTtCtT YYYY
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 8University of Applied Sciences
Approaches to Impact Evaluation
appropriate impact evaluation designs are often reject as unnecessarily sophisticated or because of ethical concerns
various realistic ways in which quasi-experimental designs can be introduced in an ethically and politically acceptable manner:
- Matching on Observables- Regression Discontinuity - Propensity Score Matching (PSM)- Pipeline Approach- Multiple Comparison Group Design
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 9University of Applied Sciences
Possible Approaches in Practice
Matching on Observables: - characteristics (access tor services, economic level, type of
housing, etc.) on which the comparison group should match the program group (individuals, households or areas) are identified carefully
- often easily observable or identifiable characteristics- unobservable differences has to be kept in mind- control group is build out of those individuals, households or areas
which match best- quasi-experimental design “pretest-posttest-comparison with post-
test non-equivalent control group” (3) or at least “static group comparison” (4) is possible single-difference (SD) possible
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 10University of Applied Sciences
Possible Approaches in Practice
Regression Discontinuity:- if a program is assigned using a clear threshold for eligibility
comprised for one ore more criteria (age, income less than…)- control group is built out of those just above the threshold and
hence not eligible for the program- those individuals will have comparable characteristics- quasi-experimental design “pre-test post-test non-equivalent
control group design” (2) possible! double-difference (DD) possible!
participants
individuals
elig
ibili
ty c
riter
ia
not eligiblethreshold
control group
participants
individuals
elig
ibili
ty c
riter
ia
not eligiblethreshold
control group
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 11University of Applied Sciences
Possible Approaches in Practice
Pipeline Approach:- if large programs (housing or community infrastructure,
immunization, …) are introduced in phases over several years- when there are no major differences between the characteristics of
families, communities scheduled for each phase and- when there is no selection criteria for participants of the first phase
(the poorest families, communities, …) participants of phase 2 & 3
= control group for participants phase 1
quasi-experimental design “pre-test post-test non-equivalent control group design” (2) possible!
double-difference (DD) possiblet2t1 t3
program term
phas
e
phase 1 for group 1
phase 2 for group 2
phase 3 for group 3
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 12University of Applied Sciences
Important Remarks
The international discussion about RIE refers just to a small aspect of evaluation: the causal attribution of impact
Impact is measured at the level of target groups/participants because target groups are typically large, for this evaluation step quantitative methods are necessary (representativeness vs. profundity)
other evaluation methods are not condemned! causal attribution is necessary but not sufficient ‘black box’
remains: why does a program have impact (or does not) comprehensive meaningful and reliable impact evaluations need the
use of mixed method, i.e. use of quantitative and qualitative methods
Fachhochschule Frankfurt am Main – Alexandra Caspari, 31/03/2009, Cairo Slide 13University of Applied Sciences
