Joost de LaatStrategic Impact Evaluation Fund, World Bank

Why bother with impact evaluations?

Education example: Computers in school: good idea?

That small miracle can be replicated in every school, rich and poor, across America ... Every child in American deserves a

chance to participate in the information revolution.

President Clinton (1998)

One Laptop Per Child -Peru

It depends…some rigorous impact evaluations find no impact…

• Peru: “One laptop per child” had no sizeable impact on learning.

• Colombia: “Computers for Education” program: no impact on learning

• Israel: the “tomorrow-98”, placing 35,000 computers across elementary and middles schools across Israel had no measurable impact.

…Computer-based learning was not integrated into the curriculum.

• India: but integrated computer assisted learning in India raised student performance

We need more than programs

we need programs that work

857 estimates of program impact from 207 evaluations

We need more than programs

we need programs that work

We need more than programs

we need programs that work

Other conclusions…

• Human capital programs (e.g. training): Zero (or negative) effects in short-run, larger effects in long-run

• Public sector work programs: Negative to zero effects across all time horizons

• Females and long-term unemployed: on average, larger employment impacts (compared with males and short-term unemployed)

• Youth and elderly: on average, smaller impacts than middle-aged people

Strong growth in impact evaluations, also in low and middle-income countries

0

50

100

150

200

250

300

350

400

2006 2007 2008 2009 2010 2011 2012

110141

170

269298

364 366

All

All

Example: International Impact Evaluation Initiative (3ie) database

Using impact evaluations to make program decisions

Sustain and Expand Successful Programs

Example: Mexico PROGRESA

Using IE to Sustain and Expand Programs

Countries implementing CCT programs in

1997

Using IE to Sustain and Expand Programs

Countries implementing CCT programs in

2011

Or Cut Non-Performing Programs

• Federal US program created to prevent drug use, gang membership, and violent behavior

• Police officers visit 5th and 6th grade classrooms to educate students

Evaluations show no significant impact → funding reduced(GAO, 2003; Weiss, 2003; West and O’Neil, 2004)

D.A.R.E Program (USA)

How do impact evaluations compare to other evaluations?

Our Toolbox: Monitoring and Evaluation Methods

1Needs Assessment

2Process Evaluation

3Impact Evaluation

Analysis of Beneficiary Needs

Analysis of Program

Implementation

Measures How Much the Program

Impacts Beneficiaries

1. Services being delivered?

2. Clients satisfied?3. Services reaching

target population?

1. Youth skills improved?

2. Employment rates higher?

1. Why are we doing the project?2. Who is the target population?3. Why do we think this approach will work in this context?

How to Go from Program to Results?

Education PROGRAM• Inputs: people,

money, supplies

• Activities: Teacher training

• Output: 200 Teacher trained

INTER-MEDIATE OUTCOMES

• Teacher’ knowledge increased?

• Teachers change classroom behavior?

OUTCOMES

• Student learning improved?

• Graduation rates increased?

• Employment rates improved?

How to Go from Program to Results?

Employment

PROGRAM• Inputs: people,

money, supplies

• Activities: Job search assistance

• Output: 600 Unemployed counselled

INTER-MEDIATE OUTCOMES

• Better soft skills for searching?

• Increased search activity?

OUTCOMES

• Higher employment rate?

• Higher job satisfaction?

How to measure impact?

Children’s Learning Outcomes

Time

Intervention: extra tutoring

Impact?

Before-After comparison ok?

Before After

Treatment schools

Time

Intervention

Counterfactual

Actual Impact

=Treatment

-Control

Need a comparison group (“control”) to measure impact

Control needs to be a good proxy for the counterfactual

Before After

Treatment schools

Control schools

Children’s learning

outcomes

Quasi-experimental

methods (require more assumptions)

Randomized Assignment

Difference-in-Differences

Instrumental Variables

Regression Discontinuity Design

Matching

IE Methods

Toolbox

Randomized Assignment(or “randomized control trials; RCTs”)

Why randomize? Non-random program selection leads to

comparisons of apples and oranges

Often impossible to measure impact!

Randomization creates the right comparison group to measure impact

Good so-called

“Internal Validity”

Randomization creates the right comparison group to measure impact

“Internal Validity” still

good

But less “External Validity”

Ethics and randomization:2 scenarios

1: Lottery is fair when demand for program exceeds program resources

Target population (e.g. poor communities or schools, unemployed etc.)

Treatment

Control

2. Lottery is fair when not everyone can be served at the same time

TimeInterventionBefore

evaluation period

After

Keep in Mind

Randomized Assignment

In Randomized Assignment, large enough samples produce two statistically equivalent groups.

We have identified the perfect group match!

• Sometimes we want to compare two or more different kinds of treatments– E.g., compare cost-effectiveness of efdifferent program intensities

such as 10 versus 20 weeks of training, etc.

PS.: Multi-treatment experiments

r1/3 1

T1 C

2/3

T2

1

PopulationSample

Thank you!Who wants to learn more about impact

evaluations?