+ All Categories
Home > Documents > Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program...

Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program...

Date post: 04-Jan-2016
Category:
Upload: cori-higgins
View: 220 times
Download: 0 times
Share this document with a friend
Popular Tags:
44
Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)
Transcript
Page 1: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

Nigeria Impact EvaluationCommunity of Practice

Abuja, Nigeria, April 2, 2014

Measuring Program Impacts ThroughRandomization

David Evans (World Bank)

Page 2: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

2

Objective

Evaluate the causal impact of a program or an intervention on some outcome

Examples How much did free distribution of

bednets decrease malaria incidence?

Which of two supply chain models was most effective at eliminating drug shortages?

Page 3: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

3

Counterfactual Criteria

Treated & comparison groups…Have identical average

characteristics (observed & unobserved)

The only difference is the treatment

Therefore the only reason for any difference in outcomes is the treatment

Key question: What would participant look like if she hadn’t received the program?

Page 4: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

4

Perfect Experiment

1. Identify target beneficiaries

2. Clone them!

Identical on the outside (observable)

Identical on the inside (unobservable)

Chief Ahun

We’re both middle-aged

chiefs

We both love to take up new

health interventions!

Chief Batun

Page 5: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

5

Perfect Experiment

Give the intervention to one set of clones

Ahun Batun

Page 6: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

6

Perfect Experiment

Observe some time later

Because the groups are identical (inside & out), the difference is due to the bednets!

Ahun

Batun

Page 7: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

7

Back to Reality

What would Batun look like if he didn’t receive the bednet?

Room For Improvement Control Groups

Before – After Participants – Non Participants

???

Page 8: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

RFI: Before-After

BEFORE BEDNETS

6 malaria episodes in 6 months

AFTER BEDNETS

2 malaria episodes in 6 months

What else might be going on besides the bednets?

• Seasonal differences• Rising incomes: Households invest in other measures

Too many other factors!

Impact of bednets = ???

Page 9: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

RFI: Before-After

Important to monitor before-after Insufficient to show impact of

programToo many factors changing over time

Example of cash transfers in Nicaragua!

Counterfactual: What would have happened in the absence of the project, with everything else the same

Page 10: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

10

RFI: Participants vs Non-Participants

Compare recipients of a program to People who were not eligible for the

program People who chose not to enroll in the

program

Home births Clinic births

Example: Complications in childbirth

Impact of clinic births?

What else might explain the difference?

Page 11: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

11

RFI: Participants vs Non-Participants

Observable differences Income Education

Unobservable differences Heard rumor about hospitals Neighbor available to care for other

children

Mercy Patience

Page 12: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

12

RFI: Participants vs Non-Participants

How much of difference is because of clinic?

Impact of clinic births = ???

Home births Clinic births

Example: Complications in childbirth

Impact of clinic births

Other factors!

Page 13: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

13

Selection bias

People who choose to join the program are different!

If we cannot account completely for those differences in our data…

We never can How do you capture attitudes toward

health systems? Initiative? …then our comparison will not show

the true impact of the program

Page 14: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

What should we do?

Gold standard:Randomized experimental design

Page 15: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

15

Randomized Experimental Design

Randomly assign potential beneficiaries to be in the treatment or comparison group

Treatment and comparison have the same characteristics (observed and unobserved), on average, so…

Any difference in outcomes is due to treatment

Page 16: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

16

Why Randomization Works

Randomization with two doesn’t work!

But differences average out in a big sample

On average, same number of Ahuns and Patiences Observable AND unobservable

Result: Measure true impact of program

ComparisonTreatment

Comparison

Treatment

Page 17: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

17

Random Sample or Assignment?

RANDOM SAMPLE

Select randomly who to gather data on

Gives unbiased average of the group NOT of impact

If take random sample of group: Half women, half men – Sample should be about ½ women, ½ men

RANDOM ASSIGNMENT

Randomly assign who gets the program

Gives unbiased estimate of program impact

Why? Treatment & comparison

are IDENTICAL (on average)

T

T

C

C

Page 18: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

18

LET’S RANDOMIZE!

1. Identify the eligible participants

What is the impact of

receiving a new car on body-mass

index?

Page 19: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

19

LET’S RANDOMIZE!

2. Generate a random number for each one

Page 20: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

20

LET’S RANDOMIZE!

2. Generate a random number for each one

Page 21: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

21

LET’S RANDOMIZE!

3. Re-order based on the random number

Page 22: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

22

LET’S RANDOMIZE!

4. Assign the first ten to receive cars

I really wanted a car!

Page 23: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

23

LET’S RANDOMIZE!

5. Check for balance across treatment & control

Treatment# Drs: 4

Control# Drs: 6

Total observations21

Not a very big sample!

How close? 2/3

Page 24: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

24

LET’S RANDOMIZE!

What if we had 500 observations?

Treatment# Drs: 114

Control# Drs: 116

How close? 98/100

Page 25: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

25

Is there more?

That’s a simple way to randomize Works with BIG samples

You can help randomization by stratifying Randomize within each sub-group Ensure that each group is equally

represented

Page 26: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

26

Stratifying

1. Identify the characteristic(s) to stratify on

Page 27: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

27

Stratifying

2. Sort on those characteristics

Page 28: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

28

Stratifying

3. Generate those random numbers

Page 29: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

29

Stratifying

4. Sort on those characteristics

Page 30: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

30

Stratifying

5. Assign treatment within each sub-group

Result: Equal doctors in each group

Page 31: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

31

Can we really randomize?

Randomization does not mean denying people the benefits of the project

Usually existing constraints in project roll-out allow randomization

Randomization often the fairest way to allocate treatment

Page 32: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

32

Use Staggered Roll-out

Roll-out to 200 clinicsRoll-out to 400 more

clinics

Jan 2014

July 2014

Jan 2015

• Randomize the order in which clinics receive program

• Compare Jan 2014 group to Jan 2015 group at end of first year

Page 33: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

33

Some groups must get the program

Page 34: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

34

Vary treatment

INTENSITY OF TREATMENT

Malaria Information Campaign•100 villagesMalaria Information Campaign + SMS Reminders•100 villages

NATURE OF TREATMENT

Radio campaign•100 villages

Newspaper campaign•100 villages

Page 36: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

36

What if randomization is impossible?

Think again: It often is possible on some level, and it’s the best way to get a clear measure of impact Always begin the IE with imagining what the

ideal would look like

With a national policy Use randomization to test implementation

Page 37: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

37

Key takeaway #1

The single best way to evaluate the unbiased average impact of an activity is by randomizing treatment.

Page 38: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

38

Key takeaway #2

It is more ethical to test programs rigorously before universally implementing them than it is to use scarce public resources to implement a universal program with uncertain benefits.

Page 39: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

39

Key takeaway #3

Randomization is more flexible than you think:

It does not require withholding of benefits.

It can take advantage of necessary staggered roll-out.

It can test different reforms or packages of services across groups at the same time (so all receive at least some package).

Page 40: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

40

Let’s randomize

Page 41: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

41

Thank you!

Page 42: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

BONUS SLIDES

Page 43: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

43

Page 44: Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

44

LET’S RANDOMIZE!

What if we had 100 observations?

Treatment# Drs: 26

Control# Drs: 22

Total observations21

Not a very big sample!


Recommended