1

The Need for Control: Learning what ESF achieves

Robert Walker

Identifying impact

Time

Outcome

Control orcounterfactual

Policy success?

Introduction of policy

Impact

Defining the counterfactual

Time

Outcome

Introduction of policy

Control group

Action group

(Treatment group)

Defining control groups

• Random assignment

• Random encouragement designs

• Matching areas

• Individual matching

• Propensity score matching

• Difference in difference

• Regression discontinuity designs

Defining the control groupRandom assignment

12

6

9

53 4

10

12

87

11

Action group

Control group

Random number =

Defining the control groupRandom assignment

1

26

9

5

3

4

10

8

7

11

Action group

Control group

12

Random = 7 = ControlNumber Group

Advantages of random assignment

• Action group and controls same (with large numbers) with respect to:– Observed characteristics– Unobserved characteristics

• Unbiased estimate of mean impact• Possible to state degree of confidence that

estimate is a true measure of impact

Non- participants

Eligibleindividuals

General population

Participants

Controlgroup

Programme group

Random assignment

Determineeligibility

Ineligibleindividuals

Random assignment

Adapted from Orr (1999)

Non- participants

Eligibleindividuals

Jobseekers Finland

Participants

Controlgroup

Jobsearch training

Random assignment

Determineeligibility

Ineligibleindividuals

Jobsearch training - Finland

Malmberg-Heimonen and Vuori (2005)

If staff thought would benefit &Did not express a preference for training

677

19 employment offices

338

430 247

20.3% 19.5% find employment

Seek informed consent 18% asked by post, 28% asked face to face agreed

Jobcentre Plus office

New Deal25+

New Deal forLone Parents

Working Tax Credit

ERADEmployment Retention and Advancement

Demonstration

P C

RA

Mandatory WFI

Voluntary JC+ visit

P C

RA

P C

RA

JB+ initiated contact

Participants

Targetpopulation

Non-participants

Encourage to participate

No encourage-ment

Random assignment

Participants Non-participants

Randomised encouragement design

IMPACT

Defining the control group

Geographic areas(Limited equivalence)Population

Area matchingUK - Educational Maintenance Allowance

Propensity Score Matching

Programme

Control

Criteria for inclusion in control

Have the combination of characteristics

making them likely to be included in the programme group

PSM Norway, Rehab and

Activation

Rønsen and Skarðhamar (2009)

Months to finding work

Participants

0 5 10 15 20

Controls

(Social security recipients)

Months until 25% find

work

Controls 20

Participants 11

Difference in difference design O

utc

om

e

TimeT1 T2

Programme group

Control group

CounterfactualDifference at T2 (D2)

Difference at T1 (D1)

Difference at T1 (D1)

IMPACT(D2 - D1)

Regression discontinuity design

Measure of need/eligibility (at time T1)

Outcome (at time T2)

(Measure of need)

High

High

Low

Low

Regression discontinuity design

Measure of need/eligibility (at time T1)

Outcome (at time T2)

(Measure of need)

High

High

Low

Low

Threshold

Regression discontinuity design

Measure of need/eligibility (at time T1)

Outcome (at time T2)

(Measure of need)

High

High

Low

Low

Threshold

Non-participantsParticipants

IMPACT

Regression discontinuity design: Extended UB entitlement, Austria

IMPACT

14.8 weeks extra unemployment

Unemploymentduration

Age

Lalive (2008)

Characteristic Random assignmen

t

Random encouragement

Matching RegressionDiscontinuity

Individuallevel

Areacontrol

s

Propensity score

matching

Characteristics of policy intervention (1)Measure systemic effects

No(Yes, for Cluster

RA)

No No Yes No Yes

Multiple target groups

Possible for a few

Difficult but possible for a

few

Possible for a few

Not easy

Yes Yes

Multiple policy components

Possible for a few

Difficult but possible for a

few

Not easy

Competing policy designs

Possible for a few

Difficult but possible for a

few

Not easy

Characteristic Random assignmnt

Random encouragement

Matching RegressionDiscontinuity

Individuallevel

Areacontrol

s

Propensity score

matching

Characteristics of policy intervention (2)

Need to measure second order effects

No(or only by using quasi experimental methods)

Yes(but only by further supplementary individual level matching, i.e. further quasi-experimental

methods)

Targeted on hard to reach group

Difficult Not easy Difficult Not easy

Difficult Not easy

Targeted at areas

No (Yes, for Cluster RA)

Not easy Yes Yes Yes

Characteristc Random assignme

nt

Random encouragemnt

Matching RegressionDiscontin-ty

Individuallevel

Areacontrol

Propnsty score

matching

Ease of implementationDisruption of existing policy

Yes No Yes No No No

Requirement for informed consent

Yes No Yes No No No

Ethical concerns

Real for all prospective evaluation since people are placed at risk of harm due to possible negative effects of previously untested policy for the

common good

Scale: sample sizes

Small Small Very large

Mod-erate

Large Moderate

Data requirements

Low Low Moderate Mod-erate

High Moderate

Characteristic Random assignmnt

Random encouragement

Matching RegressionDiscontinuit

yIndividuallevel

Areacontrol

s

Propensity score

matching

Quality of impact estimatesUnbiased impactEstimate

Yes Yes No No No No(yes at margin)

Contamintion of control group

Moderate Moderate Moderate Low Low Low

Control for unobservable variables

Yes Yes No No No No

Overall control Excellent Excellent Poor Poor Moderate Moderate

External validity

Limited Limited Moderate Limited Higher Higher

Conclusion• No evaluation method is perfect• Evaluation without a counterfactual is very, very

likely to wrong and misleading• Only random assignment (and regression

discontinuity designs) produce unbiased estimates

• It is impossible to be confident that controls are adequate

• Control groups are generally better than no control group – they force evaluators to think about the characteristics of a good counterfactual