Policy Evaluation

Antoine BozioInstitute for Fiscal Studies

University of Oxford - January 2008

Outline

I. Why is policy evaluation important?1. Policy view2. Academic view

II. What are the evaluation problems? 1. The Quest for the Holy Grail: causality2. The generic problem: the counterfactual3. Specific problems: selection, endogeneity

III. The evaluation methods1. Randomised social experiments2. Controlling for observables: regression, matching3. Natural experiments: diff-in-diff, regression discontinuity4. Instrument variable strategy

I. Why is policy evaluation important? (policy view)1. Policy interventions are very expensive: UK

government spends 41.5 % of GDP

Need to know whether money is well spent

There are many alternative policies possible

2. Evaluation is key to modern democracies

– Citizens can differ in their preferences, in the goals they want policy to achieve => politics

– Policies are means to achieve these goals– Citizens need to be informed on the efficiency of

these means => policy evaluation

I. Why is policy evaluation important? (policy view)

3. It’s rarely obvious to know what are a policy’s effects

– Unless you have strong beliefs (ideology), policies can have wide-ranging effects (economy is very complex and hard to predict)

– Economic theory is very useful but leaves many of the policy conclusions indeterminate => it depends on parameters (individuals behaviour, markets…)

– Correlation is not causation…

I. Why is policy evaluation important? (academic view)

Evaluation is now a crucial part of applied economics:- To estimate parameters - To test models and theoriesEvaluation’s techniques have become a field in

themselves- Many advances in the last ten years- Turning away from descriptive correlations, and

aiming at “causal relationship”

I. Why is policy evaluation important? (academic view)Many fields of economics now rely heavily on

evaluation’s work• Labour market policies• Impact of taxes• Impact of savings incentives• Education policies• Aid to developing countries use of micro data different from macro analysis (cross-country…)

II. What are the evaluation problems?

1. The Quest for the Holy Grail: causal relationships

2. The generic problem: the counterfactuals are missing

3. Specific obstacles: selection, endogeneity

II -1. The Quest for Causality

Correlation is not causality!

• “Post hoc, ergo propter hoc” : looking at what happens after the introduction of a policy is not proper evaluation.

– Long term trends

– Macroeconomics changes

– Selection effects

• Rubin’s model for causal inference: the experience setting and language

- See Holland (1986)

II- 1. The Quest for Causality

We want to establish causal inference of a policy T on a population U composed agents u

To measure how treatment or the cause (T) affects the outcome of interest (Y)

Define Y(u) as the outcome of interest defined over U (It can measure income, employment status, health, …)

II – 2. Looking for counterfactuals

Looking for counterfactual: What would have happened to this person’s behaviour under an alternative policy ?

E.G.: - Do people work more when marginal taxes are

lower ?- Do people earn more when they have more

education ?- Do unemployed find more easily a job when

unemployment benefit duration is reduced ?

II – 2. Looking for counterfactuals

• Treated: agent that has been exposed to treatment (T)

• Control: agent that has not been exposed to treatment (C, non T)

• The role of Y is to measure the effect of treatment (causes have effects)

YT(u) and YC(u)

outcome that would be outcome that would be observed had unit observed had unit uu been exposedbeen exposed to to treatmenttreatment

outcome that would be outcome that would be observed had unit observed had unit uu not not been exposedbeen exposed to to treatmenttreatment

II – 2. Looking for counterfactuals

• The causal effect of treatment T on unit u as measured by outcome Y is

α(u) = YT(u) - YC(u)

• It’s a missing data problem

• Fundamental problem of causal inferenceIt is impossible to observe YT(u) and YC(u)

simultaneously on the same unit

Therefore, it is impossible to observe α(u) for any unit u

II – 2. Looking for counterfactuals

timetime

Y(uY(u))

kk

TT

CC

α(u) = YT(u)-YC(u)

The statistical solution

• Use population to compute the average causal effect of T over U

Need data on many individuals (micro data)

• Average outcome of the treated:

E[YT(u) | T]• Average outcome of the control :

E[YC(u) | C]

• Compute the difference between averages:

D = E[YT(u) | T] - E[ YC(u)| C]

II – 2. Selection bias

Compute the difference between averages:

D = E[YT(u) | T] - E[ YC(u)| C]

D= E[YT(u) - YC(u)| T ] + E[YC(u) | T] - E[ YC(u)| C]

D = α + E[YC(u) | T] - E[ YC(u)| C]

D = average causal effect + selection bias

Illustration: impact of advanced course in maths

Treatment: advanced course in maths (against standard course in maths)

Treated: students scoring above x in maths test at beginning of year

Outcome of interest: score in maths test at end of year

Measure impact of treatment by comparing average score of treated and controls by the end of year

Problem? • On average, treated students are better at maths than control

students

• Best students would always perform better on average!!

Selection bias : E[YC(u) | T] - E[ YC(u)| C] > 0 Overestimation of the true effect of the course

Illustration: impact of advanced course in maths (cont)

Compare before (t-1) and after treatment (t+1): before the advanced class and after.

D = E[YT(u) | T, t+1] - E[ YT(u)| T, t-1]

Problems? -Many other things might change : student grow older, smarter (trend issue)-Hard to disentangle the impact of the advanced class from the regular class-Grading before and after might not be equivalent…

=> Estimates likely to be biased

III. Evaluation methods: how to construct the counterfactual

1. Randomised social experiments

2. Controlling for observables: OLS, matching

3. Natural experiment: Difference in difference, regression discontinuity design

4. Instrument variable methodology

5. Other methods : Selection model and structural estimation

III- 1. Randomized social experiments

• Experiments solve the selection problem by randomly assigning units to treatment

• Because assignment to treatment is not based on any criterion related with the characteristics of the units, it will be independent of possible outcomes

• E[YC(u) | C] = E[YC(u) | T] now holds

D = E[YT(u) | T] - E[YC(u) | C] D = α = E[YT(u) - YC(u)] = causal effect

Convincing results More and more randomized policies

Reform to incapacity benefits

20%

25%

30%

35%

40%

45%

50%

Jul-

99

Oct

-99

Jan

-00

Ap

r-0

0

Jul-

00

Oct

-00

Jan

-01

Ap

r-0

1

Jul-

01

Oct

-01

Jan

-02

Ap

r-0

2

Jul-

02

Oct

-02

Jan

-03

Ap

r-0

3

Jul-

03

Oct

-03

Jan

-04

Ap

r-0

4

Jul-

04

Oct

-04

Jan

-05

Ap

r-0

5

Time of benefit start

Pe

rce

nta

ge

Non Pathways areas

Source: House of Commons Work and Pensions Select Committee Report (2006).

October2003

April2004

Six-month off-flow rate from incapacity benefits

Is the policy working?

20%

25%

30%

35%

40%

45%

50%

Jul-

99

Oct

-99

Jan

-00

Ap

r-0

0

Jul-

00

Oct

-00

Jan

-01

Ap

r-0

1

Jul-

01

Oct

-01

Jan

-02

Ap

r-0

2

Jul-

02

Oct

-02

Jan

-03

Ap

r-0

3

Jul-

03

Oct

-03

Jan

-04

Ap

r-0

4

Jul-

04

Oct

-04

Jan

-05

Ap

r-0

5

Time of benefit start

Pe

rce

nta

ge

Non Pathways areas

Phase 1 areas (pre)

Source: House of Commons Work and Pensions Select Committee Report (2006).

October2003

April2004

Six-month off-flow rate from incapacity benefits

Is the policy working?

20%

25%

30%

35%

40%

45%

50%

Jul-

99

Oct

-99

Jan

-00

Ap

r-0

0

Jul-

00

Oct

-00

Jan

-01

Ap

r-0

1

Jul-

01

Oct

-01

Jan

-02

Ap

r-0

2

Jul-

02

Oct

-02

Jan

-03

Ap

r-0

3

Jul-

03

Oct

-03

Jan

-04

Ap

r-0

4

Jul-

04

Oct

-04

Jan

-05

Ap

r-0

5

Time of benefit start

Pe

rce

nta

ge

Non Pathways areas

Phase 1 areas (pre)

Phase 1 areas (post)

Source: House of Commons Work and Pensions Select Committee Report (2006).

October2003

April2004

Six-month off-flow rate from incapacity benefits

Is the policy working?

20%

25%

30%

35%

40%

45%

50%

Jul-

99

Oct

-99

Jan

-00

Ap

r-0

0

Jul-

00

Oct

-00

Jan

-01

Ap

r-0

1

Jul-

01

Oct

-01

Jan

-02

Ap

r-0

2

Jul-

02

Oct

-02

Jan

-03

Ap

r-0

3

Jul-

03

Oct

-03

Jan

-04

Ap

r-0

4

Jul-

04

Oct

-04

Jan

-05

Ap

r-0

5

Time of benefit start

Pe

rce

nta

ge

Non Pathways areasPhase 1 areas (pre)Phase 1 areas (post)Phase 2 areas (pre)Phase 2 areas (post)

Source: House of Commons Work and Pensions Select Committee Report (2006).

October2003

April2004

Six-month off-flow rate from incapacity benefits

Drawbacks to social experiments• Experiments are costly and therefore rare (less rare

now)– Cost and ethical reasons, feasibility

• Threats to internal validity– Non response bias. Non-random dropouts

– Substitution between treated and control

• Threats to external validity– Limited duration

– Experiment specificity (region, timing…)

– Agents know they are observed

– General equilibrium effects

• Threats to power– Small sample

Non-experimental approaches

• Aim at recovering randomisation, thus recovering the missing counterfactual

E(YC|T)

• This is done in different ways by different methods

• Which one is more appropriate depends on the treatment being studied, question of interest and available data

III 2. Controlling on observables

1/ Regression analysis (OLS)Y = a + b X1 + c X2 + d X3

Problems : a/ Omitted variables lead to bias and some variables may be

unobservableExample: Effect of education on earnings

Ability or preference to work is hardly observable

b/ Explanatory variables might be endogenous Example: Effect of unemployment benefit durationWhen unemployment increases, policies tend to

increase unemployment benefit duration=> Correlation is NOT causality !!

III 2. Controlling on observables

2/ Matching on observables: It is possible to compare groups that are similar relative to

the variables we observe (same education, income…)• Explores all observable information

• Take X to represent the observed characteristics of the units other than Y and D

• It assumes that units with the same X are identical with respect to Y except possibly for the treatment status

• Formally, what is being assumed isE[YC|D=T,X] = E[YC|D=C,X]

Matching (cont)

timetime

YY

kk k+1k+1

αM(X) =

YD=T(k+1,X) – YD=C(k+1,X)

αT|D=C,XT|D=C,X

T|D=T,XT|D=T,X

C|D=CC|D=CUse X characteristics to Use X characteristics to ensure comparable units are ensure comparable units are being comparedbeing compared

C|D=C,XC|D=C,X

III - 3. Natural experiments

• Explore sudden changes or spatial variation in the rules governing behaviour

• Typically involve one group that is affected by the phenomena (the treated) and one other group that is not affected (the control)

• Observe how behaviour (outcome of interest) changes as compared to change in unaffected group

Difference in difference Regression discontinuity estimation

Difference in differences (DD)

• Suppose a change in policy occurs at time k

• We observe agents affected by policy change before and after the policy change, say at times k-1 and k+1 : A= E[YT|t=k+1] – E[YT|t=k-1]

• We also observe agents not affected by the policy change at the same time periods: B=E[YD|t=k+1] – E[YD|t=k-1]

• DD =A- B = true effect of the policy

Difference in differencesDifference in differences

timetime

YY

kk k+1k+1k-1k-1

αT|D=CT|D=C

T|D=TT|D=T

C|D=CC|D=CUse fact that difference Use fact that difference between T and C remains between T and C remains fixed over timefixed over time in C regimein C regime

Under certain conditions

1. No composition changes within groups

2. Common time effects across groups

Checking the strategy- Checks the DD strategy before the reform- Use different control groups- Use an outcome variable not affected by the reform

Has to be a careful study ! Can take into account unobservable variables !

Difference in differences: differential Difference in differences: differential trendstrends

timetime

YY

kk k+1k+1k-1k-1

E[YC(k+1) - YC(k-1)|D=C] ≠

E[YC(k+1) - YC(k-1)|D=T]

αT|D=CT|D=C

T|D=TT|D=T

C|D=CC|D=CTime trend in T and C Time trend in T and C groups are differentgroups are different

Difference in differences: Difference in differences: Ashenfelter’s dipAshenfelter’s dip

timetime

YY

kk k+1k+1k-1k-1

DD assumption holds only in certain periods

αT|D=CT|D=C

T|D=TT|D=T

C|D=CC|D=CT often experience particularly T often experience particularly bad shocks before deciding to bad shocks before deciding to enrol into treatmentenrol into treatment

k-2k-2

Diff-in-Diff : Esther Duflo (AER 2001)

• Policy: school construction in Indonesia• Regional difference: low and high• Children young enough to be affected =

treated• Children too old = control Estimate the impact of building school on

education Estimate the impact of education on earnings

Diff-in-Diff : Esther Duflo (AER 2001)

Diff-in-Diff : Esther Duflo (AER 2001)

Common problems with DD

• Long-term versus reliability trade-off:- Impact most reliable in the short term- True impact might take time

• Heterogeneous behavioural response– Average effect might hide high/low effect for certain groups

• Local estimation – Truly DD estimates are hard to generalize

Need many estimations to establish general causal effect

Regression discontinuity design (RD)

• Use the discontinuity in the treatment

Example: New deal for young people in the UK

Program targeted to young unemployed aged 18 to 24

Unemployed just older than 25 are in the control group

Unemployed just younger than 25 are in the treated group

De Giorgi (2006) : New deal for young people

III - 4. Instrumental variable

• Use the fact that a variable Z (the instrument) might be correlated with the endogenous variable X

• BUT not with the outcome Y• Except through the variable X

E.G.: number of student per class is endogenous to outcomes “test scores”

=> How to find a good instrument ?

Angrist and Lavy (QJE 1999)

III - 5. Other methods

• Matching mixed with diff-in-diff• Selection estimator• Structural estimation There is a trade-off between reliability of the

causal inference (identification) and the generalization of the results

Conclusion

• Policy evaluation is crucial– For conducting efficient policies– For improving scientific knowledge

• Correlation is not causality• Beware of the selection effect or of

endogenous variable !• Methods to draw causal inference are

available=> Need careful analysis !