TBILISI METRO EXTENSION PROJECT

IMPACT EVALUATION

July 11, 2012

Tomomi Tamaki: Principal Regional Economist, CWRD

Nina Fenton: Young Professional, CWRD

THIS PRESENTATION

1. The Tbilisi Metro Extension Project

2. The Theory: Difference-in-Differences

3. The Reality: A Modified Evaluation Design

4. Some Reflections

PART ONE

THE TBILISI METRO EXTENSION PROJECT

THE TBILISI METRO EXTENSION PROJECT

Part of Multi-tranche Financing Facility (MFF). Four

components of first tranche:

(i) Completion of a 1.2 km Tbilisi metro extension on

Vazha Pshavela Av. to the university district,

serving 150,000 inhabitants

(ii) Redevelopment of Gorgasali embankment in Tbilisi,

including pedestrianized access to the river from the

old city

(iii) Upgrading of 20 km of the Mestia urban area road

network

(iv) Urban renewal of main avenues in Kutaisi through the

introduction of a 26-km cycle network.

THE TBILISI METRO EXTENSION PROJECT

THE TBILISI METRO EXTENSION PROJECT

IE study to focus on the Tbilisi metro extension.

Expected impacts

Improved urban environment

Local economic development

Better health

Poverty reduction.

Through:

Better access to employment

Lower transport cost and travel time

Reduced pollution

PART TWO

THE THEORY: DIFFERENCE IN DIFFERENCES

A DIFFERENCE-IN-DIFFERENCES APPROACH

Infrastructure projects are not (usually) “randomly” allocated

Difference-in-differences compares: ➙ Mean outcome value before and after the

intervention (first difference) ➙…..for beneficiaries and a comparison group

(second difference).

So, differences in outcomes between project-affected population and others cannot be attributed to the project under evaluation: selection bias

THE PARALLEL TREND ASSUMPTION

The groups

develop at the

same speed in the

absence of the

project…

FAILURE OF THE ASSUMPTION

Assumption fails,

groups develop at

different speeds

THE TBILISI METRO EXTENSION: IE

Planned Methodology

Difference in difference of households, students

and businesses

Looking at key impact variables

The Problem

Failure of the parallel-trend assumption

PARALLEL TRENDS IN TBILISI?

Widespread

development in Tbilisi:

violates parallel trend

assumption

Lack of data:

hard to

identify a

“similar” area

Different parts

of town have

very different

characteristics

PART THREE

THE REALITY: A MODIFIED EVALUATION DESIGN

THE STUDENT SURVEY

1. Student survey

Difference-in-differences design

Treatment group: TSU Maglivi students (2015)

Control groups:

Cohort approach. Students entering 2014 as control, students entering 2015 as treatment group (benefit from metro)

TSU downtown students (no metro access) ILIA state students

HOUSEHOLDS AND BUSINESSES

Some outcomes more directly attributable to the metro than others, even in the absence of a control group: Travel time, travel cost.

2. Household survey

Single difference survey of 300 households before and after project completion. Measuring along the causal chain – time use, use of transportation, socio-economic indicators.

3. Enterprise survey

• Single difference survey of 300 businesses before and after project completion. Measuring revenue, profit, customer characteristics.

4. Air quality survey

Technical survey of air pollution by new station and in Delisi.

PART FOUR

SOME REFLECTIONS

BE CLEAR ABOUT THE CAUSAL CHAIN

Lay out project theory and causal chain

clearly

DMF is a starting point but, depending on

the project and the impacts we are

interested in may not be enough

Being clear about causal mechanisms can

really help with evaluation design

LOGFRAME: IMPACTS ON STUDENTS

New Metro

Less travel time

More time for other activities incl. studying

Fewer delays, higher

attendance rates

Better test scores

More convenient

travel

Fewer travel costs

More money for other things

Better well-being

THEORY-BASED IMPACT EVALUATION

Deciding what to measure means looking

at the Theory of Change:

”An explicit theory or model of how an intervention

contributes to a chain of intermediary results and

finally to the intended or observed outcomes”

”About the central processes or drivers by which

change comes about for individuals, groups or

communities”

IMPLEMENTING IMPACT EVALUATION

Expect:

Lack of data

Project delays

Difficulty in identifying control groups, especially in

rapidly developing DMCs

Be creative

Pay close attention to survey design and

enumerator training

Consider including small-scale qualitative surveys

PLANNING FOR IMPACT EVALUATION Important to consider:

→Is the budget sufficient? How will you cover the end-line? What is the project is delayed?

→Who will manage and lead the evaluation: staff

time and commitment as well as good consultants →Is there real commitment and interest from the

DMC?

→Is it likely that the results will be used to inform future project design or policymaking?

Thank you!