Assessment of Transport Projects: Risk Analysis … Kim Salling.pdfAssessment of Transport Projects:...

Assessment of Transport Projects: Risk Analysis and Decision Support

Assistant Professor

Kim Bang Salling

Presentation at the RISK Conference 2011:

December 1st 2011

DGI Byen, Copenhagen

DTU Transport, Kim Bang Salling 2

Outline

• Background

– Introduction

• Methodologies

–Cost Benefit Analysis (CBA)

–Quantitative Risk Analysis (QRA)

• Feasibility Risk Assessment (FRA)

–Accumulated Descending Graphs (ADG)

• The UNITE-DSS Decision Support Model

–Uncertainties in Transport Project Evaluation

• Conclusion

• Perspective


Background

• The Manual for socio-economic analysis in the transport sector (2003)

–Unique guidelines for evaluating transport infrastructure projects

–Lack of uncertainty handling

–Expected revision 2012-2013

• Building decision support ”with a twist”

–Rational decision making involves the assessment of both the benefits and the losses (costs)

–The need for making ”good” decisions in transport planning and evaluation are vital


Transport Planning and Assessment

Decision

support

Ongoing

transport

planning:

- Societal goals

as, for example

networks and

mobility,

sustainable

development,

etc.

- Prognoses/

forecasts

- Urban &

regional

planning

- Design

standards, etc.

Transport

infrastructure

project

proposal

Traffic

models

Impact

models

Multi-criteria

analysis

(MCA)

Cost-benefit

analysis

(CBA)

Research:

- Concepts as

for example

Feasibility Risk

Assessment

(FRA) and

Accumulated

Descending

Graphs (ADG)

- The CBA-DK

model and

@RISK software

- Case examples

related to

different modes

- Findings and

recom-

mendations


Introduction

• CBA & MCA produce single point estimates

• Informativ decision support

– Feasibility Risk Assessment (FRA)

– Accumulated Descending Graphs (ADG)

• Normally, uncertainties are handled by sensitivity tests

• Historical overview of uncertainties

– Construction cost ’overrun’

– Traffic forecast ’underrun’ (traffic modelling)


Construction Cost Overruns (fixed prices)

Construction Cost Overruns

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Channel Tunnel,

UK & France

Øresund Access

links, DK &

Sweden

Great belt link, DK Øresund coast-to-

coast link, DK &

Sweden

Co

st

Ov

err

un

(%

)

Construction cost overruns

0%

200%

400%

600%

800%

1000%

1200%

1400%

1600%

1800%

2000%

Su

ez C

an

al

Syd

ne

y

Op

era

Ho

use

Co

nco

rde

Su

pe

rso

nic

Ae

rop

lan

e

Bo

sto

n's

Art

ery

/Tu

nn

el

Pro

ject,

US

A

Hu

mb

er

Bri

dg

e,

UK

Bo

sto

n-

Wa

sh

ing

ton

-

Ne

w Y

ork

Gre

at

Be

lt

Ra

il T

un

ne

l,

DK

A6

Mo

torw

ay

Ch

ap

el-

en

-le

-

Fri

th/W

ha

ley

Sh

inka

nse

n

Jo

ets

u R

ail

line

, Ja

pa

n

Wa

sh

ing

ton

me

tro

, U

SA

Ch

an

ne

l

Tu

nn

el, U

K &

Fra

nce

Ka

rlsru

he

-

Bre

tte

n lig

ht

rail,

Ge

rma

ny

Øre

su

nd

Acce

ss lin

ks,

DK

& S

we

de

n

Me

xic

o c

ity

me

tro

lin

e,

Me

xic

o

Pa

ris-A

ub

er-

Na

nte

rre

ra

il

line

, F

ran

ce

Co

st

Ov

err

un

s (

%)


Cost-Benefit Analysis (CBA)

• Method for evaluating the ”goodness” of investments

– A systematic approach in listing costs and benefits

– Selection of the ”best” performing alternative(s)

• Inputs derived from a lot of external sources

– Traffic models and impact models

– Key figue catalogues

• Output based upon single point criteria

– Net present value (NPV)

– Benefit cost ratio (BCR)

• Transferred model uncertainties!?!?


Uncertainty in transport appraisal

• Unit price principles are assumed ”certain”

• Two types of impacts stands out: – Travel time savings -> Benefit

– Construction costs -> Cost

• Literature supports the latter

impacts by the so-called:

Optimism Bias

Sources of Uncertainty

Unit Pricing

PrinciplesModel Uncertainty

Relies on the key figure

catalogue in calibrating

and determining unit

price settings.

Relies on the model

build up of impact and

traffic models that

provide the input

towards decision

support models.

Randomness of the

systemLack of knowledge


Optimism Bias and Reference Class Forecasting

The Transport Planning Phase: Adapted from the British Department for Transport (DfT) (2004)

Reference Class Forecasting: Optimism Bias

Inside View Outside View

”Uniqueness” of Project

”The Planning Fallacy”

Reference Class

Forecasting

Forecasting of particular

projects

Forecasting from a group

of projects

(1) Identification of

relevant reference

classes

(2) Establishing

probability

distribution

(3) Placing and

comparing the

project

Optimism Bias UpliftsCurrent Situation


Optimism Bias and uplifts

• Deriving uplifts is highly dependet on large data-sets

–Flyvbjerg from (AAU) has since 2003 developed a large database

–Unfortunately, it looks upon mega-projects

• Uplift values were derived on basis of Reference Class Forecasting i.e. statistical measurements on various project pools

• Applying uplifts still produces single point rate of returns

• BUT, the data collected can be transformed and used in another way….

Risk Analysis


Risk ”Control” – Infrastructure assessment

General information

(Technical, political,

economical, etc.)

”SAFETY” SOCIETAL GOALS:

PHILOSOPHY:Definition of goals,

fundaments for priorities

and standards

ACCEPTANCE

CRITERIA:Societal acceptance,

budgetary constraints etc.

Appraising the

information brought

above

RISK ANALYSIS (TRADITIONAL):

RISK

IDENTIFICATION:Definition of risk

components - impacts

RISK

ASSESSMENT:Describe and quantify risk

by evaluation

RISK

EVALUATION:Compare risk to

acceptable standards

TRANSPORT INFRASTRUCTURE PROJECT:


Monte Carlo Simulation


Input Distributions

• Distinction between non-parametric and parametric

–Non-parametric is used when experts have to make the judgments

–Parametric are used when data and/or theory underpins the judgments

• Non-Parametric distributions:

–Uniform

–Triangular/Trigen

• Parametric distributions:

–Normal

–Erlang (Gamma) –> Construction Cost

–PERT (Beta) -> Travel time savings


Level of Knowledge (LoK)

• The LoK ranges from low to medium to high

• Distinction between Parametric and Non-Parametric distributions


PERT Distribution

• Based upon a beta distribution with the assumption that the mean can be derived from:

• This makes it ideal for modelling experts opinion

–Stands out compared to the Triangular distribution

6

4 MaxModeMinMeanPERT

Triangular

Beta-PERT

3

MaxModeMinMeanTriang

vs


Data fit (Rail) – Demand forecasts

• Demand forecasts (user benefits) are set against prior Reference classes derived from Flyvbjerg et al. (2003)

• 27 rail projects were compared where the inaccuracy on average were 39% lower than predicted

• I have fitted a PERT curve around the data from Flyvbjerg et al. (2003)


Data fit (Road) – Demand forecasts

• 183 road projects were compared where the inaccuracy on average were 9% lower than predicted

Fit Comparison for Inaccuracy in Traffic ForecastsRiskPERT(-78.5;9.6%;179.34%)

1,057-0,4875,0%90,0%5,0%

-150%

-125%

-100%

-75%

-50%

-25%

0%

25%

50%

75%

100%

125%

150%

175%

200%

Input Beta-PERT


Erlang Distribution

• Based upon a gamma distribution defined upon a shape and a scale parameter (k, )

• The shape parameter, k, depicts the skewness of the distribution whereas the scale, , is based upon data

0

0.5

1

1.5

2

0 0.5 1 1.5 2 2.5 3

K=2

K=5

K=10

K=20


Data fit (Rail) – Investment costs

• Flyvbjerg et al. Compared 58 rail projects

• Approximately 88% of the probability mass is above 0 which indicates that rail type projects are underestimated

• The fitted probability distribution contributes to the fact that an Erlang distribution is very well suited


Data fit (Road) – Investment costs

• 167 road projects were compared where the inaccuracy on average were 20% lower than predicted, with k = 8

Fit Comparison for Cost Overrun for Road ProjectsRiskErlang(8;0.09) -> (-33.6%;20.2%;222.6%)

0,569-0,1565,0%90,0%5,0%

-100%

-75%

-50%

-25%

0%

25%

50%

75%

100%

125%

150%

175%

200%

225%

250%

Input Erlang


Recommendation – High level of knowledge

• Risk analysis in decision support:

– Combination of data from Flyvbjerg, Successive Principle and Risk Analysis: Large-scale implementation in UNITE

– Definition of distributions

– Empirical data to feed the distributions

• Assigning probability distributions:

– Investment Cost – Gamma (Erlang) distribution

– Travel Time Savings – Beta (PERT) distribution

Mode Impact Distribution Low High

Rail Travel time savings PERT -90% 140%

Rail Construction cost Erlang (k = 23) -40% 120%

Road Travel time savings PERT -80% 180%

Road Construction cost Erlang (k = 8) -30% 120%

A negative sign for travel time savings means that benefits have been overestimated and a negative sign for construction costs

means that costs have been underestimated

DTU Transport, Kim Bang Salling

Uncertainties in Transport Project Evaluation (UNITE)

22

Uncertainties in Transport Project Evaluation (UNITE): the five Work-Packages

(5) Evaluation methodology

WP5 project leader: Steen Leleur (DMG)

(4) Uncertainty calculation in transport models

WP4 project leader: Otto Anker Nielsen (TMG)

(2) Organizational context of Modelling, an

empirical study

WP2 project leader: Petter Næss (AAU)

(3) Uncertainty calculation of cost

estimates

WP3 project leader: Bo Friis Nielsen

(DTU Informatics)

(1) Systematic biases in transport models (recognized ignorance), an empirical study

WP1 project leader: Bent Flyvbjerg (Oxford University)


The Case Study: HH-Connection

• Connecting Denmark with Sweden: Scandinavian link

–Currently, close to the capacity limit on Oresund

HH-Connection

(alternatives)

Description

(Alignment of connection)

Cost

(million DKK)

Alternative 1 Tunnel for rail (2 tracks) person traffic only 7,700

Alternative 2 Tunnel for rail (1 track) goods traffic only 5,500

Alternative 3 Bridge for road and rail (2x2 lanes & 2 tracks) 11,500

Alternative 4 Bridge for road (2x2 lanes) 6,000

Note! 1 € 7.5 DKK


The UNITE DSS Modelling Framework

The UNITE-DSS Decision Support Model for Risk Assessment

Determinstic Calculation

I) Cost-benefit analysis

Results: Point estimates in

terms of NPV, BCR, IRR

II) Optimism Bias Uplifts

Impact: Investment costs

Stochastic Calculation

III) Reference Class

Forecasting

Determination of Beta-PERT

distribution

Impact: Travel time savings

Results: Certainty graphs

and certainty values

Results: Point estimates in

terms of NPV, BCR, IRR

Determination of inputs to the

Beta-PERT distribution

IV) Reference Scenario

Forecasting

Determination of scenarios

and triple estimates

Impact: Travel time savings

Results: Certainty graphs

and values for scenarios

Trtiple estimate parameters

to the Beta-PERT distribution


Deterministic Module – Entry data


Results: Cost-Benefit Analysis

• Construction costs – by far the largest contributor of costs

• User Benefits – by far the largest contributor of benefits

– Consists of Ticket revenue and time savings

– Relies on the prognosis of future number of passengers i.e. demand forecasts

HH-Connection

(alternatives)

Cost

(million DKK)

BCR NPV

(million DKK)

Alternative 1 7,700 1.50 5,530

Alternative 2 5,500 0.16 -6,640

Alternative 3 11,500 2.71 28,240

Alternative 4 6,000 3.08 17,860


Results : Optimism Bias Uplifts

• The BCR are lower, however, still point estimates towards DM

–Moreover an advanced form of sensitivity analysis

• Imply to introduce risk analysis and Monte Carlo simulation

HH-Connection

(alternatives)

Cost (uplifted)

(million DKK)

BCR (orig.)

(from slide 8)

BCR (uplifts):

80% uplift

Alternative 1 12,090 1.50 0.97

Alternative 2 8,640 0.16 0.10

Alternative 3 15,180 2.71 1.75

Alternative 4 7,920 3.08 1.98


Stochastic module - @RISK

• The UNITE-DSS model is assigned an add-on software model named @RISK

• A range of distribution functions are shown

• ’Two’ non-parametric distrbutions have been tested/applied (green)

• ’Three’ parametric distributions have been tested/applied (orange)


Input in UNITE-DSS – Construction cost

• Shape parameter k = 8 for road projects and k = 23 for rail projects (including air)

• The mean () and standard (std) deviation is calculated

• The scale parameter () is calculated on basis of the succesive principle

k


Results (RCF): Monte Carlo simulation


Reference Scenario Forecasting

• Accomodates scenario analysis and RCF

• Vertical regime: Economic development due to link

• Horizontal regime: Integration between borders


Results from RSF


The coupling of methodologies in achieving feasibility risk assessment


Conclusions

• The UNITE-DSS model has been developed and functions as a flexible assessment tool applicable for wider risk oriented assessment for transport projects across different modes.

• The developed type of accumulated descending graph is found to be useful to inform about uncertainty relating to assessment of transport projects.

• Dependent on the information available parameter-based or parameter-free input probability distributions should be applied.

• It is possible to accommodate the recent results stemming from Optimism Bias and Reference Class Forecasting to produce relevant input to the PDFs for travel time savings and construction costs.


Perspectives

• Investigation of introducing non-monetary aspects to the modelling framework as discussed in some of the papers is highly relevant

• Correlations between impacts are under review as to whether a general implementation is possible/needed

• The distinguishing between ”lack of knowledge” (uncertainty) and ”inherent randomness of the system” (variability) uncertainty should be investigated further

• Finally, the combinations of Optimism Bias and Risk Analysis needs further implementation – especially, the need for reference classes are obvious

DTU Transport, Kim Bang Salling

www.transport.dtu.dk/unite

• Large-scale investigation of uncertainties

• New up-to-date database information with regard to demand forecasts (and transport models)

• Involvement of researcher from Princeton and Oxford Universities

• Cross-disciplinarian research with practical applicability

36

http://www.transport.dtu.dk/unite


Thank you for listening!

Extra slides for presentation if needed


Scenario Trend Development

Scenario Trend DevelopmentEconomic Growth and Level of Integration

90

100

110

120

130

140

150

160

2024 2029 2034 2039 2044 2049 2054 2059 2064 2069 2074

Years of evaluation

Inte

gra

tio

n level (I

nd

ex 1

00 in

2024)

High

Middle

Low


Separation of Uncertainty

Nature of Uncertainty

Uncertainty (Epistemic):

Due to lack of Knowledge

Variability Uncertainty

(Ontological):

Due to inherent variability

within the system

Traditional aspects of modelling and policy

analysis:

- Limited and inaccurate data

- Measurement error

- Incomplete knowledge

- Limited uncerstanding

- Imperfect Models

- Subjective judgments

- Ambiguities

- etc.

Behavioural variability

(Micro)

Societal variability

(Meso & Macro)Natural randomness


Cost-Benefit Analysis

Q’Q

P’’

E

B

A

P’

P

P

rice

- P

Quantity - QQ’’

''2

1''

2

1'

QQPPQQPPQPPB

TravellersGeneratedNewlyTravellersExisting


Large changes in the Demand Curve

• ADT shift from 865 before to 10.000 after

• Cost per car before 300 DKK cost after 100 DKK

Demand curve: Cars (Øresund Fixed Link)

0

100

200

300

400

500

0 2000 4000 6000 8000 10000 12000

ADT

Co

sts

831,2 1045,4 PQPkQ


Cost-Benefit Analysis

• Strengths:

– Transparency – all aspects are included in the analysis

– Comparable – Consistent, mostly due to the new manual

– Systematical data collection

• Weaknesses:

– ”False” sense of transparency – how to decide and undcover all aspects

– Practical measuring problem – models and unit prices

– Generations equity – same value today as last century

– Social equity (we are all a-like)

• Individual welfare

• Aggregation of individual welfare


Dispute of criteria NPV vs. IRR

• As shown before the IRR expression is a polynomial equation with several roots

– Gradient and discount rate determines the choice

• IRR is independent from r

• NPV is dependent on r

• Hence, changing r to r* creates problems from the two projects suggested A and B.

IRR

NP

V

r r*

A B

IRRA IRRB

NP

VA

NP

VB


Dispute about criteria NPV vs. BCR

• Given the system below with respectively costs and benefits for three system alternatives

• For a very short evaluation period of 1 year the NPV and B/C-rate are calculated


Public vs. Private

• Tax distortion of 1.2 is introduced due to the financing of projects through taxes:

– E.g. Person A willing to perform a job for 100 DKK

– Person B is willing to pay to get the job done for 110 DKK

– 50% tax would endure that Person B would pay 55 DKK

– Society loses the actual surplus of 10 DKK

• Net Taxation factor is introduced of 1.17:

– Since we operate with market prices, a private company would endure duties, taxes etc. on commodities

– The State obviously does not have to pay that

– 17% has been found as an average


Research Outcomes


Full scale uplifts from COWI and Flyvbjerg


Beta Distribution

• Typically parameterized by two shape parameters [, ]:


Gamma Distribution

• Typically parameterized by a shape and scale parameters [k, ]:

xotherforxfandkxex

k

kxf kxk

k

0)( ....4,3,2,0,!1

)( 1

kiancethewhilek

k

k 1var var

1


Succesive Calculation

Post Beskrivelse Mængde Enhed a b C m s varians*10-6

1 Opstartsarbejde 1 stk. 37.500 187.500 450.000 210.000 82.500 6.806

2 Boldbaner 50.000 m2 30 75 120 3.750.000 900.000 810.000

3 Andre græsarealer 25.000 m2 8 15 30 412.500 112.500 12.656

4 Parkanlæg 20.000 m2 8 23 60 540.000 210.000 44.100

5 Befæstede arealer 15.000 m2 90 225 330 3.285.000 720.000 518.400

6 Afsluttende arbejde 1 stk. 37.500 150.000 375.000 172.500 67.500 4.556

7 Generelle forhold 8.370.000 Sum -10 % 0 % 20 % 167.400 502.200 252.205

Kalkuleret middelværdi 8.537.400 1.648.724

Tilhørende spredning, beregnet som kvadratroden af

summen af variansen

1.284.026

Post Beskrivelse Mængde Enhed a b c m s varians*10-6

1 Opstartsarbejde 1 stk. 37.500 187.500 450.000 210.000 82.500 6.806

2 Boldbaner 50.000 m2 3.006.000 5.234

2.1 Rydning og afretning 50.000 m2 11,25 12,3 13,35 615.000 21.000 441

2.2 Dræn 50.000 m2 14,7 16,5 18,75 829.000 40.500 1.640

2.3 Vandingssystem 50.000 m2 9,75 12,75 13,5 615.000 37.500 1.406

2.4 Muld og planering 50.000 m2 12 13,5 15,9 684.000 39.000 1.521

2.5 Såning 50.000 m2 4,5 5,25 6 262.500 15.000 225

3 Andre græsarealer 25.000 m2 7,5 15 30 412.500 112.500 12.656

4 Parkanlæg 20.000 m2 7,5 22,5 60 540.000 210.000 44.100

5 Befæstede arealer 15.000 m2 90 225 330 3.285.000 720.000 518.400

6 Afsluttende arbejde 1 stk. 37.500 150.000 375.000 172.500 67.500 4.556

7 Generelle forhold 7.626.000 sum -10 % 0 % 20 % 152.520 457.560 209.361

Kalkuleret middelværdi 7.778.520 801.114

Tilhørende spredning, beregnet som kvadratroden af summen af

variansen

895.050


Data fitting

• The data fits are conducted by Maximum likelihood estimators:

– Estimates the distribution parameters

– Maximum likelihood parameter estimation is to determine the parameters that maximize the probability (likelihood) of the sample data

• The goodness of fits interpreted by using Chi-squared [2] statistics:

– The sum of differences between observed and expected outcomes

– where O is an observed outcome

– and E is an expected frequency

E

EO2

2


Background literature (international)

2002 2003 2004


Background literature (National)

2007 2007 2008


Back et al. (2000)

• Four bullet points for estimating construction costs with probability distributions have been proposed in:

– Upper and lower limits which ensures that the analyst is relatively certain values does not exceed. Consequently, a closed-ended distribution is desirable.

– The distribution must be continuous

– The distribution will be unimodal; presenting a most likely value

– The distribution must be able to have a greater freedom to be higher than lower with respect to the estimation – skewness must be expected.


Composite Model for Assessment

Alt. 1

Alt. 2

Alt. 3

CBA

MCA

.

.

.

.

.

A

B

C

D

.

.

.

SMART

AHP

B/C


A Brief History

• 1950’s: Introduction of CBA in USA – Highway’s connecting East-West

• 1960’s: CBA Methodology reaches Europe – New Motorway Schemes

• 1970’s: Traditional traffic impacts are introduced

• 1980’s: The methodology reaches Denmark together with widespread impacts within the Multi-Criteria methodology

• 1990’s: Full implementation in Denmark a general acceptance of CBA & MCA

• 2003: The Danish Ministry of Transport published in 2003 a guideline for making socio-economic analysis in the Danish Transport Sector

Date post:	15-May-2018
Category:	Documents
Upload:	ledat
View:	216 times
Download:	1 times

Assessment of Transport Projects: Risk Analysis … Kim Salling.pdfAssessment of Transport Projects:...

Documents