CENTER FOR AIR CENTER FOR AIR TRANSPORTATION SYSTEMSTRANSPORTATION SYSTEMS

FAA-NEXTOR

NAS/ATM Performance Indexes

Dr. Alexander (Sasha) Klein

2

CATSCATSRR

This research was funded by the FAA-NEXTOR-GMU contract #DTFAWA-04-D-00013

Many thanks to our FAA sponsors:

• Steve Bradford, Rich Jehlen, Diana Liang – FAA

Also to:

• Stephane Mondoloni – CSSI

• Glenn Roberts - MITRE CAASD

• George Donohue, Lance Sherry – GMU

• Barry Davis, Carlton Wine – FAA; Doug Williamson - Crown

Acknowledgments

3

CATSCATSRR

Project Objectives

Develop a framework for assessing NAS/ATM performance on a recurring / daily basis• Come up with a simple yet informative index of weather-related ATM

performance on a given day (“one number”)

• Produce charts for each season

• Compare different seasons: “Did we do better this year than last year?”

Account for major external factors:• Weather

• Traffic Demand

Enhance existing methods for NAS performance analysis• Refine computation of the effects of both en-route and

terminal weather

• Consider additional metrics alongside Delay

NAS

Traffic Demand

(Schedules)

ProceduresFacilities

Weather

ATM

Delays

Costs

4

CATSCATSRR

Operational Response Index (ORI), 2004 Components, Excluding Highest-Cost Days

ORI Component Analysis, 2004 (excluding a few highest-cost snowstorm outliers)

$0

$2,000,000

$4,000,000

$6,000,000

$8,000,000

$10,000,000

$12,000,000

$14,000,000

1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 321 341 361

Day, from lowest to highest cost

5-Pe

riod

Mov

ing

Ave

rage

of $

$ C

ost

Excess Distance (navy blue)

Diversions (yellow)Excess Block Time (pink)

Cancellations (light blue)

ORI = Total daily OPSNet cost of • Excess block time vs. Schedule, • Excess distance vs. Flight-planned, • Cancellations, and • Diversions

per flight

ORI:

Direct airline operating costs per flight for an “averaged” narrowbody fleet

All OPSNet flights daily

A cost-derivedmetric

Computation details here

5

CATSCATSRR

ORI for 366 Days 01/01 – 12/31/2004, sorted by Date

ORI, 2004, All days

0

100

200

300

400

500

600

700

800

900

1,000

1/1/200

41/1

5/2004

1/29/2

0042/1

2/2004

2/26/2

0043/1

1/2004

3/25/2

0044/8

/2004

4/22/2

0045/6

/2004

5/20/2

0046/3

/2004

6/17/2

0047/1

/2004

7/15/2

0047/2

9/2004

8/12/2

0048/2

6/2004

9/9/200

49/2

3/2004

10/7/

2004

10/21

/2004

11/4/

2004

11/18

/2004

12/2/

2004

12/16

/2004

12/30

/2004

Snowstorms

Quiet period

Convective season

Hurricanes and convective Wx

Snowstorms

Somewhat quieter period

Volatile Wx in early spring

6

CATSCATSRR

Weather-Impacted Traffic Index (WITI)Combined En-Route and Terminal Wx

En-Route WITI:

• Find intersections of each flow (GC track) with hex cells where convective Wx was reported

• Multiply by each hex cell’s total NCWD count (reflects Wx duration) and by # of daily flights on this flow

• Add up all flows: En-Route WITI

Terminal WITI:

• Hourly surface Wx observations at major airports

• Capacity degradation % for each Wx type * hourly movement rate

• Add up all airports: Terminal WITI

Combined WITI (CWITI):

• Weighted sum of En-route and Terminal WITI

• Reflects “front-end” impact of Wx on intended flights

7

CATSCATSRR

Combined WITI and NAS Performance MetricsExample: ORI & Delays, June-Oct 2004 Including Outliers

ASPM Arr Delays and Operational Response Index (ORI) vs. Combined WITI

0

10000000

20000000

30000000

40000000

50000000

60000000

70000000

80000000

90000000

0 10000000 20000000 30000000 40000000 50000000 60000000 70000000

Combined WITI

ASPM Arrival Delay

ORI

9/49/5

Hurricane Frances

9/26Hurricane Jeanne

9/16Hurricane Ivan

9/3

Even in ideal weather, there are significant “residual” delays and costs

8

CATSCATSRR

ORI and Delays vs. Combined WITIExample 1: Convective Weather

ASPM Arr Delay and ORI vs. Combined WITIOutliers (Exceedingly high costs on or around hurricane days) are removed

0

10,000,000

20,000,000

30,000,000

40,000,000

50,000,000

60,000,000

70,000,000

80,000,000

90,000,000

0 10,000,000 20,000,000 30,000,000 40,000,000 50,000,000

Combined WITI

July 14, 2004

Very high ORI ($485/flight)

Very high delays

Medium-high WITI

Checking…

ASPM Arrival Delay

ORI

Jun-Nov 2004

9

CATSCATSRR

Zooming In on July 14, 2004…ORI, En-Route and Terminal WITI

En-route WITI is high (July…)

Terminal WITI is low

Combined WITI is medium-high

Very high % of cancellations: 3x the usual

High “operational response cost” (ORI) was caused by en-route thunderstorms leading to delays and cancellations

NAS performance was worse than usual on this day

0

10,000,000

20,000,000

30,000,000

40,000,000

50,000,000

60,000,000

70,000,000

80,000,000

90,000,000

100,000,000

7080

4

7090

4

7100

4

7110

4

7120

4

7130

4

7140

4

7150

4

7160

4

7170

4

7180

4

WITI ATerminal WITIORICombined WITI

T-WITI

E-WITI

Comb-WITIORI

10

CATSCATSRR

ORI and Delays vs. Combined WITIExample 2: Non-Convective Weather

ASPM Arr Delay and ORI vs. Combined WITIOutliers (Exceedingly high costs on or around hurricane days) are removed

0

10,000,000

20,000,000

30,000,000

40,000,000

50,000,000

60,000,000

70,000,000

80,000,000

90,000,000

0 10,000,000 20,000,000 30,000,000 40,000,000 50,000,000

Combined WITI

October 20, 2004

ORI is high but in line with average trend

Relatively low delays

Very high WITI

Checking…

ASPM Arrival Delay

ORI

Jun-Nov 2004

11

CATSCATSRR

Zooming In on October 20, 2004…ORI, En-Route and Terminal WITI

0

10,000,000

20,000,000

30,000,000

40,000,000

50,000,000

60,000,000

70,000,00010

1504

1016

04

1017

04

1018

04

1019

04

1020

04

1021

04

1022

04

1023

04

1024

04

1025

04

WITI ATerminal WITIORICombined WITI

T-WITI

E-WITI

Comb WITI

ORI

En-route WITI is low (late October)

But Terminal WITI is very high (rain, low ceilings etc)

So the Combined WITI is high

That is, high ORI ($/flight) and delays were caused mostly by terminal Wx

NAS performance was actually goodfor this “IMC day”

(Better in terms of delays than costs)

12

CATSCATSRR

ORI and Delays vs. Combined WITIExample 3: Two Metrics Yield Different Results

ASPM Arr Delay and ORI vs. Combined WITIOutliers (Exceedingly high costs on or around hurricane days) are removed

0

10,000,000

20,000,000

30,000,000

40,000,000

50,000,000

60,000,000

70,000,000

80,000,000

90,000,000

0 10,000,000 20,000,000 30,000,000 40,000,000 50,000,000

Combined WITI

November 23, 2004

High CWITI

High delays but just average ORI

NAS performance could be judged as “poor” if only delays were considered

But considering costs (ORI), it was about average given the weather and the demand

ASPM Arrival Delay

ORI

Jun-Nov 2004

13

CATSCATSRR

Comparing Delays for 2004 and 2005May-September

ASPM Arrival Delays (avg delay for all flights, ASPM 55 airports, vs. schedule)

0

5

10

15

20

25

30

35

40

5/1 5/11

5/21

5/31

6/10

6/20

6/30

7/10

7/20

7/30 8/9 8/19

8/29 9/8 9/18

9/28

2005 delays were on average about the same as in 2004 (1% diff.)

But, weather (CWITI) was on average better in May-Sep 2005

(If 2004 average = 100, then 2005 average = 83)

20042005

14

CATSCATSRR

Normalized Delay-to-Wx Ratio ComparisonDelay-Based NAS/ATM Performance Index, ‘05 vs.’04

Normalized ASPM Arr Delay vs Weather, May-Sep 2004 and 2005Against 2004 Trendline (all days, including hurricane-impacted)

0

50

100

150

200

250

300

350

5/1/20

045/8

/2004

5/15/2

0045/2

2/2004

5/29/2

0046/5

/2004

6/12/2

0046/1

9/2004

6/26/2

0047/3

/2004

7/10/2

0047/1

7/2004

7/24/2

0047/3

1/2004

8/7/20

048/1

4/2004

8/21/2

0048/2

8/2004

9/4/20

049/1

1/2004

9/18/2

0049/2

5/2004

20042005

2004 benchmark 2005 average

2004 average Delay vs. 2004 WITI = 1002005 average Delay vs. 2005 WITI vs. 2004 WITI = 120

15

CATSCATSRR

Delays and Traffic Demand Taking Exponential Delay-vs.-Demand Factor into Account

• Looking at 1990-2005 historical monthly averages…

• 2005: a 10% traffic increase

• 10% increase in traffic (from 4.3M to 4.7M ops) can lead to a 45% increase in delays (from 1.25M to 1.8M minutes)

• This factor ought to be taken into account when we talk about NAS / ATM performance

• The trend doesn’t depend on weather

• Adjusted chart is shown on next slide

Monthly total delays vs operations, Jan 1990 - Aug 2005

0

500000

1000000

1500000

2000000

2500000

3000000

1.2E+0

63.5

E+06

3.7E+0

63.9

E+06

4.0E+0

64.1

E+06

4.2E+0

64.3

E+06

4.4E+0

64.6

E+06

Monthly instrument ops, NAS OPSNet

Mon

thly

del

ay to

tal,

min

utes

16

CATSCATSRR

ATM Performance Index, 2005 vs. 2004Adjusted by Exponential Delay-vs-Demand Factor

2004 benchmark = 1002005 average adjusted for Weather only = 120

Adjustment factor: divide by 145% (exponential delay increase rate), multiply by 110% (traffic increase rate; need to pro-rate 2005 back to 2004)

2005 adjusted-for-Weather-and-Demand average = 120 / (1.45 / 1.1) = 91

Normalized ASPM Arr Delay vs Weather, May-Sep 2004 and 2005Against 2004 Trendline, Adjusted by Delay-vs-Demand Factor (all days, including hurricane-impacted)

0

50

100

150

200

250

300

350

5/1/

2004

5/8/

2004

5/15

/200

4

5/22

/200

4

5/29

/200

4

6/5/

2004

6/12

/200

4

6/19

/200

4

6/26

/200

4

7/3/

2004

7/10

/200

4

7/17

/200

4

7/24

/200

4

7/31

/200

4

8/7/

2004

8/14

/200

4

8/21

/200

4

8/28

/200

4

9/4/

2004

9/11

/200

4

9/18

/200

4

9/25

/200

4

20042005

2004 benchmark

2005 adjusted average

2005 adjusted average = 91

17

CATSCATSRR

DiscussionDelays

Did the NAS/ATM do 9% better in 2005 than in 2004?• NAS delays were similar; delays vs. weather were worse in 2005…

• But, relative to weather and traffic demand, the ATM component of the NAS did do better in 2005 than in 2004

• D-RVSM and other measures may have helped

Even so, • We are on the ascending slope of the

exponential delay curve

• Peak delays in bad weather (July 2005) were highest ever

• Delay variance is significant

• The exact proportion (45% delay increase due to 10% traffic demand growth) needs to be fine tuned

0

500000

1000000

1500000

2000000

2500000

3000000

Monthly instrument ops, NAS OPSNetM

onth

ly d

elay

tota

l, m

inut

es

We are here

18

CATSCATSRR

00

50

00

50

00

50

0060104 61104 62104 70104 71104 72104 73104 81004 82104 83104 91404 9300

Normalized Indices, Jun-Sep 2004"Cost", "Weather", and "NAS Cost-to-Wx Index", normalized (Current-day / Average)

Outliers (hurricane-impacted days) removedStandard Deviation: Cost 0.22, Weather 0.4, NAS Cost-to-Wx Index 0.6

-3.00

-2.50

-2.00

-1.50

-1.00

-0.50

0.00

0.50

1.00

1.50

2.00

2.50

3.00

60104 61104 62104 70104 71104 72104 73104 81004 82104 83104 91404 9300

ORI: Cost-to-Wx Ratio The Ruler for ORI (100) is an Averaged Day

Normalized Wx

-300

-250

-200

-150

-100

-50

0

50

100

150

200

250

300

60

0

50

100

150

200

250

300

6-300

-250

-200

-150

-100

-50

0

50

100

150

200

250

300

60

0

50

100

150

200

250

300

6

Normalized-Cost-to-Normalized-Wx Ratio(is roughly inversely proportional to ORI)

Normalized Cost (ORI)

Values below 100 are goodHigh peaks are bad

19

CATSCATSRR

Conclusions

Delay, cost (ORI) and weather (WITI) metrics computed for 2004 and 2005• Delay metric can be normalized vs. seasonal-average (e.g. 2004’s)

• Normalized cost (ORI) is a useful additional metric

WITI calculation refined for both en-route and terminal parts

Delay/Cost metrics should account for traffic demand, not just weather, if used as NAS/ATM performance indicators• 10-15% traffic demand increase can cause 45-60% increase in delays

• Slightly better NAS/ATM performance in 2005 if both weather andtraffic demand are taken into account

These metrics can advance our understanding of NAS response to external impacts

20

CATSCATSRR

NAS Response to External Impacts

Traffic Demand

Avoidable

Inefficiencies

Excess Demand vs. Capacity

Unavoidable

A T M

NAS

Unavoidable NAS Response(Delays, Costs)

WeatherImpact

Inefficiencies

Excess D vs. C

UnavoidableNAS/ATM Efficiency Improvements

1) What portion of delays/costs is due to system inefficiencies as opposed to unavoidable weather and traffic demand outside ATM’s control?

2) Can we quantify positive impact of NAS/ATM efficiency improvements?

21

CATSCATSRR

Back-up Slides

22

CATSCATSRR

Operational Response Index (ORI) Components

Using direct carrier costs only• Passenger impact (value of pax time, ‘ill will’, re-issuing tickets etc) excluded

Flights per day: OPSNet daily totals (varies between 37,000 and 50,000)• Simplifying assumption: all aircraft are narrowbodies

Cost of 1 Minute of Delay • Used $22/min (based on total non-fuel operating costs averaged for a narrowbody jet)

Cost of 1 Extra Mile Flown (expressed in $/min)• Equivalent to $18/min (based on 2004 fuel cost average for a narrowbody in cruise at

$1.25 / gallon)

Cost of a [Narrowbody] Cancellation• US carrier-reported average cost was $4,500 in ’94 which equates to $6,000 per

cancellation in 2004

Cost of a [Narrowbody] Diversion• Assuming 4 hrs extra block time and a $2,500 hourly operating cost for a narrowbody,

we get $10,000 per diversion

Sources: OIG; BTS; MITRE CAASD; FAA APO; FAA OPSNet database

23

CATSCATSRR

Operational Response Index (ORI) Calculation

ORI = (Num_fl * Avg ∆dist * Avg_fuelburn * Fuel_cost +

Num_fl * Avg ∆time * Avg_nonfuel_oper_cost

Num_diversions * Avg_cost_of_diversion +

Num_Cancellations * Avg_cost_of_cancellation) / Num_fl

where:∆dist = average excess distance per flight (actual vs. flight-planned)

∆time = average excess block time per flight (actual vs. scheduled)

Avg_fuelburn = fuelburn for a generic narrowbody jet in cruise at FL330

Num_fl = daily number of OPSNet flights

Sources of data: FAA APO Lab; FAA ASPM

24

CATSCATSRR

Operational Response Index – Reality CheckComparison with $$ Quoted in Literature

For OPSNet flights:• Total “excess airline cost” for 2004 (all flights, all days) is $4.6B

• For a baseline “ideal” day (ORI = $150/flight):

if all days in 2004 were like it, total cost would have been $2.7B

Difference = $1.9B in direct operating costs

References show comparable excess-cost estimates:• “Some figures have indicated that the total average direct annual costs of the irregular

operations of ten U.S. major airlines for the period 1996-1999 have been about $1.9B” (M.Janic – TRB Report, 2003)

• “…the Air Transport Association estimated that delays cost the air carriers approximately $2.0B in direct operating costs in 1999”: OIG Report, 2000

• “The Air Transport Association's amount increases to nearly $5B when indirect costs and the value of passengers' lost time are included”: OIG Report, 2000 (extrapolation of our calculations to include indirect costs produces comparable numbers – AK)

• Total operating costs of delays: $1.8-2.4B in 1987-94: FAA APO-130, “Total Cost for Air Carrier Delay Report”, 1996

• 1999 total cost of disruptions estimated at $1.8B (Z.Shavell – Effects of Schedule Disruptions on the Economics of Airline Operations. In: Air Transportation Systems Engineering, 2001, Chapter 8).

25

CATSCATSRR

2005 Delay/Wx: Linear vs. Exponential TrendA Sign of a Worsening Delay Situation?

Exponential trendline: a better fit for 2005 Delay-vs-Weather Plot?

Linear trendline is a better fit for 2004 data

ASPM Arr Delay vs CWITI, 2004-2005 May-SepExcluding hurricane-impacted days

y = 0.832x + 30.734

0

50

100

150

200

250

0 50 100 150 200 250

CWITI, Normalized to 2004 average

Arr

Del

ay p

er F

light

, N

orm

aliz

ed to

200

4 av

erag

e

2005Linear (2005)

ASPM Arr Delay vs CWITI, 2004-2005 May-SepExcluding hurricane-impacted days

y = 47.823e0.0078x

0

50

100

150

200

250

0 50 100 150 200 250

CWITI, Normalized to 2004 average

Arr

Del

ay p

er F

light

, N

orm

aliz

ed to

200

4 av

erag

e

2005Expon. (2005)

Linear trendline Exponential trendline

26

CATSCATSRR

“Quiet Period” Monthly ASPM Delays vs. Ops1995-2005

10-Year Trend: NAS Delays vs Ops, "Quiet months" (Mar,Apr,Oct,Nov)

0

200000

400000

600000

800000

1000000

1200000

1400000

1600000

1800000

2000000

2200000

2400000

2600000

2800000

3000000

3200000

3887

061

3919

373

3949

738

4014

168

4043

193

4076

275

4097

886

4117

945

4130

877

4166

976

4197

472

4231

055

4254

236

4306

868

4331

362

4358

776

4390

865

4451

586

4499

093

4571

369

4662

789

Monthly ops

Tota

l mon

thly

del

ay m

inut

es v

s. S

ched

ule

27

CATSCATSRR

Terminal Capacity Degradation

Weather factor Available airport capacity, % nominal

• THUNDERSTORM 10

• HEAVY_SNOW 30

• HIGH_WIND (>30 kt)* 30

• HEAVY_RAIN 40

• LOW_VISIBILITY 70

• LOW_CEILING 70

• SNOW 70

• RAIN 70

• WIND (20-30 kt) 70

• NO_WEATHER 100

*sustained wind above 30 kt, higher gusts

28

CATSCATSRR

“Flows” and Actual TracksSimilarity