Estimating Flow Rates in Convective Weather ... - ATM Seminar · Estimating Flow Rates in...

Estimating Flow Rates in Convective

Weather: A Simulation-Based Approach

20 June 2019

James Jones, Yan Glina

DISTRIBUTION STATEMENT A. Approved for public release: distribution unlimited.

ATM Reinforcement Learning - 2

JJ 6/20/2019

This material is based upon work supported by the National

Aeronautics and Space Administration under Air Force Contract No.

FA8702-15-D-0001. Any opinions, findings, conclusions or

recommendations expressed in this material are those of the

author(s) and do not necessarily reflect the views of the National

Aeronautics and Space Administration.

© 2019 Massachusetts Institute of Technology.

Delivered to the U.S. Government with Unlimited Rights, as defined

in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any

copyright notice, U.S. Government rights in this work are defined by

DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use

of this work other than as specifically authorized by the U.S.

Government may violate any copyrights that exist in this work.


JJ 6/20/2019

• Collaborative Trajectory Options Program (CTOP) assigns delay and/or reroutes around one or more Flow Constrained Area-based airspace constraints in order to balance demand with available capacity

• NASA’s Integrated Demand Management (IDM) program is exploring ways to use CTOP to precondition demand into time-based metering programs at airports in the Northeast United States

• Estimates from strategic decision support systems (TFMS) may be inconsistent with delivery capability of tactical decision support systems (TBFM)

• Good estimates of airport/airspace capacity are needed to effectively control demand to the appropriate levels

• Proposed Approach

– Leverage reinforcement learning and integer programming to estimate airport and terminal airspace capacity

– Use fast-time simulation to evaluate performance of algorithms

Background and Approach

TFMS = Traffic Flow Management System TBFM= Time-Based Flow Management


JJ 6/20/2019

• If actual capacity exceeds planned arrival rate: may under-deliver

• Incur costs due to unused airspace capacity and unnecessary ground delays

Under-delivery

t

Capacity

UnnecessaryGround Delay

Cplanned

Cactual


JJ 6/20/2019

• If actual capacity is less than the planned arrival rate: may over-deliver

• May need tactical intervention (e.g. holding, miles-in-trail restrictions)

• Incur costs due to airborne delay, diversion, …

Over-delivery

t

Capacity

Cplanned

Cactual

Airborne Delay


JJ 6/20/2019

• When we over-deliver, the achieved throughput serves as a decent initial estimate for capacity (otherwise we would not be holding)

• This estimate is performed under operationally undesirable conditions, however, so it may not be sustainable

Baseline Capacity

t

Capacity

Cplanned

Cactual

Airborne Delay


JJ 6/20/2019

t

Capacity

• If we can adjust the Cplanned to produce comparable throughput while reducing holding to a manageable level, then the planned rate serves as a good estimate for the true capacity

Optimization Goal

Optimized Capacity Estimate

Cplanned

Cactual


JJ 6/20/2019

Example Traffic Management Initiative for this Paper: Airspace Flow Program (AFP)

• AFPs reduce demand by issuing ground delays for

flights planned to cross Flow Constrained Areas (FCAs)

– Acceptance rate = # aircraft that can cross FCA per hour

• AFP sets acceptance rates for each hour

• Flights wait based on their order in the schedule

• Example AFP:

• Large search space, determining “optimal” rates is very challenging

FCA1

FCA2

12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00

FCA1 59 59 59 57 58 63 61 62 60

FCA2 61 63 64 62 62 60 60 60 60

Can machine learning and integer programming help us identify the

appropriate solutions?

Acceptance rates


JJ 6/20/2019

Outline

• Background

• Modeling and simulation approach

• Optimization methods

• Results

• Summary


JJ 6/20/2019

Simulation Details

Inputs

CIWS/CoSPA Weather

Flight Schedule

Weather-Based

Capacity Constraints

Sector-Based

Constraints

Aircraft Separation

Constraints

Runway Separation

Constraints

NASPlay Simulation Outputs

Ground Delay

Airborne Delay

Number of

Arrivals

Airborne Holding

Diversions

Cancellations


JJ 6/20/2019

Simulation Details

NASPlay provides

high fidelity

simulation of national

airspace but it is

difficult to generate

good solutions due to

large search space

Can we use

optimization methods

to expedite the

process?

Inputs

CIWS/CoSPA Weather

Flight Schedule

Weather-Based

Capacity Constraints

Sector-Based

Constraints

Aircraft Separation

Constraints

Runway Separation

Constraints

NASPlay Simulation Outputs

Ground Delay

Airborne Delay

Number of

Arrivals

Airborne Holding

Diversions

Cancellations


JJ 6/20/2019

• May 15, 2018: Airspace capacity in

Northeast was compromised due

to weather

• Case study: Estimate airport

capacity at Newark Liberty

International Airport (EWR) from

11:00am to 00:45am GMT

• NASPlay Baseline: no control

strategy (“do nothing”)

– 388 aircraft landed in total (number

of Arrivals = 34-40 per hour)

– 161 aircraft accrued at least 15

minutes of holding before landing

• Goal: Find AFPs that achieve

comparable throughput but reduce

holding

Case Study

AFP = Airspace Flow Program

EWR


JJ 6/20/2019

• Hourly rates need to be specified for each of three FCAs

• Each solution has 30 dimensions: (3 FCAs) x (10 hr)

• Optimization covers a 30-dimensional space

Case Study Search Space

15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 0:00

FCA1 14 14 14 15 15 17 17 16 16 17

FCA2 10 10 10 10 10 8 8 9 9 10

FCA3 13 14 14 13 13 12 12 12 12 10

FCA1

FCA2

FCA3

FCA = Flow Constrained Area


JJ 6/20/2019

Search Space Description

• Choose rates for each FCA from a normal distribution: 𝒓𝒊𝒕~𝑵(𝝁𝒊, 𝝈𝒊

𝟐)

• Search space is constrained so that rates do not change drastically each hour

– Hourly AFP rates are correlated to the previous hour

– FCA hourly rates sum to a target rate for the airport R

Hour 1 2 3 4 5 6 7 8 9 T = 10

FCA1 r11 r12 r13 r14 r15 r16 r17 r18 r19 r110

FCA2 r21 r22 r23 r24 r25 r26 r27 r28 r29 r210

FCA3 r31 r32 r33 r34 r35 r36 r37 r38 r39 r310

FCA = Flow Constrained Area


𝑠. 𝑡.

𝑖

𝑟𝑖𝑡 = 𝑅 ∀𝑡


JJ 6/20/2019

Optimizing Airspace Flow Programs

• X = array of AFP rates

– X is 30-dimensional in above example

• y = Number of Arrivals

– Compute by running NASPlay simulation and calculating number of aircraft that landed per hour (# arrivals)

• Want to identify the AFP rate with maximum number of arrivals while achieving low holding levels

AFP Rates (X)

# o

f A

rriv

als

(y) Global

maximum


15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 0:00

FCA1 14 14 14 15 15 17 17 16 16 17

FCA2 10 10 10 10 10 8 8 9 9 10

FCA3 13 14 14 13 13 12 12 12 12 10


JJ 6/20/2019

Outline

• Background on demand / capacity balancing



– Reinforcement Learning (RL)

– Integer Programming (IP)

– Integer Programming with Random Exploration (IPRE)

• Results

• Summary


JJ 6/20/2019

Major Categories of Machine Learning

Supervised Learning Unsupervised Learning Reinforcement Learning

Approach• Train model given labeled

(truth) dataset

• Train model given unlabeled

dataset

• Train model based on trial and

error

Example

Applications

• Classification

• Pattern recognition

• Data fusion

• Clustering

• Anomaly detection

• Dimensionality reduction

• Game theory / strategy

• Optimal control

Example

Techniques

• Support Vector Machine

• Random Forest

• Convolutional Neural Network

• Principal Component

Analysis

• K-means clustering

• Dynamic programming

• ε-greedy algorithms

• Monte Carlo Tree Search

Air Traffic

Management

Example

Offshore Weather Prediction Trajectory Clustering ACAS X Collision Avoidance

-400 -300 -200 -100 0 100 200 300

-300

-200

-100

0

100

200

300


JJ 6/20/2019

Major Categories of Machine Learning

Supervised Learning Unsupervised Learning Reinforcement Learning

Approach• Train model given labeled

(truth) dataset

• Train model given unlabeled

dataset

• Train model based on trial and

error

Example

Applications

• Classification

• Pattern recognition

• Data fusion

• Clustering

• Anomaly detection

• Dimensionality reduction

• Game theory / strategy

• Optimal control

Example

Techniques

• Support Vector Machine

• Random Forest

• Convolutional Neural Network

• Principal Component

Analysis

• K-means clustering

• Dynamic programming

• ε-greedy algorithms

• Monte Carlo Tree Search

Air Traffic

Management

Example

Offshore Weather Prediction Trajectory Clustering ACAS X Collision Avoidance

-400 -300 -200 -100 0 100 200 300

-300

-200

-100

0

100

200

300


JJ 6/20/2019

e-Greedy Approach

Sampling entire space is intractable due to:

• Size of search space

• Time required to run each simulation

global maximum

Alternative is to search for optimum

using combination of two strategies:

ExplorationFind new point by random

sampling

ExploitationFind new point by fitting model f

to current sample points and

taking optimum

• Earlier iterations emphasize exploration

• Later iterations emphasize exploitationFor iteration 𝒊, 𝑷 𝒆𝒙𝒑𝒍𝒐𝒓𝒂𝒕𝒊𝒐𝒏 =

𝟓

𝟒 + 𝒊

X

Y

If sampling entire space were tractable, we

would simulate all possible points and

find global optimum

X

Y

y = f(x)

X

Y

AFP rate settings

# o

f A

rriv

als


JJ 6/20/2019

• Run 5 iterations with random sampling to initialize RL algorithm

• Run subsequent iterations with RL-selected samples

– RL algorithm selects exploration or exploitation

– If exploitation is selected:

• Surrogate model is generated using gradient tree boosting

• Surrogate model predicts number of arrivals for 100,000 randomly sampled points

• Choose sample point that maximizes predicted number of arrivals while limiting holding

Reinforcement Learning (RL) Details

15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 00:00

FCA1 14 15 15 16 16 16 15 16 17 16

FCA2 8 9 9 10 10 10 10 11 11 11

FCA3 14 13 13 12 12 12 13 10 9 10

• # Arrivals

• # Holdings

Set of acceptance rates

NASPlay simulation

Performance

metrics


JJ 6/20/2019









• # Arrivals

• # Holdings


NASPlay simulation

Performance

metrics

15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 00:00

FCA1 14 15 15 16 16 16 15 16 17 16

FCA2 8 9 9 10 10 10 10 11 11 11

FCA3 14 13 13 12 12 12 13 10 9 10


JJ 6/20/2019









• # Arrivals

• # Holdings


NASPlay simulation

Performance

metrics

15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 00:00

FCA1 14 15 15 16 16 16 15 16 17 16

FCA2 8 9 9 10 10 10 10 11 11 11

FCA3 14 13 13 12 12 12 13 10 9 10


JJ 6/20/2019









• # Arrivals

• # Holdings


NASPlay simulation

Performance

metrics

15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 00:00

FCA1 14 15 15 16 16 16 15 16 17 16

FCA2 8 9 9 10 10 10 10 11 11 11

FCA3 14 13 13 12 12 12 13 10 9 10


JJ 6/20/2019









• # Arrivals

• # Holdings

Reinforcement Learning


NASPlay simulation

Performance

metrics

15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 00:00

FCA1 14 15 15 16 16 16 15 16 17 16

FCA2 8 9 9 10 10 10 10 11 11 11

FCA3 14 13 13 12 12 12 13 10 9 10


JJ 6/20/2019

• Epsilon greedy (RL) approach is one way to derive good acceptance rates

– Model-free approach

– Initial solutions are likely to perform poorly

– May require many simulations to provide sufficient data to train the model

• Integer programming (IP) offers an alternative approach to identifying acceptance

rates

– Model-based approach

– Uses estimates of demand and capacity

– Initial solutions are likely to outperform e-greedy if the estimates for objective function and

constraints are accurate

• Two variants

– IP: Stochastic Integer Programming

– IPRE: Use Stochastic Integer Programming for baseline solution, then Random Exploration

(RE) to produce alternative acceptance rates

Alternative Method: Integer Programming


JJ 6/20/2019

Terminal AFP Planning Model for Integer Program

TMI = Traffic Management Initiative

15:00 16:00 17:00 18:00

FCA1 14 15 15 16

FCA2 8 9 9 10

FCA3 14 13 13 12

15:00 16:00 17:00 18:00

FCA1 15 13 15 14

FCA2 9 11 12 12

FCA3 14 13 10 12

Perturb

Randomly

Inputs:

• List of flights and scheduled times of arrival

• System cost of ground and air delay

Problem 1 (Estimation)

Quantify Model Uncertainty

• Estimate Airspace Capacity in

nearby en route airspace

• Generate Airport Demand based on

historical data

Inputs:

• Convective weather models

• Historical airspace throughput

Problem 2 (Optimization)

Objective: Minimize total expected cost of

ground and air delay

• All flights scheduled to take off must either

take off or be delayed on the ground

• All scheduled arrivals that have taken off

must land at their scheduled times or be

delayed in the air

• Number of arrivals cannot exceed the

assigned capacity (estimate based on

nearby airspace capacity)

• Decision Variable: Number of flights that

take off in a given time period

AFP rates


JJ 6/20/2019

Outline

• Background on demand / capacity balancing



– Reinforcement Learning (RL)

– Integer Programming (IP)

– Integer Programming with Random Exploration (IPRE)

• Results

• Summary


JJ 6/20/2019

• “Do Nothing” scenario resulted in 388

arrivals and 161 flights held for greater

than 15 minutes

Results


JJ 6/20/2019



than 15 minutes

• We’d like to achieve a comparable

throughput level with lower holding

Results


JJ 6/20/2019



than 15 minutes



Results

Ideal

performance


JJ 6/20/2019



than 15 minutes



• Typically, AFP programs identified

through random sampling will reduce

holding at the expense of the number

of arrivals

Results



JJ 6/20/2019



than 15 minutes






of arrivals

Results



JJ 6/20/2019



than 15 minutes






of arrivals

• Algorithm-based samples perform

considerably better than random

sampling

Results

AFP = Airspace Flow ProgramIP = Integer Programming

IPRE = IP with Random Exploration


JJ 6/20/2019



than 15 minutes






of arrivals



sampling

Results




JJ 6/20/2019



than 15 minutes






of arrivals



sampling

– The e-greedy algorithm outperforms other

candidates’ approaches

Results




JJ 6/20/2019



than 15 minutes






of arrivals



sampling

– The e-greedy algorithm outperforms other

candidates’ approaches

Results

Best

selected

point




JJ 6/20/2019

• Fit distributions to number of arrivals and holdings

Variance in Results

Holding increases with target rate on e-greedy algorithm but

no clear relationship for IPRE.e-greedy algorithm outperforms IPRE when rates

are comparable. A target rate of 37 flights/hour

performs best for both methods.

IP = Integer Programming



JJ 6/20/2019

• Reinforcement learning method demonstrates that effective balance of flight demand and

airport capacity is achievable

– Optimal strategy results in 98% of the original throughput with 76% reduction in holding

– Airport can sustain 33-41 arrivals/hr even when compromised with significant convective weather

Results Summary

Do-Nothing Best Selected


JJ 6/20/2019

Summary

• Approaches successfully applied to estimate flow rates in terminal airspace

– Demonstrated ability to efficiently find Airspace Flow Program (AFP) rates that provide high

throughput with low holding

– Resulting AFP provides a good proxy for airspace capacity at the arrival gate level

• Reinforcement learning is effective in optimizing over large search space when the

appropriate objective function and constraints cannot be captured in closed form

– Performance is significantly better than random selection

– Outperforms integer programming approaches

• Opportunities for future study

– Arrival/Departure balancing with traffic from neighboring airports

– Developing regional strategies to manage traffic through multiple airports

– Mapping resulting algorithmic solutions to more intuitive and interpretable control strategies


JJ 6/20/2019

Acknowledgements

• We would like to thank our sponsors William Chan, Paul Lee and Nancy Smith of NASA Ames for providing excellent technical oversight, discussions and instrumental feedback in the examined area.

Back-up


JJ 6/20/2019

Variance in Results

• e-greedy approach provides more consistent performance with respect to throughput

• IPRE outperforms IP when the target rate is not well aligned, but performance degrades slightly with good rate alignment

Selection

Method

Individual Throughput and Holding Performance

Number of

Arrivals

Mean (STD)

Number of

Holds Mean

(STD)

Maximum

Arrivals Case

Arrivals/Holds

IP

Solution

IPRE 35 354.1 (4.99) 35.0 (14.34) 362 / 19 357 / 89

IPRE 36 357.5 (4.66) 33.0 (15.16) 365 / 52 364 / 77

IPRE 37 365.2 (6.25) 32.3 (14.64) 371 / 45 372 / 44

e-greedy 35 365.4 (2.37) 20.3 (15.25) 371 / 15 N/A

e-greedy 36 365.8 (1.22) 18.2 (12.59) 368 / 51 N/A

e-greedy 37 371.9 (14.15) 42.8 (10.87) 381 / 39N/A

Selection method

Aggregate Throughput and Holding

Performance

Number of

Arrivals Mean

(STD)

Number of

Holds Mean

(STD)

Maximum

Arrivals Case

Arrivals/Holds

Random 356.27 (9.50) 62.75 (46.96) 374 / 145

IPRE 359.14 (6.80) 33.34 (14.74) 371 / 45

e-greedy 367.70 (4.90) 27.10 (16.92) 381 / 39

IP = Integer Programming



JJ 6/20/2019

• Epsilon greedy (Reinforcement Learning) approach is one way to derive good

acceptance rates





rates







JJ 6/20/2019

• Epsilon greedy (Reinforcement Learning) approach is one way to derive good

acceptance rates





rates






Will the IP approach yield better solutions overall?


JJ 6/20/2019

Terminal AFP Planning Model for Integer Program

Inputs:

• List of flights and scheduled times of arrival

• List of historical days for TMI intervention

Inputs:

• Convective weather models

• Historical airspace throughput

Problem 1 (Machine Learning)

Quantify Model Uncertainty

• Estimate Airspace Capacity in

nearby en route airspace

• Generate Airport Demand based on

historical data

FCA capacity

estimates

and demand

scenarios

TMI = Traffic Management Initiative


JJ 6/20/2019

• Traffic Flow Impact* translates convective weather blockage forecast at a resource to a percentage known as permeability

• Idea: Use the permeability forecast at an upstream en route resource as a proxy for permeability through terminal FCAs

Initial Capacity Estimate for Integer Program

Traffic Flow Impact

* M. Matthews, M. Veillette, J. Venuti, R. DeLaura and J. Kuchar, "Heterogeneous Convective Weather Forecast

Translation into Airspace Permeability with Prediction Intervals.," Journal of Air Transportation, pp. 1-14, 2016.


JJ 6/20/2019

• Traffic Flow Impact* translates convective weather blockage forecast at a resource to a percentage known as permeability

• Idea: Use the permeability forecast at an upstream en route resource as a proxy for permeability through terminal FCAs

• Multiply en route permeability by target rate Cjt on terminal FCA and normalize values so that the flow rate through the 3 FCAs is equals target rate for airport C

Initial Capacity Estimate for Integer Program

Traffic Flow Impact

𝑅𝑗𝑡 = 𝑃𝑒𝑟𝑚𝑗𝑡 ∗ 𝐶𝑗𝑡 ∀𝑗, 𝑡

𝑠. 𝑡.

𝑗

𝐶𝑗𝑡 = 𝐶 ∀𝑡

* M. Matthews, M. Veillette, J. Venuti, R. DeLaura and J. Kuchar, "Heterogeneous Convective Weather Forecast

Translation into Airspace Permeability with Prediction Intervals.," Journal of Air Transportation, pp. 1-14, 2016.

Date post:	14-Jul-2020
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

Estimating Flow Rates in Convective Weather ... - ATM Seminar · Estimating Flow Rates in...

Documents