Using Program Generation to Support V&V in the Next Generation Air
Transportation System Ewen Denney
(joint work with Xavier Bouyssounouse
Johann Schumann
Karen Tung-Cate)
August, 2007
NGATS
• US air transportation system (airports and airspace) rapidly reaching capacity
• Major effort to develop integrated solutions for a safe, efficient, and high-capacity airspace system– Dynamic Airspace Configuration– Traffic Flow Management– Separation Assurance – Safety analysis
Trajectory Prediction
Trajectory Prediction predicts the future progress of individual aircraft on the basis of
• the current aircraft state,• estimates of pilot and controller intent,• expected environmental conditions, and• models of aircraft performance and procedures
• What is a trajectory?– Actual flight path (“ground truth”) seldom known– Tracking data: radar, altitude, weather– Intent: flight-plan (pilot/ATC), way points, jet-routes– Route constraints
TP Client Applications
• Flight Plan Distribution and Sector Loading: prediction of the Center Sequence (i.e., Air Traffic Control facilities to be transited)
• Flight Data Processing Systems: estimation of hand-over conditions, collision avoidance
• Basic Decision Support Tools: monitor flight plan and tracking data to advise controllers of suggested metering/sequencing and potential conflicts
• Advanced Decision Support Tools: directly suggest solutions (e.g., instructions for conflict resolution, accurate prediction of cruise-descent and climb-cruise profiles in presence of tactical maneuvers)
Sources of Uncertainty
• Data errors: incomplete, inaccurate, corrupted or untimely data (e.g., flight plan and radar track data inputs, airspace/navigation database)
• Modeling errors: simplifications, omissions and uncertainty in the mathematical models used in the system (e.g., aircraft performance modeling database)
• Intent errors: unpredictability of pilot and controller intent
Tracking Database
• Trajectory info in NAS– Track reading (radar
position and barometric altitude): noisy measurement of actual AC movement
– Flight plan: “intended path” of the AC
• Initial flight plan• Amendments to flight
plan (often not in system)
actual flightpath
flightplan
radar reading
CM_SIM database assembles aircraft trajectory data from multiple noisy sources: raw, cleaned, and derived data, weather, pilot intent, etc.
Tracking Data
• AC_DATA: radar tracking data– Noisy radar readings (every 12 secs)
• ADD_FLIGHT_PLAN: flight plan data• AMEND_FLIGHT_INFO: additional intent data• weather and waypoint data are separate
AMEND_FLIGHT_INFO 15 FDX1169/AFW.0015 000000002000 -NS- F -NS- AFW/1043 GRABE/1049 OKMX/1049 GRABE012038/1052 TIKYS260015/1055 EAKER/1057 ATOKA313002/1059 MLC304033/1104 LOBOE085022/1104 LOBOE058028/1106 69194/1106 69195/1107 TUL200033/1107 69184/1109 TUL/1111ADD_FLIGHT_PLAN 15 AAL8/PHNL.0016 ZFW AAL8 0016 -NS- -NS- 1114 H/B763/QHNL./.GATES247077..3254N/9702W..KDFW TXO221069 1100 390 455 -NS- ESTIMATED_FP ATC_ARRIVALAC_DATA 52 BKA302/ATL.0002 806.6335 540.9157 333850.00N 0913712.00W 28100 0.00 400.65 0.00 259.26 0.00 41.29 1 -NS- 28 N N F F ZFW 0.0000 403.0000 259.2621AC_DATA 52 AAL8/PHNL.0016 217.1337 539.1776 333634.00N 1032302.00W 39000 0.00 450.52 0.00 97.61 0.00 52.69 2 93 93 N N F F ZFW 0.0000 510.0000 97.6074ASSIGN_CONFIG 15 AAL8/PHNL.0016 CM KDFW SOUTH_4_VFR
Trajectory Synthesis
• Mature discipline with complex algorithms to– integrate equations of motion, aircraft models– analyze weather– interpret pilot intent
• Can represent trajectories using Generalized Altitude Profile– Domain-specific language for trajectories: GenAlt– Input request constrains possible trajectories– TS tool outputs parametric trajectory and point sequence of
states
V&V Challenge
• TS tool highly safety-critical but difficult to test– Complex algorithms– Lots of domain knowledge– Many different “kinds” of trajectories– Many different “kinds” of requests
• Huge database of raw data– US national airspace tracking data– Weather data– Aircraft manufacturer models– Field tests
• Need flexible data selection and analysis mechanism• Want to write analysis queries at high-level• Goal:
– develop DSL for trajectory analysis– use program synthesis to generate analysis code
V&V Goals
• Validate TS algorithm– Test with representatives and edge-cases of important trajectory
classes • Populate trajectory database (based on tracking data, etc.)
– Contractor building database infrastructure– Use for testing/assessment of trajectory prediction algorithms
• Determine faulty assumptions in models• Compare trajectory predictors• Initial analysis techniques
– Clustering wrt.• Point data• Trajectory data
– Change-point detection• Decompose trajectories into segments• Can use as features for clustering
Clustering
• Given: data-points that originate from 3 different sources/processes
• Data points are:– iid– Gaussian distributed
Problem: you don’t see the colors
Clustering
• Clustering task: Estimate the parameters µ, such that
P(x| µ,) is maximal
Clustering also produces most likely class-assignment (“coloring”) of data points and class frequency .
Usually, an iterative machine-learning algorithm (EM, k-means) is used
AutoBayes• Customizable Bayesian data
analysis code generator• Synthesizes custom algorithms
– C/C++, Matlab, …– Different model domains:
MLE, MAP, mixture models, change-point detections, etc.
– Different PDFs– Different algorithms (EM, k-
means, kd-trees)– Analysis or sampling (Monte
Carlo simulation)– Can be extended with new
algorithms
• Input is statistical model of analysis problem
Example Data Model
• Specify:– num. classes– num.
datapoints– distribution of
data– goal function
Trajectory Features• Each trajectory (i.e., sequence of AC
locations over time) can be characterized by a set of scalar features: – % in level flight, climbing, descent– Endpoints– Min distance to other AC– Min/max/mean v/speed, g/speed, g/accel,
Mach/CAS/TAS– Diff raw and cleaned data– Turning maneuvers:
• Total, max, sum• Altitude and heading• Could smooth wrt. turbulence
Clustering Analyses
• Simple clustering– Cluster into disjoint sets of ACs: each represents
set of AC trajectories– Cluster into disjoint time segments: each
represents all AC info during that segment– Cluster into disjoint sets of ACs at specific time
point
• Cluster on– Tracking data– GenAlt requests – Synthesized trajectories
Example Analysis
• Approx. 300x300nm around DFW
• Approx. 1 hour of data
• 215 AC trajectories• Cluster wrt. several
features• Blue and cyan are
landing and approach to DFW
Example: Endpoints
• DFW forms one cluster (cyan)
• Other points not obviously clustered– through flights– cut-off by time
Example: Close AC-distance
• AC in descent to/departure from DFW: small distance to airport (?) (d~1.2nm)
• En route flights which exhibit small minimal distances to other AC (d~0.97nm)
Change-point Detection
• Data analysis technique for time series• Detect (rapid) changes in
– Intensity– Noise– etc.
Mach/CAS Transition
• Mach number– Velocity relative to sound
(M = v/a)– Depends on altitude
• Calibrated airspeed (CAS)– Indicated airspeed,
corrected for errors• Mach/CAS transition point
– During ascent: constant CAS then constant Mach
Climb/Descend Profiles
mach
cas
alt
Generate code to find transition points: Mach/CAS, climb/cruise, …
• Accurate AC models for climb profiles are necessary for accurate trajectory prediction
• Mach/CAS profiles can be extracted from radar trajectory data – Convert ground speed to
Mach/CAS profile– Cluster according to
Mach/CAS values for different aircraft types and maneuvers
– Change-point detection in climb/descend maneuvers
Finding Transition Points
• Pre-process to find “climbs”
• Find where transition from slope to constant occurs
• Noisy data• Should also find
best Mach/CAS transition
const nat N, t_0; const double sigma_sqdouble a,b,c.data double mach(0..N-1).mach(t) ~ N( if(T < t_0)? a+b*t : c, sigma_sq).max pr(mach|{a,b,c}) for {a,b,c}.
a = 0.69428b = 0.0072091c = 0.44532t_0 = 35Red: actual trajectory
Blue: estimated profile
Next Steps• Characterize important trajectory features• Generate test cases for TS from clusters• Combine extracted transitions and parameters with other
features• Covariance analysis wrt. AC types• Extension of statistical model by domain knowledge
(“priors”)– e.g., airport locations
• Incorporation of other data– flight plans– weather data (e.g., for Mach/CAS)
• Relating physical trajectories (tracking data), synthesized trajectories, and TS requests
• Recursive clustering algorithms (large data sets)
Longer Term
• Domain-specific language to describe– analysis types – trajectory features– data filters
• weather conditions• changes of flight plans• specific airports, flight types, flight segments
• Integrate generator with– trajectory database– TS tool