978-1-4244-6618-4/10/$26.00 ©2010 IEEE 2.C.2-1

EVALUATION OF SEPARATION PERFORMANCE WITH ADS-B AT THE PHILADELPHIA KEY SITE

Michael W. Castle, Aurora Sciences, LLC, Washington, DC Tan Trinh, Colin Mayer, and Christine Parry, Massachusetts Institute of Technology Lincoln

Laboratory, Lexington, MA

Abstract The Federal Aviation Administration’s (FAA)

Surveillance and Broadcast Services (SBS) Program Office has implemented one of the first enablers of the Next Generation Air Transportation System (NextGen) by installing a ground infrastructure that supports the Automatic Dependent Surveillance - Broadcast (ADS-B) data service. One of the main applications for this infrastructure is to enhance air traffic control (ATC) separation through more frequent and accurate data about aircraft. As part of the initial assessment activities, the ADS-B data was incorporated into four different automation platforms in four different key sites. One of these key sites was the Philadelphia Terminal Radar Approach Control (PHL TRACON) which uses Standard Terminal Automation Replacement System (STARS) to process and display radar data to controllers. The Philadelphia STARS was updated to accept ADS-B data in its native format. Beyond just accepting new surveillance data, however, PHL TRACON is the first location that went operational where ATC used system track positions (so called- “fused” targets) published to the ATC display to allow controllers to separate aircraft, a paradigm shift from current operations where controllers use radar “plot” positions.

As part of the SBS evaluation, a working group was created that worked specifically to evaluate the end-end performance of separation services with the new data flow - ADS-B avionics, ADS-B surveillance service on the ground, and the updated STARS software. The evaluation used a comparative approach with current monopulse secondary surveillance radar (MSSR) systems as a baseline; if the new system performed as good or better than MSSR systems in separation applications, then the system was acceptable for operation. This paper describes the efforts of the SBS working group to evaluate the operational performance in Philadelphia, including the metrics used, assumptions made,

operational scenario development, and results from several analyses, including simulation, flight testing, and targets of opportunity operating in the TRACON airspace.

Introduction NextGen systems are being implemented that

are changing ATC surveillance and aircraft separation services. In the past, state data on aircraft (position, altitude, speed and heading, and identification) presented to ATC originated exclusively from Primary and Secondary Surveillance Radar systems in the US National Airspace System (NAS). Now, two new surveillance systems, ADS-B and Wide-Area Multilateration, have been given Initial Operating Capability (IOC) designations at 6 key sites (WAM in Colorado and Juneau, Alaska, and ADS-B at Louisville TRACON, Philadelphia TRACON, and the Anchorage and Houston ARTCCs) to provide surveillance data to ATC for separation services.

To execute this rollout of new surveillance technologies, the FAA created the SBS Program in September 2005 to develop a multi-segment, life-cycle managed, and performance-based strategy that aligns with the NextGen and generates value for the NAS.

In June 2006, the SBS program office created the Separation Standards Working Group (WG), to develop and evaluate the technical aspects of the integration of ADS-B systems into the operations, procedures, and separation standards used in the NAS.

Scope of Work This report summarizes the FAA SBS Program's

performance evaluation of the separation of targets where fused radar and ADS-B data is provided to the ATC display STARS. The purpose is to document the performance analysis of the end-to-end ATC system at the Philadelphia key site, as a stand-in for

2.C.2-2

any generic STARS site, to support the request for approval to use current terminal separation procedures documented in FAA Order 7110.65.

This evaluation is distinct from other efforts in the SBS program to verify and validate that the system performs in accordance with specifications (System Test, Operational Test, Automation Requirements) or the Safety Risk Management Panel’s evaluation of hazards. These inputs were worked in parallel as part of the approval request process within SBS.

In the long term to enable the reduction of separation standards, the ADS-B surveillance system cannot use a comparative approach, and will need to be evaluated against a target level of safety. For example, an effort is currently underway to evaluate the safety of separating aircraft by 3 NM in enroute domains.

It should be noted that the material presented here is not the full evaluation done by the WG. Other aspects of the system that require more setup and understanding from a technical perspective were omitted for space considerations.

End-End System Avionics

Onboard aircraft, two systems are relevant to the evaluation of separation standards:

• the source of state vector data (state vector data can be defined as position and velocity of the aircraft, the time at which that information is applicable, and the associated integrity and accuracy of that data), and

• the ADS-B transmit subsystem that assembles, processes, formats, and transmits the data over RF channels.

The FAA rulemaking for ADS-B “Out” defines the minimum level of performance for several key parameters that ADS-B avionics systems must achieve in order to permit ATC to use the data for separation.[1] The technical performance requirements for all components of the end-end system are defined in the SBS final program requirements (fPR) document. [2] In the analysis to follow, the minimum performance is modeled in simulations to ensure that the system can support the worst-case end-to-end specified requirements.

In addition to the performance required for separation, avionics standards govern the technical characteristics of ADS-B systems. The ADS-B Rule defines RTCA standards for 1090 Extended Squitter (1090ES) [3] and Universal Access Transceiver (UAT) [4] as required for separation services.

Note that the analysis done in this report is based on RTCA DO-260A and DO-282A certified systems. However, the performance characteristics of these systems were found to be identical to the expected performance of the more current “B” versions of these links. These systems will be evaluated when new avionics systems are certified to these standards to ensure this assumption holds true.

Ground Infrastructure In August 2007, the ITT Corporation was

selected by the FAA as the prime contractor to provide ADS-B data being broadcast from aircraft in Enroute, Terminal, and Surface domains. This data is being used as surveillance data input to ATC in 4 key-sites currently. ITT Corporation is building a NAS-wide network of ground stations they own and operate, and the FAA pays subscription charges for the ADS-B data that is supplied to air traffic control facilities.

Because of the performance based specifications and the service acquisition, the details of how the ground infrastructure is architected and constructed are not pertinent to the analysis. The important details for the evaluation of separation services is whether the system can be shown to support the requisite level of performance given the technical approach discussed below.

Automation STARS implemented in Philadelphia uses

“fusion” of multiple surveillance data inputs, including multiple radars and ADS-B, to form a target position on the ATC display. STARS (and most other automation systems) have used fused track positions in their safety functions in the past. However, using this fused track position to indicate aircraft positions on a terminal display had not been accomplished prior to the SBS IOC for CARTS in Louisville, which occurred in November 2010.[5]

This evaluation was done on the horizontal separation between targets. In STARS, system tracking is divided between the horizontal plane, which utilizes an Interacting Multiple Model Kalman

2.C.2-3

filter architecture, and a separate tracker to estimate vertical motion. The details of this implementation are not discussed in this paper.

The system track contains the track's computed horizontal position, velocity and turn rate, and the track's established altitude. The display uses that information, along with an extrapolation algorithm, to position the track on the relative to the current time and update that position each display update interval (e.g. at a 3 second adaptable display update rate).

Accuracy from the STARS lateral IMM is used to determine the horizontal separation rules that apply to each aircraft. STARS uses the symbology sizes as specified by SBS requirements documents to determine what to present to ATC indicating whether terminal, en route, or non-radar separation applies.

Technical Approach ICAO documentation recommends two

approaches to assessing the safety of separation standards: comparing performance with a reference system (comparative analysis) or evaluating system risk against a threshold (target level of safety analysis). [5] The approach taken by the FAA is to conduct a comparative analysis of the ADS-B systems to current radar surveillance systems. The baseline used for the modeling is monopulse secondary surveillance radar (MSSR) performance, which typifies the best performance available to ATC from radar systems today (as opposed to sliding window secondary surveillance radar (SWSSR) or primary radar systems).

The initial objective of the SBS Program was to obtain approval for the current set of NAS separation standards using ADS-B as a source of surveillance data. This report documents the evaluation of the performance of these current terminal separation operations with ADS-B data in STARS automation.

Radar data supporting current terminal separation operations is presented on the display unaltered from the data reported by the radar in slant range and azimuth. No extrapolation is made to account for system latencies. The system deployed at Philadelphia, and expected to be deployed to future STARS sites, shows targets on a stereographic (x,y) plane whose positions are extrapolated to account for system delays and refreshed on the display on a periodic basis.

Separation Error Metric The Separation Standard WG uses horizontal

separation error as the principal metric in its evaluations; more precisely, separation error will be used as a key measure to determine whether the performance of ATC separation services with ADS-B data is equivalent to that with radar systems in operation today.

Horizontal separation error is the difference, ignoring altitude, between the true separation of two aircraft in the airspace and the separation that is displayed to an air traffic controller at any given time.

Figure 1 shows what is meant by horizontal separation error (hereafter referred to as separation error), which is denoted by Esep. The top part of Figure 1 depicts the real separation experienced by the aircraft. The middle section shows how the symbols (depicted as radar extents – not as it appears in the new PHL system) would appear on an ATC display, where showing the measurements of aircraft positions are published at a variable time after the data is relayed to ATC and the necessary processing has been performed by ground surveillance systems. The bottom part of Figure 1 shows how Esep is calculated based on these two separations.

Figure 1. Depiction of Actual Separation (Top),

Reported Separation (Middle), and the Calculation of Separation Error (Yellow Box)

Ideally, a model of separation error would calculate the error nearly continuously and would randomly choose a large sampling of display times to provide a continuous measurement.

The WG’s evaluation is intended to validate that the new ADS-B sensor data will perform equivalently

2.C.2-4

to or better than radar data used in current systems for ATC separation services. ATC personnel use the data on the display to make decisions about how to control traffic; controllers monitor the apparent aircraft separation on the display to decide when to issue commands and what commands should be given. For this, the controller is constantly monitoring how far apart the symbols representing aircraft are separated now and how far apart they will be separated in the future.

Separation error can be thought of as a measure of the accuracy of these symbols on the ATC display, and it is a crucial decision making factor for personnel performing separation services.

Another useful metric is horizontal position accuracy. The error related to this metric measure how accurately a given aircraft is displayed to ATC compared to the aircraft’s real position. However, this was not used (except in cases where only one aircraft is present in the scenario) as the principal metric because this metric unfairly penalizes current radar performance for characteristics that are deemed safe for operation; current radar systems exhibit biases with respect to a given aircraft’s position. The separation indication on the ATC display is accurate because the bias is common to both aircraft targets in most cases of separation when a single sensor is updating the targets.

For example, imagine two different systems providing position data to ATC. The first system has perfect accuracy, but the individual aircraft targets are updated asynchronously and probabilistically; for example, each target is updated every 3 seconds, but on different schedules and with an individual probability of updating each time of 50%. The second system has positional accuracy errors that consist of a (relatively) large bias but a small random variance around that bias, but the targets are updated periodically on a 6-second sweep (similar to current terminal radar). The first system would likely not be acceptable to a controller charged with providing separation services because of the independent, uncorrelated movement of the targets, even though there are no positional errors. The second system is that currently used by ATC and the separation between aircraft displayed to the controller is deemed accurate enough to maintain safety.

Simulation

Approach and Assumptions SBS and FAA’s Terminal Automation

Engineers worked collaboratively to execute and evaluate simulated scenarios of separation performance to stress the minimum conditions in the requirements for implementation of a generic STARS site. The goal was to evaluate performance using minimum acceptable parameter settings in order to ensure the system supported separation operations even when getting minimum performance from each of the component subsystems.

The performance assessment is based on comparison of the proposed new system’s performance to the current safe system’s performance. The baseline used for comparative analysis is a single monopulse radar system providing slant-range and azimuth on the terminal display. Separation error statistics from the displayed positions are compared with separation error statistics from the baseline display. Four aircraft scenarios with varying speeds and geometries are utilized in an effort to bound the performance throughout the terminal area. The spectrum of scenarios allows the analyst to weigh the benefits of fusion against any possible deficiencies as compared to the baseline.

The main assumptions applied when choosing the simulation scenarios and error models are as follows:

• The terminal radar is an Airport Surveillance Radar – 9 (ASR-9) with a MSSR.

• The En-Route radar uses a SWSSR (Sliding Window Secondary Surveillance Radar), a lower performing radar system.

• All aircraft are Mode-C equipped. Targets in automation with no associated altitude incur additional error due to difficulties converting from radar to system plane coordinates and were analyzed separately in the WG evaluation

• All aircraft are equipped with Mode S transponders. Modes S transponders have smaller range errors compared to Air Traffic Control Radar Beacon System (ATCRBS) transponders. It is assumed the

2.C.2-5

majority of aircraft will be Mode S equipped in the TRACON airspace.

ADS-B Error Modeling The ADS-B model parameters are summarized

in Table 1. ADS-B surveillance reports are modeled using the minimum allowable accuracy of NACp = 8 (95%<125 ft in either X/Y direction). The update interval is 3 seconds, to reflect the slowest data rate expected from a throttled feed from ITT’s ADS-B ground infrastructure.

Table 1. ADS-B Error Model Parameters

Source ADS-B/1090ES NACP 8 95% X/Y Error (ft) 125 Update Interval (sec) 3.0

Radar Error Modeling Two radars were used in the simulation; the

salient characteristics are summarized in Table 2.

Table 2. Summary Characteristics of the Radars Used in Simulation

Characteristic PHL QIE

Type ASR-9 /Mode S

Long-Range SWSSR

Sweep Period (sec) 5 13 Standard Deviation of Azimuth Jitter (°) 0.06 0.23

Azimuth Bias (when applied) 1 ACP 1 ACP

Two major drivers of performance were

explicitly investigated in the simulations: the effect of radar bias on the results and the effect from having different levels of sensor inputs into the fusion tracker. Four permutations were performed for each factor in the 3 NM scenarios, for a total of 64 cases. The bias errors include the sensor azimuth bias, range bias, and X/Y registration bias. The values were fixed to specific values that designed to bound the system performance between the best case (all 3 bias errors are zero or cancel out) and the worst case, where bias values combine to arrive at the largest possible error.

Table 3 summarizes the bias and sensor diversity for each run. Runs 1 and 2 had no bias errors, while runs 3 and 4 had maximum biases. Runs 1 and 3 had only the PHL radar, while runs 2 and 4 had PHL and QIE contributing to the tracks. In all 4 runs, ADS-B is also an input into the fusion tracker.

Table 3. Runs and Biases

Run 1 Run 2 Run 3 Run 4

Bias None None Max Max Radars PHL PHL + QIE PHL PHL + QIE

Simulation Analysis Methodology

Position error and separation error results are easily calculated using the simulated data provided by Raytheon. The data provides the precise location where a controller would view a target on the display and the true position of the aircraft at that time.

The fused tracker recalculates position every time sensor data is updated (either radar or ADS-B). However, these tracker positions must be extrapolated forward to the time of display updates. These positions at the time of display update are “extrapolated positions.”

There are two sources of data for analysis:

• Truth data output, “TruOut.” file: contains the truth and sensor reports for ADS-B and radar surveillance sources. The system X/Y coordinates for all data, range-azimuth for radar data, and lat-lon for ADS-B data are available, among others.

• Fused data records, “Extrap” file: contains displayed positions of fused data that has been extrapolated to the display update times. The positions are available in system X/Y coordinates.

The TruOut file is used to verify that the input data is modeled correctly and that the prescribed bias errors were applied. In addition, the TruOut file is used to approximate the position displayed in the baseline slant-range and azimuth display. Although STARS has the capability to perform dynamic radar registration corrections for range and azimuth, this capability was disabled in order to gauge the effect of radar biases on the STARS tracker accuracy.

The Extrap file contains the positions that are used to calculate the position and separation errors of the fusion prototype. The system coordinates are used to calculate position and separation errors. The position error is calculated by:

𝐸𝑝𝑜𝑠2 = �𝑥𝑒𝑥𝑡𝑟𝑎𝑝 − 𝑥𝑡𝑟𝑢𝑒�2 + �𝑦𝑒𝑥𝑡𝑟𝑎𝑝 − 𝑦𝑡𝑟𝑢𝑒�

2

and the separation error is calculated by

2.C.2-6

𝐸𝑠𝑒𝑝 = 𝑆𝑒𝑝𝑒𝑥𝑡𝑟𝑎𝑝 − 𝑆𝑒𝑝𝑡𝑟𝑢𝑒

where the “Sep” quantities are calculated by the simple distance formula between two points (for the two aircraft).

Figure 2 illustrates the complex timings involved in position error calculation. In the upper diagram, truth is represented by a red line. The green slashes represent sensor (radar or ADS-B) updates, every 4.8 seconds. The unfilled black circles represent tracker updates, which occur when new sensor information comes in. The blue line represents

the tracker extrapolation. Note that the extrapolation resets at every new tracker update, and attempts to predict position forward in time until the next tracker update. Lastly, the filled blue circles (which are on the extrapolated blue line) occur every 3 seconds, and represent the position that would be displayed on the screen. The lower diagram shows the error sampling for position error, which occurs every 0.5 seconds. Current truth is measured against what is displayed on the screen, so there is a latency associated with how often the screen is updated.

Figure 2. Illustration of Position Error Calculation Methodology

Note that the separation errors are not necessarily sampled at the time of radar or ADS-B position updates. Even if there was no position error, this can result in considerable separation errors when the separation between two aircraft is changing rapidly and the true separation is sampled some time before the displayed separation is updated. The true separation could be significantly different compared to the displayed separation because of the latency in position updates. This is one of the major advantages of fusion, which allows more frequent updates using the most recent data.

The baseline “single sensor” separation error is calculated using slant-range and azimuth that has been projected onto a Cartesian coordinate system. These X and Y coordinates of slant-range and azimuth were provided in the “TruOut” file.

Scenarios The 4 standard scenarios developed by the

Separation Standards Working Group are illustrated in Figure 3, and summarized below.

Figure 3. Four Standard Scenarios for Simulation Overlaid in Each of 4 Different Runs Conducted

2.C.2-7

• Holding: One aircraft maintains a standard oval holding pattern 35 NM from the PHL radar. The other aircraft flies past the holding pattern spaced 10 NM laterally. Both aircraft have speeds of 200 knots.

• Parallel overtake: One aircraft is flying a radial towards PHL at 100 knots. A second aircraft overtakes the first aircraft, and is on a parallel track 3 NM from the first, with a speed of 120 knots.

• In-trail tangential: The trailing aircraft flies 3 NM in-trail behind the lead aircraft. The aircraft fly past PHL with a minimum range of 20 NM at 400 knots.

• Merging: A trailing aircraft flies 10 NM in-trail behind a lead aircraft. A third aircraft merges into the stream of the first two aircraft, by performing a 90° standard rate turn. All aircraft fly at 180 knots.

Selected Results Figure 4 shows a four by four grid of the results

for the RMS separation error of the fused simulations compared to the MSSR baseline. The baseline results are shown as horizontal black lines across the graphs. The colored columns depict the 4 different cases of bias and sensor inputs to each of the scenarios (for a total of 64 permutations, all tolled). Where the columns are beneath the MSSR performance line, the tracker exhibits lower separation errors, and vice versa.

Figure 5 shows a drill-down of position error for the passing aircraft in the holding scenario for run 3 (max bias and only ADS-B and PHL contributing to

the track). The 4 cases are shown as cumulative distribution functions, and the statistics shown on the graph. The errors are all quite low, since the passing aircraft is a simple straight flight.

The results in the evaluation showed that the fused track output was equivalent to or better than the MSSR single-sensor baseline results for 61 of 64 RMS separation errors, and (not shown) for 54 of 64 cases of the 5th percentile separation error. Cases where MSSR has small errors (straight flight towards the radar) were found hard to improve with tracking. However, improvement was seen in maneuvering cases and scenarios where separation is changing.

When fused track output errors exceeded baseline, they exceeded their baseline by a relatively small amount, less than 0.02NM in RMS error and less than 0.09NM in 5% error.

In addition to these original simulations that all included a short-range sensor contributing to the fused track solution, additional simulations were performed with ADS-B as the sole contributor to the fused track and with ADS-B augmented with a long-range sensor as the contributors to the fused track. The results of these simulations showed that all four representative scenarios performed better than the MSSR baseline, when under ADS-B-Only or ADS-B+LRR sensor configurations.

Based on this analysis of STARS-provided simulation data, the prototype STARS Multisensor Fusion Tracker (MFT) can support 3 NM terminal separation performance.

2.C.2-8

Figure 4. RMS Separation Errors for All Scenarios and Runs

Figure 5. Cumulative Distribution & Error Statistics of Position Error for the Passing Aircraft in

Run 3 Holding Scenario

2.C.2-9

Flight Testing Controlled flight tests were performed on

November 17th, November 18th and December 17th, 2009 at Philadelphia International Airport (PHL). The purpose of these tests was to analyze the accuracy of the system display used by controllers to direct aircraft in terminal environments.

Approach The two aircraft used for testing were Beechcraft

King Air 90s owned by Ohio University with tail numbers N200U and FLC56. The baseline used for comparison against the fused system data was taken from the PHL sensor.

The data used in flight test analysis came from three primary sources: GPS, the STARS tracking system, and MSSR sensor reports. The GPS data was recorded by augmented systems onboard aircraft. Its accuracy is sufficient to be used as a reference source in analysis.

Table 4 shows the 6 scenarios evaluated, along with the acronyms and descriptions of the scenario.

Table 4. Flight Test Scenario Descriptions

Name Description In-Trail Tangential

IT Straight flight past radar, min. dist. from sensor is 20 NM, a/c are in-trail formation

Merge into Arrival

MA AC1 makes 90° turn to merge in front of AC2 in final approach pattern, AC2 trails AC1 by at least 3 NM after merge

Parallel Approach

PA Straight flight starting at least 10 NM from runway threshold, side-by-side in approach

Passing Holding

PH Racetrack pattern and a passing radial flight; AC1 completes holding pattern, AC2 performs a radial flight past AC1

In-trail Approach

PI Straight flight towards radar starting at least 10 NM from runway threshold; AC2 trails AC1 by at least 2.5 NM

Parallel Radial

PR Straight flight towards radar; Side-by-side flight pattern, trailing aircraft (AC2) flying at a higher velocity to overtake AC1

The scenarios were repeated four times, each with a different aircraft equipage combination, as shown in Table 5. Note that the “Mixed” case refers to equipage of both 1090ES (ADS-B) and Mode S.

Table 5. Aircraft Equipage for Flight Test Profiles

Profile Equipage Aircraft 1 Aircraft 2

1 Mode S Only Mode S Only 2 Mixed Mode S Only 3 Mixed Mixed 4 Mixed UAT

For the remainder of this section, the six flight patterns used in the flight tests will be referred to as a scenario, and the individual runs of each scenario will be referred to as profiles. Specific profiles will be referred to by their two-letter scenario abbreviation followed by the profile number. For example, the profile in the Passing Holding scenario in which both aircraft are surveilled with both MSSR and 1090ES will be referred to as PH3.

Finally, the error calculation is performed in the exact manner as was described in the Simulation section. Refer to Figure 2 for details on the calculation of position error that feeds separation error.

Selected Results Figure 6 and Table 6 show the results for the

RMS separation errors from the flight tests for the 24 scenario/profile combinations.

The results from the flight test analysis were as follows:

• The RMS values for the separation errors using the STARS fused track display system were smaller than the baseline for 17 of the 22 profiles. In the remaining five, four of these five were essentially equivalent to the baseline, exceeding that error by less than 0.01 NM, or approximately 60 feet. The final case (PI1) was less than 0.02 NM different.

• Vast improvements were seen in cases with maneuvering aircraft – the Merging (MA) and Holding (PH) scenarios.

2.C.2-10

The conclusion of the WG was that the results for the STARS fused track display system showed

equivalent or better performance in the scenario profiles than the baseline system.

Table 6. RMS Separation Errors in Feet from Flight Testing in PHL

Profile RMS Sep. Error # of

points

Profile RMS Sep. Error # of

points Baseline Fused Baseline Fused IT1 996 944 2595 PH1 605 249 1027 IT2 919 677 3180 PH2 1187 311 1419 IT3 928 881 2771 PH3 1108 353 1549 IT4 819 767 3181 PI1 428 512 323 MA1 1027 1030 108 PI2 488 546 444 MA2 1780 777 377 PI3 485 461 405 MA3 1063 644 290 PI4 466 517 507 MA4 1546 670 349 PR1 331 294 1797 PA1 1063 863 321 PR2 436 463 1965 PA2 829 631 184 PR3 309 269 1456 PA3 782 719 193 PR4 365 228 1918

Figure 6. Column Chart of RMS Separation Errors from Flight Tests

Targets of Opportunity Analysis To complement the scenario simulation and

flight testing, analysis was conducted on Targets of Opportunity (TOO) on 22 days (~146 hours) of surveillance data collected at the PHL TRACON between Oct. 5th and Nov. 25th, 2009. The purpose of the analysis was to evaluate a large set of data from STARS fused track display system searching for any anomalies in aircraft separations as they are presented to ATC. Occasional anomalous behavior may not be

observed in simulation or flight testing because of the limited amount of data. Display reports were collected from STARS fused output mode and compared to a single sensor baseline of the sensor located at PHL. Unlike flight tests, which have truth data from independent on-board GPS systems, TOO analysis is a relative comparison and is only as accurate as the baseline system. This is not a calculation of separation error; this is more of a sanity check across the two similar systems.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

IT1

IT2

IT3

IT4

MA1

MA2

MA3

MA4 PA

1

PA2

PA3

PH1

PH2

PH3

PI1

PI2

PI3

PI4

PR1

PR2

PR3

PR4

RMS

Sepa

ratio

n Er

ror (

NM

)

Baseline

Fused

2.C.2-11

Approach and Assumptions The TOO analysis consists of searching through

surveillance reports for valid pairs of aircraft and comparing the separations as reported by STARS fused track display mode to the single sensor baseline system. The TOO pairs were filtered to select only the pairs of aircraft that might be separated by ATC during operation. A TOO pair was valid for analysis if it met all of the following criteria:

• Horizontal Separation < 10 NM • Vertical Separation < 5,000 ft • Time range that all separation criteria are

met > 1 minute (though some portions of this time are filtered out due to missing radar data)

• Update Rate < two radar sweeps (2 * 4.9 seconds)

For a direct comparison of the separation measured by STARS fused output mode to the baseline system, reports from STARS and radar for each TOO pair are linearly interpolated to a common time-step. The time-step begins at a uniformly sampled random delay of 0-1 seconds after the beginning of the overlap to avoid favoring any individual report type. The time-step is sampled at 3 second intervals. Periods where the radar update rate does not meet the two sweep threshold (delay of > 10 seconds) are removed from the time-step and the subsequent interpolation. The separation differences for each pair are determined by calculating the differences in horizontal separation as measured by STARS fused output mode and the baseline single sensor at each sample time.

Figure 7. Sample Output from Analysis of a TOO Pair

Results for each TOO pair are plotted in the format shown in Figure 7. The X position, Y position and altitude plots show the STARS tracker fused output mode extrapolated display reports (crosses)

and the PHL sensor reports (circles) for both aircraft in the pair vs. time. The separation plot shows the horizontal separation as measured by the baseline system (blue) and by STARS (green) vs. time. On the

25

30

35

40

45

X (n

mi)

30

35

40

45

Y (n

mi)

2000

3000

4000

5000

6000

Alti

tude

(ft)

25 30 35 40 4530

35

40

45

System X (nmi)

Sys

tem

Y (n

mi)

XY Position

Track 1 RadarTrack 1 TrackerTrack 2 RadarTrack 2 TrackerPHL Range (60 nmi)

4

6

8

10

Sep

arat

ion

(nm

i)

RadarTracker

3850 3900 3950 4000 4050 4100-1

-0.5

0

0.5

1

Sep

arat

ion

Diff

eren

ce (n

mi)

Time (sec)-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10

2

4

6

8

10

12

Separation Difference (nmi)

Cou

nt

Statistics:σ = 0.106 nmi (646.9 ft)µ = 0.0559 nmi (339.4 ft)rms = 0.12 nmi (68.89 ft)

max = 0.528 nmi (3207 ft)5% = -0.0683 nmi (-415 ft)95% = 0.193 nmi (1174 ft)

Pair Information:Track 1 Mode 3 = 1346 (Radar Only)Track 2 Mode 3 = 3723 (Radar Only)Date = 11/13Start Time = 01:04:36 zEnd Time = 01:08:03 z

2.C.2-12

bottom left, the separation difference plot is a discrete plot of the separation difference measured at each sample time in the time-step. Gaps in the separation difference plot denote the portions of the time-step that were removed because the aircraft separation or the delay between radar reports exceeded the thresholds. The XY plot shows the STARS and radar position reports (crosses and circles) and interpolation (lines) on the system stereographic plane. For TOO pairs close to the maximum range of the PHL sensor, a 60 NM range circle is plotted (dotted line) for reference. The final plot shows a distribution of the separation difference calculated at each sample time as well as two tables describing the distribution. The left table contains identifying information for the TOO pair: the beacon code and equipage (radar only or radar and ADS-B) of both aircraft, as well as date and time information for the pair. The right table contains a variety statistics describing the separation difference distribution. From this collection of plots it is easy to determine where and why any anomalies in separation measurement occur.

Selected Results Separation Differences

Analysis results for all of the TOO’s were summarized by the underlying surveillance source(s) for each aircraft pair. Figure 8 compares the separation differences between each combination of sources. Each column shows the average RMS separation difference (red circle) and the 95th percentile RMS difference (blue triangle). The first column on the left represents all 24,530 TOO pairs. This group is broken down into three sub-groups: Radar/Radar, TOO pairs where both aircraft are being tracked by radar only; Radar/Mixed, TOO pairs where one aircraft is being tracked by radar only and the other is being tracked by both radar and ADS-B; and Mixed/ Mixed, TOO pairs where both aircraft are being tracked by radar and ADS-B. In all cases the average and 95% RMS errors are very close to each other (<0.1 NM), indicating very narrow distributions. The RMS separation difference trends downwards, decreasing when ADS-B is present. This behavior is easily explained through observation of the data. Many separation differences occur during transition periods which stress the IMM Kalman filter. Transitions can occur either in the aircraft state (e.g. transition from straight flight to turning or vice

versa) or in the sensor inputs (e.g. entering sensor coverage or crossing sensor boundaries). When ADS-B is present, the frequency of reports is much higher and the tracker is quicker to notice transitions, resulting in smaller RMS separation differences.

Figure 8. RMS Separation Differences

Anomaly Identification and Resolution 98 of the 24,530 TOO pairs were determined to

be “outliers”; which was heuristically defined as TOO pairs whose tracks at any point showed a separation difference in excess of 0.5 NM. In all 98 outlier pairs, both aircraft were being tracked by radar only. The outliers were separated into 5 categories depending on the behavior that caused the anomalous separation differences (see Figure 9). The following section describes the categories. As part of this analysis of anomalies, 3 “discrepancy reports” were generated to the performance of the system, corresponding to “Take-offs”, “Boundary Crossing”, and “Turns” categories. The STARS program proposed resolutions to each, mostly fixed through adaptation and tracked as part of the SBS Discrepancy Review Board process.

• Take-offs(16 pairs): One or both tracks involve a plane taking off. When the aircraft is first taking off the tracker does not have sufficient data to accurately determine its position.

• Boundary Crossing(10 pairs): One or both tracks involve an aircraft entering the PHL airspace range. As the aircraft enters the PHL airspace the tracker briefly continues following the path from the previous

0

0.05

0.1

0.15

0.2

0.25

RM

S S

epar

atio

n D

iffer

ence

(nm

i)

All(24530 pairs)

(1574555 points)

Radar-Radar(23476 pairs)

(1492392 points)

Radar-Mixed(964 pairs)

(67278 points)

Mixed-Mixed(90 pairs)

(14885 points)

RMS Separation Difference STARS vs PHL ASR-922 Days

Average95%

2.C.2-13

sensor before “snapping” the track to PHL sensor reports.

Figure 9. Outlier Distribution

• Turn (21 pairs): One or both tracks contain an aircraft conducting a turn that the tracker does not identify immediately.

• Radar Dropouts (17 pairs): One or both tracks contain infrequent radar coverage or extended radar dropouts. Without fresh data the tracker cannot accurately determine aircraft position. If the aircraft turns during or immediately following the infrequent radar coverage or dropout the separation difference becomes more extreme.

• Erratic Radar Reports (34 pairs): The tracks contain erratic radar reports. At longer ranges (> 40 NM from sensors) radar reports can be erratic making it impossible to discern whether the separation differences are due to radar error or tracker error.

The outliers were investigated further to determine the cause of the anomalous behavior and resolve these issues with the STARS tracker. The STARS vendor determined that the cause of these outliers was attributed to the following:

• Registration errors between sensors • The surface tracking filter (STF) was not

adapted correctly • The ASR-11 radar from Willow Grove that

feeds the PHL STARS system had

incorrect accuracy values adapted for that sensor

These issues were corrected in STARS and the TOO data was replayed to evaluate the changes.

A subsequent TOO analysis was conducted in March 2010 for a 48 hour set of data from PHL during peak traffic periods. This data showed that the number of outlier pairs with a maximum track position greater than 0.5NM from sensor position was 25 pairs from a set of 31,288 pairs (33 points from a data sample size of 1,885,129 samples, or a rate of 2x10-5). There were no outliers attributed to boundary crossing or turns which indicate that these issues were corrected with STARS through improvements to registration and adaptation changes. There was a single outlier attributed to takeoffs with a onetime separation difference of 0.52 NM, which is not considered a risk to separating aircraft in terminal airspace.

Summary

Simulation Scenario modeling was performed to test

STARS fusion output, when fed with avionics at the minimum allowable specified values. The results showed that the fused track output is equivalent to or better than the MSSR single-sensor baseline results in the vast majority of cases – 61 of 64 for the RMS separation error and 54 of 64 for the 5th percentile separation error. Cases where MSSR has small errors (straight flight towards the radar) are hard to improve with tracking. However, turning cases and scenarios where separation is changing were improved.

Controlled Flight Testing Controlled flight tests were performed at and

around the Philadelphia International Airport on Nov. 17-18th, and Dec. 18th, 2009. Six flight scenarios were used in the flight tests, each with four profiles of varying aircraft equipage pairings. Twenty-two of the twenty-four profiles were included in the analysis, as the remaining two profiles did not contain appropriate sensor baseline information. The RMS values for separation error exhibited by the STARS system were smaller than those of the baseline (MSSR) system in 17 of the 22 profiles. In the fused configuration, four of these five were essentially equivalent to the baseline, exceeding that error by

Boundary Crossing

10%

Turns21%Radar

Dropouts17%

Take-Offs16%

Erratic Radar Reports

36%

2.C.2-14

less than 0.01 NM, or approximately 60 feet. The baseline never performed better than the fused track display system by more than 0.02 NM.

Targets of Opportunity Analysis Analysis of TOOs over 22 days (~146 hours) of

surveillance data collected within the PHL TRACON airspace between October 5th and November 25th, 2009. The purpose was to evaluate a large set of data searching for any anomalies in aircraft separations as they are presented to ATC. Display reports were collected from STARS fused output mode and compared to a single sensor baseline of the ASR-9/Mode S sensor located at PHL. TOO analysis is a relative comparison and is only as accurate as the baseline system. This was not a calculation of separation error.

For 99.6% of the TOO pairs, the fused track separation was within 0.5 NM of baseline separation measured by the MSSR. The remaining 0.4% consisted of 98 outlier pairs, all of which had both aircraft being tracked by radar only (i.e. no ADS-B). These outliers were attributed to takeoffs, boundary crossings, turns, radar dropouts, and erratic radar reports. The outliers were investigated further to determine the cause of the anomalous behavior and the issues were resolved through corrective processes in the SBS implementation process.

Other Analyses Not Described Other analyses were conducted to evaluate the

performance in the PHL airspace that are not described here in detail, including:

• Flight tests to assess performance for targets without altitude data

• Tracker stress testing using scenarios designed to ensure that the tracker appropriately determined when targets are entering a maneuver or transitioning back to straight-line flight such as to minimize track lag errors.

• Simulations of performance with ADS-B and/or long-range radar supporting the tracks (in lieu of short range radar).

• Using an analytical model of performance (Surveillance Separation Error) analytical model that includes an evaluation of GNSS faults

WG Recommendation Overall, the analytical activities showed that

STARS automation fusion performance supports terminal separation, because it has equivalent to or better accuracy than monopulse secondary surveillance radar systems in use in the NAS today. The Work Group recommends that STARS be approved to support terminal separation operations. An important contingency was placed on the approval – that DO-260B/DO-282B avionics are used for this service. The flight test and target of opportunity analysis utilized “A” versions of the avionics standards, since there was no impact to performance based on the assumptions made. However, certification of the systems at the “B” version will be necessary for operation.

Current Status On March 28th, the FAA declared Initial

Operating Capability at the Philadelphia TRACON for the use of ADS-B for air traffic separation services on the STARS automation platform. The system was used operationally to separate traffic while undergoing continuing evaluation, including Independent Operational Test and Evaluation in preparation for the upcoming SBS system In-Service Decision (ISD). At the time of this writing, the ISD is expected in October 2010.

Next Steps The Separation Standards Work Group is

continuing to evaluate the PHL airspace with STARS with a program of continued data collection and analysis. This effort will monitor the performance of the system and ensure that no anomalies occur which affect the ability of the STARS fusion system to provide separation services. The analysis is briefed to the SBS program on a quarterly basis.

References [1] Department of Transportation, Federal Aviation Administration, 14 CFR Part 91, “Automatic Dependent Surveillance - Broadcast (ADS–B) Out Performance Requirements To Support Air Traffic Control (ATC) Service; Final Rule”, FAA-2007-29305-0289, 28 May, 2010

[2] "Final Program Requirements for Surveillance and Broadcast Services", Federal Aviation

2.C.2-15

Administration Surveillance and Broadcast Services Program, Version 2.1, Aug. 6, 2007

[3] "Minimum Operational Performance Standards for 1090 MHz Extended Squitter Automatic Dependent Surveillance – Broadcast (ADS-B) and Traffic Information Services – Broadcast (TIS-B)", DO-260B, RTCA Inc., 9 December 2009

[4] "Minimum Operational Performance Standards for Universal Access Transceiver (UAT) Automatic Dependent Surveillance – Broadcast (ADS-B)", DO-282B, RTCA Inc., 2 December 2009

[5] “Evaluation of Separation Performance with ADS-B at the Louisville Key Site”, Michael Castle, Randall Sleight, Steven Handy, Proceedings of the 28th Digital Avionics Systems Conference, Oct. 2009

[6] “Manual on Airspace Planning Methodology for the Determination of Separation Minima”, ICAO Doc 9689-AN/953, 1998

Acknowledgements The authors wish to express their thanks and

appreciation to Mr. Vincent Capezzuto, SBS Program Director, to Mr. Robert Pomrink, SBS Lead SE, and the Members of the SBS ADS-B/WAM Separation Standards Working Group for their invaluable review of, and comments upon, the progress, details, and conclusions of this analysis.

29th Digital Avionics Systems Conference

October 3-7, 2010