A Glimpse at Archie: The A Glimpse at Archie: The LOSA ArchiveLOSA Archive

Ashleigh Merritt, Ph.D.Ashleigh Merritt, Ph.D.

The University of Texas Human The University of Texas Human Factors Research Project (UT)Factors Research Project (UT)

Third ICAO/IATA LOSA/TEM ConferenceThird ICAO/IATA LOSA/TEM Conference

September 13-14, 2005, Kuala LumpurSeptember 13-14, 2005, Kuala Lumpur

What is Archie?What is Archie?

““Archie” is the LOSA Archive Archie” is the LOSA Archive A database of de-identified airline data A database of de-identified airline data

which is maintained and updated by UTwhich is maintained and updated by UT

Airlines that do a LOSA with The LOSA Airlines that do a LOSA with The LOSA Collaborative contribute their data to Collaborative contribute their data to Archie for research purposes Archie for research purposes

How old is Archie?How old is Archie?

Five LOSAs in five years to Five LOSAs in five years to develop the methodology develop the methodology

Data in the Archive starts in Data in the Archive starts in 20002000

Had to wait for Archie to grow Had to wait for Archie to grow before doing any industry-before doing any industry-wide analyses.wide analyses. If we think of Archie as a dog, he If we think of Archie as a dog, he

would be a young bloodhoundwould be a young bloodhound

How big is Archie?How big is Archie?

As of August 2005, Archie has data fromAs of August 2005, Archie has data from 20 airlines (including 3 repeat LOSAs)20 airlines (including 3 repeat LOSAs) 4800 flights4800 flights

4800 flight narratives written by trained 4800 flight narratives written by trained observersobservers

4800 ratings of TEM countermeasure 4800 ratings of TEM countermeasure performanceperformance

17,500+ threats17,500+ threats 12,500+ errors12,500+ errors 2,400+ undesired aircraft states2,400+ undesired aircraft states

Archie’s Attributes 2004Archie’s Attributes 2004

1.1. Archie is loyal and Archie is loyal and faithful faithful

2.2. Archie alerts of dangerArchie alerts of danger

3.3. Archie is a tracker Archie is a tracker

4.4. Archie plays well with Archie plays well with othersothers

2005 Update on Archie2005 Update on Archie

Archie has remained loyal and Archie has remained loyal and faithful to its aviation masterfaithful to its aviation master

No airline has been publicly No airline has been publicly identifiedidentified

No individual pilots can be or ever No individual pilots can be or ever have been identifiedhave been identified

Archie continues to play well Archie continues to play well with otherswith others

2005

2004

Queries from / data-Queries from / data-sharing with: sharing with:

NOSS (ATC group)NOSS (ATC group) IATA/ICAO (ITA)IATA/ICAO (ITA) Boeing Boeing Airlines Airlines ASAP ASAP ATSBATSB

2005: Archie is learning to 2005: Archie is learning to tracktrack

Airline threats Airline threats ATC threats ATC threats Intentional Intentional

NoncomplianceNoncompliance

Airline threats: Airline threats: Simple TrackingSimple Tracking

Airline threats include aircraft malfunctions, time Airline threats include aircraft malfunctions, time pressure, errors/distractions/interruptions from pressure, errors/distractions/interruptions from Cabin, Dispatch, Maintenance, Ground, Ramp. Cabin, Dispatch, Maintenance, Ground, Ramp.

Airline threats occur on 2/3 of all flights, adding Airline threats occur on 2/3 of all flights, adding to overall workload. to overall workload.

Range: 55% - 81% across different airlinesRange: 55% - 81% across different airlines Three-quarters of airline threats occur before the Three-quarters of airline threats occur before the

aircraft has even left the gate (predeparture).aircraft has even left the gate (predeparture).

TARGETS FOR ENHANCEMENT?TARGETS FOR ENHANCEMENT?

ATC threat chains: ATC threat chains: Complex trackingComplex tracking

10 LOSAs – 2426 Flights – 9450 threats 2350 ATC threats

= ¼ of all threats = about 1 per flight

236 mismanaged ATC threats = 30% of all mismanaged threats 10% of all ATC threats were

mismanaged

Mismanaged ATC threatsMismanaged ATC threats

2426 Flights – 9450 threats

236 mismanaged ATC threats

Challenging clearances

(53%)

Runway Changes (14%)

Language difficulty (8%)

Similar call signs (5%)

ATC error (7%)

ATC threats -> ATC threats -> ErrorsErrors

2426 Flights – 9450 threats

236 mismanaged ATC threats

280 errors

Automation errors (20%) – wrong

MCP/FCU altitude setting dialed

Communication errors (27%) -

misinterpret ATC instructions

Aircraft handling (24%) –

unintentional speed, lateral, vertical

deviations

Cross-verification errors (7%)

Ground navigation errors (6%)

ATC threats -> ATC threats -> Errors -> Errors -> UASUAS

2426 Flights – 9450 threats

236 mismanaged ATC threats

280 errors

56 Automation errors

76 Communication errors

67 Aircraft handling errors

19 Cross-verification errors

16 Ground navigation errors

2UAS

58UAS

5UAS

9 UAS

21UAS

ATC threats ATC threats -> UAS-> UAS

2426 Flights – 9450 threats

236 mismanaged ATC threats

280 errors

106 UAS

Lateral deviation 17%

Speed too high 14%

Vertical deviation 10%

Unstable Approach 10%

Continued Landing 9%

Incorrect Automation configuration 9%

Taxiway/runway incursion 6%

Speed too low 6%

Incorrect Aircraft configuration 5% Other 14%

Bottom line: 4% of flights had a

UAS arising from an ATC threat that was

mismanaged. ¾ occurred during

descent/approach/land

Archie tracks intentional Archie tracks intentional noncompliancenoncompliance

Across 22 LOSAs, the average is 40% of Across 22 LOSAs, the average is 40% of flights have one or more noncompliance flights have one or more noncompliance errorserrors

Range: 23% of flights at one airline to 90% (!!) Range: 23% of flights at one airline to 90% (!!) at another at another

Most common noncompliance errors:Most common noncompliance errors: checklist performed from memory / checklist performed from memory /

nonstandard checklist usenonstandard checklist use failure to cross-verify MCP/FCU altitude alerter failure to cross-verify MCP/FCU altitude alerter

changes changes PF makes own MCP/FCU changesPF makes own MCP/FCU changes

Noncompliance correlates Noncompliance correlates with..with..

Based on 22 LOSAs (4800 flts), airlines Based on 22 LOSAs (4800 flts), airlines that have more flights with intentional that have more flights with intentional noncompliance errors also have more noncompliance errors also have more flights with:flights with:

Mismanaged threats (r = .7) Mismanaged threats (r = .7) Mismanaged handling errors (r = .9)Mismanaged handling errors (r = .9) Undesired aircraft states (r = .9)Undesired aircraft states (r = .9) Mismanaged UASs (r =.8)Mismanaged UASs (r =.8)

Archie was shocked…

Archie’s FutureArchie’s FutureArchie is maturing and getting smarter – more LOSAs allow for more analyses at the airline level, which is where the greatest differences are emerging.

Soon a sub-Archive for regional airlines

Archie is getting involved in more industry projects

If LOSA and ASAP overlap, then focusing on LOSA targets should reduce ASAP incidents…

ATC is a big threat to pilots, are pilots a big threat to ATC?

Archie is recognizing his Archie is recognizing his strengthstrength

LOSA is a macro LOSA is a macro approach for Safety approach for Safety Management Systems Management Systems

LOSA detects “gross” LOSA detects “gross” effects – strengths and effects – strengths and weaknesses in the weaknesses in the systemsystem

Management uses the Management uses the informationinformation

Self-corrections, training, Self-corrections, training, procedural changes,procedural changes, … …

If and when certain problems persist, micro If and when certain problems persist, micro approaches can investigate more closely…approaches can investigate more closely…

The University of TexasThe University of TexasHuman Factors Research ProjectHuman Factors Research Project

www.psy.utexas.edu/HumanFactors