Presented to: Committee on Assessing the Risks of UAS Integration
Presented by: Federal Aviation Administration
Date: Wednesday, September 27, 2017
Federal AviationAdministration
UAS Integration Risk AssessmentAir Traffic Organization
FEDERAL AVIATIONADMINISTRATION
What is a Safety Management System?
29/27/2017 SMS Overview
Risk Management
Safety Policies and Standards
Communication, Values, and
CultureData, Analysis, and Reporting
SMS
An integrated collection of processes, policies, procedures, and programs used to assess, define, and manage safety risk in the provision of air traffic control
and navigational services
FEDERAL AVIATIONADMINISTRATION
What Does SMS Provide?
39/27/2017 SMS Overview
Continuous monitoring,
assessment, & mitigation
Common framework to
identify & address safety hazards
and risks
Communication/ management of
potential & actual risks in the
system
Data-Informed, Expertise-Driven Management
FEDERAL AVIATIONADMINISTRATION
What are the SMS Components?
49/27/2017 SMS Overview
Safety Assurance
Safety Promotion
Safety Risk
Management
Safety Policy
FEDERAL AVIATIONADMINISTRATION
What is the SRM Process?
DIAAT: The formalized approach used by a panel of experts and affected stakeholders to identify issues, evaluate
their criticality, and determine a means to minimize or eliminate their impact
59/27/2017 SRM Process Overview
Treat the Risk
Identify the Hazards
Analyze the Risk
Assess the Risk
Describe the System
FEDERAL AVIATIONADMINISTRATION
Treat the Risk
Identify the Hazards
Analyze the Risk
Assess the Risk
Describe the System
Describe the System
9/27/2017 SRM Process Overview: Describe the System
6
FEDERAL AVIATIONADMINISTRATION
What is the 5M Model?
A tool used to describe the
system, operation, or procedures being added or
changed
79/27/2017 SRM Process Overview: Describe the System
(hu)Man/ Person
MachineManagement
Media or Environment
Mission
FEDERAL AVIATIONADMINISTRATION
Treat Risk
Identify the Hazards
Analyze Risk
Assess Risk
Describe the System
Identify Hazards
9/27/2017 SRM Process Overview: Identify Hazards 8
FEDERAL AVIATIONADMINISTRATION
Any real or potential condition that can cause injury, illness, or death to people; damage to or loss of a system, equipment, or property; or damage to the environment. A prerequisite to an accident or incident.
What is a Hazard?
99/27/2017 SRM Process Overview: Identify Hazards
Preliminary Hazard List/
What If Analysis
Procedural Changes (system or
operational)
Functional Hazard Analysis
Acquisition or modification of
equipment
Bow-Tie Diagram
All applications when it is difficult to determine hazard or
cause/effect relationship
Tool/Technique Best Use
FEDERAL AVIATIONADMINISTRATION
Treat Risk
Identify the Hazards
Analyze Risk
Assess Risk
Describe the System
Analyze Risk
9/27/2017 SRM Process Overview: Analyze Risk 10
FEDERAL AVIATIONADMINISTRATION
What Is Risk?The composite of predicted severity and likelihood of the potential effect of a hazard, before any of the
proposed mitigations are implemented
119/27/2017 SRM Process Overview: Analyze Risk
Severity
• The consequence or impact of a hazard’s effect (outcome) in terms of degree of loss or harm (less severe to more severe)
• While assessed first, determination of severity is independent of likelihood
Likelihood
• The estimated (predicted) probability or frequency, in quantitative or qualitative terms, of a hazard’s effect (outcome)
• Definitions are tailored to ATC Operations, Flight Procedures, and Systems Engineering
FEDERAL AVIATIONADMINISTRATION
Assess Risk
Treat Risk
Identify the Hazards
Analyze Risk
Assess Risk
Describe the System
9/27/2017 SRM Process Overview: Assess Risk 12
FEDERAL AVIATIONADMINISTRATION
What are Risk Levels?
139/27/2017 SRM Process Overview: Assess Risk
Risk levels are assessed for each hazard based on severity and likelihood
• Must be mitigated to a Medium or Low risk prior to implementation
High: Unacceptable
Risk
• May be implemented but safety requirements are recommended to increase the safety margin
Medium: Acceptable
Risk
• May be implemented (but safety requirements are recommended)
• Must have at least one safety performance target
Low: Acceptable
Risk
Seve
rity
/ Lik
elih
ood
FEDERAL AVIATIONADMINISTRATION
Treat Risk
Treat Risk
Identify the Hazards
Analyze Risk
Assess Risk
Describe the System
9/27/2017 SRM Process Overview: Treat Risk 14
FEDERAL AVIATIONADMINISTRATION
• Document all hazards and document and verify low-risk hazards at least once• Document a plan to implement safety requirements for all risk levels identified in the final
Hazard Analysis Worksheet (HAW)• Record monitoring activities
How do we Treat Risk?
159/27/2017 SRM Process Overview: Treat Risk
• Identify and evaluate means to manage the risk or reduce it to an acceptable level• The four risk management strategies are: Risk Control, Risk Avoidance, Risk
Transfer, and Risk Assumption
Identify feasible risk management
options
• The risk that is estimated to exist after the safety requirements are implemented, or after all avenues of risk mitigation have been explored.
Determine predicted
residual risk
• Identify measurable goals used to verify the predicted residual risk of a hazard• Determine appropriate metrics• Consider controls and safety requirements• Pre-SRM panel data analysis serves as the basis for comparison against the
post-implementation metrics
Define safety performance
targets
Develop a monitoring plan
1
2
3
4
FEDERAL AVIATIONADMINISTRATION
Evaluating UAS Risk Using SRM
169/27/2017 Evaluating UAS Risk Using SRM
FEDERAL AVIATIONADMINISTRATION
Scoping UAS Integration
9/27/2017 17Evaluating UAS Risk Using SRM
• UAS risk analysis efforts focused on the following:– Line-of-sight operations– Operations beyond visual line of sight– Commercial uses and operations (i.e., the Pathfinder
Program)– Certificate of Authorization (COA) processes– Impact on Airspace Class (e.g., operations near and
around airports and large population centers)
FEDERAL AVIATIONADMINISTRATION
Past UAS Safety Studies
– COA process for UAS low-altitude operations under Section 333 (200 feet Above Ground Level (AGL))
– COA process for UAS low-altitude operations under Section 333 at or below 400 feet AGL
– Class B
– Class A and High E
– Class A and High E– Class C– Class D– Class E– Class G– Conditions and limitations for
allowing operations in Class E Surface Area
9/27/2017 18
• SRM panels convened for each class of airspace and Pathfinders for the safe integration and testing of UAS in the National Airspace System (NAS)
Evaluating UAS Risk Using SRM
FEDERAL AVIATIONADMINISTRATION
Common Hazards Across Safety Analyses Lost Link - The loss of real-time command and control (C2) data link. If the link between the Pilot in Command (PIC) and the UAS is lost, the Unmanned Aircraft (UA) utilizes pre-programmed contingency (i.e., lost link) procedures until the link is re-established or until the UAS ends the flight in a safe manner
Fly Away - Interruption or loss of the control link, or when the pilot is unable to affect control of the aircraft and, as a result, the UA is not operating in a predicable or planned manner
Difficulty to be Seen by Manned Aircraft - Due to size of UA (e.g., Quad Copters or small fixed-wing), speed, body composition, and markings
199/27/2017 Evaluating UAS Risk Using SRM
FEDERAL AVIATIONADMINISTRATION
Loss of Visual Line of Sight - The PIC has lost visual contact with the UA as required by 14 Code of Federal Regulations Section 91.113Loss of Communications Between Pilot and Observer because of radio failure and they are not co-located
Common Hazards Across Safety Analyses (cont.)
209/26/2017 Evaluating UAS Risk Using SRM
FEDERAL AVIATIONADMINISTRATION
Worst Credible Effect
1973 MOR in 2016144 NMAC in 20162 MAC in 25 years
Most UAS operations -NMAC is the Worst Credible Effect
219/27/2017 Evaluating UAS Risk Using SRM
1.00E-11
1.00E-10
1.00E-09
1.00E-08
1.00E-07
1.00E-06
1.00E-05
1.00E-04
1.00E-03
1.00E-02
1.00E-01
1.00E+00MOR NMAC MAC
Minimal 5 Minor 4 Major 3 Hazardous 2 Catastrophic 1
FrequentA
ProbableB
RemoteC
ExtremelyRemote
D
ExtremelyImprobable
E
SMS Risk Matrix
FEDERAL AVIATIONADMINISTRATION
Current UAS Effects and Severity Classifications
22
UAS Hazard Severity ClassificationNote: Severities related to ground-based effects apply to movement areas only.
Minimal5
Minor4
Major3
Hazardous2
Catastrophic1
CONDITIONS RESULTING IN ANY ONE OF THE FOLLOWING:
Und
er 5
5 po
unds
and
100
mph
or s
low
er Discomfort to those on the ground
Low Risk Analysis Event severity, two or fewer indicators fail
Non-serious injury to three or fewer people on the ground
A reduced ability of the crew to cope with adverse operating conditions to the extent that there would be a significant reduction in safety margins
UAS crash obstructing Runway
Medium Risk Analysis Event severity, three indicators fail
Incapacitation to Unmanned Aircraft System crew
Non-serious injury to more than three people on the ground
Manned aircraft making an evasive maneuver, but proximity from Unmanned Aircraft remains greater than 100 feet
High Risk Analysis Event severity, four indicators fail
Proximity of less than 100 feet to a manned aircraftSerious injury to persons other than the Unmanned Aircraft System crew
A collision with a manned aircraft
Fatality or fatal injury to persons other than the Unmanned Aircraft System crew
9/27/2017 Evaluating UAS Risk Using SRM
FEDERAL AVIATIONADMINISTRATION
Current UAS Effects and Severity Classifications
UAS Hazard Severity ClassificationNote: Severities related to ground-based effects apply to movement areas only.
Minimal5
Minor4
Major3
Hazardous2
Catastrophic1
CONDITIONS RESULTING IN ANY ONE OF THE FOLLOWING:
Ove
r 55
poun
ds o
r fas
ter t
han
100
mph
Discomfort to those on the ground
Loss of separation leading to a Measure of Compliance greater than or equal to 66 percent
Low Risk Analysis Event severity, two or fewer indicators fail
Non-serious injury to three or fewer people on the ground
Medium Risk Analysis Event severity, three indicators fail
Non-serious injury to more than three people on the ground
A reduced ability of the crew to cope with adverse operating conditions to the extent that there would be a significant reduction in safety margins
Manned aircraft making an evasive maneuver, but proximity from Unmanned Aircraft remains greater than 500 feetUAS crash obstructing Runway
High Risk Analysis Event severity, four indicators fail
Incapacitation to Unmanned Aircraft System crew
Proximity of less than 500 feet to a manned aircraftSerious injury to persons other than the Unmanned Aircraft System crew
A collision with a manned aircraft
Fatality or fatal injury to persons other than the Unmanned Aircraft System crew
239/27/2017 Evaluating UAS Risk Using SRM
FEDERAL AVIATIONADMINISTRATION
Pros and Cons of NMACs as Credible Effects
NMACs are reported by controllers and pilots
NMACs are tracked and categorized using existing data collection
It is difficult or impossible to determine the cause of the NMAC involving a UAS
UAS operators cannot be identified
The existence and criticality of a NMAC is subjective based on its current definition (requires third-party verification)
249/27/2017 Evaluating UAS Risk Using SRM
FEDERAL AVIATIONADMINISTRATION
Monitoring UAS Encounters
25
• Quarterly, AJI collects data on UAS encounters; demographics include:– Altitude– Nearest ATC facility– Proximity to manned aircraft– Type of manned aircraft involved– UAS position in relation to manned aircraft– Class of airspace– Phase of flight (manned aircraft)– Severity of NMAC
9/27/2017 Evaluating UAS Risk Using SRM
FEDERAL AVIATIONADMINISTRATION
Trend in UAS Encounters (April–July 2017, FY17 Q3)
0
2
4
6
8
10
12
14
16
18
20
Total
Linear (Total)
269/27/2017 Evaluating UAS Risk Using SRM
FEDERAL AVIATIONADMINISTRATION
FY17 Trends in UAS Encounters
279/27/2017 Evaluating UAS Risk Using SRM
Data set – all 500ft NMACs (1/1/17 - 9/19/17)• 41% are 100ft NMACs• 5% occurred under 500ft altitude• 53% occurred over 3000ft altitude
Reports Identifying AirspaceClass A: 0.5% Class D: 14%Class B: 34% Class E: 19%Class C: 15% Class G: 5%
FEDERAL AVIATIONADMINISTRATION
Future of UAS Risk Assessment
289/27/2017 Future of UAS Risk Assessment
FEDERAL AVIATIONADMINISTRATION
UAS Integration Risk
The primary goal of the ATO is to prevent collision with manned aircraft in controlled airspace, while FAA
lines of business/other agencies are responsible for risks not related to air traffic
299/27/2017 Future of UAS Risk Assessment
Collision with manned aircraft,
people on the ground, or other
UAS
Damage to property
Environment Issues
FEDERAL AVIATIONADMINISTRATION
Target Level of Safety
309/27/2017
• The NAS is a very safe system, but it is not risk-free– The probability of collision is greater than zero– The goal is to maintain a low probability of collision, but
how low?• Most air navigation service providers have a target
level of safety.– The ATO’s minimum acceptable level of safety for a
catastrophic outcome is 1E-9• Recently, the ATO adopted an Acceptable Level of
Safety concept for rare operations, such as commercial space efforts
Future of UAS Risk Assessment
FEDERAL AVIATIONADMINISTRATION
Quantitative Risk Assessment• Risk should be assessed in relation to its
“proximity” to collision– All incidents and accidents should be analyzed with
respect to their probability of collision• Need for a Risk Continuum concept
– Employ modeling: • Use objective data • Analytical approaches; geometry and density• Minimize reliance on subject matter expert input
by employing modeling (e.g., Monte Carlo modeling)
319/27/2017 Future of UAS Risk Assessment
FEDERAL AVIATIONADMINISTRATION
Factors in Modeling• Not all collisions are created equal. This is
dependent upon:– UAS size– UAS speed– Vulnerability of aircraft
• “Credit”: Controls and mitigations should be accounted for (e.g., sense and avoid)
• “Penalty”: All credible sources of errors should be accounted for (e.g., human error)
329/27/2017 Future of UAS Risk Assessment
FEDERAL AVIATIONADMINISTRATION
Current Modeling Effort• Under the Low-Altitude Authorization and Notification
Capability (LAANC) initiative, the FAA has developed maps with pre-approved flight zones and maximum altitudes for operating drones near airports
• With the near-term focus of supporting LAANC, Volpe is working on a quantitative model to examine small UAS operating in controlled airspace around airports– Input data: Manned aircraft tracks– Output 1: Collision probability – Output 2: Risk-Adjusted Altitude
• Collision probability is negligible below, and non-negligible above, Risk-Adjusted Altitude
– Output 3: Sample Size• Used to assign confidence level to Outputs 1 and 2
339/27/2017 Future of UAS Risk Assessment
FEDERAL AVIATIONADMINISTRATION
Manned aircraft
Flight “swath” modeled by aircraft frontal profile area extruded over distance traveled
Airport
Small UAS randomlylocated in airspace
Relevant airspace examined over a given period of time, number of aircraft, etc.
Collision Probability
Aircraft frontal profiles
349/27/2017 Future of UAS Risk Assessment
Ratio of manned aircraft “swath” volumes to total airspace volumeLength of analysis time period
Unmitigated Collision Probability per Unit Time =
FEDERAL AVIATIONADMINISTRATION
Manned aircraft
• Controlled airspace divided into 1-min x 1-min grid volumes• Risk-Adjusted Altitude is defined as ”200 feet below routine
operations” in each grid volume:
Maximum altitudeof grid volume (400 feet)
Altitude of routine manneda/c operations in grid volume
Ground levelRisk-AdjustedAltitude
Highest Risk
200 feetSafety Buffer
Operatewith
caution
Risk-Adjusted Altitude
359/27/2017 Future of UAS Risk Assessment
FEDERAL AVIATIONADMINISTRATION
Routine Operations• Volpe’s model is in the early stages of development• Risk-Adjusted Altitude is calculated on a grid-by-grid
basis, rounded down to the nearest 50 feet in each grid, and presented two ways
• “Worst-case” = Lowest altitude reached manned aircraft tracks (points within tracks) over analysis time period, minus 200-foot safety buffer
• “Fifth Percentile below ceiling”: 1,200 feet selected as ceiling, below which is considered shared airspace– Filter out lowest 5 percent as “outliers” or “rare operations”– 200 feet below fifth percentile of points below 1,200 feet in each
grid volume
369/27/2017 Future of UAS Risk Assessment
FEDERAL AVIATIONADMINISTRATION
Challenges for Risk Analysis
379/27/2017 Future of UAS Risk Assessment
Insufficient data collection to
establish reliable performance monitoring
Lack of procedures for
UAS
Communication issues (e.g., non-
standard terminologies)
Difficult to quantify
mitigations
Lack of licensing requirements to
maintain a level of design reliability
Insufficient operator
proficiency
FEDERAL AVIATIONADMINISTRATION
Takeaways
38
ATO’s primary safety focus is on preventing collision between manned aircraft
9/27/2017 Future of UAS Risk Assessment
Modeling approaches focused on collision probability to determine the true risk from UAS operations in the NAS.
Improved data (collected and simulated) for monitoring performance targets.
Future StateSafety and operational experts have used the SRM process to assess UAS risk potential.
AJI is collecting data on UAS encounters with manned aircraft.
Current State
FEDERAL AVIATIONADMINISTRATION
Back-Up Slides
9/27/2017 39Back-Up Slides
FEDERAL AVIATIONADMINISTRATION
SRM Panels• Carry out the SRM process and produce safety
documents with risks and recommendations to mitigate them.
• Stakeholders and Process Participants:– Safety engineers– ATC experts– Pilots– Bargaining unit representatives– Data analysts– Industry subject matter experts
409/27/2017 Back-Up Slides
FEDERAL AVIATIONADMINISTRATION
SRM Process
419/27/2017 Back-Up Slides
SRM Document
If No Hazards Are Identified
Review Current System
Review Proposed Change or Existing
Issue
Develop Preliminary Hazard List
Develop Hazard
Analysis Worksheet
Develop Monitoring
Plan
FEDERAL AVIATIONADMINISTRATION
Link Between Cause, Hazard and Effect
Cause(s)
Ice Hazard
Possible Effect
Identify the Hazards
9/27/2017 Back-Up Slides 42
FEDERAL AVIATIONADMINISTRATION
Preliminary Hazard List
• Hazard identification tool that lists potential hazards
• The result of brainstorming (may include hazards, causes, effects, and sometimes system state or controls)
• Used to list all possible hazards within scope• Stepping stone to HAW
Identify the Hazards
Ultimately, the list will be a combination of hazards, causes, and effects that will be later categorized in the SRM process. Within/out of scope is not addressed at this time.
9/27/2017 Back-Up Slides 43
FEDERAL AVIATIONADMINISTRATION
HAW
• What is a HAW?– An analysis tool used to document the risk assessment of the
system or change– Inputs are developed from a PHL (or other Hazard
Identification tool)
• Why complete a HAW?– Required as part of the SRM process– Links identified hazards to controls and risk mitigation
activities– To document risks, hazards, system states, and safety
requirements
449/27/2017 Back-Up Slides
Identify the Hazards
FEDERAL AVIATIONADMINISTRATION
HAW: Template (Key Definitions)
(1)Hazard ID
(2)Hazard
Description
(3)Causes
(4)System State
(5)Controls
(6)Control
Justification
(7)Effects
(Credible Effects)
(8)Severity
(9)Severity
Rationale
(10)Likelihood
(11)Likelihood Rationale
(12)Initial Risk
(13)Safety
Requirements
(14)Organization
Responsible for Implementing
Safety Requirements
(15)Predicted Residual
Risk
(16)Safety
Performance Targets
* Refer to Quick Reference Guide for Additional Information.
9/27/2017 Back-Up Slides 45
Identify the Hazards
FEDERAL AVIATIONADMINISTRATION
(1)Hazard ID
(2)Hazard Description
(3)Causes
(4)System State
ABC-01 Ice accumulation on the sidewalk (slipping hazard)
Freezing rain
Snow
November-March
Temperatures below 32 °F
HAW Ice Example: Identify Hazards
469/27/2017 Back-Up Slides
FEDERAL AVIATIONADMINISTRATION
Controls: Guidelines
• Understand the impact of the control• Must be associated with the change, hazard, cause
and system state• Cite the specific version, paragraph and/or section
number(s) when using FAA Orders• Include information explaining how the control
mitigates the risk
479/27/2017 Back-Up Slides
Analyze Risk
FEDERAL AVIATIONADMINISTRATION
Effects (Credible Effects)
• Credible effect refers to the reasonable expectation that the assumed combination of conditions that define the system state will occur within the operational lifetime of a typical ATC system.
• Hazard assessments consider all credible effects (outcomes).
• Less severe effects may pose a higher risk than the worst credible effect.
489/27/2017 Back-Up Slides
Analyze Risk
FEDERAL AVIATIONADMINISTRATION
SMS Likelihood Table
49
Operations: Expected Occurrence Rate (per operation / flight hour / operational hour)
Quantitative (ATC / Flight Procedures / Systems Engineering)
Frequent (A) (Probability) ≥ 1 per 1,000
Probable (B) 1 per 1000 > (Probability) ≥ 1 per 100,000
Remote (C) 1 per 100,000 > (Probability) ≥ 1 per 10,000,000
Extremely Remote (D) 1 per 10,000,000 > (Probability) ≥ 1 per 1,000,000,000
Extremely Improbable (E) 1 per 1,000,000,000 > (Probability) ≥ 1 per 1014
9/27/2017 Back-Up Slides
Analyze Risk
FEDERAL AVIATIONADMINISTRATION
(5)Control(s)
(6)Control(s) Justification
(7)Effects
(8)Severity
(9)Severity Rationale
(10)Likelihood
(11)Likelihood Rationale
HAZARD: ABC-01
Current building safety/security plan
ABC regulations
Intervention by facility personnel
BLD PLN 001
ABC Reg 002
Facilitate personnel intervene by placing warning signs and spreading salt/ice melting agents.
Sprained Wrist
Head injury
Major (3)
Hazardous (2)
Slipping on ice can cause joint injuries, bodily harm, muscle
strains
Probable (B)
Extremely Improbable
(E)
10,000 people use the sidewalks, and approximately 5 injuries occur due to slips on ice, giving a rate of 5x10-4 per operation (one per 2000).
Over the past 10 years 36 head injuries have been reported. Estimate that a typical person makes about 100 trips per winter on foot. Total population of US is about 300 million, so rate is 36 / (10*100*3x108) = 1.2x10-11 per operation.”
Slipping on ice can cause accidental
head injury.
HAW Ice Example: Analyze Risk
509/27/2017 Back-Up Slides
Analyze Risk
FEDERAL AVIATIONADMINISTRATION
HAW Ice Example: Assess Risk
519/27/2017 Back-Up Slides
(7)Effects
(8)Severity
(10)Likelihood
(12)Initial risk
Initial Hazard Risk
Sprained WristMajor
(3)Probable
(B)
Head injuryHazardous
(2)
Extremely Improbable
(E)
3B -High
2E - Medium
3B - High
Assess Risk
FEDERAL AVIATIONADMINISTRATION
HAZARD: ABC-01
ABC-01
HAW Ice Example: Initial Risk
529/27/2017 Back-Up Slides
Assess Risk
FEDERAL AVIATIONADMINISTRATION
HAW Ice Example: Treat Risk
539/27/2017 Back-Up Slides
(13)Safety
Requirements
(14)Organization Responsible
forImplementing Safety
Requirements
(15)Predicte
dResidual
Risk
(16)Safety
Performance Targets
HAZARD: ABC-01
Establish sand buckets at each end of the sidewalk. Custodian will sand the sidewalk each hour when the hazard is present.
Post facility-wide warning of icing
Each employee will receive a briefing on the alternate route to follow if the hazard is present.Facility management will budget the cost of sidewalk anti-icing materials. Jane Smith, Director
Jeene Smith, ABC Custodian
Joe Smith, ABC Manager3C
Medium
Fewer than two falls each winter due to slipping
on ice.
Treat Risk
FEDERAL AVIATIONADMINISTRATION
ABC-01
HAW Ice Example: Predicted Residual Risk
9/27/2017 Back-Up Slides 54
Treat Risk
FEDERAL AVIATIONADMINISTRATION
55
Monitoring Plan Template
9/27/2017 Back-Up Slides 55