aarmstrong_project_January2010Page i
DEVELOPMENT OF A METHODOLOGY FOR DERIVING SAFETY METRICS FOR UAV
OPERATIONAL SAFETY
PERFORMANCE MEASUREMENT
Andrew J Armstrong
This report is submitted to satisfy the project requirements of the
Master of Science in Safety Critical Systems Engineering
at the Department of Computer Science
January 2010
Number of words = 40,088 as indicated by the Microsoft Word ‘word
count’ tool. The count includes the title page, preliminaries,
report body, and the references.
Page ii
Abstract There is increasing potential for missions utilising
Unmanned Air Vehicle Systems (UAS) that will require them to access
the same airspace as manned aircraft. Currently the accident rate
exhibited by UAS is perceived to be too high and safety
improvements are required. A literature review has identified the
following operational safety issues:
• the accident history of UAS shows that many contributory causes
are due to airworthiness and human factors (HF) issues prevalent in
operating UAS;
• operations of manned aircraft domain can be argued as
sufficiently safe by
operational safety cases (OSC) which use the Goal Structuring
Notation (GSN) to express safety claims, goals and evidence in a
structured hierarchical way; and,
• there is limited evidence of suitable metrics for UAS operational
safety parameters.
A safety performance methodology has been developed to enable the
derivation of safety metrics from fragments of an OSC GSN within
the ARP5150 safety assessment framework. The methodology has been
tailored to identify suitable performance metrics for UAS and
validated by application to a UAS case study. The project has
concluded that:
• The direct derivation of safety performance metrics from current
OSC GSN fragments is not straightforward.
• A goal based method for deriving metrics can be used in
conjunction with GSN safety
cases to more effectively identify metrics.
• Existing GSN constructs can be assessed and annotated with review
symbols to identify revisions aimed at improving metrics
derivation.
• In a case study it was demonstrated that UAS specific metrics
could be identified
using the methodology developed in the project, with the caveat
that the military aviation SMS is broadly equivalent to that
mandated in civil aviation under ICAO 9859 [Ica06].
Page iii
Statement of Ethics Ethics in Student Projects [Yor08] identifies
the basic principles as:
1. Do no harm; 2. Informed consent of human participants in
project; 3. Preserve the confidentiality of data held on
individuals.
To do no harm to anybody taking part in the project they should not
be put in a position of physical danger or asked to do anything
which is illegal or against their best interests. In producing the
project, literature and case studies have been chosen mainly from
extant material in the public domain. Where not in the public
domain, permission has been granted for use. Therefore nobody has
been put in a position of danger, illegality or been forced to act
against their best interests during the production of the project.
The project is theoretical and examines GSN argument fragments for
the purposes of defining safety performance metrics. As a result of
reading this project it is possible that someone may be influenced
to modify their approach to the work that they perform in a safety
critical industry. It must be stressed that the work in this
project reflects the personal views of the author and does not
constitute part of any regulation, standard or requirement. The
work has been undertaken for the purposes of academic study only.
Anyone who reads this project must proceed with caution if
attempting to apply any of the findings in practice and they are
strongly advised to seek independent opinion from suitably
qualified and experienced practitioners before doing so. As there
are no active participants in the project, and as it is
theoretical, no informed consent is required; no data has been
taken regarding active participants and so confidentiality of data
is not applicable.
Acknowledgements I would like to thank the following people without
whose help and encouragement I would not have been able to complete
this project. I thank my parents Andrew and Margaret who have
always encouraged me in my studies and passed on their value of
lifelong learning. I also thank Richard and Rhiannon for providing
practical assistance with a quiet space in which to work. I thank
Dave Venn and Phil Barwell of QinetiQ Ltd for supporting my request
to move into systems safety, and my colleagues Colin Blagrove, Rod
Angel and John Gallacher for their support. I thank Mike Cusack of
the Tornado Project Team for permitting the use of the Tornado
specific project GSN examples. I also thank Sqn Ldr Kevin Keen of
DARS for discussion on Operator top level safety management issues.
I thank my project supervisor Dr Mark Nicholson for his advice,
guidance and encouragement throughout the development of this
project and for the many stimulating discussions we have enjoyed,
on systems safety management, during the course Finally I would
like to thank my wife, Elizabeth, and my children Timmy, Rhydian
and William for their patience, support and giving me the time I
required to produce this project.
Page iv
2 Literature Search..................................
................................... 3 2.1 Unmanned Air Vehicle
Systems (UAS)
........................................................................
3
2.1.1 Definitions of a UAV and UAS
.............................................................................
3 2.1.2 Applications for UAVs
..........................................................................................
5 2.1.3 UAV Benefits and Capability Development
......................................................... 5 2.1.4
Constraints affecting UAV/UAS integration in non- segregated
airspace ........... 7 2.1.5 Safety Regulation of UAVs
..................................................................................
7 2.1.6 UAV Hazards and Safety Objectives
.................................................................
11 2.1.7 UAV Accident
rates............................................................................................
12 2.1.8
Autonomy...........................................................................................................
13 2.1.9 Work by York University MSc Students in UAV safety.
..................................... 16 2.1.10 Summary of Sections
2.1.1 –
2.1.9....................................................................
16
2.2 Operational Safety
Management................................................................................
17 2.2.1 The Origins of the Safety
Case..........................................................................
18 2.2.2 The Requirements for Operational Safety
Cases.............................................. 20 2.2.3 OHSAS
18001
...................................................................................................
22 2.2.4 Alternative OSC structures for
UAS...................................................................
23 2.2.5 Accident – Cause Models
..................................................................................
25 2.2.6 Military Aviation Safety Management
Systems.................................................. 26 2.2.7
Civil Aviation SMS - ICAO 9859 Safety Management Manual
.......................... 27 2.2.8 Civil Aviation SMS in the US
and Canada.........................................................
27 2.2.9 Summary of sections 2.2.1 - 2.2.8
.....................................................................
28
2.3 Safety Performance
Measurement.............................................................................
28 2.4 Safety
Monitoring........................................................................................................
30
3 Developing a Methodology for Safety Performance Measurement
........................................
...................................... 40
3.1 Introduction
.................................................................................................................
40 3.1.1 Step 1 – Consider the components of a GSN fragment
.................................... 41 3.1.2 GSN Fragment
Searching Process
...................................................................
43 3.1.3 Step 2 – Select generic GSN fragments & define
metrics................................. 45 3.1.4 Step 3 Evaluate
the results
................................................................................
53 3.1.5 Step 4 –Fixed wing instantiation of the OSC fragment G2.2
............................ 55 3.1.6 Step 5 - Evaluation and
Methodology Revisions
.............................................. 58
3.2 Summary of
Methodology...........................................................................................
60 4 Case Studies.......................................
................................... 61
4.1 Case Study 1 – High level Aircraft Operator Safety Argument
.................................. 61 4.2 Case Study UAV MQ9-
Predator B – “Counter factual” GSN example...................... 68
4.3 Summary
....................................................................................................................
76
5 Project Conclusions................................
.............................. 77 6 Future Work
........................................
................................... 79 7 References
.........................................
.................................... 80
Page v
Figure 1 - Diagram of a UAV showing the main functional
interfaces....................................... 4 Figure 2 –
Examples of the range of UAVs by weight categories replicated from
[Wei04]....... 6 Figure 3- UAV Autonomy levels from HERTI program
[Wil09]................................................ 14 Figure 4
– Aircraft system operational safety case modelled on the NATS
approach............ 24 Figure 5– ARP 5150 Safety Assessment
Process
..................................................................
34 Figure 6– Fragment of Generic
GSN.......................................................................................
41 Figure 7 – Example GSN for Systematic GSN review
process............................................... 44 Figure 8
–Process for searching nodes in the GSN fragment of
interest................................ 45 Figure 9 – Top level
goal for the generic OSC.
.......................................................................
46 Figure 10 – OSC Legislation and Regulation Pattern
............................................................. 47
Figure 11 – Generic OSC Risk Management Pattern
............................................................. 48
Figure 12 – AOA safety argument High level GSN
.................................................................
62 Figure 13 - Supporting goal structure to S11
..........................................................................
64 Figure 14 - AOA safety argument Strategy
S12......................................................................
65 Figure 15 – AOA safety argument Strategy S13
.....................................................................
66 Figure 16 – AOA safety argument Strategy S14
.....................................................................
67 Figure 17 –Predator UAV
types...............................................................................................
68 Figure 18 - Predator Ground control
station............................................................................
69 Figure 19 - Top level representation of the “counter factual”
safety argument ....................... 71 Figure 20 – Unsafe
organisational influences “counter factual” GSN
..................................... 72 Figure 21 – “Unsafe
supervision issues” counter factual GSN
............................................... 73 Figure 22 -
Preconditions leading to unsafe acts GSN
counterfactual.................................... 74 Figure 23 -
Unsafe Acts GSN counterfactual
..........................................................................
75
Table 1 –Interim classifications of UAS from CAP722 [Caa08]
................................................ 8 Table 2 –
Evaluation of the suitability of goal based metrics to a GSN based
OSC............... 36 Table 3 – example of GQM approach from [
Bas94]...............................................................
37 Table 4 – Results of metric derivation for Figure 11 OSC risk
management. ......................... 50 Table 5 - The GQM method
applied to goal G2.2a (operational risks are
identified)............. 51 Table 6 - The GQM method applied to
goal G2.2b (operational risks are assessed)............. 52 Table 7
- The GQM method applied to goal G2.2c (operational risks are
properly managed) 52 Table 8 – Proposed Metrics <Mark-up> for
GSN reviews......................................................
54 Table 9 – Description of <Operator> from [Hrm07].
................................................................ 54
Table 10 - Tornado “top level” goals for risk management using the
direct derivation of metrics method
........................................................................................................................
57 Table 11 – Analysis of nodes from Tornado OSC below Goal
G2.2.1.2................................. 58 Table 12 – Summary of
case study
requirements...................................................................
61 Table 13 – Metrics derivation for AOA safety argument (high
level strategies) ...................... 61 Table 14 - Metrics
derivation for AOA safety argument (top goal)
.......................................... 63 Table 15– Proposed
metrics and analysis for goals under Strategy S11
............................... 64 Table 16 –Proposed metrics and
analysis for goals under Strategy S12
............................... 65 Table 17 – Proposed metrics and
analysis for goals under Strategy S13 .............................
66 Table 18 - Proposed metrics and analysis for goals under
Strategy S14 ............................... 67 Table 19 –
Specifications of predator UAV
.............................................................................
68 Table 20 - HFACS analysis from Carrigan [Car08]
.................................................................
70
Page 1
1 Introduction
1.1 Background There is increasing demand and commercial potential,
for using Unmanned Air Vehicle Systems (UAS) for airborne missions
that will require them to access the same airspace as manned
aircraft. Currently the majority of UAS operations are limited to
segregated airspace where operations can be conducted without
threat of harming other airspace users or third parties on the
ground. Access to non-segregated airspace will require UAS to show
an equivalent level of safety (ELOS) as manned aircraft and appear
transparent to other users and Air Traffic Management (ATM). In
support of emerging regulation and the need to operate UAS within a
civil safety management system it is apparent that effective safety
performance measurement will be an important future enabler in
support of the demonstration of adequate safety achievement.
1.2 Motivation and Aims for the project Many civil variants of UAS
are, or will be, based on military designs and so re-use of safety
evidence to qualify against civil certification requirements can be
expected. Military UAS are operated within a safety management
system that emphasises a risk management approach to safety.
Operational Safety Cases (OSC) are used to argue that the design is
sufficiently safe to operate in a given environment for a given
task. However, in complying with emerging regulations, UAS will be
required to demonstrate adequate levels of safety in accordance
with civil standards and the SMS. Based on the above scenario, the
aim of the project was to identify and begin to address the
requirement for a safety performance measurement method based on an
existing OSC to provide information to suit the requirements of
civil standards and emerging UAS regulations. It is recognised that
OSC arguments are expressed in Goal Structuring Notation (GSN) that
links safety objectives (goals) to evidence (solutions) in a
hierarchical way. The aim of the project is to derive safety
metrics from this pre-existing material as it should enable
measurement of the key objectives within the context of the safety
case.
1.3 Project Scope and Limitations The scope of the project is to
examine fragments of GSN arguments from pre-existing operational
safety cases to determine if it is possible to identify suitable
metrics to enable safety performance measurement for UAS. A
limitation is that the extant material may not be concerned with
UAS specifically. It is assumed that this is not sufficiently
limiting to prevent a general method being developed as a first
step. Once a method has been established, a case study that
includes UAS specific safety arguments (expressed in GSN) should be
suitable to validate the method. The majority of the OSC fragments
are derived from military applications as these are the most
commonly found examples of UAS usage. This presents a limitation
for civil applications because such OSCs have been developed to fit
within a military aviation SMS. In principle it is assumed that the
basis of military and civil SMS are close enough that OSC material
from the military domain is a suitable basis on which to define
metrics for use within a civil SMS. This assumption is based on
recognition that the underpinning principles of SMS are based on
general systems safety standards that are very similar. In
particular the ICAO Safety Management Manual (SMM) 9859; Aerospace
Recommended Practice ARP5150 - Safety Assessment of Transport
Airplanes in Commercial Service; CAP722 and CAP740 are
Page 2
considered to be the primary applicable standards of interest that
define the civil environment for UAS integration in non-segregated
airspace relevant for metrics development. The scope of the project
includes utilising existing methods to derive metrics from GSN
fragments. It is considered out of scope to source information
external to the safety case GSN to define a metric, as this would
not be faithful to the objectives of the project. Within scope is
considered demonstrating inadequacies with a current OSC approach
that affect metrics derivation. It is considered to be out of scope
to create new OSC constructs or judge whether an OSC is fit for
purpose. The scope of the literature review is to: identify the key
UAS safety issues that will determine the requirements for safety
performance measurement; understand the basis for operational
safety cases; and, identify the role of safety performance
measurement within the overall SMS. This is needed in order to
understand the application and context for metrics development. The
methodology proposed by the project will be evaluated by case
study. Such case studies will need to be based on a UAS or have
applicability to a UAS.
1.4 Report Structure Section 2 covers the literature search which
includes the topics of UAVs, Operational Safety, Safety Management
Systems and Safety Performance Measurement. This section describes
the findings from the literature review and justifies the
subsequent approach taken by the project in the design and
implementation phases. Section 3 describes the development of the
methodology for the derivation of safety metrics by considering the
application of the first step of the ARP5150 [Arp03] process to
suitable GSN expressions of safety arguments. Section 4 describes
the results and evaluation of applying the devised methodology to
two different case studies. Case study 1 comprises a top level
organisational safety argument from an aircraft operator‘s
viewpoint and is based on pre-existing fragments of GSN. Case study
2 is a UAV GSN based on a “counter-factual” argument constructed by
the author based on information from published literature. A
counter factual argument is an argument demonstrating the absence
of safety. It was necessary to do this as GSN safety arguments for
UAS that were sufficiently detailed were not available. Section 5
identifies future work based on the findings of the literature
search, methodology development and evaluation by case
studies.
Page 3
2 Literature Search This section describes the findings from the
literature review and justifies the subsequent approach taken by
the project in the design and implementation phases.
2.1 Unmanned Air Vehicle Systems (UAS) The purpose of the
literature search on Unmanned Air Vehicles (UAV) is to understand
the relevant safety management issues that directly affect the
feasibility of flights in non- segregated airspace and thus
determine which areas of UAV safety performance measurement it
would be appropriate to investigate.
2.1.1 Definitions of a UAV and UAS In CAP722 [Caa08] the UK CAA
provide advisory definitions for the terms UAV and UAS, and also
list definitions used by the military as found in Joint Service
Publications (JSP) 550 [MOD06] and JSP 553 [ MOD08]. A UAV is
defined as “An aircraft which is designed to operate with no human
pilot on board as part of a UAS” [Caa08] section 2.1. Where, “An
Unmanned Aircraft System (UAS) comprises individual 'System
elements' consisting of the unmanned aerial vehicle (UAV), the
Ground Control Station (GCS) and any other UAV System Elements
necessary to enable flight, such as a Communication Link and Launch
and Recovery Element. There may be multiple UAVs, GCS or Launch and
Recovery Elements within a UAS.” CAP722 acknowledges that the GCS
may be on board a ship or land based. The European Aviation Safety
Agency (EASA) [EAS05] provides a very similar definition but
includes a description of the applicable phases of flight as well
“taxiing, takeoff and recovery/landing” Military definitions of a
UAV are very similar but include the military purposes of a UAV.
This is exemplified by JSP 553 [MOD08] which states for a UAV: “A
UAV is defined as an aircraft which does not carry personnel and:
is capable of sustained flight by aerodynamic means; is remotely
piloted or automatically flies a pre-programmed flight profile; is
reusable; is not classified as a guided weapon or similar one-shot
device designed for the delivery of munitions.” The UK MOD does not
use the acronym UAS instead it defines the term UAVS as the
Unmanned Air Vehicle System. In this project the acronym UAS will
generally be used. The US military definition of a UAV is given in
the US DoD Joint Publication 1-02 DoD Dictionary [Usd01] and is
very similar to UK definitions with the exception that it allows
for expendable use. In order to illustrate the above concepts, a
typical architecture for a military UAV is shown in Figure 1
replicated from the US Office of the Secretary of Defense (OSD)
Unmanned Aircraft Systems Roadmap [Osd05]. The UAS comprises a
vehicle and payloads, a command and control system and
communications architecture. Communications are achieved beyond
line of sight (BLOS) via a satellite communications (SATCOM) link
and by line of sight (LOS) from a control station (CS).
Page 4
Figure 1 - Diagram of a UAV showing the main functional interface s
The UAV is a fully functioning, airworthy, flight capable vehicle
comprising functions for flight control, payload control, weapons
employment and situational awareness. The flight control function
is necessary to maintain or change the required flight path and
other flight parameters such as velocities, height, and
accelerations in response to externally received commands or
internally from an autonomous control function. The payload control
and product dissemination functions serve to switch on and control
onboard sensors. This normally involves still photography or Full
Motion Video (FMV) collected from optical or Infra Red (IR) sensors
and the subsequent packaging of collected optical or video data for
transmission to a ground station. A UAV which has been designed
specifically for the military requirements of combat or strike
missions is called an Unmanned Combat Aerial Vehicle (UCAV). It
should be noted that some commentators make a distinction between a
UAV that has a weapon added as a secondary function and a purpose
designed strike capable UCAV. For example Predator B was originally
designed to be a surveillance UAV and has had weapons added (e.g.
hellfire missiles) by modification. This is primarily a UAV and
known as such. For development projects a CS can be as
straightforward as a laptop linked to a transmitter / receiver for
LOS operations conducted on a range. For production systems the CS
is likely to be more sophisticated and can comprise UAV pilot
(UAV-p) and camera controller stations linked to multiple
communication data links. Control may be achieved by the UAV-p and
a sensor operator working together to interpret information
received from the data-link. Control may be handed over from one
UAV-p to another. One UAV-p may control or supervise the operation
of the UAV during the mission phase of flight. A different pilot
may be employed during the take-off, climb, cruise and
approach/landing phases of flight. In such cases, control handovers
from one pilot to another are required to maintain continuous
flight and for endurance missions handover may be necessary to
relieve pilots in accordance with duty rosters. The UAV-p is
defined by CAP722 [Caa08] as “the person in direct control of the
UAV”. In this role the UAV-p monitors the data provided by the
flight control and situational awareness functions and provides
control commands back to the UAV to ensure that it follows the
required flight path. There is also the term UAS Commander defined
that allows for a fleet of UAVs in a UAS being commanded by a
single supervisor. The UAV-p is the direct operator of the UAV;
however the term operator has another meaning in the context of the
organisations that operate the UAS. The term Operator or Air
Operator is assumed to be the airline in the civil standard ICAO
9859 Safety Management Manual [Ica06] paragraph 2.1.13. Another
civil
Page 5
standard, CAP722 [Caa08], defines the operator as the “legal entity
operating a UAV System”. The operator is actually considered to be
one stakeholder in the complex system of aviation. In civil
aviation “operations” are described as being dependent upon service
providers in addition to, and separate from, the operator. Service
providers are described in [Ica06] as comprising air traffic
management; aerodrome operations, including airport emergency
services; airport security; and navigation and communication
aids.
2.1.2 Applications for UAVs The US Road Map [Osd05] describes the
main applications for UAVs as fulfilling the dull, dirty and
dangerous missions as this relieves humans of these categories of
missions providing economic, reliability and safety benefits. An
example military mission where UAVs can offer benefits is
persistent surveillance, conducted behind enemy lines. Such a
mission requires extended time on task, in order to capture
photographic or video evidence of potential targets or threats, and
there is a high risk of attracting hostile enemy action. Clearly
this mission can be both fatiguing and dangerous for human beings
in manned aircraft. Using a UAV for flights over enemy territory
reduces the direct safety risks to aircrew but can introduce a
security risk, if the UAV were to be captured by the enemy that may
precipitate a hazardous mission to recover or destroy it [Osd05].
There are many potential civil applications for UAVs cited in the
literature. For example De Garmo [Deg04] investigated and
summarised a number of potential civil applications reported by
others (e.g. reported Frost and Sullivan briefings not available in
the public domain). Many of these key applications for civil UAVs
in the USA derive from the need for enhanced “Homeland Security”
provisions following the 9/11 attacks on the world trade centre.
Typical security applications include border patrol, monitoring of
sensitive sites, drug surveillance and interdiction, domestic
traffic surveillance, pipeline patrol and port security. Other
roles reported by De Garmo [Deg04] include: emergency response, law
enforcement surveillance, search and rescue, forest fire
monitoring, flood mapping, nuclear biological chemical (NBC)
monitoring and chemical and petroleum spill monitoring. One notable
success for UAVs in the civil market is their use in agricultural
crop spraying in Japan where it is reported that around 2000
vehicles are used commercially [Osd05].
2.1.3 UAV Benefits and Capability Development UAVs vary
considerably in physical size, weight, range, speeds, application
and endurance. An illustration of the range of weights and types of
contemporary UAVs is shown in Figure 2 replicated from Weibel and
Hansman [Wei05]. At one extreme are small hand portable devices,
weighing a few ounces and fitted with miniature cameras, that can
be used by individual soldiers to observe what is in a building or
in the next street beyond their immediate line of sight. At the
other extreme are the much larger, heavier and more complex Medium
Altitude Long Endurance (MALE) and High Altitude Long Endurance
(HALE) UAVs, such as those used for persistent surveillance
missions.
Page 6
Figure 2 – Examples of the range of UAVs by weight categories r
eplicated from [Wei04] One set of benefits of a UAV is that it will
not experience flight control or navigation failure modes due to
pilot fatigue, failed life support systems, degraded visual
conditions or the presence of smoke in the cockpit. For a UAV that
lands or takes off under autonomous control, there is no pilot
indecision failure mode possible [Osd05]. However the removal of
the pilot may lead to the design organisation reducing the level of
system redundancy or using lower quality components than were
previously specified for aircrew safety, thereby putting
affordability before reliability and airworthiness considerations
[Osd03]. A significant contribution to combat aircraft design costs
and operating limitations is due to the need to provide a safe
environment for the human aircrew that operate the aircraft.
Removing the pilot from an aircraft removes the need for life
support systems, a cockpit or flight deck and is reported as saving
3000-5000lbs mass from an aircraft [Osd05]. This permits the UAV
designer to expand the flight envelope considerably in terms of
speeds, accelerations (g- levels) and manoeuvrability giving the
prospect of enhanced survivability for certain military missions.
[Osd05], [Wez07]. Within the last decade there have been some
noteworthy illustrations of UAV developments. In August 2001 the
NASA Helios prototype UAV set an altitude record of 96,863 ft. This
was a prototype and it remains a challenge to design a practical
air vehicle capable of managing to climb to such an altitude,
sustaining a cruise and carrying a useful payload [Cox04]. In 1998
the Aerosonde Mark 1 “Laima” became the first UAV to cross the
Atlantic Ocean achieving an altitude of 1,680 m, and completing the
journey of 3270 km in 26 hours and 40 minutes using just 7 litres
of fuel [Bar98]. This clearly demonstrated the potential
capabilities of endurance and lower cost of operations when
compared with manned aircraft. Perhaps more significantly, in terms
of technology readiness and regulatory readiness, on the 18th
August 2005 BAE Systems achieved the first CAA approved fully
autonomous mission of a UAV in UK airspace [Wil09]. Cox et al
[Cox04] provides information and analysis of US civil technology
developments. For example the NASA led Autonomous Robust Avionics
(AuRA) programme is developing technology to enable aircraft to fly
with reduced or no human intervention; to include the capability to
optimize flight over multiple regimes, and the ability to provide
maintenance on demand. Three main components of AuRA are
Intelligent Mission Management (IMM), Integrated Systems Vehicle
Management, and Adaptive Flight Controls. Autonomous control
capability is also a significant enabler for the military where one
of the aims of technology
Page 7
development is to delegate the basic flying of the vehicle to
autonomous control and thus enable humans to concentrate on mission
decision making tasks [Cox04] In the European Union the most urgent
requirements were considered to be for increased numbers and
capabilities in MALE and HALE UAVs for their long endurance
surveillance capabilities and UCAV for their enhanced performance
in dangerous missions [Wez07]. All current European UCAV programmes
are reported to be technology demonstrators. For example UK
industry is leading the development of Taranis and Corax [Wil09].
Taranis is a demonstrator programme the size of a small combat
aircraft powered by a turbofan engine and will have
intercontinental range. Corax is essentially a stealth variant of
the same UCAV and large enough to be functional. Such developments
pose a challenge for designers and operators in considering how
such future UAS can be practically catered for within the existing
airspace management arrangements. These UCAVs are likely to be
flying long range, hard to detect, and operated under autonomous
control. Their performance in terms of speeds, accelerations and
manoeuvrability is likely to be far in excess of the Typhoon
aircraft. Typhoon currently represents the upper limit of
segregated airspace protocol classification listed in CAP740
[Caa07b] Chapter 2 Annex A. It is not likely that such a protocol
will apply to all of these future UCAVs. For more immediate
requirements, the UK has made acquisitions of UAS for Intelligence
Surveillance Target Acquisition and Reconnaissance (ISTAR) roles
for defence purposes, with the intended theatres of operation
including Iraq and Afghanistan [Ukg08]. Key UK UAV programmes for
current operations have involved UOR acquisitions of Reaper
(formerly Predator B), Hermes 450 and Desert Hawk 3 (DH3). In
future scenarios it is predicted that swarms or teams of UAVs could
operate cooperatively to achieve a given mission purpose [Clo02b].
While an interesting prospect for the future (probably military
missions), there are currently many constraints limiting the use of
a single UAV in non-segregated airspace.
2.1.4 Constraints affecting UAV/UAS integration in non- segrega ted
airspace To fulfil many of the identified civil missions for UAVs
will require them to have routine access to non-segregated
airspace. However there currently exist a number of constraints
that will need to be overcome. With reference to integration of UAS
within the US National Air Space (NAS), De Garmo [Deg04] concluded
that resolution of the following key issues is necessary: 1. The
need for a consensus on operational concepts, definitions, and
classifications of
UAVs 2. The requirement for certification standards and regulations
to address UAS operations
and operator qualifications 3. The provision of effective and
affordable collision avoidance systems capable of detecting
non-cooperative airborne threats (e.g. aircraft not fitted with
transponders) 4. Improvements in the reliability of UAS and
operations 5. Provision of a suitable protected frequency spectrum
for communications – considered
out of scope for this project. 6. High insurance liability costs –
considered out of scope for this project 7. High acquisition and
operational costs – considered out of scope for this project In a
separate study, Cox et al [Cox04] investigated and summarised a
number of capability and technology issues that are required as
part of the perceived solution to achieving integration with manned
aviation. They identified that in order to achieve a similar level
of reliability to the human pilot of a manned aircraft the UAS
would need to demonstrate both system reliability (minimised
component failures) and an onboard intelligent decision making
capability. The issues identified were: autonomous mission
management, collision avoidance, intelligent system health
monitoring and reliable flight systems.
2.1.5 Safety Regulation of UAVs The CAA Directorate of Airspace
Policy has published CAP722 “Unmanned Aircraft System Operations in
UK Airspace – Guidance” [Caa08]. This document, now in its third
edition,
Page 8
provides advice to the developers of a UAS on how to identify the
route to certification and ensure that all the relevant standards
and regulations are complied with. It also provides guidance on the
safety requirements for airworthiness and operational standards
that have to be met if a UAV is to be permitted to fly in UK
airspace. Currently in the UK, UAV flights beyond “the limits of
visual control” are not permitted outside of segregated airspace.
For all UAVs that are permitted to fly, CAP722 states: “It is CAA
policy that UAS operating in the UK must meet at least the same
safety and operational standards as manned aircraft.” [Caa08]. This
is true for all aspects of UAS operations, in the air and on the
ground, that they meet the same safety standards as equivalent
classes of aircraft. This is known as Equivalent Level of Safety
(ELOS). Evans [Eva06] reviewed UAV regulations and he concluded
that whereas the majority of UAV regulation is based on the ELOS
principle, there was not much guidance provided on how this should
be achieved. Therefore a potential difficulty for designers and
operators of UAS is how to demonstrate compliance with ELOS such
that the regulators will issue permits to fly. This has
implications for the measurement of safety performance. In order to
issue a permit to fly the authorities will need to be convinced
that the safety performance will meet ELOS. However if the
contributory processes and the associated measures of success are
not fully known how can an operator provide the requisite evidence
to justify that operations will be safe? The starting point for
ELOS is to determine the correct classification of the subject UAV
against an appropriate civil aircraft category. At present there is
no internationally agreed classification system for UAS although
one recommended way forward is to base such a classification scheme
on how the UAS will be operated in civil airspace [Deg04]. The UK
CAA [Caa08] acknowledge that the process of designing a
classification system is not yet complete and as an interim measure
they have provided guidance on a suitable classification scheme as
shown in Table 1.
Table 1 –Interim classifications of UAS from CAP722 [Caa0 8]
Although the CAA show interim classifications these are not
actually related to manned aircraft categories in all cases and so
it is not possible to identify the ELOS requirements from Table 1.
Classification depends instead on calculation of kinetic energy
levels. For instance CAP722 requires UAVs in class 2 (20-150kg
mass) to have a kinetic impact energy of less than 95kJ. Guidance
published by Haddon and Whittaker [CAA02] considers equivalence
from a comparison of kinetic energies between UAVs and equivalent
aircraft classes. Their approach recognised that an air vehicle can
harm third parties on the ground in proportion to its level of
kinetic energy upon impact. There were two impact scenarios
considered in calculating levels of kinetic energy. The
“unpremeditated descent scenario” was defined as a failure
condition that results in the inability of the air vehicle to
maintain a safe altitude above the ground. The “loss of control
scenario” was defined as a failure condition that results in a loss
of control that may lead to a high impact velocity. Haddon and
Whitaker derived formulae for calculating kinetic energies for each
of these scenarios and produced results and exemplars for different
categories of air vehicle (e.g. JAR 23 and JAR25) in terms of
kinetic
Page 9
energies. In order to determine equivalence the kinetic energies
for both scenarios “unpremeditated descent” and “loss of control”
were calculated for the subject UAV and compared with the results
for the baseline aircraft categories. The ELOS principle is stated
as a set of requirements such that a UAS will appear to act and
behave to all other users (and operators) of airspace as though it
were a manned aircraft. [Caa08]. This is stated in terms of the
harm the UAS can potentially inflict to other airspace users and
third parties on the ground; and in terms of UAS behaviour
characteristics matching those of a manned aircraft (e.g. in
response to threat of collision, response to commands from ATC).
However there are significant differences between UAS and manned
aircraft that have led to the introduction of appropriate new
regulations and the reinforcement and explanation of others so that
ELOS requirements are more fully described. The key source of these
requirements in the UK is CAP722 [Caa08] which will be relevant for
the subsequent investigation of suitable safety performance
metrics. Current regulations limit the risks of operating certain
classes of UAVs by limiting their flights to within segregated
airspace, particularly where a UAV does not comply with the Air
Navigation Order (ANO) [Caa09]. However, individual flights outside
of a danger area (DA) may be permitted by the use of a Restricted
Area (Temporary) RA (T) [Caa08]. This provides exclusive use of the
airspace defined by the RA (T), to the UAS operator, and will
consequently temporarily deny airspace use to other users. This is
not a practical means for achieving routine flights in
non-segregated airspace as the use of a RA (T) is actually a form
of segregation. Generally the provision of Air Traffic Services
(ATS) must be transparent to the Aircraft Controller and s/he
should not have to do anything different in respect of
communications, rules or procedures from that required for a manned
aircraft. Furthermore the UAS is required to comply with any
instructions issued by ATS and should be equipped for the class of
airspace in which it operates. One notable difference is that the
UAS should include the word “UNMANNED” on the first communication
with ATS. [Caa08]. A further requirement is to comply with
instructions given by ATS in a similar timeframe as an equivalent
manned aircraft. This raises the question about what is a suitable
timeframe. For a manned aircraft the typical time it takes for the
pilot to respond to a time critical call from ATC (e.g. to
manoeuvre to avoid a collision) has been measured and analysed in a
study by Cardosi and Boole [Car91]. They studied the results of 80
manoeuvre calls occurring in 46 hours of recorded en-route time.
They measured the durations for each stage of the communication
process. The total communication time varied from 4 to 40 seconds
with a 50th percentile median time of 9 seconds. In theory this
sets a benchmark for a UAS to demonstrate equivalence. Some of the
individual processes within the total time are relevant for
determining the equivalence standards for UAS. The duration of the
initial call from ATC varied between 1 and 11 seconds with a mean
of 4.85 seconds. The lag before a pilot’s first response varied
from 1 to 31 seconds with a mean of 3.31 seconds and the pilot’s
response time varied between 1 and 11 seconds with a mean of 2.61
seconds. Regulations require a response from the UAS of a “few
seconds” to ATC commands [Caa08] and this agrees with the manned
pilot measures of the study [Car91]. Some other interesting aspects
of the study were to note that 6% of first calls received no pilot
response and 14% required a second call by ATC to clarify. In these
cases the total communication time took a further 4.98 seconds on
average to complete the second ATC call and pilots response.
Interestingly there was no statistical significance between high
and low workload data i.e. the time to respond was independent of
pilot workload. The study provides some interesting results for ATC
and pilot commands that could be interpreted as setting standards
that UAS have to achieve for equivalence. A UAS and a manned
aircraft both may not respond to local ATC commands. Both the UAS
and the aircraft will be tracked by ATM radar. However the UAS
presents alternative means of communication with the UAV-p as s/he
is located on the ground. What is not clear is if a non- voice
means of communication between ATM and a UAS could achieve a faster
means of time critical calling. The known benchmarks for manned
aviation set the minimum standards
Page 10
that the UAS must achieve. For a remote UAV-p the latency of the
voice communication link plus latency of the command link to the
UAV will be an additional factor in retaining control of the system
and dependability measures for communications will be an important
area for UAS [Mou01]. For manned aircraft the pilot has a role in
mitigating the hazard of an air to air collision by implementing
the rules of the air by the “see and avoid” approach. For UAVs to
be operated in non-segregated airspace, outside of the LOS of the
UAV-p, this is replaced by the “sense and avoid” requirement of
CAP722 [Caa08]. These require an equivalent capability to human
“see and avoid”, and this need is open to challenge. There is
evidence that the pilot “see and avoid” capability is significantly
unreliable as a means of preventing air to air collisions. De Garmo
[DeG 04] summarised the work of various sources from the
literature. A significant point was the reported variation between
individual pilots who may spend different times looking out of the
window, who may have different visual abilities and follow
differing scanning techniques. He reported on work conducted by
Lincoln Labs that concluded that pilots were not good at
identifying potential collisions if they were not previously aware
of aircraft in their vicinity. De Garmo quoted FAA data that showed
that many air to air collisions took place in daylight near
uncontrolled airports. Based on this evidence, De Garmo challenged
the value of basing a sense and avoid requirement on an obviously
poor human equivalent. A sense and avoid capability actually has
the potential to be much better than a human in terms of field of
view, detection reliability, persistence and tolerance to different
weather conditions. The key word appears to be potential, because
there is widespread acknowledgement that there is, as yet, no
feasible sense and avoid system available [Deg04]. In addition to
avoiding air to air collisions the UAS must be capable of avoiding
collision with the ground for obvious reasons of affording
protection to third parties. Within Chapter 2 of CAP722 the
capability to “detect terrain and other obstacles” is included
within the general policy on sense and avoid. Other aspects
included are the avoidance of hazardous weather, and performing the
functions of safe separation from other aircraft. There is little
further guidance on how to achieve a compliant sense and avoid
system and the CAA consider it is the role of industry to conduct
the necessary research and development of suitable systems for
employment in UAS. For operations in non-segregated airspace there
is a requirement for Standard Operating Procedures (SOP) for
take-off and landing procedures; en-route procedures; loss of
control data link; and, abort procedures following critical system
failure. These are designed to provide for protection against
infringing other airspace users and for protecting third parties.
These seem sensible and appropriate and are similar to current
practice for manned aircraft. Within segregated airspace, such as
UK Danger zones, the UAV has exclusive use and thus there are
greater freedoms such as enabling the controlled flights of
prototype UAVs that may be substandard in terms of airworthiness.
However requirements may be enforced such as: communicating with
ATS, and procedures for emergency recovery, loss of control link
and for the avoidance of infringing aircraft. Within a danger area,
such as a UK range, it is possible to exclude the presence of third
parties in the air and on the ground and by range control
procedures for staff operating the UAS and the range. As there are
no third parties present on the ground a suitable independent
flight termination system (FTS) can be employed. For example in the
event of certain failure modes of the UAV the FTS could cut-off the
fuel supply to the engine, trim the flight control surfaces and
return the UAV to the ground in a controlled descent by deploying a
parachute [Wei05]. CAP722 does not cover model aircraft, which are
provided for by CAP658 [Caa07a]. Some UAVs are small enough to
qualify as model aircraft and are flown as such. These UAVs are out
of scope for this project. CAP722 does not cover UAS used for the
purposes of delivering a weapon and UCAVs and UAVs with weapons
capability are outside the scope of this project.
Page 11
Summary of safety regulation issues in respect of safety monitoring
The CAP722 requirements discussed above appear to present operators
and developers with significant challenges for operating UAS in
non-segregated airspace and will influence choices made about such
factors as: the concept of operations; the location of the UAV-p;
the methods of communication employed; the levels of autonomy in
controlling the UAS; the standard operating procedures; the type
and extent of operator and UAV-p training; the integrity of
systems; and, the size and payload capability of the UAV to ensure
it can carry mandated special equipment. Consequently compelling
safety arguments and supporting evidence will be required in the
operational safety case to satisfy the above aspects together with
the identified CAP722 requirements. Additionally the safety
management system will need to include an appropriate safety
monitoring regime to enable the system to be operated safely.
2.1.6 UAV Hazards and Safety Objectives A key objective of aircraft
safety management is to achieve a target level of airworthiness in
the original design and preserve a satisfactory or improved level
of airworthiness through-life during the operation of the aircraft
[Omm08]. This is true for manned and unmanned aircraft in both
civil and military sectors. UAS activity presents a hazard to
people on other aircraft (from air to air collisions) and to people
on the ground. Ground hazard The ground hazard has been analysed by
the application of various ground impact fatality models by Weibel
and Hansman [Wei05] and Clothier and Walker [Clo06]. The approach
reported by Weibel and Hansman uses event tree analysis and a
fatality expectation model to determine the level of reliability in
the UAS necessary to achieve the required target level of safety.
The class of UAV and the area over which it is to operate
determines the required level of system reliability. A conservative
approach is to use a target probability of a fatality rate of 1 x
10-07 per flight hour. From their calculations the key drivers,
limiting where UAVs can be safely flown, are UAV mass and
population density. Micro UAVs can be safely flown in the majority
of US airspace. MALE UAVs are an “intermediate risk” and should be
able to fly over the majority of US airspace but not highly
populated areas. HALE UAVs require a reliability rate of no more
than one class A mishap per 100,000 hrs (equivalent to military
aviation safety levels) to fly over approximately 20% of the USA.
Recognising that agreement on safety targets and ELOS requirements
for UAS were yet to be harmonised Clothier and Walker [Clo06]
investigated and discussed the issues around defining appropriate
safety objectives for UAS aimed at meeting the ELOS requirements in
emerging regulations. They investigated this by reviewing the
actual occurrence of ground fatalities from a survey of National
Transportation Safety Board (NTSB) data and challenging what is
meant by equivalence. They constructed a simple model to show where
a UAV with a given mishap rate per flying hour could crash without
exceeding the safety target in terms of ground fatalities per hour.
Their analysis of the NTSB accident database [Nts05], over the
period 1984 to 2004, revealed that less than 1% of the 27,404
fatalities recorded in the NTSB database were to people on the
ground. The vast majority of recorded fatalities all occurred to
people onboard aircraft. Furthermore they found that observed
involuntary ground fatalities were 3.6 x 10 -08 per flight hour.
based on the FAA 3 year rolling average data. Using a target level
of safety of 1 x 10 -06 fatalities per flight hour they
demonstrated that, for an urban surveillance mission, using
Predator B data (mishap rate 170 X 10-06 mishaps per hour) it was
not possible to justify operations over, or within 20km of,
Brisbane central business district under any feasible safety
objectives examined. Mid-air collision hazard The calculations to
determine the risk of a mid air collision are complex because air
traffic is not uniformly distributed being much denser above large
cities with major airports and in specific air corridors. Analysis,
by Weibel and Hansman, on the probability of mid air
Page 12
collisions involving UAVs in the US National Air Space, concluded
that the ambient collision risk (without mitigation) was of the
order of 1x10-07 per hour in areas away from airways and major
flight levels [Wei05]. Thus they suggest that small UAVs could be
operated in these regions. Flying above scheduled air traffic (at
altitude > 50,000ft) is a low risk region which it is claimed
could be suitable for HALE UAVs on the assumption that suitable
procedures could be developed to enable the aircraft to ascend and
descend through the lower altitudes. In other regions additional
collision avoidance mitigation strategies will be required. Between
flight levels mitigation could be achieved by ATM separation or a
capability in the UAV to avoid other traffic. At altitudes below
5000ft the traffic density is such that ground based radar or line
of sight collision prevention may be required. Summary of Safety
objectives and hazards in respect of safety monitoring The safety
targets that should be set for UAV catastrophic accident rates are
not clear or consistent. In the literature assumptions are made as
to what the target should be and reliability data is used to
calculate where a UAV can be flown. In order to produce a reliable
measure a consistent benchmark will be required. Until a clear
target is set it will not be possible to accurately specify the
required reliability of UAVs and this risks over, or under
estimating the required safety performance for operations.
2.1.7 UAV Accident rates A presentation by Allouche at the UAV –
NET conference in Stockholm 2003 [All01] included qualitative
claims for the historical, current and future predicted
airworthiness safety performance of UAVs against the general
aviation safety level, for the Israeli Industries product range.
Four points of note emerged. First: a combination of design
features are required to drive accident rates down to equivalent
levels of safety to general aviation safety levels. In 1988 the
Pioneer design, fitted with single control analogue flight controls
and a customised engine had significantly lower safety levels than
manned aviation. The Searcher design in 1992 adopted a dual
channel, digital flight control system and redundant communications
resulting in improved safety. In later designs, adding redundancy
to safety critical systems, automating take off and landing and
using FAR 33 compliant engines were all improvements that resulted
in a UAV design matching general aviation safety levels. Second:
the safety gains made followed an exponential curve with a law of
diminishing returns applying. Thirdly: the exponential reduction in
accident rates appears to mimic the published data for manned
aircraft [Wei05] that occurred during the development of the
airline industry. Fourth: It is not clear what the context for UAV
operations was and what the mishaps were (ground or mid air
collisions). Therefore while the reduction in safety is
acknowledged as progress this does not imply that the level of
safety now achieved will be sufficient to permit flights in
non-segregated airspace over a metropolitan area. There is other
evidence in the literature that accident rates for UAVs, in the
military, are much higher than the picture Allouche presents for
civil UAVs. The mishap rates for US UAS are reported to be 10 to
100 times that of manned aircraft [Man04] and Class A mishap rates
of 32 to 334 per 100,000 flight hours are reported by the US
Department of Defense [Osd03]. However trend analysis published by
Weibel and Hansman [Wei05] showed that UAV accident rates since
1987 have been falling and, by extrapolation, are predicted to be
able to match the levels associated with general aviation. In
summary for the larger UAVs the mishap rate is considered to be too
high and this requires improvement before flights in non-
segregated airspace can be fully justified. Following analysis of
UAS accidents [Wil04] it was concluded that electrical and
mechanical reliability of the UAS were as significant as human
errors in the causes of accidents. This was attributed in part to
lower costs of design and production affecting the component
reliability, and system redundancy. The more expensive and better
engineered the air vehicle the more likely it was to be reliable.
It is not appropriate to generalise because there are different
factors and circumstances for each class of UAV [Osd03]. For
example in the case of the smaller UAVs some of the aircraft
failures were attributed to icing due to the aerodynamic
Page 13
properties of smaller sized aerofoils being affected by the
occurrence of a thin layer of ice. Furthermore the effects of
precipitation have a more damaging effect in eroding the leading
edges of thinner aerofoil sections and in penetrating less
effectively sealed compartments. Another trend is the high accident
rate observed for UAVs fitted with wooden propellers, as these are
highly susceptible to rain erosion [Osd03]. Another observation
from UAV accident analysis is the effect of Reynolds numbers
[Sto51] [Rey83] and [Osd03]. Aircraft that fly with a high Reynolds
number tend to crash less often than those with low Reynolds
number. Airliners tend to be larger than UAVs, have higher Reynolds
numbers and are less likely to crash from the associated
aerodynamic effects than UAVs. While this observation is
straightforward to make, it is acknowledged that the detail
aerodynamic properties of the low Reynolds number designs and how
this affects flying qualities are less well understood and this
inhibits the development of suitable design mitigations [Osd03]
From the accidents that Williams studied, a compilation of the
reported human factors issues showed occurrences in the following
categories of: alerts and alarms, display design, take-off error,
landing error, procedural error, aircrew co-ordination, weather,
and pilot in command [Wil04]. These are all significant from an
operational safety perspective requiring: suitable design of the
UAV and GCS; the introduction of suitable procedures and training
for operators and UAV-p; and, appropriate treatment in the safety
case. Suitable metrics for UAS could aim to measure and relate
parameters from these categories of HF issues to the measurement of
overall safety performance. In order to address the identified HF
issues one approach is to investigate the feasibility for
increasing the levels of automation involved in flying the UAV. For
example the US military are investing in research to automate the
take off and landing phases of flight to address the high
occurrence of accidents [Osd05].
2.1.8 Autonomy Employing a high level of autonomy in mission
planning and flying the UAV enables the manpower and facilities
associated with the GCS to be minimised and allows for a UAV
commander to control several UAVs [Cox04]. This has to be offset
against the requirement to respond to ATM in short timescales and
to deal with any emerging hazards of operation. Removing the pilot
from the air vehicle means that control of the UAV has to be
achieved by either: complete automation; complete direct control
from a remote pilot; or, a combination of human and automated
control. The correct specification of autonomous flight control,
navigation and payload management can serve to reduce safety risks.
For example in UAV accidents, a large proportion of the human error
causes are reported to occur during the take off and landing phases
of flight [Osd03],[Wil04]. Automating these stages should produce
consistent behaviour from the UAV as it could be programmed to
remain on the ground (if criteria for take off are not met) or to
loiter or recover to a safe landing site (if approach/
landing/weather criteria are not met). Autonomy offers the benefits
of consistent behaviour in response to events such as collision
avoidance where a consistent application of rules required by
regulations could be implemented. In theory a UAV, with a sense and
avoid capability, preparing for take off could see the approach of
other aircraft within a 360 degree field of view and in this regard
perform better than a human pilot [Deg04]. Automation can relieve
humans of the dull flight management tasks thereby giving more time
for mission tactical and strategic level decision making [Osd05].
However increasing levels of autonomy introduce new issues that may
impact upon safety elsewhere. For example a study of accident data
for Global Hawk, a UAV with relatively higher levels of autonomy
has found errors in mission planning. It is noted that mission
planning is much more complex for an autonomous UAV and can take
many days (up to 237 days is reported) to accomplish. Errors
occurred because system operators did not properly monitor the
mission planning software and thus could not detect or respond to
system errors occurring in operations [Wil04]. In order to achieve
the required levels of safety
Page 14
in operations with autonomous control there is a need for higher
integrity software and mission planning. In order to characterise
Autonomy, so called “Autonomy levels” have been presented in
various taxonomies such as Sheridan’s 10 level model [Par00] and
Clough’s Autonomous Control Levels (ACL) [Clo02a]. The ACL has 10
levels from the lowest, level 0 – remotely piloted vehicle; to the
highest, level 10 - human like, that are measured against the
parameters of: perception/situational awareness, analysis/decision
making and communication/cooperation. In the ACL, level 7 - “real
time multi-vehicle cooperation” seems to meet with the requirements
of CAP722 for ELOS. At level 7 the situational awareness is
described as the detection of other air vehicles in local airspace
with multi-threat detection and analysis capability on board. The
decision making is described as being able to compensate for
anticipated system malfunctions and hazardous weather, to be
capable of evaluating and re-planning the flight path to avoid
threats and complete the mission. The communication/cooperation
capability is described as collision avoidance, use of third party
data for de-confliction and hierarchical cooperation with other air
vehicles. The ACL taxonomy describes the required or actual
behaviour of the UAV but does not appear to describe what the human
does in cooperation with the UAV. Performance metrics will be
required not only for the technical system but also for the human
working in cooperation with the technical system elements. The
HERTI UAS illustrates the role of humans in the concept of the
taxonomy of autonomy levels. Mark Kane [Kan07] has presented a
taxonomy of levels of autonomy (based on the US Navy Office of
Naval Research and as used by SEAS DTC) used in the context of BAE
Systems research programmes aimed at demonstrating full autonomy
(air vehicle and sensor/imagery). A diagram of this taxonomy is
replicated below to illustrate what each level of autonomy means in
terms of UAV behaviour and what the role of the human “”commander”
or UAV-p is for each level of autonomy.
Figure 3- UAV Autonomy levels from HERTI program [Wil09] HERTI is
reported [Mor08] as successfully demonstrating autonomous operation
by entry to a pre-defined “Search Area”, and the automatic
generation of navigation routes to enable the sensors to search and
record images of the required area. It is claimed that the system
has also demonstrated automatic target detection and downloaded
images to the GCS. The HERTI autonomy scale description of the
highest capability level for full autonomy does not
Page 15
appear to match the higher levels of capability for multiple UAVs
that are represented in Clough’s ACL [Clo02a]. Therefore
appropriate choice of autonomy classification is an important
aspect of autonomy measurement. Different phases of flight may
involve different levels of autonomous control. The management of
the associated hazards of flight is shared by humans working in
cooperation with an autonomous system that is designed to perform
reliably and take the right decisions consistently within the
bounds of the prescribed rules. The question of monitoring arises
though. How much trust is placed in the system to perform reliably
and exhibit safe behaviour? How is the system going to be monitored
to provide information to human operators so that they know they do
not need to intervene? Is the human operator going to be able to
judge if the system has taken the right course of action and is the
human operator capable of intervening? [Lev95]. Clearly all these
questions will need addressing if a system is going to be permitted
to fly. Many of these questions have been successfully addressed in
civil aviation for many years where much of the flight is conducted
by computers implemented in auto-pilot systems. During flight,
aircrew fulfil the roles of monitoring and supervising the systems
and the flight path; communicating with ATM and intervening in
emergency conditions. Generally it is the take off and landing
phases of flight that are directly flown by the pilot. A key point
from a safety perspective is that for a manned aircraft it is still
“preferable” for the pilot to complete the take off and landing
phases of flight. For a UAV it appears “preferable” for these
stages to be automated. Autonomous operation may introduce new
errors in supervising, monitoring and intervening in the autonomous
operation of the UAV. Even in a fully autonomous situation humans
will still be required to intervene in emergency situations [Caa08]
and act with a suitable level of proficiency (e.g. by exhibiting
effective air vehicle handling skills). The human will require
alertness and need to respond in a suitable time frame “of a few
seconds” as required by CAP722 [Caa08]. This could limit the
distance that a UAV-p can be located from the geophysical location
of the UAS. For a long range UAS, handover between UAV-p will be
required to maintain the “few seconds” of response time. In
addition to the attitude and alertness of the pilot there are other
human factors issues such as those relating to the design of the
ground station “cockpit”. The UAV-p does not have the same human
sensory inputs, about how the aircraft is handling, as the pilot of
a manned aircraft would. For a UAV this information has to be
provided from machine gathered data. Such data requires effective
interpretation to be of value to the UAV-p and s/he is likely to
have reduced situational awareness. A contributing factor to poor
situational awareness is the delay in commands and information
between the operator and the UAV system. Such signal transmission
delays can be one second or more and can introduce “temporal and
spatial uncertainties” for the operator [Mou01]. Wickens has stated
that delays of one second or more can lead to significant errors
that can result in total loss of control of the vehicle [Wic92].
Metrics that address system transmission performance (e.g. latency,
fidelity, security) will be relevant for safety of UAS. A key set
of safety metrics will be concerned with measuring the
dependability of communications including the elements of human to
human and human to communication system. A pilot of a manned
aircraft can respond rapidly to changes in weather, aircraft
handling and mission environment and make decisions based on recall
of relevant data from complex previously memorised information. The
pilot has learned from experience and perhaps UAV control software
that learns is required. The results of learning could be
replicated across the fleet whereas for human pilots the learning
and application is different between each individual. The full
implications of the effects on human performance interacting within
the UAS are not fully understood or properly managed, as accident
reports show. Partly this could be due to the “static” allocation
of functions between humans and machines during the design process.
It has been noted that whereas machines have been designed to
surpass human performance, there is not the same evidence that
current systems have been designed to allow the human to surpass
machine performance [Han96]. This is a complex area that will
require the development of suitable measures to enable designs to
be assessed for their impact on human performance and the
associated contribution to safety performance of the UAS.
Page 16
2.1.9 Work by York University MSc Students in UAV safety. Following
a broad survey of the safety issues present in the UAS domain Andy
Evans [Eva06] addressed a specific hazard identification process
for UAVs based on the safety assessment approach of ARP4761. The
results of the hazard analysis could identify suitable areas for
safety monitoring regimes. Chris Hodson [Hod08] investigated the
handover procedures between control stations for UAVs He concluded
that taking the pilot out of the aircraft raises new issues to be
managed including sensory deprivation, dependency on the data link,
latency of the data link and the effect this has on the ability to
control the flight path of the UAV. Some of these issues were
discussed above.
2.1.10 Summary of Sections 2.1.1 – 2.1.9 A review of the literature
regarding UAS has established that progress will be required in
many areas (regulation, technology, airworthiness and operational
arrangements) for UAS to be considered sufficiently safe for
routine flights in non-segregated airspace. Therefore, there will
be many aspects of the UAS that will require measurement to
demonstrate adequate levels of safety. In some cases a clear
standard does not exist. For example while it is concluded that
ELOS with manned aircraft is required, there is not yet agreement
on how this translates into safety targets or measures of
performance across all aspects of UAS design and operation.
However, in principle it should be possible to derive metrics for
relevant safety related parameters as these can be identified from
the issues reported in the literature review. A key finding is that
the accident rate for UAS is too high due in part to airworthiness,
and reliability of systems and operations. These causes will need
to be analysed further and suitable metrics derived to measure
improvements. Some of these metrics may already exist or could be
adaptations of pre-existing manned aircraft metrics. For example
the airworthiness and reliability of conventional systems and air
vehicle structure could be measured and improvements identified in
common with approaches used for manned aircraft. This is outside of
the scope of the aims of the project which will are focussed on UAS
specific issues. The affect on system elements by removing the
pilot from the aircraft cockpit and relocating him/her remotely has
wide ranging ramifications for system safety. Other significant
factors are: the longer time on task a UAS can achieve and the need
to achieve ELOS and appear transparent to other airspace users.
These factors combine to introduce novel system requirements,
operating arrangements and applications of technology that
introduce new safety management issues and hence new safety metrics
in order to track performance. Section 2.1 .1 highlighted the role
of the UAV commander in controlling several UAVs by individual
UAV-p. It also identified the need to handover control from one
UAV-p to another. There will be a need to define handover
procedures and suitable metrics demonstrating the safety of such
procedures within the system. Section 2.1.3 highlighted the need
for a consistent classification system for UAS and that metrics
appropriate to kinetic energy levels were available to identify
equivalence to manned aircraft classes. Future scenarios may
require safety monitoring of teams or swarms of UAS operating under
autonomous control. This is someway into the future and out of
scope of the current project. Section 2.1.4 and 2.1.5 identified
the requirements of regulations. These included the emerging
methods for determining equivalence to manned aircraft classes. The
ELOS principle is broader than this and metrics will be required
for autonomous operations, collision avoidance, demonstrating
transparency to other airspace users (e.g. communication
effectiveness) and the effectiveness and readiness of emergency
systems (e.g. FTS).
Page 17
Section 2.1.6 reported on the suitability of safety targets and
safety objectives in the context of current UAS reliability and
concluded that there is not yet agreement on the appropriate safety
target that should apply to UAS. Section 2.1.7 identified the key
findings from accident data reported in the literature and the
significance of HF contributory causes in accidents and the need
for metrics to assist in improving safety performance. Increasing
the level of autonomy in UAV flight control, particularly in
take-off and landing phases of flight, is seen as a potential
solution. Section 2.1.8 investigated autonomy and identified that
while levels of autonomy classification exist for the technical
system, there would need to be further development to include the
human working in co-operation with the technical system and
suitable metrics to track performance and identify improvements to
design and procedures. It was noted that a UAV-p typically will not
have sufficient levels of situational awareness at all times and
can thus fall victim to latency in communications links between the
UAV and the GCS. Metrics will need to be derived to measure the
system communications performance and the level of situational
awareness the UAV-p has. Many of the issues identified in sections
2.1.1 - 2.1.8 have implications for operational safety and UAS
specific metrics will in many cases be those that measure aspects
of operational safety. This is examined further in section
2.2.
2.2 Operational Safety Management In this section of the literature
survey key concepts in operational safety were examined with regard
to how these relate to the operational safety of UAS. It can be
asserted that the majority of accidents actually occur during
system operation and therefore “operational safety is concerned
with protecting people at risk from harm during the operation of a
complex system”. [Omm08] Conducting safe operations with UAVs is of
fundamental importance and of equal significance to the inherent
airworthiness of the UAV design and the “equipment safety”
properties of all the elements of the UAS. This was shown by the
accident causes discussed in section 2.1.7. But how is operational
safety going to be managed effectively and what are the objectives
of operational safety management? In order to understand and assess
this there are lessons that can be learnt from how operational
safety is managed in other sectors of the economy: oil and gas
exploration, railway operations, nuclear power generation; and, how
military and civil aviation operations are managed for manned
aircraft which has more actual domain affinity with UAS. In the
military, JSP553 [MOD08] describes “the four pillars” of
airworthiness as: the safety management system, compliance with
recognised standards, competence (of people and organisations) and
independent assessment. Furthermore airworthiness must be managed
in- service i.e. during the operational life of the aircraft. One
of the key standards to be complied with is Def Standard 00-56
[Mod07b] which mandates the provision of an equipment safety case
that is subject to independent assessment. The safety case must be
maintained throughout the aircraft service life as changes are
introduced into the design, the equipment operation and conditions
of use. UK MOD policy on military UAVS is that they are to be
treated as UK military aircraft and are subject to the same
regulations contained in JSP 550 - Military Aviation Policy,
Regulations and Directives [Mod06]. These place requirements on the
Aircraft Operating Authority (AOA) to ensure they have provided
sufficient standard operating procedures promulgated as flying
orders. The orders applicable to UAVs are described in CAP722
[Caa08] and include “detailing the training, competency, currency,
medical requirements and crew duty considerations for all personnel
involved in the operation of UAVs.” These are all information
requirements for safety metrics.
Page 18
2.2.1 The Origins of the Safety Case The Windscale accident remains
the UK's worst nuclear accident to date and the findings of the
Windscale Accident Inquiry 1957 [Uka57] paved the way for key
safety focussed nuclear legislation enacted in The Nuclear
Installations Act 1959 [Ukg59]. This established the Nuclear
Installations Inspectorate (NII) to act as an independent regulator
of the nuclear industry and became law under the Nuclear
Installations Act 1965 [Ukg65]. Of key importance within the act is
the requirement for all nuclear installations to be licensed and
that…”the licensee shall make and implement adequate arrangements
for the production and assessment of safety cases consisting of
documentation to justify safety during the design, construction,
manufacture, commissioning, operation and decommissioning phases of
the installation.” This application of a safety case clearly covers
the operation phase as well as the design and manufacture phases.
One of the requirements of nuclear safety cases is the need for the
licensee to demonstrate that it … “understands the hazards
associated with its activities and how to control them adequately”.
In July 1988, a fire on the Piper Alpha oil rig in the North Sea
claimed one hundred and sixty seven lives [Ukd90]. Piper Alpha
produced both oil and gas and at its peak was responsible for some
ten percent of the UK’s North Sea oil production providing a strong
imperative to keep operations running. It was connected to two
other rigs, Claymore and Tartan, located 200km north-east of
Aberdeen in the Piper oilfield. The disaster was caused primarily
due to poor procedural control and a lack of safety oversight. A
maintenance task was being performed on one of two gas pumps, in
which the pressure relief valve was removed for overhaul. The
oncoming shift should have been made aware that the pump without
the pressure relief value was out of service. However, due to
procedural failings, the out of service pump was used, resulting in
a significant build up of pressure, pipe failure and a catastrophic
explosion when leaked gas ignited. The problem was exacerbated by
the sister rigs (Claymore and Tartan), who continued to pump gas to
Piper Alpha via a series of underwater pipes. A key recommendation
of “The Public Inquiry into the Piper Alpha Disaster” [Ukd90],
chaired by Lord Cullen was that “the operator or owner of every
offshore installation should be required to prepare a safety case
and submit it to HSE for acceptance.” This resulted in the
introduction, in 1992, of an additional set of regulations the
Offshore Installations (Safety Case) Regulations (OSCR). These had
the aim to “reduce the risks from major accident hazards to the
health and safety of the workforce employed on offshore
installations or in connected industries” [Hse06] . The MOD
mandates the construction of a safety case for an equipment project
during the acquisition of defence equipment and that includes
manned aircraft and UAVs. These are subject to the provisions of
Defence Standard 00-56 Issue 4 [Mod07b]. The guidance material
contained in part 2 of the standard includes definitions for: the
Safety Case (SC)and the Safety Case Report (SCR) which are
discussed further below: Safety Case (SC): “A structured argument,
supported by a body of evidence that provides a compelling,
comprehensible and valid case that a system is safe for a given
application in a given operating environment.” [Mod07b]. The
University of York Module in Hazard and Risk Management and Safety
Cases [Hrm07], contends that a safety case requires two elements:
supporting evidence and a high level argument. Supporting evidence
comprises the results of observing, analysing, testing, simulating
and estimating the properties of the system from which levels of
safety performance can be inferred. A high level argument is
required to provide an explanation of how the supporting evidence
can be interpreted as indicating the achievement of acceptable
levels of safety for the system in its operating context.
Page 19
One of the identified requirements from the OSCR, Regulation 12
[Hse06] was that the safety case must demonstrate that the
management system is sufficient to ensure that relevant statutory
arrangements are complied with. A further requirement is that all
hazards with the potential to cause major accident (i.e. loss of
life) have been identified. In the case of UAVs there will be
particular aspects of both of these argument approaches that will
require the provision of suitable evidence of compliance. It should
be expected that many aspects will be similar to manned aircraft
such as basic airworthiness and structural integrity. However there
will be key differences due to the lack of the pilot being present
in the vehicle. For example there will need to be arguments and
evidence about “sense and avoid” as opposed to “see and avoid”;
and, ensuring sufficient situational awareness to enable
appropriate responses to avoid collisions. Safety Case Report
(SCR): “A report that summarises the arguments and evidence of the
Safety Case, and documents progress against the safety programme.”
[Mod07b]. It is usual for the SC of a complex system to contain a
vast amount of data (e.g. design, trials, and analysis and test
data) that is generated throughout the project life cycle and
produced by multiple organisations and subject to review,
independent assessment and maintenance. There is likely to be
multiple cross referencing making it difficult for one person to
judge if the required safety levels have been achieved in the
system. The SCR addresses this problem by summarising the key parts
of the SC and referencing all the supporting evidence in a clear
and concise manner.[Mod07b], [Hrm07]. It effectively forms the
working documentary evidence against which the safety of the system
can be judged at a particular point in the programme. Thus the SCR
is likely to provide the quickest and most efficiently assimilated
overview about system safety and is used by project team members,
and others, as a working document for guidance about the safety of
the system. The SCR represents a complex argument by justifying a
top level claim, for example, that the system is safe to operate in
an environment by suitably qualified and experienced personnel
(SQEP) acting in accordance with prescribed operating procedures.
The thread of the argument linking evidence to the top level
claim(s) can often be obscured by the complexity and volume of
text. Consequently in order to make the safety argument
comprehensible, Kelly devised the Goal Structuring Notation (GSN)
which has found application in many industries including military
and civil aviation. [Kel99]. The GSN has significant advantages in
that it is simple, structured, hierarchical and expressive
providing clear communication on elements most important for
safety. It can be used at various stages of argument development,
and the semantics are well developed and understood. The main
disadvantages are that there is a learning curve to follow to
achieve a sufficiently competent standard at writing in the
notation and that it can not prevent bad arguments being written.
However, the notation has been expanded to offer a modular based
approach so that it can be applied to safety cases for integrated
modular avionics (IMA) for example and system of systems
applications. Safety cases expressed in GSN can be reviewed and
annotated with symbols to record objective assessments of
weaknesses or strengths of the safety argument. [Hrm07]. In brief
the GSN notation description is as follows: GSN shows how Goals are
broken down into sub-goals supported by Evidence (solutions) making
clear the Strategies adopted, the rationale for the approach
(Assumptions and Justifications) A/J and the Context in which Goals
are stated.
Page 20
2.2.2 The Requirements for Operational Safety Cases The examples of
safety cases from the nuclear power and offshore oil and gas
industries showed that these industries are regulated in order to
ensure effective operational safety management. However, in other
industries sufficient focus on the safety of operations does not
always appear to have been sufficient, as the following example
demonstrates. On 6th March 1987 the Zeebruge ferry disaster
occurred with the loss of the Herald of Free Enterprise and 193
lives. This disaster occurred primarily due to deficiencies in
operational safety management. The ship design was known to be
susceptible to capsize. On the day of the disaster the ship left
port with the bow doors open. This had become accepted practice in
certain ferry operations in order to clear fumes from the car deck
with the doors normally closed soon after leaving the dock. However
the bosun responsible for closing the doors had fallen asleep and
there was no indication in the wheelhouse of the status of the
doors being open. The ship departed nose down in the water as the
bow ballast tanks were still being pumped out. The Captain had
written memos regarding the need for indication of bow doors but no
action had been taken. Even though the ship design was not
sufficiently safe, the operator must cope with deficiencies and
could have invested in changes to improve the overall safety
performance in operation [Her09]. In the rail industry, the yellow
book [Yel05] summarises the requirements of the railway safety case
regulations. Any train or station operator must write a railway
safety case and have it accepted before starting operations and the
operator must follow their safety case. The railway safety case
must describe among other things: the operator’s policy and
arrangements for safety management, how it will monitor safety, how
the operator is organised to carry out its safety policy, and how
the operator ensures that staff are competent to do safety related
work. The definition of a Safety Case is focused on the equipment
being safe for a given application and environment. If the
environment or the application changes then the safety case will
not be valid for the present state of the system. It is essential
to keep the safety case under review and ensure that any changes
made to the system design and operational arrangements deliver the
required levels of safety [Mod07c], [Yel05], and [Omm08]. Previous
post graduate students at the University of York have investigated
OSCs: Blagrove [Bla04], Salter [Sal06] and Jones [Jon07]. Blagrove
developed a “prototype generic operational safety case pattern for
possible reuse in the UK military aerospace domain” [Bla04]. His
work