3710 McClintock Avenue, RTH 314 ~ Los Angeles, CA 90089-2902 ~ (213) 740-5514 ~ www.usc.edu/create
National Center for Risk and Economic Analysis of Terrorism Events
University of Southern California
Dynamic Aviation Risk Management System (DARMS): A Proof of Concept
Study Examining the Role of Multi-Attribute Utility
Dr. William J. Burns (Co-PI), Dr. Robin Dillon-Merrill (Co-PI) and
Dr. Richard John (Co-PI)
DRAFT Report to Transportation Security Administration (TSA)
"This research was supported by the United States Department of Homeland Security through the National Center for Risk and Economic Analysis of Terrorism Events (CREATE) under Cooperative Agreement No. 2010-ST-061-RE0001. However, any opinions, findings, and conclusions or recommendations in this document are those of the authors and do not necessarily reflect views of the United States Department of Homeland Security or the University of Southern California."
Cooperative Agreement No. 2010-ST-061-RE0001
Department of Homeland Security
March 26, 2015
ii
ABOUT CREATE
Now in its eleventh year of operation, the National Center for Risk and Economic Analysis of Terrorism
Events (CREATE) was the first university-based Center of Excellence (COE) funded by University
Programs of the Science and Technology (S&T) Directorate of the Department of Homeland Security
(DHS). CREATE started operations in March of 2004 and has since been joined by additional DHS
centers. Like other COEs, CREATE contributes university-based research to make the Nation safer by
taking a longer-term view of scientific innovations and breakthroughs and by developing the future
intellectual leaders in homeland security.
CREATE's mission is to improve our Nation's security through research and development of advanced
models and tools to evaluate risks, costs and consequences of terrorism and natural and man-made
hazards and to guide economically viable investments in homeland security. We are accomplishing our
mission through an integrated program of research, education and outreach that is designed to inform
and support decisions faced by elected officials and governmental employees at the national, state, and
local levels. We are also working with private industry, both to leverage the investments being made by
the Department of Homeland Security in these organizations, and to facilitate the transition of research
toward meeting the security needs of our nation.
CREATE employs an interdisciplinary approach merging engineers, economists, decision scientists, and
system modelers in a program that integrates research, education and outreach. This approach encourages
creative discovery by employing the intellectual power of the American university system to solve some
of the country’s most pressing problems. The Center is the lead institution where researchers from
around the country come to assist in the national effort to improve homeland security through analysis
and modeling of threats. The Center treats the subject of homeland security with the urgency that it
deserves, with one of its key goals being producing rapid results, leveraging existing resources so that
benefits accrue to our nation as quickly as possible.
CREATE develops models, analytical tools, methodologies and software, and tests these tools in case
analyses, representing critical homeland security investment and policy decisions. Due to the cross-
cutting nature of this research in risk, economics, risk management and operations research, CREATE
serves the need of many agencies at the DHS, including the Transportation Security Administration,
Customs and Border Protection, Immigration and Customs Enforcement, FEMA and the US Coast
Guard.. In addition, CREATE has developed relationships with clients in the Offices of National
Protection and Programs, Intelligence and Analysis, the Domestic Nuclear Detection Office and many
State and Local government agencies. CREATE faculty and students take both the long-term view of
how to reduce terrorism risk through fundamental research, and the near-term view of improving the cost-
effectiveness of counter-terrorism policies and investments through applied research.
Please visit www.usc.edu/create for more information.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
iii
Table of Contents
About CREATE ...................................................................................................................................................... ii
Executive Summary ............................................................................................................................................... 1
Introduction .............................................................................................................................................................. 4
Multi-Attribute Utility (MAU) Inputs .............................................................................................................. 6
MAU Analysis Output ........................................................................................................................................ 24
Modeling Uncertainty ......................................................................................................................................... 37
Adversary MAU ................................................................................................................................................... 46
Implementing DARMS: Challenges .............................................................................................................. 52
Conclusions ........................................................................................................................................................... 55
Future Research .................................................................................................................................................... 57
References .............................................................................................................................................................. 59
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
1
Executive Summary
Background. The threat of international terrorism and its focus on attacking U.S.
commercial airlines continues to loom large nearly thirteen years following the events of
September 11th
. The Transportation Security Administration (TSA) was created in 2001 to
address this threat and today has as its core mission “Protect the Nation’s transportation systems
to ensure freedom of movement for people and commerce” Inherent in this mission has been the
assessment and management of the risks surrounding threats to commercial airlines.
In the last three years the TSA has begun to consider carefully the operational costs of a
“one size fits all” screening policy. Responding to the complexity of assessing the threat posed
by passengers and cargo, the TSA could benefit from expanding the Risk-Based Security
approach to include the development of a system-wide architecture that would allow the
assessment of risk on a flight-by-flight basis and make risk-based decisions in real time. In this
regard, the TSA is now investigating the Dynamic Aviation Risk Management System
(DARMS) initiative, and hence the motivation for this study.
Goals. This proof of concept study had four goals: 1) Explore an approach to articulating
and measuring fundamental objectives that lead to an understanding of consequences the TSA
(and other stakeholders) care about; 2) Investigate the uncertainty surrounding credible threats
and flight vulnerability so as to suggest approach to calculating system wide risk to domestic
commercial aviation; 3) Illustrate how the Current and DARMS approaches to passenger
approach can be compared on objectives identified in this study and 4) Identify areas needing
further investigation.
Approach. Objectives and attributes were identified to develop a multi-attribute utility
model (MAU). The search began broadly starting with the TSA’s overarching strategic objective
“Protect the Nation’s transportation systems to ensure freedom of movement for people and
commerce.” and then focused more narrowly on objectives pertaining to aviation security and
specifically the DARMS initiative. Attribute measures, scales and consequences were selected
and assessed based on informal conversations with colleagues from TSA and Deloitte and
publically available information. Uncertainty about credible threats and flight vulnerability (the
probability a security system can be defeated assuming a credible threat) was explored by
decomposing the assessment of the probability of a successful attack into relevant component
parts (e.g., risk classification, threat detection during screening) using probability trees. A
probability function was derived to calculate system wide risk. A multi-attribute utility function
was used as an example of how to compare the Current and DARMS approaches across attribute
measures. Attribute weights (e.g., Figure 1) and other components of the model were based on
the perspective of one of the research team for illustration purposes. However, a more definitive
MAU model can be developed in the future involving experts at the TSA using the methods
outlined in this report.
Objectives and Attribute Measures. The following fundamental objectives were
proposed for the analysis presented in this study: 1) Security Effectiveness, 2) Passenger
Satisfaction, 3) Economic Costs of Security Breaches, 4) Operational Efficiency, 5) Operational
Costs, 6) TSO Job Satisfaction and 7) Aviation Industry Vitality Costs. For this study it appeared
better to think broadly and inclusively about the number of objectives and corresponding
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
2
attributes with the idea that the list could be paired down as needed at a later time. Twenty four
attribute measures of these high level objectives, along with their assessed weights, were used to
construct the multi-attribute utility functions. Examples of these attribute measures include
fatalities, passengers’ perception of security, economic costs of a significant security breach,
passenger throughput, operational costs (average cost of passenger classification and screening),
TSO morale and airline operating revenue costs.
Uncertainty: Threats and Vulnerability. Probability trees were constructed as an
illustration of how flight vulnerability could be decomposed and its component parts assessed.
Assessment of branch probabilities was guided not by sensitive information but rather by a
reasonable ordering of relative probability magnitudes (e.g., passengers posing a threat will be
much less likely to received expedited screening, standard screening lanes will have a higher rate
of threat detection than expedited screening lanes, DARMS with its proposed sophisticated
countermeasures will reduce flight vulnerability overall). As it turned out, vulnerabilities using
the probability trees calculations were similar to TSSRA assessments, and were estimated to be
about 10% (Current being slightly higher than DARMS).
Using these vulnerabilities and speculations about the average number of domestic
credible threats per year the TSA might encounter, a system wide probability of at least one
successful attack was calculated. To accomplish this, two recursive probability functions were
derived based on the binomial distribution, one assuming that credible threats operate
independently (standard assumption) and the other that their efforts are correlated in some way.
Assuming for illustration, that threats occur independently and there are two credible threats in a
given year, the system wide chance of at least one successful attack is about 19%. The recursive
expressions in equations (1) and (2) also allow for assessments regarding uncertainty of number
of credible threats i, and degree of correlated threats . These parameters permit a broad range
of system wide estimates. Notice that equation (1) is a special case of the more general
expression in equation (2) when =0. When =1 equation (2) reduces to p the flight
vulnerability.
P(X1|n,p) = i (1-(1-p)i ) for nx; 0 otherwise; i=0,1,2 …n. (1)
P(X1|n,p, ) = i {1-((1-p)(p+(1-p))i-1
)} for nx; 0 otherwise; i=1,2 …n. (2)
where X is the number of successful attacks; n is the number of credible threats; I is the
probability of n credible threats, i =1 for i=0,1,2 …n and P(X1|n,p, )=0 for n=0; p is the
flight vulnerability; and (–1 to 1) is the correlation between threats.
Key Findings. MAU calculations tend to favor a DARMS approach regardless of
whether there is a successful attack on a commercial airline or not. However, this finding is
based on the assessed attribute weights of one of the research team and was meant to illustrate
an approach to making this comparison not to yield a definitive conclusion. What this study does
demonstrate are two things worth noting. First, there appear to be conflicting objectives. That is,
the DARMS approach may do better on security effectiveness, operational efficiency and
operational costs and the Current approach may do slightly better on passenger satisfaction,
somewhat better on economic costs following a successful attack and clearly better TSO job
satisfaction regardless of attack outcome. These observations in part, emerged out of
conversations with TSA and Deloitte colleagues.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
3
Second, there is a great deal of uncertainty surrounding consequence estimates of these
attribute measures as well as probability estimates of a successful attack. One of the striking
observations was how critically dependent the probability of a successful attack is on the number
of credible threats in a given year. This speaks to the pivotal role of deterrence.
Future Research. The next step is to take the approach illustrated in this study, and
involve key stakeholders in careful elicitations. These next steps are outlined in detail in the
report.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
4
Introduction
Perspectives on Risk and Screening Procedures. The threat of international terrorism
and its focus on attacking U.S. commercial airlines continues to loom large nearly thirteen years
following the events of September 11th
. The Transportation Security Administration (TSA) was
created in 2001 to address this threat and today has as its core mission “Protect the Nation’s
transportation systems to ensure freedom of movement for people and commerce” Inherent in
this mission has been the assessment and management of the risks surrounding threats to
commercial airlines. Many agencies within the U.S. Department of Homeland Security (DHS),
including the TSA, describe the risk of terrorist attacks as a function of three components: threat,
vulnerability and consequences. Because of the difficulty of assessing this threat, and the dire
consequences that would likely follow a successful attack on an airline, the TSA initially sought
to reduce the Nation’s vulnerability by adopting a policy of screening all passengers regardless
of their individual risk level.
Risk-Based Security. In the last several years the TSA has focused on moving beyond
the “one size fits all” policy. Screening passengers according to their threat level (an assessment
of their criminal intent and capability) should allow the TSA to allocate resources in a way that
increases security and is more cost-effective in the long term. A tactical outgrowth of such a
Risk-Based Security strategy is the implementation of expedited screening for passengers who
are deemed low risk. Expedited screening can be less intrusive, more convenient, quicker and
less costly than standard screening, thus allowing more security resources to be allocated to those
passengers not receiving expedited screening. Passengers going through expedited screening can
still be subjected to random augmented screening to reduce the chance that this procedure will be
gamed. The TSA has set a strategic goal of moving significantly more of the traveling public
through expedited screening so resources can be dedicated to passengers that the TSA knows less
about in terms of their potential threat.
Dynamic Aviation Risk Management System (DARMS). Responding to the
complexity of assessing the threat posed by passengers and cargo, the TSA believes it could
benefit from expanding the Risk-Based Security approach to include the development of a
system-wide architecture that would allow the assessment of risk on a flight-by-flight basis and
make risk-based decisions in real time. Such an endeavor would require the calculation and
integration of risk over a number of dimensions and the adjustment of risk using a collection of
mitigation options on a flight-by-flight basis according to government-determined risk tolerance
levels.
Project Goal
The goal of this proof of concept study was to create an objectives and attributes
hierarchy that can serve to guide meaningful comparisons among approaches to commercial
aviation security including the Current and DARMS approaches. Hence, this report sought to
demonstrate how assessments could be made across attribute consequences to articulate the pros
and cons of each alternative.
This study sought to address two challenges:
1) Illustrate how DARMS might be examined from a strategic level perspective. As such,
the report describes the procedures used to identify strategic objectives important to the TSA and
its stakeholders and to develop performance measures of these objectives so as to compare
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
5
DARMS to what is currently being done in terms of passenger risk classification and screening.
For this purpose an example Multi-attribute utility (MAU) model is presented.
2) Illustrate how uncertainty with regard to attack outcomes and consequences can be
represented. Hence, an example probability tree that decomposes vulnerability into relevant
components is depicted.
Contribution of Multi-attribute Utility Approach
A key aspect of the game theoretic component of the DARMS approach is consequence
assessment for both the attacker and the defender. Consequences are contingent on the attacker’s
1) target selection and mode of attack (including the “no attack” option), 2) the defender’s
selection of countermeasures, and 3) the outcome of any attack initiated. How both the attacker
and the defender view these consequences is evaluated relative to each’s fundamental objectives.
Inevitably, even when one focuses on only one adversary, the fundamental objectives can still be
in conflict, and trade-offs are necessary (e.g., maximize security and maximize checkpoint
efficiency). One critical aspect of estimating consequences is determining the trade-offs among
conflicting objectives for both the attacker and defender. For most if not all consequences,
uncertainty is endemic and impacts all objectives. Additionally, accounting for risk attitudes of
both the attacker and the defender will be important. This can be done with utilities functions
that capture the preferences of decision makers.
The tasks required to develop utility functions are described below.
1. Identify attackers. It is important to identify adversaries with the fundamental objective
of attacking/crashing a passenger airplane flying within the United States. In this study
adversaries are limited to organizations with clear intent and significant capabilities.
2. Identify defender stakeholders. While the TSA is the primary stakeholder, it is important
to identify other stakeholders closely involved in aviation transportation defense that
should be included in the DARMS model. These stakeholders include passengers,
Transportation Security Officers (TSOs), and the Aviation Industry.
3. Identify fundamental objectives of an adversary and the defender. We selected objectives
both of the adversary and the defender with the guidance of the TSA and the Deloitte
team. Adversary objectives may vary by adversary, but we focused on those that include
maximizing direct and indirect economic costs to the defender, maximize deaths,
maximize one-time government costs, e.g., purchasing new security equipment, and
maximize psychological impacts to the public. Defender objectives may vary by
stakeholder group, but will likely include (but are not limited to) minimize
countermeasure costs, minimize deaths, minimize direct (short term) economic cost
associated with a successful attack, minimize indirect (long term) economic cost
associated with a successful attack, and minimize intrusiveness of countermeasures on
the flying public. Fundamental objectives for both the selected adversary and the
defender stakeholder group(s) were constructed in direct consultation with the TSA and
other SMEs and are discussed in the next section.
4. Identify attack path alternatives for specific adversaries, including specific targets and
modes of attack. Working in conjunction with Dr. Milind Tambe’s CREATE game theory
team, specific countermeasures were selected on a flight-by-flight basis by the TSA. As
noted earlier, a subset of both adversary and defender alternatives were selected for the
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
6
proof of concept demonstration. Specifically, the attack modality was construed to be an
adversary carrying a non-metallic explosive on their person or carry-on trying to board a
domestic flight.
5. Construct attribute scales. For the fundamental objectives for both the selected adversary
and selected defender, attribute scales (or simply attributes) provide a metric for
describing the consequences quantitatively. These attribute scales were constructed in
direct consultation with the TSA and other SMEs (e.g., Deloitte team).
6. Assess probability distributions. As an illustration of approach, probability distributions
describing the consequences on each attribute scale for both the selected adversary
(number of credible threats per year only) and defender stakeholder(s) were assessed.
These consequence distributions were conditional on defender “current” and “DARMS”
airport screening alternatives, and on other exogenous uncertainties related to attack
success.
7. Elicit utility functions. In consultation with the TSA and other SMEs, utility functions for
each of the attribute scales are constructed that represent characteristics of adversary and
defender preferences. All single attribute utility functions for this study are assumed to be
linear.
8. Assess trade-off weights. In consultation with the TSA and other SMEs, trade-offs among
the attribute scales associated with both adversary and defender objectives were elicited.
As an illustration, trade-off weights were developed based on the priorities of one
member of the research team. Additionally, this proof of concept assumed that attribute
scales could be aggregated using a weighted additive model.
Multi-Attribute Utility (MAU) Inputs
Defining Objectives
First, fundamental objectives were identified. We followed the procedures suggested by
Keeney and von Winterfeldt (2011) in which they developed a multi-attribute value model to
guide decisions relevant to the U.S. Department of Homeland Security. Essentially, the search
began broadly beginning with the TSA’s overarching strategic objective “Protect the Nation’s
transportation systems to ensure freedom of movement for people and commerce” (TSA, 2015)
and then focused more narrowly on objectives pertaining to aviation security and specifically the
DARMS initiative. The Deloitte consultant team to the Office of the Chief Risk Officer,
developed a DARMS business case in which they proposed six fundamental objectives critical to
TSA aviation security and the DARMS initiative:
1) Security Effectiveness,
2) Operational Efficiency,
3) Passenger Satisfaction,
4) Industry Vitality,
5) Fiscal/Policy Issues and
6) Enterprise Efficiency.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
7
Some of these objectives had multiple attribute scales. For example, Operational Efficiency was
measured by cycle time and both quality and frequency of alarms; Passenger Satisfaction was
measured by touch rate, divesture, CBRA (checked baggage resolution area) rate (i.e., how many
checked bags need to be opened and hand checked), wait time and perceptions of security;
Industry Vitality was measured by cost to the airline/airport industry to implement and
Fiscal/Policy Issues was measured by security costs per unit and regulatory/compliance costs.
Enterprise Efficiency was measured by the number of systems tracking passenger risk.
Beginning with the objectives suggested by the Deloitte team, preliminary conversations
with TSA SMEs and findings from Keeney and von Winterfeldt (2011) the following
fundamental objectives were proposed for the analysis presented in this study:
1) Security Effectiveness,
2) Passenger Satisfaction,
3) Economic Costs of Security Breaches,
4) Operational Efficiency,
5) Operational Costs,
6) TSO Job Satisfaction and
7) Aviation Industry Vitality Costs.
This study attempted to be both broad and inclusive about the number of objectives and
corresponding attribute scales with the idea that the list could be pared down as needed at a later
time. In the following paragraphs each objective and its one or more attribute scales will be
discussed. For a complete of list of objective definitions and attribute scale values please see
Tables 1 and 2.
Table 1. Definitions of TSA Objectives and Attributes
Objectives/Attribute Scales Definition
Security Effectiveness Minimize casualties and breaches of security in sterile areas in the airport and the aircraft cabin and prevent the catastrophic loss of an aircraft through the use of risk-based procedures involving deterrence, passenger pre-screening, detection and mitigating response.
Fatalities Lives lost as direct result of a security breach
Injuries Injuries incurred as direct result of a security breach
Security Breaches Inside Airport Sterile Area
Any entry into a sterile area by someone with the intent and capability of seriously harming other people within this area.
Security Breaches inside Cabin of Aircraft
Any entry into an aircraft cabin by someone with the intent and capability of seriously harming other people within this area.
Deterrence1 Risk-based procedures that prevent an attempt to breach security by
increasing the perceived costs of defeating or engaging the security system or by decreasing the perceived likelihood of defeating the security system.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
8
Objectives/Attribute Scales Definition
Detection sensitivity1 Area under a plot of P(Detection/Threat) or (screening sensitivity)
versus P(Detection/Non-Threat) or (1-screening specificity). Perfect classification has an area of 1 and random classification as an area of .5. The plot is generated for all screening thresholds. Similar plots were first used during World War II for the analysis of radar signals and are an important part of signal detection theory. The technical name of this type of plot is a receiver operating characteristic curve and is well-known as a ROC curve.
Passenger Satisfaction Maximize passenger satisfaction during security screening through the use risk-based procedures that facilitate passenger confidence, perception that they have been treated fairly and at a reasonable cost of their time, hassle and intrusion.
Average Wait time The average number of minutes a passenger can expect from the point they enter the passenger security queue until they are they are through the screening procedures (i.e. typical wait time).
Variance Wait time The variation in minutes a passenger can expect from the point they enter the passenger security queue until they are they are through the screening procedures (i.e. predictability of wait time).
Touch Rate: Passengers Touched by TSO
The percentage of passengers being touched by a TSO during the screening process (i.e. intrusiveness level).
Divesture The extent to which a passenger has to take off or take out carry-on items during the screening process (i.e. hassle factor).
Passenger Perceptions of Fairness
The extent to which a passenger feels that they have received the appropriate level of screening relative to other passengers and relative to their investment in various “trusted traveler” programs (i.e. equity level).
Passenger Perceptions of Security Effectiveness
The extent to which a passenger feels airport and airline security is effective as defined above (i.e. “system is working”).
Economic Costs of Security Breaches in Airport Sterile Areas or Aircraft Cabin
Minimize total economic costs to the U.S. due to loss of life, injuries, property damage, emergency response, business disruption, additional security measures and public response as a direct result of security breaches.
Total Consequences The sum of all economic consequences in dollars over a one-year time horizon due to security breaches.
Operational Efficiency Minimize the use of input resources such as TSOs, equipment and airport floor area to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance.
Passenger Throughput The number of passengers moving from the point they enter the passenger security queue until they are they are through the screening procedures in an hour.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
9
Objectives/Attribute Scales Definition
Passenger Cycle Time The number of minutes from the point a passenger enters the passenger security queue until they are they are through the screening procedures (same as wait time).
Passenger FTE The average number of TSOs required to screen a passenger.
Passenger False Alarm Rate
The percentage of passengers triggering either a false positive or a minor true positive (i.e. non-security breach).
TSO Utilization Rate Percentage of time TSOs spend screening passengers.
Responsiveness The number of minutes needed to reduce the average wait time to a desired level (e.g., 10 minutes).
Resilience The number hours to restore commercial aviation system to normal operating capacity following a breach of security.
Operational Costs Minimize the costs to the TSA of all procedures involved with achieving a desired level of security effectiveness, passenger satisfaction and regulatory compliance minus the costs of significant security breaches.
Total Passenger Security Costs
The sum of all costs to the TSA in dollars over a one-year time horizon due to operations relating to security effectiveness, passenger satisfaction and regulatory compliance (e.g., FTE, Equipment, FAMS, Pre-screening). Not included are the costs of responding to significant security breaches that are covered under Economic Costs.
TSO Job Satisfaction Maximize TSO job satisfaction in terms of morale, perceptions of that airport and airline security is effective and low turnover rate.
TSO Morale The extent to which a TSO feels a commitment to the TSA and its mission.
TSO Perceptions of Security Effectiveness
The extent to which a passenger feels airport and airline security is effective as defined above (i.e. “system is working”).
TSO Turnover The percentage of TSOs leaving the TSA per year.
Aviation Industry Vitality Costs
Minimize costs to U.S. airports and airlines from additional TSA security compliance requirements as the TSA attempts to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance.
Costs of Reconfiguration at Airport Checkpoints
The sum of all costs to U.S. airports and airlines over a one-year time horizon due to additional security compliance requirements from the TSA as the TSA attempts to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance (e.g., redo the checkpoints). These do not include the impact from major security breaches that are covered under Economic Costs.
Reduction of Operating Revenue from
The reduction of U.S. airline revenue from all sources (e.g., passenger, checked baggage, cargo) over a one-year time horizon due to
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
10
Objectives/Attribute Scales Definition
Additional Security Compliance
additional security compliance requirements from the TSA as the TSA attempts to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance. These do not include the impact from major security breaches that are covered under Economic Costs.
Reduction of Revenue Passenger Miles from Additional Security Compliance
2
The reduction of the number of miles flown by a paying passenger on a U.S. airline over a one-year time horizon due to additional security compliance requirements from the TSA as the TSA attempts to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance. These do not include the impact from major security breaches that are covered under Economic Costs.
Reduction of Revenue per Available Seat Mile from Additional Security Compliance
2
The reduction of U.S. airline revenue per available seat over a one-year time horizon due to additional security compliance requirements from the TSA as the TSA attempts to achieve a desired level of security effectiveness, passenger satisfaction and regulatory compliance. These do not include the impact from major security breaches that are covered under Economic Costs.
1 Are included in this list because they represent an important aspect of their corresponding objective but
were excluded from the MAU analysis because their effect was represented in the probability trees. 2 Are included in this list because they represent an important aspect of their corresponding objective but
were excluded from the MAU analysis because their effect was largely represented in the other two attributes.
Security Effectiveness. While the primary focus of this objective is to prevent the
catastrophic loss of an aircraft, it is construed more broadly to include significant security
breaches inside the sterile areas of a U.S. airport or cabin of a commercial aircraft inside the
United States. The reasoning is straightforward—significant security breaches can have large
disruptive economic effects. The attributes for security effectiveness include fatalities (0–400 per
year), injuries (0–400 per year), security breaches in sterile areas (0–90 per year), and aircraft
cabins (0–5 per year). Deterrence and area under the detection sensitivity curve (measure of
detection effectiveness depicting a plot of true positives to false positives) are included in Table
1 because they are critical components of security effectiveness. They are not included in the
MAU analysis directly because their contribution is captured in the probability trees discussed in
a later section.
Impact of DARMS: Discussions about the potential impact of the DARMS approach
suggested that within the security effectiveness objective, there should be a gain in security
effectiveness due to an optimal mix of game theory inspired countermeasures and better
intelligence shared about flight-by-flight risks. However, detection probabilities may not
necessarily improve with the DARMS approach as it seeks to adjust risk across flights. As an
example, screening settings for low risk passengers might be relaxed and hence detection
probabilities decrease accordingly.
Passenger Satisfaction. With regards to passengers, TSA must promote confidence in
the TSA’s ability to provide security while maintaining fair and not overly burdensome
procedures to board an aircraft. Attribute scales for passenger satisfaction include average wait
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
11
time (5–20 minutes per passenger), variation in wait time (0–10 minutes per passenger), touch
rate (0–10%), divesture (1–6 bins per passenger), perceptions of fairness (1–10 constructed
scale) and perceptions of security effectiveness (1–10 constructed scale).
Impact of DARMS: Discussions about the potential impact of DARMS on passenger
satisfaction were mixed across SMEs. Considering each attribute scale, average wait time is
predicted to decrease under DARMS unless the increased real time risk assessment caused
throughput delays. In contrast, variation in wait time is expected to increase under DARMS
because of the inability of passengers to fully predict whether or not they will receive expedited
screening. Touch rate and divesture are expected to decrease under DARMS largely due to the
increased percentage of passengers receiving expedited screening. Perception of fairness and
perceptions of security effectiveness are both expected to decrease under DARMS. With
increased risk classification comes the possible perception among passengers that not everyone is
being treated equally (and hence fairly) and possible confusion over the use of randomized
procedures. While a game theory guided approach to countermeasures may optimally reduce
flight risk, this approach will also be hard to explain to passengers. Surveys conducted over the
last two years by the risk perception and risk communication group at CREATE (Burns &
Dillon-Merill 2013a, 2013b) have consistently shown that expedited screening is perceived by
many as less effective than standardized screening.
Economic Costs of Security Breaches in Airport Sterile Areas or Aircraft Cabin.
Research by the economic and risk perception groups at CREATE (Burns et al., 2013; Giesecke,
Burns, Rose, Barrett, & Griffith, 2014; Giesecke et al., 2012; Rose et al., 2015) have shown the
major contributor to economic costs following a terrorist attack is public reaction. Thus, one
significant objective of the TSA in attempting to prevent terrorist attacks is to minimize the costs
associated with the catastrophic loss of an aircraft. In addition to the catastrophic event, though,
the more common and disruptive significant security breaches inside the sterile areas of a U.S.
airport or cabin of a commercial aircraft inside the United States. needs to also be considered.
This objective has only one attribute scale: the consequence in dollars ($0–$100B per year) and
is predicated on security breaches and most particularly the catastrophic loss of an aircraft.
Impact of DARMS: Barring any significant security breach, DARMS is not expected to
have a material impact on economic costs. However, should a terrorist attack occur it is predicted
the economic costs under DARMS would be considerably greater due to a significant loss of
public confidence and possible Congressional outcry (e.g., “TSA had a screening procedure in
place that completely prevented a successful attack and then switched to DARMS to reduce
operational costs”).
Operational Efficiency. TSA needs to minimize the use of resources as operationally
efficient as possible at all times. Attributes scales to capture the various operational efficiency
dimensions include passenger throughput (200–500 passengers per hour assuming two lanes
each for standard and expedited screening), passenger cycle time (5–10 minutes per passenger
screening—it is the same as passenger wait time but SMEs felt it provided added insight for
evaluating operational efficiency), passenger FTE (,02–.20 TSOs per passenger screening),
passenger false alarm rate (0–10%), TSO utilization rate (50–100%), responsiveness to reducing
unexpected increases in passenger wait times (10–60 minutes) and resilience, that is the ability to
restore system capacity following a major security breach (1–48 hours).
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
12
Impact of DARMS: DARMS is expected to improve operational efficiency across almost
all attribute scales. Throughput, cycle time, and passenger false alarms would improve because
more passengers would be receiving expedited screening. TSO utilization rate is predicted to
increase because DARMS with its more complex risk assessments would place greater demands
on TSO skills. Situational factors occasionally produce higher than desired passenger wait times.
It is thought in these situations that DARMS will have an increased capacity to respond quickly
to reduce passenger wait time. It is also thought that DARMS would improve system resilience
because of increased ability to share risk information and improvements in operational
efficiency. Passenger FTE is expected to increase due to the increased complexity of DARMS
(Note, this is the only operational efficiency attribute scale that DARMS does not improve).
Operational Costs. Given a constrained (and ever contracting budget), TSA must
minimize the costs of all procedures involved with achieving a desirable level of security
effectiveness, passenger satisfaction and regulatory compliance. The attribute scales for
operational cost are the costs of maintaining security effectiveness, passenger satisfaction and
regulatory compliance ($6–$11 per passenger screening). It does not include the economic costs
of significant security breaches (which are captured separately, see above).
Impact of DARMS: DARMS is expected to reduce operational costs due to improvements
in operational efficiency. However, there was concern raised that the complexity of DARMS
might require more personal and sophisticated procedures and thus increase operational costs
off-setting some of the improvements.
TSO Job Satisfaction. In addition to concern for the passengers and the aviation
industry, TSA needs to maximize the job satisfaction of the transportation security officers
(TSOs). The TSOs are on the frontline in airports, conduct the passenger and baggage screening
procedures and represent the TSA to the traveling public. Attribute scales for quantifying TSO
job satisfaction include TSO morale (1–10 constructed scale), perceptions of security
effectiveness (constructed scale 1–10) and TSA employee turnover (10–50%). DARMS is
expected to decrease TSO morale and perceptions of security effectiveness and hence also
increase job turnover.
Impact of DARMS: DARMS will place more complex demands on TSOs in real time and
so there could be lower job satisfaction. Likewise, the DARMS approach might appear less safe
to TSOs for the same reasons it might appear less safe to passengers.
Aviation Industry Vitality Costs. The intent is to minimize costs to airports and airlines
as they adjust to additional security requirements posed by the TSA. We include two attribute
scales to capture the potential impact to this objective: the costs of reconfiguration at airport
checkpoints ($0–$1B per year) and the reduction of operating revenue from additional security
compliance ($0–$1B per year). The former represents costs that the airports are likely to pass
along to airlines and the later represents possible reduction in passenger demand because of
increased ticket prices. Reduction of revenue passenger miles from additional security
compliance and reduction of revenue per available seat mile from additional security compliance
are measures the airline industry routinely use to measure performance, however, they are not
used in this analysis because they were thought to be highly related to the first two measures.
Impact of DARMS: DARMS is expected to have no material impact on these costs in the
absence of significant security breaches. However, like economic costs described above it is
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
13
predicted DARMS would result in higher costs to the aviation industry following a significant
security breach (especially the catastrophic loss of an aircraft) as the TSA seeks to regroup.
Table 2 summarizes the objectives, the attributes with scales, and the impact of DARMS relative
to the current system.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
14
Table 2. TSA Objectives, Attributes, Scales and DARMS Impacts
Objectives1 Metric
Scale Units
Predictions for DARMS Impact
Security Effectiveness
Fatalities Number of people/year2 0–400 Game theory model might help
reduce number for DARMS-not sure.
Injuries Number of people/year2 0–400 Game theory model might help
reduce number for DARMS-not sure.
Security Breaches Inside
Airport Sterile Area
Number of security breaches/year3 0–90 Game theory model could help
reduce number with DARMS.
Security Breaches Inside
Cabin of Aircraft
Number of security breaches/year 0–5 Game theory model could help reduce number with DARMS
Deterrence RAND Scale (“no meaningful deterrence” to “highest level of deterrence”)
4
0–5 Less deterrence if perception of security decreases, More if perception of security increases (harder to beat the randomness)
Detection Sensitivity Plot of P(Detection/Threat) versus P(Detection/Non-Threat). Larger is better. Area maximum of 1.
4a
.5–1 Game theory model predicts larger area under curve due to optimal mix of counter measures.
Passenger Satisfaction
Average Wait Time Average Minutes/passenger screening
5–20 Decrease under DARMS – the more people sent through expedited screening, the lower the average wait time
Variation Wait time Standard Deviation in Minutes/passenger screening
0–10 May Increase under DARMS if passengers sometimes have expedited screening and sometimes don’t
Touch Rate: Passengers Touched by TSO
Percentage of passengers 0–10% Decrease under DARMS – less AIT used, less alarms, lower touch rate
Divesture Number of Bins Used/passenger screening
5
1–6 Decrease under DARMS – the more people sent through expedited screening, the less bins
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
15
Objectives1 Metric
Scale Units
Predictions for DARMS Impact
Perceptions of Fairness Constructed scale (1–10)6 1–10 Perception hard to predict –
may increase (if see random selections as fair), may stay the same. Probably will decrease (if see others getting “better” treatment
Perceptions of Security Effectiveness
Constructed Scale (1–10)6 1–10 Perception hard to predict –
TSA certainly wants people to perceive as at least as safe if not more safe, DARMS probably will be perceived as less safe
Economic Costs of Security Breaches in Airport Sterile Areas or Aircraft Cabin
Total Consequences Dollars/year7 $0B–
$100B With a significant security breach DARMS might cause more costly public reaction.
Operational Efficiency
Passenger Throughput Passengers/hour8
200–500
Increase under DARMS
Passenger Cycle time Minutes/passenger screening9 5–20 Decrease under DARMS
Passenger FTE TSOs/passenger screening10
.02–.20
Pre-check lanes require less but DARMS may increase due to complexity.
Passenger False Alarm Rate
Percentage of passengers 0–10% Less with less AITs
TSO Utilization Rate Percentage time TSOs engaged in screening
50–100%
Could increase because of complexity.
Responsiveness Minutes needed to reduce passenger wait time to a desired level (e.g., 10 minutes).
10–60 Better under DARMS because of throughput increases.
Resilience Hours to restore commercial aviation system to normal operating capacity following a breach of security in a U.S. airport or aircraft cabin.
1–48 Better under DARMS (assumes increased throughput will aid in returning to normal faster)
Operational Costs
Total Passenger Security Costs (e.g., FTE, Equipment, FAMS, Pre-screening)
Dollars/passenger screening11
$6–$11
Once past fixed costs, DARMS could be operationally cheaper – less fixed equipment
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
16
Objectives1 Metric
Scale Units
Predictions for DARMS Impact
TSO Job Satisfaction
TSO Morale Constructed Scale (1–10) 1–10 Decrease under DARMS
TSO Perceptions of Security Effectiveness
Constructed Scale (1–10) 1–10 Decrease under DARMS
TSO Turnover Employee Percentage per year 10–50%
Decrease under DARMS
Aviation Industry Vitality Costs
Costs of Reconfiguration at Airport Checkpoints
Dollars/year12
$0–$1B
Larger fixed cost to get to DARMS capability
Reduction of Operating Revenue from Additional Security Compliance
Dollars/year13
$0–$1B
Some additional cost associated with sharing flight information, other costs.
1 Based largely on Keeney and von Winterfeldt (2011) and von Winterfeldt and O’Sullivan (2006) papers;
DARMS Business Case Quantification Framework-October 29 2014 (Deloitte Team, 2015). 2 Range based on available 2014 Estimates.
3 Range based on a conversation with Jeff Price, author of Practical Aviation Security (Price & Forest,
2013) in which he provided a ballpark estimate of the number of serious security breaches in airports nationwide to be from 30 to 60 per year. 4 Based on RAND scale reported in RAND Report entitled “Understanding the Role of Deterrence in
Counterterrorism Security” by Andrew R. Morral and Brian A. Jackson (2009). The scale is takes into account the perceived costs by adversaries to overcome a security system, the perceived costs to engage a security system and the perceived likelihood of defeating a security system. It ranges from “no meaningful deterrence” to “highest level of deterrence.” This attribute was not used in the MAU analysis. 4a
ROC standard for Receiver Operating Curve. Each point on the curve represents a ratio of Probability (Detection/Threat) to Probability (Alarm/No Threat). The curve is generated by computing these ratios for a wide range of signal thresholds. The larger the area under the curve the more use the detection system is. That is, across a wide array of thresholds the chance of detecting true threats is larger than false alarms. This attribute was not used in the MAU analysis. 5 Based on conversation with Kristen Best and Todd Trafford (from TSA): 1–2 bins for expedited
passenger screening and 3–4 bins for standard passenger screening.
6 This metric is based on a scale (1–10) developed to measure perceived safety of commercial airline
travel under TSA standard versus expedited screening procedures. The mean rating under in national samples is approximately 7. 7 Range based on available 2014 Estimates.
8 Based on conversation with Kristen Best and Todd Trafford (from TSA): 250 passengers per hour for
expedited passenger screening and 110–120 passengers per hour for standard passenger screening. 9 Cycle time in this case is exactly the same metric as passenger wait time and a close variation of
passenger throughput. According to Kristen Best, it provides an additional perspective to the throughput metric.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
17
10 Based on conversation with Kristen Best and Todd Trafford (from TSA): 12 TSOs for two expedited
passenger screening lanes and 13 TSOs for two standard passenger screening lanes (12/250/hr = .048 and 13/110/hr = .118 respectively). 11
Based on conversation with Kristen Best and Todd Trafford (from TSA) to look at total TSA aviation budget and passengers screened: Total TSA budget for 2013 for aviation was approximately $5.3B and about 640M domestic and international passengers were screened in 2013. Hence the average security cost per passenger was about $5.2B/640M = $8.26 assuming 100% of the aviation budget went toward security. The report can be found on
http://www.dhs.gov/sites/default/files/publications/MGMT/FY%202014%20BIB%20-%20FINAL%20-508%20Formatted%20(4).pdf 12
In 2013 for all U.S. Carriers domestically and internationally Operating Income was approximately $200B. Suppose TSA compliance requirements caused airports to pass along their checkpoint remodeling costs to the airlines by one half percent of their Operating Revenues or $1B. 13
In 2013 for all U.S. Carriers domestically and internationally Operating Income was approximately $200B. Let’s suppose that additional security compliance requirements would cause airline ticket prices to increase resulting in a one half percent decrease in Operating Revenues or $1B.
Consequences and Uncertainties of Consequences
Consequences were approximated to illustrate how a MAU analysis could be applied to
this evaluation of DARMS. The assessment did take into account much publically available
information and conversations with SMEs. Ultimately there were four alternative-scenarios of
interest: consequences under the current approach with no successful attack, consequences under
the current approach with a successful attack, consequences under DARMS with unsuccessful
attack and consequences under DARMS with a successful attack. Note that this study does not
distinguish between unsuccessful attack and no attack, an important point discussed later in the
report.
Those objectives for which it is believed there might be significant uncertainty associated
with the consequences, even in the case where a successful attack occurred, are assessed on a
three-point percentile scale (5%, 50% and 95%). For example, given that a successful attack has
occurred, the fatalities under DARMS was assessed to be 50 at the 5th
percentile, 175 at the 50th
percentile and 350 at the 95th
percentile. Three point assessments were completed for fatalities,
injuries, security breaches inside the sterile area of airports and aircraft cabin, passenger and
TSO perceptions of security effectiveness, TOS morale and turnover. These percentile
assessments were used to eventually generate a single point estimate for each attribute across all
four contingencies. Table 3 describes the three point consequence assessments and Table 4
details the single point consequence assessments. The consequences from Table 4 were used to
compute the individual attribute utilities to be discussed next.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
18
Table 3. TSA Consequence Uncertainties Assessments
Objectives Attributes
Current Unsuccessful
Attack
Current Successful
Attack
DARMS Unsuccessful
Attack
DARMS Successful
Attack
Percentile 5%/50%/95% 5%/50%/95%
Security
Effectiveness
Fatalities 0 50/175/350 0 50/175/350
Injuries 0 50/175/350 0 50/175/350
Breaches-Airport
30/45 /60 30/45/60 27/40.5/54 27/40.5/54
Breaches-Cabin
1/2.5/4 1/2.5/4 .5/1.25/2 .5/1.25/2
Passenger
Satisfaction
Perceptions
Security
5/7/9 1.5/2/5 4/6.3/8 1/1/4
Economic Costs $B
0 10/75/90 0 10/85/95
TSO Job
Satisfaction
TSO Morale 4/6/8 1/2/4 3/4/7 1/1/2
Perceptions-Security
4/6/8 1/2/4 3/4/7 1/1/2
TSO Turnover Rate
15/20/25 20/30/40 20/22/30 25/35/45
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
19
Table 4. TSA Consequences for Alternative-Outcome Contingencies
Objectives Attributes Current Unsuccessful
Attack
Current Successful
Attack
DARMS Unsuccessful
Attack
DARMS Successful
Attack
Security Effectiveness
Fatalities 0 184.25 0 184.25
Injuries 0 184.25 0 184.25
Breaches-Airport 45 45 40.5 40.5
Breaches-Cabin 2.5 2.5 1.25 1.25
Passenger Satisfaction
Average Wait Time 7.5 7.5 7.12 7.12
Variation Wait Time 5 5 5.5 5.5
Touch Rate 5 5 4 4
Divesture 2.5 2.5 2 2
Perceptions-Fairness 7.5 7.5 6 6
Perceptions-Security 7 2.46 6.19 1.56
Economic Cost $B 0 65.75 0 72.98
Operational Efficiency
Throughput 350 350 385 385
Cycle Time 7.5 7.5 7.12 7.12
Passenger FTE .08 .08 .09 .09
Passenger False Alarm 5 5 4 4
TSO Utilization Rate 75 75 90 90
Responsiveness 30 30 24 24
Resilience 2 36 1.5 24
Operational Cost $ 8.26 8.26 7.43 7.43
TSO Job Satisfaction
TSO Morale 6 2.19 4.37 1.19
Perceptions-Security 6 2.19 4.37 1.19
TSO Turnover Rate 20 30 23.11 35
Industry Vitality Cost $B
Reconfiguration Costs .05 .25 .06 .30
Reduction Revenue Costs
.05 .25 .06 .30
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
20
Single Attribute Utilities and Weights
Utility Assessment. For purposes of this study it is assumed that all utilities are linear in
consequences, which assumes that values are neither marginally increasing or decreasing in the
consequence scales units, and risk neutrality. The consequence scales for each attribute were
constructed in a way that a linear utility function assumption is reasonable. Given the size of the
U.S. transportation system, one might expect stakeholders would be risk neutral. Further
assessment would be necessary to estimate both the nature and degree of non-linearity in each
attribute scale utility function. The common assumption of risk aversion is complicated by the
natural loss frame placed on this problem, which often leads to risk seeking utility functions.
Hence, utility functions for each attribute scale were constructed as a linear mapping of the
attribute range onto a 0–1 scale respecting least preferred and most preferred end points:
U(X) = (X – worst-level) / (Best-level – Worst-level)
For example, the utility function for fatalities mapped a 0–400 range onto a 0–1 scale with the
linear function U(fatalities) = 1 – .0025*(fatalities). This utility function would assess zero
fatalities as a 1 and 184 fatalities as depicted in Table 4 would be a utility of .54. These single
attribute utilities for each of the four alternative-scenarios considered were then weighted and
combined to form a multi-attribute utility function.
Weight Assessment. An additive MAU model was assumed for purposes of this
demonstration. This model assumes that attribute scales are utility independent. Again, attributes
were constructed in a manner to make this a reasonable assumption. Additional assessments
would be required to identify attribute dependencies and estimate non-additive MAU model
parameters. There is ample research (von Winterfeldt & Edwards, 1986) that additive MAU
models are often a good approximation of non-additive MAU models, and avoids potential
assessment errors problematic for non-additive MAU models.
The weights were assessed using a swing weight approach (Clemen & Reilly, 2001;
Eisenfuhr, Weber, & Langer, 2010). The assessment began with determining high-level weights
for each of the seven objectives. This was done according to the following standard steps for
assessing swing weights:
1) Consequence ranges for each objective were considered (including the more detailed
attribute scales);
2) Each objective was imagined at its least preferred level across all its attribute scales;
3) Each objective in turn, was then imagined at its most preferred level across all its
attribute scales;
4) The eight hypothetical consequence sets (i.e. least preferred on all consequence levels)
and seven sets in which one objective was at its most preferred consequence levels were
reviewed and ranked;
5) A rating of 0 was given to the set in which all objectives were at their least preferred
levels, a 100 was given to the objective with the highest ranking and all other objectives
were given a rating between 0 and 100 consistent with their ranking and
6) The ratings were summed and then normalized to produce weights for the set of
objectives that summed to 1.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
21
This procedure was then repeated for each set of attribute scales within each objective (if there
was more than one) producing lower-level weights that summed to 1. Finally, each lower-level
weight was multiplied by its corresponding objective weight to determine its effective weight in
the multi-attribute utility function. Utilities and weights can be seen in Table 5. It should be
noted that this assessment was based on the perceptions and priorities of one researcher and not
stakeholders at the TSA and aviation industry.
Table 5. Weights and Consequence Utilities at the Attribute Scale Level
Objectives Attributes
Attribute Weight % Utilities
Current Unsuccessful
Attack
Current Successful
Attack
DARMS Unsuccessful
Attack
DARMS Successful
Attack
Security Effectiveness
Fatalities 8.9 1.00 .539 1.00 .539 Injuries 6.2 1.00 .539 1.00 .539 Brch. Airport 3.4 .500 .500 .550 .550 Brch. Cabin 4.4 .500 .500 .750 .750
Passenger Satisfaction
Avg. Wait Time
2.4 .833 .833 .858 .858
Var. Wait Time
1.6 .500 .500 .450 .450
Touch Rate 2.8 .500 .500 .600 .600 Divesture 2.0 .700 .700 .800 .800 Fairness 3.1 .722 .722 .556 .556 Security 4.1 .667 .162 .577 .062
Economic
Cost $B
13.8 1.00 .343 1.00 .270
Operational Efficiency
Throughput 1.5 .500 .500 .617 .617 Cycle Time 1.5 .833 .833 .858 .858 FTE 2.2 .694 .694 .611 .611 False Alarm 2.1 .500 .500 .600 .600 TSO Utilization
.90 .500 .500 .800 .800
Responsive 1.5 .600 .600 .720 .720 Resilience 1.8 .978 .255 .989 .510
Operational
Cost $
18.4 .548 .548 .714 .714
TSO Job Satisfaction
TSO Morale 6.0 .556 .132 .374 .021 Security 4.8 .556 .132 .374 .021 Turnover Rate 4.2 .750 .500 .672 .375
Industry Vitality Cost $B
Reconfiguration .80 .950 .750 .945 .700 Revenue Costs 1.5 .950 .750 .945 .700
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
22
Objectives Hierarchy
Figure 1 displays the seven fundamental objectives with their corresponding attribute
measures. Differing from the format in previous tables above, the objectives are displayed
according to their assessed weights (e.g., security effectiveness is shown first followed by
operational costs and so forth) and attribute scales within each objective are similarly ordered.
For example, within security effectiveness, fatalities are given the largest weight followed by
injuries, security breaches inside an aircraft cabin and security breaches inside the sterile area of
an airport. Similarly, for passenger satisfaction, perceptions of security are given the largest
weight followed by perceptions of fairness, touch rate, average wait time, divesture and lastly
variation in wait time. It should be noted that an attribute measure may have a large relative
weight within its objective but its overall influence will be also determined by the weight
assigned to the objective. For example, the attribute measure reduction of operating revenue has
a weight of .667 however the overall influence will still be relatively small because its objective
has a weight of only .023.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
23
Figure 1. Objectives Hierarchy with Attribute Scales
Fatalities
0.388
Injuries
0.271
Security Breaches inside aircraft cabin
0.194
Security Breaches inside sterile Area
0.146
Security Effectiveness
0.230
Operational Costs
0.184
Passenger Perceptions of Security
0.253
Passenger Perceptions of Fairness
0.190
Passenger Touch Rate
0.177
Avg Wait Time
0.152
Divesture
0.127
Variation in Wait Time
0.101
Passenger Satisfaction
0.161
TSO Morale
0.400
TSO Perceptions of Security
0.320
TSO Turnover
0.280
TSO Job Satisfaction
0.149
Economic Costs
0.138
Passenger FTE
0.195
Passenger False Alarms
0.182
Resilience
0.156
Passsenger Throughput
0.130
Responsiveness
0.130
Cycle Time
0.130
TSO Utilization Rate
0.078
Operational Efficiency
0.115
Reduction of Operating Revenue
0.667
Reconfiguration Costs
0.333
Industry Vitality
0.023
OVERALL
1.000
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
24
Multi-Attribute Utilities
To combine the utilities modeled at the attribute scale level into a single multi-attribute
utility function for each of the four alternative-scenarios the weighted additive model was used.
Adding the utilities assumes that each utility can be assessed without considering consequence
levels of the other attribute scales, which can be a strong assumption, but additivity has been
shown to be a good approximation of more general multiplicative or multi-linear utility
functions; examination of stakeholders’ preferences can be addressed at a later time. As an
example, using values from Table 5, an abbreviated version of the multi-attribute utility function
for the Current approach assuming a successful attack is given in equation (1):
Utility (X) = .089 * U (fatalities) + … + .015 * U (airline operation revenue costs) = .470. (1)
The corresponding multi-attribute utility for DARMS was .492. A detailed analysis of these
inputs follows in the next section.
MAU Analysis Output
Ranking of the Four Alternative-Scenarios
Figure 2 displays the overall utilities under each of the four alternative-scenarios. From
this figure, one can see in this assessment that DARMS has a higher overall utility than the
Current approach whether there is a successful attack or not, and the difference is about the same
regardless of what attack outcome occurs. The color-coded bar chart allows a comparison of the
DARMS and Current approaches across the two attack outcomes and also shows the relative
contribution of each objective to overall utility (i.e. larger bars indicate greater contribution to
overall utility). First, one can see that most objectives have a markedly lower utility if a
successful attack occurs with the exception of operational efficiency and industry vitality which
are only somewhat lower. Also, DARMS performs better on security effectiveness, operational
costs and operational efficiency over both attack outcomes. In contrast, the Current approach
performs better on TSO satisfaction over both attack outcomes. Passenger satisfaction is about
the same for DARMS and the Current approach over both outcomes (passenger perceptions of
security within this objective fares worse under DARMS should there be a successful attack).
Economic cost is the same for DARMS and the Current approach if no attack occurs, but
DARMS performs worse if an attack occurs. In this assessment, this figure suggests that
DARMS fares better overall regardless of whether an attack occurs however it does not fare best
across all objectives.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
25
Figure 2. Relative Contribution to Overall Alternative-Scenario Utilities.
Impact of Weight on Overall Utility
Figure 3 depicts the weights at the attribute scale level in order of their relative
contribution to overall utility. These weights reflect not only the relative assessed weight within
each objective but also the relative assessed weight of each objective. In this assessment,
operational costs (18.4%) and economic costs (13.8%) clearly stand out and their corresponding
consequences comprise almost a third of the overall utility. Consequences from these two
attribute scales together with fatalities (8.9%), injuries (6.2%), TSO morale (6.0%), TSO
perceptions of security effectiveness (4.8%), security breaches inside the aircraft cabin (4.5%)
and TSO turnover (4.2%) comprise two thirds of the overall utility. The consequences from these
attributes together with passenger perceptions of security effectiveness (4.1%), security breaches
inside airport sterile areas (3.4%), and passengers perceptions of fairness (3.1%) account for over
eighty percent of the overall utility-that is, half of the attributes impact 80% of the utility.
Ranking for OVERALL Objectives
AlternativeScenario
DARMS, No Successful Attack
Current, No Successful Attack
DARMS, Successful Attack
Current, Successful Attack
Utility
0.749
0.726
0.493
0.471
Security Effectiveness
TSO Job Satisfaction
Industry Vitality
Operational Costs
Economic Costs
Passenger Satisfaction
Operational Efficiency
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
26
Figure 3. Ranking of Attribute Weights
Figures 4 and 5 display the contribution of each attribute scale to overall utility. With an
unsuccessful attack, the attribute scales that most favor DARMS are operational costs and
security breaches inside the cabin of an aircraft while TSO morale and perceptions of security
most favor the Current approach. With a successful attack, operational costs and security
breaches inside the cabin of an aircraft again most favor DARMS while economic costs and TSO
morale most favor the Current approach.
Measure
Operational Costs
Economic Costs
Fatalities
Injuries
TSO Morale
TSO Perceptions of Security
Security Breaches inside aircraft cabin
TSO Turnover
Passenger Perceptions of Security
Security Breaches inside sterile Area
Passenger Perceptions of Fairness
Passenger Touch Rate
Avg Wait Time
Passenger FTE
Passenger False Alarms
Divesture
Resilience
Variation in Wait Time
Reduction of Operating Revenue
Passsenger Throughput
Responsiveness
Cycle Time
TSO Utilization Rate
Reconfiguration Costs
Weight
18.4
13.8
8.9
6.2
6.0
4.8
4.5
4.2
4.1
3.4
3.1
2.8
2.4
2.2
2.1
2.0
1.8
1.6
1.5
1.5
1.5
1.5
0.9
0.8
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
27
Figure 4. Relative Contribution of Attribute Differences to overall Alternative-Scenario
Utilities with Unsuccessful Attack
OVERALL Objectives Utility for DARMS, No Successful Attack
Current, No Successful Attack
Total Difference
0.749
0.726
0.024
Total Difference
Operational Costs
Security Breaches inside aircraft cabin
TSO Morale
TSO Perceptions of Security
Passenger Perceptions of Fairness
Passenger Perceptions of Security
TSO Turnover
Passenger Touch Rate
TSO Utilization Rate
Passenger False Alarms
Divesture
Passenger FTE
Responsiveness
Passsenger Throughput
Security Breaches inside sterile Area
Other
Difference
0.024
0.031
0.011
-0.011
-0.009
-0.005
-0.004
-0.003
0.003
0.003
0.002
0.002
-0.002
0.002
0.002
0.002
Current, No Successful Attack DARMS, No Successful Attack
Other 0.000
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
28
Figure 5. Relative Contribution of Attribute Differences to overall Alternative-Scenario
Utilities with Successful Attack
Illustrations of Tradeoffs
Economic Costs Equivalents. As an illustration of how to equate a set of attributes with
the value of a particular attribute, Table 6 displays tradeoffs between economic costs and the
other attributes as assessed by one of the research team. For each attribute an economic cost is
provided that is equivalent to moving from the attribute’s least preferred consequence to its most
preferred consequence. For example consider fatalities. In this assessment, the following two sets
of consequences are equally preferred by the team member providing the assessments: (400
fatalities/yr., $0 economic costs/yr.) and (0 fatalities/yr., $64.8B/yr.). In other words, there is a
willingness to incur an economic cost from a significant security breach of $64.8B per year to
reduce fatalities from 400 to 0 per year. The largest economic cost equivalent is $100B to move
from an operational cost of $9.75 per passenger screening to $6.00 per passenger screening. The
least economic cost equivalent is for airport screening reconfiguration costs. Here the economic
costs equivalent is $5.6B to move from $1B to $0B in reconfiguration costs. In terms of
economic costs tradeoffs, the top five attributes are operational costs, fatalities, injuries, TSO
morale and TSO perceptions of security effectiveness.
OVERALL Objectives Utility for DARMS, Successful Attack
Current, Successful Attack
Total Difference
0.493
0.471
0.022
Total Difference
Operational Costs
Security Breaches inside aircraft cabin
Economic Costs
TSO Morale
TSO Perceptions of Security
TSO Turnover
Passenger Perceptions of Fairness
Resilience
Passenger Perceptions of Security
Passenger Touch Rate
TSO Utilization Rate
Passenger False Alarms
Divesture
Passenger FTE
Responsiveness
Passsenger Throughput
Security Breaches inside sterile Area
Other
Difference
0.022
0.031
0.011
-0.010
-0.007
-0.005
-0.005
-0.005
0.005
-0.004
0.003
0.003
0.002
0.002
-0.002
0.002
0.002
0.002
Current, Successful Attack DARMS, Successful Attack
Other -0.001
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
29
Table 6. TSA Attribute Equivalents Expressed in Economic Costs
Objectives Attributes Units
Least Preferred
Most Preferred
Equivalent Economic Costs $B
Security Effectiveness
Fatalities Number of people/year 400 0 $64.8
Injuries Number of people/year 400 0 $45.3
Breach. Airport Number of security breaches/year 90 0 $24.4
Breach. Cabin Number of security breaches/year 5 0 $32.2
Passenger Satisfaction
Avg. Wait Time Avg. Minutes/passenger screening 20 5 $17.8
Var. Wait Time Standard Deviation in Minutes/passenger screening
10 0 $11.7
Touch Rate Percentage of passengers 10% 0% $20.7
Divesture Number of Bins Used/passenger screening
6 1 $14.8
Fairness Constructed scale (1–10) 1 10 $22.2
Security Constructed Scale (1–10) 1 10 $29.5
Economic
Cost $B
Dollars/year
Operational Efficiency
Throughput Passengers/hour
200 500 $10.8
Cycle Time Minutes/passenger screening 20 5 $10.8
FTE TSOs/passenger screening .2 .02 $16.2
False Alarm Percentage of passengers 10% 0% $15.1
TSO Utilization Percentage time TSOs engaged in screening
50% 100% $6.5
Responsive Minutes needed to reduce passenger wait time to a desired level (e.g., 10 minutes).
60 10 $10.8
Resilience Hours to restore commercial aviation system to normal operating capacity following a breach of security in a U.S. airport or aircraft cabin.
48 1 $13.0
Operational
Cost $
Dollars/passenger screening $9.75 $6.00 $100.0
TSO Job Satisfaction
TSO Morale Constructed Scale (1–10) 1 10 $43.2
Security Constructed Scale (1–10) 1 10 $34.6
Turnover Rate Employee Percentage per year 50% 10% $30.3
Industry Vitality Cost $B
Reconfiguration Dollars/year $1B $0B $5.6
Revenue Costs Dollars/year $1B $0B $11.1
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
30
Operational Costs Equivalents. Table 7 displays tradeoffs between operational costs
per passenger screening and the other attributes. In this assessment the baseline operational cost
per passenger screening is $6.00. As before, two sets of equally preferred consequences are
presented. For example, for fatalities the following two consequence sets are equally preferred
(400 fatalities/yr., $6.00) and (0 fatalities/yr., $8.43). In other words, there is a willingness to
incur an additional $2.43 per passenger screening to move from 400 fatalities per year to 0
fatalities per year. In terms of operational costs the top five attributes are economic costs,
fatalities, injuries, TSO morale and TSO perceptions of security effectiveness.
Table 7. TSA Attribute Equivalents Expressed in Operational Costs
Objectives Attributes Units
Least Preferred
Most Preferred
Equivalent Operational
Costs
Security
Effectiveness
Baseline $6.00/screening
Fatalities Number of people/year 400 0 $8.43
Injuries Number of people/year 400 0 $7.70
Breach. Airport Number of security breaches/year 90 0 $6.91
Breach. Cabin Number of security breaches/year 5 0 $7.21
Passenger
Satisfaction
Avg. Wait Time Avg. Minutes/passenger screening 20 5 $6.66
Var. Wait Time Standard Deviation in Minutes/passenger screening
10 0 $6.44
Touch Rate Percentage of passengers 10% 0% $6.78
Divesture Number of Bins Used/passenger screening
6 1 $6.56
Fairness Constructed scale (1–10) 1 10 $6.83
Security Constructed Scale (1–10) 1 10 $7.18
Economic
Cost $B
Dollars/year $9.75
Operational Efficiency
Throughput Passengers/hour
200 500 $6.41
Cycle Time Minutes/passenger screening 20 5 $6.41
FTE TSOs/passenger screening .2 .02 $6.61
False Alarm Percentage of passengers 10% 0% $6.57
TSO Utilization Percentage time TSOs engaged in screening
50% 100% $6.24
Responsive Minutes needed to reduce passenger wait time to a desired level (e.g., 10
60 10 $6.41
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
31
Objectives Attributes Units
Least Preferred
Most Preferred
Equivalent Operational
Costs
minutes).
Resilience Hours to restore commercial aviation system to normal operating capacity following a breach of security in a U.S. airport or aircraft cabin.
48 1 $6.49
Operational
Cost $
Dollars/passenger screening $9.75 $6.00
TSO Job
Satisfaction
TSO Morale Constructed Scale (1–10) 1 10 $7.62
Security Constructed Scale (1–10) 1 10 $7.29
Turnover Rate Employee Percentage per year 50% 10% $7.13
Industry
Vitality
Cost $B
Reconfiguration Dollars/year $1B $0B $6.21
Revenue Costs Dollars/year $1B $0B $6.42
Overall Utility and Single Attribute Equivalents. In addition to comparing the four
alternative-scenarios in terms of utility, it can be more intuitive (perhaps) to compare them in
terms of the consequences of a single attribute. This is done by setting all other attribute scales to
their most preferred levels and then moving the attribute scale of interest to a level that produces
the same overall utility as before. That is, the level of attribute scale of interest is adjusted so that
the following equality holds true:
Utility (proposed consequence settings) = Utility (all other attribute settings at their most
preferred levels, adjusted attribute of interest level).
For example, the proposed set of consequences for an unsuccessful attack under DARMS
has an overall utility of .749. Looking specifically at economic costs, if all attribute scales are set
to their preferred level and economic costs are increased to $181.7B from a proposed level of
$0B (due to unsuccessful attack)—this adjustment also produces an overall utility of .749. The
Current approach under an unsuccessful attack has a utility of .726 and an equivalent economic
cost of $198.8B. Hence the difference in utilities from DARMS to Current approach is .023 or
equivalently $17.1B in economic costs per year. The difference in utilities from DARMS and the
Current approach under a successful attack is .022 or equivalently $15.9B in economic costs per
year. In terms of operational costs, under an unsuccessful attack the difference in equivalent
operational costs per passenger screening favors DARMS by $0.63 and under a successful attack
by $0.57 per passenger screening. In terms of fatalities, under an unsuccessful attack the
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
32
difference in equivalent fatalities favors DARMS by 105 fatalities per year and under a
successful attack by 98 fatalities per year.
Table 8. TSA Overall Utility and Single Attribute Equivalents: Economic Costs, Operational Costs and Fatalities
Objectives Attributes
DARMS Unsuccessful
Attack
Current Unsuccessful
Attack
DARMS Successful
Attack
Current Successful
Attack
Overall Utility .749 .726 .493 .471
Economic Costs $B
$181.7B $198.8B $367.4B $383.3B
Operational Cost $
$12.82 $13.45 $19.77 $20.34
Fatalities 1123 1228 2270 2368
Sensitivity Analysis for Objective Weights
Security Effectiveness. It is often helpful to examine how sensitive models are to
changes in assessment inputs such as the weights. Figure 6 displays how the utility for each of
the four alternative-scenarios changes as the objective weight for security effectiveness changes.
The vertical scale is the overall utility for each of the four alternative-scenarios, the horizontal
scale is the percent weight placed on security effectiveness and the vertical bar is the proposed
weight for security effectiveness (23%). Comparing DARMS and the Current approach under no
successful attack, over the full range of weights for economic costs DARMS has a larger utility
score than the Current approach. The same is true for a successful attack.
Figure 6. Sensitivity Analysis of Security Effectiveness
Utility
Percent of Weight on Security Effectiveness Objectives
0.886
0.455
0 100
DARMS, No Successful Attack
Current, No Successful Attack
DARMS, Successful Attack
Current, Successful Attack
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
33
Passenger Satisfaction. Figure 7 displays how the utility for each of the four alternative-
scenarios as the weight for passenger satisfaction changes. The vertical bar shows the proposed
weight for passenger satisfaction (16.1%). Comparing DARMS and the Current approach under
no successful attack DARMS has a larger utility than the Current approach until the weight
approaches about 60%. For a successful attack, DARMS has a larger utility until about 50%.
Figure 7. Sensitivity Analysis of Passenger Satisfaction
Utility
Percent of Weight on Passenger Satisfaction Objectives
0.771
0.459
0 100
DARMS, No Successful Attack
Current, No Successful Attack
DARMS, Successful Attack
Current, Successful Attack
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
34
Economic Costs. Figure 8 displays how the utility for each of the four alternative-
scenarios as the weight for economic costs changes. The vertical bar shows the proposed weight
for economic costs (13.8%). Comparing DARMS and the Current approach under no successful
attack, over the full range of weights for economic costs DARMS has at least as large a utility
score as the Current approach. However, for a successful attack, as the weight for economic costs
increases past about 30% the utility for DARMS is less than that for the Current approach.
Figure 8. Sensitivity Analysis of Economic Costs
Utility
Percent of Weight on Economic Costs Measure
1.000
0.270
0 100
DARMS, No Successful Attack
Current, No Successful Attack
DARMS, Successful Attack
Current, Successful Attack
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
35
Operational Efficiency. Figure 9 displays how the utility for each of the four alternative-
scenarios as the weight for operational efficiency changes. The vertical bar shows the proposed
weight for operational efficiency (11.5%). Notice that across the full range of weights DARMS
has a larger utility than the Current approach regardless of the attack outcome.
Figure 9. Sensitivity Analysis of Operational Efficiency
Utility
Percent of Weight on Operational Efficiency Objectives
0.752
0.460
0 100
DARMS, No Successful Attack
Current, No Successful Attack
DARMS, Successful Attack
Current, Successful Attack
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
36
Operational Costs. Figure 10 displays how the utility for each of the four alternative-
scenarios as the weight for operational costs changes. The vertical bar shows the proposed
weight for operational costs (18.4%). Comparing DARMS and the Current approach, DARMS
has a higher utility than the Current approach when weight exceeds 5% regardless of the
outcome of the attack.
Figure 10. Sensitivity Analysis of Operational Costs
TSO Job Satisfaction. Figure 11 displays how the utility for each of the four alternative-
scenarios as the weight for TSO job satisfaction changes. The vertical bar shows the proposed
weight for TSO job satisfaction (14.9%). Comparing DARMS and the Current approach under
no successful attack DARMS has a larger utility than the Current approach until the weight
approaches about 25%. For a successful attack, DARMS has a larger utility until about 30%.
Figure 11. Sensitivity Analysis of TSO Job Satisfaction
Utility
Percent of Weight on Operational Costs Measure
0.766
0.443
0 100
DARMS, No Successful Attack
Current, No Successful Attack
DARMS, Successful Attack
Current, Successful Attack
Utility
Percent of Weight on TSO Job Satisfaction Objectives
0.800
0.120
0 100
DARMS, No Successful Attack
Current, No Successful Attack
DARMS, Successful Attack
Current, Successful Attack
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
37
Aviation Industry Vitality. Figure 12 displays how the utility for each of the four
alternative-scenarios as the weight for aviation industry vitality changes. The vertical bar shows
the proposed weight for aviation industry vitality (14.9%). Comparing DARMS and the Current
approach under no successful attack DARMS has a larger utility than the Current approach until
the weight approaches about 80%. For a successful attack, DARMS has a larger utility until
about 30%.
Figure 12. Sensitivity Analysis of Aviation Industry Vitality
Modeling Uncertainty
Probability Trees
Currently the TSA uses Transportation Sector Security Risk Assessment TSSRA
vulnerability estimates generated by experts within the TSA and airline industry. Essentially
these experts were asked what the chance would be of an adversary successfully executing an
attack for each of a large number of scenarios. They assumed the attackers had clear intent and
capability. That is, the threat level was assumed to be virtually certain for a particular flight.
These global assessments provide a useful perspective, especially when done by separate groups
of experts as was done in the TSSRA assessments.
In addition to global assessments, it is often helpful to decompose probabilities into
component parts that are easier to assess, explain to stakeholders and perhaps control through
targeted countermeasures. This study develops a probability tree that decomposes the probability
of an attack on a commercial flight into the following components: 1) how the passenger is
classified into low risk (receives expedited screening) vs. unknown risk (receives standardized
screening) in the current approach or more risk classifications in the DARMS approach; 2) how
well the countermeasures (e.g., machines, canines, BDOs) detect the threat (i.e., signal or no
signal) and 3) can the adversary successfully carry out the attack from the sterile area .
Utility
Percent of Weight on Industry Vitality Objectives
0.950
0.464
0 100
DARMS, No Successful Attack
Current, No Successful Attack
DARMS, Successful Attack
Current, Successful Attack
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
38
Figure 13. Probability Tree for Current Approach
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
39
Figure 14. Probability Tree for DARMS Approach
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
40
As shown in Figures 13 and 14, preliminary judgments were used to parameterize the
potential improvements that DARMS might have over the Current approach. For purposes of
illustration a number of assumptions were made as follows: 1) Passenger Classification.
Passengers that might pose a flight threat are much less likely to be classified as low risk and
hence less likely to receive expedited screening than passengers posing no risk; 2)
Countermeasures Detection. Passengers posing a flight threat will have higher detection rates
than passengers posing no threat, regardless of risk classification. Also, passengers classified as
an unknown risk (receive standard screening) will have higher detection rates than those
passengers classified as low risk and 3) Attack Risk. Even if a passenger posing a threat enters a
sterile area the chance of completing a successful attack is still about 50% (e.g., shoe bomber
attempt, Christmas day bomber attempt).
Probability Tree: Current Approach Assuming Passengers Present a Threat
Figure 13 portrays the probability tree for the Current approach. The following
assumptions are made:
Assuming a passenger poses a flight threat, the chance of being classified as a low risk
passenger is 1% and for those passengers posing no threat it might be about 50% (not
shown).
For those passengers posing a flight threat and classified as a low risk the probability of
being not detected is 40% while those posing a flight threat but classified as an unknown
risk has a probability of being not detected of 20%.
Assuming a passenger poses a flight threat, if they enter a sterile area they have about a
50% chance of succeeding in the attack.
Instructive for this assessment, given a true threat to a flight, the probability of a successful
attack or the flight vulnerability is about 10.1%. This probability will be important when
considering system wide risk.
Probability Tree: DARMS Approach Assuming Passengers Present a Threat
Figure 14 portrays the probability for the DARMS approach. The same assumptions and
calculations apply here as with the Current approach except for two differences. First, it is
assumed that DARMS does 2% better in terms of detecting threats and thwarting attack attempts
by passengers who pose a flight threat and who enter a sterile area. Second, consistent with the
intent of DARMS, passengers are classified into lower risk and lowest risk groups. DARMS has
plans to divide passengers into four low risk groups but only two groups are used here for
illustration. Given a credible threat to a flight the chance of a successful attack here is 9.9%
(about 2% improvement over the Current approach).
Both the current and DARMS probability trees assume there is a threat. The same
approach could be used to consider the misclassification of no-threat individuals as dangerous
(i.e., false alarm). While no probabilities were assigned Figure 15 demonstrates the structure of a
probability tree that could be used to examine the false alarm problem.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
41
Figure 15. Probability Tree for the No Threat Branch
System Wide Risk
Calculating System Wide Risk. Using the probability trees for the Current and DARMS
approaches, simplifying assumptions are made to calculate the system wide probability of at least
one successful attack. Here are the assumptions:
For the same attack scenario (e.g., non-metallic body bomb), the vulnerability per flight
is similar across domestic flights in airports with sufficient resources and technology
(e.g., departure from JFK, National, O’Hare);
the probability of a successful attack in any given year is a function of flight vulnerability
and the number of credible threats per year.
Assuming Credible Threats Are Independent. To compute the probability of at least one
successful attack per year system wide, a binomial distribution was used in the following way:
1) Suppose n is the number of credible threats per year;
2) p is a common flight vulnerability, that is, the probability of an adversary defeating a
flight’s countermeasures;
3) i is the probability of nth
credible threat per year where i = 0,1, 2 …n; and
4) X is the number of successful domestic attacks per year.
Assume that credible threats are independent of each other. The probability that there are no
successful attacks in a given year is expressed by the binomial formula in equation (2):
P(X=0| n,p) = (n!/x!(n-x)!) px (1-p)
n-x . (2)
For X=0 this reduces to (1-p)n. Hence, the probability of at least one successful attack for a given
number of credible threats n is expressed by equation (3):
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
42
P(X1|n,p) = 1- (1-p)n (3)
Now relaxing the assumption that the number of credible threat per year n is known with
certainty, assume n can be represented by a probability distribution characterized by I where
i=1. Then the probability of at least one successful attack in a given year is expressed by
equation (4):
P(X1|n,p) ={i (1-(1-p)i ) for nx; 0 otherwise; i=0,1,2 …n. (4)
This is a recursive formula that computes the probability for a least one successful attack for up
to n credible threats. For example, the flight vulnerability for the Current is approach is assessed
to be .101 from above (10.1%). Investigating the case for up to n=2 credible threats per year,
suppose it is believed that the chances of zero threats, one threat or two threats are 10%, 40% and
50% respectively or 0= .1, 1=.4 and 2=.5. Then the probability of at least one successful attack
in a given year is expressed by equation (5):
P(X1|n,p) = 0(1-(1-p)0
+ 1(1-(1-p)1 + 2(1-(1-p)
2 = 0+.4(1-(1-.101)+ .5(1-(1-.101)
2 = .136 or a
13.6% . (5)
Using the assessed vulnerability under DARMS probability this would be about .134 or 13.4%.
Allowing Credible Threats to be Correlated. This case relaxes the assumption that the
credible threats are independent, and reconsiders the same case but where credible threats are
allowed to correlate. Drawing upon the recursive function from a generalized binomial
distribution (Drezner & Farnum, 1993) the chance of no successful attacks for a given number of
credible threats n is expressed in equation (6) where is the correlation between credible threats
and ranges from –1 to 1 and i is the number of credible threats up to n:
P(X=0)|n,p,) = (1-p)(p+(1-p))i-1
(6)
The chance of at least one successful attack in a given year is expressed by equation (7):
P(X1|n,p, ) = {i {1-(1-p)(p+(1-p))i-1
} for nx; 0 otherwise; i=1,2 …n. (7)
Notice for the case where =0 this probability function reduces to the probability expression
under the assumption of independence. For =1, the probability function reduces to p the flight
vulnerability which assumes one credible threat. Hence, with perfect positive correlation the
probability of a successful attack does not change as a function of actual number of threats. It is
as though they all operate as one threat. For all >0, P(X1|n,p, ) < P(X1|n,p). That is, if
credible threats are positively correlated the chance of at least one successful attack is bounded
above by the probability under the assumption of independence and bounded below by p under
the assumption of perfect positive correlation.. The reverse is true for <0 then P(X1|n,p, ) >
P(X1|n,p). Consider the same case as above but credible threats are positively correlated with
=.5. Now P(X1|n,p, ) = .113 or 11.3% down from the 13.6% estimate given above under the
assumption of independence. The corresponding DARMS calculations would be .112 or 11.2%.
Summary of Probability Calculations. As a way to explore how flight vulnerability
could be used to provide insight on system wide risk to domestic commercial aviation this study
began with a few simplifying assumptions: 1) Under the Current approach flight vulnerability is
similar across flights for major domestic airports. This can be determined by expert assessment
as was done in the TSSRA report or in conjunction with a probability decomposition approach as
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
43
was illustrated in this report; 2) The probability of at least one successful attack in a given year
depends critically on the number of credible threats per year. This information rests largely in the
hands of the intelligence community. However, the probability distribution of threats can be
incorporated into broader probability calculations and this study simply described the uncertainty
about the number of credible threats as an assessed discrete probability i. Notice, that if i =1
for any value of number of threats n then the probability expression reduces to a standard
binomial calculation for fixed number of n. Also note, there are several ways to represent this
uncertainty including assuming credible threats follow a Poisson distribution as illustrated below
in the report’s simulation results; 3) The study began with the assumption that credible threats
are independent, as is required by a standard binomial distribution, and calculations of system
wide probabilities of a successful attack were computed. Then the assumption of independence
was relaxed and a recursive expression was derived to calculate the probability of at least one
successful attack assuming credible threats are correlated. Note, it appears reasonable to think
that credible threats are most likely positively correlated. With this assumption, the probability
expressions derived in this study suggest that the probability of a successful attack will be
bounded by the above calculations assuming independence and 5) Under the DARMS approach
it is not yet clear whether flight vulnerability will be similar across flights in the way the Current
approach is. This will depend on how flight risk is adjusted system wide as a matter of policy.
It’s difficult to imagine however, that countermeasures flight-by-flight will differ so markedly
that flight vulnerability varies significantly. If vulnerability across flights does differ then the
mathematical tractability of the above expressions becomes challenging. This heterogeneity
across flights can be addressed by relaxing the assumption of common vulnerability estimate p
and correlation coefficient but this would require the aid of computing algorithms to sum
across a very large number of flights.
Assuming a Fixed Number of Credible Threats and Independence Across Threats.
This study derived a probability expression for the probability of a successful attack that
incorporates uncertainty about the distribution of threats and correlation across threats. However,
this study does not estimate values for or . Hence, for purposes of illustration it is assumed
that on average there will be 2 credible threats per year. This assumption, along with the
vulnerability estimates, suggest the chance of at least one successful attack per year is about
19.2% and 18.8% for the Current and DARMS approaches respectively. This assumes with
certainty that the number of credible threats per year will be two that is 2=1 for n=2.
Simulating the Probability of a Successful Attack. To demonstrate variation about
probability estimates Monte Carlo simulations were done. Using the vulnerability calculations
from the probability tree described above and assuming credible threats follow a Poisson
distribution the probability of at least one successful attack per year was simulated. Means of 2,
10 and 25 credible threats per year were examined. Figures 16–18 show the simulated
distributions respectively. The average probabilities of a successful attack in these distributions
were 18%, 64% and 92% but of course there is considerable variation about these means and that
is the point.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
44
Figure 16. Simulation of Probability of at Least One Successful Attack with an Average of
2 Credible Threats per Year
Figure 17. Simulation of Probability of At Least One Successful Attack with an Average of
10 Credible Threats per Year
Figure 18. Simulation of Probability of At Least One Successful Attack with an Average of
25 Credible Threats per Year
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
45
Figure 19 displays the cumulative probability of at least one successful attack per year as
a function of number of credible threats and the correlation between threats under the Current
approach. These calculations are deterministic, that is, they assume the number of threats in a
given year is known. This graph demonstrates how quickly the probability of a successful attack
can increase as a function of number of credible threats per year. Notice that assuming
independence past 22 credible threats the probability of at least one successful attack in a year is
over 90%. At 40 credible threats the probability is about 99%. With a correlation of .5 these
probabilities would be about 70% and 88% respectively. This graph suggests that if adversaries
were to launch a large number separate attacks within minutes of each other, there is a sizable
and sobering chance they would succeed. This result speaks to the critical role of actionable
threat intelligence and deterrence. The best policy is to reduce the motivation to commit such
acts and to prevent those who are determined from reaching the domestic airports.
Figure 19. Probability of At Least One Successful Attack As a Function of Credible
Threats and Correlation Between Threats
Expected MAU Comparison of DARMS and Current Approaches
Figure 20 shows a probability tree of the comparison of Current and DARMS approaches
from a system wide perspective. The tree has been reduced to the four alternative-scenarios or
end nodes. The chance of at least one successful attack per year was computed for 2 credible
threats in a given year; these probabilities are 19.2% and 18.8% respectively. From Table 8
above, the multi-attribute utilities for the Current and DARMS approaches for an unsuccessful
attack are .726 and .749 respectively. Likewise, multi-attribute utilities for a successful attack are
.471 and .493. Thus, DARMS is preferred under either scenario. Aggregating over the two
scenarios, the expected utility for the Current approach is .677 and for DARMS is .701. In this
assessment, considering preferences and uncertainties, DARMS is the more attractive alternative.
0.000
0.200
0.400
0.600
0.800
1.000
0 20 40 60
Pro
ba
bil
ity
Credible Threats Per Year
Probability of Successful Attack
Independence
Correlation=.5
Correlation=.7
Correlation= 1
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
46
Figure 20. Decision Tree of Current and DARMS Approaches
Adversary MAU
Defining Objectives
Adversary objectives and attribute scales are identified in the same manner as for the TSA
described above. These are informed by Keeney and von Winterfeldt (2010) and informal
conversations with colleagues at the TSA. The objectives hierarchy is assumed to be
considerably simpler than for the TSA. Definitions of the objectives and attributes are shown in
Table 9, and scales and ranges are shown in Table 10. In this assessment there are two
objectives: Growth of the Terrorist Organization and Military Outcomes. In terms of former, the
intent is to maximize status, financial resources and support among the Muslim community. This
objective has four attribute scales:
1) status as formidable force in the middle east (1–3 constructed scale);
2) financial contributions ($0–$1M per year);
3) new recruits to attack U.S. aviation (0–120 per year) and
4) winning the hearts and minds of Muslims (1–3 constructed scale).
In terms of the latter, the intent is to maximize harm to the United States. by inflicting casualties
and economic losses as well as driving up aviation security costs and fear among the American
public. This objective has six attribute scales:
1) attacks on U.S. airports and airlines (0–5 per year);
2) U.S. fatalities (0–400 per year);
3) U.S. injuries (0–400 per year);
4) economic costs ($0–$100B per year);
5) operational costs ($0–$365M per year) and
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
47
6) and fear among U.S. public (0–60M per year).
Table 9. Definitions of Adversary Objectives and Attributes
Objectives/Attributes Definitions
Growth of Terrorist Organization
Maximize the status, financial resources, recruiting environment and number of Muslims in support of the organization goals and activities.
Status as formidable force in Middle East
The extent to which the organization is perceived in the Muslim world as capable of carrying out significant attacks on U.S. targets.
Financial contributions The number of dollars given to support organization in anticipation of or directly following an attack on a U.S. target.
New recruits to attack U S airports and
airlines
The number of new recruits willing to support or carry out organizational objectives in anticipation of or directly following an attack on a U.S. target.
Hearts and minds of Muslims
The number of Muslims who have favorable impression of organization’s goals and activities in anticipation of or directly following an attack on a U.S. target.
Military Outcomes Maximize the harm to the U.S. by inflicting casualties, economic consequences, increased aviation security operating costs and fear among American population.
Attacks on U.S. airports and airlines
The number of successful attacks per on a U.S. airport or airline.
American Fatalities Lives lost as direct result of an attack
American Injuries Injuries incurred as direct result of an attack
Economic Costs The sum of all economic consequences in dollars over a one-year time horizon due to attacks on U.S. airports and airline.
Operational Costs for aviation security
The sum of all costs to the TSA in dollars over a one-year time horizon due to operations relating to security effectiveness, passenger satisfaction and regulatory compliance (e.g., FTE, Equipment, FAMS, Pre-screening). Not included are the costs of responding to significant security breaches that are covered under Economic Costs.
Fear among Americans
The number of fearful Americans as direct result of an attack.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
48
Table 10. Adversary Objectives, Attributes, Scales and DARMS Impacts
Objectives1 Metrics
Scale Units Predictions for DARMS Impact
Growth of Terrorist Organization
Status as formidable force in middle east
Change in status/year
2
1–3 If successful attack will increase under DARMS-more significant defeat of U.S. security.
Financial contributions
Dollars/year $0–$1M If successful attack will increase under DARMS-more significant defeat of U.S. security.
New recruits to attack U.S. airports and airlines
New Recruits/year3 0–120 If successful attack will increase under DARMS-more
significant defeat of U.S. security.
Hearts and minds of Muslims
Change in number of Muslims/year
2
1–3 If successful attack will increase under DARMS-more significant defeat of U.S. security.
Military Outcomes
Successful Attacks on U.S. airports and airlines
Number of targets/year
4
0–5 Decrease under DARMS due to Game Theory countermeasures.
American Fatalities
Number of Americans/year
0–400 Possibly less number under DARMS-not sure.
American Injuries
Number of Americans/year
0–400 Possibly less number under DARMS-not sure.
Economic Costs Dollars/year $0–$100B
If successful attack will increase under DARMS due to increased public reaction.
Impact on Operational Costs for Aviation Security
Dollars/year5 $0–
$365M Minimal or no perceived impact of DARMS for adversary.
Fear among Americans
Number of fearful people/year
6
0–60M If successful attack will increase under DARMS-more significant defeat of U.S. security.
1 Based largely on the Keeney and von Winterfeldt (2010) paper.
2 Scale based on a change in status from current level. The scale is 1 for decrease in status, 2 for no
change in status and 3 for increase in status. 3 There were about 1.6B Muslims worldwide in 2010 according to a Pew study
http://www.pewresearch.org/fact-tank/2013/06/07/worlds-muslim-population-more-widespread-than-you-might-think/. Conservatively, at least seventy five percent are 18 years or older or 1.2B. Let’s suppose that following a successful attack at most one person in a million of this adult population is motivated to
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
49
carry out the goals of the organization or 1200. Suppose further ten percent of these have willingness and capability of attacking U.S. airports and airlines or 120. 4 Suppose out of the possible 90 security breaches inside the sterile areas of U.S. airports per year two
percent are significant or about 1. Likewise, suppose out of the possible 5 security breaches inside the aircraft cabin per year fifty percent are significant or about 3. Combined this would total 5. 5 This is the combined checkpoint reconfiguration and operational costs. In 2013 the aviation budget was
approximately $5.3B. Also, in Table 2 aviation reconfiguration and loss of operating revenue costs combine to range from $0–$2B. Together these costs would add to $7.3B. Suppose terrorist organizations with common goals are able to influence these costs by up to 5%/year then the costs would range from $0 to $365M per year. 6 According to the U.S. Census, there were approximately 235M U.S. residents 18 years and older. Let
suppose a successful attack caused 25% of this population to have a high level of fear for at least one month following the attack or almost 60M people. Data taken from U.S. Census: http://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?src=bkmk.
Table 11 displays the assessed consequences for adversary attributes for each of the four
alternative-scenarios. Consequences for status in the middle-east, financial contributions,
recruiting success and winning the hearts and minds of Muslims were based on reasonable
speculation. For example, unsuccessful attacks are predicted to result in minimal positive
consequences from the adversary’s perspective and were treated the same for the Current and
DARMS approaches. Also, consequences for a successful attack are predicted to be larger under
a DARMS approach because it would represent a more substantial defeat of U.S. security efforts.
Consequences for fatalities, injuries, and economic costs are predicted to be about the same as
the assessment from the TSA perspective given above. Successful attacks on airports and airlines
are treated as a combination of significant security breaches inside airport sterile areas and
aircraft cabins. As a simplification, the assumption is made that an adversary would most care
about a breach inside the cabin of commercial aircraft and so were treated the same as a cabin
breach under the TSA perspective. Fewer breaches per year are predicted under the DARMS
approach. Impact on security operational costs is predicted to be minimal with no successful
attacks. With a successful attack it is assumed that the adversary might hope for a five percent
increase in security costs to airports and airlines and this was predicted to be the same under the
Current and DARMS approaches. Lastly, under a successful attack the number of fearful
Americans is taken to be about twenty five percent of the adult population. Because a successful
attacks under DARMS might represent a more substantial defeat, fear is predicted to be greater
than under the Current approach.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
50
Table 11. Adversary Consequences for Alternative-Outcomes Contingencies
Objectives Attributes
Current Unsuccessful
Attack
Current Successful
Attack
DARMS Unsuccessful
Attack
DARMS Successful
Attack
Growth of Terrorist Organization
Status 1 2 1 3
Financial Contributions $M
$0 $.75M $0 $1M
New Recruits 0 90 0 120
Hearts and Minds 1 2 1 3
Military Outcomes
Successful Attacks-
U.S. airports and airlines
2.5 2.5 1.25 1.25
Fatalities 0 184.25 0 184.25
Injuries 0 184.25 0 184.25
Economic Cost $B $0B $65.75B $0B $72.98B
Operational Cost $M
$0M $365M $0M $365M
Fear among Americans M
0M 50M 0M 60M
Adversary Utilities
Attribute weights and utilities are displayed in Table 12. First, the weights for adversary
attributes are assessed as they were for the TSA perspective. The most weight is given to success
in recruiting people to attack U.S targets and inflicting economic costs on the United States. Far
less weight is given to impacting operational costs than under the TSA’s perspective. Computing
the multi-attribute utility for each of the four alternative-scenarios is computed as before. The
MAU calculations for the Current approach with no successful attack is .059 and for the
DARMS approach is .029. From the adversary’s perspective the Current approach is more
attractive because of the prediction that a larger number of security breaches would occur with
this security approach. However, the reverse is true under a successful attack; the Current
approach has a utility of .630 and the DARMS approach has a utility of .794. This occurs largely
because a successful attack would represent a more substantial defeat of U.S. security efforts and
the resultant consequences would be considered to be decidedly more favorable from an
adversary’s perspective. Additionally, operational costs are given less weight here than under the
TSA perspective above. Hence, DARMS is less favored on this attribute. These results suggest
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
51
the need to consider treating the defender and adversary utility functions, not as a zero-sum
game, but rather as different preference functions.
Table 12. Adversary Weights and Consequence Utilities
Objectives
Attributes
Attribute
Weight
%
Current
Unsuccessful
Attack
Current
Successful
Attack
DARMS
Unsuccessful
Attack
DARMS
Successful
Attack
Growth of Terrorist
Organization
Status 11.5 .000 .500 .000 1.00
Financial
Contributions $M
8.6 .000 .750 .000 1.00
New Recruits 14.3 .000 .750 .000 1.00
Hearts and Minds 10.0 .000 .500 .000 1.00
Military Outcomes
Successful Attacks-
U.S. airports and
airlines
11.7 .500 .500
.250 .250
Fatalities 8.8 .000 .461 .000 .461
Injuries 5.9 .000 .461 .000 .461
Economic Cost $B 14.6 .000 .658 .000 .73
Operational Cost $M 4.4 .000 1.00 .000 1.00
Fear among Americans M 10.2 .000 .833 .000 1.00
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
52
Implementing DARMS: Challenges
From this research, two significant challenges were identified that need to be considered
(and perhaps further studied) as a DARMS-approach moves from conceptual pilot modeling to
future planned implementation:
1) the homogeneity of flights and
2) the current airport operational limitations.
Heterogeneity versus Homogeneity
A DARMS-approach is better than the current approach when flights are more
heterogeneous. If flights are homogeneous (no different values/passengers for different flights),
then the current approach and a future DARMS approach from a game-theory perspective are
equivalent in terms of impact to security. The challenge is that on most days within a single
airport, flights may be largely homogeneous. Table 13 details characteristics of flights
originating from U.S. airports, specifically describing how the flight characteristic can influence
either the probability of a successful attack from the terrorist’s perspective or the consequence of
a successful attack. Additionally, some characteristics of the flight will not be knowable to the
adversary in advance (such as FFDO or FAM on-board) and some may be only knowable to the
TSA with very short notice. These characteristics are labeled in Table 13.
The differences among flights from one airport are: the destination city, the aircraft type
(correlated with destination city), the airline, the departure gate, the time of day, and the travelers
and crew on-board the flight. Some airports have multiple TSA screening areas, so in some
airports that would also be a difference. Finally, cargo could also create differences but was not
considered in this phase of the study.
Based on current perceptions of terrorist values, it seems unlikely that one airline would
be targeted versus another (i.e., Delta versus United). Additionally, departure gate is only
relevant for those airports that have multiple TSA screening areas, where passengers may receive
different screening services. Neither of these factors will create heterogeneity among flights.
The destination city and aircraft type will vary by flight but in a common pattern every
day (i.e., American Airlines uses a 737 aircraft for the daily 6:30pm IAD-LAX flight). So
screening algorithms that focus resources on larger airplanes flying to higher “valued”
destinations would still be fairly constant day-to-day.
The remaining significant difference among flights is the travelers and crew on-board.
Some will increase the risk (i.e., selectees) and some will decrease the risk (i.e., FAM, FFDO,
and armed federal and local law enforcement officers). Selectees already receive more screening,
but algorithms could increase screening for others on flights with selectees.
Because the travelers are the major significant difference for a flight, more research is needed to
determine how passengers can be further categorized into a finer gradient of risk, since this is
where a DARMS-approach can have value above the current screening methods.
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
53
Table 13. Differences Across Flights
Implication to probability of success from terrorist perspective
Implication to consequences
Knowable to adversary in advance
Knowable to TSA in advance
Other notes
Origin City Yes if smaller airports have lesser security, or overseas originated
Not separate from aircraft type (i.e., if airport only has commuter flights) unless symbolic value
Yes Yes Constant for each airport being considered
Destination City No Yes if some cities are higher value target
Yes Yes
Aircraft Type (# of passengers)
No Yes – bigger number of passengers
Yes but imperfectly – last minute aircraft substitutions
Yes Correlated with origin – destination cities, bigger planes on hub-to-hub and coast-to-coast flights
Airline No No – assuming no one airline is higher value target
Yes Yes
Passengers in different risk categories (connecting vs original, traveling military)
Yes – some passengers increase risk (connecting, high risk), some passengers lower risk – military personnel
No – unless some elaborate assassination attempt
Only those within group
Yes
Armed passengers (fed and local law enforcement)
Yes – reduce risk
No No Yes but short notice
FFDO (armed flight crew)
Yes – reduce risk
No No Yes but may be short notice
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
54
Implication to probability of success from terrorist perspective
Implication to consequences
Knowable to adversary in advance
Knowable to TSA in advance
Other notes
FAM Yes – reduce risk
No No Yes
Time of Day Yes – if more or less crowded security affects screening
No Yes Yes
Departure Gate/Concourse
No (not separate from different screening)
No Yes – may change
Yes – may change
Correlated with airline
Different screening area (if airport has different areas)
Depends on screening
No Yes – may change
Yes – may change things
Would be airport specific, DCA has separate areas by concourse
Current Operational Limitations
A member of the research team visited BWI airport and examined the separate TSA
screening checkpoints (A, B, C, and D) guided by Todd Trafford (Deputy Federal Security
Director, BWI Airport) and Kristen Best (Operations Improvement Branch, Office of Security
Operations). A significant challenge for implementing a future DARMS-approach will be
maintaining positive passenger control between the TDC (Travel Document Checker) and the
screening machines. If machines are intended to screen passengers based on an algorithmically
determined risk level, then when the passenger places their baggage on the X-ray and then either
pass through the AIT or the walk-through metal detectors, the screening devices need to know
the appropriately risk level for each individual and adjust based on risk.
As currently configured at BWI, pier D would need to be completely reconstructed. Pier
D currently has no queue area. The line wraps down the hallway. On the day of our visit, they
had capacity for 7 lanes but are losing 2 shortly for a few years because of airport remodeling to
create a concourse D/E connector. Once passengers pass the TDC, they enter a huge crowd of
travelers waiting to be sorted into lanes to approach the belt. This area appeared much like a
“mosh pit” at a pop concert. Positive passenger control would be close to impossible with the
current design unless passengers re-identified themselves at the screening machines.
On the other hand though, pier C is the newest and most efficient check point of the four
at BWI. This checkpoint has 9 lanes (they were operating 7 at our visit). Pier C has a great deal
of queueing space and correspondingly very little waiting between the TDC and X-ray belts and
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
55
passenger screening machines. Pier C could more easily be able to do some degree of positive
passenger control.
Finally, at pier A, the TSA is testing a CAT (credential authentication terminal). The
CAT is capable of providing the TDC with much more information about the passenger after
scanning their identification. Completing the piloting of such technology and deploying it in the
field will be a critical step in implementing a DARMS-approach because TDC will potentially
need more information that can be discretely encoded on a boarding pass to determine approach
screening based on risk.
Conclusions
Summary
TSA Objectives and Attributes. Seven fundamental objectives are identified based on
discussions with colleagues at the TSA and a brief review of the security literature. These were:
1) security effectiveness; 3) passenger satisfaction, 4) economic costs of a significant security
breach; 4) operational efficiency; 5) TSO job satisfaction; 6) operational costs and 7) aviation
industry vitality. Twenty eight performance measures were proposed for consideration and
twenty four were used in the MAU analysis. Likewise, attribute scales are developed and
consequence ranges are assessed for each. Objective and attribute weights are assessed from one
member of the research team as an illustration. The most weight is given to attributes such as
operational costs, economic costs of security breaches and measures of security effectiveness
such as fatalities and injuries.
Consequences and Utilities. For contingency successful attack and an unsuccessful
attack scenario, consequences were assessed for the Current approach and DARMS using
suggestions from colleagues at the TSA and discussions among the research team members.
DARMS fared better on all measures of security effectiveness and almost all measures of
operational efficiency and operational costs regardless of the outcome of significant security
breaches. The Current approach fared better on variation of passenger wait time and perceptions
of fairness and security, TSO resources (FTE), all measures of TSO job satisfaction and
measures of aviation industry vitality regardless of the outcome of significant security breaches.
Focusing on economic costs favors the Current approach should there be significant security
breach. Overall, regardless of whether a significant security breach occurs, DARMS had a higher
utility.
Calculating System Wide Probability of a Successful Attack. It is assumed that the
cumulative probability of a successful attack would follow a binomial distribution and be a
function of flight vulnerability to attack and the number of credible threats per year system wide.
As an illustration, assuming on average two credible threats, the chance of at least one successful
attack per year under the Current approach is 19.2%. Note, if both the uncertainty surrounding
number of credible threats and the possible correlation among these threats are considered than
this estimate would be smaller. On the other hand, assuming independent threats and certainty
Figure 19 above graphically portrays how the chance of a successful attack increases as a
function the number of credible threats. Past twenty two threats per year the upper bound chance
of at least one successful attack climbs to over 90%.
Adversary Objectives, Attributes and Utilities. Two objectives are identified for an
adversary: 1) growth of the terrorist organization and 2) military outcomes. Ten performance
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
56
attributes are used in the MAU analysis. Likewise, attribute scales were developed and
consequence ranges were assessed for each. Objective and attribute weights are assessed from
the perspective of one of the research team as an illustration. The most weight is given to
attributes such as success in recruiting members to attack U.S. targets and inflicting economic
costs on the United States. When comparing the Current and DARMS approaches, it is noted
with an unsuccessful attack the Current approach was more attractive, while with a successful
attack the DARMS approach is more attractive. This difference is based on a prediction that
defeating DARMS would represent a more significant victory for an adversary than the defeating
the Current approach. It also suggests the need to represent the utility function of the TSA and
adversary in a game theory context as distinct preference functions.
Caveats
This study was undertaken as a proof of concept, that DARMS could be compared to the
Current approach on a number of dimensions important to the TSA. The comparison was done at
a strategic level not tactical level. The analysis provided is not meant to provide definitive
conclusions about DARMS or the probability of a successful attack on the domestic commercial
aviation. Instead it is intended to describe a general approach to studying these issues and layout
the kinds of information, assessments and challenges that would be involved in a more complete
analysis. There were a number of simplifying assumptions that were made that need to be
addressed in future analyses.
Representing Outcomes and Consequences. The probability trees decompose a flight’s
vulnerability to a successful attack into meaningful components. This decomposition is intended
to articulate a number of uncertainties that might be useful in exploring threat levels, and
classification and detection (or false alarm) rates. The assessed probabilities are primarily
intended as an illustration, but the calculated vulnerability is quite similar to that provided by
TSA and industry experts in other contexts.
The probability trees represent passengers posing a flight threat and attack outcomes are
defined as unsuccessful attack and successful attack. This representation presumes there is some
level of threat to a flight, that is, an adversary with motivation and capability had intended to
defeat the security system connected with a particular flight. Because this study does not specify
the criminal actions, it is left undefined as to what an unsuccessful attack means apart from the
system was not defeated. In future analyses it would be important to carefully define such actions
because that would make for a more meaningful assessment of outcome consequences. For
example, even a passenger discovered outside the passenger security area with a bomb would
generate significant public response and produce economic costs. Additionally, a successful
attack is treated in the aggregate but in fact has a number of consequence levels. For example,
there are different consequences for a significant security breach inside the airport, a breach
inside the aircraft cabin and the complete loss of an aircraft.
In this study, the assessment of outcome consequences makes no distinction between no
attack and unsuccessful attack. In actuality there might be a large difference between these two,
and this should be addressed carefully in future work.
Representing Attribute Weights and Utilities. Weights were assessed by one of the
research team members with careful attention to consequence ranges and conversations with
colleagues at the TSA. However, this is not a good substitute or proxy for a careful elicitation of
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
57
the many stakeholders connected with comparing the Current and DARMS approaches.
Likewise, this study assumed a risk neutral attitude for all measures. In future work this
assumption should be examined. For example, it might well be that for many attributes the TSA
is risk neutral but there may be other attributes for which the TSA is risk averse or even risk
seeking. Finally, the study also assumes the multi-attribute utility functions are additive. That is,
preferences on individual attribute utilities are independent of levels on other attributes. In
practice, this assumption is often made as a simplifying approximation but in future work this
should be examined carefully.
Representing Probabilities. To illustrate how to calculate the system wide chances of at
least one successful attack on a commercial airline per year three assumptions are made: 1) the
flight vulnerability to attack is uniform across a wide range of flights in airports large enough to
have sufficient resources and technology (e.g., assumption might not apply to flights between
small cities like Boise and Cedar Rapids with less access to technology); 2) the number of
credible threats per year can be assessed (e.g., i) or perhaps follow a Poisson distribution and 3)
the system wide probability was calculated using a binomial distribution using estimates of
vulnerability and number of credible threats. The binomial distribution assumes credible threats
are independent. That is, the presence of one credible threat does not influence the likelihood of
witnessing other threats in a particular year. It is not clear this assumption would hold. For
example, several criminal cells could be coordinating attacks. This study addressed this
assumption by deriving a probability expression that allows for correlated threats by drawing
from the work a generalized binomial distribution (Drezner & Farnum, 1993). In this study, the
probability calculations primarily serve to illustrate an approach to calculating the system wide
probability of a successful attack and to demonstrate this probability is highly sensitive to the
number of credible threats in a given year. This speaks to the importance of deterrence.
Future Research
This study has outlined the basic types of information and analyses that could be helpful
in examining how DARMS compares with the Current approach and other alternative aviation
security approaches as articulated for example by the Deloitte team working inside the Office of
the Chief Risk Officer. What is needed in future work is to follow up on this foundation as
follows:
Fundamental Objectives and Performance Attributes. The most important task
moving forward is to achieve agreement and buy in throughout the TSA about the objectives and
performance metrics to be used to guide strategic decisions about DARMS and a number of
other related projects. Taking the list of objectives and attribute measures described here as a
starting point, stakeholders throughout the TSA and within the aviation industry should be
involved in a discussion about the relevance and completeness of the proposed list. On reflection,
additional performance attributes may be suggested and others discarded as not useful. Likewise,
it would be helpful to get greater clarity and agreement on attribute definitions. Getting
consensus on definitions is critical and also very difficult. During discussions with TSA
colleagues it was apparent that opinions differed on issues as fundamental as how to measure
security effectiveness.
With an agreed upon list of objectives and attributes the next step is to further develop the
scales used to measures these attributes. In this study some scales were natural such as number of
fatalities during a significant security breach and customer wait time in minutes. As such the
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
58
consequence ranges were straightforward to think about. Other scales such as operational costs
and economic costs were in dollars. The consequences here were also straightforward but being a
composite of several factors more analysis is needed to appropriately represent the ranges. For
example, how will indirect economic consequences from public reaction be computed? Still
other scales such as passenger perceptions of security and TSO morale were constructed and may
need to be calibrated with other behavioral measures to adequately understand consequence
ranges. The CREATE risk perception and risk communication team has done some survey work
on passenger perceptions that might be helpful in this regard.
Once consequence scales and ranges are determined, key stakeholders can be queried
regarding the relative weights given each attribute. In this study a member of the research team
used swing weighting to assess attribute weights and assumed independence of attributes.
However, a more through elicitation procedure should be used involving stakeholders at the
TSA. Preference dependencies should be examined to determine the actual structure of the multi-
attribute utility function.
Uncertainty and Outcomes. In this study uncertainty has been represented largely in
terms of probability trees that account for classification rates, detection (false alarm) rates and
ability to execute an attack once in a sterile area of an airport or aircraft. This offers a reasonable
approach to decomposing flight vulnerability into understandable and perhaps manageable
components. This procedure should be pursued with more depth with a more careful assessment
of branch probabilities generated by subject matter experts inside the TSA. Also, detection rates
were thought about very broadly here but the set of countermeasures that would materially
influence the detection rates for a particular type of threat (e.g., non-metallic explosive carried on
body) should be specifically described.
The definition of a successful attack should be developed with more fidelity than was
done in this study. While it is true that any significant criminal activity inside a airport sterile
area or aircraft could be construed as a successful attack these outcomes need to be defined more
carefully. For example, attacks carried out in the airport may have different consequences than
an equivalent attack inside the aircraft while in flight. What about near misses in which security
officials skillfully thwart an attack in progress? Also, do attempted attacks that were
unsuccessful produce the same consequences as no attempted attacks? Again, this may be
context dependent. An attempted attack that is decisively and skillfully thwarted may have
temporary impacts in terms of economic and operational costs but have long term benefits in
terms of deterrence. To properly understand the implications of different security protocols these
issues need to be investigated.
In this study consequences for the four alternative scenarios were assessed by the
research team using three factors as a guide: 1) the attribute consequence range, often a
midpoint; 2) unsuccessful versus successful attack and 3) Current versus DARMS approach.
Experts within the TSA should be queried about these consequences and a more nuanced
conversation should take place about the predicted efficacy of the DARMS approach over the
Current approach on certain critical attributes.
System wide probability of a successful attack against commercial aviation was
examined with a number of simplifying assumptions. These were preliminary calculations to
identify and better understand the possible factors driving this probability. Binomial distributions
were used assuming independent and correlated credible threats. As a start this analysis was
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
59
useful because it demonstrates how sensitive the chance of a successful attack is to number of
credible threats per year. This observation points to the importance of properly assessing the
distribution of threats.
Adversary Objectives and Attributes. The adversaries represented here were thought to
be religious extremists, organized and possess sufficient resources to be credible threats. Other
types of adversaries (e.g., lone wolf) would have different priorities and this should be
investigated. The objectives and attributes were identified and scaled in much the same manner
as for the TSA. Some attributes such as fatalities and number of new recruits had natural scales
while others such as status in the Middle East had constructed scales. As mentioned, constructed
scales need to be calibrated to better understand what impact the consequences might have. For
these, it might be useful to go to the literature on motivations behind terrorism for insight.
References
Burns, W. J., & Dillon-Merrill, R. (2013a). Public perception of TSA security checkpoint
procedures and safety: Results from a nationwide survey. Presentation at Transportation
Security Administration Headquarters on August 1, Pentagon City, Arlington County,
Virginia.
Burns, W. J., & Dillon-Merrill, R. (2013b). Public perception of TSA security checkpoint
procedures and safety: Results from two recent and independent nationwide surveys.
Presentation at Transportation Security Administration Headquarters on December 12,
Pentagon City, Arlington County, Virginia.
Burns, W. J., Slovic, P., John, R., Rosoff, H., Rose, A., Avetisyan, M., Chan,O. & Sellnow, T.
(2013). Public risk perception surveys: Risk perception, public confidence and economic
impacts of risk amplification following a terrorist event. Report delivered to
Transportation Security Administration on June 30, Pentagon City, Arlington County,
Virginia.
Clemen, R. T., & Reilly, T. (2001). Making hard decisions with decision tools. Pacific Grove,
CA: Duxbury.
Deloitte Team. (2015). DARMS Data Call Overview. Powerpoint slides describing DARMS
business case, January 21, 2015.
Drezner, Z., & Farnum, N. (1993). A generalized binomial distribution. Communications in
Statistics: Theory and Methods, 22, 3051–3063.
http://dx.doi.org/10.1080/03610929308831202
Eisenfuhr, F., Weber, M., & Langer, T. (2010). Rational decision making. Berlin, DE: Springer–
Verlag.
Giesecke , J. A., Burns, W. J., Barrett, A., Bayrak, E., Rose, A., Slovic, P., & Suher, M. (2012).
Assessment of the regional economic impacts of catastrophic events: CGE analysis of
resource loss and behavioral effects of a RDD attack scenario. Risk Analysis, 32, 583–
600. http://dx.doi.org/10.1111/j.1539-6924.2010.01567.x
Giesecke, J., Burns, W. J., Rose, A., Barrett, A., & Griffith, M. (2014). Regional dynamics under
adverse physical and behavioral shocks: The economic consequences of a chlorine
DARMS: Examining the Role of Multi-Attribute Utility (DRAFT)
60
terrorist attack in the Los Angeles financial district. In P. Nijkamp, A. Rose, & K. Kourtit
(Eds.), Regional science matters: Studies dedicated to Walter Isard. Berlin, DE:
Springer–Verlag. http://dx.doi.org/10.1007/978-3-319-07305-7_16
Keeney, G. L., & von Winterfeldt, D. (2010). Identifying and structuring the objectives of
terrorists. Risk Analysis, 30, 1803–1816. http://dx.doi.org/10.1111/j.1539-
6924.2010.01472.x
Keeney, R. L., & von Winterfeldt, D. (2011). A value model for evaluating homeland security
decisions. Risk Analysis, 31, 1470–1487. http://dx.doi.org/10.1111/j.1539-
6924.2011.01597.x
Morral, A. R., & Jackson, B. A. (2009). Understanding the role of deterrence in
counterterrorism security. RAND Corporation occasional paper series. Santa Monica,
CA: RAND Corporation. Retrieved from
http://www.rand.org/content/dam/rand/pubs/occasional_papers/2009/RAND_OP281.pdf
Price, C., & Forrest, J. (2013). Practical aviation security: Predicting and preventing future
threats. Waltham, MA: Butterwork-Heinemann.
Rose, A., Avetisyan, M., Chan, O., Rosoff, H., Burns, W. J., & Slovic, P. (2015). The role
of behavioral responses in the total economic consequences of terrorist attacks on U.S.
air travel targets. Manuscript submitted for publication.
Transportation Security Administration (TSA). (2015). Mission statement. Official website of the
U.S. Department of Homeland Security. Retrieved from http://www.tsa.gov/about-
tsa/mission on February 12, 2015.
von Winterfeldt, D., & Edwards, W. (1986). Decision analysis and behavioral research. New
York, NY: Cambridge University Press.
von Winterfeldt, D., & O’Sullivan, T. (2006). Should we protect commercial airplanes against
surface-to-air missile attacks by terrorists? Decision Analysis, 3, 63–75.