Office of Test & Evaluation
Science and Technology Directorate
Cybersecurity Test and Evaluation
Alex HooverTest Area Manager
Cyberspace & Homeland Security Enterprise Programs
202-254-5615
Policy
Practice
Threat Assessment
COI/MOE/MOP
Agenda
2
Purpose Improve operational resilience of network-enabled capabilities
and inform major acquisition decisions.
Applicability Acquisition programs subject to DOT&E oversight will
incorporate these procedures into all future TEMPs and OT&E Plans.
Programs will include cybersecurity in TEMPs
Mission context, threat description, stakeholders, evaluation framework, integrated
T&E, and resources
OTAs will include cybersecurity in OT&E concepts, plans, & reports
Realistic threat portrayal to determine mission effects
DOT&E will include cybersecurity in LOAs
Effectiveness, Suitability, and Cybersecurity
Procedures for Cybersecurity OT&E
3
Cybersecurity-Informed Acquisition
4
Is the capability sufficiently cyber secure
to enter initial production/deployment?
Refine T&E
Strategy
Conduct
Develop-
mental T&E
NEEDANALYZE/
SELECTOBTAIN
PRODUCE/
DEPLOY/
SUPPORT1 2 2B 2C 3
Test &
Evaluation
TEMP
OTEP
LOALOA
OTER OTERTEMP
Input to
Operational
Requirements
Develop T&E
StrategyConduct
Operational
T&E
LOA
TEMP
5
Lifecycle Cybersecurity T&E Activities
5
Cybersecurity Requirements
Attack SurfaceIntent
T&E
• MNS
• ORD
• CONOPS
• Threat
Assessment
• Local
• Adjacent
• Network
• Denial
• Disruption
• Modification
• Exfiltration
• Pivot
• Security T&E
• Blue & Red Team Assessments
• Realistic threats in cyber domain
Rigorous T&E is essential to close
the gap between authorities to operate and
operating securely
IOT&E FOT&E
NEEDANALYZE/
SELECTOBTAIN
PRODUCE/
DEPLOY/
SUPPORT1 32C2 LRIP
JRC 2BDT&E
ST&EST&E
Attack Surface
• Local
• Adjacent
• Network
Kill Chain
• Tactics
• Exploits
Kill Chain
• Tactics
• Exploits
Clearly define the threat(s) to the system and corresponding missions
Threat assessment should answer the following general questions:
Which threat actors may target the missions that the system supports?
What is their intent?
What do they view as the critical terrain to accomplish their intent?
What are their capabilities in terms of knowledge, tools, and operations?
What are their most likely and most dangerous attack vectors based upon their
intent and capabilities?
DHS does not have an existing process or office of primary
responsibility for program-specific threat assessments
Use Requests for Information thru Component and Department intelligence offices
DOT&E working long-term solution with DUSM, DHS I&A, and JRC
Program-Specific Threat Assessment
6
Denial – Blocking completion of mission tasks.
Degradation – Decreasing the speed, quality, or other performance
characteristics for mission tasks.
Manipulation – Altering the information available to decision makers.
Exfiltration – Gaining information about mission details to be
exploited against other assets.
Pivot – Using access to one system/network to gain access to a
partner system/network.
Intent
7
Capabilities
8
Minimal Limited Moderate Advanced
Kn
ow
led
ge
General Systems Home market hardware,
networks and, general-
purpose languages. Basic
user OS and applications.
Public cryptography/
authentication. Public exploits
of known vulnerabilities.
Common hardware, firmware, and
defensive devices. Enterprise
network and OS. Industry data
protocols. 0-day exploits of less
common/more vulnerable software,
custom software.
Custom hardware, embedded
systems, and less common
network/protocols, specialized
firmware. Biometric-based
authentication. 0-day exploits of
more common/less vulnerable
software.
Classified systems, platforms, and
software. Cross-domain devices,
cryptography and associated hardware.
0-day exploits of restricted government
systems and industrial control systems.
Target Network and
Systems
Information found from
commonly available open
sources or from external
reconnaissance of target
organization.
Knowledge of network and system
specifications and type/configuration
of host-based defenses equivalent
to an authorized user in the target
environment.
Knowledge of network and
system specifications and
type/configuration of networked
defenses equivalent to an
authorized administrator in the
target environment.
Knowledge of network and system
specifications and defenses equivalent
to an authorized domain administrator in
the target environment.
Target Operations Information found from
commonly available open
sources or from external
reconnaissance of target
organization.
Knowledge from more specialized
literature or equivalent to prior
experience with target operations,
including key information or
supporting systems.
Knowledge equivalent to
substantial prior experience with
target operations, including work
flow and sub-task objectives.
Knowledge of current target operations
equivalent to an experienced authorized
operator.
To
ols
Hardware Inexpensive home market
hardware.
Hard-ware, clusters, costing
$10,000s or dozens of man hours.
Hardware costing $100,000s or
hundreds of man hours.
Custom hardware costing $1,000,000s
or thousands of man hours.
Software Freeware and inexpensive
commercial tools.
Commercial software. Custom software, polymorphic
malware, rootkits.
Custom software, firmware-resident
malware.
Infrastructure Access through publically
available infrastructure.
Direct control of leveraged public
infrastructure.
Covert remote access tools and
loggers.
Covert close access.
Op
era
tio
ns
Planning Opportunistic actions, no
planning.
Intent and short-range plans formed
on-the-fly as needed.
Organizes one or more
operations with specific target
systems and associated effects
on target organization
Organizes multiple operations against
separate targets, synchronizing timing,
accesses, and planned second-order
effects
Procedures No demonstrated stealth, non-
attribution or efficient use of
resources
Countermeasures for common
defensive systems. Non-attribution.
Efficiency in use of resources
consistent with intent.
Advanced and custom non-
attribution tools. Efficiency in use
of resources consistent with
intent
High degree of control of defensive
infrastructure. Non-attribution, false flag
operations. Efficiency in use of resources
consistent with intent
Persistence Intermittent, directed activity. Gradual, low level passive
operations.
Repeated active operations. 24/7 monitoring and control of offensive
capabilities.
8
Critical Operational Issue
Is this capability resilient to cyber attack?
Measures of Cybersecurity
How resilient is this mission to DOS attack of this capability?
How resilient are the tasks to cyber degradation?
How resilient are the procedures to data manipulation?
How resilient is the mission to data exfiltration of the key cyber terrain?
How well does this system protect against attack from/to interfaced capabilities?
Possible Evaluation Questions
9
10
Cybersecurity
Is this capability resilient to cyber attack?
Denial of Service (Mission Impact)
- Probability of Occurrence - Repeatability
- Duration - Attack Resources
Degradation of Service (Task Impact)
- Probability of Occurrence - Degree of Degradation
- Duration - Attack Resources
- Repeatability - Defend Resources
- Probability of Detection
Data Manipulation (Task Impact)
- Probability of Occurrence - Degree of Manipulation
- Duration - Attack Resources
- Repeatability - Defend Resources
- Probability of Detection
Data Exfiltration (Enterprise Impact)
- Probability of Occurrence - Significance of Exfiltration
- Duration - Attack Resources
- Repeatability - Defend Resources
- Probability of Detection
External Pivoting (Enterprise Impact)
- Probability of Occurrence - Probability of Detection
- Duration - Attack Resources
- Repeatability - Defend Resources
Effectiveness
Measure 1
Measure 2
Measure 3
Suitability
Measure 1
Measure 2
Measure 3
Cybersecurity
Measure 1
Measure 2
Measure 3
Understand Collective
Impact on Mission/Task
Accomplishment
Sample
Cybersecurity
Evaluation Structure
Rules of Engagement
11
Purpose – Support evaluation of ...
Threat Assessment
Actors (FIS, Terrorist, Criminal, Activist, Mercenary, Hackers)
Intent
Capabilities (Historical, Projected, Surrogates)
Strategic Goals
Definition of Capability Under Test
System Boundary
Included Systems
Excluded Systems
Mission Impacts (DOS, DEG, DMAN, EXFIL, PIVOT)
Operational Objectives
Targeted Data (leads to ...) Deliberate
Targeted Systems (leads to ...) Deliberate
Targeted Networks (leads to ...) Deliberate 80 / Exploratory 20
Targeted Interfaces (leads to ...) Deliberate 50 / Exploratory 50
Relevant Vulnerabilities Deliberate 20 / Exploratory 80
Rules of Engagement (cont’d)
12
Tactical Plan
Schedule
Operational Objective, Capability (Surrogate) Start, End
Initial Access
TTP by Scheduled Event (planned and contingency)
OCO Actions
Limits of Action
Prohibited Actions
Termination Conditions/Notification
DCO Posture
Active Events – Events the DCO will carry out
Stop Events – Events where the DCO will report detection to the red team and the event will stop
Passthrough Actions – Events where the DCO will report detection and the event will proceed
Data Handling
13