+ All Categories
Home > Documents > Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office...

Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office...

Date post: 19-Mar-2020
Category:
Upload: others
View: 24 times
Download: 3 times
Share this document with a friend
41
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977 Cybersecurity Test and Evaluation Process June 2018
Transcript
Page 1: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity Test and EvaluationProcess

June 2018

Page 2: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Agenda

• Cybersecurity T&E Introduction

• Cybersecurity T&E Policy

• Cybersecurity T&E Process

• Cybersecurity T&E in the TEMP

• Cyber Ranges

• Cybersecurity T&E Guidebook

2

Page 3: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Introduction

• Many DoD systems have not proven to be cyber secure– Year after year DOT&E assessments have shown that systems remain vulnerable

• Security controls programs, such as the Risk Management Framework (RMF), are necessary but not sufficient

– These compliance measures do not adequately address threat tactics and capabilities

– These controls are frequently considered late in development

• Mission risk and operational resilience have not been properly addressed in controls based security

• There is a need for a more robust cybersecurity process– Establishing thorough cybersecurity requirements– Engineering cybersecurity into the system as opposed to adding it late– Thoroughly testing and evaluating systems and providing feedback to the

development engineers for action

• This brief describes the Cybersecurity T&E process

3

Page 4: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity T&E Process

Cybersecurity T&E is necessary and required by policy– Evaluates a system’s mission performance in the presence of cybersecurity threats – Informs acquisition decision makers regarding cybersecurity, resilience and

survivability

4

CDD Validation

Dev RFP Release Decision

IOT&EOTRR

IATT

Full RateProduction

Decision ReviewATO

Lower Fidelity Mission-Based Cyber Risk Assessments Higher Fidelity

MDD

PHASES

TechnologyMaturation & Risk

Reduction

Engineering & Manufacturing

Development Production & Deployment

Materiel SolutionAnalysis

MS CMS BMS A

Phase 2 Characterize

the Cyber Attack

Surface

Phase 4 Adversarial

Cybersecurity DT&E

Phase 5 Cooperative

Vulnerability and Penetration

Assessment

Phase 3 Cooperative

Vulnerability Identification

Phase 6 Adversarial Assessment

Phase 1 Understand

Cybersecurity Requirements

DoDI 5000.02, Enclosure 14 – planning and conducting cyber T&E

Cyber T&E analysis and planning Cyber T&E

DRAFTCDD CDD

PDR CDRCPD

TRR

Operations & Support

OTRR IOT&E

Page 5: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity T&E Policy

5

Page 6: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Policy Overview

• DoDI 5000.02, Operation of the Defense Acquisition System, August 10, 2017, incorporating Change 3 – Enclosure 14

• DoDI 5000.75, Business Systems Requirements And Acquisition, February 2, 2017

• “Procedures for Operational Test and Evaluation of Cybersecurity in Acquisition Programs,” DOT&E Memo, April 3, 2018

• DoDI 8500.01, Cybersecurity, March 14, 2014

• DoDI 8510.01, Risk Management Framework (RMF), July 28, 2017, with Change 2

• JROCM 009-17, “System Survivability KPP Update to ensure Joint Force Mission Assurance”

– Cyber Survivability Endorsement Implementation Guide (CSEIG), v1.01a

6

DoDI 5000.02 DoDI 8500.01 DoDI 8510.01Procedures for Operational T&E

JROCM 009-17DoDI 5000.75

Page 7: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

DoDI 5000.02, Enclosure 14Requires Cybersecurity T&E Planning

• General T&E Planning [Paragraph 3.b.(13)]– Work closely with the Chief Developmental Tester (CDT) and T&E WIPT to plan,

resource and conduct cybersecurity T&E– Refer to the Cybersecurity T&E Guidebook and DOT&E “Procedures for

Operational Test and Evaluation of Cybersecurity in Acquisition Programs”– Document T&E activities in TEMP, including the T&E Strategy, evaluation frameworks

(DT&E and OT&E), and resource requirements

• Requirements, Key Elements, and Resources [Paragraph 5.b.(10)]– Develop a cybersecurity T&E methodology based on derived system requirements and

draft performance specifications– Test key system elements and interfaces identified through criticality and vulnerability

analysis – Identify the cybersecurity T&E resources, (e.g., cyber ranges) for each T&E activity and

document T&E planning in the TEMP

• Cyber-Attack Surface [Paragraph 5.c.(5)] – For T&E, understand the cyber-attack surfaces and refine the T&E planning and

activities for cybersecurity

7

Page 8: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

DoDI 5000.02, Enclosure 14Requires Cybersecurity DT&E and OT&E

• Development Test & Evaluation [Enclosure 14, Paragraph 3.b.(13)(a)]– Cooperative Vulnerability Identification (CVI)

Conduct T&E activities to collect data needed to identify vulnerabilities

– Adversarial Cybersecurity Developmental Testing (ACD)Conduct a cybersecurity DT&E event using realistic threat exploitation techniques in representative operating environments

• Operational Test & Evaluation [Enclosure 14, Paragraph 3.b.(13)(b)]– Cooperative Vulnerability and Penetration Assessment (CVPA)

An overt examination of the system to identify all significant vulnerabilities and the risk of exploitation of those vulnerabilities

– Adversarial Assessment (AA)Assesses the ability of a unit equipped with a system to support its mission while withstanding cyber threat activity representative of an actual adversary

• Provide T&E feedback to engineering teams [Enclosure 14, Paragraph 3.b.(13)]– This will help avoid costly and difficult system modifications late in the

acquisition life cycle

8

Page 9: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

DoDI 5000.75Requires Cybersecurity T&E

• Defines policy and procedures, including cybersecurity, for DBS – Describes the use of the Business Capability Acquisition Cycle (BCAC) for business

systems requirements and acquisition– Outlines responsibilities the PM must implement to safeguard DoD business

systems throughout the system life cycle

• Program Office Implementation Plan must include cybersecurity processes to reduce technical risk through T&E management procedures

• Appendix 4B.2.h(2) requires:– Developmental Evaluation Framework– Cooperative vulnerability identification and adversarial cybersecurity testing in both

developmental and operational tests– A Cyber Economic Vulnerability Analysis (CEVA) - required at the discretion of

DOT&E for DoD systems whose functions include financial or fiscal/business activities or the management of funds

– Direction to Milestone Decision Authorities (MDAs) to avoid tailoring cybersecurity T&E solely to meet Authorization to Operate (ATO) requirements

9

Page 10: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

DOT&E April 3, 2018 Memo, “Procedures for Operational Test and Evaluation of Cybersecurity in

Acquisition Programs”

• Reiterates requirement for CVPA and AA• Requirement applies to all system acquisition programs under

DOT&E oversight• Operational Test Agencies may tailor procedures specifically to

support the evaluation of weapons, platforms, networks and other systems that handle or transfer data and consider:

– Operational context– System extent– System-unique attributes– Specialized components

• Testing of cybersecurity during OT&E must include representative users and an operationally representative environment

• Certification, Accreditation, and/or Authorization processes should inform OT&E, but are not substitutes for OT&E

10

Page 11: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

DoDI 8500.01 / DoDI 8510.01Planning and Conducting Cybersecurity T&E

• Cybersecurity [DoDI 8500.01]

– DASD(DT&E) and DOT&E collaborate on procedures for cybersecurity T&E− The Cybersecurity T&E Six Phase Process

– DoD Component− Provides for cybersecurity testing capability− Conducts vulnerability assessments− Ensures cybersecurity T&E is conducted throughout the acquisition lifecycle

– Defines activities for the CDT, Lead DT&E Organizations, and T&E community− Integrate RMF into DT&E − Document Cybersecurity T&E in the TEMP− Integrate with interoperability and other functional testing

• Risk Management Framework (RMF) [DoDI 8510.01]

– The RMF process will inform the acquisition process for all DoD IT, including developmental and operational T&E

– Requires integration of DT&E activities into the RMF and provides the RMF Technical Assurance Group with input as needed

– Ensure T&E of the assigned information system and information technology system is planned, resourced, and documented in the program T&E Master Plan

11

Page 12: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

DoDI 8500.01Operational Resilience is Assessed During

Cybersecurity T&E

• Acquisition programs must conduct an operational resilience evaluation during cybersecurity DT&E and OT&E– Perform cybersecurity DT&E and OT&E, including the ability to detect and

react to penetrations and exploitations and to protect and restore data and information, in order to inform acquisition and fielding decisions. [Enclosure 3, paragraph 3.b]

– Exercise system under realistic cyber conditions using test procedures and tactics to develop work-arounds and fall-backs in the face of hostility [Enclosure 3, paragraph 3.e]

12

RMF Process Does NOT Assess Operational Resilience

Page 13: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Joint Staff Guidance - System Survivability KPP

• JROCM 009-17 Item 4– Supports incorporation of cyberspace as a critical component of the SS KPP in

requirements documents– Applies to Joint programs and joint concern programs

• Service/component programs may include SS KPPs as applicable to mission context

• As part of the SS KPP assessment, the TEMP should describe: – Cyber survivability attributes – Technical performance specifications for the attributes– Countermeasures to support cyber survivability

• Programs should leverage the Cyber Survivability Endorsement Implementation Guide (CSEIG) developed by the Joint Staff/J6 in collaboration with DoD CIO, the DIA, and the NSA

– CSEIG consists of guidance that helps acquisition programs ensure cyber survivability requirements are included in system designs as early as possible

– Ensures cybersecurity is part of the operational risk trade space for functional requirements

– Incorporates cybersecurity attributes to support system survivability and operational resiliency requirements

13

Assess Cyber Survivability During DT and OT

Page 14: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity T&E Process

14

Page 15: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity T&E Process

15

Page 16: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cyber T&E analysis and planning DoDI 5000.75 Cyber T&E

Business Capability Acquisition Cycle (BCAC) and Cybersecurity T&E

BCAC

Phase 2 Characterize

the Cyber Attack

Surface

Phase 4 Adversarial

Cybersecurity DT&E

Phase 5 Cooperative

Vulnerability and Penetration

Assessment

Phase 3 Cooperative

Vulnerability Identification

Phase 6 Adversarial Assessment

Phase 1 Understand

Cybersecurity Requirements

Capability Need Identification

Business Solution Analysis

Business System Functional Requirements & Acquisition Planning

Business System Acquisition, Testing & Deployment

LimitedDeployment

ATP(s)

Acquisition ATP

Functional Requirement

ATP

Solution Analysis

ATP

Contract Award

Lower Fidelity Higher FidelityMission-Based Cyber Risk Assessments

Capability Support

Capability Support

ATP

FullDeployment

ATP

Alignment with BCAC (DoDI 5000.75) addressed in Appendix C of the Cybersecurity T&E Guidebook v2.0

16

Page 17: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

System Development and TestCybersecurity, Survivability, and Resilience (1 of 2)

• Cyber Requirements and System Design– Design for system survivability – SS KPP

− Use Cyber Survivability Endorsement Implementation Guide to design for system survivability, as required by JCIDS Manual

– Design for operational resiliency in the operational environment under expected cyber threat conditions, as required by DoDI 8500.01

– Incorporate and validate cybersecurity controls as required by DoDI 8510.01, RMF

• Cybersecurity DT&E verifies system requirements to find problems and fix them during development – T&E conducted across cybersecurity, system survivability, and operational

resiliency objectives

17

Goal: Field Systems That Withstand Cyber Threats

Page 18: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

System Development and TestCybersecurity, Survivability, and Resilience (2 of 2)

• Cybersecurity T&E (DT/OT) encompasses:– Cybersecurity assessments for software assurance, vulnerability

identification, configuration compliance and cybersecurity functionality verification (Phase 3) throughout the life cycle− Includes security controls assessment later in the life cycle for the RMF Step 4

and informs RMF Step 5− Submission to the AO

– System survivability testing to address specific cyber survivability attributes− Does the cybersecurity and system design prevent, mitigate, and recover from,

cyber-attacks (Phase 3, 4, 5 and 6)– Operational resilience testing

− Evaluates the ability of a system to successfully perform its mission in a cyber-contested environment

− When a threat is actively attempting to cause mission failure (Phase 4, 6)− Integrate with functional testing when possible

18

Page 19: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Phase 1Understanding Cybersecurity Requirements

Understand the program’s cybersecurity and resilience requirements and develop an initial approach and plan for conducting Cybersecurity T&E• Cyber Working Group (CyWG) established• Compile List of Cybersecurity and Resilience Requirements

– Measurability, testability, and achievability• Prepare for Cybersecurity T&E Events

– Develop initial Developmental Evaluation Framework (DEF)– Identify Supporting Cybersecurity T&E Resources (labs, ranges, tools and personnel)– Develop the Initial OT Evaluation Framework– Align RMF activities with the TEMP– Plan and Schedule an MBCRA

• Plan for Cybersecurity T&E– Develop Cybersecurity T&E Strategy

Informs Request for Proposal (RFP), Preliminary Design Review (PDR), Capability Development Document (CDD), Solution Analysis Authority to Proceed (ATP), Functional

Requirements ATP, Acquisition ATP

Documented in MS A TEMP, Updated in MS B TEMP19

Page 20: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Phase 2Characterize the Cyber-Attack Surface

Characterize attack surface to identify opportunities an attacker may use to exploit the system• Identify the cyber-attack surface

– Examine System Architecture, Components and Data Flows– Analyze and Decompose System Mission– Map Mission Dependencies

• Analyze the cyber-attack surface– Characterize the Cyber Threat– Select a Cyber Kill Chain– Examine Cyber Effects on the System and Mission– Perform (or Update) Mission-Based Cyber Risk Assessment

• Document Results and Update Test Planning and Artifacts• Prepare test strategy for Phase 3 and Phase 4 Cybersecurity DT&E Events

Informs RFP, PDR, Capability Development Document (CDD) validation, Critical Design review (CDR), MS B, Functional Requirements ATP, Acquisition ATP

Update the TEMP for MS B20

Page 21: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Phase 3Cooperative Vulnerability Identification

Identify the existence of any known cyber vulnerabilities in hardware, software and architecture and verify cyber survivability and resilience capabilities• Early and ongoing feedback starting at MS B• CVI is NOT a single event

– Contractor T&E– Government T&E– Continuum of vulnerability assessment activities tailored to the program

• Integrates RMF assessments• Develop test objectives, plan events and infrastructure during Phases 1 and 2 • Vulnerability testing while planning for threat testing in Phase 4

Informs CDR and Test Readiness Review, Functional Requirements ATP, Acquisition ATP, Limited Deployment ATP

CVI Events are Risk Reduction Activities21

Page 22: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Phase 4Adversarial Cybersecurity DT&E

Evaluate the system’s cybersecurity in a mission context, using realistic threat exploitation techniques while in a representative operating environment• Program must plan to replicate the system in a representative test infrastructure• Updated threat assessment• Cyber table top (CTT) exercises and service-specific MBCRAs help to inform

Phase 4 test objectives and test plans/vignettes• Execute prior to MS C and prior to the Authorization to Operate (ATO)• Cyber risk assessment describing operational mission impacts from tested cyber

attacks

Informs MS C production decision, ATO and Operational Test Readiness Review, Limited Deployment ATP

Certified Red Team NOT required22

Page 23: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Phase 5Cooperative Vulnerability and Penetration

AssessmentProvide a comprehensive characterization of the cybersecurity and resilience status of a system in a fully operational context and provide reconnaissance of the system to support adversarial testing• Uses data taken from cooperative cybersecurity test events• Early engagement with the OTA is essential for planning

– Plan and coordinate with cybersecurity vulnerability assessment team (“blue team”)• Can be integrated with CVI activities, a standalone event, a series of test events,

or an operational component of an integrated test

System vulnerability data supports adversarial testing in Phase 6, MS C, LRIP, Full Rate Production (FRP), Full Deployment Decision (FDD) and ATP

decisions

Aligned to 2018 DOT&E Memorandum

23

Page 24: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Phase 6Adversarial Assessment

Characterizes the operational effects to critical missions caused by threat-representative cyber activity against a unit trained and equipped with a system, as well as the effectiveness of defensive capabilities• Evaluate the ability of the system, tiered defenses, and defenders to

protect critical mission functions; detect and respond to cyber-attacks; and assess system resilience to survive and recover from attacks, and complete critical missions and tasks

• OTA plans testing• National Security Agency Certified Red Team performs testing

Results inform the operational effectiveness, suitability, and (in some cases) survivability of the system(s) under test due to cybersecurity vulnerabilities and the

resulting mission effects. Also informs FRP or FDD and Full Deployment ATP

Aligned to 2018 DOT&E Memorandum

24

Page 25: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Mission-Based Cyber Risk Assessment (MBCRA)

• A process of identifying, estimating, assessing and prioritizing risks based on impacts to DoD operational missions resulting from cyber effects on the system(s) being employed

• Informs RMF Steps 1-5 AND informs Cybersecurity T&E planning– Activities begin in Phase 1

• Identifies mission-impacting risks to test and mitigate– Assists in focusing and prioritizing the Cybersecurity T&E effort– Several common methodologies, including Cyber Table Tops

• Best practices described in Cybersecurity T&E Guidebook v2.0 Appendix X3 (FOUO Appendix)

25

Page 26: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

The Risk Management Framework is Necessary

• Required by policy – DoDI 8500.01 3.a and 3.h requires cybersecurity risk management– DoDI 8510.01 Risk Management Framework (RMF) implements DoD’s Risk

Management Policy • RMF provides a structured, tailorable, and repeatable process

that integrates security and risk management activities into the system development life cycle

– Considers effectiveness, efficiency, and constraints due to applicable laws, directives, Executive Orders, policies, standards, or regulations

• RMF helps ensure the appropriate “cyber hygiene” controls and security configurations are designed into the system

– Protections to help meet the goals of risk-managed Confidentiality, Integrity and Availability

– Adds continuous monitoring to system life cycle management to ensure ongoing awareness of and risk managed responses to changing threats and environments

26

RMF Does Not Replace Cybersecurity T&E

Page 27: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

RMF Alignment with T&E Process

27

PHASES

TechnologyMaturation & Risk

Reduction

Engineering & Manufacturing

Development Production & Deployment

Materiel SolutionAnalysis

MS CMS BMS A

Phase 2 Characterize

the Cyber Attack

Surface

Phase 4 Adversarial

Cybersecurity DT&E

Phase 5 Cooperative

Vulnerability and Penetration

Assessment

Phase 3 Cooperative

Vulnerability Identification

Phase 6 Adversarial Assessment

CDD Validation

Dev RFP Release Decision

IOT&EOTRR

IATT

Full RateProduction

Decision ReviewATO

Phase 1 Understand

Cybersecurity Requirements

Lower Fidelity Mission-Based Cyber Risk Assessments Higher Fidelity

DRAFTCDD CDD

PDR CDR TRRCPD

Operations & Support

MDD

RMF Step 1 Categorize

RMF Step 2Select controls

RMF Step 3 Implement Controls

RMF Step 4 Assess Controls

RMF Step 5 Authorize

New ATO ~3 years

Page 28: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity T&E in the TEMP

28

Page 29: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity in the TEMP (1of 2)(Starting at MS A)

• T&E strategy identifies Cybersecurity DT&E activities (contractor and govt) to:– Understand cybersecurity requirements - compile and analyze for

measurability, testability, and achievability (Phase 1)– Expose system's reachable and exploitable vulnerabilities to

characterize the attack surface (Phase 2)– Assess sub-components and components against potential

vulnerabilities (Phase 3)– Assess system against potential vulnerabilities (Phase 3)– Assess system resiliency against an adversarial cybersecurity

threat (Phase 4)

29

High Level Descriptions of Who, What, Where, When, Why, How

Page 30: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity in the TEMP (2 of 2)(Starting at MS A)

• DT&E Methodology includes cybersecurity and outlines the essential information needed to support programmatic, technical, and acquisition decisions

– If possible at MS A, include a Developmental Evaluation Framework (DEF) that identifies cybersecurity data for assessing progress toward cybersecurity requirements

– DEF is required at MS B• Identify when integrated cybersecurity testing (DT – OT) will

occur• Include RMF categorization and integrate RMF data needs into

DT&E activities• Include the SS KPP Cybersecurity Risk Categorization and the SS

KPP Cyber Survivability Attributes– Attributes should be available at MS B

• Define roles, responsibilities, and resources for detailed planning and execution of Cybersecurity DT&E activities, e.g., use of cyber ranges

30

TEMPs Should Not be a “Copy and Paste” of Guidance or Policy

Page 31: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cyber Test Planning in the TEMP

31

MS A TEMP MS B TEMP MS C TEMP

What to Test

Approach to conducting Phase 1 & 2 to inform cyber test events. Discussion of iterations of Phase 1 & 2 to support requirements and design changes

Test Events Planned (Who, When, Where, limitations, etc.)• Based on functions critical to

mission success that are potentially vulnerable

• Informed by mission-based cyber risk assessments

Testing conducted and testingyet to be done

How to TestDiscussion of plans for dedicatedcyber test events, integrated DT/OT events, and SCA

Test events fully documented: tools, contractor development labs, cyber ranges, etc.

Testing completed and described with references to test results

DEF - Cyber Initial DEF showing continuum of cyber test activities expected

Updated DEF showing planned cyber test activities; DEF is required at MS B

Updated DEF for future testing if needed

DataData needed to inform decision makers and system design process

Plans to collect required data during cyber testing

Test data used for mission-based risk assessment informing ACD events and TRR

ScheduleApproach to scheduling cyber test events including security controls assessment, integrated DT/OT, and functional T&E integration

Schedule for cyber tests and their estimated duration; test results inform CDR and TRR

Tests remaining to be conducted in schedule

ResourcesDiscussion of required test resources: people, test environment, processes, tools, etc.

Resources allocated to cyber testers (org that will supply testers) and test articles (HWIL, SIL, virtual systems, etc.)

Updated resources allocated to future testing

Page 32: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Decision #3

DSQ #1 DSQ #2 DSQ #3 DSQ #4 DSQ #5 DSQ #6 DSQ #7 DSQ #8

Functional evaluation areas System capability categories

Technical Reqmts

Document Reference

Description

Performance3.x.x.5 Technical Measure #1 DT #1 M&S #2 DT #4 M&S #2

3.x.x.6 Technical Measure #2 M&S #1 DT #3 DT #4 M&S #2

3.x.x.7 Technical Measure #3 DT #3 IT #1

3.x.x.8 Technical Measure #4 M&S #4 IT #1

InteroperabilityInteroperability Capability #1 3.x.x.1 Technical Measure #1 DT #3 DT #4

3.x.x.2 Technical Measure #2 IT #2 M&S #4 DT #4 M&S #2

3.x.x.3 Technical Measure #3 IT #2 IT #1 DT #3

Cybersecurity

5.x.x.1 Technical Measure #1 CTT #1 CVI #2 ACDT CVI #3

5.x.x.2 Technical Measure #2 CTT #1 SCA ACDT CVI #3

5.x.x.3 Technical Measure #3 CVI #1 SCA ACDT ACDT

5.x.x.4 Technical Measure #4 CTT #1 CVI #1 ACDT CVI #3

Reliability4.x.x.1 Technical Measure #1 M-demo

#1 IT #54.x.x.2 Technical Measure #2 M-demo

#1 IT #2 IT #54.x.x.3 Technical Measure #3 M-demo

#2 IT #24.x.x.4 Technical Measure #4 M-demo #2 IT #2

Reliability Capability #1

Reliability Capability #2

Data Sources (Test, M&S events)

System Requirements / Measures

Developmental Evaluation Objectives

Decision #1 Decision #2 Decision #4

Performance Capability #1

Performance Capability #2

Security Capability #1

Interoperability Capability #2

Security Capability #2

Cybersecurity in the Developmental Evaluation Framework

32

Decisions

Evaluation

Test / M&S

Resources

Schedule

Def

ine Inform

Def

ine Data

Def

ine Execute

Page 33: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Example Completed Cyber Portion of a DEF (Notional)

33

Developmental Evaluation Objectives

System Requirements/ Technical Measures EMD RFP Release

MS B / Contract AwardEMD Long Lead Items for A/C (A1, A2, A3) & Radars (for A/C and SIL)

Approval to Enter Gov't Led IDT&E

LRIP Long Lead Items Approval to Enter IOT&E

System Capabilities

SRD Rqmt's (Potential CTPs*)

Technical Measures

DSQ1: Did at least two Contractors provide technical

designs and information for

successful PDRs?

DSQ2: Have at least two

Contractors demonstrated

sufficient subsystem maturity?

DSQ3: Can the Aircraft meet

Requirements?

DSQ4: Can the Radar and SUT

subsystem integration meet Performance and

Processing Requirements?

DSQ5: Has the KTR demonstrated a fully integrated, functional and

stable, Radar/Comm/ C2 capability in the

SIL?

DSQ6: Has the KTR demonstrated a fully integrated, functional and

stable, Aircraft/ Radar/Comm/C2

system?

DSQ7: Do any system deficiencies

preclude an LRIP purchase?

DSQ8: Does the performance and reliability support

all required mission profiles?

DSQ9: Are cybersecurity vulnerabilities identified and

acceptable mitigations in place?

CyberSecurity Data Sources (Analysis, Test, M&S Events)

Protect; Data Security - System

Data at rest, Data in transmission

Architectural Vulnerability Analysis (AVA)

Mission Cyber Dependency

Analysis - Cyber Table Top Exercise

CVI-Data Security testing

CVI-STIG compliance verification

Security Controls Assessment (SCA)

ACD

RMF Risk Assessment/ATO

Software Assurance

Architectural Vulnerability

Analysis (AVA)Contractor T&E

CVI - Software Development Verification

CVI - Software Development Verification

PPP Supply Chain Risk Management

SCRM TAC Assessment

SCRM TAC Assessment

SCRM TAC Assessment

SCRM TAC Assessment

Hardware Assurance

Architectural Vulnerability

Analysis (AVA)Contractor T&E

CVI - Hardware Development Verification

CVI - Hardware Development Verification

Protect; Data Security –Interfaces

Critical Data Exchanges

Architectural Vulnerability Analysis (AVA)

Interoperability -Cybersecurity IT

CTT Verification Exercise

System Resilience and Survivability

SS KPP CSA

Detecting attacks (how long to

detect, how many detected versus

attempted, mission impacts)

Mission Cyber Dependency Analysis - Cyber Table Top ExerciseCVI-Cyber

Functionality Verification

CTT Verification Exercise ACD

SS KPP CSAResponding to

attacks (how long to respond)

Mission Cyber Dependency Analysis - Cyber Table Top ExerciseCVI-Cyber

Functionality Verification

CVI - Incident Response

AssessmentACD ACD

SS KPP CSA

Recovering from attacks (how long

does recovery take? Does that

impact success of the mission?)

Mission Cyber Dependency Analysis - Cyber Table Top Exercise CVI - COOP assessment ACD

ACDCVI - COOP assessment

Page 34: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cyber Ranges

34

Page 35: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cyber Range Overview

• Adequate DT&E, OT&E, and assessments may require testing on Cyber Ranges for one or more of the following reasons:

– Testing cannot occur on open operational networks– Representations of advanced cyber adversarial TTPs are not suitable for

operational networks– Scaling requirements (e.g., number of users, hosts, or interconnected systems;

amount of network traffic) cannot be otherwise achieved– Operational complexity and associated mission risk are such that impact to

operational networks should be avoided

• Planning for the use of a cyber range should begin as early as possible in the acquisition lifecycle and be reflected in the TEMP

• For more information about test range planning for Cyber T&E, refer to the Cybersecurity T&E Guidebook 2.0 Appendix X4

35

Cybersecurity T&E Performed Only During OT&E is Too Late

Page 36: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

When to Use a Cyber Range

Test Range EventLarge-scale Simulation to Train Cyber Mission

Forces and Evaluate Cyber Defensive and Offensive Operations

Pre MS A/BRequirements and Systems Security

Engineering Analysis

Test Range EventMission Thread

Testing with Blue Team

Test Range EventMission Thread Testing with Red

Team in a Realistic Threat Environment

SE/DT&EEvaluate Software and

Systems Security Architecture

Training & ExercisesEvaluate TTPs in a

Contested Environment

Test Range EventCybersecurity

Verification and Validation

RMF/DT&EVerify Baseline Cybersecurity

Requirements and Vulnerability Assessment

DT&E/OT&EEvaluate Mission Capabilities

and Interoperability in a Contested Environment

Test Range EventCybersecurity Architecture Evaluation

36

Page 37: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity T&E Guidebook

37

Page 38: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

DoD Cybersecurity T&E Guidebook

• Version 2.0 published April 2018• Describes each phase, inputs, outputs, tasks• Addresses RMF integration• Includes new appendices - FOUO appendices published

separately (June 30, 2018)• Publicly accessible links to the Guidebook:

– https://www.acq.osd.mil/dte-trmc/docs/CSTE%20Guidebook%202.0_FINAL%20(25APR2018).pdf

– https://www.dau.mil/cop/test/DAU%20Sponsored%20Documents/CSTE%20Guidebook%202.0_FINAL%20(25APR2018).pdf?Web=1

38

Page 39: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity T&E Guidebook Outline

• Introduction– Sections: Purpose, Organization, Audience– Shortened from v1.0

• Cybersecurity in the Defense Acquisition System– Overview of policy basis for Cyber T&E– Sections: DoDI 5000.02, DoDI 5000.75 Defense Business Systems, DoDI 8500.01

Cybersecurity, DoDI 8510.01 Risk management Framework (RMF), Joint Requirements Guidance, DOT&E Cybersecurity Procedures Memoranda

• Cybersecurity T&E - Phases Overview– Cyber Working Group, cyber threat assessments, DT&E and SE collaboration, early

tester involvement, MBCRA, role of Cybersecurity DT&E, DT&E and OT&E collaboration

• Phase 1 Through 6– Sections are uniform through each phase

− Purpose and schedule− Inputs – from Guidebook v1.0− Tasks – tailored for each phase, includes methods and best practices− Outputs – TEMP updates, acquisition decision informed

– RMF and MBCRA/Cyber Table Top (CTT) Exercise integration throughout each phase

39

Page 40: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity T&E Guidebook Unclassified Appendices

Appendix A (new)Phase 1-6 Quick Look– Single page for each phase showing inputs, tasks, and outputs

Appendix B (new)Incorporating Cybersecurity T&E into DoD Acquisition Contracts

Appendix C (new)Considerations for Tailoring the Cybersecurity T&E Phases

Appendix D (update)Key Program Artifacts for Cybersecurity T&E Analysis and Planning

Appendix E (update)Guidance on the Cybersecurity Portion of the Developmental Evaluation Framework

Appendix F (new)Considerations for Staffing Cybersecurity T&E Activities

Appendix G (new)Considerations for Software Assurance Testing

40

Page 41: Cybersecurity Test and Evaluation Process Sponsored Documents/Cybersecurity... · Program Office Implementation Plan must include cybersecurity ... – Ensures cybersecurity is part

CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977

Cybersecurity T&E Guidebook FOUO Appendices

For Official Use Only Appendices are accessible to government and authorized contractor personnel

– Contact DASD DT&E Cybersecurity Technical Director

Appendix X1 (significant revision)Considerations for Cybersecurity Requirements and Measures for DT&E– Cyber Survivability Endorsement Implementation for System Survivability Key Performance Parameter– STAT Metrics for Cybersecurity Test Objectives

Appendix X2 (new)Cyber Threat Assessment for Cybersecurity T&E– Integrated with phases– Supply Chain Risk Management Threat Assessment Center

Appendix X3 (new)Mission-Based Cyber Risk Assessments (MBCRAs)– Survey of MBCRA methods including Cyber Table Top exercises; how to select an MBCRA

Appendix X4 (updated)Cybersecurity Test Infrastructure and Environment Planning– Test environments from development to OT; considerations for planning cyber test infrastructure; services-specific

infrastructure; cyber range use during a MBCRA/CTT

Appendix X5 (new)Cybersecurity Test Considerations for Non-Internet Protocol (Non-IP) Systems – Introductory materials only; 1553 and 1439 (CAN) bus testing; controls systems testing

41


Recommended