Adaptable C5ISR
Instrumentation
Mission Command and Network Test Directorate
Prepared by Mr. Mark Pauls
U.S. Army Electronic Proving Ground (USAEPG)
21 May 2014
U.S. Army Electronic Proving Ground Advanced Electronics & Network Systems
Mission Command & Network Test Directorate
US Army Electronic Proving Ground
2000 Arizona Street
Fort Huachuca AZ 85613
UNCLASSIFIED
Distributed T&E Instrumentation
• C5ISR systems have evolved into very rapid Commercial-off-the-shelf (COTS) technology procurements
• Increasing use of these new and rapidly evolving technologies presents a challenge to the Test and Evaluation (T&E) community
• Shrinking procurement cycle forces the T&E cycle to also decrease
• To adequately verify these systems a robust instrumentation package must be developed and implemented to proved stimulation/simulation, data collection, reduction, and analysis
• Non-platform specific instrumentation suite of software tools that are embedded and LVC
• C5ISR system instrumentation modifications and innovative without the lengthy procurement process
• Instrumentation platform must support a wide array of standard and custom input/output (I/O) functions.
• Software tools must be reprogrammable to interpret the data streams for C5ISR system under test.
• Meet constraints of size, weight, and power (SWaP)
Challenge Solution
2
UNCLASSIFIED
Shrinking Test Technology
Resources
• Constrained Budgets
• Shrinking Number of Test Events/Increased Complexity
• Test Completion & Successful Metrics
• DOE Driven events to gain the most possible from a single event
• Aging T&E Work-force
• Limited Academia T&E Professional Programs
3
UNCLASSIFIED
Shift Left Embed Instrumentation Early
• Estimate 70% of the defects could be
discovered pre-milestone “B”
• Find defects early and avoid the
unnecessary schedule and cost over runs
• Left shift in testing takes care of the
ailments as they occur and ensures
minimal after effects.
Shift - Left
Current T&E Cycle
Improve Acquisition Outcomes
Dr. C. David Brown “T&E in Defense Acquisition” and Dr. Steve Hutcheson “Shift Left”
4
UNCLASSIFIED
Problem Definition
• Test and Evaluation Legacy
• Solutions are expensive
• Do not support the Cradle to Grave
• Do not standardized performance measurement
• Do not support lifecycle of Systems Testing
• Hardware-based solutions
• Do not leverage Systems Computing Resources
• Customized to supporting a single system test
• Increase risk with the employment of new complex systems
• Cumbersome and require maintenance
5
UNCLASSIFIED
Operational Instrumentation View
Spectrum Monitoring Sim/Stim Data Collection Data
Reduction/Analysis
6
UNCLASSIFIED
The Big Picture
Integrated Data Reduction and Analysis
•Multiple Sources of Data
• Embedded Agents
• VM Collectors
•Seamless Instrumentation Infrastructure
• Test Control
• Common DB
• Meta Data
•Data collected enterprise-wide must be fused for comprehensive analysis
Testing Advantages
•Legacy – Single unique test instrumentation
•Golden Eye Solution – Light weight hardware with installed software
•Embedded Solutions – Software embedded per milestone “B” – Life Cycle Testing – Improved Acquisition Systems
7
UNCLASSIFIED
Instrumentation Approach
Simulation
Collection Analysis
Visualization
8
UNCLASSIFIED
Solution
What is “embedded” instrumentation?
Why should we embed and virtualize
instrumentation?
9
UNCLASSIFIED
Next Generation Tools
• Machine Learning Embedded Test Tools - Software Development Kits (SDKs)
• Use of SDK’s to develop collection and analysis threads
• Use of SDK’s to validate networks
• Use of SDK’s to ensure compliance
• Use of SDK’s to support a wide array of data standard
10
UNCLASSIFIED
Application Programming Interface (API)
• Development of standard API’s will ease the work of programming graphical user components
• Over time the developed API’s would form a library that includes specifications for routines, data, structures, object classes and many other variables
• API technology enables programmers to build hybrid analytics solutions
• The use of functional API has the potential to transform unstructured data into intelligence
11
UNCLASSIFIED
Machine Learning API’s
• Selection of the “Best Model”
• Statistical Based Models
• Biased based on model used to develop the API
• Limits of the model curve not infinite
• Human Genome Models
• Un-biased based on repetition of occurrence
• A lot like the human brain learns as things occur
12
UNCLASSIFIED
Embedded Stimulation
Virtual Environment
Virtual
Switch
SUT Embedded
Stimulator
Virtual Machine
SUT Embedded
Stimulator
Virtual Machine
• Single server platform • Multiple embedded
stimulators
• Multiple virtual environments
• Stimulate message/network traffic
• Between different virtual systems
P
H
Y
S
I
C
A
L
N
E
T
W
O
R
K
Instrumentation
IMS – Network Traffic
Stimulation
SUT
Embedded
Stimulator
SUT
Embedded
Stimulator
Controlled Environment
13
UNCLASSIFIED
Network Effects Emulation System
•Network Effects • Latency, Packet loss, Packet corruption,
Packet duplication, Packet reordering, Bandwidth limitation
• NE2S provides a Cyber testing capability by simulating a wide range of computer and network-based effects that can be centrally managed and controlled during a training and/or test event
• NE2S is a TRMC program managed by the Joint Coalition Warfare Office/J7. The program is at the TRL 6 maturity level
•Host Effects • Blue Screen, Erratic Mouse, Keyboard
Lock, Forced Reboot, Sound Damage, Monitor Disruptions, CPU Effects, Memory Effects
Tester
Distributed
Network
Access
Switch
`Mission Command
Platform
`Mission Command
Platform
Host Based
Effects
Generator
(Software)
Web browser
.HT
TP
S.
SSH with public
key authentication
CPU Effects
Mem Effects
Net Effects
NE2S Master Station
Tester
Starship / StarGEN
.ST
AX
.
14
UNCLASSIFIED
EPG
Fort Dix established network
Redstone Test Center
Joint Staff J6
(C4AD)
EPG-JBLM
1: Cyber & IA Test Facility
2: Realistic Battlefield Emitters
3: Test Events: NetWarrior, Manpack, Prophet
4: Large Range, controlled airspace, controlled
RF spectrum
5: Jamming & other EW.
Fort. Hood Operational Test Command
& Central Technical Support Facility
PEO-C3T & DCGS
JITC
Distributed Test Capability
15
UNCLASSIFIED
Leveraging Academia Partnerships
for Research Concepts
Network Topology / QoS FEATURE
EXTRACTIONFEATURE
SELECTIONDATA
MININGPREPROCESSING
INTERPRETATION& EVALUATION
Log -> Flow Transformed data Reduced data Patterns Knowledge
DATA ACQUISITION
Network packet trace (log)
Data Mining
• Characterize network behavior
via statistical analysis
• Detect significant
changes/events by comparing
adjacent time windows
• Identify root cause for changes
16
UNCLASSIFIED
• Identifying S&T Gaps for Next Generation Embedded Test Tools
• Verification and Validation of Embedded Test Tools
• Adapt Policy to allow Co-Use of DT/OT Embedded Tools for Life Cycle Testing
• Testing Community/PM’s agreement to embedded test tools for life cycle testing
• Embed Cross-Domain solutions
Challenges
17
Questions?
Electronic Proving Ground
Phone: (520)538-8043
Mr. Mark Pauls
18