DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
An Enterprise Approach to Evaluating Complex Systems Using
Big Data Analytics
Ryan NormanTRMC Initiative Lead for Big Data & Knowledge Management
PM, TENA SDA [email protected]
22
Fundamental Analysis Methodologies
So far, we have identified seven fundamental questions relevant to our domain that can be addressed by Big Data Analytic Techniques:
• Anomaly Detection – Did something go wrong?• Causality Detection – What contributed to it?• Trend Analysis – What’s happening over time?• Predicting Equipment Function and Failure – When will
something go wrong?• Regression Analysis – How is today’s data different than the
past?• Data Set Comparison – Are these two large data sets equivalent?• Pattern Recognition – Are there any recognizable patterns in the
data?
These are not new or unique to T&E. So what’s the problem?
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015 3
Source: IDC’s Digital Universe Study, sponsored by EMC, Dec 2012
More information over the next two years than in the entire history of mankind!
Name (Symbol) Value
2020 40,000 EB
2005 130 EB
2010 1,250 EB
2015 8,000 EB EX
AB
YTE
SWorldwide Exponential
Growth of Data
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
Test & Evaluation Growth of Data
• Larger Test Footprints–4-on-4 test flights (more systems per test)–Much faster weapons systems–Geographic separation not as effective as it
used to be
• Demand for Shorter Acquisition Cycles
–More concurrent testing –More real-time analysis
• Increased System-of-Systems Test Complexity
–“Five Futures” (EW, UAV, NCO/W, DE, Hypersonics)
– Integrated fleet (F-18E/F, E-18G, F-35, SM VI, UAV)
– “Swarming” UAVs
Increased System Complexity
Integratedweapon system(1-20 Mbps ea.)
Integratedavionics(2-10 Mbps)
Separationvideo(1-8 Mbps)
Cockpitvideo(1-5 Mbps)
Integratedcommunications(0.5-2 Mbps)
Flight test transducers(0.5-10 Mbps)
Multi-spectralsensors(1-12 Mbps)
Integrated self-defense system(0.5-2 Mbps)
Total Throughput: 7.5Mbps – 70Mbps+
4
T&E Mission: Acquire data and discern into knowledge
55
Big Data / Knowledge Management (KM)Challenges & Needs
T&E Infrastructure Challenges:– How do we conduct T&E of increasingly complex, data-driven systems?– How do we enable more efficient & continuous system evaluation?
Need: A DoD-wide KM capability for T&E to help achieve better acquisition outcomes and reduce costs
– Trusted processes across government and industry that identify problems sooner rather than later
– Accessibility of knowledge & data to legitimate users– Discoverability of knowledge & data obtained
over time– Availability of knowledge through common tools &
technologies – including DoD T&E cloud solutions– Leverages proven Industry techniques / practices
Big Data Analytics depends on effective Knowledge Management
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
The TRMC “Blueprint”:Putting Test Capabilities on the DoD Map
Risk mitigation needsTechnology shortfalls
Risk mitigation solutions Advanced development
Capabilities
Service Modernization and Improvement Programs
Acquisition Programs and Advanced Concept Technology Demonstrations
T&E Multi-Service/Agency
Capabilities
DoD Corporate Distributed Test
Capability
TRMC Joint Investment
Programs
Transition
Requirements
Strategic Plan for DoD T&E Resources
Annual T&E Budget
Certification
(6.3 Funding) (6.6 Funding)(6.4 Activity)
DT&E / TRMC Annual Report
Defense Strategic Guidance
Service T&E Needs and Solutions Process
Acquisition Process
6
7UNCLASSIFIED – DISTRIBUTION STATEMENT A
Realizing Improved DoD T&E Knowledge Management
1. Understand and Document T&E challenges & needs– (FY12) Completed Data Management for Distributed Testing (DM-DT) Study
− Result: Developed functional requirements for T&E enterprise distributed Data Management
– (FY13) Comprehensive Review of T&E Infrastructure report published− Key Recommendation: Use DoD cloud solution for T&E data− Key Recommendation: USD(AT&L) establish a DoD-wide KM capability for T&E to help achieve
better acquisition outcomes and reduce costs
2. Execute proofs of concept that inform an enterprise approach to T&E Knowledge Management– (FY15-18) Joint Strike Fighter Knowledge Management (JSF-KM) project
− Goal: Assess KM technologies and methodologies in support of an existing acquisition program
– (FY15-17) Collected Operational Data Analytics for Continuous Test & Evaluation (CODAC-TE) project− Goal: Apply KM technologies and methodologies across the lifecycle
3. Develop investment plan that achieves strategic objectives:– Integrate T&E infrastructure into cohesive Knowledge Management enterprise– Modernize T&E practices & processes to leverage Big Data analytics techniques– Apply Big Data analytics tools & techniques to the T&E mission space
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
Existing Range Computing and Storage
Example Framework View: Big Data Software Architecture
Structured Database Unstructured/Semi-Structured Database (Hadoop)
Structured Data Engine Unstructured Data Engine
Query Engine – Federated access for both Structured and Unstructured Data
Data Analysis Packages User-Defined Analytic Plugins
Massively Parallel Tiered Computing, Storage, and Network InfrastructureAt Multiple Independent Levels of Security
Extract-Transform-
Load
Data Sources
Analytic Services
Big Data Visualization
UC S TS SAP SAR
Security
Existing Range DatabasesFlat Files
Raw Files
Setup, Configure, and Manage
Policies Security Define MetadataPrioritization
Streams
Micro-batch
Mega-batch
Parallel
Verify
Transform
Add Metadata
Index
Warehouse
Configuration
Metadata Replication
Build Queries
Quick-Look Real-Time Continuous
2D/3D/Anim
Display Reports
Design Reports
CustomizedDisplays
Display Alerts
User Interface
Authenticate
Authorize
AccessControlEnforcePoliciesEnforce
WorkflowThreat
DetectionIntrusionDetection
ActiveDefenses
Working Sets Tables
Encryption
Audit
Alerts
Load Balancing
Fault/Recovery
MILS SecureCloud
Statistics
Key-Value Store
DistributedFile System
Generate ReportsAI Tools Simulation
Analysis ToolsAlerting Scheduling/Automation Legacy Tools
SQL Services
Remote DataReplication
Big Data Analysis PackagesAnomaly Detection Trend AnalysisCausality Detection Regression AnalysisGround Truth Comparison Pattern Recognition
Filter Sort Summarize Parallelize Optimize
Machine LearningData Mining
CustomizedUIs
Structured
Unstructured
Audio/Video
Schema
ComputingResources
ComputingResources
CreateAutomated
Products
Abstraction Layer (Virtualization)
Hypervisor
Virtualized Legacy Tools
Infrastructure as a Service Platform as a Service Software as a Service
Virtualized New Tools
Simulation as a Service
Graph-Based
Schema
Audio/Video Analysis
NewDatabases
Provisioning
StreamingScripting
COTS/GOTS SoftwareNew Hardware/Network
TRMC-Developed Software
Existing Range HW/SW
Applications
Resource Mgmt
VM Library
Cloud
License
Customization
Data Services
Organization
Core
OperationsShare
Serve
MessagingMetadata
Store Retrieve
VersioningTaggingPublish/Subscribe Crawl/Index
Transfer
Transform
Catalog
Search
Verify
AdministrativeCOO/DREnforce Policies Archive ToolsDB Admin Config Mgmt
Sync Data/Video
Spatio-temporal
Ontologies
MPP Programming and Execution Engine
C/R/U/D Consistency
ExistingComputers
PipelineWorkflowRange
Protocols
TENA Data Lifecycle
Workflow CreateSoftware
IDE
SDK
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
JSF T&E Infrastructure Needs Addressed by JSF-KM Project
1. Data Warehousing: Flight test data should be stored in a government facility to expedite data access & discovery
2. Data Ingest: Current DART Pod test data ingest is too slow to meet multi-ship quick-look and quick-turn requirements– examples: 2 on 2; 4 turn 2; 4 on 4 turn 4 on 4
3. Data Access: Test data should be available for quick-look analysis during mission debrief to inform decision making
4. Video: DART Pod video should be available for quick-look analysis during mission debrief to inform decision making
5. Big Data Analytics: Analysis capabilities need to proactively identify “unknown unknowns” and other anomalies impossible for a human to discern
9
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
User
Eglin
PaxRiver
Joint Strike Fighter Knowledge Management (JSF-KM) Test Concept
• DT & OT data storage in government facilities
• Collect / Store more precise data during OT
• Search / Analyze Edwards & Nellis data from any secure location
• Bring enhanced JMETC infrastructure to JSF T&E
• Apply commercial Big Data and Knowledge Management tools to DoD requirements
• Knowledge shared acrossJSF DT / OT T&E locations
• Scalable to other JSF T&E locations
10
JPO
Potential Expansion
UNCLASSIFIED - Distribution A
JSF-KM Improvements to Existing T&E Capabilities
11
DT Today
OT Today
With JSF-KM
Parallel Data
Ingest
30 minutes (multiple aircraft)
Raw Data Available
Video/Data at Post-Mission Debrief Big Data
Analytics
Govt. Analyst Data Request
Analysis
Note: Numbers reflect single 2 hour flight mission
Data Ingest
Raw Data Available Govt. Analyst
Data Request Analysis
2 hours (per aircraft) 1 day 1 week
30 seconds
Data Ingest
Raw Data Available Govt. Analyst
Data RequestAnalysis
1-2 hours (per aircraft) 10 minutes 4-5 hours
Data Ready for Use @ (Govt)
30 seconds
90 minutes
Data Ready for Use @ LM
>20 weeks of data available online
Data Ready for Use @ (Govt)3 weeks of data available online
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
JSF-KM FY15 Success Stories
• Resolved test data time correlation issues– Time stamps in data files found to be corrupted post-mission– JSF-KM analysis tools were able to correct the time correlation issue– Without JSF-KM, at least five missions would have been re-flown
• Video available during post-mission debrief due to JSF-KM data ingest improvements from DART Pod
– Existing tools could not process video in time to support post-mission de-brief– Without JSF-KM, there would be no flight video during post-mission debrief
• Discovery of avionics box issue during first night mission– Pilot and Analyst discovered problem from video data available 30 minutes after
landing– Avionics Box was replaced before another mission was flown– Without JSF-KM, problem would not have been discovered for several days
Return on Investment has been realized before deploying any Big Data analytics capabilities
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
JSF-KM FY16 Success Stories
• Reduced data profile time from 5+ hours to 47 seconds per Query– Big Data tool enabled massive improvement to data profile generation– Without JSF-KM it would still take 5+ hours to perform data profile data runs
• Identified 2 engines that consistently performed differently than the other 9 engines within the 92 data sets analyzed– Discovered a faulty/noisy ground sensor and an unknown pattern within a known sampling rate
abnormality– Without JSF-KM these anomalies may have never been discovered
• Identified issue with ground sensor– Found anomalous points and pattern within inconsistent sensor data sampling rates– Without JSF-KM these anomalies may have never been discovered
• Identified flights which experienced propulsion component failure– During a blind analysis of 1,392 flights of propulsion data, JSF-KM data scientist was able to
identify 7 of 10 flights with JSF analyst known engine issues – Led to creation of a predictive model* for identifying future failures (*model validation pending)
– Without JSF-KM this predictive model may not have been generated
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
JSF-KM FY17 Success Stories
• Big data analytics system tested in lab using unclassified Propulsion data from PAX– KM team used data from ~6200 Power on Cycles (PUCs) to stress test the infrastructure and
fine tune the KM system– Analytics appliance deployed to Pax River in Aug 2017 – First Hands on Training conducted at Pax River Aug 28-30, follow-up event Spring 2018– First Pax River system administrator training conducted Nov 2nd
• Hands on Training example reduced 9 hour MATLAB vibration analysis process to 23 ms– Legacy MATLAB process used data from 6 sensors to calculate the avg. vibration energy per
second, cumulative vibration energy, and times of exceeded threshold values for each flight. – Grouping function in GreenPlum and standard math functions with co-located data within the
GPDB allowed for less file IO and dramatic speedup of analytics process. – New system ran same analysis process across all flights and engines in less than 5 minutes– Drastically reduced routine MATLAB analysis process at PAX prior to operational deployment– Patuxent River leadership already identifying other airframes which could use the system
• Patuxent River KM system deployed Aug 2017, Operational Nov 2017• RMF IA ATO received, Nellis KM system deployed Nov 2017*, Operational Dec 2017*• Remote demo capability created in order to showcase KM system / tools
* Planned
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
Collected Operational Data Analytics for Continuous Test & Evaluation (CODAC-TE) Proof of Concept
• Background: US Army Aberdeen Test Center (ATC) has ~30TB of underutilized data – including 20TB of in-theater operational data
• Goal: Utilize Big Data analytics across multi-commodity DT, OT, and in-theater system data to discover “unknown unknowns” for current and future Army systems
• Use Cases: Mine-Resistant Ambush Protected (MRAP) Theater Data; Camouflage Effectiveness
• Leverages High-Performance Computing Major Shared Resource Center and ATC expertise
Challenges being addressed:• Insufficient data science expertise• Current analytical systems inadequate for today’s data volume and velocity• Lacking tools and techniques for discovering unknown unknowns and conducting
complex trends analyses
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
CODAC-TE Success StoriesPhase 1
• In-theater MRAP data collected, aggregated, and analyzed as resolution, complexity, and processing requirements increased– Existing data analysis platforms and tools cannot process large enough datasets in
a reasonable time or at all– Without CODAC-TE, ATC would not be able to scale to meet growing data sizes
and processing requirements• Incorporated industry best practices for advanced data tools and
analysis– Approach allowed advanced data analysis tools to be available under rapid
development schedule– Without CODAC-TE, using big data analytics would be cost & schedule prohibitive
• Increased discoverability of multi-commodity life-cycle data and knowledge – Phase 1 created an analysis platform capable of scaling to support breadth of data
over time– Without CODAC-TE, available analysis tools could not load and process data
across commodity areas or across all events in a system's life-cycle
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
CODAC-TE Technical Accomplishments
Phase 1 System Past Systems
Size 50 billion rows 1-2 billion rows
Performance 500K in 10 seconds Hours or days for query completion
Scalability Superlinear scalability Not scalable
Usability Single homogeneous data set
Unable to query across data sets
Timeline Direct query Must batch process before query
17
Developed a platform to store and query data at large scale
Phase 1 completed on schedule and under budget
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
Big Data Initiative Summary• TRMC is acting upon the KM recommendations from the
Comprehensive Review of T&E Infrastructure. Strategic Goals:– Integrate T&E infrastructure into cohesive Knowledge Management enterprise– Modernize T&E practices & processes to leverage Big Data analytics techniques– Apply Big Data analytics tools & techniques to the T&E mission space
• TRMC-funded proofs of concept will deliver proven capabilities– Enable Big Data analytics during JSF T&E– Improve transfer of knowledge between fielded and next-gen systems– Inform T&E investment plan that advises future infrastructure, process, and
workforce decision-making• Improved T&E KM will help achieve better acquisition outcomes and
reduce costs– Identify & Diagnose problems sooner and continuously– Inform acquisition decisions through larger knowledge base– Achieve T&E infrastructure efficiencies
18
DRAFT UNCLASSIFIED – DISTRIBUTION STATEMENT AReference Number 15-S-2335; 08-14-2015
Questions?
19
Ryan NormanTRMC Initiative Lead for Big Data & Knowledge Management
PM, TENA [email protected]