Post on 09-Jan-2017
transcript
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
MACH-5 INSIGHT℠Orion High Data Rate Use Case
October 1st 2015
Haden Landhaden.land@lmco.com
Jason Lovelandjason.c.loveland@lmco.com
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Agenda Orion Program Overview High Data Rate Ingest Use Case MACH-5 INSIGHT℠ Platform SFDU and Protobuf HBase bulk loading using Spark
Streaming TRACE Visualization
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Orion OverviewOrion Multi-Purpose Crew Vehicle, NASA’s first spacecraft
designed for long-duration, human-rated deep space exploration. Orion will transport humans to interplanetary destinations beyond low Earth orbit, such as asteroids, the moon and eventually Mars, and return them safely back to Earth.
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
The Orion program will enable NASA to:
Carry out a robust human and robotic exploration program that is both sustainable and affordable
Extend human presence across the solar system to our planets, asteroids and other destinations
Develop the innovative technology, knowledge, and infrastructure needed to support more challenging human space exploration missions
Promote international and commercial participation in exploration to further advance U.S. scientific, security and economic interests
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Exploration Flight Test-1 (EFT-1) Uncrewed Orion launched from Cape Canaveral Air Force Base on a Delta IV Heavy
rocket Orbited the Earth twice, traveling approximately 3,600 miles above the Earth’s surface
• 15 times farther than the International Space Station. • Generated more than 80% of the return velocity experienced during a reentry from the moon, which allows engineers to model
expected reentries from future missions in deep space. Orion travelled twice through the Van Allen Radiation Belt, a dense radiation field of
highly energetic charged particles surrounding Earth• This exposure will help the team understand and mitigate radiation exposure, and to develop protective solutions before the first
crewed mission. At the conclusion of EFT-1, Orion reentered Earth’s atmosphere at 20,000 mph and
enduring temperatures of up to 4,000° Fahrenheit After the parachutes deployed, the spacecraft slowed to 20 mph splashing down in the
Pacific Ocean, near the Baja Peninsula.
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
How Big Is Big On Orion?• How much telemetry is there, anyway? Is it “big
data”?• Orion is transitioning to Big Data• LOTS! PB+. On board, three 1 Gb networks are kept
full• Volume, velocity, variety
• What is Spacecraft Telemetry?• Sensor measurements to monitor everything• Microsecond Timestamps• 12 discrete time sources• More complex time series• Human Space Flight requires fault tolerance
• What produces telemetry?• For Orion, the telemetry is produced in a variety of
simulation and test environments which includes at least 7 differ labs across the US.
• How many items are there to measure in Orion?• The XTCE representing the flight software telemetry
has over 3,000,000 elements, of which over 300,000 are related to currently taken measurements. There are many more when considering the ground system and other subsystems.
• How is telemetry analyzed today?• Traditionally, the responsible engineer
watches a handful of the telemetry indicators on the real time monitors as a test progresses, or in review. Certain behavior and reporting will result in further analysis on a few other measurands.
• Most values of most telemetry measurements are ignored if they are not out of limits
• Specific studies are done on some measurements in historical context, usually after detection of anomalous behavior, to determine if the behavior has been observed in the past.
• Automation has been applied to the problems of automatically collecting data, automatically running scripted tests, and automatically detecting out-of-limit values
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Test Labs
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
How Big Is Big for Orion?Orion’s next mission (EM-1) in 2018 2 weeks instead of 4 hours 4 times as many computers Twice as many instruments Subsystems that support Human Flight!
EM-1 will be orders of magnitude bigger and more complex
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Big Data Analytics Organization Technologists/Platform Developers
• Networking• Disk storage strategies• Modern languages and techniques: Scala, R• Big Data tools: Hadoop ecosystem, NoSQL databases, Stream and Batch
Processing Analytics /Data Scientist(Ponderers)
• Tools of the trade• Visualization & Communication• Data Science
StatisticsDesign of experimentsStatistical tools: SAS, SPSS, R, MATLABModelingData MiningLinear algebraData cleansing, sampling, warehousing
Subject Matter Experts• System and Subsystem Experts• Program Tools Experts
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
High Data Rate Problem DefinitionHorizontally scalable, fault tolerant prototype for processing High Data Rate telemetry from Orion Test Rigs including ingest, processing, transform, storage, advanced analytics
Satisfy Data Rate: ~300MB/s (97MB/s X 3 planes) Provide common data wire protocol that is:
Flexible, expressive and compact Programming language agnostic
Provide system to scale telemetry ingest from Test Rigs to Mach-5 within technical performance metrics
Provide solution that is continuously measurable and monitorable Provide Advanced Analytics to identify anomalies on ingest
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
MACH-5 INSIGHT℠ Architecture
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
MACH-5 INSIGHT℠ Benefits to Orion Enables Orion to teach us lessons about it’s own experience
leveraging data to tell the story Reduces time to develop and test Orion by identifying failures
early! Uses data points from one mission to help predict if mission
critical components will succeed or fail in specific future mission circumstances
Permits queries to be retrieved across historical batch and live streaming views within seconds
Streams historical data back as if it were live Overlays data across tests and missions Stores and processes much larger data sets than any other space
craft in historyScales to meet the needs of future missions to keep the conversation all the way to Mars
Data Reader/Simulator IngestPacket
Measurands (GPBs)
Kafka Message
Bus
Packet Measurands
(GPBs)Deduplica
tion (Spark)
HBase Writer(Spark)
mach5-sample ObjSplitter +
Decom (GDS)
C++ client Reads Packets and
Decommutates
Tlm Data
Packet Measurands GPB File
(represents a Packet(s) and contains
decommutated measurands)
Header Metadataapid:seqctr:time: value1
…..
apid:seqctr:time: valueN
mach5-sample (Spark)
Packet Measurands
(GPBs)
Lockheed Martin Proprietary Information
StorageAnalytics
HDFS
HFiles (HBase-RDD)
Mach-5 Data Ingest for Orion
HBase
Web/UITomcatGlassfish
Etc.
TraceFOSS
widgets
Aggregation
(Spark)
Alerting(Spark)
Limit Checking(Spark)
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
• Leverage space standards and wrapping modern data serialization technologies
• SFDU (Standard Formatted Data Unit)• CCSDS Standard
• Consultative Committee for Space Data Systems – Founded 1982
• Delimiting data and providing an internal index
• Transport and archival format• Provides ease of integration with
space systems• Should be usable 25+ years from
now• SFDU wraps serialized data, in our
case Protobuf
SFDU and ProtobufCCSD Z 001L
LMCO U 009L
V V
LMCO S 010
LMCO I 010
L
L
V
V
Header
Body
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Wire Protocol: Serialization Protocol Buffers• Compact, compressible• Many language bindings: Java, C++, Python, etc…• Self describing schema, auto-generates code• Compared to other Serialization libraries (i.e. Avro), allows for
greater data type expression. Ex: Can natively support unsigned Long and Unsigned Integer
on C++ side Allows for cleaner generated code bindings
• Widely used within the Hadoop ecosystem• Widely used within Lockheed Martin telemetry processing
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Kafka Benefits to Orion
Enables Orion to scale data architecture linearly on FOSS
Allows for multiple test rigs to simultaneously write data to be ingested at high data rates
Provides ability for analytics processing to independently pull and push data
Provides replay capabilities for guaranteed, exactly once processing semantics per partition
Provides fault tolerance and replication
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Spark Benefits to Orion Enables Orion to scale data architecture linearly on FOSS Real-time Analytics reduce time to develop and test the Orion by
identifying anomalies early! Allows for analytics to be run in batch and streaming mode Analytics libraries that scale Integration with Kafka to independently pull and push data Runs on YARN to reduce complexity of managing resources Provides guaranteed, exactly once processing semantics per
partition using Kafka Direct Connect Provides fault tolerance and check pointing Integration with HBase (SparkOnHBase)
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Using Spark to Bulk Load into HBasePull Data off of Kafka and Transform from SFDU to Protobuf HBase Column Format•Data bucketed to be aligned to appropriate Region. (HBase table is pre-split). Handled by Spark on Yarn
Sort Rows and Partitioning by HBase Regions•Sort happens locally•HFiles are guaranteed to contain all rows for a specific region•This avoids CPU intensive work by HBase
Construct HFiles Write to HDFS•HFiles are written to HDFS•Writing HFiles is much faster than HBase puts
Bulk Load into HBase•Bulk Load is called on each HFile
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
HBase Benefits to Orion
Data arrives in random order and HBase automatically sorts data by time
Enables Orion to scale data architecture linearly on FOSS
Provides random access to all telemetry with low latency access
Provides data store to perform batch processing using Mapreduce and Spark
Custom filters
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Benefits of Single Column HBase schema Single-Column Schema
• Single Colum is packed with every piece of metadata for a row. Numerous serialization formats can be used: json, protobuf, avro, etc…
Benefits• Compact, greatly decreases row footprint• Higher write throughput. Under the hood HBase performs a Put for each column of a row. We
go from ‘N’ puts in a multi column schema to 1• Caching for queries
Detractors• Increased overhead to update the column• If your column schema has the potential to change, could end up with multiple versions of the
serialized column value• Slower reads because requires that all of the data be returned and deserialized• Requires a custom HBase filter to filter on individual members of the column value• Integration with 3rd Party BI tools and the rest of the Hadoop ecosystem
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Processing and AnalyticsDetect abnormal activity during a test and notify operators early as possible. (Tests can run for weeks!) Provide limit checking on individual measurements Combine measurements to compute higher level
understanding (i.e. momentum calculation) Aggregation and statistical summaries Requirements validation Supervised and Unsupervised learning
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
TRACE View of MACH-5 INSIGHT℠ Stored Data
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation
Next Steps Analytics! We need the help of the
community to develop effective data platforms and analytics that can scale with Orion missions.
© 2015 Lockheed Martin CorporationLOCKHEED MARTIN and LOCKHEED trademarks used throughout are registered trademarks in
the U.S. Patent and Trademark Office owned by Lockheed Martin Corporation