WEBINAR
How To Get Monitoring Right For Streaming And Fast Data Systems Built With Spark,
Mesos, Akka, Cassandra and KafkaPaul Jasek, Senior Director of Global Solution Architects
Agenda1. Fast Data & Streaming Applications 2. The Challenges of Monitoring Fast Data Applications3. What To Look For In a Fast Data Application 4. Intelligent End-To-End Monitoring from Lightbend5. Live Demo 6. Questions
reactivemanifesto.org
Reactive Underpinnings: Fast Data and streaming applications often incorporate, or are based on, Reactive principles
• Real-timepersonalization
• Real-timedecision-making
• IoTdataprocessing
• Legacybatchprocessing
modernization
Growing Number Of Use Cases Across Industries
• Serveexistingcustomersbetterandreducechurn
• Attractnewonesanddrivegrowth
• Launchnewproductsmoreeasily
• Enternewmarketsmorequickly.
• Rapidly Evolving Ecosystem • Understanding the Data Pipeline • Dynamic Architectures • Intricately Interconnected • Distributed And Clustered
The Challenges of Monitoring Fast Data Applications
Apache Spark, As An Illustrative ExampleThe Challenges of Monitoring Fast Data Applications
Concern QuestionsToAsk
DataHealth(foragivenapplication)
• Throughput:isdataprocessingoccurringattheexpectedrate?• Latency:isdataprocessingoccurringwithintheexpectedtimeframe?• Error/quality:arethereproblemswiththedatabeingproduced?• Inputdata:areinputdatastreamsflowingintoSparkbehavingnormally?Forinstance,whatarethe
throughputratesforKafkatopicsfeedingintotheSparkjob?
DependencyHealth • Arethesystemsfeedinginputintothestormjob(suchasKafka)healthy?• Arethesystemsthattheapplicationisdependenton,suchasMemcacheorotherAPIendpoints,
healthy?
ServiceHealth • IstheSparkmasteroperatingnormally?Ifnot,engineeringwillbeunabletore-balanceworkloadsorrestartjobs.
ApplicationHealth • AretheapplicationKPIswithinnormaloperatingparameters?
TopologyHealth • ArethereresourcesassignedtothegivenSparktopology?• •AretheSparktasksandexecutorswell-distributedamongsttheSparkcluster?• •Aretheperformancecounters(emitted,failed,latency,etc.)forthegivenSparktopologynormal?
NodeSystemHealth • Arethekeysystemmetrics(load,CPU,memory,net-i/o,disk-i/o,diskfree)operatingnormally?
Can traditional monitoring tools help?
Why traditional monitoring tools won’t help you
• Builttomonitormonolithicapplications
• Canonlybeusedtoextractmetricsandtraceinformationbasedonasynchronousflow
• Notbuiltforasynchronousflows(i.e.inFastDataandstreamingapplications)
• Cannoteasilyhandlestreamingsystemsrunningondistributedclusters
• Deep Telemetry
• Domain Expertise
• Automated Discovery
• Real-Time Topology Visualization
• Intelligent, Rapid Troubleshooting
What users need to effectively monitor Fast Data and streaming applications
• Lightbend Monitoring takes a modern approach to instrumenting and
visualizing distributed streaming systems
• Helps users not just in production but also in development (so they can
build their applications right from Day 1)
• Shows the end-to-end status of applications, data frameworks, and the
associated infrastructure in a single view.
Intelligent, End-To-End Monitoring
• Deep Telemetry
• Domain Expertise
• Intelligent Anomaly
Detection
• Fine-Grained
Visibility, with Drill-
Down Capabilities
Data-Science Driven Anomaly Detection
• Automated Topology
Discovery
• Automatic Metric
Collection
• Real-Time Topology
Visualization
Automated Discovery, Configuration & Topology Visualization
• Single Pane of Glass
Visibility
• Rapid Root Cause
Analysis
• Reduced Mean-Time-
To-Repair (MTTR)
Intelligent, Rapid Troubleshooting
• Dramatically reduce the time and cost to identify and remediate issues across application life-cycle.
• Create happier, more satisfied customers – and lower churn
• Lower HW/infrastructure costs and reduce concerns about chargebacks & SLA penalties
• Deliver rapid time to value because everything you need for monitoring is packaged into an easy-to-use solution
Benefits for your business
On to the demo…
Upgrade your grey matter!Get the free O’Reilly book by Dr. Dean Wampler,
VP of Fast Data Engineering at Lightbend
bit.ly/lightbend-fast-data
End-To-End Monitoring For Your Fast Data And Streaming Applications From Lightbend
SET UP A 20-MIN DEMO