1© Cloudera, Inc. All rights reserved.
Hari Shreedharan, Software Engineer @ Cloudera
Committer/PMC Member, Apache Flume
Committer, Apache Sqoop
Contributor, Apache Spark
Author, Using Flume (O’Reilly)
Real Time Data Processing using Spark Streaming
2© Cloudera, Inc. All rights reserved.
Motivation for Real-Time Stream Processing
Data is being created at unprecedented rates
• Exponential data growth from mobile, web, social• Connected devices: 9B in 2012 to 50B by 2020• Over 1 trillion sensors by 2020• Datacenter IP traffic growing at CAGR of 25%
How can we harness it data in real-time?• Value can quickly degrade → capture value immediately• From reactive analysis to direct operational impact• Unlocks new competitive advantages• Requires a completely new approach...
3© Cloudera, Inc. All rights reserved.
Use Cases Across Industries
CreditIdentifyfraudulent transactions as soon as they occur.
TransportationDynamicRe-routingOf traffic orVehicle Fleet.
Retail• Dynamic InventoryManagement• Real-timeIn-storeOffers and recommendations
Consumer Internet &MobileOptimize userengagement basedon user’s currentbehavior.
HealthcareContinuouslymonitor patientvital stats and proactively identifyat-risk patients.
Manufacturing• Identifyequipmentfailures and react instantly• PerformProactivemaintenance.
SurveillanceIdentifythreatsand intrusionsIn real-time
Digital Advertising& MarketingOptimize and personalize content based on real-time information.
4© Cloudera, Inc. All rights reserved.
From Volume and Variety to Velocity
Present
Batch + Stream Processing
Time to Insight of Seconds
Big-Data = Volume + Variety
Big-Data = Volume + Variety + Velocity
PastPresent
Hadoop Ecosystem evolves as well…
Past
Big Data has evolved
Batch Processing
Time to insight of Hours
5© Cloudera, Inc. All rights reserved.
Key Components of Streaming Architectures
Data Ingestion & TransportationService
Real-Time Stream Processing Engine
Kafka Flume
System Management
Security
Data Management & Integration
Real-TimeData Serving
6© Cloudera, Inc. All rights reserved.
Canonical Stream Processing Architecture
Kafka
Data Ingest
App 1
App 2
.
.
.
Kafka Flume
HDFS HBase
Data Sources
7© Cloudera, Inc. All rights reserved.
Spark: Easy and Fast Big Data
•Easy to Develop•Rich APIs in Java, Scala, Python
• Interactive shell
•Fast to Run•General execution graphs
• In-memory storage
2-5× less codeUp to 10× faster on disk,
100× in memory
8© Cloudera, Inc. All rights reserved.
Spark Architecture
Driver
Worker
Worker
Worker
Data
RAM
Data
RAM
Data
RAM
9© Cloudera, Inc. All rights reserved.
RDDs
RDD = Resilient Distributed Datasets• Immutable representation of data• Operations on one RDD creates a new one• Memory caching layer that stores data in a distributed, fault-tolerant cache• Created by parallel transformations on data in stable storage• Lazy materialization
Two observations:a. Can fall back to disk when data-set does not fit in memoryb. Provides fault-tolerance through concept of lineage
10© Cloudera, Inc. All rights reserved.
Spark Streaming
Extension of Apache Spark’s Core API, for Stream Processing.
The Framework Provides
Fault Tolerance
Scalability
High-Throughput
11© Cloudera, Inc. All rights reserved.
Spark Streaming
• Incoming data represented as Discretized Streams (DStreams)
• Stream is broken down into micro-batches
• Each micro-batch is an RDD – can share code between batch and streaming
12© Cloudera, Inc. All rights reserved.
val tweets = ssc.twitterStream()
val hashTags = tweets.flatMap (status => getTags(status))
hashTags.saveAsHadoopFiles("hdfs://...")
flatMap flatMap flatMap
save save save
batch @ t+1batch @ t batch @ t+2tweets DStream
hashTags DStream
Stream composed of small (1-10s) batch
computations
“Micro-batch” Architecture
13© Cloudera, Inc. All rights reserved.
Use DStreams for Windowing Functions
14© Cloudera, Inc. All rights reserved.
Spark Streaming
• Runs as a Spark job
• YARN or standalone for scheduling
• YARN has KDC integration
• Use the same code for real-time Spark Streaming and for batch Spark jobs.
• Integrates natively with messaging systems such as Flume, Kafka, Zero MQ….
• Easy to write “Receivers” for custom messaging systems.
15© Cloudera, Inc. All rights reserved.
Sharing Code between Batch and Streaming
def filterErrors (rdd: RDD[String]): RDD[String] = {
rdd.filter(s => s.contains(“ERROR”))
}
Library that filters “ERRORS”
• Streaming generates RDDs periodically
• Any code that operates on RDDs can therefore be used in streaming as well
16© Cloudera, Inc. All rights reserved.
Sharing Code between Batch and Streaming
val lines = sc.textFile(…)
val filtered = filterErrors(lines)
filtered.saveAsTextFile(...)
Spark:
val dStream = FlumeUtils.createStream(ssc, "34.23.46.22", 4435)
val filtered = dStream.foreachRDD((rdd: RDD[String], time: Time) => {
filterErrors(rdd)
}))
filtered.saveAsTextFiles(…)
Spark Streaming:
17© Cloudera, Inc. All rights reserved.
Reliability
• Received data automatically persisted to HDFS Write Ahead Log to prevent data loss
• set spark.streaming.receiver.writeAheadLog.enable=true in spark conf
• When AM dies, the application is restarted by YARN
• Received, ack-ed and unprocessed data replayed from WAL (data that made it into blocks)
• Reliable Receivers can replay data from the original source, if required
• Un-acked data replayed from source.
• Kafka, Flume receivers bundled with Spark are examples
• Reliable Receivers + WAL = No data loss on driver or receiver failure!
18© Cloudera, Inc. All rights reserved.
Kafka Connectors
• Reliable Kafka DStream
• Stores received data to Write Ahead Log on HDFS for replay
• No data loss
• Stable and supported!
• Direct Kafka DStream
• Uses low level API to pull data from Kafka
• Replays from Kafka on driver failure
• No data loss
• Experimental
19© Cloudera, Inc. All rights reserved.
Flume Connector
• Flume Polling DStream
• Use Spark sink from Maven to Flume’s plugin directory
• Flume Polling Receiver polls the sink to receive data
• Replays received data from WAL on HDFS
• No data loss
• Stable and Supported!
20© Cloudera, Inc. All rights reserved.
Spark Streaming Use-Cases
• Real-time dashboards
• Show approximate results in real-time
• Reconcile periodically with source-of-truth using Spark
• Joins of multiple streams
• Time-based or count-based “windows”
• Combine multiple sources of input to produce composite data
• Re-use RDDs created by Streaming in other Spark jobs.
21© Cloudera, Inc. All rights reserved.
What is coming?
• Run on Secure YARN for more than 7 days!
• Better Monitoring and alerting
• Batch-level and task-level monitoring
• SQL on Streaming
• Run SQL-like queries on top of Streaming (medium – long term)
• Python!
• Limited support coming in Spark 1.3
22© Cloudera, Inc. All rights reserved.
Current Spark project status
• 400+ contributors and 50+ companies contributing
• Includes: Databricks, Cloudera, Intel, Yahoo! etc
• Dozens of production deployments
• Spark Streaming Survived Netflix Chaos Monkey – production ready!
• Included in CDH!
23© Cloudera, Inc. All rights reserved.
More Info..
• CDH Docs: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh5ig_spark_installation.html
• Cloudera Blog: http://blog.cloudera.com/blog/category/spark/
• Apache Spark homepage: http://spark.apache.org/
• Github: https://github.com/apache/spark
24© Cloudera, Inc. All rights reserved.
Thank you
15% Discount Code for Cloudera Training PNWCUG_15university.cloudera.com