Apache SparkThe next Generation Cluster Computing
Ivan Lozić, 04/25/2017
Ivan Lozić, software engineer & entrepreneur
Scala & Spark, C#, Node.js, Swift
Web page: www.deegloo.comE-Mail: [email protected]
LinkedIn: https://www.linkedin.com/in/ilozic/
Zagreb, Croatia
Contents
● Apache Spark and its relation to Hadoop MapReduce● What makes Apache Spark run fast● How to use Spark rich API to build batch ETL jobs● Streaming capabilities● Structured streaming
3
Apache Hadoop
44
Apache Hadoop
● Open Source framework for distributed storage and processing● Origins are in the project “Nutch” back in 2002 (Cutting, Cafarella)● 2006. Yahoo! Created Hadoop based on GFS and MapReduce ● Based on MapReduce programming model● Fundamental assumption - all the modules are built to handle
hardware failures automatically● Clusters built of commodity hardware
5
6
Apache Spark
77
Motivation
● Hardware - CPU compute bottleneck● Users - democratise access to data and improve usability● Applications - necessity to build near real time big data applications
8
Apache Spark
● Open source fast and expressive cluster computing framework designed for Big data analytics
● Compatible with Apache Hadoop● Developed at UC Berkley’s AMP Lab 2009. and donated to the Apache
Software Foundation in 2013.● Original author - Matei Zaharia● Databricks inc. - company behind Apache Spark
9
Apache Spark
● General distributed computing engine which unifies:○ SQL and DataFrames ○ Real-time streaming (Spark streaming)○ Machine learning (SparkML/MLLib)○ Graph processing (GraphX)
10
Apache Spark
● Runs everywhere - standalone, EC2, Hadoop YARN, Apache Mesos● Reads and writes from/to:
○ File/Directory○ HDFS/S3○ JDBC○ JSON○ CSV○ Parquet○ Cassandra, HBase, ...
11
Apache Spark - architecture
12
source: Databricks
Word count - MapReduce vs Spark
13
package org.myorg; import java.io.IOException; import java.util.*; import org.apache.hadoop.fs.Path; import org.apache.hadoop.conf.*; import org.apache.hadoop.io.*; import org.apache.hadoop.mapred.*; import org.apache.hadoop.util.*; public class WordCount { public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); output.collect(word, one); } } } public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterator values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { int sum = 0; while (values.hasNext()) { sum += values.next().get(); } output.collect(key, new IntWritable(sum)); } } public static void main(String[] args) throws Exception { JobConf conf = new JobConf(WordCount.class); conf.setJobName("wordcount"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(IntWritable.class); conf.setMapperClass(Map.class); conf.setCombinerClass(Reduce.class); conf.setReducerClass(Reduce.class); conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); FileInputFormat.setInputPaths(conf, new Path(args[0])); FileOutputFormat.setOutputPath(conf, new Path(args[1])); JobClient.runJob(conf); } }
val file = spark.textFile("hdfs://...")val counts = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)counts.saveAsTextFile("hdfs://...")
Hadoop ecosystem
14
Who uses Apache Spark?
15
Core data abstractions
1616
Resilient Distributed Dataset
● RDDs are partitioned collections of objects - building blocks of Spark● Immutable and provide fault tolerant computation● Two types of operations:
1. Transformations - map, reduce, sort, filter, groupBy, ...2. Actions - collect, count, take, first, foreach, saveToCassandra, ...
17
RDD
● Types of operations are based on Scala collection API● Transformations are lazily evaluated DAG (Directed Acyclic Graph)
constituents● Actions invoke DAG creation and actual computation
18
RDD
19
Data shuffling
● Sending data over the network● Slow - should be minimized as much as possible!● Typical example - groupByKey (slow) vs reduceByKey (faster)
20
RDD - the problems
● They express the how better than what● Operations and data type in clojure are black box for Spark - Spark
cannot make optimizations
21
val category = spark.sparkContext.textFile("/data/SFPD_Incidents_2003.csv") .map(line => line.split(byCommaButNotUnderQuotes)(1)) .filter(cat => cat != "Category")
Structure(Structured APIs)
22
SparkSQL
23
● Originally named “Shark” - to enable HiveQL queries● As of Spark 2.0 - SQL 2003 support
category.toDF("categoryName").createOrReplaceTempView("category")
spark.sql(""" SELECT categoryName, count(*) AS Count FROM category GROUP BY categoryName ORDER BY 2 DESC""").show(5)
DataFrame
● Higher level abstraction (DSL) to manipulate with data● Distributed collection of rows organized into named columns● Modeled after Pandas DataFrame● DataFrame has schema (something RDD is missing)
24
val categoryDF = category.toDF("categoryName")
categoryDF .groupBy("categoryName") .count() .orderBy($"Count".desc) .show(5)
DataFrame
25
Structured APIs error-check comparison
26source: Databricks
Dataset
● Extension to DataFrame● Type-safe● DataFrame = Dataset[Row]
27
case class Incident(Category: String, DayOfWeek: String)
val incidents = spark .read .option("header", "true") .csv("/data/SFPD_Incidents_2003.csv") .select("Category", "DayOfWeek") .as[Incident]
val days = Array("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")
val histogram = incidents.groupByKey(_.Category).mapGroups { case (category, daysOfWeek) => { val buckets = new Array[Int](7) daysOfWeek.map(_.DayOfWeek).foreach { dow => buckets(days.indexOf(dow)) += 1 } (category, buckets) }}
What makes Spark fast?
2828
In memory computation
● Fault tolerance is achieved by using HDFS● Easy possible to spend 90% of time in Disk I/O only
29
iter. 1
input
iter. 2 ...HDFS read HDFS write HDFS read HDFS write HDFS read
● Fault tolerance is provided by building lineage of transformations● Data is not being replicated
iter. 1
input
iter. 2 ...
Catalyst - query optimizer
30
source: Databricks
● Applies transformations to convert unoptimized to optimized query plan
Project Tungsten
● Improve Spark execution memory and CPU efficiency by:○ Performing explicit memory management instead of relying on JVM objects (Dataset
encoders)○ Generating code on the fly to fuse multiple operators into one (Whole stage codegen)○ Introducing cache-aware computation○ In-memory columnar format
● Bringing Spark closer to the bare metal
31
Dataset encoders
● Encoders translate between domain objects and Spark's internal format
32
source: Databricks
Dataset encoders
● Encoders bridge objects with data sources
33
{ "Category": "THEFT", "IncidntNum": "150060275", "DayOfWeek": "Saturday"}
case class Incident(IncidntNum: Int, Category: String, DayOfWeek: String)
Dataset benchmark
Space efficiency
34
source: Databricks
Dataset benchmark
Serialization/deserialization performance
35
source: Databricks
Whole stage codegen
● Fuse the operators together● Generate code on the fly● The idea: generate specialized code as if it was written manually to be
fast
Result: Spark 2.0 is 10x faster than Spark 1.6
36
Whole stage codegen
37
SELECT COUNT(*) FROM store_sales WHERE ss_item_sk=1000
Whole stage codegen
Volcano iterator model
38
Whole stage codegen
What if we would ask some intern to write this in c#?
39
long count = 0;foreach (var ss_item_sk in store_sales) {
if (ss_item_sk == 1000)count++;
}
Volcano vs Intern
40
Volcano
Intern
source: Databricks
Volcano vs Intern
41
Developing ETL with Spark
4242
Choose your favorite IDE
43
Define Spark job entry point
44
object IncidentsJob { def main(args: Array[String]) {
val spark = SparkSession.builder() .appName("Incidents processing job") .config("spark.sql.shuffle.partitions", "16") .master("local[4]") .getOrCreate()
{ spark transformations and actions... }
System.exit(0)}
Create build.sbt file
45
lazy val root = (project in file(".")). settings( organization := "com.mycompany", name := "spark.job.incidents", version := "1.0.0", scalaVersion := "2.11.8", mainClass in Compile := Some("com.mycompany.spark.job.incidents.main") )
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "2.0.1" % "provided", "org.apache.spark" %% "spark-sql" % "2.0.1" % "provided", "org.apache.spark" %% "spark-streaming" % "2.0.1" % "provided",
"com.microsoft.sqlserver" % "sqljdbc4" % "4.0")
Create application (fat) jar file
$ sbt compile
$ sbt test
$ sbt assembly (sbt-assembly plugin)
46
Submit job via spark-submit command
./bin/spark-submit \ --class <main-class> \ --master <master-url> \ --deploy-mode <deploy-mode> \ --conf <key>=<value> \ ... # other options <application-jar> \ [application-arguments]
47
Example workflow
48
code
1. pull content2. take build number (331)3. build & test
4. copy to cluster
job331.jar
produce job artifact
notification
5. create/schedule job job331 (http)
6. spark submit job331
Spark Streaming
4949
Apache Spark streaming
● Scalable fault tolerant streaming system● Receivers receive data streams and chop them into batches● Spark processes batches and pushes out the result
50
● Input: Files, Socket, Kafka, Flume, Kinesis...
Apache Spark streaming
51
def main(args: Array[String]) { val conf = new SparkConf() .setMaster("local[2]") .setAppName("Incidents processing job - Stream")
val ssc = new StreamingContext(conf, Seconds(1))
val topics = Set( Topics.Incident,
val directKafkaStream = KafkaUtils.createDirectStream[Array[Byte], Array[Byte], DefaultDecoder, DefaultDecoder]( ssc, kafkaParams, topics)
// process batchesdirectKafkaStream.map(_._2).flatMap(_.split(“ “))...
// Start the computation ssc.start() ssc.awaitTermination()
System.exit(0)}
Apache Spark streaming
● Integrates with the rest of the ecosystem○ Combine batch and stream processing○ Combine machine learning with streaming○ Combine SQL with streaming
52
Structured streaming
53
[Alpha version in Spark 2.1]
53
Structured streaming (continuous apps)
● High-level streaming API built on DataFrames● Catalyst optimizer creates incremental execution plan
● Unifies streaming, interactive and batch queries
● Supports multiple sources and sinks
● E.g. aggregate data in a stream, then serve using JDBC
54
Structured streaming key idea
The simplest way to perform streaming analytics is not having to reason about streaming.
55
Structured streaming
56
Structured streaming
● Reusing same API
57
val categories = spark .read .option("header", "true") .schema(schema) .csv("/data/source") .select("Category")
val categories = spark .readStream .option("header", "true") .schema(schema) .csv("/data/source") .select("Category")
finite infinite
Structured streaming
● Reusing same API
58
categories .write .format("parquet") .save("/data/warehouse/categories.parquet")
categories .writeStream .format("parquet") .start("/data/warehouse/categories.parquet")
finite infinite
Structured streaming
59
Useful resources
● Spark home page: https://spark.apache.org/● Spark summit page: https://spark-summit.org/● Apache Spark Docker image:
https://github.com/dylanmei/docker-zeppelin● SFPD Incidents:
https://data.sfgov.org/Public-Safety/Police-Department-Incidents/tmnf-yvry
60
Thank you for the attention!
61
References
62
● Michael Armbrust - STRUCTURING SPARK: DATAFRAMES, DATASETS AND STREAMING - https://spark-summit.org/2016/events/structuring-spark-dataframes-datasets-and-streaming/
● Apache Parquet - https://parquet.apache.org/ ● Spark Performance: What's Next -
https://spark-summit.org/east-2016/events/spark-performance-whats-next/ ● Avoid groupByKey -
https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/best_practices/prefer_reducebykey_over_groupbykey.html