What’s new in Spark 2.0?files.meetup.com/19070069/161129 - Michael... · ©2016 Couchbase Inc. 3...

transcript

What’snewinSpark2.0?

SparkOverview

ApacheSparkisafastandgeneralengineforlarge-scaledataprocessing.

SparkOverview

Spark2.0

§ Largelycompatiblewith1.x§ SimplifiesAPI§ 2000patchesfrom280contributors

http://www.slideshare.net/SparkSummit/simplifying-big-data-applications-with-apache-spark-20

Spark2.0

§ StructuredAPIImprovements§ Whole-stagecodegeneration§ StructuredStreaming§ SimplerSetup§ SQL2003Support§ MLlibenhancements§ EnhancedRsupport§ …

StructuredAPIImprovements

§ Dataset(typed)andDataFrame(untyped)arenowunified§  DataFrame==Dataset<Row>

§ AlsousedforStructuredStreaming

https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html

Whole-StageCodegen

§ Second-generationTungstenengine§ DepartingfromVolcanoIteratorModel§ Also:Vectorizationformoreefficientbatch-processing

https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html

Whole-StageCodegen

https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html

StructuredStreaming(Experimental)

§ Tackling“continuousapplications”§  IntegratedAPIwithbatchjobs§ Betterinteractionwithstoragesystems§ RichIntegrationintotherestofSpark

https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html

SimplerSetup

§ SparkSessionsubsumesSQLContext,HiveContext,…§ Onecommonentrypoint

SQL2003Support

§ SupportsSQL2003Standard

§ ReworkednativeSQLparser§ NativeDDLcommandimplementations§ Allkindsofsubqueriesnowsupported§ Cannowrunall99TPC-DSqueries

MLlibEnhancements

§ DataFrameasprimaryMLAPI§ ModelPersistence§  SupportforalllanguageAPIsinSpark:Scala,Java,Python&R§  SupportfornearlyallMLalgorithmsintheDataFrame-basedAPI§  SupportforsinglemodelsandfullPipelines,bothunfitted(a“recipe”)andfitted(aresult)

§  Distributedstorageusinganexchangeableformat

QAThanks!

Couchbase&Spark

EcommerserunsonCouchbase

6 10 ECOMMERCE COMPANIES

IN THE UNITED STATES

of the TOP

Online shopping

Omni channel services

TravelrunsonCouchbase

GLOBAL DISTRIBUTION SYSTEMS WORLDWIDE

of the TOP

AIRLINES

of the TOP

OnlineVideoStreamingrunsonCouchbase

6 10 NORTH AMERICAN AND

EUROPEAN BROADCAST TELEVISION COMPANIES

of the TOP

Sports&CasinoGamingrunsonCouchbase

6 10 ONLINE SPORTS AND

CASINO GAMING COMPANIES

of the TOP

FinancialServicesrunonCouchbase

3 3 CREDIT REPORTING

COMPANIES

of the TOP

WhySparkandCouchbaseOverview&Use-Cases

UseCases

Operations Analytics

§  Recommendations§  Predictiveanalytics§  Frauddetection

§  Catalog§  Personalization§  Mobileapplications

UseCase:OperationalizeAnalytics/ML

Hadoop

MLModel

Data Warehouse

Training Data

Model Online Data

Serving

Predictions

UseCase:DataIntegration

RDBMS S3 HDFS ES

StandaloneDeployment

Side-By-SideDeployment

AccessPatternsFromSparktoCouchbaseandBackAgain

Key-Value

Fetch/StorebyDocumentID

Key-Value

N1QLQuery

FetchbyCriteria“SQL”

Key-Value

N1QLQuery

Map-ReduceViews

MaterializedIndexes

(Aggregation)

Key-Value

N1QLQuery

Map-ReduceViews

MaterializedIndexes

(Aggregation)

Streaming

MutationStreamsForProcessing

Key-Value

N1QLQuery

Map-ReduceViews

MaterializedIndexes

(Aggregation)

Streaming

FullText

SearchonFreeformText

Key-Value

N1QLQuery

Map-ReduceViews

MaterializedIndexes

(Aggregation)

Streaming

CouchbaseDataPartitioning

DataLocality

§ RDDLocationHintsbasedontheClusterMap

§ NotavailableforN1QLorViews§  Roundrobin-can’tgivelocationhints§  Backendisscattergatherwith1noderesponding

N1QLQuery

§ N1QLisaSQLservicewithJSONextensions

§ UsesCouchbase’sGlobalSecondaryIndexes

§ Canrunonanynodeswithinthecluster

§ Nodeswithdifferingservicescanbeaddedandremovedasneededonthefly

DataService

Projector&Router

CouchbaseQueryArchitecture

QueryService

IndexService

SupervisorIndexmaintenance&Scancoordinator

Index#2Index#1

QueryProcessorcbq-engine

Bucket#1 Bucket#2

DCPStreamIndex#4Index#3

...Bucket#2

Bucket#1

ForestDBStorageEngine

SparkSQLSources

TableScanScanallofthedataandreturnit

PrunedScanScananindexthatmatchesonlyrelevantdatatothequeryathand.

PrunedFilteredScanScananindexthatmatchesonlyrelevantdatatothequeryathand.

PredicateConversion

SchemaInference

N1QLRelation:28 - Inferring schema from bucket travel-sample with query 'SELECT META(`travel-sample`).id as `META_ID`, `travel-sample`.* FROM `travel-sample` WHERE `type` = 'airline' LIMIT 1000'

N1QLRelation:28 - Executing generated query: 'SELECT `name`,`callsign` FROM `travel-sample` WHERE `type` = 'airline''

SchemaInference

DCPandSparkStreaming

ReplicaIndexing

StructuredStreamingSource

Adaptedfromhttps://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html

DCPStream UnboundedTable

(Un)StructuredStreaming?

StructuredStreamingSource

StructuredStreamingSink

CouchbaseSparkConnector1.2.1

§ Spark1.6.xsupport,includingDatasets§ DCPFlowControl§ EnhancedJavaAPIs

CouchbaseSparkConnector2.0.0

•  Spark2.0.xSupport•  EnhancedDCPClient•  ExperimentalStructuredStreaming

Resources

§  SparkPackageshttps://spark-packages.org/package/couchbase/couchbase-spark-connector

•  Docshttp://docs.couchbase.com

§  Sourcehttps://github.com/couchbase/couchbase-spark-connector

§  Bugshttps://issues.couchbase.com/browse/SPARKC

QAThanks!

What’s new in Spark 2.0?files.meetup.com/19070069/161129 - Michael... · ©2016 Couchbase Inc. 3...

Documents