Scalable Machine Learning with Dask

Tom Augspurger

Data Scientist

© 2018 Anaconda, Inc. All Rights Reserved. 

Hi! I’m Tom

• I work at Anaconda on dask, pandas, & ML things

2

Machine Learning WorkflowIt’s not just .fit(X, y)

Machine Learning Workflow

• Understanding the problem, objectives • Reading from data sources • Exploratory analysis • Data cleaning • Modeling • Deployment and reporting

4

Bokeh

Jake Vanderplas PyCon 2017 KeynoteThe Numeric Python Ecosystem

Machine Learning Workflow

A rich ecosystem of tools

But they don’t scale well

6

DaskParallelizing the Numeric Python Ecosystem

Dask

High Level: Parallel Pandas, NumPy, Scikit-Learn

Low Level: High performance task scheduling

8

Dask

import dask.dataframe as dd

df = dd.read_csv('s3://abc/*.csv')df.groupby(df.name).value.mean()

9

High-Level: Scalable Pandas DataFrames

Dask

import dask.array as da

x = da.random.random(…)y = x.dot(x.T) - x.mean(axis=0)

10

High-Level: Scalable NumPy Arrays

Dask

11

Low-Level: Scalable, Fine-Grained Task Scheduling

Demo

Dask

• Parallelizes libraries like NumPy, Pandas, and Scikit-Learn • Adapts to custom algorithms with a flexible task scheduler • Scales from a laptop to thousands of computers

• Integrates easily, Pure Python built from standard technology

13

Scalable Machine LearningDask-ML

Scaling Pains

15

Scaling Pains

16

Scaling Pains

17

Scaling Pains

18

Distributed Scikit-Learn

Distributed Scikit-Learn

• Use dask to distribute computation on a cluster

20

Large models, smaller datasets

Distributed Scikit-Learn

21

Single-Machine Parallelism with Scikit-Learn

from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=200, n_jobs=-1)

clf.fit(X, y)

Distributed Scikit-Learn

22

Multi-Machine Parallelism with Dask

from sklearn.ensemble import RandomForestClassifierfrom sklearn.externals import joblibimport dask_ml.joblib

clf = RandomForestClassifier(n_estimators=200, n_jobs=-1)

with joblib.parallel_backend("dask", scatter=[X, y]): clf.fit(X, y)

Distributed Scikit-Learn

• Data has to fit in RAM • Data shipped to each worker

• Each parallel task should be expensive

• There should be many parallel tasks

23

Caveats

Scalable AlgorithmsWhen your dataset is larger than RAM

First: Do you need all the data?

• Sampling may be OK • Plotting Learning

Curves from scikit-learn docs

25

Second: Parallel Meta-estimators

26

from dask_ml.wrappers import ParallelPostFitimport dask.dataframe as dd

clf = ParallelPostFit(SVC())clf.fit(X_small, y_small)

X_large = dd.read_csv("s3://abc/*.parq")y_large = clf.predict(X_large)

• Train on subset • Predict for

large dataset, in parallel

Scalable Estimators

• Scikit-Learn wasn’t designed for distributed datasets

• Dask-ML implements scalable variants of some estimators

• Works well with Dask DataFrames & Arrays

27

When the training dataset is larger than RAM

Scalable, Parallel Algorithms

28

Spectral Clustering Comparison

Scalable, Parallel Algorithms

• Distributed GLM•LogisticRegression, LinearRegression, ...

• Clustering •KMeans(init='k-means||'), SpectraclClustering, ...

•Preprocessing•QuantileTransformer, RobustScalar, ...

•Dimensionality Reduction•PCA, TruncatedSVD

•...

29

Some Notable Estimators

FamiliarWorks well with existing libraries

Familiar

• Dask-ML estimators are Scikit-Learn estimators • Dask-ML pipelines are Scikit-Learn Pipelines

31

Familiar

32

>>> from sklearn.pipeline import make_pipeline >>> from sklearn.preprocessing import FunctionTransformer

>>> pipe = make_pipeline( ... ColumnSelector(columns), ... HourExtractor(['Trip_Pickup_DateTime']), ... FunctionTransformer(payment_lowerer, validate=False), ... Categorizer(categories), ... DummyEncoder(), ... StandardScaler(scale), ... LogisticRegression(), ... )

Familiar

33

>>> from sklearn.pipeline import make_pipeline >>> from sklearn.preprocessing import FunctionTransformer

>>> pipe = make_pipeline( ... ColumnSelector(columns), ... HourExtractor(['Trip_Pickup_DateTime']), ... FunctionTransformer(payment_lowerer, validate=False), ... Categorizer(categories), ... DummyEncoder(), ... StandardScaler(scale), ... LogisticRegression(), ... )

Scikit-Learn objects

Custom transformers

Dask-ML estimators

Full Example: https://git.io/vAi7C

Distributed SystemsIntegrate with XGBoost and Tensorflow

>>> import dask_ml.xgboost as xgb>>> df = dd.read_csv("trips*.csv")>>> y = df['Tip_Amt'] > 0>>> X = df[columns]>>> booster = xgb.train(... client, params, X, y... )

Distributed System

35

Peer with systems like XGBoost or Tensorflow

Dask & Dask-ML

• Parallelizes libraries like NumPy, Pandas, and Scikit-Learn • Scales from a laptop to thousands of computers • Familiar API and in-memory computation

• https://dask.pydata.org

36

Questions?