3 DAYS ON JANUARY 16th, 17th & 18th 2015

www.globalbigdataconference.com

Twitter : @bigdataconf

Presenter: Josh Patterson

Past

Research in Swarm Algorithms

TVA / NERC

Cloudera

Today

Patterson Consulting

Skymind (Advisor)

josh@pattersonconsultingtn.com / @jpatanooga

• What is Deep Learning?• What is DL4J?• Enterprise Grade Deep Learning Workflows

WHAT IS DEEP LEARNING?

“Cooper: [When Cooper tries to reconfigure TARS] Humour 75%.TARS: 75%. Self destruct sequence in T minus 10, 9, 8...Cooper: Let's make it 65%.TARS: Knock, knock.”

--- Interstellar

We Want to be able to recognize Handwriting

This is a Hard Problem

Automated Feature Engineering• Deep Learning can be thought of as workflows for

automated feature construction– Where previously we’d consider each stage in the

workflow as unique technique• Many of the techniques have been around for years– But now are being chained together in a way that

automates exotic feature engineering• As LeCunn says:– “machines that learn to represent the world”

These are the features learned at each neuron in a Restricted Boltzmann Machine (RBMS)

These features are passed to higher levels of RBMs to learn more complicated things.

Part of the “7” digit

Learning Progressive Layers

Deep Learning Architectures• Deep Belief Networks

– Most common architecture• Convolutional Neural Networks

– State of the art in image classification• Recurrent Networks

– Timeseries • Recursive Networks

– Text / image– Can break down scenes

DL4JNext Generation Deep Learning with

What is DL4J?• “The Hadoop of Deep Learning”

– Command line driven– Java, Scala, and Python APIs

• ASF 2.0 Licensed• Java implementation

– Parallelization / GPU support• Runtime Neutral

– Local– Hadoop / YARN– Spark– AWS

• https://github.com/deeplearning4j/ deeplearning4j

14

DL4J and Parallelization

Model

Training Data

Worker 1

Master

Partial Model

Global Model

Worker 2

Partial Model

Worker N

Partial Model

Split 1 Split 2 Split 3

…

Traditional Serial TrainingModern Parallel Engine

(Hadoop / Spark)

DL4J Suite of Tools• DL4J

– Main library for deep learning• Canova

– Vectorization library• ND4J

– Linear Algebra framework – Swappable backends (JBLAS, GPUs)

• Arbiter– Model evaluation and testing platform

Canova for Command Line Vectorization

• Library of tools to take – Audio– Video– Image– Text– CSV data

• And convert the input data into vectors in a standardized format– Adaptable with custom input/output formats

• Open Source, ASF 2.0 Licensed– https://github.com/deeplearning4j/Canova– Part of DL4J suite

DEEP LEARNING WORKFLOWSEnterprise Grade

Building Deep Learning Workflows• Our terminology in data science has gotten more exotic

– But its still about gather, cleaning, visualizing, and feature construction of data

• We need to get data from a raw format into a baseline raw vector– Which is why Canova exists

• Deep Learning is not just classification– But an automated feature construction pipeline

• Together, DL4J and Canova give us the base workflow

Modeling UCI Data: Iris

• We need to vectorize the data– Possibly with some per column transformations– Which is why Canova exists• To feed raw data into a form DL4J can consume

• We then need to build a deep learning model over the data– We’ll use the DL4J lib to do this

Vectorization with Canova• Setup the configuration file• Setup the schema transforms for the input

CSV data• Generate the SVMLight vector data as the

output – with the command line interface

Canova Configuration

input.header.skip=falseinput.statistics.debug.print=falseinput.format=org.canova.api.formats.input.impl.LineInputFormat

input.directory=src/test/resources/csv/data/uci_iris_sample.txtinput.vector.schema=src/test/resources/csv/schemas/uci/iris.txtoutput.directory=/tmp/iris_unit_test_sample.txt

output.format=org.canova.api.formats.output.impl.SVMLightOutputFormat

Canova Vector Schema

@RELATION UCIIrisDataset@DELIMITER ,

@ATTRIBUTE sepallength NUMERIC !NORMALIZE @ATTRIBUTE sepalwidth NUMERIC !NORMALIZE @ATTRIBUTE petallength NUMERIC !NORMALIZE @ATTRIBUTE petalwidth NUMERIC !NORMALIZE @ATTRIBUTE class STRING !LABEL

Canova CLI./bin/canova vectorize -conf /tmp/iris_conf.txt File path already exists, deleting the old file before proceeding...Output vectors written to: /tmp/iris_svmlight.txt

DL4J from the Command Linebin/dl4jUsage:

dl4j [command] [params] Commands:

train build a deep learning model test test a deep learning model predict score new records against a deep learning

model

DL4J Configuration Fileinput.format=org.canova.api.formats.input.impl.SVMLightInputFormat

input.directory=src/test/resources/data/irisSvmLight.txt

model.config=src/test/resources/model_dbn.json

output.directory=/tmp/

JSON Model Config{"sparsity":0.0,"useAdaGrad":true,"lr":0.10000000149011612,"corruptionLevel":0.30000001192092896,"numIterations":1000,"momentum":0.5,"l2":0.0,"useRegularization":false,"momentumAfter":null,"resetAdaGradIterations":-1,"numLineSearchIterations":100,"dropOut":0.0,"applySparsity":false,"weightInit":"VI","optimizationAlgo":"CONJUGATE_GRADIENT","lossFunction":"RECONSTRUCTION_CROSSENTROPY","concatBiases":false,"constrainGradientToUnitNorm":false,"seed":123,"nIn":0,"nOut":0,"activationFunction":"sigmoid","visibleUnit":"BINARY","hiddenUnit":"BINARY","k":1,"weightShape":[0,0],"filterSize":[2,2,2,2],"numFeatureMaps":2,"featureMapSize":[2,2],"stride":[2,2],"kernel":5,"batchSize":0,"minimize":false,"rng":"org.apache.commons.math3.random.MersenneTwister","layerFactory":"org.deeplearning4j.nn.layers.factory.PretrainLayerFactory,org.deeplearning4j.models.featuredetectors.rbm.RBM","stepFunction":"org.deeplearning4j.optimize.stepfunctions.DefaultStepFunction","renderWeightIterations":-1,"dist":"org.apache.commons.math3.distribution.NormalDistribution\t{SQRT2=1.4142135623730951, mean=0.0, solverAbsoluteAccuracy=1.0E-9, DEFAULT_INVERSE_ABSOLUTE_ACCURACY=1.0E-9, standardDeviation=0.1, logStandardDeviationPlusHalfLog2Pi=-1.3836465597893728, serialVersionUID=8589540077390120676}","listeners":["org.deeplearning4j.optimize.listeners.ScoreIterationListener\t{printIterations=10, log=Logger[org.deeplearning4j.optimize.listeners.ScoreIterationListener]},"]}

DL4J CLI Training./bin/dl4j train -conf /tmp/iris_conf.txt

Skymind as DL4J Distribution

• Just as Redhat was to Linux– A distribution of Linux with enterprise grade packaging

• Just as Cloudera was to Hadoop– A distribution of Apache Hadoop with enterprise grade

packaging• Skymind is to DL4J

– A distribution of DL4J (+tool suite) with enterprise grade packaging

Thank you for your time and attention

“Deep Learning: A Practitioner’s Approach” (Oreilly, October 2015)