Date post: | 23-Jan-2018 |
Category: |
Software |
Upload: | sam-putnam-deep-learning |
View: | 156 times |
Download: | 0 times |
Deploying Enterprise Deep Learning
Sam Putnam, Enterprise Deep Learning, LLC
July 27, 2017
Day 2
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Deep Learning Made This Video
https://www.extremetech.com/g00/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they?i10c.referrer=https%3A%2F%2Fwww.google.com%2F
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
Deploying Enterprise Deep Learning
And This Song!
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
And Predicted These Housing Prices!
https://github.com/EnterpriseDeepLearning/housing-prices-wide-and-deep/blob/master/real_estate.ipynb
http://www.mirror.co.uk/news/uk-news/experts-predict-house-prices-could-10724816
Run the Code ->This Afternoon
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
OK, so what is Deep Learning?
http://p.migdal.pl/imgs/2017-04-30-learning-deep-learning/deep_learning_meme_keras.png
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Deep Learning is Neural Networks
http://p.migdal.pl/imgs/2017-04-30-learning-deep-learning/deep_learning_meme_keras.png
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Well, Artificial Neural Networks. But they are based off of how the brain works.
https://www.extremetech.com/g00/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they?i10c.referrer=https%3A%2F%2Fwww.google.com%2F
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
This is a Neuron: Electrical Signals Are Sent By Axons and Received By Dendrites
http://www.newworldencyclopedia.org/entry/Dendrite
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
This is a Single Artificial Neuron With Two Inputs and Two Weights
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
The Artificial Neuron is Also Called a “Perceptron”. It “Perceives” Inputs.
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
b is the bias, and to activate the neuron:
Sam Putnam
@edeeplearning
Deploying Enterprise Deep Learning
Click Play, Visualize Neuron’s Performance
http://playground.tensorflow.org/
Sam Putnam
@edeeplearning
Deploying Enterprise Deep Learning
Trust Me, When you add more inputs, your decision boundary becomes a “plane”
Preview - Slide Available at deeplearningconf.com
When you’re talking about Deep Learning you’re talking about Artificial Neural Networks
of Neurons
@edeeplearning
Preview - Slide Available at deeplearningconf.com
In particular, Deep Learning = Deep Artificial Neural Networks of Neurons
@edeeplearning
Preview - Slide Available at deeplearningconf.com
This it it - Just more than one layer of neurons between the input and output!
@edeeplearninghttp://blog.christianperone.com/2015/08/convolutional-neural-networks-and-feature-extraction-with-python/
Preview - Slide Available at deeplearningconf.com
You Tried The Line, Now Add A Hidden Layer To Capture More Complex Data Separations
@edeeplearninghttp://blog.christianperone.com/2015/08/convolutional-neural-networks-and-feature-extraction-with-python/
This is No Line You Are Tweaking! This Is a More Complex Function
@edeeplearninghttp://blog.christianperone.com/2015/08/convolutional-neural-networks-and-feature-extraction-with-python/
Preview - Slide Available at deeplearningconf.com
More Neurons + A Deeper Network = More Sophisticated Representations
@edeeplearninghttps://cloud.google.com/blog/big-data/2016/07/understanding-neural-networks-with-tensorflow-playground
Look at the last hidden layer - it is doing a pretty good job separating the data
@edeeplearninghttps://cloud.google.com/blog/big-data/2016/07/understanding-neural-networks-with-tensorflow-playground
Preview - Slide Available at deeplearningconf.com
This Architecture is called a Feedforward Neural Network
@edeeplearninghttps://cloud.google.com/blog/big-data/2016/07/understanding-neural-networks-with-tensorflow-playground
Preview - Slide Available at deeplearningconf.com
There’s a Lot Going on Here. Do You Really Want to Change Those Weights One at a Time?
@edeeplearninghttps://cloud.google.com/blog/big-data/2016/07/understanding-neural-networks-with-tensorflow-playground
Preview - Slide Available at deeplearningconf.com
Solution: Gradient Descent - Look How the Lowest Loss (0) Is Sought Out on the Right
@edeeplearninghttps://cloud.google.com/blog/big-data/2016/07/understanding-neural-networks-with-tensorflow-playground
What other Architectures Are There? Well, what about for Time Series Data
@edeeplearninghttps://deeplearning4j.org/lstm.html
Preview - Slide Available at deeplearningconf.com
Recurrent Neural Networks are Networks with Loops in Them
@edeeplearninghttp://colah.github.io/posts/2015-08-Understanding-LSTMs/
Preview - Slide Available at deeplearningconf.com
Information is passed from one step of the network to the next
@edeeplearninghttp://colah.github.io/posts/2015-08-Understanding-LSTMs/
Preview - Slide Available at deeplearningconf.com
Recurrent Neural Networks (RNNs) Contain A Single Layer (Yellow Box)
@edeeplearninghttp://colah.github.io/posts/2015-08-Understanding-LSTMs/
Recurrent Neural Networks (RNNs) cannot handle long term dependencies
@edeeplearninghttp://colah.github.io/posts/2015-08-Understanding-LSTMs/
Preview - Slide Available at deeplearningconf.com
LSTM Recurrent Neural Networks (RNNs) Contain Four Layers (Yellow Box)
@edeeplearninghttp://colah.github.io/posts/2015-08-Understanding-LSTMs/
Long Short Term Memory Networks (LSTMs) Can Handle Long Term Dependencies
@edeeplearninghttp://colah.github.io/posts/2015-08-Understanding-LSTMs/
Preview - Slide Available at deeplearningconf.com
@edeeplearning
Sigmoid Activation Function Scales Your Input Between 0 and 1
http://neuralnetworksanddeeplearning.com/chap4.html
Preview - Slide Available at deeplearningconf.com
Sigmoid Activation Function Scales Your Input Between 0 and 1
@edeeplearning
http://neuralnetworksanddeeplearning.com/chap4.html
Preview - Slide Available at deeplearningconf.com
Tanh Activation Function Scales Your Input Between -1 and 1
@edeeplearning
http://neuralnetworksanddeeplearning.com/chap4.html
Preview - Slide Available at deeplearningconf.com
RELU Activation Function Cuts off Negative Inputs
@edeeplearning
http://neuralnetworksanddeeplearning.com/chap4.html
Preview - Slide Available at deeplearningconf.com
What About Image Data? For That, Convolutional Neural Networks (Right)
@edeeplearninghttp://cs231n.github.io/convolutional-networks/
Convolution Just Means Picking Out Representative Elements from a Bunch of
Pixels (F represents all the below)
@edeeplearning
Preview - Slide Available at deeplearningconf.com
This is What Convolution Looks Like, a Little Boring, But Look How It Reduces the Size of
the Data (3x3 on the Right!)
@edeeplearning
Preview - Slide Available at deeplearningconf.com
So Convolution Is a Mathematical Operation (Using a Weight Filter, Let’s Look!), but Even Simpler is
Max Pooling, just take the Biggest Pixel!
@edeeplearning
A Pre-Trained Image Recognition Neural Network Looks Like This (Input At Left) - Let’s
Zoom In
@edeeplearninghttps://arxiv.org/pdf/1409.4842.pdf
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Deep Learning in Production Considerations
Help me find the source for this nice diagram? Lost it :(
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Solution - Use What I Know and Am Allowed To Use, i.e. Kaggle, AWS, TF, Jupyter
Help me find the source for this nice diagram? Lost it :(
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
What I Did - A Real Estate Agent Told Me About His Problem & Pointed Me To The Data
https://www.fortunebuilders.com/how-to-become-a-real-estate-agent/
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
I Took a Holdout Set of the Data and Left the Rest for Train/Validate
https://www.fortunebuilders.com/how-to-become-a-real-estate-agent/
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Started with Feature Selection and then Linear Regression in Good old Excel, WITH the subject matter
expert (the real estate agent). Did Pretty Well.
https://www.youtube.com/watch?v=SQkpLMLoqww
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Preview - Slide Available at deeplearningconf.com
How Well? 18 percent Mean Error and 26 percent Median Error before Feature Selection using Weighted Importance, Did some thinking:
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Feature Importance, for Feature Selection, Often Use After Deep Learning Step, to Gain Interpretability
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
https://www.kaggle.com/samdeeplearning/naive-subsample-5-10-city/output
GBM (Boosted Decision Tree) - with Hyperparameter Tweaking - Yields Strong Model for this Problem
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
https://www.kaggle.com/samdeeplearning/naive-subsample-0-25-xgb/notebook
Preview - Slide Available at deeplearningconf.com
Starts to Degrade Upon Subsampling only 1/4 of the Trees
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Want To See If Neural Network Can Improve Performance
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Building off of a preexisting housing prices regression model
https://github.com/EnterpriseDeepLearning/housing-prices-wide-and-deep/blob/master/real_estate.ipynb
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Set Input Layer (n_hidden_1 is input layer here) to number of features (27)
https://github.com/EnterpriseDeepLearning/housing-prices-wide-and-deep/blob/master/real_estate.ipynb
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Combining the features right off the bat by using a wide first hidden layer (200 nodes)
https://github.com/EnterpriseDeepLearning/housing-prices-wide-and-deep/blob/master/real_estate.ipynb
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Set number of classes to 1 for regression problem
https://github.com/EnterpriseDeepLearning/housing-prices-wide-and-deep/blob/master/real_estate.ipynb
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Multilayer Perceptron = Feedforward Perceptron We Looked At Before
https://github.com/EnterpriseDeepLearning/housing-prices-wide-and-deep/blob/master/real_estate.ipynb
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
RELU - As Mentioned in Executing Strategies Yesterday, not using negative features/weights
https://github.com/EnterpriseDeepLearning/housing-prices-wide-and-deep/blob/master/real_estate.ipynb
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Training - Can tell performance needs tweaking, pretty good on houses 2-6
https://github.com/EnterpriseDeepLearning/housing-prices-wide-and-deep/blob/master/real_estate.ipynb
123456
Estimate on LeftPrediction on Left
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Let’s Dive Into the TensorFlow Housing Code
https://www.kaggle.com/samdeeplearning/deep-neural-network-for-starters-r/edit
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
What Next? Go Deep Learning!
https://www.youtube.com/watch?v=cSKfRcEDGUs&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal&index=6
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
1) Identify an architecture for your problem
https://www.youtube.com/watch?v=cSKfRcEDGUs&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal&index=6
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
2) Identify a Framework, Be Willing to Try a New Framework
https://blog.algorithmia.com/deploying-deep-learning-cloud-services/
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
3) Build or tailor a model for your application, Don’t be Afraid to Reference Academic Code
https://www.youtube.com/watch?v=cSKfRcEDGUs&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal&index=6
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
4)Tweak the Parameters for your Model
https://www.youtube.com/watch?v=cSKfRcEDGUs&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal&index=6
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
5)Train on a (probably, Nvidia, maybe Intel) GPU if need be (for image data, you need it)
https://www.youtube.com/watch?v=2NrgPdGSXhE
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
6) Validate, test, check in the real world. Iterate!
https://medium.com/towards-data-science/train-test-split-and-cross-validation-in-python-80b61beca4b6
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Weekend trial: Josh Gordon’s Machine Learning Recipes. If the code doesn’t work for you (I am 99 %
sure it will), let me know!
https://www.youtube.com/watch?v=cSKfRcEDGUs&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal&index=6
Preview - Slide Available at deeplearningconf.com
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Like R? Try this notebook on my Kaggle
https://www.kaggle.com/samdeeplearning/deep-neural-network-for-starters-r/edit
Sam Putnam
@edeeplearning
July 27
Deploying Enterprise Deep Learning
Submit your predictions, even! (disclaimer: middle of the pack result)
https://www.kaggle.com/samdeeplearning/deep-neural-network-for-starters-r/edit
Preview - Slide Available at deeplearningconf.com
Sam Putnam
Thank you to Google and others who have published diagrams and photos. Slides are for today only.
@edeeplearning
Questions/Comments: [email protected]
Thank you
July 27
Deploying Enterprise Deep Learning