Getting Started with Neural Networks and Fundamentals of … · 2019-05-02 · Networks of Layers...

Getting Started with Neural Networksand

Fundamentals of Machine LearningMay 1, 2019

[email protected]

http://cross-entropy.net/ML410/Deep_Learning_3.pdf

mailto:[email protected]

http://cross-entropy.net/ML410/Deep_Learning_3.pdf

Agenda for Tonight

• Homework Review

• [DLP] Chapter 2: Getting Started with Neural Networks

• [DLP] Chapter 3: Fundamentals of Machine Learning

Getting Started with Neural Networks

1. Anatomy of a Neural Network

2. Introduction to Keras

3. Setting Up a Deep Learning Workstation

4. Classifying Movie Reviews: a Binary Classification Example

5. Classifying Newswires: a MultiClass Classification Example

6. Predicting House Prices: a Regression Example

7. Chapter Summary

Neural Networks

Training involves …

Anatomy of a Neural Network

Relationship between the Network, Layers, Loss Function, and Optimizer


Layers as the Lego Bricks of Deep Learning ☺

The first layer requires an input_shape parameter [the dimensions of a single observation], while additional layers do not require this parameter


Networks of Layers

• A deep learning model is a directed, acyclic graph of layers

• Most common instance is a “linear” (simple, sequential) stack of layers

• Other common instances include …• Two-branch networks; e.g. question goes down one branch and text passage

goes down another

• Multi-head networks; e.g. we have both an image and a text description as inputs

• Inception blocks; e.g. we want to use a few different convolution (input filtering) approaches in parallel


Loss Function and Optimizers

• Loss functions for this class include: cross entropy, mean squared error, mean absolute error, content loss, style loss, total variation loss, Kullback Leibler loss, temporal difference loss, actor loss, and critic loss

• Optimization functions for the class include: Stochastic Gradient Descent (SGD), Root Mean Squared (Gradient) Propagation (RMSProp), and Adaptive Moments (AdaM: RMSProp + Momentum)


Keras Features

Introduction to Keras

Google Search Interest for Deep Learning Frameworks


Deep Learning Software and Hardware Stack

• Nvidia Graphics Processing Units (GPUs) and Google Tensor Processing Units (TPUs) support efficient deep learning

• Nvidia’s Common Unified Device Architecture (CUDA) Application Programming Interface (API) and the CUDA Deep Neural Network (DNN) library provide an interface to Nvidia GPUs

• Eigen library implements the Basic Linear Algebra Subprograms (BLAS) specification, allowing tensor manipulation on Central Processing Units (CPUs)

Theano and CNTK are no longer maintained


Typical Keras Workflow


Network Definition:Sequential Model versus the Functional APISame model with both methods …


Model Configuration and Training


Two Options for Getting Keras Running

Setting Up a Deep Learning Workstation

Loading the Internet Movie DataBase (IMDB) Sentiment Analysis Datanum_words is the size of the vocabulary

Classifying Movie Reviews

Decoding a Document


Turning Lists of Integers into Tensors

Note: if more than one value in a row is one, we should refer to this as a multi-hot encoding

• one-hot encoding for identifying a class in a dense target vector

• multi-hot encoding to identify the tokens present in a document in a dense input vector


Encoding the Integer Sequences into a Binary Matrix


Architecture Decisions for Simple Feedforward Network• How many layers to use

• How many hidden units to choose for each layer

• Which activation functions to use• Do *not* forget to include activation functions: unexplained suboptimality

will ensue


Common Activation Functions

Rectified Linear Unit (ReLU): max(0,x)

[no saturation issue]

Sigmoid: 1/(1+exp(-x)

[usually used for output layer]


IMDB Network Architecture

• Features flow from bottom to top• An output is called a “head”

• Two hidden layers and an output layer with weights• It’s a deep neural network

• We’ll get to more than one hundred layers soon enough


Model Definition


Parameters and Outputs for a Dense Layer

• Parameters• (Number of Inputs from Previous Layer + 1) * (Number of “Units”)

• + 1 for bias weights: one for each “unit”

• We used to refer to “units” as neurons• The names have been changed to protect the innocent? Our approach was inspired by

neuroscience, but our brains aren’t using RMSProp ☺

• These are the same weight vectors we’ve come to know and love: projecting inputs to a new representation, one feature at a time [the number of “units” is the number of new features for the new representation]

• Output Shape• (Batch Size) x (Number of “Units”)


Why Are Activation Functions Necessary?

Try omitting activation functions from 1) the output layer and 2) hidden layers … so you can recognize this issue later


Compiling the Model


Setting Aside a Validation Set


Training the Model


Plotting the Training and Validation Loss


Where Do We Start Overfitting?


Plotting the Training and Validation Accuracy




Retraining the Model from Scratch

Why are we “retraining the model from scratch”?


Generating Predictions on New Data


Ideas for Experiments


Wrapping Up the IMDB Example


Loading the Reuters Dataset

Classifying Newswires

Decoding Newswires Back to Text


Preparing the Document Matrices


Preparing the Target Matrices


Defining the Model


Notes About the Architecture


Validating the Approach






Retraining a Model from Scratch


Why are we “retraining a model from scratch”?

Comparing to Random[and a Majority Classifier]

Nota bene (note well): 813 of the 2,356 test examples belonged to class 3

The accuracy of a majority classifier is 36.2%


Generating Predictions for New Data


Dense Versus Sparse Labels


Model With an Information Bottleneck

71% accuracy: an 8% absolute drop


Further Experiments


Wrapping Up


Loading the Boston Housing Dataset

1970s home prices in thousands of dollars

Predicting House Prices

Brief Discussion of Bias

• Boston Housing dataset has been used by many popular textbooks

• The data explicitly offers a race-related variable for modeling• Avoid using proxy variables that lead to discrimination based on race, gender,

religion, etc

• Example: don’t ask about gender if all you really want to know is whether the candidate can lift X pounds


http://lib.stat.cmu.edu/datasets/boston

http://lib.stat.cmu.edu/datasets/boston

Normalizing the Data


Model Definition


3-Fold Cross-Validation


4-Fold Cross-Validation Implementation


Cross-Validation Loop


Cross-Validation Results


Alternative Implementation [saved history]


Plotting the Average Mean Absolute Error (MAE)


Visualization Suggestions


Smoothing the Curve

The smoothed_points expression should look familiar: RMSProp (0.9 for last squared gradient) and AdaM (0.999 for last gradient)


Plotting the Smoothed MAE


Training the Final Model


Wrapping Up


Chapter Summary

Fundamentals of Machine Learning

1. Four Branches of Machine Learning

2. Evaluating Machine Learning Models

3. Data Preprocessing, Feature Engineering, and Feature Learning

4. Overfitting and Underfitting

5. The Universal Workflow of Machine Learning

Supervised Learning Examples

Four Branches of Machine Learning

Unsupervised Learning

• Dimensionality Reduction

• Clustering


Self-Supervised Learning

• Learning Without Human Annotated Labels

• Autoencoders

• Trying to predict the next word given previous words

• Trying to predict the next frame given previous frames


Reinforcement Learning

• Google Deep Mind used reinforcement learning to create a model to play Atari games

• AlphaGo was created to play Go

• Occasional rewards

• Examples of possible applications include: self-driving cars, robotics, resource management, and education


From Yann LeCun …

https://t.co/2LSb622114



Also from Yann LeCun …




Classification and Regression Glossary


Classification and Regression Glossary


Simple Hold-out Validation Split

Evaluating Machine Learning Models

Hold-out Validation Implementation[note the concatenation]


K-Fold Cross-Validation

Used for smaller data sets• If K is too small, we’ll experience high bias (underfitting)

• If K is too large, we’ll experience high variance (overfitting)


K-Fold Cross Validation Implementation


Iterated K-Fold Cross-Validation with Shuffling

• history = []

• for i in range(iterationCount):• shuffle(data)

• history.append(crossValidation(data, K = k))

• Requires building iterationCount * K + 1 models


Things to Keep in Mind


Value Normalization

• Dividing by 255 was an example of min-max normalization:

• value = (value – min(value)) / (max(value) – min(value))

• The max pixel value was 255 and the min pixel value was 0

• Alternatively, you can use center-and-scale normalization:

[-1,1] is fine too

Consider removing outliers

Data Preprocessing, Feature Engineering, and Feature Learning

Missing Values

• “In general, with neural networks, it’s safe to input missing values as 0, with the condition that zero isn’t a meaningful value”

• It’s possible to add indicator variables: 1 if missing; 0 otherwise

• If you expect missing values at test time, be sure to train with missing values:• We train like we deploy and deploy like we train


Feature Engineering Example

Three different inputs for the “What time is it?” model …

Why no radius on the polar coordinates?


Feature Engineering

• Does this mean you don’t have to worry about feature engineering as long as you’re using deep neural networks?

• No …


Original versus Lower Capacity Model

Original Model:16 units for each hidden layer

Lower Capacity Model: 4 units for each hidden layer

Overfitting and Underfitting

Original versus Lower Capacity Model

Smaller network starts overfitting later and it’s performance degrades more slowly


Original versus Higher Capacity Model:Validation Data

Validation Loss Noisierfor Higher Capacity Model(512 versus 16 units for each hidden layer)


Original versus Higher Capacity Model:Training DataMore capacity gives a model the ability to more quickly model the training data, but it also makes it susceptible to overfitting


Regularization [for Smaller Weights]


Example for Effect of Weight Regularization

Note: the goal of weight regularization is to improve generalization performance ☺


Additional Weight Regularizers for Keras


Adding Dropout(dropoutRate)


Adding Dropout to the IMDB Network


Recap of Most Common Ways to Prevent Overfitting


Define the Problem

Universal Workflow of Machine Learning

Hypothesis


Choosing a Measure of Success

• Accuracy

• Precision and Recall

• Area Under the Receiver Operating Characteristic (ROC) Curve (AUC)

• Maximize Recall subject to a constraint on the False Positive Rate?

• Mean Average Precision


Deciding on an Evaluation Protocol


Preparing Your Data


Key Choices for Your First Iteration


Choosing the Last-Layer Activation and Loss Function


How Big Should the Model Be?

Developing a model that overfits …


Regularizing the Model


Tuning the Model

We call these hyperparameters to distinguish them from the parameters of the model; i.e. the weights.

Note: We tune against validation data. Much like the “private leaderboard”, we only get one look at test perf.


Hyperas: Keras + Hyperopt

Bayesian optimization [as supported by Hyperas]• Start with random sets of hyperparameters for evaluation

• Iteratively select new hyperparameters for evaluation based on history• Partition the hyperparameter sets into two groups

• Sets where evaluated loss is lower than threshold tau

• Sets where evaluated loss is greater than or equal to threshold tau

• Hyperparameters can be viewed as having a hierarchy; e.g. we only need to optimize the number of hidden units for layer ‘k’ if we decide to add layer ‘k’

• Tree-based Parzen Estimators are used to select hyperparameter values based on

Expected Improvement, which is driven by: 𝑝 𝑥 | 𝑔𝑟𝑒𝑎𝑡𝑒𝑟𝑇ℎ𝑎𝑛𝑂𝑟𝐸𝑞𝑢𝑎𝑙

𝑝 𝑥 | 𝑙𝑒𝑠𝑠𝑇ℎ𝑎𝑛


https://en.wikipedia.org/wiki/Kernel_density_estimation

https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf

https://en.wikipedia.org/wiki/Kernel_density_estimation

https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf

Installing Hyperas and Running an Example

• pip install hyperas• wget https://github.com/maxpumperla/hyperas/blob/master/examples/mnist_readme.py

• python mnist_readme.py

model.add(Dropout({{uniform(0, 1)}}))model.add(Dense({{choice([256, 512, 1024])}}))model.add(Activation({{choice(['relu', 'sigmoid'])}}))if {{choice(['three', 'four'])}} == 'four’:

model.add(Dense(100))model.add({{choice([Dropout(0.5), Activation('linear')])}})optimizer={{choice(['rmsprop', 'adam', 'sgd'])}}batch_size={{choice([64, 128])}}


https://github.com/maxpumperla/hyperas/blob/master/examples/mnist_readme.py

Chapter Summary

Date post:	17-Mar-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Getting Started with Neural Networks and Fundamentals of … · 2019-05-02 · Networks of Layers...

Documents