The Different Forms of Machine Learning: How They Fit with ...

#vmworld

BCA1332BU

The Different Forms of Machine Learning: How They Fitwith VMware

Justin Murray, VMware, Inc.Uday Kurkure, VMware, Inc.

#BCA1332BU

VMworld 2019 Content: Not for publication or distribution

©2019 VMware, Inc.

Disclaimer

This presentation may contain product features or functionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.

2

The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or obligation to deliver any items presented herein. VMworld 2019 Content: Not for publication or distribution

©2019 VMware, Inc.

Agenda

3

Vision

AI, Machine Learning and Deep Learning defined

Two Types of Training Data for Machine Learning

Why do GPUs Help Performance?

GPU Performance Tests and Results on vSphere

Machine Learning Infrastructure on vSphere

Where to Learn More


©2019 VMware, Inc. 4

Any GPU/AcceleratorVMworld 2019 Content: Not for publication or distribution


AI, Machine Learning and Deep Learning





Machine Learning Lifecycle – Preparing the Training Dataset

Training Data Set

Labelled Examples

Prepare Training data



Machine Learning Lifecycle – Training

Training Data Set Model

Train the

Model

Labelled Examples


Training Phase



Machine Learning – Using the Testing Data

Testing Data –No labels

Classification / Prediction

Model

How Accurate

is the Model?



Machine Learning – Inference

Previously Unseen Data

from Operations


Model

Execute the model for analysis of

the new data

Inference -Operations

Phase



Machine Learning – Training and Inference

Training Data Set


from Operations


Model

Train the

Model

Labelled Examples

Execute the model for

analysis on the new data


Training Phase

Inference -Operations

Phase



Machine Learning – Deployment of the Inference

Training Data Set


from Operations

Mathematical ModelClassification /

Prediction

Mathematical ModelModel

2. Train the

Model

Labelled Examples

Execute the model for

analysis on the new data

1. Prepare Training data

Training Phase

Scoring/Inference

Phase

Re-Train



MachineLearning

DeepLearningData Lake

ON-PREM

OFF-PREM

training

The Machine Learning Infrastructure Landscape

Data Analytics

Two Main Phases in ML

• Training / Model Building

• Often very large training data sets

• Compute, storage, and network intensive

• Server-class infrastructure

• Inference / Scoring

• Apply existing models to new data

• Used for prediction

• Edge or core infrastructure



MachineLearning

DeepLearningBig Data

ON-PREM

OFF-PREM

training

inference

inference

Machine Learning Infrastructure Landscape

Data Analytics



Different Data Types in Machine LearningDifferent ML Infrastructure

Tabular Data

• Rows and columns

• Database tables

• Spreadsheets and CSV files

• CPUs may suffice

Image, Voice and Text Data • Image classification

• Video analytics,

• Advanced Financial Modeling

• GPUs apply mainly here

a b c d e f h i j k l

1 4 2 2 1 10 0 1 1 1 0

5 8 3 1 1 5 1 0 0 1 1

9 1 3 6 2 14 1 1 1 1 0


16©2019 VMware, Inc.

The First Form of ML Training DataData in Tabular or CSV Form



Training Data Can Be Tabular

Acct Number

TxnID

Education Age Sex BalanceLimit

Married Paid 1 Month Ago

Paid 2 MonthsAgo

Paid 3 Months Ago

Default

1234 45 2. 21 1 100 0 1 1 1 0

5678 89 3 31 1 5000 1 0 0 0 1

9012 150 3 61 2 1400 1 1 1 1 0

Label

Examplesxi

Features



The Training Process

Weightsw

1234

5678

9012

1234 45 2 21 1 100 80 1 1 1

5678 89 3 31 1 5000 110 0 0 1

9012 150 3 61 2 1400 50 1 0 1

Feature Values - Numerical

Row X ColumnMatrix

Multiplication



H2O.ai’s Driverless AI – Automating Training using Tabular Data



Tabular Training Data – Seen in H2O.ai’s Driverless AI Tool



H2O.ai’s Driverless AI – The Prediction Target and Accuracy



H2O.ai’s Driverless AI – Prediction Target and Accuracy



The Second Form of Training Data for MLImages, Text, Voice, Video



AI, Machine Learning and Deep Learning



Input Layer

Hidden Layers

Output Layer

A Deep Neural Network (DNN)



Data scientists commonly use GPUs for machine learning applications in

• Image classification

• Video analytics

• Financial modeling and analysis

GPUs are particularly useful in Deep Learning

• Where multi-level “deep” neural networks are being trained

Deep Neural Networks with GPUs can complete the training phase in less time than a CPU-based one would (by an order of magnitude)

GPUs and Machine Learning



General Purpose GPU

A high-end GPU has thousands of cores

These cores are used for matrix multiplication and other complex math

GPU cores are optimized exclusively for data computations



Machine Learning Infrastructure in VMware vSphere



ONE Inference for ONE Image Takes10 Millions – 30 Billions multiply add operations

Training

• Large training data (thousands to millions samples)

• Repetition (large number of epochs, iterations)

Inference

• Require real-time processing

• Can have massive amount of concurrent requests in cloud production

High Performance Requirements for Machine Learning/Deep Learning

Source of the graph: https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/README.mdVMworld 2019 Content: Not for publication or distribution


Computing Architectures for Machine Learning

GPU (Graphics Process Unit)Multi-core CPU Systems

FPGA (A field-programmable Gate Array) ASICs: TPU (Tensor Processing Unit)

M60, P100, P40, V100 cards



Performance:GPUs vs CPUsWhy do you need GPUs for ML training?



Training Workloads: Handwriting Recognition & Language Modelling

Handwriting Recognition

Neural Network: Convolutional Neural Network

Dataset:

MNIST database of handwritten digits

• Training set: 60,000 examples

• Test set: 10,000 examples

Complex Language Modeling

• Given history of words, predicts next word

Neural Network: Recurrent Neural Network

• Large Model

– 1500 Long Short Term Memory (LSTM) units /layer

• Medium

– 650 LSTM units /layer

• Small

– 200 LSTM units /layer

Dataset:

• Penn Tree Bank (PTB) Database:

– 929K training words

– 73K validation words

– 82K test words

– 10K vocabularyVMworld 2019 Content: Not for publication or distribution


Training Times with GPU vs. without GPU on a Virtualized Server

1.0

10.1

0

2

4

6

8

10

12

1 2

No

rmaliz

ed

Tra

inin

g T

ime

Lo

we

r is

be

tte

r

Handwriting Recognition with CNN on MNIST

1

7.9

0

2

4

6

8

10

1 2

No

rmaliz

ed

Tra

inin

g T

ime

Lo

we

r is

be

tte

r

Language Modeling with RNN on PTB

56 Hours for No-GPU8 Hours with vGPUEnergy Efficient



VMware vSphere with NVIDIA GPUs

Our customers are using GPUs on VMware vSphere

Accelerating 2D/3D Graphics workloads for VMware Horizon

Enabling VMware Blast Extreme protocol – Encoding / Decoding H.264 and H.265 Based

General Purpose GPU (GPGPU)– Machine learning / Deep Learning

– High performance computing workloads



Benefits of Virtualized GPUs in VMware vSphere

Virtualization Technology efficiently manages servers in the data centers

• Enables Diverse Workloads

– Windows and Linux VMs running on the same host

• Higher Consolidation Ratios

• Suspend/Resume of Virtualized GPU enabled VMs

– ML Training at night

– Interactive CAD jobs during the day



Benefits of Virtualized GPUs in VMware vSphere

vMotion of vGPU VMs

• ML Training or HPC jobs can take days

• Before server maintenance, vMotion the VMs to another host and then move them back after the maintenance. Thus, saving days of work

Combine the Power of GPUs with Management Benefits of Virtualization



Leverage GPU investment across different use cases

• ML Workloads on Linux for Data Scientist/ML researchers

• Virtual Desktop Infrastructure (VDI) for Office Workers on Windows

• 3-D CAD Workloads on Windows and Linux for Scientists

• Simulations on Linux

• End Users in Different Time Zones using GPUs at different times

• Improve Data Center Resource Utilization Using vGPUs in Data Centers

• Virtualized GPUs enable all of the above

A Typical Customer Scenario



Server

VMware Hypervisor (ESX)

Linux Virtual

Machine

Virtual Machine

Windows Virtual

Machine

Virtual Machine

What is a Virtualized NVIDIA GPU (vGPU) ?

Virtual Machine

NVIDIA GPU

H.265 Encode/Decode

Virtual Machine

NVIDIA Driver NVIDIA Driver

NVIDIA vGPU manager (vib)

NVIDIA DriverNVIDIA Driver NVIDIA Driver NVIDIA Driver

vGPU vGPUvGPU vGPU vGPU vGPU

CPUsNVIDIA

GPU

Ha

rdw

are

Vir

tua

liza

tio

n L

aye

r



Virtualized GPUs in vSphere

vSphereHypervisor

GPUGPU GPU

VMware DirectPath I/O

Virtual

Machine

Guest OS

GPU driver

Applications

Virtual

Machine

Guest OS

GPU driver

Applications

Virtual

Machine

Guest OS

GPU driver

Applications

Pass-th

rough

Pass-th

rough

Pass-th

rough

GPU

Pass-th

rough

vSphereHypervisor

vGPU

Virtual

Machine

Guest OS

GPU driver

Applications

Virtual

Machine

Guest OS

GPU driver

Applications

Virtual

Machine

Guest OS

GPU driver

Applications

Virtual

Machine

Guest OS

GPU driver

Applications

NVIDIA GRIDvGPU manager

vGPU

NVIDIA GRID vGPU

Virtual

Machine

Guest OS

GPU driver

Applications

Virtual

Machine

Guest OS

GPU driver

Applications

Virtual

Machine

Guest OS

GPU driver

Applications

vGPUvGPU

GRIDGPU

vGPU vGPU vGPU vGPU

vMotion

Sharing

vMotion

Sharing

vMotion

Sharing

vSphereHypervisor

Virtual Machine

Guest OS

VMware

GPU driver

Applications

NVIDIA Driver

GPU

vSGA

Virtual

Machine

Guest OS

VMware

GPU driver

Applications

multiple vGPUs/VMmultiple GPUs/VM

Diverse Workloads



Pascal (Virtualized for CUDA & Graphics)P40 Card with 24GB GPU Memory

40

virtualGPU type

physical board GRAPHICS CUDA

maximumvirtual GPUsper physical

GPU

GRID P40-1q Tesla P40 yes yes 24










Virtualization Benefits & Performance:vMotion for vGPUs Enabled VMs



Dell R730 – Intel Broadwell CPUs + 1 x NVIDIA P4040 cores (2 x 20-core socket) E5-2698 v4768 GB RAM

ESX: 6.7u1 NVIDIA Driver: 410.68

Dell R730 – Intel Broadwell CPUs + 1 x NVIDIA P4040 cores (2 x 20-core socket) E5-2698 v4768 GB RAM

Switch

vMotion for NVIDIA vGPU – Test-bed



Workload: SPECapc 3d Max 2015

Categories:

• Modelling

• Interactive Graphics

• Visual Effects

• GPU Rendering

• CPU Rendering

48 Tests:

• Underwater Animation

• Moving City

• Gizmo Transforms

• For complete list:

– Refer to https://www.spec.org/gwpg/apc.static/max2015info.html

43


https://www.spec.org/gwpg/apc.static/max2015info.html


vMotioning of Different vGPUs Running SPECapc



Six Concurrent vMotions of VMs Running SPECapc

45



Machine Learning Infrastructure in VMware vSphere


©2019 VMware, Inc.

Machine Learning Infrastructurein vSphere

NVIDIA vGPU

vGPUManagement

DirectPathIO

AutoScaling

DRS



Performance: Native GPU vs Virtual GPU



4% overhead for both vGPU & DirectPath I/O compared to native GPU

Performance: Training Times on Native GPU vs Virtualized GPU

11.04 1.04

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1 2 3

No

rmaliz

ed

Tra

inin

g T

ime

sL

ow

er

is b

ett

er

Language Modeling with RNN on PTB


©2019 VMware, Inc. 5016

Recommended for Virtualization

NVIDIA Data Center GPUs

V100 P40 T4 M10 P6

GPUs / Board

(Architecture)

1

(Volta)

1

(Pascal)

1

(Turing)

4

(Maxwell)

1

(Pascal)

CUDA Cores 5,120 3,840 2,5602,560

(640 per GPU)2,048

Tensor Cores 640 --- 320 --- ---

RT Cores --- --- 40 --- ---

Memory Size 32 GB/16 GB HBM2 24 GB GDDR5 16 GB GDDR632 GB GDDR5

(8 GB per GPU)16 GB GDDR5

vGPU Profiles

1 GB, 2 GB, 4 GB,

8 GB, 16 GB,

32 GB

1 GB, 2 GB, 3 GB,

4 GB, 6 GB, 8 GB,

12 GB, 24 GB

1 GB, 2 GB, 4 GB, 8 GB, 16

GB

0.5 GB, 1 GB, 2 GB,

4 GB, 8 GB

1 GB, 2 GB, 4 GB,

8 GB, 16 GB

Form FactorPCIe 3.0 Dual Slot & SXM2

(rack servers)

PCIe 3.0 Dual Slot

(rack servers)

PCIe 3.0 Single Slot (rack

servers)

PCIe 3.0 Dual Slot

(rack servers)

MXM

(blade servers)

Power 250W/300W 250W 70W 225W 90W

Thermal passive passive passive passive bare board

PERFORMANCEOptimized

DENSITYOptimized

BLADEOptimized



Turing T4 vs Pascal P40 vs Volta V100 Using Highest vGPU Profile

51

0

100

200

300

400

500

600

700

800

900

1 2 3

Tra

inin

g T

ime

s in

Se

co

nd

sL

ow

er

is b

ett

er

Training Times for Language Modelling Using RNN



rest-bed for NVIDIA on Horizon ViewML Using Containers in vSphere

52

Deep Learning Components• Machine Learning Workloads • TensorRT:19.02-py3• TensorRT-Server: 19.02-py3• TensorFlow: 1.10

Container in a VM Configuration• NVIDIA Docker: 18.09.1• vGPU T4-16Q• CentOS 7.4• ESX 6.X

Dell R730 – Intel Broadwell CPUs + Turing T4 GPU40 cores (2 x 12-core socket) E5-2698 V5 768 GB GB RAM

NVIDIA GPU CLOUD

(Container Repository)



Inferencing Workload: Image Classification Using ResNet50

Workload:– Image Classification

– 1000 classes/labels

Convolutional Neural Network

• ResNet: Residual Network

• Precision: FP 32

• 50 Layers

• Human Brain has similar structure

GPU: Turing T4 with 16GB of GPU Memory

• ResNet50 FP32 needs 2GB of GPU Memory

• T4-2Q profile => Max 8 Users Per T4 GPU

53



Fixed Share Scheduling: Image Classification Using NVIDIA TensorRT Server

54

1.00 1.00 1.00

1.16

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1 2 3 4

No

rmaliz

ed

La

ten

cy

Lo

we

r is

be

tte

r

# of VMs

Context Switching Overhead

1.00

2.00

4.00

7.00

7.71

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

1 2 3 4 5

No

rmaliz

ed

Th

rou

gh

pu

tH

igh

er

is b

ett

er

# of T4-2Q VMs

1xT4-16Q



Improving Inferencing Performance with Turing T4’s Tensor Cores

55



How to Improve Inference Latency, Throughput and Multi-tenancy Using TensorRT and vGPUs?

Uses FP32 and Needs 2 GB

Requires T4-2Q

Supports up to 8 Users on T4

Precision: FP16 or INT8 or INT4

Batch Size: specify a batch_size

Uses FP16 or INT8 or INT4

Needs 1 GB

Supports up to 16 Users on T4

Latency Improvements

Now we can support up to 16 Users on T4 with

Major Latency Improvements!VMworld 2019 Content: Not for publication or distribution


Virtualized GPUs deliver near bare metal performance

VMware vSphere support a full spectrum of workloads and users using GPUs and CPUs

Virtualization magnifies the benefits of lower and mixed precision features of Tensor Cores in GPUs by improving latency, throughput and multitenancy

For more consolidation and multitenancy, use vGPU solution

vGPUs enable concurrent diverse workloads like ML and Graphics

Huge advantage of vMotion and Suspend/Resume feature of vGPU-enabled VMs

VMware vSphere is also a great platform for traditional machine learning involving tabular datasets

VMware vSphere combines performance of GPUs and data center management featuresof virtualization

Key Takeaways



Extreme Performance Series: Sessions

HBI2526BU Performance Best Practices

BCA1482BU SQL Server, Oracle, and SAP Monster Database VMs

HBI2090BU vSphere Compute & Memory Schedulers

HBI2880BU DRS 2.0 Performance Deep Dive

HBI2090BU vSphere & Intel Optane DC PMEM=Max Performance

BCA1430BU Accelerating Application/Database Performance In the Self-Learning SDDC

HBI1421BU Innovations in vMotion: Features, Performance and Best Practices

BCA1393BU SAP HANA on vSphere 6.7u2 and Intel Cascade Lake Best Practices

MLA1594BU Optimize Virtualized Deep Learning Performance with New Intel Architectures

BCA1332BU The Different Forms of Machine Learning: How They Fit with VMware

BCA2551BU Low Latency Media & Entertainment Workloads

HCI1606BU SAP HANA on vSAN: Best Practice Recommendations and Lessons Learned

HCI1619BU Troubleshooting performance issues with vSAN Performance Diagnostics

BCA1563BU High Performance Virtualized Spark Clusters on Kubernetes for Deep Learning



Extreme Performance Series: Hands On Labs

SPL-2004-01-SDC

SPL-2004-02-CHG

ELW-2004-01-SDC

SPL-2047-01-EMT

SPL-2048-01-EMT

ELW-2048-02-EMT

SURVEY: TheVMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can improve:

www.vmware.com/go/perf

Mastering vSphere Performance

vSphere Challenge Lab

Expert Led Workshop: Mastering vSphere Performance

Accelerate Machine Learning in vSphere Using GPUs

Launch Your Machine Learning Workloads in Minutes on vSphere

Expert Led Workshop: Launch Your Machine Learning Workloads in Minutes on vSphere& Accelerate them using GPUs



VMmark ML*

• Prototype version of our popular VMmark virtualization benchmark that focuses on machine learning

• Features simple, push button deployment of Kubernetes cluster and ML applications

• Can start with single host, and scale to many

• Initial applications:

– MLPerf inference workloads

– Deep Learning image classification workloads

Continue to participate in development of MLPerf training and inference benchmarks

• See https://mlperf.org/

Further Machine Learning/Deep Learning Work

*External Release of VMmark ML will be covered in the future


https://mlperf.org/




Backup slides



Incorporate additional Kubernetes functionality into Spark tests

• Autoscaling, resource management, test on other K8s platforms

Continue to participate in development of MLPerf training and inference benchmarks

• See https://mlperf.org/

VMmarkML

• A new version of our popular VMmark virtualization benchmark that focuses on machine learning

• Features simple, push button deployment of Kubernetes cluster and ML applications

• Can start with single host, and scale to many

• Initial applications:

– MLPerf inference workloads

– Deep Learning image classification workloads used in this work

Further Machine Learning/Deep Learning Work


https://mlperf.org/

©2019 VMware, Inc.

Q& A and backup

Contact

Uday Kurkure [email protected]

Justin Murray [email protected]

Thanks to our colleagues

• Lan Vu, Hari Sivaraman, Juan Garcia-Rovetta, Ravi Soundararjan


mailto:[email protected]

mailto:[email protected]


Many Types of Neural Networks

• DNNs - Deep Neural Networks– Have multiple hidden layers, along with their input

and output layers

• CNNs - Convolutional Neural Networks – Particularly useful in image recognition

• RNNs – Recurrent Neural Networks – Used for speech recognition or NLP

`


Date post:	20-Feb-2022
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

The Different Forms of Machine Learning: How They Fit with ...

Documents