20
Containerizing Deep Learning Frameworks with Singularity Rengan Xu, Frank Han, Nishanth Dandapanthula HPC & AI Solutions Engineering, Dell EMC
Containerizing Deep Learning Frameworks with SingularityRengan Xu, Frank Han, Nishanth Dandapanthula
HPC & AI Solutions Engineering, Dell EMC
2 of 20
Agenda• Dell EMC HPC & AI Solutions Engineering
• Why use containers?
• Singularity Containers
Singularity vs Docker
Interpretability between Singularity vs Docker
Singularity workflow
• Containerizing DL frameworks
Issues and workarounds
eg. Caffe2
• Performance Results Horovod + TensorFlow
MXNet
Caffe2
3 of 20
Dell EMC HPC & AI Solutions Engineering
HeadingLorem ipsum dolor sit amet, consectetur adipiscing elit.
HeadingLorem ipsum
dolor sit amet, consectetur
adipiscing elit.
Design, develop and integrate HPC systems
Act as the focal point for joint R&D activities
HPC & AI Innovation
Lab
Prototype and evaluate advanced technologies
Conduct application performance studies and develop best practices
4 of 20
Containers and Virtualization Machine: A Recap
source: https://www.docker.com/what-container
• Container has no hypervisor• Container has no guest OS
5 of 20
Need for Containerization
• Why do we need containers?– Simplify application building– Application isolation– Faster application deployment– Validate and reproduce results– Server consolidation/Server efficiency– Can be deployed on bare metal or on virtual machines
• Benefits of Containers – Lightweight– Low overhead– Easier application sharing among users– Reproducibility
• Example containers– LXC– Docker– Singularity
6 of 20
Singularity Vs Docker
Feature Singularity DockerMultiple containers can be run on same hardware
Can be created and destroyed more quickly
Do not need entire OS, only a core run time
Transferable to other machines easily
Image format Single file Layered Image
Use with HPC schedulers X
Native Support for MPI X
Support for GPUs X
root owned Daemon process X
7 of 20
Singularity: Workflow Summary
source: http://singularity.lbl.gov/docs-flow
8 of 20
Interpretability between Singularity vs Docker
• Create Singularity image from Docker Hub• $ singularity pull docker://tensorflow/tensorflow
• Create Singularity image from Nvidia GPU Cloud Docker Registry• $ export SREGISTRY_NVIDIA_BASE="ngcr.io“• $ export SREGISTRY_CLIENT=nvidia• $ export SREGISTRY_NVIDIA_USERNAME='$oauthtoken‘• $ export SREGISTRY_NVIDIA_TOKEN='[NGC_API_KEY]‘• $ sregistry pull nvidia://tensorflow:17.11
9 of 20
Singularity MPI
• Has built-in support for all MPI implementations (OpenMPI, MPICH, Intel MPI, etc.)
• Host MPI version must be newer or equal to the version inside the container
• Example: – mpirun –np 4 singularity exec centos_ompi.img
/usr/bin/mpi_ring
9
source: https://wikihub.berkeley.edu/download/attachments/129695919/Containers_in_HPC_summary_Singularity.pdf
run on the host run in the container
10 of 20
Challenges and Workarounds
• Why containerize DL Frameworks– Every DL framework has too many dependences– Each dependent library has special version requirement– All DL frameworks are changing frequently– The friendly supported OS for most DL frameworks is Ubuntu, where as datacenter deployments are
RHEL/Centos
• Why we moved to singularity– Scaling containerized deep learning frameworks past a single node
• Issues faced with Singularity– PCIe device driver mismatch
• Workarounds– GPUs
› The container should always use the host GPU driver› Create a symbolic links for all GPU driver related files and then bind it to container› Update to latest drivers since they are backward compatible
– InfiniBand› The InfiniBand driver is kernel dependent, and the solution is to make the container OS and host OS compatible
and the container reuses the InfiniBand driver and libraries on the host
11 of 20
Singularity recipe for Caffe2
12 of 20
Building the container
13 of 20
Build Caffe2 inside the container
14 of 20
Run the container
• ${mpirun_options} ${profile_options} \
singularity exec -s /bin/bash \
-B $host_paths -B $PWD:/mnt \
-B /usr/lib64:/ibverb_libs -B /etc/libibverbs.d -B /sys/class/infiniband_verbs \
centos7_caffe2_dev_sandbox /mnt/caffe2_singularity_cmd.sh \
${WORK_DIR} ${gpu_arch} ${gpus_per_node} $network ${run_id} ${num_nodes} $epochs $profile $debug $mpi ) >& $train_log
15 of 20
Testbed
• 8 Dell EMC PowerEdge C4140 nodes.
– In process of updating to 32 nodes with NVLINK
• Nvidia V100-PCIe GPUs
• Intel Xeon Skylake CPU
• Mellanox 100Gbps EDR Infiniband
• CUDA 9.0, CUDNN 7.0, NCCL 2.0
• Dataset: ILSVRC 2012
16 of 20
Performance Results – MXNet
• In FP32 mode, batch size: 64 per GPU• In FP16 mode, batch size: 128 per GPU• IPoIB, rsync are used for nodes
communication • Speedup of 32 V100 is 29.4x in FP32 and
25.8x in FP16
Performance difference between Singularity vs bare-metal
-0.8% 0.4% -1.9%-1.1%
-0.7%
-0.7%
-1.1% -1.1%0.3%
0.5%
-0.3%
0.2%
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
1 V100 2 V100 4 V100 8 V100 16 V100 32 V100Im
aegs
/sec
MXNet Resnet50
FP32 Singularity FP32 bare-metal FP16 Singularity FP16 bare-metal
17 of 20
Performance Results – Horovod + TensorFlow
• In FP32 mode, batch size: 128 per GPU• In FP16 mode, batch size: 256 per GPU• MPI used for multi-node communication• Speedup of 32 V100 is 22.4x in FP32 and
23.7x in FP16
Performance difference between Singularity vs bare-metal
-1.6% 0.3% 0.6%0.0%
0.4%
1.2%
-1.6% 0.1%0.2%
0.2%
-0.3%
0.3%
0
2000
4000
6000
8000
10000
12000
14000
16000
1 V100 2 V100 4 V100 8 V100 16 V100 32 V100Im
ages
/sec
Horovod+TensorFlow Resnet50
FP32 Singularity FP32 bare-metal FP16 Singularity FP16 bare-metal
18 of 20
Performance Results – Caffe2
• In FP32 mode, batch size: 64 per GPU• In FP16 mode, batch size: 128 per GPU• Redis and IPoIB are used for nodes
communication• Caffe2 performance unstable on multiple
nodes
Performance difference between Singularity vs bare-metal
0.0%-0.5%
-0.3%
4.0%2.4%
-0.4%
0.2%
-0.2%
-0.1% -1.9%
-0.1%
-5.1%
0
500
1000
1500
2000
2500
3000
3500
4000
1 V100 2 V100 4 V100 8 V100 16 V100 32 V100Im
ages
/sec
Caffe2 Resnet50
FP32 Singularity FP32 bare-metal FP16 Singularity FP16 bare-metal
19 of 20
Conclusions and Future Work
• Conclusions– Singularity simplifies the building and deployment of DL in both single-node and multi-node
– Easy to use Singularity on GPU server
– Straightforward to run MPI on InfiniBand interconnect
– No performance loss compared to bare-metal
• Future Work– File system impact for DL models
– Scale impact for DL model accuracy
– Research on neural networks with model parallelism
– Case studies with appropriate DL models
• Build Optimal Solutions targeted to DL vertical.
www.hpcatdell.com
{Rengan.Xu,Frank.Han1,Nishanth.Dandapanthu}@Dell.com
![^ XE } X } v P } Ç E u } ( µ v ] h v ] r u ] o / í í õ l í ...](https://static.documents.pub/doc/80x56/622f6c617e88da2277563c60/-xe-x-v-p-e-u-v-h-v-r-u-o-.jpg)
![URegister 05 Activate Students New · 2019. 9. 17. · C w { v x 5 x v { { ? v r r hZ P ] v r L { y À í X í G x ô 2 X E } r s s r r x r](https://static.documents.pub/doc/80x56/60e5d1f777c2452fb20468fb/uregister-05-activate-students-new-2019-9-17-c-w-v-x-5-x-v-v-r-r-hz.jpg)
![E } W > ] h } v W í ñ rE } À r î ì î í í X d Z o ] } ( Z ...](https://static.documents.pub/doc/80x56/623bf0f07304b82c3a7388a5/e-w-gt-h-v-w-re-r-.jpg)
![d Z E Æ D i } / v ] v & ] o ] Ç t ] o o D /E WK/Ed í K µ Ç ... · r r r r r r r r/ u u µ v Ç u r r r r r r ^ Z o } v o X U ] } o X Z } X ô í W í ì î ñ ~ î ì ì õ ,QFLGHQFH](https://static.documents.pub/doc/80x56/5c02669d09d3f2ab198b65dc/d-z-e-a-d-i-v-v-o-c-t-o-o-d-e-wked-i-k-c-r-r-r-r.jpg)
![d Z ] u ] o Á Ç t s ] P ] v ] D ] o / v µ v ] ] ] µ Ç , o ... · G D^ } o ] Ç X W µ o ] } v E } X r í í^Kt r X í r í î ì ð î ì í ð r ì î HHQI’s Progressive Cardiovascular](https://static.documents.pub/doc/80x56/5b9ba2ac09d3f2aa588d7583/d-z-u-o-a-c-t-s-p-v-d-o-v-v-c-o-g-d-o.jpg)
![¹ B>1>. º Ø $S .x b á$×757V / D b Ì) $0¯ ¦ B Ì i...>&>1>' ) £%43 Ø È#Ø)r /¡ å ± î3 %Ê'2 d í í í í í í í í í í í í >/>.>1 >&>2>'%¼ ]%43° Ø)r %Ê'2 å](https://static.documents.pub/doc/80x56/5f099f757e708231d427b839/-b1-s-x-b-757v-d-b-oe-0-b-oe-i-1.jpg)
![î ì í í r î ì í ô } l ] o o Z ] } } l...î ì í í r î ì í ô } l ] o o z ] } } l î d o } ( } v v t o } u j x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x](https://static.documents.pub/doc/80x56/61090d284fa0f111dc4afd02/-r-l-o-o-z-l-r-l.jpg)
![E ^ Z < W/d > ] v µ ] v r ] v À u v ( ] u Z ( } o o } Á v ... · ¨ r ¨ î ¨ ð ¨ ò ¨ ô ¨ í ì ¨ í î ¨ r ¨ ì X ñ ¨ í X ì ¨ í X ñ ¨ î X ì ¨ î X](https://static.documents.pub/doc/80x56/5c02669c09d3f2ab198b65c3/e-z-wd-v-v-r-v-a-u-v-u-z-o-o-a-v-r-i.jpg)
![GBI TECHNOLOGY Pinças ISO e HSK para Spindle€¦ · GBI Technology Eireli - EP P À X / v } ^ ] o À s v v ] } U í õ õ r ^ o í ò r s } } v u l^W W í ô í í í r í ì ì](https://static.documents.pub/doc/80x56/5f9c93290cd80d0fc2650e00/gbi-technology-pinas-iso-e-hsk-para-gbi-technology-eireli-ep-p-x-v-.jpg)
![Drug Errors 2018 NWAS v1 JRAC-DIN - regional anesthesia · 2020. 8. 3. · í ôsZ í D ] ] } v } } µ P } ] v v Z ] X ~ î ì í ô r ô r í ô X Z } Z Á D í ôsZ î î X K :](https://static.documents.pub/doc/80x56/5ff6c8f7e98c6c1120189cc2/drug-errors-2018-nwas-v1-jrac-din-regional-anesthesia-2020-8-3-sz-.jpg)
![s ] Z v µ , } unityagni.com/.../Vishnu-Homam-Sanskrit-Regular.pdf · & , } u K v o ] v / v r } v o Á Á Á X v ] Ç P v ] X } u } v W = õ í õ î ô ï í í í í í ï](https://static.documents.pub/doc/80x56/5f80ff35cdf48776b064476e/s-z-v-u-k-v-o-v-v-r-v-o-x-v-p-v-x-u-.jpg)
