Post on 31-Dec-2020
transcript
Singularity A brief history
• Invented by Greg Kurtzer at LBNL to address problems specific to HPC
• Development began in October 2015
• First release April 2016
• User feedback, major changes and help from the community leads to 2.0 in June 2016
• Greg founds Sylabs.io January 1, 2018; already at 28 employees, product and revenue!
• Sylabs developers completely overhaul code base, create a suite of cloud services, and release 3.0 October 8, 2018!
Widely Adopted for Compute Needs
Singularity is the container runtime of choice for compute based workloads like
simulation and AI
As of Singularity 2.x…
● Millions container runs per day● Estimated user base of more than 25k● Installed on over 3 million sockets
HPCWire Readers and Editors Choice Awards:
● 2016: Top products to watch● 2017: Top products to watch● 2017: Best HPC Programming Tool/Tech
The Uptake of Singularity has been even faster than CentOS!
Singularity is Designed Specifically for HPC
● Compatible with traditional HPC architecture and workflows
● Supports all known resource managers
● Optimized for compute jobs and performance
● Supports GPUs, IB, OmniPath, MPI, parallel file systems, production kernels, etc...
● Users are themselves within the container
● Allows users to “BYOE”
● Blocks privilege escalation within the container
Simplicity
Singularity is designed to be simple. From its build syntax, to its image format, the runtime, and compatibility with OCI and Docker, Singularity facilitates easy and
intuitive onboarding.
SIF can squashfs, encapsulate and sign Docker and OCI image formats (encryption coming soon)
Docker, OCI, etc... Singularity
Reproducibility and Mobility
Singularity uses binary and immutable images, encapsulating the entire runtime stack so it is always 100% reproducible.
Host 1 Host 2
SCP, FTP, HTTP,Sylabs Cloud
Designed for HPC and up!
From your laptop, to HPC compute, to the cloud and edge, Singularity
facilitates extreme mobility of compute.
Singularity provides an onramp across all resources, enterprises, and clouds.
Singularity Runtime Features
Accelerated GPUs
Designed For Performance
Supports GUIs Resource Manager Agnostic
Secure
Standards Compliant
Jobs and ServicesCompatible with HPC
A year Update on Singularity
Challenges using/converting to Go
● CGo Interface :
○ Forking in Go. Certain syscalls should not be called in a multi threaded application.
● Following Go Standards (packaging, internal)
● Vendoring… dep, go mod, vendor/
● Packaging
○ Custom packaging tool makeit.
○ RPM
○ upcoming….DEB
Wins & Gains from converting to Go
● Integration with other container projects. Easy to use packages from Kubernetes, Docker/Moby, OCI etc. No need to rebuild in Python or C.
● Concurrency model of Go.
● Go is easy to learn for developers coming from other languages.
● We used CNI rather than creating our own networking stack.
Comparing Go to previous Languages (Python etc)
● Large ecosystem of packages for backend and system projects, cloud APIs.
● go test - Easy standards and framework for setting up tests.
● Easy to be productive in Go. Well designed language.
● Go is opinionated (Good and Bad). Works well but you have to follow the standards. Easy to read other codebases.
Singularity Container Format Features
Guaranteed Reproducible Archival Mobile Controls Compliant
What’s new in 3.0ー Cloud Service
Container Library• Like GitHub/DockerHub, Manage Container on Cloud, User can Pull/Push。Also
support Private Container。
Remote Builder• Sylabs provide Remote Builder Service on Cloud to the user who don’t have
privilege or environment.
KeyStore Service
• Can manage GPG Compatible Key
Container Libraryhttps://cloud.sylabs.io/library
• Pull/Push Images• Wild card search• Star• Download
• Create token - https://cloud.sylabs.io/auth• singularity push ./def.sif library://jialipassion/default/demo1.sif:latest• singularity pull demo1.sif library://jialipassion/default/demo1.sif:latest• singularity search demo1
Container LibraryDemo
Remote Buildhttps://cloud.sylabs.io/builder
• Live Output• 11GB Space
• Create token - https://cloud.sylabs.io/auth
• singularity build --remote library://sylabsed/examples/lolcow.sif:1.0 lolcow.def
• Build Definition File How To• https://www.sylabs.io/guides/3.0/user-guide/definition_files.html
Remote BuilderDemo
KeyServicehttps://cloud.sylabs.io/keystore
• GPG Compatiable• Push/Pull
• Create token - https://cloud.sylabs.io/auth• singularity keys newpair• singularity keys list• singularity keys push• singularity keys search (name/email)
KeystoreWorkflow
Container Image Integrity
Singularity container runtimes are immutable, cryptographically signed, and verifiable, ensuring absolute trust of the container environment.
Note: Singularity uses no tarballs and thus no unsigned intermediate data
• Using CNI Plugins, implement Network Virtualization。 https://www.sylabs.io/guides/3.0/user-guide/networking.html
• --dns • nslookup sylabs.io | grep Server• sudo singularity exec --dns 8.8.8.8 ubuntu.sif nslookup sylabs.io | grep Server
• --hostname• --net
• hostname -I• sudo singularity exec --net my_container.sif hostname -I
• --network• bridge• ptp• ipvlan• macvlan• User can also define their own network type
• --network-args• sudo singularity instance start --writable-tmpfs --net --network-args "portmap=8080:80/tcp"
docker://nginx web2• sudo singularity exec instance://web2 nginx• curl localhost:8080
Network VirtualizationCNI https://github.com/containernetworking/cni
• Singularity3.0 provide a lot of security functionalities for Linux Containers• --add-caps/--drop-caps
• Ping• --allow-setuid
• enable setUID (disabled by default)• --keep-privs
• When singularity.conf is being modified、can use this flag to run in full privileges
• --security• Can use the security functions from SELinux/AppArmor/seccomp• For example、using seccomp to black list some commands, you can do like
the manual in this document https://www.sylabs.io/guides/3.0/user-guide/security_options.html
Security
Native Support for cgroups
• Can use cgroups options• Sample
[memory] limit = 524288000[cpu] share = 512 period = 100000 quota = 20000 cpus = “0-1” mems = “0-1”[blockIO] weight = 1000 leftWeight = 1000[[devices]] access = “rwm” allow = false
Sign/Verify
• Can sign/verify with user key• singularity sign ubuntu.sif• singularity verify ubuntu.sif• (delete key: rm ~/.singularity/sypgp/*) singularity verify ubuntu.sif
Runtime Plugins
• plugin list• plugin compile• plugin install• plugin unable• plugin uninstall• plugin inspect
MacOS
KubernetesSingularity-CRI
Meta-SchedulingCloud native HPC?
NomadCloud native HTC
We are hiring!
• https://www.sylabs.io/about-us/we-are-hiring/
#SUG19
@SylabsIO@CarlosEArango
Support slides
$ cat recipe.defBootstrap: yumOSVersion: 7MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/x86_64/Include: yum
%postyum -y install vim-minimal
%runscriptexec /usr/bin/vi “$@”$ singularity build centos-vim.sif recipe.def…
Building a Container with Singularity
# You can use the container as follows...$ singularity shell centos-vim.sif$ singularity exec centos-vim.sif vim testfile.txt$ singularity run centos-vim.sif testfile.txt
# Singularity containers are also executable, so you can ‘run’ them directly$ ./centos-vim.sif testfile.txt
# You can move the container onto any other Linux system with Singularity installed, and use the container directly$ scp centos-vim.sif user@examplehost.com:$ ssh user@examplehost.com$ ./centos-vim.sif newtestfile.txt
Example Usage of Singularity
$ singularity sign centos-vim.sif Signing image: centos-vim.sifNo OpenPGP signing keys found, autogenerate? [Y/n] yEnter your name (e.g., John Doe) : GregEnter your email address (e.g., john.doe@example.com) : g@sylabs.ioEnter optional comment (e.g., development keys) : demokeysGenerating Entity and OpenPGP Key Pair... DoneEnter encryption passphrase :…Uploaded key successfully!Enter key passphrase: Signature created and applied to centos-vim.sif$
Cryptographically signed containers
$ singularity push centos-vim.sif library://gmk/demo/centos-vim:latestINFO: Now uploading centos-vim.sif to the library 108.16 MiB / 108.16 MiB [=============================] 100.00% 13.75 MiB/s 7sINFO: Setting tag latest$
Using the Sylabs Container Library
$ singularity pull library://gmk/demo/centos-vim:latest 108.16 MiB / 108.16 MiB [=============================] 100.00% 35.00 MiB/s 3s$ singularity verify centos-vim_latest.sif Verifying image: centos-vim_latest.sifINFO: key missing, searching key server for KeyID: 58D8405A30E12DE6...INFO: key retreived successfully!Store new public key F56D95BD3AFAC6FA3423911A58D8405A30E12DE6? [Y/n] yData integrity checked, authentic and signed by:
Greg (demokeys) <g@sylabs.io>, KeyID 58D8405A30E12DE6$
Pulling and Validating a Container
# Building a Singularity container (SIF) from DockerHub$ singularity build python.sif docker://python:latest…# Running a shell directly from DockerHub$ singularity shell docker://ubuntu:latestSingularity ubuntu_latest.sif:~/demo> cat /etc/lsb-release DISTRIB_ID=UbuntuDISTRIB_RELEASE=18.04DISTRIB_CODENAME=bionicDISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"Singularity ubuntu_latest.sif:~/demo> exit$ singularity exec docker://centos:latest cat /etc/redhat-releaseCentOS Linux release 7.5.1804 (Core)
Working with Docker
$ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu pythonPython 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import tensorflow as tf>>> x1 = tf.constant([1,2,3,4])>>> x2 = tf.constant([5,6,7,8])>>> result = tf.multiply(x1, x2)>>> print(result)Tensor("Mul:0", shape=(4,), dtype=int32)>>> exit()$
Accessing the Host’s GPU With Tensorflow
SINGULARITY IMAGES
• Singularity uses a recipe file to describe the contents of the container
• Containers created from prebuilt sources at DockerHub or SingularityHub, or your own repository
• Package, Deploy, Visualize with reproducible results
BootStrap: yumOSVersion: 7MirrorURL:<http…>Include: yum
# If you want updates then uncomment#UpdateURL:<http…>
%runscript echo "This is what happens when you run the container..."
%post echo "Hello from inside the container"
yum -y install vim-minimal
Singularity was designed with a different set of goals, that fits elegantly with other
packaging platforms for Business Intelligence and Big Data Analytics
ECOSYSTEM - OPPORTUNITY
VIRTUAL MACHINES
ENTERPRISE PERFORMANCE COMPUTING
Deep Learning Data Science
Artificial IntelligenceCompute Driven Analytics
ScienceNon-services
MPI Matlab
TensorflowInfiniBand
OmniPath
MICRO-SERVICE DERIVED
Singularity is compatible
43
• Scheduler and Resource Manager work together to orchestrate jobs in HPC
• Traditional container daemons operate outside context of the Resource Manager
• Singularity maintains original execution context
• Jobs running within Singularity containers are just like any other jobs!
ContainedProgram / Workflow
User Privilege and PID Context
Singularity
Scheduler
Container Daemon
runs as root
User’s shell / Batch Script
ContainedProgram / Workflow
HPC Resource Manager
with resource managers
Singularity / Sylabs Cloud
• The container services platform for hybrid cloud solutions• Simplify workflows, leverage signed and encrypted containers• Designed for continuous integration development practices• Brought to you by the Singularity Image Format; a single file
container package• Sylabs Cloud, your one-stop container ecosystem
ECOSYSTEM
Sylabs Cloud Value added ecosystem for Singularity
• Container Build Services• On-prem/Off-prem• Enable users to build images on secure systems
• Container Library• Object store backed service for container distribution and storage• Constant container scanning, notification, and management for DevSecOps
• Cryptographic Key Management• Singularity supports cryptographically signed containers• Trusted source of public keys• Maintainer keys are downloaded and tested on demand to validate containers
CONTAINER SERVICES
Sylabs and SingularityPROStable, Secure, Production ready
• Fully committed to open source
• SingularityPRO is to Singularity as RHEL is to Fedora
• LTS for several releases
• Stability through backported bug fixes and features
• Security though advanced security patches
• Full access to Sylabs cloud infrastructure and option for on-prem hosting