Singularity A brief history

• Invented by Greg Kurtzer at LBNL to address problems specific to HPC

• Development began in October 2015

• First release April 2016

• User feedback, major changes and help from the community leads to 2.0 in June 2016

• Greg founds Sylabs.io January 1, 2018; already at 28 employees, product and revenue!

• Sylabs developers completely overhaul code base, create a suite of cloud services, and release 3.0 October 8, 2018!

Widely Adopted for Compute Needs

Singularity is the container runtime of choice for compute based workloads like

simulation and AI

As of Singularity 2.x…

● Millions container runs per day● Estimated user base of more than 25k● Installed on over 3 million sockets

HPCWire Readers and Editors Choice Awards:

● 2016: Top products to watch● 2017: Top products to watch● 2017: Best HPC Programming Tool/Tech

The Uptake of Singularity has been even faster than CentOS!

Singularity is Designed Specifically for HPC

● Compatible with traditional HPC architecture and workflows

● Supports all known resource managers

● Optimized for compute jobs and performance

● Supports GPUs, IB, OmniPath, MPI, parallel file systems, production kernels, etc...

● Users are themselves within the container

● Allows users to “BYOE”

● Blocks privilege escalation within the container

Simplicity

Singularity is designed to be simple. From its build syntax, to its image format, the runtime, and compatibility with OCI and Docker, Singularity facilitates easy and

intuitive onboarding.

SIF can squashfs, encapsulate and sign Docker and OCI image formats (encryption coming soon)

Docker, OCI, etc... Singularity

Reproducibility and Mobility

Singularity uses binary and immutable images, encapsulating the entire runtime stack so it is always 100% reproducible.

Host 1 Host 2

SCP, FTP, HTTP,Sylabs Cloud

Designed for HPC and up!

From your laptop, to HPC compute, to the cloud and edge, Singularity

facilitates extreme mobility of compute.

Singularity provides an onramp across all resources, enterprises, and clouds.

Singularity Runtime Features

Accelerated GPUs

Designed For Performance

Supports GUIs Resource Manager Agnostic

Secure

Standards Compliant

Jobs and ServicesCompatible with HPC

A year Update on Singularity

Challenges using/converting to Go

● CGo Interface :

○ Forking in Go. Certain syscalls should not be called in a multi threaded application.

● Following Go Standards (packaging, internal)

● Vendoring… dep, go mod, vendor/

● Packaging

○ Custom packaging tool makeit.

○ RPM

○ upcoming….DEB

Wins & Gains from converting to Go

● Integration with other container projects. Easy to use packages from Kubernetes, Docker/Moby, OCI etc. No need to rebuild in Python or C.

● Concurrency model of Go.

● Go is easy to learn for developers coming from other languages.

● We used CNI rather than creating our own networking stack.

Comparing Go to previous Languages (Python etc)

● Large ecosystem of packages for backend and system projects, cloud APIs.

● go test - Easy standards and framework for setting up tests.

● Easy to be productive in Go. Well designed language.

● Go is opinionated (Good and Bad). Works well but you have to follow the standards. Easy to read other codebases.

Singularity Container Format Features

Guaranteed Reproducible Archival Mobile Controls Compliant

What’s new in 3.0ー　Cloud Service

Container Library• Like GitHub/DockerHub, Manage Container on Cloud, User can Pull/Push。Also

support Private Container。

Remote Builder• Sylabs provide Remote Builder Service on Cloud to the user who don’t have

privilege or environment.

KeyStore Service

• Can manage GPG Compatible Key

Container Libraryhttps://cloud.sylabs.io/library

• Pull/Push Images• Wild card search• Star• Download

• Create token - https://cloud.sylabs.io/auth• singularity push ./def.sif library://jialipassion/default/demo1.sif:latest• singularity pull demo1.sif library://jialipassion/default/demo1.sif:latest• singularity search demo1

Container LibraryDemo

Remote Buildhttps://cloud.sylabs.io/builder

• Live Output• 11GB Space

• Create token - https://cloud.sylabs.io/auth

• singularity build --remote library://sylabsed/examples/lolcow.sif:1.0 lolcow.def

• Build Definition File How To• https://www.sylabs.io/guides/3.0/user-guide/definition_files.html

Remote BuilderDemo

KeyServicehttps://cloud.sylabs.io/keystore

• GPG Compatiable• Push/Pull

• Create token - https://cloud.sylabs.io/auth• singularity keys newpair• singularity keys list• singularity keys push• singularity keys search (name/email)

KeystoreWorkflow

Container Image Integrity

Singularity container runtimes are immutable, cryptographically signed, and verifiable, ensuring absolute trust of the container environment.

Note: Singularity uses no tarballs and thus no unsigned intermediate data

• Using CNI Plugins, implement Network Virtualization。 https://www.sylabs.io/guides/3.0/user-guide/networking.html

• --dns • nslookup sylabs.io | grep Server• sudo singularity exec --dns 8.8.8.8 ubuntu.sif nslookup sylabs.io | grep Server

• --hostname• --net

• hostname -I• sudo singularity exec --net my_container.sif hostname -I

• --network• bridge• ptp• ipvlan• macvlan• User can also define their own network type

• --network-args• sudo singularity instance start --writable-tmpfs --net --network-args "portmap=8080:80/tcp"

docker://nginx web2• sudo singularity exec instance://web2 nginx• curl localhost:8080

Network VirtualizationCNI https://github.com/containernetworking/cni

• Singularity3.0 provide a lot of security functionalities for Linux Containers• --add-caps/--drop-caps

• Ping• --allow-setuid

• enable setUID (disabled by default)• --keep-privs

• When singularity.conf is being modified、can use this flag to run in full privileges

• --security• Can use the security functions from SELinux/AppArmor/seccomp• For example、using seccomp to black list some commands, you can do like

the manual in this document https://www.sylabs.io/guides/3.0/user-guide/security_options.html

Security

Native Support for cgroups

• Can use cgroups options• Sample

[memory]　　 limit = 524288000[cpu] share = 512 period = 100000 quota = 20000 cpus = “0-1” mems = “0-1”[blockIO] weight = 1000 leftWeight = 1000[[devices]] access = “rwm” allow = false　

Sign/Verify

• Can sign/verify with user key• singularity sign ubuntu.sif• singularity verify ubuntu.sif• (delete key: rm ~/.singularity/sypgp/*) singularity verify ubuntu.sif

Runtime Plugins

• plugin list• plugin compile• plugin install• plugin unable• plugin uninstall• plugin inspect

MacOS

KubernetesSingularity-CRI

Meta-SchedulingCloud native HPC?

NomadCloud native HTC

We are hiring!

• https://www.sylabs.io/about-us/we-are-hiring/

#SUG19

@SylabsIO@CarlosEArango

Support slides

$ cat recipe.defBootstrap: yumOSVersion: 7MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/x86_64/Include: yum

%postyum -y install vim-minimal

%runscriptexec /usr/bin/vi “$@”$ singularity build centos-vim.sif recipe.def…

Building a Container with Singularity

# You can use the container as follows...$ singularity shell centos-vim.sif$ singularity exec centos-vim.sif vim testfile.txt$ singularity run centos-vim.sif testfile.txt

# Singularity containers are also executable, so you can ‘run’ them directly$ ./centos-vim.sif testfile.txt

# You can move the container onto any other Linux system with Singularity installed, and use the container directly$ scp centos-vim.sif user@examplehost.com:$ ssh user@examplehost.com$ ./centos-vim.sif newtestfile.txt

Example Usage of Singularity

$ singularity sign centos-vim.sif Signing image: centos-vim.sifNo OpenPGP signing keys found, autogenerate? [Y/n] yEnter your name (e.g., John Doe) : GregEnter your email address (e.g., john.doe@example.com) : g@sylabs.ioEnter optional comment (e.g., development keys) : demokeysGenerating Entity and OpenPGP Key Pair... DoneEnter encryption passphrase :…Uploaded key successfully!Enter key passphrase: Signature created and applied to centos-vim.sif$

Cryptographically signed containers

$ singularity push centos-vim.sif library://gmk/demo/centos-vim:latestINFO: Now uploading centos-vim.sif to the library 108.16 MiB / 108.16 MiB [=============================] 100.00% 13.75 MiB/s 7sINFO: Setting tag latest$

Using the Sylabs Container Library

$ singularity pull library://gmk/demo/centos-vim:latest 108.16 MiB / 108.16 MiB [=============================] 100.00% 35.00 MiB/s 3s$ singularity verify centos-vim_latest.sif Verifying image: centos-vim_latest.sifINFO: key missing, searching key server for KeyID: 58D8405A30E12DE6...INFO: key retreived successfully!Store new public key F56D95BD3AFAC6FA3423911A58D8405A30E12DE6? [Y/n] yData integrity checked, authentic and signed by:

Greg (demokeys) <g@sylabs.io>, KeyID 58D8405A30E12DE6$

Pulling and Validating a Container

# Building a Singularity container (SIF) from DockerHub$ singularity build python.sif docker://python:latest…# Running a shell directly from DockerHub$ singularity shell docker://ubuntu:latestSingularity ubuntu_latest.sif:~/demo> cat /etc/lsb-release DISTRIB_ID=UbuntuDISTRIB_RELEASE=18.04DISTRIB_CODENAME=bionicDISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"Singularity ubuntu_latest.sif:~/demo> exit$ singularity exec docker://centos:latest cat /etc/redhat-releaseCentOS Linux release 7.5.1804 (Core)

Working with Docker

$ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu pythonPython 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import tensorflow as tf>>> x1 = tf.constant([1,2,3,4])>>> x2 = tf.constant([5,6,7,8])>>> result = tf.multiply(x1, x2)>>> print(result)Tensor("Mul:0", shape=(4,), dtype=int32)>>> exit()$

Accessing the Host’s GPU With Tensorflow

SINGULARITY IMAGES

• Singularity uses a recipe file to describe the contents of the container

• Containers created from prebuilt sources at DockerHub or SingularityHub, or your own repository

• Package, Deploy, Visualize with reproducible results

BootStrap: yumOSVersion: 7MirrorURL:<http…>Include: yum

# If you want updates then uncomment#UpdateURL:<http…>

%runscript echo "This is what happens when you run the container..."

%post echo "Hello from inside the container"

yum -y install vim-minimal

Singularity was designed with a different set of goals, that fits elegantly with other

packaging platforms for Business Intelligence and Big Data Analytics

ECOSYSTEM - OPPORTUNITY

VIRTUAL MACHINES

ENTERPRISE PERFORMANCE COMPUTING

Deep Learning Data Science

Artificial IntelligenceCompute Driven Analytics

ScienceNon-services

MPI Matlab

TensorflowInfiniBand

OmniPath

MICRO-SERVICE DERIVED

Singularity is compatible

43

• Scheduler and Resource Manager work together to orchestrate jobs in HPC

• Traditional container daemons operate outside context of the Resource Manager

• Singularity maintains original execution context

• Jobs running within Singularity containers are just like any other jobs!

ContainedProgram / Workflow

User Privilege and PID Context

Singularity

Scheduler

Container Daemon

runs as root

User’s shell / Batch Script

ContainedProgram / Workflow

HPC Resource Manager

with resource managers

Singularity / Sylabs Cloud

• The container services platform for hybrid cloud solutions• Simplify workflows, leverage signed and encrypted containers• Designed for continuous integration development practices• Brought to you by the Singularity Image Format; a single file

container package• Sylabs Cloud, your one-stop container ecosystem

ECOSYSTEM

Sylabs Cloud Value added ecosystem for Singularity

• Container Build Services• On-prem/Off-prem• Enable users to build images on secure systems

• Container Library• Object store backed service for container distribution and storage• Constant container scanning, notification, and management for DevSecOps

• Cryptographic Key Management• Singularity supports cryptographically signed containers• Trusted source of public keys• Maintainer keys are downloaded and tested on demand to validate containers

CONTAINER SERVICES

Sylabs and SingularityPROStable, Secure, Production ready

• Fully committed to open source

• SingularityPRO is to Singularity as RHEL is to Fedora

• LTS for several releases

• Stability through backported bug fixes and features

• Security though advanced security patches

• Full access to Sylabs cloud infrastructure and option for on-prem hosting