Containerize all the things
Agenda● Part 1: The concept of containers ● Part 2: Threats and Vulnerabilities● Part 3: Mitigating threats and best practices● Part 4: Take aways
Part 1: Docker on a calm see
What are containers ?● A way to isolate applications/processes from other
collocated processes or the host they are running on● Enables the packaging of an application and all its
dependencies● Portable unit: can run on a boat, truck, be lifted by a
tower crane,...
Containers’ history
Namespaces and cgroups● Namespaces = kernel-enforced view on different objects
from user space:– File system (mnt userspace)– Running processes (pid namespace)– Network stack (net namespace)– Users (user namespace)– Interprocess communication (ipc namespace)– Hostname configuration (uts namespace)
Namespaces and cgroups● cgroups = enforces limitation/s on a/group of
processe/s:– Memory– CPU– Network bandwidth– Disks I/O
Docker ?● A complete container management toolbox● Enables you to
– Build container images (with open specifications)– Execute/Run images as containers– Manage images and containers– Share and publish images
Docker images● Container Images are:
– A standardized unit of software, packaging● Code● Dependencies
– Packaging units/Images are● Consistent● Portable
“Package once, run everywhere”
Docker Containers● Docker Containers are:
– An instantiation of an image– Process(es) running in an isolated environment
● Docker containers should be:– Ephemeral– Stateless
“Dies like Frankenstein, resurrects as a baby”
Docker components
Containers vs Virtual Machines
Building a beautilful app
Why so hipster?
“Mature” technology with a rich ecosystem, changing the game on:
– Development and test automation– Integration and deployment– Software architecture– Infrastructure architecture
Part 2 – Containers Attack surface
Docker components
Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS
– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Usage of untrusted container images (e.g.: CVE-2019-5736)– Unlimited resources assignment
● Container abuse: Botnets, Resources abuses, (D)DoS minion– Application vulnerability– Image vulnerability– Usage of untrusted images
Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS
– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment
● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image vulnerability– Usage of untrusted images
Usage of dangerous switches:
● --privileged
● --cap-add SYS_ADMIN
● --pid=host, --network=host,…
● --user root
Usage of dangerous volumes mappings● -v /:/tmp/a
Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS
– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment
● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image vulnerability– Usage of untrusted images
Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS
– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment
● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image ecosystem vulnerability– Usage of untrusted images
Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS
– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment
● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image eco system vulnerability– Usage of untrusted images
CVE-2019-5736
Source: https://github.com/Frichetten/CVE-2019-5736-PoC
Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS
– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment
● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image ecosystem vulnerability– Usage of untrusted images
Source: https://blog.docker.com
Threats and Vulnerabilities
Source: https://blog.docker.com
Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS
– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment
● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image ecosystem vulnerability– Usage of untrusted images
Source: https://cvedetails.com
Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS
– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment
● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image eco system vulnerability– Usage of untrusted images
Containers with unlimited resources:
● CPU
● Memory
● Network Bandwidth
could abuse these resources and bring
the underlying system on its knees
Source: https://blog.docker.com
Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS
– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment
● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image eco system vulnerability– Usage of untrusted images
Containers do not magically
● Make your application flawless
● Remove Programming errors
● Patch dependencies and libraries
● ...
Threats and Vulnerabilities● Host abuse: Escape to host and Host DoS
– Poor container runtime isolation– Poor container network isolation– Unprotected runtime API– Unpatched container runtime– Unpatched Kernel/OS– Unlimited resources assignment
● Container abuse: Botnets, Resources abuses,..– Application vulnerability– Image eco system vulnerability– Usage of untrusted images
Containers do not magically
● Protect yourself from avoiding running
Untrusted components
Part 3 – Controls and mitigation
(s)SDLC
Build phase● During build
– Use scanned/tested images– Integrate images in the SDLC– Use small footprint images– Use a trusted source
● At runtime– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits
Security of images should be assessed to
● Identify unsafe settings in images manifests
● Identify vulnerable binaries/libraries
● Perform compliance checks on images
Possible solutions:
● CoreOS Clair
● Anchore
● Aqua Microscanner
● OpenScap
Build phase
Image source: https://anchore.com
Build phase● During build
– Use scanned/tested images– Integrate images in the SDLC– Use small footprint images– Use a trusted source
● At runtime– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits
How to ensure only “compliant” images are used
● Images manifest must be version controlled
● Use a private/dedicate image registry
● Security tests could make the build fail
● Only a successful build can push to the registry
● Users can only pull from registry
And now, what about tomorrow for:
● Running containers
● Already validated images
Build phase
Image source: https://www.anchore.com
Reducing the attack surface● During build
– Use scanned/tested images– Integrate images in the SDLC– Use small footprint images– Use a trusted source
● At runtime– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits
Build you images using minimal parent images
● scratch
● distroless-based
● Alpine based
Reducing attack surface● During build
– Use scanned/tested images– Integrate images in the SDLC– Use small footprint images– Use a trusted source
● At runtime– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits
Issue with minimal images (non-distro based)
● Debug sessions are tricky
● More difficult to maintain
● Scanners could fail to identify vulnerabilities
Reducing attack surface● During build
– Use scanned/tested images– Integrate images in the SDLC– Use small footprint images– Use a trusted source
● At runtime– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits
How can you trust the images you are downloading?
● Content should be audited/verified
● Image builds should be traceable
● Image should be rebuild-able
● Only pull signed images: Docker Content Trust
Reducing attack surface● At runtime
– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits
● During build– Use scanned/tested images– Integrate images in the SDLC– Use a trusted source
● Up to date/patched Operating System
● Up to date/patch container runtime
● Do not run application/containers as root
● Network isolation: iptables, NetworkPolicies,…
● Log all the things!
● Beef it up with container ‘IDS/ADS’ (eg. sysdig/falco)
Reducing attack surface● At runtime
– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits
● During build– Use scanned/tested images– Integrate images in the SDLC– Use a trusted source
● Disable intercontainers and host communication:
● dockerd --icc=false
● Re-map UID of root and other local users
● /etc/{subuid, subgid}
● Ensure docker API is not exposed and control docker
group membership
● Giving access to dockerd = giving root access
Reducing attack surface● At runtime
– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits
● During build– Use scanned/tested images– Integrate images in the SDLC– Use a trusted source
● Linux Capabilities = broken down root authorizations
● Docker by default disables a lot of capabilities
● Don’t add capabilities to your container
● Seccomp = syscall filter
● Firewall between user land and kernel
● Avoid at all costs --security-opt seccomp=unconfined
● 25 Kernel CVE prevented thanks to these layers
Source: https://docs.docker.com/engine/security/non-events/
Reducing attack surface● At runtime
– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits
● During build– Use scanned/tested images– Integrate images in the SDLC– Use a trusted source
Use kernel hardening modules
● SELinux
● AppArmor
● GRSec
Reducing attack surface● At runtime
– Apply General best practices– Docker best practices– Linux Capabilities– Seccomp– Kernel patchs– Use cgroup limits
● During build– Use scanned/tested images– Integrate images in the SDLC– Use a trusted source
Use cgroups limitations on your containers to:
● Avoid resources DoS on host
● Reduce impact by pausing the container
Take-aways (TLDL;)● Containers are:
– already in the vein of your infrastructure– Impacting software architectures and your development, build, integration and deployments
● Smooth ci/cd pipelines integrated with SSDLC– Security involved at build time: secure that deployment package– Security monitoring/detection at runtime
● Best practices also apply in container world:– System Container/Hardening– Vulnerability scanning– System and application logging– But you should/could go beyond this
Beyond docker...● Container orchestrators:
– Docker Swarm– Kubernetes
● Gvisor: Moving ring0 in ring3● Kata containers: s/docker/qemu● Cloud-Native?
Questions ?
Back-up slides
References● CVE-2019-5736:
https://blog.dragonsector.pl/2019/02/cve-2019-5736-escape-from-docker-and.html ● Dirtyc0w, container escape via vdso shared library:
https://github.com/scumjr/dirtycow-vdso ● Sysdig/Falco: https://github.com/falcosecurity/falco ● Sysdig/Images scanning tools:
https://sysdig.com/blog/container-security-docker-image-scanning/ ● Anchore: https://anchore.com● Clair:● Distroless: https://github.com/GoogleContainerTools/distroless
CVE-2019-5736
Source: https://www.twistlock.com/labs-blog/breaking-docker-via-runc-explaining-cve-2019-5736/