Inspecting Security of Dockerformatted Container Images

To find Peace of Mind

Ján Lieskovský

jlieskov@redhat.comiankko on #openscap

Docker Concepts - Introductory Notes

Docker Formatted Container ImagesInteresting Application Platform

Docker Formatted Container ImagesInteresting Application Platform

For developers

● Focus on content (rather on build process)● Data aggregation via Docker Image Specification● Simplified release management● Easy customization

Docker Formatted Container ImagesInteresting Application Platform

For users

● Abundance of applications available in official hub● Simple application deployment● Continuous application lifecycle management● Easy customization

Basic Docker TermsDocker image

● Ordered collection of root filesystem changes● Coupled with corresponding execution parameters● Doesn’t have a state● Read-only (never changes)● Set of layers stacked on top of each other

Basic Docker TermsDocker image

● Each image is derived from base image● Transformed to final image through set of steps (instructions)

○ Run a command○ Add a file or directory○ Create an environment variable○ What process to run when launching a container from this image

Docker image vs Docker containerDocker container

● Any (running / stopped) instance of Docker image● Consists of:

○ Docker image○ Execution environment○ Standard set of instructions

● It’s possible to have many running containers of the same image

Docker architecture

Virtual Machines vs Containers

Container security. And why it matters

Container Security MattersBasic security bricks of Docker daemon / containers

● Kernel security (updates, support for namespaces, cgroups)

Container Security MattersBasic security bricks of Docker daemon / containers

● Kernel security (updates, support for namespaces, cgroups)● Security of Docker daemon

Container Security MattersBasic security bricks of Docker daemon / containers

● Kernel security (updates, support for namespaces, cgroups)● Security of Docker daemon● Security of specific Dockerfile

Container Security MattersKernel namespaces, cgroups

● Form of isolation● Own network stack per container● Resource mngmt via cgroups

Container Security MattersOther kernel features applied in Docker security

● Linux kernel capabilities● GRSEC, PAX● SELinux, AppArmor

Container Security MattersBasic security bricks of Docker daemon / containers

So far looks reasonable, right?

Container Security MattersBut, what if we overlooked something?

Container Security MattersBut, what if we overlooked something?

22 April 2014 Daniel J Walsh (Red Hat)Containers do not contain

Bottom line:● Running a container not every major kernel subsystem is namespaced

○ SELinux○ Cgroups○ File system under /sys○ /proc/sys, /proc/irq, /proc/bus○ Devices and kernel modules are not namespaced

Container Security MattersBut, what if we overlooked something?

Container Security MattersBut, what if we overlooked something? Meet Shocker!

Container Security MattersBut, what if we overlooked something? Meet Shocker!

Virtual Machines vs Containers

Container Security MattersWhat the wise men have got to say?

22 Jul 2014 Jérôme Petazzoni (Docker Inc.)Is it Safe to Run Applications in Linux Containers?

Bottom line:

● Don't run things as root ● Use seccomp-bpf

● Drop capabilities ● Get a GRSEC kernel

● Enable user namespaces ● Update kernels often

● Get rid of shady SUID binaries ● Mount everything read-only

● Enable SELinux (or AppArmor) ● Ultimately, fence things in VMs

Container Security MattersWhat the wise men have got to say?

03 Sep 2014 Daniel J Walsh (Red Hat)Bringing new security features to Docker

Bottom line:● Only run applications from a trusted source● Run applications on a enterprise quality host● Install updates regularly● Drop privileges as quickly as possible● Run as non-root whenever possible● Watch your logs● setenforce 1

Container Security MattersPlenty of applications from Official Repositories

Container Security Matters(Another) Plenty of applications from Custom Repositories

Container Security MattersWho built these images?

12 Aug 2015 Introduced in Docker v1.8 using The Update Framework

Docker Content Trust Workflow

● Image producer - pushing an image to remote repository, Docker engine signs the content using publisher’s private key

● Image consumer - when pulling an image, Docker engine verifies the content of the image using publisher’s public key. If image tampering is detected, pull fails

Container Security MattersDocker Image Signing and Verification

Two types of keys known by Docker Content Trust

○ Tagging Key■ One such key is created per each new repository the publisher owns■ Intended to be shared with any person / system requiring the ability to

sign content for this repository○ Offline key

■ Can be shared across repositories■ Required to create a new repository or to rotate existing tagging keys

Container Security MattersDocker Image Signing and Verification

Provides

○ Protection against image tampering○ Protection against image replay attacks○ Protection against tagging key compromise

Container Security MattersDocker Image Signing and Verification

Container Security MattersWho built these images?

Container Security MattersWe know the publisher. But how were all these images built?

Docker image (quick recap)

● Each image is derived from base image● Transformed to final image through set of steps (instructions)

Container Security MattersWe know the publisher. But how were all these images built?

Docker image (quick recap)

● Each image is derived from base image● Transformed to final image through set of steps (instructions)

Creating new images

● Update the container (running an image)

Commit the changes to image

● Build a new image from Dockerfile

Container Security MattersHow were all these images built?

Container Security MattersWho can build / update the containers?

Container Security MattersWho can build / update the containers?

Anyone!!!

Container Security MattersWhat we trust into when pulling images?

● The base image the pulled image is derived from was secure

Container Security MattersWhat we trust into when pulling images?

● The base image the pulled image is derived from was secure

● The newly introduced changes were performed in secure way

Container Security MattersWhat we trust into when pulling images?

● The base image the pulled image is derived from was secure● The newly introduced changes were performed in secure way

● When a security flaws was found in base image or the changes, the image available in repository has been

already updated

Container Security MattersWhat we trust into when pulling images?

Ultimate goal:

● Secure container infrastructure

Container Security MattersDocker daemon / container security - Lessons Learned

Ultimate goal:

● Secure container infrastructure

Trust the design:

● We can trust Docker design to be secure

Container Security MattersDocker daemon / container security - Lessons Learned

Ultimate goal:

● Secure container infrastructure

Trust the design:

● We can trust Docker design to be secure

But act responsibly:

● Verify that all of the host, daemon and containers truly are secure

Container Security MattersDocker daemon / container security - Lessons Learned

How to verify (inspect) security of containers / images?

Inspecting Security of Containers /Images

Two separate tasks:

● Inspect presence of security flaws (vulnerability assessment)

● Verify the configuration is secure (security compliance)

Vulnerability Assessment ofContainers / Images

Vulnerability Assessment ofContainers / Images

Vulnerability Assessment ofContainers / Images

● HTML advisories are easy consumable by humans● But not suitable for machine processing

Vulnerability Assessment ofContainers / Images

● We need a standard● Security errata information available in the form of

that standard● Scanner able to perform automated scan

Vulnerability Assessment ofContainers / Images

● We need a standard to○ represent configuration information of systems○ analyze the system for presence of specified

machine state (vulnerability, configuration, …)○ report the results of the assessment back

Vulnerability Assessment ofContainers / Images

Open Vulnerability and Assessment Language

Vulnerability Assessment ofContainers / Images

● We need a standard● Security errata information available in the form of

that standard● Scanner able to perform automated scan

Vulnerability Assessment ofContainers / Images

● We need a standard● Security errata information available in the form of

that standard○ Red Hat OVAL security data○ Ubuntu OVAL security data○ …

Vulnerability Assessment ofContainers / Images

● We need a standard● Security errata information available in the form of

that standard● Scanner able to perform automated scan

Vulnerability Assessment ofContainers / Images

Vulnerability Assessmentof Containers

# dnf -y install openscap-containers

# docker pull richxsl/rhel7

# wget http://www.redhat.com/security/data/oval/Red_Hat_Enterprise_Linux_7.xml

# docker run -i -t richxsl/rhel7 /bin/bash

Vulnerability Assessmentof Containers

# oscap-docker container-cve richxsl/rhel7Definition oval:com.redhat.rhsa:def:20160695: falseDefinition oval:com.redhat.rhsa:def:20160685: trueDefinition oval:com.redhat.rhsa:def:20160676: falseDefinition oval:com.redhat.rhsa:def:20160650: falseDefinition oval:com.redhat.rhsa:def:20160612: falseDefinition oval:com.redhat.rhsa:def:20160594: falseDefinition oval:com.redhat.rhsa:def:20160534: falseDefinition oval:com.redhat.rhsa:def:20160532: true

...

Vulnerability Assessmentof Containers

# oscap-docker container-cve richxsl/rhel7Definition oval:com.redhat.rhsa:def:20160695: falseDefinition oval:com.redhat.rhsa:def:20160685: trueDefinition oval:com.redhat.rhsa:def:20160676: falseDefinition oval:com.redhat.rhsa:def:20160650: falseDefinition oval:com.redhat.rhsa:def:20160612: falseDefinition oval:com.redhat.rhsa:def:20160594: falseDefinition oval:com.redhat.rhsa:def:20160534: falseDefinition oval:com.redhat.rhsa:def:20160532: true

...

Vulnerability Assessmentof Images

Running untrusted containers might be dangerous!

Vulnerability Assessmentof Images

Running untrusted containers might be dangerous!

Let’s scan the images instead !!!

Vulnerability Assessmentof Containers

# oscap-docker image-cve richxsl/rhel7Definition oval:com.redhat.rhsa:def:20160695: falseDefinition oval:com.redhat.rhsa:def:20160685: trueDefinition oval:com.redhat.rhsa:def:20160676: falseDefinition oval:com.redhat.rhsa:def:20160650: falseDefinition oval:com.redhat.rhsa:def:20160612: falseDefinition oval:com.redhat.rhsa:def:20160594: falseDefinition oval:com.redhat.rhsa:def:20160534: falseDefinition oval:com.redhat.rhsa:def:20160532: true

...

Vulnerability Assessmentof Containers

# oscap-docker image-cve richxsl/rhel7Definition oval:com.redhat.rhsa:def:20160695: falseDefinition oval:com.redhat.rhsa:def:20160685: trueDefinition oval:com.redhat.rhsa:def:20160676: falseDefinition oval:com.redhat.rhsa:def:20160650: falseDefinition oval:com.redhat.rhsa:def:20160612: falseDefinition oval:com.redhat.rhsa:def:20160594: falseDefinition oval:com.redhat.rhsa:def:20160534: falseDefinition oval:com.redhat.rhsa:def:20160532: true

...

Vulnerability Assessmentof Containers

# oscap-docker image-cve richxsl/rhel7Definition oval:com.redhat.rhsa:def:20160695: falseDefinition oval:com.redhat.rhsa:def:20160685: trueDefinition oval:com.redhat.rhsa:def:20160676: falseDefinition oval:com.redhat.rhsa:def:20160650: falseDefinition oval:com.redhat.rhsa:def:20160612: falseDefinition oval:com.redhat.rhsa:def:20160594: falseDefinition oval:com.redhat.rhsa:def:20160534: falseDefinition oval:com.redhat.rhsa:def:20160532: true

...

Inspecting Security of Containers /Images

Two separate tasks:

● Inspect presence of security flaws (vulnerability assessment)

● Verify the configuration is secure (security compliance)

Verification if Configuration ofContainers / Images is Secure?

Verification if Configuration ofContainers / Images is Secure?

# dnf -y install scap-security-guide

Verification if Configuration ofContainers / Images is Secure?

# oscap-docker image richxsl/rhel7 xccdf eval \--profile standard --report /tmp/report.html \

/usr/share/xml/scap/ssg/content/ssg-rhel7-xccdf.xml

Verification if Configuration ofContainers / Images is Secure?

# oscap-docker image richxsl/rhel7 xccdf eval \--profile standard --report /tmp/report.html \

/usr/share/xml/scap/ssg/content/ssg-rhel7-xccdf.xml

Verification if Configuration ofContainers / Images is Secure?

Questions?

http://www.open-scap.org/tools/https://github.com/OpenSCAP

(We are hiring)Ján Lieskovskýjlieskov@redhat.comiankko on #openscap

Slightly Advanced Topics

Customizing security policy for particular use case

Example use case:● Detect unauthorised SUID binaries present in the

container

Slightly Advanced Topics

Example use case:● Detect unauthorised SUID binaries present in the

container

Modify standard SCAP Security Guide profile to contain just:

"file_permissions_unauthorized_suid"

rule

Slightly Advanced Topics

● Modify standard SCAP Security Guide profile to contain just:

"file_permissions_unauthorized_suid"

rule

● Rescan the image