Date post: | 13-Apr-2017 |
Category: |
Technology |
Upload: | delve-labs |
View: | 234 times |
Download: | 0 times |
Docker Security In Production#DevOps #Infrastructure #Deployment #Security
➔ CI/CD chain security ( git / notary / registry )
◆ … export DOCKER_CONTENT_TRUST=1
➔ Microservices architecture
◆ … secret management (Vault & al.)
◆ … Orchestration & Deployment Strategies
➔ Keeping binaries & libs. up to date in production
➔ Monitoring / Alerting / Metric / SOC / SIEM / etc.
What this talk is NOT about
Infrastructure information leak
Denial of Service
Data corruption
Software & Crypto exploit
Container escape
Root / Kernel exploit
Hypervisor escape
Hardware Implant, etc.
Reconnaissance
Loss of Availability
Loss of Integrity
Loss of Confidentiality
Privilege Escalation to Host
Host Auditability compromised
Pivot to other Host
Tin foil hat & Cryptopocalypse !
Type of attack Threat “hierarchy”
⇦
⇦
⇦
⇦
⇦
⇦
⇦
⇦
Docker builds on Kernel & Host Security
➔ Grsecurity kernelRandomization++, Bound checking,
Fork delay, Hardened seccomp BPF
➔ SELinux / AppArmorComplex execution profiles, {White,Black}-listing
➔ Sysctl settingsfd limit, IP stack, sysrq, buffers, etc.
➔ Unattended-upgradesAnd all the typical hardening
& distro compile flags!
Docker Daemon
➔ Limit docker group : docker.sockAccess to socket = root
➔ Authorization plugin APIDocker 1.10+: --authorization-pluginshould help mitigate previous issue soon
➔ docker-machine & TLSUse --tls-verify (port 2376)
➔ SELinux / AppArmor Profileapparmor.d/docker + restrictions
limit path, resources, etc.
➔ Export logs outside of host--log-driver= (syslog, fluentd, ...)
cgroups hardware resource limits
➔ Mitigate potential DoS attacksLimit memory, disk, network I/O & CPU share
➔ cgroups only limit resources share, not accessNot blocking access to:
kcore, modprobe, sysrq, mknod, eth0, ...
➔ You can define your own initial cgroup--cgroup-parent to inherit a previous context
Limiting CPU usage
➔ Limit the total or relative amount of CPU time share--cpu-shares relative weight (== cpu_shares: 100)
--cpu-period CFS (QoS) period
--cpu-quota CFS (QoS) quota
➔ Limit which CPU or RAM node can be used--cpuset-cpus CPU affinity (== cpu_set: 0,1)
--cpuset-mems Memory NUMA node (ie: 0-3, 0,1)
Limiting memory usage
➔ Limit a container’s memory usageLimit: --memory=1g (== mem_limit:)Soft Limit: --memory-reservation
➔ Limit swap usageTotal Limit: --memory-swap (== memswap_limit:)Swapiness: --memory-swapiness
** GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1" **
➔ Limit container’s kernel memory usage--kernel-memory limit
➔ Verify the Out Of Memory kernel policy--oom-kill-disable & --oom-score-adj
Device I/O & Filesystems
➔ Put docker on its own partition/var/lib/docker as a ZFS/BTRFS volume (snapshots, quotas)
➔ Minimum rights“rwm” options, i.e: --device=/dev/zero:/dev/zero:r
➔ Mount root & volumes as read-onlyFor volumes: /path:roz (Zz = SELinux label)for root (/): read_only: trueUse with --shm-size & /dev/shm for pid files, scratch, tmp, etc.--tmpfs /run:rw,noexec,nodev,nosuid,size=8m
➔ Limit allocated I/O bandwidth--device-read-bps, --device-write-bps--device-read-iops, --device-write-iops--blkio-weight-device 10 -> 1000
➔ Create an internal N-Tier architecturenetworks: ( docker-compose 1.6+ & version: ‘2’ ) || --net=
➔ Think about inter-container communication--icc=false + --link= (but deprecated), --ip-forward=
➔ Disable userland-proxy--userland-proxy=false … saves memory & faster
➔ Use iptables and tcLimit access and use QoS if necessary.
Networking
➔ Set your typical soft & hard limitsDaemon: --default-ulimit nofile=50:100
Container: --ulimit nofile=50:100
compose 1.6+: ulimit: nofile: soft:50 hard:100
➔ Prevent fork bombs: threads / process limitscompose 1.6+: ulimits: nproc: soft:32 hard:64
Docker 1.11+& Kernel 4.3+: --pids-limit (cgroup support)
➔ Think about your restart policyrestart: always? no?
System resources & ulimits
Namespaces
➔ Currently namespaced resourcesAudit, cgroups, IPC, mount, NET, PID, Syslog, UID, UTS--userns-remap=default (new in 1.10+), *but*:
Per daemon, not per container (--userns=host not yet in compose)Volumes UID/GID also remapped...Incompatible with IPC/PID/NET NS sharing...
i.e. --net=container:app1, --readonly filesystem...
➔ NOT (yet) NamespacedThe Kernel, LSM, UID (by default), keyring,ring buffer (dmesg), /proc/{sys}, /sys, /dev/{shm} ...
➔ A lot of work & cleanup still required for namespacesMany holes over the years:CVE-2010-0006, CVE-2011-2189, CVE-2013-1858, CVE-2013-1956, CVE-2013-4205, CVE-2014-4014, CVE-2014-5206, CVE-2014-5207, CVE-2014-8989, CVE-2015-8709, (!)
Capabilities
Default Capabilities
cap_chowncap_dac_overridecap_fownercap_fsetidcap_killcap_setgidcap_setuidcap_setpcapcap_net_bind_servicecap_net_rawcap_sys_chrootcap_mknodcap_audit_writecap_setfcap
➔ Useful but incomplete security modelSome are very granular: MKNOD
Others give you root: SYS_ADMIN
➔ Use whitelisting: --cap-drop=allThen --cap-add=SETUID etc, until it runs
➔ RUN setcap cap_mknod /bin/mknodUse instead of suid binaries
➔ Default Capabilities are inadequateSETUID, SETGID, MKNOD, ...
Seccomp (Secure Computing)
➔ Extremely granular filterBPF filters of syscalls + arguments
Docker default blacklist (whitelist in the future)
➔ Use tools to create profilesdockersl.im, genSeccomp.sh, etc.strace -c -f -S name ls 2>&1 >/dev/null | tail -n +3 | head -n -2 | awk '{print $(NF)}'
➔ --seccomp:/path/profile.jsonDisable default Seccomp filtering --seccomp:unconfined
➔ Use security_opt: - no-new-privilegesKeeps UID, GID & LSM Labels + can’t gain Capabilities/SUID
➔ Swarm init / joinExpose master nodes carefully (hold cluster’s secrets)
Mutually auth. TLS, AES-GCM, 12 hours key rotation (Gossip / Raft)
➔ Use overlay network encryptiondocker network create -d overlay -o encrypted mynet
- Keys shared with tasks & services, but not «docker run»
➔ Mutually authenticate your microservices tooMicroservices should not rely on overlay encryption:
Authenticate & Encrypt [container ↔ container] communications
➔ «docker-compose bundle» - experimental statusLacks support for most useful runtime security options, maybe in 1.13+?
Swarm Networking [1.12+]
➔ Never use --privilegedUse granular solutions previously described
➔ Run process as a userDon’t run inside container as root: use nobody
Remove SUID, strip unused files, etc.
➔ Layer as many security featuresNot all of them will apply, work, be enabled, etc.
➔ Don’t forget to harden applications!NGINX configs, exposed services, databases, etc.
Containers Runtime Security
References:https://www.youtube.com/watch?v=UywECF0h3eg (new in 1.10)https://www.youtube.com/watch?v=7ouzigqFUWU (defcon docker)https://www.youtube.com/watch?v=iN6QbszB1R8 (defcon container)https://www.youtube.com/watch?v=_SwxuMGQI2o (microXchg)https://docs.docker.com/engine/security/security/https://blog.docker.com/2016/02/docker-engine-1-10-security/https://github.com/konstruktoid/Docker/blob/master/Security/CheatSheet.mdhttp://linux-audit.com/docker-security-best-practices-for-your-vessel-and-containers/https://gallery.mailchimp.com/979c70339150d05eec1531104/files/Docker_Security_Red_Hat.pdfhttps://www.sans.org/reading-room/whitepapers/linux/securing-linux-containers-36142https://www.alfresco.com/blogs/devops/2015/12/03/docker-security-tools-audit-and-vulnerability-assessment/http://doger.iohttp://www.slideshare.net/Docker/docker-security-workshop-slideshttps://www.infoq.com/news/2016/08/secure-docker-microservices (Grattafiori TL;DR for youtube)https://www.youtube.com/watch?v=346WmxQ5xtk (Grattafiori Docker & High Security)
Tools:https://github.com/docker/docker-bench-security (Good practices)http://dockersl.im (Seccomp, etc.)https://github.com/konstruktoid/Docker/blob/master/Scripts/genSeccomp.sh (Seccomp Profile Generator)https://github.com/jfrazelle/bane (AppArmor)
Alexandre GuédonLEAD INFRASTRUCTURE [email protected]@peerprod