Realizing Linux Containers (LXC)Building Blocks, Underpinnings & Motivations

Boden Russell – IBM Global Technology Services(brussell@us.ibm.com)

04/07/2023 2

Definitions Linux Containers (LXC for LinuX Containers) are lightweight virtual machines (VMs)

which are realized using features provided by a modern Linux kernel – VMs without the hypervisor

Containerization of:– (Linux) Operating Systems– Single or multiple applications

LXC as a technology not to be confused with LXC (tools) which are a user space toolset for creating & managing Linux Containers

From wikipedia:LXC (LinuX Containers) is an operating system–level virtualization method for running multiple isolated Linux systems (containers) on a single control host… LXC provides operating system-level virtualization not via a virtual machine, but rather provides a virtual environment that has its own process and network space.

04/07/2023 3

Why LXC

Provision in seconds / milliseconds Near bare metal runtime performance VM-like agility – it’s still “virtualization” Flexibility

– Containerize a “system”– Containerize “application(s)”

Lightweight– Just enough Operating System (JeOS)– Minimal per container penalty

Open source – free – lower TCO Supported with OOTB modern Linux kernel Growing in popularity

“Linux Containers as poised as the next VM in our modern Cloud era…”

Manual VM LXC

Provision Time

Days

Minutes

Seconds / ms

linpack performance @ 45000

0

50

100

150

200

250

vcpus

GF

lop

s

Google trends - LXC Google trends - docker

04/07/2023 4

Hypervisors vs. Linux Containers

Hardware

Operating System

Hypervisor

Virtual Machine

Operating System

Bins / libs

App App

Virtual Machine

Operating System

Bins / libs

App App

Hardware

Hypervisor

Virtual Machine

Operating System

Bins / libs

App App

Virtual Machine

Operating System

Bins / libs

App App

Hardware

Operating System

Container

Bins / libs

App App

Container

Bins / libs

App App

Type 1 Hypervisor Type 2 Hypervisor Linux Containers

Containers share the OS kernel of the host and thus are lightweight.However, each container must have the same OS kernel.

Containers are isolated, but share OS and, where appropriate, libs / bins.

04/07/2023 5

LXC Technology Stack LXCs are built on modern kernel features

– cgroups; limits, prioritization, accounting & control– namespaces; process based resource isolation– chroot; apparent root FS directory– Linux Security Modules (LSM); Mandatory Access Control (MAC)

User space interfaces for kernel functions LXC tools

– Tools to isolate process(es) virtualizing kernel resources LXC commoditization

– Dead easy LXC– LXC virtualization

Orchestration & management– Scheduling across multiple hosts– Monitoring– Uptime

04/07/2023 6

Linux cgroups History

– Work started in 2006 by google engineers– Merged into upstream 2.6.24 kernel due to wider spread LXC usage– A number of features still a WIP

Functionality– Access; which devices can be used per cgroup– Resource limiting; memory, CPU, device accessibility, block I/O, etc.– Prioritization; who gets more of the CPU, memory, etc.– Accounting; resource usage per cgroup– Control; freezing & check pointing– Injection; packet tagging

Usage– cgroup functionality exposed as “resource controllers” (aka “subsystems”)– Subsystems mounted on FS– Top-level subsystem mount is the root cgroup; all procs on host– Directories under top-level mounts created per cgroup– Procs put in tasks file for group assignment– Interface via read / write pseudo files in group

04/07/2023 7

Linux cgroup Subsystems cgroups provided via kernel modules

– Not always loaded / provided by default– Locate and load with modprobe

Some features tied to kernel version See: https://www.kernel.org/doc/Documentation/cgroups/

Subsystem Tunable Parametersblkio - Weighted proportional block I/O access. Group wide or per device.

- Per device hard limits on block I/O read/write specified as bytes per second or IOPS per second.

cpu - Time period (microseconds per second) a group should have CPU access.- Group wide upper limit on CPU time per second.- Weighted proportional value of relative CPU time for a group.

cpuset - CPUs (cores) the group can access.- Memory nodes the group can access and migrate ability.- Memory hardwall, pressure, spread, etc.

devices - Define which devices and access type a group can use.

freezer - Suspend/resume group tasks.

memory - Max memory limits for the group (in bytes).- Memory swappiness, OOM control, hierarchy, etc..

hugetlb - Limit HugeTLB size usage.- Per cgroup HugeTLB metrics.

net_cls - Tag network packets with a class ID.- Use tc to prioritize tagged packets.

net_prio - Weighted proportional priority on egress traffic (per interface).

04/07/2023 8

Linux cgroups FS Layout

04/07/2023 9

Linux cgroups Pseudo FS Interface

/sys/fs/cgroup/my-lxc

|-- blkio| |-- blkio.io_merged| |-- blkio.io_queued| |-- blkio.io_service_bytes| |-- blkio.io_serviced| |-- blkio.io_service_time| |-- blkio.io_wait_time| |-- blkio.reset_stats| |-- blkio.sectors| |-- blkio.throttle.io_service_bytes| |-- blkio.throttle.io_serviced| |-- blkio.throttle.read_bps_device| |-- blkio.throttle.read_iops_device| |-- blkio.throttle.write_bps_device| |-- blkio.throttle.write_iops_device| |-- blkio.time| |-- blkio.weight| |-- blkio.weight_device| |-- cgroup.clone_children| |-- cgroup.event_control| |-- cgroup.procs| |-- notify_on_release| |-- release_agent| `-- tasks|-- cpu| |-- ...|-- ...`-- perf_event

echo "8:16 1048576“ > blkio.throttle.read_bps_

device

cat blkio.weight_devicedev weight8:1 2008:16 500

App

App

App

Linux pseudo FS is the interface to cgroups– Read / write to pseudo file(s) in your cgroup directory

Some libs exist to interface with pseudo FS programmatically

04/07/2023 10

Linux cgroups: CPU Usage Use CPU shares (and other controls) to prioritize jobs / containers Carry out complex scheduling schemes Segment host resources Adhere to SLAs

04/07/2023 11

Linux cgroups: CPU Pinning Pin containers / jobs to CPU cores Carry out complex scheduling schemes Reduce core switching costs Adhere to SLAs

04/07/2023 12

Linux cgroups: Device Access Limit device visibility; isolation Implement device access controls

– Secure sharing Segment device access Device whitelist / blacklist

04/07/2023 13

LXC Realization: Linux cgroups cgroup created per container (in each cgroup subsystem) Prioritization, access, limits per container a la cgroup controls Per container metrics (bean counters)

04/07/2023 14

Linux namespaces History

– Initial kernel patches in 2.4.19– Recent 3.8 patches for user namespace support – A number of features still a WIP

Functionality– Provide process level isolation of global resources

• MNT (mount points, file systems, etc.)• PID (process)• NET (NICs, routing, etc.)• IPC (System V IPC resources)• UTS (host & domain name)• USER (UID + GID)

– Process(es) in namespace have illusion they are the only processes on the system– Generally constructs exist to permit “connectivity” with parent namespace

Usage– Construct namespace(s) of desired type– Create process(es) in namespace (typically done when creating namespace)– If necessary, initialize “connectivity” to parent namespace– Process(es) in name space internally function as if they are only proc(s) on system

04/07/2023 15

Linux namespaces: Conceptual Overview

04/07/2023 16

Linux namespaces: MNT namespace Isolates the mount table – per namespace mounts mount / unmount operations isolated to namespace Mount propagation

– Shared; mount objects propagate events to one another– Slave; one mount propagates events to another, but not

vice versa– Private; no event propagation (default)

Unbindable mount forbids bind mounting itself Various tools / APIs support the mount namespace such

as the mount command– Options to make shared, private, slave, etc.– Mount with namespace support

Typically used with chroot or pivot_root for effective root FS isolation

“global” (i.e. root) namespace

“green” namespace

“red” namespace

MNT NS//proc/mnt/fsrd/mnt/fsrw/mnt/cdrom/run2

MNT NS//proc/mnt/greenfs/mnt/fsrw/mnt/cdrom

MNT NS//proc/mnt/cdrom/redns

04/07/2023 17

Linux namespaces: UTS namespace Per namespace

– Hostname– NIS domain name

Reported by commands such as hostname Processes in namespace can change UTS values – only

reflected in the child namespace Allows containers to have their own FQDN

“global” (i.e. root) namespace

“green” namespace

“red” namespace

UTS NS

globalhostrootns.com

UTS NS

greenhostgreenns.org

UTS NS

redhostredns.com

04/07/2023 18

Linux namespaces: PID namespace Per namespace PID mapping

– PID 1 in namespace not the same as PID 1 in parent namespace– No PID conflicts between namespaces– Effectively 2 PIDs; the PID in the namespace and the PID outside

the namespace Permits migrating namespace processes between hosts

while keeping same PID Only processes in the namespace are visible within the

namespace (visibility limited)

“global” (i.e. root) namespace

“green” namespace

“red” namespace

PID NS

PID COMMAND1 /sbin/init2 [kthreadd]3 [ksoftirqd]4 [cpuset]5 /sbin/udevd

PID NS

PID COMMAND1 /bin/bash2 /bin/vim

PID NS

PID COMMAND1 /bin/bash2 python3 node

04/07/2023 19

Linux namespaces: IPC namespace System V IPC object & POSIX message queue isolation

between namespaces– Semaphores– Shared memory– Message queues

Parent namespace connectivity– Signals– Memory polling– Sockets (if no NET namespace)– Files / file descriptors (if no mount namespace)– Events over pipe pair

“global” (i.e. root) namespace

“green” namespace

“red” namespace

IPC NSSHMID OWNER32452 root43321 boden

SEMID OWNER0 root1 boden

IPC NS

SHMID OWNER

SEMID OWNER0 root

IPC NS

SHMID OWNER

SEMID OWNER

MSQID OWNER

04/07/2023 20

Linux namespaces: NET namespace Per namespace network objects

– Network devices (eths)– Bridges– Routing tables– IP address(es)– ports– Etc

Various commands support network namespace such as ip Connectivity to other namespaces

– veths – create veth pair, move one inside the namespace and configure

– Acts as a pipe between the 2 namespaces LXCs can have their own IPs, routes, bridges, etc.

“global” (i.e. root) namespace

“green” namespace

“red” namespace

NET NSlo: UNKNOWN…eth0: UP…eth1: UP…br0: UP…

app1 IP:5000app2 IP:6000app3 IP:7000

NET NS

lo: UNKNOWN…eth0: UP…

app1 IP:1000app2 IP:7000

NET NS

lo: UNKNOWN…eth0: DOWN…eth1: UP

app1 IP:7000app2 IP:9000

04/07/2023 21

Linux namespaces: USER namespace A long work in progress – still development for XFS and other

FS support– Significant security impacts– A handful of security holes already found + fixed

Two major features provided:– Map UID / GID from outside the container to UID / GID inside the

container– Permit non-root users to launch LXCs– Distro’s rolling out phased support, with UID / GID mapping

typically 1st

First process in USER namespace has full CAPs; perform initializations before other processes are created– No CAPs in parent namespace

UID / GID map can be pre-configured via FS Eventually USER namespace will mitigate many perceived LXC

security concerns

“global” (i.e. root) namespace

“green” namespace

“red” namespace

USER NS

root 0:0ntp 104:109Mysql 105:110boden 106:111

USER NS

root 0:0app 106:111

USER NS

root 0:0app 104:109

04/07/2023 22

LXC Realization: Linux namespaces A set of namespaces created for the container Container process(es) “executed” in the namespace set Process(es) in the container have isolated view of resources Connectivity to parent where needed (via lxc tooling)

04/07/2023 23

Linux namespaces & cgroups: Availability

Note: user namespace support in upstream kernel 3.8+, but distributions rolling out phased support:- Map LXC UID/GID between

container and host- Non-root LXC creation

04/07/2023 24

Linux chroots Changes apparent root directory for process and children

– Search paths– Relative directories– Etc

Using chroot can be escaped given proper capabilities, thus pivot_root is often used instead– chroot; points the processes file system root to new directory – pivot_root; detaches the new root and attaches it to process root directory

Often used when building system images– Chroot to temp directory– Download and install packages in chroot– Compress chroot as a system root FS

LXC realization– Bind mount container root FS (image)– Launch (unshare or clone) LXC init process in a new MNT namespace– pivot_root to the bind mount (root FS)

04/07/2023 25

Linux chroot vs pivot_root Using pivot_root with MNT namespace addresses escaping chroot concerns The pivot_root target directory becomes the “new root FS”

04/07/2023 26

LXC Realization: ImagesLXC images provide a flexible means to deliver only what you need – lightweight and minimal footprint

Basic constraints– Same architecture– Same endian– Linux’ish Operating System; you can run different Linux distros on same host

Image types– System; images intended to virtualize Operating System(s) – standard distro root FS less the kernel– Application; images intended to virtualize application(s) – only package apps + dependencies (aka

JeOS – Just enough Operating System) Bind mount host libs / bins into LXC to share host resources Container image init process

– Container init command provided on invocation – can be an application or a full fledged init process

– Init script customized for image – skinny SysVinit, upstart, etc.– Reduces overhead of lxc start-up and runtime foot print

Various tools to build images– SuSE Kiwi– Debootstrap– Etc.

LXC tooling options often include numerous image templates

04/07/2023 27

Linux Security Modules & MAC Linux Security Modules (LSM) – kernel modules which provide a framework for

Mandatory Access Control (MAC) security implementations MAC vs DAC

– In MAC, admin (user or process) assigns access controls to subject / initiator• Most MAC implementations provide the notion of profiles• Profiles define access restrictions and are said to “confine” a subject

– In DAC, resource owner (user) assigns access controls to individual resources Existing LSM implementations include: AppArmor, SELinux, GRSEC, etc.

04/07/2023 28

Linux Capabilities & Other Security Measures Linux capabilities

– Per process privileges which define operational (sys call) access– Typically checked based on process EUID and EGID– Root processes (i.e. EUID = GUID = 0) bypass capability checks

Capabilities can be assigned to LXC processes to restrict Other LXC security mitigations

– Reduce shared FS access using RO bind mounts– Keep Linux kernel up to date– User namespaces in 3.8+ kernel

• Allow to launch containers as non-root user• Map UID / GID inside / outside of container

04/07/2023 29

LXC Realization

04/07/2023 30

LXC Tooling LXC is not a kernel feature – it’s a technology enabled via kernel features

– User space tooling required to manage LXCs effectively Numerous toolsets exist

– Then: add-on patches to upstream kernel due to slow kernel acceptance– Now: upstream LXC feature support is growing – less need for patches

More popular GNU Linux toolsets include libvirt-lxc and lxc (tools)– OpenVZ is likely the most mature toolset, but it requires kernel patches– Note: I would consider docker a commoditization of LXC

Non-GNU Linux based LXC– Solaris zones– BSD jails– Illumos / SmartOS (solaris derivatives)– Etc.

04/07/2023 31

LXC Industry Tooling

04/07/2023 32

Libvirt-lxc Perhaps the simplest to learn through a familiar virsh interface Libvirt provides LXC support by connecting to lxc:/// Many virsh commands work

• virsh -c lxc:/// define sample.xml • virsh –c lxc:/// start sample• virsh –c lxc:/// console sample• virsh –c lxc:/// shutdown sample• virsh –c lxc:/// undefine sample

No snapshotting, templates… OpenStack support since Grizzly

No VNC No Cinder support in Grizzly Config drive not supported

Alternative means of accessing metadata Attached disk rather than http calls

<domain type='lxc'> <name>sample</name> <memory>32768</memory> <os> <type>exe</type> <init>/init</init> </os> <vcpu>1</vcpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/libvirt_lxc</emulator> <filesystem type='mount'> <source dir='/opt/vm-1-root'/> <target dir='/'/> </filesystem> <interface type='network'> <source network='default'/> </interface> <console type='pty' /> </devices> </domain>

04/07/2023 33

LXC (tools) A little more functionality Supported by the major distributions LXC 1.0 recently released

– Cloning supported: lxc-clone– Templates… btrfs– lxc-create -t ubuntu -n CN creates a new ubuntu container

• “template” is downloaded from Ubuntu• Some support for Fedora <= 14• Debian is supported

– lxc-start -d -n CN starts the container– lxc-destroy -n CN destroys the container– /etc/lxc/lxc.conf has default settings– /var/lib/lxc/CN is the default place for each container

04/07/2023 34

LXC Commoditization: docker Young project with great vibrancy in the industry Currently based on unmodified LXC – but the goal is to make it dirt easy As of March 10th, 2014 at v0.9. Monthly releases, 1.0 should be ready for production use What docker adds to LXC

– Portable deployment across machines• In Cloud terms, think of LXC as the hypervisor and docker as the Open Virtualization Appliance (OVA) and the provision engine• Docker images can run unchanged on any platform supporting docker

– Application-centric• User facing function geared towards application deployment, not VM analogs [!]

– Automatic build• Create containers from build files• Builders can use chef, maven, puppet, etc.

– Versioning support• Think of git for docker containers• Only delta of base container is tracked

– Component re-use• Any container can be used as a base, specialized and saved

– Sharing• Support for public/private repositories of containers

– Tools• CLI / REST API for interacting with docker• Vendors adding tools daily

Docker containers are self contained – no more “dependency hell”

04/07/2023 35

Docker vs. LXC vs. Hypervisor

04/07/2023 36

Docker: LXC Virtualization? Docker decouples the LXC provider from the operations

– LXC provider agnostic Docker “images” run anywhere docker is supported

– Portability

04/07/2023 37

LXC Orchestration & Management Docker & libvirt-lxc in OpenStack

– Manage containers heterogeneously with traditional VMs… but not w/the level of support & features we might like

CoreOS– Zero-touch admin Linux distro with docker images as the unit of operation– Centralized key/value store to coordinate distributed environment

Various other 3rd party apps– Maestro for docker– Shipyard for docker– Fleet for CoreOS– Etc.

LXC migration– Container migration via criu

But…– Still no great way to tie all virtual resources together with LXC – e.g. storage + networking

• IMO; an area which needs focus for LXC to become more generally applicable

04/07/2023 38

Docker in OpenStack Introduced in Havana

– A nova driver to integrate with docker REST API– A Glance translator to integrate containers with Glance

• A docker container which implements a docker registry API

The claim is that docker will become a “group A” hypervisor– In it’s current form it’s effectively a “tech preview”

04/07/2023 39

LXC Evaluation Goal: validate the promise with an eye towards practical applicability Dimensions evaluated:

– Runtime performance benefits– Density / footprint– Workload isolation– Ease of use and tooling– Cloud Integration– Security– Ease of use / feature set

NOTE: tests performed in a passive manner – deeper analysis warrented.

04/07/2023 40

Runtime Performance Benefits - CPU Tested using libvirt lxc on Ubuntu 13.10 using linpack 11.1 Cpuset was used to limit the number of CPUs that the containers could use The performance overhead falls within the error of measurement of this test Actual bare metal performance is actually lower than some container results

linpack performance @ 45000

0

50

100

150

200

250

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 BM

vcpus

GF

lop

s

220.77Bare metal220.5

@32 vcpu

220.9@ 31 vcpu

04/07/2023 41

Runtime Performance Benefits – I/O I/O Tests using libvirt lxc show a < 1 % degradation Tested with a pass-through mount

Sync read I/O test

Rw=WriteSize=1024mBs=128mbdirect=1sync=1

Sync write I/O test

Rw=WriteSize=1024mBs=128mbdirect=1sync=1

I/O throughput

1711.2 1724.9 1626.4 1633.4

0

500

1000

1500

2000

lxc write bare metalwrite

lxc read bare metalread

test

MB

/s

Series1

04/07/2023 42

Runtime Performance Benefits – Block I/O Tested with [standard] AUFS

04/07/2023 43

Density & Footprint – libvirt-lxc

Starting 500 containersMon Nov 11 13:38:49 CST 2013 ... all threads done in 157(sequential I/O bound)

Stopping 500 containersMon Nov 11 13:42:20 CST 2013 ... all threads done in 162Active memory delta: 417.2 KB

Starting 1000 containersMon Nov 11 13:59:19 CST 2013 ... all threads done in 335

Stopping 1000 containersMon Nov 11 14:14:26 CST 2013 ... all threads done in 339Active memory delta: 838.4KB

Using libvirt lxc on RHEL 6.4, we found that empty container overhead was just 840 bytes. A container could be started in about 330ms, which was an I/O bound processThis represents the lower limit of lxc footprintContainers ran /bin/sh

04/07/2023 44

Density & Footprint – Docker In this test, we created 150 Docker containers with CentOS, started

apache & then removed them Average footprint was ~10MB per container Average start time was 240ms

Serially booting 150 containers which run apache – Takes on average 36 seconds– Consumes about 2 % of the CPU– Negligible HDD space– Spawns around 225 processes for create– Around 1.5 GB of memory ~ 10 MB per container– Expect faster results once docker addresses performance topics in the

next few months

Serially destroying 150 containers running apache– On average takes 9 seconds– We would expect destroy to be faster – likely a docker bug and will triage

with the docker community

Container Creation

Container Deletion

I/O profile

CPU profile

04/07/2023 45

Workload Isolation: Examples Using the blkio cgroup (lxc.cgroup.blkio.throttle.read_bps_device) to cap the I/O of a container Both the total bps and iops_device on read / write could be capped Better async BIO support in kernel 3.10+

We used fio with oflag=sync, direct to test the ability to cap the reads:

– With limit set to 6 MB / secondREAD: io=131072KB, aggrb=6147KB/s, minb=6295KB/s, maxb=6295KB/s,

mint=21320msec, maxt=21320msec

– With limit set to 60 MB / secondREAD: io=131072KB, aggrb=61134KB/s, minb=62601KB/s, maxb=62601KB/s,

mint=2144msec, maxt=2144msec

– No read limitREAD: io=131072KB, aggrb=84726KB/s, minb=86760KB/s, maxb=86760KB/s,

mint=1547msec, maxt=1547msec

04/07/2023 46

OpenStack VM OperationsNOTE: orchestration / management overheads cap LXC performance

04/07/2023 47

Who’s Using LXC Google app engine & infra is said to be using some form of LXC RedHat OpenShift dotCloud (now docker inc) CloudFoundry (early versions) Rackspace Cloud Databases

– Outperforms AWS (Xen) according to perf results

Parallels Virtuozzo (commercial product) Etc..

04/07/2023 48

LXC Gaps

There are gaps…

Lack of industry tooling / support Live migration still a WIP Full orchestration across resources (compute / storage / networking) Fears of security Not a well known technology… yet Integration with existing virtualization and Cloud tooling Not much / any industry standards Missing skillset Slower upstream support due to kernel dev process Etc.

04/07/2023 49

LXC: Use Cases For Traditional VMs

There are still use cases where traditional VMs are warranted.

Virtualization of non Linux based OSs– Windows– AIX– Etc.

LXC not supported on host VM requires unique kernel setup which is not applicable to other VMs on the host

(i.e. per VM kernel config) Etc.

04/07/2023 50

LXC Recommendations Public & private Clouds

– Increase VM density 2-3x– Accommodate Big Data & HPC type applications– Move the support of Linux distros to containers

PaaS & managed services– Realize “as a Service” and managed services using LXC

Operations management– Ease management + increase agility of bare metal components

DevOps Development & test

– Sandboxes– Dev / test envs– Etc.

If you are just starting with LXC and don’t have in-depth skillset– Start with LXC for private solutions (trusted code)

04/07/2023 51

LXC Resources https://www.kernel.org/doc/Documentation/cgroups/ http://www.blaess.fr/christophe/2012/01/07/linux-3-2-cfs-cpu-bandwidth-english-version/ http://atmail.com/kb/2009/throttling-bandwidth/ https://

access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch-Subsystems_and_Tunable_Parameters.html

http://www.janoszen.com/2013/02/06/limiting-linux-processes-cgroups-explained/ http://www.mattfischer.com/blog/?p=399 http://oakbytes.wordpress.com/2012/09/02/cgroup-cpu-allocation-cpu-shares-examples/ http://fritshoogland.wordpress.com/2012/12/15/throttling-io-with-linux/ https://lwn.net/Articles/531114/ https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt http://www.ibm.com/developerworks/library/l-mount-namespaces/ http://blog.endpoint.com/2012/01/linux-unshare-m-for-per-process-private.html http://timothysc.github.io/blog/2013/02/22/perprocess/ http://www.evolware.org/?p=293 http://s3hh.wordpress.com/2012/05/10/user-namespaces-available-to-play/ http://libvirt.org/drvlxc.html https://help.ubuntu.com/lts/serverguide/lxc.html https://linuxcontainers.org/ https://wiki.ubuntu.com/AppArmor http://linux.die.net/man/7/capabilities http://docs.openstack.org/trunk/config-reference/content/lxc.html https://wiki.openstack.org/wiki/Docker https://www.docker.io/ http://marceloneves.org/papers/pdp2013-containers.pdf http://openvz.org/Main_Page http://criu.org/Main_Page