Containers with systemd-nspawn

Gábor NyersConsultant & Trainer @Trebutgnyers@trebut.com@gabornyers

Agenda

● An example systemd-nspawn container● What is systemd-nspawn and systemd● Related Concept: Kernel CGroups● Bootable containters● Containers as Service● Advanced topic: Socket Activation

3

An example systemd-nspawn

container

4

A Simple Application Container

• Start up container

• List of processes

• Try to install package‣ Limited footprint and exposure!

• On host OS: list kernel control groups:

# systemd-nspawn -jD /srv/containers/opensuse132/ \ -M opensuse132c0 \ /bin/bash

# opensuse132c0:~ # ps -efUID PID PPID C STIME TTY TIME CMDroot 1 0 0 04:16 ? 00:00:00 -bashroot 43 1 0 04:18 ? 00:00:00 ps -ef#

opensuse132c0:~ # zypper install wget -bash: zypper: command not found#

# machinectl MACHINE CONTAINER SERVICE opensuse132c0 container nspawn

1 machines listed.physnode1:~ #

# systemd-cgls ├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21├─machine.slice│ └─machine-opensuse132c0.scope│ └─18329 -bash[…]

# ps -ef -o pid,ppid,machine,cmd

5

Create application container

• Bootstrap directory

• Install a few packages

# zypper --root /srv/containers/opensuse132/ addrepo \ http://download.opensuse.org/distribution/13.2/repo/oss/ \ repo-oss

# zypper --root /srv/containers/opensuse132/ addrepo \ http://download.opensuse.org/distribution/13.2/repo/non-oss/ \ repo-non-oss

# zypper --root /srv/containers/opensuse132/ install \ openSUSE-release-13.2 install bash procps coreutils vim

6

systemd-nspawn • What is systemd?• What is systemd-nspawn?• Adoption

7

What is systemd? 1/3

• a system- and session manager for Linux,

• provides aggressive parallelization capabilities,(no shell during boot!)

• uses socket and D-Bus activation for starting services,

• offers on-demand starting of services,

• keeps track of processes using Linux cgroups,

8

What is systemd? 2/3

• supports restoring the system's state to a predefined state,

• maintains mount and auto-mount points,

• provides dependency based service control logic,

• provides replacement for a nr. of well-known tools, e.g.: udev, automount, inetd, consolekit and syslog,

• a drop-in replacement for sysvinit

9

What is systemd? 3/3

There is a lot of criticism and opinions as well...

• “It's not the UNIX way”referring to the “do one thing and do it well” maxim

• “It's monolithic”• “It introduces too many dependencies”• (and worse)

... but we won't be addressing these today :-)

10

An aside: People and Innovation...

“If I had asked people

what they wanted, they

would have said faster

horses”

Henry Ford

11

What is systemd-nspawn?

• “chroot on steroids...”

• Invented for debug and test of systemd development

• Turns out to be a great container manager

• systemd-nspawn vs. docker‣ Management container vs. container+images‣ Inherited networking vs. Need to set up networking

12

systemd adoption

Distribution Added to repositories Enabled by default? Released as default

SUSE Linux Enterprise

v12 Yes Yes

openSUSE v11.4 Yes v12.2 (2012)

Fedora v15 (2011) Yes v15 (2011)

Red Hat Linux Enterprise

v7 (2014) Yes v7 (2014)

Debian in 2012 Yes v8 (2015)

Arch Linux in 2012 Yes 2012

Ubuntu v13.04 (2013) Yes v15.04 (2015)

see also: http://en.wikipedia.org/wiki/Systemd#Adoption_and_reception

13

Related Concept • Kernel cgroups (independent of systemd)

14

Kernel Cgroups (Control Groups)

• Linux Kernel facility allowing the grouping of processes (and their “children”) into a tree-structure hierarchy

• Each group can be assigned a quota for these system resources:‣ CPU‣ RAM‣ Disk I/O‣ Network I/O

Control groups hierarchy created by systemd

├─machine.slice│ └─machine-qemu\x2dsles1201.scope │ └─20958 /usr/bin/qemu-system-x86_64 -m...├─user.slice│ ├─user-0.slice│ │ └─user@0.service│ │ ├─4322 /usr/lib/systemd/systemd --us...│ │ └─4323 (sd-pam)│ ├─user-1000.slice│ │ ├─session-560.scope│ │ │ ├─ 2810 /usr/bin/claws-mail│ │ │ ├─ 3035 /usr/lib64/firefox/firefox│ │ │ ├─ 3086 /usr/lib/mozilla/kmozillahel...│ │ │ ├─ 5459 /bin/bash│ │ │ ├─ 7854 /usr/bin/kwalletmanager --kw...│ │ ├─session-1.scope│ │ │ ├─4179 /bin/bash ./bridge start│ │ │ └─4182 dnsmasq --conf-file=mydnsmasq...│ │ └─user@1000.service│ │ ├─1891 /usr/lib/systemd/systemd --us...│ │ └─1892 (sd-pam)│ └─user-489.slice│ └─user@489.service│ ├─1703 /usr/lib/systemd/systemd --us...│ └─1704 (sd-pam)└─system.slice ├─libvirtd.service │ └─4008 /usr/sbin/libvirtd --listen ├─rsyslog.service │ └─985 /usr/sbin/rsyslogd -n ├─apache2.service │ ├─1254 /usr/sbin/httpd2-prefork -f /et... │ └─1840 /usr/sbin/httpd2-prefork -f /et...

15

Bootable containers

16

Bootable OS container [1/4]

Bootstrap

• Host properties

• Install YUM

• Bootstrap RPM DB

• Install CentOS 7 release package

• Install a few package and their dependencies

# hostnamectl Static hostname: physnode1.trebut.com Icon name: computer-laptop Chassis: laptop Machine ID: b4ea4eb15ab7c29b6cc20a47544e5eb7 Boot ID: 3c4e7b5067d247939b89d7e7b57c1132 Operating System: openSUSE 13.2 (Harlequin) (x86_64) CPE OS Name: cpe:/o:opensuse:opensuse:13.2 Kernel: Linux 3.16.7-7-desktop Architecture: x86-64

# zypper install yum

# rpm --root /srv/containers/centos/ --initdb

# rpm --root /srv/containers/centos/ \-ihv http://mirror.centos.org/centos/7.1.1503/os/x86_64/Packages/centos-release-7-1.1503.el7.centos.2.8.x86_64.rpm

# yum -y --nogpg --releasever=7 \ --installroot=/srv/containers/centos/ \ install systemd passwd yum vim-minimal

17

Bootable OS container [2/4]

Boot container

• Boot container‣ systemd-nspawn \

-bD /srv/containers/centos/

# systemd-nspawn -bD /srv/containers/centos/systemd 208 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ)Detected virtualization 'systemd-nspawn'.

Welcome to CentOS Linux 7 (Core)!

Set hostname to <centos7c0>.[ OK ] Reached target Remote File Systems.[ OK ] Created slice Root Slice.[ OK ] Created slice User and Session Slice.[ OK ] Created slice System Slice.[ OK ] Created slice system-getty.slice.[ OK ] Reached target Slices.[ OK ] Listening on Delayed Shutdown Socket.[ OK ] Listening on /dev/initctl Compatibility Named Pipe.[ OK ] Listening on Journal Socket. Starting Journal Service...[ OK ] Started Journal Service.[ OK ] Reached target Paths. Mounting Debug File System... Mounting FUSE Control File System... Starting Create static device nodes in /dev... Mounting POSIX Message Queue File System...[...][ OK ] Started Login Service.[ OK ] Started Permit User Sessions. Starting Console Getty...[ OK ] Started Console Getty.[ OK ] Reached target Login Prompts.[ OK ] Reached target Multi-User System.

CentOS Linux 7 (Core)Kernel 3.16.7-7-desktop on an x86_64

centos7c0 login:

18

Bootable OS container [3/4]

Instance propertiesOS Properties from inside the containerCentOS Linux 7 (Core)Kernel 3.16.7-7-desktop on an x86_64

centos7c0 login: rootPassword: Last login: Sat Apr 11 23:22:04 on console-bash-4.2# -bash-4.2# hostnamectl Static hostname: centos7c0 Icon name: computer-container Chassis: container Machine ID: afb4a0719ad842c99dd7cc704919a2fe Boot ID: 7c03b147c9114632b96bbeb2a462cf5a Virtualization: systemd-nspawn Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 3.16.7-7-desktop Architecture: x86_64-bash-4.2#

Container properties

# machinectl MACHINE CONTAINER SERVICE centos container nspawn

1 machines listed.physnode1:~

# systemd-cgls ├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21├─machine.slice│ └─machine-centos.scope│ ├─10159 /usr/lib/systemd/systemd│ └─system.slice│ ├─dbus.service│ │ └─10184 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation│ ├─systemd-journald.service│ │ └─10167 /usr/lib/systemd/systemd-journald│ ├─systemd-logind.service│ │ └─10183 /usr/lib/systemd/systemd-logind│ └─console-getty.service│ └─10189 /sbin/agetty --noclear --keep-baud console 115200 38400 9600├─system.slice

19

Bootable OS container [4/4]

Shutdown container

• Shutdown container from the inside:‣ Type: `init 0` or `poweroff`

Note: will require running init in container

‣ Type: ^]^]^] ( 3x CTRL+[ )

• Shutdown container from the host‣ machinectl terminate $CONT

-bash-4.2# init 0 [ OK ] Removed slice user-0.slice.[ OK ] Removed slice system-getty.slice. Stopping Hostname Service...[ OK ] Stopped target Graphical Interface.[ OK ] Stopped target Multi-User System.[ OK ] Stopped target Login Prompts. Stopping Console Getty... Stopping Login Service... Stopping D-Bus System Message Bus...[ OK ] Stopped Login Service.[ OK ] Stopped D-Bus System Message Bus.[ OK ] Stopped Console Getty. Stopping Permit User Sessions...[ OK ] Stopped Permit User Sessions.[ OK ] Stopped target Remote File Systems.[ OK ] Stopped Hostname Service.[ OK ] Stopped target Basic System.[ OK ] Stopped target Slices.[ OK ] Removed slice User and Session Slice.[ OK ] Stopped target Paths.[ OK ] Stopped target Timers.[ OK ] Stopped target Sockets.[ OK ] Closed D-Bus System Message Bus Socket.[ OK ] Stopped target System Initialization.[ OK ] Stopped target Encrypted Volumes. Stopping Load/Save Random Seed... Stopping Update UTMP about System Reboot/Shutdown...[ OK ] Stopped target Swap.[ OK ] Stopped Update UTMP about System Reboot/Shutdown.[ OK ] Stopped Load/Save Random Seed. Stopping Create Volatile Files and Directories...[ OK ] Stopped Create Volatile Files and Directories.[ OK ] Reached target Shutdown.

physnode1:/srv/containers #

20

Networking and systemd-nspawn containers

Networking in container-bash-4.2# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever2: wlp12s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff inet 10.1.2.27/21 brd 10.1.7.255 scope global dynamic wlp12s0 valid_lft 14611sec preferred_lft 14611sec inet6 fe80::224:d6ff:fe89:521e/64 scope link valid_lft forever preferred_lft forever3: enp0s25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000 link/ether 00:aa:bb:cc:dd:ee brd ff:ff:ff:ff:ff:ff

-bash-4.2# md5sum /etc/resolv.conf a92a6e440cd677ad17748aa29c5a7333 /etc/resolv.conf

‣ By default the nspawn container will inherit the network settings‣ /etc/resolv.conf will be copied into container

Networking at Host OSphysnode1:~ # ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever2: wlp12s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff inet 10.1.2.27/21 brd 10.1.7.255 scope global dynamic wlp12s0 valid_lft 14433sec preferred_lft 14433sec inet6 fe80::224:d6ff:fe89:521e/64 scope link valid_lft forever preferred_lft forever3: enp0s25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 link/ether 00:aa:bb:cc:dd:ee brd ff:ff:ff:ff:ff:ff

physnode1:~ # md5sum /etc/resolv.confa92a6e440cd677ad17748aa29c5a7333 /etc/resolv.conf

21

More advanced networking

‣ Create a virtual ethernet device, with name “vb-$machinename”‣ Connect veth device to bridge “virbr0”

systemd-nspawn -bD /srv/containers/opensuse132/ \

--network-bridge=virbr0 --network-veth

virbr0

veth(host0)

veth(vb-opensuse132c0)

opensuse132

physnode1

opensuse132c0:~ # ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group [...]2: host0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 36:e3:35:8d:8e:95 brd ff:ff:ff:ff:ff:ffopensuse132c0:~ #

physnode1:~ # ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group [...]29: vb-opensuse132c0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000 link/ether 0a:62:90:a4:b5:72 brd ff:ff:ff:ff:ff:ffphysnode1:~ #

22

journald and systemd-nspawn containers

• Integrating the journal of the host and the container

# systemd-nspawn \ -bD /srv/containers/centos \ --link-journal=host

23

Containers as Service

24

Container as service

• Install Apache and a few other packages

• Create a machine-id for the container

• Create systemd unit file

#install Apache zypper --root /srv/containers/opensuse132/ install \ openSUSE-release-13.2 apache2 timezone iproute2 rsyslog

# set up machine-id systemd-nspawn -D /srv/containers/opensuse132/ \ systemd-machine-id-setup

# unit file: cat <<EOF > /etc/systemd/system/opensuse132c0.service [Unit] Description=Start an openSUSE 13.2 container Wants=network.target nss-lookup.target After=network.target nss-lookup.target

[Service] Type=notify PrivateTmp=true ExecStart=/usr/bin/systemd-nspawn -M opensuse132c0 \ -jD /srv/containers/opensuse132/ ExecStop=/usr/bin/machinectl terminate opensuse132c0

[Install] WantedBy=machines.target EOF

25

Managing containers

nsenter

• nsenter - run program with namespaces of other processes

# machinectl MACHINE CONTAINER SERVICE opensuse132c0 container nspawn

1 machines listed.

# machinectl status opensuse132c0opensuse132c0 Since: Sun 2015-04-12 03:54:18 CEST; 37s ago Leader: 17717 (systemd) Service: nspawn; class container Root: /srv/containers/opensuse132 Unit: machine-opensuse132c0.scope ├─17717 /usr/lib/systemd/systemd └─system.slice ├─dbus.service […]

# nsenter --target 17717 --mount --uts --ipc --net –pidopensuse132c0:/ #

opensuse132c0:/ # systemctl disable rsyslog rm '/etc/systemd/system/multi-user.target.wants/rsyslog.service'rm '/etc/systemd/system/syslog.service'opensuse132c0:/

26

Summary

systemd-nspawn • Makes containers easy• Everyone familiar with “chroot” instantly “gets” systemd-nspawn• Does not have special dependencies, like e.g. docker• It is available on all modern Linux distro's

Thank you.

27

Questions?