Your first dive into systemd!

Your first dive into systemd!

1

ver1.4 Etsuji NakaiTwitter @enakai00

Open Cloud Campus

Your first diveinto systemd!

Open Cloud Campus2


$ who am i

–The author of “Professional Linux Systems” series.• Available only in Japanese. Translation offering from publishers are welcomed ;-)

Self-study LinuxDeploy and Manage by yourself

Professional Linux SystemsDeployment and Management

Professional Linux SystemsNetwork Management

Etsuji Nakai– Senior solution architect and cloud

evangelist at Red Hat.

Professional Linux SystemsTechnology for Next Decade

Open Cloud Campus3


Contents

Review: SysVinit & Upstart System boot process with systemd Operating systemd Log management with journald Unit configuration files Hint and tips References

(*) These contents are based on Fedora 19


4

Review: SysVinit & Upstart

Open Cloud Campus5


Linux System Boot Process (1)

“System BIOS” reads the boot loader program (GRUB) into memory from the boot disk.

GRUB displays the boot kernel selection menu, reads the specified kernel and initrd into memory, and start the kernel execution.

Kernel extracts the contents of initrd into the RAM disk area of memory, and kicks the “init script.”

– initrd contains device drivers necessary for root file system access and the “init script.”

Boot Loader （GRUB）

/boot filesystem

(2) Read by the boot loader

(3) Extracted into the RAM disk area

Boot DiskPhysical Memory

Linux Kernel

initrd

Linux Kernel

initrd

RAM disk area- Device Drivers- init script- Other commands

(1) Executed by System BIOS

Open Cloud Campus6


Linux System Boot Process (2)

“init script” loads appropriate device drivers and mounts the root filesystem. Then it executes the initial process “/sbin/init”.

– Strictly speaking, “init script” is replaced by “/sbin/init” with “exec” command.

This “/sbin/init” is the main body of the traditional SysVinit or Upstart. Under the systemd environment, this will be “/usr/bin/systemd” instead.

(*) Since initrd is a cpio+gzip archive file, you can extract it with pax command.– # pax -rzf /boot/initramfs-2.6.32-131.0.15.el6.x86_64.img

#!/bin/sh## Licensed under the GPLv2## Copyright 2008-2009, Red Hat, Inc.# Harald Hoyer <[email protected]># Jeremy Katz <[email protected]>...snip...# start up udev and trigger cold plugsudevd --daemon –resolve-names=never...snip... exec switch_root "$NEWROOT" "$INIT" $initargs || { warn "Something went very badly wrong in the initramfs. Please " warn "file a bug against dracut." emergency_shell }fi

udevd loads necessary device drivers.

This process is replaced by /sbin/init with exec.

Example of init script from RHEL6

Open Cloud Campus7


How SysVinit works

SysVinit / Upstart does user land system initialization and service startups. RHEL5's /sbin/init (SysVinit) does the following tasks according to “/etc/inittab”.

id:5:initdefault:

# System initialization.si::sysinit:/etc/rc.d/rc.sysinit

l0:0:wait:/etc/rc.d/rc 0l1:1:wait:/etc/rc.d/rc 1l2:2:wait:/etc/rc.d/rc 2l3:3:wait:/etc/rc.d/rc 3l4:4:wait:/etc/rc.d/rc 4l5:5:wait:/etc/rc.d/rc 5l6:6:wait:/etc/rc.d/rc 6...snip...

# Run gettys in standard runlevels1:2345:respawn:/sbin/mingetty tty12:2345:respawn:/sbin/mingetty tty23:2345:respawn:/sbin/mingetty tty34:2345:respawn:/sbin/mingetty tty45:2345:respawn:/sbin/mingetty tty56:2345:respawn:/sbin/mingetty tty6

# Run xdm in runlevel 5x:5:respawn:/etc/X11/prefdm -nodaemon

Example of /etc/inittab from RHEL5– Execute the system initialization script “/etc/rc.d/rc.sysinit”

which does fsck and mount of filesystems, and other various initialization works.

– Execute the service startup script “/etc/rc.d/rc” which starts services according to the current runlevel.

• The actual startup tasks are done by the separate service scripts under /etc/init.d/.

– Start mingetty processes which accepts the console user login. And start the GUI login screen for runlevel 5.

mingettyprefdmrc.sysinit

System initialization Service startups Accept user login

rc

# /etc/init.d/<service> start

Open Cloud Campus8


Upstart's Job flow

RHEL6's ”/sbin/init” is an event based job management system called “Upstart.”– Only the default runlevel is written in “/etc/inittab”. Various jobs are executed according to

the job configuration files under /etc/init/ on the system startup.

However, the actual tasks done by those jobs are almost the same as in RHEL5.

Job: "rcS"

rc.sysinittelinit $runlevel

Event "startup" is issued

/etc/init/rcS.conf

Event "runlevel X" is issued

/etc/init/rc.conf

Event "stopped rc" is issued

Job: "start-ttys"

initctl start tty TTY=$tty

/etc/init/start-ttys.conf

Job: "prefdm"

/etc/X11/prefdm -nodaemon

/etc/init/prefdm.conf

If RUNLEVEL=5

Job: "tty"

/sbin/mingetty TTY

Job: "rc"

/etc/rc.d/rc $RUNLEVEL

/etc/init/tty.conf

Execute /sbin/init

Open Cloud Campus9


Service startup scripts

Service startup scripts “/etc/init.d/<service>” should accept the following options at least.

– start : Starting the service.– stop : Stopping the service.– status : Replying the service status through the return code.

Since they are mere shell scripts, they could accept non-standard options, or could be used for purposes other than the service(daemon) startup.

– PostgreSQL's database cluster initialization. (Non-standard option.)• # service postgresql initdb

– Running the configuration wizard at the first boot time. (Not for service startup.)• # service firstboot start

Open Cloud Campus10


Objectives of systemd

Reducing the system startup time.– Shell scripts execute commands in serial order, which is not efficient under the modern multi

core processors. If various tasks in the script can be split and executed in parallel, it will reduce the system startup time.

Handling dynamic system configuration changes.– Not only at the system startup time, but also at system configuration changes, services need

to be dynamically started/stopped. Providing the standard method for process shutdown.

– There has been no standard method to stop processes in the service script. Some use the PID file while others rely on a process name to determine which processes should be stopped. systemd should provide the standard methods with tracking the forked daemons by itself.

Controlling the process execution environment– Process execution environment such as resource management with cgroups and directory

access control should be centrally managed by systemd.

http://0pointer.de/blog/projects/systemd.html


11

System boot process with systemd

Open Cloud Campus12


The minimum execution unit of systemd

Various tasks in the SysVinit's system initialization scripts are extracted and defined as independent “Unit”s.

rc.sysinit rc

systemd-remount-fs.service systemd-udevd.service・・・ chronyd.service

crond.servicedbus.serviceirqbalance.servicemdmonitor.serviceNetworkManager.servicerngd.servicerpcbind.servicersyslog.servicesshd.service・・・

console-getty.servicesystemd-logind.service

dev-hugepages.mountproc-sys-fs-binfmt_misc.mounttmp.mount ・・・

mingettyprefdm

# /etc/init.d/<service> start

sys-devices-pci00...0:00:03.0-virtio0-net-eth0.devicesys-devices-pci00...4.0-virtio1-block-vda-vda1.devicesys-devices-pci00...4.0-virtio1-block-vda-vda2.devicesys-devices-pci00...:00:04.0-virtio1-block-vda.device dev-dm\x2d1.swap ・・・

These are all “Unit”.

Open Cloud Campus13


Unit types

Unit has various types which are indicated by their name extensions. The followings are some major units.

– .service：Service type• When activated, an associated daemon is started.

– .target：Target type• Do nothing. This is used to group other units when defining unit dependencies, or to

provide a timing synchronization point when defining order relationships.– .mount：Mount point type

• When activated, an associated filesystem is mounted.– .swap：Swap area type

• When activated, an associated swap area is enabled.– .device：Device type

• When udev recognizes a new device, the associated unit is defined and activated automatically.

Some units are dynamically generated. You don't need to define all units by hand.– .service and .target are defined through config files by hand.– .mount and .swap are automatically generated from /etc/fstab.– .device is automatically generated by udev.

Open Cloud Campus14


Location of unit configuration files

Unit configuration files are placed in two locations.– Under /etc/systemd/system/ : Admin customized files.– Under /usr/lib/systemd/system/ : System default files installed from RPM packages.

If configuration files of the same filename are in both places, one in /etc/systemd/system is used. (One in /usr/lib/systemd/system is simply ignored.)

– When modifying the system default configuration, you need to copy the configuration file in /usr/lib/systemd/system to /etc/systemd/system, and modify the copied one.

The name of configuration file corresponds to its unit name.– So, you can create an alias name just by creating a symlink to the original configuration file.– Directories “<unit name>.wants” are used to define unit dependencies, which is explained

later.

basic.target.wantsdbus-org.fedoraproject.FirewallD1.service -> /usr/lib/systemd/system/firewalld.servicedbus-org.freedesktop.Avahi.service -> /usr/lib/systemd/system/avahi-daemon.servicedbus-org.freedesktop.NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.servicedefault.target -> /lib/systemd/system/multi-user.targetdefault.target.wantsgetty.target.wantsmulti-user.target.wantssockets.target.wantssysinit.target.wantssyslog.service -> /usr/lib/systemd/system/rsyslog.servicesystem-update.target.want

Unit configuration files under /etc/systemd/system (Sample)

Open Cloud Campus15


Unit dependencies and orders

There are two independent definitions among units. One is “dependencies” and the other is “orders”

– Dependency means “if unit A is activated, unit B must be activated, too.”– Order means “unit A must be activated after unit B.”

At a startup, systemd tries to activate the unit named “default.target”. Then all units under its dependency tree are automatically activated.

– default.target is a symlink to another target such as multi-user.target or graphical.target. Changing this symlink corresponds to changing the “default runlevel.”

– The drawing shows a typical dependency tree where the backbone is made of target units and other real units are under those targets.

default.target multi-user.target basic.target

NetworkManager.service avahi-daemon.service irqbalance.service remote-fs.target rsyslog.service ・・・

symlink

fedora-autorelabel-mark.servicefedora-autorelabel.servicefedora-configure.servicefedora-loadmodules.service

sys-kernel-config.mountsys-kernel-debug.mountsystemd-journald.servicesystemd-modules-load.servicesystemd-random-seed-load.servicesystemd-sysctl.servicesystemd-udev-trigger.servicesystemd-udevd.service・・・

sysinit.target

Open Cloud Campus16


Overview of major dependencies

multi-user.target

symlink

graphical.target

rescue.target

Services activatedunder runlevel 1



basic.target

default.target

sysinit.target

Services independentof the runlevel

runlevelに依存せず起動するサービス

Tasks traditionallyHandled by rc.sysinit

symlink can bechanged.

local-fs.target

swap.target

Enabling swap ares

Mounting filesystems

Open Cloud Campus17


Defining dependencies and orders (1)

At startup, systemd tries to figure out all units which should be activated as a whole, based on their dependencies.

– Note that the other relationship “orders” is defined independent of “dependencies.” They are totally orthogonal.

– systemd tries to activate multiple (dependent) units in parallel unless they have some order relationships in order to accelerate the system startup speed.

Dependencies are defined in following ways.– Use “Wants=” or “Requires=” to specify the pre-req units (i.e. units which must be activated

together) in the [Unit] section of the configuration file.– Create a symlink to the configuration files of the pre-req units in “<unit name>.wants” or

“<unit name>.requires” directory.– “Requires” means if the pre-req unit fails to start, this unit should not start either.– “Wants” means even if the pre-req unit fails to start, this unit should start anyway.– In addition, you can use “Conflicts=” option to specify the conflicting units which should not

be activated at the same time with this unit.

Open Cloud Campus18



“<unit name>.wants” directories are used to configure autostartup of services.

– When autostartup is enabled with the systemctl command, a symlink to the service's configuration file is created under the .wants directory specified by “WantedBy=” option.

# systemctl disable sshd.servicerm '/etc/systemd/system/multi-user.target.wants/sshd.service'# systemctl enable sshd.serviceln -s '/usr/lib/systemd/system/sshd.service' '/etc/systemd/system/multi-user.target.wants/sshd.service'

# ls -l /etc/systemd/system/multi-user.target.wants/total 0lrwxrwxrwx. 1 root root 46 Oct 30 12:35 NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.servicelrwxrwxrwx. 1 root root 35 Oct 30 12:36 atd.service -> /usr/lib/systemd/system/atd.servicelrwxrwxrwx. 1 root root 38 Oct 30 12:36 auditd.service -> /usr/lib/systemd/system/auditd.servicelrwxrwxrwx. 1 root root 44 Oct 30 12:35 avahi-daemon.service -> /usr/lib/systemd/system/avahi-daemon.servicelrwxrwxrwx. 1 root root 39 Oct 30 12:36 chronyd.service -> /usr/lib/systemd/system/chronyd.service...snip...lrwxrwxrwx. 1 root root 36 Nov 27 20:38 sshd.service -> /usr/lib/systemd/system/sshd.service

Sample of enabling / disabling autostartup of sshd.service

[Unit]Description=OpenSSH server daemonAfter=syslog.target network.target auditd.service

[Service]EnvironmentFile=/etc/sysconfig/sshdExecStartPre=/usr/sbin/sshd-keygenExecStart=/usr/sbin/sshd -D $OPTIONSExecReload=/bin/kill -HUP $MAINPIDKillMode=process

[Install]WantedBy=multi-user.target

/usr/lib/systemd/system/sshd.service

Open Cloud Campus19



Unit orders are defined with “After=” and “Before=” options in the unit configuration files. – “After=A B C” means this unit needs to be activated after “A”, “B” and “C”.– “Before=A B C” means this unit needs to be activated before “A”, “B” and “C”.

target units are used to define synchronization points of multiple units.– For example, “network.target” is used as a synchronization point of “units providing network

functions” and “units using network functions”.

network.targetUnits to provide

network functionsUnits to use

network functionsBefore=network.targetAfter=network.target

[Unit]Description=firewalld - dynamic firewall daemonBefore=network.targetBefore=libvirtd.serviceBefore=NetworkManager.service・・・

[Unit]Description=The Apache HTTP ServerAfter=network.target remote-fs.target nss-lookup.target・・・

/usr/lib/systemd/system/firewalld.service/usr/lib/systemd/system/httpd.service

httpd is assured to startafter firewall has been configured.

Open Cloud Campus20



The following commands show the dependencies and orders of currently active units.– # systemctl list-dependencies <unit name>

• Show units which are pre-req of the specified unit (“default target” unless specified). • If the pre-req unit is of target type, its pre-req units are shown recursively. • –-all option shows all units recursively.

# systemctl list-dependencies default.target├─atd.service├─auditd.service...snip...├─basic.target│ ├─fedora-autorelabel-mark.service...snip...│ ├─firewalld.service│ ├─paths.target│ ├─sockets.target│ │ ├─avahi-daemon.socket│ │ ├─dbus.socket...snip...│ ├─sysinit.target│ │ ├─dev-hugepages.mount│ │ ├─dev-mqueue.mount│ │ ├─lvm2-monitor.service│ │ ├─plymouth-read-write.service│ │ ├─plymouth-start.service...snip...├─getty.target│ └─[email protected]└─remote-fs.target

– # systemctl list-dependencies <unit name> --after • Show units which are activated before the specified

unit. (i.e. the specified one is activated after them.)• --all option works as above.

– # systemctl list-dependencies <Unit名> --before • Show units which are activated after the specified

unit. (i.e. the specified one is activated before them.)• --all option works as above.

Open Cloud Campus21


Dynamic unit generation (1)

Generator modules under /usr/lib/systemd/system-generators/ provide dynamic unit generation and modification according to system configuration changes.

– Example:– systemd-cryptsetup-generator

• Generate [email protected] according to /etc/crypttab.– systemd-fstab-generator

• Generate mount and swap type units according to /etc/fstab.• For a mount type unit, its name corresponds to the mount point directory path.

(/ is replaced with _.) – systemd-rc-local-generator

• Enable autostart of rc-local.service if /etc/rc.d/rc.local exists as an executable file.

Configuration files of dynamically generated units are placed under /run/systemd/generator/.

# ls /run/systemd/generator/-.mountboot.mountdev-disk-by\x2duuid-89cd76be\x2d8d59\x2d441c\x2d9165\x2dfe8ff338266b.device.wantsdev-mapper-fedora\x2dswap.device.wantsdev-mapper-fedora\x2dswap.swaplocal-fs.target.requiresswap.target.wants

Open Cloud Campus22


Dynamic unit generation (2)

When udev assigns “systemd” tag to a new device, the corresponding unit is generated and activated.

– Its name corresponds to the device's /sys filesystem path. (/ is replaced with _.)

bluetooth.target, printer.target, smartcard.target and sound.target will be set as pre-req of dynamically generated device units - bluetooth controllers, printers, smartcards and soundcards, respectively.

– This means, for example, you can configure some units which requires soundcards as pre-req of sound.target so that those units are automatically activated when a soundcard device is attached to the system.

# systemctl list-units --type=device --fullUNIT LOAD ACTIVE SUB DESCRsys-devices-pci0000:00-0000:00:01.1-ata2-host1-target1:0:0-1:0:0:0-block-sr0.device loaded active plugged QEMU_sys-devices-pci0000:00-0000:00:03.0-virtio0-net-eth0.device loaded active plugged Virtisys-devices-pci0000:00-0000:00:04.0-virtio1-block-vda-vda1.device loaded active plugged /sys/sys-devices-pci0000:00-0000:00:04.0-virtio1-block-vda-vda2.device loaded active plugged /sys/sys-devices-pci0000:00-0000:00:04.0-virtio1-block-vda.device loaded active plugged /sys/sys-devices-platform-serial8250-tty-ttyS1.device loaded active plugged /sys/sys-devices-platform-serial8250-tty-ttyS2.device loaded active plugged /sys/sys-devices-platform-serial8250-tty-ttyS3.device loaded active plugged /sys/sys-devices-pnp0-00:04-tty-ttyS0.device loaded active plugged /sys/sys-devices-virtual-block-dm\x2d0.device loaded active plugged /sys/sys-devices-virtual-block-dm\x2d1.device loaded active plugged /sys/sys-module-configfs.device loaded active plugged /sys/sys-subsystem-net-devices-eth0.device loaded active plugged Virti


23

Operating systemd

Open Cloud Campus24


Listing units (1)

# systemctl list-unit-files– List all defined units and their statuses.– --type option lists only the specified type.

Status Description

enabled “WantedBy=” entry exists.Autostart enabled.

disabled “WantedBy=” entity exists.Autostart disabled.

static “WantedBy=” entity doesn't exists.

# systemctl list-unit-files --type=serviceUNIT FILE STATE arp-ethers.service disabledatd.service enabled auditd.service enabled [email protected] disabledavahi-daemon.service enabled blk-availability.service disabledchrony-wait.service disabledchronyd.service enabled console-getty.service disabledconsole-shell.service disabledcrond.service enabled dbus-org.fedoraproject.FirewallD1.service enabled dbus-org.freedesktop.Avahi.service enabled dbus-org.freedesktop.hostname1.service static dbus-org.freedesktop.locale1.service static dbus-org.freedesktop.login1.service static dbus-org.freedesktop.NetworkManager.service enabled dbus-org.freedesktop.timedate1.service static dbus.service static ...snip...

Similar to the traditional“chkconfig --list”

You can use systemctl command to enable/disable autostart.

Open Cloud Campus25


Listing units (2)

# systemctl list-units （”list-units” can be omitted.)– List currently active (or should be active) units and their statuses.– --type option lists only the specified type.

# systemctl --type=serviceUNIT LOAD ACTIVE SUB DESCRIPTIONatd.service loaded active running Job spooling toolsauditd.service loaded active running Security Auditing Serviceavahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stackchronyd.service loaded active running NTP client/servercrond.service loaded active running Command Schedulerdbus.service loaded active running D-Bus System Message Busfedora-readonly.service loaded active exited Configure read-only root suppfirewalld.service loaded active running firewalld - dynamic firewall [email protected] loaded active running Getty on tty1irqbalance.service loaded active running irqbalance daemonlvm2-lvmetad.service loaded active running LVM2 metadata daemonlvm2-monitor.service loaded active exited Monitoring of LVM2 mirrors, smcelog.service loaded active running Machine Check Exception LoggiNetworkManager.service loaded active running Network Managerpolkit.service loaded active running Authorization Managerrngd.service loaded failed failed Hardware RNG Entropy Gathererrpcbind.service loaded active running RPC bind service・・・systemd-v...le-setup.service loaded active exited Setup Virtual Console

LOAD = Reflects whether the unit definition was properly loaded.ACTIVE = The high-level unit activation state, i.e. generalization of SUB.SUB = The low-level unit activation state, values depend on unit type.

31 loaded units listed. Pass --all to see loaded but inactive units, too.To show all installed unit files use 'systemctl list-unit-files'.

Open Cloud Campus26


Basic unit operations (1)

# systemctl enable/disable <unit name>– Enable/disable autostart of the specified unit. (It defines/undefines dependency from the

unit specified with “WantedBy=” option in the background.)

# systemctl start/stop/restart <unit name>– Start/stop/restart the specified unit.– “reload” can be used only when “ExecReload=” is defined in the unit configuration file.

# systemctl status <unit name>– Show running status of the specified unit.

# systemctl status httpd.servicehttpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled) Active: active (running) since Sun 2013-11-03 15:59:37 JST; 4 weeks 2 days ago Main PID: 4621 (httpd) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" CGroup: name=systemd:/system/httpd.service ├─4621 /usr/sbin/httpd -DFOREGROUND ├─4622 /usr/sbin/httpd -DFOREGROUND ├─4623 /usr/sbin/httpd -DFOREGROUND ├─4624 /usr/sbin/httpd -DFOREGROUND ├─4625 /usr/sbin/httpd -DFOREGROUND └─4626 /usr/sbin/httpd -DFOREGROUND

Nov 03 15:59:36 fedora19 systemd[1]: Starting The Apache HTTP Server...Nov 03 15:59:36 fedora19 httpd[4621]: AH00557: httpd: apr_sockaddr_info_get...19Nov 03 15:59:36 fedora19 httpd[4621]: AH00558: httpd: Could not reliably de...geNov 03 15:59:37 fedora19 systemd[1]: Started The Apache HTTP Server.

Recent entries of daemon logs

daemon processes of this service

Open Cloud Campus27


Basic unit operations (2)

# systemctl daemon-reload– Let systemd reload configuration files. You need to run this when you modified a unit

configuration file.

chkconfig and service commands are translated to corresponding systemctl commands.

Non-standard options in service startup scripts cannot be used with systemd command.

– For example, you need to use postgresql-setup for database cluster initialization of PostgreSQL.

• RHEL6: # service postgresql initdb• Fedora19: # postgresql-setup initdb

# chkconfig sshd offNote: Forwarding request to 'systemctl disable sshd.service'.rm '/etc/systemd/system/multi-user.target.wants/sshd.service'

# chkconfig sshd onNote: Forwarding request to 'systemctl enable sshd.service'.ln -s '/usr/lib/systemd/system/sshd.service' '/etc/systemd/system/multi-user.target.wants/sshd.service'

# service sshd stopRedirecting to /bin/systemctl stop sshd.service

# service sshd startRedirecting to /bin/systemctl start sshd.service

Open Cloud Campus28


Listing cgroups

systemd creates a cgroups' group for each service.– # systemd-cgls : Show cgroups' group tree.– # systemd-cgtop : Display resource usage per group in real time.

# systemd-cgls├─system│ ├─1 /usr/lib/systemd/systemd --system --deserialize 17│ ├─httpd.service│ │ ├─4621 /usr/sbin/httpd -DFOREGROUND│ │ ├─4622 /usr/sbin/httpd -DFOREGROUND│ │ ├─4623 /usr/sbin/httpd -DFOREGROUND│ │ ├─4624 /usr/sbin/httpd -DFOREGROUND│ │ ├─4625 /usr/sbin/httpd -DFOREGROUND│ │ └─4626 /usr/sbin/httpd -DFOREGROUND│ ├─sm-client.service│ │ └─560 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue│ ├─sendmail.service│ │ └─495 sendmail: accepting connections│ ├─rpcbind.service│ │ └─461 /sbin/rpcbind -w│ ├─sshd.service│ │ └─4440 /usr/sbin/sshd -D・・・│ └─systemd-journald.service│ └─285 /usr/lib/systemd/systemd-journald└─user └─0.user ├─108.session │ ├─4521 sshd: root@pts/1 │ ├─4525 -bash │ ├─4654 systemd-cgls │ └─4655 cat

Subgroup for each unit is createdunder this parent group.

Subgroup for each user account iscreated under this parent group.

Open Cloud Campus29


Sending signal to process groups

You can send a process signal to all processes in the same cgroups' group.– Example：– # systemctl kill -s9 sshd.service

• Send KILL signal (signal# 9) to all processes in the group for sshd.service.– You can kill all the processes related to a specific service with this.– With “--kill-who=main” option, it sends the signal only to the main process which has started

first in the group.

This mechanism is used when stopping a unit with “systemctl stop”, too.– systemd first execute the command specified with “ExecStop=” in the configuration file. If

there remains some processes in the group, it sends “SIGTERM” to them. If there still remains some processes, it sends “SIGKILL”.


30

Log management with journald

Open Cloud Campus31


Log management mechanism of systemd

systemd intercepts messages to stdout/stderr and rsyslogd from daemon processes which have been started as service units. The messages are sent to its own logging service “systemd-journald.service”, or “journald” in short.

– journald records those messages in binary files under /var/log/journal/ with adding some meta information of sender processes.

– You can use the “journalctl” command to search the binary log.

rsyslogd should be configured accordingly.– In the traditional environment, rsyslogd receives processes' system log messages through the

unix socket “/dev/log”. Under the systemd environment, systemd receives messages from “/dev/log” and send it again to “/run/systemd/journal/syslog”. So rsyslogd needs to receive it through “/run/systemd/journal/syslog”.

rsyslogd

/dev/log

Process

/dev/log

Process

rsyslogd

/run/systemd/journal/syslog

systemd

$SystemLogSocketName /run/systemd/journal/syslog

/etc/rsyslog.d/listen.conf

Traditional system log flow

System log flowunder systemd

Open Cloud Campus32


Log search with journalctl

The followings are major options of journalctl command.– -u <unit name> : Show log messages related to this unit.– --since="YYYY-MM-DD hh:mm:ss" : Show log messages since this date/time.– --until="YYYY-MM-DD hh:mm:ss" : Show log messages until this date/time.– -b : Show log messages since the last system boot.– -f : Works as “tail -f”– --no-pager: Don't use the pager(less)– -a : Don't cut long messages.

# journalctl -u sshd.service -b --no-pager -a-- Logs begin at Wed 2013-10-30 12:40:32 JST, end at Tue 2013-12-03 21:18:59 JST. --Oct 30 12:59:39 fedora19 systemd[1]: Starting OpenSSH server daemon...Oct 30 12:59:39 fedora19 systemd[1]: Started OpenSSH server daemon.Oct 30 12:59:40 fedora19 sshd[454]: Server listening on 0.0.0.0 port 22.Oct 30 12:59:40 fedora19 sshd[454]: Server listening on :: port 22.Oct 30 12:59:50 fedora19 sshd[824]: Accepted password for root from 192.168.122.1 port 42680 ssh2Nov 01 16:08:57 fedora19 sshd[2037]: Accepted password for root from 192.168.122.1 port 52666 ssh2Nov 02 20:16:31 fedora19 sshd[3215]: Accepted password for root from 192.168.122.1 port 57418 ssh2Nov 03 09:23:13 fedora19 sshd[3855]: Accepted password for root from 192.168.122.1 port 33521 ssh2Nov 03 14:55:54 fedora19 sshd[4301]: Accepted password for root from 192.168.122.1 port 33634 ssh2Nov 03 15:49:23 fedora19 systemd[1]: Stopping OpenSSH server daemon...Nov 03 15:49:23 fedora19 sshd[454]: Received signal 15; terminating.Nov 03 15:49:24 fedora19 systemd[1]: Stopped OpenSSH server daemon.Nov 03 15:49:25 fedora19 systemd[1]: Starting OpenSSH server daemon...Nov 03 15:49:25 fedora19 systemd[1]: Started OpenSSH server daemon.Nov 03 15:49:25 fedora19 sshd[4440]: Server listening on 0.0.0.0 port 22.Nov 03 15:49:25 fedora19 sshd[4440]: Server listening on :: port 22.Nov 03 15:56:39 fedora19 sshd[4461]: Accepted password for root from ::1 port 51491 ssh2


33

Unit configuration files

Open Cloud Campus34


General configuration options (1)

Unit configuration file has multiple sections such as [Unit], [Install], [Service], etc.– [Unit] : Common options for all unit types such as dependency and order.– [Install]：Options related to “systemctl enable/disable” operation.– [Service]：Options specific to the service type.– Each type has its own section in general.

Major options in [Unit] section.

– If you need to specify multiple items in the same option, they can be space separated. You can use the same option multiple times, too. (This rule applies to all sections.)

Option Description

Description Short documentation of this unit.

Documentation Document URL

Requires/Wants(*) Pre-req units of this unit.

After This unit will be activated after these units.

Before This unit will be activated before these units.

(*) ”Requires” means this unit won't be activated if pre-req units failed to start.　 “Wants” means this unit will be activated even if pre-req units fails to start.

Open Cloud Campus35


General configuration options (2)

Major options in [Install] section.

A

– “WantedBy” and “RequiredBy” is to specify the unit to which this unit becomes a pre-req when autostart is enabled.

Option Description

WantedBy When enabled, symlink to the configuration files is created under this unit's .wants directory.

RequiredBy When enabled, symlink to the configuration files is created under this unit's .required directory.

Also When enabled/disabled, this unit is enabled/disabled, too.




This service is activated aftersyslog/network/auditd is ready.

This service is enabled asa pre-req of multi-user.target


Open Cloud Campus36


Configuration options for service type unit (1)

Major options in [service] section (1)

– “ExecXXX” options specify the commands to start/reload/stop the service. Environment variables from “EnvironmentFile” and the special variable “$MAINPID” can be used. Details are explained later.

– systemd judges if the service is successfully started based on the result of ExecStart command. The result of ExecStartPre/ExecStartPost commands are not considered.

– ExecStopPost command is executed not only when the service is stopped with ExecStop but also when the service aborts.

Option Description

ExecStart Command to start the service

ExecReload Command to reload the service

ExecStop Command to stop the service

ExecStartPreExecStartPost

Additional commands to be executed before/after ExecStart.

ExecStopPost Command to be executed when the service stops including the case of abort.

EnvironmentFile Read environment variables from this file.

KillMode Specify how remaining processes are handled after ExecStop.

Open Cloud Campus37




– Type option specifies the timing when systemd considers that the service is successfully started.

• Type=simple : Used when the specified command runs in foreground. systemd considers it's successfully started just when the command is executed.

• Type=forking : Used when the specified command forks a child process and exits.　systemd considers it's successfully started when the command exits.

• Type=oneshot : Used when the specified command does oneshot task and exits. systemd considers it's successfully executed and the service has been finished. (If “RemainAfterExit=yes” is set, it considers that the service is still active.）

• Type=notify : Used when the specified command uses the systemd's library function “sd_notify()”. The process needs to be designed to use sd_notify()(*).

• Type=dbus : Used when the service uses D-Bus (Inter process messaging bus). systemd considers it's successfully started whtn the connection name specified with “BusName” is registered in D-Bus.

Option Description

Type Specify how the service startup is detected.（Default: simple)

PIDFile PID file of the main process of “fork” type service.

BusName Bus connection name of “D-Bus” type service.

Restart Specify whether the service is restarted when it aborts.（Default: no)

(*) https://fedoraproject.org/wiki/User:Johannbg/QA/Systemd/Sd_notify

Open Cloud Campus38



systemd tracks PID of the main process which has been started first in the process group of the service. This PID can be referred with $MAINPID in ExecXXX options.

– You use it for sending HUP signal to the main process in ExecReload, for example.– For “type=simple”, the main process is the one started with ExecStart.– For “type=forking”, systemd detects it from the PID file specified with PIDFile option. (The

service startup command needs to create the PID file by itself.）

When the process aborts, systemd takes the following actions. (Corresponds to the respawn configuration in the traditional case.)

– Restart option specified if systemd tries to restart service.• Restart=no : It doesn't restart service.• Restart=always : It tries to restart service.

– In default, if the service restarted more than five times in ten seconds, systemd doesn't restart it for next ten seconds. In general, if the service restarted more than “StartLimitBurst” times in “StartLimitInterval”, systemd doesn't restart it for next “StartLimitInterval”. (i.e. The default is StartLimitInterval=10s and StartLimitBurst=5.)

Open Cloud Campus39



systemd creates a cgroups' group for each service to track the related processes.– When the service is stopped with ExecStop command, systemd takes the following actions

according to “KillMode” option if there are some remaining processes in the group.• KillMode=none : The remaining processes are left untouched.• KillMode=process : If the main process is still running, systemd stops it with

SIGTERM/SIGKILL. Other processes are left untouched.• KillMode= control-group : All remaining processes in the group are stopped with

SIGTERM/SIGKILL.




Generate hostkey before starting the service.

Reload is done by sendingHUP to the main process.


With -D option (daemon mode), the initialcommand keeps running in foreground.

Hence, “Type=simple” (default) should be used.Child processes are left

when the service is stopped.

Open Cloud Campus40



Use of “Type=simple/notify” is recommended instead of “Type=forking” in order to avoid the use of PID file.

– For example, httpd.service uses “Type=forking” in Fedora17, but it's replaced with “Type=notify” in Fedora19.

[Service]Type=notifyEnvironmentFile=/etc/sysconfig/httpdExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUNDExecReload=/usr/sbin/httpd $OPTIONS -k gracefulExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop

KillSignal=SIGCONTPrivateTmp=true

/usr/lib/systemd/system/httpdd.service in Fedora19

[Service]Type=forkingPIDFile=/var/run/httpd/httpd.pidEnvironmentFile=/etc/sysconfig/httpdExecStart=/usr/sbin/httpd $OPTIONS -k startExecReload=/usr/sbin/httpd $OPTIONS -tExecReload=/bin/kill -HUP $MAINPIDExecStop=/usr/sbin/httpd $OPTIONS -k stopPrivateTmp=true

/usr/lib/systemd/system/httpdd.service in Fedora17

The main process is tracked with PID file.

This command exits after forkingchild process in background.

httpd is integrated with systemd.(*)

With -DFOREGROUND option, this commandkeeps running in foreground. The service startup

is notified with sd_notify().

(*) httpd uses mod_systemd module for systemd integration. http://bit.ly/1bTIoKg

Open Cloud Campus41




– User/Group specifies user/group to execute commands in “ExecXXX=” options.• If “PermissionsStartOnly=yes” is set, only “ExecStart=” command is executed with this

user/group.– Other options are used to limit directory access from the processes of this service.

• These features use “filesystem namespace” separation mechanism in background. Because of this, the bind-mount done by the processes in this service are invisible from other processes(*).

(*) For example, if processes in the service use “ip netns”, this may cause a problem because this command bind-mounts network namespace information under /proc onto /var/run/netns to share it with other processes.

Bug 872689 - Quantum: root cannot access network namespaces created by Quantum service https://bugzilla.redhat.com/show_bug.cgi?id=872689

Option Description

User / Group user/group to start processes.

PrivateTmp Prepare private /tmp and /var/tmp for this service.

ReadOnlyDirectories The specified directory becomes read only.

InaccessibleDirectories The specified directory becomes inaccessible.

RootDirectory chroot-ed to the specified directory.


42

Hint and tips

Open Cloud Campus43


Disable with masking

# systemctl mask/unmask <unit name>– Disable/enable the specified unit with masking it. When disabled with masking, it cannot be

activated with “systemctl start”.– When disabled with masking, its configuration file is created as a symlink to /dev/null

under /etc/systemd/system. If a configuration files already exits in /etc/systemd/system/, it cannot be masked.

# systemctl mask firewalld.serviceln -s '/dev/null' '/etc/systemd/system/firewalld.service'

# systemctl unmask firewalld.servicerm '/etc/systemd/system/firewalld.service'

Open Cloud Campus44


Template style configuration file

A configuration file with the name “[email protected]” works as a configuration for multiple services as “foo@<arbitrary string>.service”

– # systemctl start [email protected]• Service “[email protected]” is activated according to the configuration file “[email protected]”.

The parameter “%I” in the configuration file is replaced by “<arbitrary sting>” (“bar” in this case).（”%i” can also be used. It is an escaped version of “%I”.）

– # systemctl enable [email protected]• A symlink from “[email protected]” to “[email protected]” in the .wants directory. When systemd

tires to autostart “[email protected]”, it actually uses the configuration file “[email protected]”.

– In the example below, “[email protected]”, “[email protected]”, etc. can be used.

[Service]# the VT is cleared by TTYVTDisallocateExecStart=-/sbin/agetty --noclear %I 38400 linuxType=idleRestart=alwaysRestartSec=0UtmpIdentifier=%ITTYPath=/dev/%ITTYReset=yesTTYVHangup=yesTTYVTDisallocate=yesKillMode=processIgnoreSIGPIPE=no

/lib/systemd/system/[email protected]（Snippet from [service] section)


45

References

Open Cloud Campus46


References

freedesktop.org - systemd System and Service Manager– Project home of systemd– http://www.freedesktop.org/wiki/Software/systemd/

The systemd for Administrators Blog Series– Blog series by the original developer Lennart (Links are on the project home.)

Rethinking PID 1– Lennart's blog about the background idea of systemd– http://0pointer.de/blog/projects/systemd.html

man page– systemd is supplemented with useful man pages. In addition to a man page for each command,

you can follow “SEE ALSO” section of systemd(1).


47

Etsuji NakaiTwitter @enakai00

Open Cloud Campus

Let's study the latestLinux features with Fedora!

Date post:	08-Sep-2014
Category:	Technology
Upload:	etsuji-nakai
View:	3,849 times
Download:	7 times