systemd @ FB – a year later

Davide CavalcaProduction Engineer

• Recap• Tracking upstream• Resource management• Service monitoring• Case studies• Advocacy

Agenda

Recap

• 100% of the bare metal feet on CentOS 7!• Migrated countless services to systemd• libsystemd integration in our build system• Containers: see Zeal’s talk later today!

RecapCentOS 7 migration

Tracking upstream

• systemd 231 232 233 (234 235)→ → → →• Also tracking util-linux, dbus, etc.• Published our Rawhide-based backports on:

https://github.com/facebookincubator/rpm-backports• Binary RPMs based on it on:

https://copr.fedorainfracloud.org/coprs/jsynacek/systemd-backports-for-centos-7/

Tracking upstreamStaying up to date

• Not specifc to systemd• Duplicate systemd RPMs: package-cleanup wrapper• rpmdb corruption: dcrpm• Mismatch between systemd and systemd-libs

Tracking upstreamRPM issues

if ldd /usr/lib/systemd/systemd | grep ‘systemd.*not found$’ yum reinstall -y $systemd_packagesfi

• Rebuild packaging for the Meson transition• Backported meson, ninja-build in CentOS• Standalone systemd-compat-libs

https://github.com/facebookincubator/systemd-compat-libs

Tracking upstreamMeson and compat-libs

Tracking upstreamtty woes with 234

• When rolling 234 we discovered a race in the kernel tty subsystem (repros all the way back to 4.0)• Turns out both systemd and Tupperware use the real tty0• Investigation still in progress, likely a use-after-free bug• Tupperware should probably just use a pty here

Resource management

• See Chris’s talk tomorrow for all things cgroup2!• Using systemd to partition services and apply limits• Lightweight daemon to collect metrics from /sys/fs/cgroup• Chef API to apply confgurations and manage experiments

Resource managementRolling out cgroup2

Resource managementSlice hierarchy

/||-system.slice||-workload.slice| || +-critical-wdb.slice|+-tbd.slice

Service monitoring

Service monitoring

• systemd exposes lots of useful metrics over dbus• Unit properties (e.g. *Timestamp*, NRestarts)• Status events (e.g. unit state changes)• Options: python-dbus, sd-bus, coreos/go-systemd/dbus

Getting metrics out of systemd

Service monitoring

• Lightweight daemon to feed systemd metrics to various monitoring systems• Polling for unit properties, subscriptions for status events• Initial implementation in golang

systemdmon

Service monitoring

• Thin Cython wrapper on top of sd-bus• Expose systemd dbus object model• ipython REPL for prototyping• Will be opensourced together with systemdmon

pystemd

Case studies

Case studiesdbus reliability

• Issues with dbus-daemon or the system bus afect systemd• systemctl hanging or failing Chef failing→• Easy to DoS the bus, especially with user services• Hard to remediate without a reboot

• Looking forward to dbus-broker!

Case studiesrpm macros for systemd services

• By default RPM macros will restart units on upgrade...• …which is a problem if you’ve also setup Chef to restart

• Solution: knob in our internal packaging tool to optionally disable the restart macro

Case studiesLogging

• Journald setup: 10MB in memory logging feeding rsyslog• journalctl is awesome• Double writing problem• No way to set per-unit limits

Case studiesUnit loops

• Easy to create loops with x-systemd-requires in fstab• systemd will delete a random unit to break loops

• Solution: add _netdev to the fstab entry• systemd-analyze to help debugging

systemd-tmpfiles-setup.service: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with smc_proxy.service/start

Case studiesTransient unit creep

• systemd-run creates units in /run/systemd/transient• If the unit fails, it sticks around in ‘failed’ state• 10k failed units 50% cpu usage for pid 1→• 30k failed units 100% cpu usage for pid 1→• Fix: call systemctl reset-failed periodically

Case studiesKillMode=process

• KillMode=process may leave stray processes in the cgroup• Changes to unit slices don’t apply unless the old slice is

empty• Fix: move to use KillMode=control-group

Case studiesUnit escaping

• Escape logic relies on shell control characters:/dev/dm0 dev-dm\x2d1.swap→• Chef fx: https://github.com/chef/chef/pull/6230• path_to_unit wrapper in fb_systemd

Advocacy

• Announce core packages updates widely• Tailor documentation to customer usecases• Encourage people to engage upstream directly• Tech talks

Advocacy

Questions?