Building a skarnet.org s6 Init System

February 21, 2021

Per skarnet.org/software/ s6 is “skarnet.org’s small and secure supervision software suite. Comes with an ultra-fast init replacement, process management tools, an asynchronous locking library, and more.” After a little bit of reading and a lot of tinkering, I was able to assemble an s6 based system and learned a lot about linux userspace initialization in the process.

s6 is the core of an s6 based init system, but it is only one of several components that must be assembled to build a complete system. Per Laurent Bercot, the creator of the s6 software toolset, a “complete” system consists of the following components (ref: The s6 supervision suite A modern alternative to systemd):

/sbin/init: The program that the linux kernel executes to start up userspace. (Actually, /sbin/init is the first of several defaults that the linux kernel tries to execute, and another init program may be specified on the kernel command line.)
pid 1: The process that is the root of the linux userspace and never terminates. Initially this is /sbin/init, but /sbin/init can exec into another program which will then be pid 1.
Process Supervisor: This component monitors a set of long running processes (daemons) and ensures that if they ever terminate, they are restarted in a well defined environment.
Service Manager: This component, brings services up and down when required at boot time, shutdown or any other time the state of userspace needs to change. In the s6 world, services can be either long running daemons, or short lived programs with side effects (e.g, mounting: the service is up when the mount point is mounted and down when it is unmounted). A service manager also has knowledge of the dependency relationships between all of the services (if specified by the administrator) and ensures services are brought up or down in the correct sequence during a state change.

In an s6 based linux init system, component 1 is provided by s6-linux-init which is a part of the s6-linux-init package. This is the non-portable component in the system that is specific to the linux kernel. It performs some linux specific initializations and execs into the s6-svscan program…

Component 2 and 3: pid 1 and the process supervisor, are provided by s6-svscan which is a part of the s6 package. s6-svscan runs from the time /sbin/init execs into it during boot up until it terminates at shutdown. s6-svscan is the root of the process supervision tree.

Component 4, the service manager, is provided by s6-rc. s6-rc can start and stop services and can change the state of the machine (set of running/up services). It delegates the job of supervising its long running service processes to the service supervisor, s6-svscan.

s6-linux-init

Below is an outline of what s6-linux-init does taken directly from here.

It prints a banner to /dev/console.
It chdirs to /.
It sets the umask to initial_umask.
It becomes a session leader.
It mounts a devtmpfs on slashdev, if requested.
It uses /dev/null as its stdin (instead of /dev/console). /dev/console is still used, for now, as stdout and stderr.
It unmounts /run (or the directory you have given to the tmpfsdir configure option at package build time), just in case; then it creates a tmpfs on it. Alternatively, it remounts /run, or does not touch it at all.
It copies the whole basedir/run-image hierarchy to /run (or your chosen tmpfsdir).
It reads the initial environment from basedir/env.
If required, it stores the kernel environment into env_store.
It performs “the fifo trick”, i.e. it redirects its stdout to the catch-all logger’s fifo, without blocking, before the catch-all logger is even up (because it’s a service that will be spawned a bit later, when s6-svscan is executed).
It forks a child.
- The child scans the kernel command line to find a suitable runlevel (default, 2, 3, 4, or 5). If it doesn’t find any kernel command line argument that defines a runlevel, it uses initdefault.
- The child becomes a session leader.
- The child blocks until the catch-all logger runs.
It also makes the catch-all logger’s fifo its stderr.
It traps the ctrl-alt-del keyboard combination.
It execs into s6-svscan with /run/service as its scandir (or tmpfsdir/service).
s6-svscan spawns the early services that are defined in basedir/run-image/service, and have been copied into /run/service (or tmpfsdir/service).
One of those early services is s6-svscan-log, which is the catch-all logger. When this service is up, s6-linux-init’s child unblocks.
The child execs into basedir/scripts/rc.init. The first argument to rc.init is the chosen runlevel. The kernel command line, as given by the kernel to s6-linux-init (i.e. without the key=value arguments, which were passed into s6-linux-init’s environment and were stored into env_store), makes for the rest of the arguments given to rc.init.

As you can see, the actual s6-linux-init program is not a long lived daemon. It execs into s6-svscan which then becomes pid 1. It is s6-svscan that is running as pid 1 for most of the time the machine is running and fills the role of process supervisor for all long lived daemons. s6-svscan initially only starts a few daemons that are the bare essentials required for ensuring no logs are lost, an emergency tty is available, and bootstapping a service manager.

The other item to note is that s6-linux-init also spawns a child that waits until the catch-all logger is running then execs into basedir/scripts/rc.init which is a script that is created by the administrator and typically just initializes the service manager and tells the service manager to transition the system to the default state (i.e., ensure default set of services the administrator wants up/running are brought up along with all of their dependencies).

Mechanism vs Policy

All of the s6 components generally attempt to provide mechanism but not policy . However, with the s6-linux-init component some policy has crept in probably for the sake of ease of use. For example, it dictates a particular organization of the basedir and the relative location and name of the rc.init it’s child execs into. In theory, that could be an administrator defined script that could be specified in a command line parameter of s6-linux-init. Likewise, the env directory it reads could be specified by a comand line parameter rather than assuming basedir/env same goes for the run-image directory.

s6-linux-init provides a program called s6-linux-init-maker that makes it relatively easy to setup a basedir with all required scripts and essential early service definitions setup. It is by necessity, opinionated and does dictate policy. Personally, I think the s6-linux-init would be better if it eliminated the concept of basedir and allowed configuring each of its required scripts and directories individually. Then, policy could be isolated to the s6-linux-init-maker program.

s6-linux-init: A Demo

Ok, enough generalized description, lets get into a concrete example. My goal with the example is to provide a barebones demonstration of s6-linux-init bringing up a s6-svscan based process supervision tree.

First, create a chroot environment per this post. I’m going to call mine s6-linux-init-demo. Briefly:

mkdir -pv ~/chroots
cd ~/chroots
wget https://dl-cdn.alpinelinux.org/alpine/v3.13/releases/x86_64/alpine-minirootfs-3.13.1-x86_64.tar.gz
mkdir s6-linux-init-demo
tar -C s6-linux-init-demo -xf alpine-minirootfs-*.tar.gz
wget https://dash9.dev/posts/chroot-1/start-chroot.sh
chmod +x ./start-chroot.sh
sudo ./start-chroot.sh ./s6-linux-init-demo

We are now in a chroot jail that we can use to build our root filesystem that we will boot.

Install required packages:

apk add gcc make musl-dev linux-headers e2fsprogs

Create a raw file that we will use as our disk in qemu and mount it so we can populate it. Using dd with seek like this creates a sparse file meaning it looks like its huge but only takes up as much disk space as we write to it. We are creating a couple of scripts to mount and unmount for convenience.

cd
dd if=/dev/null of=/root/rootfs.img bs=1M seek=512
mkfs.ext4 -F /root/rootfs.img

cat << EOF > /root/mount.sh
mount -t ext4 -o loop /root/rootfs.img /mnt
EOF
chmod +x /root/mount.sh

./mount.sh
rm -rv /mnt/lost+found  # created by mkfs.ext4, don't need

cat << EOF > /root/umount.sh
umount /mnt
EOF
chmod +x /root/umount.sh

Build the s6-linux-init and its prerequisites. We install them in the chroot jail so we can use them to build other packages and we install all except skalibs in our image mounted at /mnt/. We are building everything as statically linked so we don’t need any shared libraries in our final image.

skalibs:

cd
wget https://skarnet.org/software/skalibs/skalibs-2.10.0.1.tar.gz
tar -xvf skalibs-*.tar.gz
cd skalibs-*
./configure --disable-shared
make
make install

execline:

cd
wget https://skarnet.org/software/execline/execline-2.7.0.1.tar.gz
tar -xvf execline-*.tar.gz
cd execline-*
./configure --enable-static-libc
make
make install
make install DESTDIR=/mnt

s6:

cd
wget https://skarnet.org/software/s6/s6-2.10.0.1.tar.gz
tar -xvf s6-*.tar.gz
cd s6-*
./configure --enable-static-libc
make
make install
make install DESTDIR=/mnt

s6-linux-init:

cd
wget https://skarnet.org/software/s6-linux-init/s6-linux-init-1.0.6.0.tar.gz
tar -xvf s6-linux-init-*.tar.gz
cd s6-linux-init-*
./configure --enable-static-libc
make
make install
make install DESTDIR=/mnt

Use s6-linux-init-maker to generate service directories and scripts required by s6-linux-init:

s6-linux-init-maker -1 -G "/sbin/getty 38400 ttyS0" /mnt/etc/s6-linux-init/current
mkdir /mnt/sbin/
cp /mnt/etc/s6-linux-init/current/bin/* /mnt/sbin/

The s6-linux-init-maker created a directory like this:

.
├── bin
│   ├── halt
│   ├── init
│   ├── poweroff
│   ├── reboot
│   ├── shutdown
│   └── telinit
├── env
├── run-image
│   ├── service
│   │   ├── .s6-svscan
│   │   │   ├── SIGINT
│   │   │   ├── SIGPWR
│   │   │   ├── SIGQUIT
│   │   │   ├── SIGTERM
│   │   │   ├── SIGUSR1
│   │   │   ├── SIGUSR2
│   │   │   ├── SIGWINCH
│   │   │   ├── crash
│   │   │   └── finish
│   │   ├── s6-linux-init-early-getty
│   │   │   └── run
│   │   ├── s6-linux-init-runleveld
│   │   │   ├── notification-fd
│   │   │   └── run
│   │   ├── s6-linux-init-shutdownd
│   │   │   ├── fifo
│   │   │   └── run
│   │   └── s6-svscan-log
│   │       ├── fifo
│   │       ├── notification-fd
│   │       └── run
│   └── uncaught-logs
└── scripts
    ├── rc.init
    ├── rc.shutdown
    ├── rc.shutdown.final
    └── runlevel

To alow us to actually do something in our minimal demo we’ll download a static build of busybox and setup a few symlinks to it to let us login and look around.

cd /mnt/bin
wget https://busybox.net/downloads/binaries/1.30.0-i686/busybox
chmod +x busybox
ln -s /bin/busybox /mnt/bin/sh
ln -s /bin/busybox /mnt/bin/ls
ln -s /bin/busybox /mnt/bin/login
ln -s /bin/busybox /mnt/bin/cat
ln -s /bin/busybox /mnt/sbin/getty

Create files required to login. Note, this is setup for a passwordless login for root. Not exactly secure.

cat << EOF > /mnt/etc/passwd
root::0:0:root:/:/bin/sh
EOF

cat << EOF > /mnt/etc/group
root:x:0:root
EOF

Create a couple of directories we need to boot:

mkdir /mnt/run
mkdir /mnt/dev

We are done creating our root filesystem image, so we can unmount it and exit the chroot jail.

cd
umount /mnt

We are missing one critical component… a linux kernel. So lets build one!

Install some prerequisites for building the linux kernel.

apk add build-base ncurses-dev bison flex elfutils-dev openssl-dev xz elfutils diffutils perl

Download, configure and build the kernel.

cd
wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.16.tar.xz
tar xf linux-*.tar.xz
cd linux-*
make mrproper
make defconfig
make kvm_guest.config
make bzImage -j`nproc`

This results in a compressed kernel image at linux-*/arch/x86/boot/bzImage. Now we point qemu at this kernel boot our root filesystem image.

apk add qemu-system-x86_64
cd
cat << EOF > qemu.sh
#!/bin/sh
qemu-system-x86_64 -kernel /root/linux-5.10.16/arch/x86/boot/bzImage \
  -drive format=raw,file=rootfs.img \
  -nographic \
  -enable-kvm \
  -append "console=ttyS0 root=/dev/sda quiet"
EOF
chmod +x qemu.sh
./qemu.sh

If all went well, we should be at a login prompt. Type root for the username and we are now at a command prompt.

To exit qemu, either type poweroff then press enter, or to force qemu to quit press crtrl-a then x. When using poweroff there will be a s6-linux-init-umountall: fatal: unable to open /proc/mounts: No such file or directory error because we did not mount the /proc virtual file system.

s6-rc

s6-rc is the service manager component in the s6 toolset. It is not a long running process itself, but relies on s6-svscan to supervise all of its services that are long running processes.

s6-rc services are configured using a simple “source” format. The source format (described here) consists of a directory with a subdirectory for each service. Each service directory consists of a collection of files. Some files are mandatory, some are optional. Each file is either a script or a file with either a single value or one value per line. s6 tools typically avoid parsing so the configuration files are kept very simple.

The source format is not used on the live system. It must be compiled to a compiled service database first using s6-rc-compile. This avoids parsing at runtime completely and ensures the source directory is valid before using it.

This has been an extremely terse overview of s6-rc. For a more thourough understanding of it, please read the official docs .

s6-rc: A Demo

To build upon the s6-linux-init demo, we are going to add s6-rc. The default s6-linux-init sripts basedir/scripts/rc.init, basedir/scripts/runlevel, and basedir/scripts/rc.shutdown have commented out hooks for initializing s6-rc, calling s6-rc to change states, and calling s6-rc to stop all services respectively. So all we have to do is:

Install s6-rc.
Create a source directory.
Compile the source directory.
Update the rc.init, runlevel, and rc.shutdown scripts to use s6-rc.

I’m assuming we are still in our chroot jail. First install s6-rc into our rootfs.img.

mount -t ext4 -o loop /root/rootfs.img /mnt
cd
wget https://skarnet.org/software/s6-rc/s6-rc-0.5.2.1.tar.gz
tar -xvf s6-rc-*.tar.gz
cd s6-rc-*
./configure --enable-static-libc
make
make install
make install DESTDIR=/mnt

Create some links to some tools we need.

ln -sv /bin/busybox /mnt/bin/echo
ln -sv /bin/busybox /mnt/bin/sleep

Create a longrun service.

mkdir -pv /mnt/etc/s6-rc/source/demo-longrun-1
echo "longrun" > /mnt/etc/s6-rc/source/demo-longrun-1/type

cat << EOF > /mnt/etc/s6-rc/source/demo-longrun-1/run
#!/bin/sh
while :; do echo "demo-longrun-1 is running"; sleep 10; done
EOF

chmod +x /mnt/etc/s6-rc/source/demo-longrun-1/run

Create a oneshot service.

mkdir -pv /mnt/etc/s6-rc/source/demo-oneshot-1
echo "oneshot" > /mnt/etc/s6-rc/source/demo-oneshot-1/type

cat << EOF > /mnt/etc/s6-rc/source/demo-oneshot-1/up
#!/bin/sh
echo "demo-oneshot-1 is up"
EOF

chmod +x /mnt/etc/s6-rc/source/demo-oneshot-1/up

cat << EOF > /mnt/etc/s6-rc/source/demo-oneshot-1/down
#!/bin/sh
echo "demo-oneshot-1 is down"
EOF

chmod +x /mnt/etc/s6-rc/source/demo-oneshot-1/down

Make the longrun service dependent upon the oneshot.

echo "demo-oneshot-1" > /mnt/etc/s6-rc/source/demo-longrun-1/dependencies

Create a default bundle and add our longrun service to it. By default, s6-rc looks for a service or bundle named default to start. This name can be overridden either when running s6-linux-init-maker or by editing /sbin/init and adding a -D option with the desired bundle or service name to the s6-linux-init command line.

mkdir -pv /mnt/etc/s6-rc/source/default
echo "bundle" > /mnt/etc/s6-rc/source/default/type
echo "demo-longrun-1" >> /mnt/etc/s6-rc/source/default/contents

Compile the database into a versioned directory and then link the versioned directory to the compiled directory.

stamp=`date -Iseconds`
s6-rc-compile /mnt/etc/s6-rc/compiled-$stamp /mnt/etc/s6-rc/source
ln -s compiled-$stamp /mnt/etc/s6-rc/compiled

Update the rc.init and runlevel scripts to use s6-rc.

cat << "EOF" >> /mnt/etc/s6-linux-init/current/scripts/rc.init
s6-rc-init /run/service
exec /etc/s6-linux-init/current/scripts/runlevel "$rl"
EOF

cat << "EOF" >> /mnt/etc/s6-linux-init/current/scripts/runlevel
exec s6-rc -v2 -up change "$1"
EOF

cat << "EOF" >> /mnt/etc/s6-linux-init/current/scripts/rc.shutdown
exec s6-rc -v2 -bDa change
EOF

cd
./umount.sh
./qemu.sh

If all went well, we should have a login prompt just like before, but we should have some log messages from s6-rc:

.
s6-linux-init version 1.0.6.0

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service demo-oneshot-1: starting
demo-oneshot-1 is up
s6-rc: info: service demo-oneshot-1 successfully started
s6-rc: info: service demo-longrun-1: starting
s6-rc: info: service demo-longrun-1 successfully started
demo-longrun-1 is running

(none) login:

Then every 10 seconds it should print demo-longrun-1 is running. When we login as root then poweroff by typing poweroff and pressing enter we should see additional logs showing our services shutting down:

~ # s6-rc: info: service demo-longrun-1: stopping
s6-rc: info: service demo-longrun-1 successfully stopped
s6-rc: info: service demo-oneshot-1: stopping
demo-oneshot-1 is down
s6-rc: info: service demo-oneshot-1 successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
s6-linux-init-shutdownd: info: sending all processes the TERM signal...
s6-linux-init-shutdownd: info: sending all processes the KILL signal...
s6-linux-init-umountall: fatal: unable to open /proc/mounts: No such file or directory
[   21.730251] reboot: Power down

To exit the chroot, just type exit and press enter.

Wrap Up

I really enjoyed studying the the s6 init system. It is apparent that Laurent Bercot put a lot of time and thought into all aspects of its design, and the result is an init system that just feels correct.

Although we’ve just touched the surface of the s6 system’s capabilities, we’ve successfully built a small linux system with reliable process monitoring and service managment. I hope this whets your appetite to learn more and continue exploring.