Building a skarnet.org s6 Init System
Per skarnet.org/software/ s6 is “skarnet.org’s small and secure supervision software suite. Comes with an ultra-fast init replacement, process management tools, an asynchronous locking library, and more.” After a little bit of reading and a lot of tinkering, I was able to assemble an s6 based system and learned a lot about linux userspace initialization in the process.
s6 is the core of an s6 based init system, but it is only one of several components that must be assembled to build a complete system. Per Laurent Bercot, the creator of the s6 software toolset, a “complete” system consists of the following components (ref: The s6 supervision suite A modern alternative to systemd):
- /sbin/init: The program that the linux kernel executes to start up userspace. (Actually, /sbin/init is the first of several defaults that the linux kernel tries to execute, and another init program may be specified on the kernel command line.)
- pid 1: The process that is the root of the linux userspace and never terminates. Initially this is /sbin/init, but /sbin/init can exec into another program which will then be pid 1.
- Process Supervisor: This component monitors a set of long running processes (daemons) and ensures that if they ever terminate, they are restarted in a well defined environment.
- Service Manager: This component, brings services up and down when required at boot time, shutdown or any other time the state of userspace needs to change. In the s6 world, services can be either long running daemons, or short lived programs with side effects (e.g, mounting: the service is up when the mount point is mounted and down when it is unmounted). A service manager also has knowledge of the dependency relationships between all of the services (if specified by the administrator) and ensures services are brought up or down in the correct sequence during a state change.
In an s6 based linux init system, component 1 is provided by s6-linux-init which is a part of the s6-linux-init package. This is the non-portable component in the system that is specific to the linux kernel. It performs some linux specific initializations and execs into the s6-svscan program…
Component 2 and 3: pid 1 and the process supervisor, are provided by s6-svscan which is a part of the s6 package. s6-svscan runs from the time /sbin/init execs into it during boot up until it terminates at shutdown. s6-svscan is the root of the process supervision tree.
Component 4, the service manager, is provided by s6-rc. s6-rc can start and stop services and can change the state of the machine (set of running/up services). It delegates the job of supervising its long running service processes to the service supervisor, s6-svscan.
Below is an outline of what s6-linux-init does taken directly from here.
It prints a banner to /dev/console.
It chdirs to /.
It sets the umask to initial_umask.
It becomes a session leader.
It mounts a devtmpfs on slashdev, if requested.
It uses /dev/null as its stdin (instead of /dev/console). /dev/console is still used, for now, as stdout and stderr.
It unmounts /run (or the directory you have given to the tmpfsdir configure option at package build time), just in case; then it creates a tmpfs on it. Alternatively, it remounts /run, or does not touch it at all.
It copies the whole basedir/run-image hierarchy to /run (or your chosen tmpfsdir).
It reads the initial environment from basedir/env.
If required, it stores the kernel environment into env_store.
It performs “the fifo trick”, i.e. it redirects its stdout to the catch-all logger’s fifo, without blocking, before the catch-all logger is even up (because it’s a service that will be spawned a bit later, when s6-svscan is executed).
It forks a child.
- The child scans the kernel command line to find a suitable runlevel (default, 2, 3, 4, or 5). If it doesn’t find any kernel command line argument that defines a runlevel, it uses initdefault.
- The child becomes a session leader.
- The child blocks until the catch-all logger runs.
It also makes the catch-all logger’s fifo its stderr.
It traps the ctrl-alt-del keyboard combination.
It execs into s6-svscan with /run/service as its scandir (or tmpfsdir/service).
s6-svscan spawns the early services that are defined in basedir/run-image/service, and have been copied into /run/service (or tmpfsdir/service).
One of those early services is s6-svscan-log, which is the catch-all logger. When this service is up, s6-linux-init’s child unblocks.
The child execs into basedir/scripts/rc.init. The first argument to rc.init is the chosen runlevel. The kernel command line, as given by the kernel to s6-linux-init (i.e. without the key=value arguments, which were passed into s6-linux-init’s environment and were stored into env_store), makes for the rest of the arguments given to rc.init.
As you can see, the actual s6-linux-init program is not a long lived daemon. It execs into s6-svscan which then becomes pid 1. It is s6-svscan that is running as pid 1 for most of the time the machine is running and fills the role of process supervisor for all long lived daemons. s6-svscan initially only starts a few daemons that are the bare essentials required for ensuring no logs are lost, an emergency tty is available, and bootstapping a service manager.
The other item to note is that s6-linux-init also spawns a child that waits until the catch-all logger is running then execs into basedir/scripts/rc.init which is a script that is created by the administrator and typically just initializes the service manager and tells the service manager to transition the system to the default state (i.e., ensure default set of services the administrator wants up/running are brought up along with all of their dependencies).
Mechanism vs Policy
All of the s6 components generally attempt to provide mechanism but not policy . However, with the s6-linux-init component some policy has crept in probably for the sake of ease of use. For example, it dictates a particular organization of the basedir and the relative location and name of the rc.init it’s child execs into. In theory, that could be an administrator defined script that could be specified in a command line parameter of s6-linux-init. Likewise, the env directory it reads could be specified by a comand line parameter rather than assuming basedir/env same goes for the run-image directory.
s6-linux-init provides a program called s6-linux-init-maker that makes it relatively easy to setup a basedir with all required scripts and essential early service definitions setup. It is by necessity, opinionated and does dictate policy. Personally, I think the s6-linux-init would be better if it eliminated the concept of basedir and allowed configuring each of its required scripts and directories individually. Then, policy could be isolated to the s6-linux-init-maker program.
s6-linux-init: A Demo
Ok, enough generalized description, lets get into a concrete example. My goal with the example is to provide a barebones demonstration of s6-linux-init bringing up a s6-svscan based process supervision tree.
First, create a chroot environment per this post. I’m going to call mine s6-linux-init-demo. Briefly:
mkdir -pv ~/chroots cd ~/chroots wget https://dl-cdn.alpinelinux.org/alpine/v3.13/releases/x86_64/alpine-minirootfs-3.13.1-x86_64.tar.gz mkdir s6-linux-init-demo tar -C s6-linux-init-demo -xf alpine-minirootfs-*.tar.gz wget https://dash9.dev/posts/chroot-1/start-chroot.sh chmod +x ./start-chroot.sh sudo ./start-chroot.sh ./s6-linux-init-demo
We are now in a chroot jail that we can use to build our root filesystem that we will boot.
Install required packages:
apk add gcc make musl-dev linux-headers e2fsprogs
Create a raw file that we will use as our disk in qemu and mount it so we can populate it. Using dd with seek like this creates a sparse file meaning it looks like its huge but only takes up as much disk space as we write to it. We are creating a couple of scripts to mount and unmount for convenience.
cd dd if=/dev/null of=/root/rootfs.img bs=1M seek=512 mkfs.ext4 -F /root/rootfs.img cat << EOF > /root/mount.sh mount -t ext4 -o loop /root/rootfs.img /mnt EOF chmod +x /root/mount.sh ./mount.sh rm -rv /mnt/lost+found # created by mkfs.ext4, don't need cat << EOF > /root/umount.sh umount /mnt EOF chmod +x /root/umount.sh
Build the s6-linux-init and its prerequisites. We install them in the chroot jail so we can use them to build other packages and we install all except skalibs in our image mounted at /mnt/. We are building everything as statically linked so we don’t need any shared libraries in our final image.
cd wget https://skarnet.org/software/skalibs/skalibs-18.104.22.168.tar.gz tar -xvf skalibs-*.tar.gz cd skalibs-* ./configure --disable-shared make make install
cd wget https://skarnet.org/software/execline/execline-22.214.171.124.tar.gz tar -xvf execline-*.tar.gz cd execline-* ./configure --enable-static-libc make make install make install DESTDIR=/mnt
cd wget https://skarnet.org/software/s6/s6-126.96.36.199.tar.gz tar -xvf s6-*.tar.gz cd s6-* ./configure --enable-static-libc make make install make install DESTDIR=/mnt
cd wget https://skarnet.org/software/s6-linux-init/s6-linux-init-188.8.131.52.tar.gz tar -xvf s6-linux-init-*.tar.gz cd s6-linux-init-* ./configure --enable-static-libc make make install make install DESTDIR=/mnt
Use s6-linux-init-maker to generate service directories and scripts required by s6-linux-init:
s6-linux-init-maker -1 -G "/sbin/getty 38400 ttyS0" /mnt/etc/s6-linux-init/current mkdir /mnt/sbin/ cp /mnt/etc/s6-linux-init/current/bin/* /mnt/sbin/
The s6-linux-init-maker created a directory like this:
. ├── bin │ ├── halt │ ├── init │ ├── poweroff │ ├── reboot │ ├── shutdown │ └── telinit ├── env ├── run-image │ ├── service │ │ ├── .s6-svscan │ │ │ ├── SIGINT │ │ │ ├── SIGPWR │ │ │ ├── SIGQUIT │ │ │ ├── SIGTERM │ │ │ ├── SIGUSR1 │ │ │ ├── SIGUSR2 │ │ │ ├── SIGWINCH │ │ │ ├── crash │ │ │ └── finish │ │ ├── s6-linux-init-early-getty │ │ │ └── run │ │ ├── s6-linux-init-runleveld │ │ │ ├── notification-fd │ │ │ └── run │ │ ├── s6-linux-init-shutdownd │ │ │ ├── fifo │ │ │ └── run │ │ └── s6-svscan-log │ │ ├── fifo │ │ ├── notification-fd │ │ └── run │ └── uncaught-logs └── scripts ├── rc.init ├── rc.shutdown ├── rc.shutdown.final └── runlevel
To alow us to actually do something in our minimal demo we’ll download a static build of busybox and setup a few symlinks to it to let us login and look around.
cd /mnt/bin wget https://busybox.net/downloads/binaries/1.30.0-i686/busybox chmod +x busybox ln -s /bin/busybox /mnt/bin/sh ln -s /bin/busybox /mnt/bin/ls ln -s /bin/busybox /mnt/bin/login ln -s /bin/busybox /mnt/bin/cat ln -s /bin/busybox /mnt/sbin/getty
Create files required to login. Note, this is setup for a passwordless login for root. Not exactly secure.
cat << EOF > /mnt/etc/passwd root::0:0:root:/:/bin/sh EOF cat << EOF > /mnt/etc/group root:x:0:root EOF
Create a couple of directories we need to boot:
mkdir /mnt/run mkdir /mnt/dev
We are done creating our root filesystem image, so we can unmount it and exit the chroot jail.
cd umount /mnt
We are missing one critical component… a linux kernel. So lets build one!
Install some prerequisites for building the linux kernel.
apk add build-base ncurses-dev bison flex elfutils-dev openssl-dev xz elfutils diffutils perl
Download, configure and build the kernel.
cd wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.16.tar.xz tar xf linux-*.tar.xz cd linux-* make mrproper make defconfig make kvm_guest.config make bzImage -j`nproc`
This results in a compressed kernel image at linux-*/arch/x86/boot/bzImage. Now we point qemu at this kernel boot our root filesystem image.
apk add qemu-system-x86_64 cd cat << EOF > qemu.sh #!/bin/sh qemu-system-x86_64 -kernel /root/linux-5.10.16/arch/x86/boot/bzImage \ -drive format=raw,file=rootfs.img \ -nographic \ -enable-kvm \ -append "console=ttyS0 root=/dev/sda quiet" EOF chmod +x qemu.sh ./qemu.sh
If all went well, we should be at a login prompt. Type root for the username and we are now at a command prompt.
To exit qemu, either type poweroff then press enter, or to force qemu to quit press crtrl-a then x. When using poweroff there will be a s6-linux-init-umountall: fatal: unable to open /proc/mounts: No such file or directory error because we did not mount the /proc virtual file system.
s6-rc is the service manager component in the s6 toolset. It is not a long running process itself, but relies on s6-svscan to supervise all of its services that are long running processes.
s6-rc services are configured using a simple “source” format. The source format (described here) consists of a directory with a subdirectory for each service. Each service directory consists of a collection of files. Some files are mandatory, some are optional. Each file is either a script or a file with either a single value or one value per line. s6 tools typically avoid parsing so the configuration files are kept very simple.
The source format is not used on the live system. It must be compiled to a compiled service database first using s6-rc-compile. This avoids parsing at runtime completely and ensures the source directory is valid before using it.
This has been an extremely terse overview of s6-rc. For a more thourough understanding of it, please read the official docs .
s6-rc: A Demo
To build upon the s6-linux-init demo, we are going to add s6-rc. The default s6-linux-init sripts basedir/scripts/rc.init, basedir/scripts/runlevel, and basedir/scripts/rc.shutdown have commented out hooks for initializing s6-rc, calling s6-rc to change states, and calling s6-rc to stop all services respectively. So all we have to do is:
- Install s6-rc.
- Create a source directory.
- Compile the source directory.
- Update the rc.init, runlevel, and rc.shutdown scripts to use s6-rc.
I’m assuming we are still in our chroot jail. First install s6-rc into our rootfs.img.
mount -t ext4 -o loop /root/rootfs.img /mnt cd wget https://skarnet.org/software/s6-rc/s6-rc-0.5.2.1.tar.gz tar -xvf s6-rc-*.tar.gz cd s6-rc-* ./configure --enable-static-libc make make install make install DESTDIR=/mnt
Create some links to some tools we need.
ln -sv /bin/busybox /mnt/bin/echo ln -sv /bin/busybox /mnt/bin/sleep
Create a longrun service.
mkdir -pv /mnt/etc/s6-rc/source/demo-longrun-1 echo "longrun" > /mnt/etc/s6-rc/source/demo-longrun-1/type cat << EOF > /mnt/etc/s6-rc/source/demo-longrun-1/run #!/bin/sh while :; do echo "demo-longrun-1 is running"; sleep 10; done EOF chmod +x /mnt/etc/s6-rc/source/demo-longrun-1/run
Create a oneshot service.
mkdir -pv /mnt/etc/s6-rc/source/demo-oneshot-1 echo "oneshot" > /mnt/etc/s6-rc/source/demo-oneshot-1/type cat << EOF > /mnt/etc/s6-rc/source/demo-oneshot-1/up #!/bin/sh echo "demo-oneshot-1 is up" EOF chmod +x /mnt/etc/s6-rc/source/demo-oneshot-1/up cat << EOF > /mnt/etc/s6-rc/source/demo-oneshot-1/down #!/bin/sh echo "demo-oneshot-1 is down" EOF chmod +x /mnt/etc/s6-rc/source/demo-oneshot-1/down
Make the longrun service dependent upon the oneshot.
echo "demo-oneshot-1" > /mnt/etc/s6-rc/source/demo-longrun-1/dependencies
Create a default bundle and add our longrun service to it. By default, s6-rc looks for a service or bundle named default to start. This name can be overridden either when running s6-linux-init-maker or by editing /sbin/init and adding a -D option with the desired bundle or service name to the s6-linux-init command line.
mkdir -pv /mnt/etc/s6-rc/source/default echo "bundle" > /mnt/etc/s6-rc/source/default/type echo "demo-longrun-1" >> /mnt/etc/s6-rc/source/default/contents
Compile the database into a versioned directory and then link the versioned directory to the compiled directory.
stamp=`date -Iseconds` s6-rc-compile /mnt/etc/s6-rc/compiled-$stamp /mnt/etc/s6-rc/source ln -s compiled-$stamp /mnt/etc/s6-rc/compiled
Update the rc.init and runlevel scripts to use s6-rc.
cat << "EOF" >> /mnt/etc/s6-linux-init/current/scripts/rc.init s6-rc-init /run/service exec /etc/s6-linux-init/current/scripts/runlevel "$rl" EOF cat << "EOF" >> /mnt/etc/s6-linux-init/current/scripts/runlevel exec s6-rc -v2 -up change "$1" EOF cat << "EOF" >> /mnt/etc/s6-linux-init/current/scripts/rc.shutdown exec s6-rc -v2 -bDa change EOF cd ./umount.sh ./qemu.sh
If all went well, we should have a login prompt just like before, but we should have some log messages from s6-rc:
. s6-linux-init version 184.108.40.206 s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service demo-oneshot-1: starting demo-oneshot-1 is up s6-rc: info: service demo-oneshot-1 successfully started s6-rc: info: service demo-longrun-1: starting s6-rc: info: service demo-longrun-1 successfully started demo-longrun-1 is running (none) login:
Then every 10 seconds it should print demo-longrun-1 is running. When we login as root then poweroff by typing poweroff and pressing enter we should see additional logs showing our services shutting down:
~ # s6-rc: info: service demo-longrun-1: stopping s6-rc: info: service demo-longrun-1 successfully stopped s6-rc: info: service demo-oneshot-1: stopping demo-oneshot-1 is down s6-rc: info: service demo-oneshot-1 successfully stopped s6-rc: info: service s6rc-oneshot-runner: stopping s6-rc: info: service s6rc-oneshot-runner successfully stopped s6-linux-init-shutdownd: info: sending all processes the TERM signal... s6-linux-init-shutdownd: info: sending all processes the KILL signal... s6-linux-init-umountall: fatal: unable to open /proc/mounts: No such file or directory [ 21.730251] reboot: Power down
To exit the chroot, just type exit and press enter.
I really enjoyed studying the the s6 init system. It is apparent that Laurent Bercot put a lot of time and thought into all aspects of its design, and the result is an init system that just feels correct.
Although we’ve just touched the surface of the s6 system’s capabilities, we’ve successfully built a small linux system with reliable process monitoring and service managment. I hope this whets your appetite to learn more and continue exploring.