Setting up s6 on Gentoo

S. Gilles

2017-01-22

This is a recap of the steps I took to replace OpenRC on one of my Gentoo
boxes with s6, while I still remember how things went.  The system now

 o has a crummy ARM processor,
 o runs off eMMC (speeds comparable to a soldered in SD card)
 o uses s6 for init,
 o uses s6 for process supervision,
 o does not use OpenRC at all,
 o uses udev (eudev, to be precise),
 o doesn't use /etc/inittab,
 o takes ~2s to transition from bootloader to login prompt,
 o prints nothing for eix -I openrc sysvinit,
 o has no problems (portage-wise) with this.

I consider that boot time impressive enough that I'll want to do it on
my future systems, and this document is intended to help me recall how
to do that later.  It may be also of interest to others.  Much of this
document could be summarized as “I followed the documentation skarnet
wrote, and it did what he said it would do.”.  Nonetheless:

What is s6?

s6 is an init system, or a framework for init systems, or a collection
of small, quality programs, depending how you look at it.  Reading [0] is
probably the best way to decide for yourself.  It follows the daemontools
tradition, which is a good thing.

In principle, I have no problem with SysV-style init, but the results
tend to be either dodgy (synchronization by way of sleep(1)) or slow (my
desktop machine won't let me log in until ntpd and nfsd both start). s6's
selling point to me is that I don't have to choose between those two. The
downside is that, since it's a niche system, you have to write all your
daemon initialization files by hand.

The second downside to using s6 is the excessive choice involved.
The initial effort of s6 was to write a library from which init systems
could be created.  As a result, there are a lot of questions when setting
up s6 on a system, the answer to almost all of which is “Sure, if you
want”. My choices are focused on a rather totalitarian s6 system.

Dependencies

The packages

 sys-apps/s6
 sys-apps/s6-linux-init
 sys-apps/s6-portable-utils
 sys-apps/s6-rc

in portage should be sufficient.  I added virtual/s6-suite to my system
to bundle them because it makes me feel better. The kernel also needs
DEVTMPFS_MOUNT set.

PID 1

It's probably a good idea to read [1], the terminology of which I'll
follow. As a note, I also tried suckless' init[2], which worked fine.
In the end, I wanted s6-svscan running anyway, so I decided to cut out
a middleman.

As root, I ran (the -g and -G options seem to have switched at some point)

    s6-linux-init-maker -t 3 -g "/sbin/agetty 38400 tty1" foo
    mkdir /etc/s6-linux-init
    cp -a foo/* /etc/s6-linux-init
    ln -s /etc/s6-linux-init/init /sbin/init.s6

None of this destroyed my existing init system.  I then looked inside
/etc/s6-linux-init and read it until I was reasonably sure how it a
ll worked (this is a rather killer feature for me). The script is,
in total:

    #!/bin/execlineb -P
    /bin/export PATH "/usr/bin:/usr/sbin:/bin:/sbin"
    /bin/cd /
    s6-setsid -qb --
    umask 022
    if { s6-mount -nwt tmpfs -o mode=0755 tmpfs /run }
    if { s6-hiercopy /etc/s6-linux-init/run-image /run }
    emptyenv -p
    s6-envdir -I -- "/etc/s6-linux-init/env"
    redirfd -r 0 /dev/null
    redirfd -wnb 1 /run/service/s6-svscan-log/fifo
    background
    {
      s6-setsid --
      redirfd -w 1 /run/service/s6-svscan-log/fifo
      fdmove -c 1 2
      /etc/rc.init
    }
    unexport !
    cd /run/service
    fdmove -c 2 1
    s6-svscan -s -t 0

That's actually everything for stage 1.  I believe at this step I could
have edited my kernel command line to load /sbin/init.s6 and gotten a
fallback shell, but the depthcharge bootloader doesn't make that trivial,
so I don't remember doing so.

Importantly, this also made /etc/s6-linux-init/run-image, which gets
copied to a tmpfs at /run on boot. The daemon-controlling bits of s6
will also use this directory, as will the fallback logger (that's the
bit where init.s6 redirects stdout and stderr to a fifo before starting
anything that could potentially spam errors).

rc.init

The next part, setting up Stage 2 of the system, was writing /etc/rc.init
and /etc/rc.tini (the first and last parts of the system that don't run
as PID 1). This was the most challenging part, full of questions of “Do
I need this magic, or is it the job of something else?”). I ended up
modifying the example stage2 init (examples/ROOT/etc/s6-init/init-stage2
from the s6 repository[3]), with some added components from morpheus'
init[4] (which is intended to work with suckless init). I later found [5],
which might have proven helpful as well. The final file is as follows:

    #!/bin/execlineb -P
    if -nt
    {
      if { mount -n -t proc -o nosuid,noexec,nodev proc /proc }
      if { mount -n -t sysfs -o nosuid,noexec,nodev sysfs /sys }
      if { mkdir -p /dev/pts }
      if { mount -n -t devpts -o gid=5,mode=0620 devpts /dev/pts }
      if { mkdir -p /dev/shm }
      if { mount -n -t tmpfs -o nosuid,noexec,nodev,mode=1777 shm /dev/shm }
      if { ln -sf /proc/mounts /etc/mtab }
      if { ip addr add 127.0.0.1/8 dev lo broadcast + scope host }
      if { ip route add 127.0.0.0/8 dev lo scope host }
      if { ip link set lo up }
      if { s6-rc-init /run/service }
      if { s6-rc change normal-startup }
    }
    redirfd -w 1 /dev/console
    s6-echo "\ninit failed; dropping to fallback shell\n"

This could be shorter: most of the magic could go into startup scripts of
the init system proper. My motivation for including it here is partially
to meet some expectations of udev that I still am not 100% sure of (one
day I may switch to mdev or sdev in order to resolve these lingering
doubts), and partially to give myself a list of implicit assumptions
for daemon startup later.

The counterpart, /etc/rc.tini, is honestly the part of the system
I'm least comfortable with, since the killalls and halts (and the
Linux-specific magic they hide) are abstracted a bit far away from
my eyes.  However, here it is:

    #!/bin/execlineb -P
    s6-rc -da change

This still isn't a fully usable system (this ‘s6-rc’ thing is still
not fully set up), but it's getting there.  I could have booted to this,
logged in on the fallback shell, started udev/ntpd/sshd/etc. by hand,
and gotten some work done.  I'd probably have to hard-kill the machine,
though.

Usable daemons

Now comes the fun/tedious part: figuring out a dependency tree for daemons
like sshd and figuring out how to make binaries like sshd play nicely
with s6-rc.  I disovered at least one bug with upstream daemons this way,
but was able to work around it.

s6-rc is inspired by daemontools[6], like runit[7], daemontools-encore[8],
and perp[9]. Setups for daemons written for any one of these systems can
probably be adapted to the others without too much work.  A very useful
resource for me was Avery Payne's supervision-scripts[10] repository. I
didn't end up using anything directly, as he was missing a few of the
daemons I wanted and the myriads of symlinks were intimidating, but
seeing the proper arguments for e.g. ntpd was immensely helpful.

Logging was slightly challenging. I wanted an unprivileged user to be
responsible for writing world-readable (or at least wheel-readable)
logs, and I wanted those logs to be in /var/log where I'm used to them.
Ultimately, I created a ‘syslog’ user (out of fear of over-using
‘nobody’) and gave them ownership of /var/log/.  An example daemon
is ntpd:

    % pwd
    /etc/s6-rc/source
    % find ntpd -type f -exec sh -c "printf '===%s===\n' '{}'; cat '{}'; printf '\n'" \;
    ===ntpd/producer-for===
    ntpd-log
    
    ===ntpd/pipeline-name===
    ntpd-pipeline
    
    ===ntpd/dependencies===
    dhcpcd
    
    ===ntpd/type===
    longrun
    
    ===ntpd/run===
    #!/bin/execlineb -P
    fdmove -c 2 1
    exec -c
    /usr/sbin/ntpd -d
    
    % find ntpd-log -type f -exec sh -c "printf '===%s===\n' '{}'; cat '{}'; printf '\n'" \;
    ===ntpd-log/pipeline-name===
    ntpd-pipeline
    
    ===ntpd-log/dependencies===
    fsck
    
    ===ntpd-log/consumer-for===
    ntpd
    
    ===ntpd-log/type===
    longrun
    
    ===ntpd-log/run===
    #!/bin/execlineb -P
    s6-setuidgid syslog
    foreground { mkdir -p /var/log/ntpd }
    exec -c
    s6-log t s1000000 n20 /var/log/ntpd

This spits out logs to /var/log/ntpd/current, with automatic rollover
and all that.

The ‘fsck’ target is one which I'm not sure I handled correctly
(expecially given the filesystem-specific magic in /etc/rc.init, some
of which was added so that I could have this target). It's a oneshot
with the up script:

    #!/bin/execlineb -P
    exec -c
    redirfd -a 2 /dev/null
    redirfd -a 1 /dev/null
    foreground { mount -o remount,ro / }
    foreground { ifelse { fsck -p -A -T } { true } true }
    foreground { mount -o remount,rw / }
    foreground { mount -a }

I've never had any FS problems so far, which is good, but it means this
hasn't been tested in an error condition, and I'm unsure of the sanity.

I won't dump my entire /etc/s6-rc/source, but it contains targets which

 o set the hostname
 o chgrp video /dev/fb0 and chmod 660 /dev/fb0 so that my user can use
   the framebuffer (I would probably do audio ownership of various
   devices if I needed sound from this machine)
 o start gettys on tty2 and tty3, with prettier fonts
 o load kernel modules which don't trust to be loaded automagically
 o seed /dev/urandom from /etc/random-seed just like sysvinit
 o start dhcpcd with a hook to start wpa_supplicant

This doesn't get compiled automatically, and adding a service definition
at runtime requires two or three more commands than I'd prefer (as it
may completely change the dependency graph), but I haven't had to modify
this in a long time.

Getting rid of the old stuff

Once the system doesn't need OpenRC, or sysvinit, those packages should
be removable.  I don't mind having them around, but I don't really like
having packages I don't need, especially when the hard drive is tiny
(16GiB).

The problem is that the default Gentoo profiles mark these packages
as required, and the solution is that Gentoo allows creating custom
profiles. That's definitely “Here There Be Dragons” territory,
but that line was crossed long ago.

Portage determines profile by the symlink /etc/portage/make.profile,
which is usually a symlink into /usr/portage/profiles/ somewhere.
This system was originally a bit of an oddball, since the profile
was /usr/portage/profiles/hardened/linux/musl/arm/armv7a. To make
a custom profile, I created a new profile directory, which I put at
/usr/local/portage/profiles/hardened/linux/musl/arm/armv7a/s6 .

There are only three files I needed in this directory:

    % find /etc/portage/make.profile/ -type f -exec sh -c "printf '===%s===\n' '{}'; cat '{}'; printf '\n'" \;
    ===/etc/portage/make.profile/parent===
    /usr/portage/profiles/hardened/linux/musl/arm/armv7a
    
    ===/etc/portage/make.profile/packages===
    *virtual/s6-suite
    -*sys-apps/openrc
    -*sys-apps/sysvinit
    
    ===/etc/portage/make.profile/eapi===
    6

Importantly, sysvinit contains other programs, like halt and reboot. Not
having these programs available means I don't mistype them when I should
be using ‘s6-svscanctl’.

[0] http://scarnet.org/software/s6
[1] http://skarnet.org/software/s6-linux-init/s6-linux-init-maker.html
[2] http://core.suckless.org/sinit
[3] http://git.skarnet.org/cgi-bin/cgit.cgi/s6
[4] http://git.2f30.org/ports/file/fs/bin/rc.init.html
[5] http://troubleshooters.com/linux/diy/suckless_init_on_plop.htm
[6] http://cr.yp.to/daemontools.html
[7] http://smarden.org/runit/
[8] http://untroubled.org/daemontools-encore/
[9] http://b0llix.net/perp/
[10] https://bitbucket.org/avery_payne/supervision-scripts