keyboot

keyboot is a greenfield boot stage that runs after GRUB and before your OS’s init. In one boot it decrypts a LUKS keyfile inside its own initramfs, uses that key to open every LUKS device backing the root zpool, imports the pool, lets you pick a boot environment (TTY or SSH), and kexecs into that BE’s own kernel. The same artifact is also a netbootable rescue shell, an unattended installer, and a memtest launcher (selected by keyboot.mode=). It targets Debian, Gentoo, and Alpine on UEFI and BIOS.

GRUB ─▶ keyboot (unlock env) ─▶ kexec ─▶ your OS (ZFS-on-LUKS root)

Where to start

Install in one line (from a live env)

curl -fsSL https://packages.osterman.co/keyboot/install-os | sh   # musl live env (Alpine)

(For a glibc rescue, see installer-bootstrap / Hetzner deploy.)

The rest of the docs

Use the sidebar. Reference: architecture, unlock-flow, browser unlock, dataset layout, supported OSes, memtest, RAM rescue, native package repos, USB rescue, Hetzner deploy. Decisions: the ADRs record the locked design choices and the why.

SPEC.md (in the repo) is the authoritative design of record; this site is the day-2, operator-facing view.

Installing keyboot + an OS on bare metal

A step-by-step guide to take a blank bare-metal server to a keyboot-managed, ZFS-on-LUKS host running one of Debian, Gentoo, or Alpine. You pick one OS here; adding more later (multi-distro on one pool) is §6.

This is the vendor-neutral guide. For Hetzner-specific quirks (vKVM=QEMU, the rescue’s on-demand ZFS compile, BIOS-on-md) see hetzner-deploy.md. For the design, architecture.md; when it breaks, disaster-recovery.md.

What you’ll end up with

GRUB ─▶ keyboot (unlock env) ─▶ kexec ─▶ your OS (ZFS-on-LUKS root)

• Every data disk LUKS-encrypted; one passphrase unlocks all of them.
• The OS lives in a ZFS boot environment; keyboot picks + kexecs it.
• Headless-reachable: the unlock env and the OS both come up networked over SSH.

Before you start — what you need

1. A build host (x86_64, lots of cores) to build the keyboot boot image (kernel + initramfs). This is NOT the target. See “Build the boot image” below.
2. A provisioning environment on the target: a keyboot rescue USB (usb-rescue.md), netboot, or the vendor’s rescue system.
3. SSH access to that provisioning env, and your SSH public key.
4. The target disks’ device paths (/dev/sda …) or serials.
5. A passphrase for the LUKS keyfile (you’ll type it at each boot, or enroll an automation slot later — operator-guide.md §3).
6. UEFI or BIOS? Both work; it only changes the GRUB-install details (§4).

⚠️ Installing destroys the target disks. Back up anything first.

Build the boot image (on the build host)

keyboot boots its own kernel + initramfs. Build them with all NIC drivers (so the unlock env comes up networked on real hardware) and your SSH key baked in:

bash kernel/build.sh                                   # -> kernel/out/vmlinuz
INCLUDE_MODULES=all AUTHORIZED_KEYS=you.pub \
    bash ci/build-image.sh                             # -> the initramfs

Copy kernel/out/vmlinuz and the built initramfs to the target’s provisioning env (you’ll pass them to the installer as --keyboot-kernel / --keyboot-initramfs).

1. Boot the target into a provisioning environment

Boot the keyboot rescue USB / netboot, or the vendor rescue, and get an SSH shell as root. You need: the target disks visible (lsblk), network up, and enough tooling to run the installer (next step provides it).

2. Bootstrap the installer

The installer (keyboot-install) is a musl binary. How you get it depends on the provisioning env’s libc:

• musl live env (Alpine, keyboot rescue): one-liner bootstrap —

  curl -fsSL https://packages.osterman.co/keyboot/install-os | sh
  

  (fetches + minisign-verifies keyboot-install + the install-os toolkit + the be-tools). See installer-bootstrap.md.
• glibc rescue (Debian/Hetzner): the musl binary won’t link there; build it natively + stage the toolkit with the helper —

  KEYBOOT_SRC=/path/to/keyboot bash tools/hetzner-prep.sh
  

  (also recompiles ZFS for the rescue kernel; see hetzner-deploy.md).

Each bootstrap installs keyboot-install and prints the exact install-os command to copy — including the KEYBOOT_INSTALL_OS_DIR (and, for the curl|sh path, KB_BE_TOOLS_DIR) environment variables the installer needs. Run install-os the way the bootstrap tells you; don’t drop those variables. If KB_BE_TOOLS_DIR is missing when you run install-os, the install still succeeds but the new BE silently ships without the be-tools (autosnap / be-upgrade / be-rollback / snap / update-check / keys) — the GL#43 footgun.

Tip — skip the variables entirely (curl|sh): let the bootstrap run the installer for you in the same process (where the variables are already live):

curl -fsSL https://packages.osterman.co/keyboot/install-os \
  | sh -s -- --run <distro> --disk /dev/sda --hostname h --confirm   # + the §3 flags

3. Install ONE operating system

Dry-run first (omit --confirm) to see the plan; add --confirm to execute. Pick one of the following. Common flags: --disk (repeat per pool member), --hostname, --authorized-keys you.pub (so the booted OS is SSH-reachable), --passphrase-from (prompt | file:PATH | env:VAR), and the boot image you built (--keyboot-kernel / --keyboot-initramfs). Disk count picks the default pool layout (1→single, 2→mirror, 3+→raidz1); override with --template or the --topology grammar (ADR 0010).

Debian (systemd, bookworm)

keyboot-install install-os debian \
    --disk /dev/sda --disk /dev/sdb \
    --hostname my-debian --authorized-keys you.pub \
    --passphrase-from prompt \
    --keyboot-kernel /root/keyboot-vmlinuz \
    --keyboot-initramfs /root/keyboot-initramfs.img \
    --confirm

Notes: debootstraps the base, builds ZFS via zfs-dkms in the chroot, wires the re-unlock as a boot=keyboot initramfs-tools script. Pool is created compatibility=openzfs-2.1-linux so Debian’s 2.1.x ZFS can import it.

Gentoo (openrc)

keyboot-install install-os gentoo \
    --disk /dev/sda --disk /dev/sdb \
    --hostname my-gentoo --authorized-keys you.pub \
    --passphrase-from prompt \
    --keyboot-kernel /root/keyboot-vmlinuz \
    --keyboot-initramfs /root/keyboot-initramfs.img \
    --confirm

Notes: extracts a stage3, emerges gentoo-kernel-bin + zfs-kmod + a dracut 90keyboot module from the binhost. --firmware curated keeps a small server-NIC firmware set; full ships everything; none ships nothing.

Alpine (openrc)

keyboot-install install-os alpine \
    --disk /dev/sda --disk /dev/sdb \
    --hostname my-alpine --authorized-keys you.pub \
    --passphrase-from prompt \
    --keyboot-kernel /root/keyboot-vmlinuz \
    --keyboot-initramfs /root/keyboot-initramfs.img \
    --confirm

Notes: the most CI-exercised path. Curated mkinitfs BE image bundling the keyboot binary + keyfile + a be-unlock init; comes up DHCP + sshd + your key.

Declarative alternative: put any of the above in a YAML profile and run keyboot-install install-os <distro> --profile profile.yaml --confirm (SPEC §13.7; see tools/keyboot-install-os/profile.example.yaml).

What the installer does, in order: partition (GPT + mdraid1 ESP + crypt) → create the keyfile + enroll each disk → unlock as sn-<serial> → zpool create → create the BE → bootstrap the distro → configure (network/ssh/hostname) → build the BE kernel + initramfs → install GRUB + the keyboot image → snapshot → teardown. --substrate-only stops after the pool+BE (no distro).

4. First boot

Reboot the target off the installer env onto its own disks.

• UEFI: firmware reads BOOTX64.EFI from the mdraid1 ESP → GRUB → keyboot. Nothing extra needed.
• BIOS: GRUB’s core.img is installed per-disk on each ESP member (the installer does this; the array is stopped first so the member mount doesn’t EBUSY). Set the BIOS to boot the disk you installed to.

The box comes up in the keyboot unlock env, networked. SSH in (the key you baked into the boot image) and you’ll get the passphrase prompt — or, with --passphrase-from prompt, keyboot waits on both the console and SSH:

ssh root@<target>          # to the keyboot env (dropbear)
# enter the LUKS passphrase -> keyfile opens -> disks open as sn-<serial>
#   -> pool imports -> kexec -> your OS boots

After kexec the OS comes up (also networked, your key). Verify:

ssh root@<target>          # now the booted OS
zpool status               # ONLINE
cat /etc/os-release        # the distro you installed

5. Verify it’s healthy

zpool status -x            # 'all pools are healthy'
zfs list -o name,used,mountpoint -r rpool
systemctl is-active ssh    # (Debian)  / rc-status (openrc)

Add a recovery passphrase now so a forgotten daily passphrase isn’t fatal:

keyboot-install keyfile add-recovery <keyfile>     # slot 1

(Back up the keyfile container off-box — disaster-recovery.md §1.)

6. (Optional) Add more operating systems later

keyboot’s own env is the only provisioning environment you need after this. To add a second/third distro to the same pool (each its own boot environment):

• From the running OS or the keyboot provision shell, with the pool imported:

  keyboot-install install-os <distro> --add-be --pool rpool \
      --authorized-keys you.pub --confirm
  

  No repartition, no new keyfile/pool, no GRUB rewrite — keyboot auto-discovers the new BE. Reboot and pick it from the keyboot menu.
• Provision mode (boot keyboot itself into an installer shell): arm it with keyboot-install keyboot provision --confirm, reboot — keyboot unlocks + imports read-write and drops you at an install-os --add-be prompt, then reverts to normal boot. See operator-guide.md §4.

If something goes wrong

keyboot brings networking + SSH up before unlock, so most failures leave a reachable recovery shell. See disaster-recovery.md for the symptom→fix table (won’t boot, unlock fails, wrong entry, feature drift, failed disk) and hardware-test-runbook.md for the console-attended recovery model.

Installer bootstrap (curl | sh install-os) — SPEC §13.10

keyboot’s installer normally runs from keyboot’s own rescue mode (PXE/USB/ISO), where all tools are already present. When you’re instead sitting in some other live environment (Alpine live, a Debian/Ubuntu rescue shell, Gentoo/SystemRescue, a Hetzner rescue, …) and don’t want to reboot, the bootstrap sets keyboot’s installer up in place:

curl -fsSL https://packages.osterman.co/keyboot/install-os | sh
# or drive it in one shot:
curl -fsSL https://packages.osterman.co/keyboot/install-os | sh -s -- \
    --run debian --disk <serial> <serial> --hostname newhost --confirm
# or from a profile (SPEC §13.7):
curl -fsSL .../install-os | sh -s -- --run debian --profile profile.yaml --confirm

What it does (§13.10):

1. Detects the package manager (apk / apt-get / emerge / pacman / dnf) and installs keyboot’s install-time deps: cryptsetup, the ZFS userland, sgdisk/gdisk, mdadm, jq, dosfstools, minisign.
2. Fetches + minisign-verifies (against the pinned key, same as install) the keyboot-install + keyboot binaries and the keyboot-install-os.tar.gz toolkit (orchestrator + per-distro plugins).
3. Installs them and either drops you into keyboot-install install-os or, with --run, runs it directly.

Same code paths and UX as keyboot rescue mode; the only difference is that the ZFS userland came from the live env’s package manager rather than keyboot’s bundle.

Acceptable starting environments

The binding constraint is ZFS in the live env’s package manager. Known-good:

Bare-bones rescue shells with no ZFS in their package manager are not acceptable starting points — reboot into keyboot’s own rescue mode (which bundles ZFS) or install ZFS by hand first. The bootstrap fails loudly (rather than half-installing) when it can’t get the ZFS userland.

USB rescue/install image

ci/build-usb.sh turns the keyboot artifact (the initramfs from ci/build-image.sh plus its matched vmlinuz) into a self-contained, bootable raw disk image (.img). Written to a USB stick, it boots keyboot in rescue (or install) mode — the on-site / no-netboot counterpart to the netboot path (SPEC §1, §2). It is the same artifact as the netboot/boot image; the mode is chosen on the GRUB kernel cmdline.

Download the hosted image (no build needed)

A prebuilt, signed image is published on the package channel. It ships the full kernel module tree (every NIC/HBA driver) plus ethtool + lspci, so it is the quickest way to boot a box and see which network drivers bind.

base=https://packages.osterman.co/keyboot/v0.1.0
curl -fLO $base/keyboot-rescue-x86_64.img.gz
curl -fLO $base/keyboot-rescue-x86_64.img.gz.minisig

# Verify against the project signing key (same pinned key the curl|sh installer uses)
minisign -Vm keyboot-rescue-x86_64.img.gz \
  -P RWTrhJW+9h/jDQZkoGzVdqZnMvkRwjdmMeeimbc39fj/kyAg54+2J9BZ \
  -x keyboot-rescue-x86_64.img.gz.minisig

# Decompress + write to a stick (see "Write it to a stick" for the device caveat)
gunzip -c keyboot-rescue-x86_64.img.gz | sudo dd of=/dev/sdX bs=4M conv=fsync status=progress

The image boots straight into keyboot rescue mode. SSH-in works if the image was built with an authorized key baked in (the hosted one trusts the fleet keyboot_ci_deploy key); otherwise use the console.

Layout (firmware-agnostic, SPEC §13.13)

One GPT image boots on both UEFI and BIOS-MBR:

GRUB presents two menu entries: rescue (keyboot.mode=rescue, default) and unattended install (keyboot.mode=install).

One-command build (recommended)

ci/make-usb.sh is the turnkey front-end: it runs the whole kernel→initramfs→.img chain and surfaces the choices you actually make at a stick. It stops at a dd-able raw image (no gzip/sign — that’s the publish path, ci/build-usb-image.sh). Runs on an Alpine build host, as root:

sudo ci/make-usb.sh --ssh-key you.pub --mode rescue --modules all
# -> ci/qemu/out/keyboot-usb.img
ci/make-usb.sh --check --ssh-key you.pub      # preflight only (no root)

Options: --ssh-key FILE / --ssh-dir DIR (an authorized key is required), --mode rescue|install|boot|memtest, --modules all|none, --zfs/--no-zfs, --install-tools, --keyboot-bin FILE, --kernel FILE / --build-kernel, --out FILE, --size MIB, --memtest-dir DIR. It reuses a prebuilt kernel/out/vmlinuz unless --build-kernel.

Build from any distro or vendor rescue

make-usb.sh needs an Alpine host (the initramfs userland comes from apk). To build straight from a box’s own Debian/Ubuntu rescue (or any working distro), use ci/make-usb-anyhost.sh: it apk.static-bootstraps a throwaway Alpine rootfs, builds the keyboot binary in it, and runs make-usb.sh inside that chroot — same flags pass through. The host needs only root, loop devices, curl/wget, and tar, plus a checkout with a prebuilt kernel/out/vmlinuz.

sudo ci/make-usb-anyhost.sh --ssh-key you.pub --mode rescue   # from any distro/rescue

(On an Alpine host it just exec’s make-usb.sh. The image-assembly step needs a real /dev with loop devices — it won’t run in an unprivileged container with a tmpfs /dev.)

Build (manual, step by step)

Requires root, loop devices, and grub-install with both the i386-pc and x86_64-efi targets, plus sgdisk, mkfs.vfat (dosfstools), and losetup (util-linux). On Alpine — note sgdisk is its own package (the gptfdisk package ships gdisk, not sgdisk):

apk add grub grub-bios grub-efi dosfstools sgdisk util-linux

Build the inputs first (kernel + initramfs), then the image:

bash kernel/build.sh                       # -> kernel/out/vmlinuz
INCLUDE_MODULES=all INCLUDE_ZFS=yes INCLUDE_INSTALL=yes \
  AUTHORIZED_KEYS=you.pub bash ci/build-image.sh   # -> ci/qemu/out/keyboot-initramfs.cpio.gz

sudo ci/build-usb.sh                       # -> ci/qemu/out/keyboot-rescue.img

Preflight without building (checks tools + inputs, no root):

ci/build-usb.sh --check

Knobs (env): INITRAMFS, KERNEL, OUT, MODE (rescue|install), SIZE_MIB (default auto), GRUB_TIMEOUT, CONSOLE.

Write it to a stick

dd the image to the raw USB device (not a partition). This destroys everything on the target — double-check the device node.

lsblk                                       # find your stick, e.g. /dev/sdX
sudo dd if=ci/qemu/out/keyboot-rescue.img of=/dev/sdX bs=4M conv=fsync status=progress
sync

Boot the host from the stick. On UEFI it loads /EFI/BOOT/BOOTX64.EFI; on BIOS it loads GRUB from the protective MBR + the EF02 partition. Either way you land in the GRUB menu, then keyboot rescue mode.

Checking network drivers on a box

This is the headline use of the rescue stick. On boot in rescue mode keyboot runs keyboot-nic-report automatically and prints it to every console (serial + KVM video), then drops you to a shell. The report shows, per interface: name, PCI id, the bound kernel driver, link state, and speed — plus an lspci network-class listing that surfaces controllers with no driver bound (the “hardware present, kernel doesn’t drive it” case, which never shows up under /sys/class/net), and the recent NIC/driver dmesg lines.

Re-run or dig deeper from the shell:

keyboot-nic-report          # the same summary, on demand
lspci -nnk | grep -iA3 -e Ethernet -e Network   # controller + "Kernel driver in use"
ethtool enp1s0              # link, speed, driver/firmware versions
ip -br link                 # all links + up/down at a glance
dmesg | grep -iE 'eth|enp|link|firmware'        # probe/firmware errors
modprobe <driver>           # try loading a driver the autoloader missed

If a NIC shows up in lspci but has no enpXsY interface and no “Kernel driver in use”, the kernel has no driver for it — note the [vendor:device] id from lspci -nn and add/enable the matching driver in the kernel config (kernel/config-drivers-x86_64, regenerated by kernel/gen-drivers-config.sh).

Deploying keyboot to real hardware (Hetzner / generic UEFI server)

This is the field guide for installing keyboot onto a bare-metal server whose only access is a vendor rescue system + a KVM/IPMI console. It was written against Hetzner (Debian rescue + a QEMU-based “vKVM”), but the shape applies to any provider: boot a rescue, stage the installer, run install-os, reboot into the keyboot unlock environment.

The QEMU smoke tests (ci/qemu/*) get the happy path for free because everything is built-in virtio. Real hardware is where the modules and the console matter — this doc exists so the next agent doesn’t re-derive the fixes below from a dark screen.

The three environments — keep them straight

A real deploy moves through three distinct environments. They have different tooling, and conflating them is the single biggest source of wasted time.

1. Vendor rescue (Hetzner: Debian live). Where you stage and run the installer. It is not a keyboot image — it has none of keyboot’s baked-in tooling. Critically:

   • ZFS is not present. Hetzner’s zpool is a wrapper that compiles OpenZFS against the rescue kernel on first use. It must be recompiled on every fresh rescue boot (the rescue is ephemeral). hetzner-prep.sh drives this.
   • keyboot-install must be built natively here. The repo’s static-musl keyboot binary links a newer glibc than the Debian rescue ships (GLIBC_2.39 vs the rescue’s 2.36), and the install-time keyfile ops use libcryptsetup over FFI — so keyboot-install is cargo build-ed on the rescue against its own libcryptsetup. hetzner-prep.sh does this.
   • Re-flashing the ESP from here needs no ZFS. The ESP is plain vfat on an mdraid1 mirror; updating the active slot’s /keyboot/<A|B>/{vmlinuz,initramfs.img} (ADR 0009; pre-A/B installs used /EFI/keyboot/keyboot-*) is pure mount + cpio + gzip. Don’t recompile ZFS just to swap a kernel.

2. The keyboot unlock environment (the booted keyboot initramfs). What GRUB boots into post-install. It is self-contained: it bundles its own zfs.ko, a static keyboot, cryptsetup, dropbear. It does all LUKS/ZFS work itself — you never feed it ZFS from outside. This is where you SSH in to enter the passphrase.

3. The installed OS (the boot environment / BE). What keyboot kexecs into after unlocking. For Alpine this is configured by alpine.sh to come up networked with sshd (see “The installed OS must be reachable” below).

Rule of thumb: pool/ZFS/LUKS work happens in environment 2 or 3, which carry their own ZFS. In environment 1 (rescue) ZFS is a cost you pay only when you must touch the pool from outside (e.g. inspecting an install). A kernel/ESP swap is not such a case.

A fourth, sneaky one: the vendor “KVM” may be QEMU

Hetzner’s vKVM is a QEMU Q35 + OVMF virtual machine that passes the real disks through. Consequences that have bitten us:

• Disks appear as QEMU HARDDISK with synthetic serials (QM00013, QM00015), not the drives’ real serials. The real Seagate/etc. serials are only visible from the rescue or a true bare-metal boot. keyboot opens disks as sn-<serial>, so the mapper names differ between vKVM and bare metal — that’s expected, the pool imports by scanning either way.
• The vKVM gives you a video console; a plain bare-metal boot (no vKVM attached) does not. So “boots fine under vKVM” does not prove “visible on bare metal” — and vice-versa, “dark on bare metal” is often just no console, not a hang. Verify bare-metal boots over SSH into the unlock env, not by staring at a console you don’t have.

Real-hardware fixes baked into the boot image

These are all upstreamed into the repo; listed here so you know why they’re there and where to look if a new box misbehaves.

The boot image you ship must be built with INCLUDE_MODULES=all (real NIC/HBA coverage) and your SSH pubkey baked in (so you can reach the unlock env’s dropbear). Build the kernel on a real build host, the initramfs on the musl image factory (keybootvm) — see kernel/build.sh and ci/build-image.sh.

The installed OS must be reachable (net + ssh)

install-os historically produced a console-only substrate: a serial getty and a boot marker, enough for the QEMU cold-boot CI gate, but with no networking and no sshd enabled. On a headless server with no KVM that boots to an unreachable box.

alpine.sh now configures the BE as a reachable headless server:

• DHCP on eth0 (/etc/network/interfaces);
• the standard Alpine runlevel set populated via runlevel symlinks, so networking’s hard deps (localmount, hostname) are met and hwdrivers coldplugs the real NIC driver (the QEMU path only worked because virtio_net is built-in);
• sshd enabled, root login key-only (stock prohibit-password);
• the operator key from --authorized-keys <file> written to /root/.ssh/authorized_keys.

Pass --authorized-keys to install-os or the booted OS will be console-only. (Debian/Gentoo plugins need the equivalent — tracked as a follow-up.)

Two unlock modes — pick the right one for the test

keyboot stage-4 prompts for the passphrase, racing the console and any SSH session (dropbear forced-command keyboot-askpass). There is one exception: if a passphrase file is baked into the initramfs at /etc/keyboot/test/passphrase, stage-4 reads it and auto-unlocks without prompting. That file is the CI/unattended hook (set via --test-passphrase-file), not a keyboot feature you want by default.

• Interactive (default, no baked file): the box comes up in the unlock env and waits, staying reachable on the network. Use this to verify a no-KVM bare-metal boot: SSH into the unlock env, feed the passphrase, watch it unlock → import → kexec. (If the box drops off the network shortly after you feed the passphrase, that’s success — it kexeced away.)
• Auto-unlock (baked file): the box unlocks and kexecs with no prompt. Convenient, but on a no-KVM box it races past the only environment you can reach, so you can’t observe the unlock. Reserve it for CI or genuinely unattended production where the BE is itself reachable.

Note both modes concern the keyboot env’s unlock (stage-4). The BE re-unlock after kexec is now silent (ADR 0003 key hand-off): keyboot carries the payload across into the BE, so production boots with a single prompt — the one in the keyboot env. A successful real-hardware test therefore enters the passphrase once and the box rides all the way to the BE’s own sshd; if it stalls after kexec waiting for a second passphrase, the hand-off failed (check for keyboot.handoff=0 or a missing overlay). Confirm via the BE’s dmesg: keyboot be-unlock: using handed-off keyfile payload (single-prompt).

End-to-end procedure

# 0. Operator boots the box into the vendor rescue, gives you SSH access.

# 1. Build the boot image (real NIC coverage + your key). Kernel on the build
#    host; initramfs on the musl factory (keybootvm):
#      bash kernel/build.sh
#      INCLUDE_MODULES=all AUTHORIZED_KEYS=you.pub bash ci/build-image.sh

# 2. Ship the repo + boot image to the rescue (exclude build trees):
rsync -az --exclude 'tools/keyboot/target*' --exclude '.git' \
      --exclude 'kernel/.build' --exclude 'zfs/.build' \
      ./ root@<box>:/root/keyboot/
scp keyboot-vmlinuz keyboot-initramfs.cpio.gz <static-musl-keyboot> root@<box>:/tmp/

# 3. Bootstrap the rescue (ZFS recompile + apk.static + native keyboot-install):
ssh root@<box> 'cd /root/keyboot &&
  KEYBOOT_STATIC=/tmp/keyboot-static-musl bash tools/hetzner-prep.sh'

# 4. Install (DESTROYS the disks). Interactive unlock (no --test-passphrase-file):
ssh root@<box> '
  export KEYBOOT_INSTALL_OS_DIR=/root/keyboot-install-os
  export KEYBOOT=/usr/local/sbin/keyboot KEYBOOT_INSTALL=/usr/local/sbin/keyboot-install
  printf %s "<passphrase>" > /root/.kbpass
  keyboot-install install-os alpine \
    --disk /dev/sda --disk /dev/sdb \
    --passphrase-from file:/root/.kbpass \
    --keyboot-kernel /tmp/keyboot-vmlinuz \
    --keyboot-initramfs /tmp/keyboot-initramfs.cpio.gz \
    --authorized-keys /tmp/you.pub \
    --hostname <name> --no-reboot --confirm'

# 5. Operator reboots to bare metal. The box comes up in the keyboot unlock env
#    (networked, your key). SSH in and feed the passphrase:
printf %s "<passphrase>" | ssh root@<box>      # dropbear runs keyboot-askpass
#    -> keyfile open -> data disks open as sn-<serial> -> pool import -> kexec
#    -> the installed Alpine boots networked; SSH in to verify (zpool status, etc.)

Re-flashing only the ESP (no reinstall, no ZFS)

To swap just the keyboot kernel/initramfs on an already-installed box, from the rescue — e.g. to ship a new boot image without redoing the install:

mdadm --assemble --scan; mount /dev/md127 /mnt/esp        # KEYBOOT-ESP vfat
# The host keyfile is a trailing gzip member appended to the ESP initramfs.
# Split it off and re-attach to the new base image (preserves the keyfile,
# byte-for-byte) — keyfile-only (no test/passphrase) keeps interactive unlock:
python3 - <<'PY'
import zlib, gzip
d = zlib.decompressobj(16+zlib.MAX_WBITS)
img = open('/mnt/esp/keyboot/A/initramfs.img','rb').read()   # pre-A/B: /mnt/esp/EFI/keyboot/keyboot-initramfs.img
d.decompress(img)                       # consume base; unused_data = keyfile seg
open('/tmp/kf.gz','wb').write(d.unused_data)
PY
cat new-initramfs.cpio.gz /tmp/kf.gz > /mnt/esp/keyboot/A/initramfs.img
cp new-vmlinuz /mnt/esp/keyboot/A/vmlinuz
sync; umount /mnt/esp

(If the trailing segment also contains etc/keyboot/test/passphrase, extract keyfile.luks and rebuild a keyfile-only segment to drop auto-unlock.)

Adding a distro BE via provision mode (multi-distro on one pool)

The provisioning vision: keyboot’s own env is the only rescue you ever need. Boot the installed keyboot image, import the pool, and install-os --add-be any distro onto the existing rpool — no partition/keyfile/pool-create/GRUB churn, and the new BE is auto-discovered by the picker (stage-7). Build the image with INCLUDE_INSTALL=yes INCLUDE_ZFS=yes so the provision shell bundles keyboot-install + the orchestrator + zfs.ko (the shipped keyboot-install is the fully static, no-FFI binary — runs in any env with cryptsetup on PATH).

A console is required to enter provision mode on a box whose ESP grub.cfg has a fixed keyboot.mode=boot (e.g. a legacy single-slot install): there is no SSH path to change the boot mode — pick/edit it at the keyboot GRUB menu. Once the ESP carries the GL#44 one-shot grub.cfg (any A/B-staged image), keyboot-install keyboot provision --confirm arms a self-reverting provision boot with no console needed.

# At the keyboot GRUB menu (console): pick KEYBOOT - PROVISION, or edit an entry
# and append keyboot.mode=provision. keyboot unlocks (keyfile injected) -> imports
# rpool READ-WRITE -> drops to an install-os-ready shell.

# In the provision shell — add a distro into a NEW BE on the existing pool:
keyboot-install install-os gentoo --add-be --pool rpool \
    --authorized-keys /path/to/you.pub --confirm

# Reboot; the new BE is auto-discovered at the menu. The original BE is untouched
# and stays the default until you `keyboot-install be promote` the new one.

Do NOT run --add-be from a running BE over SSH: the orchestrator’s teardown ends with zpool export <pool>, which cannot run on the live root pool. --add-be belongs in the provision/rescue env (pool imported but not the live root).

Operator guide (day-2)

Routine operations on a healthy keyboot fleet. For broken hosts see disaster-recovery.md; for real-hardware test procedures see hardware-test-runbook.md. Commands assume the host-side keyboot-install and the runtime keyboot (the BE also carries the be-tools: keyboot-be-upgrade/-rollback/keyboot-autosnap/keyboot-snap).

1. Update keyboot itself (A/B self-upgrade, ADR 0009)

Is an update available? keyboot-update-check compares the installed boot-image version against the channel’s latest pointer — for admins (run it) and cron/scripts (exit code: 0 up-to-date, 10 update-available, 1 error):

keyboot-update-check            # "keyboot v0.1.4 is up to date (latest v0.1.4)"
keyboot-update-check --quiet || echo "update available"   # cron-friendly
keyboot-update-check --json    # {"status":"...","installed":"...","latest":"..."}

keyboot updates are fail-safe by construction: stage the new image into the inactive ESP slot and arm a one-attempt trial; if it doesn’t unlock+import the next boot auto-reverts to the known-good slot. A good trial commits itself.

# 1. Get the new artifact (signed package channel — packages.osterman.co):
apk upgrade keyboot keyboot-install        # Alpine (apk ships both)
apt update && apt install --only-upgrade keyboot   # Debian (runtime only)
# (or the raw installer: curl -fsSL https://packages.osterman.co/keyboot/install | sh)
# NOTE: keyboot-install (the FFI installer used in step 2) is Alpine/rescue-only;
# it is NOT packaged for Debian/Gentoo. There, build it natively
# (tools/hetzner-prep.sh) or run the staging step from the keyboot rescue env.

# 2. Stage it into the inactive slot + arm the trial:
keyboot-install install --slot inactive --promote --version-tag <ver> \
    --kernel <new-vmlinuz> --initramfs <new-initramfs.img> --confirm
keyboot-install keyboot list               # see slots/roles/trial

# 3. Reboot. The trial boots once; on a clean unlock+import it COMMITS (becomes
#    the known-good default). If it fails, boot 2 auto-reverts. No babysitting.

• Inspect/serve: keyboot-install keyboot list.
• Arm/disarm manually: keyboot-install keyboot promote <A|B> --confirm / … rollback --confirm (the panic button — pins the known-good slot).
• One-shot provision boot (add a distro from keyboot’s own env, no vendor rescue): keyboot-install keyboot provision --confirm; reboot, it enters keyboot.mode=provision once then reverts. See §4.

Validate a release across a real upgrade in QEMU first (qemu:ab-revert); the fleet should upgrade keyboot before bumping pool features (§5, ADR 0004).

2. Boot-environment lifecycle (ADR 0005/0006/0007)

Never upgrade in place — clone the BE, upgrade the clone, boot it.

# Scheduled snapshots (keyboot-autosnap = the cron engine: create + retention):
#   <ds>@<YYYY-MM-DD-HHMM>Z-<LABEL>  (UTC, ADR 0006)
keyboot-autosnap --label DAILY --keep 14   # one cron line per frequency/label

# Manage snapshots interactively (keyboot-snap = the admin front-end):
keyboot-snap list [--label L] [--json]     # inspect (name carries the UTC time)
keyboot-snap create [--label L]            # on-demand snapshot now (default MANUAL)
keyboot-snap prune --label L --keep N      # retention on demand (skips held)
keyboot-snap hold|release <UTC-LABEL>      # protect a point-in-time from pruning
keyboot-snap destroy <UTC-LABEL> --confirm # remove a point-in-time (recursive)

# Cross-release / risky upgrade (clone-chroot, original untouched):
keyboot-be-upgrade -- sh -c '<distro upgrade commands>'   # prints the clone name
# then boot it ONCE (auto-reverts to bootfs if it hangs):
#   echo 'be=<clone>' > /esp/keyboot/once.next   (ESP mounted; see the runbook)
# keep it permanently once happy:
keyboot-install be promote <clone> --pool <pool> --confirm  # set bootfs (--pool defaults to rpool)

# Boot a specific BE once (boot-once marker):
keyboot-install be boot-next <dataset>

# Roll back (pre-boot rungs, ADR 0007) — see the runbook for cmdline mechanics:
#   keyboot.rollback=<ds>@<snap>:ro        (ephemeral ro clone; zero side effects)
#   keyboot.rollback=<ds>@<snap>:destroy keyboot.rollback.confirm=1   (in place)
keyboot-be-rollback ...     # the BE-side helper (clone/destroy + ephemeral GC)

3. Key management (SPEC §7, ADRs 0003/0013)

One keyfile container, multi-slot; every slot opens the same 32-byte payload, so data-disk enrollment is untouched by passphrase changes. Slot map: 0 daily, 1 recovery, 2 automation, 3–7 reserved.

keyboot keyfile list-slots <keyfile>                       # read-only inventory
keyboot-install keyfile add-recovery <keyfile>             # slot 1 (do this early!)
keyboot-install keyfile enroll-automation <keyfile>        # slot 2 (Ansible unlock)
keyboot-install keyfile passwd <keyfile> --slot <N>        # change a slot in place
keyboot-install keyfile rekey  <keyfile> --retire-slot <N> --new-slot <M> --confirm
        # remote-safe: add new -> prove it opens -> wipe old (never zeroes a
        # working slot). The Ansible bootstrap->true-secret flow.
keyboot-install keyfile rotate <keyfile> --disk <dev>... --confirm
        # DISTINCT: regenerates the payload + re-enrolls every disk.

Back up the keyfile container off-box. It’s small and slot-encrypted; it’s the only recovery path if every passphrase is lost (§1 of disaster-recovery).

Unified front-end + operator SSH keys (keyboot-keys):

# LUKS slots (routes to the tools above; needs the keyfile present, e.g. the
# keyboot provision/rescue env, or pass --keyfile):
keyboot-keys luks list | info | add-recovery | enroll-automation | passwd | rekey | rotate

# Operator SSH keys for the UNLOCK ENVIRONMENT (the dropbear you SSH into to
# enter the passphrase). Edits an editable overlay on the ESP
# (/esp/keyboot/authorized_keys) that stage-3-ssh merges with the image-baked
# keys — so you rotate operator keys WITHOUT rebuilding the image:
keyboot-keys ssh list
keyboot-keys ssh add  "ssh-ed25519 AAAA... ops@laptop"   # or a path to a .pub
keyboot-keys ssh remove ops@laptop                       # substring match
# New keys take effect on the next boot into the keyboot unlock env.

Prefer a browser to SSH for unlock + BE selection? See §8 (web unlock UI).

4. Add a host / add a distro (SPEC §13)

# Fresh install (DESTROYS the target disks). Dry-run by default; --confirm writes.
keyboot-install install-os <debian|gentoo|alpine> --disk <dev>... \
    --hostname h --authorized-keys you.pub --confirm
# or declaratively (SPEC §13.7):
keyboot-install install-os <distro> --profile profile.yaml --confirm

# Bootstrap from any live env (no keyboot rescue needed):
curl -fsSL https://packages.osterman.co/keyboot/install-os | sh -s -- --run debian --confirm
# (musl live env; for a glibc Hetzner rescue use tools/hetzner-prep.sh)

# Multi-distro on ONE pool — add a BE without partition/keyfile/pool/GRUB churn:
keyboot-install install-os <distro> --add-be --pool rpool --confirm

# The provisioning model: boot the installed keyboot image into provision mode
# (keyboot.mode=provision; arm with `keyboot-install keyboot provision --confirm`)
# -> it unlocks + imports rpool read-write + drops to an install-os-ready shell.
# Add a BE there, reboot, pick it. keyboot's own env IS the rescue — never the
# vendor rescue after the one-time install.

Headless reachability is automatic: all three distro plugins bring up DHCP + sshd and install --authorized-keys so the booted OS is SSH-reachable. Pass --authorized-keys for --add-be too.

5. Pool health & features (ADR 0004/0011)

zpool status rpool            # health, resilver/scrub progress, errors
zpool scrub rpool            # schedule periodically (cron/systemd-timer)
zpool get bootfs rpool        # current default BE

• Feature-flag ordering (ADR 0004): keyboot’s embedded OpenZFS is the floor. Upgrade keyboot across the fleet before zpool upgrade — never enable pool features a host’s keyboot (or a BE’s ZFS) can’t read. Pools are created compatibility=openzfs-2.1-linux so older-OpenZFS BEs still import.
• ARC (ADR 0011): keyboot’s unlock env caps zfs_arc_max at 512 MiB (override keyboot.zfs_arc_max= on the cmdline); the booted OS owns production ARC tuning (/etc/modprobe.d/zfs.conf, per-host, Ansible-managed).

6. Disks (planned: grow/scrub; failed: see disaster-recovery §7)

• Replacing a failed mirror member: disaster-recovery.md §7 (partition → keyfile enroll → zpool replace sn-<old> sn-<new> → mdadm --add the ESP member → BIOS grub-install per member).
• Topology for new pools: keyboot-install topology plan + --topology/--special/ --log/--cache/--spare on install-os (ADR 0010).

7. Diagnostics

• RAM: bootable memtest86+ (memtest.md; keyboot.mode=memtest or the GRUB entry) for ALL physical RAM; the recovery-shell memtester for a quick kernel-RAM check.
• Boot debugging: keyboot.debug=1 surfaces stage logging on the console; keyboot.stop-after=<N> halts after stage N (SSH stays up ≥ stage 3).

8. Web unlock UI (ADR 0016)

A browser alternative to SSHing the dropbear to type the passphrase / pick a BE. Opt-in and additive — dropbear stays the default; nothing changes unless you set keyboot.web=1 on the keyboot menuentry cmdline. The release runtime binary (keyboot-x86_64-musl, v0.1.19+) is built with both phases compiled in. Same unlock path as everything else: the page POSTs the passphrase to the stage-4 FIFO, exactly like the SSH askpass.

Phase 1 — loopback + SSH tunnel (no setup, SSH-grade security). With keyboot.web=1 and no baked cert set, keyboot serves the UI on 127.0.0.1:8090 only. Reach it through an SSH tunnel (the boot key permits forwarding only when keyboot.web=1):

ssh -L 8090:localhost:8090 root@<host>     # then open http://localhost:8090

Security is identical to today (dropbear is still the only thing on the network).

Phase 2 — network HTTPS + mTLS (no tunnel). Bake a TLS cert set into the image and keyboot serves the UI directly on 0.0.0.0:8443 with mutual TLS — the server cert is trusted like an SSH host key, and a client must present a CA-signed cert (the authorized_keys analog) or the handshake is refused.

# 1. Generate the material (server cert + client CA + an operator client bundle).
#    SAN must list every name/IP you'll reach the box by.
SAN="IP:<host-ip>,DNS:<hostname>" OUT=web-certs ci/gen-web-certs.sh

# 2. Bake server-cert.pem + server-key.pem + client-ca.pem into the keyboot image
#    (like the SSH host key). They land at /etc/keyboot/web/ in the initramfs:
WEB_TLS_DIR=web-certs INCLUDE_MODULES=all AUTHORIZED_KEYS=you.pub ci/build-image.sh
#    then stage the rebuilt image into a slot (§1) and reboot into it.

# 3. On your client: import web-certs/client.p12 into the browser/OS keystore
#    (the mTLS identity), and trust web-certs/server-cert.pem once — or, before
#    trusting, compare its SHA-256 to the fingerprint keyboot prints on the
#    console (serial/KVM/IPMI) for a manual no-MITM proof:
openssl x509 -in web-certs/server-cert.pem -noout -fingerprint -sha256

Then browse to https://<host>:8443 and unlock. Without a valid client cert no request is ever served; rotate the cert set the same way you rotate the SSH host key (rebuild + restage the image). Keep web-certs/client-ca-key.pem offline — it signs future client certs.

See also: decisions/ (the ADRs behind each of the above).

Disaster recovery

What to do when a keyboot host is broken. Pairs with hardware-test-runbook.md (the recovery-model primer + ESP-as-control-surface mechanics apply here too) and assumes the architecture in SPEC §5/§7 and ADRs 0003/0004/0007/0009.

First principle: keyboot is designed so failures land you in a reachable place, not a brick. keyboot brings the network + dropbear up early (stage-2/3, before unlock), so a unlock/BE failure usually leaves a keyboot recovery shell over SSH (the operator key). The A/B slots (ADR 0009) and the BE timeline (ADR 0007) give you a known-good fallback for the two things that can hard-fail: keyboot itself, and the BE. Keep a console (KVM/serial) for the rest.

Quick map — symptom → section:

1. Lost or rejected passphrase

The keyfile container (SPEC §7) is multi-slot: slot 0 daily, slot 1 recovery, slot 2 automation, 3–7 reserved. Any slot opens the same 32-byte payload, so any one working passphrase recovers the host.

• Slot 0 forgotten, slot 1 known: boot, enter the recovery passphrase at the keyboot prompt — it unlocks normally. Then reset slot 0 in place: keyboot-install keyfile passwd <keyfile> --slot 0.
• Add a recovery slot before you need it: keyboot-install keyfile add-recovery <keyfile> (slot 1), … enroll-automation <keyfile> (slot 2).
• Remote-safe rotation (don’t zero a working slot until the new one proves out): keyboot-install keyfile rekey <keyfile> --retire-slot <old> --new-slot <new> --confirm — adds the new secret, verifies it opens, then wipes the old. This is the Ansible bootstrap→true-secret flow.
• All passphrases lost: the payload is the AES-256 key for every data disk; with no slot openable the data is unrecoverable unless you hold a payload backup. Back up the keyfile container (it’s small, slot-encrypted) off-box at install time — that’s the only escape hatch. With a backup: restore it to the ESP/initramfs and re-enroll a passphrase.

Never shred the last working slot. passwd/rekey keep the payload, so data enrollment is untouched; only rotate regenerates the payload (and re-enrolls every disk — keyboot-install keyfile rotate <keyfile> --disk <dev>... --confirm).

2. keyboot image won’t boot

keyboot ships as two ESP slots (ADR 0009: /keyboot/{A,B}/) with a grubenv-driven default + a decrement-before-boot trial counter.

• A bad upgrade self-reverts: a staged-and-promoted slot that fails to unlock+import spends its one trial; the next boot auto-reverts to the known-good slot. No action needed; if you’re impatient, power-cycle.
• Force the good slot now: at the GRUB menu pick KEYBOOT - slot A (or B). Or from a recovery shell / booted OS, pin it: keyboot-install keyboot rollback --confirm (clears any trial) and confirm keyboot-install keyboot list.
• Both slots corrupt / ESP grub broken: boot the USB rescue image (usb-rescue.md) or the vendor rescue, then re-stage: keyboot-install install --kernel <vmlinuz> --initramfs <img> --confirm --regenerate-grub (the GRUB seam). On BIOS, GRUB lives per-disk — see §7 for the per-member grub-install.
• Default points at a non-slot (e.g. keyboot_slot=provision) — the grub.cfg now self-heals any non-A/B value to A (GL#44); on an older image, pick slot A at the menu, then keyboot-install keyboot provision --clear --confirm heals the grubenv.

3. Unlock fails

keyboot boots but panics/loops at unlock (keyfile open or LUKS open fails). You land in the recovery shell (SSH).

• Inspect: keyboot disk discover --json, keyboot keyfile info <keyfile>, keyboot keyfile list-slots <keyfile>. Confirm the data disks are present and LUKS (keyboot disk scan-luks).
• Manual unlock to triage (proves the secret + disks): keyboot unlock <keyfile> (opens the keyfile container then every LUKS disk as sn-<serial>). If that works, the boot-time failure is environmental (a disk not yet enumerated — a coldplug/timing issue) rather than a bad secret.
• A disk renamed/replaced: devices open as sn-<serial> so bus reorder is fine, but a replaced disk has a new serial and isn’t enrolled — see §7.
• Wrong/missing keyfile in the image: re-stage the image with the host keyfile (the keyfile is a trailing gzip member on the ESP initramfs; see hetzner-deploy.md “Re-flashing only the ESP”).

4. BE won’t boot

keyboot unlocked + imported, kexec’d the BE, and the BE failed (initramfs panic, no /sysroot, no network).

• Pick another BE: keyboot’s stage-7 honors keyboot.be=<dataset> on the cmdline (GRUB e to add it), the one-shot /esp/keyboot/once.next, then bootfs. Boot a previous BE and investigate.
• Roll back to a snapshot (ADR 0007, pre-boot — no booted OS needed):
  • rung 1 (safe): keyboot.rollback=<ds>@<snap>:ro — boots an ephemeral read-only clone; original untouched.
  • rung 3 (destructive): keyboot.rollback=<ds>@<snap>:destroy keyboot.rollback.confirm=1 — zfs rollback -r in place. See the runbook for the exact cmdline mechanics + recovery.
• A bad BE upgrade: if you used keyboot-be-upgrade, the original BE + @…-PREUPGRADE snapshot are intact — zpool set bootfs=<original> and reboot, or zfs destroy -r <clone> to discard the upgrade.

5. Stuck on the wrong entry

Box keeps booting something you didn’t intend (the GL#44 class: a stale once.next, a hand-set bootfs, a corrupt keyboot_slot, or a left-in grub.cfg cmdline edit).

• From a recovery shell / booted OS, mount the ESP (see the runbook) and inspect /esp/keyboot/{keyboot.env,once.next} and /esp/grub/grub.cfg.
• Clear a stuck one-shot: rm /esp/keyboot/once.next. Fix the default: zpool set bootfs=<good-be> rpool. Heal a corrupt slot: keyboot-install keyboot provision --clear --confirm (resets non-A/B keyboot_slot to A) — or just rely on the grub.cfg A/B guard (GL#44).
• Remember GRUB’s save_env is unreliable on the mdraid1 ESP — make grubenv changes from Linux through the assembled md (mirror-consistent), not from GRUB. keyboot’s runtime writes already do this.

6. Pool feature drift

A BE’s ZFS is older than the pool’s enabled features and refuses to import (ADR 0004: keyboot’s embedded OpenZFS is the floor; hosts upgrade keyboot before bumping pool features).

• keyboot’s own env carries the newest lockstep ZFS, so keyboot still imports — you’re not locked out; you reach the recovery shell.
• Pools are created -o compatibility=openzfs-2.1-linux so a BE on an older OpenZFS (e.g. Debian’s 2.1.x) can still import. If you bumped features past a BE’s ZFS: either upgrade that BE’s ZFS (zfs-dkms/kmod) to clear the gap, or recreate the BE on the supported floor. Don’t zpool upgrade ahead of the fleet’s BE ZFS.

7. Failed disk

Mirror member died. Each disk carries an ESP md member + a LUKS crypt payload (sn-<serial>); the pool vdev is over the crypt mappers.

1. Partition the replacement like the survivors (GPT: BIOS-boot + ESP member
   • crypt) — keyboot-install partition <disk> --wipe / the install-os partition templates.
2. Enroll the new crypt partition into the keyfile (same payload): keyboot-install keyfile enroll <new-crypt-part> --confirm, then keyboot unlock opens it as sn-<new-serial>.
3. Replace in the pool: zpool replace rpool sn-<old> sn-<new>; wait for resilver (zpool status).
4. Re-add the ESP md member: mdadm --add /dev/md/keyboot-esp <new-esp-part>; it resyncs the vfat ESP.
5. BIOS only: reinstall GRUB on the new disk — grub-install --target=i386-pc <new-disk> against the raw ESP member (array stopped first so the member mount doesn’t EBUSY; see hetzner-deploy.md). UEFI needs nothing extra (firmware reads BOOTX64.EFI off the md ESP).

8. Total loss / reinstall

Disks intact but the system is unbootable beyond repair, or you’re rebuilding.

• Boot a provisioning env: USB rescue (usb-rescue.md), the installed keyboot image in keyboot.mode=provision (if it still boots), or the vendor rescue + tools/hetzner-prep.sh (glibc rescue) / curl … /keyboot/install-os | sh (musl live env).
• Data intact, OS gone: import the pool (keyboot unlock → zpool import) and add a fresh BE without touching data: keyboot-install install-os <distro> --add-be --confirm. Reboot, pick it.
• Full reinstall (DESTROYS disks): keyboot-install install-os <distro> --disk <dev>... --confirm (or a --profile). You need the keyfile backup (§1) to keep the existing payload, else the data is gone.
• After any reinstall, re-stage the be-tools-bearing toolkit so the new BE has keyboot-be-upgrade/-rollback (GL#43 — now handled by all bootstrap paths).

See also: hetzner-deploy.md, hardware-test-runbook.md, decisions/ (ADRs 0003 handoff, 0004 ZFS floor, 0007 rollback, 0009 A/B slots).

Real-hardware validation runbook (console-attended)

The remaining real-hardware test items (GL#38 memtest, GL#43 keyboot.rollback= rungs, GL#37 NIC firmware) need a human at the box: a serial/KVM console for recovery, and — for GL#37 — diverse server NICs. This runbook makes such a session fast and safe. It complements hetzner-deploy.md (install procedure + the three-environments model) and memtest.md (memtest staging). The keyboot-be-upgrade BE-upgrade flow (GL#43, first half) is already validated on real hardware — its procedure is captured below for reference because it’s the safe pattern the other tests borrow from.

Before you start — the safety model

• Have a console. On Hetzner that’s the Robot reset button (power-cycle) and the KVM/serial-over-LAN console. Most failure modes here are recoverable over SSH (see below), but a post-kexec hang or a rollback boot-loop needs the console.
• keyboot brings the network + dropbear up EARLY (stage-2/3), before the unlock/BE/rollback logic (stage-4+). So if a test fails inside keyboot, you usually still get a keyboot recovery shell over SSH (the keyboot_hw key) — from which you fix the ESP and reboot. The only truly-dark case is a successful kexec into a BE/target that then hangs (keyboot is gone).
• The ESP is the control surface. On a keyboot box the grubenv (/esp/keyboot/keyboot.env), the boot-once marker (/esp/keyboot/once.next), and the BIOS/UEFI menu config (/esp/grub/grub.cfg) live on the mdraid1 ESP. From any booted OS or the keyboot recovery shell: assemble + mount the ESP md (vfat), edit, sync, unmount. Writing through the assembled md array keeps both mirror members consistent (unlike GRUB’s own save_env on a raw member — see [grubenv on mdraid1, GL#44]).

# Mount the ESP from a booted OS / recovery shell (device name varies: md127, md0, /dev/md/keyboot-esp)
mkdir -p /mnt/esp
for d in /dev/md/keyboot-esp /dev/md127 /dev/md0; do [ -e "$d" ] && mount -t vfat "$d" /mnt/esp && break; done
ls /mnt/esp/keyboot/keyboot.env   # confirm it's the ESP
# ... edit ...
sync; umount /mnt/esp

• Boot selection precedence (keyboot stage-7): keyboot.be= cmdline > /esp/keyboot/once.next (consumed on read) > pool bootfs. Keep a known-good BE as bootfs so any failure that consumes/ignores a one-shot reverts there.

Recovering a stuck box (read this first)

Test A — keyboot-be-upgrade (GL#43, VALIDATED — reference pattern)

Clone-based cross-release BE upgrade. Safe to the running BE (it upgrades an isolated clone; the original is untouched). Validated bookworm→trixie on the Hetzner BIOS box.

# On the booted OS (be-tools are staged from first boot since GL#43 fix):
keyboot-be-upgrade -- sh -c '
  set -e; export DEBIAN_FRONTEND=noninteractive
  sed -i "s/bookworm/trixie/g" /etc/apt/sources.list /etc/apt/sources.list.d/*.list 2>/dev/null || true
  apt-get update --allow-releaseinfo-change -qq
  apt-get -y -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef full-upgrade
  apt-get -y --purge autoremove || true
  update-initramfs -u -k all'        # zfs-dkms rebuilds zfs.ko; keyboot hook persists
# -> prints the upgraded clone name, e.g. rpool/ROOT/debian-<stamp>

Safe boot-once into the upgrade (auto-reverts to bootfs on the next power-cycle if it hangs):

# bootfs stays the known-good BE; once.next selects the clone for ONE boot
mount -t vfat /dev/md127 /mnt/esp
printf 'be=%s\n' rpool/ROOT/debian-<stamp> > /mnt/esp/keyboot/once.next
sync; umount /mnt/esp
reboot

Expected: keyboot consumes once.next, kexecs the upgraded kernel, the BE initramfs re-unlocks + mounts, systemd comes up networked. Verify: cat /etc/os-release, uname -r, zpool status. PASS = new release boots + SSH reachable; original BE + @…-PREUPGRADE snapshot intact. To keep it: zpool set bootfs=rpool/ROOT/debian-<stamp> rpool. To discard: zfs destroy -r rpool/ROOT/debian-<stamp> (after reverting bootfs).

Test B — keyboot.rollback= rungs (GL#43, second half)

Validates the pre-boot rollback ladder (ADR 0007) on bare metal. CI covers it (ci/qemu/rollback-rungs-test.sh); this is the real-hw confirmation. rung 3 is destructive (zfs rollback -r, discards newer state) — do it on a test box.

There is no one-shot cmdline mechanism — keyboot.rollback= must be appended to the keyboot menuentry’s linux line in /esp/grub/grub.cfg, so keep the console up and restore grub.cfg right after each rung. Format: keyboot.rollback=<dataset>@<snap>:<rung> (:ro | :destroy); destroy also needs keyboot.rollback.confirm=1.

Prep (on the booted OS):

BE="$(awk '$2=="/"&&$3=="zfs"{print $1}' /proc/mounts)"
echo MARKER-ORIGINAL > /root/marker            # state to roll back TO
zfs snapshot "$BE@rolltest"
echo MARKER-MUTATED  > /root/marker            # newer state to discard/hide
cp /esp/grub/grub.cfg /esp/grub/grub.cfg.bak   # (mount ESP first)

rung 1 — read-only clone boot (NON-destructive)

Append keyboot.rollback=<BE>@rolltest:ro to the linux /keyboot/A/vmlinuz … line of the keyboot-A menuentry in /esp/grub/grub.cfg; sync; reboot.

• Expected: boots an ephemeral readonly=on clone of the snapshot (tagged keyboot:ephemeral=1); /root/marker reads ORIGINAL; the live BE
  • timeline are untouched (zero side effects).
• Then: from this boot, restore grub.cfg (cp …grub.cfg.bak grub.cfg), sync, reboot → back to the normal BE; /root/marker reads MUTATED (confirms rung 1 left no trace).

rung 3 — in-place destroy (DESTRUCTIVE, guarded)

Append keyboot.rollback=<BE>@rolltest:destroy (no confirm) first:

• Expected: REFUSED without keyboot.rollback.confirm=1 → normal boot, marker still MUTATED. (Also refused if a dependent clone exists — the no--R -f guard, init/lib/rollback.sh.) Then append …:destroy and keyboot.rollback.confirm=1:
• Expected: zfs rollback -r in place — same dataset (not a clone), newer state discarded; /root/marker reads ORIGINAL.
• Then: restore grub.cfg, reboot to normal.

Recovery: if a rollback boot doesn’t bring SSH up, the grub.cfg token is still in place → it will re-enter rollback every boot. Use the KVM console: at the GRUB menu press e, delete the keyboot.rollback… token, boot; then restore /esp/grub/grub.cfg from .bak.

Test C — bootable memtest (GL#38)

Confirms the kexec -l form of memtest86+ 6.x on real hardware (the one ADR 0014 unknown). Staging + entry points are in memtest.md. The box is offline during the test; you watch it on the console.

1. Stage memtest on the ESP and add the GRUB entry (re-run install with KEYBOOT_MEMTEST_DIR=<dir with memtest.{efi,bin}>, or keyboot-install install --memtest --confirm). See memtest.md.
2. Two entry points to verify:
   • GRUB menuentry (reliable): pick memtest86+ — EFI chainloader /EFI/keyboot/memtest.efi, or BIOS linux16 /EFI/keyboot/memtest.bin.
   • keyboot.mode=memtest (the kexec path under test): keyboot kexecs the ESP-staged memtest instead of a BE. This is the GL#38 unknown — confirm the kexec form actually launches memtest86+ 6.x on the metal.
3. Observe on the console (memtest86+ 6.x does serial output — console=ttyS0,115200 on the BIOS linux16 form, watch via serial-over-LAN). PASS = memtest UI runs + addresses all installed RAM (vs. the userland memtester, which only tests kernel-allocatable RAM).
4. Recovery: memtest never touches disk; just reboot (Robot reset) back to the normal entry.

Test D — curated NIC firmware (GL#37)

The curated firmware sets (Alpine subpackages, Debian firmware-*, Gentoo emerge+prune to server-NIC dirs) are only proven on virtio (which needs no firmware). Verify on real server NICs — needs hardware with Broadcom/Mellanox/QLogic/Intel-needs-firmware NICs (the Hetzner test box’s onboard Intel e1000e needs no firmware, so it can’t exercise this).

On a box with such a NIC, after installing with --firmware curated:

lspci -nn | grep -iE 'ethernet|network'        # identify the NIC
ethtool -i <iface> | grep -i firmware          # firmware-version (blob in use)
dmesg | grep -iE 'firmware.*(load|fail)'       # what loaded / what's MISSING
ls /lib/firmware/<driver-dir>                  # blob actually present?

PASS = the NIC links + the driver’s firmware loaded (no “Direct firmware load … failed” for the NIC). If a real NIC needs a dir/blob not in the curated keep-set, add it: Alpine linux-firmware-<sub> in alpine.sh; Debian firmware-<name> in debian.sh; the Gentoo keep-set in _gentoo_curate_firmware (gentoo.sh). Optionally add a non-virtio CI cell with an emulated firmware-needing NIC to catch regressions.

After the session

Record results on the issues (GL#37/#38/#43). If a curated list needed a NIC added, ship that fix. The @rolltest snapshot and any keyboot:ephemeral=1 clones from Test B can be cleaned up (zfs destroy); keyboot-be-rollback’s GC handles ephemeral clones.

Architecture

The high-level view: what keyboot is, what it isn’t, and where it sits relative to GRUB and the booted OS. Extracted from SPEC §3/§4; the locked design decisions live in CLAUDE.md and decisions/ (the ADRs). Start here, then read unlock-flow.md for the mechanism.

What keyboot is

A greenfield boot stage that runs after GRUB and before the target OS’s init. In one boot it:

1. decrypts an age-of-LUKS keyfile container inside its own initramfs,
2. uses that keyfile’s payload to open every LUKS device backing the root zpool,
3. imports the pool,
4. lets the operator pick a boot environment (TTY or SSH),
5. kexecs into that BE’s own kernel + initrd (the ZBM model).

The same artifact, selected by keyboot.mode= on the kernel cmdline, is also a netbootable rescue shell, an unattended installer, and a memtest launcher:

It targets Gentoo (openrc), Debian (systemd), and Alpine (openrc), on UEFI and BIOS-MBR.

What keyboot is NOT

• Not the bootloader. GRUB stays. keyboot is a kernel+initramfs GRUB boots like any other entry; it never replaces GRUB.
• Not a fork. Greenfield, not a patched ZBM/dracut/genkernel — those are reference points only (SPEC §3).
• Not the init. It hands off (kexec) to the BE’s own kernel + init; it does not switch_root into the OS.
• Not re-done by the OS. The booted OS does no LUKS or ZFS-import work — keyboot already opened the disks and (via the post-kexec payload handoff) the BE just re-opens + mounts. No legacy ZFS mounts in any supported distro.

Where it fits

firmware (UEFI / BIOS-MBR)
  └─ GRUB                         ← stays the bootloader; one entry = keyboot
       └─ keyboot  (its own kernel + initramfs)
            stage 1-9: unlock → import → pick BE → kexec
            └─ kexec  ──▶ the BE's OWN kernel + initrd
                              └─ BE initramfs: keyboot be-unlock (re-open + mount)
                                   └─ the distro's init (systemd / openrc)

The GRUB seam

keyboot owns its artifacts; the distro owns grub.cfg. keyboot-install install places /boot/keyboot-vmlinuz + /boot/keyboot-initramfs.img and drops an executable /etc/grub.d/10_keyboot snippet, so the distro’s own grub-mkconfig emits a menuentry 'keyboot'. On a keyboot-owned ESP (install-os) the grub.cfg is rendered directly. Either way keyboot stays one entry inside GRUB.

The three environments (keep them straight)

A keyboot host has up to three distinct ZFS/LUKS contexts — see hetzner-deploy.md:

1. keyboot’s unlock env — its own lockstep kernel + zfs.ko + the keyfile. Imports read-only, briefly. Small uniform ARC (ADR 0011).
2. the booted OS / BE — the distro’s kernel + zfs.ko, the real workload, production ARC.
3. (transient) a vendor/USB rescue — only for the one-time bootstrap; after install, keyboot’s own provision mode is the rescue (the “ENV-as-rescue” vision).

The architectural pillars (locked decisions, SPEC §3)

• Unlock = a LUKS1 keyfile container, not age. Its 32-byte payload is the AES-256 key for every data disk; multi-slot (daily/recovery/automation). LUKS1 for the container and the disks. See unlock-flow.md, SPEC §7.
• Handoff = kexec into the BE’s kernel (ADR 0003): the 32-byte payload is carried RAM-only across the kexec so the BE re-unlocks with a single prompt; keyboot.handoff=0 falls back to an independent dropbear-over-SSH re-unlock (ADR 0002).
• Devices open as sn-<serial> so pool topology survives bus reordering; one Rust source for serial discovery, shared install-time and boot-time.
• A/B ESP slots for keyboot itself (ADR 0009): two slots + a decrement-before-boot trial counter → a bad keyboot upgrade auto-reverts. The GL#44 one-shot provision builds on this.
• BE discovery enumerates any mountable dataset; no enforced layout. BE rollback is a pre-boot three-rung ladder (ADR 0007).
• ZFS in lockstep with upstream OpenZFS; keyboot’s embedded ZFS is the pool feature floor — upgrade keyboot before bumping pool features (ADR 0004).
• Delivery = CI-signed native packages (apk/ebuild/deb) in our own repos + a signed rescue USB; ansible manages the repo definitions (ADR 0008).

The pieces

Design invariants (do not break)

• The decrypted keyfile/payload is RAM-only — tmpfs, never non-volatile storage, the cmdline, dmesg, or a crashdump; zeroed after use.
• GRUB stays the bootloader; keyboot is never it.
• The target OS does no LUKS/ZFS re-work; no legacy ZFS mounts.
• Disk-serial discovery has a single source (the Rust code), shared by install and boot.

See: unlock-flow.md, operator-guide.md, disaster-recovery.md, decisions/ (the ADRs), and SPEC.md (the design of record).

The unlock flow

How keyboot gets from a passphrase to an imported, mounted ZFS root. Extracted from SPEC §5 (stages) and §7 (the keyfile model); the authoritative source is SPEC + the code under init/ and tools/keyboot/src/.

The model in one paragraph

The operator’s passphrase does not key the data disks. It opens a small LUKS1 keyfile container (a loopback file in keyboot’s initramfs); the container’s 32-byte payload is the AES-256 key for every data disk. So one passphrase → one payload → all disks. The payload (not the passphrase) is what’s re-used after kexec so the booted OS unlocks without prompting again. The decrypted payload is RAM-only, always — tmpfs, never non-volatile storage, never the kernel cmdline, never dmesg//proc, zeroed after use (SPEC §17.1, the critical invariants in CLAUDE.md).

passphrase ──▶ LUKS1 keyfile container ──▶ 32-byte payload ──┬──▶ open sn-<serial> (disk 1)
              (multi-slot; slot opens                        ├──▶ open sn-<serial> (disk 2)
               the same payload)                             └──▶ ... every data disk
                                                                      │
                                                            zpool import (read-only)
                                                                      │
                                                            pick BE ──▶ kexec BE kernel
                                                                      │  (carry payload, ADR 0003)
                                                            BE initramfs: keyboot be-unlock
                                                            ──▶ re-open LUKS ──▶ import rw ──▶ mount /sysroot

The keyfile container (SPEC §7)

• LUKS1 (not LUKS2), cipher aes-xts-plain64, key-size 512, hash sha512, iter-time 5000ms — same params for the container and the data disks.
• Multi-slot, each slot a passphrase that unlocks the same payload: 0 daily, 1 recovery, 2 automation (ADR 0013), 3–7 reserved. Adding or changing a slot never disturbs the payload, so data-disk enrollment is untouched (keyfile passwd/rekey — see operator-guide.md).
• The payload is generated once at install (keyboot-install keyfile create) and each data disk is enrolled with it (keyboot-install keyfile enroll <disk>), so the disk’s LUKS key == the payload. rotate regenerates the payload and re-enrolls every disk.

Boot-time stages (the keyboot env)

Run by /init; each init/stage-N-*.sh is sourced in order (run_stage).

The same Rust code drives the open path interactively: keyboot unlock <keyfile> opens the container then every LUKS disk (--single <dev> for one; --dry-run to report). keyboot keyfile info/list-slots inspect read-only.

Serial-based device naming

Data disks open as /dev/mapper/sn-<serial> so the pool topology survives bus reordering (a disk that moves from sda to sdc keeps its sn- name). The serial-discovery logic is a single source of truth in Rust (tools/keyboot/src/disk/serial.rs), exposed as keyboot disk discover --json and used by both the boot-time keyboot unlock and the install-time orchestrator — install and boot agree on names by construction (no separate shell discover script). On real hardware the serial comes from udevadm (the static binary shells out) with a sysfs fallback.

Re-unlock after kexec (ADR 0003)

keyboot kexecs into the BE’s own kernel (the ZBM model), so the BE’s initramfs must re-open the LUKS devices. keyboot hands the 32-byte payload (not the passphrase) across the kexec, RAM-only, as a cpio overlay appended to the BE’s initramfs; keyboot be-unlock consumes it (open keyfile→LUKS→import rw→mount /sysroot), then both key copies are overwrite+unlink’d. Single prompt for the whole boot.

# Inside the BE's initramfs, after kexec:
keyboot be-unlock --pool <P> --root-dataset <DS> [--keyfile F --root-mount /sysroot]

• keyboot.handoff=0 disables the carry and falls back to an independent BE re-unlock (dropbear-in-the-BE-initramfs, passphrase over SSH — ADR 0002). Slower (a second prompt) but carries no key across kexec; the CI qemu:install-os-handoff0 gate covers it.
• The payload must never touch non-volatile storage, the kexec cmdline, dmesg//proc, or a crashdump. This is a do-not-break invariant (CLAUDE.md).

Where it lives

• init/ — /init + stage-1..9 + init/lib/ helpers.
• tools/keyboot/src/disk/serial.rs — disk/serial discovery (single source).
• tools/keyboot/src/cli/unlock.rs, …/be_unlock.rs — the unlock / be-unlock commands.
• tools/keyboot/src/keyfile/ — keyfile container create/enroll/open/slots.

See also: operator-guide.md (key rotation), disaster-recovery.md (when unlock fails), decisions/ (ADR 0003 handoff, 0013 automation slot).

Dataset layout — a reference for --dataset

keyboot does not bake an opinionated dataset profile. By default install-os creates only the BE root (rpool/ROOT/<be>); you spell out the leaves you want with --dataset (repeatable), and an opinionated default profile is a future addition (parameterize-now, defaults-later — ADR 0005 refinement). This page is the reference for what to pass.

The rule (ADR 0005)

• Binary-coupled state stays in the BE root — anything whose state must match the installed binaries rolls back as one atomic unit. Notably /var (including the package DB) and /etc stay in the BE.
• Split out only binary-independent leaves — data whose lifetime is independent of the OS image, or that wants its own recordsize/snapshot policy. That’s the only reason to make a separate dataset.

Two classes, distinguished purely by where the dataset lives:

• BE-scoped — a child of rpool/ROOT/<be> (e.g. {BE}/home): clones + rolls back with the BE.
• Persistent — outside rpool/ROOT (e.g. rpool/var/log): survives a BE switch/rollback.

--dataset encodes this with the {BE} token: {BE}/... is BE-scoped, anything else is persistent.

Reference table

Intermediate parents (e.g. rpool/var for rpool/var/log) are auto-created with mountpoint=none, so they never shadow /.

Recommended per-dataset defaults

Set where it pays — the reason to split a leaf at all:

• compression=lz4 (cheap, near-universal win; zstd for cold/archival data)
• atime=off (avoid write amplification on reads)
• xattr=sa, acltype=posixacl (SA xattrs; needed by systemd/journald, samba)
• recordsize: leave the 128K default for general data; 8K–16K for DB datasets matching their page size
• exec=off / setuid=off / devices=off on data-only datasets (/var/log, /srv, caches) — defence in depth
• com.sun:auto-snapshot=false on discardable datasets (/var/cache, swap)

Swap (--swap, GL#35)

install-os --swap none|zvol:<size> (default none). zvol:<size> creates <pool>/swap as a ZFS volume and mkswaps it. Because the zvol lives on the pool, swap is encrypted at rest via the same LUKS keyfile as everything else — no extra key management.

The zvol uses OpenZFS swap-tuned props (sync=always, primarycache=metadata, secondarycache=none, logbias=throughput, compression=zle, 4K volblock, com.sun:auto-snapshot=false).

Boot-time activation is a dedicated keyboot-swap service, not fstab. At boot the zvol’s device nodes fire their uevents inside keyboot’s initramfs (before the BE’s real udev), and minimal BEs lack the ZFS udev rules that make /dev/zvol/<pool>/swap, so a plain swapon -a from fstab races a non-existent symlink. Instead /usr/local/sbin/keyboot-swapon re-triggers udev and swapons the swap-signed zdN device directly (validated by swapon itself, so no blkid flag dependence). It runs via an openrc init service in the boot runlevel or a systemd oneshot (keyboot-swap.service, WantedBy=multi-user). The install marker prints KEYBOOT-SWAP <active-count> so the boot test can assert swap actually came up.

Caveat — swap-on-zvol can deadlock under heavy memory pressure (ZFS may need to allocate memory to write out swap — the memory it’s trying to free). It’s fine for moderate overcommit, but for swap-heavy workloads a raw, keyfile-encrypted swap partition is safer. That backing (--swap partition:<size>) is deferred to GL#34: it can’t live on the pool, so it needs the partition-role work.

Example: a typical server

install-os debian --disk … \
  --dataset '{BE}/home:/home' \
  --dataset 'rpool/var/log:/var/log:exec=off' \
  --dataset 'rpool/var/cache:/var/cache:com.sun:auto-snapshot=false' \
  --dataset 'rpool/data/docker:/var/lib/docker'

This table is a recommendation, not a baked default. Fleet policy belongs in Ansible (the installer accepts a spec; ADR 0005 / 0015). An opinionated one-flag default profile is deferred to a future pass.

What keyboot can boot — the supported-OS boundary

keyboot is a launcher, not a container. After it opens the encrypted pool it kexecs into the OS’s own kernel + initramfs and exits — once your OS is running, keyboot is gone. That one fact decides which operating systems keyboot can boot, and why.

The BE initramfs requirement

Your root filesystem is the encrypted ZFS pool — there is no plaintext /, and even /boot lives inside the encrypted dataset. So the only path to a mountable root runs through: load dm-crypt + the ciphers, open the LUKS volumes, load zfs.ko, import the pool, mount the dataset. After kexec, the thing that must do all of that is the OS’s own initramfs.

Therefore every boot environment needs its own initramfs carrying crypto + zfs + the keyboot be-unlock hook, plus a kernel that supports dm-crypt and a zfs.ko matching that kernel. keyboot’s install-os plugins bundle exactly these (Alpine mkinitfs, Debian initramfs-tools boot=keyboot, Gentoo dracut 90keyboot). An OS image whose initramfs lacks them will not boot — it has no way to reach its own root.

The single-prompt handoff carries the key, not the tools. keyboot passes the 32-byte keyfile payload across kexec so the BE doesn’t re-prompt — but the BE still has to open the disks and import the pool itself, which needs the crypto + zfs + be-unlock bits in its initramfs. A no-crypto/no-zfs image gets handed a key it has nothing to use.

The boundary

The decision rule for “can this run as a keyboot OS” is not “does it have zfs/crypto modules” — it’s “is its root a mountable ZFS dataset, and can I get a zfs.ko for its kernel.”

Why immutable/image distros don’t fit

Two walls, and the modules are the least of them:

1. No kernel-matched zfs.ko. ZFS is out-of-tree; it must be compiled against that exact kernel. These distros ship no ZFS and are precisely the kind you can’t build it on in place (read-only squashfs appliance; rpm-ostree-immutable).
2. Their root isn’t a mountable ZFS dataset. LibreELEC loop-mounts a SYSTEM squashfs with its own init; ostree selects a deployment via an ostree= karg; SteamOS/ChromeOS use A/B image + verity. be-unlock mounting rpool/ROOT/<be> at / satisfies none of those — the blocker is what the OS expects / to be, not which modules are present.

So an immutable distro is fine as an install target you put on a different system, but it can’t be a keyboot BE without porting it to a ZFS-dataset root — a per-distro project, not a config tweak.

A durability gotcha: keep the BE initramfs keyboot-aware

The fragile moment is whenever the OS regenerates its own initramfs (a kernel upgrade). The new initramfs must still carry crypto + zfs + the keyboot hook, or the next boot can’t mount root:

• Debian (initramfs-tools) & Gentoo (dracut): self-healing. The integration is a persistent hook/module, so a kernel bump re-includes it automatically.
• Alpine (mkinitfs): NOT self-healing. mkinitfs has no hook mechanism; the apk linux-lts trigger rebuilds the initramfs with the default init, dropping keyboot’s be-init. A kernel bump outside keyboot-be-upgrade (which rebuilds it) leaves an unbootable image — a known footgun on Alpine.

Also: zfs.ko must match the running kernel after an upgrade (a failed dkms/akmod rebuild → no pool import), and don’t uninstall cryptsetup/zfs (they feed the next initramfs regeneration).

This is intrinsic to encrypted-ZFS-on-LUKS root, not a keyboot quirk: the initramfs is always the bridge between kexec and your encrypted root. keyboot’s contributions are doing it uniformly across the three initramfs frameworks and the single-prompt key hand-off — it satisfies the requirement, it doesn’t remove it.

memtest86+ — staging, entry points, netboot (ADR 0014)

keyboot ships no memtest binaries (memtest86+ is GPLv2 but a build artifact, not vendored). You stage them once; every entry point below then lights up. Absent binaries, the entries are simply not emitted / fail gracefully — nothing breaks.

Getting the binaries

Download the latest memtest86+ 6.x release from https://memtest.org/ (or build from https://github.com/memtest86plus/memtest86plus). Two artifacts matter:

Put both in one directory, named exactly memtest.efi / memtest.bin:

mkdir -p ~/memtest && cd ~/memtest
# from the release zips: mt86plus_<ver>.{efi,bin}.zip
unzip mt86plus_*.efi.zip && mv memtest64.efi memtest.efi
unzip mt86plus_*.bin.zip && mv memtest64.bin memtest.bin

Staging (KEYBOOT_MEMTEST_DIR)

Set KEYBOOT_MEMTEST_DIR=<that dir> and the build/install paths stage the images at /EFI/keyboot/memtest.{efi,bin} on the ESP and emit a memtest86+ GRUB menuentry (EFI chainload; BIOS linux16):

# USB rescue stick (ci/build-usb.sh)
sudo KEYBOOT_MEMTEST_DIR=~/memtest ci/build-usb.sh

# Fresh install (the install-os orchestrator grub step)
KEYBOOT_MEMTEST_DIR=~/memtest keyboot-install install-os alpine ... --confirm

On an existing host, copy the files onto the keyboot ESP yourself and drop the chainload snippet with keyboot-install install --memtest --confirm (writes /etc/grub.d/11_keyboot_memtest).

Entry points

1. GRUB menuentry — the reliable path; chainloads memtest.efi before keyboot even starts (BIOS: linux16 memtest.bin).
2. keyboot.mode=memtest — keyboot kexecs the ESP-staged memtest.bin right after stage-1. SSH/IPMI-scriptable on a headless box; best-effort until verified on real hardware (GL#38).
3. stage-8 menu m — same kexec, for an operator already in keyboot.
4. Netboot (GL#39) — no local footprint at all: the published netboot.ipxe menu (ci/gen-netboot.sh) boots ${BASE_URL}/memtest.{efi,bin} straight over the network — for burning in a box before it has any install. Publish the two binaries next to keyboot-vmlinuz/keyboot-initramfs.img at the release URL:

BASE_URL=http://packages.osterman.co/keyboot/v0.1.0 ci/gen-netboot.sh > netboot.ipxe

Serial output (console=ttyS0,115200) is baked into every memtest append so progress reaches IPMI serial-over-LAN.

Native package repositories

Besides the signed curl | sh channel (see ci/install.sh), keyboot’s userland tools are published as native packages on packages.osterman.co — an apk repo, an apt repo, and a Gentoo overlay. All three are built and signed in CI on a release tag (.gitlab-ci.yml: package:apk / package:deb / package:overlay → publish:repos), from the same artifacts as the raw channel.

Packages provided (the three-package split, ADR 0008):

Alpine (apk)

# Trust the repo signing key, then add the repo and install.
wget -O /etc/apk/keys/keyboot-pkg.rsa.pub https://packages.osterman.co/alpine/keyboot-pkg.rsa.pub
echo "https://packages.osterman.co/alpine/v3.23" >> /etc/apk/repositories
apk update
apk add keyboot                 # runtime tools
apk add keyboot-install         # + the installer (rescue hosts)

Debian / Ubuntu (apt)

# Trust the archive key (dearmored into the keyrings dir).
curl -fsSL https://packages.osterman.co/debian/keyboot-archive-keyring.asc \
  | gpg --dearmor -o /usr/share/keyrings/keyboot-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/keyboot-archive-keyring.gpg] https://packages.osterman.co/debian stable main" \
  > /etc/apt/sources.list.d/keyboot.list
apt-get update
apt-get install keyboot

The Debian repo ships only the keyboot package (the running-host userland — a fully static binary that runs on glibc). keyboot-install (FFI rescue binary) and keyboot-boot (kernel + initramfs) are Alpine/rescue-only; get them from the apk repo or the rescue image.

Gentoo (overlay)

The package host is static HTTP, so the overlay is distributed as a snapshot tarball you add as a local overlay (the ebuilds fetch the prebuilt musl statics from the package host, so there is nothing to compile):

The overlay tarball is published per release; use the current version (from https://packages.osterman.co/keyboot/latest):

ver=$(wget -qO- https://packages.osterman.co/keyboot/latest)   # e.g. v0.1.12
mkdir -p /var/db/repos/keyboot
wget -O - "https://packages.osterman.co/gentoo-overlay/keyboot-overlay-${ver#v}.tar.gz" \
  | tar -xz -C /var/db/repos/keyboot --strip-components=1
cat > /etc/portage/repos.conf/keyboot.conf <<'EOF'
[keyboot]
location = /var/db/repos/keyboot
masters = gentoo
auto-sync = no
EOF
emerge -av app-admin/keyboot
emerge -av app-admin/keyboot-install

Signing

• apk: the repository APKINDEX.tar.gz is abuild-signed with the keyboot apk RSA key; its public half is /alpine/keyboot-pkg.rsa.pub.
• apt: Release is GPG-signed (InRelease + Release.gpg) with the keyboot apt key; public half /debian/keyboot-archive-keyring.asc.
• gentoo: thin-manifest DIST entries carry per-distfile BLAKE2B + SHA512; the distfiles themselves are the raw channel artifacts (which are minisign-covered by the release SHA256SUMS).

The signing keys live as Protected CI variables (APK_SIGN_PRIVKEY, APT_GPG_KEY); back them up offline like the minisign key.

docs/decisions/

Architecture Decision Records. One file per decision, immutable once merged; supersede an old ADR with a new one rather than editing.

Filename format: NNNN-slug.md, zero-padded sequence, kebab-case slug. Example: 0001-bash-not-ash-in-init.md.

Each ADR documents: context, decision, alternatives considered, consequences. Keep them short — link to SPEC sections rather than restating them.

Index

• 0001 — MIT license.
• 0002 — BE re-unlock passphrase over SSH (dropbear in the BE). Superseded by 0003 (retained as the no-handoff fallback path).
• 0003 — BE re-unlock by handing the keyfile payload across kexec (single prompt). Reverses SPEC §16.1/§17.1 to choice (b).
• 0004 — OpenZFS version & pool-feature lifecycle (keyboot is the ZFS floor): compatibility= pin, gated zpool upgrade shim, boot-time feature check, ESP read-only by default. accepted.
• 0005 — Dataset layout & boot-environment scope: /var+pkgdb stay in the BE; BE-scoped vs persistent datasets; the two-homes rule. accepted.
• 0006 — Snapshot naming (@<UTC>Z-<LABEL>), cross-distro autosnapshot tool, and TZ surfacing (BE carries a TZ; keyboot bundles tzdata). accepted.
• 0007 — BE rollback model: three rungs (read-only / clone / destroy), pre-boot, recursive set, dependent-clone guard. accepted.
• 0008 — Packaging & delivery: signed curl|sh from packages.osterman.co first, then apk/deb/overlay; keyboot/keyboot-boot/keyboot-install split; ESP staged-not-committed on upgrade. accepted. (ci/install.sh) Boot-package post-install refined to auto-promote by 0009.
• 0009 — keyboot self-upgrade: A/B ESP slots with boot-count recovery (trial slot + keyboot.env grubenv; keyboot commits on unlock+import; auto-revert to the known-good slot on failure). Closes 0004’s keyboot-self-upgrade open boundary. accepted.
• 0010 — Multi-disk vdev topology grammar (data + special/log/cache/spare): count form mirror 2 or explicit mirror sn-a sn-b; per-disk partition role (data/spare bootable, aux crypt-only); warn+confirm on no-redundancy. accepted. (keyboot-install topology plan)
• 0011 — ZFS module params: keyboot’s unlock env caps zfs_arc_max to 512 MiB (uniform; it only RO-imports) with a keyboot.zfs_arc_max= override; the booted OS owns production ARC (Ansible). accepted. (init/lib/zfs.sh)
• 0012 — Boot-environment naming: <pool>/ROOT/<stem>-<YYYY-MM-DD-HHMM>Z (UTC, ADR 0006 stamp); stem defaults to distro (--be-label overrides); upgrade/rollback re-stamp the stem (no stacking); provenance in user props. accepted. Staged: helper + upgrade stem-strip now, install-default flip follow-up. (keyboot-install-os/lib/benaming.sh)
• 0013 — Automation keyslot: slot 2 is the canonical, independently-revocable automation passphrase for unattended/Ansible unlock (keyfile enroll-automation); Ansible feeds it to the existing dropbear askpass path. accepted. (keyfile enroll-automation, ansible/keyboot_unlock/)
• 0014 — Bootable memtest86+: ESP-staged; entry points keyboot.mode=memtest
  • stage-8 m (kexec, best-effort/unverified) + a GRUB 11_keyboot_memtest chainload seam (reliable). accepted. (init/lib/memtest.sh, keyboot-install install --memtest)
• 0015 — Ansible roles: keyboot_unlock (done) + keyboot_substrate (rescue→install) + keyboot_deploy (clone→boot-once→health-gate→promote, revert via boot-once not the role). Health gate = an operator command. accepted. (ansible/keyboot_{substrate,deploy}/)
• 0016 — Web UI for unlock + BE selection: additive/opt-in, never replaces dropbear. Trust = bake a self-signed TLS cert like the SSH host keys + import once (browser then auto-detects MITM), console-fingerprint as the bootstrap fallback (browsers can’t expose TLS channel-binding to JS, so no in-page MITM proof). Phase 1 = localhost-only + ssh -L (SSH-grade security); Phase 2 = network-exposed mTLS/passkey, opt-in. Rust+rustls, same unlock code. proposed / DRAFT.
• 0017 — RAM rescue (keyboot.mode=ramrescue): kexec a SEPARATE, fuller rescue image (real coreutils/parted/lvm/smartctl + zfs + cryptsetup + keyboot-install + the install-os orchestrator) entirely into RAM — keyboot’s “own env IS the rescue” vision, full and disk-independent. Built via ci/build-image.sh RESCUE_FULL=yes; staged on the ESP (like memtest) or netbooted; reuses the memtest kexec seam. Normal boot stays small; the big rescue loads on demand. accepted — BUILT (v-next; keyboot.mode=ramrescue, RESCUE_FULL=yes, the ESP KEYBOOT - RAM RESCUE entry + ci/stage-ramrescue.sh, qemu:ramrescue green).
• 0018 — Passwordless WebAuthn-PRF unlock (ADR 0016 Phase 3): touch a hardware authenticator in the browser → its PRF (FIDO2 hmac-secret) output is a reserved LUKS keyslot → the pool unlocks. No SSH, no typed passphrase, no cert files; phishing-resistant + hardware-backed. Boot-path verify = ring (P-256 + SHA-256) over a plain binary authenticatorData (no CBOR at boot); COSE pubkey extracted + baked at enrollment. Over the Phase-2 baked-TLS channel. proposed.
• 0019 — Off-disk rescue verification (TPM-free measured launch): on an unexpected reboot, verify the on-disk keyboot from an INDEPENDENTLY-sourced rescue (Hetzner’s off-disk rescue) against an OPERATOR-signed manifest (pinned key supplied out-of-band), then kexec the verified bytes directly (no reboot ⇒ no TOCTOU). The TPM-free substitute for measured boot when there’s no TPM/SB/ netboot-control; defeats file-rewrite tamper, residual = a fully-malicious provider rescue (bigger lift). Tool: tools/keyboot-verify. proposed/BUILT (verifier).

Backlog: the design Q&A from 2026-05-26 captured in SPEC §3 (Decision Log) should be backfilled here as individual ADRs so the rationale survives independent of SPEC revisions.

0001 — MIT license for keyboot

Context

The license was carried as TBD in Cargo.toml, the repo-layout listing, and SPEC §17.7, which blocked native packaging (pkg/{alpine,gentoo,debian} all need a declared license) and any distribution. SPEC §17.7 leaned MIT but deferred the choice while the repo stayed private.

keyboot links no GPL/CDDL code into its own artifact: it builds a vanilla LTS kernel and the OpenZFS module from upstream sources at build time, and the operator assembles the bootable image themselves. The keyboot source itself is host-side tooling plus init scripts — there is no derived-work entanglement that would force a copyleft license.

The west17m fleet otherwise uses BSL 1.1 (orcai, obooks) or proprietary all-rights-reserved (td, omessage, summer-school). keyboot differs: it is low-level boot infrastructure with no commercial-hosting angle to protect, and the permissive path matches SPEC §17.7’s “users assemble the artifact themselves” framing.

Decision

License keyboot under the MIT License. Copyright holder: Travis Osterman / west17m, 2026.

Alternatives considered

• BSL 1.1 — matches orcai/obooks, but its source-available/time-delayed model exists to protect a hosted product. keyboot has no such product; the extra friction buys nothing here.
• Proprietary / all-rights-reserved — matches td/omessage. Fine while private, but forecloses the reuse SPEC §17.7 anticipates and still needs a real text before any release.
• Apache-2.0 — permissive like MIT with an explicit patent grant. Heavier than warranted for a tool of this size; MIT is the fleet-simplest permissive option and is unambiguously CDDL/GPL-compatible at the boundary.

Consequences

• LICENSE (MIT) added at repo root; Cargo.toml sets license = "MIT".
• SPEC §17.7 and the §14 layout note updated from TBD to MIT.
• Packaging recipes can now declare a license.
• The MIT grant is one-way; relicensing later (e.g. to add a patent grant) requires a superseding ADR but no permission from downstream users.

0002 — BE re-unlock passphrase: dropbear in the BE, entered over SSH

Status: superseded by 0003 (2026-06-04)

Superseded. ADR 0003 adopts single-prompt key-handoff across kexec (SPEC §17.1 choice (b)). The dropbear-in-the-BE mechanism below is retained as the fallback path (keyboot.handoff=0, or whenever no handoff key is present — recovery, opt-out), so the implementation here stays live; it is just no longer the primary boot path.

Context

After keyboot kexecs into the boot environment, the BE’s own initramfs re-opens the LUKS-on-ZFS pool from scratch (SPEC §5 Stage 9 / §16.1 chose option (a): the BE has its own independent keyfile/passphrase flow — the keyfile/keys are deliberately NOT carried across the kexec boundary). That flow needs the keyfile-container passphrase, and SPEC §15.1 left how the BE obtains it open. The real-hardware bring-up (Hetzner, 2026-06-04) forced the question:

• keyboot be-unlock calls passphrase::read() → prompts on the console / reads stdin. On a headless box whose serial console is dead (common — see docs/hetzner-deploy.md), that blocks forever: the BE initramfs has no SSH, so there’s no remote way to answer.
• The only thing that made an unattended boot work was a baked /etc/keyboot/test/passphrase in the BE initramfs. But the BE initramfs lives unencrypted on the ESP/pool metapath, so a baked passphrase defeats the encryption — fine for CI (--test-passphrase-file), unacceptable for production.

The keyboot unlock environment already solves the equivalent problem well (SPEC §5 Stages 2–4): bring up the network, run dropbear, and let the operator enter the passphrase over SSH (or the console) via a shared FIFO. The BE re-unlock should reuse that model.

Decision

The BE initramfs gets the same network + dropbear + askpass path the keyboot unlock environment uses. No baked passphrase in production.

Mechanism (mirrors the unlock env, kept minimal for the BE):

1. be-init brings up the NIC (udhcpc, reusing the coldplug already added for AHCI) and starts dropbear with keyboot’s host keys + the operator’s authorized_keys, forced-command = an askpass that writes the entered passphrase to a FIFO.
2. be-init runs keyboot be-unlock … < <FIFO> — be-unlock’s existing passphrase::read() already reads a single line from stdin when there’s no TTY, so the SSH-entered passphrase flows straight in. The console prompt stays as a fallback (now usable thanks to the writable-console fix).
3. --test-passphrase-file continues to short-circuit this for CI only (read_passphrase() already checks /etc/keyboot/test/passphrase first).

The operator therefore enters the passphrase twice per boot — once for the unlock env, once for the BE. Accepted for v1 (see Consequences).

Alternatives considered

• Baked passphrase (status quo / CI): simplest, but the BE initramfs is unencrypted, so it stores the passphrase in clear. Test-only. Rejected for production.
• Console-only prompt: no new code, but blocks on headless / dead-serial boxes and offers no remote operator path. Rejected as the only mechanism; retained as a fallback.
• Keyring handoff across kexec (SPEC §15.1 option b): keyboot stashes the keyfile/passphrase in a kernel-keyring slot the BE reads → a single prompt per boot. Attractive UX, but it carries key material across the kexec boundary, which §16.1’s v1 invariant explicitly avoids; needs a focused security review. Deferred — revisit as the single-prompt follow-up.
• TPM-sealing the keyfile: fully unattended, but requires a TPM + a measured-boot story keyboot doesn’t have yet. Deferred.

Consequences

• Two prompts per boot (unlock env + BE). The keyring-handoff alternative above is the documented path to one prompt later.
• BE initramfs grows by dropbear + busybox net tooling + host/authorized keys (~a few hundred KB). Acceptable.
• The BE initramfs now needs networking — reuses the unlock env’s udhcpc/ network grammar; static/VLAN/bond support tracks the unlock-env milestone.
• install-os stops baking the passphrase for real installs; --authorized-keys (already added) supplies the BE’s operator key, and keyboot’s SSH host keys are reused. CI keeps --test-passphrase-file.
• Supersedes the “baked test passphrase” stopgap recorded in docs/hetzner-deploy.md; that doc’s open item is closed by this ADR.

Implementation note

Lands in tools/keyboot-install-os/alpine.sh (be-init + the keyboot.files mkinitfs feature: dropbear, the askpass, host/authorized keys) and is covered by extending ci/qemu/install-os-boot-test.sh to feed the passphrase over SSH to the BE instead of relying on the baked file. Debian/Gentoo plugins need the equivalent be-init wiring.

0003 — BE re-unlock: hand the keyfile payload across kexec (single prompt)

Status: accepted (2026-06-04) Supersedes: 0002 Reverses: SPEC §17.1 / §16.1 v1 choice — from (a) independent re-unlock to (b) hand-off across the kexec boundary.

Context

keyboot kexecs into each boot environment’s own kernel + initramfs (the ZBM model, SPEC §3) — that is what makes “pick between several root filesystems and kernels” possible. The cost of kexec is that the new kernel starts from zero: the dm-crypt mappings and the imported pool that keyboot set up live in the old kernel’s RAM and are gone the instant the BE kernel boots. So the BE’s initramfs must re-open the LUKS disks and re-import the pool from scratch.

ADR 0002 resolved this with (a): the BE re-derives everything itself — dropbear in the BE initramfs, operator enters the keyfile passphrase a second time over SSH. That is correct and shipped, but it forces the operator to type the same passphrase twice per boot. For a fleet of Debian/Gentoo/Alpine BEs that is the normal path, not an edge case, and the double prompt is a standing UX tax plus a second attack-relevant passphrase-entry point.

The operator has decided the destination is single-prompt: decrypt once in keyboot, then operate (rollback / sync / BE-pick) and boot any BE without re-entering the secret. That requires carrying the already-derived key across the kexec boundary — SPEC §17.1’s deferred choice (b). This ADR adopts it and pins the mechanism and the security envelope.

Threat model (why this is safe enough)

Anchor everything against the baseline: while the box is unlocked, the cleartext key is already in RAM — the 32-byte keyfile payload, and the per-disk LUKS volume keys which dm-crypt keeps resident in kernel memory the whole time the disks are open. Handoff does not introduce “key in RAM”; it adds one more transient copy for the window between kexec and the BE consuming it.

• Attacker who can read RAM while unlocked (cold-boot, DMA/PCILeech, malicious hypervisor): already wins in both designs — the volume keys are in kernel memory regardless, and on a hosted box the host can read guest RAM at will. Handoff’s marginal delta is ~nil.
• Attacker who can tamper at rest: the BE’s kernel+initramfs live inside the encrypted pool (keyboot reads them only after unlocking), so they cannot be pre-tampered without already holding the key — the attack eats its own tail. The one tamperable artifact is keyboot’s own initramfs on the unencrypted ESP; an attacker who owns that captures the passphrase at entry in both designs. So the independent-re-unlock invariant never protected against this attacker either.

Conclusion: handoff does not meaningfully widen attacker capability. The only genuine residual risk is leakage by our own handoff code — an auditable, bounded checklist (see Consequences), not an open-ended exposure. This is what §16.1’s caution was really guarding; we discharge it with the mechanism below plus the audit checklist, rather than by paying the double-prompt forever.

Decision

keyboot hands the 32-byte keyfile payload (not the passphrase, not the per-disk volume keys) to the BE, RAM-only, via an initramfs overlay on the kexec’d image. The BE uses it to open the disks directly and then shreds it.

Mechanism:

1. Capture (keyboot unlock, stage-5). While the keyfile container is open, keyboot unlock --emit-key <path> copies the 32-byte payload from /dev/mapper/keyboot-key to a tmpfs path (/run/keyboot/handoff.key, mode 0400) before closing the container. tmpfs = RAM; never hits disk.
2. Inject (stage-9, at kexec). stage-9 builds a tiny uncompressed cpio containing etc/keyboot/handoff.key and appends it to the BE initramfs image in tmpfs (the Linux initramfs loader concatenates archives; later entries overlay earlier — the same mechanism as early-microcode cpios). It kexec-loads the combined image, then shreds /run/keyboot/handoff.key and the staged copy. The payload never touches the kexec cmdline (which would be world-readable in /proc/cmdline forever).
3. Consume (BE, keyboot be-unlock). If /etc/keyboot/handoff.key exists, be-unlock uses it directly as the --key-file for the per-disk cryptsetup luksOpen (open_enrolled already takes a key-file path) — no passphrase, no container open, no dropbear — then shreds it (overwrite + unlink) immediately after the opens, before the pool import.
4. Hand the payload, not the passphrase. The reusable human secret (slot 0) never crosses kexec; only the per-install, rotatable machine key does. This is strictly less exposure of the human secret than re-prompting.

The transmitted unit is the keyfile payload, so rotating it (re-enrolling disks) invalidates any captured copy — unlike the passphrase.

Kill switch / fallback. keyboot.handoff=0 on the keyboot cmdline disables capture+inject; the BE then finds no handoff.key and falls back to ADR 0002’s independent re-unlock (dropbear-over-SSH / console). The 0002 path is retained in full as the recovery and opt-out path, not deleted.

Alternatives considered

• Independent re-unlock only (ADR 0002 / §17.1(a)): the status quo. Correct, but two prompts per boot forever. Demoted to fallback, not removed.
• Kernel-keyring handoff (as literally worded in SPEC §17.1(b)): infeasible as written — a kernel keyring is per-kernel and does not survive kexec (the BE boots a fresh kernel with an empty keyring). Rejected on mechanism; the initrd overlay achieves §17.1(b)’s intent by a means that actually crosses the boundary.
• Reserved-memory region preserved across kexec (memmap=/kho): works, but the key sits at a predictable physical address and relies on correct reserve+clear semantics; more moving parts and more leak surface than a tmpfs file inside a trusted initramfs. Deferred unless the initrd overlay proves insufficient.
• Hand the passphrase instead of the payload: simplest BE change, but moves the reusable human secret across the boundary and into a second consumer. Rejected — strictly worse than handing the rotatable payload.
• TPM-seal the keyfile (no handoff, fully unattended): the better long-term story for unattended boxes, but needs a TPM + measured-boot keyboot doesn’t have yet. Deferred; tracked separately.

Consequences

• One prompt per boot. The operator decrypts in keyboot; the BE re-unlock is silent. dropbear in the BE is no longer on the hot path (fallback only).
• Reverses a “do not relitigate” invariant. CLAUDE.md’s “never carry keyboot’s keyfile across the kexec boundary” and SPEC §16.1/§17.1 are updated to choice (b) with this ADR as the rationale of record.
• The handoff code is now safety-critical and must pass this audit checklist (CI/review gate):
  1. The payload is never written outside tmpfs and never placed on the kexec cmdline, in kernel log/dmesg, /proc, or a crash/kdump path.
  2. Both copies (/run/keyboot/handoff.key and the BE’s /etc/keyboot/handoff.key) are overwritten then unlinked after use — keyboot’s after kexec-load, the BE’s right after the disk opens.
  3. The overlay cpio holds only handoff.key, mode 0400, owner root.
  4. keyboot.handoff=0 cleanly yields the 0002 fallback (verified by a test).
  5. A wrong/blank/short handoff key fails closed to the fallback, never to an open pool with a wrong key.
• Exposure envelope: a single 32-byte buffer, in RAM, from kexec until be-unlock zeroes it — strictly less than what dm-crypt already keeps resident, and never the passphrase.
• CI: the existing baked---test-passphrase-file path still works (be-unlock prefers handoff > test > prompt). A new gate asserts (i) single-prompt boot via handoff and (ii) keyboot.handoff=0 falls back to the 0002 path.

Implementation note

• tools/keyboot/src/cli/unlock.rs: open_enrolled/try_open take a key-file path (already do, internally — generalize the /dev/mapper/<name> construction to an arbitrary path); add --emit-key <path> to the unlock command (copy 32 bytes from the open keyfile mapper before close).
• tools/keyboot/src/cli/be_unlock.rs: prefer /etc/keyboot/handoff.key → key-file path → open_enrolled, skipping passphrase+container; shred after. Order: handoff > baked test passphrase > stdin/prompt.
• init/stage-5-luks.sh: pass --emit-key /run/keyboot/handoff.key (unless keyboot.handoff=0).
• init/stage-9-kexec.sh: build the overlay cpio, append to the staged BE initrd, kexec the combined image, shred both key copies.
• tools/keyboot-install-os/alpine.sh: be-init runs be-unlock non-interactively when handoff.key is present; the dropbear/network/FIFO block (ADR 0002) becomes the no-handoff fallback. Debian/Gentoo BE wiring follows.

0004 — OpenZFS version & pool-feature lifecycle (keyboot is the ZFS floor)

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-04

Context

A pool that has enabled OpenZFS feature flags can only be imported by a ZFS implementation that understands them. If the recovery environment’s ZFS is older than the pool’s enabled features, the pool won’t import — the classic “recovery USB’s zfs is too old to import my pool” lockout, and a continent-away brick.

keyboot is the recovery environment. It carries its own zfs.ko + userland (SPEC §3, build via zfs/build.sh, bundled by ci/build-image.sh INCLUDE_ZFS=yes), independent of whatever the booted BE ships. Source/rolling distros make this sharp: Gentoo can keyword-unmask and emerge a newer sys-fs/zfs, and an operator can run zpool upgrade and silently push the pool past what keyboot can import. SPEC §3 already locks the doctrine — “ZFS coupling tracks upstream OpenZFS in lockstep; hosts upgrade keyboot before bumping pool features.” This ADR pins the concrete guards that make the doctrine hold.

Key fact that bounds the risk: upgrading the ZFS software is safe; only enabling pool features is dangerous. OpenZFS never auto-enables features, and no distro’s package upgrade runs zpool upgrade. Newer ZFS imports an older-feature pool fine. Lockout happens only when features get enabled beyond the recovery env’s capability.

This ADR protects keyboot’s recovery floor on two fronts: the ZFS feature lockout above, and the integrity of the ESP — the mdraid1 vfat that holds keyboot’s own kernel/initramfs + GRUB. The ESP is the only unencrypted boot artifact; if a stray write corrupts it, the recovery floor itself is gone, so it gets the same “protect by default, change only deliberately” treatment.

Decision

1. keyboot’s embedded OpenZFS is the floor. The pool’s enabled feature set must never exceed what keyboot can import. Release ordering: upgrade keyboot (its embedded zfs.ko/userland) before bumping pool features.
2. Pin compatibility= on every pool (default openzfs-2.1-linux or a keyboot-blessed set; the compat files are bundled into the rescue image and shipped in the BE). With the property set, zpool upgrade can only enable features within that set — so the pool structurally cannot outrun keyboot as long as keyboot ≥ the pinned baseline. This is the real safety net.
3. zpool upgrade is gated, never automatic. Ship a wrapper shim (/usr/local/sbin/zpool, earlier in PATH than the real binary) in the BE that intercepts only the upgrade subcommand: it checks the resulting feature set against keyboot’s capability / the pinned compatibility and refuses or warns, then execs the real zpool (located by absolute path, not PATH, to avoid recursion) for upgrade once cleared and for all other subcommands untouched. This is an accident guardrail, not a security control — explicit /sbin/zpool calls and automation bypass it; guard (2) is the structural guarantee.
4. Boot-time feature-vs-capability check. At import, keyboot compares the pool’s required features against its own zfs.ko and warns loudly (and declines a silent rw import that would worsen the gap) if the pool has outrun it — surfacing the problem at the next boot, while the pool is still importable, instead of at the moment it bricks.
5. Mount the ESP read-only by default. The ESP holds keyboot + GRUB — the recovery floor itself — and is not needed at runtime (verified: it isn’t even mounted in a normally-booted BE). So the BE mounts it ro by default; a gated keyboot/GRUB update is the only thing that remounts it rw, does its write, and drops back to ro. This blocks a stray dd, a distro grub-install/os-prober, or a fat-fingered /boot/efi write from corrupting the one artifact that must survive for recovery to work. (Pairs with the keyboot self-upgrade boundary — A/B keyboot images on the ESP so even a botched keyboot update is recoverable — now decided in ADR 0009, which carves out one bounded ESP-rw exception: keyboot’s trial-commit grubenv write at unlock+import.)

Alternatives considered

• Discipline only (no pin, no shim): relies on the operator remembering the ordering; one zpool upgrade bricks recovery. Rejected.
• Hard-block zpool upgrade entirely: too rigid — feature upgrades are legitimate after keyboot is upgraded. Rejected in favour of gate-with-check.
• Shim zfs to police zfs snapshot/other subcommands: breaks replication, sanoid, zfs send, and our own snapshot tool. The upgrade gate lives on zpool; the zfs side is warn-only at most (see ADR 0006).
• TPM/attested version negotiation: overkill for v1. Deferred.

Consequences

• A keyboot release that bumps OpenZFS must ship before fleet pools enable the new features — a documented release-ordering rule.
• The shim is best-effort (PATH order); the compatibility= pin is the binding guarantee. Both, plus the boot-time check, are defense-in-depth.
• keyboot’s import path (stage-6/7) gains the feature-vs-capability comparison + a loud warning channel.
• Gentoo unmask + emerge of newer ZFS is safe; the discipline is “bump keyboot’s embedded ZFS in the same step, and treat zpool upgrade as gated.”
• Ties into the upgrade story (SPEC §14); the gate belongs with keyboot-install upgrade.

0005 — Dataset layout & boot-environment scope

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-04

Context

/boot lives inside each BE on the encrypted pool (ADR 0003 / SPEC §3), so a BE rolls back / clones as a unit. That raises the layout question: which datasets travel with the BE (clone + roll back with it) and which are persistent (survive a BE switch/rollback)? SPEC §3 locks “discover any mountable dataset; no enforced layout” for boot-time discovery — this ADR sets the installer default + the consistency rules, not a hard requirement keyboot enforces at boot.

Two correctness constraints drive it:

• /usr ↔ package DB lockstep. /var/lib/{dpkg,rpm,portage} records what is installed; if it can roll back independently of /usr, a rolled-back system has a package DB that disagrees with its binaries. So the package DB must move with the BE.
• zfs rollback is per-dataset (ADR 0007): there is no atomic roll-back-these-five-datasets, so the fewer datasets that must move together, the simpler and safer rollback is.

Decision

BE root dataset = rpool/ROOT/<be>, and it INCLUDES /var (so the package DB rolls back in lockstep with /usr). Only binary-independent leaves are split out.

Two dataset classes, distinguished purely by where the dataset lives:

• BE-scoped — children of rpool/ROOT/<be> (e.g. rpool/ROOT/<be>/home): cloned and rolled back with the BE. This is the default for /home.
• Persistent — outside rpool/ROOT (e.g. rpool/data/home, rpool/var/log, rpool/data/...): survive BE switch/rollback. For state that must not revert, and for datasets that want their own recordsize/props.

Installer default profile (extends the SPEC §13 interactive sketch; today install-os creates only rpool/ROOT/<distro>):

• rpool/ROOT/<be> — root incl. /usr, /etc, /var (+ package DB)
• rpool/ROOT/<be>/home — /home, BE-scoped by default
• rpool/var/log — persistent, sync=always, exec=off
• (opt) rpool/data/home — persistent /home (the opt-out)
• (opt) rpool/var/lib/docker, rpool/data/... — persistent, tuned per use

recordsize and other props are set per dataset where it pays (e.g. a DB dataset at 16K vs the 128K default) — the reason to split a leaf at all.

The /var line: stays in the BE — package DB, and anything whose state must match the binaries. Split out only genuinely-independent leaves: /var/log, /var/cache, /var/lib/<db>, /var/lib/docker.

Refinement (2026-06-05, GL#17): parameterize; defer the default profile

The profile above is the recommended reference, not what the installer bakes in. Per operator decision, the layout is parameterized and the installer default stays minimal (rpool/ROOT/<distro> only, as today). Operators / Ansible spell out the leaves they want:

• install-os --dataset '<ds>[:<mnt>][:<props>]' (repeatable). The {BE} token expands to the BE dataset, so --dataset '{BE}/home:/home' is BE-scoped and --dataset 'rpool/var/log:/var/log:exec=off' is persistent. Props (incl. recordsize, compression, user props like com.sun:auto-snapshot=false) are passed through. Datasets are created after the BE is mounted, canmount=on so they auto-mount on first boot.
• Swap is a separate param (--swap, follow-up: per-distro fstab + the zvol-deadlock caveat).

This avoids the installer making fleet policy decisions; an opinionated default profile (turning the table above into a one-flag preset) is deferred to a later pass. So /home placement is no longer a baked default — it’s whichever --dataset form the operator passes (the two-homes rule, expressed explicitly).

Alternatives considered

• /var fully separate / persistent: breaks the /usr↔package-DB lockstep. Rejected — split only binary-independent leaves.
• /home always persistent (outside the BE) by default: safest against data loss, but a BE rollback then can’t give a consistent point-in-time (home stays current). Made the opt-out (rpool/data/home), not the default. The BE-scoped default is safe because the default rollback verb is non-destructive clone-and-boot (ADR 0007) — current home is never destroyed.
• Flat single dataset (no splits): no sub-system rollback, no per-dataset tuning. Rejected.

Consequences

• A BE-scoped /home forks per BE (an upgrade clone gets a CoW copy; work in the new BE isn’t in the old). Acceptable given clone-and-boot rollback + frequent snapshots (ADR 0006/0007); workstation operators who dislike it flip to rpool/data/home.
• install-os must grow a dataset profile (it currently creates only rpool/ROOT/<distro>). The profile is operator-overridable.
• The “what stays in /var vs splits out” rule is documented so rollbacks stay internally consistent.
• Boot-time discovery stays layout-agnostic (SPEC §3); this ADR governs the installer + the consistency rules only.

0006 — Snapshot naming convention, autosnapshot tool, and TZ surfacing

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-04

Context

keyboot’s pre-boot rollback UI (ADR 0007) needs snapshots that it can (a) read unambiguously during recovery, (b) sort chronologically for retention, (c) group into a consistent point-in-time across the recursive dataset set, and (d) show to a human. A cross-distro userland tool (Debian/Alpine/Gentoo) creates them; the name is an interface contract between that tool and keyboot.

Decision

Name: <dataset>@<YYYY-MM-DD-HHMM>Z-<LABEL> — UTC, with an explicit Z, 24-hour. LABEL ∈ {FREQUENT, HOURLY, DAILY, WEEKLY, MONTHLY} (the zfs-auto-snapshot model). Example: rpool/ROOT/gentoo@2026-06-04-1600Z-DAILY.

• UTC, not local. Local time is non-monotonic across DST (the fall-back hour repeats), which breaks lexical-sort retention and “keep newest N”, and is ambiguous exactly when you’re recovering. UTC names sort lexically == chronologically, making retention a trivial correct sort | head -n -N | xargs zfs destroy. The Z removes the “is 1600 UTC?” ambiguity at a glance.

Recursive + lockstep. Snapshots are taken zfs snapshot -r over the BE subtree so every dataset shares the name atomically (same txg) — a point-in-time is always complete. Retention/purge is recursive on matching names so no per-child snapshot is orphaned (which would leave a rollback with a hole). Per-LABEL retention counts (--keep=N), cron- or timer-driven.

The tool. A cross-distro POSIX-sh utility ships snap/list/purge verbs, driven by cron (Alpine busybox-crond / Gentoo cronie / Debian) or systemd timers where present. Installed by the install-os plugins (or a keyboot package). It is the blessed snapshot path. The zfs binary is not shimmed to force naming (that would break zfs send/sanoid/replication, ADR 0004); at most a warn-only hint on non-conforming manual snaps.

TZ surfacing (UTC on disk, local at display). Names stay UTC (source of truth, always shown); keyboot additionally renders operator-local time. Chain:

1. The BE carries a configured timezone (zone name, e.g. America/Chicago). → Requirement: install-os sets/captures a TZ on the BE.
2. install-os records the zone into keyboot’s /etc/keyboot/host.conf on the ESP (timezone=America/Chicago).
3. keyboot bundles tzdata (zoneinfo) in its initramfs.
4. The menu renders 2026-06-04 16:00Z (11:00 America/Chicago).

DST-correct local display needs the zone name + tzdata, not a fixed offset (a fixed offset is wrong half the year — the very confusion we’re removing).

Contract. The naming scheme is a versioned interface between the snapshotter and keyboot; changing it requires a superseding ADR.

Alternatives considered

• Local-time names: friendlier in isolation, but DST-ambiguous + non-monotonic → breaks retention and recovery reading. Rejected; local is a display-time convenience only.
• Fixed UTC-offset display (no tzdata): avoids bundling zoneinfo but is wrong across DST. Rejected; bundle tzdata.
• Shim zfs snapshot to enforce the convention: breaks replication / sanoid / zfs send / our own tool. Rejected (warn-only at most).

Consequences

• keyboot’s initramfs grows by tzdata (small, version-matched to the zoneinfo release).
• host.conf gains a timezone= field; install-os must write it; the BE must have a timezone set (a new install step / profile field).
• Non-conforming snapshots (replication, manual) still display by UTC mtime fallback; only conforming names group into the DAILY/WEEKLY point-in-time tiers in the menu.
• The recursive-set naming is what ADR 0007’s clone/rollback verbs key off — this ADR is load-bearing for those.

0007 — Boot-environment rollback model (three rungs, pre-boot)

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-04

Context

System rollbacks happen pre-boot, inside keyboot — the one place the datasets are idle (so zfs rollback/clone can’t fail on a busy mount) and a half-running system can’t fight you or be corrupted. This is why ZFSBootMenu rolls back from its own environment, not the booted OS.

zfs rollback is per-dataset and in-place, and its -r/-R flags only destroy newer snapshots/clones of the same dataset — they do not recurse into children (a common trap; zfs snapshot -r is recursive, rollback is not). So any whole-system operation must iterate the recursive snapshot set (ADR 0006 names), and rollback is not a single atomic command.

Decision

keyboot’s BE menu offers an escalating ladder, all operating on the recursive snapshot set (root + BE-scoped /var + BE-scoped /home at the same name; persistent rpool/data/* untouched):

1. Boot read-only — boot the snapshot RO; zero side effects. Inspect.
2. Rollback-clone (default, recommended) — clone the recursive snapshot set into a new BE and boot it. The original BE + timeline are untouched; reversible (destroy the clone). Adopt with keyboot-install be promote.
3. Rollback-destroy — zfs rollback -r each dataset in the set; discards all newer snapshots. Irreversible; explicit, unmistakable confirmation.

• Default system-rollback = consistent point-in-time across the BE-scoped set (ADR 0005). Mixing versions (e.g. old /var, current /home) is a deliberate recovery-shell, manual action — and is always safe because the only separately-rollbackable datasets are binary-independent (ADR 0005 keeps the package DB inside the BE).

Guards (the “many checks”)

• Rollback-destroy must detect dependent clones and REFUSE. zfs rollback -r fails if a newer snapshot has a clone; the only force-past is -R -f, which destroys those clones — which may be other boot environments. The script must never auto-escalate to -R -f; it refuses with a clear message (“snapshot X is the origin of BE gentoo-test; promote or destroy it first”).
• Clone → promote lifecycle. A clone is tethered to its origin snapshot; you can’t reclaim the old timeline until be promote flips the origin. Keep the clone if good (be promote), destroy it if not (original pristine).
• Visual unmistakability. The menu marks rung 3 as irreversible and separates it from the safe rungs.

Alternatives considered

• Only zfs rollback (destructive): loses the forward timeline and footguns on dependent clones. Kept as rung 3 only, guarded.
• Only clone-and-boot: can’t “just undo” or reclaim space immediately. Kept as rung 2 (default).
• In-OS rollback (from the booted system): dataset busy, can corrupt a running system. Rejected — pre-boot only.
• Recurse children automatically via some rollback -r magic: does not exist in ZFS. Rejected (script iterates the set).

Consequences

• The rollback engine is a checks-heavy script in keyboot’s recovery/menu stage (stage-8), using the full ZFS toolkit keyboot already ships.
• keyboot-install be {clone, promote, destroy} (SPEC §14) are the clone lifecycle primitives this builds on.
• Depends on ADR 0006’s recursive, consistently-named snapshot set.
• The BE-scoped-/home default (ADR 0005) is safe because rung 2 is the default — clone-and-boot never destroys current /home; only the explicit rung 3 could, and only with confirmation.

0008 — Packaging & delivery (curl|sh first, then native repos)

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-04

Context

SPEC §3/§10 commit keyboot to “CI-signed artifacts shipped as native packages (apk, ebuild, deb) in our own repos; ansible manages the repo definitions,” with a signed curl | sh bootstrap (§13.10) and a host at packages.osterman.co. None of that exists yet, and two new userland tools (keyboot-autosnap, keyboot-be-upgrade) plus the boot image now need a distribution story. This ADR fixes the artifact set, the package split, the channels, and the staging order so we can build the lowest-friction path first without re-architecting when the native repos land.

Decision

Artifact set & package split

Three packages from the build, split so the dangerous/heavy bits are isolated:

• keyboot — runtime keyboot binary + keyboot-autosnap + keyboot-be-upgrade + the grub.d/10_keyboot seam. Installs on the running host. (be-upgrade is host-safe: it clones, never wipes.)
• keyboot-boot — the kernel + initramfs (large, per-kernel). Its post-install stages to /var/lib/keyboot/staged/ first, then — per the superseding ADR 0009 (A/B ESP slots) — auto-promotes: writes the new keyboot into the inactive A/B slot and arms a boot-once trial, so a failure auto-reverts to the known-good slot. This answers the §10.3/§16.3 “package scripts touching /boot make operators nervous” without leaving the commit manual: the boot-counter is the safety net the original “stage-and-operator- commits” stance lacked. KEYBOOT_AUTOPROMOTE=0 opts back into staging-only (the operator then commits with keyboot install). ADR 0009 closes ADR 0004’s keyboot-self-upgrade boundary.
• keyboot-install — rescue/install only; never auto-installed on a running host (SPEC §3 binary split).

The two shell tools are arch-independent; keyboot/keyboot-install are x86_64-musl static (aarch64 later). Shell stays shell deliberately (zero-dep, runs in recovery, easy to audit); Rust stays for crypto/FFI.

Channels & staging order

Phase A — packages.osterman.co static host + signed curl | sh (build first). A small static site (nginx container behind the existing proxy; new /root/layout entry + proxy route) that CI publish rsyncs into:

packages.osterman.co/keyboot/<ver>/{keyboot-x86_64-musl, keyboot-install-x86_64-musl,
    keyboot-autosnap, keyboot-be-upgrade, keyboot-vmlinuz, keyboot-initramfs.img,
    manifest.json, SHA256SUMS, SHA256SUMS.minisig}
packages.osterman.co/keyboot/install            # the bootstrap (trailing minisig)
packages.osterman.co/keyboot/latest             # version pointer

• Signing: minisign for this channel (one keypair, trailing-block-friendly, tiny verifier). Pubkey baked into the keyboot image + documented for pre-fetch. The bootstrap verifies before exec / before install.
• ci/install.sh is the bootstrap: detect arch/distro → fetch latest (or a pinned KEYBOOT_VERSION) → verify SHA256SUMS.minisig → install keyboot-autosnap+cron, keyboot-be-upgrade, and the keyboot binary; --rescue also pulls keyboot-install + the boot image. Idempotent.

Phase B — native repos (§10.3 end state).

• Alpine abuild → signed .apk + APKINDEX at packages.osterman.co/alpine/ <branch>/; pubkey to /etc/apk/keys/. install-os can pre-add the repo so BEs auto-update.
• Debian .deb → reprepro/aptly (InRelease GPG) at …/debian/<codename>/.
• Gentoo ebuild overlay git repo git.osterman.co/west17m/keyboot-overlay (+ Manifest GPG; optional binhost).
• Ansible role adds the repo def + key per host.

Signing identities

minisign for curl|sh; native repos use their own (apk RSA, apt GPG, gentoo Manifest GPG) — all keys in GitLab CI variables (§10.2). Open: unify GPG across apt+gentoo.

Version ↔ ZFS-floor coupling

keyboot/keyboot-boot package versions encode the OpenZFS feature floor (ADR 0004), so the package manager can refuse a pool-feature bump until keyboot is upgraded. manifest.json carries the zfs version + compat baseline.

Alternatives considered

• Native repos first: higher friction to stand up (per-distro signing + index tooling) before anything is installable. curl|sh from a static host is the faster MVP and serves every distro at once; native repos layer on.
• GitLab Package Registry only: fine for generic binary pulls, awkward for the predictable repo-index paths apk/apt expect. Use a static host.
• Rewrite the shell tools in Rust for one package format: loses the audit/recovery-friendliness; the tools must run in minimal envs. Rejected.
• Auto-commit the ESP on package upgrade: originally rejected as the operator-nervous footgun (a bad write bricks the boot stage). ADR 0009 revisits this: with A/B ESP slots + a boot-once trial that auto-reverts, the staged image can be auto-promoted safely — the failure mode the footgun worried about is now caught by the boot-counter. KEYBOOT_AUTOPROMOTE=0 keeps the manual stage-then-commit for the conservative operator.

Consequences

• New infra: packages.osterman.co (static + minisign) must be stood up — a /root/layout entry + proxy route + a CI publish stage. Until then, ci/install.sh can point at a KEYBOOT_PKG_BASE override (e.g. a GitLab artifact URL) so the bootstrap is testable before the host exists.
• The build pipeline (§10.1) gains package (apk/deb/ebuild + tarballs) and publish stages after sign.
• ESP integrity: keyboot-boot’s post-install is now governed by ADR 0009 (A/B ESP slots, auto-promote with boot-once auto-revert) — no longer an open boundary.
• install-os already installs the BE tools directly (build-image staging); the native packages are the update path for already-running hosts.

0009 — keyboot self-upgrade: A/B ESP slots with boot-count recovery

Status: accepted (2026-06-10) — install side implemented (GL#29: slots + grubenv + keyboot-install install --slot / keyboot list|promote|rollback, two-entry snippet + A/B ESP grub.cfg, single-slot adoption; ESP sizing GL#32). Runtime trial-commit implemented too (GL#30: keyboot slot-commit + init/lib/slot.sh, called by stage-6 after unlock+import; fail-safe no-commit on any error). The auto-revert QEMU gate is authored (GL#31: ci/qemu/ab-revert-test.sh, CI job qemu:ab-revert — broken trial auto-reverts, good trial self-commits).

Context

keyboot-be-upgrade (ADR 0007 neighbourhood) upgrades the booted OS. Nothing yet upgrades keyboot itself — the boot stage’s kernel + initramfs + the lockstep zfs.ko baked into that initramfs (ADR 0004). Today the GRUB seam (tools/keyboot/src/install/host.rs) places a single copy at <ESP>/keyboot-vmlinuz + keyboot-initramfs.img and emits one menuentry 'keyboot'. install_file is atomic per file (write .kbtmp → rename), but there is one slot: a keyboot update overwrites the only copy, and you cannot fix a boot stage that will not boot. A bad kernel-config change, a zfs.ko that won’t import the pool, a broken initramfs — any of these bricks the host until someone shows up with the USB rescue image (GL#1).

keyboot is pre-OS, so “did it work?” has a precise, self-contained answer: did it open the LUKS disks and import the pool? That makes boot-count A/B recovery — the systemd-boot / Fedora boot counting / greenboot pattern, but for the boot stage — the right model. ADR 0004 already flagged “keyboot self-upgrade … ideally A/B keyboot” as an open boundary; this ADR closes it.

Decision

Two ESP slots

keyboot lives in two slots, A and B, on the keyboot-owned mdraid1 vfat ESP:

<ESP>/keyboot/A/{vmlinuz, initramfs.img}
<ESP>/keyboot/B/{vmlinuz, initramfs.img}
<ESP>/keyboot/keyboot.env          # GRUB grubenv: the A/B state

A slot is a {kernel, initramfs} pair; the lockstep zfs.ko and the host keyfile container are inside that slot’s initramfs (so staging a slot re-injects the keyfile, exactly as the single-slot install does today). The slots live on the ESP, not the encrypted pool, because GRUB must read keyboot before unlock.

grubenv state (three keys)

A GRUB grubenv (keyboot.env, the fixed 1024-byte block GRUB reads/writes in place) holds the whole state machine:

• keyboot_slot — the known-good slot GRUB boots by default (A|B).
• keyboot_try — a slot on trial (A|B|unset).
• keyboot_try_left — remaining boot attempts for the trial (starts at 1).

Two menuentries (keyboot-A, keyboot-B) are always emitted, each tagged with a keyboot.slot=A|B cmdline token, so either slot is reachable by hand from the GRUB menu as a human backstop. Which one boots by default is driven by grubenv, evaluated in the 10_keyboot snippet:

if [ -n "$keyboot_try" ] && [ "$keyboot_try_left" -gt 0 ]; then
    decr keyboot_try_left; save_env; default=keyboot-$keyboot_try
else
    default=keyboot-$keyboot_slot
fi

Decrement-before-boot is the load-bearing trick: a trial that hangs or panics and never returns has already spent its attempt, so the next reset falls through to keyboot_slot (the known-good) with no human in the loop.

“Good” = unlock + pool import (pre-kexec)

When the trial keyboot reaches its core milestone — LUKS open + pool imported, i.e. its own job is done — and only then, it commits the trial: briefly mount the ESP rw, write keyboot_slot=<trial>, clear keyboot_try / keyboot_try_left, fsync, return the ESP to ro, then kexec. The trial slot is now the default. A trial that never reaches unlock never commits, so recovery is automatic (above).

This is a deliberate, bounded carve-out from ADR 0004’s ESP-read-only-by- default: a single grubenv write, only on a trial boot, mount-rw → write → sync → ro. Documented here so it is not a silent erosion of that invariant.

Layering boundary (explicit): “good” means keyboot works, not that the OS works. A keyboot that unlocks fine but kexecs into a broken BE is not caught here — that is the BE’s own A/B rollback (ADR 0007), a different layer. keyboot self-upgrade protects against a broken keyboot; BE rollback protects against a broken OS. They compose; neither substitutes for the other.

Auto-promote on package upgrade

The keyboot-boot package post-install (ADR 0008) auto-promotes: stage the new keyboot into the inactive slot (never the running/known-good one), then arm the trial (keyboot_try=<inactive>, keyboot_try_left=1) and let GRUB boot it next reset. This is safe precisely because the boot-counter auto-reverts on failure — the safety net ADR 0008’s “stage-and-operator-commits” stance lacked. KEYBOOT_AUTOPROMOTE=0 opts out to stage-only (0008’s original manual-commit path remains for the conservative operator). This refines ADR 0008 for the boot package: staging still lands in /var/lib/keyboot/staged/ first, but the commit step is now an A/B promote, not an ESP overwrite.

keyboot-install surface (deferred — this ADR is design only)

The planned verbs, to be implemented in the follow-up issues below:

• keyboot-install install --slot {A|B|inactive} [--promote] — write a slot, optionally arm the trial. (Evolves today’s single-slot install_host.)
• keyboot-install keyboot list — slots, versions, which is good/trial, counter.
• keyboot-install keyboot promote <slot> / rollback — arm a trial / pin keyboot_slot back to the current good (the panic button).
• The 10_keyboot snippet grows from one entry to the two-entry + grubenv form above; keyboot gains slot-awareness (keyboot.slot=) and the commit step (ship grub-editenv, or a ~30-line grubenv writer, in the initramfs).

Alternatives considered

• Manual two-entry only — both slots in the menu, operator flips the default by hand. Rejected as the primary mechanism (recovery needs a console; fleet- hostile), but kept as the backstop (the entries are always present).
• GRUB fallback only — default=new, one-shot fallback to old. Rejected: catches a hard panic but not a keyboot that boots-but-won’t-unlock, which would loop. The boot-counter + success-commit closes that gap.
• Full-BE-boot success signal — mark good only after the OS comes up and signals back. Rejected for now: needs an OS→ESP feedback path and a be-tool; heavier, and conflates the keyboot and OS layers. unlock+import is the minimal self-contained signal that keyboot actually did its job.
• TPM-measured / sealed keyboot — out of scope; a future ADR.

Consequences

• ESP capacity doubles. Two full keyboot images, and the firmware-laden initramfs is hundreds of MiB (GL#20 / BACKLOG B5). The KEYBOOT-ESP partition template must be sized for ≥2× an initramfs; this couples self-upgrade to the firmware-slimming work. Note the dependency at install time.
• Both-slots-bad is still possible but narrowed: we never stage over the currently-good slot, so a single bad upgrade is always recoverable to the prior good. Two successive bad upgrades (the second staged before the first was proven) could strand you → mitigation: the always-present manual menu entries and the USB rescue image (GL#1) remain the ultimate backstop.
• grubenv tooling. keyboot’s commit step needs to write a grubenv on the ESP — ship grub-editenv (or a tiny fixed-block writer) in the keyboot initramfs; a missing writer must fail safe (don’t commit → auto-revert), never brick.
• The single-slot install_host and its 10_keyboot snippet are superseded by the two-slot form; the migration (first A/B install adopts the existing single copy as slot A) is a follow-up detail.

Follow-ups (file as issues)

1. Implement install-side A/B: keyboot-install install --slot/--promote, keyboot keyboot list|promote|rollback, the two-entry grubenv-driven 10_keyboot snippet, single-slot→A/B migration. (The bulk.)
2. keyboot trial-commit step: slot-awareness + the bounded ESP-rw grubenv write at unlock+import; ship a grubenv writer in the initramfs; fail-safe on its absence.
3. QEMU gate: stage a deliberately-broken slot, prove the box auto-reverts to the known-good slot (the headline guarantee).
4. ESP partition sizing for two slots (couples GL#20 firmware slim).
5. Reconcile ADR 0008’s keyboot-boot post-install to the auto-promote flow (+ the KEYBOOT_AUTOPROMOTE=0 opt-out).

0010 — Multi-disk vdev topology grammar (data + aux vdevs)

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-05

Context

install-os picks one of three fixed pool layouts from disk count (tools/keyboot-install-os/lib/vdev.sh: 1→single, 2→mirror, 3+→raidz1), each a single top-level vdev. That can’t express what real boxes need: an 8-disk host should be describable as RAID10 (4×2-mirror), raidz2(8), 2×raidz1(4), etc., and many hosts want a special (metadata) vdev, an SLOG, an L2ARC, or a hot spare. SPEC §7/§13.13 endorse arbitrary topology + aux vdevs; this ADR fixes the grammar, the disk→role mapping (which drives partitioning), and the validation so the installer can build any layout over the sn-<serial> crypt mappers without re-architecting later.

Partitioning today is uniform (every pool disk: 1 MiB bios-boot + 512 MiB ESP member + LUKS payload; the ESP is an mdraid1 across all disks, partition.rs). Aux devices break that assumption — an L2ARC SSD is not a bootable data disk — so the topology must decide each disk’s partition role, not just the zpool string.

Decision

Grammar

A pool is a sequence of data vdev groups plus optional aux vdevs:

--topology "<group> | <group> | ..."     # data vdevs (pipe-separated)
--special  "<group>"                      # metadata/small-blocks vdev (one group)
--log      "<group>"                      # SLOG
--cache    "<devspec>"                    # L2ARC (single-disk vdevs)
--spare    "<devspec>"                    # hot spares

Each group is <type> <devspec> where type ∈ {stripe, mirror, raidz1, raidz2, raidz3} and <devspec> is one of two forms (both supported, not mixed within a single invocation):

• Count form — an integer: mirror 2, raidz2 8. Disks are consumed from the resolved sn-<serial> list in discovery order, group by group.
• Explicit form — a device list: mirror sn-a sn-b. Names must resolve to members of the discovered set; you control exactly which physical disk lands in which vdev (fault-domain placement).

stripe N (or stripe <devs>) expands to N separate single-disk top-level vdevs (a ZFS stripe — no redundancy). Count and explicit forms may not be mixed in one --topology (error), to keep disk accounting unambiguous.

Examples:

--topology "mirror 2 | mirror 2"                 # 4-disk RAID10
--topology "raidz2 8" --special "mirror 2" --log "mirror 2" --cache "sn-z"
--topology "raidz1 sn-a sn-b sn-c | raidz1 sn-d sn-e sn-f"

The legacy --template single|mirror|raidz1 stays as sugar (maps to a single group); count-from-disk-count remains the default when neither is given.

Disk → partition role (this is the new partitioning rule)

Each discovered disk gets exactly one role, which fixes its partition layout:

Rationale: a spare may be auto-promoted to replace a data disk, so it must be partitioned identically (incl. an ESP member, to preserve boot redundancy). special/log/cache hold pool data/metadata and so are encrypted (crypt) — but they are never booted, so they carry no bios-boot/ESP. The ESP mdraid1 therefore spans data + spare disks only (boot redundancy follows the bootable disks, not the cache SSD). Every aux device is still opened as sn-<serial> and keyed by the same keyfile payload — encryption is uniform even though partitioning is not.

zpool expansion

zpool create -f -o ashift=12 -o compatibility=openzfs-2.1-linux <pool> \
    <data groups...> \
    [special <special group>] [log <log group>] \
    [cache <cache devs>] [spare <spare devs>]

over the resolved /dev/mapper/sn-<serial> crypt paths.

Validation

• Per-vdev minimums (ZFS hard floors): mirror ≥2, raidz1 ≥2, raidz2 ≥3, raidz3 ≥4. Warn (not error) below the recommended width (raidz1 3, raidz2 4, raidz3 5).
• Disk accounting: every discovered disk is assigned exactly once across all data + aux vdevs. Unassigned or double-assigned disks are an error. (Count form: the counts must sum to the disk total; explicit form: a bijection.)
• No-redundancy guard (operator decision: warn + require --confirm): any stripe, single-disk data vdev, or otherwise non-redundant top-level data vdev prints a loud warning in the plan; the existing --confirm is the gate (no extra flag). A non-redundant --special vdev gets an extra-loud warning — losing it loses the whole pool, not just redundancy — but still proceeds under --confirm per the uniform policy.
• At least one data vdev is required; aux-only is an error.

Implementation locus

The grammar parser + validator + role mapping live in Rust (keyboot-install), exposed as keyboot-install topology plan --json and consumed by the shell orchestrator — mirroring the existing disk-discovery/partition split (Rust is the single source of truth, the orchestrator shells to it). partition.rs gains a per-disk role so it emits the right partitions and the ESP array spans only data+spare. lib/vdev.sh’s three-template path is reimplemented in terms of the grammar (kept as sugar).

Alternatives considered

• Expanded named menu (raid10/raidz2/raidz3 templates, no grammar) — rejected: can’t express 3×raidz1(4) or aux vdevs; just postpones the grammar.
• Count form only — rejected: loses fault-domain control (which disk in which mirror), which matters on multi-controller/multi-enclosure boxes.
• Aux vdevs as a follow-up — considered; operator chose to include them now, accepting the partitioning rework (the crypt-only role) this pass.
• Grammar in shell (vdev.sh) — rejected: two device forms × multiple groups × aux × disk-accounting is too error-prone for bash; Rust is testable and already owns partitioning.

Consequences

• Partitioning is no longer uniform: partition.rs and the partition --json contract grow a per-disk role; the ESP member set is a subset of disks. The install-os boot path (CI-verified) must be re-proven with a multi-disk + aux QEMU scenario before this is trusted on real hardware.
• Encryption stays uniform: every disk (data/spare/special/log/cache) is LUKS-keyed by the keyfile payload, so an aux SSD never holds plaintext pool data.
• Special-vdev redundancy is the sharp edge: a non-redundant special vdev is a pool-loss footgun; we warn extra-loud but honour the warn+confirm policy. Operators wanting a hard stop can be given --refuse-no-redundancy later.
• L2ARC encryption: cache is crypt-backed; ZFS L2ARC of an encrypted dataset is itself encrypted, so this is belt-and-suspenders but consistent and costs nothing operationally.
• Follow-ups: a multi-disk+aux QEMU install gate; --refuse-no-redundancy opt; per-vdev ashift / --special-small-blocks tuning (deferred).

0011 — ZFS module params: keyboot-env ARC cap vs OS-owned tuning

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-05

Context

There are two distinct zfs.ko load contexts on a keyboot host, and they have opposite tuning needs:

• keyboot’s unlock env — its own lockstep zfs.ko (ADR 0004). It opens LUKS, RO-imports the pool (stage-6-import.sh: zpool import … -o readonly=on), reads BE metadata, and kexecs. It touches almost no data, so it needs almost no ARC. Today it sets no zfs_arc_max, so the default (~50% of RAM) applies — on a big-RAM box that reserves a lot of RAM during a step that reads kilobytes.
• the booted OS/BE — the distro’s zfs.ko, running the real workload. Its ARC is a per-host, workload-dependent production tuning decision.

These must not share a policy: a fleet-uniform small cap is right for the first, wrong for the second.

Decision

keyboot’s unlock env caps zfs_arc_max to 512 MiB, uniform across the fleet. stage-1-early.sh writes /etc/modprobe.d/keyboot-zfs.conf (options zfs zfs_arc_max=<bytes>) before zfs.ko loads — so the cap applies whether zfs is in /etc/keyboot/modules or autoloaded by zpool import (the kernel’s autoload runs userspace modprobe, which reads modprobe.d). The value is resolved by the pure helper keyboot_zfs_arc_max_bytes (init/lib/zfs.sh): empty → 512 MiB; accepts a raw byte count or a K/M/G suffix; malformed → warn-to-default.

Override: keyboot.zfs_arc_max=<v> on the kernel cmdline (e.g. a low-RAM box, or a recovery shell doing a scrub that wants more ARC). Parameterized with a sensible default — the operator can override, but doesn’t have to.

The booted OS owns production ARC. keyboot never writes the installed OS’s /etc/modprobe.d/zfs.conf. Production ARC is per-host and workload- dependent → Ansible-managed (a baseline + per-host override). install-os bakes no OS-level ARC default; the installed system uses the distro/ZFS default until Ansible tunes it. (Consistent with “parameterize choices, defer opinionated defaults”.)

The zfs.ko binaries are already independent per context (keyboot’s lockstep build vs the distro’s) — this ADR is only about params, and confirms they are owned separately too: keyboot-uniform-and-small here, OS-and-Ansible there. Not “per-OS params inside keyboot.”

Alternatives considered

• A fraction of RAM (e.g. 1/16) for the keyboot cap — rejected: needs RAM detection in the initramfs and is less predictable/uniform; a metadata-only RO import doesn’t benefit from scaling ARC with host RAM.
• No cap (status quo) — rejected: ARC balloons toward ~50% RAM during the unlock on big-RAM hosts for no benefit.
• keyboot setting an OS-level ARC default — rejected: production ARC is fleet policy; baking it into the installer is exactly what Ansible should own.
• 256 MiB cap — considered; 512 MiB chosen for a little headroom on very large pools’ metadata / a recovery-shell scrub, still tiny + uniform.

Consequences

• keyboot’s unlock-env RAM footprint is bounded + uniform regardless of host RAM; recovery-shell heavy ops override via keyboot.zfs_arc_max=.
• The cap is set before any zfs load path, so it can’t be missed by autoload.
• No change to the booted OS — its ARC is whatever the distro/Ansible sets.
• init/lib/zfs.sh is the home for future keyboot-env module params (it already pairs with the cmdline override plumbing).

0012 — Boot-environment naming convention (stem + UTC stamp)

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-05

Context

The install default names the BE <pool>/ROOT/<distro> (e.g. rpool/ROOT/gentoo), but keyboot-be-upgrade already names its clone <be>-<UTCstamp> (date -u +%Y-%m-%d-%H%MZ, matching ADR 0006’s snapshot stamp) → rpool/ROOT/gentoo-2026-06-05-1823Z. So after the first upgrade a host has a mixed convention: gentoo next to gentoo-2026-…Z.

Worse, the upgrade tool derives the clone leaf from ${BE##*/} — the full leaf, including any existing stamp — so a second upgrade produces gentoo-2026-06-05-1823Z-2026-07-01-0900Z: the stamp stacks. (Latent today because the install BE is unstamped; it bites on the second upgrade.)

Decision

Canonical BE name: <pool>/ROOT/<stem>-<YYYY-MM-DD-HHMM>Z — UTC with an explicit Z, identical to ADR 0006’s snapshot stamp (so BE and snapshot times read the same, and sort within a stem is chronological).

• <stem> is the stable human anchor, carried across the BE’s whole upgrade lineage. It defaults to the distro (gentoo); install-os --be-label <stem> overrides it (e.g. gentoo-prod / gentoo-test) so one host can run differently-purposed lineages. (Parameterize with a sensible default.)
• Re-stamp the stem, never append. Upgrade/rollback derive the stem by stripping a trailing -<YYYY-MM-DD-HHMM>Z from the current leaf, then add a fresh stamp: gentoo-<old>Z → gentoo-<new>Z, not …-<old>Z-<new>Z.
• --be <full-dataset> still overrides everything (no stamp added) for operators who want a plain rpool/ROOT/gentoo.
• Provenance lives in ZFS user properties, not the name: keyboot:origin= install|upgrade|rollback, keyboot:created=<UTC>, and for rollbacks keyboot:rolled-back-from=<snapshot>. Names stay short + sortable; keyboot-install be list surfaces origin/when/active. “Which is current” is bootfs + keyboot’s picker — the name never encodes it.

The convention is specified once here and implemented by a small pure helper pair (keyboot_be_stem / keyboot_be_make_name, lib/benaming.sh, bats-tested). Because the install orchestrator (rescue env) and the keyboot-be-* tools (booted BE) run in different runtime contexts that can’t always share a sourced lib, the stem-strip is mirrored inline in the tools — this ADR is the canonical spec they track.

Alternatives considered

• Keep <distro> for the install BE, stamp only clones — rejected: that’s the current mixed convention; the first BE stays the special-case “no stamp”.
• Stem = always the distro (no custom label) — rejected as too rigid; a host may want app-prod / app-test lineages. The label is optional (defaults to distro), so simple hosts are unaffected.
• Provenance in the name (-upgrade / -rb-<tag> suffixes) — rejected: bloats names and breaks chronological sort; user properties are queryable and keep the name a clean <stem>-<UTC>Z.

Consequences

• Fixes the stamp-stacking bug in keyboot-be-upgrade (re-stamp the stem).
• Staged rollout (avoids one risky change to the CI-verified install path):
  1. done (b42d46f) — locked the convention, shipped lib/benaming.sh + tests, fixed the upgrade stem-strip.
  2. done (GL#36) — flipped the install-os default BE to <stem>-<UTC>Z, added --be-label, aligned keyboot-be-rollback’s clone naming to <stem>-<UTC>Z + keyboot:origin/keyboot:rolled-back-from provenance props. KEYBOOT_BE_STAMP pins the stamp for deterministic installs/tests; the QEMU harnesses already pin --be explicitly, so the default flip doesn’t ripple into them.
• keyboot-install be list becomes the place “which BE, when, how made, which is active” is answered (origin/created/bootfs), since the name no longer carries provenance.

0013 — Automation keyslot (slot 2) for unattended/Ansible unlock

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-05

Context

The keyfile container has slot 0 (daily passphrase), slot 1 (recovery), and slots 2-7 reserved (SPEC §7). Reboots are rare and usually attended, so fully hands-off TPM unlock isn’t urgent — but upgrade/deploy cycles want a scriptable unlock that does not require putting the human daily passphrase into an Ansible vault. That secret must be independently revocable: a leaked automation credential should be rotatable without touching slot 0 or slot 1.

Decision

Slot 2 is the automation keyslot, fleet-wide. A dedicated automation passphrase is enrolled there and used only by automation.

• Enroll: keyboot-install keyfile enroll-automation <file> — sugar that adds a passphrase to slot 2 (same mechanism as add-recovery, authorized by an existing daily/recovery passphrase). The convention lives in the tool, not just the docs.
• Rotate / revoke (GL#14): rotate in place with keyfile passwd --slot 2; move/retire with keyfile rekey --retire-slot 2 --new-slot 3. The 32-byte payload is untouched, so data-disk enrollment survives — only the automation passphrase changes. Slot 0 (daily) and slot 1 (recovery) are unaffected by any automation-slot rotation.
• Unlock flow (no new mechanism): Ansible (vaulted passphrase) connects to keyboot’s pre-unlock dropbear and feeds the slot-2 passphrase to the authorized-keys forced command /sbin/keyboot-askpass, which reads it from stdin and submits it to stage-4’s FIFO — exactly the existing SSH-unlock path (stage-3-ssh.sh / stage-4-passphrase.sh). A sample role lives at ansible/keyboot_unlock/.
• Single canonical slot, not a range. LUKS1 has no per-slot labels, so a range (2-7) would need external inventory to track “which slot is what”. One canonical slot keeps the vault entry and rotation trivial; slots 3-7 stay reserved (e.g. as the target of a rekey rotation, or future use).

TPM-sealing (truly hands-off, measured boot) remains the deferred path.

Alternatives considered

• Reserved range 2-7 for multiple automation identities — rejected as the default: needs out-of-band tracking (no LUKS1 labels) for little gain on a fleet where one automation secret per host is the norm. rekey already lets you stage a rotation into slot 3 and kill slot 2 when needed.
• Reuse the daily passphrase for automation — rejected: puts the human secret in a vault and couples revocation (rotating automation would force a daily-passphrase change).
• TPM/measured-boot unlock — the right hands-off answer eventually; deferred (needs sealing policy + PCR design; out of scope here).

Consequences

• A leaked automation credential → passwd --slot 2 (or rekey) and you’re done; daily/recovery untouched.
• No change to the unlock runtime — automation reuses the proven SSH askpass path; the only new code is the enroll-automation sugar + the sample role.
• Pairs with the future keyboot_deploy role (GL#24): the deploy loop unlocks via this slot, then clones/boots-next/health-gates.

0014 — Bootable memtest86+ (ESP-staged, kexec + GRUB seam)

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-05

Context

SPEC §3 ships a userland memtester in the recovery shell, but it only tests kernel-allocatable RAM. A bootable memtest86+ tests all physical RAM, and it’s the one diagnostic you want before any disk/crypto/pool exists — so it should run the moment keyboot’s env is up, disk-independent. The standout use: trigger it over SSH on a headless box (reboot → keyboot comes up networked, pre-unlock → SSH in → memtest), watching progress over IPMI serial-over-LAN.

memtest86+ ships two relevant artifacts: an EFI app (memtest.efi, for a firmware/GRUB chainload) and a kexec-loadable image (memtest.bin, bzImage-compatible). They suit two different launch paths.

Decision

Stage memtest on the keyboot-owned ESP at /EFI/keyboot/memtest.{efi,bin} (read pre-unlock — the ESP is already mounted ro at /esp in stage-1 for the boot-once marker). Keeps the keyboot initramfs lean (relevant to the A/B ESP slots, ADR 0009) and lets memtest be updated per-host without rebuilding the initramfs. Three entry points:

1. keyboot.mode=memtest — keyboot runs stage-1 (console + udev + ESP mount), then kexecs memtest.bin instead of unlocking. Scriptable + SSH/IPMI-friendly; the headless killer feature.
2. stage-8 menu action m — for an operator already at the console/SSH during a normal boot, an m) memtest action kexecs it.
3. GRUB 11_keyboot_memtest seam — a grub.d snippet emitting a menuentry 'memtest86+' that chainloads /EFI/keyboot/memtest.efi directly. No kexec, so it’s the reliable fallback that works before keyboot even starts.

Serial by default: the kexec append carries console=ttyS0,115200 so memtest86+ 6.x’s serial output reaches IPMI SoL on a headless box.

The kexec form is best-effort/unverified

kexec -l memtest.bin --append=… ; kexec -e is the documented memtest86+ 6.x kexec path, but the exact loader behaviour (bzImage vs multiboot, EFI vs BIOS image) is only confirmable on real hardware (QEMU/OVMF is not a faithful proxy). Per the operator decision we ship it now as best-effort with a clear log marker; if a box can’t kexec it, the GRUB seam (chainload) is the reliable path and is unaffected. Real-hardware verification + correction is a follow-up.

Sourcing the binaries

memtest86+ is GPLv2 but a build artifact, not vendored here. Staging is conditional on KEYBOOT_MEMTEST_DIR (a dir holding memtest.efi/memtest.bin) at build/install time; absent it, the memtest entries are simply not emitted (graceful). Docs point at memtest.org for the images.

Alternatives considered

• Bundle in the initramfs — always-available even with no readable ESP, but costs initramfs size (× the A/B slots) for a rarely-used tool; the ESP is already mounted pre-unlock, so ESP-staging gets ~the same availability cheaper.
• Netboot/iPXE only — zero local footprint, but needs the network + a netboot server; doesn’t cover “diagnose this box right now, no net”. (The iPXE menu remains a good addition — deferred, not chosen this pass.)
• kexec the .efi — rejected: kexec loads kernels/bzImages, not EFI apps; the .efi is for the GRUB/firmware chainload path.

Consequences

• Two artifacts staged per host (efi + bin), a few MiB on the ESP.
• The kexec entry points (mode + stage-8) are unverified until real hardware; the GRUB chainload seam is the dependable path meanwhile.
• Pure bits (the kexec argv builder, the GRUB snippet render) are unit-tested; the kexec execution is what needs hardware.
• Follow-ups: real-hardware kexec verification; the iPXE netboot menu (keyboot/memtest/rescue); auto-staging memtest in the build/CI image.

0015 — Ansible roles: substrate, unlock, deploy

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-05

Context

keyboot’s fleet story needs Ansible (SPEC §3: “ansible manages the repo definitions”). The primitives exist — keyboot-install install-os, the BE verbs (be clone / boot-next / promote / destroy / gc), the automation keyslot (ADR 0013) and the keyboot_unlock role (GL#19). This ADR pins the role set and, crucially, the deploy state machine so a remote upgrade is safe by construction.

Decision

Three roles under ansible/:

• keyboot_unlock (shipped, ADR 0013) — feed the vaulted slot-2 automation passphrase to keyboot’s pre-unlock dropbear over SSH. The reboot/unlock building block the other roles include.

• keyboot_substrate — drive a host already in a rescue env through install-os to a fresh encrypted-ZFS system: stage keyboot-install (the signed curl|sh, ADR 0008), run install-os <distro> --disk … --confirm, reboot, then keyboot_unlock + wait for the OS. Getting the host into rescue is vendor-specific (Hetzner rescue, netboot, IPMI) and stays an operator prerequisite — the role asserts rescue reachability rather than pretending to be generic.

• keyboot_deploy — the safe remote-upgrade loop:

  be clone (running -> new)        # a fresh BE off the running one
  [prepare hook]                   # operator mutates the clone (pkgs/config); optional
  be boot-next --once <new>        # boot-ONCE: not yet the permanent default
  reboot
  keyboot_unlock                   # automation-slot unlock of the keyboot env
  wait for the OS sshd
  run health_cmd on the new BE     # operator-defined; exit 0 = healthy
    healthy  -> be promote <new>   # make it the permanent default
    unhealthy-> do NOT promote     # leave it; boot-once auto-reverts next reboot
  

Health gate = an operator command

“Healthy” is whatever the operator’s workload says — so keyboot_deploy_health_cmd is a command run on the booted new BE, exit 0 = promote. Default systemctl is-system-running --wait (systemd BEs). openrc BEs (Alpine/Gentoo) must override it (e.g. rc-status -c or a service-specific probe) — documented in the role defaults. Parameterize-with-a-default; no baked fleet policy.

Revert = keyboot’s boot-once net, not the role

On an unhealthy BE the role simply doesn’t promote. Because the clone was booted with boot-once, the next reboot returns to the old (still-bootfs) BE automatically. This is strictly safer than the role actively reverting: it also covers the BE that won’t boot at all or hangs before SSH — cases an “Ansible sets bootfs back” approach can’t reach because Ansible never gets a connection. The role may optionally reboot to enact the revert immediately; the safety itself is keyboot’s, not Ansible’s.

Alternatives considered

• Role-driven revert (set bootfs back + reboot) — rejected as the primary mechanism: it can’t recover a BE that never reaches SSH; boot-once already handles every failure mode including “won’t boot”. Kept only as an optional immediate enactment of the revert boot-once would do anyway.
• HTTP-only health gate — rejected as the default: not universal (DB/host with no endpoint); a command covers HTTP (curl) and everything else.
• Generic keyboot_substrate incl. rescue entry — rejected: rescue entry is irreducibly vendor-specific; faking it would be a lie. The role starts from “host is in rescue and reachable”.

Consequences

• A remote upgrade can’t strand a box: worst case it boots the new BE, fails the gate (or never comes up), and the next reboot is the old BE. Pairs with ADR 0009 (keyboot self-upgrade A/B) — different layer, same boot-once discipline.
• The deploy loop depends on the automation keyslot (ADR 0013) for unattended unlock and on be boot-next --once honoring boot-once semantics (verify in the deploy round-trip test, the rung-1/promote follow-up).
• keyboot_substrate’s rescue-entry prerequisite should grow per-vendor helper docs over time (Hetzner first, since that’s the validated platform).

0016 — Web UI for unlock + BE selection (in the keyboot env)

Status: accepted — Phase 1 + Phase 2 BUILT (2026-06-25; proposed 2026-06-24)

Context

Today the operator unlocks the pool and (eventually) picks a boot environment by SSHing into the pre-boot dropbear (ADR 0002 fallback path / the interactive prompt), or at the TTY. SSH is a strong default: its security rests on public/private-key auth plus server host-key verification, all in one small audited daemon already in the image. The downside is convenience — it needs an SSH client and a typed command; a browser-based unlock (“open a page, type the passphrase, click the BE”) would be friendlier, especially for less CLI-comfortable operators.

Prior art is essentially nonexistent. Network/early-boot unlock today is: SSH-in-initramfs (dropbear; what we do), automated key-fetch over HTTPS (tqdev’s LUKS-over-HTTPS keyscript — machine automation, not a UI), network-bound auto-unlock (Clevis/Tang), or Mandos. No project ships an interactive browser UI to unlock + select a BE at boot. So this is novel — exciting, and a reason for extra caution: no one has hardened this pattern for us.

This ADR records the design thinking so we don’t trade away SSH’s security for UX. No code is committed by this ADR.

The constraint that shapes the whole design

Browsers do not expose the TLS channel/cert to JavaScript. Therefore the appealing “the page itself cryptographically proves there’s no MITM” approach (PAKE / TLS channel-binding, RFC 5929) cannot be done in a plain browser — the page can’t read its own TLS cert, and a MITM can rewrite anything the page displays. “Prove no MITM” must come from either (a) the browser’s own cert validation or (b) a human comparing a fingerprint out-of-band. There is no fully-automatic in-page version.

What SSH gives for free (the bar to clear)

1. Strong mutual auth — client key and server host-key (TOFU / known_hosts).
2. Encrypted transport before any secret crosses the wire.
3. A tiny, audited, already-present server (dropbear).

A web unlock must re-solve all three, and the convenient version (“browse to https://host, type passphrase”) is exactly where each is hardest.

Threats specific to a web unlock

• Passphrase capture (catastrophic). The LUKS passphrase unlocks every disk. Plain HTTP, or HTTPS the user was trained to click through, → an on-path attacker captures it. SSH never reveals the passphrase until the channel and server identity are proven.
• Server impersonation / phishing — a fake unlock page harvests the passphrase.
• Pre-boot RCE — an HTTP+TLS stack in the initramfs runs as root, holding the keys; a parser bug is game-over. Dropbear is one small audited thing; a web stack is more surface.
• Browser-trust UX — a raw self-signed cert trains users to click through the exact “not secure” warning that would otherwise catch a MITM.

Decision (proposed)

Pursue it in layers, none of which weaken the SSH default; the web UI is always additive and opt-in, and dropbear remains a supported gate.

Trust model: bake the TLS cert like an SSH host key

The middle path that fits keyboot’s existing model: keyboot already bakes persistent SSH host keys into the image (SSH_DIR). Do the same for TLS — bake a persistent self-signed server cert + key into the image. The operator imports that cert into their browser/OS trust store once (from the image they built — the same already-trusted source as the SSH keys). Thereafter: no warning, and the browser performs MITM detection automatically every boot (cert mismatch = hard block). Its leak profile is identical to the baked SSH host key (image leak → key leak), a trade already accepted for SSH — so this is consistent, not a regression.

Bootstrap / no-import-yet fallback: keyboot prints the cert’s SHA-256 fingerprint to the console (serial/KVM/IPMI — already written there); the operator compares it to the browser’s cert viewer on first connect (TLS equivalent of verifying an SSH host key). Manual, but a real proof of no-MITM.

Authorization (who may unlock)

A trusted channel still must gate who can unlock (don’t expose the unlock UI to every host on the subnet). The authorized_keys analog, in preference order: mTLS client cert (issued from the same baked trust — cleanest model, clunky browser enrollment) → WebAuthn/passkey (nicest touch-to-auth, more moving parts) → baked bearer token (simplest, weakest).

Rollout

• Phase 1 (recommended first): web server bound to 127.0.0.1 only, reached via an SSH tunnel (ssh -L). Security is identical to today (dropbear is still the only thing on the network); the win is a real UI (BE grid, pool/disk status, click-to-unlock) at near-zero new risk. Validates the UX and the server skeleton.
• Phase 2 (opt-in, non-default): network-exposed HTTPS using the bake-and-trust-once cert + client-cert/passkey auth above, behind its own cmdline/config flag, documented threat model, and never replacing dropbear.

Implementation invariants

• Server in Rust + rustls, minimal hand-rolled routes — no general web framework; keep the pre-boot-root attack surface tiny.
• Drives the same keyboot unlock / BE-discovery code paths — no parallel unlock logic.
• The passphrase is handled with the same care as elsewhere (never logged, never to non-volatile storage; zeroized after use).

Alternatives considered (rejected)

• Plain HTTP / network-exposed self-signed with clickthrough — passphrase capture via MITM/sniff; trains clickthrough. Strictly worse than SSH. No.
• Automated HTTPS key-fetch (tqdev) / Clevis-Tang — these are automatic unlock (key from a server / network presence). Different feature; doesn’t give an interactive operator UI and changes the threat model (a network service holds/gates the key). Out of scope here.
• In-page PAKE / channel-binding for automatic MITM proof — impossible in a plain browser (no JS access to the TLS channel; see constraint above).

Consequences

• A genuinely novel, friendlier unlock path without lowering the SSH baseline (it stays; the web UI is additive/opt-in).
• New code on a pre-boot-root surface — mitigated by Rust/rustls + minimal routes
  • Phase-1 localhost-only start.
• A new baked secret (TLS cert/key) with the same lifecycle/leak handling as the SSH host key (rotation story should mirror it).
• Operator one-time cert import for the warning-free path; console-fingerprint fallback otherwise.

Status — what was built (2026-06-25)

Both phases are implemented, CI-gated, and Phase 2 was validated on real hardware.

• Phase 1 (keyboot web, loopback + ssh -L): tools/keyboot/src/cli/web.rs (--features web, tiny_http). Shared router web::dispatch (GET /, /api/status; POST /api/unlock, /api/select-be). /api/unlock writes the passphrase to the stage-4 FIFO exactly like keyboot-askpass (one unlock path); BE selection writes /run/keyboot/be.selected validated against the candidates. CI: clippy:web + qemu:web (boots keyboot.web=1, tunnels, asserts "phase":1).
• Phase 2 (network HTTPS + mTLS): tools/keyboot/src/cli/web_tls.rs (--features tls, rustls with the ring provider — no aws-lc/cmake, so it builds into the static musl binary). The shipped keyboot-x86_64-musl is built --features web,tls (build:static-be), so the release binary is Phase-2-capable out of the box. Server cert baked like the SSH host key; WebPkiClientVerifier requires a client cert signed by the baked client CA (the authorized_keys analog). Same dispatch router over a hand-rolled HTTP/1.1 read on the TLS stream.
• Boot wiring: init/stage-3-ssh.sh launches the server when keyboot.web=1 (init/lib/cmdline.sh) — HTTPS+mTLS on 0.0.0.0:8443 if /etc/keyboot/web/{server-cert,server-key,client-ca}.pem are baked, else loopback Phase 1. no-port-forwarding on the boot key is relaxed only under keyboot.web=1 so the ssh -L tunnel works. ci/build-image.sh bakes the cert set from WEB_TLS_DIR; ci/gen-web-certs.sh generates server/CA/client material.
• Real-hardware validation (.11, 88.99.137.11, 2026-06-25): the static --features tls binary served HTTPS+mTLS on :8443 over the public internet — no client cert ⇒ TLS handshake rejected (certificate required); the CA-signed operator client cert ⇒ /api/status JSON. (Ran in userspace on the installed OS; the in-pre-boot launch is the same QEMU-gated stage-3 path.)

Open questions

• mTLS client cert vs passkey vs token for authorization — resolved: mTLS client cert (Phase 2, built). Passkey/token remain possible future additions.
• Cert rotation/management UX (parallel to SSH host-key rotation).
• Whether Phase 2 is worth building at all, or Phase 1 (localhost + tunnel) already captures most of the value at a fraction of the risk.