Deploying keyboot to real hardware (Hetzner / generic UEFI server)

This is the field guide for installing keyboot onto a bare-metal server whose only access is a vendor rescue system + a KVM/IPMI console. It was written against Hetzner (Debian rescue + a QEMU-based “vKVM”), but the shape applies to any provider: boot a rescue, stage the installer, run install-os, reboot into the keyboot unlock environment.

The QEMU smoke tests (ci/qemu/*) get the happy path for free because everything is built-in virtio. Real hardware is where the modules and the console matter — this doc exists so the next agent doesn’t re-derive the fixes below from a dark screen.

The three environments — keep them straight

A real deploy moves through three distinct environments. They have different tooling, and conflating them is the single biggest source of wasted time.

1. Vendor rescue (Hetzner: Debian live). Where you stage and run the installer. It is not a keyboot image — it has none of keyboot’s baked-in tooling. Critically:

   • ZFS is not present. Hetzner’s zpool is a wrapper that compiles OpenZFS against the rescue kernel on first use. It must be recompiled on every fresh rescue boot (the rescue is ephemeral). hetzner-prep.sh drives this.
   • keyboot-install must be built natively here. The repo’s static-musl keyboot binary links a newer glibc than the Debian rescue ships (GLIBC_2.39 vs the rescue’s 2.36), and the install-time keyfile ops use libcryptsetup over FFI — so keyboot-install is cargo build-ed on the rescue against its own libcryptsetup. hetzner-prep.sh does this.
   • Re-flashing the ESP from here needs no ZFS. The ESP is plain vfat on an mdraid1 mirror; updating the active slot’s /keyboot/<A|B>/{vmlinuz,initramfs.img} (ADR 0009; pre-A/B installs used /EFI/keyboot/keyboot-*) is pure mount + cpio + gzip. Don’t recompile ZFS just to swap a kernel.

2. The keyboot unlock environment (the booted keyboot initramfs). What GRUB boots into post-install. It is self-contained: it bundles its own zfs.ko, a static keyboot, cryptsetup, dropbear. It does all LUKS/ZFS work itself — you never feed it ZFS from outside. This is where you SSH in to enter the passphrase.

3. The installed OS (the boot environment / BE). What keyboot kexecs into after unlocking. For Alpine this is configured by alpine.sh to come up networked with sshd (see “The installed OS must be reachable” below).

Rule of thumb: pool/ZFS/LUKS work happens in environment 2 or 3, which carry their own ZFS. In environment 1 (rescue) ZFS is a cost you pay only when you must touch the pool from outside (e.g. inspecting an install). A kernel/ESP swap is not such a case.

A fourth, sneaky one: the vendor “KVM” may be QEMU

Hetzner’s vKVM is a QEMU Q35 + OVMF virtual machine that passes the real disks through. Consequences that have bitten us:

• Disks appear as QEMU HARDDISK with synthetic serials (QM00013, QM00015), not the drives’ real serials. The real Seagate/etc. serials are only visible from the rescue or a true bare-metal boot. keyboot opens disks as sn-<serial>, so the mapper names differ between vKVM and bare metal — that’s expected, the pool imports by scanning either way.
• The vKVM gives you a video console; a plain bare-metal boot (no vKVM attached) does not. So “boots fine under vKVM” does not prove “visible on bare metal” — and vice-versa, “dark on bare metal” is often just no console, not a hang. Verify bare-metal boots over SSH into the unlock env, not by staring at a console you don’t have.

Real-hardware fixes baked into the boot image

These are all upstreamed into the repo; listed here so you know why they’re there and where to look if a new box misbehaves.

The boot image you ship must be built with INCLUDE_MODULES=all (real NIC/HBA coverage) and your SSH pubkey baked in (so you can reach the unlock env’s dropbear). Build the kernel on a real build host, the initramfs on the musl image factory (keybootvm) — see kernel/build.sh and ci/build-image.sh.

The installed OS must be reachable (net + ssh)

install-os historically produced a console-only substrate: a serial getty and a boot marker, enough for the QEMU cold-boot CI gate, but with no networking and no sshd enabled. On a headless server with no KVM that boots to an unreachable box.

alpine.sh now configures the BE as a reachable headless server:

• DHCP on eth0 (/etc/network/interfaces);
• the standard Alpine runlevel set populated via runlevel symlinks, so networking’s hard deps (localmount, hostname) are met and hwdrivers coldplugs the real NIC driver (the QEMU path only worked because virtio_net is built-in);
• sshd enabled, root login key-only (stock prohibit-password);
• the operator key from --authorized-keys <file> written to /root/.ssh/authorized_keys.

Pass --authorized-keys to install-os or the booted OS will be console-only. (Debian/Gentoo plugins need the equivalent — tracked as a follow-up.)

Two unlock modes — pick the right one for the test

keyboot stage-4 prompts for the passphrase, racing the console and any SSH session (dropbear forced-command keyboot-askpass). There is one exception: if a passphrase file is baked into the initramfs at /etc/keyboot/test/passphrase, stage-4 reads it and auto-unlocks without prompting. That file is the CI/unattended hook (set via --test-passphrase-file), not a keyboot feature you want by default.

• Interactive (default, no baked file): the box comes up in the unlock env and waits, staying reachable on the network. Use this to verify a no-KVM bare-metal boot: SSH into the unlock env, feed the passphrase, watch it unlock → import → kexec. (If the box drops off the network shortly after you feed the passphrase, that’s success — it kexeced away.)
• Auto-unlock (baked file): the box unlocks and kexecs with no prompt. Convenient, but on a no-KVM box it races past the only environment you can reach, so you can’t observe the unlock. Reserve it for CI or genuinely unattended production where the BE is itself reachable.

Note both modes concern the keyboot env’s unlock (stage-4). The BE re-unlock after kexec is now silent (ADR 0003 key hand-off): keyboot carries the payload across into the BE, so production boots with a single prompt — the one in the keyboot env. A successful real-hardware test therefore enters the passphrase once and the box rides all the way to the BE’s own sshd; if it stalls after kexec waiting for a second passphrase, the hand-off failed (check for keyboot.handoff=0 or a missing overlay). Confirm via the BE’s dmesg: keyboot be-unlock: using handed-off keyfile payload (single-prompt).

End-to-end procedure

# 0. Operator boots the box into the vendor rescue, gives you SSH access.

# 1. Build the boot image (real NIC coverage + your key). Kernel on the build
#    host; initramfs on the musl factory (keybootvm):
#      bash kernel/build.sh
#      INCLUDE_MODULES=all AUTHORIZED_KEYS=you.pub bash ci/build-image.sh

# 2. Ship the repo + boot image to the rescue (exclude build trees):
rsync -az --exclude 'tools/keyboot/target*' --exclude '.git' \
      --exclude 'kernel/.build' --exclude 'zfs/.build' \
      ./ root@<box>:/root/keyboot/
scp keyboot-vmlinuz keyboot-initramfs.cpio.gz <static-musl-keyboot> root@<box>:/tmp/

# 3. Bootstrap the rescue (ZFS recompile + apk.static + native keyboot-install):
ssh root@<box> 'cd /root/keyboot &&
  KEYBOOT_STATIC=/tmp/keyboot-static-musl bash tools/hetzner-prep.sh'

# 4. Install (DESTROYS the disks). Interactive unlock (no --test-passphrase-file):
ssh root@<box> '
  export KEYBOOT_INSTALL_OS_DIR=/root/keyboot-install-os
  export KEYBOOT=/usr/local/sbin/keyboot KEYBOOT_INSTALL=/usr/local/sbin/keyboot-install
  printf %s "<passphrase>" > /root/.kbpass
  keyboot-install install-os alpine \
    --disk /dev/sda --disk /dev/sdb \
    --passphrase-from file:/root/.kbpass \
    --keyboot-kernel /tmp/keyboot-vmlinuz \
    --keyboot-initramfs /tmp/keyboot-initramfs.cpio.gz \
    --authorized-keys /tmp/you.pub \
    --hostname <name> --no-reboot --confirm'

# 5. Operator reboots to bare metal. The box comes up in the keyboot unlock env
#    (networked, your key). SSH in and feed the passphrase:
printf %s "<passphrase>" | ssh root@<box>      # dropbear runs keyboot-askpass
#    -> keyfile open -> data disks open as sn-<serial> -> pool import -> kexec
#    -> the installed Alpine boots networked; SSH in to verify (zpool status, etc.)

Re-flashing only the ESP (no reinstall, no ZFS)

To swap just the keyboot kernel/initramfs on an already-installed box, from the rescue — e.g. to ship a new boot image without redoing the install:

mdadm --assemble --scan; mount /dev/md127 /mnt/esp        # KEYBOOT-ESP vfat
# The host keyfile is a trailing gzip member appended to the ESP initramfs.
# Split it off and re-attach to the new base image (preserves the keyfile,
# byte-for-byte) — keyfile-only (no test/passphrase) keeps interactive unlock:
python3 - <<'PY'
import zlib, gzip
d = zlib.decompressobj(16+zlib.MAX_WBITS)
img = open('/mnt/esp/keyboot/A/initramfs.img','rb').read()   # pre-A/B: /mnt/esp/EFI/keyboot/keyboot-initramfs.img
d.decompress(img)                       # consume base; unused_data = keyfile seg
open('/tmp/kf.gz','wb').write(d.unused_data)
PY
cat new-initramfs.cpio.gz /tmp/kf.gz > /mnt/esp/keyboot/A/initramfs.img
cp new-vmlinuz /mnt/esp/keyboot/A/vmlinuz
sync; umount /mnt/esp

(If the trailing segment also contains etc/keyboot/test/passphrase, extract keyfile.luks and rebuild a keyfile-only segment to drop auto-unlock.)

Adding a distro BE via provision mode (multi-distro on one pool)

The provisioning vision: keyboot’s own env is the only rescue you ever need. Boot the installed keyboot image, import the pool, and install-os --add-be any distro onto the existing rpool — no partition/keyfile/pool-create/GRUB churn, and the new BE is auto-discovered by the picker (stage-7). Build the image with INCLUDE_INSTALL=yes INCLUDE_ZFS=yes so the provision shell bundles keyboot-install + the orchestrator + zfs.ko (the shipped keyboot-install is the fully static, no-FFI binary — runs in any env with cryptsetup on PATH).

A console is required to enter provision mode on a box whose ESP grub.cfg has a fixed keyboot.mode=boot (e.g. a legacy single-slot install): there is no SSH path to change the boot mode — pick/edit it at the keyboot GRUB menu. Once the ESP carries the GL#44 one-shot grub.cfg (any A/B-staged image), keyboot-install keyboot provision --confirm arms a self-reverting provision boot with no console needed.

# At the keyboot GRUB menu (console): pick KEYBOOT - PROVISION, or edit an entry
# and append keyboot.mode=provision. keyboot unlocks (keyfile injected) -> imports
# rpool READ-WRITE -> drops to an install-os-ready shell.

# In the provision shell — add a distro into a NEW BE on the existing pool:
keyboot-install install-os gentoo --add-be --pool rpool \
    --authorized-keys /path/to/you.pub --confirm

# Reboot; the new BE is auto-discovered at the menu. The original BE is untouched
# and stays the default until you `keyboot-install be promote` the new one.

Do NOT run --add-be from a running BE over SSH: the orchestrator’s teardown ends with zpool export <pool>, which cannot run on the live root pool. --add-be belongs in the provision/rescue env (pool imported but not the live root).