Real-hardware validation runbook (console-attended)

The remaining real-hardware test items (GL#38 memtest, GL#43 keyboot.rollback= rungs, GL#37 NIC firmware) need a human at the box: a serial/KVM console for recovery, and — for GL#37 — diverse server NICs. This runbook makes such a session fast and safe. It complements hetzner-deploy.md (install procedure + the three-environments model) and memtest.md (memtest staging). The keyboot-be-upgrade BE-upgrade flow (GL#43, first half) is already validated on real hardware — its procedure is captured below for reference because it’s the safe pattern the other tests borrow from.

Before you start — the safety model

• Have a console. On Hetzner that’s the Robot reset button (power-cycle) and the KVM/serial-over-LAN console. Most failure modes here are recoverable over SSH (see below), but a post-kexec hang or a rollback boot-loop needs the console.
• keyboot brings the network + dropbear up EARLY (stage-2/3), before the unlock/BE/rollback logic (stage-4+). So if a test fails inside keyboot, you usually still get a keyboot recovery shell over SSH (the keyboot_hw key) — from which you fix the ESP and reboot. The only truly-dark case is a successful kexec into a BE/target that then hangs (keyboot is gone).
• The ESP is the control surface. On a keyboot box the grubenv (/esp/keyboot/keyboot.env), the boot-once marker (/esp/keyboot/once.next), and the BIOS/UEFI menu config (/esp/grub/grub.cfg) live on the mdraid1 ESP. From any booted OS or the keyboot recovery shell: assemble + mount the ESP md (vfat), edit, sync, unmount. Writing through the assembled md array keeps both mirror members consistent (unlike GRUB’s own save_env on a raw member — see [grubenv on mdraid1, GL#44]).

# Mount the ESP from a booted OS / recovery shell (device name varies: md127, md0, /dev/md/keyboot-esp)
mkdir -p /mnt/esp
for d in /dev/md/keyboot-esp /dev/md127 /dev/md0; do [ -e "$d" ] && mount -t vfat "$d" /mnt/esp && break; done
ls /mnt/esp/keyboot/keyboot.env   # confirm it's the ESP
# ... edit ...
sync; umount /mnt/esp

• Boot selection precedence (keyboot stage-7): keyboot.be= cmdline > /esp/keyboot/once.next (consumed on read) > pool bootfs. Keep a known-good BE as bootfs so any failure that consumes/ignores a one-shot reverts there.

Recovering a stuck box (read this first)

Test A — keyboot-be-upgrade (GL#43, VALIDATED — reference pattern)

Clone-based cross-release BE upgrade. Safe to the running BE (it upgrades an isolated clone; the original is untouched). Validated bookworm→trixie on the Hetzner BIOS box.

# On the booted OS (be-tools are staged from first boot since GL#43 fix):
keyboot-be-upgrade -- sh -c '
  set -e; export DEBIAN_FRONTEND=noninteractive
  sed -i "s/bookworm/trixie/g" /etc/apt/sources.list /etc/apt/sources.list.d/*.list 2>/dev/null || true
  apt-get update --allow-releaseinfo-change -qq
  apt-get -y -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef full-upgrade
  apt-get -y --purge autoremove || true
  update-initramfs -u -k all'        # zfs-dkms rebuilds zfs.ko; keyboot hook persists
# -> prints the upgraded clone name, e.g. rpool/ROOT/debian-<stamp>

Safe boot-once into the upgrade (auto-reverts to bootfs on the next power-cycle if it hangs):

# bootfs stays the known-good BE; once.next selects the clone for ONE boot
mount -t vfat /dev/md127 /mnt/esp
printf 'be=%s\n' rpool/ROOT/debian-<stamp> > /mnt/esp/keyboot/once.next
sync; umount /mnt/esp
reboot

Expected: keyboot consumes once.next, kexecs the upgraded kernel, the BE initramfs re-unlocks + mounts, systemd comes up networked. Verify: cat /etc/os-release, uname -r, zpool status. PASS = new release boots + SSH reachable; original BE + @…-PREUPGRADE snapshot intact. To keep it: zpool set bootfs=rpool/ROOT/debian-<stamp> rpool. To discard: zfs destroy -r rpool/ROOT/debian-<stamp> (after reverting bootfs).

Test B — keyboot.rollback= rungs (GL#43, second half)

Validates the pre-boot rollback ladder (ADR 0007) on bare metal. CI covers it (ci/qemu/rollback-rungs-test.sh); this is the real-hw confirmation. rung 3 is destructive (zfs rollback -r, discards newer state) — do it on a test box.

There is no one-shot cmdline mechanism — keyboot.rollback= must be appended to the keyboot menuentry’s linux line in /esp/grub/grub.cfg, so keep the console up and restore grub.cfg right after each rung. Format: keyboot.rollback=<dataset>@<snap>:<rung> (:ro | :destroy); destroy also needs keyboot.rollback.confirm=1.

Prep (on the booted OS):

BE="$(awk '$2=="/"&&$3=="zfs"{print $1}' /proc/mounts)"
echo MARKER-ORIGINAL > /root/marker            # state to roll back TO
zfs snapshot "$BE@rolltest"
echo MARKER-MUTATED  > /root/marker            # newer state to discard/hide
cp /esp/grub/grub.cfg /esp/grub/grub.cfg.bak   # (mount ESP first)

rung 1 — read-only clone boot (NON-destructive)

Append keyboot.rollback=<BE>@rolltest:ro to the linux /keyboot/A/vmlinuz … line of the keyboot-A menuentry in /esp/grub/grub.cfg; sync; reboot.

• Expected: boots an ephemeral readonly=on clone of the snapshot (tagged keyboot:ephemeral=1); /root/marker reads ORIGINAL; the live BE
  • timeline are untouched (zero side effects).
• Then: from this boot, restore grub.cfg (cp …grub.cfg.bak grub.cfg), sync, reboot → back to the normal BE; /root/marker reads MUTATED (confirms rung 1 left no trace).

rung 3 — in-place destroy (DESTRUCTIVE, guarded)

Append keyboot.rollback=<BE>@rolltest:destroy (no confirm) first:

• Expected: REFUSED without keyboot.rollback.confirm=1 → normal boot, marker still MUTATED. (Also refused if a dependent clone exists — the no--R -f guard, init/lib/rollback.sh.) Then append …:destroy and keyboot.rollback.confirm=1:
• Expected: zfs rollback -r in place — same dataset (not a clone), newer state discarded; /root/marker reads ORIGINAL.
• Then: restore grub.cfg, reboot to normal.

Recovery: if a rollback boot doesn’t bring SSH up, the grub.cfg token is still in place → it will re-enter rollback every boot. Use the KVM console: at the GRUB menu press e, delete the keyboot.rollback… token, boot; then restore /esp/grub/grub.cfg from .bak.

Test C — bootable memtest (GL#38)

Confirms the kexec -l form of memtest86+ 6.x on real hardware (the one ADR 0014 unknown). Staging + entry points are in memtest.md. The box is offline during the test; you watch it on the console.

1. Stage memtest on the ESP and add the GRUB entry (re-run install with KEYBOOT_MEMTEST_DIR=<dir with memtest.{efi,bin}>, or keyboot-install install --memtest --confirm). See memtest.md.
2. Two entry points to verify:
   • GRUB menuentry (reliable): pick memtest86+ — EFI chainloader /EFI/keyboot/memtest.efi, or BIOS linux16 /EFI/keyboot/memtest.bin.
   • keyboot.mode=memtest (the kexec path under test): keyboot kexecs the ESP-staged memtest instead of a BE. This is the GL#38 unknown — confirm the kexec form actually launches memtest86+ 6.x on the metal.
3. Observe on the console (memtest86+ 6.x does serial output — console=ttyS0,115200 on the BIOS linux16 form, watch via serial-over-LAN). PASS = memtest UI runs + addresses all installed RAM (vs. the userland memtester, which only tests kernel-allocatable RAM).
4. Recovery: memtest never touches disk; just reboot (Robot reset) back to the normal entry.

Test D — curated NIC firmware (GL#37)

The curated firmware sets (Alpine subpackages, Debian firmware-*, Gentoo emerge+prune to server-NIC dirs) are only proven on virtio (which needs no firmware). Verify on real server NICs — needs hardware with Broadcom/Mellanox/QLogic/Intel-needs-firmware NICs (the Hetzner test box’s onboard Intel e1000e needs no firmware, so it can’t exercise this).

On a box with such a NIC, after installing with --firmware curated:

lspci -nn | grep -iE 'ethernet|network'        # identify the NIC
ethtool -i <iface> | grep -i firmware          # firmware-version (blob in use)
dmesg | grep -iE 'firmware.*(load|fail)'       # what loaded / what's MISSING
ls /lib/firmware/<driver-dir>                  # blob actually present?

PASS = the NIC links + the driver’s firmware loaded (no “Direct firmware load … failed” for the NIC). If a real NIC needs a dir/blob not in the curated keep-set, add it: Alpine linux-firmware-<sub> in alpine.sh; Debian firmware-<name> in debian.sh; the Gentoo keep-set in _gentoo_curate_firmware (gentoo.sh). Optionally add a non-virtio CI cell with an emulated firmware-needing NIC to catch regressions.

After the session

Record results on the issues (GL#37/#38/#43). If a curated list needed a NIC added, ship that fix. The @rolltest snapshot and any keyboot:ephemeral=1 clones from Test B can be cleaned up (zfs destroy); keyboot-be-rollback’s GC handles ephemeral clones.