Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

0009 — keyboot self-upgrade: A/B ESP slots with boot-count recovery

Status: accepted (2026-06-10) — install side implemented (GL#29: slots + grubenv + keyboot-install install --slot / keyboot list|promote|rollback, two-entry snippet + A/B ESP grub.cfg, single-slot adoption; ESP sizing GL#32). Runtime trial-commit implemented too (GL#30: keyboot slot-commit + init/lib/slot.sh, called by stage-6 after unlock+import; fail-safe no-commit on any error). The auto-revert QEMU gate is authored (GL#31: ci/qemu/ab-revert-test.sh, CI job qemu:ab-revert — broken trial auto-reverts, good trial self-commits).

Context

keyboot-be-upgrade (ADR 0007 neighbourhood) upgrades the booted OS. Nothing yet upgrades keyboot itself — the boot stage’s kernel + initramfs + the lockstep zfs.ko baked into that initramfs (ADR 0004). Today the GRUB seam (tools/keyboot/src/install/host.rs) places a single copy at <ESP>/keyboot-vmlinuz + keyboot-initramfs.img and emits one menuentry 'keyboot'. install_file is atomic per file (write .kbtmp → rename), but there is one slot: a keyboot update overwrites the only copy, and you cannot fix a boot stage that will not boot. A bad kernel-config change, a zfs.ko that won’t import the pool, a broken initramfs — any of these bricks the host until someone shows up with the USB rescue image (GL#1).

keyboot is pre-OS, so “did it work?” has a precise, self-contained answer: did it open the LUKS disks and import the pool? That makes boot-count A/B recovery — the systemd-boot / Fedora boot counting / greenboot pattern, but for the boot stage — the right model. ADR 0004 already flagged “keyboot self-upgrade … ideally A/B keyboot” as an open boundary; this ADR closes it.

Decision

Two ESP slots

keyboot lives in two slots, A and B, on the keyboot-owned mdraid1 vfat ESP:

<ESP>/keyboot/A/{vmlinuz, initramfs.img}
<ESP>/keyboot/B/{vmlinuz, initramfs.img}
<ESP>/keyboot/keyboot.env          # GRUB grubenv: the A/B state

A slot is a {kernel, initramfs} pair; the lockstep zfs.ko and the host keyfile container are inside that slot’s initramfs (so staging a slot re-injects the keyfile, exactly as the single-slot install does today). The slots live on the ESP, not the encrypted pool, because GRUB must read keyboot before unlock.

grubenv state (three keys)

A GRUB grubenv (keyboot.env, the fixed 1024-byte block GRUB reads/writes in place) holds the whole state machine:

  • keyboot_slot — the known-good slot GRUB boots by default (A|B).
  • keyboot_try — a slot on trial (A|B|unset).
  • keyboot_try_left — remaining boot attempts for the trial (starts at 1).

Two menuentries (keyboot-A, keyboot-B) are always emitted, each tagged with a keyboot.slot=A|B cmdline token, so either slot is reachable by hand from the GRUB menu as a human backstop. Which one boots by default is driven by grubenv, evaluated in the 10_keyboot snippet:

if [ -n "$keyboot_try" ] && [ "$keyboot_try_left" -gt 0 ]; then
    decr keyboot_try_left; save_env; default=keyboot-$keyboot_try
else
    default=keyboot-$keyboot_slot
fi

Decrement-before-boot is the load-bearing trick: a trial that hangs or panics and never returns has already spent its attempt, so the next reset falls through to keyboot_slot (the known-good) with no human in the loop.

“Good” = unlock + pool import (pre-kexec)

When the trial keyboot reaches its core milestone — LUKS open + pool imported, i.e. its own job is done — and only then, it commits the trial: briefly mount the ESP rw, write keyboot_slot=<trial>, clear keyboot_try / keyboot_try_left, fsync, return the ESP to ro, then kexec. The trial slot is now the default. A trial that never reaches unlock never commits, so recovery is automatic (above).

This is a deliberate, bounded carve-out from ADR 0004’s ESP-read-only-by- default: a single grubenv write, only on a trial boot, mount-rw → write → sync → ro. Documented here so it is not a silent erosion of that invariant.

Layering boundary (explicit): “good” means keyboot works, not that the OS works. A keyboot that unlocks fine but kexecs into a broken BE is not caught here — that is the BE’s own A/B rollback (ADR 0007), a different layer. keyboot self-upgrade protects against a broken keyboot; BE rollback protects against a broken OS. They compose; neither substitutes for the other.

Auto-promote on package upgrade

The keyboot-boot package post-install (ADR 0008) auto-promotes: stage the new keyboot into the inactive slot (never the running/known-good one), then arm the trial (keyboot_try=<inactive>, keyboot_try_left=1) and let GRUB boot it next reset. This is safe precisely because the boot-counter auto-reverts on failure — the safety net ADR 0008’s “stage-and-operator-commits” stance lacked. KEYBOOT_AUTOPROMOTE=0 opts out to stage-only (0008’s original manual-commit path remains for the conservative operator). This refines ADR 0008 for the boot package: staging still lands in /var/lib/keyboot/staged/ first, but the commit step is now an A/B promote, not an ESP overwrite.

keyboot-install surface (deferred — this ADR is design only)

The planned verbs, to be implemented in the follow-up issues below:

  • keyboot-install install --slot {A|B|inactive} [--promote] — write a slot, optionally arm the trial. (Evolves today’s single-slot install_host.)
  • keyboot-install keyboot list — slots, versions, which is good/trial, counter.
  • keyboot-install keyboot promote <slot> / rollback — arm a trial / pin keyboot_slot back to the current good (the panic button).
  • The 10_keyboot snippet grows from one entry to the two-entry + grubenv form above; keyboot gains slot-awareness (keyboot.slot=) and the commit step (ship grub-editenv, or a ~30-line grubenv writer, in the initramfs).

Alternatives considered

  • Manual two-entry only — both slots in the menu, operator flips the default by hand. Rejected as the primary mechanism (recovery needs a console; fleet- hostile), but kept as the backstop (the entries are always present).
  • GRUB fallback only — default=new, one-shot fallback to old. Rejected: catches a hard panic but not a keyboot that boots-but-won’t-unlock, which would loop. The boot-counter + success-commit closes that gap.
  • Full-BE-boot success signal — mark good only after the OS comes up and signals back. Rejected for now: needs an OS→ESP feedback path and a be-tool; heavier, and conflates the keyboot and OS layers. unlock+import is the minimal self-contained signal that keyboot actually did its job.
  • TPM-measured / sealed keyboot — out of scope; a future ADR.

Consequences

  • ESP capacity doubles. Two full keyboot images, and the firmware-laden initramfs is hundreds of MiB (GL#20 / BACKLOG B5). The KEYBOOT-ESP partition template must be sized for ≥2× an initramfs; this couples self-upgrade to the firmware-slimming work. Note the dependency at install time.
  • Both-slots-bad is still possible but narrowed: we never stage over the currently-good slot, so a single bad upgrade is always recoverable to the prior good. Two successive bad upgrades (the second staged before the first was proven) could strand you → mitigation: the always-present manual menu entries and the USB rescue image (GL#1) remain the ultimate backstop.
  • grubenv tooling. keyboot’s commit step needs to write a grubenv on the ESP — ship grub-editenv (or a tiny fixed-block writer) in the keyboot initramfs; a missing writer must fail safe (don’t commit → auto-revert), never brick.
  • The single-slot install_host and its 10_keyboot snippet are superseded by the two-slot form; the migration (first A/B install adopts the existing single copy as slot A) is a follow-up detail.

Follow-ups (file as issues)

  1. Implement install-side A/B: keyboot-install install --slot/--promote, keyboot keyboot list|promote|rollback, the two-entry grubenv-driven 10_keyboot snippet, single-slot→A/B migration. (The bulk.)
  2. keyboot trial-commit step: slot-awareness + the bounded ESP-rw grubenv write at unlock+import; ship a grubenv writer in the initramfs; fail-safe on its absence.
  3. QEMU gate: stage a deliberately-broken slot, prove the box auto-reverts to the known-good slot (the headline guarantee).
  4. ESP partition sizing for two slots (couples GL#20 firmware slim).
  5. Reconcile ADR 0008’s keyboot-boot post-install to the auto-promote flow (+ the KEYBOOT_AUTOPROMOTE=0 opt-out).