Disaster recovery

What to do when a keyboot host is broken. Pairs with hardware-test-runbook.md (the recovery-model primer + ESP-as-control-surface mechanics apply here too) and assumes the architecture in SPEC §5/§7 and ADRs 0003/0004/0007/0009.

First principle: keyboot is designed so failures land you in a reachable place, not a brick. keyboot brings the network + dropbear up early (stage-2/3, before unlock), so a unlock/BE failure usually leaves a keyboot recovery shell over SSH (the operator key). The A/B slots (ADR 0009) and the BE timeline (ADR 0007) give you a known-good fallback for the two things that can hard-fail: keyboot itself, and the BE. Keep a console (KVM/serial) for the rest.

Quick map — symptom → section:

1. Lost or rejected passphrase

The keyfile container (SPEC §7) is multi-slot: slot 0 daily, slot 1 recovery, slot 2 automation, 3–7 reserved. Any slot opens the same 32-byte payload, so any one working passphrase recovers the host.

• Slot 0 forgotten, slot 1 known: boot, enter the recovery passphrase at the keyboot prompt — it unlocks normally. Then reset slot 0 in place: keyboot-install keyfile passwd <keyfile> --slot 0.
• Add a recovery slot before you need it: keyboot-install keyfile add-recovery <keyfile> (slot 1), … enroll-automation <keyfile> (slot 2).
• Remote-safe rotation (don’t zero a working slot until the new one proves out): keyboot-install keyfile rekey <keyfile> --retire-slot <old> --new-slot <new> --confirm — adds the new secret, verifies it opens, then wipes the old. This is the Ansible bootstrap→true-secret flow.
• All passphrases lost: the payload is the AES-256 key for every data disk; with no slot openable the data is unrecoverable unless you hold a payload backup. Back up the keyfile container (it’s small, slot-encrypted) off-box at install time — that’s the only escape hatch. With a backup: restore it to the ESP/initramfs and re-enroll a passphrase.

Never shred the last working slot. passwd/rekey keep the payload, so data enrollment is untouched; only rotate regenerates the payload (and re-enrolls every disk — keyboot-install keyfile rotate <keyfile> --disk <dev>... --confirm).

2. keyboot image won’t boot

keyboot ships as two ESP slots (ADR 0009: /keyboot/{A,B}/) with a grubenv-driven default + a decrement-before-boot trial counter.

• A bad upgrade self-reverts: a staged-and-promoted slot that fails to unlock+import spends its one trial; the next boot auto-reverts to the known-good slot. No action needed; if you’re impatient, power-cycle.
• Force the good slot now: at the GRUB menu pick KEYBOOT - slot A (or B). Or from a recovery shell / booted OS, pin it: keyboot-install keyboot rollback --confirm (clears any trial) and confirm keyboot-install keyboot list.
• Both slots corrupt / ESP grub broken: boot the USB rescue image (usb-rescue.md) or the vendor rescue, then re-stage: keyboot-install install --kernel <vmlinuz> --initramfs <img> --confirm --regenerate-grub (the GRUB seam). On BIOS, GRUB lives per-disk — see §7 for the per-member grub-install.
• Default points at a non-slot (e.g. keyboot_slot=provision) — the grub.cfg now self-heals any non-A/B value to A (GL#44); on an older image, pick slot A at the menu, then keyboot-install keyboot provision --clear --confirm heals the grubenv.

3. Unlock fails

keyboot boots but panics/loops at unlock (keyfile open or LUKS open fails). You land in the recovery shell (SSH).

• Inspect: keyboot disk discover --json, keyboot keyfile info <keyfile>, keyboot keyfile list-slots <keyfile>. Confirm the data disks are present and LUKS (keyboot disk scan-luks).
• Manual unlock to triage (proves the secret + disks): keyboot unlock <keyfile> (opens the keyfile container then every LUKS disk as sn-<serial>). If that works, the boot-time failure is environmental (a disk not yet enumerated — a coldplug/timing issue) rather than a bad secret.
• A disk renamed/replaced: devices open as sn-<serial> so bus reorder is fine, but a replaced disk has a new serial and isn’t enrolled — see §7.
• Wrong/missing keyfile in the image: re-stage the image with the host keyfile (the keyfile is a trailing gzip member on the ESP initramfs; see hetzner-deploy.md “Re-flashing only the ESP”).

4. BE won’t boot

keyboot unlocked + imported, kexec’d the BE, and the BE failed (initramfs panic, no /sysroot, no network).

• Pick another BE: keyboot’s stage-7 honors keyboot.be=<dataset> on the cmdline (GRUB e to add it), the one-shot /esp/keyboot/once.next, then bootfs. Boot a previous BE and investigate.
• Roll back to a snapshot (ADR 0007, pre-boot — no booted OS needed):
  • rung 1 (safe): keyboot.rollback=<ds>@<snap>:ro — boots an ephemeral read-only clone; original untouched.
  • rung 3 (destructive): keyboot.rollback=<ds>@<snap>:destroy keyboot.rollback.confirm=1 — zfs rollback -r in place. See the runbook for the exact cmdline mechanics + recovery.
• A bad BE upgrade: if you used keyboot-be-upgrade, the original BE + @…-PREUPGRADE snapshot are intact — zpool set bootfs=<original> and reboot, or zfs destroy -r <clone> to discard the upgrade.

5. Stuck on the wrong entry

Box keeps booting something you didn’t intend (the GL#44 class: a stale once.next, a hand-set bootfs, a corrupt keyboot_slot, or a left-in grub.cfg cmdline edit).

• From a recovery shell / booted OS, mount the ESP (see the runbook) and inspect /esp/keyboot/{keyboot.env,once.next} and /esp/grub/grub.cfg.
• Clear a stuck one-shot: rm /esp/keyboot/once.next. Fix the default: zpool set bootfs=<good-be> rpool. Heal a corrupt slot: keyboot-install keyboot provision --clear --confirm (resets non-A/B keyboot_slot to A) — or just rely on the grub.cfg A/B guard (GL#44).
• Remember GRUB’s save_env is unreliable on the mdraid1 ESP — make grubenv changes from Linux through the assembled md (mirror-consistent), not from GRUB. keyboot’s runtime writes already do this.

6. Pool feature drift

A BE’s ZFS is older than the pool’s enabled features and refuses to import (ADR 0004: keyboot’s embedded OpenZFS is the floor; hosts upgrade keyboot before bumping pool features).

• keyboot’s own env carries the newest lockstep ZFS, so keyboot still imports — you’re not locked out; you reach the recovery shell.
• Pools are created -o compatibility=openzfs-2.1-linux so a BE on an older OpenZFS (e.g. Debian’s 2.1.x) can still import. If you bumped features past a BE’s ZFS: either upgrade that BE’s ZFS (zfs-dkms/kmod) to clear the gap, or recreate the BE on the supported floor. Don’t zpool upgrade ahead of the fleet’s BE ZFS.

7. Failed disk

Mirror member died. Each disk carries an ESP md member + a LUKS crypt payload (sn-<serial>); the pool vdev is over the crypt mappers.

1. Partition the replacement like the survivors (GPT: BIOS-boot + ESP member
   • crypt) — keyboot-install partition <disk> --wipe / the install-os partition templates.
2. Enroll the new crypt partition into the keyfile (same payload): keyboot-install keyfile enroll <new-crypt-part> --confirm, then keyboot unlock opens it as sn-<new-serial>.
3. Replace in the pool: zpool replace rpool sn-<old> sn-<new>; wait for resilver (zpool status).
4. Re-add the ESP md member: mdadm --add /dev/md/keyboot-esp <new-esp-part>; it resyncs the vfat ESP.
5. BIOS only: reinstall GRUB on the new disk — grub-install --target=i386-pc <new-disk> against the raw ESP member (array stopped first so the member mount doesn’t EBUSY; see hetzner-deploy.md). UEFI needs nothing extra (firmware reads BOOTX64.EFI off the md ESP).

8. Total loss / reinstall

Disks intact but the system is unbootable beyond repair, or you’re rebuilding.

• Boot a provisioning env: USB rescue (usb-rescue.md), the installed keyboot image in keyboot.mode=provision (if it still boots), or the vendor rescue + tools/hetzner-prep.sh (glibc rescue) / curl … /keyboot/install-os | sh (musl live env).
• Data intact, OS gone: import the pool (keyboot unlock → zpool import) and add a fresh BE without touching data: keyboot-install install-os <distro> --add-be --confirm. Reboot, pick it.
• Full reinstall (DESTROYS disks): keyboot-install install-os <distro> --disk <dev>... --confirm (or a --profile). You need the keyfile backup (§1) to keep the existing payload, else the data is gone.
• After any reinstall, re-stage the be-tools-bearing toolkit so the new BE has keyboot-be-upgrade/-rollback (GL#43 — now handled by all bootstrap paths).

See also: hetzner-deploy.md, hardware-test-runbook.md, decisions/ (ADRs 0003 handoff, 0004 ZFS floor, 0007 rollback, 0009 A/B slots).