0015 — Ansible roles: substrate, unlock, deploy

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-05

Context

keyboot’s fleet story needs Ansible (SPEC §3: “ansible manages the repo definitions”). The primitives exist — keyboot-install install-os, the BE verbs (be clone / boot-next / promote / destroy / gc), the automation keyslot (ADR 0013) and the keyboot_unlock role (GL#19). This ADR pins the role set and, crucially, the deploy state machine so a remote upgrade is safe by construction.

Decision

Three roles under ansible/:

keyboot_unlock (shipped, ADR 0013) — feed the vaulted slot-2 automation passphrase to keyboot’s pre-unlock dropbear over SSH. The reboot/unlock building block the other roles include.
keyboot_substrate — drive a host already in a rescue env through install-os to a fresh encrypted-ZFS system: stage keyboot-install (the signed curl|sh, ADR 0008), run install-os <distro> --disk … --confirm, reboot, then keyboot_unlock + wait for the OS. Getting the host into rescue is vendor-specific (Hetzner rescue, netboot, IPMI) and stays an operator prerequisite — the role asserts rescue reachability rather than pretending to be generic.

keyboot_deploy — the safe remote-upgrade loop:

be clone (running -> new)        # a fresh BE off the running one
[prepare hook]                   # operator mutates the clone (pkgs/config); optional
be boot-next --once <new>        # boot-ONCE: not yet the permanent default
reboot
keyboot_unlock                   # automation-slot unlock of the keyboot env
wait for the OS sshd
run health_cmd on the new BE     # operator-defined; exit 0 = healthy
  healthy  -> be promote <new>   # make it the permanent default
  unhealthy-> do NOT promote     # leave it; boot-once auto-reverts next reboot

Health gate = an operator command

“Healthy” is whatever the operator’s workload says — so keyboot_deploy_health_cmd is a command run on the booted new BE, exit 0 = promote. Default systemctl is-system-running --wait (systemd BEs). openrc BEs (Alpine/Gentoo) must override it (e.g. rc-status -c or a service-specific probe) — documented in the role defaults. Parameterize-with-a-default; no baked fleet policy.

Revert = keyboot’s boot-once net, not the role

On an unhealthy BE the role simply doesn’t promote. Because the clone was booted with boot-once, the next reboot returns to the old (still-bootfs) BE automatically. This is strictly safer than the role actively reverting: it also covers the BE that won’t boot at all or hangs before SSH — cases an “Ansible sets bootfs back” approach can’t reach because Ansible never gets a connection. The role may optionally reboot to enact the revert immediately; the safety itself is keyboot’s, not Ansible’s.

Alternatives considered

Role-driven revert (set bootfs back + reboot) — rejected as the primary mechanism: it can’t recover a BE that never reaches SSH; boot-once already handles every failure mode including “won’t boot”. Kept only as an optional immediate enactment of the revert boot-once would do anyway.
HTTP-only health gate — rejected as the default: not universal (DB/host with no endpoint); a command covers HTTP (curl) and everything else.
Generic keyboot_substrate incl. rescue entry — rejected: rescue entry is irreducibly vendor-specific; faking it would be a lie. The role starts from “host is in rescue and reachable”.

Consequences

A remote upgrade can’t strand a box: worst case it boots the new BE, fails the gate (or never comes up), and the next reboot is the old BE. Pairs with ADR 0009 (keyboot self-upgrade A/B) — different layer, same boot-once discipline.
The deploy loop depends on the automation keyslot (ADR 0013) for unattended unlock and on be boot-next --once honoring boot-once semantics (verify in the deploy round-trip test, the rung-1/promote follow-up).
keyboot_substrate’s rescue-entry prerequisite should grow per-vendor helper docs over time (Hetzner first, since that’s the validated platform).

Keyboard shortcuts