Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Operator guide (day-2)

Routine operations on a healthy keyboot fleet. For broken hosts see disaster-recovery.md; for real-hardware test procedures see hardware-test-runbook.md. Commands assume the host-side keyboot-install and the runtime keyboot (the BE also carries the be-tools: keyboot-be-upgrade/-rollback/keyboot-autosnap/keyboot-snap).

1. Update keyboot itself (A/B self-upgrade, ADR 0009)

Is an update available? keyboot-update-check compares the installed boot-image version against the channel’s latest pointer — for admins (run it) and cron/scripts (exit code: 0 up-to-date, 10 update-available, 1 error):

keyboot-update-check            # "keyboot v0.1.4 is up to date (latest v0.1.4)"
keyboot-update-check --quiet || echo "update available"   # cron-friendly
keyboot-update-check --json    # {"status":"...","installed":"...","latest":"..."}

keyboot updates are fail-safe by construction: stage the new image into the inactive ESP slot and arm a one-attempt trial; if it doesn’t unlock+import the next boot auto-reverts to the known-good slot. A good trial commits itself.

# 1. Get the new artifact (signed package channel — packages.osterman.co):
apk upgrade keyboot keyboot-install        # Alpine (apk ships both)
apt update && apt install --only-upgrade keyboot   # Debian (runtime only)
# (or the raw installer: curl -fsSL https://packages.osterman.co/keyboot/install | sh)
# NOTE: keyboot-install (the FFI installer used in step 2) is Alpine/rescue-only;
# it is NOT packaged for Debian/Gentoo. There, build it natively
# (tools/hetzner-prep.sh) or run the staging step from the keyboot rescue env.

# 2. Stage it into the inactive slot + arm the trial:
keyboot-install install --slot inactive --promote --version-tag <ver> \
    --kernel <new-vmlinuz> --initramfs <new-initramfs.img> --confirm
keyboot-install keyboot list               # see slots/roles/trial

# 3. Reboot. The trial boots once; on a clean unlock+import it COMMITS (becomes
#    the known-good default). If it fails, boot 2 auto-reverts. No babysitting.
  • Inspect/serve: keyboot-install keyboot list.
  • Arm/disarm manually: keyboot-install keyboot promote <A|B> --confirm / … rollback --confirm (the panic button — pins the known-good slot).
  • One-shot provision boot (add a distro from keyboot’s own env, no vendor rescue): keyboot-install keyboot provision --confirm; reboot, it enters keyboot.mode=provision once then reverts. See §4.

Validate a release across a real upgrade in QEMU first (qemu:ab-revert); the fleet should upgrade keyboot before bumping pool features (§5, ADR 0004).

2. Boot-environment lifecycle (ADR 0005/0006/0007)

Never upgrade in place — clone the BE, upgrade the clone, boot it.

# Scheduled snapshots (keyboot-autosnap = the cron engine: create + retention):
#   <ds>@<YYYY-MM-DD-HHMM>Z-<LABEL>  (UTC, ADR 0006)
keyboot-autosnap --label DAILY --keep 14   # one cron line per frequency/label

# Manage snapshots interactively (keyboot-snap = the admin front-end):
keyboot-snap list [--label L] [--json]     # inspect (name carries the UTC time)
keyboot-snap create [--label L]            # on-demand snapshot now (default MANUAL)
keyboot-snap prune --label L --keep N      # retention on demand (skips held)
keyboot-snap hold|release <UTC-LABEL>      # protect a point-in-time from pruning
keyboot-snap destroy <UTC-LABEL> --confirm # remove a point-in-time (recursive)

# Cross-release / risky upgrade (clone-chroot, original untouched):
keyboot-be-upgrade -- sh -c '<distro upgrade commands>'   # prints the clone name
# then boot it ONCE (auto-reverts to bootfs if it hangs):
#   echo 'be=<clone>' > /esp/keyboot/once.next   (ESP mounted; see the runbook)
# keep it permanently once happy:
keyboot-install be promote <clone> --pool <pool> --confirm  # set bootfs (--pool defaults to rpool)

# Boot a specific BE once (boot-once marker):
keyboot-install be boot-next <dataset>

# Roll back (pre-boot rungs, ADR 0007) — see the runbook for cmdline mechanics:
#   keyboot.rollback=<ds>@<snap>:ro        (ephemeral ro clone; zero side effects)
#   keyboot.rollback=<ds>@<snap>:destroy keyboot.rollback.confirm=1   (in place)
keyboot-be-rollback ...     # the BE-side helper (clone/destroy + ephemeral GC)

3. Key management (SPEC §7, ADRs 0003/0013)

One keyfile container, multi-slot; every slot opens the same 32-byte payload, so data-disk enrollment is untouched by passphrase changes. Slot map: 0 daily, 1 recovery, 2 automation, 3–7 reserved.

keyboot keyfile list-slots <keyfile>                       # read-only inventory
keyboot-install keyfile add-recovery <keyfile>             # slot 1 (do this early!)
keyboot-install keyfile enroll-automation <keyfile>        # slot 2 (Ansible unlock)
keyboot-install keyfile passwd <keyfile> --slot <N>        # change a slot in place
keyboot-install keyfile rekey  <keyfile> --retire-slot <N> --new-slot <M> --confirm
        # remote-safe: add new -> prove it opens -> wipe old (never zeroes a
        # working slot). The Ansible bootstrap->true-secret flow.
keyboot-install keyfile rotate <keyfile> --disk <dev>... --confirm
        # DISTINCT: regenerates the payload + re-enrolls every disk.

Back up the keyfile container off-box. It’s small and slot-encrypted; it’s the only recovery path if every passphrase is lost (§1 of disaster-recovery).

Unified front-end + operator SSH keys (keyboot-keys):

# LUKS slots (routes to the tools above; needs the keyfile present, e.g. the
# keyboot provision/rescue env, or pass --keyfile):
keyboot-keys luks list | info | add-recovery | enroll-automation | passwd | rekey | rotate

# Operator SSH keys for the UNLOCK ENVIRONMENT (the dropbear you SSH into to
# enter the passphrase). Edits an editable overlay on the ESP
# (/esp/keyboot/authorized_keys) that stage-3-ssh merges with the image-baked
# keys — so you rotate operator keys WITHOUT rebuilding the image:
keyboot-keys ssh list
keyboot-keys ssh add  "ssh-ed25519 AAAA... ops@laptop"   # or a path to a .pub
keyboot-keys ssh remove ops@laptop                       # substring match
# New keys take effect on the next boot into the keyboot unlock env.

Prefer a browser to SSH for unlock + BE selection? See §8 (web unlock UI).

4. Add a host / add a distro (SPEC §13)

# Fresh install (DESTROYS the target disks). Dry-run by default; --confirm writes.
keyboot-install install-os <debian|gentoo|alpine> --disk <dev>... \
    --hostname h --authorized-keys you.pub --confirm
# or declaratively (SPEC §13.7):
keyboot-install install-os <distro> --profile profile.yaml --confirm

# Bootstrap from any live env (no keyboot rescue needed):
curl -fsSL https://packages.osterman.co/keyboot/install-os | sh -s -- --run debian --confirm
# (musl live env; for a glibc Hetzner rescue use tools/hetzner-prep.sh)

# Multi-distro on ONE pool — add a BE without partition/keyfile/pool/GRUB churn:
keyboot-install install-os <distro> --add-be --pool rpool --confirm

# The provisioning model: boot the installed keyboot image into provision mode
# (keyboot.mode=provision; arm with `keyboot-install keyboot provision --confirm`)
# -> it unlocks + imports rpool read-write + drops to an install-os-ready shell.
# Add a BE there, reboot, pick it. keyboot's own env IS the rescue — never the
# vendor rescue after the one-time install.

Headless reachability is automatic: all three distro plugins bring up DHCP + sshd and install --authorized-keys so the booted OS is SSH-reachable. Pass --authorized-keys for --add-be too.

5. Pool health & features (ADR 0004/0011)

zpool status rpool            # health, resilver/scrub progress, errors
zpool scrub rpool            # schedule periodically (cron/systemd-timer)
zpool get bootfs rpool        # current default BE
  • Feature-flag ordering (ADR 0004): keyboot’s embedded OpenZFS is the floor. Upgrade keyboot across the fleet before zpool upgrade — never enable pool features a host’s keyboot (or a BE’s ZFS) can’t read. Pools are created compatibility=openzfs-2.1-linux so older-OpenZFS BEs still import.
  • ARC (ADR 0011): keyboot’s unlock env caps zfs_arc_max at 512 MiB (override keyboot.zfs_arc_max= on the cmdline); the booted OS owns production ARC tuning (/etc/modprobe.d/zfs.conf, per-host, Ansible-managed).

6. Disks (planned: grow/scrub; failed: see disaster-recovery §7)

  • Replacing a failed mirror member: disaster-recovery.md §7 (partition → keyfile enroll → zpool replace sn-<old> sn-<new> → mdadm --add the ESP member → BIOS grub-install per member).
  • Topology for new pools: keyboot-install topology plan + --topology/--special/ --log/--cache/--spare on install-os (ADR 0010).

7. Diagnostics

  • RAM: bootable memtest86+ (memtest.md; keyboot.mode=memtest or the GRUB entry) for ALL physical RAM; the recovery-shell memtester for a quick kernel-RAM check.
  • Boot debugging: keyboot.debug=1 surfaces stage logging on the console; keyboot.stop-after=<N> halts after stage N (SSH stays up ≥ stage 3).

8. Web unlock UI (ADR 0016)

A browser alternative to SSHing the dropbear to type the passphrase / pick a BE. Opt-in and additive — dropbear stays the default; nothing changes unless you set keyboot.web=1 on the keyboot menuentry cmdline. The release runtime binary (keyboot-x86_64-musl, v0.1.19+) is built with both phases compiled in. Same unlock path as everything else: the page POSTs the passphrase to the stage-4 FIFO, exactly like the SSH askpass.

Phase 1 — loopback + SSH tunnel (no setup, SSH-grade security). With keyboot.web=1 and no baked cert set, keyboot serves the UI on 127.0.0.1:8090 only. Reach it through an SSH tunnel (the boot key permits forwarding only when keyboot.web=1):

ssh -L 8090:localhost:8090 root@<host>     # then open http://localhost:8090

Security is identical to today (dropbear is still the only thing on the network).

Phase 2 — network HTTPS + mTLS (no tunnel). Bake a TLS cert set into the image and keyboot serves the UI directly on 0.0.0.0:8443 with mutual TLS — the server cert is trusted like an SSH host key, and a client must present a CA-signed cert (the authorized_keys analog) or the handshake is refused.

# 1. Generate the material (server cert + client CA + an operator client bundle).
#    SAN must list every name/IP you'll reach the box by.
SAN="IP:<host-ip>,DNS:<hostname>" OUT=web-certs ci/gen-web-certs.sh

# 2. Bake server-cert.pem + server-key.pem + client-ca.pem into the keyboot image
#    (like the SSH host key). They land at /etc/keyboot/web/ in the initramfs:
WEB_TLS_DIR=web-certs INCLUDE_MODULES=all AUTHORIZED_KEYS=you.pub ci/build-image.sh
#    then stage the rebuilt image into a slot (§1) and reboot into it.

# 3. On your client: import web-certs/client.p12 into the browser/OS keystore
#    (the mTLS identity), and trust web-certs/server-cert.pem once — or, before
#    trusting, compare its SHA-256 to the fingerprint keyboot prints on the
#    console (serial/KVM/IPMI) for a manual no-MITM proof:
openssl x509 -in web-certs/server-cert.pem -noout -fingerprint -sha256

Then browse to https://<host>:8443 and unlock. Without a valid client cert no request is ever served; rotate the cert set the same way you rotate the SSH host key (rebuild + restage the image). Keep web-certs/client-ca-key.pem offline — it signs future client certs.

See also: decisions/ (the ADRs behind each of the above).