0004 — OpenZFS version & pool-feature lifecycle (keyboot is the ZFS floor)

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-04

Context

A pool that has enabled OpenZFS feature flags can only be imported by a ZFS implementation that understands them. If the recovery environment’s ZFS is older than the pool’s enabled features, the pool won’t import — the classic “recovery USB’s zfs is too old to import my pool” lockout, and a continent-away brick.

keyboot is the recovery environment. It carries its own zfs.ko + userland (SPEC §3, build via zfs/build.sh, bundled by ci/build-image.sh INCLUDE_ZFS=yes), independent of whatever the booted BE ships. Source/rolling distros make this sharp: Gentoo can keyword-unmask and emerge a newer sys-fs/zfs, and an operator can run zpool upgrade and silently push the pool past what keyboot can import. SPEC §3 already locks the doctrine — “ZFS coupling tracks upstream OpenZFS in lockstep; hosts upgrade keyboot before bumping pool features.” This ADR pins the concrete guards that make the doctrine hold.

Key fact that bounds the risk: upgrading the ZFS software is safe; only enabling pool features is dangerous. OpenZFS never auto-enables features, and no distro’s package upgrade runs zpool upgrade. Newer ZFS imports an older-feature pool fine. Lockout happens only when features get enabled beyond the recovery env’s capability.

This ADR protects keyboot’s recovery floor on two fronts: the ZFS feature lockout above, and the integrity of the ESP — the mdraid1 vfat that holds keyboot’s own kernel/initramfs + GRUB. The ESP is the only unencrypted boot artifact; if a stray write corrupts it, the recovery floor itself is gone, so it gets the same “protect by default, change only deliberately” treatment.

Decision

1. keyboot’s embedded OpenZFS is the floor. The pool’s enabled feature set must never exceed what keyboot can import. Release ordering: upgrade keyboot (its embedded zfs.ko/userland) before bumping pool features.
2. Pin compatibility= on every pool (default openzfs-2.1-linux or a keyboot-blessed set; the compat files are bundled into the rescue image and shipped in the BE). With the property set, zpool upgrade can only enable features within that set — so the pool structurally cannot outrun keyboot as long as keyboot ≥ the pinned baseline. This is the real safety net.
3. zpool upgrade is gated, never automatic. Ship a wrapper shim (/usr/local/sbin/zpool, earlier in PATH than the real binary) in the BE that intercepts only the upgrade subcommand: it checks the resulting feature set against keyboot’s capability / the pinned compatibility and refuses or warns, then execs the real zpool (located by absolute path, not PATH, to avoid recursion) for upgrade once cleared and for all other subcommands untouched. This is an accident guardrail, not a security control — explicit /sbin/zpool calls and automation bypass it; guard (2) is the structural guarantee.
4. Boot-time feature-vs-capability check. At import, keyboot compares the pool’s required features against its own zfs.ko and warns loudly (and declines a silent rw import that would worsen the gap) if the pool has outrun it — surfacing the problem at the next boot, while the pool is still importable, instead of at the moment it bricks.
5. Mount the ESP read-only by default. The ESP holds keyboot + GRUB — the recovery floor itself — and is not needed at runtime (verified: it isn’t even mounted in a normally-booted BE). So the BE mounts it ro by default; a gated keyboot/GRUB update is the only thing that remounts it rw, does its write, and drops back to ro. This blocks a stray dd, a distro grub-install/os-prober, or a fat-fingered /boot/efi write from corrupting the one artifact that must survive for recovery to work. (Pairs with the keyboot self-upgrade boundary — A/B keyboot images on the ESP so even a botched keyboot update is recoverable — now decided in ADR 0009, which carves out one bounded ESP-rw exception: keyboot’s trial-commit grubenv write at unlock+import.)

Alternatives considered

• Discipline only (no pin, no shim): relies on the operator remembering the ordering; one zpool upgrade bricks recovery. Rejected.
• Hard-block zpool upgrade entirely: too rigid — feature upgrades are legitimate after keyboot is upgraded. Rejected in favour of gate-with-check.
• Shim zfs to police zfs snapshot/other subcommands: breaks replication, sanoid, zfs send, and our own snapshot tool. The upgrade gate lives on zpool; the zfs side is warn-only at most (see ADR 0006).
• TPM/attested version negotiation: overkill for v1. Deferred.

Consequences

• A keyboot release that bumps OpenZFS must ship before fleet pools enable the new features — a documented release-ordering rule.
• The shim is best-effort (PATH order); the compatibility= pin is the binding guarantee. Both, plus the boot-time check, are defense-in-depth.
• keyboot’s import path (stage-6/7) gains the feature-vs-capability comparison + a loud warning channel.
• Gentoo unmask + emerge of newer ZFS is safe; the discipline is “bump keyboot’s embedded ZFS in the same step, and treat zpool upgrade as gated.”
• Ties into the upgrade story (SPEC §14); the gate belongs with keyboot-install upgrade.