0011 — ZFS module params: keyboot-env ARC cap vs OS-owned tuning

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-05

Context

There are two distinct zfs.ko load contexts on a keyboot host, and they have opposite tuning needs:

• keyboot’s unlock env — its own lockstep zfs.ko (ADR 0004). It opens LUKS, RO-imports the pool (stage-6-import.sh: zpool import … -o readonly=on), reads BE metadata, and kexecs. It touches almost no data, so it needs almost no ARC. Today it sets no zfs_arc_max, so the default (~50% of RAM) applies — on a big-RAM box that reserves a lot of RAM during a step that reads kilobytes.
• the booted OS/BE — the distro’s zfs.ko, running the real workload. Its ARC is a per-host, workload-dependent production tuning decision.

These must not share a policy: a fleet-uniform small cap is right for the first, wrong for the second.

Decision

keyboot’s unlock env caps zfs_arc_max to 512 MiB, uniform across the fleet. stage-1-early.sh writes /etc/modprobe.d/keyboot-zfs.conf (options zfs zfs_arc_max=<bytes>) before zfs.ko loads — so the cap applies whether zfs is in /etc/keyboot/modules or autoloaded by zpool import (the kernel’s autoload runs userspace modprobe, which reads modprobe.d). The value is resolved by the pure helper keyboot_zfs_arc_max_bytes (init/lib/zfs.sh): empty → 512 MiB; accepts a raw byte count or a K/M/G suffix; malformed → warn-to-default.

Override: keyboot.zfs_arc_max=<v> on the kernel cmdline (e.g. a low-RAM box, or a recovery shell doing a scrub that wants more ARC). Parameterized with a sensible default — the operator can override, but doesn’t have to.

The booted OS owns production ARC. keyboot never writes the installed OS’s /etc/modprobe.d/zfs.conf. Production ARC is per-host and workload- dependent → Ansible-managed (a baseline + per-host override). install-os bakes no OS-level ARC default; the installed system uses the distro/ZFS default until Ansible tunes it. (Consistent with “parameterize choices, defer opinionated defaults”.)

The zfs.ko binaries are already independent per context (keyboot’s lockstep build vs the distro’s) — this ADR is only about params, and confirms they are owned separately too: keyboot-uniform-and-small here, OS-and-Ansible there. Not “per-OS params inside keyboot.”

Alternatives considered

• A fraction of RAM (e.g. 1/16) for the keyboot cap — rejected: needs RAM detection in the initramfs and is less predictable/uniform; a metadata-only RO import doesn’t benefit from scaling ARC with host RAM.
• No cap (status quo) — rejected: ARC balloons toward ~50% RAM during the unlock on big-RAM hosts for no benefit.
• keyboot setting an OS-level ARC default — rejected: production ARC is fleet policy; baking it into the installer is exactly what Ansible should own.
• 256 MiB cap — considered; 512 MiB chosen for a little headroom on very large pools’ metadata / a recovery-shell scrub, still tiny + uniform.

Consequences

• keyboot’s unlock-env RAM footprint is bounded + uniform regardless of host RAM; recovery-shell heavy ops override via keyboot.zfs_arc_max=.
• The cap is set before any zfs load path, so it can’t be missed by autoload.
• No change to the booted OS — its ARC is whatever the distro/Ansible sets.
• init/lib/zfs.sh is the home for future keyboot-env module params (it already pairs with the cmdline override plumbing).