0010 — Multi-disk vdev topology grammar (data + aux vdevs)

Status: accepted (2026-06-22) — ratified by maintainer; proposed 2026-06-05

Context

install-os picks one of three fixed pool layouts from disk count (tools/keyboot-install-os/lib/vdev.sh: 1→single, 2→mirror, 3+→raidz1), each a single top-level vdev. That can’t express what real boxes need: an 8-disk host should be describable as RAID10 (4×2-mirror), raidz2(8), 2×raidz1(4), etc., and many hosts want a special (metadata) vdev, an SLOG, an L2ARC, or a hot spare. SPEC §7/§13.13 endorse arbitrary topology + aux vdevs; this ADR fixes the grammar, the disk→role mapping (which drives partitioning), and the validation so the installer can build any layout over the sn-<serial> crypt mappers without re-architecting later.

Partitioning today is uniform (every pool disk: 1 MiB bios-boot + 512 MiB ESP member + LUKS payload; the ESP is an mdraid1 across all disks, partition.rs). Aux devices break that assumption — an L2ARC SSD is not a bootable data disk — so the topology must decide each disk’s partition role, not just the zpool string.

Decision

Grammar

A pool is a sequence of data vdev groups plus optional aux vdevs:

--topology "<group> | <group> | ..."     # data vdevs (pipe-separated)
--special  "<group>"                      # metadata/small-blocks vdev (one group)
--log      "<group>"                      # SLOG
--cache    "<devspec>"                    # L2ARC (single-disk vdevs)
--spare    "<devspec>"                    # hot spares

Each group is <type> <devspec> where type ∈ {stripe, mirror, raidz1, raidz2, raidz3} and <devspec> is one of two forms (both supported, not mixed within a single invocation):

• Count form — an integer: mirror 2, raidz2 8. Disks are consumed from the resolved sn-<serial> list in discovery order, group by group.
• Explicit form — a device list: mirror sn-a sn-b. Names must resolve to members of the discovered set; you control exactly which physical disk lands in which vdev (fault-domain placement).

stripe N (or stripe <devs>) expands to N separate single-disk top-level vdevs (a ZFS stripe — no redundancy). Count and explicit forms may not be mixed in one --topology (error), to keep disk accounting unambiguous.

Examples:

--topology "mirror 2 | mirror 2"                 # 4-disk RAID10
--topology "raidz2 8" --special "mirror 2" --log "mirror 2" --cache "sn-z"
--topology "raidz1 sn-a sn-b sn-c | raidz1 sn-d sn-e sn-f"

The legacy --template single|mirror|raidz1 stays as sugar (maps to a single group); count-from-disk-count remains the default when neither is given.

Disk → partition role (this is the new partitioning rule)

Each discovered disk gets exactly one role, which fixes its partition layout:

Rationale: a spare may be auto-promoted to replace a data disk, so it must be partitioned identically (incl. an ESP member, to preserve boot redundancy). special/log/cache hold pool data/metadata and so are encrypted (crypt) — but they are never booted, so they carry no bios-boot/ESP. The ESP mdraid1 therefore spans data + spare disks only (boot redundancy follows the bootable disks, not the cache SSD). Every aux device is still opened as sn-<serial> and keyed by the same keyfile payload — encryption is uniform even though partitioning is not.

zpool expansion

zpool create -f -o ashift=12 -o compatibility=openzfs-2.1-linux <pool> \
    <data groups...> \
    [special <special group>] [log <log group>] \
    [cache <cache devs>] [spare <spare devs>]

over the resolved /dev/mapper/sn-<serial> crypt paths.

Validation

• Per-vdev minimums (ZFS hard floors): mirror ≥2, raidz1 ≥2, raidz2 ≥3, raidz3 ≥4. Warn (not error) below the recommended width (raidz1 3, raidz2 4, raidz3 5).
• Disk accounting: every discovered disk is assigned exactly once across all data + aux vdevs. Unassigned or double-assigned disks are an error. (Count form: the counts must sum to the disk total; explicit form: a bijection.)
• No-redundancy guard (operator decision: warn + require --confirm): any stripe, single-disk data vdev, or otherwise non-redundant top-level data vdev prints a loud warning in the plan; the existing --confirm is the gate (no extra flag). A non-redundant --special vdev gets an extra-loud warning — losing it loses the whole pool, not just redundancy — but still proceeds under --confirm per the uniform policy.
• At least one data vdev is required; aux-only is an error.

Implementation locus

The grammar parser + validator + role mapping live in Rust (keyboot-install), exposed as keyboot-install topology plan --json and consumed by the shell orchestrator — mirroring the existing disk-discovery/partition split (Rust is the single source of truth, the orchestrator shells to it). partition.rs gains a per-disk role so it emits the right partitions and the ESP array spans only data+spare. lib/vdev.sh’s three-template path is reimplemented in terms of the grammar (kept as sugar).

Alternatives considered

• Expanded named menu (raid10/raidz2/raidz3 templates, no grammar) — rejected: can’t express 3×raidz1(4) or aux vdevs; just postpones the grammar.
• Count form only — rejected: loses fault-domain control (which disk in which mirror), which matters on multi-controller/multi-enclosure boxes.
• Aux vdevs as a follow-up — considered; operator chose to include them now, accepting the partitioning rework (the crypt-only role) this pass.
• Grammar in shell (vdev.sh) — rejected: two device forms × multiple groups × aux × disk-accounting is too error-prone for bash; Rust is testable and already owns partitioning.

Consequences

• Partitioning is no longer uniform: partition.rs and the partition --json contract grow a per-disk role; the ESP member set is a subset of disks. The install-os boot path (CI-verified) must be re-proven with a multi-disk + aux QEMU scenario before this is trusted on real hardware.
• Encryption stays uniform: every disk (data/spare/special/log/cache) is LUKS-keyed by the keyfile payload, so an aux SSD never holds plaintext pool data.
• Special-vdev redundancy is the sharp edge: a non-redundant special vdev is a pool-loss footgun; we warn extra-loud but honour the warn+confirm policy. Operators wanting a hard stop can be given --refuse-no-redundancy later.
• L2ARC encryption: cache is crypt-backed; ZFS L2ARC of an encrypted dataset is itself encrypted, so this is belt-and-suspenders but consistent and costs nothing operationally.
• Follow-ups: a multi-disk+aux QEMU install gate; --refuse-no-redundancy opt; per-vdev ashift / --special-small-blocks tuning (deferred).