USB rescue/install image

ci/build-usb.sh turns the keyboot artifact (the initramfs from ci/build-image.sh plus its matched vmlinuz) into a self-contained, bootable raw disk image (.img). Written to a USB stick, it boots keyboot in rescue (or install) mode — the on-site / no-netboot counterpart to the netboot path (SPEC §1, §2). It is the same artifact as the netboot/boot image; the mode is chosen on the GRUB kernel cmdline.

Download the hosted image (no build needed)

A prebuilt, signed image is published on the package channel. It ships the full kernel module tree (every NIC/HBA driver) plus ethtool + lspci, so it is the quickest way to boot a box and see which network drivers bind.

base=https://packages.osterman.co/keyboot/v0.1.0
curl -fLO $base/keyboot-rescue-x86_64.img.gz
curl -fLO $base/keyboot-rescue-x86_64.img.gz.minisig

# Verify against the project signing key (same pinned key the curl|sh installer uses)
minisign -Vm keyboot-rescue-x86_64.img.gz \
  -P RWTrhJW+9h/jDQZkoGzVdqZnMvkRwjdmMeeimbc39fj/kyAg54+2J9BZ \
  -x keyboot-rescue-x86_64.img.gz.minisig

# Decompress + write to a stick (see "Write it to a stick" for the device caveat)
gunzip -c keyboot-rescue-x86_64.img.gz | sudo dd of=/dev/sdX bs=4M conv=fsync status=progress

The image boots straight into keyboot rescue mode. SSH-in works if the image was built with an authorized key baked in (the hosted one trusts the fleet keyboot_ci_deploy key); otherwise use the console.

Layout (firmware-agnostic, SPEC §13.13)

One GPT image boots on both UEFI and BIOS-MBR:

GRUB presents two menu entries: rescue (keyboot.mode=rescue, default) and unattended install (keyboot.mode=install).

One-command build (recommended)

ci/make-usb.sh is the turnkey front-end: it runs the whole kernel→initramfs→.img chain and surfaces the choices you actually make at a stick. It stops at a dd-able raw image (no gzip/sign — that’s the publish path, ci/build-usb-image.sh). Runs on an Alpine build host, as root:

sudo ci/make-usb.sh --ssh-key you.pub --mode rescue --modules all
# -> ci/qemu/out/keyboot-usb.img
ci/make-usb.sh --check --ssh-key you.pub      # preflight only (no root)

Options: --ssh-key FILE / --ssh-dir DIR (an authorized key is required), --mode rescue|install|boot|memtest, --modules all|none, --zfs/--no-zfs, --install-tools, --keyboot-bin FILE, --kernel FILE / --build-kernel, --out FILE, --size MIB, --memtest-dir DIR. It reuses a prebuilt kernel/out/vmlinuz unless --build-kernel.

Build from any distro or vendor rescue

make-usb.sh needs an Alpine host (the initramfs userland comes from apk). To build straight from a box’s own Debian/Ubuntu rescue (or any working distro), use ci/make-usb-anyhost.sh: it apk.static-bootstraps a throwaway Alpine rootfs, builds the keyboot binary in it, and runs make-usb.sh inside that chroot — same flags pass through. The host needs only root, loop devices, curl/wget, and tar, plus a checkout with a prebuilt kernel/out/vmlinuz.

sudo ci/make-usb-anyhost.sh --ssh-key you.pub --mode rescue   # from any distro/rescue

(On an Alpine host it just exec’s make-usb.sh. The image-assembly step needs a real /dev with loop devices — it won’t run in an unprivileged container with a tmpfs /dev.)

Build (manual, step by step)

Requires root, loop devices, and grub-install with both the i386-pc and x86_64-efi targets, plus sgdisk, mkfs.vfat (dosfstools), and losetup (util-linux). On Alpine — note sgdisk is its own package (the gptfdisk package ships gdisk, not sgdisk):

apk add grub grub-bios grub-efi dosfstools sgdisk util-linux

Build the inputs first (kernel + initramfs), then the image:

bash kernel/build.sh                       # -> kernel/out/vmlinuz
INCLUDE_MODULES=all INCLUDE_ZFS=yes INCLUDE_INSTALL=yes \
  AUTHORIZED_KEYS=you.pub bash ci/build-image.sh   # -> ci/qemu/out/keyboot-initramfs.cpio.gz

sudo ci/build-usb.sh                       # -> ci/qemu/out/keyboot-rescue.img

Preflight without building (checks tools + inputs, no root):

ci/build-usb.sh --check

Knobs (env): INITRAMFS, KERNEL, OUT, MODE (rescue|install), SIZE_MIB (default auto), GRUB_TIMEOUT, CONSOLE.

Write it to a stick

dd the image to the raw USB device (not a partition). This destroys everything on the target — double-check the device node.

lsblk                                       # find your stick, e.g. /dev/sdX
sudo dd if=ci/qemu/out/keyboot-rescue.img of=/dev/sdX bs=4M conv=fsync status=progress
sync

Boot the host from the stick. On UEFI it loads /EFI/BOOT/BOOTX64.EFI; on BIOS it loads GRUB from the protective MBR + the EF02 partition. Either way you land in the GRUB menu, then keyboot rescue mode.

Checking network drivers on a box

This is the headline use of the rescue stick. On boot in rescue mode keyboot runs keyboot-nic-report automatically and prints it to every console (serial + KVM video), then drops you to a shell. The report shows, per interface: name, PCI id, the bound kernel driver, link state, and speed — plus an lspci network-class listing that surfaces controllers with no driver bound (the “hardware present, kernel doesn’t drive it” case, which never shows up under /sys/class/net), and the recent NIC/driver dmesg lines.

Re-run or dig deeper from the shell:

keyboot-nic-report          # the same summary, on demand
lspci -nnk | grep -iA3 -e Ethernet -e Network   # controller + "Kernel driver in use"
ethtool enp1s0              # link, speed, driver/firmware versions
ip -br link                 # all links + up/down at a glance
dmesg | grep -iE 'eth|enp|link|firmware'        # probe/firmware errors
modprobe <driver>           # try loading a driver the autoloader missed

If a NIC shows up in lspci but has no enpXsY interface and no “Kernel driver in use”, the kernel has no driver for it — note the [vendor:device] id from lspci -nn and add/enable the matching driver in the kernel config (kernel/config-drivers-x86_64, regenerated by kernel/gen-drivers-config.sh).