Skip to content

VFIO (experimental): berth vfio status and berth vfio run - #130

Merged
T-Crypt merged 10 commits into
mainfrom
T-Crypt/glm-v2-vfio-cli
Oct 21, 2026
Merged

T-Crypt merged 10 commits into
mainfrom
T-Crypt/glm-v2-vfio-cli

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Second half of VFIO passthrough support, after #129. Berth coordinates; the operator's own script switches drivers (with sudo) and runs the VM. Berth never binds a driver, writes to sysfs, asks for a password or stops a process.

  • berth vfio status shows IOMMU on or off, the card's IOMMU group with each function's driver (anything besides the card's own functions blocks passthrough), the processes holding /dev/nvidia* or the card's DRM nodes, and what berthd reports. It changes nothing. It takes an address, because nvidia-smi cannot see a card bound to vfio-pci.
  • berth vfio run [--reason NAME] -- CMD:
    • refuses when IOMMU is off or the group is not clean;
    • takes the VFIO hold and lets engines drain;
    • refuses, naming the PIDs, if any process still holds the card;
    • runs the script with the terminal attached and BERTH_VFIO_DEVICES set;
    • releases the hold only once the card is back on its host driver and answering nvidia-smi.

Found in review and fixed here:

  • berthd held the card itself. Its probe is one long-running nvidia-smi -lms, so VFIO (experimental): berthd shows a card passed through to a VM and stays off it during a switch #129's skipped reads still left the device open, and the unbind would block on berthd. Stream.Pause kills the loop for the whole hold, the broker calls it every tick, and vfio run refuses a probe still open after the settle time.
  • sudo would hang. The script was started in its own process group, as berth run does for engines. A process outside the terminal's foreground group that reads the terminal is stopped (SIGTTIN), so sudo's prompt would freeze. I reproduced it with a pty, where the child sat in state T. The script now runs in berth's own group.
  • Kernel threads flooded the openers list. On aphotic they were 400 of 430 unreadable entries. They are skipped now; root daemons are still listed as not proven clear.
  • The docs. They claimed a real run that has not happened. The example script now moves each function with driver_override, which the card's audio function needs (it is not bound to nvidia). Checked with sh -n and shellcheck.

Proof:

  • Tests with a temp sysfs tree, a fake /proc and a stand-in nvidia-smi: each refusal, the happy path, the env var, the hold kept while the card is out, an unreadable process, kernel threads skipped. TestStreamPauseStopsTheLoop (a real child process: killed on pause, not restarted, restarted on resume). TestVFIOHoldPausesTheProbe. TestVfioRunRefusesALingeringProbe.
  • berth vfio status on aphotic lists exactly the four processes I had found by hand (llama-swap's and aether's nvidia-smi loops, Quickshell, berthd's probe). berth vfio run there refuses because IOMMU is off.
  • Full proof green: build, gofmt, vet, go test ./..., -race on gpu, broker, cli and vfio, Windows build and vet, golangci-lint (0 issues). mkdocs build --strict passes.

…dupe nodes; status stays readable with IOMMU off
A script in its own process group that reads the terminal is stopped by
SIGTTIN, so sudo's password prompt would hang (reproduced with a pty: the
child sat in state T). The script now stays in berth's process group, and a
Ctrl-C reaches it from the terminal. Only ExitCode stays exported from the
runner. The docs no longer claim a real run, and the example script moves
each function with driver_override, which the card's audio function needs.
berthd's probe is one long-running nvidia-smi -lms, not a call per tick, so
skipping the read during a hold still left berthd holding the card open, and
the driver unbind would block on berthd itself. Found by running berth vfio
status on aphotic. Stream.Pause kills the loop and keeps it stopped until
the hold ends; the broker calls it every tick. berth vfio run now refuses a
probe still open after the settle time, from a berthd that does not pause.

Kernel threads (a readable, empty cmdline) are skipped in the openers scan:
on aphotic they were 400 of the 430 unreadable entries.
They passed on a machine with nvidia-smi and failed on CI without one.
@T-Crypt
T-Crypt merged commit b9e839d into main Oct 21, 2026
4 checks passed
@T-Crypt
T-Crypt deleted the T-Crypt/glm-v2-vfio-cli branch October 21, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant