Skip to content

VFIO (experimental): berthd shows a card passed through to a VM and stays off it during a switch - #129

Merged
T-Crypt merged 6 commits into
mainfrom
T-Crypt/glm-v1-vfio-daemon
Oct 21, 2026
Merged

T-Crypt merged 6 commits into
mainfrom
T-Crypt/glm-v1-vfio-daemon

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Oct 21, 2026

Copy link
Copy Markdown
Owner

First half of VFIO passthrough support (design: Berth coordinates and shows the card; the operator's own script switches drivers and runs the VM). The command, berth vfio run, comes in the next PR.

  • While the governed card is bound to vfio-pci, berth status shows "passed through to a VM (vfio-pci)" instead of memory figures, and the JSON has a passthrough object. card.passthrough and card.returned events fire once per change.
  • While a VFIO hold is granted (proto.GroupVFIO, a pinned acquire at hold priority), berthd does not run its probe. Its nvidia-smi call opens the card and can block the driver unbind, so it reads the driver from sysfs instead.
  • Nothing new is granted while the card is away, and each waiter is told why.
  • A card switched by hand with no hold shows the same state instead of a probe-failed warning. A probe failure on a card still bound to nvidia warns as before.
  • Reads sysfs only. Nothing writes to /sys, binds a driver or loads a module. Linux only. Elsewhere the reads return nothing and behavior is unchanged.

Proof:

  • Tests with a fake sysfs tree and a counting probe: the probe does not run while the hold is granted; detaching, passthrough and returned each fire one event; a waiter gets the passthrough reason; the card is found by scan when berthd never saw it; one, none or two vfio-bound cards; a failing probe on nvidia still warns. TestVFIOHoldDrainsTheHolderFirst covers a hold asked for while an engine holds the card: it drains the engine and is granted.
  • Full proof green: build, gofmt, vet, go test ./..., Windows build and vet, golangci-lint (0 issues). mkdocs build --strict passes.

DriverFor takes the basename of the function's driver symlink under
/sys/bus/pci/devices, empty when unbound or unreadable or off Linux.
SysfsAddr turns nvidia-smi's bus id into the sysfs device name.
FindVFIOCard names the one NVIDIA display function bound to vfio-pci,
empty when none or several match. The sysfs root is a package variable
so tests use a temp dir. BusID is now exported for the broker.
…vice during a switch

While a granted lease in group vfio guards the switch, Tick does not run
the probe at all: it reads the driver bound to the governed card's PCI
function instead. Bound to vfio-pci the card is passthrough; with the
hold granted and another or no driver there, it is detaching. A probe
failure with no hold is read the same way when the card is bound to
vfio-pci (a hand switch), and keeps today's warning otherwise. Neither
state grants anything new; waiters hear where the card is in their queue
place, and the hold itself stays HOT. card.passthrough and
card.returned events mark each transition once, and the status carries
passthrough with the state and driver, omitted when the card is normal.
The card's sysfs address is remembered from the last successful
snapshot, or found by scan when the probe fails and no reading ever
came.
…n-first test

proto.GroupVFIO replaces the broker's private constant; berth vfio run uses
the same name. TestVFIOHoldDrainsTheHolderFirst covers the common case: an
engine holds the card when the hold is asked for, drains, and the hold is
granted.
@T-Crypt
T-Crypt merged commit 3a2e21d into main Oct 21, 2026
4 checks passed
@T-Crypt
T-Crypt deleted the T-Crypt/glm-v1-vfio-daemon branch October 21, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant