Skip to content

VFIO: --before and --after around the session, and a libvirt example in contrib/vfio - #131

Merged
T-Crypt merged 1 commit into
mainfrom
T-Crypt/vfio-before
Oct 21, 2026
Merged

T-Crypt merged 1 commit into
mainfrom
T-Crypt/vfio-before

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Follows #129 and #130. The first real launcher, run on the reference machine, showed what was missing.

The problem. Services the operator runs beside the engines can hold the card without a lease: a monitor that runs nvidia-smi in a loop, a desktop shell with a GPU widget. They have to stop after the engines drain and before Berth checks that nothing holds the card. Without a hook, the launcher had to take a plain hold, stop them, queue berth vfio run, and then release the plain hold.

The change.

  • --before CMD runs right after the grant, before that check.
  • --after CMD runs once the hold has ended, whenever --before ran: after the card comes back, after a refusal, or after a failed --before. What --before stopped always comes back, and never while the card is in the VM.
  • Both run with sh -c, the terminal attached and BERTH_VFIO_DEVICES set. Berth still stops nothing itself.

Also fixed. A script that ran and exited 127 (a typo inside it) read the same as one that never started, which would have skipped the card-back check and released the hold with the card possibly still in the VM. The start error is now its own value, so the check always runs.

Example. contrib/vfio/ has vm-session.sh (a launcher built on these flags, shellcheck clean) and domain-hostdev.xml (the GPU's managed hostdev entries, so libvirt switches the drivers). docs/vfio.md covers both flags and links the example. The help text no longer says berthd's probe may hold the card.

Proof:

  • New tests: --before stops a holder and --after runs after release; --after runs when the check refuses; a failing --before releases the hold; a script exiting 127 still gets the card-back check.
  • The handoff the launcher replaces ran on aphotic's live daemon (VFIO (experimental): berth vfio status and berth vfio run #130's checks): the VFIO hold took the card, berthd's probe let go, and the check named exactly the three services --before now stops.
  • Full proof green: build, gofmt, vet, go test ./... (license headers included), -race on the vfio tests, Windows build and vet, golangci-lint (0 issues). mkdocs build --strict passes.

Not here: the --help usage line drops the parent command for every subcommand (berth vfio run --help prints berth run). It is an older, general bug, tracked separately.

A VM session has to stop the operator's own services that hold the card (a
monitor running nvidia-smi in a loop, a desktop widget) after the engines
drain and before the check that nothing holds it. Without a hook, a launcher
had to take a plain hold, stop them, queue the VFIO hold and release the
plain one. --before runs right after the grant; --after runs once the hold
has ended, whenever --before ran, so what it stopped always comes back and
never while the card is in the VM.

A script that ran and exited 127 no longer reads as one that never started:
the start error is its own value, so the card-back check always runs.

contrib/vfio has a launcher and the domain's managed hostdev entries.
@T-Crypt
T-Crypt merged commit edc6cd9 into main Oct 21, 2026
4 checks passed
@T-Crypt
T-Crypt deleted the T-Crypt/vfio-before branch October 22, 2026 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant