Repository navigation
VFIO: --before and --after around the session, and a libvirt example in contrib/vfio - #131
Merged
Merged
Conversation
A VM session has to stop the operator's own services that hold the card (a monitor running nvidia-smi in a loop, a desktop widget) after the engines drain and before the check that nothing holds it. Without a hook, a launcher had to take a plain hold, stop them, queue the VFIO hold and release the plain one. --before runs right after the grant; --after runs once the hold has ended, whenever --before ran, so what it stopped always comes back and never while the card is in the VM. A script that ran and exited 127 no longer reads as one that never started: the start error is its own value, so the card-back check always runs. contrib/vfio has a launcher and the domain's managed hostdev entries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #129 and #130. The first real launcher, run on the reference machine, showed what was missing.
The problem. Services the operator runs beside the engines can hold the card without a lease: a monitor that runs
nvidia-smiin a loop, a desktop shell with a GPU widget. They have to stop after the engines drain and before Berth checks that nothing holds the card. Without a hook, the launcher had to take a plain hold, stop them, queueberth vfio run, and then release the plain hold.The change.
--before CMDruns right after the grant, before that check.--after CMDruns once the hold has ended, whenever--beforeran: after the card comes back, after a refusal, or after a failed--before. What--beforestopped always comes back, and never while the card is in the VM.sh -c, the terminal attached andBERTH_VFIO_DEVICESset. Berth still stops nothing itself.Also fixed. A script that ran and exited 127 (a typo inside it) read the same as one that never started, which would have skipped the card-back check and released the hold with the card possibly still in the VM. The start error is now its own value, so the check always runs.
Example.
contrib/vfio/hasvm-session.sh(a launcher built on these flags, shellcheck clean) anddomain-hostdev.xml(the GPU's managed hostdev entries, so libvirt switches the drivers).docs/vfio.mdcovers both flags and links the example. The help text no longer says berthd's probe may hold the card.Proof:
--beforestops a holder and--afterruns after release;--afterruns when the check refuses; a failing--beforereleases the hold; a script exiting 127 still gets the card-back check.--beforenow stops.go test ./...(license headers included),-raceon the vfio tests, Windows build and vet, golangci-lint (0 issues).mkdocs build --strictpasses.Not here: the
--helpusage line drops the parent command for every subcommand (berth vfio run --helpprintsberth run). It is an older, general bug, tracked separately.