VFIO (experimental): berth vfio status and berth vfio run

CHANGELOG.md

@@ -12,6 +12,11 @@ has a matching GitHub release whose body comes from its section below.
   `berth status` and its JSON show it as passed through to a VM, and waiters are told why nothing is
   granted. While a VFIO hold is granted, berthd does not open the card, so a driver switch can run.
   `card.passthrough` and `card.returned` events mark each change.
+- `berth vfio status` and `berth vfio run` (experimental, Linux only) for passing the governed card to a
+  VM. `status` shows IOMMU, the card's IOMMU group with each function's driver, and the processes holding
+  the card. `run -- CMD` drains the card, stops berthd's own probe for the hold, refuses while any process holds it, runs your own switch
+  script with `BERTH_VFIO_DEVICES` set, and releases the hold only when the card is back on its host
+  driver. Berth never switches drivers or asks for passwords. See docs/vfio.md.
 - Gateway fallback routing: an `[[engine]]` may name a `fallback`, another engine in the file that
   needs no card (a CPU llama-server, an Ollama on another box), and `fallback_after` (10 s by
   default). A request that has waited that long for its own engine is proxied to the fallback when

README.md

@@ -215,6 +215,7 @@ The same pages are a site with search at <https://t-crypt.github.io/berth/>.
 | [Install](docs/install.md) | The agent prompt, manual install, `berth.toml`, uninstall |
 | [Engines](docs/engines.md) | Every engine Berth reads, API keys, ComfyUI as a suite |
 | [Leasing, games and holds](docs/leasing.md) | Priorities, drains, stuck releases, the juggle guard, minimum turns, claims and `berth hold` |
+| [VFIO passthrough](docs/vfio.md) (experimental, Linux) | `berth vfio status`, `berth vfio run`, the IOMMU requirements, an example switch script |
 | [Models](docs/models.md) | `berth pull`, integrity checks, shelves, `berth storage`, the page cache and `berth prefetch` |
 | [Gateway](docs/gateway.md) | One OpenAI-compatible port, cold starts, fallbacks, KV snapshots, switch costs, wake pages, park in RAM |
 | [The harbor and telemetry](docs/harbor.md) | Stowaways, the collisions board, alerts, every harbor panel, what Berth measures |

docs/README.md

@@ -9,6 +9,7 @@ has the overview, the quick start and screenshots. These pages are the reference
 | [Install](install.md) | The agent prompt, manual install, `berth.toml`, uninstall |
 | [Engines](engines.md) | Every engine Berth reads, API keys, ComfyUI as a suite |
 | [Leasing, games and holds](leasing.md) | Priorities, drains, stuck releases, the juggle guard, minimum turns, claims and `berth hold` |
+| [VFIO passthrough](vfio.md) (experimental, Linux) | `berth vfio status`, `berth vfio run`, the IOMMU requirements, an example switch script |
 | [Models](models.md) | `berth pull`, integrity checks, shelves, `berth storage`, the page cache and `berth prefetch` |
 | [Gateway](gateway.md) | One OpenAI-compatible port, cold starts, fallbacks, KV snapshots, switch costs, wake pages, park in RAM |
 | [The harbor and telemetry](harbor.md) | Stowaways, the collisions board, alerts, every harbor panel, what Berth measures |

docs/harbor.md

@@ -70,7 +70,7 @@ network you trust.
   `berth status` shows "passed through to a VM (vfio-pci)" in place of its memory figures, and the JSON
   has a `passthrough` object with the state and the bound driver. The state reads `detaching` while a
   switch is under way. `card.passthrough` and `card.returned` mark each change once in the event log.
-  While a VFIO hold is granted (`berth vfio run`), berthd does not open the card, because
+  While a VFIO hold is granted ([`berth vfio run`](vfio.md)), berthd does not open the card, because
   its own probe would block the driver switch; it reads the driver from sysfs instead. Nothing new is
   granted while the card is away, and each waiter's reason says so ("gpu0 is passed through to a VM
   (vfio-pci); it comes back when the VM's hold ends"). A card switched by hand, with no hold, shows the

docs/vfio.md

@@ -0,0 +1,99 @@
+# VFIO passthrough (experimental)
+
+Linux only and experimental. It has passed its tests but has not yet run on a real passthrough, so the
+commands and messages can still change.
+
+Passing the governed card to a virtual machine through VFIO takes it away from the host for the whole
+VM session. The NVIDIA driver lets go, `vfio-pci` takes the card, and nothing on the host may touch it
+until the VM is done. Berth coordinates its side of that handover. It drains your engines, holds the
+card, runs your own switch script, and keeps the next engine waiting until the card is back on its host
+driver.
+
+## What Berth does
+
+- Drains every engine on the card at its boundary, as for any other lease.
+- Holds the card for the whole session with a pinned lease at hold priority, so nothing else starts on
+  it and nothing drains it.
+- Stops berthd's own `nvidia-smi` probe for the whole hold, then checks which processes still hold
+  `/dev/nvidia*` or the card's DRM nodes. Any holder stops the run, and Berth names it by PID and
+  device node.
+- Runs your script with the terminal attached, so its sudo prompt and the VM's console reach you, and
+  with `BERTH_VFIO_DEVICES` set to the card's PCI functions.
+- After the script exits, waits until every function of the card is off `vfio-pci` and the card
+  answers `nvidia-smi`, and only then releases the hold.
+
+## What Berth leaves to you
+
+- The driver switch. Berth never binds or unbinds a driver and never writes to sysfs.
+- Passwords. Your script calls sudo, and the prompt comes to your terminal.
+- The VM, libvirt, QEMU and every other device you pass through.
+- Processes that hold the card. Berth names them and refuses; it never stops them.
+
+## Requirements
+
+- An IOMMU. Turn on VT-d (Intel) or AMD-Vi in the firmware, and add `intel_iommu=on iommu=pt` (or
+  `amd_iommu=on iommu=pt`) to the kernel command line.
+- A second GPU or an iGPU for the host's display, because the passed-through card leaves the host.
+- A clean IOMMU group. The card's group may hold only the card's own functions, such as its audio
+  function. `berth vfio status` lists the group's members and their drivers, and `berth vfio run`
+  refuses while anything else is in the group.
+- An NVIDIA card. AMD cards are not covered yet.
+
+## The commands
+
+`berth vfio status` reads the card's state and changes nothing. It shows whether IOMMU is on, the IOMMU
+group and the driver bound to each function, the processes holding the card's device nodes, and what
+berthd reports. While the card is bound to `vfio-pci`, `nvidia-smi` cannot see it, so pass the card's
+address yourself (`berth vfio status 0000:01:00.0`).
+
+`berth vfio run [--reason NAME] -- CMD` holds the card and runs your script:
+
+```sh
+berth vfio run --reason galaxy -- ~/bin/gpu-to-galaxy
+```
+
+Engines drain first. Once the card is granted, berthd stops its probe, and after a few seconds Berth
+reads the device holders. If any process still holds the card, Berth names it, releases the hold and
+exits 1. A process whose open files Berth cannot read (another user's, usually
+root's) gets a warning instead, because Berth cannot prove it clear. Such a process can block the unbind.
+
+When your script exits, Berth releases the hold only if the card is back. Otherwise it keeps the hold,
+names the function still on `vfio-pci`, and waits. Each Ctrl-C checks again and releases only once the
+card is back. `berth vfio run` exits with your script's exit code.
+
+While the hold is granted, berthd stops opening the card and `berth status` shows it as passed through
+to a VM. Waiting requests are told why.
+
+## An example script to adapt
+
+This is an example, not something Berth installs. It moves every function the generic way
+(`driver_override`), so it works for the card's audio function too, and it reads the addresses from
+`BERTH_VFIO_DEVICES`. If your libvirt domain marks the card's `hostdev` as `managed='yes'`, libvirt
+switches the drivers itself, and the script only starts the VM and waits.
+
+```sh
+#!/bin/sh
+# gpu-to-galaxy: give the card to the VM, run it, take the card back.
+# Run it through: berth vfio run --reason galaxy -- ~/bin/gpu-to-galaxy
+set -eu
+devs="${BERTH_VFIO_DEVICES:?run this through berth vfio run}"
+
+to() { # to DRIVER: move every function of the card to DRIVER ("" = its own driver)
+  for d in $devs; do
+    echo "$1" | sudo tee "/sys/bus/pci/devices/$d/driver_override" >/dev/null
+    if [ -e "/sys/bus/pci/devices/$d/driver" ]; then
+      echo "$d" | sudo tee "/sys/bus/pci/devices/$d/driver/unbind" >/dev/null
+    fi
+    echo "$d" | sudo tee /sys/bus/pci/drivers_probe >/dev/null
+  done
+}
+
+sudo modprobe vfio-pci
+to vfio-pci
+sudo virsh start galaxy
+while [ "$(sudo virsh domstate galaxy)" != "shut off" ]; do sleep 5; done
+to ""
+```
+
+If the script dies with the card still on `vfio-pci`, the hold stays, and the next engine waits until
+the card is back.

internal/broker/broker.go

@@ -624,6 +624,11 @@ func (b *Broker) Tick(ctx context.Context) {
 	b.mu.Lock()
 	held := b.vfioHold()
 	b.mu.Unlock()
+	if p, ok := b.probe.(gpu.Pauser); ok {
+		// A probe that keeps nvidia-smi running would hold the card open
+		// through the driver switch, so it stops for the whole hold.
+		p.Pause(held != nil)
+	}
 	var snap gpu.Snapshot
 	var err error
 	if held == nil {

internal/broker/vfio_test.go

@@ -374,3 +374,44 @@ func TestVFIOHoldDrainsTheHolderFirst(t *testing.T) {
 		t.Fatalf("card state = %+v after the grant, want detaching", card)
 	}
 }
+
+// pausingProbe records what the broker asked of a probe that keeps a
+// process on the card between reads.
+type pausingProbe struct {
+	fakeProbe
+	mu     sync.Mutex
+	paused []bool
+}
+
+func (p *pausingProbe) Pause(v bool) {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	if n := len(p.paused); n == 0 || p.paused[n-1] != v {
+		p.paused = append(p.paused, v)
+	}
+}
+
+func (p *pausingProbe) changes() []bool {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	return append([]bool(nil), p.paused...)
+}
+
+// A VFIO hold stops the probe's own process on the card for the whole hold,
+// and the probe starts again once the hold ends.
+func TestVFIOHoldPausesTheProbe(t *testing.T) {
+	skipNonLinux(t)
+	sysfsCard(t, "0000:01:00.0", "nvidia")
+	p := &pausingProbe{fakeProbe: fakeProbe{busID: "00000000:01:00.0"}}
+	b := vfioBroker(t, p)
+	b.Tick(context.Background())
+	hold := must(t, acquire(b, vfioRequest()))
+	b.Tick(context.Background())
+	hold.Release()
+	for i := 0; i < 3; i++ {
+		b.Tick(context.Background())
+	}
+	if got := p.changes(); len(got) != 3 || got[0] || !got[1] || got[2] {
+		t.Fatalf("pause calls = %v, want running, paused for the hold, running again", got)
+	}
+}

internal/cli/commands.go

@@ -561,6 +561,53 @@ same reference again repairs only what does not match.`,
 		Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
 		Run:     runVersion,
 	},
+	{
+		Name:    "vfio",
+		Summary: "pass the card to a VM through VFIO: watch it, wrap the switch script",
+		Help: `Berth drains the engines, holds the card while your own script switches it to
+vfio-pci and runs the VM, and refuses to let the next engine load until the card
+is back on its host driver and answering nvidia-smi. Berth never switches a
+driver itself, never asks for a password, and never stops a process: your script
+(sudo, virsh) owns the switch and the VM. Experimental, Linux only;
+docs/vfio.md has the requirements and an example script.`,
+		Sub: []*Command{
+			{
+				Name:    "run",
+				Summary: "hold the card, run the switch script, wait for the card back",
+				Args:    "[--reason NAME] [--wait D] -- CMD [ARGS...]",
+				Help: `Acquire the VFIO hold the way berth hold does: engines drain at their
+boundaries and none starts. Then the script runs with the terminal attached, so
+its sudo prompt reaches you, and with BERTH_VFIO_DEVICES set to the card's PCI
+functions, so it hardcodes no address.
+
+Before the script starts, the openers are re-read: anyone but berthd's own
+probe holding /dev/nvidia* or the card's DRM nodes refuses the run, names every
+PID and node, releases the hold and exits 1. Berth never stops a process.
+
+When the script exits, the hold ends only when every function of the card is
+off vfio-pci and the card answers nvidia-smi again (up to a minute). Until
+then the hold stays, the message names the function that is out and what to do,
+and Ctrl-C rechecks: it releases only when the card is back, so the next engine
+never loads onto a card that is out.`,
+				Run: runVfioRun,
+			},
+			{
+				Name:    "status",
+				Summary: "the card's VFIO state: IOMMU, group, drivers, openers, berthd's view",
+				Args:    "[ADDR] [--socket P] [--format F]",
+				Help: `Print the card's VFIO state and change nothing: whether IOMMU is on, the card's
+IOMMU group and the driver each function in it is bound to (anything but the
+card's own functions blocks the passthrough), the processes holding the card's
+device nodes open, and what berthd reports about the card.
+
+ADDR is the card's PCI address, as in 0000:01:00.0. Without it the card berthd
+leases is used, which needs nvidia-smi to answer; nvidia-smi cannot see a card
+bound to vfio-pci, so pass the address while the VM runs.`,
+				Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
+				Run:     runVfioStatus,
+			},
+		},
+	},
 	{
 		Name:    "wire",
 		Summary: "make another engine's whole roster run under berth",