Cgroup attribution closes the escaped-child stowaway window (chaos scenario 3)

CHANGELOG.md

@@ -6,6 +6,19 @@ has a matching GitHub release whose body comes from its section below.
 
 ## [Unreleased]
 
+### Added
+
+- Cgroup attribution closes the one known stowaway window (chaos scenario 3: a child
+  whose parent exits within one 2 s tick). On Linux with a systemd user manager,
+  `berth run` execs the engine in its own transient cgroup scope (`systemd-run --user
+  --scope`, unit `berth-run-<name>-<rand>`), which every child inherits on fork: a
+  child that escapes its process group and reparents stays the lease's, the lease
+  turns stuck naming it, `berth reclaim` lists it, and a whole-card request is never
+  granted `contested` over it. A GPU process in a scope no live lease recorded is a
+  stowaway as before, and a process outside every scope is never attributed to a
+  scopeless lease. Without a user manager, and on Windows, the run starts as before
+  and the window stands.
+
 ## [v0.3.3] - 2026-10-21
 
 Fixes from the Windows testers and two gateway additions. On a single-GPU Windows desktop, the

README.md

@@ -420,9 +420,12 @@ startup, so a change to the roots applies on the next restart.
   ones, SIGTERM first and SIGKILL to the survivors after `--grace`, re-checking each
   PID and start time immediately before every signal.
 - **Process identity is PID plus start time**, so a reused PID never inherits a lease.
-- **Children stay on the lease.** Berth records every descendant of an engine's wrapper each
-  tick, so a child that escapes its process group and reparents keeps the lease stuck under its
-  own name instead of turning into a stowaway.
+- **Children stay on the lease.** On Linux with a systemd user manager, `berth run`
+  gives the engine its own cgroup scope, which every child inherits on fork: a child
+  that escapes its process group and reparents keeps the lease stuck under its own
+  name instead of turning into a stowaway. Without a user manager, and on Windows,
+  Berth records every descendant of the wrapper each tick, which covers everything
+  except a fork whose parent exits inside one tick.
 - **Whole-card or sized leases.** `--needs-mib N` lets engines that fit share a card.
 - **Fail-open, never silent.** If the daemon is down, `berth run` still starts the engine
   but records it as an *unprotected run* that status shows (`--fail-closed` refuses).
@@ -1162,8 +1165,6 @@ engine's files during a drain (`prefetch`) and serves measured switch costs at `
 
 In the next v0.3.x releases. Each item has its building blocks merged or a spec written:
 
-- Cgroup attribution, which closes the one known stowaway window (chaos scenario 3: a child
-  whose parent exits within one 2 s tick).
 - KV snapshots wired into the drain and start paths, so a sequence of agents skips the prefill
   after each switch.
 - Measured switch costs on the live daemon.

contrib/chaos/README.md

@@ -45,7 +45,7 @@ the log live; the expectations below quote its event kinds.
 |---|---|---|---|
 | 1 | Clean drain | `berth run --as a -- python hog.py 8 polite`, then, once a is in, `berth run --as b --priority foreground -- python hog.py 8 polite` | a is asked to drain, gets SIGTERM and exits; b is granted only after the driver shows a's 8 GiB gone. The event log gets `lease.draining`, `lease.released`, `lease.freed`, `lease.granted` in that order. |
 | 2 | Ignores SIGTERM | as scenario 1 with `ignore-term` on the first run | SIGTERM goes to a's own process group, nothing happens for the 30 s grace, then SIGKILL goes to that same group. No other process is signalled. After a dies, the handoff is as in scenario 1. |
-| 3 | Escaped child | as scenario 1 with `escape-child` on the first run; the parent exits immediately, so start b at any time (`berth run --as b --priority foreground -- python hog.py 8 polite`) | Every tick Berth records each descendant of a's wrapper (PID plus start time), so a child that escapes its group and reparents stays a's: once a exits, the lease turns `stuck` after the 30 s release grace and the log names the child's PID and MiB; b waits and is told why. `berth reclaim` lists it, and `berth reclaim --yes` stops exactly that child. Nothing stops it on its own. One window remains: a child that forks and whose parent exits inside one tick (2 s) before any record shows as a stowaway, and a whole-card b is granted `contested` with a warning. |
+| 3 | Escaped child | as scenario 1 with `escape-child` on the first run; the parent exits immediately, so start b at any time (`berth run --as b --priority foreground -- python hog.py 8 polite`) | On Linux with a systemd user manager `berth run` execs the engine in its own transient cgroup scope, which every child inherits on fork: a child that leaves the group and reparents stays a's, once a exits the lease turns `stuck` after the 30 s release grace and the log names the child's PID and MiB, and b waits and is told why, never granted `contested` over a stuck lease. `berth reclaim` lists it, and `berth reclaim --yes` stops exactly that child. Nothing stops it on its own. Without a user manager, and on Windows, the run has no scope of its own and only the per-tick descendant records stand, where the old window remains: a child that forks and whose parent exits inside one tick (2 s) before any record shows as a stowaway, and a whole-card b is granted `contested` with a warning. |
 | 4 | Stowaway | `python hog.py 8 polite` on its own (no `berth run`), then `berth run --as b --needs-mib 20000 -- python hog.py 4 polite` on a 24 GB card | The hog shows up as a stowaway with 8192 MiB in status, the events log and the harbor. b waits with `squatters hold 8192 MiB`; it is never granted while the hog holds memory, and the hog is never signalled. |
 | 5 | Unrelated workload survives | run scenario 2 while `glxgears` (or a game) is running | The kill ladder reaches only a's process group: the unrelated window keeps drawing through every step, and the handoff finishes around it. |
 | 6 | Busy forever | `berth run --as a --drain-slots http://127.0.0.1:18080 --drain-max 10m -- python hog.py 8 busy-slots`, then `berth run --as b --priority foreground -- sleep 30` | The drain waits on a's `/slots`, which is busy forever. Without `drain_max` b waits without limit; the wrapper prints `draining a: 1 slot busy for 6m12s` each minute, so the journal says why. With `drain_max` (the flag, or `drain_max` in a's berth.toml entry) the wait ends after that long: the wrapper says `a still busy after 10m0s (1 slot processing); draining anyway`, and the drain runs the normal ladder, so b gets the card. |

internal/broker/broker.go

@@ -154,6 +154,11 @@ type Config struct {
 	// on Linux); "" = a host process. Tests fake it. A GPU process whose
 	// container matches an attached lease's belongs to that lease.
 	Containers func(pid int) (string, bool)
+	// Scopes reads a process's Berth scope (procs.Scope on Linux): the
+	// path of the berth-*.scope cgroup it runs in, "" for a process
+	// outside one. Tests fake it. A GPU process in the scope an attached
+	// lease recorded belongs to that lease.
+	Scopes func(pid int) (string, bool)
 	// ContainerName resolves a container id to its runtime name (docker,
 	// then podman, inspect); "" when neither answers. Used to name a
 	// stowaway's container in the report; never affects scheduling.
@@ -214,13 +219,18 @@ type lease struct {
 	// ("" for a host process), containerName the name the operator gave.
 	container     string
 	containerName string
-	gpu           map[ident]gpu.Proc
-	desc          map[ident]struct{} // descendants recorded while the lease ran
-	contested     uint64             // squatter bytes on the card when granted
-	peak          uint64             // most memory its processes were measured holding
-	arrived       time.Time
-	granted       time.Time
-	released      time.Time
+	// scope is the berth-*.scope path the attached engine runs in
+	// ("" when it runs outside one). A cgroup is inherited on fork, so
+	// every process under that path is the lease's, however its process
+	// tree is shaped.
+	scope     string
+	gpu       map[ident]gpu.Proc
+	desc      map[ident]struct{} // descendants recorded while the lease ran
+	contested uint64             // squatter bytes on the card when granted
+	peak      uint64             // most memory its processes were measured holding
+	arrived   time.Time
+	granted   time.Time
+	released  time.Time
 	// releasedRead and parkedRead are the card reads started when the lease
 	// was released or its park call was done: only a tick whose read is
 	// later frees it or marks it parked. Clock-free, so a coarse clock
@@ -516,6 +526,9 @@ func New(cfg Config, probe gpu.Probe, tree procs.Tree) *Broker {
 	if cfg.Containers == nil {
 		cfg.Containers = procs.Container
 	}
+	if cfg.Scopes == nil {
+		cfg.Scopes = procs.Scope
+	}
 	if cfg.ContainerName == nil {
 		cfg.ContainerName = containerNameCLI
 	}
@@ -623,13 +636,15 @@ func (b *Broker) Tick(ctx context.Context) {
 	b.probeErr = nil
 	b.snap = snap
 	b.recordDescendants()
+	b.refreshScopes()
 
 	present := map[ident]bool{}
 	claimed := map[ident]claimedProc{}
 	for _, p := range snap.Procs {
 		id := b.identify(p.PID)
 		present[id] = true
-		if l := b.owner(id, b.containerOf(p.PID)); l != nil {
+		scope := b.scopeOf(p.PID)
+		if l := b.owner(id, b.containerOf(p.PID), scope); l != nil {
 			if l.state != active && l.state != draining && b.runFor(p.PID) != nil {
 				// A released lease keeps blocking while its processes hold
 				// memory, unless a run record says they were left behind on
@@ -638,6 +653,13 @@ func (b *Broker) Tick(ctx context.Context) {
 				delete(l.gpu, id)
 			} else {
 				l.gpu[id] = p
+				if scope != "" && l.scope == scope {
+					// In the lease's scope but never recorded: a child that
+					// escaped the tree before any tick. Record it now, so
+					// reclaim and the stuck report name it exactly like a
+					// recorded descendant.
+					l.desc[id] = struct{}{}
+				}
 				l.peak = max(l.peak, l.used())
 				if s, ok := b.squatters[id]; ok {
 					b.reportGone(s)
@@ -773,10 +795,11 @@ func (b *Broker) reportGone(s *squatter) {
 
 // owner finds the lease a GPU process belongs to: one it was already
 // attributed to, else a recorded descendant of an attached lease, else a
-// container the lease names, else the lease whose wrapped engine it
-// descends from. The engine's PID only counts while its start time still
-// matches, so a reused PID never inherits a lease.
-func (b *Broker) owner(id ident, container string) *lease {
+// scope the lease recorded, else a container the lease names, else the
+// lease whose wrapped engine it descends from. The engine's PID only
+// counts while its start time still matches, so a reused PID never
+// inherits a lease. An empty scope never matches an empty one.
+func (b *Broker) owner(id ident, container, scope string) *lease {
 	for _, l := range b.leases {
 		if _, ok := l.gpu[id]; ok {
 			return l
@@ -787,6 +810,9 @@ func (b *Broker) owner(id ident, container string) *lease {
 			return l
 		}
 	}
+	if l := b.scopeLease(scope); l != nil {
+		return l
+	}
 	if container != "" {
 		for _, l := range b.leases {
 			if (l.state == active || l.state == draining || l.state == releasing) && l.container == container {
@@ -1250,10 +1276,14 @@ func (l *Lease) Contested() uint64 {
 }
 
 // Attach records the engine holding the berth, by PID plus start time.
+// Its Berth scope is read on the next tick, not here: systemd-run moves
+// itself into the scope only after a round trip to the user manager, so
+// at attach the engine still sits in its wrapper's cgroup.
 func (l *Lease) Attach(pid int) {
 	start := l.b.tree.StartTime(pid)
 	l.b.mu.Lock()
 	l.l.pid, l.l.pidStart = pid, start
+	l.l.scope = ""
 	l.b.mu.Unlock()
 }
 
@@ -1334,6 +1364,7 @@ func (b *Broker) Reattach(r Request, pid int, start uint64) (*Lease, error) {
 	if pid <= 0 || start == 0 || b.tree.StartTime(pid) != start {
 		return nil, fmt.Errorf("%w: pid %d is not the process that started as %s", ErrRefused, pid, r.Engine)
 	}
+	scope, _ := b.cfg.Scopes(pid)
 	b.mu.Lock()
 	defer b.mu.Unlock()
 	for _, l := range b.leases {
@@ -1350,6 +1381,7 @@ func (b *Broker) Reattach(r Request, pid int, start uint64) (*Lease, error) {
 	}
 	l.state, l.reattach = active, true
 	l.pid, l.pidStart = pid, start
+	l.scope = scope
 	l.granted = now
 	l.grantedN = b.order()
 	close(l.grantCh)

internal/broker/chaos_test.go

@@ -177,3 +177,122 @@ func TestChaosScenario4Stowaway(t *testing.T) {
 		t.Fatalf("reports = %q, want a single appearance", got)
 	}
 }
+
+// Scenario 3, escaped child, on a machine with a systemd user manager:
+// `berth run` execs the engine into a berth-run scope, the engine's
+// child forks a grandchild that takes the memory and exits itself
+// before any tick records a descendant. A cgroup is inherited on fork,
+// so the grandchild is still in the scope the lease recorded: it
+// belongs to the lease from its first sighting, the lease turns stuck
+// after the release grace naming the grandchild, reclaim lists it
+// under the lease, and a whole-card requester waits, never granted
+// contested.
+func TestChaosScenario3EscapedChildInScope(t *testing.T) {
+	tree := procs.Fake{Parents: map[int]int{100: 1, 102: 1}, Starts: map[int]uint64{100: 1, 102: 5}}
+	b, c, r := setup(t, tree)
+	b.cfg.ReleaseGrace = time.Millisecond
+	table := []procs.Proc{
+		{PID: 100, Parent: 1, Start: 1, Name: "berth"},
+		{PID: 102, Parent: 1, Start: 5, Name: "python"},
+	}
+	b.cfg.ProcTable = func() []procs.Proc { return table }
+	b.cfg.Scopes = func(pid int) (string, bool) {
+		if pid == 100 || pid == 102 {
+			return runScopeA, true
+		}
+		return "", false
+	}
+
+	a := must(t, acquire(b, Request{Engine: "a"}))
+	a.Attach(100)
+	_, live, stop := b.Events(0, true)
+	defer stop()
+
+	// The intermediate child is gone before the first tick; the
+	// grandchild takes the card with the process tree already escaped.
+	c.set(gpu.Proc{PID: 102, Name: "python", Used: 8 * GiB})
+	b.Tick(context.Background())
+
+	st := b.Status()
+	if len(st.Leases) != 1 || st.Leases[0].Used != 8*GiB {
+		t.Fatalf("leases = %+v, want the scope's memory under a", st.Leases)
+	}
+	if len(st.Squatters) != 0 || strings.Join(r.seen, ",") != "" {
+		t.Fatalf("squatters = %+v, reports = %v, want no stowaway", st.Squatters, r.seen)
+	}
+
+	// a lets the lease go; the grandchild keeps holding, so the lease
+	// turns stuck after the grace and names it in the event.
+	a.Release()
+	b.Tick(context.Background())
+	time.Sleep(5 * time.Millisecond)
+	b.Tick(context.Background())
+
+	st = b.Status()
+	if len(st.Leases) != 1 || st.Leases[0].State != "stuck" || st.Leases[0].Used != 8*GiB {
+		t.Fatalf("leases = %+v, want a stuck lease holding the escaped grandchild", st.Leases)
+	}
+	deadline := time.Now().Add(time.Second)
+	for {
+		select {
+		case e := <-live:
+			if e.Kind == proto.EvStuck {
+				if !strings.Contains(e.Detail, "pid 102 (python)") {
+					t.Fatalf("stuck event %q does not name the grandchild", e.Detail)
+				}
+				goto listed
+			}
+		case <-time.After(time.Until(deadline)):
+			t.Fatal("no stuck event arrived")
+		}
+	}
+listed:
+	hs, err := b.Reclaim(a.ID())
+	if err != nil {
+		t.Fatalf("reclaim: %v", err)
+	}
+	if len(hs) != 1 || hs[0].PID != 102 || hs[0].Lease != a.ID() || hs[0].Name != "python" {
+		t.Fatalf("holders = %+v, want the grandchild under a's lease", hs)
+	}
+
+	// A whole-card requester waits: the stuck lease holds memory, so
+	// nothing is granted, contested or otherwise.
+	breq := acquire(b, Request{Engine: "b", Priority: Foreground})
+	pending(t, breq)
+	if st := b.Status(); len(st.Leases) != 1 {
+		t.Fatalf("leases = %+v, want only the stuck lease", st.Leases)
+	}
+}
+
+// Scenario 3 without a user manager (no scope to record): the same
+// escaped grandchild is a stowaway as today, and once a's wrapper lets
+// the lease go a whole-card requester is granted contested. Nothing
+// matches an empty scope, so a scopeless process is never silently
+// attributed to the lease that also has no scope.
+func TestChaosScenario3EscapedChildWithoutScope(t *testing.T) {
+	tree := procs.Fake{Parents: map[int]int{100: 1, 102: 1}, Starts: map[int]uint64{100: 1, 102: 5}}
+	b, c, r := setup(t, tree)
+	b.cfg.ProcTable = func() []procs.Proc { return []procs.Proc{{PID: 100, Parent: 1, Start: 1, Name: "berth"}} }
+	b.cfg.Scopes = func(int) (string, bool) { return "", false }
+
+	a := must(t, acquire(b, Request{Engine: "a"}))
+	a.Attach(100)
+	c.set(gpu.Proc{PID: 102, Name: "python", Used: 8 * GiB})
+	b.Tick(context.Background())
+
+	st := b.Status()
+	if len(st.Squatters) != 1 || st.Squatters[0].PID != 102 {
+		t.Fatalf("squatters = %+v, want the escaped grandchild listed", st.Squatters)
+	}
+	if got := strings.Join(r.seen, ","); got != "appeared:python" {
+		t.Fatalf("reports = %q, want a single appearance", got)
+	}
+
+	a.Release()
+	b.Tick(context.Background())
+
+	whole := must(t, acquire(b, Request{Engine: "b", Priority: Foreground}))
+	if whole.Contested() != 8*GiB {
+		t.Fatalf("contested = %d, want the stowaway counted", whole.Contested())
+	}
+}

internal/broker/reclaim.go

@@ -52,7 +52,7 @@ func (b *Broker) Reclaim(id string) ([]proto.Holder, error) {
 	var hs []proto.Holder
 	for _, p := range b.snap.Procs {
 		pid := b.identify(p.PID)
-		l := b.owner(pid, b.containerOf(p.PID))
+		l := b.owner(pid, b.containerOf(p.PID), b.scopeOf(p.PID))
 		switch {
 		case l != nil && want[l.id]:
 			hs = append(hs, proto.Holder{PID: p.PID, PIDStart: pid.start, Lease: l.id, Name: p.Name, Used: p.Used})

internal/broker/scope.go

@@ -0,0 +1,57 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package broker
+
+// Cgroup attribution. `berth run` starts its engine in a berth-run
+// scope and the gateway starts its runs in berth-engine scopes; a
+// cgroup is inherited on fork, so every process under a lease's
+// recorded scope is the lease's, even a child that left the process
+// tree before any tick could record it (chaos scenario 3). Nothing is
+// signalled because of a scope that Berth did not start: the scope
+// holds exactly the processes a Berth wrapper execed into it.
+
+// scopeLease is the one live lease that recorded scope, nil when none
+// did or when more than one did: a scope two leases share names neither,
+// so attribution falls through to the process tree.
+func (b *Broker) scopeLease(scope string) *lease {
+	if scope == "" {
+		return nil
+	}
+	var found *lease
+	for _, l := range b.leases {
+		if (l.state == active || l.state == draining || l.state == releasing) && l.scope == scope {
+			if found != nil {
+				return nil
+			}
+			found = l
+		}
+	}
+	return found
+}
+
+// refreshScopes reads each attached engine's scope while its PID plus
+// start time is still the engine: the engine enters its scope a moment
+// after it starts, so the read at attach would name the wrapper's
+// cgroup. Once the engine is gone the last scope read stays, which is
+// what names a child it left behind. Called with b.mu held.
+func (b *Broker) refreshScopes() {
+	for _, l := range b.leases {
+		if (l.state != active && l.state != draining && l.state != releasing) || l.pid <= 0 ||
+			b.tree.StartTime(l.pid) != l.pidStart {
+			continue
+		}
+		if s, ok := b.cfg.Scopes(l.pid); ok {
+			l.scope = s
+		}
+	}
+}
+
+// scopeOf is a process's Berth scope path, "" when it runs outside one.
+func (b *Broker) scopeOf(pid int) string {
+	s, ok := b.cfg.Scopes(pid)
+	if !ok {
+		return ""
+	}
+	return s
+}