Skip to content

fix(runner): preserve notify readiness across child scopes - #116

Closed
Bacongineer wants to merge 1 commit into
T-Crypt:mainfrom
Bacongineer:fix/scoped-notify-readiness
Closed

Bacongineer wants to merge 1 commit into
T-Crypt:mainfrom
Bacongineer:fix/scoped-notify-readiness

Conversation

@Bacongineer

Copy link
Copy Markdown
Contributor

Problem

After #112, a child launched by berth run can live in a sibling systemd scope. Its sd_notify sender credentials then belong to that scope rather than the original Type=notify service. Even with NotifyAccess=all, the service can remain activating while its child is already running, and ordered helper services cannot start.

Change

  • Keep the engine's child scope. On Linux, give the child a private Unix datagram socket and relay notifications from the service-owned Berth wrapper to the original manager socket. Apply the same command setup to leased and unleased runs.
  • Preserve datagram payloads and SCM_RIGHTS, including synchronous notification barriers and FDSTORE. Receive descriptors close-on-exec, close received copies after forwarding, and reject truncated payloads or ancillary data.
  • Surface setup, forwarding and private-directory cleanup failures. Forwarding failure uses the existing request-boundary-aware stop path; a pending error at child exit remains visible without draining an already-exited child.
  • Leave the wrapper's environment unchanged. Only the child receives the private socket. Stop and join the relay before removing its private state. No relay is created without NOTIFY_SOCKET; other platforms retain their existing behavior.

This does not emit READY=1 on the child's behalf, change unit ordering, weaken NotifyAccess, remove the engine scope, add configuration, or add dependencies.

Verification

The sender-attribution, pending-error-at-exit and cleanup-failure regressions each failed before their corresponding correction and passed afterward. Protocol tests cover leased/unleased sender identity, absolute/abstract destination sockets, barriers, stored-descriptor contents, malformed setup, failed starts, truncation, transport failure and idempotent cleanup.

Local checks passed on the committed tree:

  • go build ./..., formatting and go vet ./...
  • go test -timeout 0 -count=1 ./...
  • go test -race -timeout 0 -count=3 ./internal/runner ./internal/sysd
  • Windows cross-build and vet
  • The CI-pinned golangci-lint invocation

A standalone built executable also passed a CPU-only native user-service lifecycle fixture. The owning notify service reached active before its ordered helper started. Helper lifetime, signal/stop cleanup, broker reattachment and transaction recovery, and fast cold-start ComfyUI completion/release passed. Fixture state was cleaned and protected process identities stayed unchanged.

That native check matters because Go tests bypass real systemd scope creation under testing.Testing(). It used isolated CPU stand-ins, not real GPU applications or model workloads. No production deployment is part of this PR.

Goal: Keep a notify service ready and its ordered dependents startable while the leased engine lives in its own scope.

How: Give the child a private Unix datagram socket and relay notifications from the service-owned wrapper to the original manager socket.

Key details: Preserve datagram payloads and descriptor rights, including barriers and stored descriptors. Receive rights close-on-exec, clean private state on shutdown and failed starts, and surface transport failures through the existing boundary-aware drain path. Cover leased and unleased sender identity, setup, cleanup, descriptor forwarding, and pending errors at child exit.
@T-Crypt

T-Crypt commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Thanks for catching this. You were right, and I reproduced it here: a Type=notify unit whose child sends READY=1 from its own scope sits in activating, and the same unit without the scope goes active.

I went with a smaller fix in #117. When NOTIFY_SOCKET is set, berth run leaves the engine in the service's cgroup, as it did before #112. Engines under a notify service lose the scope, and we don't take on a socket relay with fd passing to maintain. One behavior here worried me as well: a forwarding error drains the engine, so a relay bug could stop a model that was serving fine.

Do you run a setup where the engine needs its own scope and also sits under a notify service? If so, #117 misses it and I want to know. If not, I'll close this one once #117 merges. Your native systemd fixture notes saved me time checking this, thank you.

@Bacongineer

Bacongineer commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for reproducing this. I agree that #117 is the better solution for our current setup.

To answer your question directly: no, we do not currently require an engine to have its own separate scope while also running under a Type=notify service. The owning service's cgroup is sufficient for our notify-based applications. Our requirement is that readiness activates the owning unit so its ordered helper can start.

For context, Kodi is the GPU lease holder in our playback setup. SVP is a tracked helper, not another lease holder, ordered after Kodi's service and tied to its lifetime.

The unit runs berth run, then systemd-notify --ready, then execs Kodi. Readiness is a lease-admission barrier, not UI readiness. With Type=simple, the wrapper could count as started while still waiting for a lease, allowing the helper to start too early.

I reviewed #117's change. Keeping the child in the service cgroup when NOTIFY_SOCKET is set addresses that attribution failure directly, while other launches retain their scopes. It also avoids the relay/FD-lifecycle machinery and the additional failure-driven drain path you identified.

I understand the trade-off: notify tenants do not get #112's escaped-child scope attribution. For our current requirements, that does not justify maintaining #116's relay instead of taking your smaller fix.

@T-Crypt

T-Crypt commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Closing in favor of #117, now merged. Thanks again for finding this and for the clear repro.

@T-Crypt T-Crypt closed this Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants