[sejori]

Fourth of five stacked PRs for the worker validation gates DEP (ai-dynamo/dynamo#15897). Based on #3. The code doesn't depend on #3, but the stack is kept linear so #5's end-to-end test covers the card hold and this endpoint together.

Why

A gated worker is unreachable by design: its card is held, so no frontend routes to it. Its checks still need to send requests that go through the frontend's OpenAI preprocessing, tool-call and reasoning parsers.

What

• New lib/llm/src/entrypoint/input/local_openai.rs. When DYN_WORKER_VALIDATION_GATES is set, LocalModel::attach serves the worker's first base-model card that takes tokens and has a chat or completions surface. It listens on 127.0.0.1:${DYN_WORKER_VALIDATION_LOCAL_PORT}, where 0 (the default) means an ephemeral port.
  • It uses the same pipeline as the in-process token HTTP input, via serve_in_process_tokens, extracted from http.rs with no behaviour change.
  • On each request it looks up the engine in the LocalEndpointRegistry through RegistryEngine. This is a serde round trip, the inverse of JsonProbeAdapter, so the parsers see exactly what they'd see over the request plane. The endpoint returns 503 until the engine is registered.
  • No listener opens when gates are unset.
• Gate trigger. Gates now start only once this endpoint is serving and the engine is registered. They get OPENAI_BASE_URL=http://127.0.0.1:<port>/v1 and DYN_VALIDATION_MODEL. This replaces feat(runtime): worker validation gates #2's start-on-first-local-engine trigger, which could also fire on load_lora or clear_kv_blocks registrations.
• Endpoint lifetime. The endpoint closes on release, and stays up after a failed gate so the held worker can be debugged.
• Local engine registration. The unified Rust worker (backend-common) and the dyn:// token input (the mocker and run_input) now register a local engine when gates are set, even without a canary payload. Python workers already register one for every served endpoint, so the bindings don't change.
• enable_rl. HttpServiceConfig::enable_rl is now Option<bool>, so the loopback service can turn RL discovery off. DYN_ENABLE_RL is also a worker env var.
• One endpoint per process. WorkerValidation::claim_local_endpoint() ensures there's only one, even with multiple cards or LoRA attaches.

Tests

• lib/llm/tests/local_openai_validation.rs:
  • loopback_matches_frontend_on_the_mocker runs with hold=health, so the frontend can see the gated worker to compare against. It starts a real mocker worker with scripted output against a real frontend (ModelWatcher + HttpService). Unary and streaming chat responses from both are identical, with reasoning (deepseek_r1) and tool calls (hermes) parsed. Gates get the right URL and model, and the endpoint closes on release. Stripping the tool parser on the loopback side makes the test fail.
  • ungated_worker_opens_no_listener.
  • endpoint_is_unavailable_until_the_engine_registers: 503 and gates pending until the engine registers; then the gates start.
• Unit tests in local_openai.rs cover card eligibility, port parsing, 503 before registration, and the JSON round trip.
• The runtime validation tests are updated for the moved trigger.
• cargo fmt, cargo clippy -p dynamo-llm -p dynamo-runtime -p dynamo-backend-common --all-targets -D warnings and cargo check --workspace are clean.

Constraints

• There's no in-pod signal for hostNetwork, so it isn't rejected here. Under hostNetwork the endpoint is reachable from the node while gates run; an operator-side check is needed.
• Prefill, encode, text-input and embeddings-only workers get no endpoint, so their gates never start (fail closed). Don't gate them for now.
• Decode workers are served, but their requests bypass the prefill router.
• Only the base model is served; LoRA adapters aren't.
• Parity is with a default frontend. Per-frontend flags and Python chat processors aren't mirrored.
• Error chunks are shaped as Annotated errors rather than LLMEngineOutput with finish_reason=Error. Both surface as errors, but engine-error bodies may differ slightly from the frontend's.

Stack

1. refactor(runtime): serve /live and /health from separate handlers #1 Split /live from /health
2. feat(runtime): worker validation gates #2 Runtime validation gates
3. feat(llm): hold model-card discovery writes until validation releases #3 Hold the model card until release
4. This PR: loopback OpenAI endpoint
5. Reference check runner, docs and an example DGD

🤖 Generated with Claude Code

[cubic-dev-ai]

1 issue found across 12 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="lib/runtime/src/validation/mod.rs">

<violation number="1" location="lib/runtime/src/validation/mod.rs:111">
P2: This guard is scoped to one `WorkerValidation`, not the process, so multiple `DistributedRuntime` instances can each claim a listener. With an ephemeral port this creates multiple validation endpoints; with `DYN_WORKER_VALIDATION_LOCAL_PORT` set, the later worker fails to bind. Use process-wide coordination for the one-endpoint-per-process contract.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: This guard is scoped to one WorkerValidation, not the process, so multiple DistributedRuntime instances can each claim a listener. With an ephemeral port this creates multiple validation endpoints; with DYN_WORKER_VALIDATION_LOCAL_PORT set, the later worker fails to bind. Use process-wide coordination for the one-endpoint-per-process contract.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At lib/runtime/src/validation/mod.rs, line 111:

<comment>This guard is scoped to one `WorkerValidation`, not the process, so multiple `DistributedRuntime` instances can each claim a listener. With an ephemeral port this creates multiple validation endpoints; with `DYN_WORKER_VALIDATION_LOCAL_PORT` set, the later worker fails to bind. Use process-wide coordination for the one-endpoint-per-process contract.</comment>

<file context>
@@ -108,6 +108,7 @@ pub struct WorkerValidation {
     release_hooks: parking_lot::Mutex<Vec<ReleaseHook>>,
     released: watch::Sender<bool>,
     started: AtomicBool,
+    local_endpoint_claimed: AtomicBool,
     metrics: std::sync::OnceLock<Metrics>,
 }
</file context>

[sejori]

Not changing this. One DistributedRuntime per worker process is the supported shape; two runtimes in one process each parse the env and each own their gates, so each gets its own endpoint. With a fixed port the second attach fails loudly at bind rather than two workers silently sharing one endpoint, which is the behaviour I'd want.

[sejori]

Superseded by the cut-down stack #6 → #7 → #8 → #9 → #10 → #11 (~4,950 → ~2,000 lines): #2 is split into exec gates (#7) and sidecar url gates (#8), the url contract is now 200 = pass, and metrics, the fixed loopback port, the virtual-planner change and the parity harness are dropped. The review threads here stay for reference.

🤖 Generated with Claude Code