[sejori]

Part 6 of 6 of the worker validation gates stack (DEP ai-dynamo/dynamo#15897). Based on part 5.

Summary

• dynamo.validation.completion: a built-in check that one chat completion through the loopback endpoint returns content.
• dynamo.validation.runner: a stateless check runner for url gates, run as a sidecar. POST /validate/<name> runs that check with the gate's environment and answers 200 on exit 0, else 500 with the end of stderr. Loopback only, standard library only.
• A docs page and an example DGD: an exec gate plus a sidecar url gate, with max-unavailable: 0.
• Codeowners: components/src/dynamo/validation/ goes to the runtime area.

Validation

• E2E (tests/fault_tolerance/test_worker_validation_gates.py, CPU mocker workers, real frontend, etcd and NATS), 3 passed on this stack: a failing gate holds the worker (/health 503 with its reason, /live 200, no client requests reach it) while the old worker serves; passing gates release it, including a completion check through the loopback endpoint while the card is held; a restarted worker is checked again.
• Runner unit test: 200 on pass, 500 with the stderr tail on fail, 404 for an unknown check.
• pre-commit clean; codeowners strict build at 100%.

🤖 Generated with Claude Code

[cubic-dev-ai]

4 issues found across 11 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="components/src/dynamo/validation/__init__.py">

<violation number="1" location="components/src/dynamo/validation/__init__.py:7">
P3: The completion check also passes with empty `content` when `reasoning_content` or `tool_calls` is present, so this description overstates what the gate guarantees. Describe the requirement as a non-empty message, or make the check require `content`.</violation>
</file>

<file name="docs/fern/pages/kubernetes/fault-tolerance/validation-gates.md">

<violation number="1" location="docs/fern/pages/kubernetes/fault-tolerance/validation-gates.md:49">
P2: `card` mode cannot guarantee this for every router: the scope section below lists `worker_id`-pinned standalone-router traffic as uncovered. Replace “under any router” with a discovery-routing qualifier and retain that bypass warning.</violation>
</file>

<file name="components/src/dynamo/validation/runner.py">

<violation number="1" location="components/src/dynamo/validation/runner.py:42">
P2: `run_gate` drops the URL request at its deadline, but this blocking `subprocess.run` does not observe the disconnect, so a hung check and its descendants continue after the worker records failure. Run checks in a cancellable handler and terminate their process group on cancellation.

(Based on your team's feedback about process-group cancellation.)</violation>
</file>

<file name="components/src/dynamo/validation/completion.py">

<violation number="1" location="components/src/dynamo/validation/completion.py:35">
P2: `urlopen` uses urllib's default proxy and redirect handlers, so this check can leave the worker's loopback API; a proxy without a `NO_PROXY` exception can fail the gate, while a redirect can let another endpoint satisfy it. Use an opener with proxies and redirects disabled.

(Based on your team's feedback about keeping validation clients loopback-confined.)</violation>
</file>

<sub>Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: card mode cannot guarantee this for every router: the scope section below lists worker_id-pinned standalone-router traffic as uncovered. Replace “under any router” with a discovery-routing qualifier and retain that bypass warning.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/fern/pages/kubernetes/fault-tolerance/validation-gates.md, line 49:

<comment>`card` mode cannot guarantee this for every router: the scope section below lists `worker_id`-pinned standalone-router traffic as uncovered. Replace “under any router” with a discovery-routing qualifier and retain that bypass warning.</comment>

<file context>
@@ -0,0 +1,81 @@
+`DYN_WORKER_VALIDATION_HOLD` sets what a held worker keeps back:
+
+- `card` (default): the model card isn't written to discovery until release, so no frontend routes to
+  the worker under any router. `/health` reports 503.
+- `health`: the card is written at once and only `/health` is held. Use it when the deployment keeps
+  traffic off the worker by other means.
</file context>

[cubic-dev-ai]

P2: run_gate drops the URL request at its deadline, but this blocking subprocess.run does not observe the disconnect, so a hung check and its descendants continue after the worker records failure. Run checks in a cancellable handler and terminate their process group on cancellation.

(Based on your team's feedback about process-group cancellation.)

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At components/src/dynamo/validation/runner.py, line 42:

<comment>`run_gate` drops the URL request at its deadline, but this blocking `subprocess.run` does not observe the disconnect, so a hung check and its descendants continue after the worker records failure. Run checks in a cancellable handler and terminate their process group on cancellation.

(Based on your team's feedback about process-group cancellation.) </comment>

<file context>
@@ -0,0 +1,80 @@
+            except ValueError:
+                body = {}
+            env = {key: str(body[key]) for key in GATE_ENV_KEYS if key in body}
+            result = subprocess.run(
+                argv,
+                env={**os.environ, **env},
</file context>

[cubic-dev-ai]

P2: urlopen uses urllib's default proxy and redirect handlers, so this check can leave the worker's loopback API; a proxy without a NO_PROXY exception can fail the gate, while a redirect can let another endpoint satisfy it. Use an opener with proxies and redirects disabled.

(Based on your team's feedback about keeping validation clients loopback-confined.)

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At components/src/dynamo/validation/completion.py, line 35:

<comment>`urlopen` uses urllib's default proxy and redirect handlers, so this check can leave the worker's loopback API; a proxy without a `NO_PROXY` exception can fail the gate, while a redirect can let another endpoint satisfy it. Use an opener with proxies and redirects disabled.

(Based on your team's feedback about keeping validation clients loopback-confined.) </comment>

<file context>
@@ -0,0 +1,55 @@
+        headers={"Content-Type": "application/json"},
+    )
+    try:
+        with urllib.request.urlopen(request, timeout=120) as response:
+            message = json.loads(response.read())["choices"][0]["message"]
+    except urllib.error.HTTPError as error:
</file context>

[cubic-dev-ai]

P3: The completion check also passes with empty content when reasoning_content or tool_calls is present, so this description overstates what the gate guarantees. Describe the requirement as a non-empty message, or make the check require content.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At components/src/dynamo/validation/__init__.py, line 7:

<comment>The completion check also passes with empty `content` when `reasoning_content` or `tool_calls` is present, so this description overstates what the gate guarantees. Describe the requirement as a non-empty message, or make the check require `content`.</comment>

<file context>
@@ -0,0 +1,10 @@
+"""Reference check and check runner for worker validation gates.
+
+* ``python -m dynamo.validation.completion``: one chat completion through the
+  worker's loopback OpenAI endpoint must return content.
+* ``python -m dynamo.validation.runner``: a check runner for ``url`` gates, run
+  as a sidecar with its own image.
</file context>

[sejori]

Superseded by ai-dynamo#16524 (prep) and ai-dynamo#16525 (gates), which carry this series rebased on current upstream main, including the per-gate phase that replaces DYN_WORKER_VALIDATION_HOLD (see DEP ai-dynamo#15897). Closing so there's one copy to maintain.