[sejori]

Part 2 of 6 of the worker validation gates stack (DEP ai-dynamo/dynamo#15897). Based on part 1. Together with parts 3–6 it replaces #2–#5, cut down from ~4,950 to ~2,000 lines.

Summary

A worker can run checks against itself before it reports ready, so a broken worker stalls a rollout instead of serving.

DYN_WORKER_VALIDATION_GATES='{"smoke": {"exec": ["python3", "/checks/smoke.py"], "timeout_secs": 900}}'

• Each gate runs once, in parallel, when the worker's first local engine registers (part 5 moves this to the loopback endpoint). Exit 0 is a pass; otherwise the end of stderr is the reason. A gate past its deadline fails and its process is killed.
• /health reports 503 with each gate's state and reason until every gate passes. /live is unchanged, so a held worker isn't restarted.
• Verdicts live in memory: a restarted container is checked again, and a failed gate holds the worker until the pod is deleted.
• Startup fails on bad config, without the system status server, or in sidecar mode.

Unset, nothing changes.

Validation

Config parsing and error table, a failing gate holding the worker, the gate deadline, /health held while /live stays 200 for passing and failing gates, and the status-server requirement.

🤖 Generated with Claude Code

[cubic-dev-ai]

5 issues found across 8 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="lib/runtime/src/validation/runner.rs">

<violation number="1" location="lib/runtime/src/validation/runner.rs:25">
P2: A timed-out check's stderr is discarded, so `/health` reports only `no verdict within ...` even when the process wrote a useful failure reason. Preserve the captured tail when returning the timeout verdict.</violation>

<violation number="2" location="lib/runtime/src/validation/runner.rs:39">
P2: A gate can leave its subprocesses running after cancellation or timeout: `kill_on_drop` terminates only the direct child. Start each check in its own process group and kill the group when the future is dropped.

(Based on your team's feedback about process-group cleanup on cancellation.)</violation>

<violation number="3" location="lib/runtime/src/validation/runner.rs:45">
P1: A verbose check can exhaust worker memory: `read_to_end` buffers all stderr even though the code retains only the last 2 KiB. Drain stderr into a bounded rolling tail buffer.</violation>
</file>

<file name="lib/runtime/src/validation/config.rs">

<violation number="1" location="lib/runtime/src/validation/config.rs:51">
P1: `std::env::var` returns `NotUnicode` for a non-UTF-8 value, and `.ok()` turns that into `None`, disabling validation and starting the worker unchecked. Treat only `NotPresent` as no configuration; return an error for `NotUnicode`.</violation>
</file>

<file name="lib/runtime/src/distributed.rs">

<violation number="1" location="lib/runtime/src/distributed.rs:498">
P1: This check verifies only that the status server is enabled; a bind failure still reaches the Worker log-and-continue arm, so a gated worker starts without `/health`. Return the bind error for gated workers while preserving the existing non-gated behavior.

(Based on your team's feedback about gated status server bind failures.)</violation>
</file>

<sub>Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P1: A verbose check can exhaust worker memory: read_to_end buffers all stderr even though the code retains only the last 2 KiB. Drain stderr into a bounded rolling tail buffer.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At lib/runtime/src/validation/runner.rs, line 45:

<comment>A verbose check can exhaust worker memory: `read_to_end` buffers all stderr even though the code retains only the last 2 KiB. Drain stderr into a bounded rolling tail buffer.</comment>

<file context>
@@ -0,0 +1,74 @@
+
+    let mut stderr = Vec::new();
+    let mut pipe = child.stderr.take().expect("stderr is piped");
+    let (status, _) = tokio::join!(child.wait(), pipe.read_to_end(&mut stderr));
+    let status = status.map_err(|error| format!("waiting for {:?}: {error}", argv[0]))?;
+    if status.success() {
</file context>

[cubic-dev-ai]

P1: std::env::var returns NotUnicode for a non-UTF-8 value, and .ok() turns that into None, disabling validation and starting the worker unchecked. Treat only NotPresent as no configuration; return an error for NotUnicode.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At lib/runtime/src/validation/config.rs, line 51:

<comment>`std::env::var` returns `NotUnicode` for a non-UTF-8 value, and `.ok()` turns that into `None`, disabling validation and starting the worker unchecked. Treat only `NotPresent` as no configuration; return an error for `NotUnicode`.</comment>

<file context>
@@ -0,0 +1,144 @@
+    /// Read the configuration from the environment. `Ok(None)` when no gates
+    /// are configured, which keeps today's behaviour unchanged.
+    pub fn from_env() -> Result<Option<Self>> {
+        Self::parse(std::env::var(env_validation::DYN_WORKER_VALIDATION_GATES).ok())
+    }
+
</file context>

Suggested change

	Self::parse(std::env::var(env_validation::DYN_WORKER_VALIDATION_GATES).ok())
	let gates = match std::env::var(env_validation::DYN_WORKER_VALIDATION_GATES) {
	Ok(gates) => Some(gates),
	Err(std::env::VarError::NotPresent) => None,
	Err(error) => return Err(error.into()),
	};
	Self::parse(gates)

[cubic-dev-ai]

P1: This check verifies only that the status server is enabled; a bind failure still reaches the Worker log-and-continue arm, so a gated worker starts without /health. Return the bind error for gated workers while preserving the existing non-gated behavior.

(Based on your team's feedback about gated status server bind failures.)

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At lib/runtime/src/distributed.rs, line 498:

<comment>This check verifies only that the status server is enabled; a bind failure still reaches the Worker log-and-continue arm, so a gated worker starts without `/health`. Return the bind error for gated workers while preserving the existing non-gated behavior.

(Based on your team's feedback about gated status server bind failures.) </comment>

<file context>
@@ -477,6 +486,43 @@ impl DistributedRuntime {
+            return Ok(None);
+        };
+        anyhow::ensure!(
+            config.system_server_enabled(),
+            "worker validation gates hold /health, so they need the system status server; \
+             set DYN_SYSTEM_PORT"
</file context>

[cubic-dev-ai]

P2: A timed-out check's stderr is discarded, so /health reports only no verdict within ... even when the process wrote a useful failure reason. Preserve the captured tail when returning the timeout verdict.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At lib/runtime/src/validation/runner.rs, line 25:

<comment>A timed-out check's stderr is discarded, so `/health` reports only `no verdict within ...` even when the process wrote a useful failure reason. Preserve the captured tail when returning the timeout verdict.</comment>

<file context>
@@ -0,0 +1,74 @@
+        .await
+        .unwrap_or_else(|_| {
+            Err(format!(
+                "no verdict within timeout_secs={}",
+                gate.timeout.as_secs()
+            ))
</file context>

[cubic-dev-ai]

P2: A gate can leave its subprocesses running after cancellation or timeout: kill_on_drop terminates only the direct child. Start each check in its own process group and kill the group when the future is dropped.

(Based on your team's feedback about process-group cleanup on cancellation.)

View Feedback

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At lib/runtime/src/validation/runner.rs, line 39:

<comment>A gate can leave its subprocesses running after cancellation or timeout: `kill_on_drop` terminates only the direct child. Start each check in its own process group and kill the group when the future is dropped.

(Based on your team's feedback about process-group cleanup on cancellation.) </comment>

<file context>
@@ -0,0 +1,74 @@
+        .env("DYN_VALIDATION_GATE", gate)
+        .stdin(Stdio::null())
+        .stderr(Stdio::piped())
+        .kill_on_drop(true)
+        .spawn()
+        .map_err(|error| format!("could not run {:?}: {error}", argv[0]))?;
</file context>

[sejori]

Superseded by ai-dynamo#16524 (prep) and ai-dynamo#16525 (gates), which carry this series rebased on current upstream main, including the per-gate phase that replaces DYN_WORKER_VALIDATION_HOLD (see DEP ai-dynamo#15897). Closing so there's one copy to maintain.