[sejori]

Part 4 of 6 of the worker validation gates stack (DEP ai-dynamo/dynamo#15897). Based on part 3.

Summary

Under etcd discovery the frontend routes to any endpoint with a model card, whatever the pod's readiness, so holding /health alone doesn't keep traffic off a worker under validation.

• DYN_WORKER_VALIDATION_HOLD=card (default): register_model_card returns as usual, but the discovery write waits. When every gate passes, held cards are written in registration order, then the worker reports ready.
• DYN_WORKER_VALIDATION_HOLD=health: the card is written at once and only /health is held, for deployments that keep traffic off the worker by other means.
• A held card can be unregistered but not edited: taint updates are refused until release.
• If a held card can't be written, the worker shuts down so its gates run again on restart; its discovery lease removes any card already written.

The runtime gains a single release hook that runs after the gates pass and before release; a failed hook keeps the worker held.

Validation

A card is written only on release and taint updates are refused while it's held; hold=health writes it at once; a card unregistered while held is never written; a failed release hook keeps the worker held; config rejects HOLD without gates or with an unknown value.

🤖 Generated with Claude Code

[cubic-dev-ai]

Running ultrareview automatically — this PR appears to change concurrency-sensitive code such as locks, queues, or retries, where a missed race may only surface under production load, so a deeper multi-pass review is worth running. I'll post findings when complete.

[cubic-dev-ai]

Ultrareview ran automatically — this PR appears to change concurrency-sensitive code such as locks, queues, or retries, where a missed race may only surface under production load, so a deeper multi-pass review is worth running.

3 issues found across 6 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="lib/runtime/src/validation/config.rs">

<violation number="1" location="lib/runtime/src/validation/config.rs:133">
P2: Reject the gate name `publish` during parsing because the release flow reserves it for the synthetic release-hook gate. Otherwise a user-defined gate can collide with that lifecycle gate and produce incorrect validation/release behavior.</violation>
</file>

<file name="lib/llm/src/local_model.rs">

<violation number="1" location="lib/llm/src/local_model.rs:493">
P1: This hold path accepts a second registration for the same endpoint/suffix without conflict checking or replacement. A different model then shuts down the worker only at release, while a same-name update publishes the stale first card; reject conflicts immediately and replace same-name held entries.

(Based on your team's feedback about held-card registration conflicts.)</violation>

<violation number="2" location="lib/llm/src/local_model.rs:493">
P1: A release failure can leave cards already written by that release in discovery, especially on Kubernetes, because `publish` shuts down without rolling them back. Unregister every successfully written card before returning the release error.

(Based on your team's feedback about release rollback.)</violation>
</file>

<sub>Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P1: This hold path accepts a second registration for the same endpoint/suffix without conflict checking or replacement. A different model then shuts down the worker only at release, while a same-name update publishes the stale first card; reject conflicts immediately and replace same-name held entries.

(Based on your team's feedback about held-card registration conflicts.)

View Feedback

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At lib/llm/src/local_model.rs, line 493:

<comment>This hold path accepts a second registration for the same endpoint/suffix without conflict checking or replacement. A different model then shuts down the worker only at release, while a same-name update publishes the stale first card; reject conflicts immediately and replace same-name held entries.

(Based on your team's feedback about held-card registration conflicts.) </comment>

<file context>
@@ -486,9 +486,14 @@ pub async fn register_model_card(
-    let _instance = discovery.register(spec).await?;
+    // Under worker validation the discovery write may be held until release.
+    if let Some(spec) =
+        crate::validation_hold::hold_card(endpoint, model_suffix.as_deref(), spec).await
+    {
+        discovery.register(spec).await?;
</file context>

[cubic-dev-ai]

P1: A release failure can leave cards already written by that release in discovery, especially on Kubernetes, because publish shuts down without rolling them back. Unregister every successfully written card before returning the release error.

(Based on your team's feedback about release rollback.)

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At lib/llm/src/local_model.rs, line 493:

<comment>A release failure can leave cards already written by that release in discovery, especially on Kubernetes, because `publish` shuts down without rolling them back. Unregister every successfully written card before returning the release error.

(Based on your team's feedback about release rollback.) </comment>

<file context>
@@ -486,9 +486,14 @@ pub async fn register_model_card(
-    let _instance = discovery.register(spec).await?;
+    // Under worker validation the discovery write may be held until release.
+    if let Some(spec) =
+        crate::validation_hold::hold_card(endpoint, model_suffix.as_deref(), spec).await
+    {
+        discovery.register(spec).await?;
</file context>

[cubic-dev-ai]

P2: Reject the gate name publish during parsing because the release flow reserves it for the synthetic release-hook gate. Otherwise a user-defined gate can collide with that lifecycle gate and produce incorrect validation/release behavior.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At lib/runtime/src/validation/config.rs, line 133:

<comment>Reject the gate name `publish` during parsing because the release flow reserves it for the synthetic release-hook gate. Otherwise a user-defined gate can collide with that lifecycle gate and produce incorrect validation/release behavior.</comment>

<file context>
@@ -99,7 +130,7 @@ impl ValidationConfig {
             })
             .collect::<Result<_>>()?;
-        Ok(Some(ValidationConfig { gates }))
+        Ok(Some(ValidationConfig { gates, hold }))
     }
 }
</file context>

Suggested change

	Ok(Some(ValidationConfig { gates, hold }))
	if gates.iter().any(|gate| gate.name == "publish") {
	bail!("validation gate name \"publish\" is reserved");
	}
	Ok(Some(ValidationConfig { gates, hold }))

[sejori]

Superseded by ai-dynamo#16524 (prep) and ai-dynamo#16525 (gates), which carry this series rebased on current upstream main, including the per-gate phase that replaces DYN_WORKER_VALIDATION_HOLD (see DEP ai-dynamo#15897). Closing so there's one copy to maintain.