[tedzhouhk]

Overview

A DGDR v1beta2 search can now replay a recorded request trace instead of synthetic static traffic. This is the input side of a planner retune in a live deployment: the frontend records traces to a shared claim, and a Request evaluates candidates against one recorded window. The planner side is out of scope here.

spec:
  workload:
    trace:
      format: dynamo                    # required; only dynamo is admitted
      source:
        pvc:
          claimName: request-traces     # PVC in the Request namespace
          path: retune/w-1.jsonl.gz     # one file, relative to the claim root
      replay:
        arrivalSpeedupRatio: 2.0        # optional

Key Changes

API (api/v1beta2)

• New spec.workload.trace{format, source.pvc{claimName, path}, replay.arrivalSpeedupRatio}, exclusive with static. The shape follows the trace workload in DEP DEP: Evolve DGDR for replay-backed multi-candidate search #13092; format is limited to dynamo, which can widen compatibly later.
• The shared schema and CEL enforce the rules on Requests and Runs: claimName is a DNS-1123 subdomain; path is relative, has no empty, . or .. segment, no NUL, and at most 1024 characters; the speedup is greater than 0.
• The Run snapshot rule from feat(operator): integrate DGDR v1beta2 candidate flow #17308, which required workload.static, now accepts static or trace.
• Defaulting adds neither a static workload nor a format to a trace.

Validation (internal/webhook/validation)

• Structural validators for the shared v1beta2 workload types in shared_v1beta2.go mirror the CRD rules. They matter for a v1beta1 carrier stored without admission, whose unchanged trace CEL ratchets on the next v1beta2 write.
• inputOverrides cannot set traffic.source.type, format or paths, or replace traffic or traffic.source with a non-object. Every other traffic override, traffic.stop included, is AISimulate input that AISimulate validates.
• The Request controller no longer validates the trace. A Run create the CRD rejects as Invalid releases the reservation and reports InvalidSpec for that generation instead of retrying.

Lowering (internal/aisimulate)

• traffic.source{type: trace, format, paths: [<fixed mount path>]} and traffic.load.type: trace_timestamps, plus traffic.load.speedup when set. No traffic.stop.requests is added. The Run controller restores the typed source after the override merge.

Run controller and Job

• The Job mounts only the trace file, read-only through subPath at /var/run/aisimulate/trace/trace.jsonl.gz (or trace.jsonl), into the sweeper alone. The kubelet resolves the path inside the claim and refuses an absolute symlink or one that leaves the claim.
• The Run Job's pod sets no fsGroup, so neither kubelet nor a CSI driver changes the claim's ownership or permissions. A trace Job's pod gets group 65532 as a supplemental group. The base's runAsNonRoot, seccomp profile and numeric sweeper UID are kept.
• Claim protection. Before it creates a trace Job, the Run adds its own finalizer nvidia.com/dgdr-run-<run-uid> to the claim with an optimistic-lock patch, so a stale cache gets a Conflict instead of a Job. No Job is created for a claim that is being deleted.
• Deletion is cancellation. When the claim is being deleted (or disappears after the Run created its Job), the Run is cancelled whatever its Pod's state, through the foreground stop path that Request supersession uses (stopAndCompleteRun, which replaces cancelSupersededRun). The Run fails with Completed=True/TraceSourceDeleted only after the API server confirms the Job is gone, and then releases its finalizer. A Job that reached JobComplete or JobFailed on its own keeps its result. Stop deletes carry UID and resourceVersion preconditions, so a stale read never stops a Job that has since finished.
• Missing, block or lost claim before the first Job. A missing claim keeps the Run open (Pending, Completed=False/TraceSourceMissing), and a claim event wakes it when the claim appears. A block or lost claim fails the Run with TraceSourceUnavailable, through the existing durable permanent-failure marker.
• Waiting containers are diagnostics only. No status message drives reconciliation any more. While a Run's Job is not finished, the first sweeper or publisher container waiting for a reason other than ContainerCreating or PodInitializing sets Completed=False/ContainerWaiting with the kubelet's reason and message copied verbatim. It never ends the Run or deletes the Job.
• Release. A completed Run stops any unfinished Job it still owns (for example one a stale pass created), confirms through the API that none is left, and releases its finalizer. A new claim-keyed reconciler, DynamoGraphDeploymentRunTraceClaimReconciler, releases the finalizer of a Run that is gone once no Job of that Run is left, so release converges after Run deletion and operator restarts.
• New watches: PVCs, mapped through a Run index on the claim name; pods of Run Jobs, for the diagnostic; Run and Job deletions, for the claim reconciler. RBAC: persistentvolumeclaims get;list;watch;patch;update, pods get;list;watch, jobs delete. The generated role.yaml changes; the chart's hand-written operator rules already grant these.

Generated files and docs

• CRDs, deepcopy and the Kubernetes API reference are regenerated. The v1beta2 type-count baseline in test_gen_kubernetes_api.py goes from 35 to 40.
• The DGDR reference page has a section, "Replay a recorded trace (v1beta2)": claim and file requirements, permissions, symlinks, the mount, the lowered input and overrides. Its "Trace claim lifecycle" subsection covers protection, deletion as cancellation, the missing, block and lost cases, the ContainerWaiting diagnostic and how to recover or cancel, and removing leftover finalizers by hand.

Validation

On head f73f4bbd0b:

Build and lint

• golangci-lint v2.15.0 run ./...: 0 issues. go vet with no tags, clustertest, and clustertest,upgradecontract: clean.
• make generate manifests generate-api-docs generate-helm-docs leaves the tree unchanged. pre-commit run on the changed files and git diff --check pass.

Go tests (envtest on k8s 1.30.0): go test ./internal/... ./api/... passes (48 packages). Coverage includes:

• The admission-chain table: CEL accept and reject for Requests, carrier rows for every structural rule (with the ratcheting warnings and error aggregation), and override ownership. The Run CRD accepts a trace snapshot and rejects bad paths.
• The Run controller: the read-only single-file subPath mount and the security context; protection before Job creation, including patch failures and Conflicts; cancellation on claim deletion across Job and Pod states, with the finalizer kept until the Job is gone; the finished-Job exemption, including a Job the cache does not show yet; release after completion and the stale-pass backstop; two Runs sharing one claim; a missing claim that appears later; block and lost claims; waiting containers (missing file, escaping symlink, image pull, publisher) that keep the Job.
• The claim reconciler: release after Run deletion, holders, watch mappings and predicates.
• A pod-cache contract subtest that pins the Pod fields the diagnostic reads after the production projection.
• An envtest showing the API server returns Conflict for a fenced patch and Invalid for a new finalizer on a deleting claim, and that both are retried.
• The Request controller: an Invalid Run create is reported once and not re-reserved.

Cluster (clustertest, only in CI's Operator Integration): the vllm-trace candidate scenario, now through the default AIC generator, and the escaping-symlink test, which now expects ContainerWaiting and cleans up by deleting the claim. Not run locally.

AIC renderer: the vllm-trace scenario's Run Job command, run in the CI planner image with AISimulate 0.13.0.dev202610290000000087, reaches phase: Succeeded through the default AIC generator and renders --block-size 64 with the TODO(#17581) workaround. Without the workaround it fails on #17581.

Development cluster: before this lifecycle change, at ad3bd3050e, this PR ran with #17562 on an H200 development cluster. Frontends recorded traces to the claim, and every Run Job read its window file and succeeded. #17562's description has the details.

Not verified: real foreground garbage collection of the Job and its Pods on claim deletion, release across a live operator restart, and RBAC in a non-system:masters install.

Issues and Ongoing Work

• Relates to DEP: Evolve DGDR for replay-backed multi-candidate search #13092. feat(planner): enable live retune through DGDR v1beta2 #17562 (live retune) builds on this PR.
• Not in this PR: the planner executor that records windows and creates Requests, and its RBAC (feat(planner): enable live retune through DGDR v1beta2 #17562); the frontend trace-recording recipe; a Run Job deadline.
• Known limitations:
  • A missing trace file on a volume type whose node directory the kubelet can write, such as hostPath, makes the kubelet create an empty directory at the path; the Run then fails in AISimulate.
  • A Run whose container cannot start, for example because the trace file is missing or its symlink leaves the claim, has no timeout. It stays open with the ContainerWaiting diagnostic until the cause is fixed or the Run is cancelled by deleting the claim or changing the Request.
  • Pods of finished Jobs keep a deleted claim Terminating through the upstream kubernetes.io/pvc-protection finalizer until their Runs are retired.
  • If the operator is removed while trace Runs are in flight, their nvidia.com/dgdr-run-* finalizers stay on the claim; the DGDR reference shows how to remove them.
  • Until AISimulate DGD generation rejects candidates without an explicit block size #17581 is fixed, a v1beta2 Request without an explicit engine.workers.<role>.kv_cache.block_size fails in the default AIC renderer, trace or static. The trace scenario carries the same TODO(#17581) workaround as the static vllm scenario.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Configure workloads to replay recorded Dynamo traces from a PVC-backed file, with optional arrival-speed adjustment.
  • Trace files are mounted read-only. Runs report an unavailable trace source when the claim or file cannot be used.
• Bug Fixes
  • Trace workloads no longer receive static traffic defaults, and trace-controlled source fields cannot be overridden.
  • Invalid requests are reported without repeatedly attempting to create a Run.
• Documentation
  • Added API reference guidance for configuring trace workloads, file requirements, and override behavior.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a18efd-7dbe-75c9-9807-0fe1b489527c.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

✅ Dynamo PR CI passed — run 41139800682 (attempt 1) on 6e8087d359

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 41139800682. Updated on every full-CI run of this PR.}

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: ai-dynamo/dynamo
• Branch: hzhou/dgdr-trace-pvc
• File: .coderabbit.yaml
• Commit: ad3bd30
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: a59131dc-73dd-4111-8fd8-ee8fe898dff7

Commits

Reviewing files that changed from the base of the PR and between dff7c99 and ad3bd30.

Files selected for processing: 33

• deploy/operator/api/roundtrip_fuzz_test.go
• deploy/operator/api/v1beta2/dynamographdeploymentrequest_types.go
• deploy/operator/api/v1beta2/dynamographdeploymentrun_types.go
• deploy/operator/api/v1beta2/zz_generated.deepcopy.go
• deploy/operator/config/crd/bases/nvidia.com_dynamographdeploymentrequests.yaml
• deploy/operator/config/crd/bases/nvidia.com_dynamographdeploymentruns.yaml
• deploy/operator/internal/aisimulate/defaults.go
• deploy/operator/internal/aisimulate/doc.go
• deploy/operator/internal/aisimulate/input.go
• deploy/operator/internal/aisimulate/paths.go
• deploy/operator/internal/aisimulate/trace.go
• deploy/operator/internal/aisimulate/trace_test.go
• deploy/operator/internal/controller/clusterenv_test.go
• deploy/operator/internal/controller/dgdr_v1beta2_trace_clustertest_test.go
• deploy/operator/internal/controller/dynamographdeploymentrequest_v1beta2_controller.go
• deploy/operator/internal/controller/dynamographdeploymentrequest_v1beta2_controller_test.go
• deploy/operator/internal/controller/dynamographdeploymentrun_v1beta2_controller.go
• deploy/operator/internal/controller/dynamographdeploymentrun_v1beta2_controller_test.go
• deploy/operator/internal/controller/testdata/dgdr-v1beta2/vllm-trace/input.yaml
• deploy/operator/internal/controller/testdata/dgdr-v1beta2/vllm-trace/output.yaml
• deploy/operator/internal/controller/testdata/dgdr-v1beta2/vllm-trace/trace.jsonl
• deploy/operator/internal/webhook/defaulting/dynamographdeploymentrequest_v1beta2.go
• deploy/operator/internal/webhook/defaulting/suite_envtest_test.go
• deploy/operator/internal/webhook/validation/dynamographdeploymentrequest_carrier_envtest_test.go
• deploy/operator/internal/webhook/validation/dynamographdeploymentrequest_v1beta2.go
• deploy/operator/internal/webhook/validation/dynamographdeploymentrequest_v1beta2_input_overrides.go
• deploy/operator/internal/webhook/validation/dynamographdeploymentrequest_validation_envtest_test.go
• deploy/operator/internal/webhook/validation/shared_helpers.go
• deploy/operator/internal/webhook/validation/shared_v1beta2.go
• docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md
• docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment-request.mdx
• docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx
• docs/fern/scripts/tests/test_gen_kubernetes_api.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The v1beta2 API now supports PVC-backed Dynamo trace workloads alongside static workloads. Admission validates trace configuration and defaults, AISimulate lowering creates trace input, and the Run controller mounts and monitors trace files.

Changes

DGDR v1beta2 trace workloads

Layer / File(s)	Summary
Trace workload API and schema deploy/operator/api/*, deploy/operator/config/crd/bases/*, docs/fern/pages/reference/kubernetes-api/*, docs/fern/scripts/tests/test_gen_kubernetes_api.py	The API and CRD schemas add a Dynamo-format trace source backed by a PVC, optional arrival speedup, and static-versus-trace exclusivity. The references, examples, generated deep-copy methods, and fuzz tests cover the new workload.
Admission defaults and validation deploy/operator/internal/webhook/defaulting/*, deploy/operator/internal/webhook/validation/*, deploy/operator/internal/controller/dynamographdeploymentrequest_v1beta2_controller*	Defaulting skips static traffic defaults for trace workloads. Validation checks trace fields, PVC paths, override restrictions, and conversion-carried workloads. Request reconciliation records rejected Run creation as InvalidSpec.
Trace input lowering deploy/operator/internal/aisimulate/*, deploy/operator/internal/controller/testdata/dgdr-v1beta2/vllm-trace/*	AISimulate lowering sets trace source and timestamp-based load fields. It maps source paths to fixed mount filenames and omits the request-count default for trace traffic. Tests and fixtures cover lowered trace input and a vLLM trace candidate.
Run trace mounting and failure handling deploy/operator/internal/controller/dynamographdeploymentrun_v1beta2_controller*, deploy/operator/internal/controller/dgdr_v1beta2_trace_clustertest_test.go, deploy/operator/internal/controller/clusterenv_test.go	The Run controller mounts the source file read-only in the sweeper, watches PVC and Job-pod events, and records trace-source failures. It deletes unfinished Jobs after those failures. Controller and cluster tests cover claim availability, mount errors, overrides, and file integrity.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: Minimal · up to ad3bd

This change adds PVC-backed recorded-trace replay for DGDR v1beta2. No remaining merge-blocking risk was found in the supplied evidence. Cluster-level behavior is reported as verified only by CI's Operator Integration, which is normal validation uncertainty and not a concrete defect.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 39.39% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly and concisely describes the main change: replaying recorded traces from a PVC in DGDR v1beta2 Runs.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description is complete and follows the required template. It includes Overview, Key Changes, Validation, and Issues and Ongoing Work sections, with detailed scope, testing, limitations, and follo…

Full details: Description check

Explanation

The description is complete and follows the required template. It includes Overview, Key Changes, Validation, and Issues and Ongoing Work sections, with detailed scope, testing, limitations, and follow-up work. The checklist is present but remains unchecked, and no closing issue link is provided; these are non-critical for this description check.

Full details: Docstring Coverage

Explanation

Docstring coverage is 39.39% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 66 functions across 25 files. (8 skipped: 8 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[sttts]

Three follow-up comments on trace-PVC lifetime/cancellation, diagnostic-only Pod errors, and exercising the default AIC renderer. Source review only; no tests were executed.

[jthomson04]

Reviewed at 6e8087d. The finalizer and cancellation lifecycle holds up: every finalizer and status write is resourceVersion-fenced, finalizers are keyed by Run UID, and Job absence is confirmed through the uncached reader. Five nits inline, none blocking.

[jthomson04]

Nothing calls Update on a PVC; every write here and in the claim reconciler is a Patch. Drop update so the generated role.yaml doesn't grant it cluster-wide.

[jthomson04]

When the cache shows no Jobs, this lists every Job in the namespace from the API, then stopOwnedJobs (:363) lists them all again because jobs is still empty, once per dgdrRunExecutionCacheRetry. Pass the confirmed result through. A Run-UID label on the Job would let this and runHoldsClaim use a selector instead of an unfiltered list.

[jthomson04]

Every claim event re-queues every Run on the claim, completed ones included. With live retune (#17562) all Runs share one claim, so each finalizer add or remove fans out to all of them: O(N²) reconciles. Enqueue only Runs that aren't Completed or still hold a finalizer on the claim.

[jthomson04]

On this path no Job doesn't mean no pod. A Run deleted directly has no finalizer, so GC deletes its Jobs with background propagation and the release can land while the pod still has the subPath mounted. kubernetes.io/pvc-protection keeps the object, so nothing breaks, but the claimName doc's "until its execution stops" overstates it. Either foreground-delete the gone Run's Jobs here or narrow the doc.

[jthomson04]

This uncached, unfiltered namespace Job list runs once per stale finalizer. Hoist it out of the loop, or select on a Run-UID label.