Repository navigation
Conversation
Reads the pod-local dgdr_run_snapshot.yaml written by the dgdr_run output adapter and publishes it on the DGDR run: creates immutable DGDCs with stable, rank-independent ids, patches run status (message, progress, ordered candidate refs), then deletes candidates that left the top N, in that order. The terminal snapshot is the completion barrier; sweeper exit codes and a missing or invalid terminal snapshot map to publisher exit codes 0/1/3. The v1beta2 types are placeholders under internal/dgdrrunpublisher/placeholderapi until the DGDR v1beta2 API lands.
…hot; trim redundant tests/comments
…/manifests, reject regressing snapshots
Writes an atomic pod-local snapshot for the DGDR run publisher: stable rank-independent candidate ids, bounded and coalesced updates, and a terminal snapshot as the completion barrier. Candidate-to-DGD rendering reuses dgd.renderers. Live updates need aisimulate#384 (subscribe(config, context)); without it the adapter writes a single snapshot from write(). Routes output/dgdr_run/ to the planner and operator owners, like output/dgd/.
…ncomplete candidates
…cs, bounded errors, require lifecycle hooks
…omments and tests
…terminal snapshot; trim tests
…ntroller Split the publisher into an input-observation loop and a Reconciler that share one workqueue key (the Run's namespace/name). The observer stores semantic snapshot changes atomically and backs off while input is unchanged. The Reconciler creates missing candidates with server-side apply, populates candidate status, updates run progress and ordered references only where they differ, then deletes obsolete candidates. The manager cache is scoped to this run, its candidates and its own pod. Sidecar lifecycle, SweeperState check, exit semantics and Completed ownership are unchanged. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…ion, write-echo handling, reconciler tests, ctrl logger)
…from observed state Remove written/applied/lastProgress from the Reconciler. Each pass decides from the cached run and candidates; status writes stay constrained to the observed resourceVersion. CreateCandidate returns the server's copy so a stale cache cannot make the publisher rewrite populated candidate status.
Defaulting and validation for native v1beta2 DynamoGraphDeploymentRequests, on top of the conversion API (ai-dynamo#13744). Admission runs through the shared v1beta1 webhooks (matchPolicy: Equivalent) and is routed by the original request's API version: a native v1beta2 request is decoded from spec.v1beta2, defaulted, validated and re-encoded; a foreign-version carrier is never mutated, direct carrier creates are rejected, and an unchanged foreign view can still be updated. Defaults: backends, workload (static ISL 1024, OSL 128, concurrency 10), objective (Optimize, Throughput), recommendation (maxCandidates 1), hardware discovery on CREATE only, and the operator's AISimulate image. The structural defaults are applied in admission, not the CRD schema, because a schema default would also be applied to an object that carries spec.v1beta1 and conversion would then reject it; only nested defaults stay in the schema. Explicit empty objects (workload.static: {}, objective.sla: {}) are kept. The AISimulate input is derived from the typed spec with derived defaults (request_count, sweep.max_trials, strict_sla) computed first and spec.aisimulate.parameters merged last, once. The base's early returns for requests carrying spec.v1beta2 are removed so those requests are validated. internal/runtimeversion is renamed internal/versionoverride. Tests: defaulting and routing unit tests, a CRD drift test (no top-level defaults; nested defaults equal the webhook's), admission envtest rows, an API-server test that carrier objects stay writable through every served version, and a Helm unit test for matchPolicy. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…unchanged carrier Objective.metric: the Optimize default is Throughput, not ThroughputPerGPU. Workload: the fixed-concurrency default creates neither sweep.max_trials nor a KV-load ratio search. The carrier test now also writes the unchanged foreign view back with a full update and requires the spec to be byte-stable and the generation unchanged, in both directions. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…ssion Validate the lowered input, so hardware given only in spec.aisimulate.parameters counts and a null that deletes discovered hardware does not. Updates validate only when the spec changed. Rewrite the aisimulate package contract to match. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…dline Split the flag description (lll). Drop the constant kind parameter of the test helper (unparam). Match the lowercased error text for the DCGM-not-enabled reason, which never matched. Give the DCGM phase of GPU discovery half of the remaining deadline so the node-label fallback keeps time to run. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…a deadline The DCGM phase reserved time for the fallback by calling NodeLabels() before DCGM ran. Callers without a deadline, such as the v1beta1 reconciler, never needed that and could hit a callback that was not configured for them. Evaluate it only when there is a deadline to share, and test that it is not asked otherwise. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
Create one immutable Run per Request generation, with the effective AISimulate input frozen into aisimulate.parameters and the version resolved. Wait for an active Run before starting the next, record the active and the five most recent terminal Runs in the status, and register the controller only when --dgdr-v1beta2-enabled is set and the cluster serves the v1beta2 APIs. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…ming Label Runs with the Request UID (label values are limited to 63 characters), refuse to adopt a Run of the same name owned by another object, and report an over-long derived Run name as InvalidSpec. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…against the v1beta2 CRDs Add an envtest for the publisher's writer: candidate and run-status payloads are accepted by the real schema, the publisher owns the fields it writes, resourceVersion constraints produce conflicts, omitted fields are removed, and candidate deletion honors the UID precondition. Register the package for make envtest and wrap two long lines in the publisher command. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
Publish the Accepted condition, gate the next Run on the Run's Completed condition, use the controller OwnerReference (with a cache index) as the only ownership source, derive the Run generation from the immutable Run name, publish status before deleting retired Runs, and add InputForRun so Run execution reads the frozen input instead of lowering the Run again. Replace the direct-Reconcile envtest with a manager-backed one and cover coalesced generations, null-deleted input keys and status-patch failure. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
Keep <request-name>-<generation> when it fits; otherwise truncate the Request name and add a stable hash so long names stay valid and distinct. Validate a Run's generation by recomputing its expected name. Register the v1beta2 scheme once in the manager. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…ssion Admission validates the typed spec and the image/version selection only. Drop the parameters object check and the effective-hardware validation; hardware discovery reads spec.hardware alone. A Request whose discovery fails is admitted and reported as Accepted=False/InvalidSpec by the Request controller. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…aults
Remove the +kubebuilder:default markers for workload.static, objective.mode
and recommendation.maxCandidates, and the test that compared them with the
webhook constants. Default in the order of the Go API types: each parent
creates its documented default child and fills its own scalars, and
workload.static: {} stays an opt-out.
Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…ange spec.hardware Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…re the GPU discoverer Default through the NativeSpec and NativeSearchInput types in declaration order. Make the aisimulate native paths package-private and drop the unused ones. HardwareDiscoverer now has the signature of gpu.Discoverer, so the shared discoverer satisfies it directly and the catalog mapping lives in resolveHardware. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…nd CEL Backend uniqueness is +listType=set and the kvLoadRatio range order is CEL on the type. Native model presence and the model name length were already enforced by the representation CEL and MinLength, so the webhook no longer repeats any of them. The operatorenv table covers the exact error paths. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…ssion table Move the externally observable defaulting cases into the admission table and delete v1beta2SchemaDefaulted and the direct-function matrix. The unit tests that remain cover resolver and envelope internals. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…dary Resolve the admission request once in Validate and ValidateUpdate and fail when it is absent. Compare the requested version with the spec.v1beta2 carrier directly and drop RequestUsesV1Beta2, RequestIsNative and requestIsNative. The mutating handler returns the missing-request error instead of skipping defaulting. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
…uest controller Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VnCXbx7DMQHVwxwN8Dnutz Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
The admission change made the AISimulate path variables private. The Request controller read two of them to check for a GPU SKU and budget. Expose that answer as aisimulate.HardwareNamed instead. Signed-off-by: devivasudevan <49675305+devivasudevan@users.noreply.github.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
sttts
force-pushed
the
sttts-dgh-1426-dgdr-run-controller
branch
4 times, most recently
from
October 28, 2026 17:32
b6c7d16 to
5ba92a0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On top of ai-dynamo#17195, keep DGDR Run execution on the standard
aisimulate recommend --stack dynamoCLI path while making opaque Planner image tags usable.CI and integration builds tag the configured Planner image with a commit SHA rather than a Dynamo semantic version. The
dgdr_runoutput adapter needs the canonical Dynamo version to derive runtime images. The operator already owns that version, so this change passes its normalizedMAJOR.MINOR.PATCHvalue into the Run reconciler and emitsdgdr_run.runtime_version_overrideonly when the Planner image tag cannot provide one.There is deliberately no custom runner or alternate AISimulate schema path. The companion Admission/lowering PR changes the generated input to the public recommendation schema consumed by this existing CLI command.
Tracks DGH-1426.
Compatibility
dgdr_runoutput adapter remain the execution boundary.Validation
go test ./cmd/... ./internal/controller -count=1with Kubernetes envtest assets: passed