[devivasudevan]

Summary

Adds admission for native v1beta2 DynamoGraphDeploymentRequest objects: defaulting (mutating) and validation (validating) webhooks. After conversion, a v1beta2 Request reaches both webhooks as a v1beta1 hub object carrying spec.v1beta2.

Design

Defaults have one owner: admission. Backends, workload.static, objective.mode and the scalar metric, recommendation.maxCandidates, cluster hardware and the AISimulate image are all defaulted by the webhook. The CRD carries no default markers for them, because the hub holds v1beta2 as an opaque envelope and the API server defaults in the webhook's version. The traversal mirrors the Go API tree (Spec → NativeSpec → NativeSearchInput → children, in declaration order). It preserves explicit values including empty objects, is idempotent, and treats workload.static: {} as an opt-out.

Hardware discovery uses the shared gpu.Discoverer through a consumer-owned interface with its exact signature. The AISimulate catalog mapping (unknown GPU model → empty SKU) happens at the v1beta2 boundary.

• CREATE discovers every omitted typed hardware field.
• An UPDATE that leaves spec.hardware unchanged does not contact the cluster or mutate the spec, metadata-only updates included.
• An UPDATE that changes spec.hardware discovers only the missing fields.
• Explicit values are always kept, and spec.aisimulate.parameters is never consulted.
• Failed discovery leaves the hardware unresolved and the Request is admitted.

Validation covers the typed API and the image and version selection: an AISimulate image is required, and versionOverride is required when the image tag has no parseable version. It runs on every CREATE and UPDATE, with no whole-spec equality shortcut. Rules the source schema can express stay with the schema: model presence (representation CEL), model.name length, backend uniqueness (+listType=set) and the kvLoadRatio range order (CEL).

spec.aisimulate.parameters is opaque to admission. It is stored untouched and never decides a default. Lowering (typed values and derived defaults as the underlay, parameters as an RFC 7386 merge patch applied last) happens in the reconciler (#17097), which freezes the result into the Run.

Conversion carriers. The requested version is resolved once from the admission request, and a missing request fails the call. A foreign-version view is read-only: an unchanged spec round-trips, any change is rejected.

Supporting changes

• Shared GPU discovery package used by the v1beta1 reconciler and v1beta2 admission
• AISimulate default-image flag and Helm value
• v1beta2 scheme registration
• Regenerated CRDs and API docs

Tests

• operatorenv admission table: model-only result, explicit values, empty parents (workload, workload.static, objective, sla, recommendation), Pareto objective, hardware lifecycle (failed discovery, recovery, metadata-only update, removing budget, SKU or all hardware), schema and CEL error paths, a stored invalid spec rejected again on update
• Unit tests for the internal resolver (discovery filtering, budget cap, failure, timeout, per-operation rules), old-spec decoding and envelope stability
• AISimulate lowering and merge-patch tests; Helm matchPolicy tests

Behaviour changes to review

• CRD default markers for workload.static, objective.mode and recommendation.maxCandidates are removed, so the Run CRD loses them too. This is harmless because the controller always writes a fully resolved Run.
• Failed discovery no longer rejects the Request. Hardware can stay unresolved when it is not in parameters; this surfaces when the Run executes.
• A whitespace-only model.name now passes, because the schema enforces only a minimum length of 1.

Out of scope

• Per-version validation of the lowered AISimulate input, to be driven by generated JSON Schema from the pinned AISimulate models and ratcheting only that result
• Rebasing onto refactor(operator): structure admission defaulting #16540 once it merges; this code lives in the _v1beta2 files and calls nothing that PR changes

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[dynamo-ops]

/ok to test 79d7fde

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a162e7-5bd1-729d-bd19-314bc7ed592a.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

✅ Dynamo PR CI passed — run 39753620443 (attempt 1) on 629636cbf6

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 39753620443. Updated on every full-CI run of this PR.}

[sttts]

Detailed re-review findings for the v1beta2 DGDR admission implementation.

[dynamo-ops]

/ok to test c3058df

[dynamo-ops]

/ok to test ea4eb12

[dynamo-ops]

/ok to test 7a00fc2

[sttts]

CI is currently blocked before admission envtests run

At HEAD cb03208943, the operator job fails in its generated-file check. The generator removes stale CRD-derived defaults from the checked-in operator API references, including objective.mode, recommendation.maxCandidates, and workload.static, and changes the workload text from schema defaulting to admission defaulting.

Please regenerate and commit the CRD/API documentation from the source definitions rather than editing generated files manually. Because that check fails first, the later operator Go/Python integration step is skipped; consequently the new admission/operatorenv table has not actually run in CI on this HEAD. After updating the generated artifacts, please rerun the operator job and confirm that the admission envtests execute and pass.

[sttts]

The mixed-node count is now tracked as a shared v1beta1/v1beta2 follow-up. The AISimulate compatibility version still needs an explicit planner-version mapping in this PR.

[sttts]

Decision: AISimulate execution is operator-owned and version-locked

We resolved the compatibility question by removing per-Request AISimulate artifact selection. A DGDR is not a supported compatibility boundary to an independently versioned or skewed AISimulate installation.

The operator will always run the AISimulate version contained in the planner image configured for that operator release. The later Job reconciler reads that image directly from operator/Helm configuration; it does not copy the image or an AISimulate version into the Request or Run API.

1. Fixed release bundle

The two-container Job topology remains. What changes is ownership of the versions: both containers are selected by the running operator release, rather than one of them being selected by the DGDR.

flowchart LR
    HELM["Helm/operator configuration<br/>operator release + matching planner image"]
    OP["Running operator<br/>admission + controllers"]
    DGDR["DGDR v1beta2<br/>typed search intent<br/>aisimulate.inputOverrides"]
    RUN["Immutable DGDRRun<br/>accepted native input snapshot"]
    JOB["Fixed two-container Job"]
    PUB["Publisher/helper container<br/>operator-owned image"]
    EXEC["AISimulate execution container<br/>operator-configured planner image"]
    BUNDLE["One tested release bundle<br/>Python host<br/>AISimulate<br/>Dynamo simulator/providers<br/>DGD generator/output adapter"]
    STATUS["Run status<br/>aisimulate.input"]
    CAND["Candidate resources<br/>DGD + metrics"]

    HELM --> OP
    HELM --> JOB
    DGDR --> OP
    OP --> RUN
    RUN --> JOB
    JOB --> PUB
    JOB --> EXEC
    EXEC --> BUNDLE
    BUNDLE --> STATUS
    BUNDLE --> CAND

Loading

The output adapter may eventually be consumed natively from the planner image or be exported by the publisher container. Because the whole Job is version-locked to one Dynamo/operator release, that choice is a reconciler packaging detail, not a cross-version compatibility feature and not a DGDR API option.

2. Kubernetes API contract

The user-facing AISimulate field contains only an optional native-input override:

spec:
  model:
    name: meta-llama/Llama-3.1-8B-Instruct
  aisimulate:
    inputOverrides:
      workload:
        request_count: 2000

inputOverrides is an RFC 7386 JSON Merge Patch over the operator-generated input. It uses upstream AISimulate field names, has highest precedence, and null deletes a lowered field.

The immutable Run preserves the accepted typed input and the override. The exact document used to invoke AISimulate is observable separately:

status:
  aisimulate:
    input:
      model:
        name: meta-llama/Llama-3.1-8B-Instruct
      workload:
        request_count: 2000
      # ...the complete effective native input...

status.aisimulate.input is output/provenance of the execution, not another override surface.

3. Lowering contract

static native defaults
        +
typed DGDR underlay
        +
spec.aisimulate.inputOverrides   (RFC 7386; last writer wins; null deletes)
        =
effective native AISimulate input
        |
        +--> status.aisimulate.input
        |
        +--> exact same document passed to the Job

These are the intentional boundaries:

Boundary	Contract
Kubernetes API	Typed DGDR fields, spec.aisimulate.inputOverrides, and status.aisimulate.input
Operator to Job	The effective native input plus the operator-configured planner image
Inside the planner image	Python ABI, AISimulate, simulator/providers, result types, generator and output adapter as one tested implementation bundle
Job output to controller	Generated DGD/Candidate data and metrics

Private AISimulate structures such as SweepResult, Candidate.config, provenance snapshots and Python package internals do not become persisted Kubernetes APIs.

4. Removed API and admission behavior

This decision removes:

• spec.aisimulate.image;
• spec.aisimulate.versionOverride;
• corresponding Run fields;
• admission defaults and CLI/Helm flags that persisted those values;
• image-tag-to-AISimulate-version resolution and a Dynamo-to-AISimulate compatibility table;
• the native-versus-injected adapter choice as a user-visible compatibility mechanism.

The generic runtime-version resolver remains specific to Dynamo runtime images; AISimulate is not a Dynamo runtime.

5. Follow-up ownership

The Job reconciler must consume the planner image from operator/Helm configuration directly, lower the immutable Run input exactly once, write the resulting document to status.aisimulate.input, and pass that same document to the execution Job.

Validation of inputOverrides against the JSON Schema generated from the bundled AISimulate/Pydantic model remains a follow-up. It only needs to target the one AISimulate version shipped with the operator-owned planner image; it does not need a per-DGDR compatibility matrix or version selector.

6. Implementation

Implemented in devivasudevan/dynamo#4.

[dynamo-ops]

Automatic CI Approval Skipped

Automatic approval requires every commit to have a signature that GitHub reports as verified. The following commits do not meet that requirement:

Commit	Reason
f2c2a7e	unsigned

To use automatic approval, sign the listed commits and push again. Alternatively, a maintainer can review this head and comment /ok to test 82b1eabd48ecfd6f1a2e9f397800f5b041ddb3f6; you do not need to rewrite the commits for that manual approval path.

[Pranjulcr7]

For the TODO(#17192) in hardware budget defaulting: #17262 adds GPUInfo.TotalGPUs to both shared discovery paths. It sums the selected SKU across nodes (8 + 4 → 12) and counts duplicate DCGM exporters on one node only once, while keeping GPUsPerNode as the maximum. Once available, the budget calculation here can use found.TotalGPUs instead of found.GPUsPerNode * found.NodesWithGPUs, retaining the consumer's cap and explicit-override behavior. #17192 remains open for that v1beta2 consumer and its admission regression.