Skip to content

fix(operator): separate inventory from selected-shape GPU capacity - #1

Closed
sttts wants to merge 4 commits into
Pranjulcr7:pranjul/fix-17192-aggregate-gpu-countfrom
sttts:sttts-dgh-17192-selected-shape-capacity
Closed

sttts wants to merge 4 commits into
Pranjulcr7:pranjul/fix-17192-aggregate-gpu-countfrom
sttts:sttts-dgh-17192-selected-shape-capacity

Conversation

@sttts

@sttts sttts commented Oct 28, 2026 •

Copy link
Copy Markdown

Incorporated upstream; closing as superseded. This selected-shape correction landed through ai-dynamo/dynamo#17308, including both DGDR consumers. The source PR #17262 is also superseded. The separate remaining asyncio change is tracked in #17640. Branches and review history are preserved.

Summary

This is an on-top fix for ai-dynamo/dynamo#17262 and the heterogeneous-node review finding in discussion_r4403133517.

The source PR correctly discovers total same-GPU-type inventory, but used that total directly as the executable profiling budget. A cluster with 8-GPU and 4-GPU nodes of the same type therefore produced a budget of 12 even though one deployment shape cannot schedule 12 GPUs across those unequal nodes.

This change separates the two concepts without adding a Kubernetes API field:

  • GPUInfo.TotalGPUs remains diagnostic same-type inventory.
  • GPUInfo.SelectedShapeGPUs counts only nodes matching the selected GPU type, GPUs per node, and VRAM per GPU.
  • v1beta1 automatic spec.hardware.totalGpus uses selected-shape capacity, capped at 32 as before.
  • Explicit totalGpus remains authoritative.
  • Unknown GPU types are grouped by normalized model identity rather than treating every unknown model as the same empty SKU.

Relates to ai-dynamo#17192.

API behavior

All hardware fields remain optional. No Kubernetes schema field is added.

spec:
  hardware:
    gpuSku: h100_sxm        # OPTIONAL; discovered when omitted
    numGpusPerNode: 8       # OPTIONAL; selected shape
    vramMb: 81920           # OPTIONAL; selected shape
    # totalGpus: omitted    # OPTIONAL; defaults to selected-shape capacity, capped at 32

With same-type 8-GPU and 4-GPU nodes, discovery records inventory 12 but defaults totalGpus to 8.

A caller may choose another shape by supplying its budget explicitly:

spec:
  hardware:
    gpuSku: h100_sxm
    numGpusPerNode: 4
    totalGpus: 4

A shape override without an explicit budget is rejected because discovery cannot derive executable capacity for that different shape:

spec:
  hardware:
    gpuSku: h100_sxm
    numGpusPerNode: 4
    # INVALID when discovery selected 8 GPUs per node: totalGpus is required
Discovered nodes Inventory Selected-shape capacity Automatic budget
8 + 4, same type/VRAM 12 8 8
8 + 8, same type/VRAM 16 16 16
8 + 8, same type/different VRAM 16 8 8
8 + 4, explicit totalGpus: 64 12 8 64

Generated artifacts

The v1beta1 CRD, Pydantic model, raw/full API references, and curated DGDR reference are regenerated or aligned with the Go API comments.

Validation

  • go test ./internal/gpu -count=1
  • discovery table covers both 8+4 and 4+8, proving the maximum selected shape is independent of node observation order
  • KUBEBUILDER_ASSETS=... go test ./internal/controller -count=1 (73.511s)
  • golangci-lint v2.15.0 run --timeout=5m ./... (0 issues)
  • make manifests generate-pydantic generate-api-docs
  • python3 docs/fern/scripts/gen_kubernetes_api.py --check
  • python3 docs/fern/scripts/docs_lint.py (0 errors; existing advisory warnings only)
  • git diff --check

Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
@sttts
sttts temporarily deployed to external_collaborator October 28, 2026 19:02 — with GitHub Actions Inactive
@sttts
sttts temporarily deployed to external_collaborator October 28, 2026 19:02 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

👋 Hi sttts! Thank you for contributing to Pranjulcr7/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
@sttts
sttts deployed to external_collaborator October 28, 2026 19:23 — with GitHub Actions Active
sttts added a commit to ai-dynamo/dynamo that referenced this pull request Oct 28, 2026
…ulcr7#1]

Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
sttts added a commit to ai-dynamo/dynamo that referenced this pull request Oct 28, 2026
…ulcr7#1]

Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
sttts added a commit to ai-dynamo/dynamo that referenced this pull request Oct 28, 2026
…ulcr7#1]

Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
@sttts
sttts deployed to external_collaborator October 28, 2026 22:00 — with GitHub Actions Active
@sttts
sttts marked this pull request as draft October 28, 2026 22:05
sttts added a commit to ai-dynamo/dynamo that referenced this pull request Oct 28, 2026
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
sttts added a commit to ai-dynamo/dynamo that referenced this pull request Oct 29, 2026
…ulcr7#1]

Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
sttts added a commit to ai-dynamo/dynamo that referenced this pull request Oct 29, 2026
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
sttts added a commit to ai-dynamo/dynamo that referenced this pull request Oct 29, 2026
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
@sttts
sttts deployed to external_collaborator October 29, 2026 12:41 — with GitHub Actions Active
sttts added a commit to ai-dynamo/dynamo that referenced this pull request Oct 29, 2026
…ulcr7#1]

Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
sttts added a commit to ai-dynamo/dynamo that referenced this pull request Oct 29, 2026
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
@Pranjulcr7 Pranjulcr7 closed this Oct 30, 2026

This branch was successfully deployed

1 active deployment
external_collaborator — 8c369f2a Deployed Oct 29, 2026 by sttts via ok-to-test #4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants