Skip to content

fix(operator): harden DGDR run publication - #2

Merged
devivasudevan merged 2 commits into
devivasudevan:dgdr-run-publisher-clean2from
sttts:sttts-dgdr-run-publisher-review-fixes
Oct 23, 2026
Merged

devivasudevan merged 2 commits into
devivasudevan:dgdr-run-publisher-clean2from
sttts:sttts-dgdr-run-publisher-review-fixes

Conversation

@sttts

@sttts sttts commented Oct 23, 2026 •

Copy link
Copy Markdown

Summary

Harden the DGDR run publisher introduced by ai-dynamo#15894 around ownership, event handling, and terminal outcomes.

  • Create missing Candidates atomically. Existing Candidates are eligible for server-side apply only after a live controller-owner check, and the apply enforces both the observed UID and resourceVersion so concurrent mutation or delete/recreate cannot cause adoption.
  • Treat the deterministic Candidate name plus controller ownership as reconciliation identity. Candidate watches and lists remain namespace-scoped rather than depending on mutable labels; owned label drift is repaired and obsolete owned objects remain visible for deletion.
  • Restore normal Candidate update watches so externally cleared status metrics trigger reconciliation and repair.
  • Preserve the producer's terminal message and error, fail a nominally successful run when no Candidate was materialized, and write the bounded UTF-8 failure detail to /dev/termination-log for the future Run controller.

This changes reconciliation behavior only; it does not change the DGDR, Run, or Candidate API schemas.

Validation

  • go test -tags=clustertest ./internal/dgdrrunpublisher ./cmd/dgdr-run-publisher
  • KUBEBUILDER_ASSETS="$PWD/bin/k8s/1.30.0-darwin-arm64" go test ./internal/dgdrrunpublisher ./cmd/dgdr-run-publisher
  • go test -race -tags=clustertest ./internal/dgdrrunpublisher ./cmd/dgdr-run-publisher
  • go vet ./internal/dgdrrunpublisher ./cmd/dgdr-run-publisher
  • git diff --check

Envtest covers a foreign object already occupying the deterministic name, a foreign create that wins the name race, delete/recreate between the ownership read and SSA, repair and cleanup after label drift, and Candidate status repair caused by an update event after the controller has converged and become quiet.

Tracking

This is an on-top fix for review findings on ai-dynamo#15894. No matching issue is linked from that PR or was found in the repository issue search.

Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
@github-actions

Copy link
Copy Markdown

👋 Hi sttts! Thank you for contributing to devivasudevan/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@sttts sttts left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inline findings cover the remaining ownership atomicity, cache visibility, and update-watch test gaps in this on-top PR.

Comment thread deploy/operator/internal/dgdrrunpublisher/writer.go Outdated
Comment thread deploy/operator/internal/dgdrrunpublisher/reconciler.go
Comment thread deploy/operator/internal/dgdrrunpublisher/writer_envtest_test.go Outdated
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
@devivasudevan
devivasudevan merged commit 5c9eeb3 into devivasudevan:dgdr-run-publisher-clean2 Oct 23, 2026
24 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants