[estroz]

Overview

Adds a standard implementation of the Kubernetes Multi-Cluster Services API (KEP-1645) for the Global Router as a new Go module, deploy/global-discovery-mcs, with its Helm chart, CI wiring and docs-site pages. It replaces Karmada under the Global Discovery Service's --pool-discovery mcs mode: the service already selects EndpointSlices by multicluster.kubernetes.io/service-name and multicluster.kubernetes.io/source-cluster and reads each endpoint's targetRef name, and the controller here writes exactly those, so no Rust change is needed. With #17604 the service also reads the topology.kubernetes.io/region label the controller stamps on imported slices.

Pool discovery stays with the Global Discovery Service's HTTP contract; this module only moves Services and endpoints between clusters.

Three binaries ship in one image (global-discovery-mcs):

• mcs-registry: stateless registry server. Keeps clusters (with leases), ServiceExports and endpoint sets in memory or one DynamoDB table (Global Table across regions), merges the exports of a Service from every cluster into one ServiceImport using the KEP-1645 conflict rules (oldest export wins), and serves the dynamo.mcs.v1 API over gRPC (:9090) and REST through grpc-gateway (:8080), including watches with opaque revisions.
• mcs-controller: runs in every member cluster. Registers the cluster with a lease, exports Services labelled dynamo.nvidia.com/mcs-export: "true" (or with a ServiceExport) with their endpoints, and materialises ServiceImports, derived-<hash> Services and EndpointSlices with the KEP labels, the source cluster's region and per-endpoint targetRef.
• mcs-admin: creates the DynamoDB table, describes a store, deletes a cluster.

Key Changes

• New Go module deploy/global-discovery-mcs (github.com/ai-dynamo/dynamo/deploy/global-discovery-mcs, Go 1.27.3, k8s.io v0.36.3, controller-runtime v0.24.1, mcs-api v0.5.2, no replace directives).
  • cmd/mcs-registry: registry server with memory and DynamoDB stores, ServiceImport merge with the KEP-1645 conflict rules, watches with revisions, leases evaluated at read time, gRPC plus grpc-gateway REST plus OpenAPI.
  • cmd/mcs-controller: cluster registration and lease, ServiceExport reconciler, label-driven auto-export, importer writing ServiceImports, derived-<hash> Services and EndpointSlices.
  • cmd/mcs-admin: create-table, describe, delete-cluster; takes the store flags and MCS_ env fallbacks.
  • api/proto/dynamo/mcs/v1 contract (cluster, common, mcs, server) and generated Go, gateway and OpenAPI under api/gen.
  • pkg/client (failover Go client with watch loop and metrics) and pkg/fakeregistry (in-memory server for consumer tests).
  • Imported EndpointSlices carry the KEP-1645 labels, topology.kubernetes.io/region of the source cluster and a per-endpoint targetRef naming the exported pod, which is what the Global Discovery Service's mcs backend reads.
  • Labels dynamo.nvidia.com/mcs-export, mcs-managed-by, mcs-clusterset; finalizer dynamo.nvidia.com/mcs-service-export; leader Lease and EndpointSlice managed-by mcs-controller.dynamo.nvidia.com; metrics dynamo_mcs_registry_*, dynamo_mcs_controller_*, dynamo_mcs_client_*. KEP-1645 and ClusterProperty keys unchanged.
  • Makefile (build lint test envtest test-integration manifests proto proto-check sync-crds tidy-check check docker-build), Dockerfile with linter, tester, builder and distroless runtime stages (USER 65532, entrypoint /mcs-registry, image global-discovery-mcs), .golangci.yml from deploy/operator plus goheader.
  • README.md, AGENTS.md with CLAUDE.md, docs/ (api, mcs-controller, consistency, consistency-dynamodb, storage, iam), pkg/client/README.md. The module .gitignore re-includes internal/server/, which the root server/ pattern would otherwise drop.
• New Helm chart deploy/helm/charts/global-discovery-mcs: registry.enabled and controller.enabled, one image block, ClusterRole rendered from the generated files/controller-role.yaml, leader-election Role, CRDs under crds.install with helm.sh/resource-policy: keep (set crds.install=false where Karmada or another implementation already owns them), values.schema.json, helm-docs README, 57 helm-unittest cases. Resource names are fixed (mcs-registry, mcs-controller), so one release per namespace.
• Docs site: user guide kubernetes/kv-aware-routing/multi-cluster-services.md, reference reference/components/global-discovery-mcs-configuration.md, design page developer-guide/knowledge-base/modular-components/router/multi-cluster-services-registry.md, nav entries, one cross-link from the pool relay page.
• CI: filter global_discovery_mcs in .github/filters.yaml and FILTERS.md; pre-merge jobs for the module (check-deploy-component plus make proto-check check and make test-integration) and the chart (helm lint, unittest); pr.yaml and post-merge-ci.yml image builds via build-deploy-component for amd64 and arm64; check-deploy-component and build-deploy-component accept the new component.
• Code ownership: module, chart, docs pages and the shared Dockerfile claimed under global-router in .github/codeowners/areas.yaml; CODEOWNERS regenerated.
• Not included: compliance and SBOM image stages, and a CI job for the kind e2e (the e2e runs locally with make e2e).

Validation

From a fresh checkout, in deploy/global-discovery-mcs:

• make lint: 0 issues. make test (unit plus envtest, -race, coverage gate 85 on ./internal/...): 95.7 percent. make test-integration against amazon/dynamodb-local: all pass, none skipped. make proto-check and make check (tidy, manifests, proto, CRD copies): no drift.
• docker buildx build --target linter and --target tester exit 0; the runtime image builds and --help works for all three binaries.
• Chart: make -C deploy/helm/charts/global-discovery-mcs lint test under Helm 3.17.3 and 3.21: lint strict clean, 57 unittest cases pass; server-side dry run of every object against a kube-apiserver 1.30 with CRDs on and off; a real install, upgrade with crds.install=false and uninstall keeps the CRDs. The controller ran with the chart's rendered args as the chart's ServiceAccount (RBAC enforced), took the leader Lease, registered the cluster and exported a labelled Service with Valid=True, Ready=True, Conflict=False.
• CI metadata: build_codeowners.py --strict 7514/7514 owned, emit_codeowners.py no diff, test-filters.js 87/87, actionlint adds no finding beyond the existing runner-label warning.
• Docs: docs_lint.py 0 errors, fern check 0 errors, fern docs broken-links --strict passes.
• pre-commit on all changed files passes.

Not run here: the GitHub workflow jobs themselves (CI on this branch needs a maintainer's /ok to test <sha>) and the PowerShell copyright check (pwsh unavailable; headers verified with a script).
End-to-end (make -C deploy/global-discovery-mcs e2e E2E_PHASES="1 2 3", three kind clusters with routed pod subnets, the registry in the hub, a controller per cluster, and the Global Discovery Service built from container/Dockerfile.global-discovery-service running in the hub with --pool-discovery mcs): 99 passed, 0 failed, 1 known gap. Phase 1 (23): registration, label-driven export, imports with derived Services, EndpointSlices with KEP labels, region and targetRef pod names in every cluster, a pod in each importing cluster reaching the exporter through its derived ClusterIP. Phase 2 (57): endpoint updates, a v1alpha1 ServiceExport, conflicting exports (oldest wins, Conflict=True on the loser), lease expiry, export deletion. Phase 3 (19): GET /pools lists the pool from cluster a with its relay identity, GET /routers lists the hub's two router pods, scaling the relay to zero removes the pool, deleting the export removes it. The known gap: the service reports the pool region as the cluster id because the branch predates #17604, which adds the label read; E2E_STRICT=1 turns the gap into a failure.

Review slices

This PR is kept open as the complete, validated source of truth. For review it is split into eight independent PRs against the same base branch; each carries one part and links back here for the pieces it depends on. Merge conflicts between them are expected only in areas.yaml and CODEOWNERS.

• feat(global-router): scaffold the global-discovery-mcs module and its CI #17629: module scaffolding, Makefile, Dockerfile, README, CI wiring and code ownership
• feat(global-router): add the Multi-Cluster Services registry API contract #17630: the dynamo.mcs.v1 proto contract, generated code and the label and condition constants
• feat(global-router): add mcs-registry, its stores and mcs-admin #17631: mcs-registry (view, merge rules, memory and DynamoDB stores, handlers, server) and mcs-admin
• feat(global-router): add the MCS registry Go client and fake registry #17632: pkg/client and pkg/fakeregistry
• feat(global-router): add mcs-controller #17633: mcs-controller (registration and lease, export reconciler, auto-export, importer, CRDs, RBAC)
• feat(global-router): add the global-discovery-mcs Helm chart #17634: the Helm chart
• docs(global-router): document Multi-Cluster Services for the Global Router #17635: the docs-site pages
• test(global-router): add the kind end-to-end test for global-discovery-mcs #17636: the kind end-to-end test

Issues and Ongoing Work

• Closes Ship a standard Multi-Cluster Services implementation for the Global Router #17601
• Targets the feat(global-router): add the Global Discovery HTTP service #17371 branch because the docs and e2e run that service in --pool-discovery mcs. Companion gap PRs: feat(global-router): take a pool's region from its EndpointSlice label in the mcs backend #17604 (region label from the slice), fix(global-router): install mcs-api CRDs that serve v1alpha1 and v1beta1 in the CDK example #17605 (mcs-api v0.5.2 CRDs in the CDK), fix(operator): accept any served ServiceExport version for multi-cluster export #17606 (operator accepts either served ServiceExport version), docs(dep): add the standard Multi-Cluster Services implementation to DEP 103 enhancements#104 (DEP 103 wording).
• Known follow-ups (module README): the exporter does not yet send targetRef.namespace; the image build passes no version ldflags so binaries report dev; a memory-store registry restart makes importers garbage-collect imports until controllers re-register (deferring GC on an empty snapshot from a new view epoch would avoid the gap).

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a18e73-c576-71e8-a5c5-5eb77efa6017.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

❌ Dynamo PR CI failed — run 41121078903 (attempt 1) on 1a07d2f1be

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Framework	Build
vLLM	❌ 1
SGLang	❌ 1

Other	Jobs
dynamo-runtime	❌ 1
frontend	❌ 1
Global Discovery MCS	❌ 1
Operator	❌ 1
planner	❌ 1
Power Agent	❌ 1
triton-runtime	❌ 1

Failure details

All 9 failed jobs are image builds that fail at the push step with the same infrastructure error: the CI ECR repository ai-dynamo/dynamo has hit its 100,000-image limit, so every push is denied. No code or test failure appears in the logs; rerunning failed jobs should be enough once registry space is freed (a rerun before cleanup will likely fail the same way).

❌ Power Agent: ECR push denied: repository image limit (100000) reached

Job: Power Agent · Failed step: Build and push power agent · Logs: gh run view --job 124287457639 -R ai-dynamo/dynamo --log-failed

ERROR: failed to build: failed to solve: failed to push ***.dkr.ecr.us-west-2.amazonaws.com/ai-dynamo/dynamo: denied: The repository with name 'ai-dynamo/dynamo' in registry with id '***' already has the maximum allowed number of images which is '100000'
##[error]Process completed with exit code 1.

The image build completed but pushing to ECR was denied because the repository is at its maximum image count (infrastructure, not caused by the PR). Rerun failed jobs after the registry is cleaned up.

❌ Global Discovery MCS: ECR push denied: repository image limit (100000) reached

Job: Global Discovery MCS · Failed step: Build and push the MCS registry image · Logs: gh run view --job 124287457672 -R ai-dynamo/dynamo --log-failed

ERROR: failed to build: failed to solve: failed to push ***.dkr.ecr.us-west-2.amazonaws.com/ai-dynamo/dynamo: denied: The repository with name 'ai-dynamo/dynamo' in registry with id '***' already has the maximum allowed number of images which is '100000'
##[error]Process completed with exit code 1.

The image build completed but pushing to ECR was denied because the repository is at its maximum image count (infrastructure, not caused by the PR). Rerun failed jobs after the registry is cleaned up.

❌ Operator: ECR push denied: repository image limit (100000) reached

Job: Operator · Failed step: Build and push operator · Logs: gh run view --job 124287457745 -R ai-dynamo/dynamo --log-failed

ERROR: failed to build: failed to solve: failed to push ***.dkr.ecr.us-west-2.amazonaws.com/ai-dynamo/dynamo: denied: The repository with name 'ai-dynamo/dynamo' in registry with id '***' already has the maximum allowed number of images which is '100000'
##[error]Process completed with exit code 1.

The image build completed but pushing to ECR was denied because the repository is at its maximum image count (infrastructure, not caused by the PR). Rerun failed jobs after the registry is cleaned up.

❌ vllm-runtime / Build multi-arch cuda13.0: ECR push denied: repository image limit (100000) reached

Job: vllm-runtime / Build multi-arch cuda13.0 · Failed step: Build and Push Image · Logs: gh run view --job 124287457995 -R ai-dynamo/dynamo --log-failed

ERROR: failed to build: failed to solve: failed to push ***.dkr.ecr.us-west-2.amazonaws.com/ai-dynamo/dynamo: denied: The repository with name 'ai-dynamo/dynamo' in registry with id '***' already has the maximum allowed number of images which is '100000'
##[error]Process completed with exit code 1.

The image build completed but pushing to ECR was denied because the repository is at its maximum image count (infrastructure, not caused by the PR). Rerun failed jobs after the registry is cleaned up.

❌ sglang-runtime / Build multi-arch cuda13.0: ECR push denied: repository image limit (100000) reached

Job: sglang-runtime / Build multi-arch cuda13.0 · Failed step: Build and Push Image · Logs: gh run view --job 124287458149 -R ai-dynamo/dynamo --log-failed

ERROR: failed to build: failed to solve: failed to push ***.dkr.ecr.us-west-2.amazonaws.com/ai-dynamo/dynamo: denied: The repository with name 'ai-dynamo/dynamo' in registry with id '***' already has the maximum allowed number of images which is '100000'
##[error]Process completed with exit code 1.

The image build completed but pushing to ECR was denied because the repository is at its maximum image count (infrastructure, not caused by the PR). Rerun failed jobs after the registry is cleaned up.

❌ planner / Build multi-arch cpu: ECR push denied: repository image limit (100000) reached

Job: planner / Build multi-arch cpu · Failed step: Build and Push Image · Logs: gh run view --job 124287458163 -R ai-dynamo/dynamo --log-failed

ERROR: failed to build: failed to solve: failed to push ***.dkr.ecr.us-west-2.amazonaws.com/ai-dynamo/dynamo: denied: The repository with name 'ai-dynamo/dynamo' in registry with id '***' already has the maximum allowed number of images which is '100000'
##[error]Process completed with exit code 1.

The image build completed but pushing to ECR was denied because the repository is at its maximum image count (infrastructure, not caused by the PR). Rerun failed jobs after the registry is cleaned up.

❌ triton-runtime / Build multi-arch cpu: ECR push denied: repository image limit (100000) reached

Job: triton-runtime / Build multi-arch cpu · Failed step: Build and Push Image · Logs: gh run view --job 124287458166 -R ai-dynamo/dynamo --log-failed

ERROR: failed to build: failed to solve: failed to push ***.dkr.ecr.us-west-2.amazonaws.com/ai-dynamo/dynamo: denied: The repository with name 'ai-dynamo/dynamo' in registry with id '***' already has the maximum allowed number of images which is '100000'
##[error]Process completed with exit code 1.

The image build completed but pushing to ECR was denied because the repository is at its maximum image count (infrastructure, not caused by the PR). Rerun failed jobs after the registry is cleaned up.

❌ frontend / Build multi-arch cpu: ECR push denied: repository image limit (100000) reached

Job: frontend / Build multi-arch cpu · Failed step: Build and Push Image · Logs: gh run view --job 124287458320 -R ai-dynamo/dynamo --log-failed

denied: The repository with name 'ai-dynamo/dynamo' in registry with id '***' already has the maximum allowed number of images which is '100000'
ERROR: failed to build: failed commit on ref "index-sha256:8c0e6eeb...": unexpected status from PUT request to https://***.dkr.ecr.us-west-2.amazonaws.com/v2/ai-dynamo/dynamo/manifests/1a07d2f1be90d8fd87c5b9ec0e99cf3253f7f730-dynamo-frontend: 403 Forbidden
##[error]Process completed with exit code 1.

Layers built, but the manifest PUT to ECR returned 403 because the repository is at its maximum image count (infrastructure, not caused by the PR). Rerun failed jobs after the registry is cleaned up.

❌ dynamo-runtime / image / Build multi-arch cuda13.0: ECR push denied: repository image limit (100000) reached

Job: dynamo-runtime / image / Build multi-arch cuda13.0 · Failed step: Build and Push Image · Logs: gh run view --job 124287458428 -R ai-dynamo/dynamo --log-failed

ERROR: failed to build: failed to solve: failed to push ***.dkr.ecr.us-west-2.amazonaws.com/ai-dynamo/dynamo: denied: The repository with name 'ai-dynamo/dynamo' in registry with id '***' already has the maximum allowed number of images which is '100000'
##[error]Process completed with exit code 1.

The image build completed but pushing to ECR was denied because the repository is at its maximum image count (infrastructure, not caused by the PR). Rerun failed jobs after the registry is cleaned up.

For agents

{"pr": 17615, "run_id": 41121078903, "run_attempt": 1, "head_sha": "1a07d2f1be90d8fd87c5b9ec0e99cf3253f7f730", "failures": [{"job": "Power Agent", "job_id": 124287457639, "failed_step": "Build and push power agent", "signature": "ECR push denied: repository image limit (100000) reached", "tests": [], "log_cmd": "gh run view --job 124287457639 -R ai-dynamo/dynamo --log-failed"}, {"job": "Global Discovery MCS", "job_id": 124287457672, "failed_step": "Build and push the MCS registry image", "signature": "ECR push denied: repository image limit (100000) reached", "tests": [], "log_cmd": "gh run view --job 124287457672 -R ai-dynamo/dynamo --log-failed"}, {"job": "Operator", "job_id": 124287457745, "failed_step": "Build and push operator", "signature": "ECR push denied: repository image limit (100000) reached", "tests": [], "log_cmd": "gh run view --job 124287457745 -R ai-dynamo/dynamo --log-failed"}, {"job": "vllm-runtime / Build multi-arch cuda13.0", "job_id": 124287457995, "failed_step": "Build and Push Image", "signature": "ECR push denied: repository image limit (100000) reached", "tests": [], "log_cmd": "gh run view --job 124287457995 -R ai-dynamo/dynamo --log-failed"}, {"job": "sglang-runtime / Build multi-arch cuda13.0", "job_id": 124287458149, "failed_step": "Build and Push Image", "signature": "ECR push denied: repository image limit (100000) reached", "tests": [], "log_cmd": "gh run view --job 124287458149 -R ai-dynamo/dynamo --log-failed"}, {"job": "planner / Build multi-arch cpu", "job_id": 124287458163, "failed_step": "Build and Push Image", "signature": "ECR push denied: repository image limit (100000) reached", "tests": [], "log_cmd": "gh run view --job 124287458163 -R ai-dynamo/dynamo --log-failed"}, {"job": "triton-runtime / Build multi-arch cpu", "job_id": 124287458166, "failed_step": "Build and Push Image", "signature": "ECR push denied: repository image limit (100000) reached", "tests": [], "log_cmd": "gh run view --job 124287458166 -R ai-dynamo/dynamo --log-failed"}, {"job": "frontend / Build multi-arch cpu", "job_id": 124287458320, "failed_step": "Build and Push Image", "signature": "ECR push denied: repository image limit (100000) reached", "tests": [], "log_cmd": "gh run view --job 124287458320 -R ai-dynamo/dynamo --log-failed"}, {"job": "dynamo-runtime / image / Build multi-arch cuda13.0", "job_id": 124287458428, "failed_step": "Build and Push Image", "signature": "ECR push denied: repository image limit (100000) reached", "tests": [], "log_cmd": "gh run view --job 124287458428 -R ai-dynamo/dynamo --log-failed"}]}

_{Posted automatically by Devin for run 41121078903. Updated on every full-CI run of this PR.}