Repository navigation
feat(global-router): integrate Karmada and MCS support - #17270
Merged
sachalmalick merged 5 commits intoOct 25, 2026
Merged
sachalmalick merged 5 commits into
sachalmalick merged 5 commits into
Conversation
A deployment with `multiCluster` set is ready for Karmada and the SIG Multicluster (MCS) API: - An ingress rule from each peer VPC range on the EKS cluster security group, since MCS EndpointSlices list pod addresses. - The peer ranges are excluded from source NAT in the VPC CNI (`AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS` on `aws-node`), so traffic between the clusters keeps its pod address. - The MCS ServiceExport and ServiceImport CRDs, copied from Karmada 1.19.0. - In the hub only, the Karmada 1.19.0 control plane from the release Helm chart, with every image pinned. Its API is an internal NLB on TCP 5443 that only the member VPCs can reach, and its certificate covers `*.elb.<region>.amazonaws.com`. Ohio is the hub (`aws-ohio`) and West a member (`aws-west`); the names match the Global Router's site IDs. The peering stacks can also route the public subnets, so a bastion can reach the other VPC; existing private routes keep their logical IDs. Members join with `karmadactl register`, using a short-lived token from the hub, so the join is a documented step after deployment. The README also covers moving the running clusters off the hand-installed Karmada. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com> (cherry picked from commit 220c0ed)
Set `publicRouteTableIds` for both POC VPCs. In each VPC these are the only two route tables with an internet gateway route, which belong to publicSubnet1 and publicSubnet2. `deployment=both` now also adds a route to the peer VPC in each of them, so a bastion can reach the other VPC, including the Karmada API load balancer, without a proxy pod. The synthesized peering stacks gain OhioToWestPublicRoute1-2 and WestToOhioPublicRoute1-2 and change nothing else; the existing routes keep their logical IDs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com> (cherry picked from commit e0115f7)
MCS lists pod addresses, so a Global Router that takes a Relay address from discovery connects to the Relay pod directly. Without the peer ingress rule, Ohio reached a West Relay only through its NLB. The rule reaches pods only because the node launch templates set no security groups, so EKS gives the nodes the cluster security group that the rule is on. Assert both, so a launch template change cannot silently break direct pod traffic between the clusters. The README now says why the rule exists, and no longer says that no broad worker ingress rule is added. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com> (cherry picked from commit a1749b0)
- Install the Karmada control plane only with `-c karmada=true`. The running clusters still have a Karmada that was installed by hand, so every routine Ohio deployment would have tried, and failed, to install the chart. - Keep the chart's setup Job for 15 minutes and lower the Helm timeout to 10. Helm 4 waits for every resource to be ready at once and counts the Job, deleted 10 seconds after it finishes, as never ready, so the first install would hang until the kubectl Lambda timed out. - Keep the MCS CRDs and the Karmada release when they leave the stack or a deployment rolls back. The CRDs may predate the stack, deleting one deletes every ServiceExport and ServiceImport, and the chart's uninstall hook deletes the CRDs and its own namespace. - Turn on the controller's PropagateDeps feature gate, which the chart turns off. Without it, a propagated ServiceImport never brings its derived Service and EndpointSlices. - Admit ICMP "fragmentation needed" from the peer VPC, so Path MTU Discovery works over inter-Region peering. - Fix the join steps: run the register command that `token create` prints, with `--cluster-name`, and on Ohio with `--namespace karmada-agent`, because `karmada-system/karmada-kubeconfig` belongs to the control plane there. Cleanups: peer ranges must be valid VPC ranges that do not overlap the cluster's own VPC; the Karmada version must match the vendored CRDs; the component image tags that repeated the release chart's own pins are gone, and kubectl uses the chart's digest field; the CRD refresh loop no longer overwrites the files on a failed download, and crds/README.md keeps the upstream license notice; the tests now fail when a dependency, image pin, patch type or peering ID changes; and the comments, stack descriptions, MCS wording and a moved docs link are corrected. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com> (cherry picked from commit 5b05c6a)
The README said the ServiceImport's PropagationPolicy must set `propagateDeps: true`. Karmada's PropagationPolicy webhook already sets it for any policy that selects a ServiceImport, which is why the manual test on the POC clusters worked without it. What the import needs is the controller's PropagateDeps feature gate, which CDK turns on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com> (cherry picked from commit cf40408)
github-actions
Bot
added
feat
documentation
sachalmalick
marked this pull request as ready for review
October 25, 2026 07:35
sachalmalick
merged commit Oct 25, 2026
0997e8b
into
codex/global-router-reserved-gpu-compute
24 of 27 checks passed
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cherry-pick all five commits from #16963 onto the reserved GPU infrastructure branch, preserving source authors and cherry-pick references. Resolve conflicts with the operator-config cleanup so public examples retain synthetic identifiers and the CPU entry point continues to load explicitly selected local deployment settings.
This is the integration layer of a PR stack: GPU infrastructure → Karmada/MCS integration → existing four-region fleet setup and live findings. No merge to global-router-v2 or main is performed.
Part of #17269.
Validation