Repository navigation
feat(global-router): validate MCS across four existing regional clusters - #17271
Merged
sachalmalick merged 8 commits intoOct 25, 2026
Conversation
A deployment with `multiCluster` set is ready for Karmada and the SIG Multicluster (MCS) API: - An ingress rule from each peer VPC range on the EKS cluster security group, since MCS EndpointSlices list pod addresses. - The peer ranges are excluded from source NAT in the VPC CNI (`AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS` on `aws-node`), so traffic between the clusters keeps its pod address. - The MCS ServiceExport and ServiceImport CRDs, copied from Karmada 1.19.0. - In the hub only, the Karmada 1.19.0 control plane from the release Helm chart, with every image pinned. Its API is an internal NLB on TCP 5443 that only the member VPCs can reach, and its certificate covers `*.elb.<region>.amazonaws.com`. Ohio is the hub (`aws-ohio`) and West a member (`aws-west`); the names match the Global Router's site IDs. The peering stacks can also route the public subnets, so a bastion can reach the other VPC; existing private routes keep their logical IDs. Members join with `karmadactl register`, using a short-lived token from the hub, so the join is a documented step after deployment. The README also covers moving the running clusters off the hand-installed Karmada. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com> (cherry picked from commit 220c0ed)
Set `publicRouteTableIds` for both POC VPCs. In each VPC these are the only two route tables with an internet gateway route, which belong to publicSubnet1 and publicSubnet2. `deployment=both` now also adds a route to the peer VPC in each of them, so a bastion can reach the other VPC, including the Karmada API load balancer, without a proxy pod. The synthesized peering stacks gain OhioToWestPublicRoute1-2 and WestToOhioPublicRoute1-2 and change nothing else; the existing routes keep their logical IDs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com> (cherry picked from commit e0115f7)
MCS lists pod addresses, so a Global Router that takes a Relay address from discovery connects to the Relay pod directly. Without the peer ingress rule, Ohio reached a West Relay only through its NLB. The rule reaches pods only because the node launch templates set no security groups, so EKS gives the nodes the cluster security group that the rule is on. Assert both, so a launch template change cannot silently break direct pod traffic between the clusters. The README now says why the rule exists, and no longer says that no broad worker ingress rule is added. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com> (cherry picked from commit a1749b0)
- Install the Karmada control plane only with `-c karmada=true`. The running clusters still have a Karmada that was installed by hand, so every routine Ohio deployment would have tried, and failed, to install the chart. - Keep the chart's setup Job for 15 minutes and lower the Helm timeout to 10. Helm 4 waits for every resource to be ready at once and counts the Job, deleted 10 seconds after it finishes, as never ready, so the first install would hang until the kubectl Lambda timed out. - Keep the MCS CRDs and the Karmada release when they leave the stack or a deployment rolls back. The CRDs may predate the stack, deleting one deletes every ServiceExport and ServiceImport, and the chart's uninstall hook deletes the CRDs and its own namespace. - Turn on the controller's PropagateDeps feature gate, which the chart turns off. Without it, a propagated ServiceImport never brings its derived Service and EndpointSlices. - Admit ICMP "fragmentation needed" from the peer VPC, so Path MTU Discovery works over inter-Region peering. - Fix the join steps: run the register command that `token create` prints, with `--cluster-name`, and on Ohio with `--namespace karmada-agent`, because `karmada-system/karmada-kubeconfig` belongs to the control plane there. Cleanups: peer ranges must be valid VPC ranges that do not overlap the cluster's own VPC; the Karmada version must match the vendored CRDs; the component image tags that repeated the release chart's own pins are gone, and kubectl uses the chart's digest field; the CRD refresh loop no longer overwrites the files on a failed download, and crds/README.md keeps the upstream license notice; the tests now fail when a dependency, image pin, patch type or peering ID changes; and the comments, stack descriptions, MCS wording and a moved docs link are corrected. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com> (cherry picked from commit 5b05c6a)
The README said the ServiceImport's PropagationPolicy must set `propagateDeps: true`. Karmada's PropagationPolicy webhook already sets it for any policy that selects a ServiceImport, which is why the manual test on the POC clusters worked without it. What the import needs is the controller's PropagateDeps feature gate, which CDK turns on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com> (cherry picked from commit cf40408)
Signed-off-by: Sachal Malick <s@chal.ai>
github-actions
Bot
added
feat
documentation
…ation Signed-off-by: Sachal Malick <s@chal.ai>
Base automatically changed from
codex/global-router-mcs-integration
to
codex/global-router-reserved-gpu-compute
October 25, 2026 07:35
sachalmalick
marked this pull request as ready for review
October 25, 2026 07:36
Signed-off-by: Sachal Malick <s@chal.ai>
sachalmalick
merged commit Oct 25, 2026
4afd75b
into
codex/global-router-reserved-gpu-compute
23 of 26 checks passed
![dynamo-review-agent[bot]](https://avatars.githubusercontent.com/u/201626793?s=60&v=4){kind=small}
| .map((p) => p.cidr) as [string, ...string[]]; | ||
| new MultiClusterSupport(this, "Mcs", { | ||
| cluster, | ||
| config: { memberName: m.siteId, peerCidrs: peers }, |
There was a problem hiding this comment.
Deploying member mode to the existing Ohio/West clusters re-runs MultiClusterSupport for every peer, including their externally owned Ohio-West link. Those clusters' original stacks already invoked this construct with the opposite CIDR, so this stack attempts to create identical all-TCP and ICMP security-group rules; EC2 rejects them as duplicate permissions and the member deployment fails. Separate the complete peer list needed by the CNI patch from the ingress rules owned by this stack, excluding externally managed links from the latter.
🤖 AI Fix
Add a separately managed ingress-CIDR input to MultiClusterSupport and pass only peers whose links are not externally owned, while retaining all peer CIDRs for the CNI exclusion value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an independent CDK entry point that imports existing clusters and kubectl providers. Separate private peering, route and member setup stacks preserve the running Karmada hub, regional foundations and reserved GPU compute. Existing externally managed links keep their original owners.
Extend the private mesh with five connections and 48 additive private-CIDR routes; install MCS CRDs/CNI exclusions and peer security rules without deploying the CPU/GPU stacks. Document fresh-member requirements found live: the agent lease namespace must exist on the Karmada API, and ServiceExport discovery needs an applied propagated ServiceExport Work. Keep Services member-local, skip namespace auto-propagation for independently managed namespaces, and explicitly set the certificate-rotation Secret namespace when registering agents outside karmada-system.
Closes #17269.
Validation
The live run reused the existing hub. It did not migrate its hostPath-backed etcd to Helm, install clusterset.local DNS, wire the Global Router discovery consumer, or redeploy the original Ohio–West public-route extension.