Skip to content

fix(global-router): preserve MCS import fields and add conformance suite - #17681

Open
sttts wants to merge 1 commit into
estroz/mcs-port-modulefrom
sttts-mcs-upstream-conformance
Open

sttts wants to merge 1 commit into
estroz/mcs-port-modulefrom
sttts-mcs-upstream-conformance

Conversation

@sttts

@sttts sttts commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Overview

MCS imports dropped the selected internalTrafficPolicy and trafficDistribution, and never published status.endpointSliceObjects. This preserves the selected policies through registry responses and into both imported Kubernetes resources, and reports slice-object presence after successful reconciliation. It also integrates the unmodified upstream MCS conformance suite and records baseline and post-fix results.

Stacked directly on #17615 (estroz/mcs-port-module), rebased onto its complete implementation at 1a07d2f1be90d8fd87c5b9ec0e99cf3253f7f730. The dated baseline below used the earlier 0fd4c96b4e source.

Key Changes

Core logic: 43 added nonblank, non-comment production lines, excluding generated code, tests, documentation and test infrastructure.

  • Policy projection and API: add protobuf response fields ServiceImport.internal_traffic_policy = 15 and traffic_distribution = 16; preserve winner values in full/lite registry views, the fake registry, ServiceImport and derived Service. Existing exports, storage representation and merge ownership already carry the values.
  • Importer: patch endpointSliceObjects to Present for a nonempty successfully applied slice set and Absent for an empty set. Include it in status comparison and patching; object presence is independent of endpoint readiness.
  • Conformance runner: pin the unmodified upstream suite to the existing MCS API v0.5.2 commit, reuse e2e deployment/routing helpers on two isolated kind clusters, save reports/diagnostics, preserve failing exit status and clean up. A manual workflow uploads reports even on failure.

API and Compatibility

Previously these policy fields and the EndpointSlice status were omitted from imported objects. For example, a source Service can explicitly select:

apiVersion: v1
kind: Service
metadata:
  name: inference
  namespace: llm
spec:
  selector:
    app: inference
  internalTrafficPolicy: Local
  trafficDistribution: PreferClose
  ports:
    - name: http
      port: 80
      targetPort: 8080

When its export wins the existing merge policy, the controller now writes the selected values to both the ServiceImport and derived Service. Example ServiceImport after slice reconciliation:

apiVersion: multicluster.x-k8s.io/v1beta1
kind: ServiceImport
metadata:
  name: inference
  namespace: llm
spec:
  type: ClusterSetIP
  ips: [10.251.0.20]
  ports:
    - name: http
      protocol: TCP
      port: 80
  internalTrafficPolicy: Local
  trafficDistribution: PreferClose
status:
  endpointSliceObjects: Present
  clusters:
    - cluster: source-a

The controller owns these imported fields. Existing export validation and oldest-active-export selection apply. Local retains Kubernetes node-local semantics, including exclusion of remote endpoints from node-local routing; traffic distribution remains a preference interpreted by Kubernetes.

Both protobuf fields are additive. Older clients ignore them; newer controllers reading older registries leave unspecified policy and empty distribution omitted. The derived Service then defaults its policy to Cluster. Removing a distribution from the selected export removes the previously owned value via server-side apply. Upgrade registry and controller together to propagate non-default policies; no stored-record, DynamoDB-table, CRD, RBAC or Helm schema change is needed.

Validation

Executed the complete, unmodified upstream suite locally on fresh IPv4 kind v0.33.0 / Kubernetes v1.35.8 clusters with the memory registry. The fix run tests signed commit 918d5228ab6dba417dc30e8c69c39df9eef615f4 and image global-discovery-mcs-conformance:e2e-0d629aebc7dc.

Upstream group Baseline passed / failed After fixes passed / failed
Required 12 / 3 15 / 0
Optional 3 / 10 4 / 9
Total 15 / 13 19 / 9

Both runs execute 28 of 28 specifications with 0 skipped or pending. The post-fix suite took 1023.343 seconds (17m03s). Its binary exits 1 and Make exits 2 because the optional DNS-dependent cases still fail.

Previously failing requirement Before After
status.endpointSliceObjects Empty Required check passes with Present/Absent
spec.internalTrafficPolicy nil Matches exported policy
spec.trafficDistribution nil Matches exported preference
Optional EndpointSlice labels/lifecycle Blocked by missing status field Passes

The headless StatefulSet DNS case also gets past its old status-field blocker and now reaches the DNS lookup. The nine remaining failures are five DNS cases and four DNS-dependent connectivity cases. The deployment has no clusterset.local DNS provider; ordinary local Service DNS passes. Those failures cannot independently diagnose endpoint reachability or per-port routing. Upstream classifies them as execution/assertion failures with conformance status unknown; a raw entry with failed: true and conformant: true is not a passing result.

The original baseline and post-fix analysis remain as dated Markdown summaries with environment and provenance. Generated YAML reports are no longer checked into the current tree; both summaries link immutable historical report snapshots, and future workflow runs upload generated reports as artifacts.

The real-registry/envtest regression TestImportTrafficPoliciesAndEndpointSliceStatus passes in 0.63 seconds with the race detector. It covers projection into both resources, changing Local to Cluster, clearing a distribution, omitted-policy defaulting, correcting only slice status on a registry event, zero slices and unready endpoints. Pinned Kubernetes 1.30 envtest enables the ServiceTrafficDistribution feature gate so the field is exercised instead of pruned. The full module unit/envtest suite passes at 95.7% internal statement coverage. It was rerun for the review changes, alongside lint and conformance-binary compilation. The configurable report URL was checked with a failing stub for default/override handling, preservation of exit 42 and stale-report cleanup. Generated-file checks, workflow lint and pre-commit checks pass on the clean rebased tree. The recorded 17-minute conformance run predates this rebase; it was not repeated for the test/report/metadata-only review changes.

Both dedicated kind clusters were removed; complete HTML/JSON/JUnit reports and diagnostics remain in ignored local artifacts. No suite filters, assertion changes, expected-failure conversions or DNS query rewrites are used. GitHub's manual workflow has not been run; the evidence is local. This run does not certify IPv6, DynamoDB, multi-region consistency, partition recovery, or full MCS conformance with DNS deployed.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill, including independent API/view, validation and adversarial reviews.

@github-actions github-actions Bot added test documentation Improvements or additions to documentation actions labels Oct 30, 2026
@sttts
sttts marked this pull request as ready for review October 30, 2026 12:23
@sttts
sttts requested review from a team as code owners October 30, 2026 12:23
@devin-ai-integration

devin-ai-integration Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 41446969249 (attempt 1) on 03b0d74f01

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 41446969249. Updated on every full-CI run of this PR.

@sttts sttts changed the title test(global-router): add upstream MCS conformance suite fix(global-router): preserve MCS import fields and add conformance suite Oct 30, 2026
@github-actions github-actions Bot added fix and removed test labels Oct 30, 2026

@estroz estroz left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for running the upstream suite, this is exactly the signal we were missing, and the two api fixes close your own threads on #17630/#17633 so i'll pull this into #17615 and re-slice rather than fix them twice. tests pass locally at head; the red ci job is operator integration, which the workflow touch triggers and is unrelated. a few small things inline, all around the harness. one ask: squash to a single conventional commit before merge, there's a fixup! in there and the titles aren't in the repo's type(scope): ... form.

name: MCS Conformance

on:
workflow_dispatch:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

workflow_dispatch only, so nobody will run it. your coverage ask on #17629 (a ci path for the kind suite) is the same problem; i'd like one scheduled workflow on the base branch that runs both the e2e phases and this suite nightly and uploads the reports, plus a label trigger for prs. fine as a follow-up once #17636 is in, but let's plan it as one thing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have escalated the combined CI trigger/scope decision before expanding this workflow. The full upstream run takes about 17 minutes plus image build/two-cluster setup; the custom e2e suite also builds GDS and uses three clusters. Nightly is not technically required, and affected-path PR CI or an explicit PR label can provide earlier feedback. No schedule/label change is included in this push. Leaving this thread open pending agreement on the combined CI plan.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ai-dynamo/dynamo-ops-codeowners could you weigh in on the trigger policy for a combined MCS e2e + upstream conformance workflow: affected-path PR runs, an explicit PR label, nightly, or a combination?

For sizing: the full upstream conformance suite took about 17 minutes locally, excluding image build and setup of its two kind clusters. The custom e2e suite additionally builds GDS and uses three clusters. These tests need no GPUs. The upstream suite currently passes all 15 required checks but still reports nine optional DNS-dependent failures because the harness does not deploy clusterset.local DNS; those results and artifacts remain visible.

Eric suggested one nightly workflow plus a PR-label trigger, potentially as a follow-up after #17636. Before choosing the cadence, we'd like Ops guidance on the repository's CI conventions and runner cost. Keeping this thread open for that decision.

@estroz
estroz force-pushed the estroz/mcs-port-module branch from 0fd4c96 to 1a07d2f Compare October 30, 2026 20:09
@estroz
estroz requested review from a team as code owners October 30, 2026 20:09
Preserve traffic policies through registry projections and imported resources,
and publish EndpointSlice object presence after successful reconciliation.

Integrate the unmodified upstream MCS conformance suite on isolated kind
clusters, with artifact reporting and dated baseline/post-fix summaries.
Address review feedback on projection coverage, report URLs and artifacts.

Part of #17601; refs DGH-2319.

Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
@sttts
sttts force-pushed the sttts-mcs-upstream-conformance branch from 5ccd7b6 to 03b0d74 Compare November 2, 2026 12:42

sttts commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current #17615 head 1a07d2f1be and squashed to one conventional commit: 03b0d74f01. GitHub verifies its signature; the PR is conflict-free.

Addressed the projection-test, report-URL and generated-report feedback, confirmed EndpointSlice status semantics, and replied to/resolved those four threads. Validation: race-enabled module tests (95.7% coverage), lint, conformance-binary build, generated-file checks, workflow lint and pre-commit all pass. The 17-minute upstream run remains the dated evidence; it was not rerun for these test/report/build-metadata changes.

The combined e2e/conformance CI trigger decision is escalated and remains open in its review thread; this push adds no nightly or label trigger.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions documentation Improvements or additions to documentation fix size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants