Skip to content

feat(relay): add edge and core cluster roles over TLS qmux - #4731

Closed
kixelated wants to merge 2 commits into
quest/m1/cluster-routing/READMEfrom
quest/m1/cluster-routing/roles
Closed

kixelated wants to merge 2 commits into
quest/m1/cluster-routing/READMEfrom
quest/m1/cluster-routing/roles

Conversation

@kixelated

@kixelated kixelated commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Edges must pull each path through one regional core without becoming transit between cores. The existing mesh has no explicit admission or dialing roles, and qmux TCP is plaintext.

Approach

Add explicit edge/core configuration and peer metadata, validate link roles, require peer grants on cores, and publish only local ingress on edge cluster dials. Add encrypted tls:// qmux TCP with the existing certificate/trust configuration and request-target forwarding. Preserve legacy mesh behavior when the own role is unset.

Impact

  • Add cluster.role, --cluster-role, and optional peer-object role metadata (cluster::Role, Peer::role, Peer::with_role).
  • Add listen.tcp.tls, --listen-tcp-tls, and tls://host:port dialing. TLS uses existing qmux framing and MoQ ALPNs; no MoQ wire change.
  • Core media admission requires grant.peer; edge admission refuses inbound peer sessions. Explicit edges dial declared cores over TLS only. Role-aware LAN discovery is refused.
  • TLS TCP reports the existing TCP auth transport. It refuses mTLS roots because qmux does not expose the verified peer certificate to relay authorization.

Validation

  • Full selected-package just rs check-test: 2,120 passed, seven existing skips; format, docs, stats docs, dependency hygiene, and sort checks passed.
  • Exact-head focused nextest: 15 passed, including TLS server assertions, startup refusal, SETUP request-target forwarding, role admission, edge split horizon, two-region deduplication, and mocked-time path failover. The filtered check wrapper passed test-only -E to cargo doc; rerunning package-only just rs _lint passed.
  • Registered split-horizon benchmark completed all six release cells. Local ingress: 19.8/22.9/177.1 microseconds at 100 paths + 2 cores / 1,000 + 2 / 1,000 + 20. Filtered core ingress: 9.2/14.4/28.3 microseconds. Short ten-sample runs were noisy; these are descriptive measurements, not an optimization claim.
  • TCP-only feature lint (--no-default-features --features tcp, warnings denied) and final package-only lint passed. Existing CI package recipes discover the tests; nightly runs all registered moq-net benchmark smoke checks. MoQ framing and protocol messages are unchanged, so the cross-language wire matrix was not run.

Alternatives

Keep the legacy mesh by leaving the role unset. Region inventories and lower-name dial direction stay in the peer-list generator, as the quest specifies.

Follow-ups

The quest remains present pending one maintainer decision: it says a one-edge region's edge is its core, but also says cores refuse user sessions. This draft uses exclusive roles. Recommend separate edge/core instances even on one host; alternatives are an explicit combined role or an edge allowed to act as its region's core. Do not merge before that topology is settled.

(written by GPT-6)

kixelated and others added 2 commits October 2, 2026 08:26
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Implemented exclusive edge/core configuration, admission, edge split horizon, and encrypted qmux TCP. The draft remains blocked on the quest's singleton-region contradiction: an edge must serve users, while a core must refuse them. Recommend separate edge/core instances; no combined role was silently introduced. The quest remains present.

Validation: 2,120 package tests passed (seven existing skips), 15 exact-head focused tests passed, final format/docs/dependency lint passed, and TCP-only feature lint passed with warnings denied. Six release benchmark cells over path count and core count completed; short samples were noisy, and the PR records the estimates. A test-filter flag was incompatible with the combined lint wrapper; the independent final lint passed. No MoQ wire messages or versions changed. Cross-language wire interop was not run.

TLS TCP cannot expose the verified client certificate through qmux's accept API, so mTLS roots are refused instead of losing the identity. JWT peer grants are supported. Draft head: aceb5b4.

(written by GPT-6)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Closing this per a maintainer decision.

✅ Close #4731 and carry the tls:// qmux piece into the Upstream links implementation. Rejected: rework in place; park.

Main replaced edge/core roles with a per-link upstream attribute (#4683, #4694): a drone must transit for its mesh but never for its CDN uplink, which no per-node role can express. See quest/m1/cluster-routing/transit.md.

The tls:// qmux work here (moq-tokio listener, dialer, request-target forwarding) is still wanted. #4752 adds a line to transit.md's Plan pointing at this PR as prior art; its commits stay reachable from this PR after the quest/m1/cluster-routing/roles branch is deleted.

(Written by Claude Opus 5.5)

@kixelated kixelated closed this Oct 3, 2026
@kixelated
kixelated deleted the quest/m1/cluster-routing/roles branch October 3, 2026 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant