feat(relay): upstream cluster links and tls:// qmux

[kixelated]

Quest: quest/m1/cluster-routing/transit.md (deleted here as finished).

Problem

moq.pro's edge tier and a drone uplink need one routing rule: a relay must never carry traffic between two of its upstream links (an edge between two cores, a drone between two CDN relays), while everything else still transits. A per-node switch cannot express a drone that transits for its mesh but not for its CDN uplink, so the mark belongs to the link. Edge links also want qmux over TLS on TCP, which only existed in plaintext (tcp://).

Approach

• moq-net: the origin's private peer: bool route flag becomes a private Link { Local, Peer, Upstream }. New origin::Producer::upstream() marks a handle's routes as learned upstream (a peer, for Source purposes). A Consumer taken from an upstream handle hides every upstream-learned route from both announced and request_broadcast, so the best route it offers skips a better upstream one. One mark on the link's handle covers both directions.
• moq-relay: a peer entry's object form gains upstream (config and connect_api). A dialed upstream peer gets self.origin.upstream() instead of .peer(). An accepted session is upstream when its grant sets upstream: true (with peer: true), so an edge treats a core that dials it as upstream too. Cluster::publisher/subscriber mark the link once in mounted(). A recheck that flips upstream closes the session, as a flipped peer already does. run_remote_once/run_remote_session now take the DialTarget instead of four loose args.
• moq-auth: Grant::upstream, refused by Grant::validate without peer (Error::UpstreamWithoutPeer). Mirrored in js/auth's zod schema and the cross-language vector.
• moq-tokio: tls://host:port/path dials qmux over TLS on TCP with the connect TLS settings (fingerprint pins and redirect ranking treat it like https). listen.tcp.tls = true (--listen-tcp-tls) serves the TCP listener over TLS with the listen certificate, reloading it. It asks for no client certificate: qmux's TLS accept keeps no peer identity for the auth server, so neither a tls.root nor pinned peers meant for QUIC apply there, and a peer on it authenticates with a token.

Tests:

• origin::tests::upstream_view_hides_upstream_routes (moq-net) and cluster::tests::upstream_grant_hides_other_upstreams.
• rs/moq-relay/tests/upstream_links.rs, all over tls://: a two-region tiered cluster (both edges pick the same core for every path, an edge broadcast reaches the far relay, losing a core moves only its paths), a core-to-core no-transit check through both a forward-dialed and a reverse-dialed edge, and a mesh/uplink/CDN line that transits both ways. Sensitivity: disabling the filter fails edges_never_transit_between_cores with core 1's route at core 2 via the reverse-dialed edge ([11, 21]).
• tcp::tests::tls_carries_qmux_and_refuses_an_untrusted_certificate, with QUIC CA roots and with pinned peers configured (the pinned case fails with CertificateRequired without the fix), and tcp::tests::tls_without_a_bind_is_refused.

Decisions (settled by the maintainer)

1. An accepted session is marked upstream by a grant claim upstream: true beside peer: true, mirroring how peer already classifies accepted sessions.
2. upstream without peer is refused by Grant::validate (fail loud), rather than implying peer.
3. The tls:// TCP listener asks for no client certificate, so it coexists with QUIC mTLS and peers on it present a token.

Impact

Public API (all additive):

• moq_net::origin::Producer::upstream(): new.
• moq_relay::cluster::Peer::{upstream, with_upstream} and the upstream field on peer objects (cluster.connect config, connect_api JSON): new.
• moq_relay::auth::Token::upstream: new public field.
• moq_auth::Grant::upstream (JSON upstream) and moq_auth::Error::UpstreamWithoutPeer: new. Grant is #[non_exhaustive]; Error already is.
• @moq/auth GrantSchema: optional upstream, refined to require peer.
• moq_tokio::tcp::Config::tls (--listen-tcp-tls, MOQ_LISTEN_TCP_TLS, listen.tcp.tls) and the tls:// client scheme: new. A TLS TCP session still reports Transport::Tcp to the auth server.

Wire: none. No message, field, or ALPN changes; tls:// carries the existing qmux-01 framing, negotiating qmux-01.<moq alpn> as the TLS ALPN, and no draft changes. Relays from before the mark treat every link as transit, so a cluster migrates a region at a time.

Cross-package sync: relay config and behavior are documented in doc/bin/relay/{cluster,config,auth}.md; the grant claim lives in moq-auth, mirrored in js/auth (there is no js/token counterpart for grants); doc/lib/rs/{moq-auth,moq-net}.md updated. No bindings or drafts touched.

Alternatives

• How an accepted session is marked upstream: (a) the acceptor matches the dialer against its own peer list by identity (needs a node identity on accept, which plain sessions lack today); (b) a relay-wide "every accepted peer is upstream" switch (wrong for a drone that accepts mesh peers); (c) a SETUP hint from the dialer (a wire change, and the dialer should not decide the acceptor's policy).
• upstream without peer: implying peer would be one less rule for auth-server authors, but hides a misconfigured grant.
• Consumer side naming: no separate public consumer method; the view comes from Producer::upstream().consume(). A Consumer::upstream()-style filter would read like "the upstream routes", the opposite of what it shows.
• mTLS on tls://: refusing tls.root alongside --listen-tcp-tls (as the closed feat(relay): add edge and core cluster roles over TLS qmux #4731 did) would forbid a relay with mTLS QUIC peers from also serving TLS edge links. Carrying the identity would mean doing the TLS accept ourselves instead of through qmux::tls::Server.

Follow-ups

• Peer identity (mTLS) on the tls:// listener, so cluster links on it can authenticate by certificate instead of a token.
• The down-only bit stays with Routes and announces; Selection's rendezvous hash is what will spread paths over equally ranked upstream links (today's path-vector FNV tie-break already keeps every edge on the same core).

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

[kixelated]

Background quest run complete; leaving this as a draft for the maintainer.

Open decisions (recommendation first):

1. Accepted peers are marked upstream by a grant claim upstream: true beside peer: true. Recommended. Alternatives: match the dialer against the acceptor's peer list by identity, a relay-wide "accepted peers are upstream" switch, or a SETUP hint (wire change).
2. upstream without peer is refused by Grant::validate. Recommended (fail loud); the alternative is to let upstream imply peer.
3. The tls:// TCP listener asks for no client certificate, so it coexists with QUIC mTLS and peers on it present a token. Recommended over refusing tls.root with --listen-tcp-tls (as feat(relay): add edge and core cluster roles over TLS qmux #4731 did).

Suggested follow-up: mTLS peer identity on the tls:// listener (own the TLS accept instead of qmux::tls::Server).

(Written by Claude Opus 5.5)

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 5c29d1a6-ddf7-471d-859e-10e854fe2670

📥 Commits

Reviewing files that changed from the base of the PR and between 95c2182 and 620ba4c.

📒 Files selected for processing (27)

• doc/bin/relay/auth.md
• doc/bin/relay/cluster.md
• doc/bin/relay/config.md
• doc/lib/rs/moq-auth.md
• doc/lib/rs/moq-net.md
• js/auth/src/contract.ts
• js/auth/src/interop.test.ts
• quest/m1/cluster-routing/README.md
• quest/m1/cluster-routing/routes.md
• quest/m1/cluster-routing/selection.md
• quest/m1/cluster-routing/transit.md
• rs/moq-auth/src/error.rs
• rs/moq-auth/src/grant.rs
• rs/moq-auth/src/request.rs
• rs/moq-net/src/model/origin.rs
• rs/moq-relay/src/auth.rs
• rs/moq-relay/src/cluster.rs
• rs/moq-relay/tests/upstream_links.rs
• rs/moq-tokio/Cargo.toml
• rs/moq-tokio/src/client.rs
• rs/moq-tokio/src/connect.rs
• rs/moq-tokio/src/connection.rs
• rs/moq-tokio/src/lib.rs
• rs/moq-tokio/src/listen.rs
• rs/moq-tokio/src/server.rs
• rs/moq-tokio/src/settings.rs
• rs/moq-tokio/src/tcp.rs

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[kixelated]

Automated review by review (OpenAI)

Reviewed commit: 9959e26

Direction: the per-link upstream horizon is well scoped, including reverse-dial grants, alternate-route selection and revalidation. The TLS listener needs one correction before it delivers its documented token-only behavior.

[P2] Clear the pinned-peer policy as well as CA roots for TCP TLS (rs/moq-tokio/src/server.rs:1020–1022). TcpTls::new clones Listen and clears root, but retains peers. Listen::client_auth prioritizes peers (tls.rs:1394–1400), and PeerVerifier requires a client certificate. An embedder sharing a Listen configured with a QUIC mesh fingerprint allowlist therefore rejects a token-only tls:// client during the handshake, before SETUP/auth can run, despite this listener explicitly promising no client-certificate requirement. Set listen.peers = None on this TCP-specific clone and add coverage with peers: Some(...), not only root.

Verification: static review of all 27 changed-file patches, surrounding TLS verifier code, and discussion. No tests or live handshakes were run; existing TLS coverage sets root but not peers.

[kixelated]

Automated review: 620ba4c7

Reviewed head 620ba4c78f51437fe9b3d440f7f2ebb420f95f91 (full review; no earlier Grok review on this PR).

The upstream horizon is small and well placed: Link replaces the peer bool, Horizon::upstream is set once in Consumer::from_producer, and scope/mount/excluding/local all carry it, so both announced and request_broadcast agree (the Unroutable assertion in upstream_view_hides_upstream_routes pins that). Fronts are keyed by (path, Horizon), so upstream readers never share a front with clients. The latest push fixes the pinned-peers gap on the tls:// listener (TcpTls::new now clears peers as well as root), and the round-trip test covers both. Head CI is still pending; the previous head 9959e267 was green.

No blocking code issues.

Should fix before merge

1. The PR description belongs to feat(net)!: bound peer-declared lengths and per-session requests #4820. The body describes the request caps work (lite 64 KiB message cap, session::Limits, MAX_REQUEST_ID, "Implements quest/m0/request-caps.md"), none of which is in this diff. The real public surface isn't listed anywhere: origin::Producer::upstream(), moq_auth::Grant::upstream with Error::UpstreamWithoutPeer (plus the js/auth GrantSchema field and refine), cluster::Peer::{upstream, with_upstream}, auth::Token::upstream, listen.tcp.tls / --listen-tcp-tls / MOQ_LISTEN_TCP_TLS, and the tls:// scheme. All of these are additive on #[non_exhaustive] types, so no ! is needed, but the squash message and changelog would be wrong as is.

Non-blocking

2. Two sessions to the same peer can disagree on the mark, and the unmarked one leaks transit (rs/moq-relay/src/cluster.rs:1212 mounted, :1749 run_remote_session; doc cluster.md:97). The mark is per session, not per peer. Say an edge E lists core C1 with upstream = true, and C1 also lists E (the docs allow this as a redundant second session, cluster.md:76). If E's auth grant for C1's inbound dial doesn't set upstream, C1's routes also arrive as Link::Peer. E's upstream view toward C2 then admits them, so E carries C1 to C2 traffic, which is exactly what the mark is meant to stop. Either say in "Upstream links" that every session with an upstream must carry the mark (the dialed entry and the grant for its inbound dial), or treat an accepted peer whose hop matches a dialed upstream target as upstream. A two-session case in upstream_links.rs would pin whichever you pick.

3. The reverse-dial mark has no shipped way to be granted, and the doc example conflicts with tls:// (cluster.md:97-101, :115). moq auth serve never sets peer (so it can't set upstream), so only a custom auth server or an embedded decider can mark an accepted upstream. The example "an auth server grants that to core certificates" also can't work for a core that dials over tls://, since that listener never asks for a client certificate (request.tls is None). Suggest saying the grant has to come from the token (or a custom decider), and pointing at the QUIC listener when certificate identity is wanted.

4. The auth contract can't tell tls:// from plaintext tcp:// (rs/moq-relay/src/auth.rs:727). Both map to Transport::Tcp, and the TLS accept reports no SNI or peer identity. Until now, transport: "tcp" effectively meant a loopback or private worker (auth.md:391, "native-only paths for gateways and stats publishers on the same host"). A policy keyed on that would now admit remote tls:// clients the same way. moq_auth::Transport is #[non_exhaustive], so a Tls variant (or an encrypted flag) is cheap now and awkward later. At minimum, note in auth.md that tcp now covers remote TLS sessions.

5. tls:// rebuilds the rustls client config on every dial (rs/moq-tokio/src/tcp.rs:192, client.rs:77). tcp_tls stores the tls::Connect settings and calls build() per attempt. On each reconnect that re-reads custom root PEMs and rebuilds the platform verifier, and with insecure = true it logs the "man-in-the-middle" warning every time. A bad connect.tls also only surfaces at the first tls:// dial instead of at init. When noq or websocket is compiled in, Client::new already builds one (client.rs:125); reusing it, or building lazily once in a OnceLock, keeps the "plaintext-only builds need no crypto provider" property.

6. Minor test gap. tiered_regions asserts that both edges pick the same core per path, but not the quest's stated outcome that the far relay serves one cross-region subscription per broadcast. A subscriber count on r for paths[0] after both edges read would close that.

Cross-PR

• feat(relay)!: delete the removed cluster mesh and linger flags #4818 (delete the removed cluster mesh and linger flags) also edits rs/moq-relay/src/cluster.rs and is already conflicting with main, so expect a rebase on whichever lands second.

Verdict: MERGE once the description is replaced and CI is green. Items 2 to 4 are worth a doc pass in this PR or a follow-up.

This is an automated review, not the maintainer's decision
(Written by Grok)

[kixelated]

Automated review by review (OpenAI)

Reviewed commit: 620ba4c

Follow-up to 9959e26, separating the main merge from the two-file TLS fix. The prior P2 is addressed: rs/moq-tokio/src/server.rs:1020–1023 clears both root and peers on the TCP-only clone, including the reload path. rs/moq-tokio/src/tcp.rs:365–417 now covers both QUIC client-auth policies and still checks rejection of an untrusted server certificate.

Direction: this now matches the documented token-authenticated TLS listener without changing QUIC peer authentication. No additional actionable defect found in this follow-up.

Verification: static inspection of the fix, surrounding configuration/reload and handshake test paths, and current discussion. I did not run tests or live TLS handshakes, or verify current CI; this is a COMMENT, not merge approval.

[kixelated]

Merge summary.

Decisions (settled by the maintainer, as recommended):

1. Accepted peers are marked upstream by the grant claim upstream: true beside peer: true.
2. upstream without peer is refused by Grant::validate.
3. The tls:// TCP listener asks for no client certificate; peers on it present a token.

Changes since the last review round:

• Merged main.
• Fixed the review finding: the TCP listener's TLS clone now clears pinned peers as well as root, so a QUIC fingerprint allowlist no longer rejects token-only tls:// clients with CertificateRequired. The test covers both policies and fails without the fix.

Impact: additive public API only (Producer::upstream, Grant::upstream, Error::UpstreamWithoutPeer, relay peer upstream, --listen-tcp-tls, the tls:// scheme); no wire or draft change.

Follow-up: mTLS peer identity on the tls:// listener.

(Written by Claude Opus 5.5)