From 7dcb0c46e694cbb4485fe8fa53ae437fd0726377 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 1 Oct 2026 17:18:16 -0700 Subject: [PATCH 1/4] quest(m1): cluster routing as generic per-node routes and path-less announces Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 6 +- quest/m1/cluster-routing/README.md | 212 +++++++++++----------- quest/m1/cluster-routing/inter-cluster.md | 36 ---- quest/m1/cluster-routing/link-quality.md | 34 ++++ quest/m1/cluster-routing/path-hunting.md | 51 ------ quest/m1/cluster-routing/route-trust.md | 39 ++++ quest/m1/cluster-routing/routes.md | 88 +++++++++ quest/m1/cluster-routing/selection.md | 20 +- quest/m1/cluster-routing/sim.md | 19 ++ quest/m1/cluster-routing/transit.md | 101 ++++++----- quest/m1/cross-relay-bursts.md | 2 +- quest/m1/hop-removal.md | 8 +- quest/m1/ietf-cluster-off.md | 2 +- quest/m2/ietf-cluster-peers.md | 5 +- quest/m3/routing-cost-domains.md | 13 +- 15 files changed, 371 insertions(+), 265 deletions(-) delete mode 100644 quest/m1/cluster-routing/inter-cluster.md create mode 100644 quest/m1/cluster-routing/link-quality.md delete mode 100644 quest/m1/cluster-routing/path-hunting.md create mode 100644 quest/m1/cluster-routing/route-trust.md create mode 100644 quest/m1/cluster-routing/routes.md create mode 100644 quest/m1/cluster-routing/sim.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 80f417690a..e50d2d483d 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -23,7 +23,9 @@ QUIC studies there on that rule. ## Required - [moq-transport peers are plain clients](/quest/m1/ietf-cluster-off.md) - no moq-transport session negotiates the cluster extension, so none can splice a pool's members on a Hop ID label -- [Cluster routing](/quest/m1/cluster-routing/README.md) - edge and core tiers carry each broadcast into a region once over path vector, with the backbone hidden from end users +- [Epoch primitive](/quest/m1/epoch.md) - one `Epoch` type in moq-net and @moq/net, carried as a trailing `@` path segment, shared by e2ee and broadcast epochs +- [Broadcast epochs](/quest/m1/broadcast-epoch/README.md) - each publish of a name gets a fresh `@` epoch, viewers follow the newest live one at once, and bare names still resolve on every version +- [Cluster routing](/quest/m1/cluster-routing/README.md) - any node routes toward a broadcast's origin over CDN and P2P links alike, with per-origin routes and path-less announces - [Remove `--hop`](/quest/m1/hop-removal.md) - on `dev`, redundant publishers share an explicit `@`, and `--hop` and the publisher's Hop ID are gone - [One route cost](/quest/m1/route-cost.md) - on `dev`, Warm and Cold collapse to one static route cost - [Delete removed cluster flags](/quest/m1/cluster-shims.md) - on `dev`, once a release has carried their refusals, `mesh` and `linger` leave `cluster::Config` @@ -147,9 +149,7 @@ QUIC studies there on that rule. - [Admission bench](/quest/m1/admission-bench.md) - the driver's per-event admission walk is benchmarked over tracks and copies - [Publish channel count](/quest/m1/publish-audio-channel-count.md) - forcing a channel count on an Audio.Capture stops costing the subscriber gaps of silence - [JS abandonment](/quest/m1/js-subscribe-abandonment.md) - a viewer returning during IETF subscribe setup keeps its track across microtasks -- [Epoch primitive](/quest/m1/epoch.md) - one `Epoch` type in moq-net and @moq/net, carried as a trailing `@` path segment, shared by e2ee and broadcast epochs - [E2EE](/quest/m1/e2ee/README.md) - TypeScript and Rust peers interoperate over encrypted broadcasts no relay can decrypt -- [Broadcast epochs](/quest/m1/broadcast-epoch/README.md) - each publish of a name gets a fresh `@` epoch, viewers follow the newest live one at once, and bare names still resolve on every version - [#3056](/quest/m1/3056-watch-video-decoder-captures-the-rewind-generation-at.md) - watch: the video decoder resets on a declared discontinuity - [#933](/quest/m1/933-video-rotation-metadata-not-propagated-from-mobile-camera.md) - the catalog rotation follows the live camera's orientation - [#2848](/quest/m1/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md) - the Opus producer follows its bandwidth grant through the settled `moq_mux::rate::Control` diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index 9ab73cc01e..464dd50723 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -2,137 +2,139 @@ ## Goal -A cluster carries each broadcast into a region once and never exposes its -backbone to end users. An operator builds edge and core tiers from one -relay: an edge runs with transit off, serves end users, and spreads paths -over its region's cores; cores link to the cores of other regions over -configured links with static costs; and routing between them stays path -vector. A broadcast under overlapping -prefixes routes to one origin deterministically. Redundancy an operator -configures may deliver more than one copy. - -Non-goals: edges acting as each other's intermediates (it makes edges a -bigger DDoS target), cache-aware route switching, and a permanently -mixed-version cluster. +Any MoQ node routes a subscription toward a broadcast's origin over whatever +links it holds, and a change in the network costs messages in proportion to +the origins it touches, not the broadcasts behind them. One protocol serves +every layout: moq.pro's edge and core tiers, a customer's own cluster, and a +drone swarm where some drones have an uplink, the rest reach it over a lossy +radio mesh, and every drone proxies for its neighbours. A node holding both a +direct link and a CDN link knows which one reaches an origin, so it uses the +CDN only when no direct route exists. The CDN never exposes its backbone to +end users, and a broadcast under overlapping prefixes routes to one origin +deterministically. + +Non-goals: link-state topology, cache-aware route switching, TURN (the CDN +route is the fallback), and a permanently mixed-version cluster. ## Plan ### Decisions -Settled in the 2026-09-30 wildcard audit (cache tiers), replacing the -link-state and existence-split design planned earlier that day: - -- Edge and core are tiers an operator configures, not relay roles - (decided 2026-10-01). The relay's one setting is `cluster.transit`: off, it - never re-advertises a route from one cluster link to another. An edge runs - with it off and dials every core in its region over qmux on TLS - (intra-region links are not congested), picking one per path by rendezvous - hashing, so a broadcast crosses into a region once. A region may have - several cores, which do not link to each other. Cores dial the cores of - the regions they link to, only those with lower names, so each pair has - one connection. A one-node region is a transit relay that also serves end - users. Cores are hidden from end users by admission, which is the DDoS - shield. See [Non-transit relays](/quest/m1/cluster-routing/transit.md). -- Core links are configured and static, and may skip PoPs when the RTT is - short: Seattle pulls from San Jose directly rather than via Oregon. Nothing - switches routes on cache state, so Warm and Cold collapse to one route cost - ([One route cost](/quest/m1/route-cost.md)), and the wildcard line's route - upgrade quest is deleted. -- No link-state topology. With split-horizon edges and a sparse core graph, - path vector with hop lists stays loop-free, path hunting is confined to the - cores, liveness flooding (the simulator's dominant link-state cost) is never - paid, and failure detection sets the outage window either way. The parked - implementation is #4631. Stale paths after a withdrawal are ended by - per-origin seqnos on cluster links - ([Path hunting](/quest/m1/cluster-routing/path-hunting.md)); a lite-06 - hold-down (#4644) was rejected because it breaks seamless failover, and - #4642's cursor hold mitigates production meanwhile. -- The line lands on `main`: its children are additive. The two breaking - changes left it for `dev` on their own: - [Remove `--hop`](/quest/m1/hop-removal.md) and - [One route cost](/quest/m1/route-cost.md). -- Cluster links are moq-lite only; moq-transport peers are plain clients - ([moq-transport peers are plain clients](/quest/m1/ietf-cluster-off.md)). -- An epoch-qualified concrete path (`foo/@`) is a source's identity: - origins that announce the same one are interchangeable, which is how a - redundant pair is expressed. A path a claim produces keeps Wildcard's - per-origin identity. See [Selection](/quest/m1/cluster-routing/selection.md). -- Between clusters, announcements stay path vector with cluster ids as hops, - in the last child. -- Anything specific to moq.pro's deployment (generating peer lists and tiers - from its inventory, refusing clients on its cores, its simulator) is - planned in moq.pro, not here. +Settled in the 2026-10-01 audit (#TBD), replacing the per-broadcast path +vector the 2026-09-30 cache-tiers audit kept: + +- **Two layers on one stream.** A ROUTE advertises reachability of one origin + node (node id, seqno, metric); an ANNOUNCE says a prefix lives at a route's + node and carries no path. Both ride the session's announce stream, so a + ROUTE always precedes the ANNOUNCEs naming it. A link flap updates one ROUTE + per affected origin instead of re-announcing every broadcast through it, + and ending a broadcast is one fact with nothing to path-hunt. The hop list + and the per-broadcast seqno plan (#4650) are gone. See + [Routes and announces](/quest/m1/cluster-routing/routes.md). +- **Loop freedom per node, not per prefix.** Routes use Babel's feasibility + condition (RFC 8966) keyed by node id. Babel was set aside earlier because + it is not loop-free for a prefix with several origins (RFC 8966 s2.7); + routing to a node removes that case, and a redundant pair is two ANNOUNCEs + of one epoch path. +- **Next-hop authority.** A node stores every neighbour's announces but treats + as live only those from its current next hop toward the origin. Babel's next + hops toward one origin form a tree, so an END is final and no cycle of + neighbours can keep a dead broadcast alive, while losing a neighbour just + moves the next hop and adopts the stored copies. +- **Every hop re-selects.** No origin pinning in SUBSCRIBE: a pin breaks + subscription aggregation and has no good answer when stale. Nearest-origin + selection over consistent metrics is still a shortest-path tree, so it is + loop-free once converged, and a transient cycle collapses into one + aggregated subscription until re-selection moves it. The reply names the + serving node for the splice rule. +- **Policy per link, not roles.** A link marked upstream never receives + routes learned on another upstream link, and a route learned upstream keeps + a down-only mark across other links so a mesh with two uplinks cannot leak + CDN routes back into the CDN. An edge marks its core links upstream; a + Starlink drone marks its CDN link upstream. See + [Upstream links](/quest/m1/cluster-routing/transit.md). +- **One node id space, no cluster ids.** A customer cluster is nodes behind + upstream or trusted links. Loop detection by cluster id would drop a + partitioned swarm's traffic to itself through the CDN (BGP's partitioned-AS + problem). Folding a large customer cluster into one id at its boundary is + [Routing cost domains](/quest/m3/routing-cost-domains.md)'s. +- **Every session speaks it**, relays, apps, and browsers, in the wip lite + version; a plain client with one link advertises only itself. Node ids are + opaque randoms, so routes reveal no backbone addresses. +- **Trust.** Cluster-peer links may advertise any node; a client link's node + ids stay scoped to its session. See + [Route trust](/quest/m1/cluster-routing/route-trust.md). +- **Metric on the wire, cost policy local.** One additive metric. Static + configured costs stay the default (the CDN); a radio link may measure its + own ([Link quality](/quest/m1/cluster-routing/link-quality.md)). +- **Topology now, wire later.** moq.pro's edge and core migration runs on + today's lite-06 path vector with upstream links; the route layer lands in + the wip version afterwards with no re-layout, once + [the simulator](/quest/m1/cluster-routing/sim.md) has compared it. + +Kept from 2026-09-30: core links are configured and may skip PoPs; Warm and +Cold collapse to one cost ([One route cost](/quest/m1/route-cost.md)); +epoch-qualified paths are a source's identity +([Selection](/quest/m1/cluster-routing/selection.md)); cluster links are +moq-lite only ([moq-transport peers are plain clients](/quest/m1/ietf-cluster-off.md)); +`--hop` removal and One route cost land on `dev` on their own. Children +changing the wip lite version count as additive and land on `main`. +Anything specific to moq.pro's deployment is planned in moq.pro. ### Simulator findings -Measured on the earlier flat mesh; they motivated the link-state design the -tiers decision replaced, and still bound what path vector costs. +Measured on the earlier flat mesh with moq.pro's `just rs sim` (live's graph) +and `just rs sim sweep` (synthetic regional graphs of 34, 340, and 1020 +relays). -The report is moq.pro's `just rs sim` (every scenario on live's graph) and -`just rs sim sweep` (synthetic regional graphs of 34, 340, and 1020 relays). -It carries messages and bytes by kind, per relay and cross-region, -convergence, loop, stall, and failover windows, and state per relay and per -registry. - -- Existence costs about one message per relay per event flooded, and one per - interested relay plus one per remote registry with registries. On live, a +- Existence costs about one message per relay per event flooded. On live, a wide customer prefix with `.stats` from every relay cost 2.5M announces - today, 4k flooded, and 3.5k with registries, whose relays held 63 records - on average against 94. -- Liveness is the split's dominant cost at scale: flooding every link repeats - each report once per neighbour. At 1020 relays (mean degree 35), a link - loss and restore plus a relay loss and restart cost about 290 MiB of - liveness flooding and 54 MiB of session-up digests against under 1 MiB of - existence for twenty publishes, while a publish costs exactly one message - per relay. -- Unbatched, one relay restart at 340 relays sent half a million liveness - messages; batched for 50 ms, 27k. + under path vector and 4k flooded. +- Link-state liveness flooding dominates at scale: at 1020 relays (mean + degree 35), a link loss and restore plus a relay loss and restart cost + about 290 MiB of liveness against under 1 MiB of existence for twenty + publishes. Per-node distance vector sends a route change only where the + best route changes, which is what the split candidate must show. - Failure detection, not routing, sets every outage window: a silent link or - relay loss is noticed only after the QUIC idle timeout in every candidate, - and subscribes through it go nowhere until then. -- The simulator saw no loop while views agreed, and HRW split an equal-cost - pool 63/49 where a hash of the announced prefix sent all of it to one - sibling (Wildcard has since keyed the tie on the requested path). On live's graph no disagreement looped across 20 seeds of link, - cost, and relay churn. -- One registry per region and two cost about the same; two halves the - busiest registry's load. A registration sent on a dead registry session - that nobody has noticed yet waits for the failover. + relay loss is noticed only after the QUIC idle timeout. +- HRW split an equal-cost pool 63/49 where a hash of the announced prefix + sent all of it to one sibling. - [#4399](https://github.com/moq-dev/moq/pull/4399) trades path hunting for - flapping. In its own full mesh of 8 it retracts each of 100 withdrawn - broadcasts exactly once per relay, as its tests show. On live's graph, with - each writer coalescing for 1 ms, ending three paths loops subscribes for - 9 s instead of 41 s and edges report the ended paths for 38 s summed - instead of 93, but it costs about the same 60k messages, and clients see - 17k updates instead of 12k, with ended paths re-announced 4185 times - instead of 163: each re-announce by a peer revives every route through it, - stale ones included. Written without coalescing, the revivals cascade, and - the same scenario costs 18.6M messages instead of 189k. Per-origin seqnos, - above, end both. + flapping: each re-announce by a peer revives every route through it, stale + ones included, and without coalescing the revivals cascade (18.6M messages + instead of 189k). ANNOUNCEs that carry no path and follow the next hop end + both. ### Remaining work Once every child has landed: -- An end-to-end test: a two-region cluster in `rs/moq-relay/tests` where a - publish, an end, a core loss, and a redundant-pair failover each reach a - subscriber on a far edge, with time mocked. -- Rewrite `doc/bin/relay/cluster.md` into the operator's view (edge and core - layout with `cluster.transit`, dial rules, TLS edge links, link costs, idle timeout, redundant - pairs), and add a routing page under `doc/concept`. +- End-to-end tests in `rs/moq-relay/tests` with time mocked: a two-region + tiered cluster (publish, end, core loss, redundant-pair failover reaching a + far edge), and a drone mesh with two uplinks that partitions and heals + (a broadcast from a drone without an uplink reaches a CDN subscriber, and + each island reaches the other through the CDN). +- Rewrite `doc/bin/relay/cluster.md` into the operator's view (upstream + links, tiers, dial rules, TLS edge links, link costs, redundant pairs), and + add a routing page under `doc/concept` that walks both the CDN tiers and the + drone mesh through the same ROUTE and ANNOUNCE rules. ## Required -- [Non-transit relays](/quest/m1/cluster-routing/transit.md) - a relay with `cluster.transit` off never carries routes between its cluster links, which is what an edge tier needs +- [Upstream links](/quest/m1/cluster-routing/transit.md) - a link marked upstream never receives routes learned on another upstream link, which builds edge tiers and drone uplinks from one rule - [Selection](/quest/m1/cluster-routing/selection.md) - a broadcast under overlapping prefixes routes to one origin deterministically, and same-epoch origins are one source -- [Path hunting](/quest/m1/cluster-routing/path-hunting.md) - per-origin seqnos on cluster links end stale re-announces without breaking seamless failover -- [Between clusters](/quest/m1/cluster-routing/inter-cluster.md) - announcements crossing a cluster boundary stay path vector with cluster ids as hops +- [Simulate the split](/quest/m1/cluster-routing/sim.md) - moq.pro's simulator compares the route layer with path vector before the wire is written +- [Routes and announces](/quest/m1/cluster-routing/routes.md) - ROUTE per origin node and path-less ANNOUNCE on one stream, loop-free by Babel feasibility +- [Route trust](/quest/m1/cluster-routing/route-trust.md) - a peer grant lets a client link advertise the nodes behind it as one identity +- [Link quality](/quest/m1/cluster-routing/link-quality.md) - a radio link's cost follows its measured quality without flapping routes ## Related - [Wildcard](/quest/m0/wildcard/README.md) - the longest-prefix rule, pool spread, and reply identity Selection builds on +- [P2P](/quest/m2/p2p/README.md) - browser and native peers that become routing nodes over this layer - [Remove `--hop`](/quest/m1/hop-removal.md) - on `dev`: redundant publishers share an explicit `@` - [One route cost](/quest/m1/route-cost.md) - on `dev`: Warm and Cold collapse to one static cost - [Same-hop importers](/quest/m1/hop-aligned-import.md) - the importer half of a redundant pair; `--hop` removal re-keys it to a shared epoch - [Broadcast epochs](/quest/m1/broadcast-epoch/README.md) - a redundant pair shares one epoch -- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - its #4349 report shows closed broadcasts announced for up to 229 s, evidence for per-origin seqnos -- [Routing cost domains](/quest/m3/routing-cost-domains.md) - cost across the cluster boundaries this keeps path vector +- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - its #4349 report shows closed broadcasts announced for up to 229 s +- [Routing cost domains](/quest/m3/routing-cost-domains.md) - policy and aggregation at boundaries between operators diff --git a/quest/m1/cluster-routing/inter-cluster.md b/quest/m1/cluster-routing/inter-cluster.md deleted file mode 100644 index 7bc07c072f..0000000000 --- a/quest/m1/cluster-routing/inter-cluster.md +++ /dev/null @@ -1,36 +0,0 @@ -# [M] Routing between clusters - -## Goal - -Announcements crossing a cluster boundary stay path vector with cluster ids -as the hops, like BGP between autonomous systems. A customer's on-prem -cluster is one hop, and an announcement naming the receiving cluster is -dropped. Inside a cluster, hop lists name relays; across a boundary they name -clusters. - -## Plan - -- A boundary is a configured link to a relay of another cluster; decide how - a relay knows its own cluster id and its peer's. -- What crosses a boundary is a cluster's announcement, plus its cluster-id - list. An imported record keeps that list as provenance, so - whichever boundary relay exports it appends its own cluster id to the full - list; stripping it on import would let A → B → C → A re-enter A. -- The remote origin is outside the receiving cluster, so the - importing boundary relay announces the record inside its cluster as the - origin, with the boundary link's cost added. The serving origin's identity - still rides the reply, so re-originating does not merge two sources. -- Cluster-id lists are short, so they need no `Hop Base`/`Hop Keep` - compression. -- Cost across the boundary is plain configured link cost; business policy - and incomparable costs are - [Routing cost domains](/quest/m3/routing-cost-domains.md)'s. - -Wire: the boundary announcement in the current wip lite version, with the -draft updated in the same PR. Tests: two clusters with two boundary links -and an origin that is not a boundary relay, and three clusters in a cycle with different ingress and egress relays, where -an announcement neither loops nor re-enters its origin cluster. - -## Related - -- [Routing cost domains](/quest/m3/routing-cost-domains.md) - designs policy on this path vector diff --git a/quest/m1/cluster-routing/link-quality.md b/quest/m1/cluster-routing/link-quality.md new file mode 100644 index 0000000000..4fc0310774 --- /dev/null +++ b/quest/m1/cluster-routing/link-quality.md @@ -0,0 +1,34 @@ +# [M] Link quality + +## Goal + +A link can take its cost from its measured quality instead of configuration, +so a drone mesh steers around a fading radio link without manual tuning, +while a small wobble never flips a route. Static configured cost stays the +default and is what the CDN runs; measured cost is opt-in per link. This is +local policy feeding the one metric on the wire, with no wire change. + +## Plan + +Decided 2026-10-01: the wire carries one additive metric and each node +computes its own link costs. + +- Derive the cost from what QUIC already measures (smoothed RTT, loss); an + ETX-style estimate (Babel RFC 8966 Appendix A, B.A.T.M.A.N.'s TQ) is the + reference. Pick the formula from the harness, not from intuition. +- Hysteresis: only advertise a cost change past a threshold or after it + holds, so the route layer sees a few changes per real shift. Report how + many ROUTE changes a fading link causes. +- Never mix it into business pricing: moq.pro's + [priced topology](https://github.com/moq-dev/moq.pro/blob/main/quest/m3/priced-topology.md) + keeps RTT out of monetary costs, and that stays true because measured cost + is opt-in on links the operator chooses. +- A lossy-link fixture over the mock transport with mocked time, reusable by + the line's drone-mesh end-to-end test. + +Public API: a per-link way to ask for measured cost in the peer entry. +Wire: none. + +## Required + +- [Routes and announces](/quest/m1/cluster-routing/routes.md) - a cost change costs one ROUTE per origin there, instead of a re-announce per broadcast diff --git a/quest/m1/cluster-routing/path-hunting.md b/quest/m1/cluster-routing/path-hunting.md deleted file mode 100644 index 3304e7d144..0000000000 --- a/quest/m1/cluster-routing/path-hunting.md +++ /dev/null @@ -1,51 +0,0 @@ -# [M] Per-origin sequence numbers end path hunting - -## Goal - -When a broadcast ends, each relay retracts it once and never re-announces a -stale alternative, and a failover where the origin still publishes stays -seamless: no downstream front loses its route while a working one exists. - -## Plan - -History (2026-09-30 to 10-01): moq-dev/moq.pro#2116 measured ended names -drawing 575 to 57,805 mesh announce starts on Kick dev, and reproduced it on -live's 34-relay graph. #4642's 300 ms announce-cursor hold mitigates it and -is what production runs; it still leaves 16 of 34 publishers hunting in the -repro. #4644 added a cluster-link hold-down (retract at once, re-announce a -replacement after 1 s) and measured 0 resurrections, but the maintainer -rejected it: the immediate retraction ends a downstream peer's front on -`best: None` when its only path ran through the relay, so subscribers drop -for about 1.1 s while the relay keeps serving. Holding the replacement -without retracting brought hunting back. A lite-06 announce-side fix cannot -avoid that retraction. - -Decided: carry a per-origin seqno, scoped to the origin's incarnation, with -each announcement on cluster links in the wip lite version, and never apply -an event older than one already seen for that path and origin (DSDV and -Babel feasibility). A withdrawal then outranks every stale copy without a -hold-down or a retraction of live alternatives. - -- Start from #4644's branch (`quest/m0/path-hunting`): its live-graph - regression tests (withdrawal at several latencies, and failover) fail on - `main` and stay the acceptance tests. Their fixture hard-codes lite-06; - run them on the wip version. Add a check that a downstream in-flight - subscription survives the failover. -- Keep an ended path's seqno long enough to outlive delayed copies of its - start; a new incarnation clears it. -- Decide whether #4642's cursor hold can then be removed. -- Clients and lite-06 cluster peers are unchanged; only wip-version cluster - links carry the field. - -Wire: a field on ANNOUNCE_START, ANNOUNCE_UPDATE, and ANNOUNCE_END in the -wip lite version, with `drafts/draft-lcurley-moq-lite.md` updated in the -implementation PR. A failover that switches origin or advances its seqno -travels as ANNOUNCE_UPDATE (`AnnounceBroadcast::Restart`), so the update -carries its own seqno; forcing END then START would bring back the -retraction this quest removes. Add a regression that advances the seqno via -an update without retracting the live advertisement. - -## Related - -- [Non-transit relays](/quest/m1/cluster-routing/transit.md) - non-transit edges shrink the graph this hunts on to the cores -- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - closed broadcasts announced for minutes, the same symptom diff --git a/quest/m1/cluster-routing/route-trust.md b/quest/m1/cluster-routing/route-trust.md new file mode 100644 index 0000000000..592e6bd8fe --- /dev/null +++ b/quest/m1/cluster-routing/route-trust.md @@ -0,0 +1,39 @@ +# [M] Route trust + +## Goal + +A session may advertise routes only to nodes it is trusted for, so an +authorized publisher cannot claim a cheap route to a CDN relay's node and +attract its subscriptions. Cluster-peer links may advertise any node. A +client link's node ids stay scoped to that session, unless its peer grant +names the nodes it may speak for: then a Starlink drone carries its fleet's +nodes as their real identities, and two uplinks of one swarm reach one origin +instead of two unrelated ones. + +## Plan + +Decided 2026-10-01: + +- Session scoping is the default and ships with + [Routes and announces](/quest/m1/cluster-routing/routes.md): a scoped id + resolves only that session's own authorized announces and never shadows a + route learned on a trusted link. +- Promotion rides [Peer grants](/quest/m1/auth/peer-grant.md), the + hop-bound credential a direct session already presents; extend it to name + the node ids (or an id range or prefix of them) the holder may advertise. + Decide the shape with the auth line. +- A promoted link is still subject to its upstream attribute and the + down-only bit. + +Test: a client advertising a cheap route to a cluster node does not move any +other session's subscription; two uplinks of one mesh holding grants for the +same drone node fail over between each other at a group boundary instead of +ending the subscription. + +Public API: the grant's node scope (`moq-token`, `js/token`, and the token +CLI docs). Wire: none beyond the grant. + +## Required + +- [Routes and announces](/quest/m1/cluster-routing/routes.md) - the node ids this scopes +- [Peer grants](/quest/m1/auth/peer-grant.md) - the credential this extends diff --git a/quest/m1/cluster-routing/routes.md b/quest/m1/cluster-routing/routes.md new file mode 100644 index 0000000000..b3cd5b3f6d --- /dev/null +++ b/quest/m1/cluster-routing/routes.md @@ -0,0 +1,88 @@ +# [XL] Routes and announces + +## Goal + +Routing info splits into two layers on the session's announce stream, in the +wip lite version, for every session. A ROUTE advertises reachability of one +origin node; an ANNOUNCE says a prefix lives at a route's node and carries no +path. A link flap or relay loss sends one ROUTE change per origin whose best +route changed, never a re-announce per broadcast; ending a broadcast reaches +each node about once and nothing hunts; failover to another neighbour is +seamless; and no state of disagreeing neighbours keeps a dead broadcast +alive. + +## Plan + +Decided 2026-10-01 (see the [line's decisions](/quest/m1/cluster-routing/README.md)): + +- Wire sketch, flexible in detail. ROUTE_START carries the node id, a seqno, + the metric, and a down-only bit, and implicitly takes the next stream-local + Route ID, as ANNOUNCE_START takes the next Announce ID today. ROUTE_UPDATE + (Route ID, seqno, metric) and ROUTE_END (Route ID) reference it. + ANNOUNCE_START carries the prefix (keeping today's Path Base/Keep + compression) and a Route ID. An ANNOUNCE naming an unknown Route ID is a + protocol violation, and ROUTE_END ends that stream's ANNOUNCEs on the + route. The hop list and per-announce cost leave this version. + + ```text + ROUTE_START node=0x7a3f seqno=41 metric=12 -> route 0 + ANNOUNCE_START prefix=drone/cam1 route=0 -> announce 0 + ROUTE_UPDATE route=0 seqno=42 metric=15 + ROUTE_END route=0 # ends announce 0 too + ``` +- The node id is global and opaque: a relay's `cluster.id` or a random id, + and an app's handshake id. It is needed so routes from two neighbours to + one origin are recognized as one, which carries loop freedom, + deduplication, the reply's serving node, and P2P dialing (an app maps an + announce's node to a roster peer). +- Loop freedom is Babel's feasibility condition (RFC 8966) keyed by node: + accept a route if its seqno is newer, or equal with a metric below the + feasibility distance. Only the origin advances its seqno. Retraction is an + infinite metric or ROUTE_END; there is no count to infinity. +- Next-hop authority: store every neighbour's ANNOUNCEs, but a prefix is live + at a node only as its current next hop toward the origin announces it, the + neighbour a subscribe for that path would go to (metric, then the path-keyed + rendezvous hash among equal next hops). When the next hop changes, adopt + the new neighbour's stored set and send downstream only the difference. + No per-announce seqno. +- Down-only bit: set on a route learned on an upstream link, kept across + other links, and a route carrying it is never sent on an upstream link + ([Upstream links](/quest/m1/cluster-routing/transit.md)). +- A plain client with one link advertises a ROUTE for itself and the + ANNOUNCEs it publishes; its node id is scoped to its session (shared + identity across sessions is [Route trust](/quest/m1/cluster-routing/route-trust.md)). + A relay advertising routes to a client sends them as usual; node ids reveal + nothing about the backbone. +- Every hop re-selects; SUBSCRIBE names no origin. The reply names the + serving node, which is the identity Selection's splice rule uses. +- Mixed versions: a lite-06 peer keeps today's path vector, translated at the + relay that speaks both, for the rollout window only. + +Open, for the implementer to settle and record: + +- Seqno lifetime across restarts of a node with a configured stable id + (persist it, fold an incarnation into the id, or Babel's seqno request). +- Whether starvation recovery needs a request message, or session restart + covers it. +- The window where an origin's END and a next-hop change cross in flight, + and whether it needs more than the next END to arrive. +- How much of today's route trie, fronts, and `route_order` survives intact. + +Tests, time mocked: #4644's live-graph withdrawal and failover tests +(`quest/m0/path-hunting` branch) as acceptance, run on the wip version; a +link flap with many broadcasts behind one origin costs one ROUTE change per +link; a ring of three where a stale neighbour cannot revive an ended +broadcast; a two-uplink mesh that never carries CDN routes back to the CDN; +an island reached through the CDN after a partition; an unknown Route ID +refused. + +Wire: `drafts/draft-lcurley-moq-lite.md` in the same PR, and `js/net` +encodes, decodes, and resolves it (JS transit stays in +[P2P](/quest/m2/p2p/README.md)). Public API: the route-change surface on +`broadcast::Route` and its bindings will likely change; report it. This may +split at start (Rust and draft, then JS), as long as both land in one +release. + +## Required + +- [Simulate the split](/quest/m1/cluster-routing/sim.md) - the numbers that confirm the design before the wire is written diff --git a/quest/m1/cluster-routing/selection.md b/quest/m1/cluster-routing/selection.md index 7006e8e7fc..7fecbfbe12 100644 --- a/quest/m1/cluster-routing/selection.md +++ b/quest/m1/cluster-routing/selection.md @@ -3,8 +3,9 @@ ## Goal A broadcast under overlapping announcements routes to one origin chosen -deterministically, and an edge spreads paths over its region's cores so -every edge sends a given path the same way. +deterministically, and a relay spreads paths over equally ranked next hops +(an edge over its region's cores) so every relay sends a given path the same +way. Origins that announce the same epoch-qualified concrete path are one source, so a subscriber moves between them at a group boundary when the incumbent ends or becomes unreachable. @@ -27,14 +28,20 @@ Decided: announce the same epoch-qualified path and must not pool. This extends Wildcard's resume rule to concrete same-epoch origins. -Candidate mechanics (re-scoped to path vector by the 2026-09-30 tiers -decision; there is no topology to forward by): +Candidate mechanics: - Route selection stays `route_order` over the routes a relay holds: longest prefix, then cost, then a rendezvous hash (HRW) of the requested path and the route's origin. At an edge that spreads paths over the region's cores and makes every edge pick the same core for a path; failover moves only the paths the lost core won. +- Every hop re-selects (decided 2026-10-01): SUBSCRIBE names no origin, since + a pin breaks subscription aggregation. Once + [Routes and announces](/quest/m1/cluster-routing/routes.md) lands, the + candidates are the origin nodes announcing the path, ranked by longest + prefix, then route metric to the node, then HRW, and the reply's Origin is + the serving node id. Write the ranking so that change swaps its inputs, not + its shape. - A refusal follows Wildcard's refusal rule: every refusal is terminal, and an origin sheds load by withdrawing or re-pricing its route instead. - The serving origin's identity rides the Origin field of the reply, per @@ -54,9 +61,6 @@ subscribers end and resubscribe rather than splice. ## Required -- [Non-transit relays](/quest/m1/cluster-routing/transit.md) - the edges and cores this spreads over +- [Upstream links](/quest/m1/cluster-routing/transit.md) - the edges and cores this spreads over - [Wildcard](/quest/m0/wildcard/README.md) - the longest-prefix rule, pool spread, and reply identity this builds on - -## Related - - [Epoch primitive](/quest/m1/epoch.md) - parses the `@` segment that makes a path a source identity diff --git a/quest/m1/cluster-routing/sim.md b/quest/m1/cluster-routing/sim.md new file mode 100644 index 0000000000..b013d47433 --- /dev/null +++ b/quest/m1/cluster-routing/sim.md @@ -0,0 +1,19 @@ +# [XS] Simulate the split + +## Goal + +Waits on moq.pro's routing simulator. The condition clears when moq.pro's +[Simulate the edge and core layout](https://github.com/moq-dev/moq.pro/blob/main/quest/m1/routing-sim-tiers.md) +reports a ROUTE and ANNOUNCE split candidate (per-node distance vector with +Babel feasibility, path-less announces following the next hop) beside path +vector, on the tiered live layout, the tiered synthetic sweeps, and a +synthetic drone mesh (lossy links, two uplinks, a partition that heals). + +Per scenario (publish, end, link flap, relay loss and restart, uplink loss): +messages and bytes by kind, convergence time, stale-path seconds, transient +subscribe loops, and whether a two-uplink mesh ever carries CDN traffic. + +To check: look for that report in a merged moq.pro PR. To advance: ask the +maintainer to start the moq.pro quest. Delete this quest once the report is +in and [Routes and announces](/quest/m1/cluster-routing/routes.md) is +confirmed or revised by it. diff --git a/quest/m1/cluster-routing/transit.md b/quest/m1/cluster-routing/transit.md index 8ed950fcc9..a938e75adb 100644 --- a/quest/m1/cluster-routing/transit.md +++ b/quest/m1/cluster-routing/transit.md @@ -1,62 +1,65 @@ -# [M] Non-transit relays +# [M] Upstream links ## Goal -A relay can be configured never to carry routes between its cluster links, -which is what lets an operator build edge and core tiers out of one relay -with no roles. A relay with `cluster.transit` off advertises only what -entered locally to its cluster links and never re-advertises a route learned -from one link to another. It still serves its own clients every route it -knows. Edge, core, and one-node region are an operator's names for -configurations, not relay modes. +A cluster link can be marked upstream, and a relay never re-advertises a +route learned on one upstream link to another upstream link. Everything else +still transits. That one rule builds moq.pro's edge tier (an edge marks its +region's cores upstream, so it is never transit between them) and a drone +uplink (a Starlink drone marks its CDN link upstream: the mesh reaches the +CDN and the CDN reaches the mesh, but the drone never carries CDN traffic +between CDN relays). Edge, core, and one-node region are an operator's names +for configurations, not relay modes. Clients still see every route a relay +knows. ## Plan -Decided 2026-10-01, replacing the edge and core roles the 2026-09-30 -wildcard audit (cache tiers) planned: - -- `cluster.transit` is the node's own setting, on by default, from config or - alongside the peer list the connect API returns. Off, every cluster-peer - session publishes only `origin::Consumer::local()`, accepted as well as - dialed (today they take different paths in `cluster.rs`), so no cluster - link routes through it whichever side dialed. The grant's peer - classification tells a peer from a client, and clients still see - everything it learned from its links. - A typical edge runs with it off and dials its region's cores; a core or a - one-node region runs with it on. -- No roles. Who dials whom is the peer list, and whatever generates the list - applies the operator's rules (an edge dials its region's cores, cores dial - only lower-named cores of linked regions). The relay does not check its - links against a topology; the generator is where that is tested. -- Refusing clients on a hidden node is admission, not routing: the auth - server or embedder already decides every session and sees its verified - client certificate, so it refuses a session that is not a cluster peer. - The relay adds no client switch. -- Relays from before this setting are transit, so a cluster migrates one - region at a time with no legacy mode. A permanently mixed-version cluster - stays a non-goal. -- Spreading paths over equally ranked links by the path-keyed rendezvous - hash is [Selection](/quest/m1/cluster-routing/selection.md)'s, for every - relay. With transit off it is what makes every edge of a region send a - given path to the same core. +Decided 2026-10-01, replacing the per-node `cluster.transit` setting (#4683) +with a per-link attribute, because a drone must transit for its mesh but never +for its CDN uplink, which no per-node switch can express (BGP's +customer/provider export rule, RFC 9234, reduced to one bit): + +- `upstream` is an attribute of a peer entry, from config or the connect + API's peer list, off by default. It applies to the link whichever side + dialed: an edge that is dialed by a core still treats that core as + upstream, so accepted peer sessions need a way to be marked (the peer's + grant classification or identity is the likely source; pick the simplest + shape that covers the reverse-dial test). +- An upstream link is sent only routes that did not arrive on another + upstream link. On today's path vector that is the routes that entered + locally or over non-upstream links. +- The down-only mark that stops a two-uplink mesh from leaking CDN routes + back into the CDN needs a wire bit, so it lands with + [Routes and announces](/quest/m1/cluster-routing/routes.md), not here. +- No roles and no topology check. Who dials whom is the peer list, and + whatever generates it applies the operator's rules (an edge dials its + region's cores, cores dial only lower-named cores of linked regions); the + generator is where that is tested. +- Refusing clients on a hidden core is admission, not routing: the auth + server or embedder refuses a session that is not a cluster peer. +- Relays from before this attribute treat every link as transit, so a cluster + migrates one region at a time with no legacy mode. +- Spreading paths over equally ranked upstream links by the path-keyed + rendezvous hash is [Selection](/quest/m1/cluster-routing/selection.md)'s; + it is what makes every edge of a region send a given path to the same core. - Edge links run qmux over TLS on TCP, not WebSocket and not QUIC: intra-region links are not congested. qmux over TCP exists today only in plaintext (`tcp://`, moq-tokio's `tcp.rs`); add `tls://` beside it on - moq-tokio's `tls` helpers. - (Scheme name decided 2026-09-30.) - -Test: a two-region cluster in `rs/moq-relay/tests`. One region has two -non-transit relays dialing two transit relays; the other is one transit relay -that also serves clients. Subscribers on both non-transit relays cause one -cross-region subscription per broadcast, a route from one transit relay never -reaches the other through a non-transit relay, a client on the far relay -gets a broadcast published on a non-transit relay, a transit relay that -dials a non-transit relay still gets no routes through it, and losing a -transit relay moves only its paths. - -Public API: `cluster.transit` (default on) and a TLS qmux URL scheme. + moq-tokio's `tls` helpers. (Scheme name decided 2026-09-30.) + +Test: a two-region cluster in `rs/moq-relay/tests`. One region has two edges +whose links to two cores are upstream; the other is one relay that also +serves clients. Subscribers on both edges cause one cross-region subscription +per broadcast, a route from one core never reaches the other through an edge, +a client on the far relay gets a broadcast published on an edge, a core that +dials an edge still gets no routes through it, and losing a core moves only +its paths. Add a three-node line (mesh node, uplink node, CDN relay) where the +mesh node's broadcast reaches the CDN and a CDN broadcast reaches the mesh +node. + +Public API: an `upstream` peer attribute and a TLS qmux URL scheme. `cluster.tier` is unrelated: it stays the billing and stats label. -Wire: none expected. +Wire: none. ## Related diff --git a/quest/m1/cross-relay-bursts.md b/quest/m1/cross-relay-bursts.md index a7a31241d4..3dc14f1030 100644 --- a/quest/m1/cross-relay-bursts.md +++ b/quest/m1/cross-relay-bursts.md @@ -27,4 +27,4 @@ cluster-routing evidence, not this quest's scope. ## Related -- [Cluster routing](/quest/m1/cluster-routing/README.md) - owns the stale and flapping announcements from the same report +- [Routes and announces](/quest/m1/cluster-routing/routes.md) - owns the stale and flapping announcements from the same report diff --git a/quest/m1/hop-removal.md b/quest/m1/hop-removal.md index 40213ba3a0..f6cf874015 100644 --- a/quest/m1/hop-removal.md +++ b/quest/m1/hop-removal.md @@ -19,15 +19,17 @@ connection cannot publish several broadcasts with different identities. an epoch a caller supplies ([Broadcast epochs](/quest/m1/broadcast-epoch/README.md)). Give `moq` a flag for it if Broadcast epochs has not. - Relays keep `cluster.id` as their identity; `--hop` fills it today - (`rs/moq-cli/src/args.rs`), so split that coupling. Decide whether relays - still need the Hop setup parameter or learn identity from topology. + (`rs/moq-cli/src/args.rs`), so split that coupling. Relays and P2P peers + keep the Hop setup parameter: it becomes the node id of + [Routes and announces](/quest/m1/cluster-routing/routes.md). Only the + publisher-facing use goes. - Re-key [Same-hop importers](/quest/m1/hop-aligned-import.md)'s docs and 1+1 tests to a shared epoch. - Docs: rewrite "Redundant publishers" in `doc/bin/cli.md` and `doc/concept/use-case/contribution.md`, then search the repo for `--hop`, `MOQ_HOP`, and `--cluster-id` examples, including demo recipes. -Open: whether anything declares an incumbent dead faster than the cluster +Open: whether anything declares an incumbent dead faster than the QUIC idle timeout (a failover service that retracts it, or active-active delivery to the relay). Neither is required to land this. diff --git a/quest/m1/ietf-cluster-off.md b/quest/m1/ietf-cluster-off.md index b0da33252f..3aeda66a88 100644 --- a/quest/m1/ietf-cluster-off.md +++ b/quest/m1/ietf-cluster-off.md @@ -36,5 +36,5 @@ negotiating an optional extension. ## Related -- [Non-transit relays](/quest/m1/cluster-routing/transit.md) - cluster links are lite-only +- [Upstream links](/quest/m1/cluster-routing/transit.md) - cluster links are lite-only - [Wildcard](/quest/m0/wildcard/README.md) - the pool spread that makes a Hop ID label unsafe to splice on diff --git a/quest/m2/ietf-cluster-peers.md b/quest/m2/ietf-cluster-peers.md index 24c9d70849..a0803b2505 100644 --- a/quest/m2/ietf-cluster-peers.md +++ b/quest/m2/ietf-cluster-peers.md @@ -18,8 +18,9 @@ The extension must carry what a lite cluster link carries by then: FETCH_OK Origin, so a downstream stitches failover on the member serving the path rather than the advertisement's first Hop ID. The draft's "Several Publishers of One Namespace" section changes with it. -- Whatever the [cluster routing line](/quest/m1/cluster-routing/README.md) - settles for topology and selection, since cluster links are lite-only there. +- The route layer of the [cluster routing line](/quest/m1/cluster-routing/README.md) + (per-node ROUTEs with seqno and metric, path-less announces, the down-only + bit) and its selection, since cluster links are lite-only there. Test: two workers behind a pool relay behind a moq-transport downstream relay; killing the serving worker ends the downstream subscription rather than diff --git a/quest/m3/routing-cost-domains.md b/quest/m3/routing-cost-domains.md index f32615b6ae..abb942b3e6 100644 --- a/quest/m3/routing-cost-domains.md +++ b/quest/m3/routing-cost-domains.md @@ -3,9 +3,11 @@ ## Goal Settle how independently operated MoQ networks exchange reachability without -adding incomparable costs, at the cluster boundaries where -[Cluster routing](/quest/m1/cluster-routing/README.md) keeps path vector with cluster -ids as hops. Cost inside one cluster is cluster-routing's. Produce a +adding incomparable costs, at the boundaries between operators on the one node id space +[Cluster routing](/quest/m1/cluster-routing/README.md) routes over. Cost inside +one operator's network is cluster-routing's. This includes whether a boundary +may fold a large customer cluster's nodes into one advertised node, and how +that keeps loop safety. Produce a reviewed design and scoped implementation quests, not a protocol implementation. Cloudflare, moq.pro, and self-hosted relays can retain their own business policy; no RTT/loss-driven repricing or automatic performance @@ -45,8 +47,7 @@ The design must work through examples of two operators using different scales, multiple entrances to one domain, asymmetric charges, mixed-provider nodes in one domain, unknown or untrusted peers, and a route leaving and re-entering a domain. Preserve publisher identity and loop safety across any metric rewrite. -Explain how warm-route marginal savings interact with border policy without -pretending those savings erase upstream delay. State tradeoffs, migration and +State tradeoffs, migration and mixed-version behavior, and the limits of any convergence claim. Reconcile with cluster-routing's configured link costs rather than creating @@ -56,7 +57,7 @@ quests. Open wire/API choices belong to this design exercise. ## Required -- [Routing between clusters](/quest/m1/cluster-routing/inter-cluster.md) - this designs on its inter-cluster path vector +- [Cluster routing](/quest/m1/cluster-routing/README.md) - the route layer this designs policy on ## Related From 4107367dbbc307fd720d0609ad5b016080e3e96c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 1 Oct 2026 17:18:34 -0700 Subject: [PATCH 2/4] quest: link the audit PR Co-Authored-By: Claude Opus 5.5 --- quest/m1/cluster-routing/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index 464dd50723..52093b96ae 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -20,7 +20,7 @@ route is the fallback), and a permanently mixed-version cluster. ### Decisions -Settled in the 2026-10-01 audit (#TBD), replacing the per-broadcast path +Settled in the 2026-10-01 audit (#4694), replacing the per-broadcast path vector the 2026-09-30 cache-tiers audit kept: - **Two layers on one stream.** A ROUTE advertises reachability of one origin From eefaaa977f201b05d02c43d816d8126afa935ddf Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 1 Oct 2026 17:28:01 -0700 Subject: [PATCH 3/4] quest: direct P2P peers win only as origins; multi-CDN endpoints Co-Authored-By: Claude Opus 5.5 --- quest/m1/cluster-routing/README.md | 8 ++- quest/m1/cluster-routing/multi-cdn.md | 49 +++++++++++++++ quest/m1/cluster-routing/selection.md | 4 +- quest/m2/p2p/README.md | 38 ++++++------ quest/m2/p2p/cost-scopes.md | 85 ++++++++++++--------------- quest/m2/p2p/harness.md | 1 - quest/m2/p2p/signal.md | 3 +- quest/m2/p2p/transit.md | 35 ----------- quest/m2/p2p/watch.md | 21 +++---- 9 files changed, 125 insertions(+), 119 deletions(-) create mode 100644 quest/m1/cluster-routing/multi-cdn.md delete mode 100644 quest/m2/p2p/transit.md diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index 52093b96ae..a96900f2eb 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -9,7 +9,8 @@ every layout: moq.pro's edge and core tiers, a customer's own cluster, and a drone swarm where some drones have an uplink, the rest reach it over a lossy radio mesh, and every drone proxies for its neighbours. A node holding both a direct link and a CDN link knows which one reaches an origin, so it uses the -CDN only when no direct route exists. The CDN never exposes its backbone to +CDN only when no direct route exists, and an endpoint linked to several CDNs +prefers its primary and fails over to the next. The CDN never exposes its backbone to end users, and a broadcast under overlapping prefixes routes to one origin deterministically. @@ -64,6 +65,10 @@ vector the 2026-09-30 cache-tiers audit kept: - **Trust.** Cluster-peer links may advertise any node; a client link's node ids stay scoped to its session. See [Route trust](/quest/m1/cluster-routing/route-trust.md). +- **Multi-CDN by link preference.** A node may rank its links (moq.pro + primary, Cloudflare secondary) ahead of the metric, so metrics from + different operators are never compared; a CDN that speaks no ROUTE is just + a link. See [Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md). - **Metric on the wire, cost policy local.** One additive metric. Static configured costs stay the default (the CDN); a radio link may measure its own ([Link quality](/quest/m1/cluster-routing/link-quality.md)). @@ -123,6 +128,7 @@ Once every child has landed: - [Upstream links](/quest/m1/cluster-routing/transit.md) - a link marked upstream never receives routes learned on another upstream link, which builds edge tiers and drone uplinks from one rule - [Selection](/quest/m1/cluster-routing/selection.md) - a broadcast under overlapping prefixes routes to one origin deterministically, and same-epoch origins are one source +- [Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md) - an endpoint holds sessions to several CDNs, uses its preferred one, and fails over to the next - [Simulate the split](/quest/m1/cluster-routing/sim.md) - moq.pro's simulator compares the route layer with path vector before the wire is written - [Routes and announces](/quest/m1/cluster-routing/routes.md) - ROUTE per origin node and path-less ANNOUNCE on one stream, loop-free by Babel feasibility - [Route trust](/quest/m1/cluster-routing/route-trust.md) - a peer grant lets a client link advertise the nodes behind it as one identity diff --git a/quest/m1/cluster-routing/multi-cdn.md b/quest/m1/cluster-routing/multi-cdn.md new file mode 100644 index 0000000000..f073b061ec --- /dev/null +++ b/quest/m1/cluster-routing/multi-cdn.md @@ -0,0 +1,49 @@ +# [M] Multi-CDN endpoints + +## Goal + +A client or publisher holds sessions to several CDNs at once, ranked by its +own preference: moq.pro as primary, Cloudflare as secondary. A subscription +goes over the most preferred link that has a route and moves to the next one +when that route goes away; a publisher pushes the same epoch to every CDN so +the move splices instead of restarting. The secondary needs nothing from us: +a moq-transport relay that speaks no ROUTE is just a link. + +## Plan + +Decided 2026-10-01 (moq-dev/moq#4694): + +- Ranking: longest prefix, then the link's preference, then the metric, then + the rendezvous hash. Preference is a per-link rank the node sets (lower + wins, equal by default), separate from the additive link cost, so metrics + from different operators are never added together or compared; within one + preference the metric decides as before. Babel permits any choice among + feasible routes, so loop freedom is unchanged. The ranking lives in + [Selection](/quest/m1/cluster-routing/selection.md); this quest adds the + preference input and the endpoint side. +- A session that speaks no ROUTE (moq-transport, older lite) is a link whose + announces resolve to a session-scoped origin at that link's cost. +- Identity: two CDNs never share node ids, so failover across them splices + only for an epoch-qualified path; a bare path is a discontinuity. The + publisher side is the redundant pair of + [Selection](/quest/m1/cluster-routing/selection.md), from one process. +- API: the Rust client config and `js/net` accept several upstream URLs, each + with its preference (and cost), feeding one shared origin. Mirror the shape + in the bindings that expose connect config. Pick the smallest shape that + covers both publish and subscribe. + +Test with mocked time: a subscriber linked to a lite relay (preferred) and a +moq-transport relay (secondary), with a publisher pushing one epoch to both; +losing the preferred relay moves the subscription at a group boundary with +no timestamp rewind, and its return moves it back. A bare path ends and +resubscribes instead. + +Docs: a multi-CDN section in the routing concept page and the client +connect docs. + +Public API: per-upstream preference in client config, in Rust, JS, and the +bindings. Wire: none. + +## Required + +- [Selection](/quest/m1/cluster-routing/selection.md) - the ranking this adds the preference input to diff --git a/quest/m1/cluster-routing/selection.md b/quest/m1/cluster-routing/selection.md index 7fecbfbe12..8091855e8c 100644 --- a/quest/m1/cluster-routing/selection.md +++ b/quest/m1/cluster-routing/selection.md @@ -39,7 +39,9 @@ Candidate mechanics: a pin breaks subscription aggregation. Once [Routes and announces](/quest/m1/cluster-routing/routes.md) lands, the candidates are the origin nodes announcing the path, ranked by longest - prefix, then route metric to the node, then HRW, and the reply's Origin is + prefix, then the link's preference + ([Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md)), then route + metric to the node, then HRW, and the reply's Origin is the serving node id. Write the ranking so that change swaps its inputs, not its shape. - A refusal follows Wildcard's refusal rule: every refusal is terminal, and diff --git a/quest/m2/p2p/README.md b/quest/m2/p2p/README.md index f6e40ddb04..5067a3e8de 100644 --- a/quest/m2/p2p/README.md +++ b/quest/m2/p2p/README.md @@ -5,9 +5,10 @@ Opted-in moq-lite clients serve each other directly, over WebRTC data channels between any two peers and over iroh between native ones, while the relay stays the rendezvous and the path of last resort. A client subscribes through the -relay at once, negotiates peers in the background, and each subscription -migrates to a peer when the origin ranks that route cheaper and back to the -relay when the peer goes away. Nothing waits on ICE and nothing breaks when it +relay at once, negotiates peers in the background, and a subscription +migrates to a direct peer when that peer is the broadcast's origin (or a +routing node such as `moq-cli --p2p`) and back to the relay when the peer +goes away. A browser never carries another peer's traffic. Nothing waits on ICE and nothing breaks when it fails: a symmetric NAT, a denied local-network prompt, or a filtered multicast segment leaves the client exactly where it started. @@ -93,23 +94,19 @@ the same independence without fighting the browser. ### Routes and migration -Rust already does the serving-side work: `best_route` re-runs on every table -change and a live subscription re-splices onto a cheaper route with the same -first hop at a group boundary, while an anonymous chain never wins. The JS -origin ranks the same way (`compareRoutes` in `js/net/src/origin.ts`: cost, -then fewest hops, then newest). The JS handshake -already declares a random hop id, so browser hops are identified; the roster -id is that hop id, held once per origin rather than once per session. +Routing is the [cluster routing line](/quest/m1/cluster-routing/README.md)'s: +every session speaks per-node ROUTEs and path-less ANNOUNCEs, so a broadcast's +announce names its origin node, the roster maps that node to a peer, and the +route metric over the node's own link costs picks the direct link or the +relay. The roster id is the node id, held once per origin rather than once +per session. -Watcher-to-watcher offload needs a tab to forward what it receives, which the -JS origin does not do today: [transit](/quest/m2/p2p/transit.md). - -How a P2P route outranks the relay's is deliberately open. Costs today live -in one scope, a relay mesh pricing its own links; a P2P link is neither free -nor the CDN's egress, and `warm` only accumulates, so a tab forwarding the -relay's route ties the relay and loses on chain length. -[Cost across scopes](/quest/m2/p2p/cost-scopes.md) writes the rule before -the watcher depends on it. +Decided 2026-10-01: a direct link wins only when the peer is the origin or a +routing node; a watching tab never re-serves what it receives. Offloading a +room of watchers is a customer relay's job (`moq-cli --p2p` is the small +one), which keeps warm and cold costs and cache-aware routing out of the +protocol. [Direct peers win](/quest/m2/p2p/cost-scopes.md) sets the cost +knobs and defaults. No Rust + WASM in-tab hop: [rs2ts](/quest/m1/rs2ts/remove-wasm.md) removes the WASM build, so the browser side stays TypeScript. @@ -132,8 +129,7 @@ WASM build, so the browser side stays TypeScript. - [Signaling and policy](/quest/m2/p2p/signal.md) - opted-in peers find each other under the prefix, the application picks who to dial, and the roster-size gate decides whether STUN is used - [Native data channel transport](/quest/m2/p2p/webrtc.md) - `moq-tokio` holds a moq-net session with a browser over str0m with a full ICE agent - [moq-cli joins](/quest/m2/p2p/cli.md) - `--p2p` publishes a roster entry with its iroh endpoint, dials iroh between native peers, and serves browsers as a transit hop -- [Transit in the JS origin](/quest/m2/p2p/transit.md) - a tab forwards the routes it receives to its peers with split horizon and its hop id appended -- [Cost across scopes](/quest/m2/p2p/cost-scopes.md) - the written rule for how relay egress, mesh links, and P2P links compare, so a peer that already carries a broadcast wins +- [Direct peers win](/quest/m2/p2p/cost-scopes.md) - a direct link wins only when the peer is the origin or a routing node, by link costs the node sets itself - [Watch opts in](/quest/m2/p2p/watch.md) - one attribute turns it on in the demo and the watcher migrates to the cheapest route - [Harness](/quest/m2/p2p/harness.md) - the Playwright harness and the numbers behind every mapping decision - [Unordered qmux](/quest/m2/p2p/unordered.md) - qmux tolerates reordering so the data channel runs unordered and a loss stalls one stream diff --git a/quest/m2/p2p/cost-scopes.md b/quest/m2/p2p/cost-scopes.md index fd0a2e5b2c..c6223259a3 100644 --- a/quest/m2/p2p/cost-scopes.md +++ b/quest/m2/p2p/cost-scopes.md @@ -1,52 +1,45 @@ -# [S] Cost across scopes +# [S] Direct peers win ## Goal -A written rule, in `doc/concept/transport.md` and the drafts, for how a route -through a P2P peer compares with the relay's own route, such that a peer -already carrying a broadcast wins and an idle peer never has traffic pulled -through it. The rule is implementable by the Rust and JS origins with the -existing `Cost { warm, cold }` and the per-link price a publisher declares in -SETUP. +A subscriber uses a direct P2P link only when the peer at the other end is +the broadcast's origin or a routing node (a `moq-cli --p2p` hop or a relay +the customer hosts), and the CDN otherwise. A browser never carries another +peer's traffic. The rule is the route layer's ordinary metric over link +costs the node sets itself, with defaults that make a direct link beat the +relay, so a watcher next to the publishing tab pulls from it and everyone +else pulls from the CDN. ## Plan -This is a planning quest; it produces the rule and rewrites the -implementation quests that depend on it. - -What is settled today: a publisher declares an egress price in SETUP on -lite-06 and the receiver charges it per link, with local policy able to -override the peer's declaration; `warm` and `cold` accumulate per link and -compare in that order; a chain containing an anonymous hop loses to any -identified chain; Rust migrates a live subscription only to a route with the -same first hop. - -What is not: costs live in one scope, a mesh pricing its own links. A P2P -link is neither free (peer uplink, reliability) nor the CDN's egress, and -nothing flattens `warm` when an origin already carries the content, so a tab -forwarding the relay's route at the relay's price plus its own ties the relay -and loses on chain length. Candidate rules to weigh, with the numbers from -the [harness](/quest/m2/p2p/harness.md) where they exist: - -- a warm discount: an origin actively receiving a broadcast re-announces it - at `warm` 0 and `cold` unchanged, over every session, in both languages, - which is what the two magnitudes were designed for; -- explicit prices per scope: the relay's client-facing link priced as egress, - the P2P link priced by the application, and a documented comparison between - them; -- a scope tag on the route, so a P2P route is compared with a relay route by - a rule rather than by subtraction. - -Decide who sets the P2P link price (the application, per `Peers` knob), what -the relay's client-facing default is, whether the first-hop migration gate -holds when the first hop is a publisher tab, and whether the warm discount -belongs in the mesh too. Write the rule beside route selection in -`doc/concept/transport.md`, mirror it in `draft-lcurley-moq-lite.md` and -`draft-lcurley-moq-cluster.md` where the wire carries it, and pass -`just drafts check`. Then update [watch](/quest/m2/p2p/watch.md) and -[transit](/quest/m2/p2p/transit.md) with the chosen rule and open the -implementation quest it needs. - -## Related - -- [Cluster routing](/quest/m1/cluster-routing/README.md) - the mesh-side use of cost +Decided 2026-10-01 (moq-dev/moq#4694): + +- No warm or cold cost and no cache state. A peer that merely watches a + broadcast does not offer it to other peers; deduplicating a site's internet + traffic is a customer relay's job, which is a routing node like any other. + This deleted the JS transit quest. +- Link costs are the node's own policy: `Peers` sets the cost of P2P links + and of the relay session, mirrored by `moq-cli --p2p`. The relay declares + nothing per client. Pick defaults so one direct LAN hop beats the relay + route, and a relay route beats two P2P hops through a routing node only + when the application says so. +- The comparison needs the node ids and metrics of + [Routes and announces](/quest/m1/cluster-routing/routes.md): a broadcast's + ANNOUNCE names its origin node, and the roster maps that node to a peer the + application may dial, so an app learns whom to dial from the announce. +- Switching between the peer and the relay splices for an epoch-qualified + path (same source), and is a discontinuity otherwise, per + [Selection](/quest/m1/cluster-routing/selection.md). + +Deliverables: the `Peers` and `moq-cli` cost knobs with their defaults, the +rule written beside route selection in the routing concept page the cluster +routing line adds, and tests with the mock transport: a watcher with a direct +link to the publishing tab uses it and returns to the relay when it drops; a +watcher whose only direct peer is another watcher stays on the relay. + +Public API: cost knobs on `Peers` and `moq-cli`. Wire: none beyond the route +layer. + +## Required + +- [Routes and announces](/quest/m1/cluster-routing/routes.md) - the origin node ids and metrics this compares diff --git a/quest/m2/p2p/harness.md b/quest/m2/p2p/harness.md index 0863d93dd2..383581bfff 100644 --- a/quest/m2/p2p/harness.md +++ b/quest/m2/p2p/harness.md @@ -38,4 +38,3 @@ points. - [moq-cli joins](/quest/m2/p2p/cli.md) - [Signaling and policy](/quest/m2/p2p/signal.md) -- [Transit in the JS origin](/quest/m2/p2p/transit.md) - the watcher-to-watcher row diff --git a/quest/m2/p2p/signal.md b/quest/m2/p2p/signal.md index c088426611..2857a6881c 100644 --- a/quest/m2/p2p/signal.md +++ b/quest/m2/p2p/signal.md @@ -59,8 +59,7 @@ failure is terminal for the pair until either side re-announces; no retry loop. Each established connection goes through `connect` with the supplied transport on the dialing side and `accept` on the answering side, both with `publish: origin.consume()` and `consume: origin`, so the tab serves what it -publishes and, once [transit](/quest/m2/p2p/transit.md) lands, what it -receives. +publishes, never what it receives (a tab is not transit). Trust: publishing under the prefix proves only that the relay admitted the peer to the prefix, not that it may read everything this tab can. Tokens diff --git a/quest/m2/p2p/transit.md b/quest/m2/p2p/transit.md deleted file mode 100644 index cde7482763..0000000000 --- a/quest/m2/p2p/transit.md +++ /dev/null @@ -1,35 +0,0 @@ -# [S] Transit in the JS origin - -## Goal - -A test proves a tab forwards the routes it receives on one session to the -sessions it serves, so a watcher can re-serve a broadcast to a peer and a room -of watchers can cost the relay one egress stream. Split horizon holds: a route -is never announced back to the session it arrived on, and every forwarded -chain carries the tab's hop id. - -## Plan - -Decided in the 2026-09-30 audit: shrink to a test. The Rust origin already -does transit (`with_publisher` and `with_subscriber` splice and share remote -routes), and [Generated lite](/quest/m1/rs2ts/lite.md) replaces the -hand-written JS origin with one generated from it, so there is no JS -forwarding step to write by hand. - -The hop id must be one per origin, not per session, so the roster id from -[signaling](/quest/m2/p2p/signal.md) and the id in forwarded chains agree. - -Tests with the mock transport pair against the generated origin: a route -received on A appears on B with the id appended and never on A; a chain -already containing the id or at `MAX_HOPS` is dropped; a retraction on A -retracts on B; two watchers of one tab share one upstream subscription. If a -case fails, the fix goes in the Rust origin. - -## Required - -- [Generated lite](/quest/m1/rs2ts/lite.md) - the JS origin generated from moq-net, which already does transit - -## Related - -- [Watch opts in](/quest/m2/p2p/watch.md) - the first topology that needs a forwarding tab -- [Cost across scopes](/quest/m2/p2p/cost-scopes.md) - adds the receiving link's cost to forwarded routes diff --git a/quest/m2/p2p/watch.md b/quest/m2/p2p/watch.md index 8307f6c064..631b74ef2a 100644 --- a/quest/m2/p2p/watch.md +++ b/quest/m2/p2p/watch.md @@ -3,27 +3,24 @@ ## Goal One attribute turns P2P on in the demo. A watcher tab keeps subscribing -through the shared origin, its subscriptions migrate to a peer when the -origin ranks that route cheaper, and they return to the relay when the peer -goes away, with no visible interruption. +through the shared origin, its subscriptions migrate to a direct peer that is +the broadcast's origin or a routing node, and they return to the relay when +the peer goes away, with no visible interruption. ## Plan `hang-watch` and `hang-publish` gain a `p2p` attribute that constructs `Peers` on the shared connection's origin with the demo's ICE servers and a `max` from the page; `demo/web` exposes the toggle. The route pick is the -origin's, from its cost ranking (`compareRoutes` in `js/net/src/origin.ts`) under the rule from -[cost across scopes](/quest/m2/p2p/cost-scopes.md): a peer already carrying -the broadcast wins, and its retraction falls back to the relay. +origin's, under the rule from [Direct peers win](/quest/m2/p2p/cost-scopes.md): +a direct link to the origin or a routing node wins, and losing it falls back +to the relay. -Three topologies and the demo shows all of them: a publishing tab serving -watcher tabs directly, watcher tabs pulling from a `moq-cli --p2p` hop, and a -watcher tab re-serving to another watcher through -[transit](/quest/m2/p2p/transit.md). +Two topologies and the demo shows both: a publishing tab serving watcher tabs +directly, and watcher tabs pulling from a `moq-cli --p2p` hop. ## Required - [Signaling and policy](/quest/m2/p2p/signal.md) - [moq-cli joins](/quest/m2/p2p/cli.md) - the native hop the second topology shows -- [Transit in the JS origin](/quest/m2/p2p/transit.md) -- [Cost across scopes](/quest/m2/p2p/cost-scopes.md) +- [Direct peers win](/quest/m2/p2p/cost-scopes.md) From 6e6d04315a121200ab9f7ed1f6e5026f4637f49b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 1 Oct 2026 17:35:36 -0700 Subject: [PATCH 4/4] quest: address review on the route split (seed cost, +1 per hop, down-only in UPDATE, lazy ROUTEs) Co-Authored-By: Claude Opus 5.5 --- quest/m1/auth/peer-grant.md | 1 + quest/m1/cluster-routing/README.md | 5 +++- quest/m1/cluster-routing/routes.md | 43 +++++++++++++++++++++++++----- quest/m1/cluster-routing/sim.md | 4 ++- quest/m1/route-cost.md | 3 +++ 5 files changed, 47 insertions(+), 9 deletions(-) diff --git a/quest/m1/auth/peer-grant.md b/quest/m1/auth/peer-grant.md index bdf539b257..0641129a7c 100644 --- a/quest/m1/auth/peer-grant.md +++ b/quest/m1/auth/peer-grant.md @@ -67,3 +67,4 @@ Additive. ## Related - [Signaling and policy](/quest/m2/p2p/signal.md) - the first consumer +- [Route trust](/quest/m1/cluster-routing/route-trust.md) - extends the grant with the node ids its holder may advertise routes for, a second purpose beside serving paths diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index a96900f2eb..63a54c88c0 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -69,7 +69,10 @@ vector the 2026-09-30 cache-tiers audit kept: primary, Cloudflare secondary) ahead of the metric, so metrics from different operators are never compared; a CDN that speaks no ROUTE is just a link. See [Multi-CDN endpoints](/quest/m1/cluster-routing/multi-cdn.md). -- **Metric on the wire, cost policy local.** One additive metric. Static +- **Metric on the wire, cost policy local.** One additive metric on ROUTE, + where each hop adds its link cost plus one so it strictly increases; the + origin's per-prefix cost rides ANNOUNCE untouched (`transcode/**` at 10 and + `transcode/foobar` at 1 from one node). Static configured costs stay the default (the CDN); a radio link may measure its own ([Link quality](/quest/m1/cluster-routing/link-quality.md)). - **Topology now, wire later.** moq.pro's edge and core migration runs on diff --git a/quest/m1/cluster-routing/routes.md b/quest/m1/cluster-routing/routes.md index b3cd5b3f6d..511beded23 100644 --- a/quest/m1/cluster-routing/routes.md +++ b/quest/m1/cluster-routing/routes.md @@ -18,18 +18,38 @@ Decided 2026-10-01 (see the [line's decisions](/quest/m1/cluster-routing/README. - Wire sketch, flexible in detail. ROUTE_START carries the node id, a seqno, the metric, and a down-only bit, and implicitly takes the next stream-local Route ID, as ANNOUNCE_START takes the next Announce ID today. ROUTE_UPDATE - (Route ID, seqno, metric) and ROUTE_END (Route ID) reference it. + (Route ID, seqno, metric, down-only bit) and ROUTE_END (Route ID) reference + it; the bit rides UPDATE so a route moving between a mesh path and a CDN + fallback changes it without retracting its announces. ANNOUNCE_START carries the prefix (keeping today's Path Base/Keep - compression) and a Route ID. An ANNOUNCE naming an unknown Route ID is a - protocol violation, and ROUTE_END ends that stream's ANNOUNCEs on the - route. The hop list and per-announce cost leave this version. + compression), a Route ID, and the origin's cost for that prefix. + ANNOUNCE_UPDATE re-prices it and ANNOUNCE_END ends one broadcast while the + route stays up. An announce never changes its Route ID: another origin + serving the same path is another ANNOUNCE. An ANNOUNCE naming an unknown + Route ID is a protocol violation, and ROUTE_END ends that stream's + ANNOUNCEs on the route. The hop list leaves this version. ```text ROUTE_START node=0x7a3f seqno=41 metric=12 -> route 0 - ANNOUNCE_START prefix=drone/cam1 route=0 -> announce 0 + ANNOUNCE_START prefix=transcode/ route=0 cost=10 -> announce 0 + ANNOUNCE_START prefix=transcode/foobar route=0 cost=1 -> announce 1 ROUTE_UPDATE route=0 seqno=42 metric=15 + ANNOUNCE_END announce=1 # one broadcast ends ROUTE_END route=0 # ends announce 0 too ``` +- A stream carries a ROUTE only while some ANNOUNCE on it references it: sent + just before the first, ended after the last. That bounds a client session + to the origins of what it asked for (announce interest already scopes it), + not the CDN's whole node table. +- The announce cost is the origin's per-prefix seed (`transcode/**` at 10 and + `transcode/foobar` at 1 from one node, decided 2026-10-01) and nobody + changes it in transit; a path's cost through a route is that seed plus the + route metric. Link changes therefore touch only ROUTE, and feasibility runs + on the route metric alone. This is [One route cost](/quest/m1/route-cost.md)'s + single cost, which stops accumulating per hop once this lands. +- Each hop adds its link cost plus one to the metric (decided 2026-10-01), so + the metric strictly increases as Babel requires while operators keep + configuring cost 0 for a free link. - The node id is global and opaque: a relay's `cluster.id` or a random id, and an app's handshake id. It is needed so routes from two neighbours to one origin are recognized as one, which carries loop freedom, @@ -58,14 +78,23 @@ Decided 2026-10-01 (see the [line's decisions](/quest/m1/cluster-routing/README. - Mixed versions: a lite-06 peer keeps today's path vector, translated at the relay that speaks both, for the rollout window only. +Why not a hold-down: #4644 retracted at once and re-announced a replacement +after 1 s, which ended a downstream front whose only path ran through the +relay, dropping subscribers for about a second while the relay kept serving; +holding without retracting brought hunting back (moq-dev/moq.pro#2116 has the +measurements, and #4642's 300 ms cursor hold is the production mitigation). +Separating "the path changed" (ROUTE) from "the broadcast ended" (ANNOUNCE) +removes that trade, and the failover tests must show no front loses its +route while a working one exists. Decide whether #4642's hold can then go. + Open, for the implementer to settle and record: - Seqno lifetime across restarts of a node with a configured stable id (persist it, fold an incarnation into the id, or Babel's seqno request). - Whether starvation recovery needs a request message, or session restart covers it. -- The window where an origin's END and a next-hop change cross in flight, - and whether it needs more than the next END to arrive. +- The window where an origin's ANNOUNCE_END and a next-hop change cross in + flight, and whether it needs more than the next END to arrive. - How much of today's route trie, fronts, and `route_order` survives intact. Tests, time mocked: #4644's live-graph withdrawal and failover tests diff --git a/quest/m1/cluster-routing/sim.md b/quest/m1/cluster-routing/sim.md index b013d47433..d474d82282 100644 --- a/quest/m1/cluster-routing/sim.md +++ b/quest/m1/cluster-routing/sim.md @@ -13,7 +13,9 @@ Per scenario (publish, end, link flap, relay loss and restart, uplink loss): messages and bytes by kind, convergence time, stale-path seconds, transient subscribe loops, and whether a two-uplink mesh ever carries CDN traffic. -To check: look for that report in a merged moq.pro PR. To advance: ask the +moq-dev/moq.pro#2136 rewrites that quest for the split; until it merges the +moq.pro quest still asks the older path-vector question. To check: look for +the report in a merged moq.pro PR. To advance: land #2136, then ask the maintainer to start the moq.pro quest. Delete this quest once the report is in and [Routes and announces](/quest/m1/cluster-routing/routes.md) is confirmed or revised by it. diff --git a/quest/m1/route-cost.md b/quest/m1/route-cost.md index e058957d87..b5bddafe6e 100644 --- a/quest/m1/route-cost.md +++ b/quest/m1/route-cost.md @@ -24,6 +24,9 @@ state, so Warm has no job. - Replace `Cost { warm, cold }` in `rs/moq-net/src/model/origin.rs` and its JS mirror, and delete what computes Warm from cache state. - The moq-transport cluster extension already carries one cost. +- Once [Routes and announces](/quest/m1/cluster-routing/routes.md) lands, this + single cost stays on ANNOUNCE as the origin's per-prefix seed and stops + accumulating per hop; link costs move to the ROUTE metric. Public API: `Cost` changes shape (a break, so it lands on `dev`). Wire: the wip version drops a field; lite-06 is unchanged.