quest: plan cluster routing, update dedupe, and the local origin #4213
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # [S] Skip unchanged announce updates | ||
|
|
||
| ## Goal | ||
|
|
||
| A publisher sends an announce update only when what the peer would decode | ||
| differs from what it last sent for that announcement. A local change the wire | ||
| cannot express (the route's source session, `served`, captures) sends nothing. | ||
| This applies on every version, lite and IETF, in Rust and JS, and it is wire | ||
| compatible. | ||
|
|
||
| ## Plan | ||
|
|
||
| Each announce cursor dedupes its best route on `(hops, cost, source)` plus | ||
| `served` and captures (`rs/moq-net/src/model/origin.rs`, the `Updated` kind), | ||
| but the lite publisher only remembers each suffix's Announce ID | ||
| (`rs/moq-net/src/lite/publisher.rs`, the `self.live` branch) and re-sends | ||
| `Restart` for every `Updated` it sees. Only the hops and cost reach the wire, | ||
| so a flip of any other field sends an identical update, to every peer, for | ||
| every covered broadcast. This comes from reading the code, not from a test. | ||
|
|
||
| - Reproduce first: a test that flips a route's source session with the same | ||
| hops and cost, and asserts the peer receives no update. | ||
| - Keep the last-sent `(hops, cost)` beside the Announce ID and skip equal | ||
| updates. Check the IETF publisher's re-pricing path and the JS publisher for | ||
|
Comment on lines
+21
to
+24
There was a problem hiding this comment.
On legacy Lite versions, storing the full Useful? React with 👍 / 👎. |
||
| the same pattern. | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| - Fix the stale comments on the way: `lite/announce.rs` says restarts are only | ||
| ever received, and the `restart_announce` doc in `lite/subscriber.rs` says it | ||
| compares the first hop. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Cluster routing](/quest/m1/cluster-routing.md) - removes the other big source of updates, reroutes that change only the hop chain | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| # [M] Local origin | ||
|
|
||
| ## Goal | ||
|
|
||
| A localhost worker (a Voice agent, a recorder) connects to the relay's | ||
| internal listener and gets a moq session whose origin holds only the | ||
| broadcasts this relay ingested from its own customer sessions, never those | ||
| learned from cluster peers. Workers stop electing on hop chains ("the first | ||
| internal hop is me"), which [Cluster routing](/quest/m1/cluster-routing.md) | ||
| removes inside a cluster. | ||
|
|
||
| ## Plan | ||
|
|
||
| - The view exists: `origin::Consumer::local()` (`rs/moq-net/src/model/origin.rs`) | ||
| hides every route a `Producer::peer` handle announced, which is how cluster | ||
| peers attach (`rs/moq-relay/src/cluster.rs`), and | ||
| `local_view_hides_peer_routes` covers it. The work is serving that view. | ||
| - The internal listener (`rs/moq-relay/src/internal.rs`) is plain HTTP | ||
| today (`/metrics`, `/health`, `/nodes`, `/sessions`). Serve a moq session on | ||
| it over the WebSocket transport the relay already accepts. It is | ||
| unauthenticated, so serve it only when the internal listener binds a | ||
| loopback address. The listener may also bind a private-overlay address, | ||
| which would expose customer media to the overlay; config load fails if the | ||
| local origin is enabled there. | ||
| - The session is read-only. Open question: Voice publishes responses. Decide | ||
| whether this session may publish into the relay origin, and under which | ||
| grant, before it accepts a publish. | ||
| - Document it in `doc/bin/relay/`. | ||
|
|
||
| Tests: a two-relay cluster where each relay's local session lists only its own | ||
| publishers, a publisher moving relays moves between the two views, and a | ||
| peer-learned route never appears. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Voice on the local origin](https://github.com/moq-dev/moq.pro/blob/main/quest/m0/voice-local-origin.md) - moq.pro's Voice and recorder switch to this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| # [XL] Cluster routing | ||
|
|
||
| ## Goal | ||
|
|
||
| A broadcast event reaches each relay at most once, and a relay learns only the | ||
| prefixes its own clients asked for. An announcement says a path exists at an | ||
| origin relay, at a cost; how to reach that origin comes from a shared relay | ||
| topology, so no announcement inside a cluster carries a hop list. This quest | ||
| records the design; its wire and implementation quests are planned from the | ||
| simulator's report. | ||
|
|
||
| Non-goals: warm re-origination (a warm relay would be one more origin with a | ||
| cost, so leave room for it), and a permanently mixed-version cluster. | ||
|
|
||
| ## Plan | ||
|
|
||
| ### Why not path vector or Babel | ||
|
|
||
| Today every relay advertises its best route to every peer not already in the | ||
| hop chain. One publish costs about R·(d-1) announces for R relays of mesh | ||
| degree d, every relay learns every broadcast (`.stats` and `.internal` | ||
| included), and a link change rewrites every route crossing it. On moq.pro's | ||
| live fleet (26 PoPs, average degree about 5) each relay receives every event | ||
| about five times. Babel ([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966)) | ||
| shrinks the message and skips equal-cost reroutes, but it is still distance | ||
| vector: the same fan-out, the same global knowledge, and no loop freedom | ||
| when several sources claim a prefix (section 2.7), which pools and wildcards | ||
| make MoQ's common case. | ||
|
|
||
| ### Decisions | ||
|
|
||
| - Existence is split from reachability. An announcement carries the path, its | ||
| origin relay, and the origin's cost, and nothing about the path to it. | ||
| - An existence event carries the origin's seqno, scoped to its incarnation. A | ||
| relay applies an event only when it is newer than the last it applied for | ||
| that path and origin, and keeps an ended path's seqno, so a start delayed on | ||
| a stale tree or a failed-over registry cannot revive it. Babel keeps | ||
| feasibility past withdrawal for the same reason | ||
| ([RFC 8966 section 3.7.3](https://www.rfc-editor.org/rfc/rfc8966#section-3.7.3)). | ||
| - The topology is configured: `--cluster-connect` or the connect API gives the | ||
| relay graph and link costs. Relays flood per-link liveness among themselves | ||
| with a per-link seqno. The seqno is scoped to the relay's incarnation, so a | ||
| restarted relay's links supersede its stale ones instead of looking older. | ||
| Gossip discovery (`cluster.mesh`) stays for zero-config self-hosting and | ||
| derives the topology from what it discovers; it need not scale. | ||
| - A relay picks the origin with the lowest shortest-path distance plus origin | ||
| cost, ties broken by rendezvous hashing (HRW) of the requested path and the | ||
| origin id, and forwards along its shortest path. Distance compares cost, | ||
| then hop count, so every hop strictly shortens it even across `?cost=0` | ||
| links. That is a shortest path to a virtual node linked to every origin, so | ||
| it is loop-free whenever relays agree on the topology. Specificity still | ||
| ranks first, per [Wildcard](/quest/m0/wildcard/README.md). | ||
| - The first relay's choice rides the SUBSCRIBE, and transit relays forward | ||
| toward that origin by topology alone, never re-selecting. Re-selection | ||
| against another existence view loops: a relay that lost a specific claim | ||
| falls back to a broader one through a relay still routing to the specific | ||
| one ([RFC 8966 section 3.5.4](https://www.rfc-editor.org/rfc/rfc8966#section-3.5.4)). | ||
| If the origin no longer serves the path, it refuses, and the first relay | ||
| selects again. | ||
| - SUBSCRIBE and FETCH carry a visited-relay list end to end. It catches loops | ||
| while liveness views disagree and names the path for stats. Narrowing it to | ||
| cluster hops is later work. The serving origin's identity rides the reply, | ||
| per Wildcard's Spread quest. | ||
| - Announcements are on demand. A relay forwards only the union of its clients' | ||
| ANNOUNCE_REQUEST prefixes, never the empty prefix. A wide prefix that many | ||
| edges' viewers request is that customer's cost. `.stats` becomes ordinary | ||
| demand. | ||
| - Registries are an optional, configured tier: moq-relay in a registry mode, | ||
| one or more per region. | ||
| - An ingest relay registers its broadcasts with its nearest registry, and an | ||
| edge sends its ANNOUNCE_REQUEST there. | ||
| - Registries form a small full mesh and flood existence among themselves, so | ||
| an event crosses an ocean once per remote registry, not once per relay. | ||
| Announce latency is about one round trip to the nearest registry, | ||
| whatever the path length. | ||
| - A relay fails over to the next-nearest registry and reconciles its view | ||
| instead of treating the lost session as ends, so a registry failure never | ||
| reports a live broadcast offline. | ||
| - With no registry reachable, a relay freezes: it keeps its view, learns | ||
| nothing new, and alerts. Falling back to flooding would cascade the | ||
| failure. | ||
| - Without registries (self-hosting), existence floods along the shortest-path | ||
| tree, one copy per relay. A relay forwards an event only when it changes its | ||
| view, so a duplicate copy, from trees built on disagreeing liveness, stops | ||
| there. | ||
| - Between clusters, announcements stay path vector with cluster ids as the | ||
| hops, like BGP between autonomous systems. A customer's on-prem cluster is | ||
| one hop, and an announcement naming the receiving cluster is dropped. | ||
| - The cluster switches versions as a whole; older lite and IETF sessions stay | ||
| at its edges. | ||
|
|
||
| ### Open questions | ||
|
|
||
| - Sharding registries by HRW over a prefix key once one registry cannot hold | ||
| everything, and what that key is. | ||
| - A mixed-version bridge, if a fleet cannot switch at once. | ||
| - How long a relay keeps an ended path's seqno. A new origin incarnation | ||
| clears it; within one, it must outlive every delayed copy of the start. | ||
| - How an edge routes a SUBSCRIBE for a path none of its clients asked to | ||
| announce. It holds no route for it, and asking a registry first adds a round | ||
| trip before the first byte. | ||
| - What a cold ANNOUNCE_REQUEST reports as live. Answering from the local view | ||
| keeps a relay from waiting on peers but reports an empty set until the | ||
| registry's replay lands, on every new prefix rather than in a rare race. | ||
| - What remains of announce compression's hop-tail half (`Hop Base` and | ||
| `Hop Keep` in the lite draft) once only cluster boundaries carry hops. | ||
|
|
||
| ## Required | ||
|
|
||
| - moq.pro's routing simulator reports ([quest](https://github.com/moq-dev/moq.pro/blob/main/quest/m1/routing-simulator.md)) | ||
|
|
||
| ## Related | ||
|
|
||
| - [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - cuts duplicate updates on today's routing | ||
| - [Local origin](/quest/m0/local-origin.md) - workers stop reading hop chains before they go | ||
| - [Wildcard](/quest/m0/wildcard/README.md) - the specificity, pool spread, and reply identity this selection builds on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Promoting Wildcard to m0 makes its Resolve child appear ready even though the line explicitly relies on the m1 Path patterns quest to add versioned publish patterns and the containment semantics used for authorization. Since neither the line nor Resolve lists that quest under
Required,quest readycan direct work to start before its required representation exists; add the dependency so the promoted line cannot be scheduled prematurely. (Written by GPT-5.6 Sol)AGENTS.md reference: quest/AGENTS.md:L63-L66
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Declining: the dependency is gone on the line branch (#4037). Resolve and Demand already landed there, only Spread remains, and the matcher exists in
rs/moq-pattern(used by moq-net and moq-auth). Path patterns stays Related, not Required.(Written by Claude Opus 5.5)