From 3a3fe029a307be4f8fcd49760783b2c827deafb8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 16:39:39 -0700 Subject: [PATCH 1/5] quest: plan announcement spam reduction (Babel routing, counters, update dedupe) Promote the wildcard line to m0 and add Babel routing, announce counters, unchanged-update dedupe, and the relay's local origin for localhost workers. Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 5 ++ quest/m0/announce-counters.md | 35 +++++++++++ quest/m0/announce-update-dedupe.md | 33 ++++++++++ quest/m0/babel/README.md | 94 ++++++++++++++++++++++++++++ quest/m0/babel/js.md | 24 +++++++ quest/m0/babel/rust.md | 40 ++++++++++++ quest/m0/local-origin.md | 31 +++++++++ quest/{m1 => m0}/wildcard/README.md | 8 +-- quest/{m1 => m0}/wildcard/demand.md | 4 +- quest/{m1 => m0}/wildcard/resolve.md | 0 quest/m1/README.md | 1 - quest/m1/archive/README.md | 2 +- quest/m1/broadcast-epoch/README.md | 2 +- quest/m1/path-patterns.md | 2 +- quest/m1/pop-skipping/README.md | 2 +- quest/m1/processor/README.md | 2 +- 16 files changed, 273 insertions(+), 12 deletions(-) create mode 100644 quest/m0/announce-counters.md create mode 100644 quest/m0/announce-update-dedupe.md create mode 100644 quest/m0/babel/README.md create mode 100644 quest/m0/babel/js.md create mode 100644 quest/m0/babel/rust.md create mode 100644 quest/m0/local-origin.md rename quest/{m1 => m0}/wildcard/README.md (98%) rename quest/{m1 => m0}/wildcard/demand.md (94%) rename quest/{m1 => m0}/wildcard/resolve.md (100%) diff --git a/quest/m0/README.md b/quest/m0/README.md index a37d03ba03..4c9a3eebdf 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -94,6 +94,11 @@ do not add another media abstraction or a renderer crate during stabilization. ## Quests - [Release](/quest/m0/release.md) - the release moq.pro adopts: binding docs, an upgrade page, and a staging soak gate it rather than the merge +- [Announce counters](/quest/m0/announce-counters.md) - relay `/metrics` counts announce starts, ends, updates, and wire bytes per tier, so announce traffic is measurable +- [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - a publisher sends an announce update only when the wire route changed +- [Wildcard](/quest/m0/wildcard/README.md) - a relay resolves subscriptions against advertised prefixes, a service claims the prefix it could serve and refuses the rest instead of enumerating broadcasts, and the browser player treats a covering claim as availability +- [Babel routing](/quest/m0/babel/README.md) - lite-07 announcements drop the hop list and stay loop-free by source, seqno, and cost, so reroutes stop flooding the mesh +- [Local origin](/quest/m0/local-origin.md) - localhost workers read only the broadcasts their relay ingested, from the internal listener - [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the audio playout target is a measured estimate of arrival timing in both languages, not a round-trip guess - [A/V clock](/quest/m0/plan-av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles - [SD rendition for bbb](/quest/m0/bbb-sd.md) - `just pub bbb` publishes a pre-encoded 360p rung beside the 720p source, for localhost demos and moq.pro's fleet demo diff --git a/quest/m0/announce-counters.md b/quest/m0/announce-counters.md new file mode 100644 index 0000000000..e307f32ffb --- /dev/null +++ b/quest/m0/announce-counters.md @@ -0,0 +1,35 @@ +# [S] Announce counters + +## Goal + +A relay's `/metrics` counts its announcement traffic per stats tier (customer +and cluster) and direction: starts, ends, and in-place updates, plus the +encoded bytes its announce streams carry. An operator can then tell announce +traffic apart from subscribe and stats traffic, which live cannot today. + +## Plan + +On moq.pro's live fleet (2026-09-25), 93% of relay egress was not media (about +15 MB/s), and it tracked mesh degree rather than customer load. No announce +series exists anywhere, so whether announces drive it is unknown. Subscriptions +are already counted (`moq_relay_subscriptions_opened_total` in +`rs/moq-relay/src/internal.rs`); add the announce counters beside it, with the +same tier labels. + +- Count at the lite and IETF announce writers and readers in `rs/moq-net`, into + the stats model the session already reports through, so the relay only + exports them. +- Bytes are the encoded announce-stream bytes, not `announced_bytes` (which sums + raw path lengths and stays invariant under compression). Otherwise the + counters cannot show what [Announce compression](/quest/m1/announce-compression.md) + or [Babel routing](/quest/m0/babel/README.md) save. +- Document the series in `doc/bin/relay/`. + +Tests: a two-relay cluster with one publisher counts one start per hop on +publish, one update per reroute, and one end on unpublish, each on the right +tier. + +## Related + +- [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - the first saving these counters should show +- [Fleet announce panels](https://github.com/moq-dev/moq.pro/blob/main/quest/m0/announce-panels.md) - charts these on live and records the baseline diff --git a/quest/m0/announce-update-dedupe.md b/quest/m0/announce-update-dedupe.md new file mode 100644 index 0000000000..fd566d5968 --- /dev/null +++ b/quest/m0/announce-update-dedupe.md @@ -0,0 +1,33 @@ +# [S] Skip unchanged announce updates + +## Goal + +A publisher sends an announce update only when what the peer would decode +differs from what it last sent for that announcement. A local change the wire +cannot express (the route's source session, `served`, captures) sends nothing. +This applies on every version, lite and IETF, in Rust and JS, and it is wire +compatible. + +## Plan + +Each announce cursor dedupes its best route on `(hops, cost, source)` plus +`served` and captures (`rs/moq-net/src/model/origin.rs`, the `Updated` kind), +but the lite publisher only remembers each suffix's Announce ID +(`rs/moq-net/src/lite/publisher.rs`, the `self.live` branch) and re-sends +`Restart` for every `Updated` it sees. Only the hops and cost reach the wire, +so a flip of any other field sends an identical update, to every peer, for +every covered broadcast. This comes from reading the code, not from a test. + +- Reproduce first: a test that flips a route's source session with the same + hops and cost, and asserts the peer receives no update. +- Keep the last-sent `(hops, cost)` beside the Announce ID and skip equal + updates. Check the IETF publisher's re-pricing path and the JS publisher for + the same pattern. +- Fix the stale comments on the way: `lite/announce.rs` says restarts are only + ever received, and the `restart_announce` doc in `lite/subscriber.rs` says it + compares the first hop. + +## Related + +- [Announce counters](/quest/m0/announce-counters.md) - shows the saving on a live fleet +- [Babel routing](/quest/m0/babel/README.md) - removes the other big source of updates, reroutes that change only the hop chain diff --git a/quest/m0/babel/README.md b/quest/m0/babel/README.md new file mode 100644 index 0000000000..571bc7fd13 --- /dev/null +++ b/quest/m0/babel/README.md @@ -0,0 +1,94 @@ +# Babel routing + +## Goal + +moq-lite-07 announcements drop the hop list. A route carries a source id, a +per-route sequence number, and its warm and cold cost, and relays stay +loop-free with Babel's feasibility condition +([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966) section 3.5) instead of +path inspection. A reroute that leaves a relay's advertised cost unchanged no +longer propagates, and every announce stops paying for its path. lite-06 and +older, and the IETF cluster extension, keep today's hop lists. + +Non-goals: fewer routes per relay (tree routing was dropped), and a +permanently mixed mesh. Mixing pre-07 and lite-07 relays is a rollout window. + +## Plan + +Today the hop list does three jobs: loop prevention (a relay drops a path +containing itself), request exclusion (never advertise or serve through the +requester, anywhere in the chain), and stitching identity (the first hop). Any +change to the chain emits an update, and a relay forwards that update to every +peer that isn't excluded. [Wildcard](/quest/m0/wildcard/README.md)'s Spread +quest moves identity onto the subscribe and fetch reply, so this line only +replaces the first two jobs. + +### Wire (lite-07, still `moq-lite-07-wip`) + +- ANNOUNCE_START carries `Source ID`, `Seqno`, and the warm and cold costs in + place of `Hops`. The source is fixed for the life of an Announce ID: a new + source is END and START. +- ANNOUNCE_UPDATE carries `Seqno` and costs. +- New ANNOUNCE_REFRESH (subscriber to publisher, on the announce stream): + `Announce ID` and the wanted `Seqno`. +- An anonymous flag replaces the 0 Hop ID, so anonymous routes still rank last. + The encoding is the implementer's call. +- The Cost Parameter's link cost is at least 1 and the wire cannot express 0 + (encode it minus one). Every hop then strictly raises both costs, which + feasibility needs. +- [Announce compression](/quest/m1/announce-compression.md) lands first; this + deletes its hop-tail half (`Hop Base`/`Hop Keep`) and keeps path compression. + +### Routing + +- A seqno covers one route: a source keeps one per announced prefix, so a + refresh re-announces only the route that starved. +- A relay keeps a feasibility distance per (source, prefix): the best + `(seqno, warm, cold)` it has advertised. It accepts a route only with a newer + seqno, or the same seqno and a lexicographically lower cost. A saturated cost + cannot drop, so it is infeasible, which is safe. +- A relay with no feasible route sends ANNOUNCE_REFRESH on the streams offering + an infeasible one. A publisher that already holds a new enough seqno + re-announces. Otherwise it forwards the request toward its own upstream, and + the source bumps the route's seqno. At most one refresh per route per + upstream is outstanding. +- Exclusion is adjacent-only: never advertise or serve a route back to the + peer it arrived from. Deep exclusion goes with the chain. +- Selection keeps specificity, then anonymity, then warm and cold cost. The + shortest-path tie-break goes away, since cost already grows every hop. +- An ingest relay mints a random source id for a publisher that declares none + (lite-02/03, IETF without the cluster extension) and marks it anonymous. +- [Warm advertise](/quest/m1/pop-skipping/warm-advertise.md) re-originates the + exact path as the relay's own source, so its warm-zero price is compatible. + +### Bridge + +Routes from a pre-07 or IETF peer take `hops[0]` (or a minted id) as source. +Routes sent to one carry a hop list the bridge synthesizes. The bridge may be +conservative, but it must never hold a persistent loop. + +### Line work + +This README owns the draft and the end-to-end proof: + +- Lite draft Routing, ANNOUNCE_START/UPDATE/REFRESH, Cost Parameter, and the + lite-07 changelog. +- `doc/concept/moq-lite.md`. +- A mixed ring of pre-07 and lite-07 relays that converges with no persistent + loop, fails over when a mid-path relay dies, and recovers a starved route + through REFRESH. + +## Quests + +- [Rust routing](/quest/m0/babel/rust.md) - moq-net's route model, the lite-07 codec, ANNOUNCE_REFRESH, and the pre-07/IETF bridge +- [JS codec](/quest/m0/babel/js.md) - `@moq/net` speaks the lite-07 route fields and answers ANNOUNCE_REFRESH as a source + +## Required + +- [Wildcard](/quest/m0/wildcard/README.md) - its Spread quest moves stitching identity onto the reply, which this line stops carrying in announcements + +## Related + +- [Local origin](/quest/m0/local-origin.md) - gives localhost workers locally ingested broadcasts without reading hop chains +- [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - the other half of cutting update churn +- [Announce counters](/quest/m0/announce-counters.md) - measures the saving diff --git a/quest/m0/babel/js.md b/quest/m0/babel/js.md new file mode 100644 index 0000000000..e1dc2b9526 --- /dev/null +++ b/quest/m0/babel/js.md @@ -0,0 +1,24 @@ +# [S] JS codec + +## Goal + +`@moq/net` decodes and encodes the lite-07 route fields from the +[Babel routing](/quest/m0/babel/README.md) line. As a publisher it answers +ANNOUNCE_REFRESH by bumping the route's seqno, and it tells a reroute from a +takeover without reading a hop list. + +## Plan + +- `js/net/src/lite/`: ANNOUNCE_START/UPDATE fields and ANNOUNCE_REFRESH. +- Every connection mints a random hop today (`lite/connection.ts`). The source + id is per origin instead, so a reconnecting browser publisher keeps its + identity. +- `lite/subscriber.ts` separates a reroute from a takeover with + `hops[0] ?? responderOrigin`. Key it on the source id, or on the reply's + origin once Spread lands. +- The tie-break on hop count in `origin.ts` goes away. +- Interop: `just test interop --all` against the Rust side. + +## Required + +- [Rust routing](/quest/m0/babel/rust.md) - interop needs the Rust side of the wire diff --git a/quest/m0/babel/rust.md b/quest/m0/babel/rust.md new file mode 100644 index 0000000000..104650b10d --- /dev/null +++ b/quest/m0/babel/rust.md @@ -0,0 +1,40 @@ +# [L] Rust routing + +## Goal + +`moq-net` routes lite-07 announcements by source, per-route seqno, and cost, +under the feasibility condition and adjacent-only exclusion that the +[Babel routing](/quest/m0/babel/README.md) line specifies. Pre-07 and IETF +sessions keep hop lists through the bridge. + +## Plan + +- `rs/moq-net/src/model/origin.rs`: `Route` gains source, seqno, and the + anonymous flag. `route_order` drops the hop-length and `fnv_key(prefix, + hops)` tie-breaks. `RouteEntry::visible_to` excludes only the adjacent peer, + so `Horizon` narrows to that peer. The feasibility table lives beside the + routes and garbage-collects entries for sources with no live route. +- `rs/moq-net/src/lite/`: the codec for the new ANNOUNCE_START/UPDATE fields + and ANNOUNCE_REFRESH. The publisher answers or forwards refreshes; the + subscriber applies feasibility and sends refreshes when starved. It builds on + the stateful codec from [Announce compression](/quest/m1/announce-compression.md) + and deletes its hop-tail half. +- Cost Parameter: on lite-07 the link cost is at least 1 by construction. +- Bridge: pre-07 lite and IETF (`ietf/cluster.rs`) sessions map hops to and + from source routes. +- Consumers: the relay's `/nodes` (`rs/moq-relay/src/nodes.rs`) reports source + and cost in place of the path. `Route` is exposed through `moq-ffi` and + `libmoq`, so this is a published API break: retarget to `dev` if `hops` has + to leave the public type. + +Tests: codec round-trips and violations, feasibility accept/refuse, a starved +relay recovering through REFRESH, adjacent exclusion on both advertise and +serve, and anonymous minting. + +Benchmark: announce messages and bytes per publish, reroute, and relay loss +on a simulated mesh, swept over relay count and degree, lite-06 against +lite-07. + +## Related + +- [JS codec](/quest/m0/babel/js.md) - the browser side of the same wire diff --git a/quest/m0/local-origin.md b/quest/m0/local-origin.md new file mode 100644 index 0000000000..620cd0e964 --- /dev/null +++ b/quest/m0/local-origin.md @@ -0,0 +1,31 @@ +# [M] Local origin + +## Goal + +A localhost worker (a Voice agent, a recorder) connects to the relay's +internal listener and gets a moq session whose origin holds only the +broadcasts this relay ingested from its own customer sessions, never those +learned from cluster peers. Workers stop electing on hop chains ("the first +internal hop is me"), which [Babel routing](/quest/m0/babel/README.md) removes. + +## Plan + +- The relay already knows each route's arriving session and its tier: cluster + peers attach through `origin.peer()` (`rs/moq-relay/src/cluster.rs`). Build + an `origin::Consumer` that admits only routes from customer-tier sessions, + without copying the table. +- The internal listener (`rs/moq-relay/src/internal.rs`) is localhost HTTP + today (`/metrics`, `/health`, `/nodes`, `/sessions`). Serve a moq session on + it over the WebSocket transport the relay already accepts. It is + unauthenticated, like `/sessions`, because only local processes reach it. +- Open question: Voice publishes responses. Decide whether this session + may publish into the relay origin, and under which grant. +- Document it in `doc/bin/relay/`. + +Tests: a two-relay cluster where each relay's local session lists only its own +publishers, a publisher moving relays moves between the two views, and a +peer-learned route never appears. + +## Related + +- [Voice on the local origin](https://github.com/moq-dev/moq.pro/blob/main/quest/m0/voice-local-origin.md) - moq.pro's Voice and recorder switch to this diff --git a/quest/m1/wildcard/README.md b/quest/m0/wildcard/README.md similarity index 98% rename from quest/m1/wildcard/README.md rename to quest/m0/wildcard/README.md index e7be970d69..13795b6eff 100644 --- a/quest/m1/wildcard/README.md +++ b/quest/m0/wildcard/README.md @@ -64,7 +64,7 @@ the requester's excluded hop, ordered by `route_order` (`rs/moq-net/src/model/origin.rs:633`), served on demand by the session that announced it and cached per prefix in `ServeState.served` (`:764`). That is the split-horizon-safe lookup the old `origin::Dynamic` could not provide, and it is what -[Resolve](/quest/m1/wildcard/resolve.md) now extends rather than replaces. +[Resolve](/quest/m0/wildcard/resolve.md) now extends rather than replaces. Request resolution, by contrast, is still prefix-only (`best_server` in `rs/moq-net/src/model/origin.rs`). The pattern matcher itself exists: @@ -154,7 +154,7 @@ field. composer waiting for an announcement that only demand would produce. The browser player currently enforces the opposite (`js/watch`'s `#isPathAnnounced` hides a catalog rendition with no exact-path - announcement); [Demand](/quest/m1/wildcard/demand.md) makes a covering wildcard count as + announcement); [Demand](/quest/m0/wildcard/demand.md) makes a covering wildcard count as availability there. - **Refusal is a typed stream reset, with no negative cache.** An advertiser resets a subscribe it will not serve, and the reset carries which KIND of @@ -258,9 +258,9 @@ than announce state. ## Quests -- [Resolve](/quest/m1/wildcard/resolve.md) - a relay resolves a subscribe or +- [Resolve](/quest/m0/wildcard/resolve.md) - a relay resolves a subscribe or FETCH for an unannounced path against the best matching wildcard -- [Demand](/quest/m1/wildcard/demand.md) - the browser player subscribes to a +- [Demand](/quest/m0/wildcard/demand.md) - the browser player subscribes to a catalog-referenced broadcast a wildcard covers, breaking the lazy-rendition deadlock diff --git a/quest/m1/wildcard/demand.md b/quest/m0/wildcard/demand.md similarity index 94% rename from quest/m1/wildcard/demand.md rename to quest/m0/wildcard/demand.md index 9cd10f88fd..80be0b35ed 100644 --- a/quest/m1/wildcard/demand.md +++ b/quest/m0/wildcard/demand.md @@ -28,7 +28,7 @@ Do not simply delete the gate. It exists so the player does not subscribe to absent broadcasts and so renditions appear and disappear reactively with announcements. The Rust side needs nothing here: `moq-mux::Source` resolves references through `request_broadcast`, which -[resolve](/quest/m1/wildcard/resolve.md) teaches to consult patterns. +[resolve](/quest/m0/wildcard/resolve.md) teaches to consult patterns. Two existing soft spots to not reintroduce: the first evaluation runs before the announcement stream has populated, briefly hiding cross-broadcast @@ -45,5 +45,5 @@ nothing visibly. ## Required -- [Resolve](/quest/m1/wildcard/resolve.md) - recognizing the wildcard is useless +- [Resolve](/quest/m0/wildcard/resolve.md) - recognizing the wildcard is useless until the relay routes the resulting subscribe through it diff --git a/quest/m1/wildcard/resolve.md b/quest/m0/wildcard/resolve.md similarity index 100% rename from quest/m1/wildcard/resolve.md rename to quest/m0/wildcard/resolve.md diff --git a/quest/m1/README.md b/quest/m1/README.md index f3b9d5c9da..ce9beca1eb 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -47,7 +47,6 @@ transport, benchmark tooling); worktrees isolate commits, not semantics. - [Play tune-in backpressure](/quest/m1/play-tunein-backpressure.md) - moq play: a tune-in burst larger than the video queue parks the decoder, so the clock never reaches live at a wide `--delay` - [JavaScript FETCH](/quest/m1/js-fetch.md) - generic on-demand group serving and IETF FETCH for browser publishers - [Archive](/quest/m1/archive/README.md) - record selected tracks to any object_store and replay them over FETCH or derived HLS, on the catalog and store the release ships -- [Wildcard](/quest/m1/wildcard/README.md) - a relay resolves subscriptions against advertised prefixes, a service claims the prefix it could serve and refuses the rest instead of enumerating broadcasts, and the browser player treats a covering claim as availability - [Tooling](/quest/m1/tooling/README.md) - justfiles become a one-line menu over `sh/`, one impact map scopes CI, and every workflow step runs a recipe - [Path patterns](/quest/m1/path-patterns.md) - one matcher for every predicate over broadcast paths: tokens, origins, interest - [In-band auth](/quest/m1/auth/README.md) - a session tells its peer what it may publish and subscribe to, unions tokens presented in band, and fails loud on an out-of-scope publish diff --git a/quest/m1/archive/README.md b/quest/m1/archive/README.md index 8c388a19bc..0ee8107407 100644 --- a/quest/m1/archive/README.md +++ b/quest/m1/archive/README.md @@ -116,5 +116,5 @@ owned by that prerequisite, not duplicated in archive storage. - [Catalog track identity](/quest/m2/catalog-tracks.md) - explore immutable definitions or explicit version binding independently of archives -- [wildcard](/quest/m1/wildcard/README.md) - catch-all routing exposes an archive at its stable replay path +- [wildcard](/quest/m0/wildcard/README.md) - catch-all routing exposes an archive at its stable replay path - [e2ee](/quest/m1/e2ee/README.md) - protected broadcasts are excluded initially diff --git a/quest/m1/broadcast-epoch/README.md b/quest/m1/broadcast-epoch/README.md index a0a9551ecc..0c40773fe3 100644 --- a/quest/m1/broadcast-epoch/README.md +++ b/quest/m1/broadcast-epoch/README.md @@ -42,7 +42,7 @@ Decided: prefix route. Document this rather than promise it works. - Derived output lives under the epoch it came from (`pid/foo.hang/@e/transcode.pro`), so nested epochs must parse. This moves - the [wildcard](/quest/m1/wildcard/README.md) line's derived-output example + the [wildcard](/quest/m0/wildcard/README.md) line's derived-output example down one segment, and its suffix patterns still match. This README owns: diff --git a/quest/m1/path-patterns.md b/quest/m1/path-patterns.md index fff74556a5..90b43fb304 100644 --- a/quest/m1/path-patterns.md +++ b/quest/m1/path-patterns.md @@ -98,5 +98,5 @@ matches, containment refusal, and old-version behavior. ## Related -- [Wildcard advertisements](/quest/m1/wildcard/README.md) - routing adopts the +- [Wildcard advertisements](/quest/m0/wildcard/README.md) - routing adopts the matcher while retaining its own cost, pool, refusal, and resolution work diff --git a/quest/m1/pop-skipping/README.md b/quest/m1/pop-skipping/README.md index c1399c0b19..89a3652246 100644 --- a/quest/m1/pop-skipping/README.md +++ b/quest/m1/pop-skipping/README.md @@ -154,5 +154,5 @@ costs of one bidirectional session, which one `?cost=` cannot split. ## Related - [drain](/quest/m1/drain/README.md) - a second relay per PoP makes the same-PoP link price and its connection cardinality operationally important -- [wildcard](/quest/m1/wildcard/README.md) - it reuses this questline's route cost, and needs a cluster on Lite06 +- [wildcard](/quest/m0/wildcard/README.md) - it reuses this questline's route cost, and needs a cluster on Lite06 - [relay-memory](/quest/m1/relay-memory.md) - a denser mesh multiplies whatever a non-selected route costs diff --git a/quest/m1/processor/README.md b/quest/m1/processor/README.md index 93041435b6..dcc625f468 100644 --- a/quest/m1/processor/README.md +++ b/quest/m1/processor/README.md @@ -30,5 +30,5 @@ and custom transforms use the same worker lifecycle. - [Reference vision worker](/quest/m3/processor-vision.md) - a runnable worker publishes frame-correlated detections and proves demand, reconnect, failover, and teardown end to end -- [Wildcard advertisements](/quest/m1/wildcard/README.md) - lets a dormant +- [Wildcard advertisements](/quest/m0/wildcard/README.md) - lets a dormant processor advertise what it could serve without enumerating live sources From 88f3a473d72cc00d85fc99b7e874291cf77fc81a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 19:23:15 -0700 Subject: [PATCH 2/5] quest: gate Babel routing on a routing simulator The Babel line's first deliverable is now a deterministic simulator that compares today's path vector, Babel, and a topology split under the same workloads, and checks the actual subscribe/fetch forwarding decisions for loops. The wire quests require it. Also from review: feasibility state expires on a retention timer instead of with the last route, withdrawn prefixes are held unreachable, the mixed-mesh bridge becomes an all-at-once cluster switch with legacy sessions at the edge, and the local origin is served only on a loopback or Unix-socket internal listener. Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 2 +- quest/m0/babel/README.md | 64 +++++++++++++++++++++++++------------ quest/m0/babel/rust.md | 17 +++++++--- quest/m0/babel/simulator.md | 51 +++++++++++++++++++++++++++++ quest/m0/local-origin.md | 7 ++-- 5 files changed, 113 insertions(+), 28 deletions(-) create mode 100644 quest/m0/babel/simulator.md diff --git a/quest/m0/README.md b/quest/m0/README.md index 4c9a3eebdf..e7e89c0ea3 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -97,7 +97,7 @@ do not add another media abstraction or a renderer crate during stabilization. - [Announce counters](/quest/m0/announce-counters.md) - relay `/metrics` counts announce starts, ends, updates, and wire bytes per tier, so announce traffic is measurable - [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - a publisher sends an announce update only when the wire route changed - [Wildcard](/quest/m0/wildcard/README.md) - a relay resolves subscriptions against advertised prefixes, a service claims the prefix it could serve and refuses the rest instead of enumerating broadcasts, and the browser player treats a covering claim as availability -- [Babel routing](/quest/m0/babel/README.md) - lite-07 announcements drop the hop list and stay loop-free by source, seqno, and cost, so reroutes stop flooding the mesh +- [Babel routing](/quest/m0/babel/README.md) - a routing simulator picks the algorithm, then lite-07 announcements drop the hop list and stay loop-free by source, seqno, and cost, so reroutes stop flooding the mesh - [Local origin](/quest/m0/local-origin.md) - localhost workers read only the broadcasts their relay ingested, from the internal listener - [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the audio playout target is a measured estimate of arrival timing in both languages, not a round-trip guess - [A/V clock](/quest/m0/plan-av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles diff --git a/quest/m0/babel/README.md b/quest/m0/babel/README.md index 571bc7fd13..cd5177315a 100644 --- a/quest/m0/babel/README.md +++ b/quest/m0/babel/README.md @@ -2,16 +2,17 @@ ## Goal -moq-lite-07 announcements drop the hop list. A route carries a source id, a -per-route sequence number, and its warm and cold cost, and relays stay -loop-free with Babel's feasibility condition -([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966) section 3.5) instead of -path inspection. A reroute that leaves a relay's advertised cost unchanged no -longer propagates, and every announce stops paying for its path. lite-06 and -older, and the IETF cluster extension, keep today's hop lists. +A route change stops flooding the cluster with announce updates, and the +routing algorithm that achieves it is chosen by simulation before any wire +format is committed. Babel is the leading candidate: moq-lite-07 announcements +drop the hop list, and a route carries a source id, a per-route sequence +number, and its warm and cold cost. Relays stay loop-free with Babel's +feasibility condition ([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966) +section 3.5) instead of path inspection. lite-06 and older, and the IETF +cluster extension, keep today's hop lists. Non-goals: fewer routes per relay (tree routing was dropped), and a -permanently mixed mesh. Mixing pre-07 and lite-07 relays is a rollout window. +permanently mixed mesh. ## Plan @@ -23,6 +24,24 @@ peer that isn't excluded. [Wildcard](/quest/m0/wildcard/README.md)'s Spread quest moves identity onto the subscribe and fetch reply, so this line only replaces the first two jobs. +### Gate + +The [Routing simulator](/quest/m0/babel/simulator.md) decides whether the wire +quests below go ahead as written. Babel proceeds if, on the same workloads, it +sends clearly fewer announce messages than today's path vector, never holds a +persistent forwarding loop, and bounds transient loops and unavailability. If +the simulator shows fan-out driven by topology dominates instead (every relay +re-advertising every route to every peer), propose a line that separates relay +topology from broadcast reachability, and replace the wire quests with it. + +Babel's weak spot is the MoQ common case. RFC 8966 section 2.7 does not +guarantee loop freedom when several routers originate the same prefix, and +MoQ has overlapping prefixes, several publishers, warm relays that +re-originate, and per-request pool selection. Lite SUBSCRIBE carries no path, +so a forwarding loop is not caught at subscribe time; it stalls the +subscription until the route changes. The proof obligation is on the actual +subscribe and fetch forwarding decisions, not on converged announce tables. + ### Wire (lite-07, still `moq-lite-07-wip`) - ANNOUNCE_START carries `Source ID`, `Seqno`, and the warm and cold costs in @@ -47,6 +66,12 @@ replaces the first two jobs. `(seqno, warm, cold)` it has advertised. It accepts a route only with a newer seqno, or the same seqno and a lexicographically lower cost. A saturated cost cannot drop, so it is infeasible, which is safe. +- Feasibility state outlives the route (RFC 8966 section 3.7.3): an entry is + dropped on a timer long enough for any delayed advertisement of that seqno + to have expired, never because the last route went away. +- A withdrawn prefix is held unreachable (RFC 8966 section 3.5.4) until a + feasible route returns or every neighbour has stopped routing through this + relay, so a covering prefix cannot take over and loop back. - A relay with no feasible route sends ANNOUNCE_REFRESH on the streams offering an infeasible one. A publisher that already holds a new enough seqno re-announces. Otherwise it forwards the request toward its own upstream, and @@ -61,11 +86,14 @@ replaces the first two jobs. - [Warm advertise](/quest/m1/pop-skipping/warm-advertise.md) re-originates the exact path as the relay's own source, so its warm-zero price is compatible. -### Bridge +### Rollout -Routes from a pre-07 or IETF peer take `hops[0]` (or a minted id) as source. -Routes sent to one carry a hop list the bridge synthesizes. The bridge may be -conservative, but it must never hold a persistent loop. +The cluster switches to lite-07 routing as a whole; pre-07 and IETF sessions +stay at its edges, where a route from one takes `hops[0]` (or a minted id) as +its source and a route to one carries a single-hop list. Mixed pre-07 and +lite-07 transit inside the cluster is out of scope. If a fleet cannot switch +at once, the bridge is designed and proven in the simulator before either +wire quest starts. ### Line work @@ -74,19 +102,15 @@ This README owns the draft and the end-to-end proof: - Lite draft Routing, ANNOUNCE_START/UPDATE/REFRESH, Cost Parameter, and the lite-07 changelog. - `doc/concept/moq-lite.md`. -- A mixed ring of pre-07 and lite-07 relays that converges with no persistent - loop, fails over when a mid-path relay dies, and recovers a starved route - through REFRESH. +- A lite-07 ring that converges with no persistent loop, fails over when a + mid-path relay dies, and recovers a starved route through REFRESH. ## Quests -- [Rust routing](/quest/m0/babel/rust.md) - moq-net's route model, the lite-07 codec, ANNOUNCE_REFRESH, and the pre-07/IETF bridge +- [Routing simulator](/quest/m0/babel/simulator.md) - a deterministic simulator runs today's path vector, Babel, and a topology split under the same workloads, and decides whether the wire quests go ahead +- [Rust routing](/quest/m0/babel/rust.md) - moq-net's route model, the lite-07 codec, ANNOUNCE_REFRESH, and the cluster edge - [JS codec](/quest/m0/babel/js.md) - `@moq/net` speaks the lite-07 route fields and answers ANNOUNCE_REFRESH as a source -## Required - -- [Wildcard](/quest/m0/wildcard/README.md) - its Spread quest moves stitching identity onto the reply, which this line stops carrying in announcements - ## Related - [Local origin](/quest/m0/local-origin.md) - gives localhost workers locally ingested broadcasts without reading hop chains diff --git a/quest/m0/babel/rust.md b/quest/m0/babel/rust.md index 104650b10d..0dc79c8afd 100644 --- a/quest/m0/babel/rust.md +++ b/quest/m0/babel/rust.md @@ -5,7 +5,7 @@ `moq-net` routes lite-07 announcements by source, per-route seqno, and cost, under the feasibility condition and adjacent-only exclusion that the [Babel routing](/quest/m0/babel/README.md) line specifies. Pre-07 and IETF -sessions keep hop lists through the bridge. +sessions keep hop lists at the cluster edge. ## Plan @@ -13,15 +13,16 @@ sessions keep hop lists through the bridge. anonymous flag. `route_order` drops the hop-length and `fnv_key(prefix, hops)` tie-breaks. `RouteEntry::visible_to` excludes only the adjacent peer, so `Horizon` narrows to that peer. The feasibility table lives beside the - routes and garbage-collects entries for sources with no live route. + routes and expires entries on a retention timer, never when the last route + goes; withdrawn prefixes are held unreachable. - `rs/moq-net/src/lite/`: the codec for the new ANNOUNCE_START/UPDATE fields and ANNOUNCE_REFRESH. The publisher answers or forwards refreshes; the subscriber applies feasibility and sends refreshes when starved. It builds on the stateful codec from [Announce compression](/quest/m1/announce-compression.md) and deletes its hop-tail half. - Cost Parameter: on lite-07 the link cost is at least 1 by construction. -- Bridge: pre-07 lite and IETF (`ietf/cluster.rs`) sessions map hops to and - from source routes. +- Cluster edge: pre-07 lite and IETF (`ietf/cluster.rs`) sessions map hops to + and from source routes, as the line README's Rollout section describes. - Consumers: the relay's `/nodes` (`rs/moq-relay/src/nodes.rs`) reports source and cost in place of the path. `Route` is exposed through `moq-ffi` and `libmoq`, so this is a published API break: retarget to `dev` if `hops` has @@ -29,12 +30,18 @@ sessions keep hop lists through the bridge. Tests: codec round-trips and violations, feasibility accept/refuse, a starved relay recovering through REFRESH, adjacent exclusion on both advertise and -serve, and anonymous minting. +serve, anonymous minting, and the simulator's scenarios against the real +implementation. Benchmark: announce messages and bytes per publish, reroute, and relay loss on a simulated mesh, swept over relay count and degree, lite-06 against lite-07. +## Required + +- [Routing simulator](/quest/m0/babel/simulator.md) - decides whether this goes ahead as written +- [Wildcard](/quest/m0/wildcard/README.md) - its Spread quest moves stitching identity onto the reply, which lite-07 stops carrying in announcements + ## Related - [JS codec](/quest/m0/babel/js.md) - the browser side of the same wire diff --git a/quest/m0/babel/simulator.md b/quest/m0/babel/simulator.md new file mode 100644 index 0000000000..4acb8b2d5f --- /dev/null +++ b/quest/m0/babel/simulator.md @@ -0,0 +1,51 @@ +# [M] Routing simulator + +## Goal + +A deterministic simulator runs cluster routing algorithms on the same +topologies and workloads, and reports announce cost and forwarding safety for +each. Its result decides whether the [Babel routing](/quest/m0/babel/README.md) +wire quests go ahead as written. No wire format or public API changes. + +## Plan + +Candidates: + +- Today's path vector, as `rs/moq-net/src/model/origin.rs` and the lite + publisher implement it, including hop exclusion and the "prefer shorter hop" + tie-break. +- Babel as the line README specifies it: per-(source, prefix) seqno, + feasibility with retention, the unreachable hold, ANNOUNCE_REFRESH, and + adjacent-only exclusion. +- Topology split: relays share link costs, broadcasts announce only their + origin relays, and a relay forwards an announcement to a peer only when it + lies on that peer's shortest path to the origin. + +The model is abstract (no QUIC, no codec), with a seeded scheduler that can +delay, reorder, and drop messages on a link. Each relay decides how it would +forward a subscribe or fetch at every step, using the same rules as serving: +specificity, anonymity, cost, adjacent exclusion, and per-request pool +selection. It flags any forwarding cycle and how long it lasts, and any +interval where a reachable broadcast has no route. + +Workloads, swept over relay count and mesh degree: + +- publish and unpublish, including a prefix covered by a broader one +- several publishers of one broadcast, and a warm relay re-originating it +- peer reconnect (full replay), link cost change, relay loss and restart +- the two counterexamples from #4213's review: a three-relay ring where a + delayed same-seqno advertisement returns after withdrawal, and a relay + falling back to a broader prefix that still routes through it + +Report per algorithm and workload: announce messages and bytes by kind +(start, end, update), convergence time, loop and unavailability intervals, +and retained state per relay. Write the findings into the line README's Gate +and adjust the wire quests to match. + +It lives in an unpublished workspace crate (`publish = false`). The workloads +run as tests in nightly CI; the sweep is a benchmark. The scenarios later +become regression tests against the real implementation. + +## Related + +- [Announce counters](/quest/m0/announce-counters.md) - live numbers to check the simulated baseline against diff --git a/quest/m0/local-origin.md b/quest/m0/local-origin.md index 620cd0e964..0418d3a38d 100644 --- a/quest/m0/local-origin.md +++ b/quest/m0/local-origin.md @@ -14,10 +14,13 @@ internal hop is me"), which [Babel routing](/quest/m0/babel/README.md) removes. peers attach through `origin.peer()` (`rs/moq-relay/src/cluster.rs`). Build an `origin::Consumer` that admits only routes from customer-tier sessions, without copying the table. -- The internal listener (`rs/moq-relay/src/internal.rs`) is localhost HTTP +- The internal listener (`rs/moq-relay/src/internal.rs`) is plain HTTP today (`/metrics`, `/health`, `/nodes`, `/sessions`). Serve a moq session on it over the WebSocket transport the relay already accepts. It is - unauthenticated, like `/sessions`, because only local processes reach it. + unauthenticated, so serve it only when the internal listener is loopback or + a Unix socket. The listener may also bind a private-overlay address, which + would expose customer media to the overlay; config load fails if the local + origin is enabled there. - Open question: Voice publishes responses. Decide whether this session may publish into the relay origin, and under which grant. - Document it in `doc/bin/relay/`. From 839d10698b2e1853bb1af27ba391ab97a5045816 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 26 Sep 2026 08:53:00 -0700 Subject: [PATCH 3/5] quest: replace the Babel line with a cluster routing design Babel keeps distance-vector fan-out and global knowledge, so plan the redesign instead: announcements carry origin and cost only, relays route over a configured topology, and announce interest is served on demand through optional per-region registries. moq.pro's routing simulator gates the wire quests. Drop the announce counters quest; the simulator replaces measurement. Keep the local-origin session loopback-only and read-only. Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 2 - quest/m0/announce-counters.md | 35 --------- quest/m0/announce-update-dedupe.md | 3 +- quest/m0/babel/README.md | 118 ----------------------------- quest/m0/babel/js.md | 24 ------ quest/m0/babel/rust.md | 47 ------------ quest/m0/babel/simulator.md | 51 ------------- quest/m0/local-origin.md | 16 ++-- quest/m1/README.md | 1 + quest/m1/cluster-routing.md | 90 ++++++++++++++++++++++ 10 files changed, 101 insertions(+), 286 deletions(-) delete mode 100644 quest/m0/announce-counters.md delete mode 100644 quest/m0/babel/README.md delete mode 100644 quest/m0/babel/js.md delete mode 100644 quest/m0/babel/rust.md delete mode 100644 quest/m0/babel/simulator.md create mode 100644 quest/m1/cluster-routing.md diff --git a/quest/m0/README.md b/quest/m0/README.md index e7e89c0ea3..0edf3b3aff 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -94,10 +94,8 @@ do not add another media abstraction or a renderer crate during stabilization. ## Quests - [Release](/quest/m0/release.md) - the release moq.pro adopts: binding docs, an upgrade page, and a staging soak gate it rather than the merge -- [Announce counters](/quest/m0/announce-counters.md) - relay `/metrics` counts announce starts, ends, updates, and wire bytes per tier, so announce traffic is measurable - [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - a publisher sends an announce update only when the wire route changed - [Wildcard](/quest/m0/wildcard/README.md) - a relay resolves subscriptions against advertised prefixes, a service claims the prefix it could serve and refuses the rest instead of enumerating broadcasts, and the browser player treats a covering claim as availability -- [Babel routing](/quest/m0/babel/README.md) - a routing simulator picks the algorithm, then lite-07 announcements drop the hop list and stay loop-free by source, seqno, and cost, so reroutes stop flooding the mesh - [Local origin](/quest/m0/local-origin.md) - localhost workers read only the broadcasts their relay ingested, from the internal listener - [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the audio playout target is a measured estimate of arrival timing in both languages, not a round-trip guess - [A/V clock](/quest/m0/plan-av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles diff --git a/quest/m0/announce-counters.md b/quest/m0/announce-counters.md deleted file mode 100644 index e307f32ffb..0000000000 --- a/quest/m0/announce-counters.md +++ /dev/null @@ -1,35 +0,0 @@ -# [S] Announce counters - -## Goal - -A relay's `/metrics` counts its announcement traffic per stats tier (customer -and cluster) and direction: starts, ends, and in-place updates, plus the -encoded bytes its announce streams carry. An operator can then tell announce -traffic apart from subscribe and stats traffic, which live cannot today. - -## Plan - -On moq.pro's live fleet (2026-09-25), 93% of relay egress was not media (about -15 MB/s), and it tracked mesh degree rather than customer load. No announce -series exists anywhere, so whether announces drive it is unknown. Subscriptions -are already counted (`moq_relay_subscriptions_opened_total` in -`rs/moq-relay/src/internal.rs`); add the announce counters beside it, with the -same tier labels. - -- Count at the lite and IETF announce writers and readers in `rs/moq-net`, into - the stats model the session already reports through, so the relay only - exports them. -- Bytes are the encoded announce-stream bytes, not `announced_bytes` (which sums - raw path lengths and stays invariant under compression). Otherwise the - counters cannot show what [Announce compression](/quest/m1/announce-compression.md) - or [Babel routing](/quest/m0/babel/README.md) save. -- Document the series in `doc/bin/relay/`. - -Tests: a two-relay cluster with one publisher counts one start per hop on -publish, one update per reroute, and one end on unpublish, each on the right -tier. - -## Related - -- [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - the first saving these counters should show -- [Fleet announce panels](https://github.com/moq-dev/moq.pro/blob/main/quest/m0/announce-panels.md) - charts these on live and records the baseline diff --git a/quest/m0/announce-update-dedupe.md b/quest/m0/announce-update-dedupe.md index fd566d5968..c428466a89 100644 --- a/quest/m0/announce-update-dedupe.md +++ b/quest/m0/announce-update-dedupe.md @@ -29,5 +29,4 @@ every covered broadcast. This comes from reading the code, not from a test. ## Related -- [Announce counters](/quest/m0/announce-counters.md) - shows the saving on a live fleet -- [Babel routing](/quest/m0/babel/README.md) - removes the other big source of updates, reroutes that change only the hop chain +- [Cluster routing](/quest/m1/cluster-routing.md) - removes the other big source of updates, reroutes that change only the hop chain diff --git a/quest/m0/babel/README.md b/quest/m0/babel/README.md deleted file mode 100644 index cd5177315a..0000000000 --- a/quest/m0/babel/README.md +++ /dev/null @@ -1,118 +0,0 @@ -# Babel routing - -## Goal - -A route change stops flooding the cluster with announce updates, and the -routing algorithm that achieves it is chosen by simulation before any wire -format is committed. Babel is the leading candidate: moq-lite-07 announcements -drop the hop list, and a route carries a source id, a per-route sequence -number, and its warm and cold cost. Relays stay loop-free with Babel's -feasibility condition ([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966) -section 3.5) instead of path inspection. lite-06 and older, and the IETF -cluster extension, keep today's hop lists. - -Non-goals: fewer routes per relay (tree routing was dropped), and a -permanently mixed mesh. - -## Plan - -Today the hop list does three jobs: loop prevention (a relay drops a path -containing itself), request exclusion (never advertise or serve through the -requester, anywhere in the chain), and stitching identity (the first hop). Any -change to the chain emits an update, and a relay forwards that update to every -peer that isn't excluded. [Wildcard](/quest/m0/wildcard/README.md)'s Spread -quest moves identity onto the subscribe and fetch reply, so this line only -replaces the first two jobs. - -### Gate - -The [Routing simulator](/quest/m0/babel/simulator.md) decides whether the wire -quests below go ahead as written. Babel proceeds if, on the same workloads, it -sends clearly fewer announce messages than today's path vector, never holds a -persistent forwarding loop, and bounds transient loops and unavailability. If -the simulator shows fan-out driven by topology dominates instead (every relay -re-advertising every route to every peer), propose a line that separates relay -topology from broadcast reachability, and replace the wire quests with it. - -Babel's weak spot is the MoQ common case. RFC 8966 section 2.7 does not -guarantee loop freedom when several routers originate the same prefix, and -MoQ has overlapping prefixes, several publishers, warm relays that -re-originate, and per-request pool selection. Lite SUBSCRIBE carries no path, -so a forwarding loop is not caught at subscribe time; it stalls the -subscription until the route changes. The proof obligation is on the actual -subscribe and fetch forwarding decisions, not on converged announce tables. - -### Wire (lite-07, still `moq-lite-07-wip`) - -- ANNOUNCE_START carries `Source ID`, `Seqno`, and the warm and cold costs in - place of `Hops`. The source is fixed for the life of an Announce ID: a new - source is END and START. -- ANNOUNCE_UPDATE carries `Seqno` and costs. -- New ANNOUNCE_REFRESH (subscriber to publisher, on the announce stream): - `Announce ID` and the wanted `Seqno`. -- An anonymous flag replaces the 0 Hop ID, so anonymous routes still rank last. - The encoding is the implementer's call. -- The Cost Parameter's link cost is at least 1 and the wire cannot express 0 - (encode it minus one). Every hop then strictly raises both costs, which - feasibility needs. -- [Announce compression](/quest/m1/announce-compression.md) lands first; this - deletes its hop-tail half (`Hop Base`/`Hop Keep`) and keeps path compression. - -### Routing - -- A seqno covers one route: a source keeps one per announced prefix, so a - refresh re-announces only the route that starved. -- A relay keeps a feasibility distance per (source, prefix): the best - `(seqno, warm, cold)` it has advertised. It accepts a route only with a newer - seqno, or the same seqno and a lexicographically lower cost. A saturated cost - cannot drop, so it is infeasible, which is safe. -- Feasibility state outlives the route (RFC 8966 section 3.7.3): an entry is - dropped on a timer long enough for any delayed advertisement of that seqno - to have expired, never because the last route went away. -- A withdrawn prefix is held unreachable (RFC 8966 section 3.5.4) until a - feasible route returns or every neighbour has stopped routing through this - relay, so a covering prefix cannot take over and loop back. -- A relay with no feasible route sends ANNOUNCE_REFRESH on the streams offering - an infeasible one. A publisher that already holds a new enough seqno - re-announces. Otherwise it forwards the request toward its own upstream, and - the source bumps the route's seqno. At most one refresh per route per - upstream is outstanding. -- Exclusion is adjacent-only: never advertise or serve a route back to the - peer it arrived from. Deep exclusion goes with the chain. -- Selection keeps specificity, then anonymity, then warm and cold cost. The - shortest-path tie-break goes away, since cost already grows every hop. -- An ingest relay mints a random source id for a publisher that declares none - (lite-02/03, IETF without the cluster extension) and marks it anonymous. -- [Warm advertise](/quest/m1/pop-skipping/warm-advertise.md) re-originates the - exact path as the relay's own source, so its warm-zero price is compatible. - -### Rollout - -The cluster switches to lite-07 routing as a whole; pre-07 and IETF sessions -stay at its edges, where a route from one takes `hops[0]` (or a minted id) as -its source and a route to one carries a single-hop list. Mixed pre-07 and -lite-07 transit inside the cluster is out of scope. If a fleet cannot switch -at once, the bridge is designed and proven in the simulator before either -wire quest starts. - -### Line work - -This README owns the draft and the end-to-end proof: - -- Lite draft Routing, ANNOUNCE_START/UPDATE/REFRESH, Cost Parameter, and the - lite-07 changelog. -- `doc/concept/moq-lite.md`. -- A lite-07 ring that converges with no persistent loop, fails over when a - mid-path relay dies, and recovers a starved route through REFRESH. - -## Quests - -- [Routing simulator](/quest/m0/babel/simulator.md) - a deterministic simulator runs today's path vector, Babel, and a topology split under the same workloads, and decides whether the wire quests go ahead -- [Rust routing](/quest/m0/babel/rust.md) - moq-net's route model, the lite-07 codec, ANNOUNCE_REFRESH, and the cluster edge -- [JS codec](/quest/m0/babel/js.md) - `@moq/net` speaks the lite-07 route fields and answers ANNOUNCE_REFRESH as a source - -## Related - -- [Local origin](/quest/m0/local-origin.md) - gives localhost workers locally ingested broadcasts without reading hop chains -- [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - the other half of cutting update churn -- [Announce counters](/quest/m0/announce-counters.md) - measures the saving diff --git a/quest/m0/babel/js.md b/quest/m0/babel/js.md deleted file mode 100644 index e1dc2b9526..0000000000 --- a/quest/m0/babel/js.md +++ /dev/null @@ -1,24 +0,0 @@ -# [S] JS codec - -## Goal - -`@moq/net` decodes and encodes the lite-07 route fields from the -[Babel routing](/quest/m0/babel/README.md) line. As a publisher it answers -ANNOUNCE_REFRESH by bumping the route's seqno, and it tells a reroute from a -takeover without reading a hop list. - -## Plan - -- `js/net/src/lite/`: ANNOUNCE_START/UPDATE fields and ANNOUNCE_REFRESH. -- Every connection mints a random hop today (`lite/connection.ts`). The source - id is per origin instead, so a reconnecting browser publisher keeps its - identity. -- `lite/subscriber.ts` separates a reroute from a takeover with - `hops[0] ?? responderOrigin`. Key it on the source id, or on the reply's - origin once Spread lands. -- The tie-break on hop count in `origin.ts` goes away. -- Interop: `just test interop --all` against the Rust side. - -## Required - -- [Rust routing](/quest/m0/babel/rust.md) - interop needs the Rust side of the wire diff --git a/quest/m0/babel/rust.md b/quest/m0/babel/rust.md deleted file mode 100644 index 0dc79c8afd..0000000000 --- a/quest/m0/babel/rust.md +++ /dev/null @@ -1,47 +0,0 @@ -# [L] Rust routing - -## Goal - -`moq-net` routes lite-07 announcements by source, per-route seqno, and cost, -under the feasibility condition and adjacent-only exclusion that the -[Babel routing](/quest/m0/babel/README.md) line specifies. Pre-07 and IETF -sessions keep hop lists at the cluster edge. - -## Plan - -- `rs/moq-net/src/model/origin.rs`: `Route` gains source, seqno, and the - anonymous flag. `route_order` drops the hop-length and `fnv_key(prefix, - hops)` tie-breaks. `RouteEntry::visible_to` excludes only the adjacent peer, - so `Horizon` narrows to that peer. The feasibility table lives beside the - routes and expires entries on a retention timer, never when the last route - goes; withdrawn prefixes are held unreachable. -- `rs/moq-net/src/lite/`: the codec for the new ANNOUNCE_START/UPDATE fields - and ANNOUNCE_REFRESH. The publisher answers or forwards refreshes; the - subscriber applies feasibility and sends refreshes when starved. It builds on - the stateful codec from [Announce compression](/quest/m1/announce-compression.md) - and deletes its hop-tail half. -- Cost Parameter: on lite-07 the link cost is at least 1 by construction. -- Cluster edge: pre-07 lite and IETF (`ietf/cluster.rs`) sessions map hops to - and from source routes, as the line README's Rollout section describes. -- Consumers: the relay's `/nodes` (`rs/moq-relay/src/nodes.rs`) reports source - and cost in place of the path. `Route` is exposed through `moq-ffi` and - `libmoq`, so this is a published API break: retarget to `dev` if `hops` has - to leave the public type. - -Tests: codec round-trips and violations, feasibility accept/refuse, a starved -relay recovering through REFRESH, adjacent exclusion on both advertise and -serve, anonymous minting, and the simulator's scenarios against the real -implementation. - -Benchmark: announce messages and bytes per publish, reroute, and relay loss -on a simulated mesh, swept over relay count and degree, lite-06 against -lite-07. - -## Required - -- [Routing simulator](/quest/m0/babel/simulator.md) - decides whether this goes ahead as written -- [Wildcard](/quest/m0/wildcard/README.md) - its Spread quest moves stitching identity onto the reply, which lite-07 stops carrying in announcements - -## Related - -- [JS codec](/quest/m0/babel/js.md) - the browser side of the same wire diff --git a/quest/m0/babel/simulator.md b/quest/m0/babel/simulator.md deleted file mode 100644 index 4acb8b2d5f..0000000000 --- a/quest/m0/babel/simulator.md +++ /dev/null @@ -1,51 +0,0 @@ -# [M] Routing simulator - -## Goal - -A deterministic simulator runs cluster routing algorithms on the same -topologies and workloads, and reports announce cost and forwarding safety for -each. Its result decides whether the [Babel routing](/quest/m0/babel/README.md) -wire quests go ahead as written. No wire format or public API changes. - -## Plan - -Candidates: - -- Today's path vector, as `rs/moq-net/src/model/origin.rs` and the lite - publisher implement it, including hop exclusion and the "prefer shorter hop" - tie-break. -- Babel as the line README specifies it: per-(source, prefix) seqno, - feasibility with retention, the unreachable hold, ANNOUNCE_REFRESH, and - adjacent-only exclusion. -- Topology split: relays share link costs, broadcasts announce only their - origin relays, and a relay forwards an announcement to a peer only when it - lies on that peer's shortest path to the origin. - -The model is abstract (no QUIC, no codec), with a seeded scheduler that can -delay, reorder, and drop messages on a link. Each relay decides how it would -forward a subscribe or fetch at every step, using the same rules as serving: -specificity, anonymity, cost, adjacent exclusion, and per-request pool -selection. It flags any forwarding cycle and how long it lasts, and any -interval where a reachable broadcast has no route. - -Workloads, swept over relay count and mesh degree: - -- publish and unpublish, including a prefix covered by a broader one -- several publishers of one broadcast, and a warm relay re-originating it -- peer reconnect (full replay), link cost change, relay loss and restart -- the two counterexamples from #4213's review: a three-relay ring where a - delayed same-seqno advertisement returns after withdrawal, and a relay - falling back to a broader prefix that still routes through it - -Report per algorithm and workload: announce messages and bytes by kind -(start, end, update), convergence time, loop and unavailability intervals, -and retained state per relay. Write the findings into the line README's Gate -and adjust the wire quests to match. - -It lives in an unpublished workspace crate (`publish = false`). The workloads -run as tests in nightly CI; the sweep is a benchmark. The scenarios later -become regression tests against the real implementation. - -## Related - -- [Announce counters](/quest/m0/announce-counters.md) - live numbers to check the simulated baseline against diff --git a/quest/m0/local-origin.md b/quest/m0/local-origin.md index 0418d3a38d..bb435c2954 100644 --- a/quest/m0/local-origin.md +++ b/quest/m0/local-origin.md @@ -6,7 +6,8 @@ A localhost worker (a Voice agent, a recorder) connects to the relay's internal listener and gets a moq session whose origin holds only the broadcasts this relay ingested from its own customer sessions, never those learned from cluster peers. Workers stop electing on hop chains ("the first -internal hop is me"), which [Babel routing](/quest/m0/babel/README.md) removes. +internal hop is me"), which [Cluster routing](/quest/m1/cluster-routing.md) +removes inside a cluster. ## Plan @@ -17,12 +18,13 @@ internal hop is me"), which [Babel routing](/quest/m0/babel/README.md) removes. - The internal listener (`rs/moq-relay/src/internal.rs`) is plain HTTP today (`/metrics`, `/health`, `/nodes`, `/sessions`). Serve a moq session on it over the WebSocket transport the relay already accepts. It is - unauthenticated, so serve it only when the internal listener is loopback or - a Unix socket. The listener may also bind a private-overlay address, which - would expose customer media to the overlay; config load fails if the local - origin is enabled there. -- Open question: Voice publishes responses. Decide whether this session - may publish into the relay origin, and under which grant. + unauthenticated, so serve it only when the internal listener binds a + loopback address. The listener may also bind a private-overlay address, + which would expose customer media to the overlay; config load fails if the + local origin is enabled there. +- The session is read-only. Open question: Voice publishes responses. Decide + whether this session may publish into the relay origin, and under which + grant, before it accepts a publish. - Document it in `doc/bin/relay/`. Tests: a two-relay cluster where each relay's local session lists only its own diff --git a/quest/m1/README.md b/quest/m1/README.md index ce9beca1eb..ef7595ebc8 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -21,6 +21,7 @@ transport, benchmark tooling); worktrees isolate commits, not semantics. - [JS fetch answer](/quest/m1/js-fetch-answer.md) - js/net's lite fetch settles on the publisher's answer, and a JS publisher's miss resets with NotFound - [libmoq hidden opt-in](/quest/m1/libmoq-hidden.md) - `moq_origin_announced` takes a `hidden` flag so C callers can list `.`-named broadcasts - [Announce compression](/quest/m1/announce-compression.md) - a lite-07 announce reuses the path head and hop-chain tail of a live announcement on its stream instead of resending them +- [Cluster routing](/quest/m1/cluster-routing.md) - an announcement says where a broadcast originates, not how to reach it, and a relay hears only the prefixes its clients asked for - [lite-07 count settle](/quest/m1/lite-count-settle.md) - moq-lite-07 subscribers stop waiting for a subscription's tail once SUBSCRIBE_END's stream count is reached - [Session death error](/quest/m1/session-death-error.md) - a dying session ends its tracks with its own error in Rust and JS, never a clean end, `Dropped`, or `Cancel` - [JS bare FIN](/quest/m1/js-bare-fin.md) - a `@moq/net` subscriber aborts a track whose subscribe stream FINs before its declared end, like Rust diff --git a/quest/m1/cluster-routing.md b/quest/m1/cluster-routing.md new file mode 100644 index 0000000000..8045d36911 --- /dev/null +++ b/quest/m1/cluster-routing.md @@ -0,0 +1,90 @@ +# [XL] Cluster routing + +## Goal + +A broadcast event reaches each relay at most once, and a relay learns only the +prefixes its own clients asked for. An announcement says a path exists at an +origin relay, at a cost; how to reach that origin comes from a shared relay +topology, so no announcement inside a cluster carries a hop list. This quest +records the design; its wire and implementation quests are planned from the +simulator's report. + +Non-goals: warm re-origination (a warm relay would be one more origin with a +cost, so leave room for it), and a permanently mixed-version cluster. + +## Plan + +### Why not path vector or Babel + +Today every relay advertises its best route to every peer not already in the +hop chain. One publish costs about R·(d-1) announces for R relays of mesh +degree d, every relay learns every broadcast (`.stats` and `.internal` +included), and a link change rewrites every route crossing it. On moq.pro's +live fleet (26 PoPs, average degree about 5) each relay receives every event +about five times. Babel ([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966)) +shrinks the message and skips equal-cost reroutes, but it is still distance +vector: the same fan-out, the same global knowledge, and no loop freedom +when several sources claim a prefix (section 2.7), which pools and wildcards +make MoQ's common case. + +### Decisions + +- Existence is split from reachability. An announcement carries the path, its + origin relay, and the origin's cost, and nothing about the path to it. +- The topology is configured: `--cluster-connect` or the connect API gives the + relay graph and link costs. Relays flood per-link liveness among themselves + with a per-link seqno. Gossip discovery (`cluster.mesh`) stays for + zero-config self-hosting and derives the topology from what it discovers; it + need not scale. +- A relay picks the origin with the lowest shortest-path distance plus origin + cost, ties broken by rendezvous hashing (HRW) of the requested path and the + origin id, and forwards along its shortest path. That is a shortest path to + a virtual node linked to every origin, so it is loop-free whenever relays + agree on the topology. Specificity still ranks first, per + [Wildcard](/quest/m0/wildcard/README.md). +- SUBSCRIBE and FETCH carry a visited-relay list end to end. It catches loops + while liveness views disagree and names the path for stats. Narrowing it to + cluster hops is later work. The serving origin's identity rides the reply, + per Wildcard's Spread quest. +- Announcements are on demand. A relay forwards only the union of its clients' + ANNOUNCE_REQUEST prefixes, never the empty prefix. A wide prefix that many + edges' viewers request is that customer's cost. `.stats` becomes ordinary + demand. +- Registries are an optional, configured tier: moq-relay in a registry mode, + one or more per region. + - An ingest relay registers its broadcasts with its nearest registry, and an + edge sends its ANNOUNCE_REQUEST there. + - Registries form a small full mesh and flood existence among themselves, so + an event crosses an ocean once per region. Announce latency is about one + round trip to the nearest registry, whatever the path length. + - A relay fails over to the next-nearest registry and reconciles its view + instead of treating the lost session as ends, so a registry failure never + reports a live broadcast offline. + - With no registry reachable, a relay freezes: it keeps its view, learns + nothing new, and alerts. Falling back to flooding would cascade the + failure. +- Without registries (self-hosting), existence floods along the shortest-path + tree, one copy per relay. +- Between clusters, announcements stay path vector with cluster ids as the + hops, like BGP between autonomous systems. A customer's on-prem cluster is + one hop, and an announcement naming the receiving cluster is dropped. +- The cluster switches versions as a whole; older lite and IETF sessions stay + at its edges. + +### Open questions + +- Sharding registries by HRW over a prefix key once one registry cannot hold + everything, and what that key is. +- A mixed-version bridge, if a fleet cannot switch at once. +- What remains of [Announce compression](/quest/m1/announce-compression.md)'s + hop-tail half once only cluster boundaries carry hops. + +## Required + +- moq.pro's routing simulator reports ([quest](https://github.com/moq-dev/moq.pro/blob/main/quest/m1/routing-simulator.md)) + +## Related + +- [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - cuts duplicate updates on today's routing +- [Local origin](/quest/m0/local-origin.md) - workers stop reading hop chains before they go +- [Wildcard](/quest/m0/wildcard/README.md) - the specificity, pool spread, and reply identity this selection builds on From 7435e35c770c6622efd54d2315cf83f9cabb2890 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 26 Sep 2026 08:53:36 -0700 Subject: [PATCH 4/5] quest: drop the hop-list plan that cluster routing supersedes Co-Authored-By: Claude Opus 5.5 --- quest/m2/README.md | 1 - quest/m2/plan-routing-origin.md | 41 --------------------------------- 2 files changed, 42 deletions(-) delete mode 100644 quest/m2/plan-routing-origin.md diff --git a/quest/m2/README.md b/quest/m2/README.md index 80fa09dd1b..d80049439f 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -53,7 +53,6 @@ upstream release waits in [m4](/quest/m4/README.md). - [Compressed tracks](/quest/m2/flate/README.md) - any track compresses per group from every language, not only the JSON modes - [Binary delta stats](/quest/m2/stats-delta.md) - an on-demand varint delta flavor of every stats track, if relay encode CPU still matters after the JSON fixes - [#3115](/quest/m2/3115-moqsink-the-publication-has-no-generation-so-a-flush.md) - moqsink: a flushing restart after EOS opens a new publication generation -- [Plan: routing without a hop list](/quest/m2/plan-routing-origin.md) - whether announcements can drop the hop list and stay loop-free once stitching keys on the subscribe reply - [Redundant ingest](/quest/m2/redundant-ingest.md) - decide whether two publishers sharing one epoch may splice, and who declares the incumbent dead before the keep-alive does - [Multipath spike](/quest/m2/multipath-spike.md) - whether bonded contribution over multipath QUIC is worth building, given it needs noq on both ends - [Receive timestamps](/quest/m2/quic-receive-ts.md) - per-packet arrival times in ACKs, the feedback GCC and deadlines need diff --git a/quest/m2/plan-routing-origin.md b/quest/m2/plan-routing-origin.md deleted file mode 100644 index 5a0c7f74f3..0000000000 --- a/quest/m2/plan-routing-origin.md +++ /dev/null @@ -1,41 +0,0 @@ -# [M] Plan: routing without a hop list - -## Goal - -Decide whether moq-lite announcements can drop the full hop list and still -route loop-free, then write the decision into quests. Today every -announcement carries the Hop IDs it traversed. That path does three jobs: -loop prevention (a relay discards a path containing itself), request -exclusion (never serve a request back through the peer that made it), and -route identity (the first hop decides whether a failover may stitch -seamlessly). The hop list was designed before prefix claims, and once the Spread quest in -[Wildcard advertisements](/quest/m1/wildcard/README.md) moves stitching -identity onto the subscribe reply, only the first two jobs remain. - -## Plan - -Open questions, to settle with the maintainer: - -- Loop freedom without a path. Rising cost alone ("cost + 1, never - advertise lower") counts to infinity after a withdrawal, as two relays - offer each other the stale route at ever higher cost. Remembering each - route's immediate neighbor and never echoing a route back to it (split - horizon) stops two-relay loops, but not three-relay loops. It also hides - backups: a relay never learns an alternative that passes back through the - neighbor it came from. Babel's feasibility condition (RFC 8966: a - per-origin sequence number; a route is feasible when its sequence number is - newer, at any cost, or equal and cheaper than the best seen for it) is - loop-free with only the origin on the wire. Weigh it and - any simpler scheme against what the relay cluster actually needs. -- What replaces request exclusion, and whether it still matters once - announcements are loop-free. -- What the stats and sidecar consumers lose without a path, and whether - anything besides the origin must stay on the wire. -- The version it lands in (lite-07 is `moq-lite-07-wip` today) and the - bridge to versions that still carry a hop list. lite-07 already compresses - each announcement's hop chain against a live base (#4196, - `rs/moq-net/src/lite/compress.rs`); dropping the hop list removes that too. - -## Related - -- [Wildcard advertisements](/quest/m1/wildcard/README.md) - its Spread quest moves stitching identity to the subscribe reply, which this builds on From 363ea972061f459a4ff3111249f14bde55514218 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 26 Sep 2026 19:38:24 -0700 Subject: [PATCH 5/5] quest: loop-free distance, incarnation seqnos, flood dedupe in cluster routing Also records two open questions where on-demand announcements meet the rule that a relay never waits on peers. Co-Authored-By: Claude Opus 5.5 --- quest/m1/cluster-routing.md | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/quest/m1/cluster-routing.md b/quest/m1/cluster-routing.md index 2c04067884..dbaef2bb8d 100644 --- a/quest/m1/cluster-routing.md +++ b/quest/m1/cluster-routing.md @@ -33,15 +33,17 @@ make MoQ's common case. origin relay, and the origin's cost, and nothing about the path to it. - The topology is configured: `--cluster-connect` or the connect API gives the relay graph and link costs. Relays flood per-link liveness among themselves - with a per-link seqno. Gossip discovery (`cluster.mesh`) stays for - zero-config self-hosting and derives the topology from what it discovers; it - need not scale. + with a per-link seqno. The seqno is scoped to the relay's incarnation, so a + restarted relay's links supersede its stale ones instead of looking older. + Gossip discovery (`cluster.mesh`) stays for zero-config self-hosting and + derives the topology from what it discovers; it need not scale. - A relay picks the origin with the lowest shortest-path distance plus origin cost, ties broken by rendezvous hashing (HRW) of the requested path and the - origin id, and forwards along its shortest path. That is a shortest path to - a virtual node linked to every origin, so it is loop-free whenever relays - agree on the topology. Specificity still ranks first, per - [Wildcard](/quest/m0/wildcard/README.md). + origin id, and forwards along its shortest path. Distance compares cost, + then hop count, so every hop strictly shortens it even across `?cost=0` + links. That is a shortest path to a virtual node linked to every origin, so + it is loop-free whenever relays agree on the topology. Specificity still + ranks first, per [Wildcard](/quest/m0/wildcard/README.md). - SUBSCRIBE and FETCH carry a visited-relay list end to end. It catches loops while liveness views disagree and names the path for stats. Narrowing it to cluster hops is later work. The serving origin's identity rides the reply, @@ -55,8 +57,9 @@ make MoQ's common case. - An ingest relay registers its broadcasts with its nearest registry, and an edge sends its ANNOUNCE_REQUEST there. - Registries form a small full mesh and flood existence among themselves, so - an event crosses an ocean once per region. Announce latency is about one - round trip to the nearest registry, whatever the path length. + an event crosses an ocean once per remote registry, not once per relay. + Announce latency is about one round trip to the nearest registry, + whatever the path length. - A relay fails over to the next-nearest registry and reconciles its view instead of treating the lost session as ends, so a registry failure never reports a live broadcast offline. @@ -64,7 +67,9 @@ make MoQ's common case. nothing new, and alerts. Falling back to flooding would cascade the failure. - Without registries (self-hosting), existence floods along the shortest-path - tree, one copy per relay. + tree, one copy per relay. A relay forwards an event only when it changes its + view, so a duplicate copy, from trees built on disagreeing liveness, stops + there. - Between clusters, announcements stay path vector with cluster ids as the hops, like BGP between autonomous systems. A customer's on-prem cluster is one hop, and an announcement naming the receiving cluster is dropped. @@ -76,6 +81,12 @@ make MoQ's common case. - Sharding registries by HRW over a prefix key once one registry cannot hold everything, and what that key is. - A mixed-version bridge, if a fleet cannot switch at once. +- How an edge routes a SUBSCRIBE for a path none of its clients asked to + announce. It holds no route for it, and asking a registry first adds a round + trip before the first byte. +- What a cold ANNOUNCE_REQUEST reports as live. Answering from the local view + keeps a relay from waiting on peers but reports an empty set until the + registry's replay lands, on every new prefix rather than in a rare race. - What remains of announce compression's hop-tail half (`Hop Base` and `Hop Keep` in the lite draft) once only cluster boundaries carry hops.