diff --git a/quest/m0/README.md b/quest/m0/README.md index 865a53a06d..16f16f47b5 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -95,6 +95,9 @@ do not add another media abstraction or a renderer crate during stabilization. - [Auth parity](/quest/m0/auth-parity.md) - 0.14 configs and tokens keep working or are refused with the fix named, and every credential a session presents is evaluated or refused - [Release](/quest/m0/release.md) - the release moq.pro adopts: binding docs, an upgrade page, and a staging soak gate it rather than the merge +- [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - a publisher sends an announce update only when the wire route changed +- [Wildcard](/quest/m0/wildcard/README.md) - a relay resolves subscriptions against advertised prefixes, a service claims the prefix it could serve and refuses the rest instead of enumerating broadcasts, and the browser player treats a covering claim as availability +- [Local origin](/quest/m0/local-origin.md) - localhost workers read only the broadcasts their relay ingested, from the internal listener - [Audio quality harness](/quest/m0/audio-quality-harness/README.md) - a playout latency regression fails a run instead of arriving as a bug report, and its recorder supplies the jitter target's replay traces - [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the audio playout target is a measured estimate of arrival timing in both languages, not a round-trip guess - [A/V clock](/quest/m0/plan-av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles diff --git a/quest/m0/announce-update-dedupe.md b/quest/m0/announce-update-dedupe.md new file mode 100644 index 0000000000..c428466a89 --- /dev/null +++ b/quest/m0/announce-update-dedupe.md @@ -0,0 +1,32 @@ +# [S] Skip unchanged announce updates + +## Goal + +A publisher sends an announce update only when what the peer would decode +differs from what it last sent for that announcement. A local change the wire +cannot express (the route's source session, `served`, captures) sends nothing. +This applies on every version, lite and IETF, in Rust and JS, and it is wire +compatible. + +## Plan + +Each announce cursor dedupes its best route on `(hops, cost, source)` plus +`served` and captures (`rs/moq-net/src/model/origin.rs`, the `Updated` kind), +but the lite publisher only remembers each suffix's Announce ID +(`rs/moq-net/src/lite/publisher.rs`, the `self.live` branch) and re-sends +`Restart` for every `Updated` it sees. Only the hops and cost reach the wire, +so a flip of any other field sends an identical update, to every peer, for +every covered broadcast. This comes from reading the code, not from a test. + +- Reproduce first: a test that flips a route's source session with the same + hops and cost, and asserts the peer receives no update. +- Keep the last-sent `(hops, cost)` beside the Announce ID and skip equal + updates. Check the IETF publisher's re-pricing path and the JS publisher for + the same pattern. +- Fix the stale comments on the way: `lite/announce.rs` says restarts are only + ever received, and the `restart_announce` doc in `lite/subscriber.rs` says it + compares the first hop. + +## Related + +- [Cluster routing](/quest/m1/cluster-routing.md) - removes the other big source of updates, reroutes that change only the hop chain diff --git a/quest/m0/local-origin.md b/quest/m0/local-origin.md new file mode 100644 index 0000000000..60937e64ac --- /dev/null +++ b/quest/m0/local-origin.md @@ -0,0 +1,36 @@ +# [M] Local origin + +## Goal + +A localhost worker (a Voice agent, a recorder) connects to the relay's +internal listener and gets a moq session whose origin holds only the +broadcasts this relay ingested from its own customer sessions, never those +learned from cluster peers. Workers stop electing on hop chains ("the first +internal hop is me"), which [Cluster routing](/quest/m1/cluster-routing.md) +removes inside a cluster. + +## Plan + +- The view exists: `origin::Consumer::local()` (`rs/moq-net/src/model/origin.rs`) + hides every route a `Producer::peer` handle announced, which is how cluster + peers attach (`rs/moq-relay/src/cluster.rs`), and + `local_view_hides_peer_routes` covers it. The work is serving that view. +- The internal listener (`rs/moq-relay/src/internal.rs`) is plain HTTP + today (`/metrics`, `/health`, `/nodes`, `/sessions`). Serve a moq session on + it over the WebSocket transport the relay already accepts. It is + unauthenticated, so serve it only when the internal listener binds a + loopback address. The listener may also bind a private-overlay address, + which would expose customer media to the overlay; config load fails if the + local origin is enabled there. +- The session is read-only. Open question: Voice publishes responses. Decide + whether this session may publish into the relay origin, and under which + grant, before it accepts a publish. +- Document it in `doc/bin/relay/`. + +Tests: a two-relay cluster where each relay's local session lists only its own +publishers, a publisher moving relays moves between the two views, and a +peer-learned route never appears. + +## Related + +- [Voice on the local origin](https://github.com/moq-dev/moq.pro/blob/main/quest/m0/voice-local-origin.md) - moq.pro's Voice and recorder switch to this diff --git a/quest/m1/wildcard/README.md b/quest/m0/wildcard/README.md similarity index 98% rename from quest/m1/wildcard/README.md rename to quest/m0/wildcard/README.md index e7be970d69..13795b6eff 100644 --- a/quest/m1/wildcard/README.md +++ b/quest/m0/wildcard/README.md @@ -64,7 +64,7 @@ the requester's excluded hop, ordered by `route_order` (`rs/moq-net/src/model/origin.rs:633`), served on demand by the session that announced it and cached per prefix in `ServeState.served` (`:764`). That is the split-horizon-safe lookup the old `origin::Dynamic` could not provide, and it is what -[Resolve](/quest/m1/wildcard/resolve.md) now extends rather than replaces. +[Resolve](/quest/m0/wildcard/resolve.md) now extends rather than replaces. Request resolution, by contrast, is still prefix-only (`best_server` in `rs/moq-net/src/model/origin.rs`). The pattern matcher itself exists: @@ -154,7 +154,7 @@ field. composer waiting for an announcement that only demand would produce. The browser player currently enforces the opposite (`js/watch`'s `#isPathAnnounced` hides a catalog rendition with no exact-path - announcement); [Demand](/quest/m1/wildcard/demand.md) makes a covering wildcard count as + announcement); [Demand](/quest/m0/wildcard/demand.md) makes a covering wildcard count as availability there. - **Refusal is a typed stream reset, with no negative cache.** An advertiser resets a subscribe it will not serve, and the reset carries which KIND of @@ -258,9 +258,9 @@ than announce state. ## Quests -- [Resolve](/quest/m1/wildcard/resolve.md) - a relay resolves a subscribe or +- [Resolve](/quest/m0/wildcard/resolve.md) - a relay resolves a subscribe or FETCH for an unannounced path against the best matching wildcard -- [Demand](/quest/m1/wildcard/demand.md) - the browser player subscribes to a +- [Demand](/quest/m0/wildcard/demand.md) - the browser player subscribes to a catalog-referenced broadcast a wildcard covers, breaking the lazy-rendition deadlock diff --git a/quest/m1/wildcard/demand.md b/quest/m0/wildcard/demand.md similarity index 94% rename from quest/m1/wildcard/demand.md rename to quest/m0/wildcard/demand.md index 9cd10f88fd..80be0b35ed 100644 --- a/quest/m1/wildcard/demand.md +++ b/quest/m0/wildcard/demand.md @@ -28,7 +28,7 @@ Do not simply delete the gate. It exists so the player does not subscribe to absent broadcasts and so renditions appear and disappear reactively with announcements. The Rust side needs nothing here: `moq-mux::Source` resolves references through `request_broadcast`, which -[resolve](/quest/m1/wildcard/resolve.md) teaches to consult patterns. +[resolve](/quest/m0/wildcard/resolve.md) teaches to consult patterns. Two existing soft spots to not reintroduce: the first evaluation runs before the announcement stream has populated, briefly hiding cross-broadcast @@ -45,5 +45,5 @@ nothing visibly. ## Required -- [Resolve](/quest/m1/wildcard/resolve.md) - recognizing the wildcard is useless +- [Resolve](/quest/m0/wildcard/resolve.md) - recognizing the wildcard is useless until the relay routes the resulting subscribe through it diff --git a/quest/m1/wildcard/resolve.md b/quest/m0/wildcard/resolve.md similarity index 100% rename from quest/m1/wildcard/resolve.md rename to quest/m0/wildcard/resolve.md diff --git a/quest/m1/README.md b/quest/m1/README.md index e78483fdba..343826bb77 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -19,6 +19,7 @@ transport, benchmark tooling); worktrees isolate commits, not semantics. - [Drill sensitivity](/quest/m1/drill-sensitivity.md) - the nightly drill-sensitivity job passes: the subscriber-leaks-broadcasts mutation applies to the current lite subscriber again - [BBR classic ECN](/quest/m1/bbr-classic-ecn.md) - Startup and bandwidth probing respond to CE marks before the bottleneck drops packets +- [Cluster routing](/quest/m1/cluster-routing.md) - an announcement says where a broadcast originates, not how to reach it, and a relay hears only the prefixes its clients asked for - [lite-07 count settle](/quest/m1/lite-count-settle.md) - moq-lite-07 subscribers stop waiting for a subscription's tail once SUBSCRIBE_END's stream count is reached - [Dropped sources](/quest/m1/dropped-sources.md) - track consumers see the producer's real error on every end path, never `Dropped` - [Track tail interop](/quest/m1/track-tail-interop.md) - a Rust publisher ending a track with a group in flight is read to its end by the JS subscriber, and the reverse, in `just test interop` @@ -75,7 +76,6 @@ transport, benchmark tooling); worktrees isolate commits, not semantics. - [JS IETF datagrams](/quest/m1/js-ietf-datagram.md) - `@moq/net` sends and receives datagram groups over moq-transport, like Rust - [JavaScript FETCH](/quest/m1/js-fetch.md) - generic on-demand group serving and IETF FETCH for browser publishers - [Archive](/quest/m1/archive/README.md) - record selected tracks to any object_store and replay them over FETCH or derived HLS, on the catalog and store the release ships -- [Wildcard](/quest/m1/wildcard/README.md) - a relay resolves subscriptions against advertised prefixes, a service claims the prefix it could serve and refuses the rest instead of enumerating broadcasts, and the browser player treats a covering claim as availability - [Tooling](/quest/m1/tooling/README.md) - justfiles become a one-line menu over `sh/`, one impact map scopes CI, and every workflow step runs a recipe - [Path patterns](/quest/m1/path-patterns.md) - one matcher for every predicate over broadcast paths: tokens, origins, interest - [In-band auth](/quest/m1/auth/README.md) - a session tells its peer what it may publish and subscribe to, unions tokens presented in band, and fails loud on an out-of-scope publish diff --git a/quest/m1/archive/README.md b/quest/m1/archive/README.md index b772e4d8f2..2e8b5508de 100644 --- a/quest/m1/archive/README.md +++ b/quest/m1/archive/README.md @@ -117,5 +117,5 @@ owned by that prerequisite, not duplicated in archive storage. - [Catalog track identity](/quest/m2/catalog-tracks.md) - explore immutable definitions or explicit version binding independently of archives -- [wildcard](/quest/m1/wildcard/README.md) - catch-all routing exposes an archive at its stable replay path +- [wildcard](/quest/m0/wildcard/README.md) - catch-all routing exposes an archive at its stable replay path - [e2ee](/quest/m1/e2ee/README.md) - protected broadcasts are excluded initially diff --git a/quest/m1/broadcast-epoch/README.md b/quest/m1/broadcast-epoch/README.md index a0a9551ecc..0c40773fe3 100644 --- a/quest/m1/broadcast-epoch/README.md +++ b/quest/m1/broadcast-epoch/README.md @@ -42,7 +42,7 @@ Decided: prefix route. Document this rather than promise it works. - Derived output lives under the epoch it came from (`pid/foo.hang/@e/transcode.pro`), so nested epochs must parse. This moves - the [wildcard](/quest/m1/wildcard/README.md) line's derived-output example + the [wildcard](/quest/m0/wildcard/README.md) line's derived-output example down one segment, and its suffix patterns still match. This README owns: diff --git a/quest/m1/cluster-routing.md b/quest/m1/cluster-routing.md new file mode 100644 index 0000000000..13d66d28a8 --- /dev/null +++ b/quest/m1/cluster-routing.md @@ -0,0 +1,116 @@ +# [XL] Cluster routing + +## Goal + +A broadcast event reaches each relay at most once, and a relay learns only the +prefixes its own clients asked for. An announcement says a path exists at an +origin relay, at a cost; how to reach that origin comes from a shared relay +topology, so no announcement inside a cluster carries a hop list. This quest +records the design; its wire and implementation quests are planned from the +simulator's report. + +Non-goals: warm re-origination (a warm relay would be one more origin with a +cost, so leave room for it), and a permanently mixed-version cluster. + +## Plan + +### Why not path vector or Babel + +Today every relay advertises its best route to every peer not already in the +hop chain. One publish costs about R·(d-1) announces for R relays of mesh +degree d, every relay learns every broadcast (`.stats` and `.internal` +included), and a link change rewrites every route crossing it. On moq.pro's +live fleet (26 PoPs, average degree about 5) each relay receives every event +about five times. Babel ([RFC 8966](https://www.rfc-editor.org/rfc/rfc8966)) +shrinks the message and skips equal-cost reroutes, but it is still distance +vector: the same fan-out, the same global knowledge, and no loop freedom +when several sources claim a prefix (section 2.7), which pools and wildcards +make MoQ's common case. + +### Decisions + +- Existence is split from reachability. An announcement carries the path, its + origin relay, and the origin's cost, and nothing about the path to it. +- An existence event carries the origin's seqno, scoped to its incarnation. A + relay applies an event only when it is newer than the last it applied for + that path and origin, and keeps an ended path's seqno, so a start delayed on + a stale tree or a failed-over registry cannot revive it. Babel keeps + feasibility past withdrawal for the same reason + ([RFC 8966 section 3.7.3](https://www.rfc-editor.org/rfc/rfc8966#section-3.7.3)). +- The topology is configured: `--cluster-connect` or the connect API gives the + relay graph and link costs. Relays flood per-link liveness among themselves + with a per-link seqno. The seqno is scoped to the relay's incarnation, so a + restarted relay's links supersede its stale ones instead of looking older. + Gossip discovery (`cluster.mesh`) stays for zero-config self-hosting and + derives the topology from what it discovers; it need not scale. +- A relay picks the origin with the lowest shortest-path distance plus origin + cost, ties broken by rendezvous hashing (HRW) of the requested path and the + origin id, and forwards along its shortest path. Distance compares cost, + then hop count, so every hop strictly shortens it even across `?cost=0` + links. That is a shortest path to a virtual node linked to every origin, so + it is loop-free whenever relays agree on the topology. Specificity still + ranks first, per [Wildcard](/quest/m0/wildcard/README.md). +- The first relay's choice rides the SUBSCRIBE, and transit relays forward + toward that origin by topology alone, never re-selecting. Re-selection + against another existence view loops: a relay that lost a specific claim + falls back to a broader one through a relay still routing to the specific + one ([RFC 8966 section 3.5.4](https://www.rfc-editor.org/rfc/rfc8966#section-3.5.4)). + If the origin no longer serves the path, it refuses, and the first relay + selects again. +- SUBSCRIBE and FETCH carry a visited-relay list end to end. It catches loops + while liveness views disagree and names the path for stats. Narrowing it to + cluster hops is later work. The serving origin's identity rides the reply, + per Wildcard's Spread quest. +- Announcements are on demand. A relay forwards only the union of its clients' + ANNOUNCE_REQUEST prefixes, never the empty prefix. A wide prefix that many + edges' viewers request is that customer's cost. `.stats` becomes ordinary + demand. +- Registries are an optional, configured tier: moq-relay in a registry mode, + one or more per region. + - An ingest relay registers its broadcasts with its nearest registry, and an + edge sends its ANNOUNCE_REQUEST there. + - Registries form a small full mesh and flood existence among themselves, so + an event crosses an ocean once per remote registry, not once per relay. + Announce latency is about one round trip to the nearest registry, + whatever the path length. + - A relay fails over to the next-nearest registry and reconciles its view + instead of treating the lost session as ends, so a registry failure never + reports a live broadcast offline. + - With no registry reachable, a relay freezes: it keeps its view, learns + nothing new, and alerts. Falling back to flooding would cascade the + failure. +- Without registries (self-hosting), existence floods along the shortest-path + tree, one copy per relay. A relay forwards an event only when it changes its + view, so a duplicate copy, from trees built on disagreeing liveness, stops + there. +- Between clusters, announcements stay path vector with cluster ids as the + hops, like BGP between autonomous systems. A customer's on-prem cluster is + one hop, and an announcement naming the receiving cluster is dropped. +- The cluster switches versions as a whole; older lite and IETF sessions stay + at its edges. + +### Open questions + +- Sharding registries by HRW over a prefix key once one registry cannot hold + everything, and what that key is. +- A mixed-version bridge, if a fleet cannot switch at once. +- How long a relay keeps an ended path's seqno. A new origin incarnation + clears it; within one, it must outlive every delayed copy of the start. +- How an edge routes a SUBSCRIBE for a path none of its clients asked to + announce. It holds no route for it, and asking a registry first adds a round + trip before the first byte. +- What a cold ANNOUNCE_REQUEST reports as live. Answering from the local view + keeps a relay from waiting on peers but reports an empty set until the + registry's replay lands, on every new prefix rather than in a rare race. +- What remains of announce compression's hop-tail half (`Hop Base` and + `Hop Keep` in the lite draft) once only cluster boundaries carry hops. + +## Required + +- moq.pro's routing simulator reports ([quest](https://github.com/moq-dev/moq.pro/blob/main/quest/m1/routing-simulator.md)) + +## Related + +- [Skip unchanged announce updates](/quest/m0/announce-update-dedupe.md) - cuts duplicate updates on today's routing +- [Local origin](/quest/m0/local-origin.md) - workers stop reading hop chains before they go +- [Wildcard](/quest/m0/wildcard/README.md) - the specificity, pool spread, and reply identity this selection builds on diff --git a/quest/m1/path-patterns.md b/quest/m1/path-patterns.md index fff74556a5..90b43fb304 100644 --- a/quest/m1/path-patterns.md +++ b/quest/m1/path-patterns.md @@ -98,5 +98,5 @@ matches, containment refusal, and old-version behavior. ## Related -- [Wildcard advertisements](/quest/m1/wildcard/README.md) - routing adopts the +- [Wildcard advertisements](/quest/m0/wildcard/README.md) - routing adopts the matcher while retaining its own cost, pool, refusal, and resolution work diff --git a/quest/m1/pop-skipping/README.md b/quest/m1/pop-skipping/README.md index c1399c0b19..89a3652246 100644 --- a/quest/m1/pop-skipping/README.md +++ b/quest/m1/pop-skipping/README.md @@ -154,5 +154,5 @@ costs of one bidirectional session, which one `?cost=` cannot split. ## Related - [drain](/quest/m1/drain/README.md) - a second relay per PoP makes the same-PoP link price and its connection cardinality operationally important -- [wildcard](/quest/m1/wildcard/README.md) - it reuses this questline's route cost, and needs a cluster on Lite06 +- [wildcard](/quest/m0/wildcard/README.md) - it reuses this questline's route cost, and needs a cluster on Lite06 - [relay-memory](/quest/m1/relay-memory.md) - a denser mesh multiplies whatever a non-selected route costs diff --git a/quest/m1/processor/README.md b/quest/m1/processor/README.md index 93041435b6..dcc625f468 100644 --- a/quest/m1/processor/README.md +++ b/quest/m1/processor/README.md @@ -30,5 +30,5 @@ and custom transforms use the same worker lifecycle. - [Reference vision worker](/quest/m3/processor-vision.md) - a runnable worker publishes frame-correlated detections and proves demand, reconnect, failover, and teardown end to end -- [Wildcard advertisements](/quest/m1/wildcard/README.md) - lets a dormant +- [Wildcard advertisements](/quest/m0/wildcard/README.md) - lets a dormant processor advertise what it could serve without enumerating live sources diff --git a/quest/m2/README.md b/quest/m2/README.md index 5f1a80717d..33a1ac3dd6 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -55,7 +55,6 @@ upstream release waits in [m4](/quest/m4/README.md). - [Compressed tracks](/quest/m2/flate/README.md) - any track compresses per group from every language, not only the JSON modes - [Binary delta stats](/quest/m2/stats-delta.md) - an on-demand varint delta flavor of every stats track, if relay encode CPU still matters after the JSON fixes - [#3115](/quest/m2/3115-moqsink-the-publication-has-no-generation-so-a-flush.md) - moqsink: a flushing restart after EOS opens a new publication generation -- [Plan: routing without a hop list](/quest/m2/plan-routing-origin.md) - whether announcements can drop the hop list and stay loop-free once stitching keys on the subscribe reply - [Redundant ingest](/quest/m2/redundant-ingest.md) - decide whether two publishers sharing one epoch may splice, and who declares the incumbent dead before the keep-alive does - [Multipath spike](/quest/m2/multipath-spike.md) - whether bonded contribution over multipath QUIC is worth building, given it needs noq on both ends - [Receive timestamps](/quest/m2/quic-receive-ts.md) - per-packet arrival times in ACKs, the feedback GCC and deadlines need diff --git a/quest/m2/plan-routing-origin.md b/quest/m2/plan-routing-origin.md deleted file mode 100644 index 5a0c7f74f3..0000000000 --- a/quest/m2/plan-routing-origin.md +++ /dev/null @@ -1,41 +0,0 @@ -# [M] Plan: routing without a hop list - -## Goal - -Decide whether moq-lite announcements can drop the full hop list and still -route loop-free, then write the decision into quests. Today every -announcement carries the Hop IDs it traversed. That path does three jobs: -loop prevention (a relay discards a path containing itself), request -exclusion (never serve a request back through the peer that made it), and -route identity (the first hop decides whether a failover may stitch -seamlessly). The hop list was designed before prefix claims, and once the Spread quest in -[Wildcard advertisements](/quest/m1/wildcard/README.md) moves stitching -identity onto the subscribe reply, only the first two jobs remain. - -## Plan - -Open questions, to settle with the maintainer: - -- Loop freedom without a path. Rising cost alone ("cost + 1, never - advertise lower") counts to infinity after a withdrawal, as two relays - offer each other the stale route at ever higher cost. Remembering each - route's immediate neighbor and never echoing a route back to it (split - horizon) stops two-relay loops, but not three-relay loops. It also hides - backups: a relay never learns an alternative that passes back through the - neighbor it came from. Babel's feasibility condition (RFC 8966: a - per-origin sequence number; a route is feasible when its sequence number is - newer, at any cost, or equal and cheaper than the best seen for it) is - loop-free with only the origin on the wire. Weigh it and - any simpler scheme against what the relay cluster actually needs. -- What replaces request exclusion, and whether it still matters once - announcements are loop-free. -- What the stats and sidecar consumers lose without a path, and whether - anything besides the origin must stay on the wire. -- The version it lands in (lite-07 is `moq-lite-07-wip` today) and the - bridge to versions that still carry a hop list. lite-07 already compresses - each announcement's hop chain against a live base (#4196, - `rs/moq-net/src/lite/compress.rs`); dropping the hop list removes that too. - -## Related - -- [Wildcard advertisements](/quest/m1/wildcard/README.md) - its Spread quest moves stitching identity to the subscribe reply, which this builds on