diff --git a/quest/m0/README.md b/quest/m0/README.md index 5ec1c79c84..8263867ee7 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -47,7 +47,6 @@ Published API or wire breaks still land on dev; each quest's Plan says so. - [quest check everywhere](/quest/m0/quest-check-everywhere.md) - `quest check` guards `main`, `dev`, and the line branches on push and PR, not only PRs into `main` - [noq reassembly cap](/quest/m0/noq-reassembly-cap.md) - noq carries quinn's stream reassembly cap and the connection receive window is finite by default - [qmux reset race](/quest/m0/qmux-reset-race.md) - qmux handles RESET_STREAM under one lock instead of panicking -- [Path hunting](/quest/m0/path-hunting.md) - a withdrawn path is retracted about once per relay instead of hunting stale alternatives, with a hold-down on route updates and no wire change - [Shared fronts](/quest/m0/shared-fronts.md) - viewer sessions share a front, so fronts scale with peers, not viewers - [Frame alloc budget](/quest/m0/frame-alloc-budget.md) - frame buffers pre-allocate within a per-session budget and otherwise grow with bytes received - [Handshake deadline](/quest/m0/handshake-deadline.md) - an unfinished handshake or slow HTTP header times out diff --git a/quest/m0/path-hunting.md b/quest/m0/path-hunting.md deleted file mode 100644 index 7ca416b246..0000000000 --- a/quest/m0/path-hunting.md +++ /dev/null @@ -1,49 +0,0 @@ -# [M] A withdrawn path stops hunting - -## Goal - -When a broadcast ends, each relay in a lite-06 mesh retracts it about once, -instead of re-announcing stale alternatives. Today one withdrawal on live's -34-relay graph costs hundreds to over a thousand extra announce starts, and -on Kick dev an ended name drew 575 to 57,805 mesh announce starts (median -about 13,000) while live names drew one flood (moq-dev/moq.pro#2116). New -and removed routes still propagate at once, and no wire changes. - -## Plan - -Promoted from the cluster routing line to m0 on 2026-09-30: it is today's -partial-mesh hunting that #4399 left open, and it is hurting production. - -Decided (2026-09-30), lite-06, no wire change: - -- A route *update*, where a relay's advertised route for a path switches to a - different route, waits a short hold-down before it is re-advertised. A new - path (nothing advertised yet) and a removal (nothing left) go out at once. -- So when the best route is withdrawn and an alternative remains, the relay - retracts at once and re-announces the alternative only if it survives the - hold-down. Stale alternatives are withdrawn inside that window and never - spread. A genuine failover, where a link dies but the origin still - publishes, is delayed by the hold-down; pick the value from the repro, and - report the failover cost it adds. -- Check the interaction with #4399's withdrawal hiding and with - subscriptions already pinned to a route (fronts keep serving from a live - source; only advertisements are held). - -Test first: land moq.pro's in-process repro upstream (real lite-06 sessions -over the mock transport on live's 34-relay graph, mocked time), asserting -about one retraction per relay for a single withdrawal from a non-hub -publisher, plus a failover test where a link dies and the path comes back -through another route after the hold-down. It must fail on `main` today. - -Then moq.pro moves its pin and confirms about one retraction wave per ended -name on Kick dev after a deploy the user approves (moq.pro's own quest). - -If no hold-down closes it, the fallback is a per-origin seqno on cluster -links in the wip lite version; report before building it. - -Public API: none expected. Wire: none. - -## Related - -- [Cluster routing](/quest/m1/cluster-routing/README.md) - tiers shrink the core graph this hunts on -- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - closed broadcasts announced for minutes, the same symptom diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index 386f8655e9..abbf20e513 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -39,9 +39,11 @@ link-state and existence-split design planned earlier that day: path vector with hop lists stays loop-free, path hunting is confined to the cores, liveness flooding (the simulator's dominant link-state cost) is never paid, and failure detection sets the outage window either way. The parked - implementation is #4631. If stale paths remain on the tiered layout, the - fix is [Path hunting](/quest/m0/path-hunting.md), promoted to m0: a hold-down - on route updates, with no wire change. + implementation is #4631. Stale paths after a withdrawal are ended by + per-origin seqnos on cluster links + ([Path hunting](/quest/m1/cluster-routing/path-hunting.md)); a lite-06 + hold-down (#4644) was rejected because it breaks seamless failover, and + #4642's cursor hold mitigates production meanwhile. - The line lands on `main`: its children are additive. The two breaking changes left it for `dev` on their own: [Remove `--hop`](/quest/m1/hop-removal.md) and @@ -118,6 +120,7 @@ Once every child has landed: - [Edge and core](/quest/m1/cluster-routing/roles.md) - relays take an explicit edge or core role; edges spread paths over their region's cores and are never transit - [Selection](/quest/m1/cluster-routing/selection.md) - a broadcast under overlapping prefixes routes to one origin deterministically, and same-epoch origins are one source +- [Path hunting](/quest/m1/cluster-routing/path-hunting.md) - per-origin seqnos on cluster links end stale re-announces without breaking seamless failover - [Between clusters](/quest/m1/cluster-routing/inter-cluster.md) - announcements crossing a cluster boundary stay path vector with cluster ids as hops ## Related diff --git a/quest/m1/cluster-routing/path-hunting.md b/quest/m1/cluster-routing/path-hunting.md new file mode 100644 index 0000000000..15e91afe4f --- /dev/null +++ b/quest/m1/cluster-routing/path-hunting.md @@ -0,0 +1,51 @@ +# [M] Per-origin sequence numbers end path hunting + +## Goal + +When a broadcast ends, each relay retracts it once and never re-announces a +stale alternative, and a failover where the origin still publishes stays +seamless: no downstream front loses its route while a working one exists. + +## Plan + +History (2026-09-30 to 10-01): moq-dev/moq.pro#2116 measured ended names +drawing 575 to 57,805 mesh announce starts on Kick dev, and reproduced it on +live's 34-relay graph. #4642's 300 ms announce-cursor hold mitigates it and +is what production runs; it still leaves 16 of 34 publishers hunting in the +repro. #4644 added a cluster-link hold-down (retract at once, re-announce a +replacement after 1 s) and measured 0 resurrections, but the maintainer +rejected it: the immediate retraction ends a downstream peer's front on +`best: None` when its only path ran through the relay, so subscribers drop +for about 1.1 s while the relay keeps serving. Holding the replacement +without retracting brought hunting back. A lite-06 announce-side fix cannot +avoid that retraction. + +Decided: carry a per-origin seqno, scoped to the origin's incarnation, with +each announcement on cluster links in the wip lite version, and never apply +an event older than one already seen for that path and origin (DSDV and +Babel feasibility). A withdrawal then outranks every stale copy without a +hold-down or a retraction of live alternatives. + +- Start from #4644's branch (`quest/m0/path-hunting`): its live-graph + regression tests (withdrawal at several latencies, and failover) fail on + `main` and stay the acceptance tests. Their fixture hard-codes lite-06; + run them on the wip version. Add a check that a downstream in-flight + subscription survives the failover. +- Keep an ended path's seqno long enough to outlive delayed copies of its + start; a new incarnation clears it. +- Decide whether #4642's cursor hold can then be removed. +- Clients and lite-06 cluster peers are unchanged; only wip-version cluster + links carry the field. + +Wire: a field on ANNOUNCE_START, ANNOUNCE_UPDATE, and ANNOUNCE_END in the +wip lite version, with `drafts/draft-lcurley-moq-lite.md` updated in the +implementation PR. A failover that switches origin or advances its seqno +travels as ANNOUNCE_UPDATE (`AnnounceBroadcast::Restart`), so the update +carries its own seqno; forcing END then START would bring back the +retraction this quest removes. Add a regression that advances the seqno via +an update without retracting the live advertisement. + +## Related + +- [Edge and core](/quest/m1/cluster-routing/roles.md) - tiers shrink the core graph this hunts on +- [Cross-relay delivery under bursts](/quest/m1/cross-relay-bursts.md) - closed broadcasts announced for minutes, the same symptom