From 01675e131654cc152254eebb86831ae73fe49b7d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 10 Oct 2026 15:30:00 -0700 Subject: [PATCH] quest: plan pinned requests, catalog epochs, and fetch budgets Co-Authored-By: GPT-6 --- quest/m0/broadcast-epoch/README.md | 25 ++++++- .../m0/broadcast-epoch/catalog-references.md | 74 +++++++++++++++++++ .../m0/broadcast-epoch/js-consume-identity.md | 1 + quest/m0/broadcast-epoch/resume-duplicates.md | 38 ++++++++++ quest/m0/broadcast-epoch/retired-requests.md | 65 ++++++++++++++++ quest/m0/broadcast-epoch/source-pin.md | 5 ++ quest/m1/README.md | 2 + quest/m1/cache-max-age.md | 9 ++- quest/m1/catalog-track-id.md | 1 + quest/m1/catalog-tracks.md | 1 + quest/m1/fetch-max-delay.md | 74 +++++++++++++++++++ quest/m1/ietf-epochs.md | 4 +- quest/m1/ietf-fill-timeout.md | 55 ++++++++++++++ quest/m1/js-fetch.md | 2 + quest/m1/js-group-handover.md | 22 ++++-- quest/m1/lite07-finalize.md | 1 + quest/m1/pipeline-requests/fetch.md | 2 + quest/m1/resume-reorder.md | 5 ++ quest/m1/subscribe-drop.md | 19 ++++- quest/m1/subscribe-ranges/README.md | 7 +- quest/m1/subscribe-ranges/ietf.md | 2 + quest/m1/subscribe-ranges/model.md | 13 +++- 22 files changed, 410 insertions(+), 17 deletions(-) create mode 100644 quest/m0/broadcast-epoch/catalog-references.md create mode 100644 quest/m0/broadcast-epoch/resume-duplicates.md create mode 100644 quest/m0/broadcast-epoch/retired-requests.md create mode 100644 quest/m1/fetch-max-delay.md create mode 100644 quest/m1/ietf-fill-timeout.md diff --git a/quest/m0/broadcast-epoch/README.md b/quest/m0/broadcast-epoch/README.md index 9398e937e9..8f2e7d0521 100644 --- a/quest/m0/broadcast-epoch/README.md +++ b/quest/m0/broadcast-epoch/README.md @@ -65,8 +65,13 @@ Decided: lite-07 claim's answer carries the served broadcast's own epoch, no longer gates this line (decided 2026-10-08): it is a lite-07 opt-in, so it moved to m1. -- A catalog `broadcast` reference by name follows the newest epoch, since a - path cannot name one. +- Decided in the 2026-10-10 resume audit: a catalog reference pins its first + resolved sibling for the catalog instance's lifetime in players and + exporters. Optional reference epochs constrain the first resolution on + epoch-aware transports; on epochless transports they remain best effort. + This supersedes following the newest sibling by name. The + [catalog references quest](/quest/m0/broadcast-epoch/catalog-references.md) + owns the schemas, readers, and compatibility documentation. - Every first-party publisher that can restart mints its own: the apps, moq-boy, the ingest gateways, moqsink, and the bindings below. Players (`moq play`, `@moq/watch`, demo/web) follow the announce `Restart`. moq-stats mints one per @@ -103,11 +108,27 @@ its `doc/concept/stats.md` sentence from `release`; until then, the maintainer's pre-release merge of `release` into `main` keeps `main`'s `rs/moq-stats` and `doc/concept/stats.md`. MoQ Pro's VOD `storage.json` moves from the same seed to its own epoch. +The 2026-10-10 resume audit adds release gates for newly issued requests +through retired handles and for late duplicate delivery after same-epoch +handover. Keep existing reads sticky; stopping new unpinned wire work is a +separate boundary. The catalog-reference work applies the same pinning +policy to players and exporters. Wall-clock expiry, live DROP dispositions, +FETCH budgets, seamless JS handover, and the IETF extension remain m1. + +Extend the line's relay/media tests to hold catalog and media from A while B +replaces it, exercise later metadata/subscription/FETCH operations, and prove +that following the replacement starts a fresh catalog with fresh pins. Check +exact and covering routes, matching/different/absent epochs, and legacy +End+Start. The child quests own their focused regressions. + ## Required +- [Retired requests](/quest/m0/broadcast-epoch/retired-requests.md) - a held broadcast or track cannot open unpinned wire work against its replacement +- [Resume duplicates](/quest/m0/broadcast-epoch/resume-duplicates.md) - a delayed group is delivered once even beyond the old 1024-entry history - [JS consume identity](/quest/m0/broadcast-epoch/js-consume-identity.md) - requests carry the serving prefix's epoch and new winners cannot reuse an old cached broadcast - [JS restart keeps the request](/quest/m0/broadcast-epoch/js-restart-keeps-request.md) - a resolved request remains on its old instance, matching Rust's sticky subscriptions - [Source pin](/quest/m0/broadcast-epoch/source-pin.md) - `Source` is built from the resolved catalog broadcast and every later request stays on it, so an epochless replacement never splices into the old program +- [Catalog references](/quest/m0/broadcast-epoch/catalog-references.md) - players and exporters pin siblings consistently, with optional reference epochs and documented legacy best effort - [Publish catalog restart](/quest/m0/broadcast-epoch/publish-catalog-restart.md) - `@moq/publish` never reuses catalog group numbers under one name and epoch after a re-announce - [Bindings](/quest/m0/broadcast-epoch/bindings.md) - moq-ffi and every wrapper expose the epoch and let a publisher announce one - [Stats totals and prefix tracks](/quest/m0/broadcast-epoch/stats-split.md) - the same release retires the per-path stats maps for totals and on-demand prefix tracks (decided 2026-10-05) diff --git a/quest/m0/broadcast-epoch/catalog-references.md b/quest/m0/broadcast-epoch/catalog-references.md new file mode 100644 index 0000000000..3939fe58b1 --- /dev/null +++ b/quest/m0/broadcast-epoch/catalog-references.md @@ -0,0 +1,74 @@ +# [L] Catalog references pin their broadcast instances + +## Goal + +Catalog, media, and data requests use the same pinned broadcast instance +in players and exporters. Each sibling stays on its first resolved +instance for the catalog instance's lifetime. Optional catalog epochs let +a reference name its intended sibling before that first resolution. + +## Plan + +Decided in the 2026-10-10 resume audit: + +- Apply the same pinning policy to players and exporters. This replaces + the earlier rule that a named sibling follows the newest epoch, including + `@moq/watch`'s test that follows a republished rendition independently of + its catalog. A sibling replacement does not silently retarget the old + catalog's reference. A new catalog instance or explicit export stitch + starts fresh pins; same-epoch recovery preserves identity. +- Add optional `epoch` beside every catalog broadcast reference: video, + audio, text, JSON, and binary configs, plus the archive replay reference. + Reuse the canonical UUIDv7 representation and validation from MoQ epochs. + There is no new top-level catalog epoch. Self-references use the already + pinned catalog broadcast; an explicit value must not contradict its + known identity. +- An explicit reference epoch constrains the initial resolution and later + requests wherever the transport can express and verify it. A known + mismatch fails instead of falling back to the current winner. Conflicting + identities for an already-pinned path cannot replace that pin through a + catalog update. +- Omission remains compatible with existing catalogs: pin the first + resolved instance, with no promise about an unresolved sibling's past. +- On epochless transports, explicit catalog epochs are best effort + (maintainer choice). Keep the requested identity in the reference, use + whatever identity the route can verify, and pin the resolved handle. + Unknown identity alone is not a refusal. Document that this cannot prove + the requested epoch, and never promote an unverified catalog value into + transport evidence permitting seamless resume. Older readers can ignore + the additive field and do not gain its guarantee. + +Use the shared reference resolution paths rather than adding one rule per +media type. Mirror schemas and behavior in `rs/hang`, `js/hang`, native +readers/exporters and `@moq/watch`; carry the field through catalog producers, +conversions and bindings that expose these configs. A conversion that +cannot preserve the requested identity must report that limitation rather +than silently discard it. Coordinate with catalog rendition IDs, which +changes the neighboring track-name field but is not a prerequisite. + +Test all reference-bearing sections in both languages: omitted/matching/ +mismatched/malformed epochs, same and sibling paths, a replacement before +first resolution and after pinning, late rendition additions, updates that +conflict with a pin, new catalog instances, and an epochless transport with +an explicit reference. Catalog and media must never independently resolve +the same path across a replacement. Include archive replay references and +concurrent old/new consumers. Use existing unit, media and interop CI lanes; +run `just check`, `just test interop --all`, and `just drafts check`. + +Public API: additive optional catalog fields and changed player sibling +following behavior. Wire: additive Hang catalog fields, no transport +framing change. Update the Hang draft, existing catalog/identity docs, and +upgrade notes inline; no separate guide. Explain both the strong epoch-aware +guarantee and the selected legacy best-effort behavior. + +## Required + +- [Source pin](/quest/m0/broadcast-epoch/source-pin.md) - exporters share their resolved catalog handle +- [JS consume identity](/quest/m0/broadcast-epoch/js-consume-identity.md) - JS resolves the actual serving identity + +## Related + +- [Retired requests](/quest/m0/broadcast-epoch/retired-requests.md) - held handles cannot emit unpinned requests after retirement +- [Catalog rendition IDs](/quest/m1/catalog-track-id.md) - the adjacent optional wire-track field +- [Catalog track identity](/quest/m1/catalog-tracks.md) - catalog updates cannot redefine a track's content +- [IETF epochs](/quest/m1/ietf-epochs.md) - verified catalog identity over negotiated IETF sessions diff --git a/quest/m0/broadcast-epoch/js-consume-identity.md b/quest/m0/broadcast-epoch/js-consume-identity.md index 292476c65a..33935f89e5 100644 --- a/quest/m0/broadcast-epoch/js-consume-identity.md +++ b/quest/m0/broadcast-epoch/js-consume-identity.md @@ -45,3 +45,4 @@ no framing change, and older versions stay epochless. ## Related - [JS track handover](/quest/m1/js-group-handover.md) - preserves open tracks across routes of the same explicit identity +- [Retired requests](/quest/m0/broadcast-epoch/retired-requests.md) - separately prevents old callable handles from opening wire work against replacements; this quest owns fresh-consumer selection and cache reuse diff --git a/quest/m0/broadcast-epoch/resume-duplicates.md b/quest/m0/broadcast-epoch/resume-duplicates.md new file mode 100644 index 0000000000..3fa1765f03 --- /dev/null +++ b/quest/m0/broadcast-epoch/resume-duplicates.md @@ -0,0 +1,38 @@ +# [M] Resume never delivers a late group twice + +## Goal + +An unordered reader receives each group once across same-epoch route +changes, even when a duplicate arrives very late. A genuinely unseen late +group remains eligible within the reader's bounds and budget. + +## Plan + +Reproduced in the 2026-10-10 audit: read group 0 from A, switch to B, read +groups 1 through 1024, then deliver B's delayed copy of group 0. The reader +receives group 0 twice. `model/resume.rs` forgets delivered IDs at +`MAX_DELIVERED = 1024` without proving those groups can no longer arrive. + +Decided: this data-correctness fix gates m0. Preserve both duplicate +suppression and unordered delivery; advancing a floor past every late +group would hide the duplication by losing valid data. Bound bookkeeping +using the actual reader/copy lifetime, retained ranges, and terminal +dispositions rather than an arbitrary number of recently delivered IDs. +Choose the representation during implementation; do not retain one entry +per delivered group for the lifetime of an endless broadcast. + +Land the failing regression, a genuinely unseen late group beyond the old +window, repeated route flaps, and ordered-reader controls. Cover readers +that keep old groups open and change their group bounds, and prove finished +copies release their bookkeeping. Tests use mocked time and existing CI +lanes. Extend the resume benchmark over readers and retained/reordered +groups so memory and delivery costs are measured. + +Public API: duplicate suppression is corrected; no signature changes. +Wire: none. Update stale resume comments inline. JS handover must preserve +the corrected contract instead of copying the fixed-size history. + +## Related + +- [Resume reorder](/quest/m1/resume-reorder.md) - an unseen reordered group is not mistaken for a missing one +- [JS track handover](/quest/m1/js-group-handover.md) - the same delivery contract in JS diff --git a/quest/m0/broadcast-epoch/retired-requests.md b/quest/m0/broadcast-epoch/retired-requests.md new file mode 100644 index 0000000000..52d448d232 --- /dev/null +++ b/quest/m0/broadcast-epoch/retired-requests.md @@ -0,0 +1,65 @@ +# [L] Retired handles never request a replacement's content + +## Goal + +A Rust or JS broadcast or track handle keeps its resolved instance for +TRACK, SUBSCRIBE, and FETCH. Once that instance retires, a new wire request +cannot resolve the replacement by path. Existing reads and cached content +from the old instance remain usable within their normal lifetime and budgets. + +## Plan + +Reproduced in the 2026-10-10 resume audit: resolve catalog and media from A, +publish B at the same path, wait for the replacement announcement, then +FETCH a missing group through A's held media track. Rust returns B's bytes +on lite-05/06, epochless lite-07, and IETF drafts 14-22. Explicit-epoch +lite-07 refuses the stale request. JS also accepts a new TRACK_INFO through +a held epochless broadcast after End+Start on lite-06/07. + +Decided by the maintainer in that audit: + +- This correctness fix gates m0. Supporting epochless peers on updated + relays remains required; neither new paths nor the IETF epoch extension + is a prerequisite. +- Preserve instance validity through every handle and deferred operation, + including a track handed out before retirement, coalesced requests, and + a request whose setup races withdrawal. A fresh request may proceed only + when it still names the held instance. An explicit wire epoch can prove + that; an epochless path after retirement cannot. +- Keep already-open reads sticky and allow old cached bytes. Do not close + every old subscription to prevent new requests. A rejected operation must + settle its waiter and release its demand, without migrating it to B. +- Same explicit epoch permits route replacement and recovery. Absent + epochs never prove continuity. Before an epochless peer observes a + replacement, its wire cannot identify the old instance; do not claim + this fixes arbitrary legacy caches or that unavoidable ambiguity. + +Fix validity at the request/source boundary shared by these operations, +rather than patching individual exporters. `Source` handle pinning and the +JS consume cache fix select the right initial handle but do not prevent a +surviving handle from opening new path-based wire work. Coordinate with +those active quests without duplicating their cache-selection changes. +IETF standalone FETCH resolves a path, whereas joining FETCH reads the +saved subscription cache; preserve that distinction and check both. + +Add mocked-time regressions with A still alive after B is announced: held +catalog plus held media FETCH, late track lookup, re-subscription, pending +TRACK/FETCH setup, concurrent old/new consumers, exact and covering routes, +and source withdrawal without replacement. Test matching, different, and +absent epochs. Keep the explicit lite-07 epoch control and the failing +lite-05/06, epochless lite-07, and IETF14-22 matrix in CI; cover supported +legacy request forms and refusals too. Run `just check` and +`just test interop --all`. Benchmark any routing or invalidation fanout over +both routes and held consumers. + +Public API: corrected request lifetime and refusal behavior; preserve +readable old subscriptions. Wire: none; published framing is unchanged. +Update existing identity and request-lifetime documentation inline. + +## Related + +- [Source pin](/quest/m0/broadcast-epoch/source-pin.md) - the exporter holds the exact catalog broadcast +- [JS consume identity](/quest/m0/broadcast-epoch/js-consume-identity.md) - fresh consumers select the serving announcement's identity +- [JS restart keeps the request](/quest/m0/broadcast-epoch/js-restart-keeps-request.md) - an existing resolved request stays sticky +- [IETF epochs](/quest/m1/ietf-epochs.md) - explicit wire identity removes the epochless ambiguity +- [Pipelined first FETCH](/quest/m1/pipeline-requests/fetch.md) - staged requests must retain the same instance validity diff --git a/quest/m0/broadcast-epoch/source-pin.md b/quest/m0/broadcast-epoch/source-pin.md index 2b04faa45f..e6f1656822 100644 --- a/quest/m0/broadcast-epoch/source-pin.md +++ b/quest/m0/broadcast-epoch/source-pin.md @@ -63,3 +63,8 @@ of resolving the current route. Document it on `Source`, fix stale comments inline, and add a line for the `Source::new` break to the Unreleased section of `doc/setup/upgrade.md`; no new doc page (decided 2026-10-10). Wire: none. + +## Related + +- [Retired requests](/quest/m0/broadcast-epoch/retired-requests.md) - held track FETCH and JS wire operations also need instance validity after retirement; handle pinning alone does not fix them +- [Catalog references](/quest/m0/broadcast-epoch/catalog-references.md) - players adopt the same sibling policy, and optional catalog epochs constrain initial resolution where supported diff --git a/quest/m1/README.md b/quest/m1/README.md index efb4f35a3f..eb9c3857c9 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -43,6 +43,7 @@ bench all require its crate. ## Required - [One max_age meaning](/quest/m1/cache-max-age.md) - a superseded group goes stale on wall clock since its successor arrived or on media time, whichever is first, in Rust and js/net; fixes the untimed failover stall +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - historical readers obey publisher max-age and an optional tighter reader budget, including while holding a group - [Upstream position regression](/quest/m1/largest-regression.md) - a relay copy that sees upstream's largest group go backwards on moq-transport or epochless lite-07 fails loud instead of serving the old instance's cache - [Untimed decisions](/quest/m1/untimed-decisions.md) - the maintainer decides whether moq-archive keeps refusing untimed tracks, and whether a malformed FETCH object ends its track - [Untimed by default in Rust](/quest/m1/rust-untimed-default.md) - an undeclared Rust timescale means untimed, and shared-clock publishers declare milliseconds @@ -84,6 +85,7 @@ bench all require its crate. - [JSON window timestamps](/quest/m1/json-window-timestamps.md) - Rust and JS window consumers return each event's frame timestamp like the snapshot and stream consumers - [IETF object gaps](/quest/m1/ietf-object-gaps.md) - a gapped object ID is refused loudly like a subgroup in Rust and JS, and an overflowing one closes the session in Rust - [JavaScript FETCH](/quest/m1/js-fetch.md) - browser publishers answer IETF FETCH through the JS ranges request surface +- [IETF fill timeout](/quest/m1/ietf-fill-timeout.md) - native FILL_TIMEOUT bounds upstream waiting on drafts 18-22, alongside local content-age enforcement - [Relay session limits](/quest/m1/relay-session-limits.md) - moq-relay sets per-session request limits, tighter for clients than peers, and the bindings name a refused request - [Churn with held subscriptions](/quest/m1/session-churn-held.md) - opening and closing a request costs the same with 1 or 1,024 held subscriptions - [Catalog track identity](/quest/m1/catalog-tracks.md) - a track's codec and description never change for its name; resolution changes in band below ceilings fixed at creation, and anything else mints a new rendition or epoch diff --git a/quest/m1/cache-max-age.md b/quest/m1/cache-max-age.md index 71f34afd6c..5a9db9dedf 100644 --- a/quest/m1/cache-max-age.md +++ b/quest/m1/cache-max-age.md @@ -94,8 +94,12 @@ Facts, and work carried over: until the track ends; the replaced copy's lease stays alive too. The wall term convicts it. Settle what "nothing pending" means against the recovery fetch (`a_pending_recovery_fetch_does_not_disable_expiry`), so a fetch - about to fill the group isn't cut short. A FETCH reader has no budget and - stays as it is. + about to fill the group is still subject to its reader's deadline. Do not + make a pending fetch disable expiry. Decided in the 2026-10-10 audit: + FETCH also obeys publisher max-age and an optional reader max-delay; + omission uses the publisher's limit alone. This supersedes the old FETCH + exemption. [FETCH max-delay](/quest/m1/fetch-max-delay.md) owns that API, + request propagation and per-reader enforcement, building on this rule. - Start resolution: the untimed model starts an untimed track at the latest group (`TrackState::untimed_start`). Replace that special case with the normal rule, replaying the cached groups that aren't stale. Until then a @@ -126,3 +130,4 @@ encoding change; Max Age semantics in the lite draft change. - [JS track handover](/quest/m1/js-group-handover.md) - mirrors the failover rule in JS - [Cache expiry growth](/quest/m1/cache-expiry-growth.md) - relay memory past the expiry window, in the same cache - [Generated @moq/net](/quest/m1/rs2ts/README.md) - retires js/net's hand-written model +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - applies the shared rule to historical requests and reconciles native IETF retention clocks diff --git a/quest/m1/catalog-track-id.md b/quest/m1/catalog-track-id.md index fcaa8188a8..d19255fb29 100644 --- a/quest/m1/catalog-track-id.md +++ b/quest/m1/catalog-track-id.md @@ -56,3 +56,4 @@ Rejected: an ID plus a required `track`. - [Media stats](/quest/m1/stats/README.md) - keys stats and feedback by this ID +- [Catalog references](/quest/m0/broadcast-epoch/catalog-references.md) - optional epoch beside broadcast across every reference-bearing section, with shared instance pinning diff --git a/quest/m1/catalog-tracks.md b/quest/m1/catalog-tracks.md index fb6817cc94..6d355e1da0 100644 --- a/quest/m1/catalog-tracks.md +++ b/quest/m1/catalog-tracks.md @@ -77,3 +77,4 @@ the hang catalog's field semantics, documented in its draft. - [Catalog rendition IDs](/quest/m1/catalog-track-id.md) - lets a catalog list a new track name for a changed rendition - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a restart is a new epoch rather than a changed track - [Archive](/quest/m1/archive/README.md) - storage and replay consume the identity contract +- [Catalog references](/quest/m0/broadcast-epoch/catalog-references.md) - an update cannot retarget an already-pinned sibling instance diff --git a/quest/m1/fetch-max-delay.md b/quest/m1/fetch-max-delay.md new file mode 100644 index 0000000000..067b7b6fa5 --- /dev/null +++ b/quest/m1/fetch-max-delay.md @@ -0,0 +1,74 @@ +# [L] FETCH enforces publisher and reader age limits + +## Goal + +A historical FETCH obeys publisher `max_age` and an optional reader +`max_delay` in Rust and JS, including while a returned group is being read. +Omitting max-delay uses publisher max-age alone. An explicit delay can +only tighten the publisher's limit. Range subscriptions that replace FETCH +preserve these semantics. + +## Plan + +Decided in the 2026-10-10 resume audit: FETCH is not exempt from content +age limits. The maintainer explicitly requested max-delay on FETCH. The +public Rust `group::Fetch` and JS `FetchGroupOptions` have no such option +today, and FETCH readers bypass the live subscription's expiry guard. + +- Add optional `max_delay` / `maxDelay` to the request surfaces and carry + it through local, relayed, and resumed fetches. Omission means no extra + reader cap, not SUBSCRIBE's zero-delay default. An explicit zero remains + a zero freshness budget. Neither omission nor a large reader budget + overrides publisher retention. +- Use the settled successor-based staleness rule from One max_age meaning: + either wall age or media drift can expire a superseded group; the newest + group has no successor and retains that exception. Apply the budget to + cache hits, misses, in-flight recovery, and handed-out readers. A pending + upstream fetch cannot suppress a deadline wake or resurrect expired data. +- Shared FETCH work must not let a tolerant caller relax another caller's + budget, nor let one expired/cancelled caller kill data another still needs. + Reuse the model's existing per-reader budget and aggregate-demand rules. +- Preserve published lite and IETF framing. Enforce the budget locally on + every supported wire. Add the field to unpublished lite-07 FETCH if it + still exists when this lands; if ranges already replaced it, carry it + through their existing budget instead. Do not reintroduce removed FETCH. +- IETF upstream waiting uses native FILL_TIMEOUT where supported, through + the separate quest below. It is an approximation for waiting, not proof + of freshness. Exact content-age enforcement stays local. No custom + IETF max-delay extension is planned. +- Check the IETF adapter's [MAX_CACHE_DURATION](https://www.ietf.org/archive/id/draft-ietf-moq-transport-22.html#section-10.3) interpretation: that native + property bounds caching from object receipt, not from a group's successor. + Honor the native publisher bound as well; do not claim numeric conversion + into `max_age` makes the two clocks identical or permits retaining an IETF + object past its native bound. + +Update all fetch callers and exposed bindings under Cross-Package Sync, +including CLI, relay, archive and media readers. Keep the API cohesive with +range subscriptions rather than adding a parallel historical-only budget. +Archive replay still works for content its replay publisher makes available +within its declared retention; stale live-cache data is not an exemption. + +With mocked time, test omitted, zero, tighter and looser budgets; cache hit +and cold fetch; timed/untimed tracks; no writes after a successor; reordered +or aborted successors; reading past expiry; route replacement; and coalesced +callers with different budgets and cancellation. Test publisher max-age +clamping at each supported wire boundary and IETF native retention clocks. +Run existing Rust/JS and `just test interop --all` lanes, `just check`, and +`just drafts check` for lite changes. Benchmark budgeted fanout if the +implementation changes shared expiry or request bookkeeping. + +Public API: new optional FETCH budget across Rust, JS and exposed wrappers; +historical reads can now expire. Wire: only unpublished lite-07 gains a +field if needed; published formats are unchanged. Update API, concept, +CLI and matching draft documentation inline, including the changed FETCH +exemption and the distinction between age and upstream wait budgets. + +## Required + +- [One max_age meaning](/quest/m1/cache-max-age.md) - one expiry rule and deadline wake mechanism + +## Related + +- [IETF fill timeout](/quest/m1/ietf-fill-timeout.md) - upstream waiting bounded through the existing FETCH parameter +- [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - historical ranges retain this optional budget and publisher clamp +- [Retired requests](/quest/m0/broadcast-epoch/retired-requests.md) - identity stays pinned independently of age diff --git a/quest/m1/ietf-epochs.md b/quest/m1/ietf-epochs.md index 65925d301e..c756ab0bee 100644 --- a/quest/m1/ietf-epochs.md +++ b/quest/m1/ietf-epochs.md @@ -79,4 +79,6 @@ extension; no changes to unnegotiated published versions or broadcast paths. - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - current lite-07 identity and legacy restart behavior - [Claim epochs](/quest/m1/claim-epochs.md) - shared broadcast identity and lite-07 response identity - [IETF claim epochs](/quest/m1/ietf-claim-epochs.md) - learn per-output identity from an IETF claim response -- [JS track handover](/quest/m1/js-group-handover.md) - a stable JS track pump for same-epoch route changes +- [JS track handover](/quest/m1/js-group-handover.md) - stable JS readers across same-epoch route changes +- [Catalog references](/quest/m0/broadcast-epoch/catalog-references.md) - optional requested epochs become verifiable on this negotiated transport; epochless fallback remains best effort +- [IETF fill timeout](/quest/m1/ietf-fill-timeout.md) - native wait-budget support is separate from epoch negotiation and adds no age extension diff --git a/quest/m1/ietf-fill-timeout.md b/quest/m1/ietf-fill-timeout.md new file mode 100644 index 0000000000..e730cdfb21 --- /dev/null +++ b/quest/m1/ietf-fill-timeout.md @@ -0,0 +1,55 @@ +# [L] IETF FETCH honors its fill timeout + +## Goal + +Rust and JS use IETF FILL_TIMEOUT to bound waiting for missing FETCH +objects on drafts 18-22, and return the draft's timeout-gap disposition +without hanging or discarding objects that are available. Drafts 14-17 +keep their existing wire; local FETCH age enforcement works on all versions. + +## Plan + +Decided in the 2026-10-10 resume audit: reuse the native parameter instead +of introducing a negotiated content-age extension. DELIVERY_TIMEOUT and +its later OBJECT_/SUBGROUP_ forms are subscription parameters, not FETCH +parameters. [FILL_TIMEOUT](https://www.ietf.org/archive/id/draft-ietf-moq-transport-22.html#section-9.20.5) +bounds the total wait for missing upstream objects; zero is cache-only. +It does not judge how old an immediately available object is. Keep exact +max-delay/max-age enforcement in the model alongside this approximation. + +Today Rust decodes only FILL_TIMEOUT's presence and refuses such requests +because it cannot emit the required gap statuses. Preserve the value and +honor its semantics before advertising or sending support. JS's IETF FETCH +responder comes from its existing quest; build on that surface. + +Propagate an explicit FETCH max-delay as the upstream wait budget where +the draft supports it. For omitted max-delay, retain the wire's omitted +FILL_TIMEOUT behavior; publisher retention still applies locally. A zero +mapped budget means cache-only on this wire, an accepted difference from +a pure content-age test. Do not reset the total wait on every missing +object or group. Shared upstream work retains each caller's own deadline +and cancellation, as it retains each age budget. + +Implement each draft's correct timed-out/unknown range status and the +receiving side's terminal disposition; a timeout must not become a claim +that the content never existed. Do not wait forever for the timed-out +range, reset the entire session, or silently reinterpret FILL_TIMEOUT as +publisher retention. Test standalone and applicable joining/fill forms, +reordered arrivals, cache-only requests, partial availability, no subsequent +writes, cancellation, shared callers, and an older peer without this field. +Use mocked time and the existing Rust/JS and interop CI lanes; run +`just check` and `just test interop --all`. + +Public API: reuse FETCH's max-delay option; no new content-age extension +configuration. Wire: support existing IETF parameters and gap statuses on +their applicable versions. Update existing interop/deviation and FETCH docs +inline, clearly distinguishing upstream wait time from content age. + +## Required + +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - per-caller age budgets and their public option +- [JavaScript FETCH](/quest/m1/js-fetch.md) - the JS responder this extends + +## Related + +- [moq-transport ranges](/quest/m1/subscribe-ranges/ietf.md) - sparse range forwarding and gap handling diff --git a/quest/m1/js-fetch.md b/quest/m1/js-fetch.md index 36c7e9189a..fb0cdc25e7 100644 --- a/quest/m1/js-fetch.md +++ b/quest/m1/js-fetch.md @@ -45,3 +45,5 @@ claims that change. ## Related - [Browser archive](/quest/m3/archive-browser.md) - supplies memory or OPFS archive data through the same request surface +- [IETF fill timeout](/quest/m1/ietf-fill-timeout.md) - native timeout budgets and gap dispositions build on this responder +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - local per-reader age limits also apply to historical data diff --git a/quest/m1/js-group-handover.md b/quest/m1/js-group-handover.md index c7719e71cf..40e0fe60bb 100644 --- a/quest/m1/js-group-handover.md +++ b/quest/m1/js-group-handover.md @@ -22,13 +22,14 @@ old relay drops its upstream pull, the new relay subscribes upstream from scratch at the live edge. A group boundary that lands inside that window loses the group in flight: about 1 run in 5 at 100 ms groups. -Rust solves this with #4741's single-writer pump (`model/resume.rs`): each track a -front serves is one producer, a route change subscribes the new route from -the first frame the logical track lacks (mid-group), an open group is -continued in place, duplicates are dropped by frame index, and the old route -is cancelled once the new one feeds the track. Mirror that shape and naming -in JS rather than inventing a second model. Things to -settle along the way: +Current Rust (`model/resume.rs`) reads route copies directly through each +reader. It retains replaced copies for unread and in-flight groups, bounds +them at replacement, and recovers a group from the new copy at the first +missing frame and byte. It does not copy through the earlier single-writer +pump or immediately cancel every old copy. Decided in the 2026-10-10 audit: +mirror the current identity, reader, recovery and lifetime contract, including +the late-duplicate correction, rather than porting the obsolete pump. +Things to settle along the way: - Whether the request's `active` broadcast stays the same object across a swap, with its tracks re-sourced underneath. That is the Rust behavior and the @@ -65,3 +66,10 @@ subscriptions across a swap. Report it in the PR. ## Required - [One max_age meaning](/quest/m1/cache-max-age.md) - the rule that gives up a resumed group no route continues +- [Resume duplicates](/quest/m0/broadcast-epoch/resume-duplicates.md) - the corrected once-only delivery contract for reordered copies + +## Related + +- [Retired requests](/quest/m0/broadcast-epoch/retired-requests.md) - old handles cannot open unpinned requests after an identity change +- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - a terminal group disposition stops recovery even for the newest group +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - recovery fetches retain the reader's budget diff --git a/quest/m1/lite07-finalize.md b/quest/m1/lite07-finalize.md index 1c56860b04..7592edb0bf 100644 --- a/quest/m1/lite07-finalize.md +++ b/quest/m1/lite07-finalize.md @@ -76,6 +76,7 @@ is published; older versions are unchanged. - [In-band auth](/quest/m1/auth/README.md) - lite-07 carries the Auth Stream and UNAUTHORIZED (0x3B) - [Routes and announces](/quest/m1/cluster-routing/routes.md) - lite-07 carries the route layer: ROUTE per origin node and path-less ANNOUNCE, with the hop list gone - [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - lite-07 restores SUBSCRIBE_DROP in place of `Stream Count` +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - historical requests have a reader budget, carried by FETCH while present and by ranges after its removal - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - SUBSCRIBE carries ranges and an order and lite FETCH is gone, in Rust and JS - [Untimed lite-07](/quest/m1/lite-untimed.md) - an untimed track crosses the wire untimed - [Claim-served epochs](/quest/m1/claim-epochs.md) - TRACK_INFO carries the epoch of the instance that answered diff --git a/quest/m1/pipeline-requests/fetch.md b/quest/m1/pipeline-requests/fetch.md index adf8d457ff..b8a23f2b11 100644 --- a/quest/m1/pipeline-requests/fetch.md +++ b/quest/m1/pipeline-requests/fetch.md @@ -63,3 +63,5 @@ Public API: none. Wire: none (ordering only). - [Pipelined SUBSCRIBE](/quest/m1/pipeline-requests/subscribe.md) - the same change for subscriptions - [moq-transport ranges](/quest/m1/subscribe-ranges/ietf.md) - adjacent code in `ietf/subscriber.rs` and `model/origin.rs`; this lands first and ranges rebases +- [Retired requests](/quest/m0/broadcast-epoch/retired-requests.md) - staged requests retain their selected instance; retries cannot resolve a replacement by path +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - setup and recovery do not reset or bypass the reader's age budget diff --git a/quest/m1/resume-reorder.md b/quest/m1/resume-reorder.md index 7f54330663..db77e0e813 100644 --- a/quest/m1/resume-reorder.md +++ b/quest/m1/resume-reorder.md @@ -31,3 +31,8 @@ Public API: behavior only. Wire: none. ## Required - [Tail arrivals](/quest/m1/tail-arrivals.md) - the per-track arrival record this verdict reads + +## Related + +- [Resume duplicates](/quest/m0/broadcast-epoch/resume-duplicates.md) - retain once-only delivery without excluding genuinely unseen late groups +- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - explicit terminal dispositions end recovery without waiting for a newer group diff --git a/quest/m1/subscribe-drop.md b/quest/m1/subscribe-drop.md index 78263dc676..89228a6114 100644 --- a/quest/m1/subscribe-drop.md +++ b/quest/m1/subscribe-drop.md @@ -11,8 +11,11 @@ brings it back in place of `Stream Count`. ## Plan -Today SUBSCRIBE_DROP is on the wire for lite-03 through lite-06, and the Rust -and `@moq/net` subscribers account for it. Once a track ends, the Rust +Today SUBSCRIBE_DROP is defined on the wire for lite-03 through lite-06. +The Rust and `@moq/net` lite-05/06 response paths account for it in the +subscription tail. That accounting is not a terminal group disposition for +live readers or recovery; the JS lite-03/04 path does not run that response +loop. Once a track ends, the Rust publisher on lite-05 and lite-06 names every sequence from its SUBSCRIBE_START to the end that the subscription never got (skipped, stale, or missing its head), just before its FIN. Still missing: JS @@ -44,7 +47,11 @@ Decided: delivered, so a lost one leaves an uncovered hole that waits out the tail grace, as today. - A resumed group that is the new copy's latest ends with the DROP's error - when the copy drops it. + when the copy drops it. Audit clarification (2026-10-10): test this while + the subscription is still open, with no successor and no later writes. + Both Rust and JS must settle the group wait on DROP, not only mark the + tail accounted at SUBSCRIBE_END. A pending recovery fetch cannot hide + that disposition or restart an unbounded route-retry loop. - A dropped or aborted group is visible to readers, not silently skipped. #4533 found the Rust model releases an aborted group's sequence and skips it, so a truncated first object is indistinguishable from a group never sent. @@ -87,3 +94,9 @@ subscribers both settle on SUBSCRIBE_DROP through the relay, so a group the publisher skipped or never opened ends the track without waiting out the grace. Decided in the 2026-09-30 audit: the case moved here so the basic tail interop could land first. + +## Related + +- [Tail arrivals](/quest/m1/tail-arrivals.md) - per-track live dispositions survive cache eviction +- [Resume reorder](/quest/m1/resume-reorder.md) - late stream headers are not proof that a group was dropped +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - age expiry and DROP independently settle pending recovery diff --git a/quest/m1/subscribe-ranges/README.md b/quest/m1/subscribe-ranges/README.md index 4f0a972d8d..603e024a3e 100644 --- a/quest/m1/subscribe-ranges/README.md +++ b/quest/m1/subscribe-ranges/README.md @@ -19,7 +19,10 @@ relay bounds a sparse FETCH only if the model can request ranges: (an open end means live) in place of Group Start/End, plus `order: asc | desc` (desc, newest first, is today's rule and the default). Subscriber Max Age (the model's `max_delay`) stays as a cap alongside the - ranges, since a group's timestamp isn't always known. SUBSCRIBE_UPDATE + ranges, since a group's timestamp isn't always known. Decided 2026-10-10: + historical requests also obey that cap and publisher max-age. An omitted + FETCH budget means publisher max-age alone; translation to a range must + not accidentally give it the live SUBSCRIBE zero-delay default. SUBSCRIBE_UPDATE replaces the list. Lite FETCH is removed from lite-07; published versions keep it, answered through the same model. - **Holes.** SUBSCRIBE_DROP (restored in lite-07 by @@ -66,3 +69,5 @@ every sequence delivered or dropped. ## Related - [Track priority scope](/quest/m1/track-priority-scope.md) - group order within a track now follows the subscription +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - the age contract applies to past ranges and held readers too +- [IETF fill timeout](/quest/m1/ietf-fill-timeout.md) - bounds upstream waiting independently of content age diff --git a/quest/m1/subscribe-ranges/ietf.md b/quest/m1/subscribe-ranges/ietf.md index ec206c9ef5..392a1ef008 100644 --- a/quest/m1/subscribe-ranges/ietf.md +++ b/quest/m1/subscribe-ranges/ietf.md @@ -38,3 +38,5 @@ a live track's Largest Object, and test both cases. ## Related - [Pipelined first FETCH](/quest/m1/pipeline-requests/fetch.md) - sends TRACK_STATUS alongside the FETCH in adjacent code (`ietf/subscriber.rs`, `model/origin.rs`); decided 2026-10-08 it lands first, since it is unblocked and this is not, and this rebases onto it +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - preserve local age enforcement and the omitted-budget default when adapting IETF FETCH to ranges +- [IETF fill timeout](/quest/m1/ietf-fill-timeout.md) - handles native timeout gaps; do not mistake them for never-existing objects or media-age limits diff --git a/quest/m1/subscribe-ranges/model.md b/quest/m1/subscribe-ranges/model.md index 37e8ae8c85..cecc7ccbf8 100644 --- a/quest/m1/subscribe-ranges/model.md +++ b/quest/m1/subscribe-ranges/model.md @@ -22,9 +22,20 @@ Guidance: (`fetch_group`, `requested_group`), moq-c (`poll_requested_group`), every binding wrapper and its `doc/lib` page, and `moq fetch` through `moq_relay::fetch_group` (added in the 2026-10-05 audit). -- `max_delay` caps every range. +- Decided in the 2026-10-10 audit: `max_delay` caps every range, including + historical FETCH replacements, and publisher `max_age` always clamps it. + A FETCH with no reader delay uses publisher max-age alone, not the live + SUBSCRIBE zero-delay default. Preserve that distinction when translating + FETCH into a range request. Sharing a range never relaxes a caller's + budget or lets one expired caller cancel another still-valid reader. + [FETCH max-delay](/quest/m1/fetch-max-delay.md) owns the FETCH option and + enforcement; reuse its contract rather than restoring a historical exemption. - Benchmark range count and span as separate axes (AGENTS.md fan-out rule). ## Required - [lite-07 Live flag](/quest/m1/lite-live.md) - reshapes the same `Subscription` first + +## Related + +- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - historical request budgets and per-caller enforcement