From 22c17531083fa8ad046b851cab969e25a8cac35a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 29 Sep 2026 12:42:00 -0700 Subject: [PATCH 1/2] quest(m1): plan catalog estimate rate limiting and CI runner stalls Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 ++ quest/m1/catalog-estimate-rate.md | 38 +++++++++++++++++++++++++++++++ quest/m1/ci-runner-stalls.md | 29 +++++++++++++++++++++++ 3 files changed, 69 insertions(+) create mode 100644 quest/m1/catalog-estimate-rate.md create mode 100644 quest/m1/ci-runner-stalls.md diff --git a/quest/m1/README.md b/quest/m1/README.md index e657b29d32..a4a305a204 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -45,6 +45,8 @@ transport, benchmark tooling); worktrees isolate commits, not semantics. - [Watch and publish under CSP](/quest/m1/csp-assets.md) - blob workers stay the default; strict-CSP apps host the files and set a base URL - [More tests under load](/quest/m1/test-flakes-2.md) - the second round of load-only failures, fixed at the cause - [Interop audio cold start](/quest/m1/interop-audio-cold-start.md) - the interop audio tone check stops failing on cold start, fixed at its cause +- [CI runner stalls](/quest/m1/ci-runner-stalls.md) - the 0.4 to 0.8 s freezes of both interop tracks on CI are attributed from a week of nightlies and fixed or told apart from playback bugs +- [Catalog estimate rate](/quest/m1/catalog-estimate-rate.md) - a rising `jitter`/`delay` estimate republishes the catalog at most once a second, in js/publish and moq-mux - [Auth outage clock](/quest/m1/auth-outage-clock.md) - the relay and moq-auth outage tests run on a paused clock again and assert both bounds of `expires` - [Legacy end overshoot](/quest/m1/legacy-end-overshoot.md) - browser playback survives a group that starts inside the previous group's estimated end - [UnknownSession log flood](/quest/m1/unknown-session-logs.md) - streams reset before their WebTransport header stop being reported as UnknownSession at WARN diff --git a/quest/m1/catalog-estimate-rate.md b/quest/m1/catalog-estimate-rate.md new file mode 100644 index 0000000000..2aae44d0c6 --- /dev/null +++ b/quest/m1/catalog-estimate-rate.md @@ -0,0 +1,38 @@ +# [S] A rising catalog estimate republishes at most once a second + +## Goal + +A publisher whose `jitter` or `delay` estimate keeps rising under load stops +republishing the catalog on every 1 ms step. Estimate-driven republishes are +rate-limited to one per second in `js/publish` and `moq-mux`: the first rise +publishes at once, and later rises inside the window coalesce into one publish +at the window's end carrying the latest value. Track additions, removals, and +config changes still publish immediately. + +## Plan + +Found in [#4529](https://github.com/moq-dev/moq/pull/4529): under load, the +lifetime-maximum `Estimator` (`js/publish/src/jitter.ts`, mirroring +`moq_mux::catalog::Estimator`) rises 1 ms at a time and each rise republishes +the catalog, which makes every player send a subscribe update on every track. + +Decided with the maintainer: + +- Rate limit, not quantization, so the advertised value stays exact. +- Leading and trailing edge, 1 s window: a first estimate arrives without + delay and the catalog never trails the estimate by more than a window. +- Estimate-driven changes only; a structural catalog change never waits. +- Keep the draft's never-lower rule. #4529 also saw an estimate grow to about + 586 ms on an overloaded machine, so resuming refilled that whole delay. That + is the rule working as written ("a burst it emitted once it may emit + again"), not a bug for this quest. + +A pending trailing publish folds into any structural publish that happens +first. Unit tests drive the window on a mocked clock in both languages. + +Public API: none expected. Wire: none; catalog contents are unchanged, only +how often they are sent. + +## Related + +- [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the player-side target the advertised jitter floors diff --git a/quest/m1/ci-runner-stalls.md b/quest/m1/ci-runner-stalls.md new file mode 100644 index 0000000000..3d6f4df3bd --- /dev/null +++ b/quest/m1/ci-runner-stalls.md @@ -0,0 +1,29 @@ +# [S] Interop freezes on CI runners are attributed + +## Goal + +The 0.4 to 0.8 s freezes where both interop tracks stop at once on CI are +attributed to the runner, the relay, or the player, with evidence from the +nightly traces. A cause in our code is fixed at its source; a runner stall is +documented, and the interop checks tell it apart from a playback bug instead +of failing on it as one. + +## Plan + +Found in [#4529](https://github.com/moq-dev/moq/pull/4529) while chasing the +cold-start tone check: some CI traces show both tracks freezing together, +which that fix shortens but cannot remove. #4529 added the player's `delay` to +the interop trace, so a week of nightlies after it lands shows the freezes +without the delay-flush noise that hid them. + +Collect the failing and passing nightly interop traces for that week, line up +each freeze against relay and player timestamps, and look for what the +runner was doing (CPU steal, GC, disk). Decided with the maintainer: diagnose +first from real nightlies rather than assume a runner stall. + +Public API: none. Wire: none. + +## Required + +- A week of nightly interop runs on `main` after #4529 merges +- [Interop audio cold start](/quest/m1/interop-audio-cold-start.md) - #4529, which adds the `delay` column these traces need From 20af14b116cea0c982e80d0849beedc9513fc768 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 29 Sep 2026 13:43:48 -0700 Subject: [PATCH 2/2] quest: require a tracks x subscribers bench for the catalog rate limit Co-Authored-By: Claude Opus 5.5 --- quest/m1/catalog-estimate-rate.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/quest/m1/catalog-estimate-rate.md b/quest/m1/catalog-estimate-rate.md index 2aae44d0c6..3aa26cf7fc 100644 --- a/quest/m1/catalog-estimate-rate.md +++ b/quest/m1/catalog-estimate-rate.md @@ -30,6 +30,12 @@ Decided with the maintainer: A pending trailing publish folds into any structural publish that happens first. Unit tests drive the window on a mocked clock in both languages. +The churn fans out (every rise reaches every subscriber on every track), so +add a benchmark swept over tracks x subscribers that drives a rising estimate +and counts catalog publishes and subscribe updates. It shows the saving now and +catches a regression later as a slope. `moq-mux` has no benches yet; add one +there or extend `rs/moq-net/benches/session.rs`. + Public API: none expected. Wire: none; catalog contents are unchanged, only how often they are sent.