Repository navigation
quest(m1): plan catalog estimate rate limiting and CI runner stalls #4535
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # [S] A rising catalog estimate republishes at most once a second | ||
|
|
||
| ## Goal | ||
|
|
||
| A publisher whose `jitter` or `delay` estimate keeps rising under load stops | ||
| republishing the catalog on every 1 ms step. Estimate-driven republishes are | ||
| rate-limited to one per second in `js/publish` and `moq-mux`: the first rise | ||
| publishes at once, and later rises inside the window coalesce into one publish | ||
| at the window's end carrying the latest value. Track additions, removals, and | ||
| config changes still publish immediately. | ||
|
|
||
| ## Plan | ||
|
|
||
| Found in [#4529](https://github.com/moq-dev/moq/pull/4529): under load, the | ||
| lifetime-maximum `Estimator` (`js/publish/src/jitter.ts`, mirroring | ||
| `moq_mux::catalog::Estimator`) rises 1 ms at a time and each rise republishes | ||
| the catalog, which makes every player send a subscribe update on every track. | ||
|
|
||
| Decided with the maintainer: | ||
|
|
||
| - Rate limit, not quantization, so the advertised value stays exact. | ||
| - Leading and trailing edge, 1 s window: a first estimate arrives without | ||
| delay and the catalog never trails the estimate by more than a window. | ||
| - Estimate-driven changes only; a structural catalog change never waits. | ||
| - Keep the draft's never-lower rule. #4529 also saw an estimate grow to about | ||
| 586 ms on an overloaded machine, so resuming refilled that whole delay. That | ||
| is the rule working as written ("a burst it emitted once it may emit | ||
| again"), not a bug for this quest. | ||
|
|
||
| A pending trailing publish folds into any structural publish that happens | ||
| first. Unit tests drive the window on a mocked clock in both languages. | ||
|
|
||
| The churn fans out (every rise reaches every subscriber on every track), so | ||
| add a benchmark swept over tracks x subscribers that drives a rising estimate | ||
| and counts catalog publishes and subscribe updates. It shows the saving now and | ||
| catches a regression later as a slope. `moq-mux` has no benches yet; add one | ||
| there or extend `rs/moq-net/benches/session.rs`. | ||
|
|
||
| Public API: none expected. Wire: none; catalog contents are unchanged, only | ||
| how often they are sent. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Audio jitter target](/quest/m0/audio-jitter-target/README.md) - the player-side target the advertised jitter floors | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| # [S] Interop freezes on CI runners are attributed | ||
|
|
||
| ## Goal | ||
|
|
||
| The 0.4 to 0.8 s freezes where both interop tracks stop at once on CI are | ||
| attributed to the runner, the relay, or the player, with evidence from the | ||
| nightly traces. A cause in our code is fixed at its source; a runner stall is | ||
| documented, and the interop checks tell it apart from a playback bug instead | ||
| of failing on it as one. | ||
|
|
||
| ## Plan | ||
|
|
||
| Found in [#4529](https://github.com/moq-dev/moq/pull/4529) while chasing the | ||
| cold-start tone check: some CI traces show both tracks freezing together, | ||
| which that fix shortens but cannot remove. #4529 added the player's `delay` to | ||
| the interop trace, so a week of nightlies after it lands shows the freezes | ||
| without the delay-flush noise that hid them. | ||
|
|
||
| Collect the failing and passing nightly interop traces for that week, line up | ||
| each freeze against relay and player timestamps, and look for what the | ||
|
Comment on lines
+19
to
+20
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The passing traces required by this plan will not exist: Useful? React with 👍 / 👎. |
||
| runner was doing (CPU steal, GC, disk). Decided with the maintainer: diagnose | ||
| first from real nightlies rather than assume a runner stall. | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Required | ||
|
|
||
| - A week of nightly interop runs on `main` after #4529 merges | ||
| - [Interop audio cold start](/quest/m1/interop-audio-cold-start.md) - #4529, which adds the `delay` column these traces need | ||
Uh oh!
There was an error while loading. Please reload this page.