Repository navigation
test(audio-quality): grade browser audio playout against a budget, nightly - #4426
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstreams the browser audio quality lane from fperex's fork (fperex/moq, debug-findings-solution) as test/audio-quality/, without the fork's player, estimator, or shaper changes. Plays a tone in headless Chromium through moq-shaper and grades underruns, skip-aheads, silence, and the resolved delay against budgets.json. Runs nightly under --enforce. Co-Authored-By: fperex <270788582+fperex@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on PRs Ceilings come from two full local runs plus one short one: the fixed-250 control measured zero on every counter, and every auto row underruns on today's shallow target. The page builds into the run directory so concurrent runs cannot empty each other's dist/, the JS check scope covers the new workspace, and the nightly job sets MOQ_TEST_RUNS on the step, where the runner context is allowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Matches wasm.yml, which moved its Playwright lane to the ARM runners that cache.yml warms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
29b0a64
into
quest/m0/audio-quality-harness/README
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8dc1be3bf0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| detail: `the AudioContext clock ran at ${round(renderRate, 3)}x the viewer's: throttled or stalled`, | ||
| }); | ||
| } | ||
| const toViewer = (render: Ms) => render + (renderOffset ?? 0); |
There was a problem hiding this comment.
When the AudioContext clock runs at any accepted non-unit rate, renderRate is calculated but ignored here, so render timestamps are translated using only a constant offset. At the allowed 1% drift over a 60-second row, a gap can be placed hundreds of milliseconds from its viewer-clock position and cross the warmup or end boundary, changing the graded underrun totals. Use the fitted affine mapping for toViewer, not just the median offset.
Useful? React with 👍 / 👎.
| s.timestamp === undefined || s.stalled ? undefined : s.at - s.timestamp - (mediaRate ?? 0) * (s.at - base), | ||
| ); | ||
| const skips = skipAheads(lags); | ||
| const stalledShare = window.length > 0 ? window.filter((s) => s.stalled).length / window.length : null; |
There was a problem hiding this comment.
When a stall begins and ends between two 250 ms probe samples, the exact changes are present in Sample.stalls, but this calculation sees only the periodic s.stalled snapshots and reports no stalled time. Consequently stalled_share can remain zero despite repeated brief stalls and pass its budget; compute the duration of the clipped stallSpans instead of counting sampled booleans.
Useful? React with 👍 / 👎.
| /** Gaps closed since the last call. A gap still open is reported once a full quantum ends it. */ | ||
| take(): Gap[] { | ||
| return this.#closed.splice(0, this.#closed.length); |
There was a problem hiding this comment.
When playback enters an underflow near the end of a row and never produces another full quantum, #open is never moved into #closed, so every call to take() omits that final episode before the page is closed. This can exclude hundreds of milliseconds from all underrun and gap metrics, while still staying below the 1% silence ceiling, allowing a real trailing glitch to pass; expose or finalize the open gap at shutdown.
Useful? React with 👍 / 👎.
| if (environment.transport !== "webtransport") { | ||
| refuse("transport", `negotiated ${environment.transport}, which never crosses the UDP shaper`); | ||
| } | ||
| if (environment.crossOriginIsolated !== (ring === "isolated")) { | ||
| refuse("ring", `asked for ${ring}, but crossOriginIsolated is ${environment.crossOriginIsolated}`); |
There was a problem hiding this comment.
When the publisher or catalog regresses to a different codec or sample rate, the page already reports those values in environment, but the driver validates only transport and ring. The run is then graded under the requested row's codec/rate budget even though another matrix cell actually ran, which can produce a false pass or failure; compare the reported codec and rate with the row before accepting the measurement.
AGENTS.md reference: AGENTS.md:L17-L17
Useful? React with 👍 / 👎.
1 participant
Quest:
/quest/m0/audio-quality-harness/browser.md(deleted here). Part of the questline #4395.Summary
Adds
just test audio-quality: a tone played in headless Chromium throughmoq-shaper, graded against a checked-in budget, nightly.This upstreams the lane @fperex built on their fork for #3477 (
fperex/moq, branchesdebug/rt-audioanddebug-findings-solution): the black-box probe, the skip-ahead rule, the Playwright matrix overmoq-shaper, and the--enforcebudget grading. Only the harness lands. The fork's player, estimator, and shaper changes do not. fperex is credited as co-author on the main commit and intest/audio-quality/README.md.near-zero,mild,wide,fixed-250) x ring (isolatedSharedArrayBuffer,plainpostMessage). That is 16 rows at 60 s each.clients/js/src/schema.tsdefines every counter, its unit, its clock, and its aggregation. The budget key names the aggregation (underruns_per_min), and the stages are exclusive spans with a namedunaccountedremainder. The render clock is reduced to the viewer clock. Publisher and relay stages are reported unmeasured rather than guessed.--enforce.budgets.jsonis keyed by the whole row (runtime, codec, rate, profile, ring). Thenotefield says how the ceilings were measured.Audio qualityjob runs under--enforceand uploads a failing run directory for 7 days. The header comment explains why this is not a PR gate.test/audio-quality/clients/js, so the analyzer's 18 unit tests and its type check run on PRs.dist/.What it measured (local, seed 7)
The fixed control is clean on every counter, which is the check that the tap does not invent underruns. Every auto row underruns, including on an unimpaired path. The resolved target is 43 ms for Opus, which is the #3477 finding reproduced by a harness. So the auto-row ceilings (twice the worse run) record today's floor rather than a good one. The jitter target line is expected to tighten them.
Scope split
The previous attempt split two parts of the original quest into new quests on this line instead of landing them here:
traces.md: record real arrival traces and grade a replay. No public signal carries a per-frame arrival, so recording needs either a@moq/watchdebug hook or a native recorder, and the fork's replay module is a player change.shaper-profiles.md: theburstyand mid-runstepprofiles. These needmoq-shaperbatch and step features that exist on the fork and not onmain.audio-jitter-target/watch.mdnow depends ontraces.mdinstead of this quest.Public API / wire impact
None. The new workspace package
@moq/audio-qualityis private. No wire changes.Verification
just checkpasses.--enforcepasses against the recorded budgets. A tightened budget copy fails as expected.Open decisions
Suggested follow-ups
audio-jitter-target/watch.md, not a new quest.(Written by Claude Opus 5.5)
🤖 Generated with Claude Code