Repository navigation
quest(m0): plan the nightly release-link, fuzz LTO, and audio budget fixes #5394
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # [S] Audio quality budgets come from the nightly runner | ||
|
|
||
| ## Goal | ||
|
|
||
| The nightly `Audio quality` job passes on `main`'s current player, and a breach | ||
| means a regression again. The browser rows' ceilings in | ||
| `test/audio-quality/budgets.json` are measured on the nightly runner, not on a | ||
| developer machine, and the file's `note` says how. | ||
|
|
||
| Non-goals: improving playout to meet tighter ceilings. That stays with | ||
| [Watch](/quest/m1/audio-jitter-target/watch.md). The replay rows, which PRs | ||
| already enforce, keep their own budgets. | ||
|
|
||
| ## Plan | ||
|
|
||
| Seen 2026-10-02 to 2026-10-12: the job has failed every night since #4395 | ||
| added it. The 2026-10-12 run (38276133321) breached 27 keys across the | ||
| Chromium rows, for example `chromium-aac-44100-bursty-isolated` at a | ||
| `stalled_share` of 0.324 against 0.01, and `target_ms_p95` on most Opus | ||
| rows. The browser ceilings were recorded locally (#4426), but nightly | ||
| measures on an `ubuntu-24.04-arm` runner. A red job nobody can act on hides | ||
| real regressions, so this moved out of Watch into m0 (maintainer, | ||
| 2026-10-13). | ||
|
|
||
| - Measure several nightly runs, or a matrix dispatched on the runner from | ||
| the branch, before setting ceilings. A single sample's noise would make | ||
| the job flaky. Failing runs keep their summaries as the | ||
| `audio-quality-run` artifact for 7 days. | ||
| - Set each ceiling from the observed spread with modest headroom, and record | ||
| the method in `budgets.json`'s `note`. Each loosened ceiling needs its | ||
| reason in review (see `test/audio-quality/README.md`). | ||
| - Some rows may be genuine defects rather than runner noise. The AAC bursty | ||
| rows stalling a third of the time look like one. Budget what the player | ||
| does today, and file each suspected defect as a follow-up quest, so the job | ||
| goes green without hiding it. | ||
| - Rows that are too noisy to bound on a shared runner can be dropped from | ||
| `--enforce` instead. That decision needs the maintainer. | ||
|
Comment on lines
+36
to
+37
Contributor
There was a problem hiding this comment. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n -C 5 -- '--enforce|exclude|skip|filter|void row|no budget' \
test/audio-quality .github/workflows/nightly.ymlRepository: moq-dev/moq Length of output: 41831 🏁 Script executed: set -euo pipefail
printf '%s\n' '--- quest ---'
nl -ba quest/m0/audio-quality-budgets.md | sed -n '1,80p'
printf '%s\n' '--- README candidates ---'
rg --files test/audio-quality | sort | sed -n '1,120p'
printf '%s\n' '--- README enforcement/selection references ---'
rg -n -C 8 -F -- '--enforce' test/audio-quality
rg -n -C 5 -e 'exclude|skip|filter|enabled|disabled|void|no budget|budget' test/audio-quality/README* test/audio-quality/clients/js/src test/audio-quality/run.sh || test "$?" -eq 1
printf '%s\n' '--- grade entrypoints and relevant implementation ---'
rg -n -F -- 'grade.ts' test/audio-quality
ast-grep outline test/audio-quality/clients/js/src/grade.ts --view expanded
sed -n '1,280p' test/audio-quality/clients/js/src/grade.ts
printf '%s\n' '--- budget schema and row identity ---'
rg -n -C 5 -e 'interface.*Row|type.*Row|rows|runtime|codec|profile|ring|budget' test/audio-quality/clients/js/src/schema.ts test/audio-quality/budgets.json | sed -n '1,260p'Repository: moq-dev/moq Length of output: 41763 🏁 Script executed: set -euo pipefail
nl -ba test/audio-quality/clients/js/grade.ts | sed -n '1,115p'
printf '%s\n' '--- schema budget declarations ---'
nl -ba test/audio-quality/clients/js/src/schema.ts | sed -n '390,475p'Repository: moq-dev/moq Length of output: 7601 Use a supported row-selection mechanism. The harness has no way to exclude individual rows from nightly enforcement. Omitting a row’s budget also fails enforcement because 🤖 Prompt for AI Agents
Collaborator
Author
There was a problem hiding this comment. Agreed that (Written by Claude Opus 5.5)
Contributor
There was a problem hiding this comment.
Would you like a separate GitHub issue for the follow-up, or keep it in the quest? You are interacting with an AI system. |
||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Watch](/quest/m1/audio-jitter-target/watch.md) - tightens these ceilings once playout improves | ||
| - [Audio quality native](/quest/m1/audio-quality-native.md) - reuses the same budget file for a native lane | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| # [XS] Fuzz targets build under the release profile's LTO | ||
|
|
||
| ## Goal | ||
|
|
||
| Every nightly `Fuzz (<target>)` job builds and runs its target again. A | ||
| release-profile change can't silently break the fuzz build, because the | ||
| fuzz build opts out of what sanitizer coverage can't link. | ||
|
|
||
| ## Plan | ||
|
|
||
| Seen in the 2026-10-12 nightly (run 38276133321): all six fuzz targets fail | ||
| to link with `undefined reference to '__sancov_gen_.N'` from | ||
| `asan.module_dtor`. The 2026-10-11 nightly passed. | ||
|
|
||
| Cause: #5274 (be8d0024d) set `lto = "fat"`, `codegen-units = 1` and | ||
| `strip = "symbols"` on the workspace `[profile.release]`. `cargo fuzz` | ||
| builds `--release`, and `rs/moq-net/fuzz` is a workspace member, so it | ||
| inherits fat LTO, which drops the sanitizer-coverage globals the ASan module | ||
| destructor references. Reproduced locally on the dev shell's rustc with | ||
| `RUSTC_BOOTSTRAP=1` and cargo-fuzz's flags (`cargo build --manifest-path | ||
| rs/moq-net/fuzz/Cargo.toml --release --bin lite`): fails as in CI, and links | ||
| with `CARGO_PROFILE_RELEASE_LTO=false`. | ||
|
|
||
| Likely a one-line fix: `sh/rs/fuzz.sh` already exports `RUSTFLAGS`, so it can | ||
| override the profile the same way (for example | ||
| `CARGO_PROFILE_RELEASE_LTO=false`). Consider `strip = "none"` too, so ASan | ||
| and libFuzzer crash reports keep symbol names. Prefer whatever keeps the | ||
| fuzz build's settings next to the fuzz build rather than another profile in | ||
| the root `Cargo.toml`, unless cargo-fuzz can select one cleanly. | ||
|
|
||
| Verify with `just rs fuzz <target> -- -max_total_time=10` for each target. | ||
| The nightly is the only CI that builds fuzz targets, so the PR can't prove | ||
| it on CI; say so in the PR. | ||
|
Comment on lines
+32
to
+33
There was a problem hiding this comment.
When implementing the proposed profile override, this should not stop at the local ten-second runs: Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment. Disagree, leaving the quest as is. The maintainer's policy is to never dispatch the Nightly workflow from a branch, so a branch fuzz run isn't available to the implementer. The quest already asks for every target to be run locally with (Written by Claude Opus 5.5) |
||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Release builds link again](/quest/m0/release-link.md) - the other build break in the same nightly, from a different cause | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # [S] Release builds link again on the current Nix toolchain | ||
|
|
||
| ## Goal | ||
|
|
||
| The Linux release binaries (`moq-cli`, `moq-relay`) and the `moq-gst` | ||
| .deb/.rpm packages build with `cargo zigbuild` on `main`'s pinned Nix inputs, | ||
| for x86_64 and aarch64, so the next release cut from `main` can publish. | ||
|
|
||
| ## Plan | ||
|
|
||
| Seen in the 2026-10-12 nightly (run 38276133321): `moq-cli / release / | ||
| Build`, `moq-relay / release / Build` and `moq-gst / Package .deb/.rpm` | ||
| fail on both Linux targets. Linking `iroh-relay`'s cdylib fails with | ||
| `ld.lld: error: undefined symbol:` (an empty name), referenced by | ||
| `bcm.o:(aws_lc_0_45_0_OPENSSL_cpuid_setup)` in `aws-lc-sys`. The 2026-10-11 | ||
| nightly passed. macOS and Windows release builds still pass. | ||
|
|
||
| Onset: the nixpkgs bump in #5316 (d0736ec04), not #5274's LTO change, which | ||
| landed the same night. Reproduced with | ||
| `cargo zigbuild --locked --release --target x86_64-unknown-linux-gnu.2.34 -p iroh-relay --lib`: | ||
|
|
||
| - Current `flake.lock`: fails as in CI, also with | ||
| `CARGO_PROFILE_RELEASE_LTO=false` and with `CARGO_PROFILE_RELEASE_STRIP=none`. | ||
| - `flake.lock` from 717f3fecc (before #5316), same `Cargo.toml` and | ||
| `Cargo.lock`: links. | ||
| - Both shells have zig 0.16.0 and rustc 1.98.1. What changed is GCC | ||
| 15.3 -> 16.2, glibc 2.42 -> 2.44, and smaller bumps. | ||
| - `bcm.o` in the new build has 292 undefined `SECTION` symbols in | ||
| `readelf -s`; the old one has none. Both report clang 21.1.8 in `.comment`, | ||
| so something other than the compiler zig picks differs, possibly a step in | ||
| `aws-lc-sys`'s build that reaches the host toolchain. | ||
|
|
||
| Find which tool produces the broken object and fix it at the source: an | ||
| `aws-lc-sys` or cargo-zigbuild environment fix, or an upstream report if the | ||
| bug is theirs. Reverting nixpkgs is a stopgap, not the fix; if it's needed | ||
| to unblock a release, make it its own quest. | ||
|
|
||
| `release` still pins the pre-bump nixpkgs, so shipping releases are fine | ||
| today; the next `main` -> `release` merge carries the break. PR CI never runs | ||
| zigbuild, so nothing caught it before the nightly. Consider a cheap PR-time | ||
| check when `flake.lock` changes, if the fix shows one is worth it. | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Fuzz targets build under LTO](/quest/m0/fuzz-lto.md) - the other build break in the same nightly, from a different cause |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.