From de09b9c4caed89993d136589157197f67ec87632 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 12 Oct 2026 21:18:42 -0700 Subject: [PATCH 1/2] quest(m0): plan the nightly release-link and fuzz LTO breaks Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 9 ++++++++ quest/m0/fuzz-lto.md | 39 +++++++++++++++++++++++++++++++++ quest/m0/release-link.md | 47 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 95 insertions(+) create mode 100644 quest/m0/fuzz-lto.md create mode 100644 quest/m0/release-link.md diff --git a/quest/m0/README.md b/quest/m0/README.md index 0747a5de22..b144617a5e 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -55,12 +55,21 @@ Audio playout: the jitter target is default in both languages (#4162); the 2026-10-08 audit, since only a manual browser proof and a native trace replay remain and no release waits on them. +Nightly on main (triaged 2026-10-13): two build breaks from 2026-10-11 get +quests here. The nixpkgs bump broke the Linux zigbuild release builds, and +fat LTO in the release profile broke the fuzz build. The other red jobs +have fixes on `main` or in open PRs: macOS tests (#5335), +drill-sensitivity (#5338), loom (#5389) and Windows tests (#5339). Interop +is planned separately. + ## Required - [Self-hosted CI](/quest/m0/self-hosted-ci.md) - same-repo Check and Test run on a self-hosted NixOS runner with a main-written local cache, behind a `CI_RUNNER` kill switch - [CI host](/quest/m0/ci-host.md) - the maintainer brings up the spare desktop as the `moq-ci` and `moq-gpu` runner host - [Draft-22 media on 0.17](/quest/m0/release-22/README.md) - a 0.17.x with the LOCATION_FILTER and FIRST_OBJECT fixes and moq-noq 1.3.5, before Seattle, then draft-20+ FETCH - [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - Interop's TS compliance captures the whole round-tripped stream again, fixing a regression that turns Interop red on main +- [Release builds link](/quest/m0/release-link.md) - the Linux zigbuild release binaries and moq-gst packages link again after the 2026-10-11 nixpkgs bump, so the next release cut can publish +- [Fuzz builds under LTO](/quest/m0/fuzz-lto.md) - the nightly fuzz targets build again despite the release profile's fat LTO - [web-transport releases the qmux fixes](/quest/m0/qmux-credit-upstream.md) - waiting on moq-dev/web-transport#412 and #413 to merge and ship, which qmux credit bumps to - [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription diff --git a/quest/m0/fuzz-lto.md b/quest/m0/fuzz-lto.md new file mode 100644 index 0000000000..f97aec2a9e --- /dev/null +++ b/quest/m0/fuzz-lto.md @@ -0,0 +1,39 @@ +# [XS] Fuzz targets build under the release profile's LTO + +## Goal + +Every nightly `Fuzz ()` job builds and runs its target again. A +release-profile change can't silently break the fuzz build, because the +fuzz build opts out of what sanitizer coverage can't link. + +## Plan + +Seen in the 2026-10-12 nightly (run 38276133321): all six fuzz targets fail +to link with `undefined reference to '__sancov_gen_.N'` from +`asan.module_dtor`. The 2026-10-11 nightly passed. + +Cause: #5274 (be8d0024d) set `lto = "fat"`, `codegen-units = 1` and +`strip = "symbols"` on the workspace `[profile.release]`. `cargo fuzz` +builds `--release`, and `rs/moq-net/fuzz` is a workspace member, so it +inherits fat LTO, which drops the sanitizer-coverage globals the ASan module +destructor references. Reproduced locally on the dev shell's rustc with +`RUSTC_BOOTSTRAP=1` and cargo-fuzz's flags (`cargo build --manifest-path +rs/moq-net/fuzz/Cargo.toml --release --bin lite`): fails as in CI, and links +with `CARGO_PROFILE_RELEASE_LTO=false`. + +Likely a one-line fix: `sh/rs/fuzz.sh` already exports `RUSTFLAGS`, so it can +override the profile the same way (for example +`CARGO_PROFILE_RELEASE_LTO=false`). Consider `strip = "none"` too, so ASan +and libFuzzer crash reports keep symbol names. Prefer whatever keeps the +fuzz build's settings next to the fuzz build rather than another profile in +the root `Cargo.toml`, unless cargo-fuzz can select one cleanly. + +Verify with `just rs fuzz -- -max_total_time=10` for each target. +The nightly is the only CI that builds fuzz targets, so the PR can't prove +it on CI; say so in the PR. + +Public API: none. Wire: none. + +## Related + +- [Release builds link again](/quest/m0/release-link.md) - the other build break in the same nightly, from a different cause diff --git a/quest/m0/release-link.md b/quest/m0/release-link.md new file mode 100644 index 0000000000..7a6804b860 --- /dev/null +++ b/quest/m0/release-link.md @@ -0,0 +1,47 @@ +# [S] Release builds link again on the current Nix toolchain + +## Goal + +The Linux release binaries (`moq-cli`, `moq-relay`) and the `moq-gst` +.deb/.rpm packages build with `cargo zigbuild` on `main`'s pinned Nix inputs, +for x86_64 and aarch64, so the next release cut from `main` can publish. + +## Plan + +Seen in the 2026-10-12 nightly (run 38276133321): `moq-cli / release / +Build`, `moq-relay / release / Build` and `moq-gst / Package .deb/.rpm` +fail on both Linux targets. Linking `iroh-relay`'s cdylib fails with +`ld.lld: error: undefined symbol:` (an empty name), referenced by +`bcm.o:(aws_lc_0_45_0_OPENSSL_cpuid_setup)` in `aws-lc-sys`. The 2026-10-11 +nightly passed. macOS and Windows release builds still pass. + +Onset: the nixpkgs bump in #5316 (d0736ec04), not #5274's LTO change, which +landed the same night. Reproduced with +`cargo zigbuild --locked --release --target x86_64-unknown-linux-gnu.2.34 -p iroh-relay --lib`: + +- Current `flake.lock`: fails as in CI, also with + `CARGO_PROFILE_RELEASE_LTO=false` and with `CARGO_PROFILE_RELEASE_STRIP=none`. +- `flake.lock` from 717f3fecc (before #5316), same `Cargo.toml` and + `Cargo.lock`: links. +- Both shells have zig 0.16.0 and rustc 1.98.1. What changed is GCC + 15.3 -> 16.2, glibc 2.42 -> 2.44, and smaller bumps. +- `bcm.o` in the new build has 292 undefined `SECTION` symbols in + `readelf -s`; the old one has none. Both report clang 21.1.8 in `.comment`, + so something other than the compiler zig picks differs, possibly a step in + `aws-lc-sys`'s build that reaches the host toolchain. + +Find which tool produces the broken object and fix it at the source: an +`aws-lc-sys` or cargo-zigbuild environment fix, or an upstream report if the +bug is theirs. Reverting nixpkgs is a stopgap, not the fix; if it's needed +to unblock a release, make it its own quest. + +`release` still pins the pre-bump nixpkgs, so shipping releases are fine +today; the next `main` -> `release` merge carries the break. PR CI never runs +zigbuild, so nothing caught it before the nightly. Consider a cheap PR-time +check when `flake.lock` changes, if the fix shows one is worth it. + +Public API: none. Wire: none. + +## Related + +- [Fuzz targets build under LTO](/quest/m0/fuzz-lto.md) - the other build break in the same nightly, from a different cause From f3eb46016724d7574006f37a742018b179c5398e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 12 Oct 2026 21:33:35 -0700 Subject: [PATCH 2/2] quest(m0): move the audio quality budget re-record out of Watch Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 5 ++- quest/m0/audio-quality-budgets.md | 44 +++++++++++++++++++++++++++ quest/m1/audio-jitter-target/watch.md | 7 +++-- 3 files changed, 52 insertions(+), 4 deletions(-) create mode 100644 quest/m0/audio-quality-budgets.md diff --git a/quest/m0/README.md b/quest/m0/README.md index b144617a5e..b341d8bd35 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -57,7 +57,9 @@ remain and no release waits on them. Nightly on main (triaged 2026-10-13): two build breaks from 2026-10-11 get quests here. The nixpkgs bump broke the Linux zigbuild release builds, and -fat LTO in the release profile broke the fuzz build. The other red jobs +fat LTO in the release profile broke the fuzz build. Audio quality has been +red since it landed, against budgets recorded locally; re-measuring them on +the runner moved here from Watch (maintainer, 2026-10-13). The other red jobs have fixes on `main` or in open PRs: macOS tests (#5335), drill-sensitivity (#5338), loom (#5389) and Windows tests (#5339). Interop is planned separately. @@ -70,6 +72,7 @@ is planned separately. - [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - Interop's TS compliance captures the whole round-tripped stream again, fixing a regression that turns Interop red on main - [Release builds link](/quest/m0/release-link.md) - the Linux zigbuild release binaries and moq-gst packages link again after the 2026-10-11 nixpkgs bump, so the next release cut can publish - [Fuzz builds under LTO](/quest/m0/fuzz-lto.md) - the nightly fuzz targets build again despite the release profile's fat LTO +- [Audio quality budgets](/quest/m0/audio-quality-budgets.md) - the nightly Audio quality job grades against ceilings measured on its own runner, so a breach means a regression again - [web-transport releases the qmux fixes](/quest/m0/qmux-credit-upstream.md) - waiting on moq-dev/web-transport#412 and #413 to merge and ship, which qmux credit bumps to - [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription diff --git a/quest/m0/audio-quality-budgets.md b/quest/m0/audio-quality-budgets.md new file mode 100644 index 0000000000..97d8de8544 --- /dev/null +++ b/quest/m0/audio-quality-budgets.md @@ -0,0 +1,44 @@ +# [S] Audio quality budgets come from the nightly runner + +## Goal + +The nightly `Audio quality` job passes on `main`'s current player, and a breach +means a regression again. The browser rows' ceilings in +`test/audio-quality/budgets.json` are measured on the nightly runner, not on a +developer machine, and the file's `note` says how. + +Non-goals: improving playout to meet tighter ceilings. That stays with +[Watch](/quest/m1/audio-jitter-target/watch.md). The replay rows, which PRs +already enforce, keep their own budgets. + +## Plan + +Seen 2026-10-02 to 2026-10-12: the job has failed every night since #4395 +added it. The 2026-10-12 run (38276133321) breached 27 keys across the +Chromium rows, for example `chromium-aac-44100-bursty-isolated` at a +`stalled_share` of 0.324 against 0.01, and `target_ms_p95` on most Opus +rows. The browser ceilings were recorded locally (#4426), but nightly +measures on an `ubuntu-24.04-arm` runner. A red job nobody can act on hides +real regressions, so this moved out of Watch into m0 (maintainer, +2026-10-13). + +- Measure several nightly runs, or a matrix dispatched on the runner from + the branch, before setting ceilings. A single sample's noise would make + the job flaky. Failing runs keep their summaries as the + `audio-quality-run` artifact for 7 days. +- Set each ceiling from the observed spread with modest headroom, and record + the method in `budgets.json`'s `note`. Each loosened ceiling needs its + reason in review (see `test/audio-quality/README.md`). +- Some rows may be genuine defects rather than runner noise. The AAC bursty + rows stalling a third of the time look like one. Budget what the player + does today, and file each suspected defect as a follow-up quest, so the job + goes green without hiding it. +- Rows that are too noisy to bound on a shared runner can be dropped from + `--enforce` instead. That decision needs the maintainer. + +Public API: none. Wire: none. + +## Related + +- [Watch](/quest/m1/audio-jitter-target/watch.md) - tightens these ceilings once playout improves +- [Audio quality native](/quest/m1/audio-quality-native.md) - reuses the same budget file for a native lane diff --git a/quest/m1/audio-jitter-target/watch.md b/quest/m1/audio-jitter-target/watch.md index 1cce2e7f9e..fe80fac882 100644 --- a/quest/m1/audio-jitter-target/watch.md +++ b/quest/m1/audio-jitter-target/watch.md @@ -32,9 +32,9 @@ about 22 s with skip-aheads on the way down. - Decide whether those re-stall refills are what the goal wants, using the replay rows as the measurement. Tighten the replay budgets with any fix. - Re-record the browser rows' `test/audio-quality/budgets.json` from the - nightly runner's first runs on `main` before tightening them, since they were - recorded locally (#4426) and nightly only runs `main`'s code. + Tighten the browser rows only from the ceilings + [Audio quality budgets](/quest/m0/audio-quality-budgets.md) measures on the + nightly runner. - The replay sizes every frame as the trace's median spacing, which a trace missing most of its frames defeats; record the codec's frame duration in the trace and replay that instead. @@ -59,3 +59,4 @@ release. ## Related - [A/V clock](/quest/m1/av-clock.md) - reshapes `SyncInput` around the per-track target this line produces +- [Audio quality budgets](/quest/m0/audio-quality-budgets.md) - measures the browser rows' ceilings on the nightly runner, which this quest tightens