diff --git a/quest/m0/README.md b/quest/m0/README.md index a1453262e5..1f8649e0e4 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -62,6 +62,14 @@ released wire-compat step aborts on planned breaks the last release made stale. Both stay in m0 (maintainer, 2026-10-13) and rank with TS duration fidelity: a red Interop hides real regressions. +Nightly on main (triaged 2026-10-13): two build breaks from 2026-10-11 get +quests here. The nixpkgs bump broke the Linux zigbuild release builds, and +fat LTO in the release profile broke the fuzz build. Audio quality has been +red since it landed, against budgets recorded locally; re-measuring them on +the runner moved here from Watch (maintainer, 2026-10-13). The other red jobs +have fixes on `main`: macOS tests (#5335), drill-sensitivity (#5338), loom +(#5389) and Windows tests (#5339). + ## Required - [Self-hosted CI](/quest/m0/self-hosted-ci.md) - same-repo Check and Test run on a self-hosted NixOS runner with a main-written local cache, behind a `CI_RUNNER` kill switch @@ -70,6 +78,9 @@ fidelity: a red Interop hides real regressions. - [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - Interop's TS compliance captures the whole round-tripped stream again, fixing a regression that turns Interop red on main - [Upgrade restarts epochless broadcasts](/quest/m0/upgrade-unroutable.md) - a WebSocket-to-QUIC upgrade retires epochless handles only once their `Restart` is delivered, and first-party consumers follow it, fixing Interop's `unroutable` rounds - [Wire compat on main](/quest/m0/wire-compat-green.md) - the nightly released-wire step runs again: planned breaks re-measured against the new releases, and a released-API client for the released JS lanes +- [Release builds link](/quest/m0/release-link.md) - the Linux zigbuild release binaries and moq-gst packages link again after the 2026-10-11 nixpkgs bump, so the next release cut can publish +- [Fuzz builds under LTO](/quest/m0/fuzz-lto.md) - the nightly fuzz targets build again despite the release profile's fat LTO +- [Audio quality budgets](/quest/m0/audio-quality-budgets.md) - the nightly Audio quality job grades against ceilings measured on its own runner, so a breach means a regression again - [web-transport releases the qmux fixes](/quest/m0/qmux-credit-upstream.md) - waiting on moq-dev/web-transport#412 and #413 to merge and ship, which qmux credit bumps to - [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription diff --git a/quest/m0/audio-quality-budgets.md b/quest/m0/audio-quality-budgets.md new file mode 100644 index 0000000000..97d8de8544 --- /dev/null +++ b/quest/m0/audio-quality-budgets.md @@ -0,0 +1,44 @@ +# [S] Audio quality budgets come from the nightly runner + +## Goal + +The nightly `Audio quality` job passes on `main`'s current player, and a breach +means a regression again. The browser rows' ceilings in +`test/audio-quality/budgets.json` are measured on the nightly runner, not on a +developer machine, and the file's `note` says how. + +Non-goals: improving playout to meet tighter ceilings. That stays with +[Watch](/quest/m1/audio-jitter-target/watch.md). The replay rows, which PRs +already enforce, keep their own budgets. + +## Plan + +Seen 2026-10-02 to 2026-10-12: the job has failed every night since #4395 +added it. The 2026-10-12 run (38276133321) breached 27 keys across the +Chromium rows, for example `chromium-aac-44100-bursty-isolated` at a +`stalled_share` of 0.324 against 0.01, and `target_ms_p95` on most Opus +rows. The browser ceilings were recorded locally (#4426), but nightly +measures on an `ubuntu-24.04-arm` runner. A red job nobody can act on hides +real regressions, so this moved out of Watch into m0 (maintainer, +2026-10-13). + +- Measure several nightly runs, or a matrix dispatched on the runner from + the branch, before setting ceilings. A single sample's noise would make + the job flaky. Failing runs keep their summaries as the + `audio-quality-run` artifact for 7 days. +- Set each ceiling from the observed spread with modest headroom, and record + the method in `budgets.json`'s `note`. Each loosened ceiling needs its + reason in review (see `test/audio-quality/README.md`). +- Some rows may be genuine defects rather than runner noise. The AAC bursty + rows stalling a third of the time look like one. Budget what the player + does today, and file each suspected defect as a follow-up quest, so the job + goes green without hiding it. +- Rows that are too noisy to bound on a shared runner can be dropped from + `--enforce` instead. That decision needs the maintainer. + +Public API: none. Wire: none. + +## Related + +- [Watch](/quest/m1/audio-jitter-target/watch.md) - tightens these ceilings once playout improves +- [Audio quality native](/quest/m1/audio-quality-native.md) - reuses the same budget file for a native lane diff --git a/quest/m0/fuzz-lto.md b/quest/m0/fuzz-lto.md new file mode 100644 index 0000000000..f97aec2a9e --- /dev/null +++ b/quest/m0/fuzz-lto.md @@ -0,0 +1,39 @@ +# [XS] Fuzz targets build under the release profile's LTO + +## Goal + +Every nightly `Fuzz ()` job builds and runs its target again. A +release-profile change can't silently break the fuzz build, because the +fuzz build opts out of what sanitizer coverage can't link. + +## Plan + +Seen in the 2026-10-12 nightly (run 38276133321): all six fuzz targets fail +to link with `undefined reference to '__sancov_gen_.N'` from +`asan.module_dtor`. The 2026-10-11 nightly passed. + +Cause: #5274 (be8d0024d) set `lto = "fat"`, `codegen-units = 1` and +`strip = "symbols"` on the workspace `[profile.release]`. `cargo fuzz` +builds `--release`, and `rs/moq-net/fuzz` is a workspace member, so it +inherits fat LTO, which drops the sanitizer-coverage globals the ASan module +destructor references. Reproduced locally on the dev shell's rustc with +`RUSTC_BOOTSTRAP=1` and cargo-fuzz's flags (`cargo build --manifest-path +rs/moq-net/fuzz/Cargo.toml --release --bin lite`): fails as in CI, and links +with `CARGO_PROFILE_RELEASE_LTO=false`. + +Likely a one-line fix: `sh/rs/fuzz.sh` already exports `RUSTFLAGS`, so it can +override the profile the same way (for example +`CARGO_PROFILE_RELEASE_LTO=false`). Consider `strip = "none"` too, so ASan +and libFuzzer crash reports keep symbol names. Prefer whatever keeps the +fuzz build's settings next to the fuzz build rather than another profile in +the root `Cargo.toml`, unless cargo-fuzz can select one cleanly. + +Verify with `just rs fuzz -- -max_total_time=10` for each target. +The nightly is the only CI that builds fuzz targets, so the PR can't prove +it on CI; say so in the PR. + +Public API: none. Wire: none. + +## Related + +- [Release builds link again](/quest/m0/release-link.md) - the other build break in the same nightly, from a different cause diff --git a/quest/m0/release-link.md b/quest/m0/release-link.md new file mode 100644 index 0000000000..7a6804b860 --- /dev/null +++ b/quest/m0/release-link.md @@ -0,0 +1,47 @@ +# [S] Release builds link again on the current Nix toolchain + +## Goal + +The Linux release binaries (`moq-cli`, `moq-relay`) and the `moq-gst` +.deb/.rpm packages build with `cargo zigbuild` on `main`'s pinned Nix inputs, +for x86_64 and aarch64, so the next release cut from `main` can publish. + +## Plan + +Seen in the 2026-10-12 nightly (run 38276133321): `moq-cli / release / +Build`, `moq-relay / release / Build` and `moq-gst / Package .deb/.rpm` +fail on both Linux targets. Linking `iroh-relay`'s cdylib fails with +`ld.lld: error: undefined symbol:` (an empty name), referenced by +`bcm.o:(aws_lc_0_45_0_OPENSSL_cpuid_setup)` in `aws-lc-sys`. The 2026-10-11 +nightly passed. macOS and Windows release builds still pass. + +Onset: the nixpkgs bump in #5316 (d0736ec04), not #5274's LTO change, which +landed the same night. Reproduced with +`cargo zigbuild --locked --release --target x86_64-unknown-linux-gnu.2.34 -p iroh-relay --lib`: + +- Current `flake.lock`: fails as in CI, also with + `CARGO_PROFILE_RELEASE_LTO=false` and with `CARGO_PROFILE_RELEASE_STRIP=none`. +- `flake.lock` from 717f3fecc (before #5316), same `Cargo.toml` and + `Cargo.lock`: links. +- Both shells have zig 0.16.0 and rustc 1.98.1. What changed is GCC + 15.3 -> 16.2, glibc 2.42 -> 2.44, and smaller bumps. +- `bcm.o` in the new build has 292 undefined `SECTION` symbols in + `readelf -s`; the old one has none. Both report clang 21.1.8 in `.comment`, + so something other than the compiler zig picks differs, possibly a step in + `aws-lc-sys`'s build that reaches the host toolchain. + +Find which tool produces the broken object and fix it at the source: an +`aws-lc-sys` or cargo-zigbuild environment fix, or an upstream report if the +bug is theirs. Reverting nixpkgs is a stopgap, not the fix; if it's needed +to unblock a release, make it its own quest. + +`release` still pins the pre-bump nixpkgs, so shipping releases are fine +today; the next `main` -> `release` merge carries the break. PR CI never runs +zigbuild, so nothing caught it before the nightly. Consider a cheap PR-time +check when `flake.lock` changes, if the fix shows one is worth it. + +Public API: none. Wire: none. + +## Related + +- [Fuzz targets build under LTO](/quest/m0/fuzz-lto.md) - the other build break in the same nightly, from a different cause diff --git a/quest/m1/audio-jitter-target/watch.md b/quest/m1/audio-jitter-target/watch.md index 1cce2e7f9e..fe80fac882 100644 --- a/quest/m1/audio-jitter-target/watch.md +++ b/quest/m1/audio-jitter-target/watch.md @@ -32,9 +32,9 @@ about 22 s with skip-aheads on the way down. - Decide whether those re-stall refills are what the goal wants, using the replay rows as the measurement. Tighten the replay budgets with any fix. - Re-record the browser rows' `test/audio-quality/budgets.json` from the - nightly runner's first runs on `main` before tightening them, since they were - recorded locally (#4426) and nightly only runs `main`'s code. + Tighten the browser rows only from the ceilings + [Audio quality budgets](/quest/m0/audio-quality-budgets.md) measures on the + nightly runner. - The replay sizes every frame as the trace's median spacing, which a trace missing most of its frames defeats; record the codec's frame duration in the trace and replay that instead. @@ -59,3 +59,4 @@ release. ## Related - [A/V clock](/quest/m1/av-clock.md) - reshapes `SyncInput` around the per-track target this line produces +- [Audio quality budgets](/quest/m0/audio-quality-budgets.md) - measures the browser rows' ceilings on the nightly runner, which this quest tightens