diff --git a/quest/m1/media-audio-tone.md b/quest/m1/media-audio-tone.md index c1b8f1bc51..553eb78554 100644 --- a/quest/m1/media-audio-tone.md +++ b/quest/m1/media-audio-tone.md @@ -4,7 +4,8 @@ The `just test media` audio-tone check passes under load, fixed at its cause. It failed with 66 of 75 samples audible in #4719's runs and 72 of 81 in -the 2026-10-06 late-join loops (#4914), and passed on rerun. +the 2026-10-06 late-join loops (#4914), 68 of 81 once in 45 late-join runs +on 2026-10-09 under `stress-ng --cpu 20`, and passed on rerun. ## Plan @@ -20,5 +21,4 @@ loaded check. ## Related - [More tests under load](/quest/m1/test-flakes-2/README.md) - shares the loaded-check validation -- [Media late join](/quest/m1/test-flakes-2/media-late-join.md) - shares the media fixture, but tracks a separate failure - [Forced channel count](/quest/m1/publish-audio-channel-count.md) - another source of audio gaps on the JS publish path; rule it in or out first diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index 1b43c6790c..3488db9e6b 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -32,7 +32,6 @@ Public API: none. Wire: none. ## Required -- [Media late join](/quest/m1/test-flakes-2/media-late-join.md) - a late joiner shows video promptly and catches up to live, and the check asserts that under load - [Relay restart rebind](/quest/m1/test-flakes-2/relay-restart-rebind.md) - the crash drill restarts on its original UDP address under concurrent load - [Impaired handshake](/quest/m1/test-flakes-2/impaired-handshake.md) - the impaired cluster drills' clients never time out while connecting - [Media audio tone](/quest/m1/media-audio-tone.md) - the `just test media` audio-tone check passes under load, fixed at its cause diff --git a/quest/m1/test-flakes-2/media-late-join.md b/quest/m1/test-flakes-2/media-late-join.md deleted file mode 100644 index e22bb0d9d8..0000000000 --- a/quest/m1/test-flakes-2/media-late-join.md +++ /dev/null @@ -1,54 +0,0 @@ -# [S] Media late join catches up to live - -## Goal - -A viewer joining a broadcast through a relay after a demand gap shows video -promptly and reaches the live edge within a bound, and the late-join check -asserts exactly that and passes under load. - -## Plan - -It failed once after [#4181](https://github.com/moq-dev/moq/pull/4181) ("16 -frames behind" a 15-frame GOP), and the interop browser check failed the -same way on `main`. `just test media` is that browser lane through a Rust -`moq-relay`; there is no separate Rust media test. -[#4719](https://github.com/moq-dev/moq/pull/4719) now compares the -latecomer against the published keyframe instead of the painted counter. - -It is real, not fixture sampling: the latecomer is handed a GOP from before -the demand gap. Reproduced by looping `bun media.ts --cases late-join` -against a local relay with debug logs. In the fixture, the old viewer -leaves, the publisher pauses (closing its open GOP and writing an empty -marker group), and the latecomer joins about a second later. - -[#4914](https://github.com/moq-dev/moq/pull/4914) fixed two relay windows, -each with a deterministic test: - -- A received group stays hidden from readers until its first frame lands or - its stream ends, and an idle copy goes live only once the cache shows the - route's answer (`rejoin_waits_for_the_answers_first_frame`). -- A rejoin's answer is judged against the groups that survived the leave, - not the idle snapshot, since the cancel resets the group in flight - (`an_answer_past_a_reset_group_skips_the_cache`). - -What remains (1 of 30 runs after both fixes): the latecomer's max delay rises -to its playout delay (about 145ms) right after it subscribes. When the -pre-gap GOP was short, the GOP before it still reaches within that budget of -the marker, so the relay hands it over by design. The player then presents -from that GOP's keyframe, not from its playout position. Relay log: groups -7, 8, 9 served in 1ms with a 145ms budget, and the player shows frames 360ms -behind the published keyframe. - -Decided 2026-10-06 by the maintainer: a joiner showing the reachable pre-gap -GOP at once and then fast-forwarding to live is the wanted behavior, so no -player or consumer change. The check is what's too strict: it asserts the -first presented frame is at or after the newest published keyframe. - -Remaining work: the late-join check (`just test media` and the interop -browser lane) asserts what a viewer cares about instead: video shows within a -start bound, and the player reaches the live edge within a catch-up bound. -Set both bounds from measurements, looped under synthetic CPU load (the -maintainer approved a bounded load generator such as `stress-ng` for these -runs), and keep the relay fixes' deterministic tests as the regression guard. - -Public API: none. Wire: none. diff --git a/test/interop/README.md b/test/interop/README.md index 3359d53444..8fc5d53e08 100644 --- a/test/interop/README.md +++ b/test/interop/README.md @@ -186,10 +186,11 @@ Each run covers, against a real local relay: The fake device is not physical hardware, and the headless permission decision is not a person clicking a browser prompt. - **pause and resume**, **unsubscribe and rejoin**, **detach and reattach**, - **publisher stop and same-path republish**, and **late join**. The late join - must present the newest published GOP or a newer one. Its lower bound is the - encoded keyframe timestamp sampled before the existing viewer closes, since - painting the canvas does not mean capture and encoding have finished. + **publisher stop and same-path republish**, and **late join**. The late + joiner may first show the GOP from before the demand gap, but it must present + a frame within 5s of loading and then come within 1.5s of the fixture's + painted frame within 2s, both bounds about twice the worst seen in looped runs + under CPU load. - **resources return to baseline** - the page wraps `WebTransport`, `WebSocket`, `AudioContext`, and `Worker` to count live instances, so a detach that leaks a session is visible rather than merely invisible. diff --git a/test/interop/clients/js/media.ts b/test/interop/clients/js/media.ts index 2d24122eb4..3570b6ea1b 100644 --- a/test/interop/clients/js/media.ts +++ b/test/interop/clients/js/media.ts @@ -47,7 +47,6 @@ import { import { type CaptureState, FAULTS, - lateJoinStartsLive, leakedPlayerStarted, SAMPLE_MS, SAMPLE_RATE, @@ -166,6 +165,61 @@ async function waitFrozen(page: Page, errors: BrowserErrors, assertion: string, throw new Failure(assertion, `${description}: the presented frame is still advancing, now ${frame}`); } +/** + * A presented frame at most this far behind the fixture's painted counter counts as live. + * + * The lag spans capture, encode, relay, playout delay, and decode. Loaded runs settle at 4 to 24 + * frames with spikes to 33, while a player stuck on the GOP from before a demand gap sits 60 or more + * behind. + */ +const LIVE_LAG_FRAMES = 45; + +/** How long a latecomer may take to present its first frame, from its page loading. Loaded runs take up to 2.6s. */ +const LATE_START_MS = 5000; + +/** How long a latecomer may take to reach live after its first frame. Loaded runs take up to 0.3s. */ +const LATE_CATCHUP_MS = 2000; + +const framesToMs = (frames: number) => Math.round((frames * 1000) / Pattern.FPS); + +/** + * Wait until the player presents a frame within {@link LIVE_LAG_FRAMES} of the one the fixture last + * painted, and report how far behind its first reading was. + * + * Both pages are read at once, in separate browsers, so the lag is what a viewer would see against + * the source. + */ +async function waitForLive( + publisher: Page, + publisherErrors: BrowserErrors, + player: Page, + playerErrors: BrowserErrors, +): Promise<{ first: number; lag: number; after: number }> { + const since = Date.now(); + let first: number | undefined; + let lag: number | undefined; + while (Date.now() < since + LATE_CATCHUP_MS) { + throwPageErrors(publisherErrors); + throwPageErrors(playerErrors); + const [fixture, state] = await Promise.all([readFixtureState(publisher), readPlayerState(player)]).catch( + () => [], + ); + if (fixture && state?.frameId !== undefined) { + lag = fixture.frameId - state.frameId; + first ??= lag; + if (lag <= LIVE_LAG_FRAMES) return { first, lag, after: Date.now() - since }; + } + await sleep(POLL_INTERVAL_MS); + } + throwPageErrors(publisherErrors); + throwPageErrors(playerErrors); + throw new Failure( + "late join reaches live", + `still ${lag === undefined ? "?" : framesToMs(lag)}ms behind the fixture ${LATE_CATCHUP_MS}ms after the first frame, ` + + `against a ${framesToMs(LIVE_LAG_FRAMES)}ms bound`, + ); +} + // One sample as a line: elapsed time, the frame on the canvas, the tone step heard against the one // that frame belongs to, how far the tone stood above the noise floor, and whether audio was // arriving at all, which separates a silent player from a publisher that stopped sending. @@ -691,30 +745,29 @@ try { } // ── late join ──────────────────────────────────────────────────────────── - // A fresh page against a broadcast that has been running for a while: the player has to tune in - // at the live edge rather than replay what it missed. + // A fresh page against a broadcast that has been running for a while, after a demand gap. The + // relay may hand over the GOP from before the gap, so the first frame can be stale; the player + // has to show it promptly and then fast-forward to live rather than replay what it missed. if (wants("late-join")) { console.error("=== late join ==="); - const live = await command(publisher, "liveGop"); await player.close(); [player, playerErrors] = await subscriber(broadcast, "latecomer"); + const joined = Date.now(); await gesture(player); - const joined = await waitForState(player, playerErrors, { - deadline: Date.now() + timeoutMs, - assertion: "late join presents media", - description: "the latecomer to present the fixture", - predicate: (state) => state.frameId !== undefined && state.audioContext === "running", + + const shown = await waitForState(player, playerErrors, { + deadline: joined + LATE_START_MS, + assertion: "late join shows video", + description: `a presented fixture frame within ${LATE_START_MS}ms of the latecomer loading`, + predicate: (state) => state.frameId !== undefined, }); - // requestFrame() is asynchronous, and capture timestamps decide keyframes. The painted - // counter can therefore be over 15 ticks ahead of a still-current encoded GOP under load. - // Compare against its actual published keyframe, sampled while the old viewer held demand. - check( - lateJoinStartsLive(live, joined.videoTimestamp), - "late join starts live", - () => - `joined at frame ${joined.frameId}, timestamp ${joined.videoTimestamp}ms before the current GOP's ${live.timestamp}ms keyframe`, + const startMs = Date.now() - joined; + + const live = await waitForLive(publisher, publisherErrors, player, playerErrors); + console.error( + ` showed frame ${shown.frameId} after ${startMs}ms, ${framesToMs(live.first)}ms behind the fixture; ` + + `live ${live.after}ms later, ${framesToMs(live.lag)}ms behind`, ); - console.error(` joined at frame ${joined.frameId}, timestamp ${joined.videoTimestamp}ms, current GOP began at ${live.timestamp}ms`); assertMedia(await collect(player, playerErrors, WINDOW_MS), "late join"); } diff --git a/test/interop/clients/js/src/contract.test.ts b/test/interop/clients/js/src/contract.test.ts index 33deb26acc..d22d56c5a5 100644 --- a/test/interop/clients/js/src/contract.test.ts +++ b/test/interop/clients/js/src/contract.test.ts @@ -1,6 +1,5 @@ import { expect, test } from "bun:test"; -import * as Moq from "@moq/net"; -import { lateJoinStartsLive, leakedPlayerStarted, readLiveGop, type Resources } from "./contract"; +import { leakedPlayerStarted, type Resources } from "./contract"; const playing: Resources = { transports: 1, sockets: 0, audioContexts: 1, workers: 0 }; @@ -19,36 +18,3 @@ test("unchanged counts are not a leak start", () => { expect(leakedPlayerStarted(playing, playing)).toBe(false); }); - -test("late join uses the published GOP when painting is ahead of capture", async () => { - const broadcast = new Moq.Broadcast.Producer(); - const track = broadcast.createTrack("video", { timescale: Moq.Time.Timescale.MILLI }); - try { - const old = track.appendGroup(); - old.writeFrame({ payload: new Uint8Array([1]), timestamp: Moq.Time.Timestamp.fromMillis(3000) }); - old.close(); - const current = track.appendGroup(); - current.writeFrame({ payload: new Uint8Array([2]), timestamp: Moq.Time.Timestamp.fromMillis(3700) }); - current.writeFrame({ payload: new Uint8Array([3]), timestamp: Moq.Time.Timestamp.fromMillis(4199) }); - - // Painted counter 127 can precede capture while GOP 111 remains current. The old - // 127 - 111 <= 15 check rejects a valid join; an earlier published GOP must fail. - const viewer = track.subscribe(); - const gop = await readLiveGop(broadcast.track("video")); - expect(gop.timestamp).toBe(3700); - expect(lateJoinStartsLive(gop, 3700)).toBe(true); - expect(lateJoinStartsLive(gop, 4199)).toBe(true); - expect(lateJoinStartsLive(gop, 3000)).toBe(false); - expect(lateJoinStartsLive(gop, undefined)).toBe(false); - expect(track.demand().used.peek()).toBe(true); - const viewed = await viewer.recvGroup(); - expect((await viewed?.readFrame())?.payload).toEqual(new Uint8Array([2])); - expect((await viewed?.readFrame())?.payload).toEqual(new Uint8Array([3])); - viewed?.close(); - viewer.close(); - await track.demand().unused(); - expect(track.demand().used.peek()).toBe(false); - } finally { - broadcast.close(); - } -}); diff --git a/test/interop/clients/js/src/contract.ts b/test/interop/clients/js/src/contract.ts index 1211e7d087..cb5562dcb6 100644 --- a/test/interop/clients/js/src/contract.ts +++ b/test/interop/clients/js/src/contract.ts @@ -8,8 +8,6 @@ * @module */ -import type * as Moq from "@moq/net"; - // ── the deterministic publisher ───────────────────────────────────────────── /** A deliberate defect, used to prove an assertion can fail. */ @@ -47,36 +45,6 @@ export type FixtureState = { videoEnabled: boolean; }; -/** The newest published video's GOP, sampled independently of canvas capture and encoding. */ -export type LiveGop = { - /** The timestamp of its keyframe, in milliseconds on the publisher's media clock. */ - timestamp: number; -}; - -/** Read the newest published GOP without retaining a subscription or changing its frames. */ -export async function readLiveGop(track: Moq.Track.Consumer): Promise { - const subscriber = track.subscribe(); - try { - const group = await subscriber.recvGroup(); - if (!group) throw new Error("the fixture video has no published GOP"); - try { - const frame = await group.readFrame(); - if (!frame) throw new Error("the fixture GOP has no keyframe"); - if (!frame.timestamp) throw new Error("the fixture video arrived untimed"); - return { timestamp: frame.timestamp.asMillis() }; - } finally { - group.close(); - } - } finally { - subscriber.close(); - } -} - -/** True when the presented frame belongs to the sampled GOP or a newer one. */ -export function lateJoinStartsLive(gop: LiveGop, timestamp: number | undefined): boolean { - return timestamp !== undefined && timestamp >= gop.timestamp; -} - /** The camera publisher state mirrored onto its element for Playwright. */ export type CaptureState = { videoError?: string; @@ -204,8 +172,6 @@ export type CloseState = * owns them rather than the driver reaching in. */ export type InteropControl = { - /** Read the current video GOP while an existing viewer is still pulling media. */ - liveGop(): Promise; /** Stop the fixture publisher, releasing its session. */ stop(): void; /** Start (or restart) the fixture publisher on the same broadcast path. */ diff --git a/test/interop/clients/js/src/fixture.ts b/test/interop/clients/js/src/fixture.ts index d8f141a78f..dbbe25ab67 100644 --- a/test/interop/clients/js/src/fixture.ts +++ b/test/interop/clients/js/src/fixture.ts @@ -17,7 +17,7 @@ import { Time } from "@moq/net"; import * as Publish from "@moq/publish"; import { Effect, Signal } from "@moq/signals"; import type { Fault, FixtureState } from "./contract"; -import { KEYFRAME_INTERVAL_MS, OFFSET_STEPS, readLiveGop, SAMPLE_RATE } from "./contract"; +import { KEYFRAME_INTERVAL_MS, OFFSET_STEPS, SAMPLE_RATE } from "./contract"; import * as Pattern from "./pattern"; /** Cap the encoder rather than letting it track a bandwidth estimate, so runs are comparable. */ @@ -47,7 +47,6 @@ export class Fixture { readonly host: HTMLElement; readonly #signals = new Effect(); - readonly #broadcast: Publish.Broadcast; readonly #audio: AudioContext; readonly #frameId = new Signal(-1); readonly #audioState = new Signal("suspended"); @@ -97,8 +96,6 @@ export class Fixture { }); this.#signals.cleanup(() => broadcast.close()); - this.#broadcast = broadcast; - const video = new Publish.Video.Encoder("video", { broadcast, capture, @@ -220,13 +217,6 @@ export class Fixture { this.#videoEnabled.set(enabled); } - /** Sample the current published keyframe before the viewer closes its subscription. */ - async liveGop() { - const broadcast = this.#broadcast.net.peek(); - if (!broadcast) throw new Error("the fixture is not publishing"); - return readLiveGop(broadcast.track("video")); - } - /** Stop publishing and release the capture, audio graph, and session. */ close(): void { this.#signals.close(); diff --git a/test/interop/clients/js/src/setup.ts b/test/interop/clients/js/src/setup.ts index c6525e4ae1..46bc44b22f 100644 --- a/test/interop/clients/js/src/setup.ts +++ b/test/interop/clients/js/src/setup.ts @@ -57,10 +57,6 @@ if (role === "publish") { // The driver stops and restarts the publisher in place to exercise a same-path republish, which // has to reuse this page so the audio context keeps its user activation. publish({ - liveGop: () => { - if (!fixture) throw new Error("the fixture is stopped"); - return fixture.liveGop(); - }, stop: () => { fixture?.close(); fixture = undefined;