Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
01c8024
chore(quest): start test stability questline
kixelated Oct 1, 2026
5900438
Merge remote-tracking branch 'origin/main' into quest/m1/test-flakes-…
kixelated Oct 1, 2026
8aa6017
test(tokio): make broadcast transport race deterministic (#4664)
kixelated Oct 1, 2026
baeaf3e
test(shaper): run the unit tests on paused time over in-memory socket…
kixelated Oct 1, 2026
c63c01e
quest: claim test-flakes-2/warn-capture
kixelated Oct 1, 2026
113bb17
quest: claim mux-debounce-clock
kixelated Oct 1, 2026
d965302
moq-mux: run the TS SI debounce on the tokio clock
kixelated Oct 2, 2026
c8db5fb
moq-net: count drop WARNs through a global subscriber, per thread
kixelated Oct 2, 2026
1265ae0
quest: plan moq-mux live import clock and tracing capture audit
kixelated Oct 2, 2026
7533eb3
Merge pull request #4687 from moq-dev/quest/m1/test-flakes-2/mux-debo…
kixelated Oct 2, 2026
5d03e8c
Merge remote-tracking branch 'origin/quest/m1/test-flakes-2/README' i…
kixelated Oct 2, 2026
f1c5e78
Merge remote-tracking branch 'origin/quest/m1/test-flakes-2/README' i…
kixelated Oct 2, 2026
ac25d88
quest: address review on live import clock and tracing capture audit
kixelated Oct 2, 2026
7a93493
moq-net: rebuild callsite interest at each WARN capture
kixelated Oct 2, 2026
ca455c8
Merge pull request #4693 from moq-dev/quest/plan-flake-followups
kixelated Oct 2, 2026
7f096f7
Merge remote-tracking branch 'origin/quest/m1/test-flakes-2/README' i…
kixelated Oct 2, 2026
3873e5d
quest: settle the moq-mux Clock boundary
kixelated Oct 2, 2026
04fca7e
Merge pull request #4690 from moq-dev/quest/m1/test-flakes-2/warn-cap…
kixelated Oct 2, 2026
1ab08a5
Merge remote-tracking branch 'origin/quest/m1/test-flakes-2/README' i…
kixelated Oct 2, 2026
bbc16bb
Merge pull request #4700 from moq-dev/quest/plan-clock-boundary
kixelated Oct 2, 2026
126c6a8
chore: claim media late-join quest
kixelated Oct 2, 2026
70321a0
test(media): measure late join against the published GOP
kixelated Oct 2, 2026
0165839
quest: claim media-late-join
kixelated Oct 10, 2026
b4b39cd
test(media): late join asserts a prompt start and a bounded catch-up
kixelated Oct 10, 2026
921af44
Merge remote-tracking branch 'origin/quest/m1/test-flakes-2/media-lat…
kixelated Oct 10, 2026
e8760b9
Merge remote-tracking branch 'origin/main' into media-late-join-merge
kixelated Oct 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions quest/m1/media-audio-tone.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@

The `just test media` audio-tone check passes under load, fixed at its cause.
It failed with 66 of 75 samples audible in #4719's runs and 72 of 81 in
the 2026-10-06 late-join loops (#4914), and passed on rerun.
the 2026-10-06 late-join loops (#4914), 68 of 81 once in 45 late-join runs
on 2026-10-09 under `stress-ng --cpu 20`, and passed on rerun.

## Plan

Expand All @@ -20,5 +21,4 @@ loaded check.
## Related

- [More tests under load](/quest/m1/test-flakes-2/README.md) - shares the loaded-check validation
- [Media late join](/quest/m1/test-flakes-2/media-late-join.md) - shares the media fixture, but tracks a separate failure
- [Forced channel count](/quest/m1/publish-audio-channel-count.md) - another source of audio gaps on the JS publish path; rule it in or out first
1 change: 0 additions & 1 deletion quest/m1/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ Public API: none. Wire: none.

## Required

- [Media late join](/quest/m1/test-flakes-2/media-late-join.md) - a late joiner shows video promptly and catches up to live, and the check asserts that under load
- [Relay restart rebind](/quest/m1/test-flakes-2/relay-restart-rebind.md) - the crash drill restarts on its original UDP address under concurrent load
- [Impaired handshake](/quest/m1/test-flakes-2/impaired-handshake.md) - the impaired cluster drills' clients never time out while connecting
- [Media audio tone](/quest/m1/media-audio-tone.md) - the `just test media` audio-tone check passes under load, fixed at its cause
54 changes: 0 additions & 54 deletions quest/m1/test-flakes-2/media-late-join.md

This file was deleted.

9 changes: 5 additions & 4 deletions test/interop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,10 +186,11 @@ Each run covers, against a real local relay:
The fake device is not physical hardware, and the headless permission decision
is not a person clicking a browser prompt.
- **pause and resume**, **unsubscribe and rejoin**, **detach and reattach**,
**publisher stop and same-path republish**, and **late join**. The late join
must present the newest published GOP or a newer one. Its lower bound is the
encoded keyframe timestamp sampled before the existing viewer closes, since
painting the canvas does not mean capture and encoding have finished.
**publisher stop and same-path republish**, and **late join**. The late
joiner may first show the GOP from before the demand gap, but it must present
a frame within 5s of loading and then come within 1.5s of the fixture's
painted frame within 2s, both bounds about twice the worst seen in looped runs
under CPU load.
- **resources return to baseline** - the page wraps `WebTransport`, `WebSocket`,
`AudioContext`, and `Worker` to count live instances, so a detach that leaks a
session is visible rather than merely invisible.
Expand Down
89 changes: 71 additions & 18 deletions test/interop/clients/js/media.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@ import {
import {
type CaptureState,
FAULTS,
lateJoinStartsLive,
leakedPlayerStarted,
SAMPLE_MS,
SAMPLE_RATE,
Expand Down Expand Up @@ -166,6 +165,61 @@ async function waitFrozen(page: Page, errors: BrowserErrors, assertion: string,
throw new Failure(assertion, `${description}: the presented frame is still advancing, now ${frame}`);
}

/**
* A presented frame at most this far behind the fixture's painted counter counts as live.
*
* The lag spans capture, encode, relay, playout delay, and decode. Loaded runs settle at 4 to 24
* frames with spikes to 33, while a player stuck on the GOP from before a demand gap sits 60 or more
* behind.
*/
const LIVE_LAG_FRAMES = 45;

/** How long a latecomer may take to present its first frame, from its page loading. Loaded runs take up to 2.6s. */
const LATE_START_MS = 5000;

/** How long a latecomer may take to reach live after its first frame. Loaded runs take up to 0.3s. */
const LATE_CATCHUP_MS = 2000;

const framesToMs = (frames: number) => Math.round((frames * 1000) / Pattern.FPS);

/**
* Wait until the player presents a frame within {@link LIVE_LAG_FRAMES} of the one the fixture last
* painted, and report how far behind its first reading was.
*
* Both pages are read at once, in separate browsers, so the lag is what a viewer would see against
* the source.
*/
async function waitForLive(
publisher: Page,
publisherErrors: BrowserErrors,
player: Page,
playerErrors: BrowserErrors,
): Promise<{ first: number; lag: number; after: number }> {
const since = Date.now();
let first: number | undefined;
let lag: number | undefined;
while (Date.now() < since + LATE_CATCHUP_MS) {
throwPageErrors(publisherErrors);
throwPageErrors(playerErrors);
const [fixture, state] = await Promise.all([readFixtureState(publisher), readPlayerState(player)]).catch(
() => [],
);
if (fixture && state?.frameId !== undefined) {
lag = fixture.frameId - state.frameId;
first ??= lag;
if (lag <= LIVE_LAG_FRAMES) return { first, lag, after: Date.now() - since };
}
await sleep(POLL_INTERVAL_MS);
}
throwPageErrors(publisherErrors);
throwPageErrors(playerErrors);
throw new Failure(
"late join reaches live",
`still ${lag === undefined ? "?" : framesToMs(lag)}ms behind the fixture ${LATE_CATCHUP_MS}ms after the first frame, ` +
`against a ${framesToMs(LIVE_LAG_FRAMES)}ms bound`,
);
}

// One sample as a line: elapsed time, the frame on the canvas, the tone step heard against the one
// that frame belongs to, how far the tone stood above the noise floor, and whether audio was
// arriving at all, which separates a silent player from a publisher that stopped sending.
Expand Down Expand Up @@ -691,30 +745,29 @@ try {
}

// ── late join ────────────────────────────────────────────────────────────
// A fresh page against a broadcast that has been running for a while: the player has to tune in
// at the live edge rather than replay what it missed.
// A fresh page against a broadcast that has been running for a while, after a demand gap. The
// relay may hand over the GOP from before the gap, so the first frame can be stale; the player
// has to show it promptly and then fast-forward to live rather than replay what it missed.
if (wants("late-join")) {
console.error("=== late join ===");
const live = await command(publisher, "liveGop");
await player.close();
[player, playerErrors] = await subscriber(broadcast, "latecomer");
const joined = Date.now();
Comment on lines 754 to +755

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Start the late-join deadline before opening the page

subscriber() does not return until page.goto(..., { waitUntil: "load" }) completes and waitForWatch() observes the player's first probe sample, which can take up to 30 seconds. Starting joined afterward means a page that spends, for example, 20 seconds initializing and already has a frame when sampling begins still passes this 5-second check, so the test does not enforce its stated bound from page loading. Capture the start time before opening the latecomer, or return the navigation start time from subscriber().

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disagree, leaving this as is. waitForWatch() cannot take long here: data-interop-ready is set synchronously by attach(el) in src/setup.ts, in the same module script that creates the player, so it is already present when page.goto(..., { waitUntil: "load" }) resolves and waitForWatch() returns on its first poll. The window that is not covered runs from module execution to the load event on a local page, which is the "page loading" the bound names. Starting the clock before subscriber() would also count launching a fresh Chromium (browserFor()), which is not player behavior and is the slow, noisy part on a small CI runner. The measured start values (max 2.6s under load) were taken with this placement.

(Written by Claude Opus 5.5)

await gesture(player);
const joined = await waitForState(player, playerErrors, {
deadline: Date.now() + timeoutMs,
assertion: "late join presents media",
description: "the latecomer to present the fixture",
predicate: (state) => state.frameId !== undefined && state.audioContext === "running",

const shown = await waitForState(player, playerErrors, {
deadline: joined + LATE_START_MS,
assertion: "late join shows video",
description: `a presented fixture frame within ${LATE_START_MS}ms of the latecomer loading`,
predicate: (state) => state.frameId !== undefined,
});
// requestFrame() is asynchronous, and capture timestamps decide keyframes. The painted
// counter can therefore be over 15 ticks ahead of a still-current encoded GOP under load.
// Compare against its actual published keyframe, sampled while the old viewer held demand.
check(
lateJoinStartsLive(live, joined.videoTimestamp),
"late join starts live",
() =>
`joined at frame ${joined.frameId}, timestamp ${joined.videoTimestamp}ms before the current GOP's ${live.timestamp}ms keyframe`,
const startMs = Date.now() - joined;

const live = await waitForLive(publisher, publisherErrors, player, playerErrors);
console.error(
` showed frame ${shown.frameId} after ${startMs}ms, ${framesToMs(live.first)}ms behind the fixture; ` +
`live ${live.after}ms later, ${framesToMs(live.lag)}ms behind`,
);
console.error(` joined at frame ${joined.frameId}, timestamp ${joined.videoTimestamp}ms, current GOP began at ${live.timestamp}ms`);
assertMedia(await collect(player, playerErrors, WINDOW_MS), "late join");
}

Expand Down
36 changes: 1 addition & 35 deletions test/interop/clients/js/src/contract.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import { expect, test } from "bun:test";
import * as Moq from "@moq/net";
import { lateJoinStartsLive, leakedPlayerStarted, readLiveGop, type Resources } from "./contract";
import { leakedPlayerStarted, type Resources } from "./contract";

const playing: Resources = { transports: 1, sockets: 0, audioContexts: 1, workers: 0 };

Expand All @@ -19,36 +18,3 @@ test("unchanged counts are not a leak start", () => {
expect(leakedPlayerStarted(playing, playing)).toBe(false);
});


test("late join uses the published GOP when painting is ahead of capture", async () => {
const broadcast = new Moq.Broadcast.Producer();
const track = broadcast.createTrack("video", { timescale: Moq.Time.Timescale.MILLI });
try {
const old = track.appendGroup();
old.writeFrame({ payload: new Uint8Array([1]), timestamp: Moq.Time.Timestamp.fromMillis(3000) });
old.close();
const current = track.appendGroup();
current.writeFrame({ payload: new Uint8Array([2]), timestamp: Moq.Time.Timestamp.fromMillis(3700) });
current.writeFrame({ payload: new Uint8Array([3]), timestamp: Moq.Time.Timestamp.fromMillis(4199) });

// Painted counter 127 can precede capture while GOP 111 remains current. The old
// 127 - 111 <= 15 check rejects a valid join; an earlier published GOP must fail.
const viewer = track.subscribe();
const gop = await readLiveGop(broadcast.track("video"));
expect(gop.timestamp).toBe(3700);
expect(lateJoinStartsLive(gop, 3700)).toBe(true);
expect(lateJoinStartsLive(gop, 4199)).toBe(true);
expect(lateJoinStartsLive(gop, 3000)).toBe(false);
expect(lateJoinStartsLive(gop, undefined)).toBe(false);
expect(track.demand().used.peek()).toBe(true);
const viewed = await viewer.recvGroup();
expect((await viewed?.readFrame())?.payload).toEqual(new Uint8Array([2]));
expect((await viewed?.readFrame())?.payload).toEqual(new Uint8Array([3]));
viewed?.close();
viewer.close();
await track.demand().unused();
expect(track.demand().used.peek()).toBe(false);
} finally {
broadcast.close();
}
});
34 changes: 0 additions & 34 deletions test/interop/clients/js/src/contract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,6 @@
* @module
*/

import type * as Moq from "@moq/net";

// ── the deterministic publisher ─────────────────────────────────────────────

/** A deliberate defect, used to prove an assertion can fail. */
Expand Down Expand Up @@ -47,36 +45,6 @@ export type FixtureState = {
videoEnabled: boolean;
};

/** The newest published video's GOP, sampled independently of canvas capture and encoding. */
export type LiveGop = {
/** The timestamp of its keyframe, in milliseconds on the publisher's media clock. */
timestamp: number;
};

/** Read the newest published GOP without retaining a subscription or changing its frames. */
export async function readLiveGop(track: Moq.Track.Consumer): Promise<LiveGop> {
const subscriber = track.subscribe();
try {
const group = await subscriber.recvGroup();
if (!group) throw new Error("the fixture video has no published GOP");
try {
const frame = await group.readFrame();
if (!frame) throw new Error("the fixture GOP has no keyframe");
if (!frame.timestamp) throw new Error("the fixture video arrived untimed");
return { timestamp: frame.timestamp.asMillis() };
} finally {
group.close();
}
} finally {
subscriber.close();
}
}

/** True when the presented frame belongs to the sampled GOP or a newer one. */
export function lateJoinStartsLive(gop: LiveGop, timestamp: number | undefined): boolean {
return timestamp !== undefined && timestamp >= gop.timestamp;
}

/** The camera publisher state mirrored onto its element for Playwright. */
export type CaptureState = {
videoError?: string;
Expand Down Expand Up @@ -204,8 +172,6 @@ export type CloseState =
* owns them rather than the driver reaching in.
*/
export type InteropControl = {
/** Read the current video GOP while an existing viewer is still pulling media. */
liveGop(): Promise<LiveGop>;
/** Stop the fixture publisher, releasing its session. */
stop(): void;
/** Start (or restart) the fixture publisher on the same broadcast path. */
Expand Down
12 changes: 1 addition & 11 deletions test/interop/clients/js/src/fixture.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import { Time } from "@moq/net";
import * as Publish from "@moq/publish";
import { Effect, Signal } from "@moq/signals";
import type { Fault, FixtureState } from "./contract";
import { KEYFRAME_INTERVAL_MS, OFFSET_STEPS, readLiveGop, SAMPLE_RATE } from "./contract";
import { KEYFRAME_INTERVAL_MS, OFFSET_STEPS, SAMPLE_RATE } from "./contract";
import * as Pattern from "./pattern";

/** Cap the encoder rather than letting it track a bandwidth estimate, so runs are comparable. */
Expand Down Expand Up @@ -47,7 +47,6 @@ export class Fixture {
readonly host: HTMLElement;

readonly #signals = new Effect();
readonly #broadcast: Publish.Broadcast;
readonly #audio: AudioContext;
readonly #frameId = new Signal(-1);
readonly #audioState = new Signal<AudioContextState>("suspended");
Expand Down Expand Up @@ -97,8 +96,6 @@ export class Fixture {
});
this.#signals.cleanup(() => broadcast.close());

this.#broadcast = broadcast;

const video = new Publish.Video.Encoder("video", {
broadcast,
capture,
Expand Down Expand Up @@ -220,13 +217,6 @@ export class Fixture {
this.#videoEnabled.set(enabled);
}

/** Sample the current published keyframe before the viewer closes its subscription. */
async liveGop() {
const broadcast = this.#broadcast.net.peek();
if (!broadcast) throw new Error("the fixture is not publishing");
return readLiveGop(broadcast.track("video"));
}

/** Stop publishing and release the capture, audio graph, and session. */
close(): void {
this.#signals.close();
Expand Down
4 changes: 0 additions & 4 deletions test/interop/clients/js/src/setup.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,6 @@ if (role === "publish") {
// The driver stops and restarts the publisher in place to exercise a same-path republish, which
// has to reuse this page so the audio context keeps its user activation.
publish({
liveGop: () => {
if (!fixture) throw new Error("the fixture is stopped");
return fixture.liveGop();
},
stop: () => {
fixture?.close();
fixture = undefined;
Expand Down
Loading