feat(moq-mux): catalog delay measures cross-rendition encoder lateness

doc/bin/gstreamer.md

@@ -77,7 +77,8 @@ is at least 1, and a rate is the delta over any window you sample. Unlike
 Set `encoder=true` on audio and video pads a local encoder feeds
 (`x264enc`, `opusenc`, ...). The pad then measures how late each frame reaches
 the sink behind its running time and raises the catalog `jitter` by the spread,
-so players buffer for an encoder that delivers irregularly. Leave it off, the
+and `delay` by how far it trails the earliest such pad, so players buffer for an
+encoder that delivers irregularly or behind the others. Leave it off, the
 default, for file, demuxed, and network media: their arrival reflects the disk
 or the network, not the original encoder, and a GStreamer segment cannot tell
 the two apart. Text and opaque pads refuse it.

doc/bin/obs.md

@@ -35,7 +35,8 @@ OBS Studio install.
 
 OBS reports each locally encoded packet's handoff to libmoq against the shared
 broadcast media clock. Each track's catalog `jitter` is the largest measured
-delay above that track's own recent minimum, rounded up to milliseconds.
+delay above that track's own recent minimum, and its `delay` is how far that
+minimum trails the earliest track, both rounded up to milliseconds.
 
 ## Source quality and moq-transcode
 

doc/concept/audio-jitter.md

@@ -271,6 +271,39 @@ is the one thing a measurement cannot be.
 Note that `js/watch` today adds the two rather than taking the maximum, which
 over-buffers a bursty publisher by its own flush span.
 
+## Across renditions
+
+Every track plays from one clock, anchored on the earliest lateness any track
+has shown. The estimator measures each track against its own fastest frame, so a
+constant offset between tracks cancels out of `measured`: a video encoder that
+flushes 200 ms behind the audio encoder produces the same video target as one
+that does not, and its frames would all arrive 200 ms late for the shared clock.
+
+The catalog `delay` field carries that offset: how far a rendition's minimum
+flush lateness trails the broadcast's earliest rendition, measured by the
+publisher and never lowered. It is an addend, not a floor, because nothing on
+this page contains it. The shared playout is the largest requirement among the
+renditions actually subscribed:
+
+```
+playout = max over subscribed renditions of (delay + target)
+```
+
+A receiver **must not** subtract one rendition's `delay` from another's. Each is
+a lifetime maximum taken against a sliding baseline, so two values need not
+share an origin: if the earliest rendition later drifts behind another, their
+difference understates the real spread. The maximum never under-buffers,
+because the subscribed renditions' spread is measured from an earliest baseline
+no earlier than the broadcast's, so it never exceeds the largest `delay` among
+them. The cost is over-buffering by the smallest subscribed `delay` when the
+broadcast's earliest rendition is not subscribed.
+
+The playout is recomputed when a rendition is subscribed, unsubscribed, or its
+catalog entry rises, so dropping a slow rendition lowers latency. The shared
+reference itself only ever moves earlier, so once the earliest subscribed track
+leaves, the clock stays anchored to it until playback re-anchors; that errs on
+the safe side.
+
 ## Rise, fall, startup, and the ceiling
 
 There is **no separate rise or fall limiter**. The histogram is the only

doc/concept/hang.md

@@ -55,6 +55,7 @@ A few things the catalog can express beyond decoder config:
 - **Labels.** Any rendition may carry a human-readable `label` for a track picker. The map key stays the track name used to subscribe, so labels need not be unique and renaming one doesn't rename the track.
 - **Renditions in another broadcast.** A rendition may point at a relative broadcast path, so a transcoder can publish a ladder that adds low rungs and references the source's original rendition without re-publishing its bytes. The path resolves against where the consumer found the catalog, so a reference that escapes above the root names nothing and the catalog is rejected.
 - **Jitter.** A rendition can say how far its frames fell behind the media clock before the publisher flushed them, in whole milliseconds rounded up. Encoders report the spread of lateness above each rendition's own recent minimum, so a constant encoder delay is not jitter; container imports estimate batch spans without counting ingest delay. It describes the publisher, never the network, only grows over the life of a stream, and a player sizes its buffer to at least this much. A `0` is read as absent.
+- **Delay.** A rendition can also say how far its frames reach the transport behind the broadcast's earliest rendition, measured the same way from each rendition's minimum lateness, so a video encoder running 200 ms behind audio advertises `delay: 200` on video. It follows the same rules as jitter. A player holds the largest `delay + jitter` among the renditions it subscribes to, and never subtracts one rendition's `delay` from another's.
 - **Stalled renditions.** A publisher can flag a rendition as temporarily bad so players prefer another one without the track disappearing. First-party video publishers set this flag after more than three frame intervals of source silence or encoding lag while subscribed, and clear it after three on-time completed frames or when idle. Browser and native capture poll while waiting; FLV and MPEG-TS importers observe video silence as container data arrives. The shared detector is `hang::catalog::stalled::Detector` in Rust and `Catalog.Stalled.Detector` in JavaScript. It is a playback diagnostic, not an authorization or routing signal.
 - **Archive.** A broadcast may advertise an `archive` entry naming its timeline track (a small index of each complete aligned segment) and, if recorded, the replay MoQ path, object-store URL, and format version. The timeline is what lets the [HLS gateway](/bin/hls) build playlists without subscribing to media.
 - **Clock.** The optional root `clock` maps PTS zero to wall time so every media track and the archive index share one fixed epoch after timescale conversion. It is independent of `archive`, so a live-only publisher can expose wall-clock timing without creating a segment index.

doc/lib/rs/moq-mux.md

@@ -41,9 +41,11 @@ retires the entry. Calling `modify` before the first `set` returns
 `Error::NotPublished`. Container writes measure bitrate; importers can also
 measure batch span or reorder delay for jitter. Locally encoded frames call
 `container::Producer::flush(timestamp, Instant::now())`; jitter is the spread
-above that track's own recent minimum lateness, published as soon as it rises.
-Generic imports remain clock-free. Invalid or decreasing jitter is rejected
-before the edit is retained, including while the initial catalog is reserved.
+above that track's own recent minimum lateness, and delay is how far that
+minimum trails the earliest track on the same catalog. Both are published as
+soon as they rise. Generic imports remain clock-free. Invalid or decreasing
+jitter or delay is rejected before the edit is retained, including while the
+initial catalog is reserved.
 Codec importers propagate catalog and media errors through their configuration
 and frame-writing methods.
 

drafts/draft-lcurley-moq-hang.md

@@ -481,6 +481,7 @@ type CommonExtensions = {
   "label": string | undefined,
   "container": Container,
   "jitter": number | undefined,
+  "delay": number | undefined,
 }
 ~~~
 
@@ -532,6 +533,19 @@ For example:
 - A fragment or packet batch contributes the media span between its earliest timestamp and flush point.
 - Reordered frames contribute the delay they were held before flushing, without treating a decode-order presentation timestamp gap as delay by itself.
 
+### delay {#field-delay}
+The maximum amount, in milliseconds, by which a rendition's minimum flush lateness ({{field-jitter}}) has trailed the smallest minimum among the broadcast's renditions that measure it.
+If absent, a consumer SHOULD assume the rendition does not trail the others.
+
+A publisher measures each rendition's minimum over the same recent window it uses for `jitter`, from one clock shared by every rendition, and advertises the largest difference observed.
+A container importer does not measure `delay`.
+The rounding, `0`, and never-lower rules of `jitter` apply unchanged.
+
+A consumer SHOULD hold at least the largest `delay` plus `jitter` among the renditions it plays together.
+A consumer MUST NOT subtract one rendition's `delay` from another's: each is a maximum over the life of the stream, so two values need not share an origin.
+
+For example, a video encoder that flushes 200 milliseconds after the audio encoder for the same media time advertises a video `delay` of 200 and no audio `delay`.
+
 # Container {#container}
 Audio, video, and text tracks use a container to encapsulate the media payload.
 A rendition declares its container via the `container` field of its catalog entry ({{common}}):
@@ -1085,6 +1099,7 @@ A publisher MAY estimate an unknown final duration from the frame cadence, but M
 - An audio endpoint bounds only the terminal packets that follow it in its own group.
 - Replaced the archive timeline `wall` field with a root `clock` section (`wall` plus `timescale`): one fixed broadcast mapping every track and the archive index convert into, independent of any archive. Zero timescales and walls past the JSON-safe integer range are refused.
 - Added optional `bitrate` and `jitter` fields to `json` and `binary` track entries.
+- Added the optional `delay` rendition field: how far a rendition's minimum flush lateness trails the broadcast's earliest rendition, never lowered once advertised and never subtracted across renditions.
 - Recommended namespaced keys for application root sections.
 
 # Acknowledgments

js/hang/src/catalog/audio.ts

@@ -59,6 +59,17 @@ export const AudioConfigSchema = z.object({
 			z.transform((value) => (value === 0 ? undefined : value)),
 		),
 	),
+
+	// How far this rendition's frames reach the transport behind the broadcast's earliest
+	// rendition, in whole milliseconds rounded up. A player holds `delay + jitter` for it and never
+	// subtracts one rendition's `delay` from another's. Absent on the earliest rendition. It only
+	// ever grows over the life of a stream.
+	delay: z.optional(
+		z.pipe(
+			u53Schema,
+			z.transform((value) => (value === 0 ? undefined : value)),
+		),
+	),
 });
 
 /** Schema for the catalog audio section: a map of track name to rendition config. */

js/hang/src/catalog/root.test.ts

@@ -1,6 +1,7 @@
 import { expect, test } from "bun:test";
 import * as z from "@zod/mini";
 import { ARCHIVE_VERSION } from "./archive.ts";
+import { u53 } from "./integers.ts";
 import type { RelativeBroadcast } from "./path.ts";
 import { RootSchema } from "./root.ts";
 
@@ -95,6 +96,29 @@ test("legacy zero jitter is absent for audio and video", () => {
 	expect(JSON.stringify(parsed)).not.toContain('"jitter"');
 });
 
+test("delay parses beside jitter and zero is absent", () => {
+	const parsed = RootSchema.parse({
+		audio: {
+			renditions: {
+				audio: {
+					codec: "opus",
+					container: { kind: "legacy" },
+					sampleRate: 48000,
+					numberOfChannels: 2,
+					delay: 0,
+				},
+			},
+		},
+		video: {
+			renditions: { video: { codec: "avc1.64001f", container: { kind: "legacy" }, jitter: 34, delay: 200 } },
+		},
+		text: { renditions: { captions: { format: "vtt", container: { kind: "legacy" }, delay: 120 } } },
+	});
+	expect(parsed.audio?.renditions.audio?.delay).toBeUndefined();
+	expect(parsed.video?.renditions.video?.delay).toBe(u53(200));
+	expect(parsed.text?.renditions.captions?.delay).toBe(u53(120));
+});
+
 test("clock round-trips at the root", () => {
 	const parsed = RootSchema.parse({
 		clock: { wall: 1_751_846_400_000_000, timescale: 1_000_000 },

js/hang/src/catalog/text.ts

@@ -58,6 +58,17 @@ export const TextConfigSchema = z.object({
 			z.transform((value) => (value === 0 ? undefined : value)),
 		),
 	),
+
+	// How far this rendition's frames reach the transport behind the broadcast's earliest
+	// rendition, in whole milliseconds rounded up. A player holds `delay + jitter` for it and never
+	// subtracts one rendition's `delay` from another's. Absent on the earliest rendition. It only
+	// ever grows over the life of a stream.
+	delay: z.optional(
+		z.pipe(
+			u53Schema,
+			z.transform((value) => (value === 0 ? undefined : value)),
+		),
+	),
 });
 
 /** Schema for the catalog text section: a map of track name to rendition config. */

js/hang/src/catalog/video.ts

@@ -73,6 +73,17 @@ export const VideoConfigSchema = z.object({
 			z.transform((value) => (value === 0 ? undefined : value)),
 		),
 	),
+
+	// How far this rendition's frames reach the transport behind the broadcast's earliest
+	// rendition, in whole milliseconds rounded up. A player holds `delay + jitter` for it and never
+	// subtracts one rendition's `delay` from another's. Absent on the earliest rendition. It only
+	// ever grows over the life of a stream.
+	delay: z.optional(
+		z.pipe(
+			u53Schema,
+			z.transform((value) => (value === 0 ? undefined : value)),
+		),
+	),
 });
 
 /**

js/msf/src/catalog.test.ts

@@ -185,6 +185,30 @@ test("preserves SAP fields through decode and encode", () => {
 	expect(wireTracks[0].jitter).toBe(15);
 });
 
+test("preserves delay through decode and encode", () => {
+	const catalog = decode(
+		encodeJson({
+			version: "draft-01",
+			tracks: [
+				{
+					name: "video0",
+					packaging: "loc",
+					isLive: true,
+					role: "video",
+					codec: "avc1.640028",
+					delay: 200,
+				},
+			],
+		}),
+	);
+
+	expect(catalog.tracks[0].delay).toBe(200);
+
+	const wire = decodeJson(encode(catalog));
+	const wireTracks = wire.tracks as { delay?: number }[];
+	expect(wireTracks[0].delay).toBe(200);
+});
+
 test.each([
 	["omitted", undefined],
 	["false", false],

js/msf/src/catalog.ts

@@ -48,6 +48,10 @@ const trackShape = {
 	// The player's buffer must be at least this large to avoid underruns.
 	// Mirrors the `jitter` field in the hang catalog.
 	jitter: z.optional(z.number()),
+
+	// Non-standard: how far this rendition trails the broadcast's earliest, in milliseconds.
+	// Mirrors hang `delay`. A player holds `delay + jitter` and does not subtract across renditions.
+	delay: z.optional(z.number()),
 };
 
 /** Zod schema describing a single track entry in an MSF catalog. */

js/publish/src/catalog.test.ts

@@ -117,10 +117,37 @@ test("a reconnecting subscriber is seeded with the full current catalog", async
 	effect.close();
 });
 
-test("catalog producer refuses zero jitter before retaining an edit", () => {
-	const catalog = new CatalogProducer();
+for (const field of ["jitter", "delay"] as const) {
+	test(`catalog producer refuses zero ${field} before retaining an edit`, () => {
+		const catalog = new CatalogProducer();
+		for (const section of ["audio", "video", "text"] as const) {
+			expect(() =>
+				catalog.mutate((value) => {
+					Object.assign(value, {
+						[section]: {
+							renditions: {
+								media: {
+									codec: "opus",
+									container: { kind: "legacy" },
+									sampleRate: 48000,
+									numberOfChannels: 2,
+									[field]: 0,
+								},
+							},
+						},
+					});
+				}),
+			).toThrow(`omit ${field}`);
+		}
+		catalog.mutate((value) => {
+			expect(value.audio).toBeUndefined();
+			expect(value.video).toBeUndefined();
+		});
+	});
+
 	for (const section of ["audio", "video", "text"] as const) {
-		expect(() =>
+		test(`catalog refuses ${section} ${field} decreases without retaining them`, () => {
+			const catalog = new CatalogProducer();
 			catalog.mutate((value) => {
 				Object.assign(value, {
 					[section]: {
@@ -130,70 +157,45 @@ test("catalog producer refuses zero jitter before retaining an edit", () => {
 								container: { kind: "legacy" },
 								sampleRate: 48000,
 								numberOfChannels: 2,
-								jitter: 0,
+								[field]: 100,
 							},
 						},
 					},
 				});
-			}),
-		).toThrow("omit jitter");
-	}
-	catalog.mutate((value) => {
-		expect(value.audio).toBeUndefined();
-		expect(value.video).toBeUndefined();
-	});
-});
-
-for (const section of ["audio", "video", "text"] as const) {
-	test(`catalog refuses ${section} jitter decreases without retaining them`, () => {
-		const catalog = new CatalogProducer();
-		catalog.mutate((value) => {
-			Object.assign(value, {
-				[section]: {
-					renditions: {
-						media: {
-							codec: "opus",
-							container: { kind: "legacy" },
-							sampleRate: 48000,
-							numberOfChannels: 2,
-							jitter: 100,
-						},
-					},
-				},
 			});
-		});
 
-		// The section is optional on the loose root type, so re-read it through a guard.
-		const retained = (value: Catalog.Root) => {
-			const sectionValue = value[section];
-			if (!sectionValue) throw new Error(`expected a retained ${section} section`);
-			return sectionValue;
-		};
-		for (const jitter of [Catalog.u53(50), undefined]) {
-			expect(() =>
+			// The section is optional on the loose root type, so re-read it through a guard.
+			const retained = (value: Catalog.Root) => {
+				const sectionValue = value[section];
+				if (!sectionValue) throw new Error(`expected a retained ${section} section`);
+				return sectionValue;
+			};
+			for (const estimate of [Catalog.u53(50), undefined]) {
+				expect(() =>
+					catalog.mutate((value) => {
+						retained(value).renditions.media[field] = estimate;
+					}),
+				).toThrow(`${field} cannot decrease`);
 				catalog.mutate((value) => {
-					retained(value).renditions.media.jitter = jitter;
-				}),
-			).toThrow("jitter cannot decrease");
+					expect(retained(value).renditions.media[field]).toBe(Catalog.u53(100));
+				});
+			}
 			catalog.mutate((value) => {
-				expect(retained(value).renditions.media.jitter).toBe(Catalog.u53(100));
+				delete retained(value).renditions.media;
 			});
-		}
-		catalog.mutate((value) => {
-			delete retained(value).renditions.media;
-		});
-		catalog.mutate((value) => {
-			Object.assign(retained(value).renditions, {
-				media: {
-					codec: "opus",
-					container: { kind: "legacy" },
-					sampleRate: 48000,
-					numberOfChannels: 2,
-					jitter: 50,
-				},
+			catalog.mutate((value) => {
+				Object.assign(retained(value).renditions, {
+					media: {
+						codec: "opus",
+						container: { kind: "legacy" },
+						sampleRate: 48000,
+						numberOfChannels: 2,
+						[field]: 50,
+					},
+				});
 			});
 		});
-	});
+	}
 }
 
 for (const section of ["json", "binary"] as const) {