feat(moq-mux): import ts --passthrough carries the multiplex whole

doc/bin/cli.md

@@ -76,7 +76,10 @@ each as its own broadcast (`event.hang` becomes `event/1.hang`,
 `event/2.hang`). A damaged packet is dropped on its own PID, so video freezes
 until its next keyframe while the other tracks carry on. The importer also
 logs feed health (stalled streams, TR 101 290 errors) without changing what it
-publishes. FLV covers H.264 + AAC.
+publishes. `--passthrough` publishes the multiplex whole instead, every packet
+in order on one [`m2ts`](/concept/hang#transport-streams) track with every
+program aboard, for what demultiplexing cannot carry, such as a scrambled
+service; `export ts` does not read it. FLV covers H.264 + AAC.
 
 ## Export
 

doc/concept/hang.md

@@ -126,6 +126,27 @@ Name the section with a namespaced key such as
 `com.example.mavlink`. A generic consumer only finds tracks in `json` and
 `binary`.
 
+## Transport streams
+
+A broadcast can carry an MPEG transport stream whole instead of demultiplexed
+into renditions, for what demultiplexing cannot carry, such as a scrambled
+service. The root `m2ts` section names the one track holding it:
+
+```json
+{
+  "m2ts": { "track": "0.m2ts", "randomAccess": true, "muxRate": 2499999 }
+}
+```
+
+Each frame is a run of whole 188-byte packets in input order, timestamped on
+the multiplex's own PCR clock, which never steps back: a source whose clock
+steps back is a new broadcast. `randomAccess` says whether every group begins
+where a receiver can start decoding, and `muxRate` is the constant multiplex
+rate in bits per second, absent for a variable-rate source. A player has
+nothing to decode here. `moq import ts --passthrough` writes this section (see
+[moq-cli](/bin/cli)); the encoding is in
+[draft-lcurley-moq-mpegts](/draft/moq-mpegts).
+
 ## Container
 
 The `container.kind` on each rendition says how frames are framed:

drafts/draft-lcurley-moq-mpegts.md

@@ -50,6 +50,7 @@ informative:
 
 This document defines the `mpegts` catalog section, which records what demultiplexing an MPEG-2 Transport Stream {{mpeg2}} into a MoQ broadcast would otherwise lose: each track's PID and PMT descriptors, the program identity, the service information tables, and a carriage record for every elementary stream the publisher did not decode.
 It is a root member of either the hang catalog {{hang}} or the MSF catalog {{msf}}, so a subscriber that ignores it still plays the broadcast and one that reads it can rebuild the source multiplex.
+It also defines the `m2ts` section of the hang catalog, which names a track carrying the transport stream whole, and maps that track onto MSFTS {{msfts}}.
 
 --- note_Note_to_Readers
 
@@ -68,16 +69,17 @@ A **verbatim** track carries an elementary stream the publisher did not decode,
 
 # Introduction {#introduction}
 A transport stream reaches MoQ in one of two shapes.
-A publisher can leave the multiplex intact and carry the packet stream as opaque payload, which is what {{msfts}} specifies.
-Or it can demultiplex, which is what this document addresses: each elementary stream becomes its own MoQ track with a codec description, so a relay can drop and prioritize per track and a subscriber can decode without a transport-stream parser ({{comparison}}).
+A publisher can leave the multiplex intact and carry the packet stream as opaque payload, which is what {{msfts}} specifies for MSF and the `m2ts` section ({{m2ts}}) describes in a hang catalog.
+Or it can demultiplex, which is what most of this document addresses: each elementary stream becomes its own MoQ track with a codec description, so a relay can drop and prioritize per track and a subscriber can decode without a transport-stream parser ({{comparison}}).
 
 Demultiplexing loses everything that is not media: the PID layout, the PMT descriptors, the program identity, the service information, and any stream the publisher has no decoder for.
 The `mpegts` section records that.
 It defines no packaging and no container: decoded media keeps the container its catalog entry declares, and verbatim tracks use the framing in {{verbatim}}.
 
 
-# Comparison with m2ts Packaging {#comparison}
+# Comparison with mpeg2ts Packaging {#comparison}
 {{msfts}} carries the transport stream as it arrives: each object is a run of whole 188- or 192-byte packets, and the receiver demultiplexes as it always has.
+The `m2ts` section ({{m2ts}}) carries the same shape in a hang catalog: one track per multiplex, with a timestamp before each object's packets.
 This document takes the stream apart at the publisher and discards the packetization layer, because MoQ already provides what it was there for:
 
 - **Packet framing** interleaves elementary streams onto one serial pipe. MoQ delivers each track separately.
@@ -88,17 +90,17 @@ This document takes the stream apart at the publisher and discards the packetiza
 
 What remains is media in a codec-neutral catalog, plus the signaling a demultiplexer cannot reconstruct: that is the `mpegts` section.
 
-| | m2ts packaging | mpegts section |
+| | mpeg2ts packaging | mpegts section |
 |:--------|:--------|:--------|
-| MoQ track | one per program | one per elementary stream |
+| MoQ track | one per program or multiplex | one per elementary stream |
 | Object payload | whole source packets | one media frame, or one verbatim PES payload or section |
-| Packaging | `m2ts` | none defined; media keeps its own |
-| Catalog | `m2ts*` members on each track | one `mpegts` root member |
+| Packaging | `mpeg2ts` | none defined; media keeps its own |
+| Catalog | `mpeg2ts*` members on each track | one `mpegts` root member |
 | Relay | caches and drops whole objects | drops and prioritizes per elementary stream |
 
-The two overlap only on program identity: `m2tsProgramNumber` and `m2tsPmtPid` against {{field-program}}, and `m2tsPsiInterval` against a per-PID `interval` ({{field-si}}).
+The two overlap on program identity, `mpeg2tsProgramNumber` against {{field-program}}, and on the multiplex rate, `mpeg2tsMuxRate` against {{field-mux-rate}}.
 They do not collide, since those members sit on a track and this section sits at the root.
-A track is one shape or the other.
+A track is one shape or the other, and a broadcast may carry either.
 
 
 # The mpegts Section {#section}
@@ -245,11 +247,89 @@ A verbatim track has no codec, dimensions, or decoder configuration, so a publis
 Nothing there declares a container, so its frames use hang's `legacy` container ({{hang}}): a varint microsecond timestamp followed by the payload.
 
 
+# The m2ts Section {#m2ts}
+A publisher that carries a transport stream whole, without demultiplexing it, names its track in an `m2ts` root member of the hang catalog ({{carriage-hang}}):
+
+~~~
+type M2ts = {
+  "track": TrackName,
+  "randomAccess": boolean,
+  "muxRate": number | undefined,
+}
+~~~
+
+`track` names the one track carrying the multiplex, in the same broadcast as the catalog.
+A second multiplex is a second broadcast.
+
+`randomAccess` says that every group starts at a random access point ({{m2ts-groups}}).
+A publisher MUST write it, and a consumer MUST read an absent `randomAccess` as false.
+
+`muxRate` follows {{field-mux-rate}}, measured over every packet of the track.
+
+The section records no program, PID, descriptor, or table: they all reach the consumer inside the packets, as the publisher received them.
+A track named here MUST NOT also be described as a rendition or in an `mpegts` section.
+
+## Objects {#m2ts-objects}
+Each frame is a run of whole 188-byte packets in the order the publisher received them, in hang's `legacy` container ({{hang}}) with the timestamp of {{m2ts-timestamps}}.
+A publisher MUST refuse a source of 192- or 204-byte packets rather than reframe it, since removing the extra bytes would change the stream.
+
+The multiplex is paced on one clock PID: the `PCR_PID` in the PMT of the first program the PAT lists, unless the publisher is configured with another.
+A frame runs from one packet carrying a PCR on the clock PID up to the next, and is also cut wherever a group starts.
+
+A publisher publishes nothing until it has read the PAT, the PMT of the clock's program, and a PCR, and its first group then waits for the next group start ({{m2ts-groups}}).
+It drops the bytes before that group, and any bytes off the 188-byte packet grid, and they are outside the track's identity with its input.
+From the first group on, the frames of consecutive groups concatenate to the input: a publisher MUST NOT drop, reorder, or modify a packet after that point.
+
+## Groups {#m2ts-groups}
+A group starts at:
+
+- a packet with `random_access_indicator` set on the video PID of the clock's program (the first PID its PMT declares with a video `stream_type`), or on the clock PID when that program has no video;
+- a PCR on the clock PID with `discontinuity_indicator` set that steps forward ({{m2ts-timestamps}}), since a group MUST NOT contain a PCR discontinuity;
+- otherwise, the first PCR at least one second of PCR time after the group's start;
+- the frame that would take a group past a size bound the publisher chooses below what a relay caches of one group; frames are never split to meet it.
+
+`randomAccess` is true only while the PAT lists one program, its PMT declares at most one video stream, and the clock PID is that PMT's `PCR_PID`, and only if every group so far has started at a `random_access_indicator`.
+The first group that starts any other way, or that carries packets while that layout does not hold, republishes the section with `randomAccess` false for the rest of the track.
+With more programs or video streams the random access points can stagger, so a group start is not one for all of them.
+
+Both indicators sit in the adaptation field, which TS-level scrambling leaves in the clear, so a scrambled multiplex groups like its clear twin.
+A publisher parses the PAT and the PMT and nothing else past the adaptation field.
+
+## Timestamps {#m2ts-timestamps}
+A frame's timestamp is the PCR time of its first byte, in microseconds rounded down.
+A PCR gives the time of the byte holding the last bit of its `program_clock_reference_base` ({{mpeg2}}), ten bytes into its packet.
+A byte between two PCRs is placed at the rate between them; a byte past the last PCR of a timebase takes the rate of the interval before it, and one before the first PCR of a timebase takes the rate of the interval after it.
+
+PCRs are unwrapped into a 64-bit count of 27 MHz ticks, and timestamps never decrease.
+A step forward of at most half the 2^33 × 300 tick period adds to the count, across the wrap included, so a wrap is not a discontinuity.
+A PCR with `discontinuity_indicator` set whose step is forward by that measure adds its step too, after the bytes before it are timed at the rate before it, and the group it starts MUST follow a skipped group sequence, hang's break marker.
+A PCR that steps backwards MUST end the track with an error, flagged or not: a flagged one is a restart, which is new content, so a publisher that carries on publishes it as a new broadcast.
+
+The timeline depends only on the bytes since the publisher's first PCR, so two publishers that start at the same packet of one input publish identical frames.
+
+## Mapping onto MSFTS {#m2ts-msfts}
+Stripping each frame's timestamp leaves an {{msfts}} object of whole source packets, and the groups carry over unchanged.
+An MSF catalog does not list the `m2ts` track itself, whose frames hold more than packets; a converter that republishes the stripped objects describes its track as follows:
+
+| MSFTS track field | Value |
+|:--------|:--------|
+| `packaging` | `mpeg2ts` |
+| `mpeg2tsMode` | `unmodified-program` when the PAT lists one program, otherwise `unmodified-multiplex` |
+| `mpeg2tsPacketSize` | 188 |
+| `mpeg2tsRandomAccess` | `randomAccess` |
+| `mpeg2tsMuxRate` | `muxRate`, in `unmodified-program` mode only |
+| `mpeg2tsProgramNumber`, `mpeg2tsPcrPid`, `mpeg2tsScte35Pid` | read from the PAT and the PMT in the packets, where {{msfts}} permits them in the mode |
+| `mpeg2tsTimestampMode`, `mpeg2tsEsPid`, `mpeg2tsSiPids` | absent |
+
+{{msfts}} asks for the PAT and the PMT in `initDataList` when random access is true; the `m2ts` section does not carry them, so a converter takes them from the packets.
+
+
 # Carriage {#carriage}
 The section is the same JSON in either catalog format.
 
 ## hang {#carriage-hang}
 A root member named `mpegts`, alongside `video` and `audio` ({{hang}}).
+The `m2ts` section ({{m2ts}}) is a root member of the hang catalog only.
 hang carries a decoder config's raw bytes as hex while every binary field here is base64; the alphabets overlap, so the encoding cannot be detected and is stated per field above.
 
 ## MSF {#carriage-msf}
@@ -258,7 +338,7 @@ A publisher MUST NOT name a section after a member MSF itself defines.
 
 The keys of the section's own `tracks` member are MSF track names; a decoded track also has an MSF track object, a verbatim track does not.
 
-This document defines no packaging value; {{msfts}} registers `m2ts` for the passthrough shape ({{comparison}}).
+This document defines no packaging value; {{msfts}} defines `mpeg2ts` for the passthrough shape ({{comparison}}), and {{m2ts-msfts}} maps an `m2ts` track onto it.
 
 
 # Rebuilding a Transport Stream {#rebuild}
@@ -279,6 +359,7 @@ A consumer MUST NOT derive one that contradicts a recorded descriptor.
 
 # Security Considerations
 Every binary field here is re-emitted without inspection, by a publisher that never parsed it and a relay that never looked, so a consumer MUST treat all of it as untrusted input.
+The same holds for every packet of an `m2ts` track, which the publisher forwards as it received them.
 
 Nothing bounds the number of entries in the section, nor the size of an SI track: a publisher controls both, and an EPG is large by nature.
 A consumer MUST bound what it accepts of each, and MUST reject a catalog it cannot bound rather than truncate it into a stream that silently differs from what was described.
@@ -346,6 +427,7 @@ A broadcast demultiplexed from a DVB transport stream: video and audio described
 - A consumer refuses a catalog with an unrecognized `framing` or an invalid `si` map key.
 - Added `muxRate`, the source's constant multiplex rate.
 - A consumer may read the pre-`table_id` inline `sections` form; writing stays track-only.
+- Added the `m2ts` section, a transport stream carried whole, with its mapping onto MSFTS; the comparison uses MSFTS's `mpeg2ts` names.
 
 
 # Acknowledgments

quest/m1/README.md

@@ -155,7 +155,7 @@ were done or not worth their cost.
 - [Catalog warmup](/quest/m1/catalog-warmup.md) - `warmup` on video and audio renditions, in the catalog and the draft
 - [Audio warmup](/quest/m1/audio-warmup.md) - a viewer joining an Opus rendition mid-stream never hears the unconverged first 80 ms
 - [#3021](/quest/m1/3021-moq-gst-anchor-generated-media-timelines-to-wall-clock.md) - moq-gst picks the broadcast wall epoch; a restarted source is a new epoch, not a forward re-anchor
-- [TS passthrough](/quest/m1/ts-passthrough.md) - `--passthrough` carries the multiplex as whole packets, listed in an `m2ts` catalog section, and writes it back byte-identical (less late drops) on a fixed delay
+- [TS passthrough export](/quest/m1/ts-passthrough-export.md) - `export ts --passthrough` writes the `m2ts` track back byte-identical (less late drops) on a fixed delay
 - [FLV and MKV export delay](/quest/m1/export-delay.md) - FLV and MKV interleave through the shared jitter buffer on a fixed delay, breaking the CLI once
 - [MKV lacing](/quest/m1/mkv-lacing.md) - laced MKV blocks import as one timed frame each, refused without DefaultDuration
 - [Release profile](/quest/m1/release-profile.md) - every release build gets fat LTO, one codegen unit, and stripping from the workspace profile instead of three script exports

quest/m1/ts-passthrough-export.md

@@ -0,0 +1,54 @@
+# [M] TS passthrough export writes the multiplex back byte-identical on a fixed delay
+
+## Goal
+
+`moq export ts --passthrough --delay <dur>` reads the track a broadcast's
+`m2ts` catalog section names and writes the same bytes back out, paced on the
+source's own PCR and released at a fixed delay like an SRT receiver. From the
+first group it releases, the output is byte-identical to the input, less any
+object dropped late, so it is exactly as T-STD-conformant as the source.
+There is no JS player for it.
+
+## Plan
+
+Decided (2026-10-01), from a discussion with t0ms, and split from the TS
+passthrough quest on #5003, whose import half it completes:
+
+- The export reuses the TS export's jitter buffer (`rs/moq-mux/src/jitter.rs`),
+  keyed on each object's PCR time instead of a DTS, which also spreads each
+  object's bytes at the PCR-implied rate. It must pace on the source's PCR,
+  not on arrival: a pacer that re-clocks on arrival moves the PCR-to-PTS
+  offset over a long capture and fails the decoder buffers, even though every
+  byte is intact.
+- An object that misses its deadline is dropped and counted, as the release
+  stage does. The output shows a continuity error there; passthrough never
+  rewrites continuity counters.
+- Source clock drift is handled by the fixed-delay release's clock recovery,
+  shared with the demultiplexed export, not here.
+- A flagged forward jump arrives as a break marker before the group it
+  starts; the release keeps the jump in the output, since the bytes carry it.
+- Passthrough gives 1+1 identity for free: two exporters fed the same objects
+  emit identical TS packets for every object both release, continuity
+  counters included; an object one leg drops late is a gap in that leg only.
+  Aligning the legs in time is left to the `--sync` anchor planned for the
+  demultiplexed export's 2022-7 legs, which passthrough can adopt. That is TS
+  identity, not ST 2022-7 recovery, which also needs matching RTP headers from
+  a coordinated RTP egress.
+- Rust only.
+
+Test: an export of a broadcast capture is byte-identical to the input from the
+first released group, and the strict T-STD check gives the same verdict on
+output and input. Two exporters fed the same objects with different arrival
+skew emit identical bytes; when only one misses a deadline, its output is the
+other's less that object's packets. A dropped object is counted, and the rest
+still go out on time. Rerun the #4613 netem rig (10% loss, 120 s) against it.
+
+Update `doc/bin/cli.md` for the flag, which today says `export ts` does not
+read the passthrough track.
+
+Public API: a new export mode; nothing existing breaks.
+
+## Related
+
+- [TS hitless](/quest/m2/ts-hitless.md) - the demultiplexed lane's 2022-7 legs and the `--sync` anchor
+- [T-STD TS export](/quest/m1/tstd/README.md) - the jitter buffer and clock recovery this release shares