Skip to content

import ts corrupts a multi-program transport stream without an error, and export ts rebuilds only one program #4353

Description

@t0ms

Summary

Feeding a real MPTS to moq import ts does not produce an error on current main. With the programs
on independent clocks, which is how they arrive from separate encoders, the output timeline is
scrambled and most of one program is lost. With the programs on a common clock the video survives,
but audio has multi-second holes and most SCTE-35 sections are lost. The publisher exits 0 in both
cases and logs nothing about it. On either build, the exporter emits one program: a PAT with one
entry and one PMT listing every elementary stream. The carried SDT and EIT still describe the other
services, so the service layer contradicts itself.

The importer already scopes itself to single-program input in a comment (below), so this is not a
request for MPTS support as such. The ask is that an MPTS fails loudly, or is reduced to one program
on purpose, rather than being corrupted without a word.

Input

A 60 s MPTS built with TSDuck from three single-program clips: three programs, each with its own PMT,
PCR PID and elementary streams, plus one PAT and one SDT listing all three, and EIT present/following
for each. Clean at source: 0 continuity errors, and every PCR within 13 ticks (pcrverify).

Program PMT PID PCR PID Elementary streams
1 100 111 H.264 111, MP2 121, AC-3 123, teletext 131, SCTE-35 141–143
2 512 529 H.264 529
3 768 785 H.264 785, MP2 786

It comes in two variants. In the first, the clocks are independent: program 1's PCR starts near
25,631 s and the other two near 3 s. In the second, programs 2 and 3 are re-timed by stream-copy remux
so that all three start within 0.82 s of each other.

What happens

On main (2b689c24), independent clocks

The publisher runs to the end and exits 0. The subscriber receives 15.6 MB, where the common-clock
variant delivers about 86 MB:

Stream Packets at source Packets at egress
Program 1 video (111) 359,960 8,424
Program 1 MP2 audio (121) 8,055 92
Program 2 video (529) 74,441 73,023
Program 3 video (785) 14,398 546

Program 2 arrives whole, on its own timeline. The rest is re-timed across clocks:

  • PCR on PID 111 jumps by more than 5 s thirteen times, and spans 60,970 s for a 60 s input.
  • Program 3's video arrives stamped on program 1's clock (about 25,641 s).
  • Audio and teletext PTS land at values that exist nowhere in the source: 39,780 s, 76,897 s and
    83,968 s, against source clocks near 25,631 s and 3 s.

The only warnings, both from the subscriber:

WARN moq_mux::container::consumer: current group evicted; skipping to next buffered group track="1.avc3" group=1 error=Hang(Moq(Old))
WARN moq_mux::container::consumer: current group evicted; skipping to next buffered group track="7.avc3" group=0 error=Hang(Moq(Old))

On main, common clock (two runs)

The publisher exits 0 and video is intact on all three programs. The rest is not:

  • SCTE-35: 1–2 and 4–7 packets per PID reach the egress, against about 60 at source.
  • Program 1 audio: its MP2 PID carries 5,408 and 6,376 packets, against 9,688 when program 1
    is sent alone through the same build. In the second run the PTS show eight holes of 2.7–4.1 s,
    each about 0.45 s shorter than the last. AC-3 on program 1 shows nine holes, and program 3's MP2 three (0.8–1.8 s).
  • EIT: the carriage rate moves a long way from source: +139 % in one run and −76 % in the other.

No warning is logged while any of this happens; the only warnings come at teardown.

Control: program 1 alone through the same build, extracted with
tsp -P zap 1 --stuffing --eit, is clean. Its SCTE-35 arrives (57 of 60 packets per PID), audio has
no holes, the EIT rate is within 9 % and PCR intervals stay at or below 25 ms. The losses need
several programs in one input.

On 0.12.1 (ffa5b81b)

With independent clocks the publisher aborts about 2 s in, in two runs out of two:

WARN moq_net::model::resume: no route can serve the rest of this group group=0 frame=17 err=transport: frame timestamp is below the live edge
Error: frame timestamp is below the live edge

With a common clock it completes without the losses above. This is a better outcome than main's,
because it fails, but the error does not say that the input has more than one program.

The exporter, on both builds

Whenever enough survives to export, the output looks like this:

PAT: program 1 -> PMT 100                (source: 1 -> 100, 2 -> 512, 3 -> 768)
PMT 100: program 1, PCR PID 111
  ES 111, 121, 123, 131, 141, 142, 143, 529, 785, 786

PMT PIDs 512 and 768 do not appear, and PIDs 529 and 785 carry no PCR. The SDT and EIT are carried
verbatim, so they still list services 2 and 3, which the PAT no longer has.

Why, as far as I can read it

  • One section clock. Private sections and SI snapshots are stamped with last_pts
    (rs/moq-mux/src/container/ts/import.rs). Every program's video writes that one field, and the
    comment on it states the limit:

    /// SPTS scope: one clock for the whole input. Under MPTS every program's video
    /// advances it, so a cue could be stamped with another program's PTS.
    

    On 0.12.1 the alternation between program clocks trips TimestampRewind on the section and SI
    tracks, which is the abort.

  • Re-anchoring and one broadcast clock. Between the two builds, import.rs gained per-stream
    re-anchoring below the live edge (fix(moq-mux): every TS elementary stream re-anchors below the live edge #3997) and publication on the broadcast clock (feat(cli): publish stdin imports on the broadcast clock #4122). My reading,
    which I have not verified by bisecting, is that one or both now treat a jump between program clocks
    as a wrap or restart and re-anchor rather than fail. That would explain the independent-clock
    output, and possibly the audio and SCTE-35 losses on a common clock, where the clocks still differ
    by up to 0.82 s.

  • One program record. record_program_identity keeps the first non-zero entry in the PAT. The
    PAT arm then adds every PMT PID to pmt_pids, so the elementary streams of every program become
    tracks of one broadcast. The exporter builds its PAT and PMT from that single record
    (table: vec![ProgramAssociation { .. }] in export.rs).

Reproducer

  • Fixture recipe:
    make-real-mpts.sh
    (TSDuck 3.44 and ffmpeg). Set P2_OFFSET/P3_OFFSET for the common-clock variant; the values
    are in the script header.
  • Rig:
    t10-moq.sh
    runs relay, export ts to file, and import ts fed once at source rate by
    tsp -P regulate --pcr-synchronous.
  • Grader:
    t10-grade.py
    grades each program for PAT, PMT, SDT, EIT, continuity, PCR and every elementary stream.
make-real-mpts.sh mpts3.ts 60
BIN=<dir with moq and moq-relay> t10-moq.sh mpts3.ts out mpts
t10-grade.py mpts3.ts out/egress.ts

Any two programs whose clocks start far apart should reproduce the independent-clock case. The
second program does not need audio or SI.

What would help, from where I sit

In rough order of cost; the first alone would close the silent part:

  1. Refuse an MPTS explicitly. When the PAT lists more than one program, exit with an error that
    says so, before anything is published. tsp -P zap <program> in front of import ts already
    gives a clean single-program feed. All three programs came through intact that way on 0.12.1,
    and program 1 again on main, so a refusal costs users nothing but a pipeline stage.
  2. Or select one program. A --program <n> option that routes one program's PMT and drops the
    other programs' PIDs would make the reduction deliberate instead of accidental. It would also give
    one broadcast per program, which is what the catalog model supports today.
  3. If MPTS belongs in scope: one section clock per program, keyed by the PMT that lists the PID,
    and one program record per PMT in the catalog, so the exporter can rebuild the PAT and every PMT.
    Only this one preserves the multiplex, and it is plainly the largest.

For whichever you choose, a regression test in the shape of the existing import tests: a synthetic
input with two programs whose video clocks start far apart. It should either fail with an error
naming the second program, or publish two programs that export as two.

Activity

  1. t0ms commented on Sep 28, 2026

    @t0ms
    ContributorAuthor

    Bisected both halves of this, first-parent between ffa5b81b and 2b689c24, with the fixtures and
    grader from the report and a
    bisect step
    whose thresholds were fixed from the two endpoints before starting:

    Property Last good First changed Before After
    Independent clocks refused cf0ef442 2299622e (#3997) publisher exit 1 exit 0
    Common clock clean 2f2ff7d3 c87159d8 (#4122) SCTE-35 168 packets, PID 121 audio 9,416 SCTE-35 6, audio 5,504

    Each parent was graded directly. At 66440a6c the independent-clock input already exits 0 and the
    common-clock input is still clean, so it is two changes, not one. #4122 also takes programme 3's video
    on independent clocks from 12,836 packets to 0.

    So my guess in the report was half right. #3997 turns the refusal into a re-timing. #4122 is the one
    that loses content, and I think it is the shared anchor. Every track of the input maps through one
    Anchor, and a backwards step over MAX_REORDER (500 ms) on any lane re-anchors the whole source.
    The tracks stamped from last_pts step back each time the programmes alternate, and on
    mpts3-cc.ts they are up to 0.82 s apart. That part is from reading the diff, not measured; a
    common-clock fixture with offsets under 500 ms would test it. Either way, the ask in the report is
    unchanged: refusing a second PMT up front would make all of this moot.

  2. t0ms commented on Sep 28, 2026

    @t0ms
    ContributorAuthor

    Tested the anchor guess from my last comment, and it's wrong. I modelled the section clock over the fixture's bytes: on mpts3-cc.ts the SCTE-35 lanes never step back more than 500 ms. They start a section about once a second, so the forward interval hides the offset between programmes. Only EIT does step back (9 times, up to 720 ms).

    Then I built more common-clock fixtures, some with no section lane stepping back past 500 ms, and ran each one on 2f2ff7d3, c87159d8 and 2b689c24:

    Fixture EIT steps > 500 ms P2 / P3 median offset SCTE-35 (141–143) parent / #4122 / main
    b7 0 (largest 160 ms) −40 / +40 ms 170 / 153 / 152
    b3 0 (320 ms) +130 / −120 ms 168 / 71 / 38
    b5 0 (280 ms) +320 / +60 ms 168 / 24 / 14
    b8 1 (520 ms) 0 / −420 ms 170 / 12 / 9
    b4 8 (920 ms) −20 / +760 ms 167 / 5 / 9
    b6 10 (1,760 ms) −1,520 / +200 ms 160 / 3 / 3

    The parent is clean on all of them, so it's still #4122. But b3 and b5 lose most of their SCTE-35 with nothing crossing MAX_REORDER, and nothing is logged. Loss grows with the offset between programmes. Audio doesn't follow it (b4 loses 91 % of programme 1's MPEG audio, b1 none), so I don't know what drops the frames. One run per cell. Details and the model script are in T10. The ask is unchanged.

  3. kixelated commented on Sep 28, 2026

    @kixelated
    Collaborator

    K so a hang broadcast currently assumes a single clock. I'm going to add a flag to select a specific program, or publish all programs as separate broadcasts.

  4. added
    questBeing tracked/planned in a quest. See `quest/`
    on Sep 28, 2026
  5. t0ms commented on Sep 29, 2026

    @t0ms
    ContributorAuthor

    Found it. It's not the section lanes, it's the lane that feeds them. last_pts is advanced through media_unwrap for every video PES, whichever programme it belongs to. With programmes alternating in decode order, that lane steps back more than 500 ms, and since it's always on the current generation, each step re-anchors the whole source forward.

    Two patched builds of 2b689c24: A only logs re-anchors, and B also gives the section clock one lane per video PID (HashMap<Pid, PtsUnwrap> in place of media_unwrap at that call site). One run per cell:

    Fixture A re-anchors A anchor drift A SCTE-35 / P1 audio B re-anchors B SCTE-35 / P1 audio parent
    b7 7 +4.1 s 153 / 9,348 0 171 / 9,532 170 / 9,532
    b3 33 +19.8 s 62 / 7,496 0 169 / 9,540 168 / 9,240
    b5 195 +113.8 s 21 / 6,584 0 171 / 9,532 168 / 9,536
    mpts3-cc 1,044 +688 s 17 / 7,056 0 174 / 9,620 168 / 9,420
    programme 1 alone 0 — 171 / 9,688 0 177 / 9,820 —

    Every re-anchor in A comes from that call site. B removes all of them, and SCTE-35 and audio come back to the parent's level. I haven't traced how it reaches the audio lanes. ts-programs or remove-live would moot this, so it's for the record rather than a fix request. The patch and T10 have the details.

  6. added a commit that references this issue on Sep 29, 2026
    c0c45d7
  7. t0ms commented on Oct 9, 2026

    @t0ms
    ContributorAuthor

    Closing: #4505 answered this as asked. An unselected multiplex is refused, --program <n> imports one programme, and --program all publishes each as its own broadcast; #4580 then scoped the SI to the selected programme. On main at 6f1a9e33, with the fixtures from this report, every programme was carried from both the independent-clock and the common-clock input, in four of four runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questBeing tracked/planned in a quest. See `quest/`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions