Skip to content

TS import ends the ingest on one malformed packet (PES header, NAL header) instead of dropping it #4581

Description

@t0ms

What happens

moq import ts, and the SRT gateway through it, end the whole ingest on the first packet they cannot
parse. One damaged packet at 20 s into an otherwise clean 60 s off-air capture (H.264, MP2, AC-3 and
teletext, 9.95 Mb/s, PCR-paced through tsp -P regulate into moq import ts):

Damage at 20 s dev @ 9a80e87 main @ 6a01640
One video PES header: flag and timestamp bytes zeroed, transport_error_indicator clear exits 1: InvalidInput: Unexpected marker bits same
One H.264 NAL header: forbidden_zero_bit set exits 1: h264: forbidden zero bit is not zero same
The same PES damage with transport_error_indicator set carries on carries on
About two dozen lost 7-packet datagrams over 60 s (continuity gaps) carries on carries on

The importer is unchanged from 9a80e875 to the current dev tip. The SRT gateway ends the
connection on the same errors (SRT ingest ended with error … mux: …). A caller that redials turns
that into an outage of about a second, and a subscriber that does not --linger loses its output
for good.

Why it matters

transport_error_indicator covers the errors a demodulator detected. It does not cover a bit error
that got past FEC, a multiplexer bug, or a sender quirk. On a 24/7 contribution feed, one such packet
in days of input is enough to end the ingest.

One sender quirk showed how easy it is to hit. srt-live-transmit 1.5.6, fed from a pipe, pads every
short stdin read with zeros out to its full chunk. Its default chunk is 1456 bytes, which is not a
multiple of 188. The result is packets with a valid header over zeros or over the next packet's bytes.
In 4-minute runs on a loaded host, the gateway ended 10 to 15 sessions on Unexpected marker bits,
Expected stuffing byte 0xFF, Expected packet start code prefix, CRC32 mismatch and
h264: forbidden zero bit is not zero. That is a defect in the sender, not here, and a libsrt
receiver gets the same zero-filled payloads. But it is exactly the kind of damage a broadcast
demultiplexer is expected to survive.

The watch side already follows this principle: quest/m1/watch-decoder-recovery.md rebuilds the
decoder on one malformed packet instead of ending playback. This asks for the ingest equivalent.

Where

In rs/moq-mux/src/container/ts/import.rs, decode runs
while let Some(packet) = self.reader.read_ts_packet()? { self.handle_packet(packet)?; }. So a
PES-header or adaptation-field error from mpeg2ts, and a codec error raised from handle_packet,
both end the import. The transport_error_indicator branch in the same loop already drops a flagged
packet, and it is the model for the rest.

Suggestion

Treat a parse error confined to one packet, PES or access unit as damage to that unit:

  • drop the unit;
  • count it in the importer's stats, alongside the audio resync counters;
  • where the codec needs one, have the track wait for its next keyframe;
  • carry on.

Keep a fatal error for input that cannot be TS at all, such as sync that never comes back.

Reproduce

Zero one PES header's flag and timestamp bytes on the video PID after 20 s (TEI left clear):

import sys
data = bytearray(open(sys.argv[1], "rb").read())
pid, at = int(sys.argv[3], 0), int(float(sys.argv[2]) * len(data) / 188 / 72.3)  # 72.3 s clip
for i in range(at, len(data) // 188):
    p = i * 188
    if ((data[p + 1] & 0x1F) << 8 | data[p + 2]) == pid and data[p + 1] & 0x40:
        off = 4 if (data[p + 3] >> 4) & 3 == 1 else 5 + data[p + 4]
        if data[p + off : p + off + 3] == b"\0\0\1":
            data[p + off + 6 : p + off + 19] = bytes(13)
            break
sys.stdout.buffer.write(data)
python3 corrupt.py clip.ts 20 0x6f \
  | tsp -I file - -P regulate --pcr-synchronous -O file - \
  | moq --connect <relay> --broadcast damaged.hang import ts
# exits 1 at 20 s: Error: InvalidInput: Unexpected marker bits

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questBeing tracked/planned in a quest. See `quest/`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions