Skip to content

feat(client): add AVC+AAC publish support (send_video / send_audio / MultiPublisher) - #3

Open
urlynn wants to merge 10 commits into
torresjeff:mainfrom
urlynn:feat/avc-aac-publish
Open

feat(client): add AVC+AAC publish support (send_video / send_audio / MultiPublisher)#3
urlynn wants to merge 10 commits into
torresjeff:mainfrom
urlynn:feat/avc-aac-publish

Conversation

@urlynn

@urlynn urlynn commented Aug 5, 2026

Copy link
Copy Markdown

Summary

Adds high-level publish APIs for AVC (H.264) and AAC, enabling RTMP push without manual FLV tag construction. Also adds a MultiPublisher for fanning out one stream to multiple endpoints with per-platform isolation.

Background

The existing RtmpPublisher (from #2) exposes send_audio_raw(Bytes) for raw FLV audio tag bodies, but has no video publish path and no high-level audio API that derives the FLV format byte from AudioSpecificConfig. This PR fills both gaps, as pure transmux (no encode/decode).

Changes

Media layer (src/media/)

  • H264Data::to_flv_tag_body() — pure builder, returns FLV video tag body
  • AacData::to_flv_tag_body(format_byte) — pure builder, returns FLV audio tag body
  • AudioSpecificConfig::flv_format_byte() — derives the first byte from ASC

Connector (src/client/connector.rs)

  • send_video_data() — symmetric with existing send_audio_data()
  • Fix: send_message now carries cmd.stream_id only for Command/CommandAmf3; protocol-control messages stay at stream 0

Publisher (src/client/publisher.rs)

  • send_video(&H264Data, ts) — high-level video publish
  • send_audio(&AacData, ts) — high-level audio publish (derives + caches format byte)
  • send_audio_raw(Bytes, ts) — renamed from send_audio (low-level, no derivation)
  • connect() resets cached state; proper error variants (ConnectionClosed, MediaError::MissingSequenceHeader)

MultiPublisher (src/client/multi_publisher.rs)

  • Per-platform tokio task + bounded mpsc buffer (256 capacity); send_video / send_audio / send_audio_raw fan out to all endpoints
  • Concurrent connect with a per-target connect_timeout — one slow or unreachable endpoint never delays the others
  • Event-driven readiness: PlatformConnected / PlatformConnectFailed / PlatformDisconnected
  • set_sequence_headers() — caches AVC/AAC sequence headers and replays them to each endpoint before its first raw frame, on both connect and reconnect
  • reconnect(id) for manual recovery (caller controls backoff)
  • send_* are synchronous (enqueue, not delivery) — result reflects buffer acceptance, not server acknowledgment

Client connection fixes (required for reliable push)

  • Acknowledgement is now sent at half the advertised window, so a server that enforces its window never stalls the push
  • drain_incoming(): the push path reads the socket (non-blocking) and handles control messages (SetChunkSize, WindowAckSize, SetPeerBandwidth, PingRequestPingResponse), so a remote server's control traffic never backpressures the push
  • protocol/chunk: force chunk format 0 (absolute timestamp) when a message's timestamp goes backward, instead of a wrapped 32-bit delta that peers misparse as an extended-timestamp marker (DTS discontinuity)

Usage

let (mut publisher, mut events) = RtmpPublisher::new(ClientConfig::new(url));
publisher.connect().await?;

// Send sequence headers first
publisher.send_video(&h264_seq_header, 0).await?;
publisher.send_audio(&aac_seq_header, 0).await?;

// Then frames
publisher.send_video(&keyframe, 33).await?;
publisher.send_audio(&audio_frame, 23).await?;

Testing

The publish path was exercised end-to-end in an RTMP→RTMP relay
(PS5 → rtmp-rs server → MultiPublisher → platform ingest URLs), covering
both single- and multi-platform configurations.

Single platform. Pushing to one live ingest yields a clean stream with no
stalls over sustained runs.

Multi-platform. The same stream is pushed to two live platforms
simultaneously (e.g. Twitch plus a second live ingest); frame rate and
latency on one are unaffected by the other.

Robustness — one broken platform never blocks the others:

  • Unreachable/fake URL: with two healthy targets plus one unreachable URL,
    the bad target fails within connect_timeout (default 10s) and emits
    PlatformConnectFailed; the two healthy targets start immediately and push
    at full rate throughout — the unusable URL never blocks the stream.
  • Malformed URL: rejected at config parse (Configuration error: Invalid RTMP URL) with no retry loop and no impact on the other targets.
  • Mid-stream disconnect: when a target's server closes the connection,
    PlatformDisconnected is emitted and the remaining targets keep pushing.
    MultiPublisher::reconnect restores a target; the cached sequence headers
    are replayed before its first raw frame (set_sequence_headers), so the
    server sees a clean stream.
  • Slow target: when an endpoint's bounded buffer fills, frames are dropped
    for that endpoint only (Err(BufferFull)); every other target still
    receives every frame.

These scenarios exercise the isolation guarantees of MultiPublisher:
parallel per-target connects with individual timeouts, per-endpoint error
events, and an event-driven readiness/reconnect protocol.

Automated. cargo test — 379 lib unit tests + doc-tests pass (media
tag-builders, MultiPublisher dispatch semantics, and the new publish APIs).

Notes

  • MultiPublisher::set_sequence_headers() replays the cached AVC/AAC sequence headers to each endpoint before its first raw frame, on both connect and reconnect; call it whenever a new sequence header appears (e.g. stream restart)
  • MultiPublisher::send_* return Vec<Result<()>> (one per endpoint); Err(BufferFull) means enqueue failed, Err(ConnectionClosed) means endpoint task exited — caller decides drop/retry/reconnect

Follow-ups (separate PRs)

  • ChunkEncoder state reset on stream end, extended-timestamp edge cases, server-side acknowledgement tuning

urlynn added 10 commits August 5, 2026 09:59
Builds the FLV video tag body (header byte + AVCPacketType + SI24
composition time + payload) from parsed AVC data so a publisher can
transmux AVC without re-encoding.
Builds the FLV audio tag body (0xAF header + AAC packet type + payload)
from parsed AAC data for transmux-only forwarding.
Add send_video_data (mirrors send_audio_data, uses CSID_VIDEO/MSG_VIDEO)
for the publish path.

Also fix send_message, which hardcoded stream_id=0 for every chunk and
ignored the command's own stream id. publish/play commands must carry the
createStream result id, otherwise the rtmp-rs server marks no publishing
stream and drops the first media frame. Fixing this makes client publish
and play interoperate with rtmp-rs's own server.
High-level counterpart to send_audio: takes parsed H264Data, builds the
FLV tag body internally, and sends it on the published stream.
Pushes one stream to several RTMP endpoints at once (e.g. multiple
platforms). Each endpoint uses its own RtmpPublisher; connect/send/
disconnect fan out and return per-endpoint Results so a failure on one
platform is isolated from the others.
- publisher: add send_audio(&AacData) + send_audio_raw, cache aac_format_byte
- publisher: reset state on connect, use proper error variants
- aac: add flv_format_byte() derivation, to_flv_tag_body(format_byte)
- h264: preallocate capacity, explicit SI24 shifts
- error: add BufferFull variant
- multi_publisher: concurrent connect with per-target timeout, event-driven
  readiness (PlatformConnected / PlatformConnectFailed), shared senders
- aac: test flv_format_byte for 44100/22050/5512/96000 Hz
- aac: test to_flv_tag_body for SequenceHeader and Raw frame
- h264: test to_flv_tag_body for SequenceHeader/keyframe/P-frame/EndOfSequence
The client never sent RTMP Acknowledgement (BytesRead) messages, so a
peer that enforces its window ack size paused sending every window
(~8s at typical bitrates): periodic stutter, latency reset to the window
size on every resume. The puller was the most affected path.

- track bytes_received in read_message
- save the server's advertised WindowAckSize instead of dropping it
- send Acknowledgement once half the window is received (half-window,
  earlier than the peer's full window so the sender never stalls)
- u32 wrap-safe sequence arithmetic
RtmpPublisher only writes after publish; the server's control traffic
(SetChunkSize, WindowAckSize, SetPeerBandwidth, PingRequest) piled up
unconsumed, and on remote targets the server's send buffer backpressures
the push into periodic stalls (same family as the missing-ACK issue).

- drain_incoming(): non-blocking read + decode; feeds control messages to
  handle_incoming, counts bytes, sends ACK (half window)
- handle_incoming(): apply SetChunkSize/WindowAckSize, echo WindowAckSize
  on SetPeerBandwidth, answer PingRequest with PingResponse
- call drain_incoming before each send_audio_data
- the non-blocking read polls the socket once with a no-op waker instead
  of timeout(Duration::ZERO, read): the timer version cost ~1ms per call
  (timer tick), which at 73 frames/s adds ~95ms/s of blocking and makes
  the push fall behind real time
select_format picked a delta format (1/2/3) even when the new timestamp
was smaller than the previous one. The wrapped 32-bit delta (~4G) was
then truncated by write_u24; if the low 24 bits were 0xFFFFFF the peer
misread it as an extended-timestamp marker and reconstructed a ~2^24 DTS,
which breaks players and gets live streams rejected as a DTS
discontinuity on a second publish.

Detect regression and force format 0 (absolute timestamp) so the peer
resynchronizes. Add roundtrip test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant