feat(client): add AVC+AAC publish support (send_video / send_audio / MultiPublisher) - #3
Open
urlynn wants to merge 10 commits into
Open
feat(client): add AVC+AAC publish support (send_video / send_audio / MultiPublisher)#3urlynn wants to merge 10 commits into
urlynn wants to merge 10 commits into
Conversation
Builds the FLV video tag body (header byte + AVCPacketType + SI24 composition time + payload) from parsed AVC data so a publisher can transmux AVC without re-encoding.
Builds the FLV audio tag body (0xAF header + AAC packet type + payload) from parsed AAC data for transmux-only forwarding.
Add send_video_data (mirrors send_audio_data, uses CSID_VIDEO/MSG_VIDEO) for the publish path. Also fix send_message, which hardcoded stream_id=0 for every chunk and ignored the command's own stream id. publish/play commands must carry the createStream result id, otherwise the rtmp-rs server marks no publishing stream and drops the first media frame. Fixing this makes client publish and play interoperate with rtmp-rs's own server.
High-level counterpart to send_audio: takes parsed H264Data, builds the FLV tag body internally, and sends it on the published stream.
Pushes one stream to several RTMP endpoints at once (e.g. multiple platforms). Each endpoint uses its own RtmpPublisher; connect/send/ disconnect fan out and return per-endpoint Results so a failure on one platform is isolated from the others.
- publisher: add send_audio(&AacData) + send_audio_raw, cache aac_format_byte - publisher: reset state on connect, use proper error variants - aac: add flv_format_byte() derivation, to_flv_tag_body(format_byte) - h264: preallocate capacity, explicit SI24 shifts - error: add BufferFull variant - multi_publisher: concurrent connect with per-target timeout, event-driven readiness (PlatformConnected / PlatformConnectFailed), shared senders
- aac: test flv_format_byte for 44100/22050/5512/96000 Hz - aac: test to_flv_tag_body for SequenceHeader and Raw frame - h264: test to_flv_tag_body for SequenceHeader/keyframe/P-frame/EndOfSequence
The client never sent RTMP Acknowledgement (BytesRead) messages, so a peer that enforces its window ack size paused sending every window (~8s at typical bitrates): periodic stutter, latency reset to the window size on every resume. The puller was the most affected path. - track bytes_received in read_message - save the server's advertised WindowAckSize instead of dropping it - send Acknowledgement once half the window is received (half-window, earlier than the peer's full window so the sender never stalls) - u32 wrap-safe sequence arithmetic
RtmpPublisher only writes after publish; the server's control traffic (SetChunkSize, WindowAckSize, SetPeerBandwidth, PingRequest) piled up unconsumed, and on remote targets the server's send buffer backpressures the push into periodic stalls (same family as the missing-ACK issue). - drain_incoming(): non-blocking read + decode; feeds control messages to handle_incoming, counts bytes, sends ACK (half window) - handle_incoming(): apply SetChunkSize/WindowAckSize, echo WindowAckSize on SetPeerBandwidth, answer PingRequest with PingResponse - call drain_incoming before each send_audio_data - the non-blocking read polls the socket once with a no-op waker instead of timeout(Duration::ZERO, read): the timer version cost ~1ms per call (timer tick), which at 73 frames/s adds ~95ms/s of blocking and makes the push fall behind real time
select_format picked a delta format (1/2/3) even when the new timestamp was smaller than the previous one. The wrapped 32-bit delta (~4G) was then truncated by write_u24; if the low 24 bits were 0xFFFFFF the peer misread it as an extended-timestamp marker and reconstructed a ~2^24 DTS, which breaks players and gets live streams rejected as a DTS discontinuity on a second publish. Detect regression and force format 0 (absolute timestamp) so the peer resynchronizes. Add roundtrip test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds high-level publish APIs for AVC (H.264) and AAC, enabling RTMP push without manual FLV tag construction. Also adds a
MultiPublisherfor fanning out one stream to multiple endpoints with per-platform isolation.Background
The existing
RtmpPublisher(from #2) exposessend_audio_raw(Bytes)for raw FLV audio tag bodies, but has no video publish path and no high-level audio API that derives the FLV format byte fromAudioSpecificConfig. This PR fills both gaps, as pure transmux (no encode/decode).Changes
Media layer (
src/media/)H264Data::to_flv_tag_body()— pure builder, returns FLV video tag bodyAacData::to_flv_tag_body(format_byte)— pure builder, returns FLV audio tag bodyAudioSpecificConfig::flv_format_byte()— derives the first byte from ASCConnector (
src/client/connector.rs)send_video_data()— symmetric with existingsend_audio_data()send_messagenow carriescmd.stream_idonly for Command/CommandAmf3; protocol-control messages stay at stream 0Publisher (
src/client/publisher.rs)send_video(&H264Data, ts)— high-level video publishsend_audio(&AacData, ts)— high-level audio publish (derives + caches format byte)send_audio_raw(Bytes, ts)— renamed fromsend_audio(low-level, no derivation)connect()resets cached state; proper error variants (ConnectionClosed,MediaError::MissingSequenceHeader)MultiPublisher (
src/client/multi_publisher.rs)send_video/send_audio/send_audio_rawfan out to all endpointsconnect_timeout— one slow or unreachable endpoint never delays the othersPlatformConnected/PlatformConnectFailed/PlatformDisconnectedset_sequence_headers()— caches AVC/AAC sequence headers and replays them to each endpoint before its first raw frame, on both connect and reconnectreconnect(id)for manual recovery (caller controls backoff)send_*are synchronous (enqueue, not delivery) — result reflects buffer acceptance, not server acknowledgmentClient connection fixes (required for reliable push)
Acknowledgementis now sent at half the advertised window, so a server that enforces its window never stalls the pushdrain_incoming(): the push path reads the socket (non-blocking) and handles control messages (SetChunkSize,WindowAckSize,SetPeerBandwidth,PingRequest→PingResponse), so a remote server's control traffic never backpressures the pushprotocol/chunk: force chunk format 0 (absolute timestamp) when a message's timestamp goes backward, instead of a wrapped 32-bit delta that peers misparse as an extended-timestamp marker (DTS discontinuity)Usage
Testing
The publish path was exercised end-to-end in an RTMP→RTMP relay
(PS5 → rtmp-rs server →
MultiPublisher→ platform ingest URLs), coveringboth single- and multi-platform configurations.
Single platform. Pushing to one live ingest yields a clean stream with no
stalls over sustained runs.
Multi-platform. The same stream is pushed to two live platforms
simultaneously (e.g. Twitch plus a second live ingest); frame rate and
latency on one are unaffected by the other.
Robustness — one broken platform never blocks the others:
the bad target fails within
connect_timeout(default 10s) and emitsPlatformConnectFailed; the two healthy targets start immediately and pushat full rate throughout — the unusable URL never blocks the stream.
Configuration error: Invalid RTMP URL) with no retry loop and no impact on the other targets.PlatformDisconnectedis emitted and the remaining targets keep pushing.MultiPublisher::reconnectrestores a target; the cached sequence headersare replayed before its first raw frame (
set_sequence_headers), so theserver sees a clean stream.
for that endpoint only (
Err(BufferFull)); every other target stillreceives every frame.
These scenarios exercise the isolation guarantees of
MultiPublisher:parallel per-target connects with individual timeouts, per-endpoint error
events, and an event-driven readiness/reconnect protocol.
Automated.
cargo test— 379 lib unit tests + doc-tests pass (mediatag-builders, MultiPublisher dispatch semantics, and the new publish APIs).
Notes
MultiPublisher::set_sequence_headers()replays the cached AVC/AAC sequence headers to each endpoint before its first raw frame, on both connect and reconnect; call it whenever a new sequence header appears (e.g. stream restart)MultiPublisher::send_*returnVec<Result<()>>(one per endpoint);Err(BufferFull)means enqueue failed,Err(ConnectionClosed)means endpoint task exited — caller decides drop/retry/reconnectFollow-ups (separate PRs)
ChunkEncoderstate reset on stream end, extended-timestamp edge cases, server-side acknowledgement tuning