Skip to content

obs-outputs: Windows network optimizations turn off for RTMPS streams #2220

Closed

Description

@rdoupe

Summary

On Windows, plugins/obs-outputs/rtmp-stream.c.h turns off "Enable network optimizations" (the new socket loop) for every rtmps:// URL:

// ugly hack for now, can be removed once new loop is reworked
if (stream->new_socket_loop && !strncmp(stream->path.array, "rtmps://", 8)) {
	warn("Disabling network optimizations, not compatible with RTMPS");
	stream->new_socket_loop = false;
}

Low-latency mode depends on that loop, so it goes off too. Both settings are off by default.

With #2219, Twitch, Vimeo, niconico, BoxCast, Bitmovin, Eventials and EventLive.pro connect over RTMPS. A Windows user who turned on network optimizations for one of them now loses both settings, with only the warning line in the log. YouTube - RTMPS, Facebook and Amazon IVS users already lose them today.

Proposal

Make the new socket loop work over TLS: send through the Mbed TLS context (mbedtls_ssl_write) instead of send() on the raw socket, and handle MBEDTLS_ERR_SSL_WANT_WRITE the way the loop handles WSAEWOULDBLOCK. Then remove the RTMPS check above.

Testing

Windows only, so this needs the Windows box or the merge-queue Windows lane:

  • a unit test of the send loop against a local TLS listener (partial writes, would-block, disconnect),
  • a real stream over RTMPS with network optimizations and low-latency mode on, compared with plain RTMP at the same settings (dropped frames, CPU).

Activity

  1. zackees commented on Oct 25, 2026

    @zackees
    Member

    Triage: same as upstream (the "ugly hack" in rtmp-stream.c.h), so not a port regression. It becomes more visible as #2205 moves services to RTMPS, because Windows users with "network optimizations" on silently lose them. A fix means making the new socket loop work over the TLS layer. That's a design change in upstream code, so it needs a plan before it's Ready: how the loop would drive mbedTLS's non-blocking reads and writes, and how to test it (an RTMPS ingest with the new loop on Windows, compared for throughput and latency). Until then, a small Ready alternative is to make the existing warn visible in the UI. P3, size:M.
    /investigate

  2. zackees commented on Oct 25, 2026

    @zackees
    Member

    /investigate

  3. moved this from Triage to Investigating in obs-rust Multiviewon Oct 25, 2026
  4. zackees commented on Oct 25, 2026

    @zackees
    Member

    Plan (from reading plugins/obs-outputs/rtmp-windows.c.h, librtmp and mbedtls 3.6.4's library/ssl_msg.c; not run on Windows). It covers the missing piece triage asked for: how the loop would drive mbedtls's non-blocking I/O.

    What happens today if the RTMPS check is removed

    The issue's proposal is partly done already: write_data() sends through RTMPSockBuf_Send(), which is mbedtls_ssl_write() when sb_ssl is set. So sends are encrypted. The breakage is elsewhere:

    1. Would-block is not recognized. On the non-blocking socket, mbedtls_ssl_write() returns MBEDTLS_ERR_SSL_WANT_WRITE (-0x6880), not -1. write_data() only checks ret == -1 (WSAEWOULDBLOCK) and ret == 0. Any other negative falls through both branches: no error and can_write stays true, so the loop spins instead of waiting for FD_WRITE. Real TLS errors (MBEDTLS_ERR_SSL_*) are not fatal either, and WSAGetLastError() means nothing for them.
    2. A retry must repeat the same length, or data is corrupted. After WANT_WRITE (or a partial record), mbedtls keeps the encrypted record in out_left. The next mbedtls_ssl_write() only flushes it and then returns the new call's len (capped at the max fragment length), not the old one (ssl_write_real, "we expect … to be called with the same parameters"). Between retries, write_data() passes write_buf_len, which grows as the send thread appends (or min(latency_packet_size, len) in low-latency mode). A bigger retry length makes the loop memmove away bytes that were never encrypted, so the RTMP stream silently loses data.
    3. Reads bypass TLS. On FD_READ, socket_event() calls raw recv() and throws the bytes away. In plain RTMP that only drops server messages, which this loop ignores anyway. Over TLS it also drops records mbedtls must see: close_notify and alerts, so errors show up late as a generic FD_CLOSE. With TLS 1.3 it also drops post-handshake messages: NewSessionTicket (harmless) and KeyUpdate with update_requested (RFC 8446 §4.6.3 requires us to answer with our own KeyUpdate).

    Fix

    • Send: write_data() remembers the length of a TLS write that hasn't finished (stream->tls_pending_len). While it's set, a retry passes exactly that length from the same write_buf start. That holds because write_buf is only memmoved after a successful send, and the send thread only appends to its end. The send result maps like this:
      • WANT_WRITE → would-block: can_write = false, wait for FD_WRITE;
      • WANT_READ → would-block until FD_READ;
      • any other negative → fatal, logging the mbedtls code (mbedtls_strerror);
      • 0 → clear the pending length and continue as today.

    • Read: the FD_READ discard loop reads through TLS_read() (mbedtls_ssl_read) when sb_ssl is set, so mbedtls processes alerts and post-handshake messages. WANT_READ / WANT_WRITE means would-block; MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY or 0 means closed.
    • Then delete the "ugly hack" check in rtmp-stream.c.h.

    Testing

    • Linux, shared code: pull the result mapping and the pending-length rule into a small shared helper. A cmocka test can then pin them on Linux: WANT_WRITE / WANT_READ / fatal / positive, and a retry after WANT_WRITE uses the first length even when the buffer has grown.
    • Linux, against mbedtls: a cmocka or Tier 3 test that drives a real mbedtls client over a socketpair or loopback, with a tiny send buffer and a server that reads slowly. It checks that the bytes the server decrypts equal the bytes queued, which is the corruption case in point 2.
    • Windows: the issue's own checks, on the Windows box or the merge-queue Windows lane. A real RTMPS stream with network optimizations and low-latency mode on, compared with plain RTMP at the same settings.

    Not claiming yet. The fix touches upstream code paths, so this needs a maintainer OK first; the issue is marked /investigate.

  5. zackees commented on Oct 27, 2026

    @zackees
    Member

    /ready

    Triage asked for a plan for driving Mbed TLS through the new socket loop before this issue could be Ready; the plan in the comment above (2026-10-25) supplies it, and I checked its load-bearing claims against the tree at 3ed8f10b4:

    • plugins/obs-outputs/librtmp/rtmp.c:4532-4551 — RTMPSockBuf_Send() returns TLS_write(), which is mbedtls_ssl_write() (plugins/obs-outputs/librtmp/rtmp.h:104) when sb_ssl is set. So sends already go through TLS and the break is in the result handling, as the plan says.
    • plugins/obs-outputs/rtmp-windows.c.h:160-201 — write_data() treats only ret == -1 + WSAEWOULDBLOCK and ret == 0 as would-block/fatal, so MBEDTLS_ERR_SSL_WANT_WRITE (-0x6880) falls through with can_write still true and WSAGetLastError() is queried for a code that is not a Winsock error.
    • plugins/obs-outputs/rtmp-windows.c.h:152-163 — the send length is recomputed from write_buf_len on every call (or min(latency_packet_size, len) in low-latency mode), which is the retry-length hazard in the plan's point 2.
    • plugins/obs-outputs/rtmp-windows.c.h:58-87 — the FD_READ path reads with raw recv() and discards, bypassing TLS.
    • plugins/obs-outputs/rtmp-stream.c.h:1266-1270 — the RTMPS check to delete.

    Acceptance criteria for the PR:

    1. With network optimizations on, starting an rtmps:// stream no longer logs Disabling network optimizations, not compatible with RTMPS, and new_socket_loop and low_latency_mode both stay true (rtmp-stream.c.h:1260-1274); the check at 1266-1270 is gone.
    2. RED then GREEN on Linux: a cmocka test of the extracted send-result mapping pins WANT_WRITE to "clear can_write, wait for FD_WRITE", WANT_READ to waiting for FD_READ, any other negative to fatal with the Mbed TLS code logged via mbedtls_strerror, and > 0 to success; plus a retry after WANT_WRITE that reuses the first call's length after the buffer has grown.
    3. A real RTMPS run with both settings on, compared with plain RTMP at the same settings, on Windows, with the log in the PR (Tier 4). No PR lane runs the Windows app — full OBS app builds run only on schedule, workflow_dispatch, tags and the release gate (ci/ci_scope.py, "full: never from a PR path") — so this cannot come from PR CI.

    Three corrections and limits for whoever claims it:

    • The plan cites Mbed TLS 3.6.4; this tree requires at least 3.6.7 and fails configure below it on Windows, macOS and Flatpak (plugins/obs-outputs/CMakeLists.txt:11-17).
    • Nothing in the loop can run under Linux Tier 2 until the mapping and the pending-length rule are pulled into a platform-neutral helper: the whole loop is inside #ifdef _WIN32 (rtmp-windows.c.h:3), and the POSIX host has no socket loop at all — test/cmocka/test_obs_outputs_rtmp_host.c:67 asserts rtmp_host_has_socket_loop() is false. That extraction is the RED commit; a new test also has to be added to the cmocka aggregate target and must build on Windows (docs/rust-port/testing-policy.md, "A shared .c test selects Linux Tier 2, not Windows").
    • master's Windows Tier 2 lane was red in run 40409686763 (2026-10-26 22:05, LNK2005 matrix4_* symbols in test_graphics_state_stacks_rust.exe); 1a90124a3 (test/cmocka: Fix master's Windows and macOS Rust Tier 2 lanes #2534) turns that Rust test off under MSVC, with test/cmocka: Run test_graphics_state_stacks_rust on Windows again (LNK2005 on matrix4_*) #2533 to restore it. Re-check the lane before reading a Windows failure as this PR's fault.

    Labels: P3 and size:M are already right for two Windows files plus one shared helper and its test — no label change.

  6. added
    readyMeets the definition of Ready; claimable
    and removed
    needs-investigationUnderdefined: research before it can be Ready
    on Oct 27, 2026
  7. moved this from Investigating to Ready in obs-rust Multiviewon Oct 27, 2026
  8. zackees commented on Oct 28, 2026

    @zackees
    Member

    Claiming: #2220 — make the new socket loop (network optimizations) work over RTMPS.

    1. RED: pull the send-result mapping and the pending-length rule into a platform-neutral helper
      (plugins/obs-outputs/rtmp-send-result.h) and add Linux cmocka cases to the existing
      test_obs_outputs_rtmp_host binary: WANT_WRITE -> clear can_write/wait for FD_WRITE,
      WANT_READ -> wait for FD_READ, any other negative -> fatal with the Mbed TLS code,
      > 0 -> success, and a retry after WANT_WRITE that reuses the first call's length after
      write_buf_len has grown.
    2. GREEN: write_data() classifies through the helper (no more WSAGetLastError() for a
      non-Winsock code), the FD_READ discard reads through TLS when sb_ssl is set, and the
      rtmps:// check in rtmp-stream.c.h is deleted so new_socket_loop and
      low_latency_mode both stay on.
    3. ./lint plus the focused cmocka binary on Linux. The PR says Refs #2220: acceptance
      item 3 (a real RTMPS run on Windows with both settings on) still needs the Windows lane
      or hardware, and no PR lane builds the Windows app.

    First PR: within the hour.

  9. 8 remaining items

  10. moved this from Ready to In progress in obs-rust Multiviewon Oct 28, 2026
  11. removed their assignment
    on Oct 29, 2026
  12. zackees commented on Oct 29, 2026

    @zackees
    Member

    Status from the worker tick: the code half of this issue is on master, and the only thing left is the hardware run.

    On master now (2fe262aa3):

    • The rtmps:// carve-out is gone from plugins/obs-outputs/rtmp-stream.c.h; grep -rn "not compatible with RTMPS" plugins/ finds nothing, so new_socket_loop and low_latency_mode now stay on for an RTMPS URL.
    • The loop sends through mbedtls_ssl_write()'s result classes (plugins/obs-outputs/rtmp-send-result.h), pinned by Linux Tier 2 in the crate's existing test_obs_outputs_rtmp_host binary (12 cases, green locally).
    • Windows evidence recorded on obs-outputs: Run the new socket loop over RTMPS (#2220) #2778: verify-only run 40684548610 built the Windows app for x64 and arm64.

    Still open on this issue: acceptance item 3 — a real RTMPS stream on Windows with network optimizations and low-latency mode on, compared with plain RTMP at the same settings (dropped frames, CPU), log pasted here. No checkout in this fleet has a Windows host, and no PR lane builds the Windows app, so a worker cannot produce it. Labelling this human-review-needed to match the other hardware-gated issues (#2424, #2383, #2950).

    When that run is pasted here, the issue can close: nothing in the code path still needs converting.

  13. moved this from In progress to Needs decision in obs-rust Multiviewon Oct 29, 2026
  14. zackees commented on Oct 30, 2026

    @zackees
    Member

    Decided: close #2220 as completed; the Windows RTMPS live-stream comparison (acceptance item 3) becomes a release-validation check instead of holding this issue open.

    Reasons:

    • The code change this issue asks for is on master: obs-outputs: Run the new socket loop over RTMPS (#2220) #2778 (4ed317257) removed the rtmps:// carve-out in plugins/obs-outputs/rtmp-stream.c.h, and plugins/obs-outputs/rtmp-send-result.h now handles MBEDTLS_ERR_SSL_WANT_WRITE/WANT_READ and the same-length retry. RED→GREEN is pinned in Linux Tier 2 (test_obs_outputs_rtmp_host), as the testing policy requires.
    • The Windows lanes that can run it are green: Rust Tier 2 (windows-2025-vs2026) and Rust Tests (windows-2025-vs2026) on obs-outputs: Run the new socket loop over RTMPS (#2220) #2778, and verify-only run 40684548610 built the Windows app for x64 and arm64.
    • What is left needs Windows hardware, and no worker or PR lane has it (AGENTS.md: the full app builds run only on schedule, dispatch, tags and the release gate). Keeping a merged feature open for a manual run leaves it in the escalated list for good. That run belongs with the release checks, where a Windows app is built anyway.
    • Easy to undo: if the release check on Windows shows a regression (the "Disabling network optimizations" warning, dropped frames or CPU clearly worse than plain RTMP at the same settings), reopen this issue with the log, or file a bug that links here.

    Removing human-review-needed.

  15. moved this from Needs decision to Done in obs-rust Multiviewon Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low priorityenhancementNew feature or requestreadyMeets the definition of Ready; claimablesize:MAbout a day

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions