Skip to content

bug: obs-websocket: WebSocketServer::Stop reads _sessions without the session mutex, and waits on it unbounded #2713

Closed

Description

@zackees

Found while investigating #2432 (the arm64 packaged-startup SIGSEGV). This is a standalone defect; it is not claimed to be the cause of that crash.

The bug

plugins/obs-websocket/src/websocketserver/WebSocketServer.cpp guards _sessions with _sessionMutex everywhere except one place, and then waits on that unguarded state without a bound.

_sessions is a std::map (WebSocketServer.h:99), _sessionMutex is its lock (WebSocketServer.h:98). Locked uses: GetWebSocketSessions (:235), onOpen insert (:277-278), onClose erase (:348-357), Stop's close loop (:185-200).

Stop()'s wait loop is not locked (:205-206):

std::unique_lock<std::mutex> lock(_sessionMutex);
for (auto const &[hdl, session] : _sessions) { ... }
lock.unlock();
_threadPool.waitForDone();
while (_sessions.size() > 0)                       // :205 — no lock
    std::this_thread::sleep_for(std::chrono::milliseconds(10));
_serverThread.join();                              // :208

The IO thread (ServerRunner, :104-117) calls onClose, which erases from the same map under the mutex, while the main thread reads size() with no lock. Concurrent size() against std::map::erase (node rebalancing) is a data race and undefined behaviour.

It is also unbounded: Stop() returns only after _sessions is empty, so a session whose close handshake never completes keeps obs_module_unload (src/obs-websocket.cpp:134-163) from returning. In the packaged-startup smoke test that shows up as the harness's 30 s shutdown budget expiring (test/packaged-startup.py:601) rather than as a crash.

Why it matters

Stop() is on the process-shutdown path (obs_module_unload), so a race or a stuck wait here fails the Native macOS/Windows DMG startup and Ubuntu/Windows app lanes at the very end of an otherwise green run. #2432's arm64 -11 log stops immediately after [obs_module_unload] Shutting down..., which is this function, before any of its three completion lines — a sample of the x86_64 hang in that same lane shows the other teardown race (rtmp-services' update join, #2672, already fixed).

Proposed fix

In obs-rust/obs-websocket (its own PR, then a submodule pointer bump here — never commit the pointer separately from the bump PR):

  1. Test the predicate under the mutex, releasing it between polls, and prefer a condition_variable: onClose notifies after erasing, Stop() waits on !_sessions.empty().
  2. Bound the wait (a few seconds) and log a warning naming the sessions that did not close, so unload cannot hang.
  3. Keep _serverThread.join() after the wait; do not hold _sessionMutex across join().

Tests

  • A unit test is impossible against the real class without a client, so the RED is a Tier 2/OBS-level one: two clients, one of them never answering the close frame, and deinit must return within the bound with a warning instead of hanging. Today it spins forever (a sample of the hang shows the busy-wait loop).
  • TSan would flag the unsynchronized size(): sanitizers.yaml -f tools=tsan on a build with a client that disconnects during Stop().

Activity

  1. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    /ready

    Triaged against the definition of Ready (WORKFLOW.md §Definition of Ready):

    1. Failing-test sketch: yes — a Tier 2 / OBS-level test with two clients where one never answers the close frame, asserting deinit returns within the bound and logs a warning; plus a TSan run (sanitizers.yaml -f tools=tsan) for the unsynchronized size().
    2. Acceptance criteria: read off the fix plan in the report — (a) _sessions is only read while holding _sessionMutex (or via a condition_variable predicate), (b) Stop() cannot wait unbounded: it bounds the wait and logs which sessions did not close, (c) _serverThread.join() still happens after the wait and is never called while holding _sessionMutex, (d) the busy-wait while (_sessions.size() > 0) is gone.
    3. Dependencies: none open. The fix lands in obs-rust/obs-websocket as its own PR, then a submodule pointer bump here in the same PR — no blocker to link.
    4. No open decision: none raised.
    5. Labels: added P2 (a teardown-path race that fails the packaged DMG/app lanes at the very end of an otherwise green run, not a red master) and size:S (one function, one submodule bump).

    Scope note for whoever claims it: the arm64 SIGSEGV in #2432 is not claimed to be this bug; this is a standalone defect found while investigating it.

  2. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Claiming: fix WebSocketServer::Stop's unguarded, unbounded _sessions wait

    Plan, two PRs (the submodule owns the fix; this repo owns the pointer bump):

    1. obs-rust/obs-websocket: RED first — a test that drives the real WebSocketServer with a client that never answers the close frame and asserts Stop() returns within the bound instead of spinning (plus a TSan-visible predicate read under the lock). GREEN: condition_variable notified from onClose after the erase, polled under _sessionMutex, bounded wait with a warning that names the sessions still open, _serverThread.join() kept after the wait and never under the lock.
    2. obs-rust/obs-studio: bump the plugins/obs-websocket pointer to that commit, and record the teardown behaviour through the packaged-startup harness (test/packaged-startup.py) or state plainly what could not be run here.

    Neither PR advances master; both go through the queue on their own repo's terms.

  3. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Status: the submodule fix is up for review — obs-rust/obs-websocket#22 (fix/websocket-server-stop-drain, head f1d1417).

    What it does, matching this issue's proposed fix:

    • onClose() notifies a std::condition_variable immediately after it erases from _sessions (and after releasing _sessionMutex).
    • Stop() replaces the unlocked while (_sessions.size() > 0) sleep(10ms) poll with wait_for on that variable, predicate evaluated under _sessionMutex, bounded at 5 s.
    • On expiry it logs LOG_WARNING ... %zu session(s) still open after 5000 ms, continuing shutdown: <remote addresses> so an abandoned shutdown is visible in the log instead of being a silent hang, and the module still unloads.
    • _serverThread.join() stays after the wait; the mutex is not held across it.

    Verified so far: clang-format --dry-run --Werror clean, and clang++ -fsyntax-only -Wall -std=c++17 clean against the real headers (websocketpp 0.8.2 / asio 1.36 / Qt 6.11 / OBS libobs+frontend-api). Not verified at runtime: that repo has no test target and no build CI, and the racy path needs a live client, so the RED/GREEN has to come from the obs-studio side (Tier 2 + packaged startup load and shut this plugin down, and the new warning distinguishes "drained slowly" from "hung").

    Next, once #22 is merged: the pointer bump PR here — plugins/obs-websocket to that merge commit plus a ci/test_*.py pinning this shape (drain predicate read under the mutex, the wait is bounded, join() outside the lock), RED against the current pointer. I am not committing the pointer on its own (PR CI rejects a pointer that does not descend from master's).

    Separate defect noticed while reading onClose(): SessionPtr session = _sessions[hdl]; uses operator[], so a close for a handle that was never opened (e.g. onOpen() returned early because the config was unavailable) inserts a null SessionPtr and the next line dereferences it. Not folded into #22; I will file it as its own issue.

  4. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Follow-up landed on the obs-studio side: #2722 bumps plugins/obs-websocket 30d6f511 → 3253206 (merge of obs-rust/obs-websocket#22) and pins the shape offline.

    RED ca8abb5f3 (test only): uv run --no-project -m unittest ci/test_websocket_submodule.py →
    FAILED (failures=3, errors=1) — the pointer predates #22, and SessionDrainShapeTest
    fails on the unlocked while (_sessions.size() > 0) poll, the missing
    _sessionsDrained.wait_for( / kSessionDrainTimeout, and the missing "still open after"
    warning.

    GREEN 819d3f5cd: Ran 5 tests ... OK (skipped=1); ruff check clean; ./lint ok (34.5 s).
    The one skip is the existing bounded-parser ancestry check (shallow submodule); the new
    pointer check no longer skips merely because the checkout is shallow — it only skips when the
    fix commit object is absent too.

    Honest gap, per the issue's test plan: no runtime RED/GREEN here. That path needs a live
    client that ignores the close frame, and obs-rust/obs-websocket has no test target. What the
    packaged-startup and Tier 2 lanes now give us is a distinguishing signal instead of a hang:
    [WebSocketServer::Stop] <n> session(s) still open after 5000 ms, continuing shutdown: <addresses>.
    TSan was not run either.

    docs/features.md carries the ledger line ("pending merge"). I have not closed this issue:
    the runtime/Tier 2 evidence and the TSan run are still open, and #2722 is in the merge queue
    (handed off to the coordinator).

  5. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    The parent-repo pointer this issue needs is #2723: plugins/obs-websocket 3253206 -> 777071b (submodule master, i.e. #22 plus #23's onClose() null-session fix for #2720). #2722 stopped at 3253206; the submodule moved one commit further after that.

    uv run --no-project -m unittest ci/test_websocket_submodule.py passes at 777071b (the bounded-parser, bounded-drain and SessionDrainShapeTest pins), so #22's fix is present as well. Auto-merge is on. Once #2723 is on master, this issue's code change is complete and #2432 can close on the next daily/verify run if the -11 does not return.

  6. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Closed: the fix is on master.

    No compiled lane covers this submodule (Tier 2 configures ENABLE_WEBSOCKET=OFF), so the remaining evidence is the release-gate app build; the crash symptom this was found from is tracked separately in #2432.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Normal prioritybugSomething isn't workingreadyMeets the definition of Ready; claimablesize:SHours

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions