Skip to content

bug: obs-websocket: onClose() inserts a null session with _sessions[hdl] and dereferences it #2720

Closed

Description

@zackees

Found while fixing #2713 (WebSocketServer::Stop's unguarded session wait). Standalone defect, same file, not folded into that PR.

The defect

plugins/obs-websocket/src/websocketserver/WebSocketServer.cpp:346-355 (onClose):

std::unique_lock<std::mutex> lock(_sessionMutex);
SessionPtr session = _sessions[hdl];       // :347 — operator[] INSERTS a default SessionPtr
uint64_t eventSubscriptions = session->EventSubscriptions();   // :348 — null dereference

_sessions is std::map<connection_hdl, SessionPtr> (WebSocketServer.h:99), so operator[] on a missing key default-constructs a null SessionPtr and inserts it, and the very next line calls a member on it. Every other read in the class that can miss uses at() in a try (onMessage, :356-360 in the same file).

How a missing key happens

onOpen() (:219-233) is the only insert path, and it can return after websocketpp has accepted the connection:

  • :223-227 — GetConfig() returns null (module unloading, config pointer unavailable) → return before _sessions[hdl] = ... at :231;
  • the select_subprotocol/session->Set* calls between accept and insert are on the same path.

onClose() is still called for that connection by the IO thread, so the erase path reads a handle that was never inserted. Today that is a null shared_ptr dereference, i.e. undefined behaviour, on the shutdown path.

Second effect, even when the key exists

Because the insert is invisible, a close for a never-opened handle leaves a null entry in the map that only _sessions.erase(hdl) (the next line) removes — but only after the dereference has already happened. Any GetWebSocketSessions() iteration in between would also dereference null (:189-198).

Fix

Read without inserting and bail out of the bookkeeping the session never had:

auto it = _sessions.find(hdl);
if (it == _sessions.end())
    return;
SessionPtr session = it->second;
_sessions.erase(it);

and, when the entry is removed, still notify whatever waits for the drain (#2713's _sessionsDrained), otherwise a Stop() concurrent with this path could wait on the full 5 s bound.

Tests

  • RED: a Tier 2 / OBS-level case that opens a connection while the config is unavailable and asserts unload completes with no null dereference (a build with -fsanitize=address fails today at :348). Cheap alternative if the config-injection path is hard to drive: a unit-level case on WebSocketServer that calls onClose() with a handle that was never opened (needs the private accessor route the plugin tests already use for PLUGIN_TESTS).
  • Not claimed: nothing has been run for this report beyond reading the code paths; the file:line references are against obs-websocket 5de9d09 (the pointer in master today).

Activity

  1. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    /ready

    Triaged against the definition of Ready (WORKFLOW.md §Definition of Ready):

    1. Failing-test sketch: yes — onClose() for a handle that was never inserted dereferences a default-constructed null SessionPtr; an ASan build fails at WebSocketServer.cpp:348.
    2. Scope is bounded: one function in one file of obs-rust/obs-websocket, plus the _sessionsDrained notification that bug: obs-websocket: WebSocketServer::Stop reads _sessions without the session mutex, and waits on it unbounded #2713 introduced; no public API, no protocol change.
    3. Evidence: the file:line references were verified against obs-websocket 5de9d09 and again on 3253206 (the #22 merge, which kept _sessions[hdl] in place).
    4. Owner: claimed by a worker next; the submodule owns the change and this repo only carries the pointer bump.
  2. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Claiming: fix onClose() inserting and dereferencing a null session (_sessions[hdl])

    Plan, two PRs (the submodule owns the fix; this repo owns the pointer bump):

    1. obs-rust/obs-websocket — replace _sessions[hdl] in onClose() with a find() that returns early when the handle was never inserted, keeping the _sessionsDrained notification for the removed entry (bug: obs-websocket: WebSocketServer::Stop reads _sessions without the session mutex, and waits on it unbounded #2713's Stop() bound depends on it). Branch fix/onclose-null-session.
    2. obs-rust/obs-studio — submodule pointer bump once that merges (carrying libobs: Preserve extensions and UTF-8 boundaries in generated filenames #22 and this fix together), with the docs/features.md line in the same PR.

    First PR expected within the hour.

  3. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Status: the submodule fix is merged — obs-rust/obs-websocket#23 (fix/onclose-null-session, squash 777071b5db3dc3cf0fd62f3cea5a2a3ac7992af2, merged 02:20 UTC).

    What it does, matching the Fix section here:

    Verification, honestly: clang-format --dry-run --Werror clean; Code Quality and Lint Markdown green. The repo has no compile lane (its workflows are lint/docs/crowdin only) and no test harness, so the RED from this issue (ASan, a close for a never-opened handle) is not in that PR — the compile gate is the obs-studio pointer bump.

    Pointer bump: obs-rust/obs-studio#2722 already bumps to 3253206; I asked its author to move it to 777071b so one bump carries both fixes. This issue stays open until that PR lands, then it closes with the bump.

  4. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    PR #2723 points plugins/obs-websocket at 777071b (submodule master, obs-rust/obs-websocket#23) — the parent-repo half of this fix.

    When #2723 merges, this issue's fix is in the parent repo and it can close.

  5. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Closed: the fix is on master.

    No compiled lane covers this submodule (Tier 2 configures ENABLE_WEBSOCKET=OFF), so the release-gate app build is the compile evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingreadyMeets the definition of Ready; claimable

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions