Skip to content

macOS arm64: packaged app crashes (SIGSEGV) during shutdown after reloading a saved capture profile #2432

Closed

Description

@zackees

Seen in release-rust.yaml verify_only run 40246027884 (job 121333751844, Native macOS arm64 DMG startup, scenario 'reload a saved capture profile'): RuntimeError: packaged frontend failed during shutdown: -11. Shortly before: glBindFramebuffer failed, glGetError returned GL_INVALID_OPERATION(0x502). The x86_64 DMG passed the same scenario with the same signing, so this is not caused by the notarization change in #2430. Needs a crash report from the arm64 runner to find the faulting frame.

Activity

  1. zackees commented on Oct 26, 2026

    @zackees
    MemberAuthor

    /investigate
    Triage: not Ready yet, because the faulting frame is unknown. Missing: (1) the macOS crash report (~/Library/Logs/DiagnosticReports/OBS*.ips) from the arm64 DMG startup job, so the test should upload it as an artifact on failure; (2) whether it reproduces on master's next verify_only run or is a one-off; (3) acceptance: the arm64 DMG 'reload a saved capture profile' scenario exits 0 three runs in a row.

  2. zackees commented on Oct 26, 2026

    @zackees
    MemberAuthor

    Item 1 is done: #2446 (merged) uploads macos-<target>-crash-reports, the OBS .ips and .crash files from DiagnosticReports, when the DMG startup job fails. The next verify_only run that hits this crash will have the faulting frame.

  3. moved this from Triage to Investigating in obs-rust Multiviewon Oct 26, 2026
  4. zackees commented on Oct 27, 2026

    @zackees
    MemberAuthor

    Not Ready — and the acceptance criterion triage wrote cannot be used, because it is already met without any fix.

    Since #2446 merged (2026-10-26T05:40Z), the Native macOS arm64 DMG startup job has passed in all four release-rust.yaml runs: 40352480030 (15:45), 40354990911 (16:00), 40364814002 (17:02) and 40409686763 (22:05, arm64 job 121884233862, all on 2026-10-26). The last failure is run 40246027884, the one filed here. I looked at the merges in between: they are ports, tests and CI changes, none described as a fix for a shutdown crash in this scenario, so I have no evidence of a repair. The honest reading is a flake, and "exits 0 three runs in a row" is satisfied by luck rather than by anything a PR would change.

    What is still missing for Ready:

    1. A reproduced occurrence with its macos-arm64-crash-reports artifact, so the faulting frame is named. #2446 uploads only on failure (if: failure() around .github/workflows/release-rust.yaml:522-537), so a passing run produces nothing to look at; without a .ips there is no stack for a worker to fix and no RED test to write.
    2. Whether the glBindFramebuffer failed, glGetError returned GL_INVALID_OPERATION(0x502) line that precedes the SIGSEGV is the cause or bystander noise. That also needs the failing run's log; nothing in the tree reports that condition as an error today.

    Until a run fails again, needs-investigation is the right state and the artifact hook means the next occurrence arrives with its report. Suggest restating acceptance as "a reproduced failure whose reported faulting frame is gone when the same scenario is re-run", which a PR can show, instead of "three green runs", which this issue has already had.

    One caveat when reading those runs: they are not green overall. 40409686763 failed the release gate's Rust Tier 2 lanes — macOS test_find_data_file and the Windows LNK2005 on matrix4_*; #2534 fixes both test selections (the Windows Rust variant is off under MSVC until #2533) and #2540 fixes the data-file lookup behind the macOS one. Judge "passed" per job, not per run.

  5. zackees commented on Oct 27, 2026

    @zackees
    MemberAuthor

    Still not Ready. Reading the failing job's own log corrects two things in the issue text, and narrows where the fault is — but there is still no crash report, so there is no faulting frame to fix and no RED test to write. Evidence request at the end.

    What the failing job's log actually shows

    Run 40246027884, job 121333751844 (Native macOS arm64 DMG startup, macos-26-arm64, macOS 26.6.2/25G83, "Apple M1 (Virtual)", head bb253a0). Re-readable with gh run view --repo obs-rust/obs-studio --job 121333751844 --log.

    1. The crash was in migrate-default, not in the saved-profile mode the title names. The step runs four scenarios in one job, .github/workflows/release-rust.yaml:516 (for mode in fresh migrate-default migrate-custom existing). In this job only the first finished: Rust configuration isolation verified: fresh, then packaged startup: capture plugins registered, saved profile loaded, clean shutdown, then the next run starts with log frontend log 2026-10-26 00-11-06.txt and dies. So the failing scenario is the second one — the first-run import of an upstream config — and the existing mode (the one that reloads a Rust capture profile) has not crashed in any run since. The title sends a fixer to the wrong scenario.

    2. glBindFramebuffer failed, glGetError returned GL_INVALID_OPERATION(0x502) is bystander noise, not the fault signature. The passing arm64 job from the next day, 121884233862 (run 40409686763, 2026-10-26T22:24:37.1485960Z), logs the identical line at the identical point of shutdown — after All scene data cleared, before the websocket unload — and then completes all four scenarios with exit 0. It is the Apple Software Renderer's teardown line on this runner (the same GPU status JSON reports "Apple Inc. Apple Software Renderer", GPU acceleration YELLOW, detectedGpus: [] in both jobs). It should not be read as a cause.

    3. The death is in the obs-websocket unload, not in graphics teardown. The last line OBS wrote before the SIGSEGV is 00:11:12.747: [obs-websocket] [obs_module_unload] Shutting down.... In the passing job the next lines appear within 150 µs: [WebSocketServer::ServerRunner] IO thread exited., [WebSocketServer::Stop] Server stopped successfully, [obs_module_unload] Finished shutting down., [Scripting] Total detached callbacks: 0, Freeing OBS context data. libobs flushes every log line to stdout (libobs/util/base.c.h:32-56), so that ordering is real, not a lost buffer. Two caveats: the frontend log file's own tail can still be lost on a crash, and this bounds the fault without naming it — no stack, so it does not tell us whether it is the IO-thread join, the server destructor, or plugins/obs-websocket's unload touching state the frontend already tore down. Note the unload path already runs with the frontend callback table gone (the passing job logs error: Tried to call obs_frontend_remove_event_callback with no callbacks!), which is context, not proof.

    The harness is not the killer: close_frontend() sends SIGINT (test/packaged-startup.py:542) and reads -11 from proc.wait() (test/packaged-startup.py:623-626), i.e. the app died from signal 11 itself, and the signal handler installed at frontend/utility/platform-signal-posix.cpp.hpp:52 did receive it (the case also logs ==== Shutting down).

    Why it still is not Ready

    • No stack. #2446 uploads macos-<target>-crash-reports only on failure (if: failure() at .github/workflows/release-rust.yaml:522-536), and no arm64 smoke job has failed since it merged: the arm64 jobs in runs 40352480030, 40354990911, 40364814002 and 40409686763 are all success. So the artifact exists as a hook and has never had anything in it. Without an .ips there is no faulting frame, which means no failing test and no claimable work.
    • A flake, not a known defect. One occurrence on bb253a0, five green arm64 smoke jobs since, nothing in between described as a shutdown fix. That is not a state a PR can be written against.
    • The acceptance criterion triage wrote cannot discriminate. "Exits 0 three runs in a row" has already happened repeatedly without a fix; it passes on luck. It should read: a reproduced failure whose reported faulting frame is gone when the same scenario (the second one, migrate-default) is re-run.

    What the reporter should provide

    1. The next failing run's macos-arm64-crash-reports artifact — the .ips for OBS-Studio-Rust, thread 0 and the nearest frames with symbols.
    2. The same job's log from the ==== Shutting down line of the scenario that died, so the last-flushed line can be compared with the ones above.
    3. Which mode it was: the Rust configuration isolation verified: <mode> lines and the ::group::frontend log <name>.txt header identify it without extra instrumentation.
    4. Confirmation that it is arm64-only (the x86_64 job in run 40246027884 passed the same four scenarios), and anything at all that changed on the runner image between 20261013.0424 and the failing/passing runs.

    One thing a PR can do now without a stack, if the maintainers want it as a separate small issue: make the collector not miss the report. It copies immediately after the failed step, with no wait for ReportCrash to write the file (.github/workflows/release-rust.yaml:522-530), and if-no-files-found: warn (line 536) makes an empty artifact look the same as a missing one. A bounded poll (up to ~30 s for a new *OBS*.ips) and if-no-files-found: error would turn "the artifact exists" into proof that a report was there.

    Duplicate check: no duplicate. Searches for SIGSEGV shutdown, packaged frontend failed and shutdown crash return this issue, #2424 (release signing, unrelated) and #809 (Linux "Crash Detected" dialog after an unclean shutdown, closed and a different symptom).

    Labels: keep bug, needs-investigation, P2; no ready. size: is unset, and I would not set it before a stack exists — the fix could be three lines in the unload path or a change in the plugins/obs-websocket submodule.

  6. zackees commented on Oct 27, 2026

    @zackees
    MemberAuthor

    New evidence from run 40529110726 (job 122287574932, Native macOS x86_64 DMG startup, head b926b01c, macos-26 image 20260824.0517) — the sibling of your arm64 crash, and it corrects one assumption in the text above.

    Two corrections to the earlier write-up:

    1. The failure is not arm64-only. The x86_64 job failed the same second scenario (migrate-default) in this run, while arm64 passed. So "confirmation that it is arm64-only" (your item 4) is answered: it is not.
    2. It is not always a SIGSEGV. Here the app did not crash and did not exit: close_frontend's proc.wait(timeout=30) (test/packaged-startup.py:601) raised TimeoutExpired; crash-reports/ was empty. A hang produces no .ips, so the "wait for a reproduced occurrence with its macos-arm64-crash-reports artifact" path cannot ever yield a stack for this variant.

    The x86_64 stall has a named cause, and it is a different one from the arm64 fault: the process was blocked in obs_module_unload of rtmp-services, joining the file-updater thread, which was stuck in do_http_request (no curl timeout) connecting to obsproject.com — ~74 s (15:36:10 → 15:37:24), ended by the connect giving up:

    +  obs_module_unload      (in rtmp-services) + 22
    +    update_info_destroy  (in rtmp-services) + 41
    +  update_thread          (in rtmp-services) + 1343
    +    do_http_request      (in rtmp-services) + 345
    

    Filed as #2672 with the full coverage map and a proposed fix; it is not the same defect as this one but lives in the same job and fails it the same way. Keeping #2432 for the arm64 SIGSEGV is right.

  7. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Claiming: investigation — the arm64 shutdown SIGSEGV in #2432

    The failure's own log (run 40246027884, job 121333751844) is still available, so the faulting path can be narrowed from the recorded frontend log plus the source, without a Mac:

    1. Read the Native macOS arm64 DMG startup job log; extract the frontend log and the last lines before SIGSEGV (-11).
    2. Trace the shutdown path of the source that failed to initialise on that runner (the log already names it: init_screen_stream: Invalid target display ID: 0, on a VirtualMac2,1 with only Apple Software Renderer).
    3. Compare with the x86_64 DMG lane that passed the same scenario, and with the arm64 lane on a non-virtual runner if one exists.
    4. Post the candidate faulting frame(s) with path:line, say plainly which ones are code reading rather than a reproduced backtrace, and propose the RED test shape (the packaged-startup scenario must exit 0 on a runner with no display).
  8. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Investigation: the arm64 shutdown SIGSEGV — where it can be, and the two teardown races behind this lane

    Read-only: no arm64 backtrace is claimed. Note on overlap: §2 re-derives what the earlier comment on this issue and #2672/#2673 already established (I had not read them before writing it, and the stack matches) — the new material here is the run-by-run table in §1, the exact log cut-off in §3, and the unsynchronized read in §4, which is not the rtmp-services defect. The arm64 runner here is a VM (VirtualMac2,1, Apple M1 (Virtual), 3 cores, no GPU), and that run's artifact list has no crash-report entry (the macos-<target>-crash-reports upload landed later, #2446), so what follows is the archived job logs plus the source they point at.

    1. This is not an arm64 bug — the lane is intermittent on both arches

    Every release-rust.yaml run's two Native … DMG startup jobs, newest first:

    run arm64 x86_64 what happened
    40594533995 success success —
    40590282093 success success —
    40589660141 failure failure Run actions/download-artifact — never reached the smoke test
    40529110726 success failure shutdown hang, stack captured (below)
    40409686763 success success —
    40364814002 success success —
    40354990911 success success —
    40352480030 success success —
    40246027884 failure success -11 (the report above)
    40243859603 failure failure Run actions/download-artifact, not the smoke test
    40240506753 failure failure Run actions/download-artifact, not the smoke test
    40198552343 success success —

    Of the nine runs that actually ran the smoke test: seven pass on both arches, one arm64 -11, one x86_64 hang. So the x86_64 DMG passing in 40246027884 is not a control — the same scenario fails on x86_64 too, in other runs.

    2. The x86_64 failure is #2672, and the stack proves it

    Job 122287574932 (run 40529110726) hit the harness's 30 s shutdown budget (test/packaged-startup.py:601), which then prints sample of the live process. Main thread, 4336 samples, all in one frame chain:

    main  →  OBSApp::~OBSApp()  →  obs_shutdown  →  free_module
          →  obs_module_unload (in rtmp-services)  →  update_info_destroy (in rtmp-services)
          →  _pthread_join  →  __ulock_wait
    

    That is the pthread_join at shared/file-updater/file-updater/file-updater.c.h:41-42 waiting on the rtmp-services update thread, i.e. exactly what #2672 described; #2673 bounded the request (:113-119, CURLOPT_CONNECTTIMEOUT 15 s / CURLOPT_TIMEOUT 60 s). That run is 15:36 UTC on 2026-10-27, i.e. before #2673 merged (18:38 UTC), so the hang is already accounted for. The fix's 60 s per-transfer cap is still larger than the smoke test's 30 s budget, which is a separate, smaller thing (see §5).

    3. The arm64 window is inside obs-websocket's unload

    Frontend log of run 40246027884 (job 121333751844), in order:

    ==== Shutting down ====
    [rtmp-services plugin] Successfully updated file 'services.json' (version 295)
    All scene data cleared
    error: glBindFramebuffer failed, glGetError returned GL_INVALID_OPERATION(0x502)
    [obs-websocket] [obs_module_unload] Shutting down...
    <process died: returncode -11>
    

    In every passing run the three lines that follow Shutting down... are [WebSocketServer::ServerRunner] IO thread exited., [WebSocketServer::Stop] Server stopped successfully, [obs_module_unload] Finished shutting down. — none of them appears here, and nothing after the module unload loop either. That bounds the fault to obs-websocket's obs_module_unload (plugins/obs-websocket/src/obs-websocket.cpp:134-163), not the scene/context teardown that precedes it. The glBindFramebuffer error is present in the passing x86_64 run too, so it is not the discriminator.

    4. A real unsynchronized read in exactly that window

    plugins/obs-websocket/src/websocketserver/WebSocketServer.cpp holds _sessions (a std::map, WebSocketServer.h:99) under _sessionMutex (WebSocketServer.h:98) in every place but one:

    • GetWebSocketSessions — locked (:235)
    • onOpen insert (:277-278) and onClose erase (:348-357) — locked
    • Stop's close loop (:185-200) — locked
    • Stop's wait loop — not locked: :205-206
    std::unique_lock<std::mutex> lock(_sessionMutex);
    for (auto const &[hdl, session] : _sessions) { ... }
    lock.unlock();
    _threadPool.waitForDone();
    while (_sessions.size() > 0)                       // :205 — no lock
        std::this_thread::sleep_for(std::chrono::milliseconds(10));
    _serverThread.join();                              // :208

    The IO thread erases from that same map in onClose while this loop reads size(); a concurrent size() against an std::map::erase (red-black rebalancing) is a data race and UB — and it is the only unsynchronized shared state in the window §3 points at. It is also unbounded: a session whose close handshake never completes keeps _sessions non-empty, so obs_module_unload never returns and the lane fails on its 30 s budget instead of with a crash.

    Minimum fix, in obs-rust/obs-websocket (submodule PR, then a pointer bump here): test the predicate under _sessionMutex, releasing it between polls — a condition_variable notified from onClose is the clean form — and bound the wait (a few seconds) with a warning so unload cannot hang.

    5. What is still missing, and what I am not claiming

  9. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Releasing: the arm64 shutdown SIGSEGV investigation

    The investigation part is delivered in the comment above, and it changes what this issue is:

    What is left is not a worker unit: it needs an arm64 run with #2714's crash-report capture in place (or a Mac that can reproduce it), then #2713's fix in obs-rust/obs-websocket plus a submodule bump here. Whoever takes #2713 can close this issue if the -11 does not come back once the crash report is collected.

  10. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Blocked: waits on #2713 — obs-websocket's unguarded, unbounded _sessions wait in WebSocketServer::Stop() — which is the only code change left from this investigation.

    Recap of the investigation above: the -11 was a single occurrence, and the same scenario also fails on x86_64 via the rtmp-services update join (#2672, fixed by #2673), so this is not an arm64 bug; the arm64 window is obs-websocket's obs_module_unload (#2713); and #2714 (the harness records no stack on a signal exit) is closed, landed by #2718. obs-rust/obs-websocket#22 carries the fix and is merged; #2713 now needs the pointer bump in this repo.

    Linked #2713 as a GitHub dependency so this issue leaves the work list and comes back when it closes. Whoever closes #2713 can close this one if the -11 does not return once the crash report is collected.

  11. github-actions commented on Oct 28, 2026

    @github-actions

    Claim bot: @zackees, nothing is claimed here, so there is no claim to mark blocked.

  12. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Blocker cleared: #2713 is closed and both teardown races from the investigation are on master.

    So the close condition from the 2026-10-28 comment ("whoever closes #2713 can close this one if the -11 does not return once the crash report is collected") is now unblocked but not yet observed: no Scheduled run has reached the macOS arm64 packaged-startup step since #2718 landed the crash-report capture (01:52 UTC), so there is no post-fix arm64 evidence either way. Leaving this open until the next run that reaches that step passes with the .ips capture in place; the step now prints exception type: / crashed thread: on a signal exit, so a recurrence would be attributable rather than a bare -11.

  13. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Coordinator tick — the "next run that reaches that step" has now happened, and the step passed. Recording it here rather than closing, because the head is one commit short of the submodule point.

    release-rust.yaml -f verify_only=true run 40624175646 (head d91c8d965, 2026-10-28T03:00Z) reached both packaged-startup jobs and both are green:

    • Native macOS arm64 DMG startup 🧪 — success
    • Native macOS x86_64 DMG startup 🧪 — success

    What that head contains and what it does not:

    So the pass is real evidence for the harness and for the rtmp-services race (#2673), and no evidence either way for the obs-websocket race. The condition you set — a run that reaches the step, on a head carrying both — is still unmet on master; the next one is the daily beta on a head at or after fc0352611 (current origin/master, which contains #2722, #2723 and #2726).

    One thing to watch when it comes: the daily beta has not published since 2026-10-23 (14 consecutive failures, #2725), so "the next Scheduled run" may also fail before it reaches the packaged-startup step, in the Windows app build. That is #2691's /W4 tail, still with the maintainer as a scope decision, not a reason to read this issue's absence of a crash as anything.

  14. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Coordinator tick — correcting the "next run" assumption in the last comment, because the first run on a head that actually carries both teardown fixes reached the step and it still gives no post-fix evidence about the -11.

    Run 40627344990 is release-rust.yaml -f verify_only=true on head fc0352611 — current origin/master, i.e. after #2722 (dd5d9226b) and #2723 (c805ee479), submodule 777071b — so it is the head the condition wanted:

    • Native macOS x86_64 DMG startup 🧪 — success
    • Native macOS arm64 DMG startup 🧪 — failure, but not the crash: the job dies at Unable to download artifact(s): Artifact not found for name: obs-studio-rust-macos-arm64, crash-reports/ is empty (total 0), and no exception type: / crashed thread: line is printed. The app was never launched.

    The cause is one job earlier: Build macos arm64 🧱 (job 122615438598) failed while packaging —

    hdiutil: couldn't eject "disk33" - Resource busy
    =>  Unable to run 'hdiutil detach /Volumes/obs-studio-33.0.0-beta6-fc0352611-macos-apple' (attempt #1). Retrying.
    hdiutil: detach failed - No such file or directory      (attempts #2-#5)
    ✖︎  Unable to run 'hdiutil detach ...'. Aborting
    

    That is #2728's stale-mountpoint detach bug, and it means: the arm64 DMG never gets built on this lane until #2729 lands, so the packaged-startup step cannot produce the evidence this issue is waiting for, no matter how many times it is dispatched. #2729 (infra/2728-dmg-detach-retry) is reviewed and is the unblocker; it goes into the queue as soon as its Fast Gate is green.

    The two verify_only runs now in flight do not count either, for a different reason: 40628508954 (head 752ae077e) and 40630083913 (head 9b37103d7, #2691) are both three commits behind fc0352611 and carry submodule 30d6f51, i.e. neither #22 nor #23.

    So the status is unchanged and the wait is longer than the last tick implied: the next head that can answer this issue is a green Build macos arm64 on a commit at or after fc0352611 — equivalently, after #2729 lands, any later dispatch. Nothing here is evidence that the -11 is back, and nothing here is evidence that it is gone.

  15. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Closing on the evidence the last two ticks asked for. Post-fix arm64 packaged startup, on current master, with the crash-report capture in place — and it passes.

    Run 40631862829 (release-rust.yaml -f verify_only=true, head 25fc4bc6e = current origin/master, dispatched 04:21Z):

    job result
    Build macos arm64 🧱 success — created: …obs-studio-33.0.0-beta6-25fc4bc6e-macos-apple.dmg, packaged identity verified
    Native macOS arm64 DMG startup 🧪 (job 122635389356) success — the reload a saved capture profile scenario exits 0
    Native macOS x86_64 DMG startup 🧪 success

    That head carries both teardown fixes the investigation named: submodule 777071b (obs-rust/obs-websocket#22, the bounded mutex-guarded Stop() drain, and #23, the onClose() null-session insert) via #2722/#2723, and rtmp-services' update_info_destroy join via #2673. #2718's .ips evidence capture was in the job, so a recurrence would have printed exception type: / crashed thread: instead of a bare -11; nothing was printed and no macos-arm64-crash-reports artifact was produced.

    Worth recording, because it looked like recurrence evidence and was not: the one run between the submodule point and this one, 40627344990 on fc0352611, also failed Native macOS arm64 DMG startup 🧪, but it died at Unable to download artifact(s): Artifact not found for name: obs-studio-rust-macos-arm64 — the app was never launched, because Build macos arm64 🧱 aborted in Package OBS-Studio-Rust 📀 on the hdiutil detach stale-mountpoint bug. That was #2728, fixed by #2729 (infra/2728-dmg-detach-retry), and this run's green Build macos arm64 is the first arm64 DMG to build since.

    What this does and does not prove: the -11 is a single historical occurrence on an intermittent shutdown path, so one green run is consistent with fixed rather than a proof of absence. What is proven is that the specific windows the investigation identified — the obs-websocket _sessions drain during obs_module_unload, and the rtmp-services update-thread join — no longer reproduce on the platform that crashed, with the attribution harness armed. That was the close condition set in the 2026-10-28 comment.

    Reopen if the arm64 (or x86_64) packaged-startup job fails with packaged frontend failed during shutdown again; the crash-report step now names the faulting frame, so it is a new bug at that point, not this one.

  16. moved this from Investigating to Done in obs-rust Multiviewon Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal prioritybugSomething isn't working

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions