Skip to content

release: the daily beta has not published since 2026-10-24 — every release-gate run fails #2725

Closed

Description

@zackees

Summary

The daily OBS-Studio-Rust beta has not published since the 2026-10-23 14:09 run (39967273880). Every release-rust.yaml run since 40097160881 (2026-10-24 13:12) has failed: 14 consecutive runs, and the 2026-10-28 01:17 dispatch was cancelled before it could finish.

The publishing jobs never run because the app builds they consume fail, so this is one outage with two independent causes, not a series of unrelated reds.

Evidence (runs and jobs, read from the logs)

run head started failing jobs
40097160881 af31b65c9 10-24 13:12 first failure of the series
40198552343 38deb4d26 10-25 13:38
40240506753 7992b0b89 10-25 22:30
40352480030 4729f1270 10-26 15:45
40409686763 107b24733 10-26 22:05
40529110726 b926b01c4 10-27 15:15
40590282093 8219c7f1f 10-27 21:52 Build windows x64/arm64, Build linux x86_64, gate Flatpak/Windows/macOS/Ubuntu, Launch packaged Windows x64/arm64, Rust Tier 2 windows
40594533995 8646ef97b 10-27 22:24 same set, no Flatpak

Cause 1 — Linux app build: -Werror=pedantic on the SWIG Lua sources (already fixed on master)

Build linux x86_64 🧱 (job 122507234793, run 40594533995):

shared/obs-scripting/obs-scripting-lua-frontend.c.h:23:75: error: ISO C does not support '__FUNCTION__' predefined identifier [-Werror=pedantic]
shared/obs-scripting/obs-scripting-lua-source.c.h:110:81: error: ... [-Werror=pedantic]
shared/obs-scripting/obs-scripting-lua.h:200:80: error: ... [-Werror=pedantic]
cc1: all warnings being treated as errors

This is #2711, fixed by #2712 (d82a994e4, merged 2026-10-28T00:10Z) — after that run started, so it is stale. The next release run should pass this job; I did not re-run it to confirm.

Cause 2 — Windows app build: C2220: the following warning is treated as an error

Build windows x64 🧱 (job 122507234744) fails in three files, and Build windows arm64 🧱 with it:

file:line warning note
plugins/win-capture/graphics-hook/d3d11-capture.cpp.hpp(14,10), (20,10) C4201 nameless struct/union the anonymous unions in struct d3d11_data
plugins/win-capture/graphics-hook/d3d12-capture.cpp.hpp(24,10) C4201 same shape
libobs/obs-windows.c.h(1129,25) C4113 parameter-list mismatch GetProcAddress/os_dlsym assigned through void *
libobs/obs-win-crash-handler.c.h(345,1), (513,1) C4701 possibly-uninitialized func_offset, C4702 unreachable code

CMAKE_COMPILE_WARNING_AS_ERROR defaults to ON (cmake/common/compiler_common.cmake:95), so these are hard errors in the app lanes. None of them is visible to PR CI: no PR path builds the OBS app (ci/ci_scope.py), so master can be unpublishable with every PR green.

These are the same files the /W4 ratchet PR #2691 already changes (libobs/obs-windows.c.h, libobs/obs-win-crash-handler.c.h, plugins/win-capture/graphics-hook/d3d11-capture.cpp.hpp, d3d12-capture.cpp.hpp), where they are being closed wave by wave. So the Windows half of this outage clears when #2691 lands; the open question is whether the beta waits for it (see below).

Cause 3 — downstream, not independent

Release gate 🚦 (macOS arm64/x86_64, Windows x64, Ubuntu) and Launch packaged Windows x64/arm64 🧪 fail with the builds they consume missing. Release gate 🚦 / Flatpak 📦 fails in Build Flatpak Manifest 🧾 with the known-flaky The process '/app/bin/xvfb-run' failed with exit code 1 (#2702 note), which also fails on heads that predate any branch under test.

Not verified

Proposal

  1. Land build(msvc): Compile C/C++ at /W4 and drop the dead /WX link option (#2626) #2691 (its changed files are exactly this failure's sites), or cherry-pick those four fixes to master if the full /W4 wave is not ready — the beta is blocked either way.
  2. Add a scheduled-release smoke signal: a cheap workflow_dispatch-only job that builds the Windows app from master on every push to master for the paths the app build compiles (libobs/**, plugins/**, shared/**, cmake/**), so a master-only compile break cannot sit for two days behind green PR runs.
  3. Re-dispatch release-rust.yaml on master once (1) lands and confirm the beta publishes.

Activity

  1. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Claiming: investigation — verify the two causes in this report against current master and pin down what still blocks the beta.

    Plan:

    1. Check whether master's Linux app build is green after build(obs-scripting): Exempt the Lua SWIG sources from -Wpedantic (#2711) #2712 (d82a994e4) by reading the newest release-rust.yaml run on master, not by inference.
    2. Read the macOS gate job logs from 40594533995 (the report explicitly does not claim a cause there) and say whether those reds are the missing builds or an independent failure.
    3. Confirm the Windows reds are exactly the sites build(msvc): Compile C/C++ at /W4 and drop the dead /WX link option (#2626) #2691 changes, and report whether build(msvc): Compile C/C++ at /W4 and drop the dead /WX link option (#2626) #2691 is on track (its verify_only dispatch is running now) — no duplicate fix.
    4. Post a verified status comment with the job ids read; file a separate bounded follow-up only if something actionable is left that is not already owned.
  2. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Investigation verdict (worker tick, obs-studio3): the two "unknown" parts of the report are now resolved, and neither is a master defect. The Windows half is the whole remaining blocker and it is already owned by #2691.

    Read from the job logs of run 40594533995 (head 8646ef97b, branch wip/2626-w4-enumerate), not inferred:

    macOS gate reds are not the missing builds — they are a stale head

    Release gate 🚦 / macOS 🍏 (arm64) (job 122508468974) and (x86_64) (122508469017) both die in the step Run soldr rustup toolchain install:

    /Users/runner/work/_temp/…:1: command not found: soldr
    ##[error]Process completed with exit code 127.
    

    Build macos arm64 🧱 (122507234584) and Build macos x86_64 🧱 (122507234814) succeeded in the same run, so the gate red is independent of the packaged app — it is exactly the #2699 shape (soldr called with no zackees/setup-soldr in the job), and it is already fixed on master:

    On current master (fc0352611) build-project.yaml's macOS job carries Set Up soldr 🦀 before Add the Rust x86_64 macOS target 🦀, and a scan of every workflow with ci/test_soldr_rustup.py's soldr_jobs_without_setup() reports zero jobs that call soldr without setup-soldr (build-project.yaml 10 calls, release-rust.yaml 2, no composite action calls soldr). So no new pin is warranted — that invariant test already exists and is file-scoped but currently green everywhere.

    Ubuntu gate red is the #2712 cause, also stale

    Release gate 🚦 / Ubuntu 🐧 (OBS-Studio-Rust) (122508469032):

    shared/obs-scripting/obs-scripting-lua.c.h:45:88: error: ISO C does not support '__FUNCTION__' [-Werror=pedantic]
    cc1: all warnings being treated as errors
    

    That is #2711, fixed by d82a994e4 (#2712), merged 2026-10-28T00:00Z — also not an ancestor of 8646ef97b. Same for Cross Compile 🧰 (122508469080), which stops in the same Lua SWIG TU plus libobs TUs from the same branch.

    Windows is the only live blocker, and #2691 owns it

    On current master the sites the report lists are still unfixed, and they are byte-for-byte the files #2691 changes:

    site master today in #2691's diff
    plugins/win-capture/graphics-hook/d3d11-capture.cpp.hpp:12 anonymous union { present yes
    plugins/win-capture/graphics-hook/d3d12-capture.cpp.hpp:23 anonymous union { present yes
    libobs/obs-windows.c.h:1119-1120 = (void *)GetProcAddress(...) present yes
    libobs/obs-win-crash-handler.c.h:287 DWORD64 func_offset; present yes

    fix/2626-msvc-w4 (PR #2691, CLEAN, draft) has a verify_only dispatch queued since 03:00Z — run 40624175646 — which is the build that decides whether the beta can publish. I did not duplicate it.

    Not verified by me

    Recommendation (no PR from me for this)

    The beta unblocks when #2691 lands and release-rust.yaml is re-dispatched on master — nothing else in this report is master-side work. On proposal 2 (a push-to-master Windows app-build smoke signal): that adds a ~12-minute app build to every master push, so it is a maintainer cost decision, not a worker edit; if it is wanted, the cheapest shape is a workflow_dispatch-only job in build-project.yaml reusing the existing windows-build steps behind a path filter, not a new workflow.

  3. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Releasing: investigation — verdict posted above; no worker PR follows from it.

    The remaining unit is the Windows app build, which lives in #2691 (fix/2626-msvc-w4, CLEAN, verify_only dispatch 40624175646 queued since 03:00Z). The Linux and macOS gate reds in this outage are stale heads (8646ef97b predates both fc97f5fce #2699/#2701 and d82a994e4 #2711/#2712), not master defects. Whoever picks this issue up next only needs the post-#2691 master dispatch and the beta confirmation.

  4. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Claiming: investigation — the report's Cause 2 does not apply to master, so the beta is not waiting on #2691. I dispatched the end-to-end proof on current master and will report its result.

    The Windows app build on master is green; Cause 2 is branch-only

    Run 40529110726 (head b926b01c4, the last Scheduled release run) job table, read from the API:

    job conclusion
    Build windows x64 🧱 success
    Build windows arm64 🧱 success
    Build linux x86_64 🧱 success
    Build macos arm64 🧱 / Build macos x86_64 🧱 success / success
    Release gate 🚦 / Windows 🪟 (x64) and (arm64) success / success
    Launch packaged Windows x64 🧪 / arm64 🧪 success / success

    The C2220 sites in Cause 2 (d3d11-capture.cpp.hpp C4201, obs-windows.c.h C4113, obs-win-crash-handler.c.h C4701/C4702) were reported by run 40594533995, whose head is wip/2626-w4-enumerate — a branch that has /W4 on. Master is still /W3, where those are warnings, not -Werror. So the Windows half of this outage is not a master defect and does not block publishing; raising the level is #2691's change to make, on its own schedule.

    The two reds on that master run are both fixed after its head

    1. Fast Gate — libobs/media-io/audio-math.h:38:25: error: conversion from 'double' to 'float' may change value [-Werror=float-conversion] (cargo:warning= from obs-c-oracle). Fixed by 058448ae7 (Drop the double cast in db_to_mul, fix(media-io): Drop the double cast in db_to_mul that -Wconversion rejects on MinGW (#516) #2651/build: Maximum C/C++ warning strictness with -Werror on every platform (cross-compiled with soldr) #516), which is in origin/master and not in b926b01c4.
    2. Native macOS x86_64 DMG startup 🧪 — subprocess.TimeoutExpired: … timed out after 30 seconds; the live-process sample is parked in __semwait_signal, i.e. the shutdown hang, not a missing artifact (no .ips was written). Both known teardown races in that window are already fixed on master by the submodule bumps dd5d9226b (bounded session drain, fix(obs-websocket): Bump the submodule to the bounded session drain #2722) and c805ee479 (onClose null-session, bug: obs-websocket: onClose() inserts a null session with _sessions[hdl] and dereferences it #2720) — again absent from b926b01c4.

    Action taken

    Dispatched release-rust.yaml -f verify_only=true on master fc0352611 → run 40627344990 (queued 03:33Z). It builds and gates the candidate packages and publishes nothing, so it is the harmless answer to "does the beta publish on today's master". Publishing itself stays a maintainer/schedule action.

    If that run is green, the outage closes with the next daily and nothing needs cherry-picking from #2691. Cause 1 (-Werror=pedantic on the Lua SWIG sources) is likewise stale, fixed by d82a994e4 (#2711/#2712), and the macOS gate command not found: soldr reds by fc97f5fce (#2699/#2701).

  5. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Result of the master dispatch (run 40627344990, head fc0352611): the two known causes are confirmed fixed on master, and one new master-only red appeared — in macOS packaging, not in the app build.

    job result
    Release gate 🚦 / Fast Gate ⚡ success (-Wfloat-conversion in audio-math.h gone)
    Build windows x64 🧱 / arm64 🧱 success / success
    Release gate 🚦 / Windows 🪟 (x64) and (arm64) success / success
    Launch packaged Windows x64 🧪 / arm64 🧪 success / success
    Native macOS x86_64 DMG startup 🧪 success — the 30 s shutdown hang from the previous run does not reproduce; the dd5d9226b + c805ee479 submodule bumps are the fix
    Build macos arm64 🧱 failure — Package OBS-Studio-Rust 📀
    Native macOS arm64 DMG startup 🧪 failure, downstream only: Unable to download artifact(s): Artifact not found for name: obs-studio-rust-macos-arm64

    The arm64 build failure is not a compile problem (compilation, codesign and the .app all succeeded). hdiutil detach /Volumes/obs-studio-…-macos-apple failed once with Resource busy, the half-ejected volume removed the mountpoint, and the retry loop then failed all five attempts on the now-missing path and aborted with "Disk image creation failed". Filed as #2728 with the fix in PR #2729 (detach retried by device node; already-unmounted counts as success).

    So: Cause 1 and the macOS gate/startup reds are stale-head artifacts; Cause 2 was never a master defect; publishing is blocked only by this intermittent packaging abort. Once #2729 lands, re-dispatch release-rust.yaml on master and the beta should publish.

  6. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    #2729 (infra/2728-dmg-detach-retry) merged at 04:18Z, so master is now 25fc4bc6e (frontend: Save the last dock layout that was not clamped (#2727)), and the re-dispatch this comment asked for is running:

    • run 40631862829, release-rust.yaml -f verify_only=true, head 25fc4bc6e, dispatched 04:21Z.

    verify_only publishes nothing, so this is the probe and not the beta itself. What it settles:

    • Build macos arm64 🧱 / Package OBS-Studio-Rust 📀 — whether the device-node detach survives the Resource busy → half-ejected-volume sequence that aborted 40627344990. If this job is green, the release-only red is gone and a publishing run is the maintainer's call, not another verify.
    • Native macOS arm64 DMG startup 🧪 — the first arm64 packaged startup on a head that carries both teardown fixes (submodule 777071b), which is also the evidence macOS arm64: packaged app crashes (SIGSEGV) during shutdown after reloading a saved capture profile #2432 is waiting on.

    If Package aborts again, the new log line to look for is a hdiutil: detach failed that repeats past attempt #1 on the path form (device resolution returning empty), not the original No such file or directory on all five attempts.

  7. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    The macOS red from the master dispatch (run 40627344990, Build macos arm64 🧱) is fixed on master: #2729 merged as e1f0c8b-tip at 04:18 UTC, so hdiutil detach retries the device instead of a path the first attempt destroyed. Nothing else from that run is open: the Windows app build and the beta's publication stay in #2691's queue path.

  8. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Correction to the hash in my previous comment: #2729 merged as 25fc4bc6e1 (25fc4bc) at 04:18 UTC. Everything else in that comment stands.

  9. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    The packaging blocker is gone: #2729 landed on master as 25fc4bc6e (detach retried by device node; an already-unmounted volume counts as success).

    State of the outage on master today:

    item status
    Cause 1, Lua SWIG -Werror=pedantic fixed by d82a994e4 (#2712), verified green in verify_only run 40627344990
    macOS gate command not found: soldr fixed by fc97f5fce (#2701)
    Fast Gate -Wfloat-conversion in audio-math.h fixed by 058448ae7 (#2651), green in the same run
    macOS x86_64 DMG 30 s shutdown hang did not reproduce in 40627344990 (the dd5d9226b + c805ee479 bumps)
    macOS arm64 packaging abort fixed by #2729, merged
    Windows app build green on master (Cause 2 was /W4-branch-only, owned by #2691)

    So nothing on master blocks publishing now; the next release-rust.yaml run (scheduled or dispatched) should publish. Publishing itself I left to the schedule.

    verify_only run 40627344990 covered everything except the arm64 packaging path that #2729 fixes, so a fresh run is the remaining confirmation. I did not dispatch another one: run 40630083913 (on #2691's branch) has been sitting in the release workflow's queue for over an hour, so the queue is backed up and another dispatch would only add to it.

  10. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Result of the re-dispatch on master 25fc4bc6e (run 40631862829, verify_only=true): the packaging abort is gone and the arm64 DMG builds again.

    job result
    Build macos arm64 🧱 (job 122630072405) success — created: …obs-studio-33.0.0-beta6-25fc4bc6e-macos-apple.dmg; packaged identity verified, volume OBS-Studio-Rust 0.1.0-beta.20261028 (Apple), 215 MB
    Native macOS arm64 DMG startup 🧪 success
    Native macOS x86_64 DMG startup 🧪 success
    Release gate 🚦 / macOS 🍏 (arm64) success

    This is the same job that aborted at Package OBS-Studio-Rust 📀 in 40627344990. In this run the detach succeeded on the first attempt (no Resource busy, so the new retry branch was not exercised on the runner) — the honest reading is that the lane is unblocked and the fix does not regress, not that the retry path was demonstrated.

    Nothing else is red in the macOS or Windows app lanes so far; the Linux gate jobs are still running at the time of writing.

    So the four causes this issue tracked are now all addressed on master: the stale-head -Wfloat-conversion red, the Windows /W4 tail (still a scope decision on #2626), the rtmp-services teardown join, and this macOS packaging abort. Publishing the beta is now a maintainer call, not a blocked step — verify_only publishes nothing, so a real release-rust.yaml dispatch on 25fc4bc6e or later is what actually restores the daily beta. If the packaging abort returns on that run, the log to read is Package OBS-Studio-Rust 📀 for a hdiutil: detach failed that repeats on the path form, plus whether hdiutil info produced a /dev/disk* device for the mountpoint at all.

  11. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Addendum — run 40631862829 finished green end to end (conclusion success, 04:59Z), with zero failed jobs:

    • both macOS app builds + both DMG startups, both Windows app builds + both packaged launches, the AppImage and its five distro test containers;
    • every release-gate lane: Fast Gate, Rust Tests (macos-26, windows-2025-vs2026), Tier 2 (Linux, macos-26, windows-2025-vs2026), Rust Cross Lint, Qt Frontend Linux, Ubuntu (OBS-Studio-Rust), Flatpak 📦 / Flatpak Saved Profile 🧪;
    • the CEF job is absent because verify_only skips publishing.

    That is the whole gate this issue said was unreachable, on head 25fc4bc6e. So the state is: nothing red is left on the release path, and the only remaining step is a real publishing dispatch (verify_only publishes nothing). Recommendation: dispatch release-rust.yaml on master as-is — if the beta publishes, this issue closes with it.

  12. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    Status: coordinator tick — dispatched the real publishing run on master 25fc4bc6e after the green verify_only run 40631862829.

    Run: https://github.com/obs-rust/obs-studio/actions/runs/40644942239 (workflow_dispatch, verify_only=false, started 06:37Z).

    This is the end-to-end proof this issue asked for: if the beta publishes, the outage is closed with it. I will leave this issue open until the run's publishing job succeeds; if the packaging abort returns, the log to read is Package OBS-Studio-Rust for a repeated hdiutil: detach failed.

  13. zackees commented on Oct 28, 2026

    @zackees
    MemberAuthor

    The beta published — the outage is closed with the proof this issue asked for.

    Run 40644942239 (release-rust.yaml, verify_only=false, head 25fc4bc6e) finished success at 07:14Z, and release obs-studio-rust-v0.1.0-beta.20261028 was created at 07:13Z with every asset:

    linux-x86_64.AppImage (+ .zsync), linux-x86_64.flatpak, ubuntu-26.04-x86_64.deb, macos-arm64.dmg, macos-x86_64.dmg, windows-x64.zip, windows-arm64.zip, both -symbols.zip, and SHA256SUMS.

    What fixed it, by cause:

    cause fix
    Lua SWIG -Werror=pedantic on the Linux app build #2712 (d82a994e4)
    macOS gate command not found: soldr #2699/#2701 (fc97f5fce)
    macOS packaging abort — hdiutil detach on a path the first attempt destroyed #2729 (25fc4bc6e)

    Verified first with the green verify_only run 40631862829 on the same head, then with this publishing run, so the publish is no longer resting on inference. The last good beta was 2026-10-23; the gap was 2026-10-24 → 2026-10-28.

    Nothing is left from this report. The Pre-release flag on the new tag and 20261023 staying Latest are the release workflow's normal beta behaviour, not a regression from this outage.

  14. moved this from Investigating to Done in obs-rust Multiviewon Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

infraBuild, gate, CI and toolingneeds-investigationUnderdefined: research before it can be Ready

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions