You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This one failure is test_recording_shutdown_wait in Rust Tier 2 (ON): test_wait_ends_on_stop_and_keeps_ui_running (out.active.load() is not false, so the wait returned while the output was still active) and then test_wait_times_out_without_stop. That's the early return from #1391. The fix, #1392, merged at 04:47 UTC, after this run had started at 04:35 on 827bd13, so this run didn't have it.
On current master (with #1392), Rust lane (ENABLE_RUST_LIBOBS=ON): ctest -R test_recording_shutdown_wait --repeat until-fail:20 passed 20/20, and it passed in a full C-lane ctest -j4 too. Tonight's scheduled run should confirm it.
@zackees, here is a CI audit of the scheduled and master runs as of 5bda2e6.
What's broken
Release OBS-Studio-Rust: there has been no release since v0.1.0-beta.20261018. The last green run was 39115087613 (10-18). In 39284837053 (10-19), every job passed except Release gate / Rust Cross Lint (release targets): all 5 builds, the AppImage and its 4 distro tests, the Windows and macOS smokes, and the rest of the gate. Publish was skipped only because of that one job.
Nightly full run (this issue): 39200819508 (10-19) has the same single failure. The 10-18 failure was test_recording_shutdown_wait, which fix: recording shutdown waits for inactive after deactivate #1392 fixed (see the comment above). Tier 2 ON has been green since then.
Sanitizers: asan, tsan and valgrind have all failed in "Install dependencies" every night since 10-16 (first red 38896638310, last green 38731778779 on 10-15).
Push on master is green.
Root causes (two, both in code, not infra; the runner image is the same on the green and red runs)
rust/tools/sanitize/install-ubuntu-deps.sh scrapes the package list from rust/tools/linux-validate/Dockerfile. ci: Run Tier 2 Linux natively on the runner #967 moved the list to apt-packages.txt, so the scrape now finds nothing (no package list found in .../Dockerfile), and Triage then fails with "No results".
What we're fixing
(1) is fixed in obs-util: Allow target-dependent wchar_t and c_char casts #1485. Its local cross-release lane is green. Its Fast Gate is red because of a flaky test that is already on master: callback::decl_parity::random_pieces_match_oracle drops the Rust-side log line when the whole obs-tests-core binary runs in parallel. It fails about 1 in 10 runs locally and passes when run alone. We'll look at that next.
(2) is ci: Install the sanitizer deps from apt-packages.txt #1487. The script reads apt-packages.txt, and a new ci/ test runs it against a stubbed apt and fails on the old script. A local asan build past the install step found one error the install failure had been hiding: test_ws_save_changes (obs-websocket: Save the scene collection after changes #812) fails to link under UBSan with undefined reference to `typeinfo for WebSocketServer'. UBSan's vptr check needs that typeinfo, and the test stubs the server, so the same branch builds that one target with -fno-sanitize=vptr. The local build also hit a GCC -Wformat-truncation false positive in libobs/util/pipe-variadic.c.h:16, with a newer GCC than the runner has. A Sanitizers run on ci: Install the sanitizer deps from apt-packages.txt #1487's head will show whether CI hits it too.
Whether to run the cross-release lane (about 3.5 min) on master push or on rust/** PRs, so target-only lints show up the day they land. fix: 10 bug fixes across Rust, CI and video output #1462 touches obs-util/src/platform/unix/nix.rs and won't get that lane on its PR either.
Full validation of master f1783cf, run 39691274808, completed with two failures. All native app builds (including the upgraded Flatpak action), Qt, Fast Gate and all Tier 2 OFF/ON jobs passed. This is not a full-green validation result.
Release cross-lint job 119537886574: the Linux ARM64 target fails in rust/obs-util/build.rs because its target sysroot has no gio-2.0.pc. uv run ci/local_gate.py --lane cross-release reproduces this locally after the Windows ARM64 and macOS x64 checks pass. The GIO-enabled code must stay enabled; provide a proper target dependency discovery/setup rather than disabling the probe or pointing a target link at host libraries.
macOS Rust job 119540175722: Application-category native_build::mixed_objective_languages_compile_load_and_rebuild_with_unity_and_pch fails its rebuild/load assertion (33 cases pass, 1 fails in that executable). This matches the Objective-C dylib lifetime fixture problem already addressed by the unmerged final head of build: preserve source options and add the macOS Cargo core graph #1669 (7c03fac), whose description explains the fresh-process correction. No duplicate fix or test relaxation is proposed here.
Reporting these under the existing full-run incident; my active issue remains #1269's CI cache retention work. No workflow rerun, target skip, feature disablement, C test edit, or gate bypass was performed.
Confirmed the existing release cross-lint GIO setup failure again while qualifying #1269's opt-in cache-budget measurement path (#1681). This was one defined experiment dispatch, not a blind rerun:
The job log's Linux ARM64 obs-util build reports GIO feature needs gio-2.0, pkg-config exit 1, and missing gio-2.0.pc from the target setup. The Windows ARM64/macOS x64 checks preceding it passed in that lane.
The full run remains active for independent Windows native experiments and other coverage. Their eventual results must not be called a green full run. No GIO code/check/target was disabled, no host GIO library was substituted for the target, and no fix to this setup failure is claimed by #1681. I remain on #1269 rather than claiming a parallel #1395 implementation.
Triage of the 2026-10-23 05:18 failure (run 39902551202, f948723): the only failed job is Fast Gate, lane rust-test, test native_build::native_test_graph_replaces_legacy_utility_definitions. That is the pinned core cmocka count going stale when two test PRs landed (#1908). #1910 (merged after that run) removed the pin, so the next scheduled run should not repeat it. The Linux ARM64 release cross-lint GIO sysroot failure from 10-21/10-22 is separate and still open here.
Triage of run 40063131689 (56732d6): the only failed job is Rust Tier 2 macos-26, test_context_names segfaulting in test_canvas_sources teardown. That is #2049, a use after free in obs_canvas_destroy from #2041; #2052 (queued) fixes it.
Status: both causes of the 10-25 failure are fixed on master: #2355 (the Qt suite count) and #2359 (the Windows PCH include order). This tracker gets a comment from the bot on every failed nightly; leave it open until the next scheduled full run passes, then close it. Preventing this class of break at PR time is #1294 part 2. P1 (master's full build), size:S.
Triage of run 40270992228 (2026-10-26). Every job passed except CEF Chromium Support, so both causes of 10-25's failure are fixed (#2355, #2359). The Windows x64 app build passes again.
The one failure is the version-floor guard working as intended. ci/cef_support.py reports that macos-arm64, macos-x86_64, ubuntu-aarch64, windows-arm64 and windows-x64 still ship CEF 7871, which is older than Chrome's extended stable (Chromium 156, branch 8078). That is #1864: #2417 is the Windows cross-build (blocked on a Windows toolchain package and disk), #2418 is macOS (needs a Mac), and linux aarch64 is the remaining Ready part.
So the nightly stays red until those archives exist. Nothing else here needs a fix. I suggest closing this tracker and letting #1864 own the remaining failure, unless you want the guard made non-blocking until then.
The failures originally tracked here have fixes on master: #1392, #1485/#1487, #1910, #2052, #2355, and #2359. The latest full run passed the previously failing app and test jobs; its remaining CEF Chromium Support failure is tracked by #1864 and its platform issues. Closing this incident tracker so #1864 owns the current release-floor failure.
Activity
rdoupe commentedon Oct 18, 2026
This one failure is
test_recording_shutdown_waitin Rust Tier 2 (ON):test_wait_ends_on_stop_and_keeps_ui_running(out.active.load() is not false, so the wait returned while the output was still active) and thentest_wait_times_out_without_stop. That's the early return from #1391. The fix, #1392, merged at 04:47 UTC, after this run had started at 04:35 on 827bd13, so this run didn't have it.On current master (with #1392), Rust lane (
ENABLE_RUST_LIBOBS=ON):ctest -R test_recording_shutdown_wait --repeat until-fail:20passed 20/20, and it passed in a full C-lanectest -j4too. Tonight's scheduled run should confirm it.github-actions commentedon Oct 19, 2026
The scheduled full run failed on d5101a9: https://github.com/obs-rust/obs-studio/actions/runs/39200819508
PRs skip these checks (#992), so a recent merge may have broken them.
rdoupe commentedon Oct 20, 2026
@zackees, here is a CI audit of the scheduled and master runs as of 5bda2e6.
What's broken
v0.1.0-beta.20261018. The last green run was 39115087613 (10-18). In 39284837053 (10-19), every job passed exceptRelease gate / Rust Cross Lint (release targets): all 5 builds, the AppImage and its 4 distro tests, the Windows and macOS smokes, and the rest of the gate. Publish was skipped only because of that one job.test_recording_shutdown_wait, which fix: recording shutdown waits for inactive after deactivate #1392 fixed (see the comment above). Tier 2 ON has been green since then.Root causes (two, both in code, not infra; the runner image is the same on the green and red runs)
unnecessary_castonaarch64-unknown-linux-gnu, wherewchar_tisu32:rust/obs-util/src/platform/unix/file.rs:55and:158, from util: Port platform-file.c helpers to obs-util #1444. obs-util: port cf-parser (#1466) #1467 adds a third cast of the same kind inffi/cf_parser.rs. The cross-release lane only runs in the full gate (ci: Cross Dylint targets: PR gate win-x64/mac-arm64/linux-x64; release gate adds win-arm64, mac-x64, linux-arm64 #955), so PR and push CI can't see it.rust/tools/sanitize/install-ubuntu-deps.shscrapes the package list fromrust/tools/linux-validate/Dockerfile. ci: Run Tier 2 Linux natively on the runner #967 moved the list toapt-packages.txt, so the scrape now finds nothing (no package list found in .../Dockerfile), and Triage then fails with "No results".What we're fixing
callback::decl_parity::random_pieces_match_oracledrops the Rust-side log line when the wholeobs-tests-corebinary runs in parallel. It fails about 1 in 10 runs locally and passes when run alone. We'll look at that next.apt-packages.txt, and a newci/test runs it against a stubbed apt and fails on the old script. A local asan build past the install step found one error the install failure had been hiding:test_ws_save_changes(obs-websocket: Save the scene collection after changes #812) fails to link under UBSan withundefined reference to `typeinfo for WebSocketServer'. UBSan's vptr check needs that typeinfo, and the test stubs the server, so the same branch builds that one target with-fno-sanitize=vptr. The local build also hit a GCC-Wformat-truncationfalse positive inlibobs/util/pipe-variadic.c.h:16, with a newer GCC than the runner has. A Sanitizers run on ci: Install the sanitizer deps from apt-packages.txt #1487's head will show whether CI hits it too.What needs you
workflow_dispatchcan't target a branch on a fork, so we can't start it ourselves), or the next cron after it merges.rust/**PRs, so target-only lints show up the day they land. fix: 10 bug fixes across Rust, CI and video output #1462 touchesobs-util/src/platform/unix/nix.rsand won't get that lane on its PR either.ci: Install the sanitizer deps from apt-packages.txt (obs-rust#1487)
zackees commentedon Oct 21, 2026
Full validation of master f1783cf, run 39691274808, completed with two failures. All native app builds (including the upgraded Flatpak action), Qt, Fast Gate and all Tier 2 OFF/ON jobs passed. This is not a full-green validation result.
uv run ci/local_gate.py --lane cross-releasereproduces this locally after the Windows ARM64 and macOS x64 checks pass. The GIO-enabled code must stay enabled; provide a proper target dependency discovery/setup rather than disabling the probe or pointing a target link at host libraries.Reporting these under the existing full-run incident; my active issue remains #1269's CI cache retention work. No workflow rerun, target skip, feature disablement, C test edit, or gate bypass was performed.
zackees commentedon Oct 22, 2026
Confirmed the existing release cross-lint GIO setup failure again while qualifying #1269's opt-in cache-budget measurement path (#1681). This was one defined experiment dispatch, not a blind rerun:
obs-utilbuild reportsGIO feature needs gio-2.0, pkg-config exit 1, and missinggio-2.0.pcfrom the target setup. The Windows ARM64/macOS x64 checks preceding it passed in that lane.The full run remains active for independent Windows native experiments and other coverage. Their eventual results must not be called a green full run. No GIO code/check/target was disabled, no host GIO library was substituted for the target, and no fix to this setup failure is claimed by #1681. I remain on #1269 rather than claiming a parallel #1395 implementation.
github-actions commentedon Oct 23, 2026
The scheduled full run failed on f948723: https://github.com/obs-rust/obs-studio/actions/runs/39902551202
PRs skip these checks (#992), so a recent merge may have broken them.
zackees commentedon Oct 23, 2026
Triage of the 2026-10-23 05:18 failure (run 39902551202, f948723): the only failed job is Fast Gate, lane rust-test, test
native_build::native_test_graph_replaces_legacy_utility_definitions. That is the pinned core cmocka count going stale when two test PRs landed (#1908). #1910 (merged after that run) removed the pin, so the next scheduled run should not repeat it. The Linux ARM64 release cross-lint GIO sysroot failure from 10-21/10-22 is separate and still open here.github-actions commentedon Oct 24, 2026
The scheduled full run failed on 56732d6: https://github.com/obs-rust/obs-studio/actions/runs/40063131689
PRs skip these checks (#992), so a recent merge may have broken them.
zackees commentedon Oct 24, 2026
Triage of run 40063131689 (56732d6): the only failed job is Rust Tier 2 macos-26,
test_context_namessegfaulting intest_canvas_sourcesteardown. That is #2049, a use after free in obs_canvas_destroy from #2041; #2052 (queued) fixes it.github-actions commentedon Oct 25, 2026
The scheduled full run failed on bd029c8: https://github.com/obs-rust/obs-studio/actions/runs/40162888607
PRs skip these checks (#992), so a recent merge may have broken them.
zackees commentedon Oct 25, 2026
Coordinator triage of run 40162888607 (bd029c8): three failed jobs, two causes.
Rust Testson macOS and Windows,canonical_remux_queue_runs_unchanged(left: 2, right: 1): frontend: Import a profile from a path with a trailing slash #2312 added a second Qt-suite test. Fixed by obs-build: Count the Qt suite from the graph in the remux queue test #2355 (merged).Windows 🪟 (x64)app build, C2086/C2146 inshellapi.h: frontend: obs-main Windows startup, crash dialog and portable mode behind host files (#865) #2176's newfrontend-windows-obs-mainPCH includesshellapi.hbeforewindows.h. Fix in frontend: Include windows.h before shellapi.h in the obs-main Windows PCH #2359. PR CI doesn't build the app, so the next full run confirms it.zackees commentedon Oct 25, 2026
/ready
Status: both causes of the 10-25 failure are fixed on master: #2355 (the Qt suite count) and #2359 (the Windows PCH include order). This tracker gets a comment from the bot on every failed nightly; leave it open until the next scheduled full run passes, then close it. Preventing this class of break at PR time is #1294 part 2. P1 (master's full build), size:S.
zackees commentedon Oct 26, 2026
Triage of run 40270992228 (2026-10-26). Every job passed except
CEF Chromium Support, so both causes of 10-25's failure are fixed (#2355, #2359). The Windows x64 app build passes again.The one failure is the version-floor guard working as intended.
ci/cef_support.pyreports thatmacos-arm64,macos-x86_64,ubuntu-aarch64,windows-arm64andwindows-x64still ship CEF 7871, which is older than Chrome's extended stable (Chromium 156, branch 8078). That is #1864: #2417 is the Windows cross-build (blocked on a Windows toolchain package and disk), #2418 is macOS (needs a Mac), and linux aarch64 is the remaining Ready part.So the nightly stays red until those archives exist. Nothing else here needs a fix. I suggest closing this tracker and letting #1864 own the remaining failure, unless you want the guard made non-blocking until then.
zackees commentedon Oct 27, 2026
The failures originally tracked here have fixes on master: #1392, #1485/#1487, #1910, #2052, #2355, and #2359. The latest full run passed the previously failing app and test jobs; its remaining CEF Chromium Support failure is tracked by #1864 and its platform issues. Closing this incident tracker so #1864 owns the current release-floor failure.