Skip to content

ci: Nightly full run failed on master #1395

Closed

Description

@github-actions

The scheduled full run failed on 827bd13: https://github.com/obs-rust/obs-studio/actions/runs/39097646214

PRs skip these checks (#992), so a recent merge may have broken them.

Activity

  1. rdoupe commented on Oct 18, 2026

    @rdoupe
    Collaborator

    This one failure is test_recording_shutdown_wait in Rust Tier 2 (ON): test_wait_ends_on_stop_and_keeps_ui_running (out.active.load() is not false, so the wait returned while the output was still active) and then test_wait_times_out_without_stop. That's the early return from #1391. The fix, #1392, merged at 04:47 UTC, after this run had started at 04:35 on 827bd13, so this run didn't have it.

    On current master (with #1392), Rust lane (ENABLE_RUST_LIBOBS=ON): ctest -R test_recording_shutdown_wait --repeat until-fail:20 passed 20/20, and it passed in a full C-lane ctest -j4 too. Tonight's scheduled run should confirm it.

  2. github-actions commented on Oct 19, 2026

    @github-actions
    Author

    The scheduled full run failed on d5101a9: https://github.com/obs-rust/obs-studio/actions/runs/39200819508

    PRs skip these checks (#992), so a recent merge may have broken them.

  3. rdoupe commented on Oct 20, 2026

    @rdoupe
    Collaborator

    @zackees, here is a CI audit of the scheduled and master runs as of 5bda2e6.

    What's broken

    • Release OBS-Studio-Rust: there has been no release since v0.1.0-beta.20261018. The last green run was 39115087613 (10-18). In 39284837053 (10-19), every job passed except Release gate / Rust Cross Lint (release targets): all 5 builds, the AppImage and its 4 distro tests, the Windows and macOS smokes, and the rest of the gate. Publish was skipped only because of that one job.
    • Nightly full run (this issue): 39200819508 (10-19) has the same single failure. The 10-18 failure was test_recording_shutdown_wait, which fix: recording shutdown waits for inactive after deactivate #1392 fixed (see the comment above). Tier 2 ON has been green since then.
    • Sanitizers: asan, tsan and valgrind have all failed in "Install dependencies" every night since 10-16 (first red 38896638310, last green 38731778779 on 10-15).
    • Push on master is green.

    Root causes (two, both in code, not infra; the runner image is the same on the green and red runs)

    1. clippy unnecessary_cast on aarch64-unknown-linux-gnu, where wchar_t is u32: rust/obs-util/src/platform/unix/file.rs:55 and :158, from util: Port platform-file.c helpers to obs-util #1444. obs-util: port cf-parser (#1466) #1467 adds a third cast of the same kind in ffi/cf_parser.rs. The cross-release lane only runs in the full gate (ci: Cross Dylint targets: PR gate win-x64/mac-arm64/linux-x64; release gate adds win-arm64, mac-x64, linux-arm64 #955), so PR and push CI can't see it.
    2. rust/tools/sanitize/install-ubuntu-deps.sh scrapes the package list from rust/tools/linux-validate/Dockerfile. ci: Run Tier 2 Linux natively on the runner #967 moved the list to apt-packages.txt, so the scrape now finds nothing (no package list found in .../Dockerfile), and Triage then fails with "No results".

    What we're fixing

    • (1) is fixed in obs-util: Allow target-dependent wchar_t and c_char casts #1485. Its local cross-release lane is green. Its Fast Gate is red because of a flaky test that is already on master: callback::decl_parity::random_pieces_match_oracle drops the Rust-side log line when the whole obs-tests-core binary runs in parallel. It fails about 1 in 10 runs locally and passes when run alone. We'll look at that next.
    • (2) is ci: Install the sanitizer deps from apt-packages.txt #1487. The script reads apt-packages.txt, and a new ci/ test runs it against a stubbed apt and fails on the old script. A local asan build past the install step found one error the install failure had been hiding: test_ws_save_changes (obs-websocket: Save the scene collection after changes #812) fails to link under UBSan with undefined reference to `typeinfo for WebSocketServer'. UBSan's vptr check needs that typeinfo, and the test stubs the server, so the same branch builds that one target with -fno-sanitize=vptr. The local build also hit a GCC -Wformat-truncation false positive in libobs/util/pipe-variadic.c.h:16, with a newer GCC than the runner has. A Sanitizers run on ci: Install the sanitizer deps from apt-packages.txt #1487's head will show whether CI hits it too.

    What needs you

  4. zackees commented on Oct 21, 2026

    @zackees
    Member

    Full validation of master f1783cf, run 39691274808, completed with two failures. All native app builds (including the upgraded Flatpak action), Qt, Fast Gate and all Tier 2 OFF/ON jobs passed. This is not a full-green validation result.

    1. Release cross-lint job 119537886574: the Linux ARM64 target fails in rust/obs-util/build.rs because its target sysroot has no gio-2.0.pc. uv run ci/local_gate.py --lane cross-release reproduces this locally after the Windows ARM64 and macOS x64 checks pass. The GIO-enabled code must stay enabled; provide a proper target dependency discovery/setup rather than disabling the probe or pointing a target link at host libraries.
    2. macOS Rust job 119540175722: Application-category native_build::mixed_objective_languages_compile_load_and_rebuild_with_unity_and_pch fails its rebuild/load assertion (33 cases pass, 1 fails in that executable). This matches the Objective-C dylib lifetime fixture problem already addressed by the unmerged final head of build: preserve source options and add the macOS Cargo core graph #1669 (7c03fac), whose description explains the fresh-process correction. No duplicate fix or test relaxation is proposed here.

    Reporting these under the existing full-run incident; my active issue remains #1269's CI cache retention work. No workflow rerun, target skip, feature disablement, C test edit, or gate bypass was performed.

  5. zackees commented on Oct 22, 2026

    @zackees
    Member

    Confirmed the existing release cross-lint GIO setup failure again while qualifying #1269's opt-in cache-budget measurement path (#1681). This was one defined experiment dispatch, not a blind rerun:

    The full run remains active for independent Windows native experiments and other coverage. Their eventual results must not be called a green full run. No GIO code/check/target was disabled, no host GIO library was substituted for the target, and no fix to this setup failure is claimed by #1681. I remain on #1269 rather than claiming a parallel #1395 implementation.

  6. github-actions commented on Oct 23, 2026

    @github-actions
    Author

    The scheduled full run failed on f948723: https://github.com/obs-rust/obs-studio/actions/runs/39902551202

    PRs skip these checks (#992), so a recent merge may have broken them.

  7. zackees commented on Oct 23, 2026

    @zackees
    Member

    Triage of the 2026-10-23 05:18 failure (run 39902551202, f948723): the only failed job is Fast Gate, lane rust-test, test native_build::native_test_graph_replaces_legacy_utility_definitions. That is the pinned core cmocka count going stale when two test PRs landed (#1908). #1910 (merged after that run) removed the pin, so the next scheduled run should not repeat it. The Linux ARM64 release cross-lint GIO sysroot failure from 10-21/10-22 is separate and still open here.

  8. github-actions commented on Oct 24, 2026

    @github-actions
    Author

    The scheduled full run failed on 56732d6: https://github.com/obs-rust/obs-studio/actions/runs/40063131689

    PRs skip these checks (#992), so a recent merge may have broken them.

  9. zackees commented on Oct 24, 2026

    @zackees
    Member

    Triage of run 40063131689 (56732d6): the only failed job is Rust Tier 2 macos-26, test_context_names segfaulting in test_canvas_sources teardown. That is #2049, a use after free in obs_canvas_destroy from #2041; #2052 (queued) fixes it.

  10. github-actions commented on Oct 25, 2026

    @github-actions
    Author

    The scheduled full run failed on bd029c8: https://github.com/obs-rust/obs-studio/actions/runs/40162888607

    PRs skip these checks (#992), so a recent merge may have broken them.

  11. zackees commented on Oct 25, 2026

    @zackees
    Member

    Coordinator triage of run 40162888607 (bd029c8): three failed jobs, two causes.

  12. zackees commented on Oct 25, 2026

    @zackees
    Member

    /ready

    Status: both causes of the 10-25 failure are fixed on master: #2355 (the Qt suite count) and #2359 (the Windows PCH include order). This tracker gets a comment from the bot on every failed nightly; leave it open until the next scheduled full run passes, then close it. Preventing this class of break at PR time is #1294 part 2. P1 (master's full build), size:S.

  13. zackees commented on Oct 26, 2026

    @zackees
    Member

    Triage of run 40270992228 (2026-10-26). Every job passed except CEF Chromium Support, so both causes of 10-25's failure are fixed (#2355, #2359). The Windows x64 app build passes again.

    The one failure is the version-floor guard working as intended. ci/cef_support.py reports that macos-arm64, macos-x86_64, ubuntu-aarch64, windows-arm64 and windows-x64 still ship CEF 7871, which is older than Chrome's extended stable (Chromium 156, branch 8078). That is #1864: #2417 is the Windows cross-build (blocked on a Windows toolchain package and disk), #2418 is macOS (needs a Mac), and linux aarch64 is the remaining Ready part.

    So the nightly stays red until those archives exist. Nothing else here needs a fix. I suggest closing this tracker and letting #1864 own the remaining failure, unless you want the guard made non-blocking until then.

  14. zackees commented on Oct 27, 2026

    @zackees
    Member

    The failures originally tracked here have fixes on master: #1392, #1485/#1487, #1910, #2052, #2355, and #2359. The latest full run passed the previously failing app and test jobs; its remaining CEF Chromium Support failure is tracked by #1864 and its platform issues. Closing this incident tracker so #1864 owns the current release-floor failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Next upreadyMeets the definition of Ready; claimablesize:SHours

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions