Skip to content

Flaky: real-time audio cmocka tests lose audio under machine load (test_vlc_playlist_gap, test_group_hide_transition_audio) #838

Open

Description

@zackees

Two Tier 2 tests that mix real-time audio failed once each on 2026-10-15 in local rust/tools/linux-validate/run.sh runs on a loaded shared machine, then passed on rerun with no code change. Neither run touched the code under test (#828/#832 and #833/#835).

test_vlc_playlist_gap (ENABLE_RUST_LIBOBS=ON)

playlist_of_three_plays_every_file_to_the_end, test_vlc_playlist_gap.c:370:

debug: Audio timestamp for 'vlc' exceeded TS_SMOOTHING_THRESHOLD, diff=128596000 ns, expected 18243622000, input 18372218000
file 2: 3.008 s of 3.0 s in the mix, heard from 0.014 s to 1.114 s of the file, mix 3.093 s - 6.229 s
file 2: the last 1.886 s are missing from the mix
playlist track boundary cut audio

test_group_hide_transition_audio (ENABLE_RUST_LIBOBS=OFF)

All four cases failed with no program audio was mixed (test_group_hide_transition_audio.c:216 first).

Likely cause

Both tests play audio in real time and assert on what reached the mix. When the process is starved (another Docker build on the host), a ~130 ms timestamp jump past TS_SMOOTHING_THRESHOLD or a late audio thread drops audio, and the assertion fails. That is the test environment, not the code under test.

Options

  • make the tests drive the audio clock instead of wall time, or
  • widen what they accept for a timestamp jump (and assert the jump was logged), or
  • run them in a lane that is not shared with parallel builds.

Running them under ctest --repeat until-fail:N while another build loads the machine should reproduce it.

Activity

  1. zackees commented on Oct 15, 2026

    @zackees
    MemberAuthor

    Two more one-off failures in a loaded local Tier 2 run (OFF build, 2026-10-15 ~22:50Z, unrelated change #918):

    • test_async_delay_audio (test_async_delay_audio.c:466), another real-time audio test.
    • test_v4l2_busy_signal force_camera_closes_the_holder_and_opens_the_camera (test_v4l2_busy_signal.c:296, 10185 != 0): the forced holder process had not been reaped yet when the test checked. Not audio, but the same pattern: a wall-clock race that loses under load.
  2. zackees commented on Oct 16, 2026

    @zackees
    MemberAuthor

    Another one, video this time: test_deinterlace_2x_field_order (discard_2x_bottom_first_60i_into_50, yadif_2x_top_first_unbuffered_60i_into_60; test_deinterlace_2x_field_order.c:370) failed once in a loaded local Tier 2 run (OFF build, 2026-10-16 ~01:50Z, unrelated change 965): 12 fields skipped, up to 2 in a row (allowed 1). Same wall-clock pattern: real-time frame cadence loses under machine load.

  3. zackees commented on Oct 16, 2026

    @zackees
    MemberAuthor

    Another one: test_vlc_playlist_gap failed (28.6 s, ***Failed) in Tier 2 Linux (ON) on the merge-queue run for #936 (job 116819150784, 2026-10-16 05:13 UTC). #936 is a text-freetype2 refactor and touches nothing in vlc-video. It knocked #936 out of the queue; I re-queued it.

  4. zackees commented on Oct 16, 2026

    @zackees
    MemberAuthor

    Add test_media_restart_stale_frame to the list: failed once in a heavily loaded local Tier 2 run (OFF, 2026-10-16 ~05:50Z, unrelated change 997), together with test_async_delay_audio and test_deinterlace_2x_field_order. Separately, test_v4l2_busy_signal (also logged here earlier) was a fixture race rather than load sensitivity; #953 / 997 fixes it.

  5. zackees commented on Oct 16, 2026

    @zackees
    MemberAuthor

    These flakes are now costing merges: #936 (text-freetype2, no media code) was removed from the merge queue twice for failed_checks. The 05:19Z removal was test_vlc_playlist_gap stop_at_track_end_is_prompt (test_vlc_playlist_gap.c:476) in merge-group run 38850208674, Rust Tier 2 Linux (ON). The queue's Tier 2 runs every real-time test against a busy runner, so each of these is a coin flip on every merge.

  6. zackees commented on Oct 16, 2026

    @zackees
    MemberAuthor

    test_vlc_playlist_gap stop_at_track_end_is_prompt is not load sensitivity: it is a real race in vlc-video (an async libVLC Opening event resets drain_abort after Stop signals it), filed and claimed as 1011. The other cases here (audio/video cadence under load) are still open.

  7. zackees commented on Oct 16, 2026

    @zackees
    MemberAuthor

    test_media_restart_stale_frame clear_still_clears is a crash, not a timing flake: obs_shutdown() segfaults after obs_reset_video() failed ("Unable to open X display"). Filed as 1032.

  8. zackees commented on Oct 16, 2026

    @zackees
    MemberAuthor

    A common cause for part of this list: in loaded local Tier 2 runs (2026-10-16 ~07:30Z and ~08:10Z), single cases fail setup with info: Unable to open X display while the neighbouring cases in the same binary pass, e.g. test_v4l2_busy_signal force_camera_opens_while_the_holder_is_still_exiting ("Could not run test: Test setup failed") and test_media_restart_stale_frame (which then crashes in obs_shutdown, #1032). So the Xvfb server from xvfb-run -a intermittently isn't reachable while ctest runs tests in parallel (-j4). That's an environment fault, not load sensitivity in the tested code; worth fixing in the harness (one Xvfb per test, or retry on display-open failure) alongside #982. Separately, test_vlc_playlist_gap stop_at_track_end_is_prompt was a real vlc-video race: #1011, fixed in #1037.

  9. zackees commented on Oct 16, 2026

    @zackees
    MemberAuthor

    Correction to my earlier comment: "Unable to open X display" is not evidence that Xvfb was unreachable. It is printed by obs_nix_x11_log_info() (libobs/obs-nix-x11.c:39) whenever obs_get_nix_platform_display() is NULL, i.e. whenever nothing called obs_set_nix_platform_display(), which only the frontend does. Every cmocka test that calls obs_startup logs it, including passing ones (the next test_v4l2_busy_signal case logged it and then registered X11 hotkey keycodes fine). So the "Xvfb dropping out under parallel ctest" explanation I gave is wrong and should be disregarded; the cause of these failures is still unknown. For the list here: test_v4l2_busy_signal's "Test setup failed" case and test_media_restart_stale_frame's segfault both still need a real cause.

  10. zackees commented on Oct 16, 2026

    @zackees
    MemberAuthor

    Another hit, 2026-10-16 19:40 UTC, in the merge queue: test_vlc_playlist_gap / playlist_of_three_plays_every_file_to_the_end failed in Rust Tier 2 Linux (ON 2/3) of run 38956680388 (merge group for #1111, a linux-pipewire change unrelated to VLC). Log: Audio timestamp for 'vlc' exceeded TS_SMOOTHING_THRESHOLD, diff=127413000 ns, then file 1: the last 1.056 s are missing from the mix. The OFF lane passed. This kicked #1111 out of the queue; it's re-queued.

  11. zackees commented on Oct 17, 2026

    @zackees
    MemberAuthor

    Another one: test_media_seek_keyframe / playing_seek_between_keyframes failed at test_media_seek_keyframe.c:629 in Rust Tier 2 macos-26 on the merge-queue run for #1292 (run 39055672620). #1292 only touches plugins/frontend-tools, so this is unrelated to it. Real-time media playback under queue load, like the others here.

  12. zackees commented on Oct 17, 2026

    @zackees
    MemberAuthor

    Another observed instance: master run 39078519174, OFF job 117532901565, failed test_group_hide_transition_audio (9.14 s). Three subcases (hide_child_with_transitions, hide_group_with_child_transitions, hide_group_during_child_show_transition) reported no program audio was mixed, failing line 216 after the fixed one-second mix observation window. 293 CTest registrations; one executable failed.

    OBS startup succeeded, created the main canvas and initialized llvmpipe OpenGL. This is separate from #1355's locally reproduced XOpenDisplay/startup failure. The informational Unable to open X display message precedes successful startup here too. No new Xvfb attribution, skip, assertion change or blind rerun was made. Raw log is retained under ignored build_ci_audit/master-e89-off-failure.log during the #1286 audit.

  13. zackees commented on Oct 18, 2026

    @zackees
    MemberAuthor

    Additional live failure in master run 39084293679, revision d27d69f585eec876b3d90a18b74ca69194422cba, OFF job 117549418999: unchanged test_group_hide_transition_audio failed all four cases at line 216 with no program audio was mixed. Suite registered 295 tests, one failing executable. Startup-failure diagnostics passed; the run failed this OFF job. No retry, skip or assertion relaxation.

    This recurs after a different full manual run (39082733551) passed both modes. A green full run therefore does not establish absence of this intermittent integrity failure. This is separate from #1355's verified Xvfb startup fix and #1283's recording-shutdown state race. Logs retained locally at build_ci_audit/master-1373-off-failure-117549418999.log while continuing #1286 cache validation. No cache-layer root cause is claimed.

  14. zackees commented on Oct 18, 2026

    @zackees
    MemberAuthor

    Claiming: investigate only the test_group_hide_transition_audio scheduling portion while completing #1286. Compare the unchanged executable in isolated versus concurrent execution on the existing local Linux build, preserve all C assertions, and record skips/environment failures separately. If resource isolation is supported by reproduction, add focused scheduling regression coverage and a narrow CTest change; do not blanket-serialize media tests or close the unrelated VLC/seek cases.

  15. 31 remaining items

  16. moved this from Needs decision to In progress in obs-rust Multiviewon Oct 30, 2026
  17. zackees commented on Oct 30, 2026

    @zackees
    MemberAuthor

    Part 1 landed: #3000 (squash-merged) re-anchors program_level()'s wait on the mix's own clock. The window keeps its original anchors, so it still measures the same 400 ms of program audio; what changed is that it waits for the mix to reach the window's end (bounded at 5 s) instead of sleeping 1000 ms.

    Evidence in the PR, from the Tier 2 harness on a host at load average 34: 1.00 on --repeat until-fail:100 for this test with ENABLE_RUST_LIBOBS=OFF and ON, and the unmodified 623-test suite green in both modes. The first attempt on that branch opened the window on the first arriving frame and the 100-repeat run caught a finished show transition inside it (0.89 of the tone) at 14/100, so the anchors stayed as they are.

    Still mine, next in this order per the decision: part 2 test_async_delay_audio listen(), part 3 test_vlc_playlist_gap (deterministic media clock).

  18. zackees commented on Oct 30, 2026

    @zackees
    MemberAuthor

    Claiming: test_async_delay_audio determinism (part 2 of #838)

    Part 1 (test_group_hide_transition_audio, #3000) landed; this is the second of the three PRs the Decided: on this issue names.

    1. Re-anchor listen() (test/cmocka/test_async_delay_audio.c:427-441) on the mix's own clock: it records the first program-audio packet's data->timestamp after listening starts and stops after LISTEN_MS of mixed audio has gone past, bounded by a deadline, instead of sleeping LISTEN_MS of wall time.
    2. Every assertion and threshold stays (samples >= 90% of RATE * LISTEN_MS, the tone-break and click-delay checks); on a starved machine the measurement waits instead of recording a shorter window.
    3. Test-timing only, no product code. RED is the unmodified test under taskset plus a parallel build, GREEN is --repeat until-fail:100 on both ENABLE_RUST_LIBOBS=ON and OFF, in the warm Tier 2 volume.
    4. Part 3 (test_vlc_playlist_gap, deterministic media clock) follows in its own PR.
  19. zackees commented on Oct 30, 2026

    @zackees
    MemberAuthor

    Worker tick (obs-studio4): part 2 measured, and the decided approach does not match this test's failure. Reporting before writing code, and releasing the claim.

    What I ran

    Warm Tier 2 volume, ctest -R ^test_async_delay_audio$ under xvfb-run, on this checkout's branch off 08749f47c (master). I built both variants in the same volume, so only the test source varies.

    Stressor Variant Runs Failures
    --cpus=2, 3 busy loops unmodified master 3 0
    --cpus=1, 4 busy loops unmodified master 3 1 (tone_continuous_with_3s_video_delay)
    --cpus=1, 3 busy loops, test at nice -n 19 unmodified master 3 1 (same case)
    --cpus=0.15, no hogs unmodified master 2 0
    --cpus=1, 3 busy loops re-anchored listen() (below) 3 1 (same case)

    Failing output, both times identical in shape:

    video delay 3000 ms: 192512 samples heard, 120321 loud, tone broken 1 times, mix skipped 0 times
    video delay 3000 ms: the tone was broken up 1 times in 4000 ms
    

    Why this is not the part-1 class

    • 192512 samples over RATE = 48000 is 4.01 s of mix, i.e. the whole LISTEN_MS window arrived; the samples < 90% assertion never fired in any of the 11 local runs. So the wall-clock window is not what truncates this test.
    • The case that fails is the upstream-13830 assertion, not the "did not run" one: exactly one discontinuity inside the tone, with a full-length window.
    • I implemented the part-1 shape anyway to test the hypothesis: listen() records the first program packet's data->timestamp and waits for LISTEN_MS of mix time (bounded +5 s) instead of sleeping wall time, and the raw-audio callback primes the sine check again when the mix jumps more than one packet. Under the same stressor it still fails on the same case, and mix skipped 0 times says there is no timestamp skip to attribute the discontinuity to. The mix genuinely contains one gap.

    So a test-side clock change does not remove this flake, and shipping it as "part 2" would be a PR that claims a fix the evidence does not support.

    What the choice is

    1. Treat the one-off gap as the environment — accept at most one isolated tone discontinuity per 4 s window (the original #13830 bug broke the tone repeatedly, so a bound of 1 keeps the guard), and report the count in the failure text. This is the Decided: "test-side determinism" intent applied to the assertion that actually fires.
    2. Treat it as product audio loss under CPU starvation — then the fix is in the mixer/delay path, not the test, and Flaky: real-time audio cmocka tests lose audio under machine load (test_vlc_playlist_gap, test_group_hide_transition_audio) #838's remaining scope changes.

    I do not think a worker should pick either unilaterally: option 1 widens an assertion that exists to catch a real upstream bug, and the Decided: above only covers the wall-clock window.

    Releasing: test_async_delay_audio determinism (part 2 of #838) — no branch pushed, nothing left in flight from this tick. Part 3 (test_vlc_playlist_gap, deterministic media clock) is untouched and can be taken independently if you would rather defer this one.

  20. zackees commented on Oct 30, 2026

    @zackees
    MemberAuthor

    Releasing: test_async_delay_audio determinism (part 2 of #838)

    No branch pushed; see the measurement above. Happy to take it back once the assertion-vs-product choice is made.

  21. zackees commented on Oct 31, 2026

    @zackees
    MemberAuthor

    Claiming: test_vlc_playlist_gap determinism (part 3 of #838)

    Part 1 (#3000) and the part-2 measurement are done; this is the last of the three PRs the Decided: names. Taking it as an investigation first, because part 2 showed the decided shape can miss the actual failure:

    1. Reproduce test_vlc_playlist_gap in this checkout's warm Tier 2 volume under CPU starvation (--cpus=1 plus hogs, ctest --repeat until-fail:N), and record which of the two mechanisms fires: libVLC genuinely delivering fewer decoded frames (the Decided: comment's expectation) or the test mis-attributing samples when data->timestamp jumps past TS_SMOOTHING_THRESHOLD.
    2. Only then write the test-side change: drive file position from the mix's own sample sequence if the samples are all there, and keep every assertion and threshold. That is the deterministic media clock the decision asks for, and it is test-timing only, with ENABLE_RUST_LIBOBS=OFF as the evidence.
    3. RED is the unmodified test failing under the stressor; GREEN is --repeat until-fail:100 under the same stressor on both ENABLE_RUST_LIBOBS=ON and OFF. If the reproduction shows libVLC really dropping audio, I will report that and release instead of widening a tolerance, the same way part 2 was handled.

    No branch pushed yet; first PR this session.

  22. zackees commented on Oct 31, 2026

    @zackees
    MemberAuthor

    Worker tick (obs-studio4): part 3 measured, and the deterministic-clock approach has no lever in this test. Reporting before writing code, and releasing the claim.

    What I ran

    Warm Tier 2 volume for this checkout (obs-rust-linux-validate-build-e829c156d95d), current binary from 2026-10-30, host load 10-13 from the other workers, ctest -R '^test_vlc_playlist_gap$' --repeat until-fail:N -V under xvfb-run, container CPU capped with --cpus:

    Container cap Repeats Failures Timestamp jumps logged
    --cpus=1 8 0 9
    --cpus=0.25 4 0 several

    (The 8-repeat run: 244 s total, 30.55 s per repeat, every repeat [ PASSED ] 8 test(s).)

    So no RED, even at a quarter of a core, which is harsher than the 2-vCPU arm64 condition in the issue.

    Why the decided shape does not fit this test

    test_vlc_playlist_gap's measurement is already clock-free: analyze() (test/cmocka/test_vlc_playlist_gap.c:185-235) classifies each 1024-frame mix block by its peak level, derives the position inside the file from that level, and builds the per-file stats from the sequence of blocks, only using cap->mix_frames as an index. data->timestamp is never read. That is why the 129 ms TS_SMOOTHING_THRESHOLD jumps happen in almost every run here and the case still passes - the log lines are not what the assertions see.

    The failure in the issue (file 2: the last 1.886 s are missing from the mix) is therefore blocks that never arrived, not blocks attributed to the wrong file. Making the media clock deterministic from the test side cannot invent audio libVLC did not deliver, and the only way to stop libVLC being real-time is to stop playing through libVLC, which would stop testing the product path.

    The loss mechanism the Decided: comment names ("a starved media thread genuinely delivers fewer frames") is consistent with everything above; my runs did not reach it, so I have no RED to gate a change against, and per the policy in the same comment I am not widening TOLERANCE without one.

    Suggested next step for the maintainer

    • (b) as written in the 2026-10-28 escalation: RUN_SERIAL (or a lane that is not shared with parallel builds) for this real-time binary - the cost is wall time on the gate, but it is the only lever that touches the actual mechanism.
    • Or accept the flake rate and reopen on the next occurrence; 11/11 under load is not evidence that 2026-10-15 cannot recur, only that it is rare enough that a stress loop does not find it here.
    • If a RED is wanted first, the condition to reproduce still looks like I/O + CPU (libVLC demux/decode on a starved filesystem), which neither of my runs applied; a --device-read-bps cap on the container would be the next thing to try.

    No branch pushed. Releasing: test_vlc_playlist_gap determinism (part 3 of #838).

  23. removed their assignment
    on Nov 1, 2026
  24. moved this from In progress to Ready in obs-rust Multiviewon Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal priorityreadyMeets the definition of Ready; claimablesize:MAbout a day

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions