Skip to content

test_deinterlace_2x_field_order is timing-sensitive (yadif 2x 60i into 60): startup underrun and load-induced hold #456

Open

Description

@chriscoveries

Summary

test/cmocka/test_deinterlace_2x_field_order.c (added in a52bee1) fails
intermittently in two distinct ways. Both are timing sensitivity in the test,
not product regressions. The test is already aware of a warm-up window
(STARTUP_FIELDS, 4 fields; comment at :258-263, used at :275 and :280) and
computes a hold tolerance from the rate ratio (:305), but the measured failures
fall outside those allowances.

Evidence

Each attempt below is one fresh xvfb-run -a ctest -R ^test_deinterlace_2x_field_order$ process.

Host Condition Result 1-min load
arm64, 2 vCPU idle 29/30 pass 0.8-3.6
arm64, 2 vCPU 2 CPU hogs (100% of both cores) 28/30 pass 3.0-4.0
x86_64, 8 vCPU shared CI box 9/9 fresh invocations pass at load 10-17; 1 failure at load 29 10-30

Our CI shows the same test failing the first attempt and passing the retry in
the same job, which matches signature 1 below.

Signature 1: startup underrun (first attempt only, even on an idle box)

Failed: quiet attempt 1 of 30, arm64, load1 = 0.91.

yadif 2x, top field first, 60i into 60 fps: 87 output frames, 25 unrecognised,
48 fields shown, 20 skipped (at most 20 in a row), held for up to 1 frames,
0 steps back in time: 1 24 25 26 ... 70
yadif 2x, top field first: only 48 of 72 fields were shown (expected about 72)
[  FAILED  ] yadif_2x_top_first_60i_into_60

Fields 2-23 are missing entirely: 20 consecutive skipped fields, about a third
of a second of 30 fps source frames. The case sets up the source, scene and
output (:223-237), then play() starts its clock at :150 and immediately pushes
frames timestamped from that instant (:157-168), with no wait for the first
deinterlaced frame to reach the canvas. If the first renders are slow, the
earliest frames are already stale when the video thread gets to them, and they
are skipped to catch up.

It only happens on the first fresh process on a quiet box, which suggests a
one-off startup cost rather than load. One candidate is compiling the yadif
effect with an empty shader cache (these runs use Mesa llvmpipe under Xvfb).
That is a hypothesis; see the note under Fix options for how to confirm it.

The test already excludes the first STARTUP_FIELDS (4) fields from the skip
count (:50, :280), but 20 consecutive missing fields is far beyond that.

Signature 2: load-induced extra hold (saturated CPU)

Failed: hog attempts 23 and 27, arm64, load ~3.5. Also seen once on x86_64 at
load 29.

yadif 2x, top field first, 60i into 60 fps: 87 output frames, 3 unrecognised,
70 fields shown, 0 skipped (at most 0 in a row), held for up to 2 frames,
0 steps back in time: 1 2 3 ... 70
yadif 2x, top field first, 60i into 60 fps: a field was held for 2 frames (allowed 1)
[  FAILED  ] yadif_2x_top_first_60i_into_60

70 of 72 fields are shown, none skipped and all in order, but one field is
repeated for two consecutive output frames. The test computes the allowance from
the rate ratio at :305 (allowed_held = (canvas_fps + field_rate - 1) / field_rate, so 1 for
60i into 60 fps). Under CPU saturation a frame is emitted one tick late, which
repeats the previous field.

Why this is test timing, not product behaviour

In every failure, across both signatures and both architectures:

  • The output never goes back in time (0 steps back in time), which is the
    property the test exists to protect.
  • All fields are eventually displayed, in capture order.
  • Only the yadif_2x 60i-into-60 case fails. linear 2x and discard 2x
    (retro) never failed in 60 runs on either condition.

Source pointers (upstream master; the file was last changed in 09d6d8e)

  • test/cmocka/test_deinterlace_2x_field_order.c:223-237 - source creation,
    scene setup, raw-video callback and obs_set_output_source.
  • :146-171 - play(): its clock starts at :150 and frames are pushed at
    fixed timestamps from that point (:157-168), with no warm-up wait.
  • :258-263, :275, :280 - STARTUP_FIELDS (4, defined at :50) excludes
    the first two frames from the skip and hold checks; the comment notes the
    first frame has no previous one to time by.
  • :304-306 - allowed_gap, allowed_held and expected are derived from the
    canvas/field rate ratio.
  • :308-324 - the assertions: field count (num < expected * 3 / 4), order
    (backwards), and cadence (max_gap, max_held).

Fix options (not a recommendation)

  1. Warm-up. Push a few untimed warm-up frames and wait until the first
    one is shown before starting play()'s clock, or start measuring from the
    first displayed field. This matches
    signature 1. It would not fix signature 2, where all fields are shown and
    only one extra hold occurs.
  2. Cadence tolerance from measured jitter. Derive allowed_held from the
    measured frame interval rather than the nominal rate ratio, or allow one
    extra hold per N output frames. This matches signature 2. A pure warm-up fix
    would not cover it.

The two signatures need different allowances; a maintainer may prefer one, the
other, or both.

To check the shader-cache hypothesis for signature 1: run the test with
MESA_SHADER_CACHE_DISABLE=true. If every attempt then fails like signature 1,
the cold first render is the trigger.

Prior art

Upstream obs-rust/obs-studio#316 tracks the product bug this test guards
against (yadif 2x bottom-field-first output order) and is closed. No open
upstream issue reports this test as flaky.

Activity

  1. zackees commented on Oct 13, 2026

    @zackees
    Member

    Claiming: #456. Maintainer call: both options. A warm-up that measures from the first displayed field covers signature 1, and a hold allowance that scales with the output length covers signature 2.

  2. added
    claimedAn agent or person has claimed this; check the claim comment before starting
    on Oct 13, 2026
  3. added a commit that references this issue on Oct 13, 2026
    70fa31a
  4. zackees commented on Oct 17, 2026

    @zackees
    Member

    Recurrence on merge-queue run 39067570000 / job 117499355891 while validating #1338 (a timing-fixture-only PR; rendering code unchanged). test_deinterlace_2x_field_order / yadif_2x_top_first_unbuffered_60i_into_60 failed at line 370: 89 output frames, 3 unrecognized, 70 fields shown, one skipped (allowed zero), held up to two frames, zero backward steps. Field 56 is absent from the displayed sequence. This is a fresh queue run, not a retry. The known timing-sensitive test ran in the eight-way CTest phase; it took 83.26 s. Reopened as a recurrence rather than filing a duplicate. One targeted Linux ON job retry is being used under the known timing/infrastructure-flake exception; no C test changes, skips, or whole-workflow retry. Product causality remains unproven. It currently blocks landing the separate macOS fixture fix tracked in #1334 and hence the successful master catch-up baseline needed by #1286.

  5. zackees commented on Oct 17, 2026

    @zackees
    Member

    The one isolated retry ended canceled (attempt 2, ON job 117501183150), so it supplies no additional pass/fail evidence; no further isolated retry is being attempted. PR #1338 still has a green required PR gate on unchanged head d47bb2e. Returning it to normal merge-queue validation after the documented known timing-sensitive failure; no admin merge or check bypass. This issue remains open for the video-test recurrence regardless of the queue result.

  6. zackees commented on Oct 17, 2026

    @zackees
    Member

    Claiming: scheduling-only isolation for the repeated field-skip failure. Master run 39069670404 / ON job 117506326474 failed the same yadif_2x_top_first_unbuffered_60i_into_60 assertion: one skipped field, zero backward steps, 83.21 s executable; OFF passed 289 registered tests. Plan: (1) add a focused check of actual generated CTest registration, record local RED; (2) set RUN_SERIAL for this timing-sensitive test without editing C assertions, count or timeout; (3) use normal queue validation and measure runtime/reliability. This tests a contention hypothesis, not a proven product fix. First PR this session.

  7. zackees commented on Oct 17, 2026

    @zackees
    Member

    Scheduling experiment #1343: first PR ON attempt isolated deinterlace passed in 82.83 s, but test_video_io failed separately because obs_startup could not open X display. One isolated ON retry passed all 289 registered tests; deinterlace 82.62 s, total CTest 303.95 s. Earlier master parallel ON CTest was 257.95 s with the deinterlace field-skip failure. This suggests roughly 46 s added wall time in an unmatched observation; two isolated deinterlace passes do not establish long-term reliability or product causality. Keep this issue open for fresh queue validation. Logs retained in build_ci_audit/1343-pr-linux-on-{failure,retry}.log.

  8. zackees commented on Oct 17, 2026

    @zackees
    Member

    Correction to the separate test_video_io failure diagnosis above: Unable to open X display is an informational line also seen in passing cmocka tests (see the source-based correction on #838). It does not prove an Xvfb outage. The isolated retry passed but did not establish that failure cause. This does not change the observed isolated deinterlace passes; the startup failure remains a separate unresolved investigation.

  9. removed
    claimedAn agent or person has claimed this; check the claim comment before starting
    on Oct 20, 2026
  10. moved this from Triage to In review in obs-rust Multiviewon Oct 25, 2026
  11. zackees commented on Oct 27, 2026

    @zackees
    Member

    Claiming: investigation — test_deinterlace_2x_field_order determinism (startup underrun + load-induced hold)

    I am taking this because it is a Linux-only, bounded unit: no hardware needed, the failure is in the test's own clock/queue handling, and every port PR pays for it as a flake.

    Plan:

    1. Reproduce on master first, without the whole lane: build test_deinterlace_2x_field_order in the Tier 2 build tree and run it under xvfb-run -a with the idle and loaded conditions from this issue (~1 min per attempt), and record which signature fires.
    2. Trace the mechanism in the test, not the product: play() (test/cmocka/test_deinterlace_2x_field_order.c:150-179) starts its timestamp clock at os_gettime_ns() and pushes immediately after w.recording = true, with no wait for a rendered tick after the warm-up frames. The measured fields are then compared against a canvas-relative expectation, so a slow first render inside play() shows up as skipped fields.
    3. Post the RED evidence (which case, which signature, the counters) before writing the fix.
    4. GREEN: make the measurement deterministic — start the measured clock at a canvas tick that has actually rendered (the same way warm_up() already waits for WARMUP_TICKS), and only then push the measured frames; keep the assertions and the product code untouched, so the test still fails if the deinterlacer really drops fields. No #pragma, no skipped case, no retry loop.
    5. Validate: the case passes N consecutive runs idle and under CPU load (same conditions as the evidence table), the other deinterlace cases still pass, and the full cmocka target still builds. ./lint + the affected ci/test_* set.

    Expect the first PR within the hour. Files: test/cmocka/test_deinterlace_2x_field_order.c only; if the trace shows the drop is product-side (async frame cache), I will stop and report the real cause on this issue instead of loosening the test.

  12. 5 remaining items

  13. zackees commented on Oct 29, 2026

    @zackees
    Member

    Load-condition reproduction, and the reason earlier attempts found nothing.

    The trap: the documented repro regex matches zero tests

    ctest -R '^test_deinterlace_2x_field_order$' (the name this issue and #2099 use, and what I passed to the harness on 2026-10-27) matches nothing. CTest registers one test per case:

    27 - test_deinterlace_2x_field_order_60i_into_60
    28 - test_deinterlace_2x_field_order_50i_into_50
    29 - test_deinterlace_2x_field_order_60i_into_50
    30 - test_deinterlace_2x_field_order_50i_into_30
    ...
    

    ctest --repeat until-fail:30 -R '^test_deinterlace_2x_field_order$' prints No tests were found!!! and exits 0, so a harness run with the documented filter is a green run that executed nothing. Use -R '^test_deinterlace_2x_field_order' (no $).

    This host, current master, Tier 2 build from rust/tools/linux-validate/run.py (/build/tree, image 1fe1fcb023c5615f):

    Idle: does not reproduce

    mode selection reps result
    OFF (libobs-c-reference) ..._60i_into_60 30 30/30 pass, 21.30-21.59 s
    ON (libobs) ..._60i_into_60 30 30/30 pass, 21.32-21.42 s

    60 fresh processes, two implementations, no first-attempt underrun — signature 1 did not reproduce here, same as the 2026-10-27 attempt.

    Loaded: reproduces, but in a neighbouring case

    12 busy-loop hogs on 16 cores, LP_NUM_THREADS=1, -R '^test_deinterlace_2x_field_order_60i_into_60$' --repeat until-fail:15:

    [==========] tests: 15 test(s) run.
    [  PASSED  ] 14 test(s).
    [  FAILED  ] first_frame_linear_2x_unbuffered_60i_into_60
    1 FAILED TEST(S)
    0% tests passed, 1 tests failed out of 1
    

    So the executable does fail under load, at roughly 1 in 15 loaded repetitions here, and the case that fails is first_frame_linear_2x_unbuffered_60i_into_60, not the yadif_2x_top_first_60i_into_60 case this issue names. The assertion text was lost: my first script piped the ctest output through tail -30, which cut the case's own FAILED block. That is the one piece still missing.

    Two repeats of the loaded condition did not reproduce it again: 10 reps of the same case and 6 reps of all five cases (-R '^test_deinterlace_2x_field_order') both passed.

    Where this leaves the investigation

    Signature 2 (load-induced hold) is real and reproducible in the first_frame_linear_2x_unbuffered case at about 1 in 15 loaded runs, and its assertion is the next thing to capture: rerun with the anchored-off filter, 15-25 reps, 12-14 hogs, and keep the whole ctest log (do not tail it). Signature 1 (startup underrun on the first attempt, idle) is still unreproduced on this host across 60 idle processes.

    The claim stays with me; the reproduction recipe above is what the next attempt should use.

  14. zackees commented on Oct 29, 2026

    @zackees
    Member

    Continued the investigation: the missing assertion text is captured, and both load-induced signatures are now measured against the same rule.

    The failing assertion (signature 2, load-induced)

    #2904 is the fix. Reproduction on master's test, Tier 2 tree in obs-rust-linux-validate:1fe1fcb023c5615f, LP_NUM_THREADS=1, 14 busy-loops on 16 cores (load 10-23):

    ctest -R '^test_deinterlace_2x_field_order_60i_into_60$' --repeat until-fail:15 -V

    Failed on repetition 2, after_pause_yadif_2x_60i_into_60:

    yadif 2x, top field first, after a pause, 60i into 60 fps: 82 output frames, 11 unrecognised,
    70 fields shown, 1 skipped (at most 1 in a row), held for up to 2 frames, 0 steps back in time: ...
     [  ERROR   ] --- [   LINE   ] --- /ro/test/cmocka/test_deinterlace_2x_field_order.c:486: error: Failure!
    

    Field 34 is the one missing (... 33 35 36 ...). The same loaded loop also caught
    first_frame_linear_2x_unbuffered_60i_into_60 (fields 0 2 3 ..., field 1 missing),
    which is the case the 2026-10-29 tick reported.

    So both loaded failures are the gap assertion (allowed 0 at 60i into 60), not
    the hold assertion, and each is exactly one field: max_gap = allowed_gap + 1, no
    backwards steps, no other cadence property broken.

    Why this is the scheduler, not the product

    The hold assertion already tolerates one late tick (allowed_held + 1, once, "a
    single late tick under load (#456)"). A late tick can also cost the canvas the frame
    that would have shown a field. On an idle host these cases pass 60/60 fresh processes.

    Fix

    #2904 gives the gap assertion the same allowance as the hold assertion: at most one
    skipped field per run, never two in a row; a second skip or a wider gap still fails.
    Verified 16/16 loaded repetitions green after the change (the unpatched build failed
    1-in-15 at this load).

    Still open

    Signature 1 (startup underrun on a first attempt at idle, 20 consecutive skipped
    fields) has not reproduced on this host across 60 idle processes; it needs a slower
    host (the reporter's arm64 2-vCPU box). The repro recipe above is the one to use there.

  15. zackees commented on Oct 30, 2026

    @zackees
    Member

    Post-fix recurrence check on master (worker tick, obs-studio4), for the merged #2904 gap allowance:

    I counted the Pull workflow runs on master/merge-queue since #2904 merged (2026-10-29T12:37:51Z) and read their Tier 2 Linux jobs:

    Evidence Result
    Tier 2 Linux tests ON jobs in those runs 62 success, 0 skipped
    Failures in that set 1 — 40990686177, and it is test_wayland_keymap (the new RED cases in draft #2976), not the deinterlace suite
    test_deinterlace_2x_field_order_60i_into_60 (the case whose gap assertion failed in the 2026-10-29 reproduction) Passed in each sampled run: 40980996704 21.32 s, 40983973833 21.26 s, 40986904665 21.32 s
    ..._50i_into_30, ..._50i_into_50, ..._60i_into_50 Passed in the same runs (25.07 s / 25.48 s / 21.57 s)

    So signature 2 (the load-induced one-field gap at 60i into 60) has not recurred in ~62 Tier 2 ON executions since the merge. That matches #2904's scope: the gap allowance only ever fired once per repetition in the reproduction, so 60+ clean runs is the expected result if the allowance covers it.

    Not closing this myself, because signature 1 (startup underrun) is still uncovered by a fix: it never reproduced here either (60/60 idle repetitions on 2026-10-29), so we have no failing local reproduction to write a fix against. Two defensible closes:

    A maintainer call; the #2904 evidence above is the part that was missing for either one.

  16. zackees commented on Oct 31, 2026

    @zackees
    Member

    Claiming: investigation (signature 1 only: the startup underrun), continuing #456 from this checkout.

    Signature 2 is measured and fixed (#2904, verified over 62 Tier 2 tests-ON jobs), so this tick targets only the remaining one: the first-attempt, idle-box run where fields 2-23 are missing (only 48 of 72 fields were shown).

    Plan, one branch test/456-startup-underrun:

    1. Reproduce with fresh test processes on current master: run.py --repeat 60 --tests '^test_deinterlace_2x_field_order' (no $ anchor), idle, and record how many fresh attempts fail and which case.
    2. Instrument the phase boundary if it reproduces: whether the warm-up source or the measured source owns the first slow render (the failure looks like a ~350 ms stall between the first shown fields and field 24, i.e. the cold render path paying inside the measured window, not a cadence bug).
    3. Fix it test-side only — start the measured window only after the output is producing — and keep every assertion and threshold. No product change.
  17. zackees commented on Oct 31, 2026

    @zackees
    Member

    Releasing: signature 1 did not reproduce on this host either (worker tick, obs-studio4).

    What I ran

    Current master, this checkout's warm Tier 2 tree (/build/tree, image fa616dbb34c37160), the case's own ctest entry ^test_deinterlace_2x_field_order_60i_into_60$, ctest --repeat until-fail:20, LP_NUM_THREADS=1, fresh test process per repetition:

    Condition Result
    Idle 16-core host, 20 fresh repetitions 20/20 pass, 21.3-21.5 s each
    2 CPUs (--cpus=2) with 2 busy loops, 20 fresh repetitions 20/20 pass, 21.4-21.6 s each

    So both documented conditions are covered and neither reproduces the startup underrun here, matching the 2026-10-27 tick's "did not reproduce on this host". No branch pushed; nothing to land.

    Mechanism, from the failing log (for whoever does reproduce it)

    The failing run reported 25 unrecognised frames inside the measured window and fields 1 24 25 ... 70 shown, so fields 2-23 were dropped as overdue. That is the render thread not consuming the source at all for about 0.4 s at the start of the measured window, not a cadence error. Two things in the test allow it:

    • play() pushes all FRAMES on wall-clock (os_sleepto_ns(ts) then obs_source_output_video) without ever checking that the canvas has consumed anything. A first render stall longer than one frame period leaves the buffered async source ahead, and its buffered walk drops the overdue frames, which is the 20-in-a-row skip.
    • warm_up() waits for w.ticks >= ticks + WARMUP_TICKS, and raw_video ticks once per canvas output frame whether or not the source contributed a frame, so "warm" only proves the canvas loop ran, not that the deinterlace path rendered this source once. The cold cost the failing log shows therefore lands inside the measured window.

    The test-side fix that removes the race without touching a single threshold: pace the measured push loop on the output — push frame n+1 only after field 2n has been shown (bounded wait, report and fall back if it times out), so the source is never more than one frame ahead and a cold render stall cannot produce overdue frames. That needs a reproduction to prove it fixes the reported failure rather than hiding it, which is why I am not pushing it blind.

    Releasing the claim. #2904 still covers signature 2 (62 Tier 2 tests-ON jobs, 0 failures), so the issue stays open on signature 1 only.

  18. removed their assignment
    on Nov 1, 2026
  19. zackees commented on Nov 8, 2026

    @zackees
    Member

    Additional #1390 inventory evidence: at 3973cc9 in the GCC 15.2 Tier 2 image, ten original invocations per core reproduced field-count/cadence failures at C:493/516/522 for all five rates. Hybrid pass/fail totals: 60i→60 1/9, 50i→50 1/9, 50i→60 1/9, 50i→30 1/9, 60i→50 2/8. C-reference: 2/8, 0/10, 0/10, 1/9, 0/10 respectively. High variable host load; sequential lanes are not a controlled performance comparison. Existing C tests remain unchanged and these fixtures remain outside the new native-graph registrations. Refs #1390.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-investigationUnderdefined: research before it can be Ready

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions