Repository navigation
Description
Summary
test/cmocka/test_deinterlace_2x_field_order.c (added in a52bee1) fails
intermittently in two distinct ways. Both are timing sensitivity in the test,
not product regressions. The test is already aware of a warm-up window
(STARTUP_FIELDS, 4 fields; comment at :258-263, used at :275 and :280) and
computes a hold tolerance from the rate ratio (:305), but the measured failures
fall outside those allowances.
Evidence
Each attempt below is one fresh xvfb-run -a ctest -R ^test_deinterlace_2x_field_order$ process.
| Host | Condition | Result | 1-min load |
|---|---|---|---|
| arm64, 2 vCPU | idle | 29/30 pass | 0.8-3.6 |
| arm64, 2 vCPU | 2 CPU hogs (100% of both cores) | 28/30 pass | 3.0-4.0 |
| x86_64, 8 vCPU | shared CI box | 9/9 fresh invocations pass at load 10-17; 1 failure at load 29 | 10-30 |
Our CI shows the same test failing the first attempt and passing the retry in
the same job, which matches signature 1 below.
Signature 1: startup underrun (first attempt only, even on an idle box)
Failed: quiet attempt 1 of 30, arm64, load1 = 0.91.
yadif 2x, top field first, 60i into 60 fps: 87 output frames, 25 unrecognised,
48 fields shown, 20 skipped (at most 20 in a row), held for up to 1 frames,
0 steps back in time: 1 24 25 26 ... 70
yadif 2x, top field first: only 48 of 72 fields were shown (expected about 72)
[ FAILED ] yadif_2x_top_first_60i_into_60
Fields 2-23 are missing entirely: 20 consecutive skipped fields, about a third
of a second of 30 fps source frames. The case sets up the source, scene and
output (:223-237), then play() starts its clock at :150 and immediately pushes
frames timestamped from that instant (:157-168), with no wait for the first
deinterlaced frame to reach the canvas. If the first renders are slow, the
earliest frames are already stale when the video thread gets to them, and they
are skipped to catch up.
It only happens on the first fresh process on a quiet box, which suggests a
one-off startup cost rather than load. One candidate is compiling the yadif
effect with an empty shader cache (these runs use Mesa llvmpipe under Xvfb).
That is a hypothesis; see the note under Fix options for how to confirm it.
The test already excludes the first STARTUP_FIELDS (4) fields from the skip
count (:50, :280), but 20 consecutive missing fields is far beyond that.
Signature 2: load-induced extra hold (saturated CPU)
Failed: hog attempts 23 and 27, arm64, load ~3.5. Also seen once on x86_64 at
load 29.
yadif 2x, top field first, 60i into 60 fps: 87 output frames, 3 unrecognised,
70 fields shown, 0 skipped (at most 0 in a row), held for up to 2 frames,
0 steps back in time: 1 2 3 ... 70
yadif 2x, top field first, 60i into 60 fps: a field was held for 2 frames (allowed 1)
[ FAILED ] yadif_2x_top_first_60i_into_60
70 of 72 fields are shown, none skipped and all in order, but one field is
repeated for two consecutive output frames. The test computes the allowance from
the rate ratio at :305 (allowed_held = (canvas_fps + field_rate - 1) / field_rate, so 1 for
60i into 60 fps). Under CPU saturation a frame is emitted one tick late, which
repeats the previous field.
Why this is test timing, not product behaviour
In every failure, across both signatures and both architectures:
- The output never goes back in time (
0 steps back in time), which is the
property the test exists to protect. - All fields are eventually displayed, in capture order.
- Only the yadif_2x 60i-into-60 case fails.
linear 2xanddiscard 2x
(retro) never failed in 60 runs on either condition.
Source pointers (upstream master; the file was last changed in 09d6d8e)
test/cmocka/test_deinterlace_2x_field_order.c:223-237- source creation,
scene setup, raw-video callback andobs_set_output_source.:146-171-play(): its clock starts at :150 and frames are pushed at
fixed timestamps from that point (:157-168), with no warm-up wait.:258-263,:275,:280-STARTUP_FIELDS(4, defined at :50) excludes
the first two frames from the skip and hold checks; the comment notes the
first frame has no previous one to time by.:304-306-allowed_gap,allowed_heldandexpectedare derived from the
canvas/field rate ratio.:308-324- the assertions: field count (num < expected * 3 / 4), order
(backwards), and cadence (max_gap,max_held).
Fix options (not a recommendation)
- Warm-up. Push a few untimed warm-up frames and wait until the first
one is shown before startingplay()'s clock, or start measuring from the
first displayed field. This matches
signature 1. It would not fix signature 2, where all fields are shown and
only one extra hold occurs. - Cadence tolerance from measured jitter. Derive
allowed_heldfrom the
measured frame interval rather than the nominal rate ratio, or allow one
extra hold per N output frames. This matches signature 2. A pure warm-up fix
would not cover it.
The two signatures need different allowances; a maintainer may prefer one, the
other, or both.
To check the shader-cache hypothesis for signature 1: run the test with
MESA_SHADER_CACHE_DISABLE=true. If every attempt then fails like signature 1,
the cold first render is the trigger.
Prior art
Upstream obs-rust/obs-studio#316 tracks the product bug this test guards
against (yadif 2x bottom-field-first output order) and is closed. No open
upstream issue reports this test as flaky.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsIn review
Activity
zackees commentedon Oct 13, 2026
Claiming: #456. Maintainer call: both options. A warm-up that measures from the first displayed field covers signature 1, and a hold allowance that scales with the output length covers signature 2.
test: Warm up and allow one late hold in the deinterlace 2x field ord…
zackees commentedon Oct 17, 2026
Recurrence on merge-queue run 39067570000 / job 117499355891 while validating #1338 (a timing-fixture-only PR; rendering code unchanged). test_deinterlace_2x_field_order / yadif_2x_top_first_unbuffered_60i_into_60 failed at line 370: 89 output frames, 3 unrecognized, 70 fields shown, one skipped (allowed zero), held up to two frames, zero backward steps. Field 56 is absent from the displayed sequence. This is a fresh queue run, not a retry. The known timing-sensitive test ran in the eight-way CTest phase; it took 83.26 s. Reopened as a recurrence rather than filing a duplicate. One targeted Linux ON job retry is being used under the known timing/infrastructure-flake exception; no C test changes, skips, or whole-workflow retry. Product causality remains unproven. It currently blocks landing the separate macOS fixture fix tracked in #1334 and hence the successful master catch-up baseline needed by #1286.
zackees commentedon Oct 17, 2026
The one isolated retry ended canceled (attempt 2, ON job 117501183150), so it supplies no additional pass/fail evidence; no further isolated retry is being attempted. PR #1338 still has a green required PR gate on unchanged head d47bb2e. Returning it to normal merge-queue validation after the documented known timing-sensitive failure; no admin merge or check bypass. This issue remains open for the video-test recurrence regardless of the queue result.
zackees commentedon Oct 17, 2026
Claiming: scheduling-only isolation for the repeated field-skip failure. Master run 39069670404 / ON job 117506326474 failed the same yadif_2x_top_first_unbuffered_60i_into_60 assertion: one skipped field, zero backward steps, 83.21 s executable; OFF passed 289 registered tests. Plan: (1) add a focused check of actual generated CTest registration, record local RED; (2) set RUN_SERIAL for this timing-sensitive test without editing C assertions, count or timeout; (3) use normal queue validation and measure runtime/reliability. This tests a contention hypothesis, not a proven product fix. First PR this session.
zackees commentedon Oct 17, 2026
Scheduling experiment #1343: first PR ON attempt isolated deinterlace passed in 82.83 s, but test_video_io failed separately because obs_startup could not open X display. One isolated ON retry passed all 289 registered tests; deinterlace 82.62 s, total CTest 303.95 s. Earlier master parallel ON CTest was 257.95 s with the deinterlace field-skip failure. This suggests roughly 46 s added wall time in an unmatched observation; two isolated deinterlace passes do not establish long-term reliability or product causality. Keep this issue open for fresh queue validation. Logs retained in build_ci_audit/1343-pr-linux-on-{failure,retry}.log.
zackees commentedon Oct 17, 2026
Correction to the separate test_video_io failure diagnosis above: Unable to open X display is an informational line also seen in passing cmocka tests (see the source-based correction on #838). It does not prove an Xvfb outage. The isolated retry passed but did not establish that failure cause. This does not change the observed isolated deinterlace passes; the startup failure remains a separate unresolved investigation.
zackees commentedon Oct 27, 2026
Claiming: investigation — test_deinterlace_2x_field_order determinism (startup underrun + load-induced hold)
I am taking this because it is a Linux-only, bounded unit: no hardware needed, the failure is in the test's own clock/queue handling, and every port PR pays for it as a flake.
Plan:
test_deinterlace_2x_field_orderin the Tier 2 build tree and run it underxvfb-run -awith the idle and loaded conditions from this issue (~1 min per attempt), and record which signature fires.play()(test/cmocka/test_deinterlace_2x_field_order.c:150-179) starts its timestamp clock atos_gettime_ns()and pushes immediately afterw.recording = true, with no wait for a rendered tick after the warm-up frames. The measured fields are then compared against a canvas-relative expectation, so a slow first render insideplay()shows up as skipped fields.warm_up()already waits forWARMUP_TICKS), and only then push the measured frames; keep the assertions and the product code untouched, so the test still fails if the deinterlacer really drops fields. No#pragma, no skipped case, no retry loop../lint+ the affectedci/test_*set.Expect the first PR within the hour. Files:
test/cmocka/test_deinterlace_2x_field_order.conly; if the trace shows the drop is product-side (async frame cache), I will stop and report the real cause on this issue instead of loosening the test.5 remaining items
zackees commentedon Oct 29, 2026
Load-condition reproduction, and the reason earlier attempts found nothing.
The trap: the documented repro regex matches zero tests
ctest -R '^test_deinterlace_2x_field_order$'(the name this issue and #2099 use, and what I passed to the harness on 2026-10-27) matches nothing. CTest registers one test per case:ctest --repeat until-fail:30 -R '^test_deinterlace_2x_field_order$'printsNo tests were found!!!and exits 0, so a harness run with the documented filter is a green run that executed nothing. Use-R '^test_deinterlace_2x_field_order'(no$).This host, current master, Tier 2 build from
rust/tools/linux-validate/run.py(/build/tree, image1fe1fcb023c5615f):Idle: does not reproduce
libobs-c-reference)..._60i_into_60libobs)..._60i_into_6060 fresh processes, two implementations, no first-attempt underrun — signature 1 did not reproduce here, same as the 2026-10-27 attempt.
Loaded: reproduces, but in a neighbouring case
12 busy-loop hogs on 16 cores,
LP_NUM_THREADS=1,-R '^test_deinterlace_2x_field_order_60i_into_60$' --repeat until-fail:15:So the executable does fail under load, at roughly 1 in 15 loaded repetitions here, and the case that fails is
first_frame_linear_2x_unbuffered_60i_into_60, not theyadif_2x_top_first_60i_into_60case this issue names. The assertion text was lost: my first script piped the ctest output throughtail -30, which cut the case's own FAILED block. That is the one piece still missing.Two repeats of the loaded condition did not reproduce it again: 10 reps of the same case and 6 reps of all five cases (
-R '^test_deinterlace_2x_field_order') both passed.Where this leaves the investigation
Signature 2 (
load-induced hold) is real and reproducible in thefirst_frame_linear_2x_unbufferedcase at about 1 in 15 loaded runs, and its assertion is the next thing to capture: rerun with the anchored-off filter, 15-25 reps, 12-14 hogs, and keep the whole ctest log (do nottailit). Signature 1 (startup underrun on the first attempt, idle) is still unreproduced on this host across 60 idle processes.The claim stays with me; the reproduction recipe above is what the next attempt should use.
zackees commentedon Oct 29, 2026
Continued the investigation: the missing assertion text is captured, and both load-induced signatures are now measured against the same rule.
The failing assertion (signature 2, load-induced)
#2904 is the fix. Reproduction on master's test, Tier 2 tree in
obs-rust-linux-validate:1fe1fcb023c5615f,LP_NUM_THREADS=1, 14 busy-loops on 16 cores (load 10-23):ctest -R '^test_deinterlace_2x_field_order_60i_into_60$' --repeat until-fail:15 -VFailed on repetition 2,
after_pause_yadif_2x_60i_into_60:Field 34 is the one missing (
... 33 35 36 ...). The same loaded loop also caughtfirst_frame_linear_2x_unbuffered_60i_into_60(fields0 2 3 ..., field 1 missing),which is the case the 2026-10-29 tick reported.
So both loaded failures are the gap assertion (
allowed 0at 60i into 60), notthe hold assertion, and each is exactly one field:
max_gap = allowed_gap + 1, nobackwards steps, no other cadence property broken.
Why this is the scheduler, not the product
The hold assertion already tolerates one late tick (
allowed_held + 1, once, "asingle late tick under load (#456)"). A late tick can also cost the canvas the frame
that would have shown a field. On an idle host these cases pass 60/60 fresh processes.
Fix
#2904 gives the gap assertion the same allowance as the hold assertion: at most one
skipped field per run, never two in a row; a second skip or a wider gap still fails.
Verified 16/16 loaded repetitions green after the change (the unpatched build failed
1-in-15 at this load).
Still open
Signature 1 (startup underrun on a first attempt at idle, 20 consecutive skipped
fields) has not reproduced on this host across 60 idle processes; it needs a slower
host (the reporter's arm64 2-vCPU box). The repro recipe above is the one to use there.
test(deinterlace): allow one skipped field for a single late tick (#456)
test(deinterlace): count skips past the cadence, not fields skipped (#…
zackees commentedon Oct 30, 2026
Post-fix recurrence check on master (worker tick,
obs-studio4), for the merged #2904 gap allowance:I counted the
Pullworkflow runs on master/merge-queue since #2904 merged (2026-10-29T12:37:51Z) and read their Tier 2 Linux jobs:40990686177, and it istest_wayland_keymap(the new RED cases in draft #2976), not the deinterlace suitetest_deinterlace_2x_field_order_60i_into_60(the case whose gap assertion failed in the 2026-10-29 reproduction)4098099670421.32 s,4098397383321.26 s,4098690466521.32 s..._50i_into_30,..._50i_into_50,..._60i_into_50So signature 2 (the load-induced one-field gap at 60i into 60) has not recurred in ~62 Tier 2 ON executions since the merge. That matches #2904's scope: the gap allowance only ever fired once per repetition in the reproduction, so 60+ clean runs is the expected result if the allowance covers it.
Not closing this myself, because signature 1 (startup underrun) is still uncovered by a fix: it never reproduced here either (60/60 idle repetitions on 2026-10-29), so we have no failing local reproduction to write a fix against. Two defensible closes:
A maintainer call; the #2904 evidence above is the part that was missing for either one.
test(deinterlace): allow one skipped field for a single late tick (ob…
zackees commentedon Oct 31, 2026
Claiming: investigation (signature 1 only: the startup underrun), continuing #456 from this checkout.
Signature 2 is measured and fixed (#2904, verified over 62 Tier 2 tests-ON jobs), so this tick targets only the remaining one: the first-attempt, idle-box run where fields 2-23 are missing (
only 48 of 72 fields were shown).Plan, one branch
test/456-startup-underrun:run.py --repeat 60 --tests '^test_deinterlace_2x_field_order'(no$anchor), idle, and record how many fresh attempts fail and which case.zackees commentedon Oct 31, 2026
Releasing: signature 1 did not reproduce on this host either (worker tick,
obs-studio4).What I ran
Current master, this checkout's warm Tier 2 tree (
/build/tree, imagefa616dbb34c37160), the case's own ctest entry^test_deinterlace_2x_field_order_60i_into_60$,ctest --repeat until-fail:20,LP_NUM_THREADS=1, fresh test process per repetition:--cpus=2) with 2 busy loops, 20 fresh repetitionsSo both documented conditions are covered and neither reproduces the startup underrun here, matching the 2026-10-27 tick's "did not reproduce on this host". No branch pushed; nothing to land.
Mechanism, from the failing log (for whoever does reproduce it)
The failing run reported 25 unrecognised frames inside the measured window and
fields 1 24 25 ... 70shown, so fields 2-23 were dropped as overdue. That is the render thread not consuming the source at all for about 0.4 s at the start of the measured window, not a cadence error. Two things in the test allow it:play()pushes allFRAMESon wall-clock (os_sleepto_ns(ts)thenobs_source_output_video) without ever checking that the canvas has consumed anything. A first render stall longer than one frame period leaves the buffered async source ahead, and its buffered walk drops the overdue frames, which is the 20-in-a-row skip.warm_up()waits forw.ticks >= ticks + WARMUP_TICKS, andraw_videoticks once per canvas output frame whether or not the source contributed a frame, so "warm" only proves the canvas loop ran, not that the deinterlace path rendered this source once. The cold cost the failing log shows therefore lands inside the measured window.The test-side fix that removes the race without touching a single threshold: pace the measured push loop on the output — push frame
n+1only after field2nhas been shown (bounded wait, report and fall back if it times out), so the source is never more than one frame ahead and a cold render stall cannot produce overdue frames. That needs a reproduction to prove it fixes the reported failure rather than hiding it, which is why I am not pushing it blind.Releasing the claim.
#2904still covers signature 2 (62 Tier 2 tests-ON jobs, 0 failures), so the issue stays open on signature 1 only.test: Warm up the cold-start deinterlace case like the others (obs-ru…
zackees commentedon Nov 8, 2026
Additional #1390 inventory evidence: at 3973cc9 in the GCC 15.2 Tier 2 image, ten original invocations per core reproduced field-count/cadence failures at C:493/516/522 for all five rates. Hybrid pass/fail totals: 60i→60 1/9, 50i→50 1/9, 50i→60 1/9, 50i→30 1/9, 60i→50 2/8. C-reference: 2/8, 0/10, 0/10, 1/9, 0/10 respectively. High variable host load; sequential lanes are not a controlled performance comparison. Existing C tests remain unchanged and these fixtures remain outside the new native-graph registrations. Refs #1390.