Repository navigation
Description
Summary
The Rust unit test step of Fast Gate runs 4,926 tests in 418 s on 4 test threads. 26 tests take 10 s or more each. Together they take 802 s, which is 48 % of the 1,672 s of test time. The slowest test alone takes 148 s, so it sets a lower limit for the step. Fast Gate runs on every merge-queue run and every master push (and on PRs without a local-gate stamp).
There is no time limit for one test. ci/nextest.toml sets no slow-timeout, so nextest's default applies: a test is marked SLOW after 60 s and never fails. docs/rust-port/testing-policy.md has no time rule. Thus a slow test passes without a warning, and nothing stops the next one.
This issue proposes (A) a fix for each of the 26 tests, (B) a time budget as an acceptance criterion that CI enforces, and (C) a local measurement of time, memory and CPU in the worker skill, with the numbers in the PR. It is part B of #2593 for single tests ("parity tests are most of the Rust test time"); it does not change which tiers are required.
Evidence
Merge-queue run 40836910827 (#2882 on master c010b3fae8), job "Build / Fast Gate", 2026-10-29: Summary [ 418.390s] 4926 tests run: 4926 passed (2 slow), 6 skipped. The two SLOW tests are audio_discard_parity::random_cases_match and video_tick_parity::parity_proptest_tick_runs.
Most of the time comes from four causes:
- Proptests with many cases: 2,000 to 20,000 cases, where the default is 256. 18 of the 26 tests.
- Full 2^32 sweeps: every
u32or everyf32bit pattern, on all CPUs, while other tests run. 2 tests. - Large fixed grids: for example
output_stop_parity::callers_match_cruns 2 x 16 x 2,048 x 9 = 589,824 inputs on three sides. 5 tests. - A real compile:
native_build::mixed_objective_languages_…compiles C, C++ and Objective-C with unity and PCH. 1 test.
A. Proposed fix for each test
The time scales about linearly with the cases or the grid size. A fix keeps the exhaustive and edge-value parts of a test and keeps every assertion. Each fix must still kill the deliberate-break mutants that its PR reported. Run them again, and do not reduce a test below the point where a mutant survives.
| Time | Test | Source | Cause | Proposed fix |
|---|---|---|---|---|
| 147.9 s | obs_core::audio_discard_parity::random_cases_match |
#2404 | 20,000 cases | 2,000 cases |
| 101.2 s | source::video_tick_parity::parity_proptest_tick_runs |
#2448 | 20,000 cases | 2,000 cases |
| 50.1 s | hotkey::translations_parity::random_structs_match_c |
#1682 | 2,000 cases, large structs | 256 cases |
| 43.9 s | hotkey::translations::store_calls_follow_members |
#1682 | 2,000 cases | 256 cases |
| 38.5 s | source::video_tick::edge_proptest_tick_runs |
#2448 | 20,000 cases | 2,000 cases |
| 31.8 s | obs_core::audio_mix_parity::mix_on_every_start_frame |
#1909 | all 1,024 start frames x 3 x 2 rates | the first and last 8 frames and every 61st frame |
| 30.9 s | render::state_stacks_parity::sequences |
#2528 | 5,000 cases of up to 60 calls | 1,000 cases |
| 27.4 s | source::audio_timing_parity::sequences_match_the_c |
#1936 | 10,000 cases | 1,000 cases |
| 26.7 s | output::output_stop_parity::callers_match_c |
#2536 | 589,824 inputs x 3 sides | each of the 11 bits alone, 0, all set, and 64 random words, instead of all 2,048 |
| 25.9 s | source::render_route_parity::filter_decisions_every_filter |
#2208 | full product of 9 axes | a pairwise (covering) set of the same values, plus the full product of the edge values |
| 22.4 s | tests::video::video_fourcc_parity::every_u32_matches_c_oracle |
#978 | every u32 |
every known FourCC, each byte value in each byte position, and 2^20 random values; the full sweep in a scheduled run (see B4) |
| 22.3 s | source::audio_input::edge_downmix_model |
#2021 | per-case cost (8 x 40 samples) | half the cases, or 16 frames instead of 40 |
| 22.0 s | linear::round::exhaustive_matches_libm |
#1398 | every f32 |
the special values, every exponent with edge mantissas, and 2^20 random bit patterns; the full sweep in a scheduled run |
| 21.8 s | output::output_slots_parity::proptest_callers |
#2475 | 20,000 cases | 2,000 cases |
| 21.6 s | obs_core::audio_mix_parity::mix_matches |
#1909 | 4,096 cases | 512 cases |
| 21.3 s | source::audio_place_parity::callers_match_on_packet_sequences |
#2469 | 20,000 cases | 2,000 cases |
| 18.3 s | source::audio_input_parity::laws_random |
#2021 | 20,000 cases | 2,000 cases |
| 17.8 s | source::audio_input_parity::downmix_random |
#2021 | 20,000 cases | 2,000 cases |
| 16.9 s | source::deinterlace_pacing_parity::parity_sequences |
#1954 | 3,000 cases | 500 cases |
| 16.4 s | native_build::mixed_objective_languages_compile_load_and_rebuild_with_unity_and_pch |
#1333 | real compiles | keep; an exception in the budget (B2) |
| 15.2 s | source::deinterlace_pacing_parity::buffered_walk_sequences |
#1954 | 2,000 fixed sequences | 300 sequences |
| 14.8 s | output::encoder_bind_parity::callers_match_c_over_sequences |
#2511 | 20,000 cases | 2,000 cases |
| 12.6 s | source::deinterlace_pacing_parity::sequences_reach_every_branch |
#1954 | 3,000 cases | 500 cases, if every branch is still reached (the test asserts it) |
| 12.3 s | source::async_pacing_parity::queues_near_the_clock |
#2046 | 20,000 cases | 2,000 cases |
| 11.6 s | scene::item_transform_parity::compose_flags_exhaustive |
#2392 | 2,000 cases x 448 compositions | 200 cases (the 448 flag combinations stay exhaustive) |
| 10.7 s | hotkey::translations_parity::large_templates_match_c |
#1682 | 1 MiB template, 100,000 %1 |
64 KiB and 10,000 %1 (still past every fixed buffer) |
Expected result: these 26 tests drop from 802 s to about 100 s of test time. With 4 test threads, the step drops from about 418 s to about 240 s, and the longest single test drops from 148 s to the budget.
Proposed split: one PR for each crate's tests (obs-core, obs-source, obs-hotkey, obs-output, obs-graphics, obs-media-io, obs-scene). Each PR reports, for each test, the time before and after and the mutants it still kills. The authors of the tests know best which inputs matter, so each author is the natural owner of their crate's PR. We take the obs-core and obs-graphics tests (#2404, #1909, #2528).
B. Time budget as an acceptance criterion
We check the time of new tests locally before we open a PR. Not every contributor does, so CI must check it.
- CI enforces a budget. In
ci/nextest.toml:slow-timeout = { period = "10s", terminate-after = 3 }. nextest marks a test SLOW at 10 s and fails it at 30 s. The tests above that stay over 30 s until their fix lands get a temporary override that names this issue. - Exceptions are explicit. A test that needs more time gets a
[[profile.default.overrides]]entry with a filter for that test and a comment with the reason. Example: the native build test, which compiles real sources. - The testing policy gets a "Test time" section. A new test takes less than about 5 s in the CI debug build. Choose the proptest cases so that every branch is reached and every reported mutant is killed, not as many as possible. A new test is not a reason to add an exception.
- Full sweeps run on a schedule. The 2^32 sweeps, and full-size runs of the reduced proptests, run in the nightly security and release validation, which already runs outside PR and merge-group CI. For example: an
#[ignore]test run with--run-ignored only, or a case count read from an environment variable that only that run sets.
C. Measure new tests locally and report the numbers in the PR
The CI budget (B) finds a slow test only after the PR is open, and it does not see memory. A test that allocates too much does more damage than a slow one: in #2514 a parity test allocated up to 17 GB on one seed (one run had measured 39 MB), and it killed the 16 GB Fast Gate runner three times in three days (#2593 part A). A test that starts its own threads, such as the two 2^32 sweeps, also takes CPUs from the other test threads.
Proposed change to .claude/skills/obs-worker/SKILL.md (section 5, "Local checks") and to the PR description (section 4), for each PR that adds or changes Rust tests:
- Time: run each new or changed test module alone, in the debug build, and record its longest run. Over the budget (B3): reduce it before you push.
- Memory: record the peak RSS of the test process over many runs, not one: 20 to 50 runs for proptests, because the inputs change with each seed (
/usr/bin/time -von Linux,/usr/bin/time -lon macOS). Cap every generated size that drives an allocation. - CPU: state whether a test starts its own threads, and how many. A test that uses all CPUs gets a fixed thread count or moves to the scheduled run.
- The PR description gets a "Test cost" line with the three numbers, for example: "new tests: longest 0.7 s, peak RSS 18 MB over 25 runs, no own threads".
The obs-coordinator skill gets one check in its review: a PR that adds Rust tests and has no "Test cost" line gets a comment that asks for it.
These steps are what we do now before each PR (#2882 is an example). In the skill, every worker does them, and a reviewer can see the numbers.
Decisions needed
- The budget numbers: SLOW at 10 s and fail at 30 s, and about 5 s for a new test.
- Whether the full sweeps and full-size proptests run in the nightly run (B4), or whether the reduced versions are enough.
- Whether
obs-c-oraclebenches and other test executables (Tier 2 ctest) get the same budget. This issue covers only the nextest step. - A memory limit for one test (for example 1 GB peak RSS), and whether CI enforces it (for example a cgroup or
ulimitaround the test step), or the "Test cost" line (C) is enough.
🤖 Generated with Claude Code
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
vpacher commentedon Oct 29, 2026
The four rows of part A that are ours (#2404
random_cases_match, #1909mix_on_every_start_frameandmix_matches, #2528sequences, 233 s together) are in #2891, which we fix now. They do not depend on the budget decision here.zackees commentedon Oct 29, 2026
Triage (coordinator): not Ready yet — four open decisions, and the scope is XL. Everything else about the issue meets the definition of Ready: part A is a concrete, testable table (each row names the test, the cause and the target case count), the evidence is a real merge-group run, and the child slice #2891 is already Ready and being implemented (#2898 bounds the four obs-core/obs-graphics tests). No blocker issues are named, so nothing to link as
blocked_by.The four "Decisions needed" are project policy (CI cost, what the nightly run owns, what a new test must cost), so I have escalated them rather than set them:
human-review-neededis added. My recommendation, for a one-word confirmation:slow-timeout = { period = "10s", terminate-after = 3 }and the ~5 s rule for a new test. The numbers are in line with the current distribution (4,926 tests, 2 already SLOW at the 60 s default, and the fixes in part A landing at 0.5–3 s), and both are reversible inci/nextest.toml.obs-c-oraclebenches and the Tier 2 ctest executables have a different cost profile (real compiles); a single budget across them would either be vacuous or noisy. A second issue can take them once the nextest budget has run for a while.ulimitin CI. The obs-graphics: Port the libnsgif state machine and public API (#2510) #2514 case (17 GB on one seed) is a generated-size bug that a cap would hide behind a crashed test rather than point at the test; and a hard cap needs a number nobody has measured across the suite yet. Revisit enforcement once the line has run for a few weeks and the numbers exist.Once these are answered (or the labels changed), the remaining work splits cleanly along the issue's own proposal — one PR per crate, plus one PR for
ci/nextest.toml+ testing-policy "Test time" and one for the worker/coordinator skill text — and each is Ready on its own.vpacher commentedon Oct 29, 2026
Data for this issue: the result of #2898 (our four rows of part A, merged) and the daily cost of the slow tests. Measurements only, for the decisions above.
#2898 in the merge queue
Before: merge-queue run 40836910827 (#2882). After: merge-queue run 40846144867 (#2898). Fast Gate, same runner image.
obs_core::audio_discard_parity::random_cases_matchobs_core::audio_mix_parity::mix_on_every_start_frame(nowmix_on_sampled_start_frames)render::state_stacks_parity::sequencesobs_core::audio_mix_parity::mix_matchesThe runner of the after run was about 2x faster: the 4,645 tests that both runs have and #2898 did not change took 1,390 s before and 703 s after (factor 1.98; 2.06 for the 47 of them over 5 s). At the speed of the before run, the four tests take about 10 s instead of 233 s, so #2898 saves about 223 s of test time per Fast Gate run, and about 55 s of the test step's wall time (4 test threads).
The test step itself went from 418 s to 178 s, but most of that is the faster runner, and the after run did not select 3 of the 8 test binaries (
obs-tests-gpu,obs-tests-preflight,obs-tests-app: 275 tests, 49 s before). The slowest test of the step is nowsource::video_tick_parity::parity_proptest_tick_runs(51 s in the after run, 101 s in the before run). On the after runner, it is the only test over 30 s.Daily cost
Rust test jobs that ran to the end on 2026-10-28 (cancelled jobs not counted):
The macOS and Windows lanes run only for changes to their platforms, and in PR runs only for the changed crates, so not every one of those jobs runs a given test. The local gate (
ci-lint local-gate run, lane rust-test) runs the same tests again before each stamped PR.At the speed of the before run (418 s test step, 1,672 s of test time), Fast Gate alone on that day:
The macOS and Windows lanes add up to 94 runs to this for a change that selects those crates.
Limits of these numbers: one day (a busy one); runner speed changes by about 2x between runs; wall time is estimated as test time / 4, the step's thread count.
How #2898 chose its counts
The count of each test comes from the mutants of its source PR, rebuilt and run again with the test alone at increasing case counts, 5 runs each. 16 of the 19 mutants of #2404, 7 of 8 of #1909 (mix side) and all 15 of #2528 fail within 64 to 125 cases in every run; the others need an exact boundary, and other tests of the same module (the chosen cases, the sampled start frames) find them. In
audio_discard_parity, most of the time was the test's own setup (one deque push per sample): building each deque with one push took the test from 58 s to 11 s locally at the old case count. The details are in #2898. Other rows of part A could have the same kind of setup cost, which is worth a look before the case count is cut.zackees commentedon Oct 30, 2026
Decided: adopt the coordinator's four recommendations — (1)
slow-timeout = { period = "10s", terminate-after = 3 }with named per-test overrides citing this issue, and ~5 s for a new test; (2) full sweeps and full-size proptests run on schedule/dispatch/release, reduced in PR/queue/master; (3) nextest only for now; (4) memory via the "Test cost" PR line, no CI cgroup/ulimit yet.Reasons:
video_tick_parity::parity_proptest_tick_runs, and the bounded rows land at 0.7–1.8 s (data comment above).ci/nextest.tomlanddocs/rust-port/testing-policy.mdhave no time rule today, so nothing stops the next slow test; both numbers are one-line, reversible edits.OBS_PARITY_CASES(reducedon PR/merge_group/master push,fullon schedule, dispatch, tags and release branches). The 2^32 sweeps here should use the same switch rather than a second mechanism. Exhaustive coverage is kept, just not on every merge.Next: split per the issue — one PR for
ci/nextest.toml+ testing-policy "Test time", one for the obs-worker/obs-coordinator skill text (part C), and one per crate for the remaining part A rows. Each is Ready on its own.zackees commentedon Oct 30, 2026
Claiming: the remaining items, batched with #2072 (build-job setup + real-time tests) and #1592 (prepared Tier 2 image) into one PR to cut CI runs (maintainer direction: ~3 similar issues per PR). All are on the merge-queue critical path. Starts after #2991 merges (same workflow file).
zackees commentedon Oct 30, 2026
#3012 merged. It adds the nextest budget (10 s SLOW, 30 s fail, with named overrides that cite this issue and a
[profile.full]for the full parity level), addsparity::sweepandparity::keeponOBS_PARITY_CASESfor the 2^32 sweeps and the big grids, reduces the last 6 tests over 10 s (104 s to about 14 s of test time per Fast Gate run), and adds the testing-policy "Test time" section and the Test cost line in the worker and coordinator skills. Parts A, B and C are done. The mutants of the reduced tests were not rerun.vpacher commentedon Oct 30, 2026
Thanks for #3012. Two items of this issue are still open, so I suggest that it stays open until they are done or have their own issues.
1. Six rows of part A are still over 10 s
In the merge-queue run of #3012 itself (41015283649, reduced parity level, nextest summary "5186 passed (6 slow)"), these tests are still SLOW. For three of them, the cost is not a proptest case count (per-case work, fixed sequences, a large template), so
OBS_PARITY_CASES,parity::sweepandparity::keepdo not change it. The other three are proptests that are still over 10 s at the reduced level.source::audio_input::edge_downmix_modeloutput::encoder_bind_parity::callers_match_c_over_sequencessource::deinterlace_pacing_parity::buffered_walk_sequencessource::deinterlace_pacing_parity::sequences_reach_every_branchhotkey::translations_parity::large_templates_match_c%1(still past every fixed buffer)source::video_tick_parity::parity_proptest_tick_runsparity::cases(#2991); just over the mark, a few more cases off would doThe runner speed changes by up to 2x between runs, so the two time columns are approximate. The native build test (14.9 s) has its override, as decided.
@rdoupe, the six tests come from your ports (#2021, #2511, #1954, #1682, #2448). The original proposal of part A was one PR per crate by the owner of the tests. Can you take them? The "Test time" section of the testing policy now applies: about 5 s or less, and the reduced counts must still kill the mutants that the source PR reported.
2. The mutant check of the six tests that #3012 reduced
#3012 says that the mutants of the reduced tests were not run again. The
Decided:above requires it ("every reduction must still kill the mutants its source PR reported, as #2898 showed"), and so does the new "Test time" section. The six tests:linear::round::exhaustive_matches_libm,video_fourcc_parity::every_u32_matches_c_oracle,output_stop_parity::callers_match_c,render_route_parity::filter_decisions_every_filter,video_tick::edge_proptest_tick_runs,hotkey::translations::store_calls_follow_members.I can pick it up later with the method of #2898: build each mutant of the source PR again and run the test alone at the reduced level, 5 runs each. I will file it as its own issue and claim it then.
zackees commentedon Oct 30, 2026
Reproduced: no — current
masteralready carries the per-test nextest budget from #3012.Evidence:
ci/nextest.toml:7-9sets the default budget to SLOW at 10 s and termination after 30 s.ci/test_nextest_budget.py:25-39regression-tests that budget and requires named, issue-citing overrides.docs/rust-port/testing-policy.md:489-508documents the policy and enforcement scope.Verified on
b924e5e8a:uv run python -m unittest ci/test_nextest_budget.py— 2 passed.(cd ci && uv run python -m unittest test_rust_execution.py)— 9 passed, including reduced/full profile selection.Closing as already fixed by #3012.
test: Bound our four slowest parity tests (obs-rust#2891) (obs-rust#2898
infra(ci): per-test time budget and reduced sweeps (obs-rust#2888, ob…
rdoupe commentedon Oct 31, 2026
The four rows of part A that are still ours (
callers_match_c_over_sequences,buffered_walk_sequences,sequences_reach_every_branch,parity_proptest_tick_runs; the other two were fixed in #3103 and #3110) are tracked in #3167, as suggested above.Most of their time was not the tests themselves: proptest's ChaCha20 generator runs unoptimized in the debug build. Built optimized, the same counts take 0.4 to 2.5 s each locally (7.4 s to 0.7 s for
callers_match_c_over_sequences), with no case count reduced. Numbers and the PR will follow on #3167.