Skip to content

perf(ci): 26 Rust tests take 48 % of the Fast Gate test time; add a per-test time budget #2888

Closed

Description

@vpacher

Summary

The Rust unit test step of Fast Gate runs 4,926 tests in 418 s on 4 test threads. 26 tests take 10 s or more each. Together they take 802 s, which is 48 % of the 1,672 s of test time. The slowest test alone takes 148 s, so it sets a lower limit for the step. Fast Gate runs on every merge-queue run and every master push (and on PRs without a local-gate stamp).

There is no time limit for one test. ci/nextest.toml sets no slow-timeout, so nextest's default applies: a test is marked SLOW after 60 s and never fails. docs/rust-port/testing-policy.md has no time rule. Thus a slow test passes without a warning, and nothing stops the next one.

This issue proposes (A) a fix for each of the 26 tests, (B) a time budget as an acceptance criterion that CI enforces, and (C) a local measurement of time, memory and CPU in the worker skill, with the numbers in the PR. It is part B of #2593 for single tests ("parity tests are most of the Rust test time"); it does not change which tiers are required.

Evidence

Merge-queue run 40836910827 (#2882 on master c010b3fae8), job "Build / Fast Gate", 2026-10-29: Summary [ 418.390s] 4926 tests run: 4926 passed (2 slow), 6 skipped. The two SLOW tests are audio_discard_parity::random_cases_match and video_tick_parity::parity_proptest_tick_runs.

Most of the time comes from four causes:

  1. Proptests with many cases: 2,000 to 20,000 cases, where the default is 256. 18 of the 26 tests.
  2. Full 2^32 sweeps: every u32 or every f32 bit pattern, on all CPUs, while other tests run. 2 tests.
  3. Large fixed grids: for example output_stop_parity::callers_match_c runs 2 x 16 x 2,048 x 9 = 589,824 inputs on three sides. 5 tests.
  4. A real compile: native_build::mixed_objective_languages_… compiles C, C++ and Objective-C with unity and PCH. 1 test.

A. Proposed fix for each test

The time scales about linearly with the cases or the grid size. A fix keeps the exhaustive and edge-value parts of a test and keeps every assertion. Each fix must still kill the deliberate-break mutants that its PR reported. Run them again, and do not reduce a test below the point where a mutant survives.

Time Test Source Cause Proposed fix
147.9 s obs_core::audio_discard_parity::random_cases_match #2404 20,000 cases 2,000 cases
101.2 s source::video_tick_parity::parity_proptest_tick_runs #2448 20,000 cases 2,000 cases
50.1 s hotkey::translations_parity::random_structs_match_c #1682 2,000 cases, large structs 256 cases
43.9 s hotkey::translations::store_calls_follow_members #1682 2,000 cases 256 cases
38.5 s source::video_tick::edge_proptest_tick_runs #2448 20,000 cases 2,000 cases
31.8 s obs_core::audio_mix_parity::mix_on_every_start_frame #1909 all 1,024 start frames x 3 x 2 rates the first and last 8 frames and every 61st frame
30.9 s render::state_stacks_parity::sequences #2528 5,000 cases of up to 60 calls 1,000 cases
27.4 s source::audio_timing_parity::sequences_match_the_c #1936 10,000 cases 1,000 cases
26.7 s output::output_stop_parity::callers_match_c #2536 589,824 inputs x 3 sides each of the 11 bits alone, 0, all set, and 64 random words, instead of all 2,048
25.9 s source::render_route_parity::filter_decisions_every_filter #2208 full product of 9 axes a pairwise (covering) set of the same values, plus the full product of the edge values
22.4 s tests::video::video_fourcc_parity::every_u32_matches_c_oracle #978 every u32 every known FourCC, each byte value in each byte position, and 2^20 random values; the full sweep in a scheduled run (see B4)
22.3 s source::audio_input::edge_downmix_model #2021 per-case cost (8 x 40 samples) half the cases, or 16 frames instead of 40
22.0 s linear::round::exhaustive_matches_libm #1398 every f32 the special values, every exponent with edge mantissas, and 2^20 random bit patterns; the full sweep in a scheduled run
21.8 s output::output_slots_parity::proptest_callers #2475 20,000 cases 2,000 cases
21.6 s obs_core::audio_mix_parity::mix_matches #1909 4,096 cases 512 cases
21.3 s source::audio_place_parity::callers_match_on_packet_sequences #2469 20,000 cases 2,000 cases
18.3 s source::audio_input_parity::laws_random #2021 20,000 cases 2,000 cases
17.8 s source::audio_input_parity::downmix_random #2021 20,000 cases 2,000 cases
16.9 s source::deinterlace_pacing_parity::parity_sequences #1954 3,000 cases 500 cases
16.4 s native_build::mixed_objective_languages_compile_load_and_rebuild_with_unity_and_pch #1333 real compiles keep; an exception in the budget (B2)
15.2 s source::deinterlace_pacing_parity::buffered_walk_sequences #1954 2,000 fixed sequences 300 sequences
14.8 s output::encoder_bind_parity::callers_match_c_over_sequences #2511 20,000 cases 2,000 cases
12.6 s source::deinterlace_pacing_parity::sequences_reach_every_branch #1954 3,000 cases 500 cases, if every branch is still reached (the test asserts it)
12.3 s source::async_pacing_parity::queues_near_the_clock #2046 20,000 cases 2,000 cases
11.6 s scene::item_transform_parity::compose_flags_exhaustive #2392 2,000 cases x 448 compositions 200 cases (the 448 flag combinations stay exhaustive)
10.7 s hotkey::translations_parity::large_templates_match_c #1682 1 MiB template, 100,000 %1 64 KiB and 10,000 %1 (still past every fixed buffer)

Expected result: these 26 tests drop from 802 s to about 100 s of test time. With 4 test threads, the step drops from about 418 s to about 240 s, and the longest single test drops from 148 s to the budget.

Proposed split: one PR for each crate's tests (obs-core, obs-source, obs-hotkey, obs-output, obs-graphics, obs-media-io, obs-scene). Each PR reports, for each test, the time before and after and the mutants it still kills. The authors of the tests know best which inputs matter, so each author is the natural owner of their crate's PR. We take the obs-core and obs-graphics tests (#2404, #1909, #2528).

B. Time budget as an acceptance criterion

We check the time of new tests locally before we open a PR. Not every contributor does, so CI must check it.

  1. CI enforces a budget. In ci/nextest.toml: slow-timeout = { period = "10s", terminate-after = 3 }. nextest marks a test SLOW at 10 s and fails it at 30 s. The tests above that stay over 30 s until their fix lands get a temporary override that names this issue.
  2. Exceptions are explicit. A test that needs more time gets a [[profile.default.overrides]] entry with a filter for that test and a comment with the reason. Example: the native build test, which compiles real sources.
  3. The testing policy gets a "Test time" section. A new test takes less than about 5 s in the CI debug build. Choose the proptest cases so that every branch is reached and every reported mutant is killed, not as many as possible. A new test is not a reason to add an exception.
  4. Full sweeps run on a schedule. The 2^32 sweeps, and full-size runs of the reduced proptests, run in the nightly security and release validation, which already runs outside PR and merge-group CI. For example: an #[ignore] test run with --run-ignored only, or a case count read from an environment variable that only that run sets.

C. Measure new tests locally and report the numbers in the PR

The CI budget (B) finds a slow test only after the PR is open, and it does not see memory. A test that allocates too much does more damage than a slow one: in #2514 a parity test allocated up to 17 GB on one seed (one run had measured 39 MB), and it killed the 16 GB Fast Gate runner three times in three days (#2593 part A). A test that starts its own threads, such as the two 2^32 sweeps, also takes CPUs from the other test threads.

Proposed change to .claude/skills/obs-worker/SKILL.md (section 5, "Local checks") and to the PR description (section 4), for each PR that adds or changes Rust tests:

  1. Time: run each new or changed test module alone, in the debug build, and record its longest run. Over the budget (B3): reduce it before you push.
  2. Memory: record the peak RSS of the test process over many runs, not one: 20 to 50 runs for proptests, because the inputs change with each seed (/usr/bin/time -v on Linux, /usr/bin/time -l on macOS). Cap every generated size that drives an allocation.
  3. CPU: state whether a test starts its own threads, and how many. A test that uses all CPUs gets a fixed thread count or moves to the scheduled run.
  4. The PR description gets a "Test cost" line with the three numbers, for example: "new tests: longest 0.7 s, peak RSS 18 MB over 25 runs, no own threads".

The obs-coordinator skill gets one check in its review: a PR that adds Rust tests and has no "Test cost" line gets a comment that asks for it.

These steps are what we do now before each PR (#2882 is an example). In the skill, every worker does them, and a reviewer can see the numbers.

Decisions needed

  • The budget numbers: SLOW at 10 s and fail at 30 s, and about 5 s for a new test.
  • Whether the full sweeps and full-size proptests run in the nightly run (B4), or whether the reduced versions are enough.
  • Whether obs-c-oracle benches and other test executables (Tier 2 ctest) get the same budget. This issue covers only the nextest step.
  • A memory limit for one test (for example 1 GB peak RSS), and whether CI enforces it (for example a cgroup or ulimit around the test step), or the "Test cost" line (C) is enough.

🤖 Generated with Claude Code

Activity

  1. vpacher commented on Oct 29, 2026

    @vpacher
    MemberAuthor

    The four rows of part A that are ours (#2404 random_cases_match, #1909 mix_on_every_start_frame and mix_matches, #2528 sequences, 233 s together) are in #2891, which we fix now. They do not depend on the budget decision here.

  2. zackees commented on Oct 29, 2026

    @zackees
    Member

    Triage (coordinator): not Ready yet — four open decisions, and the scope is XL. Everything else about the issue meets the definition of Ready: part A is a concrete, testable table (each row names the test, the cause and the target case count), the evidence is a real merge-group run, and the child slice #2891 is already Ready and being implemented (#2898 bounds the four obs-core/obs-graphics tests). No blocker issues are named, so nothing to link as blocked_by.

    The four "Decisions needed" are project policy (CI cost, what the nightly run owns, what a new test must cost), so I have escalated them rather than set them: human-review-needed is added. My recommendation, for a one-word confirmation:

    1. Budget numbers — adopt slow-timeout = { period = "10s", terminate-after = 3 } and the ~5 s rule for a new test. The numbers are in line with the current distribution (4,926 tests, 2 already SLOW at the 60 s default, and the fixes in part A landing at 0.5–3 s), and both are reversible in ci/nextest.toml.
    2. Full sweeps on a schedule — yes: the 2^32 sweeps and the full-size proptest runs move to the nightly validation, with the reduced versions in PR/merge-group CI. Exhaustive coverage is worth keeping somewhere, and the nightly lane already exists for it; the alternative (reduced only) silently drops inputs that found real bugs before.
    3. Scope of the budget — nextest only for now, as filed. The obs-c-oracle benches and the Tier 2 ctest executables have a different cost profile (real compiles); a single budget across them would either be vacuous or noisy. A second issue can take them once the nextest budget has run for a while.
    4. Memory — the "Test cost" line (C) first, not a cgroup/ulimit in CI. The obs-graphics: Port the libnsgif state machine and public API (#2510) #2514 case (17 GB on one seed) is a generated-size bug that a cap would hide behind a crashed test rather than point at the test; and a hard cap needs a number nobody has measured across the suite yet. Revisit enforcement once the line has run for a few weeks and the numbers exist.

    Once these are answered (or the labels changed), the remaining work splits cleanly along the issue's own proposal — one PR per crate, plus one PR for ci/nextest.toml + testing-policy "Test time" and one for the worker/coordinator skill text — and each is Ready on its own.

  3. vpacher commented on Oct 29, 2026

    @vpacher
    MemberAuthor

    Data for this issue: the result of #2898 (our four rows of part A, merged) and the daily cost of the slow tests. Measurements only, for the decisions above.

    #2898 in the merge queue

    Before: merge-queue run 40836910827 (#2882). After: merge-queue run 40846144867 (#2898). Fast Gate, same runner image.

    Test Before After
    obs_core::audio_discard_parity::random_cases_match 147.9 s 1.26 s
    obs_core::audio_mix_parity::mix_on_every_start_frame (now mix_on_sampled_start_frames) 31.8 s 0.68 s
    render::state_stacks_parity::sequences 30.9 s 1.25 s
    obs_core::audio_mix_parity::mix_matches 21.6 s 1.79 s
    Total 233 s 5.0 s

    The runner of the after run was about 2x faster: the 4,645 tests that both runs have and #2898 did not change took 1,390 s before and 703 s after (factor 1.98; 2.06 for the 47 of them over 5 s). At the speed of the before run, the four tests take about 10 s instead of 233 s, so #2898 saves about 223 s of test time per Fast Gate run, and about 55 s of the test step's wall time (4 test threads).

    The test step itself went from 418 s to 178 s, but most of that is the faster runner, and the after run did not select 3 of the 8 test binaries (obs-tests-gpu, obs-tests-preflight, obs-tests-app: 275 tests, 49 s before). The slowest test of the step is now source::video_tick_parity::parity_proptest_tick_runs (51 s in the after run, 101 s in the before run). On the after runner, it is the only test over 30 s.

    Daily cost

    Rust test jobs that ran to the end on 2026-10-28 (cancelled jobs not counted):

    Lane PR Merge queue Master push Nightly and manual Total
    Fast Gate (Linux) 131 76 37 2 246
    Rust Tests (macos-26) 20 8 7 1 36
    Rust Tests (windows-2025-vs2026) 33 13 11 1 58

    The macOS and Windows lanes run only for changes to their platforms, and in PR runs only for the changed crates, so not every one of those jobs runs a given test. The local gate (ci-lint local-gate run, lane rust-test) runs the same tests again before each stamped PR.

    At the speed of the before run (418 s test step, 1,672 s of test time), Fast Gate alone on that day:

    • The test step: 246 x 418 s, about 28.6 h of runner time.
    • The 26 tests of part A: 802 s of test time per run, about 200 s of the step's wall time with 4 threads, so about 13.7 h of runner time.
    • test: Bound our four slowest parity tests (#2891) #2898 (4 of the 26): about 55 s per run, so about 3.8 h of runner time.
    • The other 22 rows of part A: 569 s of test time per run, about 142 s of wall time, so about 9.7 h of runner time, if they are bounded as the four were.

    The macOS and Windows lanes add up to 94 runs to this for a change that selects those crates.

    Limits of these numbers: one day (a busy one); runner speed changes by about 2x between runs; wall time is estimated as test time / 4, the step's thread count.

    How #2898 chose its counts

    The count of each test comes from the mutants of its source PR, rebuilt and run again with the test alone at increasing case counts, 5 runs each. 16 of the 19 mutants of #2404, 7 of 8 of #1909 (mix side) and all 15 of #2528 fail within 64 to 125 cases in every run; the others need an exact boundary, and other tests of the same module (the chosen cases, the sampled start frames) find them. In audio_discard_parity, most of the time was the test's own setup (one deque push per sample): building each deque with one push took the test from 58 s to 11 s locally at the old case count. The details are in #2898. Other rows of part A could have the same kind of setup cost, which is worth a look before the case count is cut.

  4. moved this from Triage to Needs decision in obs-rust Multiviewon Oct 29, 2026
  5. zackees commented on Oct 30, 2026

    @zackees
    Member

    Decided: adopt the coordinator's four recommendations — (1) slow-timeout = { period = "10s", terminate-after = 3 } with named per-test overrides citing this issue, and ~5 s for a new test; (2) full sweeps and full-size proptests run on schedule/dispatch/release, reduced in PR/queue/master; (3) nextest only for now; (4) memory via the "Test cost" PR line, no CI cgroup/ulimit yet.

    Reasons:

    Next: split per the issue — one PR for ci/nextest.toml + testing-policy "Test time", one for the obs-worker/obs-coordinator skill text (part C), and one per crate for the remaining part A rows. Each is Ready on its own.

  6. zackees commented on Oct 30, 2026

    @zackees
    Member

    Claiming: the remaining items, batched with #2072 (build-job setup + real-time tests) and #1592 (prepared Tier 2 image) into one PR to cut CI runs (maintainer direction: ~3 similar issues per PR). All are on the merge-queue critical path. Starts after #2991 merges (same workflow file).

  7. moved this from Needs decision to In progress in obs-rust Multiviewon Oct 30, 2026
  8. zackees commented on Oct 30, 2026

    @zackees
    Member

    #3012 merged. It adds the nextest budget (10 s SLOW, 30 s fail, with named overrides that cite this issue and a [profile.full] for the full parity level), adds parity::sweep and parity::keep on OBS_PARITY_CASES for the 2^32 sweeps and the big grids, reduces the last 6 tests over 10 s (104 s to about 14 s of test time per Fast Gate run), and adds the testing-policy "Test time" section and the Test cost line in the worker and coordinator skills. Parts A, B and C are done. The mutants of the reduced tests were not rerun.

  9. vpacher commented on Oct 30, 2026

    @vpacher
    MemberAuthor

    Thanks for #3012. Two items of this issue are still open, so I suggest that it stays open until they are done or have their own issues.

    1. Six rows of part A are still over 10 s

    In the merge-queue run of #3012 itself (41015283649, reduced parity level, nextest summary "5186 passed (6 slow)"), these tests are still SLOW. For three of them, the cost is not a proptest case count (per-case work, fixed sequences, a large template), so OBS_PARITY_CASES, parity::sweep and parity::keep do not change it. The other three are proptests that are still over 10 s at the reduced level.

    Test Part A #3012 queue run Source Proposed fix (part A)
    source::audio_input::edge_downmix_model 22.3 s 21.8 s #2021 half the cases, or 16 frames instead of 40 (the cost is per case: 8 x 40 samples)
    output::encoder_bind_parity::callers_match_c_over_sequences 14.8 s 15.4 s #2511 2,000 cases instead of 20,000
    source::deinterlace_pacing_parity::buffered_walk_sequences 15.2 s 13.9 s #1954 300 fixed sequences instead of 2,000
    source::deinterlace_pacing_parity::sequences_reach_every_branch 12.6 s 12.3 s #1954 500 cases, if every branch is still reached (the test asserts it)
    hotkey::translations_parity::large_templates_match_c 10.7 s 11.0 s #1682 a 64 KiB template and 10,000 %1 (still past every fixed buffer)
    source::video_tick_parity::parity_proptest_tick_runs 101.2 s 10.4 s #2448 already reduced by parity::cases (#2991); just over the mark, a few more cases off would do

    The runner speed changes by up to 2x between runs, so the two time columns are approximate. The native build test (14.9 s) has its override, as decided.

    @rdoupe, the six tests come from your ports (#2021, #2511, #1954, #1682, #2448). The original proposal of part A was one PR per crate by the owner of the tests. Can you take them? The "Test time" section of the testing policy now applies: about 5 s or less, and the reduced counts must still kill the mutants that the source PR reported.

    2. The mutant check of the six tests that #3012 reduced

    #3012 says that the mutants of the reduced tests were not run again. The Decided: above requires it ("every reduction must still kill the mutants its source PR reported, as #2898 showed"), and so does the new "Test time" section. The six tests: linear::round::exhaustive_matches_libm, video_fourcc_parity::every_u32_matches_c_oracle, output_stop_parity::callers_match_c, render_route_parity::filter_decisions_every_filter, video_tick::edge_proptest_tick_runs, hotkey::translations::store_calls_follow_members.

    I can pick it up later with the method of #2898: build each mutant of the source PR again and run the test alone at the reduced level, 5 runs each. I will file it as its own issue and claim it then.

  10. zackees commented on Oct 30, 2026

    @zackees
    Member

    Reproduced: no — current master already carries the per-test nextest budget from #3012.

    Evidence:

    • ci/nextest.toml:7-9 sets the default budget to SLOW at 10 s and termination after 30 s.
    • ci/test_nextest_budget.py:25-39 regression-tests that budget and requires named, issue-citing overrides.
    • docs/rust-port/testing-policy.md:489-508 documents the policy and enforcement scope.

    Verified on b924e5e8a:

    • uv run python -m unittest ci/test_nextest_budget.py — 2 passed.
    • (cd ci && uv run python -m unittest test_rust_execution.py) — 9 passed, including reduced/full profile selection.

    Closing as already fixed by #3012.

  11. moved this from In progress to Done in obs-rust Multiviewon Oct 30, 2026
  12. rdoupe commented on Oct 31, 2026

    @rdoupe
    Collaborator

    The four rows of part A that are still ours (callers_match_c_over_sequences, buffered_walk_sequences, sequences_reach_every_branch, parity_proptest_tick_runs; the other two were fixed in #3103 and #3110) are tracked in #3167, as suggested above.

    Most of their time was not the tests themselves: proptest's ChaCha20 generator runs unoptimized in the debug build. Built optimized, the same counts take 0.4 to 2.5 s each locally (7.4 s to 0.7 s for callers_match_c_over_sequences), with no case count reduced. Numbers and the PR will follow on #3167.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

infraBuild, gate, CI and toolingreadyMeets the definition of Ready; claimable

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions