Skip to content

test: Proptest's RNG is most of our parity tests' time in the debug build #3172

Closed

Description

@rdoupe

Summary

Follow-up to #2888 (closed): the Fast Gate log names only tests over 10 s, so this measures every test locally. The workspace suite ran at OBS_PARITY_CASES=reduced in the debug build (ci/nextest.toml, the default profile, 6 test threads, --status-level all), on master 538e920. The run took 75 s, and the per-test times add up to 399 s.

Most of the cost in our proptests isn't the test. Profiled with perf, 60 to 70 % of encoder_lifecycle_parity::callers_match_c_over_sequences, output_slots::edge_proptest_stand_in and caption_text::no_panic_on_raw_bytes is proptest's ChaCha RNG (rand_chacha refill and the SSE2 intrinsics behind it), compiled at opt-level 0 like every dependency in the dev profile. Building proptest, rand_core, rand_chacha and ppv-lite86 at opt-level 3 in the dev profile changes no test, case count or input distribution. The workspace already does this for obs-media-io and obs-graphics.

Tests over about 3 s (one full-suite run, so the times include contention)

The four tests of #3167 and the native_build::* tests (which have a named override in ci/nextest.toml) are listed for reference only.

Test Time Owner (PR) Cause Fix
output::encoder_lifecycle_parity::callers_match_c_over_sequences 4.8 s ours (#2521) the RNG: 20,000 cases written as a struct literal opt-level 3 for the RNG
output::output_start_parity::callers_match_c_over_sequences 4.4 s ours (#2532) the RNG: 20,000 cases written as a struct literal opt-level 3 for the RNG
scene::item_order_parity::reorder_check_matches_c_for_every_short_order 4.4 s ours (#2899) each of about 1.2 million checks formats a Debug string of both orders for a message it shows only on failure format the message only on failure
output::output_slots::edge_proptest_stand_in 4.3 s ours (#2475) the RNG: with_cases(4096) opt-level 3 for the RNG
output::caption_text::no_panic_on_raw_bytes 3.9 s ours (#1620) the RNG: up to 20,000 random bytes for each of 1,024 cases opt-level 3 for the RNG
source::audio_timing_parity::sequences_reach_every_branch 3.3 s ours (#1936) the RNG opt-level 3 for the RNG
hotkey::nix_x11_keys_parity::random_keyboards 3.3 s ours (#2903) the RNG, partly opt-level 3 for the RNG
output::delay_parity::parity_random_sequences 3.0 s ours (#1567) the RNG opt-level 3 for the RNG
hotkey::translations_parity::random_structs_match_c 3.0 s ours (#1682) the RNG opt-level 3 for the RNG
output::output_stop_parity::callers_match_c_over_sequences 2.9 s ours (#2536) the RNG: 20,000 cases written as a struct literal opt-level 3 for the RNG
output::output_stop::edge_stand_in_every_route 4.4 s ours (#2536) a fixed grid of 294,912 stand-ins through 11 callers on both sides; the stand-in's event log, which the test compares, is most of the cost not an easy win: left as is
source::monitoring_parity::balance_bit_patterns 3.4 s ours (#2629) about 117 million f32 bit patterns, 100 million of them in its exhaustive edge ranges not an easy win (any cut is in the edges): left as is
source::frame_copy_parity::plan_exhaustive_over_format_and_line_size_relation 3.0 s ours (#2215) an exhaustive grid not an easy win: left as is
tests::platform::pipe_parity::process::dead_pipe_write_survives_concurrent_spawns (obs-util) 7.1 s not ours (the dead_pipe_write_survives_concurrent_spawns RED commit, 5677125) 3,000 dead-pipe spawns against 16 spawning threads for its owner
source::deinterlace_pacing_parity::buffered_walk_sequences 9.3 s #3167
output::encoder_bind_parity::callers_match_c_over_sequences 9.2 s #3167
source::deinterlace_pacing_parity::sequences_reach_every_branch 7.9 s #3167
source::video_tick_parity::parity_proptest_tick_runs 5.3 s #3167
native_build::* (3 tests) 4.4 to 14.9 s named override

Several more of our proptest modules still write with_cases(N) or cases: N instead of obs_c_oracle::parity::cases(N). Once the RNG is optimized none of them is slow, so this issue cuts no count; routing them through parity::cases would cut coverage at the reduced level for no time that matters.

Plan

One PR, with no change to any case count:

  1. Build proptest, rand_core, rand_chacha and ppv-lite86 at opt-level 3 in the dev profile (Cargo.toml, next to the existing obs-media-io and obs-graphics entries).
  2. item_order_parity: format the check's message only when it fails.

Acceptance

  • Before and after per-test times in the PR, with the "Test cost" line.
  • No case count, input or assertion changes, so the source PRs' mutants see the same tests.

Refs #2888

Activity

  1. zackees commented on Oct 31, 2026

    @zackees
    Member

    Claiming: #3172 (build proptest's RNG dependencies optimized in the dev profile)

    Plan, one PR:

    1. Reproduce the current debug-profile cost locally for a representative parity proptest, and record the RED evidence (test wall time plus perf showing rand_chacha/ppv-lite86 at opt-level 0).
    2. Add opt-level = 3 overrides for proptest, rand_core, rand_chacha and ppv-lite86 in the workspace Cargo.toml, with the same rationale comment as the existing obs-media-io/obs-graphics overrides.
    3. Re-run the same tests and the full ./unittest Rust tiers; confirm case counts and inputs are unchanged and record the GREEN timings in the PR. Add a ci/ check that pins the four overrides so they are not dropped.

    Expected first PR: within this session.

  2. ghost assigned on Oct 31, 2026
  3. zackees commented on Oct 31, 2026

    @zackees
    Member

    Fixed by #3173 (6fdc264), merged to master.

    • Cargo.toml: proptest, rand_core, rand_chacha and ppv-lite86 build at opt-level 3 in the dev profile, next to the existing obs-media-io and obs-graphics overrides.
    • ci/test_cargo_dev_profile.py: pins those four overrides (RED before the manifest change).
    • rust/obs-scene/tests/scene/item_order_parity.rs: check_reorder formats both orders only in assert_eq!'s failure path instead of on every passing check.

    Focused OBS_PARITY_CASES=reduced runs, before → after: encoder lifecycle 4.49 → 0.46 s, output slots 4.04 → 0.51 s, caption text 3.92 → 0.73 s, reorder check 5.15 → 3.86 s. No case count, input or assertion changed. reorder_check_matches_c_for_every_short_order: median 3.53 s, peak RSS 13.6 MB over 20 runs, no own threads.

    Nothing left on this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

infraBuild, gate, CI and toolingreadyMeets the definition of Ready; claimable

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions