Skip to content

CI perf: test (windows/macos) — redundant cargo build step + full debuginfo linking ~240 test binaries (~930 job-min/day, ~300 macOS) #1173

Closed as not planned
Closed as not planned
CI perf: test (windows/macos) — redundant cargo build step + full debuginfo linking ~240 test binaries (~930 job-min/day, ~300 macOS)#1173
Labels
agent:triagedci-perfCI / merge-queue performance finding (profiler routine)priority:p3

Description

@mikolalysenko
Mikola Lysenko (mikolalysenko)
opened on Oct 8, 2026

Measurement

Window: 2026-10-08 02:16Z → 19:36Z, all CI runs.

test job Build step (cargo build --workspace)

OS Legs/day Avg per leg Total
Windows 408 1.54 min 628 job-min/day
macOS 179 1.71 min 306 job-min/day

test job duration, 7 days, p50 / p90

OS Shard 1 + shard 2 combined
Windows 21.7 / 26.9 min
macOS 15.9 / 21.8 min

Merge_group run 37828932401

  • windows shard 2:
    • Build took 84 s.
    • Run tests took 13.6 min: 5 min 28 s compiling and linking (only socket-patch-core/socket-patch-cli show Compiling, so deps came from the rust-cache) plus 8.2 min running 159 test binaries.
  • windows shard 1:
    • Build took 91 s.
    • Core lib unit tests took 210 s, against 57 s on macOS and 48 s under coverage.
  • macOS shard 1 / macOS shard 2: Build took 85 s / 74 s.

Where the time goes

  • Build compiles the workspace bins with the dev profile. cargo test in the next step recompiles the same crates as test targets and links them.
  • Linking about 240 integration-test binaries with full DWARF/PDB debuginfo dominates the 5.5 min compile phase on Windows.

Root cause

  • ci.yml test job, line ~285: - name: Build / run: cargo build --workspace runs before scripts/ci-test-shard.py.
  • cargo test already builds every [[bin]] that integration tests use via CARGO_BIN_EXE_*, so the separate build only helps if some test locates target/debug/socket-patch by path.
  • The dev profile's default debug = true makes every test binary carry full debuginfo, which CI never uses. Panics still give file:line with line-tables-only.

Proposed fix

  1. Check that no test needs a pre-built binary outside CARGO_BIN_EXE_*: grep for target/debug / env!("CARGO_BIN_EXE under crates/*/tests. Then delete the Build step from the test job.
    • On push to main it can stay (or become cargo test --no-run) if the rust-cache save wants bin artifacts.
  2. Set CARGO_PROFILE_DEV_DEBUG: line-tables-only in the test job env, and in e2e-build if it isn't already off. This shrinks link inputs for the ~240 test binaries on Windows and macOS.
  3. Follow-up, separate: find out why core lib unit tests take 210 s on Windows against 57 s on macOS. Likely filesystem-heavy tests on NTFS/Defender, so check for tempdir churn.

Expected saving

  • Measured: removing Build saves ~628 Windows + ~306 macOS job-min/day, and about 1.5 min off each test leg's wall time.
  • Estimated, not yet measured: line-tables-only should save another 1–2 min per Windows/macOS leg, i.e. about 400–800 Windows and 180–360 macOS job-min/day. Please measure this on the PR.
  • test legs aren't on the ci-ok critical path today (Gradle e2e is), so the critical-path effect is about 0. The gain is macOS/Windows runner time and slot occupancy.

Coverage and risk

  • No test is removed. If some test does depend on a pre-built binary, step 1 shows that immediately as a test failure on the PR.
  • line-tables-only keeps backtrace line numbers.
  • Required checks ci-ok and clippy are unchanged.

Effort

S

ROI

ROI = weighted saving × confidence / effort.

  • Weighted saving: (2 × 0.63 Windows + 3 × 0.31 macOS) = 2.2k job-min/day.
  • Confidence: 0.6.
  • Effort: S, scored as 1.
  • ROI = 1.3

Activity

  1. Mikola Lysenko (mikolalysenko)
    added
    ci-perfCI / merge-queue performance finding (profiler routine)
    agent:triaged
    priority:p3
    on Oct 8, 2026

  2. mikolalysenko commented on Oct 8, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 8, 2026
    CollaboratorAuthor

    [agent] Triaged as priority:p3 (CI-only). No open PR references this yet; it is not a duplicate of the other CI-perf reports (#1170–#1178 each target a different workflow cost).

    Generated by Claude Code

  3. mikolalysenko commented on Oct 9, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 9, 2026
    CollaboratorAuthor

    Profiler refresh, 2026-10-09 00:17 UTC

    test (windows-latest, 2) became the last job before ci-ok in merge_group runs right after #1143 merged. It was last in 4 of the 6 successful merge_group runs created between 21:35 and 22:25 UTC, at 21–23 min. Since then it has dropped to ~18 min per run. Here is the in-job split: Build takes 2.1–2.4 min (the redundant step this issue targets) and Run tests takes 14.8–15.4 min. Examples: 37858663723, 37859440298.

    Across the 24h sample, Run tests cost 12.5 min per Windows leg (n=143) and 9.7 min per macOS leg (n=117). Removing Build saves ~2 min per leg and up to ~2 min of merge-queue critical path in the runs where this leg finishes last.

    Generated by Claude Code

  4. mikolalysenko commented on Oct 11, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 11, 2026
    CollaboratorAuthor

    [agent] Re-verified against origin/main (578b63e) on 2026-10-11: still technically applies, but the cost is now small. The redundant cargo build --workspace step is still there (ci.yml:352-353), but the test job (macOS + Windows, 2 shards) now runs only on full scope / schedule / dispatch (ci.yml:302), so it costs ~4 legs × ~1.6 min ≈ 6 job-min per nightly run instead of ~930 job-min/day. Leaving open as a low-value cleanup for the nightly tier; it does not affect PR or merge-queue runner usage.

    Generated by Claude Code

  5. mikolalysenko commented on Oct 11, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 11, 2026
    CollaboratorAuthor

    [agent] Claiming this issue (shared root cause: redundant cargo build --workspace step in ci.yml's test job). Branch: agent/fix-ci-test-redundant-build. Claim-ID: 2026-10-11T18:50:49Z-60a4f2

    Generated by Claude Code

  6. Mikola Lysenko (mikolalysenko)
    added
    agent:claimed
    on Oct 11, 2026

  7. mikolalysenko commented on Oct 11, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 11, 2026
    CollaboratorAuthor

    [agent] Fix PR: #1396. The debuginfo half is already fixed on main (Cargo.toml:132, debug = "line-tables-only"); the PR drops the redundant build step.

    Generated by Claude Code

  8. mikolalysenko commented on Oct 11, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 11, 2026
    CollaboratorAuthor

    [agent] Re-verified on origin/main (41659c1) and closing as not planned.

    • Debuginfo: already fixed. [profile.dev] uses debug = "line-tables-only" (Cargo.toml:130-132).
    • "Redundant" cargo build --workspace (ci.yml:375-376): it isn't redundant. On macOS/Windows it is the only step that compiles and links socket-patch-node (cdylib, test = false) and the non-test socket-patch-bench binary. Locally, after the workspace build (209 crates), cargo test --workspace --no-run recompiled only 55, so most of the step's time is reused by Run tests rather than duplicated.
    • Cost: the test job runs only on full scope / schedule / dispatch (ci.yml:325), so PR and merge-queue usage is zero. Removing the step would save well under ~2 min × 4 legs per nightly run and lose the addon link check. A trial removal (CI: drop redundant build in nightly test job (#1173) #1396) was closed for this reason.

    Generated by Claude Code

  9. Mikola Lysenko (mikolalysenko)
    removed
    agent:claimed
    on Oct 11, 2026
  10. Mikola Lysenko (mikolalysenko)
    closed this as not plannedon Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:triagedci-perfCI / merge-queue performance finding (profiler routine)priority:p3

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions