Repository navigation
[experiment] ci(backend): 5 xdist workers - #129365
Closed
sentry-junior[bot] wants to merge 1 commit into
Closed
sentry-junior[bot] wants to merge 1 commit into
sentry-junior[bot] wants to merge 1 commit into
Conversation
Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
github-actions
Bot
added
the
Scope: Backend
sentry-junior
Bot
added
the
Trigger: Override Selective Testing
Contributor
Author
|
Result: no gain. Normalized for runner CPU model, 5 workers came out -0.1% vs. 4 workers (SE about 2%). CPU pressure rose from 34 to 49, setup took about 7s longer, and the slowest model-adjusted shard was worse (305s vs. 282s). Raw times looked faster only because this run happened to get 8 of the fastest runners, against 1 for the 4-worker run. Closing; 4 stays. |
joshuarli
added a commit
that referenced
this pull request
Nov 2, 2026
…04 (#129338) Runs backend tests with **4 xdist workers per shard**, on both pull requests and master. Before this, pull requests used 2 and master used 3. The PR also removes most of the extra CPU load that made a 4th worker a wash at first, and adds resource monitoring so future tuning can be measured. The `backend test` and `acceptance` jobs also move from `ubuntu-24.04` to `ubuntu-26.04`; acceptance stays at 2 workers and gets about 5% faster per job (see Acceptance below). Next step, stacked on this PR: #129387 moves both jobs to `ubuntu-26.04-arm` on pull requests, which cuts backend test time by another ~27% and acceptance jobs by ~10%. ## Rollout order Land these in order. Each is stacked on the one before, so after one merges, retarget the next PR's base to `master` and rebase it. Each can be reverted on its own. 1. **#129338: 4 xdist workers, `ubuntu-26.04`** **← this PR** - **Scope:** backend tests on pull requests and master (PRs 2 → 4 workers, master 3 → 4). `backend test` and `acceptance` move to x64 `ubuntu-26.04`. Also adds the Kafka healthcheck shim, the assert-rewrite prewarm, Postgres with durability off, and the resource monitor. - **Expected impact:** backend test time about **−6% (±2%)** compared with 3 workers on the same code, after correcting for runner CPU type. PRs gain more, since they ran 2 workers. Acceptance jobs about **−5%** compared with master. - **getsentry:** not affected. It has its own scripts and Namespace runners. 2. **#129387: Arm runners for pull requests, faster Selenium waits** - **Scope:** `backend test` and `acceptance` run on `ubuntu-26.04-arm` **for pull requests only**; master stays on amd64. Also includes the Chromium snap for Arm acceptance, the compiled Arm pydantic wheels in `uv.lock`, and in `selenium.py` 0.1s wait polling plus debug capture only on failure. - **Expected impact on PRs, compared with 4-worker x64:** backend test time **−27%**, slowest shard **−33%**, median job **−25%**. Acceptance is about **−10%** compared with master. - **Selenium change:** cuts acceptance test time about **−7%** more on Arm. It isn't Arm-specific, so it also applies to master's x64 acceptance runs and to getsentry's acceptance tests, which use the same fixture. The `uv.lock` change only adds Arm wheels, which x64 installs ignore. 3. **#129398: size selective shards by test duration** - **Scope:** only selective backend runs on PRs. Full runs (master, and PRs with `Trigger: Override Selective Testing`) are unchanged. - **Expected impact:** measured at a 45s per-worker budget, the slowest selective job went from 267–369s to **191–215s** (−31% / −41%). The final 40s budget targets about **3 minutes**; that's modelled, not measured. - **Runner time:** −23% on a large selection, +40% on a small one that includes the 794s `test_organization_agent_token.py`. - **getsentry:** not affected. It has its own shard scripts. **Whole stack, on pull requests:** backend test time is about **−40%** compared with master today, the slowest shard about −45% and the median job about −31%. Roughly a quarter of the gain comes from the 4th worker and most of the rest from Arm. Selective PR runs should land near a 3-minute slowest job. ## Result With 4 workers, total test-step time is **about 6% lower than with 3 workers on the same code**, after correcting for runner CPU type (margin of error about ±2%). In every comparison, the slowest shard was also faster. The 5th worker adds nothing (see the experiments below), so 4 is the sweet spot on these 4-vCPU, 16 GB runners. Test step time (`Run backend test`) across all 30 shards: | Run | Workers | Code | Test step, summed over 30 shards | Slowest shard | |---|---|---|---|---| | master, several recent runs | 3 | master | 7,000–7,500s | 321–324s | | [41108345005](https://github.com/getsentry/sentry/actions/runs/41108345005) | 4 | before any fixes | 7,479s | 322s | | [41123286212](https://github.com/getsentry/sentry/actions/runs/41123286212) | 3 | final code | 7,057s | 307s | | [41127942844](https://github.com/getsentry/sentry/actions/runs/41127942844) | 3 | final code | 6,947s | 289s | | [41119271090](https://github.com/getsentry/sentry/actions/runs/41119271090) | 4 | Kafka + prewarm fix | 6,535s | 259s | | [41121010265](https://github.com/getsentry/sentry/actions/runs/41121010265) | 4 | same + autovacuum off (reverted) | 6,397s | 283s | | [41124537028](https://github.com/getsentry/sentry/actions/runs/41124537028) attempt 1 | 4 | final, `226e1191` | 6,194s | 282s | | [41124537028](https://github.com/getsentry/sentry/actions/runs/41124537028) attempt 2 | 4 | final, `226e1191` | 6,533s | 253s | Whole job on the final code: median 344–353s, slowest 394–408s. Master's slowest job was around 430s. Setup takes about 100s with either worker count. **Runner hardware matters more than you'd think.** GitHub-hosted `ubuntu-24.04` runners came with four CPU types. Fitting test time per shard and per CPU model gives these relative times (lower is faster): - AMD EPYC 7763: 1.00, and the most common - Intel Xeon 8370C: 0.80 - AMD EPYC 9V74: 0.82 - AMD EPYC 9V45: 0.64 The same shard can take up to 1.9× longer depending on the runner it lands on, so single-run wall times are only directional. The resource monitor now records each shard's CPU model so runs can be compared fairly. ## Acceptance `acceptance` moves to `ubuntu-26.04` too, and gets the same resource monitor and Kafka healthcheck. It stays at **2 xdist workers**, because more workers made it slower: | Acceptance, 5 shards, x64 | Test step, summed | Slowest test step | Median job | Slowest job | |---|---|---|---|---| | master (`ubuntu-24.04`, 9 recent runs, median) | 856s | 200s | 313s | 335s | | this PR ([41140601391](https://github.com/getsentry/sentry/actions/runs/41140601391), [41142191454](https://github.com/getsentry/sentry/actions/runs/41142191454)) | 826–830s (−3%) | 172–193s (−9%) | 293–299s (−5%) | 314–340s (−2%) | Worker count, with all runs at 10 shards and corrected for runner CPU type (residual noise about 5% per shard): | Workers | Test step | Job | |---|---|---| | 2 (#129395) | baseline | baseline | | 3 (#129389, two runs) | −2.3% (noise) | +4.8% | | 4 (#129390) | +5.3% | +14.0% | The runner is already CPU-bound with Chrome plus per-worker Snuba, so each extra worker mostly adds another Snuba bootstrap. On Arm (#129387) acceptance gets about 10% faster across the board. ## What changed - **`XDIST_WORKERS: '4'`** for backend tests on pull requests and master. - **Assert-rewrite prewarm** (`prewarm_assert_rewrite.py`, started from `pytest-selected.py`). Every xdist worker used to rewrite and compile the shard's test files, conftests and `src/sentry/testutils` at the same moment. That was about 11.5s of CPU per worker, all at startup. A background process now writes pytest's rewritten pycs once while the workers start up. It compiles with the same absolute path pytest uses, because pytest doesn't fix `co_filename` in cached code. An earlier version used relative paths, which changed traceback paths and broke `test_thread_leak_strict_mode_raises`. `test_prewarm_assert_rewrite.py` runs real pytest against a prewarmed file to guard this. The prewarm only runs with xdist. - **Kafka healthcheck** (`kafka-topics-healthcheck.sh`). `sentry-shared-kafka`'s healthcheck runs `kafka-topics --list` every 5s, which starts a JVM each time. That cost about **0.35 cores** during tests. After everything that waits on Kafka's health has finished, CI answers that exact command with a TCP probe. Any other `kafka-topics` call still runs the real binary. Kafka dropped to 0.02 cores and still reports healthy. The step is `continue-on-error`. Used in both backend and acceptance. - **Test Postgres durability off**: `fsync=off`, `synchronous_commit=off`, `full_page_writes=off`. This is a throwaway CI database. - **Resource monitor** (`monitor-resources.py`, uploaded as `backend-resource-usage-<run>-<shard>` and `acceptance-resource-usage-<run>-<shard>`). Every 5s it records: - CPU, memory, swap and load - CPU pressure (PSI `some`; the `full` line is always 0 system-wide) - user/system/iowait/steal split - CPU-seconds per container, per xdist worker and for the pytest controller - CPU from short-lived processes, read from cgroup `cpu.stat` - the runner's CPU model - **`runs-on: ubuntu-26.04`** for `backend test` and `acceptance`. On the same CPU types it was within ~2% of 24.04, which is noise (#129378, [results](#129378 (comment))). It's here as the upgrade, and because the ARM runners in #129387 are 26.04. - **py-spy profiling is disabled in CI.** The `Install py-spy` step is commented out in `backend.yml`, and `profile-backend-tests.sh` stays in the repo for ad-hoc profiling. The comment above the step says how to turn it back on. It costs about 0.37 cores and skews timings. ## Where the CPU goes now The runner is about 93% busy during tests, with CPU pressure around 34. Mean cores per shard during the test phase: | | Before (4 workers, first instrumented runs) | After | |---|---|---| | xdist workers | 2.17 | 2.38–2.39 | | Postgres (live + exited backends) | about 0.55 | 0.60 | | ClickHouse | 0.35 | 0.38 | | Kafka (mostly healthcheck JVMs) | 0.35 | 0.02 | | Snuba gateways (4 combined) | 0.16 | 0.17 | Postgres's "exited" CPU is query work done on test connections that later closed. It isn't autovacuum: turning autovacuum off changed nothing, so that change was reverted. From py-spy (first profiling run): - collection is about 20% of worker samples, mostly imports - assertion rewriting was about 11%, now prewarmed - hybrid-cloud outbox draining and `create_organization` fixtures are large chunks of test time ## Experiments (stacked draft PRs, run at the same time on the same base) Backend experiments ran the full suite, green on all 30 shards. Results are normalized for runner CPU model against a same-time rerun of this PR (4 workers). The margin of error is about ±2%. | PR | Change | Result | |---|---|---| | #129365 | 5 workers | **-0.1%**, no gain. CPU pressure rose from 34 to 49, workers got 2.48 cores vs. 2.38, the slowest shard (adjusted for CPU type) was 305s vs. 282s, and setup took about 7s longer. Memory peaked at 43.5% with 8.8 GB free, so CPU is the limit, not memory. | | #129366 | 3 workers | **+6.1%** slower (about 3× the margin of error). The slowest shard (adjusted for CPU type) was 353s vs. 282s. | | #129367 | `PIECES_PER_SHARD` 48 → 24 | Total 1.2% lower, which is inside the noise. The slowest shard was worse (309s vs. 282s), so this stays at 48. | | #129369 | ClickHouse `log_queries=0` and query profiler off via `users.d` | Didn't take effect: query_log still recorded about 23k rows per shard, most likely because Snuba's connections already existed. ClickHouse CPU went from 0.382 to 0.373 cores, which is noise. | | #129378 | `ubuntu-26.04` x64 | Within ~2% of 24.04 on the same CPU types, which is noise. Adopted here as the OS upgrade. | | #129389 | acceptance, 3 workers | Test step −2.3% (noise), job +4.8%. Stays at 2. | | #129390 | acceptance, 4 workers | Test step +5.3%, job +14.0%. | | #129395 | acceptance, 2 workers at 10 shards | Control for the two rows above. | | #129372 | `ubuntu-26.04-arm` | Summed test time −27%, slowest shard −33%, median job −25%. Shipping in #129387. | Earlier single runs on this branch also showed: - py-spy overhead: 0.37 cores - Postgres `autovacuum=off`: no effect, reverted ## Follow-ups (not in this PR) - Move `backend test` and `acceptance` to `ubuntu-26.04-arm` on pull requests: #129387, stacked on this PR. #129398 then sizes selective shards for a ~3 minute slowest job. In the #129372 experiment, ARM cut summed test time 6,233s → 4,550s (−27%), the slowest shard 290s → 195s (−33%) and the median job 345s → 259s (−25%), all with 4 workers. - Make `sentry-shared-kafka`'s healthcheck cheap or less frequent upstream, for example with a long `interval` plus `start_interval`. That would replace the CI shim and also help local dev. - ClickHouse uses about 0.4 cores. Its queries (mostly about 5,700 test-reset truncates and inserts per shard) account for a bit over half of that. Cutting the background part would need a server config change plus a restart before the per-worker bootstrap. At most about 0.15 cores is in play. - The remaining worker CPU is test and fixture work, such as outbox draining and org creation. Reducing it needs test-code changes. <!-- junior-request-attribution:start --> via @joshuarli. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AD0BQC9WSQ2E%3A1793386300.592079 --> -- [View Session](https://junior-prod.sentry.dev/conversations/slack%3AD0BQC9WSQ2E%3A1793386300.592079) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AD0BQC9WSQ2E%3A1793386300.592079/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge. A stacked experiment on #129338: it changes only
XDIST_WORKERSfrom 4 to 5.The question is whether the spare memory (about 10 GB free at 4 workers) lets a 5th worker help, even though the runner is already about 93% CPU-busy. Compare the summed test-step time and the slowest shard against #129338's run on
226e1191.via @joshuarli.
--
View Session [Sentry]