[joshuarli]

Runs backend tests with 4 xdist workers per shard, on both pull requests and master. Before this, pull requests used 2 and master used 3. The PR also removes most of the extra CPU load that made a 4th worker a wash at first, and adds resource monitoring so future tuning can be measured. The backend test and acceptance jobs also move from ubuntu-24.04 to ubuntu-26.04; acceptance stays at 2 workers and gets about 5% faster per job (see Acceptance below).

Next step, stacked on this PR: #129387 moves both jobs to ubuntu-26.04-arm on pull requests, which cuts backend test time by another ~27% and acceptance jobs by ~10%.

Rollout order

Land these in order. Each is stacked on the one before, so after one merges, retarget the next PR's base to master and rebase it. Each can be reverted on its own.

1. ci(backend): Run 4 xdist workers per backend test shard on ubuntu-26.04 #129338: 4 xdist workers, ubuntu-26.04 ← this PR
   • Scope: backend tests on pull requests and master (PRs 2 → 4 workers, master 3 → 4). backend test and acceptance move to x64 ubuntu-26.04. Also adds the Kafka healthcheck shim, the assert-rewrite prewarm, Postgres with durability off, and the resource monitor.
   • Expected impact: backend test time about −6% (±2%) compared with 3 workers on the same code, after correcting for runner CPU type. PRs gain more, since they ran 2 workers. Acceptance jobs about −5% compared with master.
   • getsentry: not affected. It has its own scripts and Namespace runners.
2. ci(backend): Run backend test shards on ubuntu-26.04-arm #129387: Arm runners for pull requests, faster Selenium waits
   • Scope: backend test and acceptance run on ubuntu-26.04-arm for pull requests only; master stays on amd64. Also includes the Chromium snap for Arm acceptance, the compiled Arm pydantic wheels in uv.lock, and in selenium.py 0.1s wait polling plus debug capture only on failure.
   • Expected impact on PRs, compared with 4-worker x64: backend test time −27%, slowest shard −33%, median job −25%. Acceptance is about −10% compared with master.
   • Selenium change: cuts acceptance test time about −7% more on Arm. It isn't Arm-specific, so it also applies to master's x64 acceptance runs and to getsentry's acceptance tests, which use the same fixture. The uv.lock change only adds Arm wheels, which x64 installs ignore.
3. ci(backend): Size selective shards by test duration for a ~3 minute slowest job #129398: size selective shards by test duration
   • Scope: only selective backend runs on PRs. Full runs (master, and PRs with Trigger: Override Selective Testing) are unchanged.
   • Expected impact: measured at a 45s per-worker budget, the slowest selective job went from 267–369s to 191–215s (−31% / −41%). The final 40s budget targets about 3 minutes; that's modelled, not measured.
   • Runner time: −23% on a large selection, +40% on a small one that includes the 794s test_organization_agent_token.py.
   • getsentry: not affected. It has its own shard scripts.

Whole stack, on pull requests: backend test time is about −40% compared with master today, the slowest shard about −45% and the median job about −31%. Roughly a quarter of the gain comes from the 4th worker and most of the rest from Arm. Selective PR runs should land near a 3-minute slowest job.

Result

With 4 workers, total test-step time is about 6% lower than with 3 workers on the same code, after correcting for runner CPU type (margin of error about ±2%). In every comparison, the slowest shard was also faster.

The 5th worker adds nothing (see the experiments below), so 4 is the sweet spot on these 4-vCPU, 16 GB runners.

Test step time (Run backend test) across all 30 shards:

Run	Workers	Code	Test step, summed over 30 shards	Slowest shard
master, several recent runs	3	master	7,000–7,500s	321–324s
41108345005	4	before any fixes	7,479s	322s
41123286212	3	final code	7,057s	307s
41127942844	3	final code	6,947s	289s
41119271090	4	Kafka + prewarm fix	6,535s	259s
41121010265	4	same + autovacuum off (reverted)	6,397s	283s
41124537028 attempt 1	4	final, 226e1191	6,194s	282s
41124537028 attempt 2	4	final, 226e1191	6,533s	253s

Whole job on the final code: median 344–353s, slowest 394–408s. Master's slowest job was around 430s. Setup takes about 100s with either worker count.

Runner hardware matters more than you'd think. GitHub-hosted ubuntu-24.04 runners came with four CPU types. Fitting test time per shard and per CPU model gives these relative times (lower is faster):

• AMD EPYC 7763: 1.00, and the most common
• Intel Xeon 8370C: 0.80
• AMD EPYC 9V74: 0.82
• AMD EPYC 9V45: 0.64

The same shard can take up to 1.9× longer depending on the runner it lands on, so single-run wall times are only directional. The resource monitor now records each shard's CPU model so runs can be compared fairly.

Acceptance

acceptance moves to ubuntu-26.04 too, and gets the same resource monitor and Kafka healthcheck. It stays at 2 xdist workers, because more workers made it slower:

Acceptance, 5 shards, x64	Test step, summed	Slowest test step	Median job	Slowest job
master (ubuntu-24.04, 9 recent runs, median)	856s	200s	313s	335s
this PR (41140601391, 41142191454)	826–830s (−3%)	172–193s (−9%)	293–299s (−5%)	314–340s (−2%)

Worker count, with all runs at 10 shards and corrected for runner CPU type (residual noise about 5% per shard):

Workers	Test step	Job
2 (#129395)	baseline	baseline
3 (#129389, two runs)	−2.3% (noise)	+4.8%
4 (#129390)	+5.3%	+14.0%

The runner is already CPU-bound with Chrome plus per-worker Snuba, so each extra worker mostly adds another Snuba bootstrap. On Arm (#129387) acceptance gets about 10% faster across the board.

What changed

• XDIST_WORKERS: '4' for backend tests on pull requests and master.
• Assert-rewrite prewarm (prewarm_assert_rewrite.py, started from pytest-selected.py). Every xdist worker used to rewrite and compile the shard's test files, conftests and src/sentry/testutils at the same moment. That was about 11.5s of CPU per worker, all at startup. A background process now writes pytest's rewritten pycs once while the workers start up. It compiles with the same absolute path pytest uses, because pytest doesn't fix co_filename in cached code. An earlier version used relative paths, which changed traceback paths and broke test_thread_leak_strict_mode_raises. test_prewarm_assert_rewrite.py runs real pytest against a prewarmed file to guard this. The prewarm only runs with xdist.
• Kafka healthcheck (kafka-topics-healthcheck.sh). sentry-shared-kafka's healthcheck runs kafka-topics --list every 5s, which starts a JVM each time. That cost about 0.35 cores during tests. After everything that waits on Kafka's health has finished, CI answers that exact command with a TCP probe. Any other kafka-topics call still runs the real binary. Kafka dropped to 0.02 cores and still reports healthy. The step is continue-on-error. Used in both backend and acceptance.
• Test Postgres durability off: fsync=off, synchronous_commit=off, full_page_writes=off. This is a throwaway CI database.
• Resource monitor (monitor-resources.py, uploaded as backend-resource-usage-<run>-<shard> and acceptance-resource-usage-<run>-<shard>). Every 5s it records:
  • CPU, memory, swap and load
  • CPU pressure (PSI some; the full line is always 0 system-wide)
  • user/system/iowait/steal split
  • CPU-seconds per container, per xdist worker and for the pytest controller
  • CPU from short-lived processes, read from cgroup cpu.stat
  • the runner's CPU model
• runs-on: ubuntu-26.04 for backend test and acceptance. On the same CPU types it was within ~2% of 24.04, which is noise (ci(backend): experiment: backend test shards on ubuntu-26.04 #129378, results). It's here as the upgrade, and because the ARM runners in ci(backend): Run backend test shards on ubuntu-26.04-arm #129387 are 26.04.
• py-spy profiling is disabled in CI. The Install py-spy step is commented out in backend.yml, and profile-backend-tests.sh stays in the repo for ad-hoc profiling. The comment above the step says how to turn it back on. It costs about 0.37 cores and skews timings.

Where the CPU goes now

The runner is about 93% busy during tests, with CPU pressure around 34. Mean cores per shard during the test phase:

	Before (4 workers, first instrumented runs)	After
xdist workers	2.17	2.38–2.39
Postgres (live + exited backends)	about 0.55	0.60
ClickHouse	0.35	0.38
Kafka (mostly healthcheck JVMs)	0.35	0.02
Snuba gateways (4 combined)	0.16	0.17

Postgres's "exited" CPU is query work done on test connections that later closed. It isn't autovacuum: turning autovacuum off changed nothing, so that change was reverted.

From py-spy (first profiling run):

• collection is about 20% of worker samples, mostly imports
• assertion rewriting was about 11%, now prewarmed
• hybrid-cloud outbox draining and create_organization fixtures are large chunks of test time

Experiments (stacked draft PRs, run at the same time on the same base)

Backend experiments ran the full suite, green on all 30 shards. Results are normalized for runner CPU model against a same-time rerun of this PR (4 workers). The margin of error is about ±2%.

PR	Change	Result
#129365	5 workers	-0.1%, no gain. CPU pressure rose from 34 to 49, workers got 2.48 cores vs. 2.38, the slowest shard (adjusted for CPU type) was 305s vs. 282s, and setup took about 7s longer. Memory peaked at 43.5% with 8.8 GB free, so CPU is the limit, not memory.
#129366	3 workers	+6.1% slower (about 3× the margin of error). The slowest shard (adjusted for CPU type) was 353s vs. 282s.
#129367	PIECES_PER_SHARD 48 → 24	Total 1.2% lower, which is inside the noise. The slowest shard was worse (309s vs. 282s), so this stays at 48.
#129369	ClickHouse log_queries=0 and query profiler off via users.d	Didn't take effect: query_log still recorded about 23k rows per shard, most likely because Snuba's connections already existed. ClickHouse CPU went from 0.382 to 0.373 cores, which is noise.
#129378	ubuntu-26.04 x64	Within ~2% of 24.04 on the same CPU types, which is noise. Adopted here as the OS upgrade.
#129389	acceptance, 3 workers	Test step −2.3% (noise), job +4.8%. Stays at 2.
#129390	acceptance, 4 workers	Test step +5.3%, job +14.0%.
#129395	acceptance, 2 workers at 10 shards	Control for the two rows above.
#129372	ubuntu-26.04-arm	Summed test time −27%, slowest shard −33%, median job −25%. Shipping in #129387.

Earlier single runs on this branch also showed:

• py-spy overhead: 0.37 cores
• Postgres autovacuum=off: no effect, reverted

Follow-ups (not in this PR)

• Move backend test and acceptance to ubuntu-26.04-arm on pull requests: ci(backend): Run backend test shards on ubuntu-26.04-arm #129387, stacked on this PR. ci(backend): Size selective shards by test duration for a ~3 minute slowest job #129398 then sizes selective shards for a ~3 minute slowest job. In the ci(backend): experiment: backend test shards on ubuntu-26.04-arm #129372 experiment, ARM cut summed test time 6,233s → 4,550s (−27%), the slowest shard 290s → 195s (−33%) and the median job 345s → 259s (−25%), all with 4 workers.
• Make sentry-shared-kafka's healthcheck cheap or less frequent upstream, for example with a long interval plus start_interval. That would replace the CI shim and also help local dev.
• ClickHouse uses about 0.4 cores. Its queries (mostly about 5,700 test-reset truncates and inserts per shard) account for a bit over half of that. Cutting the background part would need a server config change plus a restart before the per-worker bootstrap. At most about 0.15 cores is in play.
• The remaining worker CPU is test and fixture work, such as outbox draining and org creation. Reducing it needs test-code changes.

via @joshuarli.

--

View Session [Sentry]