[sentry-junior]

Stacked on #129338 (4 xdist workers, ubuntu-26.04). Land that first.

Runs backend test and acceptance on ubuntu-26.04-arm for pull requests only. Master and every other event stay on amd64 ubuntu-26.04.

Headline: on pull requests, backend test time drops 27% (slowest shard −33%, median job −25%), and acceptance jobs get about 10% faster. Faster Selenium polling (below) takes another 7% off acceptance test time.

Rollout order

Land these in order. Each is stacked on the one before, so after one merges, retarget the next PR's base to master and rebase it. Each can be reverted on its own.

1. ci(backend): Run 4 xdist workers per backend test shard on ubuntu-26.04 #129338: 4 xdist workers, ubuntu-26.04
   • Scope: backend tests on pull requests and master (PRs 2 → 4 workers, master 3 → 4). backend test and acceptance move to x64 ubuntu-26.04. Also adds the Kafka healthcheck shim, the assert-rewrite prewarm, Postgres with durability off, and the resource monitor.
   • Expected impact: backend test time about −6% (±2%) compared with 3 workers on the same code, after correcting for runner CPU type. PRs gain more, since they ran 2 workers. Acceptance jobs about −5% compared with master.
   • getsentry: not affected. It has its own scripts and Namespace runners.
2. ci(backend): Run backend test shards on ubuntu-26.04-arm #129387: Arm runners for pull requests, faster Selenium waits ← this PR
   • Scope: backend test and acceptance run on ubuntu-26.04-arm for pull requests only; master stays on amd64. Also includes the Chromium snap for Arm acceptance, the compiled Arm pydantic wheels in uv.lock, and in selenium.py 0.1s wait polling plus debug capture only on failure.
   • Expected impact on PRs, compared with 4-worker x64: backend test time −27%, slowest shard −33%, median job −25%. Acceptance is about −10% compared with master.
   • Selenium change: cuts acceptance test time about −7% more on Arm. It isn't Arm-specific, so it also applies to master's x64 acceptance runs and to getsentry's acceptance tests, which use the same fixture. The uv.lock change only adds Arm wheels, which x64 installs ignore.
3. ci(backend): Size selective shards by test duration for a ~3 minute slowest job #129398: size selective shards by test duration
   • Scope: only selective backend runs on PRs. Full runs (master, and PRs with Trigger: Override Selective Testing) are unchanged.
   • Expected impact: measured at a 45s per-worker budget, the slowest selective job went from 267–369s to 191–215s (−31% / −41%). The final 40s budget targets about 3 minutes; that's modelled, not measured.
   • Runner time: −23% on a large selection, +40% on a small one that includes the 794s test_organization_agent_token.py.
   • getsentry: not affected. It has its own shard scripts.

Whole stack, on pull requests: backend test time is about −40% compared with master today, the slowest shard about −45% and the median job about −31%. Roughly a quarter of the gain comes from the 4th worker and most of the rest from Arm. Selective PR runs should land near a 3-minute slowest job.

Acceptance results (2 workers, 5 shards)

	master (ubuntu-24.04 x64, 9 runs, median)	#129338 (ubuntu-26.04 x64)	this PR (Arm, 41141328891, 41142216797)
Test step, summed	856s	826–830s	754–778s (−10% vs master)
Slowest test step	200s	172–193s	172–174s (−14%)
Job, median	313s	293–299s	279–281s (−10%)
Job, slowest	335s	314–340s	300–301s (−10%)

The Arm runner image has no Chrome, so acceptance installs the Chromium snap behind google-chrome and chromedriver wrappers. The install takes about 66s, but it runs alongside the rest of setup, so it only delays the ~15s webpack step that waits on it.

Acceptance: faster Selenium waits

src/sentry/testutils/pytest/selenium.py, from the experiment in #129404:

• Browser.wait_until* and wait_until_script_execution poll every 0.1s instead of Selenium's default 0.5s.
• pytest_runtest_makereport captures the URL, screenshot, page source and browser logs only when a test phase fails. It used to do this for every phase of every test.

Arm, 2 workers, 5 shards	before (41142216797)	after (41145962251)
Test step, summed	779s	728s (−7%)
Slowest test step	169s	160s
Job, median / slowest	286s / 308s	275s / 284s
Result	195 passed, 7 skipped	195 passed, 7 skipped, no reruns

This file is shared test code, so it applies on x64 and in getsentry's acceptance tests too. On #129404, getsentry dispatch and getsentry / backend passed. Reusing one Chrome per worker (#129405, #129406) saved more but made the TOTP auth tests flaky, so it isn't included.

Backend results (4 workers, full suite, 30 shards, run at the same time as an x64 baseline)

From the experiment in #129372 (run 41138038705, comparison):

	x64	ARM	Δ
Test step, summed over shards	6,233s	4,550s	−27%
Slowest shard, test step	290s	195s	−33%
Job duration, median	345s	259s	−25%
Job duration, slowest	422s	300s	−29%

The latest full-suite run on this PR (41142216945, rebased on master) matched: 4,620s summed, slowest shard 199s.

• Every ARM shard got the same CPU (Neoverse N2) and averaged 152s of test time. x64 shards depend on which CPU they get: 244s on EPYC 7763 (the most common), 205s on 9V74 and 150s on 9V45. ARM removes that slow tail.
• ubuntu-26.04 on x64 was within noise of 24.04 on the same CPU types (ci(backend): experiment: backend test shards on ubuntu-26.04 #129378), so the speedup comes from the ARM hardware, not the OS image.
• Compiled pydantic on ARM accounts for about 2% of the gain.

Changes

• .github/workflows/backend.yml and acceptance.yml: runs-on: ${{ github.event_name == 'pull_request' && 'ubuntu-26.04-arm' || 'ubuntu-26.04' }} for backend test and acceptance.
• .github/workflows/backend.yml: the odiff download picks the arm64 or x64 binary based on uname -m. It was hard-coded to x64.
• .github/workflows/acceptance.yml: on Arm, install the Chromium snap (retried up to 5 times; the snap store returned a transient HTTP 408 once) and point google-chrome/chromedriver at it.
• .github/workflows/scripts/monitor-resources.py: ARM kernels don't report a model name, so the monitor records the CPU implementer and part IDs instead. A test for this is in test_monitor_resources.py.
• src/sentry/testutils/pytest/selenium.py: 0.1s wait polling and failure-only debug capture (see above).
• uv.lock: adds the compiled linux aarch64 pydantic 1.10.26 wheels, now published on the internal index by build: Build a compiled pydantic 1.10.26 wheel for linux arm64 pypi#2693. Without them, ARM installs pure-Python pydantic and test_pydantic_1x_compiled fails. The change only adds wheel entries, so x64 is unaffected.

All images the backend tests use already have ARM builds. Only image-mirror-google-cloud-sdk is x64-only, and these tests don't use it.

Next in the stack: #129398 sizes selective backend shards for a ~3 minute slowest job on these runners.

Checks

• test_monitor_resources.py passes locally, and backend.yml parses as YAML.
• Before the Selenium commit, CI on this PR was green: backend full suite on all 30 shards and acceptance on all 5 shards on Arm. CI is re-running on 0aa275df.
• selenium.py: ruff check/format and mypy pass locally.

via @joshuarli.

--

View Session [Sentry]