Repository navigation
ci: experiment: time a selective backend run on Arm (issues endpoints) - #129391
Closed
sentry-junior[bot] wants to merge 1 commit into
Closed
sentry-junior[bot] wants to merge 1 commit into
sentry-junior[bot] wants to merge 1 commit into
Conversation
github-actions
Bot
added
the
Scope: Backend
sentry-junior
Bot
force-pushed
the
ci/backend-tests-arm
branch
from
October 30, 2026 23:04
0d3bafe to
26f6011
Compare
sentry-junior
Bot
force-pushed
the
ci/xdist-exp-selective-issues
branch
from
October 30, 2026 23:05
c1c0888 to
bd6d4dc
Compare
Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
sentry-junior
Bot
force-pushed
the
ci/xdist-exp-selective-issues
branch
from
October 30, 2026 23:20
bd6d4dc to
155e36f
Compare
sentry-junior
Bot
changed the base branch from
ci/backend-tests-arm
to
ci/backend-selective-shard-timing
October 30, 2026 23:20
Contributor
Author
|
Result: closing. The data from this PR drove #129398. Same selected files (51 files: 2,894 passed, 17 xfailed in every run), Arm, 4 workers:
Before, 5 of the 9 shards each ran a slice of |
joshuarli
added a commit
that referenced
this pull request
Nov 2, 2026
…lowest job (#129398) Stacked on #129387 (Arm runners for PRs), which is stacked on #129338 (4 xdist workers). Aims for a slowest selective backend job of about 3 minutes on the Arm PR runners. **Headline:** on two real PR selections, the slowest selective backend job dropped from 267–369s to 191–215s (−31% and −41%), running exactly the same tests. ## Rollout order Land these in order. Each is stacked on the one before, so after one merges, retarget the next PR's base to `master` and rebase it. Each can be reverted on its own. 1. **#129338: 4 xdist workers, `ubuntu-26.04`** - **Scope:** backend tests on pull requests and master (PRs 2 → 4 workers, master 3 → 4). `backend test` and `acceptance` move to x64 `ubuntu-26.04`. Also adds the Kafka healthcheck shim, the assert-rewrite prewarm, Postgres with durability off, and the resource monitor. - **Expected impact:** backend test time about **−6% (±2%)** compared with 3 workers on the same code, after correcting for runner CPU type. PRs gain more, since they ran 2 workers. Acceptance jobs about **−5%** compared with master. - **getsentry:** not affected. It has its own scripts and Namespace runners. 2. **#129387: Arm runners for pull requests, faster Selenium waits** - **Scope:** `backend test` and `acceptance` run on `ubuntu-26.04-arm` **for pull requests only**; master stays on amd64. Also includes the Chromium snap for Arm acceptance, the compiled Arm pydantic wheels in `uv.lock`, and in `selenium.py` 0.1s wait polling plus debug capture only on failure. - **Expected impact on PRs, compared with 4-worker x64:** backend test time **−27%**, slowest shard **−33%**, median job **−25%**. Acceptance is about **−10%** compared with master. - **Selenium change:** cuts acceptance test time about **−7%** more on Arm. It isn't Arm-specific, so it also applies to master's x64 acceptance runs and to getsentry's acceptance tests, which use the same fixture. The `uv.lock` change only adds Arm wheels, which x64 installs ignore. 3. **#129398: size selective shards by test duration** **← this PR** - **Scope:** only selective backend runs on PRs. Full runs (master, and PRs with `Trigger: Override Selective Testing`) are unchanged. - **Expected impact:** measured at a 45s per-worker budget, the slowest selective job went from 267–369s to **191–215s** (−31% / −41%). The final 40s budget targets about **3 minutes**; that's modelled, not measured. - **Runner time:** −23% on a large selection, +40% on a small one that includes the 794s `test_organization_agent_token.py`. - **getsentry:** not affected. It has its own shard scripts. **Whole stack, on pull requests:** backend test time is about **−40%** compared with master today, the slowest shard about −45% and the median job about −31%. Roughly a quarter of the gain comes from the 4th worker and most of the rest from Arm. Selective PR runs should land near a 3-minute slowest job. ## Why test counts weren't enough Two selective runs on #129387 (files from #129352 and #129254): | | #129391 (51 files) | #129392 (173 files) | |---|---|---| | Shards (2,550 / 6,893 tests ÷ 300) | 9 | 23 | | Slowest test step / median | 161s / 143s | 261s / 99s | | Slowest job | 267s | 369s | - **Shards are packed by test count, not time.** The 369s job had 12 files worth 468s in the durations file; the median shard had about 150s. - **A split file runs on one xdist worker.** With `--dist=loadfile`, each shard that got a slice of `tests/sentry/seer/endpoints/test_organization_agent_token.py` (794s, picked by both PRs) ran it on 1 of 4 workers for ~150s. - **About 145s of every job is fixed**: ~100s setup, ~10s teardown and ~35s pytest startup and collection. A 3-minute job leaves each worker about 40–45s of tests, timed as in `backend-test-durations.json`. Arm runs them about 25% faster than that. ## What changes - `pytest-selected.py`: selective runs use the same plan as full runs, `plan_full_shards`, over just the selected files and weighted by duration. Slow files get sliced across shards so every slice can go to a different worker. Slices are coarser than in full runs (`SELECTED_PIECES_PER_SHARD = 8` instead of 48), so shards don't all import the same files. Without the durations file, it falls back to the old count-based plan. - `calculate-backend-test-shards.py`: when it has the selected files, it picks enough shards that no xdist worker gets more than `WORKER_SECONDS_PER_SHARD = 40` seconds, and enough for the slowest file's slices to fit too. The 30-shard cap stays. If the selected files are missing, the item-count path (`TESTS_PER_SHARD = 300`) is still the fallback. - `count_test_items.py`: `estimate_file_seconds`, shared by both scripts. It's the existing `file_weights` logic moved so `calculate-backend-test-shards.py` can use it without pytest. - `backend.yml`: `calculate test shards` downloads the selected-tests artifact. Full runs (master, and PRs with `Trigger: Override Selective Testing`) are unchanged. ## Measured with a 45s budget (both experiments re-run on this branch, same selected files, Arm, 4 workers) | | #129391 files: before (2 runs) | after ([41143869940](https://github.com/getsentry/sentry/actions/runs/41143869940)) | #129392 files: before (2 runs) | after ([41143871561](https://github.com/getsentry/sentry/actions/runs/41143871561)) | |---|---|---|---|---| | Shards | 9 | 18 | 23–24 | 22 | | Slowest job | 267–288s | **191s (−31%)** | 358–369s | **215s (−41%)** | | Median job | 247–253s | 163s | 202–207s | 178s | | Slowest test step | 161–172s | 75s | 254–261s | 111s | | Test step, summed | 1,107–1,157s | 1,162s | 2,515–2,632s | 1,697s | | Runner minutes | 35–36 | 49 (+40%) | 84–86 | 66 (−23%) | | Tests run | 2,894 passed, 17 xfailed | identical | 7,231 passed, 20 skipped, 21 xfailed, 2 xpassed | identical | - The model predicted ~184s and ~193s. The real slowest jobs landed 7–22s above that, so the final commit lowers `WORKER_SECONDS_PER_SHARD` from 45 to **40**. On the same two selections that plans 20 and 25 shards (from 18 and 22), to put the slowest job at about 3 minutes. That last step is modelled, not re-measured. - The bigger selection gets cheaper as well as faster, because shards no longer wait on a single worker. - The smaller one costs more runner time, all from slicing `test_organization_agent_token.py` (794s) into 18 pieces. Speeding up that file would cut both the shard count and about 5% of all backend CI time. <!-- junior-request-attribution:start --> via @joshuarli. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AD0BQC9WSQ2E%3A1793386300.592079 --> -- [View Session](https://junior-prod.sentry.dev/conversations/slack%3AD0BQC9WSQ2E%3A1793386300.592079) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AD0BQC9WSQ2E%3A1793386300.592079/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Joshua Li <14209781+joshuarli@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment, not for merge. Stacked on #129387.
Adds a trailing comment to the same
.pyfiles #129352 changed, so selective testing picks a comparable set of tests. That PR ran 9 selective shards on amd64 with 2 workers. This runs the same selection on Arm with 4 workers, to calibrateTESTS_PER_SHARDfor a ~3-minute slowest shard. The results will be posted here, then this PR will be closed.via @joshuarli.
--
View Session [Sentry]