Repository navigation
ci(acceptance): experiment: poll waits every 0.1s, debug capture only on failure - #129404
Closed
sentry-junior[bot] wants to merge 1 commit into
Closed
sentry-junior[bot] wants to merge 1 commit into
sentry-junior[bot] wants to merge 1 commit into
Conversation
… only on failure Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
github-actions
Bot
added
the
Scope: Backend
Contributor
Author
Experiment results (Arm, 5 acceptance shards)Acceptance test-step seconds per shard, from the GitHub Actions step timestamps:
Flakiness: all 4 Chrome-reuse runs had a rerun on shard 3, and 3 of them were Other checks: Backend, backend typing (after the fix on #129405/#129406), Cost: each job still spends about 2 minutes on setup, so job time drops less than test time. The job median goes from 286s to 275s (poll), 241–246s (reuse), and 237s (reuse + 3 workers). This PR (#129404): a small, clean win with no new flakes. Recommended to fold into the stack. Closing as an experiment. |
This was referenced Oct 31, 2026
joshuarli
added a commit
that referenced
this pull request
Nov 2, 2026
Stacked on #129338 (4 xdist workers, ubuntu-26.04). Land that first. Runs `backend test` and `acceptance` on `ubuntu-26.04-arm` **for pull requests only**. Master and every other event stay on amd64 `ubuntu-26.04`. **Headline:** on pull requests, backend test time drops 27% (slowest shard −33%, median job −25%), and acceptance jobs get about 10% faster. Faster Selenium polling (below) takes another 7% off acceptance test time. ## Rollout order Land these in order. Each is stacked on the one before, so after one merges, retarget the next PR's base to `master` and rebase it. Each can be reverted on its own. 1. **#129338: 4 xdist workers, `ubuntu-26.04`** - **Scope:** backend tests on pull requests and master (PRs 2 → 4 workers, master 3 → 4). `backend test` and `acceptance` move to x64 `ubuntu-26.04`. Also adds the Kafka healthcheck shim, the assert-rewrite prewarm, Postgres with durability off, and the resource monitor. - **Expected impact:** backend test time about **−6% (±2%)** compared with 3 workers on the same code, after correcting for runner CPU type. PRs gain more, since they ran 2 workers. Acceptance jobs about **−5%** compared with master. - **getsentry:** not affected. It has its own scripts and Namespace runners. 2. **#129387: Arm runners for pull requests, faster Selenium waits** **← this PR** - **Scope:** `backend test` and `acceptance` run on `ubuntu-26.04-arm` **for pull requests only**; master stays on amd64. Also includes the Chromium snap for Arm acceptance, the compiled Arm pydantic wheels in `uv.lock`, and in `selenium.py` 0.1s wait polling plus debug capture only on failure. - **Expected impact on PRs, compared with 4-worker x64:** backend test time **−27%**, slowest shard **−33%**, median job **−25%**. Acceptance is about **−10%** compared with master. - **Selenium change:** cuts acceptance test time about **−7%** more on Arm. It isn't Arm-specific, so it also applies to master's x64 acceptance runs and to getsentry's acceptance tests, which use the same fixture. The `uv.lock` change only adds Arm wheels, which x64 installs ignore. 3. **#129398: size selective shards by test duration** - **Scope:** only selective backend runs on PRs. Full runs (master, and PRs with `Trigger: Override Selective Testing`) are unchanged. - **Expected impact:** measured at a 45s per-worker budget, the slowest selective job went from 267–369s to **191–215s** (−31% / −41%). The final 40s budget targets about **3 minutes**; that's modelled, not measured. - **Runner time:** −23% on a large selection, +40% on a small one that includes the 794s `test_organization_agent_token.py`. - **getsentry:** not affected. It has its own shard scripts. **Whole stack, on pull requests:** backend test time is about **−40%** compared with master today, the slowest shard about −45% and the median job about −31%. Roughly a quarter of the gain comes from the 4th worker and most of the rest from Arm. Selective PR runs should land near a 3-minute slowest job. ## Acceptance results (2 workers, 5 shards) | | master (`ubuntu-24.04` x64, 9 runs, median) | #129338 (`ubuntu-26.04` x64) | this PR (Arm, [41141328891](https://github.com/getsentry/sentry/actions/runs/41141328891), [41142216797](https://github.com/getsentry/sentry/actions/runs/41142216797)) | | --- | ---: | ---: | ---: | | Test step, summed | 856s | 826–830s | 754–778s (−10% vs master) | | Slowest test step | 200s | 172–193s | 172–174s (−14%) | | Job, median | 313s | 293–299s | 279–281s (−10%) | | Job, slowest | 335s | 314–340s | 300–301s (−10%) | The Arm runner image has no Chrome, so acceptance installs the Chromium snap behind `google-chrome` and `chromedriver` wrappers. The install takes about 66s, but it runs alongside the rest of setup, so it only delays the ~15s webpack step that waits on it. ## Acceptance: faster Selenium waits `src/sentry/testutils/pytest/selenium.py`, from the experiment in #129404: - `Browser.wait_until*` and `wait_until_script_execution` poll every 0.1s instead of Selenium's default 0.5s. - `pytest_runtest_makereport` captures the URL, screenshot, page source and browser logs only when a test phase fails. It used to do this for every phase of every test. | Arm, 2 workers, 5 shards | before ([41142216797](https://github.com/getsentry/sentry/actions/runs/41142216797/attempts/2)) | after ([41145962251](https://github.com/getsentry/sentry/actions/runs/41145962251)) | | --- | ---: | ---: | | Test step, summed | 779s | 728s (−7%) | | Slowest test step | 169s | 160s | | Job, median / slowest | 286s / 308s | 275s / 284s | | Result | 195 passed, 7 skipped | 195 passed, 7 skipped, no reruns | This file is shared test code, so it applies on x64 and in getsentry's acceptance tests too. On #129404, `getsentry dispatch` and `getsentry / backend` passed. Reusing one Chrome per worker (#129405, #129406) saved more but made the TOTP auth tests flaky, so it isn't included. ## Backend results (4 workers, full suite, 30 shards, run at the same time as an x64 baseline) From the experiment in #129372 ([run 41138038705](https://github.com/getsentry/sentry/actions/runs/41138038705), [comparison](#129372 (comment))): | | x64 | ARM | Δ | | --- | ---: | ---: | ---: | | Test step, summed over shards | 6,233s | 4,550s | −27% | | Slowest shard, test step | 290s | 195s | −33% | | Job duration, median | 345s | 259s | −25% | | Job duration, slowest | 422s | 300s | −29% | The latest full-suite run on this PR ([41142216945](https://github.com/getsentry/sentry/actions/runs/41142216945), rebased on master) matched: 4,620s summed, slowest shard 199s. - Every ARM shard got the same CPU (Neoverse N2) and averaged 152s of test time. x64 shards depend on which CPU they get: 244s on EPYC 7763 (the most common), 205s on 9V74 and 150s on 9V45. ARM removes that slow tail. - `ubuntu-26.04` on x64 was within noise of 24.04 on the same CPU types (#129378), so the speedup comes from the ARM hardware, not the OS image. - Compiled pydantic on ARM accounts for about 2% of the gain. ## Changes - `.github/workflows/backend.yml` and `acceptance.yml`: `runs-on: ${{ github.event_name == 'pull_request' && 'ubuntu-26.04-arm' || 'ubuntu-26.04' }}` for `backend test` and `acceptance`. - `.github/workflows/backend.yml`: the odiff download picks the `arm64` or `x64` binary based on `uname -m`. It was hard-coded to x64. - `.github/workflows/acceptance.yml`: on Arm, install the Chromium snap (retried up to 5 times; the snap store returned a transient HTTP 408 once) and point `google-chrome`/`chromedriver` at it. - `.github/workflows/scripts/monitor-resources.py`: ARM kernels don't report a `model name`, so the monitor records the CPU implementer and part IDs instead. A test for this is in `test_monitor_resources.py`. - `src/sentry/testutils/pytest/selenium.py`: 0.1s wait polling and failure-only debug capture (see above). - `uv.lock`: adds the compiled linux aarch64 pydantic 1.10.26 wheels, now published on the internal index by getsentry/pypi#2693. Without them, ARM installs pure-Python pydantic and `test_pydantic_1x_compiled` fails. The change only adds wheel entries, so x64 is unaffected. All images the backend tests use already have ARM builds. Only `image-mirror-google-cloud-sdk` is x64-only, and these tests don't use it. Next in the stack: #129398 sizes selective backend shards for a ~3 minute slowest job on these runners. ## Checks - `test_monitor_resources.py` passes locally, and `backend.yml` parses as YAML. - Before the Selenium commit, CI on this PR was green: backend full suite on all 30 shards and acceptance on all 5 shards on Arm. CI is re-running on `0aa275df`. - `selenium.py`: ruff check/format and mypy pass locally. <!-- junior-request-attribution:start --> via @joshuarli. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AD0BQC9WSQ2E%3A1793386300.592079 --> -- [View Session](https://junior-prod.sentry.dev/conversations/slack%3AD0BQC9WSQ2E%3A1793386300.592079) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AD0BQC9WSQ2E%3A1793386300.592079/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: Joshua Li <joshuarli98@gmail.com> Co-authored-by: Joshua Li <14209781+joshuarli@users.noreply.github.com> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment stacked on #129387 (acceptance on Arm for PRs, 2 xdist workers). Not meant to merge. I'll post timings and then close it.
Changes to
src/sentry/testutils/pytest/selenium.py:Browser.wait_until*andwait_until_script_executionnow poll every 0.1s instead of Selenium's default 0.5s.tests/acceptancehas about 400 wait calls, and everyAcceptanceTestCase.tearDownadds 3 more.pytest_runtest_makereportused to take a screenshot, the page source and the browser logs for every phase of every test, including passing ones. It now does that only when a phase fails.Baseline is #129387 on Arm. An acceptance job there takes about 279s: about 115s of setup, 12s of pytest startup and 140s of tests (about 40 tests per shard, about 6.3s per test per worker). During the test step the runner is only about 67% busy and the pytest workers use about 0.3 cores, so most of the time goes to waiting on the browser.
This is one of a set of three, each built on the one before:
All three are based on #129387, so each one runs the same 5 acceptance shards.
via @joshuarli.
--
View Session [Sentry]