Repository navigation
ci(backend): experiment: backend test shards on ubuntu-26.04-arm - #129372
Draft
sentry-junior[bot] wants to merge 11 commits into
Draft
sentry-junior[bot] wants to merge 11 commits into
sentry-junior[bot] wants to merge 11 commits into
Conversation
Bump XDIST_WORKERS to 4 on pull requests and master to speed up backend test shards.
Sample cpu/mem every 5s per shard and upload resource-usage artifacts with a headroom verdict for one more xdist worker.
…y-spy Break each resource sample down into CPU-seconds per docker container, xdist worker, pytest controller, and other host process, plus user/system/ iowait/steal split. Fix PSI parsing, which read the always-zero full line. Sample the pytest controller and its xdist workers with py-spy and upload the collapsed stacks with the resource artifact. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…-process CPU - Rewrite the shard's test modules, conftests and testutils once in a background process instead of in each xdist worker. - Run the throwaway test postgres with fsync, synchronous_commit and full_page_writes off. - Recover CPU of processes that exit between samples from their cgroup's cpu.stat, and record reaped children's CPU by parent. - Turn py-spy profiling off by default; it costs ~0.4 cores. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…Ms, count postgres connections Prewarmed pycs were compiled with relative paths while pytest uses the absolute spec origin and doesn't fix co_filename, so tracebacks changed. sentry-shared-kafka's healthcheck starts a JVM every 5s, about 0.25 cores during tests. Once everything that waits on Kafka is up, answer that exact command with a TCP probe. Log postgres connections and autovacuum runs to name the ~0.3 cores of short-lived postgres process CPU. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…d clickhouse activity Autovacuum ran ~190 vacuums/analyzes per shard during tests, a likely source of the ~0.3 cores of short-lived postgres process CPU. Move the service diagnostics to a script and add ClickHouse query/part CPU breakdowns to explain its ~0.4 cores. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
… postgres experiments Same commit as the 4-worker runs except XDIST_WORKERS, to see whether the fourth worker still pays off after the CPU fixes. Will be set back to 4. autovacuum=off didn't change postgres CPU (exited backends are closed test connections, not autovacuum), so it and the connection logging go. Same-shard runtimes differ up to 1.9x between runs with identical CPU-seconds ratios, so record each runner's CPU model. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
The 3-worker control on the same commit was ~9% slower in total test time (7057s vs 6397-6535s across 30 shards) and its slowest shard 24-48s slower, so keep 4. The ClickHouse query breakdown has answered its question. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Picks the odiff binary for the runner's architecture, and records an Arm runner's CPU implementer/part ids, since Arm kernels report no model name. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
github-actions
Bot
added
the
Scope: Backend
sentry-junior
Bot
added
the
Trigger: Override Selective Testing
Contributor
Author
|
First run, 41132145862: ARM is much faster. 29 of 30 shards passed. Shard 21 failed only on Compared against a re-run of #129338 at the same time on
|
getsentry/pypi#2693 published compiled pydantic 1.10.26 wheels for linux aarch64; refresh uv.lock so Arm runners install them instead of the purelib wheel. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Contributor
Author
|
Second run, with compiled pydantic (41138038705, Compared against a re-run of #129338 at the same time on
|
This was referenced Oct 30, 2026
sentry-junior
Bot
force-pushed
the
ci/add-xdist-worker-backend-tests
branch
from
October 30, 2026 23:04
83ebd20 to
06e6ecc
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment stacked on #129338 (4 xdist workers). Not for merge.
Runs the
backend testjob onubuntu-26.04-arm(Arm64, 4 vCPU / 16 GB for public repos) instead ofubuntu-24.04x64.Changes:
backend.yml:runs-on: ubuntu-26.04-armforbackend testonly. The odiff download now picksodiff-linux-arm64orodiff-linux-x64fromuname -m.monitor-resources.py: Arm kernels have nomodel namein/proc/cpuinfo, so the monitor recordsCPU implementer/CPU partinstead, plus a test for it.Pre-checked: the backend-ci service images (snuba, symbolicator, objectstore, cbtemulator, docker-redis-cluster) publish arm64 manifests, and actions/python-versions ships Python 3.13.17 for 26.04 arm64. The venv cache key includes the Python path, which contains the arch, so x64 and arm64 venvs don't collide. Still unverified: arm64 wheels for every locked package on the internal PyPI, and the Kafka/ClickHouse/Postgres images pulled through devservices' remote configs.
How it'll be measured: full suite (
Trigger: Override Selective Testing), run at the same time as a re-run of #129338 onubuntu-24.04x64.via @joshuarli.
--
View Session [Sentry]