Repository navigation
ci(backend): experiment: backend test shards on ubuntu-26.04 - #129378
Draft
sentry-junior[bot] wants to merge 10 commits into
Draft
sentry-junior[bot] wants to merge 10 commits into
sentry-junior[bot] wants to merge 10 commits into
Conversation
Bump XDIST_WORKERS to 4 on pull requests and master to speed up backend test shards.
Sample cpu/mem every 5s per shard and upload resource-usage artifacts with a headroom verdict for one more xdist worker.
…y-spy Break each resource sample down into CPU-seconds per docker container, xdist worker, pytest controller, and other host process, plus user/system/ iowait/steal split. Fix PSI parsing, which read the always-zero full line. Sample the pytest controller and its xdist workers with py-spy and upload the collapsed stacks with the resource artifact. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…-process CPU - Rewrite the shard's test modules, conftests and testutils once in a background process instead of in each xdist worker. - Run the throwaway test postgres with fsync, synchronous_commit and full_page_writes off. - Recover CPU of processes that exit between samples from their cgroup's cpu.stat, and record reaped children's CPU by parent. - Turn py-spy profiling off by default; it costs ~0.4 cores. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…Ms, count postgres connections Prewarmed pycs were compiled with relative paths while pytest uses the absolute spec origin and doesn't fix co_filename, so tracebacks changed. sentry-shared-kafka's healthcheck starts a JVM every 5s, about 0.25 cores during tests. Once everything that waits on Kafka is up, answer that exact command with a TCP probe. Log postgres connections and autovacuum runs to name the ~0.3 cores of short-lived postgres process CPU. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…d clickhouse activity Autovacuum ran ~190 vacuums/analyzes per shard during tests, a likely source of the ~0.3 cores of short-lived postgres process CPU. Move the service diagnostics to a script and add ClickHouse query/part CPU breakdowns to explain its ~0.4 cores. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
… postgres experiments Same commit as the 4-worker runs except XDIST_WORKERS, to see whether the fourth worker still pays off after the CPU fixes. Will be set back to 4. autovacuum=off didn't change postgres CPU (exited backends are closed test connections, not autovacuum), so it and the connection logging go. Same-shard runtimes differ up to 1.9x between runs with identical CPU-seconds ratios, so record each runner's CPU model. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
The 3-worker control on the same commit was ~9% slower in total test time (7057s vs 6397-6535s across 30 shards) and its slowest shard 24-48s slower, so keep 4. The ClickHouse query breakdown has answered its question. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
sentry-junior
Bot
added
the
Trigger: Override Selective Testing
github-actions
Bot
added
the
Scope: Backend
Contributor
Author
|
Result: Mean test-step time per shard, by runner CPU type:
|
This was referenced Oct 30, 2026
sentry-junior
Bot
force-pushed
the
ci/add-xdist-worker-backend-tests
branch
from
October 30, 2026 23:04
83ebd20 to
06e6ecc
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment stacked on #129338 (4 xdist workers). Not for merge.
Changes only the
backend testjob's runner fromubuntu-24.04toubuntu-26.04(GA since 2026-09-17, kernel 7.0 vs 6.17). Everything else matches #129338 at226e1191.Question: is the same 4-vCPU / 16 GB x64 runner faster on the newer image (kernel, systemd, Docker)? The ARM experiment is #129372.
How it'll be measured: full suite (
Trigger: Override Selective Testing), run at the same time as a re-run of #129338 onubuntu-24.04. The comparison is grouped by CPU model, because runner CPU types change shard times by up to 1.9×.Note: the venv cache key (
getsentry/action-setup-venv) doesn't include the OS version, so this run may restore a venv built on 24.04. The wheels are manylinux and 26.04's glibc is newer, so that should be fine.via @joshuarli.
--
View Session [Sentry]