Repository navigation
ci(acceptance): experiment: 4 xdist workers - #129390
Closed
sentry-junior[bot] wants to merge 12 commits into
Closed
sentry-junior[bot] wants to merge 12 commits into
sentry-junior[bot] wants to merge 12 commits into
Conversation
Bump XDIST_WORKERS to 4 on pull requests and master to speed up backend test shards.
Sample cpu/mem every 5s per shard and upload resource-usage artifacts with a headroom verdict for one more xdist worker.
…y-spy Break each resource sample down into CPU-seconds per docker container, xdist worker, pytest controller, and other host process, plus user/system/ iowait/steal split. Fix PSI parsing, which read the always-zero full line. Sample the pytest controller and its xdist workers with py-spy and upload the collapsed stacks with the resource artifact. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…-process CPU - Rewrite the shard's test modules, conftests and testutils once in a background process instead of in each xdist worker. - Run the throwaway test postgres with fsync, synchronous_commit and full_page_writes off. - Recover CPU of processes that exit between samples from their cgroup's cpu.stat, and record reaped children's CPU by parent. - Turn py-spy profiling off by default; it costs ~0.4 cores. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…Ms, count postgres connections Prewarmed pycs were compiled with relative paths while pytest uses the absolute spec origin and doesn't fix co_filename, so tracebacks changed. sentry-shared-kafka's healthcheck starts a JVM every 5s, about 0.25 cores during tests. Once everything that waits on Kafka is up, answer that exact command with a TCP probe. Log postgres connections and autovacuum runs to name the ~0.3 cores of short-lived postgres process CPU. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…d clickhouse activity Autovacuum ran ~190 vacuums/analyzes per shard during tests, a likely source of the ~0.3 cores of short-lived postgres process CPU. Move the service diagnostics to a script and add ClickHouse query/part CPU breakdowns to explain its ~0.4 cores. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
… postgres experiments Same commit as the 4-worker runs except XDIST_WORKERS, to see whether the fourth worker still pays off after the CPU fixes. Will be set back to 4. autovacuum=off didn't change postgres CPU (exited backends are closed test connections, not autovacuum), so it and the connection logging go. Same-shard runtimes differ up to 1.9x between runs with identical CPU-seconds ratios, so record each runner's CPU model. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
The 3-worker control on the same commit was ~9% slower in total test time (7057s vs 6397-6535s across 30 shards) and its slowest shard 24-48s slower, so keep 4. The ClickHouse query breakdown has answered its question. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…profiling Same-CPU comparisons put ubuntu-26.04 within noise of 24.04, and the ARM runners we want next are 26.04. The py-spy step is commented out; the wrapper script stays for ad-hoc profiling. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…afka healthcheck Same changes as the backend test job: ubuntu-26.04, the resource monitor (uploaded as acceptance-resource-usage-<run>-<shard>), and the kafka-topics healthcheck shim. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
sentry-junior
Bot
force-pushed
the
ci/add-xdist-worker-backend-tests
branch
from
October 30, 2026 23:04
83ebd20 to
06e6ecc
Compare
Contributor
Author
|
Result: 4 acceptance workers is slower than 3 on x64. Closing. Both runs used 10 shards and got the same runner CPU mix (2× EPYC 9V45, 5× EPYC 7763, 3× EPYC 9V74), all green.
Chrome plus the per-worker Snuba containers already keep the 4-vCPU runner busy at 3 workers, so a 4th worker only adds contention and another Snuba bootstrap. |
This was referenced Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment, not for merge. Stacked on #129338.
Sets acceptance
XDIST_WORKERSfrom 2 to 4 onubuntu-26.04(amd64), to compare acceptance test time with #129338 (2 workers) and #129389 (3 workers). The results will be posted here, then this PR will be closed.via @joshuarli.
--
View Session [Sentry]