Skip to content

ci: queue on a busy Dell instead of falling back to slower hosted runners - #5291

Open
zackees wants to merge 4 commits into
masterfrom
infra/select-overcommit
Open

zackees wants to merge 4 commits into
masterfrom
infra/select-overcommit

Conversation

@zackees

@zackees zackees commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Queue on a busy Dell when it is expected to finish before hosted CI, while preventing PR runs from understating the jobs they send to the Dell.

MQ_RUNNER_MAX_RUNS defaults to 3. Queue runs retain the idle fast case, then use the busy Dell below the cap; at the cap or without enough online capacity for one run, use hosted. Reports 22/23 measured Dell/hosted medians 27.8/46.3 minutes and Dell queue wait p90 349 s, maximum 615 s. Dell's median plus that maximum wait is 38.05 minutes, 8.25 minutes ahead of hosted's median. The default is a bounded first step above the previous two-run limit and needs deployment measurement.

A PR override previously reduced its admission reservation while its full job graph still received Dell labels. The manager's MQ_PR_DEMAND=2, MQ_PR_RESERVE=3 configuration admitted run 44617744203 onto six runners, which then occupied all six with five further jobs queued. The actual expanded PR graph has 13 routed jobs. Make that count a hard floor for both admission and in-flight PR weights; a smaller positive override is logged and clamped to the inventory, while larger overrides remain conservative. Job placement and artifact handoffs are unchanged. The graph regression expands matrix legs from the real workflow and checks effective admission against the count, including the manager's exact configuration.

Six-runner configuration: keep DELL_PR_ROUTING=off, MQ_PR_DEMAND=13, MQ_PR_RESERVE=3. No demand override safely admits the current thirteen-job graph with that reserve on six runners. At least sixteen idle/online runners, or a separately bounded routing subset, are needed. This PR changes no live repository variables.

Rebased onto #5279 and preserve PR-only idle admission and the MQ reserve. Queue overcommit never relaxes PR admission. Mixed commitments retain full event-specific capacity weights. Missing run/job routing fields reserve capacity conservatively, with missing keys logged. Unknown current ordering reserves every undecided peer, and missing identities hold distinct slots across event listings without invalid jobs requests. Run 44586086466/job135779273252 logged a bare KeyError after entries=2, total_count=3; its original API payload/key was not retained, so tests reconstruct possible missing-current/field shapes rather than claiming an exact historical payload.

Validation:

  • Original RED a242d4601: five local fake-API failures for busy admission, caps, snapshot/current visibility and missing labels; RED/GREEN published together.
  • PR-budget RED 100bffa: four assertion failures reproducing the 6/2/3 admission bug and a graph reservation smaller than 13. GREEN enforces the graph floor and rejects undersized fleets even with small overrides.
  • 202 targeted tests pass: 107 selector, 65 CI scope, 30 member-PR routing; complexity and diff checks pass. One local reviewer, reused for integration and the added scope: clud-review: clean.
  • Previous proof passed analysis on all six supported targets and all 399 repository check files, but host execution had 9,263 passed/1 failed/6 skipped. Existing bounded_free_space::paths_do_not_block_each_other failed (0 versus 42); that optional host proof was stopped at its 40-minute budget with documentation tests unfinished. No complete host proof was claimed. Fresh push evidence is recorded below.

No new workflow files, native implementation changes, public header changes, C-test changes or WebSocket plugin changes.

Refs #4839 #4896.

Observed graph evidence: the jobs API for run 44617744203 contains 13 completed self-hosted jobs, matching the expanded workflow regression. The initial six busy plus five queued jobs were followed by the two Tier 2 test jobs after their build.

Fresh final-tree evidence: all six supported targets completed analysis, and all 399 repository check files passed. The follow-up optional host-test attempt was bounded to 60 seconds because this update changes only selector/tests/docs and the previous full host result is already recorded above. It was stopped during compilation without a host-test receipt; ./push retained partial Ci-Attestation evidence, used its ordinary required-CI fallback, and updated this PR with an exact force-with-lease. No complete host evidence or passing host rerun is claimed.

@zackees
zackees enabled auto-merge November 19, 2026 22:27
@zackees
zackees disabled auto-merge November 19, 2026 22:31
@zackees zackees added the pool-active Pool worker owns this PR; automated rebases must skip it label Nov 19, 2026
…ners

Use MQ_RUNNER_MAX_RUNS (default 3) to bound Dell queue admission and reserve capacity conservatively when Actions run/job snapshots omit routing fields. Refs #4839 #4896.

Ci-Attestation: {"at":1795124615,"gate":"data/all/compatibility","host":"linux-x86_64","key":"","lane":"client-checks","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":333,"stamp":"3d7f96fa0deb690633fb183a1bfde7be","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124615,"gate":"data/all/flatpak-manifest","host":"linux-x86_64","key":"","lane":"client-checks","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":333,"stamp":"af36f585875b056ace30024b75c87b58","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124615,"gate":"data/all/qt-xml","host":"linux-x86_64","key":"","lane":"client-checks","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":333,"stamp":"479a9281b73e1619b3facd0fada0a5ad","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124615,"gate":"data/all/services-schema","host":"linux-x86_64","key":"","lane":"client-checks","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":333,"stamp":"91f3cd6325884a3878002b5671c4ef61","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124274,"gate":"general/all/client-lint","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"909f385a8eb8ead160ca21677d59f005","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124615,"gate":"python/all/self-test","host":"linux-x86_64","key":"","lane":"client-checks","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":333,"stamp":"ce7aeab25e6549943d36b82252a8ce1c","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124277,"gate":"rust/aarch64-apple-darwin/client-clippy","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"70ee60c57683a2a5ce12960dce491d63","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124277,"gate":"rust/aarch64-apple-darwin/client-dylint","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"efeca24625038dcee597450d21a6c645","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124276,"gate":"rust/aarch64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"20a4e15dc2c1d58f550788df7fee169d","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124276,"gate":"rust/aarch64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"8efad7afb28bce42c9af540b2e5e589f","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124275,"gate":"rust/aarch64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"2f766aa6e1d871a90e7f86dda3753552","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124275,"gate":"rust/aarch64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"54a8bee0425907a6a09525cab4cad7ca","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124276,"gate":"rust/x86_64-apple-darwin/client-clippy","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"c04ceb9782ee9630d65cf962285c6e8c","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124277,"gate":"rust/x86_64-apple-darwin/client-dylint","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"a04f741b2b1cf8332606bc5607302c7d","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124275,"gate":"rust/x86_64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"daba282e1ff9d338d47b6d2da59b222c","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124276,"gate":"rust/x86_64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"d4a657f3e2f65995a55524b0d862773b","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124275,"gate":"rust/x86_64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"33a0b4aba3ff09679b3d72ef0f9ae80f","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795124275,"gate":"rust/x86_64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"95938cab5fe9002e5edcf05669f9e2b671d5b55fe3a4fa72913f00639d2e56c4","lane":"client-lint","parents":["33b325cd83f264ab110d27b159cc87c41301252b"],"secs":855,"stamp":"08c96492373aa9d482995d930a9e51ca","tree":"27a11807feb7c708ba27346496b7dacdee092350","v":1,"via":"run"}
Ci-Attestation: {"at":1795128501,"gate":"data/all/compatibility","host":"linux-x86_64","key":"","lane":"client-checks","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":328,"stamp":"74c2e968040a84527f2fa6841446d2e3","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128501,"gate":"data/all/flatpak-manifest","host":"linux-x86_64","key":"","lane":"client-checks","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":328,"stamp":"1ddd1a7f46af483ee079c7a9310759f0","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128501,"gate":"data/all/qt-xml","host":"linux-x86_64","key":"","lane":"client-checks","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":328,"stamp":"3e0cfffa0aafd715cdad594a2fec1a84","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128501,"gate":"data/all/services-schema","host":"linux-x86_64","key":"","lane":"client-checks","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":328,"stamp":"72800a9f4ad23076b05e874c9b5865e7","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128151,"gate":"general/all/client-lint","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"7b840cdee2d6f72bb174de565f25ef68","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128501,"gate":"python/all/self-test","host":"linux-x86_64","key":"","lane":"client-checks","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":328,"stamp":"7b20a0fe906c980d6fe4a339abc60185","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128158,"gate":"rust/aarch64-apple-darwin/client-clippy","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"5ceeb3b21e0b74764085cd5b5aa28e87","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128158,"gate":"rust/aarch64-apple-darwin/client-dylint","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"ab22e88090859f7a4da76717edd38f82","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128156,"gate":"rust/aarch64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"b40e9aa3269892bd4e926436d88a3957","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128156,"gate":"rust/aarch64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"943ec41d0943903077bc9294154e9f6e","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128153,"gate":"rust/aarch64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"a3d03c418ab8d85c169c6167a27a2d1f","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128153,"gate":"rust/aarch64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"81a60316dd45df48efe034fbf81c5d0c","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128157,"gate":"rust/x86_64-apple-darwin/client-clippy","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"cafb67a9f90bbcec594899b81030d729","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128157,"gate":"rust/x86_64-apple-darwin/client-dylint","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"13f5abe93818cb3d2c47a087d54ecc8c","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128154,"gate":"rust/x86_64-pc-windows-msvc/client-clippy","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"dc1f2e0a54c5b9b7a1eaf7a242e14d26","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128155,"gate":"rust/x86_64-pc-windows-msvc/client-dylint","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"e980345646924e776dbcf44ec5817679","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128152,"gate":"rust/x86_64-unknown-linux-gnu/client-clippy","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"ade6a476ea3420b7b51b259e28dcffcf","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
Ci-Attestation: {"at":1795128152,"gate":"rust/x86_64-unknown-linux-gnu/client-dylint","host":"linux-x86_64","key":"d837fa7b9cb3365a6d62f7f0cdb63795197c895654a82abd3b77a61e66ad58b8","lane":"client-lint","parents":["100bffadfa89448db069d9540cefc954950f7ab2"],"secs":149,"stamp":"2a1456c2ae3ae00e633bb63a9ef13ce4","tree":"f68580677252587b53c2c821db2b0bcb3f5e5f70","v":1,"via":"run"}
@zackees
zackees force-pushed the infra/select-overcommit branch from 343b23d to 3131a37 Compare November 19, 2026 22:50
@zackees
zackees enabled auto-merge November 19, 2026 22:52
@zackees zackees removed the pool-active Pool worker owns this PR; automated rebases must skip it label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant