Skip to content

ci: run CI Gate and Cache Janitor on ubuntu-latest, not the contended ubuntu-26.04 pool #5256

Closed

Description

@zackees

Refs #5224, Refs #5237

Problem

In merge_group runs since #5231 (2026-11-19 13:49Z), the ubuntu-26.04 hosted label waits far longer for a runner than ubuntu-latest. Queue wait is job created_at to started_at, successful jobs only:

label n median p90 max over 60 s
ubuntu-26.04 120 58 s 257 s 568 s 60
ubuntu-latest 51 19 s 76 s 176 s 9

CI Gate is a few-second job (checkout, uv, ci/queue_report.py, ci/gate_jobs.py) that starts only after every lane finishes, and the merge happens at its end. It sat in the ubuntu-26.04 queue for 280 s (run 44536141725), 274 s (44536716046) and 196 s (44536711256) while three or four groups ran together. In the other four green runs it waited 3 to 11 s. That is 750 s of pure runner wait across 7 merged PRs, about 110 s per PR on average and up to 280 s at the tail. Cache Janitor waited 342 s in the same run.

Proposal

Switch ci-gate (ci.yml, runs-on: ubuntu-26.04) and cache-janitor to ubuntu-latest. Neither uses a 26.04-only feature: both run uv run Python scripts and gh. Keep jobs that consume the 26.04 container or runtime (tier2-setup-*, native-cache-consumer, flatpak-runtime-reuse) as they are.

Estimate

About 110 s saved per merged PR on average (280 s when the hosted pool is saturated), with no change to what the gate checks.

Verify

Compare gate created_at to started_at on the next 10 merge_group runs. Median should be under 20 s.

Activity

  1. added
    readyMeets the definition of Ready; claimable
    ciContinuous integration
    on Nov 19, 2026
  2. zackees commented on Nov 19, 2026

    @zackees
    MemberAuthor

    Claiming: CI Gate and Cache Janitor runner labels (pool slot obs-studio2).

    1. Audit every step and runner-label policy/self-test for 26.04 requirements.
    2. Record local RED evidence, switch only the two lightweight jobs to ubuntu-latest, and update the existing policy pin.
    3. Run focused CI self-tests and full lint, publish subject to backpressure, and hand off the ready PR with pinned auto-merge.

    First PR expected within this session.

  3. zackees commented on Nov 19, 2026

    @zackees
    MemberAuthor

    #5264 merged (6c3417a): ci-gate and cache-janitor now run on ubuntu-latest, and the required context is unchanged. Leaving this open until the next E640 check confirms the gate's queue wait on 10 merge_group runs (median target under 20 s).

  4. zackees commented on Nov 19, 2026

    @zackees
    MemberAuthor

    Verified by E640 check 23: on the merge_group runs after #5264, the CI Gate queue wait was 31, 80, 4, 3, 2 and 4 s, a median of 3.5 s. On ubuntu-26.04 it had been a 96 s median with a 310 s maximum. Cache Janitor waits dropped from up to 583 s to 2-113 s. Target met.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ciContinuous integrationreadyMeets the definition of Ready; claimable

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions