Skip to content

CI perf: full tier (41 legs) runs on every push to main, not just nightly #1414

Closed

Description

The ci.yml header says the full tier is nightly, but e2e-full, yarn-berry-full and cargo-vex-matrix-full also run on every push to main: 41 legs per landing, most done in under a minute, which came to 1,845 of the 2,330 push-run jobs across 45 pushes on Oct 10-11. Gating them like e2e-docker (schedule, workflow_dispatch and the v5 integration PR) keeps the full tier on the nightly and release-qa dispatches; the trade-off is that a regression only that tier catches shows up at the next nightly instead of on the landing. PRs here are collaborators-only, so could a maintainer cherry-pick this signed commit (it also updates the two scheduling tests that pinned the old condition, and scripts/tests pass) or grant PR access? main...austenstone:socket-patch:ci-full-tier-nightly

Activity

  1. mikolalysenko commented on Oct 12, 2026

    @mikolalysenko
    Collaborator

    [agent] Triage: confirmed on origin/main 1779317. .github/workflows/ci.yml:2032 (e2e-full), :2286 (yarn-berry-full) and :2426 (cargo-vex-matrix-full) only exclude pull_request and merge_group, so they run on every push to main. Push run 38197830089: 41 of its 50 non-skipped jobs are these legs (~21 runner-minutes wall, ~41 billed job-minutes). The nightly schedule and workflow_dispatch already cover this tier.

    [agent] Claiming this issue (shared root cause: full-tier jobs gated on 'not PR/queue' instead of nightly/dispatch). Branch: agent/fix-ci-full-tier-nightly. Claim-ID: 2026-10-12T03:50:35Z-8f4246


    Generated by Claude Code

  2. mikolalysenko commented on Oct 12, 2026

    @mikolalysenko
    Collaborator

    [agent] Fix in #1415.


    Generated by Claude Code

  3. mikolalysenko commented on Oct 12, 2026

    @mikolalysenko
    Collaborator

    CI profiler — fresh numbers (window 2026-10-11 15:38 → 2026-10-12 04:00 UTC, 17 push-to-main CI runs, all jobs fetched with filter=all)

    per push run per day (~20 CI pushes/24h)
    Non-skipped jobs 50.0 ~1,000
    of which e2e-full* / yarn-berry-full / cargo-vex-matrix-full 29.0 ~580
    Full-tier wall job-min (Depot Linux) 14.2 ~285
    Full-tier job-min rounded up per job 29.1 ~580
    All other push jobs (coverage 1.9, hosted-e2e 1.4, clippy 1.3, yarn-berry 1.3 ×2, e2e-build 1.1, …) 14.8 ~300
    • Full-tier legs are short: p50 0.47 min, p90 0.75 min, so most of their cost is per-job setup.
    • push created → ci-ok: p50 3.43 / p90 3.82 min. An e2e-full leg is the last job before ci-ok in 10 of 17 runs, so gating it should also take a little off the push verdict (unmeasured; coverage would become last).
    • Merge queue and PR runs are unaffected: no full-tier job ran on any of the 22 merge_group or 12 full PR runs sampled.
    • Expected effect of CI: run the full e2e tier nightly, not per push (#1414) #1415: about half of push-to-main job time, ~285 Depot job-min/day (~580 rounded per job). Nightly and release-qa dispatches keep the tier, so coverage stays at nightly.

    Example runs: 38197830089, 38187527055, 38186163932. The profiler will report the before/after on the dashboard (#1182) once #1415 merges.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions