From b045fde329fff6c65137214a463438a42824d016 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 26 Oct 2026 06:45:10 +0000 Subject: [PATCH 1/2] ci: run main's full battery every 4 hours and CodeQL daily (maintainer ruling Q6 = C) ci.yml and lint.yml schedule '0 * * * *' -> '0 */4 * * *', job logic unchanged. codeql.yml drops push: main and runs daily at 02:00 UTC (the Monday pass folds in), with workflow_dispatch. The scheduled-full-run card identity moves to `4-hourly` and keeps the hourly title and marker readable through CADENCE_PREDECESSORS, so the open card #25438 is refreshed, never orphaned or duplicated; pinned in the self-test with its real title and marker line. The scheduled runs' Test Core run-summary and timing artifacts are kept 4 days so the weekly shard-timings refresh still sees about 24 runs. Readers, runbook and messages that said hourly follow. Co-authored-by: Claude Claude-Session: https://claude.ai/code/session_01AjoNWTpihGQZrTamr1vd4W --- .github/workflows/ci.yml | 56 +++-- .github/workflows/codeql.yml | 28 +-- .github/workflows/lint.yml | 26 +-- .github/workflows/release.yml | 2 +- .github/workflows/scheduled-full-run-card.yml | 24 ++- .github/workflows/shard-timings-refresh.yml | 31 +-- AGENTS.md | 2 +- docs/releases-maintenance.md | 23 +- scripts/ci/scheduled-full-run.mjs | 201 ++++++++++++++---- scripts/ci/select-gate-families.selftest.sh | 4 +- scripts/ci/select-gate-families.sh | 6 +- scripts/ci/select-shard-packages.selftest.sh | 2 +- scripts/ci/select-shard-packages.sh | 4 +- scripts/ci/select-shard-timings-run.mjs | 35 +-- scripts/partition-test-shards.mjs | 7 +- 15 files changed, 297 insertions(+), 154 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4b6eb0680e9..9009b73df0c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -38,7 +38,7 @@ on: # branch-protection-required check MUST carry this trigger, or queue builds # wait forever on a check that never reports. merge_group: - # ── THE HOURLY FULL RUN (#16467) ───────────────────────────────────────── + # ── THE SCHEDULED FULL RUN (#16467), every 4 hours since #25771 ────────── # `push` above now computes the Test Core package set with `--affected` # against `github.event.before`, so a merge no longer re-tests the whole # workspace it just tested in the queue. Something still has to run the FULL @@ -48,16 +48,22 @@ on: # This is that run. Minute 0 rather than an offset minute ON PURPOSE, and # against the sibling convention in test-nightly-tiers.yml: the reading this # feeds — `scripts/ci/select-shard-timings-run.mjs` — wants one complete run - # per hour far more than it wants a short queue wait, and a run that starts - # a few minutes late is still the hour's run. What it must NOT do is collide - # with the next hour's, which the concurrency group below decides. + # per window far more than it wants a short queue wait, and a run that starts + # a few minutes late is still that window's run. What it must NOT do is + # collide with the next one, which the concurrency group below decides. + # + # EVERY 4 HOURS, not hourly (#25771, maintainer ruling Q6 = C): the job logic + # is unchanged — the run is still the full battery and still finds flakes on + # an unchanged SHA — at six runs a day instead of twenty-four, most of which + # re-tested a SHA an earlier run had already tested. Every runner-minute it + # spends competes with PR and merge-queue capacity. # # ⛔ This is a NEW TRIGGER KEY, not a widening of `push:`. The ⛔ above still # binds: `push:` stays filtered to `main`. schedule: - - cron: '0 * * * *' + - cron: '0 */4 * * *' # The same full battery, on demand: an operator who has just landed a fix for - # a red hourly run should not have to wait up to an hour to see it answered. + # a red scheduled run should not have to wait up to 4 hours to see it answered. # Treated as `full` by the selection script for the same reason `schedule` is. workflow_dispatch: @@ -68,7 +74,7 @@ on: # run that tests ITS OWN merge: Test Core computes `--affected` against # `github.event.before` (see `on.schedule` above), so a push run that does not # finish leaves that merge's diff with no Test Core verdict on `main` until the -# next hourly run. Main merges land every ~9–16 min and a push run takes ~27, +# next scheduled full run. Main merges land every ~9–16 min and a push run takes ~27, # so under the old shared key `ci-CI-push-refs/heads/main` the next merge # cancelled it: 31 of 40 push runs on 2026-10-23 01:03Z–11:03Z, most of them # inside "Run this shard's tests". (The Turbo seed is NOT what was lost: the @@ -86,7 +92,7 @@ on: # ⭐ `github.event_name` IS LOAD-BEARING TOO (#16467). A `schedule` and a # `workflow_dispatch` run on `main` both fall through to `refs/heads/main`; # without the event name they share one group and an on-demand run cancels the -# hourly one (and, under any key that drops `github.event.after`, the next +# scheduled one (and, under any key that drops `github.event.after`, the next # merge cancels both, which is how 36 of 60 hourly-era push runs died). # # ⛔ Do not "simplify" either segment away. `scripts/ci/scheduled-full-run.mjs @@ -98,8 +104,8 @@ on: # those events too). # # `cancel-in-progress` stays `true`: with a per-commit push group it never -# fires on a push, and for everything else it is the policy. Two hourly runs -# overlapping means one of them is over an hour old, and an hour-old +# fires on a push, and for everything else it is the policy. Two scheduled +# runs overlapping means one of them is over 4 hours old, and a 4-hour-old # measurement of `main` is strictly the worse of the two. Test Core's wall clock # is well inside the cadence (~39 shard-minutes across the shards), so this is # the exceptional path, not the normal one. @@ -109,7 +115,7 @@ concurrency: # ── THE TWO SPEEDS (#22998) ────────────────────────────────────────────────── # A `pull_request` or `merge_group` run carries the FAST SET: `Build Core`, -# `Test Core` (affected-only) and `Build Docs`. `main`'s hourly `schedule` run, +# `Test Core` (affected-only) and `Build Docs`. `main`'s 4-hourly `schedule` run, # its `push` runs and an on-demand `workflow_dispatch` carry the FULL BATTERY: # the fast set plus the three jobs below that spell # @@ -243,7 +249,7 @@ jobs: # That failure direction is the dangerous one: every filter output would # come back `false`, `|| 'true'` supplies nothing (the output exists and # says `false`), and every downstream `!= 'false'` guard SKIPS — so the - # hourly full run would have been an entirely green, entirely empty run + # scheduled full run would have been an entirely green, entirely empty run # of nothing at all. An allow-list makes each new event fail toward # THE FILTER CONTRACT's half 1 instead: the step is skipped, its outputs # are the empty string, `|| 'true'` fills them in, and everything runs. @@ -628,7 +634,7 @@ jobs: # (p + s + 10m) and the cap-deferred path (p + s + 20m) -- land before the # new value on every observation, AND the slowest `Test Core (N/8)` job # wall is at or under 80% of it. Read the pull_request and merge_group - # runs, not the hourly schedule alone: scheduled runs mostly replay the + # runs, not the 4-hourly schedule alone: scheduled runs mostly replay the # cache their commit's push run saved. It carries no other expiry — a # raise with no revert condition beside it becomes permanent by # forgetting — and the stall guard stays the primary hang detector. @@ -772,7 +778,7 @@ jobs: # `main` was on before this merge landed (#16467). A zero # sha -- a first push, or a force-push that rewrote # history -- falls back to the FULL list, loudly. - # schedule the FULL list: the hourly run IS the full battery, and + # schedule the FULL list: the 4-hourly run IS the full battery, and # workflow_ the on-demand rerun of it. Neither carries a diff base, # dispatch and neither should: they are the run that says whether # the whole workspace is green on `main`. @@ -1266,6 +1272,11 @@ jobs: # attestation pair below to "match" this step — that pair is the # credential, not diagnostics, and check:shard-attestation pins its # unguarded shape statically. + # Retention: 4 days on a `schedule` run, 1 day otherwise (#25771). The + # weekly shard-timings refresh accumulates the scheduled runs whose + # summaries are still downloadable; hourly, one day held 24 of them, and + # at every 4 hours it takes four days to hold the same 24. Every other + # event's summary is read only inside its own run, so it stays at 1 day. - name: Publish this shard's turbo run summary if: always() continue-on-error: true @@ -1274,7 +1285,7 @@ jobs: name: test-core-run-summary-${{ matrix.shard }}-of-8 path: .turbo/runs/ if-no-files-found: ignore - retention-days: 1 + retention-days: ${{ github.event_name == 'schedule' && 4 || 1 }} # ── Predicted vs measured: this shard's timing drift (#16173, #16465) ── # @@ -1490,10 +1501,11 @@ jobs: --shard "${{ matrix.shard }}/8" \ --out "$RUNNER_TEMP/test-timings/shard-${{ matrix.shard }}.json" - # 1 day, matching the run-summary upload above: this is an intermediate - # the aggregator consumes minutes later in the SAME run. The 14-day - # retention this card asks for is on the MERGED table, published by - # `Test Core`. + # Retention matches the run-summary upload above (4 days on `schedule`, + # 1 day otherwise, #25771): the aggregator consumes this minutes later in + # the SAME run, and the shard-timings refresh reads the scheduled runs' + # captures beside their summaries. The 14-day retention this card asks + # for is on the MERGED table, published by `Test Core`. - name: Publish this shard's test timings if: always() continue-on-error: true @@ -1502,7 +1514,7 @@ jobs: name: test-core-timings-${{ matrix.shard }}-of-8 path: ${{ runner.temp }}/test-timings/ if-no-files-found: ignore - retention-days: 1 + retention-days: ${{ github.event_name == 'schedule' && 4 || 1 }} # A stall's full diagnostic reports (JS stacks, libuv handles, heap # summary per process) outlive the in-log digest — keep them so a #4250 @@ -1836,7 +1848,7 @@ jobs: # driver-memory, driver-mongodb, service-analytics) — the other backends the # temporal conformance matrix holds to one standard. Its NAME still says # "live PG + MySQL" on purpose: it was a required check until #22998 moved - # the job to the full battery, and its readers (the hourly run's red card, + # the job to the full battery, and its readers (the scheduled run's red card, # merge-queue triage history, every instruction that names it) find it by # that literal — renaming it is a contract change, not a tidy-up. temporal-conformance: @@ -2733,7 +2745,7 @@ jobs: # too (`no caches are enabled`) and re-executes everything. # # Credentials go only to runs whose code has landed or is landing: - # `merge_group`, and any run on refs/heads/main (push, the hourly + # `merge_group`, and any run on refs/heads/main (push, the 4-hourly # schedule, workflow_dispatch on main). A `pull_request` run, and a # dispatch on any other branch, receives neither TURBO_TOKEN nor the # signing key, so code a PR adds can neither upload to the remote nor diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index b96600be018..d1afaf14ce8 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -1,22 +1,24 @@ name: CodeQL Security Analysis # Deliberately NOT on pull_request: the analysis took ~4½ runner-minutes per -# PR sync, and everything a PR introduces is analyzed on main within minutes -# of the merge anyway (multi-agent merge cadence). Alerts surface on main's -# runs + the weekly deep pass instead of blocking/occupying PR runners. -# If branch protection ever required the "Analyze (javascript)" check, that -# requirement must be dropped along with this trigger. +# PR sync. If branch protection ever required the "Analyze (javascript)" check, +# that requirement must be dropped along with this trigger. +# +# DAILY, not on every push to main (#25771, maintainer ruling Q6 = C). The push +# trigger ran 100 analyses in 25.9 h, 86 of them cancelled by the next merge, +# and each one competed with PR and merge-queue runners. One daily run of +# `main` analyzes everything that landed the day before; the Monday deep pass +# it replaces ran the same query suite, so it folds into the daily one. +# `workflow_dispatch` re-analyzes `main` on demand, e.g. after landing a fix. on: - push: - branches: - - main schedule: - # Run at 02:00 UTC every Monday - - cron: '0 2 * * 1' + # 02:00 UTC every day + - cron: '0 2 * * *' + workflow_dispatch: -# A superseded analysis of an outdated commit has no value — cancel it. A newer -# main push cancels only an in-flight main analysis (the newer commit still -# gets analyzed). +# A superseded analysis of an outdated commit has no value — cancel it. An +# on-demand run cancels an in-flight analysis of the same ref (the newer +# `main` head still gets analyzed). concurrency: group: codeql-${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 9cfb886055b..272e424827b 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -25,8 +25,10 @@ on: # queue builds or the queue stalls. This workflow has no PR-only steps, so # the trigger alone is enough. merge_group: - # ── THE HOURLY FULL RUN (#16467), the half this file owns ──────────────── - # The card's ruling is that the hourly run covers BOTH required-context + # ── THE SCHEDULED FULL RUN (#16467), the half this file owns ───────────── + # Every 4 hours since #25771 (maintainer ruling Q6 = C), the same cadence as + # ci.yml's half and for the reason stated there; no step changed. + # The card's ruling is that the scheduled run covers BOTH required-context # files, not just ci.yml, and the reason is measured rather than symmetric: # the push-on-`main` run of THIS workflow is the only post-merge full-battery # run of the families `scripts/ci/select-gate-families.sh` scopes away on @@ -44,7 +46,7 @@ on: # # ⛔ A NEW TRIGGER KEY, not a widening of `push:`; see ci.yml's `on:` block. schedule: - - cron: '0 * * * *' + - cron: '0 */4 * * *' workflow_dispatch: # ── MEASURED 2026-08-25 (#12211) — a NEGATIVE result, recorded so it is not @@ -95,7 +97,7 @@ on: # # ⭐ `github.event_name` is in the key for the reason ci.yml's concurrency block # states in full (#16467): `schedule` and `workflow_dispatch` both fall through -# to `refs/heads/main`, so without it an on-demand run cancels the hourly one. +# to `refs/heads/main`, so without it an on-demand run cancels the scheduled one. # `scripts/ci/scheduled-full-run.mjs --check-concurrency` evaluates THIS # expression too and reds if two pushes, or a push and a schedule run, share a # group, or if two schedule, dispatch or same-PR runs stop sharing one. @@ -3514,8 +3516,8 @@ jobs: run: pnpm check:select-shard-packages # Rationale: `scripts/ci/scheduled-full-run.mjs` header (#16467). - - name: Hourly full run self-test and concurrency gate - id: hourly-full-run-self-test-and-concurrency + - name: Scheduled full run self-test and concurrency gate + id: scheduled-full-run-self-test-and-concurrency continue-on-error: true run: | node scripts/ci/scheduled-full-run.mjs --self-test @@ -4426,7 +4428,7 @@ jobs: # only a name that DISAPPEARS can turn a live declaration dead, which is # a structural change the selector already runs everything for. A group # confined to product source, tests, docs or changesets skips it here and - # pays for it on `push` to main and on the hourly run. + # pays for it on `push` to main and on the 4-hourly run. - name: A declared gate population reaches the tree id: a-declared-gate-population-reaches-the-tree continue-on-error: true @@ -4542,7 +4544,7 @@ jobs: # ROOT_DIR_WATCH_HINTS declaration, held against the root it really walks # by its own self-test — so a group that changes no file under scripts/ # (and no root configuration) skips it. The selection step at the top of - # this job decides that, `push` on main and the hourly run keep it + # this job decides that, `push` on main and the 4-hourly run keep it # unconditional, and the ruling that authorizes moving a tooling # self-test off the PR path is quoted in the selector's header. - name: scripts/ entry guards go through one predicate @@ -4693,7 +4695,7 @@ jobs: # anyway, and the trade is explicit rather than reasoned away: it imports # the same derivation as the dispatch-gates and watch-hint literal steps, so its inputs are the # workflow tree and the gate sources, and on a group touching neither the - # gap does move — to `push` on main and the hourly full run, which keep + # gap does move — to `push` on main and the 4-hourly full run, which keep # the whole battery. Gate weakening is a maintainer floor; the sentence # that authorizes this one is quoted in the selector's header. - name: PM bare-root worklist self-test @@ -4747,7 +4749,7 @@ jobs: # `scripts/**/*.mjs`, `scripts/**/*.mts` and `scripts/**/*.sh`, and the # rest of its read-set is the workflow tree and `.github/actions` it # discovers the runnable self-tests from — so a group touching none of - # those skips it here and runs it on `push` to main and on the hourly + # those skips it here and runs it on `push` to main and on the 4-hourly # full run. - name: Self-test workflow-command gate id: self-test-workflow-command-gate @@ -4913,13 +4915,13 @@ jobs: # three-file `packages/spec` diff, and ruling #208 on #19491 (R4) took # those whole-tree reads off the PR path on the maintainer's sentence, # quoted in the selector's header. ⛔ A defect in that wider set can now - # first appear on `main`; the push-on-main and hourly full runs are what + # first appear on `main`; the push-on-main and 4-hourly full runs are what # bound it, and widening the skip further is again a maintainer call. # # ⚠️ Since #22991 the battery is TIERED by event, inside the gate rather # than here: on `pull_request` and `merge_group` `pnpm check:pm-dispatch-gates` # runs the fast tier; on `schedule`, `push` and any other event it runs - # both. Every case still runs at least hourly on the full run, and a + # both. Every case still runs at least every 4 hours on the full run, and a # slow-tier red is a red on `main`. The step below is unchanged; the # decision and its argument live in `scripts/pm/check-dispatch-gates.mjs`. # diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 7e2f1530374..1aab183bce1 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1507,7 +1507,7 @@ jobs: full-battery: name: Full battery green at the version commit # A PR or queue build carries ci.yml's FAST SET since #22998; the dogfood - # and temporal legs run only on main's push, hourly and + # and temporal legs run only on main's push, 4-hourly schedule and # dispatch runs. So "tested before a release" is read HERE, as a gate: the # newest completed `schedule` or `push` runs of `CI` and `Lint & Type # Check` whose head is the version commit or a first-parent ancestor of it diff --git a/.github/workflows/scheduled-full-run-card.yml b/.github/workflows/scheduled-full-run-card.yml index cb179467c29..91c064504d5 100644 --- a/.github/workflows/scheduled-full-run-card.yml +++ b/.github/workflows/scheduled-full-run-card.yml @@ -1,6 +1,6 @@ name: Scheduled Full Run Card -# The reader for the hourly full run (#16467). +# The reader for the scheduled full run (#16467; every 4 hours since #25771). # # ══════════════════════════════════════════════════════════════════════════════ # WHY THIS EXISTS: A RED SCHEDULED RUN HAD NOBODY LOOKING AT IT. @@ -8,7 +8,7 @@ name: Scheduled Full Run Card # # Since #16467 a `push` to `main` computes the Test Core package set with # `--affected`, and `scripts/ci/select-gate-families.sh` already scopes several -# `Lint & Repo Gates` families away on merge groups. The hourly `schedule` run +# `Lint & Repo Gates` families away on merge groups. The scheduled `schedule` run # of `CI` and `Lint & Type Check` is therefore the ONLY run that exercises the # whole battery on `main`. # @@ -30,11 +30,13 @@ name: Scheduled Full Run Card # `scripts/ci/scheduled-full-run.mjs`, driven offline by its `--self-test`. That # is deliberate: a de-dup rule exercised only by the live workflow gets its # first real test on the night it files its second duplicate, and this one fires -# hourly. +# six times a day. # # IDENTITY a fixed title prefix per workflow — -# `hourly full run: red on main (CI)` — plus a PLAIN-TEXT body -# marker. ⛔ Never an HTML comment: this platform's body sanitizer +# `4-hourly full run: red on main (CI)` — plus a PLAIN-TEXT body +# marker. The hourly spellings an older card carries +# (`hourly full run: red on main (CI)`, #25438) are still matched +# (`CADENCE_PREDECESSORS`), so a cadence change never orphans it. ⛔ Never an HTML comment: this platform's body sanitizer # is measured to eat short angle-bracket fragments, and a de-dup # key that can be swallowed files a duplicate an hour. # DE-DUP scan OPEN issues for that prefix or that marker, bounded pages; @@ -101,14 +103,14 @@ concurrency: jobs: file: - name: File or refresh the hourly full run card + name: File or refresh the scheduled full run card # ⭐ THREE guards, and each one is load-bearing. # # event == 'schedule' — a push, PR or merge_group run of these workflows # is read by branch protection and by merge-queue-triage.yml. Filing on # those would mint a card for every red PR in the repo. # conclusion in (failure, timed_out) — NOT `!= 'success'`. `cancelled` - # and `skipped` are not reds: a cancelled hourly run measured nothing, + # and `skipped` are not reds: a cancelled scheduled run measured nothing, # and a card saying "main is red" on the strength of a run that did not # finish is a false statement that costs somebody an investigation. # head_branch == 'main' — belt and braces. A scheduled run can only be on @@ -181,7 +183,7 @@ jobs: }); const hit = previous.data.workflow_runs?.[0]; if (hit) previousGreen = { run_id: hit.id, head_sha: hit.head_sha }; - else rangeNote = 'No previous GREEN `schedule` run of this workflow is in the API window — either this is the first one, or every hourly run in the window was red.'; + else rangeNote = 'No previous GREEN `schedule` run of this workflow is in the API window — either this is the first one, or every scheduled run in the window was red.'; } catch (error) { rangeNote = `The previous-green lookup failed (${error.message}).`; } @@ -272,13 +274,13 @@ jobs: core.info(`${outcome.action} ${identity.marker} card #${outcome.number} (${body.length} chars)`); if (outcome.block) core.info(`block: ${outcome.blockSummary}`); const moved = outcome.block === 'expired' ? ' (every blocker closed: pm:blocked moved to pm:queue)' : ''; - core.notice(`${run.name} is red on the hourly full run — card #${outcome.number} ${outcome.action}${moved}.`); + core.notice(`${run.name} is red on the scheduled full run — card #${outcome.number} ${outcome.action}${moved}.`); core.setOutput('action', outcome.action); core.setOutput('number', String(outcome.number)); - name: Publish the verdict to the run summary # always(): when the step above threw — a refused scan, a rate limit — - # the run must still say what it was reacting to, or the red hourly run + # the run must still say what it was reacting to, or the red scheduled run # is invisible in both places at once. if: always() env: @@ -289,7 +291,7 @@ jobs: WATCHED_CONCLUSION: ${{ github.event.workflow_run.conclusion }} run: | { - echo "### Hourly full run: \`$WATCHED\` concluded \`$WATCHED_CONCLUSION\` on \`main\`" + echo "### Scheduled full run: \`$WATCHED\` concluded \`$WATCHED_CONCLUSION\` on \`main\`" echo echo "- Run: $WATCHED_URL" if [ -n "${CARD_ACTION:-}" ]; then diff --git a/.github/workflows/shard-timings-refresh.yml b/.github/workflows/shard-timings-refresh.yml index 218846ee817..2d7a80a3d00 100644 --- a/.github/workflows/shard-timings-refresh.yml +++ b/.github/workflows/shard-timings-refresh.yml @@ -70,7 +70,10 @@ # no refresh because it stamps a fresh `measuredAt` on numbers nobody measured. # That script carries the argument and the self-test; this file only drives it. # -# ⚠ WHICH RUNS: THE HOURLY `schedule` RUN OF ci.yml, NEVER A PUSH RUN (#16467). +# ⚠ WHICH RUNS: THE SCHEDULED RUN OF ci.yml, NEVER A PUSH RUN (#16467). Hourly +# until #25771 moved it to every 4 hours; the reasoning below is unchanged. +# Its summaries are retained 4 days (1 day on every other event), so the +# window still holds about 24 of them. # Until #16467 a push to `main` re-ran the FULL Test Core battery, so "a green # push run on main" and "a measurement of the workspace" were the same thing. # They are not any more: a push run computes its package set with `--affected` @@ -342,9 +345,9 @@ jobs: GITHUB_TOKEN: ${{ github.token }} run: | # 24, not 15 (#16467). The window that matters is the artifact - # retention window — 1 day — and the runs inside it are now the - # hourly scheduled ones, so 24 is "everything still downloadable" - # rather than an arbitrary depth. Coverage and sample depth are both + # retention window of the scheduled runs — 4 days at one run every + # 4 hours since #25771, 1 day at one an hour before it — so 24 is + # "everything still downloadable" rather than an arbitrary depth. Coverage and sample depth are both # ACCUMULATED across runs (see the next step), so examining fewer # than the window holds is samples left on the table; the cost is two # API reads per run examined, and the loop below stops at the first @@ -366,8 +369,8 @@ jobs: # landed the hourly run, this lane ran with `event=schedule` before # that trigger existed on main, printed "NO ELIGIBLE RUN among the 0" # and named three causes none of which had occurred. It also does not - # clear at merge — there is a bootstrap window of at least an hour - # before the first hourly run finishes. + # clear at merge — there is a bootstrap window of up to 4 hours + # before the first scheduled run finishes. # # `|| SELECT_EXIT=$?` and not a bare call: this step runs under # `bash -e`, where the non-zero would kill it before the code could @@ -388,7 +391,7 @@ jobs: # summary section and a job-level notice, and every one of them says # that a PERSISTENT 3 is a defect rather than a steady state. echo "not_measured=true" >> "$GITHUB_OUTPUT" - echo "::warning::Shard timings NOT MEASURED: no completed \`schedule\` run of ci.yml exists on main yet. The dataset was left untouched. If this repeats once the hourly full run has been live for a few hours, the trigger is gone or every hourly run is being cancelled — file it." + echo "::warning::Shard timings NOT MEASURED: no completed \`schedule\` run of ci.yml exists on main yet. The dataset was left untouched. If this repeats once the scheduled full run has been live for a day, the trigger is gone or every scheduled run is being cancelled — file it." { echo "### Shard timings: NOT MEASURED (exit 3) — nothing was regenerated, and nothing failed" echo @@ -398,12 +401,12 @@ jobs: echo "\`scripts/test-shard-timings.json\` is untouched, which is the correct outcome for this" echo "reading, and this job is green because nothing went wrong — not because anything passed." echo - echo "Expected exactly once: while the hourly full run bootstraps. The trigger has to be on" - echo "\`main\` and one run has to finish, and run-summary artifacts live 1 day." + echo "Expected exactly once: while the scheduled full run bootstraps. The trigger has to be on" + echo "\`main\` and one run has to finish, and run-summary artifacts live 4 days on a scheduled run." echo echo "⛔ **A PERSISTENT NOT MEASURED IS A DEFECT, NOT A STEADY STATE.** If this section is" - echo "still here after the hourly run has been live for a few hours, the \`schedule\` trigger" - echo "has been removed from \`ci.yml\` or every hourly run is being cancelled — and the" + echo "still here after the scheduled run has been live for a day, the \`schedule\` trigger" + echo "has been removed from \`ci.yml\` or every scheduled run is being cancelled — and the" echo "balancing dataset is quietly ageing out while this job reports green. File a card." echo echo "The selector's own refusal text, verbatim:" @@ -437,7 +440,7 @@ jobs: # is a HIT — and the generator refuses hits rather than recording a # replayed ~0.1s window as a suite's cost. "Download the shard artifacts # from any green run" therefore measures a SLICE of the workspace, never all of it. - # Since #16467 the runs read here are the HOURLY `schedule` runs, which are + # Since #16467 the runs read here are the `schedule` runs (every 4 hours since #25771), which are # the full battery; the accumulation stays because the cache argument above # is unchanged by which event ran the suite. # @@ -645,7 +648,7 @@ jobs: # The per-file half of the regression readings (#22949 ruling E, item 3). # The dataset holds no per-file durations; the window's runs do, in the # per-shard captures `report-test-timings.mjs --capture` uploads as - # `test-core-timings-N-of-M`, retained 1 day like the run summaries the + # `test-core-timings-N-of-M`, retained like the run summaries the # generator read. The selector listed their ids beside the summaries' # (`timing_artifact_ids`), so this is a read of the SAME runs, never a new # measurement, and it feeds the PR body alone. @@ -845,7 +848,7 @@ jobs: echo echo "## Source" echo - echo "Measured across $RUN_COUNT accumulated run(s) of the HOURLY \`schedule\` run of CI on" + echo "Measured across $RUN_COUNT accumulated run(s) of the 4-hourly \`schedule\` run of CI on" echo "\`main\` — the full-battery run (#16467). A \`push\` run on \`main\` is affected-only and" echo "is not a measurement of the workspace, so no push run feeds this file." echo diff --git a/AGENTS.md b/AGENTS.md index 4e05826ad5f..966544bb94f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -529,7 +529,7 @@ Even inside your own worktree, operate defensively: that lands rides `main`'s merge ref into every later PR until stanched. A required context is matched by check-run name, so a rename detaches its gate silently — treat those eight names as contract. **Two speeds** (ruling record 6120820653): PR and queue runs carry the fast set, 10 minutes of - wall per queue entry; `main`'s hourly and push runs carry the full battery — dogfood, e2e, + wall per queue entry; `main`'s scheduled (4-hourly) and push runs carry the full battery — dogfood, e2e, `Temporal Conformance (live PG + MySQL)` — read by the publish lane before any approval. **Main-red:** a full-battery red is its lane's first work — triage files it under the main-red convention, the lane's next PR fixes it, and ⛔ no lane PR arms the queue meanwhile. diff --git a/docs/releases-maintenance.md b/docs/releases-maintenance.md index 098289a2206..d7358713ac9 100644 --- a/docs/releases-maintenance.md +++ b/docs/releases-maintenance.md @@ -478,11 +478,11 @@ how many first-parent commits before the version commit that run's head sits. Th gate reads only *completed* runs and never waits (by design: `selectFullBatteryRun` in `scripts/ci/scheduled-full-run.mjs`), so a run still in progress is invisible to it. Now open **Actions → CI** on `main`: if a run on the version commit, or on any -commit between it and the one the `CI` line names, is still in progress — normally -the hourly `schedule` run (`cron: '0 * * * *'` in `ci.yml`), occasionally a `push` -run — **wait for it to complete before approving**. That is about 25–30 minutes from -the top of the hour (the 20 hourly runs up to 2026-10-23 00:20Z took 14–39 minutes, -median 28.7, 14 of them 25–31). Green: approve. Red: do not approve. It is the reading the gate would +commit between it and the one the `CI` line names, is still in progress — a `push` +run (every merge to `main` gets its own since #25373, so the version commit's own push +run is the usual one) or the `schedule` run — **wait for it to complete before +approving**. A run takes about 25–30 minutes (the 20 hourly scheduled runs up to +2026-10-23 00:20Z took 14–39 minutes, median 28.7, 14 of them 25–31). Green: approve. Red: do not approve. It is the reading the gate would have refused had it been complete, so treat it as that refusal (recovery: the `full-battery` job's header in `release.yml`). `CI` is the long battery, so it is the window that matters in practice; an in-flight `Lint & Type Check` or `Spec Shape @@ -496,6 +496,19 @@ still running. That run concluded **failure** at 12:30Z, minutes after the gate it was refused. The gate was right about everything it could see; the gap is only the window in which a nearer run is in flight, and only a human can wait it out. +**The scheduled run is every 4 hours** (`cron: '0 */4 * * *'` in `ci.yml` and +`lint.yml`, hourly until #25771). Two consequences for the reading above. A version +commit can wait **up to 4 hours** for the first *scheduled* full run on or after it; +the gate does not wait for one, because a completed `push` run nearer the version +commit decides first (`FULL_BATTERY_EVENTS` reads both), and no window in the gate +is a clock — it reads the nearest completed run at or before the version commit, +however old. And a `push` run's `Test Core` is affected-only (#16467), so when the +nearest verdict is a push run it is the narrower one. If you want the whole-workspace +verdict on the version commit itself and `main` has not moved past it, run **Actions → +CI → Run workflow** on `main` (and the same for **Lint & Type Check**) and wait for it +rather than for the next scheduled run; once `main` has moved, a dispatched run tests a +later tree and the gate rightly ignores it. + **This is a wait, not a merge freeze.** `main` keeps moving the whole time: the approval publishes the version commit's tree and nothing that landed after it, so runs on later commits are not what you are waiting for. The snapshot flow above diff --git a/scripts/ci/scheduled-full-run.mjs b/scripts/ci/scheduled-full-run.mjs index 7d46e326236..64b25636746 100644 --- a/scripts/ci/scheduled-full-run.mjs +++ b/scripts/ci/scheduled-full-run.mjs @@ -17,8 +17,8 @@ // ## Why one file holds the first two halves // // `push` on `main` now tests the AFFECTED set. The thing that keeps "main is -// green" a statement about the whole workspace is the hourly `schedule` run of -// ci.yml and lint.yml. Two properties have to hold for that to be true, and +// green" a statement about the whole workspace is the `schedule` run of +// ci.yml and lint.yml (hourly from #16467, every 4 hours since #25771). Two properties have to hold for that to be true, and // neither is visible in the run that would be wrong: // // 1. THE RUN HAS TO SURVIVE. On a `schedule` event `github.event.pull_ @@ -68,7 +68,7 @@ import { isEntrypoint } from '../invoked-as.mjs'; // `Unlock-action:` key exactly as the patrol reads it. import { blockerTargetsFor, directiveValues, h19BlockOutlivedBlocker, labelNames } from '../pm/check-half-states.mjs'; -// The workflows that carry the hourly full run. Spelled as quoted repo-relative +// The workflows that carry the scheduled full run (every 4 hours, #25771). Spelled as quoted repo-relative // literals: the dispatch derivation reads a gate's path literals as the // population it watches, so a card touching either file schedules this family. export const WATCHED_WORKFLOWS = Object.freeze([ @@ -169,7 +169,7 @@ export function eventContext( }; const events = { push: { ref: 'refs/heads/main', before: 'f'.repeat(40), after: sha }, - schedule: { schedule: '0 * * * *' }, + schedule: { schedule: '0 */4 * * *' }, workflow_dispatch: { ref: 'refs/heads/main', inputs: {} }, pull_request: { action: 'synchronize', number: pr, after: sha, pull_request: { number: pr } }, merge_group: { action: 'checks_requested', merge_group: { head_sha: sha } }, @@ -205,7 +205,7 @@ export function concurrencyVerdict(groupExpression, { workflow = 'W' } = {}) { if (schedule === push) { findings.push( `a \`schedule\` run and a \`push\` run share the concurrency group '${schedule}', so the next ` + - 'merge to main cancels the hourly full run. Put `github.event_name` in the key.' + 'merge to main cancels the scheduled full run. Put `github.event_name` in the key.' ); } if (dispatch === push) { @@ -217,13 +217,13 @@ export function concurrencyVerdict(groupExpression, { workflow = 'W' } = {}) { if (dispatch === schedule) { findings.push( `a \`workflow_dispatch\` run and a \`schedule\` run share the concurrency group '${dispatch}', so ` + - 'an on-demand full run cancels the hourly one it was meant to pre-empt.' + 'an on-demand full run cancels the scheduled one it was meant to pre-empt.' ); } // EVERY PUSH RUN FINISHES (#25373). A push run is the only run that tests // its own merge's `--affected` set; a cancelled one leaves that merge with - // no verdict on main until the next hourly run. Sharing a group between two + // no verdict on main until the next scheduled full run. Sharing a group between two // merges lets the second cancel the first -- and `cancel-in-progress: false` // does not save it, because GitHub keeps ONE pending run per group and // cancels that one when a third arrives. So two pushes of two different @@ -241,7 +241,7 @@ export function concurrencyVerdict(groupExpression, { workflow = 'W' } = {}) { // inequality above is to put something run-unique in the key -- // `github.run_id`, `github.sha` -- which also switches `cancel-in-progress` // off for every OTHER event in the file. So everything that is not a push - // must still collapse two runs onto one group: two hourly runs at two `main` + // must still collapse two runs onto one group: two scheduled runs at two `main` // heads, two on-demand runs, and two heads of the SAME pull request. if (group('schedule', SECOND) !== schedule) { findings.push('two `schedule` runs on two `main` heads do not share a group (the key is run-unique off push)'); @@ -273,10 +273,10 @@ export function concurrencyVerdict(groupExpression, { workflow = 'W' } = {}) { // that can be swallowed files a duplicate an hour. // // Both prefixes carry the watched workflow's CADENCE (#23880): the card for a -// nightly workflow must not call itself hourly. For an hourly workflow they -// spell exactly what they always did -- `hourly full run: red on main` and -// `os-hourly-full-run` -- so an open card filed before the cadence existed is -// still found by its title and by its marker, and refreshed, never duplicated. +// nightly workflow must not call itself hourly. When a workflow's cadence +// CHANGES, the card already open under the old one must still be found -- by +// its old title and by its old marker -- and refreshed, never duplicated; that +// is what `CADENCE_PREDECESSORS` below is for. const markerPrefix = (cadence) => `os-${cadence}-full-run`; const titlePrefix = (cadence) => `${cadence} full run: red on main`; @@ -287,20 +287,34 @@ const titlePrefix = (cadence) => `${cadence} full run: red on main`; // and the card is the only reader of that run. Adding a workflow to the card's // watched list therefore means adding its cadence here in the same change. export const CARD_CADENCES = Object.freeze({ - CI: 'hourly', - 'Lint & Type Check': 'hourly', + CI: '4-hourly', + 'Lint & Type Check': '4-hourly', }); -// The words each cadence puts in the body. The `hourly` entry is the text the -// body has always carried, byte for byte. +// THE CADENCES A CADENCE REPLACED, newest first. `CI` and `Lint & Type Check` +// ran hourly from #16467 until #25771 (maintainer ruling Q6 = C) moved both to +// every 4 hours, and the CI card open at that change (#25438) carries the +// hourly title `hourly full run: red on main (CI)` and the hourly marker +// `os-hourly-full-run-ci`. `cardIdentity` turns each predecessor into a +// `legacy` title and marker that `matchesCard` still reads, so that card is +// refreshed, never orphaned beside a duplicate. A refresh rewrites the body +// with the CURRENT marker and leaves the title alone (a title is the seat's to +// edit, and `updateIssue` writes only the body), so the old title is matched +// for as long as such a card stays open. ⛔ Never drop an entry while a card +// it names may be open. +export const CADENCE_PREDECESSORS = Object.freeze({ + '4-hourly': Object.freeze(['hourly']), +}); + +// The words each cadence puts in the body. const CADENCE_WORDS = Object.freeze({ - hourly: Object.freeze({ - every: 'every hour', - previous: 'an hour ago', + '4-hourly': Object.freeze({ + every: 'every 4 hours', + previous: '4 hours ago', about: (name) => [ - `\`${name}\` runs the FULL battery on \`main\` every hour (#16467). A \`push\` run on`, + `\`${name}\` runs the FULL battery on \`main\` every 4 hours (#16467, #25771). A \`push\` run on`, '`main` tests only the packages the merge touched, and a merge-queue build tests only what the group', - 'contained — so this hourly run is the only thing that says whether the WHOLE tree is green, and', + 'contained — so this scheduled run is the only thing that says whether the WHOLE tree is green, and', 'this card is the only channel that reports it. Nothing is blocked by it.', ], }), @@ -343,19 +357,26 @@ export function cardIdentity(workflowName) { cadence, title: `${titlePrefix(cadence)} (${name})`, marker: `${markerPrefix(cadence)}-${slug}`, + // Read by `matchesCard` only; a card is never FILED under a legacy identity. + legacy: (CADENCE_PREDECESSORS[cadence] ?? []).map((old) => + Object.freeze({ title: `${titlePrefix(old)} (${name})`, marker: `${markerPrefix(old)}-${slug}` }) + ), labels: ['bug', 'domain:devx', 'priority:p1'], }; } // Is this issue THIS card? Two ways in, because either can be edited away: the -// title prefix, and the body marker. A PULL REQUEST is never this card -- +// title prefix, and the body marker -- each in the current cadence's spelling +// or in a predecessor's (`identity.legacy`). A PULL REQUEST is never this card -- // `listForRepo` returns PRs too, and a PR titled after this card would // otherwise swallow the file and the real red would never be reported. export function matchesCard(issue, identity) { if (!issue || issue.pull_request) return false; const title = String(issue.title ?? ''); const body = String(issue.body ?? ''); - return title.startsWith(identity.title) || body.includes(identity.marker); + return [identity, ...(identity.legacy ?? [])].some( + (spelling) => title.startsWith(spelling.title) || body.includes(spelling.marker) + ); } // The de-dup decision, over a PAGED listing of OPEN issues. @@ -663,7 +684,7 @@ export function withBlockNote(body, noteLines) { // The card body. The run link and the commit range are the whole product: a red // scheduled run says "main broke somewhere since the last green one" -- the last -// hour for an hourly workflow, the last night for a nightly one -- and the +// 4 hours for a 4-hourly workflow, the last night for a nightly one -- and the // commits between the previous GREEN scheduled run and this one are that span. export function renderBody({ identity, @@ -753,7 +774,7 @@ export function renderBody({ // // A PR or queue build carries the FAST SET since #22998 (ci.yml's THE TWO // SPEEDS); the dogfood and temporal legs run only on `main`'s -// push, hourly schedule and dispatch runs. "Tested before a release" is +// push, 4-hourly schedule and dispatch runs. "Tested before a release" is // therefore a statement about those runs, and this half makes it a gate: // release.yml's `full-battery` job, which `publish` needs, so a refusal here // means no deployment is queued at the `release` environment at all. @@ -778,7 +799,7 @@ export function renderBody({ // // ⛔ It never waits and never reads a run AFTER the release commit: a run on a // later `main` head tested a different tree. ⚠️ A `push` run's Test Core is -// affected-only (#16467), so a push run is a narrower verdict than an hourly +// affected-only (#16467), so a push run is a narrower verdict than a scheduled // one; the ruled rule reads both events, and that narrowing is recorded on // #22998 rather than decided here. @@ -808,8 +829,12 @@ export const RELEASE_GATE_JOB = 'full-battery'; export const RELEASE_PUBLISH_JOB = 'publish'; // How many first-parent commits the window spans, and how many pages of 100 -// completed runs per event per workflow are read. Generous against an hourly -// cadence: the nearest verdict is normally a handful of commits back. +// completed runs per event per workflow are read. Generous against the 4-hourly +// schedule (#25771): every merge's own push run carries a verdict since #25373, +// so the nearest verdict is normally a handful of commits back, and 300 +// schedule runs are 50 days of them. Neither number is a clock: the gate never +// asks how OLD the deciding run is, only that it is the nearest one at or +// before the release commit -- so the cadence moved no freshness window here. export const RELEASE_GATE_WINDOW = 400; export const RELEASE_GATE_PAGES = 3; @@ -1047,7 +1072,7 @@ async function releaseGate(argv) { // remedy is to find what stopped registering, never to lower the number. const SELF_TEST_BATTERIES = Object.freeze({ 'scheduled-full-run concurrency isolation': 32, - 'scheduled-full-run card identity and de-dup': 34, + 'scheduled-full-run card identity and de-dup': 42, 'card refresh: a stale pm:blocked (the measured #23885 sequence)': 22, 'release gate: the selection rule on the recorded 2026-10-12 listing': 8, 'release gate: the selection rule on synthetic runs': 17, @@ -1282,10 +1307,10 @@ async function selfTest() { const lint = cardIdentity('Lint & Type Check'); await check(() => { - if (ci.title !== 'hourly full run: red on main (CI)') throw new Error(`identity: CI title is '${ci.title}'`); + if (ci.title !== '4-hourly full run: red on main (CI)') throw new Error(`identity: CI title is '${ci.title}'`); }); await check(() => { - if (lint.marker !== 'os-hourly-full-run-lint-type-check') throw new Error(`identity: lint marker is '${lint.marker}'`); + if (lint.marker !== 'os-4-hourly-full-run-lint-type-check') throw new Error(`identity: lint marker is '${lint.marker}'`); }); await check(() => { if (ci.marker === lint.marker || ci.title === lint.title) { @@ -1372,9 +1397,22 @@ async function selfTest() { if (found?.number !== 42) throw new Error('de-dup: a multi-page scan lost the card'); }); - // -- The cadence (#23880): each cadence, the back-compat pins for the card - // already open under the hourly identity, and the refusal of a workflow - // the closed map does not name. + // -- The cadence (#23880, #25771): each cadence, the back-compat pins for the + // card already open under the HOURLY identity (#25438, filed before the + // move to every 4 hours), and the refusal of a workflow the closed map + // does not name. + // + // THE CHOICE #25771 MADE, pinned here: the identity MOVES to the new + // cadence (`4-hourly full run: red on main (CI)`, `os-4-hourly-full-run-ci`) + // and the hourly spellings stay READABLE through `CADENCE_PREDECESSORS`. + // A new card files under the new identity; the open hourly card is found + // by its old title or its old marker, refreshed, and never duplicated. + const CURRENT_CI_TITLE = '4-hourly full run: red on main (CI)'; + const CURRENT_CI_MARKER = 'os-4-hourly-full-run-ci'; + const CURRENT_CI_MARKER_LINE = + 'os-4-hourly-full-run-ci — machine-findable marker for this generated card. ⛔ Do not delete this line: ' + + 'it is how the 4-hourly full run finds this card instead of filing a new one every 4 hours.'; + // #25438 as it stood on 2026-10-26, byte for byte: title and first body line. const OPEN_CI_TITLE = 'hourly full run: red on main (CI)'; const OPEN_CI_MARKER = 'os-hourly-full-run-ci'; const OPEN_CI_MARKER_LINE = @@ -1388,36 +1426,105 @@ async function selfTest() { renderBody({ identity: identityUnderTest, runUrl: 'u', headSha: 'a', conclusion: 'failure', sweptAt: 't', rangeNote: 'n' }), ]; await check(() => { - if (ci.cadence !== 'hourly' || ci.title !== OPEN_CI_TITLE || ci.marker !== OPEN_CI_MARKER) { - throw new Error(`cadence: the CI identity moved (${ci.cadence}, '${ci.title}', '${ci.marker}') -- the open card would be duplicated`); + if (ci.cadence !== '4-hourly' || ci.title !== CURRENT_CI_TITLE || ci.marker !== CURRENT_CI_MARKER) { + throw new Error(`cadence: the CI identity is (${ci.cadence}, '${ci.title}', '${ci.marker}'), not the 4-hourly one`); + } + }); + await check(() => { + const legacy = ci.legacy ?? []; + if (!legacy.some((l) => l.title === OPEN_CI_TITLE && l.marker === OPEN_CI_MARKER)) { + throw new Error(`cadence: the CI identity lost the hourly spelling #25438 carries (${JSON.stringify(legacy)}) -- that card would be orphaned`); } }); await check(() => { for (const b of bodiesOf(ci)) { - if (b.split('\n')[0] !== OPEN_CI_MARKER_LINE) throw new Error(`cadence: the CI marker line moved: '${b.split('\n')[0]}'`); + if (b.split('\n')[0] !== CURRENT_CI_MARKER_LINE) throw new Error(`cadence: the CI marker line is '${b.split('\n')[0]}'`); } }); await check(async () => { - // The open CI card exactly as it was filed before the cadence existed. - const open = { number: 23885, title: OPEN_CI_TITLE, body: `${OPEN_CI_MARKER_LINE}\n# ${OPEN_CI_TITLE}` }; + // #25438 exactly as filed under the hourly identity: found by its title. + const open = { number: 25438, title: OPEN_CI_TITLE, body: `${OPEN_CI_MARKER_LINE}\n# ${OPEN_CI_TITLE}` }; const found = await findExistingCard({ identity: cardIdentity('CI'), listPage: page([issue(7), open]) }); - if (found?.number !== 23885) throw new Error(`cadence: the open CI card was not found by its title (${found?.number})`); + if (found?.number !== 25438) throw new Error(`cadence: #25438 was not found by its hourly title (${found?.number})`); }); await check(async () => { - const renamed = { number: 23885, title: 'renamed by a human', body: `${OPEN_CI_MARKER_LINE}\n` }; + // ...and by its hourly marker alone, after a human renamed it. + const renamed = { number: 25438, title: 'renamed by a human', body: `${OPEN_CI_MARKER_LINE}\n` }; const found = await findExistingCard({ identity: cardIdentity('CI'), listPage: page([renamed]) }); - if (found?.number !== 23885) throw new Error('cadence: the open CI card was not found by its marker alone'); + if (found?.number !== 25438) throw new Error('cadence: #25438 was not found by its hourly marker alone'); + }); + await check(async () => { + // A red 4-hourly run REFRESHES #25438 and files no second card; the + // refreshed body carries the current marker, so it is found by that too. + const card = { number: 25438, title: OPEN_CI_TITLE, body: `${OPEN_CI_MARKER_LINE}\n# ${OPEN_CI_TITLE}`, labels: [] }; + const filed = []; + const outcome = await fileOrRefreshCard({ + identity: ci, + body: bodiesOf(ci)[0], + ownerRepo: 'objectstack-ai/objectstack', + listPage: async (n) => (n === 1 ? [card] : []), + createIssue: async (made) => { + filed.push(made); + return { number: 1 }; + }, + updateIssue: async ({ number, body: b }) => { + if (number === card.number) card.body = b; + }, + listComments: async () => [], + getIssue: async () => ({}), + addLabels: async () => {}, + removeLabel: async () => {}, + }); + if (outcome.action !== 'refreshed' || outcome.number !== 25438 || filed.length !== 0) { + throw new Error(`cadence: a red 4-hourly run did not refresh #25438 (${outcome.action}, ${outcome.number}, filed ${filed.length})`); + } + if (card.body.split('\n')[0] !== CURRENT_CI_MARKER_LINE) throw new Error('cadence: the refreshed #25438 body lost the current marker'); + const again = await findExistingCard({ identity: ci, listPage: page([{ ...card, title: 'renamed by a human' }]) }); + if (again?.number !== 25438) throw new Error('cadence: the refreshed #25438 is not found by the current marker'); + }); + await check(async () => { + // A legacy spelling is READ, never FILED: an empty board files the new title. + const filed = []; + await fileOrRefreshCard({ + identity: ci, body: 'b', ownerRepo: 'objectstack-ai/objectstack', + listPage: async () => [], + createIssue: async (made) => { + filed.push(made); + return { number: 1 }; + }, + updateIssue: async () => {}, listComments: async () => [], getIssue: async () => ({}), + addLabels: async () => {}, removeLabel: async () => {}, + }); + if (filed[0]?.title !== CURRENT_CI_TITLE) throw new Error(`cadence: a new card was filed as '${filed[0]?.title}'`); + }); + await check(async () => { + // The Lint legacy spelling stays Lint's: CI's hourly card is not Lint's card. + const found = await findExistingCard({ identity: lint, listPage: page([{ number: 25438, title: OPEN_CI_TITLE, body: OPEN_CI_MARKER_LINE }]) }); + if (found) throw new Error(`cadence: Lint's identity matched CI's hourly card #${found.number}`); }); await check(() => { - if (lint.cadence !== 'hourly' || lint.title !== 'hourly full run: red on main (Lint & Type Check)') { - throw new Error(`cadence: Lint & Type Check is not hourly ('${lint.title}')`); + if (lint.cadence !== '4-hourly' || lint.title !== '4-hourly full run: red on main (Lint & Type Check)') { + throw new Error(`cadence: Lint & Type Check is not 4-hourly ('${lint.title}')`); + } + if (!(lint.legacy ?? []).some((l) => l.title === 'hourly full run: red on main (Lint & Type Check)' && l.marker === 'os-hourly-full-run-lint-type-check')) { + throw new Error('cadence: Lint & Type Check lost its hourly spelling'); } }); await check(() => { for (const b of bodiesOf(ci)) { - if (/nightly|every night|the night before/.test(b)) throw new Error(`cadence: an hourly card body says nightly:\n${b}`); + if (/nightly|every night|the night before/.test(b)) throw new Error(`cadence: a 4-hourly card body says nightly:\n${b}`); + if (/every hour\b|an hour ago|this hourly run/.test(b)) throw new Error(`cadence: a 4-hourly card body still says hourly:\n${b}`); } }); + for (const file of WATCHED_WORKFLOWS) { + await check(() => { + // The cadence the card prints is the cadence the workflow runs. + const crons = [...readFileSync(file, 'utf8').matchAll(/^\s*-\s*cron:\s*'([^']+)'/gm)].map((m) => m[1]); + if (crons.length !== 1 || crons[0] !== '0 */4 * * *') { + throw new Error(`cadence: ${file} declares ${JSON.stringify(crons)}, but CARD_CADENCES says 4-hourly ('0 */4 * * *')`); + } + }); + } await check(() => { let message = ''; try { @@ -1671,7 +1778,7 @@ async function selfTest() { }); await check(() => { const lines = measured.card.body.split('\n'); - if (lines[0] !== OPEN_CI_MARKER_LINE) throw new Error('measured: the refreshed body lost its marker line'); + if (lines[0] !== CURRENT_CI_MARKER_LINE) throw new Error('measured: the refreshed body lost its marker line'); if (lines.at(-1) !== CARD_FOOTER) throw new Error('measured: the note was not placed above the footer'); for (const line of NEW_BODY.split('\n')) { if (!lines.includes(line)) throw new Error(`measured: the refreshed body dropped '${line}'`); @@ -2338,7 +2445,7 @@ function checkConcurrency() { if (bad > 0) { console.error( `scheduled-full-run: ${bad} workflow file(s) carry a concurrency key that lets one main run cancel ` + - 'another: the hourly full run is the only thing testing the whole workspace on main since push ' + + 'another: the scheduled full run is the only thing testing the whole workspace on main since push ' + "went affected-only, and a push run is the only test of its own merge's affected set, so this is a " + 'coverage refusal, not a style one.' ); diff --git a/scripts/ci/select-gate-families.selftest.sh b/scripts/ci/select-gate-families.selftest.sh index 2ba748c1e1b..c4a2a529c06 100644 --- a/scripts/ci/select-gate-families.selftest.sh +++ b/scripts/ci/select-gate-families.selftest.sh @@ -304,7 +304,7 @@ expect_reason entry_guard "event 'push' is not scoped" expect_reason bare_root_worklist "event 'push' is not scoped" expect_line 'Gate families: 9 run, 0 skipped' -run_case 'schedule: every family runs (the hourly full run keeps the battery)' "$REPO" schedule '' '' +run_case 'schedule: every family runs (the 4-hourly full run keeps the battery)' "$REPO" schedule '' '' expect_rc 0 expect_warnings '' '' expect_all_run @@ -321,7 +321,7 @@ expect_all_run # The backstop, held over the very change set the PR path now lets the PM # self-test skip (#22076): an unscoped event never reads the diff. S=$(scenario M:scripts/slot-lookup-baseline.json M:AGENTS.md M:skills/x/SKILL.md) -run_case 'schedule: a change the PR path skips the PM self-test for still runs it -- the hourly full run is the backstop (#22076 control)' "$REPO" schedule '' '' +run_case 'schedule: a change the PR path skips the PM self-test for still runs it -- the 4-hourly full run is the backstop (#22076 control)' "$REPO" schedule '' '' expect_rc 0 expect_warnings '' '' expect_all_run diff --git a/scripts/ci/select-gate-families.sh b/scripts/ci/select-gate-families.sh index ab5e541d810..c286ae23ad2 100644 --- a/scripts/ci/select-gate-families.sh +++ b/scripts/ci/select-gate-families.sh @@ -167,7 +167,7 @@ # # ⛔ So a defect these families would have caught can now first appear on # `main` instead of on the PR that wrote it. What keeps that bounded is the -# unscoped half, unchanged by this: `push` on main and the hourly scheduled run +# unscoped half, unchanged by this: `push` on main and the 4-hourly scheduled run # take the `*)` branch above and run the whole battery, and lint.yml's `on:` # block owns that argument. Widening this set further -- moving another step # behind the selector, or making one of these read-sets smaller -- is again a @@ -195,7 +195,7 @@ # # For a MODIFICATION that is the ratchet-grade claim, not #19498's weaker one: # no case reads these files' bytes or anything computed from them, so no case -# can change verdict. What moves from PR time to push-on-main and the hourly +# can change verdict. What moves from PR time to push-on-main and the 4-hourly # run is the NAME half -- an ADDED file in these two classes, which the # tracked-name sweep sees and which, like an added file in every other class # since #19498, is judged here by its class and not its status. @@ -504,7 +504,7 @@ family_reads() { # for the compound-anchor sweep, the nested .gitignore files the # exposed-scratch-dir sweep consults, and the tracked NAME sweep that # made an ADDED path anywhere run it. Those reads are real; they run on - # push-on-main and on the hourly full run, and not on a PR. + # push-on-main and on the 4-hourly full run, and not on a PR. case "$path" in scripts/pm/*|.claude/skills/pm-dispatch/*) return 0 ;; esac diff --git a/scripts/ci/select-shard-packages.selftest.sh b/scripts/ci/select-shard-packages.selftest.sh index 11fac31bb02..b844d0762bb 100644 --- a/scripts/ci/select-shard-packages.selftest.sh +++ b/scripts/ci/select-shard-packages.selftest.sh @@ -367,7 +367,7 @@ expect_turbo 'affected only' "$(affected_call "$C1")" expect_union 'not reached' '' # ── the full-battery events (#16467) ──────────────────────────────────────── -# `schedule` is the hourly full run and `workflow_dispatch` is the same battery +# `schedule` is the 4-hourly full run and `workflow_dispatch` is the same battery # on demand. Neither carries a diff base, and neither may acquire one by # accident: both cases below hand the script every base variable it knows and # pin that it still asks `turbo ls` for the whole workspace. diff --git a/scripts/ci/select-shard-packages.sh b/scripts/ci/select-shard-packages.sh index 09afd7187d3..e8452a36c99 100755 --- a/scripts/ci/select-shard-packages.sh +++ b/scripts/ci/select-shard-packages.sh @@ -35,7 +35,7 @@ # A ZERO before sha (a branch's first push, or a force-push that # rewrote history) has no previous commit to diff from and falls # back to the FULL list, loudly. -# schedule the FULL list. This is the hourly run (#16467) that keeps +# schedule the FULL list. This is the 4-hourly run (#16467, #25771) that keeps # workflow_ "main is green" a statement about the WHOLE workspace, and it # dispatch -- not the push run -- is the measurement the shard-timings # refresh reads (scripts/ci/select-shard-timings-run.mjs). @@ -279,7 +279,7 @@ else # refresh reads. Both halves of that are retired (#16467): the full run cost # ~90 machine-hours a day to re-test a tree the merge group had just verified, # and 36 of 60 of those runs never finished, so it was not ground truth for - # anything -- the HOURLY `schedule` run is, and `scripts/ci/select-shard- + # anything -- the 4-HOURLY `schedule` run is, and `scripts/ci/select-shard- # timings-run.mjs` now reads that event by name. Since #16467 a push lands # here only when its `before` sha is absent, zero, or unresolvable, and says # so just below; merge groups likewise since #16453. What still takes this diff --git a/scripts/ci/select-shard-timings-run.mjs b/scripts/ci/select-shard-timings-run.mjs index 2258e312bf5..704c26323c8 100644 --- a/scripts/ci/select-shard-timings-run.mjs +++ b/scripts/ci/select-shard-timings-run.mjs @@ -19,9 +19,9 @@ // Test Core finishes: 36 of the last 60 push runs were censored that way. // Such a run leaves SOME artifacts, so "the artifacts exist" is not the // test -- every one of the run's Test Core jobs must have concluded -// `success`. The hourly run this now reads has its OWN concurrency group +// `success`. The scheduled run this now reads has its OWN concurrency group // (#16467, `github.event_name` is in ci.yml's key), so a merge cannot -// censor it -- but a second hourly run can, which is why this test stays. +// censor it -- but a second scheduled run can, which is why this test stays. // // 2. CACHE REPLAYS. A run whose shard jobs all concluded `success` can still // have replayed most of the workspace from the turbo cache: shards were @@ -34,7 +34,8 @@ // catches this, and it catches it by looking at what was MEASURED rather // than at a duration threshold nobody can defend. // -// 3. EXPIRED ARTIFACTS. Retention is 1 day, so the eligible window is +// 3. EXPIRED ARTIFACTS. Retention is 4 days on a scheduled run (1 day on +// every other event, #25771), so the eligible window is // genuinely short and an empty candidate list is a normal Monday if main // was quiet -- it is reported as a loud, named refusal rather than as a // no-op, because "no PR opened" and "no run was good enough" must not read @@ -63,8 +64,8 @@ // one package at a time, which is exactly the silent rot this lane exists to // end. // -// So the event is named, and it is the HOURLY SCHEDULED RUN: `schedule` on -// ci.yml, minute 0, the one event the selection script treats as FULL by +// So the event is named, and it is the SCHEDULED RUN: `schedule` on ci.yml, +// minute 0 every 4 hours (hourly until #25771), the one event the selection script treats as FULL by // construction. `DEFAULT_RUN_EVENT` below is that name, spelled once. // // Usage: @@ -441,17 +442,17 @@ export async function listCandidates({ // exist. So the two outcomes are split, with two exits and two messages: // // EMPTY CANDIDATE LIST -- the API has no completed run of this event at all. -// The PREREQUISITE (an hourly run has happened, recently enough to still +// The PREREQUISITE (a scheduled run has happened, recently enough to still // hold its artifacts) is NOT MET, nothing was measured, and the code is the // repo-wide `EXIT_PREREQUISITE_NOT_MET` -- ⛔ never a finding's -// `EXIT_FINDINGS`. This is the bootstrap window: at least an hour between -// the trigger landing and the first hourly run finishing. +// `EXIT_FINDINGS`. This is the bootstrap window: up to 4 hours between +// the trigger landing and the first scheduled run finishing. // // CANDIDATES EXISTED, NONE WAS ELIGIBLE -- every one really was censored, // failed, or lost its artifacts. That IS a finding, it keeps exit 1, and it // keeps the sentence naming those causes, because now they are justified. // -// ⛔ A persistent NOT MEASURED is NOT a steady state. Once the hourly run has +// ⛔ A persistent NOT MEASURED is NOT a steady state. Once the scheduled run has // been live for a while, "zero completed schedule runs" stops meaning bootstrap // and starts meaning the trigger was removed or every run is being cancelled -- // a defect to file, not a green day. Both the message below and the workflow @@ -468,13 +469,13 @@ export function noCandidatesText({ event, workflow, limit }) { '\n so no run was censored, none failed, and none lost its artifacts. The dataset was left exactly' + '\n as it is, which is the correct outcome for this reading.' + '\n' + - `\n Expected while the hourly \`${event}\` run is bootstrapping: the trigger has to land on main and` + - '\n one run has to finish before anything is downloadable, and run-summary artifacts are retained' + - '\n for 1 day.' + + `\n Expected while the scheduled \`${event}\` run is bootstrapping: the trigger has to land on main and` + + '\n one run has to finish before anything is downloadable, and a scheduled run\'s run-summary' + + '\n artifacts are retained for 4 days.' + '\n' + '\n ⛔ PERSISTENTLY NOT MEASURED IS A DEFECT, NOT A STEADY STATE. If this keeps saying zero after' + - `\n the hourly run has been live for a few hours, the \`${event}\` trigger is gone from ${workflow} or` + - '\n every hourly run is being cancelled -- and the balancing dataset is quietly ageing out. File it.' + + `\n the scheduled run has been live for a day, the \`${event}\` trigger is gone from ${workflow} or` + + '\n every scheduled run is being cancelled -- and the balancing dataset is quietly ageing out. File it.' + '\n' + `\n (Exit code ${EXIT_PREREQUISITE_NOT_MET}, distinct from a finding's ${EXIT_FINDINGS} -- capture it BEFORE any pipe:` + '\n `node scripts/ci/select-shard-timings-run.mjs --candidates > /tmp/candidates.log 2>&1; echo "EXIT=$?"`.' + @@ -489,7 +490,7 @@ export function noEligibleRunText({ examined, event, workflow }) { return ( `select-shard-timings-run: NO ELIGIBLE RUN among the ${examined} most recent completed ` + `\`${event}\` runs of ${workflow} on main. Every one was censored, failed, or has lost its ` + - 'run-summary artifacts to the 1-day retention window. This is a refusal, not a no-op: nothing ' + + 'run-summary artifacts to the retention window (4 days on a scheduled run). This is a refusal, not a no-op: nothing ' + 'was regenerated.' ); } @@ -501,7 +502,7 @@ export function noEligibleRunText({ examined, event, workflow }) { export async function runCandidates({ repo, token, limit, event, workflow = 'ci.yml', fetchImpl = fetch }) { const notes = [ `select-shard-timings-run: examining the ${limit} most recent completed \`${event}\` runs of ` + - `${workflow} on main. The hourly \`schedule\` run is the FULL battery; a \`push\` run has been ` + + `${workflow} on main. The \`schedule\` run is the FULL battery; a \`push\` run has been ` + 'affected-only since #16467 and is not a measurement of the workspace.', ]; const examined = await listCandidates({ repo, token, limit, event, workflow, fetchImpl }); @@ -1059,7 +1060,7 @@ async function selfTest() { if (DEFAULT_RUN_EVENT === 'push') throw new Error('default event: the affected-only event is back as the default'); }); - // THE READING: by default the selector asks for the hourly scheduled run and + // THE READING: by default the selector asks for the scheduled run and // gets it, and never asks for a push run at all. const defaultCall = await call(); check(() => { diff --git a/scripts/partition-test-shards.mjs b/scripts/partition-test-shards.mjs index d2ecd156af6..d3757cbb9f4 100644 --- a/scripts/partition-test-shards.mjs +++ b/scripts/partition-test-shards.mjs @@ -401,9 +401,10 @@ export const FILE_SHARDED_PACKAGES = Object.freeze({ // matching neither map is refused as before. // // Set it to the OUTGOING map in the same PR that changes FILE_SHARDED_PACKAGES. -// Its only readers are run summaries, which ci.yml keeps for one day -// (`retention-days: 1` on `test-core-run-summary-*`), so a day after that PR -// lands no retained summary predates the change and this map decodes nothing. +// Its only readers are run summaries, which ci.yml keeps for four days on a +// scheduled run and one day otherwise (`retention-days` on +// `test-core-run-summary-*`, #25771), so four days after that PR lands no +// retained summary predates the change and this map decodes nothing. // The outgoing map of #22075's third round was empty -- the runs before it // carried no slice -- so this one is empty too. export const PREVIOUS_FILE_SHARDED_PACKAGES = Object.freeze({}); From ac31cc8cbfde9b0996f24c90316ede13e49d79fc Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 26 Oct 2026 06:49:19 +0000 Subject: [PATCH 2/2] ci: the two remaining present-tense "hourly" sentences follow the 4-hourly schedule shard-timings-refresh.yml's Save-Turbo-cache paragraph and docs-deploy.yml's cron-collision reasoning (the 0/6/12/18 set still holds: the 4-hourly ci/lint runs meet it at 00 and 12 only). Co-authored-by: Claude Claude-Session: https://claude.ai/code/session_01AjoNWTpihGQZrTamr1vd4W --- .github/workflows/docs-deploy.yml | 6 ++++-- .github/workflows/shard-timings-refresh.yml | 11 ++++++----- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/.github/workflows/docs-deploy.yml b/.github/workflows/docs-deploy.yml index 8e00922bf1e..d1320f414c0 100644 --- a/.github/workflows/docs-deploy.yml +++ b/.github/workflows/docs-deploy.yml @@ -57,8 +57,10 @@ on: # snapshot owns :07 of 02, 08, 14 and 20. Of the six six-hour sets, only # 0/6/12/18 and 1/7/13/19 have hours with nothing scheduled after :00; the # first is taken because its :00 neighbours are short (release.yml's - # version-pr refresh, the hourly ci/lint runs), while 07:00 starts the - # showcase smoke. The MINUTE is off the top of the hour for the reason every + # version-pr refresh, the ci/lint full runs), while 07:00 starts the + # showcase smoke. Since #25771 those ci/lint runs are 4-hourly + # (00/04/08/12/16/20 :00), so they now meet this set only at 00 and 12 and + # 06/18 lose that neighbour; the choice still holds and nothing moved. The MINUTE is off the top of the hour for the reason every # patrol here gives: scheduled runs queue behind everyone else's :00 cron. # # This job spends almost nothing of the per-repository GITHUB_TOKEN budget diff --git a/.github/workflows/shard-timings-refresh.yml b/.github/workflows/shard-timings-refresh.yml index 2d7a80a3d00..97fdbc87c45 100644 --- a/.github/workflows/shard-timings-refresh.yml +++ b/.github/workflows/shard-timings-refresh.yml @@ -88,11 +88,12 @@ # # ⛔ Do NOT make ci.yml's `Save Turbo cache (main only)` step fire on `schedule` # as well. It is `github.event_name == 'push'` on purpose and that is now -# load-bearing here: if the hourly run seeded the cache it restores, the next -# hourly run would replay almost the whole workspace, the generator would refuse -# every replayed task, and this lane would measure nothing — by construction, -# every hour, forever. Affected-only pushes seeding a narrower cache is the -# direction that HELPS: it leaves more real misses for the hourly run to time. +# load-bearing here: if the scheduled run seeded the cache it restores, the next +# scheduled run would replay almost the whole workspace, the generator would +# refuse every replayed task, and this lane would measure nothing — by +# construction, every 4 hours, forever. Affected-only pushes seeding a narrower +# cache is the direction that HELPS: it leaves more real misses for the +# scheduled run to time. # # ⚠ AND IT IS "RUNS", PLURAL, WHICH THE DOCUMENTED PROCEDURE DOES NOT SAY. # Measured on this lane's first live run (34083991141), when the runs it read