Skip to content

Commit 03243fc

Browse files
ci: prune each Test Core Turbo seed to its own run's tasks before saving it (#25523)
<!-- claude-projects-attribution: {"github_login":"os-elon-musk"} --> _Requested by **Elon** · [project thread](https://claude.ai/code/project/chan_01WedDzV26nniwxv7jC88B9D?thread=cmsg_01WedDzV26nniwxv7jC88B9DTQm7fHdg9y6VYJBP9Rc2ab)_ Fixes #25452 Clause-②: no Metadata-protocol: none Before each Test Core shard saves its Turbo seed on a push to `main`, the seed is now pruned to the cache entries its own run used. The Test Core seeds were not 2.4 GB of useful cache. They were one seed lineage that grew by one run per merge and was never trimmed. Pruning fixes this at the source, so the card's change (b) is the only change here. Change (a), dropping the Build Core / Type Check seeds, and change (c), the Next.js cache, are left alone: the measurement below shows those seeds are 0.05 GB each, not where the pool goes. ## Step 1: the measurement (save-step and restore-step logs; the pool API is refused from the cloud container) **Bytes saved per merge** (push run 39982654692, `main` at `05cc3ee953`, a completed run; read from each save step's `Sent N of N (100.0%)` line): | seed key prefix | per seed | seeds per merge | per merge | | :-- | --: | --: | --: | | `Linux-turbo-test-<1..6>-main-` | 2.410 – 2.441 GB | 6 | **14.60 GB** | | `Linux-turbo-build-core-main-` | 0.048 GB | 1 | 0.048 GB | | `Linux-turbo-dogfood-<1..3>-main-` | 0.048 GB | 3 | 0.145 GB | | `Linux-turbo-typecheck-main-` (lint.yml run 39982654477) | 0.054 GB | 1 | 0.054 GB | | **total** | | | **≈ 14.85 GB** (Test Core = 98 %) | Each merge writes about 1.5× the repository's 10 GB default pool. The jobs API shows every one of the six Test Core save steps succeeding on five more push runs from 16:23Z to 17:56Z (39986349408, 39989699728, 39994571067, 39996442855, 39999263178), including the cancelled runs. **Eviction, seen from a PR restore.** On PR run 39999049930 (17:55Z), `Test Core (1/6)` (job 120570291429) restored `Linux-turbo-test-3-main-fa06169c…` (2.50 GB). That is **shard 3's** seed, reached through the bare `Linux-turbo-test-` restore key. The push at 17:36Z (run 39996442855) had saved a shard-1 seed, but no `Linux-turbo-test-1-*` entry was left 19 minutes later. **Why 2.4 GB: the seed snowballs.** A full 76-task workspace build is 0.048 GB in Build Core's seed, and a shard's closure is a subset of that build. The restore step takes the newest seed by prefix, turbo never evicts its local cache, and the save step stored the whole directory. So every seed was the previous seed plus one run, and the bare-prefix fallback carried the lineage from shard to shard. That is why all six shards are the same size. The sizes keep growing: shard 3 was 2.441 GB at `05cc3ee9` (16:20Z) and 2.50 GB at `fa06169c` (17:36Z). The same growth reproduces on a two-package turbo 2.11.5 workspace: one input change took `.turbo/cache` from 9 files to 18, and pruning to the run summary took it back to 9 with a full `3 cached, 3 total` replay. **Pool total and largest entries: pending.** `GET /actions/cache/usage` and `/actions/caches` answer 403 from a cloud container ("Access to this GitHub Actions path is not permitted through this proxy"). The devx seat has asked the maintainer for the two reads, and they will be added here when they arrive. ## What changed - **`scripts/ci/prune-turbo-cache.mjs` (new).** It reads every run summary (`--summarize`) in `--run-summaries`, plus any `--extra-summaries`, and collects each task's `hash`. It then deletes every `<hash>.tar.zst` / `<hash>-meta.json` / `<hash>-manifest.json` in `.turbo/cache` whose hash no summary names, and appends `save=true|false` to `GITHUB_OUTPUT`. - **No readable run summary means `save=false`.** That covers a cancelled test step or an empty shard. An extra summary alone (the closure build's) does not count, because saving builds without test results would make the newest seed one that replays no tests. - A file in any other shape is **kept and reported**, never deleted, so a turbo layout change shows up as a count and does not empty the cache. - The script exits 0 on every run-time path, so it cannot fail the shard or withhold its attestation. Malformed arguments exit 3. - `--self-test` covers 25 cases across 5 batteries with a pinned battery floor and the verdict handshake. - **`ci.yml`, Test Core:** a new step, `Prune the Turbo cache to this run's tasks (main only)` (`if: always() && github.event_name == 'push'`), runs before `Save Turbo cache (main only)`. It reads `.turbo/runs` (the test step's summary) plus `$RUNNER_TEMP/turbo-remote-cache` (the closure build's summary, moved there by the remote-cache assertion). The save step's `if:` gains `&& steps.prune-turbo-cache.outputs.save == 'true'`. - **`ci.yml`, Build Core:** a new `prune-turbo-cache self-test` step next to `assert-turbo-remote-cache self-test`, and the script is added to the `core` filter's guard rows (`check-ci-filter-parity` and `check-self-test-wired` both require it). **Kept on purpose: the bare `Linux-turbo-test-` restore key.** Once seeds are pruned, a cross-shard restore brings at most another shard's pruned seed, about one closure's worth. The shared build entries still replay, and the prune removes the rest before the next save, so the fallback can no longer carry a lineage forward. Removing it would also take away the warm start its comment describes for a new shard number. ## Expected after Each Test Core seed should hold its shard's closure builds plus its test results, which is well under Build Core's 0.048 GB full build plus test logs. A merge would then write about 0.3–0.6 GB in total instead of about 14.85 GB. A side effect: PR shards stop downloading about 2.4 GB in `Restore Turbo cache`, which measured 15–46 s per shard on the PR runs above. The first push after landing still restores the old snowball once, prunes it, and saves the small seed. The old 2.4 GB entries then age out of the pool. ## Acceptance (from the card) - **Before:** the per-merge table above. The pool total and its four shares will be added when the maintainer's `gh api` reads arrive. - **After, owed after ≥ 5 merges to `main` post-landing:** the Test Core save-step sizes, the pool total (target ≤ 8 GB at steady state), and two readings an hour apart. Over the next 10 PR runs whose closure matches `main`: the median of the shard "Build this shard's dependency closure" step ≤ 1.7 min and Build Core's build step ≤ 2.5 min. The `assert-turbo-remote-cache` hit counts will be recorded before and after. - ⛔ The pool limit is not raised here. If the pool is still over budget after this lands, the billing option goes back to the maintainer with the numbers. ## Risks and rollback - **Test replay on `main` depends on this seed.** `test` / `test:repo` are off the remote cache (Build Core's PINS), so a pushed shard replays unchanged packages' tests only from the Actions seed. The prune keeps every hash the test step's summary names, which covers every test task and every build it depended on, so the replay set is unchanged. A test task that failed has no entry either way. - **A shard whose test step left no summary saves nothing.** The previous seed stays the newest, and the next completed push writes a fresh one. - **Rollback:** revert this PR. It touches no package and changes no restore key. ## Local verification (head fcae6e3) - `node scripts/ci/prune-turbo-cache.mjs --self-test`: OK, 25 cases across 5 batteries. **Firing proof:** with the keep rule ablated to keep everything (a temporary copy of the script), the self-test exits 1 with 5 failures (`every file of an unnamed hash is dropped`, `an inherited entry no summary names is deleted`, …). The copy was deleted afterwards. - A real turbo 2.11.5 workspace: prune to the run summary, delete `dist/` and the summaries, re-run, and the result is `Cached: 3 cached, 3 total` with outputs restored. - All exit 0: `check-self-test-wired` (+ `--self-test`), `check-ci-filter-parity` (+ `--self-test`), `check-step-collectors` (+ `--self-test`), `check-self-test-workflow-commands`, `check-published-list-mirrors`, `check-declaration-mirrors`, `check-platform-checklist-watchdog`, `check-empty-changeset --base origin/main`, `check-adr-0087-registration --base origin/main`, `check-aggregator-roster`, `check-closing-keyword-parity`, `check-comment-mask-corpus`, `check-shard-attestation --self-test`, and `pnpm check:` `workflow-status-functions`, `workflow-step-name-quoting`, `nul-bytes`, `merged-result`, `pm-governed-merges`, `pm-expected-skips`, `single-claim-paths`, `pnpm-filter-targets`, `pnpm-flag-spellings`, `scripts-symbol-anchors`, `runner-env-posture`, `node-version`, `pnpm-acquisition`, `select-shard-packages`, `shard-attestation`, `required-contexts`, `stall-guard-budget`, `stall-guard-headroom`, `issue-citations`, `optional-error-sink`, `doc-authoring`. The rest of the farm is left to CI. - `ci.yml` parses (`yaml`); eslint is clean on the new script. - `check-governed-merges.mjs --test .github/workflows/ci.yml scripts/ci/prune-turbo-cache.mjs`: GOVERNED, landing tier H (`.github/workflows/**` ×1), 331 changed lines. - `skip-changeset`: no published package is touched (`.github/workflows/**`, `scripts/ci/**`). ## Acceptance notes - **Branch name.** The branch is `claude/project-thread-53z7nt`, the name the session was given. The claim (comment 6371859957) names it. - **Card changes (a) and (c) not taken.** Measured at 0.05 GB per seed, they are not where the pool goes. Revisit them only if the after-reading still exceeds 8 GB. ## 维护者速读(草稿) **改了什么**:main 每次合并后,Test Core 的 6 个分片会各存一份 Turbo 缓存种子。现在存之前先把种子裁剪到「这次运行真正用到的任务」,其余旧条目删掉。如果测试步骤没有跑完(被取消或空分片),这次就不存。 **为什么改**:实测每份种子 2.4 GB,每次合并共写约 14.6 GB,而仓库的 Actions 缓存池只有 10 GB,结果是每合并一次,上一次的种子就被挤掉。原因是种子在滚雪球:每次都在恢复的旧种子上再加一轮,从不清理。完整构建一次的产物其实只有 0.05 GB。裁剪后预计每次合并只写 0.3–0.6 GB,PR 运行也不用再下载 2.4 GB 的种子。 **风险与代价(含回滚)**:main 上测试结果的复用只靠这份种子,所以本次运行用到的条目(含全部测试结果)一律保留,复用范围不变。没跑完的运行不存种子,旧种子继续可用。回滚只需 revert,不动任何包和恢复键。缓存池总量的读数还在等,拿到后补进正文。 **席位意见**:同意落地(devx 席位已在 head fcae6e3 复核,ACCEPT 见 #25452)。日志已证实种子在滚雪球:每次合并约 14.6 GB,远超 10 GB 的缓存池。修剪只保留本次运行用过的任务,失败模式偏保守:认不出的文件保留,没有运行摘要就不存。改动前的缓存池总量等你本机读出后补进正文,不影响审批。 **你要做的**:本 PR 改了 `.github/workflows/**`,属于 Tier H,需要 GOVERNED_APPROVERS 中的一位批准后,席位才会翻 ready 并落地。 --- _Generated by [Claude Code](https://claude.ai/code/session_01SawdJgiuoz8deqVdM51jAE)_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01SawdJgiuoz8deqVdM51jAE Co-authored-by: Claude <noreply@anthropic.com>
1 parent 0949fe5 commit 03243fc

2 files changed

Lines changed: 410 additions & 1 deletion

File tree

‎.github/workflows/ci.yml‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,7 @@ jobs:
294294
- 'scripts/check-lean-entry-closure.mjs'
295295
- 'scripts/check-sourcemap-no-sources-content.mjs'
296296
- 'scripts/ci/assert-turbo-remote-cache.mjs'
297+
- 'scripts/ci/prune-turbo-cache.mjs'
297298
- 'scripts/population-floor.mjs'
298299
# Build inputs of the vendored Console SPA — see the Console Pin Gate
299300
# job at the bottom of this file. `.objectui-sha` is a ROOT DOTFILE, so
@@ -1459,12 +1460,33 @@ jobs:
14591460
if-no-files-found: ignore
14601461
retention-days: 14
14611462

1463+
# ⛔ PRUNE THE SEED TO THIS RUN'S TASKS BEFORE SAVING IT (#25452). The
1464+
# restore step above takes the newest seed by prefix and turbo never
1465+
# evicts its local cache, so a seed saved whole was every seed before it
1466+
# plus this run: on push run 39982654692 all six shard seeds were
1467+
# 2.41-2.44 GB, against 0.048 GB for Build Core's full 76-task build --
1468+
# ~14.6 GB per merge into the 10 GB Actions cache pool, which evicted the
1469+
# seeds PR runs restore. The script keeps the entries the test step's run
1470+
# summary (`.turbo/runs/`) and the closure build's (moved to
1471+
# $RUNNER_TEMP/turbo-remote-cache by the assertion above) name, and drops
1472+
# the rest. No run summary from the test step -- a cancelled run, or an
1473+
# empty shard -- means no save: such a run has no test results to seed,
1474+
# and the previous seed stays the newest. Exits 0 on every run-time path,
1475+
# so it cannot withhold the attestation below.
1476+
- name: Prune the Turbo cache to this run's tasks (main only)
1477+
id: prune-turbo-cache
1478+
if: always() && github.event_name == 'push'
1479+
run: |
1480+
node scripts/ci/prune-turbo-cache.mjs --site 'Test Core (${{ matrix.shard }}/6)' \
1481+
--cache .turbo/cache --run-summaries .turbo/runs \
1482+
--extra-summaries "$RUNNER_TEMP/turbo-remote-cache"
1483+
14621484
# Seed the shared Turbo cache from main only (see the restore step
14631485
# above). always(): keep the seed fresh even when a test fails, matching
14641486
# the old actions/cache post-step behavior. Shard-scoped key so the two
14651487
# matrix jobs don't collide.
14661488
- name: Save Turbo cache (main only)
1467-
if: always() && github.event_name == 'push'
1489+
if: always() && github.event_name == 'push' && steps.prune-turbo-cache.outputs.save == 'true'
14681490
uses: actions/cache/save@v6
14691491
with:
14701492
path: .turbo/cache
@@ -2619,6 +2641,12 @@ jobs:
26192641
- name: assert-turbo-remote-cache self-test
26202642
run: node scripts/ci/assert-turbo-remote-cache.mjs --self-test
26212643

2644+
# Test Core's seed prune (the step before its "Save Turbo cache") runs
2645+
# on main pushes only, six times; its self-test runs here, once, on
2646+
# every event, so a PR that breaks it reds before the push that uses it.
2647+
- name: prune-turbo-cache self-test
2648+
run: node scripts/ci/prune-turbo-cache.mjs --self-test
2649+
26222650
# ── Turbo remote cache (#21186) ─────────────────────────────────────
26232651
# Vercel's managed Turborepo remote cache. objectstack-ai/cloud's
26242652
# test.yml builds this repo at its pinned SHA with the same command and

0 commit comments

Comments
 (0)