Repository navigation
Commit b9ac35c
ci(test-leftovers): report what each test job leaves in the tree; retire Rerun Safety (#24211)
Fixes #23880
Clause-②: no
## 维护者速读
**改了什么**
- **新增(只报告,永不让任务失败):** CI 里每个跑包测试的任务,即 `Test Core` 的 6 个分片,以及 `main`
全量跑里的 `Dogfood Regression Gate` 3 个分片,在构建之后、测试之前给工作树拍一次快照(包括被
`.gitignore`
忽略的文件);测试结束后列出这次测试**新增**的每一个路径,按包目录分组,写成任务摘要里的一张表,每个包一条警告注解。脚本在任何情况下都以 0
退出,步骤上也没有 `continue-on-error`:它不是门禁,不会让任何任务变红。
- **退役:** `Rerun Safety`
夜间双跑(在同一个工作树里把整个测试套件跑两遍)整体删除:工作流文件、判定脚本、`package.json` 别名、lint
里的自测步骤、红灯卡片的登记(`CARD_CADENCES` 条目与 `scheduled-full-run-card.yml`
的监听名单),以及其它文件里对它的引用。
**为什么改**
裁决 乙(本卡评论
`6235054403`)。维护者先指出:「这个问题不是应该派卡解决吗?但测试运行时常会往工作目录里写东西:临时数据库、.objectstack/data/*.json、缓存、生成文件。写完不清理,下一次在同一个目录里跑,就会带着上次的残留开始。」席位列出甲、乙、丙三个选项后,维护者回复「同意
乙」。
双跑在它被读过的三十个夜里没有抓到一次残留污染,真实的失败全是负载下的超时;它测的是"后果"(第二遍变红),从不点名文件。而
`.objectstack` 在 `.gitignore` 里,`git status`
根本看不见这一类残留。新报告直接点名"哪个包留下了哪个文件",每个 PR 都跑。
**风险与代价(含回滚)**
- **看不见的:** 写到仓库目录**之外**的状态,例如
`$HOME`、系统临时目录(`/tmp`)、数据库服务器。快照前已经存在、又被测试改写的文件,以及被测试改动的已跟踪文件(这一类 `git
status` 本来就能看到),也不在报告里。
- **失去的:**
双跑能看到的"跨包污染的后果",即一个包的测试弄脏了另一个包读到的状态、第二遍才变红。新报告看原因(谁留下了文件),不看后果。
- **报告的盲区:** 从 turbo 缓存回放的测试任务并没有真正运行,也就不会留下文件。`main`
每小时全量跑里多数测试任务是回放,所以 CI 上的报告只覆盖真正执行了的包。下文的本地读数是强制全部执行后得到的。
- **成本:** 每个测试分片多两步,本地实测快照约 3 秒、报告约 2 秒。
- **回滚:** revert 这一个 PR,夜间双跑、它的判定脚本和红灯卡片全部恢复。
**席位意见**(`domain:devx` 2 号席位,2026-10-17T19:24Z 定稿)
- **同意合并。** 席位逐项核对过:
- 报告脚本在任何路径都以 0 退出,包括崩溃和未知参数;两个步骤都没有 `continue-on-error`。
- 快照放在构建之后、测试之前,报告带 `if: always()`。
- 退役后 grep 只剩 4 处过去式的历史说明,没有任何仍在生效的引用。
- CI 卡片和 Lint 卡片的标题、标记逐字节不变。
- **开发提出的问题:** `temporal-conformance` 不接报告,席位答 A。它只在全量跑里出现,测的套件 Test
Core 已经覆盖。
- **报告的盲区要知道:** turbo 从缓存回放的测试任务没有真正执行,所以 `main`
上的报告只覆盖改动过的包。完整读数以下文的本地强制全量为准。
- **今天唯一一处残留:**
`examples/app-showcase/.objectstack/data/showcase_external.db`,来自
dogfood 测试。落地后按裁决交给下一任席位开卡。
**你要做的**
- 这是 Tier H PR(改动了 `.github/workflows/**`),需要 `os-zhuang` 或 `hotlong`
审批。
- 落地后,席位读 `main` 上第一次 CI 的报告,按包在各自的 lane 开卡(本地读数见下)。
- 报告是否升级为阻断门禁,等 `main` 报告干净后由你决定;本 PR 不做这件事。
---
## What the report says today: the local full-suite listing
On `015e6eab7` (base `cb8756b00`; the later merge of `origin/main`
brought 13 commits, none touching a file in this diff). Commands: a full
build (`pnpm turbo run build --filter=!@objectstack/docs
--concurrency=2`, 76 of 76 tasks), then `report-test-leftovers.mjs
snapshot` (88275 paths), then every test and `test:repo` task forced to
execute (`pnpm turbo run test test:repo --only --force --continue
--concurrency=2`, under `os-verify-lock`, 2h48m), then `report`.
| package directory | paths added | path | who wrote it |
| :-- | --: | :-- | :-- |
| `examples/app-showcase` | 1 |
`examples/app-showcase/.objectstack/data/showcase_external.db` | mtime
16:54:26Z, while `@objectstack/dogfood#test` was the only task running.
The showcase's `showcase_external` datasource and `external-fixture.ts`
resolve `.objectstack/data/showcase_external.db` against the process
cwd. |
- Not counted: 82 turbo task logs and 76 vitest results caches.
- Rewritten instead of added: 0. Every one of the 88275 snapshot paths
was stat'ed after the run, and none had an mtime later than the
snapshot, so nothing was hiding as a rewrite.
- Measurement limits:
- `@objectstack/organizations#test` was killed by the container's memory
cgroup at 16:41 (`Memory cgroup out of memory: Killed process 26265
(node)`, exit 137), so its row is partial.
- `@objectstack/cli#test` ran to the end with 3 failed files out of 464
(`data-commands.absent-database.integration`,
`hook-timeout-override-refusal`, `boot-preparation-parity`), under a
load average of about 8 to 10 on 4 cores. This diff touches the CLI by
one comment line in another file, and the two unit files pass when
re-run alone at `076add1f6` (`2 passed`, `41 passed`); the integration
file was not re-run.
- 80 of 82 tasks passed.
- In CI, the report groups by where a path lands, not by which test
wrote it. In a dogfood shard the writer is the dogfood suite even though
the row reads `examples/app-showcase`.
## What this PR does
**Adds a report-only leftover list to every job in `ci.yml` that runs
package tests**, and **retires Rerun Safety completely**, per ruling 乙
(comment `6235054403`). No product code changes, and no test behaviour
changes.
### `scripts/ci/report-test-leftovers.mjs` (new)
- `snapshot --out FILE` lists every untracked path in the tree, ignored
paths included, into a file under `$RUNNER_TEMP` (outside the tree; a
path inside the tree is refused, since the report would count it).
- `report --before FILE --label TEXT` lists the tree again and prints
every path the test run ADDED, grouped by the nearest workspace package
directory (`(root)` for anything outside one): a table in
`$GITHUB_STEP_SUMMARY`, the paths in the log (50 per package, then a
count), and one `::warning::` annotation per package with its count and
up to 5 example paths. GitHub shows 10 warnings per step, so past 10
packages the last annotation is a roll-up naming the rest; the summary
always has every row.
- **The listing is `git ls-files --others -z` with no exclude flag.**
Without `--exclude-standard` git applies no ignore rules, so ignored
files are listed one by one. `--exclude-standard` is blind to
`.objectstack/`; `--ignored --directory` collapses an ignored directory
to one entry, hiding a new file inside an existing `.objectstack/data/`;
`git status --porcelain --ignored --untracked-files=all` lists the same
paths but also refreshes the index and diffs tracked files. Measured on
this worktree: 85066 paths in 0.4 s after install, 88275 after a full
build.
- **It never fails a job.** `snapshot` and `report` exit 0 on findings,
on bad arguments, on a missing snapshot and on a crash; a failure to
measure is one `NOT MEASURED` warning. The ci.yml steps carry no
`continue-on-error`. The one failure the script cannot catch is its own
source not loading; its `--self-test` loads it on every pull request in
`Lint & Repo Gates`, so that breakage is red there first.
- **Not counted, a closed list, each counted in the summary:**
`turbo-state` (`.turbo/` at the root: turbo's cache and the
`--summarize` run summaries Test Core reads back and uploads);
`turbo-task-log` (`PKG/.turbo/turbo-TASK.log`); `vitest-results-cache`
(`PKG/node_modules/.vite/vitest/`, vitest 4.1.11's per-file duration and
pass/fail record, read only to order the next run); `own-build-output`
(the declared outputs of a task a test task depends on without `^`,
derived from `turbo.json` on every run: today `@objectstack/cli#build`,
`@objectstack/metadata#build` and
`@objectstack/spec#gen:migration-registry`).
### `ci.yml`
- `Test Core (k/6)`: `Snapshot the working tree before this shard's
tests` sits after both dependency-closure builds and before `Run this
shard's tests`; `Report what this shard's tests left in the working
tree` sits directly after the test step with `if: always()`, above the
attestation pair.
- `Dogfood Regression Gate (k/3)` (full battery only): the same pair
around `Boot example apps and exercise real user flows`.
- Not wired, by name: `dogfood-verify` runs the `objectstack verify`
CLI, not a package test suite; `build-core`, `build-docs` and
`console-pin` run no package tests; `temporal-conformance` is a fork,
below.
### Retirement
- Deleted `.github/workflows/rerun-safety-nightly.yml` and
`scripts/rerun-safety-verdict.mjs`; removed the
`check:rerun-safety-verdict` script from `package.json` and its
`Rerun-safety verdict table self-test` step from `lint.yml`.
- `scripts/ci/scheduled-full-run.mjs`: removed the `Rerun Safety` entry
of `CARD_CADENCES`, the `cardIdentity('Rerun Safety')` setup line and
the three self-test cases that pinned it (its title/marker, its nightly
body words, and the nightly-vs-hourly non-match). The `CI` and `Lint &
Type Check` cards are byte-identical (proof below).
- `scheduled-full-run-card.yml`: `Rerun Safety` removed from
`workflow_run.workflows`; the comment keeps the rule that every watched
name needs a `CARD_CADENCES` entry.
- `scripts/check-stall-guard-budget.mjs`: the population paragraph no
longer names `rerun-safety`; the census and self-test find sites from
the workflow files, so no self-test case pinned it.
- Re-pointed schedule comments in `board-snapshot.yml`,
`docs-deploy.yml`, `prerelease-pin-watch.yml` and
`test-nightly-tiers.yml`; past tense in `scripts/docs-audit/README.md`
and in the comment of
`packages/cli/test/published-entry-node-env-source-reroute.test.ts`
(comment only).
- `lint.yml` gains `Test-leftover report self-test` in `Lint · CI,
release & engine gates`, beside the other ci-script self-tests, which is
what `check:self-test-wired` requires for a script CI runs.
**Retirement grep** at `076add1f6`, `git grep -n -i
"rerun.safety\|rerun-safety"`: 4 hits, 0 live. All four are history
sentences in past tense, kept on purpose:
| hit | why it stays |
| :-- | :-- |
|
`packages/cli/test/published-entry-node-env-source-reroute.test.ts:112`
| where a timeout measurement was taken ("a since-retired nightly that
ran the whole suite twice") |
| `scripts/check-stall-guard-budget.mjs:193` | the dated census
"Measured when this landed"; dropping one term would make its totals
false |
| `scripts/ci/report-test-leftovers.mjs:20` | the new header naming what
it replaces ("retired by the same ruling") |
| `scripts/docs-audit/README.md:902` | the 2026-08-18 list of scheduled
workflows, now "on that date" and "(retired since)" |
#24158 is not addressed by this diff; the seat retires it at landing, as
the ruling says.
## The premise, measured per job (ruling item 5)
**`Test Core (k/6)`** separates test-created paths from build outputs:
- Before the snapshot: `Build this shard's dependency closure`
(`--filter=PKG^...` per package) and `Build the sliced package's
dependency closure` (`--filter=PKG` per slice).
- After it, the test step schedules `test test:repo` with their
`dependsOn`. The `^build` closure is already on disk, so a replay
rewrites existing paths and adds none. Three tasks without `^` can run
after the snapshot: `@objectstack/cli#build`,
`@objectstack/metadata#build` and
`@objectstack/spec#gen:migration-registry`, whose output exists from
`pnpm install`. Their declared outputs are the `own-build-output` rule,
derived from `turbo.json`.
- Turbo cache restore happens before the snapshot, and test tasks
declare `outputs: []`, so a replayed test restores only its log.
- `--summarize` writes `.turbo/runs/`, which the drift check, the
timings capture and the artifact upload read: `turbo-state`.
- Coverage: no `vitest.config.*` enables it (`git grep -n "enabled:
true" -- '*vitest.config.*'` gives 0 hits); coverage runs in
`coverage-nightly.yml`.
- vitest writes its results cache under
`PKG/node_modules/.vite/vitest/`: `vitest-results-cache`.
**`Dogfood Regression Gate (k/3)`**: `Build the dogfood package's
dependency closure` runs before the snapshot, and
`@objectstack/dogfood#test` depends on `^build` alone, so nothing is
built after it. There is no `--summarize`.
**Measured: the own builds.** On `015e6eab7`, after deleting
`packages/cli/dist` and `packages/metadata/dist`: snapshot, then `pnpm
turbo run build --filter=@objectstack/cli --filter=@objectstack/metadata
--only --force`, then report. 688 paths were added, all 688 under
`own-build-output`, 0 counted. The build was forced, so it wrote
everything it writes, not only what a replay restores.
**Not wired, reported as a fork: `temporal-conformance`** (full battery
only). It runs package tests, but it interleaves four builds with four
test runs: build `driver-sql...` then test it; build the non-SQL
backends then test them; build `metadata-protocol...` then test; build
`runtime...` then test. One snapshot after the first build would charge
the next three builds' `dist/` to the tests. The ruling names the per-PR
path and lists Test Core and the dogfood jobs, so this PR leaves it
unwired and names it here instead of shipping a partial check. The
options are in the report's open question.
## Proof
- **Self-test:** `node scripts/ci/report-test-leftovers.mjs --self-test`
at `076add1f6` prints `report-test-leftovers --self-test: OK (47 cases
over 6 batteries)`, exit 0. Batteries: the diff (the four dispatch
cases: a new ignored `.objectstack/data/x.json` under its package, build
outputs in both snapshots not reported, a deleted path not reported,
root paths as `(root)`); the fixed exclusions; the `turbo.json`
derivation; rendering; a real throwaway git tree; and every in-job
failure path exiting 0.
- **Ablation** on blob `30c07df010bc` with
`scripts/ablation-replace.mjs`: the listing gained `--exclude-standard`,
so it missed ignored paths (anchor 1 to 0, blob `30c07df010bc` to
`873a382177b1`). Result: `report-test-leftovers --self-test: 5 FAILED
(of 47 cases)`, including "a NEW ignored file inside an ignored
directory that already existed is listed". Restored: blob equals HEAD
and `git diff HEAD` is empty. The direction was red, as expected.
- **`scheduled-full-run`:** `--self-test` prints `scheduled-full-run:
self-test OK` and `--check-concurrency` passes for both workflows. A
one-off byte comparison of the module at `cb8756b00` against the new
one, for `CI` and `Lint & Type Check` (workflowName, cadence, title,
marker, labels, three body shapes), gave `byte-identical: 16 of 16`, and
the new module refuses `cardIdentity('Rerun Safety')` by name.
- **`check:stall-guard-budget`:** `OK (40 workflow file(s), 75 job(s),
832 step(s), 9 guard-wrapped step(s); ...)`, and its `--self-test`
passes with 61 assertions (62 before; the real-repository battery lost
one job group and sits at its floor of 13).
- **YAML:** every edited workflow parses with `yaml` (`0 error(s), 0
warning(s)`). In `ci.yml` the pair sits at steps 10 and 12 of 22 in
`test` and 8 and 10 of 16 in `dogfood`; the report step has `if:
always()` and no `continue-on-error`, and the last two steps are still
the attestation pair.
## Gates
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` at `076add1f6` (15 paths against merge base
`d9a749bae`) derived 105 commands. Each ran with its exit code captured
before any pipe, and all 105 exited 0. They include
`check:stall-guard-budget`, `check:self-test-wired`,
`check:self-test-workflow-commands`, `check:shard-attestation`,
`check:workflow-status-functions`, `check:workflow-step-name-quoting`,
`check:declared-population-live`, `check:pm-dispatch-gates` (411 s) and
the new self-test. `--ran` with exit codes printed `Run reconciliation —
105 derived, 105 run, 0 NOT-MEASURED, 0 UNRUN.`
- An earlier pass at `95219d01e` had one red.
`check:declared-population-live` read the snapshot schema id
`test-leftovers-snapshot/1` as a path declaration. `015e6eab7` fixes it
with the `no-path-population` marker (the self-test reads no tracked
file) and the id `test-leftovers-snapshot.v1`.
- Edited gate scripts, their own suites: `check:stall-guard-budget` runs
its `--self-test` first, and `scheduled-full-run.mjs --self-test` is in
the derived list. No `*.test.*` file names either script.
- `node scripts/pm/check-governed-merges.mjs --branch
claude/issue-23880-test-leftover-report`: GOVERNED, landing tier H
(`.github/workflows/**` ×8), exit 3. Size: 2333 changed lines (+1008 /
-1325) over 15 files.
- Changeset: none. Scripts, workflows and one test comment publish
nothing, so the seat applies `skip-changeset`.
- CI check-runs on this PR were not waited on; CI convergence is the
seat's.
## NOT MEASURED
- The report's real output in CI. Read it on the first `main` CI run
after landing: Test Core on the hourly full run, and Dogfood on the full
battery. This PR's own `Test Core` shards run the new steps for the
affected set, so their summaries are a first reading.
- On the hourly run most test tasks replay from the turbo cache and add
nothing, so a CI reading covers executed packages only. The forced local
listing above is the complete one.
## Acceptance notes
- `CADENCE_WORDS` in `scheduled-full-run.mjs` keeps its `nightly`
vocabulary, with no watched workflow mapped to it after this PR. The
dispatch scoped the removal to the `CARD_CADENCES` entry and the
self-test cases pinning it. A later nightly card can reuse it, or a
later edit can drop it. Carrier: none.
- The `scheduled-full-run` identity battery keeps its floor of 34 with
36 cases left. The stall-guard real-repository battery is now exactly at
its floor.
- `test-nightly-tiers.yml` and `coverage-nightly.yml` also run package
tests, outside `ci.yml`. The ruling scopes the report to `ci.yml`.
- GitHub shows 10 warning annotations per step, so past 10 packages the
report uses a roll-up annotation.
---
_Generated by [Claude
Code](https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent aa31569 commit b9ac35c
15 files changed
Lines changed: 1008 additions & 1325 deletions
File tree
- .github/workflows
- packages/cli/test
- scripts
- ci
- docs-audit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
152 | 152 | | |
153 | 153 | | |
154 | 154 | | |
155 | | - | |
156 | | - | |
157 | | - | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
158 | 158 | | |
159 | 159 | | |
160 | 160 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
828 | 828 | | |
829 | 829 | | |
830 | 830 | | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
831 | 849 | | |
832 | 850 | | |
833 | 851 | | |
| |||
1056 | 1074 | | |
1057 | 1075 | | |
1058 | 1076 | | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
1059 | 1098 | | |
1060 | 1099 | | |
1061 | 1100 | | |
| |||
2106 | 2145 | | |
2107 | 2146 | | |
2108 | 2147 | | |
| 2148 | + | |
| 2149 | + | |
| 2150 | + | |
| 2151 | + | |
| 2152 | + | |
| 2153 | + | |
| 2154 | + | |
2109 | 2155 | | |
2110 | 2156 | | |
2111 | 2157 | | |
| |||
2147 | 2193 | | |
2148 | 2194 | | |
2149 | 2195 | | |
| 2196 | + | |
| 2197 | + | |
| 2198 | + | |
| 2199 | + | |
| 2200 | + | |
| 2201 | + | |
| 2202 | + | |
| 2203 | + | |
| 2204 | + | |
| 2205 | + | |
2150 | 2206 | | |
2151 | 2207 | | |
2152 | 2208 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
55 | | - | |
56 | | - | |
| 55 | + | |
| 56 | + | |
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3728 | 3728 | | |
3729 | 3729 | | |
3730 | 3730 | | |
| 3731 | + | |
| 3732 | + | |
| 3733 | + | |
| 3734 | + | |
| 3735 | + | |
| 3736 | + | |
| 3737 | + | |
| 3738 | + | |
| 3739 | + | |
| 3740 | + | |
| 3741 | + | |
| 3742 | + | |
| 3743 | + | |
| 3744 | + | |
| 3745 | + | |
| 3746 | + | |
| 3747 | + | |
3731 | 3748 | | |
3732 | 3749 | | |
3733 | 3750 | | |
| |||
5184 | 5201 | | |
5185 | 5202 | | |
5186 | 5203 | | |
5187 | | - | |
5188 | | - | |
5189 | | - | |
5190 | | - | |
5191 | | - | |
5192 | | - | |
5193 | | - | |
5194 | | - | |
5195 | | - | |
5196 | | - | |
5197 | | - | |
5198 | | - | |
5199 | | - | |
5200 | | - | |
5201 | | - | |
5202 | | - | |
5203 | | - | |
5204 | | - | |
5205 | | - | |
5206 | | - | |
5207 | | - | |
5208 | | - | |
5209 | 5204 | | |
5210 | 5205 | | |
5211 | 5206 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | | - | |
61 | | - | |
| 60 | + | |
| 61 | + | |
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| |||
0 commit comments