From 9d31264b96f7060e44960e65b536113e28dacce2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 17 Oct 2026 14:45:45 +0000 Subject: [PATCH 1/3] ci(test-leftovers): report what each test job leaves in the tree; retire Rerun Safety MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ruling 乙: a report-only leftover check on the per-PR test path replaces the nightly double run. - scripts/ci/report-test-leftovers.mjs: `snapshot` lists every untracked and ignored path after the build (`git ls-files --others`, no exclude flags), `report` lists what the test run added, grouped by package directory, as a job-summary table and one warning per package. Both modes exit 0 on every path. A closed exclusion list (turbo state, turbo task logs, vitest's results cache, and the declared outputs of a build the test step runs itself, derived from turbo.json) is counted in the summary. Its --self-test runs in Lint & Repo Gates. - ci.yml: the snapshot and report steps in Test Core (k/6) and in the dogfood shards, above the attestation pair, no continue-on-error. - Rerun Safety retired: the workflow, its verdict script, its package.json alias and lint.yml step, its CARD_CADENCES entry and the self-test cases pinning it, its watched-list entry, and every reference a grep finds. Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t Co-authored-by: Claude --- .github/workflows/board-snapshot.yml | 6 +- .github/workflows/ci.yml | 56 ++ .github/workflows/docs-deploy.yml | 4 +- .github/workflows/lint.yml | 39 +- .github/workflows/prerelease-pin-watch.yml | 4 +- .github/workflows/rerun-safety-nightly.yml | 298 ------ .github/workflows/scheduled-full-run-card.yml | 11 +- .github/workflows/test-nightly-tiers.yml | 4 +- package.json | 1 - ...shed-entry-node-env-source-reroute.test.ts | 4 +- scripts/check-stall-guard-budget.mjs | 11 +- scripts/ci/report-test-leftovers.mjs | 887 ++++++++++++++++ scripts/ci/scheduled-full-run.mjs | 25 - scripts/docs-audit/README.md | 6 +- scripts/rerun-safety-verdict.mjs | 952 ------------------ 15 files changed, 983 insertions(+), 1325 deletions(-) delete mode 100644 .github/workflows/rerun-safety-nightly.yml create mode 100644 scripts/ci/report-test-leftovers.mjs delete mode 100644 scripts/rerun-safety-verdict.mjs diff --git a/.github/workflows/board-snapshot.yml b/.github/workflows/board-snapshot.yml index 1b88c2fd7b3..81c0824df16 100644 --- a/.github/workflows/board-snapshot.yml +++ b/.github/workflows/board-snapshot.yml @@ -152,9 +152,9 @@ on: # platform-checklist watchdog at :51 — 44 minutes after this run starts — # plus CodeQL on Mondays. `4,10,16,22` was the obvious alternative and is # worse on the same measurement: it drops an 800-request run into the busiest - # scheduled hour on this board (04:00 rerun-safety, 04:19 release-coverage - # patrol, 04:23 required-set patrol, 04:37 branch reaper on Mondays, 04:41 - # create-smoke) and shares hour 16 with the required-set patrol. + # scheduled hour on this board (04:19 release-coverage patrol, 04:23 + # required-set patrol, 04:37 branch reaper on Mondays, 04:41 create-smoke) + # and shares hour 16 with the required-set patrol. # # The minute is off the top of the hour on purpose, as in every patrol here: # scheduled workflows queue behind everyone else's :00 cron. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c6f374e89b7..d4806e6bd25 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -828,6 +828,24 @@ jobs: pnpm turbo run build "--filter=$PKG" --concurrency=4 --log-order=stream done < "$RUNNER_TEMP/shard-packages.txt" + # ── Test leftovers, step 1 of 2: the snapshot (#23880, ruling 乙) ────── + # + # REPORT ONLY, never a gate; the rationale, the listing and the closed + # exclusion list are in the `scripts/ci/report-test-leftovers.mjs` header. + # Taken HERE, after both dependency-closure builds above and before the + # tests, so the report after the tests can tell what the test run added + # from what the builds wrote. A package's OWN build (a test task that + # turbo.json makes depend on `build` without `^`) still runs inside the + # test step; the report excludes its declared outputs, read from + # turbo.json on every run. The snapshot goes to $RUNNER_TEMP, outside the + # tree, or the report would count it. + # + # ⛔ No `if:` and no `continue-on-error`: the script exits 0 on every + # path, by construction (a failure prints one NOT MEASURED warning), so + # this step cannot fail the shard or withhold its attestation. + - name: Snapshot the working tree before this shard's tests + run: node scripts/ci/report-test-leftovers.mjs snapshot --out "$RUNNER_TEMP/test-leftovers/before.json" + # --concurrency=4: turbo's default (10) oversubscribes the 4-vCPU # hosted runner; matching the core count bounds peak memory and the # job is CPU-bound anyway. @@ -1056,6 +1074,27 @@ jobs: done exit $STATUS + # ── Test leftovers, step 2 of 2: the report (#23880, ruling 乙) ──────── + # + # Every path this shard's test run ADDED to the working tree since the + # snapshot, ignored paths included -- `.objectstack/` is gitignored, so + # `git status` cannot see that class -- grouped by package directory: a + # table in the job summary and one `::warning::` annotation per package. + # Placed directly after the test step so no later step's writes are + # charged to the tests, and `if: always()` so a red shard still reports. + # + # ⛔ REPORT ONLY by ruling, never a gate: the script exits 0 on findings, + # on a missing snapshot and on a crash, so the step carries no + # `continue-on-error` and cannot fail the shard. Making it block is the + # maintainer's word, not an edit here. It stays ABOVE the attestation + # pair at the bottom of the job (#6082), like every step after the tests. + - name: Report what this shard's tests left in the working tree + if: always() + run: | + node scripts/ci/report-test-leftovers.mjs report \ + --before "$RUNNER_TEMP/test-leftovers/before.json" \ + --label "Test Core (${{ matrix.shard }}/6)" + # --summarize above costs nothing at runtime and writes # `.turbo/runs/.json`: one per-task record with the execution window # turbo actually observed. That file IS the balancing input for the next @@ -2106,6 +2145,13 @@ jobs: --report-dir "$RUNNER_TEMP/stall-reports" -- \ pnpm turbo run build --filter=@objectstack/dogfood --concurrency=4 --log-order=stream + # Test leftovers, step 1 of 2 (#23880, ruling 乙): the snapshot, after the + # closure build and before the tests. Report only; the same pair as Test + # Core's, whose comments carry the rationale. `dogfood#test` depends on + # `^build` alone, so the step above builds everything the tests need. + - name: Snapshot the working tree before the dogfood tests + run: node scripts/ci/report-test-leftovers.mjs snapshot --out "$RUNNER_TEMP/test-leftovers/before.json" + # Boots real example apps in-process (in-memory SQLite) and exercises them # through the real HTTP + service stack — catches runtime regressions that # build / unit tests / spec-liveness pass over (e.g. the #2018 tz-bucketing @@ -2147,6 +2193,16 @@ jobs: --report-dir "$RUNNER_TEMP/stall-reports" -- \ pnpm turbo run test --filter=@objectstack/dogfood --log-order=stream + # Test leftovers, step 2 of 2 (#23880, ruling 乙): what the dogfood test + # run added to the working tree. Report only -- exits 0 on every path, no + # `continue-on-error` -- and `if: always()`; see Test Core's pair. + - name: Report what the dogfood tests left in the working tree + if: always() + run: | + node scripts/ci/report-test-leftovers.mjs report \ + --before "$RUNNER_TEMP/test-leftovers/before.json" \ + --label "Dogfood Regression Gate (${{ matrix.shard }}/3)" + # Dogfood boots real apps in-process, so a native/OOM abort is likelier # here than in the unit suites — and a shard that dies silently looks like # a shard that had less work. diff --git a/.github/workflows/docs-deploy.yml b/.github/workflows/docs-deploy.yml index 3fe7d08ba17..d2c2c3fa5da 100644 --- a/.github/workflows/docs-deploy.yml +++ b/.github/workflows/docs-deploy.yml @@ -52,8 +52,8 @@ on: # The HOURS keep out of this repository's crowded scheduled windows, read # from `grep -rn 'cron:' .github/workflows/` on the day this landed: 01–05 # UTC carry most of the nightlies (stale, CodeQL, validate-deps, - # rerun-safety, coverage, the release-coverage and required-set patrols, - # the smokes and nightly tiers), 07 carries showcase-smoke, and the board + # coverage, the release-coverage and required-set patrols, the smokes and + # nightly tiers), 07 carries showcase-smoke, and the board # snapshot owns :07 of 02, 08, 14 and 20. Of the six six-hour sets, only # 0/6/12/18 and 1/7/13/19 have hours with nothing scheduled after :00; the # first is taken because its :00 neighbours are short (release.yml's diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 5b1aabebfb9..a6ac802cfea 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -3728,6 +3728,23 @@ jobs: continue-on-error: true run: node scripts/report-test-timings.mjs --self-test + # The test-leftover report's own instrument (#23880, ruling 乙). ci.yml's + # test jobs run `scripts/ci/report-test-leftovers.mjs snapshot` / `report`, + # which exit 0 on every path BY RULING -- report only, never a gate -- so + # nothing there can go red when the report stops seeing what it exists to + # see: an ignored path a test added. This self-test is where that breaks + # loudly: fixture snapshots for the diff, the closed exclusion list and the + # turbo.json own-build derivation, plus a real throwaway git tree that + # proves the listing includes IGNORED paths and that every in-job failure + # is a warning with exit 0. It also loads the script on every pull + # request, the one breakage its in-job modes cannot catch about + # themselves. Invoked as `node` per the GATE INVOCATION IDIOM note at the + # top of this file. Offline; about a second. + - name: Test-leftover report self-test + id: test-leftover-report-self-test + continue-on-error: true + run: node scripts/ci/report-test-leftovers.mjs --self-test + # The per-package suite-duration ratchet's own instrument (#16468). The # gate runs in ci.yml's `Test Core` aggregator, where a clean tree with # every suite under its ceiling reads OK whether or not the over-ceiling, @@ -5184,28 +5201,6 @@ jobs: continue-on-error: true run: pnpm check:stall-guard-budget - # The `Rerun Safety` verdict table (#18590) — self-test half only, and the - # asymmetry is the same one the step below explains: the measurement half - # takes two log paths and two exit codes that exist only inside that - # nightly job, so there is nothing here for a bare invocation to read. - # - # ⭐ It is wired HERE rather than left in the nightly deliberately. - # `rerun-safety-nightly.yml` declares `schedule` and `workflow_dispatch` - # and no PR-time event, so a gate family reachable only from it would be - # this tree's first SCHEDULED-ONLY family — which `check:pm-dispatch-gates` - # refuses by name, on the ground that a board sweep must not be routed into - # a per-PR gate list. - # - # WHAT REDDENS IT: its own script's parsing rules and verdict table. Those - # are matching rules, so their production verdict is identical before and - # after they break (see `check-self-test-wired.mjs`) — this step is the - # only instrument watching them. Reads nothing but its own fixtures; no - # build, no network, no token, sub-second. - - name: Rerun-safety verdict table self-test - id: rerun-safety-verdict-table-self-test - continue-on-error: true - run: pnpm check:rerun-safety-verdict - # Rationale: `scripts/measure-stall-guard-headroom.mjs` header (#13246, #13121). # ⛔ `--self-test` only: never append a bare run (it needs a token). - name: Stall-guard headroom self-test (self-test half only, never the measurement) diff --git a/.github/workflows/prerelease-pin-watch.yml b/.github/workflows/prerelease-pin-watch.yml index 38e27f7a4bb..361fe37111a 100644 --- a/.github/workflows/prerelease-pin-watch.yml +++ b/.github/workflows/prerelease-pin-watch.yml @@ -57,8 +57,8 @@ name: Prerelease Pin Watch on: schedule: - # 06:00 UTC nightly — after Rerun Safety (04:00) and Spec Coverage (05:00), - # before Showcase Smoke (07:00). Nightly, not weekly: see the note above. + # 06:00 UTC nightly — after Spec Coverage (05:00), before Showcase Smoke + # (07:00). Nightly, not weekly: see the note above. - cron: '0 6 * * *' workflow_dispatch: pull_request: diff --git a/.github/workflows/rerun-safety-nightly.yml b/.github/workflows/rerun-safety-nightly.yml deleted file mode 100644 index e8c49d88481..00000000000 --- a/.github/workflows/rerun-safety-nightly.yml +++ /dev/null @@ -1,298 +0,0 @@ -# Rerun safety, nightly. -# -# ## The blind spot this closes -# -# Every other job in this repo runs on a FRESH CLONE. That makes CI structurally -# incapable of seeing a whole class of defect: a suite that pollutes its own -# working tree and therefore passes exactly once. CI always runs pass #1, so it -# is always green; only a human or an agent running the suite twice in one -# checkout ever sees the failure. -# -# #4065 was exactly that. `InMemoryDriver` defaulted to `persistence: 'auto'`, -# which on Node means a file adapter — so a suite seeding two rows with fixed -# ids wrote them to `.objectstack/data/memory-driver.json`, and the next run -# loaded them back and asserted four. Run N read 2N rows. It sat in the repo -# green through every CI run it ever had, and surfaced only because someone ran -# the full suite a second time in the same tree while doing unrelated work — at -# which point it looked like their own change had broken something. -# -# This job runs the full suite TWICE in one checkout. The first pass is allowed -# to fail loudly like any other; what this gate is really asserting is that the -# SECOND pass agrees with the first. -# -# ## Why pass 1 no longer ends the job (#18590) -# -# That last sentence was true of the design and false of the implementation. A -# red `pass 1` ENDED THE JOB, which skipped `pass 2` — so on every night the -# suite was red on main, the one thing this workflow exists to assert was not -# asserted. The job went red anyway, and its red is indistinguishable from the -# red it exists to report. -# -# It ran that way for a 17-night streak ending 2026-08-31: thirteen of those -# nights died in pass 1, under FOUR different failing packages and two different -# failure kinds, with the killer moving from `verify` to `cli` as each was -# fixed. Only four of the seventeen nights ever reached pass 2. "Keep pass 1 -# green" is not a reachable end state on a trunk that goes red like any other, -# so the measurement is no longer conditional on it: both passes always run, -# and `scripts/rerun-safety-verdict.mjs` compares their failing task sets and -# says, in one line, whether the rerun property was MEASURED and what it said. -# -# ⛔ Nothing here is weakened. A red pass 1 still fails this job. What changed -# is only that a broken trunk now costs the instrument its BASELINE instead of -# its READING. -# -# ## Why `--force` -# -# Turbo would replay cached results on the second pass and report green without -# executing anything, which is precisely the wrong answer here. `--force` -# bypasses the cache so the second pass genuinely re-runs. -# -# ## Why `--only` (#23880) -# -# `--force` applies to every task in the run, and `test` depends on `^build`, so -# each pass also re-executed the whole build closure -- 69 `build` tasks and -# `@objectstack/spec#gen:migration-registry` (`turbo run test --dry=json` at -# 98ba1bf1d2) -- that `Build packages` had just built. `--only` runs the 83 `test` -# tasks and nothing they depend on; all 69 of those builds are in `pnpm build`'s -# own plan, and the spec's `test` script regenerates its migration registry -# itself. It also stops pass 2 from rebuilding `dist/` over whatever pass 1 left -# there, which is a working-tree state this job exists to see, not to repair. -name: Rerun Safety - -on: - workflow_dispatch: - schedule: - - cron: '0 4 * * *' # 04:00 UTC nightly — an hour before Spec Coverage - -permissions: - contents: read - -jobs: - rerun-safety: - name: Full suite twice in one working tree - runs-on: ubuntu-latest - # Two full passes of everything, dogfood included. Generous, but bounded — - # an unbounded job is what let a hung Test Core sit for GitHub's 6h default - # while the PR read as "still running" rather than broken. Since #4314 the - # primary stall detector is run-with-stall-guard on each pass (labeled red - # after 15 min of frozen output); this is the pure backstop. - # - # ⚠️ Raised 120 -> 170 (#18590). 120 had stopped being a backstop and become - # a second way for this instrument to go dark: the last four GREEN two-pass - # runs on main took 74, 89, 103 and **115** minutes, so a healthy run was - # within five minutes of being killed by its own bound — and #18590 makes - # pass 2 run on the nights pass 1 is red too, which is exactly when a - # 115-minute run is most likely. ⛔ This is NOT a stall budget and must never - # be read as one: run-with-stall-guard still declares a stall after 15 - # minutes of frozen output on EITHER pass, so a genuine hang is still caught - # in 15 minutes and never at this number. Raising it can therefore hide no - # hang; all it stops is a slow-but-live run being reported as a failure of - # the thing under test. - # - # ⚠️ 170 no longer holds two full passes, and raising it again is not the - # fix (#23880). Measured from the job's own step timestamps: pass 1 took 38 - # minutes on 2026-09-26 (the last green night), 66 on 10-02, and 97-113 on - # every night from 10-12 to 10-16, while the queue-tier suite's measured - # task time (scripts/test-shard-timings.json) grew from 124 to 194 minutes - # between the 09-28 and 10-13 refreshes, the CLI's alone from 733 to 2206 - # seconds. Not a hang: on 10-16 pass 2 was still printing when the job - # timeout cancelled it 67 minutes in. A night finished at all only because - # pass 2 stopped at its first failure; one that did not fail was CANCELLED, - # with no verdict and a conclusion the red-card filer does not read. - # - # So each pass now runs under the time the job has LEFT (JOB_TIMEOUT_MINUTES - # below, which must equal this number, minus PASS_RESERVE_MINUTES for the - # verdict), and a pass that runs out is stopped by `timeout` instead: the - # verdict step still runs, reads the pass's 124/137 exit (or a log with no - # turbo summary) as INCOMPLETE -- NOT MEASURED, and the job concludes - # `failure`, never a silent `cancelled`. That bound decides only WHO reports - # a slow night; how two full passes fit again is a capacity decision - # recorded on #23880. - timeout-minutes: 170 - env: - JOB_TIMEOUT_MINUTES: 170 - # The verdict and the steps after it take seconds; the reserve also covers - # `timeout --kill-after=2m`, the grace a pass gets between SIGTERM and SIGKILL. - PASS_RESERVE_MINUTES: 5 - steps: - - name: Record the job's deadline - run: echo "JOB_DEADLINE_EPOCH=$(( $(date +%s) + JOB_TIMEOUT_MINUTES * 60 ))" >> "$GITHUB_ENV" - - - name: Checkout repository - uses: actions/checkout@v7 - - - name: Setup Node.js - uses: actions/setup-node@v7 - with: - node-version: '22' - - - name: Setup pnpm - uses: ./.github/actions/setup-pnpm - - - name: Get pnpm store directory - shell: bash - run: echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV - - # Restore-only: scheduled runs read main's store cache; the per-push - # workflows own saving it. - - name: Restore pnpm cache - uses: actions/cache/restore@v6 - with: - path: ${{ env.STORE_PATH }} - key: ${{ runner.os }}-pnpm-store-v3-${{ hashFiles('**/pnpm-lock.yaml') }} - restore-keys: | - ${{ runner.os }}-pnpm-store-v3- - - - name: Install dependencies - run: pnpm install --frozen-lockfile - - - name: Build packages - run: pnpm build - - # ⭐ Pass 1 RECORDS its exit code instead of ending the job with it - # (#18590), and that change is what makes this workflow an instrument - # again rather than a light that has been on for seventeen nights. - # - # It used to be spelled "deliberately NOT `continue-on-error`", on the - # reasoning that a red pass 1 means the suite is broken on main and that - # is worth failing on. The reasoning still holds and the job still fails - # — see the verdict step, which owns the exit status now. What did NOT - # hold is the consequence nobody wrote down: a step that ends the job also - # skips `pass 2`, so on every night pass 1 was red the ONE measurement - # this workflow exists to take was not taken at all. The job went red - # either way, and that red is indistinguishable from the red it exists to - # report — "rerun safety is failing" and "rerun safety was never measured" - # printed the same colour. - # - # Measured over the 17-night red streak that ended 2026-08-31 (#18590): - # thirteen nights died here in pass 1, and the killer MOVED — six nights - # `@objectstack/verify#test` (an AssertionError, since fixed on main), - # then four nights `@objectstack/cli#test` (5000ms vitest timeouts), with - # `plugin-auth` and earlier `cli` nights in between. The remaining four - # nights reached pass 2 and are the only rerun readings the streak - # contains. So "keep pass 1 green" was never a reachable end state on a - # trunk that is red from time to time like any other — and every night it - # was not reached cost the reading. - # - # The stall guard (#4250, #4314) is unchanged: it bounds a frozen-output - # hang at 15 min instead of letting a nightly nobody watches sit to the - # job timeout; on a stall it exits 75 and its banner names the last output - # line. --log-order=stream is mandatory under the guard (see the Test Core - # comment in ci.yml): grouped log order flushes a task's output only when - # the task ends, so a solo tail task longer than the budget reads as - # frozen while healthy. The guard refuses a turbo run without it. - # - # The time bound (#23880) sits INSIDE the guard, as the command it wraps: - # the guard runs that command in its own process group, so `timeout` is - # that group's leader and its SIGTERM reaches turbo, which stops its tasks. - # Measured locally at 98ba1bf1d2 (guard -> timeout 45s -> turbo over core, - # types and formula): exit 124 came back through the guard, and the log - # ended at "Shutting down Turborepo tasks..." / "Force killed Turborepo - # tasks: @objectstack/core#test" with no `Tasks:` summary. The verdict reads - # 124 (or 137, after --kill-after) as a pass that did not finish. Wrapped - # the other way round, `timeout` would kill only the guard and leave its - # detached group running into the next step. - - name: Test suite — pass 1 - run: | - # Bound vitest's INNER worker pool (#11958). A no-op on a runner with - # <= 5 cores (it only ever LOWERS vitest's own `cores - 1` default); - # it exists so a larger runner cannot multiply turbo's outer - # --concurrency by a host-sized inner pool. Empty on failure, which is - # vitest's own "use the default" signal. Needs turbo.json's - # globalPassThroughEnv entry or turbo strips it — see the script header. - export VITEST_MAX_WORKERS="$(node scripts/vitest-worker-cap.mjs)" - budget=$(( (JOB_DEADLINE_EPOCH - $(date +%s)) / 60 - PASS_RESERVE_MINUTES )) - echo "Pass 1 may run ${budget} minutes before the job's own timeout would end it." - status=0 - if [ "$budget" -lt 1 ]; then - : > "$RUNNER_TEMP/rerun-pass1.log" - status=124 - else - node scripts/run-with-stall-guard.mjs --log "$RUNNER_TEMP/rerun-pass1.log" --stall-minutes 15 -- \ - timeout --kill-after=2m "${budget}m" \ - pnpm turbo run test --only --concurrency=4 --force --log-order=stream || status=$? - fi - echo "PASS1_EXIT=$status" >> "$GITHUB_ENV" - if [ "$status" -eq 124 ] || [ "$status" -eq 137 ]; then - echo "::warning::Pass 1 ran out of the job's time (exit $status) before it finished." - echo "::warning::That is a capacity reading, not a suite reading; the verdict step" - echo "::warning::reports it as INCOMPLETE and will fail this job." - elif [ "$status" -ne 0 ]; then - echo "::warning::Pass 1 exited $status — the suite is red on main. Recorded as the" - echo "::warning::BASELINE and NOT failed here: pass 2 runs anyway so the rerun" - echo "::warning::property still gets measured tonight. The verdict step owns the" - echo "::warning::job's exit status and will still fail this job." - fi - - # Informational only. A stray `.objectstack/` is how the #4065 class shows - # up on disk, so printing what pass 1 left behind turns a pass-2 failure - # from "something is flaky" into a named file to go look at. Some entries - # here are legitimate (an example app's own project directory), which is - # why this reports rather than fails — pass 2 is the assertion. - - name: Report on-disk state left by pass 1 - if: always() - run: | - echo "Directories named .objectstack under the checkout:" - find . -path ./node_modules -prune -o -type d -name '.objectstack' -print || true - echo - echo "Files inside them:" - find . -path ./node_modules -prune -o -type d -name '.objectstack' -exec find {} -type f \; || true - - # ⭐ `if: always()` is the other half of #18590 — pass 2 is the assertion, - # so it runs whatever pass 1 did. A red pass 1 now costs this instrument - # its BASELINE, not its reading. - # - # This step no longer renders the verdict either. It cannot: with a red - # baseline allowed, "pass 2 failed" is no longer the same fact as - # "rerun-unsafe" — a task that was already failing in pass 1 and fails - # again in pass 2 is ordinary breakage on main, and printing the #4065 - # text over it would send whoever triages it hunting for state leaks that - # are not there. That comparison needs both passes' failing task sets, so - # it belongs one step down, in the verdict. - - name: Test suite — pass 2 (same working tree) - if: always() - run: | - # Bound vitest's INNER worker pool (#11958). A no-op on a runner with - # <= 5 cores (it only ever LOWERS vitest's own `cores - 1` default); - # it exists so a larger runner cannot multiply turbo's outer - # --concurrency by a host-sized inner pool. Empty on failure, which is - # vitest's own "use the default" signal. Needs turbo.json's - # globalPassThroughEnv entry or turbo strips it — see the script header. - export VITEST_MAX_WORKERS="$(node scripts/vitest-worker-cap.mjs)" - # Whatever pass 1 left of the job's time; see pass 1 for the bound. - budget=$(( (JOB_DEADLINE_EPOCH - $(date +%s)) / 60 - PASS_RESERVE_MINUTES )) - echo "Pass 2 may run ${budget} minutes before the job's own timeout would end it." - status=0 - if [ "$budget" -lt 1 ]; then - : > "$RUNNER_TEMP/rerun-pass2.log" - status=124 - else - node scripts/run-with-stall-guard.mjs --log "$RUNNER_TEMP/rerun-pass2.log" --stall-minutes 15 -- \ - timeout --kill-after=2m "${budget}m" \ - pnpm turbo run test --only --concurrency=4 --force --log-order=stream || status=$? - fi - echo "PASS2_EXIT=$status" >> "$GITHUB_ENV" - - # ⭐ THE VERDICT, and the job's exit status (#18590). - # - # It answers the question the exit code alone has never been able to - # express — was the rerun property MEASURED tonight — and only then what - # it measured. `scripts/rerun-safety-verdict.mjs` carries the table and a - # `--self-test`; its header is the authority on detail. - # - # ⛔ This step does not make the job green. A red pass 1 still fails it, - # exactly as before; what is new is that the failure now SAYS which of - # "broken on main" and "rerun-unsafe" it is, and a broken-on-main night no - # longer swallows the reading. - # ⛔ The self-test is NOT invoked here. It runs at PR time as - # `pnpm check:rerun-safety-verdict` in lint.yml, because this workflow - # declares only `schedule` and `workflow_dispatch`: a gate family reachable - # from here alone would be the tree's first SCHEDULED-ONLY family, which - # `check:pm-dispatch-gates` refuses by name. That step's comment carries - # the reasoning. - - name: Rerun-safety verdict - if: always() - run: | - node scripts/rerun-safety-verdict.mjs \ - --pass1-log "$RUNNER_TEMP/rerun-pass1.log" --pass1-exit "${PASS1_EXIT:-1}" \ - --pass2-log "$RUNNER_TEMP/rerun-pass2.log" --pass2-exit "${PASS2_EXIT:-1}" diff --git a/.github/workflows/scheduled-full-run-card.yml b/.github/workflows/scheduled-full-run-card.yml index 6f3461efbeb..51c73b4a651 100644 --- a/.github/workflows/scheduled-full-run-card.yml +++ b/.github/workflows/scheduled-full-run-card.yml @@ -73,13 +73,10 @@ on: # Byte-exact workflow NAMES (the `name:` at the top of each file), not # paths. Quoted because `Lint & Type Check` starts a YAML alias unquoted. # - # `Rerun Safety` (rerun-safety-nightly.yml, #23880) runs once a night, and - # its scheduled runs from 2026-09-17 to 10-16 concluded `success` once in - # thirty nights with no filer at all. It gets its own card by the same - # identity rule, in its own cadence's words: every name listed here must - # have an entry in `CARD_CADENCES` (scripts/ci/scheduled-full-run.mjs), - # which refuses a name it does not hold rather than default one. - workflows: ['CI', 'Lint & Type Check', 'Rerun Safety'] + # Every name listed here must have an entry in `CARD_CADENCES` + # (scripts/ci/scheduled-full-run.mjs), which refuses a name it does not + # hold rather than default one, so a card never prints the wrong cadence. + workflows: ['CI', 'Lint & Type Check'] types: [completed] permissions: {} diff --git a/.github/workflows/test-nightly-tiers.yml b/.github/workflows/test-nightly-tiers.yml index bca2a3cbca5..6d3fe90ea21 100644 --- a/.github/workflows/test-nightly-tiers.yml +++ b/.github/workflows/test-nightly-tiers.yml @@ -76,8 +76,8 @@ on: # Nightly. The minute is offset off the top of the hour ON PURPOSE (scheduled # workflows queue behind everyone else's `:00` cron) and off every sibling # patrol's minute, so two of them never contend for the same runner minute; - # 05:29 sits after `rerun-safety-nightly` (04:00) and `coverage-nightly` - # (05:00) have started and well before `showcase-smoke` (07:00). + # 05:29 sits after `coverage-nightly` (05:00) has started and well before + # `showcase-smoke` (07:00). # # ⚠️ Two readers, one fact. This value is restated as `PATROL_SCHEDULE` in # the `env:` block below, where the report step computes the deadline the diff --git a/package.json b/package.json index 51626e48ec6..6457158513a 100644 --- a/package.json +++ b/package.json @@ -172,7 +172,6 @@ "check:select-gate-families": "bash scripts/ci/select-gate-families.selftest.sh", "check:shard-attestation": "node scripts/check-shard-attestation.mjs --self-test && node scripts/check-shard-attestation.mjs", "check:required-contexts": "node scripts/check-required-contexts.mjs --self-test && node scripts/check-required-contexts.mjs", - "check:rerun-safety-verdict": "node scripts/rerun-safety-verdict.mjs --self-test", "check:cross-package-test-inputs": "node scripts/check-cross-package-test-inputs.mjs --self-test && node scripts/check-cross-package-test-inputs.mjs", "check:live-db-isolation": "node scripts/check-live-db-isolation.mjs --self-test && node scripts/check-live-db-isolation.mjs", "check:tier-file-adoption": "node scripts/check-tier-file-adoption.mjs --self-test && node scripts/check-tier-file-adoption.mjs", diff --git a/packages/cli/test/published-entry-node-env-source-reroute.test.ts b/packages/cli/test/published-entry-node-env-source-reroute.test.ts index 59112f9b4ff..72a9dcc663c 100644 --- a/packages/cli/test/published-entry-node-env-source-reroute.test.ts +++ b/packages/cli/test/published-entry-node-env-source-reroute.test.ts @@ -109,8 +109,8 @@ const NEUTRALISER = resolve(HERE, 'fixtures/published-entry-auto-transpile-neutr * vitest's 5000ms DEFAULT, which nobody here chose. Against costs of 2.5-3.6s * that is a margin thinner than the run-to-run variance on an idle machine, so * WHICH leg reddens is decided by how busy the box is and by nothing this file - * tests. Both halves of that were measured: in `Rerun Safety` (the workflow - * that deliberately runs the whole suite twice on one runner) the three + * tests. Both halves of that were measured: in `Rerun Safety` (a since-retired + * nightly that ran the whole suite twice on one runner) the three * non-neutralised legs timed out while both CONTROL legs passed; locally, on a * quiet box, the opposite happened — the neutralised CONTROL leg took 5710ms * and was the only one to fail. Same file, same commit, disjoint casualties. diff --git a/scripts/check-stall-guard-budget.mjs b/scripts/check-stall-guard-budget.mjs index ae55686316b..07fb3d71d30 100644 --- a/scripts/check-stall-guard-budget.mjs +++ b/scripts/check-stall-guard-budget.mjs @@ -143,12 +143,11 @@ * No value of `T` closes either one. Raising a job timeout buys headroom for * item 3; it does not extend the guard's reach. * - * The affected population is 7 guard-wrapped STEPS across 5 jobs, not 4 jobs: - * TWO of those jobs carry two guarded steps each on ONE job clock -- - * `temporal-conformance` in ci.yml, and `rerun-safety` in - * rerun-safety-nightly.yml, whose two sites are consecutive full passes of the - * suite -- so in each pair the second site's `p` includes the first one's - * entire runtime. + * The affected population is counted in guard-wrapped STEPS, not jobs: some + * jobs carry more than one guarded step on ONE job clock -- in ci.yml today + * `test`, `temporal-conformance` and `dogfood` (`--list` prints the current + * census) -- so within each such job a later site's `p` includes the earlier + * sites' entire runtime. * * Each site's `slack` is still computed independently, which is correct for the * criterion; on its own it also READ as more independent than the clock diff --git a/scripts/ci/report-test-leftovers.mjs b/scripts/ci/report-test-leftovers.mjs new file mode 100644 index 00000000000..8057d674cd1 --- /dev/null +++ b/scripts/ci/report-test-leftovers.mjs @@ -0,0 +1,887 @@ +#!/usr/bin/env node +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// +// report-test-leftovers -- list every path a job's test run ADDED to the +// working tree, ignored paths included. REPORT ONLY: it never fails a job. +// +// node scripts/ci/report-test-leftovers.mjs snapshot --out +// node scripts/ci/report-test-leftovers.mjs report --before [--label ] +// [--summary ] [--json ] +// node scripts/ci/report-test-leftovers.mjs --self-test +// +// ## Why it exists (#23880, the maintainer's ruling 乙, comment 6235054403) +// +// A test that writes state into the working tree -- a temporary database, a +// `.objectstack/data/*.json` file, a cache, a generated file -- and does not +// clean it up hands that state to the next run in the same checkout, which then +// starts from the residue instead of from the tree. Nothing on the per-PR path +// looked for it: `.objectstack` is in `.gitignore`, so `git status` cannot see +// the class at all. The nightly double run that used to stand in for this +// (`Rerun Safety`, retired by the same ruling) measured consequences -- a red +// second pass -- and named no file. +// +// So each job in ci.yml that runs package tests takes a snapshot of the tree's +// untracked and ignored paths after its build and before its tests, and a +// report after its tests lists what the test run added, grouped by package +// directory: a table in the job summary and one `::warning::` annotation per +// package. Existing leftovers become cards in each package's own lane; whether +// the report ever blocks anything is the maintainer's decision, not this file's. +// +// ## It never fails a job, by construction +// +// `snapshot` and `report` exit 0 on every path: on findings, on bad arguments, +// on a missing snapshot, on a crash. A failure to measure prints one +// `::warning::` saying the report is NOT MEASURED for this job, and exits 0. +// The workflow steps carry no `continue-on-error` -- the ruling asked for a +// script that cannot fail, not a failure that is discarded. The one thing this +// file cannot catch is its own source failing to load; `--self-test` loads it +// on every pull request in `Lint & Repo Gates`, so that breakage reddens there +// before it can reach a test job. `--self-test` is the only mode that exits 1. +// +// ## The listing: `git ls-files --others -z`, with NO exclude flags +// +// Without `--exclude-standard` git applies no ignore rules, so `--others` lists +// every untracked file, ignored ones included, file by file. That is the whole +// point: `.objectstack/` is ignored. The alternatives were measured against it: +// +// * `git ls-files --others --exclude-standard` lists only NON-ignored paths, +// so it is blind to exactly the class this exists for. +// * `--ignored --directory` collapses an ignored directory to one entry, so a +// new file inside an ignored directory that already existed (a second +// `.objectstack/data/*.json`) is invisible. +// * `git status --porcelain --ignored --untracked-files=all` lists the same +// paths but also refreshes the index and diffs every tracked file, work the +// listing does not need; `ls-files` is the plumbing for exactly this list. +// +// Cost, measured on a fresh worktree of this repository after `pnpm install`: +// 85066 paths in 0.4 s, 83769 of them under `node_modules/.pnpm`. Git does not +// follow symlinks, so each package's `node_modules/` link is one entry. +// +// ## What is NOT counted -- a closed list, each rule with its reason +// +// Every rule below is counted in the summary, so nothing is dropped silently. +// +// turbo-state `.turbo/` at the repository root: turbo's local cache, +// and the `--summarize` run summaries the Test Core job +// reads back and uploads in its later steps. +// turbo-task-log `/.turbo/turbo-.log`: turbo writes every +// task's log there, the test task's included. +// vitest-results-cache `/node_modules/.vite/vitest/`: vitest records each +// test file's duration and pass/fail there, and the next +// run reads it only to order its files. +// own-build-output the declared `outputs` of a task a test task depends on +// WITHOUT `^` in turbo.json (today `@objectstack/cli#test` +// and `@objectstack/metadata#test` depend on their own +// `build`). The job's dependency-closure step builds the +// `^build` closure before the snapshot, but a package's +// OWN build runs inside the test step, after it. Derived +// from turbo.json on every run, never hand-listed. +// +// ## What it cannot see -- the boundary of an empty report +// +// * State written OUTSIDE the repository tree: `$HOME`, the OS temp directory, +// a database server. The snapshot is of the tree only. +// * A path that existed before the snapshot and that a test rewrote: only +// paths that are NEW count as added. +// * A tracked file a test modified (that one `git status` already shows). +// * Empty directories, which git does not list. +// * A package whose test task REPLAYED from the turbo cache ran no test, so it +// can add nothing; an empty row for it is not a clean bill. +// * A file written inside one of the excluded locations above. + +import { execFileSync, spawnSync } from 'node:child_process'; +import { appendFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, isAbsolute, join, relative, resolve } from 'node:path'; +import process from 'node:process'; +import { fileURLToPath } from 'node:url'; + +import { isEntrypoint } from '../invoked-as.mjs'; +import { workspacePackages } from '../workspace-enumerator.mjs'; + +const SELF = fileURLToPath(import.meta.url); + +export const SNAPSHOT_SCHEMA = 'test-leftovers-snapshot/1'; +export const ROOT_GROUP = '(root)'; +export const EXAMPLES_PER_PACKAGE = 5; + +// GitHub shows at most 10 warning annotations per step and drops the rest +// without saying so. Past that many packages the last slot becomes one roll-up +// annotation naming the rest, so no package is dropped silently; the job +// summary always carries every row. +export const ANNOTATIONS_PER_STEP = 10; + +// The turbo tasks the test steps of ci.yml run (`turbo run test test:repo`, +// `turbo run test`). Their non-`^` dependencies run after the snapshot. +export const TEST_TASKS = Object.freeze(['test', 'test:repo']); + +// The turbo configuration the own-build rule reads, a repo-relative literal. +export const TURBO_CONFIG = 'turbo.json'; + +// The fixed rules. Each is pure over a repo-relative POSIX path. The +// own-build rule needs the turbo-derived context and lives in `classify`. +export const FIXED_EXCLUSIONS = Object.freeze([ + Object.freeze({ + id: 'turbo-state', + label: "turbo's own state under `.turbo/` at the repository root", + test: (path) => path.startsWith('.turbo/'), + }), + Object.freeze({ + id: 'turbo-task-log', + label: 'turbo task logs (`.turbo/turbo-TASK.log` in a package)', + test: (path) => /(^|\/)\.turbo\/turbo-[^/]+\.log$/.test(path), + }), + Object.freeze({ + id: 'vitest-results-cache', + label: "vitest's results cache (`node_modules/.vite/vitest/` in a package)", + test: (path) => /(^|\/)node_modules\/\.vite\/vitest\//.test(path), + }), +]); + +export const OWN_BUILD_EXCLUSION = Object.freeze({ + id: 'own-build-output', + label: 'declared outputs of a build the test step runs itself (turbo.json `dependsOn` without `^`)', +}); + +// --------------------------------------------------------------------------- +// Listing and snapshots +// --------------------------------------------------------------------------- + +/** + * Every untracked path under `root`, ignored ones included, repo-relative and + * sorted. See the header for why there is no exclude flag. + */ +export function listUntracked(root) { + const out = execFileSync('git', ['-C', root, 'ls-files', '--others', '-z'], { + encoding: 'utf8', + maxBuffer: 1024 * 1024 * 1024, + stdio: ['ignore', 'pipe', 'pipe'], + }); + return out.split('\0').filter(Boolean).sort(); +} + +export function repoRoot(cwd = process.cwd()) { + return execFileSync('git', ['-C', cwd, 'rev-parse', '--show-toplevel'], { + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + }).trim(); +} + +function isInside(root, path) { + const rel = relative(root, path); + return rel === '' || (!rel.startsWith('..') && !isAbsolute(rel)); +} + +// --------------------------------------------------------------------------- +// turbo.json -> the build outputs a test step writes after the snapshot +// --------------------------------------------------------------------------- + +/** A turbo output glob, relative to its package directory, as a RegExp. */ +export function globToRegExp(glob) { + let re = ''; + for (let i = 0; i < glob.length; i += 1) { + const ch = glob[i]; + if (ch === '*') { + if (glob[i + 1] === '*') { + i += 1; + if (glob[i + 1] === '/') { + i += 1; + re += '(?:.*/)?'; + } else { + re += '.*'; + } + } else { + re += '[^/]*'; + } + } else if (ch === '?') { + re += '[^/]'; + } else { + re += ch.replace(/[.+^${}()|[\]\\]/g, '\\$&'); + } + } + return new RegExp(`^${re}$`); +} + +function taskConfig(tasks, pkgName, task) { + return tasks[`${pkgName}#${task}`] ?? tasks[task] ?? null; +} + +/** + * For every workspace package whose test task depends on a task WITHOUT `^`, + * the declared outputs of that task (followed transitively through further + * non-`^` dependencies), keyed by the directory they are written under. + * + * @param {object} turbo parsed turbo.json + * @param {Array<{dir: string, manifest: object}>} packages + * @returns {Map} + */ +export function deriveOwnBuild(turbo, packages) { + const tasks = turbo?.tasks ?? turbo?.pipeline ?? {}; + const dirOf = new Map(); + for (const { dir, manifest } of packages) { + if (typeof manifest?.name === 'string') dirOf.set(manifest.name, dir); + } + const result = new Map(); + const add = (pkgName, task) => { + const dir = dirOf.get(pkgName); + if (dir === undefined) return; + const cfg = taskConfig(tasks, pkgName, task); + const entry = result.get(dir) ?? { tasks: [], include: [], exclude: [] }; + const id = `${pkgName}#${task}`; + if (entry.tasks.includes(id)) return; + entry.tasks.push(id); + for (const glob of cfg?.outputs ?? []) { + if (typeof glob !== 'string' || glob === '') continue; + if (glob.startsWith('!')) entry.exclude.push(globToRegExp(glob.slice(1))); + else entry.include.push(globToRegExp(glob)); + } + result.set(dir, entry); + for (const dep of cfg?.dependsOn ?? []) follow(pkgName, dep); + }; + const follow = (pkgName, dep) => { + if (typeof dep !== 'string' || dep.startsWith('^') || dep.startsWith('$')) return; + const hash = dep.lastIndexOf('#'); + if (hash > 0) add(dep.slice(0, hash), dep.slice(hash + 1)); + else add(pkgName, dep); + }; + for (const { manifest } of packages) { + const name = manifest?.name; + if (typeof name !== 'string') continue; + for (const testTask of TEST_TASKS) { + if (typeof manifest?.scripts?.[testTask] !== 'string') continue; + for (const dep of taskConfig(tasks, name, testTask)?.dependsOn ?? []) follow(name, dep); + } + } + for (const [dir, entry] of result) if (entry.include.length === 0) result.delete(dir); + return result; +} + +// --------------------------------------------------------------------------- +// The report itself -- pure over two listings +// --------------------------------------------------------------------------- + +/** The nearest enclosing package directory, or ROOT_GROUP. */ +export function packageOf(path, packageDirs) { + let best = null; + for (const dir of packageDirs) { + if ((path === dir || path.startsWith(`${dir}/`)) && (best === null || dir.length > best.length)) best = dir; + } + return best ?? ROOT_GROUP; +} + +export function classify(path, { packageDirs, ownBuild }) { + for (const rule of FIXED_EXCLUSIONS) { + if (rule.test(path)) return { excluded: rule.id }; + } + const pkg = packageOf(path, packageDirs); + const built = pkg === ROOT_GROUP ? undefined : ownBuild?.get(pkg); + if (built) { + const rel = path.slice(pkg.length + 1); + if (built.include.some((re) => re.test(rel)) && !built.exclude.some((re) => re.test(rel))) { + return { excluded: OWN_BUILD_EXCLUSION.id }; + } + } + return { pkg }; +} + +/** + * @param {{before: string[], after: string[], packageDirs: string[], ownBuild?: Map}} input + */ +export function computeReport({ before, after, packageDirs, ownBuild = new Map() }) { + const had = new Set(before); + const now = new Set(after); + const added = [...now].filter((p) => !had.has(p)).sort(); + const removed = [...had].filter((p) => !now.has(p)).length; + const byPackage = new Map(); + const excluded = new Map(); + for (const path of added) { + const verdict = classify(path, { packageDirs, ownBuild }); + if (verdict.excluded) { + excluded.set(verdict.excluded, (excluded.get(verdict.excluded) ?? 0) + 1); + continue; + } + const list = byPackage.get(verdict.pkg) ?? []; + list.push(path); + byPackage.set(verdict.pkg, list); + } + const packages = [...byPackage] + .map(([dir, paths]) => ({ dir, count: paths.length, paths })) + .sort((a, b) => b.count - a.count || a.dir.localeCompare(b.dir)); + const counted = packages.reduce((n, p) => n + p.count, 0); + return { added: added.length, removed, counted, packages, excluded }; +} + +// --------------------------------------------------------------------------- +// Rendering +// --------------------------------------------------------------------------- + +/** Workflow-command data escaping: `%` first, or it would re-escape the others. */ +export function escapeData(text) { + return String(text).replace(/%/g, '%25').replace(/\r/g, '%0D').replace(/\n/g, '%0A'); +} + +/** Workflow-command property escaping (`title=`). */ +export function escapeProperty(text) { + return escapeData(text).replace(/:/g, '%3A').replace(/,/g, '%2C'); +} + +function code(text) { + return `\`${String(text).replace(/`/g, "'").replace(/\|/g, '\\|')}\``; +} + +function exclusionLines(report) { + const labels = new Map([...FIXED_EXCLUSIONS, OWN_BUILD_EXCLUSION].map((r) => [r.id, r.label])); + return [...report.excluded].map(([id, n]) => `${n} ${labels.get(id) ?? id}`); +} + +export function renderAnnotations(report, { label }) { + const one = (pkg) => { + const examples = pkg.paths.slice(0, EXAMPLES_PER_PACKAGE).join(', '); + const more = pkg.count > EXAMPLES_PER_PACKAGE ? `, and ${pkg.count - EXAMPLES_PER_PACKAGE} more` : ''; + return ( + `::warning title=${escapeProperty(`Test leftovers: ${pkg.dir}`)}::` + + escapeData( + `${pkg.dir}: ${pkg.count} path(s) added to the working tree by the test run of ${label}, e.g. ` + + `${examples}${more}. Report only -- this never fails the job; the job summary lists every row.`, + ) + ); + }; + if (report.packages.length <= ANNOTATIONS_PER_STEP) return report.packages.map(one); + const shown = report.packages.slice(0, ANNOTATIONS_PER_STEP - 1); + const rest = report.packages.slice(ANNOTATIONS_PER_STEP - 1); + const restPaths = rest.reduce((n, p) => n + p.count, 0); + return [ + ...shown.map(one), + `::warning title=${escapeProperty('Test leftovers: more packages')}::` + + escapeData( + `${rest.length} more package(s) left ${restPaths} path(s) in the working tree during the test run of ${label}: ` + + `${rest.map((p) => `${p.dir} (${p.count})`).join(', ')}. GitHub shows ${ANNOTATIONS_PER_STEP} warnings per ` + + 'step, so these share one; the job summary lists every row.', + ), + ]; +} + +export function renderSummary(report, { label, notes = [] }) { + const lines = [`### Test leftovers: ${label}`, '']; + lines.push( + 'Report only: this step never fails the job. Every path the test run ADDED to the working tree since the ' + + 'snapshot taken after the build, ignored paths included, grouped by package directory.', + '', + ); + if (report.packages.length === 0) { + lines.push(`No path was added by the test run (${report.added} new path(s) in all, none outside the rules below).`, ''); + } else { + lines.push(`| package | paths added | examples (up to ${EXAMPLES_PER_PACKAGE}) |`, '| :-- | --: | :-- |'); + for (const pkg of report.packages) { + lines.push(`| ${code(pkg.dir)} | ${pkg.count} | ${pkg.paths.slice(0, EXAMPLES_PER_PACKAGE).map(code).join(', ')} |`); + } + lines.push(''); + } + const excl = exclusionLines(report); + lines.push( + excl.length > 0 + ? `Not counted, each by a fixed rule in \`scripts/ci/report-test-leftovers.mjs\`: ${excl.join('; ')}.` + : 'Not counted: nothing (no new path fell under a fixed rule).', + ); + for (const note of notes) lines.push('', note); + lines.push( + '', + 'Not seen at all: state written outside the repository tree (`$HOME`, the OS temp directory), a path that ' + + 'existed before the snapshot and was rewritten, and a tracked file a test modified.', + '', + ); + return lines.join('\n'); +} + +export function renderLog(report, { label }) { + const lines = []; + if (report.packages.length === 0) { + lines.push(`test-leftovers: ${label} -- no path added by the test run (report only).`); + } else { + lines.push( + `test-leftovers: ${label} -- ${report.packages.length} package(s) left ${report.counted} path(s) in the ` + + 'working tree (report only; this step never fails the job).', + ); + for (const pkg of report.packages) { + lines.push(` ${pkg.dir} ${pkg.count}`); + for (const path of pkg.paths) lines.push(` ${path}`); + } + } + const excl = exclusionLines(report); + if (excl.length > 0) lines.push(` not counted: ${excl.join('; ')}`); + return lines; +} + +// --------------------------------------------------------------------------- +// The two in-job modes -- neither can exit non-zero +// --------------------------------------------------------------------------- + +function notMeasured(mode, label, message) { + console.log( + `::warning title=${escapeProperty('Test leftovers: NOT MEASURED')}::` + + escapeData(`report-test-leftovers ${mode}${label ? ` (${label})` : ''}: ${message} -- the leftover list for this job is NOT MEASURED. Report only; the job is unaffected.`), + ); +} + +function parseArgs(argv) { + const opts = { _: [] }; + for (let i = 0; i < argv.length; i += 1) { + const arg = argv[i]; + if (arg.startsWith('--')) { + const key = arg.slice(2); + const next = argv[i + 1]; + if (next === undefined || next.startsWith('--')) opts[key] = true; + else { + opts[key] = next; + i += 1; + } + } else opts._.push(arg); + } + return opts; +} + +export function runSnapshot(opts) { + const out = typeof opts.out === 'string' ? resolve(opts.out) : null; + if (!out) throw new Error('snapshot needs --out FILE (outside the repository tree)'); + const root = typeof opts.root === 'string' ? resolve(opts.root) : repoRoot(); + if (isInside(root, out)) { + throw new Error(`--out ${out} is inside the repository tree, so the report would count the snapshot itself as a leftover`); + } + const paths = listUntracked(root); + mkdirSync(dirname(out), { recursive: true }); + writeFileSync(out, `${JSON.stringify({ schema: SNAPSHOT_SCHEMA, root, takenAt: new Date().toISOString(), paths })}\n`); + console.log(`test-leftovers: snapshot of ${paths.length} untracked or ignored path(s) under ${root} -> ${out}`); +} + +export function runReport(opts) { + const label = typeof opts.label === 'string' ? opts.label : 'this job'; + const before = typeof opts.before === 'string' ? resolve(opts.before) : null; + if (!before) throw new Error('report needs --before FILE, written by the snapshot step'); + const summaryTarget = typeof opts.summary === 'string' ? opts.summary : process.env.GITHUB_STEP_SUMMARY; + if (!existsSync(before)) { + // The snapshot step did not run (an earlier step failed) or could not + // write. Not a finding about the tests, so no annotation: one log line and + // one summary line. + const line = `test-leftovers: ${label} -- NOT MEASURED, no snapshot at ${before} (the snapshot step did not run or did not complete).`; + console.log(line); + if (summaryTarget) appendFileSync(summaryTarget, `### Test leftovers: ${label}\n\n${line}\n\n`); + return; + } + const snap = JSON.parse(readFileSync(before, 'utf8')); + if (snap?.schema !== SNAPSHOT_SCHEMA || !Array.isArray(snap.paths)) { + throw new Error(`${before} is not a ${SNAPSHOT_SCHEMA} document`); + } + const root = typeof opts.root === 'string' ? resolve(opts.root) : repoRoot(); + if (snap.root !== root) throw new Error(`the snapshot was taken under ${snap.root}, this report runs under ${root}`); + const after = listUntracked(root); + const packages = workspacePackages(root); + const notes = []; + let ownBuild = new Map(); + try { + ownBuild = deriveOwnBuild(JSON.parse(readFileSync(join(root, TURBO_CONFIG), 'utf8')), packages); + } catch (error) { + notes.push(`⚠ ${TURBO_CONFIG} could not be read (${error.message}), so the own-build rule was NOT applied: a build output below may be counted as a leftover.`); + } + const report = computeReport({ before: snap.paths, after, packageDirs: packages.map((p) => p.dir), ownBuild }); + for (const line of renderLog(report, { label })) console.log(line); + for (const line of renderAnnotations(report, { label })) console.log(line); + for (const note of notes) console.log(` ${note}`); + if (summaryTarget) appendFileSync(summaryTarget, renderSummary(report, { label, notes })); + if (typeof opts.json === 'string') { + writeFileSync( + resolve(opts.json), + `${JSON.stringify({ label, ...report, excluded: Object.fromEntries(report.excluded), ownBuild: Object.fromEntries([...ownBuild].map(([d, e]) => [d, e.tasks])) }, null, 2)}\n`, + ); + } +} + +function inJob(mode, opts) { + const label = typeof opts.label === 'string' ? opts.label : ''; + const onCrash = (error) => { + notMeasured(mode, label, error?.message ?? String(error)); + process.exit(0); + }; + process.on('uncaughtException', onCrash); + process.on('unhandledRejection', onCrash); + try { + if (mode === 'snapshot') runSnapshot(opts); + else runReport(opts); + } catch (error) { + notMeasured(mode, label, error?.message ?? String(error)); + } + process.exitCode = 0; +} + +// --------------------------------------------------------------------------- +// Self-test +// --------------------------------------------------------------------------- + +// Set only after the verdict line prints; the dispatch refuses when it is unset, +// so a `return` above the verdict cannot pass for a green run. +let selfTestReachedVerdict = false; + +// The battery roster, by NAME, each with its minimum case count. A floor, not +// an equality: adding cases is ordinary work. A battery below its floor means +// cases stopped running. +const SELF_TEST_BATTERIES = Object.freeze({ + '1. The diff: what counts as ADDED, and under which package': 7, + '2. The fixed exclusions: each one, and only its own paths': 8, + '3. turbo.json: the builds a test step runs after the snapshot': 8, + '4. Rendering: the table, the annotations, the escaping': 9, + '5. The real listing: ignored paths are seen (a real git tree)': 5, + '6. Never fails a job: the in-job modes exit 0 on every path': 9, +}); + +// Deleting a roster entry deletes its floor too, so the roster size is pinned. +const SELF_TEST_BATTERY_FLOOR = 6; + +const UNATTRIBUTED_BATTERY = '(no battery open)'; + +function selfTest() { + const failures = []; + const seen = new Map(); + let open = UNATTRIBUTED_BATTERY; + const battery = (name) => { + open = name; + }; + const assert = (label, condition, detail = '') => { + seen.set(open, (seen.get(open) ?? 0) + 1); + if (!condition) failures.push(`[${open}] ${label}${detail ? `\n | ${String(detail).split('\n').join('\n | ')}` : ''}`); + }; + const dirs = ['packages/runtime', 'packages/cli', 'packages/qa/dogfood', 'examples/app-crm']; + + // ── 1. The diff ───────────────────────────────────────────────────────── + battery('1. The diff: what counts as ADDED, and under which package'); + { + const before = ['packages/runtime/dist/index.js', 'packages/cli/dist/run.js', 'stale.tmp']; + const after = [ + 'packages/runtime/dist/index.js', + 'packages/cli/dist/run.js', + 'packages/runtime/.objectstack/data/x.json', + 'leftover-at-root.db', + 'examples/app-crm/.objectstack/data/sys_user.json', + 'packages/qa/dogfood/tmp/run-1/state.json', + ]; + const r = computeReport({ before, after, packageDirs: dirs }); + const rows = Object.fromEntries(r.packages.map((p) => [p.dir, p.paths])); + assert( + 'a new IGNORED .objectstack/data/x.json is reported under its own package', + JSON.stringify(rows['packages/runtime']) === JSON.stringify(['packages/runtime/.objectstack/data/x.json']), + JSON.stringify(rows), + ); + assert( + 'build outputs present in BOTH snapshots are not reported', + !JSON.stringify(rows).includes('/dist/'), + JSON.stringify(rows), + ); + assert('a path deleted by the run is not reported as added', !JSON.stringify(rows).includes('stale.tmp'), JSON.stringify(rows)); + assert('...but the deletion is counted as removed', r.removed === 1, String(r.removed)); + assert( + 'a path at the repository root is grouped as (root)', + JSON.stringify(rows[ROOT_GROUP]) === JSON.stringify(['leftover-at-root.db']), + JSON.stringify(rows), + ); + assert( + 'a nested package directory wins over a shorter prefix (packages/qa/dogfood, not the root)', + Array.isArray(rows['packages/qa/dogfood']) && packageOf('packages/qa/other/x', dirs) === ROOT_GROUP, + JSON.stringify(rows), + ); + assert('an unchanged tree reports nothing', computeReport({ before, after: before, packageDirs: dirs }).packages.length === 0); + } + + // ── 2. The fixed exclusions ───────────────────────────────────────────── + battery('2. The fixed exclusions: each one, and only its own paths'); + { + const excluded = (path) => classify(path, { packageDirs: dirs, ownBuild: new Map() }).excluded ?? null; + assert('.turbo/ at the root (cache and run summaries) is turbo state', excluded('.turbo/runs/2abc.json') === 'turbo-state'); + assert('.turbo/cache entries are turbo state', excluded('.turbo/cache/0f1e2d.tar.zst') === 'turbo-state'); + assert('a package task log is a turbo task log', excluded('packages/runtime/.turbo/turbo-test.log') === 'turbo-task-log'); + assert('...and so is the test:repo one', excluded('packages/runtime/.turbo/turbo-test:repo.log') === 'turbo-task-log'); + assert( + 'any other file under a package .turbo/ is COUNTED, not swallowed by the log rule', + excluded('packages/runtime/.turbo/state.json') === null, + ); + assert( + "vitest's results cache is excluded", + excluded('packages/runtime/node_modules/.vite/vitest/da39a3ee/results.json') === 'vitest-results-cache', + ); + assert( + 'anything else under node_modules/.vite is COUNTED', + excluded('packages/runtime/node_modules/.vite/deps/_metadata.json') === null, + ); + assert('a .objectstack/ path is never excluded', excluded('packages/runtime/.objectstack/data/x.json') === null); + } + + // ── 3. turbo.json -> own-build outputs ────────────────────────────────── + battery('3. turbo.json: the builds a test step runs after the snapshot'); + { + const turbo = { + tasks: { + build: { dependsOn: ['^build'], outputs: ['dist/**', '.next/**', '!.next/cache/**'] }, + test: { dependsOn: ['^build'], outputs: [] }, + 'test:repo': { dependsOn: ['^build'], outputs: [] }, + '@objectstack/cli#test': { dependsOn: ['build'], outputs: [] }, + '@objectstack/spec#gen:migration-registry': { dependsOn: [], outputs: ['src/migrations/registry.ts'] }, + '@objectstack/spec#test:repo': { dependsOn: ['^build', 'gen:migration-registry'], outputs: [] }, + }, + }; + const packages = [ + { dir: 'packages/cli', manifest: { name: '@objectstack/cli', scripts: { test: 'vitest run', build: 'tsc' } } }, + { dir: 'packages/runtime', manifest: { name: '@objectstack/runtime', scripts: { test: 'vitest run', build: 'tsup' } } }, + { dir: 'packages/spec', manifest: { name: '@objectstack/spec', scripts: { 'test:repo': 'vitest run' } } }, + { dir: 'packages/quiet', manifest: { name: '@objectstack/quiet', scripts: { build: 'tsup' } } }, + ]; + const own = deriveOwnBuild(turbo, packages); + assert('a test that depends on its own build yields that build', own.has('packages/cli'), JSON.stringify([...own.keys()])); + assert('a test that depends on ^build only yields nothing', !own.has('packages/runtime'), JSON.stringify([...own.keys()])); + assert('a test:repo dependency on a generator yields its outputs', own.has('packages/spec'), JSON.stringify([...own.keys()])); + assert('a package with no test script yields nothing', !own.has('packages/quiet')); + const ctx = { packageDirs: [...dirs, 'packages/spec'], ownBuild: own }; + assert('the own build output is excluded', classify('packages/cli/dist/commands/run.js', ctx).excluded === 'own-build-output'); + assert( + 'the SAME path shape in a package that does not build in the test step is COUNTED', + classify('packages/runtime/dist/index.js', ctx).pkg === 'packages/runtime', + ); + assert('a negated output glob is honoured', classify('packages/cli/.next/cache/x', ctx).pkg === 'packages/cli'); + assert( + 'a non-output path of the own-build package is COUNTED', + classify('packages/cli/.objectstack/data/x.json', ctx).pkg === 'packages/cli', + ); + } + + // ── 4. Rendering ──────────────────────────────────────────────────────── + battery('4. Rendering: the table, the annotations, the escaping'); + { + const many = Array.from({ length: 7 }, (_, i) => `packages/runtime/.objectstack/data/t${i}.json`); + const r = computeReport({ before: [], after: [...many, 'root.db'], packageDirs: dirs }); + const notes = renderAnnotations(r, { label: 'Test Core (3/6)' }); + assert('one annotation per package', notes.length === 2, notes.join('\n')); + assert( + 'an annotation names its package, its count and at most 5 examples', + notes[0].includes('packages/runtime: 7 path(s)') && + notes[0].includes('t4.json') && + !notes[0].includes('t5.json') && + notes[0].includes('and 2 more'), + notes[0], + ); + assert('an annotation starts with the warning command at column 0', notes.every((n) => n.startsWith('::warning title='))); + assert('the title property escapes its colon', notes[0].startsWith('::warning title=Test leftovers%3A packages/runtime::'), notes[0]); + assert('message data escapes % and newlines', escapeData('50% done\nnext') === '50%25 done%0Anext'); + const wide = computeReport({ + before: [], + after: Array.from({ length: 12 }, (_, i) => `packages/p${String(i).padStart(2, '0')}/.objectstack/x`), + packageDirs: Array.from({ length: 12 }, (_, i) => `packages/p${String(i).padStart(2, '0')}`), + }); + const capped = renderAnnotations(wide, { label: 'L' }); + assert( + `past ${ANNOTATIONS_PER_STEP} packages: ${ANNOTATIONS_PER_STEP - 1} annotations plus one roll-up naming the rest`, + capped.length === ANNOTATIONS_PER_STEP && capped.at(-1).includes('3 more package(s)') && capped.at(-1).includes('packages/p11 (1)'), + capped.join('\n'), + ); + assert('an empty report raises no annotation', renderAnnotations(computeReport({ before: [], after: [], packageDirs: dirs }), { label: 'L' }).length === 0); + const md = renderSummary(computeReport({ before: [], after: ['packages/runtime/a|b.json'], packageDirs: dirs }), { label: 'L' }); + assert('the summary table escapes a pipe in a path', md.includes('`packages/runtime/a\\|b.json`'), md); + const excl = renderSummary( + computeReport({ before: [], after: ['.turbo/runs/x.json', 'packages/runtime/.turbo/turbo-test.log'], packageDirs: dirs }), + { label: 'L' }, + ); + assert( + 'the summary COUNTS what it did not count, rule by rule', + excl.includes('1 turbo') && excl.includes('No path was added') && excl.includes('Not counted, each by a fixed rule'), + excl, + ); + } + + const scratch = mkdtempSync(join(tmpdir(), 'report-test-leftovers-')); + try { + const git = (cwd, ...args) => + execFileSync('git', ['-c', 'user.email=t@example.invalid', '-c', 'user.name=t', '-c', 'init.defaultBranch=main', ...args], { + cwd, + stdio: ['ignore', 'pipe', 'pipe'], + encoding: 'utf8', + }); + const write = (root, path, text = 'x') => { + mkdirSync(dirname(join(root, path)), { recursive: true }); + writeFileSync(join(root, path), text); + }; + const makeRepo = (name) => { + const root = join(scratch, name); + mkdirSync(root, { recursive: true }); + git(root, 'init', '-q'); + write(root, '.gitignore', 'node_modules\ndist/\n.objectstack\n.turbo\n'); + write(root, 'pnpm-workspace.yaml', 'packages:\n - packages/*\n'); + write(root, 'package.json', '{"name":"fixture-root","private":true}\n'); + write(root, 'packages/a/package.json', '{"name":"@fixture/a","scripts":{"test":"vitest run"}}\n'); + write(root, 'packages/a/src/index.ts', 'export {};\n'); + write(root, 'turbo.json', `${JSON.stringify({ tasks: { test: { dependsOn: ['^build'] } } })}\n`); + git(root, 'add', '-A'); + git(root, 'commit', '-q', '-m', 'fixture'); + write(root, 'packages/a/dist/index.js', 'built'); + write(root, 'packages/a/.objectstack/data/old.json', '{}'); + return realpathOf(root); + }; + + // ── 5. The real listing ─────────────────────────────────────────────── + battery('5. The real listing: ignored paths are seen (a real git tree)'); + { + const root = makeRepo('listing'); + const before = listUntracked(root); + assert('the listing includes an IGNORED file that existed before', before.includes('packages/a/.objectstack/data/old.json'), before.join('\n')); + write(root, 'packages/a/.objectstack/data/x.json', '{}'); + write(root, 'packages/a/scratch.txt', 'untracked, not ignored'); + rmSync(join(root, 'packages/a/dist/index.js')); + const after = listUntracked(root); + assert( + 'a NEW ignored file inside an ignored directory that already existed is listed', + after.includes('packages/a/.objectstack/data/x.json'), + after.join('\n'), + ); + const r = computeReport({ before, after, packageDirs: workspacePackages(root).map((p) => p.dir) }); + const row = r.packages.find((p) => p.dir === 'packages/a'); + assert( + 'the report over two real listings names both new paths under packages/a', + JSON.stringify(row?.paths) === JSON.stringify(['packages/a/.objectstack/data/x.json', 'packages/a/scratch.txt']), + JSON.stringify(r.packages), + ); + assert('the tracked files are never listed', !after.includes('packages/a/src/index.ts') && !after.includes('turbo.json'), after.join('\n')); + assert('the deleted build output is not reported', !JSON.stringify(r.packages).includes('dist/index.js'), JSON.stringify(r.packages)); + } + + // ── 6. Never fails a job ────────────────────────────────────────────── + battery('6. Never fails a job: the in-job modes exit 0 on every path'); + { + const root = makeRepo('cli'); + // GITHUB_STEP_SUMMARY is blanked so a child never writes into the summary + // of the job running this self-test, and git is fenced at the scratch dir + // so "not a repository" stays true wherever the temp dir lives. + const env = { ...process.env, GITHUB_STEP_SUMMARY: '', GIT_CEILING_DIRECTORIES: scratch }; + const runIn = (cwd, ...args) => spawnSync(process.execPath, [SELF, ...args], { cwd, encoding: 'utf8', env }); + const run = (...args) => runIn(root, ...args); + const snapFile = join(scratch, 'out', 'before.json'); + const summary = join(scratch, 'summary.md'); + writeFileSync(summary, ''); + const snap = run('snapshot', '--out', snapFile); + assert('snapshot exits 0 and writes its file', snap.status === 0 && existsSync(snapFile), snap.stdout + snap.stderr); + write(root, 'packages/a/.objectstack/data/new.json', '{}'); + write(root, 'packages/a/.turbo/turbo-test.log', 'log'); + const rep = run('report', '--before', snapFile, '--label', 'Fixture (1/1)', '--summary', summary); + assert('report WITH findings exits 0', rep.status === 0, rep.stdout + rep.stderr); + assert( + 'report prints one warning annotation for the package, at column 0', + rep.stdout.split('\n').filter((l) => l.startsWith('::warning ')).length === 1 && + rep.stdout.includes('packages/a: 1 path(s) added'), + rep.stdout, + ); + const md = readFileSync(summary, 'utf8'); + assert( + 'report appends the table to the summary file, with the task log counted as not counted', + md.includes('| `packages/a` | 1 |') && md.includes('1 turbo task logs'), + md, + ); + const missing = run('report', '--before', join(scratch, 'absent.json'), '--summary', summary); + assert( + 'report with no snapshot exits 0, says NOT MEASURED, and raises no annotation', + missing.status === 0 && missing.stdout.includes('NOT MEASURED') && !missing.stdout.includes('::warning'), + missing.stdout + missing.stderr, + ); + const garbage = join(scratch, 'garbage.json'); + writeFileSync(garbage, '{ not json'); + const crash = run('report', '--before', garbage); + assert( + 'a crash (an unparseable snapshot) is ONE NOT MEASURED warning and exit 0', + crash.status === 0 && crash.stdout.startsWith('::warning title=Test leftovers%3A NOT MEASURED::'), + crash.stdout + crash.stderr, + ); + const inside = run('snapshot', '--out', join(root, 'snap.json')); + assert( + 'a snapshot aimed INSIDE the tree is refused with a warning, exit 0, and no file', + inside.status === 0 && inside.stdout.includes('NOT MEASURED') && !existsSync(join(root, 'snap.json')), + inside.stdout + inside.stderr, + ); + const bare = run('report'); + assert('report with no arguments exits 0 with a warning', bare.status === 0 && bare.stdout.includes('NOT MEASURED'), bare.stdout + bare.stderr); + const notARepo = join(scratch, 'not-a-repo'); + mkdirSync(notARepo, { recursive: true }); + const outside = runIn(notARepo, 'report', '--before', snapFile); + assert( + 'report outside any git tree exits 0 with a warning', + outside.status === 0 && outside.stdout.includes('NOT MEASURED'), + outside.stdout + outside.stderr, + ); + } + } finally { + rmSync(scratch, { recursive: true, force: true }); + } + + // ── The floor: every declared battery ran, at or above its pin ─────────── + const declared = Object.keys(SELF_TEST_BATTERIES); + if (declared.length < SELF_TEST_BATTERY_FLOOR) { + failures.push(`floor: SELF_TEST_BATTERIES declares ${declared.length} batteries, below the pinned ${SELF_TEST_BATTERY_FLOOR}`); + } + for (const [name, count] of seen) { + if (!declared.includes(name)) failures.push(`floor: battery "${name}" registered ${count} case(s) but is not declared`); + } + for (const name of declared) { + const count = seen.get(name) ?? 0; + if (count < SELF_TEST_BATTERIES[name]) { + failures.push( + count === 0 + ? `floor: battery "${name}" DID NOT RUN (0 cases, ${SELF_TEST_BATTERIES[name]} pinned)` + : `floor: battery "${name}" registered ${count} case(s), below its floor of ${SELF_TEST_BATTERIES[name]}`, + ); + } + } + + const total = [...seen.values()].reduce((n, c) => n + c, 0); + if (failures.length > 0) { + for (const f of failures) console.error(` ✗ ${f}`); + console.error(`report-test-leftovers --self-test: ${failures.length} FAILED (of ${total} cases)`); + return 1; + } + console.log(`report-test-leftovers --self-test: OK (${total} cases over ${declared.length} batteries)`); + selfTestReachedVerdict = true; + return 0; +} + +function realpathOf(path) { + return execFileSync('git', ['-C', path, 'rev-parse', '--show-toplevel'], { encoding: 'utf8' }).trim(); +} + +// --------------------------------------------------------------------------- + +function usage() { + return [ + 'usage:', + ' node scripts/ci/report-test-leftovers.mjs snapshot --out FILE', + ' node scripts/ci/report-test-leftovers.mjs report --before FILE [--label TEXT] [--summary FILE] [--json FILE]', + ' node scripts/ci/report-test-leftovers.mjs --self-test', + ].join('\n'); +} + +function main(argv) { + if (argv.includes('--self-test')) { + const code = selfTest(); + if (!selfTestReachedVerdict && code === 0) { + console.error('report-test-leftovers: the self-test never reached its verdict -- refusing to report a pass.'); + return 1; + } + return code; + } + const opts = parseArgs(argv); + const mode = opts._[0]; + if (mode === 'snapshot' || mode === 'report') { + inJob(mode, opts); + return 0; + } + if (opts.help) { + console.log(usage()); + return 0; + } + // An unknown mode in a workflow step is still not allowed to fail the job. + notMeasured(String(mode ?? '(none)'), '', `unknown mode -- ${usage().replace(/\n/g, ' ')}`); + return 0; +} + +if (isEntrypoint(import.meta.url)) { + process.exitCode = main(process.argv.slice(2)); +} diff --git a/scripts/ci/scheduled-full-run.mjs b/scripts/ci/scheduled-full-run.mjs index c33af68bc2e..5b0cfda6410 100644 --- a/scripts/ci/scheduled-full-run.mjs +++ b/scripts/ci/scheduled-full-run.mjs @@ -229,7 +229,6 @@ const titlePrefix = (cadence) => `${cadence} full run: red on main`; export const CARD_CADENCES = Object.freeze({ CI: 'hourly', 'Lint & Type Check': 'hourly', - 'Rerun Safety': 'nightly', }); // The words each cadence puts in the body. The `hourly` entry is the text the @@ -958,7 +957,6 @@ async function selfTest() { // -- The cadence (#23880): each cadence, the back-compat pins for the card // already open under the hourly identity, and the refusal of a workflow // the closed map does not name. - const rerun = cardIdentity('Rerun Safety'); const OPEN_CI_TITLE = 'hourly full run: red on main (CI)'; const OPEN_CI_MARKER = 'os-hourly-full-run-ci'; const OPEN_CI_MARKER_LINE = @@ -997,24 +995,6 @@ async function selfTest() { throw new Error(`cadence: Lint & Type Check is not hourly ('${lint.title}')`); } }); - await check(() => { - if ( - rerun.cadence !== 'nightly' || - rerun.title !== 'nightly full run: red on main (Rerun Safety)' || - rerun.marker !== 'os-nightly-full-run-rerun-safety' - ) { - throw new Error(`cadence: Rerun Safety is not nightly ('${rerun.title}', '${rerun.marker}')`); - } - }); - await check(() => { - for (const b of bodiesOf(rerun)) { - if (/hourly|every hour|an hour ago/.test(b)) throw new Error(`cadence: a nightly card body says hourly:\n${b}`); - if (!b.includes('every night') || !b.includes('## What landed since the last green nightly run')) { - throw new Error(`cadence: a nightly card body lacks its own cadence words:\n${b}`); - } - } - if (!bodiesOf(rerun)[1].includes('green the night before')) throw new Error('cadence: the nightly flake reading is not nightly'); - }); await check(() => { for (const b of bodiesOf(ci)) { if (/nightly|every night|the night before/.test(b)) throw new Error(`cadence: an hourly card body says nightly:\n${b}`); @@ -1037,11 +1017,6 @@ async function selfTest() { throw new Error('cadence: renderBody defaulted the cadence of an identity that carries none'); } }); - await check(() => { - if (matchesCard(issue(1, { title: ci.title, body: ci.marker }), rerun) || matchesCard(issue(1, { title: rerun.title, body: rerun.marker }), ci)) { - throw new Error("cadence: the nightly and hourly identities matched each other's cards"); - } - }); // -- THE ACCEPTANCE SHAPE, driven end to end against a mutable board: one red // run files exactly one card, and the NEXT red run refreshes that card and diff --git a/scripts/docs-audit/README.md b/scripts/docs-audit/README.md index c5911531a04..c9a098359b2 100644 --- a/scripts/docs-audit/README.md +++ b/scripts/docs-audit/README.md @@ -897,10 +897,10 @@ monthly / per-release) catches drift the CI gate missed — it runs the re-deriving them: - **GitHub Actions** — no workflow runs this audit at all. Of the 31 workflows - registered on the repo, the 10 carrying a `schedule:` trigger (`codeql`, + registered on the repo on that date, the 10 carrying a `schedule:` trigger (`codeql`, `coverage-nightly`, `engine-split-metric`, `prerelease-pin-watch`, `publish-smoke`, - `rerun-safety-nightly`, `scaffold-e2e`, `showcase-smoke`, `stale`, `validate-deps`) - are all unrelated. Derive that list with `grep -rl '^\s*schedule:' .github/workflows/` + `rerun-safety-nightly` (retired since), `scaffold-e2e`, `showcase-smoke`, `stale`, + `validate-deps`) were all unrelated. Derive that list with `grep -rl '^\s*schedule:' .github/workflows/` rather than copying it: an unanchored grep for `schedule:` also matches `cut-rc.yml`, whose only hit is a comment saying it deliberately has no schedule. ⚠️ **Read the registered workflow list and its run history, not the YAML on disk.** diff --git a/scripts/rerun-safety-verdict.mjs b/scripts/rerun-safety-verdict.mjs deleted file mode 100644 index 824354c8be3..00000000000 --- a/scripts/rerun-safety-verdict.mjs +++ /dev/null @@ -1,952 +0,0 @@ -#!/usr/bin/env node -// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. -// -// rerun-safety-verdict -- decide what the two passes of `Rerun Safety` actually -// measured, and SAY whether the rerun property was measured at all. -// -// ## The blind spot this closes -// -// `rerun-safety-nightly.yml` runs the full suite twice in one working tree to -// catch a suite that pollutes its own tree and therefore passes exactly once -// (#4065). Pass 2 is the assertion; pass 1 is only the baseline it is compared -// against. -// -// Until this script existed, a red pass 1 ENDED THE JOB -- `pass 2` never ran, -// so on every such night the rerun property was not measured at all. The job -// still went red, and that red is indistinguishable from the red it exists to -// report: "rerun safety is failing" and "rerun safety is unmeasured" printed -// the same colour. Measured over the 17-night red streak that ended 2026-08-31: -// -// nights step that died failing task kind -// 4 pass 1 @objectstack/cli#test 5000ms vitest timeouts -// 6 pass 1 @objectstack/verify#test AssertionError -// 1 pass 1 @objectstack/plugin-auth#test -// 2 pass 1 @objectstack/cli#test -// 4 pass 2 @objectstack/core#test, @objectstack/objectql#test -// -// Read that table the way it has to be read: the pass-1 killer MOVED. It was -// `verify` for six nights, then `verify` was fixed on main and `cli` inherited -// the job. So "get pass 1 green and keep it green" is not a reachable end state -// on a trunk that is red from time to time like any other -- and every night it -// is not reached, the ONE measurement this workflow exists to take is skipped. -// -// The repair is to stop making the measurement conditional on the baseline -// being clean. Pass 2 now runs whatever pass 1 did, and this script compares -// the two, so a red trunk costs the instrument its baseline -- not its reading. -// -// ## What a pass's log says, and the two facts it used to blur (#23880) -// -// Turbo states two different facts about a task, in two different spellings: -// -// FAILED `Failed: @objectstack/dogfood#test` (the summary footer) and -// `@objectstack/dogfood#test: ERROR command (...) exited (1)` -// (the per-task error line). Turbo prints these only for a task -// that exited non-zero ON ITS OWN. -// INTERRUPTED `@objectstack/cli:test: ELIFECYCLE Test failed.` with no -// ERROR line and no footer entry. Without `--continue`, turbo ends -// the run at its first failed task and stops every task still -// running; pnpm then prints ELIFECYCLE for each of them in the very -// words a real failure uses. An interrupted task did not fail. It -// was not allowed to finish, so it is UNMEASURED in that pass. -// -// This script used to take the union of all three spellings as the failing -// set, on the theory that the footer under-reports. Read against the job logs -// of the nights #23880 covers, the "under-reporting" was that interruption -// every time -- the extra ELIFECYCLE lines land 0.1-0.5 s after the real -// failure's, and one of the interrupted suites logs `Received SIGINT`: -// -// night pass turbo's footer (FAILED) ELIFECYCLE only (INTERRUPTED) -// 10-02 1 @objectstack/dogfood @objectstack/cli -// 10-02 2 @objectstack/core driver-sql, spec, lint -// 10-13 2 @objectstack/runtime http-conformance, verify -// 10-14 1 @objectstack/dogfood @objectstack/cli -// 10-14 2 @objectstack/core spec, lint -// 10-15 2 @objectstack/runtime verify, cli, organizations -// -// So `@objectstack/cli#test`, printed as a pass-1 failure on every one of those -// nights, was the suite turbo stopped when dogfood failed; and the "different -// package set each night" on pass 2 was each night's first real failure plus -// whatever happened to be running beside it. -// -// Three more facts are read for the same reason -- so the verdict never claims -// more than the passes measured: -// -// FINISHED turbo's closing ` Tasks: N successful, M total`. A pass log -// without it did not finish -- stopped by its time budget, the -// job timeout or a crash -- and nothing it had not finished was -// measured. `M` counts the tasks turbo STARTED, so a pass that -// stopped at its first failure shows a smaller `M` than one that -// ran to the end (10-14: pass 1 started 143, pass 2 only 49). -// A pass the workflow's time bound stopped exits 124 or 137 -// (`timeout`). Measured once, locally: that log ended at -// `Force killed Turborepo tasks: ...` with no summary at all. -// A turbo that did print one on its way down could name the -// tasks the signal killed in its footer, so for such a pass the -// exit code decides, and only a task whose own vitest run -// reported a failed test or suite counts as failed. -// RAN a task that printed vitest's `Test Files` line finished its -// suite in that pass. A pass-1 failure that pass 2 never finished -// was NOT re-measured; it did not "recover". -// TIMEOUT-ONLY a failed task whose every failed test is vitest's -// `Error: Test timed out in Nms`. Every real failure on the nights -// above has this shape. A working-tree leak fails an assertion, not -// a clock, so a timeout-only task that fails only on pass 2 points -// at load before it points at a leak, and the verdict says how to -// tell the two apart. -// -// A leading ISO timestamp and ANSI colour codes are stripped from every line, so -// a log downloaded from the Actions API (which stamps each line, and keeps -// vitest's colours) reads the same as the stall guard's own tee. -// -// ## What it does NOT do -// -// It does not make the job green. A red pass 1 still fails the job: the -// workflow's own header calls that deliberate ("if pass 1 is red the suite is -// simply broken on main and that is worth failing on"), and nothing here -// relaxes it. What changes is only that the job now also reports whether the -// second pass agreed with the first, and how much of the suite either pass -// actually covered. -// -// ## Usage -// -// node scripts/rerun-safety-verdict.mjs \ -// --pass1-log --pass1-exit \ -// --pass2-log --pass2-exit -// -// Exit status: 0 only for RERUN-SAFE; 75 for a stall (EX_TEMPFAIL, the same -// code run-with-stall-guard uses and for the same reason -- rerun it); 1 for -// every verdict that is a real red, INCOMPLETE included. - -import { readFileSync } from 'node:fs'; - -import { isEntrypoint } from './invoked-as.mjs'; - -const STALL_EXIT_CODE = 75; // EX_TEMPFAIL -- matches run-with-stall-guard.mjs -// What `timeout` (coreutils) exits with when it stopped the command at the -// deadline, and what it exits with when the command then needed SIGKILL. -const BUDGET_EXIT_CODES = Object.freeze([124, 137]); - -// --------------------------------------------------------------------------- -// Reading one pass out of its log -// --------------------------------------------------------------------------- - -// The ESC-[...m colour sequences vitest prints, spelled with the escape -- never -// as the raw control byte (scripts/check-nul-bytes.mjs has the reason). -const ANSI_RE = /\u001b\[[0-9;]*[A-Za-z]/g; -const ISO_PREFIX_RE = /^\d{4}-\d{2}-\d{2}T[\d:.]+Z\s/; - -const FAILED_FOOTER_RE = /^\s*Failed:\s+(.+?)\s*$/; -const TASK_ERROR_RE = /^\s*(\S+?)#(\S+?):\s+ERROR\s+command\b/; -const TASKS_SUMMARY_RE = /^\s*Tasks:\s+(\d+)\s+successful,\s+(\d+)\s+total\b/; -// turbo's streaming prefix: `:: `. -const TASK_LINE_RE = /^(\S+?):([A-Za-z0-9_-]+(?::[A-Za-z0-9_-]+)*?):\s(.*)$/; - -const ELIFECYCLE_RE = /^\s*ELIFECYCLE\b/; -const VITEST_TEST_FILES_RE = /^\s*Test Files\s/; -// The rule characters around vitest's summary headers are U+23AF. -const VITEST_FAILED_TESTS_RE = /^\s*⎯+\s*Failed Tests (\d+)\s*⎯+\s*$/; -const VITEST_FAILED_SUITES_RE = /^\s*⎯+\s*Failed Suites (\d+)\s*⎯+\s*$/; -// Anchored at the start of the task's output on purpose: the github-actions -// reporter repeats the message inside a `::error file=...` annotation, and that -// copy must not count as a second timed-out test. -const VITEST_TIMEOUT_RE = /^\s*Error: Test timed out in \d+ms\b/; - -function cleanLines(logText) { - return String(logText ?? '') - .split(/\r?\n/) - .map((line) => line.replace(ANSI_RE, '').replace(ISO_PREFIX_RE, '')); -} - -/** - * Everything one pass's log states about its tasks. Every list is a sorted - * array of `pkg#task` ids so two passes compare and print stably. - * - * failing turbo reported the task as failed (footer or ERROR line) - * interrupted ELIFECYCLE only: turbo stopped it after another task failed - * ran the task's vitest run printed its `Test Files` line - * timeoutOnly failing tasks whose every failed test hit vitest's timeout - * finished turbo printed its closing `Tasks:` summary - * started that summary's `M total` (null when unfinished) - * successful that summary's `N successful` (null when unfinished) - */ -export function readPass(logText) { - const failing = new Set(); - const elifecycle = new Set(); - const ran = new Set(); - const counters = new Map(); - const counter = (id) => { - if (!counters.has(id)) counters.set(id, { failedTests: 0, failedSuites: 0, timeouts: 0 }); - return counters.get(id); - }; - let summary = null; - - for (const line of cleanLines(logText)) { - let m = FAILED_FOOTER_RE.exec(line); - if (m) { - // turbo separates several ids with commas and/or whitespace. - for (const id of m[1].split(/[,\s]+/)) { - const trimmed = id.trim(); - if (trimmed.includes('#')) failing.add(trimmed); - } - continue; - } - m = TASK_ERROR_RE.exec(line); - if (m) { - failing.add(`${m[1]}#${m[2]}`); - continue; - } - m = TASKS_SUMMARY_RE.exec(line); - if (m) { - summary = { successful: Number(m[1]), started: Number(m[2]) }; - continue; - } - m = TASK_LINE_RE.exec(line); - if (!m) continue; - const id = `${m[1]}#${m[2]}`; - const output = m[3]; - if (ELIFECYCLE_RE.test(output)) { - elifecycle.add(id); - continue; - } - if (VITEST_TEST_FILES_RE.test(output)) ran.add(id); - const tests = VITEST_FAILED_TESTS_RE.exec(output); - if (tests) counter(id).failedTests += Number(tests[1]); - const suites = VITEST_FAILED_SUITES_RE.exec(output); - if (suites) counter(id).failedSuites += Number(suites[1]); - if (VITEST_TIMEOUT_RE.test(output)) counter(id).timeouts += 1; - } - - const timeoutOnly = [...failing].filter((id) => { - const c = counters.get(id); - return Boolean(c) && c.failedTests > 0 && c.failedSuites === 0 && c.timeouts === c.failedTests; - }); - // Failing tasks whose own vitest run REPORTED a failed test or suite. When a - // signal stops the whole run, turbo reports every task it killed as failed; - // only these failed on their own. - const reported = [...failing].filter((id) => { - const c = counters.get(id); - return Boolean(c) && c.failedTests + c.failedSuites > 0; - }); - - return { - failing: [...failing].sort(), - interrupted: [...elifecycle].filter((id) => !failing.has(id)).sort(), - ran: [...ran].sort(), - timeoutOnly: timeoutOnly.sort(), - reported: reported.sort(), - finished: summary !== null, - started: summary?.started ?? null, - successful: summary?.successful ?? null, - }; -} - -/** The tasks turbo reported as FAILED in one pass's log (interruptions excluded). */ -export function failingTasks(logText) { - return readPass(logText).failing; -} - -// --------------------------------------------------------------------------- -// The verdict -// --------------------------------------------------------------------------- - -const without = (list, ...others) => list.filter((t) => !others.some((o) => o.includes(t))); - -// A red pass is PARTIAL unless its own summary accounts for every task it -// started AND it started as many as the other pass did. Without `--continue` -// turbo stops scheduling at the first failure, so the tasks it never started -// appear in no count of that pass at all -- only the other pass's `M total` -// can show them missing. -function stoppedShort(pass, maxStarted) { - if (!pass.finished || pass.exit === 0) return false; - if (pass.interrupted.length > 0) return true; - if (pass.started === null || pass.successful === null) return true; - return pass.successful + pass.failing.length < pass.started || pass.started < maxStarted; -} - -function timeoutAdvice(ids, timeoutOnly) { - return ids - .filter((id) => timeoutOnly.includes(id)) - .map((id) => { - const [pkg, task] = id.split('#'); - return ( - `${id}: every failed test hit vitest's test timeout -- load-shaped. A leak fails an ` + - `assertion, not a clock: run \`pnpm --filter ${pkg} run ${task}\` twice in one tree on ` + - 'its own; a leak fails the second run there, a load timeout does not.' - ); - }); -} - -function coverageNotes(pass1, pass2, maxStarted) { - const notes = []; - for (const [n, p] of [[1, pass1], [2, pass2]]) { - if (!stoppedShort(p, maxStarted)) continue; - notes.push( - p.started !== null && p.started < maxStarted - ? `Pass ${n} stopped at its first failure (turbo runs without --continue) after starting ` + - `${p.started} of the ${maxStarted} tasks the longer pass started; every task it did not ` + - 'finish is unmeasured in that pass.' - : `Pass ${n} stopped at its first failure (turbo runs without --continue); the tasks still ` + - 'running then were interrupted and are unmeasured in that pass.', - ); - } - return notes; -} - -/** - * The verdicts, and the one question each answers. - * - * `coverage` is the field this whole script exists for -- `full`, `partial` or - * `none` -- because "rerun safety is failing" and "rerun safety was never (or - * only partly) measured" are different facts the exit code alone cannot carry. - * `measured` is `coverage !== 'none'`. - * - * Each pass is `{ exit, ...readPass(log) }`. - */ -export function verdict({ pass1, pass2 }) { - const result = (code, coverage, exit, headline, detail) => ({ - code, - coverage, - measured: coverage !== 'none', - exit, - headline, - detail, - }); - - if (pass1.exit === STALL_EXIT_CODE || pass2.exit === STALL_EXIT_CODE) { - return result('STALL', 'none', STALL_EXIT_CODE, 'STALLED -- frozen output, killed by run-with-stall-guard', [ - 'This is NOT a rerun-safety verdict: a stall says nothing about', - 'working-tree pollution. Rerun the workflow; if it stalls again at the', - 'same file, note the occurrence on #4250.', - ]); - } - - // A pass stopped by its time budget did not finish even when turbo, on its - // way down, printed a summary: the exit code is the authority there. - const byBudget = (p) => BUDGET_EXIT_CODES.includes(p.exit); - const unfinished = [[1, pass1], [2, pass2]].filter(([, p]) => !p.finished || byBudget(p)); - if (unfinished.length > 0) { - const named = unfinished.map(([n, p]) => - byBudget(p) - ? `Pass ${n} (exit ${p.exit}) was stopped by its share of the job's time before it finished.` - : `Pass ${n} (exit ${p.exit}) printed no turbo summary: it was stopped before it finished (job timeout or crash).`, - ); - const found = [[1, pass1], [2, pass2]].flatMap(([n, p]) => { - const own = byBudget(p) ? p.reported : p.failing; - const killed = byBudget(p) ? without(p.failing, p.reported) : []; - const when = unfinished.some(([u]) => u === n) ? ' before it stopped' : ''; - return [ - ...(own.length > 0 ? [`Failed in pass ${n}${when}: ${own.join(', ')}`] : []), - ...(killed.length > 0 ? [`Stopped mid-run in pass ${n} by the time bound (unmeasured): ${killed.join(', ')}`] : []), - ]; - }); - return result('INCOMPLETE', 'none', 1, 'INCOMPLETE -- a pass did not finish, so the rerun property was NOT MEASURED', [ - ...named, - 'Running out of time is a capacity reading, not a suite reading: compare the', - 'two pass steps\' durations with the job\'s timeout before reading anything', - 'into the failures below.', - ...found, - ]); - } - - const maxStarted = Math.max(pass1.started ?? 0, pass2.started ?? 0); - const coverage = - stoppedShort(pass1, maxStarted) || stoppedShort(pass2, maxStarted) ? 'partial' : 'full'; - const notes = coverageNotes(pass1, pass2, maxStarted); - - const newlyFailing = without(pass2.failing, pass1.failing); - const recovered = pass1.failing.filter((t) => pass2.ran.includes(t) && !pass2.failing.includes(t)); - const notRemeasured = without(pass1.failing, pass2.failing, pass2.ran); - const interruptedNote = [[1, pass1], [2, pass2]] - .filter(([, p]) => p.interrupted.length > 0) - .map( - ([n, p]) => - `Interrupted by turbo in pass ${n} after another task failed (not failures, unmeasured): ${p.interrupted.join(', ')}`, - ); - const notRemeasuredNote = - notRemeasured.length > 0 - ? [`Failed on pass 1 and never finished by pass 2 (NOT re-measured, not recovered): ${notRemeasured.join(', ')}`] - : []; - const advice = timeoutAdvice(newlyFailing, pass2.timeoutOnly); - - if (pass1.exit === 0 && pass2.exit === 0) { - return result('RERUN_SAFE', coverage, 0, 'RERUN-SAFE -- the second pass agreed with the first', [ - 'Both passes of the full suite were green in one working tree.', - ]); - } - - if (pass1.exit === 0 && pass2.exit !== 0) { - return result('RERUN_UNSAFE', coverage, 1, 'RERUN-UNSAFE -- the suite passes once and fails on a second run', [ - 'Something under test writes state into the working tree and reads it', - 'back on the next run. Ordinary CI cannot see this -- every other job is', - 'a fresh clone, so every other job is always run #1.', - `Newly failing on pass 2: ${newlyFailing.join(', ') || '(none named in the log)'}`, - ...advice, - ...interruptedNote, - ...notes, - 'See the on-disk state reported above, and #4065 for the pattern.', - ]); - } - - if (pass1.exit !== 0 && pass2.exit === 0) { - return result('BASELINE_FLAKY', coverage, 1, 'SUITE BROKEN ON MAIN (pass 1 only) -- and it passed on rerun', [ - 'Pass 1 failed and pass 2 was green in the same tree, so this is NOT the', - '#4065 rerun-unsafe pattern -- it is order-dependence or flake, pointing', - 'the opposite way down the same axis.', - `Failed on pass 1, green on pass 2: ${pass1.failing.join(', ') || '(none named in the log)'}`, - ...interruptedNote, - ...notes, - ]); - } - - if (newlyFailing.length > 0) { - return result( - 'RERUN_UNSAFE_ON_BROKEN_BASELINE', - coverage, - 1, - 'RERUN-UNSAFE, on top of a suite that is already broken on main', - [ - 'Both passes failed, but pass 2 failed tasks that pass 1 did not. The', - 'extra ones are the rerun-safety CANDIDATES; the shared ones are ordinary', - 'breakage on main and belong to whoever owns them.', - `Newly failing on pass 2: ${newlyFailing.join(', ')}`, - ...advice, - `Failing in both passes: ${pass1.failing.filter((t) => pass2.failing.includes(t)).join(', ') || '(none)'}`, - ...notRemeasuredNote, - ...interruptedNote, - ...notes, - ], - ); - } - - const headline = - coverage === 'full' - ? 'SUITE BROKEN ON MAIN -- rerun property measured, and it is CLEAN' - : 'SUITE BROKEN ON MAIN -- no task failed only on the rerun, as far as pass 2 got'; - return result('BASELINE_BROKEN', coverage, 1, headline, [ - 'No task failed in pass 2 that had not already failed in pass 1. The red', - 'belongs to ordinary breakage on main, NOT to working-tree pollution -- do', - 'not go hunting for state leaks on the strength of this run.', - `Failing in both passes: ${pass1.failing.filter((t) => pass2.failing.includes(t)).join(', ') || '(none)'}`, - ...(recovered.length > 0 ? [`Failed on pass 1 only, green on pass 2: ${recovered.join(', ')}`] : []), - ...notRemeasuredNote, - ...interruptedNote, - ...notes, - ]); -} - -// --------------------------------------------------------------------------- -// Reporting -// --------------------------------------------------------------------------- - -const COVERAGE_WORDS = Object.freeze({ full: 'MEASURED', partial: 'PARTIALLY MEASURED', none: 'NOT MEASURED' }); - -function passLine(n, p) { - const failed = p.failing - .map((id) => (p.timeoutOnly.includes(id) ? `${id} (timeout-only)` : id)) - .join(', '); - const lines = [` pass ${n} exit ${p.exit}, failing tasks: ${failed || '(none)'}`]; - if (p.interrupted.length > 0) lines.push(` interrupted by turbo (unmeasured): ${p.interrupted.join(', ')}`); - lines.push( - p.finished - ? ` turbo: ${p.successful} successful of ${p.started} started` - : ' turbo: no closing summary -- this pass did not finish', - ); - if (BUDGET_EXIT_CODES.includes(p.exit)) lines.push(' stopped by its share of the job\'s time'); - return lines; -} - -export function renderVerdict(v, { pass1, pass2 }) { - const lines = []; - lines.push('='.repeat(72)); - lines.push(`Rerun Safety verdict: ${v.code}`); - lines.push('='.repeat(72)); - lines.push(v.headline); - lines.push(''); - for (const d of v.detail) lines.push(` ${d}`); - lines.push(''); - lines.push(` rerun property: ${COVERAGE_WORDS[v.coverage]}`); - lines.push(...passLine(1, pass1)); - lines.push(...passLine(2, pass2)); - lines.push('='.repeat(72)); - return lines.join('\n'); -} - -// --------------------------------------------------------------------------- -// The self-test's own battery roster and floor (#13489) -// --------------------------------------------------------------------------- -// -// `failed === 0` is not a success condition on its own -- "every case held" and -// "the cases never ran" print the same line. What is pinned here is the -// registered NAMES with a per-battery floor, so a battery that stops running -// names ITSELF in the refusal instead of vanishing into a smaller total. - -const SELF_TEST_BATTERIES = Object.freeze({ - 'parse: turbo Failed: footer': 1, - 'parse: turbo per-task ERROR line': 1, - 'parse: an ELIFECYCLE line alone is INTERRUPTED, never failing': 2, - 'parse: 2026-10-14 pass 1 -- dogfood failed, cli interrupted by fail-fast': 5, - 'parse: 2026-10-15 pass 2 -- runtime failed; verify, cli, organizations interrupted': 3, - 'parse: an API-downloaded log with ISO timestamps reads the same': 1, - 'parse: a green log names no failing task and is finished': 2, - 'parse: timeout-only needs EVERY failed test to be a timeout': 2, - 'parse: the ::error annotation copy is not a second timeout': 1, - 'parse: a log without turbo\'s closing summary did not finish': 2, - 'verdict: both green -> RERUN_SAFE, measured, exit 0': 1, - 'verdict: pass1 green + pass2 red -> RERUN_UNSAFE, exit 1': 1, - 'verdict: pass1 red + pass2 green -> BASELINE_FLAKY, measured': 1, - 'verdict: both red, same set -> BASELINE_BROKEN, measured, CLEAN': 1, - 'verdict: both red, pass2 worse -> RERUN_UNSAFE_ON_BROKEN_BASELINE': 1, - 'verdict: a stall in pass 1 -> STALL, NOT MEASURED, exit 75': 1, - 'verdict: a stall in pass 2 -> STALL, NOT MEASURED, exit 75': 1, - 'verdict: a pass that did not finish -> INCOMPLETE, NOT MEASURED, exit 1': 4, - 'verdict: every finished, non-stall verdict reports measured=true': 1, - 'verdict: 2026-10-14 end to end -- no interrupted task is newly failing, coverage PARTIAL': 4, - 'verdict: a pass-1 failure pass 2 never finished is not re-measured, never recovered': 1, - 'render: the verdict line names MEASURED, PARTIALLY MEASURED or NOT MEASURED': 1, - 'render: a timeout-only newly failing task carries the twice-on-its-own instruction': 1, -}); - -const SELF_TEST_BATTERY_FLOOR = 23; - -// Set by `selfTest()` only after its verdict is printed, and read at the -// dispatch: a `return` above that line prints nothing and still exits 0 -- a -// self-test that never finished, reported as one that passed. -let selfTestReachedVerdict = false; - -// Fixture spellings. The colour codes and the U+23AF rules are written as -// escapes, exactly as the job log carries them once decoded. -const ESC = '\u001b'; -const RULE = '⎯'.repeat(7); -const failedTestsHeader = (prefix, n) => - `${prefix} ${ESC}[31m${RULE}${ESC}[39m${ESC}[1m${ESC}[41m Failed Tests ${n} ${ESC}[49m${ESC}[22m${ESC}[31m${RULE}${ESC}[39m`; -const timedOut = (prefix) => `${prefix} ${ESC}[31m${ESC}[1mError${ESC}[22m: Test timed out in 5000ms.`; -const testFiles = (prefix, failed, passed) => - `${prefix} ${ESC}[2m Test Files ${ESC}[22m ${ESC}[1m${ESC}[31m${failed} failed${ESC}[39m${ESC}[22m${ESC}[2m | ${ESC}[22m${ESC}[1m${ESC}[32m${passed} passed${ESC}[39m`; -const elifecycle = (pkg) => `${pkg}:test: ELIFECYCLE Test failed. See above for more details.`; -const taskError = (pkg) => `${pkg}#test: ERROR command (/w/${pkg}) /x/pnpm run test exited (1)`; -const tasksSummary = (ok, total) => ` Tasks: ${ok} successful, ${total} total`; - -// The 2026-10-14 night (run 38509291145), cut to the lines this script reads. -const NIGHT_1014_PASS1 = [ - '@objectstack/dogfood:test: ✓ shared-showcase test/showcase-public-read-owd.dogfood.test.ts (3 tests) 1876ms', - failedTestsHeader('@objectstack/dogfood:test:', 1), - '@objectstack/dogfood:test: FAIL isolated test/activity-text-predicate.dogfood.test.ts > [#21154] ... > control: the unmasking reader filters and groups by every column of a pinned parent, as before', - timedOut('@objectstack/dogfood:test:'), - testFiles('@objectstack/dogfood:test:', 1, 256), - '@objectstack/dogfood:test: ::error file=/w/packages/qa/dogfood/test/activity-text-predicate.dogfood.test.ts,title=[isolated] test/activity-text-predicate.dogfood.test.ts::Error: Test timed out in 5000ms.', - elifecycle('@objectstack/dogfood'), - elifecycle('@objectstack/cli'), - '##[error]command (/w/packages/qa/dogfood) /x/pnpm run test exited (1)', - taskError('@objectstack/dogfood'), - '', - tasksSummary(141, 143), - 'Cached: 0 cached, 143 total', - ' Time: 1h38m20.709s ', - 'Failed: @objectstack/dogfood#test', -].join('\n'); - -const NIGHT_1014_PASS2 = [ - failedTestsHeader('@objectstack/core:test:', 1), - '@objectstack/core:test: FAIL local src/security/auth-gate.test.ts > auth-gate (ADR-0069 session gate) > BOUNDARY', - timedOut('@objectstack/core:test:'), - testFiles('@objectstack/core:test:', 1, 96), - elifecycle('@objectstack/core'), - '@objectstack/lint:test: ✓ src/validate-expressions.field-conditional-formatting.test.ts (41 tests) 489ms', - elifecycle('@objectstack/spec'), - elifecycle('@objectstack/lint'), - taskError('@objectstack/core'), - tasksSummary(24, 49), - 'Failed: @objectstack/core#test', -].join('\n'); - -// The 2026-10-15 night's pass 2 (run 38679312462), API-downloaded: stamped. -const NIGHT_1015_PASS2 = [ - failedTestsHeader('@objectstack/runtime:test:', 1), - timedOut('@objectstack/runtime:test:'), - testFiles('@objectstack/runtime:test:', 1, 382), - '@objectstack/verify:test: 2026-10-15T06:47:11.224Z INFO Received SIGINT - initiating graceful shutdown', - elifecycle('@objectstack/runtime'), - elifecycle('@objectstack/verify'), - elifecycle('@objectstack/cli'), - elifecycle('@objectstack/organizations'), - taskError('@objectstack/runtime'), - tasksSummary(132, 142), - 'Failed: @objectstack/runtime#test', -] - .map((line, i) => `2026-10-15T06:47:${String(10 + i).padStart(2, '0')}.0000000Z ${line}`) - .join('\n'); - -// A synthetic pass: `failing` tasks fail on their own, `ran` tasks finish -// green, and the summary accounts for every task unless `started` says more. -function logOf({ failing = [], interrupted = [], ran = [], started = 10, finished = true } = {}) { - const lines = []; - for (const id of [...ran, ...failing]) lines.push(testFiles(`${id.split('#')[0]}:test:`, failing.includes(id) ? 1 : 0, 3)); - for (const id of [...failing, ...interrupted]) lines.push(elifecycle(id.split('#')[0])); - for (const id of failing) lines.push(taskError(id.split('#')[0])); - if (finished) lines.push(tasksSummary(started - failing.length - interrupted.length, started)); - if (finished && failing.length > 0) lines.push(`Failed: ${failing.join(', ')}`); - return lines.join('\n'); -} -const passOf = (exit, log) => ({ exit, ...readPass(log) }); - -function selfTest() { - const registered = new Map(); - const registerCase = (label) => registered.set(label, (registered.get(label) ?? 0) + 1); - let failed = 0; - const check = (label, ok, note = '') => { - registerCase(label); - if (ok) { - console.log(` ok ${label}`); - } else { - failed += 1; - console.log(` FAIL ${label}${note ? ` -- ${note}` : ''}`); - } - }; - - // -- parsing ------------------------------------------------------------- - check('parse: turbo Failed: footer', failingTasks('Failed: @objectstack/cli#test\n').join() === '@objectstack/cli#test'); - check( - 'parse: turbo per-task ERROR line', - failingTasks(`${taskError('@objectstack/verify')}\n`).join() === '@objectstack/verify#test', - ); - { - const p = readPass(`${elifecycle('@objectstack/plugin-auth')}\n`); - check('parse: an ELIFECYCLE line alone is INTERRUPTED, never failing', p.failing.length === 0, `failing ${p.failing}`); - check( - 'parse: an ELIFECYCLE line alone is INTERRUPTED, never failing', - p.interrupted.join() === '@objectstack/plugin-auth#test', - `interrupted ${p.interrupted}`, - ); - } - { - const p = readPass(NIGHT_1014_PASS1); - const label = 'parse: 2026-10-14 pass 1 -- dogfood failed, cli interrupted by fail-fast'; - check(label, p.failing.join() === '@objectstack/dogfood#test', `failing ${p.failing}`); - check(label, p.interrupted.join() === '@objectstack/cli#test', `interrupted ${p.interrupted}`); - check(label, p.timeoutOnly.join() === '@objectstack/dogfood#test', `timeoutOnly ${p.timeoutOnly}`); - check(label, p.ran.join() === '@objectstack/dogfood#test', `ran ${p.ran}`); - check(label, p.finished && p.started === 143 && p.successful === 141, `summary ${p.successful}/${p.started}`); - } - { - const p = readPass(NIGHT_1015_PASS2); - const label = 'parse: 2026-10-15 pass 2 -- runtime failed; verify, cli, organizations interrupted'; - check(label, p.failing.join() === '@objectstack/runtime#test', `failing ${p.failing}`); - check( - label, - p.interrupted.join() === '@objectstack/cli#test,@objectstack/organizations#test,@objectstack/verify#test', - `interrupted ${p.interrupted}`, - ); - check(label, p.timeoutOnly.join() === '@objectstack/runtime#test' && p.started === 142, `timeoutOnly ${p.timeoutOnly}`); - } - { - const stamped = - `2026-09-17T04:46:54.4228362Z ${taskError('@objectstack/cli')}\n` + - '2026-09-17T04:46:54.4229431Z Failed: @objectstack/cli#test\n'; - check( - 'parse: an API-downloaded log with ISO timestamps reads the same', - failingTasks(stamped).join() === '@objectstack/cli#test', - `got ${failingTasks(stamped).join(', ')}`, - ); - } - { - const p = readPass(' Tasks: 140 successful, 140 total\n Time: 39m50.01s\n'); - check('parse: a green log names no failing task and is finished', p.failing.length === 0 && p.interrupted.length === 0); - check('parse: a green log names no failing task and is finished', p.finished && p.started === 140); - } - { - const mixed = [ - failedTestsHeader('@objectstack/core:test:', 2), - timedOut('@objectstack/core:test:'), - '@objectstack/core:test: AssertionError: expected 4 to be 2', - taskError('@objectstack/core'), - ].join('\n'); - check('parse: timeout-only needs EVERY failed test to be a timeout', readPass(mixed).timeoutOnly.length === 0); - const suite = [ - `@objectstack/core:test: ${RULE} Failed Suites 1 ${RULE}`, - failedTestsHeader('@objectstack/core:test:', 1), - timedOut('@objectstack/core:test:'), - taskError('@objectstack/core'), - ].join('\n'); - check('parse: timeout-only needs EVERY failed test to be a timeout', readPass(suite).timeoutOnly.length === 0); - } - { - const annotated = [ - failedTestsHeader('@objectstack/core:test:', 2), - timedOut('@objectstack/core:test:'), - '@objectstack/core:test: AssertionError: expected 4 to be 2', - '@objectstack/core:test: ::error file=/w/a.test.ts,title=a::Error: Test timed out in 5000ms.', - taskError('@objectstack/core'), - ].join('\n'); - check('parse: the ::error annotation copy is not a second timeout', readPass(annotated).timeoutOnly.length === 0); - } - { - // What a pass looks like when its budget's SIGTERM reached turbo. - const cut = readPass(`${testFiles('@objectstack/core:test:', 0, 96)}\nShutting down Turborepo tasks...\n`); - check('parse: a log without turbo\'s closing summary did not finish', cut.finished === false && cut.started === null); - check('parse: a log without turbo\'s closing summary did not finish', readPass('').finished === false); - } - - // -- the verdict table --------------------------------------------------- - const V = (p1, p2, f1 = [], f2 = []) => - verdict({ - pass1: passOf(p1, logOf({ failing: f1, ran: ['a#test', 'b#test', 'c#test'] })), - pass2: passOf(p2, logOf({ failing: f2, ran: ['a#test', 'b#test', 'c#test'] })), - }); - - { - const v = V(0, 0); - check('verdict: both green -> RERUN_SAFE, measured, exit 0', v.code === 'RERUN_SAFE' && v.coverage === 'full' && v.exit === 0, v.code); - } - { - const v = V(0, 1, [], ['@objectstack/core#test']); - check( - 'verdict: pass1 green + pass2 red -> RERUN_UNSAFE, exit 1', - v.code === 'RERUN_UNSAFE' && v.exit === 1 && v.detail.some((d) => d.includes('@objectstack/core#test')), - v.code, - ); - } - { - const v = V(1, 0, ['@objectstack/cli#test'], []); - check( - 'verdict: pass1 red + pass2 green -> BASELINE_FLAKY, measured', - v.code === 'BASELINE_FLAKY' && v.measured === true && v.exit === 1, - v.code, - ); - } - { - const same = ['@objectstack/cli#test']; - const v = V(1, 1, same, same); - check( - 'verdict: both red, same set -> BASELINE_BROKEN, measured, CLEAN', - v.code === 'BASELINE_BROKEN' && v.coverage === 'full' && v.headline.includes('CLEAN'), - `${v.code} ${v.coverage}`, - ); - } - { - const v = V(1, 1, ['@objectstack/cli#test'], ['@objectstack/cli#test', '@objectstack/core#test']); - check( - 'verdict: both red, pass2 worse -> RERUN_UNSAFE_ON_BROKEN_BASELINE', - v.code === 'RERUN_UNSAFE_ON_BROKEN_BASELINE' && - v.detail.some((d) => d.includes('Newly failing on pass 2: @objectstack/core#test')), - v.code, - ); - } - { - const v = V(STALL_EXIT_CODE, 0); - check( - 'verdict: a stall in pass 1 -> STALL, NOT MEASURED, exit 75', - v.code === 'STALL' && v.measured === false && v.exit === STALL_EXIT_CODE, - v.code, - ); - } - { - const v = V(0, STALL_EXIT_CODE); - check( - 'verdict: a stall in pass 2 -> STALL, NOT MEASURED, exit 75', - v.code === 'STALL' && v.measured === false && v.exit === STALL_EXIT_CODE, - v.code, - ); - } - { - // The shape measured locally (guard -> timeout -> turbo, stopped at 45 s): - // exit 124 through the guard, and the log ends at turbo's shutdown lines. - const pass1 = passOf(1, NIGHT_1014_PASS1); - const cut = passOf( - 124, - [ - testFiles('@objectstack/core:test:', 0, 96), - 'Shutting down Turborepo tasks...', - 'Force killed Turborepo tasks: @objectstack/core#test', - ].join('\n'), - ); - const v = verdict({ pass1, pass2: cut }); - check( - 'verdict: a pass that did not finish -> INCOMPLETE, NOT MEASURED, exit 1', - v.code === 'INCOMPLETE' && v.coverage === 'none' && v.measured === false && v.exit === 1, - `${v.code} ${v.coverage}`, - ); - check( - 'verdict: a pass that did not finish -> INCOMPLETE, NOT MEASURED, exit 1', - v.detail.some((d) => d.includes('Pass 2 (exit 124)') && d.includes("share of the job's time")) && - v.detail.includes('Failed in pass 1: @objectstack/dogfood#test'), - v.detail.join(' | '), - ); - } - { - // A budget stop that turbo still summarised: every task it killed is in - // its footer, and only the one whose vitest run reported a failure failed. - const budgetStopped = passOf( - 124, - [ - failedTestsHeader('@objectstack/core:test:', 1), - timedOut('@objectstack/core:test:'), - taskError('@objectstack/core'), - taskError('@objectstack/runtime'), - tasksSummary(40, 42), - 'Failed: @objectstack/core#test, @objectstack/runtime#test', - ].join('\n'), - ); - const v = verdict({ pass1: passOf(1, NIGHT_1014_PASS1), pass2: budgetStopped }); - check( - 'verdict: a pass that did not finish -> INCOMPLETE, NOT MEASURED, exit 1', - v.code === 'INCOMPLETE' && v.coverage === 'none', - `${v.code} ${v.coverage}`, - ); - check( - 'verdict: a pass that did not finish -> INCOMPLETE, NOT MEASURED, exit 1', - v.detail.includes('Failed in pass 2 before it stopped: @objectstack/core#test') && - v.detail.includes('Stopped mid-run in pass 2 by the time bound (unmeasured): @objectstack/runtime#test'), - v.detail.join(' | '), - ); - } - { - // Every FINISHED, non-stall combination still reports a reading. - const combos = [V(0, 0), V(0, 1, [], ['a#test']), V(1, 0, ['a#test'], []), V(1, 1, ['a#test'], ['a#test']), V(1, 1, ['a#test'], ['a#test', 'b#test'])]; - check( - 'verdict: every finished, non-stall verdict reports measured=true', - combos.every((v) => v.measured === true), - combos.map((v) => `${v.code}=${v.measured}`).join(' '), - ); - } - { - const v = verdict({ pass1: passOf(1, NIGHT_1014_PASS1), pass2: passOf(1, NIGHT_1014_PASS2) }); - const label = 'verdict: 2026-10-14 end to end -- no interrupted task is newly failing, coverage PARTIAL'; - const newly = v.detail.find((d) => d.startsWith('Newly failing on pass 2:')) ?? ''; - check(label, v.code === 'RERUN_UNSAFE_ON_BROKEN_BASELINE', v.code); - // The pre-#23880 reading named core, lint AND spec here. - check(label, newly === 'Newly failing on pass 2: @objectstack/core#test', newly); - check(label, v.coverage === 'partial' && v.measured === true, v.coverage); - check( - label, - v.detail.some((d) => d.includes('Interrupted by turbo in pass 1') && d.includes('@objectstack/cli#test')), - v.detail.join(' | '), - ); - } - { - const v = verdict({ pass1: passOf(1, NIGHT_1014_PASS1), pass2: passOf(1, NIGHT_1014_PASS2) }); - check( - 'verdict: a pass-1 failure pass 2 never finished is not re-measured, never recovered', - v.detail.some((d) => d.startsWith('Failed on pass 1 and never finished by pass 2') && d.includes('@objectstack/dogfood#test')) && - !v.detail.some((d) => d.includes('green on pass 2')), - v.detail.join(' | '), - ); - } - - // -- rendering ----------------------------------------------------------- - { - const full = { pass1: passOf(1, logOf({ failing: ['a#test'] })), pass2: passOf(1, logOf({ failing: ['a#test'] })) }; - const partial = { pass1: passOf(1, NIGHT_1014_PASS1), pass2: passOf(1, NIGHT_1014_PASS2) }; - const stall = { pass1: passOf(STALL_EXIT_CODE, ''), pass2: passOf(0, logOf()) }; - const text = (args) => renderVerdict(verdict(args), args); - check( - 'render: the verdict line names MEASURED, PARTIALLY MEASURED or NOT MEASURED', - text(full).includes('rerun property: MEASURED') && - text(partial).includes('rerun property: PARTIALLY MEASURED') && - text(stall).includes('rerun property: NOT MEASURED'), - ); - } - { - const args = { pass1: passOf(1, NIGHT_1014_PASS1), pass2: passOf(1, NIGHT_1014_PASS2) }; - const out = renderVerdict(verdict(args), args); - check( - 'render: a timeout-only newly failing task carries the twice-on-its-own instruction', - out.includes('pnpm --filter @objectstack/core run test') && out.includes('@objectstack/core#test (timeout-only)'), - out, - ); - } - - // -- the floor ----------------------------------------------------------- - const declared = Object.keys(SELF_TEST_BATTERIES); - let floorBroken = 0; - if (declared.length < SELF_TEST_BATTERY_FLOOR) { - console.log( - ` FAIL roster floor: ${declared.length} declared batteries, pinned at ${SELF_TEST_BATTERY_FLOOR}`, - ); - floorBroken += 1; - } - for (const [label, min] of Object.entries(SELF_TEST_BATTERIES)) { - const ran = registered.get(label) ?? 0; - if (ran < min) { - console.log(` FAIL battery "${label}" DID NOT RUN (registered ${ran}, floor ${min})`); - floorBroken += 1; - } - } - for (const label of registered.keys()) { - if (!(label in SELF_TEST_BATTERIES)) { - console.log(` FAIL case "${label}" names no declared battery`); - floorBroken += 1; - } - } - - const bad = failed + floorBroken; - console.log( - bad === 0 - ? `rerun-safety-verdict --self-test: OK (${declared.length} batteries, ${registered.size} registered)` - : `rerun-safety-verdict --self-test: ${bad} FAILED`, - ); - selfTestReachedVerdict = true; - return bad === 0 ? 0 : 1; -} - -// --------------------------------------------------------------------------- -// CLI -// --------------------------------------------------------------------------- - -function readArg(argv, name) { - const i = argv.indexOf(name); - return i === -1 ? undefined : argv[i + 1]; -} - -function main(argv) { - if (argv.includes('--self-test')) { - const status = selfTest(); - if (!selfTestReachedVerdict) { - console.error('rerun-safety-verdict: the self-test never reached its verdict -- refusing to report a pass.'); - return 1; - } - return status; - } - - const pass1Log = readArg(argv, '--pass1-log'); - const pass2Log = readArg(argv, '--pass2-log'); - const pass1Exit = Number(readArg(argv, '--pass1-exit')); - const pass2Exit = Number(readArg(argv, '--pass2-exit')); - - if (!pass1Log || !pass2Log || !Number.isInteger(pass1Exit) || !Number.isInteger(pass2Exit)) { - console.error( - 'usage: rerun-safety-verdict.mjs --pass1-log --pass1-exit --pass2-log --pass2-exit ', - ); - return 2; - } - - // A log this script cannot read is a reading it cannot stand behind -- say so - // rather than reporting an empty failing set, which would read as "clean". - const read = (path) => { - try { - return readFileSync(path, 'utf8'); - } catch (e) { - console.error(`rerun-safety-verdict: cannot read ${path} -- ${e.message}`); - return null; - } - }; - const t1 = read(pass1Log); - const t2 = read(pass2Log); - if (t1 === null || t2 === null) return 2; - - const args = { pass1: passOf(pass1Exit, t1), pass2: passOf(pass2Exit, t2) }; - const v = verdict(args); - console.log(renderVerdict(v, args)); - if (v.exit !== 0) { - console.log(`::error::Rerun Safety: ${v.code} -- ${v.headline}`); - for (const d of v.detail) console.log(`::error::${d}`); - } - return v.exit; -} - -// This module EXPORTS `readPass` / `failingTasks` / `verdict` / `renderVerdict` -// so the workflow's verdict step and any future reader can use them, so its top -// level must not run on import -- an importer would otherwise be ended -// mid-import by the dispatch below (`pnpm check:entry-guard`). -if (isEntrypoint(import.meta.url)) { - process.exitCode = main(process.argv.slice(2)); -} From 95219d01ee46d39728487c7d50c5c6e0bda506ca Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 17 Oct 2026 14:48:57 +0000 Subject: [PATCH 2/3] ci(test-leftovers): cap the per-package log listing; clearer empty summary The job log lists at most 50 paths per package and counts the rest, so one package that leaves thousands of files cannot bury the others; the self-test pins it. Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t Co-authored-by: Claude --- scripts/ci/report-test-leftovers.mjs | 29 +++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/scripts/ci/report-test-leftovers.mjs b/scripts/ci/report-test-leftovers.mjs index 8057d674cd1..068540303ff 100644 --- a/scripts/ci/report-test-leftovers.mjs +++ b/scripts/ci/report-test-leftovers.mjs @@ -105,6 +105,11 @@ export const SNAPSHOT_SCHEMA = 'test-leftovers-snapshot/1'; export const ROOT_GROUP = '(root)'; export const EXAMPLES_PER_PACKAGE = 5; +// The job log lists each package's paths in full up to this many, then says how +// many more: one package that leaves thousands of files must not bury the rest. +// `--json` always carries every path. +export const LOG_PATHS_PER_PACKAGE = 50; + // GitHub shows at most 10 warning annotations per step and drops the rest // without saying so. Past that many packages the last slot becomes one roll-up // annotation naming the rest, so no package is dropped silently; the job @@ -369,7 +374,12 @@ export function renderSummary(report, { label, notes = [] }) { '', ); if (report.packages.length === 0) { - lines.push(`No path was added by the test run (${report.added} new path(s) in all, none outside the rules below).`, ''); + lines.push( + report.added === 0 + ? 'No path was added by the test run.' + : `No path was added by the test run outside the rules below (${report.added} new path(s), every one under a rule).`, + '', + ); } else { lines.push(`| package | paths added | examples (up to ${EXAMPLES_PER_PACKAGE}) |`, '| :-- | --: | :-- |'); for (const pkg of report.packages) { @@ -404,7 +414,8 @@ export function renderLog(report, { label }) { ); for (const pkg of report.packages) { lines.push(` ${pkg.dir} ${pkg.count}`); - for (const path of pkg.paths) lines.push(` ${path}`); + for (const path of pkg.paths.slice(0, LOG_PATHS_PER_PACKAGE)) lines.push(` ${path}`); + if (pkg.count > LOG_PATHS_PER_PACKAGE) lines.push(` ... and ${pkg.count - LOG_PATHS_PER_PACKAGE} more under ${pkg.dir}`); } } const excl = exclusionLines(report); @@ -527,7 +538,7 @@ const SELF_TEST_BATTERIES = Object.freeze({ '1. The diff: what counts as ADDED, and under which package': 7, '2. The fixed exclusions: each one, and only its own paths': 8, '3. turbo.json: the builds a test step runs after the snapshot': 8, - '4. Rendering: the table, the annotations, the escaping': 9, + '4. Rendering: the table, the annotations, the escaping': 10, '5. The real listing: ignored paths are seen (a real git tree)': 5, '6. Never fails a job: the in-job modes exit 0 on every path': 9, }); @@ -678,6 +689,18 @@ function selfTest() { capped.length === ANNOTATIONS_PER_STEP && capped.at(-1).includes('3 more package(s)') && capped.at(-1).includes('packages/p11 (1)'), capped.join('\n'), ); + const flood = computeReport({ + before: [], + after: Array.from({ length: LOG_PATHS_PER_PACKAGE + 3 }, (_, i) => `packages/runtime/.objectstack/f${String(i).padStart(3, '0')}`), + packageDirs: dirs, + }); + const logLines = renderLog(flood, { label: 'L' }); + assert( + `the log lists at most ${LOG_PATHS_PER_PACKAGE} paths of a package and counts the rest`, + logLines.filter((l) => l.startsWith(' packages/runtime/')).length === LOG_PATHS_PER_PACKAGE && + logLines.some((l) => l.includes('... and 3 more under packages/runtime')), + logLines.slice(-3).join('\n'), + ); assert('an empty report raises no annotation', renderAnnotations(computeReport({ before: [], after: [], packageDirs: dirs }), { label: 'L' }).length === 0); const md = renderSummary(computeReport({ before: [], after: ['packages/runtime/a|b.json'], packageDirs: dirs }), { label: 'L' }); assert('the summary table escapes a pipe in a path', md.includes('`packages/runtime/a\\|b.json`'), md); From 015e6eab7d4c21362172d52ef98a42e86c28043a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 17 Oct 2026 15:21:15 +0000 Subject: [PATCH 3/3] ci(test-leftovers): declare no path population; de-path the schema id check:declared-population-live read the snapshot schema id as a path declaration. The self-test reads no tracked file and the in-job modes run only inside ci.yml's test jobs, so the script now says so with the no-path-population marker, and the id no longer looks like a path. Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t Co-authored-by: Claude --- scripts/ci/report-test-leftovers.mjs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/ci/report-test-leftovers.mjs b/scripts/ci/report-test-leftovers.mjs index 068540303ff..30c07df010b 100644 --- a/scripts/ci/report-test-leftovers.mjs +++ b/scripts/ci/report-test-leftovers.mjs @@ -99,9 +99,11 @@ import { fileURLToPath } from 'node:url'; import { isEntrypoint } from '../invoked-as.mjs'; import { workspacePackages } from '../workspace-enumerator.mjs'; +// dispatch-gates: no-path-population -- the --self-test (the one invocation a pull request runs, in Lint & Repo Gates) drives in-memory fixture snapshots and throwaway git trees under the OS temp directory and reads no tracked file; the snapshot and report modes read turbo.json, the workspace manifests and the untracked tree only inside ci.yml's test jobs, which ci.yml's own filter already starts for a change to packages or scripts, so a hint here would be a second spelling of one population + const SELF = fileURLToPath(import.meta.url); -export const SNAPSHOT_SCHEMA = 'test-leftovers-snapshot/1'; +export const SNAPSHOT_SCHEMA = 'test-leftovers-snapshot.v1'; export const ROOT_GROUP = '(root)'; export const EXAMPLES_PER_PACKAGE = 5;