Repository navigation
Commit 2d2e3ce
ci(nightly-tiers): the nightly states its own slicing, one CLI slice per shard (#23444)
Fixes #23407
Clause-②: no
## What was wrong (measured)
The nightly tiers shard the CLI's e2e tier with Test Core's partitioner,
and until now with Test Core's slice map, `FILE_SHARDED_PACKAGES`. That
map has changed twice for Test Core's own reasons, and each change took
the nightly's second shard away without any step going red:
| tree | `FILE_SHARDED_PACKAGES` | `--shard 1/2` | `--shard 2/2` |
|---|---|---|---|
| `fd96a847` (`9ff74285^`) | cli: 2 | `@objectstack/cli 1/2` |
`@objectstack/cli 2/2` |
| `9ff74285` (2026-10-02, retired the entry) | empty |
`@objectstack/cli` | (nothing) |
| `3ca71b6e` (`59194834^`) | empty | `@objectstack/cli` | (nothing) |
| `59194834` (2026-10-09, sliced 3 ways) | cli: 3 | `@objectstack/cli` |
(nothing): "3 slices cannot spread across this run's shards" |
| `e4ec6ec1a5` (run 38517756588) and `4f81f31526` (base) | cli: 3 |
`@objectstack/cli` | (nothing): same refusal |
Each row comes from `node scripts/nightly-tiers.mjs --packages` and then
`node scripts/partition-test-shards.mjs DOC --shard k/2`, run against
that tree's own scripts.
The nightly runs agree with that history. This is the "Run this shard's
nightly tiers" step's duration, read from the Actions jobs API:
- 2026-09-30 and 2026-10-01: both shards non-empty, 1183s / 1190s and
1196s / 1168s.
- 2026-10-04 onward: shard 2 ran 0s every night. On seven of eight green
nights shard 1 took 2361-2615s; 2026-10-07 read 1589s.
- 2026-10-14 (run 38517756588): shard 1 was cancelled at the 45-minute
limit. Shard 2's log reads "No packages on this shard — nothing to
test."
So the workflow header's premise (FILE_SHARDED_PACKAGES gives the CLI
two slices, one per shard) stopped holding at `9ff74285`, a week before
`59194834`. `59194834` did not bring the second slice back, because 3
slices cannot spread over 2 shards.
## The fix
The nightly now states its own slicing. Test Core's map no longer
reaches it.
- `scripts/nightly-tiers.mjs --packages` writes a `sliceAcrossShards`
key naming every tier-owning package. Today that is
`["@objectstack/cli"]`.
- When a document carries that key, `scripts/partition-test-shards.mjs`
cuts each named package into as many slices as the run has shards (the N
of the run's own `--shard k/N`). `FILE_SHARDED_PACKAGES` does not apply
to that run. `planShards()` (NEEDED / SPREADS / WITHIN CAP) and the
vitest file floor still apply.
- `turbo ls` never writes the key, so every Test Core document reads it
as absent and slices exactly as before (proof below). When the key is
present it is checked strictly: it must be an array of distinct names,
and each name must be an item of the same document.
- The planning in `main()` moved into one exported function,
`planRun()`, in the same order. `main()` prints its result, and the
self-test drives that same function.
- No workflow change. The nightly keeps 2 shards. Because the count
comes from the run's own `--shard k/N`, changing the nightly matrix
later re-slices it with no planner edit.
New output on the base tree's inputs: `--shard 1/2` gives
`@objectstack/cli 1/2` and `--shard 2/2` gives `@objectstack/cli 2/2`,
at 1102.9s predicted each. A third log line reads `slicing map: the
document's sliceAcrossShards -- @objectstack/cli cut 2 ways, one slice
per shard of this run; FILE_SHARDED_PACKAGES (Test Core's map) does not
apply to this run`. At `--shard k/3` the output is `@objectstack/cli
1/3`, `2/3` and `3/3`.
Expected wall time per shard: the whole tier ran 2361-2615s on one
shard, and the 2-slice runs on 10-01 ran 1196s / 1168s. That points to
roughly 1200-1310s of tests per shard (about 20-22 min), plus about 1
min of setup, against the 45-minute limit.
## Pin: a planner change that empties a shard again goes red
`partition-test-shards.mjs --self-test` has a new battery, "a document
states its own slicing: the nightly tiers", with 16 cases. lint.yml runs
this self-test on every pull request. The battery covers:
- An absent key slices by FILE_SHARDED_PACKAGES through `planRun()`.
- Five malformed shapes are refused by name.
- The run map is N ways, and a one-shard run cuts nothing.
- One package at 2, 3 and 4 shards lands one distinct slice per shard.
- A control case: 3 configured slices over 2 shards leaves one shard
empty, so the predicate can go red.
- `test-nightly-tiers.yml`'s matrix size equals the N in its `--shard`
argument, and the shards are partitioned from `nightly-tiers.mjs
--packages`.
- Three live pins drive the nightly's own document from the live tree
through `planRun()` at the workflow's shard count: the key names every
tier package; no shard is empty; the CLI lands as one distinct slice on
each shard.
Red shown once, then restored. Both mutations were applied with
`scripts/ablation-replace.mjs` on the committed fix (`e11d8343`); each
restore was proven by blob hash equal to HEAD and an empty `git diff
HEAD`:
1. Planner-side. `const sliced = acrossShards === null ?
FILE_SHARDED_PACKAGES : sliceMapForRun(acrossShards, shardCount);` was
replaced with `const sliced = FILE_SHARDED_PACKAGES;` (anchor 1 to 0,
blob `6bc0cdf7948b` to `e08f8b5ec718`). The self-test exited 1 with:
`nightly: the live document at the workflow's 2 shards leaves shard(s) 2
with nothing to run, while the split is [@objectstack/cli | ]
(@objectstack/cli: whole (... but 3 slices cannot spread across this
run's shards))`. That is the production failure. Restored to
`6bc0cdf7948b`.
2. Producer-side. The key was dropped from `packageListDocument()` (blob
`d75822731830` to `232c14a0299f`). The self-test exited 1 with `nightly
document: sliceAcrossShards is undefined for the tier-owning packages
["@objectstack/cli"] ...`. Restored to `d75822731830`.
## Test Core's partition is byte-identical
Test Core's partition did not change. Check: same command on both trees,
base `4f81f315` (origin/main) and head `448c8745`, and `cmp` on stdout,
stderr and exit code. The command is `node
scripts/partition-test-shards.mjs DOC --shard k/6 --exclude
@objectstack/dogfood`, for k = 1..6, over five documents:
- the full `pnpm turbo ls --output=json` list (83 packages);
- an affected subset with the CLI;
- an affected subset without it;
- the CLI alone;
- an empty list.
Result: 90 of 90 files identical. The concatenated sha256 is
`885ce341b5b2d363...` on both trees, and all 30 exits are 0. The full
list still puts `@objectstack/cli 1/3`, `2/3` and `3/3` on shards 2, 3
and 4.
## Gates (all on head `448c8745`)
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derived 32 commands from the 2-path change
set. All 32 ran with exit 0, including `pnpm check:pm-dispatch-gates`
(fast tier, 1841 cases). `--ran` reconciliation: 32 derived, 32 run, 0
NOT-MEASURED, 0 UNRUN.
- The edited scripts' own suites: `nightly-tiers.mjs --self-test` (47
cases) and `partition-test-shards.mjs --self-test`, both exit 0. The
partitioner's module graph gained an import, so the self-tests of the
scripts that import it also ran: `check-test-completeness`,
`check-test-suite-ceilings`, `measure-test-shard-timings` and
`check-tier-file-adoption`, all exit 0.
- Lint, narrowed to the 2 changed files: `npx eslint --no-inline-config
--format json` read 2 files with 0 errors and 0 warnings. Both files are
in eslint's population (`isPathIgnored` is false for each;
`eslint.config.mjs` covers `**/*.{...,mjs,...}`). Type-aware linting is
not enabled (no `project` / `projectService` in `eslint.config.mjs`), so
this diff cannot change the verdict on any untouched file. Full `pnpm
lint` is left to CI.
- `skip-changeset`: this diff publishes nothing. The root `package.json`
is private, `scripts/` is not shipped, and 0 public `files[]` entries
name these scripts. As a positive control, 72 of 72 public manifests
have a `dist` entry.
- This PR triggers the nightly workflow itself (its `pull_request` paths
filter names `scripts/nightly-tiers.mjs`). That run on real runners is
the first two-shard reading. It writes nothing to the board.
## Acceptance notes
- NOT MEASURED locally: whether the two vitest slices are disjoint.
`vitest list --filesOnly --shard=k/2` ignores the shard: both slices
listed all 82 tier files. The evidence that the `--shard=k/n`
passthrough partitions the tier is the 2026-10-01 run (1196s / 1168s).
This PR's own nightly run is the next reading.
- The workflow header's "Sharding — TWO shards" paragraph is stale. So
is the `timeout-minutes` comment ("about 13 min a shard"). The header
says the shard count is spelled in three places; it is four, because the
artifact name `nightly-tiers-reports-...-of-2` is the fourth. Correcting
these is a governed (Tier H) workflow edit, so it is left out of this
scripts-only PR. Carrier: the next PR that touches
`test-nightly-tiers.yml`.
- Nothing outside the workflow reads the nightly's shard count. The
`report` job downloads with `pattern: nightly-tiers-reports-*`, and
`check-test-completeness.mjs --scheduled` reads only its own shard's
list. Neither depends on the count. The new self-test reads the matrix,
so a mismatch between matrix and `--shard` now goes red.
- An empty nightly shard still exits 0 in the workflow ("No packages on
this shard"). That green is how this stayed hidden for twelve nights.
With this change the planner-side cause is pinned on every pull request.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 4369677 commit 2d2e3ce
2 files changed
Lines changed: 309 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
70 | 98 | | |
71 | 99 | | |
72 | 100 | | |
| |||
195 | 223 | | |
196 | 224 | | |
197 | 225 | | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
198 | 231 | | |
199 | 232 | | |
200 | 233 | | |
201 | | - | |
| 234 | + | |
202 | 235 | | |
203 | 236 | | |
204 | 237 | | |
| |||
462 | 495 | | |
463 | 496 | | |
464 | 497 | | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
465 | 504 | | |
466 | 505 | | |
467 | 506 | | |
| |||
497 | 536 | | |
498 | 537 | | |
499 | 538 | | |
500 | | - | |
| 539 | + | |
501 | 540 | | |
502 | 541 | | |
503 | 542 | | |
| |||
0 commit comments