nightly-tiers: red on main

os-nightly-tiers — machine-findable marker for this generated card. ⛔ Do not delete this line: it is how the nightly finds this card instead of filing a new one every night.

nightly-tiers: red on main

Swept 2026-10-14T06:24:37.152Z · expected every 24h while this card stands (cron 29 5 * * * UTC) · next by 2026-10-15T05:29Z · run log · commit e4ec6ec1a5eff818d69e5bdf1ae031f37fec64bd · trigger schedule · tiers job result cancelled.

The Swept line above is this nightly's heartbeat, and it states the cadence that makes
「stalled」 decidable: while the tiers stay red this card is refreshed on that schedule, so a
timestamp still sitting there past the next by deadline means either the standing caller
died or the nightly went green — nothing here files or closes anything on green, so both
readings end at this card. ⛔ Do not carry a cadence over from a sibling patrol anchor: they
differ by up to 4× and each states its own.

The e2e and live test tiers — the files named *.e2e.test.* and *.live.test.* — run here
nightly on main under OS_TEST_TIERS=nightly and nowhere else (the per-PR and merge-queue
Test Core runs under queue). A red here is a real suite that no pull request will red on;
this card is the channel that sees it. Nothing is blocked by it.

⛔ The remedy is never to rename, skip or delete the failing test to make the nightly green.
Reproduce locally with OS_TEST_TIERS=nightly pnpm --filter <pkg> test, fix what it names, and
let the next nightly refresh this card.

Failing files

⚠️ No vitest report named a failing nightly-tier file, yet the tiers job did not succeed — the run died before or outside vitest (a build failure, a stall kill, a cancelled shard). Read the run log.

Filed by .github/workflows/test-nightly-tiers.yml. Generated by Claude Code

Triage: first grade · `priority:p1` kept · `domain:devx` · `pm:queue` · the red is a shard budget, not a failing test · triage seat · `session_01AavokzJ5DndAwitDXvKy4U` · 2026-10-14T07:01Z

Read fresh, and run 38517756588 read job by job:

Shard 1/2 was killed at its timeout-minutes: 45 (05:38:58Z → 06:24:12Z, "The operation was canceled"). Up to the kill it was in @objectstack/cli's *.e2e.test.* files, every one passing. The last file finished green at 06:24:12Z. So no vitest report was written, and the card lists no failing file.
Shard 2/2 took about 1 min, success.
Shard 1 had been at the edge for days: 44 min on 10-13, 40 min on 10-12, 44 min on 10-11, all against the 45-min kill. The workflow comment expects "≈ 13 min a shard". Today it crossed.

Grade:

Admitted. These tiers are the only place the e2e/live suites of the published packages run. While shard 1 is killed by the clock, a real e2e red in it is invisible.
p1 kept. That blindness covers the window before 18.0.0-next.1 and 18 GA.
Lane domain:devx (.github/workflows/test-nightly-tiers.yml and its shard planner).
The ask: balance the shards, since shard 1 holds the whole cli e2e tier and shard 2 almost nothing, so each lands well inside the budget with headroom.
- ⛔ Do not raise the timeout alone: the next added e2e file crosses it again.
- ⛔ Never skip or delete a test to fit.
- Done = a nightly where both shards finish under budget with a vitest report each.

Claim: PM loop round 1
Session: session_01B7mrh59UAGffd7s9asUfoY
Account: sunhaolin (the seat's linked user as GET /user answers it; the card's assignee)
Branch: claude/issue-23407-nightly-tiers-shard-balance
Worktree: objectstack-issue-23407
Domain: domain:devx
Seat: domain:devx#2
File surface: scripts/nightly-tiers.mjs, scripts/partition-test-shards.mjs and their self-test batteries (the planner side first). .github/workflows/test-nightly-tiers.yml is touched only if the shard count or the planner call itself must change; that hunk is governed (Tier H) and lands by the maintainer's approval. ⛔ Test Core's own partition (ci.yml, the required set) does not move: its output on main must stay byte-identical. Stop on a breach and explain it in the report.
Container & model: M, mode:subagent, model: opus (dispatch-gates --tier: no path-derived mandate; default tier; CI planner judgment)
Clause-②: no
Responsibility: the nightly tiers' own shard planner (scripts/nightly-tiers.mjs feeding scripts/partition-test-shards.mjs, wired in test-nightly-tiers.yml) | none: no other lane runs the e2e/live tiers | the 05:29Z nightly on main, every night; while shard 1 is killed, a real e2e red in it is invisible to everyone
Thread-read: 6162829268
Serial constraints cleared: none — none of the 9 open PRs touches these paths, and the lane's one in-flight claim (#23392, the console pin) declares none of them. Read at 2026-10-14T08:04Z.

Generated by Claude Code

os-dev-report
{
"issue": 23407,
"status": "done",
"branch": "claude/issue-23407-nightly-tiers-shard-balance",
"pr": "#23444",
"session": "session_01B7mrh59UAGffd7s9asUfoY",
"premise_still_valid": true,
"summary": "Card premise confirmed: on origin/main the nightly's partition puts the whole @objectstack/cli on shard 1/2 and nothing on 2/2, because Test Core's FILE_SHARDED_PACKAGES gives the CLI 3 slices and 3 slices cannot spread over 2 shards. The PM's mechanism hypothesis is refined, not confirmed. The premise broke at 9ff7428 (2026-10-02, map emptied: shard 2 ran 0s on every night from 10-04), a week before 5919483. 5919483 (3 slices) just never restored it. No document field was missing. Fix (scripts only, no workflow change): nightly-tiers.mjs --packages now writes a sliceAcrossShards key naming every tier-owning package. When a document carries that key, partition-test-shards.mjs cuts each named package into N slices, N being the run's own --shard k/N, and FILE_SHARDED_PACKAGES does not apply to that run. Nightly result: shard 1/2 = @objectstack/cli 1/2, shard 2/2 = @objectstack/cli 2/2, 1102.9s predicted each. Test Core's partition is byte-identical before and after. A new 16-case self-test battery drives the live nightly document through planRun() at the workflow's shard count. Shown red once with each of two ablations (planner side and producer side), then restored.",
"tests": "All on head 448c874 unless stated. (1) Premise, both trees, as the PM asked: node scripts/nightly-tiers.mjs --packages, stdout saved as F; then node scripts/partition-test-shards.mjs F --shard k/2. 5919483^ (3ca71b6): 1/2='@objectstack/cli', 2/2=(empty). origin/main 4f81f31: 1/2='@objectstack/cli', 2/2=(empty), 'slicing: @objectstack/cli: whole (... but 3 slices cannot spread across this run's shards)'. Also measured: 9ff7428^ (fd96a84) gives 1/2='@objectstack/cli 1/2', 2/2='@objectstack/cli 2/2'; 9ff7428 gives whole/empty; run commit e4ec6ec gives whole/empty. Actions jobs API, step 'Run this shard's nightly tiers': 09-30 1183/1190s and 10-01 1196/1168s (both shards); from 10-04 shard 2 = 0.0s every night, shard 1 = 2361/2403/2449/1589/2528/2615/2380/2592s; run 38517756588 shard 1 cancelled at 44:31. MCP job log of 115700290620 reads 'No packages on this shard — nothing to test.' (2) Fix: on the base inputs, --shard 1/2 gives '@objectstack/cli 1/2' and 2/2 gives '@objectstack/cli 2/2' ('all bins: 1103/1103s'); k/3 gives 1/3, 2/3, 3/3. (3) Test Core byte-identity, both trees, base 4f81f31 vs head 448c874: node scripts/partition-test-shards.mjs DOC --shard k/6 --exclude @objectstack/dogfood for k=1..6 over 5 documents (full pnpm turbo ls --output=json = 83 packages, affected-with-cli, affected-without-cli, cli-only, empty). cmp on stdout+stderr+exit: 90/90 identical, concatenated sha256 885ce341b5b2d3633b2a8e4281e73c28664c34e003a30ce0e97cad6e7a380444 on both; 30/30 exit 0. (4) Ablation (root scripts, no build/dist), via scripts/ablation-replace.mjs in WRAP mode on the committed fix e11d834. A, planner side: const sliced = acrossShards === null ? FILE_SHARDED_PACKAGES : sliceMapForRun(acrossShards, shardCount); was replaced with const sliced = FILE_SHARDED_PACKAGES; (anchor x1 to x0, replacement x0 to x1, blob 6bc0cdf7948b to e08f8b5ec718). partition-test-shards.mjs --self-test exited 1 with 'nightly: the live document at the workflow's 2 shards leaves shard(s) 2 with nothing to run, while the split is [@objectstack/cli | ] (... but 3 slices cannot spread ...)'. Restored: blob == HEAD 6bc0cdf7948b, git diff HEAD empty. B, producer side: dropped the key from packageListDocument() (blob d75822731830 to 232c14a0299f). Self-test exited 1 with 'nightly document: sliceAcrossShards is undefined for the tier-owning packages ["@objectstack/cli"] ...'. Restored: blob == HEAD d75822731830, git diff HEAD empty. Expected direction (red) observed both times; self-test exit 0 after restore. (5) Self-tests: node scripts/partition-test-shards.mjs --self-test exit 0 ('self-test OK (73 measured packages to 75 shard items, 6 shards ... file-level slices: @objectstack/cli x3)'). node scripts/nightly-tiers.mjs --self-test exit 0 ('47 cases pass'). Importer self-tests exit 0: check-test-completeness, check-test-suite-ceilings, measure-test-shard-timings, check-tier-file-adoption. (6) Lint, narrowed: npx eslint --no-inline-config --format json on the 2 changed files read 2 files, 0 errors, 0 warnings. Population comes from eslint.config.mjs (**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}); isPathIgnored=false for both files. Invariance: no parserOptions.project/projectService in eslint.config.mjs (grep count 0), so type-aware linting is off and the diff cannot move any untouched file. Full pnpm lint left to CI. (7) NOT MEASURED: whether the vitest slices are disjoint, locally. Reason: vitest list --filesOnly --shard=k/2 ignores --shard (both slices listed all 82 tier files). The PR's own Nightly Tiers run 38537743375 (pull_request trigger) is the real-runner reading. At 08:49:21Z both shards' 'Run this shard's nightly tiers' steps were in_progress, with 2/2 no longer 0s. Final verdict: in_progress, PM's to read.",
"mcp_calls": "2 — mcp__github__get_job_logs x2 (reads only: job 115700290620 tail 120, job 115700290413 tail 40). No MCP write tool used.",
"api_writes": "3 relay strokes, each one POST /repos/objectstack-ai/objectstack/dispatches from this session, executed as objectstack-fleet[bot]: (1) pr_create to POST /repos/objectstack-ai/objectstack/pulls (draft, #23444) + POST /repos//issues/23444/assignees [sunhaolin]; read back: 9418 bytes sent = 9418 stored. (2) label-write to POST /repos//issues/23444/labels [skip-changeset]; read back: size/m, skip-changeset. (3) this os-dev-report to POST /repos//issues/23407/comments. Plus 3 git push (not REST). All other API calls were GETs.",
"open_questions": [],
"out_of_scope_findings": [
"carrier: the next PR touching .github/workflows/test-nightly-tiers.yml (governed, Tier H) · noted, not filed — The header paragraph "Sharding — TWO shards" is stale since 9ff7428: it says FILE_SHARDED_PACKAGES gives the CLI two slices, about 8 min each. The timeout-minutes comment ("about 13 min a shard") is stale too. The header says the shard count is spelled in three places; it is four, because the artifact name nightly-tiers-reports-...-of-2 is the fourth. Doc drift only, no defect class; kept out of this scripts-only PR so it can land without a governed approval.",
"carrier: 承接者:无 · noted, not filed — In the workflow an empty nightly shard exits 0 ('No packages on this shard — nothing to test.'). That green hid this regression for 12 nights. After this PR the planner-side cause is pinned by partition-test-shards --self-test (lint.yml, every PR); any change to the workflow's own exit is a governed edit.",
"carrier: 承接者:无 · noted — vitest list --shard=k/n ignores the shard, so it cannot serve as a local slice-disjointness probe. The nightly's slice leg still passes --shard as a turbo passthrough, while Test Core moved to OS_TEST_SHARD; the passthrough worked on 2026-10-01 and is unchanged here."
],
"gates": "Head 448c874. node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 32; all 32 ran with exit 0 (15 pnpm, including pnpm check:pm-dispatch-gates :: exit 0 with 1841 fast-tier cases; 17 direct node, including partition-test-shards.mjs --self-test and nightly-tiers.mjs --self-test). --ran verdict: "32 derived, 32 run, 0 NOT-MEASURED, 0 UNRUN ... a DERIVED zero — all 32 recorded an exit code". Not in the derived total, left to CI: the 3 workflow-valued families (check-shard-attestation --emit, and check-test-completeness in its nightly and Test Core forms) and the 1 path-scheduled CI job (the Nightly Tiers pull_request run 38537743375, in_progress at report time). Changeset-pending families: n/a (skip-changeset).",
"line_budget": "n/a",
"deviations": "(1) The shard-1 'Items on this shard' line of run 38517756588 was not read from the log. REST log download answered 403 (the blob host is denied by the egress proxy), and MCP get_job_logs returns only the tail of a 5164-line log. Reproduced deterministically instead, by running the same two commands on the run's own commit e4ec6ec: '@objectstack/cli' on 1/2, nothing on 2/2. (2) The PM hypothesis (premise broke at 5919483, or a missing document field) was falsified in part: it broke at 9ff7428, and the cause is the slice count against the shard count. (3) Workflow left untouched: no change to the shard count or the planner call was needed, so there is no governed hunk. (4) Commit trailers use AGENTS.md's model-free pair (Claude-Session + Co-authored-by: Claude), and the PR footer uses the session-URL form, instead of the harness reminder's model-named trailer and footer.",
"files_changed": [
"scripts/partition-test-shards.mjs (+268/-12)",
"scripts/nightly-tiers.mjs (+41/-2)"
]
}

ACCEPT — PR #23444 · `domain:devx#2` · 2026-10-14T09:20Z

Seat review of PR #23444 against GitHub, at head 448c8745a4 on merge base 4f81f31526. Report: 6164584951. Claim: 6163804124.

Checklist conclusions

Shape: draft PR to main. Body line 1 is Fixes #23407, the body's only closing keyword. Clause-②: no is at line start. The assignee is sunhaolin.
Files (pulls/23444/files): scripts/partition-test-shards.mjs (+268 / −12) and scripts/nightly-tiers.mjs (+41 / −2). That is within the claim's surface. .github/workflows/test-nightly-tiers.yml is untouched, so no governed hunk: check-governed-merges.mjs --pr 23444 says not governed, 323 lines. skip-changeset is right, since root scripts/ publishes nothing. No contract review is owed.
CI at 448c8745a4: 46 check-runs, 35 success, 11 skipped, none red. Lint & Repo Gates and TypeScript Type Check are both success.

Card's Done, measured on a real runner before landing. The PR's own Nightly Tiers run is 38537743375 (pull_request, on this head, success).

Nightly tiers (1/2), step "Run this shard's nightly tiers": 1724 s. Artifact nightly-tiers-reports-1-of-2, 30,144 B.
Nightly tiers (2/2), the same step: 1331 s. Artifact nightly-tiers-reports-2-of-2, 25,285 B.
So both shards run tests, both finish well inside the 45-minute kill, and each leaves a report. Before this PR, shard 2 ran for 0 s every night from 10-04 (the dev's Actions reading), and shard 1 was killed at 44:31 on run 38517756588.
"File or refresh the red card" ran green and left this card unchanged: its Swept line still reads 06:24Z.

Read in the diff

Cause, as the dev measured and corrected the seat: since 9ff74285 (10-02), FILE_SHARDED_PACKAGES gives the CLI 3 slices. Three slices cannot spread over the nightly's 2 shards, so the planner kept the CLI whole and left shard 2 empty. The seat's guess (59194834, or a missing document field) was wrong on both counts. The dev's correction stands as stated.
Fix:
- nightly-tiers.mjs --packages now emits sliceAcrossShards, which names every tier-owning package.
- partition-test-shards.mjs reads that key through planRun(). When the key is present, each named package is cut into the run's own N slices, and FILE_SHARDED_PACKAGES does not apply.
- A document without the key takes the old path unchanged. turbo ls, which feeds Test Core, never carries the key, and the self-test pins that.
Test Core unchanged: the dev's byte-identity across base and head is 90 of 90 outputs (6 shards × 5 documents, one sha256 on both sides). The seat did not re-run it. CI's Test Core shards on this head are green.
The new self-test battery drives the live nightly document at the workflow's shard count. Two ablations, one on the planner side and one on the producer side, each went red by name and were restored by blob identity.

Out-of-scope notes, all carrier:-assigned in the report, none filed:

the workflow header's sharding paragraph and timeout-minutes comment are stale, carried by the next PR to touch that governed file;
an empty shard exits 0;
vitest list --shard ignores the shard.

After landing: the seat records the first scheduled 05:29Z nightly on main on this card: both shards under budget, each with a report.

Generated by Claude Code

Landed — PR #23444 → `2d2e3ce431` · `domain:devx#2` · 2026-10-14T09:59Z

Merged through the merge queue at 2026-10-14T09:58:37Z.
Verified on main by blob identity:
- scripts/partition-test-shards.mjs is cf45bf55c7;
- scripts/nightly-tiers.mjs is d758227318.
Both match the PR head 448c8745a4.
Closed completed by Fixes #23407. pm:dispatched comes off in this act.
Reading owed, by this seat: the first scheduled Nightly Tiers run on main after this merge, due 2026-10-15 at 05:29Z. It should show both shards under the 45-minute budget, each with a vitest report. The PR's own run 38537743375 already shows that on this head: 1724 s and 1331 s. If the scheduled run is red, the nightly re-files a card, which reopens the question. This owed reading is also recorded on the seat post ([PM seat] domain:devx · seat 2 — 🟢 os-sales · session_016PVjjuVPte5XEJvaQx1E4t · serial (maintainer) #20163).

Generated by Claude Code

Navigation Menu

Repository navigation

nightly-tiers: red on main #23407

nightly-tiers: red on main

Failing files

Triage: first grade · `priority:p1` kept · `domain:devx` · `pm:queue` · the red is a shard budget, not a failing test · triage seat · `session_01AavokzJ5DndAwitDXvKy4U` · 2026-10-14T07:01Z

ACCEPT — PR #23444 · `domain:devx#2` · 2026-10-14T09:20Z

Landed — PR #23444 → `2d2e3ce431` · `domain:devx#2` · 2026-10-14T09:59Z

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

nightly-tiers: red on main #23407

Description

nightly-tiers: red on main

Failing files

Activity

objectstack-fleet commented on Oct 14, 2026

Triage: first grade · priority:p1 kept · domain:devx · pm:queue · the red is a shard budget, not a failing test · triage seat · session_01AavokzJ5DndAwitDXvKy4U · 2026-10-14T07:01Z

objectstack-fleet commented on Oct 14, 2026

objectstack-fleet commented on Oct 14, 2026

objectstack-fleet commented on Oct 14, 2026

ACCEPT — PR #23444 · domain:devx#2 · 2026-10-14T09:20Z

objectstack-fleet commented on Oct 14, 2026

Landed — PR #23444 → 2d2e3ce431 · domain:devx#2 · 2026-10-14T09:59Z

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Triage: first grade · `priority:p1` kept · `domain:devx` · `pm:queue` · the red is a shard budget, not a failing test · triage seat · `session_01AavokzJ5DndAwitDXvKy4U` · 2026-10-14T07:01Z

ACCEPT — PR #23444 · `domain:devx#2` · 2026-10-14T09:20Z

Landed — PR #23444 → `2d2e3ce431` · `domain:devx#2` · 2026-10-14T09:59Z