Repository navigation
[Decision] CI 的 Test (shard 1/8) 又逼近 20 分钟上限,并已超时取消一次:分片怎么改才不再复发 #12208
Description
Protects: this repository's required Test context, the published check-run name every PR and merge group of @object-ui/* passes through; this card keeps the name and keeps its shards under the cap.
Ruled: 6121816480 · letter B (staged) · 2026-10-12T09:49Z
Q2: 否 · the maintainer's word verbatim 「同意」
Blocked-by: #12214
立卡门类别:② 只有维护者能做的决定,兼 ③ 维护者直派。维护者在 domain:ui 座位 1 的会话(session_016djJF12Qt14ejKR5Vjt4sK)里的原话:「这个开决策卡优化,避免以后遇到类似的问题:这不是本 PR 的问题:超时前的测试全部通过,被打断的是和本 PR 无关的 plugin-form 测试。」
- 动手的读者: 裁定之后,改动落在
domain:devx@ objectui 车道,即.github/workflows/ci.yml和 vitest 配置。该车道座位目前空缺(objectui#5748),所以由分诊或总监席按裁定派发。Q2 若选「是」,改动落在 objectstack 的scripts/pm/fleet-write/,由那边的席位承接。 - 查重: 两次 objectui issue 搜索(含 closed)。① "Test shard exceeded maximum execution time 20m timeout flaky signature ledger" 命中 2 条;② "Test (shard 1/8) cancelled timeout-minutes 20 runs close to ceiling" 命中同样 2 条:objectui#9499(closed,本类问题的 4 分片版本)和 objectui#6640(closed,无关)。没有开着的同题卡。
- 立卡席位:
domain:ui@ objectui 座位 1(座位贴 objectui#5560)。
一句话问题
一个只改了一行 import 的 PR,测试全部通过,却因为第 1 个测试分片跑满 20 分钟被系统取消,被判为 CI 失败,卡在合并前,只能等有权限的人手动点重跑。这个分片平时就跑到 17 分钟以上,而测试总量还在快速增长,不改的话这种情况会越来越频繁。
背景
- 这次的实例是 PR objectui#12196(objectui#12184,一个模块改名)。 head
5a2a86c306上,Test (shard 1/8)跑了 20m16s,被 "The job has exceeded the maximum execution time of 20m0s" 取消(job114699823457,run38215707699)。- 被打断时它正在跑
packages/plugin-form的测试,此前跑过的文件全部 ✓。 - 汇总检查
Test把cancelled读成failure。座位的说明评论是6119717818。
- 被打断时它正在跑
- 座位没有重跑手段。 车队写入通道 relay 按设计拒绝 run 的重跑(
scripts/pm/fleet-write/ops.mjs的 "What is refused BY CONSTRUCTION" 一节:"a run cancel / re-run / delete")。所以每次都要人来点 "Re-run failed jobs"。 - 这是 objectui#9499 的同一类问题,第二次出现。 2026-09-15 的裁决(objectui#9499
5682566291,字母 C)做了三件事:加了单一的Test汇总检查;分片从 4 加到 8;dist pins 拆成独立 job。PR objectui#9584 于 2026-09-20 合并,+1118/−103,16 个文件。三周后,8 分片里最重的那片又回到了上限边缘。
Governing text
ci.yml里 type-check job 的注释:"⛔ Raisingtimeout-minutes: 20is not the fix and was ruled out on the card: a larger ceiling only buys a longer hang, still ending incancelled, and lifting a gate's ceiling weakens the gate."ci.yml里 test job 的注释(记录的是 objectui#9499 的裁决):"⛔ The ceiling is NOT raised …, ⛔ no test is skipped, disabled or quarantined, and ⛔ the relevance gate is not widened."- 所以「提高上限」「跳过或隔离慢测试」「放宽相关性门」都与现行裁决冲突,不作为本卡选项。另外,PR 只跑受影响包的做法不能单独解决问题,因为合并队列(
merge_group)始终跑全量,而合并队列上的最长分片也已到 1087 秒。
协议声明
不涉及 spec 或元数据协议,不改协议。
前提(每条附复核命令)
-
P1 · 第 1 片一直是最重的,而且贴着上限。
- 读数取自 CI workflow(id
223145786)最近 40 次pull_request和merge_group运行,时间窗 2026-10-11T22:38Z 至 2026-10-12T07:24Z。
分片 中位数(秒) 最大(秒) 1 1038 1216(超时那次) 2 904 999 3 829 967 4 934 1084 5 863 1017 6 768 981 7 855 1026 8 770 959 - 第 1 片的 p90 是 1070 秒。8 片中位数的平均值是 870 秒,第 1 片比它高约 19%。
- 按事件分:
pull_request25 次,中位 1047、最大 1216;merge_group15 次,中位 938、最大 1087。 - 时间几乎全花在 "Run tests" 一步(run
38215231563里 1038 秒),安装等准备步骤约 50 秒。 - 复核:
gh api "repos/objectstack-ai/objectui/actions/workflows/223145786/runs?per_page=60",对每个 run 再读/jobs,按Test (shard N/8)统计completed_at − started_at。
- 读数取自 CI workflow(id
-
P2 · 增长很快。
merge_group运行里的最长分片:2026-09-22 抽样 608 秒;2026-09-29 六次 783–830 秒;2026-10-05 六次 790–936 秒;现在中位 938 秒。也就是 20 天涨了约 54%,平均每天约 +16 秒。- 复核:同上的 API,加
&event=merge_group&created=<日期>。
-
P3 · 分片是按文件路径哈希做的,不按耗时。
- 依据是
ci.yml的注释:"Vitest shards by hashing each test file's path …"。所以哪一片偏重是随机的,文件越多越会漂移。 - 复核:
git grep -n "Vitest shards by hashing" -- .github/workflows/ci.yml。
- 依据是
-
P4 · vitest 允许自定义 sequencer 的
shard()(选项 B 依赖这一点)。⚠️ 未验证:本座位的检出里没有安装 vitest(仓库用^4.1.10)。- 复核:装好依赖后执行
grep -rn "shard(" node_modules/vitest/dist/*.d.ts,确认BaseSequencer带有shard方法。不成立就走回退项。
- 复核:装好依赖后执行
-
P5 · relay 不能重跑。
- 复核:
git grep -n "re-run" -- scripts/pm/fleet-write/ops.mjs(objectstack 仓库)。
- 复核:
具体问题
- Q1(防止): 用什么办法让测试分片在测试量持续增长时不再撞上 20 分钟上限?
- Q2(兜底): 要不要给座位一条带围栏的「重跑超时分片」能力?
选项 × 真实代价 × 工作量
Q1
| 选项 | 做什么 | 客户可感知的后果 | 工作量 |
|---|---|---|---|
| A. 只加宽 | 矩阵从 8 片加到 12 片,只改宽度。 | 立刻缓解。按比例估算,最重的一片约 690 秒(未实测)。按 P2 的增速大约 6 周后回到上限,到时要再加一次。每次运行多 4 台 runner,每台约 50 秒的准备开销。 | domain:devx · 1 个 PR。先例:objectui#9584 做过 4→8(连同汇总检查和 dist pins,+1118/−103);只改宽度这部分是一行加注释。 |
| B. 按实测耗时均衡分片,宽度跟着数据走 | 自定义 vitest sequencer 的 shard(),按一份每文件耗时数据装箱,不再按哈希平均分。宽度按「每片目标约 10 分钟」由总耗时算出,现在约 6961 秒,对应 12 片。耗时数据定期从 merge_group 运行刷新。 |
各片接近平均值,不再有一片单独偏重。测试量增长时,改的是一个由数据算出的数字,不用每次重新立卡。 | domain:devx · 2 个 PR:先加宽,再做 sequencer、数据和刷新。objectui 内无先例。姊妹先例在 objectstack:scripts/partition-test-shards.mjs 加 scripts/test-shard-timings.json 加 shard-timings-refresh.yml(objectstack#4859、#10472),按包分片;在 objectstack HEAD 上,连同测量与选择脚本共 5 个文件、7627 行。objectui 只需按文件分,预计更轻,但未实测。 |
| C. 只做兜底,不动分片 | 只做 Q2 的「是」。 | 超时照样发生,每次多等一轮 20 分钟,但座位能自己恢复。 | 见 Q2。 |
Q2
| 选项 | 做什么 | 后果 | 工作量 |
|---|---|---|---|
| 是 | 在 relay 的操作表加一行,围栏四道:只限 objectui 的 CI workflow;只限失败 job 全部是带 "maximum execution time" 注释的 cancelled Test (shard N/8);每个 run 只能用一次;不用于 merge_group。 |
座位可以不等人就恢复超时。但车队 token 会多出 actions: write,这是权限面。 |
objectstack · 1 个 PR。先例:同一个文件里的 workflow_dispatch 行就是这样围起来的。 |
| 否 | 保持现状:超时之后由人点重跑。 | 罕见时要等人。 | 0 |
业务含义直译
- A: 就像仓库爆满时再加一排货架,够用一阵子,满了再加。
- B: 按每件货的实际大小装车,车的数量由货的总量算出来。货再多,也只是多派几辆车,不用每次重新规划。
- C / Q2「是」: 不解决装不下,但给司机一把备用钥匙,卡住了能自己再开一次。
四轴(从业务立场)
- 项目长远合理性:
- 测试量随每张卡的 pin 持续增长,20 天涨 54%。A 是三周前做过的同一个补丁,还会第三次、第四次出现。
- B 把「分片超时」变成一个由数据决定的常量(每片目标时长),是主流做法。参照:CircleCI 的
tests split --split-by=timings、Knapsack Pro,以及我们自己 objectstack 的partition-test-shards.mjs。
- 实际业务拉动:
- 今天就撞上了:objectui#12196 测试全绿却卡着,要等人工重跑。
- 2026-09 在 objectui#9499 期间,同一类问题多次把已经通过的 PR 踢出合并队列,每次阻塞整条队列约 20 分钟。
- 防 AI 犯错(出错时谁看到什么):
- B 的耗时数据过期时,新文件按中位数估,结果是不够均衡,但仍可见,不会静默。
- Q2「是」属于容错:一个真实变慢的分片,可能被一次重跑掩盖成绿。所以 Q2 荐「否」。
- 创业阶段不扩散:
- B 新增一个 sequencer 和一份数据文件(新零件),但不新增门禁;A 零新零件。
- Q2「是」新增权限面,默认从紧。
推荐
- 只看长远选 Q1 = B,Q2 = 否。
- 两年后这个平台的 CI 应该是:按实测耗时分片,宽度由总量自动推出,不会有一片因为哈希运气不好单独超时。这和 objectstack 已经在用的形态一致。
- 分期(只调时序,不换字母): B 的第一个 PR 就是把宽度从 8 改到 12,当天可落,立刻解除当前风险;第二个 PR 做耗时均衡与刷新。
- 回退: 如果 P4 不成立(vitest 4 的自定义 sequencer 不能接管
shard()),退到 A,并在Test汇总的摘要里输出最长分片的余量,作为只报告的读数(⛔ 不是新门禁),让下一次加宽有数据可依。 - 置信缺口:
- P4 未验证。
- 12 片后的实际时长是按比例估的,未实测。
- PR 事件的运行比
merge_group慢约 100 秒,原因(runner 争用还是别的)未查。
- 自检:「只看①选 B;②③④ 是否翻转:否。」B 的工作量明显大于 A(2 个 PR 加数据刷新,对 1 个 PR),所以把 A 的那一步排在 B 的第一个 PR 里先落。
裁后执行
- 选 B:
- 本卡转入
pm:queue,由分诊定domain:devx。 - PR-1:只改
ci.yml的矩阵宽度(8→12)和对应注释,并在一次merge_group运行上验证。 - PR-2:sequencer、每文件耗时数据和刷新通道。验收是连续 10 次
merge_group运行里,最长分片不超过上限的 60%(720 秒)。Test汇总输出最长分片余量,只报告。
- 本卡转入
- 选 A: 只做 PR-1,加上这行余量读数。
- Q2 选「是」: objectstack 那边另派一个 PR,按上表的四道围栏加一行操作,由 objectstack 的 PM 工具席位承接。
- 当前的 PR objectui#12196: 不等本卡,仍需要有人在 run
38215707699上点一次 "Re-run failed jobs"。
os-decision-facets
- ① 项目长远合理性:B 让分片时长由数据决定,缩小特例;A 是第二次同样的补丁,问题会再回来。
- ② 实际业务拉动:今天 objectui#12196 测试全绿却被卡;objectui#9499 期间同类问题多次把已通过的 PR 踢出合并队列。
- ③ 防 AI 犯错:B 的数据过期时退化成可见的不均衡,不静默;Q2「是」的自动重跑会掩盖真实变慢,故荐否。
- ④ 创业阶段不扩散:B 新增 sequencer 和数据文件,但不新增门禁;Q2「是」新增权限面,默认从紧。
Prior rulings read: shard, timeout-minutes, ceiling, cancelled, merge queue → 5 hits (objectui#9499 ruling 5682566291 letter C; objectui#9584; the two ci.yml notes quoted above; objectui#6640 unrelated); ADR none; thread: none
相关单与 PR
objectui#9499(closed,裁决 5682566291)· PR objectui#9584(4→8 分片与 Test 汇总)· PR objectui#12196(本次实例,说明评论 6119717818)· objectui#12184 · objectstack-ai/objectstack#4859、objectstack-ai/objectstack#10472(objectstack 的耗时均衡分片先例)· objectui#5748(domain:devx 座位贴)
Generated by Claude Code
Activity
objectstack-fleet commentedon Oct 12, 2026
Triage: decision-box entry check, with one completion. The 「维护者速读」 block is added, and Q2 is folded so the question is one letter
Triage seat (seat post objectstack-ai/objectstack#6015) ·
session_01AavokzJ5DndAwitDXvKy4U· 2026-10-12T08:08Z. ⛔ Not a claim, ⛔ no letter chosen.actions: writeto the fleet token, a permission widening that defaults to no, and the card recommends "no". So Q2 rides as "no" under A and B, and Q2 "yes" is letter C.domain:devx@ objectui (its seat is vacant, objectui#5748). The dispatch follows the ruling.维护者速读
objectui 的 CI 第 1 个测试分片平时就跑到 17 分钟以上,已经有一个测试全绿的 PR 因为跑满 20 分钟被取消,只能等人手动重跑。测试量 20 天涨了约 54%,不改会越来越频繁。三条路:
席位推荐 B,其次 A。
选 A、B 还是 C?
os-decision-facets
Prior rulings read: as listed on this card (objectui#9499 C, and
ci.yml's "⛔ the ceiling is NOT raised" notes).objectstack-fleet commentedon Oct 12, 2026
Ruling: batch #328 item 1 · letter B (staged) · Q2 否 · maintainer 「同意」 2026-10-12T09:49Z
Director seat, summon #37,
session_01LqNgdPZB2Jnr5HpGbaqLeB(GitHubos-zhuang; written asobjectstack-fleet[bot]via the relay). The maintainer asked for this card on the direct channel (「这个你也帮我分析,一起接管」); this seat measured the shards itself, presented the card's three letters with its own reading and the recommendation "Q1 = B staged, Q2 = 否, PR-1 folded into objectui#12214's in-flight PR, PR-2 after it lands", and the maintainer answered 「同意」.Premises re-read in this act (300
CIruns, 1,728Test (shard n/8)jobs over 7 days, the jobs API): shard wall median 14.4 min, p90 17.0, p95 17.4, max 20.3 (the one cancellation, run 38215707699, shard 1); 13 % of headroom at p95; the card's growth reading (+54 % in 20 days) is not re-measured here. One premise of the card changed since it was written: under objectui#12214 (claimed, dev in flight; the doctrine objectstack#22998)pull_requestandmerge_groupruns carry only the affected packages' tests, so "the merge queue always runs the whole suite" no longer holds after it lands; the whole-suite run moves tomain'spushand hourlyschedule, where this card's cap problem would otherwise reappear as a main-red. B therefore serves the full-battery runs, not every PR.The ruling
Q1 → B, staged.
testmatrix goes from 8 to 12 shards with its comment updated; every pin of the shard count (scripts/check-test-shard-results.mjs --shards …in theTestaggregator, and any other) moves with it; the coverage matrix stays 4. This is a mechanical change to the same file and job the 12214 dev is editing, so it rides that PR as one commit (Refs objectui#12208 (PR-1)), on the fold rule's five gates: same defect shape (test wall against a fixed cap), same file and job, both ruled, independently checkable (the shard count), nothing else folded.ci.yml): shards balanced by measured per-file duration (a vitest sequencershard()that bin-packs from a per-file timing file; if vitest 4's sequencer cannot takeshard(), the fallback the card names: width only, plus the headroom reading), the width derived from the total measured duration at a target of about 10 minutes per shard, the timing file refreshed frommain's full-suite runs on a schedule. TheTestaggregator prints the longest shard's headroom against the cap as a report-only reading (⛔ not a gate; the doctrine's timing-gate rule). Acceptance as the card states: ten consecutive full-suite runs with the longest shard under 60 % of the cap.timeout-minutes(ruled out on the card and inci.yml's own comments), skipping or quarantining a test, widening the relevance gate.Q2 → 否. No fleet-token
actions: writefor re-running timed-out shards: a permission widening, and a re-run can mask a shard that really got slower. A timed-out shard stays a human re-run until PR-1 and objectui#12214 remove the cause.os-decision-facets
Prior rulings read: objectui#9499 C (5682566291),
ci.yml's "⛔ the ceiling is NOT raised" notes, objectstack#22998 (two-speed CI; timing gates report-only; test budgets), objectui#12214 (claim 6121064652), objectstack#22949 C2; thread: 6120293503 (triage's fold of Q2 into the letters).After this ruling
needs-user-decision→pm:blocked,Blocked-by: objectstack-ai/objectui#12214(PR-2 waits for that PR; PR-1 rides it). Lanedomain:devx@ objectui; this seat dispatches PR-2 on the maintainer's standing order 「三仓的 ci 优化卡你负责派发」 when the block clears.objectstack-fleet commentedon Oct 12, 2026
Triage seat (seat post objectstack-ai/objectstack#6015) ·
session_01AavokzJ5DndAwitDXvKy4U· 2026-10-12T13:10Z. ⛔ Not a claim, ⛔ not a dispatch. Unlock:pm:blocked→pm:queue.Blocked-by: objectstack-ai/objectui#12214is closed (completed), and PR-1 rode it as the ruling6121816480set out.objectstack-fleet commentedon Oct 12, 2026
Claim: PM loop round 1 · the director seat dispatching on objectui#12208's ruling (6121816480: "this seat dispatches PR-2 when the block clears") and the maintainer's standing order 「三仓的 ci 优化卡你负责派发」 (this seat's session, 2026-10-12T14:18Z)
Session:
session_01LqNgdPZB2Jnr5HpGbaqLeBAccount:
os-zhuangBranch:
claude/issue-12208-duration-balanced-shardsWorktree:
objectui-issue-12208Domain:
domain:devxSeat:
domain:devx#9File surface:
.github/workflows/ci.yml(thetestandtest-coveragejobs and theTest/Test (coverage)aggregators only: matrix widths, the sequencer wiring, the headroom reading),vitest.config.mts(the sequencer hook), a newscripts/test-shard-sequencer.*and its test, a new per-file timing file underscripts/and the script that refreshes it frommain's full-suite runs (plus itsschedulewiring inci.ymlor a sibling workflow),scripts/check-test-shard-results.mjsand every pin of the shard counts; ⛔ not thetype-checkjob (objectui#12222, in flight), ⛔ notspec-main-shape-gate.yml(objectui#12252), ⛔ notimeout-minutesraise, ⛔ no skipped or quarantined test, ⛔ no change tocoverage.thresholds(stop on breach; explain in the report)Container & model:
L,mode:subagent,model: opus(the default judgment tier: a sequencer and a width rule are design work). Tier H posture:.github/workflows/**(the round's posture; objectui#12255 is queued for the table row): the PR stays a draft until aGOVERNED_APPROVERSmember approves it.Clause-②: no
Responsibility:
Test (coverage)onmain's full-battery runs (objectui#12256, the main-red anchor folded here, fold note 6126024274: a coverage shard at its 40-minute ceiling on 7 of 25pushruns) | the shard walls for both matrices | thedomain:devx@ objectui lane's PRs, which do not arm the queue while the red standsThread-read: 6121816480
Ruling-ref: objectui#12208 ruling 6121816480 (Q1 → B staged, the maintainer's word verbatim 「同意」; PR-1 landed in PR #12232); objectui#12256 triage 6125671110 (the same remedy for the coverage lane); objectstack#22998 (timing readings report-only; test budgets)
Serial constraints cleared: PR #12232 (objectui#12214, PR-1) is MERGED (
21fdeb34); objectui#12222 (in flight) edits thetype-checkjob ofci.ymlandturbo.json, disjoint hunks from this claim's two jobs, and whichever lands second mergesorigin/mainfirst; objectui#12252 (claimed, not started) editsspec-main-shape-gate.ymlonly. Mergeorigin/mainbefore opening the PR.objectstack-fleet commentedon Oct 12, 2026
os-dev-report$TURBO_ROOT$ /scripts/test-shard-*)",
{
"issue": 12208,
"status": "done",
"branch": "claude/issue-12208-duration-balanced-shards",
"pr": "#12263",
"session": "session_01LqNgdPZB2Jnr5HpGbaqLeB",
"premise_still_valid": true,
"summary": "Draft PR #12263 (head 0e1a41c, mergeable clean, assignee os-zhuang) carries both halves. (1) First commit 4b3daf9, which opened the PR: the coverage matrix goes from 4 to 8. Every spelling of the width moves with it. The job name, the step name and --shard=N/8 change in test-coverage. coverage-report now reads one job variable, COVERAGE_SHARDS '8', for both the blob glob and the count it refuses to go below. Dependabot gate names, the census regex, the doc rows and the comments move too, and a new pin, coverage-shard-width.test.ts, holds every spelling to the matrix. timeout-minutes 40, the tests and coverage.thresholds are unchanged. (2) PR-2 proper (5192aa3, 9b1bea3, 0e1a41c): a vitest sequencer (scripts/test-shard-sequencer.mjs, root sequence.sequencer) whose shard() bin-packs files longest-first from a committed per-file dataset (scripts/test-shard-timings.json, keyed by a path hash). A per-file timing reporter runs on the full-battery test legs (scripts/test-file-timings.mjs; the push and schedule legs upload test-file-timings-N). A weekly refresh workflow (test-shard-timings-refresh.yml) regenerates the dataset from main's runs and opens a PR. Both aggregators print a report-only headroom reading (scripts/shard-headroom.mjs, continue-on-error, exit 0 whatever it reads). Main-red objectui#12256 stops at the first full-battery run after landing if H1 holds; that is NOT MEASURED until then.",
"widths": {
"test-coverage": "4 to 8, derived. On the full-battery push/schedule runs since PR-1 (21fdeb3), the 6 finished runs summed their 4 coverage job walls to 7458-9252 s, and their longest shard ran 2325-2364 s against the 2400 s cap. ceil(max total / 1200 s target) = ceil(7.7) = 8. 5 of those 11 runs were cancelled at the coverage cap.",
"test": "Kept at 12, the ruled number, but the derivation answers 13. Across the 11 main runs since PR-1, the total of the 12 job walls was 6759-8052 s (median about 7311). At the 600 s target that is 13 by the median and 14 by the max. The PR's own headroom reading printed '13 shard(s) (the matrix has 12)' on both runs where it ran. See open_questions.",
"measured_ratio": "Coverage vs test, from the same 6 finished main runs: the sum of 'Run tests' step walls is 1.09-1.41x, median 1.26 (job walls: median 1.20). The two lanes ran at different widths (4 vs 12), so per-shard fixed costs differ. The ratio enters only the width derivation. Both matrices bin-pack with the same per-file weights, and a uniform multiplier does not move a bin-packed partition."
},
"sequencer_decision_H2": "H2 holds on vitest 4.1.10. BaseSequencer has shard(files) and sort(files). The root config's sequence.sequencer is constructed once per run, and its shard() is called with every specification whenever --shard is set (pool executeTests, before sort). Implemented as an adapter extending BaseSequencer: shard() is replaced, sort() is inherited. The pure partition lives in scripts/test-shard-partition.mjs: greedy longest-first, deterministic tie-breaks (path, project, lowest leg), an unknown file weighed at the dataset median, and a 1 ms floor so every leg gets a file. A dataset holding no file falls back to super.shard(), byte for byte. Checked on vitest's REAL spec list (4979 specifications, node API): every leg's bin, at 12 and at 8, places each specification exactly once (missing 0, twice 0). H3 holds in part. The built-in json reporter's per-file span covers only the tests, so the writer reads TestModule.diagnostic() instead (environment + prepare + setup + collect + run), the cloud#2806 shape. One dataset serves both matrices. Refresh choice: a committed file plus a weekly PR, not an artifact the legs fetch. Every leg computes the whole plan and keeps its bin, so one run's legs must read the same weights; legs fetching 'the newest artifact' could straddle a refresh and run files twice or never, every leg green. A committed file also reproduces a CI leg on a laptop. The dataset sits in the root config's import closure, so the PR that moves it runs the full suite (checked with select-affected-tests: a dataset-only change is mode=full). Cost: the refresh PR is opened with GITHUB_TOKEN, so it has no CI until someone pushes to it once (objectui#5397 refused a PAT). The workflow header and the doc say so.",
"test_walls": {
"without_sequencer": "Hash split, empty dataset, PR runs (pull_request, mode=full, 12 legs). Run 38268236293 (a7f6018): 413-696 s, longest shard 11. Run 38269820609 (5874484): 414-773 s, longest shard 1. Main baseline, 11 push/schedule runs since PR-1: longest leg 697-796 s, 9 of 11 above 720 s.",
"with_sequencer": "Run 38271639400 (9b1bea3, path-keyed one-run dataset): 412-753 s, and shard 2 red (the sweep collision below). Run 38274458184 (0e1a41c, hashed two-run dataset): 425-731 s, all 12 green. The aggregator printed 'longest Test (shard 3/12): 12m11s = 61% of the 20-minute cap, 7m49s of headroom; total 122m49s over 12 shard(s); at the 10-minute target per shard that is 13 shard(s)'.",
"reading": "Two runs each way cannot show a balancing gain. On main, the same leg index varies by about 100 s (stdev per leg over 11 runs) on a near-identical file set. 84 of 132 legs ran 602-796 s and 48 ran 377-597 s, which looks like runner speed rather than file assignment. That variation is as large as the imbalance the hash split produced. The ten-run acceptance after landing is the measurement. Both balanced runs had a leg over 720 s, so at 12 legs that acceptance is unlikely to pass (open_questions). Per-file sums predicted their own run's leg walls tightly: leg wall / sum of file walls = 0.347-0.363 across the 12 legs of run 38269820609. One run's numbers carry each leg's runner speed into every file it measured, so the bootstrap uses the median of two runs and the refresh uses the median of up to six."
},
"tests": "All at HEAD 0e1a41c (git rev-parse --short HEAD), through os-verify-lock. (1) pnpm exec vitest run scripts/tests/ packages/types/src/tests/ai-insights-retired-8800.test.ts: 'Test Files 188 passed | 2 skipped (190)', 'Tests 5622 passed | 2 skipped (5624)', VERDICT command-exit 0. (2) pnpm type-check:scripts and pnpm type-check:vitest-config: exit 0. tsc --listFiles counts the 4 new test files in the scripts program. (3) eslint --format json over the 12 changed scripts and tests: 12 files, 0 errors. vitest.config.mts is outside eslint's configured population, so it is not linted. (4) Gates, node scripts/X.mjs, each exit 0: check-control-bytes, check-action-ref-convention, check-pre-install-import-graph (39 pre-install steps, every leaf a builtin, the refresh workflow's node-only scripts included), check-shell-escape-residue, check-doc-links, check-doc-fence-languages, check-new-cross-file-line-citations (0 new), check-changeset-presence (no changeset owed), check-test-path-roots. check-governed-queue-guard --test on ci.yml, the refresh workflow and vitest.config.mts: NOT GOVERNED. (5) YAML parses for both workflows. actionlint is not installed: NOT MEASURED. (6) Ablations through objectstack's ablation-replace.mjs, each restore verified (blob == HEAD, git diff HEAD empty). COVERAGE_SHARDS '8' set to '4': width pin red, 1 failed. Coverage matrix set to [1,2,3,4]: red, 3 of 5. sequence: line removed from vitest.config.mts: wiring pin red. Headroom --cap-minutes 20 set to 30: headroom pin red. Longest-first replaced by dealing in sort order: 1 red, the heavy-file placement pin; the two balance-bound tests stay green, because dealing in descending weight also satisfies them, so they pin balance, not longest-first. (7) Local 8-way coverage merge, ten script test files: 8 blobs (blob-k-8.json, k = 1-8), --merge-reports reads all 8 ('Test Files 10 passed (10)'); over the merged map a lines threshold of 70 exits 0, and 71 exits 1 with 'ERROR: Coverage for lines (70.57%) does not meet global threshold (71%)'. (8) CI on the PR head 0e1a41c: CI success (Type Check, Build & E2E, Build Docs, 12 test shards, dist pins, Test), plus Lint, Repo Gates, Changeset Presence, Line Citation Gate, Governed Surface Guard, Live E2E and Test Shard Timings Refresh, all success. The refresh's PR dry run printed its NOT MEASURED warning and stayed green, as designed before main has artifacts. Spec Main Shape Gate (advisory) was in_progress when read. NOT MEASURED: coverage walls at 8 shards (push/schedule only, read on the first full-battery run after landing); the refresh's real PR-opening leg (schedule or dispatch only); the 10-run acceptance.",
"gates": "scripts/tests suite + objectui#8800 pin, type-check:scripts, type-check:vitest-config, eslint over the changed scripts, check-control-bytes, check-action-ref-convention, check-pre-install-import-graph (with --self-test), check-shell-escape-residue, check-doc-links, check-doc-fence-languages, check-new-cross-file-line-citations, check-changeset-presence, check-test-path-roots, check-governed-queue-guard --test. All are green at 0e1a41c. check-test-shard-results reads only Test (shard N/M), never coverage, so its --shards stays 12. pnpm lint over the repo was not run locally (CI's Lint is green).",
"files_changed": [
".github/workflows/ci.yml (test-coverage 4 to 8; coverage-report COVERAGE_SHARDS, permissions, a sparse checkout and the headroom step; test: timing reporter, upload step, the sequencer note; test-aggregate: headroom step)",
".github/workflows/test-shard-timings-refresh.yml (new)",
"vitest.config.mts (sequence.sequencer)",
"turbo.json (one test input:
"scripts/test-shard-sequencer.mjs, scripts/test-shard-partition.mjs, scripts/test-file-timings.mjs, scripts/shard-headroom.mjs, scripts/test-shard-timings.json (new)",
"scripts/dependabot-merge-gate.mjs (8 coverage shard names; the refresh check as OPTIONAL)",
"scripts/coverage-red-cause-census.mjs (any-width regex; shardOf note)",
"scripts/tests/coverage-shard-width.test.ts, test-shard-partition.test.ts, test-file-timings.test.ts, shard-headroom.test.ts (new); coverage-red-cause-census.test.ts, coverage-shard-reporter-readability.test.ts (edited)",
"content/docs/guide/ci-cd-pipeline.md (coverage rows, test/test-aggregate rows, a new workflow section and inventory row)"
],
"deviations": [
"File surface additions beyond the claim's list: turbo.json and content/docs/guide/ci-cd-pipeline.md (each required by a pin), plus the partition, timings and headroom scripts. turbo.json gains one test input, because turbo-test-inputs.test.ts requires the root config's new relative imports in turbo's test inputs. That line is disjoint from objectui#12222's hunks, and it merged cleanly after PR #12251 landed. The doc changes are required by ci-cd-pipeline-doc.test.ts, which wants every command a job runs and every workflow file named.",
"Two TEMPORARY commits (5192aa3's notice step, and 5874484 splitting it into three steps) carried the first per-file measurement out as notice annotations. This container cannot download an artifact: the blob host answered 403 on artifact 11751093289. The runner kept 10 notices per step, so run 38268236293 was partial. 9b1bea3 removes the steps, and a squash leaves nothing of them. This cost two extra full PR runs, and a third run re-measured after the key change.",
"The dataset is keyed by sha1(path)[0:16], not by the path. The first path-keyed full run (38271639400) went red because objectui#8800's retired-spelling pin found its own file name in the dataset. Hashing closes the whole class, and a new pin asserts that no tracked test file name occurs in the dataset text. My own source comment named the spelling and the pin caught it locally; it was reworded.",
"Merged origin/main once (a7f6018), after objectui#12222's PR #12251 landed. The CI doc's job table conflicted on adjacent rows: main's type-check row was taken, and this branch's test row. Each side had changed only its own row (checked three ways against the base).",
"Commit trailers use the model-free pair per objectui AGENTS.md (its rule takes precedence over the harness reminder). The PR body carries line 3 'Clause-②: no', copied from the claim; nothing in objectui reads it.",
"The PR body was written once, at open, on commit 1. Its 'commits that follow' section says 'planned'. Corrections are below in pr_body_corrections for the seat; the dev does not PATCH the body.",
"Labels: zero writes. objectui labels are path-driven and the dispatch named none."
],
"pr_body_corrections": "Replace the planned-commits section with what landed. 5192aa3: sequencer, reporter, refresh, headroom. 9b1bea3: first dataset, temporary steps removed. 0e1a41c: hashed keys, two-run dataset. Add the test_walls readings and the bootstrap channel. Add the sweep-collision fix. In 维护者速读, add the finding that per-leg runner variation is about as large as the imbalance, and the test-width question (12 kept; the derivation says 13).",
"not_measured": [
"Coverage shard walls at 8: the lane runs on push/schedule only. Read on the first full-battery run after landing (H1).",
"Ten consecutive full-suite runs under target plus margin: after landing, read by the seat.",
"The refresh workflow's write leg (push the bot branch, gh pr create): it runs on schedule or dispatch only. Its PR leg ran green with NOT MEASURED, because no main run carries test-file-timings artifacts before landing.",
"actionlint: not installed in this container."
],
"mcp_calls": "0",
"api_writes": "2 — both through the fleet-write relay (POST /repos/objectstack-ai/objectstack/dispatches). (1) pr_create on objectstack-ai/objectui: draft, assignees os-zhuang, giving #12263; relay run 38264541514; read-back 5784 bytes identical. (2) This report comment on objectui#12208 plus a one-line pointer comment on objectui#12256, as one stroke of two actions. git push is not REST: the empty-branch probe, 4b3daf9, 5192aa3, a7f6018, 5874484, 9b1bea3, 0e1a41c.",
"open_questions": [
{
"question": "The test matrix width: keep the ruled 12, or move to the width the ruling's own rule derives (13 at the 10-minute target)?",
"options": [
"A keep 12. Cost: none now. The derivation answers 13. Both balanced PR runs had a leg over the card's 720 s acceptance bound (753 s, 731 s), as did 9 of 11 unbalanced main runs. The ten-run acceptance likely fails, and growth widens the gap.",
"B 13. One mechanical commit on this PR or a follow-up: ci.yml name and both --shard flags, test-aggregate --shards, the headroom and refresh --shards, the 13 Dependabot NOT_A_GATE names, doc and comments. One more runner per run (about 40 s setup each). Expected longest leg about 12/13 of today's.",
"C 14. The derivation at the max total (8052 s). Overshoots the median reading."
],
"recommendation": "B. Every reading of the ruled rule (width from the measured total at about 10 minutes per leg) answers 13: the main median, and the PR's own headroom line, twice. A cannot pass the card's acceptance as measured. Every full-battery run prints the implied width, so moving later to 14 is a data-driven edit and not a new card. Four axes, briefly. Long-term: the width follows the data, which is the ruling's B. Business pull: today's legs sit at 61-66% of the cap. AI error: report-only readings, no gate. Startup scope: one runner, no new part."
}
],
"out_of_scope_findings": [
"carrier: the next AGENTS.md edit (governed surface) · noted, not filed. AGENTS.md section 9's test recipe still gives '--shard=1/4' as its example (H5); the coverage lane is 8 and the test lane is 12, and the sequencer now decides membership.",
"carrier: whoever next edits .github/workflows/changeset-release.yml · noted, not filed. Two comments there still say the push lane ran 'coverage across four shards'.",
"carrier: whoever next edits scripts/tests/check-unused-dependencies.test.ts · noted, not filed. A comment there says 'on the merged 4-shard report'."
]
}
Generated by Claude Code
objectstack-fleet commentedon Oct 12, 2026
Review: ACCEPT · PR #12263 (draft, head
0e1a41c9) · director seat dispatching on the card's ruling (6121816480, B staged) and the objectui#12256 fold (6126024274), 2026-10-12T16:27ZChecked on GitHub and on the head's tree, not on the report's say-so:
main(merge-basee5c8ea7f;origin/mainmerged once ata7f6018aafter PR ci(type-check): the turbo cache replays unchanged packages, and pull-request and queue runs type-check only what the change affects (objectui#12222) #12251 landed), body lines 1–2Fixes #12208/Fixes #12256(the only closing keywords),Clause-②: no, assigneeos-zhuang; 18 files, +7,082/−79 (5,004 of them the committed dataset), six commits (4b3daf92the width,5192aa3athe sequencer + reporter + refresh + headroom,a7f6018athe merge,58744849and9b1bea38the temporary bootstrap steps in and out,0e1a41c9the hashed keys and the two-run dataset).ci.yml:test-coveragematrix[1..8], job nameTest (coverage shard N/8),--shard=N/8(line 1641),timeout-minutes: 40unchanged (1525);coverage-reportsays the width once asCOVERAGE_SHARDS: '8'and reads it for the blob glob and the count it refuses below (1734, 1792–1797); no literal 4 is left near the coverage lane;coverage.thresholdsuntouched (thevitest.config.mtsdiff is the sequencer import and thesequence:line only); the Dependabot gate names and the census regex moved; the width pincoverage-shard-width.test.tsis new. The local 8-way merge in the body shows the thresholds read the merged map (lines 70 passes, 71 fails).scripts/test-shard-sequencer.mjsextends vitest'sBaseSequencer, replacesshard()only and falls back tosuper.shard()on an empty dataset;test-shard-partition.mjsbin-packs longest-first with the median for an unknown file and a 1 ms floor;test-file-timings.mjsis the reporter on the full-battery legs (push/scheduleuploadtest-file-timings-N);scripts/test-shard-timings.jsonis keyed by a path hash (0 test-file names in its text, checked);turbo.jsongains one test input; both aggregators' headroom steps areif: always()+continue-on-error: true(1429–1434, 1864–1869; report-only, as ruled); the refresh workflow runs47 6 * * 1,workflow_dispatchand apull_requestdry run, and writes withGITHUB_TOKEN(no PAT; objectui#5397 kept, so the bot's PR needs one push before CI runs on it, said in the header and the doc).type-checkjob (lines 146–929) has no hunk; the twoon:hunks are comments. H2 held on vitest 4.1.10 (shard()called with every specification whenever--shardis set; every specification placed exactly once at 12 and at 8 over the real 4,979-file list). H3 held in part: the built-in json reporter's per-file span covers the tests only, so the reporter readsTestModule.diagnostic()(the cloud#2806 shape). The refresh choice (a committed file + a weekly PR, not an artifact the legs fetch) is accepted for the reason given: every leg must read the same weights, and legs fetching "the newest artifact" could straddle a refresh.0e1a41c9, read at this act: all green:Test(12 shards, dist pins, the aggregator),Lint,Repo Gates,Type Check,Build & E2E,Build Docs,Changeset Declaration,Changeset Claim Re-read,Changeset Fixed Group Check,Governed Surface Queue Guard,Line Citation Gate,Live E2E,Regenerate the per-file test timings(the refresh lane's dry run, NOT MEASURED by design beforemainhas artifacts),Spec Main Shape Gate(advisory);Test (coverage)skipped by design on a PR run. The dev's local union: 188 test files / 5,622 tests green under the verify lock;type-check:scriptsandtype-check:vitest-configexit 0; eslint over the 12 changed scripts 0 errors; ninecheck-*gates exit 0;check-governed-queue-guard --testNOT GOVERNED on the three paths; five ablations red in the predicted direction (restores proven).testlegs with the sequencer 425–731 s (run 38274458184, all green) and 412–753 s (run 38271639400, one leg red on the sweep collision, fixed by the hashed keys); without it 413–696 s and 414–773 s;main's baseline since PR-1: longest leg 697–796 s, 9 of 11 runs above 720 s. Two runs each way cannot show the balancing gain: the per-leg runner variation onmain(about 100 s stdev per leg index) is as large as the hash split's imbalance; the per-file sums predict a leg's own wall tightly (wall ÷ sum of file walls 0.347–0.363 across 12 legs). NOT MEASURED until landing: the coverage walls at 8 (the lane runs onpush/scheduleonly; the main-red stops at the first full-battery run if H1 holds), the ten-run acceptance, the refresh's write leg, actionlint..github/workflows/**and the size (7,082 lines > 3000) both say the maintainer approves and the seat lands. Contract review: none of the three faces touched → the seat's own reading, no second agent.turbo.jsonandcontent/docs/guide/ci-cd-pipeline.md(each required by a pin); the two temporary bootstrap commits, removed in9b1bea38(a squash leaves nothing; two extra PR runs were the cost of measuring from a container that cannot download artifacts); the hashed dataset keys after objectui#8800's retired-spelling pin found a file name in the first dataset (a new pin asserts no tracked test-file name occurs in the dataset text); the oneorigin/mainmerge with a two-row doc conflict resolved each side's own row; zero label writes (path-driven). Writes: 2 relay strokes (pr_create; the report 6128243551 with the pointer 6128244348 on objectui#12256);mcp_calls0.AGENTS.mdsection 9 still gives--shard=1/4(H5; governed, the lane's nextAGENTS.mdedit); two comments inchangeset-release.ymlsay "four shards"; a comment inscripts/__tests__/check-unused-dependencies.test.tssays "4-shard report". None filed by this seat.A decision for the maintainer, not a blocker to landing: the
testmatrix width. The ruled number is 12 and the PR keeps it; the ruling's own rule (width = the measured total at about 10 min per leg) answers 13 onmain's median (about 7,311 s over 12 legs) and on the PR's own headroom line twice, 14 at the max (8,052 s). Both balanced runs had a leg over the card's 720 s bound (753 s, 731 s), as did 9 of 11 unbalancedmainruns, so the ten-run acceptance at 12 is unlikely to pass as measured. The dev recommends B (13 now, one mechanical commit: the job name, both--shardflags,test-aggregate --shards, the headroom and refresh--shards, the 13 Dependabot names, the doc). The seat's recommendation: land as delivered now, then let the headroom line onmain's full-battery runs decide: if it keeps answering 13 (or 14) over the first runs, one mechanical follow-up commit moves the width, a data-driven edit under ruling B rather than a new card, and the acceptance in the landing record names the width it was read at. Four axes: long-term the width follows the data either way; business pull: landing now stops the coverage reds at the next full-battery run and unblocks the objectui lane's PRs waiting behind the main-red; AI error: the headroom reading is report-only; scope: at most one runner, on a public repository. A B-now ruling costs one more PR cycle (about 15 min) before landing; the maintainer's word decides.Landing: on the maintainer's Approve the seat readies the PR and arms the queue. After merge the seat reads the first full-battery run (the eight coverage walls against the 40-minute cap; both headroom lines) and then the ten-run acceptance; objectui#12256 closes with the PR's
Fixesline and its landing is recorded there. This card stayspm:dispatcheduntil the landing record.objectstack-fleet commentedon Oct 12, 2026
LANDED — PR #12263 is MERGED (2026-10-12T22:00Z, approved by the maintainer, review 5496197961 on head
0e1a41c9, readied and auto-merge armed by hand withmainmerged in (8937ca97); the queue merged it as1fb1ced3) and read in place onorigin/main1fb1ced3. Director seat, summon #37,session_01LqNgdPZB2Jnr5HpGbaqLeB, 2026-10-12T22:01Z.origin/mainci.ymltest-coverage: matrix[1..8], nameTest (coverage shard N/8),COVERAGE_SHARDS: '8',timeout-minutes: 40vitest.config.mts:sequence: { sequencer: TestShardSequencer }scripts/test-shard-sequencer.mjs,test-shard-partition.mjs,test-file-timings.mjs,shard-headroom.mjs,scripts/test-shard-timings.json(schemaobjectui.test-shard-timings/2),.github/workflows/test-shard-timings-refresh.ymlreport-only,continue-on-error: true)type-checkjob untouched by this merge (hunks against7af0ab37are comments)Fixesline (closed_by_pull_requestslists PR ci(test): the coverage lane runs eight shards, then both test matrices shard by measured duration (objectui#12208, objectui#12256) #12263 alone), as was objectui#12256 (its own landing note is on that card);pm:dispatchedstripped and the assignee released in this act. Thetestmatrix stays at the ruled 12: the maintainer approved the PR as delivered; the width follows the headroom line onmain(ACCEPT 6128336930).pushrun onmainafter this merge (1fb1ced3, in progress at this act): the eight coverage shards' walls against the 40-minute cap, both headroom lines (the longest leg and the width the measured total implies), thetest-file-timings-Nartifacts; then ten consecutive full-suite runs with the longesttestleg under 720 s, or the data-driven width edit if the headroom line keeps answering 13. The refresh workflow's first scheduled PR (Monday 06:47 UTC) needs one push from the seat before CI runs on it (objectui#5397's posture).AGENTS.mdsection 9 still gives--shard=1/4; two comments inchangeset-release.ymland one inscripts/__tests__/check-unused-dependencies.test.tsstill say four coverage shards.objectstack-fleet commentedon Oct 12, 2026
Reading 1 on objectui#12208 (the first full-battery
pushrun after1fb1ced3: run 38314070610, 22:00–22:23Z, conclusion success), read on the jobs API and the aggregators' notices by the director seat, 2026-10-12T22:34Z.Test(12, the sequencer's first run onmain)Test (coverage)(8, the first run at eight)testleg 697 s, under the 720 s bound (before: 697–796 s onmain, 9 of 11 runs above 720). The headroom line's implied width is 12, not the 13 the PR's own runs printed: the sequencer's firstmainrun spread the legs 414–697 s (the PR runs' two balanced readings were 425–731 and 412–753). The width question stays with the data: nine more runs.Test (coverage)green: "the merged 8-shard report was produced for 1fb1ced…, the thresholds passed over the whole map". The coverage reds stopped on the first run, as H1 said. Recorded on main-red:Test (coverage)fails on main's push lane: a coverage shard hits its 40-minute ceiling on 7 of the last 25CIpush runs, so the coverage gate is not evaluated on those commits #12256 as well.test-file-timings-Nartifacts of this run are the refresh workflow's first input (Monday 06:47 UTC).