[objectstack-fleet]

Requested by Elon · project thread

domain:server#3 · os-elon-musk · session_016WV4czGci1wfPnGZhkvrcF

Fixes #28271

Clause-②: no (a re-issue on a store with more than 1000 parked runs now proves the pause as it already does below that; no schema, accept set, error code or result field changes)
Metadata-protocol: none

维护者速读

• 改了什么：审批的「恢复后重发」(continueRestoredRun) 的 GUARD 3 改为按 run id 键读这个 run 停在哪个节点（新增引擎成员 AutomationEngine.suspensionOf(runId)），不再在 listSuspendedRunsDurable() 的 1000 行首页里找。
• 为什么改：实测 main（69ce01d7）：两个副本共用一个 ObjectStoreSuspendedRunStore，前面已挂起 1000 条别的 run，副本 a 搁浅并恢复一个驳回决定，副本 b 重发：每次都以 INVALID_STATE "parked at no node this engine can see" 拒绝，连试 3 次一样，决定永远落不了地；998 条时正常。
• 风险与回滚：只换了「run 停在哪」这一步读法；期望节点的推导、不在暂停或停在别的节点照旧拒绝、重发本身都没变。新成员只读，一次按 id 读，用的是 hasSuspendedRun 同一个严格读法，存储读不出时抛错。回滚即 revert 本 PR。
• 先看：packages/services/service-automation/src/engine.ts 的 suspensionOf，与 approval-service.ts 里 continueRestoredRun 的 GUARD 3。
• 两分钟验证：先 build @objectstack/service-automation，再 pnpm --filter @objectstack/plugin-approvals exec vitest run src/restored-approval-continuation.test.ts（预期 20 passed）与 pnpm --filter @objectstack/service-automation exec vitest run src/suspension-of.test.ts（预期 4 passed）。
• 需要你做的：无，席位在 CI 绿后按落地授权落地。

What changes

• packages/services/service-automation/src/engine.ts: one new public read, suspensionOf(runId). It loads the run's suspension by id through loadSuspendedRunStrict, the store-authoritative loader hasSuspendedRun uses, and answers { runId, flowName, nodeId, correlation }, or undefined when the run is not suspended. One keyed read, never a listing. It rejects when the store cannot be read, as hasSuspendedRun does. It sits beside suspendedMapParentOf (fix(plugin-approvals): a revise-window recall on a batch item finds its batch run past the first 1000 parked runs #28268). No other engine hunk.
• packages/plugins/plugin-approvals/src/approval-service.ts. ApprovalResumeSurface declares suspensionOf? (optional, fail-closed: an engine without it is refused, as an engine without the listing was). GUARD 3 of continueRestoredRun reads suspensionOf(runId)?.nodeId instead of the run's entry in listSuspendedRunsDurable(). The refusal texts are unchanged except the missing-reader one, which now names suspensionOf. listSuspendedRunsDurable? stays declared (the interface is exported); its docblock now says this service no longer reads it.
• Tests. suspension-of.test.ts (new, service-automation) pins the engine member. restored-approval-continuation.test.ts (plugin-approvals) gains a #28271 block over two replicas and one ObjectStoreSuspendedRunStore, reusing the file's existing engine double, so scripts/engine-double-contract.pinned.json is untouched.
• .changeset/28271-restored-continuation-past-first-page.md. @objectstack/plugin-approvals and @objectstack/service-automation, patch.

Why

Before (measured at origin/main 69ce01d7)

Replicas a and b share one ObjectStoreSuspendedRunStore and one table set. N paused rows of another flow are stored first. On a, a deal_approval run parks, its reject decision strands (the reject branch throws), and restoreConsumedSuspension re-arms the pause. On b, continueRestoredRun re-issues it.

	N = 1000, before	N = 998, before	N = 1000, after
b.hasSuspendedRun(run)	true	true	true
run on b.listSuspendedRunsDurable()	no	yes	no (no longer read)
continueRestoredRun, 3 tries	refused each time: INVALID_STATE: run '…' is parked at no node this engine can see, …	resumed: true	resumed: true, the reject branch runs

The listing reads one 1000-row page of paused rows in driver order, and a process adds only what it parked itself. So the run is missed exactly when another process (a replica, or this one before a restart) parked it behind more than 1000 runs, and a retry reads the same page.

Why this path

Triage 6535230031: a keyed read by run id, the same kind as suspendedMapParentOf, still fail-closed when the run is not parked. The strict loader is the one GUARD 2 (hasSuspendedRun) already reads, so GUARD 2 and GUARD 3 now ask the same store the same question; the listing's own merge also let the durable row win (#14332), so below the page nothing changes. On an outage GUARD 2 already throws before GUARD 3 is reached.

Other listSuspendedRunsDurable() callers

Grepped (git grep -n "listSuspendedRunsDurable\|readSuspendedRuns", non-test). GUARD 3 was the only caller that looks up a single run. The rest:

• engine.ts listSuspendedRunsDurable() itself — the operability listing; a page is its contract.
• subflow-guards.ts parkedRunsOf → readSuspendedRuns('throw', flowNames) — per-flow, read to its end with listByFlow (service-automation: ADR-0126 §7.3's packaged-subflow guard holds on the toggle door only — creating, republishing or deleting a flow can still arm a packaged caller onto a disabled packaged subflow #20725); not a page.
• runtime/src/domains/automation.ts:1628 reads the synchronous in-memory listSuspendedRuns(), not the durable listing, for a listing, not a single-run lookup.
• The revise-window recall's batch lookup moved off the listing in fix(plugin-approvals): a revise-window recall on a batch item finds its batch run past the first 1000 parked runs #28268.

Tests

suspension-of.test.ts (4 cells): a parked run answers its node, also on a second engine over the same store with nothing in memory; the answer follows the run to its next pause on another engine and is undefined once that engine completes it (no stale memory answer); an unknown id answers undefined; a store whose load throws rejects.

restored-approval-continuation.test.ts, #28271 block (5 cells):

1. past the first page, the pause a re-armed is proved on b and the decision is re-issued; the reject branch runs (red on main: refused "parked at no node this engine can see");
2. CONTROL: within the first page (998) it is re-issued as before;
3. CONTROL: past the first page, a run parked at another node (the revise window after a send-back) is still refused, and the refusal names wait_revision (red on main: it named no node);
4. CONTROL: past the first page, a run that is not parked is still refused — by GUARD 2, and by GUARD 3 on an engine without hasSuspendedRun;
5. CONTROL: an engine without suspensionOf is refused with INVALID_STATE, and the pause is left in place.

Ablations through scripts/ablation-replace.mjs, each restored to the HEAD blob with git diff HEAD empty:

Ablation	Red
A1: GUARD 3 reads the listing again	#28271 cells 1 and 3
A2: GUARD 3 drops the node comparison	PIN 7, #28271 cells 3 and 4
A3: GUARD 3 drops the missing-reader refusal	#28271 cell 5
A4: suspensionOf answers from this process's memory, not the store	engine cells 1 and 2; #28271 cells 1, 2 and 3

Verification

Head c4e208a4, after a whole-tree turbo run build, pnpm install --frozen-lockfile and an unshallow fetch; @objectstack/service-automation rebuilt after the engine edit (plugin-approvals tests import its dist).

• pnpm --filter @objectstack/service-automation run test: 236 files, 2973 passed. pnpm --filter @objectstack/plugin-approvals run test: 106 files, 1366 passed. Both packages' typecheck (including check:test-typecheck): exit 0.
• dispatch-gates --commands: 64 families, all exit 0. --ran: 64 derived, 64 run, 0 NOT-MEASURED, 0 UNRUN. check:engine-double-contract and check:objectql-double-limit are green with no ledger edit.
• pnpm lint: exit 0. check-commit-card-trailers --range origin/main..HEAD: exit 0.

Skills-docs: none. git grep -n "listSuspendedRunsDurable\|parked at no node\|continueRestoredRun\|suspendedMapParentOf" -- skills content docs has no hit; git grep -il "restoreConsumedSuspension\|re-issue the continuation" -- skills content hits only content/docs/automation/flows.mdx, whose restore paragraph does not describe the listing or a page cap.

Acceptance notes

• run-history.ts:463-465 (readSuspendedRuns' degrade-path comment) still says "plugin-approvals' restored-continuation proof refuses on an absent entry, so a short list costs it a retry". After this PR that proof no longer reads the listing, so the sentence is stale; the comment's conclusion (keep the level at warn) still holds. Outside this card's file surface; not edited.

Human Minimum

• 1. Schemas start as Zod; packages/spec holds no runtime logic. (Not touched.)
• 2. Every name I added uses the casing for its kind.
• 3. No formatter ran; pnpm lint passes.
• 4. One issue, one branch off an up-to-date main, one pull request.
• 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset.
• 6. packages/spec touched: its generated artifacts are regenerated and committed. (Not touched.)
• 7. No hand edits to content/docs/references/, packages/*/CHANGELOG.md or content/docs/releases/.
• 8. Title, body and comments are in English (the 维护者速读 block is the lane's maintainer summary).
• The Before Submitting list in CONTRIBUTING.md is done.

🤖 Generated with Claude Code

https://claude.ai/code/session_016WV4czGci1wfPnGZhkvrcF

[github-actions]

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/plugin-approvals, @objectstack/service-automation, touching 5 documentable anchor(s).

⛔ 3 release-owned page(s) name something this change touched. These are read-only:

• content/docs/releases/v16.mdx (via AutomationEngine (symbol, a top-level class))
• content/docs/releases/v17/17-0.mdx (via AutomationEngine (symbol, a top-level class))
• content/docs/releases/v17/17-6.mdx (via AutomationEngine (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see

• the SDK route bridge reached 59 of 211 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
• a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 57f8ca2f0afbc0b08df9b02543ce32691e0ede34 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 9bd5e9ea46cd317f3712d9f6abdcd7d252a59abb — the merge of head c4e208a4554b6c9a1075ef6f50b4c9b77eb0e33b into base 57f8ca2f0afbc0b08df9b02543ce32691e0ede34, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 9bd5e9ea46cd317f3712d9f6abdcd7d252a59abb && git checkout 9bd5e9ea46cd317f3712d9f6abdcd7d252a59abb
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 57f8ca2f0afbc0b08df9b02543ce32691e0ede34 c4e208a4554b6c9a1075ef6f50b4c9b77eb0e33b && git checkout -B drift-repro 57f8ca2f0afbc0b08df9b02543ce32691e0ede34 && git merge --no-ff c4e208a4554b6c9a1075ef6f50b4c9b77eb0e33b

node scripts/docs-audit/affected-docs.mjs --json 57f8ca2f0afbc0b08df9b02543ce32691e0ede34

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 57f8ca2f0afbc0b08df9b02543ce32691e0ede34 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

[objectstack-fleet]

domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia

Seat ACCEPT · PR #28314 · head c4e208a4554b6c9a1075ef6f50b4c9b77eb0e33b · 2026-10-31T04:34Z

维护者速读

• 改了什么： 审批的"恢复后续发"在证明运行停在哪个节点时，改为按运行 id 直接读一条（新的 AutomationEngine.suspensionOf(runId)），不再翻挂起列表的第一页。
• 为什么改： 数据库存储上那份列表只读前 1000 条。别的副本挂起的运行如果排在 1000 条之后，续发每次都被拒，已记录的审批结论永远不生效（与 plugin-approvals: past the 1000-row suspended-run page, a revise-window recall on a batch item cannot find its batch run and leaves it parked #28230 同类）。
• 风险与回滚： Clause-②: no；1000 条以内行为不变；没挂起、挂在别的节点、引擎缺这个读取，都照旧拒绝；存储读不了时现在直接报错（仍是拒绝）。回滚即还原本 PR。
• 先看： approval-service.ts GUARD 3 和 engine.ts 的 suspensionOf。
• 两分钟验证： 跑 restored-approval-continuation.test.ts（前面挂 1000 条时续发通过）。
• 需要你做的： 无（Tier S，未命中受管面，无契约复核义务，按本期落地授权由席位落地）。

席位核对

• Shape: draft → main, Fixes #28271. 5 files (+322/−23). check-governed-merges --pr 28314: NOT governed (0 of 5 paths).
• Premise: measured on main @ 69ce01d7 by the card thread: with 1000 rows parked ahead, a second replica's re-issue is refused "parked at no node this engine can see" on 3 of 3 tries; with 998 it passes.
• File surface: within the claim (6538070000): approval-service.ts, engine.ts, tests beside them, one changeset.
• Seat diff read: suspensionOf is one keyed read through the same strict loader hasSuspendedRun uses, next to suspendedMapParentOf (fix(plugin-approvals): a revise-window recall on a batch item finds its batch run past the first 1000 parked runs #28268). GUARD 3 stays fail-closed in every direction: no reader, no pause, a pause at another node, or a store that cannot be read. plugin-approvals and service-automation are in the same changeset fixed group and pinned workspace:*, so the new optional member carries no version skew. The card thread found no other single-run listSuspendedRunsDurable() caller and names the search in the PR.
• Declarations: Clause-②: no, re-derived from the diff: an optional member on an existing internal interface plus one public engine method; no exported type, status, error code or result field. Metadata-protocol: none. Patch changesets. Skills-docs: none with the grep shown.
• Local (card thread): dispatch-gates 64/64, 0 NOT-MEASURED; pnpm lint 0; both packages' tests and typecheck green; 4 engine cells and 5 approvals cells pinned, each ablation red. Report 6538884932.
• Gate check-runs on this head: 48 total, 35 success, 13 skipped by the path filter, 0 failed. mergeable_state: clean.
• Landing: Tier S, under the maintainer's landing grant for this term (2026-10-29T22:05Z, seat thread). Ready + auto-merge through the relay.

ACCEPT