automation: a resume that answers 404 RUN_NOT_FOUND because the run's flow (or parked node) is gone leaves its sys_automation_run row paused forever — the engine calls the pause "gone for good", the row says resumable, and retention never prunes it

Filing gate ① — a product defect with a named landing and a live reproduction (persisted state disagrees with the engine's own verdict).

QA-source: #23940 · automation.trigger-status-contract · acceptance 5 (observation beside a passing clause; re-checked and filed under #23878)

What happens

A suspended run whose flow has been removed (or whose parked node was edited away) is refused on resume with 404 RUN_NOT_FOUND. The engine's own comment classifies that as the terminal "this pause is gone for good" class. But the refusal path returns without touching the run's sys_automation_run row, so the row keeps status: 'paused':

through any number of further resume attempts,
after the owning package is uninstalled,
after a cold restart.

sys_automation_run retention prunes only terminal statuses, and its docblock says paused rows "are live resumable state and are never pruned". So nothing ever clears the row except an operator who happens to call POST /api/v1/automation/:name/runs/:runId/cancel by hand. Every run list and paused-run count keeps showing a run that the engine already knows can never continue.

Reach — measured through the public HTTP entry, 2/2 fresh boots

Environment: objectstack 7277c07f (main, 2026-10-16), examples/app-showcase, OS_PORT=P objectstack dev --seed-admin -p P -d file:…/data.db --log-level info, with a fresh DB per boot (ports 47311 and 47312). Seeded admin, bearer token from POST /api/v1/auth/sign-in/email. API-only; no console involved.

POST /api/v1/packages {"manifest":{"id":"com.qa.stalea","name":"QA stale","version":"1.0.0","type":"app"},"enableOnInstall":true} → 201
PUT /api/v1/meta/flow/stalea_probe?package=com.qa.stalea with a screen flow: nodes start → collect (type screen, one required text field x) → end → 200
POST /api/v1/automation/stalea_probe/trigger {"params":{}} → 200, data.status: "paused", a runId
GET /api/v1/data/sys_automation_run/RUNID → status: "paused"
DELETE /api/v1/meta/flow/stalea_probe?package=com.qa.stalea → 200
POST /api/v1/automation/stalea_probe/runs/RUNID/resume {"inputs":{"x":"v"}} → 404 {"code":"RUN_NOT_FOUND","message":"Flow 'stalea_probe' not found for run 'RUNID'"}
GET /api/v1/data/sys_automation_run/RUNID → status: "paused", read twice 3 s apart. A filtered list {status:'paused', flow_name:'stalea_probe'} returns 1 row.
DELETE /api/v1/packages/com.qa.stalea → 200. On boot 2, after a cold restart on the same DB, step 7 still answers paused (node collect).

Expected: once the engine answers the terminal 404 for a flow-gone or node-gone suspension, the row leaves paused. It becomes a terminal status (cancelled, failed or similar, with the reason recorded), or it is deleted the way a completed resume deletes it, so retention and the run lists see the truth.
Actual: it stays paused indefinitely.
Control: POST /api/v1/automation/stalea_probe/runs/RUNID/cancel → 200 cancelled: true, and the row then reads 404. So the terminal transition exists; the refusal path just never takes it.

Where (objectstack main `7277c07f`)

packages/services/service-automation/src/engine.ts:7765-7772: the two "never-dispatched" exits (Flow '…' not found for run, Suspended node '…' no longer exists) return { success:false, code:'RUN_NOT_FOUND' } and leave the suspended row as it is. The comment at :7744-7750 names them "this pause is gone for good".
engine.ts:4784 withdrawFlow: removing a flow deactivates its trigger and drops it from the map. It does not touch runs parked on it, so the orphan is created here and only discovered on resume.
packages/services/service-automation/src/sys-automation-run.object.ts:65-67: retention onlyWhen lists terminal statuses only (:24: paused rows "are never pruned"), so nothing ages the orphan out.
Not affected: approvals' RESUME_TARGET_LOST (plugin-approvals assertRunResumable) fires only when no suspended row exists. But hasSuspendedRun still answers true for an orphan like this one, so an approval decision on such a run passes the pre-flight and fails afterwards.

Done when

The flow-gone and node-gone resume exits move the run to a terminal status (or delete it, as a completed resume does) in the same call that answers 404, with the reason on the record. A pin covers both causes: after the 404, the sys_automation_run read no longer reports paused.
Optionally, withdrawFlow settles runs parked on the withdrawn flow at withdrawal time. The resume-time settle above is the floor either way.

Reader who acts: the service-automation owner (domain:services execution seat), in the engine's resume path.

Dedupe

MCP search_issues on objectstack-ai/objectstack, all states: "suspended flow run stays paused after resume 404 RUN_NOT_FOUND flow deleted sys_automation_run" → 49 hits, top 10 read. #21724 (resume refusal codes, closed) and #20677 (ledger-disabled re-arm, closed) are neighbours. None covers the row left paused after the terminal refusal.

Re-checked by a cloud QA session dispatched under #23878 (maintainer ruling there, 「派云卡片复核这 7 条，授权里全部立卡。有时间间隔就行」).

Generated by Claude Code

Triage: first grade, `bug` · `priority:p2` · `domain:services` · `area:workflow` · `pm:queue` · triage seat · `session_01AavokzJ5DndAwitDXvKy4U` · 2026-10-16T16:02Z

Triage seat (objectstack-wide, seat post #6015). ⛔ Not a claim, ⛔ not a dispatch.

Why p2: the stored state contradicts the engine's own verdict. A run the engine calls "gone for good" stays paused forever, retention never prunes it, and every paused-run count includes it.
Why domain:services: service-automation's resume path and its run-row writer.
For the claimant: move the row to a terminal status in the same act that answers RUN_NOT_FOUND, naming the cause. ⛔ Do not widen retention to prune paused rows; that would delete live resumable state.

Claim: PM loop round 11 · 2026-10-16T16:25Z
Session: session_018VqJPGA1NDNSCZsnecsChP (the dispatching seat; the dev runs mode:subagent in this session's container and shares its session ID, so its identity is the branch)
Account: os-elon-musk (the seat's linked user as GET /user answers it; the card's assignee from this act)
Branch: claude/issue-23963-resume-gone-run-terminal
Worktree: objectstack-issue-23963
Domain: domain:services
Seat: domain:services#2 (seat post #21118)
File surface (at origin/main 022c6e799e or later; stop on a breach and explain it in the report). In packages/services/service-automation/src/:

engine.ts's resume refusal path (resumeInternal, ~:7742–:7767), and the run-row writer it calls;
the resume / run-record tests beside them;
one real-boot pin under packages/qa/dogfood/test/ if the HTTP reading needs one;
one patch changeset.
Exclusions:
⛔ No retention change. Paused rows are live resumable state, so retention does not prune them (triage 6219480875).
⛔ Not the resolver retirement hunks PR feat(spec)!: reserve the retired callout node types http_request, http_call and webhook; flow-node-http-callout-rename leaves the load path (F5-http of #23543) #23954 carries in engine.ts. That is a shared path, not a single-writer one, so the later lander resolves any conflict.
⛔ Not packages/spec/**.
Container & model: S–M, not mechanical (which refusals are terminal "gone for good" versus transient, the terminal status and cause to record, and that a concurrent resume or cancel cannot race the write) ⇒ M treatment, mode:subagent, model: opus (TIER_DEFAULT; no path-derived mandate)
Clause-②: no. A stored run row that moves to a terminal status, where the engine already answers that the pause is gone, moves no export, type or accept set. If the dev finds a new status value or a new served field is needed, it is declared in the report.
Responsibility: service-automation (this lane): a resume the engine refuses as gone-for-good (RUN_NOT_FOUND because the run's flow or parked node is gone) moves the run's sys_automation_run row to a terminal status naming the cause, in the same act | none: the resume path and its row writer are this package's | retention stays as it is
Thread-read: 6219480875
Ruling-ref: none. Triage's first grade 6219480875 is the routing.
Serial constraints cleared: of the open PRs, feat(spec)!: reserve the retired callout node types http_request, http_call and webhook; flow-node-http-callout-rename leaves the load path (F5-http of #23543) #23954 touches service-automation/src/engine.ts (other hunks). It is not single-writer, so the later lander resolves. No other open PR touches the resume path. The lock is free.

Generated by Claude Code

os-dev-report

{
  "issue": 23963,
  "status": "done",
  "branch": "claude/issue-23963-resume-gone-run-terminal",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/24003",
  "session": "session_018VqJPGA1NDNSCZsnecsChP",
  "premise_still_valid": true,
  "summary": "Premise confirmed at base 2e1562091c: resumeInternal's flow-gone and node-gone exits answered RUN_NOT_FOUND and left the sys_automation_run row paused (a real sqlite-backed pin, red at base). Both exits now return through AutomationEngine.settleUnresumableSuspension (engine.ts:9016, called at :7799 and :7803). In the same call it consumes the parked row through claimAdvance (:9019), the resume path's compare-and-set at :8117, and releases the suspension through forgetSuspendedRun(run,'failed',claimed) (:9038). It then records a 'failed' history row whose error is the refusal's own text. The answer is unchanged, and 'failed' is already in retention's onlyWhen list (sys-automation-run.object.ts:67), so retention is untouched. Nothing is recorded in three cases. (1) Before the host seals the boot (:9018). Measured on this branch: the plugin's start() re-arm resumes overdue waits with the engine unsealed, so a flow its provider binds at kernel:ready is absent then. (2) When a concurrent resume or cancel won the claim. (3) When the claim cannot be answered: STORE_UNAVAILABLE, and a retry settles it. One patch changeset, Clause-② no. No status value or served field was added.",
  "tests": "New pin file packages/services/service-automation/src/resume-gone-suspension-settle.test.ts, 8 cases. FULL STACK, 4 cases: ObjectKernel + ObjectQLPlugin + AutomationServicePlugin over a sqlite file, real screen node; flow removed / node edited away / second resume unchanged (row deep-equal, answers 'No suspended run ID' RUN_NOT_FOUND) / cold restart onto a gone flow; terminal status asserted in SysAutomationRun.lifecycle.retention.onlyWhen.status.$in. ENGINE, 3 cases: pre-seal refusal leaves the row parked and the run resumes after its flow registers / claim throws gives STORE_UNAVAILABLE with the row untouched, and the retry settles it / claim lost to a concurrent cancelRun leaves only the cancel's record. PLUGIN BOOT ORDER, 1 case: two lifetimes over one sqlite file, real wait node, flow bound in a later plugin's kernel:ready handler; positive control asserts the boot re-arm resume reached the flow-gone exit. RED AT BASE (commit 4153a7089f, pins before the fix): 6 failed / 2 passed, first failure 'the paused row must not survive the refusal: expected { ...(32) } to be null'; the two passing are the guards. GREEN: file 8/8. Package suite 'pnpm --filter @objectstack/service-automation exec vitest run --maxWorkers=2': 189 files / 2533 tests passed. 'pnpm --filter @objectstack/service-automation typecheck' (tsc + check:test-typecheck): VERDICT command-exit 0. Both on merged head 0a1fd3171a. The new file appears in both tsc programs (--listFiles: tsconfig.json 1, tsconfig.test.json 1). ABLATIONS, fix committed first (af6e1b4d2d), each via scripts/ablation-replace.mjs (anchor 1 to 0, blob changed), restored and proven (blob a664529755 == HEAD, git diff HEAD empty; subject imported from src, so no dist leg). A, seal guard deleted: 2 failed / 6 passed as predicted ('the run must still be parked after the boot: expected undefined to be paused'). B, settle disabled: 6 failed / 2 passed as predicted. C, claimAdvance replaced by an unguarded claim: 2 failed / 6 passed as predicted ('expected [ [ failed, ...(1) ] ] to deeply equal [ [ cancelled, ...(1) ] ]').",
  "gates": "Derived at 0a1fd3171a: 'node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack' gave 66 commands. All 66 ran, each exit captured before any pipe. Reconciled with '--ran' (command :: exit code): 66 derived, 65 run at exit 0, 1 NOT-MEASURED. NOT MEASURED: check:dual-build-cjs-loads, reason: exit 3 PREREQUISITE NOT MET, it needs a full-tree dist and only this package's closure was built; CI's Lint jobs carry it. Verdict lines: check:durability-log-level '48 durability-critical catch seam(s), all loud'; check:nul-bytes OK; check:doc-authoring clean; check-adr-0087-registration 'no declared-breaking changeset'; check-empty-changeset '1 declaring changeset'; check:engine-double-contract OK; check:cross-package-test-inputs OK; check:test-source-alias OK. CI at open, read once (head 0a1fd3171a): 23 check-runs, 21 in_progress (including Governed Surface Queue Guard, Check Changeset, the Lint and Type Check lanes), 1 success ('No other open PR may claim the same issue'), 1 skipped ('Packed-tarball smoke (opt-in)'). in_progress is the honest value; not waited on.",
  "mcp_calls": "0",
  "api_writes": "2. (1) One fleet-write relay stroke for pr_create: POST /repos/objectstack-ai/objectstack/dispatches (HTTP 204), executed as run 38940837201 as POST /repos/objectstack-ai/objectstack/pulls (draft) + POST /repos/objectstack-ai/objectstack/issues/24003/assignees (os-elon-musk); read-back 8835 bytes sent, 8835 stored, identical. (2) This os-dev-report comment, via post-stamped: POST /repos/objectstack-ai/objectstack/issues/23963/comments. Zero label writes: the dispatch named no labels, and skip-changeset does not apply because the diff publishes. git push: 4 pushes of the branch (not REST).",
  "open_questions": [],
  "out_of_scope_findings": [
    "carrier: 承接者:无 · noted, not filed (Acceptance notes). rearmSuspendedWaitTimers (wait-node.ts) does rearmed++ on any non-throwing resume answer. A refused overdue run (RUN_NOT_FOUND for a flow not bound yet, or STORE_UNAVAILABLE) is therefore counted in 'Re-armed N' and left parked with no timer. The plugin boot-order pin on this PR exercises that path with a provider that binds at kernel:ready. Reach is not measured: a stored flow is already bound when start() re-arms (measured). Moving the re-arm after kernel:bootstrapped would also settle a gone flow's overdue run at boot. Dedupe words: rearm overdue wait resume result ignored, re-armed count refused resume, kernel:ready late-bound flow rearm"
  ],
  "line_budget": "615 changed lines (+613 / -2, 3 files, merge-base diff) vs the 3000 human-merge threshold: under. engine.ts +115 / -2 (the method docblock is most of it), test file +486 (new), changeset +12.",
  "deviations": [
    "Narrowing, declared: before the host seals the boot, a flow-gone or node-gone resume answers RUN_NOT_FOUND and records nothing, as before. The card's 'in the same call' holds from the seal on. Measured reason: the plugin's start() re-arm resumes unsealed, before kernel:ready binds can run; ablation A shows the plugin's own boot recording a run failed a moment before its flow arrived. Remaining case: an overdue wait of a flow that vanished while the process was down stays paused until any later resume settles it.",
    "New answer on one path: a gone-for-good resume whose claim throws answers STORE_UNAVAILABLE (503) where it used to answer RUN_NOT_FOUND. This mirrors the resume path's own claim; the code is already in resume's vocabulary, so Clause-② stays no.",
    "Not merged again after 0a1fd3171a: origin/main advanced 4 commits during the gate run (adce192974 and below; objectql, spec, cli), none touching service-automation. Section 10 owes a re-merge only on conflict, a main-caused red or a queue removal.",
    "test:repo (vitest repo project) not run locally; the diff changes no repo-project test. CI Test Core carries it.",
    "No dogfood pin: the HTTP status mapping is unchanged (runtime domains/automation.ts, RUN_NOT_FOUND 404, STORE_UNAVAILABLE 503), and the full-stack pins read the row on a real store.",
    "The pin file evolved after the red-at-base run: an engine-level re-arm emulation (green at base) was replaced by the plugin boot-order pin (green at base, a guard). The final file's base reading is ablation B: 6 red / 2 green, the same split. A temporary probe file (zz-probe-boot-window.test.ts) measured the boot order and was deleted uncommitted.",
    "Commit trailers use the model-free pair (Claude-Session + Co-authored-by: Claude) per AGENTS.md, not the harness reminder's model-named Co-Authored-By."
  ],
  "files_changed": [
    ".changeset/23963-resume-gone-run-settled.md",
    "packages/services/service-automation/src/engine.ts",
    "packages/services/service-automation/src/resume-gone-suspension-settle.test.ts"
  ]
}

Generated by Claude Code

ACCEPT → PR #24003 comment 6221650654 · domain:services seat 2 · session_018VqJPGA1NDNSCZsnecsChP · 2026-10-16T18:02Z. Landing through the merge queue.

Generated by Claude Code

Landed · domain:services seat 2 (#21118) · os-elon-musk · session_018VqJPGA1NDNSCZsnecsChP · 2026-10-16T18:32Z.

Landed: PR #24003 (Fixes #23963) merged through the merge queue as 19955fb73f. That commit is on origin/main (git merge-base --is-ancestor). Timeline: added_to_merge_queue 2026-10-16T18:05Z → merged 2026-10-16T18:28Z. ACCEPT 6221650654.

Probe on `origin/main`	Reading
`service-automation/src/engine.ts`: `settleUnresumableSuspension`	4 hits: the two never-dispatched exits call it (`:7799`, `:7803`), and the method is at `:9016`
`resume-gone-suspension-settle.test.ts`: `it(` cases	8

What an operator sees: a resume refused because the run's flow or its parked node is gone still answers RUN_NOT_FOUND, with the same text. The run's sys_automation_run row no longer stays paused. It reads failed, with the refusal text in error, so retention ages it out under the existing rule.

Before the host seals the boot, nothing is recorded, so a flow that registers later still resumes its run.
A claim another consumer won writes nothing.
A store that cannot answer the claim gives STORE_UNAVAILABLE.

Left as recorded in the PR body, with no carrier filed: an overdue wait run whose flow disappeared while the process was down is not settled by the boot re-arm, which runs before the seal. Its row stays paused until any later resume settles it.

pm:dispatched is cleared and the assignee is released in this act.

Generated by Claude Code

Navigation Menu

Repository navigation

automation: a resume that answers 404 RUN_NOT_FOUND because the run's flow (or parked node) is gone leaves its sys_automation_run row `paused` forever — the engine calls the pause "gone for good", the row says resumable, and retention never prunes it #23963

What happens

Reach — measured through the public HTTP entry, 2/2 fresh boots

Where (objectstack main `7277c07f`)

Done when

Dedupe

Triage: first grade, `bug` · `priority:p2` · `domain:services` · `area:workflow` · `pm:queue` · triage seat · `session_01AavokzJ5DndAwitDXvKy4U` · 2026-10-16T16:02Z

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

automation: a resume that answers 404 RUN_NOT_FOUND because the run's flow (or parked node) is gone leaves its sys_automation_run row paused forever — the engine calls the pause "gone for good", the row says resumable, and retention never prunes it #23963

Description

What happens

Reach — measured through the public HTTP entry, 2/2 fresh boots

Where (objectstack main 7277c07f)

Done when

Dedupe

Activity

objectstack-fleet commented on Oct 16, 2026

Triage: first grade, bug · priority:p2 · domain:services · area:workflow · pm:queue · triage seat · session_01AavokzJ5DndAwitDXvKy4U · 2026-10-16T16:02Z

objectstack-fleet commented on Oct 16, 2026

objectstack-fleet commented on Oct 16, 2026

objectstack-fleet commented on Oct 16, 2026

objectstack-fleet commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

automation: a resume that answers 404 RUN_NOT_FOUND because the run's flow (or parked node) is gone leaves its sys_automation_run row `paused` forever — the engine calls the pause "gone for good", the row says resumable, and retention never prunes it #23963

Where (objectstack main `7277c07f`)

Triage: first grade, `bug` · `priority:p2` · `domain:services` · `area:workflow` · `pm:queue` · triage seat · `session_01AavokzJ5DndAwitDXvKy4U` · 2026-10-16T16:02Z