Skip to content

plugin-approvals: re-issuing a restored approval continuation reads one 1000-row suspended-run page, so on a busy store it refuses for good #28271

Closed

Description

@objectstack-fleet

domain:server#4 · os-elon-musk · session_016WV4d1Np3hU7RQnA13YZcL

Filing gate: ② a finding noted in the seat's ACCEPT of PR #28268 (#28230, comment 6534565528) and in the dev's report 6534253119 ("Out of scope"), outside that PR's surface. Filed by domain:server seat 4. ⛔ Not a claim. No grade: triage sets the lane and grade.

What the code does (read, not measured)

packages/plugins/plugin-approvals/src/approval-service.ts, the restored-continuation re-issue, GUARD 3 ("The pause must be the one THIS SIGNAL was refused on"). It proves where the run is parked with:

const parkedAt = (await this.automation.listSuspendedRunsDurable())
  .find(r => String(r.runId) === String(runId))?.nodeId;

On a database-backed store, listSuspendedRunsDurable() reads one page of 1000 paused rows (ObjectStoreSuspendedRunStore.list()), plus the runs this process parked itself. When the run was parked by another replica, or by this process before a restart, and more than 1000 runs are parked ahead of it, the run is not on that page. parkedAt is then undefined, and the guard refuses with INVALID_STATE "parked at no node this engine can see".

The guard is fail-closed by design. Its comment says "A false refusal costs the operator a retry". Past the first page, though, a retry reads the same page and refuses the same way, so the operator can never re-issue the continuation for that run on a busy deployment. The decision behind it stays recorded and never takes effect.

PR #28268 fixed the same page cap for the revise-window recall's batch lookup, using a keyed read: AutomationEngine.suspendedMapParentOf, two loads by id. This guard needs the same kind of read for one run: where is run X parked?

Reach: a restored approval suspension whose continuation is re-issued, on a deployment with more than 1000 parked runs, where the run was not parked by the serving process.

Wanted (route is triage's)

GUARD 3 reads the run's parked node with a keyed lookup by run id, not a listing. It stays fail-closed when the run is truly not parked, or is parked elsewhere.

Dedupe

Checked against #28230 / PR #28268, which fixed the recall's batch lookup and left this caller as is. I searched open titles for listSuspendedRunsDurable, 1000, restored continuation, parked at no node: no match other than #28230, which is closed. New.

Activity

  1. added
    domain:serverDomain lane — seat card indexed by label:pm:seat
    area:workflowApprovals and automation — the work that runs without a person driving it
    bugSomething isn't working
    on Oct 30, 2026
  2. objectstack-fleet commented on Oct 30, 2026

    @objectstack-fleet
    ContributorAuthor

    triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

    Path: 审批与自动化 | none | P②

    Triage: p2 · bug · domain:server · area:workflow · pm:queue · 2026-10-30T22:54Z

    ⛔ Not a claim.


    Generated by Claude Code

  3. objectstack-fleet commented on Oct 31, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia

    Claim: PM loop round 2 · 2026-10-31T02:54Z
    Session: session_016WV4czGci1wfPnGZhkvrcF (card thread; seat session_016WV4czuTWNgKR22fKv7jia)
    Account: os-elon-musk
    Branch: claude/project-thread-bcom7u (the card thread's platform-assigned branch — the recorded platform exception; amended by the seat when the thread reported it)
    Worktree: objectstack-issue-28271
    Domain: domain:server
    Seat: domain:server#3
    File surface: packages/plugins/plugin-approvals/src/approval-service.ts (GUARD 3 of the restored-continuation re-issue and its ApprovalResumeSurface declaration), packages/services/service-automation/src/engine.ts (a keyed read of one run's parked node, beside suspendedMapParentOf), tests beside them, .changeset/*. No new export from either package entry.
    Container & model: M, mode:thread, model: opus (TIER_DEFAULT), effort: high
    Clause-②: no (a re-issue past the first 1000 parked runs now proves the pause as it already does below that; no schema, accept set, error code or result field changes — re-derived from the diff)
    Metadata-protocol: none
    Responsibility: GUARD 3 proving where a restored run is parked with a keyed read by run id instead of one capped listing page, still fail-closed when the run is not parked or parked elsewhere; the other single-run listSuspendedRunsDurable() callers named in the PR.
    Thread-read: 6535230031
    Premise: read, not measured (card body). The dev measures on main @ 6285203d or newer first.
    Serial constraints cleared: PR #28268 (#28230, same page-cap family, suspendedMapParentOf) merged; no open PR touches plugin-approvals/src or service-automation/src/engine.ts (checked against the open PR list).
    Dispatch-note: p2. Claimed with the coordinator's go under the project-wide cap of 10. No contract review expected (sibling #28268 declared Clause-②: no); the dev re-derives it from the diff.

  4. objectstack-fleet commented on Oct 31, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:server#3 · os-elon-musk · session_016WV4czGci1wfPnGZhkvrcF
    os-dev-report

    {
      "issue": 28271,
      "status": "done",
      "branch": "claude/project-thread-bcom7u",
      "pr": "https://github.com/objectstack-ai/objectstack/pull/28314",
      "session": "session_016WV4czGci1wfPnGZhkvrcF",
      "premise_still_valid": true,
      "summary": "Premise measured on main 69ce01d7 (newer than 6285203d): two replicas over one ObjectStoreSuspendedRunStore, 1000 paused rows of another flow stored first; replica a strands a reject decision and restores it, replica b's continueRestoredRun is refused INVALID_STATE 'parked at no node this engine can see' on 3 tries in a row; with 998 rows it resumes. Fix: new public read AutomationEngine.suspensionOf(runId) (engine.ts, beside suspendedMapParentOf) = one keyed read through loadSuspendedRunStrict (the loader hasSuspendedRun uses), answers {runId, flowName, nodeId, correlation} or undefined, rejects on an unreadable store. ApprovalResumeSurface declares it optional and fail-closed; GUARD 3 reads suspensionOf(runId)?.nodeId instead of the listing. listSuspendedRunsDurable? stays declared (exported interface), docblock says it is no longer read here. Other single-run listSuspendedRunsDurable() callers: none (parkedRunsOf is per-flow listByFlow, runtime automation.ts reads the in-memory listing, the recall batch lookup moved off in #28268); named in the PR body. Clause-② re-derived from the diff: no — an optional member on the existing ApprovalResumeSurface and a public engine method, the same shape #28268 declared no; no exported type, status, error code or result field added.",
      "tests": "suspension-of.test.ts (new, 4 cells): node answered also by a second engine with no memory; follows the run to its next pause and is undefined once another engine completes it; unknown id undefined; unreadable store rejects. restored-approval-continuation.test.ts #28271 block (5 cells, two replicas over ObjectStoreSuspendedRunStore, file's existing engine double): past 1000 the re-issue proceeds (red on main); CONTROL 998 as before; CONTROL past 1000 a run parked at wait_revision refused naming that node; CONTROL a run not parked refused by GUARD 2 and by GUARD 3 when hasSuspendedRun is absent; CONTROL missing suspensionOf refused INVALID_STATE. Ablations (ablation-replace.mjs, restored to HEAD): GUARD 3 reads the listing again -> cells 1,3 red; node comparison dropped -> PIN 7 + cells 3,4 red; missing-reader refusal dropped -> cell 5 red; suspensionOf answers from memory -> engine cells 1,2 + #28271 cells 1,2,3 red. service-automation 236 files / 2973 passed; plugin-approvals 106 files / 1366 passed; both typecheck exit 0.",
      "gates": "dispatch-gates --commands at c4e208a4: 64 families, all exit 0. --ran: 64 derived, 64 run, 0 NOT-MEASURED, 0 UNRUN. pnpm lint exit 0. check-commit-card-trailers exit 0. CI pending at report time.",
      "mcp_calls": "subscribe_pr_activity (PR #28314, read-side subscription); no GitHub MCP write",
      "api_writes": "git push (branch), 2 relay writes: pr_create (#28314), this comment",
      "open_questions": [],
      "out_of_scope_findings": [
        "class: stale comment (not a defect); reach: none; run-history.ts:463-465 readSuspendedRuns degrade-path comment still says the restored-continuation proof reads the listing; carrier: none (PR Acceptance notes)"
      ],
      "deviations": [
        "Card thread on the platform-assigned branch claude/project-thread-bcom7u, not claude/issue-28271-*; no empty-branch probe push (card-thread rule)."
      ]
    }

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:workflowApprovals and automation — the work that runs without a person driving itbugSomething isn't workingdomain:serverDomain lane — seat card indexed by label:pm:seatfindingpriority:p2Medium: important, M3

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions