Skip to content

Audit terminal intent completion with bounded read-only checks - #539

Open
JWThewes wants to merge 24 commits into
fix/blocked-unit-completionfrom
fix/terminal-completion-consistency
Open

JWThewes wants to merge 24 commits into
fix/blocked-unit-completionfrom
fix/terminal-completion-consistency

Conversation

@JWThewes

@JWThewes JWThewes commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Add an operator-run, read-only audit for terminal intents whose persisted work may be incomplete or unverifiable. The audit reports intent/run IDs, reasons, unit counts and summary metrics; it never repairs records or reruns execution.

Discovery is bounded by an explicit project, terminal status, UTC execution-start interval, page/execution/record limits and a read-request budget. Scope-bound checkpoints support safe resume and reruns. Current META/plan rechecks distinguish changed or unreadable evidence from verified completion. The CLI rejects scans and writes and includes a credential-free fixture mode. The existing authorized audit page shows the same assessment and remaining work.

Depends directly on #534's canonical completion/provenance evaluator. Completion enforcement now belongs to #534; delivery-specific safeguards belong to #537. This PR has no delivery or terminal-preservation dependency and can merge independently of #537 after #534. #536 is deferred with terminal-preservation recovery.

Original correction validation at 8afa750 on Node 24 (one backend worker):

  • Affected shared evaluator/audit/API/real local DynamoDB suites: 385 tests passed. The serial no-unit audit regression preserves the records and reports no unfinished units; limits, pagination, scope denial, changed ownership and unknown evidence remain covered.
  • CLI tests: 2 passed. Rendered audit/intent/unit-lane tests: 90 passed.
  • Lint, formatting, secretlint, dependency/SDK checks and docs build passed. Normal commit hooks: 1,065 tests passed.
  • The corrected Prevent incomplete intents from reaching success and publication #534 base passed its full backend suite (5,302 passed, 3 existing Bun-dependent skips). The audit's earlier full-suite/build/release validation predates this base update; affected checks above were rerun on the current tree.

Cancellation amendment at c216d43 inherits #534's shared correction through an additive merge. Unpinned CANCELLED executions and recorded current-run aborts now report aborted before workflow resolution. API/audit/CLI regressions preserve unfinished unit evidence, count aborted rather than unknown, and return CLI exit 0 without labeling work complete. Focused shared/audit/API/frozen-replay checks 399 passed; CLI 4 passed; release tooling 101 passed; frontend 839 passed and production build passed; normal commit hooks 1,076 passed. Lint, formatting, secretlint, dependency/SDK checks passed. Earlier full-backend counts above remain historical for the preceding heads.

The independent request-budget/checkpoint review finding and minor fetch suggestions remain unresolved. This cancellation amendment does not change audit pagination/budget behavior or claim approval of this PR.

No infrastructure, scheduling, status migration, production audit or automatic continuation is added. The audit observes current terminal records, not every retired history. Ambiguous legacy unit-plan scope, explicit incomplete repository evidence, read limits and concurrent changes remain unknown or incomplete. Pre-deploy MERGED rows with no integrationResults attribute retain their supported integrated meaning; legacy no-unit serial records do not acquire false unit findings; GSI discovery can lag and supports bounded reruns. Finished incomplete intents still require ordinary rewind (which reruns affected work) or a new intent; preservation recovery is deferred.

@JWThewes
JWThewes force-pushed the fix/terminal-completion-consistency branch from 27c8635 to f9be52b Compare October 15, 2026 15:31
@JWThewes
JWThewes removed this pull request from stack #551 October 15, 2026 15:36
@JWThewes JWThewes changed the title Guard final completion and add bounded consistency auditing Audit terminal intent completion with bounded read-only checks Oct 15, 2026
@JWThewes
JWThewes changed the base branch from fix/intent-delivery-status-retry to fix/blocked-unit-completion October 15, 2026 15:36
@JWThewes

Copy link
Copy Markdown
Contributor Author

@jeromevdl This PR is now limited to the bounded, read-only audit at f9be52b. It targets #534 directly and can follow it independently of #537; there is no #536 dependency.

Completion enforcement and the canonical evaluator have moved into #534. Delivery-specific safeguards have moved into #537. This increment contains audit traversal, bounded indexed reads, scope-bound resume checkpoints, the CLI, and the existing authorized audit-page presentation, all using #534's shared evaluator. It does not repair records or rerun execution; missing or ambiguous historical evidence stays unverified.

Validation: 5,299 backend tests passed (3 existing skips), 838 frontend tests and production build, 99 release tests, and the actual offline CLI fixture returned the expected incomplete result. All seven current-head CI checks pass. The audit also passed combined validation with the independent delivery sibling. No production audit was run.

Could you re-review this audit-only scope?

@jeromevdl jeromevdl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 — --max-requests can make the audit skip intents permanently

Location: scripts/audit-completion.mjs:19-33 (readBudget), lambda/shared/completion-audit.js:84-109

What goes wrong when the limit is reached partway through a page:

  1. Each remaining intent on the current page fails with the limit error. auditCompletion treats that error like any other read failure. Those intents are reported as unknown / completion_read_failed and counted as examined.
  2. The loop then moves cursor to the end of the page. The next page request also fails on the limit, so discoveryUnverified is set.
  3. The checkpoint is saved with the cursor after the page, so it skips intents that were never actually audited.

Reproduced with one page of 4 intents and the limit reached during the third:

e1:complete  e2:complete  e3:unknown:completion_read_failed  e4:unknown:completion_read_failed
checkpoint cursor: after-e4

Impact. The documented workflow says to rerun with --checkpoint until finished: true. Following it, the operator gets a "finished" audit in which e3 and e4 were never checked, and the reported reason (read failure) is wrong. This is a realistic case: each intent costs roughly 5–20 queries plus the plan load, so the defaults (100 intents, 5,000 requests) can run out on large intents.

Why the option isn't needed. Total reads are already bounded by --limit (intents), --max-pages / --page-size (discovery), --record-limit (records per intent) and maxAttempts: 2 (SDK retries). The global counter adds a second limit that overlaps the others, and the bug only exists because of it.

Suggested fix (KISS). Remove --max-requests and keep the read-only command guard:

If a hard per-run request cap really is needed, make reaching it a clean stop instead:

  • Rethrow the limit error out of the per-intent try/catch instead of turning it into completion_read_failed.
  • Save the checkpoint at the last fully audited intent, using that page item's GSI key as ExclusiveStartKey, not page.cursor.
  • Report a distinct reason such as budget_exhausted.
  • Add a regression test where the limit is reached mid-page.

Minor:

  • The audit page cache key includes detail.status/completedAt. On first load, detail is undefined and then arrives, so /audit is fetched twice, each time resolving the plan. Fetching only once detail exists would avoid that.
  • /audit computes completion for RUNNING intents too, even though the UI only shows it for terminal ones. Gating it on terminal status would save a plan load.

@JWThewes
JWThewes force-pushed the fix/blocked-unit-completion branch from 9288d89 to b6883ab Compare October 26, 2026 13:06
@jeromevdl

Copy link
Copy Markdown
Collaborator

@JWThewes what's the status of this? I don't think my latest finding (--max-requests) is fixed

@jeromevdl jeromevdl added area/frontend React application in frontend/ area/agents AgentCore runtime: CLI drivers, MCP tools, workspace and Git operations area/backend Lambda APIs, lambda/shared, v2-orchestrator area/ci GitHub Actions, hooks, lint, release tooling, E2E harness size/XL 3000+ changed lines (lockfiles excluded), consider splitting labels Oct 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/agents AgentCore runtime: CLI drivers, MCP tools, workspace and Git operations area/backend Lambda APIs, lambda/shared, v2-orchestrator area/ci GitHub Actions, hooks, lint, release tooling, E2E harness area/frontend React application in frontend/ size/XL 3000+ changed lines (lockfiles excluded), consider splitting

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants