Repository navigation
Conversation
d970348 to
27c8635
Compare
27c8635 to
f9be52b
Compare
|
@jeromevdl This PR is now limited to the bounded, read-only audit at f9be52b. It targets #534 directly and can follow it independently of #537; there is no #536 dependency. Completion enforcement and the canonical evaluator have moved into #534. Delivery-specific safeguards have moved into #537. This increment contains audit traversal, bounded indexed reads, scope-bound resume checkpoints, the CLI, and the existing authorized audit-page presentation, all using #534's shared evaluator. It does not repair records or rerun execution; missing or ambiguous historical evidence stays unverified. Validation: 5,299 backend tests passed (3 existing skips), 838 frontend tests and production build, 99 release tests, and the actual offline CLI fixture returned the expected incomplete result. All seven current-head CI checks pass. The audit also passed combined validation with the independent delivery sibling. No production audit was run. Could you re-review this audit-only scope? |
jeromevdl
left a comment
There was a problem hiding this comment.
P2 — --max-requests can make the audit skip intents permanently
Location: scripts/audit-completion.mjs:19-33 (readBudget), lambda/shared/completion-audit.js:84-109
What goes wrong when the limit is reached partway through a page:
- Each remaining intent on the current page fails with the limit error.
auditCompletiontreats that error like any other read failure. Those intents are reported asunknown / completion_read_failedand counted as examined. - The loop then moves
cursorto the end of the page. The next page request also fails on the limit, sodiscoveryUnverifiedis set. - The checkpoint is saved with the cursor after the page, so it skips intents that were never actually audited.
Reproduced with one page of 4 intents and the limit reached during the third:
e1:complete e2:complete e3:unknown:completion_read_failed e4:unknown:completion_read_failed
checkpoint cursor: after-e4
Impact. The documented workflow says to rerun with --checkpoint until finished: true. Following it, the operator gets a "finished" audit in which e3 and e4 were never checked, and the reported reason (read failure) is wrong. This is a realistic case: each intent costs roughly 5–20 queries plus the plan load, so the defaults (100 intents, 5,000 requests) can run out on large intents.
Why the option isn't needed. Total reads are already bounded by --limit (intents), --max-pages / --page-size (discovery), --record-limit (records per intent) and maxAttempts: 2 (SDK retries). The global counter adds a second limit that overlaps the others, and the bug only exists because of it.
Suggested fix (KISS). Remove --max-requests and keep the read-only command guard:
If a hard per-run request cap really is needed, make reaching it a clean stop instead:
- Rethrow the limit error out of the per-intent
try/catchinstead of turning it intocompletion_read_failed. - Save the checkpoint at the last fully audited intent, using that page item's GSI key as
ExclusiveStartKey, notpage.cursor. - Report a distinct reason such as
budget_exhausted. - Add a regression test where the limit is reached mid-page.
Minor:
- The audit page cache key includes detail.status/completedAt. On first load, detail is undefined and then arrives, so /audit is fetched twice, each time resolving the plan. Fetching only once detail exists would avoid that.
- /audit computes completion for RUNNING intents too, even though the UI only shows it for terminal ones. Gating it on terminal status would save a plan load.
9288d89 to
b6883ab
Compare
|
@JWThewes what's the status of this? I don't think my latest finding (--max-requests) is fixed |
Add an operator-run, read-only audit for terminal intents whose persisted work may be incomplete or unverifiable. The audit reports intent/run IDs, reasons, unit counts and summary metrics; it never repairs records or reruns execution.
Discovery is bounded by an explicit project, terminal status, UTC execution-start interval, page/execution/record limits and a read-request budget. Scope-bound checkpoints support safe resume and reruns. Current META/plan rechecks distinguish changed or unreadable evidence from verified completion. The CLI rejects scans and writes and includes a credential-free fixture mode. The existing authorized audit page shows the same assessment and remaining work.
Depends directly on #534's canonical completion/provenance evaluator. Completion enforcement now belongs to #534; delivery-specific safeguards belong to #537. This PR has no delivery or terminal-preservation dependency and can merge independently of #537 after #534. #536 is deferred with terminal-preservation recovery.
Original correction validation at 8afa750 on Node 24 (one backend worker):
Cancellation amendment at c216d43 inherits #534's shared correction through an additive merge. Unpinned CANCELLED executions and recorded current-run aborts now report aborted before workflow resolution. API/audit/CLI regressions preserve unfinished unit evidence, count aborted rather than unknown, and return CLI exit 0 without labeling work complete. Focused shared/audit/API/frozen-replay checks 399 passed; CLI 4 passed; release tooling 101 passed; frontend 839 passed and production build passed; normal commit hooks 1,076 passed. Lint, formatting, secretlint, dependency/SDK checks passed. Earlier full-backend counts above remain historical for the preceding heads.
The independent request-budget/checkpoint review finding and minor fetch suggestions remain unresolved. This cancellation amendment does not change audit pagination/budget behavior or claim approval of this PR.
No infrastructure, scheduling, status migration, production audit or automatic continuation is added. The audit observes current terminal records, not every retired history. Ambiguous legacy unit-plan scope, explicit incomplete repository evidence, read limits and concurrent changes remain unknown or incomplete. Pre-deploy MERGED rows with no integrationResults attribute retain their supported integrated meaning; legacy no-unit serial records do not acquire false unit findings; GSI discovery can lag and supports bounded reruns. Finished incomplete intents still require ordinary rewind (which reruns affected work) or a new intent; preservation recovery is deferred.