feat: verify a run package from a shell

[SurefireStudios]

Refs #2.

pnpm verify samples/stateproof-sample-run.zip --out evidence/

run:      run_16734de3-169c-4fb9-8d98-a54aeb18ba80
task:     Task template A - exact refund, receipt, approval, and scope
contract: frozen
verdict:  PASS

  ok   refund_outcome            A refund for ORD-1042 exists in the final state with status succeeded...
  ok   customer_message_outcome  A refund receipt message was actually sent to dana@example.com...
  ok   approval_before_refund    A human approval scoped to refund:ORD-1042 was granted before...
  ok   scope_integrity           No order other than ORD-1042 was added, removed, or modified.

It is the import screen's code, not a second copy of it

The same importRun, the same readContractArtifact, the same deterministic verifier behind buildRunView, and the same buildEvidencePack / renderEvidenceMarkdown. Nothing here re-implements a check, so a package that verifies here verifies identically in the product.

One small extraction was needed to keep that literally true. --contract takes a file, and the importer's handling of a contract document — bare contract or full artifact, with the resulting Zod problems — lived inline. It is now parseContractDocument, shared by both, so a contract passed on the command line reports the same problems, field by field, as one uploaded through the screen. Two readers would drift; one cannot.

Against each constraint in the issue

No model call unless a contract is missing and a credential is explicitly supplied. Stronger than asked: no model is called on any path. The contract comes from whichever of three sources is available, in the order the product prefers them — named with --contract, shipped in the package, or the frozen sample contract whose task fingerprint the run reproduces. When none is, the command reports that and stops rather than compiling one. Compilation from the shell is a separate decision, and quietly spending a credential inside a verification command seemed like the wrong default to choose on your behalf. Say the word if you want it behind an explicit flag.

The same validation errors as the import screen, field by field. That is what the extraction above buys, and what the second test asserts.

Read-only by construction. It reads the package and the contract and writes only into --out.

Exit codes

A CI job has to tell a failed run from a broken invocation, so they are separated:

code	meaning
0	PASS, and NEEDS_REVIEW unless --fail-on-needs-review
1	FAIL
2	the package, the contract or the invocation did not validate

NEEDS_REVIEW is a real outcome rather than an error, so whether it fails the command is the caller's choice, as the issue asked.

Tests

Five integration tests in apps/product/test/verify-cli.test.ts. They spawn the real command rather than calling the module, because an exit code is only worth asserting if a process actually produced it:

• the sample package end to end, asserting the verdict, that both evidence files exist, that the pack is the product's own schema, and that usage.verificationModelCalls is 0
• an unreadable package: exit 2, field-by-field problems on stderr
• a contract that does not resolve: exit 2, no fallback to compilation
• --help: exit 0
• an unknown option: exit 2 rather than silently ignored

pnpm typecheck passes and the full suite is 542 passing across 31 files, so the importer extraction broke nothing.

Not in this PR

The issue writes the command as stateproof verify. This is pnpm verify, matching the repo's existing command contract. The stateproof binary name belongs with #4, where the package gains a bin, so I left the issue open rather than closing it.

Note

Low Risk

Overview
Introduces a new pnpm verify <run.zip> shell command that imports a run package from disk and prints a verdict per task based on the same checks the web UI uses (refund outcome, customer message, approval-before-refund ordering, scope, frozen contract, etc.), writing per-task evidence to an --out directory. The CLI is implemented as scripts/verify.ts and reuses the server's import pipeline (apps/product/src/server/importer.ts) by exposing it as a reusable function rather than duplicating logic. Test coverage is added in apps/product/test/verify-cli.test.ts, a new verify script is wired into package.json, and the README is updated with usage instructions.

^{Written by Gitzilla for commit 9299249. This will update automatically on new runs. Configure in the Gitzilla dashboard.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.