Skip to content

feat(cli): stepfork run — a tri-state verdict and a runner-owned receipt - #20

Merged
utsab345 merged 5 commits into
utsab345:mainfrom
how2how2how2-arch:export-run-receipt
Oct 19, 2026
Merged

utsab345 merged 5 commits into
utsab345:mainfrom
how2how2how2-arch:export-run-receipt

Conversation

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

Follow-up to #14, implementing the part you asked for in your last comment: the tri-state verdict, the runner-owned receipt, and stepfork run --receipt.

Why the receipt has to belong to the runner

stepfork export writes a pytest file. Run that file with pytest directly and its exit status cannot tell these cases apart. Measured on pytest 9.1.1 against the file the export actually produces:

case pytest exit JUnit kind
the agent's behaviour changed 1 failure
the trace bundle is missing 1 failure
a dependency is absent 1 failure
a fixture cannot be built 1 error
collection failed 2 error
no tests were collected 5 (none)

The first two rows are identical in both the status and the kind, so a pipeline keying on either one reports a missing artifact as a caught regression — the case a reader hits the first time they copy the test somewhere without the bundle, which is exactly the path docs/pytest-export.md sends them down.

What this adds

src/stepfork/export/receipt.py — the runner:

  • resolves the bundle the test names before pytest is launched, so a missing or unreadable bundle is could_not_execute and no failure is manufactured for it;
  • runs pytest as a subprocess with --junit-xml and derives the verdict from the counts, not from the exit status alone;
  • writes the receipt itself, after the child exits, from what the child produced. The generated test never writes a receipt: the interesting failures here are the ones where the check does not run at all.
receipt verdict exit code when
passed 0 exit 0, no failures, no errors
failed 1 the check ran and at least one test failed
could_not_execute 2 the artifacts are missing, pytest errored, nothing was collected, or pytest itself did not finish

reason is set whenever the verdict is could_not_execute and names the way it failed: bundle_missing, bundle_unreadable, test_unreadable, check_error, no_tests_collected, report_missing, pytest_unavailable.

The receipt records the verdict, the raw exit_status, the JUnit counts, the bundle_digest, the test_sha256, the command, the duration, the stepfork version and created_at. bundle_digest hashes the bundle's file contents in sorted relative-path order rather than the directory's metadata, so copying a bundle between machines leaves the digest unchanged and editing any file in it changes it.

src/stepfork/cli/run.py

stepfork run TEST [--bundle PATH] [--receipt PATH] [--json]
                  [--pytest-arg ARG]... [--timeout-seconds N]

docs/cli.md — the run section, the measured table above, and the exit-code convention updated to say that 2 also covers a check that could not execute.

What it does not catch

  • It pre-checks the artifacts it can name from the test's literals — the bundle. It does not pre-check what the test body imports: an entrypoint the local tree cannot resolve, or a dependency imported inside the test, raises inside the body, and pytest reports an exception raised there as failure, still indistinguishable from a real verdict. Resolving ENTRYPOINT before the launch the same way the bundle is resolved would close that; it is deliberately not in this PR.
  • It does not change the replay/export semantics discussed in Tester report (v0.2.0 @ 6507281): replay is solid; exported test can't pass when the fix changes tool args + minimize --mode bug #17. A test exported from a failure trace whose fix changes later tool arguments still replays as a divergence and still comes out failed — correctly, because it really did fail to replay. This PR makes the un-run case visible, not the unpassable one.

Tests

tests/unit/test_cli_run.py, 9 tests: each verdict path (missing bundle, pass, fail, collection error, empty file), the receipt naming the test and the bundle it rests on, the digest changing with the bundle's content, --json, and an explicit --bundle.

Full suite in a fresh venv: 550 passed, 4 failed — all four fail on main too, on ModuleNotFoundError: No module named 'langchain_core' (the optional dependency is not installed in this venv), and none of them touch the new code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants