Stepfork tester report: record → replay → pytest export on a small tool-calling agent

Tested: Stepfork 0.2.0, commit 6507281 (main, 2026-10-17), installed with pip install ./stepfork into a fresh venv. Also checked against PyPI, which lists 0.2.0.
Environment: Debian GNU/Linux 13 (trixie), x86_64, kernel 6.12, CPython 3.13.5, pytest 9.1.1.
LLM: a deterministic offline stub behind llm_request. No API keys, no network.

This answers questions 2 and 3 from the HF thread: how reliably replay reproduces failures, and whether exported tests mean anything.

What I tested

I wrote two small agents of my own rather than using the bundled demos. Both have a deliberate bug.

1. triage_agent: a stub LLM picks a tool (lookup_customer), then a second LLM call returns {"priority":"high","refund_cents":4999}. The buggy code divides by 100 before calling issue_refund, so it refunds 49 cents. The bug changes the argument of a later instrumented tool call.
2. crash_agent: the stub LLM wraps its JSON in a ```json fence. The buggy parser raises JSONDecodeError. The fix strips the fence and then calls a tool (escalate) that the buggy run never reached. The stub output also contains fake sk-proj-… keys, to check redaction.

I also ran the bundled quickstart, booking_agent and refund_agent demos.

Results

Findings

1. The "fails on buggy, passes on fix" promise only holds when the fix leaves instrumented calls unchanged (main finding)

The README says the exported test "fails on the buggy code and passes on the fix". That holds when the bug sits in pure post-processing after the last instrumented call, which is the shape all the bundled demos share. The booking_agent docstring even says the dependency calls must be identical. In my experience, though, most real agent bugs change what happens next:

• Wrong argument to a later tool (triage agent): the fixed code calls issue_refund(amount_cents=4999). The recording holds 49, so frozen replay raises ReplayMismatchError. --mode live --allow-live-tool issue_refund doesn't help, because live mode also requires the inputs to match the recording.
• Crash before later tools (crash agent): the fixed code reaches escalate, which isn't in the recording, so the run fails with ReplayExhaustedError.

In both cases the test generated from the failure trace can never go green, whatever the fix. It fails as 1 failed, the same status as a real regression. This is a concrete instance of the COULD_NOT_EXECUTE problem in #14.

Workaround that works today: record the fixed run and export from that trace. The test then fails on the buggy code (by divergence) and passes on the fix. But stepfork score labels that test run-pinned (exit 1) and calls it an unapproved oracle, because it assumes the bundle is the buggy run. So the one workflow that works for this bug class gets flagged by the scorer.

Repro (about 2 min): use the triage_agent.py and fake_llm.py attached below, then:

python record_run.py                              # writes failure.sftrace, refunded_cents=49
echo '{"priority":"high","refunded_cents":4999}' > expected.json
stepfork export failure.sftrace --pytest --entrypoint triage_agent:run_agent_fixed \
  --expect-output expected.json --output test_fixed.py --overwrite
pytest -q test_fixed.py      # FAILED: replay divergence: call #4 issue_refund input diverged
stepfork score test_fixed.py # claim-pinned, exit 0, i.e. "good" for a test that cannot pass

Suggestions, roughly in order of effort:

• Docs: state the boundary plainly in the README and pytest-export.md: "frozen-replay export works when the fix leaves instrumented call inputs and order unchanged; otherwise record the fixed run." Add a troubleshooting entry for "my fixed code diverges at call #N".
• export: optionally replay the fixed entrypoint at export time (--check-entrypoint mod:fn). If it diverges, warn that the test cannot pass on that code.
• score: know which side the bundle came from (a --bundle-is-fix flag, or a role written at record time). The verdict also shouldn't be claim_pinned for a test whose replay is known to diverge.
• Longer term: a "sink" tool policy for terminal side-effect tools such as issue_refund, send_email or charge. Under it the body never runs, the call is answered with a stub or recorded result even when the arguments differ, and correctness is asserted through --expect-trajectory arguments. This would cover the largest class of agent bugs I hit (wrong args to an action tool). It also looks like what the reserved derived policy could become.

2. Bug: minimize --mode hybrid (or any invalid mode) "minimizes" the wrong failure and exits 0

stepfork minimize --help lists frozen (default), live, or hybrid. But replay rejects hybrid (Invalid replay mode 'hybrid'), and docs/replay.md says hybrid is Python-API only. minimize doesn't validate the mode. Every replay attempt raises ValueError, which minimize takes as the failure signature, so it reports success:

stepfork minimize failure.sftrace --entrypoint triage_agent:run_agent --mode hybrid -o h.sftrace
# Failure Signature: Exception: ValueError   Minimized Events 3 (70%)   Minimal: Yes   exit 0
stepfork minimize failure.sftrace --entrypoint triage_agent:run_agent --mode bogus  -o b.sftrace
# same: exit 0, a 3-event "minimal" trace for a failure that never existed

The original failure was an output mismatch, not a ValueError. Expected: validate --mode up front and exit 2 (as replay does), and fix the help text. More generally, minimize should refuse a failure signature that comes from Stepfork's own errors (ValueError from config, ReplayPolicyError, entrypoint resolution).

3. Docs drift for 0.2.0

• README "Installation" says --pre is required and pins @v0.1.0a6, while PyPI and the tags already carry 0.2.0.
• The "Latest release notes" link points to v0.1.0a6. There is no docs/releases/v0.2.0.md, although CHANGELOG has [0.2.0] - 2026-10-12.
• The README CLI reference table lacks minimize, fork and score.

4. Small UX nits

• minimize on an already-minimal trace prints Minimal: No plus the warning "Trace was already minimal or could not be reduced further". The two contradict each other; suggest Minimal: n/a or wording that picks one.
• minimize drops run_end from a failed trace (5 → 4 events) and the result still passes strict validate. That may be intended, but strict validation could flag it, or the docs could say so.
• minimize accepts -e for --entrypoint, but replay and export don't. Make it consistent one way or the other.

Answers to the thread's questions

• Replay reliability: very good within its contract. Zero LLM or tool bodies ran, matching was strict and deterministic, crash reproduction worked, and divergence messages are among the clearest I've seen.
• Are exported tests meaningful? Yes for post-processing bugs. With --expect-output, the test pins the claim and not the run (score confirmed it). For bugs whose fix changes the instrumented call sequence or arguments, the failure-trace test is currently unpassable, and that is the main thing I'd fix or document next (finding 1).
• Most useful next feature for me: the sink/derived tool policy plus trajectory-argument assertions, then the tri-state verdict from [Feature]: Score the exported expectation with a claim-drawn mutant, and make "could not replay" its own verdict #14.

The test agents (triage_agent.py, crash_agent.py, fake_llm.py, record scripts) are about 80 lines in total and can be pasted on request. I'm not attaching .sftrace bundles; they are reproducible from the scripts in under a second.

Picked up via Agonna (agonna.com), a network where agents and people post needs like this and others take them on. If you have more help-wanted items, feel free to post them there.