fix(cli,minimize): validate --mode upfront and reject invalid modes (exit 2)

README.md

@@ -31,7 +31,7 @@ If you're writing evals, Stepfork complements them: evals measure model quality
 </p>
 
 <p align="center">
-  Install the alpha: <code>pip install --pre stepfork</code>
+  Install the current release: <code>pip install stepfork</code>
 </p>
 
 Product goal:
@@ -117,21 +117,20 @@ change an external system; the allowlist does not sandbox it. See
 
 ## Installation
 
-Stepfork is published on PyPI. Install the current alpha with:
+Stepfork is published on PyPI. Install the current release with:
 
 ```bash
 # pip
-pip install --pre stepfork
+pip install stepfork
 
 # uv
-uv pip install --pre stepfork
+uv pip install stepfork
 ```
 
-The `--pre` flag is required while Stepfork is a pre-release. You can also
-install from the tagged GitHub repository:
+You can also install from source or a tagged release:
 
 ```bash
-pip install "git+https://github.com/utsab345/stepfork.git@v0.1.0a6"
+pip install "git+https://github.com/utsab345/stepfork.git@v0.2.0"
 ```
 
 Requires Python 3.11, 3.12, or 3.13. This installs the `stepfork` CLI and the
@@ -325,9 +324,12 @@ business outcome - it lives in `expected.json`, defined by a human.
 |---|---|
 | `stepfork validate [PATH] [--partial] [--verify-integrity]` | Check a `.sftrace` bundle structurally (and its integrity). |
 | `stepfork inspect [PATH] [--json] [--events] [--errors-only] [--step N]` | Review a bundle locally. |
-| `stepfork replay [PATH] --entrypoint MODULE:FUNCTION [--mode M]` | Rerun the entrypoint with dependencies answered from the trace. |
+| `stepfork replay [PATH] --entrypoint MODULE:FUNCTION [--mode M] [--allow-live-tool NAME] [--verbose]` | Rerun the entrypoint with dependencies answered from the trace. |
+| `stepfork minimize [PATH] --entrypoint MODULE:FUNCTION [--expect-output FILE] [--mode M] [--output PATH] [--overwrite] [--timeout SECONDS] [--attempts N]` | Minimize a failing trace to the smallest reproducible subset. |
+| `stepfork fork [PATH] [--at-step N \| --at-event ID] [--output PATH] [--overwrite]` | Fork a trace at a specific step (Python API for overrides). |
+| `stepfork score [TEST] [--expect-output PATH] [--bundle PATH] [--json]` | Score an exported expectation against a mutant battery. |
 | `stepfork diff [BASELINE] [CANDIDATE] [--json]` | Compare the behavior of two runs. |
-| `stepfork export [PATH] --pytest --entrypoint MODULE:FUNCTION [--expect-output JSON] [--expect-trajectory JSON] [--output FILE] [--overwrite]` | Generate an executable pytest regression test. |
+| `stepfork export [PATH] --pytest --entrypoint MODULE:FUNCTION [--expect-output JSON] [--expect-trajectory JSON] [--output FILE] [--overwrite] [--mode M] [--allow-live-tool NAME]` | Generate an executable pytest regression test. |
 
 The full reference (arguments, examples, expected output, and common errors)
 is in [docs/cli.md](docs/cli.md).
@@ -379,16 +381,16 @@ sources live in `docs/`.
 - [Release checklist](docs/release-checklist.md)
 - [PyPI publishing checklist](docs/pypi-publishing.md)
 - [Early adopter guide](docs/community/early-adopter-guide.md)
-- [Latest release notes (v0.1.0a6)](docs/releases/v0.1.0a6.md)
+- [Latest release notes (v0.2.0)](docs/releases/v0.2.0.md)
 - [Changelog](CHANGELOG.md)
 
 ## Security and Limitations
 
 Stepfork is experimental. The API, CLI, and `.sftrace` trace format may change
 before v1.0.
 
-- **Alpha on PyPI.** `pip install --pre stepfork` installs the current
-  pre-release; the API and trace format may still change.
+- **Beta status.** `pip install stepfork` installs the current release; the API
+  and trace format may still change before v1.0.
 - **Limited integrations.** Optional adapters cover the OpenAI Python SDK
   (synchronous and asynchronous, non-streaming `chat.completions.create`) and
   LangGraph (synchronous and asynchronous `BaseChatModel` generation and

docs/cli.md

@@ -1,7 +1,7 @@
 # CLI Reference
 
 The `stepfork` CLI manages `.sftrace` bundles. This page documents every
-command as of v0.1.0a6. Run `stepfork --help` for an up-to-date summary.
+command as of v0.2.0. Run `stepfork --help` for an up-to-date summary.
 
 ```text
 Usage: stepfork [OPTIONS] COMMAND [ARGS]...
@@ -22,6 +22,8 @@ Commands:
 - [inspect](#inspect)
 - [replay](#replay)
 - [minimize](#minimize)
+- [fork](#fork)
+- [score](#score)
 - [diff](#diff)
 - [export](#export)
 
@@ -261,7 +263,7 @@ Options:
 | Option | Description |
 |---|---|
 | `--expect-output FILE` | JSON file with expected output for comparison. |
-| `--mode`, `-m` | Replay mode: `frozen`, `live`, or `hybrid`. Default: `frozen`. |
+| `--mode`, `-m` | Replay mode: `frozen`, `live`, `forbidden`, `manual`, or `derived`. Default: `frozen`. |
 | `--output`, `-o` | Path for the minimized `.sftrace` bundle. Default: `<path>-minimized.sftrace`. |
 | `--overwrite` | Overwrite the output bundle if it exists. Default: `false`. |
 | `--timeout SECONDS` | Timeout in seconds for each replay attempt. Default: `30.0`. |

docs/community/early-adopter-guide.md

@@ -32,14 +32,13 @@ Stepfork is on PyPI. Install the current alpha:
 ```bash
 python -m venv .venv
 source .venv/bin/activate
-pip install --pre stepfork
+pip install stepfork
 pip install pytest
 stepfork --version
 ```
 
-The `--pre` flag is required while Stepfork is a pre-release. Alternatively,
-install the tagged repository build:
-`pip install "git+https://github.com/utsab345/stepfork.git@v0.1.0a6"`.
+Alternatively, install the tagged repository build:
+`pip install "git+https://github.com/utsab345/stepfork.git@v0.2.0"`.
 
 Requires Python 3.11, 3.12, or 3.13.
 

docs/getting-started.md

@@ -6,17 +6,16 @@ This guide installs Stepfork and walks through the full workflow with a
 
 ## Install
 
-Install the current alpha from PyPI:
+Install the current release from PyPI:
 
 ```bash
-pip install --pre stepfork
+pip install stepfork
 ```
 
-The `--pre` flag is required while Stepfork is a pre-release. To install from
-the tagged GitHub repository instead:
+To install from the tagged GitHub repository instead:
 
 ```bash
-pip install "git+https://github.com/utsab345/stepfork.git@v0.1.0a6"
+pip install "git+https://github.com/utsab345/stepfork.git@v0.2.0"
 ```
 
 Requires Python 3.11, 3.12, or 3.13. This installs the `stepfork` CLI and the

docs/index.md

@@ -73,4 +73,4 @@ in this repository.
 
 Stepfork is experimental. See the
 [changelog](https://github.com/utsab345/stepfork/blob/main/CHANGELOG.md) and
-the [v0.1.0a6 release notes](releases/v0.1.0a6.md) for what shipped.
+the [v0.2.0 release notes](releases/v0.2.0.md) for what shipped.

docs/minimize.md

@@ -85,7 +85,7 @@ stepfork minimize <trace> --entrypoint MODULE:FUNCTION [OPTIONS]
 | Option | Description | Default |
 |--------|-------------|---------|
 | `--expect-output FILE` | JSON file with expected output for comparison | None |
-| `--mode`, `-m` | Replay mode: `frozen`, `live`, or `hybrid` | `frozen` |
+| `--mode`, `-m` | Replay mode: `frozen`, `live`, `forbidden`, `manual`, or `derived` | `frozen` |
 | `--output`, `-o` | Path for the minimized `.sftrace` bundle | `<trace>-minimized.sftrace` |
 | `--overwrite` | Overwrite the output bundle if it exists | `false` |
 | `--timeout SECONDS` | Timeout for each replay attempt | `30.0` |

docs/pytest-export.md

@@ -9,6 +9,37 @@ A trace records what the agent *did*, including the bug. If you generate a test
 that asserts the recorded output, it passes against the very code that produced
 the recording, which is not a regression test.
 
+## Frozen-replay contract and limitations
+
+Stepfork's pytest export uses frozen replay: the generated test executes your
+trusted entrypoint with instrumented tool and LLM calls answered from the recorded
+trace. The contract is strict to preserve determinism and prevent silent
+acceptance of unexpected behavior.
+
+**Supported when:** the fixed implementation makes the same instrumented calls in
+the same order with the same inputs as recorded in the failure trace. The
+regression test can then assert the corrected output against the trace.
+
+**Not supported when:** the fix legitimately changes later tool arguments,
+invokes a tool not present in the recorded trace, or omits/moves recorded calls.
+In these cases, frozen replay will raise a divergence (e.g., `ReplayMismatchError`
+for changed inputs, or `ReplayExhaustedError`/unmatched calls for sequence
+changes). The test generated from the failure trace **cannot pass** on the fixed
+code because replay cannot satisfy the recorded sequence.
+
+When a fix changes the call sequence or arguments, record the *fixed* run and
+export a test from that trace instead. The exported test will fail on the buggy
+code (by replay divergence) and pass on the fixed code. `stepfork score` can help
+assess whether an expectation is claim-pinned (rejects the recorded run) or
+run-pinned (accepts it).
+
+`score` assumes the bundle is the failing run. Against a test exported from the
+*fixed* run, an expectation copied from that run reads run-pinned (exit 1) by
+construction. That verdict describes where the bundle came from, not whether the
+test is worth keeping.
+
+---
+
 The regression test needs a separate **expected outcome**: the behavior you want
 after the bug is fixed. Pass it with `--expect-output`.
 

docs/releases/v0.2.0.md

@@ -0,0 +1,46 @@
+# Stepfork v0.2.0 - Behavioral Replay
+
+This current release adds trajectory assertions, async instrumented boundaries, and an
+opt-in hybrid mode for selected live LLM calls. Existing `.sftrace` bundles
+remain compatible; frozen replay remains the default.
+
+## Changes
+
+- Assert tool calls, absence, counts, JSON arguments, order, sequences, and a
+  maximum dependency-step count against the execution that actually replayed.
+  Generated pytest tests can include a reviewed JSON trajectory expectation.
+- Record and frozen-replay async tools, LLM requests, and agent entrypoints.
+  Supported OpenAI and LangGraph adapters now include async paths.
+- Re-execute selected LLMs in hybrid replay while recorded tools stay frozen.
+  The [offline model-decision example](https://github.com/utsab345/stepfork/tree/v0.2.0/examples/model_decision)
+  uses a deterministic fake provider and shows a failing regression before a
+  prompt fix and a passing one afterward. It does not establish real-model
+  quality improvement.
+- Live tool execution now requires `allow_live_tools={"name"}` or
+  `--allow-live-tool name`. Previously, `mode="live"` ran every matching tool.
+  See [migration guidance](../replay.md#modes).
+
+Hybrid replay is **not deterministic**. Selected live models can incur network
+calls, token costs, and latency. The tool allowlist controls instrumented tool
+bodies by name; it does not sandbox arbitrary code or uninstrumented side
+effects. Concurrent async calls still match one strict invocation sequence,
+while task scheduling remains outside Stepfork's control.
+
+## Installation
+
+```bash
+pip install stepfork
+```
+
+Requires Python 3.11, 3.12, or 3.13.
+
+## Verification
+
+- 444 tests passed on Python 3.11, 3.12, and 3.13. Python 3.12 combined
+  statement/branch coverage was 92%.
+- Ruff, MyPy, strict documentation builds, six offline demos, source and wheel
+  builds, and an installed-wheel smoke of the new APIs passed locally.
+
+## Feedback
+
+https://github.com/utsab345/stepfork/issues

src/stepfork/cli/minimize.py

@@ -49,7 +49,7 @@ def minimize_command(
         typer.Option(
             "--mode",
             "-m",
-            help="Replay mode: frozen (default), live, or hybrid",
+            help="Replay mode: frozen (default), live, forbidden, manual, or derived",
         ),
     ] = "frozen",
     output: Annotated[
@@ -126,6 +126,13 @@ def minimize_command(
 
     # Run minimization
     console.print("[yellow]Detecting failure signature...[/yellow]")
+    try:
+        from stepfork.trace.replay_policy import ReplayPolicy
+
+        ReplayPolicy(mode)
+    except ValueError:
+        console.print(f"[red]Invalid replay mode {mode!r}.[/red]")
+        raise typer.Exit(2) from None
     engine = MinimizationEngine(
         trace,
         entrypoint,

src/stepfork/minimize/engine.py

@@ -88,6 +88,20 @@ def minimize(self) -> MinimizationResult:
         """
         # Step 1: Detect the original failure signature
         try:
+            try:
+                from stepfork.trace.replay_policy import ReplayPolicy
+
+                ReplayPolicy(self.mode)
+            except ValueError as exc:
+                return MinimizationResult(
+                    success=False,
+                    original_event_count=len(self.trace.events),
+                    minimized_event_count=None,
+                    replay_attempts=self.total_replay_attempts,
+                    preserved_failure=False,
+                    is_minimal=False,
+                    error_message=f"Invalid replay mode {self.mode!r}: {exc}",
+                )
             original_signature = detect_failure_signature(
                 self.trace,
                 self.entrypoint,

src/stepfork/minimize/signature.py

@@ -18,6 +18,7 @@
     ReplayError,
     ReplayExhaustedError,
     ReplayMismatchError,
+    ReplayPolicyError,
     ReplaySession,
 )
 from stepfork.trace.canonical import canonical_json_bytes
@@ -59,15 +60,29 @@ def detect_failure_signature(
             f"cannot resolve entrypoint {entrypoint!r}: {exc}"
         ) from exc
 
+    # Harness setup errors (invalid mode, bad allowlists) must surface as
+    # detection failures, never become a minimizable signature.
+    try:
+        session = ReplaySession.from_trace(trace, mode=mode)
+    except Exception as exc:
+        raise FailureDetectionError(f"cannot create replay session: {exc}") from exc
+
     # Attempt replay and capture the failure
     exception: BaseException | None = None
     actual_output: Any = None
     divergence_error: ReplayError | None = None
 
     try:
-        with ReplaySession.from_trace(trace, mode=mode) as replay:
+        with session as replay:
             actual_output = entrypoint_fn()
             divergence_error = replay.divergence
+    except ReplayPolicyError as exc:
+        # The replay harness refused execution (forbidden/manual/derived mode,
+        # unauthorized live tool). The agent never ran its check, so this is
+        # not a failure signature to minimize. See issue #17.
+        raise FailureDetectionError(
+            f"replay policy refused the attempt: {exc}"
+        ) from exc
     except RecordedDependencyError as exc:
         exception = exc
     except ReplayMismatchError as exc:
@@ -80,6 +95,13 @@ def detect_failure_signature(
         # Application exception during replay
         exception = exc
 
+    # A policy refusal the agent swallowed internally still poisons the
+    # outcome; refuse instead of recording it as a divergence signature.
+    if isinstance(divergence_error, ReplayPolicyError):
+        raise FailureDetectionError(
+            f"replay policy refused the attempt: {divergence_error}"
+        ) from divergence_error
+
     # Determine the failure signature based on what happened
     if exception is not None:
         return _signature_from_exception(exception, divergence_error)