Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,4 @@ examples/model_decision/.artifacts/

# Quickstart demo artifacts
examples/quickstart/.artifacts/
research_langgraph.py
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,35 @@ All notable changes to Stepfork are recorded here. The format follows
[Keep a Changelog](https://keepachangelog.com/), and this project adheres to
[Semantic Versioning](https://semver.org/).

## [Unreleased] - v0.2.0

Automatic failure minimization using dependency-aware delta debugging.

### Added

- **Failure minimization** via `stepfork minimize` command that automatically
finds the smallest reproducible trace preserving the original failure.
- Dependency-aware delta debugging algorithm that respects parent-child event
relationships and structural integrity.
- Failure signature detection supporting exception-based, replay-divergence,
and output-mismatch failures.
- Minimization orchestrator with timeout handling, nondeterminism detection,
and iteration limits.
- Comprehensive unit tests, integration tests, and CLI tests for minimization.
- Complete documentation in `docs/minimize.md` and updated CLI reference.

### Changed

- README roadmap updated to reflect v0.2 implementation status.

### Limitations

- Minimization requires replay to fail consistently. Highly nondeterministic
failures may not minimize reliably.
- Delta debugging uses binary search with dependency preservation. Very large
traces (1000+ events) may require significant replay attempts.
- Timeout handling uses SIGALRM on Unix systems; Windows support is limited.

## [0.1.0a6] - 2026-10-10

Behavioral assertions and selective live-model evaluation, with no trace
Expand Down
11 changes: 4 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -374,9 +374,6 @@ before v1.0.

- **Alpha on PyPI.** `pip install --pre stepfork` installs the current
pre-release; the API and trace format may still change.
- **No automatic failure minimization.** The exported regression test uses the
recorded steps. Reducing it to the smallest failing subset and fork-at-step
are planned work, not automatic behavior.
- **Limited integrations.** Optional adapters cover the OpenAI Python SDK
(synchronous and asynchronous, non-streaming `chat.completions.create`) and
LangGraph (synchronous and asynchronous `BaseChatModel` generation and
Expand Down Expand Up @@ -422,11 +419,11 @@ bundles, are in [docs/security.md](docs/security.md).
- Offline examples for both adapters
- Example GitHub Actions regression workflow

### v0.2
### v0.2 (Current)

- Failure minimization
- Fork-at-step
- Baselines
- **Failure minimization** - Automatic delta debugging to find minimal traces
- Fork-at-step (planned)
- Baselines (planned)

### v0.3

Expand Down
87 changes: 87 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Commands:
- [validate](#validate)
- [inspect](#inspect)
- [replay](#replay)
- [minimize](#minimize)
- [diff](#diff)
- [export](#export)

Expand Down Expand Up @@ -240,6 +241,92 @@ Common errors:

---

## minimize

Minimize a failing trace to the smallest reproducible subset.

```text
Usage: stepfork minimize [OPTIONS] PATH --entrypoint MODULE:FUNCTION
```

Arguments:

| Argument | Description |
|---|---|
| `PATH` | Path to the `.sftrace` directory bundle to minimize. Required. |
| `--entrypoint`, `-e` | Agent entrypoint as `MODULE:FUNCTION`. Required. |

Options:

| Option | Description |
|---|---|
| `--expect-output FILE` | JSON file with expected output for comparison. |
| `--mode`, `-m` | Replay mode: `frozen`, `live`, or `hybrid`. Default: `frozen`. |
| `--output`, `-o` | Path for the minimized `.sftrace` bundle. Default: `<path>-minimized.sftrace`. |
| `--overwrite` | Overwrite the output bundle if it exists. Default: `false`. |
| `--timeout SECONDS` | Timeout in seconds for each replay attempt. Default: `30.0`. |
| `--attempts N` | Number of attempts per subset test (for nondeterminism). Default: `1`. |
| `--help` | Show command help and exit. |

Minimization uses dependency-aware delta debugging to find the smallest set of
events that preserves the original failure. The process:

1. Detects the failure signature by replaying the original trace
2. Systematically tests smaller event subsets while preserving dependencies
3. Verifies the minimized trace reproduces the same failure
4. Saves the minimal trace and reports reduction statistics

Example:

```bash
echo '{"order_id": "ORD-1001", "should_notify": true}' > expected.json

stepfork minimize failure.sftrace \
--entrypoint quickstart:run_agent \
--expect-output expected.json \
--output minimal.sftrace
```

```text
Stepfork Minimize

Original trace: failure.sftrace
Agent: notify-agent
Events: 8
Entrypoint: quickstart:run_agent
Mode: frozen

Detecting failure signature...
Minimizing trace...

╭─ Minimization Report ─────────╮
│ Original Events 8 │
│ Minimized Events 5 │
│ Reduction 37.5% │
│ Replay Attempts 15 │
│ Failure Signature Output Mismatch │
│ Minimal Yes │
╰────────────────────────────────╯

✓ Minimized trace saved to: minimal.sftrace
```

Exit codes:

- `0` minimization succeeded and produced a valid minimal trace.
- `1` minimization failed (no failure detected, or minimization error).
- `2` invalid input, unreadable trace, or unresolvable entrypoint.

Common errors:

- `Failed to detect original failure signature: trace replay completed successfully; no failure to minimize` - the trace does not fail when replayed.
- `cannot resolve entrypoint ...` - the entrypoint specification is invalid or the module cannot be imported.
- `Minimized trace does not reproduce the original failure` - rare internal error during delta debugging.

See [docs/minimize.md](minimize.md) for detailed documentation and examples.

---

## diff

Compare the behavior of two recorded agent runs.
Expand Down
Loading
Loading