Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -437,6 +437,91 @@ Common errors:

---

## score

Score an exported expectation against a mutant battery derived from the
recorded run, without executing anything.

```text
Usage: stepfork score [OPTIONS] TEST
```

An exported test asserts `EXPECTATION`. Two things can satisfy that assertion
and pass the same green run: the claim you meant to check, or a copy of the run
the recording came from. `score` derives the battery that separates them -- the
claim-drawn row is the recorded run's own output, and the code-drawn rows
perturb the expectation's leaves -- and reports which rows the expectation
rejects.

The test file is **read, never executed**: it is parsed with `ast`, JSON-shaped
values are read with `ast.literal_eval`, and the bundle path is read from the
expressions the exporter writes. Nothing is imported and no entrypoint runs.

Arguments:

| Argument | Description |
|---|---|
| `TEST` | Exported pytest regression test. Required. |

Options:

| Option | Description |
|---|---|
| `--expect-output <path>` | Score this expectation instead of the one the test embeds. |
| `--bundle <path>` | Bundle to read the recorded run from. Default: the `TRACE_PATH` the test names. |
| `--json` | Print the score as JSON. |
| `--help` | Show command help and exit. |

A claim-pinned expectation rejects the recorded run:

```text
battery: 1 claim-drawn + 6 code-drawn mutant(s), derived from test_notify_regression.py and notify.sftrace

claim the recorded run killed (first difference at carrier)
code carrier: with a suffix killed (first difference at carrier)
code order_id: with a suffix killed (first difference at order_id)
code should_notify: flipped killed (first difference at should_notify)

claim-pinned: rejected 7/7 mutants, including every claim-drawn mutant
pins: 1 live (carrier); 2 leaf(s) reproduce the run (order_id, should_notify)
```

An expectation that has drifted into the recording accepts it, which is the
finding: the generated test would pass against the code that produced the
recording.

```text
run-pinned: the expectation accepts the claim-drawn mutant(s) the recorded run, so it encodes the recorded run rather than the claim
The generated test passes against the code that produced the recording. Treat the expectation as an unapproved oracle.
pins: 0 live (none); 3 leaf(s) reproduce the run (carrier, order_id, should_notify)
```

The `pins` line answers a second question the verdict does not: which leaves
carry a value the run did not produce (live pins), and which merely reproduce
the run. A leaf that reproduces the run is not a defect -- a fix does not change
every field -- but it is not evidence of anchoring either.

Exit codes:

| Code | Meaning |
|---|---|
| `0` | `claim_pinned`: the expectation rejects the recorded run. |
| `1` | `run_pinned`: the expectation accepts it; it is an unapproved oracle. |
| `2` | Cannot answer: no expectation to score, no recorded run to derive a claim-drawn mutant from, or a test file that cannot be read. |

Common errors:

- `<path>:<line>: EXPECTATION is not a literal value ...` -- the test was
hand-edited; `score` reads the literals the export wrote.
- `cannot read the bundle path from this expression; pass --bundle ...` -- the
`TRACE_PATH` expression is outside the shapes the exporter writes.
- `trace bundle not found: <path>` -- the bundle the test names is not at that
path. Copy it beside the test, or pass `--bundle`.
- `no expectation to score` -- the test asserts the recorded run without an
`--expect-output`, so there is no claim for a mutant to separate from it.

---

## Error handling

Command output is sanitized before display: secrets in trace-derived text
Expand Down
56 changes: 56 additions & 0 deletions docs/pytest-export.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,62 @@ not embed or copy the fixture. If moving the test to another project or CI,
copy the bundle with it and verify `TRACE_PATH` resolves there. These are test
environment requirements; `pytest` is not a Stepfork runtime dependency.

## Is the expectation still anchored? (`stepfork score`)

An expectation can be two different things and pass the same green run: the
claim you meant to assert, or a copy of the run you wrote it from. `stepfork
score` separates them, and it does it by derivation rather than by execution.

```bash
stepfork score test_booking_regression.py
stepfork score test_booking_regression.py --expect-output expected.json --json
```

Two values are already at hand when the export is prepared, and both are read
back rather than re-derived:

- the **expectation** the generated test asserts;
- the **recorded run's output** (`RunEnd.output` in the bundle), which is the
value the buggy code that produced the recording actually returned.

The claim-drawn row is that second value verbatim: the run's own output is the
most plausible output of a variant implementation, because an implementation did
produce it. It is the row that discriminates. A claim-pinned expectation rejects
it -- the run it came from is the bug, so the expectation must fail against it.
A run-pinned expectation accepts it, and that acceptance is the finding: the
granularity of the comparison means the generated test would pass against the
code that produced the recording.

The remaining rows are **controls**, not discriminators: perturbations of the
expectation's own leaves (an integer moved by one, a string suffixed, a value
replaced by `null`). A run-pinned expectation rejects those too, so they cannot
separate the two readings by themselves; what they do is make the contrast
legible, because a battery in which every control is killed and the claim-drawn
row survives reads differently from one in which everything is killed.

`score` also reports **pin coverage**: which leaves of the expectation carry a
value the run did not produce (live pins), and which merely reproduce the run.
A leaf that reproduces the run is not a defect -- a fix does not change every
field -- but it is not evidence of anchoring either, which is why it is reported
rather than counted.

Exit codes: `0` when the expectation is claim-pinned, `1` when it is run-pinned,
`2` when the question cannot be answered.

Three limits are worth stating plainly.

- **No recorded run, no claim-drawn row.** A trace whose run did not complete
carries no output, so there is nothing to draw the discriminating row from.
The battery is then the controls alone and the verdict is `undetermined` -- a
reading, not a failure, and the honest one for a battery that cannot separate
the two.
- **The bundle must be there.** `score` reads the recorded run from the bundle
the test names, so it has the same requirement the test has. Pass `--bundle`
to point at one elsewhere.
- **Nothing is executed.** Neither the test file nor the entrypoint runs: the
file is parsed, the literals are read, and each row is scored with the same
canonical comparison the generated test uses.

## Generated test

The generated file is produced from a fixed, reviewed template. Trace payloads
Expand Down
12 changes: 2 additions & 10 deletions src/stepfork/cli/export.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,9 @@
ExportExistsError,
export_pytest_test,
)
from stepfork.export.mutants import recorded_output
from stepfork.trace import (
JsonValue,
RunEnd,
RunStatus,
Trace,
TraceStorageError,
validate_bundle,
Expand Down Expand Up @@ -156,7 +155,7 @@ def export_command(
expectation = _load_expectation(expect_output)
has_expectation = True
else:
expectation = _recorded_output(trace)
expectation = recorded_output(trace)
has_expectation = expectation is not None
warning = RECORDED_OUTPUT_WARNING

Expand Down Expand Up @@ -223,13 +222,6 @@ def _load_expectation(path: Path) -> JsonValue:
return redact_json(payload).value


def _recorded_output(trace: Trace) -> JsonValue | None:
for event in reversed(trace.events):
if isinstance(event, RunEnd) and event.run_status is RunStatus.COMPLETED:
return event.output
return None


def _safe_stem(path: Path) -> str:
stem = re.sub(r"\W+", "_", path.stem).strip("_")
return stem or "trace"
Expand Down
2 changes: 2 additions & 0 deletions src/stepfork/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from stepfork.cli.inspect import inspect_command
from stepfork.cli.minimize import minimize_command
from stepfork.cli.replay import replay_command
from stepfork.cli.score import score_command
from stepfork.cli.validate import validate_command

app = typer.Typer(
Expand All @@ -28,6 +29,7 @@
app.command("inspect")(inspect_command)
app.command("minimize")(minimize_command)
app.command("replay")(replay_command)
app.command("score")(score_command)
app.command("validate")(validate_command)


Expand Down
Loading
Loading