minimize: refuse signatures produced before any recorded call is consumed #22

Activity

1. how2how2how2-arch commented on Oct 19, 2026

   

   how2how2how2-arch

   on Oct 19, 2026

   Contributor

   More actions

   Measured the arms before arguing about the rule. Two of them turned out to be the same arm, which changes what the decision is about.

   Method. Recorded a two-call trace with the library's own record() (an llm_request, a @trace_tool lookup_customer, then a ValueError — so the recorded failure comes after both calls, 2/2), and ran seven entrypoints through minimize on this branch. Consumption read from the public ReplaySession.pending (no private _cursor needed; len(session.calls) gives the total), recorded failure position from trace.events — count of llm_request/tool_call events before the error event, which is 2 for my trace and 1 for tests/golden/failed_tool_run.sftrace.

   arm	consumed	exception	recorded type	exit today
   A import error	0/2	ModuleNotFoundError	ValueError	0
   B unrelated setup error	0/2	RuntimeError	ValueError	0
   H setup error, recorded type	0/2	ValueError	ValueError	0
   E early failure, recorded type	0/2	ValueError	ValueError	0
   F early failure, other type	0/2	KeyError	ValueError	0
   G all calls consumed, other type	2/2	TypeError	ValueError	0
   C recorded failure reproduced	2/2	ValueError	ValueError	0
   D returns without failing	0/2	—	ValueError	1 (refused)

   A and H are not distinguishable, and neither are E and H. All three fail before any recorded call; H and E even raise the recorded type. E and H differ only in the story I attach to them ("the fix broke early" vs "setup could not build the fixture") — from inside minimize they are the same observation. So the question isn't "which exceptions are harness errors", it is "may this run be reported as a minimization at all", and on that question every arm with consumed == 0 while the recorded failure came after ≥1 call answers no.

   What the current behaviour costs is not just a wrong label. For arm A the emitted bundle is

   original:  7 events  run_start llm_request llm_response tool_call tool_result error run_end
   minimized: 3 events  run_start llm_request llm_response
   

   The recording's tool_call, its tool_result and the recorded error are gone, and the bundle is written with exit 0. A "minimal" trace was derived from a failure that never reached those events — the artifact loses the events that carry the recorded failure.

   The interesting part is that the invariant already exists one level down. minimize/engine.py:189 refuses any subset that cannot reproduce an execution failure:

   if not subset.includes_dependency_calls:
       # Subsets without any dependency calls cannot reproduce execution failures
       return False

   That is the same intent as the suggestion in the issue, expressed as "the subset contains a dependency call". The missing half is "the replay reached it" — which is why arm A's bundle keeps an llm_request the entrypoint never consumed. Applying the same intent to the original attempt closes the class instead of one instance of it.

   Proposal, small enough to be a patch rather than a policy:

   refuse when consumed == 0 and calls_before_recorded_error >= 1 and no divergence
     → exit 2, message names both readings, write no bundle
   

   Wording that keeps it honest: "the replay failed before reaching any recorded call: either the fixture could not be built, or the change moved the failure earlier than the recording. No bundle written." Exit 2 rather than 1 so it is separable from "no failure to minimize" (arm D), which really does mean the trace replays clean.

   This does refuse arm E, a real regression — unavoidable, since E and H are the same observation. That is the cost of not silently rewriting the trace: the caller gets told which two things it might be, instead of getting a 3-event bundle labelled minimal.

   Arms C and G keep working under this rule (both consume the recorded calls), and a trace whose own error event precedes every call has calls_before_recorded_error == 0, so the condition degenerates to "never refuse" there.
2. utsab345 commented on Oct 19, 2026

   

   utsab345

   on Oct 19, 2026

   Last edited by utsab345

   OwnerAuthor

   More actions

   @how2how2how2-arch Accepted, thanks for the measurement. The rule as proposed:

   refuse when consumed == 0 and calls_before_recorded_error >= 1 and no divergence
   exit 2, message names both readings, write no bundle
   

   Two notes for the patch:

   1. Exit 2 already means invalid mode in the CLI, the message differs so sharing the code is fine. The ReplayPolicyError refusals from fix(cli,minimize): validate --mode upfront and reject invalid modes (exit 2) #19 exit 1, that is a different class (divergence is set), so leave them as they are.
   2. detect_failure_signature reads session.divergence only on the success path today. The patch should read pending and divergence after the attempt even when an exception propagated.

   Your arm table and wording are the acceptance criteria, including refusing arm E. If you would like to open the PR, happy to review it.
3. utsab345

   closed this as completedin #23on Oct 19, 2026