Repository navigation
Fix: don't count a check with a masked exit status as a pass - #189
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #150
Extraction strips trailing filters like
| tail -20,|| trueand; echo okfrom a check before it classifies and stores it, but it still took the exit code of the whole shell line as the check's result. Withoutpipefailthat is the status oftailorecho, so a failing test run counted as "the check passed after the last edit". A session that editedsrc/paging.rsand then rancargo test 2>&1 | head -6, whose output stopped atrunning 2 testsand one... okline, became an active procedure, its task was recorded as succeeded, and an injection earlier in that task would have been scored as having helped.Capture already reads the output of a masked check and flips the outcome when it finds a runner summary, but the trace did not say whether that happened, so extraction could not tell a pass shown by
test result: ok.from an exit code that nobody checked. Tool calls now carryconfirmed_by_output, set where the outcome is assembled (the shared trace builder for transcripts, and the hook journal) whenever the output decided it. The field is omitted when false, so existing traces and journals read as before.Extraction now counts a successful check as a pass only when its exit status is really the check's, or when its output confirmed it. A run whose line continues past the check with a pipe,
||,;,&or a newline and whose output showed nothing conclusive is treated as unknown: it is not the passing check after the last edit, it does not clear an earlier failure of the same check, and it is not offered as the command that "worked" in an avoid line. A chain with&&still counts, since anything after it only runs when the check passed, and so docd dir && ...prefixes and redirects like2>&1,&>and>&2, which hide nothing. Separators inside quotes are ignored, sopytest -k 'paging | totals'is still a plain check. When the only check after the last edit hid its status, the task is rejected with "the last check hid its exit status:..." and its outcome is left unjudged rather than marked failed, because nothing is known either way. An output-confirmed pass, such ascargo test 2>&1 | tail -20ending intest result: ok., verifies the task like a plain run, and an output-confirmed failure stays a failure.So a masked check now counts only when its output confirms the result. Which output counts as confirmation is decided at capture, and that rule needs #187, which should be merged first. Today cargo's
Finished `test` profile ... target(s)line is read as a pass for any masked cargo command, includingcargo test, where it only means the build finished. #187 limits that line to build commands, so a test run has to show its own summary.Traces captured before this change have no marker, so a masked pass in them stays unknown if they are extracted again. That only loses procedures, it never invents one.
To reproduce, I built the binary, used a scratch
TRODDEN_HOMEand ingested Claude Code transcripts with an edit followed by a maskedcargo test. On main, a run ofcargo test 2>&1 | head -6whose output stopped atrunning 2 testsand one... okline became an active procedure. With this change and the same output minus cargo's compile line, the task shows up introdden rejectedas "the last check hid its exit status:cargo test 2>&1 | head -6" and nothing is learned. If the output also has theFinished `test` profile ... target(s) in 0.42sline, the task is still learned on this branch alone, because capture reads that line as a pass; that is the case #187 fixes. A run ofcargo test 2>&1 | tail -3ending intest result: ok. 2 passedwas rejected the same way while only extraction knew about masking, and with the marker it is learned as a procedure that verifies withcargo test.The test that asserted
npm test || trueverifies a task now expects the opposite. New tests cover masked checks not passing on their own, not clearing an earlier failure and not hiding an earlier real pass, output-confirmed passes and failures, a task whose only check was masked being left unjudged, a masked run not being taught as a replacement, the journal recording when output decided a masked check, and which shell lines hide a check's status and which do not.