Skip to content

Fix: don't count a check with a masked exit status as a pass - #189

Merged
aayank13 merged 2 commits into
mainfrom
fix/masked-check-status
Oct 10, 2026
Merged

aayank13 merged 2 commits into
mainfrom
fix/masked-check-status

Conversation

@aayank13

@aayank13 aayank13 commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Fixes #150

Extraction strips trailing filters like | tail -20, || true and ; echo ok from a check before it classifies and stores it, but it still took the exit code of the whole shell line as the check's result. Without pipefail that is the status of tail or echo, so a failing test run counted as "the check passed after the last edit". A session that edited src/paging.rs and then ran cargo test 2>&1 | head -6, whose output stopped at running 2 tests and one ... ok line, became an active procedure, its task was recorded as succeeded, and an injection earlier in that task would have been scored as having helped.

Capture already reads the output of a masked check and flips the outcome when it finds a runner summary, but the trace did not say whether that happened, so extraction could not tell a pass shown by test result: ok. from an exit code that nobody checked. Tool calls now carry confirmed_by_output, set where the outcome is assembled (the shared trace builder for transcripts, and the hook journal) whenever the output decided it. The field is omitted when false, so existing traces and journals read as before.

Extraction now counts a successful check as a pass only when its exit status is really the check's, or when its output confirmed it. A run whose line continues past the check with a pipe, ||, ;, & or a newline and whose output showed nothing conclusive is treated as unknown: it is not the passing check after the last edit, it does not clear an earlier failure of the same check, and it is not offered as the command that "worked" in an avoid line. A chain with && still counts, since anything after it only runs when the check passed, and so do cd dir && ... prefixes and redirects like 2>&1, &> and >&2, which hide nothing. Separators inside quotes are ignored, so pytest -k 'paging | totals' is still a plain check. When the only check after the last edit hid its status, the task is rejected with "the last check hid its exit status: ..." and its outcome is left unjudged rather than marked failed, because nothing is known either way. An output-confirmed pass, such as cargo test 2>&1 | tail -20 ending in test result: ok., verifies the task like a plain run, and an output-confirmed failure stays a failure.

So a masked check now counts only when its output confirms the result. Which output counts as confirmation is decided at capture, and that rule needs #187, which should be merged first. Today cargo's Finished `test` profile ... target(s) line is read as a pass for any masked cargo command, including cargo test, where it only means the build finished. #187 limits that line to build commands, so a test run has to show its own summary.

Traces captured before this change have no marker, so a masked pass in them stays unknown if they are extracted again. That only loses procedures, it never invents one.

To reproduce, I built the binary, used a scratch TRODDEN_HOME and ingested Claude Code transcripts with an edit followed by a masked cargo test. On main, a run of cargo test 2>&1 | head -6 whose output stopped at running 2 tests and one ... ok line became an active procedure. With this change and the same output minus cargo's compile line, the task shows up in trodden rejected as "the last check hid its exit status: cargo test 2>&1 | head -6" and nothing is learned. If the output also has the Finished `test` profile ... target(s) in 0.42s line, the task is still learned on this branch alone, because capture reads that line as a pass; that is the case #187 fixes. A run of cargo test 2>&1 | tail -3 ending in test result: ok. 2 passed was rejected the same way while only extraction knew about masking, and with the marker it is learned as a procedure that verifies with cargo test.

The test that asserted npm test || true verifies a task now expects the opposite. New tests cover masked checks not passing on their own, not clearing an earlier failure and not hiding an earlier real pass, output-confirmed passes and failures, a task whose only check was masked being left unjudged, a masked run not being taught as a replacement, the journal recording when output decided a masked check, and which shell lines hide a check's status and which do not.

@aayank13
aayank13 merged commit 1b9b50e into main Oct 10, 2026
4 of 7 checks passed
@aayank13
aayank13 deleted the fix/masked-check-status branch October 10, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A check whose exit status is masked by | tail, || true or ; echo counts as a pass

1 participant