Skip to content

Fix: judge a command by its output only when it is a masked check - #187

Merged
aayank13 merged 2 commits into
mainfrom
fix/output-verdict-only-for-checks
Oct 10, 2026
Merged

aayank13 merged 2 commits into
mainfrom
fix/output-verdict-only-for-checks

Conversation

@aayank13

@aayank13 aayank13 commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Fixes #151

Any command with ;, |, & or a newline was treated as hiding its exit code, so its output was scanned for test-runner summaries even when the command was not a check at all. sed -n '1,40p' src/report.rs; echo done exits 0 and prints a source file. If that file had a line like error: unknown customer id, the call was marked failed and the line was kept as its error signature. Ingesting such a session put error: unknown customer id into the learned procedure's trigger entities, so error recall could later match on a line of the user's own code.

The output now decides only when the exit code is masked and the first stage runs a check: a known test, build or lint tool (cargo test, pytest, jest, tsc, go test and so on, also behind npx or uv run), a project task runner (npm run, make, just, mvn, gradle, rake), or a script whose name says it checks (./scripts/test.sh, bash ci.sh). Everything else keeps the exit code the agent reported, in both directions: a read can no longer be turned into a failure, and a failed command can no longer be turned into a success because it printed test result: ok.. The output summaries are the formats of those runners, so tying the verdict to them is what makes it trustworthy.

Cargo's Finished ... target(s) line now counts as a pass only for build-type commands (cargo build, cargo check, cargo clippy). For cargo test or cargo nextest it only means the build finished, so a masked cargo test 2>&1 | head -6 cut off after running 2 tests was judged a pass; test runners now need a real result line such as test result: ok..

sed and awk reads now count as reads too, unless they edit in place (sed -i, sed --in-place, gawk -i inplace). That also keeps the Codex and Copilot fallbacks, which flag a run with an error-looking line when no exit code is known, from flagging a file printed with sed.

The list of check runners overlaps with the one the extractor uses to spot verify steps. I kept a separate list here because capture cannot depend on the extractor; moving the shared part into capture can be a follow-up. A masked check that starts with something else, like echo start; cargo test | tail, is no longer judged by its output and keeps its exit code.

To reproduce, I built the binary and ingested a Claude Code transcript into a scratch home: a sed -n '1,40p' src/report.rs; echo done call with is_error: false whose output contains error: unknown customer id, then an edit and a passing cargo test. Before the fix, trodden show listed error: unknown customer id as an error_signature entity of the new procedure. After it, the procedure has only the path, symbol and command entities. New tests cover reads and prints with failing-looking output staying out of the verdict, masked checks (including behind cd, npx, uv run, make and scripts) still being judged, the Claude Code outcome of the sed call, and sed/awk being classified as reads unless they edit in place, and a masked cargo test | head whose output has the Finished line and partial test output but no result line not being judged a pass. All of them fail without the fix.

@aayank13
aayank13 merged commit 20b7cdb into main Oct 10, 2026
5 of 8 checks passed
@aayank13
aayank13 deleted the fix/output-verdict-only-for-checks branch October 10, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Output heuristics turn a command with a known exit code 0 into a failure

1 participant