Repository navigation
Fix: judge a command by its output only when it is a masked check - #187
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #151
Any command with
;,|,&or a newline was treated as hiding its exit code, so its output was scanned for test-runner summaries even when the command was not a check at all.sed -n '1,40p' src/report.rs; echo doneexits 0 and prints a source file. If that file had a line likeerror: unknown customer id, the call was marked failed and the line was kept as its error signature. Ingesting such a session puterror: unknown customer idinto the learned procedure's trigger entities, so error recall could later match on a line of the user's own code.The output now decides only when the exit code is masked and the first stage runs a check: a known test, build or lint tool (
cargo test,pytest,jest,tsc,go testand so on, also behindnpxoruv run), a project task runner (npm run,make,just,mvn,gradle,rake), or a script whose name says it checks (./scripts/test.sh,bash ci.sh). Everything else keeps the exit code the agent reported, in both directions: a read can no longer be turned into a failure, and a failed command can no longer be turned into a success because it printedtest result: ok.. The output summaries are the formats of those runners, so tying the verdict to them is what makes it trustworthy.Cargo's
Finished ... target(s)line now counts as a pass only for build-type commands (cargo build,cargo check,cargo clippy). Forcargo testorcargo nextestit only means the build finished, so a maskedcargo test 2>&1 | head -6cut off afterrunning 2 testswas judged a pass; test runners now need a real result line such astest result: ok..sedandawkreads now count as reads too, unless they edit in place (sed -i,sed --in-place,gawk -i inplace). That also keeps the Codex and Copilot fallbacks, which flag a run with an error-looking line when no exit code is known, from flagging a file printed withsed.The list of check runners overlaps with the one the extractor uses to spot verify steps. I kept a separate list here because capture cannot depend on the extractor; moving the shared part into capture can be a follow-up. A masked check that starts with something else, like
echo start; cargo test | tail, is no longer judged by its output and keeps its exit code.To reproduce, I built the binary and ingested a Claude Code transcript into a scratch home: a
sed -n '1,40p' src/report.rs; echo donecall withis_error: falsewhose output containserror: unknown customer id, then an edit and a passingcargo test. Before the fix,trodden showlistederror: unknown customer idas anerror_signatureentity of the new procedure. After it, the procedure has only the path, symbol and command entities. New tests cover reads and prints with failing-looking output staying out of the verdict, masked checks (including behindcd,npx,uv run,makeand scripts) still being judged, the Claude Code outcome of thesedcall, andsed/awkbeing classified as reads unless they edit in place, and a maskedcargo test | headwhose output has theFinishedline and partial test output but no result line not being judged a pass. All of them fail without the fix.