fix(storage): exclude zero-valued Codex shell echoes from unfiltered search and relax IDF - #89
Merged
Merged
Conversation
…search and relax IDF The requeue path learned the Codex shell wrapper form in PR #87, but the query-time exclusion paths never did: isDirectBackscrollSearchEcho (Go) and directBackscrollSearchEchoSQL (SQL, used by recallFrequency) recognized only the bash and exec_command three-token prefixes. A pre-#80 tool row stored as search_echo=0 with a serialized 'shell command=["<sh>","-c"|"-lc", "backscroll search ..."]' call therefore leaked into unfiltered result pages and inflated unfiltered --relax document-frequency counting until an owner sync replayed the source — the same gap class fixed for exec_command in PR #86 (bs-echo-exec-cmd-gap-r1). Following PR #87's split, the Go predicate now decodes the JSON argv via the shared pendingSearchEchoShellMatches / directsearch.IsCodexDirectSearchCall instead of token-shape matching, and recallFrequency subtracts shell echo rows with the same broad-SQL-prefilter plus strict-Go-predicate split rather than attempting an unbounded SQL GLOB enumeration. Regression coverage: E2E with a real CodexReader round-trip and forced search_echo=0 (cmd/backscroll/echo_shell_zero_query_gap_test.go), per-row Go/SQL/IDF boundary cases including wrong-command, wrong-flag, extra-argv and non-search guards, a dedicated zero-valued shell IDF test, and shell rows in the at-scale SQL-vs-Go-scan cross-check.
Review round 1 on PR #89 found that the pre-existing len(fields) < 3 guard ran before the new shell branch: a Codex shell call whose argv separators are ALL JSON control escapes (\t, \n, …) serializes to only two whitespace-separated tokens, never reached the shell check, and leaked into unfiltered result pages — while recallFrequency's IDF path, which has no such floor, excluded the identical row. The fix made the two paths disagree where at base they had both been wrong. The shell check now runs on the first whitespace-delimited token before any token-count floor, case-insensitively to match the SQL LIKE prefilters. Regression coverage: boundary-table row and E2E files built through a real CodexReader round-trip with a tab-separated argv (asserted to serialize to exactly two tokens), both mutation-checked RED without the hoist; docs and the pendingSearchEchoShellMatches comment corrected to describe the page-side Go matching and the two shape gates accurately.
pablontiv
added a commit
that referenced
this pull request
Sep 12, 2026
…d-text chokepoint (#90) * fix(storage): unify direct-search echo detection behind one serialized-text chokepoint The echo-exclusion bug class (#64, PRs #86/#87/#89) was caused by 5-6 independent recognizers of "is this row a direct backscroll search echo" across three representations, kept in sync only by prose comments: - isDirectBackscrollSearchEcho: hand-rolled bash/exec_command token checks plus a shell-only decode fallback - directBackscrollSearchEchoSQL (recallFrequency IDF): SQL GLOB with an ASCII-only separator alphabet, shell form not matched at all - unmarkedDirectSearchCallSQL (PendingSearchEchoPaths requeue): the same GLOB, plus a separate 'text LIKE shell %' prefilter for shell - excludeDirectBackscrollSearchEchoes: dead duplicate with no production caller strings.Fields accepts every unicode.IsSpace rune (NBSP, U+2028/2029, U+3000, ...), so any separator outside the SQL alphabet — or any shell row with leading whitespace under the anchored LIKE — made pages exclude a row that IDF counting and requeue detection kept. Point-patching the SQL whitespace alphabet cannot close this class. Replace all of it with the broad-SQL-prefilter + strict-Go-predicate split PRs #87/#89 proved for the shell form, generalized to every shape: - directsearch.IsSerializedDirectSearchCall is the single strict predicate owning all three serialized shapes (bash, exec_command, shell argv decode); the shell decode moved here unchanged. - recallFrequency and PendingSearchEchoPaths share one provable-superset prefilter (content_type='tool', search_echo zero, text LIKE '%backscroll%') and apply the chokepoint in Go; no SQL-side shape recognizer remains to drift. - isDirectBackscrollSearchEcho is now SearchEcho || chokepoint, deleting the token indexing and the guard-ordering hazard. - Dead excludeDirectBackscrollSearchEchoes removed; its boundary test rewired onto the production refill path. Tests: new internal/storage/echo_parity_test.go asserts three-way page/IDF/requeue agreement over shape x separator alphabet (ASCII controls, NBSP, U+2028/2029, U+3000, raw vs JSON-escaped) x leading/trailing runs (RED on the old code: 48 diverging subtests, GREEN now); the boundary corpus in relaxation_echo_idf_test.go lost its SQL-equivalence column and gained the requeue leg; the mislabeled "shell NBSP separator" fixture now carries a real U+00A0 via an explicit escape (verified bytewise). Spike note: docs/research/2026-09-12-echo-chokepoint-spike.md. No schema migration. Ordinary search behavior/output unchanged; all prior echo-exclusion regressions (#85-#89) still green. * fix(directsearch): guard serialized-call predicate on literal backscroll substring Independent review of the chokepoint PR found the broad SQL prefilter (text LIKE '%backscroll%') was a provable superset of the strict Go predicate for the bash/exec_command shapes but not for the Codex shell shape: the argv is stored as raw, un-decoded JSON bytes, so a JSON \u-escaped letter of "backscroll" (e.g. \u0062ackscroll) would be accepted by IsSerializedDirectSearchCall after decoding yet missed by the prefilter — a one-row-wide three-way divergence. The guard makes the documented provable-superset claim true for every shape; it folds ASCII case to match LIKE semantics. Regression coverage: unit case in directsearch plus a three-way parity corpus entry asserting page, --relax IDF, and requeue agree (on not excluding the row). * fix(directsearch): make the superset guard case-sensitive Round-2 review found the round-1 guard folded case the wrong way for a superset guarantee: Go's strings.ToLower folds U+212A KELVIN SIGN to 'k', but SQLite LIKE folds only ASCII A-Z, so a row with an escaped-letter shell argv plus a BAC<KELVIN>SCROLL near-miss elsewhere in the text passed the guard while remaining invisible to the prefilter — the same three-way divergence one code point over. Case-sensitive strings.Contains(text, "backscroll") needs no folding reasoning at all: literal-lowercase-substring present implies LIKE matches, full stop, and every real-encoder row accepted by the predicate contains the literal by construction (verified non-narrowing by the reviewer's 391,300-input differential). Regression fixture added to the parity negatives; verified RED against the round-1 guard, GREEN here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the Codex
shellwrapper half of the zero-valuedsearch_echoquery-time gap — the same bug class fixed forexec_commandin #86 (bs-echo-exec-cmd-gap-r1).#87 taught the requeue path (
unmarkedDirectSearchCallSQL+pendingSearchEchoShellMatches+internal/directsearch) to recognize Codexshell command=["<sh>","-c"|"-lc","backscroll search ..."]rows stored assearch_echo=0. But the functions that exclude such a row from unfiltered results right now, before an owner sync replays the source, were never touched:isDirectBackscrollSearchEcho(Go) — used byexcludeDirectBackscrollSearchEchoes/refillCandidatesWithoutDirectEchoesfor unfilteredsearch --textresult pagesdirectBackscrollSearchEchoSQL(SQL) — used byrecallFrequencyfor--relaxIDF document-frequency countingBoth recognized only the
bash command=backscroll searchandexec_command cmd=backscroll searchthree-token prefixes.Reproduction (spike, hermetic)
New E2E
TestZeroValuedCodexShellEchoExcludedBeforeReplay(followingecho_zero_replay_e2e_test.go) ingests real Codexshellrollouts through the actualCodexReader(serialized shape asserted), forcessearch_echo=0to reproduce the pre-#80 state, then searches as a startup-lock follower so no owner replay can converge the rows first. Before the fix:search --text "violet handshake"page (ranks 2–9), while the identically-treatedexec_commandcontrol rows were correctly excludedsearch --relax --text "violet handshake adaptation"returned zero rows: the leaked shell rows inflated DF(violet)/DF(handshake) to 9 > DF(adaptation)=4, inverting the drop orderFix
Reuses the already-reviewed PR #87 logic instead of a new SQL/GLOB implementation:
isDirectBackscrollSearchEchogains ashellbranch calling the sharedpendingSearchEchoShellMatches(locates thecommand=token inside the sorted key=value list,json.Decoder-decodes exactly one JSON array value, feedsdirectsearch.IsCodexDirectSearchCall— the exact predicate the reader uses at ingest).directBackscrollSearchEchoSQLdeliberately stays pure-prefix (the JSON argv's separator byte-sequences are provably unbounded for SQL GLOB — the lesson of PR fix(search): requeue search_echo=0 Codex shell wrapper calls #87's three failed enumeration rounds).recallFrequencynow subtracts shell echo rows using the same broad-SQL-prefilter (text LIKE 'shell %') + strict-Go-predicate split, so unfiltered IDF counts exactly the row set the Go predicate keeps.docs/search.md#query-echo-handlingand the--relaxIDF rule now enumerate the shell argv form and document the query-time zero-valued fallback.Regression coverage
--relaxIDF drop order, withexec_commandcontrol in the same stateTestRecallFrequencySQLEchoPredicatePreservesBoundaries: 11 new shell rows asserting Go predicate, SQL predicate, andrecallFrequencyDF per row — including false-positive guards (wrong command, wrong flag-x, 4-element argv,backscroll status,env-wrapper and absolute-path inside argv, NBSP separator via realjson.Marshal)TestRelaxationUnfilteredIDFExcludesZeroValuedCodexShellEchoes: dedicated IDF + relaxed-recall testTestRecallFrequencyUnfilteredSQLMatchesGoScan: shell echo rows added to the 8000-row SQL-vs-Go-scan agreement checkexec_command/NULL/search_echo=1cases stay green in both statesjust cigreen (aggregate coverage 86.6% ≥ 85%).Note:
TestSearchRelaxationQueryEchoesDoNotInvertIDF(pre-existing, untouched here) has a latent fragility — itsstrings.Contains(out, "echo-")guard trips whenGOTMPDIRcontains the substringecho-(e.g. a worktree/branch namedfm/bs-echo-...). Environmental, not caused by this change; flagged for a possible follow-up.