Skip to content

fix(storage): exclude zero-valued Codex shell echoes from unfiltered search and relax IDF - #89

Merged
pablontiv merged 3 commits into
mainfrom
fm/bs-echo-shell-query-gap-r1
Sep 12, 2026
Merged

pablontiv merged 3 commits into
mainfrom
fm/bs-echo-shell-query-gap-r1

Conversation

@pablontiv

Copy link
Copy Markdown
Owner

Summary

Fixes the Codex shell wrapper half of the zero-valued search_echo query-time gap — the same bug class fixed for exec_command in #86 (bs-echo-exec-cmd-gap-r1).

#87 taught the requeue path (unmarkedDirectSearchCallSQL + pendingSearchEchoShellMatches + internal/directsearch) to recognize Codex shell command=["<sh>","-c"|"-lc","backscroll search ..."] rows stored as search_echo=0. But the functions that exclude such a row from unfiltered results right now, before an owner sync replays the source, were never touched:

  • isDirectBackscrollSearchEcho (Go) — used by excludeDirectBackscrollSearchEchoes / refillCandidatesWithoutDirectEchoes for unfiltered search --text result pages
  • directBackscrollSearchEchoSQL (SQL) — used by recallFrequency for --relax IDF document-frequency counting

Both recognized only the bash command=backscroll search and exec_command cmd=backscroll search three-token prefixes.

Reproduction (spike, hermetic)

New E2E TestZeroValuedCodexShellEchoExcludedBeforeReplay (following echo_zero_replay_e2e_test.go) ingests real Codex shell rollouts through the actual CodexReader (serialized shape asserted), forces search_echo=0 to reproduce the pre-#80 state, then searches as a startup-lock follower so no owner replay can converge the rows first. Before the fix:

  • 8 shell echo rows leaked into the unfiltered search --text "violet handshake" page (ranks 2–9), while the identically-treated exec_command control rows were correctly excluded
  • search --relax --text "violet handshake adaptation" returned zero rows: the leaked shell rows inflated DF(violet)/DF(handshake) to 9 > DF(adaptation)=4, inverting the drop order

Fix

Reuses the already-reviewed PR #87 logic instead of a new SQL/GLOB implementation:

  • Go: isDirectBackscrollSearchEcho gains a shell branch calling the shared pendingSearchEchoShellMatches (locates the command= token inside the sorted key=value list, json.Decoder-decodes exactly one JSON array value, feeds directsearch.IsCodexDirectSearchCall — the exact predicate the reader uses at ingest).
  • SQL/IDF: directBackscrollSearchEchoSQL deliberately stays pure-prefix (the JSON argv's separator byte-sequences are provably unbounded for SQL GLOB — the lesson of PR fix(search): requeue search_echo=0 Codex shell wrapper calls #87's three failed enumeration rounds). recallFrequency now subtracts shell echo rows using the same broad-SQL-prefilter (text LIKE 'shell %') + strict-Go-predicate split, so unfiltered IDF counts exactly the row set the Go predicate keeps.
  • Docs: docs/search.md#query-echo-handling and the --relax IDF rule now enumerate the shell argv form and document the query-time zero-valued fallback.

Regression coverage

  • E2E (real reader round-trip, forced-zero state, follower lock): unfiltered page exclusion + --relax IDF drop order, with exec_command control in the same state
  • TestRecallFrequencySQLEchoPredicatePreservesBoundaries: 11 new shell rows asserting Go predicate, SQL predicate, and recallFrequency DF per row — including false-positive guards (wrong command, wrong flag -x, 4-element argv, backscroll status, env-wrapper and absolute-path inside argv, NBSP separator via real json.Marshal)
  • TestRelaxationUnfilteredIDFExcludesZeroValuedCodexShellEchoes: dedicated IDF + relaxed-recall test
  • TestRecallFrequencyUnfilteredSQLMatchesGoScan: shell echo rows added to the 8000-row SQL-vs-Go-scan agreement check
  • Mutation-checked: with the fix stashed, all new tests fail RED; existing Claude/exec_command/NULL/search_echo=1 cases stay green in both states

just ci green (aggregate coverage 86.6% ≥ 85%).

Note: TestSearchRelaxationQueryEchoesDoNotInvertIDF (pre-existing, untouched here) has a latent fragility — its strings.Contains(out, "echo-") guard trips when GOTMPDIR contains the substring echo- (e.g. a worktree/branch named fm/bs-echo-...). Environmental, not caused by this change; flagged for a possible follow-up.

…search and relax IDF

The requeue path learned the Codex shell wrapper form in PR #87, but the
query-time exclusion paths never did: isDirectBackscrollSearchEcho (Go) and
directBackscrollSearchEchoSQL (SQL, used by recallFrequency) recognized only
the bash and exec_command three-token prefixes. A pre-#80 tool row stored as
search_echo=0 with a serialized 'shell command=["<sh>","-c"|"-lc",
"backscroll search ..."]' call therefore leaked into unfiltered result
pages and inflated unfiltered --relax document-frequency counting until an
owner sync replayed the source — the same gap class fixed for exec_command
in PR #86 (bs-echo-exec-cmd-gap-r1).

Following PR #87's split, the Go predicate now decodes the JSON argv via the
shared pendingSearchEchoShellMatches / directsearch.IsCodexDirectSearchCall
instead of token-shape matching, and recallFrequency subtracts shell echo
rows with the same broad-SQL-prefilter plus strict-Go-predicate split rather
than attempting an unbounded SQL GLOB enumeration.

Regression coverage: E2E with a real CodexReader round-trip and forced
search_echo=0 (cmd/backscroll/echo_shell_zero_query_gap_test.go), per-row
Go/SQL/IDF boundary cases including wrong-command, wrong-flag, extra-argv
and non-search guards, a dedicated zero-valued shell IDF test, and shell
rows in the at-scale SQL-vs-Go-scan cross-check.
Review round 1 on PR #89 found that the pre-existing len(fields) < 3 guard
ran before the new shell branch: a Codex shell call whose argv separators
are ALL JSON control escapes (\t, \n, …) serializes to only two
whitespace-separated tokens, never reached the shell check, and leaked into
unfiltered result pages — while recallFrequency's IDF path, which has no
such floor, excluded the identical row. The fix made the two paths disagree
where at base they had both been wrong.

The shell check now runs on the first whitespace-delimited token before any
token-count floor, case-insensitively to match the SQL LIKE prefilters.

Regression coverage: boundary-table row and E2E files built through a real
CodexReader round-trip with a tab-separated argv (asserted to serialize to
exactly two tokens), both mutation-checked RED without the hoist; docs and
the pendingSearchEchoShellMatches comment corrected to describe the
page-side Go matching and the two shape gates accurately.
@pablontiv
pablontiv merged commit e7e03b8 into main Sep 12, 2026
8 checks passed
@pablontiv
pablontiv deleted the fm/bs-echo-shell-query-gap-r1 branch September 12, 2026 13:45
pablontiv added a commit that referenced this pull request Sep 12, 2026
…d-text chokepoint (#90)

* fix(storage): unify direct-search echo detection behind one serialized-text chokepoint

The echo-exclusion bug class (#64, PRs #86/#87/#89) was caused by 5-6
independent recognizers of "is this row a direct backscroll search echo"
across three representations, kept in sync only by prose comments:

- isDirectBackscrollSearchEcho: hand-rolled bash/exec_command token checks
  plus a shell-only decode fallback
- directBackscrollSearchEchoSQL (recallFrequency IDF): SQL GLOB with an
  ASCII-only separator alphabet, shell form not matched at all
- unmarkedDirectSearchCallSQL (PendingSearchEchoPaths requeue): the same
  GLOB, plus a separate 'text LIKE shell %' prefilter for shell
- excludeDirectBackscrollSearchEchoes: dead duplicate with no production
  caller

strings.Fields accepts every unicode.IsSpace rune (NBSP, U+2028/2029,
U+3000, ...), so any separator outside the SQL alphabet — or any shell row
with leading whitespace under the anchored LIKE — made pages exclude a row
that IDF counting and requeue detection kept. Point-patching the SQL
whitespace alphabet cannot close this class.

Replace all of it with the broad-SQL-prefilter + strict-Go-predicate split
PRs #87/#89 proved for the shell form, generalized to every shape:

- directsearch.IsSerializedDirectSearchCall is the single strict predicate
  owning all three serialized shapes (bash, exec_command, shell argv
  decode); the shell decode moved here unchanged.
- recallFrequency and PendingSearchEchoPaths share one provable-superset
  prefilter (content_type='tool', search_echo zero, text LIKE
  '%backscroll%') and apply the chokepoint in Go; no SQL-side shape
  recognizer remains to drift.
- isDirectBackscrollSearchEcho is now SearchEcho || chokepoint, deleting
  the token indexing and the guard-ordering hazard.
- Dead excludeDirectBackscrollSearchEchoes removed; its boundary test
  rewired onto the production refill path.

Tests: new internal/storage/echo_parity_test.go asserts three-way
page/IDF/requeue agreement over shape x separator alphabet (ASCII controls,
NBSP, U+2028/2029, U+3000, raw vs JSON-escaped) x leading/trailing runs
(RED on the old code: 48 diverging subtests, GREEN now); the boundary
corpus in relaxation_echo_idf_test.go lost its SQL-equivalence column and
gained the requeue leg; the mislabeled "shell NBSP separator" fixture now
carries a real U+00A0 via an explicit escape (verified bytewise).

Spike note: docs/research/2026-09-12-echo-chokepoint-spike.md.
No schema migration. Ordinary search behavior/output unchanged; all prior
echo-exclusion regressions (#85-#89) still green.

* fix(directsearch): guard serialized-call predicate on literal backscroll substring

Independent review of the chokepoint PR found the broad SQL prefilter
(text LIKE '%backscroll%') was a provable superset of the strict Go
predicate for the bash/exec_command shapes but not for the Codex shell
shape: the argv is stored as raw, un-decoded JSON bytes, so a JSON
\u-escaped letter of "backscroll" (e.g. \u0062ackscroll) would be
accepted by IsSerializedDirectSearchCall after decoding yet missed by
the prefilter — a one-row-wide three-way divergence.

The guard makes the documented provable-superset claim true for every
shape; it folds ASCII case to match LIKE semantics. Regression coverage:
unit case in directsearch plus a three-way parity corpus entry asserting
page, --relax IDF, and requeue agree (on not excluding the row).

* fix(directsearch): make the superset guard case-sensitive

Round-2 review found the round-1 guard folded case the wrong way for a
superset guarantee: Go's strings.ToLower folds U+212A KELVIN SIGN to 'k',
but SQLite LIKE folds only ASCII A-Z, so a row with an escaped-letter shell
argv plus a BAC<KELVIN>SCROLL near-miss elsewhere in the text passed the
guard while remaining invisible to the prefilter — the same three-way
divergence one code point over.

Case-sensitive strings.Contains(text, "backscroll") needs no folding
reasoning at all: literal-lowercase-substring present implies LIKE matches,
full stop, and every real-encoder row accepted by the predicate contains
the literal by construction (verified non-narrowing by the reviewer's
391,300-input differential). Regression fixture added to the parity
negatives; verified RED against the round-1 guard, GREEN here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant