Skip to content

fix(recall): stabilize deterministic FTS recall across equivalent query forms #1976

Open
Open
fix(recall): stabilize deterministic FTS recall across equivalent query forms#1976
Assignees
guamiao
Labels
bugSomething isn't working

Description

@AlexStocks
AlexStocks
opened on Oct 13, 2026

Scope

Implementation task for the reproducible Atomic Memory FTS recall defect in #1912. Make the deterministic analyzer-to-FTS path stable for documented equivalent query forms before considering any model-assisted query preparation.

The public operations in scope are POST /v1/memory/search and POST /v1/context/prepare when they select Atomic Memory through FTS. Shared analyzer code may be changed only where necessary to deliver this contract; do not expand the task into Experience, Skill, Topic Memory, vector, hybrid, or generic cross-family search behavior.

Implementation map

  • Start with the deterministic analyzer and FTS-requirement construction in builtin/artifacts/search.py, then trace its callers into Atomic Memory search and the SQLite/OceanBase Atomic Memory index adapters.
  • Keep fixed-pool fusion and PreparedContext rendering out of the root-cause change unless the trace proves they receive different FTS eligibility inputs.
  • Add the minimal regression fixture at the public search/prepare boundary as well as focused unit coverage at the identified analyzer/FTS seam.

Problem

#1912 demonstrates that one stored fact can change from a hit to zero hits when a Chinese natural-language query gains a short grammatical prefix, a question form, or punctuation. The behavior is currently not explainable from semantic intent or query length, so every integration is pushed to invent its own brittle query-shortening workaround.

Required work

  1. Create a deterministic same-Scope fixture with the stored reference from bug(recall): identical fact is returned for one query form and missed for another — retrieval flips on 1-2 character prefixes (CJK) #1912 and a result matrix. The following forms are mandatory positive cases for that reference:

    PowerContext Server 地址
PowerContext Server 地址是
PowerContext Server 地址是什么
PowerContext Server 地址是什么？
的 PowerContext Server 地址是什么？
我之前让你记住的 PowerContext Server 地址是什么？

    Each must return the stored reference through FTS search and prepare_context when FTS is available.

  2. Trace the current analyzer-to-FTS pipeline and identify the precise cause before selecting a fix. The fix may normalize or derive bounded lexical requirements, but it must not silently broaden every query into an unbounded OR search.

  3. Define and test the supported form-invariance contract:

    • leading/trailing whitespace and ordinary punctuation do not change recall;
    • the documented interrogative/wrapper forms and leading function words do not turn a previously matching literal fact into zero candidates;
    • identifiers, paths, code-like tokens, operators, quoted phrases, and FTS-special-character handling retain their current safety and precision guarantees.

  4. Add explicit negative controls: unrelated queries remain empty, and an English query with no literal/equivalent deterministic evidence is not promised a semantic FTS hit. English variants are diagnostic and escaping-regression coverage, not an embedding or LLM semantic-retrieval requirement.

  5. Apply the same deterministic FTS candidate eligibility to direct Atomic Memory search and PreparedContext preparation for the same Scope, query, and capability. Their final rendered content may differ because Context Pack still applies its existing byte budget.

  6. Preserve the original query as the authoritative caller input. Any internal normalization is bounded, deterministic, non-persistent, and does not create a Source, Memory, Artifact, Scope, authorization, or tool-action claim.

Acceptance criteria

  • The mandatory bug(recall): identical fact is returned for one query form and missed for another — retrieval flips on 1-2 character prefixes (CJK) #1912 CJK positive matrix consistently returns the stored reference through both operations wherever FTS is available.
  • The regression suite covers punctuation, the required wrappers/function words, English diagnostic cases, unrelated queries, and technical-token/FTS escaping cases.
  • The fix does not turn known unrelated queries into hits merely to avoid an empty result.
  • memory/search and context/prepare agree on FTS candidate eligibility for the same request inputs; rendering and byte-budget omission remain separate concerns.
  • Existing FTS injection/escaping and query-length protections remain covered by focused tests.
  • SQLite and OceanBase behavior is checked wherever the backend exposes the same FTS capability; capability absence is reported separately rather than treated as a failed semantic assertion.

Non-goals

  • No LLM query rewrite, embedding requirement, model provider, schema/index migration, trigram-index migration, new HTTP field, or query-text retention.
  • No whole-query synonym expansion, hidden cross-Scope search, or change to Memory authority/ranking policy.
  • No claim that all natural-language paraphrases or cross-language questions are semantically equivalent; this task fixes reproducible deterministic form sensitivity around the same lexical intent.

Follow-up boundary

RFC #1770 permits model-assisted query preparation only as a host/context-boundary helper in shadow evaluation, with original-query fallback and replay evidence. It is not part of this fix. A future experiment may be considered only after this deterministic baseline is stable and measured.

Validation

Run focused analyzer/FTS, Atomic Memory search, and PreparedContext tests; include the #1912 minimal reproduction and report backend/capability coverage explicitly.

References

Activity

  1. AlexStocks
    added
    bugSomething isn't working
    on Oct 13, 2026
  2. github-project-automation
    added this to PowerContexton Oct 13, 2026
  3. github-project-automation
    moved this to Todo in PowerContexton Oct 13, 2026

  4. guamiao commented on Oct 13, 2026

    @guamiao
    guamiao
    on Oct 13, 2026
    Contributor

    Hi, I'd like to work on this issue. Could you please assign it to me? Thanks!

  5. AlexStocks
    assigned
    guamiao
    on Oct 13, 2026

  6. CX33071 commented on Oct 13, 2026

    @CX33071
    CX33071
    on Oct 13, 2026

    Hi, I'd like to work on this issue. Could you please assign it to me? I’ll investigate the existing implementation and submit a PR.

  7. guamiao commented on Oct 14, 2026

    @guamiao
    guamiao
    on Oct 14, 2026
    Contributor

    Result

    I attempted this as a deterministic, query-side fix and ran four rounds of adversarial review with measurements. The documented forms can be made to pass, but the contract as written does not look reachable from the query side. Two acceptance criteria below could not be met, and my best attempt made multi-row precision worse than master. Details and the reasoning follow; happy to share the harness.

    How the failure was traced

    The stored projection is Analyzer v1: every CJK character becomes a unigram (u_<cp>) plus a bigram with its neighbour (b_<cp>_<cp>). Admission then requires

    required = max(2, ceil(n_terms * 0.25))

    A Chinese wrapper is analysed into many more terms than the keywords it wraps, so required grows with the wrapper while the stored fact supplies a fixed number of matching terms.

    Measured against one stored entry, PowerContext Server 地址是什么？ vs 我之前让你记住的 PowerContext Server 地址是什么？:

    • short form — 11 terms, required 3, 5 matching → hit
    • wrapped form — 26 terms, required 7, 6 matching → 0 hits

    That is the "long prefix degrades to zero" report, and it is a property of the requirement formula, not of the query text.

    What was implemented and verified

    Query-side Jieba segmentation, opt-in per caller, storage projection and indexes untouched (so no migration):

    • segment the query's Han runs; a multi-character word contributes its adjacent bigrams only, a single-character word its unigram;
    • drop word-level stopwords; keep negations;
    • do not let a single shared character, or one digit of a dotted version, count toward the requirement;
    • non-Han text is passed through unchanged.

    Verified: all six documented forms and four undocumented wrappers reach the reference through both POST /v1/memory/search and POST /v1/context/prepare. Exact identifiers (POWERCONTEXT_SERVER_WORKSPACE, server_unavailable, café, full-width, dotted versions) stay recall-equivalent to master.

    What was tried, and where each attempt failed

    approach where it failed (measured)
    CJK function-character filter on the query drops content morphemes (使/的/之); the query collapses to a single term, required falls to 1, and unrelated queries become hits (使用 hits 费用报销流程说明)
    bounded required for CJK queries unrelated long CJK queries share four common characters (的/在/本/定) — 4/20 unrelated sentences hit
    Jieba query-side segmentation (above) (a) a query for 周会时间 no longer finds 团队的周会固定在周四下午三点。 — segmentation splits 周会 into single characters, which the requirement then ignores; (b) wrappers beyond ~66 characters still return 0, because required still grows with the wrapper's own content words; (c) on a ten-row corpus, unrelated-sentence hits move above master (17/53 vs 15/53)

    Acceptance criteria from this issue that could not be met

    • "unrelated queries remain empty" — not met at scale: multi-row precision is worse than master, including two HTTP-reproducible cases where two shared common words, or one shared three-character word, satisfy the requirement.
    • "documented wrapper forms … do not turn a matching fact into zero candidates" — met for the documented forms, not met generally: the same mechanism reappears past ~66 characters.
    • "identifiers, paths, code-like tokens … retain their current safety and precision" — met, but only after scoping segmentation to Han runs; segmenting identifiers directly regressed them (POWERCONTEXT_SERVER_WORKSPACE 1 → 0 hits).
    • Recall neutrality — not met: the weak-evidence rule costs legitimate queries (周会时间, 记住, 现在, 告诉).

    Why, structurally

    Against a character-level projection, every query-side rewrite balances on a fuzzy signal: tighten the requirement and recall drops, loosen it and common single characters get admitted. Word boundaries have to exist in the index for a word-level query to mean anything — which is exactly what #1979 does, by segmenting both sides and refreshing the existing lexical projections at startup.

    This issue's non-goals forbid that ("No … schema/index migration, trigram-index migration"), so the constraint and a robust fix appear to be in tension.

    Suggestion

    Either relax that non-goal to allow a projection refresh (the route #1979 already takes), or scope this contract explicitly to the documented forms and record the residual boundaries as known limits — wrappers beyond ~66 characters, multi-row precision, and the query/requirement coupling — pinned by tests, rather than closing the issue as fully fixed.

    Happy to share the reproduction scripts and the per-approach measurement harness.

    Given #1979 already segments both sides and refreshes existing projections at startup, is that the intended fix for this issue? If so, should the "no index migration" non-goal be relaxed to allow a projection refresh, rather than this being solved on the query side? I'll keep the analysis here and can unassign myself if that branch already covers it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions