You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(search): unify word analysis across backends - #1979
Apply consistent Chinese word segmentation and query filtering across built-in searchable Families.
What changes are included in this PR?
Use Jieba segmentation and bundled English/Chinese stopwords, preserving identifiers, negations and quoted stopwords.
Refresh existing lexical projections at startup without changing records, history or vectors.
Include stopword attribution, public search and upgrade acceptance tests, and bilingual documentation.
Are there any user-facing changes?
Chinese queries are matched by words. Unquoted stopwords are filtered; negations and quoted stopwords remain searchable.
APIs, v1 formats and ranking defaults remain unchanged.
Stop existing Server and Worker processes before upgrading. Startup refreshes searchable text and may take longer for larger databases.
How was this change tested?
uv run --locked prek run -a
uv run --locked pytest tests/e2e/test_unified_lexical_search.py tests/e2e/test_lexical_projection_upgrade.py -k sqlite -q
make docs-test
AI usage statement
Implemented and validated with OpenAI Codex (GPT-6).
Stopwords-only applies #1979’s lists to master’s existing tokens. Full #1979 adds substantial speed and storage savings, with no statistically clear nDCG gain over stopwords-only; it also includes encoding and query-policy changes.
Tradeoffs: ~39 MiB additional installed dependency size, slower indexing, and refreshing existing token projections while retaining the v1 record format.
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 65d40d6822284f9c7ea7c3beca0b2f67314bf012. The query-bound issue remains reproducible (see my reply in the existing thread). One additional processing failure is detailed inline. Please address both before merging.
ab9aaeda and 65b44047 address these issues: lost quoted or trailing query terms, failures on valid Topic content containing only stopwords, slow SQLite FTS reconciliation, and repeated resegmentation at startup.
On synthetic SQLite fixtures (2 CPUs, 4 GiB):
FTS reconciliation with 10,000 unchanged rows: 66.17 s → 0.036 s.
Subsequent startup with 100,000 Atomic rows: about 38 s → 0.13 s.
DuRetrieval with 100,001 documents and the first 500 official queries showed unchanged Recall@10 (62.19%) and nDCG@10 (60.80%).
Single-quoted contractions lose the quoted-stopword exemption: lexical_query_words("'I'm'") returns (), while the double-quoted equivalent retains i'm. The quotation regex does not recognize a single-quoted span containing an apostrophe, so this new filter removes the contraction and leaves no FTS evidence. Make the shared quotation parser apostrophe-aware and add a regression for single-quoted contractions.
All-directive queries fail to fall back to searchable terms
Use only supplied context. Do not call tools. now produces no lexical terms, although each sentence separately remains searchable. The fallback tests the entire query as one directive, so FTS cannot retrieve an identical stored policy. Restore the fallback when all constituent sentences are directives, while continuing to exclude directives accompanying other query evidence.
Preserve please as a query stopword. The previous filter included it, but neither bundled list does. As a result, PowerContext Server admits text containing only PowerContext, while please PowerContext Server requires two matches and rejects that same text. Add please to the effective stopwords so a polite prefix does not change admission.
Restore fallback for queries with no lexical evidence
A query consisting of multiple execution-policy sentences loses all lexical evidence. For example, Do not call tools. Do not read files. returns no words and cannot retrieve an identical stored policy: each sentence is filtered, but the combined string fails _is_execution_sentence, so the fallback never runs. Restore the fallback when filtering leaves no evidence, while still removing stopwords.
Reset lexical-version markers after restoring an older export
docs/en/docs/operate/troubleshoot.md:271
Reset the target lexical-version markers after restoring an older export. Step 4 starts the current Server on an empty target, which writes current-version markers. If the source predates this table, omitting its import leaves those markers intact. On restart, refresh_lexical_projections() skips the restored legacy Topic text, so new Chinese query tokens cannot match it. In both locale guides, instruct operators to run DELETE FROM pc_lexical_projection_versions; on the target after restoration and before restarting the Server.
🧠 Review effort: Balanced
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hidb4ai
left a comment
Which issue or RFC does this PR close?
Related to #1969.
Rationale for this change
Apply consistent Chinese word segmentation and query filtering across built-in searchable Families.
What changes are included in this PR?
Are there any user-facing changes?
Chinese queries are matched by words. Unquoted stopwords are filtered; negations and quoted stopwords remain searchable.
APIs, v1 formats and ranking defaults remain unchanged.
Stop existing Server and Worker processes before upgrading. Startup refreshes searchable text and may take longer for larger databases.
How was this change tested?
uv run --locked prek run -auv run --locked pytest tests/e2e/test_unified_lexical_search.py tests/e2e/test_lexical_projection_upgrade.py -k sqlite -qmake docs-testAI usage statement
Implemented and validated with OpenAI Codex (GPT-6).