feat(search): a full-text index over chunks and a global search palette - #169
Merged
Merged
Conversation
Chat retrieval and the search box were both vector-only. Embeddings are measurably bad at exact tokens — model names, ids, symbols, numbers — so there was no way to find a literal string, and a note could not be searched at all. Build the sparse index #77 also needs, and give it a user-facing surface: - `chunks_fts` (FTS5) is created and backfilled by `ensureChunksFts()`. The backfill runs on every ensure, so a library indexed before FTS existed, or a crash between the chunk write and the FTS write, heals itself. - The index is written with the chunks in the ingestion pipeline and deleted with the document, so it tracks re-index and delete. - `KnowledgeService.searchText()` returns the same `SearchResult` shape as the vector path, so provenance and the jump-to-source chain are not duplicated. - User text is quoted term by term before it reaches `MATCH`: an operator-looking query (`-`, `NEAR(`, `col:value`, an unbalanced quote) returns no rows instead of a syntax error, which would read as "no results rather than "bad query". - `SearchPalette` (Ctrl/Cmd+K) shows **matching text** and **related passages** as two labelled groups and never fuses them. Chat retrieval (#77) does fuse them, because the model needs one ordered list; the two must not share ranking semantics, which is why the shared part is the index, not the ranking. Verified: npm run typecheck; npm test (391 pass, incl. FTS5 integration against the bundled SQLite); npm run check:design; npm run build; `npm run eval` unchanged and byte-identical; `electron . --smoke-test` PASS — 27 checks, including a new one that a literal search finds the term after indexing and still finds exactly one row after a re-index. Not verified: the palette was not clicked in a running app (markup is covered by typecheck and the design guard).
Merged
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds
chunks_fts(FTS5/BM25) and a Ctrl/Cmd+K search palette that shows literal and semantic results as two separate signals. This is the sparse index #77 will fuse with dense.Why?
Both chat retrieval and the search box were vector-only. Embeddings are measurably bad at exact tokens — model names, ids, symbols, numbers — so there was no way to find a literal string, and a note could not be searched at all. #96 also builds the index #77 needs for BM25/hybrid, so the two share an index but not a ranking.
Related issue
Fixes #96
Related to #154, #77
What changed
chunks_fts(FTS5).ensureChunksFts()creates it and backfills missing rows on every call, so a library indexed before FTS existed — or a crash between the chunk write and the FTS write — heals itself.KnowledgeService.searchText()returns the sameSearchResultshape as the vector path, so provenance and the jump-to-source chain are not duplicated.MATCH, so-,NEAR(,col:value,a AND bor an unbalanced quote return no rows instead of a syntax error — a syntax error reads as "no results", not "bad query".SearchPalette(Ctrl/Cmd+K): matching text and related passages as labelled groups, never fused. A hit opens the reader at its passage (the locator travels with the result).KnowledgeSearchResultgained an optionallocator, and the preload API asearchTextmethod.How was this tested?
npm run typecheck— passes.npm test— 391 pass, including FTS5 integration against the bundled SQLite: BM25 ordering, notebook isolation, document-scoped filter, operator-looking queries not throwing, snippet extraction.npm run check:design— no violations.npm run build— passes.npm run eval— output byte-identical tobaseline-v1.5.json(FTS does not touch dense retrieval).electron . --smoke-test— PASS, 27 checks, including a new one: a literal search finds the term after indexing, and finds exactly one row after a re-index.Not verified
Checklist
npm run typecheckpasses.npm run buildpasses.Desktop / build changes
IF NOT EXISTS; no drizzle migration).