feat(retrieval): fusion candidate layer, BM25/hybrid strategies, and the retrieval experiment - #170
Merged
Merged
Conversation
mrsibe
added
enhancement
…d the experiment #77 asked whether BM25, hybrid or a reranker beats dense, measured on the frozen chunk baseline. Implementing it exposed a bug that made BM25 silently useless. The experiment (`scripts/eval-retrieval.mjs`, `docs/eval/retrieval-v1.5.md`) runs the real harness once per strategy, chunking held at 1000/100: strategy R@1 R@5 MRR nDCG@10 p95 dense 0.8333 1.0000 0.9278 0.9437 18.35 ms sparse 0.7333 0.8667 0.8056 0.8184 2.73 ms hybrid 0.8667 1.0000 0.9444 0.9561 28.52 ms Hybrid is better on Recall@1, MRR and nDCG@10, but the frozen rule requires Recall@5 to *improve*, and dense is already saturated at 1.0000 — no strategy can meet that condition on this corpus. **Dense stays the default**, and the result is recorded as inconclusive rather than adopted or rejected on a metric that cannot move. The report says this plainly. The bug: `buildFtsMatchQuery` ANDed the terms, which is the right default for a lookup box but wrong for a question. A natural-language question's terms virtually never all appear in one chunk, so sparse scored 0.0000 on every metric and hybrid silently degenerated to dense — a "hybrid" that was dense with extra latency. Terms are now ORed; BM25 still ranks a chunk matching more terms higher. - `candidates.ts`: `rrfFuse` over `chunkId + rank` (cosine and BM25 are not comparable, which is why only ranks are used). - `HybridRetriever` serves dense / sparse / hybrid from one path; `DenseRetriever` gained `candidateHits()` so the dense channel is not duplicated. - `RetrievalRequest.strategy`, `SearchOptions.strategy`, `--eval-retrieval=`. - Reranking is **not evaluated**: a cross-encoder model is not available offline and inventing its numbers would defeat the harness. Stated in the report. Verified: npm run typecheck; npm test (398 pass, incl. RRF tests); npm run check:design; npm run build; `npm run eval` byte-identical to baseline-v1.5.json (dense is still the default); `electron . --smoke-test` PASS (27 checks).
mrsibe
force-pushed
the
spike/77-retrieval-experiments
branch
from
September 28, 2026 10:32
0d38922 to
fd1cc3a
Compare
This was referenced Sep 28, 2026
mrsibe
added a commit
that referenced
this pull request
Sep 30, 2026
…odel (#198) The harness measured the retriever but never the window the prompt actually gets. `evidenceK: 5` was a separate constant from the `contextK: 3` production uses, so the one metric that looked at a window looked at a different one than the product does. Child 5 of #192. **`contextK` is now the window for both context metrics.** - `contextPrecision@contextK` — of the first `contextK` passages, the share covering ground truth. This is what `evidencePrecisionAt5` was, at the production width. - `contextRecall@contextK` — the share of needed ground-truth blocks that made it into that window. Distinct from `Recall@10`: a block found at rank 4 is invisible when `contextK = 3`, and that is a product fact, not a ranking fact. Both are **deterministic**: the dataset says which blocks answer the question, so no model is needed to score a window. Together they are the trade-off a `contextK` decision makes — a wider window finds more and carries more noise — which is what the sweep in the next child needs. Baseline: `contextPrecision@3 = 0.3556`, `contextRecall@3 = 1.0000` — the needed evidence is always inside the top 3 on this corpus, but only about a third of what is inside the window is relevant. That second number is the one that says the window is paying for passages that do not answer the question. **What is deliberately not here.** Faithfulness, completeness, answer correctness and noise sensitivity need a generative model. The harness runs offline with only the pinned embedding model — the same constraint that keeps the reranker unmeasured (#170) — so this PR does not add them and does not fake them: the report's Definitions section now says so, and the "Not evaluated" line in `eval-retrieval.mjs` points at the same constraint. Adding an LLM judge is a separate change that has to solve model pinning first, not a line of code. `evidenceK` is removed from the harness config, so there is exactly one context width. ## Testing - `npm run typecheck` — clean - `npm test` — 497 pass - `npm run eval` twice — byte-identical `docs/eval/baseline-v1.6.json` - `npm run eval:retrieval` — regenerated; hybrid still clears the amended rule Part of #192 (child 5, deterministic half).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Implements the candidate/fusion layer and the BM25 / hybrid strategies, runs the retrieval experiment #77 asked for, and fixes the bug that made BM25 silently useless.
Related issue
Fixes #77
Related to #154, #96, #78
The result
Every strategy runs the real harness over the same corpus and 30 questions, chunking held fixed at the frozen baseline (
baseline-v1.5.json):Hybrid is better on Recall@1, MRR and nDCG@10 — but the frozen rule requires Recall@5 to improve, and dense is already saturated at 1.0000, so no strategy can satisfy it here. Dense stays the default, and the result is recorded as inconclusive rather than adopted or rejected on a metric that cannot move. The report says this plainly.
The bug it exposed
buildFtsMatchQueryANDed the terms. That is the right default for a lookup box but wrong for a question: a natural-language question's terms virtually never all appear in one chunk, so sparse scored 0.0000 on every metric and hybrid silently degenerated to dense — a "hybrid" that was dense with extra latency. Terms are now ORed; BM25 still ranks a chunk matching more terms higher.What changed
candidates.ts:rrfFuseoverchunkId + rank. Scores are not comparable across channels (cosine vs BM25), which is exactly why only the ranks are used.HybridRetrieverservesdense/sparse/hybridfrom one path;DenseRetrievergainedcandidateHits()so the dense channel is not duplicated for fusion.RetrievalRequest.strategy,SearchOptions.strategy,--eval-retrieval=,scripts/eval-retrieval.mjs,npm run eval:retrieval.docs/eval/retrieval-v1.5.{md,json}are generated and added to.prettierignore.Not evaluated
Reranking. The issue lists "hybrid + reranker", but a cross-encoder model is not available offline and inventing its numbers would defeat the harness. It stays open until a model can be pinned the way the embedding model is. Stated in the report.
How was this tested?
npm run typecheck— passes.npm test— 398 pass (new RRF tests: agreement wins, single-channel hits still rank, empty channels, rank-not-score).npm run check:design— no violations.npm run build— passes.npm run eval— byte-identical tobaseline-v1.5.json(dense is still the default, so the frozen numbers do not move).electron . --smoke-test— PASS, 27 checks, withHybridRetrievernow the app's retriever.Checklist
npm run typecheckpasses.npm run buildpasses.Desktop / build changes