fix: cosine distance space and boundary-aware paragraph overlap - #41
Merged
Conversation
…aragraphs Chroma collection now created with hnsw:space=cosine to match embedding models' cosine training objective (existing collections need delete + re-ingest to pick this up). _split_long_paragraph in the structural chunker previously raw-sliced oversized paragraphs with zero overlap and no boundary awareness; it now reuses chunk_overlap_chars and snaps splits to sentence/word boundaries, falling back to raw slice only when no boundary is found.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hnsw:space=cosine, matching the cosine-similarity objective embedding models are trained against (existing deployments need a delete + re-ingest to pick this up; documented indocs/rag-retrieval.md)._split_long_paragraphin the structural chunker no longer raw-slices oversized paragraphs with zero overlap: it now reuseschunk_overlap_charsand snaps split points to sentence/word boundaries, falling back to a raw slice only when no boundary exists in range.Prompted by comparing the current RAG pipeline against the Claude RAG course notes — retrieval/hybrid-fusion side already matched or exceeded the course; these two gaps were the concrete deltas found in chunking/vector-store correctness.
Test plan
uv run pytest -q(95 passed, 9 skipped)uv run ruff check .uv run mypy localrag(pre-existing untyped-lib warnings only, unrelated to this change)