Repository navigation
Phase 10: the vector ranking by default, and episode chunks - #30
Merged
Merged
Conversation
bench.retrieval takes --ablate, which also ranks each question with the fused query less one arm at a time: bm25-content, content-cue, and bm25. bench/ablation.py builds those queries. With every arm, its query is the query of ranking="fused".
…h chunks HotdataStore ranks by content vector alone by default, as MemoryStore does, and ranking="fused" stays as an option. Memory.capture takes chunk_chars with episode: it cuts the text at blank lines into chunks, stores chunk n under <episode>-<n>, and each fact cites every chunk. The extractor still reads the whole text. The skill capture script takes --chunk-chars, and the bench rankings default to vector. docs/benchmarks.md reports the ablation of task 1 and the memory arm with each ranking.
bench.e2e takes --chunk-chars with --session-episodes, and capture stores each session as chunks of that size. The chunk size is part of the kept name. bench.diagnose maps a chunk key to its session.
A fact captured in chunks cites every chunk, so its line in the block grew by about 300 characters at 1,000-character chunks. The block now shows a run of sources <prefix>-<n>@<revision>, with n counting up by 1 and one revision, as <prefix>-<first>…<last>@<revision>. The stored sources do not change.
The bench README names chunks in the kept name and limits the BM25 warnings to the fused ranking. The benchmarks report which keep runs paid for extraction, and round each median down. The guarantees, contracts, and changelog state the range rule in full. The plan gives task 3 as it was built.
rohan-hotdata
requested review from
shefeek-jinnah
and removed request for
a team
October 18, 2026 14:41
rohan-hotdata
added a commit
that referenced
this pull request
Oct 21, 2026
The roadmap closes phase 10 with PR #30 and records what it changed for later phases: the extractor prompt moves with the block line, a reuse at a new budget pays for the reader, and the framework listing reads only 100 databases. Two items move to Deferred: precise chunk citations, and the first-page listing (hotdata-dev/sdk-python-framework#89). The owner paused the workstream before phase 11 on 2026-10-19. The plan drops phase 10, adds its numbers to the starting point, and notes what phase 10 changed for phase 11.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes the vector ranking the default of
HotdataStore. It also letscapturestore a session as chunks, so the reader gets the parts of a session that match. Closes #26.How
HotdataStore(ranking="vector")is the default now, asMemoryStoreranks."fused"stays as an option. The bench runner defaults follow.Memory.capture(..., episode=key, chunk_chars=n)cuts the text at blank lines outside fences into chunks of at mostncharacters and stores chunk i under<key>-<i>. It does not use the heading rules ofload, because an assistant turn often holds a Markdown heading. The extractor reads the whole text once, and each fact cites every chunk. A retry writes nothing, and an edited chunk gets its next revision only. No newStoremethod or record field.<prefix>-<n>@<rev>(same prefix, n counting up by 1, same revision) as<prefix>-<first>…<last>@<rev>. Other sources show as before. The stored sources do not change. A rule that folded by key prefix alone was rejected: a caller's own keyschat-0002andchat-0003look like chunks.bench.retrieval --ablateranks with the fused query less one arm.bench.e2e --chunk-charsstores chunks, and the chunk size is part of the kept name.bench.diagnosemaps a chunk key to its session.scripts/capture.py --chunk-chars.Why
The ablation on the 419 retrieval questions showed that the BM25 arm makes the whole gap between the rankings: fused
recall_all@100.9570, vector 0.9928, BM25 alone 0.8926. Episodes have no cues, so fused is BM25 plus the content vector. Fusion with equal weights pushes evidence sessions that BM25 ranks 23 to 40 out of the top 10. End to end on sample A at 64,000 characters, vector scored 46 and fused 45.A session is one episode, so one needed turn brought the whole session into the block. With chunks of 1,000 characters at a budget of 16,000:
The full grid (1,000 / 2,000 / 4,000 at 8,000 / 16,000 / 32,000) and the diagnosis are in
docs/benchmarks.md.Verification
make verifyandmake integration(144 hotdata tests) on the final library commit.MemoryStoreandHotdataStorewith chunks, for"search"and"cited". Two more prove that capture searches with the store's ranking, and that the default is vector. The skill test runs the new--chunk-charscommand.docs/benchmarks.mdcomes from a result file inbench/results/. Phase 10 spent $12.77 of the $50 cap.Notes
HotdataStorewithoutrankingnow get the vector ranking. The block format also changes for runs of chunk sources. Both are inCHANGELOG.md."cited"is unchanged. A fact cites every chunk of its session, so"cited"still gives the whole session. Only"search"saves tokens with chunks.k=30and not the budget was the limit.hotdata-framework0.14.1list_managed_databasesreads only the first 100 databases, andHotdataStore.provisionuses it. Filed as list_managed_databases and resolve_managed_database read only the first page of databases sdk-python-framework#89. Not fixed here.