Skip to content

Phase 10: the vector ranking by default, and episode chunks - #30

Merged
rohan-hotdata merged 7 commits into
mainfrom
phase-10-ranking-and-episode-chunks
Oct 19, 2026
Merged

rohan-hotdata merged 7 commits into
mainfrom
phase-10-ranking-and-episode-chunks

Conversation

@rohan-hotdata

Copy link
Copy Markdown
Collaborator

This PR makes the vector ranking the default of HotdataStore. It also lets capture store a session as chunks, so the reader gets the parts of a session that match. Closes #26.

How

  • Default ranking. HotdataStore(ranking="vector") is the default now, as MemoryStore ranks. "fused" stays as an option. The bench runner defaults follow.
  • Chunks. Memory.capture(..., episode=key, chunk_chars=n) cuts the text at blank lines outside fences into chunks of at most n characters and stores chunk i under <key>-<i>. It does not use the heading rules of load, because an assistant turn often holds a Markdown heading. The extractor reads the whole text once, and each fact cites every chunk. A retry writes nothing, and an edited chunk gets its next revision only. No new Store method or record field.
  • Block display. A fact that cites 11 chunks showed 11 ids in its block line. The block now shows a run of adjacent ids <prefix>-<n>@<rev> (same prefix, n counting up by 1, same revision) as <prefix>-<first>…<last>@<rev>. Other sources show as before. The stored sources do not change. A rule that folded by key prefix alone was rejected: a caller's own keys chat-0002 and chat-0003 look like chunks.
  • Bench. bench.retrieval --ablate ranks with the fused query less one arm. bench.e2e --chunk-chars stores chunks, and the chunk size is part of the kept name. bench.diagnose maps a chunk key to its session.
  • Skill. scripts/capture.py --chunk-chars.

Why

The ablation on the 419 retrieval questions showed that the BM25 arm makes the whole gap between the rankings: fused recall_all@10 0.9570, vector 0.9928, BM25 alone 0.8926. Episodes have no cues, so fused is BM25 plus the content vector. Fusion with equal weights pushes evidence sessions that BM25 ranks 23 to 40 out of the top 10. End to end on sample A at 64,000 characters, vector scored 46 and fused 45.

A session is one episode, so one needed turn brought the whole session into the block. With chunks of 1,000 characters at a budget of 16,000:

Sample Chunks of 1,000, 16,000 Phase 8, whole sessions, 64,000
A 48 of 54, 4,336 reader tokens 45, 12,849 tokens
B 46 of 54, 4,360 tokens 45, 12,676 tokens

The full grid (1,000 / 2,000 / 4,000 at 8,000 / 16,000 / 32,000) and the diagnosis are in docs/benchmarks.md.

Verification

  • make verify and make integration (144 hotdata tests) on the final library commit.
  • New tests cover chunked capture on both drivers, retries, edits, refusals, and the range display. Others prove the same block from MemoryStore and HotdataStore with chunks, for "search" and "cited". Two more prove that capture searches with the store's ranking, and that the default is vector. The skill test runs the new --chunk-chars command.
  • Benchmarks: every number in docs/benchmarks.md comes from a result file in bench/results/. Phase 10 spent $12.77 of the $50 cap.
  • A docs audit of every shipped doc against the final diff.

Notes

  • Behavior change. Callers of HotdataStore without ranking now get the vector ranking. The block format also changes for runs of chunk sources. Both are in CHANGELOG.md.
  • "cited" is unchanged. A fact cites every chunk of its session, so "cited" still gives the whole session. Only "search" saves tokens with chunks.
  • Extraction with chunks is not identical. A fact line with a range is a little longer. The known-fact list that the extractor reads holds 2,000 characters, so it can hold one fact less. Part of one keep run paid for extraction again.
  • At 32,000 with chunks of 1,000, the block held 29.9 chunks, so k=30 and not the budget was the limit.
  • BM25 favors long sessions a little (length percentile 0.645 against 0.583 for vector). This was raised at the task 1 stop, and judged a cost of the arm, not of the data.
  • Found on the way: hotdata-framework 0.14.1 list_managed_databases reads only the first 100 databases, and HotdataStore.provision uses it. Filed as list_managed_databases and resolve_managed_database read only the first page of databases sdk-python-framework#89. Not fixed here.
  • The roadmap closes phase 10 in a separate PR after this merges.

bench.retrieval takes --ablate, which also ranks each question with the fused query
less one arm at a time: bm25-content, content-cue, and bm25. bench/ablation.py builds
those queries. With every arm, its query is the query of ranking="fused".
…h chunks

HotdataStore ranks by content vector alone by default, as MemoryStore does, and
ranking="fused" stays as an option. Memory.capture takes chunk_chars with episode: it
cuts the text at blank lines into chunks, stores chunk n under <episode>-<n>, and each
fact cites every chunk. The extractor still reads the whole text. The skill capture
script takes --chunk-chars, and the bench rankings default to vector.

docs/benchmarks.md reports the ablation of task 1 and the memory arm with each ranking.
bench.e2e takes --chunk-chars with --session-episodes, and capture stores each session
as chunks of that size. The chunk size is part of the kept name. bench.diagnose maps a
chunk key to its session.
A fact captured in chunks cites every chunk, so its line in the block grew by about
300 characters at 1,000-character chunks. The block now shows a run of sources
<prefix>-<n>@<revision>, with n counting up by 1 and one revision, as
<prefix>-<first>…<last>@<revision>. The stored sources do not change.
The bench README names chunks in the kept name and limits the BM25 warnings to the fused
ranking. The benchmarks report which keep runs paid for extraction, and round each median
down. The guarantees, contracts, and changelog state the range rule in full. The plan
gives task 3 as it was built.
@rohan-hotdata
rohan-hotdata requested a review from a team as a code owner October 18, 2026 14:41
@rohan-hotdata
rohan-hotdata requested review from shefeek-jinnah and removed request for a team October 18, 2026 14:41
@rohan-hotdata
rohan-hotdata merged commit a0145ee into main Oct 19, 2026
3 checks passed
rohan-hotdata added a commit that referenced this pull request Oct 21, 2026
The roadmap closes phase 10 with PR #30 and records what it changed for later phases:
the extractor prompt moves with the block line, a reuse at a new budget pays for the
reader, and the framework listing reads only 100 databases. Two items move to Deferred:
precise chunk citations, and the first-page listing (hotdata-dev/sdk-python-framework#89).
The owner paused the workstream before phase 11 on 2026-10-19. The plan drops phase 10,
adds its numbers to the starting point, and notes what phase 10 changed for phase 11.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Phase 10: the ranking, and episode chunks

1 participant