Repository navigation
Phase 9: faster evaluation - #28
Merged
Merged
Conversation
With --keep, the memory arm runs each question over a local database of its own, named by the sample, the extractor variant and a digest of its prompt, how the facts cite their session, the memory ranking, the last commit that changed src/hotmemory, and the question id. The database stays after the question, and a question that raises deletes it. The runner refuses --keep when src/hotmemory has changes that are not committed, when a kept database of the run exists already, and with an arm other than memory. The result file records the name. plan.md task 1 now says one database for each question: BM25 ranks the whole table before the namespace filter, so a shared table changes the block.
With --reuse, the memory arm opens the kept database of each question, skips capture, and runs recall, the reader, and the judge. The runner refuses when a kept database of the run does not exist, and never deletes a kept database in a reused run. local_store takes `delete` (always, on_error, never) in place of `keep`. A reused row has None for facts written and sessions at the limit, and each memory row now records the SHA-256 of its block. Also fixes the name of the database of bench.diagnose, which the task 1 commit passed without its prefix.
When the result file names kept databases (a run with --keep or --reuse), bench.diagnose opens the kept database of each missed question and reads its memory, with no capture replay. It refuses when a kept database does not exist, and never deletes one. A result file without kept databases replays capture as before. The diagnosis of one miss moves into `diagnose`, and the output records the kept name.
With --smoke, the runner runs the 18 questions of bench/sample-smoke.json, 3 of each type drawn from sample A with seed 20261016, over one MemoryStore with a fixed clock, in the process. The result file records the smoke sample, the memory driver, and the vector ranking, and is named e2e-smoke-<tag>.json. --smoke refuses --keep, --reuse, --workers, and --sample. MemoryStore ranks by vector only, so capture shows the extractor a different list of known facts than a run on the local engine, and the first smoke run of a configuration pays for its extraction (observed: a replay stops at the first extractor call).
The runner writes bench/results/<name>.rows.jsonl as it runs: a header line with the arms, the configuration, the sample, the limit, and the kept name, then one line for each finished question, appended under a file lock. With --resume, it reads the file, refuses a header of another run, and runs only the questions that the file does not hold. Without --resume, it refuses to start over an existing rows file. The result file is written, and the rows file deleted, only when every question finished. A question that raises is listed as failed, its namespaces are emptied, and the next question runs. The error text crosses the process pool as a string, so an APIConnectionError no longer breaks the pool. CacheMiss still stops the shard. A resumed --keep run deletes the kept databases of the questions that did not finish before it starts again.
docs/benchmarks.md records the proof. On sample A, from the caches, with 4 workers: search at 64,000 with --keep took 762.4 s and equals the phase 8 run on all 54 rows. search at 32,000 and cited at 64,000 with --reuse took 13.0 s and 15.3 s, and equal their phase 8 runs on all 54 rows. search at 64,000 with --reuse took 11.1 s, with the same block SHA-256 as the --keep run for each question. The 54 kept databases hold 192 MB. The diagnosis of the --keep run read the kept databases in 7 s and equals the phase 8 diagnosis on all 9 misses.
bench/README.md gets the options --keep, --reuse, --smoke, and --resume, and a section "Faster runs": the name of a kept database and its parts, the refusals, the smoke sample and why its first run pays for extraction, and the rows file. The diagnosis paragraph says that a result file with kept databases is read without capture. The result file name of a smoke run and the new field block_sha256 are listed.
docs/benchmarks.md records the smoke run: the first run paid $1.07 for 842 extractor calls in 2,080 s, and the replay took 12.7 s at $0 with the same answer, verdict, and block SHA-256 for each of the 18 questions. The first replay command waited 563 s in all, for a cause that was not found, and a later one took 14.9 s. The docs audit: the docstring of bench.e2e names --keep, --reuse, the rows file, and --resume, and bench/README.md says that a worker stops if it cannot empty the namespaces of a failed question.
--smoke now accepts --workers. Each worker runs its questions over a MemoryStore of its own, with the fixed clock, and empties the memory of each question after it, so a question gives the same row in any worker. A replay of the smoke run with 4 workers took 3.8 s and gave the same answer, verdict, block SHA-256, and count of facts for each of the 18 questions as the first run. A first run with 4 workers, which pays for extraction, was not measured.
rohan-hotdata
requested review from
eddietejeda
and removed request for
a team
October 16, 2026 11:48
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
rohan-hotdata
added a commit
that referenced
this pull request
Oct 16, 2026
The roadmap closes phase 9 with PR #28 and records what it changed for later phases: one kept database for each question, kept memory that a library commit makes unusable, and a first smoke run that pays for extraction. Phase 10 is open as issue #26. The plan drops phase 9, and phase 10 says how its runs use --keep, --reuse, and --smoke --workers 4. Phase 11 gets a part that measures recall over a narrow scope in a shared table, because BM25 ranks the whole table before the namespace filter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes the end-to-end runner fast for runs that change only
recallor the reader, adds an 18-question smoke run, and keeps finished work when a run stops. It changesbench/and the docs only; no file insrc/hotmemorychanges. Closes #25.How
--keep(task 1): the memory arm runs each question over its own local database and keeps it. The name holds what decides the memory:hotmemory-kept-<sample>-<extractor><prompt digest>-<cites>-<ranking>-<last src/hotmemory commit>-<question id>. Settings that act only inrecall(episodes, budget, k) stay out of the name, so one kept memory serves many runs. It refuses whensrc/hotmemoryhas uncommitted changes or the databases exist; a failed question deletes its database.captureshows the extractor. Agreed on Phase 9: faster evaluation #25.--reuse(task 2): opens the kept databases, skipscapture, refuses when one is missing, never deletes one.local_storetakesdelete(always,on_error,never). Each memory row recordsblock_sha256.bench.diagnose(task 3) reads a kept database when the result file names one, with no capture replay.--smoke(task 4): the 18 questions ofbench/sample-smoke.json(3 per type from sample A, seed 20261016) onMemoryStore, one store in each worker.--resume(task 5): each finished row is appended tobench/results/<name>.rows.jsonlunder a file lock, after a header with the configuration.--resumeskips the rows it holds and refuses another configuration's file. A failed model or engine call fails only its question, and its error crosses the process pool as text, which fixes theBrokenProcessPoolof phase 8. A run with a failed question writes no result file.Verification
All observed on the local stack, sample A,
--extractor detail --session-episodes, from the caches at $0 except where a cost is given (docs/benchmarks.md, "Faster evaluation"):--keep, 4 workers--reuse--reuse--reuse--reuse--keeprun on 54 of 54--keeprunThe 54 kept databases hold 192 MB. A replay killed after its first row kept that row, and
--resumefinished the other 2 with answers equal to phase 8.make verify(252 passed) andmake integration(120 passed) pass.Not verified: a real loss of the internet connection (a unit test raises
ConnectionErrorin the reader), and a first smoke run with 4 workers. The first cached smoke command waited 563 s in all for a cause that was not found; later tries took about 15 s.Notes
src/hotmemorymakes every kept database unusable, by design:capturecallsrecallfor the known facts, so any library change can change the memory. Such a change also makes the next smoke run pay for extraction again. Kept databases are deleted by hand.MemoryStoreranks by vector only, so the first smoke run of a configuration misses the extractor cache of the engine runs and pays.--smoketakes--workers, which widens the "in the process" of task 4 in Phase 9: faster evaluation #25.