Repository navigation
Phase 11, part 1: a large memory store, the sorted index made optional, and LongMemEval-S in full - #32
Conversation
…error, and compose drops the retired SQL writes setting
…fault A write into a table of 188,893 rows took 8 to 20 s with the sorted index, about 1.4 s with BM25 alone, and about 0.4 s with no index, for 1 row as for 12 rows, by INSERT or by load. No query of HotdataStore needs the index. provision and open take sorted_index; a database that has the index opens without the option.
…of a background run bench/scale.py --prefill fills background/lme_s with the chunks of the sessions outside samples A and B. bench/e2e.py --background runs each question in that database, recalls over the question and the background, and records the time of each write. --sample all runs every question. The result file records chunk_chars, and bench/diagnose.py rebuilds the block of a background run, checks its SHA-256, and counts the episodes kept by as_of.
…mEval-S in full LongMemEval-S, all 500 questions, at the phase 10 configuration: 439 of 500 (87.8%), and 88.0% on the 392 questions outside samples A and B. Samples A and B over a store with 188,893 chunks of other sessions: 39 and 35, against 48 and 46 from a fresh store, with the diagnosis of the loss. The cost of the BM25 and sorted indexes on writes. The plan and the roadmap move the large store test to LongMemEval-M.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| A store holds a lock across the read and the load of each write, so one store writes in | ||
| order. One process writes to a database. Two processes that put the same key can both read | ||
| the same current revision and both write the next one. Then the later load replaces the |
There was a problem hiding this comment.
nit: State that a guarded INSERT keeps the earlier row, not the later one (not blocking).
A put of a new key now goes through the guarded INSERT in _load (src/hotmemory/hotdata.py:590). Two processes that put the same new key both write id@1. The second INSERT finds the id and writes 0 rows, so the earlier write wins. "The later load replaces the earlier row" holds only for the upsert path. The same sentence appears in docs/guarantees.md:41. AGENTS.md requires both documents to match the behavior.
There was a problem hiding this comment.
The full diff was omitted from the review prompt. The patch was fetched with gh pr diff. Reviewed in full: src/hotmemory/hotdata.py, tests/test_hotdata.py, tests/test_hotdata_listing.py, tests/test_hotdata_retry.py, bench/scale.py, and the docs/contracts.md concurrency section. The other bench and docs files were not read line by line. CI checks were still pending when this review started.
… the same new key The first revision of a new key, in a write of at most 5 new rows, goes by the guarded INSERT, which skips an id that the table holds. A later revision, or a write of more than 5 rows, goes by upsert load, where the later load wins.
There was a problem hiding this comment.
The full diff was omitted from the review prompt. This cycle reviewed the diff since adaa611 (docs/contracts.md, docs/guarantees.md) and checked it against src/hotmemory/hotdata.py:524-552. The earlier nit on the guarded INSERT is fixed: both documents now match the insert/upsert choice in _load_records.
What this does
Measures hotmemory with a large memory store, makes the sorted index on
created_atanoption (off by default) because it made every write into a large table slow, and runs
LongMemEval-S on all 500 questions: 439 of 500 (87.8%).
How
HotdataStore.provision/opentakesorted_index(src/hotmemory/hotdata.py). BM25stays; vector indexes stay experimental. Existing databases open as before.
INSERTfor writes of ≤5 new rows,provisionpages every database, current revisions read in batches of 500 keys, BM25 scope filter
inside the search.
bench/scale.py(scale harness,--prefill),bench/e2e.py(--background,--sample all, write timings,chunk_charsrecorded),bench/diagnose.py(background runs, block SHA-256 check,as_ofcounts),bench/retrieval.py --data m.Why
BM25 alone, ~0.4 s with none, for INSERT and load alike. No HotdataStore query needs it.
were too noisy.
Verification
make verify; driver tests against the local engine (26 passed, 1 expected failure),including 2 new tests for
sorted_index.with a diagnosis of 25 misses whose blocks all matched their run's SHA-256.
Notes
recallappliesas_ofafter the top k, so a store with many laterrecords can leave a block half empty (planned in phase 13). One
RESOURCE_LOCKEDwith asingle writer, cause unknown.
store scopes apart. See docs/benchmarks.md.