From cddebb8c0f1666159d82b642dc794dceed216a9d Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 21 Oct 2026 11:55:28 +0530 Subject: [PATCH 01/14] Docs: fixes after phase 10 --- README.md | 9 ++++++--- docs/contracts.md | 3 +++ docs/guarantees.md | 2 +- docs/internal/architecture.md | 30 ++++++++++++++++++++++-------- docs/internal/plan.md | 4 ++-- 5 files changed, 34 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 60d39a8..0c75831 100644 --- a/README.md +++ b/README.md @@ -151,13 +151,16 @@ sample B on 2026-10-14. Later rows give their own date. The reader and the judge | The block of `recall`, after `capture` with a `gpt-4o-mini` extractor and the `baseline` prompt | 19 of 54 (35.2%) | 27 of 54 (50.0%) | | The same, on 2026-10-14, with the `detail` prompt and a session source on each fact | 31 of 54 (57.4%) | 32 of 54 (59.3%) | | The same, on 2026-10-15, with each session stored as an episode, and `recall` with `episodes="search"` and a budget of 64,000 characters, about 12,800 tokens | 45 of 54 (83.3%) | 45 of 54 (83.3%) | +| The same, on 2026-10-18, with each session stored as chunks of 1,000 characters (`capture` with `chunk_chars=1000`), and `recall` with `episodes="search"` and a budget of 16,000 characters, about 4,300 tokens | 48 of 54 (88.9%) | 46 of 54 (85.2%) | Facts alone score below the full history. With the `baseline` prompt, the extractor did not store the answer in 19 of the 35 misses on sample A, and in 16 of the 27 misses on sample B. With the `detail` prompt, these counts are 16 of 23 and 14 of 22. With the -episodes that a search adds after the facts, the memory arm scores above the full history -and equal to the 10 sessions on sample A, with less than half of their tokens. With 54 -questions, each score can be about 10 points off. With the `baseline` prompt, the memory +whole sessions that a search adds after the facts, the extractor did not store the answer +in 5 misses on each sample. With chunks of 1,000 characters, this count is 2 on each +sample. The chunk row scores above the full history on both samples. It scores 3 more +than the 10 sessions on sample A and 2 less on sample B, with about one seventh of their +tokens. With 54 questions, each score can be about 10 points off. With the `baseline` prompt, the memory arm differed by 14.8 points between the two samples. In the cloud, a search took 361 ms at p50 with the vector ranking, and 998 ms with the fused ranking. diff --git a/docs/contracts.md b/docs/contracts.md index fe7b985..cd39d53 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -213,6 +213,9 @@ with that name, or creates it. It is safe to run again. - Two processes that provision the same new name at the same moment can both create a database, because the platform has no conditional create. The driver does not prevent this. +- `provision` reads only the first page of the database list, at most 100 databases, newest + first. In a workspace with more than 100 databases, it can miss an existing database with + the name and create a second one. This was read from source and not observed. `HotdataStore.open(database_id, ...)` opens a database by its id, after the same checks. `client` is a `HotdataClient` from `hotdata-framework`, and defaults to diff --git a/docs/guarantees.md b/docs/guarantees.md index 624af0f..a4db71e 100644 --- a/docs/guarantees.md +++ b/docs/guarantees.md @@ -51,7 +51,7 @@ A row that no test proves yet names the phase that will prove it. | A record is past its `forget_after`. Does a `put` of the same content write it again? | Yes. Deduplication applies only to a current revision that `list` can return. The put writes a new revision, and `list` and `search` return it. | answered | `test_put_after_forget_after_writes_a_new_revision` | | Can a caller change a stored record? | No. A record is frozen, and the store copies `payload` on the way in and on the way out. A change to a dict that the caller holds does not reach the store. | answered | `test_stored_record_cannot_change` | | Where does `HotdataStore` keep an episode? | A `put` of kind `episode` writes to `episode_v1`, and every other kind writes to `memory_v1`. `get`, `history`, `list`, `search`, `delete`, `sweep`, and `list_namespaces` read both tables. The cue vector of a record of either table is a row in `cue_v1`. | answered | `test_episode_goes_to_its_own_table`, `test_key_cannot_cross_the_episode_line` | -| Is `provision` safe to run again? | Yes. If one database has the name, `provision` opens it after a check of its tables, columns, indexes, and the recorded model and vector size. If more than one database has the name, or the one found has another layout, it raises `LayoutError` and creates nothing. Two processes that provision the same new name at the same moment can both create a database, because the platform has no conditional create. | answered | `test_provision_again_opens_the_same_database`, `test_provision_refuses_a_duplicate_name`, `test_provision_refuses_another_layout`, `test_provision_refuses_another_embedding_model` | +| Is `provision` safe to run again? | Yes. If one database has the name, `provision` opens it after a check of its tables, columns, indexes, and the recorded model and vector size. If more than one database has the name, or the one found has another layout, it raises `LayoutError` and creates nothing. Two processes that provision the same new name at the same moment can both create a database, because the platform has no conditional create. `provision` reads only the first page of the database list, at most 100 databases, so in a workspace with more than 100 databases it can miss the existing one and create a second one. This was read from source and not observed. | answered | `test_provision_again_opens_the_same_database`, `test_provision_refuses_a_duplicate_name`, `test_provision_refuses_another_layout`, `test_provision_refuses_another_embedding_model` | | Does a plain vector index serve rows loaded after its build? | Only without a filter. With a `WHERE` filter in the same query, the engine searched the index and returned only the rows that were there at the build. So the vector rankings of `HotdataStore` do not use the index. They filter first and rank every row in scope by a scan, so a search in a narrow scope returns the top k rows of that scope. At 100,000 rows of 64 dimensions, a search in a scope of 1 percent of the rows took a median of 97 ms fused and 26 ms by content vector alone. | answered [measured, local] 2026-10-08 | `test_filtered_search_finds_every_later_write`, `test_search_in_a_narrow_scope_returns_its_top_k` | | Does `HotdataStore` agree with `MemoryStore`? | Ranked by content vector alone, it returns the same records in the same order, with distances equal within 1e-6. With the fused ranking, it returns the same set when k covers every record. On the test fixture, the largest distance difference was 4.0e-8. | answered [measured, local] 2026-10-08 | `test_vector_ranking_matches_the_oracle`, `test_fused_ranking_returns_the_oracle_set` | | What does `load` store for a document? | One episode for each chunk, under the key `-`, with the first heading and the heading path at the top. Each fact that the extractor returns names the episode of its chunk in `sources`. A fact with no `valid_from` is valid from the `observed_at` of the load. A heading inside a fenced code block does not start a chunk, and no chunk body is longer than `chunk_chars`, unless one line is. | answered | `test_load_stores_each_chunk_as_an_episode_and_links_its_facts`, `test_a_loaded_fact_is_valid_from_the_observed_time`, `test_a_heading_inside_a_fence_is_not_a_heading`, `test_every_body_fits_the_size`, `test_a_cut_table_repeats_its_header` | diff --git a/docs/internal/architecture.md b/docs/internal/architecture.md index 5622bb1..0986cb1 100644 --- a/docs/internal/architecture.md +++ b/docs/internal/architecture.md @@ -10,7 +10,7 @@ the SQL, and the loads that carry them out. Each claim has one of these sources: -- The code of hotmemory at the commit that adds this page. Such a claim has no mark. +- The code of hotmemory at commit `a0145ee`, the merge of phase 10. Such a claim has no mark. - A measurement in [contracts.md](../contracts.md) or [guarantees.md](../guarantees.md), marked [measured] with its date. - A look at the local stack on 2026-10-12, marked [observed, local]. @@ -223,6 +223,12 @@ name and the vector size must equal the arguments. Two processes that provision the same new name at the same moment can both create a database. The platform has no conditional create, and the driver does not prevent this. +`list_managed_databases` of `hotdata-framework` 0.14.1 reads only the first page of the +list, newest first, and a page holds at most 100 databases. [read from source, not +observed] Thus, in a workspace with more than 100 databases, `provision` can miss an +existing database with the name and create a second one. The issue is +hotdata-dev/sdk-python-framework#89. + ## A write Every write goes through one method, `_write_many`. A `put` sends it one draft. A writer @@ -335,7 +341,7 @@ sequenceDiagram end C->>M: capture(text, scope, extractor, actor, observed_at) M->>H: search(text, [scope], kinds fact, profile, procedure, k=10) - H->>API: one vector SELECT on memory_v1 + H->>API: one SELECT with the ranking of the store API-->>H: ranked rows H-->>M: hits Note over M: recall keeps the lines that fit in 2,000 characters @@ -393,19 +399,25 @@ flowchart TD eps -- "no" --> out["Return the records and the block"] eps -- "cited" --> get["get each episode that the facts
of the block name in sources"] eps -- "search" --> find["search the episodes,
kind episode, top k"] - get --> whole["Add whole episodes while they fit
in what the facts left"] - find --> whole + get --> epasof{"as_of given?"} + find --> epasof + epasof -- "yes" --> epkeep["Keep the episodes valid at as_of"] + epasof -- "no" --> whole + epkeep --> whole["Add whole episodes while they fit
in what the facts left"] whole --> out ``` -With `episodes="cited"`, `recall` sends one `get` for each distinct source that is an -episode id in the allowed scopes. With `episodes="search"`, it sends a second search with +With `episodes="cited"`, `recall` sends one `get` for each distinct source that is a +valid record id in an allowed scope. Then it drops each record of another kind than +`episode`. With chunks, a fact cites every chunk of its session, so a fact that cites 11 +chunks costs 11 `get` calls. With `episodes="search"`, it sends a second search with the filter `kind` = `episode`, so that search reads `episode_v1`. With chunks, each chunk is an episode, so that search ranks each chunk on its own. The episodes that `capture` and `load` write have no cues, so the fused ranking of that search is BM25 and the content vector. -The `as_of` filter runs in the process, after the search. Thus `recall` with `as_of` can +The `as_of` filter runs in the process, after the search. It runs on the facts, and then +on the episodes. Thus `recall` with `as_of` can return fewer than k records. The search keeps k records before the filter, so a record that is valid at `as_of` but ranks below k is missing. @@ -602,7 +614,7 @@ sequenceDiagram else a new or changed chunk Note over M: The episode id is known before the write:
scope/key@next revision M->>H: search for the chunk (as capture does) - H->>API: vector SELECT on memory_v1 + H->>API: one SELECT with the ranking of the store M->>X: extractor(chunk, observed_at, current records) X-->>M: facts Note over M: Add the episode id to the sources of each fact @@ -666,3 +678,5 @@ These claims on this page come from reading source, or were not measured: - Where the cloud keeps the catalog, the parquet files, and the index files. - Whether the sorted index on `created_at` serves `list` and `sweep`. - The cost of a whole document `load` in the cloud. +- The first page of `list_managed_databases`, and a second database that `provision` can + create with more than 100 databases in the workspace. diff --git a/docs/internal/plan.md b/docs/internal/plan.md index b515703..a20e331 100644 --- a/docs/internal/plan.md +++ b/docs/internal/plan.md @@ -157,8 +157,8 @@ What phase 10 changed for this phase: - The default ranking is the vector ranking. The loss of BM25 hits in a narrow scope now applies only with `ranking="fused"`. -- With chunks, a session is many episode rows: about 11 for a session of 10,800 - characters, in chunks of 1,000. A store with chunks reaches each size sooner. +- With chunks, a session is many episode rows: at least one row for each 1,000 characters + of the session, with chunks of 1,000. A store with chunks reaches each size sooner. - `list_managed_databases` of `hotdata-framework` 0.14.1 reads only the first 100 databases, and `HotdataStore.provision` uses it. A run that makes many databases, or a workspace that holds more than 100, can make `provision` miss a database and create a From 314a08cb0c823b2d694205234a26740c86f870bd Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 21 Oct 2026 13:03:23 +0530 Subject: [PATCH 02/14] Open phase 11: a large memory store --- docs/internal/plan.md | 183 +++++++++++++++++++++++++++++++-------- docs/internal/roadmap.md | 23 +++-- 2 files changed, 163 insertions(+), 43 deletions(-) diff --git a/docs/internal/plan.md b/docs/internal/plan.md index a20e331..ffd92f0 100644 --- a/docs/internal/plan.md +++ b/docs/internal/plan.md @@ -5,7 +5,8 @@ Status: agreed, 2026-10-13. The owner agreed to each decision and to the target three phases after it: faster evaluation, the ranking with episode chunks, and a large memory store. The later phases moved to 12 to 15. Phase 9 closed on 2026-10-16 with PR #28. Phase 10 closed on 2026-10-19 with PR #30. On 2026-10-19 the owner paused this -workstream before phase 11, to work on other things first, so no phase is open. This file +workstream before phase 11, to work on other things first. On 2026-10-21 the owner +resumed it with phase 11, a large memory store. This file holds the phases still to come, 11 to 15. Each phase becomes one GitHub issue and one pull request, worked in task order. When a phase closes, `roadmap.md` gets its paragraph, and this file keeps only the phases still open. The plan of phase 5 is in git history, at commit `4a6fb36`. The plan of phase 6, with the decisions taken during @@ -128,41 +129,144 @@ The owner agreed to each item on 2026-10-13. ## Phase 11. A large memory store -Goal: numbers for hotmemory at 10,000, 100,000, and 1,000,000 memories: the time of a -write and of a search, and the accuracy of retrieval when the store is large. The owner -asked for it on 2026-10-16. Phase 10 closed on 2026-10-19, and the owner paused the -workstream before this phase. The tasks are planned when the owner resumes it. - -The parts discussed so far: - -- A synthetic scale test, with no reader model: load 10,000, 100,000, and 1,000,000 - records, from the facts and sessions that the benchmark already extracted. Measure the - rate of writes with table loads, and with native DML when the engine has it. Measure - search time at p50 and p95 for the fused and the vector ranking, for scopes that hold 1, - 10, and 100 percent of the rows, on the local stack and in the cloud. -- The index check. Today the vector rankings filter first and scan, because a filtered - search through the vector index misses rows loaded after the build. When the engine - fixes that, the vector rankings move to the index, and tests prove that every write, - revision, and delete is found through the vector index and the BM25 index. -- Retrieval at scale: LongMemEval-M, about 500 sessions for each question, scored by - `recall_any@k` with no reader model. -- Recall over a narrow scope in a shared table. BM25 ranks the whole table before the - namespace filter, so a scope that holds a small share of the rows can lose BM25 hits. - Phase 9 found this in the benchmark runner. Measure the score at each scope size. -- A comparison with other memory layers in one harness, such as MemoryBench, with the - same questions, reader, and judge. It needs accounts for each service and the owner's - yes on its cost. - -What phase 10 changed for this phase: - -- The default ranking is the vector ranking. The loss of BM25 hits in a narrow scope now - applies only with `ranking="fused"`. -- With chunks, a session is many episode rows: at least one row for each 1,000 characters - of the session, with chunks of 1,000. A store with chunks reaches each size sooner. -- `list_managed_databases` of `hotdata-framework` 0.14.1 reads only the first 100 - databases, and `HotdataStore.provision` uses it. A run that makes many databases, or a - workspace that holds more than 100, can make `provision` miss a database and create a - second one with the same name. It is filed as hotdata-dev/sdk-python-framework#89. +Goal: proof that hotmemory recalls well from a store that already holds many memories. +The owner set the aim on 2026-10-21: one store with 100,000 memories or more, where +search stays fast and exact, and where the right memory still comes back. The phase +gives numbers at 10,000, 100,000, and 1,000,000 rows: the time of a write and of a +search, the accuracy of retrieval, and the accuracy of answers from a shared store. + +A large store here means many memories in the store during a run. It does not mean only +many questions. The owner's hypothesis: hotmemory does well at 100,000 to 1,000,000 +memories now that the engine accepts SQL writes. A second hypothesis, also to test: cues +help retrieval on facts. In phase 10 the fused ranking lost to the vector ranking on +episodes, but episodes have no cues. + +What the research of 2026-10-21 found (read from source and the public datasets, not +observed unless marked): + +- The engine accepts `INSERT` and `CREATE TABLE` on managed tables, with no setting, + since 2026-10-18. It refuses `UPDATE`, `DELETE`, `MERGE`, `ON CONFLICT`, and + `RETURNING`. No work on `UPDATE` or `DELETE` has started. Whether the cloud accepts + `INSERT` is not observed. An `INSERT` goes through the same append path as a load. +- `execute_sql` of `hotdata-framework` 0.14.1 retries a dropped connection up to 3 + times, and the engine does not drop a repeated `INSERT`. So a retry can write a row + twice. +- Each write reads the current revision of each key first. With every question in one + table of 19,864 rows, a write took 1.91 s at p50, against 0.29 s with one question per + table [observed, phase 9]. +- Of the writes of `HotdataStore`, an `INSERT` can take over rows that are all first revisions, new cue rows, and the seed + and meta rows. A closed old revision, `forget`, and `sweep` still need keyed loads. +- With the vector ranking, a shared table gave the same sessions in the same order as one + question per table [observed, phase 9]. So a shared store with the vector ranking + keeps the capture cache, and one store can hold every question. +- LongMemEval-M has about 500 sessions for each question. Its file is 2.74 GB, under the + MIT license. All 500 questions of LongMemEval-S in one store give about 150,000 memory + rows. BEAM, with conversations of 1 million and 10 million tokens, is the other public + test of a large store. +- The filtered vector index defect is still open in the engine, so the vector rankings + keep their scan in this phase. + +Unit costs for the estimates, from the phase 10 result files: capture costs about +$0.0005 and 1.06 s for each session with 4 workers. The reader and the judge cost about +$0.015 for each question at a budget of 16,000. Only the run of 54 questions on +LongMemEval-S is observed. Every other estimate below is extrapolated from these. + +Tasks: + +1. SQL writes in the cloud. Send one `INSERT` to a throwaway cloud database, and run M4 + again: one hundred single-row `INSERT` statements against one hundred single-row + loads, then the same at flushes of 100 and 1,000 rows of 1,536 dimensions. Record + the size limit of a query request if one shows. Correct the claims that the cloud + refuses SQL writes in `docs/guarantees.md`, `docs/internal/architecture.md`, and + `docs/internal/brief.md`. Remove the retired `RUNTIMEDB_ENGINE__SQL_WRITES` from + `compose.yaml` and `scripts/measure_local.py`. No model cost. About 15 minutes. +2. An `INSERT` path in `HotdataStore`, only if task 1 shows a clear gain. A flush whose + rows are all first revisions, and the new cue rows, go by `INSERT`. Rows that close a + revision stay on the upsert load. The path must not write a row twice on a retry: + either send the statement with no retry, or check by id before a second try. A flush + that mixes both kinds of row sends the upsert load last, so a crash leaves no second + current revision. Update `docs/contracts.md` and `docs/guarantees.md` in the same + commit, with tests in the conformance suite and `make integration`. +3. `provision` reads every page of databases. It pages the raw API instead of + `list_managed_databases`, so a workspace with more than 100 databases does not get a + second database with one name. This works around + hotdata-dev/sdk-python-framework#89 until the framework fixes it. +4. The scale harness, a script outside the package. It builds a store from real text, + not random vectors: + - The rows: LongMemEval sessions as chunks of 1,000 characters, and the facts that + earlier runs extracted, from the reply cache. Sessions of LongMemEval-M fill the + store past the rows of LongMemEval-S. No row is a copy of another. + - The embeddings: `text-embedding-3-small` at 1,536 dimensions, through the bench + cache. + - The scopes: namespaces that hold 1, 10, and 100 percent of the rows. + - The questions: a fixed set of LongMemEval questions whose evidence sessions are in + the store. + + At each checkpoint of 10,000, 100,000, and 1,000,000 rows, the script records: + - The write rate with loads, and with `INSERT` if task 2 took it, and the time of the + read before each write. + - The search time at p50 and p95, by vector and fused, at each scope size. + - For each search, whether the top k equals a brute-force top k in Python over the + same rows. While the rankings scan, any difference is a defect. + - The `recall_any@k` against the evidence sessions. + - For the fused ranking, the share of BM25 hits that pass the namespace filter. + + It runs on the local stack first, then in the cloud. Embeddings for 1,000,000 rows + cost about $2 to $5. The time of the load is not known. The script stops after + 10,000 rows, and the owner gets the extrapolated time of 100,000 and 1,000,000 rows + before it goes on. +5. Retrieval on LongMemEval-M. Score `recall_any@k` with no reader model in two forms: + one shared table with a namespace for each question, and one pool of the unique + sessions, where the sessions of other questions are distractors. The vector ranking + first, and the fused ranking beside it. About $1.60 of embeddings, and 1.5 to 2.5 + hours on the local stack. The 2.74 GB file stays out of git. +6. Answers from one shared store. The runner gains a mode that keeps every question in + one database, with a namespace for each question, and the vector ranking. + - First, sample A captures into the shared store from the cache. Each row must equal + its phase 10 row, at no model cost. If a row differs, stop and diagnose. + - Then the other questions of LongMemEval-S capture into the same store, to about + 150,000 rows. About $9 to $10 for capture, because about 108 questions are in the + cache, and $1.20 for embeddings. About 5.5 to 7 hours. + - Then samples A and B answer from the full store at the phase 10 configuration. + About $1.60 for the reader and the judge. +7. The cue test on facts. Compare the fused ranking, with cues, and the vector ranking on + facts only, with one question for each database, on samples A and B. The fused ranking + changes the known facts that the extractor reads, so part of capture pays again. Up to + about $2 for each sample, and about 45 minutes for each. +8. Record the numbers in `docs/benchmarks.md`, the measured claims in + `docs/guarantees.md` and `docs/internal/architecture.md`, and the phase in + `roadmap.md`. + +The phase spends about $20 to $25 in all, under the cost limit of $50. Each paid or long +run stops first for the owner's yes, with its estimate. + +Acceptance: + +- Each number of task 4 at 10,000 and 100,000 rows, locally and in the cloud, and at + 1,000,000 rows if the owner agrees to its time. +- Every brute-force check of task 4 passes. +- `recall_any@k` on LongMemEval-M, in both forms of task 5. +- Samples A and B answered from a store of about 150,000 rows, next to their phase 10 + scores of 48 and 46. +- The cue test answers whether the fused ranking beats the vector ranking on facts. + +Stop conditions: + +- If the shared replay of sample A differs from phase 10, stop before more capture. +- If the extrapolated load of 1,000,000 rows takes more than one night, stop and ask. +- If the cloud refuses `INSERT`, task 2 drops, and the phase goes on with loads. + +Not in this phase, under Deferred in `roadmap.md`: + +- The vector rankings on the vector index. They wait for the engine's fix of the filtered + search. When it lands, tests must prove that every write, revision, and delete is found + through the vector index and the BM25 index. +- A BM25 search that applies the namespace filter inside the search. It needs a change in + the engine, and the request goes to the engine team with the numbers of task 4. +- `UPDATE` and `DELETE` by SQL, until the engine has them. +- The comparison with other memory layers, such as MemoryBench. It needs an account and a + cost for each service, and it does not test a large store. +- BEAM at 10 million tokens. ## Phase 12. Supersede in capture @@ -296,8 +400,7 @@ Acceptance criteria: These wait, and `roadmap.md` lists them under Deferred: - Latency: a conditional write, text loaded only for ranked rows, and the per-request - time of the cloud. Native DML and the filtered vector index move to phase 11, where - they are measured at scale. + time of the cloud. Phase 11 measures SQL writes and the search at scale. - A model-backed extractor in an extra, and decision calls with a probability, such as a classifier that decides supersede or routes a recall. - The other ideas from the hindsight study of 2026-10-09: a chunk-hash delta on @@ -316,4 +419,8 @@ These wait, and `roadmap.md` lists them under Deferred: ## Questions for the owner +- Phase 11: the cloud workspace for the run at 1,000,000 rows, and its cost limit apart + from the model spend. +- Phase 11: a target for the search time, such as p95 in the cloud at 1,000,000 rows. + - Phase 15: the first version number, and whether to publish to TestPyPI first. diff --git a/docs/internal/roadmap.md b/docs/internal/roadmap.md index a85e4ae..4af4349 100644 --- a/docs/internal/roadmap.md +++ b/docs/internal/roadmap.md @@ -28,7 +28,9 @@ conditions of each phase. The library stays deterministic: no code in `src/hotme calls a model. Latency is not a goal, and its work waits under Deferred, except the measurements at scale of phase 11. On 2026-10-16 the owner added phases 9 to 11, and the later phases moved to 12 to 15. On 2026-10-19, after phase 10 closed, the owner paused the -workstream to work on other things first. Phase 11 opens when the owner resumes it. +workstream to work on other things first. On 2026-10-21 the owner resumed it with phase +11, to test hotmemory as a large memory store: recall from one store that holds 100,000 +memories or more. | Phase | Delivers | Status | |---|---|---| @@ -37,7 +39,7 @@ workstream to work on other things first. Phase 11 opens when the owner resumes | 8. Episodes behind facts | `capture` stores its text as an episode that its facts cite. `recall` can add the cited or ranked episodes to its block, within one budget. | closed 2026-10-15, PR #24 | | 9. Faster evaluation | The runner keeps the memory of a run and reuses it for runs that change only `recall` or the reader. A smoke run of 18 questions on `MemoryStore`. A run keeps its finished questions when it stops. No library change. | closed 2026-10-16, PR #28 | | 10. The ranking, and episode chunks | An ablation of the fused ranking, and a default ranking chosen by the benchmark. `capture` stores a session as chunks, so the block holds the chunks that match and not whole sessions, for fewer reader tokens. | closed 2026-10-19, PR #30 | -| 11. A large memory store | The time of a write and a search at 10,000, 100,000, and 1,000,000 memories, by scope size, locally and in the cloud. The vector rankings on the vector index once the engine fixes the filtered search, with tests that every record is found through both indexes. Retrieval on LongMemEval-M. Then a comparison with other memory layers in one harness. | proposed | +| 11. A large memory store | Recall from one store that holds 100,000 memories or more. SQL `INSERT` measured in the cloud, and taken for appends if it is faster. The time of a write and a search at 10,000, 100,000, and 1,000,000 rows built from real text, by scope size, locally and in the cloud, with each top k checked against a brute-force ranking. Retrieval on LongMemEval-M. Samples A and B answered from one store of every LongMemEval-S question. A test of cues on facts. | open | | 12. Supersede in capture | A fact can name the current record that it replaces, and `remember` closes that record. The extractor decides, and the library applies the rule. | proposed | | 13. Search over history | `recall(as_of=T)` returns the revision valid at T, with the as-of filter in SQL, and a date window from the caller. | proposed | | 14. The full benchmark | The final configuration on all 500 LongMemEval-S questions, and a LoCoMo smoke test. | proposed | @@ -53,8 +55,7 @@ taken up now. and is on PyPI. - Latency, from phase 5: a conditional write for more than one writer, text loaded only for ranked rows, and the time of each cloud request. The second workstream measures - accuracy first. Native DML for writes and the filtered vector index search moved to - phase 11 on 2026-10-16, to be measured at scale. + accuracy first. Phase 11 measures SQL writes and the search at scale. - Model calls, from the discussion of 2026-10-13: a model-backed extractor in an extra, and decision calls with a probability, for example to decide a supersede or to route a recall. The library stays deterministic in the second workstream. @@ -71,7 +72,19 @@ taken up now. - The first-page listing of `hotdata-framework`, from phase 10. `list_managed_databases` and `resolve_managed_database` read only the first 100 databases, and `HotdataStore.provision` uses the first. It is filed as - hotdata-dev/sdk-python-framework#89, and waits for the framework fix or for phase 11. + hotdata-dev/sdk-python-framework#89. Phase 11 works around it in `provision`. +- The vector rankings on the vector index, from phase 2. A filtered search through the + vector index misses rows loaded after the build. The defect is reported to the engine + team. Until it is fixed, the vector rankings scan, and phase 11 measures that cost. +- A BM25 search that applies the namespace filter inside the search, from phase 9. BM25 + ranks the whole table before the filter, so a narrow scope in a shared table loses + hits. It needs a change in the engine. Phase 11 measures the loss. +- SQL `UPDATE` and `DELETE`, from phase 11. The engine refuses them, so a closed + revision, `forget`, and `sweep` stay on keyed loads. +- The comparison with other memory layers in one harness, such as MemoryBench, from + phase 11. It needs an account and a cost for each service, and it does not test a + large store. +- BEAM at 10 million tokens, from phase 11, as a later test of one very large store. - The ideas of the hindsight study of 2026-10-09 that the second workstream does not take, for later planning: - A chunk-hash delta on re-capture. From 129e50104b95b062e1e52840fb461249ff243796 Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 21 Oct 2026 13:10:02 +0530 Subject: [PATCH 03/14] Phase 11: provision reads every page of the database list --- CHANGELOG.md | 3 ++ docs/contracts.md | 5 ++-- docs/guarantees.md | 2 +- docs/internal/architecture.md | 10 +++---- src/hotmemory/hotdata.py | 18 ++++++++++- tests/test_hotdata_listing.py | 56 +++++++++++++++++++++++++++++++++++ 6 files changed, 84 insertions(+), 10 deletions(-) create mode 100644 tests/test_hotdata_listing.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ef831a..6642539 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -56,6 +56,9 @@ each one, and its failure message points to this file. - `Memory.recall` and `Memory.profile` read the records of kind `fact`, `profile`, and `procedure` only. They never return an episode, and the counts of `profile` leave episodes out. +- `HotdataStore.provision` reads every page of the database list. Before, it read the + first 100 databases, and in a larger workspace it could create a second database with + the name. - `Memory.capture` takes `episode`, a key. With it, `capture` also stores the text as an episode under that key, after the facts, and each fact names the episode in `sources`. A capture of the same text under the same key writes nothing and calls no extractor. diff --git a/docs/contracts.md b/docs/contracts.md index cd39d53..823793b 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -213,9 +213,8 @@ with that name, or creates it. It is safe to run again. - Two processes that provision the same new name at the same moment can both create a database, because the platform has no conditional create. The driver does not prevent this. -- `provision` reads only the first page of the database list, at most 100 databases, newest - first. In a workspace with more than 100 databases, it can miss an existing database with - the name and create a second one. This was read from source and not observed. +- `provision` reads every page of the database list, so it finds a database with the name + in a workspace with more than 100 databases. `HotdataStore.open(database_id, ...)` opens a database by its id, after the same checks. `client` is a `HotdataClient` from `hotdata-framework`, and defaults to diff --git a/docs/guarantees.md b/docs/guarantees.md index a4db71e..7a607aa 100644 --- a/docs/guarantees.md +++ b/docs/guarantees.md @@ -51,7 +51,7 @@ A row that no test proves yet names the phase that will prove it. | A record is past its `forget_after`. Does a `put` of the same content write it again? | Yes. Deduplication applies only to a current revision that `list` can return. The put writes a new revision, and `list` and `search` return it. | answered | `test_put_after_forget_after_writes_a_new_revision` | | Can a caller change a stored record? | No. A record is frozen, and the store copies `payload` on the way in and on the way out. A change to a dict that the caller holds does not reach the store. | answered | `test_stored_record_cannot_change` | | Where does `HotdataStore` keep an episode? | A `put` of kind `episode` writes to `episode_v1`, and every other kind writes to `memory_v1`. `get`, `history`, `list`, `search`, `delete`, `sweep`, and `list_namespaces` read both tables. The cue vector of a record of either table is a row in `cue_v1`. | answered | `test_episode_goes_to_its_own_table`, `test_key_cannot_cross_the_episode_line` | -| Is `provision` safe to run again? | Yes. If one database has the name, `provision` opens it after a check of its tables, columns, indexes, and the recorded model and vector size. If more than one database has the name, or the one found has another layout, it raises `LayoutError` and creates nothing. Two processes that provision the same new name at the same moment can both create a database, because the platform has no conditional create. `provision` reads only the first page of the database list, at most 100 databases, so in a workspace with more than 100 databases it can miss the existing one and create a second one. This was read from source and not observed. | answered | `test_provision_again_opens_the_same_database`, `test_provision_refuses_a_duplicate_name`, `test_provision_refuses_another_layout`, `test_provision_refuses_another_embedding_model` | +| Is `provision` safe to run again? | Yes. If one database has the name, `provision` opens it after a check of its tables, columns, indexes, and the recorded model and vector size. If more than one database has the name, or the one found has another layout, it raises `LayoutError` and creates nothing. Two processes that provision the same new name at the same moment can both create a database, because the platform has no conditional create. `provision` reads every page of the database list, so a workspace with more than 100 databases does not get a second database with the name. An offline test proves the paging against a fake listing. No workspace with more than 100 databases was tried. | answered | `test_named_follows_the_cursor`, `test_provision_again_opens_the_same_database`, `test_provision_refuses_a_duplicate_name`, `test_provision_refuses_another_layout`, `test_provision_refuses_another_embedding_model` | | Does a plain vector index serve rows loaded after its build? | Only without a filter. With a `WHERE` filter in the same query, the engine searched the index and returned only the rows that were there at the build. So the vector rankings of `HotdataStore` do not use the index. They filter first and rank every row in scope by a scan, so a search in a narrow scope returns the top k rows of that scope. At 100,000 rows of 64 dimensions, a search in a scope of 1 percent of the rows took a median of 97 ms fused and 26 ms by content vector alone. | answered [measured, local] 2026-10-08 | `test_filtered_search_finds_every_later_write`, `test_search_in_a_narrow_scope_returns_its_top_k` | | Does `HotdataStore` agree with `MemoryStore`? | Ranked by content vector alone, it returns the same records in the same order, with distances equal within 1e-6. With the fused ranking, it returns the same set when k covers every record. On the test fixture, the largest distance difference was 4.0e-8. | answered [measured, local] 2026-10-08 | `test_vector_ranking_matches_the_oracle`, `test_fused_ranking_returns_the_oracle_set` | | What does `load` store for a document? | One episode for each chunk, under the key `-`, with the first heading and the heading path at the top. Each fact that the extractor returns names the episode of its chunk in `sources`. A fact with no `valid_from` is valid from the `observed_at` of the load. A heading inside a fenced code block does not start a chunk, and no chunk body is longer than `chunk_chars`, unless one line is. | answered | `test_load_stores_each_chunk_as_an_episode_and_links_its_facts`, `test_a_loaded_fact_is_valid_from_the_observed_time`, `test_a_heading_inside_a_fence_is_not_a_heading`, `test_every_body_fits_the_size`, `test_a_cut_table_repeats_its_header` | diff --git a/docs/internal/architecture.md b/docs/internal/architecture.md index 0986cb1..26b5a78 100644 --- a/docs/internal/architecture.md +++ b/docs/internal/architecture.md @@ -225,9 +225,9 @@ database. The platform has no conditional create, and the driver does not preven `list_managed_databases` of `hotdata-framework` 0.14.1 reads only the first page of the list, newest first, and a page holds at most 100 databases. [read from source, not -observed] Thus, in a workspace with more than 100 databases, `provision` can miss an -existing database with the name and create a second one. The issue is -hotdata-dev/sdk-python-framework#89. +observed] The issue is hotdata-dev/sdk-python-framework#89. So `provision` does not use +it. It pages the list of the SDK with the name as the search text, keeps each database +whose name is exactly the name, and reads each of those by its id. ## A write @@ -678,5 +678,5 @@ These claims on this page come from reading source, or were not measured: - Where the cloud keeps the catalog, the parquet files, and the index files. - Whether the sorted index on `created_at` serves `list` and `sweep`. - The cost of a whole document `load` in the cloud. -- The first page of `list_managed_databases`, and a second database that `provision` can - create with more than 100 databases in the workspace. +- `provision` in a workspace with more than 100 databases. An offline test proves the + paging against a fake listing. diff --git a/src/hotmemory/hotdata.py b/src/hotmemory/hotdata.py index 51bd104..cd316a7 100644 --- a/src/hotmemory/hotdata.py +++ b/src/hotmemory/hotdata.py @@ -20,6 +20,7 @@ import pyarrow as pa import pyarrow.parquet as pq +from hotdata.api.databases_api import DatabasesApi from hotdata.api.indexes_api import IndexesApi from hotdata_framework import HotdataClient, ManagedDatabase from hotdata_framework.client import IndexType, ManagedLoadMode, VectorMetric @@ -62,6 +63,7 @@ MIN_DEPTH = 100 DEPTH_PER_RESULT = 10 LOAD_ATTEMPTS = 8 +LIST_PAGE = 100 FIRST_BACKOFF = 0.25 MAX_BACKOFF = 4.0 @@ -202,7 +204,7 @@ def provision( """ _check_settings(model, dimensions) client = HotdataClient.from_env() if client is None else client - found = [db for db in client.list_managed_databases() if db.description == name] + found = _named(client, name) if len(found) > 1: ids = ", ".join(sorted(db.id for db in found)) raise LayoutError(f"{len(found)} databases are named {name!r}: {ids}") @@ -567,6 +569,20 @@ def _check_settings(model: str, dimensions: int) -> None: raise ValueError("dimensions must be 1 or more") +def _named(client: HotdataClient, name: str) -> builtins.list[ManagedDatabase]: + """Return every database named exactly `name`, from every page of the listing.""" + databases = DatabasesApi(client.api) + ids: builtins.list[str] = [] + cursor: str | None = None + while True: + page = databases.list_databases(limit=LIST_PAGE, cursor=cursor, search=name) + ids.extend(str(summary.id) for summary in page.databases if summary.name == name) + if not page.has_more or not page.next_cursor: + break + cursor = page.next_cursor + return [client.resolve_managed_database(database_id) for database_id in ids] + + def _create(client: HotdataClient, name: str, model: str, dimensions: int) -> ManagedDatabase: """Create the database, load a seed row into each table, index it, and remove the seeds. diff --git a/tests/test_hotdata_listing.py b/tests/test_hotdata_listing.py new file mode 100644 index 0000000..a50f3ae --- /dev/null +++ b/tests/test_hotdata_listing.py @@ -0,0 +1,56 @@ +"""Offline tests of the database listing of `HotdataStore.provision`, with a fake client.""" + +from types import SimpleNamespace +from typing import Any + +from hotdata_framework import ManagedDatabase + +from hotmemory import hotdata + + +class FakeDatabases: + """Serves `names` in pages of `size`, newest first, with the search of the API.""" + + def __init__(self, names: list[str], size: int) -> None: + self.names = names + self.size = size + self.calls: list[str | None] = [] + + def list_databases(self, limit: int, cursor: str | None, search: str) -> Any: + self.calls.append(cursor) + matches = [ + SimpleNamespace(id=f"db-{n}", name=name) + for n, name in enumerate(self.names) + if search.lower() in name.lower() + ] + start = int(cursor or 0) + end = start + min(limit, self.size) + more = end < len(matches) + return SimpleNamespace( + databases=matches[start:end], has_more=more, next_cursor=str(end) if more else None + ) + + +class FakeClient: + api = None + + def resolve_managed_database(self, database_id: str) -> ManagedDatabase: + return ManagedDatabase(id=database_id, description="", default_connection_id="c") + + +def test_named_reads_every_page(monkeypatch: Any) -> None: + names = [f"other-{n}" for n in range(150)] + ["store", "store-2", "store"] + fake = FakeDatabases(names, size=100) + monkeypatch.setattr(hotdata, "DatabasesApi", lambda api: fake) + found = hotdata._named(FakeClient(), "store") # type: ignore[arg-type] + assert [db.id for db in found] == ["db-150", "db-152"] + assert fake.calls == [None] + + +def test_named_follows_the_cursor(monkeypatch: Any) -> None: + names = ["store"] * 3 + ["store-old"] * 150 + ["store"] + fake = FakeDatabases(names, size=100) + monkeypatch.setattr(hotdata, "DatabasesApi", lambda api: fake) + found = hotdata._named(FakeClient(), "store") # type: ignore[arg-type] + assert [db.id for db in found] == ["db-0", "db-1", "db-2", "db-153"] + assert fake.calls == [None, "100"] From 9e967533eb34af358fe16bcb908edfbb411750f7 Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 21 Oct 2026 13:35:02 +0530 Subject: [PATCH 04/14] Phase 11: SQL INSERT measured in the cloud --- docs/guarantees.md | 34 +++++++- docs/internal/architecture.md | 8 +- docs/internal/brief.md | 9 +- scripts/measure_local.py | 151 +++++++++++++++++++++++++++++----- 4 files changed, 172 insertions(+), 30 deletions(-) diff --git a/docs/guarantees.md b/docs/guarantees.md index 7a607aa..a43bd6c 100644 --- a/docs/guarantees.md +++ b/docs/guarantees.md @@ -132,9 +132,35 @@ alone. ### M4. What does a SQL INSERT cost against a file load? -With `RUNTIMEDB_ENGINE__SQL_WRITES=true`, time one hundred single-row INSERT statements and -one hundred single-row upload-and-load calls into the same table shape. Record the cost per -call of each. +Time one hundred single-row INSERT statements and one hundred single-row upload-and-load +calls into the same table shape. Record the cost per call of each. From 2026-10-21, also +time five flushes of 1, 10, 100, and 1,000 rows with 1,536-dimension vectors, by INSERT +and by upload and load. Run it with `scripts/measure_local.py --only m4`. + +Cloud result, 2026-10-21, `api.hotdata.dev`: the engine accepted INSERT. The engine ran +SQL writes without a setting from 2026-10-18. Two runs gave the same picture, and the +table gives the second. Each value is the median. + +| Write | By INSERT | By upload and load | +|---|---|---| +| One row, no vector, 100 calls | 494 ms | 1,693 ms | +| One row with a vector, 5 calls | 531 ms | 1,682 ms | +| 10 rows with vectors, 5 calls | 1,744 ms | 1,811 ms | +| 100 rows with vectors, 5 calls | 13,338 ms | 2,526 ms | +| 1,000 rows with vectors, 5 calls | refused, `Payload Too Large` | 8,323 ms | + +The vectors go into the SQL text as array literals. So a flush of 100 rows is 3.2 MB of +SQL, against 1.0 MB of parquet, and a flush of 1,000 rows is 32 MB, over the size limit of +a query request. An INSERT is about three times faster than a load for one row, the same +for 10 rows, and slower above that. Every table held the expected row count after the +calls. + +Local result, 2026-10-21, the local stack with an engine image of 2026-10-08: one row by +INSERT took a median of 22 ms, against 30 ms by load. A flush of 100 rows with vectors +took 874 ms by INSERT and 93 ms by load. A flush of 1,000 rows by INSERT was refused with +`Payload Too Large`, and took 248 ms by load. + +The results of 2026-10-05 follow. Cloud result, 2026-10-05, `api.hotdata.dev`: the engine refused the first INSERT with `Bad Request`. Production does not turn on SQL writes, so this is the expected result. One @@ -148,7 +174,7 @@ of 29 ms, and a mean of 24 ms. Both tables held 101 rows after the calls. Locally, a load costs about 24 ms, against 2,136 ms in the cloud. So almost all of the cloud cost is outside the engine. An INSERT is about 30 percent cheaper than a load -locally, but production does not accept it. +locally, but production did not accept it on 2026-10-05. ### M5. At what size does an index beat a scan? diff --git a/docs/internal/architecture.md b/docs/internal/architecture.md index 26b5a78..52acdf9 100644 --- a/docs/internal/architecture.md +++ b/docs/internal/architecture.md @@ -309,8 +309,9 @@ fails, the record has no cue row. If the record load fails, a cue row is left wi record, and the search drops it. Each load uploads its file once, and then calls the load API. Only the load call is -retried on 409. A load costs about 2.1 seconds in the cloud, and the cost does not change -with row count. [measured] So hotmemory buffers puts in a writer, and each flush pays for +retried on 409. A load of rows with no vector costs about 2.1 seconds in the cloud, and +the cost does not change with row count. [measured] With 1,536-dimension vectors, a load +of 100 rows took 2.5 s, and 1,000 rows 8.3 s. [measured, M4, 2026-10-21] So hotmemory buffers puts in a writer, and each flush pays for one load for each table, not one load for each record. If a load takes more than 10 seconds, the framework asks the API to run it as a job, and @@ -319,7 +320,8 @@ object storage and records them in the catalog. The order of those two steps was observed. hotmemory never sends `INSERT`, `UPDATE`, or `DELETE` in SQL. Every change to a row is a -keyed load. The cloud refuses SQL writes. [measured] +keyed load. The cloud accepts `INSERT` and refuses `UPDATE` and `DELETE`. One row by +`INSERT` took 531 ms in the cloud, against 1,682 ms by load. [measured, M4, 2026-10-21] ## Remember and capture diff --git a/docs/internal/brief.md b/docs/internal/brief.md index 8eb0b6c..f2d46fd 100644 --- a/docs/internal/brief.md +++ b/docs/internal/brief.md @@ -65,9 +65,10 @@ These come from measurements against Hotdata between 2026-08-27 and 2026-09-18. - A write is a file load in every released engine and in production. The caller uploads parquet and calls a keyed load in replace, append, upsert, update, or delete mode. - [measured] On RuntimeDB main since 2026-09-25, a SQL write path exists behind the - `[engine] sql_writes` flag, off by default, and production does not set it. INSERT and - CREATE TABLE, CREATE TABLE AS, and DROP TABLE are implemented. An INSERT is an append + [measured] On RuntimeDB main since 2026-09-25, a SQL write path exists. It ran behind + the `[engine] sql_writes` flag until 2026-10-18, and runs for every instance after. + The cloud accepted INSERT on 2026-10-21. [measured, M4] INSERT and CREATE TABLE, CREATE + TABLE AS, and DROP TABLE are implemented. UPDATE and DELETE are refused. An INSERT is an append load whose source is a query: the planned rows are staged as parquet and appended through the same locked load path, so it holds the same per-table lock and maintains indexes the same way. A SQL write needs the read and write claims and can target only the database's @@ -320,7 +321,7 @@ because the bare container refuses managed tables. All six were run on 2026-10-0 index still returns it. - M3. Run two processes that load the same table at once with the serialized writer in each. Record how many 409s occur and whether the retry bound of the driver is enough. -- M4. With `RUNTIMEDB_ENGINE__SQL_WRITES=true`, time one hundred single-row INSERT +- M4. With SQL writes on, time one hundred single-row INSERT statements and one hundred single-row upload-plus-load calls into the same table shape. Record both per-call costs. - M5. Load one thousand, ten thousand, and one hundred thousand rows into the `memory` diff --git a/scripts/measure_local.py b/scripts/measure_local.py index 6ab174e..e030cec 100644 --- a/scripts/measure_local.py +++ b/scripts/measure_local.py @@ -8,8 +8,9 @@ """Run measurements M6, M4, and M5 against a local RuntimeDB container or the cloud. By default the script targets a running local container. It does not start the -container. Start it with `make local-up`, which sets RUNTIMEDB_ENGINE__SQL_WRITES=true -for M4. With --cloud, it targets the Hotdata cloud API instead. +container. Start it with `make local-up`. With --cloud, it targets the Hotdata cloud API +instead. With --only and a comma-separated list, such as --only m4, it runs only those +measurements. Reads the connection from the environment: @@ -56,6 +57,9 @@ SCHEMA = "public" M4_CALLS = 100 +M4_FLUSHES = (1, 10, 100, 1000) +M4_FLUSH_RUNS = 5 +M4_DIMENSIONS = 1536 M5_DIMENSIONS = 64 M5_RUNS = 5 M5_DEPTH = 100 @@ -305,6 +309,77 @@ def measure_insert(client: HotdataClient, work: Path, prefix: str) -> dict[str, out[f"{table}_rows"] = int(result.rows[0][0]) finally: client.delete_managed_database(db) + out["flushes"] = measure_flushes(client, work, prefix) + return out + + +def flush_rows(size: int, start: int, rng: random.Random) -> pa.Table: + return pa.table( + { + "id": [f"row-{start + n}" for n in range(size)], + "content": [f"row {start + n} {rng.choice(WORDS)}" for n in range(size)], + "embedding": [[rng.uniform(-1, 1) for _ in range(M4_DIMENSIONS)] for _ in range(size)], + }, + schema=pa.schema( + [ + ("id", pa.string()), + ("content", pa.string()), + ("embedding", pa.list_(pa.float32())), + ] + ), + ) + + +def insert_sql(table: str, rows: pa.Table) -> str: + values = ", ".join( + f"('{row['id']}', '{row['content']}', CAST({vector_literal(row['embedding'])} AS REAL[]))" + for row in rows.to_pylist() + ) + return f"INSERT INTO {table_ref(table)} (id, content, embedding) VALUES {values}" + + +def measure_flushes(client: HotdataClient, work: Path, prefix: str) -> dict[int, dict[str, Any]]: + """Time flushes of M4_FLUSHES rows with vectors, by INSERT and by upload and load.""" + rng = random.Random(4) + out: dict[int, dict[str, Any]] = {} + db = client.create_managed_database(f"{prefix}-m4f", tables=["by_load", "by_sql"]) + try: + seed = flush_rows(1, -1, rng) + for table in ("by_load", "by_sql"): + client.load_managed_table( + db, table, file=write_parquet(work, f"m4f_{table}", seed), mode="replace" + ) + start = 0 + for size in M4_FLUSHES: + entry: dict[str, Any] = {"sql": [], "load": []} + for _run in range(M4_FLUSH_RUNS): + rows = flush_rows(size, start, rng) + start += size + sql = insert_sql("by_sql", rows) + entry["sql_bytes"] = len(sql.encode()) + try: + entry["sql"].append(timed(lambda sql=sql: client.execute_sql(sql, database=db))) + except RuntimeError as e: + entry["sql_error"] = short(e) + break + for run in range(M4_FLUSH_RUNS): + rows = flush_rows(size, start, rng) + start += size + path = write_parquet(work, f"m4f_load_{size}_{run}", rows) + entry["parquet_bytes"] = Path(path).stat().st_size + entry["load"].append( + timed( + lambda path=path: client.load_managed_table( + db, "by_load", file=path, mode="append" + ) + ) + ) + out[size] = entry + for table in ("by_load", "by_sql"): + result = client.execute_sql(f"SELECT count(*) FROM {table_ref(table)}", database=db) + out.setdefault(0, {})[f"{table}_rows"] = int(result.rows[0][0]) + finally: + client.delete_managed_database(db) return out @@ -472,11 +547,14 @@ def summary(times: list[float]) -> str: def main() -> int: provider = os.environ.get("HOTMEMORY_EMBEDDING_PROVIDER", "sys_emb_openai") sizes = [int(s) for s in os.environ.get("HOTMEMORY_M5_SIZES", "1000,10000,100000").split(",")] - cloud = "--cloud" in sys.argv[1:] + args = sys.argv[1:] + cloud = "--cloud" in args + only = set(args[args.index("--only") + 1].split(",")) if "--only" in args else set() + wanted = only or {"m4", "m5", "m6"} if cloud: prefix = env("HOTMEMORY_MEASURE_DB") client = connect_cloud() - names = {f"{prefix}-{m}" for m in ("m4", "m5", "m6")} + names = {f"{prefix}-{m}" for m in ("m4", "m4f", "m5", "m6")} taken = sorted(names & {d.description for d in client.list_managed_databases()}) if taken: sys.exit( @@ -488,28 +566,41 @@ def main() -> int: started_at = now() with tempfile.TemporaryDirectory() as tmp: work = Path(tmp) - bare = measure_bare(client, work, provider, prefix) + bare = measure_bare(client, work, provider, prefix) if "m6" in wanted else [] print("m6 done", file=sys.stderr) - try: - insert = measure_insert(client, work, prefix) - except RuntimeError as e: - insert = {"setup_error": short(e)} + insert: dict[str, Any] = {} + if "m4" in wanted: + try: + insert = measure_insert(client, work, prefix) + except RuntimeError as e: + insert = {"setup_error": short(e)} print("m4 done", file=sys.stderr) - try: - scale = measure_scale(client, work, sizes, provider, prefix) - except RuntimeError as e: - scale = {"sizes": {}, "coexistence": [], "setup_error": short(e)} + scale: dict[str, Any] = {"sizes": {}, "coexistence": []} + if "m5" in wanted: + try: + scale = measure_scale(client, work, sizes, provider, prefix) + except RuntimeError as e: + scale = {"sizes": {}, "coexistence": [], "setup_error": short(e)} print("m5 done", file=sys.stderr) print(f"## {'Cloud' if cloud else 'Local'} measurements of M4 to M6, {started_at}") print() print(f"Host: {client.host}.") print() - print("### M6") - print() - for label, outcome in bare: - print(f"- {label}: {outcome}") - print() + if "m6" in wanted: + print("### M6") + print() + for label, outcome in bare: + print(f"- {label}: {outcome}") + print() + if "m4" in wanted: + print_insert(insert) + if "m5" in wanted: + print_scale(scale) + return 0 + + +def print_insert(insert: dict[str, Any]) -> None: print("### M4") print() if "setup_error" in insert: @@ -522,7 +613,30 @@ def main() -> int: f"- Rows after the calls: by_sql {insert.get('by_sql_rows')}, " f"by_load {insert.get('by_load_rows')} (each includes one seed row)." ) + flushes = insert.get("flushes", {}) + for size, entry in flushes.items(): + if size == 0: + continue + print( + f"- Flush of {size} rows with {M4_DIMENSIONS}-dimension vectors, by INSERT " + f"({entry.get('sql_bytes', 0) / 1e6:.1f} MB of SQL): {summary(entry['sql'])}." + ) + if "sql_error" in entry: + print(f" - error: {entry['sql_error']}") + print( + f"- Flush of {size} rows by upload and append load " + f"({entry.get('parquet_bytes', 0) / 1e6:.1f} MB of parquet): " + f"{summary(entry['load'])}." + ) + if 0 in flushes: + print( + f"- Rows after the flushes: by_sql {flushes[0].get('by_sql_rows')}, " + f"by_load {flushes[0].get('by_load_rows')} (each includes one seed row)." + ) print() + + +def print_scale(scale: dict[str, Any]) -> None: print("### M5") print() if "setup_error" in scale: @@ -545,7 +659,6 @@ def main() -> int: print() for label, outcome in scale["coexistence"]: print(f"- {label}: {outcome}") - return 0 if __name__ == "__main__": From 8a86ca7b377c253a10743e110bb0a93ef3319f80 Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 21 Oct 2026 13:35:02 +0530 Subject: [PATCH 05/14] Phase 11: read current revisions with IN lists in batches of 500 keys --- src/hotmemory/hotdata.py | 37 +++++++++++++++++++++++++------------ tests/test_hotdata.py | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 12 deletions(-) diff --git a/src/hotmemory/hotdata.py b/src/hotmemory/hotdata.py index cd316a7..1c29305 100644 --- a/src/hotmemory/hotdata.py +++ b/src/hotmemory/hotdata.py @@ -64,6 +64,7 @@ DEPTH_PER_RESULT = 10 LOAD_ATTEMPTS = 8 LIST_PAGE = 100 +SLOT_BATCH = 500 FIRST_BACKOFF = 0.25 MAX_BACKOFF = 4.0 @@ -455,21 +456,33 @@ def _write_many(self, drafts: Sequence[Draft]) -> builtins.list[str]: def _current( self, slots: set[tuple[tuple[str, ...], str]] ) -> tuple[dict[tuple[tuple[str, ...], str], Record], dict[str, builtins.list[float]]]: - """Return the current revision of each slot that has one, and its content vector.""" + """Return the current revision of each slot that has one, and its content vector. + + Each query matches a list of namespaces and a list of at most SLOT_BATCH keys, and + the rows of other slots that it also matches are dropped. + """ current: dict[tuple[tuple[str, ...], str], Record] = {} vectors: dict[str, builtins.list[float]] = {} - if not slots: - return current, vectors - where = " OR ".join(f"({_slot_condition(*slot)})" for slot in sorted(slots)) + ordered = sorted(slots) columns = ", ".join(RECORD_SCHEMA.names) - for table in RECORD_TABLES: - sql = f"SELECT {columns} FROM {_ref(table)} WHERE superseded_by IS NULL AND ({where})" - for row in self._sql(sql).to_records(): - record = _record(row) - slot = (record.namespace, record.key) - if slot not in current or current[slot].revision < record.revision: - current[slot] = record - vectors[record.id] = row["content_embedding"] + for start in range(0, len(ordered), SLOT_BATCH): + part = ordered[start : start + SLOT_BATCH] + paths = sorted({"/".join(namespace) for namespace, _ in part}) + keys = sorted({key for _, key in part}) + where = ( + f"superseded_by IS NULL AND namespace IN ({', '.join(map(_text, paths))}) " + f"AND key IN ({', '.join(map(_text, keys))})" + ) + for table in RECORD_TABLES: + sql = f"SELECT {columns} FROM {_ref(table)} WHERE {where}" + for row in self._sql(sql).to_records(): + record = _record(row) + slot = (record.namespace, record.key) + if slot not in slots: + continue + if slot not in current or current[slot].revision < record.revision: + current[slot] = record + vectors[record.id] = row["content_embedding"] return current, vectors def _load_records( diff --git a/tests/test_hotdata.py b/tests/test_hotdata.py index af9595a..48a2524 100644 --- a/tests/test_hotdata.py +++ b/tests/test_hotdata.py @@ -18,9 +18,13 @@ MEMORY_TABLE, META_TABLE, RECORD_SCHEMA, + SLOT_BATCH, HotdataStore, LayoutError, + _bm25_text, _record, + _ref, + _text, ) from .conftest import DIMENSIONS, FakeClock, fake_embedder @@ -332,6 +336,39 @@ def test_search_in_a_narrow_scope_returns_its_top_k( assert sorted(hit.record.key for hit in hits) == sorted(f"narrow-{n}" for n in range(10)) +def test_bm25_applies_the_scope_filter_before_its_depth(client: HotdataClient, name: str) -> None: + store = provision(client, name) + narrow = ("team", "narrow") + with store.writer() as writer: + for n in range(300): + writer.put(NS, f"wide-{n}", kind="fact", content=f"disk disk disk report {n}") + for n in range(10): + writer.put(narrow, f"narrow-{n}", kind="fact", content=f"disk usage {n}") + + where = store._listed([narrow], None) + sql = ( + f"SELECT key FROM bm25_search('{_ref(MEMORY_TABLE)}', 'content', " + f"{_text(_bm25_text('disk'))}, 100) WHERE {where}" + ) + keys = sorted(row[0] for row in client.execute_sql(sql, database=store.database).rows) + assert keys == sorted(f"narrow-{n}" for n in range(10)) + + +def test_a_flush_larger_than_a_slot_batch_reads_every_current_revision( + client: HotdataClient, name: str +) -> None: + store = provision(client, name) + size = 2 * SLOT_BATCH + 100 + for content in ("first", "second"): + with store.writer(max_rows=size + 1) as writer: + for n in range(size): + writer.put(NS if n % 2 else (*NS, "other"), f"k-{n}", kind="fact", content=content) + for n in (0, 1, size - 1): + namespace = NS if n % 2 else (*NS, "other") + assert [r.content for r in store.history(namespace, f"k-{n}")] == ["first", "second"] + assert len(store.list(NS, limit=size + 1)) == size + + def test_two_stores_write_one_database_at_once(client: HotdataClient, name: str) -> None: first = provision(client, name) second = HotdataStore.open( From d3300cc2782e29249c94ffe2d240379dfd3c4527 Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 21 Oct 2026 13:35:02 +0530 Subject: [PATCH 06/14] Phase 11: the scale harness --- bench/scale.py | 346 ++++++++++++++++++++++++++++++++++++++ bench/tests/test_scale.py | 55 ++++++ pyproject.toml | 2 +- uv.lock | 6 + 4 files changed, 408 insertions(+), 1 deletion(-) create mode 100644 bench/scale.py create mode 100644 bench/tests/test_scale.py diff --git a/bench/scale.py b/bench/scale.py new file mode 100644 index 0000000..601b8d3 --- /dev/null +++ b/bench/scale.py @@ -0,0 +1,346 @@ +"""The scale test: one store filled with real chunks, timed and checked at each size. + +Run it from the root of the repository, with the local stack up: + + uv run --group bench --env-file .env python -m bench.scale --sizes 10000 + +The store holds the sessions of LongMemEval-S as chunks of 1,000 characters, one +session after another, with no copy of a row. The evidence sessions of samples A and B +come first, so every size holds them. Each session goes into one of 100 buckets, under +the namespace `scale//`, so a scope of one bucket holds about 1 percent of +the rows and a scope of one tens label about 10 percent. + +At each size in `--sizes`, the run searches each scored question of the two samples in +each scope, by each ranking, and records the time of each search. For the vector +ranking, it compares the top k with a brute-force ranking in numpy over the same rows. +Over the whole store, it scores `recall_any@k` against the evidence sessions. Each flush +records its time and the time of the read of the current revisions before it. It writes +one JSON file into `bench/results/`. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import statistics +import time +import uuid +from collections.abc import Iterator, Sequence +from dataclasses import dataclass +from datetime import UTC, datetime, timedelta +from pathlib import Path +from typing import Any + +import numpy as np + +from bench import embed, prompts +from bench.data import DATA_DIR, DATA_FILE, Question, Session, load +from bench.e2e import episode_key, sample_ids +from bench.episodes import ACTOR, session_of +from bench.metrics import score +from bench.retrieval import RESULTS_DIR, _commit +from hotmemory import Embedder, Filter, Store +from hotmemory._chunks import chunk_text + +CHUNK_CHARS = 1000 +BUCKETS = 100 +FLUSH = 1000 +K = 10 +KS = (5, 10) +ROOT = ("scale",) +TIE = 1e-5 +EPISODES = Filter(kind="episode") + + +@dataclass(frozen=True) +class Row: + """One chunk of a session, with its key, namespace, and session id.""" + + namespace: tuple[str, ...] + key: str + content: str + session: str + + +def bucket(session_id: str) -> int: + """Return the bucket of a session, from 0 to BUCKETS - 1, from the SHA-256 of its id.""" + return int(hashlib.sha256(session_id.encode()).hexdigest()[:8], 16) % BUCKETS + + +def namespace_of(number: int) -> tuple[str, ...]: + """Return the namespace of bucket `number`: `scale//`.""" + return (*ROOT, str(number // 10), f"{number:02d}") + + +def scopes() -> dict[str, tuple[str, ...]]: + """Return the three scopes: one bucket, one tens label, and the whole store.""" + return {"1%": namespace_of(0), "10%": (*ROOT, "0"), "100%": ROOT} + + +def pool(questions: Sequence[Question], first: Sequence[Question]) -> list[Session]: + """Return each session of `questions` once, in three groups, each sorted by id. + + First the evidence sessions of `first`, then the other sessions of `first`, then the + rest. A session with no turns is left out. + """ + seen: dict[str, Session] = {} + for question in (*first, *questions): + for session in question.sessions: + if session.turns and session.id not in seen: + seen[session.id] = session + evidence = sorted({i for q in first for i in q.evidence if i in seen}) + near = sorted({s.id for q in first for s in q.sessions if s.id in seen} - set(evidence)) + rest = sorted(set(seen) - set(evidence) - set(near)) + return [seen[i] for i in (*evidence, *near, *rest)] + + +def rows_of(sessions: Sequence[Session]) -> Iterator[Row]: + """Yield the chunks of each session in order, as `capture` stores them in phase 10.""" + for session in sessions: + namespace = namespace_of(bucket(session.id)) + key = episode_key(session.id) + for number, chunk in enumerate(chunk_text(prompts.session_text(session), CHUNK_CHARS), 1): + yield Row(namespace, f"{key}-{number:04d}", chunk, session.id) + + +def spread(values: Sequence[float]) -> dict[str, float]: + """Return the count, p50, p95, and highest of `values`, rounded to 0.001.""" + if not values: + return {"count": 0} + ordered = sorted(values) + p95 = ordered[min(len(ordered) - 1, int(len(ordered) * 0.95))] + return { + "count": len(values), + "p50": round(statistics.median(ordered), 3), + "p95": round(p95, 3), + "highest": round(ordered[-1], 3), + } + + +def in_scope(row: Row, prefix: tuple[str, ...]) -> bool: + return row.namespace[: len(prefix)] == prefix + + +class Exact: + """The brute-force cosine ranking over every row written so far, in numpy.""" + + def __init__(self, capacity: int) -> None: + self.capacity = capacity + self.vectors: Any = None + self.rows: list[Row] = [] + + def add(self, rows: Sequence[Row], vectors: Sequence[Sequence[float]]) -> None: + start = len(self.rows) + block = np.asarray(vectors, dtype=np.float32) + if self.vectors is None: + self.vectors = np.zeros((self.capacity, block.shape[1]), dtype=np.float32) + norms = np.linalg.norm(block, axis=1, keepdims=True) + self.vectors[start : start + len(rows)] = block / np.where(norms == 0, 1, norms) + self.rows.extend(rows) + + def scope(self, prefix: tuple[str, ...]) -> Any: + """Return the positions of the rows under `prefix`.""" + mask = np.fromiter((in_scope(row, prefix) for row in self.rows), bool, len(self.rows)) + return np.flatnonzero(mask) + + def top(self, query: Sequence[float], index: Any, k: int) -> list[tuple[str, float]]: + """Return the keys and distances of the k nearest rows among the positions `index`.""" + q = np.asarray(query, dtype=np.float32) + q = q / (np.linalg.norm(q) or 1.0) + distances = 1.0 - self.vectors[index] @ q + order = np.argsort(distances, kind="stable")[:k] + return [(self.rows[index[i]].key, float(distances[i])) for i in order] + + +def same_top(found: Sequence[tuple[str, float]], exact: Sequence[tuple[str, float]]) -> str: + """Return `equal`, `ties` if the keys differ only among equal distances, or `differ`.""" + if [key for key, _ in found] == [key for key, _ in exact]: + return "equal" + if len(found) == len(exact) and all( + abs(a - b) <= TIE for (_, a), (_, b) in zip(found, exact, strict=True) + ): + return "ties" + return "differ" + + +class Run: + """The fill and the checkpoints of one scale run over `stores`, one per ranking.""" + + def __init__( + self, + stores: dict[str, Store], + embedder: Embedder, + questions: Sequence[Question], + capacity: int, + ) -> None: + self.stores = stores + self.first = next(iter(stores.values())) + self.embedder = embedder + self.questions = questions + self.exact = Exact(capacity) + self.flushes: list[dict[str, float]] = [] + self.reads: list[float] = [] + read = getattr(self.first, "_current", None) + if read is not None: + + def timed_read(slots: Any) -> Any: + started = time.perf_counter() + try: + return read(slots) + finally: + self.reads.append(time.perf_counter() - started) + + self.first._current = timed_read # type: ignore[attr-defined] + + def fill(self, rows: Sequence[Row]) -> None: + """Write `rows` in flushes of FLUSH rows, and time each flush.""" + for start in range(0, len(rows), FLUSH): + part = rows[start : start + FLUSH] + vectors = self.embedder([row.content for row in part]) + started = time.perf_counter() + with self.first.writer(max_rows=len(part) + 1, interval=timedelta.max) as writer: + for row in part: + writer.put( + row.namespace, + row.key, + kind="episode", + content=row.content, + sources=(f"session:{row.session}",), + actor=ACTOR, + ) + seconds = time.perf_counter() - started + read = self.reads[-1] if self.reads else 0.0 + self.exact.add(part, vectors) + total = len(self.exact.rows) + self.flushes.append({"rows": len(part), "total": total, "s": seconds, "read_s": read}) + print(f"wrote {total} rows, flush {seconds:.2f} s, read {read:.2f} s", flush=True) + + def checkpoint(self) -> dict[str, Any]: + """Search each question in each scope by each ranking, and return the results.""" + result: dict[str, Any] = {"rows": len(self.exact.rows), "scopes": {}} + for name, prefix in scopes().items(): + index = self.exact.scope(prefix) + entry: dict[str, Any] = {"rows": len(index)} + for ranking, store in self.stores.items(): + times: list[float] = [] + checks = {"equal": 0, "ties": 0, "differ": 0} + scored: list[dict[str, float]] = [] + for question in self.questions: + started = time.perf_counter() + hits = store.search(question.question, [prefix], EPISODES, k=K) + times.append((time.perf_counter() - started) * 1000) + found = [(hit.record.key, hit.distance or 0.0) for hit in hits] + if ranking == "vector": + query = self.embedder([question.question])[0] + checks[same_top(found, self.exact.top(query, index, K))] += 1 + if name == "100%": + sessions = list( + dict.fromkeys(session_of(hit.record.sources) for hit in hits) + ) + scored.append(score(sessions, question.evidence, KS)) + entry[ranking] = {"search_ms": spread(times)} + if ranking == "vector": + entry[ranking]["exact"] = checks + if scored: + entry[ranking]["metrics"] = { + f"recall_any@{k}": round( + sum(s[f"recall_any@{k}"] for s in scored) / len(scored), 4 + ) + for k in KS + } + result["scopes"][name] = entry + return result + + +def hotdata_stores(embedder: Embedder, location: str, url: str, name: str) -> Any: + """Return the client, the database, and one store for each ranking.""" + from hotdata_framework import HotdataClient + + from hotmemory.hotdata import HotdataStore + + if location == "local": + client = HotdataClient("local", "local", host=url) + else: + client = HotdataClient.from_env() + store = HotdataStore.provision( + name, embedder=embedder, model=embed.MODEL, dimensions=embed.DIMENSIONS, client=client + ) + options: dict[str, Any] = { + "embedder": embedder, + "model": embed.MODEL, + "dimensions": embed.DIMENSIONS, + } + stores: dict[str, Store] = { + ranking: HotdataStore(client, store.database, ranking=ranking, **options) + for ranking in ("vector", "fused") + } + return client, store.database, stores + + +def main(argv: Sequence[str] | None = None) -> Path: + parser = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + parser.add_argument("--sizes", default="10000", help="comma-separated row counts") + parser.add_argument("--location", choices=("local", "cloud"), default="local") + parser.add_argument("--url", default="http://localhost:3000") + parser.add_argument("--keep", action="store_true", help="keep the database after the run") + parser.add_argument("--tag", default="") + args = parser.parse_args(argv) + sizes = sorted(int(size) for size in args.sizes.split(",")) + + questions = load(DATA_DIR / DATA_FILE) + by_id = {question.id: question for question in questions} + chosen = [by_id[i] for name in ("a", "b") for i in sample_ids(name, questions)] + scored = [question for question in chosen if question.is_scored] + rows = list(rows_of(pool(questions, chosen))) + if sizes[-1] > len(rows): + raise SystemExit(f"the pool holds {len(rows)} rows, fewer than {sizes[-1]}") + + embedder = embed.openai_embedder() + run_id = uuid.uuid4().hex[:8] + name = f"hotmemory-scale-{run_id}" + client, database, stores = hotdata_stores(embedder, args.location, args.url, name) + run = Run(stores, embedder, scored, sizes[-1]) + started = datetime.now(UTC) + checkpoints = [] + try: + written = 0 + for size in sizes: + run.fill(rows[written:size]) + written = size + checkpoints.append(run.checkpoint()) + print(json.dumps(checkpoints[-1]), flush=True) + finally: + if not args.keep: + client.delete_managed_database(database) + result = { + "config": { + "started": started.isoformat(), + "location": args.location, + "sizes": sizes, + "pool_rows": len(rows), + "questions": len(scored), + "chunk_chars": CHUNK_CHARS, + "flush": FLUSH, + "k": K, + "buckets": BUCKETS, + "commit": _commit(), + "database": name if args.keep else None, + }, + "flushes": run.flushes, + "flush_s_by_size": { + str(size): spread([f["s"] for f in run.flushes if prev < f["total"] <= size]) + for prev, size in zip([0, *sizes], sizes, strict=False) + }, + "checkpoints": checkpoints, + } + tag = f"-{args.tag}" if args.tag else "" + path = RESULTS_DIR / f"scale-{args.location}{tag}-{run_id}.json" + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(result, indent=1) + "\n") + print(path) + return path + + +if __name__ == "__main__": + main() diff --git a/bench/tests/test_scale.py b/bench/tests/test_scale.py new file mode 100644 index 0000000..00aaca8 --- /dev/null +++ b/bench/tests/test_scale.py @@ -0,0 +1,55 @@ +from pathlib import Path + +from bench.data import load +from bench.scale import BUCKETS, Run, bucket, namespace_of, pool, rows_of, same_top, scopes +from hotmemory import MemoryStore +from tests.conftest import fake_embedder + +FIXTURE = Path(__file__).with_name("fixture.json") + + +def test_pool_puts_the_first_questions_first_and_keeps_each_session_once() -> None: + questions = load(FIXTURE, expected_sha256=None) + first = questions[-1:] + sessions = pool(questions, first) + ids = [session.id for session in sessions] + assert len(ids) == len(set(ids)) + evidence = first[0].evidence + assert set(ids[: len(evidence)]) == evidence + near = {session.id for session in first[0].sessions if session.turns} - evidence + assert set(ids[len(evidence) : len(evidence) + len(near)]) == near + + +def test_rows_share_the_bucket_of_their_session() -> None: + questions = load(FIXTURE, expected_sha256=None) + for row in rows_of(pool(questions, [])): + assert row.namespace == namespace_of(bucket(row.session)) + assert 0 <= bucket(row.session) < BUCKETS + + +def test_scopes_nest() -> None: + found = scopes() + assert found["1%"][: len(found["10%"])] == found["10%"] + assert found["10%"][: len(found["100%"])] == found["100%"] + + +def test_same_top_tells_ties_from_differences() -> None: + assert same_top([("a", 0.1), ("b", 0.2)], [("a", 0.1), ("b", 0.2)]) == "equal" + assert same_top([("b", 0.1), ("a", 0.1)], [("a", 0.1), ("b", 0.1)]) == "ties" + assert same_top([("c", 0.3)], [("a", 0.1)]) == "differ" + + +def test_run_on_memory_store_matches_the_exact_ranking() -> None: + questions = load(FIXTURE, expected_sha256=None) + scored = [question for question in questions if question.is_scored] + rows = list(rows_of(pool(questions, scored))) + store = MemoryStore(embedder=fake_embedder) + run = Run({"vector": store}, fake_embedder, scored, len(rows)) + run.fill(rows) + result = run.checkpoint() + assert result["rows"] == len(rows) + whole = result["scopes"]["100%"] + assert whole["rows"] == len(rows) + assert whole["vector"]["exact"]["differ"] == 0 + assert set(whole["vector"]["metrics"]) == {"recall_any@5", "recall_any@10"} + assert len(run.flushes) == 1 diff --git a/pyproject.toml b/pyproject.toml index 952fb88..1e51bbb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,7 +27,7 @@ dev = [ "ruff", { include-group = "bench" }, ] -bench = ["hotmemory[hotdata,openai]", "tiktoken"] +bench = ["hotmemory[hotdata,openai]", "numpy", "tiktoken"] [tool.ruff] line-length = 100 diff --git a/uv.lock b/uv.lock index 6035e19..763b34f 100644 --- a/uv.lock +++ b/uv.lock @@ -322,11 +322,15 @@ openai = [ [package.dev-dependencies] bench = [ { name = "hotmemory", extra = ["hotdata", "openai"] }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, { name = "tiktoken" }, ] dev = [ { name = "hotmemory", extra = ["hotdata", "openai"] }, { name = "mypy" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, { name = "pytest" }, { name = "pytest-socket" }, { name = "ruff" }, @@ -344,11 +348,13 @@ provides-extras = ["hotdata", "openai"] [package.metadata.requires-dev] bench = [ { name = "hotmemory", extras = ["hotdata", "openai"] }, + { name = "numpy" }, { name = "tiktoken" }, ] dev = [ { name = "hotmemory", extras = ["hotdata", "openai"] }, { name = "mypy" }, + { name = "numpy" }, { name = "pytest" }, { name = "pytest-socket" }, { name = "ruff" }, From 7cf383635a1c11222aadea84fb042dff6a3bb439 Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 21 Oct 2026 13:36:07 +0530 Subject: [PATCH 07/14] Phase 11: BM25 applies the scope filter inside the search --- bench/README.md | 10 +++++----- docs/benchmarks.md | 8 +++++--- docs/contracts.md | 4 ++-- docs/guarantees.md | 1 + docs/internal/architecture.md | 8 +++++--- docs/internal/plan.md | 3 --- docs/internal/roadmap.md | 3 --- src/hotmemory/hotdata.py | 8 ++++---- 8 files changed, 22 insertions(+), 23 deletions(-) diff --git a/bench/README.md b/bench/README.md index 037b863..0f351fb 100644 --- a/bench/README.md +++ b/bench/README.md @@ -103,8 +103,8 @@ The options are these: Each worker gets the questions in turn. Without `--keep` or `--reuse`, it runs them over a new local database of its own, and empties the namespaces of each question after it. Each table then holds one question at a time, as in a run with one worker. With -`--memory-ranking fused`, BM25 ranks over the whole table before the scope filter, so the -rows of a second question in a table can change the ranking. An extractor variant names a prompt and a model. Each fact carries +`--memory-ranking fused`, the BM25 scores can depend on every row of the table, so the +rows of a second question in a table can change the ranking. This was not measured. An extractor variant names a prompt and a model. Each fact carries the source `session:`, where the hash is the first 12 hex digits of the SHA-256 of the session id. The block shows the sources to the reader, and the id of an evidence session holds `answer`, so the source does not show the id. If an extractor reply reaches @@ -136,9 +136,9 @@ uv run --group bench --env-file .env python -m bench.e2e --run --arms memory --e ``` With `--keep`, each question runs over a new local database of its own, and the database -stays after the run. One database holds one question. With `--memory-ranking fused`, BM25 -ranks the whole table before the namespace filter, so the memory of a second question -would change the block. The +stays after the run. One database holds one question. With `--memory-ranking fused`, the +BM25 scores can depend on every row of the table, so the memory of a second question can +change the block. This was not measured. The name of the database is `hotmemory-kept------`: | Part | What it holds | diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 2726d3e..f621fce 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -73,9 +73,11 @@ number on S compares with this table. ### Every question in one table A second run kept the episodes of all 419 questions in the same tables, 19,864 rows, and -searched each question inside its own namespace. BM25 in the fused ranking takes its top -rows from the whole table before the namespace filter applies, so this run tests that -limit. On the local stack, the fused ranking scored higher at the top 5 and slightly lower +searched each question inside its own namespace. This run was planned to test whether BM25 +loses rows of the namespace, because it was thought to rank the whole table before the +filter. On 2026-10-21, a check showed that the engine applies the filter inside +`bm25_search`, so BM25 loses no rows of the namespace. The cause of the small differences +below was not measured. On the local stack, the fused ranking scored higher at the top 5 and slightly lower at the top 10. At the top 10, 1 more question had no evidence session. The vector ranking gave the same sessions in the same order in both runs. File: `retrieval-hotdata-local-shared-fused-vector.json`. diff --git a/docs/contracts.md b/docs/contracts.md index 823793b..e373118 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -280,8 +280,8 @@ With `ranking="fused"`, the query has three parts. `forget_after`, and the `Filter`. The two vector rankings apply the filters first and then rank every row that is left by a scan. Thus a narrow scope keeps its recall. They do not use the vector index, because a filtered search through the index misses rows - loaded after its build. BM25 fetches its top rows from the whole table and then - applies the filters. Thus a narrow filter can leave BM25 with fewer rows than k. + loaded after its build. The engine applies the filters inside the BM25 search, so BM25 + fetches its top rows from the rows that the filters keep. 3. Reciprocal rank fusion adds `1 / (60 + rank)` from each ranking, and the query returns the k rows with the highest sum. Ties go to the smaller content distance, then to the newer row. diff --git a/docs/guarantees.md b/docs/guarantees.md index a43bd6c..2647ca5 100644 --- a/docs/guarantees.md +++ b/docs/guarantees.md @@ -53,6 +53,7 @@ A row that no test proves yet names the phase that will prove it. | Where does `HotdataStore` keep an episode? | A `put` of kind `episode` writes to `episode_v1`, and every other kind writes to `memory_v1`. `get`, `history`, `list`, `search`, `delete`, `sweep`, and `list_namespaces` read both tables. The cue vector of a record of either table is a row in `cue_v1`. | answered | `test_episode_goes_to_its_own_table`, `test_key_cannot_cross_the_episode_line` | | Is `provision` safe to run again? | Yes. If one database has the name, `provision` opens it after a check of its tables, columns, indexes, and the recorded model and vector size. If more than one database has the name, or the one found has another layout, it raises `LayoutError` and creates nothing. Two processes that provision the same new name at the same moment can both create a database, because the platform has no conditional create. `provision` reads every page of the database list, so a workspace with more than 100 databases does not get a second database with the name. An offline test proves the paging against a fake listing. No workspace with more than 100 databases was tried. | answered | `test_named_follows_the_cursor`, `test_provision_again_opens_the_same_database`, `test_provision_refuses_a_duplicate_name`, `test_provision_refuses_another_layout`, `test_provision_refuses_another_embedding_model` | | Does a plain vector index serve rows loaded after its build? | Only without a filter. With a `WHERE` filter in the same query, the engine searched the index and returned only the rows that were there at the build. So the vector rankings of `HotdataStore` do not use the index. They filter first and rank every row in scope by a scan, so a search in a narrow scope returns the top k rows of that scope. At 100,000 rows of 64 dimensions, a search in a scope of 1 percent of the rows took a median of 97 ms fused and 26 ms by content vector alone. | answered [measured, local] 2026-10-08 | `test_filtered_search_finds_every_later_write`, `test_search_in_a_narrow_scope_returns_its_top_k` | +| Does BM25 rank the rows of a narrow scope in a shared table? | Yes. The engine applies the scope filter inside `bm25_search`, before its depth. In a table of 2,020 rows, a depth of 100 in a scope of 20 rows returned all 20, although each scored below the other 2,000 rows. Whether rows of other scopes change the BM25 scores was not measured. | answered [measured, local and cloud] 2026-10-21 | `test_bm25_applies_the_scope_filter_before_its_depth` | | Does `HotdataStore` agree with `MemoryStore`? | Ranked by content vector alone, it returns the same records in the same order, with distances equal within 1e-6. With the fused ranking, it returns the same set when k covers every record. On the test fixture, the largest distance difference was 4.0e-8. | answered [measured, local] 2026-10-08 | `test_vector_ranking_matches_the_oracle`, `test_fused_ranking_returns_the_oracle_set` | | What does `load` store for a document? | One episode for each chunk, under the key `-`, with the first heading and the heading path at the top. Each fact that the extractor returns names the episode of its chunk in `sources`. A fact with no `valid_from` is valid from the `observed_at` of the load. A heading inside a fenced code block does not start a chunk, and no chunk body is longer than `chunk_chars`, unless one line is. | answered | `test_load_stores_each_chunk_as_an_episode_and_links_its_facts`, `test_a_loaded_fact_is_valid_from_the_observed_time`, `test_a_heading_inside_a_fence_is_not_a_heading`, `test_every_body_fits_the_size`, `test_a_cut_table_repeats_its_header` | | Does a second `load` of the same text write anything? | No. A chunk whose episode is current with the same content is skipped, and the extractor is not called for it, so this holds for an extractor that calls a model. A changed chunk writes the next revision of its episode. | answered | `test_a_second_load_of_the_same_text_writes_nothing_new`, `test_load_of_an_edited_chunk_writes_its_next_revision` | diff --git a/docs/internal/architecture.md b/docs/internal/architecture.md index 52acdf9..0399672 100644 --- a/docs/internal/architecture.md +++ b/docs/internal/architecture.md @@ -479,9 +479,11 @@ the table at the build of the index. [measured, local, 2026-10-08] The query nam table with an alias and names the distance, and that form makes the engine scan. [measured, local, 2026-10-08] -BM25 needs its index. `bm25_search` refuses to run on a table without one. [measured] It -fetches its top rows from the whole table before the filters apply. So a narrow scope can -leave BM25 with fewer than k rows, while the vector rankings still rank every row in scope. +BM25 needs its index. `bm25_search` refuses to run on a table without one. [measured] The +engine applies the filters inside `bm25_search`, so BM25 fetches its top rows from the +rows in scope. In a table of 2,020 rows, a depth of 100 in a scope of 20 rows returned all +20, although each scored below the 2,000 other rows. [measured, local and cloud, +2026-10-21] Whether the rows of other scopes change the BM25 scores was not measured. BM25 ranks each record table on its own, because the BM25 scores of two tables do not compare. A fused search with an `episode` filter and another kind thus has four rankings. diff --git a/docs/internal/plan.md b/docs/internal/plan.md index ffd92f0..78cdd4a 100644 --- a/docs/internal/plan.md +++ b/docs/internal/plan.md @@ -209,7 +209,6 @@ Tasks: - For each search, whether the top k equals a brute-force top k in Python over the same rows. While the rankings scan, any difference is a defect. - The `recall_any@k` against the evidence sessions. - - For the fused ranking, the share of BM25 hits that pass the namespace filter. It runs on the local stack first, then in the cloud. Embeddings for 1,000,000 rows cost about $2 to $5. The time of the load is not known. The script stops after @@ -261,8 +260,6 @@ Not in this phase, under Deferred in `roadmap.md`: - The vector rankings on the vector index. They wait for the engine's fix of the filtered search. When it lands, tests must prove that every write, revision, and delete is found through the vector index and the BM25 index. -- A BM25 search that applies the namespace filter inside the search. It needs a change in - the engine, and the request goes to the engine team with the numbers of task 4. - `UPDATE` and `DELETE` by SQL, until the engine has them. - The comparison with other memory layers, such as MemoryBench. It needs an account and a cost for each service, and it does not test a large store. diff --git a/docs/internal/roadmap.md b/docs/internal/roadmap.md index 4af4349..9392ebe 100644 --- a/docs/internal/roadmap.md +++ b/docs/internal/roadmap.md @@ -76,9 +76,6 @@ taken up now. - The vector rankings on the vector index, from phase 2. A filtered search through the vector index misses rows loaded after the build. The defect is reported to the engine team. Until it is fixed, the vector rankings scan, and phase 11 measures that cost. -- A BM25 search that applies the namespace filter inside the search, from phase 9. BM25 - ranks the whole table before the filter, so a narrow scope in a shared table loses - hits. It needs a change in the engine. Phase 11 measures the loss. - SQL `UPDATE` and `DELETE`, from phase 11. The engine refuses them, so a closed revision, `forget`, and `sweep` stay on keyed loads. - The comparison with other memory layers in one harness, such as MemoryBench, from diff --git a/src/hotmemory/hotdata.py b/src/hotmemory/hotdata.py index 1c29305..c46e00e 100644 --- a/src/hotmemory/hotdata.py +++ b/src/hotmemory/hotdata.py @@ -802,10 +802,10 @@ def _fused_query(tables: Sequence[str], where: str, vector: str, text: str, k: i Each ranking keeps only the rows that `where` keeps, numbers them from 1, and adds 1 / (RRF_CONSTANT + rank) to the score of each row. BM25 ranks each table on its own, - because BM25 scores of two tables do not compare. BM25 fetches its depth before the - filter applies, so a narrow filter can leave it with fewer rows. The vector rankings - filter first and rank by a scan, in the form of `_vector_query`, so they rank every - row that `where` keeps. + because BM25 scores of two tables do not compare. The engine applies `where` inside + `bm25_search`, so BM25 fetches its depth from the rows that `where` keeps. The vector + rankings filter first and rank by a scan, in the form of `_vector_query`, so they rank + every row that `where` keeps. """ depth = _depth(k) rank = "ROW_NUMBER() OVER (ORDER BY {order}) AS rank" From f25d7ea9e41f84bb2deb8e99c1c01d73de65a3d0 Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 21 Oct 2026 13:53:22 +0530 Subject: [PATCH 08/14] Phase 11: the retrieval runner reads LongMemEval-M --- bench/README.md | 1 + bench/data.py | 57 +++++++++++++++++++++++++++++++--------- bench/retrieval.py | 15 +++++++---- bench/tests/test_data.py | 12 +++++++++ 4 files changed, 68 insertions(+), 17 deletions(-) diff --git a/bench/README.md b/bench/README.md index 0f351fb..8cca2f1 100644 --- a/bench/README.md +++ b/bench/README.md @@ -37,6 +37,7 @@ The options are these: | `--location` | `local` or `cloud` | `local` | | `--url` | the address of the local engine | `http://localhost:3000` | | `--limit` | the first N scored questions | every scored question | +| `--data` | `s` for LongMemEval-S, or `m` for LongMemEval-M, about 500 sessions for each question. The M file is 2.74 GB. The runner reads it one question at a time and keeps only the user turns. | `s` | | `--ablate` | also rank with the fused query less one arm at a time: `bm25-content`, `content-cue`, and `bm25`; needs `--driver hotdata` | off | With `--location cloud`, the runner reads the connection from `HOTDATA_API_KEY`, diff --git a/bench/data.py b/bench/data.py index af69cc7..55a177d 100644 --- a/bench/data.py +++ b/bench/data.py @@ -1,4 +1,4 @@ -"""The LongMemEval-S data: the download, its checks, and the questions in date order.""" +"""The LongMemEval data, S and M: the download, its checks, and the questions in date order.""" from __future__ import annotations @@ -6,6 +6,7 @@ import json import re import urllib.request +from collections.abc import Iterator from dataclasses import dataclass from datetime import UTC, datetime, time from pathlib import Path @@ -19,6 +20,9 @@ f"{DATA_REVISION}/{DATA_FILE}" ) DATA_SHA256 = "d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442" +M_FILE = "longmemeval_m_cleaned.json" +M_SHA256 = "9d79e5524794a2e6900a3aa9cb7d9152c5a3e8319c9a87c25494ba1eacee495f" +DATASETS = {"s": (DATA_FILE, DATA_SHA256), "m": (M_FILE, M_SHA256)} DATE_FORMAT = "%Y/%m/%d (%a) %H:%M" END_OF_DAY = time(23, 59, 59, tzinfo=UTC) QUESTION_TYPES = ( @@ -176,39 +180,68 @@ def sha256(path: Path) -> str: return digest.hexdigest() -def download(directory: Path = DATA_DIR) -> Path: - """Download the data file into `directory` unless it is there, and return its path. +def download(directory: Path = DATA_DIR, dataset: str = "s") -> Path: + """Download the data file of `dataset` into `directory` unless it is there. - Raises ValueError, and keeps no file, if the download has another SHA-256. + Returns its path. Raises ValueError, and keeps no file, if the download has another + SHA-256. """ - path = directory / DATA_FILE + data_file, expected = DATASETS[dataset] + url = DATA_URL.removesuffix(DATA_FILE) + data_file + path = directory / data_file if path.exists(): return path directory.mkdir(parents=True, exist_ok=True) partial = path.with_suffix(".part") - with urllib.request.urlopen(DATA_URL) as response, partial.open("wb") as file: + with urllib.request.urlopen(url) as response, partial.open("wb") as file: while block := response.read(_CHUNK): file.write(block) found = sha256(partial) - if found != DATA_SHA256: + if found != expected: partial.unlink() - raise ValueError(f"{DATA_URL} has SHA-256 {found}, expected {DATA_SHA256}") + raise ValueError(f"{url} has SHA-256 {found}, expected {expected}") partial.rename(path) return path -def load(path: Path, expected_sha256: str | None = DATA_SHA256) -> list[Question]: +def load( + path: Path, expected_sha256: str | None = DATA_SHA256, user_only: bool = False +) -> list[Question]: """Return the questions of the data file at `path`, in file order. - Raises ValueError if `expected_sha256` is not None and the file has another SHA-256, - or if two questions share an id. + With `user_only`, the file is read one question at a time, and each session keeps only + its user turns. Raises ValueError if `expected_sha256` is not None and the file has + another SHA-256, or if two questions share an id. """ if expected_sha256 is not None: found = sha256(path) if found != expected_sha256: raise ValueError(f"{path} has SHA-256 {found}, expected {expected_sha256}") - questions = [parse_question(entry) for entry in json.loads(path.read_text())] + if user_only: + questions = [parse_question(_user_turns(entry)) for entry in _entries(path)] + else: + questions = [parse_question(entry) for entry in json.loads(path.read_text())] ids = [question.id for question in questions] if len(set(ids)) != len(ids): raise ValueError(f"{path} repeats a question id") return questions + + +def _entries(path: Path) -> Iterator[dict[str, Any]]: + """Yield the entries of the JSON array in the file at `path`, one at a time.""" + text = path.read_text() + decoder = json.JSONDecoder() + position = text.index("[") + 1 + while True: + while position < len(text) and text[position] in " \t\r\n,": + position += 1 + if position >= len(text) or text[position] == "]": + return + entry, position = decoder.raw_decode(text, position) + yield entry + + +def _user_turns(entry: dict[str, Any]) -> dict[str, Any]: + """Return `entry` with only the user turns of each session.""" + sessions = [[t for t in body if t["role"] == "user"] for body in entry["haystack_sessions"]] + return {**entry, "haystack_sessions": sessions} diff --git a/bench/retrieval.py b/bench/retrieval.py index c33f912..79e9c56 100644 --- a/bench/retrieval.py +++ b/bench/retrieval.py @@ -25,7 +25,7 @@ from typing import Any, Literal from bench import embed -from bench.data import DATA_DIR, DATA_FILE, DATA_REVISION, DATA_SHA256, Question, load +from bench.data import DATA_DIR, DATA_REVISION, DATASETS, Question, load from bench.episodes import scope, session_of, write_episodes from bench.metrics import KS, mean, score from hotmemory import Embedder, Filter, Hit, MemoryStore, Store @@ -227,14 +227,17 @@ def main(argv: Sequence[str] | None = None) -> Path: parser.add_argument("--location", choices=("local", "cloud"), default="local") parser.add_argument("--url", default="http://localhost:3000") parser.add_argument("--limit", type=int, default=None, help="the first N scored questions") + parser.add_argument("--data", choices=sorted(DATASETS), default="s", help="LongMemEval-S or -M") args = parser.parse_args(argv) + data_file, data_sha256 = DATASETS[args.data] rankings = RANKINGS if args.ranking == "both" else (args.ranking,) if args.ablate: if args.driver == "memory": parser.error("--ablate needs --driver hotdata") rankings = (*rankings, *ABLATED) - questions = [question for question in load(DATA_DIR / DATA_FILE) if question.is_scored] + loaded = load(DATA_DIR / data_file, data_sha256, user_only=args.data == "m") + questions = [question for question in loaded if question.is_scored] questions = questions[: args.limit] commit = _commit() started = datetime.now(UTC) @@ -255,10 +258,10 @@ def main(argv: Sequence[str] | None = None) -> Path: summaries = {name: summarize(rows, name) for name in rankings} result = { "config": { - "benchmark": "LongMemEval-S", - "data_file": DATA_FILE, + "benchmark": f"LongMemEval-{args.data.upper()}", + "data_file": data_file, "data_revision": DATA_REVISION, - "data_sha256": DATA_SHA256, + "data_sha256": data_sha256, "granularity": "session, user turns only", "embedder": { "model": embed.MODEL, @@ -288,6 +291,8 @@ def main(argv: Sequence[str] | None = None) -> Path: embedder.close() RESULTS_DIR.mkdir(exist_ok=True) name = f"retrieval-{target.driver}-{target.location}-{args.mode}-{'-'.join(rankings)}" + if args.data != "s": + name = name.replace("retrieval-", f"retrieval-{args.data}-", 1) if args.limit is not None: name += f"-first{args.limit}" path = RESULTS_DIR / f"{name}.json" diff --git a/bench/tests/test_data.py b/bench/tests/test_data.py index 58c94ba..f34ce29 100644 --- a/bench/tests/test_data.py +++ b/bench/tests/test_data.py @@ -103,3 +103,15 @@ def test_write_episodes(questions: dict[str, Question]) -> None: assert {session_of(hit.record.sources) for hit in hits} == { session.id for session in question.episodes() } + + +def test_load_user_only_streams_the_same_questions_with_user_turns() -> None: + fixture = Path(__file__).with_name("fixture.json") + full = load(fixture, expected_sha256=None) + lean = load(fixture, expected_sha256=None, user_only=True) + assert [q.id for q in lean] == [q.id for q in full] + for whole, part in zip(full, lean, strict=True): + assert [s.id for s in part.sessions] == [s.id for s in whole.sessions] + assert all(t.role == "user" for s in part.sessions for t in s.turns) + assert [s.user_text for s in part.sessions] == [s.user_text for s in whole.sessions] + assert part.evidence == whole.evidence From 59ac1412629e3e1661900f94d04a1715d70ae12c Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 21 Oct 2026 14:24:54 +0530 Subject: [PATCH 09/14] Phase 11: INSERT for small writes, and the vector indexes as an experimental option --- CHANGELOG.md | 11 ++ bench/scale.py | 90 ++++++++----- docs/contracts.md | 44 ++++--- docs/guarantees.md | 3 +- docs/internal/architecture.md | 36 ++++-- src/hotmemory/hotdata.py | 234 ++++++++++++++++++++++++++-------- tests/test_hotdata.py | 81 ++++++++++++ tests/test_hotdata_retry.py | 92 ++++++++++++- 8 files changed, 479 insertions(+), 112 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6642539..1fd98ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -56,6 +56,17 @@ each one, and its failure message points to this file. - `Memory.recall` and `Memory.profile` read the records of kind `fact`, `profile`, and `procedure` only. They never return an episode, and the counts of `profile` leave episodes out. +- `HotdataStore.provision` builds no vector index by default, so each write skips the cost + of the index. `provision` and `open` take `vector_index`, an experimental option: with + it, `provision` builds the three vector indexes, and the vector rankings search through + them. A filtered search through a vector index can miss rows loaded after its build. +- `HotdataStore` writes a table by SQL `INSERT` when its rows in the write are all new and + number at most 5, as in a `put` of a new fact. The `INSERT` writes only the ids that the + table does not hold. Other writes stay keyed loads. If the engine refuses the `INSERT`, + the store writes by load from then on. +- `HotdataStore` reads the current revisions before a write with lists of at most 500 + keys. Before, one condition for each key could overflow the stack of the engine on a + flush of about 2,000 rows. - `HotdataStore.provision` reads every page of the database list. Before, it read the first 100 databases, and in a larger workspace it could create a second database with the name. diff --git a/bench/scale.py b/bench/scale.py index 601b8d3..eb4ce8f 100644 --- a/bench/scale.py +++ b/bench/scale.py @@ -253,7 +253,9 @@ def checkpoint(self) -> dict[str, Any]: return result -def hotdata_stores(embedder: Embedder, location: str, url: str, name: str) -> Any: +def hotdata_stores( + embedder: Embedder, location: str, url: str, name: str, vector_index: bool +) -> Any: """Return the client, the database, and one store for each ranking.""" from hotdata_framework import HotdataClient @@ -264,7 +266,12 @@ def hotdata_stores(embedder: Embedder, location: str, url: str, name: str) -> An else: client = HotdataClient.from_env() store = HotdataStore.provision( - name, embedder=embedder, model=embed.MODEL, dimensions=embed.DIMENSIONS, client=client + name, + embedder=embedder, + model=embed.MODEL, + dimensions=embed.DIMENSIONS, + client=client, + vector_index=vector_index, ) options: dict[str, Any] = { "embedder": embedder, @@ -272,7 +279,9 @@ def hotdata_stores(embedder: Embedder, location: str, url: str, name: str) -> An "dimensions": embed.DIMENSIONS, } stores: dict[str, Store] = { - ranking: HotdataStore(client, store.database, ranking=ranking, **options) + ranking: HotdataStore( + client, store.database, ranking=ranking, vector_index=vector_index, **options + ) for ranking in ("vector", "fused") } return client, store.database, stores @@ -285,6 +294,9 @@ def main(argv: Sequence[str] | None = None) -> Path: parser.add_argument("--url", default="http://localhost:3000") parser.add_argument("--keep", action="store_true", help="keep the database after the run") parser.add_argument("--tag", default="") + parser.add_argument( + "--vector-index", action="store_true", help="provision with the experimental vector index" + ) args = parser.parse_args(argv) sizes = sorted(int(size) for size in args.sizes.split(",")) @@ -299,10 +311,29 @@ def main(argv: Sequence[str] | None = None) -> Path: embedder = embed.openai_embedder() run_id = uuid.uuid4().hex[:8] name = f"hotmemory-scale-{run_id}" - client, database, stores = hotdata_stores(embedder, args.location, args.url, name) + client, database, stores = hotdata_stores( + embedder, args.location, args.url, name, args.vector_index + ) run = Run(stores, embedder, scored, sizes[-1]) started = datetime.now(UTC) - checkpoints = [] + checkpoints: list[dict[str, Any]] = [] + tag = f"-{args.tag}" if args.tag else "" + path = RESULTS_DIR / f"scale-{args.location}{tag}-{run_id}.json" + path.parent.mkdir(parents=True, exist_ok=True) + config = { + "started": started.isoformat(), + "location": args.location, + "sizes": sizes, + "pool_rows": len(rows), + "questions": len(scored), + "chunk_chars": CHUNK_CHARS, + "flush": FLUSH, + "k": K, + "buckets": BUCKETS, + "vector_index": args.vector_index, + "commit": _commit(), + "database": name if args.keep else None, + } try: written = 0 for size in sizes: @@ -310,36 +341,37 @@ def main(argv: Sequence[str] | None = None) -> Path: written = size checkpoints.append(run.checkpoint()) print(json.dumps(checkpoints[-1]), flush=True) + write_result(path, config, run.flushes, sizes, checkpoints) finally: if not args.keep: client.delete_managed_database(database) - result = { - "config": { - "started": started.isoformat(), - "location": args.location, - "sizes": sizes, - "pool_rows": len(rows), - "questions": len(scored), - "chunk_chars": CHUNK_CHARS, - "flush": FLUSH, - "k": K, - "buckets": BUCKETS, - "commit": _commit(), - "database": name if args.keep else None, - }, - "flushes": run.flushes, - "flush_s_by_size": { - str(size): spread([f["s"] for f in run.flushes if prev < f["total"] <= size]) + print(path) + return path + + +def write_result( + path: Path, + config: dict[str, Any], + flushes: Sequence[dict[str, float]], + sizes: Sequence[int], + checkpoints: Sequence[dict[str, Any]], +) -> None: + """Write the configuration, the flushes, and the checkpoints so far to `path`.""" + + def by_size(field: str) -> dict[str, dict[str, float]]: + return { + str(size): spread([f[field] for f in flushes if prev < f["total"] <= size]) for prev, size in zip([0, *sizes], sizes, strict=False) - }, - "checkpoints": checkpoints, + } + + result = { + "config": config, + "flushes": list(flushes), + "flush_s_by_size": by_size("s"), + "read_s_by_size": by_size("read_s"), + "checkpoints": list(checkpoints), } - tag = f"-{args.tag}" if args.tag else "" - path = RESULTS_DIR / f"scale-{args.location}{tag}-{run_id}.json" - path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps(result, indent=1) + "\n") - print(path) - return path if __name__ == "__main__": diff --git a/docs/contracts.md b/docs/contracts.md index e373118..0e4bf1a 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -17,7 +17,9 @@ API owns and loads from files. The facts below shape the contracts. A fact marke 2026-09-18. - A write is a file load. The caller uploads parquet and calls a keyed load in replace, - append, upsert, update, or delete mode. [measured] + append, upsert, update, or delete mode. [measured] From 2026-10-18, a SQL `INSERT` + appends rows too. It is about three times faster than a load for one row, and slower + above 10 rows. [measured, 2026-10-21] The engine refuses SQL `UPDATE` and `DELETE`. - A write costs about 2.1 seconds per call, and the cost does not change with row count. One row costs 1.75 seconds. Ten thousand rows cost 2.5 seconds. Call count is the only cost that a caller controls. [measured] @@ -170,9 +172,9 @@ Version 1 ships two drivers. with. `records()` returns every revision that a store holds, so a caller can save a store and build it again. `MemoryStore` ranks by the cosine distance between the query and `content` only. It does not rank by BM25 or by `cues`. -- `HotdataStore` uses one managed database, keyed loads, a lock for its writes, and the - retrieval query below. It needs the `hotdata` extra. With the local RuntimeDB stack, it - is also the development driver. The section on `HotdataStore` below gives the details. +- `HotdataStore` uses one managed database, keyed loads and small SQL inserts, a lock + for its writes, and the retrieval query below. It needs the `hotdata` extra. With the + local RuntimeDB stack, it is also the development driver. The section on `HotdataStore` below gives the details. ### Tables and retrieval @@ -234,18 +236,28 @@ timestamp in UTC to the microsecond. The embedding model and the vector size are each database. If an embedder returns a vector of another size, the write or search raises `ValueError`, and a write sends nothing. -`provision` builds three indexes in each record table: BM25 on `content`, plain cosine -vector on `content_embedding`, and sorted on `created_at`. In `cue_v1`, it -builds a plain cosine vector index on `cues_embedding`. The engine refuses an index on an -empty table. So `provision` loads one seed row into each table, builds the indexes, and -deletes the seed rows. The vector indexes stay built, but no ranking uses them until the -engine fixes the filtered search through the index. +`provision` builds two indexes in each record table: BM25 on `content`, and sorted on +`created_at`. The engine refuses an index on an empty table. So `provision` loads one seed +row into each table, builds the indexes, and deletes the seed rows. + +`vector_index=True`, in `provision` and `open`, is experimental. With it, `provision` also +builds a plain cosine vector index on `content_embedding` in each record table, and on +`cues_embedding` in `cue_v1`. The vector rankings then search through these indexes, with +the filters inside the search. The engine can miss rows loaded after the index build in a +filtered search, so such a search can miss recent writes. Each vector index also makes +each write slower: a load of 1,000 rows into a table of 10,000 rows took about 8.5 s with +the index and about 1 s without it in the cloud. [measured, 2026-10-21] A database that has +the vector indexes opens without the option too, and its vector rankings then scan. A +database without them raises `LayoutError` with the option. A `put` reads the current revision, then writes the new row and the superseded row in one -load. If the record has cues, a second load writes its cue row into `cue_v1` at the same -time. The two loads are not atomic. If the cue load fails, the record has no cue row, and +load. If the record has cues, a second write puts its cue row into `cue_v1` at the same +time. A table whose rows in the write are all new, at most 5 of them, gets one SQL +`INSERT` and not a load. The `INSERT` writes only the ids that the table does not hold, so +a second try writes no row twice. A store whose engine refuses the `INSERT` writes by +upsert load from then on. The two writes are not atomic. If the cue load fails, the record has no cue row, and the cue ranking misses it until a retry. If the record load fails, the cue row is left, -and the search drops it. A writer sends one load per table at each flush. `delete` and +and the search drops it. A writer sends one write per table at each flush. `delete` and `sweep` delete the rows of every revision from the record table and from `cue_v1`. A store holds a lock across the read and the load of each write, so one store writes in @@ -253,9 +265,9 @@ order. One process writes to a database. Two processes that put the same key can the same current revision and both write the next one. Then the later load replaces the earlier row. This rule stays until the engine has a conditional write. -If the engine refuses a load with `409 RESOURCE_LOCKED`, the driver tries it again, up to -8 attempts in all. The wait starts at 0.25 seconds and doubles to at most 4 seconds. After -the last attempt, the driver raises the error. +If the engine refuses a load or an `INSERT` with `409 RESOURCE_LOCKED`, the driver tries +it again, up to 8 attempts in all. The wait starts at 0.25 seconds and doubles to at most +4 seconds. After the last attempt, the driver raises the error. A `search` with query text runs one SQL query over both record tables, and over `cue_v1` with `ranking="fused"`. If diff --git a/docs/guarantees.md b/docs/guarantees.md index 2647ca5..6d905f8 100644 --- a/docs/guarantees.md +++ b/docs/guarantees.md @@ -35,6 +35,7 @@ A row that no test proves yet names the phase that will prove it. | After a synchronous `put` returns, does `list` see the record? | Yes. A read after a write was never stale in 30 trials. | answered [measured] | `test_synchronous_put_is_visible_to_list` | | After a synchronous `put` returns, does `search` see the record? | Yes. Without an index, the retrieval query scans the table. With a provider-backed vector index or a BM25 index, the first search after the load returned the new row. | answered [measured], M1 | `test_synchronous_put_is_visible_to_search` | | After a buffered `put` returns, is the record visible? | No. It is visible after the writer flushes. The writer records the ids that it flushed. | answered | `test_buffered_put_is_visible_after_flush` | +| Can a retried `INSERT` write a row twice? | No. `HotdataStore` sends an `INSERT` only for at most 5 new rows of one table, and the statement keeps only the ids that the table does not hold. A second run of the same statement wrote 0 rows. If the engine refuses the `INSERT` for a reason other than a lock, the store writes by upsert load from then on. | answered [measured, local] 2026-10-21 | `test_a_second_insert_writes_nothing`, `test_a_refused_insert_falls_back_to_the_load_from_then_on` | | Two processes write to the same table. What happens? | The engine refuses the second load with 409 `RESOURCE_LOCKED`. The driver retries up to 8 attempts, with a backoff from 0.25 seconds that doubles to at most 4 seconds, and then raises the error. Inside one process, the store sends one load at a time for each table. The integration test runs two stores in one process, which the engine cannot tell apart from two processes. | answered [measured], M3 | `test_two_stores_write_one_database_at_once`, `test_a_locked_load_retries_with_a_doubling_backoff`, `test_a_load_stops_after_the_last_attempt` | | Two writers in one process put the same key. What happens? | The last writer wins at the row level. Revisions are new rows, so both revisions exist and the later one is current. This holds inside one process only. The next row gives the rule for two processes. | answered | `test_last_writer_wins_on_one_key`, `test_one_store_serializes_writes_on_one_key` | | Can two processes write to one database? | Not safely. Two processes that put the same key can both read the same current revision. Both then write the same next revision, and the later load replaces the earlier row, so one revision is lost. One process writes to a database. Other processes can read it. This rule stays until the engine has a conditional write. | answered | `test_one_store_serializes_writes_on_one_key` | @@ -52,7 +53,7 @@ A row that no test proves yet names the phase that will prove it. | Can a caller change a stored record? | No. A record is frozen, and the store copies `payload` on the way in and on the way out. A change to a dict that the caller holds does not reach the store. | answered | `test_stored_record_cannot_change` | | Where does `HotdataStore` keep an episode? | A `put` of kind `episode` writes to `episode_v1`, and every other kind writes to `memory_v1`. `get`, `history`, `list`, `search`, `delete`, `sweep`, and `list_namespaces` read both tables. The cue vector of a record of either table is a row in `cue_v1`. | answered | `test_episode_goes_to_its_own_table`, `test_key_cannot_cross_the_episode_line` | | Is `provision` safe to run again? | Yes. If one database has the name, `provision` opens it after a check of its tables, columns, indexes, and the recorded model and vector size. If more than one database has the name, or the one found has another layout, it raises `LayoutError` and creates nothing. Two processes that provision the same new name at the same moment can both create a database, because the platform has no conditional create. `provision` reads every page of the database list, so a workspace with more than 100 databases does not get a second database with the name. An offline test proves the paging against a fake listing. No workspace with more than 100 databases was tried. | answered | `test_named_follows_the_cursor`, `test_provision_again_opens_the_same_database`, `test_provision_refuses_a_duplicate_name`, `test_provision_refuses_another_layout`, `test_provision_refuses_another_embedding_model` | -| Does a plain vector index serve rows loaded after its build? | Only without a filter. With a `WHERE` filter in the same query, the engine searched the index and returned only the rows that were there at the build. So the vector rankings of `HotdataStore` do not use the index. They filter first and rank every row in scope by a scan, so a search in a narrow scope returns the top k rows of that scope. At 100,000 rows of 64 dimensions, a search in a scope of 1 percent of the rows took a median of 97 ms fused and 26 ms by content vector alone. | answered [measured, local] 2026-10-08 | `test_filtered_search_finds_every_later_write`, `test_search_in_a_narrow_scope_returns_its_top_k` | +| Does a plain vector index serve rows loaded after its build? | Only without a filter. With a `WHERE` filter in the same query, the engine searched the index and returned only the rows that were there at the build. So the vector rankings of `HotdataStore` do not use the index. They filter first and rank every row in scope by a scan, so a search in a narrow scope returns the top k rows of that scope. At 100,000 rows of 64 dimensions, a search in a scope of 1 percent of the rows took a median of 97 ms fused and 26 ms by content vector alone. `provision` builds no vector index by default. With the experimental `vector_index=True`, the vector rankings search through the index, and a test marked as an expected failure checks that they find every later write. | answered [measured, local] 2026-10-08 | `test_filtered_search_finds_every_later_write`, `test_search_in_a_narrow_scope_returns_its_top_k`, `test_a_search_through_the_vector_index_finds_every_later_write` | | Does BM25 rank the rows of a narrow scope in a shared table? | Yes. The engine applies the scope filter inside `bm25_search`, before its depth. In a table of 2,020 rows, a depth of 100 in a scope of 20 rows returned all 20, although each scored below the other 2,000 rows. Whether rows of other scopes change the BM25 scores was not measured. | answered [measured, local and cloud] 2026-10-21 | `test_bm25_applies_the_scope_filter_before_its_depth` | | Does `HotdataStore` agree with `MemoryStore`? | Ranked by content vector alone, it returns the same records in the same order, with distances equal within 1e-6. With the fused ranking, it returns the same set when k covers every record. On the test fixture, the largest distance difference was 4.0e-8. | answered [measured, local] 2026-10-08 | `test_vector_ranking_matches_the_oracle`, `test_fused_ranking_returns_the_oracle_set` | | What does `load` store for a document? | One episode for each chunk, under the key `-`, with the first heading and the heading path at the top. Each fact that the extractor returns names the episode of its chunk in `sources`. A fact with no `valid_from` is valid from the `observed_at` of the load. A heading inside a fenced code block does not start a chunk, and no chunk body is longer than `chunk_chars`, unless one line is. | answered | `test_load_stores_each_chunk_as_an_episode_and_links_its_facts`, `test_a_loaded_fact_is_valid_from_the_observed_time`, `test_a_heading_inside_a_fence_is_not_a_heading`, `test_every_body_fits_the_size`, `test_a_cut_table_repeats_its_header` | diff --git a/docs/internal/architecture.md b/docs/internal/architecture.md index 0399672..057c789 100644 --- a/docs/internal/architecture.md +++ b/docs/internal/architecture.md @@ -66,7 +66,7 @@ token. [local.md](../local.md) describes the local stack. | Concurrent writes | One lock in each store, and a retry on `409 RESOURCE_LOCKED` | Locks a table for each load, and refuses a second load with 409. [measured] | | Expiry | Hides a record past `forget_after` in SQL, and deletes it in `sweep` | Deletes nothing. [measured] | | Rows on disk | Builds each batch of rows as a parquet file | Stores the rows as parquet files and records them in DuckLake metadata. [observed, local] | -| Indexes | Asks for seven indexes at provision | Builds and stores them. [observed, local] | +| Indexes | Asks for four indexes at provision, or seven with `vector_index=True` | Builds and stores them. [observed, local] | | Queries | Writes every SQL query, with values as literals | Plans and runs the SQL. | ## Where the data lives @@ -87,7 +87,7 @@ erDiagram int64 revision string kind "fact, profile, or procedure" string content "BM25 index" - list_float content_embedding "vector index, cosine" + list_float content_embedding "vector index, cosine, with vector_index" timestamp created_at "sorted index" string superseded_by timestamp valid_from @@ -99,12 +99,12 @@ erDiagram string id PK string kind "episode" string content "BM25 index" - list_float content_embedding "vector index, cosine" + list_float content_embedding "vector index, cosine, with vector_index" timestamp created_at "sorted index" } cue_v1 { string id PK "the id of a record row" - list_float cues_embedding "vector index, cosine" + list_float cues_embedding "vector index, cosine, with vector_index" } meta_v1 { string id PK "meta" @@ -121,9 +121,9 @@ UTC to the microsecond. | Table | Rows | Indexes | |---|---|---| -| `memory_v1` | The records of kind `fact`, `profile`, and `procedure` | BM25 on `content`, plain cosine vector on `content_embedding`, sorted on `created_at` | -| `episode_v1` | The records of kind `episode` | The same three | -| `cue_v1` | One row for each record that has cues: its id and the vector of its cues | Plain cosine vector on `cues_embedding` | +| `memory_v1` | The records of kind `fact`, `profile`, and `procedure` | BM25 on `content`, sorted on `created_at`, and with `vector_index=True` plain cosine vector on `content_embedding` | +| `episode_v1` | The records of kind `episode` | The same | +| `cue_v1` | One row for each record that has cues: its id and the vector of its cues | None, or with `vector_index=True` plain cosine vector on `cues_embedding` | | `meta_v1` | One row: the schema version, the embedding model name, and the vector size | None | Every table has the key column `id`. A keyed load replaces or deletes rows by this column. @@ -157,13 +157,13 @@ flowchart TD start["provision(name)"] --> list["List the managed databases
and keep those whose description is name"] list --> count{"How many?"} count -- "more than one" --> dup["Raise LayoutError
and create nothing"] - count -- "one" --> check["Check the tables, the columns,
the seven indexes, and the meta row"] + count -- "one" --> check["Check the tables, the columns,
the indexes, and the meta row"] check --> same{"Same layout, model,
and vector size?"} same -- "yes" --> open["Return the store"] same -- "no" --> bad["Raise LayoutError"] count -- "none" --> create["Create the database
with four tables keyed by id"] create --> seed["Load one seed row into
memory_v1, episode_v1, and cue_v1"] - seed --> index["Build the seven indexes"] + seed --> index["Build the four indexes,
or seven with vector_index"] index --> unseed["Delete the seed rows"] unseed --> meta["Load the meta row into meta_v1"] meta --> open @@ -199,7 +199,7 @@ sequenceDiagram F->>S: PUT the parquet file F->>API: finish the upload, then load it end - loop seven indexes + loop four indexes, or seven with vector_index P->>F: create_index(table, column, type, metric) F->>API: submit the index build F->>API: poll every 0.2 s until it is done @@ -303,7 +303,11 @@ key in one flush write revisions n+1 and n+2. ### The loads The new rows and the superseded rows of one table go into one parquet file and one upsert -load. The cue vectors of the new rows go into a second load, on `cue_v1`. The loads of one +load. The cue vectors of the new rows go into a second load, on `cue_v1`. A table whose +rows are all new, at most 5 of them, gets a SQL `INSERT` in place of the load. So a `put` +of a new fact with cues sends two `INSERT` statements and no upload. The `INSERT` selects +its rows from a `VALUES` list, and keeps only the ids that the table does not hold, so a +retried `INSERT` writes no row twice. The loads of one batch run at the same time, one thread for each load. They are not atomic. If the cue load fails, the record has no cue row. If the record load fails, a cue row is left without a record, and the search drops it. @@ -319,8 +323,8 @@ then polls the job. [read from source, not observed] The engine then writes the object storage and records them in the catalog. The order of those two steps was not observed. -hotmemory never sends `INSERT`, `UPDATE`, or `DELETE` in SQL. Every change to a row is a -keyed load. The cloud accepts `INSERT` and refuses `UPDATE` and `DELETE`. One row by +hotmemory sends `INSERT` only for a small write of new rows. Every other change to a row +is a keyed load. The cloud accepts `INSERT` and refuses `UPDATE` and `DELETE`. One row by `INSERT` took 531 ms in the cloud, against 1,682 ms by load. [measured, M4, 2026-10-21] ## Remember and capture @@ -479,6 +483,12 @@ the table at the build of the index. [measured, local, 2026-10-08] The query nam table with an alias and names the distance, and that form makes the engine scan. [measured, local, 2026-10-08] +With `vector_index=True`, each vector ranking sends one subquery for each table with no +alias, the filters in its `WHERE`, and `ORDER BY cosine_distance(...) LIMIT n`. That is +the form that the engine serves from the vector index. The test that every later write +is found through the index is marked as an expected failure, so it fails when the engine +fixes the filtered search. + BM25 needs its index. `bm25_search` refuses to run on a table without one. [measured] The engine applies the filters inside `bm25_search`, so BM25 fetches its top rows from the rows in scope. In a table of 2,020 rows, a depth of 100 in a scope of 20 rows returned all diff --git a/src/hotmemory/hotdata.py b/src/hotmemory/hotdata.py index c46e00e..cd29452 100644 --- a/src/hotmemory/hotdata.py +++ b/src/hotmemory/hotdata.py @@ -65,6 +65,7 @@ LOAD_ATTEMPTS = 8 LIST_PAGE = 100 SLOT_BATCH = 500 +INSERT_ROWS = 5 FIRST_BACKOFF = 0.25 MAX_BACKOFF = 4.0 @@ -98,8 +99,9 @@ ] ) READ_COLUMNS = tuple(name for name in RECORD_SCHEMA.names if name != "content_embedding") -_Load = tuple[str, pa.Table, ManagedLoadMode] -"""One load: the table, the rows, and the load mode.""" +WriteMode = ManagedLoadMode | Literal["insert"] +_Load = tuple[str, pa.Table, WriteMode] +"""One write: the table, the rows, and the load mode, or `insert` for a SQL INSERT.""" CUE_SCHEMA = pa.schema([("id", pa.string()), ("cues_embedding", _VECTOR)]) META_SCHEMA = pa.schema( @@ -130,8 +132,10 @@ class _Index: INDEXES = ( *(_Index(table, "bm25", "content") for table in RECORD_TABLES), - *(_Index(table, "vector", "content_embedding", "cosine") for table in RECORD_TABLES), *(_Index(table, "sorted", "created_at") for table in RECORD_TABLES), +) +VECTOR_INDEXES = ( + *(_Index(table, "vector", "content_embedding", "cosine") for table in RECORD_TABLES), _Index(CUE_TABLE, "vector", "cues_embedding", "cosine"), ) @@ -156,6 +160,11 @@ class HotdataStore: as `MemoryStore` does. `ranking="fused"` ranks by reciprocal rank fusion of BM25 over `content`, cosine distance over the content vectors, and cosine distance over the cue vectors. In both, `Hit.distance` is the content cosine distance. + + `vector_index=True` is experimental. The vector rankings then search through the + vector indexes, with the filters inside the index search, and do not scan. The engine + can miss rows loaded after the index build in such a search, so a search can miss + recent writes. The default scans every row in scope, which is exact. """ def __init__( @@ -168,6 +177,7 @@ def __init__( dimensions: int, clock: Clock = utc_now, ranking: Ranking = "vector", + vector_index: bool = False, ) -> None: if ranking not in ("vector", "fused"): raise ValueError(f"ranking must be 'vector' or 'fused', got {ranking!r}") @@ -178,8 +188,10 @@ def __init__( self._dimensions = dimensions self._clock = clock self._ranking = ranking + self._vector_index = vector_index self._lock = threading.Lock() self._sleep: Callable[[float], None] = time.sleep + self._inserts = True @classmethod def provision( @@ -191,6 +203,7 @@ def provision( dimensions: int, client: HotdataClient | None = None, clock: Clock = utc_now, + vector_index: bool = False, ) -> HotdataStore: """Open the database named `name`, or create it if no database has that name. @@ -201,6 +214,10 @@ def provision( database has the name or the one found has another layout. Two processes that provision the same new name at the same moment can both create a database. + With `vector_index=True`, an experimental option, a new database also gets the + three vector indexes, and the store searches through them. A database found + without them raises LayoutError. Each vector index makes each write slower. + `client` defaults to `HotdataClient.from_env()`. """ _check_settings(model, dimensions) @@ -211,11 +228,17 @@ def provision( raise LayoutError(f"{len(found)} databases are named {name!r}: {ids}") if found: database = found[0] - _check_layout(client, database, model, dimensions) + _check_layout(client, database, model, dimensions, vector_index) else: - database = _create(client, name, model, dimensions) + database = _create(client, name, model, dimensions, vector_index) return cls( - client, database, embedder=embedder, model=model, dimensions=dimensions, clock=clock + client, + database, + embedder=embedder, + model=model, + dimensions=dimensions, + clock=clock, + vector_index=vector_index, ) @classmethod @@ -228,6 +251,7 @@ def open( dimensions: int, client: HotdataClient | None = None, clock: Clock = utc_now, + vector_index: bool = False, ) -> HotdataStore: """Open the database with the id `database_id`, after the checks of `provision`. @@ -238,9 +262,15 @@ def open( database = client.resolve_managed_database(database_id) if database.id != database_id: raise KeyError(f"no database has the id {database_id!r}") - _check_layout(client, database, model, dimensions) + _check_layout(client, database, model, dimensions, vector_index) return cls( - client, database, embedder=embedder, model=model, dimensions=dimensions, clock=clock + client, + database, + embedder=embedder, + model=model, + dimensions=dimensions, + clock=clock, + vector_index=vector_index, ) def put( @@ -336,9 +366,9 @@ def search( return [] vector = _vector(self._embed([query])[0]) if self._ranking == "vector": - sql = _vector_query(tables, where, vector, k) + sql = _vector_query(tables, where, vector, k, self._vector_index) else: - sql = _fused_query(tables, where, vector, _bm25_text(query), k) + sql = _fused_query(tables, where, vector, _bm25_text(query), k, self._vector_index) return [Hit(_record(row), float(row["distance"])) for row in self._sql(sql).to_records()] def delete(self, namespace: Sequence[str], key: str) -> None: @@ -488,7 +518,11 @@ def _current( def _load_records( self, records: Sequence[Record], vectors: dict[str, builtins.list[float]] ) -> None: - """Upsert `records` into their tables, and the cue rows of the new ones.""" + """Write `records` into their tables, and the cue rows of the new ones. + + A table whose rows are all new and number at most INSERT_ROWS gets a SQL INSERT, + and so do the cue rows. Every other table gets one upsert load. + """ fresh = [record for record in records if record.id not in vectors] with_cues = [record for record in fresh if record.cues] embedded = self._embed( @@ -500,13 +534,15 @@ def _load_records( for record in records: vector = vectors[record.id] if record.id in vectors else content[record.id] by_table.setdefault(_table(record.kind), []).append(_row(record, vector)) - loads: builtins.list[_Load] = [ - (table, pa.Table.from_pylist(rows, schema=RECORD_SCHEMA), "upsert") - for table, rows in by_table.items() - ] + loads: builtins.list[_Load] = [] + for table, rows in by_table.items(): + new = all(row["id"] not in vectors for row in rows) + mode: WriteMode = "insert" if new and len(rows) <= INSERT_ROWS else "upsert" + loads.append((table, pa.Table.from_pylist(rows, schema=RECORD_SCHEMA), mode)) if cues: cue_rows = {"id": builtins.list(cues), "cues_embedding": builtins.list(cues.values())} - loads.append((CUE_TABLE, pa.table(cue_rows, schema=CUE_SCHEMA), "upsert")) + mode = "insert" if len(cues) <= INSERT_ROWS else "upsert" + loads.append((CUE_TABLE, pa.table(cue_rows, schema=CUE_SCHEMA), mode)) self._load_all(loads) def _embed(self, texts: Sequence[str]) -> builtins.list[builtins.list[float]]: @@ -536,25 +572,44 @@ def _load_all(self, loads: Sequence[_Load]) -> None: for future in futures: future.result() - def _load(self, table: str, rows: pa.Table, mode: ManagedLoadMode) -> None: - """Upload `rows` once and load them, with a retry when the table is locked. + def _load(self, table: str, rows: pa.Table, mode: WriteMode) -> None: + """Write `rows` by INSERT or by one upload and load, with a retry when locked. - A load refused with 409 RESOURCE_LOCKED is tried up to LOAD_ATTEMPTS times, with - a backoff from FIRST_BACKOFF seconds that doubles up to MAX_BACKOFF seconds. + A write refused with 409 RESOURCE_LOCKED is tried up to LOAD_ATTEMPTS times, with + a backoff from FIRST_BACKOFF seconds that doubles up to MAX_BACKOFF seconds. The + INSERT writes only the ids that the table does not hold, so a second try writes + nothing twice. If the engine refuses the INSERT for another reason, this store + writes by upsert load from then on. """ + if mode == "insert" and self._inserts: + sql = _insert_sql(table, rows) + try: + self._retry_locked(lambda: self._sql(sql)) + return + except RuntimeError as error: + if _is_locked(error): + raise + self._inserts = False + load_mode: ManagedLoadMode = "upsert" if mode == "insert" else mode with tempfile.TemporaryDirectory() as tmp: upload_id = self.client.upload_parquet(_Files(Path(tmp)).write(rows)) + self._retry_locked( + lambda: self.client.load_managed_table( + self.database, + table, + schema=SCHEMA, + upload_id=upload_id, + mode=load_mode, + key=["id"], + ) + ) + + def _retry_locked(self, call: Callable[[], object]) -> None: + """Run `call`, and run it again while the table is locked, up to LOAD_ATTEMPTS.""" delay = FIRST_BACKOFF for attempt in range(1, LOAD_ATTEMPTS + 1): try: - self.client.load_managed_table( - self.database, - table, - schema=SCHEMA, - upload_id=upload_id, - mode=mode, - key=["id"], - ) + call() return except RuntimeError as error: if attempt == LOAD_ATTEMPTS or not _is_locked(error): @@ -596,7 +651,9 @@ def _named(client: HotdataClient, name: str) -> builtins.list[ManagedDatabase]: return [client.resolve_managed_database(database_id) for database_id in ids] -def _create(client: HotdataClient, name: str, model: str, dimensions: int) -> ManagedDatabase: +def _create( + client: HotdataClient, name: str, model: str, dimensions: int, vector_index: bool +) -> ManagedDatabase: """Create the database, load a seed row into each table, index it, and remove the seeds. The meta row is loaded last, so a database without it did not finish provisioning. @@ -609,7 +666,7 @@ def _create(client: HotdataClient, name: str, model: str, dimensions: int) -> Ma for table in (*RECORD_TABLES, CUE_TABLE): seed = files.write(_seed_rows(table, dimensions)) client.load_managed_table(database, table, schema=SCHEMA, file=seed, mode="replace") - for index in INDEXES: + for index in (*INDEXES, *(VECTOR_INDEXES if vector_index else ())): client.create_index( database, index.table, @@ -650,9 +707,17 @@ def _seed_rows(table: str, dimensions: int) -> pa.Table: def _check_layout( - client: HotdataClient, database: ManagedDatabase, model: str, dimensions: int + client: HotdataClient, + database: ManagedDatabase, + model: str, + dimensions: int, + vector_index: bool = False, ) -> None: - """Raise LayoutError unless `database` has the tables, columns, indexes, and meta row.""" + """Raise LayoutError unless `database` has the tables, columns, indexes, and meta row. + + The vector indexes are checked only with `vector_index`. A database that has them + opens without it too. + """ columns = { info.table: [column.name for column in info.columns or []] for info in client.iter_tables( @@ -675,7 +740,7 @@ def _check_layout( for table in (*RECORD_TABLES, CUE_TABLE) for index in indexes.list_indexes(database.default_connection_id, SCHEMA, table).indexes } - for wanted in INDEXES: + for wanted in (*INDEXES, *(VECTOR_INDEXES if vector_index else ())): if (wanted.table, wanted.index_type, (wanted.column,), wanted.metric) not in built: raise LayoutError( f"table {wanted.table} of database {database.id} has no " @@ -781,23 +846,36 @@ def _union(tables: Sequence[str], select: str, where: str) -> str: ) -def _vector_query(tables: Sequence[str], where: str, vector: str, k: int) -> str: - """Return the query that ranks by the content cosine distance alone, filters first. +def _vector_query( + tables: Sequence[str], where: str, vector: str, k: int, indexed: bool = False +) -> str: + """Return the query that ranks by the content cosine distance alone. - The table alias and the named distance keep the engine from the vector index, so the - query scans and stays exact. A filtered search through the index misses rows loaded - after the index build. + By default, the table alias and the named distance keep the engine from the vector + index, so the query filters first, scans, and stays exact. A filtered search through + the index misses rows loaded after the index build. With `indexed`, each table gets + the form that the engine serves from the vector index, with `where` inside it. """ columns = ", ".join(f"scanned.{column}" for column in READ_COLUMNS) - rows = " UNION ALL ".join( - f"SELECT {columns}, cosine_distance(scanned.content_embedding, {vector}) AS distance " - f"FROM {_ref(table)} scanned WHERE {where}" - for table in tables - ) + if indexed: + rows = " UNION ALL ".join( + f"SELECT * FROM (SELECT {', '.join(READ_COLUMNS)}, " + f"cosine_distance(content_embedding, {vector}) AS distance FROM {_ref(table)} " + f"WHERE {where} ORDER BY cosine_distance(content_embedding, {vector}) LIMIT {k})" + for table in tables + ) + else: + rows = " UNION ALL ".join( + f"SELECT {columns}, cosine_distance(scanned.content_embedding, {vector}) AS distance " + f"FROM {_ref(table)} scanned WHERE {where}" + for table in tables + ) return f"SELECT * FROM ({rows}) ORDER BY distance ASC, {NEWEST_FIRST} LIMIT {k}" -def _fused_query(tables: Sequence[str], where: str, vector: str, text: str, k: int) -> str: +def _fused_query( + tables: Sequence[str], where: str, vector: str, text: str, k: int, indexed: bool = False +) -> str: """Return the query that fuses BM25, content vector, and cue vector rankings. Each ranking keeps only the rows that `where` keeps, numbers them from 1, and adds @@ -818,11 +896,30 @@ def _fused_query(tables: Sequence[str], where: str, vector: str, text: str, k: i f"bm25_search('{_ref(table)}', 'content', {_text(text)}, {depth}) WHERE {where})" ) rankings.append(name) - nearest = " UNION ALL ".join( - f"SELECT scanned.id, cosine_distance(scanned.content_embedding, {vector}) AS distance " - f"FROM {_ref(table)} scanned WHERE {where}" - for table in tables - ) + if indexed: + nearest = " UNION ALL ".join( + f"SELECT * FROM (SELECT id, cosine_distance(content_embedding, {vector}) AS distance " + f"FROM {_ref(table)} WHERE {where} " + f"ORDER BY cosine_distance(content_embedding, {vector}) LIMIT {depth})" + for table in tables + ) + cues = ( + f"SELECT * FROM (SELECT id, cosine_distance(cues_embedding, {vector}) AS distance " + f"FROM {_ref(CUE_TABLE)} WHERE id IN ({_union(tables, 'id', where)}) " + f"ORDER BY cosine_distance(cues_embedding, {vector}) LIMIT {depth}) " + f"ORDER BY distance ASC, id ASC LIMIT {depth}" + ) + else: + nearest = " UNION ALL ".join( + f"SELECT scanned.id, cosine_distance(scanned.content_embedding, {vector}) AS distance " + f"FROM {_ref(table)} scanned WHERE {where}" + for table in tables + ) + cues = ( + f"SELECT scanned.id, cosine_distance(scanned.cues_embedding, {vector}) AS distance " + f"FROM {_ref(CUE_TABLE)} scanned WHERE scanned.id IN ({_union(tables, 'id', where)}) " + f"ORDER BY distance ASC, id ASC LIMIT {depth}" + ) parts.append( f"content_rank AS (SELECT candidate.id, " f"{rank.format(order='candidate.distance ASC, candidate.id ASC')} FROM (" @@ -832,9 +929,7 @@ def _fused_query(tables: Sequence[str], where: str, vector: str, text: str, k: i parts.append( f"cue_rank AS (SELECT candidate.id, " f"{rank.format(order='candidate.distance ASC, candidate.id ASC')} FROM (" - f"SELECT scanned.id, cosine_distance(scanned.cues_embedding, {vector}) AS distance " - f"FROM {_ref(CUE_TABLE)} scanned WHERE scanned.id IN ({_union(tables, 'id', where)}) " - f"ORDER BY distance ASC, id ASC LIMIT {depth}) candidate)" + f"{cues}) candidate)" ) rankings.append("cue_rank") ranked = " UNION ALL ".join( @@ -860,6 +955,41 @@ def _slot_condition(namespace: tuple[str, ...], key: str) -> str: return f"namespace = {_text('/'.join(namespace))} AND key = {_text(key)}" +def _insert_sql(table: str, rows: pa.Table) -> str: + """Return the INSERT of `rows` into `table`, for the ids that the table does not hold.""" + names = rows.schema.names + types = [rows.schema.field(name).type for name in names] + values = ", ".join( + "(" + + ", ".join(_literal(row[name], kind) for name, kind in zip(names, types, strict=True)) + + ")" + for row in rows.to_pylist() + ) + columns = ", ".join(names) + ids = ", ".join(_text(value) for value in rows.column("id").to_pylist()) + return ( + f"INSERT INTO {_ref(table)} ({columns}) SELECT * FROM (VALUES {values}) AS new({columns}) " + f"WHERE new.id NOT IN (SELECT id FROM {_ref(table)} WHERE id IN ({ids}))" + ) + + +def _literal(value: Any, kind: pa.DataType) -> str: + """Return `value` as a SQL literal of the column type `kind`.""" + if value is None: + return "NULL" + if kind == _TEXTS: + if not value: + return "CAST(ARRAY[] AS VARCHAR[])" + return "ARRAY[" + ", ".join(_text(item) for item in value) + "]" + if kind == _VECTOR: + return f"CAST({_vector(value)} AS REAL[])" + if kind == _TIME: + return _timestamp(value) + if pa.types.is_integer(kind): + return str(int(value)) + return _text(str(value)) + + def _row(record: Record, vector: Sequence[float]) -> dict[str, Any]: """Return the table row of `record`, with `vector` as its content embedding.""" return { diff --git a/tests/test_hotdata.py b/tests/test_hotdata.py index 48a2524..36acaf5 100644 --- a/tests/test_hotdata.py +++ b/tests/test_hotdata.py @@ -8,7 +8,9 @@ from datetime import UTC, datetime, timedelta from typing import Any +import pyarrow as pa import pytest +from hotdata.api.indexes_api import IndexesApi from hotdata_framework import HotdataClient from hotmemory import Filter @@ -22,8 +24,10 @@ HotdataStore, LayoutError, _bm25_text, + _insert_sql, _record, _ref, + _row, _text, ) @@ -369,6 +373,83 @@ def test_a_flush_larger_than_a_slot_batch_reads_every_current_revision( assert len(store.list(NS, limit=size + 1)) == size +def test_a_second_insert_writes_nothing(client: HotdataClient, name: str) -> None: + store = provision(client, name) + store.put(NS, "disk", kind="fact", content="disk full", cues=("disk?",)) + record = store.get(NS, "disk") + assert record is not None + rows = pa.Table.from_pylist( + [_row(record, fake_embedder(["disk full"])[0])], schema=RECORD_SCHEMA + ) + result = client.execute_sql(_insert_sql(MEMORY_TABLE, rows), database=store.database) + assert result.rows[0][0] == 0 + assert [r.id for r in store.history(NS, "disk")] == [record.id] + assert store.search("disk", [NS], k=5)[0].record == record + + +def built_indexes(client: HotdataClient, store: HotdataStore) -> set[tuple[str, str]]: + api = IndexesApi(client.api) + connection = store.database.default_connection_id + return { + (table, index.index_type) + for table in (MEMORY_TABLE, EPISODE_TABLE, CUE_TABLE) + for index in api.list_indexes(connection, "public", table).indexes + } + + +def test_provision_builds_vector_indexes_only_when_asked(client: HotdataClient, name: str) -> None: + store = provision(client, name) + assert (MEMORY_TABLE, "vector") not in built_indexes(client, store) + assert (MEMORY_TABLE, "bm25") in built_indexes(client, store) + with pytest.raises(LayoutError, match="vector index"): + HotdataStore.provision( + name, + embedder=fake_embedder, + model=MODEL, + dimensions=DIMENSIONS, + client=client, + vector_index=True, + ) + + +def test_a_store_with_vector_indexes_opens_without_the_option( + client: HotdataClient, name: str +) -> None: + indexed = HotdataStore.provision( + name, + embedder=fake_embedder, + model=MODEL, + dimensions=DIMENSIONS, + client=client, + vector_index=True, + ) + found = built_indexes(client, indexed) + assert {(MEMORY_TABLE, "vector"), (EPISODE_TABLE, "vector"), (CUE_TABLE, "vector")} <= found + plain = provision(client, name) + assert plain.database.id == indexed.database.id + + +@pytest.mark.xfail( + strict=True, reason="a filtered search through the vector index misses later rows" +) +def test_a_search_through_the_vector_index_finds_every_later_write( + client: HotdataClient, name: str +) -> None: + store = HotdataStore.provision( + name, + embedder=fake_embedder, + model=MODEL, + dimensions=DIMENSIONS, + client=client, + vector_index=True, + ) + keys = [f"fact-{n}" for n in range(8)] + for key in keys: + store.put(NS, key, kind="fact", content=f"The disk fills, {key} report.") + hits = store.search("disk fills report", [NS], Filter(kind="fact"), k=10) + assert sorted(hit.record.key for hit in hits) == sorted(keys) + + def test_two_stores_write_one_database_at_once(client: HotdataClient, name: str) -> None: first = provision(client, name) second = HotdataStore.open( diff --git a/tests/test_hotdata_retry.py b/tests/test_hotdata_retry.py index 5f11f8a..789036e 100644 --- a/tests/test_hotdata_retry.py +++ b/tests/test_hotdata_retry.py @@ -1,12 +1,20 @@ """Offline tests of the load retry in `HotdataStore`, with a fake client.""" +from datetime import UTC, datetime from typing import Any import pyarrow as pa import pytest from hotdata_framework import ManagedDatabase -from hotmemory.hotdata import LOAD_ATTEMPTS, MEMORY_TABLE, HotdataStore +from hotmemory.hotdata import ( + INSERT_ROWS, + LOAD_ATTEMPTS, + MEMORY_TABLE, + HotdataStore, + _insert_sql, + _literal, +) from .conftest import DIMENSIONS, FakeClock, fake_embedder @@ -88,3 +96,85 @@ def test_another_refusal_is_not_retried(status: int, body: str) -> None: store(client, sleeps)._load(MEMORY_TABLE, ROWS, "upsert") assert len(client.loads) == 1 assert sleeps == [] + + +class FakeSqlClient(FakeClient): + """Refuses the first `sql_refusals` statements with `status` and `body`, then runs them.""" + + def __init__(self, sql_refusals: int, status: int = 409, body: str = LOCKED) -> None: + super().__init__(refusals=0) + self.sql_refusals = sql_refusals + self.sql_status = status + self.sql_body = body + self.statements: list[str] = [] + + def execute_sql(self, sql: str, database: Any = None) -> Any: + if sql.startswith("SELECT"): + return Empty() + self.statements.append(sql) + if len(self.statements) <= self.sql_refusals: + raise RuntimeError("Conflict") from Refusal(self.sql_status, self.sql_body) + return None + + +class Empty: + rows: list[Any] = [] + + def to_records(self) -> list[Any]: + return [] + + +INSERTED = pa.table({"id": ["team/disk@1", "team/cpu@1"], "count": [3, None]}) + + +def test_insert_sql_writes_only_the_ids_that_the_table_lacks() -> None: + sql = _insert_sql(MEMORY_TABLE, INSERTED) + assert sql.startswith(f"INSERT INTO default.public.{MEMORY_TABLE} (id, count) SELECT * FROM") + assert "VALUES ('team/disk@1', 3), ('team/cpu@1', NULL)" in sql + assert sql.endswith("WHERE id IN ('team/disk@1', 'team/cpu@1'))") + assert "new.id NOT IN (SELECT id FROM" in sql + + +def test_literal_of_each_column_type() -> None: + texts = pa.list_(pa.string()) + vector = pa.list_(pa.float32()) + time = pa.timestamp("us", tz="UTC") + assert _literal(["a", "it's"], texts) == "ARRAY['a', 'it''s']" + assert _literal([], texts) == "CAST(ARRAY[] AS VARCHAR[])" + assert _literal([0.5, 1.0], vector) == "CAST(ARRAY[0.5, 1.0] AS REAL[])" + when = datetime(2026, 10, 21, 8, 0, 1, 5, tzinfo=UTC) + assert _literal(when, time) == "TIMESTAMP '2026-10-21T08:00:01.000005Z'" + assert _literal(None, time) == "NULL" + assert _literal("{}", pa.string()) == "'{}'" + + +def test_a_locked_insert_retries_and_never_uploads() -> None: + client = FakeSqlClient(sql_refusals=2) + sleeps: list[float] = [] + store(client, sleeps)._load(MEMORY_TABLE, ROWS, "insert") + assert len(client.statements) == 3 + assert sleeps == [0.25, 0.5] + assert client.uploads == 0 + + +def test_a_refused_insert_falls_back_to_the_load_from_then_on() -> None: + client = FakeSqlClient(sql_refusals=1, status=400, body="Bad Request") + sleeps: list[float] = [] + target = store(client, sleeps) + target._load(MEMORY_TABLE, ROWS, "insert") + target._load(MEMORY_TABLE, ROWS, "insert") + assert len(client.statements) == 1 + assert client.loads == ["upload-1", "upload-2"] + + +def test_a_small_write_of_new_rows_inserts_and_a_large_one_loads() -> None: + client = FakeSqlClient(sql_refusals=0) + target = store(client, []) + target.put(("team",), "disk", kind="fact", content="disk full", cues=("disk?",)) + assert len(client.statements) == 2 + assert client.uploads == 0 + with target.writer(max_rows=100) as writer: + for n in range(INSERT_ROWS + 1): + writer.put(("team",), f"key-{n}", kind="fact", content=f"fact {n}") + assert len(client.statements) == 2 + assert client.uploads == 1 From a00cc4dbe85b1dbb48bde77c6cc03d04024fbb1b Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Wed, 21 Oct 2026 14:53:13 +0530 Subject: [PATCH 10/14] Phase 11: the scale harness retries a flush after a transient upload error, and compose drops the retired SQL writes setting --- bench/scale.py | 36 ++++++++++++++++++++++++++---------- compose.yaml | 1 - 2 files changed, 26 insertions(+), 11 deletions(-) diff --git a/bench/scale.py b/bench/scale.py index eb4ce8f..175567c 100644 --- a/bench/scale.py +++ b/bench/scale.py @@ -51,6 +51,8 @@ ROOT = ("scale",) TIE = 1e-5 EPISODES = Filter(kind="episode") +FLUSH_ATTEMPTS = 3 +TRANSIENT = ("upload transfer to storage failed", "Remote end closed connection") @dataclass(frozen=True) @@ -180,6 +182,7 @@ def __init__( self.questions = questions self.exact = Exact(capacity) self.flushes: list[dict[str, float]] = [] + self.retries = 0 self.reads: list[float] = [] read = getattr(self.first, "_current", None) if read is not None: @@ -199,16 +202,15 @@ def fill(self, rows: Sequence[Row]) -> None: part = rows[start : start + FLUSH] vectors = self.embedder([row.content for row in part]) started = time.perf_counter() - with self.first.writer(max_rows=len(part) + 1, interval=timedelta.max) as writer: - for row in part: - writer.put( - row.namespace, - row.key, - kind="episode", - content=row.content, - sources=(f"session:{row.session}",), - actor=ACTOR, - ) + for attempt in range(1, FLUSH_ATTEMPTS + 1): + try: + self.write(part) + break + except RuntimeError as error: + if attempt == FLUSH_ATTEMPTS or not any(t in str(error) for t in TRANSIENT): + raise + self.retries += 1 + print(f"flush failed, try {attempt + 1}: {error}", flush=True) seconds = time.perf_counter() - started read = self.reads[-1] if self.reads else 0.0 self.exact.add(part, vectors) @@ -216,6 +218,19 @@ def fill(self, rows: Sequence[Row]) -> None: self.flushes.append({"rows": len(part), "total": total, "s": seconds, "read_s": read}) print(f"wrote {total} rows, flush {seconds:.2f} s, read {read:.2f} s", flush=True) + def write(self, part: Sequence[Row]) -> None: + """Put `part` in one flush. A put of a row that is already current writes nothing.""" + with self.first.writer(max_rows=len(part) + 1, interval=timedelta.max) as writer: + for row in part: + writer.put( + row.namespace, + row.key, + kind="episode", + content=row.content, + sources=(f"session:{row.session}",), + actor=ACTOR, + ) + def checkpoint(self) -> dict[str, Any]: """Search each question in each scope by each ranking, and return the results.""" result: dict[str, Any] = {"rows": len(self.exact.rows), "scopes": {}} @@ -341,6 +356,7 @@ def main(argv: Sequence[str] | None = None) -> Path: written = size checkpoints.append(run.checkpoint()) print(json.dumps(checkpoints[-1]), flush=True) + config["flush_retries"] = run.retries write_result(path, config, run.flushes, sizes, checkpoints) finally: if not args.keep: diff --git a/compose.yaml b/compose.yaml index 5ef96ee..0533fa5 100644 --- a/compose.yaml +++ b/compose.yaml @@ -42,7 +42,6 @@ services: environment: RUNTIMEDB_SECRET_KEY: ${RUNTIMEDB_SECRET_KEY:-} RUNTIMEDB_AUTH__ALLOW_UNAUTHENTICATED: "true" - RUNTIMEDB_ENGINE__SQL_WRITES: "true" RUNTIMEDB_CATALOG__TYPE: postgres RUNTIMEDB_CATALOG__HOST: catalog RUNTIMEDB_CATALOG__PORT: "5432" From bf71046a314795018dedcee31a8e608b42eaea1a Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Thu, 22 Oct 2026 20:49:21 +0530 Subject: [PATCH 11/14] Phase 11: the sorted index on created_at becomes an option, off by default A write into a table of 188,893 rows took 8 to 20 s with the sorted index, about 1.4 s with BM25 alone, and about 0.4 s with no index, for 1 row as for 12 rows, by INSERT or by load. No query of HotdataStore needs the index. provision and open take sorted_index; a database that has the index opens without the option. --- CHANGELOG.md | 4 ++++ docs/contracts.md | 17 +++++++++---- docs/guarantees.md | 5 +++- docs/internal/architecture.md | 13 +++++----- docs/internal/brief.md | 7 ++++-- src/hotmemory/hotdata.py | 45 +++++++++++++++++++++++++---------- tests/test_hotdata.py | 31 ++++++++++++++++++++++++ 7 files changed, 95 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fd98ea..1299d4e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -60,6 +60,10 @@ each one, and its failure message points to this file. of the index. `provision` and `open` take `vector_index`, an experimental option: with it, `provision` builds the three vector indexes, and the vector rankings search through them. A filtered search through a vector index can miss rows loaded after its build. +- `HotdataStore.provision` builds no sorted index on `created_at` by default, because a + load into a table with it takes longer as the table grows. `provision` and `open` take + `sorted_index`: with it, `provision` builds the sorted index in each record table. + Databases provisioned before this change keep the index and open as before. - `HotdataStore` writes a table by SQL `INSERT` when its rows in the write are all new and number at most 5, as in a `put` of a new fact. The `INSERT` writes only the ids that the table does not hold. Other writes stay keyed loads. If the engine refuses the `INSERT`, diff --git a/docs/contracts.md b/docs/contracts.md index 0e4bf1a..2dee18e 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -236,9 +236,17 @@ timestamp in UTC to the microsecond. The embedding model and the vector size are each database. If an embedder returns a vector of another size, the write or search raises `ValueError`, and a write sends nothing. -`provision` builds two indexes in each record table: BM25 on `content`, and sorted on -`created_at`. The engine refuses an index on an empty table. So `provision` loads one seed -row into each table, builds the indexes, and deletes the seed rows. +`provision` builds one index in each record table: BM25 on `content`. The engine refuses +an index on an empty table. So `provision` loads one seed row into each table, builds the +indexes, and deletes the seed rows. + +`sorted_index=True`, in `provision` and `open`, also builds a sorted index on `created_at` +in each record table. No query of `HotdataStore` needs it. Each load into a table with it +takes longer as the table grows: a load of 12 rows into a table of 188,893 episodes took +7.9 to 20.1 s with it, 1.2 to 2.1 s with BM25 alone, and 0.1 to 0.2 s with no index, on +the local stack. [measured, 2026-10-21] In the cloud, M5 found that the sorted index cut a +newest-first filter scan at 100,000 rows from 695 ms to 407 ms. [measured 2026-10-05] A +database that has the sorted indexes opens without the option too. `vector_index=True`, in `provision` and `open`, is experimental. With it, `provision` also builds a plain cosine vector index on `content_embedding` in each record table, and on @@ -307,8 +315,7 @@ index saves engine time from about ten thousand rows. In the cloud, one request 400 ms at every size. That cost hides the saving up to at least one hundred thousand rows. [measured 2026-10-05] The fused query costs more than that. On LongMemEval-S in the cloud, a fused search took 998 ms at p50, and a vector search took 361 ms. About 50 rows were in -scope. [measured 2026-10-10, [docs/benchmarks.md](benchmarks.md)] Whether the sorted index on `created_at` serves `list` and `sweep` -was not observed. +scope. [measured 2026-10-10, [docs/benchmarks.md](benchmarks.md)] ## The memory contract diff --git a/docs/guarantees.md b/docs/guarantees.md index 6d905f8..79ae1e1 100644 --- a/docs/guarantees.md +++ b/docs/guarantees.md @@ -214,7 +214,10 @@ dominates. These results lead to four conclusions. - The only clear gain is the sorted index on `created_at`. It cut the newest-first filter - scan at one hundred thousand rows from 695 ms to 407 ms. + scan at one hundred thousand rows from 695 ms to 407 ms. Phase 11 found that a load into + a table with this index takes longer as the table grows: 7.9 to 20.1 s for 12 rows into + 188,893 rows, locally. So `provision` builds it only with `sorted_index=True`. + [measured, local, 2026-10-21] - The vector index gave no gain at any size. A scan of one hundred thousand vectors of 64 dimensions is already inside the 400 ms floor. - `bm25_search` refuses to run without a BM25 index, so the fused query cannot run without diff --git a/docs/internal/architecture.md b/docs/internal/architecture.md index 057c789..17ab08c 100644 --- a/docs/internal/architecture.md +++ b/docs/internal/architecture.md @@ -66,7 +66,7 @@ token. [local.md](../local.md) describes the local stack. | Concurrent writes | One lock in each store, and a retry on `409 RESOURCE_LOCKED` | Locks a table for each load, and refuses a second load with 409. [measured] | | Expiry | Hides a record past `forget_after` in SQL, and deletes it in `sweep` | Deletes nothing. [measured] | | Rows on disk | Builds each batch of rows as a parquet file | Stores the rows as parquet files and records them in DuckLake metadata. [observed, local] | -| Indexes | Asks for four indexes at provision, or seven with `vector_index=True` | Builds and stores them. [observed, local] | +| Indexes | Asks for two BM25 indexes at provision, two more sorted indexes with `sorted_index=True`, and three vector indexes with `vector_index=True` | Builds and stores them. [observed, local] | | Queries | Writes every SQL query, with values as literals | Plans and runs the SQL. | ## Where the data lives @@ -88,7 +88,7 @@ erDiagram string kind "fact, profile, or procedure" string content "BM25 index" list_float content_embedding "vector index, cosine, with vector_index" - timestamp created_at "sorted index" + timestamp created_at "sorted index, with sorted_index" string superseded_by timestamp valid_from timestamp valid_until @@ -100,7 +100,7 @@ erDiagram string kind "episode" string content "BM25 index" list_float content_embedding "vector index, cosine, with vector_index" - timestamp created_at "sorted index" + timestamp created_at "sorted index, with sorted_index" } cue_v1 { string id PK "the id of a record row" @@ -121,7 +121,7 @@ UTC to the microsecond. | Table | Rows | Indexes | |---|---|---| -| `memory_v1` | The records of kind `fact`, `profile`, and `procedure` | BM25 on `content`, sorted on `created_at`, and with `vector_index=True` plain cosine vector on `content_embedding` | +| `memory_v1` | The records of kind `fact`, `profile`, and `procedure` | BM25 on `content`, with `sorted_index=True` sorted on `created_at`, and with `vector_index=True` plain cosine vector on `content_embedding` | | `episode_v1` | The records of kind `episode` | The same | | `cue_v1` | One row for each record that has cues: its id and the vector of its cues | None, or with `vector_index=True` plain cosine vector on `cues_embedding` | | `meta_v1` | One row: the schema version, the embedding model name, and the vector size | None | @@ -163,7 +163,7 @@ flowchart TD same -- "no" --> bad["Raise LayoutError"] count -- "none" --> create["Create the database
with four tables keyed by id"] create --> seed["Load one seed row into
memory_v1, episode_v1, and cue_v1"] - seed --> index["Build the four indexes,
or seven with vector_index"] + seed --> index["Build the two BM25 indexes,
and the optional ones"] index --> unseed["Delete the seed rows"] unseed --> meta["Load the meta row into meta_v1"] meta --> open @@ -199,7 +199,7 @@ sequenceDiagram F->>S: PUT the parquet file F->>API: finish the upload, then load it end - loop four indexes, or seven with vector_index + loop two BM25 indexes, and the sorted and vector indexes when asked P->>F: create_index(table, column, type, metric) F->>API: submit the index build F->>API: poll every 0.2 s until it is done @@ -690,7 +690,6 @@ These claims on this page come from reading source, or were not measured: - The order in which the engine writes parquet files and records them in the catalog. - Whether each load writes exactly one parquet file. - Where the cloud keeps the catalog, the parquet files, and the index files. -- Whether the sorted index on `created_at` serves `list` and `sweep`. - The cost of a whole document `load` in the cloud. - `provision` in a workspace with more than 100 databases. An offline test proves the paging against a fake listing. diff --git a/docs/internal/brief.md b/docs/internal/brief.md index f2d46fd..9c9bf72 100644 --- a/docs/internal/brief.md +++ b/docs/internal/brief.md @@ -236,7 +236,9 @@ Retrieval runs in one SQL query over the `memory` table, in three stages. a provider-backed vector ranking alone, not both. `bm25_search` ranks the whole table and the filters of stage 1 apply after it, so the BM25 fetch depth must be wide enough to survive the filters. -3. A sorted index on `created_at` for recency ordering and for the sweeper. +3. A sorted index on `created_at` for recency ordering and for the sweeper. Since phase 11 + it is an option, `sorted_index=True`, because a write into a table of 188,893 rows took + 8 to 20 s with it and about 1.4 s without it. [measured, local, 2026-10-22] `cues` is the part that is not in any surveyed system and is cheap here. A cue is the question a record answers, written at capture time by the extractor or by the caller. A @@ -253,7 +255,8 @@ request costs about 400 ms at every size, and that floor hides the saving up to one hundred thousand rows. Only the sorted index on `created_at` showed a cloud gain, 695 ms to 407 ms at one hundred thousand rows. The driver therefore builds the BM25 index whatever the size, because `bm25_search` needs it, and treats the vector and sorted indexes as an -optimisation that matters from about ten thousand rows. [measured, M5] +optimisation that matters from about ten thousand rows. [measured, M5] Phase 11 found the +cost of that optimisation on writes, so both are options, off by default. ## 4. The memory contract diff --git a/src/hotmemory/hotdata.py b/src/hotmemory/hotdata.py index cd29452..e32b364 100644 --- a/src/hotmemory/hotdata.py +++ b/src/hotmemory/hotdata.py @@ -130,10 +130,8 @@ class _Index: metric: VectorMetric | None = None -INDEXES = ( - *(_Index(table, "bm25", "content") for table in RECORD_TABLES), - *(_Index(table, "sorted", "created_at") for table in RECORD_TABLES), -) +INDEXES = tuple(_Index(table, "bm25", "content") for table in RECORD_TABLES) +SORTED_INDEXES = tuple(_Index(table, "sorted", "created_at") for table in RECORD_TABLES) VECTOR_INDEXES = ( *(_Index(table, "vector", "content_embedding", "cosine") for table in RECORD_TABLES), _Index(CUE_TABLE, "vector", "cues_embedding", "cosine"), @@ -165,6 +163,9 @@ class HotdataStore: vector indexes, with the filters inside the index search, and do not scan. The engine can miss rows loaded after the index build in such a search, so a search can miss recent writes. The default scans every row in scope, which is exact. + + `sorted_index=True` in `provision` and `open` adds a sorted index on `created_at` in + each record table. A load into a table with it takes longer as the table grows. """ def __init__( @@ -204,6 +205,7 @@ def provision( client: HotdataClient | None = None, clock: Clock = utc_now, vector_index: bool = False, + sorted_index: bool = False, ) -> HotdataStore: """Open the database named `name`, or create it if no database has that name. @@ -218,6 +220,9 @@ def provision( three vector indexes, and the store searches through them. A database found without them raises LayoutError. Each vector index makes each write slower. + With `sorted_index=True`, a new database also gets a sorted index on `created_at` in + each record table, and a database found without it raises LayoutError. + `client` defaults to `HotdataClient.from_env()`. """ _check_settings(model, dimensions) @@ -228,9 +233,9 @@ def provision( raise LayoutError(f"{len(found)} databases are named {name!r}: {ids}") if found: database = found[0] - _check_layout(client, database, model, dimensions, vector_index) + _check_layout(client, database, model, dimensions, vector_index, sorted_index) else: - database = _create(client, name, model, dimensions, vector_index) + database = _create(client, name, model, dimensions, vector_index, sorted_index) return cls( client, database, @@ -252,6 +257,7 @@ def open( client: HotdataClient | None = None, clock: Clock = utc_now, vector_index: bool = False, + sorted_index: bool = False, ) -> HotdataStore: """Open the database with the id `database_id`, after the checks of `provision`. @@ -262,7 +268,7 @@ def open( database = client.resolve_managed_database(database_id) if database.id != database_id: raise KeyError(f"no database has the id {database_id!r}") - _check_layout(client, database, model, dimensions, vector_index) + _check_layout(client, database, model, dimensions, vector_index, sorted_index) return cls( client, database, @@ -652,7 +658,12 @@ def _named(client: HotdataClient, name: str) -> builtins.list[ManagedDatabase]: def _create( - client: HotdataClient, name: str, model: str, dimensions: int, vector_index: bool + client: HotdataClient, + name: str, + model: str, + dimensions: int, + vector_index: bool, + sorted_index: bool = False, ) -> ManagedDatabase: """Create the database, load a seed row into each table, index it, and remove the seeds. @@ -666,7 +677,7 @@ def _create( for table in (*RECORD_TABLES, CUE_TABLE): seed = files.write(_seed_rows(table, dimensions)) client.load_managed_table(database, table, schema=SCHEMA, file=seed, mode="replace") - for index in (*INDEXES, *(VECTOR_INDEXES if vector_index else ())): + for index in _wanted(vector_index, sorted_index): client.create_index( database, index.table, @@ -712,11 +723,12 @@ def _check_layout( model: str, dimensions: int, vector_index: bool = False, + sorted_index: bool = False, ) -> None: """Raise LayoutError unless `database` has the tables, columns, indexes, and meta row. - The vector indexes are checked only with `vector_index`. A database that has them - opens without it too. + The vector indexes are checked only with `vector_index`, and the sorted indexes only + with `sorted_index`. A database that has them opens without the option too. """ columns = { info.table: [column.name for column in info.columns or []] @@ -740,7 +752,7 @@ def _check_layout( for table in (*RECORD_TABLES, CUE_TABLE) for index in indexes.list_indexes(database.default_connection_id, SCHEMA, table).indexes } - for wanted in (*INDEXES, *(VECTOR_INDEXES if vector_index else ())): + for wanted in _wanted(vector_index, sorted_index): if (wanted.table, wanted.index_type, (wanted.column,), wanted.metric) not in built: raise LayoutError( f"table {wanted.table} of database {database.id} has no " @@ -761,6 +773,15 @@ def _check_layout( ) +def _wanted(vector_index: bool, sorted_index: bool) -> tuple[_Index, ...]: + """Return the indexes of a database with the options `vector_index` and `sorted_index`.""" + return ( + *INDEXES, + *(SORTED_INDEXES if sorted_index else ()), + *(VECTOR_INDEXES if vector_index else ()), + ) + + def _table(kind: Kind) -> str: """Return the table that holds records of `kind`.""" return EPISODE_TABLE if kind == "episode" else MEMORY_TABLE diff --git a/tests/test_hotdata.py b/tests/test_hotdata.py index 36acaf5..71eb4f4 100644 --- a/tests/test_hotdata.py +++ b/tests/test_hotdata.py @@ -412,6 +412,37 @@ def test_provision_builds_vector_indexes_only_when_asked(client: HotdataClient, ) +def test_provision_builds_sorted_indexes_only_when_asked(client: HotdataClient, name: str) -> None: + store = provision(client, name) + assert (MEMORY_TABLE, "sorted") not in built_indexes(client, store) + assert (EPISODE_TABLE, "bm25") in built_indexes(client, store) + with pytest.raises(LayoutError, match="sorted index"): + HotdataStore.provision( + name, + embedder=fake_embedder, + model=MODEL, + dimensions=DIMENSIONS, + client=client, + sorted_index=True, + ) + + +def test_a_store_with_sorted_indexes_opens_without_the_option( + client: HotdataClient, name: str +) -> None: + indexed = HotdataStore.provision( + name, + embedder=fake_embedder, + model=MODEL, + dimensions=DIMENSIONS, + client=client, + sorted_index=True, + ) + assert {(MEMORY_TABLE, "sorted"), (EPISODE_TABLE, "sorted")} <= built_indexes(client, indexed) + plain = provision(client, name) + assert plain.database.id == indexed.database.id + + def test_a_store_with_vector_indexes_opens_without_the_option( client: HotdataClient, name: str ) -> None: From 987e2b4df27c4455cfd1a052b8149c63ff19fb1c Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Thu, 22 Oct 2026 20:49:25 +0530 Subject: [PATCH 12/14] Phase 11: the prefilled store, the full benchmark, and the diagnosis of a background run bench/scale.py --prefill fills background/lme_s with the chunks of the sessions outside samples A and B. bench/e2e.py --background runs each question in that database, recalls over the question and the background, and records the time of each write. --sample all runs every question. The result file records chunk_chars, and bench/diagnose.py rebuilds the block of a background run, checks its SHA-256, and counts the episodes kept by as_of. --- bench/README.md | 14 +++-- bench/diagnose.py | 128 ++++++++++++++++++++++++++++++++++++++++++---- bench/e2e.py | 124 ++++++++++++++++++++++++++++++++++++++++++-- bench/scale.py | 88 +++++++++++++++++++++++++++++-- 4 files changed, 330 insertions(+), 24 deletions(-) diff --git a/bench/README.md b/bench/README.md index 8cca2f1..ab14624 100644 --- a/bench/README.md +++ b/bench/README.md @@ -50,7 +50,8 @@ sessions of each question. Git ignores `bench/results/`, and ## The end-to-end score -This command runs three arms on a sample of 54 questions, 9 of each question type: +This command runs three arms on a sample of 54 questions, 9 of each question type. With +`--sample all`, it runs every question of the data file: ```sh uv run --group bench --env-file .env python -m bench.e2e @@ -81,7 +82,7 @@ The options are these: | Option | Values | Default | |---|---|---| -| `--sample` | `a` or `b` | `a` | +| `--sample` | `a`, `b`, or `all` for every question of the data file | `a` | | `--arms` | a comma-separated list of `full`, `retrieval`, and `memory` | every arm | | `--retrieval-ranking` | `vector` or `fused`, the ranking of the retrieval arm | `vector` | | `--memory-ranking` | `vector` or `fused`, the ranking of `capture` and `recall` | `vector` | @@ -97,6 +98,7 @@ The options are these: | `--keep` | keep the memory of each question in a local database of its own; needs `--arms memory` | off | | `--reuse` | recall from the kept memory, and skip `capture`; needs `--arms memory` | off | | `--smoke` | run the 18 questions of the smoke sample on `MemoryStore`, one store in each worker | off | +| `--background` | a local database that `python -m bench.scale --prefill` filled: run every question in it, and recall over the namespace of the question and `background/lme_s`; needs `--arms memory` | off | | `--resume` | run only the questions that the rows file of an earlier run does not hold | off | | `--tag` | a suffix for the name of the result file | none | | `--limit` | the first N questions of the sample | every question | @@ -121,7 +123,7 @@ the block shows the sources as one range, `session--0001…@1`. The holds no run id, so a second run with another `--episodes` or `--recall-budget` replays `capture` from the cache. The result file lists the episode keys of each block in `episodes_in_block`, and the SHA-256 of each block in `block_sha256`. The result file is -`bench/results/e2e-sample-[-].json`, or `bench/results/e2e-smoke[-].json` +`bench/results/e2e-sample-[-].json`, or `bench/results/e2e-smoke[-].json` for a smoke run, and it records each option. ## Faster runs @@ -205,7 +207,11 @@ The command runs `capture` again with the ranking, the `k`, the `budget`, and th variant that the result file records. The replay calls no model, and it stops at the first reply that the cache does not hold. If the result file names kept databases, from a run with `--keep` or `--reuse`, the command reads the memory of each question from its kept -database and does not run `capture`. GPT-4o marks the stored facts that hold each answer. +database and does not run `capture`. If the result file names a background database, from +a run with `--background`, the command captures again in that database, recalls over the +background as the run did, and records whether the block has the SHA-256 of the run. +`--ids` limits the diagnosis to the listed misses. A result file written before the runner +recorded `chunk_chars` needs `--chunk-chars`. GPT-4o marks the stored facts that hold each answer. Each miss then gets one of four causes: | Cause | Meaning | diff --git a/bench/diagnose.py b/bench/diagnose.py index 199a06a..e09248e 100644 --- a/bench/diagnose.py +++ b/bench/diagnose.py @@ -12,22 +12,31 @@ If the result file names kept databases (a run with `--keep` or `--reuse`), this reads the memory of each question from its kept database and does not run `capture`. +If the result file names a background database (a run with `--background`), this runs +`capture` again in that database and recalls over the namespace of the question and +`BACKGROUND`, as the run did. The block is the block of that recall, and each row records +whether its SHA-256 equals the one of the run, and how many episodes the search found, kept +after `as_of`, and showed, from the question and from the background. `--ids` limits the +diagnosis to the listed misses. + uv run --group bench --env-file .env python -m bench.diagnose --result """ from __future__ import annotations import argparse +import hashlib import json import uuid from collections import Counter from collections.abc import Sequence from datetime import UTC, datetime from pathlib import Path -from typing import Any +from typing import Any, cast from bench.data import DATA_DIR, DATA_FILE, QUESTION_TYPES, Question, load from bench.e2e import ( + BACKGROUND, BASELINE, KEPT_RUN, RECALL_BUDGET, @@ -44,7 +53,7 @@ from bench.models import JUDGE, CachedModels, Offline, Remote from bench.retrieval import RANKINGS, RESULTS_DIR, _commit from hotmemory import Filter, Memory, Record, Store -from hotmemory.contract import RECALLED_KINDS, fit_lines, is_valid_at, render_line +from hotmemory.contract import RECALLED_KINDS, Episodes, fit_lines, is_valid_at, render_line NOT_EXTRACTED = "not extracted" BELOW_K = "extracted, ranked below k" @@ -115,16 +124,22 @@ def verdict(holding: set[str], best_rank: int | None, block: set[str], k: int = return BELOW_K -def memory_config(result: dict[str, Any]) -> Config: +def memory_config(result: dict[str, Any], chunk_chars: int | None = None) -> Config: """Return the configuration of the memory arm that a result file records. A result file of phase 5 records no `memory` section, and its run used `BASELINE`. A result file of phase 6 records no `fact_sources`, and its facts carried no source. A - result file before phase 8 records no `session_episodes` and no `episodes`. + result file before phase 8 records no `session_episodes` and no `episodes`. A result + file before phase 11 records no `chunk_chars`, and `chunk_chars` gives it. Raises + ValueError if the run stored chunks and neither gives their size. """ found = result["config"].get("memory") if found is None: return BASELINE + size = found.get("chunk_chars", chunk_chars) + described = result["config"].get("arms", {}).get("memory", "") + if size is None and "in chunks of" in described: + raise ValueError("the run stored chunks of a size that the file does not record") return Config( memory_ranking=found["ranking"], k=found["k"], @@ -133,7 +148,7 @@ def memory_config(result: dict[str, Any]) -> Config: fact_sources=found.get("fact_sources", False), session_episodes=found.get("session_episodes", False), episodes=found.get("episodes", "none"), - chunk_chars=found.get("chunk_chars"), + chunk_chars=size, ) @@ -186,12 +201,26 @@ def main(argv: Sequence[str] | None = None) -> Path: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--result", type=Path, default=RESULTS_DIR / "e2e-sample-a.json") parser.add_argument("--url", default="http://localhost:3000") + parser.add_argument("--ids", default="", help="comma-separated misses; all if empty") + parser.add_argument( + "--chunk-chars", type=int, default=None, help="the chunk size of a run that lacks it" + ) args = parser.parse_args(argv) commit = _commit() result = json.loads(args.result.read_text()) - config = memory_config(result) + try: + config = memory_config(result, args.chunk_chars) + except ValueError as error: + parser.error(f"{error}: pass --chunk-chars") missed = [row["id"] for row in result["questions"] if not row["arms"]["memory"]["correct"]] + if args.ids: + wanted = [question_id for question_id in args.ids.split(",") if question_id] + unknown = sorted(set(wanted) - set(missed)) + if unknown: + parser.error(f"not a miss of the result file: {', '.join(unknown)}") + missed = [question_id for question_id in missed if question_id in wanted] + saved = {row["id"]: row["arms"]["memory"].get("block_sha256") for row in result["questions"]} in_block = { row["id"]: row["arms"]["memory"].get("episodes_in_block", []) for row in result["questions"] } @@ -199,10 +228,20 @@ def main(argv: Sequence[str] | None = None) -> Path: replay = CachedModels(Offline(), CACHE_DIR / "replies.sqlite") marker = CachedModels(Remote(), CACHE_DIR / "replies.sqlite") kept: str | None = result["config"].get("memory", {}).get("kept") + background: dict[str, str] | None = result["config"].get("memory", {}).get("background") rows: list[dict[str, Any]] = [] - def diagnosed(question: Question, stores: dict[str, Store], namespace: tuple[str, ...]) -> None: - rows.append(diagnose(question, stores, namespace, config, marker, in_block[question.id])) + def diagnosed( + question: Question, + stores: dict[str, Store], + namespace: tuple[str, ...], + recalled: Sequence[Record] | None = None, + extra: dict[str, Any] | None = None, + ) -> None: + rows.append( + diagnose(question, stores, namespace, config, marker, in_block[question.id], recalled) + | (extra or {}) + ) print( f"{len(rows)}/{len(missed)} {question.id} {group(rows[-1])} {rows[-1]['best_rank']}", flush=True, @@ -218,6 +257,36 @@ def diagnosed(question: Question, stores: dict[str, Store], namespace: tuple[str with local_store(args.url, name, delete="never") as (stores, _): namespace: tuple[str, ...] = ("bench", "lme_s", KEPT_RUN, "memory", question_id) diagnosed(questions[question_id], stores, namespace) + elif background is not None: + if not existing_databases(args.url, [background["database"]]): + parser.error(f"the background database {background['database']} does not exist") + run_id = f"diagnose-{datetime.now(UTC):%Y%m%d%H%M%S}-{uuid.uuid4().hex[:6]}" + with local_store(args.url, background["database"], delete="never") as (stores, empty): + memory = Memory(stores[config.memory_ranking]) + for question_id in missed: + question = questions[question_id] + namespace = ("bench", "lme_s", run_id, "memory", question.id) + try: + for session in question.sessions: + capture_session(memory, namespace, session, replay, config, []) + records, block = memory.recall( + question.question, + [namespace, BACKGROUND], + as_of=question.as_of, + budget=config.budget, + k=config.k, + episodes=None + if config.episodes == "none" + else cast(Episodes, config.episodes), + ) + sha = hashlib.sha256(block.encode()).hexdigest() + extra = {"block_reproduced": sha == saved[question.id]} + extra |= episode_counts( + stores[config.memory_ranking], question, namespace, records, config.k + ) + diagnosed(question, stores, namespace, records, extra) + finally: + empty(namespace) else: run_id = f"diagnose-{datetime.now(UTC):%Y%m%d%H%M%S}-{uuid.uuid4().hex[:6]}" with local_store(args.url, f"hotmemory-bench-{run_id}") as (stores, empty): @@ -243,8 +312,11 @@ def diagnosed(question: Question, stores: dict[str, Store], namespace: tuple[str "fact_sources": config.fact_sources, "session_episodes": config.session_episodes, "episodes": config.episodes, + "chunk_chars": config.chunk_chars, }, "kept": kept, + "background": background, + "ids": args.ids or None, "marker": JUDGE, "hotmemory_commit": commit, "spent_usd": round(marker.spent, 4), @@ -268,10 +340,12 @@ def diagnose( config: Config, marker: CachedModels, shown: Sequence[str], + recalled: Sequence[Record] | None = None, ) -> dict[str, Any]: """Return the diagnosis of one miss, from the memory that `namespace` holds. - `shown` holds the episode keys that the block of the run held. + `shown` holds the episode keys that the block of the run held. With `recalled`, the + records of a block from `Memory.recall`, the facts of the block are those records. """ sessions = {fact_source(session.id): session.id for session in question.sessions} sessions |= {episode_key(session.id): session.id for session in question.sessions} @@ -284,8 +358,11 @@ def diagnose( ranked = _search(stores[name], question.question, namespace, len(facts)) found = [n for n, r in enumerate(ranked, start=1) if r.id in holding] ranks[name] = found[0] if found else None - top = _search(store, question.question, namespace, config.k) - block = block_ids(top, question.as_of, config.k, config.budget) + if recalled is None: + top = _search(store, question.question, namespace, config.k) + block = block_ids(top, question.as_of, config.k, config.budget) + else: + block = {record.id for record in recalled if record.kind != "episode"} cause = verdict(holding, ranks[config.memory_ranking], block, config.k) present = None if cause == RECALLED: @@ -307,6 +384,35 @@ def diagnose( } +def episode_counts( + store: Store, + question: Question, + namespace: tuple[str, ...], + recalled: Sequence[Record], + k: int, +) -> dict[str, Any]: + """Return the episodes of the search of `recall`, of those valid at `as_of`, and of the + block, each counted from the namespace of the question and from the background.""" + + def split(records: Sequence[Record]) -> dict[str, int]: + own = sum(record.namespace == namespace for record in records) + return {"own": own, "background": len(records) - own} + + found = [ + hit.record + for hit in store.search( + question.question, [namespace, BACKGROUND], Filter(kind="episode"), k=k + ) + ] + valid = [record for record in found if is_valid_at(record, question.as_of)] + shown = [record for record in recalled if record.kind == "episode"] + return { + "episodes_found": split(found), + "episodes_valid": split(valid), + "episodes_shown": split(shown), + } + + def session_of_fact(fact: Record, sessions: dict[str, str]) -> str | None: """Return the session id of `fact` from its sources, or None if no source names one. diff --git a/bench/e2e.py b/bench/e2e.py index 3ceedcb..c059035 100644 --- a/bench/e2e.py +++ b/bench/e2e.py @@ -9,6 +9,13 @@ uv run --group bench --env-file .env python -m bench.e2e --replay --arms memory uv run --group bench --env-file .env python -m bench.e2e --run --smoke --arms memory +With `--background `, every question runs in that local database, which +`python -m bench.scale --prefill` filled, and the memory arm recalls over the namespace of +the question and the namespace `BACKGROUND`. Capture still reads and writes only the +namespace of the question. Each row records the time of each write of the memory arm. + +With `--sample all`, it runs every question of the data file, in file order. + With `--smoke`, it runs the 18 questions of `bench/sample-smoke.json` on `MemoryStore`, one store in each worker. With `--keep`, it keeps the memory of each question, and `--reuse` recalls from it with no `capture`. It appends the row of each finished question to a rows @@ -75,6 +82,7 @@ COST_LIMIT = 50.0 EXTRACT_OVERHEAD = 700 ACTOR = "longmemeval-extractor" +BACKGROUND = ("background", "lme_s") Models = Callable[..., Reply] @@ -141,6 +149,8 @@ class Sample: "b": Sample("bench/sample-b.json", 20261013, excludes=("a",)), } SMOKE = Sample("bench/sample-smoke.json", 20261016) +ALL = "all" +"""The name of the run over every question of the data file, in file order.""" SMOKE_PER_TYPE = 3 @@ -304,12 +314,14 @@ def run_question( arms: Sequence[str] = ARMS, config: Config = BASELINE, capture: bool = True, + background: Sequence[str] | None = None, ) -> dict[str, Arm]: """Run each arm on `question`, and empty its namespaces after. `stores` maps each ranking to a store, and every store holds the same records. Without `capture`, the memory arm recalls from the records that its namespace holds already, - and its count of facts written and of sessions at the limit is None. + and its count of facts written and of sessions at the limit is None. With + `background`, the final recall of the memory arm also reads that namespace. """ result: dict[str, Arm] = {} if "full" in arms: @@ -338,7 +350,7 @@ def run_question( ) records, block = memory.recall( question.question, - [namespace], + [namespace] if background is None else [namespace, background], as_of=question.as_of, budget=config.budget, k=config.k, @@ -573,6 +585,7 @@ class Job: reuse: bool = False smoke: bool = False rows_file: Path | None = None + background: str | None = None @dataclass(frozen=True) @@ -657,6 +670,43 @@ def row_of(question: Question, result: dict[str, Arm]) -> dict[str, Any]: } +class WriteTimer: + """Records the seconds and the drafts of each `_write_many` call of a HotdataStore.""" + + def __init__(self, store: Store) -> None: + self.writes: list[tuple[int, float]] = [] + write = store._write_many # type: ignore[attr-defined] + + def timed(drafts: Sequence[Any]) -> list[str]: + started = time.perf_counter() + try: + ids: list[str] = write(drafts) + return ids + finally: + self.writes.append((len(drafts), time.perf_counter() - started)) + + store._write_many = timed # type: ignore[attr-defined] + + def take(self) -> dict[str, Any]: + """Return a summary of the writes since the last call, and forget them. + + The summary holds the count of writes, the count of drafts, and the p50, p95, + highest, and sum of the seconds. + """ + writes, self.writes = self.writes, [] + seconds = sorted(s for _, s in writes) + if not seconds: + return {"count": 0} + return { + "count": len(seconds), + "drafts": sum(n for n, _ in writes), + "p50_s": round(seconds[len(seconds) // 2], 3), + "p95_s": round(seconds[min(len(seconds) - 1, int(len(seconds) * 0.95))], 3), + "highest_s": round(seconds[-1], 3), + "sum_s": round(sum(seconds), 3), + } + + def run_shard(job: Job, shard: Shard) -> Done: """Run the questions of `shard` in order, over a new local database of its own. @@ -668,7 +718,9 @@ def run_shard(job: Job, shard: Shard) -> Done: opens its own model cache, embedder, and store, so it can run in its own process. It prints one line for each question, and appends the row of each finished question to `job.rows_file`. A question that raises is listed in `failed`, its namespaces are - emptied, and the next question runs. CacheMiss stops the shard. + emptied, and the next question runs. CacheMiss stops the shard. With `job.background`, + the questions run in that database, which is never deleted, and each row holds the + writes of the memory arm and the time of the question. """ from bench.models import Offline, Remote @@ -677,8 +729,12 @@ def run_shard(job: Job, shard: Shard) -> Done: rows: list[dict[str, Any]] = [] failed: list[str] = [] - def finish(question: Question, result: dict[str, Arm]) -> None: + def finish( + question: Question, result: dict[str, Arm], timing: dict[str, Any] | None = None + ) -> None: rows.append(row_of(question, result)) + if timing is not None: + rows[-1]["timing"] = timing if job.rows_file is not None: append_row(job.rows_file, rows[-1]) marks = " ".join( @@ -715,6 +771,34 @@ def empty(namespace: Sequence[str]) -> None: clean(empty, shard.run_id, question) continue finish(question, result) + elif job.background is not None: + with local_store(job.url, job.background, embedder, "never") as (stores, emptied): + timer = WriteTimer(stores[job.config.memory_ranking]) + for question in shard.questions: + started = time.perf_counter() + try: + result = run_question( + question, + models, + stores, + emptied, + shard.run_id, + job.arms, + job.config, + background=BACKGROUND, + ) + except CacheMiss: + raise + except Exception as error: + timer.take() + fail(question, error) + clean(emptied, shard.run_id, question) + continue + timing = { + "writes": timer.take(), + "question_s": round(time.perf_counter() - started, 1), + } + finish(question, result, timing) elif job.kept is not None: delete: Delete = "never" if job.reuse else "on_error" for question in shard.questions: @@ -799,7 +883,12 @@ def main(argv: Sequence[str] | None = None) -> Path | None: parser.add_argument( "--limit", type=int, default=None, help="the first N questions of the sample" ) - parser.add_argument("--sample", choices=tuple(SAMPLES), default="a", help="the sample to run") + parser.add_argument( + "--sample", + choices=(*SAMPLES, ALL), + default="a", + help="the sample to run, or all for every question", + ) parser.add_argument( "--smoke", action="store_true", @@ -855,6 +944,11 @@ def main(argv: Sequence[str] | None = None) -> Path | None: action="store_true", help="skip the questions that the rows file of an earlier run of this command holds", ) + parser.add_argument( + "--background", + default=None, + help="a local database that bench.scale --prefill filled: run in it, and recall over it", + ) parser.add_argument("--tag", default="", help="a suffix for the name of the result file") parser.add_argument("--url", default="http://localhost:3000") args = parser.parse_args(argv) @@ -867,6 +961,10 @@ def main(argv: Sequence[str] | None = None) -> Path | None: parser.error("pass --keep or --reuse, not both") if args.smoke and (args.keep or args.reuse or args.sample != "a"): parser.error("--smoke runs alone: no --keep, --reuse, or --sample") + if args.background is not None and (args.keep or args.reuse or args.smoke): + parser.error("--background runs alone: no --keep, --reuse, or --smoke") + if args.background is not None and arms != ("memory",): + parser.error("--background needs --arms memory") if (args.keep or args.reuse) and arms != ("memory",): parser.error(f"--{'keep' if args.keep else 'reuse'} needs --arms memory") try: @@ -889,6 +987,8 @@ def main(argv: Sequence[str] | None = None) -> Path | None: all_questions = {question.id: question for question in load(DATA_DIR / DATA_FILE)} if args.smoke: sample = smoke_ids(list(all_questions.values())) + elif args.sample == ALL: + sample = list(all_questions) else: sample = sample_ids(args.sample, list(all_questions.values())) questions = [all_questions[question_id] for question_id in sample][: args.limit] @@ -929,6 +1029,8 @@ def main(argv: Sequence[str] | None = None) -> Path | None: "kept": kept, "reuse": args.reuse, } + if args.background is not None: + header["background"] = args.background earlier: list[dict[str, Any]] = [] if args.resume: if not rows_file.exists(): @@ -962,6 +1064,9 @@ def main(argv: Sequence[str] | None = None) -> Path | None: print(f"deleted {len(partial)} kept databases of questions that did not finish") print(f"{'keeping' if args.keep else 'reusing'} the memory as {kept}-") + if args.background is not None and not existing_databases(args.url, [args.background]): + parser.error(f"the local database {args.background} does not exist") + commit = _commit() started = datetime.now(UTC) run_id = f"{started:%Y%m%d%H%M%S}-{uuid.uuid4().hex[:6]}" @@ -980,6 +1085,7 @@ def main(argv: Sequence[str] | None = None) -> Path | None: args.reuse, args.smoke, rows_file, + args.background, ) if len(shards) <= 1: done = [run_shard(job, shard) for shard in shards] @@ -1017,6 +1123,8 @@ def main(argv: Sequence[str] | None = None) -> Path | None: "from": SAMPLES["a"].file, } if args.smoke + else {"name": ALL, "questions": len(all_questions), "order": "data file"} + if args.sample == ALL else { "name": args.sample, "file": SAMPLES[args.sample].file, @@ -1054,8 +1162,14 @@ def main(argv: Sequence[str] | None = None) -> Path | None: "fact_sources": config.fact_sources, "session_episodes": config.session_episodes, "episodes": config.episodes, + "chunk_chars": config.chunk_chars, "kept": kept, "reused": args.reuse, + "background": ( + None + if args.background is None + else {"database": args.background, "namespace": "/".join(BACKGROUND)} + ), }, "arms": { "full": "every session of the question", diff --git a/bench/scale.py b/bench/scale.py index 175567c..776da27 100644 --- a/bench/scale.py +++ b/bench/scale.py @@ -16,6 +16,12 @@ Over the whole store, it scores `recall_any@k` against the evidence sessions. Each flush records its time and the time of the read of the current revisions before it. It writes one JSON file into `bench/results/`. + +With `--prefill`, the run fills the namespace `BACKGROUND` of `bench.e2e` instead, with +the chunks of every session that no history of samples A and B holds, each with the date +of its session as `observed_at`. It keeps the database, runs no search, and writes the +flushes into its result file. `python -m bench.e2e --background ` then recalls +over it. """ from __future__ import annotations @@ -36,7 +42,7 @@ from bench import embed, prompts from bench.data import DATA_DIR, DATA_FILE, Question, Session, load -from bench.e2e import episode_key, sample_ids +from bench.e2e import BACKGROUND, episode_key, sample_ids from bench.episodes import ACTOR, session_of from bench.metrics import score from bench.retrieval import RESULTS_DIR, _commit @@ -63,6 +69,7 @@ class Row: key: str content: str session: str + observed_at: datetime | None = None def bucket(session_id: str) -> int: @@ -106,6 +113,14 @@ def rows_of(sessions: Sequence[Session]) -> Iterator[Row]: yield Row(namespace, f"{key}-{number:04d}", chunk, session.id) +def background_rows(sessions: Sequence[Session]) -> Iterator[Row]: + """Yield the chunks of each session in order, under BACKGROUND, with the session date.""" + for session in sessions: + key = episode_key(session.id) + for number, chunk in enumerate(chunk_text(prompts.session_text(session), CHUNK_CHARS), 1): + yield Row(BACKGROUND, f"{key}-{number:04d}", chunk, session.id, session.date) + + def spread(values: Sequence[float]) -> dict[str, float]: """Return the count, p50, p95, and highest of `values`, rounded to 0.001.""" if not values: @@ -175,12 +190,14 @@ def __init__( embedder: Embedder, questions: Sequence[Question], capacity: int, + exact: bool = True, ) -> None: self.stores = stores self.first = next(iter(stores.values())) self.embedder = embedder self.questions = questions - self.exact = Exact(capacity) + self.exact = Exact(capacity) if exact else None + self.written = 0 self.flushes: list[dict[str, float]] = [] self.retries = 0 self.reads: list[float] = [] @@ -213,8 +230,10 @@ def fill(self, rows: Sequence[Row]) -> None: print(f"flush failed, try {attempt + 1}: {error}", flush=True) seconds = time.perf_counter() - started read = self.reads[-1] if self.reads else 0.0 - self.exact.add(part, vectors) - total = len(self.exact.rows) + if self.exact is not None: + self.exact.add(part, vectors) + self.written += len(part) + total = self.written self.flushes.append({"rows": len(part), "total": total, "s": seconds, "read_s": read}) print(f"wrote {total} rows, flush {seconds:.2f} s, read {read:.2f} s", flush=True) @@ -229,10 +248,13 @@ def write(self, part: Sequence[Row]) -> None: content=row.content, sources=(f"session:{row.session}",), actor=ACTOR, + observed_at=row.observed_at, ) def checkpoint(self) -> dict[str, Any]: """Search each question in each scope by each ranking, and return the results.""" + if self.exact is None: + raise RuntimeError("a run with no exact ranking has no checkpoint") result: dict[str, Any] = {"rows": len(self.exact.rows), "scopes": {}} for name, prefix in scopes().items(): index = self.exact.scope(prefix) @@ -312,7 +334,14 @@ def main(argv: Sequence[str] | None = None) -> Path: parser.add_argument( "--vector-index", action="store_true", help="provision with the experimental vector index" ) + parser.add_argument( + "--prefill", + action="store_true", + help="fill the background namespace with the sessions outside samples A and B, and keep it", + ) args = parser.parse_args(argv) + if args.prefill: + return prefill(args.location, args.url, args.vector_index, args.tag) sizes = sorted(int(size) for size in args.sizes.split(",")) questions = load(DATA_DIR / DATA_FILE) @@ -365,6 +394,57 @@ def main(argv: Sequence[str] | None = None) -> Path: return path +def prefill(location: str, url: str, vector_index: bool, tag: str) -> Path: + """Fill BACKGROUND in a new database, keep it, and return the path of the result file. + + The background holds the chunks of every session of LongMemEval-S that no history of + samples A and B holds. A run stopped with SIGINT deletes the database. + """ + questions = load(DATA_DIR / DATA_FILE) + by_id = {question.id: question for question in questions} + chosen = [by_id[i] for name in ("a", "b") for i in sample_ids(name, questions)] + held = {session.id for question in chosen for session in question.sessions} + sessions = [session for session in pool(questions, chosen) if session.id not in held] + rows = list(background_rows(sessions)) + + embedder = embed.openai_embedder() + run_id = uuid.uuid4().hex[:8] + name = f"hotmemory-prefill-{run_id}" + client, database, stores = hotdata_stores(embedder, location, url, name, vector_index) + run = Run({"vector": stores["vector"]}, embedder, [], len(rows), exact=False) + started = datetime.now(UTC) + suffix = f"-{tag}" if tag else "" + path = RESULTS_DIR / f"prefill-{location}{suffix}-{run_id}.json" + path.parent.mkdir(parents=True, exist_ok=True) + config: dict[str, Any] = { + "started": started.isoformat(), + "location": location, + "database": name, + "namespace": "/".join(BACKGROUND), + "sessions": len(sessions), + "rows": len(rows), + "left_out_sessions": len(held), + "chunk_chars": CHUNK_CHARS, + "flush": FLUSH, + "vector_index": vector_index, + "commit": _commit(), + } + print(f"filling {name} with {len(rows)} rows of {len(sessions)} sessions", flush=True) + finished = False + try: + run.fill(rows) + finished = True + finally: + config["flush_retries"] = run.retries + config["finished"] = finished + config["duration_s"] = round((datetime.now(UTC) - started).total_seconds(), 1) + write_result(path, config, run.flushes, [len(rows)], []) + if not finished: + client.delete_managed_database(database) + print(path) + return path + + def write_result( path: Path, config: dict[str, Any], From adaa611e5a660acffbd7c8b8ee0cea3f927a7a6b Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Thu, 22 Oct 2026 20:49:31 +0530 Subject: [PATCH 13/14] Phase 11: the numbers of the large store, the index costs, and LongMemEval-S in full LongMemEval-S, all 500 questions, at the phase 10 configuration: 439 of 500 (87.8%), and 88.0% on the 392 questions outside samples A and B. Samples A and B over a store with 188,893 chunks of other sessions: 39 and 35, against 48 and 46 from a fresh store, with the diagnosis of the loss. The cost of the BM25 and sorted indexes on writes. The plan and the roadmap move the large store test to LongMemEval-M. --- README.md | 10 +- docs/benchmarks.md | 242 ++++++++++++++++++++++++++++++++- docs/internal/plan.md | 284 ++++++++++++++++++++------------------- docs/internal/roadmap.md | 6 +- 4 files changed, 398 insertions(+), 144 deletions(-) diff --git a/README.md b/README.md index 0c75831..1fb8aec 100644 --- a/README.md +++ b/README.md @@ -135,13 +135,19 @@ hotmemory ran LongMemEval-S, a public benchmark of 500 questions over long chat on 2026-10-10. [docs/benchmarks.md](docs/benchmarks.md) gives every number with its configuration. +On all 500 questions, on 2026-10-22, the memory arm scored 439 of 500 (87.8%). It ran +`capture` with chunks of 1,000 characters and a `gpt-4o-mini` extractor, and `recall` with +a budget of 16,000 characters, about 4,350 tokens of reader input. The reader and the judge +were `gpt-4o-2024-08-06`. This is one run, and its 95% confidence interval is about 2.9 +points on each side. The 392 questions that no tuning used scored 88.0%. + Retrieval needs no model except the embedder. The runner stored each session of a question as an episode in `HotdataStore` and searched with the question. The 419 scored questions had these results with the vector ranking. For 98.3% of them, an evidence session was in the top 5. For 99.3% of them, every evidence session was in the top 10. -The end-to-end score ran on two samples of 54 questions, sample A on 2026-10-10 and -sample B on 2026-10-14. Later rows give their own date. The reader and the judge were +Before that run, the end-to-end score ran on two samples of 54 questions, sample A on +2026-10-10 and sample B on 2026-10-14. Later rows give their own date. The reader and the judge were `gpt-4o-2024-08-06`. | What the reader got | Sample A | Sample B | diff --git a/docs/benchmarks.md b/docs/benchmarks.md index f621fce..bbcdd12 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -2,7 +2,8 @@ This page reports the numbers of hotmemory on LongMemEval-S, a public benchmark of 500 questions over long chat histories. Each number on this page was observed on 2026-10-10, -except where a section gives another date. Each section names the result file that a run wrote into `bench/results/`, with its +except where a section gives another date. The section "The full benchmark" gives the score +on all 500 questions. Each section names the result file that a run wrote into `bench/results/`, with its numbers, its configuration, and the commit of hotmemory that ran it. Git ignores that folder, so this page is the record of the numbers. [bench/README.md](../bench/README.md) gives the commands that repeat each run and write the files again. @@ -607,11 +608,242 @@ The retrieval score with the vector ranking, rerun on the 419 questions at the s gave the same sessions in the same order as the vector ranking of the ablation, on 419 of 419 questions: `recall_all@10` 0.9928. File: `retrieval-hotdata-local-isolated-vector.json`. +## The full benchmark + +On 2026-10-22, the best configuration of phase 10 ran on all 500 questions of +LongMemEval-S, the memory arm only. Each worker had a new local database of its own, as in +phase 10. File: `e2e-sample-all-p11-full-chunks1000-16k.json`. + +| Item | Value | +|---|---| +| Data | `longmemeval_s_cleaned.json`, all 500 questions, 30 of them abstention questions | +| Capture | Each session in date order, stored as chunks of at most 1,000 characters, and the facts that the extractor writes from the whole session | +| Extractor | `gpt-4o-mini-2024-07-18`, the `detail` prompt, up to 4,000 output tokens, one call for each session | +| Embedder | `text-embedding-3-small`, 1,536 dimensions | +| Ranking | The vector ranking, an exact scan, no vector index | +| Recall | k of 30 for the facts, then `episodes="search"` with k of 30 for the chunks, and `as_of` at 23:59:59 on the date of the question | +| Budget | 16,000 characters. The block had 15,561 characters at the median. | +| Reader input | 4,346 tokens at the median and 4,588 at the 95th percentile, with the prompt | +| Reader and judge | `gpt-4o-2024-08-06`, temperature 0, the prompts of the reference code | +| Engine | The local stack, RuntimeDB image `sha256:75403e84eb47` | + +The budget is in characters, not tokens. Compare it with another system by the reader +input in tokens. + +| Question type | Questions | Correct | Accuracy | Samples A and B, phase 10 | +|---|---|---|---|---| +| single-session-user | 70 | 68 | 97.1% | 18 of 18 | +| single-session-assistant | 56 | 56 | 100.0% | 18 of 18 | +| single-session-preference | 30 | 22 | 73.3% | 13 of 18 | +| multi-session | 133 | 103 | 77.4% | 15 of 18 | +| knowledge-update | 78 | 67 | 85.9% | 14 of 18 | +| temporal-reasoning | 133 | 123 | 92.5% | 16 of 18 | +| All | 500 | 439 | 87.8% | 94 of 108 | + +The abstention questions scored 27 of 30. Phases 8 to 10 chose the configuration on +samples A and B. The 392 questions outside the two samples scored 345, 88.0%, and the 108 +questions of the samples scored 94, 87.0%. The 95% confidence interval of a score of 87.8% +on 500 questions is about 2.9 points on each side. This is one run. + +The model calls cost $14.84: $2.23 with 4 workers for the first 80 questions, and $12.61 +with 8 workers for the other 420. 77,703 texts missed the embedding cache, and their cost +was not measured. The run took 80 minutes with 4 workers and 16,189 s with 8 workers. The +run was stopped once to change the number of workers, and `--resume` kept the 80 finished +questions. 7 sessions reached the token limit of the extractor and gave no facts. + +The run used `src/hotmemory` at `a00cc4d` with the change of the sorted index of phase 11, +not yet committed. Before the run, a replay of samples A and B on new databases called no +model and gave 48 and 46 again, with the same blocks. Files: +`e2e-sample-a-p11-fresh-replay.json` and `e2e-sample-b-p11-fresh-replay.json`. + +## A large memory store + +Phase 11 measured recall from one store that holds many memories. Each number below was +observed on 2026-10-21. The local runs used the RuntimeDB image `sha256:75403e84eb47`, +built on 2026-10-19, unless a paragraph names another. The cloud run used +`api.hotdata.dev`. + +### The scale test + +`bench/scale.py` fills one store with the sessions of LongMemEval-S as chunks of 1,000 +characters, as `capture` stores them. Each session is stored once, with no copy of a row. +The pool holds 19,195 sessions and 254,025 chunks. The evidence sessions of samples A and +B come first, so every size holds them. Each session goes into one of 100 namespaces, so a +scope of one namespace holds about 1 percent of the rows, and a scope of ten about 10 +percent. The run searches the 85 scored questions of samples A and B in each scope, by each +ranking, with k of 10. Over the whole store, it scores `recall_any@k` against the evidence +sessions. So every other session of the pool is a distractor, also the sessions of other +questions. For the vector ranking, it compares each top 10 with a ranking by brute force in +numpy over the same rows. The store has no vector index, the default of phase 11. + +Local stack, file `scale-local-local-new-engine-436c7217.json`. Times are the median. + +| Rows | Flush of 1,000 rows | Vector search, 1 / 10 / 100 percent | Fused search, 1 / 10 / 100 percent | `recall_any@10`, vector / fused | +|---|---|---|---|---| +| 10,000 | 1.3 s | 37 / 42 / 45 ms | 102 / 111 / 119 ms | 0.8588 / 0.8824 | +| 50,000 | 3.4 s | 52 / 99 / 116 ms | 146 / 242 / 286 ms | 0.6588 / 0.6941 | +| 100,000 | 5.4 s | 93 / 157 / 276 ms | 216 / 418 / 504 ms | 0.5647 / 0.6000 | +| 254,025 | 14.3 s | 199 / 421 / 1,486 ms | 498 / 899 / 1,297 ms | 0.4235 / 0.4824 | + +Cloud, file `scale-cloud-cloud-no-vector-index-ec9d7e68.json`. Times are the median. + +| Rows | Flush of 1,000 rows | Vector search, 1 / 10 / 100 percent | Fused search, 1 / 10 / 100 percent | `recall_any@10`, vector / fused | +|---|---|---|---|---| +| 10,000 | 7.8 s | 342 / 341 / 353 ms | 677 / 710 / 696 ms | 0.8588 / 0.8824 | +| 50,000 | 9.9 s | 344 / 452 / 461 ms | 778 / 805 / 822 ms | 0.6588 / 0.6941 | +| 100,000 | 13.1 s | 354 / 362 / 412 ms | 722 / 841 / 809 ms | 0.5647 / 0.6000 | + +These results lead to five conclusions. + +- The vector ranking is exact at every size. In both runs, no top 10 differed from the + brute-force ranking. Some differed only in the order of rows with the same distance: 12 + of 85 at 50,000 rows, 19 at 100,000, and 36 at 254,025, over the whole store. +- Recall falls as the store grows, because more sessions compete with the evidence. Over + the whole store, `recall_any@10` of the vector ranking went from 0.8588 at 10,000 rows + to 0.4235 at 254,025. The cloud and the local stack gave the same scores. +- The fused ranking keeps more recall at each size above 10,000 rows. At 254,025 rows it + scored 0.4824, against 0.4235 for the vector ranking. On the small histories of + LongMemEval-S, the vector ranking scored higher. +- A scoped search stays fast. In a scope of 1 percent at 254,025 rows, a vector search + took 199 ms locally. A search over the whole store took 1,486 ms, with a p95 of + 2,396 ms. In the cloud, each search stayed near the cost of one request up to 100,000 + rows. +- A flush gets slower as the table grows. The read of the current revisions before each + flush took 0.2 s at 10,000 rows and 1.4 s at 254,025 rows. The rest of the flush grew + too. The cause was not measured. + +The cloud run retried 9 flushes after a transient upload failure, `the upload transfer to +storage failed before any response`. No flush failed three times. + +On the local image of 2026-10-08, the engine returned fewer than k rows for some searches +over 50,000 rows or more, when the query selected `content`. For one question at 100,000 +rows, `LIMIT 10` returned 7 rows, and a query of `id` and the distance alone returned 10. +The image of 2026-10-19 and the cloud did not show it. + +### The cost of a vector index on writes + +A probe loaded 1,000 rows of 1,536 dimensions at a time into one table, and timed the load +call alone, without the upload. Local is the image of 2026-10-08. + +| Index on the table | Local, table of 10,000 rows | Cloud, table of 10,000 rows | +|---|---|---| +| None | 0.1 to 0.2 s | 0.8 to 1.3 s | +| BM25 | 0.2 to 0.3 s | not measured | +| Plain cosine vector | 6.4 to 7.7 s | 8.2 to 8.9 s | + +Upsert and append loads cost the same. No query of `HotdataStore` reads the vector index, +because a filtered search through it misses rows loaded after its build. So phase 11 made +the vector indexes an experimental option, `vector_index=True`, off by default. + +### The cost of the BM25 and sorted indexes on writes + +On 2026-10-22, a probe filled four tables with the same 188,893 chunks of 1,000 characters +and built a different set of indexes on each. Then it wrote 1, 5, and 12 new rows into +each table by SQL `INSERT`, by upsert load, and by append load, twice for each size. The +time is that of the `INSERT` statement or of the load call. File: +`probe-insert-local-task6.json`. + +| Index on the table | `INSERT` | Upsert load | Append load | Full build of the index | +|---|---|---|---|---| +| None | 0.38 s | 0.38 s | 0.33 s | | +| BM25 on `content` | 1.43 s | 1.36 s | 1.42 s | 3.3 s | +| Sorted on `created_at` | 11.8 s | 14.1 s | 12.0 s | 22.2 s | +| BM25 and sorted | 11.7 s | 13.1 s | 12.0 s | 2.9 s and 17.8 s | + +The times are the median over the three sizes and the two trials. A write of 1 row cost +about the same as a write of 12 rows. After each of the 72 writes, a read by id, a +newest-first query, and `bm25_search` for a word of the new rows returned every new row at +once. So the engine keeps the BM25 and sorted indexes complete after each write, and the +write pays for that by the size of the table, not by its own size. That is inferred from +the times, not observed in the engine. A first probe on 2026-10-21 gave the same result +for loads of 12 rows: 7.9 to 20.1 s with the sorted index, 1.2 to 2.1 s with BM25 alone, +and 0.11 to 0.21 s with no index. File: `probe-index-local-task6.json`. + +No query of `HotdataStore` needs the sorted index. So phase 11 made it an option, +`sorted_index=True`, off by default. Over the 188,893 chunks, a vector search, a fused +search, and a newest-first `list` of 100 records had the same times with and without the +sorted index, within the spread of three runs: 340 to 426 ms, 771 to 1,305 ms, and 61 to +89 ms at the median. + +### Retrieval on LongMemEval-M + +LongMemEval-M has the same 500 questions as LongMemEval-S, with the same evidence +sessions, and a history of 472 sessions at the median for each question. The other +sessions differ from those of LongMemEval-S. The run is the retrieval run above with the +file `longmemeval_m_cleaned.json`, SHA-256 `9d79e552...`. Each question has its own +namespace, one at a time. File: `retrieval-m-hotdata-local-isolated-fused-vector.json`. +It took 1,057 s for 419 questions. 31,832 session texts missed the embedding cache. + +| Ranking | Split | `recall_any@5` | `recall_all@5` | `recall_any@10` | `recall_all@10` | `ndcg_any@10` | +|---|---|---|---|---|---|---| +| fused | M | 0.9189 | 0.7637 | 0.9451 | 0.8616 | 0.8128 | +| vector | M | 0.9045 | 0.7589 | 0.9451 | 0.8616 | 0.7952 | +| fused | S | 0.9690 | 0.9045 | 0.9928 | 0.9570 | 0.9311 | +| vector | S | 0.9833 | 0.9475 | 1.0000 | 0.9928 | 0.9479 | + +On M, the fused ranking scored higher at the top 5 and on `ndcg_any@10`, and the two +rankings tied at the top 10. On S, the vector ranking scored higher on every metric. A +search took 116 ms fused and 43 ms by vector at the median. A write of the sessions of one +question took 2.0 s at the median. + +| Question type | Questions | fused `recall_all@10` | vector `recall_all@10` | +|---|---|---|---| +| knowledge-update | 72 | 0.986 | 0.931 | +| multi-session | 121 | 0.777 | 0.835 | +| single-session-assistant | 5 | 1.000 | 1.000 | +| single-session-preference | 30 | 0.733 | 0.833 | +| single-session-user | 64 | 1.000 | 0.938 | +| temporal-reasoning | 127 | 0.827 | 0.811 | + +### Samples A and B over a prefilled store + +This test answers samples A and B from one store that holds many other memories. A +background namespace, `background/lme_s`, held the chunks of 1,000 characters of the 14,343 +sessions of LongMemEval-S that no history of samples A and B holds: 188,893 rows, with +each session date as `observed_at`. The sessions of the questions themselves stayed out, +so the background holds no copy of their evidence. The fill took 468 s in flushes of +1,000 rows, without the sorted index. File: `prefill-local-task6-nosorted-a424b237.json`. + +Each question ran as in phase 10, in the same database: `capture` wrote only the namespace +of the question, and the final `recall` read that namespace and the background. The +namespace of the question was emptied after it. Observed on 2026-10-21 and 2026-10-22. + +| Sample | Fresh store, phase 10 | Over the prefilled store | File | +|---|---|---|---| +| A | 48 of 54 | 39 of 54 | `e2e-sample-a-p11-prefilled-chunks1000-16k.json` | +| B | 46 of 54 | 35 of 54 | `e2e-sample-b-p11-prefilled-chunks1000-16k.json` | + +The two runs cost $1.53, with each `capture` from the extractor cache. A question took +96 s on sample A and 125 s on sample B at the median, and a write 1.1 s and 1.3 s. With +the sorted index, two questions of sample A took 719 s and 704 s, and a write 9.9 s and +9.2 s at the median. + +Each history of LongMemEval is a different simulated user. So the background holds the +facts of other users about "the user", and some of them contradict the facts of a +question. 761 of the background sessions are evidence sessions of other questions. A real +store keeps each user in a scope of its own, so this test is harder than one large memory +of one user. + +The diagnosis covered the 23 questions that the prefilled store lost against phase 10, +and 2 other misses with a short block. It ran `capture` again in the same database and +recalled again, and each of the 25 blocks had the SHA-256 of its run. Files: the result +files above with `-diagnosis` before `.json`. They cost $0.16, after a first try of $0.14 +that was wrong and was repeated. + +- In 16 of the 25 questions, the top 30 chunks held no chunk of the question. In the other + 9, 1 to 4 of them did. +- In 13 questions, no fact held the answer, and the chunks of the evidence were out of the + block. In 2 more, the fact that held the answer ranked below k. +- In 10 questions, the block held facts with the answer, 6 of them with every value that + the answer needs, and the reader still gave another answer. +- In 3 questions, only 3, 4, and 7 of the top 30 chunks were valid at `as_of`, so the block + used about half of its budget. `recall` applies `as_of` after the search takes its top k. + ## Known limits -- The end-to-end numbers come from samples of 54 questions, so one question moves an arm - by 1.9 points. The memory arm differs by 14.8 points between sample A and sample B. A - run on all 500 questions waits for the owner's agreement. +- Most end-to-end numbers come from samples of 54 questions, so one question moves an arm + by 1.9 points. The memory arm differs by 14.8 points between sample A and sample B in + phase 5. Only the memory arm of phase 10 ran on all 500 questions, once. - The full-context arm gives the reader every session. The reference code keeps the last 50 sessions, so the two differ for histories longer than 50 sessions. - The memory arm tried one extractor model, `gpt-4o-mini-2024-07-18`, with two prompts. A @@ -623,5 +855,7 @@ gave the same sessions in the same order as the vector ranking of the ablation, - In the runs of phase 8, the local engine refused one fused search once, because it found a BM25 index of `memory_v1` stale against its table. A second try of the same run did not show the error. The cause was not found. +- `recall` applies `as_of` after the search takes its top k. In a store with many records + dated after `as_of`, the block can hold fewer records than the budget allows. - `recall` keeps the records that are valid at `as_of`, and a search over history does not exist yet. A question about a value before it changed can need a superseded record. diff --git a/docs/internal/plan.md b/docs/internal/plan.md index 78cdd4a..d0276b7 100644 --- a/docs/internal/plan.md +++ b/docs/internal/plan.md @@ -129,140 +129,153 @@ The owner agreed to each item on 2026-10-13. ## Phase 11. A large memory store -Goal: proof that hotmemory recalls well from a store that already holds many memories. -The owner set the aim on 2026-10-21: one store with 100,000 memories or more, where -search stays fast and exact, and where the right memory still comes back. The phase -gives numbers at 10,000, 100,000, and 1,000,000 rows: the time of a write and of a -search, the accuracy of retrieval, and the accuracy of answers from a shared store. - -A large store here means many memories in the store during a run. It does not mean only -many questions. The owner's hypothesis: hotmemory does well at 100,000 to 1,000,000 -memories now that the engine accepts SQL writes. A second hypothesis, also to test: cues -help retrieval on facts. In phase 10 the fused ranking lost to the vector ranking on -episodes, but episodes have no cues. - -What the research of 2026-10-21 found (read from source and the public datasets, not -observed unless marked): - -- The engine accepts `INSERT` and `CREATE TABLE` on managed tables, with no setting, - since 2026-10-18. It refuses `UPDATE`, `DELETE`, `MERGE`, `ON CONFLICT`, and - `RETURNING`. No work on `UPDATE` or `DELETE` has started. Whether the cloud accepts - `INSERT` is not observed. An `INSERT` goes through the same append path as a load. -- `execute_sql` of `hotdata-framework` 0.14.1 retries a dropped connection up to 3 - times, and the engine does not drop a repeated `INSERT`. So a retry can write a row - twice. -- Each write reads the current revision of each key first. With every question in one - table of 19,864 rows, a write took 1.91 s at p50, against 0.29 s with one question per - table [observed, phase 9]. -- Of the writes of `HotdataStore`, an `INSERT` can take over rows that are all first revisions, new cue rows, and the seed - and meta rows. A closed old revision, `forget`, and `sweep` still need keyed loads. -- With the vector ranking, a shared table gave the same sessions in the same order as one - question per table [observed, phase 9]. So a shared store with the vector ranking - keeps the capture cache, and one store can hold every question. -- LongMemEval-M has about 500 sessions for each question. Its file is 2.74 GB, under the - MIT license. All 500 questions of LongMemEval-S in one store give about 150,000 memory - rows. BEAM, with conversations of 1 million and 10 million tokens, is the other public - test of a large store. -- The filtered vector index defect is still open in the engine, so the vector rankings - keep their scan in this phase. - -Unit costs for the estimates, from the phase 10 result files: capture costs about -$0.0005 and 1.06 s for each session with 4 workers. The reader and the judge cost about -$0.015 for each question at a budget of 16,000. Only the run of 54 questions on -LongMemEval-S is observed. Every other estimate below is extrapolated from these. - -Tasks: - -1. SQL writes in the cloud. Send one `INSERT` to a throwaway cloud database, and run M4 - again: one hundred single-row `INSERT` statements against one hundred single-row - loads, then the same at flushes of 100 and 1,000 rows of 1,536 dimensions. Record - the size limit of a query request if one shows. Correct the claims that the cloud - refuses SQL writes in `docs/guarantees.md`, `docs/internal/architecture.md`, and - `docs/internal/brief.md`. Remove the retired `RUNTIMEDB_ENGINE__SQL_WRITES` from - `compose.yaml` and `scripts/measure_local.py`. No model cost. About 15 minutes. -2. An `INSERT` path in `HotdataStore`, only if task 1 shows a clear gain. A flush whose - rows are all first revisions, and the new cue rows, go by `INSERT`. Rows that close a - revision stay on the upsert load. The path must not write a row twice on a retry: - either send the statement with no retry, or check by id before a second try. A flush - that mixes both kinds of row sends the upsert load last, so a crash leaves no second - current revision. Update `docs/contracts.md` and `docs/guarantees.md` in the same - commit, with tests in the conformance suite and `make integration`. -3. `provision` reads every page of databases. It pages the raw API instead of - `list_managed_databases`, so a workspace with more than 100 databases does not get a - second database with one name. This works around - hotdata-dev/sdk-python-framework#89 until the framework fixes it. -4. The scale harness, a script outside the package. It builds a store from real text, - not random vectors: - - The rows: LongMemEval sessions as chunks of 1,000 characters, and the facts that - earlier runs extracted, from the reply cache. Sessions of LongMemEval-M fill the - store past the rows of LongMemEval-S. No row is a copy of another. - - The embeddings: `text-embedding-3-small` at 1,536 dimensions, through the bench - cache. - - The scopes: namespaces that hold 1, 10, and 100 percent of the rows. - - The questions: a fixed set of LongMemEval questions whose evidence sessions are in - the store. - - At each checkpoint of 10,000, 100,000, and 1,000,000 rows, the script records: - - The write rate with loads, and with `INSERT` if task 2 took it, and the time of the - read before each write. - - The search time at p50 and p95, by vector and fused, at each scope size. - - For each search, whether the top k equals a brute-force top k in Python over the - same rows. While the rankings scan, any difference is a defect. - - The `recall_any@k` against the evidence sessions. - - It runs on the local stack first, then in the cloud. Embeddings for 1,000,000 rows - cost about $2 to $5. The time of the load is not known. The script stops after - 10,000 rows, and the owner gets the extrapolated time of 100,000 and 1,000,000 rows - before it goes on. -5. Retrieval on LongMemEval-M. Score `recall_any@k` with no reader model in two forms: - one shared table with a namespace for each question, and one pool of the unique - sessions, where the sessions of other questions are distractors. The vector ranking - first, and the fused ranking beside it. About $1.60 of embeddings, and 1.5 to 2.5 - hours on the local stack. The 2.74 GB file stays out of git. -6. Answers from one shared store. The runner gains a mode that keeps every question in - one database, with a namespace for each question, and the vector ranking. - - First, sample A captures into the shared store from the cache. Each row must equal - its phase 10 row, at no model cost. If a row differs, stop and diagnose. - - Then the other questions of LongMemEval-S capture into the same store, to about - 150,000 rows. About $9 to $10 for capture, because about 108 questions are in the - cache, and $1.20 for embeddings. About 5.5 to 7 hours. - - Then samples A and B answer from the full store at the phase 10 configuration. - About $1.60 for the reader and the judge. -7. The cue test on facts. Compare the fused ranking, with cues, and the vector ranking on - facts only, with one question for each database, on samples A and B. The fused ranking - changes the known facts that the extractor reads, so part of capture pays again. Up to - about $2 for each sample, and about 45 minutes for each. -8. Record the numbers in `docs/benchmarks.md`, the measured claims in - `docs/guarantees.md` and `docs/internal/architecture.md`, and the phase in - `roadmap.md`. - -The phase spends about $20 to $25 in all, under the cost limit of $50. Each paid or long -run stops first for the owner's yes, with its estimate. - -Acceptance: - -- Each number of task 4 at 10,000 and 100,000 rows, locally and in the cloud, and at - 1,000,000 rows if the owner agrees to its time. -- Every brute-force check of task 4 passes. -- `recall_any@k` on LongMemEval-M, in both forms of task 5. -- Samples A and B answered from a store of about 150,000 rows, next to their phase 10 - scores of 48 and 46. -- The cue test answers whether the fused ranking beats the vector ranking on facts. - -Stop conditions: - -- If the shared replay of sample A differs from phase 10, stop before more capture. -- If the extrapolated load of 1,000,000 rows takes more than one night, stop and ask. -- If the cloud refuses `INSERT`, task 2 drops, and the phase goes on with loads. - -Not in this phase, under Deferred in `roadmap.md`: - -- The vector rankings on the vector index. They wait for the engine's fix of the filtered - search. When it lands, tests must prove that every write, revision, and delete is found - through the vector index and the BM25 index. +Status on 2026-10-22: open, on the branch `phase-11-large-store` from `main`. Tasks 1 to 6 +are done, and the full LongMemEval-S run is done. The numbers are in `docs/benchmarks.md`, +in the sections "The full benchmark" and "A large memory store". The work so far goes into +one pull request. No issue exists for this phase. The next task is task 7, the large store +on LongMemEval-M, in the form that the owner chose on 2026-10-22. + +Goal: proof that hotmemory keeps its accuracy when the memory layer is large. The owner set +the aim on 2026-10-21. A large store means many sensible memories in the layer, from +100,000 to 1,000,000, that recall can reach, and new memories added to it during the run. +It does not mean only many questions. The test that answers it runs samples A and B as in +phase 10, over a store that is filled first, and compares the scores with the phase 10 +scores of 48 of 54 and 46 of 54 from a fresh store. The full LongMemEval sets add variety to +the evaluation, and the scale test is the volume test of the layer. + +Two hypotheses to test. The owner's: hotmemory does well at 100,000 to 1,000,000 memories +now that the engine accepts SQL writes. The second: cues help retrieval on facts. In phase +10, the fused ranking lost to the vector ranking on episodes, which have no cues. + +### Decisions of the owner on 2026-10-21 + +- The work runs on one branch, with the doc fixes after phase 10 as its first commit. +- `provision` builds no vector index by default. `vector_index=True` is an experimental + option that builds the three vector indexes and searches through them. It stays, so the + index can come back when the engine updates an index as data arrives and fixes the + filtered search through it. Work on both is under way in the engine, by report. +- The local stack runs the latest engine image, `sha256:75403e84eb47`, built 2026-10-19. +- Task 6 changes from "capture every question into one database" to the prefilled store + below, because a namespace that recall never reads cannot change an answer. +- The numbers go into the docs now, and the commit waits for the later numbers. + +### Decisions of the owner on 2026-10-22 + +- The background of task 6 holds only chunks of sessions that no history of samples A and + B holds. The cached facts stay out, because they copy the facts of the questions. +- `provision` builds no sorted index by default. `sorted_index=True` builds it. +- LongMemEval-S runs in full, on new databases, as the comparable score. +- The large store test moves to LongMemEval-M, where each question has one consistent user. + BEAM comes later. +- The runs move to a Kubernetes namespace later, for a known and stable machine. + +### What the phase found + +Each item was observed on 2026-10-21 unless marked. + +- The cloud accepts SQL `INSERT`. One row with a vector took 531 ms by `INSERT` and + 1,682 ms by load. At 10 rows they were equal. At 100 rows the load was five times + faster. At 1,000 rows the `INSERT` was refused with `Payload Too Large`, at 32 MB of SQL. + The engine refuses `UPDATE` and `DELETE`, and no work on them has started. +- The engine applies the scope filter inside `bm25_search`, before its depth, locally and + in the cloud. So BM25 loses no rows of a narrow scope in a shared table. The plan of + phase 9 to measure that loss is void. +- A plain vector index is the main cost of a write. A load of 1,000 rows into a table of + 10,000 rows took about 8.5 s with the index and about 1 s without it, in the cloud. +- The read of the current revisions before a write once sent one condition for each key. + On a flush of 2,020 rows, the local engine stopped with a stack overflow. The read now + uses lists of at most 500 keys. +- A query result holds at most 10,000 rows. A load that is in flight when its process + dies still commits. +- The local image of 2026-10-08 returned fewer than k rows for a vector search over 50,000 + rows or more, when the query selected `content`. The image of 2026-10-19 and the cloud + did not. +- The scale test, over one pool of every LongMemEval-S session as chunks, up to 254,025 + rows: the vector ranking matched a brute-force ranking at every size. `recall_any@10` + over the whole pool fell from 0.8588 at 10,000 rows to 0.4235 at 254,025 by vector, and + to 0.4824 fused. A scoped search stayed fast: 199 ms at 1 percent of 254,025 rows. A + flush of 1,000 rows grew from 1.3 s to 14.3 s, and the cause past the read was not + measured. +- LongMemEval-M retrieval, 419 questions, each in its own namespace of about 470 sessions: + `recall_any@5` 0.9189 fused and 0.9045 by vector. The two tied at `recall_any@10`, at + 0.9451. On LongMemEval-S the vector ranking scored higher on every metric. So the fused + ranking gains as a store grows. +- Transient upload failures, `the upload transfer to storage failed before any response`, + happened about once in every 100 cloud loads. +- Observed on 2026-10-21 and 2026-10-22: the sorted index on `created_at` made each write + into a table of 188,893 rows take 8 to 20 s, for 1 row as for 12 rows. BM25 alone took + about 1.4 s, and no index about 0.4 s. SQL `INSERT` paid the same as a load. Every write + was visible at once to a read by id, a newest-first query, and `bm25_search`. So the + engine rebuilds the index after each write [inferred]. This was the cause of the growth + of a flush in the scale test. +- Samples A and B over a store with 188,893 chunks of other sessions scored 39 and 35, + against 48 and 46 from a fresh store. Other users' chunks took the places of the + question's chunks, and contradicted its facts. `recall` applies `as_of` after the top k, + so 3 blocks were half empty. +- LongMemEval-S in full: 439 of 500, 87.8%. The 392 questions outside the samples scored + 88.0%. +- One write into the episode table was refused with `RESOURCE_LOCKED` after 8 tries in + about 16 s, with one writer only. The cause was not found. + +### Tasks + +1. Done, `9e96753`. SQL writes in the cloud: `scripts/measure_local.py --only m4`, with + flushes of 1, 10, 100, and 1,000 rows. The results are in `docs/guarantees.md`, M4. +2. Done, `59ac141`. `HotdataStore` writes a table by a guarded `INSERT` when its rows in + the write are all new and number at most 5 (`INSERT_ROWS`). The `INSERT` writes only + the ids that the table lacks, so a retry writes nothing twice. A refused `INSERT` makes + the store write by load from then on. +3. Done, `129e501`. `provision` reads every page of the database list, so + hotdata-dev/sdk-python-framework#89 no longer affects it. +4. Done, `d3300cc` and later. The scale harness, `bench/scale.py`. Results: + `scale-local-local-new-engine-436c7217.json` and + `scale-cloud-cloud-no-vector-index-ec9d7e68.json` in `bench/results/`. +5. Done, `f25d7ea`. Retrieval on LongMemEval-M: `python -m bench.retrieval --data m`. The + file is in `bench/data/`, 2.74 GB, SHA-256 `9d79e552...`, and the loader keeps only the + user turns. Result: `retrieval-m-hotdata-local-isolated-fused-vector.json`. +6. Done. The prefilled store: `bench/scale.py --prefill`, `bench/e2e.py --background`, + the time of each write in each row, and `bench/diagnose.py` for a background run. The + sorted index became an option. Samples A and B scored 39 and 35, and a diagnosis of 25 + misses explains the loss. Cost $1.56 for the runs and $0.30 for the diagnosis. +7. Done. LongMemEval-S in full: `bench/e2e.py --sample all`, after a replay of samples A + and B on new databases that gave 48 and 46 at $0. 439 of 500. Cost $14.84. +8. Next. The large store on LongMemEval-M. Two arms on samples A and B, each question with + its own namespace of about 500 sessions: the chunks alone, with no extracted facts, and + the full memory arm on sample A only. The full arm on both samples costs about $46 in + extractor calls [extrapolated], too close to the cap of $50. Write the cost and time + estimates before each run. +9. The cue test on facts. Compare the fused ranking, with cues, and the vector ranking on + facts only, on samples A and B. Up to about $2 for each sample if the fused ranking + changes the capture prompt. +10. Record the numbers in `docs/benchmarks.md`, `docs/guarantees.md`, and + `docs/internal/architecture.md`, close the phase in `roadmap.md`, and commit. + +### How to run and check + +- `make local-up` starts the local stack with a random `RUNTIMEDB_SECRET_KEY`. The engine + image of 2026-10-19 refuses an empty key. To swap only the engine, run + `RUNTIMEDB_SECRET_KEY="$(openssl rand -base64 32)" docker compose up -d --no-deps runtimedb`. +- Scale: `uv run --group bench --env-file .env python -m bench.scale --sizes 10000,100000`, + with `--location cloud`, `--keep`, `--vector-index`, and `--tag`. It writes its result + file after each size, and retries a flush three times after a transient upload error. +- Run one heavy job on the local engine at a time, or the timings mix. +- Kill a run with SIGINT, so that it deletes its database. SIGKILL leaves the database. A + run started with `&` from a shell that is not interactive ignores SIGINT. Stop such a + run with SIGTERM to its process group, then delete its worker databases by name. +- Ask the owner before you stop, restart, or delete anything that a run uses. +- The local databases `agent-memory` and `local-memory` stay untouched. Each held one + memory row and one meta row before and after the engine upgrade. + +### Not in this phase + +These are under Deferred in `roadmap.md`: + +- The vector rankings on the vector index by default. They wait for the engine's fix of + the filtered search. When it lands, the expected failure + `test_a_search_through_the_vector_index_finds_every_later_write` fails, and the index can + become the default. - `UPDATE` and `DELETE` by SQL, until the engine has them. -- The comparison with other memory layers, such as MemoryBench. It needs an account and a - cost for each service, and it does not test a large store. +- The comparison with other memory layers, such as MemoryBench. - BEAM at 10 million tokens. ## Phase 12. Supersede in capture @@ -416,8 +429,9 @@ These wait, and `roadmap.md` lists them under Deferred: ## Questions for the owner -- Phase 11: the cloud workspace for the run at 1,000,000 rows, and its cost limit apart - from the model spend. +- Phase 11: the yes to build and run task 6, the prefilled store. +- Phase 11: the cloud workspace for a run at 1,000,000 rows, and its cost limit apart + from the model spend. The cloud runs so far used the workspace of `.env`. - Phase 11: a target for the search time, such as p95 in the cloud at 1,000,000 rows. - Phase 15: the first version number, and whether to publish to TestPyPI first. diff --git a/docs/internal/roadmap.md b/docs/internal/roadmap.md index 9392ebe..dc6a752 100644 --- a/docs/internal/roadmap.md +++ b/docs/internal/roadmap.md @@ -39,10 +39,10 @@ memories or more. | 8. Episodes behind facts | `capture` stores its text as an episode that its facts cite. `recall` can add the cited or ranked episodes to its block, within one budget. | closed 2026-10-15, PR #24 | | 9. Faster evaluation | The runner keeps the memory of a run and reuses it for runs that change only `recall` or the reader. A smoke run of 18 questions on `MemoryStore`. A run keeps its finished questions when it stops. No library change. | closed 2026-10-16, PR #28 | | 10. The ranking, and episode chunks | An ablation of the fused ranking, and a default ranking chosen by the benchmark. `capture` stores a session as chunks, so the block holds the chunks that match and not whole sessions, for fewer reader tokens. | closed 2026-10-19, PR #30 | -| 11. A large memory store | Recall from one store that holds 100,000 memories or more. SQL `INSERT` measured in the cloud, and taken for appends if it is faster. The time of a write and a search at 10,000, 100,000, and 1,000,000 rows built from real text, by scope size, locally and in the cloud, with each top k checked against a brute-force ranking. Retrieval on LongMemEval-M. Samples A and B answered from one store of every LongMemEval-S question. A test of cues on facts. | open | +| 11. A large memory store | Recall from one store that holds 100,000 memories or more. SQL `INSERT` measured in the cloud, and taken for appends if it is faster. The time of a write and a search at 10,000, 100,000, and 1,000,000 rows built from real text, by scope size, locally and in the cloud, with each top k checked against a brute-force ranking. Retrieval on LongMemEval-M. Samples A and B answered over a store prefilled with 100,000 or more memories, against their scores from a fresh store. LongMemEval-S in full: 439 of 500 on 2026-10-22. The large store on LongMemEval-M. A test of cues on facts. | open | | 12. Supersede in capture | A fact can name the current record that it replaces, and `remember` closes that record. The extractor decides, and the library applies the rule. | proposed | -| 13. Search over history | `recall(as_of=T)` returns the revision valid at T, with the as-of filter in SQL, and a date window from the caller. | proposed | -| 14. The full benchmark | The final configuration on all 500 LongMemEval-S questions, and a LoCoMo smoke test. | proposed | +| 13. Search over history | `recall(as_of=T)` returns the revision valid at T, with the as-of filter in SQL, and a date window from the caller. The filter in SQL also keeps a block full when many records are dated after `as_of`, as phase 11 found. | proposed | +| 14. The full benchmark | The final configuration on all 500 LongMemEval-S questions, and a LoCoMo smoke test. The LongMemEval-S run of phase 11 is the first. | proposed | | 15. Release on PyPI | The release flow of the sibling repository `hotdata-langchain`, and the first release. | proposed | ## Deferred From 4bdcdd57cbc3f573098afe3f866b9114067eac8d Mon Sep 17 00:00:00 2001 From: Rohan Dsouza Date: Thu, 22 Oct 2026 21:23:07 +0530 Subject: [PATCH 14/14] Docs: a guarded INSERT keeps the earlier row when two processes write the same new key The first revision of a new key, in a write of at most 5 new rows, goes by the guarded INSERT, which skips an id that the table holds. A later revision, or a write of more than 5 rows, goes by upsert load, where the later load wins. --- docs/contracts.md | 9 +++++++-- docs/guarantees.md | 2 +- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/contracts.md b/docs/contracts.md index 2dee18e..95b0fa5 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -270,8 +270,13 @@ and the search drops it. A writer sends one write per table at each flush. `dele A store holds a lock across the read and the load of each write, so one store writes in order. One process writes to a database. Two processes that put the same key can both read -the same current revision and both write the next one. Then the later load replaces the -earlier row. This rule stays until the engine has a conditional write. +the same current revision and both write the next one, with the same id. Which row stays +depends on how the store writes it. The first revision of a new key, in a write of at most +5 new rows, goes by the guarded `INSERT`, so the earlier write stays and the later one +writes nothing. A later revision rewrites the row of the revision before it, so it goes by +upsert load, and so does a write of more than 5 rows: then the later load replaces the +earlier row. In both cases one of the two writes is lost. This rule stays until the engine +has a conditional write. If the engine refuses a load or an `INSERT` with `409 RESOURCE_LOCKED`, the driver tries it again, up to 8 attempts in all. The wait starts at 0.25 seconds and doubles to at most diff --git a/docs/guarantees.md b/docs/guarantees.md index 79ae1e1..08d264c 100644 --- a/docs/guarantees.md +++ b/docs/guarantees.md @@ -38,7 +38,7 @@ A row that no test proves yet names the phase that will prove it. | Can a retried `INSERT` write a row twice? | No. `HotdataStore` sends an `INSERT` only for at most 5 new rows of one table, and the statement keeps only the ids that the table does not hold. A second run of the same statement wrote 0 rows. If the engine refuses the `INSERT` for a reason other than a lock, the store writes by upsert load from then on. | answered [measured, local] 2026-10-21 | `test_a_second_insert_writes_nothing`, `test_a_refused_insert_falls_back_to_the_load_from_then_on` | | Two processes write to the same table. What happens? | The engine refuses the second load with 409 `RESOURCE_LOCKED`. The driver retries up to 8 attempts, with a backoff from 0.25 seconds that doubles to at most 4 seconds, and then raises the error. Inside one process, the store sends one load at a time for each table. The integration test runs two stores in one process, which the engine cannot tell apart from two processes. | answered [measured], M3 | `test_two_stores_write_one_database_at_once`, `test_a_locked_load_retries_with_a_doubling_backoff`, `test_a_load_stops_after_the_last_attempt` | | Two writers in one process put the same key. What happens? | The last writer wins at the row level. Revisions are new rows, so both revisions exist and the later one is current. This holds inside one process only. The next row gives the rule for two processes. | answered | `test_last_writer_wins_on_one_key`, `test_one_store_serializes_writes_on_one_key` | -| Can two processes write to one database? | Not safely. Two processes that put the same key can both read the same current revision. Both then write the same next revision, and the later load replaces the earlier row, so one revision is lost. One process writes to a database. Other processes can read it. This rule stays until the engine has a conditional write. | answered | `test_one_store_serializes_writes_on_one_key` | +| Can two processes write to one database? | Not safely. Two processes that put the same key can both read the same current revision. Both then write the same next revision, with the same id, so one revision is lost. For the first revision of a new key, in a write of at most 5 new rows, the guarded `INSERT` keeps the earlier row. For a later revision, or a write of more than 5 rows, the upsert load keeps the later row. One process writes to a database. Other processes can read it. This rule stays until the engine has a conditional write. | answered | `test_one_store_serializes_writes_on_one_key` | | Does `delete` remove retained revisions and their embeddings? | Yes. The content vector is a column of its row, and `HotdataStore` deletes the cue rows of every revision with the record rows. After a keyed delete, neither a provider-backed vector index nor a BM25 index returned the deleted row. | answered [measured], M2 | `test_delete_removes_every_revision`, `test_delete_removes_rows_and_cues` | | Which filters work in `list` and `search`? | Equality on `kind`, `subject`, `tags`, and `actor`. `kind` also takes a tuple of kinds and matches any kind named. A range on `valid_from`, `valid_until`, `created_at`, and `expired_at`. A prefix on namespace labels. Any other filter raises an error. | answered | `test_filter_matches_by_equality`, `test_filter_kind_tuple_matches_any_kind_named`, `test_filter_matches_a_time_range`, `test_unknown_filter_key_raises`, `test_prefix_matches_whole_labels` | | Does a higher score mean more relevant? | The store returns a distance, and a lower distance is closer. The memory contract returns records in order, with no score. An adapter that needs a score converts the distance. | answered | `test_search_returns_distance_closest_first` |