Phase 11, part 1: a large memory store, the sorted index made optional, and LongMemEval-S in full

CHANGELOG.md

@@ -56,6 +56,24 @@ each one, and its failure message points to this file.
 - `Memory.recall` and `Memory.profile` read the records of kind `fact`, `profile`, and
   `procedure` only. They never return an episode, and the counts of `profile` leave
   episodes out.
+- `HotdataStore.provision` builds no vector index by default, so each write skips the cost
+  of the index. `provision` and `open` take `vector_index`, an experimental option: with
+  it, `provision` builds the three vector indexes, and the vector rankings search through
+  them. A filtered search through a vector index can miss rows loaded after its build.
+- `HotdataStore.provision` builds no sorted index on `created_at` by default, because a
+  load into a table with it takes longer as the table grows. `provision` and `open` take
+  `sorted_index`: with it, `provision` builds the sorted index in each record table.
+  Databases provisioned before this change keep the index and open as before.
+- `HotdataStore` writes a table by SQL `INSERT` when its rows in the write are all new and
+  number at most 5, as in a `put` of a new fact. The `INSERT` writes only the ids that the
+  table does not hold. Other writes stay keyed loads. If the engine refuses the `INSERT`,
+  the store writes by load from then on.
+- `HotdataStore` reads the current revisions before a write with lists of at most 500
+  keys. Before, one condition for each key could overflow the stack of the engine on a
+  flush of about 2,000 rows.
+- `HotdataStore.provision` reads every page of the database list. Before, it read the
+  first 100 databases, and in a larger workspace it could create a second database with
+  the name.
 - `Memory.capture` takes `episode`, a key. With it, `capture` also stores the text as an
   episode under that key, after the facts, and each fact names the episode in `sources`.
   A capture of the same text under the same key writes nothing and calls no extractor.

README.md

@@ -135,13 +135,19 @@ hotmemory ran LongMemEval-S, a public benchmark of 500 questions over long chat
 on 2026-10-10. [docs/benchmarks.md](docs/benchmarks.md) gives every number with its
 configuration.
 
+On all 500 questions, on 2026-10-22, the memory arm scored 439 of 500 (87.8%). It ran
+`capture` with chunks of 1,000 characters and a `gpt-4o-mini` extractor, and `recall` with
+a budget of 16,000 characters, about 4,350 tokens of reader input. The reader and the judge
+were `gpt-4o-2024-08-06`. This is one run, and its 95% confidence interval is about 2.9
+points on each side. The 392 questions that no tuning used scored 88.0%.
+
 Retrieval needs no model except the embedder. The runner stored each session of a question
 as an episode in `HotdataStore` and searched with the question. The 419 scored questions
 had these results with the vector ranking. For 98.3% of them, an evidence session was in
 the top 5. For 99.3% of them, every evidence session was in the top 10.
 
-The end-to-end score ran on two samples of 54 questions, sample A on 2026-10-10 and
-sample B on 2026-10-14. Later rows give their own date. The reader and the judge were
+Before that run, the end-to-end score ran on two samples of 54 questions, sample A on
+2026-10-10 and sample B on 2026-10-14. Later rows give their own date. The reader and the judge were
 `gpt-4o-2024-08-06`.
 
 | What the reader got | Sample A | Sample B |
@@ -151,13 +157,16 @@ sample B on 2026-10-14. Later rows give their own date. The reader and the judge
 | The block of `recall`, after `capture` with a `gpt-4o-mini` extractor and the `baseline` prompt | 19 of 54 (35.2%) | 27 of 54 (50.0%) |
 | The same, on 2026-10-14, with the `detail` prompt and a session source on each fact | 31 of 54 (57.4%) | 32 of 54 (59.3%) |
 | The same, on 2026-10-15, with each session stored as an episode, and `recall` with `episodes="search"` and a budget of 64,000 characters, about 12,800 tokens | 45 of 54 (83.3%) | 45 of 54 (83.3%) |
+| The same, on 2026-10-18, with each session stored as chunks of 1,000 characters (`capture` with `chunk_chars=1000`), and `recall` with `episodes="search"` and a budget of 16,000 characters, about 4,300 tokens | 48 of 54 (88.9%) | 46 of 54 (85.2%) |
 
 Facts alone score below the full history. With the `baseline` prompt, the extractor did
 not store the answer in 19 of the 35 misses on sample A, and in 16 of the 27 misses on
 sample B. With the `detail` prompt, these counts are 16 of 23 and 14 of 22. With the
-episodes that a search adds after the facts, the memory arm scores above the full history
-and equal to the 10 sessions on sample A, with less than half of their tokens. With 54
-questions, each score can be about 10 points off. With the `baseline` prompt, the memory
+whole sessions that a search adds after the facts, the extractor did not store the answer
+in 5 misses on each sample. With chunks of 1,000 characters, this count is 2 on each
+sample. The chunk row scores above the full history on both samples. It scores 3 more
+than the 10 sessions on sample A and 2 less on sample B, with about one seventh of their
+tokens. With 54 questions, each score can be about 10 points off. With the `baseline` prompt, the memory
 arm differed by 14.8 points between the two samples. In the cloud, a search took 361 ms at
 p50 with the vector ranking, and 998 ms with the fused ranking.
 

bench/README.md

@@ -37,6 +37,7 @@ The options are these:
 | `--location` | `local` or `cloud` | `local` |
 | `--url` | the address of the local engine | `http://localhost:3000` |
 | `--limit` | the first N scored questions | every scored question |
+| `--data` | `s` for LongMemEval-S, or `m` for LongMemEval-M, about 500 sessions for each question. The M file is 2.74 GB. The runner reads it one question at a time and keeps only the user turns. | `s` |
 | `--ablate` | also rank with the fused query less one arm at a time: `bm25-content`, `content-cue`, and `bm25`; needs `--driver hotdata` | off |
 
 With `--location cloud`, the runner reads the connection from `HOTDATA_API_KEY`,
@@ -49,7 +50,8 @@ sessions of each question. Git ignores `bench/results/`, and
 
 ## The end-to-end score
 
-This command runs three arms on a sample of 54 questions, 9 of each question type:
+This command runs three arms on a sample of 54 questions, 9 of each question type. With
+`--sample all`, it runs every question of the data file:
 
 ```sh
 uv run --group bench --env-file .env python -m bench.e2e
@@ -80,7 +82,7 @@ The options are these:
 
 | Option | Values | Default |
 |---|---|---|
-| `--sample` | `a` or `b` | `a` |
+| `--sample` | `a`, `b`, or `all` for every question of the data file | `a` |
 | `--arms` | a comma-separated list of `full`, `retrieval`, and `memory` | every arm |
 | `--retrieval-ranking` | `vector` or `fused`, the ranking of the retrieval arm | `vector` |
 | `--memory-ranking` | `vector` or `fused`, the ranking of `capture` and `recall` | `vector` |
@@ -96,15 +98,16 @@ The options are these:
 | `--keep` | keep the memory of each question in a local database of its own; needs `--arms memory` | off |
 | `--reuse` | recall from the kept memory, and skip `capture`; needs `--arms memory` | off |
 | `--smoke` | run the 18 questions of the smoke sample on `MemoryStore`, one store in each worker | off |
+| `--background` | a local database that `python -m bench.scale --prefill` filled: run every question in it, and recall over the namespace of the question and `background/lme_s`; needs `--arms memory` | off |
 | `--resume` | run only the questions that the rows file of an earlier run does not hold | off |
 | `--tag` | a suffix for the name of the result file | none |
 | `--limit` | the first N questions of the sample | every question |
 
 Each worker gets the questions in turn. Without `--keep` or `--reuse`, it runs them over a
 new local database of its own, and empties the namespaces of each question after it. Each
 table then holds one question at a time, as in a run with one worker. With
-`--memory-ranking fused`, BM25 ranks over the whole table before the scope filter, so the
-rows of a second question in a table can change the ranking. An extractor variant names a prompt and a model. Each fact carries
+`--memory-ranking fused`, the BM25 scores can depend on every row of the table, so the
+rows of a second question in a table can change the ranking. This was not measured. An extractor variant names a prompt and a model. Each fact carries
 the source `session:<hash>`, where the hash is the first 12 hex digits of the SHA-256 of
 the session id. The block shows the sources to the reader, and the id of an evidence
 session holds `answer`, so the source does not show the id. If an extractor reply reaches
@@ -120,7 +123,7 @@ the block shows the sources as one range, `session-<hash>-0001…<last>@1`. The
 holds no run id, so a second run with another `--episodes` or `--recall-budget` replays
 `capture` from the cache. The result file lists the episode keys of each block in
 `episodes_in_block`, and the SHA-256 of each block in `block_sha256`. The result file is
-`bench/results/e2e-sample-<a or b>[-<tag>].json`, or `bench/results/e2e-smoke[-<tag>].json`
+`bench/results/e2e-sample-<a, b, or all>[-<tag>].json`, or `bench/results/e2e-smoke[-<tag>].json`
 for a smoke run, and it records each option.
 
 ## Faster runs
@@ -136,9 +139,9 @@ uv run --group bench --env-file .env python -m bench.e2e --run --arms memory --e
 ```
 
 With `--keep`, each question runs over a new local database of its own, and the database
-stays after the run. One database holds one question. With `--memory-ranking fused`, BM25
-ranks the whole table before the namespace filter, so the memory of a second question
-would change the block. The
+stays after the run. One database holds one question. With `--memory-ranking fused`, the
+BM25 scores can depend on every row of the table, so the memory of a second question can
+change the block. This was not measured. The
 name of the database is `hotmemory-kept-<sample>-<extractor><digest>-<cites>-<ranking>-<commit>-<question id>`:
 
 | Part | What it holds |
@@ -204,7 +207,11 @@ The command runs `capture` again with the ranking, the `k`, the `budget`, and th
 variant that the result file records. The replay calls no model, and it stops at the first
 reply that the cache does not hold. If the result file names kept databases, from a run
 with `--keep` or `--reuse`, the command reads the memory of each question from its kept
-database and does not run `capture`. GPT-4o marks the stored facts that hold each answer.
+database and does not run `capture`. If the result file names a background database, from
+a run with `--background`, the command captures again in that database, recalls over the
+background as the run did, and records whether the block has the SHA-256 of the run.
+`--ids` limits the diagnosis to the listed misses. A result file written before the runner
+recorded `chunk_chars` needs `--chunk-chars`. GPT-4o marks the stored facts that hold each answer.
 Each miss then gets one of four causes:
 
 | Cause | Meaning |

bench/data.py

@@ -1,11 +1,12 @@
-"""The LongMemEval-S data: the download, its checks, and the questions in date order."""
+"""The LongMemEval data, S and M: the download, its checks, and the questions in date order."""
 
 from __future__ import annotations
 
 import hashlib
 import json
 import re
 import urllib.request
+from collections.abc import Iterator
 from dataclasses import dataclass
 from datetime import UTC, datetime, time
 from pathlib import Path
@@ -19,6 +20,9 @@
     f"{DATA_REVISION}/{DATA_FILE}"
 )
 DATA_SHA256 = "d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442"
+M_FILE = "longmemeval_m_cleaned.json"
+M_SHA256 = "9d79e5524794a2e6900a3aa9cb7d9152c5a3e8319c9a87c25494ba1eacee495f"
+DATASETS = {"s": (DATA_FILE, DATA_SHA256), "m": (M_FILE, M_SHA256)}
 DATE_FORMAT = "%Y/%m/%d (%a) %H:%M"
 END_OF_DAY = time(23, 59, 59, tzinfo=UTC)
 QUESTION_TYPES = (
@@ -176,39 +180,68 @@ def sha256(path: Path) -> str:
     return digest.hexdigest()
 
 
-def download(directory: Path = DATA_DIR) -> Path:
-    """Download the data file into `directory` unless it is there, and return its path.
+def download(directory: Path = DATA_DIR, dataset: str = "s") -> Path:
+    """Download the data file of `dataset` into `directory` unless it is there.
 
-    Raises ValueError, and keeps no file, if the download has another SHA-256.
+    Returns its path. Raises ValueError, and keeps no file, if the download has another
+    SHA-256.
     """
-    path = directory / DATA_FILE
+    data_file, expected = DATASETS[dataset]
+    url = DATA_URL.removesuffix(DATA_FILE) + data_file
+    path = directory / data_file
     if path.exists():
         return path
     directory.mkdir(parents=True, exist_ok=True)
     partial = path.with_suffix(".part")
-    with urllib.request.urlopen(DATA_URL) as response, partial.open("wb") as file:
+    with urllib.request.urlopen(url) as response, partial.open("wb") as file:
         while block := response.read(_CHUNK):
             file.write(block)
     found = sha256(partial)
-    if found != DATA_SHA256:
+    if found != expected:
         partial.unlink()
-        raise ValueError(f"{DATA_URL} has SHA-256 {found}, expected {DATA_SHA256}")
+        raise ValueError(f"{url} has SHA-256 {found}, expected {expected}")
     partial.rename(path)
     return path
 
 
-def load(path: Path, expected_sha256: str | None = DATA_SHA256) -> list[Question]:
+def load(
+    path: Path, expected_sha256: str | None = DATA_SHA256, user_only: bool = False
+) -> list[Question]:
     """Return the questions of the data file at `path`, in file order.
 
-    Raises ValueError if `expected_sha256` is not None and the file has another SHA-256,
-    or if two questions share an id.
+    With `user_only`, the file is read one question at a time, and each session keeps only
+    its user turns. Raises ValueError if `expected_sha256` is not None and the file has
+    another SHA-256, or if two questions share an id.
     """
     if expected_sha256 is not None:
         found = sha256(path)
         if found != expected_sha256:
             raise ValueError(f"{path} has SHA-256 {found}, expected {expected_sha256}")
-    questions = [parse_question(entry) for entry in json.loads(path.read_text())]
+    if user_only:
+        questions = [parse_question(_user_turns(entry)) for entry in _entries(path)]
+    else:
+        questions = [parse_question(entry) for entry in json.loads(path.read_text())]
     ids = [question.id for question in questions]
     if len(set(ids)) != len(ids):
         raise ValueError(f"{path} repeats a question id")
     return questions
+
+
+def _entries(path: Path) -> Iterator[dict[str, Any]]:
+    """Yield the entries of the JSON array in the file at `path`, one at a time."""
+    text = path.read_text()
+    decoder = json.JSONDecoder()
+    position = text.index("[") + 1
+    while True:
+        while position < len(text) and text[position] in " \t\r\n,":
+            position += 1
+        if position >= len(text) or text[position] == "]":
+            return
+        entry, position = decoder.raw_decode(text, position)
+        yield entry
+
+
+def _user_turns(entry: dict[str, Any]) -> dict[str, Any]:
+    """Return `entry` with only the user turns of each session."""
+    sessions = [[t for t in body if t["role"] == "user"] for body in entry["haystack_sessions"]]
+    return {**entry, "haystack_sessions": sessions}