Repository navigation
Phase 12A: BEAM at 1 million tokens - #45
Conversation
bench/beam.py pins the 100K and 1M files of Mohammadta/BEAM by revision and SHA-256, parses each row into dated batches and probes, strips the index markers from user turns, maps the gold field of each ability to one field, and cuts each batch into windows of whole turn pairs of at most 10,000 characters. A longer pair is cut at blank lines. On the real files it counts 1,606 windows at 100K and 21,003 at 1M.
Each sample is 5 conversations from 5 different domains, drawn with a fixed seed, and sample B leaves out the conversations of sample A. Sample A is conversations 2, 6, 12, 17, 18 (444 windows), and sample B is 1, 4, 7, 14, 19 (386 windows).
bench/beam_score.py scores a response to a BEAM probe as the official code at commit b2da22e does: one judge call for each rubric nugget with the unified judge prompt and its JSON parse, and for event ordering the line match and Kendall tau-b, reported as tau_norm. The official answer and judge prompts and the MIT license are in bench/beam_prompts/. The test runs a copy of the official functions beside ours on fixed and random replies with a fake model, and needs equal scores. The model calls can now send a system message, which the official match call uses. The cache key of a call without one is unchanged. The bench group gains scipy and json-repair.
…ersation python -m bench.beam_e2e captures each conversation once into a local database that keeps its memory, then for each probe recalls once for each arm (memory, and chunks as the RAG baseline), reads the block with each reader (gpt-4.1-nano and gpt-4.1-mini) and the official answer prompt, and scores each answer with bench/beam_score.py. Workers take the conversation with the most windows first. The reads and judge calls of a conversation run in a thread pool, with one reply cache connection for each thread. A rows file holds a capture row for each conversation and a row for each probe, so --resume skips both, and a stopped capture goes on over its kept database. --reuse reads with no capture. The runner deletes no database, returns each error as text, and stops a conversation that spends twice its estimate. Each probe row records which windows hold its source turns and the share of them that each block shows. bench/models.py gains the prices of the gpt-4.1 models and the embedder, read 2026-10-26.
The new section gives the commands, the options, the kept database name, the two arms, the official scoring, and how a run resumes.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| } | ||
| tasks = [(probe, arm, reader) for probe in probes for arm in job.arms for reader in job.readers] | ||
| cells: dict[tuple[str, str, str], dict[str, Any]] = {} | ||
| with ThreadPoolExecutor(JUDGE_THREADS) as pool: |
There was a problem hiding this comment.
nit: After SpendLimit, every queued task still makes one paid reader call before it raises. (not blocking)
Fix: in the as_completed loop, catch the exception and call pool.shutdown(cancel_futures=True) before re-raising. Another fix sets a stop flag in ThreadModels and checks the flag before each call.
Cause: the check in ThreadModels.__call__ (line 226) runs after the call returns. Leaving the with ThreadPoolExecutor block calls shutdown(wait=True), and that call runs every pending future. A conversation has up to 80 tasks (20 probes × 2 arms × 2 readers), so the spend can pass SPEND_FACTOR times the estimate.
There was a problem hiding this comment.
The full diff was omitted from the review prompt. The diff was fetched with gh pr diff, and these files were read: bench/beam.py, bench/beam_score.py, bench/beam_e2e.py, and the bench/models.py hunk. The tests, prompts, README, and uv.lock were not read line by line. CI checks were still pending when this review started.
The API no longer serves gpt-4.1-nano. Both the alias and the snapshot gpt-4.1-nano-2025-04-14 return 404 "deprecated". The runner now has one reader, gpt-4.1-mini, the model of the judge.
There was a problem hiding this comment.
The full diff was omitted from the review prompt. This cycle reviewed the diff since the last review (bfb43f4..a6e823c) and read bench/beam_e2e.py, bench/README.md, bench/models.py, bench/tests/test_beam_e2e.py and bench/prompts/judge.txt from the checkout. No reference to the removed nano reader remains. CI checks were still pending when this review started.
ThreadModels checked the limit only after a call returned, so each queued reader task made one more paid call before it raised.
There was a problem hiding this comment.
The full diff was omitted from the review prompt. This review fetched it with gh pr diff, read the diff since the last review, and read bench/beam_e2e.py lines 180-479. The pre-call check in ThreadModels.call resolves thread #4376763669: queued tasks raise SpendLimit before any paid call.
This PR adds a BEAM runner to
bench/, for phase 12A (#40). It has a loader, two samples of the 100K split, and a scorer that follows the official BEAM code. An end-to-end runner uses one kept local database for each conversation. The full runs and their numbers come in a later PR. This PR is part of #40 and does not close it.How
bench/beam.pyloads the 100K and 1M splits fromMohammadta/BEAMat a pinned revision, and checks the SHA-256 of each file. It cuts each batch into windows of whole turn pairs, at most 10,000 characters each. It also draws samples A and B (bench/sample-beam-a.json,bench/sample-beam-b.json).bench/beam_score.pyscores as the official code at commitb2da22edoes: one judge call for each nugget, and for event ordering the line match and Kendall tau-b. The prompts are copied verbatim intobench/beam_prompts/under their MIT license.bench/beam_e2e.pycaptures each conversation, then answers each probe from two blocks:memory(recall,k30) andchunks(vector search over the chunks, the RAG baseline). It prints a cost and time estimate before a paid run, stops a conversation at twice its estimate, and resumes from a rows file.gpt-4.1-mini, the model of the judge. The API no longer servesgpt-4.1-nano: both the alias and the snapshotgpt-4.1-nano-2025-04-14return 404 "deprecated". So no reader matches the paper's GPT-4.1-nano row.bench/models.py: a model call can send a system message, because the official match call sends one. The cache key of a call without one does not change, so the cached replies still match. The cache connection can be closed from another thread.pyproject.toml: the bench group addsscipyandjson-repair.No file under
src/hotmemorychanges.Verification
make verify: 340 passed, 152 skipped.make integration: 151 passed, 1 xfailed (the known vector defect).bench/tests/test_beam_score.pyruns a copy of the official functions beside ours on fixed and 300 random replies, and requires equal results.Notes