Repository navigation

Phase 5: a public benchmark - #14

Merged

rohan-hotdata merged 22 commits into

mainfrom

phase-5-public-benchmark

Oct 12, 2026

Merged

Phase 5: a public benchmark#14
rohan-hotdata merged 22 commits into
mainfrom
phase-5-public-benchmark

Commits on Oct 12, 2026
- Plan phase 5, a public benchmark
  rohan-hotdata
  committed
  8982fae
- Record the owner's agreement to the phase 5 decisions
  rohan-hotdata
  committed
  e2d489a
- Add the benchmark data, the episodes, and the embedder cache
  rohan-hotdata
  committed
  6181795
- Port the retrieval metrics of the reference code
  rohan-hotdata
  committed
  17e010d
- Add the retrieval runner
  rohan-hotdata
  committed
  fc9d7e9
- Read the commit when a retrieval run starts
  rohan-hotdata
  committed
  dc2f79b
- Retry a dropped connection and time the first call of a retrieval run
  rohan-hotdata
  committed
  8c528cc
- Use gpt-4o-mini as the extractor of the benchmark
  rohan-hotdata
  committed
  aff898b
- Run the retrieval score and the driver check on the local stack
  rohan-hotdata
  committed
  934411d
- Record the cloud latency run of 20 questions
  rohan-hotdata
  committed
  5033cc1
- Add the end-to-end runner
  rohan-hotdata
  committed
  a98dbc3
- Run the end-to-end score on the sample of 54
  rohan-hotdata
  committed
  38c4ee9
- Add the diagnosis of the memory arm
  rohan-hotdata
  committed
  7756d25
- Mark the facts that hold an answer with GPT-4o and a strict prompt
  rohan-hotdata
  committed
  2b889b1
- Diagnose the misses of the memory arm
  rohan-hotdata
  committed
  140e98b
- Report the benchmark in docs/benchmarks.md
  rohan-hotdata
  committed
  7316fe9
- Name the rebased commits in the result files
  rohan-hotdata
  committed
  babb8e8
- Show the benchmark numbers in the README
  rohan-hotdata
  committed
  6ffb727
- Keep the result files out of git
  rohan-hotdata
  committed
  5c456b6
- Rank the diagnosis the way recall does
  rohan-hotdata
  committed
  09d6fab
- Correct the size of the largest recalled block in the plan
  rohan-hotdata
  committed
  a5c832a
- Build the diagnosed block from a search with k of 30
  rohan-hotdata
  committed
  e1c5f04

Commits

Commits on Oct 12, 2026