Fix the retrieval baseline tool, and capture spec 009's before state

[adamjohnwright]

Starting spec 009 the way its own task list says (T002) and Principle II requires: measure before changing anything. The measurement tool turned out to be broken in two ways.

It could not run

client_settings=chroma_settings,   # the factory, not a call

chroma_settings used to be a constant and became a function — because langchain_chroma mutates the Settings it is given, so a shared object leaks one store's persist directory into the next. This call site was never updated. Nothing exercises this tool, so it has been broken silently since that refactor.

Then it recorded identities that do not survive a release

doc_id prefers a Reactome stable id and falls back to a row number, with the docstring:

"Reactome stable IDs survive a bundle rebuild; row numbers do not."

Measured against its own output:

retriever	ids	stable?
vector	1000	all st_id ✓
bm25	1000	all row: ✗

Because it built BM25 from a plain CSVLoader, whose documents carry only {row, source}. A bundle rebuild happens every release, so a before/after comparison spanning one would have shown spurious total disagreement on the BM25 half — and spec 009's entire method is before/after comparison.

That is the same defect the application had, in a third caller. It is the pattern the website session named this morning — a second caller kept doing the wrong thing after the first was fixed — and the reason I now grep for the shape rather than fixing the instance I was shown. Same fix: MetaDataCSVLoader with columns read from the header.

After: 1000 of 1000 BM25 ids stable, matching the vector side.

The baseline

specs/009-collection-routing/before.json — 20 questions across all five collections (complexes, disease_variants, ewas, reactions, summations), bm25 and vector separately, at k=10. That is what collection routing will be measured against.

Context

I marked spec 009 done earlier today and was wrong — corrected in #253. Source routing landed; collection routing did not, and retrieve_documents still searches all five collections for every question. This is the first real step on it.

ruff, mypy, full suite with no API keys set.

🤖 Generated with Claude Code