Give BM25 documents the metadata the Chroma half already had

[adamjohnwright]

Found while reviewing #246, which put a local filesystem path in front of me. The guard there stopped the leak; this removes the reason it happened.

The defect

create_bm25_chroma_ensemble_retriever loads its CSVs with plain CSVLoader, so every BM25 document gets metadata of only {row, source} — while the Chroma half of the same ensemble, built from the same CSV by MetaDataCSVLoader at embedding time, carries st_id and the rest.

A document with no st_id in metadata cannot be cited, even though its stable id is sitting in the page content as st_id: R-HSA-... where nothing can reach it. On "What does CDK5 do in neurons?", of the documents the citation path walked, 50 of 62 were unattributable before and 0 are now. The sources a caller saw came only from the Chroma half.

What this does NOT do — correcting my own first claim

My first version of this said 62 retrieved documents became 14, and that 11 exact duplicates were being fed to the model twice. Both were mismeasurements, caught by the adversarial pass on this PR:

• the instrument hooked the citation loop, which stops at MAX_CITATIONS — so it counted documents examined before the cap, not documents retrieved
• and my "before" run had nothing to stash, because I had already committed, so it measured the fixed code against itself

Measured properly, at unique_documents where the model's context is actually decided, three runs per question because retrieval is not reproducible:

	docs into dedup	docs to model	distinct st_ids
before (main)	50	50	~38
after (fix)	50	50	~39

Unchanged. No duplicates are collapsed there in either version. The benefit of this change is citation completeness, and nothing else I can demonstrate.

That is consistent with the mechanism: content is untouched — MetaDataCSVLoader trims content only when content_columns is given, and it is not — so BM25 scoring is identical and retrieval should not move.

Verification

• answer sweep green, 13/13 (2 skipped for no local MCP, as expected)
• every column promoted, read from the header, so metadata cannot drift from the file
• ruff, format, mypy (130 files), full suite with no API keys set

🤖 Generated with Claude Code