fix(eval): select the strategy on validation, and stop calling candidates passages

[mrsibe]

Stacked on #203 (feat/eval-cross-lingual-corpus). Retarget to main after the chain merges.

What does this PR do?

Four follow-ups from the review. The first two are semantics that would have spread into corpus v2 if they were left alone.

1. Strategy adoption is now held out

eval-retrieval.mjs ran with split = all, so the strategy was chosen and scored on the same 44 questions — exactly the mistake the threshold experiment had already been fixed for. It now:

1. runs every strategy on validation and decides there with decideAdoption;
2. re-runs the shipped strategy and the selected one on test, and only reports them.

The choice never sees test. If validation selects nothing, test reports the shipped strategy alone and the report says there is no adoption candidate.

The consequence is a more conservative and more trustworthy result than before. On validation, hybrid is identical to dense:

	Recall@5	nDCG@10	MRR	MAP@10
dense (validation)	0.9231	0.8276	0.7788	0.7660
hybrid (validation)	0.9231	0.8276	0.7788	0.7660

So nothing is adopted — whereas the split = all run had hybrid ahead on nDCG@10 (0.8574 vs 0.8476). That difference was the choice being scored on its own questions.

2. meanRetrieved was a misleading name

The harness fetches candidateK in order to compute Recall@10; the context window is results.slice(0, contextK). So meanRetrieved = 19 never meant "19 passages go to the model" — it meant "19 candidates passed the threshold". The unanswerable group now reports both sizes:

meanCandidatesRetrieved  19   // passed the threshold, capped by candidateK
meanContextPassages       3   // actually reach the window, min(candidates, contextK)

The accurate description of the FIFA case is therefore: 19/19 chunks pass threshold: 0.5, and the top 3 irrelevant ones go into the prompt. Still a real problem, but not "19 passages are stuffed into the model". The sweep and threshold tables now carry cands and ctx as separate columns instead of conflating them.

3. It is retrieval abstention, not refusal

No generator runs in this harness, so it can show that nothing passed the threshold; it cannot show that the model would decline to answer. Fields renamed (abstentionCount, retrievalAbstentionRate) and the report states plainly that a true system refusal rate needs a generator eval.

4. The manifest rationale was wrong

The docs claimed a hash of the id "moves other questions between the sides" when a new question is added. That is false for hash(id) % 3 — it is computed per id, so it is stable. The real reasons for an explicit manifest are:

• a hash cannot stratify a small corpus, which is how multi-hop and cross-lingual ended up entirely on one side; and
• a new question would be assigned silently rather than deliberately, and test is the side a choice must not be fitted to.

Corrected in src/main/eval/types.ts, eval/README.md and test/evalSplit.test.ts.

Also

#200 was a sibling, not part of the chain. Merging #193 → … → #199 → #201 → #202 → #203 would have silently skipped it. I retargeted #200's base to feat/eval-cross-lingual-corpus so the final stack is linear and it cannot be missed.

Testing

• npm run typecheck — clean
• npm test — 504 pass
• npm run eval twice — byte-identical baseline
• npm run eval:retrieval — validation selects, test reports; no adoption candidate
• npm run eval:threshold, npm run eval:sweep — regenerated

Related

Part of #192 (review follow-ups). Next: the hard-negative corpus expansion.