Close the gate hole that answer-level questions cannot close

[adamjohnwright]

This is the T005 work I said should happen before routing goes to beta. It did not end where it was meant to, and the ending is the useful part.

There is no complexes guard question

Six candidates across two question shapes, each asked with the collection present and with it removed from what resolve_collections sees — an exact simulation of a bundle without it, and a lot cheaper than copying 3.3G.

question	with	without
Which complex contains CYBA and CYBB?	0/4	0/4
Which complex has NUP133, NUP160, NUP37?	4/4	4/4
Which complex is made of LSM10, LSM11, SNRPB?	3/4	4/4

Complex names appear throughout reactions, as the names of inputs and outputs, and throughout summations prose. So any question naming or seeking a complex is answerable without it. The one thing structurally unique to complexes — the component list — cannot be asserted on, because the same configuration returns 1 to 6 of 7 components.

This is the same finding as T007's, reached independently: a collection whose content is duplicated elsewhere cannot be guarded by asking a question, however the question is worded.

Two corrections to my own measurements

A withdrawn claim. A four-run pass appeared to show complexes making the Nup107 answer worse — 1,1,1,1 against 3,3,3,3 without it. It did not survive being run again. The variance between runs is larger than any difference between the arms, so the claim was under-powered and is withdrawn rather than quietly dropped.

A void measurement. My first attempt set the selected_collections ContextVar around the call. That no longer works now the classifier is wired — generate_answer sets it from the classifier's choice and overwrites anything set outside, so both arms searched only complexes. Caught by instrumenting what retrieval actually searched, which is the second time in one day that check has saved a result from being reported.

So the guard moves to where it can exist

One parametrised test over all five real collection names: each is reachable, and selecting it searches nothing else. Closes T005a and T007 together.

It catches what the sweep cannot see by construction — a collection routing can never reach, through a name mismatch or a lookup that silently yields nothing, while every tracked question still passes. Verified by sabotage: making resolve_collections ignore the selection fails all five.

What this means for deploying #262

The reason I gave for holding routing back was that the sweep could not detect a classifier that never routes to complexes. That is now covered at the retrieval level, which is the strongest guard available for that collection. It is a weaker signal than an end-to-end question would have been, and it is the one that exists.

Checks

506 passed, 1 skipped. mypy over all 137 files, ruff, ruff format clean.

🤖 Generated with Claude Code