Let the classifier choose collections, and measure what that is worth

[adamjohnwright]

Completes spec 009. The classifier now chooses which Reactome collections to search, and the measurement says what that is actually worth — which is less than the spec claimed, in a way worth reading.

What landed

QueryIntent gains collections; the prompt describes each collection from reactome_descriptions_info rather than a literal list, so a collection added to the bundle cannot be missing from the prompt meant to offer it; generate_answer sets the selection around retrieval.

It rides the selected_collections ContextVar, not config["configurable"] as tasks.md assumed — create_retrieval_chain gives the retriever no path for extra arguments. LangGraph copies the context into spawned tasks, so a value set around the call reaches the retriever inside them. Reset in finally: the graph reuses one asyncio task across turns, and a selection left set would narrow the next question, which nothing downstream could detect because narrowing yields a confident answer from less.

Three measurements

It holds the bar. Sweep is 13/13 with the classifier choosing — the control score — with 5 of 6 retrievals narrowed, routing exactly as the earlier measurement predicted: accession → ewas, both variant questions → disease_variants, summary → summations, broad mechanistic → left open.

It does not reduce the prompt. This corrects the spec.

question	docs all	docs narrowed	chars all	chars narrowed
TP53 accession	10	10	2,436	2,436
ABCA1 variants	10	10	6,163	5,846
Selective autophagy	10	10	22,114	21,797

Ten documents either way, context within 2% — the chain caps the fused list at ten however many collections fed it. T003's framing, that context tokens would fall, was measuring the wrong thing.

What it reduces is retrieval work: 1.82s → 1.43s median, three runs per question, timed inside aretrieve_documents. Five collections mean ten sub-retrievals; one means two. About a fifth, or 0.4s against a first token near ten seconds.

The honest case

It did not fix a failing question — the control already passed 13/13. It buys a fifth off retrieval and puts the right documents into a fixed-size context. Only the first of those is measured.

Adversarial review found two things

state["collections"] would raise. BaseState is total=False and the field is new, so a thread resumed from a pre-existing checkpoint has no key — a KeyError on the served path, invisible until someone resumes an old conversation. Now .get(...) or [], pinned by a test that deletes the key and fails against the subscript.

Six questions was a thin base to ship a router on. Probed with nine unseen and deliberately awkward ones — three broad that must not narrow, two cross-collection, four specific. Nine of nine as intended, with both cross-collection questions taking summations alongside their primary. Recorded with its limit: nine classifications, not nine answers.

Also documented: collections is list[str] and not an enum on purpose. A hallucinated name widens to all with a WARNING; constraining the schema would make structured output reject the response, turning a harmless mistake into a failed answer.

Still open

complexes has no guard question (T005), so a classifier that never routes to it would still score full marks.

Checks

501 passed, 1 skipped. ruff, ruff format, mypy clean.

🤖 Generated with Claude Code