Measure what collection narrowing costs before building the classifier

specs/009-collection-routing/research.md

@@ -114,3 +114,82 @@ concurrently and asserts neither sees the other.
 
 The data model's diagram is left as the intent; this is how it is carried.
 
+## What narrowing actually costs, measured 2026-09-19
+
+Two measurements, and only the second means anything.
+
+**Citation overlap is arithmetic, not signal.** Narrowing to two collections
+keeps 5-7 of the full top-12 citations; narrowing to one keeps 2. That looks like
+a large loss, but reciprocal rank fusion interleaves the five per-collection
+lists, so a top-12 draws about 2.4 from each. Keeping two lists predicts ~4.8,
+and 5-7 is what was observed. The number measures the interleave, not whether
+anything useful was lost. Do not use it as a quality measure.
+
+**Whether answers survive is the measurement that counts.** Re-measured
+2026-09-19 through `answer_sweep.run()` itself, after the first attempt
+reimplemented the matching and got the number wrong (see the correction below).
+Two skips on this host, both `needs_live`, in both arms.
+
+| | all five collections | narrowed to `reactions` + `summations` |
+|---|---|---|
+| passed | **13 / 13** | **10 / 13** |
+| failed | 0 | 3 |
+| skipped | 2 | 2 |
+
+Two of the three failures are real losses, and they name their collection:
+
+| question | needs | what the narrowed answer said |
+|---|---|---|
+| "What is the UniProt accession for the TP53 protein" (`P04637`) | `ewas` | "not explicitly provided in the context searched" |
+| "Which diseases involve variants of the PTEN gene" | `disease_variants` | pathway-level prose, no variant named |
+
+**The third failure was not a loss -- the check was wrong.** Narrowed, and with
+`disease_variants` excluded, the ABCA1 question named all six curated variants
+(`C1417R`, `Q537R`, `S1446L`, `N935S`, `W590S`, `R587W`) with the OMIM id. The
+expectation failed it because the pattern required `ABCA1 C1417R` adjacency and
+the answer rendered them as a numbered list of `**C1417R**`. The pattern is fixed
+and pinned by a test; the answer was correct all along.
+
+That has a consequence for this feature: **ABCA1's variants are reachable from
+`summations` prose**, so that question does not guard `disease_variants`. Only
+the PTEN question does.
+
+### The correction
+
+The first measurement reported 12/15. It was produced by a throwaway script that
+reimplemented the sweep's matching, and it was wrong twice over: it never read
+the `must` field, which nine of the fifteen expectations carry, and it counted
+the two `needs_live` questions as passes where the sweep skips them. Both errors
+push the number up. The re-run uses `run()` and `report()` from
+`src/evaluation/answer_sweep.py` and records which collections retrieval actually
+searched, so the run proves its own precondition -- the first void re-run
+searched nothing at all and would otherwise have been read as a result.
+
+### What this settles
+
+**Collections are not interchangeable, and the mapping is legible.** Accession
+questions need `ewas`; PTEN variant questions need `disease_variants`. That is
+the signal a classifier can be prompted on.
+
+It is a weaker result than the first measurement suggested, and the weakening is
+the useful part: content is duplicated across `summations` more than the guard
+table assumed. `reactions` was already known to be unguardable by answer because
+`summations` covers `Pathway OR ReactionLikeEvent`; `disease_variants` now joins
+it for at least one question.
+
+**One excluded collection is untested by construction.** Narrowing excluded
+`complexes`, `ewas` and `disease_variants`. No tracked question guards
+`complexes` (T005 is still open), so its exclusion could not have produced a
+failure. The absence of a fourth failure is not evidence.
+
+**It still justifies the fail-wide rule.** Where a narrow selection does lose the
+answer, it removes it rather than degrading it -- the TP53 answer says the
+accession is not there, and the PTEN answer falls back to pathway prose with no
+variant. Widening on uncertainty costs latency; narrowing wrongly costs the
+answer.
+
+**And it sets the acceptance bar.** Routing is worth shipping only if the sweep
+stays at the control's score with the classifier choosing -- 13/13 here, 15/15 in
+the container where MCP is configured. The TP53 and PTEN questions are the ones
+to watch, because they fail loudly and specifically. A classifier that never
+routes to `complexes` would still score full marks, which is why T005 matters.

specs/009-collection-routing/spec.md

@@ -142,13 +142,22 @@ fails without it.
 |---|---|---|---|
 | `ewas` | yes | no | **guards it** -- UniProt accessions live only there |
 | `summations` | yes | no | **guards it** -- the curated prose lives only there |
+| `disease_variants` (PTEN) | yes | no | **guards it** -- no variant is named without it |
+| `disease_variants` (ABCA1) | yes | **yes** | all six variants came back from `summations` prose |
 | `complexes` | -- | -- | no question found yet |
 | `reactions` | yes | **yes** | answered with the collection removed entirely |
 
 **No reaction name is unique to `reactions`.** The `summations` query covers
 `Pathway OR ReactionLikeEvent`, so every event name appears in both collections by
 construction. They overlap by design.
 
+**And the overlap is wider than that.** Measured 2026-09-19 (research.md), the
+ABCA1 question answered with all six curated variants while `disease_variants`
+was excluded -- the summation prose for "Defective ABCA1 does not transport CHOL"
+names them. So a collection can be partly reachable through `summations` too, and
+whether a question guards a collection has to be measured per question, not
+assumed from the collection it was written for.
+
 Two consequences for this feature.
 
 The recall risk is **lower** than this spec assumed. If a question can be answered

specs/009-collection-routing/tasks.md

@@ -9,7 +9,7 @@ before it is changed.
 ## Phase 1: Setup
 
 - [ ] T001 Create branch `009-collection-routing` from main and set `.specify/feature.json` to `specs/009-collection-routing`
-- [ ] T002 Capture the retrieval baseline before any code change: `./bin/retrieval_baseline capture --out specs/009-collection-routing/before.json` against the installed Release97 bundle
+- [x] T002 Capture the retrieval baseline before any code change: `./bin/retrieval_baseline capture --out specs/009-collection-routing/before.json` against the installed Release97 bundle
 - [ ] T003 [P] Record the current cost in `specs/009-collection-routing/quickstart.md`: documents, context tokens and retrieval seconds for one question needing no variant data
 
 ## Phase 2: Foundational (blocks every user story)
@@ -23,7 +23,7 @@ acceptance criterion cannot detect the failure the feature can cause.
 - [x] T006 [P] Add an `ewas`-dependent question to `EXPECTATIONS` in src/evaluation/answer_sweep.py (TP53 UniProt P04637; verified by removal)
 - [ ] T007 Assert at retrieval level that `reactions` was searched, in tests/retrievers/test_collection_selection.py -- no answer-level question can guard it, because every reaction name also appears in `summations`
 - [x] T008 Verify each new question FAILS when its collection is removed from the bundle copy, and passes with it present; record the evidence in the PR (method established; two of four candidates survived it)
-- [ ] T009 Pin current behaviour: a characterization test in tests/retrievers/test_collection_selection.py asserting that with no selection every collection in the bundle is searched
+- [x] T009 Pin current behaviour: a characterization test in tests/retrievers/test_collection_selection.py asserting that with no selection every collection in the bundle is searched
 - [ ] T010 Run `./bin/answer-sweep` against Release97 and confirm green before any behaviour change
 
 ## Phase 3: User Story 1 — a question searches only the collections it needs (P1)
@@ -36,9 +36,9 @@ tokens for a question needing one collection fall relative to `before.json`.
 
 - [ ] T011 [US1] Add `collections: list[str] = []` to `QueryIntent` in src/agent/tasks/intent_classifier.py, defaulting to empty so an omitted field means "all"
 - [ ] T012 [US1] Extend the classifier prompt in src/agent/tasks/intent_classifier.py to name selectable collections, sourced from `reactome_descriptions_info` rather than a literal list
-- [ ] T013 [P] [US1] Add `resolve_collections(selected, available)` to src/retrievers/csv_chroma.py implementing data-model.md: empty means all, unknown names log WARNING and return all
-- [ ] T014 [P] [US1] Unit-test `resolve_collections` in tests/retrievers/test_collection_selection.py for empty, all-valid, some-unknown and all-unknown, asserting every failure widens rather than narrows
-- [ ] T015 [US1] Filter `self.collection_retrievers` by the selection in `retrieve_documents` in src/retrievers/csv_chroma.py, reading it from `RunnableConfig["configurable"]["collections"]`
+- [x] T013 [P] [US1] Add `resolve_collections(selected, available)` to src/retrievers/csv_chroma.py implementing data-model.md: empty means all, unknown names log WARNING and return all
+- [x] T014 [P] [US1] Unit-test `resolve_collections` in tests/retrievers/test_collection_selection.py for empty, all-valid, some-unknown and all-unknown, asserting every failure widens rather than narrows
+- [x] T015 [US1] Filter `self.collection_retrievers` by the selection in `retrieve_documents` in src/retrievers/csv_chroma.py, reading it from `RunnableConfig["configurable"]["collections"]`
 - [ ] T016 [US1] Apply the identical filter in `aretrieve_documents` in src/retrievers/csv_chroma.py — this is the served path
 - [ ] T017 [US1] Extend tests/retrievers/test_sync_async_equivalence.py to assert both paths honour the same selection, and confirm it fails when only one is filtered
 - [ ] T018 [US1] Carry `collections` on `ReactToMeState` in src/agent/profiles/react_to_me.py, set in `preprocess` beside `active_sources`

src/evaluation/answer_sweep.py

@@ -119,14 +119,21 @@ class Expectation:
         must=("Tangier",),
         # A named variant, not a specific one: which of the six come back
         # depends on retrieval order, and pinning one would fail a good answer.
-        must_match=(r"\bABCA1 [A-Z]\d{2,4}[A-Z*]",),
+        # The gene name is deliberately NOT required next to the variant.
+        # Requiring "ABCA1 C1417R" failed an answer that named all six as a
+        # numbered list of "**C1417R**" -- correct, and marked a failure. The
+        # question is already anchored on the topic by `must=("Tangier",)`.
+        must_match=(r"\b[A-Z]\d{2,4}[A-Z*]\b",),
         needs_collection="disease_variants",
     ),
     Expectation(
         question="Which diseases involve variants of the PTEN gene in Reactome?",
         why="Answered with the PTEN Loss of Function pathway. Reactome curates "
         "108 PTEN variants across 86 diseases.",
-        must_match=(r"\bPTEN [A-Z]\d{2,4}[A-Z*]",),
+        # Same shape as the ABCA1 guard above, and for the same reason: the
+        # gene name is not required adjacent to the variant.
+        must=("PTEN",),
+        must_match=(r"\b[A-Z]\d{2,4}[A-Z*]\b",),
         needs_collection="disease_variants",
     ),
     # --- facts about the database, which retrieval cannot answer ------------

tests/evaluation/test_answer_sweep.py

@@ -7,6 +7,7 @@
 """
 
 import asyncio
+import re
 from collections.abc import Callable
 
 import pytest
@@ -220,3 +221,32 @@ def test_the_sweep_does_not_pay_for_a_web_search() -> None:
         "the sweep now reads additional_content, so the assertion above is no "
         "longer the right guard"
     )
+
+
+def test_a_variant_guard_accepts_a_variant_named_in_a_list() -> None:
+    # Measured 2026-09-19: narrowed to reactions+summations, the chatbot
+    # answered the ABCA1 question with all six curated variants as a numbered
+    # list -- "1. **C1417R** - Causes Tangier disease" -- and the sweep called
+    # it a failure, because the pattern required "ABCA1 C1417R" adjacency. The
+    # answer was correct; the check was wrong, and it was wrong in the
+    # direction that makes a sweep get switched off.
+    named = (
+        "The ABCA1 variants listed in Reactome are: 1. **C1417R** - Causes "
+        "Tangier disease. 2. **Q537R** - Causes Tangier disease."
+    )
+    # The historical wrong answer, which named no variant at all, must still
+    # fail: a guard that passes the bug it was written for guards nothing.
+    pathway_prose = (
+        "Defective ABCA1 causes Tangier Disease, a disorder of cholesterol "
+        "transport, described at R-HSA-5682111."
+    )
+    abca1 = next(e for e in EXPECTATIONS if "ABCA1 variants" in e.question)
+    pten = next(e for e in EXPECTATIONS if "PTEN gene" in e.question)
+    for expectation in (abca1, pten):
+        for pattern in expectation.must_match:
+            assert re.search(
+                pattern, named, re.IGNORECASE
+            ), f"{expectation.question}: {pattern} rejects a named variant"
+            assert not re.search(
+                pattern, pathway_prose, re.IGNORECASE
+            ), f"{expectation.question}: {pattern} accepts pathway prose"