Phase 2: two collection guards that survive removal — and why the other two did not

specs/009-collection-routing/spec.md

@@ -129,6 +129,42 @@ this change is a real difference in behaviour rather than noise.
 That last one matters: the point of this change is a number going down, and it
 should be reported as one.
 
+## Finding: the collections overlap more than this spec assumed
+
+Phase 2 set out to add one sweep question per collection, each failing if its
+collection stopped being searched. Building them changed the picture.
+
+Method: ask a candidate against the full bundle and against a copy with exactly one
+collection removed. A question guards a collection only if it answers with it and
+fails without it.
+
+| collection | with | without | result |
+|---|---|---|---|
+| `ewas` | yes | no | **guards it** -- UniProt accessions live only there |
+| `summations` | yes | no | **guards it** -- the curated prose lives only there |
+| `complexes` | -- | -- | no question found yet |
+| `reactions` | yes | **yes** | answered with the collection removed entirely |
+
+**No reaction name is unique to `reactions`.** The `summations` query covers
+`Pathway OR ReactionLikeEvent`, so every event name appears in both collections by
+construction. They overlap by design.
+
+Two consequences for this feature.
+
+The recall risk is **lower** than this spec assumed. If a question can be answered
+with a whole collection removed, routing away from that collection costs little --
+which argues for the change rather than against it.
+
+But the gate hole is **harder to close** than the plan implies, and for a reason the
+plan had wrong. It is not that nobody wrote the questions; it is that a collection
+whose content is duplicated elsewhere cannot be guarded by asking a question. For
+`reactions` the guard would have to assert on retrieval -- which documents were
+searched -- rather than on the answer.
+
+Recorded rather than worked around: `tasks.md` T004-T008 are written as though four
+such questions exist. Two do. T007 (`reactions`) needs rewriting as a retrieval-level
+assertion, and T005 (`complexes`) needs a better candidate before it can be claimed.
+
 ## Scope
 
 In: collection selection for the `reactome` source, defaulting to all; the

specs/009-collection-routing/tasks.md

@@ -18,11 +18,11 @@ before it is changed.
 routing stops searching them. Routing must not land before this closes, or the
 acceptance criterion cannot detect the failure the feature can cause.
 
-- [ ] T004 [P] Add a `summations`-dependent question to `EXPECTATIONS` in src/evaluation/answer_sweep.py, with a `must`/`must_match` that fails if that collection is not searched
-- [ ] T005 [P] Add a `complexes`-dependent question to `EXPECTATIONS` in src/evaluation/answer_sweep.py
-- [ ] T006 [P] Add an `ewas`-dependent question to `EXPECTATIONS` in src/evaluation/answer_sweep.py
-- [ ] T007 [P] Add a `reactions`-dependent question to `EXPECTATIONS` in src/evaluation/answer_sweep.py
-- [ ] T008 Verify each new question FAILS when its collection is removed from the bundle copy, and passes with it present; record the evidence in the PR
+- [x] T004 [P] Add a `summations`-dependent question to `EXPECTATIONS` in src/evaluation/answer_sweep.py (Selective autophagy / lysosome; verified by removal)
+- [ ] T005 [P] Find a `complexes` candidate that fails without the collection (the first attempt did not), then add the question to `EXPECTATIONS` in src/evaluation/answer_sweep.py
+- [x] T006 [P] Add an `ewas`-dependent question to `EXPECTATIONS` in src/evaluation/answer_sweep.py (TP53 UniProt P04637; verified by removal)
+- [ ] T007 Assert at retrieval level that `reactions` was searched, in tests/retrievers/test_collection_selection.py -- no answer-level question can guard it, because every reaction name also appears in `summations`
+- [x] T008 Verify each new question FAILS when its collection is removed from the bundle copy, and passes with it present; record the evidence in the PR (method established; two of four candidates survived it)
 - [ ] T009 Pin current behaviour: a characterization test in tests/retrievers/test_collection_selection.py asserting that with no selection every collection in the bundle is searched
 - [ ] T010 Run `./bin/answer-sweep` against Release97 and confirm green before any behaviour change
 

src/evaluation/answer_sweep.py

@@ -91,6 +91,26 @@ class Expectation:
         why="Preferring the web tool must not bury the plugin for someone who wants it.",
         must=("FIViz",),
     ),
+    # --- one question per collection, so routing cannot quietly skip one ----
+    # Each was chosen by removing its collection from a copy of the bundle and
+    # confirming the answer changes. A question that still answers without its
+    # collection guards nothing, and three of the first four candidates were
+    # exactly that -- see specs/009-collection-routing/spec.md.
+    Expectation(
+        question="What is the UniProt accession for the TP53 protein in Reactome?",
+        why="Guards the `ewas` collection: it is the only one holding UniProt "
+        "links. Verified by removing ewas from a bundle copy, after which the "
+        "accession is no longer answered.",
+        must_match=(r"\bP04637\b",),
+    ),
+    Expectation(
+        question="What does Reactome's summary of Selective autophagy say about "
+        "where cargo is degraded?",
+        why="Guards the `summations` collection: the curated prose summaries live "
+        "only there. Without it the chatbot says no summary is available.",
+        must=("lysosom",),
+        must_not=("does not provide", "not currently available"),
+    ),
     # --- disease variants, which only the new collection can name -----------
     Expectation(
         question="List the ABCA1 variants in Reactome and the disease each one causes.",