Repository navigation
Phase 2: two collection guards that survive removal — and why the other two did not - #229
Merged
Merged
Conversation
… not Phase 2 of spec 009 set out to add one sweep question per collection, each failing if its collection stopped being searched. Building them changed what the spec claims. Method, since the obvious approach does not work: ask a candidate against the full bundle and against a copy with exactly one collection removed. A question guards a collection only if it answers with it and fails without it. Checking string-uniqueness in the CSVs is not enough -- P04637 looked like an ewas marker and is in disease_variants too. Two of four candidates survived that test and are added here. ewas is guarded by TP53's UniProt accession, summations by the curated prose for Selective autophagy; both verified to fail with their collection removed. The other two did not, and the reason is structural rather than a matter of picking better words. Every reaction name also appears in summations, because the summations query covers Pathway OR ReactionLikeEvent -- so the two collections overlap by construction and no answer-level question can distinguish them. reactions answered correctly with its collection removed entirely. T007 is rewritten as a retrieval-level assertion; T005 needs a better complexes candidate and is left open rather than claimed. This cuts both ways for the feature and the spec now says so. Redundancy means routing away from a collection costs less recall than the spec assumed, which argues for the change. It also means the gate hole is harder to close than the plan implied: a collection whose content is duplicated elsewhere cannot be guarded by asking a question at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #228. Phase 2 of spec 009 — the blocking prerequisite, which is worth landing whether or not routing is ever built.
The method, because the obvious approach doesn't work
A question only guards a collection if the answer changes when that collection is removed. String-uniqueness in the CSVs isn't enough:
P04637looked like a cleanewasmarker and turns out to be indisease_variantstoo.So each candidate was asked twice — against the full Release97 bundle, and against a copy with exactly one collection removed.
ewassummationscomplexesreactionsTwo survived and are added here. Both verified passing against the real bundle with live MCP.
Why the other two failed, which is the actual finding
No reaction name is unique to
reactions. Thesummationsquery coversPathway OR ReactionLikeEvent, so every event name appears in both collections by construction. They overlap by design, not by accident.That cuts both ways for spec 009, and the spec now says so:
reactionsthe guard has to assert on retrieval — which documents were searched — not on the answer.Tasks
T004andT006done and ticked.T008ticked — the method is established, and it's what disqualified two candidates.T007rewritten as a retrieval-level assertion.T005left open rather than claimed:complexesneeds a better candidate, and I'd rather it stay visible than be filled with a question that guards nothing.That last point is the whole reason this phase exists — four questions that look like guards and aren't would be worse than knowing there are two.
🤖 Generated with Claude Code