Two collection guards passed against no implementation at all - #265
Merged
Merged
Conversation
The website session observed that absence tests can pass vacuously -- removing their implementation left two of four green, and both were the ones asserting that something is not there. Nothing set is also nothing seen. Two of mine were the same. Deleting the wiring from `generate_answer` left five of seven green, including both tests meant to guard the dangerous directions: that a reactome selection never leaks into the userguide bundle, and that a selection does not outlive the question. Each asserted `None`, and `None` is what an unimplemented feature produces. Both now establish the mechanism is live before asserting the absence -- a reactome call that must be seen, then the userguide call that must not be; set during, absent after. Deleting the wiring now fails four of seven instead of two. The three that still pass are the prompt and schema tests, which do not depend on the wiring and are not vacuous for passing. The general form is worth keeping: a test that asserts something is absent proves nothing until the same test has shown the thing can be present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adamjohnwright
force-pushed
the
tests/non-vacuous-collection-guards
branch
from
September 19, 2026 06:42
a5216b0 to
e015523
Compare
Reviewing the fix found one it missed. `test_the_userguide_prompt_alone_ offers_no_collections` asserted only that the block is absent from the userguide-only prompt -- which passes just as well when the block never renders for anyone. Verified by deleting it: the test stayed green while a different test in the same file caught the fault. File-level coverage was therefore fine and the test itself was worth nothing. A guard that relies on a neighbour to be non-vacuous is one refactor of the neighbour away from guarding nothing, and the neighbour is not mentioned anywhere near it. Both halves are in the one test now: the block renders for reactome, and it does not reach the userguide. Deleting the block fails it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The website session pointed out that removing their presence implementation left two of four tests green — and that both were the ones asserting absence. Nothing set is also nothing seen.
Two of mine were the same. Deleting the wiring from
generate_answerleft five of seven green, including both tests written to guard the dangerous directions:userguidebundleEach asserted
None, andNoneis exactly what an unimplemented feature produces. They would have gone on passing through any refactor that quietly dropped the wiring.Both now show the mechanism is live before asserting the absence — a reactome call that must be seen, then the userguide call that must not be; set during, absent after.
Deleting the wiring now fails four of seven, up from two. The three that still pass are the prompt and schema tests, which do not depend on the wiring and are not vacuous for passing.
The general form is worth keeping: a test that asserts something is absent proves nothing until the same test has shown the thing can be present.
501 passed, 1 skipped; mypy over all 137 files, ruff clean.
🤖 Generated with Claude Code