Repository navigation
Remove the sweep marker that retries correct answers - #259
Merged
Merged
Conversation
`TRANSIENT` is a list of phrases meaning "upstream had a problem"; a match
makes the sweep retry the question. One of the three was "could not find out"
-- and src/reactome_mcp/answer.py instructs the model to use exactly that
wording for a legitimate empty result: "If the tools do not answer the
question, say plainly what you could not find out."
So the marker matched a correct negative answer as readily as an outage, and it
resolved that ambiguity in the direction that hides regressions: a real failure
was retried and could pass on the second attempt. A gate quietly forgiving the
thing it exists to catch.
Removed, with a test that fails against the old list and still recognises the
outage literal this repository emits itself.
The two markers left are not signals either -- one is our own literal, the
other a phrase a correct answer has little reason to use. The real signal is
the tool exception in `answer_from_live_services`, which is caught, logged and
handed to the model as "This lookup failed: {exc}"; the model paraphrases it,
so nothing survives into the text. Propagating it out of the live path is
recorded as T023 in spec 010 rather than guessed at here, because it touches
the MCP module's return type and the graph state.
Found by the website session, which had just made the same mistake in the other
direction: it blocked docs.google.com as a third-party embed when four of its
pages read published Sheets from it as data. The discriminator described what
had been observed, not what the thing was for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The website session made the right challenge: if "could not find out" was
instructed by a prompt, the same question applies to the other two, and
"they read less like instructions" is not an answer.
Checked. Neither appears in any prompt, tool description or example in this
repository. "could not complete that lookup" is a literal only
`answer_from_live_services` emits; "service error" has no source here at all
and can only come from the model paraphrasing the `"This lookup failed:
{exc}"` it is handed when a tool actually raised. Both derive from real
failure paths rather than from intended output.
Recorded as evidence with its limit stated: the MCP tool descriptions come
from the remote server and are not checked here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adamjohnwright
force-pushed
the
fix/transient-marker-hides-regressions
branch
from
September 19, 2026 03:40
55cf947 to
9467b0f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The website session pointed out that my
TRANSIENTlist is the same shape as a mistake they had just made — blockingdocs.google.comin their e2e suite as a third-party embed, when four of their pages read published Sheets from it as data. The discriminator described what they had observed, not what the thing was for.Ours is worse than I first described it to them.
TRANSIENTis a list of phrases meaning "upstream had a problem"; a match makes the sweep retry the question. One of the three was"could not find out"— andsrc/reactome_mcp/answer.pyinstructs the model to use exactly that wording for a legitimate empty result:So the marker matched a correct negative answer as readily as an outage, and it resolved the ambiguity in the direction that hides regressions: a real failure got retried and could pass on the second attempt. A gate quietly forgiving the thing it exists to catch.
Removed, with a test that fails against the old list and still recognises the outage literal this repository emits itself.
What is not fixed, and why it is a task rather than a claim
The two remaining markers are not signals either — one is our own literal, the other a phrase a correct answer has little reason to use. The real signal is the tool exception in
answer_from_live_services, which is caught, logged, and handed to the model as"This lookup failed: {exc}". The model then paraphrases it, so by the time the sweep sees text there is nothing left to key on.Propagating that out of the live path is recorded as T023 in spec 010 rather than guessed at here. It touches the MCP module's return type and the graph state, which is more than this fix should carry.
Checks
461 passed, 1 skipped. ruff, ruff format and mypy clean.
Merge order: this touches
tests/evaluation/test_answer_sweep.py, as does #257. Both append, so it should merge cleanly, but #257 first is tidier.🤖 Generated with Claude Code