From 3ad234fde800f4b8817bce7e811c20b8a17c7f2b Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Fri, 18 Sep 2026 16:52:24 +0000 Subject: [PATCH 1/2] Stop denying on behalf of all of Reactome Adam clicked a search suggestion that mixed a curator's surname with the gene he was searching for, and got: "The information regarding the relationship between Tello-Ruiz, IL6, and Beta-1 is not currently available in the Reactome Knowledgebase." That statement is false. Marcela Tello-Ruiz is in Reactome as a Person record, dbId 72611. What the answer meant was "not in the pathway content I searched", and the prompt told it to say the other thing: "If no relevant information exists in Reactome, explain the information is not currently available in Reactome." The search covers pathways, reactions, complexes, proteins, disease variants and the user guide. Reactome also holds curators, authors, literature references and much else that is not in this index, so "not in Reactome" is a claim this answer is not in a position to make. Measured, four runs each, before and after: before: all four CONFABULATED -- "The relationship between Tello-Ruiz, IL6, and Beta-1 can be understood through their involvement in specific molecular..." -- inventing a role for a person's surname after: all four scoped the denial correctly and named Tello-Ruiz as the part not found The confabulation is worse than the reported overclaim and would have been missed by measuring denials alone, because inventing a relationship is not a denial. It was only visible by reading the text. My first detector matched "not found in the Reactome pathway content searched" -- the correct phrasing -- and scored the working fix as failing. Corrected with a negative lookahead and validated against four known-good and known-bad strings before being trusted. Answer sweep 15/15 with MCP configured, so normal answers are unaffected. Co-Authored-By: Claude Opus 5 --- src/retrievers/reactome/prompt.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/retrievers/reactome/prompt.py b/src/retrievers/reactome/prompt.py index 4dc788e..aaf5689 100644 --- a/src/retrievers/reactome/prompt.py +++ b/src/retrievers/reactome/prompt.py @@ -14,7 +14,9 @@ 1. Strict source discipline: Use only the information explicitly provided from Reactome. Do not invent, infer, or draw from external knowledge. - Use only information directly found in Reactome. - Do **not** supplement, infer, generalize, or assume based on external biological knowledge. - - If no relevant information exists in Reactome, explain the information is not currently available in Reactome. Do **not** answer the question. + - If the context does not answer the question, say the answer was not found **in the Reactome pathway content searched**. Do **not** say it is absent from Reactome, or from the Reactome Knowledgebase, and do **not** answer the question. + - That distinction is not pedantry. This search covers pathways, reactions, complexes, proteins, disease variants and the user guide. Reactome also holds curators and authors, literature references, and much else that is not in this index -- so "not in Reactome" is a claim you are not in a position to make, and it has been wrong: a question about a Reactome curator was answered "not currently available in the Reactome Knowledgebase" while that person was in Reactome as a Person record. + - If part of the question is unfamiliar -- a name, an acronym, a term absent from the context -- say that part was not found rather than describing it. Never assign a role, function or relationship to something the context does not describe. 2. Inline citations required: Every factual statement must include ≥1 inline anchor citation in the format: display_name - If multiple entries support the same fact, cite them together (space-separated). 3. Comprehensiveness: Capture all mechanistically relevant details available in Reactome, focusing on processes, complexes, regulations, and interactions. From 042a1f34e42ab5a04ce1e96d5da5666b60623739 Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Fri, 18 Sep 2026 18:15:15 +0000 Subject: [PATCH 2/2] Fix the same overclaim in the three other prompts The adversarial review of this branch found that I had fixed one instance of a bug that existed in four. The same instruction -- deny on behalf of an entire database -- was also in: src/retrievers/plantreactome/prompt.py "not currently available in Plant Reactome" src/retrievers/uniprot/prompt.py "not currently available in UniProt" src/agent/tasks/cross_database/summarize_reactome_uniprot.py "not currently available in Reactome or UniProt" Each is wrong for the same reason: only an indexed subset is searched, and the databases hold far more. The UniProt one is the starkest -- our index is a small slice of UniProt, so "not in UniProt" is almost always a false statement. This is the shape the website session named this morning: a second caller kept doing the wrong thing after the first was fixed. I only found it by grepping for the pattern after fixing the reported instance, which is the habit worth keeping rather than the finding. Measured only for the Reactome path: the answer sweep is 15/15 and six thinner questions show no over-refusal (0/6 both before and after) and no citation loss (51 vs 52). The Plant Reactome, UniProt and cross-database prompts are corrected by the same construction but are not exercised by the sweep, so that is an argument from consistency rather than a measurement. Co-Authored-By: Claude Opus 5 --- src/agent/tasks/cross_database/summarize_reactome_uniprot.py | 4 +++- src/retrievers/plantreactome/prompt.py | 4 +++- src/retrievers/uniprot/prompt.py | 4 +++- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/src/agent/tasks/cross_database/summarize_reactome_uniprot.py b/src/agent/tasks/cross_database/summarize_reactome_uniprot.py index 9ceb935..048d84f 100644 --- a/src/agent/tasks/cross_database/summarize_reactome_uniprot.py +++ b/src/agent/tasks/cross_database/summarize_reactome_uniprot.py @@ -9,7 +9,9 @@ Instructions: 1. Provide answers **strictly based on the given context from the Reactome and UniProt Knowledgebase**. Do **not** use or infer information from any external sources. - 2. If the answer cannot be derived from the context provided, do **not** answer the question; instead explain that the information is not currently available in Reactome or UniProt. + 2. If the answer cannot be derived from the context provided, do **not** answer the question; instead say it was not found **in the Reactome and UniProt content searched**. Do **not** say it is absent from Reactome or from UniProt. + - Only indexed subsets of both are searched here, and both hold far more, so "not in Reactome" or "not in UniProt" is a claim you are not in a position to make. + - If part of the question is unfamiliar -- a name, an acronym, a term absent from the context -- say that part was not found rather than describing it. Never assign a role, function or relationship to something the context does not describe. 3. Extract Key Insights: Identify the most relevant and accurate details from both databases; Focus on points that directly address the user’s question. 4. Merge Information: Combine overlapping information concisely while retaining key biological terminology (e.g., gene names, protein names, pathway names, disease involvement, etc.) 5. Ensure Clarity & Accuracy: diff --git a/src/retrievers/plantreactome/prompt.py b/src/retrievers/plantreactome/prompt.py index 786d510..763195d 100644 --- a/src/retrievers/plantreactome/prompt.py +++ b/src/retrievers/plantreactome/prompt.py @@ -14,7 +14,9 @@ 1. Strict source discipline: Use only the information explicitly provided from Plant Reactome. Do not invent, infer, or draw from external knowledge. - Use only information directly found in Plant Reactome. - Do **not** supplement, infer, generalize, or assume based on external biological knowledge. - - If no relevant information exists in Plant Reactome, explain the information is not currently available in Plant Reactome. Do **not** answer the question. + - If the context does not answer the question, say the answer was not found **in the Plant Reactome pathway content searched**. Do **not** say it is absent from Plant Reactome, and do **not** answer the question. + - The search covers indexed pathway content, not everything Plant Reactome holds, so "not in Plant Reactome" is a claim you are not in a position to make. + - If part of the question is unfamiliar -- a name, an acronym, a term absent from the context -- say that part was not found rather than describing it. Never assign a role, function or relationship to something the context does not describe. 2. Inline citations required: Every factual statement must include ≥1 inline anchor citation in the format: display_name - If multiple entries support the same fact, cite them together (space-separated). 3. Comprehensiveness: Capture all mechanistically relevant details available in PlantReactome, focusing on processes, complexes, regulations, and interactions. diff --git a/src/retrievers/uniprot/prompt.py b/src/retrievers/uniprot/prompt.py index 9f5cf27..b15fb82 100644 --- a/src/retrievers/uniprot/prompt.py +++ b/src/retrievers/uniprot/prompt.py @@ -8,7 +8,9 @@ When providing answers, please adhere to the following guidelines: 1. Provide answers **strictly based on the given context from the UniProt Knowledgebase**. Do **not** use or infer information from any external sources. -2. If the answer cannot be derived from the context provided, do **not** answer the question; instead explain that the information is not currently available in UniProt. +2. If the answer cannot be derived from the context provided, do **not** answer the question; instead say it was not found **in the UniProt content searched**. Do **not** say it is absent from UniProt. + - Only an indexed subset of UniProt is searched here, and UniProt holds far more, so "not in UniProt" is a claim you are not in a position to make. + - If part of the question is unfamiliar -- a name, an acronym, a term absent from the context -- say that part was not found rather than describing it. Never assign a role, function or relationship to something the context does not describe. 3. Answer the question comprehensively and accurately, providing useful background information based **only** on the context. 4. keep track of **all** the sources that are directly used to derive the final answer, ensuring **every** piece of information in your response is **explicitly cited**. 5. Create Citations for the sources used to generate the final answer according to the following: