From 93827eaa6af708d985d68d684259e6f2c6c08002 Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Fri, 2 Oct 2026 13:46:18 +0000 Subject: [PATCH] Send gene lists to over-representation, not ReactomeGSA The user-guide prompt said "gene set analysis is ReactomeGSA" with no word about gene lists, overriding the guide itself, which files a pasted identifier list under Analysis Data (over-representation) and ReactomeGSA under Analysis Gene Expression. Reported 2026-09-29: "here is my gene list TP53, ERBB3 and JAX9" was told to paste it into ReactomeGSA. The answer sweep required that wrong answer: its gene-list question had must=("ReactomeGSA",), so a correct answer would have failed a deploy. - Prompt: one added line -- a list of genes is over-representation via Analyze Data; never send a gene list to ReactomeGSA. The original ReactomeGSA sentence is kept, URL included: measured, removing reactome.org/gsa or restructuring the rule made GSEA answers mix in ReactomeFIViz's Cytoscape steps in 6/6 runs (1/6 before). The URL is real: beta.reactome.org/gsa redirects to the quantitative analysis tab. - Sweep: the gene-list question now expects over-representation, and the reported message is added. Measured, 5 runs each: old prompt 0/10 on the corrected checks, new 10/10. GSEA answers mixing FIViz steps: 1/6 before, 2/6 after (n=6, noise). Full sweep 16/16. Co-Authored-By: Claude Opus 5.5 --- src/evaluation/answer_sweep.py | 14 ++++++++++++-- src/retrievers/userguide/prompt.py | 3 ++- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/src/evaluation/answer_sweep.py b/src/evaluation/answer_sweep.py index 992915b..8f90f62 100644 --- a/src/evaluation/answer_sweep.py +++ b/src/evaluation/answer_sweep.py @@ -74,10 +74,20 @@ class Expectation: question="I have a gene list do you have a tool I can use to analyse where in " "reactome those genes are involved", why="Answered 'Reactome does not provide a specific tool' -- false, and about " - "its flagship feature.", - must=("ReactomeGSA",), + "its flagship feature. Then, until 2026-09-29, this expected ReactomeGSA: the " + "wrong tool for a list, which needs no measurements. A gene list is an " + "over-representation analysis; this gate was requiring the wrong answer.", + must_match=(r"over-?represent|analy[sz]e\s+data",), must_not=("does not provide", "not currently available"), ), + Expectation( + question="here is my gene list TP53, ERBB3 and JAX9. how can I analyse it?", + why="Reported 2026-09-29: told to paste the list into ReactomeGSA at " + "reactome.org/gsa -- a URL the user guide does not contain, for a tool that " + "needs a matrix.", + must_match=(r"over-?represent|analy[sz]e\s+data",), + must_not=("reactome.org/gsa",), + ), # --- gene set analysis should prefer the tool needing no install -------- Expectation( question="How do I run a GSEA in Reactome?", diff --git a/src/retrievers/userguide/prompt.py b/src/retrievers/userguide/prompt.py index 8b4d3b9..9a7f366 100644 --- a/src/retrievers/userguide/prompt.py +++ b/src/retrievers/userguide/prompt.py @@ -16,7 +16,8 @@ - If multiple excerpts support the same fact, cite them together (space-separated). 3. How-to focus: Give clear, actionable steps when the user asks how to perform a task. Name UI elements accurately (buttons, panels, tabs) as they appear in the context. - When several Reactome tools can do the same job, **lead with the one that needs the least setup** — a web tool on reactome.org before a desktop application, a plugin, or an R package. Mention the others afterwards as alternatives. - - Concretely: gene set analysis is **ReactomeGSA** (web, at reactome.org/gsa, nothing to install). ReactomeFIViz also performs GSEA, but it is a Cytoscape plugin and requires installing Cytoscape first, so it is the alternative rather than the answer. + - Concretely: gene set analysis on expression data is **ReactomeGSA** (web, at reactome.org/gsa, nothing to install). ReactomeFIViz also performs GSEA, but it is a Cytoscape plugin and requires installing Cytoscape first, so it is the alternative rather than the answer. + - A **list of genes or identifiers** — names only, no measurements for each sample — is not that: it is an **over-representation analysis**, run with *Analyze Data* in the Pathway Browser by pasting the list. Never send a gene list to ReactomeGSA. - Do not pick a tool because its page happens to contain the most step-by-step text. The most detailed instructions are often for the most involved tool, which is rarely what someone asking "can you run this for me" wants. 4. Tone and style: - Write in a clear, friendly, and conversational tone.