Flakey retrieval

[GFJHogue]

The following user queries lead to v1.2.0 not finding any matching documents during retrieval:

• Tell me about TP53
• Tell me about glycolysis

However some still do appear to work:

• Tell me about BRCA1

This is likely a bug introduced during the big refactor (#77) associated with adding support for profiles, along with the uniprot prototype (#78).
Certainly something in this diff: 1.1.2...1.2.0

There's probably a misnamed parameter that's not getting passed during retrieval.

---

Update

This issue is more mysterious than it initially seems.
The description above corresponded to running 1.2.0 vs 1.1.2 on my local machine, as well as observations on prod.
However I wasn't able to reproduce the issue on release.reactome.org...
Going to look into potential embedding bundle inconsistencies.

[GFJHogue]

This is bizarre - it seems to be working in all cases now.

What I've observed in the past hour along with the accounts of team members, corroborates that this issue is not connected to our 1.2.0 release nor any embeddings-bundle corruptions/inconsistencies.

[GFJHogue]

Okay, one more comment on this.
There is a likelihood that this issue is related to poor query rephrasing during the preprocessing step.

Specifically, the "Tell me about glycolysis" question is sensitive to context and seems to usually fail when it's the 1st message of a conversation, but will succeed if phrased slightly differently like "Tell me about the glycolysis pathway".

FYI @heliamoh

[GFJHogue]

[GFJHogue]

Using actual the tagged Docker image for 1.1.2 (public.ecr.aws/reactome/reactome-chatbot:2044791) gives the following:
(refusal, but at least it's giving some information)

I think this may be a key hint.

[GFJHogue]

Copying from #94 (fixes):

I identified multiple issues in the retrieval system, that together cause the inconsistent behaviours noted in #91.

BM25Retrieval was relying on the default whitespace word-splitting tokenization without any additional text processing to normalize for case (#92) and punctuation. I added case normalization and added nltk's word tokenizer. This seems to help for cases of failed queries involving gene symbols (NFAT1).
The old version of langchain-chroma we were using was not properly returning Documents from SelfQueryRetriever. This appears to have been a bug that is now fixed in the version I've upgraded to here. This caused EnsembleRetriever and MergerRetriever to miss vector search results (glycolysis).
For reasons I still do not understand, RAG would occasionally claim no results found for the 1st message, even if identical query/rephrasing results in valid responses in all other cases. I've inserted the user's original message into chat_history only for the 1st message, which strangely seems to work as a workaround.

[CodeSharingPartially]

Hello! First of all thanks for the great work to improve and keep this tool alive. Really useful.
Second of all, I am testing the chatbot and , starting with very basic questions , I got semi-right answers?
My question was:
1- how could i retrieve with a cypher query, as a json, all the pathways, subpathways in molecules for the pathway "Immune system"?
Before raising this question, I already had in mind a specific event in the Event Hierarchy - Immune system https://reactome.org/PathwayBrowser/#/R-HSA-168256

The result was straightforawrd, and it went to the closest term with pathway instead of asking me if I refere to the "event". I think this is a common behavior with no agentic workflows. Ideally you would like to clarify the question I think, so the bot first will ask if i want "the pathway or the event"? otherwise it is assuming I want the pathway immune system within the disease event instead of the immune system event. anyway i left the snapshots here, (NOTE: It feels also that it though i was talking about pathways all the time and it did not gave me any description for the Event R-HSA-168256 )

NOTE2: In the last Reactome Hackathon, we used Google ADK Toolkit to develop a chatbot with agentic workflow: opentargets-archive/hackathon-tasks#107 and I think it may improve the chatbot workflow.

[GFJHogue]

Hi @CodeSharingPartially,

The result was straightforawrd, and it went to the closest term with pathway instead of asking me if I refere to the "event". I think this is a common behavior with no agentic workflows. Ideally you would like to clarify the question I think, so the bot first will ask if i want "the pathway or the event"? otherwise it is assuming I want the pathway immune system within the disease event instead of the immune system event.

Thank you for the valuable feedback and use-case!

The agentic design of the current system certainly has room for improvement, especially where Human-In-The-Loop interactions would be desirable.
In the semi-near-term we have plans to deploy an improved retrieval system that should reduce the chat's stubbornness of focusing on pathways.

[CodeSharingPartially]

One last question, is Reactome running in AuraDB? or is it in house hosted (community version?)