Inconsistent annotation across runs

[gboscagli]

Hi all,
thanks for developing this workflow for cell type annotation.

I recently saw a blogpost of yours remarking that CyteType is now deterministic and able to retrieve the exact same annotation (and confidence score) across different runs. I then updated to v0.19.4 and followed the basic usage from the README.
However, I am still obtaining substantially different reports from the same anndata object (just re-running annotator.run with a different prefix). The same behaviour also happens with CyteTypeR v0.4.2.

Is the new approach already in place?

Best,
Gabriele

[parashardhapola]

Hi @gboscagli!

Thanks! The changes are live since this morning. Even though the workflow is fully largely deterministic and we are running LLMs at 0 temperature with very top k and top p values, there is potential for randomness, and this mainly arises from LLM inference providers. To support all our users, we ourselves rely on a pool of LLM inference providers who themselves use various underlying deployment stack that can add to non-determinism.

Im curious how different do your results look between two runs?

[gboscagli]

Hi @parashardhapola, and thanks for the quick response.

Im curious how different do your results look between two runs?

Results are similar and reliable between the two runs. Whenever the labels are quite different, I can always find a more similar one among the other candidates investigated. Still, the confidence score for the same cluster could be "High" in one run and "Moderate" in the other one, and the macro-groupings also differ in naming and size. Apart from reproducibility reasons, one may want to avoid post-hoc manual refinement and label reconciliation as much as possible.

Sorry if the word "substantially" was misleading: I didn't mean "very different biological annotation", I just wanted to pinpoint that the technical difference across runs remained of the same magnitude even when uploading to the latest version in both R and Python implementations, and I was curious to know if the changes were already live. Completion time indicated in the reports is 10AM of yesterday morning.

The changes are live since this morning.

I saw that no newer version was released since last week, do I have to update anything or set some parameter to ensure determinism?

[parashardhapola]

Hi @gboscagli,

Thanks for raising this, really helpful feedback.

Results are similar and reliable between the two runs. Whenever the labels are quite different, I can always find a more similar one among the other candidates investigated.

That's good to know, and it's one of the core reasons we expose the candidate information. Since LLMs are inherently non-deterministic, our approach is to bring determinism through convergence: CyteType evaluates many candidate annotations across multiple hypothesis and review rounds, then funnels them down to one final call. This multi-step process performs well in practice, but when results do diverge between runs, it's usually due to underlying heterogeneity in the data. This could be multiple cell types coexisting in a cluster (under-clustering), true mixed-lineage signaling, or technical artifacts. CyteType attempts to flag mixed-lineage and artifact cases explicitly, but it won't always succeed.

Still, the confidence score for the same cluster could be "High" in one run and "Moderate" in the other one, and the macro-groupings also differ in naming and size. Apart from reproducibility reasons, one may want to avoid post-hoc manual refinement and label reconciliation as much as possible.

Agreed. Macro-grouping is still a work in progress and can vary noticeably between runs. The reason is that this step happens post-hoc, after all clusters are annotated, and currently relies on a single LLM call. We're actively working on making it more robust and consistent.

Sorry if the word "substantially" was misleading: I didn't mean "very different biological annotation", I just wanted to pinpoint that the technical difference across runs remained of the same magnitude even when updating to the latest version in both R and Python implementations, and I was curious to know if the changes were already live. Completion time indicated in the reports is 10AM of yesterday morning. I saw that no newer version was released since last week, do I have to update anything or set some parameter to ensure determinism?

No action needed on your end. The Python and R packages are clients and don't run the annotation themselves. The actual workflow runs server-side on cytetype.nygen.io, which is how we're able to provide access to all the LLMs at the scale CyteType needs, along with hosted reports. The improvements were deployed to the server independently of client releases, so you'll benefit from them automatically.

We're also working on making the server changelog more visible and surfacing the server version directly in the reports. We'll update this issue once those improvements are live.

[gboscagli]

Hi @parashardhapola,

Thanks for your thorough response.

I initially opened the issue because, in the blogpost linked above, CyteType output is told to be 100% identical and consistent at every run, which seemed strange to me since it's LLM-based. Current behaviour and future steps are now much clearer, and I'm glad that this paved the way for a broader conversation that I think will come useful to many other users.

Best,
Gabriele

[josegarciamanteiga]

Hi both,
I also see this as one of the top issues in convincing bioinformaticians to use AI-native tools. The reproducibility vs accuracy test is the key. I saw you did that in the benchmarking different LLMs and it was very reproducible if used with always the same models but might be repeated in the free-tier with open/closed models random settings of providers as well, as you mentioned before, due to the differences in LLM inference providers that are used to save resources in the free-tier. I guess such a benchmark could be done with the same metrics you used and even correlated to heterogeneity and confidence flags to convince potential users that the tool, yet not deterministic is robust enough even in the free-tier version.
Best
Jose