From 6624adb2a4422a4df02829465476cd8318b3ef5b Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Wed, 9 Sep 2026 19:35:24 +0000 Subject: [PATCH] Record the ragas metrics deprecation and why it is not a rename Every evaluator run warns that ragas.metrics is deprecated in favour of ragas.metrics.collections. Reading that as an import change is wrong: two of the four are renamed, and the new classes take the judge as a constructor argument rather than through evaluate(), so a straight swap raises TypeError. Moving is a rework that may also move the scores. Written down because the warning invites a rename that does not work, and because the cheapest moment to change scoring is before anyone is comparing against the numbers -- which is now. --- src/evaluation/README.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/src/evaluation/README.md b/src/evaluation/README.md index d34abc64..c1c7fcf6 100644 --- a/src/evaluation/README.md +++ b/src/evaluation/README.md @@ -92,6 +92,26 @@ it is unrelated to the vectors in the bundle, and it is pointed at Plant Reactome host points at a self-hosted endpoint that does not serve it. `JUDGE_BASE_URL` overrides. +## A deprecation this file carries knowingly + +The four metrics are imported from `ragas.metrics`, which warns on every run: + +> Importing Faithfulness from 'ragas.metrics' is deprecated and will be removed +> in v1.0. Please use 'ragas.metrics.collections' instead. + +Not a rename. The replacements exist but two are renamed — `ResponseRelevancy` +is `AnswerRelevancy`, `LLMContextRecall` is `ContextRecall` — and they take the +judge model as a **constructor** argument rather than through `evaluate()`: + +``` +TypeError: Faithfulness.__init__() missing 1 required positional argument: 'llm' +``` + +So moving is a rework of how metrics are built, and it may change the scores, +which for a measurement tool is the part that needs care rather than the import. +Worth doing before this tool has produced numbers anyone is comparing against — +which is now. + ## Reading the output A single run has no noise floor: retrieval is not deterministic (Chroma's ANN