RAGAS answer_relevancy regression (0.488, threshold likely 0.7+)

[n0nuser]

Context

PR #47 (langchain-community<0.4 pin, merged 67506f8) fixed the ragas import breakage in CI. Its CI run completed the full 23-item offline eval and surfaced a pre-existing, unrelated failure:

faithfulness           0.942  PASS
answer_relevancy       0.488  FAIL
context_precision      0.891  PASS
context_recall         0.957  PASS

Run: https://github.com/n0nuser/LocalRAG/actions/runs/28981479102/job/86001024771

Why this is separate from #47

PR #47 touched only pyproject.toml/uv.lock (dependency pin). It did not touch prompts, retrieval, or the eval dataset. The relevancy drop is inherited from main as of the chore(deps): upgrade uv.lock commit (e9b9968) or earlier — need to bisect.

Suspects to check

• Judge model (gemma3:4b via Ollama) scoring answer_relevancy inconsistently on CPU vs prior baseline.
• Whether langchain-community 0.3.31 (post-pin) vs whatever ragas's embeddings/LLM factory resolved before e9b9968 changed embedding behavior used by answer_relevancy's cosine-similarity step.
• Dataset drift in evals/dataset.json — any recently added adversarial questions (e.g. "What is the capital of Mars?", "How do I fix ERR_LOCALRAG_42?") intentionally designed to test refusal/graceful-failure behavior that could tank relevancy scoring if the judge penalizes non-answers.
• Whether the threshold itself in evals/run_evals.py needs recalibrating for this judge model.

Next step

Bisect evals/results/*.json history if available, or re-run eval against a few recent commits to find where the score dropped.