Skip to content

[SC-16225] Add Gemini support#513

Merged
juanmleng merged 4 commits into
mainfrom
juan/sc-16225/add-llm-config-support
May 22, 2026
Merged

[SC-16225] Add Gemini support#513
juanmleng merged 4 commits into
mainfrom
juan/sc-16225/add-llm-config-support

Conversation

@juanmleng
Copy link
Copy Markdown
Contributor

Pull Request Description

What and why?

Added Gemini support across the library’s three LLM evaluation paths: prompt evaluation tests, RAGAS tests, and DeepEval scorers.

Before this change, the shared judge config only supported OpenAI/Azure and Gemini was not wired consistently across all evaluators. After this change, all three evaluation paths can use Gemini, RAGAS handles Gemini finish reasons more reliably, and the judge-configuration notebook now documents the current environment variables and defaults used by the library.

How to test

  1. Comment OPENAI_API_KEY key in your .env
  2. Set GEMINI_API_KEY, and optionally GEMINI_MODEL and GEMINI_EMBEDDINGS_MODEL
  3. Run notebooks/how_to/run_tests/configure_tests/configure_judge_llms.ipynb

What needs special review?

Dependencies, breaking changes, and deployment notes

Release notes

Added Gemini support for ValidMind prompt evaluation tests, RAGAS tests, and DeepEval scorers, and updated the judge-configuration notebook to reflect the current environment variables and defaults.

Checklist

  • What and why
  • Screenshots or videos (Frontend)
  • How to test
  • What needs special review
  • Dependencies, breaking changes, and deployment notes
  • Labels applied
  • PR linked to Shortcut
  • Unit tests added (Backend)
  • Tested locally
  • Documentation updated (if required)
  • Environment variable additions/changes documented (if required)

juanmleng and others added 2 commits May 19, 2026 14:15
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@juanmleng juanmleng self-assigned this May 19, 2026
@juanmleng juanmleng added the enhancement New feature or request label May 19, 2026
Copy link
Copy Markdown
Contributor

@johnwalz97 johnwalz97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice! lgtm

Copy link
Copy Markdown
Contributor

@AnilSorathiya AnilSorathiya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@nrichers nrichers added the support Support-related PR label May 20, 2026
juanmleng and others added 2 commits May 22, 2026 19:13
@github-actions
Copy link
Copy Markdown
Contributor

PR Summary

This PR makes extensive modifications to support an additional LLM provider (Gemini) in the ValidMind Library. Significant changes include:

• Updates to the configuration routines in the AI utilities (in validmind/ai/utils.py) so that the library now selects among OpenAI, Azure, and Gemini providers based on environment variables. The functions now consistently leverage a new helper (_get_configured_provider) and define default model names/constants (e.g. GEMINI_MODEL and GEMINI_EMBEDDINGS_MODEL).

• Refactoring of the get_client_and_model and get_judge_config functions. The judge configuration now correctly calls provider-specific methods, using a Gemini-specific implementation (_build_gemini_judge_config) when appropriate and falling back to the OpenAI method otherwise.

• Introduction of a new function, get_deepeval_model, which returns a native model object (or wraps it) appropriately for DeepEval scorers. This ensures that the Gemini provider is correctly supported, and the scoring modules in the deepeval subpackage have been updated to call get_deepeval_model in lieu of get_client_and_model.

• Updates across multiple scorer implementations (in the deepeval folder) and prompt validation tests to integrate the new provider configuration, ensuring that all evaluation paths consistently use the updated configuration logic.

• Addition of new unit tests in the tests/unit_tests directory to verify proper handling of Gemini credentials and parsing of finish signals in RAGAS tests. The test files have been augmented to mock environment variables for Gemini, confirm that the correct models are instantiated, and validate that the new finish parser behaves as expected.

In summary, this PR unifies LLM evaluation configuration across multiple usage points and extends the support to Gemini-based APIs, ensuring that users with different credentials experience consistent behavior across prompt-validation, RAGAS, and DeepEval workflows.

Test Suggestions

  • Write integration tests that set the GEMINI_API_KEY (or equivalent) environment variable and verify that get_client_and_model returns the expected model configuration.
  • Test the _ragas_is_finished_parser function with different finish reasons including 'stop', 'max_tokens', and other edge cases to ensure it returns True as expected.
  • Add tests for get_deepeval_model to ensure that when Gemini credentials are set, the function imports the appropriate module and returns an instance of the Gemini model.
  • Simulate missing dependency scenarios (e.g., when langchain_google_genai is not installed) to ensure proper ImportError messages are raised.

Copy link
Copy Markdown
Contributor

@cachafla cachafla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great 👍

@juanmleng juanmleng merged commit daca818 into main May 22, 2026
21 checks passed
@juanmleng juanmleng deleted the juan/sc-16225/add-llm-config-support branch May 22, 2026 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request support Support-related PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants