AgentGrade

Open-source evaluation framework for AI customer support agents.
Measure tone, accuracy, compliance, and resolution quality — configurable rubric, provider-agnostic, LLM-as-judge.

---

Why AgentGrade?

Most AI agent teams have no systematic way to measure support quality. AgentGrade gives you a repeatable, rubric-driven evaluation pipeline.

Built and dogfooded on GlamShelf Twin, a production AI support agent handling real WhatsApp and Instagram DMs for an e-commerce brand.

---

How It Works

1. Provide Transcripts: Input conversation logs formatted in JSON.
2. LLM-as-Judge Evaluation: AgentGrade calls an LLM with structured rubric prompts for each target dimension.
3. Structured Returns: Receives score, detailed rationale, and flagged terms.
4. Multiple Output Formats: Choose between table, JSON, and summary statistics formatting.

---

agentgrade eval examples/sample-conversation.json \
  --provider https://api.groq.com/openai/v1 \
  --model llama-3.3-70b-versatile

The compliance dimension flagged: "completely safe", "100% hypoallergenic", "absolutely never cause any burning or allergic reaction" — exact quotes from the conversation.

---

Installation

Install the package locally in development/editable mode:

pip install -e .

Quick Start

AgentGrade works with any OpenAI-compatible provider — DeepSeek, Groq (free), OpenAI, or local models.

First, set your API key. Pick your provider:

# Groq (free tier)
export GROQ_API_KEY="your-groq-key"

# DeepSeek
export DEEPSEEK_API_KEY="your-deepseek-key"

# OpenAI
export OPENAI_API_KEY="your-openai-key"

Then run:

# Using Groq's free LLaMA 3.3 70B
agentgrade eval examples/sample-conversation.json \
  --provider https://api.groq.com/openai/v1 \
  --model llama-3.3-70b-versatile

# Using DeepSeek
agentgrade eval examples/sample-conversation.json

# Using any OpenAI-compatible endpoint
agentgrade eval examples/sample-conversation.json \
  --provider https://api.openai.com/v1 \
  --model gpt-4o

CLI Command Options

Evaluate Options

• --rubric / -r PATH: Path to a custom YAML rubric file (defaults to the built-in quality rubric).
• --provider / -p BASE_URL: Base URL of the OpenAI-compatible completions endpoint (defaults to https://api.deepseek.com/v1).
• --model / -m TEXT: Model name to use for evaluation (defaults to deepseek-chat).
• --api-key / -k TEXT: API key for the provider (fallback to DEEPSEEK_API_KEY environment variable).
• --output / -o [table|json|summary]: Output formatting option (defaults to table).

Example using custom output formats:

agentgrade eval examples/sample-conversation.json --output summary
agentgrade eval examples/sample-conversation.json --output json

Rubric Initialization

You can export the default quality rubric as a starting template for your custom rubrics:

agentgrade init-rubric custom-rubric.yaml

Documentation

For full guides and reference manuals, please visit our Documentation Portal (Placeholder).