Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
580 changes: 311 additions & 269 deletions docs/demo/hero-report.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/getting-started/ab-testing-servers.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,4 +193,4 @@ Description is clear and well-targeted.
- [Comparing Configurations](comparing.md) — More comparison patterns
- [Generate Reports](../how-to/generate-reports.md) — Get AI insights on your comparison

> 📁 **Real Example:** [pydantic/test_10_ab_servers.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/pydantic/test_10_ab_servers.py) — Server version comparison and tool description impact testing
> 📁 **Real Example:** [copilot/test_10_ab_servers.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/copilot/test_10_ab_servers.py) — Configuration A/B comparison
4 changes: 2 additions & 2 deletions docs/getting-started/comparing.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,5 +116,5 @@ This helps you answer:
- [Multi-Turn Sessions](sessions.md) — Test conversations with context

> 📁 **Real Examples:**
> - [pydantic/test_01_basic.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/pydantic/test_01_basic.py) — Single agent workflows
> - [pydantic/test_04_matrix.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/pydantic/test_04_matrix.py) — Multi-dimension comparison
> - [copilot/test_01_basic.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/copilot/test_01_basic.py) — Single-eval workflows
> - [copilot/test_02_models.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/copilot/test_02_models.py) — Model comparison
2 changes: 1 addition & 1 deletion docs/getting-started/iterations.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,4 +142,4 @@ pytest tests/ --aitest-iterations=3 --aitest-min-pass-rate=80
- [Generate Reports](../how-to/generate-reports.md) — Report generation details

> **Real Examples:**
> - [pydantic/test_11_iterations.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/pydantic/test_11_iterations.py) — Iteration baseline tests
> - [copilot/test_11_iterations.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/copilot/test_11_iterations.py) — Iteration baseline tests
2 changes: 1 addition & 1 deletion docs/getting-started/skills.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ The report shows whether the skill improves performance.
- [Multi-Turn Sessions](sessions.md) — Conversations with context

> 📁 **Real Examples:**
> - [pydantic/test_05_skills.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/pydantic/test_05_skills.py) — Skill loading, metadata, and before/after comparisons
> - [copilot/test_05_skills.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/copilot/test_05_skills.py) — Skill loading and A/B comparisons

## Copilot Skills

Expand Down
59 changes: 23 additions & 36 deletions docs/how-to/complete-example.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ description: "Complete end-to-end example demonstrating all pytest-skill-enginee
This guide walks through the **hero test suite** — a comprehensive example demonstrating all pytest-skill-engineering capabilities in a single, cohesive banking scenario.

!!! tip "Generate the Report"
Run `pytest tests/showcase/ -v --aitest-html=report.html` to generate the hero report.
Run `uv run python -m pytest tests/showcase/ -v --aitest-html=report.html` to generate the hero report.

## The Scenario: Personal Finance Assistant

Expand All @@ -24,25 +24,23 @@ This realistic scenario lets us test how well an LLM can understand and coordina
tests/showcase/
├── test_hero.py # The comprehensive test suite
├── conftest.py # Shared fixtures
├── agents/ # Agent instruction files for comparison
│ ├── concise.agent.md
│ ├── detailed.agent.md
│ └── friendly.agent.md
├── instructions/ # System prompt files for comparison
│ ├── concise.md
│ ├── detailed.md
│ └── friendly.md
└── skills/
└── financial-advisor/ # Domain knowledge skill
├── SKILL.md
└── references/
└── budgeting-guide.md
└── SKILL.md
```

## Running the Hero Tests

```bash
# Run all showcase tests with HTML report
pytest tests/showcase/ -v --aitest-html=docs/demo/hero-report.html
uv run python -m pytest tests/showcase/ -v --aitest-html=docs/demo/hero-report.html

# Run a specific test class
pytest tests/showcase/test_hero.py::TestModelComparison -v
uv run python -m pytest tests/showcase/test_hero.py::TestModelComparison -v
```

## 1. Basic Tool Usage
Expand Down Expand Up @@ -187,36 +185,26 @@ The report automatically generates a **model comparison table** showing:
- Token usage and costs
- AI-generated recommendations

## 5. Agent Instruction Comparison
## 5. System Prompt Comparison

Compare how different agent instruction styles affect behavior.

Store each style as an `.agent.md` file:
Store each system prompt style as a Markdown file:

```
tests/showcase/
└── agents/
├── concise.agent.md
├── detailed.agent.md
└── friendly.agent.md
└── instructions/
├── concise.md
├── detailed.md
└── friendly.md
```

```markdown title="agents/concise.agent.md"
---
name: AGENT_CONCISE
description: Brief, to-the-point financial advice
---

```markdown title="instructions/concise.md"
You are a personal finance assistant. Be concise and direct.
Give specific numbers and actionable advice in 2-3 sentences.
```

```markdown title="agents/detailed.agent.md"
---
name: AGENT_DETAILED
description: Thorough financial analysis with explanations
---

```markdown title="instructions/detailed.md"
You are a personal finance assistant. Provide comprehensive analysis.
Explain your reasoning, show calculations, and consider multiple scenarios.
```
Expand All @@ -226,20 +214,19 @@ Then parametrize tests over them:
```python
from pathlib import Path
from pytest_skill_engineering.copilot import CopilotEval
from pytest_skill_engineering.core.evals import load_custom_agent

AGENT_PATHS = list((Path(__file__).parent / "agents").glob("*.agent.md"))
INSTRUCTION_PATHS = list((Path(__file__).parent / "instructions").glob("*.md"))


class TestPromptComparison:
"""Compare how different agent instruction styles affect financial advice."""
class TestSystemPromptComparison:
"""Compare how different system prompts affect financial advice."""

@pytest.mark.parametrize("agent_path", AGENT_PATHS, ids=lambda p: p.stem.replace(".agent", ""))
async def test_advice_style_comparison(self, copilot_eval, llm_assert, agent_path):
@pytest.mark.parametrize("instruction_path", INSTRUCTION_PATHS, ids=lambda p: p.stem)
async def test_advice_style_comparison(self, copilot_eval, instruction_path):
"""Compare concise vs detailed vs friendly advisory styles."""
agent = CopilotEval(
name=agent_path.stem,
custom_agents=[load_custom_agent(agent_path)],
name=instruction_path.stem,
instructions=instruction_path.read_text(),
)

result = await copilot_eval(
Expand Down
2 changes: 1 addition & 1 deletion docs/how-to/test-cli-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,4 +268,4 @@ CLIServer(
)
```

> 📁 **Real Example:** [pydantic/test_09_cli.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/pydantic/test_09_cli.py) — CLI server testing with ls and cat commands
> 📁 **Real Example:** [copilot/test_09_cli.py](https://github.com/sbroenne/pytest-skill-engineering/blob/main/tests/integration/copilot/test_09_cli.py) — CLI workflow testing
138 changes: 47 additions & 91 deletions tests/integration/README.md
Original file line number Diff line number Diff line change
@@ -1,121 +1,77 @@
# Integration Tests
# Copilot Integration Tests

These tests verify pytest-skill-engineering works with real LLM providers. Tests are split into two harnesses — **pydantic** (Eval + eval_run, BYOM) and **copilot** (CopilotEval + copilot_eval, GitHub Copilot SDK).
These tests validate pytest-skill-engineering with real GitHub Copilot models and tools. They use `CopilotEval` with the `copilot_eval` fixture; no mocked LLM execution is used.

## Structure

```text
tests/integration/
├── conftest.py # Shared constants and server fixtures
├── agents/ # .agent.md test fixtures
│ ├── banking-advisor.agent.md
│ ├── todo-manager.agent.md
│ └── minimal.agent.md
├── pydantic/ # Eval + eval_run tests (Azure/OpenAI, BYOM)
│ ├── conftest.py
│ ├── test_01_basic.py # Single eval, basic MCP tool calls
│ ├── test_02_models.py # Model comparison (parametrize)
│ ├── test_03_prompts.py # System prompt comparison
│ ├── test_04_matrix.py # Model × prompt 2×2 grid
│ ├── test_05_skills.py # Skill loading + skill-enhanced behavior
│ ├── test_06_sessions.py # Multi-turn sessions
│ ├── test_07_clarification.py # ClarificationDetection feature
│ ├── test_08_scoring.py # llm_score + ScoringDimension
│ ├── test_09_cli.py # CLIServer wrapping shell commands
│ ├── test_10_ab_servers.py # A/B server comparison
│ ├── test_11_iterations.py # --aitest-iterations=N reliability
│ └── test_12_custom_agents.py # Eval.from_agent_file + load_custom_agent
├── copilot/ # CopilotEval + copilot_eval tests (GitHub Copilot SDK)
│ ├── conftest.py
│ ├── test_events.py # SDK event capture
│ ├── test_01_basic.py # File create + refactor
│ ├── test_02_models.py # Model comparison
│ ├── test_03_instructions.py # Instruction differentiation + excluded_tools
│ ├── test_05_skills.py # Skill A/B comparison
│ └── test_12_custom_agents.py # Custom agents + forced subagent dispatch
└── prompts/ # Plain .md system prompt files
└── skills/ # Test skill directories
├── conftest.py # Shared system prompts and constants
├── agents/ # Custom agent test fixtures
├── prompts/ # System prompt test fixtures
├── skills/ # Skill test fixtures
└── copilot/
├── conftest.py # Copilot models, authentication, and limits
├── test_events.py # SDK event capture
├── test_01_basic.py # Basic file creation and refactoring
├── test_02_models.py # Model comparison
├── test_03_instructions.py # System prompt and tool filtering
├── test_05_skills.py # Skill A/B comparison
├── test_06_sessions.py # Multi-turn sessions
├── test_07_clarification.py # Clarification detection
├── test_08_scoring.py # LLM scoring
├── test_09_cli.py # CLI workflows
├── test_10_ab_servers.py # Configuration A/B comparison
├── test_11_iterations.py # Iteration reliability
├── test_12_custom_agents.py # Custom agent dispatch
├── test_13_plugins.py # Plugin discovery and loading
├── test_14_skill_evals.py # Skill eval execution
├── test_15_skill_refinement.py # Skill refinement
├── test_16_skill_benchmark.py # Skill benchmarking
└── test_17_plugin_skill_workflow.py # End-to-end plugin skill workflow
```

## Quick Start

### Pydantic harness (Azure OpenAI)

```bash
# Prerequisites
az login
export AZURE_API_BASE=https://your-resource.cognitiveservices.azure.com
# Run all pydantic tests
uv run python -m pytest tests/integration/pydantic/ -v
# Authenticate once
gh auth login

# Run all Copilot integration tests
uv run python -m pytest tests/integration/copilot/ -v

# Run a specific file
uv run python -m pytest tests/integration/pydantic/test_01_basic.py -v
uv run python -m pytest tests/integration/copilot/test_01_basic.py -v

# Run a specific test
uv run python -m pytest tests/integration/pydantic/test_01_basic.py::TestBankingBasic::test_balance_check_and_transfer -v
```

### Copilot harness (GitHub Copilot SDK)

```bash
# Prerequisites
uv sync --extra copilot
gh auth login
# Run all copilot tests
uv run python -m pytest tests/integration/copilot/ -v
uv run python -m pytest \
tests/integration/copilot/test_01_basic.py::TestBasicFileCreation::test_create_python_file -v
```

> **CRITICAL:** Never mix harnesses in one session. The plugin raises `pytest.UsageError` if both `eval_run` and `copilot_eval` are collected together.

## Prerequisites

1. **Azure login** (Entra ID auth — no API keys needed):

```bash
az login
export AZURE_API_BASE=https://your-resource.cognitiveservices.azure.com
```

1. **Models available** (checked 2026-02-23):

- `gpt-5.4-mini` — cheapest, use for most tests
- `gpt-5.5` — primary summary/default model
- `gpt-4.1` — alternative provider-backed model

1. **For Copilot tests only:**

```bash
uv sync --extra copilot
gh auth login # or set GITHUB_TOKEN
```

## MCP Test Servers

Built-in test servers in `src/pytest_skill_engineering/testing/`:

- `banking_mcp.py`: `get_balance`, `get_all_balances`, `transfer`, `deposit`, `withdraw`, `get_transactions` for financial workflows.
- `todo_mcp.py`: `add_task`, `complete_task`, `list_tasks`, `delete_task`, `get_task`, `update_task` for CRUD operations.
1. GitHub Copilot authentication through `gh auth login` or `GITHUB_TOKEN`.
2. A model available through the GitHub Copilot SDK.
3. Dependencies installed with `uv sync --all-groups`.

## Adding New Tests
## Adding Tests

Create evals inline using constants from `conftest.py`:
Create evals inline and use the shared constants from `copilot/conftest.py`:

```python
from pytest_skill_engineering import Eval, Provider
from ..conftest import DEFAULT_MODEL, DEFAULT_RPM, DEFAULT_TPM, DEFAULT_MAX_TURNS
from pytest_skill_engineering.copilot import CopilotEval


async def test_my_feature(eval_run, banking_server):
agent = Eval.from_instructions(
"my-agent",
"You are a banking assistant.",
provider=Provider(model=f"azure/{DEFAULT_MODEL}", rpm=DEFAULT_RPM, tpm=DEFAULT_TPM),
mcp_servers=[banking_server],
max_turns=DEFAULT_MAX_TURNS,
async def test_my_feature(copilot_eval, tmp_path):
agent = CopilotEval(
name="my-feature",
model="gpt-5.4-mini",
instructions="Create files as requested.",
working_directory=str(tmp_path),
)

result = await eval_run(agent, "What's my checking balance?")
result = await copilot_eval(agent, "Create hello.py that prints 'hello'.")

assert result.success
assert result.tool_was_called("get_balance")
assert (tmp_path / "hello.py").exists()
```
28 changes: 28 additions & 0 deletions tests/showcase/conftest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
"""Shared configuration for the showcase test suite."""

from __future__ import annotations

import sys
from typing import Any

import pytest

from tests.integration.conftest import BANKING_PROMPT


@pytest.fixture
def banking_mcp_servers() -> dict[str, dict[str, Any]]:
"""Configure the built-in banking MCP server for Copilot sessions."""
return {
"banking": {
"command": sys.executable,
"args": ["-u", "-m", "pytest_skill_engineering.testing.banking_mcp"],
"tools": ["*"],
}
}


@pytest.fixture
def banking_system_prompt() -> str:
"""Return the shared banking system prompt."""
return BANKING_PROMPT
Loading