Measure how well an MCP server works for agents, gate it in CI, and turn real agent friction into findings you can fix.
mcpeval scores any MCP server, over stdio or Streamable HTTP, with a
deterministic read-only battery. No LLM is in the loop: the same server and
manifest give the same result. Every lost point comes with the server-side fix.
A privacy-preserving shim records what real agent sessions cost, and repeated failures become findings. A finding closes after three consecutive green verification runs and reopens on regression.
Pre-1.0. The CLI, on-disk schema, and manifest format can change between minor versions. Read the changelog before upgrading.
brew install cavi-ai/tap/mcpeval
npm install --global @cavi-ai/mcp-eval
cargo install mcpeval --lockedEach installs mcpeval and the demo server mcpeval-demo. Homebrew covers
macOS and Linux; npm also covers Windows x64. Both install the release archive
after checking its pinned SHA-256. See Installation
for release archives and source builds.
Score the bundled demo server against mcpeval's standard:
mcpeval score --server demo -- mcpeval-demo
# demo readiness 91/100 protocol=100 catalog=63 context=100 error-honesty=100 reliability=91 coverage=100 standard=mcpeval-standard/2Scaffold a manifest from the live catalog, then gate on it:
mcpeval init --server demo --confirm-read-only \
--output demo.manifest.json -- mcpeval-demo
mcpeval probe --server demo --manifest demo.manifest.json -- mcpeval-demo
# demo gate 25/25 passedBreak one behavior and read the fixed failure reason and its fix:
mcpeval probe --server demo --manifest demo.manifest.json \
-- mcpeval-demo --broken stalled-cursor
# catalog-pagination pagination fail ... reason=pagination-stalled-cursor ...
mcpeval explain pagination-stalled-cursorReplace mcpeval-demo with your server's command, or pass
--url http://127.0.0.1:8080/mcp. The Quickstart
walks through the rest.
| Goal | Guide |
|---|---|
| Score readiness and fix every lost point | Readiness reporting |
| Write and run manifest cases | Capture and probe, Manifest |
| Gate pull requests and upload SARIF alerts | Continuous integration |
| Compare servers or deployments on one manifest | Comparing servers |
| Capture real agent sessions and promote findings | Capture and probe, Findings and verification |
| Let an agent probe, score, and verify over MCP | Agent loop over MCP |
| Check that instructions match tool behavior | Instruction coherence |
| See how 13 popular public servers score | State of MCP servers |
| Collect and replay your own corpus | Corpus collection and replay |
Probes are read-only. A mutating case needs a declared sandbox and
--allow-mutation, and the standard battery never calls a tool annotated as a
writer. Capture and probe records keep shaped arguments, fixed labels, and
counts, never raw argument values, response bodies, or error prose.
Records under <MCPEVAL_HOME>/store/ are content-minimized, but annotation
notes are free-form prose: review or remove them before sharing, or use
mcpeval share, which omits them by default and refuses a snapshot its
redaction sweep flags. The salt, databases, manifests, and corpus files stay
private. See Privacy and authorization.
Hosted at cavi-ai.xyz/docs/mcp-eval.
- Start: Overview, Installation, Quickstart
- Guides: Capture and probe, Readiness reporting, Findings and verification, Agent loop over MCP, Continuous integration, Instruction coherence, Corpus collection and replay, Troubleshooting
- Reference: CLI, Manifest, Evaluation dimensions
- Security and release: Privacy and authorization, Version and support
Packages: GitHub releases, npm, crates.io, Homebrew tap.
Bug reports and focused pull requests are welcome. CONTRIBUTING.md
covers development setup, quality gates, and the privacy rules for fixtures and
tests; .pr-gates lists the commands CI runs. Report
vulnerabilities through SECURITY.md.
Distributed under the MIT License.