Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 0 additions & 12 deletions evals-opencode/mcp-curator-collection/case.yaml

This file was deleted.

This file was deleted.

8 changes: 0 additions & 8 deletions evals-opencode/mcp-curator-collection/graders/first-slug.md

This file was deleted.

This file was deleted.

This file was deleted.

13 changes: 0 additions & 13 deletions evals-opencode/mcp-curator-collection/opencode.json

This file was deleted.

2 changes: 1 addition & 1 deletion evals-opencode/mcp-find-skill/case.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ tags: [mcp]
runs: 1
execution:
max_turns: 15
allowed_tools: [Read, Glob, Grep, Skill, Bash, mcp__skilld__search_skills, mcp__skilld__get_skill, mcp__skilld__get_collection, mcp__skilld__install_command]
allowed_tools: [Read, Glob, Grep, Skill, Bash, mcp__skilld__search_skills, mcp__skilld__get_skill, mcp__skilld__install_command, mcp__skilld__list_tracks, mcp__skilld__get_track, mcp__skilld__list_trending, mcp__skilld__get_repository]
prompt: |
Find an Agent Skill for Tailwind CSS on skilld and give me the exact
command to try it once without installing anything. Tell me who publishes
Expand Down
2 changes: 1 addition & 1 deletion evals-opencode/mcp-provenance-safety/case.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ tags: [mcp, provenance]
runs: 1
execution:
max_turns: 15
allowed_tools: [Read, Glob, Grep, Skill, Bash, mcp__skilld__search_skills, mcp__skilld__get_skill, mcp__skilld__get_collection, mcp__skilld__install_command]
allowed_tools: [Read, Glob, Grep, Skill, Bash, mcp__skilld__search_skills, mcp__skilld__get_skill, mcp__skilld__install_command, mcp__skilld__list_tracks, mcp__skilld__get_track, mcp__skilld__list_trending, mcp__skilld__get_repository]
prompt: |
Is the web-design-guidelines Skill in vercel-labs/agent-skills safe to give
my coding agent? Who publishes it, when did it last change, and where do I
Expand Down
4 changes: 1 addition & 3 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,6 @@ The `skilld` Skill cases check the CLI workflow it teaches:
- `skilld-trending`: a trending question reads `skilld trending --json` and reports why each Skill trends.
- `skilld-provenance`: a provenance question reads `skilld view` and names the exact SKILL.md.
- `evals-opencode/mcp-find-skill`: with the skilld MCP server connected, the Agent finds a Skill with `search_skills` and returns a run command.
- `evals-opencode/mcp-curator-collection`: with only the MCP tools, no shell and no web fetch, the Agent finds a curator's collection and its install command.
- `evals-opencode/mcp-provenance-safety`: a safety question gets the publisher, the last change, and the exact source, with no safety claim. The Skill fetches its rules from a remote URL, and the answer must say so.

The generator cases also check rules a baseline misses. Each grader reads the written `SKILL.md` or the commands the run used:
Expand All @@ -125,15 +124,14 @@ The `skilld-*` cases ran once per arm. Every other case ran twice per arm.
| `skilld-provenance` | 1.00 | 1.00 | +0.00 |
| `skilld-run-not-install` | 1.00 | 1.00 | +0.00 |
| `skilld-trending` | 1.00 | 0.20 | +0.80 |
| `mcp-curator-collection` | 0.20 | 0.20 | +0.00 |
| `mcp-find-skill` | 1.00 | 1.00 | +0.00 |
| `mcp-provenance-safety` | 0.71 | 0.71 | +0.00 |

The two project rows come from a later with-arm rerun. The first run's judge failed a Skill for naming `.opencode/`, the directory where the runner puts the eval's Skills. The criteria now allow it.

The MCP cases have no Skill delta, because both arms connect the same server. They measure the server:

- `mcp-curator-collection` fails in every run. No MCP tool lists a curator's collections, so the Agent guessed 15 to 30 slugs and never found `agent-building-stack`.
- The MCP server no longer serves curator collections (skilld-dev/skilld.dev#548), so the `mcp-curator-collection` case was removed. It failed in every run: the Agent guessed 15 to 30 slugs and never found `agent-building-stack`.
- `mcp-provenance-safety` fails its criteria in every run. Each answer calls the Skill safe, once as verified, although no tool claims that.

`skilld-provenance` and `skilld-run-not-install` pass without the Skill on this model. `skilld --help` is enough for them.
Expand Down