Skip to content

feat: add opt-in skill-routing modifier - #547

Merged
bompus merged 3 commits into
mainfrom
skill-routing-modifier
Nov 5, 2026
Merged

bompus merged 3 commits into
mainfrom
skill-routing-modifier

Conversation

@bompus

@bompus bompus commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Adds an opt-in skill-routing modifier (off by default, nothing changes for existing setups): at the start of each task the agent checks what the user is doing and picks the one available skill that best fits, keeps it for follow-ups and small edits, picks again on a new task or changed goal, and uses none (without saying so) when none fits.

Problem. With many skills installed, agents often finish a task without calling one that fits, even when the request matches the skill's description. Hosts load only each skill's description up front, so the model has to notice the match on its own.

Evidence (Claude Code 2.1.318, claude plugin eval, a 60-skill catalog, only the Skill tool allowed, 3 runs per case, tool-call graders on the raw trace, no model judge):

  • 18 existing routing cases with no other rules: skill calls on the should-fire prompts rose from 30/42 to 42/42 on Sonnet and from 13/42 to 42/42 on Haiku. Every miss without the block was "no skill called", never a wrong skill.
  • 10 new, uncoached prompts with a full set of user rules in both arms (6 that fit a skill, 4 that fit none): Sonnet 13/18 to 18/18, Haiku 11/18 to 14/18 on the five bullets in this change. The four no-skill prompts called no skill in every arm (12/12 each). A rerun of the six-bullet version gave the same counts as its first run.
  • Without the bullet that asks the agent to name the skill it used, Sonnet matched the six-bullet version and Haiku was within one trial.

Limits. Small: 3 runs and six fit-a-skill prompts in the second test; the first test's cases were written for their skills and were not held out. Only the first message of a task was tested, so the "keep the same skill for follow-ups" bullet has no measurement. The block was supplied as appended system text.

Why a modifier. Some users prefer to rely on skill descriptions alone, and the cost is a little extra reasoning at the start of each task (about 0.1 to 0.5 extra turns per trial here). It composes after Session purpose.

Changes: rules/modifiers/skill-routing.md, a README modifier row, a CHANGELOG entry under Unreleased. bun test test/ (974 pass, 0 fail) and bun run check pass locally.

@bompus bompus added author:anthropic Pull request written by a anthropic model author-model:sonnet-5-5 A model that wrote part of this pull request labels Nov 5, 2026

@bompus-ai-review bompus-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bompus AI Review (GPT-6.1 Terra, max reasoning; SWE-2 Max, max reasoning): 1 finding(s).

Comment thread rules/modifiers/skill-routing.md Resolved
@bompus
bompus force-pushed the skill-routing-modifier branch from 2b55238 to 75eb55a Compare November 5, 2026 00:59

@bompus-ai-review bompus-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bompus AI Review (GPT-6.1 Terra, max reasoning; SWE-2 Max, max reasoning): 1 finding(s).

Comment thread rules/modifiers/skill-routing.md Resolved

@bompus-ai-review bompus-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bompus AI Review (GPT-6.1 Terra, max reasoning; SWE-2 Max, max reasoning): 0 finding(s).

Review 3: only high findings and above are posted from here on.

@bompus-ai-review bompus-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bompus AI Review (GPT-6.1 Terra, max reasoning; SWE-2 Max, max reasoning): 0 finding(s).

Review 4: only high findings and above are posted from here on.

@bompus bompus closed this Nov 5, 2026
@bompus
bompus merged commit b575917 into main Nov 5, 2026
2 checks passed
@bompus
bompus deleted the skill-routing-modifier branch November 5, 2026 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author:anthropic Pull request written by a anthropic model author-model:sonnet-5-5 A model that wrote part of this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant