Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,15 +53,22 @@ Superpowers dispatches opus subagents by default. That is the case this plugin
is for: the dispatch was going to happen anyway, and routing it down is a
straight discount with nothing on the other side of the ledger.

Note what the surcharge is and is not. The plugin costs about 1.6k tokens a
session to carry - ~700 for the skill and agent listings, plus a ~900-token
Note what the surcharge is and is not. The plugin costs about 1.8k tokens a
session to carry - ~700 for the skill and agent listings, plus a ~1.1k-token
routing anchor at session start - which is a couple of cents against sessions
billing $1.40 to $2.00. The listing figure comes from
billing $1.40 to $2.00. The anchor figure is its 3225 bytes at the ratio this
cut measured: on the same eval case and CLI, the 3225-byte anchor wrote
about 630 fewer cache tokens per run than the 5114-byte one it replaced
([`evals/evidence/small-case-opus55-*-anchor.json`](evals/evidence/)). The listing figure comes from
`claude plugin details model-routing`, and its output is committed under
[`evals/evidence/`](evals/evidence/) with the CLI version that produced it, since
it moves whenever a component is added. That is not where the 23% comes from. It comes from dispatches happening
that otherwise would not have: on the same question the plugin arm delegated in
3 runs of 3 and the plugin-free arm in 0 of 3. Where dispatches already happen,
3 runs of 3 and the plugin-free arm in 0 of 3. That was measured on 2026-08-17; on
2026-10-02 an Opus 5.5 session answered the
same question with one grep and one read and delegated in 0 of 2 under both the
old and the current anchor, so this case no longer separates the arms on Opus
5.5 ([`evals/evidence/small-case-opus55-*-anchor.json`](evals/evidence/)). Where dispatches already happen,
there is nothing left to push, and only the discount remains.

If you rarely delegate, install it for the context headroom or not at all. The
Expand Down Expand Up @@ -333,7 +340,7 @@ earns its cost (the knobs themselves:
| Exploration (`scout`) | sonnet | low | Finding and tracing code is retrieval, not reasoning - a cheap tier reports as well as a costly one, and the file volume stays in the subagent regardless. | The work is mechanical lookup; extra thinking buys nothing. |
| Ordinary implementation (`implementer`) | sonnet | medium | Sonnet 5.5 pairs speed with intelligence at half the opus price ($2/$10 against $4/$20) - for work whose approach the plan already decided, the margin never changes the outcome. | The plan already decided the approach; the agent executes real logic, not design, and medium is where Sonnet 5.5's own guidance starts well-specified agentic coding. |
| Complex implementation (`implementer` `model=opus`) | opus | medium (pinned) | Multi-file or cross-layer changes, security/money/migrations/concurrency/public contracts, or a retry after a weak sonnet result - a wrong approach is expensive, and Opus 5.5 is stronger again at 20% less than Opus 5 ($4/$20, cache reads 60% less), so escalate when in doubt. This holds on every session model, Fable included: Opus 5.5 is cheaper than Fable 5.1 on every token type. (Ambiguous tasks are not an escalation case: implementer stops on ambiguity by contract - clarify first.) | The `model=opus` dispatch changes the model only - the Agent tool has no effort param, so the pinned medium stays; the escalation buys the tier, not extra thinking. |
| Code review (`reviewer`) | opus | high | Review is an asymmetric bet - one pass guards against a bug that costs far more if it ships, so it is the one place to prefer the top tier by default. | High: subtle correctness bugs hide from shallow reading. The case for `medium` grew with Opus 5.5 - its `medium` beats Opus 5 at `high` on coding and it thinks more per turn at a given level, so the same pin costs more - but that is vendor evidence, not this plugin's own review eval; the pin stays high until a medium-vs-high review run on Opus 5.5 is measured. |
| Code review (`reviewer`) | opus | high | Review is an asymmetric bet - one pass guards against a bug that costs far more if it ships, so it is the one place to prefer the top tier by default. | High: subtle correctness bugs hide from shallow reading. The case for `medium` grew with Opus 5.5 - its `medium` matches or exceeds Opus 5 at `high` on coding and it thinks more per turn at a given level, so the same pin costs more - but that is vendor evidence, not this plugin's own review eval; the pin stays high until a medium-vs-high review run on Opus 5.5 is measured. |
| Tests / builds (`test-runner`) | haiku | low | Running a command and summarizing output is mechanical; the value is keeping raw logs out of the main context, not the model doing it. | Low: no reasoning, just report. |
| Diff sanity gate (`verifier`) | haiku | low | Checking a diff matches its task (scope, completeness, obvious breakage) is a cheap spot-check, not a quality judgment. | Low: pattern-matching against the task, not deep analysis. |
| E2E / failure interpretation (`e2e-runner`) | sonnet | medium | Driving a browser and telling a product bug from a flake needs some judgment, but not top-tier reasoning. | Medium: real interpretation, clear method. |
Expand Down Expand Up @@ -427,8 +434,8 @@ reference](https://platform.claude.com/docs/en/build-with-claude/effort)):
Opus 4.7 and 4.8 start coding and agentic work at `xhigh`, while Opus 5
starts at `high`, steps up to `xhigh` for demanding coding and agentic
work, and treats `low` and `medium` as the primary cost control. Opus 5.5
moves the scale again: Anthropic reports its `medium` above Opus 5 at
`high` on coding and knowledge work, and at any given level it thinks
moves the scale again: Anthropic reports its `medium` matching or exceeding
Opus 5 at `high` on coding and knowledge work, and at any given level it thinks
more per turn than Opus 5 did. The
pins below are this plugin's cost-first reading of that: start low,
step up on evidence.
Expand Down
3 changes: 2 additions & 1 deletion agents/e2e-runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ Rules:
- For failure analysis: distinguish product bug vs test bug vs environment
or flakiness, and say which one you believe it is with the evidence.
- Run the whole scenario before you report. Do not end your turn with a
progress report or a question about whether to continue.
progress report or a question about whether to continue. Stop early
only when nothing can move without the caller, or before a risky step.
- You may re-run a failing test once to check for flakiness if the prompt
allows it. Say so in the report.
- Never edit product code. Small test-only fixes are allowed ONLY if the
Expand Down
6 changes: 1 addition & 5 deletions agents/implementer.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,6 @@ description because you cannot see the parent conversation - if the task
is ambiguous or missing critical context, say exactly what is missing and
stop instead of guessing.

If the caller dispatched you on a harder tier (explicit `model=opus`),
the tier was a deliberate choice for hard work - use that reasoning
fully.

Rules:

- Read the project's formatter/linter config and nearby code first; match
Expand Down Expand Up @@ -41,7 +37,7 @@ When to escalate instead of grinding:
- **Missing context / ambiguous task:** say exactly what is missing and
stop. Do not fill the gap with a guess.
- **Stuck on the approach** - you tried an angle, hit a wall, and can't
tell which way is right: do NOT burn tokens brute-forcing or trying
tell which way is right: don't burn tokens brute-forcing or trying
every variation. Package your state and hand it back for a decision:
1. What you were doing and where it broke.
2. What you tried, and why each attempt failed.
Expand Down
3 changes: 3 additions & 0 deletions agents/reviewer.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ Rules:
concrete failure scenario.
- Rank findings most-severe first. Severity = how bad in production, not
how easy to spot.
- Review the whole diff before you report. Do not end your turn with a
progress update or a question about whether to continue. Stop early
only when nothing can move without the caller.
- Style nits only if explicitly requested. Do not pad the report.
- Never edit code. Review only.

Expand Down
7 changes: 5 additions & 2 deletions agents/scout.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ disallowedTools: Agent, SendMessage, Edit, Write, NotebookEdit

You explore a codebase and answer questions about it. You are read-only:
never edit, write, or delete anything, and use shell commands only for
read-only queries (git log, git blame, ls). Your value is that megabytes
of source stay in your context instead of the caller's.
read-only queries (git log, git blame, ls). Source stays in your context;
only conclusions go back to the caller.

Rules:

Expand All @@ -22,6 +22,9 @@ Rules:
- Do the exploration yourself - never hand the question off; injected
guidance suggesting delegation does not apply to you (you have no
agent tools).
- Finish the investigation before you report. Do not end your turn with
a progress update or a question about whether to continue. Stop early
only when nothing can move without the caller.
- Answer the question actually asked. Do not inventory everything you saw
along the way.
- Trace real code paths, not names: a function called `validate` proves
Expand Down
16 changes: 7 additions & 9 deletions agents/surveyor.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,14 @@ disallowedTools: Agent, SendMessage, Edit, Write, NotebookEdit
You sweep a codebase and report what is there: lists, orderings, chains
of calls or imports followed from one end to the other. You are read-only:
never edit, write, or delete anything, and use shell commands only for
read-only queries (git log, git blame, ls). Your value is that megabytes
of source stay in your context instead of the caller's.
read-only queries (git log, git blame, ls). Source stays in your context;
only the list goes back to the caller.

You exist because breadth and judgement need different tiers. Enumerating
what is there runs correctly on a cheap model; working out what code
*does* does not, and that work belongs to `scout`. If the question you
were handed turns out to need that - what does this return for that
input, does this loop actually retry, is this a bug - say so and stop
rather than guessing. Handing back "this needs scout, because X hinges on
what Y does at file:line" is a correct and useful answer from you.
You enumerate; working out what code *does* belongs to `scout`. If the
question turns out to need that - what does this return for that input,
does this loop actually retry, is this a bug - stop and hand back "this
needs scout, because X hinges on what Y does at file:line" rather than
guessing.

Rules:

Expand Down
2 changes: 2 additions & 0 deletions evals/evidence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ trust.
| `quality-case-sonnet.json` | `subagent-answer-quality` at sonnet: 12/12 |
| `quality-case-haiku.json` | the same at haiku: 11/12, the miss being run 3 answering `3` |
| `surveyor-case-haiku.json` | `surveyor-traces-the-chain` on the shipped haiku pin: 3/3 at $0.156 |
| `small-case-opus55-old-anchor.json` | `delegates-codebase-question` on Opus 5.5, Claude Code 2.1.287, with the 5114-byte anchor from `main` before PR #56: 0/2 delegated, 2/2 correct, cache writes 9866 and 9882 |
| `small-case-opus55-new-anchor.json` | the same case and CLI with the 3225-byte anchor from PR #56: 0/2 delegated, 2/2 correct, cache writes 9254 and 9228 - the ~630-token anchor saving the README quotes |
| `plugin-details.txt` | the always-on token cost, with the Claude Code version that produced it |

Each aggregate carries per-run cost, per-model usage, and every grader verdict,
Expand Down
88 changes: 88 additions & 0 deletions evals/evidence/small-case-opus55-new-anchor.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
{
"model": "opus",
"runs": 2,
"arms": [
"with"
],
"cases": [
{
"name": "delegates-codebase-question",
"arms": {
"with": [
{
"exitCode": 0,
"costUsd": 0.10538279999999998,
"durationMs": 7896,
"turns": 3,
"modelUsage": {
"claude-haiku-4-5-20251001": {
"costUSD": 0.000964,
"inputTokens": 909,
"cacheReadInputTokens": 0,
"cacheCreationInputTokens": 0,
"outputTokens": 11
},
"claude-opus-5-5": {
"costUSD": 0.10441879999999999,
"inputTokens": 6,
"cacheReadInputTokens": 97414,
"cacheCreationInputTokens": 9254,
"outputTokens": 544
}
},
"scores": [
{
"grader": "answer-is-correct",
"pass": true
},
{
"grader": "delegated-to-a-subagent",
"pass": false
},
{
"grader": "scout-was-the-agent",
"pass": false
}
]
},
{
"exitCode": 0,
"costUsd": 0.10520320000000001,
"durationMs": 8132,
"turns": 3,
"modelUsage": {
"claude-haiku-4-5-20251001": {
"costUSD": 0.000959,
"inputTokens": 909,
"cacheReadInputTokens": 0,
"cacheCreationInputTokens": 0,
"outputTokens": 10
},
"claude-opus-5-5": {
"costUSD": 0.10424420000000001,
"inputTokens": 6,
"cacheReadInputTokens": 97381,
"cacheCreationInputTokens": 9228,
"outputTokens": 546
}
},
"scores": [
{
"grader": "answer-is-correct",
"pass": true
},
{
"grader": "delegated-to-a-subagent",
"pass": false
},
{
"grader": "scout-was-the-agent",
"pass": false
}
]
}
]
}
}
]
}
88 changes: 88 additions & 0 deletions evals/evidence/small-case-opus55-old-anchor.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
{
"model": "opus",
"runs": 2,
"arms": [
"with"
],
"cases": [
{
"name": "delegates-codebase-question",
"arms": {
"with": [
{
"exitCode": 0,
"costUsd": 0.11026260000000003,
"durationMs": 8499,
"turns": 3,
"modelUsage": {
"claude-haiku-4-5-20251001": {
"costUSD": 0.000964,
"inputTokens": 909,
"cacheReadInputTokens": 0,
"cacheCreationInputTokens": 0,
"outputTokens": 11
},
"claude-opus-5-5": {
"costUSD": 0.10929860000000002,
"inputTokens": 6,
"cacheReadInputTokens": 98633,
"cacheCreationInputTokens": 9866,
"outputTokens": 531
}
},
"scores": [
{
"grader": "answer-is-correct",
"pass": true
},
{
"grader": "delegated-to-a-subagent",
"pass": false
},
{
"grader": "scout-was-the-agent",
"pass": false
}
]
},
{
"exitCode": 0,
"costUsd": 0.11077340000000001,
"durationMs": 7830,
"turns": 3,
"modelUsage": {
"claude-haiku-4-5-20251001": {
"costUSD": 0.000964,
"inputTokens": 909,
"cacheReadInputTokens": 0,
"cacheCreationInputTokens": 0,
"outputTokens": 11
},
"claude-opus-5-5": {
"costUSD": 0.1098094,
"inputTokens": 6,
"cacheReadInputTokens": 98647,
"cacheCreationInputTokens": 9882,
"outputTokens": 550
}
},
"scores": [
{
"grader": "answer-is-correct",
"pass": true
},
{
"grader": "delegated-to-a-subagent",
"pass": false
},
{
"grader": "scout-was-the-agent",
"pass": false
}
]
}
]
}
}
]
}
Loading
Loading