Skip to content

feat(claude): trim each agent's skill preload and warn on oversized preloads - #137

Closed
knihit wants to merge 3 commits into
feat/agent-skill-indexfrom
feat/claude-preload-trim
Closed

knihit wants to merge 3 commits into
feat/agent-skill-indexfrom
feat/claude-preload-trim

Conversation

@knihit

@knihit knihit commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Description

Claude Code injects the full SKILL.md of every name in an agent's clientConfig.claudeCli.skills
into each subagent it spawns, and that text is re-sent on every turn of the subagent. On main
the agents together preloaded 1,807,398 B across 153 names, and k-architect declared 38 names,
past Claude Code's 32-name cap. This PR cuts the preload to what each agent needs before its first
turn and leaves every other declared skill loadable through the Skill tool.

Base: feat/agent-skill-index (#120). This PR stacks on it; feat/agent-skill-index is merged
into this branch as a merge commit. #120's skills index is Kiro-only (## Your skills, in all 11
Kiro agent files of each Kiro output) and this PR's names block is Claude-only (## Skills you can load, in all 11 Claude agent files), so no agent file carries both.

Agents:

  • k-architect, k-developer, k-product-manager, k-quality-assurance, k-researcher, k-tpm,
    konductor, konductor-mux-orchestrator, konductor-cmux-orchestrator: claudeCli.skills
    trimmed to constraints plus verification or the agent's primary checker where the prompt
    mandates it. dependencies.skills.skillNames is unchanged everywhere, so nothing becomes
    unavailable.
  • k-product-manager, k-tpm, k-quality-assurance: prompts no longer say their skills load
    automatically.
  • konductor, konductor-mux-orchestrator, konductor-cmux-orchestrator: rule 4 now asks for
    each skill in a brief's REQUIRED SKILLS by its exact skills/<name> identifier, so the
    subagent can load the right one on demand. The rule lives in the spec because the Claude Code
    orchestrator never loads delegation-protocol; that skill is unchanged from main.

CLI:

  • konductor synth warns (konductor: warning:) when an agent's preload passes 32,768 B or the
    32-name cap, and names the dropped skills. main's tree prints 9 such warnings; this branch
    prints none.
  • synth -v prints a claude preload: line per agent and --json carries a claude_preload
    array. Both use the same measuring function as the warning.
  • The generated Claude agent file ends with a "Skills you can load" block listing declared
    skills that are not preloaded.
  • New e2e test cli/konductor-rs/tests/synth_claude_preload_e2e.rs fails if any agent in the
    repository exceeds the budget or the cap.
  • cli/README.md documents the warning and the new -v / --json output.

Docs:

  • User guide and both HTML bundles: "Skills" is the declared count, with a separate preloaded
    count. check-guide-facts.py asserts both.

Preload per spawned subagent

Rendered bytes from synth -v, before = main at 7428dd9 synthed with this branch's binary,
after = this branch after merging feat/agent-skill-index. Synthing feat/agent-skill-index
itself gives the same before figures within 78 B (1,807,320 B in total; k-architect 556,639 B,
k-developer 247,820 B), and the after figures are unchanged by the merge.

Agent Before (B) Names After (B) Names Kept preload
k-architect 556,663 38 7,816 2 constraints, verification
k-browser 25,899 4 25,899 4 unchanged
k-developer 247,874 27 15,119 4 constraints, backend-review, frontend-review, verification
k-media-analyzer 2,708 1 2,708 1 unchanged
k-product-manager 174,132 16 2,708 1 constraints
k-quality-assurance 260,121 12 7,816 2 constraints, verification
k-researcher 55,255 8 6,340 2 constraints, external-research
k-tpm 112,859 14 2,708 1 constraints
konductor 141,380 13 13,063 2 constraints, claude-teams-behavior
konductor-cmux-orchestrator 114,097 10 2,708 1 constraints
konductor-mux-orchestrator 116,410 10 2,708 1 constraints
Total 1,807,398 153 89,593 21

k-architect's 38 names exceeded the 32-name cap, so on main Claude Code silently dropped six,
verification among them. Its effective preload was 511,397 B; verification now reaches it.

Benchmark

Claude Code 2.1.312, claude-sonnet-5-5, claude --agent konductor -p <prompt>. Four short cases
(c1 feature + tests, c2 one-line fix, c3 design + threat model, c4 implement + docs + review),
3 reps each, 3 arms: main (7428dd9), trim (this branch), and delete (this branch with an
empty claudeCli.skills on every k-* spec, not committed). 36 of 36 runs intact.

arm USD total USD/run mean input tokens/run mean
main 10.54 0.878 1,567,162
trim 4.32 (-59%) 0.360 531,509 (-66%)
delete 3.57 (-66%) 0.298 461,280 (-71%)

Trim vs main per case (means; min-max ranges of the two arms are disjoint in every case for both
cost and tokens):

case USD input tokens
c1 feature + tests -37% -43%
c2 one-line fix -62% -62%
c3 design + threat model -76% -86%
c4 implement + docs + review -64% -65%

First-request tokens of each spawned subagent, which carry the preload:

agent main trim
k-developer 106,959 to 107,303 (n = 18) 23,651 to 24,341 (n = 21)
k-architect 207,788 to 208,047 (n = 4) 22,395 to 22,431 (n = 3)

The drop matches the bytes removed at about 2.8 bytes per token.

Right skill reached the working agent (9 scored runs per arm; c2 has no expected skill):

arm loaded on demand available (loaded or preloaded)
main 0/9 9/9
trim 5/9 9/9
delete 5/9 5/9

On trim, k-architect loaded threat-modeling on demand in 3/3 c3 runs. The delete arm left the
working agent without the expected checker in 4 of 9 runs (c1 0/3), so an empty preload is not
proposed; this PR keeps the small preload.

Resume: a k-developer subagent resumed twice with SendMessage received the preloaded skill
bodies only on the first segment. Each resume added about 200 input tokens (23,301, 23,500,
23,712 on the first request of each segment), not the preload again. No delegation change is
needed for reviewer resume.

Outcome rubric: main 12/12, trim 9/12, delete 12/12. Each trim miss is a single rubric item
on an otherwise passing run (score 0.89 or higher): two c3 designs state concrete rate limits in
forms the regex does not match (a table, and 1 token/s), and one c4 reply reports the applied
review fix without the words the pattern expects. None of the misses coincides with a skill
failing to load.

Type of Change

  • New or updated agent spec
  • Context / system prompt update
  • CLI (cli/) change, Rust or Python
  • Documentation

Testing

  • npm test passes (runs tests/scripts/**/*.test.js): 119 of 136 pass. The 17 failures are
    identical on main (7428dd9); they need the generated
    scripts/brand-config/lib/constants.js, which a fresh checkout does not have.
  • If cli/ changed: cd cli && make test passes on the merged head: cargo test 1832
    passed, 0 failed, and the __dump_schema smoke check passes. Also on the merged head:
    make build && make synth pass with no konductor: warning: line, make -C cli lint
    passes, make test-skills-sh 85 passed, 0 failed, make test-py 223 passed,
    make claude-plugin-check passes
  • Smoke tested affected agent(s): built and installed via cli/README.md's synth/install
    steps, then ran claude --agent konductor across the 36 benchmark runs above
  • If an agent's or skill's behavior changed: ran a benchmark for the affected agents (Claude
    Code matrix above, 3 reps per case) rather than npm run benchmark
  • New or changed code files carry the required SPDX header (see AGENTS.md's License headers
    rule); formats without comment syntax (e.g. JSON) are exempt

Checklist

  • My code follows the style guidelines of this project (see AGENTS.md's Code Style & Conventions)
  • I have performed a self-review of my own changes
  • I have commented my code where necessary
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings, build is clean, and for cli/ changes,
    cd cli && make lint passes (cargo clippy -- -D warnings + cargo fmt --check)
  • I have added or updated tests that prove my change works (see Testing above for the exact commands)

Agents Affected

  • konductor
  • konductor-mux-orchestrator
  • konductor-cmux-orchestrator
  • k-developer
  • k-architect
  • k-researcher
  • k-product-manager
  • k-quality-assurance
  • k-tpm

k-browser and k-media-analyzer keep their preload unchanged.

Notes for Reviewers

  • Scope: Claude Code only for the preload. On Kiro the output changes are the prompt sentence on
    the three specialist specs and the rule 4 sentence on the three orchestrator specs.
  • The first commit's message lists a delegation-protocol change; the second commit moves that
    sentence into the orchestrator specs, so the skill's net diff against main is empty.
  • The benchmark ran on the first commit (1695160), before the relocation commit and before
    merging feat/agent-skill-index. Neither changes the Claude preload (the after bytes are
    identical on the merged head) and feat(synth): add each Kiro agent's skills and a use rule to its prompt #120 changes only Kiro output, so the figures still apply;
    the merged head was not re-benched.
  • Merge conflict: one hunk in parse_canonical.rs, where both branches added a synth warning
    loop at the same spot. Both loops are kept.
  • Sample size is n = 3 per case, short cases (30 s to 4.5 min), mechanical rubric. Cost and token
    ranges are disjoint from main in every case. On c1 one trim rep spawned 5 subagents (two
    review rounds) and used 1,408,963 input tokens: below main's c1 minimum (1,612,369) but
    less than a third under it. The other two reps cleared a third.
  • The bench does not compare konductor against a single default agent. It measures how much this
    change takes off konductor's own cost on Claude Code.
  • make guide-check has 1 failure, the same one main has today (config listed in
    WITHDRAWN_COMMANDS); no new failures.
  • No VERSION bump; the CHANGELOG entry sits under [Unreleased].

Nihit Kasabwala added 3 commits October 29, 2026 23:12
…reloads

Claude Code injects the full SKILL.md of every name in an agent's
clientConfig.claudeCli.skills into each subagent it spawns, on every
turn. Each agent now preloads only what it needs before its first turn
(constraints, plus verification or the agent's primary checker where the
prompt mandates it); every other declared skill stays loadable through
the Skill tool. dependencies.skills.skillNames is unchanged everywhere.

- synth warns (konductor: warning:) when an agent's preload passes
  32,768 B or Claude Code's 32-name cap, naming the dropped names.
- synth -v prints a "claude preload:" line per agent; --json carries a
  claude_preload array. Both use the same measuring function as the lint.
- The Claude agent file now ends with a "Skills you can load" block
  listing declared skills that are not preloaded.
- k-product-manager, k-tpm and k-quality-assurance prompts no longer say
  their skills load automatically.
- delegation-protocol asks for skills by exact skills/<name> identifier.
- User guide and both HTML bundles: Skills is the declared count, with a
  separate preloaded count; check-guide-facts.py asserts both.
- New e2e test synth_claude_preload_e2e.rs fails if any agent in this
  repository exceeds the budget or cap.

Preload bytes per spawned subagent (synth -v, before = origin/main
7428dd9 synthed with this binary, after = this commit):

  agent                        before B  names  after B  names
  k-architect                   556,663    38     7,816    2
  k-browser                      25,899     4    25,899    4
  k-developer                   247,874    27    15,119    4
  k-media-analyzer                2,708     1     2,708    1
  k-product-manager             174,132    16     2,708    1
  k-quality-assurance           260,121    12     7,816    2
  k-researcher                   55,255     8     6,340    2
  k-tpm                         112,859    14     2,708    1
  konductor                     141,380    13    13,063    2
  konductor-cmux-orchestrator   114,097    10     2,708    1
  konductor-mux-orchestrator    116,410    10     2,708    1
  total                       1,807,398   153    89,593   21

Verification run on this commit:
- make build && make synth: pass; synth prints no preload warning
  (main's tree printed 9).
- cargo test (cli/konductor-rs): 1800 passed, 0 failed.
- make -C cli lint (clippy -D warnings, fmt --check): pass.
- make test-skills-sh: pass.
- make claude-plugin-check: pass.
- make guide-check: 1 failure, the same one origin/main has today
  (`config` listed in WITHDRAWN_COMMANDS); no new failures.
- dist/kiro-cli-v2 and dist/kiro-v3 differ from main's synth only in the
  k-product-manager, k-tpm and k-quality-assurance agent files and the
  delegation-protocol skill.
…or specs

The Claude Code orchestrator follows its spec and never loads
delegation-protocol, so the sentence asking for skills by exact
skills/<name> identifier did not reach it there. It now sits on rule 4
of konductor, konductor-mux-orchestrator and konductor-cmux-orchestrator,
and delegation-protocol's REQUIRED SKILLS line is back to main's text.

Verification run on this commit:
- make build && make synth: pass, no preload warning.
- cd cli && make test: pass (cargo test 1800 passed, 0 failed;
  __dump_schema smoke check passed).
- make -C cli lint: pass.
- make test-skills-sh: 85 passed, 0 failed.
- make test-py: 223 passed.
- make claude-plugin-check: pass.
- npm test: 119 of 136 pass; the 17 failures are identical on main
  (missing generated scripts/brand-config/lib/constants.js).
- make guide-check: 1 failure, the same one main has (`config` in
  WITHDRAWN_COMMANDS).
…t/claude-preload-trim

# Conflicts:
#	cli/konductor-rs/src/cli/synth/parse_canonical.rs
@knihit

knihit commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

Replaced by #151, #152 and #153, which split this change into the preload lint, the agent preload trim, and the docs update. Closing in favor of those.

@knihit knihit closed this Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant