Skip to content

feat(synth): add each Kiro agent's skills and a use rule to its prompt - #120

Open
knihit wants to merge 13 commits into
mainfrom
feat/agent-skill-index
Open

knihit wants to merge 13 commits into
mainfrom
feat/agent-skill-index

Conversation

@knihit

@knihit knihit commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Description

Kiro agents rarely load a matching skill unless the user tells them to. An agent's own prompt had no "when" for its skills, and the agent couldn't see which skills existed until it called the deferred skills tool. This PR has synth append a short "Your skills" block to every Kiro agent's prompt. The block holds a use-matching-skills rule, how to load a skill, and the agent's own declared skills. The skills MCP server stays the load path.

Kiro only. Claude Code already lists every skill and loads it with its native Skill tool, with no MCP server involved, and the benchmark showed no measured gain from a rule there (6 against 5 of 12). So Claude Code agent output is unchanged.

Stacked on the fix/skill-name-filter PR, which makes each Kiro agent's skills server serve only that agent's skills. Review and merge that one first.

Synth (cli/konductor-rs/src/cli/synth/):

  • New skill_index.rs builds the block. kiro_cli_v2.rs and kiro_cli_v3.rs append it in their transform() loop.
  • One - <name>: <trigger> line per dependencies.skills.skillNames entry, in declared order. The trigger is the description's first Use ... or MUST USE sentence, kept whole, led by the first sentence when that is a different one, so the line keeps what the skill does. The first sentence is trimmed on a word boundary to fit 160 characters, or dropped when less than 40 would be left. With no Use sentence the trigger is the first sentence alone, capped at 160. Em and en dashes become hyphens.
  • The load line says to use get_skill on the konductor-skills MCP server, calling tool_load first if the tool is deferred. Both names come from the new cli/skills_server.rs, which install now shares, so renaming the server or tool can't leave the prompt pointing at a stale name.
  • The rule: before a non-trivial task, check whether a listed skill covers it, load at most one or two, follow their steps and output format, and skip trivial tasks. Workspace ws-* skills also count and are read from .kiro/skills/<name>/SKILL.md, matching core rule 10.
  • A skill whose trigger starts with MUST USE for every (today only constraints) is left out of the list, with no line about it. It claims every task, which would contradict the one-or-two rule. A task-specific MUST USE for <task> trigger, such as git-merge, stays in the list. Synth prints a warning to stderr when any skill other than constraints matches the rule, so a new one isn't dropped silently.
  • Orchestrators (konductor*) also get a sentence telling them to route skill-matching work to the owning specialist, and to load from the list only for their own orchestration work.
  • render_agent_file, the agent file structs, claude.rs, the agent specs and context/ are unchanged. Nothing is written to ~/.kiro/skills or .kiro/skills.

Tests:

  • Unit tests in skill_index.rs cover trigger extraction (including the first sentence leading a later Use sentence, checked against the real backend-review description), sentence splitting (AGENTS.md stays whole), the cap, dash conversion, declared order, the orchestrator paragraph, the one-or-two wording, leaving constraints out, keeping git-merge, and the every-task warning.
  • kiro_block_names_the_installed_server_and_tool checks the block names the shared server and tool constants and the ws-* path, and grants_are_built_from_the_shared_server_and_tool_names in mcp_server.rs checks the install grants are built from the same constants.
  • New tests/synth_skill_index_e2e.rs runs a real synth. For every agent, the Kiro v2 and v3 index names equal the source spec's skillNames minus constraints, in declared order, constraints is not named, and no Claude Code agent file carries a skill block. An agent with no skills must have no block, rather than failing the test.

Docs:

  • docs/user-guide/reference.md ("Behaviour worth knowing" under synth output) and cli/README.md (synth) now note the generated ## Your skills block and that skills whose description starts "MUST USE for every" (today only constraints) are left out. The published HTML guide doesn't carry this sentence, so it is unchanged.

Skills (descriptions, all harnesses):

  • progress-tracking: the Use sentence listed modes, not tasks, so its index line gave a task nothing to match. It now says "Use when validating a feature plan, tracking its progress, or checking requirements against code."
  • adversarial-code-review-pass-security, -integrity, -schema: the second sentence now reads "Use only when the adversarial-code-review coordinator spawns it", so the index doesn't present them as standalone reviews.
  • These descriptions also reach Claude Code, whose native skill matching reads them, and they changed after the benchmark ran.

Changelog:

  • One bullet under [Unreleased] / Changed. No VERSION change.

Kiro block size per agent (from make synth, identical on v2 and v3; the Skills column counts declared skills, and constraints is not listed):

Agent Skills Block (B)
k-architect 38 6,841
k-developer 27 5,128
k-product-manager 16 3,183
konductor 13 2,798
k-tpm 14 2,720
k-quality-assurance 13 2,502
konductor-mux-orchestrator 11 2,339
konductor-cmux-orchestrator 11 2,338
k-researcher 9 1,901
k-browser 5 1,133
k-media-analyzer 2 677

k-developer's block is about 1,250 tokens. For comparison, #113 adds about 17 KB to every agent.

Type of Change

  • CLI (cli/) change: Rust
  • Context / system prompt update (generated at synth for Kiro. No source prompt or context file changed.)

Testing

  • npm test: 119 of 136 pass. The same 17 fail on main (claude-code-agent-runner, generate-agent-files, run-claude-subset), and this PR touches none of their code. Not rerun after the review fixes, which change no JS.

  • cd cli && make test: 1809 passed, 0 failed. telemetry_hook_scoping, which this PR doesn't touch, failed once under parallel load on macOS and passed with --test-threads=1 and on a full rerun.

  • mcp make test: 325 passed. make -C cli lint passes. make -C mcp lint reports cargo fmt diffs in five skill-lookup test files; main reports the same five under the same toolchain. python3 -m pytest -q tests/: 222 passed. make guide-check: the one failure (config in WITHDRAWN_COMMANDS) is also on main. make test-skills-sh 10 passed, 1 skipped (existing exclude), make claude-plugin-check pass, make test-py 222 passed, kiro-power-check passed.

  • Synth output: dist/claude is byte-identical to main for every agent (diff -r clean).

  • Smoke tested: make build && make synth, then installed with --from . --no-telemetry into a scratch HOME and target. The kiro-cli v3 k-developer prompt ends with the block, and its 27 index names equal its --skill-name-filter names. .kiro/skills holds only sop-* entries, and nothing was written under HOME's .kiro.

  • Benchmark run against main. The Kiro block in that build matched this branch before the review fixes, which changed it (re-bench below). The bench numbers are from 3662fe8. The later commits (the ws-* sentence and first-sentence-led triggers) grow the 11 v2 blocks from 25,804 to 31,560 bytes (+22%; k-architect 5,464 to 6,841, k-developer 4,064 to 5,128) and have not been benchmarked. That build also had a names-only Claude Code line, which this branch dropped after the run.

    Setup: 4 tasks that each match one agent's skill (T1 backend review, T2 user stories, T3 threat model, T4 test coverage; direct, plus T1 and T2 through konductor) and a no-skill control. 2 reps per cell, project-scope install with --no-telemetry, isolated HOME per run, scored from session logs rather than agent self-reports.

    Harness Variant n Loaded a skill Right skill Followed skill format Control loaded a skill
    Kiro CLI v3 main 12 4 3 0 0/2
    Kiro CLI v3 this branch 12 9 9 4 0/2
    Claude Code main 12 5 5 4 0/2
    Claude Code names-only line (dropped) 12 6 6 4 0/2

    On Kiro the right skill loaded unprompted in 9 of 12 runs, against 3 of 12 on main. 8 of those 9 went tool_load then get_skill on the agent's own skill, and the skills server was reported unavailable in 0 of 28 Kiro runs. Mean Kiro cost per task run rose from 2.10 to 2.56 credits. On Claude Code the line gave 6 against 5 of 12, within run-to-run noise, which is why this PR leaves Claude Code alone.

    Caveats:

    • Each task cell has only 2 runs, and main varied by about 1 run in 12 between two benchmark passes.
    • Two Kiro loads came from an agent reading a project-scope SKILL.md directly (one on main, one here). Without them, Kiro is 2/12 against 8/12.
    • T1 is still weak: backend-review loaded in 1 of 8 direct runs, and konductor answered T1 itself instead of delegating in all 8 orchestrated runs.

    Follow-up comparison on Kiro CLI v3, same harness, model and scoring: X is this branch before the review fixes (constraints listed as a normal line), Y is the final head (constraints left out). Task runs only (T1 to T4, T1 and T2 also through konductor). Reps 1 and 2 are the passes above; reps 3 and 4 ran X and Y interleaved run by run, and a 60-credit cap stopped them after 21 runs (rep 4 lacks T3 for Y and T4 for both).

    Variant Runs Right skill Loaded constraints Mean credits per run
    main (reps 1 and 2) 12 3 n/a 2.10
    X, reps 3 and 4 11 6 3 2.55
    Y, reps 3 and 4 10 6 0 2.58
    X, pooled 23 15 6 2.56
    Y, pooled 22 13 0 2.68
    Task, pooled right skill X Y
    T1 direct 1/4 0/4
    T1 orchestrated 0/4 0/4
    T2 direct 4/4 4/4
    T2 orchestrated 3/4 3/4
    T3 direct 4/4 3/3
    T4 direct 3/3 3/3

    The X vs Y gap is within noise. Across the 22 matched X/Y runs, only 3 disagree: 2 favour X (T1 direct rep 2, T2 orchestrated rep 1) and 1 favours Y (T2 orchestrated rep 4). The one difference that holds is constraints: X loads it in 6 of 23 runs and Y never does. Cost per run is the same within noise; the mean gap comes from the T2 orchestrated runs, which swing between 1 and 7.5 credits. No run loaded a wrong skill, and no run reported the skills server unavailable. T1 stays the weak task on both.

  • All three new .rs files start with // SPDX-License-Identifier: Apache-2.0

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own changes
  • I have commented my code where necessary
  • I have made corresponding changes to the documentation (CHANGELOG, docs/user-guide/reference.md and cli/README.md)
  • My changes generate no new warnings: make -C cli lint (cargo clippy -- -D warnings, cargo fmt --check) passes
  • I have added or updated tests that prove my change works

Agents Affected

Every agent's Kiro CLI v2 and v3 prompt gains the block. Claude Code agents are unchanged.

  • konductor
  • konductor-mux-orchestrator
  • konductor-cmux-orchestrator
  • k-developer
  • k-architect
  • k-researcher
  • k-product-manager
  • k-quality-assurance
  • k-tpm
  • k-browser
  • k-media-analyzer

Notes for Reviewers

  • The rule paragraphs live in skill_index.rs because they name the MCP server and tool through the same constants the installer uses, so the prompt and the install can't disagree. Moving them to a template would bring that drift back. They are Kiro-only and say less than core rule 10 and the routing rules, which still govern.

  • Own commits: the first adds the block on all three harnesses, and the second drops the Claude Code part after the benchmark. After review, a merge brings in the fix/skill-name-filter review fixes, then four commits address this PR's review: shared server names (with a first take on constraints), the docs, keeping git-merge listed, and leaving constraints out of the index. Four more follow from the next round: is_orchestrator moves into synth/model.rs, the rule names ws-* skills, triggers lead with the first sentence, and the every-task rule is stated in the docs with a synth warning. Two more keep the Use sentence whole when it fits and name the expected every-task skill as a constant. Review the combined diff.

  • Overlap with fix(orchestrator): tighten the direct-work rules #114: the only textual conflict is CHANGELOG.md [Unreleased]. fix(orchestrator): tighten the direct-work rules #114 adds "delegate a step you lack the skill for" to the routing rules, and the orchestrator sentence here says the same thing for skill-matching work. Re-read the combined wording once both land.

  • Overlap with feat(synth): deny subagent spawning for specialist agents #124: feat(synth): deny subagent spawning for specialist agents #124 adds model::is_specialist (prefix k-) and changes kiro_cli_v3.rs's use super::model::CanonicalModel; to use super::model::{is_specialist, CanonicalModel};. This PR adds model::is_orchestrator (prefix konductor) to synth/model.rs, after the CanonicalModel struct, so the two predicates sit in one file. A trial git merge-tree against feat(synth): deny subagent spawning for specialist agents #124 merges model.rs and CHANGELOG.md cleanly. The one conflict is in kiro_cli_v3.rs's imports: this PR adds use super::parse_canonical::skill_names_from_dependencies; right below the line feat(synth): deny subagent spawning for specialist agents #124 rewrites, and the fix is to keep both. Once both land, is_orchestrator could be defined as the complement of is_specialist, unless a third role prefix is expected.

  • constraints is not listed. Its NEVER DO and writing rules reach agents through the core rules, and the remaining rules are a known gap tracked separately. On main Kiro agents don't see it listed either. A first fix listed it on an always-apply line outside the one-or-two budget, and agents then loaded it in 6 of 10 runs. Listed as a normal line, it loaded in 6 of 23 runs with no measurable effect on the right-skill rate (15/23 against 13/22 without it, within noise), so this PR leaves it out.

  • If the MCP server binary is missing at install, the server is skipped with a warning while the prompt still names get_skill. That install was already broken before this change.

@knihit
knihit added this pull request to stack #121 October 27, 2026 23:40
@knihit
knihit marked this pull request as ready for review October 28, 2026 02:44
knihit added a commit that referenced this pull request Oct 28, 2026
The constraints skill reaches an agent only when the agent loads it. context/k-core-rules.md is always loaded by all 11 agents, so the general engineering practices now live there and the skill keeps only the workflow rules.

Context:

    context/k-core-rules.md: added an ## Engineering practices section after Writing habits, with 8 one-line rules: read the whole file before editing, follow codebase patterns, keep backward compatibility unless the user approves a break, comment the why, structured logging with correlation IDs when adding or changing logging in service code (other code follows its existing logging style), research first when the cause chain is unclear, say whether an unverified claim that matters is documented, inferred, or a guess, and confirm tests pass before calling a code change done. Because every agent loads these, the last two are scoped: non-code agents have no tests to confirm, and an orchestrator gets test evidence from the specialist rather than running tests, which its routing rules forbid. NEVER DO and Writing habits are unchanged.

Skills:

    skills/constraints/SKILL.md: ALWAYS 1, 2, 4, 5, 6, 7, Unknown Systems, and Acknowledge Uncertainty now point at the core rules instead of restating them, so there is one copy of each rule. Maker-checker, severity ordering, the 2-strike circuit breaker, and the Quality Gate tiers stay in the skill.
    constraints description: kept the MUST USE for every interaction first sentence, because feat(synth): add each Kiro agent's skills and a use rule to its prompt #120 classifies always-apply skills by that prefix and tests it for constraints. Rewrote the second sentence, which claimed the skill defines rules that now live in the core rules.

Docs:

    CHANGELOG.md: one bullet under [Unreleased] / ### Changed. No VERSION bump.
    User guide (concepts.md, faq.md, appendix/contributing.md, skills.md): the core rules are now described as NEVER DO, writing habits, and engineering practices, and the constraints row lists what the skill keeps. The four matching entries in tools/build-user-guide/sync-html-content.py now take the current bundle text as old; both HTML bundles and the manifest are regenerated.
Base automatically changed from fix/skill-name-filter to main October 28, 2026 18:49
Nihit Kasabwala and others added 13 commits October 28, 2026 14:49
…ng-skills rule

Synth now appends a "Your skills" block to every agent's prompt.

- Kiro CLI v2 and v3: one "- <name>: <trigger>" line per declared
  dependencies.skills.skillNames entry, in declared order. The trigger
  is the description's first "Use ..."/"MUST USE" sentence, else its
  first sentence, capped at 160 chars on a word boundary, with em and
  en dashes turned into hyphens. The block tells the agent to load a
  listed skill with get_skill on the konductor-skills MCP server,
  calling tool_load first if the tool is deferred.
- Claude Code: names only, from clientConfig.claudeCli.skills, loaded
  with the Skill tool. Claude already lists every skill description
  natively, so a second description index would only add bytes.
- Every block carries the rule: before a non-trivial task, check
  whether a listed skill covers it, load at most one or two, follow
  their steps and output format, skip trivial tasks. Orchestrators
  (konductor*) also get a paragraph telling them to route
  skill-matching work to the owning specialist.

The block is appended in each transformer's transform() loop, so
render_agent_file, render_agent_md and the agent file structs are
unchanged. Nothing is written to ~/.kiro/skills or .kiro/skills.

Byte cost from make synth (Kiro block / Claude block): k-developer
(27 skills) 4,140 / 954, k-architect (38) 5,540 / 1,250, konductor
(13) 2,657 / 847, k-media-analyzer (2) 665 / 342.

Verification (TMPDIR=/tmp):
- make build, make synth: pass
- make -C cli test: 1802 passed, 0 failed (no flake reruns needed;
  resolve_cwd_json_envelope and telemetry_hook_scoping passed)
- make -C mcp test-rust: 322 passed, 0 failed
- make test-skills-sh: 10 passed, 0 failed
- make claude-plugin-check: pass
- make test-py kiro-power-check (venv, pytest 8.3.4, PyYAML 6.0.2):
  222 + 12 passed, kiro-power 27 run, 0 failed
- cargo clippy -- -D warnings and cargo fmt --check: clean
- Hermetic install (--from . --no-telemetry, scratch HOME and
  target): kiro-v3 k-developer prompt ends with the block, its 27
  index names equal the --skill-name-filter names, .kiro/skills holds
  only sop-* entries, nothing under HOME/.kiro. Claude install body
  ends with the names-only line.
Claude Code already lists every skill and loads it with its native
Skill tool, with no MCP server involved. The benchmark showed no
measured gain from the names-only block there (6 of 12 runs loaded
the right skill, against 5 of 12 on main, within noise), so it only
added bytes to every Claude agent.

- claude.rs is back to its fix/skill-name-filter content.
- skill_index.rs drops claude_skill_block, the Claude load line and
  their tests. The Kiro block is unchanged.
- The e2e test now asserts that no synthesized Claude agent carries a
  "Your skills" block or the orchestrator paragraph.
- CHANGELOG bullet says Kiro.

Verification (macOS, TMPDIR=/tmp):
- make build, make synth: pass (11 agents, 82 skills).
- dist/claude is byte-identical to a make synth of fix/skill-name-filter
  for every agent (diff -r clean). dist/kiro-cli-v2 and dist/kiro-v3
  are byte-identical to a make synth of the previous commit.
- make -C cli test: 1799 passed, 0 failed (3 Claude-block tests removed).
- make -C mcp test-rust: 322 passed, 0 failed.
- make test-skills-sh: 10 passed, 0 failed.
- make claude-plugin-check: pass.
- make test-py (venv, pytest 8.3.4): 222 passed; make kiro-power-check:
  pass (10 and 5 run, 0 failed; shellcheck not installed, skipped).
- make -C cli lint (clippy -D warnings, cargo fmt --check): pass.
… server names

The skills block listed `constraints: MUST USE for every interaction` under a
rule capping loads at one or two. Skills whose trigger starts with MUST USE
now render on a separate always-apply line outside that budget, loaded at
most once per session, and the rule reads "in addition to any skill marked
always-apply, load at most one or two".

The server and tool names now come from one shared module used by both
install and synth, so a rename cannot leave the prompt pointing at a
missing tool. The e2e test no longer panics for an agent with no skills.

Verified:
- make -C cli test (TMPDIR=/tmp): pass, 1760 unit tests plus integration suites
- make -C mcp test: pass, 325 tests
- make -C cli lint (clippy and fmt): pass
- make build and make synth: pass
- make test-skills-sh: 10 passed, 1 skipped by existing exclude list
- make claude-plugin-check: pass
- make test-py (pytest 8.3.4, PyYAML 6.0.2): 222 passed
- make kiro-power-check: pass
- dist/claude identical to a synth of main
Verified:
- make synth: pass; dist/kiro-cli-v2 prompts end with the described block
- dist/claude identical to a synth of main
Only a trigger starting with "MUST USE for every " goes on the
always-apply line. git-merge ("MUST USE for merging branches") stays a
task skill with its trigger, so the mux orchestrators no longer load it
on every session.

Verified:
- TMPDIR=/tmp make -C cli test: 1806 passed, 0 failed
- TMPDIR=/tmp make -C mcp test: 325 passed, 0 failed
- make -C cli lint (clippy, fmt): pass
- make build, make synth: pass; mux orchestrators list git-merge as a task skill
- make test-skills-sh: 10 passed, 1 skipped (excluded)
- make claude-plugin-check: pass
- make test-py (pytest 8.3.4, PyYAML 6.0.2): 222 passed
- make kiro-power-check: pass
- dist/claude identical to main (diff -r, no differences)
The always-apply line for `constraints` invited agents to load it on most
tasks. A re-bench loaded it in 6 of 10 runs and lowered the right-skill
rate. Skills whose trigger starts "MUST USE for every " are now left out
of the index, with no extra line, and the use rule is back to "If one
matches, load at most one or two". A "MUST USE for <task>" skill such as
git-merge stays listed.

Verified: make -C cli test (1805 passed; telemetry_hook_scoping, which
this branch does not touch, failed once under parallel load and passes
with --test-threads=1 and on the full rerun), make -C mcp test, make -C
cli lint, make build, make synth, make test-skills-sh, make
claude-plugin-check, make test-py (222), make kiro-power-check;
dist/claude identical to main bb7ef9c.
Agent-role predicates belong with the canonical model so other
transformers can share them.
The rule now says ws-* skills count too and where to read them, matching
core rule 10.
When the "Use ..." or "MUST USE" sentence is not the description's
first sentence, the line now leads with the first sentence, then the
160-character cap trims the pair.
The docs now give the rule itself: skills whose description starts
"MUST USE for every" are left out of the Kiro skills index. Synth warns
on stderr when a skill other than constraints matches it.
The Use sentence is what a task is matched against, so it is kept whole
and the summary sentence is trimmed to the room left, or dropped when
less than 40 characters remain. Before this, a long summary could push
the Use sentence past the 160-character cap.
…rsarial passes coordinator-only

progress-tracking's Use sentence listed modes, not tasks, so its index
line said nothing a task could match. The three adversarial-code-review
pass skills now say they run only when the coordinator spawns them, so
the index no longer presents them as standalone reviews. Unit tests use
both real descriptions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant