feat: add AI-agent assisted migrations to create-plugin update by jackw · Pull Request #2774 · grafana/plugin-tools

jackw · 2026-07-13T09:02:56Z

What this PR does / why we need it:

Lets create-plugin update leverage locally installed AI agent CLIs across the migration pipeline. Some upgrade work can't be reliably codemodded (user-authored configs, plugin source refactors); this PR gives migrations a way to ship natural-language instructions that an agent applies under user supervision.

How it works

Three migration shapes: script-only (today's codemods), prompt-only (a prompt .md instructions file), and hybrid (codemod runs first, agent finishes). All 12 existing migrations are untouched; v1 ships infrastructure plus an unregistered example prompt (prompts/000-example.md) documenting the authoring contract.
Detection & opt-in: supported agents (Claude Code, Codex) are detected on PATH via which. When a queued migration has a prompt and an agent is installed, update asks before using it; --agent[=<id>] / --no-agent decide up front.
Interactive sessions with a handoff contract: the agent runs with inherited stdio so the user can watch and redirect it. Completion is signalled by a JSON handoff file (ephemeral single-run IPC under node_modules/.cache/grafana-create-plugin/migrate-runs/, wiped every run) that create-plugin polls, then tears the session down (SIGINT → 5s → SIGKILL). Failed handoffs abort the update un-stamped; ambiguous exits ask the user.
Commit isolation: agentic runs soft-force per-migration commits (unless the user passed --commit/--no-commit themselves) so each agent step sees an isolated git diff; hybrids embed the codemod's changed-file list from the staged Context when commits are off.
CI stays safe: non-TTY short-circuits to disabled before any prompt is constructed, and the agentic module is only imported when a prompt migration is queued — existing workflows are unaffected. Skipped agent steps are printed as manual next steps.
Inside-agent inception: when update itself runs inside an agent session (env sniffing: CLAUDECODE, CODEX_THREAD_ID, etc.), no inner agent is spawned — deferred steps are printed as an XML directive block for the outer agent to complete.

Verification

381 unit tests passing (new colocated suites for every agentic module + manager/registry extensions); typecheck and lint clean.
E2E against a throwaway plugin with a temporarily registered prompt migration: CI-style non-TTY (silent defer + next steps), --no-agent, explicit --agent on non-TTY (warn + degrade), and the inside-agent directive block all verified live; version stamping and exit codes correct in every mode.
Installed claude CLI confirms the --system-prompt and --allowedTools invocation args.

Which issue(s) this PR fixes:

n/a

Special notes for your reviewer:

The full interactive agent session needs a manual run at a real terminal (scaffold a plugin, link the local build, register a test prompt migration, run update). The Codex invocation args mirror the Nx implementation but are unverified against a local install.
Deliberately out of scope (follow-ups): OpenCode support, Nx-style --validate (agent validates script-only migrations), the first real prompt/hybrid migration, and surfacing deferred next-steps in the CI auto-update PR body.

🤖 Generated with Claude Code

…gistry Migrations can now be script-only, prompt-only, or hybrid (script + prompt). Prompt-only migrations are skipped by the runner until the agentic runtime lands. Includes an unregistered example prompt documenting the authoring contract.

Prompt .md files are resolved relative to migrations.js at runtime via import.meta.resolve, so they must ship verbatim alongside the compiled migration scripts. Verified with a build smoke: dist/codemods/migrations/prompts/000-example.md exists.

AGENT_DEFINITIONS is pure data describing how to detect and interactively invoke Claude Code and OpenAI Codex. Claude invocations pre-authorize the handoff write via --allowedTools so sessions cannot stall on a permission prompt.

…ironments detectInstalledAgents probes each agent definition on the PATH via which, falling back to well-known install paths checked for the execute bit. isInsideAgent sniffs the environment variables agent CLIs set so update never spawns an agent from inside another agent.

The handoff JSON file is the completion signal for interactive agent sessions: ephemeral single-run IPC under node_modules/.cache, wiped at the start of every run, parsed with a strict valibot schema.

XML-sectioned system and user prompts with escaping of all file-derived content, a git-or-file-list hybrid change context (capped at 50 entries), an inside-agent directive block for deferred steps, and the manual next-steps list.

Spawns the agent CLI with inherited stdio, polls for the handoff file, and tears the session down with SIGINT escalating to SIGKILL. Exits are classified as handoff, user-aborted (130/143/SIGINT/SIGTERM), or ambiguous so the caller can prompt the user.

…d opt-in Resolution order: inside-agent env wins, --no-agent disables, non-TTY degrades (warning only when a flag was explicit), --agent=<id> pins and validates, bare --agent auto-picks or shows a picker, and no flag asks the user. Cancelled enquirer prompts are treated as a decline.

…, not HEAD The per-migration commit happens after the agent step so it can include the agent's edits — at agent time the codemod changes are flushed but uncommitted. Point the agent at git status/git diff instead of git show HEAD.

createAgenticRuntime resolves the mode once per run, wipes stale migrate runs, soft-forces per-migration commits when the user did not set --commit, runs prompt steps through the session runner (deferring in disabled and inside-agent modes), and prints the end-of-run directive block, next-steps list, or applied recap.

The runtime is lazily imported only when a queued migration has a prompt. Hybrid migrations run their codemod first, then the agent step before the per-migration commit so agent edits are included. Deferred prompt steps still stamp the version; a failed agent step aborts un-stamped.

The agent flag is stripped from codemodOptions so it never reaches codemod schemas, and --commit is forwarded as a tri-state so agentic runs can soft-force per-migration commits only when the user did not decide.

Covers the opt-in flow, agent flag forms, CI behavior (agent steps are skipped and surfaced as next steps in the job log), inside-agent deferral, and the prompt migration authoring contract.

joshhunt

Interesting! I initially thought it would be better to use an agent to write a deterministic codemod, but I understand your reasoning to use agents to perform actions that are tricky to determinately code up front.

Just to make sure - the intention here is that migrations within this repo are written as 000-migration.md and plugin developers in them with the create-plugin tooling? Plugin developers themselves wouldn't be writing migrations like this in their repo?

I'm unsure how to feel about this. It feels like an awkward mixing of tool fidelity. I completely understand the advantages this has, but I think if I run create-plugin update I'm expecting something deterministic to happen. Maybe, could these be surfaced as skills for users to invoke in their coding agent instead?

joshhunt · 2026-07-14T09:17:24Z

+    id: 'claude-code',
+    displayName: 'Claude Code',
+    binaryNames: ['claude'],
+    wellKnownPaths: [join(homedir(), '.claude', 'local', 'claude')],


fwiw, i think it's more straight forward to just not try to guess where else claude could be other than path. feels almost kind of sneaky to do this.

jackw · 2026-07-15T10:37:16Z

Interesting! I initially thought it would be better to use an agent to write a deterministic codemod, but I understand your reasoning to use agents to perform actions that are tricky to determinately code up front.

I've written nearly all the codemods in this repo using an agent skill. I've had claude open a PR with it here.

Just to make sure - the intention here is that migrations within this repo are written as 000-migration.md and plugin developers in them with the create-plugin tooling? Plugin developers themselves wouldn't be writing migrations like this in their repo?

Yes, migrations continue to live in this repo rather than plugin devs writing their own migrations in plugin repos.

Drop the well-known-path fallbacks from agent detection per review feedback. If an agent binary isn't on PATH it isn't set up for CLI use, so probing hardcoded install locations added complexity without a legitimate use case.

jackw · 2026-07-15T12:30:44Z

I'm unsure how to feel about this. It feels like an awkward mixing of tool fidelity. I completely understand the advantages this has, but I think if I run create-plugin update I'm expecting something deterministic to happen. Maybe, could these be surfaced as skills for users to invoke in their coding agent instead?

It is a bit of an odd one but I'm trying to think of ways to leverage our AI overlords to make updates easier on devs. I'm thinking these prompts will be able to fix the things that we can't codemod (e.g. customisations devs have made to eslint, webpack that we can't possibly know or want to cater for).

I'm not sure about skills - they don't have a migration manager and registry so a skill would need to reimplement these things to know what to run and when. Then there's the whole agent format fragmentation between claude, codex, whatever new agent where as in this way we have a single prompt file that serves them all.

Maybe we gate spawning strictly behind --agent so by default update doesn't ask and skipped prompts just get printed as next steps (with a note about the flag)?

jackw added 15 commits July 13, 2026 10:20

feat(create-plugin): add agent handoff contract and run dir lifecycle

50c5b69

The handoff JSON file is the completion signal for interactive agent sessions: ephemeral single-run IPC under node_modules/.cache, wiped at the start of every run, parsed with a strict valibot schema.

feat(create-plugin): add agentic prompt builders

088d655

XML-sectioned system and user prompts with escaping of all file-derived content, a git-or-file-list hybrid change context (capped at 50 entries), an inside-agent directive block for deferred steps, and the manual next-steps list.

fix(create-plugin): type hybrid migration fixture with a required prompt

553bf09

feat(create-plugin): parse --agent as a tri-state string flag

5c3cd14

feat(create-plugin): plumb the --agent flag through the update command

9f95321

The agent flag is stripped from codemodOptions so it never reaches codemod schemas, and --commit is forwarded as a tri-state so agentic runs can soft-force per-migration commits only when the user did not decide.

docs: document AI-assisted migrations and the --agent flag

bc20262

Covers the opt-in flow, agent flag forms, CI behavior (agent steps are skipped and surfaced as next steps in the job log), inside-agent deferral, and the prompt migration authoring contract.

github-project-automation Bot added this to Grafana Catalog Team Jul 13, 2026

github-project-automation Bot moved this to 📬 Triage in Grafana Catalog Team Jul 13, 2026

grafana-catalog-project-bot Bot moved this from 📬 Triage to 🧑‍💻 In development in Grafana Catalog Team Jul 13, 2026

jackw self-assigned this Jul 13, 2026

joshhunt reviewed Jul 14, 2026

View reviewed changes

refactor(create-plugin): detect agent CLIs on PATH only

4fc2d78

Drop the well-known-path fallbacks from agent detection per review feedback. If an agent binary isn't on PATH it isn't set up for CLI use, so probing hardcoded install locations added complexity without a legitimate use case.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add AI-agent assisted migrations to create-plugin update#2774

feat: add AI-agent assisted migrations to create-plugin update#2774
jackw wants to merge 16 commits into
mainfrom
jackw/agentic-migrations

jackw commented Jul 13, 2026 •

edited

Loading

Uh oh!

joshhunt left a comment

Uh oh!

joshhunt Jul 14, 2026

Uh oh!

jackw commented Jul 15, 2026

Uh oh!

jackw commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jackw commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshhunt left a comment

Choose a reason for hiding this comment

Uh oh!

joshhunt Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

jackw commented Jul 15, 2026

Uh oh!

jackw commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jackw commented Jul 13, 2026 •

edited

Loading