Repository navigation
feat(aws-startup-advisor): add target-only AI source-provider capability catalog - #450
Merged
Merged
Conversation
added 2 commits
October 13, 2026 19:32
…ity catalog Adds a static, target-only capability catalog for issue aws#389's "versioned capability catalog" staging step. Contract/schema/data/test-only: no runtime code, no behavior change, no phase-instruction edits, and no claim that any adapter or route is implemented. - ADD scripts/contracts/ai-capability-catalog.schema.json: Draft-07, additionalProperties:false throughout, reusing ai-workload-profile.schema.json's extensibleIdentity pattern verbatim (per-file repetition, the established convention since $ref cannot cross files in this validator). schema_version is int (per aws#389's own text), catalog_version/verified_at are a versioned snapshot pair. providers[]/gateways[] carry status (supported|restricted|unknown) and implemented_adapter/route as null-via-oneOf (the same convention as aws-design-ai.schema.json's target_bedrock_model/confidence_warning), never a stringy "unknown" sentinel. Schema description states structural validity does not imply an implemented adapter/route. - ADD skills/shared/ai/ai-capability-catalog.json seeded with anthropic (anthropic_api, supported), openai (openai_api + azure_openai, supported), google (vertex_ai, unknown — its only mapping guide is GCP-adapter-scoped, not a shared cross-skill asset the way anthropic's/openai's guides are), and the openrouter gateway (unknown — no mapping guide targets gateway routing today). implemented_adapter/route are null on every entry regardless of status. - Vendored the data file byte-identically into the two skills confirmed to vendor sdk-capability-map.json today — gcp-to-aws and azure-to-aws only (llm-to-bedrock has no vendored/ai/ tree; agent-advisor's vendored/ai/ tree carries no sdk-capability-map.json) — and added the README table row in each skill's references/vendored/README.md. - ADD tests/test_ai_capability_catalog_contracts.py: schema self-validity, the canonical file validates with zero findings, semantic invariants (no duplicate provider/gateway identity, every gateway upstream_providers[] entry exists in providers[], implemented_adapter/route are null on every entry), vendored-copy byte-identity for both copies, and negative mutations (openrouter as a provider identity, invalid status enum, malformed implemented_adapter, missing required field, a gateway upstream naming a provider absent from providers[] — documented as a semantic check, not a schema-level one, matching test_ai_workload_profile_contracts.py's schema-vs-semantic distinction). - EDIT skills/shared/ai/README.md and plugins/aws-startup-advisor/README.md to register the new canonical file, following aws#415/aws#416's exact pattern. Deliberately NOT edited: scripts/artifact-contracts.json. That file maps per-run produced artifacts (fixtures/<case>/ or .migration/<run>/ outputs) to contracts; the capability catalog is a static reference file read directly, exactly like sdk-capability-map.json, which is confirmed absent from artifact-contracts.json today. Adding an entry would be inert and would misrepresent the file's nature. Verification run from inside the worktree: 1. cd plugins/aws-startup-advisor && uv run --with pytest python -m pytest tests/ -q -> 404 passed (includes the 13 new tests in test_ai_capability_catalog_contracts.py) 2. mise run build (from worktree root) -> lint and validate tasks both exit 0 (lint:manifests, lint:md, lint:spec, lint:skill-parity all pass). The security task (gitleaks over full repo git history) reports 6 pre-existing findings in plugins/aws-core and skills/core-skills aws-observability content from commits authored 2026-09-22/24, unrelated to this change and present on upstream/main before this branch existed. 3. cd plugins/aws-startup-advisor && mise run ci -> all subtasks pass: lint:model-staleness (fresh), lint:vendored-parity (PASS - 51 vendored file(s) byte-identical to skills/shared/ and matching their README tables, including both new ai-capability-catalog.json copies), lint:artifacts (PASS - 80 artifact(s) checked, 0 errors), lint:skill-refs (PASS - 1295 references checked, 0 missing), lint:model-id (OK), lint:md (0 errors), test (404 passed), test:heroku-source-review (44 passed).
…coverage, not code The openrouter gateway entry's note previously read "No mapping guide of any kind targets gateway routing today," which could be misread as "AWS has no OpenRouter support at all." That's not true: discover-openrouter-api.md already detects OpenRouter usage and BYOK-vs-credits attribution per upstream model at runtime (shipped, unrelated to this catalog). status: unknown here reflects this catalog's narrow bar: whether a shared, cross-skill mapping GUIDE document exists (the ai-anthropic-to-bedrock.md / ai-openai-to-bedrock.md equivalent for OpenRouter upstream mapping), not whether any code handles the provider. Reworded the note to say this explicitly, so a future reader doesn't conflate missing documentation with missing functionality. Verified: data file still validates against ai-capability-catalog.schema.json, all 3 vendored copies re-synced to byte-identical (confirmed via md5), mise -C plugins/aws-startup-advisor run ci clean (lint:vendored-parity, lint:md, lint:skill-refs, test, test:heroku-source-review all pass), full suite 404 passed.
6 of 34 tasks
herosjourney
marked this pull request as ready for review
October 19, 2026 22:06
leon1418
reviewed
Oct 19, 2026
leon1418
left a comment
leon1418
left a comment
Contributor
There was a problem hiding this comment.
[🤖 AI review 🤖]
Reviewed all 9 changed files at 2a398ebe591f9c748c431d831803865563716f77 with OCR-assisted repository review and an independent general reviewer. No substantive findings. The catalog remains static and target-only; its identity patterns match the workload/design contracts, and both vendored copies match the canonical data.
Validation passed: 444 plugin tests, 323 advisor tests, 21 tool tests, 57 Heroku tests on Node 24, catalog mutation probes, artifact contracts, vendored parity, reference paths, Markdown lint, model ID/freshness checks, consent audit and git diff --check. Existing fixture/reference baselines remain unchanged. GitHub CI is green at this head; no live migration was exercised.
Approval recommended for the maintainer to submit.
leon1418
approved these changes
Oct 20, 2026
ijoel
pushed a commit
to ijoel/agent-toolkit-for-aws
that referenced
this pull request
Oct 20, 2026
… with shared mapping guides (aws#469) * feat(aws-startup-advisor): promote google and openrouter to supported in the AI capability catalog Follow-up to aws#450. That PR deliberately marked `google` and `openrouter` as `unknown` in ai-capability-catalog.json because neither had a shared, cross-skill Bedrock-mapping guide under skills/shared/ai/ — the bar the catalog uses for `supported`. Gemini's only guide lived under gcp-to-aws's adapter-local design-refs/, and no gateway-routing guide existed at all. This PR delivers those two shared guides and flips both catalog entries: - skills/shared/ai/ai-gemini-to-bedrock.md — source-cloud-neutral Gemini → Bedrock model-selection guide, modeled on the shared OpenAI/Anthropic guides (same header/vendoring contract). The Bedrock target depends on the Gemini model, not the source cloud, so the guide is canonical here and vendored into gcp-to-aws and azure-to-aws. - skills/shared/ai/ai-openrouter-to-bedrock.md — gateway-routing guide that maps each OpenRouter-fronted upstream model to Bedrock via the per-provider guides, preserving per-upstream associations and BYOK-vs-credits cost provenance. OpenRouter stays a gateway, never a provider or infrastructure source. - catalog: google -> supported, openrouter -> supported, catalog_version 1 -> 2, verified_at refreshed, notes rewritten to cite the new guides. Target-only, matching aws#450/aws#415/aws#416: no phase file reads these; no runtime behavior changes; implemented_adapter/route stay null everywhere (status reflects documentation coverage, not an implemented adapter). The gcp-to-aws adapter-local design-refs/ai-gemini-to-bedrock.md is retained; repointing its runtime routing onto the shared guide is deferred to the extraction stages (aws#389 3a-3e). Gates: vendored-parity (55 files byte-identical), skill-refs (1357 refs, 0 missing), ai-capability-catalog contract tests (13), full plugin suite (444), lint:md (803 files, 0 errors), artifacts/consent-disclosure/model-id all pass. * test(aws-startup-advisor): guard Gemini guide body-parity against adapter-local drift Addresses a review flag on this PR: gcp-to-aws now holds two Gemini → Bedrock guides — the new inert shared/vendored copy and the older adapter-local references/design-refs/ one that design-ai.md still loads at runtime — with no gate between them. The vendored-parity check only compares the vendored copy against the shared canonical, never against the adapter-local original, so a one-sided pricing edit could let them silently disagree until the aws#389 3a-3e extraction folds them together. - Make the shared guide's priced body byte-identical to the adapter-local guide's (the two had drifted by a single cosmetic table-separator character; all pricing/mapping rows already matched). Headers stay intentionally different (canonical banner + source-neutral Applies-to). - Add tests/test_gemini_guide_body_parity.py: pins both guides' bodies (from the first `## ` heading down) byte-for-byte, excluding the lead-in. Verified it fails on a one-sided pricing mutation. Delete this test when the extraction stage merges the two guides into one. No runtime routing changes; still target-only. Full `mise run ci` passes (446 tests, +2). * fix(aws-startup-advisor): align OpenRouter guide with the real Discover/Estimate contracts Addresses three P2 review findings on the ai-openrouter-to-bedrock.md guide (canonical + both vendored copies): 1. Provenance enum: the guide used invented `byok`/`credits` values and told the reader to derive a BYOK/credits volume split. The producer (discover-openrouter-api.md) actually emits `byok_passthrough`/`no_byok`/`mixed`/`uncertain`/`unknown` with a single `monthly_cost_usd` and no funding/token ratio, and llm-to-bedrock/SKILL.md forbids deriving that split. Rewrote the table to the real enum, added the `uncertain` handling, and state the mixed split is unavailable rather than fabricating one. 2. Cost baseline: the guide told the reader to compare the Bedrock target against the upstream provider's list price instead of the captured OpenRouter spend, which would swap the savings denominator. estimate-ai.md Part 1 prefers measured current_costs/usage-profile dollars and does not fall back to a direct-provider list rate for the OpenRouter portion (no source-side list row in pricing-cache.md). Made the captured spend the baseline and demoted list pricing to a separately labeled reference. 3. Endpoint/migration_path: the Cutover step sent every GPT target to Mantle, but ai-openai-to-bedrock.md can select `runtime_openai_cris` (CRIS) for GPT-5.6/GPT-6. Now carries each target's selected migration_path, endpoint/ profile ID, and matching IAM (bedrock:InvokeModel for Converse/runtime, bedrock-mantle:* for the Mantle GPT path) through cutover. Docs-only, target-only; vendored parity, skill-refs, lint:md green; 446 tests pass. --------- Co-authored-by: Logan Kleier <lkleier@amazon.com> Co-authored-by: Gen Li <i.ligen@gmail.com>
herosjourney
added a commit
to herosjourney/agent-toolkit-for-aws
that referenced
this pull request
Oct 20, 2026
…odel-lifecycle mapping-guide rule (aws#470) * feat(aws-startup-advisor): promote google and openrouter to supported in the AI capability catalog Follow-up to aws#450. That PR deliberately marked `google` and `openrouter` as `unknown` in ai-capability-catalog.json because neither had a shared, cross-skill Bedrock-mapping guide under skills/shared/ai/ — the bar the catalog uses for `supported`. Gemini's only guide lived under gcp-to-aws's adapter-local design-refs/, and no gateway-routing guide existed at all. This PR delivers those two shared guides and flips both catalog entries: - skills/shared/ai/ai-gemini-to-bedrock.md — source-cloud-neutral Gemini → Bedrock model-selection guide, modeled on the shared OpenAI/Anthropic guides (same header/vendoring contract). The Bedrock target depends on the Gemini model, not the source cloud, so the guide is canonical here and vendored into gcp-to-aws and azure-to-aws. - skills/shared/ai/ai-openrouter-to-bedrock.md — gateway-routing guide that maps each OpenRouter-fronted upstream model to Bedrock via the per-provider guides, preserving per-upstream associations and BYOK-vs-credits cost provenance. OpenRouter stays a gateway, never a provider or infrastructure source. - catalog: google -> supported, openrouter -> supported, catalog_version 1 -> 2, verified_at refreshed, notes rewritten to cite the new guides. Target-only, matching aws#450/aws#415/aws#416: no phase file reads these; no runtime behavior changes; implemented_adapter/route stay null everywhere (status reflects documentation coverage, not an implemented adapter). The gcp-to-aws adapter-local design-refs/ai-gemini-to-bedrock.md is retained; repointing its runtime routing onto the shared guide is deferred to the extraction stages (aws#389 3a-3e). Gates: vendored-parity (55 files byte-identical), skill-refs (1357 refs, 0 missing), ai-capability-catalog contract tests (13), full plugin suite (444), lint:md (803 files, 0 errors), artifacts/consent-disclosure/model-id all pass. * test(aws-startup-advisor): guard Gemini guide body-parity against adapter-local drift Addresses a review flag on this PR: gcp-to-aws now holds two Gemini → Bedrock guides — the new inert shared/vendored copy and the older adapter-local references/design-refs/ one that design-ai.md still loads at runtime — with no gate between them. The vendored-parity check only compares the vendored copy against the shared canonical, never against the adapter-local original, so a one-sided pricing edit could let them silently disagree until the aws#389 3a-3e extraction folds them together. - Make the shared guide's priced body byte-identical to the adapter-local guide's (the two had drifted by a single cosmetic table-separator character; all pricing/mapping rows already matched). Headers stay intentionally different (canonical banner + source-neutral Applies-to). - Add tests/test_gemini_guide_body_parity.py: pins both guides' bodies (from the first `## ` heading down) byte-for-byte, excluding the lead-in. Verified it fails on a one-sided pricing mutation. Delete this test when the extraction stage merges the two guides into one. No runtime routing changes; still target-only. Full `mise run ci` passes (446 tests, +2). * fix(aws-startup-advisor): align OpenRouter guide with the real Discover/Estimate contracts Addresses three P2 review findings on the ai-openrouter-to-bedrock.md guide (canonical + both vendored copies): 1. Provenance enum: the guide used invented `byok`/`credits` values and told the reader to derive a BYOK/credits volume split. The producer (discover-openrouter-api.md) actually emits `byok_passthrough`/`no_byok`/`mixed`/`uncertain`/`unknown` with a single `monthly_cost_usd` and no funding/token ratio, and llm-to-bedrock/SKILL.md forbids deriving that split. Rewrote the table to the real enum, added the `uncertain` handling, and state the mixed split is unavailable rather than fabricating one. 2. Cost baseline: the guide told the reader to compare the Bedrock target against the upstream provider's list price instead of the captured OpenRouter spend, which would swap the savings denominator. estimate-ai.md Part 1 prefers measured current_costs/usage-profile dollars and does not fall back to a direct-provider list rate for the OpenRouter portion (no source-side list row in pricing-cache.md). Made the captured spend the baseline and demoted list pricing to a separately labeled reference. 3. Endpoint/migration_path: the Cutover step sent every GPT target to Mantle, but ai-openai-to-bedrock.md can select `runtime_openai_cris` (CRIS) for GPT-5.6/GPT-6. Now carries each target's selected migration_path, endpoint/ profile ID, and matching IAM (bedrock:InvokeModel for Converse/runtime, bedrock-mantle:* for the Mantle GPT path) through cutover. Docs-only, target-only; vendored parity, skill-refs, lint:md green; 446 tests pass. * docs(aws-startup-advisor): list the OpenRouter gateway guide in the model-lifecycle mapping-guide rule The "Mapping Guides" section of ai-model-lifecycle.md enumerates the mapping guides its lifecycle rules apply to (Best-Bedrock-Match rows must be Active, no exclusion-zone or restricted models as defaults, Legacy only in notes). It listed the OpenAI, Anthropic, and Gemini guides but not the new shared OpenRouter gateway guide (ai-openrouter-to-bedrock.md), whose per-upstream recommendation rows are subject to the same rules. Add it to the heading, flagged as the gateway guide. Docs-only; vendored copies in gcp-to-aws and azure-to-aws synced byte-identical. Builds on the shared OpenRouter guide added in the catalog PR. Full `mise run ci` passes (446 tests; vendored-parity, skill-refs, lint:md all green). --------- Co-authored-by: Logan Kleier <lkleier@amazon.com> Co-authored-by: Gen Li <i.ligen@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem / impact
#389's extraction plan deliberately uses a free-form
extensibleIdentitypattern (not a closed provider enum) in #415, #416, and #449, specifically so that a later, versioned capability catalog could define what source providers / services / gateways are actually supported, without having to touch the extraction schema itself. This PR delivers that catalog: it is #389's own staging step 9, and it is target-only — it only declares which AI providers/gateways the system currently has a mapping story for, it does not implement anything.Base / diff-size note
This branch is cut directly off current
upstream/main(9f10c285, "canonicalize aws-design-ai.json contract (#416)"), which already includes #410, #415, and #416. Unlike those three PRs, which were stacked on temporary bases and explicitly flagged that their diffs would shrink once earlier PRs merged, this PR has no such caveat — the diff you see here (9 files, ~385 lines, almost entirely new catalog + schema + fixture + test files) is already close to its final size.Cross-references
extensibleIdentitypatternaws-design-ai.jsoncontract on top of test(aws-startup-advisor): define neutral AI workload profile #415What the catalog says, and why
The catalog data file (
skills/shared/ai/ai-capability-catalog.json, vendored intogcp-to-awsandazure-to-awsfor skill-local lookup) marks providers"supported"only when there's an existing, shared, cross-skill mapping guide for them underskills/shared/ai/:"supported"— has a shared Anthropic-to-Bedrock mapping guide underskills/shared/ai/."supported"— has a shared OpenAI-to-Bedrock mapping guide underskills/shared/ai/."unknown"— Gemini's only mapping guide (ai-gemini-to-bedrock.md) lives underskills/gcp-to-aws/references/design-refs/, scoped to that one skill's adapter. It is not a shared, cross-skill guide the way anthropic's and openai's are, so it doesn't clear the bar for"supported"yet."unknown"— no mapping guide of any kind targets gateway routing today.Every entry's
implemented_adapter(providers) /route(gateways) field is intentionallynullon every single entry in this first version of the catalog. Status ("supported"/"unknown") only reflects whether a mapping guide exists — it says nothing about whether an adapter is actually wired up or implemented anywhere.Not proposing
Matching the pattern from #410 / #415 / #416:
implemented_adapter/routearenulleverywhere, by design, in this version.Files changed
plugins/aws-startup-advisor/skills/shared/ai/ai-capability-catalog.schema.json(new)plugins/aws-startup-advisor/skills/shared/ai/ai-capability-catalog.json(new)plugins/aws-startup-advisor/skills/shared/ai/README.mdplugins/aws-startup-advisor/skills/gcp-to-aws/references/vendored/ai/ai-capability-catalog.json(new, vendored copy)plugins/aws-startup-advisor/skills/gcp-to-aws/references/vendored/README.mdplugins/aws-startup-advisor/skills/azure-to-aws/references/vendored/ai/ai-capability-catalog.json(new, vendored copy)plugins/aws-startup-advisor/skills/azure-to-aws/references/vendored/README.mdplugins/aws-startup-advisor/tests/test_ai_capability_catalog_contracts.py(new)plugins/aws-startup-advisor/README.md