test(aws-startup-advisor): pin Estimate and Generate AI contracts

plugins/aws-startup-advisor/README.md

@@ -39,7 +39,7 @@ Alongside the skills, the plugin ships supporting material the skills load on de
 - [`skills/shared/`](skills/shared/) — Canonical reference material shared across skills rather than an invocable skill of its own: the phase-workflow interpreter contract, estimation schemas and complexity tiers, a snapshot of AWS infrastructure rates, phase-status schema, what-if workshop invariants, shared Clarify question fragments, and the AI-migration guardrails and Bedrock model references. Each consuming skill carries a byte-identical copy under its own `references/vendored/` so the skill folder stays self-contained.
 - [`agents/`](agents/) — Seven Claude Code subagent definitions: two generic, phase-agnostic migration phase workers (read/write and read/write/shell tiers) plus five specialists for the Bedrock rewrite flow (code analyzer, code rewriter, log ingestor, prompt evaluator, report generator).
 - [`tools/`](tools/) — Contains `model-staleness.py`, which checks the Bedrock model-lifecycle registry's freshness contract (`**Last updated**` / `**Staleness window**`) and that every vendored copy under `skills/*/references/vendored/ai/` is byte-identical to `skills/shared/ai/ai-model-lifecycle.md` (`--strict` also fails on a stale snapshot).
-- [`scripts/`](scripts/) — Python helpers that validate generated artifacts, emit run summaries, and lint Bedrock model references: a migration report validator, a startup-program artifact check, a plan summary emitter (`emit-plan-json.py`), `validate-artifacts.py`, which checks every phase artifact in a run directory (or the fixture goldens, in CI) against the contract its `schema-*.md` / JSON Schema publishes — unknown keys, missing required keys, type and enum drift — and `model-id-lint.py`, which fails when a file outside the model catalog names a Bedrock model the lifecycle registry marks excluded or removed, or a fabricated dated Sonnet 4.6 / Opus 4.8 ID. `scripts/artifact-contracts.json` maps artifacts to contracts; `scripts/artifact-contracts-baseline.json` is the explained burn-down list of known contract gaps. The Heroku flow's report validator ships inside `skills/heroku-to-aws/scripts/`. `scripts/telemetry/` collects optional usage data: nothing is sent until you accept, no skill depends on it, and `/usage-data` shows the setting or opts out. `scripts/contracts/ai-workload-profile.schema.json` additionally defines a target-only, provider-neutral observation contract for a future `ai-workload-profile.json` normalization output; it has no producer yet and asserts no runtime support. `scripts/contracts/aws-design-ai.schema.json`, canonicalized from Azure's prior single-scalar `metadata.ai_source` contract into `skills/shared/ai/schema-design-aws-ai.md`, likewise defines a target-only `aws-design-ai.json` design contract whose `metadata.ai_sources[]`/`metadata.gateways[]` mirror the workload-profile contract exactly; `design-ai.md` in both skills still writes against the prior per-skill shape, so this is a target, not a claim about current output. `scripts/contracts/ai-capability-catalog.schema.json` defines the target-only contract for `skills/shared/ai/ai-capability-catalog.json`, a static source-provider/gateway capability catalog; its `implemented_adapter`/`route` fields are null everywhere and it asserts no runtime support.
+- [`scripts/`](scripts/) — Python helpers that validate generated artifacts, emit run summaries, and lint Bedrock model references: a migration report validator, a startup-program artifact check, a plan summary emitter (`emit-plan-json.py`), `validate-artifacts.py`, which checks every phase artifact in a run directory (or the fixture goldens, in CI) against the contract its `schema-*.md` / JSON Schema publishes — unknown keys, missing required keys, type and enum drift — and `model-id-lint.py`, which fails when a file outside the model catalog names a Bedrock model the lifecycle registry marks excluded or removed, or a fabricated dated Sonnet 4.6 / Opus 4.8 ID. `scripts/artifact-contracts.json` maps artifacts to contracts; `scripts/artifact-contracts-baseline.json` is the explained burn-down list of known contract gaps. The Heroku flow's report validator ships inside `skills/heroku-to-aws/scripts/`. `scripts/telemetry/` collects optional usage data: nothing is sent until you accept, no skill depends on it, and `/usage-data` shows the setting or opts out. `scripts/contracts/ai-workload-profile.schema.json` additionally defines a target-only, provider-neutral observation contract for a future `ai-workload-profile.json` normalization output; it has no producer yet and asserts no runtime support. `scripts/contracts/aws-design-ai.schema.json`, canonicalized from Azure's prior single-scalar `metadata.ai_source` contract into `skills/shared/ai/schema-design-aws-ai.md`, likewise defines a target-only `aws-design-ai.json` design contract whose `metadata.ai_sources[]`/`metadata.gateways[]` mirror the workload-profile contract exactly; `design-ai.md` in both skills still writes against the prior per-skill shape, so this is a target, not a claim about current output. `scripts/contracts/ai-capability-catalog.schema.json` defines the target-only contract for `skills/shared/ai/ai-capability-catalog.json`, a static source-provider/gateway capability catalog; its `implemented_adapter`/`route` fields are null everywhere and it asserts no runtime support. `scripts/contracts/estimation-ai.schema.json` and `scripts/contracts/generation-ai.schema.json`, canonicalized into `skills/shared/ai/schema-estimate-ai.md`/`schema-generate-ai.md`, likewise define target-only `estimation-ai.json`/`generation-ai.json` contracts whose `cost_by_source[]`/`artifacts_by_source[]` carry a per-source/per-model traceability chain back through `aws-design-ai.json`'s `metadata.ai_sources[]`/`gateways[]`; `estimate-ai.md` and `generate-ai.md`/`generate-artifacts-ai.md` in both skills still write against their prior per-skill shapes, so these are targets, not claims about current output.
 - [`fixtures/`](fixtures/) — Reference and regression fixtures for the migration report and estimation artifacts, used to keep generated output within the documented contract. `fixtures/azure-iac-terraform/` additionally carries deterministic asserters that pin the Azure Discover, Clarify, Design, and Estimate contracts against a committed synthetic estate.
 
 ## MCP Servers

plugins/aws-startup-advisor/fixtures/README.md

@@ -5,6 +5,8 @@ Also see:
 - `ai-migration-matrix/` — 14 target-only representative contract snapshots for infrastructure-only, AI-only, and combined migration modes. Under Option A, each run has zero or one infrastructure source; polycloud or multiple simultaneous infrastructure sources are deferred. Any one supported infrastructure source composes independently with many AI sources and optional gateways. `ai_sources[]` retains provider/service provenance, families, and exact normalized `models[]`; `model_evidence` distinguishes observed models from an explicitly empty not-observed set without fake IDs. One OpenRouter gateway can retain multiple structured provider/service/family/exact-model associations, while `upstream_evidence: unresolved` makes unknown upstreams explicit. Synthetic positive tests cover GCP, Azure, and Heroku composition; Heroku with direct Google/Gemini, OpenAI/GPT, and Anthropic/Claude; and one multi-provider, multi-family, multi-model OpenRouter scenario without Cartesian fixture directories. `expected-matrix.json` records represented contract coverage, not a versioned runtime capability catalog: free-form schema validity does not imply an implemented adapter or route. This is contract/test/documentation coverage only and changes no runtime behavior. Combined cases use one `.phase-status.json` and a separate fail-closed `integration-validation.json` cross-track verdict.
 - `ai-workload-profiles/` — 4 representative target-only `ai-workload-profile.json` observation fixtures (not a Cartesian matrix): AI-only multi-direct-provider (Anthropic + Google/Vertex, no gateway); OpenAI direct plus Azure OpenAI with distinct `source_service` provenance; a multi-upstream OpenRouter gateway carrying Anthropic (two exact Claude models), Google, and OpenAI simultaneously; and an unresolved/not-yet-identified OpenRouter gateway with an explicit empty upstream set. Each source retains the raw/observed model string in `model_observations[]` alongside the exact normalized `models[]` entry, with a deterministic `provenance`/`confidence` evidence vocabulary. `test_ai_workload_profile_contracts.py` proves every fixture's normalized `ai_sources[]`/`gateways[]` projects exactly into `ai-migration-scenario.schema.json`'s semantics (`test_ai_migration_matrix_contracts.py`), and exercises negative mutations (duplicate identity, a model dropped under a resolved upstream, a fabricated `unknown` ID, OpenRouter named as a direct provider, an empty upstream on an observed gateway, an invented upstream on an unresolved gateway) rather than separate fixture directories per negative. Schema validity here does not imply an implemented detector, adapter, route, model mapping, or runtime support — no producer of this artifact exists yet. `ai-workload-profile.schema.json`'s `aiSource` observed branch now also requires `model_observations` non-empty (`minItems: 1`) whenever `model_evidence == "observed"`, matching the identical rule already enforced for `aws-design-ai.schema.json`'s `metadata.ai_sources[]`.
 - `aws-design-ai/` — 2 representative target-only `aws-design-ai.json` design fixtures, nested as `<case>/aws-design-ai.json` so `lint:artifacts` discovers them (unlike a flat layout, which #415's own review flagged as invisible to the lint): `multi-provider-direct` (one Anthropic-sourced and one OpenAI-sourced `design_blocks[]` entry in a single design, no gateway) and `openrouter-gateway-multi-upstream` (one OpenRouter gateway fronting Anthropic — two exact Claude models — and OpenAI simultaneously, with a 3-tier routing strategy). `metadata.ai_sources[]`/`metadata.gateways[]` replace the retired single-scalar `metadata.ai_source` field this contract previously carried; `test_aws_design_ai_contracts.py` proves both fixtures project exactly into `ai-workload-profile.schema.json`'s semantics (`test_ai_workload_profile_contracts.py`, #415's own oracle), and exercises negative mutations — the retired single-scalar `ai_source` shape (with and without a `both`-style multi-provider implication), duplicate source identity, OpenRouter named as a direct provider, a dropped model under a resolved gateway upstream, a `design_blocks[]` row without exactly one target, and a fabricated `unknown` model ID. Schema validity here does not imply an implemented detector, adapter, route, model mapping, or runtime support, and does not claim either skill's current `design-ai.md` emits this shape — no producer has been repointed to this canonical contract in this change.
+- `estimation-ai/` — 2 representative target-only `estimation-ai.json` cost-estimate fixtures, nested as `<case>/estimation-ai.json` so `lint:artifacts` discovers them: `multi-provider-direct` and `openrouter-gateway-multi-upstream`, reusing #416's `aws-design-ai/` scenario names and the same providers/models. `cost_by_source[]` carries forward `aws-design-ai.json`'s `metadata.ai_sources[]`/`gateways[]` identity pattern so a per-source/per-model estimated cost traces back to the exact Design entry; `model_observations[]` is intentionally not carried forward (the identity is already resolved by Design). `test_estimation_ai_contracts.py` proves both fixtures project into (are a subset of) the matching `aws-design-ai/<same-case>/aws-design-ai.json`, and exercises negative mutations — duplicate/overlapping source identity, OpenRouter named as a direct provider, a fabricated untraceable model, and a model dropped under a resolved gateway upstream. Schema validity here does not imply an implemented pricing lookup, model mapping, or runtime support, and does not claim either skill's current `estimate-ai.md` emits this shape — no producer has been repointed to this canonical contract in this change.
+- `generation-ai/` — 2 representative target-only `generation-ai.json` migration-plan fixtures, nested as `<case>/generation-ai.json`, mirroring `estimation-ai/`'s two case names. `artifacts_by_source[]` carries the traceability chain one phase forward: a per-source/per-model generated-artifact reference traces back to the exact `estimation-ai.json → cost_by_source[]` entry it was produced from. `test_generation_ai_contracts.py` proves both fixtures project into the matching `estimation-ai/<same-case>/estimation-ai.json`, and exercises the same negative-mutation categories one phase forward. Schema validity here does not imply an implemented adapter, rollout mechanism, or runtime support, and does not claim either skill's current `generate-ai.md`/`generate-artifacts-ai.md` emits this shape.
 - `azure-rdfa-report*/` — producer-shaped RDfA source fixtures: inventory rows, metrics
   batches wrapped in the upstream `{ "Metrics": [...] }` envelope, corrupt/multi-ZIP
   cases, fixed/autoscaled AKS pools, canonical VM OS-disk fields, and an ML workspace.

plugins/aws-startup-advisor/fixtures/estimation-ai/multi-provider-direct/estimation-ai.json

@@ -0,0 +1,126 @@
+{
+  "phase": "estimate",
+  "timestamp": "2026-09-08T03:10:00Z",
+  "pricing_source": "cached",
+  "accuracy_confidence": "±15-25%",
+  "current_costs": {
+    "source": "mixed",
+    "monthly_ai_spend": 1200.0,
+    "services": ["anthropic_api", "openai_api"]
+  },
+  "token_volume": {
+    "source": "tier_table",
+    "monthly_input_tokens": 60000000,
+    "monthly_output_tokens": 40000000,
+    "ratio": "60/40"
+  },
+  "model_comparison": [
+    {
+      "model": "anthropic.claude-sonnet-4-5-v1:0",
+      "monthly_cost": 1320.0,
+      "vs_current": "+10%",
+      "quality": "high",
+      "capabilities_match": true,
+      "missing_capabilities": []
+    },
+    {
+      "model": "openai.gpt-5.6-terra",
+      "monthly_cost": 660.0,
+      "vs_current": "-12%",
+      "quality": "high",
+      "capabilities_match": true,
+      "missing_capabilities": []
+    }
+  ],
+  "recommended_model": {
+    "model": "anthropic.claude-sonnet-4-5-v1:0",
+    "monthly_cost": 1320.0,
+    "breakdown": {"input": 900.0, "output": 420.0, "embeddings": 0.0},
+    "rationale": "Same-model migration for the Anthropic-sourced workload; current recommended model and provider priority favor Claude."
+  },
+  "backup_model": {
+    "model": "anthropic.claude-haiku-4-5-v1:0",
+    "monthly_cost": 480.0,
+    "rationale": "Lower-cost fallback within the same model family."
+  },
+  "cost_comparison": {
+    "current_source_monthly": 1200.0,
+    "projected_bedrock_monthly": 1980.0,
+    "monthly_difference": 780.0,
+    "annual_difference": 9360.0,
+    "percent_change": 65.0
+  },
+  "migration_cost_considerations": {
+    "categories": [],
+    "complexity_factors": ["integration.pattern = \"direct_sdk\" -> moderate SDK and API pattern changes"],
+    "note": "Human and professional-services one-time migration costs are intentionally excluded from this advisor."
+  },
+  "roi_analysis": {
+    "monthly_cost_delta": 780.0,
+    "annual_cost_delta": 9360.0,
+    "justification": "Combined Anthropic and OpenAI Bedrock costs are higher than current direct-API spend; non-cost benefits justify the move.",
+    "non_cost_benefits": ["IAM/VPC/PrivateLink/KMS/CloudTrail governance", "usage counting toward existing AWS commitments"]
+  },
+  "optimization_opportunities": [
+    {
+      "opportunity": "Prompt caching",
+      "type": "prompt_caching",
+      "target_services": ["Bedrock"],
+      "potential_savings_monthly": null,
+      "potential_savings_percent": "up to 90% on cached input tokens",
+      "commitment": "none",
+      "implementation_effort": "low",
+      "available": "day_1",
+      "description": "Repeated system prompts qualify for cache-read pricing on both Claude and GPT-5.6."
+    },
+    {
+      "opportunity": "Model downsizing / tiering",
+      "type": "model_tiering",
+      "target_services": ["Bedrock"],
+      "potential_savings_monthly": null,
+      "potential_savings_percent": "60-87%",
+      "commitment": "none",
+      "implementation_effort": "low",
+      "available": "day_1",
+      "description": "High-volume, premium model selected; a lower tier may suffice for simple queries."
+    }
+  ],
+  "optimized_projection": {
+    "monthly_with_optimizations": 1400.0,
+    "vs_current": "+17%",
+    "note": "Best-case projection assuming prompt caching and selective model tiering."
+  },
+  "recommendation": {
+    "path": "migrate_optimized",
+    "path_label": "Migrate to Bedrock",
+    "migrate_if": "Governance, residency, and commitment-counting benefits outweigh the modest cost premium.",
+    "stay_if": "Cost is the only priority and no non-cost driver applies.",
+    "confidence": "high",
+    "rationale": "Both Bedrock targets carry a modest premium over direct-API pricing, offset by governance and ecosystem benefits; both source models are strong_migrate per the design assessment."
+  },
+  "cost_by_source": [
+    {
+      "provider": "anthropic",
+      "source_service": "anthropic_api",
+      "models": [
+        {
+          "model": "claude-3-5-sonnet-20240620",
+          "monthly_cost": 1320.0,
+          "vs_source_percent": 10.0
+        }
+      ]
+    },
+    {
+      "provider": "openai",
+      "source_service": "openai_api",
+      "models": [
+        {
+          "model": "gpt-4o",
+          "monthly_cost": 660.0,
+          "vs_source_percent": -12.0
+        }
+      ]
+    }
+  ],
+  "services_not_estimated": []
+}