Repository navigation
test(aws-startup-advisor): pin Estimate and Generate AI contracts - #449
Merged
leon1418 merged 137 commits intoOct 21, 2026
Merged
Conversation
…artifacts against their published schemas
The migration skills pass ~17 JSON artifacts between phases, each with a
contract published as a schema-*.md "shape" block (JSON/JSONC example) or,
for two of them, a real JSON Schema under skills/shared/. Nothing checked
that a producer writes what the next phase reads: a key could be renamed,
emitted without being documented, or an enum widened, and every DSL
_assert still passed because the model that writes the artifact evaluates
the assertion. This is the precondition the backlog names for any engine
consolidation or AI-engine extraction — moves across adapters have no
drift detection without it.
scripts/validate-artifacts.py (stdlib, Python 3.9+)
- Parses each contract document's example blocks (JSONC comments, trailing
commas, `...` ellipses, `a|b` enum strings, `// a | b | c` enum comments,
`// REQUIRED` / `(REQUIRED)` markers, `"<placeholder>"` keys = open
object, `"<key>": {...}` = template for every key) into a key-path
contract; sub-section blocks (`## services[]`, `### metadata (REQUIRED)`)
refine paths; manifest can pin a block by `@L<line>` anchor, merge other
documents' additions (graviton, workshop), or declare a row-annotation
vocabulary (azure preferences' source/reason/context/...).
- Validates real JSON Schemas with a draft-07 subset (type, required,
properties, additionalProperties, enum, const, items, pattern, min/max,
minItems, allOf/anyOf/oneOf, $ref).
- Findings: UNKNOWN_KEY, MISSING_REQUIRED, TYPE_MISMATCH, ENUM_VIOLATION,
SCHEMA_VIOLATION, SCHEMA_PARSE (the contract itself is broken),
NO_CONTRACT (info). `_`-prefixed keys are annotations.
- Modes: --self-check (contracts parse), --fixtures (CI; goldens against
the baseline), --run-dir <.migration/run> --skill <name> (users).
- scripts/artifact-contracts.json maps artifact → contract per skill.
- scripts/artifact-contracts-baseline.json: 21 explained entries covering
44 known gaps between what phases emit and what their contract documents
say, each naming the document that must change. Stale entries are
reported so the list only shrinks.
Contract documents fixed where the document contradicted itself (its own
prose named a key its example block omitted):
- gcp schema-preferences.md: the shape block was not JSON (`{ "value",
"chosen_by" }` set notation); rewritten as a JSONC row template.
- gcp schema-estimate-infra.md: root block lacked complexity_tier, which
estimation-infra.schema.json marks required.
- gcp schema-discover-iac.md § Live discovery extensions: block now shows
the metadata/resources[] additions the prose above it documents.
- heroku schema-discover-heroku.md: resources[] example carries
source/unmanaged_by_terraform/not_found_live (table at lines 93-95);
addon config example carries the per-add-on and live-enrichment fields
the bullets below it list.
- azure schema-design-aws.md: services[] block carries
routing_provenance/sizing_provenance/sizing_source/is_compute_unit/
hosted_app_azure_ids (the same document marks them REQUIRED); deferred[]
carries routing_provenance.
- azure schema-discover-azure.md: clusters[] carries pattern_status
(discover-assemble.md § Status).
- azure schema-preferences.md: top-level clarify_status (REQUIRED per
clarify-assemble.md § clarify_status).
Fixture: gcp-decision-gate's estimation-infra.json stub gains the four
keys estimation-infra.schema.json requires; check_expected_decide.py
still passes.
Wiring: mise `lint:artifacts` task, added to `validate` (CI build runs it).
README scripts/ entry updated.
Tests: tests/test_validate_artifacts.py — 16 tests: JSONC stripping,
template inference (enum/required/placeholder/any-key), each finding
code, JSON Schema subset, baseline matching + stale detection, CLI
self-check and fixtures pass on the real repo, and two negatives: an
injected off-contract key and a schema enum violation both exit 1.
…(and reject unknown schema keywords); stale baseline fails Review on aws#385, both should-fix findings valid: 1. [Propagation] phase-status.schema.json has `minProperties: 1` on `phases`; the draft-07 subset did not implement it, so `phases: {}` passed. Fixed the instance AND the class: the subset now enforces minProperties, maxProperties, exclusiveMinimum/Maximum, maxItems, uniqueItems, minLength/maxLength and `not`; and `unsupported_keywords()` walks every loaded JSON Schema at contract-build time and emits SCHEMA_PARSE for any validation keyword that is neither enforced nor a known annotation. A constraint can no longer be skipped silently — adding a keyword to a schema without teaching the validator fails the gate. 2. [Completeness] a baseline entry that matched nothing was printed as a warning and exited 0, so the burn-down list only shrank if someone read the log. STALE_BASELINE is now an error code: `--fixtures` (what `mise run lint:artifacts` runs) exits 1 on any stale entry. Tests (19): empty `phases` fails with minProperties named; the two real schemas use only enforced/annotation keywords and an invented `contentEncoding` / `dependencies` schema is rejected; a never-matching baseline entry exits 1 with STALE_BASELINE.
…ndored/ must match skills/shared/ Each migration skill is installable on its own (`npx skills add … --skill gcp-to-aws`), so shared material (DSL interpreter, pricing caches, estimate schemas, …) is vendored into skills/<skill>/references/vendored/ from the canonical skills/shared/. The vendored READMEs said: "This repository has no automated sync task for these copies — keeping them byte-identical is part of the change that touches the canonical file." All 44 copies are identical today; nothing stopped the next edit to skills/shared/ from leaving a skill running stale logic. - tools/sync-vendored.py (stdlib): same shape as sync-plugin-skills.py. Default mode copies skills/shared/<path> over every vendored copy that differs; --check exits 1 listing any copy that differs or has no canonical source (a vendored file nobody owns). README.md in a vendored dir is explanatory and skipped. The vendored tree is a subset of shared/, so files in shared/ that no skill vendors are fine. - mise `lint:vendored-parity`, added to `validate` (the CI build runs it). - The four vendored READMEs replace the manual cp + md5sum instructions with the tool and the CI rule. - tools/test_sync_vendored.py: 4 tests on a throwaway tree — drift and orphan detection (README excluded), sync repairs drift and leaves orphans as errors, CLI exit codes — plus the real repository is in sync. No skill content changes. Stacked on aws#385 only because both add a task to the same `validate` depends line.
…t; deleted copies fail; READMEs say what the gate sees Review on aws#386 (Contradictions / Completeness, should-fix): the three migration READMEs claimed "a change to skills/shared/ that forgets the copies cannot merge", but the check only walked files that still existed — a deleted vendored copy, or a new shared file never vendored, passed. Rather than only weakening the sentence, the gate now uses the record of intent each vendored README already carries: its `| Vendored path | Canonical source |` table. - check: every row must exist on disk (deleted copy → FAIL), every file on disk must have a row (README stays honest), and each row's canonical path must be skills/shared/<same path>. A README with no table is an error. - sync: restores a listed-but-missing copy from canonical before comparing. - What the gate still cannot see — a NEW skills/shared/ file no skill vendors — is now stated in the tool docstring and in all four READMEs, with the instruction "add the copy AND a table row". The three READMEs were already complete tables (gcp 14/14, heroku 8/8, azure 21/21, agent-advisor 1/1); the real repo passes unchanged. Tests (7): deleted copy caught via the table and restored by sync; unlisted file and wrong canonical path caught; a new shared file nobody vendors is NOT an error (pins the documented limit); CLI exit codes; real repo in sync. Reviewer probes re-run on a throwaway copy of the plugin: deleting agent-advisor's INTERPRETER.md → FAIL naming it; default mode restores it; adding skills/shared/brand-new.md → no error, as documented.
…l mentions must exist
The skills are prose state machines: a phase file says "Load
`references/phases/design/design-ai.md`" or "run `python3
\"<SKILL_BASE>/scripts/x.py\"`" and the agent follows the string. Nothing
checked the target exists. A renamed or never-written file is a dead end
the user hits mid-migration, and any file move during consolidation would
break references silently. This is the third gate in the family
(validate-artifacts.py: artifact contracts; sync-vendored.py: vendored
parity) and the precondition for moving phase files between skills.
tools/check-skill-refs.py (stdlib)
- Scans skills/**/*.md and agents/*.md for path tokens into the skill
tree (references/, knowledge/, scripts/, skills/, agents/, data/, and the
agent-advisor short forms phases/, shared/, design-refs/) with optional
$PLUGIN_ROOT / ${CLAUDE_PLUGIN_ROOT} / $SKILL_BASE / <SKILL_BASE> /
$GCP_BASE / ./ / ../ prefixes.
- Resolution: explicit prefix → skill-relative → file-relative →
skill/references/<short form> → plugin/repo-relative. Library-origin
files (skills/shared/*, */references/vendored/*, */references/shared/*)
speak from the consuming skill's point of view and resolve against
skills/shared and every skill; agent-advisor and llm-to-bedrock may name
gcp-to-aws's tree (documented inline execution); agents/ resolve against
llm-to-bedrock.
- Skips run-artifact paths ($MIGRATION_DIR/…, .migration/…, <PLAN_DIR>/…),
templated tokens (<slug>, NNN, *), URLs, and ./ or ../ tokens with no
skill-tree root in them (generated outputs like ./deploy.sh).
- tools/skill-refs-baseline.json: `ignore` rules (permanent policy —
scripts Generate writes into $MIGRATION_DIR/scripts/, an example path in
the user's app) and `entries` (known dead references with the fix;
STALE when they stop matching, failing the gate so the list only shrinks).
1118 references checked on the real repo; 7 dead ones baselined, each a
real user-facing defect: azure discover-iac.md points Bicep/ARM input at
extract-bicep.md / extract-arm.md that do not exist (so Bicep halts);
design-refs/patterns.md referenced from two azure files before it exists;
contextual-offers SKILL.md names references/offers.md unqualified when the
file lives in knowledge-base-for-startups; gcp design-ai.md:424 points at
shared/ai-model-lifecycle.md, which in gcp is vendored at
references/vendored/ai/.
mise `lint:skill-refs`, added to `validate`. Tests (8): each resolution
rule, each skip rule, library-origin resolution, gcp-dependent and agents
fallbacks, ignore vs entries vs stale, CLI exit codes, real repo PASS.
…inst llm-to-bedrock Review on aws#387 (Completeness / Contradictions, should-fix): the agents/ branch of the resolver searched every skill and kept the first hit, so agents/llm2bedrock-prompt-evaluator.md:233 `scripts/pyproject.toml` resolved to skills/agent-advisor/scripts/pyproject.toml (sorts first) and a tree with the file only under agent-advisor still passed — deleting the llm-to-bedrock copy would not have failed the gate. The comment above the loop said llm-to-bedrock; the loop did not. - Agent references now resolve ONLY against skills/llm-to-bedrock (_AGENTS_OWNER), including the <BDD_DIR>-style rglob, which is scoped to that skill. - Tests: scripts/pyproject.toml present only under agent-advisor is MISSING; present under llm-to-bedrock resolves to llm-to-bedrock's copy even when agent-advisor also has it; the fixture tree's agents/worker.md reference to another skill's file is now correctly MISSING. - Real repo unchanged: 1118 references, 0 missing, 7 baselined.
…for --fixtures; enums survive example merges; required table headings; real estimate/graviton/workloads paths Review round 2 on aws#385 (seven P2s, plus the Azure composite-source enum raised on aws#386/aws#387 against the same code): - The shipped baseline describes the fixture corpus (every glob is fixtures/...), so it now applies by default only to --fixtures. `--run-dir <run> --skill <s>` runs with no baseline unless --baseline names one, and then every entry must still match (a valid real run previously failed with 21 STALE_BASELINE errors). - Node.merge keeps a declared enum when another example shows a concrete member (`"confidence": "full"` no longer erases `full|reduced`); a concrete token the document shows joins the set, only a free-text example clears it. Adopted children are copied, so refining `design_constraints.cpu_architecture` no longer widens the generic `<key>` template's `chosen_by`. GCP schema-preferences.md example now lists `derived`. - `(or a "+"-joined set)` on an enum comment sets Node.enum_join: `live+terraform` is valid, `terraform+bogus` is not, closed enums elsewhere stay closed. - Heading match is exact or word-boundary prefix: `graviton` no longer selects `graviton_profile`; the Azure design manifest names `graviton block` explicitly. - Wrapped arrays (`{"workloads": [...]}` under a `workloads[]` heading) merge at the array level, so workload key/type drift is detected; bare item examples still merge into the item. - `(REQUIRED)` headings are read from every heading, not only fenced ones: Heroku `metadata` / `apps[]` / `resources[]` are now required; OPTIONAL sections unchanged. Two baseline entries record the hand-authored Heroku seed inventories that lack them. - GCP estimate: scenario_deltas, breakdown.observability, breakdown.security_baseline and breakdown.security_baseline_compliance bind to their object keys (line anchors separate the two Security Baseline blocks); nested drift inside them is still caught. - Heroku workshop `preferences_subset` is the documented open knob map (open_paths). 15 regression tests added (unit + CLI against the real contracts); each fails on the previous head. Fixture gate: 50 artifacts, 0 errors, 50 baselined, 0 stale.
… token; explicit bases are authoritative Review round 2 on aws#387 (the two findings unique to this tool): - The run-artifact skip looked 40 characters back, so an earlier, separate token (`$MIGRATION_DIR/`) swallowed the schema reference that followed it on the same line (azure design-ai.md:197) — renaming that schema passed CI. The skip now reads only the token's own contiguous prefix; `<run_dir>/…`, `.migration/<run>/…` and `$MIGRATION_DIR/…` forms are still skipped. Real repo: 1157 references (2 previously swallowed, both resolve). - An explicit base (`<SKILL_BASE>/`, `$PLUGIN_ROOT/`, `$GCP_BASE/`, `./`, `../`) is now authoritative: no library-origin / gcp-dependent / agents fallbacks after it, so a same-named file under another skill cannot make a dead `<SKILL_BASE>/scripts/x.py` look alive. Unqualified references keep every fallback; `<SKILL_BASE>/../gcp-to-aws/SKILL.md` traversal still resolves. Two regression tests added; each fails on the previous head. Real repo: PASS, 0 missing, 7 baselined, 0 stale.
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # plugins/aws-startup-advisor/skills/azure-to-aws/references/shared/schema-preferences.md
…y-gate Keep clarify_status alongside the metadata block in the preferences example, and admit the unfired licensing stub keys the assembler already writes. Co-authored-by: Cursor <cursoragent@cursor.com>
…ferences example The fast-path fixture writes licensing as the N/A stub, and the artifact check was rejecting those keys. Co-authored-by: Cursor <cursoragent@cursor.com>
An empty-array example no longer leaves the item typed as any, quoted comment enums are enforced, and a plugin without skills/ no longer crashes the vendored scan. Co-authored-by: Cursor <cursoragent@cursor.com>
…relative prefixes An empty example array kept item type any after a later object section, and a short relative path still matched the unprefixed file under the skill root. Co-authored-by: Cursor <cursoragent@cursor.com>
…ck estimate snapshots A standalone REQUIRED comment was dropped, so a discovery inventory without clarify_fast_path passed. Scenario estimate copies also skipped the active estimate shape, so an unknown pricing bucket failed on the live file and passed on the snapshot. Co-authored-by: Cursor <cursoragent@cursor.com>
…d drift values An AI-only preferences file was checked against the infra shape, numeric GCP drift failed as a string, and a plugin with no skills directory crashed the vendored-copy gate.
…y-gate # Conflicts: # plugins/aws-startup-advisor/README.md
added 2 commits
October 13, 2026 17:41
…design-ai-contract Conflicts resolved (5 files), zero functionality lost: - plugins/aws-startup-advisor/README.md: kept HEAD's scripts/ bullet (superset of upstream's, includes aws-design-ai.schema.json sentence) - plugins/aws-startup-advisor/fixtures/README.md: kept HEAD's new aws-design-ai/ bullet, placed after ai-workload-profiles/ and before heroku-workshop/ - plugins/aws-startup-advisor/scripts/artifact-contracts.json: kept HEAD's aws-design-ai.json shared-contract entry right after ai-workload-profile.json - plugins/aws-startup-advisor/mise.toml: took HEAD's empty side in both conflict regions (HEAD already defines lint:skill-refs, tasks.ci, and lint:model-id elsewhere in the file, complete and matching upstream) - plugins/aws-startup-advisor/tests/test_validate_artifacts.py: took upstream's stronger neutral_checked/neutral_findings/neutral_shared_count variant of test_neutral_matrix_shared_artifacts_extend_the_mapped_fixture_count, which calls va.validate_run_dir per unmapped fixture dir and asserts exact equality + zero findings, over HEAD's weaker count-only >= variant Verification evidence: 1) mise.toml `^[tasks` grep — every task name exactly once, no duplicates: 1 ci 1 lint:artifacts 1 lint:md 1 lint:model-id 1 lint:model-staleness 1 lint:skill-refs 1 lint:vendored-parity 1 test 1 test:heroku-source-review [tasks.ci].depends = ["lint:md","lint:model-id","lint:model-staleness", "lint:artifacts","lint:vendored-parity","lint:skill-refs","test", "test:heroku-source-review"] — complete, every name resolves to a real task. 2) python -m json.tool scripts/artifact-contracts.json — valid JSON, exit 0. 3) Isolated test (3 unmapped families: ai-migration-matrix, ai-workload-profiles, aws-design-ai): `pytest tests/test_validate_artifacts.py -k test_neutral_matrix_shared_artifacts_extend_the_mapped_fixture_count -v` -> 1 passed, 61 deselected. 4) Full plugin suite: `pytest tests/ -q` -> 391 passed. 5) `mise run build` (repo root): lint:manifests, lint:spec, lint:skill-parity, lint:md, security — all passed, exit 0. 6) `mise -C plugins/aws-startup-advisor run ci`: lint:vendored-parity (49 vendored files byte-identical), lint:artifacts (80 artifacts checked, 0 errors, 50 baselined, 14 info), lint:model-id, test (391 passed), lint:skill-refs (1293 refs checked, 0 missing), lint:md, test:heroku-source-review (44 passed) — all passed, exit 0.
Target-only, additive JSON Schemas for estimation-ai.json and generation-ai.json, stacked on merged aws#415 (ai-workload-profile) and aws#416 (aws-design-ai). cost_by_source[] / artifacts_by_source[] carry the ai_sources[]/gateways[] traceability chain forward per-source, per-model; model_observations[] intentionally not carried forward (identity already resolved by Design). No runtime wiring, no producer change — estimate-ai.md / generate-ai.md / generate-artifacts-ai.md are untouched in both skills. Added: - scripts/contracts/estimation-ai.schema.json, generation-ai.schema.json - skills/shared/ai/schema-estimate-ai.md, schema-generate-ai.md (vendored byte-identically into gcp-to-aws and azure-to-aws only) - fixtures/estimation-ai/{multi-provider-direct,openrouter-gateway-multi-upstream}/estimation-ai.json - fixtures/generation-ai/{multi-provider-direct,openrouter-gateway-multi-upstream}/generation-ai.json - tests/test_estimation_ai_contracts.py, test_generation_ai_contracts.py (positive validation, malformed/duplicate/overlapping rejection, untraceable/dropped-model negatives, cross-phase projection into aws-design-ai.json and estimation-ai.json, vendored byte-identity) Registered two 'shared' entries in artifact-contracts.json, two fixture bullets in fixtures/README.md, two rows in skills/shared/ai/README.md, and a scripts/ clause in README.md. Verification run from plugins/aws-startup-advisor (all commands executed for real, output below): uv run --with pytest python -m pytest tests/ -q -> 429 passed (from the worktree root) mise run build -> lint:manifests, lint:spec, lint:skill-parity, lint:md, security all PASS mise run ci -> lint:artifacts: PASS — 84 artifact(s) checked, 0 error(s) lint:vendored-parity: PASS — 53 vendored file(s) byte-identical lint:skill-refs: PASS — 1339 reference(s) checked, 0 missing lint:md: 0 error(s) lint:model-id: OK (18 retired IDs enforced) test: 429 passed test:heroku-source-review: 44 passed
herosjourney
marked this pull request as draft
October 14, 2026 01:30
…nerate-contracts # Conflicts: # plugins/aws-startup-advisor/README.md # plugins/aws-startup-advisor/fixtures/README.md # plugins/aws-startup-advisor/scripts/artifact-contracts.json # plugins/aws-startup-advisor/skills/azure-to-aws/references/vendored/README.md # plugins/aws-startup-advisor/skills/gcp-to-aws/references/vendored/README.md # plugins/aws-startup-advisor/skills/shared/ai/README.md
herosjourney
marked this pull request as ready for review
October 19, 2026 04:33
leon1418
requested changes
Oct 19, 2026
leon1418
left a comment
leon1418
left a comment
Contributor
There was a problem hiding this comment.
[🤖 AI review 🤖] Reviewed a0a8867 with both required review perspectives. Requesting changes for two P2 contract issues:
- Keep the target schemas separate from current GCP/Azure run validation, with compatibility tests for both existing artifact shapes.
- Represent unknown costs and uncomputable differences explicitly, with missing-baseline/missing-rate tests and matching canonical/vendored documentation.
Validation on original source: 429 plugin tests passed; artifact validation checked 84 fixtures with no new errors; vendored parity, skill references and diff checks passed. The compatibility and unknown-cost probes reproduced the failures described inline. Current GitHub checks are successful; these findings concern cases the existing tests do not cover.
Keep target-only estimate and generate schemas out of named skill run validation while retaining neutral fixture and self-check coverage. Preserve unavailable estimate costs and dependent deltas as null, with synced documentation and regression coverage. Verification: - affected pytest modules: 87 passed - startup-advisor artifact validation: 84 checked, 0 errors - vendored parity: 53 files byte-identical - startup-advisor CI: 435 Python tests passed; 57 Node tests passed; all lint and validation tasks passed
leon1418
previously approved these changes
Oct 20, 2026
# Conflicts: # plugins/aws-startup-advisor/fixtures/README.md
leon1418
approved these changes
Oct 21, 2026
5 of 9 tasks
herosjourney
pushed a commit
to herosjourney/agent-toolkit-for-aws
that referenced
this pull request
Oct 23, 2026
Addresses PR aws#483 review 5620455181 by vagmur-aws, comment 4353203474 (skills/shared/migration/bma-input-patterns.json:3 and closure.json): 4a51ac3 was a merge commit on the aws#449 branch, not an ancestor of main, and predated aws#369. - selected_base 4a51ac3 -> 9ab5063 (upstream/main) git merge-base --is-ancestor 9ab5063 upstream/main -> exit 0 - Regenerated closure.json and both bootstrap manifests with the PR's own tooling (sync-runtime-copies.py, build-bma-closure.py). - Verified every aws#369 hunk in the 7 adapted sources (clarify-ai-only, design-ai, discover-preview, estimate-ai, generate-ai, generate-artifacts-ai, openai-on-bedrock) is present in the BMA copies; no content drift to port. Vendored files pass lint:vendored-parity. - New test test_selected_base_pins_agree_and_are_consistent asserts one selected_base across closure.json, both bootstrap manifests, all three bma-input-patterns.json copies, runtime-copy-manifest.json, both fixtures, and the BASE constant in the classifier copies and tools. Hashes: closure.json file 7a58d70792bc363ab3d9e86156236dd3429905cc12ba9414c951bfa62717a783 closure.json self 457c32fc8ca1b0e6c74975500a6f921e2464aba1eb4cba0dbd279a0e261fa5da bootstrap manifest file 267fe013d216f85dd8c272126099d3f3eb2b909901324d14ab1a47ee06751b02 bootstrap manifest self 2447652f905ab40b2db3220883abb41c470106d118402f23b68fc6906624faab
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
There is no pinned target contract for the AI-authored fragments of the Estimate and Generate phases.
estimation-ai.jsonandgeneration-ai.json— the AI slices ofestimate.mdandgenerate-artifacts-ai.md— have no JSON Schema, so nothing validates a future producer against them and there is no normalized shape for a shared AI engine (issue #389's extraction) to target downstream of #415'sai-workload-profile.jsonand #416'saws-design-ai.json. Without a pinned contract here, extracting the shared AI engine out ofgcp-to-awswould have to invent the Estimate/Generate AI shapes from scratch instead of building against an already-reviewed target.Fix
Add target-only JSON Schemas and canonical vendored docs for the Estimate-phase and Generate-phase AI fragments, plus nested fixtures and a contract-projection test suite, mirroring the pattern #415 and #416 established for
ai-workload-profile.jsonandaws-design-ai.json.scripts/contracts/estimation-ai.schema.jsonandscripts/contracts/generation-ai.schema.json(Draft-07).skills/shared/ai/schema-estimate-ai.mdandskills/shared/ai/schema-generate-ai.md, vendored byte-identically undergcp-to-awsandazure-to-awswith matching README table rows.scripts/artifact-contracts.json'ssharedsection gains entries for both artifacts, andvalidate-artifacts.py(plustests/test_validate_artifacts.py) registers them solint:artifactsdiscovers and self-checks the fixtures.fixtures/estimation-ai/<case>/andfixtures/generation-ai/<case>/, covering a multi-provider-direct case and an OpenRouter-gateway multi-upstream case.tests/test_estimation_ai_contracts.pyandtests/test_generation_ai_contracts.py.Scope
Target-only, matching #410/#415/#416. This PR does not wire anything into
estimate.mdorgenerate-artifacts-ai.md, add a producer for either artifact, or change any phase behavior, question text, model mapping, pricing value, or artifact name. Schema validity here does not imply an implemented detector, adapter, route, or runtime support of any kind. It affects only the two schemas, their canonical/vendored docs, the fixtures, the artifact-contract registration, and the contract tests.Base note
The earlier stacking caveat is now resolved: #410, #415, and #416 have all merged, so this PR's base retargets to
mainand the diff has shrunk to just this PR's own 22 files (no inherited stack files remain). The branch is currently a few commits behindmain(it predates #454 and #458); amainmerge/rebase and a CI re-run on the fresh base are the only remaining step before merge — there is no conflict.In code, for reviewers
scripts/contracts/estimation-ai.schema.json,scripts/contracts/generation-ai.schema.json— the two Draft-07 target schemas.skills/shared/ai/schema-estimate-ai.md,skills/shared/ai/schema-generate-ai.md— canonical prose, with byte-identical vendored copies undergcp-to-aws/references/vendored/ai/andazure-to-aws/references/vendored/ai/and matching rows in all threeREADME.mdtables (vendored-parity gate).scripts/artifact-contracts.json+scripts/validate-artifacts.py+tests/test_validate_artifacts.py— register and self-check the two artifacts underlint:artifacts.fixtures/estimation-ai/{multi-provider-direct,openrouter-gateway-multi-upstream}/estimation-ai.jsonand the matchingfixtures/generation-ai/...— positive fixtures, including an OpenRouter gateway multi-upstream case.tests/test_estimation_ai_contracts.py,tests/test_generation_ai_contracts.py— contract-projection suites.Validation
Validation recorded for current head
dd810e147862449e45b01a90e4e6eb1d999689b4:mise -C plugins/aws-startup-advisor run cipassed; GitHub Build, Startup advisor, CodeQL, dependency-review, and PR-title checks are green on this head.pytest tests/test_estimation_ai_contracts.py tests/test_generation_ai_contracts.py— contract tests pass.lint:artifacts,lint:vendored-parity, andlint:skill-refspass (the two new artifacts are registered; vendored copies are byte-identical).The green run was against the pre-#454/#458 base this branch currently sits on; it should be re-confirmed after the
mainmerge. The tests validate the schemas/fixtures against the contract; there is no producer and no runtime path to exercise.Type of Change
Startup Advisor review checklist
By submitting this PR, I confirm that:
mise -C plugins/aws-startup-advisor run ciand all checks pass.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.