Repository navigation
feat(aws-startup-advisor): add GPT-6 family to agent-advisor model catalog - #454
Conversation
Add openai_gpt_6_astra, openai_gpt_6_sol, openai_gpt_6_luna, and openai_gpt_6_1_sol to the agent-advisor OpenAI-to-Bedrock catalog, ported from the already-verified facts in shared/ai/ai-openai-to-bedrock.md and shared/ai/bedrock-pricing-cache.md (PR aws#433). Entries follow the openai_gpt_5_6_sol template shape; placed after openai_gpt_5_4 (not before openai_gpt_5_6_sol) so the engine's dict-order fallback keeps selecting GPT-5.x candidates for GPT-5.x sources. Extended the catalog's top-level disclaimer and model-recommend.md's OpenAI routing paragraph (Step 2) to describe GPT-6 same-model routing, its narrower in-region mantle footprint (Astra: us-east-1+us-west-2; Sol/Luna/6.1 Sol: us-east-1 only), the mantle Chat Completions deviation from the GPT-5.x pattern, and GPT-6 Astra's CRIS Invoke gap. Added the four new catalog keys to test_scoring.py's _POOL_LIFECYCLE_KEYS (the one existing test that enumerates the full model-selection pool by name). Verification (see .agents/tasks/gpt6-agent-advisor-catalog/verification.md in the startups-pr repo for full detail): - python3 -m json.tool ... openai-bedrock-2026-08-21.json: valid JSON - skills/agent-advisor/scripts: uv run --with pytest python -m pytest -q -> 323 passed - mise -C plugins/aws-startup-advisor run ci -> all checks green (lint:md 0 errors, lint:model-id OK, lint:model-staleness fresh, lint:artifacts PASS, lint:vendored-parity PASS, lint:skill-refs PASS, test 391 passed, test:heroku-source-review 44 passed) Out of scope per plan/SCOPE GUARD: skills/shared/ai/*, gcp-to-aws, azure-to-aws, llm-to-bedrock skills; openai_model_recommendation.py's family-detection/candidate-order dispatch logic (new GPT-6 source IDs still fall through to 'unknown' family -- a separate, flagged gap).
…ling Review (CHANGES_REQUESTED) found two numeric data-entry errors in all four new GPT-6 catalog entries: - context_window was 272000, but the shared source facts (ai-openai-to-bedrock.md, bedrock-pricing-cache.md) state the real context window is 1.05M tokens; 272K is only the short/long-context pricing-tier boundary, the same relationship the existing openai_gpt_5_6_sol/terra/luna entries already encode as 1000000. Fixed to 1050000 across Astra, Sol, Luna, and 6.1 Sol. - output_token_ceiling was left "unknown" despite being a sourced fact (128,000 for Astra/Sol/Luna, 131,072 for 6.1 Sol). Populated, since _numeric_requirement_conflict in openai_model_recommendation.py fails closed on "unknown" and would needlessly block a hard output-token requirement the real ceiling could satisfy. Updated each entry's capability_source prose to match the corrected fields (1.05M window / 272K pricing boundary; names the populated output ceiling) without touching any other sourced claim. Verification (full detail in .agents/tasks/gpt6-agent-advisor-catalog/verification.md in the startups-pr repo): - python3 -m json.tool ... openai-bedrock-2026-08-21.json: valid JSON - mise -C plugins/aws-startup-advisor run ci -> all checks green (lint:md 0 errors, lint:model-id OK, lint:model-staleness fresh, lint:artifacts PASS, lint:vendored-parity PASS, lint:skill-refs PASS, test 391 passed, test:heroku-source-review 44 passed)
leon1418
left a comment
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Reviewed 96ee9d7cc64897a2b1503ca4ad37965fe6b9f6e4 with OCR delegation, an isolated independent reviewer, and repository/consumer checks. Two corrections remain:
- Connect the four GPT-6 catalog entries to the deterministic selector and add end-to-end routing tests. The new same-model behavior is currently unreachable.
- Correct the Sol/Luna image-input capability entries using the current model cards, so completing routing also supports their documented vision workloads.
Validation: 323 advisor tests and 391 plugin tests passed; model ID/staleness, artifact, vendored-parity, skill-reference, changed-Markdown lint and diff checks passed. A 96-case schema-valid base/head probe selected no GPT-6 model; the independent reviewer reproduced the same gap, and its 32 cases were verified against the original checkout. These are offline checks, not target-account inference validation.
leon1418
left a comment
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Follow-up review of 015b22ea040efe1cccd6246a27b84c85b73c980f completed with OCR delegation, one isolated independent reviewer, repository checks and a passing convergence gate. The original GPT-6 selection and Sol/Luna vision fixes are verified. Two P2 issues introduced by the routing changes remain: incomplete Chat-path verification/migration guidance, and exception/schema failures in early region-filtered decisions. The consolidated corrections and reproductions are in the original routing conversation.
Validation: 342 advisor tests and 391 plugin tests passed, plus model-ID/lifecycle, artifact, vendored-parity, skill-reference, changed-Markdown and diff checks. The independent reviewer passed 169 supplied tests; all 33 of its current/base/prior comparisons were reproduced against original Git source. No live AWS invocation was performed. Approval remains for the user.
Verification: - uv run --no-project --with pytest --with jsonschema python -m pytest test_openai_model_recommendation.py test_verify_model_path.py -q (90 passed) - mise -C plugins/aws-startup-advisor run ci && git diff --check (391 Python tests, 57 Node tests, all lint/parity checks passed)
leon1418
left a comment
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Follow-up review of b2bcf5d8e65d79554b59003bb57c5b035a77c759 completed with a passing hybrid/convergence gate. The previous Chat integration and region-decision findings are fixed. One P2 remains: GPT-6 runtime selection generates unsupported EU/AU CRIS IDs despite required Geo residency. This case was present at 015b22e, and I missed it in the previous review. Details, public model-card evidence and the bounded correction are in the original routing conversation.
Validation: 349 advisor and 391 plugin tests passed, plus repository contract checks and both changed Markdown files. The independent reviewer passed 152 tests; all 36 of its current/base/prior outputs were reproduced against original Git source. These are offline checks, not target-account inference validation.
Semantic union preserves GPT-6 exact routing, image input, Mantle Chat and Responses verification, region-aware options, and Converse fallback while adopting the 2026-09-24 catalogs, Opus 5.5 profile architecture, cache invalidation, and upstream hardening. Fix unsupported GPT-6 EU/AU CRIS IDs by applying model-specific profile eligibility before selection and in decision options. Verification: 152 pre-merge focused; 336 combined GPT-6/Opus; 509 agent-advisor scripts; 444 startup-advisor CI; model/staleness/vendored/skill-ref lints passed (51 vendored files, 1316 refs); root validate/build constituent tasks passed (2607 Markdown files, no leaks); 9 JSON/TOML files parsed; diff/marker checks passed.
Gap
The agent-advisor's model-recommendation catalog
(
plugins/aws-startup-advisor/skills/agent-advisor/references/models/openai-bedrock-2026-08-21.json)is a separate file tree from the migration-skills' shared AI references
(
skills/shared/ai/*). Merged PR #433 added GPT-6/UltraFast coverage to the sharedreferences, but that coverage never reached the agent-advisor's own catalog, so
agent-advisor still could not recommend any GPT-6 model.
What this PR adds
openai-bedrock-2026-08-21.json:openai_gpt_6_astra,openai_gpt_6_sol,openai_gpt_6_luna,openai_gpt_6_1_sol, ported from the facts alreadyverified in
ai-openai-to-bedrock.md/bedrock-pricing-cache.md.model-recommend.mddescribing real GPT-6 routing differences:narrower in-region footprint, Astra's CRIS
Invokegap, and the mantle Chat-Completionsdeviation from GPT-5.x (not a cosmetic reword).
disclaimerstring, and addedthe four new keys to
test_scoring.py's_POOL_LIFECYCLE_KEYSfixture so existingpool/lifecycle drift tests cover them.
context_windowas1050000(1.05M tokens, matching the conventionalready used by the sibling
openai_gpt_5_6_sol/terra/lunaentries) rather than the 272Kshort-context pricing-tier boundary, and populated
output_token_ceiling(128000forAstra/Sol/Luna,
131072for 6.1 Sol) instead of leaving it"unknown"— these were caughtand fixed in a review round, see Verification below.
Relationship to #390
This PR supersedes/fixes the gap that open draft PR #390 attempted to address. PR #390 only
reworded one sentence to claim GPT-6 Astra UltraFast support, without adding the actual
model entry to the catalog — so its claim would be false if merged as-is. This PR adds the
real catalog data and updates the routing prose to match. Recommend the maintainer close
#390 in favor of this PR (not closing it myself).
Verification
python3 -m json.tool openai-bedrock-2026-08-21.json— valid JSON._POOL_LIFECYCLE_KEYS).mise run ci): all checks green —lint:md(797 files, 0 errors),lint:model-id,lint:model-staleness,lint:artifacts(80 checked, 0 errors),lint:vendored-parity(49 files byte-identical),lint:skill-refs(1293 checked, 0missing), plugin
testsuite (391 passed),test:heroku-source-review(44 passed).context_windowandoutput_token_ceiling, see above); both were fixed and the full CI suite was re-run clean(391 passed, all lint checks green, valid JSON).
skills/shared/ai/*, other migration skills, theAnthropic catalog, or
openai_model_recommendation.py's detection/candidate-order logic.