Repository navigation
Description
Problem and requested outcome
AutoSkillit should let users and recipes name a model class (opus, sonnet, haiku) and never think about vendor model versions. Each class should resolve, per backend, to the latest model of the matching tier. "Latest" should be defined in exactly one place per backend.
Concrete model versions should not appear anywhere else unless pinning is the explicit intent: recipes, skills, agent definitions, config defaults, probes. Switching a step between Claude Code and Codex should need no model edits.
Today the translation layer exists but violates this in three ways:
- It resolves classes to hard-coded snapshots that lag the vendors.
- Concrete IDs are scattered outside the translation table.
- A concrete ID routed to the wrong backend is passed through unchanged and fails at launch.
Owner's stated intent
- Recipes and configuration say
opus(orsonnet/haiku), not a version. The class means "the latest model of that tier on whichever backend runs the step". - The latest model per backend and tier is defined once and updated in one place. Nothing else hard-codes versions, so a new model generation does not require edits across plans, recipes, agents and tests.
- Cross-vendor tier equivalence is AutoSkillit's job, not the user's:
opuson Codex →gpt-6.1-solat effortxhigh. These are not equal in capability; this is the vendor-to-vendor tier correspondence.- Luna (
haikutier on Codex) runs atxhighormax. Lower efforts are not stable enough in practice.
- No maximum version constraints. Only minimums that a real incompatibility requires.
Current behavior (verified on develop @ b1ed939f6 and 304f81179)
Translation table: hard-coded snapshots. In src/autoskillit/core/types/execution/_type_backend.py:
CLAUDE_MODEL_ALIASES(:348-354):opus→claude-opus-5-5,sonnet→claude-sonnet-5-5,haiku→claude-haiku-5-5.CODEX_MODEL_ALIASES(:356-363):opus→gpt-6-sol,sonnet→gpt-6-sol,haiku→gpt-6-luna.CODEX_MODEL_ALIASES_LAST_VERIFIED = "2026-09-23".CODEX_EFFORT_MAPPING(:378-382):sonnet→medium,opus→high,haiku→high.CODEX_VALID_MODEL_IDS:{"gpt-5.5", "gpt-6-luna", "gpt-6-sol"}.
The table already lags. codex debug models --bundled on Codex CLI 0.164.0 lists these as available (visibility: list), besides gpt-6-sol and gpt-6-luna:
gpt-6.1-sol(efforts up toultra);gpt-6-astra("Frontier intelligence for the most demanding work");gpt-5.6-sol,gpt-5.6-terra,gpt-5.6-luna,gpt-5.5.
AutoSkillit cannot use gpt-6.1-sol at all. Managed Codex issuance rejects any ID outside CODEX_VALID_MODEL_IDS with unsupported managed Codex model (src/autoskillit/execution/backends/_codex_managed_route.py:62-63).
Generations have required code changes every time.
- Raise Claude Code and Codex CLI support floors #5168 moved
opustoclaude-opus-5-5and raised floors. - Move Claude model pins to the 5.5 generation and raise the Claude Code floor #5276/Pin Claude 5.5 models and raise Claude Code floor to 2.1.296 #5318 moved
sonnet/haikuto 5.5. - On 0.10.1013 the classes still meant
claude-opus-5/claude-sonnet-5/ the barehaikualias, which resolves to Haiku 4.5.
Users therefore pinned concrete IDs in their own config to get current models. That pinning is what then breaks cross-backend use (below).
Concrete IDs outside the table.
- 10 bundled agent definitions declare
codex: model: gpt-6-lunadirectly (src/autoskillit/agents/*.md), with per-agentreasoning_effortvaluesmax(5),xhigh(4) andmedium(1). gpt-6-solliterals:src/autoskillit/execution/backends/_codex_managed_route.py:77,_probe_cache.py:49,_explorer_conformance.py:36.gpt-6-lunaliteral:src/autoskillit/core/plugins/agent_definition.py:58.src/autoskillit/execution/evidence/anomaly_detection.py:79-82lists older Claude IDs. Review whether that is intentional.
Unknown or foreign IDs are passed through unchanged.
ClaudeBackend.translate_modelreturnsCLAUDE_MODEL_ALIASES.get(base, base)(src/autoskillit/execution/backends/claude.py:273).CodexBackend.translate_modelreturnsCODEX_MODEL_ALIASES.get(base, base)(src/autoskillit/execution/backends/codex.py:382).- Nothing at config load,
open_kitchenpreflight ordoctorchecks that a pinned model belongs to the backend the step is routed to. A zero-token route dry run of a real user config with the candidate's own resolution functions shows:
| Step | Config model pin | Backend (config pins) | Sent to CLI | Outcome |
|---|---|---|---|---|
remediation.investigate |
claude-opus-5-5[1m] |
codex (packaged default; user's remediation: {} does not clear it because layers deep-merge) |
claude-opus-5-5 |
managed-join refused: unsupported managed Codex model: claude-opus-5-5 |
implementation.dry_walkthrough |
gpt-6.1-sol |
claude-code |
gpt-6.1-sol |
Claude Code is asked to run a GPT model |
Bundled recipes are already compliant: they name only classes (no claude-* / gpt-* IDs in src/autoskillit/recipes/*.yaml).
Acceptance criteria
- One authoritative definition per backend maps each class to its current latest model and effort. Every other production surface obtains models through it: agent definitions, probes, managed-route defaults, config defaults, docs.
- Concrete IDs outside it fail an architectural guard, apart from an explicit, documented allowlist for intentional pins.
- Tests reference the definition rather than restating IDs.
- Codex mapping:
opus → gpt-6.1-solatxhigh;haiku/Luna atxhighormax(owner's requirement).sonnetis chosen and documented in the same definition.- Agent definitions declare a class (plus an effort only where a deviation is intended), not
gpt-6-luna.
- Agent definitions declare a class (plus an effort only where a deviation is intended), not
- The managed Codex allowed-model set derives from the same definition, or from the installed CLI's catalog, so the defined latest model is always admissible.
- Decide and document how "latest" is maintained. Either (a) one place updated per generation, or (b) resolved at launch from the installed CLI's own catalog/aliases (Codex
debug models --bundled, which managed issuance already reads; Claude Code's aliases), with the single definition as fallback.- State the trade-off: pinned snapshots avoid silent behavior changes when a CLI updates (Bare Claude model aliases silently resolve to an outdated model: opus[1m] yields claude-opus-4-6, with no verification registry or doctor check (Codex has both) #4412), while auto-resolution tracks vendors without releases.
- Either way, no surface besides the definition needs editing for a new generation.
- Detect foreign model pins early. A class or ID that cannot be served by the backend a step is routed to is reported at config/preflight/
doctortime with the remedy (use a class), instead of failing at launch.- Concrete IDs given for a backend they do not belong to are either translated to the equivalent class or rejected clearly.
- Minimum versions only. Model-driven CLI floors stay minimums; no maximums are introduced.
- Regression tests cover class resolution per backend, latest-model admissibility in managed Codex issuance, agent-definition class resolution, and the two foreign-pin cases above.
Related
- Move Claude model pins to the 5.5 generation and raise the Claude Code floor #5276 / Pin Claude 5.5 models and raise Claude Code floor to 2.1.296 #5318: Claude 5.5 move and Claude Code floor.
- Raise Claude Code and Codex CLI support floors #5168: earlier floor and model move.
- Codex model aliases route default sessions to forbidden gpt-5.4 #4218: Codex aliases routed to a forbidden model.
- Bare Claude model aliases silently resolve to an outdated model: opus[1m] yields claude-opus-4-6, with no verification registry or doctor check (Codex has both) #4412: alias staleness.
- Codex cook session launches at medium effort with Fast (priority) service tier #5218: cook defaults
gpt-6-soltohigh. - Found during release validation of
develop@304f81179(0.10.1308).
Testing with Codex in Docker
CI cannot verify this fix. The E2E workflow routes every Codex step to Claude Code (scripts/e2e/autoskillit-config.yaml), and the unit gate never launches Codex. Verify it with a real Codex CLI in the repository's own image.
1. Build the user image for the fix commit with the Codex CLI that reproduces the bug
The Dockerfile pins CODEX_VERSION (0.156.1 at the time of writing). The defects reproduce on Codex CLI 0.164.0, so override the pin at build time. Do not edit the Dockerfile pin (see scripts/docker/README.md).
source_sha=$(git rev-parse HEAD)
git archive "$source_sha" | docker build --file scripts/docker/Dockerfile --target user \
--build-arg CODEX_VERSION=0.164.0 --tag "autoskillit-user:$source_sha-codex0.164.0" -scripts/docker/verify-image compares tool versions with the Dockerfile pins, so it rejects this override. Skip it for this image.
Also verify against the pinned version: build a second tag without --build-arg. A fix must work on both.
2. Start a throwaway container
docker run -d --name codex-fix-check --init \
--security-opt seccomp=unconfined --security-opt apparmor=unconfined \
--tmpfs /dev/shm:rw,exec,nosuid,nodev,size=4g,mode=1777 \
"autoskillit-user:$source_sha-codex0.164.0" sleep infinityseccomp/apparmormust be unconfined for Codex's Linux sandbox (bubblewrap) to run inside Docker.- The exec
/dev/shmtmpfs is required for AutoSkillit's process lifecycle. - The image user is
autoskillitwithHOME=/home/autoskillit. AutoSkillit's tool interpreter is~/.local/share/uv/tools/autoskillit/bin/python.
3. Supply your own credentials at run time (never bake them into an image)
docker exec codex-fix-check mkdir -p /home/autoskillit/.codex /home/autoskillit/.claude
docker cp ~/.codex/auth.json codex-fix-check:/home/autoskillit/.codex/auth.json
docker cp ~/.claude/.credentials.json codex-fix-check:/home/autoskillit/.claude/.credentials.json # only for recipe runs
docker exec -u root codex-fix-check sh -c 'chown -R autoskillit: /home/autoskillit/.codex /home/autoskillit/.claude && chmod 600 /home/autoskillit/.codex/auth.json /home/autoskillit/.claude/.credentials.json'- Copy credentials that were refreshed recently. The container uses the same refresh token as the host, and a refresh on either side can rotate it.
- Remove the container when you finish (
docker rm -f codex-fix-check). Its home now holds credentials.
4. Work in a local-only repository
Recipe runs can push branches and open PRs or issues. Give them a repository with no remote, so nothing can land on GitHub:
docker exec codex-fix-check bash -c '
set -e; mkdir -p /home/autoskillit/probe/sandbox /home/autoskillit/probe/tests /home/autoskillit/probe/.autoskillit
cd /home/autoskillit/probe && git init -q -b main
printf "def word_count(text: str) -> int:\n return len(text.split())\n" > sandbox/text.py
: > sandbox/__init__.py; : > tests/__init__.py
printf "branching:\n default_base_branch: main\ntest_check:\n command: [\"python3\", \"-m\", \"unittest\"]\n" > .autoskillit/config.yaml
git add -A && git -c user.name=probe -c user.email=probe@localhost commit -qm init'For autoskillit fleet run, enable the gated features in the container's user config (/home/autoskillit/.autoskillit/config.yaml):
features:
experimental_enabled: true5. Safeguards
- Order. Run the zero-token probes first. Run a live session only after they pass.
- Cost.
- Live probes are single
codex execturns: about 10k–100k input tokens, mostly cached. - A
fleet run planneralso runs a Claude Code orchestrator: about 1M cached-read tokens.
- Live probes are single
- Time limits. Use
timeouton every live command and--timeout-seconfleet run. - Evidence. Session rollouts are under
~/.local/share/autoskillit/logs/codex-sessions/and~/.codex/sessions/.session_meta/turn_contextrecord the model, effort and sandbox policy;response_itemrecords tool calls and outputs. Attach the relevant excerpts to the PR.
Verification steps for this issue
Sequence this after #5344, which changes the same Codex catalog and admission code.
A. Zero-token: what the installed Codex CLI offers.
docker exec codex-fix-check bash -c 'codex debug models --bundled | ~/.local/share/uv/tools/autoskillit/bin/python -c "
import json, sys
for m in json.load(sys.stdin)[\"models\"]:
print(m.get(\"slug\"), m.get(\"visibility\"), [e[\"effort\"] for e in m.get(\"supported_reasoning_levels\", [])])"'The model this issue defines as latest for each class must appear here, with its effort.
B. Zero-token: the effective model and effort for every Codex-routed recipe step. No model turn runs here. The script resolves each step the way run_skill does, then asks managed-join issuance for its attested model and effort:
docker exec -w /home/autoskillit/probe codex-fix-check bash -c 'cat > /tmp/route_probe.py <<EOF
import json, sys, uuid, yaml
from importlib import resources
from pathlib import Path
from autoskillit.config import load_config
from autoskillit.execution.backends.codex import CodexBackend
from autoskillit.execution.headless._headless_model import resolve_model_pin
from autoskillit.server.lifecycle._guards import _resolve_backend_override
from autoskillit.server.managed_join_prelaunch import prepare_managed_join_context
project = Path(sys.argv[1]); cfg = load_config(project)
for path in sorted((Path(str(resources.files("autoskillit"))) / "recipes").glob("*.yaml")):
recipe = yaml.safe_load(path.read_text()) or {}; name = recipe.get("name") or path.stem
for step_name, step in (recipe.get("steps") or {}).items():
if not isinstance(step, dict) or step.get("tool") != "run_skill":
continue
pin = _resolve_backend_override(step_name, name, cfg.agent_backend)
if (pin.backend if pin else cfg.agent_backend.backend) != "codex":
continue
raw = str(step.get("model") or ""); model = resolve_model_pin("" if "\${{" in raw else raw, cfg, step_name=step_name, recipe_name=name).model
out = prepare_managed_join_context(backend=CodexBackend(), configured_model=model, state_root=project,
parent_id=f"probe-{uuid.uuid4().hex[:12]}", launch_context="direct")
att = getattr(out, "managed_join_attestation", None)
print(f"{name}.{step_name}: configured={model!r} ->",
(att.resolved_model, att.resolved_reasoning_effort) if att else repr(out)[:160])
EOF
~/.local/share/uv/tools/autoskillit/bin/python /tmp/route_probe.py /home/autoskillit/probe'Run it twice:
- With the packaged defaults.
- With a user config naming only classes, e.g.
model: {recipe_overrides: {remediation: {investigate: "opus[1m]"}}}.
Expected after the fix:
- Every Codex-routed step resolves to the single definition's latest model and effort for its class. The owner's mapping:
opus→gpt-6.1-solatxhigh;haiku(Luna) atxhighormax. - No step is refused.
- A concrete Claude ID pinned on a Codex-routed step is reported clearly, not passed to Codex.
Use the state root under the project as above, not /tmp. Codex warns on stderr for a /tmp home, and the catalog probe treats any stderr as a failure.
C. Live check. After #5344 and #5345, run one headless Codex step (for example the fleet run planner from those issues). Confirm its rollout's turn_context records the expected model and effort.
Activity