Skip to content

Model classes must resolve to the latest model per backend from one definition; no concrete model versions elsewhere #5342

Open

Description

@Trecek

Problem and requested outcome

AutoSkillit should let users and recipes name a model class (opus, sonnet, haiku) and never think about vendor model versions. Each class should resolve, per backend, to the latest model of the matching tier. "Latest" should be defined in exactly one place per backend.

Concrete model versions should not appear anywhere else unless pinning is the explicit intent: recipes, skills, agent definitions, config defaults, probes. Switching a step between Claude Code and Codex should need no model edits.

Today the translation layer exists but violates this in three ways:

  1. It resolves classes to hard-coded snapshots that lag the vendors.
  2. Concrete IDs are scattered outside the translation table.
  3. A concrete ID routed to the wrong backend is passed through unchanged and fails at launch.

Owner's stated intent

  • Recipes and configuration say opus (or sonnet / haiku), not a version. The class means "the latest model of that tier on whichever backend runs the step".
  • The latest model per backend and tier is defined once and updated in one place. Nothing else hard-codes versions, so a new model generation does not require edits across plans, recipes, agents and tests.
  • Cross-vendor tier equivalence is AutoSkillit's job, not the user's:
    • opus on Codex → gpt-6.1-sol at effort xhigh. These are not equal in capability; this is the vendor-to-vendor tier correspondence.
    • Luna (haiku tier on Codex) runs at xhigh or max. Lower efforts are not stable enough in practice.
  • No maximum version constraints. Only minimums that a real incompatibility requires.

Current behavior (verified on develop @ b1ed939f6 and 304f81179)

Translation table: hard-coded snapshots. In src/autoskillit/core/types/execution/_type_backend.py:

  • CLAUDE_MODEL_ALIASES (:348-354): opus→claude-opus-5-5, sonnet→claude-sonnet-5-5, haiku→claude-haiku-5-5.
  • CODEX_MODEL_ALIASES (:356-363): opus→gpt-6-sol, sonnet→gpt-6-sol, haiku→gpt-6-luna. CODEX_MODEL_ALIASES_LAST_VERIFIED = "2026-09-23".
  • CODEX_EFFORT_MAPPING (:378-382): sonnet→medium, opus→high, haiku→high.
  • CODEX_VALID_MODEL_IDS: {"gpt-5.5", "gpt-6-luna", "gpt-6-sol"}.

The table already lags. codex debug models --bundled on Codex CLI 0.164.0 lists these as available (visibility: list), besides gpt-6-sol and gpt-6-luna:

  • gpt-6.1-sol (efforts up to ultra);
  • gpt-6-astra ("Frontier intelligence for the most demanding work");
  • gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5.

AutoSkillit cannot use gpt-6.1-sol at all. Managed Codex issuance rejects any ID outside CODEX_VALID_MODEL_IDS with unsupported managed Codex model (src/autoskillit/execution/backends/_codex_managed_route.py:62-63).

Generations have required code changes every time.

Users therefore pinned concrete IDs in their own config to get current models. That pinning is what then breaks cross-backend use (below).

Concrete IDs outside the table.

  • 10 bundled agent definitions declare codex: model: gpt-6-luna directly (src/autoskillit/agents/*.md), with per-agent reasoning_effort values max (5), xhigh (4) and medium (1).
  • gpt-6-sol literals: src/autoskillit/execution/backends/_codex_managed_route.py:77, _probe_cache.py:49, _explorer_conformance.py:36.
  • gpt-6-luna literal: src/autoskillit/core/plugins/agent_definition.py:58.
  • src/autoskillit/execution/evidence/anomaly_detection.py:79-82 lists older Claude IDs. Review whether that is intentional.

Unknown or foreign IDs are passed through unchanged.

  • ClaudeBackend.translate_model returns CLAUDE_MODEL_ALIASES.get(base, base) (src/autoskillit/execution/backends/claude.py:273).
  • CodexBackend.translate_model returns CODEX_MODEL_ALIASES.get(base, base) (src/autoskillit/execution/backends/codex.py:382).
  • Nothing at config load, open_kitchen preflight or doctor checks that a pinned model belongs to the backend the step is routed to. A zero-token route dry run of a real user config with the candidate's own resolution functions shows:
Step Config model pin Backend (config pins) Sent to CLI Outcome
remediation.investigate claude-opus-5-5[1m] codex (packaged default; user's remediation: {} does not clear it because layers deep-merge) claude-opus-5-5 managed-join refused: unsupported managed Codex model: claude-opus-5-5
implementation.dry_walkthrough gpt-6.1-sol claude-code gpt-6.1-sol Claude Code is asked to run a GPT model

Bundled recipes are already compliant: they name only classes (no claude-* / gpt-* IDs in src/autoskillit/recipes/*.yaml).

Acceptance criteria

  • One authoritative definition per backend maps each class to its current latest model and effort. Every other production surface obtains models through it: agent definitions, probes, managed-route defaults, config defaults, docs.
    • Concrete IDs outside it fail an architectural guard, apart from an explicit, documented allowlist for intentional pins.
    • Tests reference the definition rather than restating IDs.
  • Codex mapping: opus → gpt-6.1-sol at xhigh; haiku/Luna at xhigh or max (owner's requirement). sonnet is chosen and documented in the same definition.
    • Agent definitions declare a class (plus an effort only where a deviation is intended), not gpt-6-luna.
  • The managed Codex allowed-model set derives from the same definition, or from the installed CLI's catalog, so the defined latest model is always admissible.
  • Decide and document how "latest" is maintained. Either (a) one place updated per generation, or (b) resolved at launch from the installed CLI's own catalog/aliases (Codex debug models --bundled, which managed issuance already reads; Claude Code's aliases), with the single definition as fallback.
  • Detect foreign model pins early. A class or ID that cannot be served by the backend a step is routed to is reported at config/preflight/doctor time with the remedy (use a class), instead of failing at launch.
    • Concrete IDs given for a backend they do not belong to are either translated to the equivalent class or rejected clearly.
  • Minimum versions only. Model-driven CLI floors stay minimums; no maximums are introduced.
  • Regression tests cover class resolution per backend, latest-model admissibility in managed Codex issuance, agent-definition class resolution, and the two foreign-pin cases above.

Related

Testing with Codex in Docker

CI cannot verify this fix. The E2E workflow routes every Codex step to Claude Code (scripts/e2e/autoskillit-config.yaml), and the unit gate never launches Codex. Verify it with a real Codex CLI in the repository's own image.

1. Build the user image for the fix commit with the Codex CLI that reproduces the bug

The Dockerfile pins CODEX_VERSION (0.156.1 at the time of writing). The defects reproduce on Codex CLI 0.164.0, so override the pin at build time. Do not edit the Dockerfile pin (see scripts/docker/README.md).

source_sha=$(git rev-parse HEAD)
git archive "$source_sha" | docker build --file scripts/docker/Dockerfile --target user \
  --build-arg CODEX_VERSION=0.164.0 --tag "autoskillit-user:$source_sha-codex0.164.0" -

scripts/docker/verify-image compares tool versions with the Dockerfile pins, so it rejects this override. Skip it for this image.

Also verify against the pinned version: build a second tag without --build-arg. A fix must work on both.

2. Start a throwaway container

docker run -d --name codex-fix-check --init \
  --security-opt seccomp=unconfined --security-opt apparmor=unconfined \
  --tmpfs /dev/shm:rw,exec,nosuid,nodev,size=4g,mode=1777 \
  "autoskillit-user:$source_sha-codex0.164.0" sleep infinity
  • seccomp/apparmor must be unconfined for Codex's Linux sandbox (bubblewrap) to run inside Docker.
  • The exec /dev/shm tmpfs is required for AutoSkillit's process lifecycle.
  • The image user is autoskillit with HOME=/home/autoskillit. AutoSkillit's tool interpreter is ~/.local/share/uv/tools/autoskillit/bin/python.

3. Supply your own credentials at run time (never bake them into an image)

docker exec codex-fix-check mkdir -p /home/autoskillit/.codex /home/autoskillit/.claude
docker cp ~/.codex/auth.json codex-fix-check:/home/autoskillit/.codex/auth.json
docker cp ~/.claude/.credentials.json codex-fix-check:/home/autoskillit/.claude/.credentials.json   # only for recipe runs
docker exec -u root codex-fix-check sh -c 'chown -R autoskillit: /home/autoskillit/.codex /home/autoskillit/.claude && chmod 600 /home/autoskillit/.codex/auth.json /home/autoskillit/.claude/.credentials.json'
  • Copy credentials that were refreshed recently. The container uses the same refresh token as the host, and a refresh on either side can rotate it.
  • Remove the container when you finish (docker rm -f codex-fix-check). Its home now holds credentials.

4. Work in a local-only repository

Recipe runs can push branches and open PRs or issues. Give them a repository with no remote, so nothing can land on GitHub:

docker exec codex-fix-check bash -c '
  set -e; mkdir -p /home/autoskillit/probe/sandbox /home/autoskillit/probe/tests /home/autoskillit/probe/.autoskillit
  cd /home/autoskillit/probe && git init -q -b main
  printf "def word_count(text: str) -> int:\n    return len(text.split())\n" > sandbox/text.py
  : > sandbox/__init__.py; : > tests/__init__.py
  printf "branching:\n  default_base_branch: main\ntest_check:\n  command: [\"python3\", \"-m\", \"unittest\"]\n" > .autoskillit/config.yaml
  git add -A && git -c user.name=probe -c user.email=probe@localhost commit -qm init'

For autoskillit fleet run, enable the gated features in the container's user config (/home/autoskillit/.autoskillit/config.yaml):

features:
  experimental_enabled: true

5. Safeguards

  • Order. Run the zero-token probes first. Run a live session only after they pass.
  • Cost.
    • Live probes are single codex exec turns: about 10k–100k input tokens, mostly cached.
    • A fleet run planner also runs a Claude Code orchestrator: about 1M cached-read tokens.
  • Time limits. Use timeout on every live command and --timeout-sec on fleet run.
  • Evidence. Session rollouts are under ~/.local/share/autoskillit/logs/codex-sessions/ and ~/.codex/sessions/. session_meta/turn_context record the model, effort and sandbox policy; response_item records tool calls and outputs. Attach the relevant excerpts to the PR.

Verification steps for this issue

Sequence this after #5344, which changes the same Codex catalog and admission code.

A. Zero-token: what the installed Codex CLI offers.

docker exec codex-fix-check bash -c 'codex debug models --bundled | ~/.local/share/uv/tools/autoskillit/bin/python -c "
import json, sys
for m in json.load(sys.stdin)[\"models\"]:
    print(m.get(\"slug\"), m.get(\"visibility\"), [e[\"effort\"] for e in m.get(\"supported_reasoning_levels\", [])])"'

The model this issue defines as latest for each class must appear here, with its effort.

B. Zero-token: the effective model and effort for every Codex-routed recipe step. No model turn runs here. The script resolves each step the way run_skill does, then asks managed-join issuance for its attested model and effort:

docker exec -w /home/autoskillit/probe codex-fix-check bash -c 'cat > /tmp/route_probe.py <<EOF
import json, sys, uuid, yaml
from importlib import resources
from pathlib import Path
from autoskillit.config import load_config
from autoskillit.execution.backends.codex import CodexBackend
from autoskillit.execution.headless._headless_model import resolve_model_pin
from autoskillit.server.lifecycle._guards import _resolve_backend_override
from autoskillit.server.managed_join_prelaunch import prepare_managed_join_context

project = Path(sys.argv[1]); cfg = load_config(project)
for path in sorted((Path(str(resources.files("autoskillit"))) / "recipes").glob("*.yaml")):
    recipe = yaml.safe_load(path.read_text()) or {}; name = recipe.get("name") or path.stem
    for step_name, step in (recipe.get("steps") or {}).items():
        if not isinstance(step, dict) or step.get("tool") != "run_skill":
            continue
        pin = _resolve_backend_override(step_name, name, cfg.agent_backend)
        if (pin.backend if pin else cfg.agent_backend.backend) != "codex":
            continue
        raw = str(step.get("model") or ""); model = resolve_model_pin("" if "\${{" in raw else raw, cfg, step_name=step_name, recipe_name=name).model
        out = prepare_managed_join_context(backend=CodexBackend(), configured_model=model, state_root=project,
                                           parent_id=f"probe-{uuid.uuid4().hex[:12]}", launch_context="direct")
        att = getattr(out, "managed_join_attestation", None)
        print(f"{name}.{step_name}: configured={model!r} ->",
              (att.resolved_model, att.resolved_reasoning_effort) if att else repr(out)[:160])
EOF
~/.local/share/uv/tools/autoskillit/bin/python /tmp/route_probe.py /home/autoskillit/probe'

Run it twice:

  1. With the packaged defaults.
  2. With a user config naming only classes, e.g. model: {recipe_overrides: {remediation: {investigate: "opus[1m]"}}}.

Expected after the fix:

  • Every Codex-routed step resolves to the single definition's latest model and effort for its class. The owner's mapping: opus → gpt-6.1-sol at xhigh; haiku (Luna) at xhigh or max.
  • No step is refused.
  • A concrete Claude ID pinned on a Codex-routed step is reported clearly, not passed to Codex.

Use the state root under the project as above, not /tmp. Codex warns on stderr for a /tmp home, and the catalog probe treats any stderr as a failure.

C. Live check. After #5344 and #5345, run one headless Codex step (for example the fleet run planner from those issues). Confirm its rollout's turn_context records the expected model and effort.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestrecipe:remediationRoute: investigate/decompose before implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions