Skip to content

[Tracking] Codex skill admission collapse: 62 of 82 bundled skills silently refused since #4613 #4711

Closed

Description

@Trecek

Tracking status — 2026-09-10 (UTC): #4712–#4716 are closed. #4717, #4718, and #4719 remain open; this parent remains open.

The original incident narrative, measurements, test-gap descriptions, child sequencing, and stopgap observations below are an August 20 historical record, not verified current runtime state. The Current tracking update appended at the end governs remaining work and supersedes stale status/sequence statements.

What happened

Commit 55b697427 (2026-08-17, PR #4613, "Rectify: Fixed-set join contract and Claude
team-dispatch isolation", closing #4520 and #4575) changed CodexBackend.adapt_skill_semantics()
from rendering a fixed-set join contract as prose into hard-raising
SkillSemanticOperation.REQUIRED_JOIN for any skill declaring
semantic_requirements.join.required: true.

Roughly 61% of the bundled catalog declares that flag. The Codex interactive catalog collapsed
from 94 staged skills to 35. Every test in the repository stayed green.

Reproduced live in-process on develop @ 31949100d:

Configuration Backend Admitted Refused make-plan
repo root (catalog 95) claude-code 95 0 ✅
repo root (catalog 95) codex 37 58 ❌
bundled-only (catalog 82) claude-code 82 0 ✅
bundled-only (catalog 82) codex 20 62 ❌

Every refusal shares one reason: REQUIRED_JOIN.

The repo-root figure is inflated by 17 project-local skills that bypass the gate by declaring no
join: block. That 17 decomposes, and the split matters:

promote-to-main and validate-audit also shadow bundled skills but correctly declare
join.required: true, so they are refused like their bundled counterparts — healthy, leave alone.
make-arch-diag shadows a bundled skill that has no join requirement, so its override changes
nothing either way.

A clean consumer sees 20 of 82.

Corroborated by two orphaned Codex session homes straddling the change:
.autoskillit/temp/codex-sessions/6e12aeaf5d314556/add-dir/skills (2026-08-12) holds 94 skills
including make-plan and investigate; c79cb765fe7c46e7 (2026-08-19) holds 35 with neither.
Its skill-unavailability.json lists exactly 58 required_join entries.

Why it was invisible

  1. Nothing measures the admitted set. No test combines the real catalog, a real backend, and a
    set assertion. The closest test asserts refusal shape and passes identically whether 1 or 58
    skills are refused.
  2. The refusal record has no production reader. write_skill_unavailability_metadata() writes
    all 58 entries into the add-dir; the only reader anywhere is a write-correctness test. The one
    surfacing path (_session_launch.py:509-519) is reachable only from order, never cook.
  3. The end-to-end cook test neutralizes the gate under test with an unconditionally permissive
    fake (tests/fakes.py:62-84). A fake that can never refuse cannot detect an over-broad refusal.

The class of bug

fixed_set_join_capable is one of 54 BackendCapabilities fields. Exactly one (min_version)
is checked against live truth. The rest are hand-maintained literals encoding beliefs about a
backend that nothing verifies. This family installs measurement and machine-checked pairing so a
stale capability literal becomes a test failure rather than a 58-skill silent amputation.

Children

Ticket Scope Depends on
#4712 Golden admission ledger + faithful test surface —
#4713 Refusal-as-value seam + uniform surfacing #4712
#4714 Close ungated corridors + override-weakening guard #4713
#4715 Single-source agent provisioning #4714 (file conflict)
#4716 Restore Codex fixed-set join capability #4712, #4713, #4714 (gate cleared)
#4717 Migrate off deprecated $CODEX_HOME/skills discovery root #4714
#4718 Interactive-session gating family —
#4719 Remove the stopgap overrides (cleanup — must close last) #4716

#4712 and #4718 start immediately and in parallel. (#4717 was originally listed here; adversarial review found it shares the Codex skill-staging root with #4714 and must follow it.) #4713→#4714→#4715 is a chain (C and D both edit
_materialize_session). #4716 lands last. Its gate is now cleared — all three Codex blocking-hook facts were independently confirmed at pinned revision 646f7c0a91b8e327d263335da68ae8ef212895ce and in the codex-cli 0.147.0 release tree; the original refutation was a wrong-file mixup (hooks/src/registry.rs vs core/src/tools/registry.rs). See #4716 for corrected citations.

⚠️ Stopgap in effect on at least one machine — tracked by #4719

Rather than wait for the remediation chain, six skills were unblocked on Codex by planting
local-only, untracked project overrides that omit semantic_requirements, so semantic_plan
resolves to None and the skill bypasses the gate entirely
(workspace/session_skills.py:271-273) — the same mechanism that already makes 9 stale tracked
overrides work in this repo.

make-plan, investigate, dry-walkthrough, resolve-review, review-pr,
implement-worktree-no-merge. Verified effect: this checkout's Codex SESSION catalog moved from
37 admitted / 58 refused to 43 / 52.

They live under .git/info/exclude, so git status stays clean and no teammate inherits them —
which also means nothing in CI will ever surface them. That is why #4719 exists.

While in place, the fixed-set join contract is unenforced for those six on Codex. Nothing stops
the model ending its turn before children settle. That is the #4520 failure mode. Fine for
supervised interactive work; treat any unattended Codex pipeline touching those six as
untrustworthy until #4719 closes.

This does not affect #4712's ledger — that is computed with project_root=None, so
project-local overrides never enter it. The bundled-only truth remains 20 of 82.

Related existing issues

Source material

  • Investigation: .autoskillit/temp/investigate/investigation_codex_skill_admission_regression_2026-08-20_092549.md
  • Rectify plan, 5 parts (dry-walkthrough verified):
    .autoskillit/temp/rectify/rectify_codex_skill_admission_2026-08-20_105733_part_{a..e}.md

Every child ticket carries its plan part inline. All line references are as-of develop
@ 31949100d / 989497a6c and must be re-verified through the planning pipeline before
implementation — sibling tickets in this family will move them, and cli/ has already been
resequenced into cli/install/, cli/ops/, cli/prompts/ since the investigation began.


Current tracking update — 2026-09-10

Verified child status

States were fetched together from GitHub on 2026-09-10. A closed issue is a tracking fact; it is not by itself proof that a particular local installation contains the change or passes the corresponding runtime checks.

Ticket Scope Current issue state Dependency / remaining tracking
#4712 Golden admission ledger and faithful test surface Closed Original dependency: none
#4713 Refusal-as-value seam and uniform surfacing Closed Original dependency: #4712
#4714 Ungated corridors and override-weakening guard Closed Landed through #4766; original dependency: #4713
#4715 Single-source agent provisioning Closed Original dependency: #4714
#4716 Restore Codex fixed-set join capability Closed Original dependencies: #4712, #4713, #4714; installation/acceptance verification still matters for #4719
#4717 Supported Codex skill discovery and managed-root reconciliation Open #4714 sequencing prerequisite is satisfied; next step is make-plan using the updated child evidence
#4718 Interactive-session gating family Open Remains a separate open workstream under its own issue scope
#4719 Remove stopgap local skill overrides Open #4716 is closed, but the child requires the capability to have landed and the affected skills to be legitimately admitted before cleanup is considered complete

Dependency evidence for #4717: #4714 landed through PR #4766 on 2026-08-24, commit 45fbd18ac9ce9c33c441d32c1a6ea4d10ac62613. It is an ancestor of the investigated AutoSkillit revision 544deb01cb729046f7e305f36867381aca0e9fb4 (develop, version 0.10.1102). The old instructions to begin the entire #4712→#4713→#4714→#4715 chain or await #4716 as an open issue are historical.

#4717 investigation: confirmed risk and corrected scope

The updated body of #4717 contains the complete source-pinned evidence, affected consumers, test gaps, unresolved questions, and planning acceptance criteria. Its investigation update is dated 2026-09-09 in America/Los_Angeles.

  • The deprecation is confirmed. Codex 0.130.0, AutoSkillit's declared minimum at the investigated revision, and 0.147.0 both scan $CODEX_HOME/skills while explicitly marking that user-skill root deprecated. This is a latent compatibility risk; no removal date or present upstream removal was established.
  • The generated-home skills root is active. Interactive cook/prepared sessions set CODEX_HOME=<session> and discover <session>/skills. Managed headless skill sessions set CODEX_HOME=<session>/add-dir and discover <session>/add-dir/skills. --add-dir itself grants filesystem access and does not register a skill root.
  • The old ungated profile copier has already been removed by Close ungated admission corridors (profile skills, orchestrator guidance) + override-weakening guard #4714/Close ungated admission corridors #4766. Current workspace materialization creates the admitted profile/project union with profile-name precedence in the generated-home view. [Tracking] Codex skill discovery migration — Parts A–D #4717 must reconcile the remaining managed views and launch wiring while preserving those admission and ordering guarantees.
  • The replacement interface is the first planning decision. The verified arbitrary-root API, skills/extraRoots/set, is process-scoped app-server RPC in 0.147.0 and is absent in 0.130.0. No equivalent arbitrary-root setting was verified for the examined CLI launch paths. Raising the minimum version alone does not give codex exec that API. Both launch modes, prepared launch/resume, state ownership, and concurrent-session isolation must be covered; any required transport/version expansion must be explicit in the plan.
  • The missing regression is real-child discovery. Existing tests check staging layouts, CODEX_HOME, and command construction but do not establish which staged skills an actual Codex child discovers. The migration needs observed catalog names/paths and coverage of intended admission, profile precedence, refusal exclusion, and isolation.

This is deliberately deferred discovery work following #4714, not evidence that a previous migration off the deprecated root failed. The recommended next workflow for #4717 is make-plan. Another rectify pass is not warranted by the recurrence evidence gathered for this issue.

Remaining work and closure discipline

  1. [Tracking] Codex skill discovery migration — Parts A–D #4717: Resolve the supported catalog-delivery interface during planning, then plan managed-root reconciliation, affected launch wiring, and a real-child discovery regression. Preserve admission policy; do not assume a folder rename supplies discovery or that app-server adoption has already been selected.
  2. Interactive-session gating: enable_exploration bind_failed has no fallback (+ #4708/#4683/#4684) #4718: Continue the separate interactive-session gating work under its own acceptance criteria. The [Tracking] Codex skill discovery migration — Parts A–D #4717 discovery investigation did not establish that those failures are resolved.
  3. Remove stopgap local skill overrides that bypass the Codex join gate #4719: Keep the local override cleanup and verification explicitly tracked, retaining this parent's cleanup-last intent. Its own acceptance condition requires Restore Codex fixed-set join capability (gate cleared — hooks confirmed at 0.147.0) #4716's capability to have landed and the six affected skills to be legitimately admitted. Reinspect the actual overrides and .git/info/exclude on the target installation before recording cleanup. Restore Codex fixed-set join capability (gate cleared — hooks confirmed at 0.147.0) #4716's closed state alone does not establish those conditions.
  4. This parent: Remain open while [Tracking] Codex skill discovery migration — Parts A–D #4717, Interactive-session gating: enable_exploration bind_failed has no fallback (+ #4708/#4683/#4684) #4718, or Remove stopgap local skill overrides that bypass the Codex join gate #4719 is unresolved. Do not infer family completion from closure of the earlier admission/provisioning children.

Historical measurements and observations

The original 20/82, 37/58, and 43/52 figures, the staged-catalog counts, the descriptions of missing refusal surfacing/tests, and the six local stopgap overrides are retained as evidence of the original incident and its August 20 investigation. They were not remeasured or revalidated as current facts by the #4717 discovery investigation. The issue title likewise identifies the original incident.

Before publishing current admission totals or claiming that the stopgap overrides remain present or have been removed, rerun the relevant measurement/inspection against a named revision and installation. Closure of #4712–#4716 does not substitute for that evidence. The historical c79cb765fe7c46e7 directory is now absent, and the source of its reported profile view remains unverified; do not infer it from today's default profile location.

Local source report: .autoskillit/temp/investigate/investigation_issue_4717_codex_skill_discovery_2026-09-09_232547.md. The relevant findings are reproduced in #4717 so downstream planning does not depend on this local artifact being available.


Closure — 2026-11-04

Every condition this parent set for staying open has been resolved. Child states were fetched together from GitHub on 2026-11-04.

Ticket State Closed
#4712 Closed (completed) 2026-08-21
#4713 Closed (completed) 2026-08-23
#4714 Closed (completed) 2026-08-24
#4715 Closed (completed) 2026-08-26
#4716 Closed (completed), via PR #4913 2026-09-10
#4717 Closed 2026-09-16
#4718 Closed 2026-09-20
#4719 Closed 2026-09-22

Related follow-ups filed after this parent was opened are also closed: #4755 (2026-09-22), #5140 via PR #5147 (2026-09-23), and #5022 (2026-09-27).

Fix provenance. PR #4913 (775f224f5, restore required join on stock Codex) and PR #5147 (7fb5a0ce4, cook join admission and Codex catalog preparation) are both ancestors of develop @ f84bae324 (0.10.1306). Neither is in 0.10.1013 (5c4924749, 2026-08-23). Installations on releases older than these fixes still show the original required_join refusals. Upgrading resolves this; it does not require new work under this parent.

Current test evidence. The golden admission ledger (tests/contracts/_skill_admission_ledger.py) pins every shipped skill as admitted for codex in each pinned list_effective call shape, including the cook session shape. This uses managed-join launch evidence. The unfiltered test run on f84bae324 that is recorded in PR #5335 passed.

Not verified live. No interactive Codex cook session has been observed on 0.10.1306; PR #5335 ran with the Codex backend feature disabled. Managed-join issuance depends on the source home's models_cache.json, which was the #5140 failure. After upgrading, a Codex cook session's skill-unavailability record should contain no required_join entries. A project-local eval-agent, which declares a runtime_bound role, is still expected to be refused as child_spawn. That refusal is intended and is outside this family's scope.

Activity

  1. added
    bugExisting behavior is broken
    do-not-collapseIssue should not be collapsed into combined issues
    on Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugExisting behavior is brokendo-not-collapseIssue should not be collapsed into combined issues

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions