Skip to content

[Bug]: management-triggered catalog convergence reports reason:"disk" with no disk failure and discards the original error #1784

Closed

Description

@brunoflma

Client or integration

Direct HTTP/API client

Area

Catalog / models

Summary

Catalog convergence triggered from the management API fails deterministically and reports reason: "disk" when there is no disk problem. The underlying exception is discarded, so the real cause cannot be recovered from outside — and, as far as I can tell, from inside either.

PUT /api/subagent-models returns 200 ok with:

{"status":"failed","reason":"disk","phase":"gather","retryable":false,"partialWrite":false}

There is no disk failure. The Codex catalog is 871 KB, parses, holds 37 entries, and the volume has 47 GB free. partialWrite:false says nothing was written, and ocx ensure writes the same catalog successfully seconds before and after.

Three things compound here, and I think the third is the actual bug:

  1. reason: "disk" is a hard-coded label, not an observation. unexpectedCatalogFailure() in src/codex/management-convergence.ts returns reason: "disk" unconditionally, and the wrapper in src/server/management-api.ts does the same in its own catch. Nothing in either path checked a disk.

  2. retryable: false is asserted, and is wrong at least sometimes. The identical code path, driven out-of-process against the same config, returns {"status":"committed","changed":true,"degraded":true,"notices":["fallback","provider-auth"]}. An operation that succeeds when invoked differently is not non-retryable.

  3. Three unrelated causes collapse into one indistinguishable disposition, and the original error is destroyed. All of these produce byte-identical {failed, disk, gather, false}:

    • the request-shape guard in createManagementConvergeCodex (request.scope !== "catalog"),
    • the admissionFailure() fallthrough, when a thrown message matches neither "config generation is busy" nor "config generation is database",
    • the catch in management-api.ts before convergenceInvoked is set.

    None of the three logs anything. service.log contains zero lines about these failures. admissionFailure() classifying by substring match on error.message means any new or reworded throw upstream of it silently becomes a non-retryable disk failure.

I could not determine which of the three fires here, or what the underlying exception is, despite reading the source and probing the path directly. That is the point of the report: with full source access and a deliberate effort, the diagnostic is unrecoverable. An operator sees "disk failure", checks a healthy disk, and has nowhere to go.

Suggested direction: preserve the original error (log it, or carry a cause/detail field on the disposition), and distinguish the three paths — a request-shape mismatch is a programming error, an admission refusal is contention, and a write failure is disk. Only the last one is "disk".

Reproduction

Verified on 2.20.0, deterministic, 3/3.

  1. Start the proxy normally: ocx restart && ocx ensure — note that ensure reports the catalog written successfully.
  2. Read the current subagent roster and write the same list back, so the request is a no-op in content:
node -e "const fs=require('fs'),h=require('http');const T=fs.readFileSync(process.env.USERPROFILE+'/.opencodex/admin-api-token','utf8').trim();const o={host:'127.0.0.1',port:10100,headers:{authorization:'Bearer '+T}};h.get({...o,path:'/api/subagent-models'},r=>{let d='';r.on('data',c=>d+=c);r.on('end',()=>{const b=JSON.stringify({models:JSON.parse(d).chosen});const q=h.request({...o,path:'/api/subagent-models',method:'PUT',headers:{...o.headers,'content-type':'application/json','content-length':Buffer.byteLength(b)}},x=>{let e='';x.on('data',c=>e+=c);x.on('end',()=>console.log(x.statusCode,e))});q.write(b);q.end()})})"
  1. Observe catalogRefresh: {"status":"failed","reason":"disk","phase":"gather","retryable":false,"partialWrite":false}. Repeating the call reproduces it every time.

The control that makes this a defect rather than an environment problem — the same path, out-of-process, succeeds:

const { loadConfig } = await import("<pkg>/src/config.ts");
const { createCatalogConvergeRequest } = await import("<pkg>/src/codex/catalog-admission.ts");
const { createManagementConvergeCodex } = await import("<pkg>/src/codex/management-convergence.ts");
const config = loadConfig();
const outcome = await createManagementConvergeCodex(config)(createCatalogConvergeRequest({ deadlineMs: 1_000 }));
console.log(outcome.kind, JSON.stringify(outcome.catalogRefresh));

Output:

catalog-only {"status":"committed","changed":true,"degraded":true,"notices":["fallback","provider-auth"]}

Same machine, same config, same second. In-process it is failed/disk/gather 3/3; out-of-process it commits. Whatever differs is process state — which is exactly what the discarded exception would have named.

Expected: the response identifies the real cause, or at minimum the original error reaches the log so it can be identified.

Version

2.20.0

Operating system

Windows 11 Home Single Language 26200

Provider and model

Not provider-specific — the failure is in catalog convergence, before any provider is contacted.

Logs or error output

# three consecutive PUTs with identical bodies
tentativa 1: HTTP 200  catalogRefresh={"status":"failed","reason":"disk","phase":"gather","retryable":false,"partialWrite":false}
tentativa 2: HTTP 200  catalogRefresh={"status":"failed","reason":"disk","phase":"gather","retryable":false,"partialWrite":false}
tentativa 3: HTTP 200  catalogRefresh={"status":"failed","reason":"disk","phase":"gather","retryable":false,"partialWrite":false}

# the disk is fine
catalog: C:\Users\<user>\.codex\opencodex-catalog.json  871413 bytes, parses, 37 entries
volume: 465G total, 47G available

# service.log contains no line about any of these failures
$ grep -c "converge\|catalog.*fail" service.log
0

Redacted configuration

{
  "subagentModels": [
    "PROVIDER-A/model-a",
    "PROVIDER-B/model-b"
  ]
}

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. coderabbitai commented on Aug 15, 2026

    @coderabbitai
    Contributor
    🔗 Related PRs

    #354 - fix(models): surface discovery failure status when /v1/models returns 401 (#329) [open]
    #391 - feat: quota-aware subagent model fallback chain (#374) [merged]
    #1212 - feat(codex): converge account picker catalogs [merged]
    #1331 - fix(catalog): keep failover combos when live discovery omits members [merged]
    #1460 - fix(catalog): skip byte-identical catalog writes so a resync cannot mark Codex stale [open]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  2. added
    bugSomething isn't working
    catalogModel catalog, slugs, visibility, routed entries
    proxyHTTP proxy, routing, reverse-proxy / management auth
    on Aug 15, 2026
  3. added a commit that references this issue on Aug 16, 2026
    89cf04f
  4. lidge-jun commented on Aug 16, 2026

    @lidge-jun
    Owner

    Fixed on dev by #1848 (merged as 73440d4bb).

    reason: "disk" was manufactured for any unrecognized failure in two places — the management adapter's catch and the management API route — so a malformed request and a genuinely full filesystem were indistinguishable, and both were reported non-retryable.

    CatalogDisposition gains request-invalid, admission and internal alongside the existing reasons, narrowing disk to real IO. admission is the one class marked retryable, since contention is the failure worth retrying unchanged.

    The new cause field is deliberately not an error summary. Both fields are closed vocabularies: kind maps onto a fixed set rather than echoing Error.constructor.name (any thrown custom class names itself), and code is emitted only for a recognized errno token. An Error.message routinely carries paths, home directories and account identifiers, and redactSecretString masks token shapes but none of those.

    normalizeCatalogDisposition — the allowlist that rebuilds the management response — was updated in the same change. Without that the new reasons and cause would have been silently dropped and the fix would have looked implemented while changing nothing.

    Regressions: an escaping factory error is internal, a TypeError is request-invalid, and an error whose message embeds both a home path and a token-shaped string produces a response containing neither.

    Closing as completed. The change is on dev and ships with the next release.

  5. added a commit that references this issue on Sep 17, 2026
    47f917d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcatalogModel catalog, slugs, visibility, routed entriesproxyHTTP proxy, routing, reverse-proxy / management auth

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions