Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -795,6 +795,14 @@ Use `ocx provider add mine --adapter openai-chat --base-url https://example.com/

### Cached quota history

Automatic OpenAI account exhaustion checks distinguish purchased usage credits from reset
tickets. A fresh positive spendable balance or unlimited credits can keep an account eligible
after its included usage reaches 100%; an explicit upstream refusal or overage limit still
blocks that evidence. Credit evidence expires after five minutes, and usage headers do not
renew its clock. `pause-exhausted` uses this rule. Reset tickets alone never grant automatic
headroom, and a credits-only response cannot clear a request cooldown. The opt-in main-account
hard lock remains a separate local policy.

`ocx account history openai <pool-account-id> [--limit 1-200] [--json]` reads stored observations without contacting the provider. The output separates actual observation time, WHAM or response-header source, window family and usage percentage. At most 200 observations per account are retained for 30 days, with global storage bounds.

Ordinary token refresh preserves history. Reauthentication, removal or account replacement retires the old publication. Native main and probes performed before a login is published are not included. Missing history means insufficient observations, not zero usage. This command does not spend quota. Effective estimates, when supported by observations, carry the limitations below.
Expand Down
34 changes: 33 additions & 1 deletion src/codex/quota-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,31 @@ export const TERMINAL_SHORT_WINDOW_FRESHNESS_MS = 5 * 60_000;
*/
export const CODEX_EXHAUSTED_USAGE_PERCENT = 100;

/** Credits have a separate clock: ordinary usage headers do not re-observe the balance. */
export const CODEX_CREDITS_FRESHNESS_MS = 5 * 60_000;

export interface CodexCredits {
hasCredits?: boolean;
unlimited?: boolean;
balance?: number;
overageLimitReached?: boolean;
allowed?: boolean;
observedAt: number;
}

export function hasSpendableCodexCredits(
quota: Pick<StoredAccountQuota, "credits"> | null,
now = Date.now(),
): boolean {
const credits = quota?.credits;
if (!credits || !Number.isFinite(credits.observedAt)) return false;
const age = now - credits.observedAt;
if (age < 0 || age > CODEX_CREDITS_FRESHNESS_MS
|| credits.allowed === false || credits.overageLimitReached === true || credits.hasCredits === false) return false;
return credits.unlimited === true || (credits.hasCredits === true
&& typeof credits.balance === "number" && Number.isFinite(credits.balance) && credits.balance > 0);
}

/**
* Above this a value is already milliseconds; at or below it, it is Unix seconds.
*
Expand Down Expand Up @@ -80,6 +105,8 @@ export type StoredAccountQuota = {
shortWindowSeconds?: number;
customWindows?: Array<{ label: string; percent: number; resetAt?: number }>;
resetCredits?: number;
/** Spendable usage credits; resetCredits are separate, manually redeemed reset tickets. */
credits?: CodexCredits | null;
/** Monthly usage came from an explicitly monthly PRIMARY, not supplementary tertiary, window. */
monthlyIsPrimaryWindow?: boolean;
updatedAt: number;
Expand Down Expand Up @@ -114,7 +141,12 @@ export type WhamUsageResponse = {
secondary_window?: WhamUsageWindow | null;
tertiary_window?: WhamUsageWindow | null;
};
credits?: unknown;
credits?: {
has_credits?: unknown;
unlimited?: unknown;
balance?: unknown;
overage_limit_reached?: unknown;
} | null;
rate_limit_reset_credits?: { available_count: number } | null;
additional_rate_limits?: WhamAdditionalRateLimit[] | null;
};
Expand Down
67 changes: 53 additions & 14 deletions src/codex/quota.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import { getObservedMainQuotaIdentityKey, isMainQuotaWriterLive, type MainQuotaW

import { CodexQuotaHistory, QUOTA_HISTORY_LIMITS, type QuotaHistoryWindow } from "./quota-history";
import { isPoolQuotaWriterLive, poolQuotaHistoryIdentity } from "./account-store";
import { CODEX_EXHAUSTED_USAGE_PERCENT, MAIN_ACCOUNT_HARD_LOCK_MIN_PERCENT, resetAtToMs } from "./quota-types";
import { CODEX_EXHAUSTED_USAGE_PERCENT, MAIN_ACCOUNT_HARD_LOCK_MIN_PERCENT, hasSpendableCodexCredits, resetAtToMs } from "./quota-types";
import type { PoolQuotaWriter, StoredAccountQuota, WhamUsageResponse, WhamUsageWindow } from "./quota-types";

export type { StoredAccountQuota, WhamUsageResponse } from "./quota-types";
Expand Down Expand Up @@ -83,10 +83,11 @@ function mayCommitAccountQuota(accountId: string, writerGeneration: number): boo
// actually exhausted account is still eligible for threshold rotation.
export const CODEX_UNKNOWN_USAGE_SCORE = 101;
export function isCodexQuotaExhausted(
quota: Pick<StoredAccountQuota, "weeklyPercent" | "monthlyPercent" | "shortPercent"> | null,
quota: Pick<StoredAccountQuota, "weeklyPercent" | "monthlyPercent" | "shortPercent" | "credits"> | null,
plan?: unknown,
): boolean {
if (!quota) return false;
if (hasSpendableCodexCredits(quota)) return false;
// The burst window counts on EVERY plan. It is upstream-enforced independently, so an
// account at 100% there is blocked regardless of which longer window governs its plan;
// omitting it would route traffic straight into a 429 (#1791).
Expand Down Expand Up @@ -118,7 +119,7 @@ export function codexQuotaWindowForPlan(plan?: unknown): "monthly" | "weekly" {
}

export function isCompleteCodexQuotaRecoverySnapshot(
quota: Pick<StoredAccountQuota, "weeklyPercent" | "monthlyPercent" | "monthlyIsPrimaryWindow" | "shortPercent"> | null,
quota: Pick<StoredAccountQuota, "weeklyPercent" | "monthlyPercent" | "monthlyIsPrimaryWindow" | "shortPercent" | "credits"> | null,
plan?: unknown,
): boolean {
if (!quota || isCodexQuotaExhausted(quota, plan)) return false;
Expand Down Expand Up @@ -277,6 +278,10 @@ function snapshotHasCustom(quota: Omit<StoredAccountQuota, "updatedAt">): boolea
function snapshotHasUsage(quota: Omit<StoredAccountQuota, "updatedAt">): boolean {
return snapshotHasWeekly(quota) || snapshotHasMonthly(quota) || snapshotHasShort(quota) || snapshotHasCustom(quota);
}

function snapshotHasCredits(quota: Omit<StoredAccountQuota, "updatedAt">): boolean {
return quota.resetCredits !== undefined || quota.credits !== undefined;
}
/**
* Publish parsed display quota and separately validated policy evidence after writer checks.
* A null policy observation retains only the matching main identity's previous evidence;
Expand All @@ -296,7 +301,7 @@ export function setAccountQuotaFromParsed(
): void {
quota = withoutRetiredCodexQuota(quota);
policyQuota = withoutRetiredCodexQuota(policyQuota);
if (!quota || (!snapshotHasUsage(quota) && quota.resetCredits === undefined)) return;
if (!quota || (!snapshotHasUsage(quota) && !snapshotHasCredits(quota))) return;
if (!mayCommitAccountQuota(accountId, writerGeneration)) return;
const isMain = accountId === MAIN_CODEX_ACCOUNT_ID;
if (isMain && mainWriter && !isMainQuotaWriterLive(mainWriter)) return;
Expand Down Expand Up @@ -343,7 +348,7 @@ export function setAccountQuotaFromParsed(
}
schedulePersistAccountQuotas();
// Credits carry the previous usage tuple; they must not refresh its observation clock.
if (!(quota.resetCredits !== undefined && !snapshotHasUsage(quota))) {
if (!(snapshotHasCredits(quota) && !snapshotHasUsage(quota))) {
if (!isMain && policyQuota && livePoolEvidence) observeCodexLowQuota(accountId, policyQuota);
notifyCodexQuotaSnapshot(accountId, next);
}
Expand All @@ -363,7 +368,11 @@ function mergeAccountQuota(
quota = withoutRetiredCodexQuota(quota) ?? {};
existing = withoutRetiredCodexQuota(existing ?? null) ?? undefined;
const next: StoredAccountQuota = { updatedAt };
const creditsOnly = quota.resetCredits !== undefined && !snapshotHasUsage(quota);
const creditsOnly = snapshotHasCredits(quota) && !snapshotHasUsage(quota);
// An explicit empty/invalid credits summary retracts the old balance. Partial headers
// retain the original observation clock instead of extending its spending authority.
if (quota.credits !== undefined) next.credits = quota.credits;
else if (existing?.credits !== undefined) next.credits = existing.credits;

if (creditsOnly) {
if (existing?.weeklyPercent !== undefined) next.weeklyPercent = existing.weeklyPercent;
Expand All @@ -373,7 +382,8 @@ function mergeAccountQuota(
if (existing?.monthlyIsPrimaryWindow === true) next.monthlyIsPrimaryWindow = true;
assignCarriedShort(next, existing, updatedAt, policyEvidence);
if (existing?.customWindows !== undefined) next.customWindows = existing.customWindows;
next.resetCredits = quota.resetCredits;
if (quota.resetCredits !== undefined) next.resetCredits = quota.resetCredits;
else if (existing?.resetCredits !== undefined) next.resetCredits = existing.resetCredits;
return stampCodexQuotaUsageObservation(next, quota, existing);
}

Expand Down Expand Up @@ -525,7 +535,7 @@ export function withoutRetiredCodexQuota<T extends Omit<StoredAccountQuota, "upd
const next = { ...quota };
if (kept.length > 0) next.customWindows = kept;
else delete next.customWindows;
return snapshotHasUsage(next) || next.resetCredits !== undefined ? next as T : null;
return snapshotHasUsage(next) || snapshotHasCredits(next) ? next as T : null;
}

export function parseUpstreamQuotaHeaders(headers: Headers, options?: { modelId?: string }): Omit<StoredAccountQuota, "updatedAt"> | null {
Expand Down Expand Up @@ -644,6 +654,7 @@ export function updateAccountQuota(
...(existing?.monthlyResetAt !== undefined ? { monthlyResetAt: existing.monthlyResetAt } : {}),
...(existing?.customWindows !== undefined ? { customWindows: existing.customWindows } : {}),
...(existing?.resetCredits !== undefined ? { resetCredits: existing.resetCredits } : {}),
...(existing?.credits !== undefined ? { credits: existing.credits } : {}),
updatedAt: Date.now(),
};
assignCarriedShort(quota, existing, quota.updatedAt);
Expand Down Expand Up @@ -702,6 +713,20 @@ function readMainPolicyQuota(value: unknown): MainPolicyQuota | null {
if (typeof number === "number" && Number.isFinite(number) && number >= 0) quota[field] = number;
}
if (quota.monthlyPercent !== undefined && raw.monthlyIsPrimaryWindow === true) quota.monthlyIsPrimaryWindow = true;
// Retain the independent credit timestamp on restart; it still expires at its original deadline.
if (raw.credits === null) quota.credits = null;
else if (raw.credits && typeof raw.credits === "object" && !Array.isArray(raw.credits)) {
const stored = raw.credits as Record<string, unknown>;
if (typeof stored.observedAt === "number" && Number.isFinite(stored.observedAt) && stored.observedAt >= 0) {
quota.credits = { observedAt: stored.observedAt };
for (const key of ["hasCredits", "unlimited", "overageLimitReached", "allowed"] as const) {
if (typeof stored[key] === "boolean") quota.credits[key] = stored[key];
}
if (typeof stored.balance === "number" && Number.isFinite(stored.balance) && stored.balance >= 0) {
quota.credits.balance = stored.balance;
}
}
}
return { identityKey: entry.identityKey, quota };
}

Expand Down Expand Up @@ -845,7 +870,7 @@ function filterMainPolicyMonthlyQuota(
delete filtered.monthlyResetAt;
delete filtered.monthlyIsPrimaryWindow;
// Null retains the matching prior observation; an empty object would merge away evidence.
return hasKnownQuotaValue(filtered) || filtered.resetCredits !== undefined ? filtered : null;
return hasKnownQuotaValue(filtered) || snapshotHasCredits(filtered) ? filtered : null;
}

/**
Expand Down Expand Up @@ -886,11 +911,25 @@ export function parseUsageQuota(data: WhamUsageResponse): Omit<StoredAccountQuot
? data.rate_limit_reset_credits.available_count
: undefined;

if (!data.rate_limit) {
return resetCredits !== undefined ? { resetCredits } : null;
}

const quota: Omit<StoredAccountQuota, "updatedAt"> = {};
if (resetCredits !== undefined) quota.resetCredits = resetCredits;
if (data.credits !== undefined) {
quota.credits = null;
if (data.credits && typeof data.credits === "object" && !Array.isArray(data.credits)) {
const raw = data.credits;
const balance = typeof raw.balance === "number" ? raw.balance
: typeof raw.balance === "string" && raw.balance.trim() !== "" ? Number(raw.balance) : undefined;
quota.credits = {
observedAt: Date.now(),
...(typeof raw.has_credits === "boolean" ? { hasCredits: raw.has_credits } : {}),
...(typeof raw.unlimited === "boolean" ? { unlimited: raw.unlimited } : {}),
...(typeof raw.overage_limit_reached === "boolean" ? { overageLimitReached: raw.overage_limit_reached } : {}),
...(typeof data.rate_limit?.allowed === "boolean" ? { allowed: data.rate_limit.allowed } : {}),
...(typeof balance === "number" && Number.isFinite(balance) && balance >= 0 ? { balance } : {}),
};
}
}
if (!data.rate_limit) return snapshotHasCredits(quota) ? quota : null;
const thirtyDayOnly = codexQuotaWindowForPlan(data.plan_type) === "monthly";
const primaryWindow = data.rate_limit.primary_window;
const secondaryWindow = data.rate_limit.secondary_window;
Expand Down Expand Up @@ -951,7 +990,7 @@ export function parseUsageQuota(data: WhamUsageResponse): Omit<StoredAccountQuot

if (resetCredits !== undefined) quota.resetCredits = resetCredits;

return hasKnownQuotaValue(quota) || resetCredits !== undefined ? quota : null;
return hasKnownQuotaValue(quota) || snapshotHasCredits(quota) ? quota : null;
}


Expand Down
12 changes: 9 additions & 3 deletions src/codex/routing/cooldown-math.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import {
CODEX_UNKNOWN_USAGE_SCORE,
} from "../quota";
import { isThirtyDayOnlyCodexPlan } from "../plan";
import { isTerminalShortWindow } from "../quota-types";
import { hasSpendableCodexCredits, isTerminalShortWindow, type CodexCredits } from "../quota-types";
import type { CodexQuotaScope } from "./health-store";
import type { TransientProbeGrant } from "./thread-affinity";

Expand Down Expand Up @@ -107,6 +107,7 @@ export function computeCodexUsageScore(quota: {
shortPercent?: number;
shortResetAt?: number;
shortObservedAt?: number;
credits?: CodexCredits | null;
} | null, plan?: unknown, now: number = Date.now()): number {
if (!quota) return CODEX_UNKNOWN_USAGE_SCORE;
const finite = (value: unknown): value is number => typeof value === "number" && Number.isFinite(value);
Expand All @@ -126,10 +127,15 @@ export function computeCodexUsageScore(quota: {
// correct for uncertainty and wrong for a measured refusal: the account stays selected,
// `applyQuotaAutoSwitch` never fires, and the pool wedges on an exhausted credential.
if (knownLong.length === 0) {
return isTerminalShortWindow(quota, now) ? CODEX_EXHAUSTED_USAGE_PERCENT : CODEX_UNKNOWN_USAGE_SCORE;
return isTerminalShortWindow(quota, now)
? hasSpendableCodexCredits(quota, now) ? CODEX_EXHAUSTED_USAGE_PERCENT - 1 : CODEX_EXHAUSTED_USAGE_PERCENT
: CODEX_UNKNOWN_USAGE_SCORE;
}
const values = finite(quota.shortPercent) ? [...knownLong, quota.shortPercent] : knownLong;
return Math.max(...values);
const score = Math.max(...values);
// Keep included-capacity accounts ahead of a paid overage account, but do not
// classify the latter as exhausted solely from its subscription percentages.
return hasSpendableCodexCredits(quota, now) ? Math.min(score, CODEX_EXHAUSTED_USAGE_PERCENT - 1) : score;
}

// `isTerminalShortWindow` moved to ../quota-types, the leaf the dashboard can import. Routing
Expand Down
2 changes: 2 additions & 0 deletions structure/catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -596,3 +596,5 @@ Startup and explicit catalog synchronization in `src/codex/sync.ts` refresh the
gathering. Each sync waits at most two seconds for a fresh or shared fetch, then continues with the existing snapshot; the fetch retains its own abort deadline. Routed effort reads in
`src/reasoning-effort.ts` use a snapshot immediately and request a best-effort background refresh
only when an existing snapshot answers with an expired ladder. Missing or corrupt snapshots do not fetch on the request path; catalog sync owns their bootstrap.

Automatic account exhaustion and recovery use the [spendable Codex credit evidence contract](providers/openai-tiers.md#spendable-codex-credits), including independent freshness, upstream refusal, and reset-ticket separation.
2 changes: 2 additions & 0 deletions structure/codex-home.md
Original file line number Diff line number Diff line change
Expand Up @@ -465,3 +465,5 @@ Upstream API-key usage follows the [physical-attempt account attribution contrac
Stored Direct substitution follows the [credential identity contract](providers/openai-accounts.md#sidecars-management-and-ui): both synchronous and asynchronous materializers discard the caller account header before applying the stored credential; ordinary native Direct passthrough is unchanged.

Native-main owner claims and credential-generation backoff remain authoritative during [priority failback priming](providers/openai-accounts.md#ongoing-priority-failback); the preference grants no access through a fenced main profile.

Automatic account exhaustion and recovery use the [spendable Codex credit evidence contract](providers/openai-tiers.md#spendable-codex-credits), including independent freshness, upstream refusal, and reset-ticket separation.
2 changes: 1 addition & 1 deletion structure/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -498,7 +498,7 @@ Client catalog readiness observes the selected Codex runtime without creating or

Client connection metadata stores a stable `apiKeyId` and a non-secret rotation `pendingOperation`. The current data secret remains only in `service-api-token`; a bounded rotation temporarily keeps the old secret in owner-only `service-api-token.prev`. Commit or recovery clears the marker before orphan cleanup. `ocx disconnect` is local-only and leaves remote revocation to the hub's **Integrations → API Keys** page. Hub and local usage stores are not mirrored.

Codex display-cache expiry, retained blocking main-policy evidence, and reset history follow the [quota cache contract](providers/openai-tiers.md#quota-cache-and-short-window-history).
Codex display-cache expiry, retained blocking main-policy evidence, and reset history follow the [quota cache contract](providers/openai-tiers.md#quota-cache-and-short-window-history); automatic account decisions also use [spendable credit evidence](providers/openai-tiers.md#spendable-codex-credits).

`codexPool.excludedPlans` is interpreted only by automatic selection; its all-excluded and explicit-route behavior follows the [plan exclusion contract](providers/openai-accounts.md#automatic-pool-plan-exclusions). Optional `codexPool.startIdleWindows` defaults off and follows the [idle-window steering contract](providers/openai-accounts.md#idle-window-steering), using real new requests to start observed idle 5-hour windows.

Expand Down
2 changes: 2 additions & 0 deletions structure/gui-and-management-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -561,3 +561,5 @@ keep their existing no-catalog-refresh behavior. The regression suite is
> Decision record: [Durable provider PATCH](decisions/ADR-0104-durable-provider-patch.md)

> Decision record: [Publication-aware rollback](decisions/ADR-0120-provider-patch-publication-boundary.md)

Automatic account exhaustion and recovery use the [spendable Codex credit evidence contract](providers/openai-tiers.md#spendable-codex-credits), including independent freshness, upstream refusal, and reset-ticket separation.
2 changes: 2 additions & 0 deletions structure/ops/docs-and-release.md
Original file line number Diff line number Diff line change
Expand Up @@ -595,3 +595,5 @@ Linux release bundling enables Tauri verbosity on the primary attempt so linuxde

Universal macOS release builds install both aarch64-apple-darwin and x86_64-apple-darwin Rust targets. Windows builds consume the private JSON override generated by `desktop/scripts/windows-installer-config.ts`: only WiX ProductVersion uses the validated numeric public version core. Public package/application versions, tags, asset names and updater manifests retain full SemVer. The pinned Tauri MSI template permits equal-core replacement; manual MSI installation does not enforce same-core preview/stable downgrade prevention.
The existing `codex-routing`, `codex-auth-context` and `codex-quota-prime` tests cover [priority failback](../providers/openai-accounts.md#ongoing-priority-failback), including cache-default retention, stale evidence, main fencing and failed-attempt cadence.

Automatic account exhaustion and recovery use the [spendable Codex credit evidence contract](../providers/openai-tiers.md#spendable-codex-credits), including independent freshness, upstream refusal, and reset-ticket separation.
Loading
Loading