Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs-site/src/content/docs/guides/combos.md
Original file line number Diff line number Diff line change
Expand Up @@ -815,7 +815,10 @@ Combos are stored in the top-level `combos` object, keyed by combo id:
| --- | --- | --- | --- |
| `targets` | Yes | — | Non-empty ordered array of configured `{ provider, model, weight?, lastResort? }` targets. Duplicate provider/model pairs are rejected. |
| `targets[].weight` | No | `1` | Integer from 1 to 10,000. Used by round-robin and random; ignored by failover, least-used, reset-window, and JEV. |
| `targets[].reasoningEffort` | No | unset | Explicit per-target `low`, `medium`, `high`, `xhigh`, `max`, or `ultra` override. Other targets preserve the caller's effort. The operator owns target support; ordinary downstream validation still applies. |
| `targets[].serviceTier` | No | unset | Per-target `default` or `priority`. `priority` requests Fast through the destination's existing capability/wire policy; it does not enable an unsupported Fast lane. |
| `targets[].lastResort` | No | `false` | Marks an emergency-only target. Inert unless `cooldownWaitPolicy` is set. Never makes a target permanently ineligible: when no normal target can be reached it is dispatched as usual. |
| `targets[].metered` | No | `false` | Marks a pay-per-use target, such as an Anthropic API-key row placed after subscription targets. The environment token and loopback callers may reach it; an issued `apiKeys` entry may not unless it sets `allowMeteredComboTargets: true`, so its request stops at the last unmetered target and returns that target's error. |
| `strategy` | No | `"failover"` | `"failover"`, `"round-robin"`, `"random"`, `"least-used"`, `"reset-window"`, or `"jev"`. JEV decides only the initial eligible target and effort; ordinary Combo fallback owns later attempts. |
| `stickyLimit` | No | `1` | Integer from 1 to 100 successful requests per round-robin selection. Applies only to round-robin. |
| `cooldownMs` | No | unset → upstream fallback (5 s for request-rate 429 codes `1302`/`1305`, 10 min for a spent usage window or a credential/billing failure, otherwise 60 s) | Integer from 1 to 600000. When set, applies as the per-target cooldown whenever no usable upstream `Retry-After` or Codex reset signal exists, including request-rate 429s; when unset, uses the upstream fallback. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -917,6 +917,7 @@ for the subscription and client conditions the pool is meant for.
| --- | --- | --- | --- |
| `anthropicAccountPool.enabled?` | `boolean` | `false` | Enable sticky session affinity and quota-ranked new-session selection. **429 failover is not gated here**: it activates whenever two or more usable accounts are stored, exactly like every other multi-credential provider, and cannot be switched off. |
| `anthropicAccountPool.nativeMessages?` | `boolean` | `true` | Prefer native Claude Messages for eligible direct Anthropic routes when pooling is enabled. Explicit native rollout opt-outs still apply. `false` selects the legacy bridge for pooled requests; pool-off retains explicit rollout behavior. |
| `anthropicAccountPool.quotaRecheckMs?` | `number` | unset | Upper bound, in milliseconds (60000–86400000), on how long a 429-refused account stays cooled. A spent weekly window can state a reset days away; with this set the account is offered again after at most this long, answers 429 again if still spent, and serves at once if it recovered early. Applies to 429 refusals only; account 403 backoff is unchanged. |
| `anthropicAccountPool.autoSwitchThreshold?` | `number` | `80` | For new sessions, when the active account reaches this threshold, choose the lowest known cached usage in the configured window; the account chosen does not itself have to be at or above the threshold. `0` disables **proactive** usage-based switching only — new-session selection and routing recovery after an eligible 429 still consult `quotaWindow`. |
| `anthropicAccountPool.strategy?` | `"quota" \| "round-robin" \| "fill-first"` | `"quota"` | New-session strategy; `quota` ranks accounts by the window set by `quotaWindow`, and `fill-first` evaluates its drain threshold in that same window. |
| `anthropicAccountPool.quotaWindow?` | `"five-hour" \| "weekly" \| "max-utilization"` | `"five-hour"` | The cached provider-reported utilization bar used for usage-aware account selection. `five-hour` keeps the original behavior. `weekly` scores the weekly bar and skips accounts whose 5-hour bar is exhausted while another eligible account remains, but falls back to exhausted candidates when none do. `max-utilization` scores the highest known bar, so it can use 5-hour usage before weekly usage is available; if neither is known, the account follows unknown-usage ordering. Known usage ranks before unknown usage under the opt-in `weekly` and `max-utilization` windows only; an omitted or explicit `five-hour` preserves the legacy ordering. If every eligible account is unknown, selection still returns one in eligible order. After the documented lower-5-hour tie-break, exact ties preserve eligible order. A healthy affinity-bound session is not proactively rebalanced. For new-session assignment and routing recovery after an eligible 429 replacement, `quota` ranks eligible candidates directly with this window; `fill-first` advances in stable order using this window's threshold and exhaustion rules; `round-robin` ignores it. Cooldown, failover limits, and reauthentication eligibility remain separate local state. Per-account weekly bars come from usage probes or observed response headers. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ namespace, and cannot use reserved bare native families such as `gpt-*`, `o1-*`,

| Key | Type | Default | Meaning |
| --- | --- | --- | --- |
| `targets` | `{ provider: string; model: string; weight?: number; lastResort?: boolean }[]` | required | Ordered concrete routes. `weight` is 1–10000 and defaults to `1`. `lastResort` marks an emergency-only target; see `cooldownWaitPolicy`. |
| `targets` | `{ provider: string; model: string; weight?: number; lastResort?: boolean; metered?: boolean }[]` | required | Ordered concrete routes. `weight` is 1–10000 and defaults to `1`. `lastResort` marks an emergency-only target; see `cooldownWaitPolicy`. `metered` marks a pay-per-use target that issued API keys reach only with `allowMeteredComboTargets: true`; the environment token and loopback callers are not restricted. |
| `strategy?` | `"failover" \| "round-robin" \| "random" \| "least-used" \| "reset-window" \| "jev"` | `"failover"` | Selection strategy. Target order is failover priority; weights shape round-robin and random draws; least-used follows recorded successes; reset-window follows the soonest quota reset; JEV makes one bounded decision for the initial eligible target and effort, then uses ordinary ordered fallback. |
| `stickyLimit?` | `number` | `1` | Successful requests retained in one round-robin batch. Range 1–100. Applies only to round-robin. |
| `cooldownMs?` | `number` | unset → upstream fallback (5 s for request-rate 429 codes `1302`/`1305`, otherwise 60 s) | Range 1–600000. When set, applies whenever no usable upstream `Retry-After` or Codex reset signal exists, including request-rate 429s; when unset, uses the upstream fallback. Upstream signals take precedence. An explicit upstream `Retry-After` is capped at 24 hours; reset-derived, configured, and fallback cooldowns are capped at 10 minutes. |
Expand Down
1 change: 1 addition & 0 deletions scripts/test-layout/layout.json
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,7 @@
"anthropic-account-pause.test.ts": "adapters/anthropic",
"anthropic-account-threshold.test.ts": "adapters/anthropic",
"anthropic-combo-account-cooldown.test.ts": "adapters/anthropic",
"anthropic-subscription-chain.test.ts": "adapters/anthropic",
"cli-alias-json.test.ts": "cli",
"cli-anthropic-account-threshold.test.ts": "cli",
"anthropic-model-routes.test.ts": "adapters/anthropic",
Expand Down
7 changes: 7 additions & 0 deletions src/adapters/openai-responses/passthrough.ts
Original file line number Diff line number Diff line change
Expand Up @@ -574,6 +574,8 @@ export function createResponsesPassthroughAdapter(provider: OcxProviderConfig):
validateFinalCustomToolCompatibility(finalBody, provider.supportsResponsesCustomTools);
}
const body = JSON.stringify(applyGithubCopilotContextTier(finalBody, provider, parsed.modelId, incoming.providerName));
const wireReasoning = isPlainObject(finalBody) && isPlainObject(finalBody.reasoning)
? finalBody.reasoning.effort : undefined;
const releaseBodyObservation = translatorBudget.observeExternallyCapped(
"passthrough_serialization",
Buffer.byteLength(body, "utf8"),
Expand All @@ -583,6 +585,11 @@ export function createResponsesPassthroughAdapter(provider: OcxProviderConfig):
method: "POST",
headers,
body,
...(typeof wireReasoning === "string" ? { reasoningLog: {
effectiveEffort: wireReasoning,
wireField: "reasoning.effort" as const,
wireValue: wireReasoning,
} } : {}),
releaseBodyObservation,
...(convertedRoutedCustomToolNames ? { convertedRoutedCustomToolNames } : {}),
...(routedCustomToolRepairNames ? { routedCustomToolRepairNames } : {}),
Expand Down
12 changes: 11 additions & 1 deletion src/cli/combo-input.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ type ComboInput = {
reasoningEffortMode?: "strict" | "adaptive";
nativeAlias?: boolean;
};
const TARGET_KEYS = new Set(["provider", "model", "weight", "reasoningEfforts", "modelProfile", "lastResort"]);
const TARGET_KEYS = new Set(["provider", "model", "weight", "reasoningEfforts", "modelProfile", "lastResort", "metered", "reasoningEffort", "serviceTier"]);

export function parseComboTargets(value: unknown): Array<Record<string, unknown>> {
if (!Array.isArray(value) || value.length === 0) throw new CliUsageError("Targets input must be a nonempty array");
Expand All @@ -27,6 +27,13 @@ export function parseComboTargets(value: unknown): Array<Record<string, unknown>
|| !Number.isInteger(target.weight) || target.weight < 1 || target.weight > 10_000)) {
throw new CliUsageError("Target weight must be an integer from 1 to 10000");
}
if (target.reasoningEffort !== undefined
&& (typeof target.reasoningEffort !== "string" || !isCodexReasoningEffort(target.reasoningEffort))) {
throw new CliUsageError("Target reasoningEffort must be low, medium, high, xhigh, max or ultra");
}
if (target.serviceTier !== undefined && target.serviceTier !== "default" && target.serviceTier !== "priority") {
throw new CliUsageError("Target serviceTier must be default or priority");
}
if (Object.hasOwn(target, "reasoningEfforts") && (!Array.isArray(target.reasoningEfforts)
|| target.reasoningEfforts.length === 0
|| target.reasoningEfforts.some((effort: unknown) => typeof effort !== "string" || !isCodexReasoningEffort(effort))
Expand All @@ -41,6 +48,9 @@ export function parseComboTargets(value: unknown): Array<Record<string, unknown>
if (Object.hasOwn(target, "lastResort") && typeof target.lastResort !== "boolean") {
throw new CliUsageError("Target lastResort must be a boolean");
}
if (Object.hasOwn(target, "metered") && typeof target.metered !== "boolean") {
throw new CliUsageError("Target metered must be a boolean");
}
// Preserve raw slashes, target order, explicit false and omitted metadata.
return { ...target };
});
Expand Down
1 change: 1 addition & 0 deletions src/combos/failover.ts
Original file line number Diff line number Diff line change
Expand Up @@ -634,6 +634,7 @@ const DEFINITE_CONTEXT_OVERFLOW_PHRASES = [
"exceed the context window",
"context window exceeded",
"context length exceeded",
"cursor context limit exceeded",
"maximum context length",
"maximum context window",
"too many tokens",
Expand Down
11 changes: 10 additions & 1 deletion src/combos/request.ts
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,23 @@ function responsesInputNodeHasImage(value: unknown): boolean {
*/
export function concreteComboRequestBody(
body: unknown,
target: Pick<OcxComboTarget, "provider" | "model">,
target: Pick<OcxComboTarget, "provider" | "model" | "reasoningEffort" | "serviceTier">,
defaultEffort: OcxComboDefaultEffort | null,
targetReasoningEfforts: readonly string[] | undefined,
reasoningEffortMode: OcxComboReasoningEffortMode = "strict",
defaultEffortMode: OcxComboDefaultEffortMode = "fallback",
): Record<string, unknown> {
const clone = structuredClone(body) as Record<string, unknown>;
clone.model = `${target.provider}/${target.model}`;
if (target.serviceTier !== undefined) clone.service_tier = target.serviceTier;
if (target.reasoningEffort !== undefined) {
const reasoning = clone.reasoning;
const preserved = reasoning && typeof reasoning === "object" && !Array.isArray(reasoning)
? reasoning as Record<string, unknown> : {};
stripAlternativeReasoningControls(clone);
clone.reasoning = { ...preserved, effort: target.reasoningEffort };
return clone;
}
if (defaultEffortMode === "force" && (!defaultEffort || !isCodexReasoningEffort(defaultEffort))) {
throw new Error("force combo default effort requires a valid defaultEffort");
}
Expand Down
20 changes: 20 additions & 0 deletions src/combos/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,13 @@ export interface ComboValidationIssue {
export interface NormalizedComboTarget {
provider: string;
model: string;
reasoningEffort?: OcxComboDefaultEffort;
serviceTier?: "default" | "priority";
weight: number;
/** Emergency-only target, deferred under `cooldownWaitPolicy` (#5691). */
lastResort: boolean;
/** Pay-per-use target; issued keys reach it only with `allowMeteredComboTargets`. */
metered: boolean;
reasoningEfforts?: OcxComboDefaultEffort[];
/** Optional JEV decision description. */
modelProfile?: string;
Expand Down Expand Up @@ -395,6 +399,13 @@ export function comboConfigIssues(
message: `targets[${i}].weight must be an integer from 1 to 10000`,
});
}
if (target.reasoningEffort !== undefined
&& (typeof target.reasoningEffort !== "string" || !isCodexReasoningEffort(target.reasoningEffort))) {
issues.push({ path: ["targets", i, "reasoningEffort"], message: "reasoningEffort must be low, medium, high, xhigh, max or ultra" });
}
if (target.serviceTier !== undefined && target.serviceTier !== "default" && target.serviceTier !== "priority") {
issues.push({ path: ["targets", i, "serviceTier"], message: "serviceTier must be default or priority" });
}
if (target.reasoningEfforts !== undefined) {
if (!Array.isArray(target.reasoningEfforts) || target.reasoningEfforts.length === 0) {
issues.push({
Expand Down Expand Up @@ -427,6 +438,12 @@ export function comboConfigIssues(
message: `targets[${i}].lastResort must be a boolean`,
});
}
if (target.metered !== undefined && typeof target.metered !== "boolean") {
issues.push({
path: ["targets", i, "metered"],
message: `targets[${i}].metered must be a boolean`,
});
}
if (target.modelProfile !== undefined
&& (typeof target.modelProfile !== "string"
|| target.modelProfile.trim().length === 0
Expand Down Expand Up @@ -496,13 +513,16 @@ export function normalizeComboConfig(raw: OcxComboConfig): NormalizedComboConfig
provider: target.provider.trim(),
model: target.model.trim(),
weight: target.weight ?? 1,
...(target.reasoningEffort !== undefined ? { reasoningEffort: target.reasoningEffort } : {}),
...(target.serviceTier !== undefined ? { serviceTier: target.serviceTier } : {}),
...(target.reasoningEfforts !== undefined
? { reasoningEfforts: [...target.reasoningEfforts] }
: {}),
...(typeof target.modelProfile === "string" && target.modelProfile.trim()
? { modelProfile: target.modelProfile.trim() }
: {}),
lastResort: target.lastResort === true,
metered: target.metered === true,
})),
};
}
Expand Down
4 changes: 4 additions & 0 deletions src/config/schema/anthropic-account-pool.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ function poolError(value: unknown, path: string): string | undefined {
const parsed = parseAnthropicModelRoutes(pool.routes);
if (!parsed.ok) return `schema_invalid: ${path}.routes: ${parsed.error}`;
}
if (pool.quotaRecheckMs !== undefined && (typeof pool.quotaRecheckMs !== "number"
|| !Number.isInteger(pool.quotaRecheckMs) || pool.quotaRecheckMs < 60_000 || pool.quotaRecheckMs > 86_400_000)) {
return `schema_invalid: ${path}.quotaRecheckMs: must be an integer from 60000 to 86400000`;
}
return undefined;
}

Expand Down
2 changes: 2 additions & 0 deletions src/config/schema/leaf-validators.ts
Original file line number Diff line number Diff line change
Expand Up @@ -793,6 +793,8 @@ export const apiKeyEntrySchema = z.object({
// rather than promoting it.
allowedProviders: z.array(z.string().trim().min(1).max(256)).optional(),
allowedModels: z.array(z.string().trim().min(1).max(256)).optional(),
// Not `.catch`ed for the same reason: a damaged value must not grant metered spend.
allowMeteredComboTargets: z.boolean().optional(),
}).passthrough();

/**
Expand Down
10 changes: 9 additions & 1 deletion src/oauth/anthropic-routing.ts
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,12 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting
return anthropicAccountPoolConfig(config).enabled === true;
}

function anthropicQuotaRecheckMs(config: OcxConfig): number | undefined {
const value = anthropicAccountPoolConfig(config).quotaRecheckMs;
return typeof value === "number" && Number.isInteger(value) && value >= 60_000 && value <= 86_400_000
? value : undefined;
}

function anthropicAutoSwitchThreshold(config: OcxConfig): number {
const value = anthropicAccountPoolConfig(config).autoSwitchThreshold;
if (typeof value === "number" && Number.isInteger(value) && value >= 0 && value <= 100) return value;
Expand Down Expand Up @@ -1084,7 +1090,9 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting
// account is back in the rotation a minute later.
const parsedRetry = parseRetryAfterMs(retryAfterHeader, now);
const resetDerived = status === 429 && parsedRetry === undefined ? parseRateLimitReset(rateLimitHeaders, now) : undefined;
const cooldownMs = parsedRetry ?? resetDerived?.delayMs ?? (status === 403 ? ACCOUNT_REFUSAL_COOLDOWN_MS : DEFAULT_COOLDOWN_MS);
const statedCooldownMs = parsedRetry ?? resetDerived?.delayMs ?? (status === 403 ? ACCOUNT_REFUSAL_COOLDOWN_MS : DEFAULT_COOLDOWN_MS);
const recheckMs = anthropicQuotaRecheckMs(config);
const cooldownMs = status === 429 && recheckMs !== undefined ? Math.min(statedCooldownMs, recheckMs) : statedCooldownMs;
const cooldownGeneration = noteAnthropicCooldownMutation(failedAccountId);
upstreamHealth.set(failedAccountId, {
cooldownUntil: now + cooldownMs,
Expand Down
17 changes: 17 additions & 0 deletions src/server/admission-model-scope.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,23 @@ export function resolveAdmissionModelScope(
return entry ? admissionModelScopeOf(entry) : undefined;
}

/**
* May this request reach a combo target marked `metered` (pay-per-use billing)?
*
* The environment token and loopback admission are the operator's own clients, so they may.
* An issued key may not unless its entry opts in with `allowMeteredComboTargets`: a key with
* `allowedModels` naming a subscription model must not start spending pay-per-use credit the
* moment that subscription is exhausted, which is what a failover combo would otherwise do.
*/
export function admissionMayUseMeteredComboTargets(
config: Pick<OcxConfig, "apiKeys">,
admission: DataPlaneAdmission | undefined,
): boolean {
if (!admission || admission.kind !== "configured") return true;
const entry = (config.apiKeys ?? []).find(key => key.id === admission.keyId);
return entry?.allowMeteredComboTargets === true;
}

/**
* Does this scope admit this resolved destination?
*
Expand Down
Loading
Loading