[Feature]: requestPacing option to cap concurrent in-flight requests at provider and model level #5702

Area

Proxy and routing

What are you trying to accomplish?

Share one provider account (z.ai GLM coding plan) between long-running batch workloads and interactive Codex sessions through the proxy, and stay inside the upstream's concurrent-request limit so batch traffic stops eating 429s that also hurt interactive traffic.

What prevents this today?

requestPacing spaces outbound request starts: RPM becomes an even interval, minIntervalMs floors it, and dispatch waits its turn in waitForProviderRequestSlot. That works as documented, but z.ai enforces a concurrent in-flight request limit per account, not a request rate: HTTP 429, code 1302, "Rate limit reached for requests", whenever too many streaming turns are open at once. Start pacing cannot prevent that, because a long streaming turn consumes almost no request-rate budget while it holds a concurrency slot for minutes.

Evidence from a 2.54.0 proxy (single zai provider, one account, pacing enabled at 40 RPM / 1500 ms):

• A file-review batch averaging about 10 request starts per minute still lost roughly 31% of its LLM calls to 429/1302, because turns with high reasoning effort stay in flight for minutes and 8+ were open concurrently.
• Usage-ledger correlation: 429s cluster once concurrent in-flight z.ai requests pass roughly 5-8; 200s complete mostly at 1-3 in flight. The long-running high-effort glm-5.3 turns hold the slots; lighter models on the same account are collateral damage.
• A second client using the same account directly (outside the proxy, concurrency 12) made even the proxy's own 1-2-in-flight requests 429, so the cap is account-wide, not proxy-local.

What should OpenCodex do?

Offer an optional concurrency cap in requestPacing, settable at both the provider and the model level like the interval controls today (default off):

• provider-wide: requestPacing.maxConcurrentRequests as the account/provider-level semaphore;
• per-model: entries under the existing requestPacing.models map (same exact upstream model IDs) can carry their own maxConcurrentRequests; a model-specific cap tightens the provider-wide cap for that model, mirroring how per-model intervals only add delay today;
• acquired at the same waitForProviderRequestSlot boundary so HTTP, Responses WebSocket, and adapter dispatches stay covered;
• released when the upstream response body closes, including abort, error, and mid-turn key-rotation paths;
• overflow reuses RequestPacingQueueOverloadError and the existing retryable pacing 429, so clients see the same retryable shape as queue-full today;
• optionally per key when apiKeyPool is in use, since pooled keys usually carry separate caps.

Example usage or interface

Config fragment:

{
  "providers": {
    "zai": {
      "requestPacing": {
        "enabled": true,
        "requestsPerMinute": 40,
        "minIntervalMs": 1500,
        "maxConcurrentRequests": 5,
        "models": { "glm-5.3": { "maxConcurrentRequests": 2 } }
      }
    }
  }
}
}

Before/after: today a third concurrent glm-5.3 dispatch is sent upstream and returns 429/1302 after wasting a request slot; with the cap, that dispatch waits at the pacing boundary until one in-flight response body closes, then proceeds without touching the upstream limit.

Alternatives or workarounds

Keep client-side concurrency low (batch tools at concurrency 2-3) and avoid sharing one provider account between batch and interactive traffic. RPM pacing alone does not stop 1302-style errors. Both workarounds constrain the whole account for the sake of its slowest workload.

Additional context

Checked current main (4cb43cb): src/providers/request-pacing.ts tracks only next-start times; there is no in-flight accounting in the dispatch path. The release boundary (stream close vs response return, WS transport, key rotation) is the design-sensitive part, so I filed this before a PR per the contributing guide; happy to prototype once the intended boundary is confirmed.

Checks

• I searched existing issues and documentation.
• This request describes a concrete OpenCodex workflow rather than merely naming a desired technology.
• I removed secrets and personal data.