Repository navigation
_fetch_custom_model_limit never hits its cache for custom/<model> — /v1/models is fetched before every call (patch included) #183
Description
providers.py::_fetch_custom_model_limit(base_url, model, api_key) receives the full model string (custom/tools-pool)
but fills _custom_ctx_cache[base_url] with the ids the endpoint returns (tools-pool). if model in cache /
cache.get(model) therefore miss on every call, and a miss is not remembered, so both callers (resolve_max_tokens,
get_model_context_window) issue GET /v1/models before each chat completion — 2 extra requests per turn, ~4× the
number of model calls in a long session (184 GETs for 47 completions on 3.5.87). Endpoints that do not report
max_model_len/context_window (LiteLLM: max_input_tokens) are asked forever.
Fix (2 hunks): look the cache up by bare_model(model) and memoise a miss as 0 (returned as None). Patch attached;
verified with a stub server: 4 lookups → 1 request (was 4), unknown model 3 lookups → 1 request (was 3).
Patch (2 hunks, against 3.5.87 site-packages; applies to main with patch -p1):
--- a/cheetahclaws/providers.py 2026-09-20 17:14:29.456536700 -0700
+++ b/cheetahclaws/providers.py 2026-09-20 17:14:29.505910500 -0700
@@ -501,8 +501,13 @@
128000 default.
"""
cache = _custom_ctx_cache.setdefault(base_url, {})
- if model in cache:
- return cache[model]
+ # Gateway routes keep the provider prefix ("custom/tools-pool") while the
+ # endpoint lists bare ids ("tools-pool"): look the cache up by the bare id,
+ # and remember a miss (0) so a model the endpoint does not describe is
+ # asked about once per process, not before every call.
+ key = bare_model(model)
+ if key in cache:
+ return cache[key] or None
try:
url = base_url.rstrip("/") + "/models"
req = urllib.request.Request(
@@ -515,7 +520,9 @@
limit = entry.get("max_model_len") or entry.get("context_window")
if limit:
cache[mid] = int(limit)
- result = cache.get(model)
+ result = cache.get(key)
+ if result is None:
+ cache[key] = 0
# Backfill provider-level default with the most conservative value seen.
# Compaction reads PROVIDERS[provider]['context_limit'] without knowing
# base_url, so this makes the threshold see the real limit.
Activity
chauncygu commentedon Oct 12, 2026
Thanks for the report and the patch — confirmed on main. Two separate defects: the prefixed form (
custom/tools-pool, passed by compaction) never matched the bare ids the endpoint lists, and a miss was never remembered, so endpoints that report no limit were re-queried before every call.I fixed it slightly differently from the attached patch, because calling
bare_model()inside the function regresses one case:stream_openai_compatalready passes the id with the provider prefix stripped, so a vLLM-style id likeQwen/Qwen3-Codergets cut down toQwen3-Coder, misses, and is then memoised as unknown — losing a limit that resolves correctly today.What went in instead:
custom/removed.(base_url, model)in a separate set, so the limit cache only ever holds real limits.Verified with a stub endpoint: 3 lookups → 1 request for bare, prefixed, and slash-containing ids, and for a model with no reported limit.
Not covered here: reading LiteLLM's
max_input_tokens. Those endpoints are no longer re-queried, but the limit still falls back to the default — happy to take that as a follow-up.The fix will be in the next release. @linhongyu510 thanks for #187 — this supersedes it, for the slash-id reason above.