Repository navigation
[Bug]: Intermittent consecutive zero cache hits for Hermes requests through OpenCodeX #3433
Description
Client or integration
Other — Hermes Agent v0.21.0 using OpenAI-compatible Chat Completions
Area
Proxy and routing
Summary
Hermes requests routed through OpenCodeX can produce several consecutive successful responses with cached_tokens = 0, then suddenly recover to a very large cached prefix without changing the model, endpoint, or logical conversation.
This is intermittent rather than a permanent cache failure. The current evidence points to an interaction among caller-provided prompt_cache_key, session-affinity metadata, the Chat Completions → Codex Responses bridge, and long-session reconstruction/compression. It is not yet clear whether the unstable cache identity is introduced by Hermes, OpenCodeX, or the downstream route.
Expected behavior: repeated turns in the same logical conversation should retain a stable cache identity. Context compression may naturally change the prompt prefix, but ordinary continuation turns should not alternate unpredictably between a large cached prefix, a small fixed prefix, and zero.
Reproduction
The issue is intermittent and is easiest to observe in a long-running, tool-heavy Hermes conversation.
- Start OpenCodeX and route Hermes through its OpenAI-compatible
/v1endpoint. - Configure the Hermes provider with
api_mode: chat_completions. - Use one GPT-5.6 model for a long sequence of normal conversation and tool turns.
- Observe
input_tokensandcached_tokensin OpenCodeX usage logs. - Continue through normal turns and, eventually, a Hermes context-compression boundary.
A representative sequence from one logical conversation was:
input: 269,330 cached: 237,056
input: 269,925 cached: 16,896
input: 272,000 cached: 256,512
input: 273,706 cached: 16,896
input: 274,956 cached: 0
input: 278,878 cached: 0
input: 279,558 cached: 6,656
Later, without changing the model or endpoint:
input: 404,570 cached: 393,728
In a before/after observation around a Hermes update:
Before update:
measured requests: 21
weighted cache-read rate: 90.7%
zero-cache requests: 0
After update:
measured requests: 45
weighted cache-read rate: 26.0%
zero-cache requests: 26
These aggregates were restricted to the Hermes request path; unrelated client traffic present in the same OpenCodeX usage log was excluded.
Integration investigation already performed
-
OpenCodeX continued returning HTTP 200 during the zero-cache sequences.
-
The model and endpoint stayed unchanged.
-
Large cache hits later returned, so the cache path was not permanently unavailable.
-
Restoring the documented
api_mode: chat_completionsform did not eliminate the behavior. -
A custom Hermes provider plugin previously failed to load because it passed an unsupported constructor field:
supports_codex_affinity_headers=TrueRemoving that incompatible constructor field allowed the plugin to load. This likely explains one degraded interval, but not the whole issue: consecutive zero-cache responses were still observed after the plugin loaded successfully.
-
Hermes can expose
supports_prompt_cache_keyand model-scoped cache/affinity capability methods. It is not yet confirmed that the complete cache identity reaches OpenCodeX on every request. -
Automatic Hermes Background Review was moved to a different model as an isolation test while remaining enabled. The main-model conversation still produced consecutive zero-cache responses, so Background Review is not sufficient as the sole explanation.
-
Context compression and long-session reconstruction may amplify the problem, but they do not explain why later requests can abruptly recover a very large cached prefix.
Questions for OpenCodeX
Could OpenCodeX confirm, for the Chat Completions → Codex Responses path:
- Whether an inbound
prompt_cache_keyis preserved and forwarded unchanged. - Whether stable affinity data such as
session_id,x-client-request-id, or a thread/conversation identifier is preserved or generated. - Whether body-level
prompt_cache_keyalone is sufficient for stable cache routing on the GPT-5.6 Codex backend when affinity headers are absent. - Whether a request may be routed to a different backend/cache cohort despite an unchanged model and cache key.
- Whether the diagnostics can expose redacted presence/absence fields for the cache key and affinity identifiers, plus the resolved routing cohort. Raw values are not needed.
A useful reproduction matrix would compare:
A. prompt_cache_key present, affinity headers present
B. prompt_cache_key present, affinity headers absent
C. prompt_cache_key absent, affinity headers present
D. prompt_cache_key absent, affinity headers absent
For each case, recording the inbound protocol, downstream protocol, resolved route, cache-key presence, affinity presence, conversation correlation, input tokens, and cached tokens would identify which hop changes the cache identity.
Version
OpenCodeX 2.41.0
Hermes Agent v0.21.0 (upstream 63279301)
Operating system
macOS 26.5.2, arm64
Provider and model
OpenCodeX native OpenAI/Codex route, GPT-5.6 model family
Logs or error output
Provider plugin compatibility error observed during one phase:
ProviderProfile.__init__() got an unexpected keyword argument 'supports_codex_affinity_headers'
Representative successful requests from one logical conversation:
input=269330 cached=237056
input=269925 cached=16896
input=272000 cached=256512
input=273706 cached=16896
input=274956 cached=0
input=278878 cached=0
input=279558 cached=6656
Later recovery without model or endpoint change:
input=404570 cached=393728
No credentials, account labels, request IDs, local paths, or personal data are included.
Redacted configuration
{
"hermes_provider": {
"api_base": "http://127.0.0.1:<PORT>/v1",
"api_mode": "chat_completions",
"model": "gpt-5.6-<VARIANT>",
"api_key": "[REDACTED]"
},
"background_review_isolation_test": {
"enabled": true,
"model": "different GPT-5.6 variant"
}
}Checks
- I searched existing issues and documentation. Related cache-key forwarding work exists in [Bug]: Forward prompt_cache_key through the openai-chat adapter #217 and documentation in Document promptCacheKey opt-in for compatible custom openai-chat providers #2536, but I did not find an issue matching this intermittent Hermes bridge behavior.
- I removed secrets, tokens, account details, request credentials, request IDs, local user paths, and personal data.
Activity
lidge-jun commentedon Sep 4, 2026
리뷰 · 우선순위 64 / 80
설명
이 이슈는 Hermes Agent(v0.21.0)가 OpenCodeX의 OpenAI 호환
/v1/chat/completions로 GPT-5.6 계열을 길게 돌릴 때, 같은 대화인데도cached_tokens가 큰 값 → 작은 고정값 → 0 → 다시 큰 값으로 들쭉날쭉해지는 보고입니다. HTTP는 200이고 모델·엔드포인트는 안 바뀌었는데 캐시 히트만 흔들립니다. 보고자가 이미 Hermes 쪽 플러그인 생성자 오류(supports_codex_affinity_headers), Background Review 모델 분리, context compression 경계를 하나씩 빼 봤고, 그래도 연속 zero-cache가 남았다고 합니다. 질문은 명확합니다. Chat → Codex Responses 다리에서prompt_cache_key·affinity 헤더가 어디까지 살아 남는지, 키만으로 캐시 코호트가 안정적인지, 같은 키인데 다른 계정/코호트로 붙는지, 진단에 존재 여부만이라도 남길 수 있는지입니다.현재
devHEAD072df52eb(패키지 2.42.0) 코드를 보면, 이 증상이 “키를 아예 안 넘긴다”만으로 설명되진 않습니다.src/chat/inbound.ts의chatCompletionsToResponsesBody는 본문prompt_cache_key문자열을 Responses 바디로 그대로 복사합니다. 그래서 Hermes가 바디에 키를 넣으면 번역 단계에서 사라지지는 않습니다. 그런데 ChatGPT Codex 백엔드의 프롬프트 캐시 친화도는 코드 주석이 이미 다른 경로에서 밝혀 둔 대로session_id헤더 쪽에 더 크게 기대고 있습니다.src/server/claude-messages.ts는 Claude Code가 헤더를 안 보낼 때, 세션 단위prompt_cache_key로부터session_idUUID를 합성합니다. 주석에 “바디prompt_cache_key만으로는cached_tokens:0이 났다”(devlog 090)고 적혀 있습니다. 반면 Hermes가 쓰는 Chat Completions 다리(src/server/chat-completions.ts)는FORWARD_HEADERS에 있는 호출자 헤더만 복사하고, 바디 키로부터session_id를 합성하는 블록이 없습니다. Hermes가 키만 보내고 affinity 헤더를 안 보내면, Claude 경로에서 이미 막았던 “바디만 있고 헤더 없음” 구멍이 Chat 경로에 남아 있을 수 있습니다. 그게 이슈의 A/B/C/D 매트릭스와 정확히 맞습니다.두 번째 축은 계정 풀입니다. 기본
openai레지스트리 항목은adapter: openai-responses,codexAccountMode: pool입니다. 같은 모델·같은 키라도 턴마다 다른 Codex 계정으로 붙으면 업스트림 캐시 코호트가 바뀝니다. 최근 #3425(5h 소진 계정에 502로 계속 붙는 보고)와도 같은 “affinity/선택기가 대화 정체성을 얼마나 붙드느냐” 계열입니다. 보고의 간헐성·갑자기 큰 캐시 회복은 “키가 완전히 깨짐”보다 “어떤 턴은 sticky, 어떤 턴은 재바인딩/압축으로 접두가 갈림”에 더 잘 맞습니다. Hermes 쪽 context compression이 접두를 바꾸면 캐시가 떨어지는 것은 정상이고, 압축 없이 연속 0이 나는 구간만 OpenCodeX 쪽 의심 대상입니다.진단 쪽은
src/codex/affinity-debug.ts가session_id·x-client-request-id등 헤더의 존재/크기 버킷만 남기는 opt-in이 있고, 요청 로그는affinity·usage의cached_tokens를 기록합니다. 다만 이슈가 원하는 “inbound wire / cache-key 유무 / affinity 유무 / 라우팅 코호트”를 한 줄로 모아 주는 레드액트 필드는 아직 기본 usage UI에 없습니다. #217·#2536은 openai-chat 어댑터의promptCacheKey옵트인(이미 CLOSED)이라, 이번 Hermes→Codex Responses 간헐성 이슈와는 겹치되 중복 close 대상은 아닙니다. types.ts/config.ts 분할로 무효화할 이슈도 아닙니다.우선순위 64는 비용·지연이 체감되지만, 재현이 간헐적이고 클라이언트·압축·풀 로테이션이 섞여 있어 핫픽스 한 줄로 단정하기 전 조사 이슈로 두기 좋다는 뜻입니다.
경로 src/chat/inbound.ts chatCompletionsToResponsesBody - 바디
prompt_cache_key는 복사됩니다. “키가 번역에서 삭제된다”는 가설은 HEAD에서 약합니다.경로 src/server/chat-completions.ts FORWARD_HEADERS 복사 구간 - Claude 경로와 달리 바디 키 →
session_id합성이 없습니다. Hermes가 헤더 없이 키만 보내면 여기가 1순위 구멍입니다.경로 src/server/claude-messages.ts session_id 합성 주석 - ChatGPT 백엔드가 헤더 affinity에 의존한다는 기존 증거입니다. Chat 다리에 같은 정책을 쓸지 결정이 필요합니다.
경로 src/codex/affinity-debug.ts / 요청 로그 affinity·cached_tokens - 존재 여부 진단의 씨앗은 있습니다. 이슈가 원한 레드액트 매트릭스 필드는 아직 기본 노출이 아닙니다.
경로 풀 라우팅(openai registry pool) / #3425 - 계정 재바인딩만으로도 캐시 코호트가 갈릴 수 있습니다. 키 안정성과 별도로 확인하세요.
메인테이너의 판단이 필요한 지점
prompt_cache_key에서session_id를 합성할지(호출자가 보낸 헤더가 없을 때만)너의 추천
open bug로 유지하세요. 먼저 이슈에 적힌 A/B/C/D 매트릭스를 OpenCodeX usage 로그로 한 번만 재현해, Chat 다리에
session_id합성이 없는지·풀 재바인딩이 있는지부터 가릅니다. 유력하면 Claude와 같은 “메타데이터/세션 키 → session_id 합성(공유 코호트 키는 제외)”을 Chat Completions 경로에 이식하는 PR이 다음 단계입니다. 지금은 close하지 마세요. Hermes 플러그인 생성자 오류는 이미 분리됐으니 본 이슈 범위에서 빼도 됩니다.이 댓글은 grok-bot이 작성했습니다
Vivamisu commentedon Sep 4, 2026
Fresh redacted evidence: account transition, post-15:00 behavior, and
/#logs/debugI collected a fresh read-only sample from OpenCodeX
/#logsand/#logs/debugon 2026-09-04. Account labels below are anonymized as Account A and Account B; no account IDs, credentials, local paths, request IDs, or personal data are included.1. Before and after the plugin-stop/restart event
The two actions were concurrent, so this is not a single-variable plugin A/B:
Before that event, the long-context requests on Account A were:
This is 10 consecutive explicit zero-cache usage reports before the plugin-stop/restart event.
After the restart, Account A produced five long-context requests with non-zero cache reads, weighted at approximately 96.6% across that short sample:
This establishes a strong before/after correlation, but not plugin-only causality, because provider-plugin removal, Gateway restart, connection recreation, and runtime/session reconstruction happened together.
2. Account B transition and immediate behavior
The same long-running Hermes conversation then appeared on Account B. The first sequence was:
This is not just a normal one-request cold start. The same account shows zero, partial, zero, and then a large hit within minutes. It also means Account B is not simply incapable of caching.
3. Additional post-15:00 sample
From 15:00 to 15:04, Account A had 20 long-context
gpt-5.6requests:There were some partial hits in this sample, but no zero-cache request.
At 15:05 the conversation moved to Account B:
The later Account B sequence continued to alternate rather than settle:
For the full post-15:00 long-context sample:
The same Account B therefore reaches 98–99% on some turns but returns explicit zero on many neighboring turns. Account labels alone do not explain the alternation; the behavior is consistent with different cache/cohort or conversation-affinity branches being selected for different turns.
4. OpenCodeX
/#logsroute dataThe relevant requests consistently show:
The zero-cache rows are successful upstream responses, not transport failures.
5. OpenCodeX
/#logs/debugdataThe current provider-debug settings show debug, usage, and injection capture enabled by runtime override. The latest affinity-debug records for the relevant period consistently report:
{ "authKind": "pool", "accountMode": "pool", "fixedAccount": false, "credentialSubstituted": true, "inbound": [], "outbound": [], "inboundTurnMetadata": {"shape": "absent"}, "outboundTurnMetadata": {"shape": "absent"}, "inboundTurnState": {"shape": "absent"}, "outboundTurnState": {"shape": "absent"}, "status": 200 }Raw values are intentionally not requested. The important observations are that the requests use the account pool, are not fixed to one account, and the debug layer sees no inbound/outbound Codex turn metadata or turn state for these requests.
The existing request capture also showed that Hermes supplies a body-level
prompt_cache_key, while the Chat request did not contain the stable session/thread affinity headers that OpenCodeX's pool-affinity logic uses. The current debug view does not expose the raw cache-key value, which is why only presence/absence should be added to diagnostics.6. Confounding activity after 15:06
The post-15:06 interval is observational rather than a clean single-variable experiment. It included:
The main conversation still produced zero and high cache reads during these periods. This activity must be considered a confounder, not treated as proof that Background Review or the plugin is the sole cause.
7. Related direct-vs-proxy evidence from closed issue #1527
The large-context collapse portion of #1527 was verified as fixed in OpenCodeX 2.41.0 and is now closed. Its remaining cache observation was explicitly redirected here:
cacheReadTokens=489972;cached_tokens=0;cached_input_tokens=0on every observed turn.This is a separate Cursor route, not the same Hermes reproduction, but it is useful cross-route evidence that direct and proxied paths can receive different cache treatment. It should not be treated as proof that Hermes is the root cause.
Current interpretation
The evidence supports the following narrower statement:
This does not yet prove whether the primary defect is:
prompt_cache_key→session_idsynthesis in the Chat Completions bridge;A decisive next test would be a matched same-task comparison with the account fixed (no pool rebinding), while recording redacted cache-key presence, session/thread-affinity presence, resolved route/cohort, and
cached_tokensfor every turn. No raw credentials or identifiers are needed.fix(tests): repair two dev regressions from the 260904 merge train (#…
rennzhang commentedon Sep 4, 2026
Adding a measured direct-versus-proxy pair from a second route, as requested on #1527.
This is not the Hermes Chat Completions path in the original report. It is Cursor-routed turns through OpenCodex 2.41.0 on the same account, compared with
cursor-agentdirect, inbound protocolresponses. Cursor Other Models quota was exhausted on this account, so the live pairs use Cursor Models SKUs (composer-2.5, Cursor Grok 4.6). The kimi pair was captured earlier the same day, before that quota ran out.Same ~409KB prompt (409309 bytes) unless noted. OpenCodex usage on this path is
estimated: trueand never reports cache fields other thancached_tokens: 0. Directcursor-agentreportscacheReadTokens.unmeasuredRootswas absent from everyrun-requestline below.Raw JSON for these runs (usage +
run-request/ checkpoint / stream-end, output text stripped): https://gist.github.com/rennzhang/44f835f8e87849553e11948719c99df7Composer 2.5 (Cursor Models)
direct
cursor-agent --model composer-2.5:OpenCodex
cursor/composer-2.5/v1/responsesturn 1 (HTTP 200, 51.1s):OpenCodex turn 2 (
previous_response_id, HTTP 200, 26.6s):Cursor Grok 4.6 (Cursor Models)
direct
cursor-agent --model cursor-grok-4.6-high:OpenCodex
cursor/grok-4.6turn 1 (HTTP 200, 71.0s, wirecursor-grok-4.6-xhigh):OpenCodex turn 2 (
previous_response_id, HTTP 200, 13.3s):kimi-k3 (from #1527, before Other Models quota ran out)
direct
cursor-agent --model kimi-k3-max:OpenCodex
cursor/kimi-k3turn 1 (HTTP 200, 29.0s):OpenCodex turn 2 (
previous_response_id, HTTP 200, 6.7s):On the same account, real Codex app Cursor-routed turns also showed
cached_input_tokens=0every turn.Direct Cursor reports cache hits on Composer, Cursor Grok, and kimi. The OpenCodex Cursor adapter reports
cached_tokens=0on all three, including a Grok follow-up that used checkpoint continuation. This is a stable miss on the Cursor adapter / Responses path, not the intermittent Hermes Chat Completions zero-then-recover pattern in the original report.lidge-jun commentedon Sep 4, 2026
Investigated during the 260904 backlog review. The asymmetry you suspected is real, and I can point at it — but I am not shipping the obvious fix, and I want to be explicit about why.
Confirmed.
src/chat/inbound.ts:316copies a stringprompt_cache_keyinto the Responses body unchanged, so the key is not lost in translation.session_idandsession-idare both inFORWARD_HEADERS(src/adapters/openai-responses.ts:36-44), but the Chat bridge only forwards headers the caller already sent — there is no occurrence ofsession_idanywhere insrc/server/chat-completions.ts, so nothing synthesizes one from the body key.The Claude bridge does exactly that synthesis, and its comment records the measurement this issue is circling (
src/server/claude-messages.ts:757-765):So a body-only
prompt_cache_keyproducingcached_tokens: 0is a measured result on the Claude path, and Hermes is on the path that never got the fix.Why I am not simply mirroring it. Look at the first condition:
cacheKeySource === "metadata". Claude classifies its key provenance insrc/claude/inbound.ts—"metadata"for a real per-session key,"system"for a hash shared across conversations — and synthesizes only for the per-session one, because a sharedsession_idhas unproven backend semantics. The Chat bridge has no equivalent classification. A caller'sprompt_cache_keyis an opaque string, and key shape is not a reliable classifier.Mirroring the synthesis unconditionally would bind every caller sharing a cohort key onto one upstream session. That is a worse bug than the one being fixed, and it would be invisible in exactly the intermittent way this issue is already hard to pin down.
What that means for this issue. The fix needs a positive per-session provenance signal on the Chat request path first — either a header Hermes can set, or a documented convention about what a per-session key looks like on this route. That is a protocol decision, not an implementation detail, so I am marking it for a maintainer call rather than guessing.
Two things that would help meanwhile:
session_idheader directly? It is already inFORWARD_HEADERS, so it would be forwarded today with no proxy change. If cache hits stabilize, that confirms the diagnosis outright.x-codex-parent-thread-idor thesession-id+thread-idpair (src/codex/auth-context.ts:80-98), not on underscoresession_idand not on the body key. Requests without those can be reassigned across accounts, which changes the upstream cache cohort regardless of the header. Test the two separately or they will confound each other.Related, and worth cross-reading: @rennzhang reported the same shape on a Cursor route in #1527 —
cacheReadTokens=489972direct versuscached_tokens=0through the proxy on an identical prompt. A second provider showing it suggests a shared bridge cause rather than anything Hermes-specific.Vivamisu commentedon Sep 4, 2026
Vivamisu commentedon Sep 4, 2026
Fresh maintainer-directed evidence: explicit session affinity vs. body-only cache keys
I ran the requested follow-up experiments without changing the production Hermes configuration, provider plugin, or Gateway. OpenCodeX stayed in
poolmode. The sampled requests were ordinary Hermes CLI traffic ongpt-5.6-sol; all rows below were HTTP 200,inboundProtocol=chat,usageStatus=reported, andfastOutcome=not-requested. Fast/Codex traffic and Background Review were excluded. The experiments used temporaryHERMES_HOMEcopies for test-only headers.The effective sampled account stayed the same anonymized account label (Account P57); the other stored pool accounts were paused, and the native main profile was not selected by these sampled requests.
1. Hermes long-task A/B: underscore
session_idcacheReadInputTokens=0; weighted cache-read rate 11.04% (input=922,577,cache=101,888).session_idheader: 9 main-loop requests, 1 zero (the cold first request); weighted cache-read rate 71.05% (input=443,927,cache=315,392).OpenCodeX affinity debug showed the
session_idheader present inbound and outbound during the second block. All diagnostics remainedauthKind=pool,accountMode=pool,fixedAccount=false,credentialSubstituted=true, and HTTP 200; turn metadata/state remained absent.2. Pool-affinity header A/B/C
Three sequential fresh CLI tasks were run with the same pool/account conditions:
input=605,407,cache=196,608).session-id+thread-id: 8 requests, 1 zero; weighted rate 69.48% (input=342,664,cache=238,080).x-codex-parent-thread-id: 8 requests, 7 zero; weighted rate 2.91% (input=509,787,cache=14,848).The provider-debug records confirmed that OpenCodeX received and forwarded the respective header forms. Every record remained pool mode, non-fixed routing, credential substitution enabled, and HTTP 200.
This shows that “any affinity header fixes the problem” is not supported: the complete
session-id+thread-idpair helped in this sample, while the parent-thread header alone did not.3. Direct raw
/v1/chat/completionsmatrixTo remove Hermes tool-loop growth from the comparison, I ran 20 direct Chat Completions requests: four variants, five repeated identical requests per variant. Each variant used a fresh deterministic prompt prefix; every request returned HTTP 200 with the same measured input size (
27,027tokens).prompt_cache_keyonly: 4/5 zero; weighted rate 19.51%.session_idheader only: 1/5 zero (cold first request); weighted rate 78.05%.session_id: 1/5 zero (cold first request); weighted rate 78.05%.The direct matrix reproduces the important asymmetry independently of Hermes: a body-level
prompt_cache_keyis accepted but does not reliably warm the route, while a stable underscoresession_idproduces a warm repeated sequence. Adding the body key on top of the header did not improve this controlled sample further.4. Raw evidence record
The row-level redacted evidence for all experiments is included below. It contains no credentials, Authorization headers, request IDs, account IDs, or raw affinity-header values.
Raw redacted evidence JSON
{ "issue": "https://github.com/lidge-jun/opencodex/issues/3433", "collected_at": "2026-09-04", "privacy": { "credentials": "omitted", "authorization": "omitted", "request_ids": "omitted", "account_ids": "omitted", "account_label": "Account P57", "raw_header_values": "omitted" }, "controls": { "opencodex_account_mode": "pool", "hermes_model": "gpt-5.6-sol", "hermes_transport": "chat_completions", "effective_account": "one selected unpaused pool account", "http_status": 200, "inbound_protocol": "chat", "usage_status": "reported", "fast_outcome": "not-requested", "excluded": ["Fast/Codex traffic", "Background Review"], "production_config_changed": false, "test_headers_injected_only_into_temporary_HERMES_HOME": true }, "hermes_prompt_affinity_axis": { "no_affinity_header": { "rows": [ {"input_tokens": 11762, "cache_read_input_tokens": 0}, {"input_tokens": 15793, "cache_read_input_tokens": 0}, {"input_tokens": 19655, "cache_read_input_tokens": 0}, {"input_tokens": 26814, "cache_read_input_tokens": 0}, {"input_tokens": 32044, "cache_read_input_tokens": 0}, {"input_tokens": 68906, "cache_read_input_tokens": 0}, {"input_tokens": 69250, "cache_read_input_tokens": 0}, {"input_tokens": 69645, "cache_read_input_tokens": 0}, {"input_tokens": 109076, "cache_read_input_tokens": 14848}, {"input_tokens": 114346, "cache_read_input_tokens": 0}, {"input_tokens": 122780, "cache_read_input_tokens": 0}, {"input_tokens": 129098, "cache_read_input_tokens": 68096}, {"input_tokens": 133408, "cache_read_input_tokens": 18944} ], "summary": { "requests": 13, "zero_cache_requests": 10, "input_tokens": 922577, "cache_read_input_tokens": 101888, "weighted_cache_rate_pct": 11.04 } }, "constant_session_id": { "rows": [ {"input_tokens": 11759, "cache_read_input_tokens": 0}, {"input_tokens": 15788, "cache_read_input_tokens": 10752}, {"input_tokens": 19677, "cache_read_input_tokens": 14848}, {"input_tokens": 28010, "cache_read_input_tokens": 18944}, {"input_tokens": 49251, "cache_read_input_tokens": 27136}, {"input_tokens": 64532, "cache_read_input_tokens": 48640}, {"input_tokens": 65904, "cache_read_input_tokens": 64000}, {"input_tokens": 66543, "cache_read_input_tokens": 65024}, {"input_tokens": 122463, "cache_read_input_tokens": 66048} ], "summary": { "requests": 9, "zero_cache_requests": 1, "input_tokens": 443927, "cache_read_input_tokens": 315392, "weighted_cache_rate_pct": 71.05 } } }, "pool_affinity_axis": { "no_pool_affinity_header": { "rows": [ {"input_tokens": 11700, "cache_read_input_tokens": 0}, {"input_tokens": 18126, "cache_read_input_tokens": 0}, {"input_tokens": 26972, "cache_read_input_tokens": 0}, {"input_tokens": 35113, "cache_read_input_tokens": 0}, {"input_tokens": 68182, "cache_read_input_tokens": 0}, {"input_tokens": 119042, "cache_read_input_tokens": 0}, {"input_tokens": 162838, "cache_read_input_tokens": 34304}, {"input_tokens": 163434, "cache_read_input_tokens": 162304} ], "summary": { "requests": 8, "zero_cache_requests": 6, "input_tokens": 605407, "cache_read_input_tokens": 196608, "weighted_cache_rate_pct": 32.48 } }, "constant_session_id_and_thread_id": { "rows": [ {"input_tokens": 11697, "cache_read_input_tokens": 0}, {"input_tokens": 15720, "cache_read_input_tokens": 10752}, {"input_tokens": 18109, "cache_read_input_tokens": 14848}, {"input_tokens": 22224, "cache_read_input_tokens": 17920}, {"input_tokens": 35432, "cache_read_input_tokens": 22016}, {"input_tokens": 44562, "cache_read_input_tokens": 35328}, {"input_tokens": 94669, "cache_read_input_tokens": 43520}, {"input_tokens": 100251, "cache_read_input_tokens": 93696} ], "summary": { "requests": 8, "zero_cache_requests": 1, "input_tokens": 342664, "cache_read_input_tokens": 238080, "weighted_cache_rate_pct": 69.48 } }, "constant_x_codex_parent_thread_id_only": { "rows": [ {"input_tokens": 11697, "cache_read_input_tokens": 0}, {"input_tokens": 15720, "cache_read_input_tokens": 0}, {"input_tokens": 21039, "cache_read_input_tokens": 0}, {"input_tokens": 23694, "cache_read_input_tokens": 0}, {"input_tokens": 50062, "cache_read_input_tokens": 14848}, {"input_tokens": 84365, "cache_read_input_tokens": 0}, {"input_tokens": 140869, "cache_read_input_tokens": 0}, {"input_tokens": 162341, "cache_read_input_tokens": 0} ], "summary": { "requests": 8, "zero_cache_requests": 7, "input_tokens": 509787, "cache_read_input_tokens": 14848, "weighted_cache_rate_pct": 2.91 } } }, "direct_chat_completions_matrix": { "request_shape": { "endpoint": "/v1/chat/completions", "model": "gpt-5.6-sol", "requests_per_variant": 5, "input_tokens_per_request": 27027, "http_status": 200, "prompt_cache_key_values": "redacted; presence encoded by variant", "session_id_values": "redacted; presence encoded by variant" }, "body_prompt_cache_key_only": { "prompt_cache_key": "present", "session_id": "absent", "cache_read_input_tokens": [0, 0, 0, 0, 26368], "summary": { "requests": 5, "zero_cache_requests": 4, "input_tokens": 135135, "cache_read_input_tokens": 26368, "weighted_cache_rate_pct": 19.51 } }, "neither_key_nor_session_id": { "prompt_cache_key": "absent", "session_id": "absent", "cache_read_input_tokens": [0, 0, 0, 0, 0], "summary": { "requests": 5, "zero_cache_requests": 5, "input_tokens": 135135, "cache_read_input_tokens": 0, "weighted_cache_rate_pct": 0.0 } }, "session_id_only": { "prompt_cache_key": "absent", "session_id": "present", "cache_read_input_tokens": [0, 26368, 26368, 26368, 26368], "summary": { "requests": 5, "zero_cache_requests": 1, "input_tokens": 135135, "cache_read_input_tokens": 105472, "weighted_cache_rate_pct": 78.05 } }, "body_prompt_cache_key_and_session_id": { "prompt_cache_key": "present", "session_id": "present", "cache_read_input_tokens": [0, 26368, 26368, 26368, 26368], "summary": { "requests": 5, "zero_cache_requests": 1, "input_tokens": 135135, "cache_read_input_tokens": 105472, "weighted_cache_rate_pct": 78.05 } } }, "interpretation_boundary": [ "These are live sequential samples, not randomized repeated trials.", "The Hermes tool-loop request bodies were not bit-identical across every long-task run.", "The direct matrix used identical repeated requests within each variant.", "The data supports stable per-session affinity as a strong candidate; it does not prove a permanent fix or expose the hidden upstream cache cohort.", "No production Hermes configuration, plugin, or Gateway restart was performed for these experiments." ] }Narrowed interpretation
These results support the concern raised in this issue:
session_idfrom every body key, because shared/cohort keys need a positive per-session provenance signal first.session-id+thread-idappears to be the more relevant pair for OpenCodeX pool binding, while underscoresession_idappears relevant to the downstream prompt-cache route.This was a sequential live experiment rather than randomized repeated trials, and the Hermes tool outputs/context sizes were not bit-identical across all runs. I am not claiming a permanent fix yet, and I did not change the production configuration. The current evidence does justify prioritizing a maintainer-approved way for Hermes to send stable per-session affinity, plus redacted cache-key/affinity/cohort diagnostics on the same request record.
Vivamisu commentedon Sep 4, 2026
Temporary Hermes-side workaround and post-restart evidence
I tested a local Hermes provider-plugin workaround based on the cache behavior above. This is a client-side workaround, not an OpenCodeX upstream fix.
Design
For GPT/Codex models, the plugin:
supports_prompt_cache_key=True, so Hermes emits its existing rotation-stable body-levelprompt_cache_key;The plugin does not derive
session_idfrom the body cache key. It also does not sendx-codex-parent-thread-idalone, because that header did not improve the isolated test. The obsoletesupports_codex_affinity_headersconstructor field was removed so the profile loads on the current Hermes ProviderProfile API.Why this design
The matched experiments showed:
A separate direct
/v1/chat/completionsmatrix showed:This supports keeping both layers: Hermes' body cache key plus explicit per-session affinity. The plugin intentionally does not claim that either layer alone is a permanent upstream fix.
Gateway-restart verification
The Gateway was restarted at 2026-09-04 21:22:24 +0800. After restart:
For the ordinary Hermes main route (
gpt-5.6-sol, Chat Completions, Fast/Codex traffic excluded), using the same effective selected pool account throughout:The first request was:
The subsequent sequence stayed non-zero, including:
The post-restart OpenCodeX debug records remained in pool mode and observed the new affinity-header form on the relevant requests. The same effective selected pool account was used throughout this post-restart sample; no account transition was observed. Raw header values, credentials, account IDs, and request IDs are intentionally omitted.
Scope
This is a sequential live observation, not a randomized plugin-only A/B test. The evidence shows that the workaround restored stable high cache reuse in this post-restart run; it does not prove the OpenCodeX Chat → Responses cache/cohort behavior is permanently fixed. Background Review and auxiliary model traffic are separate routes and are excluded from the main-route figures.
Vivamisu commentedon Sep 4, 2026
Follow-up: split cache affinity from account-pool affinity
I changed the temporary Hermes provider plugin after finding that the first version coupled two different routing concerns.
Why the plugin changed
The first workaround sent the same stable conversation-derived value in all three headers:
That improved prompt-cache reuse, but OpenCodeX uses the hyphenated
session-id+thread-idpair (orx-codex-parent-thread-id) as account-pool affinity. Consequently, an ongoing Hermes conversation could remain bound to one account and prevent the intended round-robin assignment of new/unbound work.The revised plugin separates these concerns:
The
session_idvalue is an opaque SHA-256-derived token based on Hermes' logical conversation/cache lineage. It does not expose the original session identifier and remains stable across physical session rotation. The plugin is registered only for theopencodexprovider and no longer replaces Hermes' genericcustomprofile.The intended behavior is account-segment caching: round-robin may send new/unbound work to either pool account; each account may have an initial cold request, then reuse its own upstream cache segment.
Controlled verification
Controls:
poolround-robin195%to99%for this experiment because one account's observed long-window usage was96%prompt_cache_keyplus only the underscoresession_idWith the previous
95%threshold, all observed test requests selected one account. After raising the threshold to99%, both accounts appeared immediately in the new/unbound-session sample:This demonstrates that the revised plugin no longer prevents pool rotation. It also shows an implementation nuance: although the round-robin assignment itself does not use the proactive-switch threshold, that threshold can still affect which accounts enter the eligible candidate set before round-robin runs. Raising it above the account's observed
96%long-window usage allowed both accounts to participate.Cache observations in the bounded sample:
The zeros include first requests of independent test sessions and therefore should not all be interpreted as mid-session cache loss. The result supports the design goal only narrowly: both accounts rotate, and an account can return to high cache reuse after a cold request. It does not establish a permanent fix or guarantee that every account/session segment remains warm.
Tests
This remains a client-side workaround. No OpenCodeX source code was changed.
Vivamisu commentedon Sep 5, 2026
New controlled follow-up: no Hermes plugin,
/newversus explicit/modelI disabled the local Hermes OpenCodeX provider plugin, restarted the Gateway, and ran two sequential Telegram tests. The plugin was not loaded during these samples. This separates Hermes provider identity from the plugin's cache-key capability.
Test A — fresh
/newAfter
/new, the new Telegram session resolved as:The subsequent request sequence showed the same unstable pattern seen in the original report:
The important point is that the configured default is
opencodex, but the fresh session's live Hermes agent was created withprovider=customand the same OpenCodeX base URL. This is a separate Hermes-side default-provider identity issue.Test B — explicit
/modelselectionIn the same post-restart environment, I explicitly selected an OpenCodeX model with
/model. Hermes logged the switch as:So explicit model selection preserves
provider=opencodexeven without the plugin. However, this did not produce stable cache reuse. The complete Astra usage sequence was:All listed rows were successful HTTP 200 requests from the same reported pool account label;
conversationIdwas absent in the OpenCodeX usage records. This remains consistent with the session-affinity/cohort problem under investigation, not with a permanently bad account.Reasoning-parameter observation
For the Astra requests, Hermes' local model-switch path resolved the requested reasoning configuration as enabled/high, but OpenCodeX usage reported:
for every Astra row. The rows also showed
reasoningOutputTokensof zero on most requests. This suggests that Hermes' Chat Completions request is not transmitting the reasoning effort in the top-level field currently recognized by OpenCodeX (for example,reasoning_effort). This is separate from cache affinity, but it is relevant to interpreting these tests: the Astra sample was not confirmed to run at high reasoning effort.The OpenCodeX usage rows also showed
configuredServiceTier=prioritywhileresponseServiceTier=default; this proves the configured/requested tier reached the relay, not that upstream Priority execution was actually applied.Narrow conclusion
/newexposes a Hermes default-runtime issue: configuredopencodexbecomes livecustom./modelcan preserve liveopencodexwithout the plugin.requestedEffort=null).prompt_cache_keyhandling, but it did not explain the explicit/modelprovider identity and it did not supply the missing upstream session/cohort guarantee.These observations do not claim that OpenCodeX is solely responsible for the Hermes
customidentity issue. They suggest tracking it as a separate Hermes routing/ProviderProfile bug while continuing the OpenCodeX session-affinity investigation here.Core interpretation: why both
customandopencodexcan missThe combined evidence points to a cache-affinity problem rather than an inability of either label to cache. Reuse requires three things to remain aligned:
customandopencodexuse the same OpenCodeX base URL in this setup, but they are different Hermes provider identities. The fresh-session default path can resolve configuredopencodexto a runtimecustomprofile, which omits the OpenCodeX-specific cache-key capability. Explicit/modelpreserves runtimeopencodexeven without the plugin, but provider identity alone does not establish upstream session or account-pool affinity.Without a stable session/affinity signal, OpenCodeX's pool route may place neighboring requests into different upstream cache cohorts. That explains the observed
0% → high → partial → 0%pattern even when the model, reported account label, and HTTP status remain unchanged. Hermes tool results, skill loading, background review, compression, and model switches can also change the request prefix; these explain some partial hits but do not by themselves explain repeated successful zero-cache rows followed by high hits.The earlier controlled tests support this separation: body
prompt_cache_keyalone was much less reliable than a stablesession_id, whilesession-id+thread-idaffected pool binding as a separate axis. Therefore the Hermescustomidentity issue and the OpenCodeX session-affinity/cohort issue should be tracked separately, while recognizing that the two can compound each other.Ingwannu commentedon Sep 5, 2026
Thanks for the new no-plugin comparison. It usefully separates the Hermes provider-identity behavior from OCX cache observations, but it does not yet establish an OCX cache-cohort defect.
Two limits matter for the next sample:
/newand explicit/modelsamples change both provider identity and model. Please compare the same model, tool set, and stable prompt prefix before attributing the difference tocustomversusopencodex.The current code keeps these concepts separate:
src/codex/auth-context.tsderives pool affinity from the hyphenated session/thread identifiers, whilesrc/server/chat-native.tsreads requested effort from the incoming top-levelreasoning_effort. Please provide a small redacted field-presence comparison for one cold/warm/cold sequence: same model, whetherprompt_cache_key, underscoresession_id, hyphenatedsession-id/thread-id, andreasoning_effortwere present. For identity values, report only whether each remained equal across the requests; do not post their raw values or request contents.The
requestedEffort: nullobservation is not proof that the model ran without reasoning, and configured priority versus returned default is not proof of Priority execution. Keeping the cache investigation open and the Hermes default-provider issue separate; no configuration/threshold change or guaranteed cache-hit workaround is recommended from these samples.Vivamisu commentedon Sep 5, 2026
Controlled follow-up requested above: matched provider paths with redacted inbound field presence
I reran the comparison requested by @Ingwannu using two isolated Hermes CLI environments and a privacy-bounded local capture placed immediately before OpenCodeX. The capture recorded only field presence plus a short one-way tag for identity fields; it did not retain request contents, raw identity values, credentials, authorization headers, account IDs, or local paths.
Controls
Both groups used:
customandopencodexpointed to the same capture/relay endpoint. Each provider group used its own freshHERMES_HOMEand one continuing CLI session for the growth sequence. A final fresh-session request was used only to verify the cold boundary.Redacted inbound field comparison
For every actual streamed request in both provider groups, the capture observed:
Because every identity field above was absent, equality across turns is not applicable; there were no raw values to compare. The non-streaming model/preflight probes were excluded from this field summary and from the cache sequences below.
Same-model normal-growth cache sequences
custom:This contains a warm-to-cold transition in one continuing session:
57,856 → 0, followed later by another partial warm result. The fresh session was cold as expected.opencodex:This is the requested cold/warm/cold shape within one continuing session:
The matched inbound capture remained unchanged across that sequence: no body
prompt_cache_key, no underscoresession_id, no hyphenatedsession-id/thread-id, and top-levelreasoning_effort=mediumpresent on each actual request.Narrow interpretation
This sample fixes the earlier comparison problem: provider identity, model, reasoning effort, tool set, effective account, transport, and growth method were controlled. It also directly establishes the inbound field-presence state rather than inferring it from usage output.
The result does not prove that OpenCodeX changed an upstream cache cohort. It establishes only that:
reasoning_effort=mediumon the actual requests;No production Hermes configuration, Gateway setting, OpenCodeX threshold, or account mode was changed for this test.
Ingwannu commentedon Sep 6, 2026
This answers the field-presence comparison I requested—thank you. The earlier model/effort/provider-label confound is substantially narrowed, and the top-level
reasoning_effort=mediumobservation supersedes the earlier inference from a null usage field.One actionable boundary is now clearer: in both groups the client supplied none of the measured conversation/cache identifiers. In current
dev, the native Responses adapter forwards an incomingsession_id, while the explicitprompt_cache_key→session_idsynthesis is scoped to metadata-backed Claude inbound requests (src/server/claude-messages.ts); that is not the Chat Completions path in this experiment. These samples therefore do not show OCX dropping an identifier that Hermes sent.The next discriminating comparison would keep the same controlled request construction and add a genuinely per-conversation client identifier, stable within a session and different for a fresh session, then verify its preservation at the OCX forward boundary. A global key or one derived solely from a shared prompt prefix would conflate independent conversations and should not be used as a proxy-side workaround. This is an identity/transport check, not a guarantee of upstream cache hits or a reason to change pool thresholds.
Keeping the issue open. The warm-to-cold behavior is observed, but attributing it to an OCX cohort change still needs outbound evidence; the new sample is useful precisely because it rules out several earlier explanations without overclaiming a fix.
test(chat): cover conversation identity at native outbound boundary
test(chat): cover conversation identity at native outbound boundary
Merge pull request #4365 from lidge-jun/codex/260912-60plus-cache-hermes
fix(tests): repair two dev regressions from the 260904 merge train (l…
test(chat): cover conversation identity at native outbound boundary
Merge pull request lidge-jun#4365 from lidge-jun/codex/260912-60plus-…
lidge-jun commentedon Sep 27, 2026
Closing with the boundary this thread settled on now in place on
dev:session_idand thesession-id/thread-idpair reach the upstream headers,prompt_cache_keypasses through the body, and a shared cohort key is never turned into a session. This landed in test(chat): verify native outbound conversation identity (#3433) #4365 (db7062c37a), is pinned bytests/responses/chat-conversation-affinity.test.ts, and is recorded instructure/data-planes/inbound-compat.md.session_affinity_header: "session-id", fix(desktop,gui): bundle L5 — macOS reopen, hidden-window polling, Hermes affinity, web-search Off, quota bar #5742,dd7cb695a9). New integrations enable it and stale blocks are adopted, whichtests/clients/integrations-hermes-affinity.test.tscovers.Both suites pass on current
dev(26 tests). If zero-cache streaks still show up with a Hermes config written by the current integration, please open a new issue with the redacted outbound header presence for one cold turn, so it can be compared against this contract.