[Bug][Windows]: synchronous PowerShell app-server enumeration blocks /healthz and proxy traffic on v2 requests

Client or integration

Codex App

Area

Proxy and routing / Platform (Windows)

Summary

On Windows, OpenCodex performs synchronous PowerShell/CIM process enumeration on the proxy's main Bun event loop while building multi-agent v2 guidance.

The hot request path is:

buildMultiAgentGuidance() handles a v2 surface.
It awaits defaultCollectCatalogState().
That calls the synchronous collectCodexAppServerCatalogState().
listWindowsSnapshots() runs execFileSync(powershell.exe, ..., timeout: 8_000).

Relevant source:

On the affected machine, one enumeration blocks the Bun event loop for 3.8-5.1 seconds. During that interval, even loopback /healthz cannot be serviced. Under repeated v2 requests, these pauses can queue and make the proxy appear permanently hung: the Bun process remains alive and owns port 10100, but health probes time out and connections accumulate in CLOSE_WAIT.

I expected catalog-state discovery to be asynchronous, deduplicated/cached, or outside the request hot path so a slow CIM query cannot block unrelated proxy traffic and health checks.

This remains present in the latest stable package, 2.22.0, and in current main.

Reproduction

With the repository dependencies available on Windows, run the following with Bun:

const { listWindowsSnapshots } =
  await import("./src/codex/app-server-processes.ts");

for (let i = 1; i <= 5; i++) {
  const scheduled = performance.now();
  const timer = new Promise<number>(resolve =>
    setTimeout(() => resolve(performance.now() - scheduled), 10)
  );

  const started = performance.now();
  let count: number | null = null;
  let error: string | null = null;
  try {
    count = listWindowsSnapshots().length;
  } catch (e) {
    error = e instanceof Error ? e.message : String(e);
  }

  const callMs = performance.now() - started;
  const timerMs = await timer;
  console.log({ run: i, callMs, timerMs, count, error });
}

A 10 ms timer is delayed by essentially the full synchronous call duration:

Run	`listWindowsSnapshots`	10 ms timer fired after	Result
1	3812 ms	3813 ms	3 processes
2	3937 ms	3943 ms	3 processes
3	4080 ms	4082 ms	3 processes
4	5072 ms	5075 ms	`windows_enum_incomplete`
5	4402 ms	4406 ms	3 processes

The timer evidence makes the event-loop blocking deterministic without requiring an upstream provider request.

For the live proxy symptom:

Leave multiAgentGuidanceEnabled at its effective default (true).
Use Codex App with the multi-agent v2 surface and issue repeated turns/tool continuations.
Probe http://127.0.0.1:10100/healthz concurrently.
Observe a PowerShell child under the OpenCodex Bun process running the Get-CimInstance Win32_Process / GetOwner script.
Health requests intermittently take several seconds or time out despite the listener and Bun PID remaining alive.
Setting multiAgentGuidanceEnabled: false removes this request-path enumeration. In the observed session, two post-change probe batches passed 45/45 and no enumeration child remained.

A live stalled sample before the workaround showed:

/healthz: 5-second timeout, zero response bytes
Bun CPU increase over 2 seconds: 0.016 s (waiting, not spinning)
Bun private memory: about 1.69 GiB
listener still active on port 10100
multiple CLOSE_WAIT connections
a PowerShell child running the catalog process enumeration

These are observations from one affected installation; the deterministic timer reproduction above is the primary evidence.

Root cause

The async wrapper does not make the synchronous collector non-blocking. execFileSync prevents Bun from advancing timers, accepting/servicing health requests, or processing other proxy work until PowerShell exits or reaches the 8-second timeout.

The enumeration is especially expensive because it walks Win32_Process and calls GetOwner for candidates. Issue #1354 already measured multi-second Windows enumeration and fixed a correctness race in the same collector, but did not remove it from the event loop or request hot path.

This is analogous to #612 / #645, where synchronous icacls work blocked /healthz until that high-frequency path was made asynchronous.

Suggested safe direction

Use asynchronous process creation for Windows enumeration, or move it to a worker so the proxy event loop remains responsive.
Deduplicate concurrent discovery with a single in-flight operation.
Cache the last result with a short TTL / stale-while-revalidate policy. Catalog-state guidance is advisory; serving a recent/unknown state is safer than blocking all proxy traffic.
Add a regression test that schedules a short timer or health request while the process-enumeration seam is deliberately slow and verifies it is serviced before enumeration completes.
Optionally narrow the CIM query before owner lookups, but query optimization alone would not remove the event-loop blocking contract.

Workaround

{
  "multiAgentGuidanceEnabled": false
}

This disables OpenCodex's additional multi-agent guidance/roster injection, not Codex's built-in collaboration tools.

Version

OpenCodex 2.22.0, bundled Bun 1.3.14, Codex CLI 0.147.0.

Operating system

Windows 11 Pro, version 10.0.26200 (build 26200).

Provider and model

Not provider-specific. The blocking occurs before provider work while building v2 guidance.

Logs or error output

GET /healthz -> timeout after 5.0s, 0 bytes
PowerShell child -> Get-CimInstance Win32_Process ... Invoke-CimMethod GetOwner
listWindowsSnapshots() -> 3.8-5.1s synchronous wait

Screenshots and supporting files

Not required; the source links, deterministic timer reproduction, timings, and live probe observations are text-reproducible.

Redacted configuration

{
  "multiAgentGuidanceEnabled": true
}

Related issues

[Bug]: transient code-mode-host process makes catalog staleness check return "unknown", emitting a false "catalog changed after Codex started" restart prompt (2.11.1, Windows) #1354: correctness/race problem in the same Windows catalog-state collector
[Bug]: synchronous Windows ACL hardening stalls the proxy on response-state writes #612 / fix(windows): async ACL harden for response-state writes (#612) #645: precedent for moving a synchronous Windows child-process wait off the proxy event loop

Checks

I searched existing issues and documentation.
I removed secrets, tokens, account details, request credentials, and personal data.

🔗 Related PRs

#306 - feat(windows): add tray controls and restart-safety diagnostics [merged]
#359 - fix(codex): stable Codex runtime selection for sync/clamp (#297) [merged]
#576 - fix(codex): warn about stale Codex app-servers after a catalog write [merged]
#770 - fix: research-batch reliability and security-boundary issues [closed]
#801 - test(windows): retry one transient CIM enumeration timeout [merged]
#868 - fix(windows): settle scheduler registration verification [open]
#1154 - fix(probe): walk Windows service definition chains (Task Scheduler + WinSW) for ownership [merged]
#1279 - fix(windows): eliminate console windows from proxy-internal identity & process lookups [open]
#1347 - fix(windows): stop console popups from proxy-internal PowerShell lookups (#1278) [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

Confirmed on current dev: the v2 guidance path still awaits collectCodexAppServerCatalogState(), which reaches synchronous Windows PowerShell/CIM enumeration. The async caller does not prevent that child-process wait from blocking Bun's event loop.

I am fixing this with a focused boundary:

keep the existing synchronous collector for explicit CLI/service operations that rely on it;
add a non-blocking Windows catalog-state collector for the request path;
deduplicate concurrent refreshes and serve a short-lived last-known state so repeated v2 turns do not launch duplicate CIM walks;
fail advisory guidance closed/silent on collection failure rather than blocking unrelated proxy and /healthz traffic;
add a regression where a deliberately slow enumeration is in flight while the event loop remains responsive.

I will keep the change scoped to catalog-state guidance and add the reason/tradeoff to the relevant architecture documentation.

Implemented in #1876.

The PR keeps the existing synchronous lifecycle collector, moves only the Windows v2 request path to async trusted-System32 PowerShell, deduplicates concurrent CIM probes, caches the advisory result briefly, and generation-guards invalidation so an older in-flight probe cannot restore stale state after a catalog write. Focused tests, typecheck, privacy scan, docs build, and diff check are green; the full-suite baseline exceptions are documented in the PR.

Because this is my own PR, I will not self-approve or self-merge it. It now needs exact-head cross-platform CI and another maintainer review.

Status update — not closing this, because the part that is fixed is not the part
you reported.

#1925 (merge aa9df919a) landed a related correctness fix on dev: the top-level
Get-CimInstance Win32_Process ran under $ErrorActionPreference='SilentlyContinue'
outside the per-process try/catch, so a failure of the query itself produced no
output at all — indistinguishable from a healthy machine with no Codex process running.
The collector reported not_running for a process list it had never read. It now emits
the incomplete-enumeration sentinel and surfaces as unknown, and a state cached as
unknown expires in 250ms rather than the uniform 5s.

That is a real defect found while reviewing the fix for this issue, but it is a
correctness bug, not a latency one. Your report is about execFileSync blocking the
Bun event loop for 3.8–5.1s so that even loopback /healthz cannot be serviced, and
that call is still synchronous on the v2 guidance path. Nothing about the sentinel
change makes it faster.

The fix for the blocking is #1876, which moves discovery off the request event loop. It
is still open; its first review blocker was the fail-open described above, which is now
resolved independently, so what remains is a rebase onto current dev and a Windows
suite on the exact head.

Closing this on the strength of #1925 would have been the tidier-looking outcome and
the wrong one — the proxy would still hang on the machine you filed from.

docs(devlog): adopt close-on-dev-merge for this run only

리뷰 · 우선순위 72 / 80

Windows에서 멀티에이전트 v2 요청을 만들 때, 프록시 메인 이벤트 루프가 동기 PowerShell/CIM 프로세스 열거를 기다리면서 /healthz와 다른 프록시 트래픽까지 멈춥니다. 이슈가 말하는 증상은 Bun 프로세스는 살아 있고 10100 포트도 잡고 있는데, 헬스 프로브가 타임아웃되고 연결이 CLOSE_WAIT에 쌓여 프록시가 죽은 것처럼 보이는 것입니다.

현재 dev의 핫패스는 이슈 설명과 맞습니다. src/server/responses/collaboration.ts의 multiAgentGuidanceText()가 v2 surface이면 defaultCollectCatalogState()를 await하고, 그 함수가 collectCodexAppServerCatalogState()를 부릅니다. Windows에서는 src/codex/app-server-processes.ts의 listWindowsSnapshots()가 execFileSync(resolveTrustedWindowsPowerShellExe(), ..., { timeout: 8_000 })로 Get-CimInstance Win32_Process와 Invoke-CimMethod GetOwner를 돌립니다. async 래퍼가 있어도 execFileSync가 끝나는 동안 타이머, 헬스 요청, 다른 프록시 작업이 진행되지 않습니다.

원하는 결과는 카탈로그 상태 조회가 요청 핫패스에서 이벤트 루프를 막지 않는 것입니다. 가이던스는 조언용이라 최신 상태를 잠깐 못 받아도, 프록시 전체가 몇 초씩 멈추는 것보다 낫습니다. multiAgentGuidanceEnabled: false는 추가 가이던스만 끄고 Codex 자체 협업 도구는 남기므로, 당장 쓰는 우회로는 맞습니다. #1354는 같은 수집기의 레이스만 고쳤고, #612 / #645의 동기 icacls 선례처럼 이벤트 루프에서 빼는 작업은 아직입니다.

해결방안: Windows 열거를 비동기 spawn/worker로 옮기고, 동시에 들어오는 조회는 하나만 돌리며, 짧은 TTL 또는 stale-while-revalidate 캐시로 최근/unknown 상태를 쓰게 하는 방향이 이슈와 코드 모두와 맞습니다. 회귀 테스트는 열거 seam을 일부러 느리게 두고 짧은 타이머나 /healthz가 열거 끝나기 전에 응답하는지 보면 됩니다. CIM 쿼리를 더 좁히는 것은 도움이 되지만, 동기 대기를 없애지 않으면 계약은 그대로입니다.

이 댓글은 grok-bot이 작성했습니다

test(windows): give the async catalog enumeration a real oracle

test(windows): guard all three async wirings behind #1852

test(windows): make the #1852 oracles portable and timing-free

리뷰 · 종결

dev에 이미 머지됨. #1876이 Windows에서 카탈로그 스냅샷을 listWindowsSnapshotsAsync + 캐시로 빼서 /healthz/v2 요청을 안 막음. collectCodexAppServerCatalogStateForRequest() (src/server/responses/collaboration.ts). 동기 수집은 CLI/서비스만.

다음 배포 예정. 이 이슈는 닫음. #1925는 별개.

이 댓글은 grok-bot이 작성했습니다

Live verification on a native Windows installation

I rechecked this issue against the published OpenCodex package while investigating slow Codex responses.

OS: Windows 11 Pro
Package: @bitkyc08/opencodex 2.25.0
Runtime: native Windows, no WSL

Observed:

The Codex task experienced turns lasting several minutes before interruption.
A local health probe returned no response for more than six seconds while the OpenCodex service process remained present.
The service log contained repeated upstream connection resets and transient 503 retries against chatgpt.com.
The log also contained repeated Provider model discovery timeouts for several configured providers.

This reproduces the issue's key liveness symptom. The upstream transport retries appear to be a separate contributor that amplifies response latency.

Because #1876 is described as merged on dev, please verify whether its async catalog-state fix is included in the published 2.25.0 package. If not, please identify or publish the first release containing the fix. A useful regression test is to probe the local health endpoint while a deliberately slow Windows catalog enumeration is in flight.

fix(windows): repair a TOML path round trip and five shard-only test …

fix(windows): unblock the isolated fabric producer and unpin CI-scale…

fix(windows): repair a TOML path round trip and five shard-only test …

fix(windows): unblock the isolated fabric producer and unpin CI-scale…

fix(windows): contain the fabric producer and remove timeout-only sta…

fix(windows): unblock the isolated fabric producer and unpin CI-scale…

fix(windows): contain the fabric producer and remove timeout-only sta…

docs(devlog): adopt close-on-dev-merge for this run only

test(windows): give the async catalog enumeration a real oracle

test(windows): guard all three async wirings behind lidge-jun#1852

test(windows): make the lidge-jun#1852 oracles portable and timing-free

fix(windows): unblock the isolated fabric producer and unpin CI-scale…

fix(windows): contain the fabric producer and remove timeout-only sta…

fix(server): read Codex app-server state off the event loop for dashb…

Navigation Menu

Repository navigation

[Bug][Windows]: synchronous PowerShell app-server enumeration blocks /healthz and proxy traffic on v2 requests #1852

Client or integration

Area

Summary

Reproduction

Root cause

Suggested safe direction

Workaround

Version

Operating system

Provider and model

Logs or error output

Screenshots and supporting files

Redacted configuration

Related issues

Checks

리뷰 · 우선순위 72 / 80

리뷰 · 종결

Live verification on a native Windows installation

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Bug][Windows]: synchronous PowerShell app-server enumeration blocks /healthz and proxy traffic on v2 requests #1852

Description

Client or integration

Area

Summary

Reproduction

Root cause

Suggested safe direction

Workaround

Version

Operating system

Provider and model

Logs or error output

Screenshots and supporting files

Redacted configuration

Related issues

Checks

Activity

coderabbitai commented on Aug 16, 2026

Ingwannu commented on Aug 16, 2026

Ingwannu commented on Aug 17, 2026

lidge-jun commented on Aug 17, 2026

lidge-jun commented on Aug 19, 2026

리뷰 · 우선순위 72 / 80

lidge-jun commented on Aug 19, 2026

리뷰 · 종결

lmaonewhow commented on Aug 21, 2026

Live verification on a native Windows installation

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions