[Bug][Windows] Failed ocx update leaves a file-less package skeleton; service wrapper restart-loops 1,009 times with no integrity check or backoff

Summary

On Windows 11, an ocx update from 2.13.0 left the global package directory containing 107 directories and zero files.
Every launcher (ocx, ocx.cmd, ocx.ps1, opencodex, opencodex.cmd, opencodex.ps1) was removed from
%APPDATA%\npm, so no documented recovery command was reachable from a normal shell.

The Task Scheduler service wrapper then restarted the missing entry point every 5 seconds for 89 minutes —
1,009 consecutive child exited with code 3 events — until the package was manually rolled back.

This matches the partial-install behaviour described in #281. That issue was closed with the permanent-crash symptom
attributed to an external monitoring script terminating the restarted proxy. That attribution does not fit this case:
no external process was involved. The wrapper was relaunching a package that had no bin/ocx.mjs, no
src/cli/index.ts, and no bundled bun.exe to execute.

Environment

Item	Value
OS	Windows 11 Home, 10.0.26100 build 26100, AMD64
Node	v24.15.0
npm	11.19.0
opencodex (before/after rollback)	2.13.0
Bundled Bun	1.3.14
Service backend	Task Scheduler (`opencodex-proxy`), wrapper `opencodex-service.cmd`
Install type	npm global, `%APPDATA%\npm\node_modules\@bitkyc08\opencodex`

Evidence

1. The abandoned install contained directories only

The failed package was preserved before rollback. Recursive enumeration:

TotalDirs   : 107
TotalFiles  : 0

95 dependency folders were created under node_modules (@bufbuild, @hono,
@modelcontextprotocol, @napi-rs, @oven, bun, ...). The node_modules/bun folder
exists and is completely empty. Missing at the package root: package.json,
bin/ocx.mjs, src/cli/index.ts, node_modules/bun/bin/bun.exe.

So npm created the directory tree, then unpacked no file content and did not roll back.

2. Launchers were removed, recovery commands unreachable

%APPDATA%\npm\ocx.cmd and all sibling launchers were gone. What remained were npm's
rename-aside shims, which still pointed at the now-nonexistent target:

.ocx-cl2A1bun           .opencodex-lwm6eEVg
.ocx.cmd-RmK066DM       .opencodex.cmd-XPiSZ201
.ocx.ps1-AKozk9bb       .opencodex.ps1-p35htcAv

A stale scope directory was also left behind (same pattern as #281).

With no ocx on PATH, the documented remedies (ocx service repair, ocx doctor,
ocx update) cannot be run without manually reconstructing the Bun + CLI invocation.

3. Unbounded restart loop with no integrity check

service.log, filtered to this incident:

[16/08/2026 12:33:51.44] child exited with code 3; restarting in 5s
... 1,009 occurrences, one every ~5 seconds ...
[16/08/2026 14:02:50.11] child exited with code 3; restarting in 5s

First failure was 45 seconds after Windows boot (boot 12:33:06), so the broken package was
already on disk before the restart, and the loop began immediately at logon and never self-limited.

For contrast, code 3 appeared only 25 times in the entire preceding log history
(31/07 x1, 02/08 x13, 08/08 x11). This was not pre-existing noise.

The wrapper loop in opencodex-service.cmd is:

:loop
"%OCX_BUN%" "%OCX_CLI%" start --port 10100 >>"%OCX_SERVICE_LOG%" 2>&1
if %ERRORLEVEL% NEQ 0 (
  ping -n 6 127.0.0.1 >nul
  goto loop
)

There is no check that %OCX_BUN% or %OCX_CLI% exist, no failure counter, and no backoff.
A permanently unrecoverable condition is retried identically forever.

Reproduction

Windows, opencodex installed globally via npm and running under the Task Scheduler backend.
Run an update that triggers a global npm install while the service is live and Bun binaries are in use.
npm hits a locked-file error, leaves the package as a directory skeleton, and does not roll back.
Launchers disappear from %APPDATA%\npm; ocx is no longer on PATH.
The service wrapper loops indefinitely on the missing entry point.

Impact

All model routing through the proxy is down. Because the CLI is also gone, the failure is not
self-recoverable through the documented commands. Recovery here required restoring the previous
package directory and launcher shims manually, out of band.

Suggested fixes

Updater (relates to #281)

Stage the new version and swap atomically, so a failed unpack cannot replace a working install.
On failure, roll back to the previous package and restore launchers rather than leaving a skeleton.
Stop the service and release Bun file handles before the install step, not after.
Verify package.json, bin/ocx.mjs, src/cli/index.ts and the bundled runtime exist before declaring success.

Service wrapper (separate, arguably higher value)

Preflight the launcher and runtime paths before the first spawn; if either is missing, log one clear
"installation is incomplete, run X" line and stop rather than looping.
Add exponential backoff and a consecutive-failure ceiling.
Distinguish "crashed at runtime" from "cannot start at all" in the log.

A single existence check would have converted an 89-minute silent outage into one actionable log line.

Notes

Rolled back to 2.13.0; the proxy is healthy again and has since survived a restart
(status=protected, protection=service, rebootSafe=true).
The failed install directory has been preserved and can be inspected further if useful.
Happy to test a patched updater on this machine.

🔗 Related PRs

#306 - feat(windows): add tray controls and restart-safety diagnostics [merged]
#533 - fix(update): preserve proxy on npm cache failures [open]
#557 - fix(update): harden npm cache recovery preflight logs [open]
#803 - Codex/pr533 update recovery hardening [open]
#838 - fix(update): recover Windows dashboard proxy after self-update [merged]
#1647 - fix(tray): let the service wrapper exit 0 on an already-live proxy [open]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

Maintainer decision respected

A maintainer has reopened this issue. The automated closure has been deactivated.

Confirmed in current dev, with one important version distinction.

The current updater already stops the managed proxy before package replacement and performs registry-integrity preflight; those protections were added after the reported 2.13.0 build. However, two gaps remain:

npm can still leave an unusable global package tree on installer failure, and there is no complete transactional restore of both the package directory and npm launchers;
the current Task Scheduler wrapper still enters the same unconditional five-second loop and does not check whether its baked Bun or CLI entry exists.

I am taking the focused, independently safe wrapper slice first: verify both baked paths before the initial spawn and before every retry, log one actionable installation incomplete failure, and exit instead of retrying a condition that cannot recover in-process. Normal runtime crashes will retain the existing restart behavior. I will add source-level regressions and document why this is intentionally narrower than a transactional npm rollback.

I am not going to improvise a package-directory copy-back in this slice: restoring npm's package tree without the matching global shims and ownership/lock guarantees can make recovery less reliable. The remaining updater rollback boundary should be handled separately with a staged/verified install design.

The focused wrapper fix is now in #1877.

It prevents the exact 1,009-restart failure mode by checking the baked Bun and CLI paths before the first spawn and every retry. A missing path produces one actionable incomplete-install log entry and exits; a proxy that actually starts and later crashes keeps the existing restart behavior.

This is deliberately a partial fix: current dev already stops the proxy before replacement and performs integrity preflight, but transactional restoration of both an npm package tree and its global launchers still needs a separate staged-update design. #1849 should remain open until that updater boundary is resolved.

Split, and staying open. This report contains two defects with different fixes and
different closure conditions, so closing it on the shipped half would have hidden the one
that actually destroyed the install.

Half	Defect	Status
A	Service wrapper restart-loops against a missing install — the 1,009 × `child exited with code 3`	Fixed by #1877, released in v2.24.0 (verified by `git tag --contains 01b212579`)
B	`ocx update` deletes the working install and leaves an unbootable skeleton with no rollback	Open — now tracked in #1942

#1877 stopped the wrapper from hammering a package that is not there. It does nothing about
how the package came to be missing: the update path delegates to npm install -g, which
removes the existing install before the replacement is verified, so a mid-flight failure
leaves 107 directories, zero files, and no launcher to run a recovery command with.

#1942 carries the transactional shape (stage → verify → snapshot → atomic switch → health
probe → restore on failure), the ordering constraint that makes it work, and the
kill-at-every-boundary matrix it needs.

Keeping this issue open as the umbrella so the original evidence — the 107/0 enumeration,
the missing launchers, the 89-minute restart window — stays attached to the story rather
than being split away from it.

On the #281 comparison in the report: agreed that the earlier attribution does not fit here.
There was no external process, and a wrapper relaunching a package with no bin/ocx.mjs and
no bundled bun.exe is a different failure from the one #281 closed on.

One correction to my comment above: I wrote the npm install -g sentence in a way that reads
as present-tense causation for the 107-directories/zero-files enumeration. That evidence is
from v2.13.0, and protections have been added since — a prior comment on this issue already
qualified that.

The conclusion is unchanged, because the primitive is: npm install -g is still
destructive-in-place, and src/update/ has no staging, snapshot, or rollback anywhere on the
path. Verification does exist (checkUpdatePackageIntegrity before the proxy stops,
isRealBunBinary, and an identity-checked /healthz probe after restart) — but it is
detect-only and runs after the old install is already gone, so a failed probe has nothing
left to restore. That is the shape #1942 is about.

Merge pull request #1943 from lidge-jun/codex/wave5-wp5-triage

docs(devlog): record the WP5 outcome for lidge-jun#1849 and lidge-jun…

devlog: close the r2 blockers on the Windows unit

Status (260818 campaign sweep): the restart-storm half landed on dev — d09c752 adds missing-install guards to the Windows service wrapper (src/service.ts:1549-1556, exit /b 3 before the loop), so the 1000+-restart storm on a file-less skeleton cannot recur; a3169db scopes the wrapper killer. Still open: post-install verification that the package actually unpacked, rollback of an empty npm install, and a recovery path when launchers are gone (src/update/job.ts still only does a pre-flight registry probe). Kept open for the remaining transactional-update half — being designed together with #1942 in the campaign wp10 follow-up.

docs(devlog): wp10 follow-up designs + campaign phase records

fix(update): stage, verify, and swap npm self-updates with rollback (#…

The remaining half of this issue is fixed on dev by #2079 (1ad131a): the empty-install class is closed by staged verification (a tree that fails the manifest never becomes live), and the restart-loop-on-skeleton class now self-heals — the service wrapper restores the newest .ocx-backup-* sibling before declaring the install incomplete, and the launcher boot probe restores/reaps on the next start. Earlier halves (wrapper restart guard, atomic-replace foundations) were d09c752 and the 030/031 units.

test(codex): pin the boot-probe guard by what it wraps, not by adjacency

docs(devlog): record the WP5 outcome for lidge-jun#1849 and lidge-jun…

Merge pull request lidge-jun#1943 from lidge-jun/codex/wave5-wp5-triage

devlog: close the r2 blockers on the Windows unit

docs(devlog): wp10 follow-up designs + campaign phase records

fix(update): stage, verify, and swap npm self-updates with rollback (l…

test(codex): pin the boot-probe guard by what it wraps, not by adjacency

Navigation Menu

Repository navigation

[Bug][Windows] Failed ocx update leaves a file-less package skeleton; service wrapper restart-loops 1,009 times with no integrity check or backoff #1849

Summary

Environment

Evidence

1. The abandoned install contained directories only

2. Launchers were removed, recovery commands unreachable

3. Unbounded restart loop with no integrity check

Reproduction

Impact

Suggested fixes

Notes

Maintainer decision respected

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Bug][Windows] Failed ocx update leaves a file-less package skeleton; service wrapper restart-loops 1,009 times with no integrity check or backoff #1849

Description

Summary

Environment

Evidence

1. The abandoned install contained directories only

2. Launchers were removed, recovery commands unreachable

3. Unbounded restart loop with no integrity check

Reproduction

Impact

Suggested fixes

Notes

Activity

coderabbitai commented on Aug 16, 2026

github-actions commented on Aug 16, 2026

Maintainer decision respected

Ingwannu commented on Aug 17, 2026

Ingwannu commented on Aug 17, 2026

lidge-jun commented on Aug 18, 2026

lidge-jun commented on Aug 18, 2026

lidge-jun commented on Aug 18, 2026

lidge-jun commented on Aug 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions