Skip to content

fix(mount-io): bound token-harvester gcsfuse reads behind a MountIO guard#710

Draft
niiyeboah wants to merge 11 commits into
mainfrom
fix/gcsfuse-mount-io-guard
Draft

fix(mount-io): bound token-harvester gcsfuse reads behind a MountIO guard#710
niiyeboah wants to merge 11 commits into
mainfrom
fix/gcsfuse-mount-io-guard

Conversation

@niiyeboah

Copy link
Copy Markdown
Collaborator

Problem

The token harvester reads Claude/Codex session JSONL files directly from a gcsfuse-backed directory. When that mount stalls, read()/readdir() syscalls block indefinitely, freezing the recurring harvest loop on the first wedged file.

What this does

Introduces a reusable MountIO guard (apps/server/src/shared/mount-io/) and routes every harvester filesystem read through it:

  • Per-op timeout — a stalled op is abandoned and the caller is unblocked with a MountStallError instead of hanging forever.
  • Bounded concurrency (semaphore, default 2) — caps how much in-flight mount I/O can pile up.
  • Circuit breaker (default: open after 3 stalls, 60s cooldown) — once the mount looks wedged, harvest loops short-circuit via available() and resume on the next healthy cycle. Only MountStallError trips it, so a single missing/corrupt file never opens the process-wide circuit.
  • Streaming reads pass the AbortSignal into createReadStream so a stalled stream's fd is actually closed.

All four fs entry points (2 streaming JSONL parsers + 2 readdir discovery walks) are wrapped. Token stats become eventually-consistent during a stall (no data loss — re-harvested next cycle).

Persona review hardening

This branch was reviewed by four persona reviewers (architecture, backend-security, infra, release-readiness) — all approved. Resulting changes:

  • Idle (no-progress) timeout for streaming reads via a heartbeat() re-arm per line, so a slow-but-progressing large file over a healthy mount no longer false-trips the mount-wide breaker. readdir keeps the whole-op deadline.
  • UV_THREADPOOL_SIZE=16 in the launchd wrapper + .env.example — an uncancellable readdir syscall wedged on a stalled mount can otherwise leak I/O-pool threads and starve fs/dns/crypto work server-wide. Documented that the production Bun runtime may ignore this knob (own threadpool, not libuv), with the breaker as the cross-runtime backstop.
  • Default timeout 5s → 10s with tuning docs.
  • Breaker open/recover transition logging via mountIO.setLogger(app.log) for operator visibility.
  • Thorough run() JSDoc contract (signal mandatory; timeout abandons, not cancels; available()/semaphore track JS ops, not kernel work).

Config (all optional, defaults shown)

DISPATCH_MOUNT_IO_TIMEOUT_MS=10000
DISPATCH_MOUNT_IO_CONCURRENCY=2
DISPATCH_MOUNT_IO_BREAKER_THRESHOLD=3
DISPATCH_MOUNT_IO_BREAKER_COOLDOWN_MS=60000
UV_THREADPOOL_SIZE=16

Release posture

Code + env only. No DB migrations, schema, or API/SSE contract changes. Breaker state is in-memory; reverting fully restores prior behavior.

Testing

  • New unit tests: mount-io/{mount-io,circuit-breaker,semaphore}.test.ts and token-harvester-stall.test.ts.
  • Server tsc --noEmit clean; 17/17 mount-io + stall tests pass.
  • ⚠️ The repo-wide pnpm run check currently fails on pre-existing apps/web missing-dependency errors (refractor, react-diff-view, @testing-library/react) in files untouched by this branch.

🤖 Generated with Claude Code

niiyeboah and others added 11 commits June 30, 2026 12:12
…pool, observability

Addresses findings from backend, infra, architecture, and release-readiness
persona reviews of the MountIO guard:

- Streaming reads now use an idle (no-progress) deadline: run() hands fn a
  heartbeat() that re-arms the timer, and both JSONL parse loops call it per
  line. A slow-but-progressing large file over a healthy mount no longer trips
  the process-wide breaker — only a genuine no-data stall does. readdir keeps
  the whole-op deadline (single-shot, can't heartbeat).
- Raise default DISPATCH_MOUNT_IO_TIMEOUT_MS 5s -> 10s; document tuning.
- Set UV_THREADPOOL_SIZE=16 in the launchd wrapper + .env.example so an
  uncancellable readdir/read syscall wedged on a stalled gcsfuse mount cannot
  starve fs/dns/crypto work server-wide. Documented that production's Bun
  runtime may ignore the knob (own threadpool, not libuv) and that the breaker
  is the cross-runtime backstop.
- Log mount-io breaker open/recover transitions via the app logger
  (mountIO.setLogger in server startup) for operator visibility.
- Document run()'s contract: signal MUST be honored; timeout abandons rather
  than cancels; available()/semaphore track JS ops, not in-flight kernel work.
- Document the global-breaker single-mount assumption and the
  MountStallError-only breaker policy; comment the load-bearing work.catch.
- Replace the discoverCodexRolloutFilesForTest pass-through with a direct
  exported discoverCodexRolloutFiles (testing-only comment).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant