fix(mount-io): bound token-harvester gcsfuse reads behind a MountIO guard#710
Draft
niiyeboah wants to merge 11 commits into
Draft
fix(mount-io): bound token-harvester gcsfuse reads behind a MountIO guard#710niiyeboah wants to merge 11 commits into
niiyeboah wants to merge 11 commits into
Conversation
…pool, observability Addresses findings from backend, infra, architecture, and release-readiness persona reviews of the MountIO guard: - Streaming reads now use an idle (no-progress) deadline: run() hands fn a heartbeat() that re-arms the timer, and both JSONL parse loops call it per line. A slow-but-progressing large file over a healthy mount no longer trips the process-wide breaker — only a genuine no-data stall does. readdir keeps the whole-op deadline (single-shot, can't heartbeat). - Raise default DISPATCH_MOUNT_IO_TIMEOUT_MS 5s -> 10s; document tuning. - Set UV_THREADPOOL_SIZE=16 in the launchd wrapper + .env.example so an uncancellable readdir/read syscall wedged on a stalled gcsfuse mount cannot starve fs/dns/crypto work server-wide. Documented that production's Bun runtime may ignore the knob (own threadpool, not libuv) and that the breaker is the cross-runtime backstop. - Log mount-io breaker open/recover transitions via the app logger (mountIO.setLogger in server startup) for operator visibility. - Document run()'s contract: signal MUST be honored; timeout abandons rather than cancels; available()/semaphore track JS ops, not in-flight kernel work. - Document the global-breaker single-mount assumption and the MountStallError-only breaker policy; comment the load-bearing work.catch. - Replace the discoverCodexRolloutFilesForTest pass-through with a direct exported discoverCodexRolloutFiles (testing-only comment). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The token harvester reads Claude/Codex session JSONL files directly from a gcsfuse-backed directory. When that mount stalls,
read()/readdir()syscalls block indefinitely, freezing the recurring harvest loop on the first wedged file.What this does
Introduces a reusable
MountIOguard (apps/server/src/shared/mount-io/) and routes every harvester filesystem read through it:MountStallErrorinstead of hanging forever.available()and resume on the next healthy cycle. OnlyMountStallErrortrips it, so a single missing/corrupt file never opens the process-wide circuit.AbortSignalintocreateReadStreamso a stalled stream's fd is actually closed.All four
fsentry points (2 streaming JSONL parsers + 2readdirdiscovery walks) are wrapped. Token stats become eventually-consistent during a stall (no data loss — re-harvested next cycle).Persona review hardening
This branch was reviewed by four persona reviewers (architecture, backend-security, infra, release-readiness) — all approved. Resulting changes:
heartbeat()re-arm per line, so a slow-but-progressing large file over a healthy mount no longer false-trips the mount-wide breaker.readdirkeeps the whole-op deadline.UV_THREADPOOL_SIZE=16in the launchd wrapper +.env.example— an uncancellablereaddirsyscall wedged on a stalled mount can otherwise leak I/O-pool threads and starve fs/dns/crypto work server-wide. Documented that the production Bun runtime may ignore this knob (own threadpool, not libuv), with the breaker as the cross-runtime backstop.mountIO.setLogger(app.log)for operator visibility.run()JSDoc contract (signal mandatory; timeout abandons, not cancels;available()/semaphore track JS ops, not kernel work).Config (all optional, defaults shown)
Release posture
Code + env only. No DB migrations, schema, or API/SSE contract changes. Breaker state is in-memory; reverting fully restores prior behavior.
Testing
mount-io/{mount-io,circuit-breaker,semaphore}.test.tsandtoken-harvester-stall.test.ts.tsc --noEmitclean; 17/17 mount-io + stall tests pass.pnpm run checkcurrently fails on pre-existingapps/webmissing-dependency errors (refractor,react-diff-view,@testing-library/react) in files untouched by this branch.🤖 Generated with Claude Code