fix(perf): stop re-parsing the whole Codex session history on every refresh - #212
Merged
Conversation
…efresh Users reported Dr. Claw becoming laggy, hanging on "loading", failing to create new sessions, and sitting with a running timer that never produces output — with the PowerShell window itself looking frozen. Root cause: buildCodexSessionsIndex() re-read and JSON.parse'd every line of every transcript under ~/.codex/sessions on every call, with no cache. That directory only grows; on a regular user it had reached 577 files / 10 GB. getProjects() awaited that scan, and the sync re-ran at least every 30s, so the app spent most of its time inside a scan. Measured against that 10 GB directory: first getProjects() 19,991 ms -> 543 ms repeat codex scan ~20-30 s -> 12 ms (off the request path) Changes: - Add server/utils/jsonlTailReader.js: a byte-accurate incremental reader for append-only JSONL. Session transcripts are append-only, so their metadata is a pure fold over lines; the reader reports the offset past the last complete line so a scan can resume instead of re-reading. Handles multi-byte UTF-8 across chunk boundaries, CRLF, and unterminated trailing lines. - Memoize the Codex session index per file on (ino, size, mtimeMs), resuming the fold from the cached offset on strict size growth and re-parsing fully otherwise. Concurrent scans collapse onto one pass. - Route session open and session delete through the memoized index instead of their own full-directory re-parses, with a session_meta header scan as a fallback for transcripts the project index cannot hold (no cwd). Verify the header id before returning a filename match, so deleting a session can no longer unlink a different transcript whose name merely contains the id. - Run the Codex discovery sync in the background and publish results via a 'projects-changed' event rather than making /api/projects wait for it. - Gate per-event logging in the Codex and Claude stream loops behind DRCLAW_DEBUG. These fired thousands of times per turn; Node writes to a Windows console TTY synchronously, so a console left in QuickEdit selection mode blocks the write — and with it the event loop — indefinitely. - Serialize the watcher's project payload once instead of twice per event. Adds 20 tests covering byte-offset accuracy, incremental-vs-full equivalence, rewrites, truncation, unterminated tails, cache eviction, concurrent scans, and delete targeting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| // Get messages for a specific Codex session | ||
| async function getCodexSessionMessages(sessionId, limit = null, offset = 0) { | ||
| try { | ||
| const codexSessionsDir = path.join(os.homedir(), '.codex', 'sessions'); |
There was a problem hiding this comment.
Pull request overview
This PR addresses severe UI/server lag caused by repeatedly re-reading and JSON.parse-ing the entire Codex session history on a 30s cadence, and also mitigates event-loop stalls from hot-path console logging (notably on Windows TTYs).
Changes:
- Added an incremental, byte-offset JSONL tail reader and used it to memoize Codex session indexing across scans (with concurrency collapse and bounded cache).
- Moved Codex discovery sync off the
/api/projectsrequest path and added a background “projects changed” event to trigger client refreshes only when the project set materially changes. - Gated per-event provider stream logging behind
DRCLAW_DEBUGscopes and reduced watcher broadcast overhead by serializing payloads once.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| server/utils/logger.js | Adds scoped verbose logging gate (DRCLAW_DEBUG) and a debugLog() helper to avoid hot-path console stalls. |
| server/utils/jsonlTailReader.js | Introduces byte-accurate incremental JSONL reader with offsets/trailing-line handling for append-only logs. |
| server/utils/tests/logger.test.js | Adds unit tests covering default silence, enable-all, and scoped verbose logging behavior. |
| server/utils/tests/jsonlTailReader.test.js | Adds tests for offset accuracy, CRLF/UTF-8, trailing partial lines, and yielding behavior. |
| server/projects.js | Implements memoized Codex session index w/ incremental reads; background sync & change signatures; safer session-file resolution for open/delete. |
| server/openai-codex.js | Gates noisy per-event Codex stream logging behind the new logger controls. |
| server/index.js | Reuses a debounced projects broadcast for background discovery and avoids double JSON serialization per watcher event. |
| server/claude-sdk.js | Gates noisy per-event Claude stream logging behind the new logger controls. |
| server/tests/codex-session-index-cache.test.mjs | Regression suite validating caching behavior, incremental-vs-full equivalence, and delete targeting safety. |
Suppressed comments (1)
server/projects.js:4537
- For destructive deletes, consider resolving the transcript path with strict header verification (e.g.
findCodexSessionFilePath(sessionId, { requireHeaderMatch: true })) so a filename substring match cannot delete an unrelated transcript if the header scan fails.
const matchedFilePath = await findCodexSessionFilePath(sessionId);
if (matchedFilePath && jsonlFiles.includes(matchedFilePath)) {
await fs.unlink(matchedFilePath);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+4186
to
+4199
| await readJsonlLinesFrom(filePath, 0, (line) => { | ||
| if (sessionId || seen >= CODEX_HEADER_SCAN_LINES) { | ||
| return; | ||
| } | ||
| seen += 1; | ||
| try { | ||
| const entry = JSON.parse(line); | ||
| if (entry?.type === 'session_meta' && entry.payload?.id) { | ||
| sessionId = entry.payload.id; | ||
| } | ||
| } catch (_) { | ||
| // Skip malformed lines | ||
| } | ||
| }); |
Comment on lines
+4221
to
+4240
| async function findCodexSessionFilePath(sessionId) { | ||
| if (!sessionId) { | ||
| return null; | ||
| } | ||
|
|
||
| return null; | ||
| }; | ||
| const codexSessionsDir = path.join(os.homedir(), '.codex', 'sessions'); | ||
| const jsonlFiles = await findCodexJsonlFiles(codexSessionsDir); | ||
|
|
||
| const sessionFilePath = await findSessionFileByMetadata(); | ||
| for (const filePath of jsonlFiles) { | ||
| if (!path.basename(filePath).includes(sessionId)) { | ||
| continue; | ||
| } | ||
| // A filename can merely *contain* the id, and callers act destructively on | ||
| // what we return (deleteCodexSession unlinks it). Confirm against the | ||
| // session_meta header, which costs a handful of lines. A transcript whose | ||
| // header cannot be read is still accepted, matching the long-standing | ||
| // filename-only behaviour; only a definite mismatch is rejected. | ||
| const headerId = await readCodexSessionIdFromHeader(filePath); | ||
| if (headerId === null || headerId === sessionId) { | ||
| return filePath; |
Comment on lines
+68
to
+72
| const line = rawLine.endsWith('\r') ? rawLine.slice(0, -1) : rawLine; | ||
| if (line.trim()) { | ||
| onLine(line); | ||
| lineCount += 1; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The reports
Root cause
buildCodexSessionsIndex()re-read andJSON.parse'd every line of every transcript under~/.codex/sessionson every call, with no cache. That directory only ever grows — on a regular user's machine it had reached 577 files / 10 GB.getProjects()awaited that scan, and the sync re-ran at least every 30s, so the app spent most of its wall clock inside a scan.That explains why it felt intermittent and why it got worse over time: the cost is linear in total bytes ever written, and the 30s cooldown meant some requests were instant while others paid the full scan.
Measured, against that same 10 GB directory
getProjects()Changes
server/utils/jsonlTailReader.js(new) — a byte-accurate incremental reader for append-only JSONL. Session transcripts are append-only, so their derived metadata is a pure fold over lines; the reader reports the offset just past the last complete line, so a scan can resume rather than re-read.readlinecan't do this (it yields lines, not offsets), so this usesStringDecoder+ per-lineBuffer.byteLengthto survive multi-byte UTF-8 across chunk boundaries, CRLF, and unterminated trailing lines.Memoized Codex index — per file, keyed on
(ino, size, mtimeMs). Resumes the fold from the cached offset on strict size growth; re-parses fully on anything else (rewrite, truncation, same-size rewrite). Concurrent scans collapse onto one pass.Session open / delete no longer full-scan — both re-parsed the entire directory to map a session id to a file. They now use the memoized index, with a
session_metaheader scan as a fallback for transcripts the project index can't hold (sessions with nocwd). The header id is also verified before returning a filename match, so deleting a session can no longer unlink a different transcript whose name merely contains the id.Discovery sync moved off the request path — runs in the background and publishes via a
projects-changedevent;/api/projectsanswers from the database, which is what it was already doing.Hot-path logging gated behind
DRCLAW_DEBUG— the Codex and Claude stream loops logged once per SDK event (the Codex one fired even foritem.updatedevents that are discarded immediately after). Node writes to a Windows console TTY synchronously, so a console left in QuickEdit selection mode — one stray click in the PowerShell window — stops draining and blocks the write, and with it the event loop, indefinitely. That matches "不输出了然后一直在计时" and "powershell里面就是卡住了" precisely. Re-enable withDRCLAW_DEBUG=1orDRCLAW_DEBUG=codex,claude.Watcher payload serialized once instead of twice per event.
Testing
npm run typecheckandnpm run buildclean.cwd-less sessions unreachable) plus 1 latent delete hazard. All 5 are fixed and covered by tests; the delete test was confirmed to fail without its guard.Not addressed here
test/codex-discovery.test.mjshangs onmainas well (it setsCODEX_CLI_PATHto thenodebinary, which then waits on stdin). Pre-existing and not in CI — left alone.