Repository navigation
Agent: keep keystrokes moving while Windows holds the agent's files - #10
Merged
Merged
Conversation
On Windows, renaming over a file fails with "Access is denied" while another program (a virus scanner, the search indexer) holds it open without delete sharing. The live store, status and account writes all went through that rename and failed outright; main's Windows CI hit it in the live speed test (a 2.5 s keystroke after "the live documents couldn't be written to disk"). The rename now retries for about 1.5 s on ERROR_ACCESS_DENIED and ERROR_SHARING_VIOLATION before giving up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
Writing a document whole emptied its journal by deleting the file, and the next change made it again. On Windows another program holding the journal without delete sharing (a scanner, the indexer, a backup tool) made the delete fail with ERROR_SHARING_VIOLATION, so the document's write failed. One holding it without write sharing failed the next append the same way. The journal now stays for the life of the document: emptied in place with set_len(0) and an fsync, like a cut. Its records are in the document's file before it is emptied, so a crash before the cut is durable still loads them and skips them by client_seq, as before. Opening the journal for an append, a cut or the emptying waits for a holder for up to about 1.5 s, like write_atomic's rename (the retry is now retry_busy, shared by both). Files that do go (a document or recents file that doesn't read, and state.json and account.json when the account is forgotten or replaced) are renamed to a temporary name of their own first and then deleted (retire), so their names are free at once for the file written next. A temporary file a crash leaves is removed by the existing clean-ups. The lead was a deleted file staying "delete pending" on Windows while a scanner holds it, so that making the journal again fails with ERROR_ACCESS_DENIED. On NTFS on current Windows (checked on Windows 11, build 26200) a delete frees the name at once (POSIX semantics), so that part doesn't happen there; the test for it passed before this change too. The two journal tests and the state.json test failed before it with os error 32. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…n and p95 The test sorted the keystroke times and failed on the single worst one, so a failure didn't say which keystroke was slow or what happened around it. It now prints every keystroke in order with when it was sent, how long it took, and anything else the two documents said meanwhile (going offline, for one). It asserts the median under 100 ms, the 95th percentile under 250 ms and the worst under 1 s. A delay on every keystroke, or on every tenth one, still fails. One keystroke held up once on a busy runner doesn't. A unit test checks the bounds against those cases. The test also waits for the pc's document to come online before typing, not only the laptop's. After an import the pc catches up on its own import from the server, and a keystroke sent before that ends waits in the journal, so it timed the rest of the catch-up instead of a keystroke. The agents stop before the assertions. Before this, a failed assertion deleted the agents' temporary folders while they were still running, which printed "the live documents couldn't be written to disk" (os error 2 on Linux and macOS, os error 5 on Windows). Those lines came from the failure, they didn't cause it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
Right after a file is opened live, CI on Windows took 0.3 to 1.4 s per keystroke for about 4 s, and then settled. Two writes ran on threads that also relay every keystroke, and both waited for any other program holding the file they replace (retry_busy, up to about 1.5 s): - status.json, written on the run loop's runtime on every state change and heartbeat. That runtime also serves the local API, including the docs WebSocket that carries each keystroke from the app and to the other app. - a live document's whole file and the emptying of its journal, written on the hub's thread after a join and every 2 s while the document is dirty. That thread forwards every update to and from the server. Now status.json is written on a thread of its own (StatusWriter). Only the newest status waits there, a failed write is still tried again at the next write, and dropping the writer writes what still waits, so the loop's last status is on disk when it ends. The hub's writes that nothing waits for (flush and the 2 s flush_documents) don't wait for a holder of a document's files: the document stays dirty and is written at the next 2 s write. Its local updates are in its journal already. A journal that can't be emptied yet keeps records the file now has, and a load skips them by client_seq. Writes something does wait for still wait: an app's change written whole first, loads, recents adds, and the documents written when the agent stops. The new Windows test, a_keystroke_isnt_held_up_by_a_scanner_holding_the_ files_written_now_and_then, holds both agents' status.json and live document files open without delete sharing for 400 ms out of every 450 ms. On the previous commit it fails: median 33 ms, 95th percentile 2.0 s, keystrokes of 0.7 to 2.3 s. With this commit it passes at about 32 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
itrycodinganditsbad
changed the title
…lone On 6b955b5, Windows CI failed both latency tests with "keystroke 5 never arrived": no update within the 5 s recv_timeout. The log can't say whether the update was lost or only late, because the test stopped at 5 s. The test now waits up to a minute for a keystroke that is over 5 s late. It records what each agent has in its live folder (documents and journals, with sizes) when the keystroke is 5 s late and again when it arrives. Only one that never comes panics, with that state. A late one fails the bounds with the in-order report. CRAFT_CLOUD_TEST_LOAD=n adds n busy threads while typing, as live_autosave does on app-kit. On this PC, 30 runs with the scanner and 32 to 96 busy threads, plus 20 traced runs, gave 10 keystrokes of 3.6 to 11 s. Every one arrived; none was lost. Tracing showed each gap inside one agent thread between two consecutive steps with no I/O in between (for example, the pc's hub between appending an update and queuing it for the server, 4.0 s). That is the thread not being run under full CPU, not something in our code holding it. The two latency tests now take turns (a shared lock). Since 6b955b5 they ran at once: four agents and two servers on a CI runner's few cores, each test timing the other. A unit test checks the paths 6b955b5 added for losing data. A document write refused because another program holds the document, or one whose journal is held, leaves the document dirty or the journal as it was. A crash then reads every acked change back, each once, and the next write catches up. It passes on 6b955b5 too: the skipped writes don't lose updates. MemStore can now simulate a held file (Held), and is_busy also accepts ErrorKind::ResourceBusy, the kind MemStore returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
… thread binary.rs's a_second_run_exits_0_with_a_note failed on Windows CI: `status`, run right after a second `run`, printed only "not signed in". It found no running loop in status.json. The test, the app and a second `run` all take "the local API is up" to mean "status.json says a loop runs". That held before 6b955b5 only by accident. The first status write ran on the loop's runtime, which also serves the local API, so the API couldn't answer until the write was done. With the write on the status thread, the API answered first. The loop now writes its first status itself, before it starts the local API: Idle when signed in, SignedOut when not, Paused when the account doesn't read. If the API can't start, it writes Stopped. At the end, the loop finishes the status thread (so Stopped is on disk) before it stops the API and lets go of the data folder. Before this, the last status could land after the lock was released, over the status of a `run` started next. Everything in between still goes through the status thread. once_the_local_api_is_up_status_json_says_the_loop_runs (Windows) holds an old status.json for 1 s as the loop starts and reads status.json as soon as local-api.json appears. On ba6a69e it fails (it reads the old file). With this commit it passes. live_documents.rs's an_app_that_misbehaves_is_refused_or_dropped_and_the_ rest_go_on failed once on macOS: the app that never reads wasn't dropped. The agent drops an app whose message it can't send within 10 s, and the 60 changes of 200 KB (12 MB) only hold a send up once the kernel's loopback buffers are full. macOS can let those grow to megabytes. That app now connects with a 64 KiB receive buffer, so the agent's send waits however large the system would let buffers grow. Nothing in this branch touches the drop: the docs socket, its 10 s limit and the hub's queue are as on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main's CI failed the live latency test after the recent merges. On Windows, a keystroke took up to 2.5 s to reach the other machine in the first seconds after a document opened. The agent was writing its own files on the threads that pass keystrokes along. On Windows, those writes wait, or fail, while another program such as a virus scanner or the search indexer holds the file open.
Changes
write_atomicretries a rename that Windows refuses withERROR_ACCESS_DENIEDorERROR_SHARING_VIOLATION, for about 1.5 s.clear_journalfail, so the store failed to save.status.jsonis written on its own thread rather than on the run loop that relays updates between the agent and the apps. The first status (before the local API answers) and the last (before the data folder lock is released) are still written by the loop itself, sostatusand a secondrunread what they always did.Tests
status.json, document and journal files open the way a scanner does:cargo xtask cipasses on Linux.The "Access is denied" line in the earlier CI logs was a side effect: the failed test deleted its temporary folders while the agents still ran. The consistent ~400 ms worst case on Linux and macOS CI didn't reproduce locally, so it's not explained yet. The test now names the slow keystroke if it shows up again.
🤖 Generated with Claude Code
https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz