Repository navigation
Live documents - #8
Merged
Merged
Conversation
…etadata The live protocol gains version 2: documents of file nodes (`node_doc`, a `create` with its snapshot appended, answered `created` with `kept` appended), compaction (`compact`, `snapshot` both ways), presence both ways, and `subscribe_log` for an agent that can't read a snapshot. A recents `create` and its `created` are byte for byte what they were, and a peer that speaks only version 1 never sees the new frames. The metadata gains a document's base, its snapshot's pieces in the chunk store and their size (migration 5, SQLite and Postgres): a create by import that keeps the first snapshot, a compaction that only moves the base forward, rows kept below it but no longer counted against the cap, and documents of files counted apart from the recents caps. The local API's docs messages are in `craft-cloud-proto::local` too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
On live protocol 2 the server makes a document of a file by import (the ID must be the node's, the node a file of the space; the first snapshot is kept and stored in the chunk store), sends a `subscribe` from below the snapshot the snapshot and then the log from its base, and asks one device at a time to compact a log that passed the limits: the one whose update made it long, or one that just caught up. A device that doesn't answer in time is skipped for a while; only the snapshot a session was asked for is taken, stored first and recorded with its base in one transaction, and the rows below stay. `subscribe_log` never gets a snapshot. Presence is relayed to the document's other subscribers within its limits, never stored. A connection on protocol 1 sees none of it and is sent compacted logs whole. `live reset-doc` takes a document's ID too, and drops the snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
The replica holds documents of files beside recents, with recents' behaviour (and file format) unchanged: a generic Loro copy (`live/copy.rs`) under both, with Loro's checked import for every update (refused, the copy is as it was; Loro failing reads the document back from disk). A file opened for the first time is followed from the start to see whether the server has it; if not, the app is asked for an import, which goes up as `create` with its snapshot, and the app hears the document is live only once the server kept it, or once the agent caught up with the import that was kept instead. A document that isn't joined is never written to disk, so a never-live file opens as a plain file offline; one that has been live opens from its copy. A joined document the server lost is made again from the copy. Local changes go to a journal per document before the app is told (the document's file is written now and then, and at stop), documents of files live in files of their own kind (`.doc`) an older agent leaves alone, and at most 256 are followed (the least recently used closed one is let go). The agent answers `compact` once it holds every update up to `upto` and nothing waits, with a snapshot of exactly those updates (the log's version is tracked per update, or worked out from the copy), takes a catch-up snapshot, and follows the import and whole log when a snapshot doesn't read. Presence goes both ways. The hub opens documents by path (a file not uploaded yet gets a sync pass and up to 5 s for its node), hands remote changes and presence to the apps, and their status when it changes. The server keeps a document's import beside its snapshot, so the whole log plus the import is always there for `subscribe_log`; a document's log bytes count every row again, with the compacted part apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
The local API gains `GET /v1/docs`: JSON control messages and binary frames of `[kind][id][bytes]` for document bytes, many documents on one connection. An app's garbage binary frame ends its connection, an update over 1 MiB or an import over the snapshot limit is refused before it reaches the hub, one that doesn't read is dropped, one that goes away mid-import leaves nothing on the server (the next app to open the file is asked), and signed out every open is a plain file. `craft-cloud-app` gains `live::Documents` (one connection, its own reader thread, sends written at once from the caller's thread): `open(path, app)` gives a live document, an importer for a file that has never been live, or the reason it is a plain file; a document has its state, `send`, `presence`, `try_recv`, `recv_timeout` and `close`. Blocking, no Loro, and no new dependency. `craft-cloud-agent doc <path> [--append <text>] [--watch]` opens a file as a plain-text live document through the running agent, as an app: one Loro text under `text`, imported from the file as UTF-8 (refused if it isn't). It never writes the file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
`tests/documents_sim.rs`: machines with app stand-ins editing one document's text and map through their replicas, against the server's live sessions, with frames delivered at random moments, connections dropped, machines crashing, documents written now and then, apps closing and reopening, presence, compaction at random (a low limit per seed, requests that time out at once or never), duplicated updates and acks, backups and restores, and epoch resets. Every machine and app ends on the same document with every edit its agent took. `SIM_SEED` replays a seed. `tests/network/live.rs` runs the same work over turmoil: real WebSocket framing between the replicas and the sessions, through partitions, held links and latency. `tests/live_speed.rs`: a keystroke reaches the other machine through the crate, two agents and the server in about 20 ms with 2 ms of delay each way (the limit is 250 ms); a 5 MB document with 100,000 updates opens from snapshot and tail in under 100 ms (release). What they found, fixed: - A catch-up brought back an update of the agent's own that it still held as unacked, so the log's version worked out from the copy left it out, and a compaction snapshot lacked updates below its base. Own updates the log brings back are had now; and an agent whose catch-up from a snapshot leaves updates waiting reads the import and the whole log instead. - An app opening a document after a restart missed the agent's own unacked updates still waiting in Loro for history: the state an app gets now carries them too. - The hub wrote everything before reading again, so a long send and the server's sends could stall each other until a timeout, again and again: the live connection's writes are a task of their own now. - The crate's reader held the socket for its whole read timeout (a read timeout is the socket's, shared by its clones), delaying every send by up to 250 ms: reading and writing are WebSockets of their own over the one connection now. The server and the local API turn Nagle off. - A catch-up of a big document took each update with an import of its own: they are imported as a batch first now. - A document joined is written with the next of the documents' writes, and before its first local change (its journal needs it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
The protocol as built in `docs/design/live.md`: the local API's docs WebSocket, documents of files made by import, snapshots and compaction, catch-up from a snapshot, `subscribe_log`, presence, the caps. The `doc` command in `docs/agent.md`, the workspace map in AGENTS.md, and the README's status line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
An own update sent on an old connection and again on a new one can come back as a row of the catch-up before the second copy's ack. The row took the unacked update away, so the ack, which numbers a second row with the same bytes that this machine is never sent, found nothing: the copy waited for that row forever and every later ack piled up behind it. The recents simulation found it (seed 601). The update taken back is remembered for this connection; its ack counts as had at once, or once the rows below it came. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
A `status` event can come between an app's request and its answer (the document went online meanwhile). The test's raw reader took it for the answer on Windows; it now skips status events as an app would. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
An unacked update whose operations the server is known to hold (the rows taken and the snapshot a catch-up started with) is stored: it leaves the unacked list, whatever its ack does. An ack that came early for one is kept until the rows below it are here; one still on its way is counted when it comes. The log's version is worked out from the copy only when nothing is unacked, so a compaction after a restart never leaves out an update the log holds. A machine that dropped a row (Loro failed on it, or it can never apply) no longer answers `compact` at or above it, and says so; the file keeps the row's number (a recents file after what an older agent reads). A fresh machine joins at the end of every documents simulation and must hold the same document. Review: H1, M1, M4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…s the document When Loro fails on a catch-up's batch, the document is read back from its file and the catch-up is taken again one row at a time for the rest of the run, so the row at fault is found and only it is dropped (and said). The row Loro failed on before is left out of later batches. A second failure in one run stops the document: it is read back, no longer followed until the agent restarts (its local changes are kept on disk), and its apps are told by a `stopped` flag in `status`. Review: M3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
A journal append that fails (part-way, or at its flush) is cut back to where the journal was, by the store and again by the replica; the change is refused to the app, and the document is written whole before its journal takes another change. On load, a journal is read up to its first record that doesn't read and cut there, so what is appended next reads. A document read back from its file after a refused change is followed again at the next tick instead of staying unfollowed while the link is up. Review: M5, low 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
The Loro peers this machine's apps write a document of a file as (and the import its app made) count as the machine's own in a push, so after a restore or reset they go up without the random wait that others' operations get. A change an app makes during a push goes at once as a plain update instead of waiting behind the push's parts; once acked, its operations count as in the log for the push. An unacked update the push finds in the log at its end keeps its ack's place (as one a row brought back does). A machine that is pushing answers `compact` (its log was learnt from the start, and noted since). Review: M6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
After `live reset-doc`, the server keeps the document with an empty log
and no snapshot until its machines send it back. A machine without a
copy that opened the file then joined that empty document, and its app
could save it over the file. A join that finds the server holding
nothing of the document (head 0, no snapshot) now lets go of it and
answers `not_live {reason: "resetting"}`: the app opens the file, and a
later open joins once a machine has sent the document back.
Review: M2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
At start, only the documents of files with something unacked (in their file or journal), waiting, or a push under way are loaded; the rest stay on disk, read one at a time to decide, until an app opens one. Past 256 loaded documents, the least recently used one with nothing unacked or waiting, no push under way and no app holding it is written (if changed), unfollowed and let go of; it loads again when opened. The replica's own description says what it now does about acks, Loro failures, journals, resets and pushes. Review: M9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
… are capped Snapshots no longer travel in WebSocket frames. The agent cuts a snapshot into chunks as it does a file's content and uploads them and their manifest through the chunk API (with its body limits, hash checks and the account's stored-bytes cap) before the frame that names the manifest; the hub holds that document's later frames behind the upload. `create` and `snapshot` frames carry only the manifest ID, and the server records one only if the node is a file of the space and the manifest one the account uploaded, both checked in the metadata transaction. A catch-up's `snapshot` names the manifest; the agent downloads it (every chunk checked against its ID, the whole against the manifest), the catch-up's updates waiting for it, and follows the whole log if it can't get it. The server writes no blob for a frame, so a refused create leaves nothing behind. The live WebSocket's message cap is back to an update and its envelope, on every protocol, both ways. An account may hold updates in at most 10,000 documents of files, and every row of all its live documents, compacted ones too, counts towards a live total capped by the server's `live_bytes_per_user` (1 GiB by default, in docs/server.md); over either, the append is refused with `quota_exceeded`, which the agent handles as a full document. A reset gives the document's bytes back. A joined document the server lost whose copy is over the snapshot limit now says so instead of retrying silently. Review: H2, M7, M8, low 8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…t their edge Presence is budgeted per device and document across the device's connections (20 a second), and spends the connection's frame budget too, so a device can't multiply it by opening connections or documents. Recents files and documents of files keep their own few set-aside copies; the recents store's set-aside reuses the documents' code. The journal's cap (written into the file once past it) and the catch-up batch's cap (taken before `subscribed` once past it) have tests at their edge. Review: lows 1, 5 (in part), 7 (in part). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
Owner
Author
|
Tried 2d7943e on real machines: the server on cloud01 (database backed up before migration 0005), lptp1 (Linux,
Not tried: offline edits (I'd have to unplug lptp1), and opening a compacted document on a machine that joins fresh. One small thing: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A file in the cloud can now be opened as a live document. Every change goes to the other machines as it is made, instead of waiting for the file to be saved and synced. The server stores each change as a Loro update and sends it to every agent that follows the document. Apps reach their documents through the local agent.
What's in it
live_bytes_per_user(default 1 GiB).live reset-docfor recovery.craft_cloud_app::live::Documentstalks to the agent's local/v1/docsendpoint.craft-cloud-agent doc <path> [--append TEXT] [--watch]shows a document, appends to it or follows it. It's for trying the feature until the apps use it.docs/design/live.md, plus the server, agent and install pages.No new crates. Loro was already in the lock file through recents.
Behaviour worth knowing
not_live. It turns into a document the first time it's opened online.reset-doc, a machine with no copy of the document getsnot_live{resetting}until a machine that has one pushes it back.live_bytes_per_useronly goes up until retention exists.Tests
cargo xtask ciwith Postgres: 935 tests pass.A review and one round of fixes have been done. Known small leftovers:
reset-doccan deadlock. It's rare, and nothing is lost.🤖 Generated with Claude Code
https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz