Repository navigation
App kit: text in elements, and big changes that arrive whole - #12
Open
itrycodinganditsbad wants to merge 41 commits into
Open
itrycodinganditsbad wants to merge 41 commits into
itrycodinganditsbad wants to merge 41 commits into
Conversation
A tree node's properties (and a timeline item's or track's, and a map cell's) can now hold a rich-text story and a table, each under a key the app's codec names, as Loro mergeable child containers: two machines making a shape's text at once make one, and their typing merges character by character. - Props::text writes a nested story with the same diff, marks and paragraph attributes as a root story; its last paragraph's attributes sit in a mergeable `$last.<key>` map beside it, made only when needed. - Props::table writes a nested table: mergeable `rows`, `cols` (movable lists) and `cells`. Cells are values, or, when TableSource::cell_maps says so, property maps of their own (write_cell, cell_key) with stories. - Remote changes report the node or cell a story belongs to (TreeTouch::stories, TableTouch::stories) with Loro's deltas, so TreeView::text_patched and TableView::cell_text patch the old text as a root story is patched; PropsRead::text and PropsRead::table read whole. - Session::background_change: a change the app makes by itself (an upload that finished), sent like a local change but never an undo step. - Tests: round-trip and merge proptests over the extended test model (layer stories, tables nested in layers, a root table of map cells); typing in one node's story at once; a node deleted while typed in; cell stories with a row and a column added at once (normalise makes the crossing cell) and a row deleted while typed in; undo of one machine's typing in a shared story; a background change that undo leaves. The kit simulation edits the nested parts too. docs/kit.md and docs/design/live.md describe the layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…ke by keystroke On Windows the laptop's keystrokes reached the pc while its undo group was open, and the group then didn't undo as one step. The test now makes each keystroke its own step (as DeckCraft does) and undoes them one by one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…ter undo) Loro's undo of a tree deletion makes the nodes again under new tree IDs, so an element can come to have another node (the old one stays deleted, with the same `$id`). A remote change that switched an element to another node wasn't reported as touching it, and an app that keeps untouched elements as they were (DeckCraft keeps untouched shapes) kept the old node's content: the two machines' models differed while their documents were the same. Found by DeckCraft's live simulation. - After reindexing a tree, every element whose node is not the one it was is touched. - An event on a node the index doesn't name (deleted, or a second node with the element's ID) touches the element its meta names. - Test: a layer deleted on one machine while the other, offline, renames it, then the delete undone: both end with the same model, which is what their documents hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…machines A local change over 512 KiB is cut into several commits, each sent on its own. A receiver rebuilt its model after whatever had arrived, so an app saw half a change, repaired it (normalise) and could write the gap back with its next local edit. - Every commit of a change but the last carries the commit message `$cont`. - A session that imported another peer's change only up to a continued commit imports what comes but applies nothing to the model until the change's last commit is in; after CHANGE_WAIT (10 s) with the sender gone half-way, it applies what it has. - Test: the first commit of a 1 MB change delivered alone leaves the model as it was, a local edit meanwhile goes through, and the rest brings the whole change (fails without the wait). - docs/kit.md: the atomicity, and the known gap of undoing a deletion (the elements come back as copies; concurrent edits to them are lost). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…rts them A change over the 1 MiB frame limit was cut into several Loro commits, and a receiver applied whatever had come (or, since the last commit, imported it and waited): a document could hold part of a change. Now one update is cut at the transport: frames of at most 1 MiB, each a fragment naming its set (the first 16 bytes of the update's BLAKE3), its index and the set's count. - craft-cloud-api `fragments`: the format (it starts `ccf1`, which no Loro update does, so a program that doesn't know fragments refuses one rather than import part of a change), a deterministic split (the same update gives the same fragments, so a set sent again is the same set), join, and a Reassembler that hands over an update only once every fragment of it came and they make the update the set names; a set that stops coming is dropped after a patience (60 s), never applied. - craft-cloud-app: Document::send sends an update of up to a snapshot's limit, in fragments back to back when it is over 1 MiB, and answers by the last fragment's number (the agent's answers for the others are swallowed). Fragments that come, in an open's state or as events, are joined first: the app only ever gets whole updates. A set that stops coming, or doesn't join, is dropped and the document hears Event::Gap (open it again to catch up). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
- An app's change in fragments goes into the machine's copy, and is acked, only once it is whole (a set that never completes, say because the app went, is never taken), as its fragments: each an update of the log, as the agent's log and the server keep them. The apps join them. - set_fragment_faults: fragments on their way to a machine's apps are dropped (any but a set's first: the app sees the set come part of the way) or held back a few Hub::tick()s (so they come after later updates, out of order), from a seeded generator. - drop_connections_after(n): the agent stops after n more update frames from its apps, part of the way through a change. - Tests: a 3 MiB update goes whole and is acked once; one over a snapshot's limit isn't sent; held-back fragments come later and whole; dropped ones are a gap, and the document opened again has the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
The wait of fbf2747 imported a peer's commits as they came and held the model back until the change's last commit: a local edit meanwhile wrote against a document holding the waiting ops (deleting or reverting them, N1), a sender that never finished held every later change 10 s (N2), a save recorded frontiers its bytes lacked (N3), an undo cut by counter range still came half-way (N4), and an undo or a failed write during the wait applied and repaired the half change (N5, N6). Now a change is one commit, exported whole as one update (an undo too); craft-cloud-app sends one over 1 MiB in fragments, and every receiver joins them before anything imports them: a session never holds part of a change, so a local edit, an undo, a repair or a save meanwhile works on whole changes only. The $cont mark, the cut into commits and CHANGE_WAIT are gone. - Event::Gap (a set that stopped coming, or didn't join): the session opens the document again at once and catches up; Status::gaps counts them (N7's "count it", now that nothing is ever applied in part). - A change over a snapshot's limit (MAX_CHANGE_BYTES) can't go: the session stops, as for a change the agent refuses for good. - The reviewer's probes as tests (tests/fragments.rs), with the fake agent dropping fragments on their way to the app: rows added and a cell edited and undone and the title changed while a 1.2 MB change comes part of the way (N1, N5); the same for a node renamed and its story typed in while a 1.2 MB paste into it comes; a save meanwhile records only what the model holds (N3); a 5 MB change and its undo arrive whole, never in part (N4); a sender whose agent stops after the first fragment holds nothing up (a later change from a third machine comes at once) and sends the change again once it is back (N2, and a crash between fragments); a receiver whose agent restarts part of the way gets the change whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
Loro's import of one update takes time growing with the square of the text in it when the text builds on another peer's (every real paste does): 1 MiB took 7 s, 3 MiB 78 s, in release, on every receiver. The cut into 512 KiB commits hid most of it; one whole update made it worse. Cut into slices of at most 16,384 operations and imported one after the other, 4 MiB takes 9 ms. - craft-cloud-api fragments: a bundle (ccb1, the count, each update's length and bytes) and parts(); SLICE_OPS. - The kit exports a change as its slices in one bundle (one update: fragments carry it whole past 1 MiB), and imports a bundle's slices one after the other before the model sees any of them: the change is still never seen in part. - Test: a 300 KB paste is a bundle of slices of at most SLICE_OPS operations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…only whole The agent imports into its copy, answers for what the document holds, and makes the server's snapshots: it now does each of these only with whole changes. - The local API joins an app's fragments per document before the hub sees the change (under its last fragment's number); a set the app stops sending part of the way (it went or crashed) never reaches the hub. - A local change over a frame goes into the copy and the journal whole (one record, so a crash never leaves part of it), and up to the server as fragments, each an unacked update with a number of its own; the journal brings them back with the same numbers after a crash. - A row from the server that is a fragment counts as had and is kept (in the document's file, after what an older agent reads, so a crash between fragments loses nothing: the catch-up goes on after the kept rows); the change goes into the copy and to the apps once every fragment is there, never before. An own change's fragments count in the log from the last one's row on. - No snapshot for the server while fragments are kept, nor at a number between a change's first fragment and its last (LogVersion), so the log never loses the start of a change. - A set that never completes (only possible if its sender lost fragments it had sent) is dropped after about a week of the agent running, never applied. A fragment refused too_large takes the rest of its set with it; the push sends the change again. - Bundles (one change as its slices): imported slice by slice, their ranges merged, kept out of Loro's batch import. - Found on the way: a compaction asked while the copy holds a big change not in the log yet made Loro check the document out before it, which took over 20 s for a 3 MiB paste, holding up every app on the machine. A copy more than 65,536 operations past the asked version doesn't answer (another device does, or a later ask). - Tests: a change in fragments from the server goes in only whole, across a restart, and no snapshot is made inside it; a local change over a frame is journalled whole and sent as fragments, again with the same numbers after a crash; no snapshot from a copy far past the asked version; a bundle of a 4 MiB paste's slices imports at once. End to end, through two agents and the server: a 3 MiB paste goes across whole, and a 2 MiB one made while the other agent was stopped comes whole through its catch-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…ck (N8) - Now and then a big change: a paste of 30 to 100 KB into a new layer (a background change no edit touches again) or a fill of the grid that can be undone and redone; each goes as a bundle of slices and, past the frame limit (16 KiB in the simulation: craft-cloud-app's set_frame_limit_for_tests), in fragments. Sometimes the agent stops part of the way through taking one. - Fragments on their way to the apps are dropped (the set comes part of the way: a gap, and the session catches up) or held back a few steps (they come after later changes, out of order). - The oracle: every big paste is on every session at the end, whole (nothing a machine made is lost), besides convergence, validity and the file catching up. The run checks it made big changes and caught up from gaps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
docs/kit.md: a change is one update (slices in a bundle; fragments past 1 MiB, joined by every receiver before anything imports them), gaps and MAX_CHANGE_BYTES; the fake's fault tools; known gaps (every agent and app must know fragments and bundles; a restore's push sends a big Loro change in pieces); the import and checkout costs of big text. docs/design/live.md: the bundle and fragment formats and who joins them, instead of $cont; when an agent answers a compaction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
Found by the fragments test hanging now and then: a paste from another peer into a text this machine edited at the same moment, with this machine's peer below the sender's, makes Loro work the text out again from where the edits parted for every slice, each costing more than the last (a 1.2 MB paste took 222 s in release; with the peers the other way round, 135 ms). One update of the whole change takes it about 7 s, and detaching or a batch import clear the undo stack. So a bundle's slices still go in one by one, but once a slice takes over 50 ms, the rest is joined on a detached fork (which only takes their operations) and goes in as one update: seconds for a megabyte, undo kept. The same in the agent's copy (an app's edit there can be concurrent with the paste). The kit's tests use 64 KiB frames (set_frame_limit_for_tests), so a 300 KB paste exercises fragments. Tests: a 600 KB paste into a text edited at the same moment, in the slow peer order, takes seconds in the kit (38 s without this) and in the agent's copy (37 s without), with the text both, and the kit's undo taking back only the edit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
a_group_and_a_split_change_are_one_step_each still asked for "several updates" (the cut into commits): the 1.25 MB change is one update now, in two fragments past the 1 MiB frame. It passed on Linux only when other updates happened to land in the count; on Windows it failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
- MAX_WHOLE_BYTES (and so the kit's MAX_CHANGE_BYTES) is 8 MiB, well under the server's 16 MiB cap on a document's log above its snapshot (it was a snapshot's 36 MiB: a change could be bigger than the room a document has). - A Reassembler holds at most MAX_SETS (16) sets, and every fragment is charged 64 bytes of bookkeeping besides its part, so tiny fragments can't make the byte cap allow millions of sets. Test: a seventeenth set is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…protocol 3 (F1, F4) F1: the server stored each fragment as it came, up to the document's cap, and refused the rest; every device then kept the stored fragments, no device answered a compaction, and the document stayed full for good. - Live protocol 3 (LIVE_FRAGMENTS_PROTOCOL): the server stages an agent's fragments per connection (by document and set, at most 16 sets and two changes' bytes) and, once a set is whole and joins, stores it in one transaction with consecutive numbers (Meta::append_live_updates_counted), acking every fragment then. It counts against the document's cap only then: a set that would start in a full document is refused whole (every fragment's client_seq named), nothing stored; one that crosses the cap is stored whole, as one update that crosses it is. A connection that ends drops what it staged (nothing of it was acked: the agent sends it all again). No device ever gets part of a change from the log. - A connection on an older protocol is never sent a fragment or a bundle: its agent can't read them (it would skip them, and every later change that builds on one would wait there for good). It is refused the document with "update Craft Cloud on this computer" instead. - No migration: staging is per connection, in memory. - Tests: meta (both databases): a set is stored whole with consecutive numbers, crossing the cap whole, repeats get their numbers back, and a set that would start in a full log is refused whole. Server: nothing answered or sent until the last fragment, then three acks and the whole set to the other device; a set refused whole, every fragment named; a protocol-2 device refused the document. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
F2/F3: the agent dropped a kept set after a week, and the change was then lost on that machine (and could be compacted out of the log by it); a catch-up from a snapshot past a kept set left its fragments kept, blocking that machine's compaction answers for the week. - No time ends a kept set. Only evidence does: the set comes whole; a snapshot imported at or past its first row (the copy has the change then: the server stores a set whole, and no answer is made at or past a kept set's first row); the log learnt again from the start. - A row over what a document keeps (16 sets, or 16 MiB of fragments with 64 bytes each of bookkeeping) is not taken: it doesn't count as had, and the next connection brings it again. An own acked fragment over it leaves the log version unknown instead (worked out again from the copy). - A compaction is refused only at or past a kept set's first row; a set kept above the asked row doesn't stop the answer. - Kept sets by set id and index (no more re-parsing every kept fragment for each row); the file keeps them as rows, as before. - In a document refused as full since the last ack, this machine's own unacked operations don't count towards the check-out limit: otherwise the only device would never answer, and the document stay full for good. - A server before live protocol 3 is never sent a fragment: an app's change that needs them is refused while it is the server, and fragments made before wait unsent (with a note) until a server that takes them. - Tests: a snapshot past a kept set ends it, and the next compaction is answered; a kept set waits through 5000 ticks and a restart, answers below it and none at or past it, and goes in whole; past the sets kept a row isn't taken and comes again; a server before fragments gets none; a full document is compacted from a copy far past the asked version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
A bundle whose later slice didn't read left its earlier slices in: in the agent's copy (the app told "refused" while the copy kept them; a server row dropped with part of it in) and in the kit's document. - Agent (LiveCopy::import_any_size) and kit (import_slices): every slice's header is read (decode_import_blob_meta, with its checksum) first; one that doesn't read refuses the change whole, nothing of it in. - Tests: a bundle with a slice that doesn't read leaves the agent's copy as it was; the kit's document likewise, and takes the good slices after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…fuses its change (F5, F7, F9, T1) F9: the agent joins the server's fragments before its apps get a change, so the app library's reassembler, its 60 s patience and Event::Gap never ran in production; the sims tested them instead of what the agent does. - The app library takes every update from the agent as it is. Gone: its reassembler, fragment_patience (and its test hook), Event::Gap, the kit's Status::gaps and its reopen on a gap, the agent's doc command's arm for it. - The fake agent now does as the agent does: a machine's log keeps an app's change as its fragments (as the server's log does), and the machine joins them before its apps get the change; an open's state has only whole changes, and a set not whole yet goes once it is. FragmentFaults now stop a machine's link from the hub inside a change (sleep: SLEEP_TICKS ticks; delay: a few), as a laptop that sleeps part of the way through it; the rows after wait too, in the log's order. The kit sim counts those stops (Fake::pauses) and asserts some happened. - F5: the agent answers a refused fragment under its own number; for one that isn't its change's last the app dropped the answer, and the change waited for an ack for good. Now the app library refuses the change under its last number, once (answers still to come for it are dropped). - F7: the 5 MB test checks each note's text, not only their number. - Tests: the fake's machine that sleeps inside a change gives its apps nothing of it until it wakes, then the change whole (an app that opened meanwhile too, and the state after); inner answers aren't the app's, and a refusal of one refuses the change once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…other machines (F4)
F4: nothing told an agent from before changes in fragments apart: a new app
on one sent fragments it took as changes of their own, and a machine on
one got nothing of a big change, silently.
- Local API version 2 (LOCAL_API_FRAGMENTS): the agent takes an app's
change in fragments and sends changes whole. An app's live documents
need it: Documents::connect asks the status first and refuses an older
agent as agent_too_old ("Craft Cloud on this computer is older than this
app; update Craft Cloud to edit live"); the kit opens the file as a
plain file with NotLive::AgentTooOld for the app to say so.
- The kit's presence frame says the sender's agent version (a mark and the
version after the peer; 16 bytes before the app's, so 4 KiB less 16).
Peer::agent_api, Peer::older (none, from a kit before versions, or below
version 2) and Status::older_peer let the app show "<machine> runs an
older Craft Cloud; update it".
- (Live protocol 3 and the server refusing fragments and bundles to older
agents are in the server commit.)
- The fake's too_old machine speaks local API 1; the old-agent blob test's
stand-in too.
- Tests: an old agent's documents are refused with agent_too_old (app
library) and open as NotLive::AgentTooOld (kit); presence names the
current version, and a frame without the mark is an older peer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…ound (T1) T1: no simulation ran the agent's own path for a change in fragments. - The documents simulation (real replicas against the server's sessions and metadata database) makes some edits big, over a 2 KiB frame limit (replica::set_frame_limit_for_tests): they go up in fragments, the server stores each set whole, and the rows go down to the other machines. Machines sleep for a while (connection dropped or not), also right after a row that leaves them keeping part of a change, and crash there; catch-ups from snapshots and compaction run as before. It checks besides that no machine still keeps fragments at the end, and that big changes, kept parts and snapshot catch-ups happened. - Found and fixed: a catch-up's batch went to the apps only when it changed the copy's version. Rows Loro held back for their history (a change in fragments not whole yet below them) changed nothing, and taken on their own afterwards Loro knew them already: the apps never got them. Now every batched row with operations past what the copy had goes to the apps. - Found and fixed: acks of an own change's fragments lost with the connection after the first: the rest went again alone, never made a set on the server, and waited for good. One fragment's ack says the whole set is stored (the server stores it in one go, numbered in the fragments' order): its other fragments are acked with it. - Replica::kept_fragments (how many rows a document keeps as fragments). - Test: one ack of a set stores the whole set; nothing is sent again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…or later (F1) The end-to-end case of F1 found the deadlock the reviewer's deviation 4 names, in the order frames really come: a big change crosses the cap (stored whole, compaction asked), the sender's next big paste is already in its copy, so the ask finds the copy far past the version asked and is dropped; the paste is then refused as full, and no ask comes again (nothing more is stored). The only device never answered; the document stayed full for good. - An ask left for later (copy far past the version) is kept, and tried again at the next ack, and when the document is refused as full (where this machine's own unacked operations don't count towards the limit). - Test (end to end, server sessions and metadata with an 8 KB cap): a 12 KB change in fragments crosses the cap and is stored whole; a 70 KB paste after it is refused as full; the one device answers, the log gets below the cap, the paste goes; a machine joining at the end has both whole, and the log above the snapshot holds whole sets only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
- design/live.md: protocol 3 (the server stages a set and stores it whole, or refuses it whole; nothing in fragments to older agents, and none sent to older servers); the cap counting a set once whole, and a change at most 8 MiB; local API 2 and agent_too_old; one fragment's ack standing for the set; kept sets ended by evidence only (no time), with their bounds; the compaction rule at a kept set's first row, and the full document's exception; the agent's apps never getting a fragment; the presence frame's version. - kit.md: the agent keeps a change that comes part of the way, the app never sees part of it (no gaps any more); MAX_CHANGE_BYTES 8 MiB; the presence header of 16 bytes, Peer::older and Status::older_peer; NotLive::AgentTooOld; the fake's faults as stops of a machine's link; the known gap on versions rewritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
… no bundle goes (G1)
G1: toward a server before protocol 3 the agent skipped a held fragment
and sent what came after it (a later change, building on the held one,
waited on every other machine for history that never came), and sent
bundles, which an agent before them can't read.
- send_pending stops at the first held fragment or bundle: nothing of the
document goes ahead of it. An app's change on the live path goes the
same way (in order, behind what is held), not straight out.
- A bundle is refused on protocol < 3 as a change in fragments is ("needs
a newer Craft Cloud server; it wasn't shared").
- Whether to hold is taken from the link when sending (at subscribed, each
tick, each change), not only at the welcome: a document read back since
(after Loro failed) holds too (N-d).
- Test: a bundle in one frame refused; a small change after the held
fragments waits behind them, also after the document is read back; all
of it goes, in order, once the server takes fragments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…s; sets at most 8 MiB (G2, G3, N-c) - G2: a protocol-2 connection's fragment is refused up front (invalid, "a change in fragments needs a newer Craft Cloud on this computer: update Craft Cloud"). Stored one by one, the set of an agent from before whole sets could be stored in part near the cap, and every newer agent would keep that part for good. An agent from v0.1.0 never sends one. - G3: when a set starts staging, the fragments of it that this device has in the log already count as here (found by their client_seq: a device numbers a set's fragments one after the other, and the bytes must be that set's fragment at that index). So the rest of a set stored in part before whole sets completes it, and the rest of a set whose acks were lost does too; every fragment is acked with the number it has (Meta::live_updates_sent). - G3: `craft-cloud-server live check-sets <user>` lists the user's live documents whose log above the snapshot holds part of a change (one that never completes holds that document's compaction), each with the reset-doc that empties it. Meta::live_heads is no longer test-only. - N-c: a set that joins to more than a change's limit (8 MiB) is refused whole (too_large). - Tests: the rest of a set stored in part completes it (acks 1, 2, 3; the log joins to the change); a protocol-2 fragment refused; a set joining to 8 MiB + 10 refused whole, nothing stored; check-sets lists the partial document with its reset-doc, not the whole one, and nothing after the reset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
The ack inference of round 3 (one fragment's ack standing for its set, numbered from it) was wrong for a set partly stored before and partly now (the old part keeps its numbers). The server now completes a set from what it stored of it, so the agent no longer infers: the rest of a set whose acks were lost goes again, and is acked with the numbers it has. - After a restore (start over), the rest of an own set whose first fragments were acked is dropped from what waits: the log may not hold those any more, and sent alone the rest would never make a set. Its operations are in the copy, and the push sends them. (The documents sim found this once the inference was gone.) - Test: the rest of a set whose acks were lost goes again (2 and 3), and nothing waits once their acks come. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…s any overlap (G4) G4: the agent said local API 2, and an app from before it (which takes version 1 only) refused the agent outright: no recents, no blobs, not only no live documents. - Status::api is now the oldest version the agent speaks (1), and the new api_max its newest (2); an agent from before api_max says neither more. An app from before still reads api = 1 and connects, and gets what version 1 has (recents and blobs; its live documents still open, but a big change from a newer app, a bundle, doesn't read in it: the apps that edit live documents are updated with the agent). - An app takes an agent whose versions overlap its own (speaks), and gates what it uses by Status::api_newest: live documents need LOCAL_API_FRAGMENTS, as before. - The fake's too_old machine speaks 1 only. - Tests: an agent before api_max, a newer one, and one with nothing the app speaks; the status round-trips with and without api_max. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
The 5 MB change was twenty notes, each one value (one operation): never
more than one slice, so a regression to separate slices couldn't show in
it.
- The test also pastes 5 MB of text into a story: hundreds of slices, in
one bundle, in fragments. The other machine's story is empty or the
whole paste at every poll.
- Each change and its undo is checked in the log: one row, or every row a
fragment of one set, joining to the change (Hub::rows).
- With the reviewer's M3_K (the slices sent as separate updates instead of
a bundle) the test fails ("the laptop held part of the paste: 2277376
bytes"); without it, it passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…ht (T3, T4) T3: the documents sim reached "part of a change kept" 11 times in 3000 seeds, and passed without the kept-set compaction guard (M3_F) or without dropping kept sets on a snapshot (M3_A). - Now and then a big change is stored in part first, as a server before whole sets stored one whose sender's connection ended after its first fragment: that row alone goes into the log, and the sender's link ends; it sends the set again when it is back (the server completes it from the row it has). The other machines keep the row meanwhile, often for long, through sleeps, compactions and snapshot catch-ups. - A catch-up is taken in batches of 6 KiB (replica::set_batch_bytes_for _tests), so one that ends inside a set leaves part of it kept; a machine that keeps part of another's change after a row sleeps there more often. Big edits are 12 of every 100. - Counted per step and machine: 60,307 steps keeping part of a change in seeds 0-2999 (and 531 changes stored in part first). With the reviewer's mutations ported (env-gated, not committed): M3_A fails (seed 2639: "m1 still keeps 3 fragment(s) of a change it never got whole"), M3_F fails (seed 23: "m1 holds another document"; seed 81: "a machine joining at the end holds another document"). Without them, seeds 0-2999 pass. - Found by it: a journal read back after a crash cut a change at 1 MiB, not at the frame limit it was sent with, so in the sim its fragments came back as one update under the first one's number. Now cut at the same limit (in production the two are the same). - The sim's failure messages name each row of the server's log (fragments too). - T4: a replica test pins the catch-up batch fix: a batched row Loro holds back for its history (a change in fragments below it) goes to the apps after the change. It fails with the fix reverted (M3_B). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
N-b: once a document is full nothing more is stored, so no append asks any device to compact; a device whose ask was left for later (far past by another device's change, not its own) never answered, and the document stayed full until some device reconnected. - A quota_exceeded refusal asks the refused device to compact at the head (Session::maybe_compact: once per head, and not while another device's ask is out). - Test: the device whose change crossed the cap is asked; another device refused as full is asked once that ask times out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…ts, the full-document ask design/live.md and kit.md: the server refusing fragments from older agents and joined sets over 8 MiB; the agent holding fragments, bundles and what follows them toward an older server; a set completed from the fragments already in the log, and the rest of a set dropped after a restore; `live check-sets`; the ask on a full refusal; the local API's oldest and newest versions, and what an older app gets; the presence badge as best effort; the safe rollout order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
… documents connection Documents::connect asked the agent's status each time (one more loopback connection per reconnect). A client that connected through local-api.json knows the versions from that status already, and keeps them (Client::agent_api); one made for a fake agent is made with them. The kit sim had begun running out of loopback ports (connections in TIME_WAIT past the 4,096 ephemeral ports here) with the extra call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…arts the document over (H1, N-g) H1: completing a set from the fragments in the log found its legacy first part even at or below the snapshot's base (a device from before whole sets had compacted past it), and stored the rest above the base: a set no machine catching up from the snapshot ever gets whole, which holds back every compaction after it. - The server refuses such a fragment with resync_required (on its client_seq), storing nothing (Meta::live_updates_sent now gives each found row's number). The agent, on that refusal, drops the set from what waits (once: its other fragments are refused the same way) and starts the document over: the set's operations go up as a push. - N-g: the fragments found count against the connection's staging limit with the incoming one. - Tests: the reviewer's probe as a server test (fragment 0 stored alone, compacted past; fragments 1 and 2 refused resync_required, nothing stored above the base); a replica test (the set goes from what waits, once, and the document starts over with a push). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
Two cuts of one change (at different frame limits) share the set ID; keyed by set alone, their fragments could add up to one that looks whole. Now by set and count, each cut is its own set. Test: a document with one fragment of a 2-cut and two of a 3-cut is listed with 2 partial changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
kit.md: such an app still connects (recents, blobs), but a newer app's big change doesn't read in it, its model goes apart, and it may save that model; the agent can't tell it apart yet, so every older Craft app is quit before the agents are updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
… (H1) The documents sim (seed 2901) reached H1 with two legacy sets in one document: the second refusal started the document over again, its subscribe was refused while the first start over's catch-up ran, and the machine ended without rows another machine had sent. Now, while a start over is under way (its push to come or going), a refused set only leaves what waits: that push sends its operations too. Seeds 0-2999 pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…d; the fake answers as the agent does The app library kept a number for each of a change's earlier fragments until an answer came for exactly that number. The agent's local API never answers those (it answers the change once, under the number of the fragment that made it whole), so every change in fragments left its earlier numbers kept for the connection's life. The fake agent answered each fragment, which hid it. After a refusal, a later refusal of another of the change's numbers was passed on to the app as one for a number it never had as an answer. - An ack or a refusal under a change's last number drops what was kept for its earlier fragments. A refusal of an earlier fragment refuses the change once and marks all its numbers answered, so every later answer for any of them is dropped. A document dropped leaves nothing kept. - The fake agent answers a change in fragments once, under the number of the fragment that made it whole (ack, refusal as asked, or fragments that don't make their change), as the agent does. - Tests: through the fake, a change acked leaves nothing kept, a refused one is heard once under its last number, and a dropped document leaves nothing (fails with the cleanup taken out: 2 numbers kept); and a unit test of the answers, the refusals after one and the drop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
Completing a set looked up each of its other fragments in a query of its own, as many as the first fragment's count said (up to 4,095 for a forged one, per new set). Now one query takes the device's rows with the set's client_seqs (from this fragment's number and index, to its count), and each is matched to its index, set and count in the server (Meta::live_updates_sent takes the range, and gives each row's client_seq too). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…its function - AGENTS.md: the change writer makes one commit per change, exported as one update (a bundle of SLICE_OPS slices), cut in fragments past 1 MiB by craft-cloud-app; no more commits cut under 512 KiB. - design/live.md: GET /v1/status names api_max, and what api and api_max mean (an app takes an agent whose range overlaps its own). - server: maybe_compact's doc comment, left above ask_when_full, is back on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
At open, a bundle in the document's state that didn't read was taken as an empty change, with nothing said (after open the same case was said in Status::error). - At open and after, such a change is said in Status::error and counted in the new Status::unread (changes from other machines that didn't read here in this session, never applied). - Test: another app's bundle cut short is said and counted by the session that hears it, and by one opening the document after it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
…ne constant for fragment bookkeeping - kit undo tests: a change sent in fragments is one step (not "split into several commits"); the test is named for that. - app library: an open's state no longer counts frames apart from its updates (they are the same since it joins nothing). - meta: append_live_updates_counted says the fresh updates get consecutive numbers (repeats keep theirs). - kit writer: OP_OVERHEAD is no longer exported (nothing outside the writer uses it); the count says only whether a change wrote anything. - agent: the kept-fragment bound counts the incoming row's bookkeeping as the kept ones' are counted; the server's staging bound names FRAGMENT_OVERHEAD instead of 64. - fragments module doc: the local API joins an app's fragments in a Reassembler, the replica keeps the log's rows until they are whole. - kit.md: a stray line break. - The kit's import_slices and the agent's LiveCopy::import_any_size each point at the other, to keep their slow-slice thresholds alike. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What DeckCraft needed from the App kit to go live, and what a two-machine trial of it turned up. DeckCraft's own changes stay on the fork (
craft-cloud-live) until the bundled app PRs.What's in it
craft-cloud-server live check-sets <user>: lists live documents whose log holds part of a change from an older build, each with thereset-docthat clears it.docs/kit.mdcovers nested stories and tables, fragments, rollout order and the known gaps.Review
A whole-branch review of this branch and DeckCraft's, then five fix rounds, each re-reviewed. The first design for big changes imported part of a change and waited for the rest. The review showed that a local edit made during that wait could delete the other machine's paste, so it was replaced by joining before import. Later rounds fixed sleeping laptops holding fragments, documents filling up for good, and mixed-version rollouts.
Tests
cargo xtask cion Linux.Rollout
The server first, then every agent, then the apps. Quit old live-editing apps before the agents are updated, then run
check-sets.Known gaps
🤖 Generated with Claude Code
https://claude.ai/code/session_01RC1oqfy9kZznj94RjMojyz