diff --git a/AGENTS.md b/AGENTS.md index a0d3a8e..5c13945 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,7 +32,7 @@ crates/ store craft-cloud-store blob store: disk, S3, memory; checks every blob in and out meta craft-cloud-meta metadata on SQLite or Postgres; the batch commit; the change feed; invites and pairings (`join.rs`); live documents' logs and snapshots (`live.rs`) sync craft-cloud-sync the sync engine over `Fs`/`Transport` traits; `MemFs`; scenario tests and the simulation - kit craft-cloud-kit the app kit (`docs/kit.md`): an app's model live through the agent; the `LiveDocument` trait and `Session` (`session.rs`: open, the `local_change` hook, `poll`, undo through Loro's undo manager, reconnect and the 64 MiB queue, presence, saves), the change writer that cuts commits under 512 KiB (`writer.rs`), the mappings (`tree.rs`, `text.rs`, `table.rs`, `timeline.rs`, `opaque.rs`, `list.rs`), values, chunk references and blob stores (`value.rs`) + kit craft-cloud-kit the app kit (`docs/kit.md`): an app's model live through the agent; the `LiveDocument` trait and `Session` (`session.rs`: open, the `local_change` hook, `poll`, undo through Loro's undo manager, reconnect and the 64 MiB queue, presence, saves), the change writer: one commit per change, exported as one update (a bundle of `SLICE_OPS` slices), cut in fragments past 1 MiB by `craft-cloud-app` (`writer.rs`), the mappings (`tree.rs`, `text.rs`, `table.rs`, `timeline.rs`, `opaque.rs`, `list.rs`; stories and tables nest in a node's or a cell's map: `props.rs`), values, chunk references and blob stores (`value.rs`) egui craft-cloud-egui Craft Cloud's piece of an app's window: the account line, the shared recents, a live document's badge, and `CloudWatch` that keeps them current app craft-cloud-app what apps link: a blocking client of the agent's local API (status, sign-in, joining, computers, pause, autostart, recents, live documents at the level of bytes: `live.rs`) apps/ diff --git a/Cargo.lock b/Cargo.lock index 7b4cbe7..75a7adf 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1158,6 +1158,7 @@ dependencies = [ name = "craft-cloud-api" version = "0.1.0" dependencies = [ + "blake3", "caseless", "proptest", "serde", diff --git a/apps/craft-cloud-agent/src/doc.rs b/apps/craft-cloud-agent/src/doc.rs index 601ef33..10e2424 100644 --- a/apps/craft-cloud-agent/src/doc.rs +++ b/apps/craft-cloud-agent/src/doc.rs @@ -105,16 +105,21 @@ fn session( }; let mirror = LoroDoc::new(); mirror.import(document.state()).map_err(|e| usage(format!("the document's state doesn't read: {e}")))?; + // (An update this copy can't take is the other side's; the agent has it, and so will a fresh open. A bundle's + // slices go in one after the other.) + let import = |mirror: &LoroDoc, bytes: &[u8]| { + for part in craft_cloud_proto::fragments::parts(bytes).unwrap_or_default() { + let _ = mirror.import(part); + } + }; for update in document.state_updates() { - // (An update this copy can't take is the other side's; the agent has it, and so will a fresh open.) - let _ = mirror.import(update); + import(&mirror, update); } let text = |mirror: &LoroDoc| mirror.get_text(TEXT).to_string(); // What came right after the state (updates still waiting for their history), before anything else. let take = |mirror: &LoroDoc, event: &Event| { if let Event::Update(bytes) = event { - // (An update this copy can't take is the other side's; the agent has it, and so will a fresh open.) - let _ = mirror.import(bytes); + import(mirror, bytes); } }; while let Some(event) = document.try_recv() { @@ -148,7 +153,7 @@ fn session( loop { match document.recv_timeout(Duration::from_secs(1)) { Some(Event::Update(bytes)) => { - let _ = mirror.import(&bytes); + import(&mirror, &bytes); say("--".into()); say(text(&mirror)); } diff --git a/apps/craft-cloud-agent/src/live/copy.rs b/apps/craft-cloud-agent/src/live/copy.rs index cd8cc72..db99cd5 100644 --- a/apps/craft-cloud-agent/src/live/copy.rs +++ b/apps/craft-cloud-agent/src/live/copy.rs @@ -5,6 +5,7 @@ use std::collections::BTreeMap; +use craft_cloud_proto::fragments::{is_bundle, parts}; use craft_cloud_proto::live::MAX_UPDATE_BYTES; use loro::{ExportMode, IdSpan, LoroDoc, VersionVector}; @@ -44,9 +45,16 @@ pub fn frontiers(ids: &[(u64, u32)]) -> Option { ids.iter().map(|(peer, counter)| Some(loro::ID::new(*peer, i32::try_from(*counter).ok()?))).collect::>>().map(loro::Frontiers::from) } -/// The ranges of operations an update (or snapshot) holds, read from its header without importing it; `None` for -/// bytes Loro can't read. +/// The ranges of operations an update (or snapshot, or a bundle of updates: its slices', merged) holds, read from its +/// header without importing it; `None` for bytes Loro can't read. pub fn spans_of(bytes: &[u8]) -> Option> { + if is_bundle(bytes) { + let mut spans = Spans::new(); + for part in parts(bytes)? { + add_spans(&mut spans, &spans_of(part)?); + } + return Some(spans.into_iter().flat_map(|(peer, list)| list.into_iter().map(move |(s, e)| (peer, s, e))).collect()); + } let meta = guarded(|| LoroDoc::decode_import_blob_meta(bytes, true).map_err(|e| UpdateError::Malformed(e.to_string()))).ok()?; Some(meta.partial_end_vv.iter().map(|(peer, end)| (*peer, meta.partial_start_vv.get(peer).copied().unwrap_or(0), *end)).collect()) } @@ -94,6 +102,10 @@ fn usable_peer(peer: u64) -> u64 { /// The most bytes of updates one of Loro's batch imports takes ([`LiveCopy::import_batch`]). const BATCH_GROUP_BYTES: usize = 16 * 1024; +/// A bundle's slice whose import takes this long: the rest goes in as one update ([`LiveCopy::import_any_size`]). +/// (The kit takes slices the same way, with the same threshold: `import_slices` in `craft-cloud-kit`; keep the two +/// alike.) +const SLOW_SLICE: std::time::Duration = std::time::Duration::from_millis(50); /// `updates` in consecutive groups of at most `max` bytes (an update bigger than that alone). fn groups(updates: &[Vec], max: usize) -> Vec<&[Vec]> { @@ -188,8 +200,57 @@ impl LiveCopy { self.import_any_size(bytes) } - /// [`LiveCopy::import`] without the update limit: for a snapshot, or this machine's own stored document. + /// [`LiveCopy::import`] without the update limit: for a snapshot, or this machine's own stored document. A bundle + /// (one change as its slices) goes in slice by slice, all of it before this returns; one that doesn't read is + /// refused whole. pub fn import_any_size(&self, bytes: &[u8]) -> Result { + if is_bundle(bytes) { + let slices = parts(bytes).ok_or_else(|| UpdateError::Malformed("a bundle cut short".into()))?; + // Every slice is read before any goes in: one that doesn't read refuses the bundle whole, nothing of it in. + for slice in &slices { + guarded(|| LoroDoc::decode_import_blob_meta(slice, true).map_err(|e| UpdateError::Malformed(e.to_string())))?; + } + let (mut new, mut needs) = (false, Vec::new()); + let mut take = |taken: Taken| match taken { + Taken::Applied { new: n } => new |= n, + Taken::Waiting { needs: more } => needs.extend(more), + }; + let mut i = 0; + while let Some(slice) = slices.get(i) { + let started = std::time::Instant::now(); + take(self.import_one(slice)?); + i += 1; + // A slice this slow: Loro takes its slow way (edits in the same text concurrent with the change, in + // one order of the peers, make each slice cost more than the last): the rest goes in as one update. + let rest = slices.get(i..).unwrap_or_default(); + if rest.len() >= 2 + && started.elapsed() > SLOW_SLICE + && let Ok(one) = self.joined(rest) + { + take(self.import_one(&one)?); + break; + } + } + return Ok(if needs.is_empty() { Taken::Applied { new } } else { Taken::Waiting { needs } }); + } + self.import_one(bytes) + } + + /// `updates` (which follow on from this copy) as one update, joined on a detached fork (which only takes their + /// operations). + fn joined(&self, updates: &[&[u8]]) -> Result, UpdateError> { + self.guard(|doc| { + let fork = doc.fork(); + fork.detach(); + let before = fork.oplog_vv(); + for u in updates { + fork.import(u).map_err(|e| UpdateError::Malformed(e.to_string()))?; + } + fork.export(ExportMode::updates(&before)).map_err(|e| UpdateError::Malformed(e.to_string())) + }) + } + + fn import_one(&self, bytes: &[u8]) -> Result { let status = self.guard(|doc| { #[cfg(test)] Self::failing(&[bytes]); @@ -340,6 +401,9 @@ impl LiveCopy { /// The longest start of `changes` that fits in `max` bytes as one update (at least one change; `Invalid` if that /// one doesn't fit in `hard_max`), with how many changes it holds. + /// + /// (A Loro change is a run of one peer's commits Loro merged, not an app's change, so a big one goes in pieces + /// here: this is history the log lost, sent up again after a server's restore.) pub fn export_prefix(&self, changes: &[(u64, i32, i32)], max: usize, hard_max: usize) -> Result<(Vec, usize), UpdateError> { let Some(&(peer, start, end)) = changes.first() else { return Err(UpdateError::Invalid("nothing to send")) }; // The most operations of the first change that fit, by doubling from one and then halving the gap, so no @@ -425,11 +489,17 @@ impl LiveCopy { /// after a catch-up from the start, or worked out from the copy when nothing else is in it); every update taken after /// it adds its ranges. A version with a gap (an update whose history comes later in the log) is no version at all: /// no snapshot can be made there. +/// +/// A change that came in fragments is in the log from its last fragment's row on (its ranges are noted there), and +/// no version is offered between its first fragment's row and its last: a snapshot there would drop its first +/// fragments from the log and keep the rest, and a machine catching up later would never get it whole. #[derive(Debug, Clone, Default)] pub struct LogVersion { anchor: Option<(u64, Version)>, /// The updates taken since the anchor, in order: number and ranges. since: Vec<(u64, Ranges)>, + /// The rows of the changes that came in fragments since the anchor: first and last. + sets: Vec<(u64, u64)>, } /// The ranges of operations of an update: peer, first and end counter. @@ -441,7 +511,14 @@ const MAX_SINCE: usize = 100_000; impl LogVersion { /// Anchored at `seq` with `version`. pub fn anchored(seq: u64, version: Version) -> Self { - Self { anchor: Some((seq, version)), since: Vec::new() } + Self { anchor: Some((seq, version)), since: Vec::new(), sets: Vec::new() } + } + + /// A change that came in fragments took the rows `first` to `last`. + pub fn note_set(&mut self, first: u64, last: u64) { + if self.anchor.is_some() && last > first { + self.sets.push((first, last)); + } } /// Nothing known. @@ -469,7 +546,7 @@ impl LogVersion { /// The version the log's updates up to `upto` make, if it is known and has no gap. pub fn at(&self, upto: u64) -> Option { let (at, base) = self.anchor.as_ref()?; - if upto < *at { + if upto < *at || self.sets.iter().any(|(first, last)| *first <= upto && upto < *last) { return None; } let mut spans = Spans::new(); @@ -495,6 +572,7 @@ impl LogVersion { /// The version at `upto` is `version` (a snapshot was made there): what came before is forgotten. pub fn rebase(&mut self, upto: u64, version: Version) { self.since.retain(|(seq, _)| *seq > upto); + self.sets.retain(|(_, last)| *last > upto); self.anchor = Some((upto, version)); } } @@ -579,6 +657,91 @@ mod tests { assert_eq!(back.get_text("text").len_utf8(), 5 * 500 * 1024, "all of it"); } + /// A 4 MiB paste on top of another peer's text: whole in one update, Loro's import takes seconds (time growing + /// with the square of the text); as a bundle of slices (as the kit sends it), the copy takes it in a moment, and + /// holds all of it. A bundle's ranges are its slices'. + #[test] + fn a_big_change_as_a_bundle_of_slices_imports_at_once() { + use craft_cloud_proto::fragments::{SLICE_OPS, bundle}; + let importer = LoroDoc::new(); + importer.set_peer_id(40).unwrap(); + text_update(&importer, 0, "import"); + let import = importer.export(ExportMode::Snapshot).unwrap(); + let app = LoroDoc::new(); + app.set_peer_id(50).unwrap(); + app.import(&import).unwrap(); + app.get_text("text").insert(0, &"a".repeat(4 * 1024 * 1024)).unwrap(); + app.commit(); + let end = *app.oplog_vv().get(&50).unwrap(); + let mut slices = Vec::new(); + let mut at = 0; + while at < end { + let to = (at + SLICE_OPS).min(end); + slices.push(app.export(ExportMode::updates_in_range(vec![IdSpan::new(50, at, to)])).unwrap()); + at = to; + } + let bundle_bytes = bundle(&slices); + assert!(is_bundle(&bundle_bytes)); + assert_eq!(spans_of(&bundle_bytes), Some(vec![(50, 0, end)]), "its ranges, merged"); + let copy = LiveCopy::load(&import, 1).unwrap(); + let started = std::time::Instant::now(); + assert_eq!(copy.import_any_size(&bundle_bytes).unwrap(), Taken::Applied { new: true }); + assert!(started.elapsed() < std::time::Duration::from_secs(2), "{:?}", started.elapsed()); + assert!(copy.covers(&[(50, end)])); + let back = LoroDoc::new(); + back.import(©.snapshot().unwrap()).unwrap(); + assert_eq!(back.get_text("text").len_utf8(), 4 * 1024 * 1024 + "import".len()); + // A snapshot of the copy as it is: at once too. + let started = std::time::Instant::now(); + let at_now = copy.snapshot_at(©.version().unwrap()).unwrap(); + assert!(!at_now.is_empty()); + assert!(started.elapsed() < std::time::Duration::from_secs(2), "{:?}", started.elapsed()); + // A bundle with a slice that doesn't read is refused whole: nothing of its other slices goes in. + let fresh = LiveCopy::load(&import, 1).unwrap(); + let mut bad: Vec> = slices.iter().take(3).cloned().collect(); + bad.push(b"not an update".to_vec()); + assert!(fresh.import_any_size(&bundle(&bad)).is_err()); + assert!(!fresh.covers(&[(50, 1)]), "nothing of it in"); + } + + /// A 600 KB paste's bundle into a copy holding an app's edit of the same text made at the same moment, the edit's + /// peer below the paste's (the order Loro is slow in, slice after slice: about a minute here): a few slices one + /// by one, then the rest together, in seconds, and the copy holds both. + #[test] + fn a_bundle_into_a_text_edited_at_the_same_moment_goes_in_in_seconds() { + use craft_cloud_proto::fragments::{SLICE_OPS, bundle}; + let importer = LoroDoc::new(); + importer.set_peer_id(40).unwrap(); + text_update(&importer, 0, "import"); + let import = importer.export(ExportMode::Snapshot).unwrap(); + let typist = LoroDoc::new(); + typist.set_peer_id(10).unwrap(); + typist.import(&import).unwrap(); + let typed = text_update(&typist, 0, "hello "); + let paster = LoroDoc::new(); + paster.set_peer_id(50).unwrap(); + paster.import(&import).unwrap(); + let paste = "p".repeat(600 * 1024); + paster.get_text("text").insert(6, &paste).unwrap(); + paster.commit(); + let end = *paster.oplog_vv().get(&50).unwrap(); + let mut slices = Vec::new(); + let mut at = 0; + while at < end { + let to = (at + SLICE_OPS).min(end); + slices.push(paster.export(ExportMode::updates_in_range(vec![IdSpan::new(50, at, to)])).unwrap()); + at = to; + } + let copy = LiveCopy::load(&import, 1).unwrap(); + copy.import(&typed).unwrap(); + let started = std::time::Instant::now(); + assert_eq!(copy.import_any_size(&bundle(&slices)).unwrap(), Taken::Applied { new: true }); + assert!(started.elapsed() < std::time::Duration::from_secs(15), "{:?}", started.elapsed()); + let back = LoroDoc::new(); + back.import(©.snapshot().unwrap()).unwrap(); + assert_eq!(back.get_text("text").to_string(), format!("hello import{paste}")); + } + /// An update Loro refuses leaves the copy as it was (Ruling: validation without a scratch copy). #[test] fn a_refused_update_leaves_the_copy_as_it_was() { diff --git a/apps/craft-cloud-agent/src/live/docs.rs b/apps/craft-cloud-agent/src/live/docs.rs index ecf67fc..67a2ff4 100644 --- a/apps/craft-cloud-agent/src/live/docs.rs +++ b/apps/craft-cloud-agent/src/live/docs.rs @@ -3,8 +3,10 @@ //! turns the app's messages into [`DocsCall`]s for the hub and the hub's [`DocOut`]s into messages for the app. //! //! Everything from an app is hostile: a text message that isn't a request is answered `error`; a binary frame without -//! a header or of a kind an app doesn't send ends the connection; an update over 1 MiB, presence over 4 KiB or an -//! import over the snapshot limit is refused (and never reaches the hub). An app that doesn't take a message within 10 +//! a header or of a kind an app doesn't send ends the connection; an update frame over 1 MiB, presence over 4 KiB or an +//! import over the snapshot limit is refused (and never reaches the hub). A change over 1 MiB comes in fragments +//! (`craft-cloud-proto::fragments`): it reaches the hub only whole, under its last fragment's number; one the app +//! stops sending part of the way never does. An app that doesn't take a message within 10 //! s is dropped, and so is one whose queue at the hub fills up; its documents stay with the agent. An app that goes //! away mid-import leaves nothing behind: the import goes up only once it is whole. Signed out, every `open` is //! answered `not_live {not_signed_in}`. @@ -12,13 +14,14 @@ use std::collections::HashMap; use std::sync::Arc; use std::sync::atomic::{AtomicU64, Ordering}; -use std::time::Duration; +use std::time::{Duration, Instant}; use axum::extract::State; use axum::extract::ws::rejection::WebSocketUpgradeRejection; use axum::extract::ws::{Message, WebSocket, WebSocketUpgrade}; use axum::response::{IntoResponse, Response}; use craft_cloud_proto::api::codes as api_codes; +use craft_cloud_proto::fragments::{MAX_WHOLE_BYTES, PATIENCE, Reassembler, Taken, is_fragment}; use craft_cloud_proto::live::{MAX_PRESENCE_BYTES, MAX_SNAPSHOT_BYTES, MAX_UPDATE_BYTES}; use craft_cloud_proto::local::docs::{IMPORT, MAX_BINARY_BYTES, PRESENCE, STATE, UPDATE, decode_binary, encode_binary}; use craft_cloud_proto::local::{DocsEvent, DocsRequest, MAX_LOCAL_JSON_BYTES, NotLiveReason}; @@ -60,6 +63,10 @@ struct Link { from_hub: Option>, /// The app's updates so far, per document number (their `seq`). seqs: HashMap, + /// The fragments of the app's changes still coming, per document number: a change goes to the hub only whole + /// (under its last fragment's number, which the app's answer names); one the app stops sending part of the way + /// (it went, or crashed) never does. + sets: HashMap, } impl Link { @@ -79,7 +86,7 @@ impl Link { } async fn serve(mut socket: WebSocket, api: Arc) { - let mut link = Link { conn: NEXT_CONN.fetch_add(1, Ordering::Relaxed), hub: None, from_hub: None, seqs: HashMap::new() }; + let mut link = Link { conn: NEXT_CONN.fetch_add(1, Ordering::Relaxed), hub: None, from_hub: None, seqs: HashMap::new(), sets: HashMap::new() }; link.hub(&api).await; let end = loop { tokio::select! { @@ -161,10 +168,26 @@ async fn on_message(socket: &mut WebSocket, link: &mut Link, api: &ApiState, mes UPDATE => { let seq = link.seqs.entry(id).or_insert(0); *seq += 1; + let seq = *seq; if body.len() > MAX_UPDATE_BYTES { - return Ok(send_json(socket, &refuse(Some(*seq), format!("an update is at most {MAX_UPDATE_BYTES} bytes"))).await); + return Ok(send_json(socket, &refuse(Some(seq), format!("an update is at most {MAX_UPDATE_BYTES} bytes"))).await); + } + if is_fragment(body) { + let now = Instant::now(); + let sets = link.sets.entry(id).or_insert_with(|| Reassembler::new(2 * MAX_WHOLE_BYTES)); + // (A set the app stopped sending part of the way: dropped, never taken.) + sets.expire(now, PATIENCE); + match sets.take(body, now) { + Taken::Whole(bytes) => DocsCall::Update { conn: link.conn, id, seq, bytes }, + Taken::Waiting => return Ok(true), + Taken::Refused(why) => { + let event = DocsEvent::Error { id: Some(id), seq: Some(seq), code: api_codes::INVALID.into(), message: why.into() }; + return Ok(send_json(socket, &event).await); + } + } + } else { + DocsCall::Update { conn: link.conn, id, seq, bytes: body.to_vec() } } - DocsCall::Update { conn: link.conn, id, seq: *seq, bytes: body.to_vec() } } PRESENCE => { if body.is_empty() || body.len() > MAX_PRESENCE_BYTES { diff --git a/apps/craft-cloud-agent/src/live/replica.rs b/apps/craft-cloud-agent/src/live/replica.rs index a2cfa70..17529be 100644 --- a/apps/craft-cloud-agent/src/live/replica.rs +++ b/apps/craft-cloud-agent/src/live/replica.rs @@ -77,9 +77,10 @@ use std::collections::{BTreeMap, BTreeSet}; use craft_cloud_proto::api::codes; +use craft_cloud_proto::fragments::{FRAGMENT_OVERHEAD, MAX_SETS, MAX_WHOLE_BYTES, SetId, is_bundle, is_fragment, join, parse, split}; use craft_cloud_proto::live::{ - AppName, ClientFrame, DocKind, LIVE_DOCUMENTS_PROTOCOL, LIVE_PROTOCOL_MAX, LIVE_PROTOCOL_MIN, MAX_SNAPSHOT_BYTES, MAX_SUBSCRIPTIONS, - MAX_UPDATE_BYTES, PROTOCOL_MISMATCH, ServerFrame, node_doc, recents_doc, update_hash, + AppName, ClientFrame, DocKind, LIVE_DOCUMENTS_PROTOCOL, LIVE_FRAGMENTS_PROTOCOL, LIVE_PROTOCOL_MAX, LIVE_PROTOCOL_MIN, MAX_SNAPSHOT_BYTES, + MAX_SUBSCRIPTIONS, MAX_UPDATE_BYTES, PROTOCOL_MISMATCH, ServerFrame, node_doc, recents_doc, update_hash, }; use craft_cloud_proto::local::{NotLiveReason, RecentEntry}; use craft_cloud_proto::{DocId, Epoch, ManifestId, NodeId, SpaceId}; @@ -120,6 +121,37 @@ const DOCUMENT_FILE_FORMAT: u32 = 1; const MAX_JOURNAL_BYTES: usize = 4 * 1024 * 1024; /// A catch-up's updates are taken a batch at a time, at most this many bytes of them at once. const MAX_BATCH_BYTES: usize = 8 * 1024 * 1024; +/// Most bytes of rows of a document's log that are fragments of changes not whole yet, kept until they are (at most +/// [`MAX_SETS`] sets). Past it a row isn't taken (it doesn't count as had) and the document takes nothing more until +/// the next connection, which brings that row again. +/// +/// No time ends a kept set: a server on protocol 3 stores a set whole, so its rows come together, and a connection +/// that ends part of the way through them brings the rest with the next catch-up (the kept rows count as had). Only +/// evidence ends one: a snapshot of the log past its rows (the copy has the change then: no device snapshots inside a +/// set), or the log learnt again from the start. +pub const MAX_FRAGMENT_BYTES: usize = 2 * MAX_WHOLE_BYTES; +/// The most operations a copy may be past the version a compaction asks for and still make its snapshot (Loro checks +/// the document out there, in time growing with the square of the text it takes back). +const MAX_CHECKOUT_OPS: i64 = 64 * 1024; + +/// For tests and simulations: an app's changes over `bytes` go up in fragments of at most `bytes` from now on, in this +/// process (so changes of a few KB do what changes over 1 MiB do). At most the frame limit. +#[doc(hidden)] +pub fn set_frame_limit_for_tests(bytes: usize) { + FRAME_LIMIT.store(bytes.clamp(craft_cloud_proto::fragments::HEADER_BYTES + 1, MAX_UPDATE_BYTES), std::sync::atomic::Ordering::Relaxed); +} + +static FRAME_LIMIT: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(MAX_UPDATE_BYTES); + +/// For simulations: a catch-up's updates are taken a batch at a time once more than `bytes` of them wait, from now +/// on, in this process (so a catch-up of a few KB is taken in parts as one of megabytes is). At most +/// [`MAX_BATCH_BYTES`]. +#[doc(hidden)] +pub fn set_batch_bytes_for_tests(bytes: usize) { + BATCH_BYTES.store(bytes.min(MAX_BATCH_BYTES), std::sync::atomic::Ordering::Relaxed); +} + +static BATCH_BYTES: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(MAX_BATCH_BYTES); /// Every update of the log up to `upto` is in the document's file; `upto_hash` is the hash of the update at `upto`. #[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] @@ -211,6 +243,50 @@ struct DocumentFile { dropped: Option, } +/// What a document of a file's file holds beyond [`DocumentFile`], written after it when there is any (an older agent +/// reads the file and ignores what follows). +#[derive(Debug, Default, Serialize, Deserialize)] +struct DocumentFileMore { + /// [`Doc::fragments`]. + fragments: Vec<(u64, Vec)>, +} + +/// A document of a file's file: [`DocumentFile`], and [`DocumentFileMore`] when it has it. +fn read_document_file(bytes: &[u8]) -> postcard::Result<(DocumentFile, DocumentFileMore)> { + let (file, rest) = postcard::take_from_bytes::(bytes)?; + let more = if rest.is_empty() { DocumentFileMore::default() } else { postcard::from_bytes(rest).unwrap_or_default() }; + Ok((file, more)) +} + +/// The updates `pending` (this machine's unacked updates, in order) make whole: one as it is, a change in fragments +/// once (at its first fragment's place), and none of a change whose fragments aren't all there. +fn wholes(pending: &[Pending]) -> Vec> { + let mut out = Vec::new(); + let mut done = BTreeSet::new(); + for (i, p) in pending.iter().enumerate() { + let Some(f) = parse(&p.bytes) else { + out.push(p.bytes.clone()); + continue; + }; + if done.contains(&f.set) { + continue; + } + let parts: Vec<&[u8]> = + pending.iter().skip(i).filter(|q| parse(&q.bytes).is_some_and(|g| g.set == f.set)).map(|q| q.bytes.as_slice()).collect(); + if let Some(whole) = join(parts) { + done.insert(f.set); + out.push(whole); + } + } + out +} + +/// Imports `bytes` into `body`, an update of any size up to a whole change's (the frame limit is checked where an +/// update comes in, a whole change's when its fragments are joined). +fn import_whole(body: &Body, bytes: &[u8]) -> Result { + if bytes.len() > MAX_UPDATE_BYTES { body.import_any_size(bytes) } else { body.import(bytes) } +} + /// What a file of either kind holds, as [`Doc::from_parts`] reads it. struct Parts { id: DocId, @@ -225,6 +301,7 @@ struct Parts { own_peers: Vec, push_wait: Option, dropped: Option, + fragments: Vec<(u64, Vec)>, } /// One record of a document's journal: a local update, on disk before the app is told it is kept. @@ -547,6 +624,45 @@ struct Doc { deferred_head: Option, /// Snapshots to upload before the frames that name them go ([`Replica::take_uploads`]). uploads: Vec, + /// The rows of the log (taken from the server, or this machine's own acked) that are fragments of a change not + /// whole yet, by set. They count as had; the change goes into the copy (and to the apps) once every fragment of it + /// is here, never before. Kept in its file (see [`MAX_FRAGMENT_BYTES`] for what ends a hold). No snapshot is made + /// for the server at or past a kept set's first row. + fragments: BTreeMap, + /// The server (as of its last `welcome`) is before fragments: unacked ones wait unsent ([`send_pending`]). + whole_only: bool, + /// The change the row just taken made whole, for the apps ([`Doc::take`]). + joined_now: Option>, +} + +/// One set's fragments kept so far: their count, and each index's row number and bytes. +#[derive(Debug, Clone, Default)] +struct KeptSet { + count: u16, + rows: BTreeMap)>, +} + +impl KeptSet { + /// The row its first kept fragment took. + fn first(&self) -> u64 { + self.rows.values().map(|(s, _)| *s).min().unwrap_or(u64::MAX) + } + + fn bytes(&self) -> usize { + self.rows.values().map(|(_, b)| b.len().saturating_add(FRAGMENT_OVERHEAD)).sum() + } +} + +/// What a row that is a fragment did ([`Doc::take_fragment`]). +enum Fragmented { + /// Kept: more of its change is to come. + Waiting, + /// It was the last: the change whole, and the row its first fragment took. + Whole { first: u64, update: Vec }, + /// It doesn't read, or its change's fragments don't make the change they name: never applied. + Broken, + /// Over what a document keeps (bytes or sets): not taken. + Full, } impl Doc { @@ -617,9 +733,100 @@ impl Doc { fetching: None, deferred_head: None, uploads: Vec::new(), + fragments: BTreeMap::new(), + whole_only: false, + joined_now: None, } } + /// Row `seq` of the log is a fragment: kept until its change is whole (a row taken again is kept once). Over what + /// a document keeps, it isn't taken ([`Fragmented::Full`]). + fn take_fragment(&mut self, seq: u64, bytes: &[u8]) -> Fragmented { + let Some(f) = parse(bytes) else { return Fragmented::Broken }; + let known = self.fragments.get(&f.set); + if known.is_some_and(|s| s.rows.contains_key(&f.index)) { + return Fragmented::Waiting; + } + if known.is_none() && self.fragments.len() >= MAX_SETS { + return Fragmented::Full; + } + if self.fragment_bytes().saturating_add(bytes.len() + FRAGMENT_OVERHEAD) > MAX_FRAGMENT_BYTES { + return Fragmented::Full; + } + let set = self.fragments.entry(f.set).or_insert_with(|| KeptSet { count: f.count, ..KeptSet::default() }); + if set.count != f.count { + return Fragmented::Broken; + } + if let std::collections::btree_map::Entry::Vacant(slot) = set.rows.entry(f.index) { + slot.insert((seq, bytes.to_vec())); + self.dirty = true; + } + if set.rows.len() < usize::from(set.count) { + return Fragmented::Waiting; + } + let Some(done) = self.fragments.remove(&f.set) else { return Fragmented::Waiting }; + let first = done.first(); + match join(done.rows.values().map(|(_, b)| b.as_slice())) { + Some(update) => Fragmented::Whole { first, update }, + None => Fragmented::Broken, + } + } + + /// The fragments kept, as rows (for the file). + fn fragment_rows(&self) -> Vec<(u64, Vec)> { + let mut rows: Vec<(u64, Vec)> = self.fragments.values().flat_map(|s| s.rows.values().cloned()).collect(); + rows.sort_by_key(|(s, _)| *s); + rows + } + + /// Kept rows (from the file) back into their sets. + fn keep_rows(&mut self, rows: Vec<(u64, Vec)>) { + for (seq, bytes) in rows { + let Some(f) = parse(&bytes) else { continue }; + let (set, count, index) = (f.set, f.count, f.index); + let kept = self.fragments.entry(set).or_insert_with(|| KeptSet { count, ..KeptSet::default() }); + if kept.count == count { + kept.rows.entry(index).or_insert((seq, bytes)); + } + } + } + + /// Whether a kept set has a row at or below `upto` (a snapshot there would drop part of a change from the log). + fn keeps_at_or_below(&self, upto: u64) -> bool { + self.fragments.values().any(|s| s.first() <= upto) + } + + /// Row `seq` of the log is this machine's own update `bytes` (acked, or taken back): the server holds its + /// operations. An own change in fragments is in the log from its last fragment's row on (it is in the copy + /// already). + fn note_own_row(&mut self, seq: u64, bytes: &[u8]) { + let ranges = if self.kind.is_document() && is_fragment(bytes) { + match self.take_fragment(seq, bytes) { + Fragmented::Whole { first, update } => { + self.log.note_set(first, seq); + self.on_server(&update); + spans_of(&update).unwrap_or_default() + } + Fragmented::Waiting | Fragmented::Broken => Vec::new(), + // Not kept: the change's last row can't say it is in the log, so what the log holds isn't known + // until it is worked out from the copy again (no snapshot is made from a guess that lacks it). + Fragmented::Full => { + self.log = LogVersion::unknown(); + return; + } + } + } else { + spans_of(bytes).unwrap_or_default() + }; + self.saw(&ranges); + self.log.note(seq, ranges); + } + + /// The bytes of rows kept as fragments (with their bookkeeping). + fn fragment_bytes(&self) -> usize { + self.fragments.values().map(KeptSet::bytes).sum() + } + /// The app this document belongs to. fn app(&self) -> &AppName { self.kind.app() @@ -639,7 +846,7 @@ impl Doc { /// which; nothing waits; no push under way; no row dropped). Unknown otherwise, until those are gone. fn derive_log(&mut self) { self.log = LogVersion::unknown(); - if !self.waiting.is_empty() || self.pushing.is_some() || !self.pending.is_empty() || self.dropped.is_some() { + if !self.waiting.is_empty() || self.pushing.is_some() || !self.pending.is_empty() || self.dropped.is_some() || !self.fragments.is_empty() { return; } let Ok(mut version) = self.body.copy().version() else { return }; @@ -712,15 +919,19 @@ impl Doc { /// The answer to the server's `compact`, once this machine holds every update up to `upto`: a snapshot of exactly /// those, or nothing (something waits, it dropped a row at or below `upto`, or the version there isn't known; - /// another device is asked). The question is dropped either way once it can be answered. + /// another device is asked). The question is dropped either way once it can be answered, unless the copy is far + /// past the version asked: then it is kept, and tried again at the next ack, or when the document is refused as + /// full. fn try_compact(&mut self) -> Vec { let Some(upto) = self.compact_ask else { return Vec::new() }; if self.sub != Sub::Live || self.have.upto < upto { return Vec::new(); } self.compact_ask = None; - // (A push under way doesn't stop it: the log was learnt from the start then, and is noted from there.) - if !self.waiting.is_empty() || self.stuck || !self.joined || self.is_broken() { + // (A push under way doesn't stop it: the log was learnt from the start then, and is noted from there.) Nor is + // one made at or past a kept set's first row: the snapshot would lack that change (not whole here), and the + // log would lose its first fragments. A set kept past `upto` is left in the log, and doesn't stop it. + if !self.waiting.is_empty() || self.stuck || !self.joined || self.is_broken() || self.keeps_at_or_below(upto) { return Vec::new(); } // A row it dropped is below: its copy lacks that row, and a snapshot from it would too. @@ -731,6 +942,31 @@ impl Doc { self.derive_log(); } let Some(version) = self.log.at(upto) else { return Vec::new() }; + // Not from a copy far past that version: a snapshot of an earlier one makes Loro check the document out there, + // which for a big paste since (an own change not in the log yet) takes time growing with the square of its + // text, tens of seconds on the hub's thread. Another device answers, or the next ask finds this one there. + let Ok(now) = self.body.copy().version() else { return Vec::new() }; + let mut past: i64 = + now.iter().map(|(peer, end)| i64::from(*end) - i64::from(version.get(peer).copied().unwrap_or(0))).filter(|n| *n > 0).sum(); + // In a document refused as full since this machine's last ack, its own unacked operations don't count: they can't be acked until it is + // compacted, and with them counted the only device there is would never answer (the check-out is slow once, + // instead of the document full for good). Otherwise they are acked soon, and the next ask finds them in. + if past > MAX_CHECKOUT_OPS && self.full_pause > 0 { + for whole in wholes(&self.pending) { + for (peer, start, end) in spans_of(&whole).unwrap_or_default() { + let from = start.max(version.get(&peer).copied().unwrap_or(0)); + if end > from { + past -= i64::from(end) - i64::from(from); + } + } + } + } + if past > MAX_CHECKOUT_OPS { + // Kept: answered once its own change is acked (it is in the log then), or once the document is refused + // as full (its own operations don't count then); the server may ask another device meanwhile. + self.compact_ask = Some(upto); + return Vec::new(); + } match self.body.copy().snapshot_at(&version) { Ok(bytes) if !bytes.is_empty() && bytes.len() <= MAX_SNAPSHOT_BYTES => { // Uploaded first (through the chunk API), then named. @@ -768,6 +1004,7 @@ impl Doc { pushing: file.pushing, own_peers: file.own_peers, push_wait: file.push_wait, + fragments: Vec::new(), }; Self::from_parts(parts, peer, notes) } @@ -775,16 +1012,18 @@ impl Doc { /// [`Doc::from_file`], for a file of either kind. fn from_parts(mut file: Parts, peer: u64, notes: &mut Vec) -> Result { let (mut bad_pending, mut bad_waiting) = (BTreeSet::new(), BTreeSet::new()); + // (Unacked changes in fragments go in whole.) + let own = wholes(&file.pending); let body = 'load: loop { let body = Body::load(&file.kind, &file.snapshot, peer)?; - for (i, pending) in file.pending.iter().enumerate() { - if !bad_pending.contains(&i) && body.import_any_size(&pending.bytes) == Err(UpdateError::Panicked) { + for (i, pending) in own.iter().enumerate() { + if !bad_pending.contains(&i) && body.import_any_size(pending) == Err(UpdateError::Panicked) { bad_pending.insert(i); continue 'load; } } for (i, waiting) in file.waiting.iter().enumerate() { - if !bad_waiting.contains(&i) && body.import(&waiting.bytes) == Err(UpdateError::Panicked) { + if !bad_waiting.contains(&i) && import_whole(&body, &waiting.bytes) == Err(UpdateError::Panicked) { bad_waiting.insert(i); continue 'load; } @@ -818,6 +1057,7 @@ impl Doc { } doc.push_wait = file.push_wait; doc.dropped = file.dropped; + doc.keep_rows(file.fragments); doc.on_disk = true; doc.derive_log(); Ok(doc) @@ -827,8 +1067,8 @@ impl Doc { /// unacked updates, which are kept and sent; it catches up from the start, so nothing the server has is missed. fn from_pending(id: DocId, kind: Kind, pending: Vec, next_client_seq: u64, peer: u64) -> Self { let body = Body::new(&kind, peer); - for p in &pending { - let _ = body.import_any_size(&p.bytes); + for update in wholes(&pending) { + let _ = body.import_any_size(&update); } let pending = pending.into_iter().map(|p| Pending { acked: None, push: false, ..p }).collect(); let mut doc = Self::with(id, kind, body, Have::default(), pending, Vec::new(), next_client_seq.max(1), None, None); @@ -852,13 +1092,60 @@ impl Doc { if self.skip == Some((seq, hash)) { self.skip = None; self.drop_row(seq); + // A fragment Loro failed on (its change whole): the rest of that change goes too. + if let Some(f) = parse(bytes) { + self.fragments.remove(&f.set); + } self.log.note(seq, Vec::new()); self.advance(seq, hash); return Taken::Had; } + // A fragment of a change (a document's): kept, and counted as had, until the change is whole; only then does + // the change go into the copy, and to the apps. + let joined; + let bytes: &[u8] = if self.kind.is_document() && is_fragment(bytes) { + match self.take_fragment(seq, bytes) { + Fragmented::Waiting => { + self.log.note(seq, Vec::new()); + self.advance(seq, hash); + return Taken::Had; + } + // Not taken (not counted as had): the next connection brings the row again. + Fragmented::Full => { + self.stuck = true; + let line = format!( + "fragments of {} changes of the live document {} (or {} MiB of them) wait for the rest; this machine takes no more of them until it connects again", + MAX_SETS, + self.id, + MAX_FRAGMENT_BYTES >> 20 + ); + self.say(notes, line); + return Taken::Stuck; + } + Fragmented::Broken => { + let line = format!( + "a change of the live document {} came in fragments that don't make it; it was skipped, and this machine leaves compacting it to other devices", + self.id + ); + self.say(notes, line); + self.drop_row(seq); + self.log.note(seq, Vec::new()); + self.advance(seq, hash); + return Taken::Had; + } + Fragmented::Whole { first, update } => { + self.log.note_set(first, seq); + joined = update; + self.joined_now = Some(joined.clone()); + &joined + } + } + } else { + bytes + }; let had_history = self.body.copy().has_history(); let mut in_log = true; - match self.body.import(bytes) { + match import_whole(&self.body, bytes) { Ok(Imported::Applied { changed: list }) => { *changed |= list; if !had_history && self.rewait(changed).is_err() { @@ -934,7 +1221,7 @@ impl Doc { /// one Loro failed on (in [`Doc::waiting_failed`]). fn rewait(&mut self, changed: &mut bool) -> Result<(), ()> { for waiting in &self.waiting { - match self.body.import(&waiting.bytes) { + match import_whole(&self.body, &waiting.bytes) { Ok(Imported::Applied { changed: list } | Imported::Waiting { changed: list, .. }) => *changed |= list, Err(UpdateError::Panicked) => { self.waiting_failed = Some(waiting.hash); @@ -979,9 +1266,7 @@ impl Doc { break; }; self.have = Have { upto: next, upto_hash: Some(update_hash(&bytes)) }; - let ranges = spans_of(&bytes).unwrap_or_default(); - self.saw(&ranges); - self.log.note(next, ranges); + self.note_own_row(next, &bytes); } // Own updates the catch-up brought back count as had too. let upto = self.have.upto; @@ -1062,9 +1347,12 @@ impl Doc { }; match part { Ok(Some(bytes)) => { - let client_seq = self.next_client_seq; - self.next_client_seq = client_seq.saturating_add(1); - self.pending.push(Pending { client_seq, bytes, acked: None, push: true }); + let frames = split(&bytes, MAX_UPDATE_BYTES >> self.strikes).unwrap_or_else(|_| vec![bytes]); + for bytes in frames { + let client_seq = self.next_client_seq; + self.next_client_seq = client_seq.saturating_add(1); + self.pending.push(Pending { client_seq, bytes, acked: None, push: true }); + } self.dirty = true; send_pending(id, self) } @@ -1106,13 +1394,25 @@ impl Doc { self.pushing = Some(Spans::new()); self.push_wait = None; self.waiting.clear(); + // (The catch-up brings every row again, fragments too.) + self.fragments.clear(); self.stuck = false; + // The rest of an own set whose first fragments were acked (the log may not hold them any more: a restore) + // goes: sent alone it would never make a set. Its operations are in the copy, and the push sends them. + let whole: BTreeSet = { + let mut by_set: BTreeMap)> = BTreeMap::new(); + for f in self.pending.iter().filter_map(|p| parse(&p.bytes)) { + by_set.entry(f.set).or_insert((f.count, BTreeSet::new())).1.insert(f.index); + } + by_set.into_iter().filter(|(_, (count, got))| got.len() == usize::from(*count)).map(|(set, _)| set).collect() + }; + self.pending.retain(|p| parse(&p.bytes).is_none_or(|f| whole.contains(&f.set))); if let Ok(snapshot) = self.body.copy().snapshot() && let Ok(fresh) = Body::load(&self.kind, &snapshot, self.body.copy().peer()) { self.body = fresh; - for pending in &self.pending { - let _ = self.body.import_any_size(&pending.bytes); + for update in wholes(&self.pending) { + let _ = self.body.import_any_size(&update); } } self.forget_acks(); @@ -1175,6 +1475,10 @@ impl Doc { return Err(UpdateError::Panicked); } self.waiting.retain(|w| !self.body.copy().covers(&w.needs)); + // A kept set from at or below the base is in the snapshot whole (no device snapshots inside a set: the server + // stores one whole, and an answer is never made at or past a kept set's first row): the copy has its change + // now, and its rows go. + self.fragments.retain(|_, set| set.first() > base); self.on_server(bytes); let version = version_of(bytes).unwrap_or_default(); if base >= self.have.upto { @@ -1211,6 +1515,7 @@ impl Doc { self.have = Have::default(); self.log = LogVersion::anchored(0, Version::new()); self.forget_seen(); + self.fragments.clear(); self.compact_ask = None; self.dirty = true; self.urgent = true; @@ -1387,6 +1692,7 @@ impl Replica { && d.open == 0 && d.pending.is_empty() && d.waiting.is_empty() + && d.fragments.is_empty() && d.pushing.is_none() && !d.is_broken() && d.batch.is_empty() @@ -1784,7 +2090,8 @@ impl Replica { pub fn document_state(&self, id: DocId) -> Option<(Vec, Vec>)> { let doc = self.docs.get(&id).filter(|d| d.joined && d.kind.is_document())?; let snapshot = doc.body.copy().snapshot().ok()?; - let after = doc.waiting.iter().map(|w| w.bytes.clone()).chain(doc.pending.iter().map(|p| p.bytes.clone())).collect(); + // (Whole: an unacked change in fragments goes as the change.) + let after = doc.waiting.iter().map(|w| w.bytes.clone()).chain(wholes(&doc.pending)).collect(); Some((snapshot, after)) } @@ -1847,6 +2154,11 @@ impl Replica { Some((online, (doc.pending.len() + usize::from(doc.pushing.is_some())) as u64)) } + /// How many rows of the document's log this machine keeps as fragments of changes not whole here yet. + pub fn kept_fragments(&self, id: DocId) -> usize { + self.docs.get(&id).map_or(0, |d| d.fragments.values().map(|s| s.rows.len()).sum()) + } + /// An app has (`true`) or no longer has (`false`) the document open: one no app has open may be unfollowed past /// the device's cap. pub fn set_open(&mut self, id: DocId, open: bool) { @@ -1863,8 +2175,10 @@ impl Replica { // (After a disk error the copy may be being read back: for now, worth sending again.) return Err(LocalError::Io(std::io::Error::other("the document isn't open here (being read back from disk?)"))); }; - if bytes.is_empty() || bytes.len() > MAX_UPDATE_BYTES { - return Err(LocalError::Invalid(format!("an update is 1 to {MAX_UPDATE_BYTES} bytes"))); + // (An update over the frame limit came from the app in fragments, joined by its connection: it goes into the + // journal whole, and up to the server as fragments again.) + if bytes.is_empty() || bytes.len() > MAX_WHOLE_BYTES { + return Err(LocalError::Invalid(format!("an update is 1 to {MAX_WHOLE_BYTES} bytes"))); } // Its journal needs its file beside it (a journal alone isn't loaded): a document joined since it was last // written is written first (as it is, before this update). So is one whose journal failed: written whole, its @@ -1874,7 +2188,14 @@ impl Replica { save(&mut self.store, space, id, doc)?; doc.journal_failed = false; } - match doc.body.import(bytes) { + // Fragments of it: what goes up (each an update of the log), and its numbers. + let frames = split(bytes, FRAME_LIMIT.load(std::sync::atomic::Ordering::Relaxed)).map_err(|e| LocalError::Invalid(e.to_string()))?; + // A server before fragments would store them one by one, and its other devices couldn't read them. + // Nor a bundle: an agent before them can't read it. + if (frames.len() > 1 || is_bundle(bytes)) && self.link == Link::Up && self.protocol < LIVE_FRAGMENTS_PROTOCOL { + return Err(LocalError::Invalid("a change this big needs a newer Craft Cloud server; it wasn't shared".into())); + } + match import_whole(&doc.body, bytes) { Ok(_) => {} Err(UpdateError::Panicked) => { self.reload(id); @@ -1905,18 +2226,28 @@ impl Replica { return Err(e.into()); } } - doc.next_client_seq = client_seq.saturating_add(1); - doc.pending.push(Pending { client_seq, bytes: bytes.to_vec(), acked: None, push: false }); + // (The journal's record is the whole update under its first number; its fragments take the numbers after.) + let seqs = client_seq..client_seq.saturating_add(frames.len() as u64); + doc.next_client_seq = seqs.end; + for (seq, frame) in seqs.clone().zip(&frames) { + doc.pending.push(Pending { client_seq: seq, bytes: frame.clone(), acked: None, push: false }); + } doc.own(bytes); doc.dirty = true; let (up, epoch) = (self.link == Link::Up, self.epoch); + doc.whole_only = self.protocol < LIVE_FRAGMENTS_PROTOCOL; Ok(match doc.sub { + // A server before fragments: in order behind any fragment or bundle held back ([`send_pending`]). + Sub::Live if doc.full.is_none() && doc.whole_only => send_pending(id, doc), // (During a push too: an app's change goes at once, not behind the push's parts. Its operations count as // in the log for the push once it is acked.) - Sub::Live if doc.full.is_none() => { - doc.sent.insert(client_seq); - vec![ClientFrame::Update { doc: id, client_seq, bytes: bytes.to_vec() }] - } + Sub::Live if doc.full.is_none() => seqs + .zip(frames) + .map(|(client_seq, bytes)| { + doc.sent.insert(client_seq); + ClientFrame::Update { doc: id, client_seq, bytes } + }) + .collect(), Sub::No if up && self.protocol >= LIVE_DOCUMENTS_PROTOCOL && doc.retry.is_none() => { doc.parked = false; subscribe(id, doc, epoch) @@ -1960,7 +2291,9 @@ impl Replica { let mut idle: Vec<(u64, DocId)> = self .docs .iter() - .filter(|(_, d)| d.kind.is_document() && d.joined && !d.parked && d.open == 0 && d.pending.is_empty() && d.pushing.is_none()) + .filter(|(_, d)| { + d.kind.is_document() && d.joined && !d.parked && d.open == 0 && d.pending.is_empty() && d.pushing.is_none() && d.fragments.is_empty() + }) .map(|(id, d)| (d.last_used, *id)) .collect(); idle.sort_unstable(); @@ -2156,8 +2489,10 @@ impl Replica { fn resend_inner(&mut self) -> Vec { let (up, epoch) = (self.link == Link::Up, self.epoch); + let whole_only = self.protocol < LIVE_FRAGMENTS_PROTOCOL; let mut frames = Vec::new(); for (id, doc) in &mut self.docs { + doc.whole_only = whole_only; match doc.sub { Sub::Live => { match doc.full { @@ -2209,6 +2544,15 @@ impl Replica { self.notes.push("live documents need a newer Craft Cloud server; their changes stay on this machine until it is updated".into()); } let mut frames = Vec::new(); + for doc in self.docs.values_mut() { + doc.whole_only = protocol < LIVE_FRAGMENTS_PROTOCOL; + } + if documents && protocol < LIVE_FRAGMENTS_PROTOCOL && self.docs.values().any(|d| d.pending.iter().any(|p| is_fragment(&p.bytes))) { + self.notes.push( + "a big change of a live document needs a newer Craft Cloud server; it stays on this machine until the server is updated" + .into(), + ); + } for (id, doc) in &mut self.docs { // Documents of files only with a server that has them, and only those followed. if doc.kind.is_document() && (!documents || doc.parked || !doc.joined) { @@ -2232,7 +2576,8 @@ impl Replica { if doc.kind.is_document() && doc.sub == Sub::CatchingUp && !doc.stale { doc.batched = doc.batched.saturating_add(bytes.len()); doc.batch.push((seq, bytes)); - return Ok(if doc.batched > MAX_BATCH_BYTES { self.take_batch(id) } else { Vec::new() }); + let max = BATCH_BYTES.load(std::sync::atomic::Ordering::Relaxed); + return Ok(if doc.batched > max { self.take_batch(id) } else { Vec::new() }); } Ok(self.on_update(id, seq, bytes)) } @@ -2347,12 +2692,20 @@ impl Replica { if batch.is_empty() { return Vec::new(); } + // What the copy had before: a row with anything past it goes to the apps (one Loro holds back for its history + // too: taken again on its own after the batch, Loro has it already and says nothing of it). + let before = doc.body.copy().version().unwrap_or_default(); let new = if one_by_one { // (Each row's own import says what it brought.) false } else { - let updates: Vec> = - batch.iter().filter(|(seq, bytes)| doc.skip != Some((*seq, update_hash(bytes)))).map(|(_, bytes)| bytes.clone()).collect(); + // (Fragments are taken row by row: their change goes in once it is whole. So are bundles: their slices go + // in one after the other, which Loro's batch import can't do.) + let updates: Vec> = batch + .iter() + .filter(|(seq, bytes)| doc.skip != Some((*seq, update_hash(bytes))) && !is_fragment(bytes) && !is_bundle(bytes)) + .map(|(_, bytes)| bytes.clone()) + .collect(); match doc.body.copy().import_batch(&updates) { Ok(new) => new, Err(UpdateError::Panicked) => return self.take_one_by_one(id), @@ -2364,8 +2717,9 @@ impl Replica { if new && let Kind::Recents(app) = &kind { self.changed.insert(app.clone()); } + let past = |bytes: &[u8]| spans_of(bytes).is_none_or(|r| r.iter().any(|(peer, _, end)| *end > before.get(peer).copied().unwrap_or(0))); for (seq, bytes) in batch { - if new && kind.is_document() { + if !one_by_one && kind.is_document() && !is_fragment(&bytes) && !is_bundle(&bytes) && past(&bytes) { self.events.push(DocEvent::Remote(id, bytes.clone())); } frames.extend(self.on_update(id, seq, bytes)); @@ -2403,6 +2757,8 @@ impl Replica { } let mut changed = false; let taken = doc.take(seq, &bytes, &mut self.notes, &mut changed); + // (A row that made a change in fragments whole: the apps get the change, never a fragment.) + let joined = doc.joined_now.take(); let kind = doc.kind.clone(); let mut frames = if matches!(taken, Taken::Had) { doc.try_compact() } else { Vec::new() }; if changed { @@ -2410,7 +2766,7 @@ impl Replica { Kind::Recents(app) => { self.changed.insert(app.clone()); } - Kind::Document { .. } => self.events.push(DocEvent::Remote(id, bytes.clone())), + Kind::Document { .. } => self.events.push(DocEvent::Remote(id, joined.unwrap_or_else(|| bytes.clone()))), } } frames.extend(match taken { @@ -2464,6 +2820,8 @@ impl Replica { doc.dirty = true; self.events.push(DocEvent::Joined(id)); } + // (From the link, not from the document: one read back since the welcome has it right too.) + doc.whole_only = self.protocol < LIVE_FRAGMENTS_PROTOCOL; let mut frames = send_pending(id, doc); frames.extend(doc.push_next(id, &mut self.notes, &mut self.peers)); frames.extend(doc.try_compact()); @@ -2471,6 +2829,9 @@ impl Replica { } /// `ack {doc, client_seq, seq}`: one of this machine's updates is in the log. + /// (A fragment's ack is for it alone. One whose set's other acks were lost with a connection sends the rest again: + /// the server finds the fragments it stored already, by their numbers, and acks the set whole with the numbers it + /// has.) fn on_ack(&mut self, id: DocId, client_seq: u64, seq: u64) -> Vec { let Some(doc) = self.docs.get_mut(&id) else { return Vec::new() }; let Some(i) = doc.pending.iter().position(|p| p.client_seq == client_seq) else { @@ -2478,9 +2839,7 @@ impl Replica { // row holds them, and counts as had like any ack (or waits for the rows below it, as any ack does). let Some(bytes) = doc.taken_back.remove(&client_seq) else { return Vec::new() }; if seq == doc.have.upto.saturating_add(1) { - let ranges = spans_of(&bytes).unwrap_or_default(); - doc.saw(&ranges); - doc.log.note(seq, ranges); + doc.note_own_row(seq, &bytes); doc.advance(seq, update_hash(&bytes)); } else if seq > doc.have.upto { doc.back_acked.insert(seq, bytes); @@ -2502,9 +2861,7 @@ impl Replica { } else if seq == doc.have.upto.saturating_add(1) { let own = doc.pending.remove(i); doc.sent.remove(&client_seq); - let ranges = spans_of(&own.bytes).unwrap_or_default(); - doc.saw(&ranges); - doc.log.note(seq, ranges); + doc.note_own_row(seq, &own.bytes); doc.advance(seq, update_hash(&own.bytes)); } else if let Some(pending) = doc.pending.get_mut(i) { // Others' updates below it are on their way: counted once they are here. (A push part counts as @@ -2688,9 +3045,19 @@ impl Replica { doc.say(&mut self.notes, line); return Ok(Vec::new()); } - // Its operations go up in the parts of a push (of the whole document, if no push is under way). - doc.pending.retain(|p| p.client_seq != client_seq); - doc.sent.remove(&client_seq); + // Its operations go up in the parts of a push (of the whole document, if no push is under way). A + // fragment takes the rest of its change with it: the push sends the change whole again. + let set = doc.pending.iter().find(|p| p.client_seq == client_seq).and_then(|p| parse(&p.bytes)).map(|f| f.set); + let gone: Vec = doc + .pending + .iter() + .filter(|p| p.client_seq == client_seq || set.is_some_and(|s| parse(&p.bytes).is_some_and(|f| f.set == s))) + .map(|p| p.client_seq) + .collect(); + doc.pending.retain(|p| !gone.contains(&p.client_seq)); + for seq in gone { + doc.sent.remove(&seq); + } doc.pushing.get_or_insert_with(Spans::new); doc.dirty = true; Ok(doc.push_next(id, &mut self.notes, &mut self.peers)) @@ -2708,7 +3075,9 @@ impl Replica { |id| format!("the live document {id} is full on the server for now ({message}); new changes stay on this machine"), ); doc.say(&mut self.notes, line); - Ok(Vec::new()) + // A compaction asked of it that it left for later (far past the version asked) is answered now: the + // document gets below its cap only so, and this may be the only device there is. + Ok(doc.try_compact()) } // The update at `after` isn't the one this machine has there: the log was restored (its epoch reset hasn't // run). From the start, and the whole document up again. @@ -2716,6 +3085,26 @@ impl Replica { doc.start_over(); Ok(subscribe(id, doc, epoch)) } + // A fragment of a set whose first part the log's snapshot holds alone (a device from before whole sets + // compacted past it): the set goes from what waits, and the document starts over, so its operations go + // up as a push. (Once: the set's other fragments, refused the same way, aren't waiting any more.) + (codes::RESYNC_REQUIRED, Some((id, doc)), Some(client_seq)) => { + let Some(set) = doc.pending.iter().find(|p| p.client_seq == client_seq).and_then(|p| parse(&p.bytes)).map(|f| f.set) else { + return Ok(Vec::new()); + }; + doc.pending.retain(|p| parse(&p.bytes).is_none_or(|f| f.set != set)); + let line = + format!("a change of the live document {id} was stored in part by an older Craft Cloud; this machine sends it again whole"); + doc.say(&mut self.notes, line); + doc.dirty = true; + // (A start over under way, its push to come or going, sends these operations too: not another one, + // whose subscribe the server would refuse while the first one's catch-up runs.) + if doc.pushing.is_some() { + return Ok(Vec::new()); + } + doc.start_over(); + Ok(subscribe(id, doc, epoch)) + } // A create refused because the account is at its live-document limit (the server's cap): tried again after a // growing pause, and the recents stay here until then. The link stays up for the other lists. (codes::QUOTA_EXCEEDED, Some((_, doc)), None) => { @@ -2823,9 +3212,9 @@ impl Replica { /// the file or its journal), nothing waiting and no push under way. A file that doesn't read is loaded (and so set /// aside) as before. fn idle_on_disk(store: &S, space: SpaceId, name: &str, bytes: &[u8]) -> Option { - let file = postcard::from_bytes::(bytes).ok()?; + let (file, more) = read_document_file(bytes).ok()?; let ours = file.format <= DOCUMENT_FILE_FORMAT && file.space == space && file.doc == node_doc(space, file.node) && file.doc.to_string() == name; - let idle = file.pending.is_empty() && file.waiting.is_empty() && file.pushing.is_none(); + let idle = file.pending.is_empty() && file.waiting.is_empty() && file.pushing.is_none() && more.fragments.is_empty(); let journal = store.load_journal(name).ok()?; (ours && idle && journal_records(&journal).0.is_empty()).then_some(file.doc) } @@ -2906,8 +3295,8 @@ fn load_document( Ok(to) => (format!("a copy set aside as {to}"), true), Err(e) => (format!("no copy set aside ({e})"), false), }; - match postcard::from_bytes::(bytes) { - Ok(mut file) if file.format <= DOCUMENT_FILE_FORMAT && file.space == space && file.doc == node_doc(space, file.node) => { + match read_document_file(bytes) { + Ok((mut file, more)) if file.format <= DOCUMENT_FILE_FORMAT && file.space == space && file.doc == node_doc(space, file.node) => { if let Some(hash) = drop { file.waiting.retain(|w| w.hash != hash); } @@ -2925,8 +3314,14 @@ fn load_document( } for record in records { if record.client_seq >= next { - next = record.client_seq.saturating_add(1); - pending.push(Pending { client_seq: record.client_seq, bytes: record.bytes, acked: None, push: false }); + // A change over the frame limit is in the journal whole, under the first of its fragments' numbers: + // cut as it was cut when it was sent (the same fragments, the same numbers). + let limit = FRAME_LIMIT.load(std::sync::atomic::Ordering::Relaxed); + let frames = split(&record.bytes, limit).unwrap_or_else(|_| vec![record.bytes.clone()]); + for (k, bytes) in (0u64..).zip(frames) { + pending.push(Pending { client_seq: record.client_seq.saturating_add(k), bytes, acked: None, push: false }); + next = record.client_seq.saturating_add(k + 1); + } journalled += 1; } } @@ -2943,6 +3338,7 @@ fn load_document( own_peers: file.own_peers, push_wait: file.push_wait, dropped: file.dropped, + fragments: more.fragments, }; match Doc::from_parts(parts, peer, notes) { Ok(mut doc) => { @@ -3042,6 +3438,11 @@ fn send_pending(id: DocId, doc: &mut Doc) -> Vec { return frames; } for pending in &doc.pending { + // Fragments and bundles wait for a server that takes them, and everything after them too: a later change + // builds on them, and would wait on every other machine for history that never came. + if doc.whole_only && (is_fragment(&pending.bytes) || is_bundle(&pending.bytes)) { + break; + } // While a push is under way, only its parts go: they carry every other unacked update. if (pending.push || doc.pushing.is_none()) && doc.sent.insert(pending.client_seq) { frames.push(ClientFrame::Update { doc: id, client_seq: pending.client_seq, bytes: pending.bytes.clone() }); @@ -3112,7 +3513,11 @@ fn save_as(store: &mut S, space: SpaceId, id: DocId, doc: &mut Doc, log_only: doc.log_only, dropped: doc.dropped, }; - let bytes = postcard::to_stdvec(&file).map_err(std::io::Error::other)?; + let mut bytes = postcard::to_stdvec(&file).map_err(std::io::Error::other)?; + // (Nothing more to say without fragments: the file is as an older agent writes it.) + if !doc.fragments.is_empty() { + bytes.extend(postcard::to_stdvec(&DocumentFileMore { fragments: doc.fragment_rows() }).map_err(std::io::Error::other)?); + } let name = id.to_string(); match patience { Patience::Wait => store.save_document(&name, &bytes)?, @@ -3606,7 +4011,7 @@ mod tests { let import = importer.export(loro::ExportMode::Snapshot).unwrap(); let mut r = Replica::open(store.clone(), space, peers()).unwrap(); r.connected(); - r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); let (id, _, _) = r.open_document(node, &app()); snapshot_in(&mut r, id, 0, &import); r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); @@ -3629,7 +4034,7 @@ mod tests { let (space, node) = (SpaceId::generate(), NodeId::generate()); let mut r = Replica::open(MemStore::default(), space, peers()).unwrap(); r.connected(); - r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); let (id, _, _) = r.open_document(node, &app()); let server = text_doc(40); insert(&server, 0, "import"); @@ -3698,7 +4103,7 @@ mod tests { let nodes: Vec = (0..300).map(|_| NodeId::generate()).collect(); let mut r = Replica::open(store.clone(), space, peers()).unwrap(); r.connected(); - r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); let mut ids = Vec::new(); for node in &nodes { let (id, _, _) = r.open_document(*node, &app()); @@ -3718,7 +4123,7 @@ mod tests { let mut r = Replica::open(store.clone(), space, peers()).unwrap(); assert_eq!(r.loaded_documents(), 1, "only the one with something to send"); r.connected(); - r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); let (_, opening, frames) = r.open_document(nodes[7], &app()); assert_eq!(opening, Opening::Live, "loaded when opened, from disk"); assert!(frames.iter().any(|f| matches!(f, ClientFrame::Subscribe { doc, .. } if *doc == ids[7])), "{frames:?}"); @@ -3739,7 +4144,7 @@ mod tests { let (space, node) = (SpaceId::generate(), NodeId::generate()); let mut r = Replica::open(MemStore::default(), space, peers()).unwrap(); r.connected(); - r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); let (id, opening, _) = r.open_document(node, &app()); assert_eq!(opening, Opening::Joining); let frames = r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); @@ -3769,7 +4174,7 @@ mod tests { let import = app_doc.export(loro::ExportMode::Snapshot).unwrap(); let mut r = Replica::open(store.clone(), space, peers()).unwrap(); r.connected(); - r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); let (id, _, _) = r.open_document(node, &app()); r.on_frame(ServerFrame::error("not_found", "no such document", Some(id), None)).unwrap(); assert!(r.take_events().contains(&DocEvent::NeedsImport(id))); @@ -3784,7 +4189,7 @@ mod tests { // The server is restored from a backup that has none of it. r.disconnected(); r.connected(); - let frames = r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(2) }).unwrap(); + let frames = r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(2) }).unwrap(); assert!(frames.iter().any(|f| matches!(f, ClientFrame::Subscribe { after: 0, .. })), "{frames:?}"); let frames = r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); assert!(frames.iter().any(|f| matches!(f, ClientFrame::Update { .. })), "its own goes up at once: {frames:?}"); @@ -3878,7 +4283,7 @@ mod tests { let (mut r, id, import) = joined(&store, space, node); r.disconnected(); r.connected(); - r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); let other = text_doc(60); other.import(&import).unwrap(); let big = "y".repeat(1_000_000); @@ -3965,7 +4370,7 @@ mod tests { fn a_row_loro_fails_on_in_a_batch_is_found_and_only_it_dropped() { let space = SpaceId::generate(); let node = NodeId::generate(); - let w2 = ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }; + let w2 = ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }; let importer = text_doc(40); insert(&importer, 0, "import"); let import = importer.export(loro::ExportMode::Snapshot).unwrap(); @@ -4485,6 +4890,23 @@ mod tests { d.export(loro::ExportMode::updates(&vv)).unwrap() } + /// `s` inserted at the start, as an app's kit sends a big change: its slices in a bundle. + fn sliced(d: &loro::LoroDoc, s: &str) -> Vec { + let peer = d.peer_id(); + let start = d.oplog_vv().get(&peer).copied().unwrap_or(0); + d.get_text("text").insert(0, s).unwrap(); + d.commit(); + let end = d.oplog_vv().get(&peer).copied().unwrap_or(0); + let mut slices = Vec::new(); + let mut at = start; + while at < end { + let to = (at + craft_cloud_proto::fragments::SLICE_OPS).min(end); + slices.push(d.export(loro::ExportMode::updates_in_range(vec![loro::IdSpan::new(peer, at, to)])).unwrap()); + at = to; + } + craft_cloud_proto::fragments::bundle(&slices) + } + /// An own update the server stored (its ack lost), whose operations the server's snapshot holds on the next /// catch-up: it is stored (not sent again, nothing waits), and a compaction after a restart has it, though another /// machine's later rows don't build on it. @@ -4493,7 +4915,7 @@ mod tests { let store = MemStore::default(); let space = SpaceId::generate(); let node = NodeId::generate(); - let w2 = ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }; + let w2 = ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }; // The import (peer 40), this machine's app edit P (peer 50), another machine's concurrent edits (peer 60). let importer = text_doc(40); insert(&importer, 0, "import"); @@ -4577,7 +4999,7 @@ mod tests { let store = MemStore::default(); let space = SpaceId::generate(); let node = NodeId::generate(); - let w2 = ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }; + let w2 = ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }; let importer = text_doc(40); insert(&importer, 0, "import"); let import = importer.export(loro::ExportMode::Snapshot).unwrap(); @@ -4652,4 +5074,383 @@ mod tests { let u4 = other.add([5; 16], &entry("c.docx", 4)).unwrap(); assert!(r.on_frame(ServerFrame::Update { doc, seq: 4, bytes: u4, device: device() }).unwrap().is_empty(), "no gap at 3"); } + + /// A change in fragments from the server goes into the copy, and to the apps, only once it is whole: its first + /// fragments count as had and stay in the file across a restart (the catch-up goes on after them), and no + /// snapshot is made for the server while it comes, nor at a number between its first fragment and its last. + #[test] + fn a_change_in_fragments_goes_in_only_whole_even_across_a_restart() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + r.take_events(); + let other = text_doc(41); + other.import(&import).unwrap(); + let small = insert(&other, 0, "s "); + let big = sliced(&other, &"b".repeat(2 * MAX_UPDATE_BYTES)); + let frames = split(&big, MAX_UPDATE_BYTES).unwrap(); + assert_eq!(frames.len(), 3); + let row = |seq: u64, bytes: &[u8]| ServerFrame::Update { doc: id, seq, bytes: bytes.to_vec(), device: device() }; + r.on_frame(row(1, &small)).unwrap(); + r.on_frame(row(2, &frames[0])).unwrap(); + r.on_frame(row(3, &frames[1])).unwrap(); + assert_eq!(text_of(&r, id), "s import", "nothing of the change in the copy"); + let events = r.take_events(); + assert_eq!(events.iter().filter(|e| matches!(e, DocEvent::Remote(..))).count(), 1, "only the small change went to the apps: {events:?}"); + assert!(r.on_frame(ServerFrame::Compact { doc: id, upto: 3 }).unwrap().is_empty(), "no snapshot while it comes"); + // The agent stops and starts again: the fragments are in the file, and counted as had. + r.flush_documents().unwrap(); + drop(r); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + let mut frames_out = r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); + let (_, opening, opened) = r.open_document(node, &app()); + assert_eq!(opening, Opening::Live); + frames_out.extend(opened); + assert!(frames_out.iter().any(|f| matches!(f, ClientFrame::Subscribe { after: 3, .. })), "{frames_out:?}"); + r.on_frame(row(4, &frames[2])).unwrap(); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 4 }).unwrap(); + assert_eq!(text_of(&r, id), format!("{}s import", "b".repeat(2 * MAX_UPDATE_BYTES)), "whole once its last fragment came"); + assert!(r.take_events().contains(&DocEvent::Remote(id, big.clone())), "the apps get the change, whole"); + let last = u32::try_from(2 + 2 * MAX_UPDATE_BYTES - 1).unwrap(); + assert!(r.document_holds(id, &[(41, last)])); + // No snapshot between its first fragment and its last; one past it holds it. + assert!(r.on_frame(ServerFrame::Compact { doc: id, upto: 3 }).unwrap().is_empty(), "not inside the change's rows"); + let frames_out = r.on_frame(ServerFrame::Compact { doc: id, upto: 4 }).unwrap(); + let manifest = frames_out + .iter() + .find_map(|f| if let ClientFrame::Snapshot { upto: 4, manifest, .. } = f { Some(*manifest) } else { None }) + .expect("answered at 4"); + let fresh = loro::LoroDoc::new(); + fresh.import(&uploaded(&mut r, manifest)).unwrap(); + assert_eq!(fresh.get_text("text").len_utf8(), 2 * MAX_UPDATE_BYTES + "s import".len()); + } + + /// A catch-up's batch whose later row Loro holds back for its history (a change in fragments below it, not whole + /// until its rows are taken one by one): the row still goes to the apps, after the change it builds on. + #[test] + fn a_batched_row_loro_holds_back_reaches_the_apps() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let other = text_doc(41); + other.import(&import).unwrap(); + let big = sliced(&other, &"b".repeat(2 * MAX_UPDATE_BYTES)); + let frames = split(&big, MAX_UPDATE_BYTES).unwrap(); + let after = insert(&other, 0, "after "); + r.disconnected(); + r.connected(); + r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); + r.take_events(); + let row = |seq: u64, bytes: &[u8]| ServerFrame::Update { doc: id, seq, bytes: bytes.to_vec(), device: device() }; + for (k, f) in frames.iter().enumerate() { + r.on_frame(row(k as u64 + 1, f)).unwrap(); + } + r.on_frame(row(frames.len() as u64 + 1, &after)).unwrap(); + r.on_frame(ServerFrame::Subscribed { doc: id, head: frames.len() as u64 + 1 }).unwrap(); + let events = r.take_events(); + let remote: Vec<&Vec> = events.iter().filter_map(|e| if let DocEvent::Remote(_, b) = e { Some(b) } else { None }).collect(); + let sizes: Vec = remote.iter().map(|b| b.len()).collect(); + assert!(remote == vec![&big, &after], "the change, then the row after it: {sizes:?}"); + assert!(text_of(&r, id).starts_with("after b")); + } + + /// The rest of a set refused with `resync_required` (its first part is in the server's snapshot alone): the set + /// goes from what waits, once, and the document starts over, its operations going up as a push. + #[test] + fn a_set_refused_for_a_start_over_goes_as_a_push() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let app_doc = text_doc(50); + app_doc.import(&import).unwrap(); + let big = sliced(&app_doc, &"a".repeat(2 * MAX_UPDATE_BYTES)); + assert_eq!(r.local_update(id, &big).unwrap().len(), 3); + r.on_frame(ServerFrame::Ack { doc: id, client_seq: 1, seq: 1 }).unwrap(); + let refusal = |client_seq| ServerFrame::Error { + code: codes::RESYNC_REQUIRED.into(), + message: "part of it in the snapshot".into(), + doc: Some(id), + client_seq: Some(client_seq), + }; + let out = r.on_frame(refusal(2)).unwrap(); + assert!(out.iter().any(|f| matches!(f, ClientFrame::Subscribe { after: 0, .. })), "started over: {out:?}"); + let doc = r.docs.get(&id).unwrap(); + assert!(doc.pending.iter().all(|p| !is_fragment(&p.bytes)), "the set no longer waits"); + assert!(doc.pushing.is_some(), "its operations go as a push"); + assert!(r.on_frame(refusal(3)).unwrap().is_empty(), "once"); + } + + /// A fragment's ack is for it alone: the rest of the set, their acks lost with the connection, goes again (the + /// server finds what it stored of the set by its numbers, and acks it whole with them), and nothing waits once + /// those acks come. + #[test] + fn the_rest_of_a_set_whose_acks_were_lost_goes_again() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let app_doc = text_doc(50); + app_doc.import(&import).unwrap(); + let big = sliced(&app_doc, &"a".repeat(2 * MAX_UPDATE_BYTES)); + let sent = r.local_update(id, &big).unwrap(); + assert_eq!(sent.len(), 3); + r.on_frame(ServerFrame::Ack { doc: id, client_seq: 1, seq: 1 }).unwrap(); + assert_eq!(r.document_status(id), Some((true, 2)), "the rest waits"); + r.disconnected(); + r.connected(); + let mut out = r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); + out.extend(r.on_frame(ServerFrame::Subscribed { doc: id, head: 1 }).unwrap()); + let again: Vec = + out.iter().filter_map(|f| if let ClientFrame::Update { client_seq, .. } = f { Some(*client_seq) } else { None }).collect(); + assert_eq!(again, vec![2, 3], "{out:?}"); + for seq in [2, 3] { + r.on_frame(ServerFrame::Ack { doc: id, client_seq: seq, seq }).unwrap(); + } + assert_eq!(r.document_status(id), Some((true, 0)), "nothing waits"); + let answer = r.on_frame(ServerFrame::Compact { doc: id, upto: 3 }).unwrap(); + assert!(answer.iter().any(|f| matches!(f, ClientFrame::Snapshot { upto: 3, .. })), "its rows are in the log: {answer:?}"); + } + + /// An app's change over the frame limit: imported whole, journalled whole, and sent as fragments that fit a frame + /// (each its own number). After a crash before the file is written, the journal brings back the fragments to send + /// with the same numbers, and an app opening the document gets the change whole. + #[test] + fn a_local_change_over_a_frame_is_journalled_whole_and_sent_in_fragments() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let app_doc = text_doc(50); + app_doc.import(&import).unwrap(); + let big = sliced(&app_doc, &"a".repeat(3 * MAX_UPDATE_BYTES)); + let frames = r.local_update(id, &big).unwrap(); + let sent: Vec<(u64, usize)> = frames + .iter() + .filter_map(|f| if let ClientFrame::Update { client_seq, bytes, .. } = f { Some((*client_seq, bytes.len())) } else { None }) + .collect(); + assert_eq!(sent.iter().map(|(s, _)| *s).collect::>(), vec![1, 2, 3, 4], "{sent:?}"); + assert!(sent.iter().all(|(_, len)| *len <= MAX_UPDATE_BYTES)); + assert!(r.document_holds(id, &[(50, 1)])); + // A crash: the journal has it, the file doesn't. + drop(r); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); + let (_, opening, _) = r.open_document(node, &app()); + assert_eq!(opening, Opening::Live); + let (_, after) = r.document_state(id).unwrap(); + assert!(after.contains(&big), "an app opening it gets the change whole"); + assert!(r.document_holds(id, &[(50, 1)]), "in the copy again"); + let frames = r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); + let again: Vec = + frames.iter().filter_map(|f| if let ClientFrame::Update { client_seq, .. } = f { Some(*client_seq) } else { None }).collect(); + assert_eq!(again, vec![1, 2, 3, 4], "the same fragments, the same numbers: {frames:?}"); + for (k, seq) in again.iter().enumerate() { + r.on_frame(ServerFrame::Ack { doc: id, client_seq: *seq, seq: k as u64 + 1 }).unwrap(); + } + assert_eq!(r.document_status(id), Some((true, 0)), "nothing waits"); + let frames_out = r.on_frame(ServerFrame::Compact { doc: id, upto: 4 }).unwrap(); + assert!(frames_out.iter().any(|f| matches!(f, ClientFrame::Snapshot { upto: 4, .. })), "its rows are in the log: {frames_out:?}"); + } + + /// Asked to compact while a big change of its own isn't in the log yet, a machine doesn't answer: the snapshot + /// would have Loro check the document out before the change, which takes tens of seconds for a big paste. Once + /// the change is in the log, it answers. + #[test] + fn no_snapshot_from_a_copy_far_past_the_asked_version() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let app_doc = text_doc(50); + app_doc.import(&import).unwrap(); + let small = insert(&app_doc, 0, "s "); + r.on_frame(ServerFrame::Update { doc: id, seq: 1, bytes: small, device: device() }).unwrap(); + let big = sliced(&app_doc, &"a".repeat(2 * MAX_UPDATE_BYTES)); + let sent = r.local_update(id, &big).unwrap(); + let started = std::time::Instant::now(); + assert!(r.on_frame(ServerFrame::Compact { doc: id, upto: 1 }).unwrap().is_empty(), "not answered"); + assert!(started.elapsed() < std::time::Duration::from_secs(2), "{:?}", started.elapsed()); + let seqs: Vec = + sent.iter().filter_map(|f| if let ClientFrame::Update { client_seq, .. } = f { Some(*client_seq) } else { None }).collect(); + for (k, seq) in seqs.iter().enumerate() { + r.on_frame(ServerFrame::Ack { doc: id, client_seq: *seq, seq: k as u64 + 2 }).unwrap(); + } + let upto = seqs.len() as u64 + 1; + let frames = r.on_frame(ServerFrame::Compact { doc: id, upto }).unwrap(); + assert!(frames.iter().any(|f| matches!(f, ClientFrame::Snapshot { upto: u, .. } if *u == upto)), "{frames:?}"); + } + + /// In a document the server refused as full, a machine far past the asked version answers anyway: its own big + /// change can't be acked until the document is compacted, and with only this device it would never be. + #[test] + fn a_full_document_is_compacted_even_from_a_copy_far_past_the_asked_version() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let app_doc = text_doc(50); + app_doc.import(&import).unwrap(); + let small = insert(&app_doc, 0, "s "); + r.on_frame(ServerFrame::Update { doc: id, seq: 1, bytes: small, device: device() }).unwrap(); + let paste = sliced(&app_doc, &"a".repeat(80 * 1024)); + let sent = r.local_update(id, &paste).unwrap(); + assert!(r.on_frame(ServerFrame::Compact { doc: id, upto: 1 }).unwrap().is_empty(), "not answered while it may be acked"); + // Refused as full: the ask it kept is answered then (no other ask comes: nothing more is stored). + let mut frames = Vec::new(); + for f in &sent { + if let ClientFrame::Update { client_seq, .. } = f { + let full = + ServerFrame::Error { code: codes::QUOTA_EXCEEDED.into(), message: "full".into(), doc: Some(id), client_seq: Some(*client_seq) }; + frames.extend(r.on_frame(full).unwrap()); + } + } + let manifest = frames + .iter() + .find_map(|f| if let ClientFrame::Snapshot { upto: 1, manifest, .. } = f { Some(*manifest) } else { None }) + .expect("answered at 1"); + let fresh = loro::LoroDoc::new(); + fresh.import(&uploaded(&mut r, manifest)).unwrap(); + assert_eq!(fresh.get_text("text").to_string(), "s import", "the log's rows only, not the unacked paste"); + } + + /// A receiver that slept part of the way through a change catches up from a snapshot past it (another device + /// compacted meanwhile): the copy has the change from the snapshot, and the fragments kept go at once, so they + /// stop nothing (a compaction past them is answered). + #[test] + fn a_snapshot_past_a_kept_set_ends_it() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let other = text_doc(41); + other.import(&import).unwrap(); + let small = insert(&other, 0, "s "); + let big = sliced(&other, &"b".repeat(2 * MAX_UPDATE_BYTES)); + let frames = split(&big, MAX_UPDATE_BYTES).unwrap(); + let row = |seq: u64, bytes: &[u8]| ServerFrame::Update { doc: id, seq, bytes: bytes.to_vec(), device: device() }; + r.on_frame(row(1, &small)).unwrap(); + r.on_frame(row(2, &frames[0])).unwrap(); + r.flush_documents().unwrap(); + drop(r); + // Asleep: rows 3 and 4 came, and another device compacted at 4. + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); + r.open_document(node, &app()); + assert_eq!(r.docs.get(&id).unwrap().fragments.len(), 1, "the first fragment kept across the restart"); + let snap = other.export(loro::ExportMode::Snapshot).unwrap(); + snapshot_in(&mut r, id, 4, &snap); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 4 }).unwrap(); + assert!(r.docs.get(&id).unwrap().fragments.is_empty(), "the snapshot holds the change: its fragments go"); + assert!(text_of(&r, id).starts_with(&"b".repeat(2 * MAX_UPDATE_BYTES)), "the change is in the copy"); + let after = insert(&other, 0, "t "); + r.on_frame(row(5, &after)).unwrap(); + let answer = r.on_frame(ServerFrame::Compact { doc: id, upto: 5 }).unwrap(); + assert!(answer.iter().any(|f| matches!(f, ClientFrame::Snapshot { upto: 5, .. })), "{answer:?}"); + } + + /// A sender that slept part of the way through sending a change (an old server stored what came; this one + /// stores a set whole, but a receiver's connection can end inside it too): the receiver keeps what came for as + /// long as it takes (no time ends it), answers no compaction at or past its first row meanwhile, answers one + /// below it, and takes the change whole when the rest comes, after a restart and a new connection. + #[test] + fn a_kept_set_waits_for_its_rest_however_long_it_takes() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let other = text_doc(41); + other.import(&import).unwrap(); + let small = insert(&other, 0, "s "); + let big = sliced(&other, &"b".repeat(2 * MAX_UPDATE_BYTES)); + let frames = split(&big, MAX_UPDATE_BYTES).unwrap(); + assert_eq!(frames.len(), 3); + let row = |seq: u64, bytes: &[u8]| ServerFrame::Update { doc: id, seq, bytes: bytes.to_vec(), device: device() }; + r.on_frame(row(1, &small)).unwrap(); + r.on_frame(row(2, &frames[0])).unwrap(); + r.on_frame(row(3, &frames[1])).unwrap(); + for _ in 0..5000 { + r.resend(); + } + assert_eq!(r.docs.get(&id).unwrap().fragments.values().map(|s| s.rows.len()).sum::(), 2, "still kept"); + assert!(r.on_frame(ServerFrame::Compact { doc: id, upto: 3 }).unwrap().is_empty(), "not at or past its first row"); + let below = r.on_frame(ServerFrame::Compact { doc: id, upto: 1 }).unwrap(); + assert!(below.iter().any(|f| matches!(f, ClientFrame::Snapshot { upto: 1, .. })), "below it: {below:?}"); + r.flush_documents().unwrap(); + r.disconnected(); + drop(r); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + let mut out = r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap(); + out.extend(r.open_document(node, &app()).2); + assert!(out.iter().any(|f| matches!(f, ClientFrame::Subscribe { after: 3, .. })), "the kept rows count as had: {out:?}"); + r.on_frame(row(4, &frames[2])).unwrap(); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 4 }).unwrap(); + assert!(r.docs.get(&id).unwrap().fragments.is_empty()); + assert!(text_of(&r, id).starts_with(&"b".repeat(2 * MAX_UPDATE_BYTES)), "the change, whole"); + assert!(r.take_events().contains(&DocEvent::Remote(id, big.clone()))); + let answer = r.on_frame(ServerFrame::Compact { doc: id, upto: 4 }).unwrap(); + assert!(answer.iter().any(|f| matches!(f, ClientFrame::Snapshot { upto: 4, .. })), "{answer:?}"); + } + + /// Past [`MAX_SETS`] sets kept, a row of another set isn't taken (not counted as had): the next connection + /// brings it again, and nothing of it is lost. + #[test] + fn past_the_sets_a_document_keeps_a_row_is_not_taken() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let other = text_doc(41); + other.import(&import).unwrap(); + let row = |seq: u64, bytes: &[u8]| ServerFrame::Update { doc: id, seq, bytes: bytes.to_vec(), device: device() }; + let sets: Vec>> = (0..=MAX_SETS).map(|i| split(&insert(&other, 0, &format!("change {i} ")), 32).unwrap()).collect(); + assert!(sets.iter().all(|s| s.len() >= 2)); + for (k, set) in sets.iter().enumerate() { + r.on_frame(row(k as u64 + 1, &set[0])).unwrap(); + } + let doc = r.docs.get(&id).unwrap(); + assert_eq!(doc.fragments.len(), MAX_SETS); + assert_eq!(doc.have.upto, MAX_SETS as u64, "the row past them isn't had"); + r.disconnected(); + let mut out = r.connected(); + out.extend(r.on_frame(ServerFrame::Welcome { protocol: LIVE_FRAGMENTS_PROTOCOL, epoch: epoch(1) }).unwrap()); + assert!(out.iter().any(|f| matches!(f, ClientFrame::Subscribe { after, .. } if *after == MAX_SETS as u64)), "{out:?}"); + } + + /// A server before fragments is never sent one: an app's change that needs them is refused while it is the + /// server, and fragments made before it came wait unsent until a server that takes them. + #[test] + fn a_server_before_fragments_is_sent_none() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let app_doc = text_doc(50); + app_doc.import(&import).unwrap(); + r.disconnected(); + let big = sliced(&app_doc, &"a".repeat(2 * MAX_UPDATE_BYTES)); + assert!(r.local_update(id, &big).unwrap().is_empty(), "the link is down: kept"); + let welcome = |r: &mut Replica, protocol: u32| { + let mut out = r.connected(); + out.extend(r.on_frame(ServerFrame::Welcome { protocol, epoch: epoch(1) }).unwrap()); + out.extend(r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap()); + out + }; + let out = welcome(&mut r, LIVE_DOCUMENTS_PROTOCOL); + assert!(!out.iter().any(|f| matches!(f, ClientFrame::Update { .. })), "no fragment to the old server: {out:?}"); + let more = sliced(&app_doc, &"c".repeat(2 * MAX_UPDATE_BYTES)); + assert!(matches!(r.local_update(id, &more), Err(LocalError::Invalid(_))), "refused while it is the server"); + // A bundle in one frame too: an agent before bundles can't read it. + let bundled = sliced(&app_doc, &"d".repeat(craft_cloud_proto::fragments::SLICE_OPS as usize + 10)); + assert!(bundled.len() < MAX_UPDATE_BYTES && is_bundle(&bundled)); + assert!(matches!(r.local_update(id, &bundled), Err(LocalError::Invalid(_))), "a bundle is refused too"); + // A small change after the held one waits behind it: it builds on it. + let small = insert(&app_doc, 0, "s"); + assert!(r.local_update(id, &small).unwrap().is_empty(), "behind the held fragments"); + assert!(r.resend().iter().all(|f| !matches!(f, ClientFrame::Update { .. }))); + // Read back from its file meanwhile (as after Loro failed): still held. + r.reload(id); + r.disconnected(); + let out = welcome(&mut r, LIVE_DOCUMENTS_PROTOCOL); + assert!(!out.iter().any(|f| matches!(f, ClientFrame::Update { .. })), "a document read back holds them too: {out:?}"); + r.disconnected(); + let out = welcome(&mut r, LIVE_FRAGMENTS_PROTOCOL); + let sent = out.iter().filter(|f| matches!(f, ClientFrame::Update { .. })).count(); + assert_eq!(sent, split(&big, MAX_UPDATE_BYTES).unwrap().len() + 1, "sent once the server takes them, in order: {out:?}"); + } } diff --git a/apps/craft-cloud-agent/src/local_api.rs b/apps/craft-cloud-agent/src/local_api.rs index 3e2f8bb..25ea396 100644 --- a/apps/craft-cloud-agent/src/local_api.rs +++ b/apps/craft-cloud-agent/src/local_api.rs @@ -28,8 +28,8 @@ use craft_cloud_proto::DeviceId; use craft_cloud_proto::api::{DeviceInfo, ErrorBody, PairingStatus, codes as api_codes}; use craft_cloud_proto::join::{Code, JoinString, MAX_JOIN_BYTES, PAIRING_CHARS}; use craft_cloud_proto::local::{ - AutostartRequest, JoinRequest, LOCAL_API_FILE, LOCAL_API_MAX, LOCAL_API_SCHEMA, LocalApiFile, LoginRequest, LogoutRequest, MAX_LOCAL_JSON_BYTES, - PairingInfo, RunState, Status, codes, paths, + AutostartRequest, JoinRequest, LOCAL_API_FILE, LOCAL_API_MAX, LOCAL_API_MIN, LOCAL_API_SCHEMA, LocalApiFile, LoginRequest, LogoutRequest, + MAX_LOCAL_JSON_BYTES, PairingInfo, RunState, Status, codes, paths, }; use tokio::sync::{mpsc, oneshot}; @@ -176,7 +176,10 @@ impl Control { /// What `GET /v1/status` answers. pub fn status(&self) -> Status { self.with(|s| Status { - api: LOCAL_API_MAX, + // (The oldest it speaks first: an app from before `api_max` checks only `api`, and gets what that + // version has, its recents and blobs.) + api: LOCAL_API_MIN, + api_max: Some(LOCAL_API_MAX), signed_in: s.account.is_some(), account: s.account.as_ref().map(|a| a.username.clone()), server: s.account.as_ref().map(|a| a.server.to_string()), diff --git a/apps/craft-cloud-agent/tests/documents.rs b/apps/craft-cloud-agent/tests/documents.rs index 4ff3d0a..fe20e73 100644 --- a/apps/craft-cloud-agent/tests/documents.rs +++ b/apps/craft-cloud-agent/tests/documents.rs @@ -211,6 +211,46 @@ fn a_refused_or_unwritten_change_leaves_the_document_as_it_was() { }); } +/// A change in fragments that crosses the document's cap on the server is stored whole, never half; the next one, a +/// big paste already in the sender's copy when the compaction is asked, is refused as full and kept. The sender, the +/// only device, far past the version asked, answers once it is refused (it would never be asked again: nothing more +/// is stored), the log gets below its cap, the paste goes, and a machine joining at the end has both, whole. +#[test] +fn a_change_over_the_cap_is_stored_whole_and_the_document_gets_below_it_again() { + rt().block_on(async { + // (Fragments of 2 KiB: the change of 12 KB goes in six, as a change over 1 MiB does.) + craft_cloud_agent::live::replica::set_frame_limit_for_tests(2048); + let meta = craft_cloud_meta::Meta::open("sqlite::memory:").await.unwrap().with_live_doc_cap(8_000); + let limits = Limits { compact_updates: 3, ..Limits::default() }; + let mut w = World::with_meta(limits, meta).await; + let (a, b) = (w.machine().await, w.machine().await); + w.connect(a); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "start").await; + let app_a = w.app_of(a, doc, 501); + let big = format!("[{}]", "b".repeat(12_000)); + let paste = format!("({})", "p".repeat(70_000)); + w.edit(a, doc, &app_a.append(&big)); + w.edit(a, doc, &app_a.append(&paste)); + w.settle().await; + assert!(w.machines[a].said.iter().any(|s| s.contains("full on the server")), "refused as full: {:?}", w.machines[a].said); + for _ in 0..40 { + w.settle().await; + } + assert_eq!(w.machines[a].replica().document_status(doc).map(|s| s.1), Some(0), "nothing waits: {:?}", w.machines[a].said); + let found = w.meta.live_doc(w.space, doc).await.unwrap().unwrap(); + assert!(found.base > 0, "compacted: {found:?}"); + let rows = w.meta.live_updates(w.space, doc, found.base, 1000).await.unwrap(); + let fragments = rows.iter().filter(|r| craft_cloud_proto::fragments::is_fragment(&r.bytes)).count(); + assert!(fragments == 0 || fragments >= 35, "whole sets in the log: {fragments} fragment rows"); + w.connect(b); + w.settle().await; + joined(&mut w, b, node, "").await; + assert_eq!(w.text(b, doc), format!("start{big}{paste}"), "the other machine has both, whole"); + }); +} + /// A long log is compacted by an agent, at the server's asking; a machine opening the document later starts from /// the snapshot and the log above it, and ends on the same document. #[test] diff --git a/apps/craft-cloud-agent/tests/documents_sim.rs b/apps/craft-cloud-agent/tests/documents_sim.rs index e5f6b34..7828725 100644 --- a/apps/craft-cloud-agent/tests/documents_sim.rs +++ b/apps/craft-cloud-agent/tests/documents_sim.rs @@ -15,7 +15,12 @@ //! 1. **Convergence**: every machine holds the same document, and so does every app (its own copy, built from the //! state it opened with and the changes it was handed). //! 2. **Nothing kept is lost**: every edit an agent took from its app (on its disk before the app heard so) is in it. -//! 3. **Nothing waits**: no machine has a change the server didn't ack. +//! 3. **Nothing waits**: no machine has a change the server didn't ack, nor keeps fragments of a change it never got +//! whole. +//! +//! Some edits are big (over the simulation's 2 KiB frame limit): they go up in fragments, the server stores each +//! whole, and a machine that sleeps, drops its connection or crashes part of the way through its rows keeps what came, +//! and catches up from there or from a snapshot past it; compaction is asked meanwhile. //! //! A seed replays exactly: `SIM_SEED=17 cargo test -p craft-cloud-agent --test documents_sim`; `SIM_SEEDS=10000 //! SIM_FIRST=0` runs many (use `--release`), on `SIM_JOBS` workers. `SIM_TRACE=1` prints every step. @@ -109,9 +114,20 @@ struct Sim { last_down: Vec>, /// A backup of the document's log: its head, base and snapshot. backup: Option<(u64, Option<(u64, craft_cloud_meta::SnapshotRef)>)>, + /// Steps each machine still sleeps. + asleep: Vec, } const MACHINES: usize = 3; +/// The frame limit in the simulation: an app's change over it goes up in fragments. +const FRAME: usize = 2048; +/// Over every seed run: the big changes taken, the steps a machine kept part of one (it got some of its rows, not +/// all yet), and the catch-ups that started from the server's snapshot. All must happen for a run to say anything. +static BIG: AtomicU64 = AtomicU64::new(0); +static KEPT: AtomicU64 = AtomicU64::new(0); +static FROM_SNAPSHOT: AtomicU64 = AtomicU64::new(0); +/// Big changes stored in part first (as a server before whole sets stored them), the rest sent again later. +static PARTS: AtomicU64 = AtomicU64::new(0); impl Sim { fn say(&mut self, line: String) { @@ -193,12 +209,16 @@ impl Sim { } /// An edit by the machine's app: a token typed into the text, or a key of the map. - fn edit(&mut self, m: usize) { + async fn edit(&mut self, m: usize) { let Some(doc) = self.doc else { return }; let Some(app) = &self.apps[m].app else { return }; self.edits += 1; - let token = format!("m{m}e{};", self.edits); - let update = if self.rng.chance(70) { + // Now and then a big one, over the frame limit: it goes up in fragments, the server stores it whole, and + // every machine keeps what came of its rows until it is whole (its text is one token: never cut in two). + let big = self.rng.chance(12); + let token = + if big { format!("m{m}e{}{};", self.edits, "x".repeat(3 * FRAME + self.rng.below(4 * FRAME))) } else { format!("m{m}e{};", self.edits) }; + let update = if big || self.rng.chance(70) { // Between tokens (every token ends with `;`), so none is cut in two. let text = app.doc.get_text("text").to_string(); let mut places = vec![0]; @@ -211,8 +231,25 @@ impl Sim { match self.w.machines[m].replica().local_update(doc, &update) { Ok(frames) => { self.taken.insert(token.clone()); - self.say(format!("m{m} edits {token} ({} frames)", frames.len())); - self.w.send(m, frames); + if big { + BIG.fetch_add(1, Ordering::Relaxed); + } + self.say(format!("m{m} edits {} ({} frames)", &token[..token.len().min(12)], frames.len())); + // Now and then a big one is stored in part, as a server before whole sets stored one whose sender's + // connection ended after its first fragment: that row alone is in the log (the other machines keep + // it, maybe for long), and the sender sends the rest again once it is back. + if big && frames.len() > 1 && self.w.machines[m].conn.is_some() && self.rng.chance(60) { + if let Some(ClientFrame::Update { client_seq, bytes, .. }) = frames.first() { + let device = self.w.machines[m].auth.device; + self.w.meta.append_live_update(self.w.space, doc, device, *client_seq, bytes, 0).await.unwrap(); + self.w.state.live.publish(self.w.space, doc); + PARTS.fetch_add(1, Ordering::Relaxed); + self.say(format!("m{m}'s first fragment alone is stored, and its link ends")); + self.w.disconnect(m); + } + } else { + self.w.send(m, frames); + } } Err(e) => { // Not taken: the app's own copy has it, the agent's doesn't. It opens the document again. @@ -225,9 +262,19 @@ impl Sim { /// One random step. async fn step(&mut self) { + // (How often a machine keeps part of a change: counted every step, for every machine.) + if let Some(doc) = self.doc { + let keeping = self.w.machines.iter().filter(|x| x.replica.as_ref().is_some_and(|r| r.kept_fragments(doc) > 0)).count(); + KEPT.fetch_add(keeping as u64, Ordering::Relaxed); + } let m = self.rng.below(MACHINES); + // Asleep (a laptop's lid closed): nothing of it moves. + if self.asleep[m] > 0 { + self.asleep[m] -= 1; + return; + } match self.rng.below(100) { - 0..=24 => self.edit(m), + 0..=24 => self.edit(m).await, 25..=44 => { // One frame up (the snapshot it may name goes up before it, as the hub sends it). let api = self.w.blobs(); @@ -251,6 +298,10 @@ impl Sim { let machine = &mut self.w.machines[m]; if let (Some(frame), Some(replica)) = (machine.down.pop_front(), machine.replica.as_mut()) { let text = down_text(&frame); + if matches!(frame, ServerFrame::Snapshot { .. }) { + FROM_SNAPSHOT.fetch_add(1, Ordering::Relaxed); + } + let row = matches!(&frame, ServerFrame::Update { bytes, .. } if craft_cloud_proto::fragments::is_fragment(bytes)); self.last_down[m] = matches!(frame, ServerFrame::Update { .. } | ServerFrame::Ack { .. }).then(|| frame.clone()); match replica.on_frame(frame) { Ok(frames) => { @@ -271,6 +322,30 @@ impl Sim { } } self.say(format!("m{m} down {text}")); + // (A row of another machine's change: this one keeps part of it now. Its own acked fragments are + // kept too, but those are in its copy already.) + if let Some(doc) = self.doc + && row + && self.w.machines[m].replica.as_ref().is_some_and(|r| r.kept_fragments(doc) > 0) + { + // Part of the way through a change's rows: now and then the laptop sleeps right here, its + // connection often not outlasting it (the rest comes with a catch-up, maybe from a snapshot + // past the change), or the agent crashes (what it kept is in its file once written). + if self.rng.chance(50) { + let steps = 20 + self.rng.below(100); + self.say(format!("m{m} sleeps {steps} steps part of the way through a change")); + if self.rng.chance(80) { + self.w.disconnect(m); + } + self.asleep[m] = steps; + } else if self.rng.chance(5) { + self.say(format!("m{m} crashes part of the way through a change")); + self.close_app(m); + self.apps[m].opening = false; + self.w.crash(m); + return; + } + } self.after(m); } } @@ -316,12 +391,23 @@ impl Sim { self.apps[m].opening = false; self.w.crash(m); } - 88..=89 => { + 88 => { // Time passes: documents are written whole. if let Some(replica) = self.w.machines[m].replica.as_mut() { replica.flush_documents().unwrap(); } } + 89 => { + // The machine sleeps a while (often part of the way through a change's rows); its connection may + // not outlast it. + let steps = 20 + self.rng.below(100); + self.say(format!("m{m} sleeps {steps} steps")); + if self.rng.chance(50) && self.w.machines[m].conn.is_some() { + self.w.disconnect(m); + self.after(m); + } + self.asleep[m] = steps; + } 90..=93 => { if self.apps[m].app.is_some() && self.rng.chance(30) { self.say(format!("m{m}'s app closes it")); @@ -402,8 +488,14 @@ impl Sim { let found = meta.live_doc(space, doc).await.ok().flatten(); let mut ranges = Vec::new(); for row in &rows { - let d = loro::LoroDoc::decode_import_blob_meta(&row.bytes, true).unwrap(); - ranges.push((row.seq, format!("{:?}", d.partial_end_vv))); + let what = match craft_cloud_proto::fragments::parse(&row.bytes) { + Some(f) => format!("fragment {}/{} of {:02x}{:02x} from {}", f.index, f.count, f.set[0], f.set[1], row.device), + None => match loro::LoroDoc::decode_import_blob_meta(&row.bytes, true) { + Ok(d) => format!("{:?}", d.partial_end_vv), + Err(_) => "a bundle".into(), + }, + }; + ranges.push((row.seq, what)); } format!("{found:?}; rows {ranges:?}") } @@ -442,6 +534,10 @@ impl Sim { } async fn run(seed: u64) -> Vec { + craft_cloud_agent::live::replica::set_frame_limit_for_tests(FRAME); + // A catch-up taken in parts of a few KB, as one of megabytes is: a connection that ends inside it leaves the + // machine keeping part of a change. + craft_cloud_agent::live::replica::set_batch_bytes_for_tests(3 * FRAME); let mut rng = Rng::new(seed); let compact_updates = 2 + rng.below(30) as u64; let compact_timeout = if rng.chance(50) { Duration::ZERO } else { Duration::from_secs(3600) }; @@ -469,6 +565,7 @@ async fn run(seed: u64) -> Vec { trace, last_down: vec![None; MACHINES], backup: None, + asleep: vec![0; MACHINES], }; sim.say(format!("seed {seed}: compacting past {compact_updates} updates, asks timing out {compact_timeout:?}")); sim.open(0); @@ -514,6 +611,14 @@ async fn check(sim: &mut Sim, seed: u64) { } let (_, waiting) = sim.w.machines[m].replica().document_status(doc).unwrap(); assert_eq!(waiting, 0, "seed {seed}: m{m} has {waiting} change(s) the server never acked"); + let kept = sim.w.machines[m].replica().kept_fragments(doc); + if kept > 0 { + let server = sim.server_version(doc).await; + panic!( + "seed {seed}: m{m} still keeps {kept} fragment(s) of a change it never got whole; it said {:?}; the server: {server}", + sim.w.machines[m].said + ); + } } let view = sim.w.app_of(0, doc, 1); let text = view.text(); @@ -592,6 +697,17 @@ fn documents_converge_without_losing_an_edit() { for_each_seed(30, |seed| { block_on(run(seed)); }); + let (big, kept, snapshots) = (BIG.load(Ordering::Relaxed), KEPT.load(Ordering::Relaxed), FROM_SNAPSHOT.load(Ordering::Relaxed)); + let parts = PARTS.load(Ordering::Relaxed); + eprintln!( + "documents sim: {big} big changes ({parts} stored in part first), {kept} steps keeping part of one, {snapshots} catch-ups from a snapshot" + ); + if std::env::var("SIM_SEED").is_err() { + assert!( + big > 0 && parts > 0 && kept > 0 && snapshots > 0, + "the run tested nothing of big changes ({big}, {parts}, {kept}) or snapshots ({snapshots})" + ); + } } #[test] diff --git a/apps/craft-cloud-agent/tests/live_documents.rs b/apps/craft-cloud-agent/tests/live_documents.rs index 5a38074..9df3f96 100644 --- a/apps/craft-cloud-agent/tests/live_documents.rs +++ b/apps/craft-cloud-agent/tests/live_documents.rs @@ -22,6 +22,30 @@ fn text_snapshot(text: &str) -> Vec { craft_cloud_agent::doc::plain_text_snapshot(text).unwrap() } +/// Imports an update as an app's kit does: a bundle's slices one after the other. +fn import(loro: &LoroDoc, bytes: &[u8]) { + for part in craft_cloud_proto::fragments::parts(bytes).unwrap() { + loro.import(part).unwrap(); + } +} + +/// A big change as an app's kit sends it: `s` at the start, its slices in a bundle. +fn sliced(loro: &LoroDoc, s: &str) -> Vec { + let peer = loro.peer_id(); + let start = loro.oplog_vv().get(&peer).copied().unwrap_or(0); + loro.get_text("text").insert(0, s).unwrap(); + loro.commit(); + let end = loro.oplog_vv().get(&peer).copied().unwrap_or(0); + let mut slices = Vec::new(); + let mut at = start; + while at < end { + let to = (at + craft_cloud_proto::fragments::SLICE_OPS).min(end); + slices.push(loro.export(ExportMode::updates_in_range(vec![loro::IdSpan::new(peer, at, to)])).unwrap()); + at = to; + } + craft_cloud_proto::fragments::bundle(&slices) +} + /// An app's Loro document for an open live document. struct AppDoc { loro: LoroDoc, @@ -34,7 +58,7 @@ impl AppDoc { loro.set_peer_id(peer).unwrap(); loro.import(doc.state()).unwrap(); for u in doc.state_updates() { - loro.import(u).unwrap(); + import(&loro, u); } Self { loro, doc } } @@ -55,9 +79,7 @@ impl AppDoc { let mut others = Vec::new(); while let Some(event) = self.doc.try_recv() { match event { - Event::Update(bytes) => { - self.loro.import(&bytes).unwrap(); - } + Event::Update(bytes) => import(&self.loro, &bytes), other => others.push(other), } } @@ -156,6 +178,83 @@ async fn a_file_opened_on_one_machine_is_live_on_the_other() { run_laptop.stop().await.0.unwrap(); } +/// A change over the 1 MiB frame limit (a 3 MiB paste, as a kit's bundle of slices): the app sends it in fragments, the +/// agent takes it whole, the server stores its fragments, and the other machine's app gets it whole, never in part, +/// also when that machine's agent was stopped meanwhile and catches up from the log. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_change_over_a_frame_goes_across_whole() { + let server = Server::new().await; + let (pc, laptop) = (Agent::signed_in(&server, "pc").await, Agent::signed_in(&server, "laptop").await); + let (run_pc, run_laptop) = (pc.run(), laptop.run()); + let (on_pc, on_laptop) = (pc.client().await, laptop.client().await); + std::fs::create_dir_all(&pc.folder).unwrap(); + std::fs::write(pc.folder.join("big.txt"), "start").unwrap(); + synced(&laptop, "big.txt").await; + let c = on_pc.clone(); + let (docs_pc, app_pc) = off(move || { + let docs = c.documents().unwrap(); + let doc = open(&docs, "big.txt", "start"); + (docs, AppDoc::new(doc, 21)) + }) + .await; + let c = on_laptop.clone(); + let (docs_laptop, app_laptop) = off(move || { + let docs = c.documents().unwrap(); + let doc = open(&docs, "big.txt", "never used"); + (docs, AppDoc::new(doc, 22)) + }) + .await; + let (app_pc, app_laptop) = (Arc::new(Mutex::new(app_pc)), Arc::new(Mutex::new(app_laptop))); + let paste = "p".repeat(3 * 1024 * 1024); + let seq = { + let app = app_pc.lock().unwrap(); + let update = sliced(&app.loro, &paste); + assert!(update.len() > craft_cloud_proto::live::MAX_UPDATE_BYTES); + app.doc.send(&update).unwrap() + }; + let a = app_pc.clone(); + eventually("the pc's change is acked", move || a.lock().unwrap().pump().contains(&Event::Acked(seq)).then_some(())).await; + let want = format!("{paste}start"); + let (l, w) = (app_laptop.clone(), want.clone()); + eventually("the laptop has the paste, whole", move || { + let app = l.lock().unwrap(); + app.pump(); + let text = app.text(); + assert!(text == "start" || text == w, "the laptop held part of the change ({} bytes)", text.len()); + (text == w).then_some(()) + }) + .await; + // The laptop's agent stops; another big change is made meanwhile; the agent comes back and catches up from the log. + drop(docs_laptop); + drop(app_laptop); + run_laptop.stop().await.0.unwrap(); + let second = "q".repeat(2 * 1024 * 1024); + let seq = { + let app = app_pc.lock().unwrap(); + app.doc.send(&sliced(&app.loro, &second)).unwrap() + }; + let a = app_pc.clone(); + eventually("the second is acked", move || a.lock().unwrap().pump().contains(&Event::Acked(seq)).then_some(())).await; + let run_laptop = laptop.run(); + let c = laptop.client().await; + let want = format!("{second}{want}"); + eventually("the laptop catches up with the second, whole", move || { + let docs = c.documents().ok()?; + let doc = match docs.open("big.txt", "wordcraft").ok()? { + Opened::Live(doc) => doc, + _ => return None, + }; + let app = AppDoc::new(doc, 23); + let text = app.text(); + assert!(text.len() == want.len() || text.len() == want.len() - second.len(), "part of a change: {} bytes", text.len()); + (text == want).then_some(()) + }) + .await; + drop((docs_pc, app_pc)); + run_pc.stop().await.0.unwrap(); + run_laptop.stop().await.0.unwrap(); +} + /// The records the agent's save book holds, by content hash (hex), as the sync reads them. fn book(data: &std::path::Path) -> Vec<(String, Vec<(u64, u32)>)> { let Ok(spaces) = std::fs::read_dir(data.join("live")) else { return Vec::new() }; diff --git a/apps/craft-cloud-agent/tests/old_agent_blobs.rs b/apps/craft-cloud-agent/tests/old_agent_blobs.rs index 0aab215..0393956 100644 --- a/apps/craft-cloud-agent/tests/old_agent_blobs.rs +++ b/apps/craft-cloud-agent/tests/old_agent_blobs.rs @@ -7,12 +7,14 @@ use axum::extract::DefaultBodyLimit; use axum::http::StatusCode; use axum::response::IntoResponse; use axum::routing::get; -use craft_cloud_proto::local::{LOCAL_API_FILE, LOCAL_API_MAX, LocalApiFile, RunState, Status}; +use craft_cloud_proto::local::{LOCAL_API_FILE, LOCAL_API_MIN, LocalApiFile, RunState, Status}; #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn an_old_agent_is_too_old_for_blobs_of_any_size() { + // (An agent from before blobs speaks the first local API version.) let status = Status { - api: LOCAL_API_MAX, + api: LOCAL_API_MIN, + api_max: None, signed_in: true, account: Some("a".into()), server: Some("https://x.invalid".into()), diff --git a/apps/craft-cloud-desktop/src/model.rs b/apps/craft-cloud-desktop/src/model.rs index 83c0357..ee8d6af 100644 --- a/apps/craft-cloud-desktop/src/model.rs +++ b/apps/craft-cloud-desktop/src/model.rs @@ -855,6 +855,7 @@ pub(crate) mod tests { pub(crate) fn status(signed_in: bool) -> Status { Status { api: 1, + api_max: None, signed_in, account: signed_in.then(|| "fabio".into()), server: signed_in.then(|| "https://cloud.shadows.ch".into()), diff --git a/apps/craft-cloud-desktop/src/ui/tests.rs b/apps/craft-cloud-desktop/src/ui/tests.rs index e059f6b..d011a73 100644 --- a/apps/craft-cloud-desktop/src/ui/tests.rs +++ b/apps/craft-cloud-desktop/src/ui/tests.rs @@ -30,6 +30,7 @@ fn defaults() -> JoinDefaults { fn signed_in() -> Status { Status { api: 1, + api_max: None, signed_in: true, account: Some("fabio".into()), server: Some("https://cloud.shadows.ch".into()), diff --git a/apps/craft-cloud-server/src/cli.rs b/apps/craft-cloud-server/src/cli.rs index a9b4908..0b0fc18 100644 --- a/apps/craft-cloud-server/src/cli.rs +++ b/apps/craft-cloud-server/src/cli.rs @@ -93,6 +93,10 @@ pub enum LiveAction { /// instead of an app, that live document of a file. Its log and its snapshot go; every machine then sends its whole /// document up again, so nothing any machine holds is lost. ResetDoc { username: String, app: String }, + /// List a user's live documents whose log holds part of a change in fragments (stored before this server stored + /// a change whole): no machine compacts such a log past it, and it fills up. Each is printed with the `reset-doc` + /// that empties it (every machine then sends its whole document up again). + CheckSets { username: String }, } #[derive(Debug, Subcommand)] @@ -256,7 +260,46 @@ pub async fn run(command: Command, input: &mut impl BufRead, out: &mut impl Writ writeln!(out, "emptied the {app} recents of {}; every machine sends its own again when it next connects", user.username)?; Ok(()) } + Command::Live { config, action: LiveAction::CheckSets { username } } => { + let (meta, _) = open(&Config::load(&config_path(config)?)?).await?; + let user = user_id(&meta, &username).await?; + let mut found = 0; + for (doc, _) in meta.live_heads(user.space).await? { + let parts = partial_sets(&meta, user.space, doc).await?; + if parts > 0 { + found += 1; + writeln!( + out, + "the live document {doc} of {} holds part of {parts} change(s): `craft-cloud-server live reset-doc {} {doc}` empties it", + user.username, user.username + )?; + } + } + if found == 0 { + writeln!(out, "no live document of {} holds part of a change", user.username)?; + } + Ok(()) + } + } +} + +/// How many changes in fragments the log of `doc` holds only part of, above its snapshot (rows at or below the base +/// are in the snapshot whole). +pub async fn partial_sets(meta: &Meta, space: craft_cloud_proto::SpaceId, doc: craft_cloud_proto::DocId) -> Result { + use craft_cloud_proto::fragments::{SetId, parse}; + let Some(found) = meta.live_doc(space, doc).await? else { return Ok(0) }; + // (By set and count: two cuts of one change, at different limits, are two sets, never one that looks whole.) + let mut sets: std::collections::BTreeMap<(SetId, u16), std::collections::BTreeSet> = std::collections::BTreeMap::new(); + let mut after = found.base; + loop { + let rows = meta.live_updates(space, doc, after, craft_cloud_meta::MAX_LIVE_PAGE).await?; + let Some(last) = rows.last() else { break }; + after = last.seq; + for f in rows.iter().filter_map(|r| parse(&r.bytes)) { + sets.entry((f.set, f.count)).or_default().insert(f.index); + } } + Ok(sets.iter().filter(|((_, count), got)| got.len() < usize::from(*count)).count()) } /// `user invite`: makes the invite and prints how to use it. The code is printed only here, never logged. diff --git a/apps/craft-cloud-server/src/live.rs b/apps/craft-cloud-server/src/live.rs index 859d5c8..cdabdd0 100644 --- a/apps/craft-cloud-server/src/live.rs +++ b/apps/craft-cloud-server/src/live.rs @@ -56,9 +56,10 @@ use axum::extract::{Extension, State}; use axum::response::Response; use craft_cloud_meta::DeviceAuth; use craft_cloud_proto::api::codes; +use craft_cloud_proto::fragments::{MAX_SETS, MAX_WHOLE_BYTES, SetId, is_bundle, is_fragment, join, parse}; use craft_cloud_proto::live::{ - AppName, ClientFrame, DOCUMENT_KIND_PREFIX, DocKind, LIVE_DOCUMENTS_PROTOCOL, MAX_FRAME_BYTES, MAX_SUBSCRIPTIONS, MAX_UPDATE_BYTES, - PROTOCOL_MISMATCH, ServerFrame, node_doc, shared_version, update_hash, + AppName, ClientFrame, DOCUMENT_KIND_PREFIX, DocKind, LIVE_DOCUMENTS_PROTOCOL, LIVE_FRAGMENTS_PROTOCOL, MAX_FRAME_BYTES, MAX_SUBSCRIPTIONS, + MAX_UPDATE_BYTES, PROTOCOL_MISMATCH, ServerFrame, node_doc, shared_version, update_hash, }; use craft_cloud_proto::{DeviceId, DocId, Epoch, ManifestId, NodeId, SpaceId}; use tokio::sync::broadcast; @@ -69,6 +70,8 @@ use crate::state::{AppState, Limits}; /// Live connections one device may hold at once (a reconnect may overlap the old connection for a moment). pub const MAX_CONNECTIONS_PER_DEVICE: usize = 4; +/// The most bytes of fragments a connection stages while their sets come (two changes of the largest size). +pub const MAX_STAGED_BYTES: usize = 2 * MAX_WHOLE_BYTES + 2 * MAX_SETS * craft_cloud_proto::fragments::FRAGMENT_OVERHEAD; /// Updates (and creates) a connection may send at once before it is refilled… pub const UPDATE_BURST: u32 = 400; /// …at this many per second. @@ -350,6 +353,25 @@ pub struct Session { budget: Budget, /// The `upto` this session's agent was asked to compact each document at: only that snapshot is taken from it. asked: BTreeMap, + /// The fragments of the agent's changes still coming (protocol [`LIVE_FRAGMENTS_PROTOCOL`]), by document and set, + /// with their `client_seq`s: a set goes into the log whole once its last fragment is here, or not at all. A + /// connection that ends drops them (none was acked: the agent sends them all again). + staged: BTreeMap<(DocId, SetId), Staged>, + staged_bytes: usize, +} + +/// A set's fragments staged so far, with their `client_seq`s. +type Staged = Vec<(u64, Vec)>; + +/// An update refused: why, and the `client_seq`s refused (a set's every fragment). +type Refusal = Box<(ApiError, Vec)>; + +/// What `update` did with an update: stored (its number, the `compact` to ask for), or staged as a fragment of a set +/// still coming (no answer yet), or a set stored whole (each fragment's `client_seq` and number). +enum Stored { + One(u64, Option), + Staged, + Set(Vec<(u64, u64)>, Option), } impl Drop for Session { @@ -382,6 +404,8 @@ impl Session { dirty: BTreeSet::new(), budget, asked: BTreeMap::new(), + staged: BTreeMap::new(), + staged_bytes: 0, } } @@ -440,10 +464,25 @@ impl Session { Reply::default() } ClientFrame::Update { doc, client_seq, bytes } => match self.update(doc, client_seq, bytes).await { - Ok((seq, compact)) => { + Ok(Stored::One(seq, compact)) => { Reply { frames: std::iter::once(ServerFrame::Ack { doc, client_seq, seq }).chain(compact).collect(), close: false } } - Err(e) => one(refused(e, Some(doc), Some(client_seq))), + Ok(Stored::Staged) => Reply::default(), + Ok(Stored::Set(acks, compact)) => Reply { + frames: acks.into_iter().map(|(client_seq, seq)| ServerFrame::Ack { doc, client_seq, seq }).chain(compact).collect(), + close: false, + }, + // (A set refused: every one of its fragments is, so the agent hears it whichever it waits on.) + Err(refusal) => { + let (e, seqs) = *refusal; + // A full document is compacted only so: with nothing more stored, no append asks any device again, + // so this refusal does (once per head, and not while another device's ask is out). + let ask = if e.code == codes::QUOTA_EXCEEDED { self.ask_when_full(doc).await } else { None }; + Reply { + frames: seqs.into_iter().map(|s| ServerFrame::error(e.code, e.message.clone(), Some(doc), Some(s))).chain(ask).collect(), + close: false, + } + } }, // The frames of documents, snapshots and presence are for a connection that speaks them. ClientFrame::CreateDocument { doc, .. } | ClientFrame::SubscribeLog { doc, .. } if !self.documents() => { @@ -530,6 +569,13 @@ impl Session { .then(|| ServerFrame::Presence { doc: note.doc, bytes: note.bytes.as_ref().clone() }) } + /// A compaction ask of this device for a document refused as full, at its head ([`Session::maybe_compact`]). + async fn ask_when_full(&mut self, doc: DocId) -> Option { + let found = self.state.meta.live_doc(self.space, doc).await.ok().flatten()?; + let above = found.bytes.saturating_sub(found.compacted_bytes); + self.maybe_compact(doc, found.head, found.base, above) + } + /// The `compact` to ask this session's agent for, if the log above the snapshot is long (`upto - base` updates, /// `above` bytes) and no other device is asked already. fn maybe_compact(&mut self, doc: DocId, upto: u64, base: u64, above: u64) -> Option { @@ -635,16 +681,119 @@ impl Session { Ok(()) } - /// Appends an update; the `compact` to ask for after its ack, if its log is long now. - async fn update(&mut self, doc: DocId, client_seq: u64, bytes: Vec) -> Result<(u64, Option), ApiError> { + /// Appends an update (or stages a fragment, and appends its set once whole); the `compact` to ask for after the + /// acks, if the log is long now. A refusal names the `client_seq`s it refuses. + async fn update(&mut self, doc: DocId, client_seq: u64, bytes: Vec) -> Result { + let one = |e: ApiError| Box::new((e, vec![client_seq])); if !self.subscriptions.contains_key(&doc) { - return Err(ApiError::not_found("subscribed document")); + return Err(one(ApiError::not_found("subscribed document"))); } if bytes.is_empty() || bytes.len() > MAX_UPDATE_BYTES { - return Err(ApiError::too_large(format!("an update is 1 to {MAX_UPDATE_BYTES} bytes"))); + return Err(one(ApiError::too_large(format!("an update is 1 to {MAX_UPDATE_BYTES} bytes")))); } if !self.budget.spend() { - return Err(ApiError::rate_limited("too many updates; send again in a moment", None)); + return Err(one(ApiError::rate_limited("too many updates; send again in a moment", None))); + } + // A fragment: staged until its set is whole, then the set goes into the log in one go (and counts against the + // document's cap only then), or is refused whole: no device ever gets part of a change from the log. + // A connection before protocol 3 (an agent from before whole sets) has its fragments refused: stored one by one, + // a set could be stored in part, which the newer agents would keep for good. + if self.protocol < LIVE_FRAGMENTS_PROTOCOL && is_fragment(&bytes) { + return Err(one(ApiError::invalid("a change in fragments needs a newer Craft Cloud on this computer: update Craft Cloud"))); + } + if is_fragment(&bytes) { + let Some((set_id, count, index)) = parse(&bytes).map(|f| (f.set, f.count, f.index)) else { + return Err(one(ApiError::invalid("a fragment that doesn't read"))); + }; + let key = (doc, set_id); + let new_set = !self.staged.contains_key(&key); + if self.staged_bytes.saturating_add(bytes.len()) > MAX_STAGED_BYTES || (new_set && self.staged.len() >= MAX_SETS) { + return Err(one(ApiError::invalid("too many fragments wait on this connection"))); + } + // Fragments of the set this device has in the log already (a set stored in part before whole sets, or one + // whose acks were lost: the agent sends only the rest) count as here: the device numbered the set's + // fragments one after the other, so each is found by its number. + let mut found = Vec::new(); + if new_set { + let mut below_base = false; + // (The set's numbers, from this fragment's: one query for all of them, whatever the count says.) + let first = client_seq.saturating_sub(u64::from(index)).max(1); + let last = client_seq.saturating_add(u64::from(count).saturating_sub(u64::from(index) + 1)); + let stored = self.state.meta.live_updates_sent(self.space, doc, self.device, first, last).await.map_err(|e| one(e.into()))?; + let mut base = None; + for (c, row, b) in stored { + let Some(f) = parse(&b) else { continue }; + let at = (client_seq + u64::from(f.index)).checked_sub(u64::from(index)); + if f.set != set_id || f.count != count || f.index == index || at != Some(c) || found.iter().any(|(s, _)| *s == c) { + continue; + } + if base.is_none() { + base = Some(self.state.meta.live_doc(self.space, doc).await.map_err(|e| one(e.into()))?.map_or(0, |d| d.base)); + } + below_base |= base.is_some_and(|base| row <= base); + found.push((c, b)); + } + // Part of the set is at or below the snapshot's base (a device from before whole sets compacted past + // it): stored above it, the rest would be a set no machine ever gets whole. The agent starts this + // document over instead: it drops the set and sends its operations up as a push. + if below_base { + return Err(one(ApiError::new( + axum::http::StatusCode::GONE, + codes::RESYNC_REQUIRED, + "part of this change is in the document's snapshot, the rest not: start the document over", + ))); + } + } + // (What is found counts against the connection's staging too.) + let found_bytes: usize = found.iter().map(|(_, b): &(u64, Vec)| b.len()).sum(); + if self.staged_bytes.saturating_add(bytes.len()).saturating_add(found_bytes) > MAX_STAGED_BYTES { + return Err(one(ApiError::invalid("too many fragments wait on this connection"))); + } + let set = self.staged.entry(key).or_default(); + for (seq, b) in found { + self.staged_bytes += b.len(); + set.push((seq, b)); + } + if !set.iter().any(|(s, _)| *s == client_seq) { + self.staged_bytes += bytes.len(); + set.push((client_seq, bytes)); + } + let mut indexes: Vec = set.iter().filter_map(|(_, b)| parse(b).map(|g| g.index)).collect(); + indexes.sort_unstable(); + indexes.dedup(); + if indexes.len() < usize::from(count) { + return Ok(Stored::Staged); + } + let mut set = self.staged.remove(&key).unwrap_or_default(); + self.staged_bytes = self.staged_bytes.saturating_sub(set.iter().map(|(_, b)| b.len()).sum()); + set.sort_by_key(|(_, b)| parse(b).map(|g| g.index)); + set.dedup_by_key(|(_, b)| parse(b).map(|g| g.index)); + let seqs: Vec = set.iter().map(|(s, _)| *s).collect(); + match join(set.iter().map(|(_, b)| b.as_slice())) { + None => return Err(Box::new((ApiError::invalid("fragments that don't make the change they name"), seqs))), + Some(whole) if whole.len() > MAX_WHOLE_BYTES => { + return Err(Box::new((ApiError::too_large(format!("a change is at most {MAX_WHOLE_BYTES} bytes")), seqs))); + } + Some(_) => {} + } + let (state, space, device) = (self.state.clone(), self.space, self.device); + let appended = tokio::spawn(async move { + let rows: Vec<(u64, &[u8])> = set.iter().map(|(s, b)| (*s, b.as_slice())).collect(); + let appended = state.meta.append_live_updates_counted(space, doc, device, &rows, state.now_ms()).await?; + state.live.publish(space, doc); + Ok::<_, ApiError>(appended) + }) + .await + .map_err(|e| { + tracing::error!(error = %e, "live append task failed"); + Box::new((ApiError::internal(), seqs.clone())) + })? + .map_err(|e| Box::new((e, seqs.clone())))?; + let compact = match appended.last() { + Some(last) if !last.repeat => self.maybe_compact(doc, last.seq, last.base, last.above_bytes), + _ => None, + }; + return Ok(Stored::Set(seqs.into_iter().zip(appended.into_iter().map(|a| a.seq)).collect(), compact)); } // Appended and published in a task of its own: a connection that drops meanwhile must not skip the publish of // an update that was stored. @@ -657,12 +806,13 @@ impl Session { .await .map_err(|e| { tracing::error!(error = %e, "live append task failed"); - ApiError::internal() - })??; + one(ApiError::internal()) + })? + .map_err(one)?; // The device that sent the update that made the log long is asked first: it holds every update up to it once // those below have reached it. let compact = if appended.repeat { None } else { self.maybe_compact(doc, appended.seq, appended.base, appended.above_bytes) }; - Ok((appended.seq, compact)) + Ok(Stored::One(appended.seq, compact)) } /// `doc` got an update: if it is subscribed, the next [`Session::pump`]s send what the agent lacks. @@ -730,6 +880,20 @@ impl Session { let (page, full) = self.state.meta.live_page(self.space, doc, sub.sent, PUMP_BYTES, PUMP_PAGE).await?; let mut frames = Vec::new(); for update in page { + // An agent on an older protocol can't read a change in fragments or a bundle: it would skip it, and every + // later change that builds on it would wait there for good. It is refused the document instead, and says so. + if self.protocol < LIVE_FRAGMENTS_PROTOCOL && (is_fragment(&update.bytes) || is_bundle(&update.bytes)) { + self.subscriptions.remove(&doc); + release(&self.state.live.subscriptions, self.device, 1); + self.dirty.remove(&doc); + frames.push(ServerFrame::error( + codes::INVALID, + "this live document holds changes this Craft Cloud agent can't read: update Craft Cloud on this computer", + Some(doc), + None, + )); + return Ok(frames); + } sub.sent = sub.sent.max(update.seq); if sub.catching_up || update.device != self.device { frames.push(ServerFrame::Update { doc, seq: update.seq, bytes: update.bytes, device: update.device }); @@ -1250,6 +1414,188 @@ mod tests { assert_eq!(code(&bo.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await), Some("not_found"), "nor its document"); } + /// On protocol 3 a change's fragments are staged and stored whole: nothing is answered (or stored, or sent to the + /// other device) until the last one is here, then each is acked with consecutive numbers and the other device gets + /// them all. A set that would start in a full document is refused whole, every fragment named, nothing stored. A + /// device on protocol 2 is refused the document once it holds fragments (it can't read them). + #[tokio::test] + async fn a_change_in_fragments_is_stored_whole_or_refused_whole() { + let meta = Meta::open("sqlite::memory:").await.unwrap().with_live_doc_cap(5000); + let mut a = Account::with_meta(Default::default(), meta, "ana").await; + let (mut pc, pc_auth) = a.session(3).await; + let (mut laptop, _) = a.session(3).await; + let (mut old, _) = a.session(2).await; + let node = a.file(&pc_auth, "big.txt").await; + let doc = node_doc(a.user.space, node); + let import = a.upload(&[1; 100]).await; + pc.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: import }).await; + for s in [&mut pc, &mut laptop, &mut old] { + s.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + drain(s).await; + } + let change: Vec = (0..3000u32).map(|i| (i % 251) as u8).collect(); + let frames = craft_cloud_proto::fragments::split(&change, 1100).unwrap(); + assert_eq!(frames.len(), 3); + for (i, f) in frames.iter().enumerate().take(2) { + let reply = pc.handle(ClientFrame::Update { doc, client_seq: i as u64 + 1, bytes: f.clone() }).await; + assert!(reply.frames.is_empty(), "staged: {:?}", reply.frames); + } + laptop.note(doc); + assert!(updates(&drain(&mut laptop).await).is_empty(), "nothing of it in the log yet"); + let last = pc.handle(ClientFrame::Update { doc, client_seq: 3, bytes: frames[2].clone() }).await; + let acks: Vec<(u64, u64)> = last + .frames + .iter() + .filter_map(|f| if let ServerFrame::Ack { client_seq, seq, .. } = f { Some((*client_seq, *seq)) } else { None }) + .collect(); + assert_eq!(acks, vec![(1, 1), (2, 2), (3, 3)], "{:?}", last.frames); + laptop.note(doc); + assert_eq!(updates(&drain(&mut laptop).await), vec![1, 2, 3], "the whole set"); + old.note(doc); + let refused = drain(&mut old).await; + assert!( + matches!(refused.as_slice(), [ServerFrame::Error { code, message, .. }] if code == "invalid" && message.contains("update Craft Cloud")), + "{refused:?}" + ); + // 3000 of the cap's 5000 used: one more set of 3000 crosses it and is stored whole; the next is refused whole. + let more = craft_cloud_proto::fragments::split(&[change.clone(), vec![7]].concat(), 1100).unwrap(); + for (i, f) in more.iter().enumerate() { + pc.handle(ClientFrame::Update { doc, client_seq: 10 + i as u64, bytes: f.clone() }).await; + } + let third = craft_cloud_proto::fragments::split(&[change.clone(), vec![8]].concat(), 1100).unwrap(); + let mut answers = Vec::new(); + for (i, f) in third.iter().enumerate() { + answers.extend(pc.handle(ClientFrame::Update { doc, client_seq: 20 + i as u64, bytes: f.clone() }).await.frames); + } + let refused: Vec = answers + .iter() + .filter_map( + |f| if let ServerFrame::Error { client_seq: Some(s), code, .. } = f { (code == "quota_exceeded").then_some(*s) } else { None }, + ) + .collect(); + assert_eq!(refused, vec![20, 21, 22], "{answers:?}"); + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().head, 6, "nothing of the refused set stored"); + } + + /// A refusal as full asks the refused device to compact (the document gets below its cap only so, and no append + /// asks anyone once nothing more is stored), unless another device's ask is out. + #[tokio::test] + async fn a_device_refused_as_full_is_asked_to_compact() { + let meta = Meta::open("sqlite::memory:").await.unwrap().with_live_doc_cap(5000); + let limits = crate::state::Limits { compact_updates: 1, compact_timeout: std::time::Duration::ZERO, ..Default::default() }; + let mut a = Account::with_meta(limits, meta, "ana").await; + let (mut pc, pc_auth) = a.session(3).await; + let (mut laptop, _) = a.session(3).await; + let node = a.file(&pc_auth, "full.txt").await; + let doc = node_doc(a.user.space, node); + let import = a.upload(&[1; 100]).await; + pc.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: import }).await; + for s in [&mut pc, &mut laptop] { + s.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + drain(s).await; + } + pc.handle(ClientFrame::Update { doc, client_seq: 1, bytes: vec![1; 4000] }).await; + let crossed = pc.handle(ClientFrame::Update { doc, client_seq: 2, bytes: vec![2; 2000] }).await; + assert!(crossed.frames.iter().any(|f| matches!(f, ServerFrame::Compact { upto: 2, .. })), "{:?}", crossed.frames); + drain(&mut laptop).await; + let refused = laptop.handle(ClientFrame::Update { doc, client_seq: 1, bytes: vec![3; 10] }).await; + assert!(refused.frames.iter().any(|f| matches!(f, ServerFrame::Error { code, .. } if code == "quota_exceeded")), "{:?}", refused.frames); + assert!(refused.frames.iter().any(|f| matches!(f, ServerFrame::Compact { upto: 2, .. })), "asked: {:?}", refused.frames); + } + + /// The first part of a set stored alone, then compacted past (a device from before whole sets did): the rest, + /// sent now, is refused with `resync_required` (the agent starts the document over and sends the change as a + /// push), never stored above the base as a set no machine would get whole. + #[tokio::test] + async fn the_rest_of_a_set_whose_part_is_below_the_base_is_refused_for_a_start_over() { + let mut a = Account::new(Default::default()).await; + let (mut pc, pc_auth) = a.session(3).await; + let (_laptop, laptop_auth) = a.session(3).await; + let node = a.file(&pc_auth, "part.txt").await; + let doc = node_doc(a.user.space, node); + let import = a.upload(&[1; 100]).await; + pc.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: import }).await; + let change: Vec = (0..3000u32).map(|i| (i % 249) as u8).collect(); + let frames = craft_cloud_proto::fragments::split(&change, 1100).unwrap(); + assert_eq!(a.meta.append_live_update(a.user.space, doc, pc_auth.device, 1, &frames[0], 0).await.unwrap(), 1); + assert_eq!(a.meta.append_live_update(a.user.space, doc, laptop_auth.device, 1, &[7; 10], 0).await.unwrap(), 2); + let snap = a.upload(&[9; 20]).await; + assert!(a.meta.compact_live_doc(a.user.space, doc, 2, &snap).await.unwrap()); + pc.handle(ClientFrame::Subscribe { doc, after: 2, at: None }).await; + drain(&mut pc).await; + let mut answers = pc.handle(ClientFrame::Update { doc, client_seq: 2, bytes: frames[1].clone() }).await.frames; + answers.extend(pc.handle(ClientFrame::Update { doc, client_seq: 3, bytes: frames[2].clone() }).await.frames); + let resync: Vec> = answers + .iter() + .filter_map( + |f| if let ServerFrame::Error { code, client_seq, .. } = f { (code == "resync_required").then_some(*client_seq) } else { None }, + ) + .collect(); + assert_eq!(resync, vec![Some(2), Some(3)], "{answers:?}"); + assert!(!answers.iter().any(|f| matches!(f, ServerFrame::Ack { .. })), "{answers:?}"); + let found = a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap(); + assert_eq!((found.base, found.head), (2, 2), "nothing stored above the base"); + } + + /// A set stored in part before whole sets (an older server stored fragments one by one) is made whole when its + /// device sends the rest: the part in the log counts as here, and every fragment is acked with its number. A + /// device on protocol 2 has its fragments refused (stored one by one, a set could be stored in part). A set that + /// joins to more than a change's limit is refused whole. + #[tokio::test] + async fn the_rest_of_a_set_stored_in_part_completes_it_and_older_agents_send_none() { + let mut a = Account::new(Default::default()).await; + let (mut pc, pc_auth) = a.session(3).await; + let (mut old, _) = a.session(2).await; + let node = a.file(&pc_auth, "part.txt").await; + let doc = node_doc(a.user.space, node); + let import = a.upload(&[1; 100]).await; + pc.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: import }).await; + for s in [&mut pc, &mut old] { + s.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + drain(s).await; + } + let change: Vec = (0..3000u32).map(|i| (i % 249) as u8).collect(); + let frames = craft_cloud_proto::fragments::split(&change, 1100).unwrap(); + // Its first fragment alone in the log, as an older server stored it. + assert_eq!(a.meta.append_live_update(a.user.space, doc, pc_auth.device, 1, &frames[0], 0).await.unwrap(), 1); + pc.handle(ClientFrame::Update { doc, client_seq: 2, bytes: frames[1].clone() }).await; + let last = pc.handle(ClientFrame::Update { doc, client_seq: 3, bytes: frames[2].clone() }).await; + let acks: Vec<(u64, u64)> = last + .frames + .iter() + .filter_map(|f| if let ServerFrame::Ack { client_seq, seq, .. } = f { Some((*client_seq, *seq)) } else { None }) + .collect(); + assert_eq!(acks, vec![(1, 1), (2, 2), (3, 3)], "{:?}", last.frames); + let rows = a.meta.live_updates(a.user.space, doc, 0, 10).await.unwrap(); + assert_eq!(craft_cloud_proto::fragments::join(rows.iter().map(|r| r.bytes.as_slice())), Some(change.clone()), "whole in the log"); + // Protocol 2: refused, nothing stored. + let other = craft_cloud_proto::fragments::split(&[change.clone(), vec![5]].concat(), 1100).unwrap(); + let reply = old.handle(ClientFrame::Update { doc, client_seq: 1, bytes: other[0].clone() }).await; + assert!( + matches!(reply.frames.as_slice(), [ServerFrame::Error { code, message, client_seq: Some(1), .. }] if code == "invalid" && message.contains("update Craft Cloud")), + "{:?}", + reply.frames + ); + // Over a change's limit, once joined: refused whole. + let whole = vec![3u8; MAX_WHOLE_BYTES + 10]; + let set = blake3::hash(&whole); + let body = MAX_UPDATE_BYTES - craft_cloud_proto::fragments::HEADER_BYTES; + let chunks: Vec<&[u8]> = whole.chunks(body).collect(); + let count = u16::try_from(chunks.len()).unwrap(); + let mut answers = Vec::new(); + for (i, chunk) in (0u16..).zip(&chunks) { + let mut f = b"ccf1".to_vec(); + f.extend_from_slice(&set.as_bytes()[..16]); + f.extend_from_slice(&i.to_be_bytes()); + f.extend_from_slice(&count.to_be_bytes()); + f.extend_from_slice(chunk); + answers.extend(pc.handle(ClientFrame::Update { doc, client_seq: 10 + u64::from(i), bytes: f }).await.frames); + } + let refused = answers.iter().filter(|f| matches!(f, ServerFrame::Error { code, .. } if code == "too_large")).count(); + assert_eq!(refused, chunks.len(), "{answers:?}"); + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().head, 3, "nothing of it stored"); + } + /// Snapshots never travel in frames: a `create` or `snapshot` names a manifest the account uploaded (through the /// chunk API, charged there), and one naming a manifest it doesn't have (never uploaded, or another account's) /// records nothing. A refused `create` stores nothing either: the server writes no blob for a frame. diff --git a/apps/craft-cloud-server/tests/cli.rs b/apps/craft-cloud-server/tests/cli.rs index 4a5332b..d150b7f 100644 --- a/apps/craft-cloud-server/tests/cli.rs +++ b/apps/craft-cloud-server/tests/cli.rs @@ -93,6 +93,47 @@ async fn live_reset_doc_empties_a_full_document() { assert!(matches!(cli(&["live", "--config", c, "reset-doc", "nobody", "wordcraft"], "").await, Err(CliError::Usage(_)))); } +/// `live check-sets` lists a document whose log holds part of a change in fragments (stored before whole sets), +/// with the `reset-doc` that empties it; a whole set isn't listed. +#[tokio::test] +async fn live_check_sets_finds_part_of_a_change() { + let parent = tempfile::tempdir().unwrap(); + let dir = parent.path().join("data"); + cli(&["init", "--dir", dir.to_str().unwrap(), "--admin", "ana"], "admin password 1\n").await.unwrap(); + let config_path = dir.join("config.toml"); + let c = config_path.to_str().unwrap(); + let (meta, _) = craft_cloud_server::cli::open(&Config::load(&config_path).unwrap()).await.unwrap(); + let ana = meta.user_by_name("ana").await.unwrap().unwrap(); + let device = meta.create_device(ana.id, "pc", &[3; 32], 0).await.unwrap(); + let (whole, part) = (craft_cloud_proto::DocId::generate(), craft_cloud_proto::DocId::generate()); + let change: Vec = (0..3000u32).map(|i| (i % 247) as u8).collect(); + let frames = craft_cloud_proto::fragments::split(&change, 1100).unwrap(); + for doc in [whole, part] { + meta.create_live_doc(ana.space, doc, "recents:wordcraft", None, 0).await.unwrap(); + } + for (i, f) in frames.iter().enumerate() { + meta.append_live_update(ana.space, whole, device, i as u64 + 1, f, 0).await.unwrap(); + } + meta.append_live_update(ana.space, part, device, 1, &frames[0], 0).await.unwrap(); + // Two cuts of one change (at different limits) never make one that looks whole: by set and count, two parts. + let cuts = craft_cloud_proto::DocId::generate(); + meta.create_live_doc(ana.space, cuts, "recents:wordcraft", None, 0).await.unwrap(); + let two = craft_cloud_proto::fragments::split(&change, 1600).unwrap(); + assert_eq!((two.len(), frames.len()), (2, 3)); + meta.append_live_update(ana.space, cuts, device, 1, &two[0], 0).await.unwrap(); + meta.append_live_update(ana.space, cuts, device, 2, &frames[1], 0).await.unwrap(); + meta.append_live_update(ana.space, cuts, device, 3, &frames[2], 0).await.unwrap(); + let out = cli(&["live", "--config", c, "check-sets", "ana"], "").await.unwrap(); + assert!(out.contains(&format!("live document {cuts} of ana holds part of 2 change(s)")), "{out}"); + cli(&["live", "--config", c, "reset-doc", "ana", &cuts.to_string()], "").await.unwrap(); + assert!(out.contains(&format!("live document {part} of ana holds part of 1 change(s)")), "{out}"); + assert!(out.contains(&format!("live reset-doc ana {part}")), "{out}"); + assert!(!out.contains(&whole.to_string()), "{out}"); + cli(&["live", "--config", c, "reset-doc", "ana", &part.to_string()], "").await.unwrap(); + let out = cli(&["live", "--config", c, "check-sets", "ana"], "").await.unwrap(); + assert!(out.contains("no live document of ana holds part of a change"), "{out}"); +} + /// `live reset-doc` takes a document's ID too (a document of a file has no app name to find it by), and drops its /// snapshot with its log. #[tokio::test] diff --git a/crates/api/Cargo.toml b/crates/api/Cargo.toml index e6be988..ac17d52 100644 --- a/crates/api/Cargo.toml +++ b/crates/api/Cargo.toml @@ -8,6 +8,7 @@ rust-version.workspace = true publish = false [dependencies] +blake3.workspace = true caseless.workspace = true serde.workspace = true thiserror.workspace = true diff --git a/crates/api/src/fragments.rs b/crates/api/src/fragments.rs new file mode 100644 index 0000000..e1ad4ef --- /dev/null +++ b/crates/api/src/fragments.rs @@ -0,0 +1,395 @@ +//! Live updates over the 1 MiB frame limit, sent in fragments: one update (a Loro update of any size up to +//! [`MAX_WHOLE_BYTES`]) cut into frames of at most the limit, each a fragment naming its set (the first 16 bytes of the +//! update's BLAKE3), its index and the set's count. The agent's local API joins an app's fragments in a +//! [`Reassembler`]; its replica keeps the log's fragment rows until they are whole; either imports the update only once +//! it holds every fragment of it: no document ever holds part of a change. The +//! server stages an agent's fragments and stores a set in one go, as updates of their own, or refuses it whole; an +//! agent's log of rows keeps them as they are. Agents hand apps whole changes only. +//! +//! A fragment starts with [`MAGIC`], which no Loro update starts with (Loro's begin `loro`), so a program that doesn't +//! know fragments refuses one as an update that doesn't decode rather than importing part of a change. The cut is +//! deterministic: the same update always gives the same fragments, so a set sent again (after a reconnect) is the same +//! set, and the server keeps a fragment it already has once. +//! +//! What goes in fragments (or in one frame) may be a [`bundle`]: a big change as slices of it, imported together. + +use std::collections::{BTreeMap, HashMap}; +use std::time::{Duration, Instant}; + +use crate::limits::MAX_SNAPSHOT_BYTES; + +/// What every fragment starts with. +pub const MAGIC: [u8; 4] = *b"ccf1"; +/// What a bundle starts with: one change as several Loro updates, imported one after the other and together +/// ([`bundle`], [`parts`]). Loro's import of one update takes time growing with the square of the text in it (a 1 MiB +/// paste from another peer: seconds), so a big change goes as slices of it, which import in milliseconds, in one +/// bundle, so no receiver ever holds some of the slices and not the others. +pub const BUNDLE_MAGIC: [u8; 4] = *b"ccb1"; +/// The most operations a change's slice holds ([`bundle`]'s parts, as the kit and the agent cut them). +pub const SLICE_OPS: i32 = 16 * 1024; +/// The bytes before a fragment's part of the update: [`MAGIC`], the set (16), the index and the count (2 each, big +/// endian). +pub const HEADER_BYTES: usize = 24; +/// The largest change (one update) that may be sent in fragments: 8 MiB, well under the server's 16 MiB cap on a +/// document's log above its snapshot, so a change always fits a document that has room for updates at all. +pub const MAX_WHOLE_BYTES: usize = 8 * 1024 * 1024; +const _: () = assert!(MAX_WHOLE_BYTES <= MAX_SNAPSHOT_BYTES); +/// The most fragments one update is cut into. +pub const MAX_FRAGMENTS: usize = 4096; +/// The most sets a [`Reassembler`] (or anything that keeps fragments) holds at once: each costs bookkeeping besides its +/// bytes. +pub const MAX_SETS: usize = 16; +/// How long the agent's local API keeps the fragments of an app's change that stopped coming (the app went part of +/// the way) before it drops them, never taken. +pub const PATIENCE: Duration = Duration::from_secs(60); + +/// What a kept fragment costs beside its part of the update (its index, its map entry). +pub const FRAGMENT_OVERHEAD: usize = 64; + +/// The name of a set: the first 16 bytes of its update's BLAKE3. +pub type SetId = [u8; 16]; + +/// One fragment, as [`parse`] reads it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Fragment<'a> { + pub set: SetId, + pub index: u16, + pub count: u16, + /// Its part of the update. + pub body: &'a [u8], +} + +/// Why an update can't be sent in fragments. +#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] +pub enum SplitError { + #[error("an update is at most {MAX_WHOLE_BYTES} bytes ({0} here)")] + TooLarge(usize), + #[error("a frame of {0} bytes can't hold a fragment")] + FrameTooSmall(usize), +} + +/// The set an update's fragments name. +pub fn set_of(update: &[u8]) -> SetId { + let hash = blake3::hash(update); + let mut set = [0u8; 16]; + set.copy_from_slice(&hash.as_bytes()[..16]); + set +} + +/// `update` as frames of at most `max_frame` bytes: itself when it fits (not a fragment), else its fragments, in +/// order. Deterministic. +pub fn split(update: &[u8], max_frame: usize) -> Result>, SplitError> { + if update.len() <= max_frame { + return Ok(vec![update.to_vec()]); + } + if update.len() > MAX_WHOLE_BYTES { + return Err(SplitError::TooLarge(update.len())); + } + let room = max_frame.saturating_sub(HEADER_BYTES); + if room == 0 { + return Err(SplitError::FrameTooSmall(max_frame)); + } + let count = update.len().div_ceil(room); + let Ok(count16) = u16::try_from(count) else { return Err(SplitError::FrameTooSmall(max_frame)) }; + if count > MAX_FRAGMENTS { + return Err(SplitError::FrameTooSmall(max_frame)); + } + let set = set_of(update); + let mut out = Vec::with_capacity(count); + for (index, body) in (0u16..).zip(update.chunks(room)) { + let mut frame = Vec::with_capacity(HEADER_BYTES + body.len()); + frame.extend_from_slice(&MAGIC); + frame.extend_from_slice(&set); + frame.extend_from_slice(&index.to_be_bytes()); + frame.extend_from_slice(&count16.to_be_bytes()); + frame.extend_from_slice(body); + out.push(frame); + } + Ok(out) +} + +/// Whether `bytes` are a fragment (they start with [`MAGIC`]; [`parse`] says whether one reads). +pub fn is_fragment(bytes: &[u8]) -> bool { + bytes.starts_with(&MAGIC) +} + +/// A fragment's header and part, if `bytes` are a well-formed fragment. +pub fn parse(bytes: &[u8]) -> Option> { + if !is_fragment(bytes) || bytes.len() <= HEADER_BYTES { + return None; + } + let set: SetId = bytes.get(4..20)?.try_into().ok()?; + let index = u16::from_be_bytes(bytes.get(20..22)?.try_into().ok()?); + let count = u16::from_be_bytes(bytes.get(22..24)?.try_into().ok()?); + if count < 2 || usize::from(count) > MAX_FRAGMENTS || index >= count { + return None; + } + Some(Fragment { set, index, count, body: bytes.get(HEADER_BYTES..)? }) +} + +/// The update `parts` (one set's fragments, any order, each index at least once) make, if they are all of it and +/// they make the update the set names. +pub fn join<'a>(parts: impl IntoIterator) -> Option> { + let mut set = None; + let mut bodies: BTreeMap = BTreeMap::new(); + let mut count = 0; + for bytes in parts { + let f = parse(bytes)?; + if set.is_some_and(|s| s != f.set) || (count != 0 && count != f.count) { + return None; + } + set = Some(f.set); + count = f.count; + bodies.entry(f.index).or_insert(f.body); + } + if count == 0 || bodies.len() != usize::from(count) { + return None; + } + let whole: Vec = bodies.values().flat_map(|b| b.iter().copied()).collect(); + (Some(set_of(&whole)) == set).then_some(whole) +} + +/// One change's updates (its slices, in order) as one: the update itself when there is one, else a bundle +/// ([`BUNDLE_MAGIC`], their count, and each one's length and bytes; lengths and count are 4 bytes, big endian). +pub fn bundle(updates: &[Vec]) -> Vec { + if let [one] = updates { + return one.clone(); + } + let len = 8 + updates.iter().map(|u| 4 + u.len()).sum::(); + let mut out = Vec::with_capacity(len); + out.extend_from_slice(&BUNDLE_MAGIC); + out.extend_from_slice(&u32::try_from(updates.len()).unwrap_or(u32::MAX).to_be_bytes()); + for u in updates { + out.extend_from_slice(&u32::try_from(u.len()).unwrap_or(u32::MAX).to_be_bytes()); + out.extend_from_slice(u); + } + out +} + +/// Whether `bytes` are a bundle (they start with [`BUNDLE_MAGIC`]). +pub fn is_bundle(bytes: &[u8]) -> bool { + bytes.starts_with(&BUNDLE_MAGIC) +} + +/// The updates to import for `bytes`, in order: a bundle's, or the update itself; `None` for a bundle that doesn't +/// read (nothing of it is imported). +pub fn parts(bytes: &[u8]) -> Option> { + if !is_bundle(bytes) { + return Some(vec![bytes]); + } + let count = u32::from_be_bytes(bytes.get(4..8)?.try_into().ok()?) as usize; + let mut out = Vec::with_capacity(count.min(bytes.len() / 4)); + let mut at = 8; + for _ in 0..count { + let len = u32::from_be_bytes(bytes.get(at..at + 4)?.try_into().ok()?) as usize; + out.push(bytes.get(at + 4..at.checked_add(4 + len)?)?); + at += 4 + len; + } + (at == bytes.len()).then_some(out) +} + +/// What [`Reassembler::take`] did with a fragment. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Taken { + /// It was the last of its set: the whole update, to import. + Whole(Vec), + /// Kept: more of its set is to come. + Waiting, + /// Not kept: it doesn't read, doesn't fit its set, its set doesn't make the update it names, or too much waits. + Refused(&'static str), +} + +struct Partial { + count: u16, + parts: BTreeMap>, + bytes: usize, + last: Instant, +} + +/// The fragments of the sets still coming, by set: a set is imported only once it is whole, and dropped (never +/// applied) when no fragment of it came for a while ([`Reassembler::expire`]) or the connection it came on ends. +pub struct Reassembler { + sets: HashMap, + bytes: usize, + max_bytes: usize, +} + +impl std::fmt::Debug for Reassembler { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Reassembler").field("sets", &self.sets.len()).field("bytes", &self.bytes).finish() + } +} + +impl Reassembler { + /// One that keeps at most `max_bytes` of fragments. + pub fn new(max_bytes: usize) -> Self { + Self { sets: HashMap::new(), bytes: 0, max_bytes } + } + + /// Takes a fragment that came at `now`. + pub fn take(&mut self, frame: &[u8], now: Instant) -> Taken { + let Some(f) = parse(frame) else { return Taken::Refused("not a fragment") }; + // (Each fragment is charged its bookkeeping too, so tiny ones can't make many sets cheap.) + let cost = f.body.len().saturating_add(FRAGMENT_OVERHEAD); + let fits = self.bytes.saturating_add(cost) <= self.max_bytes; + let room = self.sets.len() < MAX_SETS; + let partial = match self.sets.get_mut(&f.set) { + Some(p) if p.count != f.count => return Taken::Refused("a fragment that doesn't fit its set"), + Some(p) => p, + None if !fits || !room => return Taken::Refused("too many fragments wait"), + None => self.sets.entry(f.set).or_insert(Partial { count: f.count, parts: BTreeMap::new(), bytes: 0, last: now }), + }; + partial.last = now; + if let std::collections::btree_map::Entry::Vacant(slot) = partial.parts.entry(f.index) { + if !fits { + return Taken::Refused("too many fragments wait"); + } + slot.insert(f.body.to_vec()); + partial.bytes += cost; + self.bytes += cost; + } + if partial.parts.len() < usize::from(partial.count) { + return Taken::Waiting; + } + let Some(done) = self.sets.remove(&f.set) else { return Taken::Waiting }; + self.bytes = self.bytes.saturating_sub(done.bytes); + let whole: Vec = done.parts.into_values().flatten().collect(); + if set_of(&whole) == f.set { Taken::Whole(whole) } else { Taken::Refused("its fragments don't make the update they name") } + } + + /// Drops the sets no fragment of which came within `patience` of `now`: how many. + pub fn expire(&mut self, now: Instant, patience: Duration) -> usize { + let before = self.sets.len(); + let mut freed = 0; + self.sets.retain(|_, p| { + let keep = now.saturating_duration_since(p.last) < patience; + if !keep { + freed += p.bytes; + } + keep + }); + self.bytes = self.bytes.saturating_sub(freed); + before - self.sets.len() + } + + /// Whether no set is coming. + pub fn is_empty(&self) -> bool { + self.sets.is_empty() + } + + /// How many sets are coming. + pub fn len(&self) -> usize { + self.sets.len() + } + + /// The bytes of fragments kept. + pub fn bytes(&self) -> usize { + self.bytes + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn update(len: usize) -> Vec { + (0..len).map(|i| u8::try_from(i * 7 % 251).unwrap_or(0)).collect() + } + + #[test] + fn an_update_that_fits_is_one_frame_and_not_a_fragment() { + let u = update(100); + assert_eq!(split(&u, 100).unwrap(), vec![u.clone()]); + assert!(!is_fragment(&u)); + } + + #[test] + fn a_big_update_is_cut_in_frames_that_fit_and_join_in_any_order() { + let u = update(10_000); + let frames = split(&u, 1000).unwrap(); + assert_eq!(frames.len(), 10_000usize.div_ceil(1000 - HEADER_BYTES)); + assert!(frames.iter().all(|f| f.len() <= 1000 && is_fragment(f))); + assert_eq!(split(&u, 1000).unwrap(), frames, "deterministic"); + let mut r = Reassembler::new(1 << 20); + let now = Instant::now(); + let mut order: Vec<&Vec> = frames.iter().rev().collect(); + // One fragment twice: kept once. + order.insert(1, &frames[frames.len() - 1]); + let last = order.pop().unwrap(); + for f in order { + assert_eq!(r.take(f, now), Taken::Waiting); + } + assert_eq!(r.take(last, now), Taken::Whole(u.clone())); + assert!(r.is_empty()); + assert_eq!(r.bytes(), 0); + assert_eq!(join(frames.iter().map(Vec::as_slice)), Some(u)); + } + + #[test] + fn a_set_that_stops_coming_is_dropped_and_never_whole() { + let frames = split(&update(5000), 1000).unwrap(); + let mut r = Reassembler::new(1 << 20); + let now = Instant::now(); + for f in &frames[..frames.len() - 1] { + assert_eq!(r.take(f, now), Taken::Waiting); + } + assert_eq!(r.expire(now + PATIENCE / 2, PATIENCE), 0); + assert_eq!(r.expire(now + PATIENCE, PATIENCE), 1); + assert!(r.is_empty()); + assert_eq!(r.bytes(), 0); + // Its last fragment alone, later: a set of its own again, still waiting. + assert_eq!(r.take(&frames[frames.len() - 1], now + PATIENCE), Taken::Waiting); + assert!(join(frames[1..].iter().map(Vec::as_slice)).is_none(), "a part missing"); + } + + #[test] + fn a_bundle_gives_its_updates_back_and_one_update_is_itself() { + let ups = vec![update(10), Vec::new(), update(70_000)]; + let b = bundle(&ups); + assert!(is_bundle(&b)); + assert_eq!(parts(&b).unwrap(), ups.iter().map(Vec::as_slice).collect::>()); + assert_eq!(bundle(&[update(5)]), update(5), "one update: itself"); + assert_eq!(parts(&update(5)).unwrap(), vec![update(5).as_slice()]); + assert!(parts(&b[..b.len() - 1]).is_none(), "cut short"); + let mut longer = b.clone(); + longer.push(0); + assert!(parts(&longer).is_none(), "junk after it"); + assert!(parts(b"ccb1\xff\xff\xff\xff").is_none()); + } + + #[test] + fn a_forged_or_broken_set_is_refused() { + let u = update(3000); + let mut frames = split(&u, 1000).unwrap(); + // A changed byte: the set doesn't make the update it names. + let last = frames.len() - 1; + if let Some(b) = frames[last].last_mut() { + *b ^= 1; + } + let mut r = Reassembler::new(1 << 20); + let now = Instant::now(); + for f in &frames[..last] { + r.take(f, now); + } + assert!(matches!(r.take(&frames[last], now), Taken::Refused(_))); + assert!(parse(b"ccf1short").is_none()); + assert!(matches!(r.take(b"loro", now), Taken::Refused(_))); + let mut small = Reassembler::new(10); + assert_eq!(small.take(&split(&u, 1000).unwrap()[0], now), Taken::Refused("too many fragments wait")); + assert_eq!(split(&update(MAX_WHOLE_BYTES + 1), 1 << 20), Err(SplitError::TooLarge(MAX_WHOLE_BYTES + 1))); + assert_eq!(split(&u, HEADER_BYTES), Err(SplitError::FrameTooSmall(HEADER_BYTES))); + } + + #[test] + fn the_sets_kept_are_bounded_in_number_as_well_as_bytes() { + let mut r = Reassembler::new(1 << 30); + let now = Instant::now(); + for i in 0..MAX_SETS { + let frames = split(&[update(300), vec![u8::try_from(i).unwrap()]].concat(), 100).unwrap(); + assert_eq!(r.take(&frames[0], now), Taken::Waiting); + } + let one_more = split(&update(301), 100).unwrap(); + assert_eq!(r.take(&one_more[0], now), Taken::Refused("too many fragments wait")); + assert_eq!(r.len(), MAX_SETS); + } +} diff --git a/crates/api/src/lib.rs b/crates/api/src/lib.rs index bb5682d..60a0b26 100644 --- a/crates/api/src/lib.rs +++ b/crates/api/src/lib.rs @@ -7,6 +7,7 @@ #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] pub mod api; +pub mod fragments; pub mod ids; pub mod limits; pub mod local; diff --git a/crates/api/src/local.rs b/crates/api/src/local.rs index c2b9874..a3337a9 100644 --- a/crates/api/src/local.rs +++ b/crates/api/src/local.rs @@ -18,8 +18,10 @@ //! documents ([`docs`]: [`DocsRequest`] and binary frames in, [`DocsEvent`] and binary frames out); `PUT /v1/blobs` //! and `GET /v1/blobs/{hash}`: binary content kept out of live documents ([`blobs`]). Errors are the HTTP API's error //! body ([`ErrorAnswer`](crate::api::ErrorAnswer) reads its code and message) with its codes and [`codes`]. -//! - **Versions**: [`Status::api`] is the local API version the agent speaks; a client refuses one outside -//! [`LOCAL_API_MIN`]..=[`LOCAL_API_MAX`] when it connects. +//! - **Versions**: [`Status::api`]..=[`Status::api_newest`] are the local API versions the agent speaks (`api` is the +//! oldest, so an app from before `api_max` still connects to a newer agent); a client refuses one whose versions +//! and its own [`LOCAL_API_MIN`]..=[`LOCAL_API_MAX`] don't overlap, and gates what it uses by `api_newest` (live +//! documents need [`LOCAL_API_FRAGMENTS`]). use std::ffi::OsString; use std::path::PathBuf; @@ -36,7 +38,11 @@ pub const LOCAL_API_FILE: &str = "local-api.json"; pub const LOCAL_API_SCHEMA: u32 = 1; /// Local API versions this build speaks. pub const LOCAL_API_MIN: u32 = 1; -pub const LOCAL_API_MAX: u32 = 1; +pub const LOCAL_API_MAX: u32 = 2; +/// The local API version from which the agent takes an app's change in fragments ([`fragments`](crate::fragments)) +/// and sends every change to its apps whole, and its server log keeps a change whole: an app's live documents need +/// an agent of at least this version (an older one is told to update Craft Cloud). +pub const LOCAL_API_FRAGMENTS: u32 = 2; /// The environment variable that overrides the agent's data folder. pub const DATA_DIR_ENV: &str = "CRAFT_CLOUD_AGENT_DIR"; /// The data folder's name inside the OS's per-user data folder. @@ -241,8 +247,13 @@ pub enum RunState { /// `GET /v1/status`. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct Status { - /// The local API version the agent speaks. + /// The oldest local API version the agent speaks: what an app from before `api_max` checks (so an older app + /// still connects to a newer agent, and gets what that version has). pub api: u32, + /// The newest local API version the agent speaks; none from an agent before it (that one speaks `api` only). + /// An app gates what it uses by it ([`Status::api_newest`]). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub api_max: Option, pub signed_in: bool, /// The account's username, when signed in. #[serde(default)] @@ -273,6 +284,13 @@ pub struct Status { pub removed: bool, } +impl Status { + /// The newest local API version the agent speaks: what an app's features are gated by. + pub fn api_newest(&self) -> u32 { + self.api_max.unwrap_or(self.api).max(self.api) + } +} + /// `POST /v1/login`: what `craft-cloud-agent login` asks for. The password goes to the agent and from there to the /// server only; it is never written anywhere. Its `Debug` output hides the password. #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] @@ -760,6 +778,7 @@ mod tests { ); let status = Status { api: 1, + api_max: Some(2), signed_in: false, account: None, server: None, @@ -783,6 +802,8 @@ mod tests { assert_eq!(older.autostart, None, "an older agent says nothing about autostart"); assert_eq!(older.last_sync_ms, None, "nor about the last sync"); assert!(!older.removed); + assert_eq!((older.api_max, older.api_newest()), (None, 1), "an older agent speaks its one version"); + assert_eq!(status.api_newest(), 2); let login: LoginRequest = serde_json::from_value(serde_json::json!({ "server": "s", "user": "u", "password": "p", "extra": 1 })).unwrap(); assert!(!login.adopt && login.folder.is_none(), "missing optional fields default; unknown ones are ignored"); } diff --git a/crates/app/src/fake.rs b/crates/app/src/fake.rs index ee5b8de..5e7cd3c 100644 --- a/crates/app/src/fake.rs +++ b/crates/app/src/fake.rs @@ -9,9 +9,16 @@ //! when the machine is back, and the other machines' updates come down then. What the bytes mean is never read: the //! state an app opens is the import followed by the updates as `update` frames (the local API allows that). //! +//! A change an app sends in fragments goes into the machine's copy (and is acked) only once it is whole, as its +//! fragments, each an update of the log (as the agent's log and the server keep them). A machine joins them before +//! its apps get the change, as the agent does: an app never gets a fragment. [`Fake::set_fragment_faults`] stops a +//! machine's link from the hub inside such a change for a while (a laptop that sleeps part of the way through it; +//! [`Hub::tick`] moves the pauses on), and [`Fake::drop_connections_after`] stops the agent part of the way through an +//! app's change. +//! //! [`Fake::restart`] drops every app connection and refuses new ones until [`Fake::start`] (an agent that stopped): -//! what the machine held stays, as the agent's files do. [`Fake::too_old`] makes a machine answer the blob calls as an -//! agent from before blobs (`not_found` for the call). [`Hub::set_live`] holds live updates back between machines while +//! what the machine held stays, as the agent's files do. [`Fake::too_old`] makes a machine answer as an agent from +//! before blobs and changes in fragments (`not_found` for the blob calls, local API version 1). [`Hub::set_live`] holds live updates back between machines while //! saved files' records still go (a proxy that blocks WebSockets). use std::collections::{HashMap, VecDeque}; @@ -22,6 +29,7 @@ use std::sync::{Arc, Mutex, PoisonError}; use std::time::Duration; use craft_cloud_api::DocId; +use craft_cloud_api::fragments::{SetId, is_fragment, join, parse}; use craft_cloud_api::limits::{MAX_BLOB_BYTES, MAX_PRESENCE_BYTES, MAX_UPDATE_BYTES}; use craft_cloud_api::local::blobs::{Stored, hex, parse_hex}; use craft_cloud_api::local::docs::{IMPORT, MAX_BINARY_BYTES, PRESENCE, STATE, UPDATE, decode_binary, encode_binary}; @@ -82,6 +90,11 @@ struct MachineDoc { hub_seen: usize, /// Positions in `log` of this machine's updates the hub doesn't have yet. pending: VecDeque, + /// Ticks the hub's rows wait for (from `hub_seen` on): the machine's link stopped inside a change in fragments + /// ([`FragmentFaults`]). + pause: u32, + /// The hub row a pause stopped at: taken without asking the faults again once the pause is over. + paused_at: Option, } struct Machine { @@ -100,6 +113,82 @@ struct Machine { refuse: VecDeque, /// Saves made here that haven't gone up yet (offline): each path's file record on this machine until they do. file_pending: HashMap, + /// What happens to fragments on their way from the hub to this machine ([`Fake::set_fragment_faults`]). + faults: Option, + /// How many times the faults stopped the link inside a change. + pauses: u64, + /// App updates (frames) left before every app connection ends ([`Fake::drop_connections_after`]). + drop_after: Option, +} + +/// What happens to the fragments of changes on their way from the hub (the server's log) to a machine, from a seeded +/// generator: before some fragment after a set's first, the machine's link stops for a while (the laptop sleeps +/// part of the way through the change, or its connection drops), and the rest of the log comes later. The machine +/// keeps what came, as the agent does, and its apps get the change whole once the rest is there, never part of it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct FragmentFaults { + pub seed: u64, + /// Of every 1000 fragments after a set's first, before how many the machine sleeps ([`SLEEP_TICKS`] ticks). + pub sleep: u32, + /// Of every 1000 fragments after a set's first, before how many the link stops a few ticks. + pub delay: u32, + /// The most ticks such a stop lasts. + pub max_delay: u32, +} + +/// How many [`Hub::tick`]s a machine sleeps inside a change ([`FragmentFaults::sleep`]). +pub const SLEEP_TICKS: u32 = 20; + +struct Faults { + spec: FragmentFaults, + state: u64, +} + +impl Faults { + fn next(&mut self) -> u64 { + self.state = self.state.wrapping_add(0x9e37_79b9_7f4a_7c15); + let mut x = self.state; + x = (x ^ (x >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9); + x = (x ^ (x >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb); + x ^ (x >> 31) + } + + /// For one fragment: how many ticks the link stops before it (0: it comes now). + fn fate(&mut self, bytes: &[u8]) -> u32 { + if parse(bytes).is_none_or(|f| f.index == 0) { + return 0; + } + let roll = u32::try_from(self.next() % 1000).unwrap_or(0); + if roll < self.spec.sleep { + SLEEP_TICKS + } else if roll < self.spec.sleep.saturating_add(self.spec.delay) { + 1 + u32::try_from(self.next() % u64::from(self.spec.max_delay.max(1))).unwrap_or(0) + } else { + 0 + } + } +} + +/// A row of a machine's log on its way to an app, joined as the agent joins: a change in fragments is kept (in +/// `kept`, by set) until it is whole, and goes whole; anything else goes as it is. `None` while it isn't whole (or +/// for a fragment that doesn't read). +fn joined_row(kept: &mut HashMap>>, bytes: &[u8]) -> Option> { + if !is_fragment(bytes) { + return Some(bytes.to_vec()); + } + let f = parse(bytes)?; + let set = kept.entry(f.set).or_default(); + set.push(bytes.to_vec()); + let mut indexes: Vec = set.iter().filter_map(|b| parse(b).map(|x| x.index)).collect(); + indexes.sort_unstable(); + indexes.dedup(); + if indexes.len() < usize::from(f.count) { + return None; + } + let mut frames = kept.remove(&f.set)?; + frames.sort_by_key(|b| parse(b).map(|x| x.index)); + frames.dedup_by_key(|b| parse(b).map(|x| x.index)); + join(frames.iter().map(Vec::as_slice)) } /// A live document's frontiers: `(peer, counter)` heads. @@ -118,12 +207,20 @@ struct Conn { told: HashMap, /// The file record each document's apps were last told. told_file: HashMap>, + /// The fragments of the app's changes still coming, by document number and set: with their numbers. A change goes + /// into the machine's copy (and is acked) only once it is whole, as the agent takes it. + sets: HashMap<(u64, SetId), Vec>, } +/// An app's update frame and its number. +type Numbered = (u64, Vec); + struct Opened { doc: DocId, sent: usize, seq: u64, + /// The fragments of the log's changes not whole yet, by set: the app gets each change whole ([`joined_row`]). + kept: HashMap>>, } fn lock(m: &Mutex) -> std::sync::MutexGuard<'_, T> { @@ -159,6 +256,9 @@ impl Hub { saves: Vec::new(), refuse: VecDeque::new(), file_pending: HashMap::new(), + faults: None, + pauses: 0, + drop_after: None, }, ); id @@ -216,11 +316,28 @@ impl Hub { lock(&self.inner).files.get(path).cloned().flatten() } + /// The updates the hub holds of the document at `path`, in order (each a row of the server's log). + pub fn rows(&self, path: &str) -> Vec> { + lock(&self.inner).docs.get(path).map(|d| d.log.iter().map(|(_, b)| b.clone()).collect()).unwrap_or_default() + } + /// How many updates the hub holds of the document at `path` (0 when it has none). pub fn updates(&self, path: &str) -> usize { lock(&self.inner).docs.get(path).map_or(0, |d| d.log.len()) } + /// A tick: a machine's link stopped inside a change ([`FragmentFaults`]) comes one tick closer to going on, and the + /// rows of the ones due come. + pub fn tick(&self) { + let mut inner = lock(&self.inner); + for m in inner.machines.values_mut() { + for d in m.docs.values_mut() { + d.pause = d.pause.saturating_sub(1); + } + } + sync(&mut inner); + } + fn serve(&self, machine: usize, mut stream: TcpStream) { let _ = stream.set_read_timeout(Some(Duration::from_secs(10))); let (token, running, too_old, name) = { @@ -247,7 +364,8 @@ impl Hub { ("GET", paths::STATUS) => { let online = lock(&self.inner).machines.get(&machine).is_some_and(|m| m.online); let status = Status { - api: LOCAL_API_MAX, + api: 1, + api_max: (!too_old).then_some(LOCAL_API_MAX), signed_in: true, account: Some("fake".into()), server: Some("https://fake.invalid".into()), @@ -346,6 +464,7 @@ impl Hub { apps: HashMap::new(), told: HashMap::new(), told_file: HashMap::new(), + sets: HashMap::new(), }, ); conn @@ -419,6 +538,18 @@ impl Hub { } UPDATE => { let Some(m) = inner.machines.get_mut(&machine) else { return }; + // An agent that stops after so many updates ([`Fake::drop_connections_after`]): this one never + // arrives, and every app connection ends. + if let Some(left) = m.drop_after.as_mut() { + if *left == 0 { + m.drop_after = None; + for (_, c) in m.conns.drain() { + let _ = c.stream.shutdown(Shutdown::Both); + } + return; + } + *left -= 1; + } let Some(c) = m.conns.get_mut(&conn) else { return }; let Some(opened) = c.open.get_mut(&id) else { return }; opened.seq += 1; @@ -427,14 +558,44 @@ impl Hub { let message = format!("an update is 1 byte to {MAX_UPDATE_BYTES} bytes"); return tell(&inner, machine, conn, &DocsEvent::Error { id: Some(id), seq: Some(seq), code: "too_large".into(), message }); } - // A refusal asked for ([`Fake::refuse_next`]): the update isn't kept. - if let Some(code) = m.refuse.pop_front() { + // A fragment: kept until its change is whole, which then goes into the copy as its fragments (each an + // update of the log, as the agent's log keeps them). As the agent's local API does, the change is + // answered once, under the number of the fragment that made it whole; the earlier ones get nothing. + let frames: Vec<(u64, Vec)> = if is_fragment(body) { + let Some(f) = parse(body) else { + let message = "a fragment that doesn't read".to_owned(); + return tell(&inner, machine, conn, &DocsEvent::Error { id: Some(id), seq: Some(seq), code: "invalid".into(), message }); + }; + let set = c.sets.entry((id, f.set)).or_default(); + set.push((seq, body.to_vec())); + let mut indexes: Vec = set.iter().filter_map(|(_, b)| parse(b).map(|x| x.index)).collect(); + indexes.sort_unstable(); + indexes.dedup(); + if indexes.len() < usize::from(f.count) { + return; + } + let mut frames = c.sets.remove(&(id, f.set)).unwrap_or_default(); + if join(frames.iter().map(|(_, b)| b.as_slice())).is_none() { + let message = "fragments that don't make the change they name".to_owned(); + return tell(&inner, machine, conn, &DocsEvent::Error { id: Some(id), seq: Some(seq), code: "invalid".into(), message }); + } + frames.sort_by_key(|(_, b)| parse(b).map(|x| x.index)); + frames.dedup_by_key(|(_, b)| parse(b).map(|x| x.index)); + frames + } else { + vec![(seq, body.to_vec())] + }; + // A refusal asked for ([`Fake::refuse_next`]): the change isn't kept. + if let Some(code) = inner.machines.get_mut(&machine).and_then(|m| m.refuse.pop_front()) { let message = format!("refused as asked ({code})"); return tell(&inner, machine, conn, &DocsEvent::Error { id: Some(id), seq: Some(seq), code, message }); } + let Some(m) = inner.machines.get_mut(&machine) else { return }; if let Some(d) = m.docs.get_mut(&doc) { - d.log.push((Origin::Conn(conn), body.to_vec())); - d.pending.push_back(d.log.len() - 1); + for (_, bytes) in &frames { + d.log.push((Origin::Conn(conn), bytes.clone())); + d.pending.push_back(d.log.len() - 1); + } } tell(&inner, machine, conn, &DocsEvent::Ack { id, seq }); sync(&mut inner); @@ -476,7 +637,10 @@ fn open(inner: &mut Inner, machine: usize, conn: u64, id: u64, path: &str) { let doc = match (known, hub_doc) { (Some(doc), _) => doc, (None, Some((doc, import))) if m.online => { - m.docs.insert(doc, MachineDoc { path: path.to_owned(), import, log: Vec::new(), hub_seen: 0, pending: VecDeque::new() }); + m.docs.insert( + doc, + MachineDoc { path: path.to_owned(), import, log: Vec::new(), hub_seen: 0, pending: VecDeque::new(), pause: 0, paused_at: None }, + ); doc } (None, None) if m.online => { @@ -496,10 +660,13 @@ fn open(inner: &mut Inner, machine: usize, conn: u64, id: u64, path: &str) { let Some(m) = inner.machines.get_mut(&machine) else { return }; let online = m.online; let Some(d) = m.docs.get(&doc) else { return }; - let state = DocState { online, waiting: d.pending.len() as u64, bytes: d.import.len() as u64, updates: d.log.len() as u64 }; - let (import, log): (Vec, Vec>) = (d.import.clone(), d.log.iter().map(|(_, b)| b.clone()).collect()); + // (Its changes whole: one whose fragments aren't all here yet is kept, and goes once it is.) + let mut kept = HashMap::new(); + let log: Vec> = d.log.iter().filter_map(|(_, b)| joined_row(&mut kept, b)).collect(); + let state = DocState { online, waiting: d.pending.len() as u64, bytes: d.import.len() as u64, updates: log.len() as u64 }; + let (import, rows) = (d.import.clone(), d.log.len()); let Some(c) = m.conns.get_mut(&conn) else { return }; - c.open.insert(id, Opened { doc, sent: log.len(), seq: 0 }); + c.open.insert(id, Opened { doc, sent: rows, seq: 0, kept }); c.told.insert(doc, (online, state.waiting)); let mut w = lock(&c.writer); let opened = DocsEvent::Opened { id, doc, state, changed_outside }; @@ -555,35 +722,50 @@ fn sync(inner: &mut Inner) { } } } - // Down: what the others sent. + // Down: what the others sent, in the log's order. The link may stop inside a change in fragments + // ([`FragmentFaults`]): the rows from there on come once the pause is over. for mid in &ids { let Some(m) = machines.get_mut(mid) else { continue }; if !m.online { continue; } - for d in m.docs.values_mut() { + let Machine { docs: machine_docs, faults, pauses, .. } = m; + for d in machine_docs.values_mut() { let Some(hub) = docs.get(&d.path) else { continue }; - if live_down { + if live_down || d.pause > 0 { continue; } - for (from, bytes) in hub.log.iter().skip(d.hub_seen) { + while let Some((from, bytes)) = hub.log.get(d.hub_seen) { if from != mid { + if d.paused_at != Some(d.hub_seen) + && let Some(ticks) = faults.as_mut().map(|f| f.fate(bytes)).filter(|t| *t > 0) + { + d.pause = ticks; + d.paused_at = Some(d.hub_seen); + *pauses += 1; + break; + } d.log.push((Origin::Remote, bytes.clone())); } + d.hub_seen += 1; } - d.hub_seen = hub.log.len(); } } - // Out to the apps: the copy's new updates (an app's own not back to it), and status when it changed. + // Out to the apps: the copy's new updates (an app's own not back to it), each change whole (as the agent joins + // the server's fragments before its apps get a change), and status when it changed. for m in machines.values_mut() { let online = m.online; - for (cid, c) in m.conns.iter_mut() { + let Machine { conns, docs: machine_docs, file_pending, .. } = m; + for (cid, c) in conns.iter_mut() { let mut w = lock(&c.writer); for (number, o) in c.open.iter_mut() { - let Some(d) = m.docs.get(&o.doc) else { continue }; + let Some(d) = machine_docs.get(&o.doc) else { continue }; for (origin, bytes) in d.log.iter().skip(o.sent) { - if *origin != Origin::Conn(*cid) { - let _ = w.send(Message::Binary(encode_binary(UPDATE, *number, bytes).into())); + if *origin == Origin::Conn(*cid) { + continue; + } + if let Some(update) = joined_row(&mut o.kept, bytes) { + let _ = w.send(Message::Binary(encode_binary(UPDATE, *number, &update).into())); } } o.sent = d.log.len(); @@ -596,7 +778,7 @@ fn sync(inner: &mut Inner) { } } // The file here: this machine's save not up yet, else the server's (offline: as it was). - let file = match m.file_pending.get(&d.path) { + let file = match file_pending.get(&d.path) { Some(v) => Some(v.clone()), None if online => files.get(&d.path).cloned().flatten(), None => c.told_file.get(&o.doc).cloned().flatten(), @@ -630,7 +812,9 @@ impl std::fmt::Debug for Fake { impl Fake { /// A client of this machine's agent, as [`Client::connect`] gives one. pub fn client(&self) -> Client { - Client::at(self.addr, self.token.clone(), std::env::temp_dir()) + // (Its versions known, as a client that connected through `local-api.json` knows them: no call of its own.) + let too_old = lock(&self.hub.inner).machines.get(&self.id).is_some_and(|m| m.too_old); + Client::at(self.addr, self.token.clone(), std::env::temp_dir(), if too_old { 1 } else { LOCAL_API_MAX }) } /// The machine reaches the hub (`true`) or not: offline, its updates and blobs wait, and others' don't come. @@ -675,6 +859,30 @@ impl Fake { } } + /// From now on (`Some`), this machine's link from the hub stops inside changes in fragments as `faults` say + /// ([`Hub::tick`] moves the pauses on); `None` stops it, and ends a pause under way (the machine wakes). + pub fn set_fragment_faults(&self, faults: Option) { + let mut inner = lock(&self.hub.inner); + if let Some(m) = inner.machines.get_mut(&self.id) { + if faults.is_none() { + for d in m.docs.values_mut() { + d.pause = 0; + } + } + m.faults = faults.map(|spec| Faults { spec, state: spec.seed }); + } + sync(&mut inner); + } + + /// After `updates` more update frames from its apps, the agent drops every app connection (as when it stops or + /// crashes): that frame never arrives, and a change whose fragments were coming is never taken (the app sends it + /// again once it is back). + pub fn drop_connections_after(&self, updates: usize) { + if let Some(m) = lock(&self.hub.inner).machines.get_mut(&self.id) { + m.drop_after = Some(updates); + } + } + /// Refuses the next `count` updates any app of this machine sends with `code`, as the agent does: `internal` when /// it can't write a change to disk (worth sending again), `invalid` when Loro doesn't take it (never). pub fn refuse_next(&self, code: &str, count: usize) { @@ -683,7 +891,8 @@ impl Fake { } } - /// Answer the blob calls as an agent from before blobs. + /// Answer as an agent from before blobs and changes in fragments: the blob calls aren't known, and the status + /// names local API version 1 (an app's live documents are refused as `agent_too_old` before they connect). pub fn too_old(&self) { if let Some(m) = lock(&self.hub.inner).machines.get_mut(&self.id) { m.too_old = true; @@ -700,6 +909,11 @@ impl Fake { let inner = lock(&self.hub.inner); inner.machines.get(&self.id).map_or(0, |m| m.docs.values().filter(|d| d.path == path).map(|d| d.pending.len()).sum()) } + + /// How many times this machine's link stopped inside a change in fragments ([`FragmentFaults`]). + pub fn pauses(&self) -> u64 { + lock(&self.hub.inner).machines.get(&self.id).map_or(0, |m| m.pauses) + } } impl Drop for Fake { @@ -828,6 +1042,11 @@ mod tests { panic!("no update came"); } + /// Whether no update comes for `doc` within about 200 ms. + fn no_update(doc: &crate::live::Document) -> bool { + (0..8).all(|_| !matches!(doc.recv_timeout(Duration::from_millis(25)), Some(Event::Update(_)))) + } + fn acked(doc: &crate::live::Document, seq: u64) { for _ in 0..200 { if let Some(Event::Acked(s)) = doc.recv_timeout(Duration::from_millis(25)) @@ -888,12 +1107,77 @@ mod tests { assert!(matches!(&put, Err(crate::Error::Refused { code, .. }) if code == "agent_too_old"), "{put:?}"); assert!(matches!(b.client().blobs().get(&stored.hash, stored.len), Err(crate::Error::Refused { code, .. }) if code == "agent_too_old")); assert!(b.client().status().is_ok(), "the rest still works"); - let big = vec![0u8; MAX_UPDATE_BYTES + 1]; - let seq = doc_a.send(&big).unwrap(); - let refused = (0..200).find_map(|_| match doc_a.recv_timeout(Duration::from_millis(25)) { - Some(Event::Refused { seq: s, .. }) => Some(s), - _ => None, - }); - assert_eq!(refused, Some(seq), "an update over 1 MiB is refused"); + assert_eq!( + b.client().documents().unwrap_err(), + crate::Error::Refused { code: "agent_too_old".into(), message: crate::live::AGENT_TOO_OLD.into() }, + "live documents need a newer agent" + ); + // An update over 1 MiB goes in fragments: it comes whole, acked once; one over a snapshot's limit isn't sent. + let big: Vec = (0..3 * MAX_UPDATE_BYTES).map(|i| (i % 253) as u8).collect(); + acked(&doc_a, doc_a.send(&big).unwrap()); + assert_eq!(next_update(&again), big); + let fragments = craft_cloud_api::fragments::split(&big, MAX_UPDATE_BYTES).unwrap().len(); + assert_eq!(fragments, 4); + assert_eq!(hub.updates("x.txt"), 4 + fragments, "its fragments, each an update of the log"); + let huge = vec![0u8; craft_cloud_api::fragments::MAX_WHOLE_BYTES + 1]; + assert!(matches!(doc_a.send(&huge), Err(crate::Error::Refused { code, .. }) if code == "too_large")); + } + + /// A change in fragments is answered once, under its last fragment's number, as by the agent: acked, the app + /// library keeps nothing of it; refused, the app hears it once, and nothing is kept once the document is + /// dropped. + #[test] + fn a_change_in_fragments_is_answered_once_and_nothing_is_kept() { + let hub = Hub::new(); + let a = hub.machine("a").unwrap(); + let docs = a.client().documents().unwrap(); + let Opened::NeedsImport(importer) = docs.open("k.txt", "test").unwrap() else { panic!("not new") }; + let doc = importer.import(b"snapshot").unwrap(); + let big: Vec = (0..2 * MAX_UPDATE_BYTES + 5).map(|i| (i % 241) as u8).collect(); + acked(&doc, doc.send(&big).unwrap()); + assert_eq!(docs.kept_numbers(), 0, "nothing kept once the change is acked"); + a.refuse_next("invalid", 1); + let last = doc.send(&big).unwrap(); + let refused: Vec = (0..40) + .filter_map(|_| match doc.recv_timeout(Duration::from_millis(25)) { + Some(Event::Refused { seq, .. }) => Some(seq), + _ => None, + }) + .collect(); + assert_eq!(refused, vec![last], "refused once, under the last number"); + drop(doc); + assert_eq!(docs.kept_numbers(), 0, "nothing kept once the document is dropped"); + } + + /// A machine whose link stops inside a change in fragments keeps what came: its apps get nothing of the change + /// (nor what comes after it in the log) until the rest is there, then the change whole; an app opening the + /// document meanwhile gets it whole once it is, never a fragment. + #[test] + fn a_machine_that_sleeps_inside_a_change_gives_its_apps_the_change_whole() { + let hub = Hub::new(); + let (a, b) = (hub.machine("a").unwrap(), hub.machine("b").unwrap()); + let docs_a = a.client().documents().unwrap(); + let Opened::NeedsImport(importer) = docs_a.open("f.txt", "test").unwrap() else { panic!("not new") }; + let doc_a = importer.import(b"snapshot").unwrap(); + let docs_b = b.client().documents().unwrap(); + let Opened::Live(doc_b) = docs_b.open("f.txt", "test").unwrap() else { panic!("not live on b") }; + let big: Vec = (0..2 * MAX_UPDATE_BYTES + 5).map(|i| (i % 251) as u8).collect(); + b.set_fragment_faults(Some(FragmentFaults { seed: 1, sleep: 1000, delay: 0, max_delay: 0 })); + doc_a.send(&big).unwrap(); + doc_a.send(b"small").unwrap(); + assert!(no_update(&doc_b), "asleep inside the change"); + let Opened::Live(meanwhile) = docs_b.open("f.txt", "test").unwrap() else { panic!("not live on b") }; + assert!(meanwhile.state_updates().is_empty(), "nothing of the change in the state"); + // Awake after the sleep: the next fragment, then asleep again before the last. + for _ in 0..SLEEP_TICKS { + hub.tick(); + } + assert!(no_update(&doc_b), "still not whole"); + b.set_fragment_faults(None); + assert_eq!(next_update(&doc_b), big, "the change, whole"); + assert_eq!(next_update(&doc_b), b"small"); + assert_eq!(next_update(&meanwhile), big, "whole for the app that opened it meanwhile too"); + let Opened::Live(again) = docs_b.open("f.txt", "test").unwrap() else { panic!("not live on b") }; + assert_eq!(again.state_updates(), [big, b"small".to_vec()], "the state has it whole"); } } diff --git a/crates/app/src/lib.rs b/crates/app/src/lib.rs index 3f5d9af..7b32732 100644 --- a/crates/app/src/lib.rs +++ b/crates/app/src/lib.rs @@ -98,6 +98,9 @@ pub struct Client { dir: PathBuf, /// Whether this agent knows `/v1/blobs`, once asked ([`blobs::Blobs`]): shared by the client's clones. blobs_known: std::sync::Arc>, + /// The newest local API version the agent speaks, once its status said ([`Client::agent_api`]): shared by the + /// client's clones. + api_known: std::sync::Arc>, } impl Client { @@ -120,9 +123,15 @@ impl Client { token: record.token, dir: dir.to_owned(), blobs_known: Default::default(), + api_known: Default::default(), }; match client.status() { - Ok(status) if (LOCAL_API_MIN..=LOCAL_API_MAX).contains(&status.api) => Ok(client), + // Any agent that speaks a version this app does: what the app uses past the oldest is gated by + // `Status::api_newest` (live documents need `LOCAL_API_FRAGMENTS`). + Ok(status) if speaks(&status) => { + let _ = client.api_known.set(status.api_newest()); + Ok(client) + } Ok(status) => Err(Error::Version(status.api)), Err(Error::Refused { code, .. }) if code == "unauthorized" => Err(Error::NotRunning(format!("{} is stale", file.display()))), Err(Error::Io(e) | Error::Protocol(e)) => Err(Error::NotRunning(format!("{} is stale ({e})", file.display()))), @@ -166,8 +175,20 @@ impl Client { /// A client of the agent at `addr` with `token` (a fake agent's, in tests), whose `local-api.json` is in `dir`. #[cfg(feature = "fake")] - pub(crate) fn at(addr: SocketAddr, token: String, dir: PathBuf) -> Self { - Self { addr, token, dir, blobs_known: Default::default() } + pub(crate) fn at(addr: SocketAddr, token: String, dir: PathBuf, api: u32) -> Self { + let api_known = std::sync::Arc::new(std::sync::OnceLock::new()); + let _ = api_known.set(api); + Self { addr, token, dir, blobs_known: Default::default(), api_known } + } + + /// The newest local API version the agent speaks: known from connecting, or asked once (a call of its own). + pub fn agent_api(&self) -> Result { + if let Some(api) = self.api_known.get() { + return Ok(*api); + } + let api = self.status()?.api_newest(); + let _ = self.api_known.set(api); + Ok(api) } /// Whether the agent is signed in, as whom, to which server, whether it is online and what it is doing. Quick @@ -283,6 +304,11 @@ impl Client { } } +/// Whether the agent speaks a local API version this app does (its range and the app's overlap). +fn speaks(status: &craft_cloud_api::local::Status) -> bool { + status.api <= LOCAL_API_MAX && status.api_newest() >= LOCAL_API_MIN +} + /// `answer` if it succeeded, else the agent's refusal (its code and message). fn refusal(answer: http::Answer) -> Result { if (200..300).contains(&answer.status) { @@ -355,6 +381,7 @@ mod tests { fn status(signed_in: bool) -> Status { Status { api: 1, + api_max: Some(LOCAL_API_MAX), signed_in, account: signed_in.then(|| "ana".into()), server: None, @@ -372,6 +399,16 @@ mod tests { serde_json::to_string(value).unwrap() } + /// An app takes any agent whose versions overlap its own: an older one (version 1 only), a newer one (from 1 on, + /// up to past this app's), never one that speaks only versions after this app's. + #[test] + fn an_agent_whose_versions_overlap_is_taken() { + let with = |api, api_max| Status { api, api_max, ..status(true) }; + assert!(speaks(&with(1, None)), "an agent before api_max"); + assert!(speaks(&with(1, Some(LOCAL_API_MAX + 5))), "a newer agent"); + assert!(!speaks(&with(LOCAL_API_MAX + 1, Some(LOCAL_API_MAX + 5))), "one with nothing this app speaks"); + } + /// Every call sends its request to its path, and reads its answer; a refusal comes back with its code. #[test] fn each_call_sends_its_request_and_reads_its_answer() { diff --git a/crates/app/src/live.rs b/crates/app/src/live.rs index 1e3eba6..4d66dab 100644 --- a/crates/app/src/live.rs +++ b/crates/app/src/live.rs @@ -44,8 +44,10 @@ use std::time::Duration; use craft_cloud_api::DocId; use craft_cloud_api::api::ErrorAnswer as ErrorBody; +use craft_cloud_api::fragments::split; +use craft_cloud_api::limits::MAX_UPDATE_BYTES; use craft_cloud_api::local::docs::{IMPORT, MAX_BINARY_BYTES, PRESENCE, STATE, UPDATE, decode_binary, encode_binary}; -use craft_cloud_api::local::{DocState, DocsEvent, DocsRequest, MAX_LOCAL_JSON_BYTES, paths}; +use craft_cloud_api::local::{DocState, DocsEvent, DocsRequest, LOCAL_API_FRAGMENTS, MAX_LOCAL_JSON_BYTES, codes, paths}; use tungstenite::client::IntoClientRequest; use tungstenite::handshake::HandshakeError; use tungstenite::protocol::WebSocketConfig; @@ -67,6 +69,9 @@ const WRITE_TIMEOUT: Duration = Duration::from_secs(10); /// How long the reader waits for data before it looks whether it is to stop. const IDLE_WAKE: Duration = Duration::from_millis(250); +/// What an agent too old for this app's live documents is refused with ([`codes::AGENT_TOO_OLD`]), for people. +pub const AGENT_TOO_OLD: &str = "Craft Cloud on this computer is older than this app; update Craft Cloud to edit live"; + /// What happens to an open document. #[derive(Debug, Clone, PartialEq, Eq)] pub enum Event { @@ -91,6 +96,24 @@ pub enum Event { Closed(String), } +/// What the agent's answer for a fragment of this app's that isn't its change's last is ([`Document::send`]). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Inner { + /// Not the app's (its change is answered under `last`), unless it is a refusal: then the change is refused. + Of(u64), + /// A number of a change refused already: its answers are dropped (until the document is). + Answered, +} + +/// For tests and simulations: updates over `bytes` go in fragments of at most `bytes` from now on, in this process (so +/// changes of a few dozen KB exercise what changes over 1 MiB do). At most the frame limit. +#[doc(hidden)] +pub fn set_frame_limit_for_tests(bytes: usize) { + FRAME_LIMIT.store(bytes.clamp(craft_cloud_api::fragments::HEADER_BYTES + 1, MAX_UPDATE_BYTES), Ordering::Relaxed); +} + +static FRAME_LIMIT: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(MAX_UPDATE_BYTES); + /// What opening a file gives. #[derive(Debug)] pub enum Opened { @@ -119,7 +142,11 @@ struct Head { } /// An open's state so far: what `opened` said, the snapshot, and the updates that are part of it. -type Filling = (Head, Vec, Vec>); +struct Filling { + head: Head, + snapshot: Vec, + updates: Vec>, +} /// What the reader hands an open in progress. enum Answer { @@ -144,6 +171,9 @@ struct Shared { numbers: Mutex>>, /// What to call after an event went to a document's channel, by number ([`Document::set_waker`]). wakers: Mutex>, + /// The numbers (document number, `seq`) of the fragments sent that aren't a change's last, and what their answers + /// are ([`Inner`]): the agent answers for a change by its last fragment's number. + inner_seqs: Mutex>, closed: AtomicBool, next_id: AtomicU64, /// A clone of the stream: dropping the connection shuts it down (which wakes the reader). @@ -171,6 +201,49 @@ impl Shared { self.send(Message::Binary(encode_binary(kind, id, bytes).into())) } + /// Sends `frames` (one change's fragments) back to back: nothing of another sender's goes between them, so their + /// numbers follow on. + fn send_frames(&self, kind: u8, id: u64, frames: &[Vec]) -> Result<(), Error> { + if self.closed.load(Ordering::Relaxed) { + return Err(Error::Io("the connection to the agent is closed".into())); + } + let mut socket = lock(&self.socket); + for frame in frames { + socket.send(Message::Binary(encode_binary(kind, id, frame).into())).map_err(|e| Error::Io(e.to_string()))?; + } + Ok(()) + } + + /// What the agent's answer for (`id`, `seq`) is, when it isn't simply the app's: a fragment that isn't a change's + /// last, or a number of a change refused already. + fn inner(&self, id: u64, seq: u64) -> Option { + lock(&self.inner_seqs).get(&(id, seq)).copied() + } + + /// The change ending at `last` was answered for (acked, or refused under its last number): what was kept for its + /// earlier fragments goes (the agent answers a change once, by its last number). + fn answered(&self, id: u64, last: u64) { + lock(&self.inner_seqs).retain(|(i, _), what| !(*i == id && *what == Inner::Of(last))); + } + + /// A fragment of a change (not its last, `last`) was refused: the change is, under its last number, once; every + /// answer still to come for any of its numbers is dropped (kept as answered until the document is dropped). + fn refuse_change(&self, id: u64, last: u64, code: String, message: String) { + { + let mut inner = lock(&self.inner_seqs); + for what in inner.iter_mut().filter(|((i, _), w)| *i == id && **w == Inner::Of(last)).map(|(_, w)| w) { + *what = Inner::Answered; + } + inner.insert((id, last), Inner::Answered); + } + self.route(id, Event::Refused { seq: last, code, message }); + } + + /// What was kept for document number `id`'s changes goes (it was dropped). + fn forget(&self, id: u64) { + lock(&self.inner_seqs).retain(|(i, _), _| *i != id); + } + fn route(&self, id: u64, event: Event) { if let Some(events) = lock(&self.routes).get(&id) { let _ = events.send(event); @@ -200,7 +273,17 @@ impl Shared { } Ok(DocsEvent::NeedsImport { id }) => self.answer(id, Answer::NeedsImport), Ok(DocsEvent::NotLive { id, reason, .. }) => self.answer(id, Answer::NotLive(reason)), - Ok(DocsEvent::Ack { id, seq }) => self.route(id, Event::Acked(seq)), + Ok(DocsEvent::Ack { id, seq }) => match self.inner(id, seq) { + None => { + self.answered(id, seq); + self.route(id, Event::Acked(seq)); + } + // (An agent that answers an earlier fragment: not the app's; the change's ack is by its last.) + Some(Inner::Of(_)) => { + lock(&self.inner_seqs).remove(&(id, seq)); + } + Some(Inner::Answered) => {} + }, Ok(DocsEvent::Status { doc, online, waiting, deleted, stopped }) => { let ids = lock(&self.numbers).get(&doc).cloned().unwrap_or_default(); for id in ids { @@ -213,7 +296,14 @@ impl Shared { self.route(id, Event::File(frontiers.clone())); } } - Ok(DocsEvent::Error { id: Some(id), seq: Some(seq), code, message }) => self.route(id, Event::Refused { seq, code, message }), + Ok(DocsEvent::Error { id: Some(id), seq: Some(seq), code, message }) => match self.inner(id, seq) { + None => { + self.answered(id, seq); + self.route(id, Event::Refused { seq, code, message }); + } + Some(Inner::Of(last)) => self.refuse_change(id, last, code, message), + Some(Inner::Answered) => {} + }, Ok(DocsEvent::Error { id: Some(id), seq: None, code, message }) => self.answer(id, Answer::Refused(Error::Refused { code, message })), Ok(DocsEvent::Error { id: None, .. }) | Err(_) => {} }, @@ -226,20 +316,21 @@ impl Shared { if head.state.updates == 0 { self.answer(id, Answer::Live { head, snapshot: body.to_vec(), updates: Vec::new() }); } else { - lock(&self.filling).insert(id, (head, body.to_vec(), Vec::new())); + lock(&self.filling).insert(id, Filling { head, snapshot: body.to_vec(), updates: Vec::new() }); } } } UPDATE => { - // An update that is part of an open's state: kept with it until the last has come. + // An update that is part of an open's state: kept with it until the last has come. (The agent + // sends changes whole, never in fragments: it joins the server's before.) let mut filling = lock(&self.filling); - if let Some((head, _, updates)) = filling.get_mut(&id) { - updates.push(body.to_vec()); - if updates.len() as u64 >= head.state.updates - && let Some((head, snapshot, updates)) = filling.remove(&id) + if let Some(f) = filling.get_mut(&id) { + f.updates.push(body.to_vec()); + if f.updates.len() as u64 >= f.head.state.updates + && let Some(f) = filling.remove(&id) { drop(filling); - self.answer(id, Answer::Live { head, snapshot, updates }); + self.answer(id, Answer::Live { head: f.head, snapshot: f.snapshot, updates: f.updates }); } return; } @@ -292,6 +383,8 @@ fn read_loop(shared: Arc, mut socket: WebSocket) { pub struct Documents { shared: Arc, reader: Option>, + /// The local API version the agent spoke when this connected. + api: u32, } impl std::fmt::Debug for Documents { @@ -302,6 +395,12 @@ impl std::fmt::Debug for Documents { impl Documents { pub(crate) fn connect(client: &Client) -> Result { + // An agent from before changes in fragments would take each fragment as a change of its own (and its apps + // and machines would get them so): live documents need an update of Craft Cloud first. + let api = client.agent_api()?; + if api < LOCAL_API_FRAGMENTS { + return Err(Error::Refused { code: codes::AGENT_TOO_OLD.into(), message: AGENT_TOO_OLD.into() }); + } let stream = TcpStream::connect_timeout(&client.addr(), CONNECT_TIMEOUT).map_err(|e| Error::NotRunning(format!("can't reach it: {e}")))?; stream.set_nodelay(true).map_err(|e| Error::Io(e.to_string()))?; stream.set_read_timeout(Some(OPEN_TIMEOUT)).map_err(|e| Error::Io(e.to_string()))?; @@ -341,6 +440,7 @@ impl Documents { filling: Mutex::new(HashMap::new()), numbers: Mutex::new(HashMap::new()), wakers: Mutex::new(HashMap::new()), + inner_seqs: Mutex::new(HashMap::new()), closed: AtomicBool::new(false), next_id: AtomicU64::new(1), stream, @@ -350,7 +450,19 @@ impl Documents { .name("craft-cloud-docs".into()) .spawn(move || read_loop(reader_shared, reader_socket)) .map_err(|e| Error::Io(e.to_string()))?; - Ok(Self { shared, reader: Some(reader) }) + Ok(Self { shared, reader: Some(reader), api }) + } + + /// How many numbers of changes in fragments the connection keeps answers for (none once each is answered). + #[cfg(test)] + pub(crate) fn kept_numbers(&self) -> usize { + lock(&self.shared.inner_seqs).len() + } + + /// The local API version the agent spoke when this connected (at least [`LOCAL_API_FRAGMENTS`]): an app tells the + /// document's other machines with its presence, so a newer one can say to update an older one. + pub fn agent_api(&self) -> u32 { + self.api } /// Opens the file at `path` (relative to the synced folder, `/`-separated, or absolute inside it) for `app` (its @@ -505,12 +617,20 @@ impl Document { self.head.changed_outside } - /// Sends this app's change (a Loro update, at most 1 MiB); returns its number, which [`Event::Acked`] names once - /// it is on the agent's disk. + /// Sends this app's change (a Loro update, of up to a snapshot's limit: one over 1 MiB goes in fragments, which the + /// agent takes only together); returns its number, which [`Event::Acked`] (or [`Event::Refused`]) names once it is + /// on the agent's disk. An update over the limit is refused here (`too_large`), and nothing is sent. pub fn send(&self, update: &[u8]) -> Result { - let seq = self.seq.fetch_add(1, Ordering::Relaxed) + 1; - self.shared.send_binary(UPDATE, self.id, update)?; - Ok(seq) + let frames = split(update, FRAME_LIMIT.load(Ordering::Relaxed)) + .map_err(|e| Error::Refused { code: craft_cloud_api::api::codes::TOO_LARGE.into(), message: e.to_string() })?; + let count = frames.len() as u64; + let first = self.seq.fetch_add(count, Ordering::Relaxed) + 1; + let last = first + count - 1; + if count > 1 { + lock(&self.shared.inner_seqs).extend((first..last).map(|seq| ((self.id, seq), Inner::Of(last)))); + } + self.shared.send_frames(UPDATE, self.id, &frames)?; + Ok(last) } /// Tells the agent the app is about to write the document's file: the content's BLAKE3 `hash`, and the live @@ -564,6 +684,7 @@ impl Drop for Document { fn drop(&mut self) { lock(&self.shared.routes).remove(&self.id); lock(&self.shared.wakers).remove(&self.id); + self.shared.forget(self.id); let mut numbers = lock(&self.shared.numbers); let doc = self.head.doc; let remaining = numbers.get_mut(&doc).map(|ids| { @@ -595,6 +716,7 @@ mod tests { filling: Mutex::new(HashMap::new()), numbers: Mutex::new(HashMap::new()), wakers: Mutex::new(HashMap::new()), + inner_seqs: Mutex::new(HashMap::new()), closed: AtomicBool::new(false), next_id: AtomicU64::new(1), stream, @@ -629,6 +751,64 @@ mod tests { assert!(events.try_recv().is_err()); } + /// The agent answers a change in fragments once, by its last number (as the agent's local API does): that ack is + /// the app's, and nothing is kept for the change after it. A refusal of an earlier fragment is the change's, + /// under its last number, once: every later answer for any of its numbers is dropped, and a document dropped + /// leaves nothing kept. + #[test] + fn a_change_is_answered_by_its_last_number_and_nothing_is_kept_after() { + let (shared, _listener) = shared(); + let (events_tx, events) = mpsc::channel(); + lock(&shared.routes).insert(7, events_tx); + let ack = |seq: u64| Message::Text(serde_json::to_string(&DocsEvent::Ack { id: 7, seq }).unwrap().into()); + let refusal = |seq: u64| { + let e = DocsEvent::Error { id: Some(7), seq: Some(seq), code: "invalid".into(), message: "no".into() }; + Message::Text(serde_json::to_string(&e).unwrap().into()) + }; + lock(&shared.inner_seqs).extend([((7, 1), Inner::Of(3)), ((7, 2), Inner::Of(3))]); + shared.on_message(ack(3)); + assert_eq!(events.try_recv().unwrap(), Event::Acked(3)); + assert!(lock(&shared.inner_seqs).is_empty(), "nothing kept after the change's ack"); + // Fragments 4 to 6: the second refused, then each of the others too (a set refused fragment by fragment). + lock(&shared.inner_seqs).extend([((7, 4), Inner::Of(6)), ((7, 5), Inner::Of(6))]); + shared.on_message(refusal(5)); + assert_eq!(events.try_recv().unwrap(), Event::Refused { seq: 6, code: "invalid".into(), message: "no".into() }); + for seq in [4, 6, 5] { + shared.on_message(refusal(seq)); + } + shared.on_message(ack(6)); + assert!(events.try_recv().is_err(), "refused once, nothing for a number the app never had as an answer"); + shared.forget(7); + assert!(lock(&shared.inner_seqs).is_empty(), "a document dropped leaves nothing"); + } + + /// The agent's answers for fragments that aren't a change's last aren't the app's; a refusal of one is the + /// change's, under its last number, once (the answers still to come for the change are dropped). + #[test] + fn inner_answers_are_not_the_apps_but_a_refusal_refuses_the_change() { + let (shared, _listener) = shared(); + let (events_tx, events) = mpsc::channel(); + lock(&shared.routes).insert(7, events_tx); + let ack = |seq: u64| Message::Text(serde_json::to_string(&DocsEvent::Ack { id: 7, seq }).unwrap().into()); + let refusal = |seq: u64| { + let e = DocsEvent::Error { id: Some(7), seq: Some(seq), code: "invalid".into(), message: "no".into() }; + Message::Text(serde_json::to_string(&e).unwrap().into()) + }; + lock(&shared.inner_seqs).extend([((7, 1), Inner::Of(3)), ((7, 2), Inner::Of(3))]); + for seq in 1..=3 { + shared.on_message(ack(seq)); + } + assert_eq!(events.try_recv().unwrap(), Event::Acked(3)); + assert!(events.try_recv().is_err()); + // The next change, of fragments 4 to 6: its second is refused. + lock(&shared.inner_seqs).extend([((7, 4), Inner::Of(6)), ((7, 5), Inner::Of(6))]); + shared.on_message(ack(4)); + shared.on_message(refusal(5)); + assert_eq!(events.try_recv().unwrap(), Event::Refused { seq: 6, code: "invalid".into(), message: "no".into() }); + shared.on_message(refusal(6)); + assert!(events.try_recv().is_err(), "refused once"); + } + /// `opened` then the state: one answer; the events after it go to the document, its status by document ID. #[test] fn an_open_is_answered_once_its_state_came() { diff --git a/crates/egui/src/lib.rs b/crates/egui/src/lib.rs index 3871173..aa38913 100644 --- a/crates/egui/src/lib.rs +++ b/crates/egui/src/lib.rs @@ -260,6 +260,7 @@ mod tests { } let status = Status { api: 1, + api_max: None, signed_in: true, account: Some("ana".into()), server: Some("https://cloud.example.com/".into()), diff --git a/crates/kit/src/lib.rs b/crates/kit/src/lib.rs index 5693589..0d12afc 100644 --- a/crates/kit/src/lib.rs +++ b/crates/kit/src/lib.rs @@ -34,20 +34,19 @@ pub use mapping::{Attrs, Mappings, OpaqueSource, RemotePatch, Span, TableSource, pub use opaque::Opaque; pub use props::{Props, PropsRead}; pub use session::{ - Agent, Badge, CATCH_UP_EVERY, LiveDocument, MAX_BACKOFF, MAX_QUEUE_BYTES, MAX_UNDO_TEXT_BYTES, NotLive, Open, PRESENCE_EVERY, PRESENCE_TTL, Peer, - Polled, REPLACE_WITH_FILE, ReadOnly, SavePoint, SaveRecord, Session, Status, UndoKind, + Agent, Badge, CATCH_UP_EVERY, LiveDocument, MAX_BACKOFF, MAX_CHANGE_BYTES, MAX_QUEUE_BYTES, MAX_UNDO_TEXT_BYTES, NotLive, Open, PRESENCE_EVERY, + PRESENCE_TTL, Peer, Polled, REPLACE_WITH_FILE, ReadOnly, SavePoint, SaveRecord, Session, Status, UndoKind, }; pub use table::{TableTouch, TableView}; pub use text::RichText; pub use timeline::{ItemRead, TimelineTouch, TimelineView}; pub use tree::{TreeTouch, TreeView}; pub use value::{AgentBlobs, BlobStore, ChunkRef, ElemId, MAX_VALUE_BYTES, MemoryBlobs, OpaqueCodec, Value}; -pub use writer::{OP_OVERHEAD, SPLIT_BYTES}; /// For an app's adapter tests: the fake agent of `craft-cloud-app` (feature `fake`). #[cfg(feature = "fake")] pub mod testing { - pub use craft_cloud_app::fake::{Fake, Hub}; + pub use craft_cloud_app::fake::{Fake, FragmentFaults, Hub, SLEEP_TICKS}; } /// For the kit's own tests and simulations: the mappings without a session. @@ -74,17 +73,20 @@ pub mod raw { } impl Replica { - /// Writes `old` → `new`; the updates (each commit's export) and how many commits. + /// Writes `old` → `new`; the update (none if nothing changed, else one, whole) and how many commits (0 or 1). pub fn write(&mut self, old: Option<&D::Doc>, new: &D::Doc) -> Result<(Vec>, usize), KitError> { let mut w = Writer::new(&self.doc, LOCAL_ORIGIN); write_diff::(&mut w, old, new, &mut self.ctx)?; w.finish() } - /// Imports updates and indexes the trees again. + /// Imports updates (a bundle's slices one after the other) and indexes the trees again. pub fn import(&mut self, updates: &[Vec], trees: &[&str]) -> Result<(), KitError> { for u in updates { - self.doc.import(u)?; + let parts = craft_cloud_api::fragments::parts(u).ok_or_else(|| KitError::Invalid("a bundle cut short".into()))?; + for part in parts { + self.doc.import(part)?; + } } self.reindex(trees); Ok(()) diff --git a/crates/kit/src/mapping.rs b/crates/kit/src/mapping.rs index e9783d7..f6c6f89 100644 --- a/crates/kit/src/mapping.rs +++ b/crates/kit/src/mapping.rs @@ -80,6 +80,20 @@ pub trait TableSource { fn changed_cells(&self) -> Option> { None } + /// Whether the cells are property maps (values and stories of their own, written by + /// [`TableSource::write_cell`]) rather than one value each. Then every cell of every row and column is one, and + /// [`TableSource::cell`] and [`TableSource::cells`] aren't used. + fn cell_maps(&self) -> bool { + false + } + /// For map cells: the same value in two models when the cell is the same; `None`: compare. + fn cell_key(&self, _row: ElemId, _column: ElemId) -> Option { + None + } + /// For map cells: writes the cell's properties and stories ([`Props::set`], [`Props::text`]). + fn write_cell(&self, _row: ElemId, _column: ElemId, _props: &mut Props<'_, '_>) -> Result<(), KitError> { + Ok(()) + } } /// An item's place on its track, in the app's time unit. @@ -178,6 +192,15 @@ impl TableSource for &T { fn changed_cells(&self) -> Option> { (**self).changed_cells() } + fn cell_maps(&self) -> bool { + (**self).cell_maps() + } + fn cell_key(&self, row: ElemId, column: ElemId) -> Option { + (**self).cell_key(row, column) + } + fn write_cell(&self, row: ElemId, column: ElemId, props: &mut Props<'_, '_>) -> Result<(), KitError> { + (**self).write_cell(row, column, props) + } } impl TimelineSource for &T { @@ -312,6 +335,10 @@ pub struct Touched { /// Loro's text deltas, by text, in the order they came (to patch the model's text rather than read it whole). pub(crate) text_deltas: BTreeMap>>, pub(crate) tables: BTreeMap, + /// Loro's text deltas of stories nested in tree nodes, by tree, then by node and story key. + pub(crate) node_text_deltas: BTreeMap, + /// Loro's text deltas of stories in map cells, by table, then by row, column and story key. + pub(crate) cell_text_deltas: BTreeMap, pub(crate) timelines: BTreeMap, pub(crate) opaques: BTreeSet, /// The schema (or the save marks) changed. @@ -354,7 +381,7 @@ impl RemotePatch<'_> { pub fn tree(&self, key: &str) -> Option> { let touch = if self.touched.all { TreeTouch::whole() } else { self.touched.trees.get(key)?.clone() }; let index = self.trees.get(key)?; - Some(TreeView::new(self.doc, key, index, touch)) + Some(TreeView::new(self.doc, key, index, touch, self.touched.node_text_deltas.get(key))) } /// The rich text `key` as it is now, if the change touched it. @@ -377,7 +404,7 @@ impl RemotePatch<'_> { /// The table `key`, if the change touched it. pub fn table(&self, key: &str) -> Option> { let touch = if self.touched.all { TableTouch::whole() } else { self.touched.tables.get(key)?.clone() }; - Some(TableView::new(self.doc, key, touch)) + Some(TableView::new(self.doc, key, touch, self.touched.cell_text_deltas.get(key))) } /// The timeline `key`, if the change touched it. diff --git a/crates/kit/src/props.rs b/crates/kit/src/props.rs index 0463bb5..7183cf6 100644 --- a/crates/kit/src/props.rs +++ b/crates/kit/src/props.rs @@ -1,8 +1,15 @@ //! A model element's properties as a Loro map: what the app's codec writes ([`Props`], only what changed) and reads //! back ([`PropsRead`]). Keys starting with `$` are the kit's own. +//! +//! Besides values, an element's map can hold a rich-text story ([`Props::text`]) and a table ([`Props::table`]), +//! each under a key the codec names: a mergeable child container of the map, so two machines making it at once make +//! one, and their edits in it merge (two people typing in one shape's text merge character by character). -use loro::{LoroMap, ValueOrContainer}; +use loro::{Container, LoroMap, StyleConfigMap, ValueOrContainer}; +use crate::mapping::{TableSource, TextSource}; +use crate::table::TableView; +use crate::text::{RichText, Story}; use crate::writer::Writer; use crate::{KitError, Value}; @@ -20,11 +27,12 @@ pub struct Props<'a, 'd> { pub(crate) map: LoroMap, pub(crate) w: &'a mut Writer<'d>, pub(crate) written: Vec, + pub(crate) styles: &'a mut StyleConfigMap, } impl<'a, 'd> Props<'a, 'd> { - pub(crate) fn new(map: LoroMap, w: &'a mut Writer<'d>) -> Self { - Self { map, w, written: Vec::new() } + pub(crate) fn new(map: LoroMap, w: &'a mut Writer<'d>, styles: &'a mut StyleConfigMap) -> Self { + Self { map, w, written: Vec::new(), styles } } /// Sets the property `key`. @@ -36,11 +44,48 @@ impl<'a, 'd> Props<'a, 'd> { Ok(()) } - /// Deletes the properties not set (the kit's own keys stay). + /// Sets the rich-text story `key` (a shape's text, a cell's): only what differs from what it holds is written, as + /// for a root story (character attributes as marks, paragraph attributes on the breaks). + pub fn text(&mut self, key: &str, source: &dyn TextSource) -> Result<(), KitError> { + check_key(key)?; + let story = match Story::existing(&self.map, key) { + Some(story) => story, + None => { + self.w.spend(crate::writer::OP_OVERHEAD)?; + Story::nested(&self.map, key)? + } + }; + crate::text::write_story(self.w, &story, None, source, self.styles)?; + self.written.push(key.to_owned()); + Ok(()) + } + + /// Sets the table `key` (a table shape's rows, columns and cells): rows and columns are movable lists of IDs, so + /// rows inserted on two machines at once are both kept; cells are as [`TableSource`] says (values, or property + /// maps with stories of their own). + pub fn table(&mut self, key: &str, source: &dyn TableSource) -> Result<(), KitError> { + check_key(key)?; + let map = match self.map.get(key) { + Some(ValueOrContainer::Container(Container::Map(map))) => map, + _ => { + self.w.spend(crate::writer::OP_OVERHEAD)?; + self.map.ensure_mergeable_map(key)? + } + }; + crate::table::write_nested(self.w, &map, source, self.styles)?; + self.written.push(key.to_owned()); + Ok(()) + } + + /// Deletes the properties not set (the kit's own keys stay, but a story's `$last.` goes with it). pub(crate) fn finish(self) -> Result<(), KitError> { let mut stale = Vec::new(); self.map.for_each(|key, _| { - if !key.starts_with('$') && !self.written.iter().any(|w| w == key) { + let gone = match key.strip_prefix("$last.") { + Some(story) => !self.written.iter().any(|w| w == story), + None => !key.starts_with('$') && !self.written.iter().any(|w| w == key), + }; + if gone { stale.push(key.to_owned()); } }); @@ -90,7 +135,32 @@ impl PropsRead { self.get(key).and_then(|v| v.as_chunk()) } - /// Every app property, sorted by key. + /// The story `key` as stored (`None`: there is none). + pub fn text(&self, key: &str) -> Option { + Story::existing(&self.map, key).map(|s| crate::text::read_story(&s)) + } + + /// The story `key`, patched from `old` (the app's text before the change) with the `deltas` a remote change + /// brought, else read whole. + pub(crate) fn text_patched(&self, key: &str, old: Option<&dyn TextSource>, deltas: Option<&Vec>>) -> Option { + let story = Story::existing(&self.map, key)?; + if let (Some(old), Some(deltas)) = (old, deltas) + && let Some(patched) = crate::text::patch_story(&story, old, deltas) + { + return Some(patched); + } + Some(crate::text::read_story(&story)) + } + + /// The table `key` as stored (`None`: there is none). Read it whole: a nested table's touch is always whole. + pub fn table(&self, key: &str) -> Option> { + match self.map.get(key)? { + ValueOrContainer::Container(Container::Map(map)) => Some(TableView::nested(&map)), + _ => None, + } + } + + /// Every app property that holds a value, sorted by key (stories and tables aren't values). pub fn all(&self) -> Vec<(String, Value)> { let mut out = Vec::new(); self.map.for_each(|key, value| { diff --git a/crates/kit/src/session.rs b/crates/kit/src/session.rs index 3a55e98..396d50f 100644 --- a/crates/kit/src/session.rs +++ b/crates/kit/src/session.rs @@ -20,6 +20,8 @@ use std::sync::mpsc::{Receiver, TryRecvError}; use std::sync::{Arc, Mutex, PoisonError}; use std::time::{Duration, Instant}; +use craft_cloud_api::fragments::parts; +use craft_cloud_api::local::LOCAL_API_FRAGMENTS; use craft_cloud_app::Client; use craft_cloud_app::live::{Document, Documents, Event, NotLiveReason, OpenOptions, Opened, Waker}; use loro::event::{Diff, DiffEvent}; @@ -41,6 +43,15 @@ const FIRST_BACKOFF: Duration = Duration::from_millis(200); /// Presence is sent again this often while set, and a peer's is dropped after this long without one. pub const PRESENCE_EVERY: Duration = Duration::from_secs(3); pub const PRESENCE_TTL: Duration = Duration::from_secs(10); +/// A presence frame names the sender's agent version after its peer: this mark, then the version (u32, big-endian), +/// then the app's bytes. A kit from before versions sends the app's bytes right after the peer. +const PRESENCE_MARK: [u8; 4] = *b"ccp2"; +/// The peer, the mark and the version before the app's bytes in a presence frame. +const PRESENCE_HEADER: usize = 16; +/// The largest change the kit sends (one update, in fragments past 1 MiB): 8 MiB, half a document's log cap on the +/// server, so the change after it still fits once the log is compacted. A change over it can't be live: the session +/// stops (the model stays editable; save it to the file). +pub const MAX_CHANGE_BYTES: usize = craft_cloud_api::fragments::MAX_WHOLE_BYTES; /// Undo steps kept. const UNDO_STEPS: usize = 500; /// The most text (inserted and deleted, in bytes) one step may change and still be undone while live. Loro's undo of a @@ -105,6 +116,9 @@ impl Agent { pub enum NotLive { /// Craft Cloud isn't running here (or refused): the message says why. NoAgent(String), + /// Craft Cloud runs here, but is older than this app's live documents: show "update Craft Cloud on this + /// computer to edit live" ([`craft_cloud_app::live::AGENT_TOO_OLD`]). + AgentTooOld, /// The agent's reason (offline before the first live open, not synced yet, …). Reason(NotLiveReason), } @@ -150,6 +164,13 @@ pub struct Status { /// The file on disk wasn't saved by a Craft app from this document (edited outside): the app may bring it in as a /// local change, or keep the live content. pub changed_outside: bool, + /// Another machine with the document open runs an older Craft Cloud ([`Peer::older`]): show " runs an + /// older Craft Cloud; update it" (its name is in its presence, as the app sends it). + pub older_peer: bool, + /// Changes from other machines that didn't read here (a bundle cut short, or one Loro refused), in this session: + /// never applied, each also said in `error`. The model may lack them; opening the document again brings what the + /// agent has. + pub unread: u64, /// The last error, for the app to show. pub error: Option, } @@ -267,6 +288,30 @@ struct Kept { pub struct Peer { pub bytes: Vec, pub seen: Instant, + /// The local API version of the agent it runs on, as its presence says; `None` from an app on a Craft Cloud from + /// before versions in presence (older than this one). + pub agent_api: Option, +} + +impl Peer { + /// Whether it runs on a Craft Cloud too old for changes in fragments: its machine should update it (it may lack + /// this machine's big changes). Show " runs an older Craft Cloud; update it". + pub fn older(&self) -> bool { + self.agent_api.is_none_or(|api| api < LOCAL_API_FRAGMENTS) + } +} + +/// A presence frame's peer, its sender's agent version (none from a kit before versions) and the app's bytes. +fn read_presence(frame: &[u8]) -> Option<(u64, Option, &[u8])> { + let (peer, rest) = frame.split_first_chunk::<8>()?; + let peer = u64::from_be_bytes(*peer); + match rest.split_first_chunk::<4>() { + Some((mark, after)) if *mark == PRESENCE_MARK => { + let (api, bytes) = after.split_first_chunk::<4>()?; + Some((peer, Some(u32::from_be_bytes(*api)), bytes)) + } + _ => Some((peer, None, rest)), + } } /// One change waiting for the agent. @@ -285,8 +330,13 @@ struct Raw { nodes: Vec, keys: Vec, tree: bool, - /// A text's delta (the text root's own). + /// A text's delta (the text root's own, or a nested story's). text: Option>, + /// The deepest tree node on the path, and the keys after it (a story nested in a node: `["text"]`). + node: Option, + under_node: Vec, + /// The keys after the root's own (a story in a root table's map cell: `[":", "text"]`). + under_root: Vec, } #[derive(Default)] @@ -317,7 +367,7 @@ pub(crate) struct Ctx { /// the agent hears it closed before the connection ends. struct Link { doc: Document, - _docs: Documents, + docs: Documents, } /// Why a reopen didn't attach. @@ -428,7 +478,7 @@ pub(crate) fn write_diff(w: &mut Writer<'_>, old: Option<&D::Do _ => None, }; let index = ctx.trees.entry((*key).to_owned()).or_default(); - crate::tree::write(w, key, o, n.as_ref(), index)?; + crate::tree::write(w, key, o, n.as_ref(), index, &mut ctx.styles)?; } Part::Text(n) => { let o = match old_part { @@ -442,14 +492,14 @@ pub(crate) fn write_diff(w: &mut Writer<'_>, old: Option<&D::Do Some(Part::Table(o)) => Some(o.as_ref()), _ => None, }; - crate::table::write(w, key, o, n.as_ref())?; + crate::table::write(w, key, o, n.as_ref(), &mut ctx.styles)?; } Part::Timeline(n) => { let o = match old_part { Some(Part::Timeline(o)) => Some(o.as_ref()), _ => None, }; - crate::timeline::write(w, key, o, n.as_ref())?; + crate::timeline::write(w, key, o, n.as_ref(), &mut ctx.styles)?; } Part::Opaque(n) => { let o = match old_part { @@ -501,6 +551,9 @@ impl Session { }; let docs = match client.documents() { Ok(d) => d, + Err(craft_cloud_app::Error::Refused { code, .. }) if code == craft_cloud_api::local::codes::AGENT_TOO_OLD => { + return Ok(Open::NotLive(NotLive::AgentTooOld)); + } Err(e) => return not_running(e), }; let file_hash = file_bytes.map(|b| *blake3::hash(b).as_bytes()); @@ -529,7 +582,7 @@ impl Session { // bytes the document was imported from are no edit either. session.status.changed_outside = snapshot.is_none() && document.changed_outside() == Some(true) && file_hash.is_some_and(|h| import_hash(&session.doc) != Some(hex(&h))); - session.attach(Link { _docs: docs, doc: document }); + session.attach(Link { docs, doc: document }); Ok(Open::Live(Box::new(session))) } @@ -559,9 +612,17 @@ impl Session { doc.import(state.0)?; // The updates that are part of the state. let mut unread = None; + let mut unread_count = 0; for update in state.1 { - if let Err(e) = doc.import(update) { + // (A bundle that doesn't read is said and counted, never taken as an empty change.) + let Some(slices) = parts(update) else { + unread = Some("a change from another machine doesn't read (a bundle cut short)".to_owned()); + unread_count += 1; + continue; + }; + if let Err(e) = import_slices(&doc, &slices) { unread = Some(format!("a change from another machine doesn't read: {e}")); + unread_count += 1; } } let collector = Arc::new(Mutex::new(Collector::default())); @@ -609,6 +670,7 @@ impl Session { session.reindex_trees(); session.check_schema(); session.status.error = unread; + session.status.unread = unread_count; if !ours { session.rebuild(Touched::everything())?; } @@ -644,7 +706,7 @@ impl Session { // random and never handed out, so whatever the agent has under it is this session's.) let ours = self.doc.peer_id(); let held = std::iter::once(link.doc.state()) - .chain(link.doc.state_updates().iter().map(Vec::as_slice)) + .chain(link.doc.state_updates().iter().flat_map(|u| parts(u).unwrap_or_default())) .filter_map(|bytes| loro::LoroDoc::decode_import_blob_meta(bytes, false).ok()?.partial_end_vv.get(&ours).copied()) .max() .unwrap_or(0); @@ -761,6 +823,57 @@ impl Session { } } + /// A change the app makes by itself rather than the user (a media upload that finished and now has its chunk + /// reference, bytes read back into the model): written and sent like [`Session::local_change`], with the same + /// rules for `old`, but never an undo step, so undo never takes it back. + pub fn background_change(&mut self, old: &Arc, new: &Arc) -> Result<(), KitError> { + if let Some(ro) = &self.status.read_only { + return Err(KitError::ReadOnly { app: ro.app.clone(), version: ro.version }); + } + if self.model_stale { + return Err(KitError::ModelStale); + } + if !Arc::ptr_eq(old, &self.model) { + return Err(KitError::Stale); + } + if Arc::ptr_eq(old, new) { + return Ok(()); + } + if self.outbox_bytes >= MAX_QUEUE_BYTES { + self.status.stalled = true; + return Err(KitError::Stalled(self.outbox_bytes)); + } + let before = self.doc.oplog_vv(); + let mut w = Writer::new(&self.doc, KIT_ORIGIN); + let wrote = write_diff::(&mut w, Some(old.as_ref()), new.as_ref(), &mut self.ctx); + let updates = match w.finish() { + Ok((updates, _)) => updates, + Err(_) => { + let mut rest = Vec::new(); + let _ = export_since(&self.doc, &before, &mut rest); + rest + } + }; + if self.doc.oplog_vv() != before { + self.changed_since_save = true; + } + self.queue(updates); + match wrote { + Ok(()) => { + self.model = new.clone(); + Ok(()) + } + Err(e) => { + if self.doc.oplog_vv() != before { + self.reindex_trees(); + let _ = self.rebuild(Touched::everything()); + } + self.status.error = Some(e.to_string()); + Err(e) + } + } + } + /// Makes every change `f` makes one undo step (typing a word, a drag's previews). pub fn group(&mut self, f: impl FnOnce(&mut Self) -> R) -> R { let outer = self.begin_group(); @@ -962,11 +1075,12 @@ impl Session { match event { Event::Update(bytes) => self.incoming.push(bytes), Event::Presence(bytes) => { - if let Some((peer, payload)) = bytes.split_first_chunk::<8>() { - let peer = u64::from_be_bytes(*peer); - if peer != self.doc.peer_id() { - self.peers.insert(peer, Peer { bytes: payload.to_vec(), seen: Instant::now() }); - } + if let Some((peer, agent_api, payload)) = read_presence(&bytes) + && peer != self.doc.peer_id() + { + let peer_now = Peer { bytes: payload.to_vec(), seen: Instant::now(), agent_api }; + self.status.older_peer |= peer_now.older(); + self.peers.insert(peer, peer_now); } } Event::Acked(seq) => { @@ -1045,7 +1159,7 @@ impl Session { self.incoming.push(doc.state().to_vec()); self.incoming.extend(doc.state_updates().iter().cloned()); self.status.error = None; - self.attach(Link { _docs: docs, doc }); + self.attach(Link { docs, doc }); if let Some(waker) = &self.waker { waker(); } @@ -1119,12 +1233,23 @@ impl Session { // means only an update Loro refuses is lost (the agent checked them all already). lock(&self.collector).on = true; for bytes in &batch { - if let Err(e) = self.doc.import(bytes) { - self.status.error = Some(format!("a change from another machine doesn't read: {e}")); + // (A bundle's slices one after the other: the model sees them only together, below.) + match parts(bytes) { + Some(updates) => { + if let Err(e) = import_slices(&self.doc, &updates) { + self.status.error = Some(format!("a change from another machine doesn't read: {e}")); + self.status.unread += 1; + } + } + None => { + self.status.error = Some("a change from another machine doesn't read (a bundle cut short)".into()); + self.status.unread += 1; + } } } let raw = std::mem::take(&mut lock(&self.collector).raw); lock(&self.collector).on = false; + // (Each update is a whole change: one that came in fragments was joined before it got here.) let touched = self.classify(raw); let was_read_only = self.status.read_only.clone(); if touched.kit { @@ -1141,6 +1266,7 @@ impl Session { fn classify(&mut self, raw: Vec) -> Touched { let mut touched = Touched::default(); let mut reindex = Vec::new(); + let mut nested: Vec<(&'static str, TreeID, String, Option>)> = Vec::new(); for r in raw { if r.root == ROOT { touched.kit = true; @@ -1164,7 +1290,20 @@ impl Session { t.structure = true; reindex.push(key); } - t.nodes.extend(r.nodes.iter().filter_map(|n| self.ctx.trees.get(key).and_then(|i| i.elem_of.get(n)).copied())); + // A node the index doesn't name (deleted, or a second node carrying an element's ID) still names its + // element in its meta: the element is touched. + let tree = self.doc.get_tree(key); + t.nodes.extend( + r.nodes.iter().filter_map(|n| { + self.ctx.trees.get(key).and_then(|i| i.elem_of.get(n)).copied().or_else(|| crate::tree::node_id(&tree, *n)) + }), + ); + // Inside a node's child container (its story, its table): named once the index is current. + if let Some(tid) = r.node + && !r.under_node.is_empty() + { + nested.push((key, tid, r.under_node.join("/"), r.text.clone())); + } } Kind::Text => { touched.texts.insert(key.to_owned()); @@ -1179,7 +1318,25 @@ impl Session { match suffix { "rows" => t.rows = true, "cols" => t.columns = true, - _ => t.cells.extend(r.keys.iter().filter_map(|k| parse_cell_key(k))), + _ => { + t.cells.extend(r.keys.iter().filter_map(|k| parse_cell_key(k))); + // A map cell's story: the cell, then the story's key. + if let (Some(cell), Some(story)) = (r.under_root.first().and_then(|k| parse_cell_key(k)), r.under_root.get(1..)) + && !story.is_empty() + { + let story = story.join("/"); + t.stories.insert((cell.0, cell.1, story.clone())); + if let Some(delta) = r.text.clone() { + touched + .cell_text_deltas + .entry(key.to_owned()) + .or_default() + .entry((cell.0, cell.1, story)) + .or_default() + .push(delta); + } + } + } } } Kind::Timeline => { @@ -1198,13 +1355,27 @@ impl Session { let tree = self.doc.get_tree(key); let index = TreeIndex::build(&tree); if let Some(t) = touched.trees.get_mut(key) { - // Created nodes: their IDs are known only now. - for (id, _) in index.tid_of.iter().filter(|(id, _)| !self.ctx.trees.get(key).is_some_and(|old| old.tid_of.contains_key(id))) { - t.nodes.insert(*id); + // Created nodes: their IDs are known only now. And an element whose node is another one now (two nodes + // came to carry its ID, and the other one wins): its content may be another, so it is touched. + let old = self.ctx.trees.get(key); + for (id, tid) in &index.tid_of { + if old.and_then(|o| o.tid_of.get(id)) != Some(tid) { + t.nodes.insert(*id); + } } } self.ctx.trees.insert(key.to_owned(), index); } + for (key, tid, story, delta) in nested { + let Some(id) = self.ctx.trees.get(key).and_then(|i| i.elem_of.get(&tid)).copied() else { continue }; + if let Some(t) = touched.trees.get_mut(key) { + t.nodes.insert(id); + t.stories.insert((id, story.clone())); + } + if let Some(delta) = delta { + touched.node_text_deltas.entry(key.to_owned()).or_default().entry((id, story)).or_default().push(delta); + } + } touched } @@ -1262,6 +1433,13 @@ impl Session { } let end = self.doc.oplog_vv().get(&self.doc.peer_id()).copied().unwrap_or(0); for bytes in updates { + if bytes.len() > MAX_CHANGE_BYTES && self.refused.is_none() { + // It can never go (nor anything after it, which needs it): the session stops, the model stays. + let why = + format!("a change of {} bytes is over what Craft Cloud takes ({MAX_CHANGE_BYTES}); save the document to the file", bytes.len()); + self.refused = Some(why.clone()); + self.status.error = Some(why); + } self.outbox_bytes = self.outbox_bytes.saturating_add(bytes.len()); self.outbox.push_back(Out { bytes, end, seq: None }); } @@ -1311,12 +1489,14 @@ impl Session { } /// Shows this app's cursor or selection to the document's other machines and apps (the bytes are the app's, at - /// most 4 KiB less 8); `None` stops showing it. Sent now and again every [`PRESENCE_EVERY`]. + /// most 4 KiB less 16); `None` stops showing it. Sent now and again every [`PRESENCE_EVERY`], with this machine's + /// agent version ([`Peer::older`] on the others). pub fn set_presence(&mut self, bytes: Option>) -> Result<(), KitError> { + let max = craft_cloud_api::limits::MAX_PRESENCE_BYTES - PRESENCE_HEADER; if let Some(b) = &bytes - && b.len() + 8 > craft_cloud_api::limits::MAX_PRESENCE_BYTES + && b.len() > max { - return Err(KitError::TooLarge { what: "a presence", len: b.len(), max: craft_cloud_api::limits::MAX_PRESENCE_BYTES - 8 }); + return Err(KitError::TooLarge { what: "a presence", len: b.len(), max }); } self.presence = bytes; self.presence_sent = None; @@ -1327,11 +1507,14 @@ impl Session { fn keep_presence(&mut self) { let now = Instant::now(); self.peers.retain(|_, p| now.duration_since(p.seen) < PRESENCE_TTL); + self.status.older_peer = self.peers.values().any(Peer::older); let (Some(link), Some(bytes)) = (&self.link, &self.presence) else { return }; if self.presence_sent.is_some_and(|at| now.duration_since(at) < PRESENCE_EVERY) { return; } let mut frame = self.doc.peer_id().to_be_bytes().to_vec(); + frame.extend_from_slice(&PRESENCE_MARK); + frame.extend_from_slice(&link.docs.agent_api().to_be_bytes()); frame.extend_from_slice(bytes); if link.doc.presence(&frame).is_ok() { self.presence_sent = Some(now); @@ -1435,12 +1618,66 @@ impl Session { Ok(self.doc.export(loro::ExportMode::Snapshot)?) } + /// For the kit's tests: an update as if it came from the agent (applied at the next poll). + #[doc(hidden)] + pub fn receive_for_tests(&mut self, update: Vec) { + self.incoming.push(update); + } + /// This session's Loro peer. pub fn peer(&self) -> u64 { self.doc.peer_id() } } +/// A slice's import this slow means Loro took its slow way: see [`import_slices`]. (The agent's copy takes slices +/// the same way, with the same threshold: `LiveCopy::import_any_size` in the agent; keep the two alike.) +const SLOW_SLICE: Duration = Duration::from_millis(50); + +/// Imports one change's slices (a bundle's, or the one update) one after the other. Each slice normally goes in in +/// milliseconds; but with this machine's own edits in the same text concurrent with the change, Loro (in one order +/// of the two peers) works the text out again from where the edits parted for each slice, so each costs more than the +/// last and a 1.2 MB paste took minutes. Once a slice is that slow, the rest goes in as one update (joined on a +/// detached fork, which only takes their operations), which Loro works out once (seconds for a megabyte). Undo is +/// kept either way. Every slice is read first: one that doesn't read refuses the change whole, nothing of it in. +/// Past that, the first error is returned; the slices after it are still imported. +fn import_slices(doc: &LoroDoc, slices: &[&[u8]]) -> Result<(), loro::LoroError> { + for slice in slices { + LoroDoc::decode_import_blob_meta(slice, true)?; + } + let mut first_error = None; + let mut i = 0; + while let Some(slice) = slices.get(i) { + let started = Instant::now(); + if let Err(e) = doc.import(slice) { + first_error.get_or_insert(e); + } + i += 1; + let rest = slices.get(i..).unwrap_or_default(); + if rest.len() >= 2 + && started.elapsed() > SLOW_SLICE + && let Some(one) = joined(doc, rest) + { + if let Err(e) = doc.import(&one) { + first_error.get_or_insert(e); + } + break; + } + } + first_error.map_or(Ok(()), Err) +} + +/// `updates` (which follow on from what `doc` holds) as one update, if they all read. +fn joined(doc: &LoroDoc, updates: &[&[u8]]) -> Option> { + let fork = doc.fork(); + fork.detach(); + let before = fork.oplog_vv(); + for u in updates { + fork.import(u).ok()?; + } + fork.export(loro::ExportMode::updates(&before)).ok() +} + /// Loro's events, noted while the collector is on (an import, an undo). fn collect(collector: &Mutex, event: DiffEvent<'_>) { let mut c = lock(collector); @@ -1451,10 +1688,22 @@ fn collect(collector: &Mutex, event: DiffEvent<'_>) { let first = diff.path.first().map(|(id, _)| id).unwrap_or(diff.target); let Some(root) = root_name(first) else { continue }; let mut raw = Raw { root, ..Raw::default() }; - for (_, index) in diff.path { + for (i, (_, index)) in diff.path.iter().enumerate() { match index { - Index::Node(t) => raw.nodes.push(*t), - Index::Key(k) => raw.keys.push(k.to_string()), + Index::Node(t) => { + raw.nodes.push(*t); + raw.node = Some(*t); + raw.under_node.clear(); + } + Index::Key(k) => { + raw.keys.push(k.to_string()); + if i > 0 { + raw.under_root.push(k.to_string()); + if raw.node.is_some() { + raw.under_node.push(k.to_string()); + } + } + } Index::Seq(_) => {} } } @@ -1470,3 +1719,65 @@ fn collect(collector: &Mutex, event: DiffEvent<'_>) { c.raw.push(raw); } } + +#[cfg(test)] +mod tests { + #![allow(clippy::unwrap_used, clippy::expect_used)] + + use super::*; + + /// A 600 KB paste from another peer into a text this machine edited at the same moment, this machine's peer below + /// the sender's (the order Loro is slow in: each slice costs more than the last, a minute for these): a few slices + /// go in one by one, then the rest together, in seconds. The text is the paste and the edit, and this machine's + /// undo step stays (it takes back only the edit). + #[test] + fn slices_that_turn_slow_go_in_together() { + let base = LoroDoc::new(); + base.set_peer_id(1).unwrap(); + base.get_text("t").insert(0, "abc").unwrap(); + base.commit(); + let snapshot = base.export(loro::ExportMode::Snapshot).unwrap(); + let sender = LoroDoc::new(); + sender.set_peer_id(9).unwrap(); + sender.import(&snapshot).unwrap(); + let paste = "p".repeat(600 * 1024); + sender.get_text("t").insert(3, &paste).unwrap(); + sender.commit(); + let end = sender.oplog_vv().get(&9).copied().unwrap(); + let mut slices = Vec::new(); + let mut at = 0; + while at < end { + let to = (at + craft_cloud_api::fragments::SLICE_OPS).min(end); + slices.push(sender.export(loro::ExportMode::updates_in_range(vec![loro::IdSpan::new(9, at, to)])).unwrap()); + at = to; + } + let here = new_doc(); + here.set_peer_id(2).unwrap(); + here.import(&snapshot).unwrap(); + let mut undo = new_undo(&here); + here.get_text("t").insert(0, "hello ").unwrap(); + here.commit(); + let started = Instant::now(); + import_slices(&here, &slices.iter().map(Vec::as_slice).collect::>()).unwrap(); + assert!(started.elapsed() < Duration::from_secs(15), "{:?}", started.elapsed()); + assert_eq!(here.get_text("t").to_string(), format!("hello abc{paste}")); + assert!(undo.undo().unwrap()); + assert_eq!(here.get_text("t").to_string(), format!("abc{paste}")); + } + + /// A change with a slice that doesn't read is refused whole: nothing of its other slices goes in. + #[test] + fn a_change_with_a_slice_that_does_not_read_goes_in_not_at_all() { + let sender = LoroDoc::new(); + sender.set_peer_id(9).unwrap(); + sender.get_text("t").insert(0, "abcdef").unwrap(); + sender.commit(); + let one = sender.export(loro::ExportMode::updates_in_range(vec![loro::IdSpan::new(9, 0, 3)])).unwrap(); + let two = sender.export(loro::ExportMode::updates_in_range(vec![loro::IdSpan::new(9, 3, 6)])).unwrap(); + let here = new_doc(); + assert!(import_slices(&here, &[one.as_slice(), b"not an update", two.as_slice()]).is_err()); + assert_eq!(here.get_text("t").to_string(), "", "nothing of it in"); + import_slices(&here, &[one.as_slice(), two.as_slice()]).unwrap(); + assert_eq!(here.get_text("t").to_string(), "abcdef"); + } +} diff --git a/crates/kit/src/table.rs b/crates/kit/src/table.rs index fd60fa5..6038d2e 100644 --- a/crates/kit/src/table.rs +++ b/crates/kit/src/table.rs @@ -1,12 +1,19 @@ //! Tables: `.rows` and `.cols` (movable lists of IDs) and `.cells` (a map from `row:column` to the -//! cell's value). A cell whose row or column is gone (deleted on one machine while edited on another) is never read -//! back, and goes with the next change to its row or column. +//! cell). A cell whose row or column is gone (deleted on one machine while edited on another) is never read back, and +//! goes with the next change to its row or column. +//! +//! A cell is a value, or, for a table whose source says so ([`TableSource::cell_maps`]), a property map of its own +//! (a mergeable child of the cell map) holding values and stories, written like a tree node's. A table can also sit +//! inside a tree node's (or a cell's) map ([`crate::Props::table`]): the same three containers, as mergeable children +//! `rows`, `cols` and `cells` of the map under the codec's key. -use std::collections::{BTreeSet, HashMap, HashSet}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; -use loro::{LoroDoc, LoroMap, LoroMovableList, ValueOrContainer}; +use loro::{Container, LoroDoc, LoroMap, LoroMovableList, StyleConfigMap, TextDelta, ValueOrContainer}; use crate::mapping::TableSource; +use crate::props::{Props, PropsRead}; +use crate::text::RichText; use crate::writer::Writer; use crate::{ElemId, KitError, Value}; @@ -31,7 +38,8 @@ pub(crate) fn parse_cell_key(key: &str) -> Option<(ElemId, ElemId)> { Some((ElemId::from_hex(r)?, ElemId::from_hex(c)?)) } -struct Containers { +#[derive(Clone)] +pub(crate) struct Containers { rows: LoroMovableList, cols: LoroMovableList, cells: LoroMap, @@ -41,6 +49,15 @@ fn containers(doc: &LoroDoc, key: &str) -> Containers { Containers { rows: doc.get_movable_list(rows_name(key)), cols: doc.get_movable_list(cols_name(key)), cells: doc.get_map(cells_name(key)) } } +/// A nested table's containers, made where missing. +fn nested_made(map: &LoroMap) -> Result { + Ok(Containers { + rows: map.ensure_mergeable_movable_list("rows")?, + cols: map.ensure_mergeable_movable_list("cols")?, + cells: map.ensure_mergeable_map("cells")?, + }) +} + fn set_cell(w: &mut Writer<'_>, cells: &LoroMap, row: ElemId, col: ElemId, value: Option) -> Result<(), KitError> { let key = cell_key(row, col); match value { @@ -49,14 +66,58 @@ fn set_cell(w: &mut Writer<'_>, cells: &LoroMap, row: ElemId, col: ElemId, value } } -/// Writes what changed from `old` to `new` (everything, when `old` is `None`). -pub(crate) fn write(w: &mut Writer<'_>, key: &str, old: Option<&dyn TableSource>, new: &dyn TableSource) -> Result<(), KitError> { +/// Writes one map cell: its property map made if missing, then the source's properties and stories. +fn write_map_cell( + w: &mut Writer<'_>, + cells: &LoroMap, + row: ElemId, + col: ElemId, + new: &dyn TableSource, + styles: &mut StyleConfigMap, +) -> Result<(), KitError> { + let key = cell_key(row, col); + let map = match cells.get(&key) { + Some(ValueOrContainer::Container(Container::Map(map))) => map, + _ => { + w.spend(crate::writer::OP_OVERHEAD + key.len())?; + cells.ensure_mergeable_map(&key)? + } + }; + let mut props = Props::new(map, w, styles); + new.write_cell(row, col, &mut props)?; + props.finish() +} + +/// Writes what changed from `old` to `new` (everything, when `old` is `None`) into the root table `key`. +pub(crate) fn write( + w: &mut Writer<'_>, + key: &str, + old: Option<&dyn TableSource>, + new: &dyn TableSource, + styles: &mut StyleConfigMap, +) -> Result<(), KitError> { if let (Some(old), Some(k)) = (old, new.key()) && old.key() == Some(k) { return Ok(()); } let c = containers(w.doc, key); + write_in(w, &c, old, new, styles) +} + +/// Writes `new` into the table under `map` (a node's or a cell's), against what it holds. +pub(crate) fn write_nested(w: &mut Writer<'_>, map: &LoroMap, new: &dyn TableSource, styles: &mut StyleConfigMap) -> Result<(), KitError> { + let c = nested_made(map)?; + write_in(w, &c, None, new, styles) +} + +fn write_in( + w: &mut Writer<'_>, + c: &Containers, + old: Option<&dyn TableSource>, + new: &dyn TableSource, + styles: &mut StyleConfigMap, +) -> Result<(), KitError> { let stored_cols = crate::list::ids(&c.cols); let stored_rows = crate::list::ids(&c.rows); let (new_rows, new_cols) = (new.rows(), new.columns()); @@ -73,6 +134,33 @@ pub(crate) fn write(w: &mut Writer<'_>, key: &str, old: Option<&dyn TableSource> set_cell(w, &c.cells, *row, *col, None)?; } } + if new.cell_maps() { + // Every cell of the table is a map: each written unless the source says it is unchanged. + let hint: Option> = match (old, new.changed_cells()) { + (Some(_), Some(hint)) => Some(hint.into_iter().collect()), + _ => None, + }; + for row in &new_rows { + for col in &new_cols { + if let Some(hint) = &hint + && !hint.contains(&(*row, *col)) + { + continue; + } + let unchanged = old.is_some_and(|old| { + let k = old.cell_key(*row, *col); + k.is_some() && k == new.cell_key(*row, *col) + }); + if !unchanged { + write_map_cell(w, &c.cells, *row, *col, new, styles)?; + } + } + } + if old.is_none() { + sweep(w, c, &new_rows, &new_cols, |_, _| false)?; + } + return Ok(()); + } match (old, new.changed_cells()) { (Some(_), Some(hint)) => { for (row, col) in hint { @@ -95,14 +183,34 @@ pub(crate) fn write(w: &mut Writer<'_>, key: &str, old: Option<&dyn TableSource> } } (None, _) => { + let mut now: HashSet<(ElemId, ElemId)> = HashSet::new(); for (row, col, value) in new.cells() { + now.insert((row, col)); set_cell(w, &c.cells, row, col, Some(value))?; } + // Against what the table holds: a cell emptied since goes. + sweep(w, c, &new_rows, &new_cols, |r, col| !now.contains(&(r, col)))?; } } Ok(()) } +/// Deletes the stored cells of rows or columns that aren't the table's (orphans of a merge), and those of its own +/// cells `stale` says go. +fn sweep(w: &mut Writer<'_>, c: &Containers, rows: &[ElemId], cols: &[ElemId], stale: impl Fn(ElemId, ElemId) -> bool) -> Result<(), KitError> { + let rows: HashSet = rows.iter().copied().collect(); + let cols: HashSet = cols.iter().copied().collect(); + let mut gone = Vec::new(); + c.cells.for_each(|key, _| match parse_cell_key(key) { + Some((r, col)) if rows.contains(&r) && cols.contains(&col) && !stale(r, col) => {} + _ => gone.push(key.to_owned()), + }); + for key in gone { + w.delete(&c.cells, &key)?; + } + Ok(()) +} + /// What a remote change touched of a table. #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct TableTouch { @@ -110,8 +218,10 @@ pub struct TableTouch { pub all: bool, pub rows: bool, pub columns: bool, - /// The cells written or removed. + /// The cells written or removed (a map cell's properties or stories changed). pub cells: BTreeSet<(ElemId, ElemId)>, + /// The stories of map cells that changed: row, column and the story's key. + pub stories: BTreeSet<(ElemId, ElemId, String)>, } impl TableTouch { @@ -120,16 +230,33 @@ impl TableTouch { } } +/// Loro's text deltas of the cells' stories, by row, column and key, in the order they came. +pub(crate) type CellDeltas = BTreeMap<(ElemId, ElemId, String), Vec>>; + /// A table as stored, after a remote change. pub struct TableView<'a> { c: Containers, touch: TableTouch, - _doc: std::marker::PhantomData<&'a LoroDoc>, + deltas: Option<&'a CellDeltas>, } impl<'a> TableView<'a> { - pub(crate) fn new(doc: &'a LoroDoc, key: &str, touch: TableTouch) -> Self { - Self { c: containers(doc, key), touch, _doc: std::marker::PhantomData } + pub(crate) fn new(doc: &'a LoroDoc, key: &str, touch: TableTouch, deltas: Option<&'a CellDeltas>) -> Self { + Self { c: containers(doc, key), touch, deltas } + } + + /// The table under a node's or cell's map, read whole. + pub(crate) fn nested(map: &LoroMap) -> TableView<'static> { + // Read without making anything: missing containers read as empty, detached ones. + let get_list = |k: &str| match map.get(k) { + Some(ValueOrContainer::Container(Container::MovableList(l))) => l, + _ => LoroMovableList::new(), + }; + let cells = match map.get("cells") { + Some(ValueOrContainer::Container(Container::Map(m))) => m, + _ => LoroMap::new(), + }; + TableView { c: Containers { rows: get_list("rows"), cols: get_list("cols"), cells }, touch: TableTouch::whole(), deltas: None } } pub fn touch(&self) -> &TableTouch { @@ -146,7 +273,7 @@ impl<'a> TableView<'a> { crate::list::ids(&self.c.cols) } - /// A cell's value (`None` when empty or unreadable). + /// A cell's value (`None` when empty, unreadable, or a map cell). pub fn cell(&self, row: ElemId, col: ElemId) -> Option { match self.c.cells.get(&cell_key(row, col))? { ValueOrContainer::Value(v) => Value::from_loro(&v), @@ -154,7 +281,24 @@ impl<'a> TableView<'a> { } } - /// Every readable cell of a row and column that are there. + /// A map cell's properties and stories (`None` when it has none: a cell a merge left out, which the app's + /// `normalise` makes again). + pub fn cell_props(&self, row: ElemId, col: ElemId) -> Option { + match self.c.cells.get(&cell_key(row, col))? { + ValueOrContainer::Container(Container::Map(map)) => Some(PropsRead { map }), + ValueOrContainer::Value(_) | ValueOrContainer::Container(_) => None, + } + } + + /// A map cell's story `key`, patched from `old` (the app's text of it before the change) when the remote change's + /// deltas fit it, else read whole. + pub fn cell_text(&self, row: ElemId, col: ElemId, key: &str, old: Option<&dyn crate::TextSource>) -> Option { + let props = self.cell_props(row, col)?; + let deltas = self.deltas.and_then(|d| d.get(&(row, col, key.to_owned()))); + props.text_patched(key, old, deltas) + } + + /// Every readable cell value of a row and column that are there. pub fn cells(&self) -> Vec<(ElemId, ElemId, Value)> { let rows: HashSet = self.rows().into_iter().collect(); let cols: HashSet = self.columns().into_iter().collect(); diff --git a/crates/kit/src/text.rs b/crates/kit/src/text.rs index 69e00a7..18ead71 100644 --- a/crates/kit/src/text.rs +++ b/crates/kit/src/text.rs @@ -11,7 +11,7 @@ use std::borrow::Cow; use std::ops::Range; use loro::cursor::PosType; -use loro::{ExpandType, LoroDoc, LoroText, StyleConfig, StyleConfigMap, TextDelta, ValueOrContainer}; +use loro::{Container, ExpandType, LoroDoc, LoroMap, LoroText, StyleConfig, StyleConfigMap, TextDelta, ValueOrContainer}; use crate::mapping::{Attrs, TextSource}; use crate::writer::Writer; @@ -27,6 +27,71 @@ pub(crate) fn last_name(key: &str) -> String { format!("{key}.last") } +/// The key, beside a nested story in its node's or cell's map, of the map holding its last paragraph's attributes. +pub(crate) fn nested_last_key(key: &str) -> String { + format!("$last.{key}") +} + +/// Where the last paragraph's attributes of a story are kept. +#[derive(Clone)] +enum Last { + /// A root story's: the root map `.last`. + Root(LoroMap), + /// A nested story's: a mergeable map beside it in its parent map, made only when there is something to keep. + Under { parent: LoroMap, key: String }, +} + +/// One rich text in the document: a root story (a part of its own), or a story nested in a tree node's or a table +/// cell's map, under a key the app's codec names. Both map, diff, mark and patch the same way. +#[derive(Clone)] +pub(crate) struct Story { + pub(crate) text: LoroText, + last: Last, +} + +impl Story { + /// The root story `key`. + pub(crate) fn root(doc: &LoroDoc, key: &str) -> Self { + Self { text: doc.get_text(key), last: Last::Root(doc.get_map(last_name(key))) } + } + + /// The story under `key` of `parent`, made if it isn't there: a mergeable child, so two machines making it at once + /// make one, and their typing merges in it. + pub(crate) fn nested(parent: &LoroMap, key: &str) -> Result { + let text = parent.ensure_mergeable_text(key)?; + Ok(Self { text, last: Last::Under { parent: parent.clone(), key: nested_last_key(key) } }) + } + + /// The story under `key` of `parent`, if there is one. + pub(crate) fn existing(parent: &LoroMap, key: &str) -> Option { + match parent.get(key)? { + ValueOrContainer::Container(Container::Text(text)) => { + Some(Self { text, last: Last::Under { parent: parent.clone(), key: nested_last_key(key) } }) + } + _ => None, + } + } + + /// The map of the last paragraph's attributes, if there is one. + fn last_map(&self) -> Option { + match &self.last { + Last::Root(map) => Some(map.clone()), + Last::Under { parent, key } => match parent.get(key)? { + ValueOrContainer::Container(Container::Map(map)) => Some(map), + _ => None, + }, + } + } + + /// The map of the last paragraph's attributes, made if it isn't there. + fn last_map_made(&self) -> Result { + match &self.last { + Last::Root(map) => Ok(map.clone()), + Last::Under { parent, key } => Ok(parent.ensure_mergeable_map(key)?), + } + } +} + /// A story as plain data: text, character runs and paragraph attributes. It is a [`TextSource`] too. #[derive(Debug, Clone, Default, PartialEq)] pub struct RichText { @@ -264,7 +329,7 @@ fn breaks(text: &str) -> Vec { text.bytes().enumerate().filter(|(_, b)| *b == b'\n').map(|(i, _)| i).collect() } -/// Writes what changed from `old` to `new` (everything, when `old` is `None`) into the text `key`. +/// Writes what changed from `old` to `new` (everything, when `old` is `None`) into the root text `key`. pub(crate) fn write( w: &mut Writer<'_>, key: &str, @@ -277,7 +342,19 @@ pub(crate) fn write( { return Ok(()); } - let text = w.doc.get_text(key); + let story = Story::root(w.doc, key); + write_story(w, &story, old, new, styles) +} + +/// Writes what changed from `old` (`None`: what the story holds) to `new` into `story`. +pub(crate) fn write_story( + w: &mut Writer<'_>, + story: &Story, + old: Option<&dyn TextSource>, + new: &dyn TextSource, + styles: &mut StyleConfigMap, +) -> Result<(), KitError> { + let text = story.text.clone(); let new_text = new.text(); let new_runs = new.runs(); check_runs(new_text, &new_runs)?; @@ -293,7 +370,7 @@ pub(crate) fn write( let (old_text, old_runs, old_paras): (Cow<'_, str>, Cow<'_, [Run]>, Cow<'_, [Attrs]>) = match old { Some(o) if text.len_utf8() == o.text().len() => (Cow::Borrowed(o.text()), o.runs(), o.paragraphs()), _ => { - stored = read(w.doc, key); + stored = read_story(story); (Cow::Owned(stored.text), Cow::Owned(stored.runs), Cow::Owned(stored.paragraphs)) } }; @@ -306,26 +383,11 @@ pub(crate) fn write( text.delete_utf8(p, old_end - p)?; w.text_bytes = w.text_bytes.saturating_add(old_end - p); } - let mut at = p; - while at < new_end { - let mut room = w.room(); - if room == 0 { - w.cut()?; - room = w.room(); - } - let mut end = (at + room.max(1)).min(new_end); - while end > at && !new_text.is_char_boundary(end) { - end -= 1; - } - if end == at { - // One character bigger than the room: it goes alone. - end = at + new_text.get(at..).and_then(|t| t.chars().next()).map_or(1, char::len_utf8); - } - let piece = new_text.get(at..end).ok_or_else(|| KitError::Invalid("the text's edit isn't on character boundaries".into()))?; + if new_end > p { + let piece = new_text.get(p..new_end).ok_or_else(|| KitError::Invalid("the text's edit isn't on character boundaries".into()))?; w.spend(piece.len())?; - text.insert_utf8(at, piece)?; + text.insert_utf8(p, piece)?; w.text_bytes = w.text_bytes.saturating_add(piece.len()); - at = end; } // Character attributes. let middle = stored_runs(&text, p, new_end)?; @@ -380,27 +442,34 @@ pub(crate) fn write( } apply(w, &text, styles, differences(&desired, ¤t), &PARA.to_string())?; } - // The last paragraph's. - let last = w.doc.get_map(last_name(key)); + // The last paragraph's (a nested story's map is made only when there is something to keep). let want = new_paras.get(new_count).cloned().unwrap_or_default(); - for (k, v) in &want { - w.set(&last, k, v)?; - } - let mut stale = Vec::new(); - last.for_each(|k, _| { - if !want.contains_key(k) { - stale.push(k.to_owned()); + let last = if want.is_empty() { story.last_map() } else { Some(story.last_map_made()?) }; + if let Some(last) = last { + for (k, v) in &want { + w.set(&last, k, v)?; + } + let mut stale = Vec::new(); + last.for_each(|k, _| { + if !want.contains_key(k) { + stale.push(k.to_owned()); + } + }); + for k in stale { + w.delete(&last, &k)?; } - }); - for k in stale { - w.delete(&last, &k)?; } Ok(()) } -/// The text `key` as stored. +/// The root text `key` as stored. pub(crate) fn read(doc: &LoroDoc, key: &str) -> RichText { - let text = doc.get_text(key); + read_story(&Story::root(doc, key)) +} + +/// A story as stored. +pub(crate) fn read_story(story: &Story) -> RichText { + let text = &story.text; let mut out = RichText::default(); for delta in text.to_delta() { let TextDelta::Insert { insert, attributes } = delta else { continue }; @@ -419,14 +488,15 @@ pub(crate) fn read(doc: &LoroDoc, key: &str) -> RichText { _ => out.runs.push((start..end, chars)), } } - out.paragraphs.push(last_paragraph(doc, key)); + out.paragraphs.push(last_paragraph(story)); out } /// The last paragraph's attributes, as stored beside the text. -fn last_paragraph(doc: &LoroDoc, key: &str) -> Attrs { +fn last_paragraph(story: &Story) -> Attrs { let mut last = Attrs::new(); - doc.get_map(last_name(key)).for_each(|k, v| { + let Some(map) = story.last_map() else { return last }; + map.for_each(|k, v| { if let ValueOrContainer::Value(v) = v && let Some(v) = Value::from_loro(&v) { @@ -535,6 +605,11 @@ fn changes<'a>(attributes: impl IntoIterator]) -> Option { + patch_story(&Story::root(doc, key), old, deltas) +} + +/// [`patch`] for any story. +pub(crate) fn patch_story(story: &Story, old: &dyn TextSource, deltas: &[Vec]) -> Option { let mut text = old.text().to_owned(); let mut runs: Vec<(Range, Attrs)> = old.runs().into_owned(); let mut paras: Vec = old.paragraphs().into_owned(); @@ -627,10 +702,10 @@ pub(crate) fn patch(doc: &LoroDoc, key: &str, old: &dyn TextSource, deltas: &[Ve paras = new_paras; } // Out of step with what Loro holds: no answer. - if text.len() != doc.get_text(key).len_utf8() || paras.len() != text.matches('\n').count() { + if text.len() != story.text.len_utf8() || paras.len() != text.matches('\n').count() { return None; } - paras.push(last_paragraph(doc, key)); + paras.push(last_paragraph(story)); Some(RichText { text, runs, paragraphs: paras }) } diff --git a/crates/kit/src/timeline.rs b/crates/kit/src/timeline.rs index 946ba1e..e1eecf5 100644 --- a/crates/kit/src/timeline.rs +++ b/crates/kit/src/timeline.rs @@ -7,7 +7,7 @@ use std::collections::{BTreeSet, HashMap, HashSet}; -use loro::{LoroDoc, LoroMap, LoroMovableList, ValueOrContainer}; +use loro::{LoroDoc, LoroMap, LoroMovableList, StyleConfigMap, ValueOrContainer}; use crate::mapping::{Span, TimelineSource, props_of}; use crate::props::{Props, PropsRead}; @@ -78,7 +78,13 @@ fn all_items(src: &dyn TimelineSource) -> HashMap { } /// Writes what changed from `old` to `new` (everything, when `old` is `None`) into the timeline `key`. -pub(crate) fn write(w: &mut Writer<'_>, key: &str, old: Option<&dyn TimelineSource>, new: &dyn TimelineSource) -> Result<(), KitError> { +pub(crate) fn write( + w: &mut Writer<'_>, + key: &str, + old: Option<&dyn TimelineSource>, + new: &dyn TimelineSource, + styles: &mut StyleConfigMap, +) -> Result<(), KitError> { if let (Some(old), Some(k)) = (old, new.key()) && old.key() == Some(k) { @@ -98,7 +104,7 @@ pub(crate) fn write(w: &mut Writer<'_>, key: &str, old: Option<&dyn TimelineSour continue; } let map = child_map(w, &c.track_props, &track.hex())?; - let mut props = Props::new(map, w); + let mut props = Props::new(map, w, styles); new.write_track(*track, &mut props)?; props.finish()?; for item in new.items(*track) { @@ -107,7 +113,7 @@ pub(crate) fn write(w: &mut Writer<'_>, key: &str, old: Option<&dyn TimelineSour if same_item { continue; } - write_item(w, &c, item, *track, old, new)?; + write_item(w, &c, item, *track, old, new, styles)?; } } // Items gone, with their keyframes. @@ -129,6 +135,7 @@ fn write_item( track: ElemId, old: Option<&dyn TimelineSource>, new: &dyn TimelineSource, + styles: &mut StyleConfigMap, ) -> Result<(), KitError> { let Some(Span { start, len }) = new.span(item) else { return Err(KitError::Invalid(format!("the item {item} has no span"))); @@ -137,7 +144,7 @@ fn write_item( w.set(&map, TRACK, &Value::Bytes(track.0.to_le_bytes().to_vec()))?; w.set(&map, START, &Value::I64(start))?; w.set(&map, LEN, &Value::I64(len))?; - let mut props = Props::new(map, w); + let mut props = Props::new(map, w, styles); new.write_item(item, &mut props)?; props.finish()?; // Keyframes: by ID; only the ones that changed. diff --git a/crates/kit/src/tree.rs b/crates/kit/src/tree.rs index 24980fe..f9822ce 100644 --- a/crates/kit/src/tree.rs +++ b/crates/kit/src/tree.rs @@ -6,12 +6,13 @@ //! the old one; under each parent it visits, it creates what is missing, moves only what is out of order (the longest //! run in order stays), and deletes, at the end, what left and went nowhere else. -use std::collections::{BTreeSet, HashMap, HashSet}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; -use loro::{LoroDoc, LoroTree, TreeID, TreeParentId, ValueOrContainer}; +use loro::{LoroDoc, LoroTree, StyleConfigMap, TextDelta, TreeID, TreeParentId, ValueOrContainer}; -use crate::mapping::{TreeSource, props_of}; +use crate::mapping::{TextSource, TreeSource, props_of}; use crate::props::{Props, PropsRead}; +use crate::text::RichText; use crate::writer::Writer; use crate::{ElemId, KitError}; @@ -56,7 +57,7 @@ impl TreeIndex { } } -fn node_id(tree: &LoroTree, tid: TreeID) -> Option { +pub(crate) fn node_id(tree: &LoroTree, tid: TreeID) -> Option { let meta = tree.get_meta(tid).ok()?; match meta.get(ID_KEY)? { ValueOrContainer::Value(v) => ElemId::from_loro(&v), @@ -71,6 +72,7 @@ pub(crate) fn write( old: Option<&dyn TreeSource>, new: &dyn TreeSource, index: &mut TreeIndex, + styles: &mut StyleConfigMap, ) -> Result<(), KitError> { if let (Some(old), Some(k)) = (old, new.key()) && old.key() == Some(k) @@ -108,7 +110,7 @@ pub(crate) fn write( skipped.push(tid); continue; } - let mut props = Props::new(tree.get_meta(tid)?, w); + let mut props = Props::new(tree.get_meta(tid)?, w, styles); new.write(*id, &mut props)?; props.finish()?; stack.push((TreeParentId::Node(tid), new.children(*id))); @@ -195,26 +197,45 @@ pub struct TreeTouch { pub all: bool, /// Nodes were created, moved or deleted: read the structure again. pub structure: bool, - /// The nodes whose properties changed (and the ones created). + /// The nodes whose properties, stories or tables changed (and the ones created). pub nodes: BTreeSet, + /// What changed inside nodes' child containers: the node and the path of keys under it (`text` for the story + /// `text`; `table/cells/:/text` for a story in a cell of the table `table`). + pub stories: BTreeSet<(ElemId, String)>, } impl TreeTouch { pub(crate) fn whole() -> Self { - Self { all: true, structure: true, nodes: BTreeSet::new() } + Self { all: true, structure: true, ..Self::default() } } } +/// Loro's text deltas of the stories nested in a tree's nodes, by node and story key, in the order they came. +pub(crate) type NodeDeltas = BTreeMap<(ElemId, String), Vec>>; + /// A tree as stored, after a remote change. pub struct TreeView<'a> { tree: LoroTree, index: &'a TreeIndex, touch: TreeTouch, + deltas: Option<&'a NodeDeltas>, } impl<'a> TreeView<'a> { - pub(crate) fn new(doc: &'a LoroDoc, key: &str, index: &'a TreeIndex, touch: TreeTouch) -> Self { - Self { tree: doc.get_tree(key), index, touch } + pub(crate) fn new(doc: &'a LoroDoc, key: &str, index: &'a TreeIndex, touch: TreeTouch, deltas: Option<&'a NodeDeltas>) -> Self { + Self { tree: doc.get_tree(key), index, touch, deltas } + } + + /// An element's story `key` as stored (`None`: it has none). + pub fn text(&self, id: ElemId, key: &str) -> Option { + self.props(id)?.text(key) + } + + /// An element's story `key`, patched from `old` (the app's text of it before the change: only what changed is + /// touched) when the remote change's deltas fit it, else read whole. `None`: it has none. + pub fn text_patched(&self, id: ElemId, key: &str, old: Option<&dyn TextSource>) -> Option { + let deltas = self.deltas.and_then(|d| d.get(&(id, key.to_owned()))); + self.props(id)?.text_patched(key, old, deltas) } pub fn touch(&self) -> &TreeTouch { diff --git a/crates/kit/src/writer.rs b/crates/kit/src/writer.rs index fc4ea0d..b839d85 100644 --- a/crates/kit/src/writer.rs +++ b/crates/kit/src/writer.rs @@ -1,82 +1,55 @@ -//! Writing one local change into Loro: the mappings write through a [`Writer`], which counts what they write and -//! commits before a commit's export would pass [`SPLIT_BYTES`] (a 5 MB paste becomes several commits, each well under -//! the agent's 1 MiB cap, one undo step together). Each commit's export is taken at once ([`export_since`]); an export -//! that is still over the limit (an undo of a big delete, which Loro makes as one commit) is cut by counter range, so -//! no update the kit sends is over the cap. +//! Writing one local change into Loro: the mappings write through a [`Writer`], which counts what they write, and the +//! change is one commit, exported as one update ([`export_since`]: a big one as a bundle of slices) whatever its size. +//! An update over the agent's 1 MiB frame limit goes in fragments (`craft-cloud-app` cuts it), and every receiver +//! imports it only once it holds all of them, so no machine ever holds part of a change. +use craft_cloud_api::fragments::{SLICE_OPS, bundle}; use loro::{IdSpan, LoroDoc, LoroMap, LoroValue, ValueOrContainer, VersionVector}; use crate::{KitError, Value}; -/// A commit's estimated export is kept at or under this (the agent refuses an update over 1 MiB). -pub const SPLIT_BYTES: usize = 512 * 1024; -/// What one operation adds to the estimate beside its payload (IDs, positions, keys). -pub const OP_OVERHEAD: usize = 32; +/// What one operation adds to the count beside its payload (IDs, positions, keys). +pub(crate) const OP_OVERHEAD: usize = 32; /// The origin of the kit's commits of local changes (the undo manager records them). pub const LOCAL_ORIGIN: &str = "craft-cloud-kit:local"; /// The origin of the kit's own repairs (a normalised merge, a save mark): never an undo step. pub const KIT_ORIGIN: &str = "craft-cloud-kit:kit"; -/// The updates one change made: the exports of its commits, in order, each at most [`SPLIT_BYTES`] (one operation -/// over it alone, at most the agent's 1 MiB cap: values and text pieces are capped below it). +/// The updates one change made: none (nothing changed) or its one update, whole. pub type Updates = Vec>; /// The write side of one change. pub struct Writer<'d> { pub(crate) doc: &'d LoroDoc, origin: &'static str, - /// The estimate of what the open commit holds. + /// What the change wrote, roughly (it only says whether it wrote anything: a change is one commit whatever its + /// size). spent: usize, - /// Where the open commit started. + /// Where the change started. from: VersionVector, - out: Updates, - commits: usize, /// The text this change inserted and deleted, in bytes (what an undo of it would have Loro go through). pub(crate) text_bytes: usize, } impl<'d> Writer<'d> { pub(crate) fn new(doc: &'d LoroDoc, origin: &'static str) -> Self { - Self { doc, origin, spent: 0, from: doc.oplog_vv(), out: Vec::new(), commits: 0, text_bytes: 0 } + Self { doc, origin, spent: 0, from: doc.oplog_vv(), text_bytes: 0 } } - /// Room left in the open commit before it is cut. - pub(crate) fn room(&self) -> usize { - SPLIT_BYTES.saturating_sub(self.spent).saturating_sub(OP_OVERHEAD) - } - - /// Counts an operation of `payload` bytes, committing first if it wouldn't fit in the open commit. + /// Counts an operation of `payload` bytes. (Never fails now that a change is never cut; kept a `Result` for the + /// mappings, which may refuse a write later.) pub(crate) fn spend(&mut self, payload: usize) -> Result<(), KitError> { - let cost = payload.saturating_add(OP_OVERHEAD); - if self.spent > 0 && self.spent.saturating_add(cost) > SPLIT_BYTES { - self.cut()?; - } - self.spent = self.spent.saturating_add(cost); - Ok(()) - } - - /// Commits what is open and takes its export. - pub(crate) fn cut(&mut self) -> Result<(), KitError> { - if self.spent == 0 { - return Ok(()); - } - self.doc.set_next_commit_origin(self.origin); - self.doc.commit(); - self.commits += 1; - export_since(self.doc, &self.from, &mut self.out)?; - self.from = self.doc.oplog_vv(); - self.spent = 0; + self.spent = self.spent.saturating_add(payload.saturating_add(OP_OVERHEAD)); Ok(()) } - /// The change's updates (the last commit made now). - pub(crate) fn finish(mut self) -> Result<(Updates, usize), KitError> { - self.cut()?; - // Anything written without being counted (there should be none) still goes out. + /// The change's update (its one commit made now), and how many commits it made (0 or 1). + pub(crate) fn finish(self) -> Result<(Updates, usize), KitError> { self.doc.set_next_commit_origin(self.origin); self.doc.commit(); - export_since(self.doc, &self.from, &mut self.out)?; - Ok((self.out, self.commits)) + let mut out = Vec::new(); + export_since(self.doc, &self.from, &mut out)?; + Ok((out, usize::from(self.spent > 0))) } /// Sets `key` of `map` to `value` unless it holds that already. @@ -116,34 +89,23 @@ impl<'d> Writer<'d> { } } -/// The export of every change since `from` (all made by this peer: a local change, an undo), cut by counter range -/// where it is over [`SPLIT_BYTES`], appended to `out`. +/// The export of every change since `from` (all made by this peer: a local change, an undo, a repair) as one update, +/// appended to `out` (nothing when there is none): its slices of at most `SLICE_OPS` operations in a bundle when it +/// has more (Loro imports one update in time growing with the square of the text in it; slices import in +/// milliseconds, and the bundle keeps them one change). pub(crate) fn export_since(doc: &LoroDoc, from: &VersionVector, out: &mut Updates) -> Result<(), KitError> { let now = doc.oplog_vv(); - let mut spans = Vec::new(); + let mut slices = Vec::new(); for (peer, end) in now.iter() { - let start = from.get(peer).copied().unwrap_or(0); - if *end > start { - spans.push(IdSpan::new(*peer, start, *end)); + let mut at = from.get(peer).copied().unwrap_or(0); + while at < *end { + let to = at.saturating_add(SLICE_OPS).min(*end); + slices.push(doc.export(loro::ExportMode::updates_in_range(vec![IdSpan::new(*peer, at, to)]))?); + at = to; } } - for span in spans { - slice(doc, span, out)?; + if !slices.is_empty() { + out.push(bundle(&slices)); } Ok(()) } - -fn slice(doc: &LoroDoc, span: IdSpan, out: &mut Updates) -> Result<(), KitError> { - let bytes = doc.export(loro::ExportMode::updates_in_range(vec![span]))?; - let (start, end) = (span.counter.start, span.counter.end); - if bytes.len() <= SPLIT_BYTES || end - start <= 1 { - // (One operation over the cap can't happen: values and text pieces are capped before they are written. Were - // it to, it still goes: the commit is made, and the agent's refusal stops the session where the app sees it, - // rather than leaving a hole every later change depends on.) - out.push(bytes); - return Ok(()); - } - let mid = start + (end - start) / 2; - slice(doc, IdSpan::new(span.peer, start, mid), out)?; - slice(doc, IdSpan::new(span.peer, mid, end), out) -} diff --git a/crates/kit/tests/fragments.rs b/crates/kit/tests/fragments.rs new file mode 100644 index 0000000..409be3a --- /dev/null +++ b/crates/kit/tests/fragments.rs @@ -0,0 +1,299 @@ +//! Changes over the agent's 1 MiB frame limit: one change is one update, which goes in fragments, and no machine +//! ever holds part of one. What the other machine does meanwhile (a local edit of the same rows, node or story, an +//! undo, a save) never sees, repairs or writes back part of it; a receiving machine that sleeps part of the way +//! through a change gives its apps the change whole once it wakes; a sender that stops part of the way holds nothing +//! up and loses nothing. + +mod common; +mod model; + +use std::sync::{Arc, Once}; +use std::time::{Duration, Instant}; + +use common::{App, Notes, change, open, until}; +use craft_cloud_api::fragments::split; +use craft_cloud_api::limits::MAX_UPDATE_BYTES; +use craft_cloud_app::fake::{FragmentFaults, Hub}; +use craft_cloud_kit::{ElemId, RichText, Session}; +use model::{Doc, Node, TestApp}; + +/// The machine sleeps after a set's first fragment (until the test wakes it): the change comes part of the way. +const FIRST_ONLY: FragmentFaults = FragmentFaults { seed: 3, sleep: 1000, delay: 0, max_delay: 0 }; + +fn setup() { + static ONCE: Once = Once::new(); + ONCE.call_once(|| { + // Frames of 64 KiB: changes of a few hundred KB go in fragments as changes over 1 MiB do. + craft_cloud_app::live::set_frame_limit_for_tests(64 * 1024); + }); +} + +/// Six rows of 200 KiB each, added at the top: about 1.2 MB, more than a frame. +fn big_rows(m: &Notes) -> Notes { + let mut big = m.clone(); + for i in 0..6 { + big = big.with_item(i, &format!("{i}{}", "x".repeat(200 * 1024))); + } + Notes { hint: None, ..big } +} + +/// Polls `sessions` for `wait`, checking `never` holds at every poll. +fn poll_for(sessions: &mut [&mut Session], wait: Duration, never: impl Fn(&[&mut Session]) -> bool) { + let end = Instant::now() + wait; + while Instant::now() < end { + for s in sessions.iter_mut() { + s.poll(); + } + assert!(never(sessions), "a model held part of a change"); + std::thread::sleep(Duration::from_millis(5)); + } +} + +/// One change is one commit and one update, whatever its size; past 1 MiB it is sent as fragments that fit a frame. +#[test] +fn a_big_change_is_one_update_and_goes_in_fragments() { + let mut r = craft_cloud_kit::raw::Replica::default(); + let base = Notes::new("t", &["first"]); + r.write::(None, &base).unwrap(); + let mut big = base.clone(); + for i in 0..15 { + big = big.with_item(i, &"y".repeat(200 * 1024)); + } + let (updates, commits) = r.write::(Some(&base), &Notes { hint: None, ..big }).unwrap(); + assert_eq!((updates.len(), commits), (1, 1)); + let frames = split(&updates[0], MAX_UPDATE_BYTES).unwrap(); + assert!(frames.len() >= 3, "{} fragments for {} bytes", frames.len(), updates[0].len()); + assert!(frames.iter().all(|f| f.len() <= MAX_UPDATE_BYTES)); + let mut r = craft_cloud_kit::raw::Replica::default(); + r.write::(None, &base).unwrap(); + let (updates, _) = r.write::(Some(&base), &big_rows(&base)).unwrap(); + assert!(updates[0].len() > MAX_UPDATE_BYTES, "the tests' 1 MB change is over a frame: {} bytes", updates[0].len()); + // A big paste of text: a bundle of slices of at most SLICE_OPS operations (Loro imports each in milliseconds). + let mut r = craft_cloud_kit::raw::Replica::default(); + let (none, _) = r.write::(None, &Doc::empty()).unwrap(); + let mut pasted = Doc::empty(); + pasted.story = Arc::new(RichText::plain(&"t".repeat(300_000))); + let (updates, _) = r.write::(Some(&Doc::empty()), &pasted).unwrap(); + assert!(none.len() <= 1 && updates.len() == 1); + let slices = craft_cloud_api::fragments::parts(&updates[0]).unwrap(); + assert!(slices.len() >= 300_000 / craft_cloud_api::fragments::SLICE_OPS as usize, "{} slices", slices.len()); + for s in slices { + let meta = loro::LoroDoc::decode_import_blob_meta(s, false).unwrap(); + let ops: i32 = meta.partial_end_vv.iter().map(|(p, e)| e - meta.partial_start_vv.get(p).copied().unwrap_or(0)).sum(); + assert!(ops <= craft_cloud_api::fragments::SLICE_OPS, "{ops} operations in a slice"); + } +} + +/// A 5 MB change (twenty 250 KiB notes) goes as one update in fragments and arrives whole: the other machine's model +/// has none or all of it at every poll, and the log has one set. Its undo goes and arrives whole too. So does a 5 MB +/// paste of text, whose slices go in one bundle (sent as separate updates, the other machine shows part of it). +#[test] +fn a_five_megabyte_change_arrives_whole_and_undoes_whole() { + setup(); + let hub = Hub::new(); + let (pc, laptop) = (hub.machine("pc").unwrap(), hub.machine("laptop").unwrap()); + let mut a = open::(&pc, "big.txt", Notes::new("big", &[])); + let mut b = open::(&laptop, "big.txt", Notes::default()); + let before = hub.updates("big.txt"); + let notes: Vec = (0..20).map(|i| format!("{i:02}{}", "x".repeat(250 * 1024))).collect(); + change(&mut a, |m| { + let mut items = (*m.items).clone(); + let mut hint = Vec::new(); + for n in ¬es { + let id = ElemId::new(); + hint.push(id); + items.push((id, n.clone())); + } + Notes { items: Arc::new(items), hint: Some(hint), ..m.clone() } + }) + .unwrap(); + until(&mut [&mut a, &mut b], "the laptop has all 5 MB", |s| { + let texts = s[1].model().texts(); + // (Its text too: a note in it, but only part of its text, is part of the change.) + assert!(texts.is_empty() || texts == notes, "the laptop held part of the change: {} notes", texts.len()); + texts.len() == 20 + }); + assert_eq!(b.model().texts(), notes); + let sent = hub.updates("big.txt") - before; + assert!(sent >= 70, "{sent} fragments of 64 KiB for one 5 MB update"); + // One update: every row it took is a fragment of one set, and they join (a change sent as separate updates, its + // slices, would arrive in parts a poll could show). + let one_set = |rows: &[Vec]| { + let sets: std::collections::BTreeSet<_> = rows.iter().map(|r| craft_cloud_api::fragments::parse(r).map(|f| f.set)).collect(); + assert_eq!(sets.len(), 1, "rows of one change, all fragments of one set: {:?}", sets.len()); + assert!(sets.iter().all(Option::is_some), "every row a fragment"); + assert!(craft_cloud_api::fragments::join(rows.iter().map(Vec::as_slice)).is_some(), "they make the change"); + }; + let rows = hub.rows("big.txt"); + one_set(&rows[before..]); + let after_change = rows.len(); + assert!(a.status().error.is_none() && b.status().error.is_none(), "{:?} {:?}", a.status().error, b.status().error); + // One undo step takes the whole change back, on both, whole. + assert!(a.undo().unwrap()); + until(&mut [&mut a, &mut b], "the undo is on both", |s| { + let texts = s[1].model().texts(); + assert!(texts.is_empty() || texts == notes, "the laptop held part of the undo: {} notes", texts.len()); + s.iter().all(|s| s.model().items.is_empty()) + }); + // The undo is one update too: one row, or the fragments of one set. + let undo = hub.rows("big.txt").split_off(after_change); + if undo.len() > 1 { + one_set(&undo); + } + assert!(!undo.is_empty()); + // A 5 MB paste of text: its operations cut into hundreds of slices, in one bundle, in fragments. The other + // machine's story is empty or the whole paste at every poll, and the log has one set. + let mut a = open::(&pc, "paste.kitdemo", Doc::empty()); + let mut b = open::(&laptop, "paste.kitdemo", Doc::empty()); + let before = hub.updates("paste.kitdemo"); + let paste = "q".repeat(5 * 1024 * 1024); + change(&mut a, |d| Doc { story: Arc::new(RichText::plain(&paste)), ..d.clone() }).unwrap(); + until(&mut [&mut a, &mut b], "the laptop has the paste", |s| { + let len = s[1].model().story.text.len(); + assert!(len == 0 || len == paste.len(), "the laptop held part of the paste: {len} bytes"); + len == paste.len() + }); + one_set(&hub.rows("paste.kitdemo")[before..]); +} + +/// The other machine gets a 1 MB change part of the way (its first fragment, then it sleeps), and meanwhile adds a +/// row, edits and undoes a cell, and changes the title: its model never shows part of the change, nothing of it is +/// written back, and once the change comes whole (the machine wakes, and its link brings the rest), every row of it +/// and every local edit are on both machines. +#[test] +fn local_edits_while_a_change_comes_in_fragments_keep_both() { + setup(); + let hub = Hub::new(); + let (pc, laptop) = (hub.machine("pc").unwrap(), hub.machine("laptop").unwrap()); + let mut a = open::(&pc, "rows.txt", Notes::new("t", &["first"])); + let mut b = open::(&laptop, "rows.txt", Notes::default()); + until(&mut [&mut a, &mut b], "the laptop has the document", |s| s[1].model().texts() == ["first"]); + change(&mut b, |m| m.with_item(1, "local")).unwrap(); + until(&mut [&mut a, &mut b], "the pc has the laptop's row", |s| s[0].model().items.len() == 2); + laptop.set_fragment_faults(Some(FIRST_ONLY)); + change(&mut a, big_rows).unwrap(); + let small = |s: &Session| s.model().items.iter().all(|(_, t)| t.len() < 1000); + poll_for(&mut [&mut a, &mut b], Duration::from_millis(200), |s| small(&*s[1])); + // Meanwhile on the laptop: a cell edited and undone, and the title. + change(&mut b, |m| m.with_edit(0, "first, edited")).unwrap(); + assert!(b.undo().unwrap()); + change(&mut b, |m| m.with_title("the laptop's")).unwrap(); + assert!(small(&b)); + laptop.set_fragment_faults(None); + until(&mut [&mut a, &mut b], "both have everything", |s| { + s.iter().all(|s| s.model().items.len() == 8 && *s.model().title == "the laptop's" && s.model().texts().contains(&"first".to_owned())) + }); + assert!(b.model().texts().contains(&"local".to_owned())); + assert_eq!(a.model(), b.model()); +} + +/// The same with the change in a tree node's story (a 300 KB paste into a layer's text): meanwhile the other machine +/// renames that node and types at the start of its story. Both keep everything. +#[test] +fn edits_of_the_same_node_and_story_while_a_paste_comes_keep_both() { + setup(); + let hub = Hub::new(); + let (pc, laptop) = (hub.machine("pc").unwrap(), hub.machine("laptop").unwrap()); + let id = ElemId::new(); + let mut start = Doc::empty(); + let mut node = Node::leaf(id, "layer"); + node.text = RichText::plain("abc"); + start.layers = Arc::new(vec![Arc::new(node)]); + let mut a = open::(&pc, "paste.kitdemo", start); + let mut b = open::(&laptop, "paste.kitdemo", Doc::empty()); + until(&mut [&mut a, &mut b], "the laptop has the layer", |s| s[1].model().layers.len() == 1); + laptop.set_fragment_faults(Some(FIRST_ONLY)); + // (300 KB: in fragments, as the test's frames are 64 KiB. Loro's import of a big paste into a text this machine + // edits at the same moment takes time growing with the square of its size, seconds for a megabyte.) + let paste = "p".repeat(300 * 1024); + change(&mut a, |d| Doc { + layers: Arc::new(model::update(&d.layers, id, &|n| Node { text: RichText::plain(&format!("abc{paste}")), ..n.clone() })), + ..d.clone() + }) + .unwrap(); + let short = |s: &Session| s.model().layers.iter().all(|n| n.text.text.len() < 1000); + poll_for(&mut [&mut a, &mut b], Duration::from_millis(200), |s| short(&*s[1])); + change(&mut b, |d| { + let renamed = model::rename(&d.layers, id, "renamed"); + Doc { layers: Arc::new(model::update(&renamed, id, &|n| Node { text: model::type_at(&n.text, 0, "hello "), ..n.clone() })), ..d.clone() } + }) + .unwrap(); + assert!(short(&b)); + laptop.set_fragment_faults(None); + let want = format!("hello abc{paste}"); + until(&mut [&mut a, &mut b], "both have the paste, the name and the typing", |s| { + s.iter().all(|s| s.model().layers.first().is_some_and(|n| n.name == "renamed" && n.text.text == want)) + }); +} + +/// A save while a change comes in fragments (its first one in): the record names only what the model holds. +#[test] +fn a_save_while_a_change_comes_records_what_the_model_holds() { + setup(); + let hub = Hub::new(); + let (pc, laptop) = (hub.machine("pc").unwrap(), hub.machine("laptop").unwrap()); + let mut a = open::(&pc, "save.txt", Notes::new("t", &["first"])); + let mut b = open::(&laptop, "save.txt", Notes::default()); + until(&mut [&mut a, &mut b], "the laptop has the document", |s| s[1].model().texts() == ["first"]); + let before = b.save_point().unwrap().frontiers().to_vec(); + laptop.set_fragment_faults(Some(FIRST_ONLY)); + change(&mut a, big_rows).unwrap(); + poll_for(&mut [&mut a, &mut b], Duration::from_millis(200), |s| s[1].model().items.len() == 1); + let during = b.save_point().unwrap(); + assert_eq!(during.frontiers(), before.as_slice(), "the record of a save made now names nothing of the change"); + assert_eq!(during.model().items.len(), 1); + assert!(!b.file_behind()); +} + +/// The pc's agent stops after the first fragment of a 1 MB change (it crashed part of the way): no machine gets any +/// of it, and nothing waits for it, so a small change from a third machine reaches the laptop at once. Once the pc's +/// agent is back, the pc's session sends the change again, and it comes whole. +#[test] +fn a_sender_stopped_part_of_the_way_holds_nothing_up_and_sends_again() { + setup(); + let hub = Hub::new(); + let (pc, laptop, tablet) = (hub.machine("pc").unwrap(), hub.machine("laptop").unwrap(), hub.machine("tablet").unwrap()); + let mut a = open::(&pc, "gone.txt", Notes::new("t", &["first"])); + let mut b = open::(&laptop, "gone.txt", Notes::default()); + let mut c = open::(&tablet, "gone.txt", Notes::default()); + until(&mut [&mut a, &mut b, &mut c], "all have the document", |s| s.iter().all(|s| s.model().texts() == ["first"])); + pc.drop_connections_after(1); + change(&mut a, big_rows).unwrap(); + pc.restart(); + poll_for(&mut [&mut b, &mut c], Duration::from_millis(300), |s| s.iter().all(|s| s.model().items.len() == 1)); + let sent = Instant::now(); + change(&mut c, |m| m.with_title("from the tablet")).unwrap(); + until(&mut [&mut b, &mut c], "the tablet's change on the laptop", |s| *s[0].model().title == "from the tablet"); + assert!(sent.elapsed() < Duration::from_secs(2), "a later change waited {:?}", sent.elapsed()); + assert_eq!(b.model().items.len(), 1, "nothing of the pc's change"); + pc.start(); + a.reconnect_now(); + until(&mut [&mut a, &mut b, &mut c], "the pc's change whole on every machine", |s| { + s.iter().all(|s| s.model().items.len() == 7 && *s.model().title == "from the tablet") + }); +} + +/// The laptop's agent restarts with only the first fragment of a 1 MB change on its way to the app: the session +/// opens the document again, and its state brings the change whole. +#[test] +fn a_receiver_restarted_part_of_the_way_gets_the_change_whole() { + setup(); + let hub = Hub::new(); + let (pc, laptop) = (hub.machine("pc").unwrap(), hub.machine("laptop").unwrap()); + let mut a = open::(&pc, "back.txt", Notes::new("t", &["first"])); + let mut b = open::(&laptop, "back.txt", Notes::default()); + until(&mut [&mut a, &mut b], "the laptop has the document", |s| s[1].model().texts() == ["first"]); + laptop.set_fragment_faults(Some(FIRST_ONLY)); + change(&mut a, big_rows).unwrap(); + poll_for(&mut [&mut a, &mut b], Duration::from_millis(100), |s| s[1].model().items.len() == 1); + laptop.restart(); + laptop.set_fragment_faults(None); + laptop.start(); + b.reconnect_now(); + until(&mut [&mut a, &mut b], "the laptop has the change whole", |s| { + let n = s[1].model().items.len(); + assert!(n == 1 || n == 7, "the laptop held part of the change"); + n == 7 + }); + assert_eq!(a.model(), b.model()); +} diff --git a/crates/kit/tests/kit_sim.rs b/crates/kit/tests/kit_sim.rs index 0c2c369..31f0208 100644 --- a/crates/kit/tests/kit_sim.rs +++ b/crates/kit/tests/kit_sim.rs @@ -1,8 +1,13 @@ //! The kit simulation: app sessions on three machines of one fake hub (two apps on the third), each making random -//! edits of every shape (tree, rich text, table, timeline, opaque), undoing and redoing, saving, while machines go -//! offline, drop their apps' connections and restart, and the live relay goes down while files still sync. Healed, -//! every session ends with the same model, every model is one the app could make (normalised), and the file catches up -//! with the autosave rule within a save per window. +//! edits of every shape (tree, rich text, table, timeline, opaque; stories and tables nested in layers, a table whose +//! cells are maps with stories), undoing and redoing, saving, while machines go +//! offline, drop their apps' connections and restart, and the live relay goes down while files still sync. Now and +//! then a big change (a paste of 30 to 100 KB, a fill of the grid that can be undone) goes as a bundle of slices, in +//! fragments past the frame limit (16 KiB here, 1 MiB for real); fragments on their way to the apps are dropped or +//! held back (out of order), and an agent +//! sometimes stops part of the way through taking one. Healed, every session ends with the same model, every model is +//! one the app could make (normalised), every big paste is whole on every session, and the file catches up with the +//! autosave rule within a save per window. //! //! `SIM_SEEDS=5000 SIM_FIRST=0 cargo test --release -p craft-cloud-kit --test kit_sim` runs seeds 0 to 4999 on //! `SIM_JOBS` threads (default 4); `SIM_SEED=17` runs one. A seed's history is random but not replayed exactly: the @@ -16,18 +21,29 @@ use std::sync::atomic::{AtomicU64, Ordering}; use std::time::{Duration, Instant}; use common::{change, open}; -use craft_cloud_app::fake::{Fake, Hub}; +use craft_cloud_app::fake::{Fake, FragmentFaults, Hub}; use craft_cloud_kit::{KitError, Session}; use model::{Doc, Rng, SHAPES, TestApp, edit, valid}; const PATH: &str = "sim/story.kitdemo"; +/// Over every seed run: the big changes made, and the times a machine's link stopped inside one (it slept part of the +/// way through). Both must happen for the run to say anything. +static BIG_CHANGES: AtomicU64 = AtomicU64::new(0); +static PAUSES: AtomicU64 = AtomicU64::new(0); const STEPS: usize = 80; /// Which machine each session runs on: the third machine has two apps open on the document. const ON: [usize; 4] = [0, 1, 2, 2]; fn run_seed(seed: u64) { + // (Frames of 16 KiB: the big changes here, of tens of KB, go in fragments as changes over 1 MiB do.) + craft_cloud_app::live::set_frame_limit_for_tests(16 * 1024); let hub = Hub::new(); let fakes: Vec = (0..3).map(|i| hub.machine(&format!("m{i}")).unwrap()).collect(); + // A machine's link stops inside a change in fragments: now and then for a long sleep, more often a few steps. The + // machine keeps what came, and its apps get the change whole once the rest is there (as the agent does). + for (i, f) in (0u64..).zip(&fakes) { + f.set_fragment_faults(Some(FragmentFaults { seed: seed.wrapping_mul(7).wrapping_add(i), sleep: 150, delay: 300, max_delay: 4 })); + } let mut rng = Rng(seed); let start = model::random_doc(&mut rng, 12); let mut sessions: Vec> = Vec::new(); @@ -35,9 +51,42 @@ fn run_seed(seed: u64) { sessions.push(open::(&fakes[*m], PATH, if k == 0 { start.clone() } else { Doc::empty() })); } let mut stopped = [false; 3]; + // The big pastes made (their layer and text): every session has each of them at the end. + let mut bigs = Vec::new(); for step in 0..STEPS { let i = rng.below(sessions.len()); let m = ON[i]; + // Now and then a big change (in fragments past 1 MiB): a paste no edit touches again, or a fill that can be + // undone; sometimes the agent stops part of the way through taking it. + if rng.chance(4) { + BIG_CHANGES.fetch_add(1, Ordering::Relaxed); + if rng.chance(30) { + fakes[m].drop_connections_after(rng.below(3)); + } + if rng.chance(60) { + let mut r = rng.clone(); + let old = sessions[i].model().clone(); + let (new, id, text) = model::big_paste(&old, &mut r); + rng = r; + match sessions[i].background_change(&old, &std::sync::Arc::new(new)) { + Ok(()) => bigs.push((id, text)), + Err(KitError::Stalled(_)) => {} + Err(e) => panic!("seed {seed} step {step}: session {i}'s big paste failed: {e}"), + } + } else { + let mut r = rng.clone(); + match change(&mut sessions[i], |d| model::big_fill(d, &mut r)) { + Ok(()) | Err(KitError::Stalled(_)) => {} + Err(e) => panic!("seed {seed} step {step}: session {i}'s big fill failed: {e}"), + } + rng = r; + } + for s in &mut sessions { + s.poll(); + } + hub.tick(); + continue; + } match rng.below(24) { 0..=11 => { let shape = SHAPES[rng.below(SHAPES.len())]; @@ -77,13 +126,19 @@ fn run_seed(seed: u64) { for s in &mut sessions { s.poll(); } + hub.tick(); } - // Healed: every machine runs and is online, the live relay is up, and every session tries at once. + // Healed: every machine runs and is online, the live relay is up, no fragment is lost or held, and every session + // tries at once. hub.set_live(true); for f in &fakes { + f.set_fragment_faults(None); f.start(); f.set_online(true); } + for _ in 0..8 { + hub.tick(); + } for s in &mut sessions { s.reconnect_now(); } @@ -105,6 +160,7 @@ fn run_seed(seed: u64) { } std::thread::sleep(Duration::from_millis(2)); } + PAUSES.fetch_add(fakes.iter().map(Fake::pauses).sum(), Ordering::Relaxed); // The file catches up on its own: one window saves, then each saves while its file is behind (as an app's // autosave does), and soon none is; the file then has the version every session has. if let Ok(point) = sessions[0].save_point() { @@ -143,6 +199,11 @@ fn run_seed(seed: u64) { if let Err(why) = valid(s.model()) { panic!("seed {seed}: session {i}'s model isn't one the app could make: {why}"); } + // Nothing a machine made is lost: every big paste is there, whole. + for (id, text) in &bigs { + let found = s.model().layers.iter().find(|n| n.id == *id); + assert!(found.is_some_and(|n| n.text.text == *text), "seed {seed}: session {i} lost the big paste {id}"); + } assert!(s.status().read_only.is_none()); // The model the session built change by change (texts patched from Loro's deltas) is the one its document // reads back whole. @@ -190,4 +251,9 @@ fn kit_sessions_converge() { if let Some((seed, message)) = failed.into_inner().unwrap() { panic!("seed {seed} failed (SIM_SEED={seed} replays it, as near as the clock allows): {message}"); } + let (big, pauses) = (BIG_CHANGES.load(Ordering::Relaxed), PAUSES.load(Ordering::Relaxed)); + eprintln!("kit sim: {seeds} seeds, {big} big changes, {pauses} stops inside one"); + if seeds >= 16 { + assert!(big > 0 && pauses > 0, "the run made {big} big changes and stopped inside {pauses}: it tested nothing of them"); + } } diff --git a/crates/kit/tests/mappings.rs b/crates/kit/tests/mappings.rs index f55eb93..8d332d5 100644 --- a/crates/kit/tests/mappings.rs +++ b/crates/kit/tests/mappings.rs @@ -8,7 +8,7 @@ mod model; use std::sync::Arc; use craft_cloud_kit::raw::Replica; -use craft_cloud_kit::{RichText, Value}; +use craft_cloud_kit::{ElemId, RichText, Value}; use model::{Doc, Grid, Item, Key, Node, Rng, SHAPES, Shape, TestApp, Timeline, edit, valid}; use proptest::prelude::*; @@ -38,7 +38,7 @@ proptest! { /// Model to Loro and back: an import, then every change after it, read back exactly. #[test] - fn every_shape_round_trips(seed in any::(), shape in 0usize..6) { + fn every_shape_round_trips(seed in any::(), shape in 0usize..8) { let mut rng = Rng(seed); let mut doc = match SHAPES.get(shape) { Some(s) => shape_doc(&mut rng, *s, 12), @@ -49,7 +49,7 @@ proptest! { r.reindex(TREES); prop_assert_eq!(&settle(&mut r), &doc); for _ in 0..12 { - let s = SHAPES.get(shape).copied().unwrap_or(SHAPES[rng.below(5)]); + let s = SHAPES.get(shape).copied().unwrap_or(SHAPES[rng.below(SHAPES.len())]); let next = edit(&doc, s, &mut rng); r.write::(Some(&doc), &next).unwrap(); doc = settle(&mut r); @@ -59,7 +59,7 @@ proptest! { /// Two replicas from one import, random edits on both, then each takes the other's: both end equal, and valid. #[test] - fn every_shape_merges(seed in any::(), shape in 0usize..6) { + fn every_shape_merges(seed in any::(), shape in 0usize..8) { let mut rng = Rng(seed); let start = model::random_doc(&mut rng, 15); let mut a = Replica::default(); @@ -71,13 +71,13 @@ proptest! { for _round in 0..3 { let (mut out_a, mut out_b) = (Vec::new(), Vec::new()); for _ in 0..rng.below(6) { - let s = SHAPES.get(shape).copied().unwrap_or(SHAPES[rng.below(5)]); + let s = SHAPES.get(shape).copied().unwrap_or(SHAPES[rng.below(SHAPES.len())]); let next = edit(&da, s, &mut rng); out_a.extend(a.write::(Some(&da), &next).unwrap().0); da = next; } for _ in 0..rng.below(6) { - let s = SHAPES.get(shape).copied().unwrap_or(SHAPES[rng.below(5)]); + let s = SHAPES.get(shape).copied().unwrap_or(SHAPES[rng.below(SHAPES.len())]); let next = edit(&db, s, &mut rng); out_b.extend(b.write::(Some(&db), &next).unwrap().0); db = next; @@ -119,7 +119,7 @@ fn unchanged_parts_write_nothing() { let doc = { let mut d = doc; if model::flatten(&d.layers).is_empty() { - d.layers = Arc::new(vec![Arc::new(Node { id: rng.id(), name: "l".into(), children: Vec::new() })]); + d.layers = Arc::new(vec![Arc::new(Node::leaf(rng.id(), "l"))]); } d }; @@ -133,6 +133,7 @@ fn unchanged_parts_write_nothing() { grid: Arc::new((*doc.grid).clone()), timeline: Arc::new((*doc.timeline).clone()), settings: Arc::new((*doc.settings).clone()), + cards: Arc::new((*doc.cards).clone()), words: doc.words, }; assert_eq!(ops(&mut r, &doc, ©), 0, "the same content writes nothing"); @@ -161,6 +162,160 @@ fn unchanged_parts_write_nothing() { let doc = celled; let settings = Doc { settings: Arc::new(vec![1, 2, 3]), ..doc.clone() }; assert_eq!(ops(&mut r, &doc, &settings), 1, "one value"); + let doc = settings; + // A keystroke in a layer's own story, and one in a card of a layer's nested table. + let typed = + Doc { layers: Arc::new(model::update(&doc.layers, first, &|n| Node { text: model::type_at(&n.text, 0, "Q"), ..n.clone() })), ..doc.clone() }; + assert_eq!(ops(&mut r, &doc, &typed), 1, "a keystroke in a node's story is one insert"); + let doc = typed; + let carded = Doc { + layers: Arc::new(model::update(&doc.layers, first, &|n| Node { cards: Some(Arc::new(model::one_card(&mut Rng(9), "c"))), ..n.clone() })), + ..doc.clone() + }; + r.write::(Some(&doc), &carded).unwrap(); + let doc = carded; + let in_card = Doc { + layers: Arc::new(model::update(&doc.layers, first, &|n| { + let mut c = (**n.cards.as_ref().unwrap()).clone(); + let card = c.cells.values_mut().next().unwrap(); + card.text = model::type_at(&card.text, 1, "!"); + Node { cards: Some(Arc::new(c)), ..n.clone() } + })), + ..doc.clone() + }; + assert_eq!(ops(&mut r, &doc, &in_card), 1, "a keystroke in a nested cell's story is one insert"); + // The same document again in new allocations writes nothing, stories and nested tables included. + let copy = Doc { layers: Arc::new(in_card.layers.iter().map(|n| Arc::new((**n).clone())).collect()), ..in_card.clone() }; + assert_eq!(ops(&mut r, &in_card, ©), 0); +} + +fn layer(id: ElemId, text: &str) -> Arc { + Arc::new(Node { text: RichText::plain(text), ..Node::leaf(id, "layer") }) +} + +fn layer_text(doc: &Doc, id: ElemId) -> Option { + model::find(&doc.layers, id).map(|n| n.text.text.clone()) +} + +fn type_in_layer(doc: &Doc, id: ElemId, at: usize, s: &str) -> Doc { + Doc { layers: Arc::new(model::update(&doc.layers, id, &|n| Node { text: model::type_at(&n.text, at, s), ..n.clone() })), ..doc.clone() } +} + +/// Two machines typing in one layer's story at once (a shape's text): it merges character by character, both words +/// whole, each where it was typed; marks made meanwhile on one side cover the same characters on both. +#[test] +fn typing_in_one_nodes_story_at_once_merges() { + let mut rng = Rng(21); + let id = rng.id(); + let (mut a, mut b, doc) = pair(&Doc { layers: Arc::new(vec![layer(id, "abcd")]), ..Doc::empty() }); + a.write::(Some(&doc), &type_in_layer(&doc, id, 1, "XY")).unwrap(); + b.write::(Some(&doc), &type_in_layer(&doc, id, 3, "12")).unwrap(); + let merged = exchange(&mut a, &mut b); + assert_eq!(layer_text(&merged, id).as_deref(), Some("aXYbc12d")); + // At one spot at once: both words, not interleaved. + a.write::(Some(&merged), &type_in_layer(&merged, id, 0, "PQR")).unwrap(); + b.write::(Some(&merged), &type_in_layer(&merged, id, 0, "789")).unwrap(); + let merged = exchange(&mut a, &mut b); + let text = layer_text(&merged, id).unwrap(); + assert!(text == "PQR789aXYbc12d" || text == "789PQRaXYbc12d", "{text}"); + // A bolds "bc" while B types inside it. + let bold = Doc { + layers: Arc::new(model::update(&merged.layers, id, &|n| Node { + text: model::format(&n.text, 9..11, "b", Some(Value::Bool(true))), + ..n.clone() + })), + ..merged.clone() + }; + a.write::(Some(&merged), &bold).unwrap(); + b.write::(Some(&merged), &type_in_layer(&merged, id, 10, "_")).unwrap(); + let merged = exchange(&mut a, &mut b); + let n = model::find(&merged.layers, id).unwrap(); + let bolded: String = n.text.text.char_indices().filter(|(i, _)| n.text.attrs_at(*i).contains_key("b")).map(|(_, c)| c).collect(); + assert_eq!(bolded, "b_c", "the bold covers what was typed inside it: {:?}", n.text); +} + +/// A layer deleted on one machine while the other types in its story: deleted on both, nothing left of it. +#[test] +fn a_node_deleted_while_typed_in_is_gone_on_both() { + let mut rng = Rng(22); + let (id, other) = (rng.id(), rng.id()); + let (mut a, mut b, doc) = pair(&Doc { layers: Arc::new(vec![layer(id, "hello"), layer(other, "stays")]), ..Doc::empty() }); + a.write::(Some(&doc), &Doc { layers: Arc::new(model::remove(&doc.layers, id).0), ..doc.clone() }).unwrap(); + let mut typed = doc.clone(); + for (k, c) in ["x", "y", "z"].into_iter().enumerate() { + let next = type_in_layer(&typed, id, 5 + k, c); + b.write::(Some(&typed), &next).unwrap(); + typed = next; + } + let merged = exchange(&mut a, &mut b); + assert!(model::find(&merged.layers, id).is_none(), "the layer is gone on both"); + assert_eq!(layer_text(&merged, other).as_deref(), Some("stays")); + // And it stays gone after more typing elsewhere. + let next = type_in_layer(&merged, other, 0, "!"); + b.write::(Some(&merged), &next).unwrap(); + let merged = exchange(&mut a, &mut b); + assert_eq!(model::flatten(&merged.layers).len(), 1); +} + +fn type_in_card(cards: &model::Cards, cell: (ElemId, ElemId), at: usize, s: &str) -> model::Cards { + let mut c = cards.clone(); + if let Some(card) = c.cells.get_mut(&cell) { + card.text = model::type_at(&card.text, at, s); + } + c +} + +/// A table whose cells are maps with stories: typing in one cell's story at once merges; a row added on one machine +/// while a column is added on the other leaves one cell nobody made, which both machines' `normalise` make alike; a +/// row deleted while its cell is typed in goes with its cells. The same in a table nested in a layer. +#[test] +fn stories_in_cells_merge() { + let mut rng = Rng(23); + let cards = model::one_card(&mut rng, "ab"); + let cell = *cards.cells.keys().next().unwrap(); + let id = rng.id(); + let start = Doc { + cards: Arc::new(cards.clone()), + layers: Arc::new(vec![Arc::new(Node { cards: Some(Arc::new(cards.clone())), ..Node::leaf(id, "with cards") })]), + ..Doc::empty() + }; + let (mut a, mut b, doc) = pair(&start); + // Typing at once in the root table's cell, and in the nested one's. + let nested = |d: &Doc, f: &dyn Fn(&model::Cards) -> model::Cards| Doc { + layers: Arc::new(model::update(&d.layers, id, &|n| Node { cards: n.cards.as_deref().map(|c| Arc::new(f(c))), ..n.clone() })), + ..d.clone() + }; + let da = nested(&Doc { cards: Arc::new(type_in_card(&doc.cards, cell, 1, "X")), ..doc.clone() }, &|c| type_in_card(c, cell, 0, "<")); + let db = nested(&Doc { cards: Arc::new(type_in_card(&doc.cards, cell, 2, "Y")), ..doc.clone() }, &|c| type_in_card(c, cell, 2, ">")); + a.write::(Some(&doc), &da).unwrap(); + b.write::(Some(&doc), &db).unwrap(); + let merged = exchange(&mut a, &mut b); + assert_eq!(merged.cards.cells[&cell].text.text, "aXbY"); + let inner = model::find(&merged.layers, id).unwrap().cards.clone().unwrap(); + assert_eq!(inner.cells[&cell].text.text, ""); + // A row on one machine, a column on the other. + let mut ca = (*merged.cards).clone(); + let r = rng.id(); + ca.rows.push(r); + ca.cells.insert((r, cell.1), model::Card::default()); + let mut cb = (*merged.cards).clone(); + let c = rng.id(); + cb.cols.push(c); + cb.cells.insert((cell.0, c), model::Card { n: 7, text: RichText::plain("new") }); + a.write::(Some(&merged), &Doc { cards: Arc::new(ca), ..merged.clone() }).unwrap(); + b.write::(Some(&merged), &Doc { cards: Arc::new(cb), ..merged.clone() }).unwrap(); + let merged = exchange(&mut a, &mut b); + assert_eq!((merged.cards.rows.len(), merged.cards.cols.len(), merged.cards.cells.len()), (2, 2, 4), "the missing card is made"); + assert_eq!(merged.cards.cells[&(cell.0, c)].text.text, "new"); + assert_eq!(merged.cards.cells[&(r, c)], model::Card::default()); + // The new row deleted on A while B types in its card. + let mut ca = (*merged.cards).clone(); + ca.rows.retain(|x| *x != r); + ca.cells.retain(|(x, _), _| *x != r); + a.write::(Some(&merged), &Doc { cards: Arc::new(ca), ..merged.clone() }).unwrap(); + b.write::(Some(&merged), &Doc { cards: Arc::new(type_in_card(&merged.cards, (r, c), 0, "lost")), ..merged.clone() }).unwrap(); + let merged = exchange(&mut a, &mut b); + assert_eq!((merged.cards.rows.len(), merged.cards.cells.len()), (1, 2)); } /// Two replicas of `doc`. @@ -272,8 +427,7 @@ fn rows_inserted_at_once_are_both_kept() { fn moves_that_would_make_a_cycle_end_in_one_tree() { let mut rng = Rng(11); let (x, y) = (rng.id(), rng.id()); - let layers = - vec![Arc::new(Node { id: x, name: "x".into(), children: Vec::new() }), Arc::new(Node { id: y, name: "y".into(), children: Vec::new() })]; + let layers = vec![Arc::new(Node::leaf(x, "x")), Arc::new(Node::leaf(y, "y"))]; let (mut a, mut b, doc) = pair(&Doc { layers: Arc::new(layers), ..Doc::empty() }); let (rest, taken) = model::remove(&doc.layers, x); let a_moves = Doc { layers: Arc::new(model::insert(&rest, Some(y), 0, taken.unwrap())), ..doc.clone() }; diff --git a/crates/kit/tests/model/mod.rs b/crates/kit/tests/model/mod.rs index 5ad551f..cfef775 100644 --- a/crates/kit/tests/model/mod.rs +++ b/crates/kit/tests/model/mod.rs @@ -49,11 +49,168 @@ impl Rng { pub struct Node { pub id: ElemId, pub name: String, + /// The layer's own text: a story nested in its node. + pub text: RichText, + /// A table of cards on the layer (map cells with stories), nested in its node. + pub cards: Option>, pub children: Vec>, } +impl Node { + /// A layer with a name and nothing else. + pub fn leaf(id: ElemId, name: &str) -> Self { + Self { id, name: name.to_owned(), text: RichText::plain(""), cards: None, children: Vec::new() } + } +} + pub type Roots = Arc>>; +// The cards: a table whose cells are property maps with a story each (`n`, and the story `t`). + +#[derive(Debug, Clone, PartialEq)] +pub struct Card { + pub n: i64, + pub text: RichText, +} + +impl Default for Card { + fn default() -> Self { + Self { n: 0, text: RichText::plain("") } + } +} + +#[derive(Debug, Clone, PartialEq, Default)] +pub struct Cards { + pub rows: Vec, + pub cols: Vec, + /// One for every row and column. + pub cells: BTreeMap<(ElemId, ElemId), Card>, +} + +pub struct CardsSrc<'a>(pub &'a Arc); + +impl TableSource for CardsSrc<'_> { + fn key(&self) -> Option { + Some(Arc::as_ptr(self.0) as usize) + } + fn rows(&self) -> Vec { + self.0.rows.clone() + } + fn columns(&self) -> Vec { + self.0.cols.clone() + } + fn cell(&self, _row: ElemId, _column: ElemId) -> Option { + None + } + fn cells(&self) -> Vec<(ElemId, ElemId, Value)> { + Vec::new() + } + fn cell_maps(&self) -> bool { + true + } + fn write_cell(&self, row: ElemId, column: ElemId, props: &mut Props<'_, '_>) -> Result<(), KitError> { + let empty = Card::default(); + let card = self.0.cells.get(&(row, column)).unwrap_or(&empty); + props.set("n", card.n)?; + props.text("t", &card.text) + } +} + +/// The cards as stored (cells a merge left out are missing: `normalise` makes them). +pub fn read_cards(view: &craft_cloud_kit::TableView<'_>, old: Option<&Cards>) -> Cards { + let (rows, cols) = (view.rows(), view.columns()); + let mut cells = BTreeMap::new(); + for r in &rows { + for c in &cols { + let Some(p) = view.cell_props(*r, *c) else { continue }; + let before = old.and_then(|o| o.cells.get(&(*r, *c))).map(|card| &card.text as &dyn TextSource); + let text = view.cell_text(*r, *c, "t", before).unwrap_or_else(|| RichText::plain("")); + cells.insert((*r, *c), Card { n: p.i64("n").unwrap_or(0), text }); + } + } + Cards { rows, cols, cells } +} + +/// The cards repaired: a card for every row and column (a plain one where a merge left none), none of rows or columns +/// that are gone, canonical texts. `None` when they are fine already. +pub fn repair_cards(c: &Cards) -> Option { + let complete = c.rows.iter().all(|r| c.cols.iter().all(|col| c.cells.contains_key(&(*r, *col)))); + let tidy = c.cells.keys().all(|(r, col)| c.rows.contains(r) && c.cols.contains(col)); + let canonical_texts = c.cells.values().all(|card| runs_canonical(&card.text)); + if complete && tidy && canonical_texts { + return None; + } + let mut out = c.clone(); + out.cells.retain(|(r, col), _| c.rows.contains(r) && c.cols.contains(col)); + for r in &c.rows { + for col in &c.cols { + out.cells.entry((*r, *col)).or_default(); + } + } + for card in out.cells.values_mut() { + if !runs_canonical(&card.text) { + card.text = canonical(&card.text); + } + } + Some(out) +} + +pub fn edit_cards(c: &Cards, rng: &mut Rng) -> Cards { + let mut c = c.clone(); + match rng.below(6) { + 0 => { + let (at, r) = (rng.below(c.rows.len() + 1), rng.id()); + c.rows.insert(at, r); + for col in &c.cols { + c.cells.insert((r, *col), Card::default()); + } + } + 1 => { + let (at, col) = (rng.below(c.cols.len() + 1), rng.id()); + c.cols.insert(at, col); + for r in &c.rows { + c.cells.insert((*r, col), Card::default()); + } + } + 2 if c.rows.len() > 1 => { + let r = c.rows.remove(rng.below(c.rows.len())); + c.cells.retain(|(row, _), _| *row != r); + } + 3 if c.cols.len() > 1 => { + let col = c.cols.remove(rng.below(c.cols.len())); + c.cells.retain(|(_, cc), _| *cc != col); + } + _ if !c.cells.is_empty() => { + let keys: Vec<(ElemId, ElemId)> = c.cells.keys().copied().collect(); + let k = keys[rng.below(keys.len())]; + if let Some(card) = c.cells.get_mut(&k) { + if rng.chance(30) { + card.n += 1; + } else { + card.text = edit_text(&card.text, rng); + } + } + } + _ => { + let (r, col) = (rng.id(), rng.id()); + c.rows.push(r); + c.cols.push(col); + for row in &c.rows { + for cc in &c.cols { + c.cells.entry((*row, *cc)).or_default(); + } + } + } + } + c +} + +/// Cards with one row, one column and one card. +pub fn one_card(rng: &mut Rng, text: &str) -> Cards { + let (r, c) = (rng.id(), rng.id()); + Cards { rows: vec![r], cols: vec![c], cells: [((r, c), Card { n: 1, text: RichText::plain(text) })].into() } +} + pub struct LayersSrc<'a> { roots: &'a Roots, index: HashMap>, @@ -87,11 +244,31 @@ impl TreeSource for LayersSrc<'_> { fn write(&self, id: ElemId, props: &mut Props<'_, '_>) -> Result<(), KitError> { if let Some(n) = self.index.get(&id) { props.set("name", n.name.as_str())?; + props.text("text", &n.text)?; + if let Some(cards) = &n.cards { + props.table("cards", &CardsSrc(cards))?; + } } Ok(()) } } +/// `roots` with the node `id` replaced by what `f` makes of it (its ancestors copied, everything else shared). +pub fn update(roots: &[Arc], id: ElemId, f: &dyn Fn(&Node) -> Node) -> Vec> { + roots + .iter() + .map(|n| { + if n.id == id { + Arc::new(f(n)) + } else if contains(n, id) { + Arc::new(Node { children: update(&n.children, id, f), ..(**n).clone() }) + } else { + n.clone() + } + }) + .collect() +} + /// Every node: id, parent, depth-first. pub fn flatten(roots: &[Arc]) -> Vec<(ElemId, Option)> { let mut out = Vec::new(); @@ -117,7 +294,7 @@ pub fn remove(roots: &[Arc], id: ElemId) -> (Vec>, Option], parent: Option, at: usize, node: Arc< if n.id == p { let mut children = n.children.clone(); children.insert(at.min(children.len()), node.clone()); - Arc::new(Node { id: n.id, name: n.name.clone(), children }) + Arc::new(Node { children, ..(**n).clone() }) } else if contains(n, p) { - Arc::new(Node { id: n.id, name: n.name.clone(), children: insert(&n.children, parent, at, node.clone()) }) + Arc::new(Node { children: insert(&n.children, parent, at, node.clone()), ..(**n).clone() }) } else { n.clone() } @@ -151,18 +328,7 @@ pub fn insert(roots: &[Arc], parent: Option, at: usize, node: Arc< } pub fn rename(roots: &[Arc], id: ElemId, name: &str) -> Vec> { - roots - .iter() - .map(|n| { - if n.id == id { - Arc::new(Node { id, name: name.to_owned(), children: n.children.clone() }) - } else if contains(n, id) { - Arc::new(Node { id: n.id, name: n.name.clone(), children: rename(&n.children, id, name) }) - } else { - n.clone() - } - }) - .collect() + update(roots, id, &|n| Node { name: name.to_owned(), ..n.clone() }) } pub fn find(roots: &[Arc], id: ElemId) -> Option<&Arc> { @@ -399,6 +565,8 @@ pub struct Doc { pub grid: Arc, pub timeline: Arc, pub settings: Arc>, + /// A root table of map cells with stories. + pub cards: Arc, /// Derived from the story, never synced. pub words: usize, } @@ -429,20 +597,33 @@ impl LiveDocument for TestApp { .table("grid", GridSrc(&doc.grid)) .timeline("timeline", TimelineSrc(&doc.timeline)) .opaque("settings", Opaque { value: Some(&*doc.settings), key: Some(Arc::as_ptr(&doc.settings) as usize) }) + .table("cards", CardsSrc(&doc.cards)) } fn apply_remote(old: &Arc, patch: &RemotePatch<'_>) -> Result, KitError> { let mut new = (**old).clone(); if let Some(tree) = patch.tree("layers") { - fn build(tree: &craft_cloud_kit::TreeView<'_>, ids: Vec) -> Vec> { + fn build(tree: &craft_cloud_kit::TreeView<'_>, ids: Vec, old: &HashMap>) -> Vec> { ids.into_iter() .map(|id| { - let name = tree.props(id).and_then(|p| p.str("name")).unwrap_or_default(); - Arc::new(Node { id, name, children: build(tree, tree.children(id)) }) + let props = tree.props(id); + let name = props.as_ref().and_then(|p| p.str("name")).unwrap_or_default(); + let before = old.get(&id); + // The node's story patched from what the model had (only what came is applied). + let text = tree.text_patched(id, "text", before.map(|n| &n.text as &dyn TextSource)).unwrap_or_else(|| RichText::plain("")); + let cards = props + .as_ref() + .and_then(|p| p.table("cards")) + .map(|view| Arc::new(read_cards(&view, before.and_then(|n| n.cards.as_deref())))); + Arc::new(Node { id, name, text, cards, children: build(tree, tree.children(id), old) }) }) .collect() } - new.layers = Arc::new(build(&tree, tree.roots())); + let index = LayersSrc::new(&old.layers).index; + new.layers = Arc::new(build(&tree, tree.roots(), &index)); + } + if let Some(view) = patch.table("cards") { + new.cards = Arc::new(read_cards(&view, Some(&old.cards))); } if let Some(story) = patch.text("story") { new.story = Arc::new(story); @@ -488,6 +669,26 @@ impl LiveDocument for TestApp { if !runs_canonical(&doc.story) { doc.story = Arc::new(canonical(&doc.story)); } + // Cards a merge left out (a row and a column added at once) are made, the same on every machine. + if let Some(fixed) = repair_cards(&doc.cards) { + doc.cards = Arc::new(fixed); + } + let mut layers = (*doc.layers).clone(); + for (id, _) in flatten(&doc.layers) { + let Some(n) = find(&layers, id) else { continue }; + let text = (!runs_canonical(&n.text)).then(|| canonical(&n.text)); + let cards = n.cards.as_deref().and_then(repair_cards); + if text.is_some() || cards.is_some() { + layers = update(&layers, id, &|n| Node { + text: text.clone().unwrap_or_else(|| n.text.clone()), + cards: cards.clone().map(Arc::new).or_else(|| n.cards.clone()), + ..n.clone() + }); + } + } + if layers != *doc.layers { + doc.layers = Arc::new(layers); + } } fn refresh_derived(doc: &mut Doc) { @@ -525,6 +726,18 @@ pub fn valid(doc: &Doc) -> Result<(), String> { if doc.words != count_words(&doc.story.text) { return Err("derived words are stale".into()); } + if repair_cards(&doc.cards).is_some() { + return Err("the cards miss a card, or keep one of a missing row or column".into()); + } + for (id, _) in &ids { + let Some(n) = find(&doc.layers, *id) else { continue }; + if canonical(&n.text) != n.text || n.text.paragraphs.len() != n.text.text.matches('\n').count() + 1 { + return Err(format!("layer {id}'s text isn't canonical")); + } + if n.cards.as_deref().is_some_and(|c| repair_cards(c).is_some()) { + return Err(format!("layer {id}'s cards miss a card")); + } + } Ok(()) } @@ -536,9 +749,13 @@ pub enum Shape { Table, Timeline, Opaque, + /// A layer's own story, or its cards (a table nested in its node). + Nested, + /// The root table of map cells with stories. + Cards, } -pub const SHAPES: [Shape; 5] = [Shape::Tree, Shape::Text, Shape::Table, Shape::Timeline, Shape::Opaque]; +pub const SHAPES: [Shape; 7] = [Shape::Tree, Shape::Text, Shape::Table, Shape::Timeline, Shape::Opaque, Shape::Nested, Shape::Cards]; /// A random edit of `shape`, as a command of the app makes it (the result is refreshed). pub fn edit(doc: &Doc, shape: Shape, rng: &mut Rng) -> Doc { @@ -549,18 +766,62 @@ pub fn edit(doc: &Doc, shape: Shape, rng: &mut Rng) -> Doc { Shape::Table => d.grid = Arc::new(edit_grid(&doc.grid, rng)), Shape::Timeline => d.timeline = Arc::new(edit_timeline(&doc.timeline, rng)), Shape::Opaque => d.settings = Arc::new((0..rng.below(20)).map(|_| rng.next() as u8).collect()), + Shape::Nested => d.layers = Arc::new(edit_nested(&doc.layers, rng)), + Shape::Cards => d.cards = Arc::new(edit_cards(&doc.cards, rng)), } TestApp::refresh_derived(&mut d); d } +/// The name every big paste's layer starts with ([`big_paste`]): random edits leave such a layer alone (it is the +/// simulation's check that nothing a machine made is lost). +pub const BIG: &str = "big:"; + +/// The layers random edits may pick: all but the big pastes' (their children are anyone's). +fn editable(roots: &[Arc]) -> Vec<(ElemId, Option)> { + flatten(roots).into_iter().filter(|(id, _)| find(roots, *id).is_none_or(|n| !n.name.starts_with(BIG))).collect() +} + +/// A big paste: a new layer at the top whose story is 30 to 100 KB of text (the change goes as slices in a bundle, and +/// past the frame limit, which the simulation sets to 16 KiB, in fragments). The layer's ID and its text, to look for +/// at the end. +pub fn big_paste(doc: &Doc, rng: &mut Rng) -> (Doc, ElemId, String) { + let id = rng.id(); + let word = format!("{} ", rng.word()); + let len = 30_000 + rng.below(70_000); + let text: String = word.repeat(len / word.len() + 1).chars().take(len).collect(); + let mut node = Node::leaf(id, &format!("{BIG}{id}")); + node.text = RichText::plain(&text); + let mut d = doc.clone(); + d.layers = Arc::new(insert(&doc.layers, None, 0, Arc::new(node))); + TestApp::refresh_derived(&mut d); + (d, id, text) +} + +/// A big change that can be undone: a column and 100 to 300 rows of 300-byte cells added to the grid (30 to 100 KB, in +/// fragments past the simulation's frame limit). +pub fn big_fill(doc: &Doc, rng: &mut Rng) -> Doc { + let mut g = (*doc.grid).clone(); + let col = rng.id(); + g.cols.push(col); + let cell = "c".repeat(300); + for _ in 0..100 + rng.below(200) { + let row = rng.id(); + g.rows.push(row); + g.cells.insert((row, col), cell.clone()); + } + let mut d = doc.clone(); + d.grid = Arc::new(g); + d +} + pub fn edit_tree(roots: &[Arc], rng: &mut Rng) -> Vec> { - let all = flatten(roots); + let all = editable(roots); let pick = |rng: &mut Rng| all.get(rng.below(all.len())).map(|(id, _)| *id); match if all.is_empty() { 0 } else { rng.below(4) } { 0 => { let parent = if rng.chance(40) { None } else { pick(rng) }; - let n = Arc::new(Node { id: rng.id(), name: rng.word(), children: Vec::new() }); + let n = Arc::new(Node::leaf(rng.id(), &rng.word())); let at = rng.below(8); insert(roots, parent, at, n) } @@ -585,6 +846,28 @@ pub fn edit_tree(roots: &[Arc], rng: &mut Rng) -> Vec> { } } +/// An edit inside a layer: its story (typing, formatting), or its cards (made, edited, removed). +pub fn edit_nested(roots: &[Arc], rng: &mut Rng) -> Vec> { + let all = editable(roots); + let Some((id, _)) = all.get(rng.below(all.len())).copied() else { return edit_tree(roots, rng) }; + let Some(node) = find(roots, id) else { return roots.to_vec() }; + let mut n = (**node).clone(); + match rng.below(10) { + 0..=5 => n.text = edit_text(&n.text, rng), + 6 if n.cards.is_some() && rng.chance(30) => n.cards = None, + _ => { + n.cards = Some(Arc::new(match &n.cards { + Some(c) => edit_cards(c, rng), + None => { + let word = rng.word(); + one_card(rng, &word) + } + })) + } + } + update(roots, id, &|_| n.clone()) +} + pub fn edit_text(t: &RichText, rng: &mut Rng) -> RichText { let len = t.text.chars().count(); let range = |rng: &mut Rng| { diff --git a/crates/kit/tests/session.rs b/crates/kit/tests/session.rs index d1b36a7..41650af 100644 --- a/crates/kit/tests/session.rs +++ b/crates/kit/tests/session.rs @@ -8,7 +8,7 @@ use std::time::{Duration, Instant}; use common::{App, Notes, NotesApp, change, open, until}; use craft_cloud_app::fake::Hub; -use craft_cloud_kit::{AgentBlobs, BlobStore, KitError, MAX_QUEUE_BYTES, Open, SPLIT_BYTES, Session}; +use craft_cloud_kit::{AgentBlobs, BlobStore, KitError, MAX_QUEUE_BYTES, NotLive, Open, Session}; #[test] fn a_file_imported_on_one_machine_is_live_on_the_other() { @@ -67,37 +67,6 @@ fn a_document_from_a_newer_app_is_read_only() { change(&mut v2, |m| m.with_title("migrated")).unwrap(); } -/// A change over the split size goes as several commits, each under it: a 5 MB change (twenty 250 KiB notes) arrives -/// whole on the other machine, and every update stays under the agent's 1 MiB cap. -#[test] -fn a_five_megabyte_change_arrives_whole() { - let hub = Hub::new(); - let (pc, laptop) = (hub.machine("pc").unwrap(), hub.machine("laptop").unwrap()); - let mut a = open::(&pc, "big.txt", Notes::new("big", &[])); - let mut b = open::(&laptop, "big.txt", Notes::default()); - let before = hub.updates("big.txt"); - let notes: Vec = (0..20).map(|i| format!("{i:02}{}", "x".repeat(250 * 1024))).collect(); - change(&mut a, |m| { - let mut items = (*m.items).clone(); - let mut hint = Vec::new(); - for n in ¬es { - let id = craft_cloud_kit::ElemId::new(); - hint.push(id); - items.push((id, n.clone())); - } - Notes { items: Arc::new(items), hint: Some(hint), ..m.clone() } - }) - .unwrap(); - until(&mut [&mut a, &mut b], "the laptop has all 5 MB", |s| s.get(1).is_some_and(|b| b.model().texts() == notes)); - let sent = hub.updates("big.txt") - before; - assert!(sent >= 10, "{sent} updates for 5 MB: it was split"); - assert!(a.status().error.is_none() && b.status().error.is_none(), "{:?} {:?}", a.status().error, b.status().error); - const { assert!(SPLIT_BYTES * 2 <= craft_cloud_api::limits::MAX_UPDATE_BYTES) }; - // One undo step takes the whole change back, on both. - assert!(a.undo().unwrap()); - until(&mut [&mut a, &mut b], "the undo is on both", |s| s.iter().all(|s| s.model().items.is_empty())); -} - /// Reconnecting: the agent drops the app; the model stays editable, changes queue, and go out once the session is /// back by itself (none lost, none twice). #[test] @@ -233,6 +202,40 @@ fn presence_goes_to_the_other_machine_and_fades() { assert!(b.peers().contains_key(&a.peer())); assert!(a.peers().is_empty(), "its own isn't shown"); assert!(a.set_presence(Some(vec![0; 4096])).is_err(), "at most 4 KiB with its header"); + // Each says its agent's version: the laptop's is current. An app on a Craft Cloud from before versions in + // presence (the bytes right after its peer) is older: the session says so. + b.set_presence(Some(b"laptop".to_vec())).unwrap(); + until(&mut [&mut a, &mut b], "the pc sees the laptop", |s| s[0].peers().contains_key(&s[1].peer())); + assert_eq!(a.peers()[&b.peer()].agent_api, Some(craft_cloud_api::local::LOCAL_API_MAX)); + assert!(!a.peers()[&b.peer()].older() && !a.status().older_peer); + let docs = laptop.client().documents().unwrap(); + let Ok(craft_cloud_app::live::Opened::Live(old)) = docs.open("p.txt", ::APP) else { panic!("not live") }; + let mut frame = 77u64.to_be_bytes().to_vec(); + frame.extend_from_slice(b"an older one"); + old.presence(&frame).unwrap(); + until(&mut [&mut a], "the pc sees an older Craft Cloud", |s| s[0].status().older_peer); + assert_eq!(a.peers()[&77].bytes, b"an older one"); + assert!(a.peers()[&77].older()); +} + +/// A change that doesn't read (a bundle cut short) is never taken as an empty one, at open or after: it is said and +/// counted. +#[test] +fn a_change_that_does_not_read_is_said_and_counted() { + let hub = Hub::new(); + let (pc, laptop) = (hub.machine("pc").unwrap(), hub.machine("laptop").unwrap()); + let mut a = open::(&pc, "u.txt", Notes::new("u", &["one"])); + // Another app on the pc sends a bundle cut short (its count says five slices; none follow). + let docs = pc.client().documents().unwrap(); + let Ok(craft_cloud_app::live::Opened::Live(raw)) = docs.open("u.txt", ::APP) else { panic!("not live") }; + let mut cut = b"ccb1".to_vec(); + cut.extend_from_slice(&5u32.to_be_bytes()); + raw.send(&cut).unwrap(); + until(&mut [&mut a], "the pc's session hears it", |s| s[0].status().unread == 1); + assert!(a.status().error.as_deref().is_some_and(|e| e.contains("doesn't read")), "{:?}", a.status().error); + let b = open::(&laptop, "u.txt", Notes::default()); + assert_eq!(b.status().unread, 1, "at open too: {:?}", b.status().error); + assert_eq!(b.model().texts(), ["one"]); } #[test] @@ -245,6 +248,9 @@ fn blobs_go_through_the_agent_and_an_old_agent_says_so() { laptop.too_old(); let e = b.get(&chunk).unwrap_err(); assert!(e.to_string().contains("agent_too_old"), "{e}"); + // Too old for live documents too: the file opens as a plain file, and the app can say to update Craft Cloud. + let opened = Session::::open(&common::agent(&laptop), "x.txt", Arc::new(Notes::default()), None).unwrap(); + assert!(matches!(opened, Open::NotLive(NotLive::AgentTooOld))); // Not running at all: the file opens as a plain file. pc.restart(); let opened = Session::::open(&common::agent(&pc), "x.txt", Arc::new(Notes::default()), None).unwrap(); diff --git a/crates/kit/tests/split.rs b/crates/kit/tests/split.rs deleted file mode 100644 index 5146fad..0000000 --- a/crates/kit/tests/split.rs +++ /dev/null @@ -1,31 +0,0 @@ -//! The split at its edge: a change whose estimate is exactly [`SPLIT_BYTES`] is one commit; one byte more is two. Every -//! update stays at or under the split size. - -mod common; - -use common::{App, Notes}; -use craft_cloud_kit::raw::Replica; -use craft_cloud_kit::{OP_OVERHEAD, SPLIT_BYTES}; - -/// The estimate of adding a note is its row's ID in the list (16 bytes) and its cell (65-byte key and the text), each -/// with the per-operation overhead: two notes of `len` bytes make the split size exactly. -fn note_len_for(total: usize) -> usize { - (total - 2 * (16 + OP_OVERHEAD) - 2 * (65 + OP_OVERHEAD)) / 2 -} - -#[test] -fn a_change_at_the_split_size_is_one_commit_and_one_byte_more_is_two() { - assert_eq!(SPLIT_BYTES % 2, 0); - for (extra, commits) in [(0, 1), (1, 2)] { - let mut r = Replica::default(); - let base = Notes::new("t", &["first"]); - r.write::(None, &base).unwrap(); - let len = note_len_for(SPLIT_BYTES); - let one = base.with_item(1, &"n".repeat(len)); - let two = one.with_item(2, &"m".repeat(len + extra)); - let next = Notes { hint: Some(two.items.iter().skip(1).map(|(id, _)| *id).collect()), ..two }; - let (updates, made) = r.write::(Some(&base), &next).unwrap(); - assert_eq!(made, commits, "{extra} byte(s) over"); - assert!(updates.iter().all(|u| u.len() <= SPLIT_BYTES), "{:?}", updates.iter().map(Vec::len).collect::>()); - } -} diff --git a/crates/kit/tests/undo.rs b/crates/kit/tests/undo.rs index b8296cd..2bcd65e 100644 --- a/crates/kit/tests/undo.rs +++ b/crates/kit/tests/undo.rs @@ -1,6 +1,5 @@ //! Undo while live, two sessions on the fake agent: undo takes back only this machine's change (the other's edit to -//! the same element stays), redo puts it back, a group is one step, and a change split into several commits is one -//! step. +//! the same element stays), redo puts it back, a group is one step, and a change sent in fragments is one step. mod common; mod model; @@ -59,8 +58,7 @@ fn undo_takes_back_only_this_machines_change_and_redo_puts_it_back() { fn undo_after_a_remote_edit_to_the_same_element_keeps_the_remote_edit() { let hub = Hub::new(); let (x, y) = (craft_cloud_kit::ElemId::new(), craft_cloud_kit::ElemId::new()); - let layers = - vec![Arc::new(Node { id: x, name: "x".into(), children: Vec::new() }), Arc::new(Node { id: y, name: "y".into(), children: Vec::new() })]; + let layers = vec![Arc::new(Node::leaf(x, "x")), Arc::new(Node::leaf(y, "y"))]; let (mut a, mut b, _pc, _laptop) = pair(&hub, Doc { layers: Arc::new(layers), ..Doc::empty() }); change(&mut a, |d| Doc { layers: Arc::new(model::rename(&d.layers, x, "renamed")), ..d.clone() }).unwrap(); until(&mut [&mut a, &mut b], "renamed on both", |s| s.iter().all(|s| model::find(&s.model().layers, x).is_some_and(|n| n.name == "renamed"))); @@ -77,11 +75,11 @@ fn undo_after_a_remote_edit_to_the_same_element_keeps_the_remote_edit() { }); } -/// Typing grouped into one step; a big change (here 1.25 MB of cells, split into several commits under the cap) is -/// one step too. (Loro's undo of one big run of text takes time growing with its square, about 10 s for 1 MB: the +/// Typing grouped into one step; a big change (here 1.25 MB of cells: one update, in fragments past the 1 MiB frame) +/// is one step too. (Loro's undo of one big run of text takes time growing with its square, about 10 s for 1 MB: the /// report's concerns, so the big change here is cells.) #[test] -fn a_group_and_a_split_change_are_one_step_each() { +fn a_group_and_a_fragmented_change_are_one_step_each() { let hub = Hub::new(); let (mut a, mut b, _pc, _laptop) = pair(&hub, Doc { story: Arc::new(RichText::plain("")), ..Doc::empty() }); a.group(|s| { @@ -105,7 +103,7 @@ fn a_group_and_a_split_change_are_one_step_each() { let sent = hub.updates("u.kitdemo"); change(&mut a, big).unwrap(); until(&mut [&mut a, &mut b], "the big change arrived", |s| s.iter().all(|s| s.model().grid.cells.len() == 5)); - assert!(hub.updates("u.kitdemo") - sent >= 3, "split into several updates"); + assert!(hub.updates("u.kitdemo") - sent >= 2, "one update over the frame limit: in fragments"); assert!(a.undo().unwrap(), "the big change"); until(&mut [&mut a, &mut b], "the big change undone in one step", |s| { s.iter().all(|s| s.model().grid.rows.is_empty() && story(s) == "one two three four") @@ -153,6 +151,103 @@ fn an_undo_during_a_gesture_waits_for_its_end() { until(&mut [&mut a, &mut b], "the same on both", |s| s.iter().all(|s| *s.model().title == "remote" && s.model().texts() == ["x", "mine"])); } +fn layer_text(s: &Session) -> String { + s.model().layers.first().map(|n| n.text.text.clone()).unwrap_or_default() +} + +fn type_in_layer(d: &Doc, at: usize, s: &str) -> Doc { + let Some(id) = d.layers.first().map(|n| n.id) else { return d.clone() }; + Doc { layers: Arc::new(model::update(&d.layers, id, &|n| Node { text: model::type_at(&n.text, at, s), ..n.clone() })), ..d.clone() } +} + +/// Typing in one layer's story on both machines at once (a shape's text box), keystroke by keystroke, then one undoes: +/// only its own typing goes, on both machines; the other's stays where it was typed. +#[test] +fn typing_in_one_nodes_story_on_both_then_one_undoes() { + let hub = Hub::new(); + let start = Doc { + layers: Arc::new(vec![Arc::new(Node { text: RichText::plain("[]"), ..Node::leaf(craft_cloud_kit::ElemId::new(), "box") })]), + ..Doc::empty() + }; + let (mut a, mut b, _pc, _laptop) = pair(&hub, start); + until(&mut [&mut a, &mut b], "the laptop has the box", |s| s.iter().all(|s| layer_text(s) == "[]")); + // The pc types "abc" inside the brackets, the laptop "XY" after them, each keystroke a step, interleaved. + for (k, (pc, laptop)) in [("a", Some("X")), ("b", Some("Y")), ("c", None)].into_iter().enumerate() { + change(&mut a, |d| type_in_layer(d, 1 + k, pc)).unwrap(); + if let Some(t) = laptop { + let at = layer_text(&b).chars().count(); + change(&mut b, |d| type_in_layer(d, at, t)).unwrap(); + } + a.poll(); + b.poll(); + } + until(&mut [&mut a, &mut b], "both have both", |s| s.iter().all(|s| layer_text(s) == "[abc]XY")); + for _ in 0..3 { + assert!(a.undo().unwrap()); + } + until(&mut [&mut a, &mut b], "the pc's typing went on both", |s| s.iter().all(|s| layer_text(s) == "[]XY")); + assert!(b.undo().unwrap()); + until(&mut [&mut a, &mut b], "the laptop's last keystroke went", |s| s.iter().all(|s| layer_text(s) == "[]X")); + for _ in 0..3 { + assert!(a.redo().unwrap()); + } + until(&mut [&mut a, &mut b], "the pc's typing back", |s| s.iter().all(|s| layer_text(s) == "[abc]X")); +} + +/// Loro's undo of a deletion makes the nodes again under new tree IDs (the old ones stay deleted, carrying the same +/// element IDs). A layer deleted on the pc while the laptop, offline, renames it, then the pc's undo: both machines +/// end with the same model, which is what their documents hold (the revived copy wins on both; the rename, made on the +/// node that stays deleted, is lost, the same on both). +#[test] +fn undoing_a_delete_while_the_other_edits_ends_the_same_on_both() { + let hub = Hub::new(); + let id = craft_cloud_kit::ElemId::new(); + let start = Doc { + layers: Arc::new(vec![Arc::new(Node::leaf(id, "before")), Arc::new(Node::leaf(craft_cloud_kit::ElemId::new(), "other"))]), + ..Doc::empty() + }; + let (mut a, mut b, _pc, laptop) = pair(&hub, start); + until(&mut [&mut a, &mut b], "in step", |s| s.iter().all(|s| s.model().layers.len() == 2)); + laptop.set_online(false); + change(&mut a, |d| Doc { layers: Arc::new(model::remove(&d.layers, id).0), ..d.clone() }).unwrap(); + change(&mut b, |d| Doc { layers: Arc::new(model::rename(&d.layers, id, "renamed offline")), ..d.clone() }).unwrap(); + assert!(a.undo().unwrap(), "the delete is undone"); + laptop.set_online(true); + until(&mut [&mut a, &mut b], "the same on both", |s| { + let (x, y) = (&s[0].model().layers, &s[1].model().layers); + x == y && x.len() == 2 + }); + // And each model is what its document holds. + for s in [&a, &b] { + let mut r = craft_cloud_kit::raw::Replica::default(); + r.import(&[s.snapshot().unwrap()], &["layers"]).unwrap(); + let back = r.read::(&Arc::new(Doc::empty())).unwrap(); + assert_eq!(model::flatten(&back.layers), model::flatten(&s.model().layers)); + assert_eq!(model::find(&back.layers, id).map(|n| n.name.clone()), model::find(&s.model().layers, id).map(|n| n.name.clone())); + } +} + +/// A change the app makes by itself (`background_change`: an upload that finished) goes to the other machine but is +/// never an undo step: undo takes back the user's change before it and leaves it. +#[test] +fn a_background_change_is_sent_but_never_undone() { + let hub = Hub::new(); + let (mut a, mut b, _pc, _laptop) = pair(&hub, Doc { story: Arc::new(RichText::plain("s")), ..Doc::empty() }); + change(&mut a, |d| type_end(d, "!")).unwrap(); + let old = a.model().clone(); + let new = Arc::new(Doc { settings: Arc::new(vec![7, 7]), ..(*old).clone() }); + a.background_change(&old, &new).unwrap(); + until(&mut [&mut a, &mut b], "the laptop has both", |s| s.iter().all(|s| story(s) == "s!" && *s.model().settings == vec![7, 7])); + assert!(a.undo().unwrap()); + until(&mut [&mut a, &mut b], "the typing went, the background change stayed", |s| { + s.iter().all(|s| story(s) == "s" && *s.model().settings == vec![7, 7]) + }); + assert!(!a.undo().unwrap(), "the background change isn't a step"); + // A stale `old` is refused, as for a local change. + let stale = Arc::new(Doc::empty()); + assert_eq!(a.background_change(&stale, &new), Err(craft_cloud_kit::KitError::Stale)); +} + /// The story with `paste` added at its end, as one change (plain text: no runs to move). fn paste(s: &mut Session, paste: &str) { change(s, |d| { diff --git a/crates/meta/src/live.rs b/crates/meta/src/live.rs index 9d8efbe..ae48694 100644 --- a/crates/meta/src/live.rs +++ b/crates/meta/src/live.rs @@ -486,56 +486,80 @@ impl Meta { bytes: &[u8], now_ms: i64, ) -> Result { - let client_seq = to_i64(client_seq)?; + let mut appended = self.append_live_updates_counted(space, doc, device, &[(client_seq, bytes)], now_ms).await?; + appended.pop().ok_or(MetaError::Invalid("no update")) + } + + /// [`Meta::append_live_update_counted`] for several updates of one device, stored together or not at all (a change + /// sent in fragments: it is in the log whole or not at all, and counts against the document's cap only then). + /// Each update sent again (as it was stored) gets its first number back; the fresh ones are stored with + /// consecutive numbers. The answers are in the order of `updates`. + pub async fn append_live_updates_counted( + &self, + space: SpaceId, + doc: DocId, + device: DeviceId, + updates: &[(u64, &[u8])], + now_ms: i64, + ) -> Result, MetaError> { let (space_key, doc_key, device_key) = (uuid_bytes(space.as_uuid()), uuid_bytes(doc.as_uuid()), uuid_bytes(device.as_uuid())); let mut w = self.begin_write().await?; - // Taking the next number locks the document's row (on Postgres), so appends to one document run one at a - // time and the check for a repeat below can't race another append of the same update. - let size = i64::try_from(bytes.len()).map_err(|_| MetaError::Invalid("an update too large"))?; + // Locking the document's row first (on Postgres), so appends to one document run one at a time and the check + // for a repeat below can't race another append of the same update. let head = sqlx::query( - "UPDATE live_docs SET head_seq = head_seq + 1, bytes = bytes + $3 WHERE space_id = $1 AND id = $2 \ - RETURNING head_seq, bytes, compacted_bytes, base_seq, kind", + "UPDATE live_docs SET head_seq = head_seq WHERE space_id = $1 AND id = $2 RETURNING head_seq, bytes, compacted_bytes, base_seq, kind", ) .bind(space_key.clone()) .bind(doc_key.clone()) - .bind(size) .fetch_optional(&mut *w.tx) .await? .ok_or(MetaError::NotFound("document"))?; - let seq: i64 = head.try_get("head_seq")?; + let head_seq: i64 = head.try_get("head_seq")?; // What counts against the cap: the log above its snapshot. let total = to_u64(head.try_get("bytes")?)?.saturating_sub(to_u64(head.try_get("compacted_bytes")?)?); let base = to_u64(head.try_get("base_seq")?)?; let recents = !head.try_get::("kind")?.starts_with(craft_cloud_proto::live::DOCUMENT_KIND_PREFIX); - let earlier = sqlx::query("SELECT seq, bytes FROM live_updates WHERE space_id = $1 AND doc = $2 AND device = $3 AND client_seq = $4") - .bind(space_key.clone()) - .bind(doc_key.clone()) - .bind(device_key.clone()) - .bind(client_seq) - .fetch_all(&mut *w.tx) - .await?; - for row in earlier { - if row.try_get::, _>("bytes")? == bytes { - let first = to_u64(row.try_get("seq")?)?; - // The number taken above goes back. - w.tx.rollback().await?; - return Ok(Appended { seq: first, base, above_bytes: total.saturating_sub(to_u64(size)?), repeat: true }); + // Which were stored before (their acks lost): their first numbers. + let mut repeats: Vec> = Vec::with_capacity(updates.len()); + for (client_seq, bytes) in updates { + let earlier = sqlx::query("SELECT seq, bytes FROM live_updates WHERE space_id = $1 AND doc = $2 AND device = $3 AND client_seq = $4") + .bind(space_key.clone()) + .bind(doc_key.clone()) + .bind(device_key.clone()) + .bind(to_i64(*client_seq)?) + .fetch_all(&mut *w.tx) + .await?; + let mut first = None; + for row in earlier { + if row.try_get::, _>("bytes")? == *bytes { + first = Some(to_u64(row.try_get("seq")?)?); + break; + } } + repeats.push(first); } - // Full is one way: once the log holds the cap, every append is refused (the one that crosses it is stored), so - // no later, smaller update can be stored past a refused one it depends on. A repeat above is still acked, so - // a full document never stalls a device whose ack was lost. Only `reset_live_doc` empties it again. - let used = total.saturating_sub(to_u64(size)?); + let fresh: Vec<&(u64, &[u8])> = updates.iter().zip(&repeats).filter(|(_, r)| r.is_none()).map(|(u, _)| u).collect(); + if fresh.is_empty() { + w.tx.rollback().await?; + return Ok(repeats.into_iter().map(|r| Appended { seq: r.unwrap_or_default(), base, above_bytes: total, repeat: true }).collect()); + } + let size = fresh.iter().map(|(_, b)| b.len()).sum::(); + let size_i = i64::try_from(size).map_err(|_| MetaError::Invalid("updates too large"))?; + // Full is one way: once the log holds the cap, every append is refused (the one that crosses it is stored, a + // set of fragments whole), so no later, smaller update can be stored past a refused one it depends on. A + // repeat above is still acked, so a full document never stalls a device whose ack was lost. Only + // `reset_live_doc` empties it again. + let used = total; if used >= self.live_doc_cap { w.tx.rollback().await?; return Err(MetaError::DocFull { used, limit: self.live_doc_cap }); } // The first update to a recents document: the space may have only so many that hold updates. Counted after the - // row lock above, in the same transaction; over the cap, nothing is stored and the number goes back, as for a - // full document. A repeat was acked above, and a document that holds updates already is never refused here - // (by its numbers, not its bytes: a compacted log holds updates, though none count against its cap). Documents - // of files aren't counted: each needs a file of the space. - if seq == 1 { + // row lock above, in the same transaction; over the cap, nothing is stored, as for a full document. A repeat + // was acked above, and a document that holds updates already is never refused here (by its numbers, not its + // bytes: a compacted log holds updates, though none count against its cap). Documents of files aren't counted: + // each needs a file of the space. + if head_seq == 0 { // Documents of files are counted apart: each needs a file of the space, and their cap is far higher. let (sql, cap) = if recents { ( @@ -557,7 +581,7 @@ impl Meta { // Every row of every live document of the space, compacted ones too, counts towards its live bytes (kept until // retention): over the cap, nothing is stored. A repeat was acked above. let live: i64 = sqlx::query("UPDATE spaces SET live_bytes = live_bytes + $1 WHERE id = $2 RETURNING live_bytes") - .bind(size) + .bind(size_i) .bind(space_key.clone()) .fetch_one(&mut *w.tx) .await? @@ -565,25 +589,43 @@ impl Meta { let live = to_u64(live)?; if live > self.live_bytes_cap { w.tx.rollback().await?; - return Err(MetaError::LiveFull { used: live.saturating_sub(to_u64(size)?), limit: self.live_bytes_cap }); + return Err(MetaError::LiveFull { used: live.saturating_sub(to_u64(size_i)?), limit: self.live_bytes_cap }); } - sqlx::query("INSERT INTO live_updates (space_id, doc, seq, bytes, device, client_seq, created_ms) VALUES ($1, $2, $3, $4, $5, $6, $7)") - .bind(space_key) - .bind(doc_key) - .bind(seq) - .bind(bytes.to_vec()) - .bind(device_key) - .bind(client_seq) - .bind(now_ms) + let n = i64::try_from(fresh.len()).map_err(|_| MetaError::Invalid("too many updates"))?; + sqlx::query("UPDATE live_docs SET head_seq = head_seq + $3, bytes = bytes + $4 WHERE space_id = $1 AND id = $2") + .bind(space_key.clone()) + .bind(doc_key.clone()) + .bind(n) + .bind(size_i) .execute(&mut *w.tx) .await?; + let above_bytes = total.saturating_add(to_u64(size_i)?); + let mut next = head_seq; + let mut out = Vec::with_capacity(updates.len()); + for ((client_seq, bytes), repeat) in updates.iter().zip(repeats) { + if let Some(first) = repeat { + out.push(Appended { seq: first, base, above_bytes, repeat: true }); + continue; + } + next += 1; + sqlx::query("INSERT INTO live_updates (space_id, doc, seq, bytes, device, client_seq, created_ms) VALUES ($1, $2, $3, $4, $5, $6, $7)") + .bind(space_key.clone()) + .bind(doc_key.clone()) + .bind(next) + .bind(bytes.to_vec()) + .bind(device_key.clone()) + .bind(to_i64(*client_seq)?) + .bind(now_ms) + .execute(&mut *w.tx) + .await?; + out.push(Appended { seq: to_u64(next)?, base, above_bytes, repeat: false }); + } w.commit().await?; - Ok(Appended { seq: to_u64(seq)?, base, above_bytes: total, repeat: false }) + Ok(out) } - /// Every live document of `space` with its last sequence number: what a backup of the database holds. A test seam - /// (the `test-support` feature), for the recents simulation's restores. - #[cfg(feature = "test-support")] + /// Every live document of `space` with its last sequence number (for the admin's checks, and the recents + /// simulation's restores). pub async fn live_heads(&self, space: SpaceId) -> Result, MetaError> { let rows = sqlx::query("SELECT id, head_seq FROM live_docs WHERE space_id = $1 ORDER BY id") .bind(uuid_bytes(space.as_uuid())) @@ -703,6 +745,31 @@ impl Meta { Ok((page, more)) } + /// The updates `device` sent to the document as `client_seq`s `from..=to` that the log holds: their `client_seq`, + /// number and bytes, in one query. (A change in fragments partly stored before: the rest of it is joined with + /// what is here.) + pub async fn live_updates_sent( + &self, + space: SpaceId, + doc: DocId, + device: DeviceId, + from: u64, + to: u64, + ) -> Result)>, MetaError> { + let rows = sqlx::query( + "SELECT client_seq, seq, bytes FROM live_updates WHERE space_id = $1 AND doc = $2 AND device = $3 AND client_seq >= $4 \ + AND client_seq <= $5 ORDER BY seq", + ) + .bind(uuid_bytes(space.as_uuid())) + .bind(uuid_bytes(doc.as_uuid())) + .bind(uuid_bytes(device.as_uuid())) + .bind(to_i64(from)?) + .bind(to_i64(to)?) + .fetch_all(&self.pool) + .await?; + rows.iter().map(|row| Ok((to_u64(row.try_get("client_seq")?)?, to_u64(row.try_get("seq")?)?, row.try_get("bytes")?))).collect() + } + /// The document's updates with a sequence number above `after`, in order, at most `limit` (and at most /// [`MAX_LIVE_PAGE`]) of them. pub async fn live_updates(&self, space: SpaceId, doc: DocId, after: u64, limit: u32) -> Result, MetaError> { diff --git a/crates/meta/tests/behaviour.rs b/crates/meta/tests/behaviour.rs index d875fc3..96b4c2a 100644 --- a/crates/meta/tests/behaviour.rs +++ b/crates/meta/tests/behaviour.rs @@ -118,6 +118,7 @@ on_every_backend!( concurrent_live_creates_make_one_document, live_pages_stop_at_their_byte_limit, a_live_document_is_capped_by_bytes, + a_set_of_updates_is_stored_whole_or_not_at_all, live_documents_holding_updates_are_capped, live_document_rows_are_capped, a_document_is_made_by_import_once, @@ -772,6 +773,30 @@ async fn live_pages_stop_at_their_byte_limit(meta: &Meta) { assert!(!meta.device_active(dev).await.unwrap(), "revoked"); } +/// A set of updates (a change's fragments) goes into the log whole, with consecutive numbers, or not at all: a set +/// that would start in a full log is refused whole, nothing of it stored; one that crosses the cap is stored whole (as +/// one update that crosses it is). Sent again, its updates get their numbers back. +async fn a_set_of_updates_is_stored_whole_or_not_at_all(meta: &Meta) { + use craft_cloud_proto::DocId; + let meta = meta.clone().with_live_doc_cap(2500); + let ana = user(&meta, "ana").await; + let (dev, other) = (device(&meta, &ana).await, device(&meta, &ana).await); + let doc = DocId::generate(); + meta.create_live_doc(ana.space, doc, "document:x", None, NOW).await.unwrap(); + meta.append_live_update(ana.space, doc, other, 1, &[9; 100], NOW).await.unwrap(); + let set: Vec<(u64, &[u8])> = vec![(1, &[1u8; 1000][..]), (2, &[2u8; 1000][..]), (3, &[3u8; 1000][..])]; + let stored = meta.append_live_updates_counted(ana.space, doc, dev, &set, NOW).await.unwrap(); + assert_eq!(stored.iter().map(|a| a.seq).collect::>(), vec![2, 3, 4], "consecutive, crossing the cap whole"); + assert!(stored.iter().all(|a| !a.repeat && a.above_bytes == 3100)); + let again = meta.append_live_updates_counted(ana.space, doc, dev, &set, NOW).await.unwrap(); + assert_eq!(again.iter().map(|a| (a.seq, a.repeat)).collect::>(), vec![(2, true), (3, true), (4, true)]); + let next: Vec<(u64, &[u8])> = vec![(4, &[4u8; 10][..]), (5, &[5u8; 10][..])]; + let full = meta.append_live_updates_counted(ana.space, doc, dev, &next, NOW).await; + assert!(matches!(full, Err(MetaError::DocFull { .. })), "{full:?}"); + let d = meta.live_doc(ana.space, doc).await.unwrap().unwrap(); + assert_eq!((d.head, d.bytes), (4, 3100), "nothing of the refused set stored"); +} + /// A document's log is capped by bytes, one way: the update that crosses the cap is /// stored, and from then on every append is refused, so no smaller later update is stored past a refused one it /// depends on. A repeat of an update already stored is still acked; the admin's reset empties the log. diff --git a/crates/proto/src/lib.rs b/crates/proto/src/lib.rs index ae04398..3bb9227 100644 --- a/crates/proto/src/lib.rs +++ b/crates/proto/src/lib.rs @@ -14,7 +14,7 @@ pub mod live; pub mod manifest; pub mod ops; -pub use craft_cloud_api::{ids, limits, local, name}; +pub use craft_cloud_api::{fragments, ids, limits, local, name}; pub use conflict::{ConflictStamp, conflict_name}; pub use hash::{BlobKind, ChunkId, FileHash, HexIdError, ManifestId}; diff --git a/crates/proto/src/live.rs b/crates/proto/src/live.rs index aa8e9ef..afb8c59 100644 --- a/crates/proto/src/live.rs +++ b/crates/proto/src/live.rs @@ -62,10 +62,15 @@ use crate::ids::{DeviceId, DocId, Epoch, NodeId, SpaceId}; /// Live protocol versions this build speaks. An agent and a server must share at least one. pub const LIVE_PROTOCOL_MIN: u32 = 1; -pub const LIVE_PROTOCOL_MAX: u32 = 2; +pub const LIVE_PROTOCOL_MAX: u32 = 3; /// The protocol that adds documents linked to files, snapshots, compaction and presence. A connection on an older one /// carries recents only. pub const LIVE_DOCUMENTS_PROTOCOL: u32 = 2; +/// The protocol that adds changes in fragments and bundles (`crate::fragments`): the server stages an agent's +/// fragments and stores a set whole or not at all (acking each fragment's `client_seq` then), and never sends a +/// fragment or a bundle to a connection on an older protocol (it refuses that connection the document instead: its +/// agent can't read them). An agent sends fragments only on it. +pub const LIVE_FRAGMENTS_PROTOCOL: u32 = 3; /// Largest update, in bytes; largest snapshot of a document (an import, or a compaction), in bytes (it travels through /// the chunk API, never in a frame); largest presence message (Loro's ephemeral state: a cursor, a selection, a machine @@ -622,7 +627,8 @@ mod tests { fn versions_are_negotiated_to_the_highest_shared() { assert_eq!(shared_version(1, 1), Some(1)); assert_eq!(shared_version(1, 5), Some(LIVE_PROTOCOL_MAX)); - assert_eq!(shared_version(2, 5), Some(LIVE_DOCUMENTS_PROTOCOL)); + assert_eq!(shared_version(2, 5), Some(LIVE_FRAGMENTS_PROTOCOL)); + assert_eq!(shared_version(1, 2), Some(LIVE_DOCUMENTS_PROTOCOL), "an agent from before fragments"); assert_eq!(shared_version(LIVE_PROTOCOL_MAX + 1, LIVE_PROTOCOL_MAX + 4), None); assert_eq!(shared_version(0, 0), None); assert_eq!(shared_version(3, 1), None, "an empty range"); diff --git a/docs/design/live.md b/docs/design/live.md index 75e9a4e..b7687b8 100644 --- a/docs/design/live.md +++ b/docs/design/live.md @@ -34,8 +34,9 @@ per app connection for live sessions. `run` serves it while signed out too. Call `status.autostart` says whether it is on (absent where the agent doesn't handle it). Signing out turns it off. The entry belongs to the agent with the default data folder only; an agent with a data folder of its own is refused (`autostart_failed`), reports nothing and leaves the entry alone when it signs out; -- `GET /v1/status` → `{api, signed_in, account, server, folder, machine, online, run_state, autostart?, - last_sync_ms?, removed}` (`last_sync_ms`: when the last pass that didn't fail finished; `removed`: the server +- `GET /v1/status` → `{api, api_max?, signed_in, account, server, folder, machine, online, run_state, autostart?, + last_sync_ms?, removed}` (`api`: the oldest local API version the agent speaks, `api_max` the newest, absent from + an agent that speaks one; an app takes an agent whose range overlaps its own; `last_sync_ms`: when the last pass that didn't fail finished; `removed`: the server refused this device at the last pass, so it was removed from the account and must sign out and join again). Something only a person can put right found as `run` starts (a damaged or newer `account.json`, a damaged token) keeps the agent up, signed out with `run_state` `paused` and the reason; every call but `logout` (which forgets the account here) and `quit` is answered @@ -55,11 +56,12 @@ per app connection for live sessions. `run` serves it while signed out too. Call such file in the synced folder) and `resetting` (the admin reset the server's document and no machine has sent it back yet: an app here would get an empty document). The app then opens and saves the file as always. A document that has been live on this machine opens offline from the agent's copy; - - a binary update from the app (its own change, at most 1 MiB) → `ack {id, seq}` once it is on the agent's disk + - a binary update from the app (its own change, at most 1 MiB a frame; a bigger one in fragments, see below, acked by + its last fragment's number) → `ack {id, seq}` once it is on the agent's disk (`seq` counts the app's updates of that document on this connection, from 1), or `error {id, seq, code, message}` when the agent refused it (`invalid`: Loro doesn't take it; `too_large`); the agent's copy is as it was; - - from the agent: remote changes as binary updates, other machines' and apps' presence as binary presence (as - sent), and `status {doc, online, waiting, deleted, stopped}` when that changes (`deleted`: the file was deleted, + - from the agent: remote changes as binary updates (each change whole, never a fragment), other machines' and + apps' presence as binary presence (as sent), and `status {doc, online, waiting, deleted, stopped}` when that changes (`deleted`: the file was deleted, here or on another machine; the document still works, and a later open of the path is a new node and a new document; a rename or move keeps the document, which belongs to the node; `stopped`: Loro failed on the document twice in this run, and the agent follows it again only once it restarts; changes made meanwhile are @@ -100,7 +102,12 @@ for opaque updates; variants are only ever appended, anything new is negotiated `craft-cloud-proto`: - `hello {min, max}` → `welcome {protocol, epoch}` (the space's epoch), or `error protocol_mismatch`. Protocol 2 adds documents of files, snapshots, compaction and presence; a connection on protocol 1 (an agent from before them, or - a new agent and an older server) carries recents only, and is sent a compacted log whole. A new agent with an older + a new agent and an older server) carries recents only, and is sent a compacted log whole. Protocol 3 adds changes + in fragments (see Fragments below): the server stages a connection's fragments and stores a set whole, and never + sends a fragment or a bundle to a connection below 3 (that agent is refused the document with "update Craft Cloud + on this computer"), and refuses a fragment from one (stored one by one, its set could be stored in part). An + agent never sends fragments or bundles to a server below 3: a change that needs them is refused while it is the + server, and what it holds of them waits, with everything after it, until the server is updated. A new agent with an older server keeps recents and says live documents need a newer server; a document live on it already keeps its changes on the machine until then. - `create {doc, kind}` → `created {doc}`; idempotent. Kinds: `recents:`, whose ID is derived from the account and @@ -148,7 +155,11 @@ for opaque updates; variants are only ever appended, anything new is negotiated update that crossed, or one that just caught up); one request per document at a time. That agent answers `snapshot {doc, upto, manifest}`, naming a Loro snapshot of exactly the updates 1..=`upto` (exported at the version those updates make as the agent took them, never including later ones: the agent tracks what the log holds, as a version - per update), once it holds them and nothing waits for its history; otherwise it doesn't answer. An agent that + per update), once it holds them, nothing waits for its history, no change it keeps fragments of starts at or below + `upto`, `upto` isn't inside a change's fragments, and its copy is at most 65,536 operations past that version + (Loro's checkout to an earlier version is slow for a big paste; its own unacked operations don't count once the + document was refused as full, and an ask it leaves for that reason is kept and answered at its next ack or + refusal); otherwise it doesn't answer. An agent that dropped a row at or below `upto` (Loro failed on it, or it never applies) never answers, and says so. The server takes only the snapshot it asked that session for, and only a manifest the account uploaded: it records `base = upto` and the manifest in one metadata transaction, only past the current base (a second snapshot for an older or @@ -177,8 +188,15 @@ for opaque updates; variants are only ever appended, anything new is negotiated device, 400 updates, creates and subscribes per connection refilled at 200/s; over them, `too_large` / `rate_limited` like the HTTP API. - Quota: once a document's log holds 16 MiB (about 30,000 recents adds), every later append is refused with - `quota_exceeded` (one-way: the update that crosses the cap is the last one stored). The agent keeps every unacked - update, retries with a pause growing to about 64 s, and says the document is full. + `quota_exceeded` (one-way: the update that crosses the cap is the last one stored). A change in fragments counts + only once its set is whole: one that would start in a full log is refused whole (every fragment's `client_seq` + named), one that crosses the cap is stored whole. A change is at most 8 MiB, half the cap (a set that joins to + more is refused whole). A refusal as full asks the refused device to compact (once per head, not while another + device's ask is out): with nothing more stored, no append would. The agent keeps every unacked update, retries + with a pause growing to about 64 s, and says the document is full. + `craft-cloud-server live check-sets ` lists the user's live documents whose log holds part of a change in + fragments (stored by a server before whole sets, and never completed: no machine compacts such a log past it), + each with the `reset-doc` that empties it. `craft-cloud-server live reset-doc ` (or, for a document of a file, ` `) empties the log, drops the snapshot and the import, and raises the space's live generation in the same transaction, so every live connection ends at its next ping and every agent starts over and pushes what the log lacks. The file sync's @@ -235,7 +253,40 @@ document's version it was made from. to `{$track, $start, $len, …}`; `k.keys`: keyframe ID to `{$item, $at, $value}`), an opaque value (the root map `$opaque`, key `k`, bytes). IDs are 16 bytes (little-endian `u128`), or 32 hex digits as map keys. Binary content is never in Loro: it is a `{$chunk, $len}` value naming a blob. -- A local change is written as commits of at most 512 KiB each (estimated: payload and 32 bytes per operation), each - exported at once; an export still over 512 KiB is cut by counter range. One change is one undo step. -- Presence an app sets is sent as `[u64 peer, big-endian][the app's bytes]`, again every 3 s, and dropped 10 s after - its peer goes quiet. +- Inside a tree node's meta map, a timeline item's or track's map, or a table's map cell, an app's codec can put a + rich text and a table under a key of its own, as Loro's mergeable child containers (the same container on every + machine that makes it, so two first edits at once merge): a story `s` is a `LoroText` at `s` laid out as a root + one, its last paragraph's attributes in a mergeable map `$last.s` beside it (made only when there are some); a + table `t` is a mergeable map at `t` holding mergeable `rows` and `cols` (movable lists of IDs) and `cells`. A cell + is a value, or, when the table's source says its cells are maps, a mergeable map at `row:column` holding the + cell's properties and stories. Deleting the key hides the child (a node's story that the app stops writing goes); + a deleted node takes everything nested in it. +- A local change is one commit, one undo step, and one update: its export cut by counter range into slices of at + most 16,384 operations (`SLICE_OPS`), several of them in a bundle (`ccb1`, a 4-byte count, then each slice's + 4-byte length and bytes, big endian), which a receiver imports slice by slice before its model sees any of it. +- **Fragments** (`craft-cloud-api::fragments`): an update over the 1 MiB frame limit goes as frames of at most 1 MiB, + each `ccf1`, the set (the first 16 bytes of the update's BLAKE3), the index and the count (2 bytes each, big + endian), then its part. The cut is deterministic, so a set sent again is the same set. The app library sends a + change's fragments back to back and answers by the last one's number (a refusal of any of them refuses the change + under that number); the agent's local API (version 2: an app's live documents need it, and refuse an older agent + as `agent_too_old`) joins them per document and hands the hub the whole change (one an app stops sending part of + the way never reaches it). The agent journals the change whole and sends its fragments up as updates of their own + (each a `client_seq`, numbered in the fragments' order). The server (protocol 3) stages a connection's fragments + (at most 16 sets and 16 MiB) and stores a set whole, in one transaction with consecutive numbers, or refuses it + whole; it acks every fragment then. A connection that ends drops what it staged; nothing of it was acked, and the + agent sends it all again. Fragments of the set that the device has in the log already (a set stored in part + before whole sets, or one whose acks were lost, of which the agent sends only the rest) count as there: they are + found by their `client_seq` (a set's fragments are numbered one after the other) and acked with the numbers they + have. After a restore, the agent drops the rest of a set whose first fragments were acked (the log may have lost + them) and pushes its operations instead. + A receiving agent keeps fragment rows (counted as had, in the document's file; at most 16 sets and 16 MiB per + document, past which a row isn't taken and the next connection brings it again) until the change is whole, then + imports it and hands it to its apps whole: the agent's apps never get a fragment, and the app library joins none. + No time ends a kept set; only evidence does: its last row comes (the rows of a set are stored together, so a + catch-up after a sleep or a dropped connection brings the rest), a snapshot at or past its first row is imported + (the copy has the change then), or the log is learnt again from the start. It makes no compaction snapshot at or + past a kept set's first row, nor at a number between a change's first fragment and its last. Nothing ever imports + part of a change. +- Presence an app's kit sets is sent as `[u64 peer, big-endian]["ccp2"][u32 local API version of its agent][the + app's bytes]`, again every 3 s, and dropped 10 s after its peer goes quiet. One without the mark comes from a kit + before versions in presence: an older Craft Cloud, which the kit says (`Peer::older`). diff --git a/docs/kit.md b/docs/kit.md index 91b2abc..2a14db8 100644 --- a/docs/kit.md +++ b/docs/kit.md @@ -81,6 +81,40 @@ impl LiveDocument for DemoApp { content never goes into the document: put it in a [`BlobStore`] ([`AgentBlobs`] is the agent's) and keep the [`ChunkRef`]. +### Text and tables inside elements + +A model with one text per element (a shape's text body, a slide's notes, a table cell's text) or a table per element +(a table shape) keeps them inside the element, under a key its codec names: + +```rust +fn write(&self, id: ElemId, props: &mut Props<'_, '_>) -> Result<(), KitError> { + let shape = self.shape(id); + props.set("xfrm", shape.xfrm_json())?; + props.text("text", &shape.story())?; // a rich text, as a root one + if let Some(table) = &shape.table { + props.table("table", &TableSrc(table))?; // rows, columns, cells + } + Ok(()) +} +``` + +- A nested story maps exactly as a root one does (marks, paragraph attributes, the diff that writes one insert per + keystroke). Two machines typing in one shape's text merge character by character, and a shape deleted on one machine + while the other types in it is deleted on both. +- A table's cells are values, or, when its source says [`TableSource::cell_maps`], property maps written by + `write_cell` like a node's, with stories of their own. Every cell of every row and column is then one; a row added + on one machine while a column is added on the other leaves the cell they cross unmade: `normalise` makes it (the + same plain cell on both, so the repairs are one). +- Both are mergeable children of the element's map: two machines making a shape's text at once make one. Write a + story whenever the element exists (empty if it has none): an element whose story stops being written loses it, + and typing that arrives in it then is lost with it. +- Reading back, [`TreeView::text_patched`] (and [`TableView::cell_text`]) patch the old text with the remote change's + deltas, as a root story is patched; [`TreeTouch::stories`] says which stories changed, [`PropsRead::table`] reads a + nested table whole. + +A change the app makes by itself rather than the user (an upload that finished and now has its chunk reference) goes +through [`Session::background_change`]: sent like a local change, never an undo step. + ## Opening ```rust @@ -108,8 +142,16 @@ if !Arc::ptr_eq(&before, &self.doc) { - `before` must be the session's model ([`KitError::Stale`] otherwise: a remote change came in between; make the change again from `session.model()`). -- A change is cut into commits under 512 KiB (the agent takes updates of up to 1 MiB): a 5 MB paste goes, and - undoes, whole. +- A change is one commit and goes as one update, whatever its size: a big one as its slices (at most 16,384 + operations each, which Loro imports in milliseconds; one update with much text takes it time growing with the + square of the text) in one bundle, and one over the agent's 1 MiB frame limit in fragments that every receiver + joins before anything imports them. A 5 MB paste goes, and undoes, whole: no machine, agent or app ever holds part + of a change, so an app never sees, repairs or writes back half of one, whatever it does meanwhile (an edit, an + undo, a save). The agent gives its apps every change whole: one that comes part of the way (its sender, or this + machine, slept or lost the connection meanwhile) is kept by the agent until the rest comes, and the app sees nothing + of it until then. A change over `MAX_CHANGE_BYTES` (8 MiB, half a document's log cap on the server) can't go: the + session stops, as for one the agent refuses for good; the model stays editable, and saving it to the file keeps the + work. - **A gesture** (a drag, an IME composition) whose `before` must stay put until it ends: `session.hold_remote(true)` while it runs; remote changes wait and come at the first poll after. - Typing: `session.group(|s| …)` (or `begin_group`/`end_group`) makes several changes one undo step. @@ -216,7 +258,13 @@ back. Saving clears `changed_outside`. ## Presence `session.set_presence(Some(bytes))` shows this app's cursor or selection to the others (the bytes are the app's, -at most 4 KiB less 8); `session.peers()` is theirs, dropped 10 s after they go quiet. Drawing them is the app's. +at most 4 KiB less 16); `session.peers()` is theirs, dropped 10 s after they go quiet. Drawing them is the app's. +Each presence also says the version of the Craft Cloud it runs on: `Peer::older()` (and `Status::older_peer`) is +true for a machine on a Craft Cloud too old for this one's big changes; show " runs an older Craft Cloud; +update it", with the machine's name from the app's own presence bytes. + +A Craft Cloud on this computer older than the kit's live documents (local API below 2) opens a file as +`Open::NotLive(NotLive::AgentTooOld)`: open it as a plain file, and say to update Craft Cloud. ## Craft Cloud's piece of the window @@ -227,14 +275,51 @@ the path); `watch.add_recent(path, saved)` puts a file on every machine's list. ## Testing the adapter With the `fake` feature, `craft_cloud_kit::testing::{Hub, Fake}` is an agent in the test's own process: machines of -one account on a shared hub, with `set_online`, `restart` and `start`, `drop_connections`, and an old-agent mode. +one account on a shared hub, with `set_online`, `restart` and `start`, `drop_connections`, and an old-agent mode +(`too_old`: before blobs and fragments). A fake machine joins a change's fragments before its apps get it, as the +agent does; `set_fragment_faults` stops a machine's link from the hub inside such a change for a while (`sleep`: +`SLEEP_TICKS` ticks, `delay`: a few; `Hub::tick` moves them on, `set_fragment_faults(None)` wakes it), and +`drop_connections_after` stops the agent part of the way through an app's change. +`craft_cloud_app::live::set_frame_limit_for_tests` makes changes of a few dozen KB go in fragments. Two sessions on two fakes test merges, undo and reconnects with no server; the demo's tests drive two windows that way with kittest. +## Known gaps + +- **Undo of a deletion makes the elements again as copies.** Loro's undo of a tree deletion creates new nodes (new + tree IDs, the same element IDs and properties); the old nodes stay deleted. What another machine changed in those + elements between the deletion and the undo went to the old nodes and is lost, the same on every machine (the models + stay alike). References other machines' repairs dropped meanwhile (a connector's end to a deleted shape) don't come + back with the undo either. A soft delete (nodes moved to a hidden root, so undo moves them back) would keep them. + +- **Every machine's agent must know fragments and bundles, and so must the server.** A server on live protocol 3 + stores a change in fragments whole and never sends a fragment or a bundle to an agent from before them: it is + refused the document with "update Craft Cloud on this computer" (and the newer machines show " runs an + older Craft Cloud" from its presence, best effort: once it is refused the document its presence stops). An agent + never sends fragments or bundles to a server before protocol 3: a change that needs them is refused while it is + the server, and what it holds of them waits on the machine, with every change after it, until it is updated. A + new app on an older agent isn't live (`NotLive::AgentTooOld`). An older app on a newer agent still connects (the + agent says its oldest local API version as `api`, and its newest as `api_max`) and gets recents and blobs. But an + older live-editing app on a new agent can't read bundles: a newer app's big change shows in it as one that doesn't + read, its model goes apart from the others', and it may save that model. The agent can't tell it apart yet (the + docs channel names no app version), so quit every older Craft app before the agents are updated, and update the + apps with them. The safe order: the server, then every agent and app. +- **After a server's restore**, a machine that sends up history the log lost sends a big Loro change in pieces (Loro + merges one peer's commits into one change, so its boundaries aren't an app's): an app open on a machine that also + lacked that history may see part of a change from before the restore. + ## Costs to know - Loro's undo of one big run of inserted text takes time growing with its square (about 10 s for a 1 MB paste). Undo of ordinary edits is instant. +- Loro's import of one update takes time growing with the square of the text in it when the text builds on another + peer's (1 MiB: 7 s); the kit's slices keep a 4 MiB paste's import at about 10 ms. So does its checkout to an earlier + version: an agent asked for a compaction while its copy is far past the asked version (its own big change not in the + log yet) leaves the answer to another device. +- A big paste into a text another machine edits at that very moment is the slow case left: in one order of the two + peers Loro works the text out again for every slice, so once a slice is slow the kit (and the agent) take the rest + as one update, which costs those seconds once (about 7 s for 1.2 MB, 0.5 s for 300 KB) on the machine that was + typing. Edits elsewhere at the same moment (another text, a property) cost nothing. - A remote change to a rich text is patched into the old text; a tree's structure change reads the tree's structure again (cheap per node, 5,000 nodes in a few milliseconds); a table or timeline is read back by the app as much as it wants.