diff --git a/AGENTS.md b/AGENTS.md index 3f35a82..c845f57 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,14 +29,15 @@ crates/ proto craft-cloud-proto wire types: names, IDs, hashes, manifests, commit ops, API bodies, limits, join codes and join strings (`join.rs`) chunker craft-cloud-chunker FastCDC `fastcdc-v1` + BLAKE3, frozen by golden vectors store craft-cloud-store blob store: disk, S3, memory; checks every blob in and out - meta craft-cloud-meta metadata on SQLite or Postgres; the batch commit; the change feed; invites and pairings (`join.rs`) + meta craft-cloud-meta metadata on SQLite or Postgres; the batch commit; the change feed; invites and pairings (`join.rs`); live documents' logs and snapshots (`live.rs`) sync craft-cloud-sync the sync engine over `Fs`/`Transport` traits; `MemFs`; scenario tests and the simulation - app craft-cloud-app what apps link: a blocking client of the agent's local API (status, sign-in, joining, computers, pause, autostart, recents) + app craft-cloud-app what apps link: a blocking client of the agent's local API (status, sign-in, joining, computers, pause, autostart, recents, live documents at the level of bytes: `live.rs`) apps/ - craft-cloud-server HTTP API (`/api/v1`), the live service (`/api/v1/live`, `live.rs`), join codes (`join.rs`: pairings, join, the `/join/{code}` page), auth, admin CLI (`init`, `serve`, `user` with `invite`, `device`, `reset-epoch`, `live`) - craft-cloud-agent the sync agent: real disk `Fs`, HTTP `Transport` over a `Connection`, saved account and state, root marker, CLI (`login`, `join`, `run`, `sync`, `status`, `logout`, `recents`, `add-computer`, `devices`, `pause`, `resume`, `quit`, `autostart`), the `run` loop, the local API (`local_api.rs`), the live hub (`live/`: recents document, replica, store, link, hub), start at login (`autostart/`: Windows, macOS, Linux), the background build (`src/bin/bg.rs`, logging to `runlog.rs`) - tests/network/ the simulation over a simulated network (turmoil): `main.rs` histories and scenarios, `harness.rs` server and agent hosts + craft-cloud-server HTTP API (`/api/v1`), the live service (`/api/v1/live`, `live.rs`: logs, documents by import, snapshots and compaction, presence), join codes (`join.rs`: pairings, join, the `/join/{code}` page), auth, admin CLI (`init`, `serve`, `user` with `invite`, `device`, `reset-epoch`, `live`) + craft-cloud-agent the sync agent: real disk `Fs`, HTTP `Transport` over a `Connection`, saved account and state, root marker, CLI (`login`, `join`, `run`, `sync`, `status`, `logout`, `recents`, `doc` (`doc.rs`), `add-computer`, `devices`, `pause`, `resume`, `quit`, `autostart`), the `run` loop, the local API (`local_api.rs`), the live hub (`live/`: a document's Loro copy, recents document, replica (recents and documents of files), store (with the documents' journals), link, hub, the local API's docs socket), start at login (`autostart/`: Windows, macOS, Linux), the background build (`src/bin/bg.rs`, logging to `runlog.rs`) + tests/network/ the simulation over a simulated network (turmoil): `main.rs` histories and scenarios, `harness.rs` server and agent hosts, `live.rs` live documents over WebSockets on it tests/recents_sim.rs the recents convergence simulation: replicas against the server's live sessions + tests/documents_sim.rs the documents convergence simulation (shares `tests/docs_common/` with `documents.rs` and `live_speed.rs`) craft-cloud-desktop the Craft Cloud app (`craft-cloud`, eframe/egui): the pure model (`model.rs`), the agent behind a trait and calls on threads (`agent.rs`), starting the agent (`start_agent.rs`), one app per user (`single_instance.rs`), the window (`ui/`, `app.rs`), the tray (`tray/`: tray-icon in the window's process on Windows and macOS; on Linux a StatusNotifierItem over zbus in a process of its own, `craft-cloud --tray`, `tray/process.rs`, since Wayland can't hide a window), the app's own start at login (`autostart.rs`), its icons (`assets/`, drawn by `cargo xtask icons`) tests/snapshots/ every screen as kittest draws it (`UPDATE_SNAPSHOTS=1` redraws them) craft-cloud-setup the Windows setup (`craft-cloud-setup.exe`): install, upgrade, move from a hand-made install, `--uninstall`; the exes built in by `cargo xtask package` diff --git a/README.md b/README.md index 76cddce..7e23ee7 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,8 @@ all your machines, and the folder `~/Craft Cloud` stays in sync, with version hi There is no hosted service and no account with anyone but you. Nothing phones home. > **Status: early development.** File sync, shared recent files and the Craft Cloud app work. Live -> documents (an edit shows on your other machines as you type) are next. +> documents (an edit shows on your other machines as you type) work in the agent and the server; the apps take +> them up next. - **Big files sync quickly.** Files are split into content-defined chunks, so a change to a 20 GB project uploads only what changed, and an interrupted upload resumes. diff --git a/apps/craft-cloud-agent/Cargo.toml b/apps/craft-cloud-agent/Cargo.toml index 6e0f6ef..c67c82d 100644 --- a/apps/craft-cloud-agent/Cargo.toml +++ b/apps/craft-cloud-agent/Cargo.toml @@ -16,6 +16,8 @@ bytes.workspace = true clap.workspace = true # `recents`, and `login`/`logout` while `run` runs, are clients of the running agent like an app. craft-cloud-app.workspace = true +# Live documents' snapshots go up as files' content does: cut into chunks. +craft-cloud-chunker.workspace = true craft-cloud-proto.workspace = true craft-cloud-sync.workspace = true file-id.workspace = true @@ -57,7 +59,6 @@ path = "src/main.rs" winresource.workspace = true [dev-dependencies] -craft-cloud-chunker.workspace = true craft-cloud-meta = { workspace = true, features = ["test-support"] } craft-cloud-server = { workspace = true, features = ["test-support"] } craft-cloud-store.workspace = true diff --git a/apps/craft-cloud-agent/src/cli.rs b/apps/craft-cloud-agent/src/cli.rs index 28405ea..e1ae35a 100644 --- a/apps/craft-cloud-agent/src/cli.rs +++ b/apps/craft-cloud-agent/src/cli.rs @@ -1,5 +1,5 @@ //! The `craft-cloud-agent` command line: `login` or `join`, `run`, `sync`, `status`, `logout`, and through the running -//! agent `recents`, `add-computer`, `devices`, `pause`, `resume`, `quit` and `autostart`. Each command is a library +//! agent `recents`, `doc` ([`crate::doc`]), `add-computer`, `devices`, `pause`, `resume`, `quit` and `autostart`. Each command is a library //! function too, so tests drive exactly what a user runs. //! //! The password is read from standard input (one line), so it never appears in the process list or shell @@ -94,6 +94,9 @@ pub enum Command { }, /// Show an app's recent files (shared by every machine of the account), or add one, through the running agent. Recents(RecentsArgs), + /// Open a file of the synced folder as a live plain-text document, through the running agent, like an app: print + /// its text, append to it, or print it again after every change. A test tool: it never writes the file. + Doc(DocArgs), /// Join a Craft Cloud with a join string: an invite from the server's admin, or a code from a signed-in computer /// (`add-computer` there). An invite for a new account asks for its password, twice, on standard input. Join(JoinArgs), @@ -142,6 +145,18 @@ pub struct RecentsArgs { pub action: Option, } +#[derive(Debug, Clone, clap::Args)] +pub struct DocArgs { + /// The file: its path in the synced folder (absolute, or relative to the folder). Plain text (UTF-8). + pub path: PathBuf, + /// Append this text to the document (on every machine that has it open). + #[arg(long)] + pub append: Option, + /// Keep printing the text, again after every change, until stopped (Ctrl-C). + #[arg(long)] + pub watch: bool, +} + #[derive(Debug, Clone, Subcommand)] pub enum RecentsAction { /// Add a file: its path in the synced folder (absolute, or relative to the folder). @@ -419,6 +434,7 @@ pub async fn run(cli: Cli, input: &mut impl BufRead, out: &mut impl Write) -> Re Ok(Exit::Done) } Command::Recents(args) => recents(&data, args, out).await, + Command::Doc(args) => crate::doc::doc(&data, args, out).await, Command::Join(args) => { if let Some(account) = data.load_account()? { return Err(already_signed_in(&account)); @@ -687,7 +703,7 @@ pub async fn main(args: Vec, input: &mut impl BufRead, out: } /// Runs a blocking call of `craft-cloud-app` off the runtime. -async fn blocking(f: impl FnOnce() -> Result + Send + 'static) -> Result { +pub(crate) async fn blocking(f: impl FnOnce() -> Result + Send + 'static) -> Result { tokio::task::spawn_blocking(f) .await .map_err(|e| AgentError::Usage(format!("the call to the running agent failed: {e}")))? @@ -695,7 +711,7 @@ async fn blocking(f: impl FnOnce() -> Result AgentError { +pub(crate) fn local_error(e: AgentError) -> AgentError { use craft_cloud_proto::local::codes; match e { AgentError::Local(craft_cloud_app::Error::Refused { code, message }) => match code.as_str() { @@ -805,7 +821,7 @@ async fn recents(data: &DataDir, args: RecentsArgs, out: &mut impl Write) -> Res /// A path in the synced folder as the local API wants it: relative, `/`-separated. An absolute one (or, on Windows, one /// with a root but no drive) must lie in the folder. -fn relative_to_folder(path: &Path, folder: Option<&Path>) -> Result { +pub(crate) fn relative_to_folder(path: &Path, folder: Option<&Path>) -> Result { let relative = if path.is_absolute() || path.has_root() { let folder = folder.ok_or(AgentError::NotSignedIn)?; let (path, folder) = (resolved(path), resolved(folder)); diff --git a/apps/craft-cloud-agent/src/daemon.rs b/apps/craft-cloud-agent/src/daemon.rs index f68dcb9..967f327 100644 --- a/apps/craft-cloud-agent/src/daemon.rs +++ b/apps/craft-cloud-agent/src/daemon.rs @@ -612,6 +612,7 @@ pub async fn run_loop(data: &DataDir, options: RunOptions, mut stop: mpsc::Unbou server: account.server.clone(), token: account.token.clone(), machine: account.device_name.clone(), + folder: account.folder.clone(), }; let hub = match Hub::start(hub_config, control.clone(), user_paused) { Ok(hub) => Some(hub), @@ -907,6 +908,12 @@ pub async fn run_loop(data: &DataDir, options: RunOptions, mut stop: mpsc::Unbou watching.watcher = None; let _ = reply.send(Ok(())); } + // An app opened a file that isn't uploaded yet: a pass soon (not while paused). + Some(Command::SyncSoon) => { + if !user_paused { + scheduler.urgent_change(); + } + } None => commands_open = false, }, () = sleep => {} @@ -1029,6 +1036,7 @@ async fn wait_needing_you( let _ = reply.send(Ok(())); return Stuck::Stop; } + Some(Command::SyncSoon) => {} None => commands_open = false, }, signal = stop.recv(), if *stop_open => match signal { @@ -1091,6 +1099,7 @@ async fn wait_for_sign_in( let _ = reply.send(Ok(())); return SignedOut::Stop; } + Some(Command::SyncSoon) => {} None => commands_open = false, }, signal = stop.recv(), if *stop_open => match signal { diff --git a/apps/craft-cloud-agent/src/doc.rs b/apps/craft-cloud-agent/src/doc.rs new file mode 100644 index 0000000..c3cb907 --- /dev/null +++ b/apps/craft-cloud-agent/src/doc.rs @@ -0,0 +1,176 @@ +//! `craft-cloud-agent doc [--append ] [--watch]`: a file of the synced folder as a live plain-text +//! document, through the running agent's local API, a client like an app (`craft-cloud-app`'s live documents). The +//! mapping is the plainest there is: one Loro text under `text`. A file that has never been live is imported from its +//! content as UTF-8 (one that isn't is refused). It prints the text, appends to it, or prints it again after every +//! change. A test tool: it never writes the file. + +use std::io::Write; +use std::path::{Path, PathBuf}; +use std::time::{Duration, Instant}; + +use craft_cloud_app::live::{Event, Opened}; +use loro::{ExportMode, LoroDoc}; + +use crate::account::DataDir; +use crate::cli::{AgentError, DocArgs, Exit, blocking, local_error, relative_to_folder}; + +/// The app this tool opens documents as. +pub const APP: &str = "plaintext"; +/// The text's container in the document. +const TEXT: &str = "text"; +/// How long an appended change may take to be on the agent's disk. +const ACK_TIMEOUT: Duration = Duration::from_secs(10); + +/// A document of one text holding `text`, as a Loro snapshot (the import of a file that has never been live). +pub fn plain_text_snapshot(text: &str) -> Result, String> { + let doc = LoroDoc::new(); + doc.get_text(TEXT).insert(0, text).map_err(|e| e.to_string())?; + doc.commit(); + doc.export(ExportMode::Snapshot).map_err(|e| e.to_string()) +} + +fn usage(message: impl Into) -> AgentError { + AgentError::Usage(message.into()) +} + +/// What the tool's thread tells the command to print. +enum Line { + Out(String), + Done(Exit), +} + +pub async fn doc(data: &DataDir, args: DocArgs, out: &mut impl Write) -> Result { + let data_path = data.path().to_owned(); + let client = blocking(move || craft_cloud_app::Client::connect_in(&data_path)).await.map_err(|e| match e { + AgentError::Local(craft_cloud_app::Error::NotRunning(why)) => { + usage(format!("the agent isn't running ({why}); start `craft-cloud-agent run` first")) + } + other => other, + })?; + let status = { + let c = client.clone(); + blocking(move || c.status()).await? + }; + let folder = status.folder.map(PathBuf::from).ok_or(AgentError::NotSignedIn)?; + let path = relative_to_folder(&args.path, Some(&folder))?; + // The crate blocks: it runs on a thread of its own, and what it has to say comes back here. + let (lines_tx, mut lines) = tokio::sync::mpsc::unbounded_channel::(); + let task = tokio::task::spawn_blocking(move || { + let said = lines_tx.clone(); + let say = move |line: String| { + let _ = said.send(Line::Out(line)); + }; + let exit = session(&client, &folder, &path, args.append.as_deref(), args.watch, &say); + if let Ok(exit) = &exit { + let _ = lines_tx.send(Line::Done(*exit)); + } + exit + }); + let mut exit = Exit::Done; + while let Some(line) = lines.recv().await { + match line { + Line::Out(text) => { + writeln!(out, "{text}").map_err(|e| AgentError::Io { path: Path::new("standard output").to_owned(), source: e })?; + let _ = out.flush(); + } + Line::Done(done) => exit = done, + } + } + task.await.map_err(|e| usage(format!("the document's session failed: {e}")))?.map_err(local_error)?; + Ok(exit) +} + +fn session( + client: &craft_cloud_app::Client, + folder: &Path, + path: &str, + append: Option<&str>, + watch: bool, + say: &dyn Fn(String), +) -> Result { + let docs = client.documents().map_err(AgentError::Local)?; + let document = match docs.open(path, APP).map_err(AgentError::Local)? { + Opened::Live(document) => document, + Opened::NeedsImport(importer) => { + let file = folder.join(path); + let bytes = std::fs::read(&file).map_err(|source| AgentError::Io { path: file.clone(), source })?; + let text = String::from_utf8(bytes).map_err(|_| usage(format!("{path} isn't UTF-8 text; `doc` reads plain text only")))?; + let snapshot = plain_text_snapshot(&text).map_err(usage)?; + importer.import(&snapshot).map_err(AgentError::Local)? + } + Opened::NotLive(reason) => { + say(format!("{path} isn't live ({reason:?}); apps open it as a plain file for now")); + return Ok(Exit::Failed); + } + }; + let mirror = LoroDoc::new(); + mirror.import(document.state()).map_err(|e| usage(format!("the document's state doesn't read: {e}")))?; + let text = |mirror: &LoroDoc| mirror.get_text(TEXT).to_string(); + // What came right after the state (updates still waiting for their history), before anything else. + let take = |mirror: &LoroDoc, event: &Event| { + if let Event::Update(bytes) = event { + // (An update this copy can't take is the other side's; the agent has it, and so will a fresh open.) + let _ = mirror.import(bytes); + } + }; + while let Some(event) = document.try_recv() { + take(&mirror, &event); + } + if let Some(addition) = append { + let before = mirror.oplog_vv(); + let t = mirror.get_text(TEXT); + t.insert(t.len_unicode(), addition).map_err(|e| usage(e.to_string()))?; + mirror.commit(); + let update = mirror.export(ExportMode::updates(&before)).map_err(|e| usage(e.to_string()))?; + let seq = document.send(&update).map_err(AgentError::Local)?; + let until = Instant::now() + ACK_TIMEOUT; + loop { + let left = until.saturating_duration_since(Instant::now()); + match document.recv_timeout(left) { + Some(Event::Acked(acked)) if acked == seq => break, + Some(Event::Refused { seq: refused, message }) if refused == seq => { + return Err(usage(format!("the agent refused the change: {message}"))); + } + Some(Event::Closed(why)) => return Err(usage(format!("the connection to the agent ended: {why}"))), + Some(event) => take(&mirror, &event), + None => return Err(usage("the agent didn't take the change in time")), + } + } + } + say(text(&mirror)); + if !watch { + return Ok(Exit::Done); + } + loop { + match document.recv_timeout(Duration::from_secs(1)) { + Some(Event::Update(bytes)) => { + let _ = mirror.import(&bytes); + say("--".into()); + say(text(&mirror)); + } + Some(Event::Status { online, waiting, deleted, stopped }) => { + let deleted = if deleted { "; its file was deleted" } else { "" }; + let stopped = if stopped { "; the agent stopped following it until it restarts" } else { "" }; + say(format!("({}, {waiting} change(s) waiting{deleted}{stopped})", if online { "online" } else { "offline" })); + } + Some(Event::Closed(why)) => { + say(format!("the connection to the agent ended: {why}")); + return Ok(Exit::Failed); + } + Some(Event::Presence(_) | Event::Acked(_) | Event::Refused { .. }) | None => {} + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_plain_text_document_holds_its_text() { + let snapshot = plain_text_snapshot("héllo\nwörld").unwrap(); + let doc = LoroDoc::new(); + doc.import(&snapshot).unwrap(); + assert_eq!(doc.get_text(TEXT).to_string(), "héllo\nwörld"); + } +} diff --git a/apps/craft-cloud-agent/src/lib.rs b/apps/craft-cloud-agent/src/lib.rs index 79c060b..ace6aa3 100644 --- a/apps/craft-cloud-agent/src/lib.rs +++ b/apps/craft-cloud-agent/src/lib.rs @@ -5,7 +5,9 @@ //! ([`account`]), guards the folder with a root marker ([`root`]) and runs it from the command line ([`cli`]): one //! pass at a time (`sync`), or on its own (`run`: [`daemon`], scheduled by [`schedule`], woken by [`watch`], with //! the stability window kept by [`settle`]). [`autostart`] registers it to start at login; the background build -//! (`craft-cloud-agent-bg`, no console on Windows) logs to [`runlog`]. +//! (`craft-cloud-agent-bg`, no console on Windows) logs to [`runlog`]. While signed in, the live hub ([`live`]) keeps +//! the account's live documents (apps' recents, and files opened live) in step with the server, and serves them to +//! apps through the local API ([`local_api`]); [`doc`] is a plain-text client of it, as a test tool. #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] pub mod account; @@ -13,6 +15,7 @@ pub mod autostart; pub mod cli; pub mod daemon; pub mod disk; +pub mod doc; pub mod http; pub mod live; pub mod local_api; diff --git a/apps/craft-cloud-agent/src/live/blobs.rs b/apps/craft-cloud-agent/src/live/blobs.rs new file mode 100644 index 0000000..822840f --- /dev/null +++ b/apps/craft-cloud-agent/src/live/blobs.rs @@ -0,0 +1,148 @@ +//! Snapshots of live documents travel as files' content does, never in a frame: cut into chunks named by a manifest +//! ([`cut`]), uploaded through the chunk API before the frame that names the manifest ([`upload`]: with that API's +//! body limits, hash checks and the account's storage cap), and downloaded the same way ([`download`]: every chunk +//! checked against its ID, the whole against the manifest's size and hash). The replica cuts and names them +//! ([`super::replica::Replica::take_uploads`], [`super::replica::Replica::take_fetches`]); the hub moves the bytes. + +use std::collections::BTreeSet; + +use bytes::Bytes; +use craft_cloud_chunker::{Chunker, FASTCDC_V1, chunk_reader}; +use craft_cloud_proto::live::MAX_SNAPSHOT_BYTES; +use craft_cloud_proto::{ChunkId, ChunkRef, ChunkerParamsId, DocId, FileHash, Manifest, ManifestId}; +use craft_cloud_sync::transport::{Transport, TransportError}; + +/// How many chunk IDs one "which are missing" question names. +const MISSING_QUERY_IDS: usize = 1000; + +/// A snapshot to upload before a frame names it: its manifest and its chunks (each once). +#[derive(Clone, PartialEq, Eq)] +pub struct Upload { + /// The document it is a snapshot of (frames of that document wait for it). + pub doc: DocId, + pub manifest: Manifest, + pub chunks: Vec<(ChunkId, Vec)>, +} + +impl std::fmt::Debug for Upload { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "Upload({}, manifest {}, {} bytes)", self.doc, self.manifest.id(), self.manifest.size()) + } +} + +impl Upload { + pub fn id(&self) -> ManifestId { + self.manifest.id() + } + + /// The snapshot again, from its chunks (for tests that look inside). + pub fn bytes(&self) -> Vec { + let mut out = Vec::new(); + for chunk in self.manifest.chunks() { + if let Some((_, data)) = self.chunks.iter().find(|(id, _)| *id == chunk.id) { + out.extend_from_slice(data); + } + } + out + } +} + +/// `bytes` (a snapshot of `doc`) cut into chunks as a file's content is, with the manifest that names them. +pub fn cut(doc: DocId, bytes: &[u8]) -> Result { + if bytes.is_empty() || bytes.len() > MAX_SNAPSHOT_BYTES { + return Err(format!("a snapshot is 1 byte to {MAX_SNAPSHOT_BYTES} bytes; this one is {}", bytes.len())); + } + let chunker = Chunker::new(FASTCDC_V1).map_err(|e| e.to_string())?; + let mut chunks: Vec<(ChunkId, Vec)> = Vec::new(); + let mut seen = BTreeSet::new(); + let summary = chunk_reader::<_, std::io::Error, _>(&chunker, bytes, |chunk| { + let id = ChunkId::from_bytes(chunk.hash); + if seen.insert(id) { + chunks.push((id, chunk.data.to_vec())); + } + Ok(()) + }) + .map_err(|e| e.to_string())?; + let refs = summary.chunks.iter().map(|c| ChunkRef { id: ChunkId::from_bytes(c.hash), len: c.len }).collect(); + let params = ChunkerParamsId::new(FASTCDC_V1.name).map_err(|e| e.to_string())?; + let manifest = Manifest::new(params, summary.size, FileHash::from_bytes(summary.file_hash), refs).map_err(|e| e.to_string())?; + Ok(Upload { doc, manifest, chunks }) +} + +/// Uploads a snapshot: the chunks the server lacks, then the manifest (and, if the server says it lacks some of them +/// after all, those once more). +pub async fn upload(transport: &T, up: &Upload) -> Result<(), TransportError> { + let ids: Vec = up.chunks.iter().map(|(id, _)| *id).collect(); + let mut missing = BTreeSet::new(); + for batch in ids.chunks(MISSING_QUERY_IDS) { + missing.extend(transport.missing_chunks(batch).await?); + } + for (id, data) in &up.chunks { + if missing.contains(id) { + transport.put_chunk(id, Bytes::from(data.clone())).await?; + } + } + let encoded = Bytes::from(up.manifest.encode()); + match transport.put_manifest(&up.id(), encoded.clone()).await { + Err(TransportError::MissingChunks(lost)) => { + for id in lost { + let (_, data) = up + .chunks + .iter() + .find(|(c, _)| *c == id) + .ok_or_else(|| TransportError::Protocol(format!("the server asked for chunk {id}, which the snapshot doesn't have")))?; + transport.put_chunk(&id, Bytes::from(data.clone())).await?; + } + transport.put_manifest(&up.id(), encoded).await + } + other => other, + } +} + +/// Downloads the snapshot `id` names: its manifest (checked against `id`), then each chunk (checked against its ID and +/// length), the whole checked against the manifest's size (at most [`MAX_SNAPSHOT_BYTES`]) and hash. +pub async fn download(transport: &T, id: &ManifestId) -> Result, TransportError> { + let bad = |what: &str| TransportError::Protocol(format!("the snapshot {id} doesn't read: {what}")); + let manifest = Manifest::decode(&transport.get_manifest(id).await?).map_err(|e| bad(&e.to_string()))?; + if manifest.id() != *id { + return Err(bad("its manifest isn't the one asked for")); + } + if manifest.size() == 0 || manifest.size() > MAX_SNAPSHOT_BYTES as u64 { + return Err(bad("its size is no snapshot's")); + } + let mut out = Vec::with_capacity(usize::try_from(manifest.size()).unwrap_or(0)); + for chunk in manifest.chunks() { + let data = transport.get_chunk(&chunk.id).await?; + if data.len() != chunk.len as usize || ChunkId::from_bytes(*blake3::hash(&data).as_bytes()) != chunk.id { + return Err(bad("a chunk isn't the one named")); + } + out.extend_from_slice(&data); + } + if out.len() as u64 != manifest.size() || FileHash::from_bytes(*blake3::hash(&out).as_bytes()) != *manifest.file_hash() { + return Err(bad("the whole isn't the one named")); + } + Ok(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A snapshot is cut as a file is (the same chunks for the same bytes), each chunk once, and reads back whole. + #[test] + fn a_snapshot_is_cut_into_chunks_and_a_manifest() { + let doc = DocId::generate(); + let bytes: Vec = (0..9_000_000u32).map(|i| (i.wrapping_mul(2_654_435_761) >> 13) as u8).collect(); + let up = cut(doc, &bytes).unwrap(); + assert!(up.chunks.len() > 1, "{} chunks", up.chunks.len()); + assert_eq!(up.manifest.size(), bytes.len() as u64); + assert_eq!(up.bytes(), bytes); + assert_eq!(cut(doc, &bytes).unwrap().id(), up.id(), "the same manifest every time"); + let repeated = vec![7u8; 9_000_000]; + let up = cut(doc, &repeated).unwrap(); + assert!(up.chunks.len() < up.manifest.chunks().len(), "a chunk the snapshot has twice goes up once"); + assert_eq!(up.bytes(), repeated); + assert!(cut(doc, &[]).is_err()); + assert!(cut(doc, &vec![1; MAX_SNAPSHOT_BYTES + 1]).is_err()); + } +} diff --git a/apps/craft-cloud-agent/src/live/copy.rs b/apps/craft-cloud-agent/src/live/copy.rs new file mode 100644 index 0000000..9970b0f --- /dev/null +++ b/apps/craft-cloud-agent/src/live/copy.rs @@ -0,0 +1,541 @@ +//! A live document's local copy: a Loro document that every machine of the account merges from opaque updates, with +//! nothing in it read here. What recents and documents of files share: importing (checked by Loro; a panic inside Loro +//! becomes an error, never a crashed agent), snapshots (whole, or at the version some updates make), and what a log +//! lacks of it, for a push after a restore. + +use std::collections::BTreeMap; + +use craft_cloud_proto::live::MAX_UPDATE_BYTES; +use loro::{ExportMode, IdSpan, LoroDoc, VersionVector}; + +/// Why an update or a stored document was refused. +#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] +pub enum UpdateError { + #[error("the update doesn't import: {0}")] + Malformed(String), + #[error("the recents document is of version {0}; this agent reads version {RECENTS_VERSION}", RECENTS_VERSION = super::recents::RECENTS_VERSION)] + Newer(i64), + #[error("the update breaks a limit: {0}")] + Invalid(&'static str), + /// Loro panicked on it: the document may be half-changed and is read back from its file. + #[error("Loro failed on the update")] + Panicked, +} + +/// What an import brought. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Taken { + /// Applied; whether it held anything this copy didn't have. + Applied { new: bool }, + /// Not wholly applied: some of it depends on updates this copy doesn't have yet. Loro keeps that part and applies + /// it as soon as they come; `needs` is how far each peer's operations must reach for it to be in + /// ([`LiveCopy::covers`]). + Waiting { needs: Vec<(u64, i32)> }, +} + +/// Ranges of each Loro peer's operations (`[start, end)` counters), as a log is known to hold them. +pub type Spans = BTreeMap>; + +/// A version: how far each peer's operations reach. +pub type Version = BTreeMap; + +/// The ranges of operations an update (or snapshot) holds, read from its header without importing it; `None` for +/// bytes Loro can't read. +pub fn spans_of(bytes: &[u8]) -> Option> { + let meta = guarded(|| LoroDoc::decode_import_blob_meta(bytes, true).map_err(|e| UpdateError::Malformed(e.to_string()))).ok()?; + Some(meta.partial_end_vv.iter().map(|(peer, end)| (*peer, meta.partial_start_vv.get(peer).copied().unwrap_or(0), *end)).collect()) +} + +/// Adds `ranges` to `spans`, merging what touches. +pub fn add_spans(spans: &mut Spans, ranges: &[(u64, i32, i32)]) { + for &(peer, start, end) in ranges { + if start >= end { + continue; + } + let list = spans.entry(peer).or_default(); + list.push((start, end)); + list.sort_unstable(); + let mut merged: Vec<(i32, i32)> = Vec::with_capacity(list.len()); + for &(s, e) in list.iter() { + match merged.last_mut() { + Some(last) if s <= last.1 => last.1 = last.1.max(e), + _ => merged.push((s, e)), + } + } + *list = merged; + } +} + +/// Runs `f`, turning a panic inside Loro into [`UpdateError::Panicked`] (never into a crashed agent). +pub(crate) fn guarded(f: impl FnOnce() -> Result) -> Result { + std::panic::catch_unwind(std::panic::AssertUnwindSafe(f)).unwrap_or(Err(UpdateError::Panicked)) +} + +#[cfg(test)] +thread_local! { + static FAIL_ON: std::cell::RefCell>> = const { std::cell::RefCell::new(Vec::new()) }; +} + +/// For tests on this thread: importing `bytes` (alone or in a batch) panics as Loro might. +#[cfg(test)] +pub(crate) fn fail_on(bytes: &[u8]) { + FAIL_ON.with(|fail| fail.borrow_mut().push(bytes.to_vec())); +} + +/// A Loro peer ID this document may use (`u64::MAX` is reserved by Loro). +fn usable_peer(peer: u64) -> u64 { + if peer == u64::MAX { peer - 1 } else { peer } +} + +/// One live document's copy on this machine. +pub struct LiveCopy { + pub(crate) doc: LoroDoc, + /// Loro panicked in a call on this document: its lock is poisoned, so the document is read back from its file and + /// this one let go of without dropping it (dropping a poisoned Loro document panics again). + poisoned: std::cell::Cell, +} + +impl std::fmt::Debug for LiveCopy { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("LiveCopy").field("peer", &self.doc.peer_id()).finish() + } +} + +impl LiveCopy { + /// An empty document, editing as Loro peer `peer` (unique per machine and run). + pub fn new(peer: u64) -> Self { + let doc = LoroDoc::new(); + // A fresh document has no history, so any peer ID is accepted. + let _ = doc.set_peer_id(usable_peer(peer)); + Self { doc, poisoned: std::cell::Cell::new(false) } + } + + /// Runs `f` on the document under [`guarded`]; a panic marks it poisoned. Every Loro call on a document goes + /// through here (it may hold any machine's data). + pub(crate) fn guard(&self, f: impl FnOnce(&LoroDoc) -> Result) -> Result { + if self.poisoned.get() { + return Err(UpdateError::Panicked); + } + let result = guarded(|| f(&self.doc)); + if matches!(result, Err(UpdateError::Panicked)) { + self.poisoned.set(true); + } + result + } + + /// Whether Loro panicked in a call on this document ([`LiveCopy::guard`]): it must be read back from its file, and + /// this one never dropped. + pub fn is_poisoned(&self) -> bool { + self.poisoned.get() + } + + /// Marks the document as one Loro failed in (for tests: a real Loro panic can't be made on demand). + #[cfg(test)] + pub(crate) fn poison(&self) { + self.poisoned.set(true); + } + + /// For tests: updates that "make Loro fail" when imported, alone or in a batch ([`fail_on`]). + #[cfg(test)] + fn failing(updates: &[&[u8]]) { + if FAIL_ON.with(|fail| fail.borrow().iter().any(|f| updates.contains(&f.as_slice()))) { + panic!("a test's update that makes Loro fail"); + } + } + + /// A stored document (a Loro snapshot), editing as `peer`. Of any size: the 1 MiB limit is on updates from the + /// server, not on this machine's own document. + pub fn load(snapshot: &[u8], peer: u64) -> Result { + let copy = Self::new(peer); + match copy.import_any_size(snapshot)? { + Taken::Applied { .. } => Ok(copy), + Taken::Waiting { .. } => Err(UpdateError::Malformed("a stored document that needs more history".into())), + } + } + + /// Merges an update from elsewhere, checked by Loro. Refused only when Loro can't decode it, or it is over the + /// update limit: those are the same on every machine. A refused update leaves the document as it was. + pub fn import(&self, bytes: &[u8]) -> Result { + if bytes.len() > MAX_UPDATE_BYTES { + return Err(UpdateError::Invalid("over 1 MiB")); + } + self.import_any_size(bytes) + } + + /// [`LiveCopy::import`] without the update limit: for a snapshot, or this machine's own stored document. + pub fn import_any_size(&self, bytes: &[u8]) -> Result { + let status = self.guard(|doc| { + #[cfg(test)] + Self::failing(&[bytes]); + doc.import(bytes).map_err(|e| UpdateError::Malformed(e.to_string())) + })?; + let needs: Vec<(u64, i32)> = status.pending.map(|p| p.iter().map(|(peer, (_, end))| (*peer, *end)).collect()).unwrap_or_default(); + let new = status.success.iter().any(|(_, (start, end))| start < end); + Ok(if needs.is_empty() { Taken::Applied { new } } else { Taken::Waiting { needs } }) + } + + /// Imports many updates at once (a catch-up's), checked by Loro as one import would: much faster than one by one + /// on a big document. Whether it brought anything new. The caller still takes each update on its own after this + /// (which is then quick), for what each one did. + pub fn import_batch(&self, updates: &[Vec]) -> Result { + if updates.iter().any(|u| u.len() > MAX_UPDATE_BYTES) { + return Err(UpdateError::Invalid("over 1 MiB")); + } + let before = self.version()?; + // (A batch Loro refuses as a whole is taken one by one by the caller, which finds the one at fault.) + let _ = self.guard(|doc| { + #[cfg(test)] + Self::failing(&updates.iter().map(Vec::as_slice).collect::>()); + doc.import_batch(updates).map_err(|e| UpdateError::Malformed(e.to_string())) + }); + Ok(self.version()? != before) + } + + /// Whether the document has any operations (Loro drops the updates waiting in a document without any when a + /// snapshot comes in: the caller imports them again). + pub fn has_history(&self) -> bool { + self.guard(|doc| Ok(!doc.oplog_vv().is_empty())).unwrap_or(false) + } + + /// The Loro peer this document edits as (a plain field read: no lock). + pub fn peer(&self) -> u64 { + self.doc.peer_id() + } + + /// Whether every peer's operations here reach as far as `needs` says (a waiting update is then in). + pub fn covers(&self, needs: &[(u64, i32)]) -> bool { + self.guard(|doc| { + let vv = doc.oplog_vv(); + Ok(needs.iter().all(|(peer, end)| vv.get(peer).is_some_and(|have| have >= end))) + }) + .unwrap_or(false) + } + + /// How far each peer's operations reach in this copy. + pub fn version(&self) -> Result { + self.guard(|doc| Ok(doc.oplog_vv().iter().map(|(peer, end)| (*peer, *end)).collect())) + } + + /// The whole document (a Loro snapshot): to store it, or to send it up again to a server that lost it. + pub fn snapshot(&self) -> Result, UpdateError> { + self.guard(|doc| doc.export(ExportMode::Snapshot).map_err(|e| UpdateError::Malformed(e.to_string()))) + } + + /// A snapshot of the document as it was at `version` (its history up to there, and the state then; nothing + /// later). `Invalid` if this copy doesn't hold every operation `version` names. + pub fn snapshot_at(&self, version: &Version) -> Result, UpdateError> { + self.guard(|doc| { + let have = doc.oplog_vv(); + if version.iter().any(|(peer, end)| *end > 0 && have.get(peer).is_none_or(|h| h < end)) { + return Err(UpdateError::Invalid("the copy doesn't hold that version")); + } + let vv: VersionVector = version.iter().filter(|(_, end)| **end > 0).map(|(peer, end)| (*peer, *end)).collect(); + let frontiers = doc.vv_to_frontiers(&vv); + doc.export(ExportMode::SnapshotAt { version: std::borrow::Cow::Owned(frontiers) }).map_err(|e| UpdateError::Malformed(e.to_string())) + }) + } + + /// The changes (Loro commits: peer, first and end counter) of this document that `has` doesn't wholly hold, in + /// causal order: by Lamport time (a change's dependencies all have a smaller one), then peer. Any prefix of it, + /// sent after what `has` holds, applies on its own. + pub fn missing(&self, has: &Spans) -> Result, UpdateError> { + self.guard(|doc| Ok(Self::missing_in(doc, has))) + } + + fn missing_in(doc: &LoroDoc, has: &Spans) -> Vec<(u64, i32, i32)> { + let mut out = Vec::new(); + let ends: Vec<(u64, i32)> = doc.oplog_vv().iter().map(|(peer, end)| (*peer, *end)).collect(); + for (peer, end) in ends { + let mut at = 0; + while at < end { + let Some(change) = doc.get_change(loro::ID::new(peer, at)) else { break }; + let (start, stop) = (change.id.counter, change.id.counter.saturating_add(i32::try_from(change.len).unwrap_or(i32::MAX))); + if stop <= at { + break; + } + // The pieces of the change `has` lacks (a change may be partly sent: a big one goes in pieces), each + // at the Lamport time of its first operation. + let mut from = start; + for &(s, e) in has.get(&peer).map(Vec::as_slice).unwrap_or_default() { + if e <= from || s >= stop { + continue; + } + if s > from { + out.push((change.lamport.saturating_add(u32::try_from(from - start).unwrap_or(0)), peer, from, s)); + } + from = from.max(e); + } + if from < stop { + out.push((change.lamport.saturating_add(u32::try_from(from - start).unwrap_or(0)), peer, from, stop)); + } + at = stop; + } + } + out.sort_unstable(); + out.into_iter().map(|(_, peer, start, stop)| (peer, start, stop)).collect() + } + + /// `changes` (whole Loro commits) as one update. + /// Each peer's pieces must follow on from each other ([`LiveCopy::export_prefix`] keeps them so): Loro panics on a + /// range export of one peer's operations with a gap. + pub fn export_changes(&self, changes: &[(u64, i32, i32)]) -> Result, UpdateError> { + let mut spans: BTreeMap = BTreeMap::new(); + for &(peer, start, end) in changes { + let span = spans.entry(peer).or_insert((start, end)); + span.0 = span.0.min(start); + span.1 = span.1.max(end); + } + let spans: Vec = spans.into_iter().map(|(peer, (start, end))| IdSpan::new(peer, start, end)).collect(); + self.guard(|doc| doc.export(ExportMode::updates_in_range(spans)).map_err(|e| UpdateError::Malformed(e.to_string()))) + } + + /// The longest start of `changes` that fits in `max` bytes as one update (at least one change; `Invalid` if that + /// one doesn't fit in `hard_max`), with how many changes it holds. + pub fn export_prefix(&self, changes: &[(u64, i32, i32)], max: usize, hard_max: usize) -> Result<(Vec, usize), UpdateError> { + let Some(&(peer, start, end)) = changes.first() else { return Err(UpdateError::Invalid("nothing to send")) }; + // The most operations of the first change that fit, by doubling from one and then halving the gap, so no + // export is much bigger than a part (a big change isn't exported whole for every piece of it). A change too big + // for a part goes in pieces: its operations depend on the ones before them, so a start of it applies once + // those before it are in. + let total = end - start; + let (mut ops, mut one) = (0, Vec::new()); + let (mut try_ops, mut over) = (1, None); + while over.is_none() && ops < total { + let n = try_ops.min(total); + let bytes = self.export_changes(&[(peer, start, start + n)])?; + if bytes.len() <= max { + (ops, one) = (n, bytes); + try_ops = n.saturating_mul(2); + } else { + over = Some(n); + } + } + if let Some(mut high) = over { + while high - ops > 1 { + let mid = ops + (high - ops) / 2; + let bytes = self.export_changes(&[(peer, start, start + mid)])?; + if bytes.len() <= max { + (ops, one) = (mid, bytes); + } else { + high = mid; + } + } + } + if ops == 0 { + // One operation over a part: sent alone if it fits an update at all. + let bytes = self.export_changes(&[(peer, start, start + 1)])?; + return if bytes.len() <= hard_max { Ok((bytes, 1)) } else { Err(UpdateError::Invalid("one operation is larger than an update may be")) }; + } + if ops < total { + return Ok((one, 1)); + } + // Only as far as each peer's pieces follow on from each other (a piece after a gap waits for a later part). + let mut ends: BTreeMap = BTreeMap::new(); + let mut usable = 0; + for &(peer, start, end) in changes { + if ends.get(&peer).is_some_and(|last| *last != start) { + break; + } + ends.insert(peer, end); + usable += 1; + } + let changes = &changes[..usable.max(1)]; + // The most changes that fit: doubling, then halving the gap. + let (mut fits, mut best) = (1, one); + let mut too_many = None; + let mut try_n = 2; + while too_many.is_none() && fits < changes.len() { + let n = try_n.min(changes.len()); + let bytes = self.export_changes(&changes[..n])?; + if bytes.len() <= max { + (fits, best) = (n, bytes); + try_n = n * 2; + } else { + too_many = Some(n); + } + } + if let Some(mut high) = too_many { + while high - fits > 1 { + let mid = fits + (high - fits) / 2; + let bytes = self.export_changes(&changes[..mid])?; + if bytes.len() <= max { + (fits, best) = (mid, bytes); + } else { + high = mid; + } + } + } + Ok((best, fits)) + } +} + +/// What the server's log holds of a document, as a version, at any sequence number from an anchor on: so the agent can +/// make a snapshot of exactly the updates 1..=`upto` when the server asks for one, without keeping the updates. +/// +/// The anchor is a number with the version of everything the log holds up to it (from the server's snapshot, from 0 +/// after a catch-up from the start, or worked out from the copy when nothing else is in it); every update taken after +/// it adds its ranges. A version with a gap (an update whose history comes later in the log) is no version at all: +/// no snapshot can be made there. +#[derive(Debug, Clone, Default)] +pub struct LogVersion { + anchor: Option<(u64, Version)>, + /// The updates taken since the anchor, in order: number and ranges. + since: Vec<(u64, Ranges)>, +} + +/// The ranges of operations of an update: peer, first and end counter. +pub type Ranges = Vec<(u64, i32, i32)>; + +/// Most updates [`LogVersion`] keeps past its anchor before it forgets it (and works one out again when it can). +const MAX_SINCE: usize = 100_000; + +impl LogVersion { + /// Anchored at `seq` with `version`. + pub fn anchored(seq: u64, version: Version) -> Self { + Self { anchor: Some((seq, version)), since: Vec::new() } + } + + /// Nothing known. + pub fn unknown() -> Self { + Self::default() + } + + pub fn is_known(&self) -> bool { + self.anchor.is_some() + } + + /// Update `seq`, holding `ranges`, is in the log (taken in order after what was noted before). + pub fn note(&mut self, seq: u64, ranges: Vec<(u64, i32, i32)>) { + let Some((at, _)) = &self.anchor else { return }; + if seq <= *at { + return; + } + if self.since.len() >= MAX_SINCE { + *self = Self::unknown(); + return; + } + self.since.push((seq, ranges)); + } + + /// The version the log's updates up to `upto` make, if it is known and has no gap. + pub fn at(&self, upto: u64) -> Option { + let (at, base) = self.anchor.as_ref()?; + if upto < *at { + return None; + } + let mut spans = Spans::new(); + let base: Vec<(u64, i32, i32)> = base.iter().map(|(peer, end)| (*peer, 0, *end)).collect(); + add_spans(&mut spans, &base); + for (_, ranges) in self.since.iter().take_while(|(seq, _)| *seq <= upto) { + add_spans(&mut spans, ranges); + } + // Each peer's operations must run from its first one without a hole. + let mut version = Version::new(); + for (peer, list) in spans { + match list.as_slice() { + [(0, end)] => { + version.insert(peer, *end); + } + [] => {} + _ => return None, + } + } + Some(version) + } + + /// The version at `upto` is `version` (a snapshot was made there): what came before is forgotten. + pub fn rebase(&mut self, upto: u64, version: Version) { + self.since.retain(|(seq, _)| *seq > upto); + self.anchor = Some((upto, version)); + } +} + +/// The version of a snapshot (or update) from its header: how far each peer's operations reach. +pub fn version_of(bytes: &[u8]) -> Option { + let ranges = spans_of(bytes)?; + let mut version = Version::new(); + for (peer, _, end) in ranges { + let reach = version.entry(peer).or_insert(0); + *reach = (*reach).max(end); + } + Some(version) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn text_update(doc: &LoroDoc, at: usize, s: &str) -> Vec { + let before = doc.oplog_vv(); + doc.get_text("text").insert(at, s).unwrap(); + doc.commit(); + doc.export(ExportMode::updates(&before)).unwrap() + } + + /// An update Loro refuses leaves the copy as it was (Ruling: validation without a scratch copy). + #[test] + fn a_refused_update_leaves_the_copy_as_it_was() { + let copy = LiveCopy::new(1); + let other = LoroDoc::new(); + other.set_peer_id(2).unwrap(); + copy.import(&text_update(&other, 0, "hello")).unwrap(); + let before = copy.snapshot().unwrap(); + let good = text_update(&other, 5, " world"); + // Damaged: a byte in the middle flipped (Loro's checksum catches it), cut short, junk, empty. + let mut flipped = good.clone(); + let mid = flipped.len() / 2; + flipped[mid] ^= 0x55; + for bad in [flipped, good[..good.len() / 2].to_vec(), vec![1, 2, 3, 4, 5], Vec::new()] { + assert!(copy.import(&bad).is_err(), "{bad:?}"); + assert_eq!(copy.snapshot().unwrap(), before, "unchanged"); + assert!(!copy.is_poisoned()); + } + assert_eq!(copy.import(&good).unwrap(), Taken::Applied { new: true }); + assert_eq!(copy.doc.get_text("text").to_string(), "hello world"); + assert_eq!(copy.import(&good).unwrap(), Taken::Applied { new: false }, "a repeat brings nothing new"); + assert!(copy.import(&vec![0; MAX_UPDATE_BYTES + 1]).is_err()); + } + + /// A snapshot at a version holds exactly what the log held there, and nothing later. + #[test] + fn a_snapshot_at_a_version_holds_nothing_later() { + let a = LoroDoc::new(); + a.set_peer_id(1).unwrap(); + let copy = LiveCopy::new(9); + let mut log = LogVersion::anchored(0, Version::new()); + let u1 = text_update(&a, 0, "one "); + let u2 = text_update(&a, 4, "two "); + let u3 = text_update(&a, 8, "three"); + for (seq, u) in [(1, &u1), (2, &u2), (3, &u3)] { + copy.import(u).unwrap(); + log.note(seq, spans_of(u).unwrap()); + } + let at2 = log.at(2).unwrap(); + let snapshot = copy.snapshot_at(&at2).unwrap(); + let fresh = LiveCopy::load(&snapshot, 5).unwrap(); + assert_eq!(fresh.doc.get_text("text").to_string(), "one two "); + fresh.import(&u3).unwrap(); + assert_eq!(fresh.doc.get_text("text").to_string(), "one two three", "the rest of the log applies after it"); + // A version the copy doesn't hold is refused. + let mut beyond = at2.clone(); + beyond.insert(1, 1000); + assert!(matches!(copy.snapshot_at(&beyond), Err(UpdateError::Invalid(_)))); + // A log with a gap (an update whose history comes after it) has no version there. + let b = LoroDoc::new(); + b.set_peer_id(2).unwrap(); + let b1 = text_update(&b, 0, "x"); + let b2 = text_update(&b, 1, "y"); + let mut gap = LogVersion::anchored(0, Version::new()); + gap.note(1, spans_of(&b2).unwrap()); + gap.note(2, spans_of(&b1).unwrap()); + assert_eq!(gap.at(1), None, "b2 without b1"); + assert!(gap.at(2).is_some(), "both"); + assert_eq!(gap.at(0), Some(Version::new())); + // Rebased: what came before is gone, what came after stays. + log.rebase(2, at2.clone()); + assert_eq!(log.at(1), None); + assert_eq!(log.at(2), Some(at2)); + assert!(log.at(3).is_some()); + assert_eq!(version_of(&snapshot).unwrap().get(&1), Some(&8)); + } +} diff --git a/apps/craft-cloud-agent/src/live/docs.rs b/apps/craft-cloud-agent/src/live/docs.rs new file mode 100644 index 0000000..87e6f99 --- /dev/null +++ b/apps/craft-cloud-agent/src/live/docs.rs @@ -0,0 +1,212 @@ +//! `GET /v1/docs`: the local API's WebSocket for live documents (the messages are in +//! `craft-cloud-proto::local::docs`). One connection per app connection; it may open many documents. The connection +//! turns the app's messages into [`DocsCall`]s for the hub and the hub's [`DocOut`]s into messages for the app. +//! +//! Everything from an app is hostile: a text message that isn't a request is answered `error`; a binary frame without +//! a header or of a kind an app doesn't send ends the connection; an update over 1 MiB, presence over 4 KiB or an +//! import over the snapshot limit is refused (and never reaches the hub). An app that doesn't take a message within 10 +//! s is dropped, and so is one whose queue at the hub fills up; its documents stay with the agent. An app that goes +//! away mid-import leaves nothing behind: the import goes up only once it is whole. Signed out, every `open` is +//! answered `not_live {not_signed_in}`. + +use std::collections::HashMap; +use std::sync::Arc; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; + +use axum::extract::State; +use axum::extract::ws::rejection::WebSocketUpgradeRejection; +use axum::extract::ws::{Message, WebSocket, WebSocketUpgrade}; +use axum::response::{IntoResponse, Response}; +use craft_cloud_proto::api::codes as api_codes; +use craft_cloud_proto::live::{MAX_PRESENCE_BYTES, MAX_SNAPSHOT_BYTES, MAX_UPDATE_BYTES}; +use craft_cloud_proto::local::docs::{IMPORT, MAX_BINARY_BYTES, PRESENCE, STATE, UPDATE, decode_binary, encode_binary}; +use craft_cloud_proto::local::{DocsEvent, DocsRequest, MAX_LOCAL_JSON_BYTES, NotLiveReason}; +use tokio::sync::mpsc; + +use super::hub::{ConnId, DOCS_QUEUE, DocOut, DocsCall, HubHandle}; +use crate::local_api::{ApiState, LocalError}; + +/// A message the app doesn't take within this long drops it. +const SEND_TIMEOUT: Duration = Duration::from_secs(10); + +static NEXT_CONN: AtomicU64 = AtomicU64::new(1); + +/// `GET /v1/docs`: the WebSocket of an app's live documents. +pub async fn docs_socket(State(api): State>, upgrade: Result) -> Response { + let upgrade = match upgrade { + Ok(upgrade) => upgrade, + Err(e) => return LocalError::new(e.status(), api_codes::INVALID, e.body_text()).into_response(), + }; + upgrade + .max_message_size(MAX_BINARY_BYTES.max(MAX_LOCAL_JSON_BYTES)) + .max_frame_size(MAX_BINARY_BYTES.max(MAX_LOCAL_JSON_BYTES)) + .on_upgrade(move |socket| serve(socket, api)) +} + +/// Why the connection ends. +enum End { + /// The app went, or broke the rules, or is too slow. + App, + /// The hub dropped the connection (or stopped: signed out). + Hub, +} + +/// One app connection. +struct Link { + conn: ConnId, + hub: Option, + /// The hub's messages for this connection (`None` until it has a hub). + from_hub: Option>, + /// The app's updates so far, per document number (their `seq`). + seqs: HashMap, +} + +impl Link { + /// The hub, connected to on first use (signed in after the app connected: it is there now). + async fn hub(&mut self, api: &ApiState) -> Option { + if self.hub.is_none() + && let Some(hub) = api.control.hub() + { + let (out, from_hub) = mpsc::channel(DOCS_QUEUE); + if hub.docs(DocsCall::Connect { conn: self.conn, out }).await.is_ok() { + self.hub = Some(hub); + self.from_hub = Some(from_hub); + } + } + self.hub.clone() + } +} + +async fn serve(mut socket: WebSocket, api: Arc) { + let mut link = Link { conn: NEXT_CONN.fetch_add(1, Ordering::Relaxed), hub: None, from_hub: None, seqs: HashMap::new() }; + link.hub(&api).await; + let end = loop { + tokio::select! { + message = socket.recv() => { + let message = match message { + Some(Ok(message)) => message, + Some(Err(_)) | None => break End::App, + }; + match on_message(&mut socket, &mut link, &api, message).await { + Ok(true) => {} + Ok(false) => break End::App, + Err(end) => break end, + } + } + out = async { match link.from_hub.as_mut() { Some(rx) => rx.recv().await, None => std::future::pending().await } } => match out { + Some(out) => { + if !send_out(&mut socket, out).await { + break End::App; + } + } + // The hub let go of this connection (it didn't keep up, or the hub stopped). + None => break End::Hub, + }, + } + }; + if let (End::App, Some(hub)) = (end, link.hub.as_ref()) { + let _ = hub.docs(DocsCall::Gone { conn: link.conn }).await; + } + let _ = socket.send(Message::Close(None)).await; +} + +/// One message from the app: `Ok(false)` ends the connection (the app is gone, or broke the rules), `Err` too, for a +/// hub that stopped. +async fn on_message(socket: &mut WebSocket, link: &mut Link, api: &ApiState, message: Message) -> Result { + match message { + Message::Text(text) => { + let request = match serde_json::from_str::(&text) { + Ok(request) => request, + Err(e) => { + let event = + DocsEvent::Error { id: None, seq: None, code: api_codes::INVALID.into(), message: format!("not a docs request: {e}") }; + return Ok(send_json(socket, &event).await); + } + }; + match request { + DocsRequest::Open { id, path, app } => match link.hub(api).await { + Some(hub) => hub.docs(DocsCall::Open { conn: link.conn, id, path, app }).await.map(|()| true).map_err(|_| End::Hub), + None => { + let event = DocsEvent::NotLive { + id, + reason: NotLiveReason::NotSignedIn, + message: Some("the Craft Cloud agent isn't signed in".into()), + }; + Ok(send_json(socket, &event).await) + } + }, + DocsRequest::Close { doc } => match link.hub.clone() { + Some(hub) => hub.docs(DocsCall::Close { conn: link.conn, doc }).await.map(|()| true).map_err(|_| End::Hub), + None => Ok(true), + }, + } + } + Message::Binary(bytes) => { + // A frame without a header, or of a kind an app never sends: the app is broken or hostile. + let Some((kind, id, body)) = decode_binary(&bytes) else { return Ok(false) }; + let Some(hub) = link.hub.clone() else { return Ok(kind == UPDATE || kind == PRESENCE || kind == IMPORT) }; + let refuse = |seq: Option, message: String| DocsEvent::Error { id: Some(id), seq, code: api_codes::TOO_LARGE.into(), message }; + let call = match kind { + UPDATE => { + let seq = link.seqs.entry(id).or_insert(0); + *seq += 1; + if body.len() > MAX_UPDATE_BYTES { + return Ok(send_json(socket, &refuse(Some(*seq), format!("an update is at most {MAX_UPDATE_BYTES} bytes"))).await); + } + DocsCall::Update { conn: link.conn, id, seq: *seq, bytes: body.to_vec() } + } + PRESENCE => { + if body.is_empty() || body.len() > MAX_PRESENCE_BYTES { + return Ok(true); + } + DocsCall::Presence { conn: link.conn, id, bytes: body.to_vec() } + } + IMPORT => { + if body.len() > MAX_SNAPSHOT_BYTES { + return Ok(send_json(socket, &refuse(None, format!("an import is at most {MAX_SNAPSHOT_BYTES} bytes"))).await); + } + DocsCall::Import { conn: link.conn, id, bytes: body.to_vec() } + } + _ => return Ok(false), + }; + hub.docs(call).await.map(|()| true).map_err(|_| End::Hub) + } + Message::Ping(_) | Message::Pong(_) => Ok(true), + Message::Close(_) => Ok(false), + } +} + +/// Sends one of the hub's messages to the app; false when the app is gone (or too slow). +async fn send_out(socket: &mut WebSocket, out: DocOut) -> bool { + match out { + DocOut::Opened { id, doc, state, snapshot, waiting } => { + send_json(socket, &DocsEvent::Opened { id, doc, state }).await && send_binary(socket, STATE, id, &snapshot).await && { + let mut ok = true; + for update in &waiting { + ok = ok && send_binary(socket, UPDATE, id, update).await; + } + ok + } + } + DocOut::NeedsImport { id } => send_json(socket, &DocsEvent::NeedsImport { id }).await, + DocOut::NotLive { id, reason, message } => send_json(socket, &DocsEvent::NotLive { id, reason, message: Some(message) }).await, + DocOut::Ack { id, seq } => send_json(socket, &DocsEvent::Ack { id, seq }).await, + DocOut::Refused { id, seq, code, message } => send_json(socket, &DocsEvent::Error { id, seq, code: code.into(), message }).await, + DocOut::Update { id, bytes } => send_binary(socket, UPDATE, id, &bytes).await, + DocOut::Presence { id, bytes } => send_binary(socket, PRESENCE, id, &bytes).await, + DocOut::Status { doc, online, waiting, deleted, stopped } => { + send_json(socket, &DocsEvent::Status { doc, online, waiting, deleted, stopped }).await + } + } +} + +async fn send_json(socket: &mut WebSocket, event: &DocsEvent) -> bool { + let Ok(text) = serde_json::to_string(event) else { return true }; + matches!(tokio::time::timeout(SEND_TIMEOUT, socket.send(Message::Text(text.into()))).await, Ok(Ok(()))) +} + +async fn send_binary(socket: &mut WebSocket, kind: u8, id: u64, bytes: &[u8]) -> bool { + let frame = encode_binary(kind, id, bytes); + matches!(tokio::time::timeout(SEND_TIMEOUT, socket.send(Message::Binary(frame.into()))).await, Ok(Ok(()))) +} diff --git a/apps/craft-cloud-agent/src/live/hub.rs b/apps/craft-cloud-agent/src/live/hub.rs index 6196cfb..9c4be03 100644 --- a/apps/craft-cloud-agent/src/live/hub.rs +++ b/apps/craft-cloud-agent/src/live/hub.rs @@ -1,26 +1,38 @@ -//! The live hub's task: owns the account's [`Replica`], answers the local API's recents calls, and keeps the live -//! connection to the server up while the agent is signed in (connecting again with backoff, from 1 s doubling to 5 -//! minutes with jitter; every 10 minutes for a server that speaks another live protocol). Offline it keeps working: -//! adds are kept on disk and reach the server once the connection is back. While syncing is paused the connection is -//! held the same way (closed, and not dialed until resumed). +//! The live hub's task: owns the account's [`Replica`], answers the local API's recents and documents calls, and keeps +//! the live connection to the server up while the agent is signed in (connecting again with backoff, from 1 s doubling +//! to 5 minutes with jitter; every 10 minutes for a server that speaks another live protocol). Offline it keeps +//! working: adds and edits are kept on disk and reach the server once the connection is back. While syncing is paused +//! the connection is held the same way (closed, and not dialed until resumed). +//! +//! **Documents of files** ([`DocsCall`]): an app opens a path; the hub finds its node in the sync state (a file not +//! uploaded yet gets a sync pass started for it, and the open waits up to [`NODE_WAIT`] for its node), and opens its +//! document through the replica. Its state goes to the app, and then every remote change and presence as they come, +//! and to the document's other apps on this machine every change one of them makes. A document's status (online, how +//! many changes wait, whether its file was deleted) goes to its apps when it changes. Nothing the hub sends an app ever +//! waits for the app: a connection whose queue is full is dropped. -use std::collections::BTreeMap; +use std::collections::{BTreeMap, HashMap}; +use std::path::PathBuf; +use std::sync::Arc; use std::time::Duration; -use craft_cloud_proto::live::{AppName, ClientFrame, MAX_SUBSCRIPTIONS, ServerFrame}; -use craft_cloud_proto::local::{MAX_MACHINE_BYTES, MAX_TITLE_BYTES, NewRecent, RecentEntry, recent_path}; +use craft_cloud_proto::DocId; +use craft_cloud_proto::live::{AppName, ClientFrame, MAX_SNAPSHOT_BYTES, MAX_SUBSCRIPTIONS, PRESENCE_PER_SEC, ServerFrame}; +use craft_cloud_proto::local::{DocState, MAX_MACHINE_BYTES, MAX_TITLE_BYTES, NewRecent, NotLiveReason, RecentEntry, recent_path}; use craft_cloud_proto::{NodeId, SpaceId}; use futures_util::{SinkExt, StreamExt}; use tokio::sync::{mpsc, oneshot, watch}; use tokio::time::Instant; use tokio_tungstenite::tungstenite::Message; +use super::blobs::{Upload, download, upload}; use super::link::{DialError, LiveSocket, dial}; -use super::replica::{AddError, LinkError, Replica}; +use super::replica::{AddError, DocEvent, LinkError, LocalError, Opening, Replica}; use super::store::DiskStore; use crate::account::DataDir; -use crate::http::{DeviceToken, ServerUrl}; +use crate::http::{DeviceToken, HttpTransport, ServerUrl}; use crate::local_api::Control; +use craft_cloud_proto::ManifestId; /// What the hub needs to know about the account. #[derive(Debug, Clone)] @@ -31,6 +43,81 @@ pub struct HubConfig { pub token: DeviceToken, /// This machine's name in the entries it adds (the device name). pub machine: String, + /// The synced folder (the paths apps open documents by are in it). + pub folder: PathBuf, +} + +/// How long an open of a file that isn't uploaded yet waits for its node (a sync pass is started for it). +pub const NODE_WAIT: Duration = Duration::from_secs(5); +/// How often the hub looks again for the nodes opens wait for. +const NODE_LOOK_EVERY: Duration = Duration::from_millis(100); +/// How often documents of files are written whole (their local changes are in their journals at once). +const DOCUMENTS_FLUSH_EVERY: Duration = Duration::from_secs(2); +/// Messages one docs connection may have queued before it is dropped (an app that doesn't read). +pub const DOCS_QUEUE: usize = 1024; + +/// One app connection of the docs WebSocket. +pub type ConnId = u64; + +/// What the hub sends a docs connection, in order. +#[derive(Debug, Clone)] +pub enum DocOut { + /// The document opened as `id` is live: its state (a snapshot, then updates still waiting for their history). + Opened { id: u64, doc: DocId, state: DocState, snapshot: Vec, waiting: Vec> }, + /// Send the file's content as an import. + NeedsImport { id: u64 }, + /// The file opens as a plain file. + NotLive { id: u64, reason: NotLiveReason, message: String }, + /// The app's update `seq` of `id` is on disk. + Ack { id: u64, seq: u64 }, + /// A request was refused. + Refused { id: Option, seq: Option, code: &'static str, message: String }, + /// A change from another machine or app. + Update { id: u64, bytes: Arc> }, + /// Presence from another machine or app. + Presence { id: u64, bytes: Arc> }, + /// How the document stands now. + Status { doc: DocId, online: bool, waiting: u64, deleted: bool, stopped: bool }, +} + +/// What a docs connection asks the hub. +#[derive(Debug)] +pub enum DocsCall { + /// A new connection: what the hub sends it goes to `out`. + Connect { + conn: ConnId, + out: mpsc::Sender, + }, + Open { + conn: ConnId, + id: u64, + path: String, + app: String, + }, + Import { + conn: ConnId, + id: u64, + bytes: Vec, + }, + Update { + conn: ConnId, + id: u64, + seq: u64, + bytes: Vec, + }, + Presence { + conn: ConnId, + id: u64, + bytes: Vec, + }, + Close { + conn: ConnId, + doc: DocId, + }, + /// The connection ended. + Gone { + conn: ConnId, + }, } /// Why a recents call was refused. @@ -63,6 +150,8 @@ enum Request { }, /// Hold the live connection (closed, and not dialed) or let it run again; answered once done. Pause(bool, oneshot::Sender<()>), + /// A docs connection's request (answered on its own channel). + Docs(DocsCall), } /// How the local API reaches the hub. Cheap to clone. @@ -93,6 +182,11 @@ impl HubHandle { self.tx.send(Request::Pause(paused, reply)).await.map_err(|_| HubError::Gone)?; rx.await.map_err(|_| HubError::Gone) } + + /// Hands a docs connection's request to the hub (its answers come on the connection's channel). + pub async fn docs(&self, call: DocsCall) -> Result<(), HubError> { + self.tx.send(Request::Docs(call)).await.map_err(|_| HubError::Gone) + } } /// The running hub; dropping it stops it (what it was told is on disk already). It runs on a thread of its own with @@ -140,15 +234,86 @@ struct NodeLookup { } impl NodeLookup { - fn node(&mut self, path: &str) -> Option { + /// The sync state, read again if `state.json` changed. + fn state(&mut self) -> Option<&craft_cloud_sync::State> { let file = self.data.path().join(crate::account::STATE_FILE); let stamp = std::fs::metadata(&file).ok().and_then(|m| Some((m.modified().ok()?, m.len()))); if stamp != self.seen { self.seen = stamp; self.state = self.data.load_state().ok().map(|s| s.state); } - self.state.as_ref()?.base_at(&craft_cloud_sync::rel(path)) + self.state.as_ref() + } + + fn node(&mut self, path: &str) -> Option { + self.state()?.base_at(&craft_cloud_sync::rel(path)) + } + + /// The node of the file at `path`, if the sync knows one there (a folder is none). + fn file_node(&mut self, path: &str) -> Option { + let state = self.state()?; + let node = state.base_at(&craft_cloud_sync::rel(path))?; + matches!(state.base.get(&node).map(|e| &e.kind), Some(craft_cloud_sync::BaseKind::File { .. })).then_some(node) } + + /// Whether the sync still has the file node `node` (`false` once it was deleted, here or elsewhere). + fn has_file(&mut self, node: NodeId) -> bool { + // (Unknown: no state to look in. Not said to be deleted.) + self.state().is_none_or(|state| state.base.contains_key(&node)) + } +} + +/// A token bucket for one app's presence in one document: [`PRESENCE_PER_SEC`] a second. +struct PresenceBudget { + tokens: f64, + at: Instant, +} + +impl PresenceBudget { + fn new() -> Self { + Self { tokens: f64::from(PRESENCE_PER_SEC), at: Instant::now() } + } + + fn spend(&mut self) -> bool { + let now = Instant::now(); + let rate = f64::from(PRESENCE_PER_SEC); + self.tokens = (self.tokens + now.saturating_duration_since(self.at).as_secs_f64() * rate).min(rate); + self.at = now; + if self.tokens >= 1.0 { + self.tokens -= 1.0; + true + } else { + false + } + } +} + +/// One docs connection: where its messages go, and the documents it has open (by its own numbers). +struct DocsConn { + out: mpsc::Sender, + handles: BTreeMap, + presence: HashMap, +} + +/// An open waiting for its file's node. +struct Lookup { + conn: ConnId, + id: u64, + path: String, + app: AppName, + until: Instant, +} + +/// The hub's documents of files: connections, opens waiting, imports asked for, and the status each was last told. +#[derive(Default)] +struct Docs { + conns: HashMap, + /// Opens waiting for a document's join, in order. + waiting: BTreeMap>, + /// The open asked for a document's import. + importer: BTreeMap, + lookups: Vec, + told: BTreeMap, } /// `text` cut to `max` bytes at a character boundary, without control characters. @@ -185,12 +350,51 @@ const PING_EVERY: Duration = Duration::from_secs(30); const IDLE_TIMEOUT: Duration = Duration::from_secs(90); const SEND_TIMEOUT: Duration = Duration::from_secs(30); +/// What a snapshot's upload or download says when it is done (for the connection `generation` it was for). +enum BlobDone { + Uploaded { generation: u64, doc: DocId, manifest: ManifestId, result: Result<(), String> }, + Downloaded { generation: u64, doc: DocId, manifest: ManifestId, result: Result, String> }, +} + +/// Snapshots on their way up or down through the chunk API, and the frames waiting for them. +struct Blobs { + /// The chunk API (`None` if it couldn't be set up: every upload and download then fails, and is said). + transport: Option>, + done: mpsc::UnboundedSender, + /// The connection the transfers under way are for: one that ended makes their results no news. + generation: u64, + /// Uploads under way, by document: the document's frames wait behind them, in order. + uploading: BTreeMap>, + held: BTreeMap>, +} + +/// The document a frame is about, if any. +fn frame_doc(frame: &ClientFrame) -> Option { + match frame { + ClientFrame::Hello { .. } => None, + ClientFrame::Create { doc, .. } + | ClientFrame::Subscribe { doc, .. } + | ClientFrame::Unsubscribe { doc } + | ClientFrame::Update { doc, .. } + | ClientFrame::Snapshot { doc, .. } + | ClientFrame::Presence { doc, .. } + | ClientFrame::SubscribeLog { doc, .. } + | ClientFrame::CreateDocument { doc, .. } => Some(*doc), + } +} + +/// Whether `frame` names the snapshot `manifest`. +fn names(frame: &ClientFrame, manifest: ManifestId) -> bool { + matches!(frame, ClientFrame::Snapshot { manifest: m, .. } | ClientFrame::CreateDocument { snapshot: m, .. } if *m == manifest) +} + struct State { config: HubConfig, replica: Replica, + blobs: Blobs, lookup: NodeLookup, lists: BTreeMap>, - socket: Option, + socket: Option, control: Control, said: Option, /// When to dial again, and the backoff that grows with every connection lost (reset once welcomed). @@ -198,6 +402,12 @@ struct State { delay: Duration, /// Syncing is paused: no connection is dialed (see [`HubHandle::set_paused`]). paused: bool, + docs: Docs, +} + +/// A refusal sent to an app. +fn refusal(id: Option, seq: Option, code: &'static str, message: impl Into) -> DocOut { + DocOut::Refused { id, seq, code, message: message.into() } } impl State { @@ -214,30 +424,397 @@ impl State { } } - /// Hands the changed lists to their followers, says the notes and writes what changed. - fn settle(&mut self) { + /// Hands the changed lists to their followers and what happened to documents to their apps, says the notes and + /// writes what changed. Frames to send (an event may call for some). + fn settle(&mut self) -> Vec { for (app, entries) in self.replica.take_changed() { if let Some(list) = self.lists.get(&app) { let push_stopped = self.replica.push_stopped(&app); list.send_replace(Listing { entries, push_stopped }); } } + let frames = self.doc_events(); + self.tell_status(); for note in self.replica.take_notes() { self.say(note); } if let Err(e) = self.replica.flush() { - self.say(format!("the recents couldn't be written to disk: {e}")); + self.say(format!("the live documents couldn't be written to disk: {e}")); } self.control.set_online(self.replica.is_up()); + frames + } + + /// Sends `message` to docs connection `conn`; one that can't take it (it doesn't read, or it is gone) is dropped. + fn tell(&mut self, conn: ConnId, message: DocOut) -> Vec { + let Some(c) = self.docs.conns.get(&conn) else { return Vec::new() }; + if c.out.try_send(message).is_ok() { + return Vec::new(); + } + self.gone(conn) + } + + /// The opens of `doc`: every connection and number it is open as. + fn handles_of(&self, doc: DocId) -> Vec<(ConnId, u64)> { + let mut out = Vec::new(); + for (conn, c) in &self.docs.conns { + out.extend(c.handles.iter().filter(|(_, d)| **d == doc).map(|(id, _)| (*conn, *id))); + } + out.sort_unstable(); + out + } + + /// Hands what happened to documents of files to the apps that have them open or wait for them. + fn doc_events(&mut self) -> Vec { + let mut frames = Vec::new(); + for event in self.replica.take_events() { + match event { + DocEvent::Joined(doc) => { + self.docs.importer.remove(&doc); + for (conn, id) in self.docs.waiting.remove(&doc).unwrap_or_default() { + frames.extend(self.deliver(conn, id, doc)); + } + } + DocEvent::NeedsImport(doc) => frames.extend(self.ask_import(doc)), + DocEvent::NotLive(doc, reason, message) => { + self.docs.importer.remove(&doc); + for (conn, id) in self.docs.waiting.remove(&doc).unwrap_or_default() { + frames.extend(self.tell(conn, DocOut::NotLive { id, reason, message: message.clone() })); + } + } + DocEvent::Remote(doc, bytes) => { + let bytes = Arc::new(bytes); + for (conn, id) in self.handles_of(doc) { + frames.extend(self.tell(conn, DocOut::Update { id, bytes: bytes.clone() })); + } + } + DocEvent::Presence(doc, bytes) => { + let bytes = Arc::new(bytes); + for (conn, id) in self.handles_of(doc) { + frames.extend(self.tell(conn, DocOut::Presence { id, bytes: bytes.clone() })); + } + } + } + } + frames + } + + /// The first open waiting for `doc` is asked for its import; with none left, the join is given up on. + fn ask_import(&mut self, doc: DocId) -> Vec { + let first = self.docs.waiting.get(&doc).and_then(|w| w.first().copied()); + match first { + Some((conn, id)) => { + self.docs.importer.insert(doc, (conn, id)); + self.tell(conn, DocOut::NeedsImport { id }) + } + None => { + self.docs.importer.remove(&doc); + self.replica.abandon_join(doc) + } + } + } + + /// Opens `doc`, live here, as `id` on `conn`: its state goes to the app. + fn deliver(&mut self, conn: ConnId, id: u64, doc: DocId) -> Vec { + let Some((snapshot, waiting)) = self.replica.document_state(doc) else { + return self.tell(conn, DocOut::NotLive { id, reason: NotLiveReason::Offline, message: "the document can't be read here".into() }); + }; + let Some(c) = self.docs.conns.get_mut(&conn) else { return Vec::new() }; + c.handles.insert(id, doc); + self.replica.set_open(doc, true); + let (online, waiting_n) = self.replica.document_status(doc).unwrap_or((false, 0)); + let state = DocState { online, waiting: waiting_n, bytes: snapshot.len() as u64 }; + self.tell(conn, DocOut::Opened { id, doc, state, snapshot, waiting }) + } + + /// A docs connection ended (or was dropped): its documents are closed, its waiting opens dropped, an import it was + /// asked for asked of the next open waiting (or the join given up on). + fn gone(&mut self, conn: ConnId) -> Vec { + let Some(c) = self.docs.conns.remove(&conn) else { return Vec::new() }; + for doc in c.handles.values() { + self.replica.set_open(*doc, false); + } + self.docs.lookups.retain(|l| l.conn != conn); + for waiting in self.docs.waiting.values_mut() { + waiting.retain(|(c, _)| *c != conn); + } + let orphaned: Vec = self.docs.importer.iter().filter(|(_, (c, _))| *c == conn).map(|(doc, _)| *doc).collect(); + let mut frames = Vec::new(); + for doc in orphaned { + frames.extend(self.ask_import(doc)); + } + self.docs.waiting.retain(|_, w| !w.is_empty()); + frames + } + + /// The path an app named, relative to the synced folder (one given absolute must be inside it). + fn relative(&self, path: &str) -> Option { + let given = std::path::Path::new(path); + if given.is_absolute() { + let inside = given.strip_prefix(&self.config.folder).ok()?; + let names: Option> = inside.components().map(|c| c.as_os_str().to_str()).collect(); + return recent_path(&names?.join("/")); + } + recent_path(path) + } + + /// An app opens `path` as `id`. + fn open(&mut self, conn: ConnId, id: u64, path: &str, app: &str) -> Vec { + let not_live = |reason, message: &str| DocOut::NotLive { id, reason, message: message.to_owned() }; + let Some(c) = self.docs.conns.get(&conn) else { return Vec::new() }; + if c.handles.contains_key(&id) + || self.docs.waiting.values().any(|w| w.contains(&(conn, id))) + || self.docs.lookups.iter().any(|l| l.conn == conn && l.id == id) + { + return self.tell(conn, refusal(Some(id), None, craft_cloud_proto::api::codes::INVALID, "that number is in use on this connection")); + } + let Ok(app) = AppName::new(app) else { + return self + .tell(conn, refusal(Some(id), None, craft_cloud_proto::api::codes::INVALID, "app names are 1-32 lowercase letters, digits or '-'")); + }; + let Some(rel) = self.relative(path) else { + return self.tell(conn, not_live(NotLiveReason::NotInFolder, "that isn't a path inside the synced folder")); + }; + match std::fs::metadata(self.config.folder.join(&rel)) { + Ok(m) if m.is_file() && m.len() > MAX_SNAPSHOT_BYTES as u64 => { + return self.tell(conn, not_live(NotLiveReason::TooLarge, "the file is over the limit for live documents")); + } + Ok(m) if m.is_file() => {} + _ => return self.tell(conn, not_live(NotLiveReason::NotInFolder, "there is no such file in the synced folder")), + } + match self.lookup.file_node(&rel) { + Some(node) => self.open_node(conn, id, node, &app), + None => { + // Not uploaded yet: a pass for it, and a few seconds for its node. + self.control.request_pass(); + self.docs.lookups.push(Lookup { conn, id, path: rel, app, until: Instant::now() + NODE_WAIT }); + Vec::new() + } + } + } + + /// Opens the document of the file node `node` as `id` on `conn`. + fn open_node(&mut self, conn: ConnId, id: u64, node: NodeId, app: &AppName) -> Vec { + let (doc, opening, mut frames) = self.replica.open_document(node, app); + match opening { + Opening::Live => frames.extend(self.deliver(conn, id, doc)), + Opening::Joining => { + self.docs.waiting.entry(doc).or_default().push((conn, id)); + // The server lacks it and the open asked for its import is gone: this one is asked. + if self.replica.needs_import(doc) && !self.docs.importer.contains_key(&doc) { + frames.extend(self.ask_import(doc)); + } + } + Opening::NotLive(reason, message) => frames.extend(self.tell(conn, DocOut::NotLive { id, reason, message })), + } + frames + } + + /// Looks again for the nodes opens wait for: found, opened; past their wait, the file opens as a plain file. + fn look_again(&mut self) -> Vec { + let now = Instant::now(); + let mut frames = Vec::new(); + for l in std::mem::take(&mut self.docs.lookups) { + match self.lookup.file_node(&l.path) { + Some(node) => frames.extend(self.open_node(l.conn, l.id, node, &l.app)), + None if now >= l.until => frames.extend(self.tell( + l.conn, + DocOut::NotLive { + id: l.id, + reason: NotLiveReason::NotSynced, + message: "the file isn't uploaded yet; open it again in a moment".into(), + }, + )), + None => self.docs.lookups.push(l), + } + } + frames + } + + /// Tells the apps of each open document how it stands, when that changed. + fn tell_status(&mut self) { + let open: Vec = { + let mut docs: Vec = self.docs.conns.values().flat_map(|c| c.handles.values().copied()).collect(); + docs.sort_unstable(); + docs.dedup(); + docs + }; + self.docs.told.retain(|doc, _| open.contains(doc)); + for doc in open { + let (online, waiting) = self.replica.document_status(doc).unwrap_or((false, 0)); + let deleted = self.replica.document_node(doc).is_some_and(|node| !self.lookup.has_file(node)); + let stopped = self.replica.is_stopped(doc); + let now = (online, waiting, deleted, stopped); + if self.docs.told.get(&doc) == Some(&now) { + continue; + } + self.docs.told.insert(doc, now); + let mut conns: Vec = self.handles_of(doc).into_iter().map(|(conn, _)| conn).collect(); + conns.dedup(); + for conn in conns { + let _ = self.tell(conn, DocOut::Status { doc, online, waiting, deleted, stopped }); + } + } + } + + /// A docs connection's request. + fn docs_call(&mut self, call: DocsCall) -> Vec { + match call { + DocsCall::Connect { conn, out } => { + self.docs.conns.insert(conn, DocsConn { out, handles: BTreeMap::new(), presence: HashMap::new() }); + Vec::new() + } + DocsCall::Open { conn, id, path, app } => self.open(conn, id, &path, &app), + DocsCall::Import { conn, id, bytes } => { + let doc = self.docs.importer.iter().find(|(_, who)| **who == (conn, id)).map(|(doc, _)| *doc); + let Some(doc) = doc else { + return self.tell(conn, refusal(Some(id), None, craft_cloud_proto::api::codes::INVALID, "no import was asked for")); + }; + match self.replica.import_document(doc, &bytes) { + Ok(frames) => frames, + Err((reason, message)) => { + self.docs.importer.remove(&doc); + let mut frames = Vec::new(); + for (conn, id) in self.docs.waiting.remove(&doc).unwrap_or_default() { + frames.extend(self.tell(conn, DocOut::NotLive { id, reason, message: message.clone() })); + } + frames + } + } + } + DocsCall::Update { conn, id, seq, bytes } => { + let Some(doc) = self.docs.conns.get(&conn).and_then(|c| c.handles.get(&id)).copied() else { + return self.tell(conn, refusal(Some(id), Some(seq), craft_cloud_proto::api::codes::NOT_FOUND, "that document isn't open here")); + }; + match self.replica.local_update(doc, &bytes) { + Ok(mut frames) => { + frames.extend(self.tell(conn, DocOut::Ack { id, seq })); + // The document's other apps on this machine get it at once. + let bytes = Arc::new(bytes); + for (other, other_id) in self.handles_of(doc) { + if (other, other_id) != (conn, id) { + frames.extend(self.tell(other, DocOut::Update { id: other_id, bytes: bytes.clone() })); + } + } + frames + } + Err(LocalError::Invalid(message)) => { + self.tell(conn, refusal(Some(id), Some(seq), craft_cloud_proto::api::codes::INVALID, message)) + } + Err(LocalError::Io(e)) => { + let message = format!("the change couldn't be written to disk: {e}"); + self.tell(conn, refusal(Some(id), Some(seq), craft_cloud_proto::api::codes::INTERNAL, message)) + } + } + } + DocsCall::Presence { conn, id, bytes } => { + let Some(c) = self.docs.conns.get_mut(&conn) else { return Vec::new() }; + let Some(doc) = c.handles.get(&id).copied() else { return Vec::new() }; + // Over the budget: dropped (the next one says where the cursor is). + if !c.presence.entry(id).or_insert_with(PresenceBudget::new).spend() { + return Vec::new(); + } + let mut frames = self.replica.presence(doc, &bytes); + let bytes = Arc::new(bytes); + for (other, other_id) in self.handles_of(doc) { + if (other, other_id) != (conn, id) { + frames.extend(self.tell(other, DocOut::Presence { id: other_id, bytes: bytes.clone() })); + } + } + frames + } + DocsCall::Close { conn, doc } => { + if let Some(c) = self.docs.conns.get_mut(&conn) { + let ids: Vec = c.handles.iter().filter(|(_, d)| **d == doc).map(|(id, _)| *id).collect(); + for id in ids { + c.handles.remove(&id); + c.presence.remove(&id); + self.replica.set_open(doc, false); + } + } + Vec::new() + } + DocsCall::Gone { conn } => self.gone(conn), + } + } + + /// Starts the snapshot uploads and downloads the replica asks for: each in a task of its own, done when it says so + /// ([`State::blob_done`]). While a document's upload is under way, its frames wait behind it. + fn start_blobs(&mut self) { + let uploads: Vec = self.replica.take_uploads(); + let fetches = self.replica.take_fetches(); + if self.socket.is_none() { + return; + } + let generation = self.blobs.generation; + for up in uploads { + let (doc, manifest) = (up.doc, up.id()); + self.blobs.uploading.entry(doc).or_default().push(manifest); + let (transport, done) = (self.blobs.transport.clone(), self.blobs.done.clone()); + tokio::spawn(async move { + let result = match transport { + Some(t) => upload(t.as_ref(), &up).await.map_err(|e| e.to_string()), + None => Err("the server's API can't be reached".into()), + }; + let _ = done.send(BlobDone::Uploaded { generation, doc, manifest, result }); + }); + } + for (doc, manifest) in fetches { + let (transport, done) = (self.blobs.transport.clone(), self.blobs.done.clone()); + tokio::spawn(async move { + let result = match transport { + Some(t) => download(t.as_ref(), &manifest).await.map_err(|e| e.to_string()), + None => Err("the server's API can't be reached".into()), + }; + let _ = done.send(BlobDone::Downloaded { generation, doc, manifest, result }); + }); + } + } + + /// A snapshot's upload or download is done: an upload's document's frames go (the one naming a snapshot that + /// didn't go up is dropped, and the replica told), a download goes to the replica. Frames to send. + fn blob_done(&mut self, done: BlobDone) -> Vec { + match done { + BlobDone::Uploaded { generation, .. } | BlobDone::Downloaded { generation, .. } if generation != self.blobs.generation => Vec::new(), + BlobDone::Uploaded { doc, manifest, result, .. } => { + if let Err(e) = &result { + self.say(format!("a live document's snapshot couldn't go up to the server: {e}")); + self.replica.upload_failed(doc, e); + if let Some(held) = self.blobs.held.get_mut(&doc) { + held.retain(|f| !names(f, manifest)); + } + } + let left = self.blobs.uploading.get_mut(&doc).map(|list| { + if let Some(i) = list.iter().position(|m| *m == manifest) { + list.remove(i); + } + list.len() + }); + if left == Some(0) { + self.blobs.uploading.remove(&doc); + return self.blobs.held.remove(&doc).unwrap_or_default(); + } + Vec::new() + } + BlobDone::Downloaded { doc, manifest, result, .. } => self.replica.on_snapshot(doc, manifest, result), + } } /// Sends frames; false when the connection broke (it is dropped). async fn send(&mut self, frames: Vec) -> bool { + self.start_blobs(); if self.socket.is_none() { return true; } let mut encoded = Vec::with_capacity(frames.len()); for frame in frames { + // Behind an upload of its document: it goes once that is done. + if let Some(doc) = frame_doc(&frame) + && self.blobs.uploading.contains_key(&doc) + { + self.blobs.held.entry(doc).or_default().push(frame); + continue; + } match frame.encode() { Ok(bytes) => encoded.push(bytes), Err(e) => self.say(format!("a live frame couldn't be encoded: {e}")), @@ -245,7 +822,7 @@ impl State { } let Some(socket) = self.socket.as_mut() else { return true }; for bytes in encoded { - if !matches!(tokio::time::timeout(SEND_TIMEOUT, socket.send(Message::Binary(bytes.into()))).await, Ok(Ok(()))) { + if !socket.send(Message::Binary(bytes.into())) { self.drop_socket(); return false; } @@ -260,6 +837,14 @@ impl State { } self.replica.disconnected(); self.control.set_online(false); + self.new_generation(); + } + + /// Transfers under way and the frames held behind them are for a connection that ended: no news any more. + fn new_generation(&mut self) { + self.blobs.generation += 1; + self.blobs.uploading.clear(); + self.blobs.held.clear(); } fn entry(&mut self, app: &AppName, new: &NewRecent) -> Result { @@ -283,6 +868,10 @@ impl State { match request { // The run loop's to handle (it owns the dialing). Request::Pause(..) => {} + Request::Docs(call) => { + let frames = self.docs_call(call); + self.send(frames).await; + } Request::Follow { app, reply } => { if !self.lists.contains_key(&app) && self.lists.len() >= MAX_SUBSCRIPTIONS { let _ = reply.send(Err(HubError::Invalid(format!("the agent follows at most {MAX_SUBSCRIPTIONS} apps' recents")))); @@ -336,9 +925,13 @@ async fn run( paused: bool, ) { let lookup = NodeLookup { data: config.data.clone(), seen: None, state: None }; + let (blob_tx, mut blob_rx) = mpsc::unbounded_channel(); + let transport = HttpTransport::new(config.server.clone(), &config.token).ok().map(Arc::new); + let blobs = Blobs { transport, done: blob_tx, generation: 0, uploading: BTreeMap::new(), held: BTreeMap::new() }; let mut state = State { config, replica, + blobs, lookup, lists: BTreeMap::new(), socket: None, @@ -347,22 +940,29 @@ async fn run( retry_at: Instant::now(), delay: RETRY_FIRST, paused, + docs: Docs::default(), }; state.settle(); let mut dialing: Option>> = None; let mut last_heard = Instant::now(); let mut ping = tokio::time::interval(PING_EVERY); let mut resend = tokio::time::interval(RESEND_EVERY); + let mut look = tokio::time::interval(NODE_LOOK_EVERY); + let mut write_documents = tokio::time::interval(DOCUMENTS_FLUSH_EVERY); loop { if state.socket.is_none() && dialing.is_none() && !state.paused && Instant::now() >= state.retry_at { let (server, token) = (state.config.server.clone(), state.config.token.clone()); dialing = Some(tokio::spawn(async move { dial(&server, &token).await })); } let (up, retry_at) = (state.socket.is_some(), state.retry_at); + let writer_gone = state.socket.as_ref().map(|s| s.gone.clone()); tokio::select! { _ = &mut stop => { - if let Some(mut socket) = state.socket.take() { - close_within(&mut socket, CLOSE_TIMEOUT).await; + if let Some(socket) = state.socket.take() { + socket.close().await; + } + if let Err(e) = state.replica.flush_documents() { + state.say(format!("the live documents couldn't be written to disk: {e}")); } return; } @@ -373,10 +973,11 @@ async fn run( if let Some(task) = dialing.take() { task.abort(); } - if let Some(mut socket) = state.socket.take() { - close_within(&mut socket, CLOSE_TIMEOUT).await; + if let Some(socket) = state.socket.take() { + socket.close().await; } state.replica.disconnected(); + state.new_generation(); // Offline before the caller hears it, so the status it reads next says so. state.control.set_online(false); } else { @@ -393,7 +994,8 @@ async fn run( dialing = None; match dialed { Ok(Ok(socket)) => { - state.socket = Some(socket); + state.new_generation(); + state.socket = Some(Conn::new(socket)); last_heard = Instant::now(); let hello = state.replica.connected(); state.send(hello).await; @@ -405,7 +1007,15 @@ async fn run( Err(_) => state.back_off(None), } } - message = async { match state.socket.as_mut() { Some(socket) => socket.next().await, None => std::future::pending().await } } => { + Some(done) = blob_rx.recv() => { + let frames = state.blob_done(done); + state.send(frames).await; + } + // The writer gave up (the server took nothing for the send timeout): the connection is gone. + () = async { match &writer_gone { Some(gone) => gone.notified().await, None => std::future::pending().await } } => { + state.drop_socket(); + } + message = async { match state.socket.as_mut() { Some(socket) => socket.incoming.next().await, None => std::future::pending().await } } => { last_heard = Instant::now(); let frame = match message { Some(Ok(Message::Binary(bytes))) => ServerFrame::decode(&bytes).map_err(|e| LinkError::Retry(format!("a live frame doesn't decode: {e}"))), @@ -422,8 +1032,8 @@ async fn run( state.send(frames).await; } Err(e) => { - if let Some(mut socket) = state.socket.take() { - close_within(&mut socket, CLOSE_TIMEOUT).await; + if let Some(socket) = state.socket.take() { + socket.close().await; } state.drop_socket(); state.back_off(None); @@ -438,7 +1048,7 @@ async fn run( if last_heard.elapsed() > IDLE_TIMEOUT { state.drop_socket(); } else if let Some(socket) = state.socket.as_mut() - && !matches!(tokio::time::timeout(SEND_TIMEOUT, socket.send(Message::Ping(Vec::new().into()))).await, Ok(Ok(()))) + && !socket.send(Message::Ping(Vec::new().into())) { state.drop_socket(); } @@ -447,15 +1057,72 @@ async fn run( let frames = state.replica.resend(); state.send(frames).await; } + _ = look.tick(), if !state.docs.lookups.is_empty() => { + let frames = state.look_again(); + state.send(frames).await; + } + _ = write_documents.tick() => { + if let Err(e) = state.replica.flush_documents() { + state.say(format!("the live documents couldn't be written to disk: {e}")); + } + } () = tokio::time::sleep_until(retry_at), if !up && dialing.is_none() && !state.paused => {} } - state.settle(); + let frames = state.settle(); + state.send(frames).await; } } /// How long closing the live connection may take before it is simply dropped. pub const CLOSE_TIMEOUT: Duration = Duration::from_secs(5); +/// The live connection: what comes in is read by the hub's loop, what goes out is written by a task of its own, so +/// the hub keeps reading while a long send (many changes made offline, a push, a snapshot) waits for the server to +/// read. (If both sides only wrote, each waiting for the other to read, the connection would stall until a send +/// timeout ended it, again and again.) A writer that can't send within [`SEND_TIMEOUT`] ends, and with it the +/// connection. +struct Conn { + out: mpsc::UnboundedSender, + incoming: futures_util::stream::SplitStream, + writer: tokio::task::JoinHandle<()>, + /// Told when the writer gave up. + gone: Arc, +} + +impl Conn { + fn new(socket: LiveSocket) -> Self { + let (mut sink, incoming) = socket.split(); + let (out, mut queue) = mpsc::unbounded_channel::(); + let gone = Arc::new(tokio::sync::Notify::new()); + let tell = gone.clone(); + let writer = tokio::spawn(async move { + while let Some(message) = queue.recv().await { + if !matches!(tokio::time::timeout(SEND_TIMEOUT, sink.send(message)).await, Ok(Ok(()))) { + tell.notify_one(); + return; + } + } + close_within(&mut sink, CLOSE_TIMEOUT).await; + }); + Self { out, incoming, writer, gone } + } + + /// Queues a message; false when the writer is gone (the connection with it). + fn send(&self, message: Message) -> bool { + !self.writer.is_finished() && self.out.send(message).is_ok() + } + + /// Closes the connection: what is queued goes first, then the close, all within [`CLOSE_TIMEOUT`]. + async fn close(self) { + let Self { out, incoming, mut writer, .. } = self; + drop(out); + drop(incoming); + if tokio::time::timeout(CLOSE_TIMEOUT, &mut writer).await.is_err() { + writer.abort(); + } + } +} + /// Closes the live connection, waiting at most `limit`: on a half-open connection with a full send buffer (a laptop /// that just changed networks) the close frame may never go out, and the hub must not hang on it. The connection is /// dropped by the caller either way. diff --git a/apps/craft-cloud-agent/src/live/mod.rs b/apps/craft-cloud-agent/src/live/mod.rs index 431a692..f786ee8 100644 --- a/apps/craft-cloud-agent/src/live/mod.rs +++ b/apps/craft-cloud-agent/src/live/mod.rs @@ -1,14 +1,23 @@ -//! The live hub: the agent's side of live documents. Today: each app's recents, shared by every machine of the account. +//! The live hub: the agent's side of live documents: each app's recents, shared by every machine of the account, and +//! documents of files, opened by apps by path. //! -//! - [`recents`]: the recents document (a CRDT merged from opaque updates) and the list apps see. +//! - [`blobs`]: snapshots up and down through the chunk API. +//! - [`copy`]: a document's local copy (a Loro document merged from opaque updates, checked by Loro) and what the +//! server's log holds of it. +//! - [`recents`]: the recents document and the list apps see. //! - [`replica`]: the protocol state for one account: documents, what it has of each server log, unacked updates. -//! - [`store`]: where documents are kept between runs (`/live//`, one file each, atomic). +//! - [`store`]: where documents are kept between runs (`/live//`, one file each, atomic; a journal +//! beside each document of a file). //! - [`link`]: the WebSocket to the server's live service. //! - [`hub`]: the task that ties them together while the agent is signed in, and serves the local API. +//! - [`docs`]: the local API's documents WebSocket. //! //! The local API's recents WebSocket ([`recents_socket`]) is a client of the hub like any other: an app subscribes //! to its list (the whole list comes now and after every change) and adds entries (answered once they are on disk). +pub mod blobs; +pub mod copy; +pub mod docs; pub mod hub; pub mod link; pub mod recents; diff --git a/apps/craft-cloud-agent/src/live/recents.rs b/apps/craft-cloud-agent/src/live/recents.rs index 4079274..5f30495 100644 --- a/apps/craft-cloud-agent/src/live/recents.rs +++ b/apps/craft-cloud-agent/src/live/recents.rs @@ -23,9 +23,13 @@ use std::collections::HashSet; use craft_cloud_proto::NodeId; -use craft_cloud_proto::live::MAX_UPDATE_BYTES; use craft_cloud_proto::local::{MAX_MACHINE_BYTES, MAX_RECENTS, MAX_TITLE_BYTES, RecentEntry, recent_path}; -use loro::{ExportMode, IdSpan, LoroDoc, LoroValue, ValueOrContainer}; +use loro::{ExportMode, LoroDoc, LoroValue, ValueOrContainer}; + +use super::copy::{LiveCopy, Taken}; +pub use super::copy::{Spans, UpdateError, add_spans, spans_of}; +#[cfg(test)] +use craft_cloud_proto::live::MAX_UPDATE_BYTES; /// The key of one add: random, never reused. pub type AddKey = [u8; 16]; @@ -39,20 +43,6 @@ const MAX_ENTRY_JSON: usize = 16 * 1024; const ENTRIES: &str = "entries"; const SCHEMA: &str = "schema"; -/// Why an update or a stored document was refused. -#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] -pub enum UpdateError { - #[error("the recents update doesn't import: {0}")] - Malformed(String), - #[error("the recents document is of version {0}; this agent reads version {RECENTS_VERSION}")] - Newer(i64), - #[error("the recents update breaks a limit: {0}")] - Invalid(&'static str), - /// Loro panicked on it: the document may be half-changed and is read back from its file. - #[error("Loro failed on the recents update")] - Panicked, -} - fn check_entry(entry: &RecentEntry) -> Result<(), UpdateError> { if recent_path(&entry.path).as_deref() != Some(entry.path.as_str()) { return Err(UpdateError::Invalid("a path that isn't a clean relative path")); @@ -70,17 +60,17 @@ fn hex(key: &AddKey) -> String { key.iter().map(|b| format!("{b:02x}")).collect() } -/// A Loro peer ID this document may use (`u64::MAX` is reserved by Loro). -fn usable_peer(peer: u64) -> u64 { - if peer == u64::MAX { peer - 1 } else { peer } +/// One app's recents document: a [`LiveCopy`] (whose generic calls it offers as its own) read as a list. +pub struct Recents { + copy: LiveCopy, } -/// One app's recents document. -pub struct Recents { - doc: LoroDoc, - /// Loro panicked in a call on this document: its lock is poisoned, so the document is read back from its file and - /// this one let go of without dropping it (dropping a poisoned Loro document panics again). - poisoned: std::cell::Cell, +impl std::ops::Deref for Recents { + type Target = LiveCopy; + + fn deref(&self) -> &LiveCopy { + &self.copy + } } impl std::fmt::Debug for Recents { @@ -158,83 +148,30 @@ pub enum Imported { Waiting { needs: Vec<(u64, i32)>, changed: bool }, } -/// Ranges of each Loro peer's operations (`[start, end)` counters), as a log is known to hold them. -pub type Spans = std::collections::BTreeMap>; - -/// The ranges of operations an update (or snapshot) holds, read from its header without importing it; `None` for -/// bytes Loro can't read. -pub fn spans_of(bytes: &[u8]) -> Option> { - let meta = guarded(|| LoroDoc::decode_import_blob_meta(bytes, true).map_err(|e| UpdateError::Malformed(e.to_string()))).ok()?; - Some(meta.partial_end_vv.iter().map(|(peer, end)| (*peer, meta.partial_start_vv.get(peer).copied().unwrap_or(0), *end)).collect()) -} - -/// Adds `ranges` to `spans`, merging what touches. -pub fn add_spans(spans: &mut Spans, ranges: &[(u64, i32, i32)]) { - for &(peer, start, end) in ranges { - if start >= end { - continue; - } - let list = spans.entry(peer).or_default(); - list.push((start, end)); - list.sort_unstable(); - let mut merged: Vec<(i32, i32)> = Vec::with_capacity(list.len()); - for &(s, e) in list.iter() { - match merged.last_mut() { - Some(last) if s <= last.1 => last.1 = last.1.max(e), - _ => merged.push((s, e)), - } - } - *list = merged; - } -} - -/// Runs `f`, turning a panic inside Loro into [`UpdateError::Panicked`] (never into a crashed agent). -fn guarded(f: impl FnOnce() -> Result) -> Result { - std::panic::catch_unwind(std::panic::AssertUnwindSafe(f)).unwrap_or(Err(UpdateError::Panicked)) -} - impl Recents { /// An empty document, editing as Loro peer `peer` (unique per machine and run). pub fn new(peer: u64) -> Self { - let doc = LoroDoc::new(); - // A fresh document has no history, so any peer ID is accepted. - let _ = doc.set_peer_id(usable_peer(peer)); - Self { doc, poisoned: std::cell::Cell::new(false) } + Self { copy: LiveCopy::new(peer) } } - /// Runs `f` on the document under [`guarded`]; a panic marks it poisoned. Every Loro call on a document goes - /// through here (it may hold any machine's data). - fn guard(&self, f: impl FnOnce(&LoroDoc) -> Result) -> Result { - if self.poisoned.get() { - return Err(UpdateError::Panicked); - } - let result = guarded(|| f(&self.doc)); - if matches!(result, Err(UpdateError::Panicked)) { - self.poisoned.set(true); - } - result + /// The document as a recents list (a copy loaded or caught up as any live document). + pub fn from_copy(copy: LiveCopy) -> Self { + Self { copy } } - /// Whether Loro panicked in a call on this document ([`Recents::guard`]): it must be read back from its file, and - /// this one never dropped. - pub fn is_poisoned(&self) -> bool { - self.poisoned.get() + /// The generic copy underneath. + pub fn into_copy(self) -> LiveCopy { + self.copy } - /// Marks the document as one Loro failed in (for tests: a real Loro panic can't be made on demand). - #[cfg(test)] - pub(crate) fn poison(&self) { - self.poisoned.set(true); + fn guard(&self, f: impl FnOnce(&LoroDoc) -> Result) -> Result { + self.copy.guard(f) } /// A stored document (a Loro snapshot), editing as `peer`. Of any size: the 1 MiB limit is on updates from the /// server, not on this machine's own document. pub fn load(snapshot: &[u8], peer: u64) -> Result { - let recents = Self::new(peer); - match recents.import_any_size(snapshot)? { - Imported::Applied { .. } => Ok(recents), - Imported::Waiting { .. } => Err(UpdateError::Malformed("a stored document that needs more history".into())), - } + LiveCopy::load(snapshot, peer).map(Self::from_copy) } /// Merges an update (or a snapshot) from elsewhere. Refused only when Loro can't decode it, or it is over the @@ -243,178 +180,25 @@ impl Recents { /// disagree about what they took, and their later updates would wait on each other for good. Entries that don't /// read, or break the rules, are only ever hidden by the list ([`read_entry`]). pub fn import(&self, bytes: &[u8]) -> Result { - if bytes.len() > MAX_UPDATE_BYTES { - return Err(UpdateError::Invalid("over 1 MiB")); - } - self.import_any_size(bytes) + let before = self.view(); + let taken = self.copy.import(bytes)?; + Ok(self.imported(taken, &before)) } /// [`Recents::import`] without the update limit: for this machine's own stored document (or a simulation merging /// machines' documents), never for an update from the server. pub fn import_any_size(&self, bytes: &[u8]) -> Result { let before = self.view(); - let status = self.guard(|doc| doc.import(bytes).map_err(|e| UpdateError::Malformed(e.to_string())))?; - let changed = self.view() != before; - let needs: Vec<(u64, i32)> = status.pending.map(|p| p.iter().map(|(peer, (_, end))| (*peer, *end)).collect()).unwrap_or_default(); - Ok(if needs.is_empty() { Imported::Applied { changed } } else { Imported::Waiting { needs, changed } }) + let taken = self.copy.import_any_size(bytes)?; + Ok(self.imported(taken, &before)) } - /// Whether the document has any operations (Loro drops the updates waiting in a document without any when a - /// snapshot comes in: the caller imports them again). - pub fn has_history(&self) -> bool { - self.guard(|doc| Ok(!doc.oplog_vv().is_empty())).unwrap_or(false) - } - - /// The Loro peer this document edits as (a plain field read: no lock). - pub fn peer(&self) -> u64 { - self.doc.peer_id() - } - - /// Whether every peer's operations here reach as far as `needs` says (a waiting update is then in). - pub fn covers(&self, needs: &[(u64, i32)]) -> bool { - self.guard(|doc| { - let vv = doc.oplog_vv(); - Ok(needs.iter().all(|(peer, end)| vv.get(peer).is_some_and(|have| have >= end))) - }) - .unwrap_or(false) - } - - /// The whole document (a Loro snapshot): to store it, or to send it up again to a server that lost it. - pub fn snapshot(&self) -> Result, UpdateError> { - self.guard(|doc| doc.export(ExportMode::Snapshot).map_err(|e| UpdateError::Malformed(e.to_string()))) - } - - /// The changes (Loro commits: peer, first and end counter) of this document that `has` doesn't wholly hold, in - /// causal order: by Lamport time (a change's dependencies all have a smaller one), then peer. Any prefix of it, - /// sent after what `has` holds, applies on its own. - pub fn missing(&self, has: &Spans) -> Result, UpdateError> { - self.guard(|doc| Ok(Self::missing_in(doc, has))) - } - - fn missing_in(doc: &LoroDoc, has: &Spans) -> Vec<(u64, i32, i32)> { - let mut out = Vec::new(); - let ends: Vec<(u64, i32)> = doc.oplog_vv().iter().map(|(peer, end)| (*peer, *end)).collect(); - for (peer, end) in ends { - let mut at = 0; - while at < end { - let Some(change) = doc.get_change(loro::ID::new(peer, at)) else { break }; - let (start, stop) = (change.id.counter, change.id.counter.saturating_add(i32::try_from(change.len).unwrap_or(i32::MAX))); - if stop <= at { - break; - } - // The pieces of the change `has` lacks (a change may be partly sent: a big one goes in pieces), each - // at the Lamport time of its first operation. - let mut from = start; - for &(s, e) in has.get(&peer).map(Vec::as_slice).unwrap_or_default() { - if e <= from || s >= stop { - continue; - } - if s > from { - out.push((change.lamport.saturating_add(u32::try_from(from - start).unwrap_or(0)), peer, from, s)); - } - from = from.max(e); - } - if from < stop { - out.push((change.lamport.saturating_add(u32::try_from(from - start).unwrap_or(0)), peer, from, stop)); - } - at = stop; - } - } - out.sort_unstable(); - out.into_iter().map(|(_, peer, start, stop)| (peer, start, stop)).collect() - } - - /// `changes` (whole Loro commits) as one update. - /// Each peer's pieces must follow on from each other ([`Recents::export_prefix`] keeps them so): Loro panics on a - /// range export of one peer's operations with a gap. - pub fn export_changes(&self, changes: &[(u64, i32, i32)]) -> Result, UpdateError> { - let mut spans: std::collections::BTreeMap = std::collections::BTreeMap::new(); - for &(peer, start, end) in changes { - let span = spans.entry(peer).or_insert((start, end)); - span.0 = span.0.min(start); - span.1 = span.1.max(end); - } - let spans: Vec = spans.into_iter().map(|(peer, (start, end))| IdSpan::new(peer, start, end)).collect(); - self.guard(|doc| doc.export(ExportMode::updates_in_range(spans)).map_err(|e| UpdateError::Malformed(e.to_string()))) - } - - /// The longest start of `changes` that fits in `max` bytes as one update (at least one change; `Invalid` if that - /// one doesn't fit in `hard_max`), with how many changes it holds. - pub fn export_prefix(&self, changes: &[(u64, i32, i32)], max: usize, hard_max: usize) -> Result<(Vec, usize), UpdateError> { - let Some(&(peer, start, end)) = changes.first() else { return Err(UpdateError::Invalid("nothing to send")) }; - // The most operations of the first change that fit, by doubling from one and then halving the gap, so no - // export is much bigger than a part (a big change isn't exported whole for every piece of it). A change too big - // for a part goes in pieces: its operations depend on the ones before them, so a start of it applies once - // those before it are in. - let total = end - start; - let (mut ops, mut one) = (0, Vec::new()); - let (mut try_ops, mut over) = (1, None); - while over.is_none() && ops < total { - let n = try_ops.min(total); - let bytes = self.export_changes(&[(peer, start, start + n)])?; - if bytes.len() <= max { - (ops, one) = (n, bytes); - try_ops = n.saturating_mul(2); - } else { - over = Some(n); - } - } - if let Some(mut high) = over { - while high - ops > 1 { - let mid = ops + (high - ops) / 2; - let bytes = self.export_changes(&[(peer, start, start + mid)])?; - if bytes.len() <= max { - (ops, one) = (mid, bytes); - } else { - high = mid; - } - } - } - if ops == 0 { - // One operation over a part: sent alone if it fits an update at all. - let bytes = self.export_changes(&[(peer, start, start + 1)])?; - return if bytes.len() <= hard_max { Ok((bytes, 1)) } else { Err(UpdateError::Invalid("one operation is larger than an update may be")) }; - } - if ops < total { - return Ok((one, 1)); - } - // Only as far as each peer's pieces follow on from each other (a piece after a gap waits for a later part). - let mut ends: std::collections::BTreeMap = std::collections::BTreeMap::new(); - let mut usable = 0; - for &(peer, start, end) in changes { - if ends.get(&peer).is_some_and(|last| *last != start) { - break; - } - ends.insert(peer, end); - usable += 1; - } - let changes = &changes[..usable.max(1)]; - // The most changes that fit: doubling, then halving the gap. - let (mut fits, mut best) = (1, one); - let mut too_many = None; - let mut try_n = 2; - while too_many.is_none() && fits < changes.len() { - let n = try_n.min(changes.len()); - let bytes = self.export_changes(&changes[..n])?; - if bytes.len() <= max { - (fits, best) = (n, bytes); - try_n = n * 2; - } else { - too_many = Some(n); - } - } - if let Some(mut high) = too_many { - while high - fits > 1 { - let mid = fits + (high - fits) / 2; - let bytes = self.export_changes(&changes[..mid])?; - if bytes.len() <= max { - (fits, best) = (mid, bytes); - } else { - high = mid; - } - } + fn imported(&self, taken: Taken, before: &[RecentEntry]) -> Imported { + let changed = self.view() != before; + match taken { + Taken::Applied { .. } => Imported::Applied { changed }, + Taken::Waiting { needs } => Imported::Waiting { needs, changed }, } - Ok((best, fits)) } /// The list apps see: newest first, one entry per file, at most [`MAX_RECENTS`]. diff --git a/apps/craft-cloud-agent/src/live/replica.rs b/apps/craft-cloud-agent/src/live/replica.rs index c5c2c6d..8d4d336 100644 --- a/apps/craft-cloud-agent/src/live/replica.rs +++ b/apps/craft-cloud-agent/src/live/replica.rs @@ -16,11 +16,13 @@ //! had as soon as it is taken. Loro applies a waiting one when its history comes; the waiting ones are kept in the //! file (Loro doesn't keep them in a snapshot) and imported again on load. An update Loro can't decode (or that made //! it fail) has no operations anything can depend on: it counts as had, and is said once. -//! - **Its own acks** may come before the updates of others numbered below them: such an ack is noted on the unacked -//! update, which stays unacked until the number is reached (then it counts as had). On every new connection those -//! notes are forgotten and the updates sent again: the server acks one it has with its number, and stores one a -//! restore lost. A gap in what the server sends otherwise (a frame lost on a connection given up on) is caught up by -//! subscribing again from the number it has. +//! - **What the server holds is known by version, not by acks**: an unacked update all of whose operations are in the +//! rows taken (and the snapshot a catch-up started with) is stored, and leaves the unacked list, whatever its ack +//! does. Its own acks may come before the updates of others numbered below them: such an ack is kept (on the unacked +//! update, or beside it once the update is stored) until the number is reached, then it counts as had. On every new +//! connection those notes are forgotten and the updates sent again: the catch-up brings the rows themselves, the +//! server acks one it has with its number, and stores one a restore lost. A gap in what the server sends otherwise +//! (a frame lost on a connection given up on) is caught up by subscribing again from the number it has. //! - **A server restored from a backup, or a log the admin emptied**: the epoch is new (a reset starts one too), or //! the update at `after` isn't the one this machine has there (`resync_required`), or `subscribed` names a head below //! the number it has. The document starts over: its waiting updates go, it catches up from the start, and then sends @@ -37,22 +39,54 @@ //! - **Waiting updates are capped** at [`MAX_WAITING_BYTES`] per document: an update past it isn't taken, and the //! document stops taking the server's updates until the next connection, said once (an honest log never comes near; //! a device that does this is revoked, and the admin resets the document). -//! - **Loro failing** on an update from the server: the document is read back from its file and that update skipped; on -//! a waiting update imported again: that one is dropped. A file that doesn't read is set aside (kept); its document -//! starts afresh from its unacked updates, which are kept and sent. +//! - **Loro failing** on an update from the server: the document is read back from its file and that update skipped (a +//! catch-up's batch Loro fails on is taken again one row at a time, so only the row at fault is); on a waiting update +//! imported again: that one is dropped. A machine that dropped a row never compacts past it. A second failure in a +//! run stops the document until the agent restarts (its local changes are kept), and its apps are told. A file that +//! doesn't read is set aside (kept); its document starts afresh from its unacked updates, which are kept and sent. //! - A newer agent's update is taken like any other: once the document says it is of a newer version, this agent only //! stops adding to it (the design's rule: an older agent is read-only on a newer document), and says why. +//! +//! **Documents of files** (with a server on live protocol 2) work the same way, with what recents never needed: +//! +//! - **Joining**: a file opened for the first time here is followed from the start to see whether the server has its +//! document ([`Replica::open_document`]). If it does, the document is live once caught up; if not (`not_found`), the +//! app is asked for an import ([`DocEvent::NeedsImport`]), which goes up as `create` with its snapshot +//! ([`Replica::import_document`]); kept, it is the document; not kept (another machine's import was first), it is +//! dropped and the document caught up from the start. The app hears it is live only once that is settled +//! ([`DocEvent::Joined`]). A document that isn't joined yet is never written to disk, so a file that has never been +//! live never opens offline from an empty copy. A joined document whose server lost it (`not_found`) is made again +//! from this machine's copy. One the server holds nothing of (an admin reset it, and its machines haven't sent it +//! back) isn't joined: the app opens the file (`not_live {resetting}`). +//! - **Local updates** ([`Replica::local_update`]) are imported into the copy with Loro's checked import (an update +//! Loro refuses leaves the copy as it was; one it fails on reads the document back from disk), appended to the +//! document's journal (durable) before the app is told, and sent at once (during a push too: the apps' peers are this +//! machine's own). The document's file itself is written now and then ([`Replica::flush_documents`]), not per +//! keystroke; the journal is cleared after. A journal append that fails is cut back and refused, and the document is +//! written whole before its journal takes another; a journal is read up to its first bad record, and cut there. +//! - **Snapshots**: a catch-up from below the server's snapshot starts with it; one this agent can't read makes it +//! follow the whole log instead (`subscribe_log`). When the server asks to compact, the agent answers once it holds +//! every update up to `upto` and nothing waits, with a snapshot of exactly those updates ([`LogVersion`]), or not at +//! all (another device is asked). +//! - **Presence** goes both ways as it is; remote updates and presence are handed to the apps ([`Replica::take_events`]). +//! - **At most [`MAX_SUBSCRIPTIONS`] followed and [`MAX_LOADED_DOCUMENTS`] in memory**: past those, the least recently +//! used document no app has open and with nothing to send or waiting is unfollowed, and then written and let go of +//! (it stays on disk, and is loaded and followed again when opened). At start, only documents with something to send +//! or waiting are loaded (and followed). use std::collections::{BTreeMap, BTreeSet}; use craft_cloud_proto::api::codes; use craft_cloud_proto::live::{ - AppName, ClientFrame, DocKind, LIVE_PROTOCOL_MAX, LIVE_PROTOCOL_MIN, MAX_UPDATE_BYTES, PROTOCOL_MISMATCH, ServerFrame, recents_doc, update_hash, + AppName, ClientFrame, DocKind, LIVE_DOCUMENTS_PROTOCOL, LIVE_PROTOCOL_MAX, LIVE_PROTOCOL_MIN, MAX_SNAPSHOT_BYTES, MAX_SUBSCRIPTIONS, + MAX_UPDATE_BYTES, PROTOCOL_MISMATCH, ServerFrame, node_doc, recents_doc, update_hash, }; -use craft_cloud_proto::local::RecentEntry; -use craft_cloud_proto::{DocId, Epoch, SpaceId}; +use craft_cloud_proto::local::{NotLiveReason, RecentEntry}; +use craft_cloud_proto::{DocId, Epoch, ManifestId, NodeId, SpaceId}; use serde::{Deserialize, Serialize}; +use super::blobs::{Upload, cut}; +use super::copy::{LiveCopy, LogVersion, Taken as CopyTaken, Version, version_of}; use super::recents::{AddKey, Imported, Recents, Spans, UpdateError, add_spans, spans_of}; use super::store::DocStore; @@ -71,11 +105,21 @@ const PUSH_PART: usize = 64 * 1024; /// first). With delivery within a tick, a part goes twice only when two machines draw the same lowest wait: about 1 in /// this many per other machine. const PUSH_PATIENCE: u32 = 8; -/// The Loro peers of this machine's latest runs a push counts as its own (older ones count as others': they still go -/// up, after a wait). One per run, so without a cap the list would grow for good. -const MAX_OWN_PEERS: usize = 16; +/// The Loro peers a push counts as this machine's own, latest first: its own copy's in its latest runs, and those its +/// apps wrote a document of a file as (older ones count as others': they still go up, after a wait). A few per run, so +/// without a cap the list would grow for good. +const MAX_OWN_PEERS: usize = 64; +/// Documents of files a replica keeps loaded (in memory) at most: past it, the least recently used one with nothing to +/// send and no app holding it is written and let go of ([`Replica::unload`]), and loaded again when opened. +pub const MAX_LOADED_DOCUMENTS: usize = 256; /// Heals of one document in a run after which its push stops (the same state fails the same way again). const MAX_HEALS: u32 = 2; +/// The format of a document of a file's file (`.doc`). +const DOCUMENT_FILE_FORMAT: u32 = 1; +/// A document's journal is written into its file (and cleared) once it holds this many bytes, besides now and then. +const MAX_JOURNAL_BYTES: usize = 4 * 1024 * 1024; +/// A catch-up's updates are taken a batch at a time, at most this many bytes of them at once. +const MAX_BATCH_BYTES: usize = 8 * 1024 * 1024; /// Every update of the log up to `upto` is in the document's file; `upto_hash` is the hash of the update at `upto`. #[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] @@ -128,6 +172,242 @@ struct DocFile { push_wait: Option, } +/// What a recents file holds beyond [`DocFile`], written after it (an older agent reads the file and ignores what +/// follows; a file an older agent wrote has nothing after it). +#[derive(Debug, Default, Serialize, Deserialize)] +struct DocFileMore { + /// [`Doc::dropped`]. + dropped: Option, +} + +/// A recents file: [`DocFile`], and [`DocFileMore`] when it has it. +fn read_doc_file(bytes: &[u8]) -> postcard::Result<(DocFile, DocFileMore)> { + let (file, rest) = postcard::take_from_bytes::(bytes)?; + let more = if rest.is_empty() { DocFileMore::default() } else { postcard::from_bytes(rest).unwrap_or_default() }; + Ok((file, more)) +} + +/// A document of a file's file (`.doc`, beside its journal): what a recents file holds, and its node. A file of its +/// own kind, so an older agent (which reads only recents files) leaves it alone. +#[derive(Debug, Serialize, Deserialize)] +struct DocumentFile { + format: u32, + space: SpaceId, + doc: DocId, + node: NodeId, + app: AppName, + have: Have, + next_client_seq: u64, + pending: Vec, + snapshot: Vec, + waiting: Vec, + epoch: Option, + pushing: Option, + own_peers: Vec, + push_wait: Option, + /// Follow it with `subscribe_log` (the server's snapshot didn't read here). + log_only: bool, + /// [`Doc::dropped`]. + dropped: Option, +} + +/// What a file of either kind holds, as [`Doc::from_parts`] reads it. +struct Parts { + id: DocId, + kind: Kind, + have: Have, + next_client_seq: u64, + pending: Vec, + snapshot: Vec, + waiting: Vec, + epoch: Option, + pushing: Option, + own_peers: Vec, + push_wait: Option, + dropped: Option, +} + +/// One record of a document's journal: a local update, on disk before the app is told it is kept. +#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)] +struct JournalRecord { + client_seq: u64, + bytes: Vec, +} + +/// A journal record as it is written: its length, the record, and the first 8 bytes of its BLAKE3 (a record cut off +/// by a crash, never acked to the app, is where the journal ends). +fn journal_record(record: &JournalRecord) -> std::io::Result> { + let body = postcard::to_stdvec(record).map_err(std::io::Error::other)?; + let len = u32::try_from(body.len()).map_err(|_| std::io::Error::other("a journal record over 4 GiB"))?; + let mut out = Vec::with_capacity(body.len() + 12); + out.extend_from_slice(&len.to_le_bytes()); + out.extend_from_slice(&body); + out.extend_from_slice(update_hash(&body).get(..8).unwrap_or_default()); + Ok(out) +} + +/// The records of a journal, up to the first that doesn't read, and how many bytes they take (where it is cut). +fn journal_records(mut bytes: &[u8]) -> (Vec, usize) { + let mut out = Vec::new(); + let mut read = 0; + while bytes.len() >= 4 { + let Some(len) = bytes.get(..4).and_then(|b| <[u8; 4]>::try_from(b).ok()).map(u32::from_le_bytes) else { break }; + let len = len as usize; + let (Some(body), Some(check)) = (bytes.get(4..4 + len), bytes.get(4 + len..12 + len)) else { break }; + if update_hash(body).get(..8) != Some(check) { + break; + } + match postcard::from_bytes::(body) { + Ok(record) => out.push(record), + Err(_) => break, + } + bytes = bytes.get(12 + len..).unwrap_or_default(); + read += 12 + len; + } + (out, read) +} + +/// Which document it is. +#[derive(Debug, Clone, PartialEq, Eq)] +enum Kind { + Recents(AppName), + Document { node: NodeId, app: AppName }, +} + +impl Kind { + fn app(&self) -> &AppName { + match self { + Kind::Recents(app) | Kind::Document { app, .. } => app, + } + } + + fn is_document(&self) -> bool { + matches!(self, Kind::Document { .. }) + } +} + +/// A document's content: an app's recents list, or a document of a file (whose content nothing here reads). +enum Body { + Recents(Recents), + Document(LiveCopy), +} + +impl Body { + fn new(kind: &Kind, peer: u64) -> Self { + match kind { + Kind::Recents(_) => Body::Recents(Recents::new(peer)), + Kind::Document { .. } => Body::Document(LiveCopy::new(peer)), + } + } + + fn load(kind: &Kind, snapshot: &[u8], peer: u64) -> Result { + Ok(match kind { + Kind::Recents(_) => Body::Recents(Recents::load(snapshot, peer)?), + Kind::Document { .. } => Body::Document(LiveCopy::load(snapshot, peer)?), + }) + } + + fn copy(&self) -> &LiveCopy { + match self { + Body::Recents(recents) => recents, + Body::Document(copy) => copy, + } + } + + /// Imports an update; `changed`: the list changed (recents), or it brought something an app hasn't seen (a + /// document: it goes to the apps). + fn import(&self, bytes: &[u8]) -> Result { + match self { + Body::Recents(recents) => recents.import(bytes), + Body::Document(copy) => copy.import(bytes).map(document_imported), + } + } + + fn import_any_size(&self, bytes: &[u8]) -> Result { + match self { + Body::Recents(recents) => recents.import_any_size(bytes), + Body::Document(copy) => copy.import_any_size(bytes).map(document_imported), + } + } + + /// A recents document a newer agent wrote: its version. + fn newer(&self) -> Option { + match self { + Body::Recents(recents) => recents.newer(), + Body::Document(_) => None, + } + } + + fn view(&self) -> Vec { + match self { + Body::Recents(recents) => recents.view(), + Body::Document(_) => Vec::new(), + } + } + + /// Marks the document as one Loro failed in (for tests). + #[cfg(test)] + fn poison(&self) { + self.copy().poison(); + } +} + +fn document_imported(taken: CopyTaken) -> Imported { + match taken { + CopyTaken::Applied { new } => Imported::Applied { changed: new }, + CopyTaken::Waiting { needs } => Imported::Waiting { needs, changed: true }, + } +} + +/// Where a document of a file that isn't live here yet stands. +#[derive(Debug, Clone, PartialEq, Eq)] +enum Join { + /// Followed from the start, to see whether the server has it. + Probing, + /// The server doesn't have it: waiting for the app's import. + NeedsImport, + /// The import went up as `create`: waiting for `created`. + Creating(Vec), + /// Another machine's import was kept: caught up from the start. + Following, +} + +/// What happened to a document of a file, for the apps that have it open (or wait to). +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum DocEvent { + /// It is live here now (caught up, or made by this machine's import): the apps that wait for it get its state. + Joined(DocId), + /// The server doesn't have it: an app is to send its import. + NeedsImport(DocId), + /// It can't be made live now: the apps that wait open the file as a plain file. + NotLive(DocId, NotLiveReason, String), + /// Another machine's change, to apply as it is. + Remote(DocId, Vec), + /// Another machine's presence, as it sent it. + Presence(DocId, Vec), +} + +/// Why a local update was refused. +#[derive(Debug, thiserror::Error)] +pub enum LocalError { + /// The update isn't taken (Loro refused it, or it is over a limit): the document is as it was. + #[error("{0}")] + Invalid(String), + #[error("the document couldn't be written to disk: {0}")] + Io(#[from] std::io::Error), +} + +/// How opening a document of a file went ([`Replica::open_document`]). +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Opening { + /// Live here: its state is ready ([`Replica::document_state`]). + Live, + /// Being joined: a [`DocEvent`] says how it ends. + Joining, + /// It has never been live here and the server can't be asked: a plain file for now. + NotLive(NotLiveReason, String), +} + /// Where a replica gets the Loro peer ID of each document it opens or makes: random in the agent (a fresh one per /// run, so a peer ID is never reused with a different history), from the seed in the simulation. pub type Peers = Box u64 + Send>; @@ -167,8 +447,9 @@ enum Taken { } struct Doc { - app: AppName, - recents: Recents, + id: DocId, + kind: Kind, + body: Body, have: Have, pending: Vec, waiting: Vec, @@ -192,11 +473,28 @@ struct Doc { full_pause: u32, /// Past the cap on waiting updates: nothing is taken until the next connection. stuck: bool, + /// Loro failed on a second row of it in this run: not followed until the agent restarts ([`Replica::stopped`]). + stopped: bool, /// Loro failed outside an import (an export): the document is read back from its file. broken: bool, sub: Sub, /// Unacked updates sent on this connection (by client_seq). sent: BTreeSet, + /// The operations the server is known to hold up to `have`: every row taken (and own row acked) and the snapshot + /// a catch-up started with, since the document was loaded or its log learnt afresh. An unacked update all of whose + /// operations are in it is stored ([`Doc::settle_pending`]), whatever its ack does. + seen: Spans, + /// Own updates sent on this connection that the log turned out to hold before their ack (a row or the server's + /// snapshot brought their operations back): the ack still comes, numbering a row that holds them and is never sent + /// to this machine, so it counts as had when it comes (by client_seq: the update's bytes). + taken_back: BTreeMap>, + /// Acks of own rows above what is had (rows below them on their way), whose updates left the unacked list: by the + /// row's number, its bytes, counted when the rows below it are here (as [`Pending::acked`]; this connection only, + /// a catch-up sends the rows themselves). + back_acked: BTreeMap>, + /// The lowest row of the log this machine dropped (Loro failed on it, or it can never apply): its copy lacks it, + /// so it never answers a `compact` at or above it (another device does). Kept in its file. + dropped: Option, /// `too_large` refusals since the last ack. strikes: u32, /// A refused create or subscribe: ticks until it is tried again, and the pause it had. @@ -208,17 +506,59 @@ struct Doc { dirty: bool, /// What was said about this document's updates already. said: BTreeSet, + /// What the server's log holds of it, as a version per number: what a compaction's snapshot is made at. + log: LogVersion, + /// The server asked for a snapshot of the updates up to this: answered once they are all here. + compact_ask: Option, + /// Live here: it has a file and opens offline. Recents always are; a document of a file once joined. + joined: bool, + /// A document of a file being joined. + join: Option, + /// A joined document the server lost is being made again from this copy (`create` sent). + creating: bool, + /// Not followed (the device's cap on subscriptions): it stays on disk. + parked: bool, + /// When an app last opened it (a counter), and how many have it open now. + last_used: u64, + open: u32, + /// Followed with `subscribe_log`: the server's snapshot didn't read here. + log_only: bool, + /// The catch-up under way was given up on (its snapshot didn't read): its `subscribed` starts the next one. + stale: bool, + /// Bytes appended to its journal since its file was last written. + journal: usize, + /// An append to its journal failed (or its end couldn't be cut): it is written whole before its journal takes + /// another change, so nothing after a bad record is ever hidden by it. + journal_failed: bool, + /// Its file must be written at the next flush (not only now and then). + urgent: bool, + /// The waiting update Loro just failed on, imported again ([`Doc::rewait`]). + waiting_failed: Option<[u8; 32]>, + /// The catch-up under way started with the server's snapshot ([`Doc::check_snapshot`]). + from_snapshot: bool, + /// It has a file (written, or loaded from one). + on_disk: bool, + /// The catch-up's updates not taken yet ([`Replica::take_batch`]), and their bytes. + batch: Vec<(u64, Vec)>, + batched: usize, + /// The catch-up started with the server's snapshot (its base and manifest), being downloaded: its updates wait in + /// `batch` (and its `subscribed`, in `deferred_head`) until it is here ([`Replica::on_snapshot`]). + fetching: Option<(u64, ManifestId)>, + deferred_head: Option, + /// Snapshots to upload before the frames that name them go ([`Replica::take_uploads`]). + uploads: Vec, } impl Doc { - fn empty(app: &AppName, peer: u64) -> Self { - Self::with(app.clone(), Recents::new(peer), Have::default(), Vec::new(), Vec::new(), 1, None, None) + fn empty(app: &AppName, peer: u64, id: DocId) -> Self { + Self::with(id, Kind::Recents(app.clone()), Body::Recents(Recents::new(peer)), Have::default(), Vec::new(), Vec::new(), 1, None, None) } #[allow(clippy::too_many_arguments)] fn with( - app: AppName, - recents: Recents, + id: DocId, + kind: Kind, + body: Body, have: Have, pending: Vec, waiting: Vec, @@ -227,9 +567,10 @@ impl Doc { pushing: Option, ) -> Self { Self { - app, - own_peers: vec![recents.peer()], - recents, + id, + own_peers: vec![body.copy().peer()], + kind, + body, have, pending, waiting, @@ -242,14 +583,168 @@ impl Doc { full: None, full_pause: 0, stuck: false, + stopped: false, broken: false, sub: Sub::No, sent: BTreeSet::new(), + seen: Spans::new(), + taken_back: BTreeMap::new(), + back_acked: BTreeMap::new(), + dropped: None, strikes: 0, retry: None, create_refused: false, dirty: false, said: BTreeSet::new(), + log: LogVersion::unknown(), + compact_ask: None, + joined: true, + join: None, + creating: false, + parked: false, + last_used: 0, + open: 0, + log_only: false, + stale: false, + journal: 0, + journal_failed: false, + urgent: false, + waiting_failed: None, + from_snapshot: false, + on_disk: false, + batch: Vec::new(), + batched: 0, + fetching: None, + deferred_head: None, + uploads: Vec::new(), + } + } + + /// The app this document belongs to. + fn app(&self) -> &AppName { + self.kind.app() + } + + /// The document as the user is told about it: `recents` names an app's recents (as every recents note always has), + /// `document` a document of a file. + fn told(&self, recents: impl FnOnce(&AppName) -> String, document: impl FnOnce(DocId) -> String) -> String { + match &self.kind { + Kind::Recents(app) => recents(app), + Kind::Document { .. } => document(self.id), + } + } + + /// What the server's log holds of the document up to `have`, worked out from the copy: everything in it, when + /// nothing in it may be out of the log (no unacked update: one may or may not be stored, and its ack doesn't say + /// which; nothing waits; no push under way; no row dropped). Unknown otherwise, until those are gone. + fn derive_log(&mut self) { + self.log = LogVersion::unknown(); + if !self.waiting.is_empty() || self.pushing.is_some() || !self.pending.is_empty() || self.dropped.is_some() { + return; + } + let Ok(mut version) = self.body.copy().version() else { return }; + version.retain(|_, end| *end > 0); + self.log = LogVersion::anchored(self.have.upto, version); + } + + /// Unacked updates whose operations the server is known to hold ([`Doc::seen`]) are stored: they leave the list + /// (and so count in a compaction, being in the log). An ack that came for one already is kept until the rows + /// below it are here; one still on its way is counted when it comes ([`Doc::taken_back`]). + fn settle_pending(&mut self) { + if self.seen.is_empty() || self.pending.is_empty() { + return; + } + let mut i = 0; + while let Some(p) = self.pending.get(i) { + let held = spans_of(&p.bytes).is_some_and(|ranges| !ranges.is_empty() && held_by(&self.seen, &ranges)); + if !held { + i += 1; + continue; + } + let own = self.pending.remove(i); + self.stored(own); + } + } + + /// An unacked update, out of the list, is stored (the log holds its operations): an ack that came for it already + /// is kept until the rows below it are here, one on its way is counted when it comes. + fn stored(&mut self, own: Pending) { + let sent = self.sent.remove(&own.client_seq); + match own.acked { + Some(n) if n > self.have.upto => { + self.back_acked.insert(n, own.bytes); + } + Some(_) => {} + None if sent => { + self.taken_back.insert(own.client_seq, own.bytes); + } + None => {} + } + self.dirty = true; + } + + /// `bytes` (an update or snapshot) were written by this machine's apps: their peers are its own in a push (which + /// sends its own operations without a wait). + fn own(&mut self, bytes: &[u8]) { + let Some(ranges) = spans_of(bytes) else { return }; + for (peer, start, end) in ranges { + if start >= end || self.own_peers.first() == Some(&peer) { + continue; + } + self.own_peers.retain(|p| *p != peer); + self.own_peers.insert(0, peer); + self.own_peers.truncate(MAX_OWN_PEERS); + self.dirty = true; + } + } + + /// The server holds `ranges` (up to `have`). + fn saw(&mut self, ranges: &[(u64, i32, i32)]) { + add_spans(&mut self.seen, ranges); + } + + /// The log is learnt afresh (from the start, or a snapshot that replaces what was known): nothing seen yet, nothing + /// dropped. + fn forget_seen(&mut self) { + self.seen = Spans::new(); + self.dropped = None; + } + + /// The answer to the server's `compact`, once this machine holds every update up to `upto`: a snapshot of exactly + /// those, or nothing (something waits, it dropped a row at or below `upto`, or the version there isn't known; + /// another device is asked). The question is dropped either way once it can be answered. + fn try_compact(&mut self) -> Vec { + let Some(upto) = self.compact_ask else { return Vec::new() }; + if self.sub != Sub::Live || self.have.upto < upto { + return Vec::new(); + } + self.compact_ask = None; + // (A push under way doesn't stop it: the log was learnt from the start then, and is noted from there.) + if !self.waiting.is_empty() || self.stuck || !self.joined || self.is_broken() { + return Vec::new(); + } + // A row it dropped is below: its copy lacks that row, and a snapshot from it would too. + if self.dropped.is_some_and(|d| d <= upto) { + return Vec::new(); + } + if !self.log.is_known() && upto == self.have.upto { + self.derive_log(); + } + let Some(version) = self.log.at(upto) else { return Vec::new() }; + match self.body.copy().snapshot_at(&version) { + Ok(bytes) if !bytes.is_empty() && bytes.len() <= MAX_SNAPSHOT_BYTES => { + // Uploaded first (through the chunk API), then named. + let Ok(upload) = cut(self.id, &bytes) else { return Vec::new() }; + self.log.rebase(upto, version); + let manifest = upload.id(); + self.uploads.push(upload); + vec![ClientFrame::Snapshot { doc: self.id, upto, manifest }] + } + Err(UpdateError::Panicked) => { + self.broken = true; + Vec::new() + } + _ => Vec::new(), } } @@ -259,57 +754,86 @@ impl Doc { /// the snapshot without it): a waiting one is dropped (the server has it, and a catch-up from the start brings it /// if it is ever needed), an unacked one stays unacked (it is sent, and its operations come back with the log). /// An error only when the snapshot itself doesn't read. - fn from_file(mut file: DocFile, peer: u64, notes: &mut Vec) -> Result { + fn from_file(file: DocFile, more: DocFileMore, peer: u64, notes: &mut Vec) -> Result { + let parts = Parts { + dropped: more.dropped, + id: file.doc, + kind: Kind::Recents(file.app), + have: file.have, + next_client_seq: file.next_client_seq, + pending: file.pending, + snapshot: file.snapshot, + waiting: file.waiting, + epoch: file.epoch, + pushing: file.pushing, + own_peers: file.own_peers, + push_wait: file.push_wait, + }; + Self::from_parts(parts, peer, notes) + } + + /// [`Doc::from_file`], for a file of either kind. + fn from_parts(mut file: Parts, peer: u64, notes: &mut Vec) -> Result { let (mut bad_pending, mut bad_waiting) = (BTreeSet::new(), BTreeSet::new()); - let recents = 'load: loop { - let recents = Recents::load(&file.snapshot, peer)?; + let body = 'load: loop { + let body = Body::load(&file.kind, &file.snapshot, peer)?; for (i, pending) in file.pending.iter().enumerate() { - if !bad_pending.contains(&i) && recents.import(&pending.bytes) == Err(UpdateError::Panicked) { + if !bad_pending.contains(&i) && body.import_any_size(&pending.bytes) == Err(UpdateError::Panicked) { bad_pending.insert(i); continue 'load; } } for (i, waiting) in file.waiting.iter().enumerate() { - if !bad_waiting.contains(&i) && recents.import(&waiting.bytes) == Err(UpdateError::Panicked) { + if !bad_waiting.contains(&i) && body.import(&waiting.bytes) == Err(UpdateError::Panicked) { bad_waiting.insert(i); continue 'load; } } - break recents; + break body; }; if !bad_pending.is_empty() || !bad_waiting.is_empty() { - notes.push(format!( - "Loro failed on {} stored recents update(s) of {}; the waiting ones were dropped, the unsent ones are kept and sent", - bad_pending.len() + bad_waiting.len(), - file.app - )); + let count = bad_pending.len() + bad_waiting.len(); + notes.push(match &file.kind { + Kind::Recents(app) => format!( + "Loro failed on {count} stored recents update(s) of {app}; the waiting ones were dropped, the unsent ones are kept and sent" + ), + Kind::Document { .. } => format!( + "Loro failed on {count} stored update(s) of the live document {}; the waiting ones were dropped, the unsent ones are kept and sent", + file.id + ), + }); } let mut i = 0; file.waiting.retain(|_| { i += 1; !bad_waiting.contains(&(i - 1)) }); - let mut doc = Self::with(file.app, recents, file.have, file.pending, file.waiting, file.next_client_seq.max(1), file.epoch, file.pushing); - doc.waiting.retain(|w| !doc.recents.covers(&w.needs)); + let mut doc = + Self::with(file.id, file.kind, body, file.have, file.pending, file.waiting, file.next_client_seq.max(1), file.epoch, file.pushing); + doc.waiting.retain(|w| !doc.body.copy().covers(&w.needs)); for peer in file.own_peers { if !doc.own_peers.contains(&peer) && doc.own_peers.len() < MAX_OWN_PEERS { doc.own_peers.push(peer); } } doc.push_wait = file.push_wait; + doc.dropped = file.dropped; + doc.on_disk = true; + doc.derive_log(); Ok(doc) } /// A document whose file doesn't read (the snapshot is damaged, or Loro fails on it): started afresh from its /// unacked updates, which are kept and sent; it catches up from the start, so nothing the server has is missed. - fn from_pending(app: AppName, pending: Vec, next_client_seq: u64, peer: u64) -> Self { - let recents = Recents::new(peer); + fn from_pending(id: DocId, kind: Kind, pending: Vec, next_client_seq: u64, peer: u64) -> Self { + let body = Body::new(&kind, peer); for p in &pending { - let _ = recents.import(&p.bytes); + let _ = body.import_any_size(&p.bytes); } let pending = pending.into_iter().map(|p| Pending { acked: None, push: false, ..p }).collect(); - let mut doc = Self::with(app, recents, Have::default(), pending, Vec::new(), next_client_seq.max(1), None, None); + let mut doc = Self::with(id, kind, body, Have::default(), pending, Vec::new(), next_client_seq.max(1), None, None); doc.dirty = true; + doc.urgent = true; doc } @@ -327,27 +851,23 @@ impl Doc { } if self.skip == Some((seq, hash)) { self.skip = None; + self.drop_row(seq); + self.log.note(seq, Vec::new()); self.advance(seq, hash); return Taken::Had; } - let had_history = self.recents.has_history(); - match self.recents.import(bytes) { + let had_history = self.body.copy().has_history(); + let mut in_log = true; + match self.body.import(bytes) { Ok(Imported::Applied { changed: list }) => { *changed |= list; - if !had_history { - // The first history, perhaps a snapshot: Loro dropped what waited; it goes in again. - for waiting in &self.waiting { - match self.recents.import(&waiting.bytes) { - Ok(Imported::Applied { changed: list } | Imported::Waiting { changed: list, .. }) => *changed |= list, - Err(UpdateError::Panicked) => return Taken::WaitingPanicked(waiting.hash), - Err(_) => {} - } - } + if !had_history && self.rewait(changed).is_err() { + return self.waiting_panicked(); } - if let Some(version) = self.recents.newer() { + if let Some(version) = self.body.newer() { let line = format!( "the recents of {} were written by a newer Craft Cloud agent (version {version}); they are shown, and this agent adds to them again once it is updated", - self.app + self.app() ); self.say(notes, line); } @@ -358,10 +878,15 @@ impl Doc { let held: usize = self.waiting.iter().map(|w| w.bytes.len()).sum(); if held.saturating_add(bytes.len()) > MAX_WAITING_BYTES { self.stuck = true; - let line = format!( - "more than {} MiB of recents updates for {} wait for history that never came; this machine takes no more of them until it connects again", - MAX_WAITING_BYTES >> 20, - self.app + let line = self.told( + |app| format!( + "more than {} MiB of recents updates for {app} wait for history that never came; this machine takes no more of them until it connects again", + MAX_WAITING_BYTES >> 20 + ), + |id| format!( + "more than {} MiB of updates of the live document {id} wait for history that never came; this machine takes no more of them until it connects again", + MAX_WAITING_BYTES >> 20 + ), ); self.say(notes, line); return Taken::Stuck; @@ -371,16 +896,63 @@ impl Doc { } Err(UpdateError::Panicked) => return Taken::Panicked, Err(e) => { - let line = format!("a recents update for {} from the server can never apply and was skipped: {e}", self.app); + in_log = false; + let line = self.told( + |app| format!( + "a recents update for {app} from the server can never apply and was skipped: {e}; this machine leaves compacting them to other devices" + ), + |id| format!( + "an update of the live document {id} from the server can never apply and was skipped: {e}; this machine leaves compacting it to other devices" + ), + ); self.say(notes, line); + self.drop_row(seq); } } - self.waiting.retain(|w| !self.recents.covers(&w.needs)); + self.waiting.retain(|w| !self.body.copy().covers(&w.needs)); self.on_server(bytes); + // What the log holds, as far as this copy has it (an update that never applies isn't in it). + let ranges = if in_log { spans_of(bytes).unwrap_or_default() } else { Vec::new() }; + self.saw(&ranges); + self.log.note(seq, ranges); + // This machine's own unacked updates the log turns out to hold (its ack was lost, or forgotten with the + // connection, or another machine pushed the same operations) are stored. + self.settle_pending(); self.advance(seq, hash); Taken::Had } + /// Row `seq` isn't in this copy (Loro failed on it, or it can never apply; the note that says so says this too): + /// this machine leaves compacting the log past it to other devices. + fn drop_row(&mut self, seq: u64) { + self.dropped = Some(self.dropped.map_or(seq, |d| d.min(seq))); + self.dirty = true; + self.urgent = true; + } + + /// After the copy's first history (perhaps a snapshot) Loro dropped what waited: it goes in again. `Err` with the + /// one Loro failed on (in [`Doc::waiting_failed`]). + fn rewait(&mut self, changed: &mut bool) -> Result<(), ()> { + for waiting in &self.waiting { + match self.body.import(&waiting.bytes) { + Ok(Imported::Applied { changed: list } | Imported::Waiting { changed: list, .. }) => *changed |= list, + Err(UpdateError::Panicked) => { + self.waiting_failed = Some(waiting.hash); + return Err(()); + } + Err(_) => {} + } + } + Ok(()) + } + + fn waiting_panicked(&mut self) -> Taken { + match self.waiting_failed.take() { + Some(hash) => Taken::WaitingPanicked(hash), + None => Taken::Panicked, + } + } + /// `bytes` are in the log: while a push is under way, their operations needn't go up. fn on_server(&mut self, bytes: &[u8]) { if let Some(has) = self.pushing.as_mut() @@ -393,11 +965,23 @@ impl Doc { /// The log is had up to `seq` (whose update has `hash`), and up to the own acks that follow it. fn advance(&mut self, seq: u64, hash: [u8; 32]) { self.have = Have { upto: seq, upto_hash: Some(hash) }; - while let Some(i) = self.pending.iter().position(|p| p.acked == Some(self.have.upto.saturating_add(1))) { - let own = self.pending.remove(i); - self.sent.remove(&own.client_seq); - self.on_server(&own.bytes); - self.have = Have { upto: self.have.upto.saturating_add(1), upto_hash: Some(update_hash(&own.bytes)) }; + self.back_acked.retain(|at, _| *at > seq); + loop { + let next = self.have.upto.saturating_add(1); + let bytes = if let Some(i) = self.pending.iter().position(|p| p.acked == Some(next)) { + let own = self.pending.remove(i); + self.sent.remove(&own.client_seq); + self.on_server(&own.bytes); + own.bytes + } else if let Some(bytes) = self.back_acked.remove(&next) { + bytes + } else { + break; + }; + self.have = Have { upto: next, upto_hash: Some(update_hash(&bytes)) }; + let ranges = spans_of(&bytes).unwrap_or_default(); + self.saw(&ranges); + self.log.note(next, ranges); } // Own updates the catch-up brought back count as had too. let upto = self.have.upto; @@ -412,7 +996,7 @@ impl Doc { /// Loro failed in this document (an import, export or read panicked): read back from its file, never dropped. fn is_broken(&self) -> bool { - self.broken || self.recents.is_poisoned() + self.broken || self.body.copy().is_poisoned() } /// Tells the user `line`, once per document. @@ -438,7 +1022,7 @@ impl Doc { return Vec::new(); } let Some(has) = &self.pushing else { return Vec::new() }; - let Ok(missing) = self.recents.missing(has) else { + let Ok(missing) = self.body.copy().missing(has) else { // Loro failed: the document is read back from its file ([`Replica::heal`]). self.broken = true; return Vec::new(); @@ -473,7 +1057,7 @@ impl Doc { } let run = missing.iter().take_while(|(p, _, _)| self.own_peers.contains(p) == own).count(); let max = PUSH_PART.min(MAX_UPDATE_BYTES >> self.strikes); - self.recents.export_prefix(&missing[..run], max, MAX_UPDATE_BYTES >> self.strikes).map(|(bytes, _)| Some(bytes)) + self.body.copy().export_prefix(&missing[..run], max, MAX_UPDATE_BYTES >> self.strikes).map(|(bytes, _)| Some(bytes)) } }; match part { @@ -486,8 +1070,15 @@ impl Doc { } Ok(None) => { let has = self.pushing.take().unwrap_or_default(); - // (One acked above what is had stays: it is counted when the numbers below it come.) - self.pending.retain(|p| p.acked.is_some() || !spans_of(&p.bytes).is_some_and(|ranges| held_by(&has, &ranges))); + // (One acked above what is had stays: it is counted when the numbers below it come. One sent and not + // acked yet is counted when its ack comes.) + let (held, keep): (Vec, Vec) = std::mem::take(&mut self.pending) + .into_iter() + .partition(|p| p.acked.is_none() && spans_of(&p.bytes).is_some_and(|ranges| held_by(&has, &ranges))); + self.pending = keep; + for own in held { + self.stored(own); + } self.dirty = true; send_pending(id, self) } @@ -497,7 +1088,10 @@ impl Doc { Vec::new() } Err(e) => { - let line = format!("the recents of {} can't go up to the server ({e}); they stay on this machine", self.app); + let line = self.told( + |app| format!("the recents of {app} can't go up to the server ({e}); they stay on this machine"), + |id| format!("the live document {id} can't go up to the server ({e}); it stays on this machine"), + ); self.say(notes, line); Vec::new() } @@ -513,16 +1107,23 @@ impl Doc { self.push_wait = None; self.waiting.clear(); self.stuck = false; - if let Ok(snapshot) = self.recents.snapshot() - && let Ok(fresh) = Recents::load(&snapshot, self.recents.peer()) + if let Ok(snapshot) = self.body.copy().snapshot() + && let Ok(fresh) = Body::load(&self.kind, &snapshot, self.body.copy().peer()) { - self.recents = fresh; + self.body = fresh; for pending in &self.pending { - let _ = self.recents.import(&pending.bytes); + let _ = self.body.import_any_size(&pending.bytes); } } self.forget_acks(); + // The log is learnt again from the start. + self.log = LogVersion::anchored(0, Version::new()); + self.forget_seen(); + self.batch.clear(); + self.batched = 0; + self.compact_ask = None; self.dirty = true; + self.urgent = true; } /// Acks of numbers not reached are forgotten: their updates go again (the server acks one it has). @@ -530,14 +1131,92 @@ impl Doc { for pending in &mut self.pending { pending.acked = None; } + self.taken_back.clear(); + self.back_acked.clear(); } - /// `subscribe` from what the document has, without `create` (the server has it: on the same connection). + /// `subscribe` from what the document has, without `create` (the server has it: on the same connection); a + /// document whose server snapshot didn't read here follows the whole log instead. fn resubscribe(&mut self, id: DocId) -> Vec { + if self.stopped { + self.sub = Sub::No; + return Vec::new(); + } self.sub = Sub::CatchingUp; + if self.log_only { + return vec![ClientFrame::SubscribeLog { doc: id, after: self.have.upto, at: self.at() }]; + } vec![ClientFrame::Subscribe { doc: id, after: self.have.upto, at: self.at() }] } + /// The server's snapshot of the document, at the start of a catch-up from below it: imported (Loro merges it with + /// what is here), and the log is had up to its base. One that doesn't read here (or that Loro fails on: the + /// document is read back from its file) is given up on: this machine follows the whole log instead + /// (`subscribe_log`), from where it was. `true` when it was taken. + fn take_snapshot(&mut self, base: u64, bytes: &[u8], notes: &mut Vec, changed: &mut bool) -> Result<(), UpdateError> { + let had_history = self.body.copy().has_history(); + match self.body.import_any_size(bytes) { + Ok(Imported::Applied { changed: list } | Imported::Waiting { changed: list, .. }) => *changed |= list, + Err(e) => { + let line = self.told( + |app| format!("the server's snapshot of the recents of {app} doesn't read here ({e}); this machine reads the whole log instead"), + |id| format!("the server's snapshot of the live document {id} doesn't read here ({e}); this machine reads the whole log instead"), + ); + self.say(notes, line); + self.log_only = true; + self.stale = true; + self.dirty = true; + self.urgent = true; + return Err(e); + } + } + if !had_history && self.rewait(changed).is_err() { + // (Read back from its file by the caller.) + return Err(UpdateError::Panicked); + } + self.waiting.retain(|w| !self.body.copy().covers(&w.needs)); + self.on_server(bytes); + let version = version_of(bytes).unwrap_or_default(); + if base >= self.have.upto { + self.have = Have { upto: base, upto_hash: if base == self.have.upto { self.have.upto_hash } else { None } }; + self.log = LogVersion::anchored(base, version.clone()); + // The rows it holds are in the copy now (Loro merged them), one this machine dropped too. + if self.dropped.is_some_and(|d| d <= base) { + self.dropped = None; + } + } + // What the server holds up to its base: this machine's unacked updates in it are stored. + let ranges: Vec<(u64, i32, i32)> = version.iter().map(|(peer, end)| (*peer, 0, *end)).collect(); + self.saw(&ranges); + self.settle_pending(); + self.from_snapshot = true; + self.dirty = true; + Ok(()) + } + + /// The end of a catch-up that started with the server's snapshot, with updates still waiting for their history: + /// the snapshot may lack some of what the rows below its base held (a device made it wrong). The whole log is + /// read again from the start, with the import (`subscribe_log`, kept for this document from now on); Loro takes what + /// it has already as no news. Frames to send, if it does. + fn check_snapshot(&mut self, id: DocId, notes: &mut Vec) -> Vec { + if !std::mem::take(&mut self.from_snapshot) || self.waiting.is_empty() || self.log_only { + return Vec::new(); + } + let line = self.told( + |app| format!("updates of the recents of {app} wait for history the server's snapshot should hold; this machine reads the whole log"), + |id| format!("updates of the live document {id} wait for history the server's snapshot should hold; this machine reads the whole log"), + ); + self.say(notes, line); + self.log_only = true; + self.have = Have::default(); + self.log = LogVersion::anchored(0, Version::new()); + self.forget_seen(); + self.compact_ask = None; + self.dirty = true; + self.urgent = true; + self.resubscribe(id) + } + fn at(&self) -> Option<[u8; 32]> { if self.have.upto > 0 { self.have.upto_hash } else { None } } @@ -589,8 +1268,25 @@ pub struct Replica { heals: BTreeMap, /// Apps whose push stopped ([`Replica::push_stopped`]). push_stopped: BTreeSet, + /// Rows of each document Loro failed on in this run ([`Replica::recover_doc`]). + failures: BTreeMap, + /// Documents whose catch-ups are taken one row at a time in this run (Loro failed on a batch of them). + one_by_one: BTreeSet, + /// Documents not followed for the rest of this run: Loro failed on a second row ([`Replica::is_stopped`]). + stopped: BTreeSet, + /// Documents of files on disk and not loaded: nothing to send, nothing waiting, no app holding them + /// ([`Replica::load`], [`Replica::unload`]). + unloaded: BTreeSet, + /// Snapshots catch-ups started with, to download ([`Replica::take_fetches`]). + fetches: Vec<(DocId, ManifestId)>, /// Documents let go of without dropping ([`Replica::forgotten`]). forgotten: usize, + /// The live protocol `welcome` named (0 while the link isn't up). + protocol: u32, + /// What happened to documents of files since [`Replica::take_events`]. + events: Vec, + /// Counts opens, for which document was used least recently. + uses: u64, } impl Drop for Replica { @@ -612,7 +1308,22 @@ impl Replica { let mut notes = Vec::new(); let mut docs = BTreeMap::new(); for (name, bytes) in files { - if let Some((id, doc)) = load_doc(&mut store, space, &name, &bytes, peers(), &mut notes) { + if let Some((id, doc)) = load_doc(&mut store, space, &name, &bytes, peers(), None, &mut notes) { + docs.insert(id, doc); + } + } + // Documents of files are loaded only when they have something to send or that waits; the rest stay on disk + // until an app opens one ([`Replica::open_document`]). + let mut unloaded = BTreeSet::new(); + for name in store.document_names()? { + let Some(bytes) = store.load_document(&name)? else { continue }; + if let Some(id) = idle_on_disk(&store, space, &name, &bytes) { + unloaded.insert(id); + continue; + } + if let Some((id, mut doc)) = load_document(&mut store, space, &name, &bytes, peers(), None, &mut notes) { + // Followed again when an app opens it, or now if it has something to send. + doc.parked = doc.pending.is_empty() && doc.pushing.is_none(); docs.insert(id, doc); } } @@ -627,10 +1338,78 @@ impl Replica { peers, heals: BTreeMap::new(), push_stopped: BTreeSet::new(), + failures: BTreeMap::new(), + one_by_one: BTreeSet::new(), + stopped: BTreeSet::new(), + unloaded, + fetches: Vec::new(), forgotten: 0, + protocol: 0, + events: Vec::new(), + uses: 0, }) } + /// Loads document `id` of a file from disk if it is there and not loaded. Whether it is loaded now. + fn load(&mut self, id: DocId) -> bool { + if self.docs.contains_key(&id) { + return true; + } + if !self.unloaded.remove(&id) { + return false; + } + let name = id.to_string(); + let Ok(Some(bytes)) = self.store.load_document(&name) else { return false }; + let peer = (self.peers)(); + match load_document(&mut self.store, self.space, &name, &bytes, peer, None, &mut self.notes) { + Some((loaded, mut doc)) if loaded == id => { + doc.parked = true; + doc.stopped = self.stopped.contains(&id); + self.docs.insert(id, doc); + true + } + _ => false, + } + } + + /// Past [`MAX_LOADED_DOCUMENTS`] documents of files in memory: the least recently used ones with nothing unacked or + /// waiting, no push under way and no app holding them are written (if changed) and let go of (unfollowed first); + /// they stay on disk, and are loaded again when opened. One that can't be written stays. Frames to send. + fn unload(&mut self) -> Vec { + let loaded = self.loaded_documents(); + let Some(extra) = loaded.checked_sub(MAX_LOADED_DOCUMENTS).filter(|n| *n > 0) else { return Vec::new() }; + let mut idle: Vec<(u64, DocId)> = self + .docs + .iter() + .filter(|(_, d)| { + d.kind.is_document() + && d.joined + && d.open == 0 + && d.pending.is_empty() + && d.waiting.is_empty() + && d.pushing.is_none() + && !d.is_broken() + && d.batch.is_empty() + }) + .map(|(id, d)| (d.last_used, *id)) + .collect(); + idle.sort_unstable(); + let mut frames = Vec::new(); + let up = self.link == Link::Up; + for (_, id) in idle.into_iter().take(extra) { + let Some(doc) = self.docs.get_mut(&id) else { continue }; + if (doc.dirty || !doc.on_disk) && save(&mut self.store, self.space, id, doc).is_err() { + continue; + } + if doc.sub != Sub::No && up { + frames.push(ClientFrame::Unsubscribe { doc: id }); + } + self.docs.remove(&id); + self.unloaded.insert(id); + } + frames + } + fn doc_id(&self, app: &AppName) -> DocId { recents_doc(self.space, app) } @@ -642,12 +1421,15 @@ impl Replica { /// `app`'s list, newest first. pub fn entries(&self, app: &AppName) -> Vec { - self.docs.get(&self.doc_id(app)).map(|d| d.recents.view()).unwrap_or_default() + self.docs.get(&self.doc_id(app)).map(|d| d.body.view()).unwrap_or_default() } /// The newest entry for a file, if the list shows it (to keep its `saved` time when it is only opened). pub fn current(&self, app: &AppName, entry: &RecentEntry) -> Option { - self.docs.get(&self.doc_id(app)).and_then(|d| d.recents.current(entry.node, &entry.path)) + self.docs.get(&self.doc_id(app)).and_then(|d| match &d.body { + Body::Recents(recents) => recents.current(entry.node, &entry.path), + Body::Document(_) => None, + }) } /// An app follows its list: its document is opened (and, with the link up, subscribed). Frames to send. @@ -662,18 +1444,19 @@ impl Replica { pub fn add(&mut self, app: &AppName, key: AddKey, entry: RecentEntry) -> Result, AddError> { let (space, up, epoch) = (self.space, self.link == Link::Up, self.epoch); let (id, doc) = doc_in(&mut self.docs, space, app, &mut self.peers); - let before = doc.recents.view(); - let bytes = doc.recents.add(key, &entry).map_err(|e| AddError::Invalid(e.to_string()))?; + let Body::Recents(recents) = &doc.body else { return Err(AddError::Invalid("not a recents document".into())) }; + let before = recents.view(); + let bytes = recents.add(key, &entry).map_err(|e| AddError::Invalid(e.to_string()))?; let client_seq = doc.next_client_seq; doc.next_client_seq = client_seq.saturating_add(1); doc.pending.push(Pending { client_seq, bytes: bytes.clone(), acked: None, push: false }); - let changed = doc.recents.view() != before; + let changed = doc.body.view() != before; // On disk before the app hears it is kept. if let Err(e) = save(&mut self.store, space, id, doc) { self.reload(id); return Err(e.into()); } - let doc_app = doc.app.clone(); + let doc_app = doc.app().clone(); let frames = match doc.sub { // (While a push is under way, its parts carry the add.) Sub::Live if doc.full.is_none() && doc.pushing.is_none() => { @@ -706,18 +1489,19 @@ impl Replica { self.forgotten += 1; } let name = id.to_string(); - let stored = self.store.load().ok().and_then(|files| files.get(&name).cloned()); - let stored = stored.map(|bytes| match (drop, postcard::from_bytes::(&bytes)) { - (Some(hash), Ok(mut file)) => { - file.waiting.retain(|w| w.hash != hash); - postcard::to_stdvec(&file).unwrap_or(bytes) - } - _ => bytes, - }); let peer = (self.peers)(); - match stored.and_then(|bytes| load_doc(&mut self.store, self.space, &name, &bytes, peer, &mut self.notes)) { + let document = self.store.load_document(&name).ok().flatten(); + let reloaded = match document { + Some(bytes) => load_document(&mut self.store, self.space, &name, &bytes, peer, drop, &mut self.notes), + None => { + let stored = self.store.load().ok().and_then(|files| files.get(&name).cloned()); + stored.and_then(|bytes| load_doc(&mut self.store, self.space, &name, &bytes, peer, drop, &mut self.notes)) + } + }; + match reloaded { Some((_, mut doc)) => { doc.dirty |= drop.is_some(); + doc.stopped = self.stopped.contains(&id); self.docs.insert(id, doc); } None => { @@ -728,27 +1512,89 @@ impl Replica { /// Loro failed on update `seq` (`hash`) of `id`: the document is read back from its file (nothing half-changed /// stays), and caught up again from there, with that update counted as had and not imported. - fn recover(&mut self, id: DocId, app: AppName, seq: u64, hash: [u8; 32]) -> Vec { + fn recover_doc(&mut self, id: DocId, kind: Kind, seq: u64, hash: [u8; 32]) -> Vec { + if self.failed_again(id) { + return self.stop(id, &kind, None); + } self.reload(id); let (up, epoch) = (self.link == Link::Up, self.epoch); - let (_, doc) = doc_in(&mut self.docs, self.space, &app, &mut self.peers); + let Some(doc) = doc_again(&mut self.docs, self.space, &mut self.peers, id, &kind) else { return Vec::new() }; doc.skip = Some((seq, hash)); - let line = format!("a recents update for {app} from the server made Loro fail; it was skipped"); + let line = doc.told( + |app| format!("a recents update for {app} from the server made Loro fail; it was skipped, and this machine leaves compacting them to other devices"), + |id| format!("an update of the live document {id} from the server made Loro fail; it was skipped, and this machine leaves compacting it to other devices"), + ); + let frames = if up { subscribe(id, doc, epoch) } else { Vec::new() }; doc.say(&mut self.notes, line); - self.changed.insert(app); - if up { subscribe(id, doc, epoch) } else { Vec::new() } + if let Kind::Recents(app) = kind { + self.changed.insert(app); + } + frames } /// Loro failed on a waiting update of `id` imported again (`hash`): that one is dropped (the incoming update is /// not to blame, and comes again), and the document read back from its file and caught up from there. - fn recover_waiting(&mut self, id: DocId, app: AppName, hash: [u8; 32]) -> Vec { + fn recover_waiting_doc(&mut self, id: DocId, kind: Kind, hash: [u8; 32]) -> Vec { + if self.failed_again(id) { + return self.stop(id, &kind, Some(hash)); + } self.reload_without(id, Some(hash)); let (up, epoch) = (self.link == Link::Up, self.epoch); - let (_, doc) = doc_in(&mut self.docs, self.space, &app, &mut self.peers); - let line = format!("a waiting recents update for {app} made Loro fail; it was dropped"); + let Some(doc) = doc_again(&mut self.docs, self.space, &mut self.peers, id, &kind) else { return Vec::new() }; + let line = doc.told( + |app| format!("a waiting recents update for {app} made Loro fail; it was dropped"), + |id| format!("a waiting update of the live document {id} made Loro fail; it was dropped"), + ); + let frames = if up { subscribe(id, doc, epoch) } else { Vec::new() }; doc.say(&mut self.notes, line); - self.changed.insert(app); - if up { subscribe(id, doc, epoch) } else { Vec::new() } + if let Kind::Recents(app) = kind { + self.changed.insert(app); + } + frames + } + + /// Counts a row Loro failed on in `id`; whether it is the second in this run. + fn failed_again(&mut self, id: DocId) -> bool { + let failures = self.failures.entry(id).or_insert(0); + *failures += 1; + *failures >= MAX_HEALS + } + + /// Loro failed on a second row of `id` in this run: read back from its file and not followed again until the agent + /// restarts (another catch-up would go round the same failures). Its local changes are kept (on disk, and sent + /// after the restart); its apps are told ([`Replica::is_stopped`]). `drop`: a waiting update to leave out. + fn stop(&mut self, id: DocId, kind: &Kind, drop: Option<[u8; 32]>) -> Vec { + self.stopped.insert(id); + self.reload_without(id, drop); + let mut frames = Vec::new(); + let Some(doc) = doc_again(&mut self.docs, self.space, &mut self.peers, id, kind) else { return frames }; + doc.stopped = true; + if doc.sub != Sub::No && self.link == Link::Up { + frames.push(ClientFrame::Unsubscribe { doc: id }); + } + doc.sub = Sub::No; + doc.retry = None; + let line = doc.told( + |app| format!("Loro failed on the recents of {app} again; this machine stops following them until the agent restarts (what was added here is kept)"), + |id| format!("Loro failed on the live document {id} again; this machine stops following it until the agent restarts (the changes made here are kept)"), + ); + doc.say(&mut self.notes, line); + if let Kind::Recents(app) = kind { + self.changed.insert(app.clone()); + } + frames + } + + /// [`Replica::recover_doc`] of `app`'s recents (as the tests call it). + #[cfg(test)] + fn recover(&mut self, id: DocId, app: AppName, seq: u64, hash: [u8; 32]) -> Vec { + self.recover_doc(id, Kind::Recents(app), seq, hash) + } + + /// [`Replica::recover_waiting_doc`] of `app`'s recents (as the tests call it). + #[cfg(test)] + fn recover_waiting(&mut self, id: DocId, app: AppName, hash: [u8; 32]) -> Vec { + self.recover_waiting_doc(id, Kind::Recents(app), hash) } /// A connection to the server is open: the first frame to send. @@ -757,18 +1603,355 @@ impl Replica { vec![ClientFrame::Hello { min: LIVE_PROTOCOL_MIN, max: LIVE_PROTOCOL_MAX }] } - /// The connection ended: everything is subscribed and sent again on the next one. + /// The connection ended: everything is subscribed and sent again on the next one. A document of a file still being + /// joined is given up on (it was never live here): the apps that wait for it open the file as a plain file. pub fn disconnected(&mut self) { self.link = Link::Down; self.epoch = None; + self.protocol = 0; + let joining: Vec = self.docs.iter().filter(|(_, d)| !d.joined).map(|(id, _)| *id).collect(); + for id in joining { + self.docs.remove(&id); + self.events.push(DocEvent::NotLive(id, NotLiveReason::Offline, "the connection to the server went".into())); + } for doc in self.docs.values_mut() { doc.sub = Sub::No; doc.sent.clear(); + doc.taken_back.clear(); + doc.back_acked.clear(); doc.retry = None; doc.create_refused = false; doc.full = None; doc.stuck = false; + doc.creating = false; + doc.compact_ask = None; + doc.stale = false; + doc.from_snapshot = false; + // (The server sends them again with the next catch-up.) + doc.batch.clear(); + doc.batched = 0; + doc.fetching = None; + doc.deferred_head = None; + // (The frames that would have named them are gone with the connection.) + doc.uploads.clear(); + } + self.fetches.clear(); + } + + /// Whether the server speaks documents of files: `None` while the link isn't up. + pub fn documents_supported(&self) -> Option { + (self.link == Link::Up).then_some(self.protocol >= LIVE_DOCUMENTS_PROTOCOL) + } + + /// Opens the document of the file node `node` for `app`: one live here already is followed again (if it was put + /// aside) and opens from its copy, online or not; one that isn't is joined, which needs the server. Frames to send. + pub fn open_document(&mut self, node: NodeId, app: &AppName) -> (DocId, Opening, Vec) { + let id = node_doc(self.space, node); + self.uses += 1; + let (up, epoch, uses) = (self.link == Link::Up, self.epoch, self.uses); + let documents = self.protocol >= LIVE_DOCUMENTS_PROTOCOL; + // One on disk is loaded now. + self.load(id); + if let Some(doc) = self.docs.get_mut(&id) { + doc.last_used = uses; + if !doc.joined { + return (id, Opening::Joining, Vec::new()); + } + let was_parked = std::mem::replace(&mut doc.parked, false); + // Room first (an unsubscribe before the subscribe that needs its slot). + let mut frames = self.unload(); + if was_parked { + frames.extend(self.evict()); + } + let Some(doc) = self.docs.get_mut(&id) else { return (id, Opening::Live, frames) }; + if up && documents && doc.sub == Sub::No && doc.retry.is_none() { + frames.extend(subscribe(id, doc, epoch)); + } + return (id, Opening::Live, frames); + } + if !up { + return (id, Opening::NotLive(NotLiveReason::Offline, "the agent can't reach the server".into()), Vec::new()); } + if !documents { + return (id, Opening::NotLive(NotLiveReason::Offline, "live documents need a newer Craft Cloud server".into()), Vec::new()); + } + let peer = (self.peers)(); + let kind = Kind::Document { node, app: app.clone() }; + let mut doc = Doc::with(id, kind.clone(), Body::new(&kind, peer), Have::default(), Vec::new(), Vec::new(), 1, epoch, None); + doc.joined = false; + doc.join = Some(Join::Probing); + doc.last_used = uses; + doc.log = LogVersion::anchored(0, Version::new()); + let subscribe = doc.resubscribe(id); + self.docs.insert(id, doc); + // Room first (an unsubscribe before the subscribe that needs its slot). + let mut frames = self.unload(); + frames.extend(self.evict()); + frames.extend(subscribe); + (id, Opening::Joining, frames) + } + + /// The app's import of a file whose document the server doesn't have (after [`DocEvent::NeedsImport`]): a Loro + /// snapshot, checked here, sent up as `create`. Frames to send, or why it was refused (the file then opens as a + /// plain file). + pub fn import_document(&mut self, id: DocId, snapshot: &[u8]) -> Result, (NotLiveReason, String)> { + let Some(doc) = self.docs.get_mut(&id) else { return Err((NotLiveReason::Offline, "the document isn't being opened any more".into())) }; + if doc.join != Some(Join::NeedsImport) { + return Err((NotLiveReason::Offline, "no import is waited for".into())); + } + let Kind::Document { node, app } = doc.kind.clone() else { return Err((NotLiveReason::NotInFolder, "not a document of a file".into())) }; + if snapshot.is_empty() || snapshot.len() > MAX_SNAPSHOT_BYTES { + self.docs.remove(&id); + return Err((NotLiveReason::TooLarge, format!("an import is 1 byte to {MAX_SNAPSHOT_BYTES} bytes"))); + } + // Checked by Loro before it goes anywhere: an import that doesn't read is never made a document. + let peer = (self.peers)(); + match LiveCopy::load(snapshot, peer) { + Ok(copy) => { + // (A copy Loro failed in is let go of without dropping.) + if copy.is_poisoned() { + std::mem::forget(copy); + } + } + Err(e) => { + self.docs.remove(&id); + return Err((NotLiveReason::Offline, format!("the import doesn't read: {e}"))); + } + } + if self.link != Link::Up { + self.docs.remove(&id); + return Err((NotLiveReason::Offline, "the connection to the server went".into())); + } + // Uploaded first (through the chunk API), then named in `create`. + let upload = match cut(id, snapshot) { + Ok(upload) => upload, + Err(e) => { + self.docs.remove(&id); + return Err((NotLiveReason::TooLarge, e)); + } + }; + let manifest = upload.id(); + doc.uploads.push(upload); + doc.join = Some(Join::Creating(snapshot.to_vec())); + Ok(vec![ClientFrame::CreateDocument { doc: id, node, app, snapshot: manifest }]) + } + + /// The snapshots to upload now, each before the frames of its document that follow it (a frame names one by its + /// manifest). One that can't be uploaded is said with [`Replica::upload_failed`]. + pub fn take_uploads(&mut self) -> Vec { + self.docs.values_mut().flat_map(|d| std::mem::take(&mut d.uploads)).collect() + } + + /// A snapshot of `doc` couldn't be uploaded (the frame that names it isn't sent): an import's join is given up on + /// (the app opens the file as it is), a joined document's making again is tried later, and a compaction's answer + /// is just not given (another device is asked). + pub fn upload_failed(&mut self, id: DocId, why: &str) { + let Some(doc) = self.docs.get_mut(&id) else { return }; + if matches!(doc.join, Some(Join::Creating(_))) { + self.docs.remove(&id); + self.events.push(DocEvent::NotLive(id, NotLiveReason::Offline, format!("the import couldn't go up to the server: {why}"))); + } else if doc.creating { + doc.creating = false; + doc.sub = Sub::No; + let pause = doc.retry.map_or(1, |(_, pause)| pause.max(1)); + doc.retry = Some((pause, pause)); + } + } + + /// The snapshots of catch-ups to download now, by document and manifest; each is handed back with + /// [`Replica::on_snapshot`]. + pub fn take_fetches(&mut self) -> Vec<(DocId, ManifestId)> { + std::mem::take(&mut self.fetches) + } + + /// An app that was to import a document is gone (it closed its connection mid-import): the join is given up on; + /// nothing was sent to the server. + pub fn abandon_join(&mut self, id: DocId) -> Vec { + match self.docs.get(&id) { + Some(doc) if !doc.joined && matches!(doc.join, Some(Join::NeedsImport | Join::Probing)) => { + let following = doc.sub != Sub::No; + self.docs.remove(&id); + if following { vec![ClientFrame::Unsubscribe { doc: id }] } else { Vec::new() } + } + _ => Vec::new(), + } + } + + /// A document of a file live here: its state for an app opening it, a Loro snapshot, and the updates to import after + /// it: the server's still waiting for their history, and this machine's own not yet acked (in the snapshot already, + /// unless they wait for history too: after a restart, the server's updates they build on may not be on disk yet, + /// and Loro applies them only once those come again; the app's Loro does the same). + pub fn document_state(&self, id: DocId) -> Option<(Vec, Vec>)> { + let doc = self.docs.get(&id).filter(|d| d.joined && d.kind.is_document())?; + let snapshot = doc.body.copy().snapshot().ok()?; + let after = doc.waiting.iter().map(|w| w.bytes.clone()).chain(doc.pending.iter().map(|p| p.bytes.clone())).collect(); + Some((snapshot, after)) + } + + /// Whether a document being joined waits for an app's import (the server doesn't have it). + pub fn needs_import(&self, id: DocId) -> bool { + self.docs.get(&id).is_some_and(|d| d.join == Some(Join::NeedsImport)) + } + + /// Whether a document of a file is live here (it opens offline). + pub fn has_document(&self, id: DocId) -> bool { + self.unloaded.contains(&id) || self.docs.get(&id).is_some_and(|d| d.joined && d.kind.is_document()) + } + + /// The node a document of a file belongs to. + pub fn document_node(&self, id: DocId) -> Option { + match self.docs.get(&id).map(|d| &d.kind) { + Some(Kind::Document { node, .. }) => Some(*node), + _ => None, + } + } + + /// How a document stands: whether its edits go out as they are made, and how many of this machine's wait for the + /// server's ack. + pub fn document_status(&self, id: DocId) -> Option<(bool, u64)> { + let doc = self.docs.get(&id)?; + let online = self.link == Link::Up && doc.sub == Sub::Live && doc.full.is_none(); + Some((online, (doc.pending.len() + usize::from(doc.pushing.is_some())) as u64)) + } + + /// An app has (`true`) or no longer has (`false`) the document open: one no app has open may be unfollowed past + /// the device's cap. + pub fn set_open(&mut self, id: DocId, open: bool) { + if let Some(doc) = self.docs.get_mut(&id) { + doc.open = if open { doc.open.saturating_add(1) } else { doc.open.saturating_sub(1) }; + } + } + + /// An app's own change to a document of a file: imported into the copy with Loro's checked import (refused, it + /// leaves the copy as it was; Loro failing reads the document back from disk), on disk in its journal before this + /// returns, and sent when the link is up. Returns the frames to send. + pub fn local_update(&mut self, id: DocId, bytes: &[u8]) -> Result, LocalError> { + let Some(doc) = self.docs.get_mut(&id).filter(|d| d.joined && d.kind.is_document()) else { + return Err(LocalError::Invalid("the document isn't open here".into())); + }; + if bytes.is_empty() || bytes.len() > MAX_UPDATE_BYTES { + return Err(LocalError::Invalid(format!("an update is 1 to {MAX_UPDATE_BYTES} bytes"))); + } + // Its journal needs its file beside it (a journal alone isn't loaded): a document joined since it was last + // written is written first (as it is, before this update). So is one whose journal failed: written whole, its + // journal starts empty. + if !doc.on_disk || doc.journal_failed { + let space = self.space; + save(&mut self.store, space, id, doc)?; + doc.journal_failed = false; + } + match doc.body.import(bytes) { + Ok(_) => {} + Err(UpdateError::Panicked) => { + self.reload(id); + self.follow_again_soon(id); + return Err(LocalError::Invalid("Loro failed on the update; it was dropped and the document read back from disk".into())); + } + Err(e) => return Err(LocalError::Invalid(e.to_string())), + } + let client_seq = doc.next_client_seq; + let before = doc.journal; + let record = JournalRecord { client_seq, bytes: bytes.to_vec() }; + let name = id.to_string(); + let written = journal_record(&record).and_then(|rec| self.store.append_journal(&name, &rec).map(|()| rec.len())); + let Some(doc) = self.docs.get_mut(&id) else { return Err(LocalError::Invalid("the document is gone".into())) }; + match written { + Ok(len) => doc.journal = doc.journal.saturating_add(len), + Err(e) => { + // Not kept: the journal is cut back to where it was (whatever part of the record got there must not + // come back after a crash, nor hide what comes after it), the copy goes back to what is on disk, the + // app is told, and the document is written whole before its journal takes another change. + let _ = self.store.truncate_journal(&name, before); + self.reload(id); + if let Some(doc) = self.docs.get_mut(&id) { + doc.journal_failed = true; + doc.urgent = true; + } + self.follow_again_soon(id); + return Err(e.into()); + } + } + doc.next_client_seq = client_seq.saturating_add(1); + doc.pending.push(Pending { client_seq, bytes: bytes.to_vec(), acked: None, push: false }); + doc.own(bytes); + doc.dirty = true; + let (up, epoch) = (self.link == Link::Up, self.epoch); + Ok(match doc.sub { + // (During a push too: an app's change goes at once, not behind the push's parts. Its operations count as + // in the log for the push once it is acked.) + Sub::Live if doc.full.is_none() => { + doc.sent.insert(client_seq); + vec![ClientFrame::Update { doc: id, client_seq, bytes: bytes.to_vec() }] + } + Sub::No if up && self.protocol >= LIVE_DOCUMENTS_PROTOCOL && doc.retry.is_none() => { + doc.parked = false; + subscribe(id, doc, epoch) + } + Sub::No | Sub::CatchingUp | Sub::Live => Vec::new(), + }) + } + + /// A document read back from its file (so unfollowed) while the link is up: followed again at the next tick. + fn follow_again_soon(&mut self, id: DocId) { + let up = self.link == Link::Up && self.protocol >= LIVE_DOCUMENTS_PROTOCOL; + if let Some(doc) = self.docs.get_mut(&id) + && up + && doc.sub == Sub::No + { + doc.parked = false; + doc.retry = Some((1, 0)); + } + } + + /// An app's presence in a document of a file, for the document's other machines (only while it is followed). + pub fn presence(&mut self, id: DocId, bytes: &[u8]) -> Vec { + match self.docs.get(&id) { + Some(doc) if self.link == Link::Up && self.protocol >= LIVE_DOCUMENTS_PROTOCOL && doc.sub == Sub::Live && doc.joined => { + vec![ClientFrame::Presence { doc: id, bytes: bytes.to_vec() }] + } + _ => Vec::new(), + } + } + + /// What happened to documents of files since the last call. + pub fn take_events(&mut self) -> Vec { + std::mem::take(&mut self.events) + } + + /// Past the device's cap on followed documents: the least recently used ones no app has open and with nothing to + /// send are unfollowed (they stay on disk). Frames to send. + fn evict(&mut self) -> Vec { + let followed = self.docs.values().filter(|d| !d.parked).count(); + let Some(extra) = followed.checked_sub(MAX_SUBSCRIPTIONS).filter(|n| *n > 0) else { return Vec::new() }; + let mut idle: Vec<(u64, DocId)> = self + .docs + .iter() + .filter(|(_, d)| d.kind.is_document() && d.joined && !d.parked && d.open == 0 && d.pending.is_empty() && d.pushing.is_none()) + .map(|(id, d)| (d.last_used, *id)) + .collect(); + idle.sort_unstable(); + let mut frames = Vec::new(); + for (_, id) in idle.into_iter().take(extra) { + if let Some(doc) = self.docs.get_mut(&id) { + if doc.sub != Sub::No { + frames.push(ClientFrame::Unsubscribe { doc: id }); + } + doc.sub = Sub::No; + doc.retry = None; + doc.parked = true; + } + } + frames + } + + /// How many documents of files are loaded (in memory), for tests and the status. + pub fn loaded_documents(&self) -> usize { + self.docs.values().filter(|d| d.kind.is_document()).count() + } + + /// Whether Loro failed on the document twice in this run: it isn't followed again until the agent restarts. + pub fn is_stopped(&self, id: DocId) -> bool { + self.stopped.contains(&id) } /// Whether the link is up (welcomed). @@ -793,7 +1976,12 @@ impl Replica { /// How `app`'s document stands, in words (for a simulation that wants to say why it waits). pub fn describe(&self, app: &AppName) -> String { - match self.docs.get(&self.doc_id(app)) { + self.describe_doc(self.doc_id(app)) + } + + /// [`Replica::describe`], of any document. + pub fn describe_doc(&self, id: DocId) -> String { + match self.docs.get(&id) { None => "no document".into(), Some(d) => format!( "{:?}, upto {}, {} unacked ({} sent, {} push, acked {:?}), pushing {:?}, wait {:?}, full {:?}, stuck {}, retry {:?}", @@ -803,7 +1991,7 @@ impl Replica { d.sent.len(), d.pending.iter().filter(|p| p.push).count(), d.pending.iter().filter_map(|p| p.acked).collect::>(), - d.pushing.as_ref().map(|has| d.recents.missing(has)), + d.pushing.as_ref().map(|has| d.body.copy().missing(has)), d.push_wait, d.full, d.stuck, @@ -847,42 +2035,67 @@ impl Replica { /// Every entry of `app`'s document that reads, by key (for a simulation that wants to look inside). pub fn readable(&self, app: &AppName) -> Vec<(String, RecentEntry)> { - self.docs.get(&self.doc_id(app)).map(|d| d.recents.readable()).unwrap_or_default() + self.docs + .get(&self.doc_id(app)) + .and_then(|d| match &d.body { + Body::Recents(recents) => Some(recents.readable()), + Body::Document(_) => None, + }) + .unwrap_or_default() } /// A document as a Loro snapshot (for a simulation that wants to look inside). pub fn snapshot_of(&self, doc: DocId) -> Option> { - self.docs.get(&doc).and_then(|d| d.recents.snapshot().ok()) + self.docs.get(&doc).and_then(|d| d.body.copy().snapshot().ok()) } /// Documents Loro failed in outside an import: read back from their files, and caught up again from there. /// After the second heal of a document in one run, its push stops (the same state would fail the same way again): /// its unacked adds go up as plain updates, and [`Replica::push_stopped`] says so for the app to show. fn heal(&mut self) -> Vec { - let broken: Vec<(DocId, AppName)> = self.docs.iter().filter(|(_, d)| d.is_broken()).map(|(id, d)| (*id, d.app.clone())).collect(); + let broken: Vec<(DocId, Kind)> = self.docs.iter().filter(|(_, d)| d.is_broken()).map(|(id, d)| (*id, d.kind.clone())).collect(); let mut frames = Vec::new(); - for (id, app) in broken { + for (id, kind) in broken { + // A document of a file that isn't joined yet has no file: it is given up on, and its apps told. + if self.docs.get(&id).is_some_and(|d| !d.joined) { + if let Some(old) = self.docs.remove(&id) { + std::mem::forget(old); + self.forgotten += 1; + } + self.events.push(DocEvent::NotLive(id, NotLiveReason::Offline, "Loro failed on the document".into())); + continue; + } self.reload(id); let heals = self.heals.entry(id).or_insert(0); *heals += 1; let stop = *heals >= MAX_HEALS; - let (up, epoch) = (self.link == Link::Up, self.epoch); - let (_, doc) = doc_in(&mut self.docs, self.space, &app, &mut self.peers); - let line = format!("Loro failed on the recents of {app}; they were read back from disk"); + let (up, epoch, documents) = (self.link == Link::Up, self.epoch, self.protocol >= LIVE_DOCUMENTS_PROTOCOL); + let Some(doc) = doc_again(&mut self.docs, self.space, &mut self.peers, id, &kind) else { continue }; + let line = doc.told( + |app| format!("Loro failed on the recents of {app}; they were read back from disk"), + |id| format!("Loro failed on the live document {id}; it was read back from disk"), + ); doc.say(&mut self.notes, line); if stop && doc.pushing.is_some() { doc.pushing = None; doc.push_wait = None; doc.dirty = true; - self.push_stopped.insert(app.clone()); - // The app's followers are told (their list event carries the status). - self.changed.insert(app.clone()); - let line = format!( - "the recents of {app} can't be sent up whole after Loro failed again; new ones are sent one by one, and what the server lacks of the rest stays on this machine" + let line = doc.told( + |app| format!( + "the recents of {app} can't be sent up whole after Loro failed again; new ones are sent one by one, and what the server lacks of the rest stays on this machine" + ), + |id| format!( + "the live document {id} can't be sent up whole after Loro failed again; new changes are sent one by one, and what the server lacks of the rest stays on this machine" + ), ); doc.say(&mut self.notes, line); + if let Kind::Recents(app) = &kind { + self.push_stopped.insert(app.clone()); + // The app's followers are told (their list event carries the status). + self.changed.insert(app.clone()); + } } - if up { + if up && (!kind.is_document() || (documents && !doc.parked)) { frames.extend(subscribe(id, doc, epoch)); } } @@ -953,11 +2166,20 @@ impl Replica { fn on_frame_inner(&mut self, frame: ServerFrame) -> Result, LinkError> { match frame { - ServerFrame::Welcome { epoch, .. } => { + ServerFrame::Welcome { protocol, epoch } => { self.link = Link::Up; self.epoch = Some(epoch); + self.protocol = protocol; + let documents = protocol >= LIVE_DOCUMENTS_PROTOCOL; + if !documents && self.docs.values().any(|d| d.kind.is_document()) { + self.notes.push("live documents need a newer Craft Cloud server; their changes stay on this machine until it is updated".into()); + } let mut frames = Vec::new(); for (id, doc) in &mut self.docs { + // Documents of files only with a server that has them, and only those followed. + if doc.kind.is_document() && (!documents || doc.parked || !doc.joined) { + continue; + } frames.extend(subscribe(*id, doc, Some(epoch))); } Ok(frames) @@ -965,67 +2187,439 @@ impl Replica { ServerFrame::Created { .. } => Ok(Vec::new()), ServerFrame::Update { doc: id, seq, bytes, .. } => { let Some(doc) = self.docs.get_mut(&id) else { return Ok(Vec::new()) }; - if seq <= doc.have.upto || doc.stuck { + // While the snapshot the catch-up started with downloads, its updates wait here (it comes first). + if doc.fetching.is_some() && doc.sub == Sub::CatchingUp { + doc.batched = doc.batched.saturating_add(bytes.len()); + doc.batch.push((seq, bytes)); return Ok(Vec::new()); } - if seq > doc.have.upto.saturating_add(1) { - // A gap: a catch-up under way brings it; else the server's updates were lost on the way (an old - // connection), and a catch-up from what it has brings them. - return Ok(if doc.sub == Sub::Live { doc.resubscribe(id) } else { Vec::new() }); - } - let mut changed = false; - let taken = doc.take(seq, &bytes, &mut self.notes, &mut changed); - let app = doc.app.clone(); - if changed { - self.changed.insert(app.clone()); + // A catch-up of a document of a file is taken a batch at a time (one Loro import for many updates is + // far quicker on a big document): the updates wait here until `subscribed`, or a few MiB of them. + if doc.kind.is_document() && doc.sub == Sub::CatchingUp && !doc.stale { + doc.batched = doc.batched.saturating_add(bytes.len()); + doc.batch.push((seq, bytes)); + return Ok(if doc.batched > MAX_BATCH_BYTES { self.take_batch(id) } else { Vec::new() }); } - Ok(match taken { - Taken::Had | Taken::Stuck => Vec::new(), - Taken::Panicked => self.recover(id, app, seq, update_hash(&bytes)), - Taken::WaitingPanicked(hash) => self.recover_waiting(id, app, hash), - }) + Ok(self.on_update(id, seq, bytes)) } ServerFrame::Subscribed { doc: id, head } => { - let Some(doc) = self.docs.get_mut(&id) else { return Ok(Vec::new()) }; - if head < doc.have.upto { - // The server has less than this machine read of it (restored without a new epoch): from the start, - // so `have` names updates of this log, and then what the log lacks goes up. - doc.start_over(); - return Ok(doc.resubscribe(id)); + if let Some(doc) = self.docs.get_mut(&id) + && doc.fetching.is_some() + && doc.sub == Sub::CatchingUp + { + doc.deferred_head = Some(head); + return Ok(Vec::new()); } - doc.sub = Sub::Live; - doc.retry = None; - doc.create_refused = false; - let mut frames = send_pending(id, doc); - frames.extend(doc.push_next(id, &mut self.notes, &mut self.peers)); + let mut frames = self.take_batch(id); + frames.extend(self.on_subscribed(id, head)); Ok(frames) } - ServerFrame::Ack { doc: id, client_seq, seq } => { + ServerFrame::Ack { doc: id, client_seq, seq } => Ok(self.on_ack(id, client_seq, seq)), + other => self.on_other(other), + } + } + + /// The snapshot `manifest` names, which the catch-up of `id` started with, downloaded (`Ok`), or not (`Err`: it + /// can't be got or doesn't read; this machine then follows the whole log instead, `subscribe_log`). Then the + /// catch-up's updates that waited for it are taken, and its `subscribed` if it came. Frames to send. + pub fn on_snapshot(&mut self, id: DocId, manifest: ManifestId, bytes: Result, String>) -> Vec { + let mut frames = self.on_snapshot_inner(id, manifest, bytes); + frames.extend(self.heal()); + frames + } + + fn on_snapshot_inner(&mut self, id: DocId, manifest: ManifestId, bytes: Result, String>) -> Vec { + let Some(doc) = self.docs.get_mut(&id) else { return Vec::new() }; + let Some((base, _)) = doc.fetching.filter(|(_, m)| *m == manifest) else { return Vec::new() }; + doc.fetching = None; + let head = doc.deferred_head.take(); + let mut changed = false; + let taken = match &bytes { + Ok(bytes) => doc.take_snapshot(base, bytes, &mut self.notes, &mut changed), + Err(why) => { + // As one that doesn't read: the whole log instead. + let line = doc.told( + |app| { + format!("the server's snapshot of the recents of {app} can't be got here ({why}); this machine reads the whole log instead") + }, + |id| { + format!("the server's snapshot of the live document {id} can't be got here ({why}); this machine reads the whole log instead") + }, + ); + doc.say(&mut self.notes, line); + doc.log_only = true; + doc.stale = true; + doc.dirty = true; + doc.urgent = true; + Err(UpdateError::Invalid("not got")) + } + }; + let kind = doc.kind.clone(); + if changed { + match &kind { + Kind::Recents(app) => { + self.changed.insert(app.clone()); + } + Kind::Document { .. } => { + if let Ok(bytes) = bytes { + self.events.push(DocEvent::Remote(id, bytes)); + } + } + } + } + match taken { + Ok(()) => { + // The updates that waited, and the end of the catch-up. + let mut frames = self.take_batch(id); + if let Some(head) = head { + frames.extend(self.on_subscribed(id, head)); + } + frames + } + // Read back from its file (Loro failed in it), and caught up again with the whole log. + Err(UpdateError::Panicked) => { + self.reload(id); + let (up, epoch) = (self.link == Link::Up, self.epoch); + match doc_again(&mut self.docs, self.space, &mut self.peers, id, &kind) { + Some(doc) if up => { + doc.log_only = true; + subscribe(id, doc, epoch) + } + _ => Vec::new(), + } + } + // Given up on: what waited goes (the next catch-up, of the whole log, brings it); its `subscribed` starts + // that one. + Err(_) => { + if let Some(doc) = self.docs.get_mut(&id) { + doc.batch.clear(); + doc.batched = 0; + } + head.map(|head| self.on_subscribed(id, head)).unwrap_or_default() + } + } + } + + /// The updates a catch-up batched: imported into the copy at once, then each taken as it would have been (Loro + /// knows them then, so that is quick). Remote changes go to the apps as they came. The row Loro failed on before + /// is left out of the batch (it is skipped when taken). If Loro fails on the batch, the document is read back + /// from its file and the catch-up taken again one row at a time for the rest of the run, so the row at fault is + /// found and only it dropped ([`Replica::recover_doc`]). + fn take_batch(&mut self, id: DocId) -> Vec { + let one_by_one = self.one_by_one.contains(&id); + let Some(doc) = self.docs.get_mut(&id) else { return Vec::new() }; + let batch = std::mem::take(&mut doc.batch); + doc.batched = 0; + if batch.is_empty() { + return Vec::new(); + } + let new = if one_by_one { + // (Each row's own import says what it brought.) + false + } else { + let updates: Vec> = + batch.iter().filter(|(seq, bytes)| doc.skip != Some((*seq, update_hash(bytes)))).map(|(_, bytes)| bytes.clone()).collect(); + match doc.body.copy().import_batch(&updates) { + Ok(new) => new, + Err(UpdateError::Panicked) => return self.take_one_by_one(id), + Err(_) => false, + } + }; + let kind = doc.kind.clone(); + let mut frames = Vec::new(); + if new && let Kind::Recents(app) = &kind { + self.changed.insert(app.clone()); + } + for (seq, bytes) in batch { + if new && kind.is_document() { + self.events.push(DocEvent::Remote(id, bytes.clone())); + } + frames.extend(self.on_update(id, seq, bytes)); + } + frames + } + + /// Loro failed on a catch-up's batch of `id`: read back from its file, and caught up again from there one row at a + /// time for the rest of the run. + fn take_one_by_one(&mut self, id: DocId) -> Vec { + let Some(kind) = self.docs.get(&id).map(|d| d.kind.clone()) else { return Vec::new() }; + self.one_by_one.insert(id); + self.reload(id); + let (up, epoch) = (self.link == Link::Up, self.epoch); + let Some(doc) = doc_again(&mut self.docs, self.space, &mut self.peers, id, &kind) else { return Vec::new() }; + if up { subscribe(id, doc, epoch) } else { Vec::new() } + } + + /// An update of a document's log from the server. + fn on_update(&mut self, id: DocId, seq: u64, bytes: Vec) -> Vec { + let Some(doc) = self.docs.get_mut(&id) else { return Vec::new() }; + // The update at the base of a snapshot the catch-up started with: its hash names it in the next `at`. + if seq == doc.have.upto && seq > 0 && doc.have.upto_hash.is_none() { + doc.have.upto_hash = Some(update_hash(&bytes)); + doc.dirty = true; + return Vec::new(); + } + if seq <= doc.have.upto || doc.stuck || doc.stopped { + return Vec::new(); + } + if seq > doc.have.upto.saturating_add(1) { + // A gap: a catch-up under way brings it; else the server's updates were lost on the way (an old + // connection), and a catch-up from what it has brings them. + return if doc.sub == Sub::Live { doc.resubscribe(id) } else { Vec::new() }; + } + let mut changed = false; + let taken = doc.take(seq, &bytes, &mut self.notes, &mut changed); + let kind = doc.kind.clone(); + let mut frames = if matches!(taken, Taken::Had) { doc.try_compact() } else { Vec::new() }; + if changed { + match &kind { + Kind::Recents(app) => { + self.changed.insert(app.clone()); + } + Kind::Document { .. } => self.events.push(DocEvent::Remote(id, bytes.clone())), + } + } + frames.extend(match taken { + Taken::Had | Taken::Stuck => Vec::new(), + Taken::Panicked => self.recover_doc(id, kind, seq, update_hash(&bytes)), + Taken::WaitingPanicked(hash) => self.recover_waiting_doc(id, kind, hash), + }); + frames + } + + /// `subscribed {doc, head}`: the end of a catch-up. + fn on_subscribed(&mut self, id: DocId, head: u64) -> Vec { + let Some(doc) = self.docs.get_mut(&id) else { return Vec::new() }; + if doc.stale { + // The end of a catch-up given up on: the next one, from where it is. + doc.stale = false; + return doc.resubscribe(id); + } + if head < doc.have.upto { + // The server has less than this machine read of it (restored without a new epoch): from the start, + // so `have` names updates of this log, and then what the log lacks goes up. + doc.from_snapshot = false; + doc.start_over(); + return doc.resubscribe(id); + } + let from_snapshot = doc.from_snapshot; + let again = doc.check_snapshot(id, &mut self.notes); + if !again.is_empty() { + return again; + } + if matches!(doc.join, Some(Join::Probing | Join::Following)) && head == 0 && !from_snapshot { + // The server has the document but nothing of it (no update, no snapshot): an admin reset it, and its + // machines haven't sent it back yet. An app here would get an empty document and might save it over its + // file: the file is what to open until then. + self.docs.remove(&id); + self.events.push(DocEvent::NotLive( + id, + NotLiveReason::Resetting, + "the server's live document was reset and isn't back yet; open the file as it is".into(), + )); + return vec![ClientFrame::Unsubscribe { doc: id }]; + } + doc.sub = Sub::Live; + doc.retry = None; + doc.create_refused = false; + if matches!(doc.join, Some(Join::Probing | Join::Following)) { + // The server has it, and this machine all of it: live here from now on. + doc.join = None; + // (Written with the next of the documents' writes: until then the server has all of it.) + doc.joined = true; + doc.dirty = true; + self.events.push(DocEvent::Joined(id)); + } + let mut frames = send_pending(id, doc); + frames.extend(doc.push_next(id, &mut self.notes, &mut self.peers)); + frames.extend(doc.try_compact()); + frames + } + + /// `ack {doc, client_seq, seq}`: one of this machine's updates is in the log. + fn on_ack(&mut self, id: DocId, client_seq: u64, seq: u64) -> Vec { + let Some(doc) = self.docs.get_mut(&id) else { return Vec::new() }; + let Some(i) = doc.pending.iter().position(|p| p.client_seq == client_seq) else { + // Stored already as far as this machine knew (the log held its operations before the ack came): the acked + // row holds them, and counts as had like any ack (or waits for the rows below it, as any ack does). + let Some(bytes) = doc.taken_back.remove(&client_seq) else { return Vec::new() }; + if seq == doc.have.upto.saturating_add(1) { + let ranges = spans_of(&bytes).unwrap_or_default(); + doc.saw(&ranges); + doc.log.note(seq, ranges); + doc.advance(seq, update_hash(&bytes)); + } else if seq > doc.have.upto { + doc.back_acked.insert(seq, bytes); + } + let mut frames = if doc.pushing.is_some() { doc.push_next(id, &mut self.notes, &mut self.peers) } else { Vec::new() }; + frames.extend(doc.try_compact()); + return frames; + }; + doc.strikes = 0; + doc.full_pause = 0; + doc.dirty = true; + let push = doc.pending.get(i).is_some_and(|p| p.push); + if let Some(bytes) = doc.pending.get(i).map(|p| p.bytes.clone()) { + doc.on_server(&bytes); + } + if seq <= doc.have.upto { + doc.pending.remove(i); + doc.sent.remove(&client_seq); + } else if seq == doc.have.upto.saturating_add(1) { + let own = doc.pending.remove(i); + doc.sent.remove(&client_seq); + let ranges = spans_of(&own.bytes).unwrap_or_default(); + doc.saw(&ranges); + doc.log.note(seq, ranges); + doc.advance(seq, update_hash(&own.bytes)); + } else if let Some(pending) = doc.pending.get_mut(i) { + // Others' updates below it are on their way: counted once they are here. (A push part counts as + // sent: the next one is computed without it.) + pending.acked = Some(seq); + pending.push = false; + } + let mut frames = if push { doc.push_next(id, &mut self.notes, &mut self.peers) } else { Vec::new() }; + frames.extend(doc.try_compact()); + frames + } + + /// The frames [`Replica::on_frame_inner`] doesn't take itself. + fn on_other(&mut self, frame: ServerFrame) -> Result, LinkError> { + match frame { + ServerFrame::Welcome { .. } | ServerFrame::Update { .. } | ServerFrame::Subscribed { .. } | ServerFrame::Ack { .. } => Ok(Vec::new()), + ServerFrame::Created { .. } => Ok(Vec::new()), + ServerFrame::Error { code, message, doc, client_seq } => self.refused(&code, &message, doc, client_seq), + ServerFrame::Compact { doc: id, upto } => { + let Some(doc) = self.docs.get_mut(&id) else { return Ok(Vec::new()) }; + doc.compact_ask = Some(upto); + Ok(doc.try_compact()) + } + ServerFrame::Snapshot { doc: id, base, manifest } => { let Some(doc) = self.docs.get_mut(&id) else { return Ok(Vec::new()) }; - let Some(i) = doc.pending.iter().position(|p| p.client_seq == client_seq) else { return Ok(Vec::new()) }; - doc.strikes = 0; - doc.full_pause = 0; + // Only as the start of a catch-up (a stray one, from a connection given up on, is no news). It is + // downloaded through the chunk API; the catch-up's updates wait for it ([`Replica::on_snapshot`]). + if doc.sub != Sub::CatchingUp || doc.stale || doc.fetching.is_some() { + return Ok(Vec::new()); + } + doc.fetching = Some((base, manifest)); + self.fetches.push((id, manifest)); + Ok(Vec::new()) + } + ServerFrame::Presence { doc: id, bytes } => { + if self.docs.get(&id).is_some_and(|d| d.joined && d.kind.is_document()) { + self.events.push(DocEvent::Presence(id, bytes)); + } + Ok(Vec::new()) + } + ServerFrame::CreatedDocument { doc: id, kept } => Ok(self.created(id, kept)), + } + } + + /// A refused `create` or `subscribe` of a document of a file, where it differs from recents: `Some(frames)` when + /// handled here. + fn refused_document(&mut self, id: DocId, code: &str, message: &str) -> Option> { + let doc = self.docs.get_mut(&id).filter(|d| d.kind.is_document())?; + let Kind::Document { node, app } = doc.kind.clone() else { return None }; + match (&doc.join, code) { + // The server doesn't have it: the app is asked for its import. + (Some(Join::Probing | Join::Following), codes::NOT_FOUND) => { + doc.sub = Sub::No; + doc.join = Some(Join::NeedsImport); + self.events.push(DocEvent::NeedsImport(id)); + Some(Vec::new()) + } + // The import was refused: the file opens as a plain file (and is tried again at the next open). + (Some(Join::Creating(_)), _) => { + let reason = match code { + codes::NOT_FOUND => NotLiveReason::NotSynced, + codes::TOO_LARGE | codes::QUOTA_EXCEEDED => NotLiveReason::TooLarge, + _ => NotLiveReason::Offline, + }; + self.docs.remove(&id); + self.events.push(DocEvent::NotLive(id, reason, format!("the server refused the import ({code}: {message})"))); + Some(Vec::new()) + } + // Refused while joining for another reason: given up on for now. + (Some(_), _) if code != codes::RATE_LIMITED => { + self.docs.remove(&id); + self.events.push(DocEvent::NotLive(id, NotLiveReason::Offline, format!("the server can't open it now ({code}: {message})"))); + Some(Vec::new()) + } + // A joined document the server doesn't have (a restore from before it was made, or a reset): made again from + // this machine's copy; nothing it holds is lost. If that is refused too (the file is gone on the server), + // tried again after a growing pause. + (None, codes::NOT_FOUND) if !doc.creating => { + let upload = doc.body.copy().snapshot().map_err(|e| e.to_string()).and_then(|s| cut(id, &s)); + let upload = match upload { + Ok(upload) => upload, + Err(e) => { + // (Over the snapshot limit, say: it can't be made again from here. Said, and tried again later + // as any refusal, in case it shrinks.) + doc.sub = Sub::No; + let pause = doc.retry.map_or(1, |(_, pause)| pause.max(1)); + doc.retry = Some((pause, pause)); + let line = + format!("the server lost the live document {id}, and this machine can't make it again ({e}); it stays on this machine"); + doc.say(&mut self.notes, line); + return Some(Vec::new()); + } + }; + doc.sub = Sub::No; + doc.creating = true; + let manifest = upload.id(); + doc.uploads.push(upload); + Some(vec![ClientFrame::CreateDocument { doc: id, node, app, snapshot: manifest }]) + } + (None, codes::NOT_FOUND | codes::QUOTA_EXCEEDED | codes::TOO_LARGE) => { + doc.creating = false; + let pause = doc.retry.map_or(1, |(_, pause)| pause.max(1)); + doc.sub = Sub::No; + doc.retry = Some((pause, pause)); + let line = format!("the server can't make the live document {id} again ({code}: {message}); it stays on this machine"); + doc.say(&mut self.notes, line); + Some(Vec::new()) + } + _ => None, + } + } + + /// `created {doc, kept}`: this machine's import is the document (kept), or another machine's is (not kept: this one + /// is dropped, and the document caught up from the start). A joined document made again from its copy (its + /// server lost it) starts over either way, so what the log lacks of it goes up. + fn created(&mut self, id: DocId, kept: bool) -> Vec { + let Some(doc) = self.docs.get_mut(&id) else { return Vec::new() }; + if doc.creating { + doc.creating = false; + doc.start_over(); + return doc.resubscribe(id); + } + let Some(Join::Creating(snapshot)) = doc.join.take() else { return Vec::new() }; + if !kept { + doc.join = Some(Join::Following); + return doc.resubscribe(id); + } + let peer = doc.body.copy().peer(); + match Body::load(&doc.kind, &snapshot, peer) { + Ok(body) => { + doc.body = body; + doc.own_peers = vec![peer]; + // The import is this machine's app's. + doc.own(&snapshot); + let version = version_of(&snapshot).unwrap_or_default(); + let ranges: Vec<(u64, i32, i32)> = version.iter().map(|(peer, end)| (*peer, 0, *end)).collect(); + doc.saw(&ranges); + doc.log = LogVersion::anchored(0, version); + doc.joined = true; doc.dirty = true; - let push = doc.pending.get(i).is_some_and(|p| p.push); - if let Some(bytes) = doc.pending.get(i).map(|p| p.bytes.clone()) { - doc.on_server(&bytes); - } - if seq <= doc.have.upto { - doc.pending.remove(i); - doc.sent.remove(&client_seq); - } else if seq == doc.have.upto.saturating_add(1) { - let own = doc.pending.remove(i); - doc.sent.remove(&client_seq); - doc.advance(seq, update_hash(&own.bytes)); - } else if let Some(pending) = doc.pending.get_mut(i) { - // Others' updates below it are on their way: counted once they are here. (A push part counts as - // sent: the next one is computed without it.) - pending.acked = Some(seq); - pending.push = false; - } - Ok(if push { doc.push_next(id, &mut self.notes, &mut self.peers) } else { Vec::new() }) + self.events.push(DocEvent::Joined(id)); + // Followed from the start: the server sends this import back, then what others add. + doc.resubscribe(id) + } + Err(_) => { + // (Checked before it went up; Loro reads it differently now.) Caught up from the server instead. + doc.join = Some(Join::Following); + doc.resubscribe(id) } - ServerFrame::Error { code, message, doc, client_seq } => self.refused(&code, &message, doc, client_seq), } } @@ -1034,6 +2628,11 @@ impl Replica { return Err(LinkError::Mismatch(message.to_owned())); } let epoch = self.epoch; + if let (Some(id), None) = (doc, client_seq) + && let Some(frames) = self.refused_document(id, code, message) + { + return Ok(frames); + } let found = doc.and_then(|id| self.docs.get_mut(&id).map(|d| (id, d))); match (code, found, client_seq) { // Sent again by `resend`, a moment later. @@ -1048,7 +2647,10 @@ impl Replica { } doc.strikes = doc.strikes.saturating_add(1); if doc.strikes > MAX_STRIKES { - let line = format!("the server refuses even small parts of the recents of {} ({message}); they stay on this machine", doc.app); + let line = doc.told( + |app| format!("the server refuses even small parts of the recents of {app} ({message}); they stay on this machine"), + |id| format!("the server refuses even small parts of the live document {id} ({message}); it stays on this machine"), + ); doc.say(&mut self.notes, line); return Ok(Vec::new()); } @@ -1067,7 +2669,10 @@ impl Replica { doc.full_pause = (doc.full_pause * 2).clamp(1, MAX_RETRY_TICKS); doc.full = Some(doc.full_pause); } - let line = format!("the server takes no more recents of {} for now ({message}); new ones stay on this machine", doc.app); + let line = doc.told( + |app| format!("the server takes no more recents of {app} for now ({message}); new ones stay on this machine"), + |id| format!("the live document {id} is full on the server for now ({message}); new changes stay on this machine"), + ); doc.say(&mut self.notes, line); Ok(Vec::new()) } @@ -1084,9 +2689,15 @@ impl Replica { doc.sub = Sub::No; doc.retry = Some((pause, pause)); doc.create_refused = true; - let line = format!( - "the account has as many live documents as the server allows; the recents of {} stay on this machine ({message})", - doc.app + let line = doc.told( + |app| { + format!("the account has as many live documents as the server allows; the recents of {app} stay on this machine ({message})") + }, + |id| { + format!( + "the account has as many live documents as the server allows; the live document {id} stays on this machine ({message})" + ) + }, ); doc.say(&mut self.notes, line); Ok(Vec::new()) @@ -1099,21 +2710,49 @@ impl Replica { doc.retry = Some((pause, pause)); // After a refused create the subscribe comes back `not_found`: that was said already. if !doc.create_refused { - let line = format!("the server can't follow the recents of {} right now ({message}); trying again shortly", doc.app); + let line = doc.told( + |app| format!("the server can't follow the recents of {app} right now ({message}); trying again shortly"), + |id| format!("the server can't follow the live document {id} right now ({message}); trying again shortly"), + ); doc.say(&mut self.notes, line); } Ok(Vec::new()) } + // A catch-up that failed on the server (its snapshot doesn't read there): tried again after a pause, with the + // whole log. + (codes::INTERNAL, Some((_, doc)), None) if doc.kind.is_document() || doc.log_only => { + let pause = doc.retry.map_or(1, |(_, pause)| pause.max(1)); + doc.sub = Sub::No; + doc.retry = Some((pause, pause)); + doc.log_only = true; + Ok(Vec::new()) + } _ => Err(LinkError::Retry(format!("the server refused a live frame ({code}: {message})"))), } } /// Writes every document changed since it was last written. Called after a batch of frames: a crash before it only /// loses what the server sends again. + /// A document of a file is written here only when it must be (it was just joined, started over, or its journal is + /// long); otherwise now and then ([`Replica::flush_documents`]): its local updates are on disk in its journal + /// already, and the server's come again. pub fn flush(&mut self) -> std::io::Result<()> { let space = self.space; for (id, doc) in &mut self.docs { - if doc.dirty { + let now = !doc.kind.is_document() || doc.urgent || doc.journal > MAX_JOURNAL_BYTES; + if doc.dirty && doc.joined && now { + save(&mut self.store, space, *id, doc)?; + } + } + Ok(()) + } + + /// Writes every document changed since it was last written, documents of files too (now and then, and when the + /// agent stops). + pub fn flush_documents(&mut self) -> std::io::Result<()> { + let space = self.space; + for (id, doc) in &mut self.docs { + if doc.dirty && doc.joined { save(&mut self.store, space, *id, doc)?; } } @@ -1135,22 +2774,45 @@ impl Replica { /// The document a file holds (`None` for a file of another space, or one that doesn't read). A file that doesn't read /// is never deleted: it is set aside (renamed, kept). If its unacked updates can still be read (Loro fails on the /// snapshot), the document starts afresh from them and catches up from the start; they are kept and sent. -fn load_doc(store: &mut S, space: SpaceId, name: &str, bytes: &[u8], peer: u64, notes: &mut Vec) -> Option<(DocId, Doc)> { +/// The document of a file `bytes` hold, if it can stay on disk unloaded: of this space, and with nothing unacked (in +/// the file or its journal), nothing waiting and no push under way. A file that doesn't read is loaded (and so set +/// aside) as before. +fn idle_on_disk(store: &S, space: SpaceId, name: &str, bytes: &[u8]) -> Option { + let file = postcard::from_bytes::(bytes).ok()?; + let ours = file.format <= DOCUMENT_FILE_FORMAT && file.space == space && file.doc == node_doc(space, file.node) && file.doc.to_string() == name; + let idle = file.pending.is_empty() && file.waiting.is_empty() && file.pushing.is_none(); + let journal = store.load_journal(name).ok()?; + (ours && idle && journal_records(&journal).0.is_empty()).then_some(file.doc) +} + +/// `drop`: a waiting update to leave out (Loro failed on it). +fn load_doc( + store: &mut S, + space: SpaceId, + name: &str, + bytes: &[u8], + peer: u64, + drop: Option<[u8; 32]>, + notes: &mut Vec, +) -> Option<(DocId, Doc)> { let aside = |store: &mut S| match store.set_aside(name) { Ok(to) => (format!("a copy set aside as {to}"), true), Err(e) => (format!("no copy set aside ({e})"), false), }; - match postcard::from_bytes::(bytes) { - Ok(file) if file.format <= DOC_FILE_FORMAT && file.space == space && file.doc == recents_doc(space, &file.app) => { + match read_doc_file(bytes) { + Ok((mut file, more)) if file.format <= DOC_FILE_FORMAT && file.space == space && file.doc == recents_doc(space, &file.app) => { + if let Some(hash) = drop { + file.waiting.retain(|w| w.hash != hash); + } let id = file.doc; let (app, pending, next) = (file.app.clone(), file.pending.clone(), file.next_client_seq); - match Doc::from_file(file, peer, notes) { + match Doc::from_file(file, more, peer, notes) { Ok(doc) => Some((id, doc)), Err(e) => { // A copy aside first (durable), then the rebuilt document over the original: a crash between them // finds the original again, and rebuilds again. let (set, copied) = aside(store); - let mut doc = Doc::from_pending(app.clone(), pending, next, peer); + let mut doc = Doc::from_pending(id, Kind::Recents(app.clone()), pending, next, peer); // (Without a copy, the original stays as it is until the document is next written.) let written = match copied.then(|| save(store, space, id, &mut doc)) { Some(Ok(())) => "written afresh".to_owned(), @@ -1182,6 +2844,100 @@ fn load_doc(store: &mut S, space: SpaceId, name: &str, bytes: &[u8] } } +/// The document of a file a `.doc` file holds, with the local updates its journal has beyond it (`None` for a file of +/// another space, or one that doesn't read at all, which is set aside and removed, as a recents file is). A file whose +/// snapshot doesn't read starts afresh from its unsent updates, which are kept and sent, and catches up from the start. +fn load_document( + store: &mut S, + space: SpaceId, + name: &str, + bytes: &[u8], + peer: u64, + drop: Option<[u8; 32]>, + notes: &mut Vec, +) -> Option<(DocId, Doc)> { + let journal = store.load_journal(name).unwrap_or_default(); + let aside = |store: &mut S| match store.set_aside_document(name) { + Ok(to) => (format!("a copy set aside as {to}"), true), + Err(e) => (format!("no copy set aside ({e})"), false), + }; + match postcard::from_bytes::(bytes) { + Ok(mut file) if file.format <= DOCUMENT_FILE_FORMAT && file.space == space && file.doc == node_doc(space, file.node) => { + if let Some(hash) = drop { + file.waiting.retain(|w| w.hash != hash); + } + let id = file.doc; + let kind = Kind::Document { node: file.node, app: file.app.clone() }; + let (mut pending, mut next) = (file.pending, file.next_client_seq.max(1)); + // The journal's records the file doesn't have yet (one cut off by a crash ends it: it was never acked). The + // journal is cut there, so what is appended next reads after the good ones. + let mut journalled = 0usize; + let (records, good) = journal_records(&journal); + if good < journal.len() + && let Err(e) = store.truncate_journal(name, good) + { + notes.push(format!("the journal of the live document {name} couldn't be cut at its end that doesn't read ({e})")); + } + for record in records { + if record.client_seq >= next { + next = record.client_seq.saturating_add(1); + pending.push(Pending { client_seq: record.client_seq, bytes: record.bytes, acked: None, push: false }); + journalled += 1; + } + } + let parts = Parts { + id, + kind: kind.clone(), + have: file.have, + next_client_seq: next, + pending: pending.clone(), + snapshot: file.snapshot, + waiting: file.waiting, + epoch: file.epoch, + pushing: file.pushing, + own_peers: file.own_peers, + push_wait: file.push_wait, + dropped: file.dropped, + }; + match Doc::from_parts(parts, peer, notes) { + Ok(mut doc) => { + doc.log_only = file.log_only; + doc.journal = good; + doc.dirty = journalled > 0; + // (Not cut: written whole before the next change goes into it.) + doc.journal_failed = good < journal.len() && store.load_journal(name).map_or(true, |j| j.len() > good); + Some((id, doc)) + } + Err(e) => { + let (set, copied) = aside(store); + let mut doc = Doc::from_pending(id, kind, pending, next, peer); + let written = match copied.then(|| save(store, space, id, &mut doc)) { + Some(Ok(())) => "written afresh".to_owned(), + Some(Err(e)) => format!("not yet written afresh ({e})"), + None => "kept in memory".to_owned(), + }; + notes.push(format!( + "the live document file {name} doesn't read ({e}); {set}; the document starts again from its {} unsent update(s), which are kept ({written}), and the rest comes from the server", + doc.pending.len() + )); + Some((id, doc)) + } + } + } + Ok(_) => None, + Err(e) => { + let (set, copied) = aside(store); + let removed = match copied.then(|| store.remove_document(name)) { + Some(Ok(())) => "and removed".to_owned(), + Some(Err(e)) => format!("not removed ({e})"), + None => "left in place".to_owned(), + }; + notes.push(format!("the live document file {name} doesn't read ({e}); {set}, {removed}; it is filled from the server again")); + None + } + } +} + /// Whether every range of `ranges` is in `has`. fn held_by(has: &Spans, ranges: &[(u64, i32, i32)]) -> bool { ranges.iter().all(|(peer, s, e)| s >= e || has.get(peer).is_some_and(|list| list.iter().any(|(hs, he)| hs <= s && e <= he))) @@ -1190,14 +2946,27 @@ fn held_by(has: &Spans, ranges: &[(u64, i32, i32)]) -> bool { /// The document of `app` in `docs`, made (empty) if there is none yet. fn doc_in<'a>(docs: &'a mut BTreeMap, space: SpaceId, app: &AppName, peers: &mut Peers) -> (DocId, &'a mut Doc) { let id = recents_doc(space, app); - let doc = docs.entry(id).or_insert_with(|| Doc::empty(app, peers())); + let doc = docs.entry(id).or_insert_with(|| Doc::empty(app, peers(), id)); (id, doc) } +/// The document `id` of `kind` after a reload: a recents document is made afresh if it had no file; a document of a +/// file without one is gone. +fn doc_again<'a>(docs: &'a mut BTreeMap, space: SpaceId, peers: &mut Peers, id: DocId, kind: &Kind) -> Option<&'a mut Doc> { + match kind { + Kind::Recents(app) => Some(doc_in(docs, space, app, peers).1), + Kind::Document { .. } => docs.get_mut(&id), + } +} + /// `create` (the server may lack the document) and `subscribe` from what it has, naming the update it has at that /// number. A document that knew another epoch (the server was restored since) catches up from the start and goes up /// whole after the catch-up. Acks of numbers not reached are forgotten: their updates go again after `subscribed`. fn subscribe(id: DocId, doc: &mut Doc, epoch: Option) -> Vec { + if doc.stopped { + doc.sub = Sub::No; + return Vec::new(); + } if let Some(epoch) = epoch && doc.epoch != Some(epoch) { @@ -1209,7 +2978,14 @@ fn subscribe(id: DocId, doc: &mut Doc, epoch: Option) -> Vec } doc.forget_acks(); doc.sent.clear(); - let mut frames = vec![ClientFrame::Create { doc: id, kind: DocKind::Recents { app: doc.app.clone() } }]; + doc.taken_back.clear(); + doc.back_acked.clear(); + // A document of a file is made by import, never by a bare `create`: one the server lost answers `not_found`, and + // is made again from this copy ([`Replica::refused_document`]). + let mut frames = match &doc.kind { + Kind::Recents(app) => vec![ClientFrame::Create { doc: id, kind: DocKind::Recents { app: app.clone() } }], + Kind::Document { .. } => Vec::new(), + }; frames.extend(doc.resubscribe(id)); frames } @@ -1230,24 +3006,61 @@ fn send_pending(id: DocId, doc: &mut Doc) -> Vec { } fn save(store: &mut S, space: SpaceId, id: DocId, doc: &mut Doc) -> std::io::Result<()> { - let file = DocFile { - format: DOC_FILE_FORMAT, - space, - doc: id, - app: doc.app.clone(), - have: doc.have.clone(), - next_client_seq: doc.next_client_seq, - pending: doc.pending.clone(), - snapshot: doc.recents.snapshot().map_err(std::io::Error::other)?, - waiting: doc.waiting.clone(), - epoch: doc.epoch, - pushing: doc.pushing.clone(), - own_peers: doc.own_peers.clone(), - push_wait: doc.push_wait, - }; - let bytes = postcard::to_stdvec(&file).map_err(std::io::Error::other)?; - store.save(&id.to_string(), &bytes)?; + let snapshot = doc.body.copy().snapshot().map_err(std::io::Error::other)?; + match &doc.kind { + Kind::Recents(app) => { + let file = DocFile { + format: DOC_FILE_FORMAT, + space, + doc: id, + app: app.clone(), + have: doc.have.clone(), + next_client_seq: doc.next_client_seq, + pending: doc.pending.clone(), + snapshot, + waiting: doc.waiting.clone(), + epoch: doc.epoch, + pushing: doc.pushing.clone(), + own_peers: doc.own_peers.clone(), + push_wait: doc.push_wait, + }; + let mut bytes = postcard::to_stdvec(&file).map_err(std::io::Error::other)?; + // (Nothing more to say: the file is as an older agent writes it.) + if doc.dropped.is_some() { + bytes.extend(postcard::to_stdvec(&DocFileMore { dropped: doc.dropped }).map_err(std::io::Error::other)?); + } + store.save(&id.to_string(), &bytes)?; + } + Kind::Document { node, app } => { + let file = DocumentFile { + format: DOCUMENT_FILE_FORMAT, + space, + doc: id, + node: *node, + app: app.clone(), + have: doc.have.clone(), + next_client_seq: doc.next_client_seq, + pending: doc.pending.clone(), + snapshot, + waiting: doc.waiting.clone(), + epoch: doc.epoch, + pushing: doc.pushing.clone(), + own_peers: doc.own_peers.clone(), + push_wait: doc.push_wait, + log_only: doc.log_only, + dropped: doc.dropped, + }; + let bytes = postcard::to_stdvec(&file).map_err(std::io::Error::other)?; + let name = id.to_string(); + store.save_document(&name, &bytes)?; + // Its journal's updates are in the file now (a crash before this finds them there and skips them). + store.clear_journal(&name)?; + doc.journal = 0; + } + } doc.dirty = false; + doc.urgent = false; + doc.on_disk = true; Ok(()) } @@ -1592,6 +3405,28 @@ mod tests { assert_eq!(r.entries(&app()).len(), 3); } + /// An own update that a row brought back before its ack (sent once on an old connection, again on this one): the + /// ack, numbering a second row with the same bytes that this machine is never sent, still counts once the rows + /// below it came, so the log goes on without a gap and nothing waits. + #[test] + fn an_ack_for_an_update_a_row_brought_back_still_counts() { + let space = SpaceId::generate(); + let doc = recents_doc(space, &app()); + let mut r = live(space, MemStore::default()); + let frames = r.add(&app(), [3; 16], entry("mine.txt", 3)).unwrap(); + let [ClientFrame::Update { bytes: mine, .. }] = frames.as_slice() else { panic!("{frames:?}") }; + r.on_frame(ServerFrame::Update { doc, seq: 1, bytes: mine.clone(), device: device() }).unwrap(); + assert_eq!(r.waiting(), 0, "the log has it"); + r.on_frame(ServerFrame::Ack { doc, client_seq: 1, seq: 3 }).unwrap(); + let other = Recents::new(77); + let u2 = other.add([5; 16], &entry("second.txt", 1)).unwrap(); + let u4 = other.add([6; 16], &entry("fourth.txt", 4)).unwrap(); + r.on_frame(ServerFrame::Update { doc, seq: 2, bytes: u2, device: device() }).unwrap(); + assert!(r.on_frame(ServerFrame::Update { doc, seq: 4, bytes: u4, device: device() }).unwrap().is_empty(), "no gap at 3"); + assert_eq!(r.entries(&app()).len(), 3); + assert_eq!(r.waiting(), 0); + } + /// A gap on a caught-up document (frames lost on an old connection): a catch-up from what it has. #[test] fn a_gap_is_caught_up() { @@ -1679,6 +3514,382 @@ mod tests { assert_eq!(r.take_notes().len(), 1); } + /// A replica on `store` with a document of a file joined from an import ("import", peer 40), its file written. + fn joined(store: &MemStore, space: SpaceId, node: NodeId) -> (Replica, DocId, Vec) { + let importer = text_doc(40); + insert(&importer, 0, "import"); + let import = importer.export(loro::ExportMode::Snapshot).unwrap(); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + let (id, _, _) = r.open_document(node, &app()); + snapshot_in(&mut r, id, 0, &import); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); + r.flush_documents().unwrap(); + (r, id, import) + } + + /// The text of document `id` as `r` has it. + fn text_of(r: &Replica, id: DocId) -> String { + let d = loro::LoroDoc::new(); + d.import(&r.snapshot_of(id).unwrap()).unwrap(); + d.get_text("text").to_string() + } + + /// The snapshot a catch-up starts with is downloaded through the chunk API: the catch-up's updates and its + /// `subscribed` wait for it. One that can't be got makes this machine follow the whole log instead. + #[test] + fn a_catch_up_waits_for_its_snapshot_and_reads_the_whole_log_without_it() { + for got in [true, false] { + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let mut r = Replica::open(MemStore::default(), space, peers()).unwrap(); + r.connected(); + r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + let (id, _, _) = r.open_document(node, &app()); + let server = text_doc(40); + insert(&server, 0, "import"); + insert(&server, 0, "<1>"); + let r2 = insert(&server, 0, "<2>"); + let snapshot = server.export(loro::ExportMode::Snapshot).unwrap(); + let r3 = insert(&server, 0, "<3>"); + let manifest = cut(id, &snapshot).unwrap().id(); + assert!(r.on_frame(ServerFrame::Snapshot { doc: id, base: 2, manifest }).unwrap().is_empty()); + assert_eq!(r.take_fetches(), vec![(id, manifest)]); + for (seq, bytes) in [(2, r2.clone()), (3, r3.clone())] { + assert!(r.on_frame(ServerFrame::Update { doc: id, seq, bytes, device: device() }).unwrap().is_empty()); + } + assert!(r.on_frame(ServerFrame::Subscribed { doc: id, head: 3 }).unwrap().is_empty(), "waits for the snapshot"); + assert!(!r.has_document(id)); + if got { + r.on_snapshot(id, manifest, Ok(snapshot.clone())); + assert!(r.take_events().contains(&DocEvent::Joined(id))); + assert_eq!(text_of(&r, id), "<3><2><1>import"); + } else { + let frames = r.on_snapshot(id, manifest, Err("gone".into())); + assert!(frames.iter().any(|f| matches!(f, ClientFrame::SubscribeLog { after: 0, .. })), "{frames:?}"); + assert!(r.take_notes().iter().any(|n| n.contains("can't be got"))); + } + } + } + + /// A journal append that fails part-way, or whose flush fails, is refused to the app and leaves nothing a crash + /// would bring back or that would hide the changes acked after it. + #[test] + fn a_failed_journal_append_hides_no_later_change() { + use crate::live::store::JournalFailure; + for how in [JournalFailure::Torn, JournalFailure::Unflushed] { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let edit = |peer: u64, s: &str| { + let d = text_doc(peer); + d.import(&import).unwrap(); + insert(&d, 0, s) + }; + store.fail_journal_as(Some(how)); + assert!(r.local_update(id, &edit(50, "")).is_err(), "{how:?}"); + store.fail_journal_as(None); + r.local_update(id, &edit(51, "")).unwrap(); + // A crash before the document is next written. + drop(r); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.open_document(node, &app()); + let text = text_of(&r, id); + assert!(text.contains(""), "{how:?}: an acked change is lost: {text:?}"); + assert!(!text.contains(""), "{how:?}: a refused change came back: {text:?}"); + } + } + + /// At start only documents with something unacked or waiting are loaded; the rest stay on disk until an app opens + /// one. Past [`MAX_LOADED_DOCUMENTS`] loaded, the least recently used one with nothing unacked or waiting and no app + /// holding it is unloaded (written first, so nothing is lost). + #[test] + fn documents_load_when_opened_and_unload_past_the_cap() { + let store = MemStore::default(); + let space = SpaceId::generate(); + let importer = text_doc(40); + insert(&importer, 0, "import"); + let import = importer.export(loro::ExportMode::Snapshot).unwrap(); + let nodes: Vec = (0..300).map(|_| NodeId::generate()).collect(); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + let mut ids = Vec::new(); + for node in &nodes { + let (id, _, _) = r.open_document(*node, &app()); + snapshot_in(&mut r, id, 0, &import); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); + ids.push(id); + } + assert!(r.loaded_documents() <= MAX_LOADED_DOCUMENTS, "{} loaded", r.loaded_documents()); + // One with a change not yet acked. + let (_, opening, _) = r.open_document(nodes[0], &app()); + assert_eq!(opening, Opening::Live); + let d = text_doc(50); + d.import(&import).unwrap(); + r.local_update(ids[0], &insert(&d, 0, "")).unwrap(); + r.flush_documents().unwrap(); + drop(r); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + assert_eq!(r.loaded_documents(), 1, "only the one with something to send"); + r.connected(); + r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + let (_, opening, frames) = r.open_document(nodes[7], &app()); + assert_eq!(opening, Opening::Live, "loaded when opened, from disk"); + assert!(frames.iter().any(|f| matches!(f, ClientFrame::Subscribe { doc, .. } if *doc == ids[7])), "{frames:?}"); + assert_eq!(r.loaded_documents(), 2); + assert!(text_of(&r, ids[7]).contains("import")); + for node in &nodes { + assert_eq!(r.open_document(*node, &app()).1, Opening::Live); + } + assert!(r.loaded_documents() <= MAX_LOADED_DOCUMENTS, "{} loaded", r.loaded_documents()); + assert!(text_of(&r, ids[0]).contains(""), "the one with a change waiting stays"); + } + + /// A machine with no copy of a document the server has but holds nothing of (an empty log, no snapshot: an admin + /// reset it, and its machines haven't sent it back yet) doesn't serve it to an app as live: the file is what to + /// open, and the join is let go of. + #[test] + fn a_reset_document_isnt_served_empty_to_a_machine_without_a_copy() { + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let mut r = Replica::open(MemStore::default(), space, peers()).unwrap(); + r.connected(); + r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + let (id, opening, _) = r.open_document(node, &app()); + assert_eq!(opening, Opening::Joining); + let frames = r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); + let events = r.take_events(); + assert!(events.iter().any(|e| matches!(e, DocEvent::NotLive(d, NotLiveReason::Resetting, _) if *d == id)), "{events:?}"); + assert!(!r.has_document(id), "not live here"); + assert!(frames.contains(&ClientFrame::Unsubscribe { doc: id }), "{frames:?}"); + // Once a machine sent it back (rows, or an import's snapshot), it joins. + let (_, opening, _) = r.open_document(node, &app()); + assert_eq!(opening, Opening::Joining); + let other = text_doc(60); + let row = insert(&other, 0, "back"); + r.on_frame(ServerFrame::Update { doc: id, seq: 1, bytes: row, device: device() }).unwrap(); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 1 }).unwrap(); + assert!(r.take_events().contains(&DocEvent::Joined(id))); + } + + /// After a restore (a new epoch) a document goes up whole: what this machine's apps wrote is its own, so it goes + /// at once (no random wait), and a change an app makes meanwhile goes at once too, not behind the push. + #[test] + fn an_apps_changes_are_its_machines_own_in_a_push() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + // This machine's app imported the file (peer 40) and then edited it (peer 50). + let app_doc = text_doc(40); + insert(&app_doc, 0, "import"); + let import = app_doc.export(loro::ExportMode::Snapshot).unwrap(); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + let (id, _, _) = r.open_document(node, &app()); + r.on_frame(ServerFrame::error("not_found", "no such document", Some(id), None)).unwrap(); + assert!(r.take_events().contains(&DocEvent::NeedsImport(id))); + let frames = r.import_document(id, &import).unwrap(); + assert!(matches!(frames.as_slice(), [ClientFrame::CreateDocument { .. }]), "{frames:?}"); + r.on_frame(ServerFrame::CreatedDocument { doc: id, kept: true }).unwrap(); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); + let d = text_doc(50); + d.import(&import).unwrap(); + r.local_update(id, &insert(&d, 0, "")).unwrap(); + r.on_frame(ServerFrame::Ack { doc: id, client_seq: 1, seq: 1 }).unwrap(); + // The server is restored from a backup that has none of it. + r.disconnected(); + r.connected(); + let frames = r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(2) }).unwrap(); + assert!(frames.iter().any(|f| matches!(f, ClientFrame::Subscribe { after: 0, .. })), "{frames:?}"); + let frames = r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); + assert!(frames.iter().any(|f| matches!(f, ClientFrame::Update { .. })), "its own goes up at once: {frames:?}"); + // A change its app makes meanwhile goes at once too, not behind the push. + let typed = r.local_update(id, &insert(&d, 0, "")).unwrap(); + assert!(matches!(typed.as_slice(), [ClientFrame::Update { .. }]), "{typed:?}"); + } + + /// A journal is written into its document's file at the next write once it holds over [`MAX_JOURNAL_BYTES`], not + /// before (the file is written now and then besides). + #[test] + fn a_journal_past_its_cap_is_written_into_the_file() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let d = text_doc(50); + d.import(&import).unwrap(); + let name = id.to_string(); + let big = "x".repeat(900_000); + for i in 0..4 { + r.local_update(id, &put(&d, i, &big)).unwrap(); + } + let held = store.journals().get(&name).map_or(0, Vec::len); + assert!(held < MAX_JOURNAL_BYTES, "{held}"); + r.flush().unwrap(); + assert_eq!(store.journals().get(&name).map_or(0, Vec::len), held, "under the cap: left for the next write"); + r.local_update(id, &put(&d, 4, &big)).unwrap(); + assert!(store.journals().get(&name).map_or(0, Vec::len) > MAX_JOURNAL_BYTES); + r.flush().unwrap(); + assert_eq!(store.journals().get(&name).map_or(0, Vec::len), 0, "over it: written into the file"); + } + + /// A catch-up's updates are taken a batch at a time: once more than [`MAX_BATCH_BYTES`] wait, they are taken + /// before `subscribed`. + #[test] + fn a_catch_up_batch_is_taken_past_its_cap() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + r.disconnected(); + r.connected(); + r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }).unwrap(); + let other = text_doc(60); + other.import(&import).unwrap(); + let big = "y".repeat(1_000_000); + let mut batched = 0; + let mut seq = 0; + while batched <= MAX_BATCH_BYTES { + let row = put(&other, seq, &big); + batched += row.len(); + seq += 1; + assert!(r.describe_doc(id).contains("upto 0"), "not taken at {batched} bytes: {}", r.describe_doc(id)); + r.on_frame(ServerFrame::Update { doc: id, seq, bytes: row, device: device() }).unwrap(); + } + assert!(r.describe_doc(id).contains(&format!("upto {seq}")), "taken past the cap: {}", r.describe_doc(id)); + } + + /// A document read back from its file after a refused change (its journal failed) is followed again at the next + /// tick, not left unfollowed while the link is up. + #[test] + fn a_document_read_back_after_a_refused_change_is_followed_again() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let d = text_doc(50); + d.import(&import).unwrap(); + store.fail_journal(true); + assert!(r.local_update(id, &insert(&d, 0, "x")).is_err()); + store.fail_journal(false); + let frames = r.resend(); + assert!(frames.iter().any(|f| matches!(f, ClientFrame::Subscribe { doc, .. } if *doc == id)), "{frames:?}"); + } + + /// A journal whose end doesn't read (cut off by a crash) is read up to there and cut there, so what is appended + /// after it reads. + #[test] + fn a_journal_is_cut_at_its_first_bad_record() { + let store = MemStore::default(); + let (space, node) = (SpaceId::generate(), NodeId::generate()); + let (mut r, id, import) = joined(&store, space, node); + let d = text_doc(50); + d.import(&import).unwrap(); + r.local_update(id, &insert(&d, 0, "")).unwrap(); + drop(r); + let name = id.to_string(); + let good = store.journals().get(&name).cloned().unwrap(); + let mut s = store.clone(); + s.append_journal(&name, &[9, 0, 0, 0, 1, 2]).unwrap(); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.open_document(node, &app()); + assert_eq!(store.journals().get(&name), Some(&good), "cut at the bad record"); + r.local_update(id, &insert(&d, 0, "")).unwrap(); + drop(r); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.open_document(node, &app()); + let text = text_of(&r, id); + assert!(text.contains("") && text.contains(""), "{text:?}"); + } + + /// Serves `rows` (numbered from 1) to every `subscribe` in `frames` until none is asked for; the rounds it took. + fn serve_catch_ups(r: &mut Replica, doc: DocId, rows: &[Vec], mut frames: Vec) -> usize { + let mut rounds = 0; + while let Some(after) = frames.iter().find_map(|f| match f { + ClientFrame::Subscribe { after, .. } => Some(*after), + _ => None, + }) { + rounds += 1; + if rounds > 10 { + break; + } + frames = Vec::new(); + for (i, row) in rows.iter().enumerate() { + let seq = i as u64 + 1; + if seq > after { + frames.extend(r.on_frame(ServerFrame::Update { doc, seq, bytes: row.clone(), device: device() }).unwrap()); + } + } + frames.extend(r.on_frame(ServerFrame::Subscribed { doc, head: rows.len() as u64 }).unwrap()); + } + rounds + } + + /// Loro fails on one row of a catch-up taken as a batch: the catch-up is taken again one row at a time, only that + /// row is dropped (and said), and the document ends live with every other row, in a few rounds. + #[test] + fn a_row_loro_fails_on_in_a_batch_is_found_and_only_it_dropped() { + let space = SpaceId::generate(); + let node = NodeId::generate(); + let w2 = ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }; + let importer = text_doc(40); + insert(&importer, 0, "import"); + let import = importer.export(loro::ExportMode::Snapshot).unwrap(); + let mut r = Replica::open(MemStore::default(), space, peers()).unwrap(); + r.connected(); + r.on_frame(w2.clone()).unwrap(); + let (id, _, _) = r.open_document(node, &app()); + snapshot_in(&mut r, id, 0, &import); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); + r.flush_documents().unwrap(); + // (Each row from a machine of its own: none needs another's operations.) + let rows: Vec> = (1..=5u64) + .map(|i| { + let d = text_doc(70 + i); + d.import(&import).unwrap(); + insert(&d, 0, &format!("<{i}>")) + }) + .collect(); + crate::live::copy::fail_on(&rows[2]); + r.disconnected(); + r.connected(); + let frames = r.on_frame(w2).unwrap(); + let rounds = serve_catch_ups(&mut r, id, &rows, frames); + assert!(rounds <= 4, "it goes round: {rounds} rounds; {}", r.describe_doc(id)); + let text = { + let d = loro::LoroDoc::new(); + d.import(&r.snapshot_of(id).unwrap()).unwrap(); + d.get_text("text").to_string() + }; + for i in [1, 2, 4, 5] { + assert!(text.contains(&format!("<{i}>")), "row {i} is in: {text:?}"); + } + assert!(!text.contains("<3>"), "{text:?}"); + assert!(r.take_notes().iter().any(|n| n.contains("made Loro fail")), "said"); + assert_eq!(r.document_status(id), Some((true, 0)), "live: {}", r.describe_doc(id)); + } + + /// Loro fails on a second row in one run: the document stops catching up (no loop), and its apps are told. + #[test] + fn a_second_loro_failure_in_a_run_stops_the_document() { + let space = SpaceId::generate(); + let doc = recents_doc(space, &app()); + let mut r = live(space, MemStore::default()); + r.flush().unwrap(); + let other = Recents::new(77); + let rows: Vec> = (1..=5u8).map(|i| other.add([i; 16], &entry(&format!("{i}.txt"), i64::from(i))).unwrap()).collect(); + crate::live::copy::fail_on(&rows[1]); + crate::live::copy::fail_on(&rows[3]); + r.disconnected(); + r.connected(); + let frames = r.on_frame(welcome(1)).unwrap(); + let rounds = serve_catch_ups(&mut r, doc, &rows, frames); + assert!(rounds <= 5, "it goes round: {rounds} rounds; {}", r.describe(&app())); + assert!(r.is_stopped(doc), "{}", r.describe(&app())); + assert!(r.take_notes().iter().any(|n| n.contains("stops following")), "said"); + assert!(r.resend().is_empty(), "nothing tried again in this run"); + r.disconnected(); + r.connected(); + assert!(!r.on_frame(welcome(1)).unwrap().iter().any(|f| matches!(f, ClientFrame::Subscribe { .. })), "nor on the next connection"); + } + /// Waiting updates are capped by bytes: past the cap the document says so and takes nothing more on this connection. #[test] fn waiting_updates_are_capped_by_bytes() { @@ -1979,7 +4190,7 @@ mod tests { r.add(&app(), [1; 16], entry("a.txt", 1)).unwrap(); r.on_frame(ServerFrame::Ack { doc, client_seq: 1, seq: 1 }).unwrap(); r.flush().unwrap(); - r.docs.get(&doc).unwrap().recents.poison(); + r.docs.get(&doc).unwrap().body.poison(); assert!(r.entries(&app()).is_empty(), "a poisoned document reads as empty, it doesn't panic"); assert!(r.wants_tick()); let frames = r.resend(); @@ -2000,7 +4211,7 @@ mod tests { r.flush().unwrap(); for round in 0..2 { assert!(r.docs.get(&doc).unwrap().pushing.is_some(), "round {round}: a push is under way"); - r.docs.get(&doc).unwrap().recents.poison(); + r.docs.get(&doc).unwrap().body.poison(); r.resend(); r.on_frame(ServerFrame::Subscribed { doc, head: 0 }).unwrap(); r.flush().unwrap(); @@ -2035,4 +4246,273 @@ mod tests { assert_eq!(r.entries(&app()).len(), 1, "the next good one applies"); assert!(matches!(r.on_frame(ServerFrame::error(PROTOCOL_MISMATCH, "old", None, None)), Err(LinkError::Mismatch(_)))); } + + /// A journal reads up to its first record that doesn't (one a crash cut off: never acked to the app). + #[test] + fn a_journal_ends_at_a_record_cut_off() { + let records: Vec> = (1..=3u64).map(|i| journal_record(&JournalRecord { client_seq: i, bytes: vec![i as u8; 10] }).unwrap()).collect(); + let whole: Vec = records.concat(); + let (read, good) = journal_records(&whole); + assert_eq!(read.iter().map(|r| r.client_seq).collect::>(), vec![1, 2, 3]); + assert_eq!(good, whole.len()); + for cut in [whole.len() - 1, whole.len() - 9, records[0].len() + 3] { + let (read, good) = journal_records(&whole[..cut]); + assert!(read.len() < 3 && read.iter().enumerate().all(|(i, r)| r.client_seq == i as u64 + 1), "cut at {cut}"); + assert_eq!(good, records[..read.len()].iter().map(Vec::len).sum::(), "where it ends"); + } + let mut damaged = whole.clone(); + let at = records[0].len() + 6; + damaged[at] ^= 1; + assert_eq!(journal_records(&damaged), (journal_records(&records[0]).0, records[0].len()), "a damaged record ends it"); + assert!(journal_records(&[]).0.is_empty()); + assert!(journal_records(&[0xff, 0xff, 0xff, 0xff, 1]).0.is_empty(), "a length past the end"); + } + + /// The version a log's updates make is worked out from the copy at load: everything but its own unsent updates + /// (when those are each peer's last operations); unknown otherwise. + #[test] + fn the_log_version_is_worked_out_from_the_copy() { + let space = SpaceId::generate(); + let node = NodeId::generate(); + let id = node_doc(space, node); + let kind = Kind::Document { node, app: app() }; + let mut doc = + Doc::with(id, kind.clone(), Body::new(&kind, 1), Have { upto: 3, upto_hash: Some([3; 32]) }, Vec::new(), Vec::new(), 1, None, None); + let other = loro::LoroDoc::new(); + other.set_peer_id(9).unwrap(); + let update = |d: &loro::LoroDoc, s: &str| { + let vv = d.oplog_vv(); + d.get_text("text").insert(0, s).unwrap(); + d.commit(); + d.export(loro::ExportMode::updates(&vv)).unwrap() + }; + let logged = update(&other, "abc"); + doc.body.import(&logged).unwrap(); + // Its own, unsent: peer 1's last operations. + let own = loro::LoroDoc::new(); + own.import(&logged).unwrap(); + own.set_peer_id(1).unwrap(); + let mine = update(&own, "xy"); + doc.body.import(&mine).unwrap(); + doc.pending.push(Pending { client_seq: 1, bytes: mine, acked: None, push: false }); + doc.derive_log(); + assert!(!doc.log.is_known(), "an unacked update may or may not be stored: unknown"); + // Once nothing is unacked, the log holds the whole copy (its own update came back as a row, or was acked). + doc.pending.clear(); + doc.derive_log(); + let version = doc.log.at(3).unwrap(); + assert_eq!(version.get(&9), Some(&3)); + assert_eq!(version.get(&1), Some(&2)); + doc.dropped = Some(2); + doc.derive_log(); + assert!(!doc.log.is_known(), "a row was dropped: unknown"); + doc.dropped = None; + doc.waiting.push(Waiting { hash: [0; 32], needs: vec![(5, 1)], bytes: vec![1] }); + doc.derive_log(); + assert!(!doc.log.is_known(), "something waits: unknown"); + } + + /// The server's snapshot `bytes` at the start of a catch-up, as the hub hands it over: the frame naming it, then + /// its bytes, downloaded. Frames to send. + fn snapshot_in(r: &mut Replica, doc: DocId, base: u64, bytes: &[u8]) -> Vec { + let manifest = cut(doc, bytes).unwrap().id(); + let mut frames = r.on_frame(ServerFrame::Snapshot { doc, base, manifest }).unwrap(); + assert_eq!(r.take_fetches(), vec![(doc, manifest)], "asked to download it"); + frames.extend(r.on_snapshot(doc, manifest, Ok(bytes.to_vec()))); + frames + } + + /// The snapshot `manifest` names, from what the replica asked to upload. + fn uploaded(r: &mut Replica, manifest: ManifestId) -> Vec { + r.take_uploads().into_iter().find(|u| u.id() == manifest).expect("uploaded before it is named").bytes() + } + + /// Sets key `k` of the map `map` to `s`: the change. + fn put(d: &loro::LoroDoc, n: u64, s: &str) -> Vec { + let vv = d.oplog_vv(); + d.get_map("map").insert(&format!("k{n}"), s).unwrap(); + d.commit(); + d.export(loro::ExportMode::updates(&vv)).unwrap() + } + + fn text_doc(peer: u64) -> loro::LoroDoc { + let d = loro::LoroDoc::new(); + d.set_peer_id(peer).unwrap(); + d + } + + fn insert(d: &loro::LoroDoc, at: usize, s: &str) -> Vec { + let vv = d.oplog_vv(); + d.get_text("text").insert(at, s).unwrap(); + d.commit(); + d.export(loro::ExportMode::updates(&vv)).unwrap() + } + + /// An own update the server stored (its ack lost), whose operations the server's snapshot holds on the next + /// catch-up: it is stored (not sent again, nothing waits), and a compaction after a restart has it, though another + /// machine's later rows don't build on it. + #[test] + fn an_update_a_snapshot_holds_is_stored_and_in_every_compaction() { + let store = MemStore::default(); + let space = SpaceId::generate(); + let node = NodeId::generate(); + let w2 = ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }; + // The import (peer 40), this machine's app edit P (peer 50), another machine's concurrent edits (peer 60). + let importer = text_doc(40); + insert(&importer, 0, "import"); + let import = importer.export(loro::ExportMode::Snapshot).unwrap(); + let mine = text_doc(50); + mine.import(&import).unwrap(); + let p = insert(&mine, 6, " P"); + let other = text_doc(60); + other.import(&import).unwrap(); + let o2 = insert(&other, 0, "O1 "); + let o3 = insert(&other, 0, "O2 "); + let server = text_doc(70); + for u in [&import, &p, &o2, &o3] { + server.import(u).unwrap(); + } + let s3 = server.export(loro::ExportMode::Snapshot).unwrap(); + // Joined from the import. + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + r.on_frame(w2.clone()).unwrap(); + let (id, opening, _) = r.open_document(node, &app()); + assert_eq!(opening, Opening::Joining); + snapshot_in(&mut r, id, 0, &import); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); + assert!(r.has_document(id)); + // P: sent, stored as row 1 by the server, its ack lost with the connection. + let frames = r.local_update(id, &p).unwrap(); + assert!(matches!(frames.as_slice(), [ClientFrame::Update { client_seq: 1, .. }]), "{frames:?}"); + r.disconnected(); + // Rows 2, 3 from the other machine; compacted to base 3. The next catch-up starts from the snapshot. + r.connected(); + r.on_frame(w2.clone()).unwrap(); + snapshot_in(&mut r, id, 3, &s3); + r.on_frame(ServerFrame::Update { doc: id, seq: 3, bytes: o3.clone(), device: device() }).unwrap(); + let frames = r.on_frame(ServerFrame::Subscribed { doc: id, head: 3 }).unwrap(); + assert!(!frames.iter().any(|f| matches!(f, ClientFrame::Update { .. })), "P is stored, not sent again: {frames:?}"); + assert_eq!(r.document_status(id), Some((true, 0)), "nothing waits"); + // The documents are written (every 2 s, or at stop) and the agent stops. + r.flush_documents().unwrap(); + drop(r); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + let mut frames = r.on_frame(w2.clone()).unwrap(); + // (Nothing to send: followed again once an app opens it.) + let (_, opening, opened) = r.open_document(node, &app()); + assert_eq!(opening, Opening::Live); + frames.extend(opened); + assert!(frames.iter().any(|f| matches!(f, ClientFrame::Subscribe { after: 3, .. })), "{frames:?}"); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 3 }).unwrap(); + // A stray ack of P (from the old connection): no news. + r.on_frame(ServerFrame::Ack { doc: id, client_seq: 1, seq: 1 }).unwrap(); + assert_eq!(r.document_status(id), Some((true, 0))); + // Rows 4, 5 from a laptop that was offline since before P (its edits don't build on P). + let laptop = text_doc(80); + laptop.import(&import).unwrap(); + let l4 = insert(&laptop, 0, "L4 "); + let l5 = insert(&laptop, 0, "L5 "); + server.import(&l4).unwrap(); + server.import(&l5).unwrap(); + r.on_frame(ServerFrame::Update { doc: id, seq: 4, bytes: l4, device: device() }).unwrap(); + r.on_frame(ServerFrame::Update { doc: id, seq: 5, bytes: l5, device: device() }).unwrap(); + // Asked to compact at 5 (base 3: the server takes it). + let frames = r.on_frame(ServerFrame::Compact { doc: id, upto: 5 }).unwrap(); + let manifest = frames + .iter() + .find_map(|f| if let ClientFrame::Snapshot { upto: 5, manifest, .. } = f { Some(*manifest) } else { None }) + .expect("answered"); + let bytes = uploaded(&mut r, manifest); + let fresh = loro::LoroDoc::new(); + fresh.import(&bytes).unwrap(); + let text = fresh.get_text("text").to_string(); + let all = server.get_text("text").to_string(); + assert_eq!(text, all, "a fresh machine from the new snapshot has every row up to 5, P too"); + } + + /// A row this machine dropped (it can never apply here) is at or below the asked number: no snapshot (its copy + /// lacks the row), also after a restart (the file keeps it); one past the base of the server's next snapshot is + /// in the copy again (`dropped` clears when a snapshot's base passes it). + #[test] + fn a_machine_that_dropped_a_row_doesnt_compact_past_it() { + let store = MemStore::default(); + let space = SpaceId::generate(); + let node = NodeId::generate(); + let w2 = ServerFrame::Welcome { protocol: 2, epoch: epoch(1) }; + let importer = text_doc(40); + insert(&importer, 0, "import"); + let import = importer.export(loro::ExportMode::Snapshot).unwrap(); + let other = text_doc(60); + other.import(&import).unwrap(); + let o1 = insert(&other, 0, "O1 "); + let o3 = insert(&other, 0, "O3 "); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + r.on_frame(w2.clone()).unwrap(); + let (id, _, _) = r.open_document(node, &app()); + snapshot_in(&mut r, id, 0, &import); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 0 }).unwrap(); + r.on_frame(ServerFrame::Update { doc: id, seq: 1, bytes: o1, device: device() }).unwrap(); + // Row 2 never applies here. + r.on_frame(ServerFrame::Update { doc: id, seq: 2, bytes: vec![0xff; 40], device: device() }).unwrap(); + r.on_frame(ServerFrame::Update { doc: id, seq: 3, bytes: o3, device: device() }).unwrap(); + assert!(r.take_notes().iter().any(|n| n.contains("leaves compacting")), "said"); + assert!(r.on_frame(ServerFrame::Compact { doc: id, upto: 3 }).unwrap().is_empty(), "no snapshot past a dropped row"); + r.flush_documents().unwrap(); + drop(r); + let mut r = Replica::open(store.clone(), space, peers()).unwrap(); + r.connected(); + r.on_frame(w2.clone()).unwrap(); + r.open_document(node, &app()); + r.on_frame(ServerFrame::Subscribed { doc: id, head: 3 }).unwrap(); + assert!(r.on_frame(ServerFrame::Compact { doc: id, upto: 3 }).unwrap().is_empty(), "nor after a restart"); + } + + /// A recents file says which row was dropped after what an older agent reads; one an older agent wrote reads too. + #[test] + fn a_recents_file_keeps_its_dropped_row_where_an_older_agent_looks_past_it() { + let space = SpaceId::generate(); + let store = MemStore::default(); + let mut r = live(space, store.clone()); + r.add(&app(), [1; 16], entry("a.docx", 1)).unwrap(); + let id = recents_doc(space, &app()); + r.docs.get_mut(&id).unwrap().dropped = Some(7); + r.docs.get_mut(&id).unwrap().dirty = true; + r.flush().unwrap(); + let bytes = store.load().unwrap().get(&id.to_string()).cloned().unwrap(); + let older = postcard::from_bytes::(&bytes).unwrap(); + assert_eq!(older.app, app(), "an older agent reads the file"); + let (_, more) = read_doc_file(&bytes).unwrap(); + assert_eq!(more.dropped, Some(7)); + let plain = postcard::to_stdvec(&older).unwrap(); + assert_eq!(read_doc_file(&plain).unwrap().1.dropped, None, "a file without it"); + let r = Replica::open(store, space, peers()).unwrap(); + assert_eq!(r.docs.get(&id).unwrap().dropped, Some(7)); + } + + /// An own update acked at a number above what is had, then a row with the same operations (another device pushed + /// them too) below it: the ack is kept until the rows below it are here, so what is had reaches the acked row and + /// the next own ack counts. + #[test] + fn an_early_ack_is_kept_when_a_row_below_it_holds_the_same_update() { + let space = SpaceId::generate(); + let doc = recents_doc(space, &app()); + let mut r = live(space, MemStore::default()); + let frames = r.add(&app(), [1; 16], entry("a.docx", 1)).unwrap(); + let ClientFrame::Update { client_seq, bytes, .. } = frames[0].clone() else { panic!("{frames:?}") }; + // Acked as row 2: row 1 is on its way. + r.on_frame(ServerFrame::Ack { doc, client_seq, seq: 2 }).unwrap(); + // Row 1 holds the same bytes (stored by another device). + r.on_frame(ServerFrame::Update { doc, seq: 1, bytes, device: device() }).unwrap(); + // The next own update, acked as row 3. + let frames = r.add(&app(), [2; 16], entry("b.docx", 2)).unwrap(); + let ClientFrame::Update { client_seq, .. } = frames[0].clone() else { panic!("{frames:?}") }; + r.on_frame(ServerFrame::Ack { doc, client_seq, seq: 3 }).unwrap(); + assert_eq!(r.waiting_for(&app()), 0, "row 2 is this machine's own (never sent to it live): {}", r.describe(&app())); + let other = Recents::new(77); + let u4 = other.add([5; 16], &entry("c.docx", 4)).unwrap(); + assert!(r.on_frame(ServerFrame::Update { doc, seq: 4, bytes: u4, device: device() }).unwrap().is_empty(), "no gap at 3"); + } } diff --git a/apps/craft-cloud-agent/src/live/store.rs b/apps/craft-cloud-agent/src/live/store.rs index 719eeec..56adc75 100644 --- a/apps/craft-cloud-agent/src/live/store.rs +++ b/apps/craft-cloud-agent/src/live/store.rs @@ -22,8 +22,56 @@ pub trait DocStore: Send { fn set_aside(&mut self, name: &str) -> io::Result; /// Deletes a document file (one that doesn't read at all, once set aside). fn remove(&mut self, name: &str) -> io::Result<()>; + + // Documents of files: files of their own kind (an older agent, which reads only the files above, leaves them + // alone), each with a journal of the local updates made since it was last written. + + // (By default a store has none, and takes none: one for recents only.) + + /// The names of every document of a file there is (each read with [`DocStore::load_document`], one at a time). + fn document_names(&self) -> io::Result> { + Ok(Vec::new()) + } + /// One document of a file, if there is one. + fn load_document(&self, _name: &str) -> io::Result>> { + Ok(None) + } + /// Writes a document of a file whole, atomically and durably. + fn save_document(&mut self, _name: &str, _bytes: &[u8]) -> io::Result<()> { + Err(io::Error::from(io::ErrorKind::Unsupported)) + } + /// Appends `record` to the document's journal, durably, before this returns. On an error the journal is cut back + /// to where it was, as far as the store can (the caller cuts it back too, and writes the document whole). + fn append_journal(&mut self, _name: &str, _record: &[u8]) -> io::Result<()> { + Err(io::Error::from(io::ErrorKind::Unsupported)) + } + /// Cuts the document's journal to its first `len` bytes (what follows doesn't read, or was never kept), durably. + fn truncate_journal(&mut self, _name: &str, _len: usize) -> io::Result<()> { + Ok(()) + } + /// The document's journal (empty if it has none). + fn load_journal(&self, _name: &str) -> io::Result> { + Ok(Vec::new()) + } + /// Empties the document's journal (its updates are in its file now). + fn clear_journal(&mut self, _name: &str) -> io::Result<()> { + Ok(()) + } + /// [`DocStore::set_aside`], for a document of a file. + fn set_aside_document(&mut self, _name: &str) -> io::Result { + Err(io::Error::from(io::ErrorKind::Unsupported)) + } + /// [`DocStore::remove`], for a document of a file (and its journal). + fn remove_document(&mut self, _name: &str) -> io::Result<()> { + Ok(()) + } } +/// Documents of files end in this… +const DOCUMENT_SUFFIX: &str = ".doc"; +/// …and their journals in this. +const JOURNAL_SUFFIX: &str = ".journal"; + /// Most set-aside files kept per store. pub const MAX_SET_ASIDE: usize = 3; @@ -41,9 +89,42 @@ impl DiskStore { pub fn open(dir: &Path) -> io::Result { make_private_dirs(dir)?; // (Interrupted writes of documents and of set-aside copies.) - remove_atomic_temps(dir, |of| of.ends_with(SUFFIX) || of.contains(ASIDE), std::time::Duration::ZERO)?; + remove_atomic_temps(dir, |of| of.ends_with(SUFFIX) || of.ends_with(DOCUMENT_SUFFIX) || of.contains(ASIDE), std::time::Duration::ZERO)?; Ok(Self { dir: dir.to_owned() }) } + + fn set_aside_suffix(&mut self, name: &str, suffix: &str) -> io::Result { + let bytes = std::fs::read(self.dir.join(format!("{name}{suffix}")))?; + // Unique: the time, then random bytes (two in one millisecond, or a clock gone back, don't collide). Written + // like a document (temporary file, fsync, rename, folder fsync). + let stamp = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_millis()); + static COUNT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + let count = COUNT.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + let to = self.dir.join(format!("{name}{suffix}{ASIDE}{stamp:013}-{count:08}-{}", crate::account::random_hex(4)?)); + write_atomic(&to, &bytes, true)?; + // Oldest first: the name sorts by its time. Each kind of file keeps its own few. + let kind = format!("{suffix}{ASIDE}"); + let mut aside: Vec<(String, PathBuf)> = std::fs::read_dir(&self.dir)? + .filter_map(Result::ok) + .filter_map(|e| { + let file = e.file_name().to_str()?.to_owned(); + let at = file.find(&kind)?; + Some((file[at..].to_owned(), e.path())) + }) + .collect(); + aside.sort(); + let extra = aside.len().saturating_sub(MAX_SET_ASIDE); + for (_, path) in aside.into_iter().take(extra) { + std::fs::remove_file(path)?; + } + Ok(to.display().to_string()) + } + + fn sync_dir(&self) -> io::Result<()> { + #[cfg(unix)] + std::fs::File::open(&self.dir)?.sync_all()?; + Ok(()) + } } /// Makes `dir` and any missing folder above it owner-only (they sit inside the data folder, which exists). @@ -78,29 +159,7 @@ impl DocStore for DiskStore { } fn set_aside(&mut self, name: &str) -> io::Result { - let bytes = std::fs::read(self.dir.join(format!("{name}{SUFFIX}")))?; - // Unique: the time, then random bytes (two in one millisecond, or a clock gone back, don't collide). Written - // like a document (temporary file, fsync, rename, folder fsync). - let stamp = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_millis()); - static COUNT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); - let count = COUNT.fetch_add(1, std::sync::atomic::Ordering::Relaxed); - let to = self.dir.join(format!("{name}{SUFFIX}{ASIDE}{stamp:013}-{count:08}-{}", crate::account::random_hex(4)?)); - write_atomic(&to, &bytes, true)?; - // Oldest first: the name sorts by its time. - let mut aside: Vec<(String, PathBuf)> = std::fs::read_dir(&self.dir)? - .filter_map(Result::ok) - .filter_map(|e| { - let file = e.file_name().to_str()?.to_owned(); - let at = file.find(ASIDE)?; - Some((file[at..].to_owned(), e.path())) - }) - .collect(); - aside.sort(); - let extra = aside.len().saturating_sub(MAX_SET_ASIDE); - for (_, path) in aside.into_iter().take(extra) { - std::fs::remove_file(path)?; - } - Ok(to.display().to_string()) + self.set_aside_suffix(name, SUFFIX) } fn remove(&mut self, name: &str) -> io::Result<()> { @@ -109,6 +168,93 @@ impl DocStore for DiskStore { std::fs::File::open(&self.dir)?.sync_all()?; Ok(()) } + + fn document_names(&self) -> io::Result> { + let mut out = Vec::new(); + for entry in std::fs::read_dir(&self.dir)? { + let entry = entry?; + let Some(name) = entry.file_name().to_str().map(str::to_owned) else { continue }; + if let Some(doc) = name.strip_suffix(DOCUMENT_SUFFIX) + && !name.starts_with('.') + && entry.file_type()?.is_file() + { + out.push(doc.to_owned()); + } + } + out.sort(); + Ok(out) + } + + fn load_document(&self, name: &str) -> io::Result>> { + match std::fs::read(self.dir.join(format!("{name}{DOCUMENT_SUFFIX}"))) { + Ok(bytes) => Ok(Some(bytes)), + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(None), + Err(e) => Err(e), + } + } + + fn save_document(&mut self, name: &str, bytes: &[u8]) -> io::Result<()> { + write_atomic(&self.dir.join(format!("{name}{DOCUMENT_SUFFIX}")), bytes, true) + } + + fn append_journal(&mut self, name: &str, record: &[u8]) -> io::Result<()> { + use std::io::Write; + let path = self.dir.join(format!("{name}{JOURNAL_SUFFIX}")); + let new = !path.exists(); + let mut file = std::fs::OpenOptions::new().create(true).append(true).open(&path)?; + let before = file.metadata()?.len(); + if let Err(e) = file.write_all(record).and_then(|()| file.sync_data()) { + // Not kept: whatever part of it got there goes (a record cut off, or one whose flush failed, must not come + // back after a crash as a change the app was told wasn't kept, nor hide the records after it). + let _ = file.set_len(before).and_then(|()| file.sync_data()); + return Err(e); + } + // A journal made just now: its name is durable too. + if new { + self.sync_dir()?; + } + Ok(()) + } + + fn truncate_journal(&mut self, name: &str, len: usize) -> io::Result<()> { + let path = self.dir.join(format!("{name}{JOURNAL_SUFFIX}")); + let file = match std::fs::OpenOptions::new().write(true).open(&path) { + Ok(file) => file, + Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(()), + Err(e) => return Err(e), + }; + if file.metadata()?.len() > len as u64 { + file.set_len(len as u64)?; + file.sync_data()?; + } + Ok(()) + } + + fn load_journal(&self, name: &str) -> io::Result> { + match std::fs::read(self.dir.join(format!("{name}{JOURNAL_SUFFIX}"))) { + Ok(bytes) => Ok(bytes), + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(Vec::new()), + Err(e) => Err(e), + } + } + + fn clear_journal(&mut self, name: &str) -> io::Result<()> { + match std::fs::remove_file(self.dir.join(format!("{name}{JOURNAL_SUFFIX}"))) { + Ok(()) => self.sync_dir(), + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e), + } + } + + fn set_aside_document(&mut self, name: &str) -> io::Result { + self.set_aside_suffix(name, DOCUMENT_SUFFIX) + } + + fn remove_document(&mut self, name: &str) -> io::Result<()> { + std::fs::remove_file(self.dir.join(format!("{name}{DOCUMENT_SUFFIX}")))?; + self.clear_journal(name)?; + self.sync_dir() + } } /// What a set-aside file's name has after the document's. @@ -120,6 +266,49 @@ pub struct MemStore { files: Arc>>>, aside: Arc>>>, set_aside_count: Arc, + documents: Arc>>>, + journals: Arc>>>, + /// How journal appends fail from now on ([`JournalFailure`] as a number; 0: they don't), for tests. + failing: Arc, +} + +/// How a [`MemStore`]'s journal appends fail, for tests. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum JournalFailure { + /// Nothing is written (a full disk). + Nothing, + /// Half the record is written, and stays (a store that couldn't cut it back). + Torn, + /// The whole record is written, and stays, but the flush fails. + Unflushed, +} + +impl MemStore { + /// Makes every journal append fail (`true`, writing nothing) or work again: a full disk, for tests. + pub fn fail_journal(&self, fail: bool) { + self.fail_journal_as(fail.then_some(JournalFailure::Nothing)); + } + + /// Makes every journal append fail as `how`, or work again (`None`). + pub fn fail_journal_as(&self, how: Option) { + let n = match how { + None => 0, + Some(JournalFailure::Nothing) => 1, + Some(JournalFailure::Torn) => 2, + Some(JournalFailure::Unflushed) => 3, + }; + self.failing.store(n, std::sync::atomic::Ordering::Relaxed); + } + + /// The journals, by document (for tests that look inside). + pub fn journals(&self) -> BTreeMap> { + self.journals.lock().unwrap_or_else(PoisonError::into_inner).clone() + } + + /// The documents of files, by name (for tests that look inside). + pub fn documents(&self) -> BTreeMap> { + self.documents.lock().unwrap_or_else(PoisonError::into_inner).clone() + } } impl MemStore { @@ -155,12 +344,97 @@ impl DocStore for MemStore { self.files.lock().unwrap_or_else(PoisonError::into_inner).remove(name); Ok(()) } + + fn document_names(&self) -> io::Result> { + Ok(self.documents().into_keys().collect()) + } + + fn load_document(&self, name: &str) -> io::Result>> { + Ok(self.documents.lock().unwrap_or_else(PoisonError::into_inner).get(name).cloned()) + } + + fn save_document(&mut self, name: &str, bytes: &[u8]) -> io::Result<()> { + self.documents.lock().unwrap_or_else(PoisonError::into_inner).insert(name.to_owned(), bytes.to_vec()); + Ok(()) + } + + fn append_journal(&mut self, name: &str, record: &[u8]) -> io::Result<()> { + let mut journals = self.journals.lock().unwrap_or_else(PoisonError::into_inner); + match self.failing.load(std::sync::atomic::Ordering::Relaxed) { + 0 => {} + 1 => return Err(io::Error::other("the disk is full (test)")), + 2 => { + journals.entry(name.to_owned()).or_default().extend_from_slice(record.get(..record.len() / 2).unwrap_or_default()); + return Err(io::Error::other("the disk filled up mid-write (test)")); + } + _ => { + journals.entry(name.to_owned()).or_default().extend_from_slice(record); + return Err(io::Error::other("the flush failed (test)")); + } + } + journals.entry(name.to_owned()).or_default().extend_from_slice(record); + Ok(()) + } + + fn truncate_journal(&mut self, name: &str, len: usize) -> io::Result<()> { + if let Some(journal) = self.journals.lock().unwrap_or_else(PoisonError::into_inner).get_mut(name) { + journal.truncate(len); + } + Ok(()) + } + + fn load_journal(&self, name: &str) -> io::Result> { + Ok(self.journals.lock().unwrap_or_else(PoisonError::into_inner).get(name).cloned().unwrap_or_default()) + } + + fn clear_journal(&mut self, name: &str) -> io::Result<()> { + self.journals.lock().unwrap_or_else(PoisonError::into_inner).remove(name); + Ok(()) + } + + fn set_aside_document(&mut self, name: &str) -> io::Result { + let bytes = self.load_document(name)?.ok_or_else(|| io::Error::other("no such file"))?; + let mut aside = self.aside.lock().unwrap_or_else(PoisonError::into_inner); + let n = self.set_aside_count.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + let to = format!("{n:013}{ASIDE}{name}{DOCUMENT_SUFFIX}"); + aside.insert(to.clone(), bytes); + while aside.len() > MAX_SET_ASIDE { + aside.pop_first(); + } + Ok(to) + } + + fn remove_document(&mut self, name: &str) -> io::Result<()> { + self.documents.lock().unwrap_or_else(PoisonError::into_inner).remove(name); + self.clear_journal(name) + } } #[cfg(test)] mod tests { use super::*; + /// Recents files and documents of files each keep their own few set-aside copies: many of one kind never push out + /// the other's. + #[test] + fn recents_and_documents_keep_set_aside_copies_of_their_own() { + let dir = tempfile::tempdir().unwrap(); + let mut store = DiskStore::open(dir.path()).unwrap(); + store.save("recents", b"damaged list").unwrap(); + store.save_document("doc", b"damaged document").unwrap(); + store.set_aside("recents").unwrap(); + for _ in 0..MAX_SET_ASIDE + 2 { + store.set_aside_document("doc").unwrap(); + } + let aside: Vec = std::fs::read_dir(dir.path()) + .unwrap() + .filter_map(|e| e.ok()?.file_name().to_str().map(str::to_owned)) + .filter(|n| n.contains(ASIDE)) + .collect(); + assert_eq!(aside.iter().filter(|n| n.contains(&format!("{SUFFIX}{ASIDE}"))).count(), 1, "the recents copy stays: {aside:?}"); + assert_eq!(aside.iter().filter(|n| n.contains(&format!("{DOCUMENT_SUFFIX}{ASIDE}"))).count(), MAX_SET_ASIDE, "{aside:?}"); + } + /// Setting a file aside copies it (renaming would lose the document to a crash until the next load rebuilt it), the /// original stays until the caller writes over it, names are unique (not just by the millisecond), and only the /// newest few are kept. diff --git a/apps/craft-cloud-agent/src/local_api.rs b/apps/craft-cloud-agent/src/local_api.rs index 2a793ee..f2af1c4 100644 --- a/apps/craft-cloud-agent/src/local_api.rs +++ b/apps/craft-cloud-agent/src/local_api.rs @@ -48,6 +48,8 @@ pub enum Command { Resume(oneshot::Sender>), /// Finish the pass under way and stop the agent. Answered at once. Quit(oneshot::Sender>), + /// Run a sync pass soon (an app opened a file that isn't uploaded yet, to make it live). Not answered. + SyncSoon, } /// What [`Status`] reports, kept up to date by the `run` loop. @@ -148,6 +150,11 @@ impl Control { self.with(|s| s.online = online); } + /// Asks the `run` loop for a sync pass soon (never waits: a loop that is busy takes it when it can). + pub fn request_pass(&self) { + let _ = self.commands.try_send(Command::SyncSoon); + } + /// The live hub that serves recents (while signed in). pub fn set_hub(&self, hub: Option) { self.with(|s| s.hub = hub); @@ -270,6 +277,11 @@ impl LocalApi { let stop = async move { let _ = shutting_down.await; }; + // No Nagle delay: a live document's update to an app goes out at once. + use axum::serve::ListenerExt; + let listener = listener.tap_io(|tcp| { + let _ = tcp.set_nodelay(true); + }); if let Err(e) = axum::serve(listener, app).with_graceful_shutdown(stop).await { crate::runlog::note(format_args!("the local API stopped: {e}")); } @@ -316,6 +328,7 @@ fn router(state: Api) -> Router { .route(paths::LOGIN, post(login)) .route(paths::LOGOUT, post(logout)) .route(paths::RECENTS, get(crate::live::recents_socket)) + .route(paths::DOCS, get(crate::live::docs::docs_socket)) .route(paths::JOIN, post(join)) .route(paths::PAIRINGS, post(create_pairing)) .route(&format!("{}/{{id}}", paths::PAIRINGS), get(pairing_status).delete(delete_pairing)) diff --git a/apps/craft-cloud-agent/tests/docs_common/blobs.rs b/apps/craft-cloud-agent/tests/docs_common/blobs.rs new file mode 100644 index 0000000..d169fd7 --- /dev/null +++ b/apps/craft-cloud-agent/tests/docs_common/blobs.rs @@ -0,0 +1,93 @@ +//! The chunk API as the server serves it, in-process (what the agent's hub reaches over HTTP to upload and download +//! live documents' snapshots): every chunk and manifest checked and recorded as the API does, under the account's +//! storage cap. Only the blob calls; the rest of the API isn't needed here. Included by `#[path]`. + +#![allow(dead_code)] + +use std::sync::Arc; + +use async_trait::async_trait; +use bytes::Bytes; +use craft_cloud_proto::api::{CommitResponse, SpaceResponse}; +use craft_cloud_proto::{BlobKind, ChangeSet, ChunkId, Manifest, ManifestId, Op, PageCursor, SpaceId}; +use craft_cloud_server::AppState; +use craft_cloud_store::BlobKey; +use craft_cloud_sync::transport::{Transport, TransportError}; + +pub struct ServerBlobs { + pub state: Arc, + pub space: SpaceId, +} + +fn failed(e: impl std::fmt::Display) -> TransportError { + TransportError::Unavailable(e.to_string()) +} + +fn meta_error(e: craft_cloud_meta::MetaError) -> TransportError { + match e { + craft_cloud_meta::MetaError::QuotaExceeded { used, limit } => TransportError::QuotaExceeded { used, limit }, + craft_cloud_meta::MetaError::MissingChunks(ids) => TransportError::MissingChunks(ids), + other => failed(other), + } +} + +#[async_trait] +impl Transport for ServerBlobs { + async fn space(&self) -> Result { + Err(TransportError::Protocol("not here".into())) + } + + async fn tree(&self, _: Option) -> Result { + Err(TransportError::Protocol("not here".into())) + } + + async fn changes(&self, _: u64, _: Option) -> Result { + Err(TransportError::Protocol("not here".into())) + } + + async fn missing_chunks(&self, ids: &[ChunkId]) -> Result, TransportError> { + self.state.meta.missing_chunks(self.space, ids).await.map_err(meta_error) + } + + async fn put_chunk(&self, id: &ChunkId, data: Bytes) -> Result<(), TransportError> { + if *blake3::hash(&data).as_bytes() != *id.as_bytes() { + return Err(TransportError::Protocol("the chunk isn't its ID".into())); + } + self.state.meta.check_room(self.space, BlobKind::Chunk, id.as_bytes(), data.len() as u64 + 1).await.map_err(meta_error)?; + self.state.store.put(BlobKey::chunk(id), &data).await.map_err(failed)?; + self.state.meta.add_chunk(self.space, id, data.len() as u64 + 1, 0).await.map_err(meta_error) + } + + async fn get_chunk(&self, id: &ChunkId) -> Result { + if !self.state.meta.missing_chunks(self.space, &[*id]).await.map_err(meta_error)?.is_empty() { + return Err(TransportError::NotFound); + } + self.state.store.get(BlobKey::chunk(id)).await.map_err(failed) + } + + async fn put_manifest(&self, id: &ManifestId, data: Bytes) -> Result<(), TransportError> { + let manifest = Manifest::decode(&data).map_err(|e| TransportError::Protocol(e.to_string()))?; + if manifest.id() != *id { + return Err(TransportError::Protocol("the manifest isn't its ID".into())); + } + let ids: Vec = manifest.chunks().iter().map(|c| c.id).collect(); + let missing = self.state.meta.missing_chunks(self.space, &ids).await.map_err(meta_error)?; + if !missing.is_empty() { + return Err(TransportError::MissingChunks(missing)); + } + self.state.meta.check_room(self.space, BlobKind::Manifest, id.as_bytes(), data.len() as u64 + 1).await.map_err(meta_error)?; + self.state.store.put(BlobKey::manifest(id), &data).await.map_err(failed)?; + self.state.meta.add_manifest(self.space, &manifest, data.len() as u64 + 1, 0).await.map_err(meta_error) + } + + async fn get_manifest(&self, id: &ManifestId) -> Result { + if !self.state.meta.has_manifest(self.space, id).await.map_err(meta_error)? { + return Err(TransportError::NotFound); + } + self.state.store.get(BlobKey::manifest(id)).await.map_err(failed) + } + + async fn commit(&self, _: &[Op]) -> Result { + Err(TransportError::Protocol("not here".into())) + } +} diff --git a/apps/craft-cloud-agent/tests/docs_common/mod.rs b/apps/craft-cloud-agent/tests/docs_common/mod.rs new file mode 100644 index 0000000..d1d015c --- /dev/null +++ b/apps/craft-cloud-agent/tests/docs_common/mod.rs @@ -0,0 +1,395 @@ +//! What the documents tests share: machines (the agent's own [`Replica`] over a store that outlasts its crashes) +//! against the server's own live [`Session`]s over the real metadata database (SQLite in memory), connections as +//! queues of frames, and app stand-ins (a Loro document the way an app's would be). Included by `#[path]`. + +#![allow(dead_code)] + +pub mod blobs; + +use std::collections::VecDeque; +use std::sync::Arc; + +use craft_cloud_agent::live::replica::{DocEvent, Opening, Peers, Replica}; +use craft_cloud_agent::live::store::MemStore; +use craft_cloud_meta::{DeviceAuth, Meta, Role, UserId}; +use craft_cloud_proto::live::{AppName, ClientFrame, ServerFrame}; +use craft_cloud_proto::{ChunkId, ChunkRef, ChunkerParamsId, DocId, FileHash, Manifest, Name, NewVersion, NodeId, Op, SpaceId, VersionId}; +use craft_cloud_server::AppState; +use craft_cloud_server::live::{PresenceNote, Session}; +use craft_cloud_server::state::Limits; +use craft_cloud_store::Store; +use craft_cloud_store::memory::MemoryBackend; +use loro::{ExportMode, LoroDoc}; +use tokio::sync::broadcast; + +pub fn app() -> AppName { + AppName::new("wordcraft").unwrap() +} + +/// Loro peer IDs for one run of a machine: distinct per document and per run. +pub fn peers(base: u64) -> Peers { + let mut n = 0u64; + Box::new(move || { + n += 1; + base.wrapping_add(n.wrapping_mul(0x9E37_79B9_7F4A_7C15)) >> 1 + }) +} + +/// An open connection: the server's session for it, what the server hears of its space, and its presence. +pub struct Conn { + pub session: Session, + pub notes: broadcast::Receiver, + pub presence: broadcast::Receiver, +} + +pub struct Machine { + pub store: MemStore, + pub replica: Option>, + pub auth: DeviceAuth, + pub conn: Option, + pub up: VecDeque, + pub down: VecDeque, + /// What happened to its documents, as its apps would hear it. + pub events: Vec, + pub said: Vec, + /// Every frame its agent was sent (for tests that look at what came). + pub heard: Vec, + /// Every frame its agent sent that reached the server. + pub said_up: Vec, + /// This machine's runs so far (each gets peers of its own). + pub runs: u64, + pub base: u64, + /// Rewrite its `hello` to name only protocol 1 (an old server, as the agent sees it). + pub old_server: bool, +} + +impl Machine { + pub fn replica(&mut self) -> &mut Replica { + self.replica.as_mut().expect("the machine runs") + } +} + +pub struct World { + pub state: Arc, + pub meta: Meta, + pub space: SpaceId, + pub user: UserId, + pub machines: Vec, + root: NodeId, + files: u32, +} + +impl World { + pub async fn new(limits: Limits) -> Self { + Self::with_meta(limits, Meta::open("sqlite::memory:").await.unwrap()).await + } + + pub async fn with_meta(limits: Limits, meta: Meta) -> Self { + let user = meta.create_user("ana", "$argon2id$stub", Role::Member, 0).await.unwrap(); + let state = Arc::new(AppState::new(meta.clone(), Store::new(MemoryBackend::new()), vec![]).unwrap().with_limits(limits)); + let root = meta.space_info(user.space).await.unwrap().root; + Self { state, meta, space: user.space, user: user.id, machines: Vec::new(), root, files: 0 } + } + + /// A new machine of the account (not connected yet). + pub async fn machine(&mut self) -> usize { + let i = self.machines.len(); + let token = [10 + i as u8; 32]; + self.meta.create_device(self.user, &format!("m{i}"), &token, 0).await.unwrap(); + let auth = self.meta.device_by_token(&token).await.unwrap().unwrap(); + let store = MemStore::default(); + let base = (1000 * (i as u64 + 1)) << 20; + let replica = Replica::open(store.clone(), self.space, peers(base)).unwrap(); + self.machines.push(Machine { + store, + replica: Some(replica), + auth, + conn: None, + up: VecDeque::new(), + down: VecDeque::new(), + events: Vec::new(), + said: Vec::new(), + heard: Vec::new(), + said_up: Vec::new(), + runs: 1, + base, + old_server: false, + }); + i + } + + /// A new file of the account's space, as the file sync would have uploaded it. + pub async fn file(&mut self) -> NodeId { + self.files += 1; + let name = format!("file{}.txt", self.files); + let chunk = ChunkId::from_bytes(*blake3::hash(name.as_bytes()).as_bytes()); + self.meta.add_chunk(self.space, &chunk, 11, 0).await.unwrap(); + let manifest = + Manifest::new(ChunkerParamsId::new("fastcdc-v1").unwrap(), 10, FileHash::from_bytes([1; 32]), vec![ChunkRef { id: chunk, len: 10 }]) + .unwrap(); + self.meta.add_manifest(self.space, &manifest, 100, 0).await.unwrap(); + let node = NodeId::generate(); + let version = NewVersion { id: VersionId::generate(), manifest: manifest.id(), mtime_ms: 0, exec: false }; + let device = self.machines.first().map_or_else(craft_cloud_proto::DeviceId::generate, |m| m.auth.device); + self.meta.commit(self.space, device, &[Op::Create { node, parent: self.root, name: Name::new(&name).unwrap(), version }], 0).await.unwrap(); + node + } + + pub fn connect(&mut self, m: usize) { + let state = self.state.clone(); + let machine = &mut self.machines[m]; + let notes = state.live.listen(machine.auth.space); + let presence = state.live.listen_presence(machine.auth.space); + machine.conn = Some(Conn { session: Session::new(state, &machine.auth), notes, presence }); + let hello = machine.replica().connected(); + machine.up.extend(hello); + } + + /// The connection goes, with whatever was in flight both ways. + pub fn disconnect(&mut self, m: usize) { + let machine = &mut self.machines[m]; + machine.conn = None; + machine.up.clear(); + machine.down.clear(); + if let Some(replica) = machine.replica.as_mut() { + replica.disconnected(); + } + machine.events.extend(machine.replica.as_mut().map(|r| r.take_events()).unwrap_or_default()); + } + + /// The machine crashes (what it hadn't written is gone) and starts again from its store. + pub fn crash(&mut self, m: usize) { + self.disconnect(m); + let machine = &mut self.machines[m]; + machine.replica = None; + machine.runs += 1; + let base = machine.base ^ (machine.runs << 40); + let mut replica = Replica::open(machine.store.clone(), self.space, peers(base)).unwrap(); + machine.said.extend(replica.take_notes()); + machine.replica = Some(replica); + } + + /// The machine stops cleanly: everything it holds is written first. + pub fn stop(&mut self, m: usize) { + if let Some(replica) = self.machines[m].replica.as_mut() { + replica.flush_documents().unwrap(); + } + self.crash(m); + } + + /// Hands frames from the agent to its connection's queue. + pub fn send(&mut self, m: usize, frames: Vec) { + let machine = &mut self.machines[m]; + if machine.conn.is_some() { + machine.up.extend(frames); + } + } + + /// The chunk API of the account's space, as the agents reach it (for snapshots). + pub fn blobs(&self) -> blobs::ServerBlobs { + blobs::ServerBlobs { state: self.state.clone(), space: self.space } + } + + /// `bytes` uploaded as a snapshot of `doc`, as an agent uploads one: the manifest that names it. + pub async fn upload(&self, doc: DocId, bytes: &[u8]) -> craft_cloud_proto::ManifestId { + let up = craft_cloud_agent::live::blobs::cut(doc, bytes).unwrap(); + craft_cloud_agent::live::blobs::upload(&self.blobs(), &up).await.unwrap(); + up.id() + } + + /// The snapshot `manifest` names, downloaded as an agent downloads one. + pub async fn download(&self, manifest: &craft_cloud_proto::ManifestId) -> Vec { + craft_cloud_agent::live::blobs::download(&self.blobs(), manifest).await.unwrap() + } + + /// One round of delivery on every connection; whether anything moved. Snapshots go up and down through the chunk + /// API as the hub moves them: each upload before the frames that follow it, each download handed back before what + /// comes next. + pub async fn step(&mut self) -> bool { + let mut moved = false; + let api = self.blobs(); + for machine in &mut self.machines { + let Some(conn) = machine.conn.as_mut() else { continue }; + if let Some(replica) = machine.replica.as_mut() { + for up in replica.take_uploads() { + if let Err(e) = craft_cloud_agent::live::blobs::upload(&api, &up).await { + replica.upload_failed(up.doc, &e.to_string()); + let id = up.id(); + machine.up.retain(|f| !matches!(f, ClientFrame::Snapshot { manifest, .. } | ClientFrame::CreateDocument { snapshot: manifest, .. } if *manifest == id)); + } + } + } + while let Some(mut frame) = machine.up.pop_front() { + moved = true; + if machine.old_server + && let ClientFrame::Hello { min, .. } = frame + { + frame = ClientFrame::Hello { min, max: 1 }; + } + machine.said_up.push(frame.clone()); + machine.down.extend(conn.session.handle(frame).await.frames); + } + while let Ok(doc) = conn.notes.try_recv() { + conn.session.note(doc); + } + while let Ok(note) = conn.presence.try_recv() { + machine.down.extend(conn.session.relay(¬e)); + } + while conn.session.has_work() { + moved = true; + machine.down.extend(conn.session.pump().await.unwrap()); + } + let Some(replica) = machine.replica.as_mut() else { continue }; + while let Some(frame) = machine.down.pop_front() { + moved = true; + machine.heard.push(frame.clone()); + match replica.on_frame(frame) { + Ok(frames) => { + machine.up.extend(frames); + for (doc, manifest) in replica.take_fetches() { + let got = craft_cloud_agent::live::blobs::download(&api, &manifest).await.map_err(|e| e.to_string()); + machine.up.extend(replica.on_snapshot(doc, manifest, got)); + } + } + Err(e) => { + machine.said.push(format!("the link ended: {e}")); + machine.conn = None; + machine.up.clear(); + machine.down.clear(); + replica.disconnected(); + break; + } + } + } + replica.flush().unwrap(); + machine.events.extend(replica.take_events()); + machine.said.extend(replica.take_notes()); + } + moved + } + + /// Delivers until nothing moves, ticking the agents (their resends) whenever things are quiet. + pub async fn settle(&mut self) { + let mut quiet = 0; + for _ in 0..100_000 { + if self.step().await { + quiet = 0; + continue; + } + quiet += 1; + if quiet > 40 { + return; + } + for m in 0..self.machines.len() { + if self.machines[m].conn.is_some() + && let Some(replica) = self.machines[m].replica.as_mut() + { + let frames = replica.resend(); + self.machines[m].up.extend(frames); + } + } + } + panic!("never settled"); + } + + /// The machine's app opens the file `node`: the replica's answer, and its frames sent. + pub fn open(&mut self, m: usize, node: NodeId) -> (DocId, Opening) { + let (doc, opening, frames) = self.machines[m].replica().open_document(node, &app()); + self.send(m, frames); + (doc, opening) + } + + /// Takes the machine's events. + pub fn events(&mut self, m: usize) -> Vec { + let machine = &mut self.machines[m]; + let mut events = std::mem::take(&mut machine.events); + if let Some(replica) = machine.replica.as_mut() { + events.extend(replica.take_events()); + } + events + } + + /// The machine's app imports `content` (a plain-text document) for `doc`. + pub fn import(&mut self, m: usize, doc: DocId, content: &str) { + let snapshot = App::new(1).with_text(content).snapshot(); + let frames = self.machines[m].replica().import_document(doc, &snapshot).unwrap(); + self.send(m, frames); + } + + /// The machine's app makes a change (on disk before this returns). + pub fn edit(&mut self, m: usize, doc: DocId, update: &[u8]) { + let frames = self.machines[m].replica().local_update(doc, update).unwrap(); + self.send(m, frames); + } + + /// The document as an app opening it on the machine would see it. + pub fn app_of(&mut self, m: usize, doc: DocId, peer: u64) -> App { + let (snapshot, waiting) = self.machines[m].replica().document_state(doc).expect("the document is live there"); + let app = App::new(peer); + app.apply(&snapshot); + for w in waiting { + app.apply(&w); + } + app + } + + /// The document's text as the machine holds it. + pub fn text(&mut self, m: usize, doc: DocId) -> String { + self.app_of(m, doc, 7).text() + } +} + +/// An app's own Loro document: a plain text under `text`, and a map under `map`. +pub struct App { + pub doc: LoroDoc, +} + +impl App { + pub fn new(peer: u64) -> Self { + let doc = LoroDoc::new(); + doc.set_peer_id(peer).unwrap(); + Self { doc } + } + + pub fn with_text(self, text: &str) -> Self { + self.doc.get_text("text").insert(0, text).unwrap(); + self.doc.commit(); + self + } + + pub fn snapshot(&self) -> Vec { + self.doc.export(ExportMode::Snapshot).unwrap() + } + + /// Types `s` at `at` (characters): the change, as an app sends it. + pub fn type_at(&self, at: usize, s: &str) -> Vec { + let before = self.doc.oplog_vv(); + let text = self.doc.get_text("text"); + let at = at.min(text.len_unicode()); + text.insert(at, s).unwrap(); + self.doc.commit(); + self.doc.export(ExportMode::updates(&before)).unwrap() + } + + /// Types at the end. + pub fn append(&self, s: &str) -> Vec { + let len = self.doc.get_text("text").len_unicode(); + self.type_at(len, s) + } + + /// Sets `key` in the map. + pub fn set(&self, key: &str, value: i64) -> Vec { + let before = self.doc.oplog_vv(); + self.doc.get_map("map").insert(key, value).unwrap(); + self.doc.commit(); + self.doc.export(ExportMode::updates(&before)).unwrap() + } + + pub fn apply(&self, bytes: &[u8]) { + self.doc.import(bytes).unwrap(); + } + + pub fn text(&self) -> String { + self.doc.get_text("text").to_string() + } +} diff --git a/apps/craft-cloud-agent/tests/documents.rs b/apps/craft-cloud-agent/tests/documents.rs new file mode 100644 index 0000000..4ff3d0a --- /dev/null +++ b/apps/craft-cloud-agent/tests/documents.rs @@ -0,0 +1,623 @@ +//! Documents of files through the agent's replica and the server's own live sessions: joining by import (one import +//! kept when two machines open a never-live file at once), edits going both ways and offline, compaction and catch-up +//! from a snapshot (one the agent can't read too), presence, the cap on followed documents, restores and resets with +//! snapshots in play, and an older server. + +#[path = "docs_common/mod.rs"] +mod common; + +use common::{App, World}; +use craft_cloud_agent::live::replica::{DocEvent, Opening}; +use craft_cloud_proto::live::{ClientFrame, MAX_SUBSCRIPTIONS, ServerFrame}; +use craft_cloud_proto::local::NotLiveReason; +use craft_cloud_server::state::Limits; + +fn small_logs() -> Limits { + Limits { compact_updates: 5, ..Limits::default() } +} + +fn rt() -> tokio::runtime::Runtime { + tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap() +} + +/// Opens `node` on `m` and settles; answers an import with `content` if asked. Returns the document. +async fn joined(w: &mut World, m: usize, node: craft_cloud_proto::NodeId, content: &str) -> craft_cloud_proto::DocId { + let (doc, opening) = w.open(m, node); + assert_eq!(opening, Opening::Joining); + w.settle().await; + let events = w.events(m); + if events.contains(&DocEvent::NeedsImport(doc)) { + w.import(m, doc, content); + w.settle().await; + assert!(w.events(m).contains(&DocEvent::Joined(doc)), "kept: live"); + } else { + assert!(events.contains(&DocEvent::Joined(doc)), "{events:?}"); + } + doc +} + +#[test] +fn a_file_opened_first_is_imported_and_the_next_machine_joins_it() { + rt().block_on(async { + let mut w = World::new(Limits::default()).await; + let (a, b) = (w.machine().await, w.machine().await); + w.connect(a); + w.connect(b); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "hello").await; + assert_eq!(w.text(a, doc), "hello"); + // The next machine joins it: it isn't asked for an import, and sees the first one's. + let (doc_b, opening) = w.open(b, node); + assert_eq!((doc_b, opening), (doc, Opening::Joining)); + w.settle().await; + let events = w.events(b); + assert!(events.contains(&DocEvent::Joined(doc)) && !events.contains(&DocEvent::NeedsImport(doc)), "{events:?}"); + assert_eq!(w.text(b, doc), "hello"); + // Edits go both ways, as the apps' own changes. + let app_a = w.app_of(a, doc, 101); + let app_b = w.app_of(b, doc, 102); + let u = app_a.append(" world"); + w.edit(a, doc, &u); + w.settle().await; + let remote: Vec> = + w.events(b).into_iter().filter_map(|e| if let DocEvent::Remote(d, bytes) = e { (d == doc).then_some(bytes) } else { None }).collect(); + assert!(!remote.is_empty(), "the change reached the other machine's apps"); + for bytes in &remote { + app_b.apply(bytes); + } + assert_eq!(app_b.text(), "hello world"); + let v = app_b.type_at(0, ">> "); + w.edit(b, doc, &v); + w.settle().await; + assert_eq!(w.text(a, doc), ">> hello world"); + assert_eq!(w.text(b, doc), ">> hello world"); + // A second open on a machine where it is live: from its copy, at once. + assert_eq!(w.open(a, node).1, Opening::Live); + }); +} + +/// Review: two machines open a never-live file at the same moment. One import wins, both end on the same document, and +/// the loser's app never saw its own import as live (it hears `joined` only once the loser has caught up with the +/// winner's). +#[test] +fn two_machines_importing_at_once_end_on_one_document() { + rt().block_on(async { + let mut w = World::new(Limits::default()).await; + let (a, b) = (w.machine().await, w.machine().await); + w.connect(a); + w.connect(b); + w.settle().await; + let node = w.file().await; + let (doc, _) = w.open(a, node); + w.open(b, node); + w.settle().await; + assert_eq!(w.events(a), vec![DocEvent::NeedsImport(doc)]); + assert_eq!(w.events(b), vec![DocEvent::NeedsImport(doc)]); + // Both imports in flight at once. + w.import(a, doc, "from a"); + w.import(b, doc, "from b"); + assert!(!w.machines[a].replica().has_document(doc) && !w.machines[b].replica().has_document(doc), "not live before the server says"); + w.settle().await; + let (ea, eb) = (w.events(a), w.events(b)); + assert!(ea.contains(&DocEvent::Joined(doc)) && eb.contains(&DocEvent::Joined(doc)), "{ea:?} {eb:?}"); + let (ta, tb) = (w.text(a, doc), w.text(b, doc)); + assert_eq!(ta, tb, "one document"); + assert!(ta == "from a" || ta == "from b", "{ta}"); + // The loser's text is nowhere: its import was dropped, not merged. + assert!(!(ta.contains("from a") && ta.contains("from b"))); + // And a third machine sees the same. + let c = w.machine().await; + w.connect(c); + w.settle().await; + joined(&mut w, c, node, "never asked").await; + assert_eq!(w.text(c, doc), ta); + }); +} + +#[test] +fn edits_made_offline_on_both_sides_merge_on_reconnect() { + rt().block_on(async { + let mut w = World::new(Limits::default()).await; + let (a, b) = (w.machine().await, w.machine().await); + w.connect(a); + w.connect(b); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "base").await; + joined(&mut w, b, node, "").await; + w.disconnect(a); + w.disconnect(b); + // Offline, both open it again from their copies and edit. + assert_eq!(w.open(a, node).1, Opening::Live); + assert_eq!(w.open(b, node).1, Opening::Live); + let (app_a, app_b) = (w.app_of(a, doc, 201), w.app_of(b, doc, 202)); + for i in 0..5 { + let u = app_a.append(&format!(" a{i}")); + w.edit(a, doc, &u); + let v = app_b.type_at(0, &format!("b{i} ")); + w.edit(b, doc, &v); + } + let (online, waiting) = w.machines[a].replica().document_status(doc).unwrap(); + assert_eq!((online, waiting), (false, 5), "offline, five changes waiting"); + // A crash before the document's file is written: the changes are in its journal. + w.crash(a); + w.connect(a); + w.connect(b); + w.settle().await; + let (ta, tb) = (w.text(a, doc), w.text(b, doc)); + assert_eq!(ta, tb); + for i in 0..5 { + assert!(ta.contains(&format!(" a{i}")) && ta.contains(&format!("b{i} ")), "{ta}"); + } + assert_eq!(w.machines[a].replica().document_status(doc).unwrap(), (true, 0)); + }); +} + +/// A file that has never been live opens as a plain file offline; one that has been opens from the copy. +#[test] +fn offline_a_never_live_file_is_a_plain_file() { + rt().block_on(async { + let mut w = World::new(Limits::default()).await; + let a = w.machine().await; + let node = w.file().await; + let (_, opening) = w.open(a, node); + assert!(matches!(opening, Opening::NotLive(NotLiveReason::Offline, _)), "{opening:?}"); + w.connect(a); + w.settle().await; + let doc = joined(&mut w, a, node, "x").await; + w.disconnect(a); + assert_eq!(w.open(a, node).1, Opening::Live); + w.stop(a); + assert_eq!(w.open(a, node).1, Opening::Live, "after a restart too"); + assert_eq!(w.text(a, doc), "x"); + // Joining when the link goes: the waiting app is told it is a plain file for now. + let other = w.file().await; + w.connect(a); + w.settle().await; + let (doc2, _) = w.open(a, other); + w.disconnect(a); + assert!(w.events(a).iter().any(|e| matches!(e, DocEvent::NotLive(d, NotLiveReason::Offline, _) if *d == doc2))); + assert!(!w.machines[a].replica().has_document(doc2), "never live: nothing kept"); + assert!(!w.machines[a].store.documents().contains_key(&doc2.to_string()), "and nothing on disk"); + }); +} + +/// Ruling: an update the agent refuses leaves the document as it was; a change it can't write to disk is taken back. +#[test] +fn a_refused_or_unwritten_change_leaves_the_document_as_it_was() { + rt().block_on(async { + let mut w = World::new(Limits::default()).await; + let a = w.machine().await; + w.connect(a); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "keep").await; + let before = w.machines[a].replica().document_state(doc).unwrap(); + for junk in [vec![1, 2, 3, 4, 5, 6, 7, 8], vec![0; 100], Vec::new()] { + assert!(w.machines[a].replica().local_update(doc, &junk).is_err()); + } + assert_eq!(w.machines[a].replica().document_state(doc).unwrap(), before); + assert!(w.machines[a].replica().local_update(doc, &vec![0; craft_cloud_proto::live::MAX_UPDATE_BYTES + 1]).is_err()); + let app = w.app_of(a, doc, 301); + let u = app.append("!"); + w.machines[a].store.fail_journal(true); + assert!(matches!(w.machines[a].replica().local_update(doc, &u), Err(craft_cloud_agent::live::replica::LocalError::Io(_)))); + assert_eq!(w.text(a, doc), "keep", "taken back"); + w.machines[a].store.fail_journal(false); + w.edit(a, doc, &u); + w.settle().await; + assert_eq!(w.text(a, doc), "keep!"); + }); +} + +/// A long log is compacted by an agent, at the server's asking; a machine opening the document later starts from +/// the snapshot and the log above it, and ends on the same document. +#[test] +fn a_long_log_is_compacted_and_a_new_machine_starts_from_the_snapshot() { + rt().block_on(async { + let mut w = World::new(small_logs()).await; + let (a, b) = (w.machine().await, w.machine().await); + w.connect(a); + w.connect(b); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "start").await; + joined(&mut w, b, node, "").await; + let (app_a, app_b) = (w.app_of(a, doc, 401), w.app_of(b, doc, 402)); + for i in 0..30 { + let u = app_a.append(&format!(" {i}")); + w.edit(a, doc, &u); + if i % 3 == 0 { + let v = app_b.set(&format!("k{i}"), i); + w.edit(b, doc, &v); + } + w.settle().await; + } + let found = w.meta.live_doc(w.space, doc).await.unwrap().unwrap(); + assert!(found.base > 0, "compacted: {found:?}"); + assert!(found.head - found.base <= 6, "and kept short: {found:?}"); + let c = w.machine().await; + w.connect(c); + w.settle().await; + joined(&mut w, c, node, "never asked").await; + assert!(w.machines[c].heard.iter().any(|f| matches!(f, ServerFrame::Snapshot { base, .. } if *base > 0)), "caught up from a snapshot"); + assert_eq!(w.text(c, doc), w.text(a, doc)); + assert_eq!(w.app_of(c, doc, 9).doc.get_deep_value(), w.app_of(a, doc, 9).doc.get_deep_value()); + assert_eq!(w.app_of(b, doc, 9).doc.get_deep_value(), w.app_of(a, doc, 9).doc.get_deep_value()); + // Its next `at` names the update at the snapshot's base (sent after it): a subscribe from there is checked. + let frames: Vec = { + let r = w.machines[c].replica(); + r.disconnected(); + r.connected(); + let welcome = ServerFrame::Welcome { protocol: 2, epoch: w.meta.live_epoch(w.space).await.unwrap() }; + r.on_frame(welcome).unwrap() + }; + assert!(frames.iter().all(|f| !matches!(f, ClientFrame::Subscribe { at: None, after, .. } if *after > 0)), "{frames:?}"); + }); +} + +/// Review: a snapshot that doesn't read on another machine (a device answered `compact` with junk). The machine that +/// can't read it says so and reads the import and the whole log instead; nothing is lost and it converges. +#[test] +fn a_snapshot_another_machine_cant_read_is_skipped_for_the_whole_log() { + rt().block_on(async { + let mut w = World::new(Limits { compact_updates: 3, ..Limits::default() }).await; + let a = w.machine().await; + w.connect(a); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "import").await; + let app = w.app_of(a, doc, 501); + // A rogue device of the account: it makes the log long and answers the compaction with junk. + let token = [99u8; 32]; + w.meta.create_device(w.user, "rogue", &token, 0).await.unwrap(); + let auth = w.meta.device_by_token(&token).await.unwrap().unwrap(); + let mut rogue = craft_cloud_server::live::Session::new(w.state.clone(), &auth); + rogue.handle(ClientFrame::Hello { min: 1, max: 2 }).await; + rogue.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + while rogue.has_work() { + rogue.pump().await.unwrap(); + } + let mut asked = None; + for i in 1..=5u64 { + let u = app.append(&format!(" r{i}")); + let reply = rogue.handle(ClientFrame::Update { doc, client_seq: i, bytes: u }).await; + asked = asked.or(reply.frames.iter().find_map(|f| if let ServerFrame::Compact { upto, .. } = f { Some(*upto) } else { None })); + } + let upto = asked.expect("the rogue was asked"); + let junk = w.upload(doc, &[0xde, 0xad, 0xbe, 0xef]).await; + rogue.handle(ClientFrame::Snapshot { doc, upto, manifest: junk }).await; + assert_eq!(w.meta.live_doc(w.space, doc).await.unwrap().unwrap().base, upto, "the junk is the snapshot now"); + w.settle().await; + // A fresh machine: the snapshot doesn't read; it reads the import and every row. + let b = w.machine().await; + w.connect(b); + w.settle().await; + joined(&mut w, b, node, "never asked").await; + assert!(w.machines[b].said.iter().any(|s| s.contains("doesn't read here")), "{:?}", w.machines[b].said); + assert!(w.machines[b].heard.iter().any(|f| matches!(f, ServerFrame::Snapshot { base: 0, .. })), "the import came"); + assert_eq!(w.text(b, doc), w.text(a, doc)); + assert!(w.text(b, doc).contains(" r5")); + // And it goes on taking edits. + let next = w.app_of(a, doc, 502).append(" more"); + w.edit(a, doc, &next); + w.settle().await; + assert_eq!(w.text(b, doc), w.text(a, doc)); + }); +} + +/// Review: a snapshot that reads but lacks some of what the rows below its base held (a device made it wrong): a machine +/// catching up from it finds updates waiting for that history, and reads the import and the whole log instead. +#[test] +fn a_snapshot_that_lacks_updates_is_caught_by_the_whole_log() { + rt().block_on(async { + let mut w = World::new(Limits { compact_updates: 3, ..Limits::default() }).await; + let a = w.machine().await; + w.connect(a); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "import").await; + let app = w.app_of(a, doc, 1501); + let token = [98u8; 32]; + w.meta.create_device(w.user, "rogue", &token, 0).await.unwrap(); + let auth = w.meta.device_by_token(&token).await.unwrap().unwrap(); + let mut rogue = craft_cloud_server::live::Session::new(w.state.clone(), &auth); + rogue.handle(ClientFrame::Hello { min: 1, max: 2 }).await; + rogue.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + while rogue.has_work() { + rogue.pump().await.unwrap(); + } + let mut asked = None; + for i in 1..=5u64 { + let u = app.append(&format!(" r{i}")); + let reply = rogue.handle(ClientFrame::Update { doc, client_seq: i, bytes: u }).await; + asked = asked.or(reply.frames.iter().find_map(|f| if let ServerFrame::Compact { upto, .. } = f { Some(*upto) } else { None })); + } + let upto = asked.expect("the rogue was asked"); + // A snapshot of the import only: it reads, but holds none of the updates up to `upto`. + let wrong = w.upload(doc, &App::new(1).with_text("import").snapshot()).await; + rogue.handle(ClientFrame::Snapshot { doc, upto, manifest: wrong }).await; + let more = app.append(" after"); + rogue.handle(ClientFrame::Update { doc, client_seq: 6, bytes: more }).await; + let b = w.machine().await; + w.connect(b); + w.settle().await; + joined(&mut w, b, node, "never asked").await; + assert!(w.machines[b].said.iter().any(|s| s.contains("reads the whole log")), "{:?}", w.machines[b].said); + assert_eq!(w.text(b, doc), "import r1 r2 r3 r4 r5 after"); + }); +} + +/// A compaction answered by a device that is behind: it answers only once it holds every update up to `upto`. +#[test] +fn a_device_behind_answers_compaction_only_once_it_holds_the_updates() { + rt().block_on(async { + let mut w = World::new(Limits { compact_updates: 3, ..Limits::default() }).await; + let (a, b) = (w.machine().await, w.machine().await); + w.connect(a); + w.connect(b); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "x").await; + joined(&mut w, b, node, "").await; + let app = w.app_of(a, doc, 601); + for i in 0..3 { + let u = app.append(&i.to_string()); + w.edit(a, doc, &u); + } + w.settle().await; + // B is asked for 4 while it holds 3 (the server's frame comes before the update it names). + let r = w.machines[b].replica(); + assert!(r.on_frame(ServerFrame::Compact { doc, upto: 4 }).unwrap().is_empty(), "not yet"); + let u = app.append("3"); + w.edit(a, doc, &u); + w.settle().await; + let answer = w.machines[b].said_up.iter().find_map(|f| match f { + ClientFrame::Snapshot { upto, manifest, .. } => Some((*upto, *manifest)), + _ => None, + }); + let (upto, manifest) = answer.expect("answered once the update came"); + assert_eq!(upto, 4); + let bytes = w.download(&manifest).await; + // A snapshot of exactly the updates 1-4: with the log above it, a fresh copy is the document. + let fresh = App::new(9); + fresh.apply(&bytes); + assert_eq!(fresh.text(), "x0123"); + let later = app.append("4"); + fresh.apply(&later); + assert_eq!(fresh.text(), "x01234"); + }); +} + +/// A snapshot made while this machine has a change of its own not yet in the log holds none of it (exactly the +/// updates up to `upto`). +#[test] +fn a_compaction_snapshot_holds_nothing_past_upto() { + rt().block_on(async { + let mut w = World::new(Limits::default()).await; + let a = w.machine().await; + w.connect(a); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "base").await; + let app = w.app_of(a, doc, 901); + let u = app.append(" acked"); + w.edit(a, doc, &u); + w.settle().await; + // Offline now: a change of its own, unsent. + w.disconnect(a); + let v = app.append(" unsent"); + w.edit(a, doc, &v); + w.connect(a); + // Asked to compact at 1 (the acked change) before its own goes up. + let r = w.machines[a].replica(); + let mut frames = Vec::new(); + frames.extend(r.on_frame(ServerFrame::Welcome { protocol: 2, epoch: w.meta.live_epoch(w.space).await.unwrap() }).unwrap()); + frames.extend(r.on_frame(ServerFrame::Update { doc, seq: 1, bytes: u.clone(), device: craft_cloud_proto::DeviceId::generate() }).unwrap()); + frames.extend(r.on_frame(ServerFrame::Subscribed { doc, head: 1 }).unwrap()); + frames.extend(r.on_frame(ServerFrame::Compact { doc, upto: 1 }).unwrap()); + let manifest = frames + .iter() + .find_map(|f| if let ClientFrame::Snapshot { upto: 1, manifest, .. } = f { Some(*manifest) } else { None }) + .expect("answered"); + let bytes = r.take_uploads().into_iter().find(|u| u.id() == manifest).expect("uploaded before it is named").bytes(); + let fresh = App::new(9); + fresh.apply(&bytes); + assert_eq!(fresh.text(), "base acked", "nothing past upto"); + }); +} + +#[test] +fn presence_goes_between_machines_and_is_never_stored() { + rt().block_on(async { + let mut w = World::new(Limits::default()).await; + let (a, b) = (w.machine().await, w.machine().await); + w.connect(a); + w.connect(b); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "x").await; + joined(&mut w, b, node, "").await; + let frames = w.machines[a].replica().presence(doc, b"cursor 3"); + w.send(a, frames); + w.settle().await; + assert!(w.events(b).contains(&DocEvent::Presence(doc, b"cursor 3".to_vec()))); + assert!(!w.events(a).iter().any(|e| matches!(e, DocEvent::Presence(..))), "not back to its sender"); + assert_eq!(w.meta.live_doc(w.space, doc).await.unwrap().unwrap().head, 0); + }); +} + +/// Past the device's cap on followed documents, the least recently used one no app has open is unfollowed; it stays +/// on disk and is followed again when opened. +#[test] +fn past_the_cap_the_least_recently_used_document_is_unfollowed() { + rt().block_on(async { + let mut w = World::new(Limits { live_burst: 100_000, ..Limits::default() }).await; + let a = w.machine().await; + w.connect(a); + w.settle().await; + let mut docs = Vec::new(); + let mut nodes = Vec::new(); + for i in 0..=MAX_SUBSCRIPTIONS { + let node = w.file().await; + let doc = joined(&mut w, a, node, &format!("d{i}")).await; + // Closed at once: no app has it open. + w.machines[a].replica().set_open(doc, false); + docs.push(doc); + nodes.push(node); + } + // (Each new file is first not found: that is how it is joined.) + let refused = w.machines[a].heard.iter().filter(|f| matches!(f, ServerFrame::Error { code, .. } if code != "not_found")).count(); + assert_eq!(refused, 0, "no subscribe was refused for the device's cap"); + // The first one went: reopening it follows it again, from its copy. + assert_eq!(w.open(a, nodes[0]).1, Opening::Live); + w.settle().await; + assert_eq!(w.text(a, docs[0]), "d0"); + let (online, _) = w.machines[a].replica().document_status(docs[0]).unwrap(); + assert!(online, "followed again"); + // (Unloaded too: past the cap on documents in memory as well.) + let followed = w.machines[a].replica().document_status(docs[1]).is_some_and(|(online, _)| online); + assert!(!followed, "the next least recent went instead"); + assert!(w.machines[a].replica().loaded_documents() <= craft_cloud_agent::live::replica::MAX_LOADED_DOCUMENTS); + }); +} + +/// Review: the server reset, or restored behind or ahead of a machine, with snapshots in play: the agent catches up +/// from the snapshot or pushes what the log lacks, and nothing any machine holds is lost. +#[test] +fn resets_and_restores_with_a_snapshot_lose_nothing() { + rt().block_on(async { + let mut w = World::new(small_logs()).await; + let (a, b) = (w.machine().await, w.machine().await); + w.connect(a); + w.connect(b); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "v").await; + joined(&mut w, b, node, "").await; + let (app_a, app_b) = (w.app_of(a, doc, 701), w.app_of(b, doc, 702)); + let mut typed = Vec::new(); + let edit = async |w: &mut World, m: usize, app: &App, s: String| { + let u = app.append(&s); + w.edit(m, doc, &u); + w.settle().await; + s + }; + for i in 0..12 { + typed.push(edit(&mut w, a, &app_a, format!(" a{i}")).await); + } + // A backup now: the log, its base and snapshot as they are. + let backup = w.meta.live_doc(w.space, doc).await.unwrap().unwrap(); + let backup_snapshot = w.meta.live_snapshot(w.space, doc).await.unwrap(); + assert!(backup.base > 0); + for i in 0..12 { + typed.push(edit(&mut w, b, &app_b, format!(" b{i}")).await); + } + assert!(w.meta.live_doc(w.space, doc).await.unwrap().unwrap().base > backup.head, "compacted past the backup"); + // Restored from the backup (both machines are ahead of it now), and its epoch reset as a restore does. + w.disconnect(a); + w.disconnect(b); + w.meta.restore_live_log(w.space, doc, Some(backup.head)).await.unwrap(); + w.meta.restore_live_snapshot(w.space, doc, backup_snapshot).await.unwrap(); + w.meta.reset_epoch(w.space).await.unwrap(); + w.connect(a); + w.connect(b); + w.settle().await; + let c = w.machine().await; + w.connect(c); + w.settle().await; + joined(&mut w, c, node, "never asked").await; + let text = w.text(c, doc); + for s in &typed { + assert!(text.contains(s.as_str()), "{s} lost after the restore: {text}"); + } + assert_eq!(w.text(a, doc), text); + assert_eq!(w.text(b, doc), text); + // The admin's reset: the log and the snapshot (the import too) go; the machines push it all back. A machine + // without a copy that opens it before they have is told to open the file (never handed an empty document). + for m in [a, b, c] { + w.disconnect(m); + } + w.meta.reset_live_doc(w.space, doc).await.unwrap(); + let d = w.machine().await; + w.connect(d); + w.settle().await; + let (_, opening) = w.open(d, node); + assert_eq!(opening, Opening::Joining); + w.settle().await; + let events = w.events(d); + assert!(events.iter().any(|e| matches!(e, DocEvent::NotLive(_, NotLiveReason::Resetting, _))), "{events:?}"); + for m in [a, b, c] { + w.connect(m); + } + w.settle().await; + joined(&mut w, d, node, "never asked").await; + assert_eq!(w.text(d, doc), text, "after the reset"); + // A restore from before the document was made at all: it is made again from a machine's copy. + for m in [a, b, c, d] { + w.disconnect(m); + } + w.meta.restore_live_log(w.space, doc, None).await.unwrap(); + assert!(w.meta.live_doc(w.space, doc).await.unwrap().is_none()); + w.meta.reset_epoch(w.space).await.unwrap(); + for m in [a, b, c, d] { + w.connect(m); + } + w.settle().await; + assert!(w.meta.live_doc(w.space, doc).await.unwrap().is_some(), "made again"); + let e = w.machine().await; + w.connect(e); + w.settle().await; + joined(&mut w, e, node, "never asked").await; + assert_eq!(w.text(e, doc), text, "after the document itself was lost"); + }); +} + +/// An agent against an older server (live protocol 1): recents go on, documents don't go live, and one that is live +/// here already keeps its changes on this machine. +#[test] +fn against_an_older_server_documents_wait_and_say_so() { + rt().block_on(async { + let mut w = World::new(Limits::default()).await; + let a = w.machine().await; + w.connect(a); + w.settle().await; + let node = w.file().await; + let doc = joined(&mut w, a, node, "x").await; + w.disconnect(a); + w.machines[a].old_server = true; + w.connect(a); + w.settle().await; + assert_eq!(w.machines[a].replica().documents_supported(), Some(false)); + let fresh = w.file().await; + let (_, opening) = w.open(a, fresh); + assert!(matches!(&opening, Opening::NotLive(NotLiveReason::Offline, m) if m.contains("newer")), "{opening:?}"); + assert_eq!(w.open(a, node).1, Opening::Live, "a document live here opens from its copy"); + let u = w.app_of(a, doc, 801).append("y"); + w.edit(a, doc, &u); + w.settle().await; + assert_eq!(w.machines[a].replica().document_status(doc), Some((false, 1)), "it waits"); + assert!(w.machines[a].said.iter().any(|s| s.contains("newer Craft Cloud server")), "{:?}", w.machines[a].said); + // Recents on: an app's list works as ever. + let app = craft_cloud_proto::live::AppName::new("gridcraft").unwrap(); + let entry = craft_cloud_proto::local::RecentEntry { + node: None, + path: "a.txt".into(), + title: "a".into(), + opened_ms: 1, + saved_ms: None, + machine: "pc".into(), + }; + let frames = w.machines[a].replica().add(&app, [1; 16], entry).unwrap(); + w.send(a, frames); + w.settle().await; + assert_eq!(w.machines[a].replica().waiting_for(&app), 0, "the add went up"); + // Back to a newer server: the change goes up. + w.disconnect(a); + w.machines[a].old_server = false; + w.connect(a); + w.settle().await; + assert_eq!(w.machines[a].replica().document_status(doc), Some((true, 0))); + }); +} diff --git a/apps/craft-cloud-agent/tests/documents_sim.rs b/apps/craft-cloud-agent/tests/documents_sim.rs new file mode 100644 index 0000000..e5f6b34 --- /dev/null +++ b/apps/craft-cloud-agent/tests/documents_sim.rs @@ -0,0 +1,603 @@ +//! Convergence simulation for documents of files: machines of one account, each the agent's own live [`Replica`] over a +//! store that outlasts its crashes, with an app stand-in editing one document (a plain text and a map) through it, +//! against the server's own live [`Session`]s over the real metadata database (SQLite in memory). The connections are +//! queues of frames, delivered in order but at random moments, so every interleaving of machines, catch-ups, fan-out +//! and compaction can come up. +//! +//! At random: apps type into the text and set keys of the map (each edit a token of its own), frames are delivered +//! either way, the server pumps a connection's log, connections drop (with whatever was in flight lost both ways) and +//! come back, machines crash (losing what they hadn't written, never what they told their app they took) and restart +//! from their store, documents are written now and then, apps close the document and open it again, presence goes +//! out, and the server's epoch is reset now and then (as after a restore). The log is compacted at random: a low limit +//! per seed, and requests that time out at once or never. Then every machine comes back, and everything is delivered +//! until nothing moves. Checks: +//! +//! 1. **Convergence**: every machine holds the same document, and so does every app (its own copy, built from the +//! state it opened with and the changes it was handed). +//! 2. **Nothing kept is lost**: every edit an agent took from its app (on its disk before the app heard so) is in it. +//! 3. **Nothing waits**: no machine has a change the server didn't ack. +//! +//! A seed replays exactly: `SIM_SEED=17 cargo test -p craft-cloud-agent --test documents_sim`; `SIM_SEEDS=10000 +//! SIM_FIRST=0` runs many (use `--release`), on `SIM_JOBS` workers. `SIM_TRACE=1` prints every step. + +#[path = "docs_common/mod.rs"] +mod common; + +use std::collections::BTreeSet; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; + +use common::{App, World}; +use craft_cloud_agent::live::replica::{DocEvent, Opening}; +use craft_cloud_proto::DocId; +use craft_cloud_proto::live::{ClientFrame, ServerFrame}; +use craft_cloud_server::state::Limits; + +struct Rng(u64); + +impl Rng { + fn new(seed: u64) -> Self { + Self(seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1) + } + + fn next(&mut self) -> u64 { + self.0 ^= self.0 >> 12; + self.0 ^= self.0 << 25; + self.0 ^= self.0 >> 27; + self.0.wrapping_mul(0x2545_F491_4F6C_DD1D) + } + + fn below(&mut self, n: usize) -> usize { + (self.next() % n.max(1) as u64) as usize + } + + fn chance(&mut self, percent: u64) -> bool { + self.next() % 100 < percent + } +} + +/// A frame as the replay log has it: what it is, without IDs (they are made afresh by every run). +fn up_text(frame: &ClientFrame) -> String { + match frame { + ClientFrame::Hello { .. } => "hello".into(), + ClientFrame::Create { .. } => "create".into(), + ClientFrame::Subscribe { after, .. } => format!("subscribe after {after}"), + ClientFrame::Unsubscribe { .. } => "unsubscribe".into(), + ClientFrame::Update { client_seq, bytes, .. } => format!("update {client_seq} ({} bytes)", bytes.len()), + ClientFrame::Snapshot { upto, .. } => format!("snapshot upto {upto}"), + ClientFrame::Presence { .. } => "presence".into(), + ClientFrame::SubscribeLog { after, .. } => format!("subscribe the log after {after}"), + ClientFrame::CreateDocument { snapshot, .. } => format!("create document (manifest {snapshot})"), + } +} + +fn down_text(frame: &ServerFrame) -> String { + match frame { + ServerFrame::Welcome { protocol, .. } => format!("welcome {protocol}"), + ServerFrame::Created { .. } => "created".into(), + ServerFrame::Update { seq, bytes, .. } => format!("update seq {seq} ({} bytes)", bytes.len()), + ServerFrame::Subscribed { head, .. } => format!("subscribed head {head}"), + ServerFrame::Ack { client_seq, seq, .. } => format!("ack {client_seq} as {seq}"), + ServerFrame::Error { code, client_seq, .. } => format!("error {code} {client_seq:?}"), + ServerFrame::Compact { upto, .. } => format!("compact upto {upto}"), + ServerFrame::Snapshot { base, .. } => format!("snapshot base {base}"), + ServerFrame::Presence { .. } => "presence".into(), + ServerFrame::CreatedDocument { kept, .. } => format!("created, kept {kept}"), + } +} + +/// One machine's app: its own copy of the document, while it has it open. +struct AppState { + app: Option, + /// Waiting for the document to be joined (it asked to open it). + opening: bool, + peers: u64, +} + +struct Sim { + w: World, + rng: Rng, + doc: Option, + node: craft_cloud_proto::NodeId, + apps: Vec, + /// Every edit an agent took: its token. + taken: BTreeSet, + edits: u64, + log: Vec, + trace: bool, + /// The last update or ack each machine was sent (to send again). + last_down: Vec>, + /// A backup of the document's log: its head, base and snapshot. + backup: Option<(u64, Option<(u64, craft_cloud_meta::SnapshotRef)>)>, +} + +const MACHINES: usize = 3; + +impl Sim { + fn say(&mut self, line: String) { + if self.trace { + eprintln!("{line}"); + } + self.log.push(line); + } + + /// Hands the machine's events to its app. + fn events(&mut self, m: usize) { + let events = self.w.events(m); + for event in events { + match event { + DocEvent::Joined(doc) => { + self.doc = Some(doc); + if self.apps[m].opening { + self.apps[m].opening = false; + self.open_app(m, doc); + } + } + DocEvent::NeedsImport(doc) => { + self.say(format!("m{m} imports")); + let snapshot = App::new(9_000 + m as u64).with_text("start;").snapshot(); + if let Ok(frames) = self.w.machines[m].replica().import_document(doc, &snapshot) { + self.w.send(m, frames); + } + } + DocEvent::NotLive(..) => { + self.say(format!("m{m} not live for now")); + self.apps[m].opening = false; + } + DocEvent::Remote(_, bytes) => { + if let Some(app) = &self.apps[m].app { + // (Loro takes updates in any order; one that waits for its history applies when it comes.) + app.doc.import(&bytes).unwrap(); + } + } + DocEvent::Presence(..) => {} + } + } + } + + /// The machine's app opens the document as its agent has it now. + fn open_app(&mut self, m: usize, doc: DocId) { + self.apps[m].peers += 1; + let peer = ((m as u64 + 1) << 32) + self.apps[m].peers; + let app = self.w.app_of(m, doc, peer); + self.w.machines[m].replica().set_open(doc, true); + self.apps[m].app = Some(app); + } + + fn close_app(&mut self, m: usize) { + if self.apps[m].app.take().is_some() + && let Some(doc) = self.doc + && let Some(replica) = self.w.machines[m].replica.as_mut() + { + replica.set_open(doc, false); + } + } + + /// The machine's app opens the file. + fn open(&mut self, m: usize) { + if self.apps[m].app.is_some() || self.apps[m].opening { + return; + } + let (doc, opening) = self.w.open(m, self.node); + match opening { + Opening::Live => { + self.say(format!("m{m} opens it from its copy")); + self.open_app(m, doc); + } + Opening::Joining => { + self.say(format!("m{m} joins it")); + self.apps[m].opening = true; + } + Opening::NotLive(..) => self.say(format!("m{m} can't open it now")), + } + } + + /// An edit by the machine's app: a token typed into the text, or a key of the map. + fn edit(&mut self, m: usize) { + let Some(doc) = self.doc else { return }; + let Some(app) = &self.apps[m].app else { return }; + self.edits += 1; + let token = format!("m{m}e{};", self.edits); + let update = if self.rng.chance(70) { + // Between tokens (every token ends with `;`), so none is cut in two. + let text = app.doc.get_text("text").to_string(); + let mut places = vec![0]; + places.extend(text.chars().enumerate().filter(|(_, c)| *c == ';').map(|(i, _)| i + 1)); + let at = places[self.rng.below(places.len())]; + app.type_at(at, &token) + } else { + app.set(&token, self.edits as i64) + }; + match self.w.machines[m].replica().local_update(doc, &update) { + Ok(frames) => { + self.taken.insert(token.clone()); + self.say(format!("m{m} edits {token} ({} frames)", frames.len())); + self.w.send(m, frames); + } + Err(e) => { + // Not taken: the app's own copy has it, the agent's doesn't. It opens the document again. + self.say(format!("m{m}'s edit refused: {e}")); + self.close_app(m); + } + } + self.events(m); + } + + /// One random step. + async fn step(&mut self) { + let m = self.rng.below(MACHINES); + match self.rng.below(100) { + 0..=24 => self.edit(m), + 25..=44 => { + // One frame up (the snapshot it may name goes up before it, as the hub sends it). + let api = self.w.blobs(); + let machine = &mut self.w.machines[m]; + if let Some(replica) = machine.replica.as_mut() + && machine.conn.is_some() + { + for up in replica.take_uploads() { + craft_cloud_agent::live::blobs::upload(&api, &up).await.unwrap(); + } + } + if let (Some(conn), Some(frame)) = (machine.conn.as_mut(), machine.up.pop_front()) { + let text = up_text(&frame); + let reply = conn.session.handle(frame).await; + machine.down.extend(reply.frames); + self.say(format!("m{m} up {text}")); + } + } + 45..=69 => { + // One frame down. + let machine = &mut self.w.machines[m]; + if let (Some(frame), Some(replica)) = (machine.down.pop_front(), machine.replica.as_mut()) { + let text = down_text(&frame); + self.last_down[m] = matches!(frame, ServerFrame::Update { .. } | ServerFrame::Ack { .. }).then(|| frame.clone()); + match replica.on_frame(frame) { + Ok(frames) => { + machine.up.extend(frames); + // A snapshot a catch-up starts with is downloaded (as the hub does). + for (doc, manifest) in replica.take_fetches() { + let got = craft_cloud_agent::live::blobs::download(&self.w.blobs(), &manifest).await.map_err(|e| e.to_string()); + let machine = &mut self.w.machines[m]; + if let Some(replica) = machine.replica.as_mut() { + let frames = replica.on_snapshot(doc, manifest, got); + machine.up.extend(frames); + } + } + } + Err(e) => { + self.say(format!("m{m} link ends: {e}")); + self.w.disconnect(m); + } + } + self.say(format!("m{m} down {text}")); + self.after(m); + } + } + 70..=79 => { + // The server hears of its space and pumps a page. + let machine = &mut self.w.machines[m]; + if let Some(conn) = machine.conn.as_mut() { + while let Ok(doc) = conn.notes.try_recv() { + conn.session.note(doc); + } + while let Ok(note) = conn.presence.try_recv() { + machine.down.extend(conn.session.relay(¬e)); + } + if conn.session.has_work() { + let frames = conn.session.pump().await.unwrap(); + let n = frames.len(); + machine.down.extend(frames); + self.say(format!("m{m} pump {n}")); + } + } + } + 80..=84 => { + // A tick of the agent. + if self.w.machines[m].conn.is_some() { + let frames = self.w.machines[m].replica().resend(); + self.w.send(m, frames); + self.after(m); + } + } + 85..=86 => { + if self.w.machines[m].conn.is_some() { + self.say(format!("m{m} drops its connection")); + self.w.disconnect(m); + self.after(m); + } else { + self.say(format!("m{m} connects")); + self.w.connect(m); + } + } + 87 => { + self.say(format!("m{m} crashes")); + self.close_app(m); + self.apps[m].opening = false; + self.w.crash(m); + } + 88..=89 => { + // Time passes: documents are written whole. + if let Some(replica) = self.w.machines[m].replica.as_mut() { + replica.flush_documents().unwrap(); + } + } + 90..=93 => { + if self.apps[m].app.is_some() && self.rng.chance(30) { + self.say(format!("m{m}'s app closes it")); + self.close_app(m); + } else { + self.open(m); + } + self.after(m); + } + 94..=95 => { + if let Some(doc) = self.doc + && self.apps[m].app.is_some() + { + let frames = self.w.machines[m].replica().presence(doc, b"cursor"); + self.w.send(m, frames); + } + } + 96 => { + if let Some(doc) = self.doc + && self.rng.chance(30) + { + // A backup of the database now (the log's head, its base and snapshot). + if let Some(found) = self.w.meta.live_doc(self.w.space, doc).await.unwrap() { + let snapshot = self.w.meta.live_snapshot(self.w.space, doc).await.unwrap(); + self.say(format!("a backup at head {}", found.head)); + self.backup = Some((found.head, snapshot)); + } + } else if self.rng.chance(20) { + // A restore from the backup (if there is one), and its step: the epoch is reset; every connection + // ends at its next ping. + if let (Some(doc), Some((head, snapshot))) = (self.doc, self.backup.take()) { + self.say(format!("the server is restored to head {head}")); + self.w.meta.restore_live_log(self.w.space, doc, Some(head)).await.unwrap(); + self.w.meta.restore_live_snapshot(self.w.space, doc, snapshot).await.unwrap(); + } + self.say("the server's epoch is reset".into()); + self.w.meta.reset_epoch(self.w.space).await.unwrap(); + for i in 0..MACHINES { + if self.w.machines[i].conn.is_some() { + self.w.disconnect(i); + self.after(i); + } + } + } + } + _ => { + // An update or ack the server sent comes twice (as from a connection given up on whose last frames + // still arrive). + let machine = &mut self.w.machines[m]; + if let (Some(frame), Some(replica)) = (self.last_down[m].clone(), machine.replica.as_mut()) + && machine.conn.is_some() + { + let text = down_text(&frame); + if let Ok(frames) = replica.on_frame(frame) { + machine.up.extend(frames); + } + self.say(format!("m{m} again {text}")); + self.after(m); + } + } + } + } + + /// After the replica did something: writes what it must, says its notes, hands its events to the app. + fn after(&mut self, m: usize) { + if let Some(replica) = self.w.machines[m].replica.as_mut() { + replica.flush().unwrap(); + let notes = replica.take_notes(); + self.w.machines[m].said.extend(notes); + } + self.events(m); + } + + /// What the server's log and snapshot hold of the document, in words (for a failure to say). + async fn server_version(&mut self, doc: DocId) -> String { + let (space, meta) = (self.w.space, self.w.meta.clone()); + let rows = meta.live_updates(space, doc, 0, 1000).await.unwrap_or_default(); + let found = meta.live_doc(space, doc).await.ok().flatten(); + let mut ranges = Vec::new(); + for row in &rows { + let d = loro::LoroDoc::decode_import_blob_meta(&row.bytes, true).unwrap(); + ranges.push((row.seq, format!("{:?}", d.partial_end_vv))); + } + format!("{found:?}; rows {ranges:?}") + } + + /// Everything comes back and is delivered until nothing moves. + async fn finish(&mut self) { + for m in 0..MACHINES { + if self.w.machines[m].conn.is_none() { + self.w.connect(m); + } + } + for round in 0..200 { + self.w.settle().await; + for m in 0..MACHINES { + self.events(m); + if self.apps[m].app.is_none() && !self.apps[m].opening { + self.open(m); + } + self.events(m); + } + let all_open = self.apps.iter().all(|a| a.app.is_some()); + let quiet = self.w.machines.iter_mut().all(|machine| machine.up.is_empty() && machine.down.is_empty()); + if all_open && quiet && round > 2 { + let waiting: u64 = + (0..MACHINES).map(|m| self.doc.and_then(|d| self.w.machines[m].replica().document_status(d)).map_or(1, |s| s.1)).sum(); + if waiting == 0 { + return; + } + } + for m in 0..MACHINES { + let frames = self.w.machines[m].replica().resend(); + self.w.send(m, frames); + } + } + } +} + +async fn run(seed: u64) -> Vec { + let mut rng = Rng::new(seed); + let compact_updates = 2 + rng.below(30) as u64; + let compact_timeout = if rng.chance(50) { Duration::ZERO } else { Duration::from_secs(3600) }; + let limits = + Limits { compact_updates, compact_timeout, live_burst: 1_000_000, live_per_sec: 1_000_000, presence_per_sec: 1_000_000, ..Limits::default() }; + let mut w = World::new(limits).await; + for _ in 0..MACHINES { + w.machine().await; + } + let node = w.file().await; + for m in 0..MACHINES { + w.connect(m); + } + let trace = std::env::var("SIM_TRACE").is_ok(); + let apps = (0..MACHINES).map(|_| AppState { app: None, opening: false, peers: 0 }).collect(); + let mut sim = Sim { + w, + rng, + doc: None, + node, + apps, + taken: BTreeSet::new(), + edits: 0, + log: Vec::new(), + trace, + last_down: vec![None; MACHINES], + backup: None, + }; + sim.say(format!("seed {seed}: compacting past {compact_updates} updates, asks timing out {compact_timeout:?}")); + sim.open(0); + let steps = 300 + sim.rng.below(900); + for _ in 0..steps { + sim.step().await; + } + sim.finish().await; + check(&mut sim, seed).await; + sim.log +} + +async fn check(sim: &mut Sim, seed: u64) { + let doc = sim.doc.unwrap_or_else(|| panic!("seed {seed}: the document was never made")); + let first = sim.w.app_of(0, doc, 1).doc.get_deep_value(); + for m in 0..MACHINES { + assert!(sim.w.machines[m].replica().has_document(doc), "seed {seed}: m{m} never joined: {:?}", sim.w.machines[m].said); + let held = sim.w.app_of(m, doc, 1).doc.get_deep_value(); + if held != first { + let heard: Vec = sim.w.machines[m].heard.iter().rev().take(40).rev().map(down_text).collect(); + let sent: Vec = sim.w.machines[m].said_up.iter().rev().take(40).rev().map(up_text).collect(); + let said = sim.w.machines[m].said.clone(); + let status = sim.w.machines[m].replica().document_status(doc); + let waiting = sim.w.machines[m].replica().waiting_bytes(); + let vv = |sim: &mut Sim, i: usize| { + let snapshot = sim.w.machines[i].replica().snapshot_of(doc).unwrap(); + let d = loro::LoroDoc::new(); + d.import(&snapshot).unwrap(); + d.oplog_vv() + }; + let (mine, theirs) = (vv(sim, m), vv(sim, 0)); + // What the server holds: every row, and its snapshot. + let server = sim.server_version(doc).await; + panic!( + "seed {seed}: m{m} holds another document; it said {said:?}; heard last {heard:?}; sent last {sent:?}; status {status:?}; waiting {waiting} bytes; vv {mine:?} vs m0 {theirs:?}; the server: {server}" + ); + } + let app = sim.apps[m].app.as_ref().unwrap_or_else(|| panic!("seed {seed}: m{m}'s app has nothing open")); + if app.doc.get_deep_value() != first { + let shown = format!("{:?}", app.doc.get_deep_value()); + let missing: Vec<&String> = sim.taken.iter().filter(|t| !shown.contains(t.as_str())).collect(); + panic!("seed {seed}: m{m}'s app shows another document: it lacks {missing:?}; m{m} said {:?}", sim.w.machines[m].said); + } + let (_, waiting) = sim.w.machines[m].replica().document_status(doc).unwrap(); + assert_eq!(waiting, 0, "seed {seed}: m{m} has {waiting} change(s) the server never acked"); + } + let view = sim.w.app_of(0, doc, 1); + let text = view.text(); + let keys: BTreeSet = { + let mut keys = BTreeSet::new(); + view.doc.get_map("map").for_each(|k, _| { + keys.insert(k.to_owned()); + }); + keys + }; + for token in &sim.taken { + assert!(text.contains(token.as_str()) || keys.contains(token), "seed {seed}: the edit {token} was taken and is lost"); + } + // A machine that never had it joins now, from the server's last snapshot and the rows above it: the same + // document (a snapshot that left out a row below its base would show here). + let fresh = sim.w.machine().await; + sim.w.connect(fresh); + sim.w.settle().await; + let (_, opening) = sim.w.open(fresh, sim.node); + assert_eq!(opening, Opening::Joining, "seed {seed}"); + sim.w.settle().await; + let joined = sim.w.events(fresh); + assert!(joined.contains(&DocEvent::Joined(doc)), "seed {seed}: the fresh machine didn't join: {joined:?}"); + let late = sim.w.app_of(fresh, doc, 1).doc.get_deep_value(); + assert_eq!(late, first, "seed {seed}: a machine joining at the end holds another document; the server: {}", sim.server_version(doc).await); +} + +fn for_each_seed(default_seeds: u64, run: impl Fn(u64) + Sync) { + if let Ok(seed) = std::env::var("SIM_SEED") { + run(seed.parse().unwrap()); + return; + } + let seeds: u64 = std::env::var("SIM_SEEDS").ok().and_then(|s| s.parse().ok()).unwrap_or(default_seeds); + let first: u64 = std::env::var("SIM_FIRST").ok().and_then(|s| s.parse().ok()).unwrap_or(0); + let jobs: usize = std::env::var("SIM_JOBS") + .ok() + .and_then(|s| s.parse().ok()) + .unwrap_or_else(|| std::thread::available_parallelism().map_or(1, |n| n.get().min(8))) + .max(1); + let next = AtomicU64::new(first); + let failures: std::sync::Mutex> = std::sync::Mutex::default(); + std::thread::scope(|scope| { + for _ in 0..jobs { + scope.spawn(|| { + loop { + let seed = next.fetch_add(1, Ordering::Relaxed); + if seed >= first + seeds { + return; + } + if std::env::var("SIM_SAY").is_ok() { + eprintln!("start {seed}"); + } + let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| run(seed))); + if let Err(payload) = result { + let message = payload.downcast_ref::().cloned().or_else(|| payload.downcast_ref::<&str>().map(|s| (*s).to_owned())); + failures.lock().unwrap().push((seed, message.unwrap_or_default())); + } + } + }); + } + }); + let mut failures = failures.into_inner().unwrap(); + failures.sort(); + if let Some((seed, message)) = failures.first() { + let others: Vec = failures.iter().skip(1).map(|(s, _)| s.to_string()).collect(); + panic!("seed {seed}: {message}{}", if others.is_empty() { String::new() } else { format!("\n(seeds {} failed too)", others.join(", ")) }); + } +} + +fn block_on(f: impl std::future::Future) -> T { + tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap().block_on(f) +} + +#[test] +fn documents_converge_without_losing_an_edit() { + for_each_seed(30, |seed| { + block_on(run(seed)); + }); +} + +#[test] +fn a_seed_replays_exactly() { + for seed in 0..10 { + let (a, b) = (block_on(run(seed)), block_on(run(seed))); + assert_eq!(a, b, "seed {seed} ran two ways"); + } +} diff --git a/apps/craft-cloud-agent/tests/live_documents.rs b/apps/craft-cloud-agent/tests/live_documents.rs new file mode 100644 index 0000000..0bc32af --- /dev/null +++ b/apps/craft-cloud-agent/tests/live_documents.rs @@ -0,0 +1,583 @@ +//! Live documents end to end: running agents of one account against a real server, the files synced by the file sync, +//! driven through `craft-cloud-app`'s live documents as an app would. A file opened on one machine is live on the +//! other; two machines importing a never-live file at once end on one document; a file renamed, moved or deleted on +//! another machine keeps its open document (and says when it was deleted); and an app that misbehaves on the local +//! API is refused or dropped while the agent and other apps go on. + +mod looping; + +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use craft_cloud_app::live::{Document, Documents, Event, NotLiveReason, Opened}; +use looping::{Agent, Server, eventually}; +use loro::{ExportMode, LoroDoc}; + +/// Blocking work off the test's runtime. +async fn off(f: impl FnOnce() -> T + Send + 'static) -> T { + tokio::task::spawn_blocking(f).await.unwrap() +} + +fn text_snapshot(text: &str) -> Vec { + craft_cloud_agent::doc::plain_text_snapshot(text).unwrap() +} + +/// An app's Loro document for an open live document. +struct AppDoc { + loro: LoroDoc, + doc: Document, +} + +impl AppDoc { + fn new(doc: Document, peer: u64) -> Self { + let loro = LoroDoc::new(); + loro.set_peer_id(peer).unwrap(); + loro.import(doc.state()).unwrap(); + Self { loro, doc } + } + + fn text(&self) -> String { + self.loro.get_text("text").to_string() + } + + /// Applies what came; the other events. + fn pump(&self) -> Vec { + let mut others = Vec::new(); + while let Some(event) = self.doc.try_recv() { + match event { + Event::Update(bytes) => { + self.loro.import(&bytes).unwrap(); + } + other => others.push(other), + } + } + others + } + + /// Types at the end and sends it: its number. + fn append(&self, s: &str) -> u64 { + let before = self.loro.oplog_vv(); + let t = self.loro.get_text("text"); + t.insert(t.len_unicode(), s).unwrap(); + self.loro.commit(); + self.doc.send(&self.loro.export(ExportMode::updates(&before)).unwrap()).unwrap() + } +} + +/// Opens `path`, importing `content` if the file has never been live. +fn open(docs: &Documents, path: &str, content: &str) -> Document { + match docs.open(path, "wordcraft").unwrap() { + Opened::Live(doc) => doc, + Opened::NeedsImport(importer) => importer.import(&text_snapshot(content)).unwrap(), + Opened::NotLive(reason) => panic!("not live: {reason:?}"), + } +} + +/// Waits for `path` to be synced to `agent`'s folder. +async fn synced(agent: &Agent, path: &str) { + let file = agent.folder.join(path); + eventually(&format!("{path} reached {}", agent.folder.display()), move || file.exists().then_some(())).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_file_opened_on_one_machine_is_live_on_the_other() { + let server = Server::new().await; + let (pc, laptop) = (Agent::signed_in(&server, "pc").await, Agent::signed_in(&server, "laptop").await); + let (run_pc, run_laptop) = (pc.run(), laptop.run()); + let (on_pc, on_laptop) = (pc.client().await, laptop.client().await); + std::fs::create_dir_all(&pc.folder).unwrap(); + std::fs::write(pc.folder.join("notes.txt"), "from the file").unwrap(); + synced(&laptop, "notes.txt").await; + // The pc opens it first: it imports it. + let c = on_pc.clone(); + let (docs_pc, app_pc) = off(move || { + let docs = c.documents().unwrap(); + let doc = open(&docs, "notes.txt", "from the file"); + (docs, AppDoc::new(doc, 11)) + }) + .await; + assert_eq!(app_pc.text(), "from the file"); + // The laptop joins it. + let c = on_laptop.clone(); + let (docs_laptop, app_laptop) = off(move || { + let docs = c.documents().unwrap(); + let doc = open(&docs, "notes.txt", "never used"); + (docs, AppDoc::new(doc, 12)) + }) + .await; + assert_eq!(app_laptop.text(), "from the file"); + // A keystroke on one shows up in the other's open document. + let (app_pc, app_laptop) = (Arc::new(Mutex::new(app_pc)), Arc::new(Mutex::new(app_laptop))); + let seq = app_pc.lock().unwrap().append(", typed on the pc"); + let a = app_pc.clone(); + eventually("the pc's change is acked", move || a.lock().unwrap().pump().contains(&Event::Acked(seq)).then_some(())).await; + let l = app_laptop.clone(); + eventually("the laptop shows it", move || { + let app = l.lock().unwrap(); + app.pump(); + (app.text() == "from the file, typed on the pc").then_some(()) + }) + .await; + // Presence goes across too. + app_laptop.lock().unwrap().doc.presence(b"cursor 4").unwrap(); + let a = app_pc.clone(); + eventually("the pc sees the laptop's cursor", move || a.lock().unwrap().pump().contains(&Event::Presence(b"cursor 4".to_vec())).then_some(())) + .await; + // Two apps on one machine share it at once (without the server's round trip). + let c = on_laptop.clone(); + let second = off(move || { + let docs = c.documents().unwrap(); + let doc = open(&docs, "notes.txt", "never used"); + (docs, AppDoc::new(doc, 13)) + }) + .await; + assert_eq!(second.1.text(), "from the file, typed on the pc"); + app_laptop.lock().unwrap().append("!"); + let s = Arc::new(Mutex::new(second)); + let s2 = s.clone(); + eventually("the other app on the laptop gets it", move || { + let app = &s2.lock().unwrap().1; + app.pump(); + app.text().ends_with('!').then_some(()) + }) + .await; + drop((docs_pc, docs_laptop, s)); + run_pc.stop().await.0.unwrap(); + run_laptop.stop().await.0.unwrap(); +} + +/// Snapshots go through the chunk API: the import goes up before `create` names it, a compaction's snapshot before +/// `snapshot` names it, and a machine that never had the document downloads the server's snapshot before the log +/// above it. The live connection never carries one. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn snapshots_go_up_and_down_through_the_chunk_api() { + let server = Server::with_limits(craft_cloud_server::state::Limits { compact_updates: 3, ..Default::default() }).await; + let pc = Agent::signed_in(&server, "pc").await; + let run_pc = pc.run(); + let on_pc = pc.client().await; + std::fs::create_dir_all(&pc.folder).unwrap(); + std::fs::write(pc.folder.join("long.txt"), "start").unwrap(); + let c = on_pc.clone(); + eventually("the file is synced", move || { + let docs = c.documents().ok()?; + matches!(docs.open("long.txt", "wordcraft").ok()?, Opened::NeedsImport(_)).then_some(()) + }) + .await; + let c = on_pc.clone(); + let (docs_pc, app) = off(move || { + let docs = c.documents().unwrap(); + let doc = open(&docs, "long.txt", "start"); + (docs, AppDoc::new(doc, 21)) + }) + .await; + let app = Arc::new(Mutex::new(app)); + let mut last = 0; + for i in 0..12 { + last = app.lock().unwrap().append(&format!(" {i}")); + } + let a = app.clone(); + eventually("every change acked", move || a.lock().unwrap().pump().contains(&Event::Acked(last)).then_some(())).await; + let node = server_node(&server, "long.txt").await.unwrap(); + let space = server.meta.user_by_name("ana").await.unwrap().unwrap().space; + let doc = craft_cloud_proto::live::node_doc(space, node); + let meta = server.meta.clone(); + let mut compacted = 0; + for _ in 0..600 { + compacted = meta.live_doc(space, doc).await.unwrap().map_or(0, |d| d.base); + if compacted > 0 { + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + } + assert!(compacted > 0, "the pc's snapshot was taken"); + let (_, snapshot) = meta.live_snapshot(space, doc).await.unwrap().unwrap(); + assert!(meta.has_manifest(space, &snapshot.manifest).await.unwrap(), "uploaded as the account's"); + // A machine that never had it: from the snapshot (downloaded) and the log above it. + let laptop = Agent::signed_in(&server, "laptop").await; + let run_laptop = laptop.run(); + let on_laptop = laptop.client().await; + synced(&laptop, "long.txt").await; + let c = on_laptop.clone(); + let (docs_laptop, joined) = off(move || { + let docs = c.documents().unwrap(); + let doc = open(&docs, "long.txt", "never used"); + (docs, AppDoc::new(doc, 22)) + }) + .await; + assert_eq!(joined.text(), "start 0 1 2 3 4 5 6 7 8 9 10 11"); + drop((docs_pc, docs_laptop, joined, app)); + run_pc.stop().await.0.unwrap(); + run_laptop.stop().await.0.unwrap(); +} + +/// A file not uploaded yet: the open starts a sync pass and waits for the file's node. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_file_not_uploaded_yet_is_synced_for_the_open() { + let server = Server::new().await; + let pc = Agent::signed_in(&server, "pc").await; + let running = pc.run(); + let on_pc = pc.client().await; + let c = on_pc.clone(); + eventually("online", move || c.status().ok()?.online.then_some(())).await; + std::fs::create_dir_all(&pc.folder).unwrap(); + std::fs::write(pc.folder.join("fresh.txt"), "new").unwrap(); + let c = on_pc.clone(); + let opened = off(move || { + let docs = c.documents().unwrap(); + let opened = docs.open("fresh.txt", "wordcraft").unwrap(); + matches!(opened, Opened::NeedsImport(_)) + }) + .await; + assert!(opened, "found once synced, and asked for its import"); + // A path outside the folder, or of no file. + let c = on_pc.clone(); + let (outside, missing) = off(move || { + let docs = c.documents().unwrap(); + (docs.open("../outside.txt", "wordcraft").unwrap(), docs.open("nothing-here.txt", "wordcraft").unwrap()) + }) + .await; + assert!(matches!(outside, Opened::NotLive(NotLiveReason::NotInFolder)), "{outside:?}"); + assert!(matches!(missing, Opened::NotLive(NotLiveReason::NotInFolder)), "{missing:?}"); + running.stop().await.0.unwrap(); +} + +/// Review: two machines open a never-live file at once and both import it: one import wins, both end on the same +/// document, and neither app is told its document is live before that is settled. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn two_machines_importing_a_file_at_once_end_on_one_document() { + let server = Server::new().await; + let (pc, laptop) = (Agent::signed_in(&server, "pc").await, Agent::signed_in(&server, "laptop").await); + let (run_pc, run_laptop) = (pc.run(), laptop.run()); + let (on_pc, on_laptop) = (pc.client().await, laptop.client().await); + std::fs::create_dir_all(&pc.folder).unwrap(); + std::fs::write(pc.folder.join("race.txt"), "x").unwrap(); + synced(&laptop, "race.txt").await; + let (c1, c2) = (on_pc.clone(), on_laptop.clone()); + // Both opens first (both find no document), then both imports at once. + let (o1, o2) = tokio::join!( + off(move || { + let docs = c1.documents().unwrap(); + let opened = docs.open("race.txt", "wordcraft").unwrap(); + (docs, opened) + }), + off(move || { + let docs = c2.documents().unwrap(); + let opened = docs.open("race.txt", "wordcraft").unwrap(); + (docs, opened) + }) + ); + let import = |(docs, opened): (Documents, Opened), content: &'static str| { + off(move || { + let doc = match opened { + Opened::NeedsImport(importer) => importer.import(&text_snapshot(content)).unwrap(), + Opened::Live(doc) => doc, + Opened::NotLive(reason) => panic!("{reason:?}"), + }; + (docs, AppDoc::new(doc, 21)) + }) + }; + let (a, b) = tokio::join!(import(o1, "the pc's import"), import(o2, "the laptop's import")); + assert_eq!(a.1.doc.id(), b.1.doc.id(), "one document"); + assert_eq!(a.1.text(), b.1.text(), "the same import on both"); + assert!(a.1.text() == "the pc's import" || a.1.text() == "the laptop's import"); + drop((a, b)); + run_pc.stop().await.0.unwrap(); + run_laptop.stop().await.0.unwrap(); +} + +/// Review: a file renamed, moved or deleted on another machine while it is open here. A rename or move keeps the +/// document (it belongs to the file's node); a delete leaves it working and says so; a new file at the same path is a +/// new document. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_file_renamed_moved_or_deleted_elsewhere_keeps_its_open_document() { + let server = Server::new().await; + let (pc, laptop) = (Agent::signed_in(&server, "pc").await, Agent::signed_in(&server, "laptop").await); + let (run_pc, run_laptop) = (pc.run(), laptop.run()); + let on_pc = pc.client().await; + laptop.client().await; + std::fs::create_dir_all(pc.folder.join("Old")).unwrap(); + std::fs::write(pc.folder.join("Old/a.txt"), "aaa").unwrap(); + synced(&laptop, "Old/a.txt").await; + let c = on_pc.clone(); + let (docs, app) = off(move || { + let docs = c.documents().unwrap(); + let doc = open(&docs, "Old/a.txt", "aaa"); + (docs, AppDoc::new(doc, 31)) + }) + .await; + let first = app.doc.id(); + // Renamed and moved on the laptop. + std::fs::create_dir_all(laptop.folder.join("New")).unwrap(); + std::fs::rename(laptop.folder.join("Old/a.txt"), laptop.folder.join("New/b.txt")).unwrap(); + synced(&pc, "New/b.txt").await; + let docs = Arc::new(docs); + let d = docs.clone(); + let again = off(move || { + let doc = open(&d, "New/b.txt", "never used"); + doc.id() + }) + .await; + assert_eq!(again, first, "the same node, the same document"); + let app = Arc::new(Mutex::new(app)); + let seq = app.lock().unwrap().append(" still here"); + let a = app.clone(); + eventually("an edit after the move is acked", move || a.lock().unwrap().pump().contains(&Event::Acked(seq)).then_some(())).await; + // Deleted on the laptop: the open document goes on, and says its file was deleted. + std::fs::remove_file(laptop.folder.join("New/b.txt")).unwrap(); + let a = app.clone(); + eventually("the document says its file was deleted", move || { + a.lock().unwrap().pump().iter().any(|e| matches!(e, Event::Status { deleted: true, .. })).then_some(()) + }) + .await; + let seq = app.lock().unwrap().append(" after the delete"); + let a = app.clone(); + eventually("it still takes edits", move || a.lock().unwrap().pump().contains(&Event::Acked(seq)).then_some(())).await; + // A new file at the old path is a new node, and a new document. + std::fs::write(pc.folder.join("New/b.txt"), "new file").unwrap(); + let file = laptop.folder.join("New/b.txt"); + eventually("the new file reached the laptop", move || file.exists().then_some(())).await; + let d = docs.clone(); + let fresh = eventually("the new file's document", move || match d.open("New/b.txt", "wordcraft").ok()? { + Opened::NeedsImport(importer) => Some(importer.import(&text_snapshot("new file")).ok()?.id()), + Opened::Live(doc) => Some(doc.id()), + Opened::NotLive(_) => None, + }) + .await; + assert_ne!(fresh, first, "a new file, a new document"); + drop((docs, app)); + run_pc.stop().await.0.unwrap(); + run_laptop.stop().await.0.unwrap(); +} + +/// The docs socket as a raw client sees it (the crate is well behaved). +mod raw { + use futures_util::{SinkExt, StreamExt}; + use tokio_tungstenite::tungstenite::Message; + use tokio_tungstenite::tungstenite::client::IntoClientRequest; + + pub type Ws = tokio_tungstenite::WebSocketStream>; + + pub async fn connect(agent: &super::Agent) -> Ws { + let api: craft_cloud_proto::local::LocalApiFile = + serde_json::from_slice(&std::fs::read(agent.data.path().join(craft_cloud_proto::local::LOCAL_API_FILE)).unwrap()).unwrap(); + let mut request = format!("ws://127.0.0.1:{}/v1/docs", api.port).into_client_request().unwrap(); + request.headers_mut().insert("authorization", format!("Bearer {}", api.token).parse().unwrap()); + tokio_tungstenite::connect_async(request).await.unwrap().0 + } + + pub async fn send_text(ws: &mut Ws, json: serde_json::Value) { + ws.send(Message::Text(json.to_string().into())).await.unwrap(); + } + + pub async fn send_binary(ws: &mut Ws, bytes: Vec) { + let _ = ws.send(Message::Binary(bytes.into())).await; + } + + /// The next message, or `None` once the connection is closed (or nothing comes for 10 s). + pub async fn next(ws: &mut Ws) -> Option { + match tokio::time::timeout(std::time::Duration::from_secs(10), ws.next()).await { + Ok(Some(Ok(Message::Close(_)))) | Ok(Some(Err(_))) | Ok(None) | Err(_) => None, + Ok(Some(Ok(message))) => Some(message), + } + } + + /// The next control message, skipping binary ones and `status` events (which come whenever the document's state + /// changes, between any two answers). + pub async fn next_json(ws: &mut Ws) -> Option { + loop { + match next(ws).await? { + Message::Text(text) => { + let value: serde_json::Value = serde_json::from_str(text.as_str()).ok()?; + if value["type"] != "status" { + return Some(value); + } + } + _ => continue, + } + } + } +} + +/// Review: an app that sends garbage binary frames is dropped; an update over 1 MiB is refused; one that never reads +/// is dropped; one that goes away mid-import leaves no half import on the server. The agent and the other apps go on. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn an_app_that_misbehaves_is_refused_or_dropped_and_the_rest_go_on() { + let server = Server::new().await; + let pc = Agent::signed_in(&server, "pc").await; + let running = pc.run(); + let on_pc = pc.client().await; + std::fs::create_dir_all(&pc.folder).unwrap(); + std::fs::write(pc.folder.join("doc.txt"), "doc").unwrap(); + std::fs::write(pc.folder.join("other.txt"), "other").unwrap(); + let c = on_pc.clone(); + eventually("both files are synced", move || { + let docs = c.documents().ok()?; + matches!(docs.open("other.txt", "wordcraft").ok()?, Opened::NeedsImport(_)).then_some(()) + }) + .await; + // A well-behaved app has a document open throughout. + let c = on_pc.clone(); + let (good_docs, good) = off(move || { + let docs = c.documents().unwrap(); + let doc = open(&docs, "doc.txt", "doc"); + (docs, AppDoc::new(doc, 41)) + }) + .await; + // Garbage binary frames: a frame too short for its header, and one of a kind an app never sends. + for junk in [vec![2, 0, 0], craft_cloud_proto::local::docs::encode_binary(1, 7, b"a state from the app")] { + let mut ws = raw::connect(&pc).await; + raw::send_binary(&mut ws, junk).await; + assert!(raw::next(&mut ws).await.is_none(), "dropped"); + } + // An update over 1 MiB: refused, and the connection goes on. + let mut ws = raw::connect(&pc).await; + raw::send_text(&mut ws, serde_json::json!({ "type": "open", "id": 1, "path": "doc.txt", "app": "wordcraft" })).await; + let opened = raw::next_json(&mut ws).await.unwrap(); + assert_eq!(opened["type"], "opened", "{opened}"); + let huge = craft_cloud_proto::local::docs::encode_binary(2, 1, &vec![0u8; craft_cloud_proto::live::MAX_UPDATE_BYTES + 1]); + raw::send_binary(&mut ws, huge).await; + let refused = raw::next_json(&mut ws).await.unwrap(); + assert_eq!( + (refused["type"].as_str(), refused["code"].as_str(), refused["seq"].as_u64()), + (Some("error"), Some("too_large"), Some(1)), + "{refused}" + ); + // An update that isn't Loro: refused, the document as it was. + raw::send_binary(&mut ws, craft_cloud_proto::local::docs::encode_binary(2, 1, b"not loro")).await; + let refused = raw::next_json(&mut ws).await.unwrap(); + assert_eq!((refused["code"].as_str(), refused["seq"].as_u64()), (Some("invalid"), Some(2)), "{refused}"); + // A request that isn't one. + futures_util::SinkExt::send(&mut ws, tokio_tungstenite::tungstenite::Message::Text("{\"type\":\"nonsense\"}".into())).await.unwrap(); + assert_eq!(raw::next_json(&mut ws).await.unwrap()["type"], "error"); + drop(ws); + // Away mid-import: asked for the import, and gone without sending it. + let mut ws = raw::connect(&pc).await; + raw::send_text(&mut ws, serde_json::json!({ "type": "open", "id": 1, "path": "other.txt", "app": "wordcraft" })).await; + assert_eq!(raw::next_json(&mut ws).await.unwrap()["type"], "needs_import"); + drop(ws); + let doc = craft_cloud_proto::live::node_doc( + server.meta.user_by_name("ana").await.unwrap().unwrap().space, + *server_node(&server, "other.txt").await.as_ref().unwrap(), + ); + tokio::time::sleep(Duration::from_millis(300)).await; + let space = server.meta.user_by_name("ana").await.unwrap().unwrap().space; + assert!(server.meta.live_doc(space, doc).await.unwrap().is_none(), "no half import on the server"); + // The next app to open it is asked again, and makes it live. + let c = on_pc.clone(); + let other = off(move || { + let docs = c.documents().unwrap(); + let doc = open(&docs, "other.txt", "other"); + (docs, AppDoc::new(doc, 42)) + }) + .await; + assert_eq!(other.1.text(), "other"); + // An app that never reads while another floods its document with big changes: dropped (it can't hold the agent up). + let mut lazy = raw::connect(&pc).await; + raw::send_text(&mut lazy, serde_json::json!({ "type": "open", "id": 1, "path": "doc.txt", "app": "wordcraft" })).await; + tokio::time::sleep(Duration::from_millis(500)).await; + let good = Arc::new(Mutex::new(good)); + let g = good.clone(); + off(move || { + let app = g.lock().unwrap(); + let chunk = "y".repeat(200_000); + for _ in 0..60 { + app.append(&chunk); + } + }) + .await; + // The well-behaved app gets every ack meanwhile, and the agent stays up. + let g = good.clone(); + eventually("the good app's changes are all acked", move || { + let app = g.lock().unwrap(); + app.pump().contains(&Event::Acked(60)).then_some(()) + }) + .await; + // Not reading for over 10 s (the agent's limit for one message) before looking. + tokio::time::sleep(Duration::from_secs(12)).await; + let mut dropped = false; + for _ in 0..200 { + match tokio::time::timeout(Duration::from_secs(30), futures_util::StreamExt::next(&mut lazy)).await { + Ok(Some(Ok(tokio_tungstenite::tungstenite::Message::Close(_)))) | Ok(Some(Err(_))) | Ok(None) => { + dropped = true; + break; + } + Ok(Some(Ok(_))) => {} + Err(_) => break, + } + } + assert!(dropped, "the app that didn't read was dropped"); + let c = on_pc.clone(); + assert!(off(move || c.status()).await.unwrap().signed_in, "the agent goes on"); + drop((good_docs, good, other)); + running.stop().await.0.unwrap(); +} + +/// The node the server has at `path` (from the root). +async fn server_node(server: &Server, path: &str) -> Option { + let space = server.meta.user_by_name("ana").await.unwrap()?.space; + let tree = server.meta.tree(space, None, 5000).await.unwrap(); + let name = path.rsplit('/').next()?; + tree.nodes.iter().find(|n| n.name.as_ref().is_some_and(|n| n.as_str() == name)).map(|n| n.id) +} + +/// Runs the agent's command line with `args`, against `agent`'s data folder. +async fn cli(agent: &Agent, args: &[&str]) -> (Result, String) { + use clap::Parser; + let data = agent.data.path().display().to_string(); + let mut argv = vec!["craft-cloud-agent", "--data-dir", data.as_str()]; + argv.extend_from_slice(args); + let parsed = craft_cloud_agent::cli::Cli::try_parse_from(argv).unwrap(); + let mut out = Vec::new(); + let exit = craft_cloud_agent::cli::run(parsed, &mut "".as_bytes(), &mut out).await; + (exit, String::from_utf8(out).unwrap()) +} + +/// The `doc` tool: imports a plain-text file, prints it, appends to it (on every machine), and refuses a file that +/// isn't UTF-8. It never writes the file. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn the_doc_command_reads_and_appends_like_an_app() { + use craft_cloud_agent::cli::{AgentError, Exit}; + let server = Server::new().await; + let (pc, laptop) = (Agent::signed_in(&server, "pc").await, Agent::signed_in(&server, "laptop").await); + let (run_pc, run_laptop) = (pc.run(), laptop.run()); + pc.client().await; + laptop.client().await; + std::fs::create_dir_all(&pc.folder).unwrap(); + std::fs::write(pc.folder.join("todo.txt"), "milk\n").unwrap(); + std::fs::write(pc.folder.join("binary.bin"), [0xff, 0xfe, 0x00]).unwrap(); + synced(&laptop, "todo.txt").await; + synced(&laptop, "binary.bin").await; + let (exit, out) = cli(&pc, &["doc", "todo.txt"]).await; + assert_eq!(exit.unwrap(), Exit::Done, "{out}"); + assert_eq!(out, "milk\n\n"); + let absolute = laptop.folder.join("todo.txt").display().to_string(); + let (exit, out) = cli(&laptop, &["doc", &absolute, "--append", "eggs\n"]).await; + assert_eq!(exit.unwrap(), Exit::Done, "{out}"); + assert_eq!(out, "milk\neggs\n\n"); + let p = Arc::new(pc); + let pp = p.clone(); + // (The change reaches the pc as it comes; `doc` there sees it once it did.) + let text = eventually("the pc has the laptop's change", move || { + let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().ok()?; + let (exit, out) = runtime.block_on(cli(&pp, &["doc", "todo.txt"])); + (exit.ok()? == Exit::Done && out.contains("eggs")).then_some(out) + }) + .await; + assert_eq!(text, "milk\neggs\n\n"); + assert_eq!(std::fs::read_to_string(p.folder.join("todo.txt")).unwrap(), "milk\n", "the file is never written"); + let (exit, _) = cli(&p, &["doc", "binary.bin"]).await; + assert!(matches!(&exit, Err(AgentError::Usage(m)) if m.contains("UTF-8")), "{exit:?}"); + run_pc.stop().await.0.unwrap(); + run_laptop.stop().await.0.unwrap(); +} + +/// Signed out, every open is answered `not_live {not_signed_in}`. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn signed_out_every_file_is_a_plain_file() { + let agent = Agent::signed_out(); + let running = agent.run(); + let client = agent.client().await; + let opened = off(move || client.documents().unwrap().open("a.txt", "wordcraft").unwrap()).await; + assert!(matches!(opened, Opened::NotLive(NotLiveReason::NotSignedIn)), "{opened:?}"); + running.stop().await.0.unwrap(); +} diff --git a/apps/craft-cloud-agent/tests/live_speed.rs b/apps/craft-cloud-agent/tests/live_speed.rs new file mode 100644 index 0000000..fcb8167 --- /dev/null +++ b/apps/craft-cloud-agent/tests/live_speed.rs @@ -0,0 +1,175 @@ +//! How fast live documents are. **Latency**: a keystroke in an app on one machine is applied in the open document on +//! another in under 250 ms, through the crate, two running agents and the server, on loopback with 2 ms of simulated +//! delay each way. **Size**: a 5 MB text document with 100,000 updates in its log opens on a machine that never had it +//! in under a second, from the server's snapshot and the log above it. The size test runs only in a release build +//! (`cargo test --release`), or with `LIVE_SIZE=1`, as the long simulations do. + +mod looping; + +#[path = "docs_common/mod.rs"] +mod docs_common; + +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +use craft_cloud_app::live::{Event, Opened}; +use looping::{Agent, Server, eventually}; +use loro::{ExportMode, LoroDoc}; + +/// Blocking work off the test's runtime. +async fn off(f: impl FnOnce() -> T + Send + 'static) -> T { + tokio::task::spawn_blocking(f).await.unwrap() +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_keystroke_reaches_the_other_machine_in_under_250_ms() { + let server = Server::delayed(Duration::from_millis(2)).await; + let (pc, laptop) = (Agent::signed_in(&server, "pc").await, Agent::signed_in(&server, "laptop").await); + let (run_pc, run_laptop) = (pc.run(), laptop.run()); + let (on_pc, on_laptop) = (pc.client().await, laptop.client().await); + std::fs::create_dir_all(&pc.folder).unwrap(); + std::fs::write(pc.folder.join("fast.txt"), "").unwrap(); + let file = laptop.folder.join("fast.txt"); + eventually("the file reached the laptop", move || file.exists().then_some(())).await; + let c = on_pc.clone(); + let (docs_pc, doc_pc) = off(move || { + let docs = c.documents().unwrap(); + let doc = match docs.open("fast.txt", "wordcraft").unwrap() { + Opened::NeedsImport(importer) => importer.import(&craft_cloud_agent::doc::plain_text_snapshot("").unwrap()).unwrap(), + Opened::Live(doc) => doc, + Opened::NotLive(reason) => panic!("{reason:?}"), + }; + (docs, doc) + }) + .await; + let c = on_laptop.clone(); + let (docs_laptop, doc_laptop) = off(move || { + let docs = c.documents().unwrap(); + let Opened::Live(doc) = docs.open("fast.txt", "wordcraft").unwrap() else { panic!("not live on the laptop") }; + (docs, doc) + }) + .await; + let mine = LoroDoc::new(); + mine.import(doc_pc.state()).unwrap(); + let theirs = LoroDoc::new(); + theirs.import(doc_laptop.state()).unwrap(); + let (pc_side, laptop_side) = (Arc::new(Mutex::new((mine, doc_pc))), Arc::new(Mutex::new((theirs, doc_laptop)))); + // Both online (the laptop's document followed). + let l = laptop_side.clone(); + eventually("the laptop's document is online", move || { + let side = l.lock().unwrap(); + (side.1.opened_state().online || std::iter::from_fn(|| side.1.try_recv()).any(|e| matches!(e, Event::Status { online: true, .. }))) + .then_some(()) + }) + .await; + let (p, l) = (pc_side.clone(), laptop_side.clone()); + let times = off(move || { + let mut times = Vec::new(); + for i in 0..40 { + let typed = { + let (mine, doc) = &*p.lock().unwrap(); + let before = mine.oplog_vv(); + let text = mine.get_text("text"); + text.insert(text.len_unicode(), &format!("{}", i % 10)).unwrap(); + mine.commit(); + let update = mine.export(ExportMode::updates(&before)).unwrap(); + let at = Instant::now(); + doc.send(&update).unwrap(); + at + }; + let want = i + 1; + let (theirs, doc) = &*l.lock().unwrap(); + loop { + match doc.recv_timeout(Duration::from_secs(5)) { + Some(Event::Update(bytes)) => { + theirs.import(&bytes).unwrap(); + if theirs.get_text("text").len_unicode() >= want { + times.push(typed.elapsed()); + break; + } + } + Some(_) => {} + None => panic!("keystroke {i} never arrived"), + } + } + // A typist's pace: a keystroke batch every 30 ms. + std::thread::sleep(Duration::from_millis(30)); + } + times + }) + .await; + let mut sorted = times.clone(); + sorted.sort(); + let (median, worst) = (sorted[sorted.len() / 2], sorted[sorted.len() - 1]); + eprintln!("keystroke to remote apply: median {median:?}, worst {worst:?} over {} keystrokes", sorted.len()); + assert!(worst < Duration::from_millis(250), "a keystroke took {worst:?}: {sorted:?}"); + drop((docs_pc, docs_laptop, pc_side, laptop_side)); + run_pc.stop().await.0.unwrap(); + run_laptop.stop().await.0.unwrap(); +} + +/// A 5 MB text document with 100,000 updates in its log (compacted, with a tail above the snapshot) opens on a fresh +/// machine in under a second: the catch-up from the snapshot and the tail, and the state an app builds from. +#[test] +fn a_big_document_opens_from_its_snapshot_and_tail_in_under_a_second() { + if cfg!(debug_assertions) && std::env::var_os("LIVE_SIZE").is_none() { + eprintln!("skipped: a release build (or LIVE_SIZE=1) runs it"); + return; + } + tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap().block_on(async { + use craft_cloud_agent::live::replica::{DocEvent, Opening}; + use craft_cloud_proto::live::{ClientFrame, ServerFrame}; + let mut w = + docs_common::World::new(craft_cloud_server::state::Limits { compact_updates: u64::MAX, compact_bytes: u64::MAX, ..Default::default() }) + .await; + let fresh = w.machine().await; + let node = w.file().await; + let doc = craft_cloud_proto::live::node_doc(w.space, node); + // The document as its machines made it: 100,000 changes of about 50 characters, 5 MB of text. + let made = LoroDoc::new(); + made.set_peer_id(1).unwrap(); + let import = made.export(ExportMode::Snapshot).unwrap(); + let token = [97u8; 32]; + w.meta.create_device(w.user, "writer", &token, 0).await.unwrap(); + let auth = w.meta.device_by_token(&token).await.unwrap().unwrap(); + let mut writer = craft_cloud_server::live::Session::new(w.state.clone(), &auth); + writer.handle(ClientFrame::Hello { min: 1, max: 2 }).await; + let import = w.upload(doc, &import).await; + let made_reply = writer.handle(ClientFrame::CreateDocument { doc, node, app: docs_common::app(), snapshot: import }).await; + assert_eq!(made_reply.frames, vec![ServerFrame::CreatedDocument { doc, kept: true }]); + let line = "the quick brown fox jumps over the lazy dog, again\n"; + let mut tail_from = None; + for i in 0..100_000u64 { + let before = made.oplog_vv(); + let text = made.get_text("text"); + text.insert(text.len_unicode(), line).unwrap(); + made.commit(); + let update = made.export(ExportMode::updates(&before)).unwrap(); + w.meta.append_live_update(w.space, doc, auth.device, i + 1, &update, 0).await.unwrap(); + if i + 1 == 99_000 { + tail_from = Some(made.export(ExportMode::Snapshot).unwrap()); + } + } + // Compacted at 99,000 (as an agent would answer), the last 1,000 above it. + let snapshot = tail_from.unwrap(); + let stored = w.upload(doc, &snapshot).await; + assert!(w.meta.compact_live_doc(w.space, doc, 99_000, &stored).await.unwrap()); + let text_len = made.get_text("text").len_utf8(); + assert!(text_len >= 5_000_000, "{text_len} bytes of text"); + // A machine that never had it opens it. + w.connect(fresh); + w.settle().await; + let started = Instant::now(); + let (_, opening) = w.open(fresh, node); + assert_eq!(opening, Opening::Joining); + w.settle().await; + assert!(w.events(fresh).contains(&DocEvent::Joined(doc))); + let joined_in = started.elapsed(); + let app = w.app_of(fresh, doc, 5); + eprintln!("joined in {joined_in:?}, the app's copy built in {:?}", started.elapsed() - joined_in); + let took = started.elapsed(); + assert_eq!(app.doc.get_text("text").len_utf8(), text_len); + eprintln!("a {text_len}-byte document with 100,000 updates opened in {took:?}"); + assert!(took < Duration::from_secs(1), "it took {took:?}"); + }); +} diff --git a/apps/craft-cloud-agent/tests/looping/mod.rs b/apps/craft-cloud-agent/tests/looping/mod.rs index 57b0932..1c9bfa5 100644 --- a/apps/craft-cloud-agent/tests/looping/mod.rs +++ b/apps/craft-cloud-agent/tests/looping/mod.rs @@ -30,7 +30,48 @@ pub struct Gate { task: tokio::task::JoinHandle<()>, } +/// Copies `from` to `to`, each piece `delay` after it was read (in order: a piece never overtakes the one before). +async fn delayed_copy(mut from: tokio::net::tcp::OwnedReadHalf, mut to: tokio::net::tcp::OwnedWriteHalf, delay: Duration) { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + let (tx, mut rx) = mpsc::unbounded_channel::<(Instant, Vec)>(); + let reader = tokio::spawn(async move { + let mut buf = vec![0u8; 64 * 1024]; + while let Ok(n) = from.read(&mut buf).await { + if n == 0 || tx.send((Instant::now() + delay, buf[..n].to_vec())).is_err() { + return; + } + } + }); + while let Some((at, piece)) = rx.recv().await { + tokio::time::sleep_until(at.into()).await; + if to.write_all(&piece).await.is_err() { + break; + } + } + reader.abort(); +} + impl Gate { + /// A gate that delays every message by `delay` each way (a LAN's latency on loopback). + async fn delayed(to: SocketAddr, delay: Duration) -> Self { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let task = tokio::spawn(async move { + while let Ok((inbound, _)) = listener.accept().await { + let _ = inbound.set_nodelay(true); + tokio::spawn(async move { + let Ok(outbound) = tokio::net::TcpStream::connect(to).await else { return }; + let _ = outbound.set_nodelay(true); + let (in_read, in_write) = inbound.into_split(); + let (out_read, out_write) = outbound.into_split(); + tokio::join!(delayed_copy(in_read, out_write, delay), delayed_copy(out_read, in_write, delay)); + }); + } + }); + let up = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(true)); + Self { addr, up, generation: std::sync::Arc::default(), task } + } + async fn new(to: SocketAddr) -> Self { use std::sync::atomic::Ordering; let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); @@ -89,12 +130,29 @@ pub struct Server { impl Server { pub async fn new() -> Self { + Self::start(None, Default::default()).await + } + + /// A server whose every message takes `delay` each way (a gate that can't go down). + pub async fn delayed(delay: Duration) -> Self { + Self::start(Some(delay), Default::default()).await + } + + /// A server with other limits (a compaction asked for early, say). + pub async fn with_limits(limits: craft_cloud_server::state::Limits) -> Self { + Self::start(None, limits).await + } + + async fn start(delay: Option, limits: craft_cloud_server::state::Limits) -> Self { let (db, blobs) = (tempfile::tempdir().unwrap(), tempfile::tempdir().unwrap()); let meta = Meta::open(&format!("sqlite:{}", db.path().join("meta.db").display())).await.unwrap(); meta.create_user("ana", &hash_password(PASSWORD).unwrap(), Role::Member, 0).await.unwrap(); - let state = AppState::new(meta.clone(), Store::new(DiskBackend::new(blobs.path()).unwrap()), vec![]).unwrap(); + let state = AppState::new(meta.clone(), Store::new(DiskBackend::new(blobs.path()).unwrap()), vec![]).unwrap().with_limits(limits); let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); - let gate = Gate::new(listener.local_addr().unwrap()).await; + let gate = match delay { + Some(delay) => Gate::delayed(listener.local_addr().unwrap(), delay).await, + None => Gate::new(listener.local_addr().unwrap()).await, + }; tokio::spawn(craft_cloud_server::serve(state, listener, std::future::pending())); Self { url: format!("http://{}", gate.addr), gate, meta, _dirs: (db, blobs) } } diff --git a/apps/craft-cloud-agent/tests/network/live.rs b/apps/craft-cloud-agent/tests/network/live.rs new file mode 100644 index 0000000..a253bd7 --- /dev/null +++ b/apps/craft-cloud-agent/tests/network/live.rs @@ -0,0 +1,694 @@ +//! Live documents over the simulated network: the documents simulation's random work (apps editing one document's +//! text and map, machines crashing and coming back, the log compacted at random) with the agents' replicas talking to +//! the server's live sessions over real WebSockets (tungstenite's framing) on turmoil's TCP, while the network +//! partitions links (both ways), holds them and delays every message. Then the network heals, everything settles, and +//! the documents simulation's checks run: every machine and every app holds the same document, every edit an agent +//! took is in it, and nothing waits for the server. +//! +//! Not simulated here: the server's axum handler and the agent's hub (`tests/live_documents.rs` runs those against a +//! loopback server); their protocol halves, the session and the replica, are the real ones. + +use std::collections::BTreeSet; +use std::future::Future; +use std::pin::Pin; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::task::{Context, Poll}; +use std::time::Duration; + +use craft_cloud_agent::live::replica::{DocEvent, Opening, Peers, Replica}; +use craft_cloud_agent::live::store::MemStore; +use craft_cloud_meta::{DeviceAuth, Meta, Role}; +use craft_cloud_proto::live::{AppName, ClientFrame, ServerFrame}; +use craft_cloud_proto::{ChunkId, ChunkRef, ChunkerParamsId, DocId, FileHash, Manifest, Name, NewVersion, NodeId, Op, SpaceId, VersionId}; +use craft_cloud_server::AppState; +use craft_cloud_server::live::Session; +use craft_cloud_server::state::Limits; +use craft_cloud_store::Store; +use craft_cloud_store::memory::MemoryBackend; +use futures_util::{SinkExt, StreamExt}; +use loro::{ExportMode, LoroDoc}; +use tokio::sync::Notify; +use tokio_tungstenite::tungstenite::Message; +use tokio_tungstenite::tungstenite::client::IntoClientRequest; + +use crate::common::sim::{Rng, tracing}; + +const SERVER: &str = "server"; +const PORT: u16 = 81; +const MACHINES: usize = 3; +/// How long the random work goes on, in simulated time; then the network heals. +const WORK: Duration = Duration::from_secs(120); + +fn app() -> AppName { + AppName::new("wordcraft").unwrap() +} + +fn peers(base: u64) -> Peers { + let mut n = 0u64; + Box::new(move || { + n += 1; + base.wrapping_add(n.wrapping_mul(0x9E37_79B9_7F4A_7C15)) >> 1 + }) +} + +/// What the hosts and the driver share. +struct World { + meta: Meta, + state: Arc, + space: SpaceId, + node: NodeId, + hosts: Vec>, + /// What happened, in order: for the trace and the replay check. + log: Mutex>, + /// A task keeps simulated time still while the database works (see the network harness's `hold_time`). + holding: AtomicBool, +} + +impl World { + fn say(&self, line: String) { + if tracing() { + eprintln!("{} {line}", stamp()); + } + self.log.lock().unwrap().push(line); + } +} + +fn stamp() -> String { + format!("[{:>8.2}s]", turmoil::sim_elapsed().unwrap_or_default().as_secs_f64()) +} + +/// One machine: its agent's replica over a store that outlasts its crashes, and its app's own copy. +struct Host { + token: String, + store: MemStore, + replica: Replica, + runs: u64, + base: u64, + app: Option, + app_runs: u64, + opening: bool, + doc: Option, + /// Every edit its agent took from its app. + taken: BTreeSet, + edits: u64, + /// Frames for the connection, from the app's side. + out: Vec, + /// It is to crash (the connection task does it). + crash: bool, + /// What its agent said. + notes: Vec, +} + +impl Host { + fn app_doc(&mut self, doc: DocId) { + self.app_runs += 1; + let Some((snapshot, after)) = self.replica.document_state(doc) else { return }; + let app = LoroDoc::new(); + app.set_peer_id((self.base >> 8) + self.app_runs).unwrap(); + app.import(&snapshot).unwrap(); + for bytes in after { + app.import(&bytes).unwrap(); + } + self.app = Some(app); + } + + /// Hands the replica's events to the app. + fn events(&mut self, node: NodeId) { + for event in self.replica.take_events() { + match event { + DocEvent::Joined(doc) => { + self.doc = Some(doc); + if std::mem::take(&mut self.opening) { + self.app_doc(doc); + } + } + DocEvent::NeedsImport(doc) => { + let start = LoroDoc::new(); + start.get_text("text").insert(0, "start;").unwrap(); + start.commit(); + if let Ok(frames) = self.replica.import_document(doc, &start.export(ExportMode::Snapshot).unwrap()) { + self.out.extend(frames); + } + } + DocEvent::NotLive(..) => self.opening = false, + DocEvent::Remote(_, bytes) => { + if let Some(app) = &self.app { + app.import(&bytes).unwrap(); + } + } + DocEvent::Presence(..) => {} + } + } + let _ = node; + self.replica.flush().unwrap(); + self.notes.extend(self.replica.take_notes()); + } + + fn open(&mut self, node: NodeId) { + if self.app.is_some() || self.opening { + return; + } + let (doc, opening, frames) = self.replica.open_document(node, &app()); + self.out.extend(frames); + match opening { + Opening::Live => self.app_doc(doc), + Opening::Joining => self.opening = true, + Opening::NotLive(..) => {} + } + } + + /// The app edits: a token between tokens of the text, or a key of the map. + fn edit(&mut self, m: usize, rng: &mut Rng) -> Option { + let (Some(doc), Some(app)) = (self.doc, self.app.as_ref()) else { return None }; + self.edits += 1; + let token = format!("m{m}e{};", self.edits); + let before = app.oplog_vv(); + if rng.below(10) < 7 { + let text = app.get_text("text"); + let mut places = vec![0]; + places.extend(text.to_string().chars().enumerate().filter(|(_, c)| *c == ';').map(|(i, _)| i + 1)); + text.insert(places[rng.below(places.len())], &token).unwrap(); + } else { + app.get_map("map").insert(&token, self.edits as i64).unwrap(); + } + app.commit(); + let update = app.export(ExportMode::updates(&before)).unwrap(); + match self.replica.local_update(doc, &update) { + Ok(frames) => { + self.taken.insert(token.clone()); + self.out.extend(frames); + Some(token) + } + Err(_) => { + self.app = None; + None + } + } + } + + /// Crashes: what it hadn't written is gone; it starts again from its store. + fn restart(&mut self, space: SpaceId) { + self.runs += 1; + self.replica = Replica::open(self.store.clone(), space, peers(self.base ^ (self.runs << 40))).unwrap(); + self.app = None; + self.opening = false; + self.out.clear(); + } +} + +/// Keeps simulated time still while the database works, as the network harness does (see its `hold_time`). +fn hold_time(world: &Arc) { + if world.holding.swap(true, Ordering::Relaxed) { + return; + } + let world = world.clone(); + tokio::spawn(async move { + struct Release(Arc); + impl Drop for Release { + fn drop(&mut self) { + self.0.holding.store(false, Ordering::Relaxed); + } + } + let _release = Release(world.clone()); + let mut idle = 0; + while idle < 4 { + if world.meta.busy() { + idle = 0; + let meta = world.meta.clone(); + let _ = tokio::task::spawn_blocking(move || { + while meta.busy() { + std::thread::sleep(Duration::from_micros(20)); + } + }) + .await; + } else { + idle += 1; + tokio::task::yield_now().await; + } + } + }); +} + +/// A future that keeps time still whenever it waits (on the database). +struct Held { + world: Arc, + inner: Pin>, +} + +impl Future for Held { + type Output = F::Output; + + fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + let this = self.get_mut(); + let out = this.inner.as_mut().poll(cx); + if out.is_pending() { + hold_time(&this.world); + } + out + } +} + +fn held(world: &Arc, f: F) -> Held { + Held { world: world.clone(), inner: Box::pin(f) } +} + +/// The server host: each connection a WebSocket, signed in with its device token, served by a live session. +async fn server(world: Arc) -> turmoil::Result { + let listener = turmoil::net::TcpListener::bind((std::net::Ipv4Addr::UNSPECIFIED, PORT)).await?; + loop { + let (stream, _) = listener.accept().await?; + tokio::spawn(serve(world.clone(), stream)); + } +} + +#[allow(clippy::result_large_err)] // (tungstenite's callback type) +async fn serve(world: Arc, stream: turmoil::net::TcpStream) { + let token = Arc::new(Mutex::new(None::)); + let seen = token.clone(); + let callback = move |request: &tokio_tungstenite::tungstenite::handshake::server::Request, response| { + let bearer = request.headers().get("authorization").and_then(|v| v.to_str().ok()).and_then(|v| v.strip_prefix("Bearer ")).map(str::to_owned); + *seen.lock().unwrap() = bearer; + Ok(response) + }; + let Ok(mut ws) = tokio_tungstenite::accept_hdr_async(stream, callback).await else { return }; + let Some(token) = token.lock().unwrap().clone() else { return }; + let hash = craft_cloud_server::auth::token_hash(&token); + let Ok(Some(auth)) = held(&world, world.meta.device_by_token(&hash)).await else { return }; + let mut notes = world.state.live.listen(auth.space); + let mut presence = world.state.live.listen_presence(auth.space); + let mut session = Session::new(world.state.clone(), &auth); + loop { + tokio::select! { + biased; + message = ws.next() => { + let bytes = match message { + Some(Ok(Message::Binary(bytes))) => bytes, + // (The pong goes out with the next frame sent.) + Some(Ok(Message::Ping(_) | Message::Pong(_))) => { + if ws.flush().await.is_err() { + break; + } + continue; + } + _ => break, + }; + let Ok(frame) = ClientFrame::decode(&bytes) else { break }; + let reply = held(&world, session.handle(frame)).await; + for frame in reply.frames { + if ws.send(Message::Binary(frame.encode().unwrap().into())).await.is_err() { + return; + } + } + if reply.close { + break; + } + } + note = notes.recv() => match note { + Ok(doc) => session.note(doc), + Err(_) => session.lagged(), + }, + note = presence.recv() => { + if let Ok(note) = note + && let Some(frame) = session.relay(¬e) + && ws.send(Message::Binary(frame.encode().unwrap().into())).await.is_err() + { + return; + } + } + () = std::future::ready(()), if session.has_work() => { + let Ok(frames) = held(&world, session.pump()).await else { break }; + for frame in frames { + if ws.send(Message::Binary(frame.encode().unwrap().into())).await.is_err() { + return; + } + } + } + } + } +} + +/// A machine's host: the app editing at random, and the agent's connection (dialed again whenever it goes). +async fn machine(world: Arc, m: usize, seed: u64, stop: Arc) -> turmoil::Result { + let wake = Arc::new(Notify::new()); + tokio::task::spawn_local(work(world.clone(), m, seed, stop.clone(), wake.clone())); + let mut rng = Rng::new(seed ^ 0xD1A1_0000 ^ m as u64); + loop { + let connected = connection(&world, m, &wake).await; + world.say(format!("m{m} {}", if connected { "lost its connection" } else { "can't connect" })); + let crashed = { + let mut host = world.hosts[m].lock().unwrap(); + host.replica.disconnected(); + host.events(world.node); + let crashed = std::mem::take(&mut host.crash); + if crashed { + host.restart(world.space); + } + crashed + }; + if crashed { + world.say(format!("m{m} crashed and started again")); + } + let backoff = 200 + rng.below(1800) as u64; + tokio::time::sleep(Duration::from_millis(backoff)).await; + } +} + +/// One connection, until it goes (or the machine crashes): whether it came up. +async fn connection(world: &Arc, m: usize, wake: &Arc) -> bool { + let token = world.hosts[m].lock().unwrap().token.clone(); + let Ok(Ok(stream)) = tokio::time::timeout(Duration::from_secs(10), turmoil::net::TcpStream::connect((SERVER, PORT))).await else { return false }; + let mut request = format!("ws://{SERVER}:{PORT}/api/v1/live").into_client_request().unwrap(); + request.headers_mut().insert("authorization", format!("Bearer {token}").parse().unwrap()); + let Ok(Ok((ws, _))) = tokio::time::timeout(Duration::from_secs(10), tokio_tungstenite::client_async(request, stream)).await else { + return false; + }; + // As the hub does: a writer task of its own, so reading goes on while a send waits. + let (mut sink, mut ws) = ws.split(); + let (to_writer, mut queue) = tokio::sync::mpsc::unbounded_channel::(); + let writer = tokio::task::spawn_local(async move { + while let Some(message) = queue.recv().await { + if !matches!(tokio::time::timeout(Duration::from_secs(30), sink.send(message)).await, Ok(Ok(()))) { + return; + } + } + }); + struct Abort(tokio::task::JoinHandle<()>); + impl Drop for Abort { + fn drop(&mut self) { + self.0.abort(); + } + } + let writer = Abort(writer); + let hello = world.hosts[m].lock().unwrap().replica.connected(); + for frame in hello { + let _ = to_writer.send(Message::Binary(frame.encode().unwrap().into())); + } + let mut tick = tokio::time::interval(Duration::from_secs(2)); + let mut last_heard = tokio::time::Instant::now(); + let api = crate::blobs::ServerBlobs { state: world.state.clone(), space: world.space }; + loop { + let (out, uploads) = { + let mut host = world.hosts[m].lock().unwrap(); + if host.crash { + return true; + } + (std::mem::take(&mut host.out), host.replica.take_uploads()) + }; + if writer.0.is_finished() { + return true; + } + // The snapshots these frames name go up first, through the chunk API (in-process here: the simulated network + // carries the live connection only). + for up in uploads { + if let Err(e) = held(world, craft_cloud_agent::live::blobs::upload(&api, &up)).await { + world.hosts[m].lock().unwrap().replica.upload_failed(up.doc, &e.to_string()); + } + } + for frame in out { + let _ = to_writer.send(Message::Binary(frame.encode().unwrap().into())); + } + tokio::select! { + message = ws.next() => { + last_heard = tokio::time::Instant::now(); + let bytes = match message { + Some(Ok(Message::Binary(bytes))) => bytes, + Some(Ok(Message::Ping(_) | Message::Pong(_))) => continue, + _ => return true, + }; + let Ok(frame) = ServerFrame::decode(&bytes) else { return true }; + let fetches = { + let mut host = world.hosts[m].lock().unwrap(); + match host.replica.on_frame(frame) { + Ok(frames) => host.out.extend(frames), + Err(_) => return true, + } + host.events(world.node); + host.replica.take_fetches() + }; + // A snapshot a catch-up starts with: downloaded (in-process), then handed back. + for (doc, manifest) in fetches { + let got = held(world, craft_cloud_agent::live::blobs::download(&api, &manifest)).await.map_err(|e| e.to_string()); + let mut host = world.hosts[m].lock().unwrap(); + let frames = host.replica.on_snapshot(doc, manifest, got); + host.out.extend(frames); + host.events(world.node); + } + } + _ = tick.tick() => { + // A connection silent for 90 s is gone (as the hub's pings find); a ping keeps a quiet one heard. + if last_heard.elapsed() > Duration::from_secs(90) { + return true; + } + let _ = to_writer.send(Message::Ping(Vec::new().into())); + let mut host = world.hosts[m].lock().unwrap(); + let frames = host.replica.resend(); + host.out.extend(frames); + host.events(world.node); + } + () = wake.notified() => {} + } + } +} + +/// The app's side of a machine: opens the document, edits it, now and then closes it, writes documents whole; and the +/// machine crashes now and then. +async fn work(world: Arc, m: usize, seed: u64, stop: Arc, wake: Arc) { + let mut rng = Rng::new(seed ^ 0xA9_0000 ^ ((m as u64) << 20)); + loop { + tokio::time::sleep(Duration::from_millis(50 + rng.below(400) as u64)).await; + let calm = stop.load(Ordering::Relaxed); + let mut host = world.hosts[m].lock().unwrap(); + host.open(world.node); + if !calm { + match rng.below(100) { + 0..=59 => { + if let Some(token) = host.edit(m, &mut rng) { + drop(host); + world.say(format!("m{m} edits {token}")); + host = world.hosts[m].lock().unwrap(); + } + } + 60..=64 => host.replica.flush_documents().unwrap(), + 65..=66 => { + host.crash = true; + } + 67..=68 => { + if let (Some(doc), Some(_)) = (host.doc, host.app.as_ref()) { + host.replica.set_open(doc, false); + host.app = None; + } + } + _ => {} + } + } + host.events(world.node); + drop(host); + wake.notify_one(); + } +} + +/// The driver: the weather while the work goes on, then calm, then the checks. +async fn drive(world: Arc, seed: u64, stop: Arc) -> Vec { + let mut rng = Rng::new(seed ^ 0x3EA7_0000_0000_0011); + let started = tokio::time::Instant::now(); + while started.elapsed() < WORK { + tokio::time::sleep(Duration::from_millis(500 + rng.below(4000) as u64)).await; + let host = format!("m{}", rng.below(MACHINES)); + match rng.below(6) { + 0 => { + world.say(format!("{host} is cut off from the server")); + turmoil::partition(host.as_str(), SERVER); + } + 1 => { + world.say(format!("{host} reaches the server again")); + turmoil::repair(host.as_str(), SERVER); + } + 2 => { + world.say(format!("{host}'s link holds")); + turmoil::hold(host.as_str(), SERVER); + } + 3 => { + turmoil::release(host.as_str(), SERVER); + } + _ => {} + } + } + for i in 0..MACHINES { + let host = format!("m{i}"); + turmoil::release(host.as_str(), SERVER); + turmoil::repair(host.as_str(), SERVER); + } + stop.store(true, Ordering::Relaxed); + world.say("the network heals".into()); + // Until every machine and app holds the same document and nothing waits (at most 10 simulated minutes). + let mut converged = false; + for _ in 0..600 { + tokio::time::sleep(Duration::from_secs(1)).await; + if settled(&world) { + converged = true; + break; + } + } + let views: Vec = (0..MACHINES).map(|m| describe(&world, m)).collect(); + assert!(converged, "seed {seed}: never settled: {views:?}"); + check(&world, seed); + world.say(format!("settled: {}", views.first().cloned().unwrap_or_default())); + std::mem::take(&mut *world.log.lock().unwrap()) +} + +fn view(host: &mut Host) -> Option<(String, String)> { + let doc = host.doc?; + let (snapshot, after) = host.replica.document_state(doc)?; + let held = LoroDoc::new(); + held.import(&snapshot).ok()?; + for bytes in after { + held.import(&bytes).ok()?; + } + let app = host.app.as_ref()?; + Some((format!("{:?}", held.get_deep_value()), format!("{:?}", app.get_deep_value()))) +} + +fn describe(world: &World, m: usize) -> String { + let mut host = world.hosts[m].lock().unwrap(); + let status = host.doc.and_then(|d| host.replica.document_status(d)); + let inside = host.doc.map(|d| host.replica.describe_doc(d)).unwrap_or_default(); + format!("m{m}: {status:?}, {:?}, said {:?}, {inside}", view(&mut host).map(|(held, _)| held.len()), host.notes) +} + +fn settled(world: &World) -> bool { + let mut first = None; + for m in 0..MACHINES { + let mut host = world.hosts[m].lock().unwrap(); + let Some(doc) = host.doc else { return false }; + if host.replica.document_status(doc) != Some((true, 0)) { + return false; + } + let Some((held, app)) = view(&mut host) else { return false }; + if held != app { + return false; + } + match &first { + None => first = Some(held), + Some(f) if *f != held => return false, + Some(_) => {} + } + } + true +} + +fn check(world: &World, seed: u64) { + let host = world.hosts[0].lock().unwrap(); + let doc = host.doc.unwrap(); + let (snapshot, after) = host.replica.document_state(doc).unwrap(); + drop(host); + let held = LoroDoc::new(); + held.import(&snapshot).unwrap(); + for bytes in after { + held.import(&bytes).unwrap(); + } + let text = held.get_text("text").to_string(); + let mut keys = BTreeSet::new(); + held.get_map("map").for_each(|k, _| { + keys.insert(k.to_owned()); + }); + for m in 0..MACHINES { + for token in &world.hosts[m].lock().unwrap().taken { + assert!(text.contains(token.as_str()) || keys.contains(token), "seed {seed}: the edit {token} was taken and is lost"); + } + } +} + +/// One seed's run; its trace. +pub fn run_seed(seed: u64) -> Vec { + let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap(); + let (world, tokens) = runtime.block_on(async { + let meta = Meta::open("sqlite::memory:").await.unwrap(); + let user = meta.create_user("ana", "$argon2id$stub", Role::Member, 0).await.unwrap(); + let mut rng = Rng::new(seed ^ 0x11FE); + let limits = Limits { + compact_updates: 2 + rng.below(30) as u64, + compact_timeout: if rng.below(2) == 0 { Duration::ZERO } else { Duration::from_secs(30) }, + ..Limits::default() + }; + let clock = Arc::new(|| 1_760_000_000_000 + turmoil::sim_elapsed().map_or(0, |t| i64::try_from(t.as_millis()).unwrap_or(i64::MAX))); + let state = Arc::new(AppState::new(meta.clone(), Store::new(MemoryBackend::new()), vec![]).unwrap().with_limits(limits).with_clock(clock)); + let mut tokens = Vec::new(); + let mut auths: Vec = Vec::new(); + for i in 0..MACHINES { + let token = format!("token-of-machine-{i}-{seed}"); + meta.create_device(user.id, &format!("m{i}"), &craft_cloud_server::auth::token_hash(&token), 0).await.unwrap(); + auths.push(meta.device_by_token(&craft_cloud_server::auth::token_hash(&token)).await.unwrap().unwrap()); + tokens.push(token); + } + // The file, as the file sync uploaded it. + let chunk = ChunkId::from_bytes(*blake3::hash(b"the file").as_bytes()); + meta.add_chunk(user.space, &chunk, 11, 0).await.unwrap(); + let manifest = + Manifest::new(ChunkerParamsId::new("fastcdc-v1").unwrap(), 10, FileHash::from_bytes([1; 32]), vec![ChunkRef { id: chunk, len: 10 }]) + .unwrap(); + meta.add_manifest(user.space, &manifest, 100, 0).await.unwrap(); + let root = meta.space_info(user.space).await.unwrap().root; + let node = NodeId::from_uuid(uuid::Uuid::from_u64_pair(seed, 7)); + let version = NewVersion { id: VersionId::from_uuid(uuid::Uuid::from_u64_pair(seed, 8)), manifest: manifest.id(), mtime_ms: 0, exec: false }; + meta.commit(user.space, auths[0].device, &[Op::Create { node, parent: root, name: Name::new("doc.txt").unwrap(), version }], 0) + .await + .unwrap(); + let hosts = (0..MACHINES) + .map(|i| { + let store = MemStore::default(); + let base = ((i as u64 + 1) << 48) ^ seed; + Mutex::new(Host { + token: tokens[i].clone(), + replica: Replica::open(store.clone(), user.space, peers(base)).unwrap(), + store, + runs: 1, + base, + app: None, + app_runs: 0, + opening: false, + doc: None, + taken: BTreeSet::new(), + edits: 0, + out: Vec::new(), + crash: false, + notes: Vec::new(), + }) + }) + .collect(); + // SQLite's connection goes back to its pool before the setup runtime ends (as the network harness does). + while meta.busy() { + tokio::time::sleep(Duration::from_millis(1)).await; + } + let world = Arc::new(World { meta, state, space: user.space, node, hosts, log: Mutex::default(), holding: AtomicBool::new(false) }); + (world, tokens) + }); + drop(runtime); + drop(tokens); + let latency = [2, 20, 100][(seed % 3) as usize]; + let mut sim = turmoil::Builder::new() + .simulation_duration(Duration::from_secs(3600)) + .tick_duration(Duration::from_millis(10)) + .min_message_latency(Duration::from_millis(1)) + .max_message_latency(Duration::from_millis(latency)) + .rng_seed(seed) + .build(); + let on_server = world.clone(); + sim.host(SERVER, move || server(on_server.clone())); + let stop = Arc::new(AtomicBool::new(false)); + for m in 0..MACHINES { + let (on_machine, stop) = (world.clone(), stop.clone()); + sim.host(format!("m{m}"), move || machine(on_machine.clone(), m, seed, stop.clone())); + } + let trace = Arc::new(Mutex::new(Vec::new())); + let (driving, out) = (world.clone(), trace.clone()); + sim.client("driver", async move { + let log = drive(driving, seed, stop).await; + *out.lock().unwrap() = log; + Ok(()) + }); + if let Err(e) = sim.run() { + panic!("seed {seed}: {e}"); + } + std::mem::take(&mut *trace.lock().unwrap()) +} diff --git a/apps/craft-cloud-agent/tests/network/main.rs b/apps/craft-cloud-agent/tests/network/main.rs index b426fc0..049b2f7 100644 --- a/apps/craft-cloud-agent/tests/network/main.rs +++ b/apps/craft-cloud-agent/tests/network/main.rs @@ -31,9 +31,12 @@ //! run; `SIM_SEEDS=1000 SIM_FIRST=2000` runs seeds 2000 to 2999 (use `--release`), in parallel on `SIM_JOBS` //! threads. `SIM_TRACE=1` prints every pass and every change of the network. +#[path = "../docs_common/blobs.rs"] +mod blobs; #[path = "../../../../crates/sync/tests/common/mod.rs"] mod common; mod harness; +mod live; use std::collections::{BTreeMap, BTreeSet}; use std::future::Future; @@ -332,6 +335,24 @@ fn a_seed_replays_exactly() { } } +/// Live documents over the same kind of network (`live.rs`): the documents simulation's random work, with the +/// agents' replicas and the server's sessions talking WebSockets through partitions, held links and latency. +#[test] +fn live_documents_converge_over_a_flaky_network() { + for_each_seed(2, |seed| { + live::run_seed(seed); + }); +} + +#[test] +fn a_live_seed_replays_exactly() { + for seed in 0..3 { + let (a, b) = (live::run_seed(seed), live::run_seed(seed)); + assert!(a.len() > 8, "seed {seed}: too little happened: {a:?}"); + assert_eq!(a, b, "seed {seed} didn't replay exactly"); + } +} + // Scenarios: what the engine and the agent do with each fault, alone. /// A scenario on a calm network (no faults but the ones it arms; pages of 5, so the feed comes in several), with two diff --git a/apps/craft-cloud-agent/tests/recents_sim.rs b/apps/craft-cloud-agent/tests/recents_sim.rs index 10f6967..c85c50f 100644 --- a/apps/craft-cloud-agent/tests/recents_sim.rs +++ b/apps/craft-cloud-agent/tests/recents_sim.rs @@ -35,6 +35,9 @@ use craft_cloud_store::Store; use craft_cloud_store::memory::MemoryBackend; use tokio::sync::broadcast; +#[path = "docs_common/blobs.rs"] +mod blobs; + struct Rng(u64); impl Rng { @@ -118,6 +121,10 @@ fn up_text(frame: &ClientFrame) -> String { ClientFrame::Subscribe { after, .. } => format!("subscribe after {after}"), ClientFrame::Unsubscribe { .. } => "unsubscribe".into(), ClientFrame::Update { client_seq, bytes, .. } => format!("update {client_seq} ({} bytes)", bytes.len()), + ClientFrame::Snapshot { upto, manifest, .. } => format!("snapshot upto {upto} (manifest {manifest})"), + ClientFrame::Presence { bytes, .. } => format!("presence ({} bytes)", bytes.len()), + ClientFrame::SubscribeLog { after, .. } => format!("subscribe the log after {after}"), + ClientFrame::CreateDocument { snapshot, .. } => format!("create a document (manifest {snapshot})"), } } @@ -129,6 +136,10 @@ fn down_text(frame: &ServerFrame) -> String { ServerFrame::Subscribed { head, .. } => format!("subscribed head {head}"), ServerFrame::Ack { client_seq, seq, .. } => format!("ack {client_seq} as {seq}"), ServerFrame::Error { code, message, client_seq, .. } => format!("error {code} {client_seq:?}: {message}"), + ServerFrame::Compact { upto, .. } => format!("compact upto {upto}"), + ServerFrame::Snapshot { base, manifest, .. } => format!("snapshot base {base} (manifest {manifest})"), + ServerFrame::Presence { bytes, .. } => format!("presence ({} bytes)", bytes.len()), + ServerFrame::CreatedDocument { kept, .. } => format!("created, kept {kept}"), } } @@ -308,6 +319,13 @@ impl Sim { /// Delivers the next frame from machine `m` to the server. async fn deliver_up(&mut self, m: usize) { + // A snapshot a frame names goes up before it, through the chunk API (as the hub sends it). + let api = blobs::ServerBlobs { state: self.state.clone(), space: self.space }; + if let Some(replica) = self.machines[m].replica.as_mut() { + for up in replica.take_uploads() { + craft_cloud_agent::live::blobs::upload(&api, &up).await.unwrap(); + } + } let Some(conn) = self.machines[m].conn.as_mut() else { return }; let Some(frame) = conn.up.pop_front() else { return }; let text = format!("m{m} -> server {}", up_text(&frame)); @@ -337,12 +355,20 @@ impl Sim { } /// Delivers the next frame from the server to machine `m`; `flush`: the machine writes what it learnt. - fn deliver_down(&mut self, m: usize, flush: bool) { + async fn deliver_down(&mut self, m: usize, flush: bool) { + let api = blobs::ServerBlobs { state: self.state.clone(), space: self.space }; let machine = &mut self.machines[m]; let (Some(conn), Some(replica)) = (machine.conn.as_mut(), machine.replica.as_mut()) else { return }; let Some(frame) = conn.down.pop_front() else { return }; let text = format!("server -> m{m} {}", down_text(&frame)); - let result = replica.on_frame(frame); + let mut result = replica.on_frame(frame); + // A snapshot a catch-up starts with is downloaded and handed back (as the hub does). + if let Ok(frames) = result.as_mut() { + for (doc, manifest) in replica.take_fetches() { + let got = craft_cloud_agent::live::blobs::download(&api, &manifest).await.map_err(|e| e.to_string()); + frames.extend(replica.on_snapshot(doc, manifest, got)); + } + } if flush { replica.flush().unwrap(); } @@ -841,7 +867,7 @@ impl Sim { self.deliver_old(m).await; self.deliver_up(m).await; self.pump(m).await; - self.deliver_down(m, true); + self.deliver_down(m, true).await; if self.machines[m].conn.is_none() { self.connect(m); } @@ -1115,7 +1141,7 @@ async fn run(seed: u64) -> Vec { 40..=54 => sim.pump(m).await, 55..=79 => { let flush = sim.rng.chance(80); - sim.deliver_down(m, flush); + sim.deliver_down(m, flush).await; } 80..=86 => sim.connect(m), 87..=90 => { diff --git a/apps/craft-cloud-server/src/cli.rs b/apps/craft-cloud-server/src/cli.rs index 03469d9..a9b4908 100644 --- a/apps/craft-cloud-server/src/cli.rs +++ b/apps/craft-cloud-server/src/cli.rs @@ -89,8 +89,9 @@ pub enum Command { #[derive(Debug, Subcommand)] pub enum LiveAction { - /// Empty a user's recents document of an app (one that is full: `quota_exceeded`). Every machine then sends its - /// whole document up again, so nothing any machine holds is lost. + /// Empty a user's recents document of an app (one that is full: `quota_exceeded`), or, given a document's ID + /// instead of an app, that live document of a file. Its log and its snapshot go; every machine then sends its whole + /// document up again, so nothing any machine holds is lost. ResetDoc { username: String, app: String }, } @@ -137,7 +138,8 @@ pub enum CliError { /// Opens the database and blob storage a config names. pub async fn open(config: &Config) -> Result<(Meta, Store), CliError> { - let meta = Meta::open(&config.database.url).await?.with_stored_bytes_cap(config.stored_bytes_per_user); + let meta = + Meta::open(&config.database.url).await?.with_stored_bytes_cap(config.stored_bytes_per_user).with_live_bytes_cap(config.live_bytes_per_user); let store = match (&config.storage, config.storage.s3()?) { (_, Some(s3)) => Store::new(S3Backend::new(&s3)?), (StorageConfig::Disk { path }, None) => Store::new(DiskBackend::new(path)?), @@ -242,6 +244,12 @@ pub async fn run(command: Command, input: &mut impl BufRead, out: &mut impl Writ Command::Live { config, action: LiveAction::ResetDoc { username, app } } => { let (meta, _) = open(&Config::load(&config_path(config)?)?).await?; let user = user_id(&meta, &username).await?; + if let Ok(id) = app.parse::() { + let doc = craft_cloud_proto::DocId::from_uuid(id); + meta.reset_live_doc(user.space, doc).await?; + writeln!(out, "emptied the live document {doc} of {}; every machine sends its own again when it next connects", user.username)?; + return Ok(()); + } let app = craft_cloud_proto::live::AppName::new(&app).map_err(|e| CliError::Usage(e.to_string()))?; let doc = craft_cloud_proto::live::recents_doc(user.space, &app); meta.reset_live_doc(user.space, doc).await?; diff --git a/apps/craft-cloud-server/src/config.rs b/apps/craft-cloud-server/src/config.rs index 017056f..725e9a4 100644 --- a/apps/craft-cloud-server/src/config.rs +++ b/apps/craft-cloud-server/src/config.rs @@ -6,6 +6,7 @@ //! trusted_proxies = [] # proxies whose X-Forwarded-For is believed; needed with behind_proxy //! # ignore_forwarded_for = true # behind a proxy without naming it: every client then counts as the proxy //! # stored_bytes_per_user = 2_000_000_000_000 # cap on what one account stores, in bytes (unset: no cap) +//! # live_bytes_per_user = 1_073_741_824 # cap on what one account's live documents hold, in bytes (1 GiB) //! # public_url = "https://cloud.example.com" # where computers reach the server; `user invite` prints it //! //! [database] @@ -78,6 +79,12 @@ pub struct Config { /// default): no cap. This is not yet the per-user quota on files' logical size (that comes with history). #[serde(default)] pub stored_bytes_per_user: Option, + /// The most bytes one account's live documents may hold together: every update in their logs, compacted ones too + /// (they are kept until retention rules come). An update that would go over it is refused (`quota_exceeded`); the + /// agent keeps it and tries again later, as for a full document. Snapshots aren't counted here: they are uploads, + /// under `stored_bytes_per_user`. Default 1 GiB. + #[serde(default = "default_live_bytes")] + pub live_bytes_per_user: u64, /// The address computers reach the server at, `https://host[:port][/prefix]` (the reverse proxy's). `user invite` /// puts it in front of the code it prints. Normalised when the config is read: the host in lower case, no /// trailing slash. Unset: `user invite` prints the bare code and says to put the address in front. @@ -87,6 +94,10 @@ pub struct Config { pub storage: StorageConfig, } +fn default_live_bytes() -> u64 { + craft_cloud_meta::DEFAULT_LIVE_BYTES_PER_SPACE +} + #[derive(Clone, PartialEq, Eq, Deserialize)] #[serde(deny_unknown_fields)] pub struct DatabaseConfig { @@ -286,6 +297,9 @@ path = "/tmp/blobs" // Under one chunk the agent would count every refusal as a full account: refused at load. assert!(matches!(Config::parse(&format!("stored_bytes_per_user = 4194303\n{DISK}")), Err(ConfigError::StoredBytesCapTooSmall(4_194_303, _)))); assert_eq!(Config::parse(&format!("stored_bytes_per_user = 4194304\n{DISK}")).unwrap().stored_bytes_per_user, Some(4_194_304)); + assert_eq!(Config::parse(DISK).unwrap().live_bytes_per_user, 1 << 30, "1 GiB by default"); + assert_eq!(Config::parse(&format!("live_bytes_per_user = 5_000_000\n{DISK}")).unwrap().live_bytes_per_user, 5_000_000); + assert!(matches!(Config::parse(&format!("live_bytes_per_user = -1\n{DISK}")), Err(ConfigError::Parse(_)))); } #[test] diff --git a/apps/craft-cloud-server/src/error.rs b/apps/craft-cloud-server/src/error.rs index d583c73..73d5a18 100644 --- a/apps/craft-cloud-server/src/error.rs +++ b/apps/craft-cloud-server/src/error.rs @@ -97,7 +97,7 @@ impl From for ApiError { MetaError::PasswordRequired => Self::password_required(), MetaError::UsernameTaken => Self::new(StatusCode::CONFLICT, codes::USERNAME_TAKEN, "the username is taken"), MetaError::QuotaExceeded { used, limit } => Self::quota_exceeded(used, limit), - MetaError::DocFull { .. } | MetaError::TooManyDocs { .. } | MetaError::TooManyDocRows { .. } => { + MetaError::DocFull { .. } | MetaError::LiveFull { .. } | MetaError::TooManyDocs { .. } | MetaError::TooManyDocRows { .. } => { Self::new(StatusCode::INSUFFICIENT_STORAGE, codes::QUOTA_EXCEEDED, e.to_string()) } MetaError::Db(_) | MetaError::Corrupt(_) | MetaError::UnsupportedUrl => { diff --git a/apps/craft-cloud-server/src/lib.rs b/apps/craft-cloud-server/src/lib.rs index 07ebd4e..97ffd43 100644 --- a/apps/craft-cloud-server/src/lib.rs +++ b/apps/craft-cloud-server/src/lib.rs @@ -29,5 +29,11 @@ pub async fn serve(state: AppState, listener: TcpListener, shutdown: impl std::f shutdown.await; state.notifier.begin_shutdown(); }; + // No Nagle delay: a live update is a few dozen bytes, and must go out at once (a keystroke reaches the other + // machines in well under 250 ms). + use axum::serve::ListenerExt; + let listener = listener.tap_io(|tcp| { + let _ = tcp.set_nodelay(true); + }); axum::serve(listener, app.into_make_service_with_connect_info::()).with_graceful_shutdown(shutdown).await } diff --git a/apps/craft-cloud-server/src/live.rs b/apps/craft-cloud-server/src/live.rs index 94a01c9..1cee86e 100644 --- a/apps/craft-cloud-server/src/live.rs +++ b/apps/craft-cloud-server/src/live.rs @@ -23,10 +23,30 @@ //! file sync's epoch doesn't). //! - A document of another account is `not_found`, exactly like one that doesn't exist. //! +//! From live protocol 2 on (a connection whose `hello` names it; one on protocol 1 never sees what follows, and is +//! sent the whole log, compacted rows too): +//! +//! - **Documents of files** are made by import (`create` with a snapshot): the server checks that the ID is the node's +//! and that the node is a file of the space, stores the snapshot in the chunk store and keeps the first one +//! ([`craft_cloud_meta::Meta::create_live_document`]); `created` says whether this one was kept. +//! - **Compaction**: once a document's log above its snapshot holds more than [`Limits::compact_updates`] updates or +//! [`Limits::compact_bytes`], the server asks one device for a snapshot (`compact {doc, upto}`): the one that sent +//! the update that crossed (its ack says it holds every update up to it once the ones below arrive), or one that just +//! caught up (its `subscribed` says so). One request per document at a time, server-wide ([`LiveHub`]); a device +//! that doesn't answer within [`Limits::compact_timeout`] is skipped for [`COMPACT_SKIP`] and the next device to +//! append or catch up is asked. Only the `snapshot` a session was asked for is taken; it is stored in the chunk store +//! first (durably), then recorded with its `base` in one metadata transaction, which takes it only past the current +//! base. The rows below stay. +//! - **Catch-up from a snapshot**: a `subscribe` from below the snapshot (or from 0, for a document made by import) is +//! sent `snapshot {doc, base, bytes}`, then the log from `base` on; `subscribe_log` never is. +//! - **Presence** is relayed to the document's other subscribers (of any device), never stored, at most +//! [`Limits::presence_per_sec`] per document per connection (more are dropped). +//! //! [`Session`] is the protocol, free of the WebSocket: the connection loop ([`connect`]) feeds it frames and //! notifications and sends what it returns. The agent's recents simulation drives sessions directly. use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Arc, Mutex, PoisonError}; use std::time::Duration; @@ -37,14 +57,15 @@ use axum::response::Response; use craft_cloud_meta::DeviceAuth; use craft_cloud_proto::api::codes; use craft_cloud_proto::live::{ - ClientFrame, DocKind, MAX_FRAME_BYTES, MAX_SUBSCRIPTIONS, MAX_UPDATE_BYTES, PROTOCOL_MISMATCH, ServerFrame, shared_version, update_hash, + AppName, ClientFrame, DOCUMENT_KIND_PREFIX, DocKind, LIVE_DOCUMENTS_PROTOCOL, MAX_FRAME_BYTES, MAX_SUBSCRIPTIONS, MAX_UPDATE_BYTES, + PROTOCOL_MISMATCH, ServerFrame, node_doc, shared_version, update_hash, }; -use craft_cloud_proto::{DeviceId, DocId, Epoch, SpaceId}; +use craft_cloud_proto::{DeviceId, DocId, Epoch, ManifestId, NodeId, SpaceId}; use tokio::sync::broadcast; use tokio::time::Instant; use crate::error::ApiError; -use crate::state::AppState; +use crate::state::{AppState, Limits}; /// Live connections one device may hold at once (a reconnect may overlap the old connection for a moment). pub const MAX_CONNECTIONS_PER_DEVICE: usize = 4; @@ -67,22 +88,64 @@ const IDLE_TIMEOUT: Duration = Duration::from_secs(90); const SEND_TIMEOUT: Duration = Duration::from_secs(60); /// Notifications a connection may fall behind by before it rereads all its documents. const NOTIFY_CAPACITY: usize = 256; +/// Presence messages a connection may fall behind by; past that the oldest are dropped (presence is ephemeral). +const PRESENCE_CAPACITY: usize = 256; +/// How long a device asked to compact has to answer before another one is asked. +pub const COMPACT_TIMEOUT: Duration = Duration::from_secs(30); +/// How long a device that didn't answer a compaction request in time isn't asked again for that document. +pub const COMPACT_SKIP: Duration = Duration::from_secs(600); + +/// A presence message on its way to a space's other connections. +#[derive(Debug, Clone)] +pub struct PresenceNote { + pub doc: DocId, + pub bytes: Arc>, + /// The session it came from (not sent back to it). + pub from: u64, +} + +/// One document's compaction: the request out (to which session and device, for which `upto`, since when), and the +/// devices that let one time out, until when they are skipped. +#[derive(Debug, Default)] +struct Compaction { + asked: Option<(u64, DeviceId, Instant)>, + skipped: HashMap, +} -/// Who is connected: per space, a channel naming each document that got an update; per device, its connections and -/// subscriptions; and a channel naming each device revoked through the API (its connections end at once). +/// Who is connected: per space, a channel naming each document that got an update, and one of presence; per device, +/// its connections and subscriptions; a channel naming each device revoked through the API (its connections end at +/// once); and the compactions under way. pub struct LiveHub { spaces: Mutex>>, + presence: Mutex>>, connections: Mutex>, subscriptions: Mutex>, revoked: broadcast::Sender, + compactions: Mutex>, + /// Each device's presence budget per document, across its connections. + presence_budgets: Mutex>, } +/// Presence budgets kept at most (a full one is as good as none, and goes first). +const MAX_PRESENCE_BUDGETS: usize = 100_000; + impl Default for LiveHub { fn default() -> Self { - Self { spaces: Mutex::default(), connections: Mutex::default(), subscriptions: Mutex::default(), revoked: broadcast::channel(64).0 } + Self { + spaces: Mutex::default(), + presence: Mutex::default(), + connections: Mutex::default(), + subscriptions: Mutex::default(), + revoked: broadcast::channel(64).0, + compactions: Mutex::default(), + presence_budgets: Mutex::default(), + } } } +/// Sessions are numbered, so a presence message isn't sent back to the one it came from. +static NEXT_SESSION: AtomicU64 = AtomicU64::new(1); + /// One of a device's live connections; frees its slot when dropped. pub struct ConnectionSlot { hub: Arc, @@ -143,6 +206,71 @@ impl LiveHub { spaces.remove(&space); } } + + /// A receiver of the presence messages of `space` from now on. + pub fn listen_presence(&self, space: SpaceId) -> broadcast::Receiver { + let mut spaces = self.presence.lock().unwrap_or_else(PoisonError::into_inner); + spaces.entry(space).or_insert_with(|| broadcast::channel(PRESENCE_CAPACITY).0).subscribe() + } + + /// Hands a presence message to the space's connections (never stored). A space nobody listens to is forgotten. + pub fn publish_presence(&self, space: SpaceId, note: PresenceNote) { + let mut spaces = self.presence.lock().unwrap_or_else(PoisonError::into_inner); + if let Some(sender) = spaces.get(&space) + && sender.send(note).is_err() + { + spaces.remove(&space); + } + } + + /// Whether `session` of `device` may be asked to compact `doc` now: no request is out for it (or the one out timed + /// out, and its device is skipped for a while), and this device isn't skipped. If so, the request is recorded. + fn ask_compaction(&self, space: SpaceId, doc: DocId, session: u64, device: DeviceId, limits: &Limits) -> bool { + let now = Instant::now(); + let mut all = self.compactions.lock().unwrap_or_else(PoisonError::into_inner); + let entry = all.entry((space, doc)).or_default(); + if let Some((_, asked, at)) = entry.asked { + if now.saturating_duration_since(at) < limits.compact_timeout { + return false; + } + entry.skipped.insert(asked, now + COMPACT_SKIP); + entry.asked = None; + } + entry.skipped.retain(|_, until| *until > now); + if entry.skipped.contains_key(&device) { + return false; + } + entry.asked = Some((session, device, now)); + true + } + + /// Whether `device` may send one more presence message for `doc` now (`per_sec` a second, across its connections). + fn spend_presence(&self, device: DeviceId, doc: DocId, per_sec: u32) -> bool { + let mut all = self.presence_budgets.lock().unwrap_or_else(PoisonError::into_inner); + if all.len() >= MAX_PRESENCE_BUDGETS { + // The ones refilled to the full are no different from new ones. + all.retain(|_, budget| !budget.is_full()); + } + all.entry((device, doc)).or_insert_with(|| Budget::of(per_sec, per_sec)).spend() + } + + /// `doc` was compacted: a later request may go to any device again. + fn compacted(&self, space: SpaceId, doc: DocId) { + self.compactions.lock().unwrap_or_else(PoisonError::into_inner).remove(&(space, doc)); + } + + /// The request out for `doc` to `session`, if any, is done with (the session ended): another device may be asked. + fn forget_ask(&self, space: SpaceId, doc: DocId, session: u64) { + let mut all = self.compactions.lock().unwrap_or_else(PoisonError::into_inner); + if let Some(entry) = all.get_mut(&(space, doc)) { + if entry.asked.is_some_and(|(s, ..)| s == session) { + entry.asked = None; + } + if entry.asked.is_none() && entry.skipped.is_empty() { + all.remove(&(space, doc)); + } + } + } } /// A token bucket: `burst` at once, refilled at `per_sec`. @@ -155,8 +283,17 @@ struct Budget { impl Budget { fn new(limits: &crate::state::Limits) -> Self { - let burst = f64::from(limits.live_burst); - Self { tokens: burst, at: Instant::now(), burst, per_sec: f64::from(limits.live_per_sec) } + Self::of(limits.live_burst, limits.live_per_sec) + } + + fn of(burst: u32, per_sec: u32) -> Self { + let burst = f64::from(burst); + Self { tokens: burst, at: Instant::now(), burst, per_sec: f64::from(per_sec) } + } + + /// Whether it has refilled to the full by now. + fn is_full(&self) -> bool { + self.tokens + Instant::now().saturating_duration_since(self.at).as_secs_f64() * self.per_sec >= self.burst } fn spend(&mut self) -> bool { @@ -181,6 +318,10 @@ struct Subscription { /// The catch-up after `subscribe` is under way: the agent's own updates are sent too (it may have lost them), and /// `subscribed` follows once the log is read to its end. catching_up: bool, + /// The catch-up starts with the document's snapshot if the agent's `after` (this) is below it, or 0. + snapshot_from: Option, + /// …with the document's import instead (`subscribe_log` from 0: the whole log follows, which doesn't hold it). + import: bool, } /// What [`Session::handle`] answers. @@ -197,17 +338,26 @@ pub struct Session { space: SpaceId, device: DeviceId, welcomed: bool, + /// The live protocol agreed at `hello` (0 before). + protocol: u32, + /// This session's number (presence isn't sent back to it). + id: u64, /// The space's epoch named in `welcome`: a connection that outlives an epoch reset ends at its next ping. epoch: Option, subscriptions: BTreeMap, /// Documents with updates the agent may not have been sent yet. dirty: BTreeSet, budget: Budget, + /// The `upto` this session's agent was asked to compact each document at: only that snapshot is taken from it. + asked: BTreeMap, } impl Drop for Session { fn drop(&mut self) { release(&self.state.live.subscriptions, self.device, self.subscriptions.len()); + for doc in self.asked.keys() { + self.state.live.forget_ask(self.space, *doc, self.id); + } } } @@ -225,13 +375,21 @@ impl Session { space: auth.space, device: auth.device, welcomed: false, + protocol: 0, + id: NEXT_SESSION.fetch_add(1, Ordering::Relaxed), epoch: None, subscriptions: BTreeMap::new(), dirty: BTreeSet::new(), budget, + asked: BTreeMap::new(), } } + /// Whether the agent speaks documents, snapshots and presence (protocol 2 on). + fn documents(&self) -> bool { + self.protocol >= LIVE_DOCUMENTS_PROTOCOL + } + /// Handles one frame from the agent. pub async fn handle(&mut self, frame: ClientFrame) -> Reply { let one = |frame| Reply { frames: vec![frame], close: false }; @@ -241,6 +399,7 @@ impl Session { Some(protocol) => match self.state.meta.live_epoch(self.space).await { Ok(epoch) => { self.welcomed = true; + self.protocol = protocol; self.epoch = Some(epoch); one(ServerFrame::Welcome { protocol, epoch }) } @@ -269,7 +428,7 @@ impl Session { match frame { ClientFrame::Hello { .. } => one(ServerFrame::error(codes::INVALID, "hello was said already", None, None)), ClientFrame::Create { doc, kind } => one(self.create(doc, kind).await), - ClientFrame::Subscribe { doc, after, at } => match self.subscribe(doc, after, at).await { + ClientFrame::Subscribe { doc, after, at } => match self.subscribe(doc, after, at, false).await { Ok(()) => Reply::default(), Err(e) => one(refused(e, Some(doc), None)), }, @@ -280,13 +439,112 @@ impl Session { self.dirty.remove(&doc); Reply::default() } - ClientFrame::Update { doc, client_seq, bytes } => one(match self.update(doc, client_seq, bytes).await { - Ok(seq) => ServerFrame::Ack { doc, client_seq, seq }, - Err(e) => refused(e, Some(doc), Some(client_seq)), - }), + ClientFrame::Update { doc, client_seq, bytes } => match self.update(doc, client_seq, bytes).await { + Ok((seq, compact)) => { + Reply { frames: std::iter::once(ServerFrame::Ack { doc, client_seq, seq }).chain(compact).collect(), close: false } + } + Err(e) => one(refused(e, Some(doc), Some(client_seq))), + }, + // The frames of documents, snapshots and presence are for a connection that speaks them. + ClientFrame::CreateDocument { doc, .. } | ClientFrame::SubscribeLog { doc, .. } if !self.documents() => { + one(refused(ApiError::invalid("that frame needs live protocol 2"), Some(doc), None)) + } + ClientFrame::Snapshot { .. } | ClientFrame::Presence { .. } if !self.documents() => Reply::default(), + ClientFrame::CreateDocument { doc, node, app, snapshot } => one(self.create_document(doc, node, app, snapshot).await), + ClientFrame::SubscribeLog { doc, after, at } => match self.subscribe(doc, after, at, true).await { + Ok(()) => Reply::default(), + Err(e) => one(refused(e, Some(doc), None)), + }, + ClientFrame::Snapshot { doc, upto, manifest } => { + self.snapshot(doc, upto, manifest).await; + Reply::default() + } + ClientFrame::Presence { doc, bytes } => { + self.presence(doc, bytes); + Reply::default() + } } } + /// `create {doc, node, app, snapshot}`: the document of a file, made by import from the snapshot `manifest` names + /// (uploaded through the chunk API already, and charged to the account there). Recorded only for a file of the + /// space and a manifest of the space, both checked in the same transaction (the file is looked at first too, so a + /// made-up node costs a read). The first snapshot is kept; a document that is there already answers `kept: false`. + async fn create_document(&mut self, doc: DocId, node: NodeId, app: AppName, manifest: ManifestId) -> ServerFrame { + if !self.budget.spend() { + return refused(ApiError::rate_limited("too many frames; send again in a moment", None), Some(doc), None); + } + if node_doc(self.space, node) != doc { + return refused(ApiError::invalid("that document ID isn't the file's"), Some(doc), None); + } + match self.state.meta.live_doc(self.space, doc).await { + Ok(Some(found)) if found.node == Some(node) && found.kind.starts_with(DOCUMENT_KIND_PREFIX) => { + return ServerFrame::CreatedDocument { doc, kept: false }; + } + Ok(Some(_)) => return refused(ApiError::invalid("the document exists with another kind"), Some(doc), None), + Ok(None) => {} + Err(e) => return refused(e.into(), Some(doc), None), + } + match self.state.meta.is_live_file(self.space, node).await { + Ok(true) => {} + Ok(false) => return refused(ApiError::not_found("file"), Some(doc), None), + Err(e) => return refused(e.into(), Some(doc), None), + } + let kind = DocKind::Document { node, app }.as_text(); + match self.state.meta.create_live_document(self.space, doc, &kind, node, &manifest, self.state.now_ms()).await { + Ok((_, kept)) => ServerFrame::CreatedDocument { doc, kept }, + Err(e) => refused(e.into(), Some(doc), None), + } + } + + /// `snapshot {doc, upto, manifest}`: the answer to this session's `compact`, taken only for the `upto` it was + /// asked, and only for a manifest the account uploaded; recorded past the current base only. Never answered: a + /// snapshot that isn't taken changes nothing, and another device is asked later. + async fn snapshot(&mut self, doc: DocId, upto: u64, manifest: ManifestId) { + if !self.subscriptions.contains_key(&doc) || self.asked.get(&doc) != Some(&upto) { + return; + } + self.asked.remove(&doc); + match self.state.meta.compact_live_doc(self.space, doc, upto, &manifest).await { + Ok(true) => self.state.live.compacted(self.space, doc), + Ok(false) => {} + Err(e) => tracing::info!(error = %e, "a live document's snapshot wasn't recorded"), + } + } + + /// `presence {doc, bytes}`: relayed to the document's other subscribers, within the device's budget for the + /// document (across its connections) and the connection's frame budget (over either, dropped). + fn presence(&mut self, doc: DocId, bytes: Vec) { + if !self.subscriptions.contains_key(&doc) { + return; + } + if !self.state.live.spend_presence(self.device, doc, self.state.limits.presence_per_sec) || !self.budget.spend() { + return; + } + self.state.live.publish_presence(self.space, PresenceNote { doc, bytes: Arc::new(bytes), from: self.id }); + } + + /// Another connection's presence message: the frame for this one's agent, if it follows the document. + pub fn relay(&self, note: &PresenceNote) -> Option { + (self.documents() && note.from != self.id && self.subscriptions.contains_key(¬e.doc)) + .then(|| ServerFrame::Presence { doc: note.doc, bytes: note.bytes.as_ref().clone() }) + } + + /// The `compact` to ask this session's agent for, if the log above the snapshot is long (`upto - base` updates, + /// `above` bytes) and no other device is asked already. + fn maybe_compact(&mut self, doc: DocId, upto: u64, base: u64, above: u64) -> Option { + let limits = &self.state.limits; + let long = upto.saturating_sub(base) > limits.compact_updates || above > limits.compact_bytes; + if !self.documents() || !long || self.asked.get(&doc).is_some_and(|asked| *asked >= upto) { + return None; + } + if !self.state.live.ask_compaction(self.space, doc, self.id, self.device, limits) { + return None; + } + self.asked.insert(doc, upto); + Some(ServerFrame::Compact { doc, upto }) + } + async fn create(&mut self, doc: DocId, kind: DocKind) -> ServerFrame { if !self.budget.spend() { return refused(ApiError::rate_limited("too many frames; send again in a moment", None), Some(doc), None); @@ -302,7 +560,9 @@ impl Session { } } - async fn subscribe(&mut self, doc: DocId, after: u64, at: Option<[u8; 32]>) -> Result<(), ApiError> { + /// `subscribe` (or, with `log_only`, `subscribe_log`): the catch-up from `after`, with the document's snapshot first + /// when `after` is below it (a connection on protocol 1, and `subscribe_log`, get the whole log instead). + async fn subscribe(&mut self, doc: DocId, after: u64, at: Option<[u8; 32]>, log_only: bool) -> Result<(), ApiError> { // A subscribe costs a read of the log: it comes out of the same budget as updates. if !self.budget.spend() { return Err(ApiError::rate_limited("too many frames; send again in a moment", None)); @@ -330,6 +590,13 @@ impl Session { } return Err(ApiError::not_found("document")); }; + // A document of a file is for an agent that reads its snapshot: one on protocol 1 never made one. + if !self.documents() && found.kind.starts_with(DOCUMENT_KIND_PREFIX) { + if new { + release(&self.state.live.subscriptions, self.device, 1); + } + return Err(ApiError::invalid("documents of files need live protocol 2")); + } // The update the agent has at `after` must be the one the log has there: if it isn't (the server was restored // from a backup and its epoch not reset yet), the agent catches up from the start and sends its whole document. if let Some(at) = at @@ -358,12 +625,18 @@ impl Session { } // An agent that holds more than the log (the server was restored from a backup) is told the log's end in // `subscribed` and sends its own state up again. - self.subscriptions.insert(doc, Subscription { sent: after.min(found.head), catching_up: true }); + let sent = after.min(found.head); + // From below the snapshot (or from 0): the snapshot first. The whole log (`subscribe_log`) from 0 of a document + // made by import: its import first, which no row holds. + let import = log_only && sent == 0 && found.kind.starts_with(DOCUMENT_KIND_PREFIX); + let snapshot_from = (self.documents() && found.has_snapshot && (import || (!log_only && (sent < found.base || sent == 0)))).then_some(sent); + self.subscriptions.insert(doc, Subscription { sent, catching_up: true, snapshot_from, import }); self.dirty.insert(doc); Ok(()) } - async fn update(&mut self, doc: DocId, client_seq: u64, bytes: Vec) -> Result { + /// Appends an update; the `compact` to ask for after its ack, if its log is long now. + async fn update(&mut self, doc: DocId, client_seq: u64, bytes: Vec) -> Result<(u64, Option), ApiError> { if !self.subscriptions.contains_key(&doc) { return Err(ApiError::not_found("subscribed document")); } @@ -376,16 +649,20 @@ impl Session { // Appended and published in a task of its own: a connection that drops meanwhile must not skip the publish of // an update that was stored. let (state, space, device) = (self.state.clone(), self.space, self.device); - tokio::spawn(async move { - let seq = state.meta.append_live_update(space, doc, device, client_seq, &bytes, state.now_ms()).await?; + let appended = tokio::spawn(async move { + let appended = state.meta.append_live_update_counted(space, doc, device, client_seq, &bytes, state.now_ms()).await?; state.live.publish(space, doc); - Ok::<_, ApiError>(seq) + Ok::<_, ApiError>(appended) }) .await .map_err(|e| { tracing::error!(error = %e, "live append task failed"); ApiError::internal() - })? + })??; + // The device that sent the update that made the log long is asked first: it holds every update up to it once + // those below have reached it. + let compact = if appended.repeat { None } else { self.maybe_compact(doc, appended.seq, appended.base, appended.above_bytes) }; + Ok((appended.seq, compact)) } /// `doc` got an update: if it is subscribed, the next [`Session::pump`]s send what the agent lacks. @@ -435,6 +712,21 @@ impl Session { self.dirty.remove(&doc); return Ok(Vec::new()); }; + // The snapshot first, as it is now (a compaction since the `subscribe` only moves it on): alone in its batch, + // then the log from its base on (the update at the base too, whose hash the agent's next `at` names). + if let Some(after) = sub.snapshot_from.take() { + let found = if sub.import { + self.state.meta.live_import_snapshot(self.space, doc).await?.map(|import| (0, import)) + } else { + self.state.meta.live_snapshot(self.space, doc).await? + }; + if let Some((base, stored)) = found.filter(|(base, _)| after < *base || after == 0) { + // The agent downloads it through the chunk API (one it can't get or read, it follows the whole log). + sub.sent = base.saturating_sub(1).max(after); + self.subscriptions.insert(doc, sub); + return Ok(vec![ServerFrame::Snapshot { doc, base, manifest: stored.manifest }]); + } + } let (page, full) = self.state.meta.live_page(self.space, doc, sub.sent, PUMP_BYTES, PUMP_PAGE).await?; let mut frames = Vec::new(); for update in page { @@ -443,14 +735,24 @@ impl Session { frames.push(ServerFrame::Update { doc, seq: update.seq, bytes: update.bytes, device: update.device }); } } + let mut caught_up = false; if !full { self.dirty.remove(&doc); if sub.catching_up { sub.catching_up = false; + caught_up = true; frames.push(ServerFrame::Subscribed { doc, head: sub.sent }); } } self.subscriptions.insert(doc, sub); + // An agent that just caught up holds the whole log: it may be asked to compact it. + if caught_up + && self.documents() + && let Some(found) = self.state.meta.live_doc(self.space, doc).await? + && found.head == sub.sent + { + frames.extend(self.maybe_compact(doc, found.head, found.base, found.above_base())); + } Ok(frames) } } @@ -466,11 +768,13 @@ pub async fn connect( return Err(ApiError::rate_limited(format!("a device may hold at most {MAX_CONNECTIONS_PER_DEVICE} live connections"), Some(5))); } let slot = ConnectionSlot { hub: state.clone(), device: auth.device }; + // No frame is over an update and its envelope (snapshots travel through the chunk API). Ok(upgrade.max_message_size(MAX_FRAME_BYTES).max_frame_size(MAX_FRAME_BYTES).on_upgrade(move |socket| async move { let _slot = slot; // Listening before the session reads anything, so no update falls between a read and the notification. let notes = state.live.listen(auth.space); - serve(socket, Session::new(state, &auth), notes).await; + let presence = state.live.listen_presence(auth.space); + serve(socket, Session::new(state, &auth), notes, presence).await; })) } @@ -497,7 +801,7 @@ async fn send_all(socket: &mut WebSocket, frames: &[ServerFrame]) -> bool { /// The connection loop: frames in, frames out, the log pumped while a document has news, pings, and an end on /// shutdown, silence or a broken connection. -async fn serve(mut socket: WebSocket, mut session: Session, mut notes: broadcast::Receiver) { +async fn serve(mut socket: WebSocket, mut session: Session, mut notes: broadcast::Receiver, mut presence: broadcast::Receiver) { let mut stopping = session.state.notifier.shutdown_signal(); let mut revocations = session.state.live.revocations(); let mut last_heard = Instant::now(); @@ -530,6 +834,18 @@ async fn serve(mut socket: WebSocket, mut session: Session, mut notes: broadcast // The hub keeps the sender while anyone listens to the space, and this connection does. Err(broadcast::error::RecvError::Closed) => break, }, + note = presence.recv() => match note { + Ok(note) => { + if let Some(frame) = session.relay(¬e) + && !send(&mut socket, &frame).await + { + break; + } + } + // Presence that fell behind is dropped: the next one says where the cursor is. + Err(broadcast::error::RecvError::Lagged(_)) => {} + Err(broadcast::error::RecvError::Closed) => break, + }, revoked = revocations.recv() => { let ours = match revoked { Ok(device) => device == session.device, @@ -747,4 +1063,505 @@ mod tests { let Some(ServerFrame::Error { code, .. }) = s.check().await else { panic!("the reset went unseen") }; assert_eq!(code, "resync_required"); } + + // ---- documents of files, snapshots, compaction and presence (protocol 2) ---- + + use craft_cloud_proto::{ChunkId, ChunkRef, ChunkerParamsId, FileHash, Manifest, Name, NewVersion, Op}; + use craft_cloud_store::BlobKey; + + /// One account's state, with devices to open sessions for. + struct Account { + state: Arc, + meta: Meta, + user: craft_cloud_meta::UserRecord, + next_token: u8, + } + + impl Account { + async fn new(limits: crate::state::Limits) -> Self { + Self::with_meta(limits, Meta::open("sqlite::memory:").await.unwrap(), "ana").await + } + + /// Another account (`name`) in the same database. + async fn with_meta(limits: crate::state::Limits, meta: Meta, name: &str) -> Self { + let user = meta.create_user(name, "$argon2id$stub", Role::Member, 0).await.unwrap(); + let state = Arc::new(AppState::new(meta.clone(), Store::new(MemoryBackend::new()), vec![]).unwrap().with_limits(limits)); + Self { state, meta, user, next_token: 1 } + } + + /// A new device of the account and a session for it, past a `hello` naming `max`. + async fn session(&mut self, max: u32) -> (Session, DeviceAuth) { + let token = [self.next_token; 32]; + self.next_token += 1; + self.meta.create_device(self.user.id, "pc", &token, 0).await.unwrap(); + let auth = self.meta.device_by_token(&token).await.unwrap().unwrap(); + let mut session = Session::new(self.state.clone(), &auth); + let reply = session.handle(ClientFrame::Hello { min: 1, max }).await; + assert!(matches!(reply.frames.as_slice(), [ServerFrame::Welcome { protocol, .. }] if *protocol == max), "{:?}", reply.frames); + (session, auth) + } + + /// `bytes` uploaded as the account's through what the chunk API does (each chunk, then the manifest naming + /// them): what an agent does with a snapshot before the frame that names it. + async fn upload(&self, bytes: &[u8]) -> ManifestId { + let space = self.user.space; + let mut chunks = Vec::new(); + for piece in bytes.chunks(craft_cloud_proto::limits::MAX_CHUNK_BYTES as usize) { + let id = ChunkId::from_bytes(*blake3::hash(piece).as_bytes()); + self.state.store.put(BlobKey::chunk(&id), piece).await.unwrap(); + self.meta.add_chunk(space, &id, piece.len() as u64 + 1, 0).await.unwrap(); + chunks.push(ChunkRef { id, len: u32::try_from(piece.len()).unwrap() }); + } + let file_hash = FileHash::from_bytes(*blake3::hash(bytes).as_bytes()); + let manifest = Manifest::new(ChunkerParamsId::new("fastcdc-v1").unwrap(), bytes.len() as u64, file_hash, chunks).unwrap(); + self.state.store.put(BlobKey::manifest(&manifest.id()), &manifest.encode()).await.unwrap(); + self.meta.add_manifest(space, &manifest, 100, 0).await.unwrap(); + manifest.id() + } + + /// A file of the account's space. + async fn file(&self, auth: &DeviceAuth, name: &str) -> NodeId { + let space = self.user.space; + let chunk = ChunkId::from_bytes(update_hash(name.as_bytes())); + self.meta.add_chunk(space, &chunk, 11, 0).await.unwrap(); + let manifest = + Manifest::new(ChunkerParamsId::new("fastcdc-v1").unwrap(), 10, FileHash::from_bytes([1; 32]), vec![ChunkRef { id: chunk, len: 10 }]) + .unwrap(); + self.meta.add_manifest(space, &manifest, 100, 0).await.unwrap(); + let root = self.meta.space_info(space).await.unwrap().root; + let node = NodeId::generate(); + let version = NewVersion { id: craft_cloud_proto::VersionId::generate(), manifest: manifest.id(), mtime_ms: 0, exec: false }; + let op = Op::Create { node, parent: root, name: Name::new(name).unwrap(), version }; + self.meta.commit(space, auth.device, &[op], 0).await.unwrap(); + node + } + } + + fn app() -> AppName { + AppName::new("wordcraft").unwrap() + } + + /// Pumps until nothing is left: every frame the agent would be sent. + async fn drain(s: &mut Session) -> Vec { + let mut out = Vec::new(); + while s.has_work() { + out.extend(s.pump().await.unwrap()); + } + out + } + + /// Makes the document of a new file by import from `s` (its snapshot uploaded first) and follows it from the start. + async fn imported(a: &Account, s: &mut Session, auth: &DeviceAuth, snapshot: &[u8]) -> DocId { + let node = a.file(auth, &format!("f{}.txt", DocId::generate())).await; + let doc = node_doc(a.user.space, node); + let snapshot = a.upload(snapshot).await; + let made = s.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot }).await; + assert_eq!(made.frames, vec![ServerFrame::CreatedDocument { doc, kept: true }]); + s.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + drain(s).await; + doc + } + + fn updates(frames: &[ServerFrame]) -> Vec { + frames.iter().filter_map(|f| if let ServerFrame::Update { seq, .. } = f { Some(*seq) } else { None }).collect() + } + + /// The snapshot a catch-up named: its base and its manifest. + fn snapshot_of(frames: &[ServerFrame]) -> Option<(u64, ManifestId)> { + frames.iter().find_map(|f| if let ServerFrame::Snapshot { base, manifest, .. } = f { Some((*base, *manifest)) } else { None }) + } + + fn compacts(frames: &[ServerFrame]) -> Vec { + frames.iter().filter_map(|f| if let ServerFrame::Compact { upto, .. } = f { Some(*upto) } else { None }).collect() + } + + fn small_logs() -> crate::state::Limits { + crate::state::Limits { compact_updates: 3, ..Default::default() } + } + + /// An agent on protocol 1 (one from before documents) works on as it did: it can't make a document of a file nor + /// follow one, presence and snapshots from it are dropped, it is never asked to compact, and it is sent a compacted + /// log whole, its compacted rows too. + #[tokio::test] + async fn an_agent_on_protocol_1_never_sees_documents() { + let mut a = Account::new(crate::state::Limits { compact_updates: 1, ..Default::default() }).await; + let (mut old, auth) = a.session(1).await; + let node = a.file(&auth, "a.txt").await; + let doc = node_doc(a.user.space, node); + let refused = old.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: a.upload(&[1; 10]).await }).await; + assert_eq!(code(&refused), Some("invalid")); + let recents = recents_doc(a.user.space, &app()); + old.handle(ClientFrame::Create { doc: recents, kind: DocKind::Recents { app: app() } }).await; + old.handle(ClientFrame::Subscribe { doc: recents, after: 0, at: None }).await; + drain(&mut old).await; + for client_seq in 1..=4 { + let reply = old.handle(ClientFrame::Update { doc: recents, client_seq, bytes: vec![client_seq as u8; 10] }).await; + assert!(matches!(reply.frames.as_slice(), [ServerFrame::Ack { .. }]), "never a compact: {:?}", reply.frames); + } + assert!(old.handle(ClientFrame::Presence { doc: recents, bytes: vec![1] }).await.frames.is_empty()); + let one = a.upload(&[1]).await; + assert!(old.handle(ClientFrame::Snapshot { doc: recents, upto: 2, manifest: one }).await.frames.is_empty()); + assert_eq!(a.meta.live_doc(a.user.space, recents).await.unwrap().unwrap().base, 0, "an unasked snapshot changes nothing"); + // Compacted by another (newer) device: the old one still gets every row, never a snapshot. + let stored = a.upload(&[9; 20]).await; + assert!(a.meta.compact_live_doc(a.user.space, recents, 3, &stored).await.unwrap()); + let (mut old2, _) = a.session(1).await; + old2.handle(ClientFrame::Subscribe { doc: recents, after: 0, at: None }).await; + let frames = drain(&mut old2).await; + assert_eq!(updates(&frames), vec![1, 2, 3, 4]); + assert!(snapshot_of(&frames).is_none()); + // Nor may it follow a document of a file. + let (mut new, auth) = a.session(2).await; + let doc = imported(&a, &mut new, &auth, &[1; 10]).await; + assert_eq!(code(&old2.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await), Some("invalid")); + } + + /// A document of a file is made by import: the first snapshot is kept, the next one (another machine's import of + /// the same file) isn't, and a machine that follows it from the start gets the snapshot first. + #[tokio::test] + async fn a_document_is_made_by_import_and_the_first_snapshot_kept() { + let mut a = Account::new(Default::default()).await; + let (mut pc, pc_auth) = a.session(2).await; + let (mut laptop, _) = a.session(2).await; + let node = a.file(&pc_auth, "doc1.txt").await; + let doc = node_doc(a.user.space, node); + let (one, two) = (a.upload(&[1; 3000]).await, a.upload(&[2; 3000]).await); + let first = pc.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: one }).await; + assert_eq!(first.frames, vec![ServerFrame::CreatedDocument { doc, kept: true }]); + let second = laptop.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: two }).await; + assert_eq!(second.frames, vec![ServerFrame::CreatedDocument { doc, kept: false }]); + laptop.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + let frames = drain(&mut laptop).await; + assert_eq!(snapshot_of(&frames), Some((0, one)), "the kept import"); + assert_eq!(frames.last(), Some(&ServerFrame::Subscribed { doc, head: 0 })); + // The ID must be the node's, the node a file of this account's space. + let other = a.file(&pc_auth, "doc2.txt").await; + let wrong = pc.handle(ClientFrame::CreateDocument { doc, node: other, app: app(), snapshot: one }).await; + assert_eq!(code(&wrong), Some("invalid")); + let missing = NodeId::generate(); + let gone = pc.handle(ClientFrame::CreateDocument { doc: node_doc(a.user.space, missing), node: missing, app: app(), snapshot: one }).await; + assert_eq!(code(&gone), Some("not_found")); + let mut b = Account::with_meta(Default::default(), a.meta.clone(), "bo").await; + b.next_token = 100; + let (mut bo, _) = b.session(2).await; + let bos = b.upload(&[1]).await; + let theirs = bo.handle(ClientFrame::CreateDocument { doc: node_doc(b.user.space, node), node, app: app(), snapshot: bos }).await; + assert_eq!(code(&theirs), Some("not_found"), "another account's file is no file of this one"); + assert_eq!(code(&bo.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await), Some("not_found"), "nor its document"); + } + + /// Snapshots never travel in frames: a `create` or `snapshot` names a manifest the account uploaded (through the + /// chunk API, charged there), and one naming a manifest it doesn't have (never uploaded, or another account's) + /// records nothing. A refused `create` stores nothing either: the server writes no blob for a frame. + #[tokio::test] + async fn a_snapshot_is_named_by_a_manifest_the_account_uploaded() { + let meta = Meta::open("sqlite::memory:").await.unwrap().with_stored_bytes_cap(Some(1000)); + let mut a = Account::with_meta(small_logs(), meta, "ana").await; + let (mut s, auth) = a.session(2).await; + // Made-up nodes: refused, nothing stored, nothing charged. + let used = a.meta.stored_bytes(a.user.space).await.unwrap(); + for i in 0..5u8 { + let node = NodeId::generate(); + let reply = s + .handle(ClientFrame::CreateDocument { + doc: node_doc(a.user.space, node), + node, + app: app(), + snapshot: ManifestId::from_bytes([i; 32]), + }) + .await; + assert_eq!(code(&reply), Some("not_found")); + } + assert_eq!(a.meta.stored_bytes(a.user.space).await.unwrap(), used); + // A file, but a manifest never uploaded, or another account's: refused, nothing recorded. + let node = a.file(&auth, "a.txt").await; + let doc = node_doc(a.user.space, node); + let never = ManifestId::from_bytes([3; 32]); + assert_eq!(code(&s.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: never }).await), Some("not_found")); + let mut b = Account::with_meta(Default::default(), a.meta.clone(), "bo").await; + b.next_token = 100; + let bos = b.upload(&[5; 10]).await; + assert_eq!(code(&s.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: bos }).await), Some("not_found")); + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap(), None); + // A compaction answer naming one it doesn't have isn't taken (and the next ask goes ahead). + let doc = imported(&a, &mut s, &auth, &[1; 10]).await; + let mut asked = Vec::new(); + for client_seq in 1..=4u64 { + asked.extend(compacts(&s.handle(ClientFrame::Update { doc, client_seq, bytes: vec![client_seq as u8; 10] }).await.frames)); + } + assert_eq!(asked, vec![4]); + s.handle(ClientFrame::Snapshot { doc, upto: 4, manifest: bos }).await; + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().base, 0, "another account's manifest isn't taken"); + } + + /// Two machines import the same never-live file at the same moment: one import is kept, and both end on it. + #[tokio::test] + async fn two_imports_at_once_keep_one() { + let mut a = Account::new(Default::default()).await; + let (mut pc, auth) = a.session(2).await; + let (mut laptop, _) = a.session(2).await; + let node = a.file(&auth, "a.txt").await; + let doc = node_doc(a.user.space, node); + let (one, two) = (a.upload(&[1; 100]).await, a.upload(&[2; 100]).await); + let (x, y) = tokio::join!( + pc.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: one }), + laptop.handle(ClientFrame::CreateDocument { doc, node, app: app(), snapshot: two }) + ); + let kept: Vec = [x, y] + .iter() + .map(|r| match r.frames.as_slice() { + [ServerFrame::CreatedDocument { kept, .. }] => *kept, + other => panic!("{other:?}"), + }) + .collect(); + assert_eq!(kept.iter().filter(|k| **k).count(), 1, "{kept:?}"); + let winner = if kept[0] { one } else { two }; + for s in [&mut pc, &mut laptop] { + s.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + assert_eq!(snapshot_of(&drain(s).await), Some((0, winner)), "both end on the kept one"); + } + } + + /// A log that grows long is compacted by the device whose update made it long: asked once, its snapshot taken for + /// the `upto` asked, and a machine catching up from below it then gets the snapshot and the log from its base. + #[tokio::test] + async fn a_long_log_is_compacted_by_the_device_that_made_it_long() { + let mut a = Account::new(small_logs()).await; + let (mut pc, auth) = a.session(2).await; + let doc = imported(&a, &mut pc, &auth, &[1; 10]).await; + let mut asked = Vec::new(); + for client_seq in 1..=5u64 { + let reply = pc.handle(ClientFrame::Update { doc, client_seq, bytes: vec![client_seq as u8; 10] }).await; + assert!(matches!(reply.frames.first(), Some(ServerFrame::Ack { seq, .. }) if *seq == client_seq), "{:?}", reply.frames); + asked.extend(compacts(&reply.frames)); + } + assert_eq!(asked, vec![4], "past 3 updates, asked once"); + // Another `upto` than asked, or the right one twice: only the asked one is taken, once. + let (s7, s8, s9) = (a.upload(&[7; 40]).await, a.upload(&[8; 40]).await, a.upload(&[9; 41]).await); + pc.handle(ClientFrame::Snapshot { doc, upto: 5, manifest: s7 }).await; + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().base, 0); + pc.handle(ClientFrame::Snapshot { doc, upto: 4, manifest: s8 }).await; + let compacted = a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap(); + assert_eq!((compacted.base, compacted.above_base(), compacted.snapshot_bytes), (4, 10, 40)); + pc.handle(ClientFrame::Snapshot { doc, upto: 4, manifest: s9 }).await; + assert_eq!(a.meta.live_snapshot(a.user.space, doc).await.unwrap().unwrap().1.manifest, s8); + // From the start: the snapshot, then the log from its base (the row at the base gives the next `at`). + let (mut laptop, _) = a.session(2).await; + laptop.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + let frames = drain(&mut laptop).await; + assert_eq!(snapshot_of(&frames), Some((4, s8))); + assert_eq!(updates(&frames), vec![4, 5]); + assert_eq!(frames.last(), Some(&ServerFrame::Subscribed { doc, head: 5 })); + // From below the base, the hash checked against the kept row. + let (mut tablet, _) = a.session(2).await; + let wrong = tablet.handle(ClientFrame::Subscribe { doc, after: 2, at: Some(update_hash(b"other")) }).await; + assert_eq!(code(&wrong), Some("resync_required")); + tablet.handle(ClientFrame::Subscribe { doc, after: 2, at: Some(update_hash(&[2; 10])) }).await; + let frames = drain(&mut tablet).await; + assert_eq!((snapshot_of(&frames).map(|s| s.0), updates(&frames)), (Some(4), vec![4, 5])); + // From the base or above: the log only. + tablet.handle(ClientFrame::Subscribe { doc, after: 4, at: Some(update_hash(&[4; 10])) }).await; + let frames = drain(&mut tablet).await; + assert_eq!((snapshot_of(&frames), updates(&frames)), (None, vec![5])); + // `subscribe_log` never gets the snapshot, but every row, after the import (which no row holds). + let (mut reader, _) = a.session(2).await; + reader.handle(ClientFrame::SubscribeLog { doc, after: 0, at: None }).await; + let frames = drain(&mut reader).await; + let import = a.meta.live_import_snapshot(a.user.space, doc).await.unwrap().unwrap().manifest; + assert_eq!((snapshot_of(&frames), updates(&frames)), (Some((0, import)), vec![1, 2, 3, 4, 5])); + reader.handle(ClientFrame::SubscribeLog { doc, after: 2, at: Some(update_hash(&[2; 10])) }).await; + let frames = drain(&mut reader).await; + assert_eq!((snapshot_of(&frames), updates(&frames)), (None, vec![3, 4, 5]), "from 2 it had the import"); + } + + /// A device that just caught up holds the whole log: if it is long and nobody is asked, it is. + #[tokio::test] + async fn a_device_that_caught_up_is_asked_when_the_log_is_long() { + let mut a = Account::new(crate::state::Limits { compact_updates: 3, compact_timeout: Duration::ZERO, ..Default::default() }).await; + let (mut pc, auth) = a.session(2).await; + let doc = imported(&a, &mut pc, &auth, &[1; 10]).await; + for client_seq in 1..=5u64 { + pc.handle(ClientFrame::Update { doc, client_seq, bytes: vec![client_seq as u8; 10] }).await; + } + // The pc was asked (at 4) and never answers: its request times out at once here. + let (mut laptop, _) = a.session(2).await; + laptop.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + let frames = drain(&mut laptop).await; + assert_eq!(compacts(&frames), vec![5], "asked after `subscribed`"); + assert!(matches!(frames.iter().rev().nth(1), Some(ServerFrame::Subscribed { head: 5, .. })), "{frames:?}"); + laptop.handle(ClientFrame::Snapshot { doc, upto: 5, manifest: a.upload(&[5; 10]).await }).await; + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().base, 5); + } + + /// A device that doesn't answer in time is skipped, and the next one to append is asked; a late answer is still + /// taken if it moves the base on, and an answer for an older `upto` than the base is ignored. + #[tokio::test] + async fn an_unanswered_compaction_goes_to_another_device() { + let mut a = Account::new(crate::state::Limits { compact_updates: 1, compact_timeout: Duration::ZERO, ..Default::default() }).await; + let (mut pc, auth) = a.session(2).await; + let (mut laptop, _) = a.session(2).await; + let doc = imported(&a, &mut pc, &auth, &[1; 10]).await; + laptop.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + drain(&mut laptop).await; + let send = async |s: &mut Session, client_seq: u64| { + compacts(&s.handle(ClientFrame::Update { doc, client_seq, bytes: vec![client_seq as u8] }).await.frames) + }; + assert!(send(&mut pc, 1).await.is_empty()); + assert_eq!(send(&mut pc, 2).await, vec![2], "the pc is asked"); + assert_eq!(send(&mut laptop, 1).await, vec![3], "it didn't answer in time: the laptop is asked"); + assert!(send(&mut pc, 3).await.is_empty(), "the pc is skipped for a while"); + // The pc answers late: its snapshot moves the base on, so it is taken. + pc.handle(ClientFrame::Snapshot { doc, upto: 2, manifest: a.upload(&[2; 10]).await }).await; + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().base, 2); + laptop.handle(ClientFrame::Snapshot { doc, upto: 3, manifest: a.upload(&[3; 10]).await }).await; + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().base, 3, "the laptop's, past it, too"); + // After a compaction anyone may be asked again, the skipped pc too (5 is two above the base). + assert_eq!(send(&mut pc, 4).await, vec![5]); + } + + /// Both devices answer at once, with snapshots for different `upto`s: the further one wins, whichever came first, + /// and a machine from the start still gets a snapshot and the rest of the log. + #[tokio::test] + async fn two_devices_answering_at_once_leave_the_furthest_snapshot() { + let mut a = Account::new(crate::state::Limits { compact_updates: 1, compact_timeout: Duration::ZERO, ..Default::default() }).await; + let (mut pc, auth) = a.session(2).await; + let (mut laptop, _) = a.session(2).await; + let doc = imported(&a, &mut pc, &auth, &[1; 10]).await; + laptop.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + drain(&mut laptop).await; + pc.handle(ClientFrame::Update { doc, client_seq: 1, bytes: vec![1] }).await; + assert_eq!(compacts(&pc.handle(ClientFrame::Update { doc, client_seq: 2, bytes: vec![2] }).await.frames), vec![2]); + assert_eq!(compacts(&laptop.handle(ClientFrame::Update { doc, client_seq: 1, bytes: vec![3] }).await.frames), vec![3]); + let (s3, s2) = (a.upload(&[3; 10]).await, a.upload(&[2; 10]).await); + tokio::join!( + laptop.handle(ClientFrame::Snapshot { doc, upto: 3, manifest: s3 }), + pc.handle(ClientFrame::Snapshot { doc, upto: 2, manifest: s2 }) + ); + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().base, 3); + let (mut fresh, _) = a.session(2).await; + fresh.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + let frames = drain(&mut fresh).await; + assert_eq!((snapshot_of(&frames), updates(&frames)), (Some((3, s3)), vec![3])); + } + + /// Past the account's cap on live bytes (every row of every live document, compacted ones too), an update is + /// refused with `quota_exceeded`, naming it, and nothing is stored. + #[tokio::test] + async fn an_update_past_the_accounts_live_bytes_is_refused() { + let meta = Meta::open("sqlite::memory:").await.unwrap().with_live_bytes_cap(25); + let mut a = Account::with_meta(small_logs(), meta, "ana").await; + let (mut pc, auth) = a.session(2).await; + let doc = imported(&a, &mut pc, &auth, &[1; 10]).await; + for client_seq in 1..=2u64 { + let reply = pc.handle(ClientFrame::Update { doc, client_seq, bytes: vec![client_seq as u8; 10] }).await; + assert!(matches!(reply.frames.first(), Some(ServerFrame::Ack { .. })), "{:?}", reply.frames); + } + let reply = pc.handle(ClientFrame::Update { doc, client_seq: 3, bytes: vec![3; 10] }).await; + assert_eq!(code(&reply), Some("quota_exceeded")); + assert!(matches!(reply.frames.as_slice(), [ServerFrame::Error { client_seq: Some(3), .. }])); + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().head, 2, "nothing stored"); + } + + /// A snapshot uploaded but never named (the agent stopped between its upload and its frame) changes nothing: the + /// document is as it was, and a later compaction works. + #[tokio::test] + async fn a_snapshot_uploaded_but_never_named_changes_nothing() { + let mut a = Account::new(small_logs()).await; + let (mut pc, auth) = a.session(2).await; + let doc = imported(&a, &mut pc, &auth, &[1; 10]).await; + let import = a.meta.live_import_snapshot(a.user.space, doc).await.unwrap().unwrap().manifest; + for client_seq in 1..=4u64 { + pc.handle(ClientFrame::Update { doc, client_seq, bytes: vec![client_seq as u8; 10] }).await; + } + a.upload(&[6; 50]).await; + let (mut laptop, _) = a.session(2).await; + laptop.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + let frames = drain(&mut laptop).await; + assert_eq!((snapshot_of(&frames), updates(&frames)), (Some((0, import)), vec![1, 2, 3, 4]), "the import and the whole log"); + pc.handle(ClientFrame::Snapshot { doc, upto: 4, manifest: a.upload(&[4; 50]).await }).await; + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().base, 4); + } + + /// Presence goes to the document's other subscribers, never back, never to one that doesn't follow it or speaks + /// protocol 1, at most so many a second per document; nothing is stored. + #[tokio::test] + async fn presence_is_relayed_within_its_limits() { + let mut a = Account::new(crate::state::Limits { presence_per_sec: 20, ..Default::default() }).await; + let (mut pc, auth) = a.session(2).await; + let (mut laptop, _) = a.session(2).await; + let (mut stranger, _) = a.session(2).await; + let doc = imported(&a, &mut pc, &auth, &[1; 10]).await; + laptop.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + drain(&mut laptop).await; + let mut heard = a.state.live.listen_presence(a.user.space); + for i in 0..25u8 { + assert!(pc.handle(ClientFrame::Presence { doc, bytes: vec![i; 100] }).await.frames.is_empty(), "never answered"); + } + let mut notes = Vec::new(); + while let Ok(note) = heard.try_recv() { + notes.push(note); + } + assert_eq!(notes.len(), 20, "20 a second, the rest dropped"); + let note = ¬es[0]; + assert_eq!(laptop.relay(note), Some(ServerFrame::Presence { doc, bytes: vec![0; 100] })); + assert_eq!(pc.relay(note), None, "not back to its sender"); + assert_eq!(stranger.relay(note), None, "not to one that doesn't follow it"); + // Presence for a document it doesn't follow is dropped. + stranger.handle(ClientFrame::Presence { doc, bytes: vec![1] }).await; + assert!(heard.try_recv().is_err()); + assert_eq!(a.meta.live_doc(a.user.space, doc).await.unwrap().unwrap().head, 0, "nothing stored"); + assert!(a.meta.live_updates(a.user.space, doc, 0, 10).await.unwrap().is_empty()); + } + + /// Presence is budgeted per device and document, across the device's connections: two connections of one device + /// share its 20 a second. + #[tokio::test] + async fn presence_is_budgeted_per_device_across_its_connections() { + let mut a = Account::new(crate::state::Limits { presence_per_sec: 20, ..Default::default() }).await; + let (mut pc, auth) = a.session(2).await; + let doc = imported(&a, &mut pc, &auth, &[1; 10]).await; + // A second connection of the same device. + let mut again = Session::new(a.state.clone(), &auth); + again.handle(ClientFrame::Hello { min: 1, max: 2 }).await; + again.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + drain(&mut again).await; + let mut heard = a.state.live.listen_presence(a.user.space); + for i in 0..15u8 { + pc.handle(ClientFrame::Presence { doc, bytes: vec![i] }).await; + again.handle(ClientFrame::Presence { doc, bytes: vec![i] }).await; + } + let mut relayed = 0; + while heard.try_recv().is_ok() { + relayed += 1; + } + assert_eq!(relayed, 20, "the device's 20, not 20 per connection"); + } + + /// The admin's reset drops the snapshot with the log; a machine that knew the document before (at 3, base 4 now) + /// is sent the snapshot, and one that knew more than the log holds is told the head, below what it has. + #[tokio::test] + async fn restores_and_resets_with_a_snapshot() { + let mut a = Account::new(small_logs()).await; + let (mut pc, auth) = a.session(2).await; + let doc = imported(&a, &mut pc, &auth, &[1; 10]).await; + for client_seq in 1..=4u64 { + pc.handle(ClientFrame::Update { doc, client_seq, bytes: vec![client_seq as u8; 10] }).await; + } + pc.handle(ClientFrame::Snapshot { doc, upto: 4, manifest: a.upload(&[4; 10]).await }).await; + let (mut behind, _) = a.session(2).await; + behind.handle(ClientFrame::Subscribe { doc, after: 3, at: Some(update_hash(&[3; 10])) }).await; + let frames = drain(&mut behind).await; + assert_eq!((snapshot_of(&frames).map(|s| s.0), updates(&frames)), (Some(4), vec![4])); + let (mut ahead, _) = a.session(2).await; + ahead.handle(ClientFrame::Subscribe { doc, after: 9, at: Some(update_hash(b"nine")) }).await; + assert_eq!(drain(&mut ahead).await, vec![ServerFrame::Subscribed { doc, head: 4 }], "the head, below what it has: it starts over"); + a.meta.reset_live_doc(a.user.space, doc).await.unwrap(); + let (mut after_reset, _) = a.session(2).await; + after_reset.handle(ClientFrame::Subscribe { doc, after: 0, at: None }).await; + assert_eq!( + drain(&mut after_reset).await, + vec![ServerFrame::Subscribed { doc, head: 0 }], + "no snapshot, no log: every machine pushes its own" + ); + } } diff --git a/apps/craft-cloud-server/src/state.rs b/apps/craft-cloud-server/src/state.rs index bc4a0ec..10d7edc 100644 --- a/apps/craft-cloud-server/src/state.rs +++ b/apps/craft-cloud-server/src/state.rs @@ -37,6 +37,14 @@ pub struct Limits { pub live_burst: u32, /// …refilled at this many per second; over it the answer is `rate_limited`. pub live_per_sec: u32, + /// A live document's log above its snapshot is compacted once it holds more than this many updates… + pub compact_updates: u64, + /// …or more than this many bytes. + pub compact_bytes: u64, + /// A device asked to compact that doesn't answer within this long is skipped, and another one asked later. + pub compact_timeout: Duration, + /// Presence messages one connection may send per document per second (more are dropped). + pub presence_per_sec: u32, } impl Default for Limits { @@ -49,6 +57,10 @@ impl Default for Limits { body_time_per_mib: Duration::from_secs(30), live_burst: crate::live::UPDATE_BURST, live_per_sec: crate::live::UPDATES_PER_SEC, + compact_updates: craft_cloud_proto::live::COMPACT_UPDATES, + compact_bytes: craft_cloud_proto::live::COMPACT_BYTES, + compact_timeout: crate::live::COMPACT_TIMEOUT, + presence_per_sec: craft_cloud_proto::live::PRESENCE_PER_SEC, } } } diff --git a/apps/craft-cloud-server/tests/api.rs b/apps/craft-cloud-server/tests/api.rs index 1053fbf..163ad16 100644 --- a/apps/craft-cloud-server/tests/api.rs +++ b/apps/craft-cloud-server/tests/api.rs @@ -173,6 +173,7 @@ on_every_backend!( live_updates_fan_out_to_the_accounts_other_devices, live_catch_up_after_a_gap, live_a_revoked_device_is_cut_off, + live_documents_carry_snapshots_and_presence, pairings_are_made_polled_and_cancelled, a_pairing_code_signs_a_new_computer_in, wrong_expired_and_used_codes_get_the_same_answer, @@ -239,6 +240,21 @@ async fn upload(s: &Server, token: &str, data: &[u8]) -> ManifestId { m.id() } +/// A file's content through the API: its manifest, then each chunk it names. +async fn download(s: &Server, token: &str, id: &ManifestId) -> Vec { + let get = |path: String| async move { + let response = s.client.get(format!("{}/{path}", s.base)).bearer_auth(token).send().await.unwrap(); + assert_eq!(response.status(), StatusCode::OK, "{path}"); + response.bytes().await.unwrap() + }; + let manifest = Manifest::decode(&get(format!("manifests/{id}")).await).unwrap(); + let mut out = Vec::new(); + for chunk in manifest.chunks() { + out.extend_from_slice(&get(format!("chunks/{}", chunk.id)).await); + } + out +} + async fn commit(s: &Server, token: &str, ops: &[Op]) -> reqwest::Response { s.client.post(format!("{}/commit", s.base)).bearer_auth(token).json(&json!({ "ops": ops })).send().await.unwrap() } @@ -1193,6 +1209,43 @@ async fn live_needs_a_device_token(s: &Server) { drop(held); } +/// Over a real WebSocket: a document of a file made by import with a snapshot bigger than an update may be (uploaded +/// through the chunk API, named by its manifest), kept once, named to another device catching up (which downloads it +/// the same way), and presence relayed between them. No message is over an update and its envelope. +async fn live_documents_carry_snapshots_and_presence(s: &Server) { + add_user(s, "ana").await; + let (pc_token, laptop_token) = (login(s, "ana").await, login(s, "ana").await); + let m = upload(s, &pc_token, b"the file's bytes").await; + let node = NodeId::generate(); + let r = root(s, &pc_token).await; + assert_eq!(commit(s, &pc_token, &[create(node, r, "doc1.txt", m)]).await.status(), StatusCode::OK); + let doc = craft_cloud_proto::live::node_doc(space_of(s, &pc_token).await.space, node); + let mut pc = live_ready(s, &pc_token).await; + let mut laptop = live_ready(s, &laptop_token).await; + let snapshot: Vec = (0..3 * 1024 * 1024u32).map(|i| (i % 251) as u8).collect(); + let manifest = upload(s, &pc_token, &snapshot).await; + let app = AppName::new("wordcraft").unwrap(); + live_send(&mut pc, ClientFrame::CreateDocument { doc, node, app: app.clone(), snapshot: manifest }).await; + assert_eq!(live_next(&mut pc).await, Some(ServerFrame::CreatedDocument { doc, kept: true })); + let other = upload(s, &laptop_token, &[1; 10]).await; + live_send(&mut laptop, ClientFrame::CreateDocument { doc, node, app, snapshot: other }).await; + assert_eq!(live_next(&mut laptop).await, Some(ServerFrame::CreatedDocument { doc, kept: false })); + let frames = live_subscribe(&mut laptop, doc, 0).await; + assert_eq!(frames, vec![ServerFrame::Snapshot { doc, base: 0, manifest }, ServerFrame::Subscribed { doc, head: 0 }]); + assert_eq!(download(s, &laptop_token, &manifest).await, snapshot, "the laptop gets the snapshot through the chunk API"); + live_subscribe(&mut pc, doc, 0).await; + live_send(&mut pc, ClientFrame::Presence { doc, bytes: b"cursor at 7".to_vec() }).await; + assert_eq!(live_next(&mut laptop).await, Some(ServerFrame::Presence { doc, bytes: b"cursor at 7".to_vec() })); + live_quiet(&mut pc).await; + // A message over an update and its envelope ends the connection, before `hello` or after. + for mut ws in [laptop, open_live(s, Some(&pc_token)).await.unwrap()] { + use futures_util::SinkExt; + let big = vec![0u8; craft_cloud_proto::live::MAX_FRAME_BYTES + 1]; + let _ = ws.send(tokio_tungstenite::tungstenite::Message::Binary(big.into())).await; + assert_eq!(live_next(&mut ws).await, None, "closed"); + } +} + async fn live_hello_checks_the_version(s: &Server) { add_user(s, "ana").await; let token = login(s, "ana").await; diff --git a/apps/craft-cloud-server/tests/cli.rs b/apps/craft-cloud-server/tests/cli.rs index 0c52cfc..4a5332b 100644 --- a/apps/craft-cloud-server/tests/cli.rs +++ b/apps/craft-cloud-server/tests/cli.rs @@ -93,6 +93,41 @@ async fn live_reset_doc_empties_a_full_document() { assert!(matches!(cli(&["live", "--config", c, "reset-doc", "nobody", "wordcraft"], "").await, Err(CliError::Usage(_)))); } +/// `live reset-doc` takes a document's ID too (a document of a file has no app name to find it by), and drops its +/// snapshot with its log. +#[tokio::test] +async fn live_reset_doc_takes_a_document_id_and_drops_its_snapshot() { + let parent = tempfile::tempdir().unwrap(); + let dir = parent.path().join("data"); + cli(&["init", "--dir", dir.to_str().unwrap(), "--admin", "ana"], "admin password 1\n").await.unwrap(); + let config_path = dir.join("config.toml"); + let c = config_path.to_str().unwrap(); + let (meta, _) = craft_cloud_server::cli::open(&Config::load(&config_path).unwrap()).await.unwrap(); + let ana = meta.user_by_name("ana").await.unwrap().unwrap(); + let device = meta.create_device(ana.id, "pc", &[3; 32], 0).await.unwrap(); + let doc = craft_cloud_proto::DocId::generate(); + meta.create_live_doc(ana.space, doc, "recents:wordcraft", None, 0).await.unwrap(); + meta.append_live_update(ana.space, doc, device, 1, &[1; 20], 0).await.unwrap(); + // A snapshot as an agent uploads one: a chunk, and the manifest naming it. + let chunk = craft_cloud_proto::ChunkId::from_bytes([7; 32]); + meta.add_chunk(ana.space, &chunk, 31, 0).await.unwrap(); + let manifest = craft_cloud_proto::Manifest::new( + craft_cloud_proto::ChunkerParamsId::new("fastcdc-v1").unwrap(), + 30, + craft_cloud_proto::FileHash::from_bytes([7; 32]), + vec![craft_cloud_proto::ChunkRef { id: chunk, len: 30 }], + ) + .unwrap(); + meta.add_manifest(ana.space, &manifest, 100, 0).await.unwrap(); + assert!(meta.compact_live_doc(ana.space, doc, 1, &manifest.id()).await.unwrap()); + let out = cli(&["live", "--config", c, "reset-doc", "ana", &doc.to_string()], "").await.unwrap(); + assert!(out.contains(&format!("emptied the live document {doc} of ana")), "{out}"); + let reset = meta.live_doc(ana.space, doc).await.unwrap().unwrap(); + assert_eq!((reset.head, reset.base, reset.has_snapshot), (0, 0, false)); + let missing = craft_cloud_proto::DocId::generate().to_string(); + assert!(cli(&["live", "--config", c, "reset-doc", "ana", &missing], "").await.is_err(), "no such document"); +} + /// `user invite` prints a join string and link to paste on the new computer (with `public_url` set), or the code /// and how to finish it (without); a new one replaces the old; `--revoke` ends it. #[tokio::test] diff --git a/crates/app/src/lib.rs b/crates/app/src/lib.rs index 3c18460..70c24e5 100644 --- a/crates/app/src/lib.rs +++ b/crates/app/src/lib.rs @@ -1,7 +1,8 @@ //! What Craft apps and the launcher link to reach the Craft Cloud agent on this machine (`craft-cloud-agent run`) //! through its local API: the account's status, signing in and out, joining with a code and adding computers (the -//! Craft Cloud app's), pausing, quitting and start-at-login, and each app's recent files, shared by every machine of -//! the account. +//! Craft Cloud app's), pausing, quitting and start-at-login, each app's recent files, shared by every machine of the +//! account, and live documents ([`live`]: files opened by path, their changes going to the other machines as they are +//! made, at the level of Loro's bytes). //! //! Blocking, with no async runtime: an egui app calls it and gets changes on a channel ([`RecentsWatch::events`]), //! which it polls each frame. @@ -12,7 +13,9 @@ //! [`Client::set_autostart`] (a few seconds at most). **Off the UI thread**, since they wait for the server (up to //! 90 s): [`Client::login`], [`Client::join`], [`Client::logout`], and the add-a-computer and computer-list calls //! ([`Client::pairing`], [`Client::pairing_state`], [`Client::cancel_pairing`], [`Client::devices`], -//! [`Client::remove_device`]). +//! [`Client::remove_device`]), and opening a live document ([`live::Documents::open`], [`live::Importer::import`]: up +//! to 30 s). A live document's own calls ([`live::Document::send`], [`live::Document::try_recv`]) don't wait for an +//! answer. //! //! ```no_run //! # fn main() -> Result<(), craft_cloud_app::Error> { @@ -42,6 +45,7 @@ #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] mod http; +pub mod live; mod recents; use std::net::{Ipv4Addr, SocketAddr}; @@ -245,6 +249,13 @@ impl Client { pub fn watch_recents(&self, app: &str) -> Result { RecentsWatch::open(self, app, true) } + + /// A connection for live documents ([`live::Documents::open`]): files of the synced folder opened by path, their + /// changes going to and coming from the account's other machines as they are made. `NotRunning` if no agent runs + /// right now. + pub fn documents(&self) -> Result { + live::Documents::connect(self) + } } #[cfg(test)] diff --git a/crates/app/src/live.rs b/crates/app/src/live.rs new file mode 100644 index 0000000..5745b97 --- /dev/null +++ b/crates/app/src/live.rs @@ -0,0 +1,513 @@ +//! Live documents through the agent's local API (`GET /v1/docs`), at the level of bytes: an app opens a file of the +//! synced folder by path and gets its state (a Loro snapshot), then sends its own changes (Loro updates) and gets the +//! other machines' and apps' as they come. What the bytes mean is the app's (its Loro document's); nothing here reads +//! them. +//! +//! One [`Documents`] is one connection, which may have many documents open. Blocking, with no async runtime: a thread +//! of its own reads the connection and hands each document's events to a channel the app polls +//! ([`Document::try_recv`]) or waits on ([`Document::recv_timeout`]); sending ([`Document::send`], +//! [`Document::presence`]) writes at once, from the caller's thread. +//! +//! ```no_run +//! # fn main() -> Result<(), craft_cloud_app::Error> { +//! use craft_cloud_app::live::{Event, Opened}; +//! let agent = craft_cloud_app::Client::connect()?; +//! let docs = agent.documents()?; +//! let document = match docs.open("Letters/doc1.txt", "wordcraft")? { +//! Opened::Live(document) => document, +//! // Never live yet: send the file's content as a Loro snapshot. +//! Opened::NeedsImport(importer) => importer.import(&[/* a Loro snapshot */])?, +//! // A plain file for now (offline, not uploaded yet, …): open and save it as always. +//! Opened::NotLive(_reason) => return Ok(()), +//! }; +//! let _state = document.state(); // Build the app's Loro document from this. +//! document.send(&[/* a Loro update */])?; +//! while let Some(event) = document.try_recv() { +//! match event { +//! Event::Update(_bytes) => { /* import into the app's document */ } +//! _ => {} +//! } +//! } +//! # Ok(()) } +//! ``` +//! +//! The connection isn't made again here when it goes ([`Event::Closed`]): an app opens its documents again on a new +//! [`Documents`]. + +use std::collections::HashMap; +use std::net::{Shutdown, TcpStream}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::mpsc::{self, Receiver, RecvTimeoutError, Sender}; +use std::sync::{Arc, Mutex, PoisonError}; +use std::thread::JoinHandle; +use std::time::Duration; + +use craft_cloud_proto::DocId; +use craft_cloud_proto::api::ErrorBody; +use craft_cloud_proto::local::docs::{IMPORT, MAX_BINARY_BYTES, PRESENCE, STATE, UPDATE, decode_binary, encode_binary}; +use craft_cloud_proto::local::{DocState, DocsEvent, DocsRequest, MAX_LOCAL_JSON_BYTES, paths}; +use tungstenite::client::IntoClientRequest; +use tungstenite::handshake::HandshakeError; +use tungstenite::protocol::WebSocketConfig; +use tungstenite::{Message, WebSocket}; + +pub use craft_cloud_proto::local::NotLiveReason; + +use crate::http::CONNECT_TIMEOUT; +use crate::{Client, Error}; + +/// How long an open may take: finding the file's node (the agent may sync it first, up to 5 s), joining the document on +/// the server, and its state. +pub const OPEN_TIMEOUT: Duration = Duration::from_secs(30); +/// How long writing one message to the agent may take (a big import too, the agent being on this machine). +const WRITE_TIMEOUT: Duration = Duration::from_secs(10); +/// How long the reader waits for data before it looks whether it is to stop. +const IDLE_WAKE: Duration = Duration::from_millis(250); + +/// What happens to an open document. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Event { + /// A change from another machine or another app on this one: import it as it is. + Update(Vec), + /// Another machine's or app's presence (Loro's ephemeral state), as it was sent. + Presence(Vec), + /// This app's change number `seq` ([`Document::send`]'s answer) is on the agent's disk. + Acked(u64), + /// This app's change number `seq` was refused (it doesn't decode, or is over 1 MiB): the agent's copy doesn't have + /// it. + Refused { seq: u64, message: String }, + /// How the document stands: `online` (edits go out as they are made), changes waiting for the server, and whether + /// its file was deleted (the document still works; a later open of the path is a new file), and whether the agent + /// stopped following it until it restarts (Loro failed on it twice; changes made meanwhile are kept). + Status { online: bool, waiting: u64, deleted: bool, stopped: bool }, + /// The connection to the agent ended. + Closed(String), +} + +/// What opening a file gives. +#[derive(Debug)] +pub enum Opened { + /// The file is live: its state, and its events from now on. + Live(Document), + /// The file has never been live: send its content to make it a live document ([`Importer::import`]). + NeedsImport(Importer), + /// The file opens as a plain file (save it as always). + NotLive(NotLiveReason), +} + +/// What the reader hands an open in progress. +enum Answer { + Live { doc: DocId, state: DocState, snapshot: Vec }, + NeedsImport, + NotLive(NotLiveReason), + Refused(Error), +} + +/// The connection's shared state: the socket (written from the caller's thread, read by the reader), and where each +/// document's events go. +struct Shared { + socket: Mutex>, + /// Each document number's events, and opens waiting for their answer. + routes: Mutex>>, + answers: Mutex>>, + /// An `opened` whose state hasn't come yet, by number. + opening: Mutex>, + /// Which numbers a document is open as (its status names the document). + numbers: Mutex>>, + closed: AtomicBool, + next_id: AtomicU64, + /// A clone of the stream: dropping the connection shuts it down (which wakes the reader). + stream: TcpStream, +} + +fn lock(m: &Mutex) -> std::sync::MutexGuard<'_, T> { + m.lock().unwrap_or_else(PoisonError::into_inner) +} + +impl Shared { + fn send(&self, message: Message) -> Result<(), Error> { + if self.closed.load(Ordering::Relaxed) { + return Err(Error::Io("the connection to the agent is closed".into())); + } + lock(&self.socket).send(message).map_err(|e| Error::Io(e.to_string())) + } + + fn send_json(&self, request: &DocsRequest) -> Result<(), Error> { + let text = serde_json::to_string(request).map_err(|e| Error::Protocol(e.to_string()))?; + self.send(Message::Text(text.into())) + } + + fn send_binary(&self, kind: u8, id: u64, bytes: &[u8]) -> Result<(), Error> { + self.send(Message::Binary(encode_binary(kind, id, bytes).into())) + } + + fn route(&self, id: u64, event: Event) { + if let Some(events) = lock(&self.routes).get(&id) { + let _ = events.send(event); + } + } + + fn answer(&self, id: u64, answer: Answer) { + if let Some(waiting) = lock(&self.answers).remove(&id) { + let _ = waiting.send(answer); + } + } + + /// One message from the agent. + fn on_message(&self, message: Message) { + match message { + Message::Text(text) => match serde_json::from_str::(text.as_str()) { + Ok(DocsEvent::Opened { id, doc, state }) => { + lock(&self.opening).insert(id, (doc, state)); + } + Ok(DocsEvent::NeedsImport { id }) => self.answer(id, Answer::NeedsImport), + Ok(DocsEvent::NotLive { id, reason, .. }) => self.answer(id, Answer::NotLive(reason)), + Ok(DocsEvent::Ack { id, seq }) => self.route(id, Event::Acked(seq)), + Ok(DocsEvent::Status { doc, online, waiting, deleted, stopped }) => { + let ids = lock(&self.numbers).get(&doc).cloned().unwrap_or_default(); + for id in ids { + self.route(id, Event::Status { online, waiting, deleted, stopped }); + } + } + Ok(DocsEvent::Error { id: Some(id), seq: Some(seq), message, .. }) => self.route(id, Event::Refused { seq, message }), + Ok(DocsEvent::Error { id: Some(id), seq: None, code, message }) => self.answer(id, Answer::Refused(Error::Refused { code, message })), + Ok(DocsEvent::Error { id: None, .. }) | Err(_) => {} + }, + Message::Binary(bytes) => { + let Some((kind, id, body)) = decode_binary(&bytes) else { return }; + match kind { + STATE => { + if let Some((doc, state)) = lock(&self.opening).remove(&id) { + lock(&self.numbers).entry(doc).or_default().push(id); + self.answer(id, Answer::Live { doc, state, snapshot: body.to_vec() }); + } + } + UPDATE => self.route(id, Event::Update(body.to_vec())), + PRESENCE => self.route(id, Event::Presence(body.to_vec())), + _ => {} + } + } + _ => {} + } + } + + /// The connection ended: every document hears it, and every open waiting gets the error. + fn end(&self, why: &str) { + self.closed.store(true, Ordering::Relaxed); + for (_, events) in lock(&self.routes).drain() { + let _ = events.send(Event::Closed(why.to_owned())); + } + for (_, waiting) in lock(&self.answers).drain() { + let _ = waiting.send(Answer::Refused(Error::Io(why.to_owned()))); + } + } +} + +/// The reader's thread, on a WebSocket of its own over the connection: every message as it comes (waking now and then +/// to look whether to stop). +fn read_loop(shared: Arc, mut socket: WebSocket) { + loop { + if shared.closed.load(Ordering::Relaxed) { + return; + } + match socket.read() { + Ok(Message::Close(_)) => return shared.end("the agent closed the connection"), + Ok(message) => shared.on_message(message), + Err(tungstenite::Error::Io(e)) if matches!(e.kind(), std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut) => {} + Err(e) => return shared.end(&e.to_string()), + } + } +} + +/// One connection to the agent's live documents. Dropping it ends the connection (its documents hear +/// [`Event::Closed`]); the agent keeps them. +pub struct Documents { + shared: Arc, + reader: Option>, +} + +impl std::fmt::Debug for Documents { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Documents").field("open", &lock(&self.shared.routes).len()).finish() + } +} + +impl Documents { + pub(crate) fn connect(client: &Client) -> Result { + let stream = TcpStream::connect_timeout(&client.addr(), CONNECT_TIMEOUT).map_err(|e| Error::NotRunning(format!("can't reach it: {e}")))?; + stream.set_nodelay(true).map_err(|e| Error::Io(e.to_string()))?; + stream.set_read_timeout(Some(OPEN_TIMEOUT)).map_err(|e| Error::Io(e.to_string()))?; + stream.set_write_timeout(Some(WRITE_TIMEOUT)).map_err(|e| Error::Io(e.to_string()))?; + let mut request = format!("ws://{}{}", client.addr(), paths::DOCS).into_client_request().map_err(|e| Error::Protocol(e.to_string()))?; + let auth = format!("Bearer {}", client.token()).parse().map_err(|_| Error::Protocol("the token can't be sent".into()))?; + request.headers_mut().insert("authorization", auth); + let limit = MAX_BINARY_BYTES.max(MAX_LOCAL_JSON_BYTES); + let config = WebSocketConfig::default().max_message_size(Some(limit)).max_frame_size(Some(limit)); + let socket = match tungstenite::client::client_with_config(request, stream, Some(config)) { + Ok((socket, _)) => socket, + Err(HandshakeError::Failure(tungstenite::Error::Http(response))) => { + let status = response.status().as_u16(); + let body = response.into_body().unwrap_or_default(); + return Err(match serde_json::from_slice::(&body) { + Ok(body) => Error::Refused { code: body.error, message: body.message }, + Err(_) => Error::Protocol(format!("HTTP {status}")), + }); + } + Err(e) => return Err(Error::Io(e.to_string())), + }; + // Two WebSockets over the one connection, a reader's and a writer's, so a write never waits for a read. (A + // read timeout is the socket's, not one handle's: the reader's wakes it now and then to look whether to + // stop.) The agent sends nothing before the app's first request, so the handshake left nothing unread; the + // reader never writes (the agent sends no pings), so writes never interleave. + let clone = |s: &TcpStream| s.try_clone().map_err(|e| Error::Io(e.to_string())); + let (read_side, write_side, stream) = (clone(socket.get_ref())?, clone(socket.get_ref())?, clone(socket.get_ref())?); + drop(socket); + read_side.set_read_timeout(Some(IDLE_WAKE)).map_err(|e| Error::Io(e.to_string()))?; + let reader_socket = WebSocket::from_raw_socket(read_side, tungstenite::protocol::Role::Client, Some(config)); + let socket = WebSocket::from_raw_socket(write_side, tungstenite::protocol::Role::Client, Some(config)); + let shared = Arc::new(Shared { + socket: Mutex::new(socket), + routes: Mutex::new(HashMap::new()), + answers: Mutex::new(HashMap::new()), + opening: Mutex::new(HashMap::new()), + numbers: Mutex::new(HashMap::new()), + closed: AtomicBool::new(false), + next_id: AtomicU64::new(1), + stream, + }); + let reader_shared = shared.clone(); + let reader = std::thread::Builder::new() + .name("craft-cloud-docs".into()) + .spawn(move || read_loop(reader_shared, reader_socket)) + .map_err(|e| Error::Io(e.to_string()))?; + Ok(Self { shared, reader: Some(reader) }) + } + + /// Opens the file at `path` (relative to the synced folder, `/`-separated, or absolute inside it) for `app` (its + /// lowercase name, as in recents). Waits for the answer, [`OPEN_TIMEOUT`] at most. + pub fn open(&self, path: &str, app: &str) -> Result { + let id = self.shared.next_id.fetch_add(1, Ordering::Relaxed); + let (events_tx, events) = mpsc::channel(); + lock(&self.shared.routes).insert(id, events_tx); + let answer = self.ask(id, || self.shared.send_json(&DocsRequest::Open { id, path: path.to_owned(), app: app.to_owned() })); + self.opened(id, answer, events) + } + + /// Sends what `send` sends and waits for the answer to `id`. + fn ask(&self, id: u64, send: impl FnOnce() -> Result<(), Error>) -> Result { + let (answer_tx, answer) = mpsc::channel(); + lock(&self.shared.answers).insert(id, answer_tx); + if let Err(e) = send() { + lock(&self.shared.answers).remove(&id); + return Err(e); + } + match answer.recv_timeout(OPEN_TIMEOUT) { + Ok(answer) => Ok(answer), + Err(RecvTimeoutError::Timeout) => { + lock(&self.shared.answers).remove(&id); + Err(Error::Io("the agent didn't answer the open in time".into())) + } + Err(RecvTimeoutError::Disconnected) => Err(Error::Io("the connection to the agent is closed".into())), + } + } + + fn opened(&self, id: u64, answer: Result, events: Receiver) -> Result { + let forget = || { + lock(&self.shared.routes).remove(&id); + }; + match answer { + Ok(Answer::Live { doc, state, snapshot }) => { + Ok(Opened::Live(Document { id, doc, state, snapshot, events, shared: self.shared.clone(), seq: AtomicU64::new(0) })) + } + Ok(Answer::NeedsImport) => Ok(Opened::NeedsImport(Importer { id, events, shared: self.shared.clone() })), + Ok(Answer::NotLive(reason)) => { + forget(); + Ok(Opened::NotLive(reason)) + } + Ok(Answer::Refused(e)) | Err(e) => { + forget(); + Err(e) + } + } + } +} + +impl Drop for Documents { + fn drop(&mut self) { + self.shared.closed.store(true, Ordering::Relaxed); + let _ = lock(&self.shared.socket).close(None); + let _ = self.shared.stream.shutdown(Shutdown::Both); + if let Some(reader) = self.reader.take() { + let _ = reader.join(); + } + self.shared.end("the connection was closed here"); + } +} + +/// A file that has never been live, waiting for its content. +pub struct Importer { + id: u64, + events: Receiver, + shared: Arc, +} + +impl std::fmt::Debug for Importer { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Importer").field("id", &self.id).finish() + } +} + +impl Importer { + /// Sends the file's content (a Loro snapshot, at most the snapshot limit) and waits until it is live: this app's + /// import, or another machine's that was there first (the [`Document::state`] is then that one). A file that can't + /// be made live now opens as a plain file ([`Error::Refused`] with `not_live`). + pub fn import(self, snapshot: &[u8]) -> Result { + let (answer_tx, answer) = mpsc::channel(); + lock(&self.shared.answers).insert(self.id, answer_tx); + if let Err(e) = self.shared.send_binary(IMPORT, self.id, snapshot) { + lock(&self.shared.answers).remove(&self.id); + return Err(e); + } + let answer = answer.recv_timeout(OPEN_TIMEOUT).map_err(|_| Error::Io("the agent didn't answer the import in time".into()))?; + match answer { + Answer::Live { doc, state, snapshot } => { + Ok(Document { id: self.id, doc, state, snapshot, events: self.events, shared: self.shared, seq: AtomicU64::new(0) }) + } + Answer::NotLive(reason) => { + Err(Error::Refused { code: "not_live".into(), message: format!("the file opens as a plain file ({reason:?})") }) + } + Answer::NeedsImport => Err(Error::Protocol("asked for the import again".into())), + Answer::Refused(e) => Err(e), + } + } +} + +/// An open live document. +pub struct Document { + id: u64, + doc: DocId, + state: DocState, + snapshot: Vec, + events: Receiver, + shared: Arc, + /// This app's changes sent so far (the agent numbers them the same way). + seq: AtomicU64, +} + +impl std::fmt::Debug for Document { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Document").field("doc", &self.doc).field("state", &self.state).finish() + } +} + +impl Document { + /// The document as it was when opened: a Loro snapshot (updates still waiting for their history come as the first + /// [`Event::Update`]s). + pub fn state(&self) -> &[u8] { + &self.snapshot + } + + /// Whether it was online, and how many changes waited, when opened. + pub fn opened_state(&self) -> DocState { + self.state + } + + /// The live document's ID. + pub fn id(&self) -> DocId { + self.doc + } + + /// Sends this app's change (a Loro update, at most 1 MiB); returns its number, which [`Event::Acked`] names once + /// it is on the agent's disk. + pub fn send(&self, update: &[u8]) -> Result { + let seq = self.seq.fetch_add(1, Ordering::Relaxed) + 1; + self.shared.send_binary(UPDATE, self.id, update)?; + Ok(seq) + } + + /// Sends this app's presence (at most 4 KiB, at most 20 a second; more are dropped). + pub fn presence(&self, bytes: &[u8]) -> Result<(), Error> { + self.shared.send_binary(PRESENCE, self.id, bytes) + } + + /// The next event, if one came. + pub fn try_recv(&self) -> Option { + self.events.try_recv().ok() + } + + /// The next event, waiting at most `timeout`. + pub fn recv_timeout(&self, timeout: Duration) -> Option { + self.events.recv_timeout(timeout).ok() + } + + /// Done with it (the agent keeps it). + pub fn close(self) { + drop(self); + } +} + +impl Drop for Document { + fn drop(&mut self) { + lock(&self.shared.routes).remove(&self.id); + let mut numbers = lock(&self.shared.numbers); + let remaining = numbers.get_mut(&self.doc).map(|ids| { + ids.retain(|id| *id != self.id); + ids.len() + }); + if remaining == Some(0) { + numbers.remove(&self.doc); + drop(numbers); + let _ = self.shared.send_json(&DocsRequest::Close { doc: self.doc }); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A shared state on a connection to nowhere (a socket pair stands in), to feed the reader's messages to. + fn shared() -> (Arc, std::net::TcpListener) { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let stream = TcpStream::connect(listener.local_addr().unwrap()).unwrap(); + let socket = WebSocket::from_raw_socket(stream.try_clone().unwrap(), tungstenite::protocol::Role::Client, None); + let shared = Arc::new(Shared { + socket: Mutex::new(socket), + routes: Mutex::new(HashMap::new()), + answers: Mutex::new(HashMap::new()), + opening: Mutex::new(HashMap::new()), + numbers: Mutex::new(HashMap::new()), + closed: AtomicBool::new(false), + next_id: AtomicU64::new(1), + stream, + }); + (shared, listener) + } + + /// `opened` then the state: one answer; the events after it go to the document, its status by document ID. + #[test] + fn an_open_is_answered_once_its_state_came() { + let (shared, _listener) = shared(); + let (answer_tx, answer) = mpsc::channel(); + lock(&shared.answers).insert(4, answer_tx); + let (events_tx, events) = mpsc::channel(); + lock(&shared.routes).insert(4, events_tx); + let doc = DocId::generate(); + let opened = DocsEvent::Opened { id: 4, doc, state: DocState { online: true, waiting: 0, bytes: 3 } }; + shared.on_message(Message::Text(serde_json::to_string(&opened).unwrap().into())); + assert!(answer.try_recv().is_err(), "not before the state"); + shared.on_message(Message::Binary(encode_binary(STATE, 4, b"abc").into())); + let Ok(Answer::Live { snapshot, .. }) = answer.try_recv() else { panic!("no answer") }; + assert_eq!(snapshot, b"abc"); + shared.on_message(Message::Binary(encode_binary(UPDATE, 4, b"u").into())); + let status = DocsEvent::Status { doc, online: false, waiting: 2, deleted: true, stopped: true }; + shared.on_message(Message::Text(serde_json::to_string(&status).unwrap().into())); + shared.on_message(Message::Binary(vec![9, 1].into())); + shared.on_message(Message::Text("not json".into())); + assert_eq!(events.try_recv().unwrap(), Event::Update(b"u".to_vec())); + assert_eq!(events.try_recv().unwrap(), Event::Status { online: false, waiting: 2, deleted: true, stopped: true }); + assert!(events.try_recv().is_err(), "junk goes nowhere"); + shared.end("gone"); + assert_eq!(events.try_recv().unwrap(), Event::Closed("gone".into())); + } +} diff --git a/crates/meta/migrations/postgres/0005_live_documents.sql b/crates/meta/migrations/postgres/0005_live_documents.sql new file mode 100644 index 0000000..55a7221 --- /dev/null +++ b/crates/meta/migrations/postgres/0005_live_documents.sql @@ -0,0 +1,24 @@ +-- Live documents linked to files, and snapshots. A document of a file has `node` set (its kind is `document:`). +-- `base_seq` is the last update its snapshot holds: the rows at or below it are compacted (kept, but no longer counted +-- against the document's cap nor sent to an agent that can read the snapshot), and `compacted_bytes` is their size +-- (`bytes` still counts every row). `snapshot` is the 32-byte ID of the manifest that names the snapshot (uploaded +-- through the chunk API as the space's) and `snapshot_bytes` is its size. A document made by import has a snapshot +-- from the start, with `base_seq` 0, and keeps that first one as `import_snapshot` (with `import_bytes`): the log +-- doesn't hold the import, so an agent that can't read a later snapshot reads the import and then the whole log. +-- `spaces.live_bytes` is the bytes every live document of the space holds (every row, compacted ones too), which its +-- cap on live bytes is checked against. +ALTER TABLE live_docs ADD COLUMN base_seq BIGINT NOT NULL DEFAULT 0; + +ALTER TABLE live_docs ADD COLUMN import_snapshot BYTEA; + +ALTER TABLE live_docs ADD COLUMN import_bytes BIGINT NOT NULL DEFAULT 0; + +ALTER TABLE live_docs ADD COLUMN compacted_bytes BIGINT NOT NULL DEFAULT 0; + +ALTER TABLE live_docs ADD COLUMN snapshot BYTEA; + +ALTER TABLE live_docs ADD COLUMN snapshot_bytes BIGINT NOT NULL DEFAULT 0; + +ALTER TABLE spaces ADD COLUMN live_bytes BIGINT NOT NULL DEFAULT 0; + +UPDATE spaces SET live_bytes = (SELECT CAST(COALESCE(SUM(d.bytes), 0) AS BIGINT) FROM live_docs d WHERE d.space_id = spaces.id) diff --git a/crates/meta/migrations/sqlite/0005_live_documents.sql b/crates/meta/migrations/sqlite/0005_live_documents.sql new file mode 100644 index 0000000..1c0e018 --- /dev/null +++ b/crates/meta/migrations/sqlite/0005_live_documents.sql @@ -0,0 +1,24 @@ +-- Live documents linked to files, and snapshots. A document of a file has `node` set (its kind is `document:`). +-- `base_seq` is the last update its snapshot holds: the rows at or below it are compacted (kept, but no longer counted +-- against the document's cap nor sent to an agent that can read the snapshot), and `compacted_bytes` is their size +-- (`bytes` still counts every row). `snapshot` is the 32-byte ID of the manifest that names the snapshot (uploaded +-- through the chunk API as the space's) and `snapshot_bytes` is its size. A document made by import has a snapshot +-- from the start, with `base_seq` 0, and keeps that first one as `import_snapshot` (with `import_bytes`): the log +-- doesn't hold the import, so an agent that can't read a later snapshot reads the import and then the whole log. +-- `spaces.live_bytes` is the bytes every live document of the space holds (every row, compacted ones too), which its +-- cap on live bytes is checked against. +ALTER TABLE live_docs ADD COLUMN base_seq BIGINT NOT NULL DEFAULT 0; + +ALTER TABLE live_docs ADD COLUMN import_snapshot BLOB; + +ALTER TABLE live_docs ADD COLUMN import_bytes BIGINT NOT NULL DEFAULT 0; + +ALTER TABLE live_docs ADD COLUMN compacted_bytes BIGINT NOT NULL DEFAULT 0; + +ALTER TABLE live_docs ADD COLUMN snapshot BLOB; + +ALTER TABLE live_docs ADD COLUMN snapshot_bytes BIGINT NOT NULL DEFAULT 0; + +ALTER TABLE spaces ADD COLUMN live_bytes BIGINT NOT NULL DEFAULT 0; + +UPDATE spaces SET live_bytes = (SELECT COALESCE(SUM(d.bytes), 0) FROM live_docs d WHERE d.space_id = spaces.id) diff --git a/crates/meta/src/blobs.rs b/crates/meta/src/blobs.rs index 94db790..45fbd78 100644 --- a/crates/meta/src/blobs.rs +++ b/crates/meta/src/blobs.rs @@ -65,16 +65,16 @@ async fn missing_in_space(conn: &mut sqlx::AnyConnection, space: SpaceId, kind: } /// One blob to record: what it is, how many bytes it is stored as, and (for a manifest) the size of its file. -struct Blob<'a> { - kind: BlobKind, - id: &'a [u8; 32], - stored_size: i64, - file_size: Option, +pub(crate) struct Blob<'a> { + pub(crate) kind: BlobKind, + pub(crate) id: &'a [u8; 32], + pub(crate) stored_size: i64, + pub(crate) file_size: Option, } /// Records that `space` has the blob. The first time, its stored size is added to the space's stored bytes, and if /// that takes them over `cap`, nothing is recorded (the caller's transaction is dropped, which rolls it back). -async fn record(tx: &mut sqlx::AnyConnection, space: SpaceId, blob: Blob<'_>, now_ms: i64, cap: Option) -> Result<(), MetaError> { +pub(crate) async fn record(tx: &mut sqlx::AnyConnection, space: SpaceId, blob: Blob<'_>, now_ms: i64, cap: Option) -> Result<(), MetaError> { let Blob { kind, id, stored_size, file_size } = blob; sqlx::query("INSERT INTO blobs (kind, id, stored_size, file_size, created_ms) VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING") .bind(kind.code()) diff --git a/crates/meta/src/lib.rs b/crates/meta/src/lib.rs index 1ec918c..ee4199c 100644 --- a/crates/meta/src/lib.rs +++ b/crates/meta/src/lib.rs @@ -27,7 +27,10 @@ use tokio::sync::{Mutex, OwnedMutexGuard}; pub use commit::CommitOutcome; pub use craft_cloud_proto::BlobKind; pub use join::{InviteFor, Redeemed}; -pub use live::{LiveDoc, LiveUpdate, MAX_LIVE_DOC_BYTES, MAX_LIVE_DOC_ROWS_PER_SPACE, MAX_LIVE_DOCS_PER_SPACE, MAX_LIVE_PAGE}; +pub use live::{ + Appended, DEFAULT_LIVE_BYTES_PER_SPACE, LiveDoc, LiveUpdate, MAX_LIVE_DOC_BYTES, MAX_LIVE_DOC_ROWS_PER_SPACE, MAX_LIVE_DOCS_PER_SPACE, + MAX_LIVE_DOCUMENTS_HOLDING_UPDATES, MAX_LIVE_DOCUMENTS_PER_SPACE, MAX_LIVE_PAGE, SnapshotRef, +}; pub use tree::SpaceInfo; pub use users::{DeviceAuth, DeviceRecord, Role, UserId, UserRecord, valid_username}; @@ -55,8 +58,13 @@ pub enum MetaError { /// `used` is what the log holds. #[error("this live document is full on the server: its log holds {used} of {limit} bytes")] DocFull { used: u64, limit: u64 }, + /// An update would take the bytes all of a space's live documents hold (compacted rows too) over its cap + /// ([`Meta::with_live_bytes_cap`]); nothing was stored. `used` is what they hold. + #[error("the account's live documents hold {used} of {limit} bytes on the server, the most they may")] + LiveFull { used: u64, limit: u64 }, /// The first update to a live document would take its space over the most documents holding updates it may have - /// ([`MAX_LIVE_DOCS_PER_SPACE`]); nothing was stored. + /// ([`MAX_LIVE_DOCS_PER_SPACE`] recents, [`MAX_LIVE_DOCUMENTS_HOLDING_UPDATES`] documents of files); nothing was + /// stored. #[error("the account has {limit} live documents holding updates on the server, the most it may have")] TooManyDocs { limit: u64 }, /// A new live document would take its space over the most rows it may have ([`MAX_LIVE_DOC_ROWS_PER_SPACE`]); @@ -99,6 +107,12 @@ pub struct Meta { live_docs_cap: u64, /// The most live document rows one space may have ([`Meta::with_live_doc_rows_cap`]). live_doc_rows_cap: u64, + /// The most documents of files one space may have ([`Meta::with_live_documents_cap`]). + live_documents_cap: u64, + /// The most documents of files holding updates one space may have ([`Meta::with_live_held_documents_cap`]). + live_held_documents_cap: u64, + /// The most bytes all of one space's live documents may hold ([`Meta::with_live_bytes_cap`]). + live_bytes_cap: u64, } impl Meta { @@ -152,6 +166,9 @@ impl Meta { live_doc_cap: MAX_LIVE_DOC_BYTES, live_docs_cap: MAX_LIVE_DOCS_PER_SPACE, live_doc_rows_cap: MAX_LIVE_DOC_ROWS_PER_SPACE, + live_documents_cap: MAX_LIVE_DOCUMENTS_PER_SPACE, + live_held_documents_cap: MAX_LIVE_DOCUMENTS_HOLDING_UPDATES, + live_bytes_cap: DEFAULT_LIVE_BYTES_PER_SPACE, }) } @@ -178,6 +195,29 @@ impl Meta { self } + /// The store with another cap on the number of documents of files one space may have (the default is + /// [`MAX_LIVE_DOCUMENTS_PER_SPACE`]); a create over it is refused with [`MetaError::TooManyDocRows`]. For tests. + pub fn with_live_documents_cap(mut self, cap: u64) -> Self { + self.live_documents_cap = cap; + self + } + + /// The store with another cap on the number of documents of files holding updates one space may have (the default + /// is [`MAX_LIVE_DOCUMENTS_HOLDING_UPDATES`]); the first update to one over it is refused with + /// [`MetaError::TooManyDocs`]. For tests. + pub fn with_live_held_documents_cap(mut self, cap: u64) -> Self { + self.live_held_documents_cap = cap; + self + } + + /// The store with another cap on the bytes all of one space's live documents may hold, compacted rows too (the + /// default is [`DEFAULT_LIVE_BYTES_PER_SPACE`]; the server's `live_bytes_per_user` sets it); an append over it is + /// refused with [`MetaError::LiveFull`]. The cap belongs to this handle and its clones. + pub fn with_live_bytes_cap(mut self, cap: u64) -> Self { + self.live_bytes_cap = cap; + self + } + /// The store with a cap on the bytes one space (one user's) stores: the stored size of every chunk and manifest /// it has, each counted once however many files use it. Recording a blob that would go over it is refused with /// [`MetaError::QuotaExceeded`]. `None` (the default): no cap. The cap belongs to this handle and its clones. diff --git a/crates/meta/src/live.rs b/crates/meta/src/live.rs index 247bb58..9d8efbe 100644 --- a/crates/meta/src/live.rs +++ b/crates/meta/src/live.rs @@ -4,28 +4,107 @@ //! write transaction that stores it, so a document's updates are numbered 1, 2, 3… without gaps. An append is //! durable when it returns (the transaction committed), which is what lets the server ack it. Everything is keyed by //! `(space, doc)`, so a document of another space is simply not found. +//! +//! **Snapshots.** A document of a file is made by import, with a snapshot ([`Meta::create_live_document`]); any +//! document's log is compacted into a new snapshot ([`Meta::compact_live_doc`]). A snapshot is stored as blobs in the +//! chunk store (pieces of at most [`SNAPSHOT_PIECE_BYTES`], each a chunk named by its BLAKE3, charged to the space +//! like any chunk); the database holds which, and the last update it holds (`base`). The rows at or below `base` are +//! kept (compacted: no longer counted against the document's cap, and not sent to an agent that reads the snapshot) +//! until the retention rules come. -use craft_cloud_proto::{DeviceId, DocId, NodeId, SpaceId}; +use craft_cloud_proto::{DeviceId, DocId, ManifestId, NodeId, SpaceId}; use sqlx::Row; -use crate::codec::{to_i64, to_u64, uuid_bytes, uuid_from}; +use crate::blobs::manifest_file_size; +use crate::codec::{flag, to_i64, to_u64, uuid_bytes, uuid_from}; use crate::{Meta, MetaError}; /// A live document. #[derive(Debug, Clone, PartialEq, Eq)] pub struct LiveDoc { pub id: DocId, - /// Its kind as text (`recents:`). + /// Its kind as text (`recents:`, `document:`). pub kind: String, /// The file node it belongs to, if any (recents have none). pub node: Option, pub created_ms: i64, /// Its last sequence number (0: no update yet). pub head: u64, - /// The bytes of updates its log holds (at or over [`Meta::live_doc_cap`]: full). + /// The bytes of updates its log holds, every row (compacted ones too). + pub bytes: u64, + /// The bytes of the rows at or below `base`: the rest ([`LiveDoc::above_base`]) counts against the cap (at or over + /// [`Meta::live_doc_cap`]: full). + pub compacted_bytes: u64, + /// The last update its snapshot holds (0: none, or a snapshot of an import only). + pub base: u64, + /// Whether it has a snapshot ([`Meta::live_snapshot`]), and its size. + pub has_snapshot: bool, + pub snapshot_bytes: u64, +} + +impl LiveDoc { + /// The bytes of the log above its snapshot: what counts against its cap. + pub fn above_base(&self) -> u64 { + self.bytes.saturating_sub(self.compacted_bytes) + } +} + +/// A snapshot: the manifest that names it (a file's content, uploaded through the chunk API and charged to the space +/// like any), and its size. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct SnapshotRef { + pub manifest: ManifestId, pub bytes: u64, } +impl SnapshotRef { + fn decode(id: &[u8], bytes: u64) -> Result { + let id = <[u8; 32]>::try_from(id).map_err(|_| MetaError::Corrupt("a snapshot's manifest ID"))?; + Ok(Self { manifest: ManifestId::from_bytes(id), bytes }) + } +} + +/// The snapshot `manifest` names, if `space` has that manifest and it is a snapshot's size; read in the caller's +/// transaction, so it is the space's when recorded. Nothing a client sends is recorded unless it was uploaded (and +/// charged) as the space's. +async fn snapshot_of(tx: &mut sqlx::AnyConnection, space: SpaceId, manifest: &ManifestId) -> Result { + let bytes = manifest_file_size(tx, space, manifest).await?.ok_or(MetaError::NotFound("snapshot manifest"))?; + if bytes == 0 || bytes > craft_cloud_proto::live::MAX_SNAPSHOT_BYTES as u64 { + return Err(MetaError::Invalid("a snapshot is 1 byte up to the snapshot limit")); + } + Ok(SnapshotRef { manifest: *manifest, bytes }) +} + +/// The most documents of files (rows) one space may have. A document of a file needs a file node of the space, so +/// they are bounded by the tree already; this is a loose cap on top, counted apart from recents. +pub const MAX_LIVE_DOCUMENTS_PER_SPACE: u64 = 65_536; + +/// The most documents of files holding updates one space may have. The first update to one more is refused with +/// [`MetaError::TooManyDocs`] (`quota_exceeded`, as for a full document); one that holds updates already never is. On +/// Postgres the count is read without a lock, so concurrent first updates can overshoot it by a few; that soft cap is +/// accepted. +pub const MAX_LIVE_DOCUMENTS_HOLDING_UPDATES: u64 = 10_000; + +/// The default for the most bytes of updates all of one space's live documents may hold together, compacted rows too +/// (they are kept until retention): 1 GiB. Over it, every append is refused with [`MetaError::QuotaExceeded`] +/// (`quota_exceeded`). The server's `live_bytes_per_user` sets it. +pub const DEFAULT_LIVE_BYTES_PER_SPACE: u64 = 1024 * 1024 * 1024; + +/// What an append did ([`Meta::append_live_update_counted`]). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Appended { + /// The update's sequence number (its first one, for a repeat). + pub seq: u64, + /// The document's snapshot holds the updates up to here. + pub base: u64, + /// The bytes of the log above `base`, with this update. + pub above_bytes: u64, + /// The update was stored before (a repeat after a lost ack). + pub repeat: bool, +} + +const DOC_COLUMNS: &str = "kind, node, created_ms, head_seq, bytes, compacted_bytes, base_seq, snapshot_bytes, CAST(CASE WHEN snapshot IS NULL THEN 0 ELSE 1 END AS BIGINT) AS has_snapshot"; + /// One stored update. #[derive(Debug, Clone, PartialEq, Eq)] pub struct LiveUpdate { @@ -68,6 +147,10 @@ fn doc_from(row: &sqlx::any::AnyRow, id: DocId) -> Result { created_ms: row.try_get("created_ms")?, head: to_u64(row.try_get("head_seq")?)?, bytes: to_u64(row.try_get("bytes")?)?, + compacted_bytes: to_u64(row.try_get("compacted_bytes")?)?, + base: to_u64(row.try_get("base_seq")?)?, + has_snapshot: flag(row, "has_snapshot")?, + snapshot_bytes: to_u64(row.try_get("snapshot_bytes")?)?, }) } @@ -96,7 +179,7 @@ impl Meta { .await? .rows_affected(); if inserted == 0 { - let row = sqlx::query("SELECT kind, node, created_ms, head_seq, bytes FROM live_docs WHERE space_id = $1 AND id = $2") + let row = sqlx::query(sqlx::AssertSqlSafe(format!("SELECT {DOC_COLUMNS} FROM live_docs WHERE space_id = $1 AND id = $2"))) .bind(space_key) .bind(id_key) .fetch_one(&mut *w.tx) @@ -104,9 +187,9 @@ impl Meta { let doc = doc_from(&row, id)?; return if doc.kind == kind { Ok(doc) } else { Err(MetaError::Invalid("the document exists with another kind")) }; } - // A new document: the space may not go over its cap on rows. Counted with this one; over the cap, the - // transaction (and the insert) is dropped uncommitted. - let count = sqlx::query("SELECT COUNT(*) AS n FROM live_docs WHERE space_id = $1") + // A new document: the space may not go over its cap on rows (of recents: documents of files have their own). + // Counted with this one; over the cap, the transaction (and the insert) is dropped uncommitted. + let count = sqlx::query("SELECT COUNT(*) AS n FROM live_docs WHERE space_id = $1 AND kind NOT LIKE 'document:%'") .bind(space_key) .fetch_one(&mut *w.tx) .await? @@ -115,7 +198,192 @@ impl Meta { return Err(MetaError::TooManyDocRows { limit: self.live_doc_rows_cap }); } w.commit().await?; - Ok(LiveDoc { id, kind: kind.to_owned(), node, created_ms: now_ms, head: 0, bytes: 0 }) + Ok(LiveDoc { + id, + kind: kind.to_owned(), + node, + created_ms: now_ms, + head: 0, + bytes: 0, + compacted_bytes: 0, + base: 0, + has_snapshot: false, + snapshot_bytes: 0, + }) + } + + /// Whether `node` is a file of `space` (not a folder, not deleted): what a document of a file needs. + pub async fn is_live_file(&self, space: SpaceId, node: NodeId) -> Result { + let row = sqlx::query("SELECT is_dir, deleted FROM nodes WHERE space_id = $1 AND id = $2") + .bind(uuid_bytes(space.as_uuid())) + .bind(uuid_bytes(node.as_uuid())) + .fetch_optional(&self.pool) + .await?; + match row { + Some(row) => Ok(!flag(&row, "is_dir")? && !flag(&row, "deleted")?), + None => Ok(false), + } + } + + /// Makes the document `id` of the file node `node` of `space` by import: `kind` (`document:`), its first + /// snapshot (the one `manifest` names, uploaded as the space's already), unless the space has the document already. + /// Returns the document and whether this snapshot was kept: the first one is, every later one (another machine + /// imported the same file at the same moment) is not, and the document stays as it is. `NotFound` if the space has + /// no such file (a missing, deleted or folder node) or no such manifest, `Invalid` for a manifest of a size no + /// snapshot has, `TooManyDocRows` over the space's cap on documents of files. + pub async fn create_live_document( + &self, + space: SpaceId, + id: DocId, + kind: &str, + node: NodeId, + manifest: &ManifestId, + now_ms: i64, + ) -> Result<(LiveDoc, bool), MetaError> { + if !kind.starts_with(craft_cloud_proto::live::DOCUMENT_KIND_PREFIX) || kind.len() > 64 { + return Err(MetaError::Invalid("a document of a file is of kind document:")); + } + let (space_key, id_key, node_key) = (uuid_bytes(space.as_uuid()), uuid_bytes(id.as_uuid()), uuid_bytes(node.as_uuid())); + let mut w = self.begin_write().await?; + let file = sqlx::query("SELECT is_dir, deleted FROM nodes WHERE space_id = $1 AND id = $2") + .bind(space_key.clone()) + .bind(node_key.clone()) + .fetch_optional(&mut *w.tx) + .await?; + match file { + Some(row) if !flag(&row, "is_dir")? && !flag(&row, "deleted")? => {} + _ => return Err(MetaError::NotFound("file")), + } + let snapshot = snapshot_of(&mut w.tx, space, manifest).await?; + let inserted = sqlx::query( + "INSERT INTO live_docs (space_id, id, kind, node, created_ms, head_seq, base_seq, snapshot, snapshot_bytes, import_snapshot, import_bytes) \ + VALUES ($1, $2, $3, $4, $5, 0, 0, $6, $7, $6, $7) ON CONFLICT (space_id, id) DO NOTHING", + ) + .bind(space_key.clone()) + .bind(id_key.clone()) + .bind(kind) + .bind(node_key) + .bind(now_ms) + .bind(snapshot.manifest.as_bytes().to_vec()) + .bind(to_i64(snapshot.bytes)?) + .execute(&mut *w.tx) + .await? + .rows_affected(); + if inserted == 0 { + let row = sqlx::query(sqlx::AssertSqlSafe(format!("SELECT {DOC_COLUMNS} FROM live_docs WHERE space_id = $1 AND id = $2"))) + .bind(space_key) + .bind(id_key) + .fetch_one(&mut *w.tx) + .await?; + let doc = doc_from(&row, id)?; + // Another machine's import (of any app) was first: this one is not kept, and nothing changes. + return if doc.kind.starts_with(craft_cloud_proto::live::DOCUMENT_KIND_PREFIX) && doc.node == Some(node) { + Ok((doc, false)) + } else { + Err(MetaError::Invalid("the document exists with another kind")) + }; + } + let count = sqlx::query("SELECT COUNT(*) AS n FROM live_docs WHERE space_id = $1 AND kind LIKE 'document:%'") + .bind(space_key) + .fetch_one(&mut *w.tx) + .await? + .try_get::("n")?; + if to_u64(count)? > self.live_documents_cap { + return Err(MetaError::TooManyDocRows { limit: self.live_documents_cap }); + } + w.commit().await?; + let doc = LiveDoc { + id, + kind: kind.to_owned(), + node: Some(node), + created_ms: now_ms, + head: 0, + bytes: 0, + compacted_bytes: 0, + base: 0, + has_snapshot: true, + snapshot_bytes: snapshot.bytes, + }; + Ok((doc, true)) + } + + /// The document's snapshot, if it has one: the last update it holds, and where it is. + pub async fn live_snapshot(&self, space: SpaceId, doc: DocId) -> Result, MetaError> { + let row = sqlx::query("SELECT base_seq, snapshot, snapshot_bytes FROM live_docs WHERE space_id = $1 AND id = $2") + .bind(uuid_bytes(space.as_uuid())) + .bind(uuid_bytes(doc.as_uuid())) + .fetch_optional(&self.pool) + .await?; + let Some(row) = row else { return Ok(None) }; + let Some(ids) = row.try_get::>, _>("snapshot")? else { return Ok(None) }; + let snapshot = SnapshotRef::decode(&ids, to_u64(row.try_get("snapshot_bytes")?)?)?; + Ok(Some((to_u64(row.try_get("base_seq")?)?, snapshot))) + } + + /// The snapshot a document of a file was made by import with (kept when its log is compacted: the log doesn't hold + /// it), if it has one. + pub async fn live_import_snapshot(&self, space: SpaceId, doc: DocId) -> Result, MetaError> { + let row = sqlx::query("SELECT import_snapshot, import_bytes FROM live_docs WHERE space_id = $1 AND id = $2") + .bind(uuid_bytes(space.as_uuid())) + .bind(uuid_bytes(doc.as_uuid())) + .fetch_optional(&self.pool) + .await?; + let Some(row) = row else { return Ok(None) }; + let Some(ids) = row.try_get::>, _>("import_snapshot")? else { return Ok(None) }; + Ok(Some(SnapshotRef::decode(&ids, to_u64(row.try_get("import_bytes")?)?)?)) + } + + /// Compacts the document's log: the snapshot `manifest` names (uploaded as the space's already, durably) holds the + /// updates 1..=`upto` (and the snapshot before it). Recorded with `base = upto` in one transaction, and the rows up + /// to `upto` stop counting against the document's cap; they are kept. Only a snapshot past the current base, and + /// not past the head, is taken: one for an older or equal `upto` (a second device answering late, or two at once) + /// is ignored, `Ok(false)`. `NotFound` if the space has no such manifest, `Invalid` for one of a size no snapshot + /// has (nothing changes). + pub async fn compact_live_doc(&self, space: SpaceId, doc: DocId, upto: u64, manifest: &ManifestId) -> Result { + let (space_key, doc_key) = (uuid_bytes(space.as_uuid()), uuid_bytes(doc.as_uuid())); + let mut w = self.begin_write().await?; + let snapshot = snapshot_of(&mut w.tx, space, manifest).await?; + let row = sqlx::query("SELECT base_seq, head_seq FROM live_docs WHERE space_id = $1 AND id = $2") + .bind(space_key.clone()) + .bind(doc_key.clone()) + .fetch_optional(&mut *w.tx) + .await? + .ok_or(MetaError::NotFound("document"))?; + let (base, head) = (to_u64(row.try_get("base_seq")?)?, to_u64(row.try_get("head_seq")?)?); + if upto <= base || upto > head { + return Ok(false); + } + // Postgres sums BIGINT as NUMERIC, which the `Any` driver can't read: cast back. + let compacted: i64 = sqlx::query( + "SELECT CAST(COALESCE(SUM(LENGTH(bytes)), 0) AS BIGINT) AS n FROM live_updates WHERE space_id = $1 AND doc = $2 AND seq > $3 AND seq <= $4", + ) + .bind(space_key.clone()) + .bind(doc_key.clone()) + .bind(to_i64(base)?) + .bind(to_i64(upto)?) + .fetch_one(&mut *w.tx) + .await? + .try_get("n")?; + // Only from the base read above: a compaction that raced this one (Postgres) leaves nothing to change here. + let done = sqlx::query( + "UPDATE live_docs SET base_seq = $3, snapshot = $4, snapshot_bytes = $5, compacted_bytes = compacted_bytes + $6 \ + WHERE space_id = $1 AND id = $2 AND base_seq = $7", + ) + .bind(space_key) + .bind(doc_key) + .bind(to_i64(upto)?) + .bind(snapshot.manifest.as_bytes().to_vec()) + .bind(to_i64(snapshot.bytes)?) + .bind(compacted) + .bind(to_i64(base)?) + .execute(&mut *w.tx) + .await? + .rows_affected(); + if done == 0 { + return Ok(false); + } + w.commit().await?; + Ok(true) } /// The most bytes of updates a live document's log may hold ([`Meta::with_live_doc_cap`]). @@ -127,8 +395,9 @@ impl Meta { /// live generation in the same transaction, so its live epoch ([`Meta::live_epoch`]) changes: a reset no agent can /// miss, however far the log grows back before it looks (its `welcome`, and every connection's ping, compare the /// live epoch). Every agent then catches up from the start and sends up what the log lacks of its document; nothing - /// any machine holds is lost. The space's own epoch, which the file sync reads, doesn't change. `NotFound` if the - /// space has no such document. + /// any machine holds is lost. The space's own epoch, which the file sync reads, doesn't change. The snapshot goes + /// too (`base` back to 0; its blobs stay in the chunk store): machines that hold the document send it up again as + /// updates. `NotFound` if the space has no such document. pub async fn reset_live_doc(&self, space: SpaceId, doc: DocId) -> Result<(), MetaError> { let (space_key, doc_key) = (uuid_bytes(space.as_uuid()), uuid_bytes(doc.as_uuid())); let mut w = self.begin_write().await?; @@ -139,11 +408,20 @@ impl Meta { .bind(space_key.clone()) .execute(&mut *w.tx) .await?; - let done = sqlx::query("UPDATE live_docs SET head_seq = 0, bytes = 0 WHERE space_id = $1 AND id = $2") + // Its rows no longer count towards the space's live bytes. + sqlx::query("UPDATE spaces SET live_bytes = live_bytes - (SELECT bytes FROM live_docs WHERE space_id = $1 AND id = $2) WHERE id = $1 AND EXISTS (SELECT 1 FROM live_docs WHERE space_id = $1 AND id = $2)") .bind(space_key.clone()) .bind(doc_key.clone()) .execute(&mut *w.tx) .await?; + let done = sqlx::query( + "UPDATE live_docs SET head_seq = 0, bytes = 0, compacted_bytes = 0, base_seq = 0, snapshot = NULL, snapshot_bytes = 0, \ + import_snapshot = NULL, import_bytes = 0 WHERE space_id = $1 AND id = $2", + ) + .bind(space_key.clone()) + .bind(doc_key.clone()) + .execute(&mut *w.tx) + .await?; if done.rows_affected() == 0 { return Err(MetaError::NotFound("document")); } @@ -174,7 +452,7 @@ impl Meta { /// The document `id` of `space`, if it has one. pub async fn live_doc(&self, space: SpaceId, id: DocId) -> Result, MetaError> { - let row = sqlx::query("SELECT kind, node, created_ms, head_seq, bytes FROM live_docs WHERE space_id = $1 AND id = $2") + let row = sqlx::query(sqlx::AssertSqlSafe(format!("SELECT {DOC_COLUMNS} FROM live_docs WHERE space_id = $1 AND id = $2"))) .bind(uuid_bytes(space.as_uuid())) .bind(uuid_bytes(id.as_uuid())) .fetch_optional(&self.pool) @@ -194,22 +472,41 @@ impl Meta { bytes: &[u8], now_ms: i64, ) -> Result { + Ok(self.append_live_update_counted(space, doc, device, client_seq, bytes, now_ms).await?.seq) + } + + /// [`Meta::append_live_update`], also saying how long the document's log is above its snapshot (whether to compact + /// it). + pub async fn append_live_update_counted( + &self, + space: SpaceId, + doc: DocId, + device: DeviceId, + client_seq: u64, + bytes: &[u8], + now_ms: i64, + ) -> Result { let client_seq = to_i64(client_seq)?; let (space_key, doc_key, device_key) = (uuid_bytes(space.as_uuid()), uuid_bytes(doc.as_uuid()), uuid_bytes(device.as_uuid())); let mut w = self.begin_write().await?; // Taking the next number locks the document's row (on Postgres), so appends to one document run one at a // time and the check for a repeat below can't race another append of the same update. let size = i64::try_from(bytes.len()).map_err(|_| MetaError::Invalid("an update too large"))?; - let head = - sqlx::query("UPDATE live_docs SET head_seq = head_seq + 1, bytes = bytes + $3 WHERE space_id = $1 AND id = $2 RETURNING head_seq, bytes") - .bind(space_key.clone()) - .bind(doc_key.clone()) - .bind(size) - .fetch_optional(&mut *w.tx) - .await? - .ok_or(MetaError::NotFound("document"))?; + let head = sqlx::query( + "UPDATE live_docs SET head_seq = head_seq + 1, bytes = bytes + $3 WHERE space_id = $1 AND id = $2 \ + RETURNING head_seq, bytes, compacted_bytes, base_seq, kind", + ) + .bind(space_key.clone()) + .bind(doc_key.clone()) + .bind(size) + .fetch_optional(&mut *w.tx) + .await? + .ok_or(MetaError::NotFound("document"))?; let seq: i64 = head.try_get("head_seq")?; - let total = to_u64(head.try_get("bytes")?)?; + // What counts against the cap: the log above its snapshot. + let total = to_u64(head.try_get("bytes")?)?.saturating_sub(to_u64(head.try_get("compacted_bytes")?)?); + let base = to_u64(head.try_get("base_seq")?)?; + let recents = !head.try_get::("kind")?.starts_with(craft_cloud_proto::live::DOCUMENT_KIND_PREFIX); let earlier = sqlx::query("SELECT seq, bytes FROM live_updates WHERE space_id = $1 AND doc = $2 AND device = $3 AND client_seq = $4") .bind(space_key.clone()) .bind(doc_key.clone()) @@ -222,7 +519,7 @@ impl Meta { let first = to_u64(row.try_get("seq")?)?; // The number taken above goes back. w.tx.rollback().await?; - return Ok(first); + return Ok(Appended { seq: first, base, above_bytes: total.saturating_sub(to_u64(size)?), repeat: true }); } } // Full is one way: once the log holds the cap, every append is refused (the one that crosses it is stored), so @@ -233,21 +530,43 @@ impl Meta { w.tx.rollback().await?; return Err(MetaError::DocFull { used, limit: self.live_doc_cap }); } - // The first update to a document: the space may have only so many that hold updates. Counted after the row lock - // above, in the same transaction; over the cap, nothing is stored and the number goes back, as for a full - // document. A repeat was acked above, and a document that holds updates already is never refused here. - if used == 0 { - let holding = sqlx::query("SELECT COUNT(*) AS n FROM live_docs WHERE space_id = $1 AND id <> $2 AND bytes > 0") - .bind(space_key.clone()) - .bind(doc_key.clone()) - .fetch_one(&mut *w.tx) - .await? - .try_get::("n")?; - if to_u64(holding)? >= self.live_docs_cap { + // The first update to a recents document: the space may have only so many that hold updates. Counted after the + // row lock above, in the same transaction; over the cap, nothing is stored and the number goes back, as for a + // full document. A repeat was acked above, and a document that holds updates already is never refused here + // (by its numbers, not its bytes: a compacted log holds updates, though none count against its cap). Documents + // of files aren't counted: each needs a file of the space. + if seq == 1 { + // Documents of files are counted apart: each needs a file of the space, and their cap is far higher. + let (sql, cap) = if recents { + ( + "SELECT COUNT(*) AS n FROM live_docs WHERE space_id = $1 AND id <> $2 AND head_seq > 0 AND kind NOT LIKE 'document:%'", + self.live_docs_cap, + ) + } else { + ( + "SELECT COUNT(*) AS n FROM live_docs WHERE space_id = $1 AND id <> $2 AND head_seq > 0 AND kind LIKE 'document:%'", + self.live_held_documents_cap, + ) + }; + let holding = sqlx::query(sql).bind(space_key.clone()).bind(doc_key.clone()).fetch_one(&mut *w.tx).await?.try_get::("n")?; + if to_u64(holding)? >= cap { w.tx.rollback().await?; - return Err(MetaError::TooManyDocs { limit: self.live_docs_cap }); + return Err(MetaError::TooManyDocs { limit: cap }); } } + // Every row of every live document of the space, compacted ones too, counts towards its live bytes (kept until + // retention): over the cap, nothing is stored. A repeat was acked above. + let live: i64 = sqlx::query("UPDATE spaces SET live_bytes = live_bytes + $1 WHERE id = $2 RETURNING live_bytes") + .bind(size) + .bind(space_key.clone()) + .fetch_one(&mut *w.tx) + .await? + .try_get("live_bytes")?; + let live = to_u64(live)?; + if live > self.live_bytes_cap { + w.tx.rollback().await?; + return Err(MetaError::LiveFull { used: live.saturating_sub(to_u64(size)?), limit: self.live_bytes_cap }); + } sqlx::query("INSERT INTO live_updates (space_id, doc, seq, bytes, device, client_seq, created_ms) VALUES ($1, $2, $3, $4, $5, $6, $7)") .bind(space_key) .bind(doc_key) @@ -259,7 +578,7 @@ impl Meta { .execute(&mut *w.tx) .await?; w.commit().await?; - to_u64(seq) + Ok(Appended { seq: to_u64(seq)?, base, above_bytes: total, repeat: false }) } /// Every live document of `space` with its last sequence number: what a backup of the database holds. A test seam @@ -287,12 +606,44 @@ impl Meta { .execute(&mut *w.tx) .await?; let sql = if head.is_some() { - "UPDATE live_docs SET head_seq = $3, bytes = (SELECT COALESCE(SUM(LENGTH(u.bytes)), 0) FROM live_updates u \ - WHERE u.space_id = $1 AND u.doc = $2) WHERE space_id = $1 AND id = $2" + // (A snapshot past the head the backup has goes back to the head: what a backup taken back then holds is + // [`Meta::restore_live_snapshot`]'s to set.) + "UPDATE live_docs SET head_seq = $3, base_seq = CASE WHEN base_seq > $3 THEN $3 ELSE base_seq END, \ + bytes = (SELECT COALESCE(SUM(LENGTH(u.bytes)), 0) FROM live_updates u WHERE u.space_id = $1 AND u.doc = $2), \ + compacted_bytes = (SELECT COALESCE(SUM(LENGTH(u.bytes)), 0) FROM live_updates u \ + WHERE u.space_id = $1 AND u.doc = $2 AND u.seq <= live_docs.base_seq AND u.seq <= $3) WHERE space_id = $1 AND id = $2" } else { "DELETE FROM live_docs WHERE space_id = $1 AND id = $2 AND $3 = 0" }; - sqlx::query(sql).bind(space_key).bind(doc_key).bind(keep).execute(&mut *w.tx).await?; + sqlx::query(sql).bind(space_key.clone()).bind(doc_key).bind(keep).execute(&mut *w.tx).await?; + sqlx::query("UPDATE spaces SET live_bytes = (SELECT CAST(COALESCE(SUM(d.bytes), 0) AS BIGINT) FROM live_docs d WHERE d.space_id = spaces.id) WHERE id = $1") + .bind(space_key) + .execute(&mut *w.tx) + .await?; + w.commit().await + } + + /// The document's snapshot and base as a backup holds them (`None`: no snapshot, base 0). A test seam (the + /// `test-support` feature), for the simulations' restores; the manifest must be the space's. + #[cfg(feature = "test-support")] + pub async fn restore_live_snapshot(&self, space: SpaceId, doc: DocId, snapshot: Option<(u64, SnapshotRef)>) -> Result<(), MetaError> { + let (space_key, doc_key) = (uuid_bytes(space.as_uuid()), uuid_bytes(doc.as_uuid())); + let mut w = self.begin_write().await?; + let (base, ids, size) = match &snapshot { + Some((base, s)) => (to_i64(*base)?, Some(s.manifest.as_bytes().to_vec()), to_i64(s.bytes)?), + None => (0, None, 0), + }; + sqlx::query( + "UPDATE live_docs SET base_seq = $3, snapshot = $4, snapshot_bytes = $5, compacted_bytes = (SELECT COALESCE(SUM(LENGTH(u.bytes)), 0) \ + FROM live_updates u WHERE u.space_id = $1 AND u.doc = $2 AND u.seq <= $3) WHERE space_id = $1 AND id = $2", + ) + .bind(space_key) + .bind(doc_key) + .bind(base) + .bind(ids) + .bind(size) + .execute(&mut *w.tx) + .await?; w.commit().await } diff --git a/crates/meta/src/schema.rs b/crates/meta/src/schema.rs index 1771af1..61cf3a3 100644 --- a/crates/meta/src/schema.rs +++ b/crates/meta/src/schema.rs @@ -10,12 +10,14 @@ const SQLITE: &[&str] = &[ include_str!("../migrations/sqlite/0002_stored_bytes.sql"), include_str!("../migrations/sqlite/0003_live.sql"), include_str!("../migrations/sqlite/0004_join.sql"), + include_str!("../migrations/sqlite/0005_live_documents.sql"), ]; const POSTGRES: &[&str] = &[ include_str!("../migrations/postgres/0001_init.sql"), include_str!("../migrations/postgres/0002_stored_bytes.sql"), include_str!("../migrations/postgres/0003_live.sql"), include_str!("../migrations/postgres/0004_join.sql"), + include_str!("../migrations/postgres/0005_live_documents.sql"), ]; /// Arbitrary fixed key for the Postgres advisory lock that serialises migrations between servers sharing one diff --git a/crates/meta/tests/behaviour.rs b/crates/meta/tests/behaviour.rs index 314cf8e..e2ed4af 100644 --- a/crates/meta/tests/behaviour.rs +++ b/crates/meta/tests/behaviour.rs @@ -117,6 +117,14 @@ on_every_backend!( a_live_document_is_capped_by_bytes, live_documents_holding_updates_are_capped, live_document_rows_are_capped, + a_document_is_made_by_import_once, + concurrent_imports_of_one_file_keep_one, + a_log_is_compacted_into_a_snapshot, + concurrent_compactions_take_one, + a_space_s_live_bytes_are_capped, + documents_of_files_have_caps_of_their_own, + a_compacted_log_still_holds_updates, + the_import_stays_when_the_log_is_compacted, a_pairing_code_works_once, an_expired_code_is_refused, a_replaced_invite_s_old_code_fails, @@ -766,6 +774,312 @@ async fn live_document_rows_are_capped(meta: &Meta) { meta.create_live_doc(bo.space, extra, "recents:wordcraft", None, NOW).await.unwrap(); } +/// A file of `u`'s space (a node with a version), at the root under `file_name`. +async fn file(meta: &Meta, u: &UserRecord, file_name: &str) -> NodeId { + let dev = device(meta, u).await; + let r = root(meta, u).await; + let m = manifest(meta, u, 10).await; + let node = NodeId::generate(); + meta.commit(u.space, dev, &[Op::Create { node, parent: r, name: name(file_name), version: version(m) }], NOW).await.unwrap(); + node +} + +/// A snapshot of `size` bytes uploaded as `u`'s (its chunks and the manifest that names them), as an agent uploads one. +async fn snapshot(meta: &Meta, u: &UserRecord, size: u64) -> craft_cloud_meta::SnapshotRef { + let max = u64::from(craft_cloud_proto::limits::MAX_CHUNK_BYTES); + let mut chunks = Vec::new(); + let mut left = size; + while left > 0 { + let len = left.min(max); + let chunk = ChunkId::from_bytes(*blake3::hash(uuid::Uuid::now_v7().as_bytes()).as_bytes()); + meta.add_chunk(u.space, &chunk, len + 1, NOW).await.unwrap(); + chunks.push(ChunkRef { id: chunk, len: u32::try_from(len).unwrap() }); + left -= len; + } + let m = Manifest::new(ChunkerParamsId::new("fastcdc-v1").unwrap(), size, FileHash::from_bytes([1; 32]), chunks).unwrap(); + meta.add_manifest(u.space, &m, 100, NOW).await.unwrap(); + craft_cloud_meta::SnapshotRef { manifest: m.id(), bytes: size } +} + +async fn a_document_is_made_by_import_once(meta: &Meta) { + use craft_cloud_proto::live::node_doc; + let ana = user(meta, "ana").await; + let bo = user(meta, "bo").await; + let node = file(meta, &ana, "a.txt").await; + let doc = node_doc(ana.space, node); + let first = snapshot(meta, &ana, 100).await; + let before = meta.stored_bytes(ana.space).await.unwrap(); + let (made, kept) = meta.create_live_document(ana.space, doc, "document:wordcraft", node, &first.manifest, NOW).await.unwrap(); + assert!(kept, "the first import is kept"); + assert_eq!((made.node, made.base, made.has_snapshot, made.snapshot_bytes, made.head), (Some(node), 0, true, 100, 0)); + assert_eq!(meta.stored_bytes(ana.space).await.unwrap(), before, "charged when it was uploaded, not again"); + // Another machine's import of the same file (even by another app): not kept, nothing changes. + let second = snapshot(meta, &ana, 50).await; + let (again, kept) = meta.create_live_document(ana.space, doc, "document:gridcraft", node, &second.manifest, NOW + 1).await.unwrap(); + assert!(!kept); + assert_eq!(again, made); + assert_eq!(meta.live_snapshot(ana.space, doc).await.unwrap(), Some((0, first)), "the first snapshot stays"); + // Only for a file of the space that is there. + let one = snapshot(meta, &ana, 1).await.manifest; + let missing = NodeId::generate(); + assert!(!meta.is_live_file(ana.space, missing).await.unwrap()); + let refused = meta.create_live_document(ana.space, node_doc(ana.space, missing), "document:wordcraft", missing, &one, NOW).await; + assert!(matches!(refused, Err(MetaError::NotFound(_))), "{refused:?}"); + let r = root(meta, &ana).await; + assert!(!meta.is_live_file(ana.space, r).await.unwrap()); + let refused = meta.create_live_document(ana.space, node_doc(ana.space, r), "document:wordcraft", r, &one, NOW).await; + assert!(matches!(refused, Err(MetaError::NotFound(_))), "a folder: {refused:?}"); + let bos = snapshot(meta, &bo, 1).await.manifest; + let refused = meta.create_live_document(bo.space, node_doc(bo.space, node), "document:wordcraft", node, &bos, NOW).await; + assert!(matches!(refused, Err(MetaError::NotFound(_))), "another account's file: {refused:?}"); + assert_eq!(meta.live_doc(bo.space, doc).await.unwrap(), None, "and another account's document"); + // Only with a snapshot the space uploaded. + let other = file(meta, &ana, "other.txt").await; + assert!(meta.is_live_file(ana.space, other).await.unwrap()); + let refused = meta.create_live_document(ana.space, node_doc(ana.space, other), "document:wordcraft", other, &bos, NOW).await; + assert!(matches!(refused, Err(MetaError::NotFound(_))), "another account's manifest: {refused:?}"); + let never = ManifestId::from_bytes([9; 32]); + let refused = meta.create_live_document(ana.space, node_doc(ana.space, other), "document:wordcraft", other, &never, NOW).await; + assert!(matches!(refused, Err(MetaError::NotFound(_))), "a manifest never uploaded: {refused:?}"); + assert_eq!(meta.live_doc(ana.space, node_doc(ana.space, other)).await.unwrap(), None, "nothing recorded"); + // A deleted file. + let gone = file(meta, &ana, "gone.txt").await; + let dev = device(meta, &ana).await; + let tree = full_tree(meta, ana.space).await; + let current = match &find(&tree.nodes, gone).kind { + NodeKind::File(state) => state.version, + _ => panic!("not a file"), + }; + meta.commit(ana.space, dev, &[Op::Delete { node: gone, expected: Some(current) }], NOW).await.unwrap(); + assert!(!meta.is_live_file(ana.space, gone).await.unwrap()); + let refused = meta.create_live_document(ana.space, node_doc(ana.space, gone), "document:wordcraft", gone, &one, NOW).await; + assert!(matches!(refused, Err(MetaError::NotFound(_))), "a deleted file: {refused:?}"); + // Of the right kind, and with a snapshot of a snapshot's size: up to the limit, not past it. + assert!(matches!(meta.create_live_document(ana.space, doc, "recents:wordcraft", node, &first.manifest, NOW).await, Err(MetaError::Invalid(_)))); + let limit = craft_cloud_proto::live::MAX_SNAPSHOT_BYTES as u64; + let empty = snapshot(meta, &ana, 0).await.manifest; + let refused = meta.create_live_document(ana.space, node_doc(ana.space, other), "document:wordcraft", other, &empty, NOW).await; + assert!(matches!(refused, Err(MetaError::Invalid(_))), "an empty one: {refused:?}"); + let over = snapshot(meta, &ana, limit + 1).await.manifest; + let refused = meta.create_live_document(ana.space, node_doc(ana.space, other), "document:wordcraft", other, &over, NOW).await; + assert!(matches!(refused, Err(MetaError::Invalid(_))), "one past the limit: {refused:?}"); + let big = snapshot(meta, &ana, limit).await; + meta.create_live_document(ana.space, node_doc(ana.space, other), "document:wordcraft", other, &big.manifest, NOW).await.unwrap(); + assert_eq!(meta.live_snapshot(ana.space, node_doc(ana.space, other)).await.unwrap(), Some((0, big))); +} + +async fn concurrent_imports_of_one_file_keep_one(meta: &Meta) { + use craft_cloud_proto::live::node_doc; + let ana = user(meta, "ana").await; + let node = file(meta, &ana, "a.txt").await; + let doc = node_doc(ana.space, node); + let mut snapshots = Vec::new(); + for i in 0..8u64 { + snapshots.push(snapshot(meta, &ana, 64 + i).await); + } + let tasks: Vec<_> = snapshots + .into_iter() + .map(|s| { + let meta = meta.clone(); + tokio::spawn(async move { + let (_, kept) = meta.create_live_document(ana.space, doc, "document:wordcraft", node, &s.manifest, NOW).await.unwrap(); + (kept, s) + }) + }) + .collect(); + let mut kept = Vec::new(); + for task in tasks { + let (was_kept, s) = task.await.unwrap(); + if was_kept { + kept.push(s); + } + } + assert_eq!(kept.len(), 1, "exactly one import is kept"); + assert_eq!(meta.live_snapshot(ana.space, doc).await.unwrap(), Some((0, kept.remove(0))), "and it is the document's"); +} + +async fn a_log_is_compacted_into_a_snapshot(meta: &Meta) { + use craft_cloud_proto::DocId; + let meta = meta.clone().with_live_doc_cap(450); + let ana = user(&meta, "ana").await; + let dev = device(&meta, &ana).await; + let doc = DocId::generate(); + meta.create_live_doc(ana.space, doc, "recents:wordcraft", None, NOW).await.unwrap(); + for i in 1..=5u64 { + let appended = meta.append_live_update_counted(ana.space, doc, dev, i, &[u8::try_from(i).unwrap(); 100], NOW).await.unwrap(); + assert_eq!((appended.seq, appended.base, appended.above_bytes, appended.repeat), (i, 0, i * 100, false)); + } + let repeat = meta.append_live_update_counted(ana.space, doc, dev, 2, &[2; 100], NOW).await.unwrap(); + assert_eq!((repeat.seq, repeat.repeat), (2, true)); + assert!(meta.append_live_update(ana.space, doc, dev, 6, &[6; 10], NOW).await.is_err(), "500 bytes: full"); + let s = snapshot(&meta, &ana, 40).await; + assert!(meta.compact_live_doc(ana.space, doc, 3, &s.manifest).await.unwrap()); + let compacted = meta.live_doc(ana.space, doc).await.unwrap().unwrap(); + assert_eq!((compacted.base, compacted.above_base(), compacted.head, compacted.snapshot_bytes), (3, 200, 5, 40), "rows 1-3 no longer count"); + assert_eq!((compacted.bytes, compacted.compacted_bytes), (500, 300), "but stay in the log"); + assert_eq!(meta.live_snapshot(ana.space, doc).await.unwrap(), Some((3, s))); + assert_eq!(meta.live_updates(ana.space, doc, 0, 10).await.unwrap().len(), 5, "the compacted rows are kept"); + // Older, equal or past the head: ignored. + let one = snapshot(&meta, &ana, 1).await.manifest; + for (upto, why) in [(3, "equal"), (2, "older"), (6, "past the head"), (0, "nothing")] { + assert!(!meta.compact_live_doc(ana.space, doc, upto, &one).await.unwrap(), "{why}"); + } + // Only with a snapshot the space uploaded. + let refused = meta.compact_live_doc(ana.space, doc, 4, &ManifestId::from_bytes([9; 32])).await; + assert!(matches!(refused, Err(MetaError::NotFound(_))), "{refused:?}"); + assert_eq!(meta.live_snapshot(ana.space, doc).await.unwrap(), Some((3, s)), "unchanged"); + // Under the cap again: appends go on. + assert_eq!(meta.append_live_update(ana.space, doc, dev, 6, &[6; 10], NOW).await.unwrap(), 6); + assert!(meta.compact_live_doc(ana.space, doc, 6, &snapshot(&meta, &ana, 50).await.manifest).await.unwrap()); + assert_eq!(meta.live_doc(ana.space, doc).await.unwrap().unwrap().above_base(), 0); + assert!(matches!(meta.compact_live_doc(ana.space, DocId::generate(), 1, &one).await, Err(MetaError::NotFound(_)))); + // A reset empties the log and drops the snapshot. + meta.reset_live_doc(ana.space, doc).await.unwrap(); + let reset = meta.live_doc(ana.space, doc).await.unwrap().unwrap(); + assert_eq!((reset.head, reset.base, reset.bytes, reset.has_snapshot, reset.snapshot_bytes), (0, 0, 0, false, 0)); + assert_eq!(meta.live_snapshot(ana.space, doc).await.unwrap(), None); +} + +async fn concurrent_compactions_take_one(meta: &Meta) { + use craft_cloud_proto::DocId; + let ana = user(meta, "ana").await; + let dev = device(meta, &ana).await; + let doc = DocId::generate(); + meta.create_live_doc(ana.space, doc, "recents:wordcraft", None, NOW).await.unwrap(); + for i in 1..=10u64 { + meta.append_live_update(ana.space, doc, dev, i, &[1; 10], NOW).await.unwrap(); + } + let mut snapshots = Vec::new(); + for i in 0..6u64 { + snapshots.push((i, snapshot(meta, &ana, 8 + i).await.manifest)); + } + let tasks: Vec<_> = snapshots + .into_iter() + .map(|(i, s)| { + let meta = meta.clone(); + let upto = 4 + (i % 2) * 4; + tokio::spawn(async move { (upto, meta.compact_live_doc(ana.space, doc, upto, &s).await.unwrap()) }) + }) + .collect(); + let mut taken = Vec::new(); + for task in tasks { + let (upto, ok) = task.await.unwrap(); + if ok { + taken.push(upto); + } + } + let after = meta.live_doc(ana.space, doc).await.unwrap().unwrap(); + assert!(taken.len() <= 2 && taken.iter().filter(|u| **u == 4).count() <= 1 && taken.iter().filter(|u| **u == 8).count() <= 1, "{taken:?}"); + assert_eq!(after.base, *taken.iter().max().unwrap(), "the base is the furthest one taken"); + assert_eq!(after.above_base(), (10 - after.base) * 10, "each row stopped counting once"); + assert_eq!(after.bytes, 100); +} + +/// Every row of every live document of a space, compacted ones too, counts towards its live bytes: over the cap, every +/// append is refused (nothing stored), and a reset gives the document's bytes back. +async fn a_space_s_live_bytes_are_capped(meta: &Meta) { + use craft_cloud_proto::DocId; + let meta = meta.clone().with_live_bytes_cap(250); + let ana = user(&meta, "ana").await; + let bo = user(&meta, "bo").await; + let dev = device(&meta, &ana).await; + let (a, b) = (DocId::generate(), DocId::generate()); + for doc in [a, b] { + meta.create_live_doc(ana.space, doc, "recents:wordcraft", None, NOW).await.unwrap(); + } + meta.append_live_update(ana.space, a, dev, 1, &[1; 100], NOW).await.unwrap(); + meta.append_live_update(ana.space, b, dev, 1, &[2; 100], NOW).await.unwrap(); + // Compacting doesn't give bytes back: the rows are kept. + assert!(meta.compact_live_doc(ana.space, a, 1, &snapshot(&meta, &ana, 10).await.manifest).await.unwrap()); + assert_eq!(meta.append_live_update(ana.space, a, dev, 2, &[3; 50], NOW).await.unwrap(), 2, "250: at the cap"); + let refused = meta.append_live_update(ana.space, b, dev, 2, &[4; 1], NOW).await; + assert!(matches!(refused, Err(MetaError::LiveFull { used: 250, limit: 250 })), "{refused:?}"); + assert_eq!(meta.live_doc(ana.space, b).await.unwrap().unwrap().head, 1, "nothing stored"); + // A repeat of a stored update is still acked. + assert_eq!(meta.append_live_update(ana.space, a, dev, 2, &[3; 50], NOW).await.unwrap(), 2); + // Another account has room of its own. + let bo_dev = device(&meta, &bo).await; + let theirs = DocId::generate(); + meta.create_live_doc(bo.space, theirs, "recents:wordcraft", None, NOW).await.unwrap(); + assert_eq!(meta.append_live_update(bo.space, theirs, bo_dev, 1, &[5; 200], NOW).await.unwrap(), 1); + // A reset gives the document's bytes back. + meta.reset_live_doc(ana.space, a).await.unwrap(); + assert_eq!(meta.append_live_update(ana.space, b, dev, 2, &[4; 100], NOW).await.unwrap(), 2); +} + +async fn documents_of_files_have_caps_of_their_own(meta: &Meta) { + use craft_cloud_proto::DocId; + use craft_cloud_proto::live::node_doc; + let meta = meta.clone().with_live_docs_cap(1).with_live_doc_rows_cap(1).with_live_documents_cap(3).with_live_held_documents_cap(2); + let ana = user(&meta, "ana").await; + let dev = device(&meta, &ana).await; + let recents = DocId::generate(); + meta.create_live_doc(ana.space, recents, "recents:wordcraft", None, NOW).await.unwrap(); + meta.append_live_update(ana.space, recents, dev, 1, b"x", NOW).await.unwrap(); + // Documents of files count neither as recents rows nor as recents holding updates. + let one = snapshot(&meta, &ana, 1).await.manifest; + let mut docs = Vec::new(); + for i in 0..3 { + let node = file(&meta, &ana, &format!("f{i}.txt")).await; + let doc = node_doc(ana.space, node); + meta.create_live_document(ana.space, doc, "document:wordcraft", node, &one, NOW).await.unwrap(); + docs.push(doc); + } + // Holding updates: two may, the third's first update is refused (and a held one goes on). + for doc in &docs[..2] { + assert_eq!(meta.append_live_update(ana.space, *doc, dev, 1, b"y", NOW).await.unwrap(), 1); + } + let refused = meta.append_live_update(ana.space, docs[2], dev, 1, b"y", NOW).await; + assert!(matches!(refused, Err(MetaError::TooManyDocs { limit: 2 })), "{refused:?}"); + assert_eq!(meta.append_live_update(ana.space, docs[0], dev, 2, b"z", NOW).await.unwrap(), 2); + let fourth = file(&meta, &ana, "f3.txt").await; + let refused = meta.create_live_document(ana.space, node_doc(ana.space, fourth), "document:wordcraft", fourth, &one, NOW).await; + assert!(matches!(refused, Err(MetaError::TooManyDocRows { limit: 3 })), "{refused:?}"); + assert!(matches!( + meta.create_live_doc(ana.space, DocId::generate(), "recents:other", None, NOW).await, + Err(MetaError::TooManyDocRows { limit: 1 }) + )); +} + +/// The log doesn't hold a document's import: compaction replaces the snapshot, never the import, which a machine that +/// can't read the snapshot reads with the whole log. A reset drops both. +async fn the_import_stays_when_the_log_is_compacted(meta: &Meta) { + use craft_cloud_proto::live::node_doc; + let ana = user(meta, "ana").await; + let dev = device(meta, &ana).await; + let node = file(meta, &ana, "a.txt").await; + let doc = node_doc(ana.space, node); + let import = snapshot(meta, &ana, 20).await; + meta.create_live_document(ana.space, doc, "document:wordcraft", node, &import.manifest, NOW).await.unwrap(); + assert_eq!(meta.live_import_snapshot(ana.space, doc).await.unwrap(), Some(import)); + meta.append_live_update(ana.space, doc, dev, 1, b"x", NOW).await.unwrap(); + let compacted = snapshot(meta, &ana, 30).await; + assert!(meta.compact_live_doc(ana.space, doc, 1, &compacted.manifest).await.unwrap()); + assert_eq!(meta.live_snapshot(ana.space, doc).await.unwrap(), Some((1, compacted))); + assert_eq!(meta.live_import_snapshot(ana.space, doc).await.unwrap(), Some(import), "the import stays"); + meta.reset_live_doc(ana.space, doc).await.unwrap(); + assert_eq!(meta.live_import_snapshot(ana.space, doc).await.unwrap(), None); + assert_eq!(meta.live_import_snapshot(ana.space, craft_cloud_proto::DocId::generate()).await.unwrap(), None); +} + +/// A recents document whose whole log was compacted still holds updates: it counts against the account's cap, and +/// its next update isn't taken for a first one. +async fn a_compacted_log_still_holds_updates(meta: &Meta) { + use craft_cloud_proto::DocId; + let meta = meta.clone().with_live_docs_cap(1); + let ana = user(&meta, "ana").await; + let dev = device(&meta, &ana).await; + let (a, b) = (DocId::generate(), DocId::generate()); + for doc in [a, b] { + meta.create_live_doc(ana.space, doc, "recents:wordcraft", None, NOW).await.unwrap(); + } + meta.append_live_update(ana.space, a, dev, 1, b"x", NOW).await.unwrap(); + assert!(meta.compact_live_doc(ana.space, a, 1, &snapshot(&meta, &ana, 1).await.manifest).await.unwrap()); + assert_eq!(meta.live_doc(ana.space, a).await.unwrap().unwrap().above_base(), 0); + assert!(matches!(meta.append_live_update(ana.space, b, dev, 1, b"y", NOW).await, Err(MetaError::TooManyDocs { limit: 1 }))); + assert_eq!(meta.append_live_update(ana.space, a, dev, 2, b"z", NOW).await.unwrap(), 2, "the compacted one goes on"); +} + async fn concurrent_commits_on_one_file_let_exactly_one_win(meta: &Meta) { let u = user(meta, "ana").await; let dev = device(meta, &u).await; @@ -1336,6 +1650,8 @@ async fn upgrading_counts_what_each_space_already_stores() { use sqlx::Connection; let mut conn = sqlx::SqliteConnection::connect(&url).await.unwrap(); sqlx::query("ALTER TABLE spaces DROP COLUMN stored_bytes").execute(&mut conn).await.unwrap(); + // And before live documents of files. + sqlx::query("ALTER TABLE spaces DROP COLUMN live_bytes").execute(&mut conn).await.unwrap(); // And before live documents. sqlx::query("DROP TABLE live_updates").execute(&mut conn).await.unwrap(); sqlx::query("DROP TABLE live_docs").execute(&mut conn).await.unwrap(); @@ -1367,5 +1683,5 @@ async fn concurrent_opens_migrate_once_on_postgres() { use sqlx::Connection; let mut conn = sqlx::PgConnection::connect(&url).await.unwrap(); let versions: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM schema_version").fetch_one(&mut conn).await.unwrap(); - assert_eq!(versions, 4); + assert_eq!(versions, 5); } diff --git a/crates/proto/src/live.rs b/crates/proto/src/live.rs index c8ba810..c73a478 100644 --- a/crates/proto/src/live.rs +++ b/crates/proto/src/live.rs @@ -21,8 +21,31 @@ //! [`crate::api::codes`]; `doc` and `client_seq` name what it refers to. A document of another account is //! `not_found`, exactly like one that doesn't exist. //! -//! Limits ([`MAX_UPDATE_BYTES`], [`MAX_SUBSCRIPTIONS`], [`MAX_FRAME_BYTES`]) are checked when a frame is decoded and -//! again by the server; over them the answer is `too_large` or `rate_limited`, like the HTTP API. +//! From protocol [`LIVE_DOCUMENTS_PROTOCOL`] on (an agent and a server that both speak it; an older peer never sees +//! these frames): +//! +//! - **Snapshots never travel in frames**: a snapshot (a Loro snapshot of a whole document, up to +//! [`MAX_SNAPSHOT_BYTES`]) goes up and down as a file's content does, through the chunk API (cut into chunks named by +//! a manifest, with that API's body limits, hash checks and the account's storage cap), and frames name it by its +//! manifest. The agent uploads it before the frame that names it; the server checks that the manifest is the +//! account's before recording anything. +//! - **Documents linked to files**: `create {doc, node, app, snapshot}` ([`ClientFrame::CreateDocument`], on the wire +//! a `create` of [`DocKind::Document`] with the snapshot's manifest appended) makes the document of a file node by +//! import; its ID is [`node_doc`]. The server keeps the first snapshot it sees and answers `created {doc, kept}` +//! ([`ServerFrame::CreatedDocument`], a `created` with `kept` appended): an agent whose import wasn't kept drops it +//! and catches up from the start. A recents `create` and its `created` are byte for byte what they were. +//! - **Snapshots and compaction**: once a document's log above its snapshot passes [`COMPACT_UPDATES`] updates or +//! [`COMPACT_BYTES`], the server asks one device that holds every update up to `upto` for `compact {doc, upto}`; it +//! answers `snapshot {doc, upto, manifest}`, naming a Loro snapshot of exactly the updates 1..=`upto`. A `subscribe` +//! from below the snapshot (or from 0) gets `snapshot {doc, base, manifest}` first (the agent downloads it), then the +//! log from `base` on (the update at `base` too, so the agent learns its hash for the next `at`). `subscribe_log` is +//! a `subscribe` that never gets a snapshot, for an agent that couldn't read one: the log's rows below the snapshot +//! are kept. +//! - **Presence**: `presence {doc, bytes}` both ways, at most [`MAX_PRESENCE_BYTES`] and [`PRESENCE_PER_SEC`] a second +//! per document per device, relayed to the document's other subscribers and never stored. +//! +//! Limits ([`MAX_UPDATE_BYTES`], [`MAX_SUBSCRIPTIONS`], [`MAX_FRAME_BYTES`], [`MAX_SNAPSHOT_BYTES`]) are checked when a +//! frame is decoded and again by the server; over them the answer is `too_large` or `rate_limited`, like the HTTP API. //! //! **Why postcard**: frames carry opaque binary updates, so a binary encoding avoids base64 (a third more bytes per //! keystroke batch, once documents are edited live). postcard (MIT OR Apache-2.0, with `cobs`, MIT OR Apache-2.0) is compact, has no schema @@ -34,11 +57,15 @@ use serde::{Deserialize, Serialize}; -use crate::ids::{DeviceId, DocId, Epoch, SpaceId}; +use crate::ManifestId; +use crate::ids::{DeviceId, DocId, Epoch, NodeId, SpaceId}; /// Live protocol versions this build speaks. An agent and a server must share at least one. pub const LIVE_PROTOCOL_MIN: u32 = 1; -pub const LIVE_PROTOCOL_MAX: u32 = 1; +pub const LIVE_PROTOCOL_MAX: u32 = 2; +/// The protocol that adds documents linked to files, snapshots, compaction and presence. A connection on an older one +/// carries recents only. +pub const LIVE_DOCUMENTS_PROTOCOL: u32 = 2; /// Largest update, in bytes. pub const MAX_UPDATE_BYTES: usize = 1024 * 1024; @@ -46,6 +73,17 @@ pub const MAX_UPDATE_BYTES: usize = 1024 * 1024; pub const MAX_SUBSCRIPTIONS: usize = 256; /// Largest encoded frame: an update and its envelope (IDs, numbers, an error message). pub const MAX_FRAME_BYTES: usize = MAX_UPDATE_BYTES + 4096; +/// Largest snapshot of a document (an import, or a compaction), in bytes (it travels through the chunk API, never in +/// a frame). +pub const MAX_SNAPSHOT_BYTES: usize = crate::limits::MAX_MANIFEST_BYTES; +/// Largest presence message (Loro's ephemeral state: a cursor, a selection, a machine name), in bytes. +pub const MAX_PRESENCE_BYTES: usize = 4096; +/// Most presence messages one device may send a second for one document (more are dropped, never stored anyway). +pub const PRESENCE_PER_SEC: u32 = 20; +/// A document's log above its snapshot is compacted once it holds more than this many updates… +pub const COMPACT_UPDATES: u64 = 1000; +/// …or more than this many bytes. +pub const COMPACT_BYTES: u64 = 1024 * 1024; /// Longest error message in a frame, in bytes. pub const MAX_MESSAGE_BYTES: usize = 1024; /// Longest error code in a frame, in bytes. @@ -103,30 +141,63 @@ impl std::fmt::Display for AppName { pub enum DocKind { /// An app's recent files: one per app per account, linked to no node. Its ID is [`recents_doc`]. Recents { app: AppName }, + /// A file of the synced folder opened live: linked to its node (a rename or move keeps it), made by the app that + /// first opened it. Its ID is [`node_doc`]. Only from protocol [`LIVE_DOCUMENTS_PROTOCOL`] on, and only created with + /// its snapshot ([`ClientFrame::CreateDocument`]). + Document { node: NodeId, app: AppName }, } +/// What a stored document kind starts with: `document:` (its node is stored beside it). +pub const DOCUMENT_KIND_PREFIX: &str = "document:"; + impl DocKind { - /// The kind as stored on the server: `recents:`. + /// The kind as stored on the server: `recents:`, `document:` (a document's node is stored beside it). pub fn as_text(&self) -> String { match self { DocKind::Recents { app } => format!("recents:{app}"), + DocKind::Document { app, .. } => format!("{DOCUMENT_KIND_PREFIX}{app}"), } } - /// The kind from its stored form, `None` for one this build doesn't know. + /// A recents kind from its stored form, `None` for one this build doesn't know (a document needs its node too: + /// [`DocKind::parse_stored`]). pub fn parse(text: &str) -> Option { let app = text.strip_prefix("recents:")?; AppName::new(app).ok().map(|app| DocKind::Recents { app }) } - /// The ID a document of this kind must have in `space`, if the kind fixes it (recents do). + /// The kind from its stored form and the node stored beside it. + pub fn parse_stored(text: &str, node: Option) -> Option { + match (text.strip_prefix(DOCUMENT_KIND_PREFIX), node) { + (Some(app), Some(node)) => AppName::new(app).ok().map(|app| DocKind::Document { node, app }), + (Some(_), None) => None, + (None, _) => Self::parse(text), + } + } + + /// The ID a document of this kind must have in `space`: every kind fixes it (recents from the app, documents from + /// the node), so two machines that make the same one offline make one document. pub fn fixed_id(&self, space: SpaceId) -> Option { match self { DocKind::Recents { app } => Some(recents_doc(space, app)), + DocKind::Document { node, .. } => Some(node_doc(space, *node)), } } } +/// The ID of the live document of the file node `node` in `space`: the first 16 bytes of a BLAKE3 hash of both (with +/// a domain of its own), as a UUID of version 8. Two machines that import the same file make the same document. +pub fn node_doc(space: SpaceId, node: NodeId) -> DocId { + let mut hasher = blake3::Hasher::new(); + hasher.update(b"craft-cloud live document v1\0"); + hasher.update(space.as_uuid().as_bytes()); + hasher.update(node.as_uuid().as_bytes()); + let hash = hasher.finalize(); + let mut bytes = [0u8; 16]; + bytes.copy_from_slice(hash.as_bytes().get(..16).unwrap_or(&[0u8; 16])); + DocId::from_uuid(uuid::Builder::from_custom_bytes(bytes).into_uuid()) +} + /// The ID of `app`'s recents document in `space`: the first 16 bytes of a BLAKE3 hash of both, as a UUID of version 8 /// (custom). Every machine of the account computes the same one, so a recents document made offline on two machines /// is one document. @@ -176,6 +247,38 @@ pub enum ClientFrame { client_seq: u64, bytes: Vec, }, + /// The answer to `compact {doc, upto}`: the manifest of a Loro snapshot of exactly the updates 1..=`upto` of the + /// document (and the snapshot it was made from), at most [`MAX_SNAPSHOT_BYTES`], uploaded already. Never answered. + Snapshot { + doc: DocId, + upto: u64, + manifest: ManifestId, + }, + /// Ephemeral state (a cursor, a selection) for the document's other subscribers, at most [`MAX_PRESENCE_BYTES`]; + /// relayed, never stored, never answered. + Presence { + doc: DocId, + bytes: Vec, + }, + /// `subscribe` that never gets a snapshot: the whole log above `after`, its compacted rows too (for an agent that + /// couldn't read the document's snapshot). + SubscribeLog { + doc: DocId, + after: u64, + at: Option<[u8; 32]>, + }, + // Frames that are only appended to on the wire go above this line. The one below is a `create` of a document with + // its snapshot's manifest appended (see [`ClientFrame::encode`]): it has no index of its own, so it stays last. + /// Make the document of the file node `node` by import, with the snapshot `snapshot` names (a Loro snapshot, at + /// most [`MAX_SNAPSHOT_BYTES`], uploaded already) as its content if the space doesn't have it yet. Answered + /// `created {doc, kept}`. On the wire: `create {doc, kind: document {node, app}}` with the manifest appended. + #[serde(skip)] + CreateDocument { + doc: DocId, + node: NodeId, + app: AppName, + snapshot: ManifestId, + }, } /// A frame from the server to an agent. Variants are only appended (their index is on the wire). @@ -195,6 +298,21 @@ pub enum ServerFrame { Ack { doc: DocId, client_seq: u64, seq: u64 }, /// A frame was refused: `code` is one of [`crate::api::codes`] or [`PROTOCOL_MISMATCH`]. Error { code: String, message: String, doc: Option, client_seq: Option }, + /// The document's log above its snapshot is long: answer `snapshot {doc, upto, bytes}` once this device holds + /// every update up to `upto` (or don't; another device is asked later). + Compact { doc: DocId, upto: u64 }, + /// The catch-up of a `subscribe` from below the document's snapshot: the manifest of the snapshot (to download), + /// which holds the updates 1..=`base` (and, for a document made by import, its import). The log from `base` on + /// follows. + Snapshot { doc: DocId, base: u64, manifest: ManifestId }, + /// Another device's presence in a subscribed document, as it sent it. + Presence { doc: DocId, bytes: Vec }, + // Frames that are only appended to on the wire go above this line; see [`ClientFrame::CreateDocument`]. + /// `create {doc, node, app, snapshot}` done: `kept` says whether this snapshot is the document's (else another + /// machine's import was there first, and this one is to be dropped). On the wire: `created {doc}` with `kept` + /// appended. + #[serde(skip)] + CreatedDocument { doc: DocId, kept: bool }, } /// A frame that can't be encoded or decoded. @@ -208,34 +326,69 @@ pub enum FrameError { Invalid(&'static str), } -fn encode(frame: &T) -> Result, FrameError> { - let bytes = postcard::to_stdvec(frame).map_err(|e| FrameError::Malformed(e.to_string()))?; - if bytes.len() > MAX_FRAME_BYTES { +fn malformed(e: postcard::Error) -> FrameError { + FrameError::Malformed(e.to_string()) +} + +/// Encodes `frame`, and `trailer` after it (a field appended to an older frame), within the frame's limit. +fn encode(frame: &T, trailer: Option<&A>, limit: usize) -> Result, FrameError> { + let mut bytes = postcard::to_stdvec(frame).map_err(malformed)?; + if let Some(trailer) = trailer { + bytes.extend(postcard::to_stdvec(trailer).map_err(malformed)?); + } + if bytes.len() > limit { return Err(FrameError::TooLarge(bytes.len())); } Ok(bytes) } -fn decode<'a, T: Deserialize<'a>>(bytes: &'a [u8]) -> Result { +/// Decodes the start of a frame, returning what follows it (a trailer, or nothing). +fn decode_head<'a, T: Deserialize<'a>>(bytes: &'a [u8]) -> Result<(T, &'a [u8]), FrameError> { if bytes.len() > MAX_FRAME_BYTES { return Err(FrameError::TooLarge(bytes.len())); } - let (frame, rest) = postcard::take_from_bytes(bytes).map_err(|e| FrameError::Malformed(e.to_string()))?; - if !rest.is_empty() { - return Err(FrameError::Malformed(format!("{} bytes after the frame", rest.len()))); + postcard::take_from_bytes(bytes).map_err(malformed) +} + +/// A trailer that must be all of `rest`. +fn decode_trailer<'a, T: Deserialize<'a>>(rest: &'a [u8]) -> Result { + let (value, after) = postcard::take_from_bytes(rest).map_err(malformed)?; + if !after.is_empty() { + return Err(FrameError::Malformed(format!("{} bytes after the frame", after.len()))); } - Ok(frame) + Ok(value) +} + +fn presence_ok(bytes: &[u8]) -> Result<(), FrameError> { + if bytes.is_empty() || bytes.len() > MAX_PRESENCE_BYTES { Err(FrameError::Invalid("a presence message is 1 byte to 4 KiB")) } else { Ok(()) } } impl ClientFrame { pub fn encode(&self) -> Result, FrameError> { self.check()?; - encode(self) + match self { + ClientFrame::CreateDocument { doc, node, app, snapshot } => { + let head = ClientFrame::Create { doc: *doc, kind: DocKind::Document { node: *node, app: app.clone() } }; + encode(&head, Some(snapshot), MAX_FRAME_BYTES) + } + frame => encode::<_, ()>(frame, None, MAX_FRAME_BYTES), + } } /// Decodes and checks a frame from the network (hostile: every length is checked). pub fn decode(bytes: &[u8]) -> Result { - let frame: Self = decode(bytes)?; + let (head, rest): (Self, &[u8]) = decode_head(bytes)?; + let frame = match head { + ClientFrame::Create { doc, kind: DocKind::Document { node, app } } => { + ClientFrame::CreateDocument { doc, node, app, snapshot: decode_trailer(rest)? } + } + head => { + if !rest.is_empty() { + return Err(FrameError::Malformed(format!("{} bytes after the frame", rest.len()))); + } + head + } + }; frame.check()?; Ok(frame) } @@ -244,6 +397,8 @@ impl ClientFrame { match self { ClientFrame::Update { bytes, .. } if bytes.len() > MAX_UPDATE_BYTES => Err(FrameError::Invalid("an update is at most 1 MiB")), ClientFrame::Update { bytes, .. } if bytes.is_empty() => Err(FrameError::Invalid("an update can't be empty")), + ClientFrame::Create { kind: DocKind::Document { .. }, .. } => Err(FrameError::Invalid("a document is created with its snapshot")), + ClientFrame::Presence { bytes, .. } => presence_ok(bytes), _ => Ok(()), } } @@ -252,12 +407,24 @@ impl ClientFrame { impl ServerFrame { pub fn encode(&self) -> Result, FrameError> { self.check()?; - encode(self) + match self { + ServerFrame::CreatedDocument { doc, kept } => encode(&ServerFrame::Created { doc: *doc }, Some(kept), MAX_FRAME_BYTES), + frame => encode::<_, ()>(frame, None, MAX_FRAME_BYTES), + } } /// Decodes and checks a frame from the network (hostile: every length is checked). pub fn decode(bytes: &[u8]) -> Result { - let frame: Self = decode(bytes)?; + let (head, rest): (Self, &[u8]) = decode_head(bytes)?; + let frame = match head { + ServerFrame::Created { doc } if !rest.is_empty() => ServerFrame::CreatedDocument { doc, kept: decode_trailer(rest)? }, + head => { + if !rest.is_empty() { + return Err(FrameError::Malformed(format!("{} bytes after the frame", rest.len()))); + } + head + } + }; frame.check()?; Ok(frame) } @@ -270,6 +437,7 @@ impl ServerFrame { ServerFrame::Error { code, message, .. } if code.len() > MAX_CODE_BYTES || message.len() > MAX_MESSAGE_BYTES => { Err(FrameError::Invalid("an error's code or message is too long")) } + ServerFrame::Presence { bytes, .. } => presence_ok(bytes), _ => Ok(()), } } @@ -298,6 +466,14 @@ impl std::fmt::Debug for ClientFrame { ClientFrame::Subscribe { doc, after, at } => write!(f, "Subscribe({doc}, after {after}, at {})", if at.is_some() { "hash" } else { "-" }), ClientFrame::Unsubscribe { doc } => write!(f, "Unsubscribe({doc})"), ClientFrame::Update { doc, client_seq, bytes } => write!(f, "Update({doc}, client_seq {client_seq}, {} bytes)", bytes.len()), + ClientFrame::Snapshot { doc, upto, manifest } => write!(f, "Snapshot({doc}, upto {upto}, manifest {manifest})"), + ClientFrame::Presence { doc, bytes } => write!(f, "Presence({doc}, {} bytes)", bytes.len()), + ClientFrame::SubscribeLog { doc, after, at } => { + write!(f, "SubscribeLog({doc}, after {after}, at {})", if at.is_some() { "hash" } else { "-" }) + } + ClientFrame::CreateDocument { doc, node, app, snapshot } => { + write!(f, "CreateDocument({doc}, node {node}, {app}, manifest {snapshot})") + } } } } @@ -311,6 +487,10 @@ impl std::fmt::Debug for ServerFrame { ServerFrame::Subscribed { doc, head } => write!(f, "Subscribed({doc}, head {head})"), ServerFrame::Ack { doc, client_seq, seq } => write!(f, "Ack({doc}, client_seq {client_seq}, seq {seq})"), ServerFrame::Error { code, message, doc, client_seq } => write!(f, "Error({code}: {message}; {doc:?}, {client_seq:?})"), + ServerFrame::Compact { doc, upto } => write!(f, "Compact({doc}, upto {upto})"), + ServerFrame::Snapshot { doc, base, manifest } => write!(f, "Snapshot({doc}, base {base}, manifest {manifest})"), + ServerFrame::Presence { doc, bytes } => write!(f, "Presence({doc}, {} bytes)", bytes.len()), + ServerFrame::CreatedDocument { doc, kept } => write!(f, "CreatedDocument({doc}, kept {kept})"), } } } @@ -445,8 +625,176 @@ mod tests { fn versions_are_negotiated_to_the_highest_shared() { assert_eq!(shared_version(1, 1), Some(1)); assert_eq!(shared_version(1, 5), Some(LIVE_PROTOCOL_MAX)); - assert_eq!(shared_version(2, 5), None); + assert_eq!(shared_version(2, 5), Some(LIVE_DOCUMENTS_PROTOCOL)); + assert_eq!(shared_version(LIVE_PROTOCOL_MAX + 1, LIVE_PROTOCOL_MAX + 4), None); assert_eq!(shared_version(0, 0), None); assert_eq!(shared_version(3, 1), None, "an empty range"); + // An agent from before documents (1-1) and a server that has them agree on 1, and so do a new agent and an old + // server (its range is 1-1): recents only. + assert_eq!(shared_version(1, 1), Some(1)); + } + + #[test] + fn a_document_has_one_id_per_space_and_node() { + let (space, other) = (SpaceId::generate(), SpaceId::generate()); + let (node, node2) = (NodeId::generate(), NodeId::generate()); + let id = node_doc(space, node); + assert_eq!(id, node_doc(space, node), "two machines importing the same file make one document"); + assert_ne!(id, node_doc(space, node2)); + assert_ne!(id, node_doc(other, node)); + assert_eq!(id.as_uuid().get_version_num(), 8); + let kind = DocKind::Document { node, app: app("wordcraft") }; + assert_eq!(kind.fixed_id(space), Some(id)); + assert_eq!(kind.as_text(), "document:wordcraft"); + assert_eq!(DocKind::parse_stored("document:wordcraft", Some(node)), Some(kind)); + assert_eq!(DocKind::parse_stored("document:wordcraft", None), None, "a document's node is stored beside it"); + assert_eq!(DocKind::parse_stored("recents:wordcraft", None), Some(DocKind::Recents { app: app("wordcraft") })); + } + + fn doc7() -> DocId { + DocId::from_uuid(uuid::Uuid::from_bytes([7; 16])) + } + + /// The frames of documents round-trip, and a document's `create` and `created` are an older `create` and + /// `created` with a field appended. + #[test] + fn document_frames_round_trip() { + let (doc, node) = (DocId::generate(), NodeId::generate()); + let manifest = ManifestId::from_bytes([0xab; 32]); + let client = [ + ClientFrame::CreateDocument { doc, node, app: app("wordcraft"), snapshot: manifest }, + ClientFrame::Snapshot { doc, upto: 1001, manifest }, + ClientFrame::Presence { doc, bytes: vec![2; 40] }, + ClientFrame::SubscribeLog { doc, after: 3, at: Some(update_hash(b"x")) }, + ]; + for frame in client { + assert_eq!(ClientFrame::decode(&frame.encode().unwrap()).unwrap(), frame); + } + let server = [ + ServerFrame::CreatedDocument { doc, kept: true }, + ServerFrame::CreatedDocument { doc, kept: false }, + ServerFrame::Compact { doc, upto: 1001 }, + ServerFrame::Snapshot { doc, base: 1001, manifest }, + ServerFrame::Presence { doc, bytes: vec![4; MAX_PRESENCE_BYTES] }, + ]; + for frame in server { + assert_eq!(ServerFrame::decode(&frame.encode().unwrap()).unwrap(), frame); + } + // `created {doc, kept}` is `created {doc}` and one byte. + let kept = ServerFrame::CreatedDocument { doc: doc7(), kept: true }.encode().unwrap(); + let mut want = ServerFrame::Created { doc: doc7() }.encode().unwrap(); + want.push(1); + assert_eq!(kept, want); + // A document's `create` is a `create` of kind 1 (`document {node, app}`), the snapshot's manifest (in hex) after + // it. + let node = NodeId::from_uuid(uuid::Uuid::from_bytes([8; 16])); + let made = ClientFrame::CreateDocument { doc: doc7(), node, app: app("ab"), snapshot: manifest }.encode().unwrap(); + let mut want = vec![1, 16]; + want.extend([7; 16]); + want.extend([1, 16]); + want.extend([8; 16]); + want.extend([2, b'a', b'b', 64]); + want.extend("ab".repeat(32).bytes()); + assert_eq!(made, want); + } + + /// What an agent and a server from before documents send (their bytes written out here) still decodes, to the same + /// frames: nothing they know changed on the wire. + #[test] + fn frames_in_the_older_encoding_still_decode() { + let doc = doc7(); + let id = |bytes: &mut Vec| bytes.extend([16].iter().chain([7u8; 16].iter())); + let mut create = vec![1]; + id(&mut create); + create.extend([0, 9]); + create.extend(b"wordcraft"); + assert_eq!(ClientFrame::decode(&create).unwrap(), ClientFrame::Create { doc, kind: DocKind::Recents { app: app("wordcraft") } }); + let mut subscribe = vec![2]; + id(&mut subscribe); + subscribe.extend([7, 0]); + assert_eq!(ClientFrame::decode(&subscribe).unwrap(), ClientFrame::Subscribe { doc, after: 7, at: None }); + let mut unsubscribe = vec![3]; + id(&mut unsubscribe); + assert_eq!(ClientFrame::decode(&unsubscribe).unwrap(), ClientFrame::Unsubscribe { doc }); + let mut created = vec![1]; + id(&mut created); + assert_eq!(ServerFrame::decode(&created).unwrap(), ServerFrame::Created { doc }); + let mut ack = vec![4]; + id(&mut ack); + ack.extend([3, 10]); + assert_eq!(ServerFrame::decode(&ack).unwrap(), ServerFrame::Ack { doc, client_seq: 3, seq: 10 }); + let mut subscribed = vec![3]; + id(&mut subscribed); + subscribed.push(9); + assert_eq!(ServerFrame::decode(&subscribed).unwrap(), ServerFrame::Subscribed { doc, head: 9 }); + let device = DeviceId::from_uuid(uuid::Uuid::from_bytes([2; 16])); + let mut update = vec![2]; + id(&mut update); + update.extend([9, 2, 0xaa, 0xbb, 16]); + update.extend([2; 16]); + assert_eq!(ServerFrame::decode(&update).unwrap(), ServerFrame::Update { doc, seq: 9, bytes: vec![0xaa, 0xbb], device }); + let mut error = vec![5, 9]; + error.extend(b"not_found"); + error.extend([2, b'n', b'o', 1]); + id(&mut error); + error.push(0); + assert_eq!(ServerFrame::decode(&error).unwrap(), ServerFrame::error("not_found", "no", Some(doc), None)); + // And the frames an older peer sends encode the same way here. + assert_eq!(ClientFrame::Create { doc, kind: DocKind::Recents { app: app("wordcraft") } }.encode().unwrap(), create); + assert_eq!(ServerFrame::Created { doc }.encode().unwrap(), created); + } + + /// Every new limit at its edge. + #[test] + fn document_frames_are_checked_at_their_limits() { + let doc = DocId::generate(); + let node = NodeId::generate(); + let presence = |n| ClientFrame::Presence { doc, bytes: vec![1; n] }; + assert!(presence(MAX_PRESENCE_BYTES).encode().is_ok()); + assert!(presence(MAX_PRESENCE_BYTES + 1).encode().is_err()); + assert!(presence(0).encode().is_err()); + let raw = postcard::to_stdvec(&presence(MAX_PRESENCE_BYTES + 1)).unwrap(); + assert!(matches!(ClientFrame::decode(&raw), Err(FrameError::Invalid(_)))); + let create = |m: [u8; 32]| ClientFrame::CreateDocument { doc, node, app: app("wordcraft"), snapshot: ManifestId::from_bytes(m) }; + // A manifest that isn't one (not 64 lowercase hex digits) doesn't decode. + let mut bad = postcard::to_stdvec(&ClientFrame::Create { doc, kind: DocKind::Document { node, app: app("wordcraft") } }).unwrap(); + bad.extend(postcard::to_stdvec(&"AB".repeat(32)).unwrap()); + assert!(ClientFrame::decode(&bad).is_err()); + // A document `create` without its snapshot, or with bytes after it, is refused; so is one of a recents kind with + // a snapshot after it. + let bare = postcard::to_stdvec(&ClientFrame::Create { doc, kind: DocKind::Document { node, app: app("x") } }).unwrap(); + assert!(ClientFrame::decode(&bare).is_err()); + assert!(ClientFrame::Create { doc, kind: DocKind::Document { node, app: app("x") } }.encode().is_err()); + let mut extra = create([1; 32]).encode().unwrap(); + assert!(ClientFrame::decode(&extra).is_ok()); + extra.push(0); + assert!(ClientFrame::decode(&extra).is_err()); + let mut recents = ClientFrame::Create { doc, kind: DocKind::Recents { app: app("x") } }.encode().unwrap(); + recents.extend(postcard::to_stdvec(&vec![1u8; 3]).unwrap()); + assert!(ClientFrame::decode(&recents).is_err()); + // Every frame is at most the small limit, either way (no frame carries a snapshot). + let mut padded = postcard::to_stdvec(&ClientFrame::Unsubscribe { doc }).unwrap(); + padded.resize(MAX_FRAME_BYTES + 1, 0); + assert!(matches!(ClientFrame::decode(&padded), Err(FrameError::TooLarge(_)))); + assert!(matches!(ServerFrame::decode(&padded), Err(FrameError::TooLarge(_)))); + // `created` with a trailer that isn't a flag. + let mut created = ServerFrame::Created { doc }.encode().unwrap(); + created.push(2); + assert!(ServerFrame::decode(&created).is_err()); + let p = ServerFrame::Presence { doc, bytes: vec![1; MAX_PRESENCE_BYTES + 1] }; + assert!(p.encode().is_err()); + } + + #[test] + fn debug_output_never_shows_document_bytes() { + let doc = DocId::generate(); + let frames = [ + format!("{:?}", ClientFrame::Update { doc, client_seq: 1, bytes: b"secret".to_vec() }), + format!("{:?}", ClientFrame::Presence { doc, bytes: b"secret".to_vec() }), + format!("{:?}", ServerFrame::Presence { doc, bytes: b"secret".to_vec() }), + ]; + for shown in frames { + assert!(!shown.contains("secret") && shown.contains("6 bytes"), "{shown}"); + } } } diff --git a/crates/proto/src/local.rs b/crates/proto/src/local.rs index 1982c7f..823a039 100644 --- a/crates/proto/src/local.rs +++ b/crates/proto/src/local.rs @@ -14,7 +14,8 @@ //! `DELETE /v1/pairings/{id}` → 204; `GET /v1/devices` → a list of [`DeviceInfo`](crate::api::DeviceInfo), //! `DELETE /v1/devices/{id}` → the list after; `POST /v1/pause`, `/v1/resume` and `/v1/quit` → [`Status`]; //! `POST /v1/autostart` ([`AutostartRequest`]) → [`Status`]; `GET /v1/recents`, upgraded to a WebSocket of JSON -//! text messages: [`RecentsRequest`] in, [`RecentsEvent`] out. Errors are the HTTP API's +//! text messages: [`RecentsRequest`] in, [`RecentsEvent`] out; `GET /v1/docs`, upgraded to a WebSocket of live +//! documents ([`docs`]: [`DocsRequest`] and binary frames in, [`DocsEvent`] and binary frames out). Errors are the HTTP API's //! [`ErrorBody`](crate::api::ErrorBody) with its codes and [`codes`]. //! - **Versions**: [`Status::api`] is the local API version the agent speaks; a client refuses one outside //! [`LOCAL_API_MIN`]..=[`LOCAL_API_MAX`] when it connects. @@ -90,6 +91,8 @@ pub mod paths { pub const LOGIN: &str = "/v1/login"; pub const LOGOUT: &str = "/v1/logout"; pub const RECENTS: &str = "/v1/recents"; + /// `GET`, upgraded to a WebSocket: live documents ([`super::docs`]). + pub const DOCS: &str = "/v1/docs"; /// `POST`: join with a join string ([`super::JoinRequest`]) → [`super::Status`]. pub const JOIN: &str = "/v1/join"; /// `POST`: a new add-a-computer code → [`super::PairingInfo`]; `GET`/`DELETE` `/v1/pairings/{id}`: its state, or @@ -365,6 +368,136 @@ pub enum RecentsEvent { }, } +/// The live documents WebSocket, `GET /v1/docs`: an app opens files of the synced folder as live documents. +/// +/// Control messages are JSON text frames ([`DocsRequest`] in, [`DocsEvent`] out); document bytes are binary frames of +/// `[u8 kind][u64 id, big-endian][bytes]` ([`encode_binary`], [`decode_binary`]), so nothing is base64. `id` is the +/// number the app gave the document in its `open`; one connection may open many documents. +/// +/// - `open {id, path, app}` → `opened {id, doc, state}` followed by the document's state (a binary [`STATE`] frame: a +/// Loro snapshot, then any updates still waiting for their history as [`UPDATE`] frames), or `needs_import {id}` +/// (the file has never been live: the app sends its content as a binary [`IMPORT`] frame, a Loro snapshot, and then +/// gets `opened`), or `not_live {id, reason}` (the app saves the file as it always does). +/// - A binary [`UPDATE`] from the app is its own change; it is answered `ack {id, seq}` once it is on the agent's disk +/// (`seq` counts the app's updates of that document on this connection, from 1), or `error {id, seq, …}` when the +/// agent refused it (it doesn't decode, or is over 1 MiB). +/// - From the agent: remote changes as binary [`UPDATE`] frames and other machines' presence as [`PRESENCE`] frames; +/// `status {doc, online, waiting, deleted}` when that changes. +/// - A binary [`PRESENCE`] from the app is relayed to the document's other machines and apps (at most 4 KiB). +/// - `close {doc}`: the app is done with it (the agent keeps it). +/// +/// An app that doesn't read for 10 s is dropped; its documents stay with the agent. +pub mod docs { + /// Binary frame kinds. + pub const STATE: u8 = 1; + pub const UPDATE: u8 = 2; + pub const PRESENCE: u8 = 3; + pub const IMPORT: u8 = 4; + /// The header of a binary frame: kind and id. + pub const HEADER_BYTES: usize = 9; + /// Largest binary frame (a state or an import: a snapshot), either way. + pub const MAX_BINARY_BYTES: usize = crate::live::MAX_SNAPSHOT_BYTES + HEADER_BYTES; + + /// A binary frame. + pub fn encode_binary(kind: u8, id: u64, bytes: &[u8]) -> Vec { + let mut out = Vec::with_capacity(bytes.len().saturating_add(HEADER_BYTES)); + out.push(kind); + out.extend_from_slice(&id.to_be_bytes()); + out.extend_from_slice(bytes); + out + } + + /// A binary frame's kind, id and bytes; `None` for one too short to have a header. + pub fn decode_binary(frame: &[u8]) -> Option<(u8, u64, &[u8])> { + let (&kind, rest) = frame.split_first()?; + let id = u64::from_be_bytes(rest.get(..8)?.try_into().ok()?); + Some((kind, id, rest.get(8..)?)) + } +} + +/// A control message from an app on the docs WebSocket ([`docs`]). +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case", tag = "type")] +pub enum DocsRequest { + /// Open the file at `path` (relative to the synced folder, `/`-separated, or absolute inside it) as a live document + /// for `app` (a lowercase app name), under the number `id` on this connection. + Open { id: u64, path: String, app: String }, + /// The app is done with the document. + Close { doc: crate::ids::DocId }, +} + +/// How a document stands, as `opened` and `status` say it. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct DocState { + /// The agent reaches the server and follows the document: edits go out as they are made. + pub online: bool, + /// Changes made on this machine the server hasn't acked yet. + pub waiting: u64, + /// The size of the binary state that follows `opened` (0 in `status`). + #[serde(default)] + pub bytes: u64, +} + +/// Why a file opens as a plain file rather than a live document. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum NotLiveReason { + /// It has never been live and the agent can't reach a server that has live documents. + Offline, + /// The file isn't uploaded yet (the agent started a sync pass for it; open it again later). + NotSynced, + /// The agent isn't signed in. + NotSignedIn, + /// The file, or its content as a document, is over the limit for live documents. + TooLarge, + /// The path isn't a file inside the synced folder. + NotInFolder, + /// The server's document was reset and its machines haven't sent it back yet: the file is what to open (it is + /// live again once one of them has). + Resetting, +} + +/// A control message from the agent on the docs WebSocket ([`docs`]). +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case", tag = "type")] +pub enum DocsEvent { + /// The document is live: its state follows as a binary frame of `state.bytes` bytes. + Opened { id: u64, doc: crate::ids::DocId, state: DocState }, + /// The file has never been live: send its content as a binary import frame. + NeedsImport { id: u64 }, + /// The file opens as a plain file. + NotLive { + id: u64, + reason: NotLiveReason, + #[serde(default)] + message: Option, + }, + /// The app's update number `seq` of document `id` is on the agent's disk. + Ack { id: u64, seq: u64 }, + /// How an open document stands, when that changes. `deleted`: its file was deleted on this or another machine + /// (the document still works; a later open of the same path is a new file and a new document). `stopped`: the + /// agent stopped following it until it restarts (Loro failed on it twice); changes made meanwhile are kept. + Status { + doc: crate::ids::DocId, + online: bool, + waiting: u64, + #[serde(default)] + deleted: bool, + #[serde(default)] + stopped: bool, + }, + /// A request or a binary frame was refused (`id` and `seq` name it when they can). `code` is one of + /// [`crate::api::codes`] or [`codes`]. + Error { + #[serde(default)] + id: Option, + #[serde(default)] + seq: Option, + code: String, + message: String, + }, +} + /// `path` as a recent file's path: `/`-separated names relative to the synced folder, each a valid [`Name`] /// (NFC-normalised; no `.`, `..`, empty or reserved name, no drive letter), at most [`MAX_RECENT_PATH_BYTES`] and /// [`MAX_TREE_DEPTH`] names. `None` for anything else. @@ -491,6 +624,23 @@ mod tests { assert!(!login.adopt && login.folder.is_none(), "missing optional fields default; unknown ones are ignored"); } + #[test] + fn docs_messages_have_a_plain_form() { + let frame = docs::encode_binary(docs::UPDATE, 0x0102_0304_0506_0708, b"abc"); + assert_eq!(frame, vec![2, 1, 2, 3, 4, 5, 6, 7, 8, b'a', b'b', b'c']); + assert_eq!(docs::decode_binary(&frame), Some((docs::UPDATE, 0x0102_0304_0506_0708, &b"abc"[..]))); + assert_eq!(docs::decode_binary(&frame[..9]), Some((docs::UPDATE, 0x0102_0304_0506_0708, &b""[..]))); + assert_eq!(docs::decode_binary(&frame[..8]), None, "too short for a header"); + assert_eq!(docs::decode_binary(&[]), None); + let open: DocsRequest = serde_json::from_value(serde_json::json!({ "type": "open", "id": 1, "path": "a.txt", "app": "wordcraft" })).unwrap(); + assert_eq!(open, DocsRequest::Open { id: 1, path: "a.txt".into(), app: "wordcraft".into() }); + let not_live = serde_json::to_value(DocsEvent::NotLive { id: 1, reason: NotLiveReason::NotSynced, message: None }).unwrap(); + assert_eq!(not_live, serde_json::json!({ "type": "not_live", "id": 1, "reason": "not_synced", "message": null })); + let status: DocsEvent = + serde_json::from_value(serde_json::json!({ "type": "status", "doc": uuid::Uuid::nil(), "online": true, "waiting": 2 })).unwrap(); + assert!(matches!(status, DocsEvent::Status { online: true, waiting: 2, deleted: false, .. })); + } + #[test] fn recent_paths_are_relative_and_clean() { assert_eq!(recent_path("Docs/report.docx").as_deref(), Some("Docs/report.docx")); diff --git a/docs/agent.md b/docs/agent.md index 1eed58e..18e2a39 100644 --- a/docs/agent.md +++ b/docs/agent.md @@ -73,3 +73,15 @@ printf '%s\n' 'your password' | ./target/release/craft-cloud-agent login https:/ An entry added offline is kept and reaches the other machines once the server is back. Each list keeps the 100 files opened last. +- A file of the synced folder as a live plain-text document (through the running agent, as an app opens one; a + test tool for live documents): + + ```sh + ./target/release/craft-cloud-agent doc Notes/todo.txt # its text + ./target/release/craft-cloud-agent doc Notes/todo.txt --append "eggs" # add to it, on every machine + ./target/release/craft-cloud-agent doc Notes/todo.txt --watch # again after every change + ``` + + The first machine to open a file makes it a live document from the file's content (UTF-8 text only); the others + join it. Changes go to every machine that has it open as they are made, and edits made offline merge when the + machine is back. `doc` never writes the file: what it shows is the live document. diff --git a/docs/design/live.md b/docs/design/live.md index be79ae6..31b5251 100644 --- a/docs/design/live.md +++ b/docs/design/live.md @@ -41,8 +41,35 @@ per app connection for live sessions. `run` serves it while signed out too. Call `run_state` `paused` and the reason; every call but `logout` (which forgets the account here) and `quit` is answered `needs_you` until then; - `GET /v1/recents` (WebSocket: subscribe per app, add an entry `{path, title?, saved}`, receive the list on change); -- later: `open` (a path → its live document, created by import when absent; returns the state and then a stream - of remote updates and presence), `update` (the app's local changes), `presence`, `close`. +- `GET /v1/docs` (WebSocket: live documents). Control messages are JSON text frames, document bytes binary frames of + `[u8 kind][u64 id, big-endian][bytes]` (kinds: 1 state, 2 update, 3 presence, 4 import), so nothing is base64; `id` + is the number the app gave the document in its `open`, and one connection may open many documents: + - `open {id, path, app}` (`path` relative to the synced folder, or absolute inside it; `app` a lowercase app name) + → `opened {id, doc, state: {online, waiting, bytes}}` followed by the binary state (a Loro snapshot) and then, as + binary updates, the updates still waiting for their history and this machine's own not yet acked; or + `needs_import {id}` (the file has never been live: the app sends a binary import, a Loro snapshot, and then gets + `opened`; the state is the import the server kept, which may be another machine's); or `not_live {id, reason, + message?}`, with `reason` one of `offline` (never live here and no server with live documents to ask), + `not_synced` (the file isn't uploaded yet: the agent starts a sync pass for it and waits up to 5 s for its + node), `not_signed_in`, `too_large` (the file, or its import, is over the snapshot limit), `not_in_folder` (no + such file in the synced folder) and `resetting` (the admin reset the server's document and no machine has sent it + back yet: an app here would get an empty document). The app then opens and saves the file as always. A document + that has been live on this machine opens offline from the agent's copy; + - a binary update from the app (its own change, at most 1 MiB) → `ack {id, seq}` once it is on the agent's disk + (`seq` counts the app's updates of that document on this connection, from 1), or `error {id, seq, code, + message}` when the agent refused it (`invalid`: Loro doesn't take it; `too_large`); the agent's copy is as it was; + - from the agent: remote changes as binary updates, other machines' and apps' presence as binary presence (as + sent), and `status {doc, online, waiting, deleted, stopped}` when that changes (`deleted`: the file was deleted, + here or on another machine; the document still works, and a later open of the path is a new node and a new + document; a rename or move keeps the document, which belongs to the node; `stopped`: Loro failed on the + document twice in this run, and the agent follows it again only once it restarts; changes made meanwhile are + kept); + - a binary presence from the app (at most 4 KiB, at most 20 a second per document) goes to the document's other + machines and apps; `close {doc}`: the app is done with it (the agent keeps it); + - a binary frame without a header, or of a kind an app doesn't send, ends the connection; a text message that + isn't a request is answered `error`. An app that doesn't take a message within 10 s is dropped (so is one whose + messages queue up past 1,024 in the agent); its documents stay with the agent. An app that goes away mid-import + leaves nothing on the server (the import goes up only once whole); the next app to open the file is asked. Errors use stable codes (`forbidden`, `wrong_password`, `already_signed_in`, `not_signed_in`, `offline`, `needs_you`, and for joining `invalid_code`, `password_required`, `weak_password`, `untrusted_certificate`, @@ -53,21 +80,44 @@ disconnected, and the crate reconnects. token like every call (at most 4 connections per device). Binary frames encoded with postcard (compact, no base64 for opaque updates; variants are only ever appended, anything new is negotiated at `hello`), types in `craft-cloud-proto`: -- `hello {min, max}` → `welcome {protocol, epoch}` (the space's epoch), or `error protocol_mismatch`. -- `create {doc, kind}` → `created {doc}`; idempotent. For now the only kind is `recents:`, whose ID is derived from - the account and the app, so two machines that create it offline make the same one. Later: `create {doc, node, - app, snapshot}` for a document made by import. +- `hello {min, max}` → `welcome {protocol, epoch}` (the space's epoch), or `error protocol_mismatch`. Protocol 2 adds + documents of files, snapshots, compaction and presence; a connection on protocol 1 (an agent from before them, or + a new agent and an older server) carries recents only, and is sent a compacted log whole. A new agent with an older + server keeps recents and says live documents need a newer server; a document live on it already keeps its changes + on the machine until then. +- `create {doc, kind}` → `created {doc}`; idempotent. Kinds: `recents:`, whose ID is derived from the account and + the app, so two machines that create it offline make the same one; and `document {node, app}`, a file of the synced + folder, whose ID is `node_doc(space, node)` (BLAKE3 of both with a domain of its own, as a UUIDv8), so two machines + that import the same file make one document. A document of a file is only made by import: `create {doc, node, app, + snapshot}` (the snapshot's manifest appended to the frame) → `created {doc, kept}` (`kept` appended): the server + checks the ID is the node's, the node a file of the space and the manifest one the account uploaded, and keeps the + first one it sees; an agent whose import wasn't kept drops it and catches up from 0, and its app is told the + document is live only once that is settled, so an app never edits a losing import. A recents `create` and `created` + are byte for byte what they were. A joined document the server lost (a restore from before it was made) is made + again from the agent's copy. +- **Snapshots never travel in frames.** A snapshot (a Loro snapshot of a whole document, at most 36 MiB) goes up as a + file's content does: cut into chunks named by a manifest, uploaded through the chunk API (with its body limits, + hash checks and the account's storage cap, `stored_bytes_per_user`) before the frame that names its manifest; the + agent's other frames for that document wait behind the upload. Down, a frame names the manifest and the agent + downloads it the same way, every chunk checked against its ID and the whole against the manifest; the catch-up's + updates wait for it. One the agent can't upload isn't named (an import's open then answers `offline`); one it can't + download, it follows the whole log instead. So no message on the live connection is over an update and its envelope + (about 1 MiB), on any protocol. - `subscribe {doc, after, at}` → the log above `after`, then `subscribed {doc, head}`, then live updates. **Catch-up is by sequence number**, not by Loro version, so the server never parses Loro: per-document numbers have no gaps, `after` is the highest number up to which the agent holds every update (an own ack that arrives before the updates below it isn't counted until they do, and its update is sent again on the next connection; the server's retry rule - makes that free), and `at` is the BLAKE3 hash of the update at `after`. + makes that free), and `at` is the BLAKE3 hash of the update at `after`. Whether one of the agent's own updates is + stored is known by version, not by acks: an unacked update all of whose operations are in the rows (and the + snapshot) the agent has taken is stored, whatever its ack does; an ack that came for it is kept until the rows + below it are there. - A restored or reset server is recognised by the epoch `welcome` names (checked again at every ping; the space's epoch hashed with its live generation), by an `at` that doesn't match its log (`error {resync_required, doc}`), or by a head behind the agent's, even when the log has grown back to the same length. The agent then catches up from 0 and sends only the operations the log lacks: parts of at most 64 KiB, built from the missing changes in Lamport order (a change too big for one part goes in pieces, and a part never spans a gap), so each part applies once the - earlier ones are in; each part is worked out when the last is acked. Its own operations go first; others' go only + earlier ones are in; each part is worked out when the last is acked. Its own operations (its apps' among them) go + without a wait, and a change an app makes meanwhile goes at once as a plain update; others' go only after a random wait of 1–8 ticks, after which what the log lacks is worked out again; any update that comes in meanwhile restarts the wait (9–16 ticks), so pushers queue behind whoever is pushing. Duplicates arise only when two waits run out before either's part arrives (at the start, or when a pusher goes quiet): expected at most @@ -75,25 +125,65 @@ for opaque updates; variants are only ever appended, anything new is negotiated `serve`. - A second `subscribe` to a document whose catch-up is still running is refused (`invalid`); catch-up is paged by bytes. +- **Snapshots and compaction**: when a document's log above its snapshot passes 1,000 updates or 1 MiB, the server + sends `compact {doc, upto}` to one subscribed device that holds every update up to `upto` (the one that sent the + update that crossed, or one that just caught up); one request per document at a time. That agent answers `snapshot + {doc, upto, manifest}`, naming a Loro snapshot of exactly the updates 1..=`upto` (exported at the version those + updates make as the agent took them, never including later ones: the agent tracks what the log holds, as a version + per update), once it holds them and nothing waits for its history; otherwise it doesn't answer. An agent that + dropped a row at or below `upto` (Loro failed on it, or it never applies) never answers, and says so. The server + takes only the snapshot it asked that session for, and only a manifest the account uploaded: it records `base = + upto` and the manifest in one metadata transaction, only past the current base (a second snapshot for an older or + equal `upto` is ignored). The rows at or below the base are kept (compacted: no longer counted against the 16 MiB + cap, and not sent to a protocol 2 agent that reads the snapshot) until the retention rules come; the retention + rules must count `snapshot` and `import_snapshot` as references to their manifests. A device that doesn't answer + within 30 s is skipped for 10 minutes and another is asked when it next appends or catches up. Recents documents + compact the same way. +- **Catch-up with a snapshot**: a `subscribe {after}` with `after < base` (or `after` 0, for a document made by + import) is sent `snapshot {doc, base, manifest}`, then the log from `base` on (the row at `base` too, so the agent + learns its hash for the next `at`). The `at` check for `after < base` is against the stored hash of row `after`; + an `after` the server can't check means `resync_required`. An agent that can't read the snapshot, or finds updates + waiting after it for history the snapshot should have held, follows the whole log instead: `subscribe_log {doc, + after, at}` is a `subscribe` that never gets a snapshot, but from 0 gets the document's import first (the log + doesn't hold it; the server keeps it beside the snapshot). - `update {doc, client_seq, bytes}` → `ack {doc, client_seq, seq}` once committed; a retry with the same `(device, client_seq, bytes)` is acked with its first `seq` and stored once. The server sends `update {doc, seq, bytes, device}` to every other subscriber. -- `presence {doc, bytes}` (later) relayed to the other subscribers, never stored. +- `presence {doc, bytes}` both ways, at most 4 KiB, at most 20 a second per document per device (more are dropped), + relayed to the document's other subscribers and never stored. The agent forwards it to the apps as it is. - `unsubscribe {doc}`; `error {code, message, doc?, client_seq?}`. A refused `create` or `subscribe` is retried with backoff. - Another account's document gets the same `not_found` as a missing one. A revoked device's sockets are closed. -- Limits: one update 1 B to 1 MiB, one snapshot at most the manifest limit, at most 256 subscribed documents per +- Limits: one update 1 B to 1 MiB, one message at most an update and its envelope, one snapshot at most 36 MiB (an + upload), at most 256 subscribed documents per device, 400 updates, creates and subscribes per connection refilled at 200/s; over them, `too_large` / `rate_limited` like the HTTP API. - Quota: once a document's log holds 16 MiB (about 30,000 recents adds), every later append is refused with `quota_exceeded` (one-way: the update that crosses the cap is the last one stored). The agent keeps every unacked update, retries with a pause growing to about 64 s, and says the document is full. - `craft-cloud-server live reset-doc ` empties the log and raises the space's live generation in the same - transaction, so every live connection ends at its next ping and every agent starts over and pushes what the log - lacks. The file sync's epoch is untouched, so it never sees a reset. Nothing a machine holds is lost. A document - whose own history is over the cap stays full until the log can be compacted, which is the real fix. -- Account caps: at most 64 of an account's live documents may hold updates; the first update to one more is refused - with `quota_exceeded`, and the agent handles it like a full document. Documents with no updates (made by following - a list) are at most 1,024 per account; a create over that is refused with `quota_exceeded`, and that list stays on - the machine and is tried again after a growing pause. A live document can't be deleted yet: `reset-doc` empties - one, but machines that hold its updates send them up again. + `craft-cloud-server live reset-doc ` (or, for a document of a file, ` `) empties the + log, drops the snapshot and the import, and raises the space's live generation in the same transaction, so every + live connection ends at its next ping and every agent starts over and pushes what the log lacks. The file sync's + epoch is untouched, so it never sees a reset. Nothing a machine holds is lost. Only the log above the snapshot + counts against the cap, so compaction keeps a document from filling up. +- Account caps: at most 64 of an account's recents documents may hold updates; the first update to one more is + refused with `quota_exceeded`, and the agent handles it like a full document. Recents documents with no updates + (made by following a list) are at most 1,024 per account; a create over that is refused with `quota_exceeded`, and + that list stays on the machine and is tried again after a growing pause. Documents of files are counted apart (each + needs a file of the space): at most 65,536 per account, at most 10,000 of them holding updates (the first update to + one more is refused with `quota_exceeded`, handled like a full document), and their snapshots count against the + account's storage (they are uploads). Every row of every live document of an account, compacted ones too, counts + towards its live bytes, at most `live_bytes_per_user` (1 GiB by default); over it, every append is refused with + `quota_exceeded`. A live document can't be deleted yet: `reset-doc` empties one (and gives its bytes back), but + machines that hold its updates send them up again. +- The agent follows at most 256 documents (the server's cap) and keeps at most 256 documents of files in memory: + past that, the least recently used document no app has open and with nothing to send or waiting is unfollowed, and + written and let go of (it stays on disk, and is loaded and followed again when opened). At start only the documents + with something to send or waiting are loaded. Its copy of each document is a file of its own beside a journal of + the apps' changes since it was written (on disk before an app hears its change is kept; every record with its + length and a checksum, read up to the first that doesn't read and cut there; an append that fails is cut back and + refused, and the document written whole before its journal takes another); an older agent leaves those files + alone. +- Loro failing on a row of a catch-up taken as a batch: the catch-up is taken again one row at a time, so only the + row at fault is dropped (and said). A second failure in a run stops the document (`status.stopped`) until the agent + restarts. diff --git a/docs/server.md b/docs/server.md index 049034e..bf2a649 100644 --- a/docs/server.md +++ b/docs/server.md @@ -33,6 +33,15 @@ full: with less than 4 MiB (one chunk) left, the agent pauses and says the accou A cap under 4 MiB is refused when the server starts, since not even one chunk would fit. Per-user quotas on file sizes come later, with history. +Live documents have a cap of their own: `live_bytes_per_user` (in bytes; 1 GiB, `1_073_741_824`, by default) is +the most one account's live documents (recent-file lists and documents of files) may hold together. It counts +every update in their logs, including the ones a snapshot has replaced: those are kept until retention rules +arrive, so the total only goes down when the admin resets a document (`live reset-doc`). An update that would go +over it is refused; the agent keeps it on the machine and tries again later, and the app shows its changes as not +yet on the server. Snapshots of live documents are uploads like files' content, so they count under +`stored_bytes_per_user` instead. An account may also hold updates in at most 10,000 documents of files; the first +change to one more is refused the same way. + To add accounts: `printf '%s\n' 'their password' | craft-cloud-server user --config cc-data/config.toml add name`. `--config` can come before or after `add`.