diff --git a/DESIGN.md b/DESIGN.md index ae9d9e4..a449088 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -205,7 +205,10 @@ There are two ways a host drives a loop: - A bounded pool: default 4 threads, configurable, lazily started, shared per process or per loop (config). - It runs file operations, DNS resolution and host-submitted `FnOnce() -> BlockingResult + Send` jobs, and completes through the notifier. - Cancellation is best-effort: a job that has started runs to the end and completes as `Cancelled` if its cancel won the race. -- File I/O goes through the pool on every platform in 0.x. io_uring and IOCP file I/O are later optimisations behind the same API. +- File I/O goes through the pool on every native platform in 0.x. io_uring and IOCP file I/O are later optimisations behind the same API. + - **Typed requests** (`Loop::fs`: open, read, write, metadata, directories, namespace changes) use this one shared pool on Linux, macOS/BSD **and Windows**. Requests on one handle run FIFO; only the head is ever queued on the pool, and each accepted request reserves its queue slot, so a full queue rejects before acceptance and a queued successor always starts. + - **Adopted descriptors** (stdio, `Detached::from_fd`/`from_handle`) keep their stream paths: reusable pool jobs on Unix, and one synchronous worker per handle on Windows. Such a handle may be a pipe or console whose read blocks indefinitely and cannot be cancelled from a shared thread; it must not occupy the pool. This is the only per-handle file worker. + - **WASI** has no threads: requests run inside `turn`, bounded by its event budget, against `wasi:filesystem` preopens. ### D9. Optional layers @@ -251,7 +254,7 @@ There are two ways a host drives a loop: - one `Atomics.waitAsync` waiter service: a single helper thread, not one per call, posting to the waiting loop Each completes on the loop that submitted. -4. **The ad-hoc threads become loop handles or pool jobs.** Child stdio, dgram, stdin, pty, fs-watch, IPC, signal wake, waitAsync, N-API async work and message acks all move off dedicated threads. The only exceptions are where the OS requires a thread: Windows non-overlapped stdio and console input (§7.3). +4. **The ad-hoc threads become loop handles or pool jobs.** Child stdio, dgram, stdin, pty, fs-watch, IPC, signal wake, waitAsync, N-API async work and message acks all move off dedicated threads. The only exceptions are where the OS requires a thread: Windows non-overlapped stdio and console input (§7.3), and macOS FSEvents, which delivers on its own dispatch queue. Filesystem watches are loop handles: inotify on epoll, EVFILT_VNODE on kqueue, ReadDirectoryChangesW on the loop's IOCP. 5. **Handle transfer:** - **Between loops:** `Loop::detach(h) -> Detached` (`Send`) and `Loop::attach(Detached, token) -> Handle`. For sockets, pipes and servers across threads or workers, it cancels in-flight ops with the usual exactly-once completions before detaching. - **Between processes:** fd passing via `SCM_RIGHTS` on Unix and `WSADuplicateSocketW` / `DuplicateHandle` on Windows, exposed on pipe handles so `child.send(msg, handle)` and cluster round-robin can move sockets (today `emitter.rs:430` drops the handle). @@ -344,7 +347,8 @@ impl Loop { pub fn signal_start(&mut self, sig: Signal, tok: Token) -> io::Result; // pool - pub fn fs(&mut self, req: FsRequest, tok: Token) -> OpId; + pub fn fs(&mut self, req: FsRequest, tok: Token) -> io::Result; // pool (native) or wasi:filesystem + pub fn fs_watch(&mut self, path: &FsPath, o: WatchOptions, tok: Token) -> io::Result; // multishot Watch pub fn resolve(&mut self, req: DnsRequest, tok: Token) -> OpId; pub fn blocking BlockingResult + Send + 'static>(&mut self, f: F, tok: Token) -> OpId; } @@ -355,7 +359,8 @@ pub enum OpResult { Read { n: usize, lease: Option }, Eof, Wrote(usize), RecvFrom { n: usize, from: SocketAddr, lease: Option }, Timer, Signal(Signal), Exited(ExitStatus), - Fs(FsResult), Resolved(DnsResult), Blocking(BlockingResult), + Fs(FsResult), Watch { events: BufLease, overflow: bool }, + Resolved(DnsResult), Blocking(BlockingResult), Cancelled, Closed, Stopped, Err(Error), } pub struct Error { pub kind: ErrorKind, pub os: Option } // host maps to ECONNRESET etc. @@ -439,7 +444,8 @@ Two backends, because both versions matter now: | TTY | termios + readiness | termios + readiness | console API reader thread, VT modes | size only | size only | unsupported | | Child processes | pidfd / SIGCHLD | EVFILT_PROC | RegisterWaitForSingleObject + Job Object | unsupported | unsupported | unsupported | | Signals | sigaction + self-pipe | EVFILT_SIGNAL | SetConsoleCtrlHandler | unsupported | unsupported | unsupported | -| Files | blocking pool | blocking pool | blocking pool | `wasi:filesystem` | `wasi:filesystem` async | OPFS if host-mapped | +| Files | blocking pool | blocking pool | blocking pool | `wasi:filesystem` preopens, run in `turn` | `wasi:filesystem` async preopens, run in `turn` | unsupported (no OPFS host mapping defined) | +| File watch | inotify (recursion by the host) | FSEvents for directories (recursive), kqueue for files | ReadDirectoryChangesW (recursive) | unsupported | unsupported | unsupported | | DNS | pool (getaddrinfo) | pool (getaddrinfo) | pool (GetAddrInfoW) | `wasi:sockets/ip-name-lookup` | same | host (via fetch) | | Host integration | epoll fd | kqueue fd | event HANDLE + helper thread | runtime-owned | runtime-owned | `HostCallback` | | Cost measurement | perf instructions:u/k | rusage ri_instructions | QueryProcessCycleTime | Wasmtime fuel / instruction counts | Wasmtime fuel | browser profiler (relative) | @@ -491,7 +497,7 @@ No other Perry change is needed for P0. 4. With queued work and **no** native operations pending, the turn makes **no OS call at all**: pure post/timer/terminal churn costs zero syscalls. 5. The no-spin rule (4a) is unchanged. - *Queued work* means completions already queued before the native step: posts, blocking-pool and external-wait results, synchronous and terminal completions, including a timer's `Cancelled`/`Closed`; timer start, reset, cancel and close are therefore pure churn. A timer expiry that is due when the turn starts is not queued work: the effective wait is zero, so the turn may spend its one call on a zero-timeout poll. *Native operations* are operations submitted on native handles (sockets, pipes, files, processes, signals, web fetch/WebSocket), plus requests a backend accepts natively (WASI 0.2 DNS). `TurnInfo::os_waits` counts blocking waits and `TurnInfo::discovery_polls` counts zero-timeout discovery polls; their sum is at most one per turn. Web callback draining and Windows Event-helper queue draining make neither kind of call. **Contract tests:** a queued post beside an idle UDP receive delivers with `os_waits == 0`, `discovery_polls == 1`, then byte-verified I/O; queued posts and timer `Cancelled`/`Closed` with no native operation record zero of both; a sustained producer keeps fresh I/O progressing within two seconds. + *Queued work* means completions already queued before the native step: posts, blocking-pool and external-wait results, synchronous and terminal completions, including a timer's `Cancelled`/`Closed`; timer start, reset, cancel and close are therefore pure churn. A timer expiry that is due when the turn starts is not queued work: the effective wait is zero, so the turn may spend its one call on a zero-timeout poll. *Native operations* are operations submitted on native handles (sockets, pipes, files, processes, signals, web fetch/WebSocket), plus requests a backend accepts natively (WASI 0.2 DNS, and WASI filesystem requests, including path requests). Typed file requests on the native blocking pool are not native operations: their results arrive as queued work, like host blocking jobs. `TurnInfo::os_waits` counts blocking waits and `TurnInfo::discovery_polls` counts zero-timeout discovery polls; their sum is at most one per turn. Web callback draining and Windows Event-helper queue draining make neither kind of call. **Contract tests:** a queued post beside an idle UDP receive delivers with `os_waits == 0`, `discovery_polls == 1`, then byte-verified I/O; queued posts and timer `Cancelled`/`Closed` with no native operation record zero of both; a sustained producer keeps fresh I/O progressing within two seconds. 4. **Ticks:** no fixed-interval ticks and no minimum wait floor. 4a. **No spin.** A turn with nothing ready and a future deadline blocks until that deadline, at the precision in §7.6. It never returns immediately and never degrades into a zero-timeout poll loop. Hosts must pass exact deadlines (`Instant`, not truncated milliseconds). The Linux A/B measured what happens otherwise: 37,607 turns for 50 timers, ~70× user instructions. **Contract test:** with an idle registered socket and a 0.5 ms / 2 ms / 10 ms timer, turns per expiry ≤ 2 and zero-event OS waits ≤ 1 per expiry, on every backend. 5. **Instruction budgets per operation** (Linux, cgu=1, `perf stat -e instructions:u,instructions:k`): TCP read / write / accept, timer start + cancel, notify + turn round trip, blocking job round trip, idle turn. **Values to be set from the attribution run of today's tokio bridge**, with a target below the tokio-bridge cost and within X % of a hand-written epoll loop. The CI gate compares against a committed baseline, with a control probe that must not move. diff --git a/crates/turnloop-contract/src/filesystem.rs b/crates/turnloop-contract/src/filesystem.rs new file mode 100644 index 0000000..1d9ec33 --- /dev/null +++ b/crates/turnloop-contract/src/filesystem.rs @@ -0,0 +1,1150 @@ +//! Typed filesystem contract scenarios, shared by native and WASI backends. +//! Each scenario creates its own directory below `root` and checks real bytes on +//! disk with `std::fs` (which WASI resolves through the same preopens). +use std::{ + path::{Path, PathBuf}, + time::Duration, +}; +use turnloop::{backend::Backend, *}; + +pub fn config() -> Config { + Config { + max_handles: 32, + max_operations: 64, + events_per_turn: 16, + ..Config::default() + } +} +fn path(p: impl AsRef) -> FsPath { + FsPath::new(p).expect("prepared path") +} +/// Wait for the terminal completion of `op`; no other completion may arrive. +pub fn wait(l: &mut Driver, op: OpId) -> std::result::Result { + let mut out = Completions::with_capacity(4); + let until = l.now() + Duration::from_secs(10); + loop { + assert!(l.now() < until, "filesystem completion deadline"); + let info = l.turn(Timeout::Until(until), &mut out).expect("turn"); + assert!( + info.os_waits + info.discovery_polls <= 1, + "one native call per turn" + ); + let mut drained = out.drain(); + let Some(c) = drained.next() else { + continue; + }; + assert!(drained.next().is_none(), "one completion per request"); + assert_eq!(c.op, Some(op)); + assert!(c.terminal); + return match c.result { + OpResult::Fs(r) => Ok(r), + OpResult::Err(e) => Err(e), + other => panic!("unexpected {other:?}"), + }; + } +} +pub fn run( + l: &mut Driver, + request: FsRequest, +) -> std::result::Result { + let op = l.fs(request, Token(7)).expect("accepted request"); + wait(l, op) +} +fn done(l: &mut Driver, request: FsRequest) { + let r = run(l, request).expect("request succeeds"); + assert!(matches!(r, FsResult::Done), "{r:?}"); +} +fn kind(l: &mut Driver, request: FsRequest) -> ErrorKind { + let e = run(l, request).expect_err("request fails"); + assert!(e.os.is_some(), "native error code retained: {e:?}"); + e.kind +} +fn stat(l: &mut Driver, p: &Path, follow_symlinks: bool) -> FileMetadata { + match run( + l, + FsRequest::Stat { + path: path(p), + follow_symlinks, + }, + ) { + Ok(FsResult::Metadata(m)) => *m, + other => panic!("stat {p:?}: {other:?}"), + } +} +fn open(l: &mut Driver, p: &Path, options: FileOptions) -> Handle { + match run( + l, + FsRequest::Open { + path: path(p), + options, + }, + ) { + Ok(FsResult::Opened(h)) => h, + other => panic!("open {p:?}: {other:?}"), + } +} +fn close(l: &mut Driver, h: Handle) { + l.close(h, Token(99)).expect("close handle"); + let mut out = Completions::with_capacity(1); + let until = l.now() + Duration::from_secs(5); + loop { + assert!(l.now() < until); + l.turn(Timeout::Until(until), &mut out).expect("turn"); + if let Some(c) = out.drain().next() { + assert_eq!(c.handle, Some(h)); + assert!(matches!(c.result, OpResult::Closed), "{c:?}"); + return; + } + } +} +fn write_buf(bytes: &'static [u8]) -> WriteBuf { + // SAFETY: static immutable bytes outlive every operation. + WriteBuf::Provided(unsafe { IoBuf::from_raw_parts(bytes.as_ptr(), bytes.len()) }) +} +fn provided(bytes: &mut [u8]) -> ReadBuf { + // SAFETY: callers keep the region in place and read it only after the terminal completion. + ReadBuf::Provided(unsafe { IoBufMut::from_raw_parts(bytes.as_mut_ptr(), bytes.len()) }) +} +fn fresh(root: &Path, name: &str) -> PathBuf { + let dir = root.join(name); + let _ = std::fs::remove_dir_all(&dir); + dir +} +fn read_names(l: &mut Driver, dir: &Path, page: usize) -> Vec<(FileType, String)> { + let d = match run(l, FsRequest::OpenDir { path: path(dir) }) { + Ok(FsResult::Opened(h)) => h, + other => panic!("opendir: {other:?}"), + }; + let mut names = Vec::new(); + let mut storage = vec![0u8; page]; + let mut pooled = false; + loop { + let buffer = if pooled { + ReadBuf::Pooled + } else { + provided(&mut storage) + }; + let (n, lease, eof) = match run(l, FsRequest::ReadDir { dir: d, buffer }) { + Ok(FsResult::Directory { n, lease, eof }) => (n, lease, eof), + other => panic!("readdir: {other:?}"), + }; + let bytes = match &lease { + Some(lease) => lease.as_slice(), + None => &storage[..n], + }; + assert_eq!(bytes.len(), n); + for (t, name) in DirEntries::new(bytes) { + names.push(( + t, + String::from_utf8(name.to_vec()).expect("test names are UTF-8"), + )); + } + if eof { + break; + } + assert!(n > 0, "a non-final page carries at least one entry"); + pooled = !pooled; + } + close(l, d); + names.sort_by(|a, b| a.1.cmp(&b.1)); + names +} + +/// Real bytes, cursor and positional I/O, metadata, links, copies and directory pages. +pub fn bytes_metadata_namespace(root: &Path) { + let mut l = Driver::::new(config()).expect("loop"); + let dir = fresh(root, "namespace"); + done( + &mut l, + FsRequest::Mkdir { + path: path(&dir), + mode: 0o755, + }, + ); + let file = dir.join("file"); + let create = FileOptions { + read: true, + write: true, + create: true, + exclusive: true, + ..FileOptions::default() + }; + let h = open(&mut l, &file, create); + assert_eq!( + kind( + &mut l, + FsRequest::Open { + path: path(&file), + options: create + } + ), + ErrorKind::AlreadyExists + ); + for (bytes, offset, n) in [ + (&b"abc"[..], None, 3), + (&b"Z"[..], Some(1), 1), + (&b"def"[..], None, 3), + ] { + let r = run( + &mut l, + FsRequest::Write { + file: h, + buffer: write_buf(bytes), + offset, + }, + ); + assert!(matches!(r, Ok(FsResult::Wrote(w)) if w == n), "{r:?}"); + } + let mut bytes = [0u8; 8]; + let r = run( + &mut l, + FsRequest::Read { + file: h, + buffer: provided(&mut bytes), + offset: Some(0), + }, + ); + assert!( + matches!(r, Ok(FsResult::Read { n: 6, lease: None })), + "{r:?}" + ); + assert_eq!(&bytes[..6], b"aZcdef"); + let r = run( + &mut l, + FsRequest::Read { + file: h, + buffer: ReadBuf::Pooled, + offset: Some(2), + }, + ); + let Ok(FsResult::Read { + n: 4, + lease: Some(lease), + }) = r + else { + panic!("pooled positional read: {r:?}") + }; + assert_eq!(lease.as_slice(), b"cdef"); + drop(lease); + let r = run( + &mut l, + FsRequest::Read { + file: h, + buffer: provided(&mut bytes), + offset: None, + }, + ); + assert!( + matches!(r, Ok(FsResult::Read { n: 0, .. })), + "cursor is at end of file: {r:?}" + ); + for data_only in [true, false] { + done(&mut l, FsRequest::Sync { file: h, data_only }); + } + let Ok(FsResult::Metadata(m)) = run(&mut l, FsRequest::Fstat { file: h }) else { + panic!("fstat") + }; + assert_eq!((m.kind, m.size), (FileType::File, 6)); + assert!(m.modified.is_some()); + #[cfg(not(target_os = "wasi"))] + assert!(m.inode.is_some() && m.device.is_some() && m.links == Some(1)); + #[cfg(unix)] + assert_eq!( + m.mode.map(|mode| mode & 0o170000), + Some(0o100000), + "regular type bits" + ); + done(&mut l, FsRequest::Truncate { file: h, size: 2 }); + assert_eq!(stat(&mut l, &file, true).size, 2); + let at = FileTime { + seconds: 1_000_000_000, + nanoseconds: 123_456_700, + }; + done( + &mut l, + FsRequest::SetTimes { + target: FsTarget::File(h), + accessed: TimeChange::Keep, + modified: TimeChange::At(at), + }, + ); + assert_eq!(stat(&mut l, &file, true).modified, Some(at)); + done(&mut l, FsRequest::Close { file: h }); + assert!( + run(&mut l, FsRequest::Fstat { file: h }).is_err(), + "closed descriptor rejects later requests" + ); + close(&mut l, h); + assert!( + l.fs(FsRequest::Fstat { file: h }, Token(1)).is_err(), + "released handle is rejected before acceptance" + ); + assert_eq!(std::fs::read(&file).expect("real bytes"), b"aZ"); + + let hard = dir.join("hard"); + done( + &mut l, + FsRequest::Link { + existing: path(&file), + link: path(&hard), + }, + ); + assert_eq!(stat(&mut l, &file, true).links, Some(2)); + let link = dir.join("link"); + done( + &mut l, + FsRequest::Symlink { + target: path("file"), + link: path(&link), + kind: SymlinkKind::File, + }, + ); + assert_eq!(stat(&mut l, &link, false).kind, FileType::Symlink); + let through = stat(&mut l, &link, true); + assert_eq!((through.kind, through.size), (FileType::File, 2)); + let r = run( + &mut l, + FsRequest::ReadLink { + path: path(&link), + buffer: ReadBuf::Pooled, + }, + ); + let Ok(FsResult::Bytes { + n: 4, + lease: Some(content), + }) = r + else { + panic!("readlink: {r:?}") + }; + assert_eq!(content.as_slice(), b"file"); + drop(content); + #[cfg(not(target_os = "wasi"))] + { + let mut canonical = [0u8; 4096]; + let r = run( + &mut l, + FsRequest::RealPath { + path: path(&link), + buffer: provided(&mut canonical), + }, + ); + let Ok(FsResult::Bytes { n, lease: None }) = r else { + panic!("realpath: {r:?}") + }; + let expected = std::fs::canonicalize(&file).expect("std canonical path"); + // std keeps the Windows verbatim prefix; RealPath reports the DOS form. + #[cfg(windows)] + let expected = PathBuf::from( + expected + .to_str() + .expect("UTF-8 path") + .strip_prefix(r"\\?\") + .expect("verbatim canonical path"), + ); + assert_eq!( + Path::new(std::str::from_utf8(&canonical[..n]).expect("UTF-8 path")), + expected + ); + } + done( + &mut l, + FsRequest::Access { + path: path(&file), + mode: AccessMode::default(), + }, + ); + let copy = dir.join("copy"); + done( + &mut l, + FsRequest::CopyFile { + from: path(&file), + to: path(©), + exclusive: true, + }, + ); + assert_eq!(std::fs::read(©).expect("copied bytes"), b"aZ"); + assert_eq!( + kind( + &mut l, + FsRequest::CopyFile { + from: path(&file), + to: path(©), + exclusive: true + } + ), + ErrorKind::AlreadyExists + ); + let moved = dir.join("moved"); + done( + &mut l, + FsRequest::Rename { + from: path(©), + to: path(&moved), + }, + ); + assert_eq!( + kind( + &mut l, + FsRequest::Stat { + path: path(©), + follow_symlinks: true + } + ), + ErrorKind::NotFound + ); + assert_eq!( + kind(&mut l, FsRequest::Rmdir { path: path(&dir) }), + ErrorKind::DirectoryNotEmpty + ); + let expected = [ + (FileType::File, "file"), + (FileType::File, "hard"), + (FileType::Symlink, "link"), + (FileType::File, "moved"), + ]; + for page in [9, 16, 4096] { + let names = read_names(&mut l, &dir, page); + let names: Vec<_> = names.iter().map(|(t, n)| (*t, n.as_str())).collect(); + assert_eq!(names, expected, "page size {page}"); + } + for name in ["file", "hard", "link", "moved"] { + done( + &mut l, + FsRequest::Unlink { + path: path(dir.join(name)), + }, + ); + } + done(&mut l, FsRequest::Rmdir { path: path(&dir) }); + assert!(!dir.exists()); + assert!(!l.alive()); +} + +/// Portable error kinds with native codes, and a page too small for any entry. +pub fn errors(root: &Path) { + let mut l = Driver::::new(config()).expect("loop"); + let dir = fresh(root, "errors"); + std::fs::create_dir(&dir).expect("fixture"); + std::fs::write(dir.join("plain"), b"x").expect("fixture"); + let missing = dir.join("missing"); + assert_eq!( + kind( + &mut l, + FsRequest::Open { + path: path(&missing), + options: FileOptions::default() + } + ), + ErrorKind::NotFound + ); + assert_eq!( + kind( + &mut l, + FsRequest::Mkdir { + path: path(&dir), + mode: 0o755 + } + ), + ErrorKind::AlreadyExists + ); + assert_eq!( + kind(&mut l, FsRequest::Rmdir { path: path(&dir) }), + ErrorKind::DirectoryNotEmpty + ); + #[cfg(not(windows))] + assert_eq!( + kind( + &mut l, + FsRequest::Stat { + path: path(dir.join("plain").join("child")), + follow_symlinks: true + } + ), + ErrorKind::NotADirectory + ); + let d = match run(&mut l, FsRequest::OpenDir { path: path(&dir) }) { + Ok(FsResult::Opened(h)) => h, + other => panic!("opendir: {other:?}"), + }; + let mut tiny = [0u8; 3]; + let e = run( + &mut l, + FsRequest::ReadDir { + dir: d, + buffer: provided(&mut tiny), + }, + ) + .expect_err("no entry fits"); + assert_eq!(e.kind, ErrorKind::ResourceLimit); + let mut page = [0u8; 64]; + let r = run( + &mut l, + FsRequest::ReadDir { + dir: d, + buffer: provided(&mut page), + }, + ); + let Ok(FsResult::Directory { n, .. }) = r else { + panic!("retained entry: {r:?}") + }; + assert_eq!( + DirEntries::new(&page[..n]).collect::>(), + [(FileType::File, &b"plain"[..])], + "the entry that did not fit is delivered next" + ); + close(&mut l, d); + assert!( + l.fs( + FsRequest::Read { + file: d, + buffer: ReadBuf::Pooled, + offset: None + }, + Token(1) + ) + .is_err() + ); + std::fs::remove_dir_all(&dir).expect("cleanup"); + assert!(!l.alive()); +} + +/// Close cancels queued requests before Closed; unstarted buffers are never touched. +pub fn fifo_cancel_close(root: &Path) { + let mut l = Driver::::new(config()).expect("loop"); + let dir = fresh(root, "cancel"); + std::fs::create_dir(&dir).expect("fixture"); + let file = dir.join("file"); + let h = open( + &mut l, + &file, + FileOptions { + read: true, + write: true, + create: true, + truncate: true, + ..FileOptions::default() + }, + ); + let write = l + .fs( + FsRequest::Write { + file: h, + buffer: write_buf(b"unwritten"), + offset: None, + }, + Token(1), + ) + .expect("write"); + let mut bytes = [0x5a; 32]; + let read = l + .fs( + FsRequest::Read { + file: h, + buffer: provided(&mut bytes), + offset: Some(0), + }, + Token(2), + ) + .expect("queued read"); + let fstat = l.fs(FsRequest::Fstat { file: h }, Token(3)).expect("fstat"); + l.close(h, Token(4)).expect("close"); + assert!(l.fs(FsRequest::Fstat { file: h }, Token(5)).is_err()); + let mut out = Completions::with_capacity(1); + let until = l.now() + Duration::from_secs(10); + let mut terminal = Vec::new(); + let mut wrote = false; + loop { + assert!(l.now() < until); + l.turn(Timeout::Until(until), &mut out).expect("turn"); + let Some(c) = out.drain().next() else { + continue; + }; + if matches!(c.result, OpResult::Closed) { + break; + } + let op = c.op.expect("request completion"); + assert!(c.terminal && !terminal.contains(&op), "exactly once"); + match c.result { + OpResult::Fs(FsResult::Wrote(9)) if op == write => wrote = true, + OpResult::Cancelled => {} + other => panic!("unexpected {other:?}"), + } + if op != write { + assert!(matches!(c.token, Token(2 | 3))); + } + terminal.push(op); + } + terminal.sort(); + let mut expected = vec![write, read, fstat]; + expected.sort(); + assert_eq!(terminal, expected, "every request precedes Closed"); + assert_eq!( + bytes, [0x5a; 32], + "an unstarted read never touched its buffer" + ); + for _ in 0..4 { + l.turn(Timeout::Now, &mut out).expect("quiet"); + assert!(out.is_empty()); + } + let content = std::fs::read(&file).expect("file"); + if wrote { + assert_eq!(content, b"unwritten"); + } else { + // D8: a write already running on a pool thread when close cancelled it + // runs to its end and still reports Cancelled; mutations are not rolled + // back. It never runs partially, and a write that never started never + // writes (the queued read above proves unstarted buffers stay untouched). + assert!( + content.is_empty() || content == b"unwritten", + "cancelled write left partial content: {content:?}" + ); + } + std::fs::remove_dir_all(&dir).expect("cleanup"); + assert!(!l.alive()); +} + +/// DESIGN §10 rule 3: a producer that queues a post before every turn cannot +/// starve filesystem requests. Queued turns never block. Backend-executed (WASI) +/// requests are native operations and get their discovery step; pool requests +/// are not, so those turns make no native call at all. +pub fn queued_posts_do_not_starve_requests(root: &Path) { + let mut l = Driver::::new(config()).expect("loop"); + let dir = fresh(root, "starvation"); + std::fs::create_dir(&dir).expect("fixture"); + let file = dir.join("file"); + std::fs::write(&file, b"payload").expect("fixture"); + let poster = l.poster(); + let mut out = Completions::with_capacity(4); + let mut posts = 0u64; + let mut serve = |l: &mut Driver, pending: &mut Vec, phase: &str| { + let until = l.now() + Duration::from_secs(2); + let mut turns = 0; + let mut results = Vec::new(); + while !pending.is_empty() { + assert!( + l.now() < until, + "{phase}: starved behind queued posts after {turns} turns: {pending:?}" + ); + poster + .post(Token(90), Payload::U64(posts)) + .expect("replenish"); + let info = l.turn(Timeout::Now, &mut out).expect("turn"); + turns += 1; + assert_eq!(info.os_waits, 0, "a queued turn never blocks"); + assert!(info.discovery_polls <= 1); + if B::FILESYSTEM == backend::Filesystem::Pool { + assert_eq!(info.discovery_polls, 0, "pool results are queued work"); + } + for c in out.drain() { + match c.result { + OpResult::Posted(Payload::U64(_)) => posts += 1, + OpResult::Fs(result) => { + let op = c.op.expect("request"); + assert!(pending.contains(&op), "exactly once"); + pending.retain(|p| *p != op); + results.push((op, result)); + } + other => panic!("unexpected {other:?}"), + } + } + } + results + }; + // Phase 1: only path requests are pending. On WASI they are the backend's + // native operations; nothing else would give the backend a turn. + let stat = l + .fs( + FsRequest::Stat { + path: path(&file), + follow_symlinks: true, + }, + Token(1), + ) + .expect("path request"); + let mkdir = l + .fs( + FsRequest::Mkdir { + path: path(dir.join("made")), + mode: 0o755, + }, + Token(2), + ) + .expect("path mutation"); + for (op, result) in serve(&mut l, &mut vec![stat, mkdir], "path requests") { + match result { + FsResult::Metadata(m) if op == stat => assert_eq!(m.size, 7), + FsResult::Done if op == mkdir => {} + other => panic!("unexpected {other:?}"), + } + } + assert!(dir.join("made").is_dir(), "the mutation ran"); + // Phase 2: an open and a handle request under the same producer. + let open_op = l + .fs( + FsRequest::Open { + path: path(&file), + options: FileOptions::default(), + }, + Token(3), + ) + .expect("open"); + let results = serve(&mut l, &mut vec![open_op], "open"); + let [(_, FsResult::Opened(handle))] = results.as_slice() else { + panic!("open: {results:?}") + }; + let handle = *handle; + let fstat = l + .fs(FsRequest::Fstat { file: handle }, Token(4)) + .expect("handle request"); + let results = serve(&mut l, &mut vec![fstat], "handle request"); + assert!(matches!(results.as_slice(), [(_, FsResult::Metadata(m))] if m.size == 7)); + assert!(posts >= 3, "the producer's posts were delivered too"); + while l + .turn(Timeout::Now, &mut out) + .expect("drain posts") + .completions + != 0 + {} + close(&mut l, handle); + std::fs::remove_dir_all(&dir).expect("cleanup"); + assert!(!l.alive()); +} + +/// A pooled read waits for a lease without spinning, then completes with real bytes. +pub fn pooled_lease_wait(root: &Path) { + let mut l = Driver::::new(Config { + pooled_buffers: 1, + pooled_buffer_size: 64, + ..config() + }) + .expect("loop"); + let dir = fresh(root, "leases"); + std::fs::create_dir(&dir).expect("fixture"); + let file = dir.join("file"); + std::fs::write(&file, [7u8; 64]).expect("fixture"); + let h = open(&mut l, &file, FileOptions::default()); + let request = |offset| FsRequest::Read { + file: h, + buffer: ReadBuf::Pooled, + offset: Some(offset), + }; + let Ok(FsResult::Read { + n: 64, + lease: Some(held), + }) = run(&mut l, request(0)) + else { + panic!("first pooled read") + }; + let waiting = l + .fs(request(32), Token(8)) + .expect("accepted while no lease"); + let mut out = Completions::with_capacity(4); + for _ in 0..3 { + let at = l.now() + Duration::from_millis(5); + let timer = l.timer(at, None, Token(9)).expect("timer"); + let mut polls = 0; + loop { + let info = l.turn(Timeout::Until(at), &mut out).expect("turn"); + polls += 1; + if !out.is_empty() { + assert_eq!(out.len(), 1); + assert!(matches!(out[0].result, OpResult::Timer), "{:?}", out[0]); + break; + } + assert_eq!(info.completions, 0); + } + assert!( + polls <= 2, + "no spin while waiting for a lease ({polls} turns)" + ); + close(&mut l, timer); + } + drop(held); + let r = wait(&mut l, waiting); + let Ok(FsResult::Read { + n: 32, + lease: Some(bytes), + }) = r + else { + panic!("released lease starts the read: {r:?}") + }; + assert_eq!(bytes.as_slice(), [7u8; 32]); + drop(bytes); + close(&mut l, h); + std::fs::remove_dir_all(&dir).expect("cleanup"); + assert!(!l.alive()); +} + +/// Collect watch records until `done` accepts the accumulated list. +#[cfg(not(target_arch = "wasm32"))] +pub fn watch_until( + l: &mut Driver, + h: Handle, + records: &mut Vec<(WatchKind, String)>, + mut done: impl FnMut(&[(WatchKind, String)]) -> bool, +) -> bool { + let mut out = Completions::with_capacity(8); + let until = l.now() + Duration::from_secs(10); + let mut overflow = false; + while !done(records) { + assert!( + l.now() < until, + "watch deadline; records so far: {records:?}" + ); + l.turn(Timeout::Until(until), &mut out).expect("turn"); + for c in out.drain() { + assert_eq!(c.handle, Some(h)); + let OpResult::Watch { + events, + overflow: lost, + } = c.result + else { + panic!("unexpected {:?}", c.result); + }; + assert!(!c.terminal); + overflow |= lost; + for (kind, name) in WatchEvents::new(events.as_slice()) { + records.push((kind, String::from_utf8_lossy(name).into_owned())); + } + } + } + overflow +} +#[cfg(not(target_arch = "wasm32"))] +fn stop(l: &mut Driver, h: Handle, stopped: bool) { + if stopped { + l.fs_watch_stop(h, Token(50)).expect("stop"); + } else { + l.close(h, Token(50)).expect("close watch"); + } + let mut out = Completions::with_capacity(4); + let until = l.now() + Duration::from_secs(10); + let mut terminal = false; + loop { + assert!(l.now() < until); + l.turn(Timeout::Until(until), &mut out).expect("turn"); + for c in out.drain() { + match c.result { + // Batches published before the stop request may still arrive first. + OpResult::Watch { .. } => assert!(!terminal), + OpResult::Stopped if stopped => terminal = true, + OpResult::Cancelled if !stopped => terminal = true, + OpResult::Closed => { + assert!(terminal, "terminal operation precedes Closed"); + return; + } + other => panic!("unexpected {other:?}"), + } + } + } +} +#[cfg(not(target_arch = "wasm32"))] +fn names(records: &[(WatchKind, String)], name: &str) -> Vec { + records + .iter() + .filter(|(_, n)| n == name) + .map(|(k, _)| *k) + .collect() +} + +/// Directory watch: liveness barrier, content change, rename ordering, stop. +#[cfg(not(target_arch = "wasm32"))] +pub fn watch_directory(root: &Path) { + use std::io::Write; + let mut l = Driver::::new(config()).expect("loop"); + let dir = fresh(root, "watched"); + std::fs::create_dir(&dir).expect("fixture"); + std::fs::write(dir.join("existing"), b"x").expect("fixture"); + let h = l + .fs_watch(&path(&dir), WatchOptions::default(), Token(1)) + .expect("watch"); + let mut records = Vec::new(); + std::fs::write(dir.join("sentinel"), b"").expect("sentinel"); + watch_until(&mut l, h, &mut records, |r| { + !names(r, "sentinel").is_empty() + }); + let barrier = records.len(); + let mut existing = std::fs::OpenOptions::new() + .append(true) + .open(dir.join("existing")) + .expect("open existing"); + existing.write_all(b"appended").expect("append"); + existing.sync_all().expect("flush"); + drop(existing); + // FSEvents reports cumulative per-path flags, so a content change to a + // recently created file can read as Rename there (as in libuv and Node). + let fsevents = cfg!(target_os = "macos"); + watch_until(&mut l, h, &mut records, |r| { + let kinds = names(&r[barrier..], "existing"); + kinds.contains(&WatchKind::Change) || (fsevents && !kinds.is_empty()) + }); + let changed = records.len(); + std::fs::rename(dir.join("existing"), dir.join("renamed")).expect("rename"); + std::fs::remove_file(dir.join("renamed")).expect("remove"); + watch_until(&mut l, h, &mut records, |r| { + let r = &r[changed..]; + names(r, "existing").contains(&WatchKind::Rename) && !names(r, "renamed").is_empty() + }); + let after = &records[changed..]; + let old = after + .iter() + .position(|(_, n)| n == "existing") + .expect("old name"); + let new = after + .iter() + .position(|(_, n)| n == "renamed") + .expect("new name"); + assert!(old < new, "native order is preserved: {after:?}"); + assert!( + records + .iter() + .all(|(_, n)| !n.contains('/') && !n.contains('\\')), + "non-recursive watch reports direct children: {records:?}" + ); + stop(&mut l, h, true); + std::fs::write(dir.join("late"), b"").expect("after stop"); + let mut out = Completions::with_capacity(4); + let at = l.now() + Duration::from_millis(50); + while l.now() < at { + l.turn(Timeout::Until(at), &mut out).expect("quiet"); + assert!( + out.is_empty(), + "no delivery after Closed: {:?}", + out.iter().collect::>() + ); + } + std::fs::remove_dir_all(&dir).expect("cleanup"); + assert!(!l.alive()); +} + +/// Watching a single file reports its base name; close cancels before Closed. +#[cfg(not(target_arch = "wasm32"))] +pub fn watch_file(root: &Path) { + use std::io::Write; + let mut l = Driver::::new(config()).expect("loop"); + let dir = fresh(root, "watched-file"); + std::fs::create_dir(&dir).expect("fixture"); + let file = dir.join("subject"); + std::fs::write(&file, b"x").expect("fixture"); + let h = l + .fs_watch(&path(&file), WatchOptions::default(), Token(1)) + .expect("file watch"); + let mut records = Vec::new(); + let mut f = std::fs::OpenOptions::new() + .append(true) + .open(&file) + .expect("open"); + f.write_all(b"more").expect("append"); + f.sync_all().expect("flush"); + drop(f); + watch_until(&mut l, h, &mut records, |r| { + names(r, "subject").contains(&WatchKind::Change) + }); + std::fs::rename(&file, dir.join("elsewhere")).expect("rename away"); + watch_until(&mut l, h, &mut records, |r| { + names(r, "subject").contains(&WatchKind::Rename) + }); + assert!( + records.iter().all(|(_, n)| n == "subject"), + "a file watch names only its subject: {records:?}" + ); + stop(&mut l, h, false); + std::fs::remove_dir_all(&dir).expect("cleanup"); + assert!(!l.alive()); +} + +/// Held leases defer delivery without spinning; loss beyond the bound is reported. +#[cfg(not(target_arch = "wasm32"))] +pub fn watch_backpressure(root: &Path) { + let mut l = Driver::::new(Config { + pooled_buffers: 1, + ..config() + }) + .expect("loop"); + let dir = fresh(root, "watch-pressure"); + std::fs::create_dir(&dir).expect("fixture"); + let h = l + .fs_watch(&path(&dir), WatchOptions::default(), Token(1)) + .expect("watch"); + std::fs::write(dir.join("first"), b"").expect("first"); + let mut out = Completions::with_capacity(8); + let until = l.now() + Duration::from_secs(10); + let held = loop { + assert!(l.now() < until); + l.turn(Timeout::Until(until), &mut out).expect("turn"); + if let Some(c) = out.drain().next() { + let OpResult::Watch { events, .. } = c.result else { + panic!("unexpected {:?}", c.result) + }; + break events; + } + }; + // Enough distinct names to exceed the bounded per-watch storage. + let count = 3000; + for i in 0..count { + std::fs::write(dir.join(format!("n{i:05}")), b"").expect("event"); + } + for _ in 0..3 { + let at = l.now() + Duration::from_millis(20); + let timer = l.timer(at, None, Token(2)).expect("timer"); + // Native deliveries (FSEvents notifies from its queue) may wake the loop: + // such turns block (`os_waits`) or, when the notification landed while the + // loop was running, make one zero-timeout discovery poll (`discovery_polls`). + // A spin is a turn that made no native call and produced nothing: the + // backend claiming work it cannot perform while every lease is held. + let (mut spins, mut turns) = (0, 0); + loop { + let info = l.turn(Timeout::Until(at), &mut out).expect("turn"); + assert!(info.os_waits + info.discovery_polls <= 1); + turns += 1; + if let Some(c) = out.drain().next() { + assert!( + matches!(c.result, OpResult::Timer), + "no batch without a lease: {c:?}" + ); + assert!(l.now() >= at); + break; + } + spins += usize::from(info.os_waits + info.discovery_polls == 0); + } + assert!( + spins <= 1, + "no spin while every lease is held ({spins} turns without a native call)" + ); + assert!( + turns <= 64, + "wakes are bounded by native deliveries, not a poll loop ({turns} turns in 20 ms)" + ); + close(&mut l, timer); + } + drop(held); + let mut records = Vec::new(); + let overflow = watch_until(&mut l, h, &mut records, |r| { + r.iter().any(|(_, n)| n.starts_with('n')) + }); + let mut overflowed = overflow; + let quiet = l.now() + Duration::from_millis(500); + while l.now() < quiet { + l.turn(Timeout::Until(quiet), &mut out).expect("turn"); + for c in out.drain() { + if let OpResult::Watch { overflow, .. } = c.result { + overflowed |= overflow; + } + } + } + assert!( + overflowed, + "{count} events while the only lease was held must report loss" + ); + stop(&mut l, h, false); + std::fs::remove_dir_all(&dir).expect("cleanup"); + assert!(!l.alive()); +} + +/// Recursive scope where native (FSEvents, ReadDirectoryChangesW), Unsupported elsewhere. +#[cfg(not(target_arch = "wasm32"))] +pub fn watch_recursive(root: &Path) { + let mut l = Driver::::new(config()).expect("loop"); + let dir = fresh(root, "watch-tree"); + std::fs::create_dir_all(dir.join("sub")).expect("fixture"); + let result = l.fs_watch(&path(&dir), WatchOptions { recursive: true }, Token(1)); + if cfg!(any(target_os = "macos", windows)) { + let h = result.expect("native recursive watch"); + std::fs::write(dir.join("sub").join("deep"), b"").expect("nested event"); + let mut records = Vec::new(); + let nested = if cfg!(windows) { + "sub\\deep" + } else { + "sub/deep" + }; + watch_until(&mut l, h, &mut records, |r| !names(r, nested).is_empty()); + stop(&mut l, h, true); + let flat = l + .fs_watch(&path(&dir), WatchOptions::default(), Token(2)) + .expect("flat watch"); + std::fs::write(dir.join("sub").join("hidden"), b"").expect("nested"); + std::fs::write(dir.join("top"), b"").expect("direct"); + let mut records = Vec::new(); + watch_until(&mut l, flat, &mut records, |r| !names(r, "top").is_empty()); + assert!( + records.iter().all(|(_, n)| !n.contains("hidden")), + "{records:?}" + ); + stop(&mut l, flat, true); + } else { + assert_eq!( + result.expect_err("no native recursion").kind, + ErrorKind::Unsupported + ); + } + std::fs::remove_dir_all(&dir).expect("cleanup"); + assert!(!l.alive()); +} + +/// WASI: paths outside every preopen are not reachable, capabilities without a +/// `wasi:filesystem` equivalent reject before acceptance, and watches are Unsupported. +#[cfg(target_os = "wasi")] +pub fn capability_scope(root: &Path) { + let mut l = Driver::::new(config()).expect("loop"); + let e = run( + &mut l, + FsRequest::Stat { + path: path("/etc/passwd"), + follow_symlinks: true, + }, + ) + .expect_err("no ambient authority"); + assert_eq!(e.kind, ErrorKind::NotFound); + let e = run( + &mut l, + FsRequest::Stat { + path: path("relative/without/dot/preopen"), + follow_symlinks: true, + }, + ) + .expect_err("relative paths need a '.' preopen"); + assert_eq!(e.kind, ErrorKind::NotFound); + let r = run( + &mut l, + FsRequest::Stat { + path: path(root), + follow_symlinks: true, + }, + ); + assert!( + matches!(&r, Ok(FsResult::Metadata(m)) if m.kind == FileType::Directory), + "{r:?}" + ); + for request in [ + FsRequest::RealPath { + path: path(root), + buffer: ReadBuf::Pooled, + }, + FsRequest::Chmod { + target: FsTarget::Path { + path: path(root), + follow_symlinks: true, + }, + mode: 0o700, + }, + FsRequest::Access { + path: path(root), + mode: AccessMode { + read: true, + ..AccessMode::default() + }, + }, + ] { + assert_eq!( + l.fs(request, Token(1)).expect_err("rejected").kind, + ErrorKind::Unsupported + ); + } + assert_eq!( + l.fs_watch(&path(root), WatchOptions::default(), Token(2)) + .expect_err("no wasi watch API") + .kind, + ErrorKind::Unsupported + ); + assert!(!l.alive()); +} diff --git a/crates/turnloop-contract/src/lib.rs b/crates/turnloop-contract/src/lib.rs index 067ff80..ecd4970 100644 --- a/crates/turnloop-contract/src/lib.rs +++ b/crates/turnloop-contract/src/lib.rs @@ -1344,6 +1344,8 @@ pub fn no_spin() { ); } +#[cfg(not(all(target_arch = "wasm32", target_os = "unknown")))] +pub mod filesystem; pub mod native_surface; #[cfg(feature = "executor")] diff --git a/crates/turnloop-contract/tests/allocations.rs b/crates/turnloop-contract/tests/allocations.rs index 880ad34..c4143f1 100644 --- a/crates/turnloop-contract/tests/allocations.rs +++ b/crates/turnloop-contract/tests/allocations.rs @@ -32,11 +32,26 @@ static ALL_THREADS_ACTIVE: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false); #[cfg(windows)] static ALL_THREADS_ALLOCS: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); -fn record() { +/// The first counted allocations on any thread: (OS thread id, size, align). +/// Recorded without allocating, printed by gates that fail, to name the source. +#[cfg(windows)] +static ALL_THREADS_SEEN: [[std::sync::atomic::AtomicUsize; 3]; 8] = + [const { [const { std::sync::atomic::AtomicUsize::new(0) }; 3] }; 8]; +fn record(layout: Layout) { #[cfg(windows)] if ALL_THREADS_ACTIVE.load(std::sync::atomic::Ordering::Relaxed) { - ALL_THREADS_ALLOCS.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + use std::sync::atomic::Ordering::Relaxed; + let n = ALL_THREADS_ALLOCS.fetch_add(1, Relaxed); + if let Some(slot) = ALL_THREADS_SEEN.get(n) { + // SAFETY: GetCurrentThreadId takes no arguments and cannot fail. + let thread = unsafe { windows_sys::Win32::System::Threading::GetCurrentThreadId() }; + slot[0].store(thread as usize, Relaxed); + slot[1].store(layout.size(), Relaxed); + slot[2].store(layout.align(), Relaxed); + } } + #[cfg(not(windows))] + let _ = layout; if ACTIVE.try_with(Cell::get).unwrap_or(false) { let _ = ALLOCS.try_with(|n| n.set(n.get() + 1)); } @@ -45,17 +60,17 @@ fn record() { // access already initialized thread-local Cells and never allocate themselves. unsafe impl GlobalAlloc for Counting { unsafe fn alloc(&self, layout: Layout) -> *mut u8 { - record(); + record(layout); // SAFETY: GlobalAlloc's caller supplies a valid layout, forwarded unchanged. unsafe { System.alloc(layout) } } unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 { - record(); + record(layout); // SAFETY: GlobalAlloc's caller supplies a valid layout, forwarded unchanged. unsafe { System.alloc_zeroed(layout) } } unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, size: usize) -> *mut u8 { - record(); + record(Layout::from_size_align(size, layout.align()).unwrap_or(layout)); // SAFETY: pointer/layout and new size are forwarded under GlobalAlloc's contract. unsafe { System.realloc(ptr, layout, size) } } @@ -1810,6 +1825,365 @@ fn windows_console_control_delivery_allocates_nothing_on_any_thread() { println!("console allocation subject: 200 deliveries, 2 stops, 2 closes"); } +/// Directory for filesystem allocation subjects: the temporary directory natively, +/// the `/turnloop-fs` preopen on WASI (see scripts/ci/wasmtime-runner.sh). +#[cfg(not(target_os = "wasi"))] +fn fs_root(name: &str) -> std::path::PathBuf { + let dir = std::env::temp_dir().join(format!("tl-alloc-fs-{}-{name}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("allocation fixture directory"); + dir +} +#[cfg(target_os = "wasi")] +fn fs_root(name: &str) -> std::path::PathBuf { + let dir = std::path::Path::new("/turnloop-fs").join(format!("alloc-{name}")); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("preopened allocation fixture directory"); + dir +} +fn count_all_threads(enabled: bool) { + #[cfg(windows)] + { + use std::sync::atomic::Ordering; + ALL_THREADS_ACTIVE.store(enabled, Ordering::SeqCst); + if enabled { + ALL_THREADS_ALLOCS.store(0, Ordering::SeqCst); + } + } + let _ = enabled; +} +/// Describe counted allocations on any thread (Windows), for failure messages. +fn all_thread_report() -> String { + #[cfg(windows)] + { + use std::sync::atomic::Ordering::SeqCst; + // SAFETY: GetCurrentThreadId takes no arguments and cannot fail. + let current = unsafe { windows_sys::Win32::System::Threading::GetCurrentThreadId() }; + let n = ALL_THREADS_ALLOCS.load(SeqCst).min(ALL_THREADS_SEEN.len()); + let seen: Vec<_> = ALL_THREADS_SEEN[..n] + .iter() + .map(|s| { + let thread = s[0].load(SeqCst); + format!( + "thread {thread}{} size {} align {}", + if thread == current as usize { + " (test)" + } else { + "" + }, + s[1].load(SeqCst), + s[2].load(SeqCst) + ) + }) + .collect(); + format!("first counted allocations: [{}]", seen.join(", ")) + } + #[cfg(not(windows))] + { + String::new() + } +} +fn all_thread_allocations() -> usize { + #[cfg(windows)] + { + ALL_THREADS_ALLOCS.load(std::sync::atomic::Ordering::SeqCst) + } + #[cfg(not(windows))] + { + 0 + } +} +fn fs_wait(l: &mut Loop, out: &mut Completions, op: OpId) -> FsResult { + let until = l.now() + Duration::from_secs(10); + loop { + assert!(l.now() < until, "filesystem allocation subject stalled"); + l.turn(Timeout::Until(until), out).expect("turn"); + if out.is_empty() { + continue; + } + assert_eq!(out.len(), 1); + let c = out.drain().next().expect("completion"); + assert_eq!(c.op, Some(op)); + match c.result { + OpResult::Fs(result) => return result, + other => panic!("unexpected {other:?}"), + } + } +} + +#[cfg(any( + unix, + windows, + all( + target_os = "wasi", + any( + target_env = "p2", + all(target_env = "p3", feature = "wasi-p3-experimental") + ) + ) +))] +#[test] +fn steady_typed_file_requests_allocate_nothing() { + let dir = fs_root("requests"); + let file = dir.join("subject"); + std::fs::write(&file, [3u8; 64]).expect("fixture"); + let mut l = Loop::new(Config::default()).expect("loop"); + let mut out = Completions::with_capacity(1); + let path = FsPath::new(&file).expect("prepared path"); + let op = l + .fs( + FsRequest::Open { + path: path.clone(), + options: FileOptions { + read: true, + write: true, + ..FileOptions::default() + }, + }, + Token(1), + ) + .expect("open"); + let FsResult::Opened(h) = fs_wait(&mut l, &mut out, op) else { + panic!("open") + }; + static INPUT: [u8; 64] = [9; 64]; + let mut output = [0u8; 64]; + let mut counts = [0usize; 6]; + for round in 0..201 { + if round == 1 { + let calibration = std::hint::black_box(Box::new([0u8; 32])); + ALLOCS.with(|n| n.set(0)); + ACTIVE.with(|v| v.set(true)); + drop(std::hint::black_box(calibration)); + let probe = std::hint::black_box(vec![1u8; 8]); + assert_eq!(ALLOCS.with(Cell::get), 1, "allocation counter is live"); + drop(probe); + ALLOCS.with(|n| n.set(0)); + count_all_threads(true); + } + // SAFETY: static immutable input outlives the request. + let buffer = WriteBuf::Provided(unsafe { IoBuf::from_raw_parts(INPUT.as_ptr(), 64) }); + let op = l + .fs( + FsRequest::Write { + file: h, + buffer, + offset: Some(0), + }, + Token(2), + ) + .expect("write"); + assert!(matches!(fs_wait(&mut l, &mut out, op), FsResult::Wrote(64))); + // SAFETY: `output` stays in place and is inspected only after completion. + let buffer = + ReadBuf::Provided(unsafe { IoBufMut::from_raw_parts(output.as_mut_ptr(), 64) }); + let op = l + .fs( + FsRequest::Read { + file: h, + buffer, + offset: Some(0), + }, + Token(3), + ) + .expect("read"); + assert!(matches!( + fs_wait(&mut l, &mut out, op), + FsResult::Read { n: 64, lease: None } + )); + assert_eq!(output, INPUT); + let op = l + .fs( + FsRequest::Read { + file: h, + buffer: ReadBuf::Pooled, + offset: Some(32), + }, + Token(4), + ) + .expect("pooled read"); + let FsResult::Read { + n: 32, + lease: Some(lease), + } = fs_wait(&mut l, &mut out, op) + else { + panic!("pooled read") + }; + assert_eq!(lease.as_slice(), &INPUT[..32]); + drop(lease); + let op = l.fs(FsRequest::Fstat { file: h }, Token(5)).expect("fstat"); + assert!(matches!( + fs_wait(&mut l, &mut out, op), + FsResult::Metadata(m) if m.size == 64 + )); + let op = l + .fs( + FsRequest::Stat { + path: path.clone(), + follow_symlinks: true, + }, + Token(6), + ) + .expect("stat"); + assert!(matches!( + fs_wait(&mut l, &mut out, op), + FsResult::Metadata(m) if m.size == 64 + )); + let op = l + .fs(FsRequest::Truncate { file: h, size: 64 }, Token(7)) + .expect("truncate"); + assert!(matches!(fs_wait(&mut l, &mut out, op), FsResult::Done)); + for (i, count) in counts.iter_mut().enumerate() { + *count += usize::from(round != 0 && i < 6); + } + } + ACTIVE.with(|v| v.set(false)); + count_all_threads(false); + assert_eq!( + ALLOCS.with(Cell::get), + 0, + "typed write/read/pooled read/fstat/stat/truncate allocations" + ); + let report = all_thread_report(); + assert_eq!( + all_thread_allocations(), + 0, + "pool-thread allocations for the same requests; {report}" + ); + assert_eq!(counts, [200; 6], "every subject executed"); + l.close(h, Token(8)).expect("close"); + l.turn(Timeout::Now, &mut out).expect("closed"); + assert!(matches!(out[0].result, OpResult::Closed)); + std::fs::remove_dir_all(&dir).expect("cleanup"); +} + +/// Append one byte through a new handle, flush and close it, without allocating. +#[cfg(windows)] +fn append_and_close(path: &[u16]) { + use windows_sys::Win32::{ + Foundation::{CloseHandle, INVALID_HANDLE_VALUE}, + Storage::FileSystem::{ + CreateFileW, FILE_APPEND_DATA, FILE_SHARE_DELETE, FILE_SHARE_READ, FILE_SHARE_WRITE, + FlushFileBuffers, OPEN_EXISTING, WriteFile, + }, + }; + // SAFETY: NUL-terminated wide path; default security, no template. + let handle = unsafe { + CreateFileW( + path.as_ptr(), + FILE_APPEND_DATA, + FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, + std::ptr::null(), + OPEN_EXISTING, + 0, + std::ptr::null_mut(), + ) + }; + assert_ne!(handle, INVALID_HANDLE_VALUE, "open subject"); + let mut written = 0; + // SAFETY: live handle and one initialized byte; synchronous write. + let ok = unsafe { WriteFile(handle, b"x".as_ptr(), 1, &mut written, std::ptr::null_mut()) }; + assert!(ok != 0 && written == 1, "append subject"); + // SAFETY: live handle owned by this function. + assert_ne!(unsafe { FlushFileBuffers(handle) }, 0, "flush subject"); + // SAFETY: closes the handle opened above exactly once. + assert_ne!(unsafe { CloseHandle(handle) }, 0, "close subject"); +} + +#[cfg(any(unix, windows))] +#[test] +fn steady_watch_batches_allocate_nothing() { + #[cfg(unix)] + use std::io::Write; + let dir = fs_root("watch"); + let subject_path = dir.join("subject"); + let subject = std::fs::File::create(&subject_path).expect("fixture"); + #[cfg(windows)] + let subject_wide: Vec = { + use std::os::windows::ffi::OsStrExt; + subject_path.as_os_str().encode_wide().chain([0]).collect() + }; + let mut l = Loop::new(Config::default()).expect("loop"); + let watch = l + .fs_watch( + &FsPath::new(&dir).expect("path"), + WatchOptions::default(), + Token(1), + ) + .expect("watch"); + let mut out = Completions::with_capacity(4); + let mut batches = 0; + let mut records = 0; + for round in 0..101 { + if round == 1 { + ALLOCS.with(|n| n.set(0)); + ACTIVE.with(|v| v.set(true)); + count_all_threads(true); + } + // Append through a fresh descriptor and close it. macOS FSEvents reports + // the content change on close, and NTFS updates the directory entry that + // ReadDirectoryChangesW observes only when a handle is flushed or closed: + // writes through a long-lived handle produce no notification there. + // std opens short Unix paths from a stack buffer; on Windows std converts + // every path to UTF-16 on the heap, so the handle is opened directly. + #[cfg(unix)] + { + let mut writer = std::fs::OpenOptions::new() + .append(true) + .open(&subject_path) + .expect("reopen"); + writer.write_all(b"x").expect("event"); + } + #[cfg(windows)] + append_and_close(&subject_wide); + let until = l.now() + Duration::from_secs(10); + loop { + assert!( + l.now() < until, + "watch allocation subject stalled in round {round}" + ); + l.turn(Timeout::Until(until), &mut out).expect("turn"); + let mut got = false; + for c in out.drain() { + let OpResult::Watch { events, .. } = c.result else { + panic!("unexpected {:?}", c.result) + }; + let n = WatchEvents::new(events.as_slice()).count(); + assert!(n > 0); + if round != 0 { + records += n; + batches += 1; + } + got = true; + } + if got { + break; + } + } + } + ACTIVE.with(|v| v.set(false)); + count_all_threads(false); + assert_eq!(ALLOCS.with(Cell::get), 0, "watch delivery allocations"); + assert_eq!( + all_thread_allocations(), + 0, + "watch allocations on any thread; {}", + all_thread_report() + ); + assert!( + batches >= 100 && records >= 100, + "{batches} batches, {records} records" + ); + l.close(watch, Token(2)).expect("close watch"); + let until = l.now() + Duration::from_secs(10); + while l.alive() { + assert!(l.now() < until); + l.turn(Timeout::Until(until), &mut out).expect("close turn"); + } + drop(subject); + std::fs::remove_dir_all(&dir).expect("cleanup"); +} + #[cfg(windows)] #[test] fn windows_sync_pipe_character_and_disk_workers_reuse_classification_without_allocating() { diff --git a/crates/turnloop-contract/tests/filesystem.rs b/crates/turnloop-contract/tests/filesystem.rs new file mode 100644 index 0000000..0452672 --- /dev/null +++ b/crates/turnloop-contract/tests/filesystem.rs @@ -0,0 +1,386 @@ +//! Native typed filesystem requests and watches (DESIGN D8, §7.6 Files). +#![cfg(all(not(loom), any(unix, windows)))] +use std::{ + path::PathBuf, + sync::{ + Arc, + atomic::{AtomicBool, AtomicUsize, Ordering}, + }, + time::Duration, +}; +use turnloop::{backend::Platform, *}; +use turnloop_contract::filesystem as contract; + +fn root(name: &str) -> PathBuf { + let dir = std::env::temp_dir().join(format!("turnloop-fs-{}-{name}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("test root"); + dir +} + +#[test] +fn bytes_metadata_namespace() { + contract::bytes_metadata_namespace::(&root("namespace")); +} +#[test] +fn errors() { + contract::errors::(&root("errors")); +} +#[test] +fn fifo_cancel_close() { + contract::fifo_cancel_close::(&root("cancel")); +} +#[test] +fn queued_posts_do_not_starve_requests() { + contract::queued_posts_do_not_starve_requests::(&root("starvation")); +} +#[test] +fn pooled_lease_wait() { + contract::pooled_lease_wait::(&root("leases")); +} +#[test] +fn watch_directory() { + contract::watch_directory::(&root("watch-dir")); +} +#[test] +fn watch_file() { + contract::watch_file::(&root("watch-file")); +} +#[test] +fn watch_backpressure() { + contract::watch_backpressure::(&root("watch-pressure")); +} +#[test] +fn watch_recursive() { + contract::watch_recursive::(&root("watch-tree")); +} + +/// Hold every shared pool thread so queued requests provably have not started. +struct Hold { + released: Arc, + jobs: Vec, +} +impl Hold { + fn new(l: &mut Loop) -> Self { + let entered = Arc::new(AtomicUsize::new(0)); + let released = Arc::new(AtomicBool::new(false)); + let threads = Config::default().blocking_pool.threads; + let jobs = (0..threads) + .map(|_| { + let (entered, released) = (entered.clone(), released.clone()); + l.blocking( + move || { + entered.fetch_add(1, Ordering::Release); + while !released.load(Ordering::Acquire) { + std::thread::park_timeout(Duration::from_millis(1)); + } + Ok(Payload::U64(1)) + }, + Token(1000), + ) + .expect("pool job") + }) + .collect(); + let until = std::time::Instant::now() + Duration::from_secs(10); + while entered.load(Ordering::Acquire) != threads { + assert!(std::time::Instant::now() < until, "pool threads started"); + std::thread::yield_now(); + } + Self { released, jobs } + } +} +impl Drop for Hold { + fn drop(&mut self) { + self.released.store(true, Ordering::Release); + } +} + +/// Queue exhaustion rejects before acceptance; every accepted request completes once. +#[test] +fn pool_backpressure_rejects_before_acceptance() { + let dir = root("backpressure"); + let mut l = Loop::new(Config::default()).expect("loop"); + let hold = Hold::new(&mut l); + let target = FsPath::new(&dir).expect("path"); + let capacity = Config::default().blocking_pool.queue_capacity; + let mut accepted = Vec::new(); + let rejected = loop { + match l.fs( + FsRequest::Stat { + path: target.clone(), + follow_symlinks: true, + }, + Token(accepted.len() as u64), + ) { + Ok(op) => accepted.push(op), + Err(e) => break e, + } + assert!(accepted.len() <= capacity, "queue bound enforced"); + }; + assert_eq!(rejected.kind, ErrorKind::ResourceLimit); + assert_eq!( + accepted.len(), + capacity, + "every reserved queue slot was usable" + ); + assert!( + l.blocking(|| Ok(Payload::U64(2)), Token(2000)).is_err(), + "reservations also bound host jobs" + ); + let jobs = hold.jobs.clone(); + drop(hold); + let mut out = Completions::default(); + let mut seen = vec![false; capacity]; + let mut blocking = 0; + let until = l.now() + Duration::from_secs(30); + while seen.iter().any(|s| !s) || blocking < jobs.len() { + assert!(l.now() < until); + l.turn(Timeout::Until(until), &mut out).expect("turn"); + for c in out.drain() { + match c.result { + OpResult::Fs(FsResult::Metadata(m)) => { + assert_eq!(m.kind, FileType::Directory); + let i = c.token.0 as usize; + assert_eq!(c.op, Some(accepted[i])); + assert!(!seen[i], "exactly once"); + seen[i] = true; + } + OpResult::Blocking(Payload::U64(1)) => blocking += 1, + other => panic!("unexpected {other:?}"), + } + } + } + assert!(!l.alive()); + // Capacity is available again. + let op = l + .fs( + FsRequest::Stat { + path: target, + follow_symlinks: true, + }, + Token(1), + ) + .expect("released reservations"); + assert!(contract::wait(&mut l, op).is_ok()); + std::fs::remove_dir_all(&dir).expect("cleanup"); +} + +/// Cancelling an unstarted request completes it at once, releases its queue +/// reservation and starts its FIFO successor; the started head reports Cancelled. +#[test] +fn cancellation_withdraws_unstarted_and_reports_started() { + let dir = root("withdraw"); + let file = FsPath::new(dir.join("file")).expect("path"); + std::fs::write(file.as_path(), b"0123456789").expect("fixture"); + let mut l = Loop::new(Config::default()).expect("loop"); + let op = l + .fs( + FsRequest::Open { + path: file.clone(), + options: FileOptions::default(), + }, + Token(1), + ) + .expect("open"); + let Ok(FsResult::Opened(h)) = contract::wait(&mut l, op) else { + panic!("open") + }; + let hold = Hold::new(&mut l); + let mut head_bytes = [0u8; 4]; + let mut middle_bytes = [0xee; 4]; + let mut tail_bytes = [0u8; 4]; + let read = |bytes: &mut [u8], offset| FsRequest::Read { + file: h, + // SAFETY: each array outlives its request and is read only after completion. + buffer: ReadBuf::Provided(unsafe { + IoBufMut::from_raw_parts(bytes.as_mut_ptr(), bytes.len()) + }), + offset: Some(offset), + }; + let head = l.fs(read(&mut head_bytes, 0), Token(2)).expect("head"); + let middle = l.fs(read(&mut middle_bytes, 4), Token(3)).expect("middle"); + let tail = l.fs(read(&mut tail_bytes, 6), Token(4)).expect("tail"); + assert!(l.cancel(middle), "unstarted request cancels"); + assert!(!l.cancel(middle), "cancellation is idempotent"); + assert!(l.cancel(head), "queued head cancels"); + let mut out = Completions::default(); + l.turn(Timeout::Now, &mut out).expect("withdrawn results"); + let mut results: Vec<_> = out.drain().map(|c| (c.op, c.result)).collect(); + // The head was handed to the (held) pool; only the middle request is withdrawn. + assert_eq!(results.len(), 1, "{results:?}"); + let (op, result) = results.pop().expect("middle"); + assert_eq!(op, Some(middle)); + assert!(matches!(result, OpResult::Cancelled)); + drop(hold); + let until = l.now() + Duration::from_secs(10); + let mut order = Vec::new(); + while order.len() < 6 { + assert!(l.now() < until); + l.turn(Timeout::Until(until), &mut out).expect("turn"); + for c in out.drain() { + match c.result { + OpResult::Blocking(_) => order.push(None), + OpResult::Cancelled => order.push(Some(c.op.expect("head"))), + OpResult::Fs(FsResult::Read { n: 4, .. }) => { + assert_eq!(c.op, Some(tail)); + order.push(c.op); + } + other => panic!("unexpected {other:?}"), + } + } + } + let requests: Vec<_> = order.into_iter().flatten().collect(); + assert_eq!( + requests, + [head, tail], + "FIFO successor runs after the cancelled head" + ); + assert_eq!( + middle_bytes, [0xee; 4], + "withdrawn request never touched its buffer" + ); + assert_eq!(&tail_bytes, b"6789"); + l.close(h, Token(5)).expect("close"); + l.turn(Timeout::Now, &mut out).expect("closed"); + assert!(matches!(out[0].result, OpResult::Closed)); + assert!(!l.alive()); + std::fs::remove_dir_all(&dir).expect("cleanup"); +} + +/// Permission errors are real OS denials (requires an unprivileged runner). +#[test] +fn permission_denied_is_reported() { + let dir = root("permission"); + let file = dir.join("locked"); + std::fs::write(&file, b"secret").expect("fixture"); + let mut l = Loop::new(contract::config()).expect("loop"); + let locked = FsPath::new(&file).expect("path"); + let r = contract::run( + &mut l, + FsRequest::Chmod { + target: FsTarget::Path { + path: locked.clone(), + follow_symlinks: true, + }, + mode: 0o444, + }, + ); + assert!(matches!(r, Ok(FsResult::Done)), "{r:?}"); + let e = contract::run( + &mut l, + FsRequest::Open { + path: locked.clone(), + options: FileOptions { + read: false, + write: true, + ..FileOptions::default() + }, + }, + ) + .expect_err("read-only file rejects writers"); + assert_eq!(e.kind, ErrorKind::PermissionDenied, "{e:?}"); + assert!(e.os.is_some()); + let e = contract::run( + &mut l, + FsRequest::Access { + path: locked.clone(), + mode: AccessMode { + write: true, + ..AccessMode::default() + }, + }, + ) + .expect_err("access reports the denial"); + assert_eq!(e.kind, ErrorKind::PermissionDenied, "{e:?}"); + let r = contract::run( + &mut l, + FsRequest::Chmod { + target: FsTarget::Path { + path: locked, + follow_symlinks: true, + }, + mode: 0o644, + }, + ); + assert!(matches!(r, Ok(FsResult::Done)), "{r:?}"); + assert!(!l.alive()); + std::fs::remove_dir_all(&dir).expect("cleanup"); +} + +/// A cancelled open exposes no handle, and dropping a loop withdraws requests +/// still queued on the shared pool: their buffers are never accessed afterwards. +#[test] +fn cancelled_open_and_loop_drop_never_touch_buffers() { + let dir = root("drop"); + let file = FsPath::new(dir.join("file")).expect("path"); + std::fs::write(file.as_path(), b"0123456789").expect("fixture"); + let mut l = Loop::new(Config::default()).expect("loop"); + let op = l + .fs( + FsRequest::Open { + path: file.clone(), + options: FileOptions::default(), + }, + Token(1), + ) + .expect("open"); + let Ok(FsResult::Opened(h)) = contract::wait(&mut l, op) else { + panic!("open") + }; + let hold = Hold::new(&mut l); + let hidden = l + .fs( + FsRequest::Open { + path: file.clone(), + options: FileOptions::default(), + }, + Token(2), + ) + .expect("queued open"); + assert!(l.cancel(hidden)); + let mut head = [0x11u8; 8]; + let mut queued = [0x22u8; 8]; + let read = |bytes: &mut [u8]| FsRequest::Read { + file: h, + // SAFETY: both arrays outlive the loop, which is dropped below. + buffer: ReadBuf::Provided(unsafe { + IoBufMut::from_raw_parts(bytes.as_mut_ptr(), bytes.len()) + }), + offset: Some(0), + }; + l.fs(read(&mut head), Token(3)).expect("head read"); + l.fs(read(&mut queued), Token(4)).expect("queued read"); + drop(l); + head.fill(0x33); + queued.fill(0x44); + let jobs = hold.jobs.len(); + drop(hold); + // Give the released pool threads time to dequeue the withdrawn job. + let mut probe = Loop::new(Config::default()).expect("probe loop"); + let mut out = Completions::default(); + let op = probe + .fs( + FsRequest::Stat { + path: file.clone(), + follow_symlinks: true, + }, + Token(5), + ) + .expect("probe request queues behind the withdrawn job"); + let until = probe.now() + Duration::from_secs(10); + let mut done = false; + while !done { + assert!(probe.now() < until); + probe.turn(Timeout::Until(until), &mut out).expect("turn"); + for c in out.drain() { + assert_eq!(c.op, Some(op)); + assert!(matches!(c.result, OpResult::Fs(FsResult::Metadata(_)))); + done = true; + } + } + assert_eq!(jobs, Config::default().blocking_pool.threads); + assert_eq!(head, [0x33; 8], "withdrawn head never ran after drop"); + assert_eq!(queued, [0x44; 8], "queued successor never ran after drop"); + drop(probe); + std::fs::remove_dir_all(&dir).expect("cleanup"); +} diff --git a/crates/turnloop-contract/tests/wasi.rs b/crates/turnloop-contract/tests/wasi.rs index ff491aa..9447326 100644 --- a/crates/turnloop-contract/tests/wasi.rs +++ b/crates/turnloop-contract/tests/wasi.rs @@ -423,3 +423,39 @@ fn stdio_streams_preserve_host_ownership() { actual.len() ); } + +/// Filesystem contracts over the `/turnloop-fs` preopen (scripts/ci/wasmtime-runner.sh). +mod filesystem { + use std::path::Path; + use turnloop::backend::Platform; + use turnloop_contract::filesystem as contract; + fn root() -> &'static Path { + let root = Path::new("/turnloop-fs"); + assert!(root.is_dir(), "the runner must preopen /turnloop-fs"); + root + } + #[test] + fn bytes_metadata_namespace() { + contract::bytes_metadata_namespace::(root()); + } + #[test] + fn errors() { + contract::errors::(root()); + } + #[test] + fn fifo_cancel_close() { + contract::fifo_cancel_close::(root()); + } + #[test] + fn pooled_lease_wait() { + contract::pooled_lease_wait::(root()); + } + #[test] + fn queued_posts_do_not_starve_requests() { + contract::queued_posts_do_not_starve_requests::(root()); + } + #[test] + fn capability_scope_and_unsupported_surface() { + contract::capability_scope::(root()); + } +} diff --git a/crates/turnloop/src/backend/epoll.rs b/crates/turnloop/src/backend/epoll.rs index e1b1060..ad57722 100644 --- a/crates/turnloop/src/backend/epoll.rs +++ b/crates/turnloop/src/backend/epoll.rs @@ -259,6 +259,7 @@ impl Poller for Epoll { & (libc::EPOLLIN | libc::EPOLLHUP | libc::EPOLLRDHUP | libc::EPOLLERR) != 0, write: flags & (libc::EPOLLOUT | libc::EPOLLHUP | libc::EPOLLERR) != 0, + vnode: 0, }); } } diff --git a/crates/turnloop/src/backend/fsevents.rs b/crates/turnloop/src/backend/fsevents.rs new file mode 100644 index 0000000..6e53f8a --- /dev/null +++ b/crates/turnloop/src/backend/fsevents.rs @@ -0,0 +1,382 @@ +//! FSEvents streams for macOS directory watches. +//! +//! CoreServices is loaded on first use (no launch-time framework dependency). +//! Streams deliver on one private serial dispatch queue per loop. A callback +//! parses events into the stream's bounded storage and publishes the handle to +//! this loop's inbox, notifying it; stopping a stream waits for the queue, so no +//! callback runs after `stop` returns. +use crate::{Error, ErrorKind, FsPath, Handle, Notifier, Result, WatchKind, fs::watch::Ring}; +use std::{ + collections::VecDeque, + ffi::{CStr, CString, c_char, c_void}, + os::unix::ffi::OsStrExt, + sync::{ + Arc, Mutex, MutexGuard, OnceLock, + atomic::{AtomicBool, Ordering}, + }, +}; + +type Ref = *const c_void; +type Callback = unsafe extern "C" fn(Ref, *mut c_void, usize, *mut c_void, *const u32, *const u64); +type StringFn = unsafe extern "C" fn(Ref, *const c_char) -> Ref; +type ArrayFn = unsafe extern "C" fn(Ref, *const Ref, isize, Ref) -> Ref; +type ReleaseFn = unsafe extern "C" fn(Ref); +type CreateFn = + unsafe extern "C" fn(Ref, Callback, *const StreamContext, Ref, u64, f64, u32) -> Ref; +type QueueFn = unsafe extern "C" fn(Ref, Ref); +type StartFn = unsafe extern "C" fn(Ref) -> u8; +#[repr(C)] +struct StreamContext { + version: isize, + info: *mut c_void, + retain: Option Ref>, + release: Option, + description: Option Ref>, +} +struct Api { + string: StringFn, + array: ArrayFn, + cf_release: ReleaseFn, + create: CreateFn, + set_queue: QueueFn, + start: StartFn, + stop: ReleaseFn, + invalidate: ReleaseFn, + release: ReleaseFn, +} +// SAFETY: the table holds immutable function pointers of thread-safe framework APIs. +unsafe impl Send for Api {} +// SAFETY: see Send; the pointers are never mutated after loading. +unsafe impl Sync for Api {} +unsafe extern "C" { + fn dispatch_queue_create(label: *const c_char, attr: Ref) -> Ref; + fn dispatch_sync_f(queue: Ref, context: *mut c_void, work: unsafe extern "C" fn(*mut c_void)); + fn dispatch_release(object: Ref); + fn dispatch_retain(object: Ref); +} +const SINCE_NOW: u64 = u64::MAX; +const NO_DEFER: u32 = 0x02; +const WATCH_ROOT: u32 = 0x04; +const FILE_EVENTS: u32 = 0x10; +const MUST_SCAN_SUBDIRS: u32 = 0x01; +const USER_DROPPED: u32 = 0x02; +const KERNEL_DROPPED: u32 = 0x04; +const ROOT_CHANGED: u32 = 0x20; +const RENAMED: u32 = 0x100 | 0x200 | 0x800; // created, removed, renamed +const MODIFIED: u32 = 0x400 | 0x1000 | 0x2000 | 0x4000 | 0x8000; +const IS_DIRECTORY: u32 = 0x20000; + +fn api() -> Result<&'static Api> { + static API: OnceLock> = OnceLock::new(); + API.get_or_init(|| { + let open = |path: &CStr| { + // SAFETY: NUL-terminated system framework path; the image stays loaded. + let image = unsafe { libc::dlopen(path.as_ptr(), libc::RTLD_LAZY | libc::RTLD_LOCAL) }; + (!image.is_null()).then_some(image) + }; + let core = open(c"/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation")?; + let services = open(c"/System/Library/Frameworks/CoreServices.framework/CoreServices")?; + macro_rules! symbol { + ($image:expr, $name:literal, $type:ty) => {{ + // SAFETY: NUL-terminated symbol name in a loaded image. + let pointer = unsafe { libc::dlsym($image, $name.as_ptr()) }; + if pointer.is_null() { + return None; + } + // SAFETY: the named framework function has exactly the declared C ABI. + unsafe { std::mem::transmute::<*mut c_void, $type>(pointer) } + }}; + } + Some(Api { + string: symbol!( + core, + c"CFStringCreateWithFileSystemRepresentation", + StringFn + ), + array: symbol!(core, c"CFArrayCreate", ArrayFn), + cf_release: symbol!(core, c"CFRelease", ReleaseFn), + create: symbol!(services, c"FSEventStreamCreate", CreateFn), + set_queue: symbol!(services, c"FSEventStreamSetDispatchQueue", QueueFn), + start: symbol!(services, c"FSEventStreamStart", StartFn), + stop: symbol!(services, c"FSEventStreamStop", ReleaseFn), + invalidate: symbol!(services, c"FSEventStreamInvalidate", ReleaseFn), + release: symbol!(services, c"FSEventStreamRelease", ReleaseFn), + }) + }) + .as_ref() + .ok_or(Error::new(ErrorKind::Unsupported)) +} + +fn lock(m: &Mutex) -> MutexGuard<'_, T> { + m.lock().unwrap_or_else(std::sync::PoisonError::into_inner) +} + +/// Handles with published records, shared by every stream of one loop. +struct Inbox { + pending: AtomicBool, + ready: Mutex>, + notifier: Mutex>, +} +struct Context { + inbox: Arc, + ring: Mutex, + queued: AtomicBool, + handle: Handle, + root: CString, + name: Box<[u8]>, + recursive: bool, +} + +unsafe extern "C" fn receive( + _stream: Ref, + info: *mut c_void, + count: usize, + paths: *mut c_void, + flags: *const u32, + _ids: *const u64, +) { + // SAFETY: the context outlives the stream (freed only after invalidation and + // a queue barrier); FSEvents passes `count` C-string paths and flags. + let (context, paths, flags) = unsafe { + ( + &*info.cast::(), + std::slice::from_raw_parts(paths.cast::<*const c_char>(), count), + std::slice::from_raw_parts(flags, count), + ) + }; + let root = context.root.as_bytes(); + let mut ring = lock(&context.ring); + let mut published = false; + for (&path, &flags) in paths.iter().zip(flags) { + if flags & (MUST_SCAN_SUBDIRS | USER_DROPPED | KERNEL_DROPPED) != 0 { + ring.lost(); + published = true; + } + if flags & ROOT_CHANGED != 0 { + ring.push(WatchKind::Rename, &context.name); + published = true; + continue; + } + // SAFETY: FSEvents paths are NUL-terminated for the callback's duration. + let path = unsafe { CStr::from_ptr(path) }.to_bytes(); + let Some(rest) = path.strip_prefix(root) else { + continue; + }; + let mut flags = flags; + // Event naming follows libuv's FSEvents backend. + let name = if rest.is_empty() { + flags &= !RENAMED; + &context.name[..] + } else if root == b"/" { + rest + } else if let Some(child) = rest.strip_prefix(b"/") { + child + } else { + continue; + }; + if !context.recursive && !rest.is_empty() && name.contains(&b'/') { + continue; + } + let kind = if flags & RENAMED == 0 && (flags & MODIFIED != 0 || flags & IS_DIRECTORY == 0) { + WatchKind::Change + } else { + WatchKind::Rename + }; + ring.push(kind, name); + published = true; + } + drop(ring); + if published && !context.queued.swap(true, Ordering::AcqRel) { + let mut ready = lock(&context.inbox.ready); + ready.push_back(context.handle); + context.inbox.pending.store(true, Ordering::Release); + drop(ready); + if let Some(notifier) = lock(&context.inbox.notifier).as_ref() { + let _ = notifier.notify(); + } + } +} +unsafe extern "C" fn barrier(_: *mut c_void) {} + +pub(super) struct Stream { + stream: Ref, + paths: Ref, + string: Ref, + /// Retained callback queue, used by Drop to wait for in-flight callbacks. + queue: Ref, + started: bool, + context: Box, +} +impl Stream { + /// Accept the next publication before this loop inspects the records. + pub fn rearm(&self) { + self.context.queued.store(false, Ordering::Release); + } + /// Access the records shared with the callback queue. + pub fn with_ring(&self, f: impl FnOnce(&mut Ring) -> R) -> R { + f(&mut lock(&self.context.ring)) + } +} +// SAFETY: every exit path runs this Drop, which stops delivery before freeing the +// context: a loop dropped with a live watch cannot leave a callback behind. +impl Drop for Stream { + fn drop(&mut self) { + let Ok(api) = api() else { + return; + }; + // SAFETY: each object is owned by this stream and released exactly once. + // A started stream is stopped and invalidated, then the loop thread (never + // running on the private queue, so no deadlock) waits for the queue: no + // callback can observe the context after this block. + unsafe { + if self.started { + (api.stop)(self.stream); + (api.invalidate)(self.stream); + } + if !self.queue.is_null() { + dispatch_sync_f(self.queue, std::ptr::null_mut(), barrier); + dispatch_release(self.queue); + } + if !self.stream.is_null() { + (api.release)(self.stream); + } + if !self.paths.is_null() { + (api.cf_release)(self.paths); + } + if !self.string.is_null() { + (api.cf_release)(self.string); + } + } + } +} + +pub(super) struct Streams { + inbox: Arc, + queue: Ref, +} +impl Streams { + pub fn new(capacity: usize) -> Self { + let inbox = Arc::new(Inbox { + pending: AtomicBool::new(false), + ready: Mutex::new(VecDeque::with_capacity(capacity * 2)), + notifier: Mutex::new(None), + }); + drop(lock(&inbox.ready)); + drop(lock(&inbox.notifier)); + Self { + inbox, + queue: std::ptr::null(), + } + } + pub fn set_notifier(&mut self, notifier: Notifier) { + *lock(&self.inbox.notifier) = Some(notifier); + } + pub fn has_work(&self) -> bool { + self.inbox.pending.load(Ordering::Acquire) + } + pub fn drain(&mut self, mut ready: impl FnMut(Handle)) { + if !self.has_work() { + return; + } + let mut inbox = lock(&self.inbox.ready); + self.inbox.pending.store(false, Ordering::Release); + while let Some(h) = inbox.pop_front() { + ready(h); + } + } + pub fn start(&mut self, h: Handle, path: &FsPath, recursive: bool) -> Result { + let api = api()?; + let root = std::fs::canonicalize(path.as_path()).map_err(Error::from)?; + let root = CString::new(root.as_os_str().as_bytes()) + .map_err(|_| Error::new(ErrorKind::InvalidInput))?; + let name = root + .as_bytes() + .rsplit(|&b| b == b'/') + .find(|part| !part.is_empty()) + .unwrap_or(b"/") + .into(); + if self.queue.is_null() { + // SAFETY: static label; a null attribute creates a serial queue. + self.queue = + unsafe { dispatch_queue_create(c"turnloop.fsevents".as_ptr(), std::ptr::null()) }; + if self.queue.is_null() { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + } + let context = Box::new(Context { + inbox: self.inbox.clone(), + ring: Mutex::new(Ring::new()), + queued: AtomicBool::new(false), + handle: h, + root, + name, + recursive, + }); + drop(lock(&context.ring)); + // SAFETY: live queue; the stream holds its own reference until Drop. + unsafe { dispatch_retain(self.queue) }; + let mut stream = Stream { + stream: std::ptr::null(), + paths: std::ptr::null(), + string: std::ptr::null(), + queue: self.queue, + started: false, + context, + }; + // SAFETY: NUL-terminated canonical path; released by Stream::drop. + stream.string = unsafe { (api.string)(std::ptr::null(), stream.context.root.as_ptr()) }; + if stream.string.is_null() { + return Err(Error::new(ErrorKind::InvalidInput)); + } + // SAFETY: one live CFString; the stream copies the array at creation. + stream.paths = + unsafe { (api.array)(std::ptr::null(), &stream.string, 1, std::ptr::null()) }; + if stream.paths.is_null() { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + let info = StreamContext { + version: 0, + info: (&*stream.context as *const Context).cast_mut().cast(), + retain: None, + release: None, + description: None, + }; + // SAFETY: valid callback, context and path array. Zero latency with NoDefer + // delivers without a driver-side interval; WatchRoot reports root moves. + stream.stream = unsafe { + (api.create)( + std::ptr::null(), + receive, + &info, + stream.paths, + SINCE_NOW, + 0.0, + NO_DEFER | WATCH_ROOT | FILE_EVENTS, + ) + }; + if stream.stream.is_null() { + return Err(Error::new(ErrorKind::Other)); + } + // SAFETY: new, unscheduled stream and a live serial queue. + unsafe { (api.set_queue)(stream.stream, self.queue) }; + stream.started = true; + // SAFETY: scheduled stream whose boxed context stays pinned until Drop. + if unsafe { (api.start)(stream.stream) } == 0 { + return Err(Error::new(ErrorKind::Other)); + } + Ok(stream) + } + /// Stop delivery, wait for in-flight callbacks and free the stream. + pub fn stop(&mut self, h: Handle, stream: Stream) { + drop(stream); + lock(&self.inbox.ready).retain(|&queued| queued != h); + } +} +impl Drop for Streams { + fn drop(&mut self) { + if !self.queue.is_null() { + // SAFETY: this reference was created by dispatch_queue_create; streams hold their own. + unsafe { dispatch_release(self.queue) }; + } + } +} diff --git a/crates/turnloop/src/backend/iocp/mod.rs b/crates/turnloop/src/backend/iocp/mod.rs index 5195aa7..b2dbe51 100644 --- a/crates/turnloop/src/backend/iocp/mod.rs +++ b/crates/turnloop/src/backend/iocp/mod.rs @@ -11,6 +11,7 @@ mod signals; mod socket; mod sync_io; mod timer; +mod watch; use crate::{ backend::{Backend, Event, Operation, Outcome, PollInfo, Request}, @@ -193,6 +194,7 @@ pub struct Iocp { deadline: Option, failure: Option, next_listener_key: usize, + watches: watch::Watches, } impl Iocp { #[cfg(test)] @@ -921,6 +923,9 @@ impl Iocp { && entry.key < port::STOP && !(base..end).contains(&entry.overlapped) { + if self.watches.completed(entry)? { + return Ok(()); + } // Listener keys are never reused in this loop. Cancellation packets // can arrive after release; ignore those without dereferencing them. for resource in self.resources.iter_mut().flatten() { @@ -994,6 +999,7 @@ unsafe impl Backend for Iocp { .iter() .map(|slot| bridge::Bridge::new(Arc::clone(&port), slot.get().cast()).map(Some)) .collect::>>()?; + let watches = watch::Watches::new(config, pool.clone(), Arc::clone(&port)); Ok(Self { resources: (0..config.max_handles).map(|_| None).collect(), ops: (0..config.max_operations).map(|_| None).collect(), @@ -1016,11 +1022,28 @@ unsafe impl Backend for Iocp { deadline: None, failure: None, next_listener_key: pipes::FIRST_KEY, + watches, }) } fn set_notifier(&mut self, notifier: Notifier) { self.notifier = Some(notifier); } + fn fs_watch(&mut self, handle: Handle, path: &FsPath, options: WatchOptions) -> Result<()> { + if self + .resources + .get(handle.index()) + .is_none_or(Option::is_some) + || self.services.contains(handle) + { + return Err(invalid()); + } + let key = self.next_listener_key; + self.next_listener_key = key + .checked_add(1) + .filter(|key| *key < port::STOP) + .ok_or(Error::new(ErrorKind::ResourceLimit))?; + self.watches.start(handle, path, options.recursive, key) + } fn spawn( &mut self, handle: Handle, @@ -1266,6 +1289,9 @@ unsafe impl Backend for Iocp { if self.services.contains(request.handle) { return self.services.submit(request); } + if self.watches.contains(request.handle) { + return self.watches.submit(&request); + } let r = self.get(request.handle)?; let kind = r.transport.kind; let valid = match &request.operation { @@ -1328,7 +1354,7 @@ unsafe impl Backend for Iocp { Ok(()) } fn cancel(&mut self, op: OpId) -> Result<()> { - if self.services.cancel(op) { + if self.services.cancel(op) || self.watches.cancel(op)? { return Ok(()); } let Some(p) = self @@ -1365,6 +1391,7 @@ unsafe impl Backend for Iocp { self.failure.is_some() || !self.ready.is_empty() || self.services.has_work() + || self.watches.has_work() || (self.pool.available() && !self.pool_waiting.is_empty()) } fn poll( @@ -1380,6 +1407,7 @@ unsafe impl Backend for Iocp { } let before = events.len(); self.services.collect(events)?; + self.watches.collect(events); self.run_ready(events); if events.len() != before || events.len() == events.capacity() || !self.ready.is_empty() { return Ok(PollInfo::default()); @@ -1420,11 +1448,13 @@ unsafe impl Backend for Iocp { cancel_result?; entries_result?; self.services.collect(events)?; + self.watches.collect(events); self.run_ready(events); Ok(info) } fn release(&mut self, h: Handle) { self.services.release(h); + self.watches.release(h); if self.get(h).is_ok() { self.workers[h.index()] = None; self.resources[h.index()] = None; @@ -1516,8 +1546,9 @@ impl Drop for Iocp { } } } + self.watches.shutdown(); let mut events = Vec::with_capacity(64); - while self.ops.iter().any(Option::is_some) { + while self.ops.iter().any(Option::is_some) || self.watches.pending() { if self.poll(None, &mut events).is_err() { std::process::abort(); } diff --git a/crates/turnloop/src/backend/iocp/watch.rs b/crates/turnloop/src/backend/iocp/watch.rs new file mode 100644 index 0000000..b8e4140 --- /dev/null +++ b/crates/turnloop/src/backend/iocp/watch.rs @@ -0,0 +1,465 @@ +//! ReadDirectoryChangesW watches completed through the loop's own IOCP. +//! +//! Each watch owns a directory handle associated with the port under a unique, +//! never-reused key, plus pinned OVERLAPPED and notification storage. A request +//! is re-armed after each completion, so the kernel buffers changes while the host +//! holds leases. A watched file is served by its parent directory, filtered by name. +//! Storage is freed only after the kernel acknowledged the last request. +use super::{ + Detached, os_error, + port::{Entry, Port}, +}; +use crate::{ + backend::{Event, Operation, Outcome, Request}, + fs::watch::Ring, + *, +}; +use std::{ + cell::UnsafeCell, + collections::VecDeque, + os::windows::io::{AsRawHandle, FromRawHandle, OwnedHandle}, + ptr, + sync::Arc, +}; +use windows_sys::Win32::{ + Foundation::{ERROR_NOT_FOUND, INVALID_HANDLE_VALUE, RtlNtStatusToDosError}, + Storage::FileSystem::*, + System::IO::{CancelIoEx, OVERLAPPED}, +}; + +/// Notification storage in DWORD-aligned words (64 KiB, the network-share limit). +const WORDS: usize = 64 * 1024 / 8; +const FILTER: u32 = FILE_NOTIFY_CHANGE_FILE_NAME + | FILE_NOTIFY_CHANGE_DIR_NAME + | FILE_NOTIFY_CHANGE_ATTRIBUTES + | FILE_NOTIFY_CHANGE_SIZE + | FILE_NOTIFY_CHANGE_LAST_WRITE + | FILE_NOTIFY_CHANGE_LAST_ACCESS + | FILE_NOTIFY_CHANGE_CREATION + | FILE_NOTIFY_CHANGE_SECURITY; + +struct Watch { + handle: Handle, + key: usize, + directory: OwnedHandle, + kernel: Box>, + buffer: Box>, + recursive: bool, + /// For a watched file: its UTF-16 name within the watched parent directory. + file: Option>, + /// Final component of the watched path, as WTF-8. + name: Box<[u8]>, + op: Option, + armed: bool, + cancelling: bool, + queued: bool, + ring: Ring, +} +impl Watch { + fn arm(&mut self) -> Result<()> { + debug_assert!(!self.armed); + // SAFETY: the previous request was dequeued (or none was issued), so the + // pinned OVERLAPPED is inactive and may be reset. + unsafe { *self.kernel.get() = std::mem::zeroed() }; + // SAFETY: live overlapped directory handle; buffer and OVERLAPPED are + // boxed, never move, and are freed only after this request's packet. + let ok = unsafe { + ReadDirectoryChangesW( + self.directory.as_raw_handle(), + self.buffer.get().cast(), + (WORDS * 8) as u32, + i32::from(self.recursive), + FILTER, + ptr::null_mut(), + self.kernel.get(), + None, + ) + }; + if ok == 0 { + return Err(os_error()); + } + self.armed = true; + Ok(()) + } + fn cancel_native(&self) -> Result<()> { + if !self.armed { + return Ok(()); + } + // SAFETY: exact live request; its acknowledgement packet is still required. + if unsafe { CancelIoEx(self.directory.as_raw_handle(), self.kernel.get()) } == 0 { + let error = os_error(); + if error.os != Some(ERROR_NOT_FOUND as i32) { + return Err(error); + } + } + Ok(()) + } + fn parse(&mut self, n: usize) { + if n == 0 { + // The kernel buffer overflowed and its contents were discarded. + self.ring.lost(); + return; + } + // SAFETY: the completed request initialized `n` bytes of the pinned buffer, + // which no kernel request accesses until the next arm. + let bytes = unsafe { std::slice::from_raw_parts(self.buffer.get().cast::(), n) }; + let mut at = 0; + loop { + if at + 12 > bytes.len() { + self.ring.lost(); + return; + } + let word = |i: usize| u32::from_le_bytes(bytes[i..i + 4].try_into().expect("word")); + let (next, action, length) = (word(at) as usize, word(at + 4), word(at + 8) as usize); + if at + 12 + length > bytes.len() { + self.ring.lost(); + return; + } + // SAFETY: records are DWORD-aligned in u64 storage, so the name is + // u16-aligned; its bytes lie within the initialized region checked above. + let units = unsafe { + std::slice::from_raw_parts(bytes.as_ptr().add(at + 12).cast::(), length / 2) + }; + let kind = if action == FILE_ACTION_MODIFIED { + WatchKind::Change + } else { + WatchKind::Rename + }; + match &self.file { + Some(file) if same_name(units, file) => self.ring.push(kind, &self.name), + Some(_) => {} + None => self.ring.push_wide(kind, units), + } + if next == 0 { + return; + } + at += next; + } + } +} +fn same_name(a: &[u16], b: &[u16]) -> bool { + let fold = |c: u16| { + if (u16::from(b'a')..=u16::from(b'z')).contains(&c) { + c - 32 + } else { + c + } + }; + a.len() == b.len() && a.iter().zip(b).all(|(&x, &y)| fold(x) == fold(y)) +} + +pub(super) struct Watches { + entries: Vec>, + active: Vec, + /// Released watches whose last request awaits its acknowledgement. + retired: Vec, + ops: Vec>, + ready: VecDeque, + finished: VecDeque<(OpId, Result<()>)>, + pool: BufferPool, + port: Arc, +} +impl Watches { + pub fn new(config: &Config, pool: BufferPool, port: Arc) -> Self { + Self { + entries: (0..config.max_handles).map(|_| None).collect(), + active: Vec::with_capacity(config.max_handles), + retired: Vec::with_capacity(config.max_handles), + ops: vec![None; config.max_operations], + ready: VecDeque::with_capacity(config.max_handles), + finished: VecDeque::with_capacity(config.max_operations), + pool, + port, + } + } + fn get_mut(&mut self, h: Handle) -> Option<&mut Watch> { + self.entries + .get_mut(h.index()) + .and_then(Option::as_mut) + .filter(|w| w.handle == h) + } + pub fn contains(&self, h: Handle) -> bool { + self.entries + .get(h.index()) + .and_then(Option::as_ref) + .is_some_and(|w| w.handle == h) + } + pub fn start(&mut self, h: Handle, path: &FsPath, recursive: bool, key: usize) -> Result<()> { + if self.entries.get(h.index()).is_none_or(Option::is_some) { + return Err(Error::new(ErrorKind::InvalidInput)); + } + let wide = &path.wide()[..path.wide().len() - 1]; + let end = wide + .iter() + .rposition(|&c| c != 0x5c && c != 0x2f) + .map_or(0, |i| i + 1); + let trimmed = &wide[..end]; + let split = trimmed.iter().rposition(|&c| c == 0x5c || c == 0x2f); + let base = match split { + Some(i) => &trimmed[i + 1..], + None => trimmed, + }; + let mut name = vec![0u8; base.len() * 3].into_boxed_slice(); + let mut used = 0; + crate::fs::wtf8(base, &mut name, &mut used); + let name: Box<[u8]> = name[..used].into(); + // SAFETY: NUL-terminated prepared path. + let attributes = unsafe { GetFileAttributesW(path.wide().as_ptr()) }; + if attributes == INVALID_FILE_ATTRIBUTES { + return Err(os_error()); + } + let (directory, file) = if attributes & FILE_ATTRIBUTE_DIRECTORY != 0 { + (path.wide().to_vec(), None) + } else { + if recursive { + return Err(Error::new(ErrorKind::Unsupported)); + } + let mut parent = match split { + Some(0) => trimmed[..1].to_vec(), + Some(i) => trimmed[..i].to_vec(), + None => vec![u16::from(b'.')], + }; + parent.push(0); + (parent, Some(base.into())) + }; + // SAFETY: NUL-terminated directory path; the new handle is owned below. + let raw = unsafe { + CreateFileW( + directory.as_ptr(), + FILE_LIST_DIRECTORY, + FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, + ptr::null(), + OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, + ptr::null_mut(), + ) + }; + if raw == INVALID_HANDLE_VALUE { + return Err(os_error()); + } + // SAFETY: CreateFileW returned a new handle owned by nobody else. + let directory = unsafe { OwnedHandle::from_raw_handle(raw) }; + // SAFETY: fresh, quiescent overlapped handle and a never-reused key. + unsafe { self.port.associate(directory.as_raw_handle(), key) }?; + let mut watch = Watch { + handle: h, + key, + directory, + // SAFETY: an all-zero OVERLAPPED is the inactive state. + kernel: Box::new(UnsafeCell::new(unsafe { std::mem::zeroed() })), + buffer: Box::new(UnsafeCell::new([0; WORDS])), + recursive, + file, + name, + op: None, + armed: false, + cancelling: false, + queued: false, + ring: Ring::new(), + }; + watch.arm()?; + self.entries[h.index()] = Some(watch); + self.active.push(h.index()); + Ok(()) + } + pub fn submit(&mut self, request: &Request) -> Result<()> { + let op = request.op; + if !matches!(request.operation, Operation::WatchFs) + || self.ops.get(op.index()).is_none_or(Option::is_some) + { + return Err(Error::new(ErrorKind::InvalidInput)); + } + let watch = self + .get_mut(request.handle) + .filter(|w| w.op.is_none()) + .ok_or(Error::new(ErrorKind::InvalidInput))?; + watch.op = Some(op); + let queue = !watch.queued && !watch.ring.is_empty(); + watch.queued |= queue; + self.ops[op.index()] = Some(request.handle); + if queue { + self.ready.push_back(request.handle); + } + Ok(()) + } + /// Begin cancellation; the terminal result follows the kernel acknowledgement. + pub fn cancel(&mut self, op: OpId) -> Result { + let Some(h) = self.ops.get(op.index()).copied().flatten() else { + return Ok(false); + }; + let Some(watch) = self.get_mut(h).filter(|w| w.op == Some(op)) else { + return Ok(false); + }; + if watch.cancelling { + return Ok(true); + } + watch.cancel_native()?; + watch.cancelling = true; + watch.ring.clear(); + if !watch.armed { + self.finish(h); + } + Ok(true) + } + /// Queue the Cancelled terminal of a cancelling watch whose request is quiescent. + fn finish(&mut self, h: Handle) { + let Some(watch) = self.get_mut(h) else { return }; + debug_assert!(watch.cancelling && !watch.armed); + if let Some(op) = watch.op.take() { + self.ops[op.index()] = None; + self.finished.push_back((op, Ok(()))); + } + } + /// Apply a port packet addressed to a watch key. Returns false for other keys. + pub fn completed(&mut self, entry: &Entry) -> Result { + if let Some(at) = self.retired.iter().position(|w| w.key == entry.key) { + if self.retired[at].kernel.get() as usize != entry.overlapped { + return Err(super::invalid()); + } + // The last request of a released watch: storage may now be freed. + drop(self.retired.swap_remove(at)); + return Ok(true); + } + let Some(&i) = self + .active + .iter() + .find(|&&i| self.entries[i].as_ref().is_some_and(|w| w.key == entry.key)) + else { + return Ok(false); + }; + let watch = self.entries[i].as_mut().expect("active watch"); + if !watch.armed || watch.kernel.get() as usize != entry.overlapped { + return Err(super::invalid()); + } + watch.armed = false; + let h = watch.handle; + if watch.cancelling { + // Cancellation acknowledged (or raced with a completion): report Cancelled. + self.finish(h); + return Ok(true); + } + if entry.status < 0 { + // Includes an unrequested STATUS_CANCELLED: the watch cannot continue. + // SAFETY: pure NTSTATUS conversion with no pointers. + let code = unsafe { RtlNtStatusToDosError(entry.status) }; + let error: Error = std::io::Error::from_raw_os_error(code as i32).into(); + if let Some(op) = watch.op.take() { + self.ops[op.index()] = None; + self.finished.push_back((op, Err(error))); + } + return Ok(true); + } + watch.parse(entry.bytes as usize); + if let Err(error) = watch.arm() { + if let Some(op) = watch.op.take() { + self.ops[op.index()] = None; + self.finished.push_back((op, Err(error))); + } + return Ok(true); + } + if watch.op.is_some() && !watch.queued && !watch.ring.is_empty() { + watch.queued = true; + self.ready.push_back(h); + } + Ok(true) + } + pub fn has_work(&self) -> bool { + !self.finished.is_empty() || (!self.ready.is_empty() && self.pool.available()) + } + pub fn collect(&mut self, events: &mut Vec>) { + while events.len() < events.capacity() { + let Some((op, result)) = self.finished.pop_front() else { + break; + }; + events.push(Event { + op, + terminal: true, + result: result.map(|()| Outcome::Cancelled), + }); + } + for _ in 0..self.ready.len() { + if events.len() == events.capacity() { + break; + } + let Some(h) = self.ready.pop_front() else { + break; + }; + let Some(watch) = self.get_mut(h) else { + continue; + }; + let Some(op) = watch.op.filter(|_| !watch.cancelling) else { + watch.queued = false; + continue; + }; + if watch.ring.is_empty() { + watch.queued = false; + continue; + } + let Some(mut lease) = self.pool.acquire() else { + self.ready.push_front(h); + break; + }; + let watch = self.get_mut(h).expect("ready watch"); + let overflow = watch.ring.take(&mut lease); + if watch.ring.is_empty() { + watch.queued = false; + } else { + self.ready.push_back(h); + } + events.push(Event { + op, + terminal: false, + result: Ok(Outcome::Watch { + events: lease, + overflow, + }), + }); + } + } + /// Release after Closed (or a failed start/submit); pending I/O retires the storage. + pub fn release(&mut self, h: Handle) { + let Some(watch) = self.entries.get_mut(h.index()).and_then(|slot| { + if slot.as_ref().is_some_and(|w| w.handle == h) { + slot.take() + } else { + None + } + }) else { + return; + }; + if let Some(at) = self.active.iter().position(|&i| i == h.index()) { + self.active.swap_remove(at); + } + if watch.queued { + self.ready.retain(|&queued| queued != h); + } + if let Some(op) = watch.op { + self.ops[op.index()] = None; + } + if watch.armed { + if watch.cancel_native().is_err() { + // Storage must outlive a request that could not be cancelled. + std::process::abort(); + } + self.retired.push(watch); + } + } + /// Cancel every request before the backend drains its port on drop. + pub fn shutdown(&mut self) { + for &i in &self.active { + let watch = self.entries[i].as_mut().expect("active watch"); + if watch.cancel_native().is_err() { + std::process::abort(); + } + watch.cancelling = true; + } + } + /// Whether any kernel request still references watch storage. + pub fn pending(&self) -> bool { + !self.retired.is_empty() + || self + .active + .iter() + .any(|&i| self.entries[i].as_ref().is_some_and(|w| w.armed)) + } +} diff --git a/crates/turnloop/src/backend/kqueue.rs b/crates/turnloop/src/backend/kqueue.rs index fdb39b5..c2cd026 100644 --- a/crates/turnloop/src/backend/kqueue.rs +++ b/crates/turnloop/src/backend/kqueue.rs @@ -220,6 +220,11 @@ impl Poller for Kqueue { key: e.udata as usize as u64, read: e.filter == libc::EVFILT_READ, write: e.filter == libc::EVFILT_WRITE, + vnode: if e.filter == libc::EVFILT_VNODE { + e.fflags + } else { + 0 + }, }); } Ok(PollInfo::native(timeout, n == 0)) diff --git a/crates/turnloop/src/backend/mod.rs b/crates/turnloop/src/backend/mod.rs index 44bbcdb..1f86321 100644 --- a/crates/turnloop/src/backend/mod.rs +++ b/crates/turnloop/src/backend/mod.rs @@ -75,6 +75,22 @@ //! it as a flag/host scheduler token. No backend creates a loop-driving thread. //! Windows' explicitly requested Integration::Event helper is the D7 exception. //! * Unsupported platforms/capabilities return errors; no fake successful I/O. +//! +//! # Filesystem (§7.6 Files) +//! +//! * `FILESYSTEM` selects where typed `FsRequest`s run. `Pool` (native default): +//! the core runs them on the shared blocking pool and never calls the backend. +//! `Backend` (WASI): `fs` accepts them under the ordinary Request rules, with a +//! FIFO per handle, and yields `Outcome::Fs`. A request that opens binds its +//! native object to the supplied new handle only on success; after a failed or +//! cancelled open the core calls `release` for that never-visible handle. +//! * `fs_watch` binds a native watch to a new handle; the core then submits one +//! multishot `Operation::WatchFs`. Each nonterminal `Outcome::Watch` carries a +//! pooled lease of records (see `WatchEvents`). Pool exhaustion retains events +//! in bounded per-watch storage; lost events set `overflow`. No backend thread +//! is created: native watch sources are registered with the loop's own wait, +//! except where the OS delivers on its own threads (FSEvents dispatch queues). +pub use crate::fs::FsOutput; use crate::{ BufLease, BufferPool, Config, Error, Handle, Integration, OpId, Open, ReadBuf, Result, WriteBuf, WriteVectored, @@ -141,6 +157,8 @@ pub enum Operation { RecvFrom(ReadBuf), /// Complete a stream write-side shutdown after preceding queued writes. Shutdown, + /// Multishot delivery of the filesystem watch bound by `Backend::fs_watch`. + WatchFs, } #[derive(Debug)] /// Backend completion awaiting translation into a host-visible completion. @@ -198,9 +216,33 @@ pub enum Outcome { }, /// Complete a stream write-side shutdown after preceding queued writes. Shutdown, + /// A batch of filesystem watch records. + Watch { + /// Initialized `WatchEvents` records. + events: BufLease, + /// Native events were lost before this batch; rescan the watched scope. + overflow: bool, + }, + /// A typed filesystem result from a `Filesystem::Backend` backend. + Fs { + /// The typed result; `Opened` refers to the request's new handle. + output: FsOutput, + /// The pooled lease holding result bytes, when one was requested. + lease: Option, + }, /// The operation was cancelled and its native buffer access has ended. Cancelled, } +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +/// Where typed filesystem requests execute (see the module documentation). +pub enum Filesystem { + /// The core runs them on the shared native blocking pool. + Pool, + /// The backend accepts them through `Backend::fs`. + Backend, + /// No filesystem: `Loop::fs` returns Unsupported. + Unsupported, +} #[derive(Clone, Copy, Debug, Default)] /// Instrumentation for the single bounded backend wait. pub struct PollInfo { @@ -243,8 +285,28 @@ pub unsafe trait Backend: Sized + 'static { type Wake: Wake; /// Owning transferable representation of an unregistered resource. type Detached: Send + 'static; + /// Where typed filesystem requests run for this backend. + const FILESYSTEM: Filesystem = if cfg!(target_arch = "wasm32") { + Filesystem::Unsupported + } else { + Filesystem::Pool + }; /// Construct backend storage and native wait resources with the supplied shared read pool. fn new(config: &Config, pool: BufferPool) -> Result; + /// Accept a typed filesystem request (only with `Filesystem::Backend`). `handle` + /// is the request's file handle, or the new handle for an open. + fn fs(&mut self, _op: OpId, _handle: Option, _request: crate::FsRequest) -> Result<()> { + Err(Error::new(crate::ErrorKind::Unsupported)) + } + /// Bind a native filesystem watch to a new handle, retaining nothing on error. + fn fs_watch( + &mut self, + _handle: Handle, + _path: &crate::FsPath, + _options: crate::WatchOptions, + ) -> Result<()> { + Err(Error::new(crate::ErrorKind::Unsupported)) + } /// Give process-wide services this loop's parking-aware notification endpoint. fn set_notifier(&mut self, _notifier: crate::Notifier) {} /// Spawn and bind the child and requested parent pipe handles atomically. @@ -349,6 +411,11 @@ pub mod iocp; #[cfg(turnloop_backend = "iocp")] pub use iocp::Iocp as Platform; +#[cfg(any( + turnloop_backend = "wasi_p2", + all(turnloop_backend = "wasi_p3", feature = "wasi-p3-experimental") +))] +mod wasi_fs; #[cfg(turnloop_backend = "wasi_p2")] pub mod wasi_p2; #[cfg(turnloop_backend = "wasi_p2")] @@ -373,3 +440,5 @@ mod signals; #[cfg(any(turnloop_backend = "kqueue", turnloop_backend = "epoll"))] mod files; +#[cfg(any(turnloop_backend = "kqueue", turnloop_backend = "epoll"))] +mod watch; diff --git a/crates/turnloop/src/backend/poller.rs b/crates/turnloop/src/backend/poller.rs index 20dd0e7..7ae9bf2 100644 --- a/crates/turnloop/src/backend/poller.rs +++ b/crates/turnloop/src/backend/poller.rs @@ -8,6 +8,9 @@ pub(crate) struct Ready { pub key: u64, pub read: bool, pub write: bool, + /// kqueue EVFILT_VNODE flags for a filesystem watch; zero otherwise. + #[cfg_attr(turnloop_backend = "epoll", allow(dead_code))] + pub vnode: u32, } pub(crate) trait Poller: Sized { type W: Wake; diff --git a/crates/turnloop/src/backend/unix.rs b/crates/turnloop/src/backend/unix.rs index 824f69a..26d8e2f 100644 --- a/crates/turnloop/src/backend/unix.rs +++ b/crates/turnloop/src/backend/unix.rs @@ -109,6 +109,7 @@ pub struct Unix { pool: BufferPool, services: super::services::Services, files: super::files::Files, + watches: super::watch::Watches, } fn direction(op: &Operation) -> usize { usize::from(!matches!( @@ -257,17 +258,25 @@ unsafe impl Backend for Unix { cancelled: VecDeque::with_capacity(config.max_operations), polled: Vec::with_capacity(config.events_per_turn), files: super::files::Files::new(config, pool.clone()), + watches: super::watch::Watches::new(config, pool.clone()), pool, services: super::services::Services::new(config), }) } fn set_notifier(&mut self, notifier: Notifier) { + self.watches.set_notifier(notifier.clone()); self.files.set_notifier(notifier.clone()); self.services.set_notifier(notifier); } fn signal(&mut self, h: Handle, signal: Signal) -> Result<()> { self.services.signal(h, signal) } + fn fs_watch(&mut self, h: Handle, path: &FsPath, options: WatchOptions) -> Result<()> { + if self.resources.get(h.index()).is_none_or(Option::is_some) || self.services.contains(h) { + return Err(Error::new(ErrorKind::InvalidInput)); + } + self.watches.start(h, path, options, &mut self.poller) + } fn prepare_close(&mut self, h: Handle) -> Result<()> { self.services.prepare_close(h) } @@ -488,6 +497,9 @@ unsafe impl Backend for Unix { if self.services.contains(h) { return self.services.submit(&request); } + if self.watches.contains(h) { + return self.watches.submit(&request); + } let r = self.get(h)?; if r.transport.kind == Kind::File { let fd = r.transport.fd.try_clone().map_err(Error::from)?; @@ -554,6 +566,9 @@ unsafe impl Backend for Unix { if self.services.cancel(op) { return Ok(()); } + if self.watches.cancel(op) { + return Ok(()); + } let p = self .ops .get(op.index()) @@ -572,6 +587,7 @@ unsafe impl Backend for Unix { || !self.cancelled.is_empty() || self.services.has_work() || self.files.has_work() + || self.watches.has_work() } fn poll( &mut self, @@ -591,6 +607,7 @@ unsafe impl Backend for Unix { } self.files.poll(events); self.services.poll(events); + self.watches.poll(events); self.run_ready(events); // Cached readiness can end in EAGAIN without producing a completion. // In that case use this turn's single OS wait with its exact timeout; @@ -602,7 +619,7 @@ unsafe impl Backend for Unix { let info = self.poller.wait(timeout, &mut self.polled)?; for i in 0..self.polled.len() { let e = self.polled[i]; - if self.services.ready(e.key) { + if self.services.ready(e.key) || self.watches.ready(e) { continue; } let Some(r) = self @@ -622,11 +639,13 @@ unsafe impl Backend for Unix { } self.files.poll(events); self.services.poll(events); + self.watches.poll(events); self.run_ready(events); Ok(info) } fn release(&mut self, h: Handle) { self.services.release(h, &mut self.poller); + self.watches.release(h, &mut self.poller); if let Ok(r) = self.get(h) { if r.transport.kind != Kind::File { let fd = r.transport.fd.as_raw_fd(); @@ -667,7 +686,9 @@ fn execute( ) -> Result, bool)>> { let fd = r.transport.fd.as_raw_fd(); match &mut p.request.operation { - Operation::ProcessExit | Operation::WatchSignal => Err(Error::new(ErrorKind::InvalidInput)), + Operation::ProcessExit | Operation::WatchSignal | Operation::WatchFs => { + Err(Error::new(ErrorKind::InvalidInput)) + } Operation::Connect => { if !r.connecting { let a = r diff --git a/crates/turnloop/src/backend/wasi_fs.rs b/crates/turnloop/src/backend/wasi_fs.rs new file mode 100644 index 0000000..d95a84e --- /dev/null +++ b/crates/turnloop/src/backend/wasi_fs.rs @@ -0,0 +1,599 @@ +//! Capability-scoped `wasi:filesystem` requests shared by the WASI 0.2 and 0.3 +//! backends. A WASI agent has no threads, so the blocking pool is replaced by +//! bounded execution inside `poll`: each turn runs at most its event budget of +//! requests, then waits as usual. Requests on one handle run FIFO. Cancelling a +//! request that has not run completes it at once without touching its buffer. +//! +//! Paths resolve only against preopened directories: an absolute path uses the +//! preopen with the longest matching name, a relative path the preopen named `.`. +use crate::{ + backend::{Event, Outcome}, + fs::{ + FileMetadata, FileOptions, FileType, FsOutput, FsRequest, FsTarget, TimeChange, put_record, + }, + *, +}; +use std::collections::VecDeque; + +/// Binding surface of one `wasi:filesystem` version. Calls may block the agent. +pub(super) trait Api { + type Descriptor; + type Appender; + type Entries; + fn preopens() -> Vec<(Self::Descriptor, String)>; + fn open_at( + dir: &Self::Descriptor, + path: &str, + options: &FileOptions, + directory: bool, + ) -> Result; + /// Positional read into `output`: (bytes, end of file). + fn read(file: &Self::Descriptor, output: &mut [u8], offset: u64) -> Result<(usize, bool)>; + fn write(file: &Self::Descriptor, bytes: &[u8], offset: u64) -> Result; + fn append( + file: &Self::Descriptor, + appender: &mut Option, + bytes: &[u8], + ) -> Result; + fn stat(file: &Self::Descriptor) -> Result; + fn stat_at(dir: &Self::Descriptor, path: &str, follow: bool) -> Result; + fn sync(file: &Self::Descriptor, data_only: bool) -> Result<()>; + fn set_size(file: &Self::Descriptor, size: u64) -> Result<()>; + fn set_times(file: &Self::Descriptor, accessed: TimeChange, modified: TimeChange) + -> Result<()>; + fn set_times_at( + dir: &Self::Descriptor, + path: &str, + follow: bool, + accessed: TimeChange, + modified: TimeChange, + ) -> Result<()>; + fn entries(dir: &Self::Descriptor) -> Result; + fn next_entry(entries: &mut Self::Entries) -> Result>; + fn create_directory_at(dir: &Self::Descriptor, path: &str) -> Result<()>; + fn remove_directory_at(dir: &Self::Descriptor, path: &str) -> Result<()>; + fn unlink_file_at(dir: &Self::Descriptor, path: &str) -> Result<()>; + fn rename_at( + dir: &Self::Descriptor, + path: &str, + new_dir: &Self::Descriptor, + new_path: &str, + ) -> Result<()>; + fn link_at( + dir: &Self::Descriptor, + path: &str, + new_dir: &Self::Descriptor, + new_path: &str, + ) -> Result<()>; + fn symlink_at(dir: &Self::Descriptor, target: &str, path: &str) -> Result<()>; + fn readlink_at(dir: &Self::Descriptor, path: &str) -> Result; +} + +/// wasi-libc errno values (identical in WASI 0.2 and 0.3 error-code naming). +pub(super) mod errno { + pub const EACCES: i32 = 2; + pub const EAGAIN: i32 = 6; + pub const EBADF: i32 = 8; + pub const EEXIST: i32 = 20; + pub const EINVAL: i32 = 28; + pub const EISDIR: i32 = 31; + pub const ENOENT: i32 = 44; + pub const ENOMEM: i32 = 48; + pub const ENOSPC: i32 = 51; + pub const ENOTDIR: i32 = 54; + pub const ENOTEMPTY: i32 = 55; + pub const ENOTSUP: i32 = 58; + pub const EPERM: i32 = 63; +} +/// Portable kind for a WASI errno, retaining the errno as the native code. +pub(super) fn error(code: i32) -> Error { + use errno::*; + let kind = match code { + EACCES | EPERM => ErrorKind::PermissionDenied, + EEXIST => ErrorKind::AlreadyExists, + ENOENT => ErrorKind::NotFound, + ENOTDIR => ErrorKind::NotADirectory, + EISDIR => ErrorKind::IsADirectory, + ENOTEMPTY => ErrorKind::DirectoryNotEmpty, + EINVAL => ErrorKind::InvalidInput, + ENOTSUP => ErrorKind::Unsupported, + EAGAIN => ErrorKind::WouldBlock, + ENOMEM | ENOSPC => ErrorKind::ResourceLimit, + _ => ErrorKind::Other, + }; + Error { + kind, + os: Some(code), + } +} + +enum Object { + File { + // Streams are children of their descriptor and must drop first. + appender: Option, + descriptor: A::Descriptor, + cursor: u64, + append: bool, + }, + Dir { + entries: Option, + descriptor: A::Descriptor, + pending: Option<(FileType, String)>, + done: bool, + }, + Closed, +} +struct Pending { + op: OpId, + handle: Option, + request: Option, + previous: Option, + next: Option, + waiting: bool, +} +pub(super) struct Files { + preopens: Option>, + objects: Vec>>, + ops: Vec>, + heads: Vec>, + tails: Vec>, + ready: VecDeque, + waiting: VecDeque, + cancelled: VecDeque, + pool: BufferPool, +} +impl Files { + pub fn new(config: &Config, pool: BufferPool) -> Self { + Self { + preopens: None, + objects: (0..config.max_handles).map(|_| None).collect(), + ops: (0..config.max_operations).map(|_| None).collect(), + heads: vec![None; config.max_handles], + tails: vec![None; config.max_handles], + ready: VecDeque::with_capacity(config.max_operations), + waiting: VecDeque::with_capacity(config.max_operations), + cancelled: VecDeque::with_capacity(config.max_operations), + pool, + } + } + pub fn submit(&mut self, op: OpId, handle: Option, request: FsRequest) -> Result<()> { + match &request { + // No wasi:filesystem equivalent exists; reject before acceptance. + FsRequest::RealPath { .. } | FsRequest::Chmod { .. } | FsRequest::Chown { .. } => { + return Err(Error::new(ErrorKind::Unsupported)); + } + FsRequest::Access { mode, .. } if *mode != AccessMode::default() => { + return Err(Error::new(ErrorKind::Unsupported)); + } + _ => {} + } + let i = op.index(); + if self.ops.get(i).is_none_or(Option::is_some) { + return Err(Error::new(ErrorKind::InvalidInput)); + } + if let Some(h) = handle { + let object = self + .objects + .get(h.index()) + .ok_or(Error::new(ErrorKind::InvalidInput))?; + if request.opens() != object.is_none() { + return Err(Error::new(ErrorKind::InvalidInput)); + } + } + let previous = handle.and_then(|h| self.tails[h.index()]); + self.ops[i] = Some(Pending { + op, + handle, + request: Some(request), + previous, + next: None, + waiting: false, + }); + if let Some(h) = handle { + match previous { + Some(tail) => self.ops[tail].as_mut().expect("FIFO tail").next = Some(i), + None => self.heads[h.index()] = Some(i), + } + self.tails[h.index()] = Some(i); + } + if previous.is_none() { + self.ready.push_back(i); + } + Ok(()) + } + fn unlink(&mut self, i: usize) { + let p = self.ops[i].as_mut().expect("linked request"); + let (previous, next, handle) = (p.previous.take(), p.next.take(), p.handle); + let Some(h) = handle else { return }; + match previous { + Some(q) => self.ops[q].as_mut().expect("predecessor").next = next, + None => self.heads[h.index()] = next, + } + match next { + Some(n) => { + self.ops[n].as_mut().expect("successor").previous = previous; + if previous.is_none() { + self.ready.push_back(n); + } + } + None => self.tails[h.index()] = previous, + } + } + pub fn cancel(&mut self, op: OpId) -> bool { + let i = op.index(); + let Some(p) = self + .ops + .get(i) + .and_then(Option::as_ref) + .filter(|p| p.op == op) + else { + return false; + }; + let (head, waiting) = (p.previous.is_none(), p.waiting); + if head { + if waiting { + self.waiting.retain(|&w| w != i); + } else { + self.ready.retain(|&r| r != i); + } + } + self.unlink(i); + self.ops[i] = None; + self.cancelled.push_back(op); + true + } + pub fn has_work(&self) -> bool { + !self.ready.is_empty() + || !self.cancelled.is_empty() + || (!self.waiting.is_empty() && self.pool.available()) + } + pub fn release(&mut self, h: Handle) { + if let Some(object) = self.objects.get_mut(h.index()) { + *object = None; + } + } + /// Run bounded ready requests, appending their terminal events. + pub fn run(&mut self, events: &mut Vec>) { + while events.len() < events.capacity() { + let Some(op) = self.cancelled.pop_front() else { + break; + }; + events.push(Event { + op, + terminal: true, + result: Ok(Outcome::Cancelled), + }); + } + while self.pool.available() { + let Some(i) = self.waiting.pop_front() else { + break; + }; + self.ops[i].as_mut().expect("waiting request").waiting = false; + self.ready.push_back(i); + } + while events.len() < events.capacity() { + let Some(i) = self.ready.pop_front() else { + break; + }; + let p = self.ops[i].as_mut().expect("ready request"); + let request = p.request.as_mut().expect("unrun request"); + let mut lease = None; + if let Some(buffer) = request.read_buffer() + && matches!(buffer, ReadBuf::Pooled) + { + let Some(mut acquired) = self.pool.acquire() else { + p.waiting = true; + self.waiting.push_back(i); + continue; + }; + let bytes = acquired.writable(); + // SAFETY: the lease lives in this frame until the event owns it; the + // request (and this region) is consumed before the event is pushed. + *buffer = ReadBuf::Provided(unsafe { + IoBufMut::from_raw_parts(bytes.as_mut_ptr(), bytes.len()) + }); + lease = Some(acquired); + } + let (op, handle) = (p.op, p.handle); + let request = p.request.take().expect("unrun request"); + if self.preopens.is_none() && needs_preopens(&request) { + self.preopens = Some(A::preopens()); + } + let preopens = self.preopens.as_deref().unwrap_or(&[]); + let result = execute::(preopens, &mut self.objects, handle, request); + self.unlink(i); + self.ops[i] = None; + events.push(Event { + op, + terminal: true, + result: result.map(|output| Outcome::Fs { output, lease }), + }); + } + } +} +fn needs_preopens(request: &FsRequest) -> bool { + match request { + FsRequest::Chmod { target, .. } + | FsRequest::Chown { target, .. } + | FsRequest::SetTimes { target, .. } => matches!(target, FsTarget::Path { .. }), + _ => request.handle().is_none(), + } +} +fn output(buffer: &mut ReadBuf) -> &mut [u8] { + let ReadBuf::Provided(buffer) = buffer else { + unreachable!("pooled buffers are provided before a request runs") + }; + // SAFETY: an accepted request exclusively owns its output region while it runs. + unsafe { std::slice::from_raw_parts_mut(buffer.as_mut_ptr(), buffer.len()) } +} +fn resolve<'a, D>(preopens: &'a [(D, String)], path: &'a FsPath) -> Result<(&'a D, &'a str)> { + let text = path.text(); + let mut best: Option<(usize, &D, &str)> = None; + for (descriptor, name) in preopens { + let name = name.trim_end_matches('/'); + let rest = if let Some(absolute) = text.strip_prefix('/') { + if name.is_empty() { + Some(absolute) + } else if let Some(rest) = name + .strip_prefix('/') + .and_then(|n| absolute.strip_prefix(n)) + { + if rest.is_empty() { + Some("") + } else { + rest.strip_prefix('/') + } + } else { + None + } + } else if name == "." { + Some(text.strip_prefix("./").unwrap_or(text)) + } else { + None + }; + if let Some(rest) = rest + && best.is_none_or(|(length, ..)| name.len() > length) + { + best = Some((name.len(), descriptor, rest)); + } + } + let (_, descriptor, rest) = best.ok_or_else(|| error(errno::ENOENT))?; + Ok((descriptor, if rest.is_empty() { "." } else { rest })) +} +fn execute( + preopens: &[(A::Descriptor, String)], + objects: &mut [Option>], + handle: Option, + request: FsRequest, +) -> Result { + let object = handle.map(|h| &mut objects[h.index()]); + let bad = || error(errno::EBADF); + match request { + FsRequest::Open { path, options } => { + let (dir, rel) = resolve(preopens, &path)?; + let descriptor = A::open_at(dir, rel, &options, false)?; + *object.expect("open handle") = Some(Object::File { + appender: None, + descriptor, + cursor: 0, + append: options.append, + }); + Ok(FsOutput::Opened) + } + FsRequest::OpenDir { path } => { + let (dir, rel) = resolve(preopens, &path)?; + let descriptor = A::open_at(dir, rel, &FileOptions::default(), true)?; + *object.expect("open handle") = Some(Object::Dir { + entries: None, + descriptor, + pending: None, + done: false, + }); + Ok(FsOutput::Opened) + } + FsRequest::Close { .. } => { + let slot = object.expect("close handle"); + match slot { + Some(Object::File { .. } | Object::Dir { .. }) => { + *slot = Some(Object::Closed); + Ok(FsOutput::Done) + } + _ => Err(bad()), + } + } + FsRequest::Read { + mut buffer, offset, .. + } => { + let Some(Some(Object::File { + descriptor, cursor, .. + })) = object + else { + return Err(bad()); + }; + let at = offset.unwrap_or(*cursor); + let (n, _eof) = A::read(descriptor, output(&mut buffer), at)?; + if offset.is_none() { + *cursor += n as u64; + } + Ok(FsOutput::Read(n)) + } + FsRequest::Write { buffer, offset, .. } => { + let Some(Some(Object::File { + appender, + descriptor, + cursor, + append, + })) = object + else { + return Err(bad()); + }; + let bytes = buffer.as_slice(); + if *append { + return A::append(descriptor, appender, bytes).map(FsOutput::Wrote); + } + let n = A::write(descriptor, bytes, offset.unwrap_or(*cursor))?; + if offset.is_none() { + *cursor += n as u64; + } + Ok(FsOutput::Wrote(n)) + } + FsRequest::Sync { data_only, .. } => { + A::sync(descriptor(object).ok_or_else(bad)?, data_only).map(|()| FsOutput::Done) + } + FsRequest::Truncate { size, .. } => { + A::set_size(descriptor(object).ok_or_else(bad)?, size).map(|()| FsOutput::Done) + } + FsRequest::Stat { + path, + follow_symlinks, + } => { + let (dir, rel) = resolve(preopens, &path)?; + A::stat_at(dir, rel, follow_symlinks).map(FsOutput::Metadata) + } + FsRequest::Fstat { .. } => { + A::stat(descriptor(object).ok_or_else(bad)?).map(FsOutput::Metadata) + } + FsRequest::ReadDir { mut buffer, .. } => { + let Some(Some(Object::Dir { + entries, + descriptor, + pending, + done, + })) = object + else { + return Err(bad()); + }; + let out = output(&mut buffer); + let mut used = 0; + loop { + if pending.is_none() && !*done { + if entries.is_none() { + *entries = Some(A::entries(descriptor)?); + } + *pending = A::next_entry(entries.as_mut().expect("entry stream"))?; + *done = pending.is_none(); + } + let Some((kind, name)) = pending.as_ref() else { + return Ok(FsOutput::Directory { n: used, eof: true }); + }; + if name == "." || name == ".." { + *pending = None; + continue; + } + if !put_record(out, &mut used, *kind as u8, name.as_bytes()) { + if used == 0 { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + return Ok(FsOutput::Directory { + n: used, + eof: false, + }); + } + *pending = None; + } + } + FsRequest::Mkdir { path, .. } => { + let (dir, rel) = resolve(preopens, &path)?; + A::create_directory_at(dir, rel).map(|()| FsOutput::Done) + } + FsRequest::Rmdir { path } => { + let (dir, rel) = resolve(preopens, &path)?; + A::remove_directory_at(dir, rel).map(|()| FsOutput::Done) + } + FsRequest::Unlink { path } => { + let (dir, rel) = resolve(preopens, &path)?; + A::unlink_file_at(dir, rel).map(|()| FsOutput::Done) + } + FsRequest::Rename { from, to } => { + let (dir, rel) = resolve(preopens, &from)?; + let (new_dir, new_rel) = resolve(preopens, &to)?; + A::rename_at(dir, rel, new_dir, new_rel).map(|()| FsOutput::Done) + } + FsRequest::Link { existing, link } => { + let (dir, rel) = resolve(preopens, &existing)?; + let (new_dir, new_rel) = resolve(preopens, &link)?; + A::link_at(dir, rel, new_dir, new_rel).map(|()| FsOutput::Done) + } + FsRequest::Symlink { target, link, .. } => { + let (dir, rel) = resolve(preopens, &link)?; + A::symlink_at(dir, target.text(), rel).map(|()| FsOutput::Done) + } + FsRequest::ReadLink { path, mut buffer } => { + let (dir, rel) = resolve(preopens, &path)?; + let content = A::readlink_at(dir, rel)?; + let out = output(&mut buffer); + let bytes = content.as_bytes(); + if bytes.len() > out.len() { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + out[..bytes.len()].copy_from_slice(bytes); + Ok(FsOutput::Bytes(bytes.len())) + } + FsRequest::Access { path, .. } => { + let (dir, rel) = resolve(preopens, &path)?; + A::stat_at(dir, rel, true).map(|_| FsOutput::Done) + } + FsRequest::SetTimes { + target, + accessed, + modified, + } => match target { + FsTarget::Path { + path, + follow_symlinks, + } => { + let (dir, rel) = resolve(preopens, &path)?; + A::set_times_at(dir, rel, follow_symlinks, accessed, modified) + .map(|()| FsOutput::Done) + } + FsTarget::File(_) => { + A::set_times(descriptor(object).ok_or_else(bad)?, accessed, modified) + .map(|()| FsOutput::Done) + } + }, + FsRequest::CopyFile { + from, + to, + exclusive, + } => { + let (dir, rel) = resolve(preopens, &from)?; + let source = A::open_at(dir, rel, &FileOptions::default(), false)?; + let (dir, rel) = resolve(preopens, &to)?; + let target = A::open_at( + dir, + rel, + &FileOptions { + read: false, + write: true, + create: true, + exclusive, + truncate: !exclusive, + ..FileOptions::default() + }, + false, + )?; + let mut chunk = [0u8; 16 * 1024]; + let mut offset = 0; + loop { + let (n, eof) = A::read(&source, &mut chunk, offset)?; + let mut written = 0; + while written < n { + written += A::write(&target, &chunk[written..n], offset + written as u64)?; + } + offset += n as u64; + if n == 0 || eof { + return Ok(FsOutput::Done); + } + } + } + FsRequest::RealPath { .. } | FsRequest::Chmod { .. } | FsRequest::Chown { .. } => { + Err(Error::new(ErrorKind::Unsupported)) + } + } +} +fn descriptor(object: Option<&mut Option>>) -> Option<&A::Descriptor> { + match object? { + Some(Object::File { descriptor, .. } | Object::Dir { descriptor, .. }) => Some(descriptor), + _ => None, + } +} diff --git a/crates/turnloop/src/backend/wasi_p2.rs b/crates/turnloop/src/backend/wasi_p2.rs index deb5c52..0eec990 100644 --- a/crates/turnloop/src/backend/wasi_p2.rs +++ b/crates/turnloop/src/backend/wasi_p2.rs @@ -1,8 +1,9 @@ //! WASI 0.2 completion backend. One poll import per turn, reusable canonical //! lists, and synchronous nonblocking I/O with generational cancellation. mod abi; +mod fs; use crate::{ - backend::{Backend, Event, Operation, Outcome, PollInfo, Request, Wake}, + backend::{Backend, Event, Filesystem, Operation, Outcome, PollInfo, Request, Wake}, *, }; use std::{ @@ -122,6 +123,7 @@ pub struct WasiP2 { indices: Vec, poll_storage: Vec, scratch: Vec, + files: super::wasi_fs::Files, pool: BufferPool, wake: Arc, } @@ -378,6 +380,10 @@ impl WasiP2 { unsafe impl Backend for WasiP2 { type Wake = WasiWake; type Detached = Detached; + const FILESYSTEM: Filesystem = Filesystem::Backend; + fn fs(&mut self, op: OpId, handle: Option, request: FsRequest) -> Result<()> { + self.files.submit(op, handle, request) + } fn new(config: &Config, pool: BufferPool) -> Result { let polls = config .max_handles @@ -396,6 +402,7 @@ unsafe impl Backend for WasiP2 { indices: Vec::with_capacity(polls), poll_storage: vec![0; polls], scratch: vec![0; 16400], + files: super::wasi_fs::Files::new(config, pool.clone()), pool, wake: Arc::new(WasiWake), }) @@ -507,6 +514,7 @@ unsafe impl Backend for WasiP2 { request.operation, Operation::ProcessExit | Operation::WatchSignal + | Operation::WatchFs | Operation::SendHandle(_) | Operation::RecvHandle ) { @@ -572,6 +580,9 @@ unsafe impl Backend for WasiP2 { Ok(()) } fn cancel(&mut self, op: OpId) -> Result<()> { + if self.files.cancel(op) { + return Ok(()); + } if self .lookups .get(op.index()) @@ -598,6 +609,7 @@ unsafe impl Backend for WasiP2 { fn has_work(&self) -> bool { !self.ready.is_empty() || !self.cancelled.is_empty() + || self.files.has_work() || self.lookups.iter().flatten().any(|l| l.ready) } @@ -617,6 +629,7 @@ unsafe impl Backend for WasiP2 { result: Ok(Outcome::Cancelled), }); } + self.files.run(events); self.run_ready(events); self.run_dns(events); if self.has_work() || !events.is_empty() || events.len() == events.capacity() { @@ -682,6 +695,7 @@ unsafe impl Backend for WasiP2 { Ok(PollInfo::native(timeout, self.indices.is_empty())) } fn release(&mut self, h: Handle) { + self.files.release(h); if self.get(h).is_ok() { self.ready.retain(|&at| at != h); self.resources[h.index()] = None; @@ -707,6 +721,7 @@ fn execute( match &mut p.request.operation { Operation::ProcessExit | Operation::WatchSignal + | Operation::WatchFs | Operation::SendHandle(_) | Operation::RecvHandle => Err(Error::new(ErrorKind::Unsupported)), Operation::Connect => { diff --git a/crates/turnloop/src/backend/wasi_p2/abi.rs b/crates/turnloop/src/backend/wasi_p2/abi.rs index 628c448..ae8fa8d 100644 --- a/crates/turnloop/src/backend/wasi_p2/abi.rs +++ b/crates/turnloop/src/backend/wasi_p2/abi.rs @@ -20,6 +20,8 @@ struct Arena { ptr: *mut u8, len: usize, used: usize, + /// Largest alignment the storage base satisfies (4 for u32 words, 1 for bytes). + align: usize, } thread_local! { static ARENA: Cell> = const { Cell::new(None) }; } struct Reset; @@ -35,6 +37,21 @@ fn scoped(storage: &mut [u32], f: impl FnOnce() -> T) -> T { ptr: storage.as_mut_ptr().cast(), len: storage.len() * 4, used: 0, + align: 4, + })); + }); + let _reset = Reset; + f() +} +/// Byte arena over caller output: a single lowered `list` lands in place. +fn scoped_bytes(storage: &mut [u8], f: impl FnOnce() -> T) -> T { + ARENA.with(|a| { + assert!(a.get().is_none(), "nested canonical return arena"); + a.set(Some(Arena { + ptr: storage.as_mut_ptr(), + len: storage.len(), + used: 0, + align: 1, })); }); let _reset = Reset; @@ -58,7 +75,7 @@ unsafe extern "C" fn cabi_realloc( if let Some(mut arena) = ARENA.get() { // These imports lower lists once, not incrementally via realloc. assert_eq!(old_len, 0); - assert!(align <= 4); + assert!(align <= arena.align); let offset = (arena.used + align - 1) & !(align - 1); assert!( len <= arena.len.saturating_sub(offset), @@ -104,6 +121,49 @@ unsafe extern "C" { fn raw_send(handle: u32, input: *const u32, len: usize, result: *mut u64); } +#[link(wasm_import_module = "wasi:filesystem/types@0.2.9")] +unsafe extern "C" { + #[link_name = "[method]descriptor.read"] + fn raw_file_read(handle: i32, length: i64, offset: i64, result: *mut u8); +} +/// `descriptor.read` lowered directly into `output`. Returns the byte count and +/// the end-of-file flag, or the raw filesystem error-code discriminant. +pub fn file_read( + descriptor: &wasip2::filesystem::types::Descriptor, + output: &mut [u8], + offset: u64, +) -> std::result::Result<(usize, bool), u8> { + let mut result = [0u32; 4]; + let base = output.as_mut_ptr(); + let capacity = output.len(); + scoped_bytes(output, || { + // SAFETY: live descriptor, bounded length, aligned 16-byte result area. The + // only list the host lowers is the returned bytes, placed in the arena. + unsafe { + raw_file_read( + descriptor.handle() as i32, + capacity as i64, + offset as i64, + result.as_mut_ptr().cast(), + ); + } + match result[0] & 0xff { + 0 => { + let (pointer, n) = (result[1] as usize as *mut u8, result[2] as usize); + assert!(n <= capacity); + if n != 0 && pointer != base { + // SAFETY: host-initialized bytes inside the output arena; ptr::copy + // tolerates overlap with the destination prefix. + unsafe { ptr::copy(pointer, base, n) }; + } + Ok((n, result[3] & 0xff != 0)) + } + 1 => Err((result[1] & 0xff) as u8), + _ => unreachable!("invalid read result"), + } + }) +} + pub fn poll(input: &[u32], storage: &mut [u32], ready: &mut Vec) { assert!(!input.is_empty()); let mut result = [0usize; 2]; diff --git a/crates/turnloop/src/backend/wasi_p2/fs.rs b/crates/turnloop/src/backend/wasi_p2/fs.rs new file mode 100644 index 0000000..eb47f39 --- /dev/null +++ b/crates/turnloop/src/backend/wasi_p2/fs.rs @@ -0,0 +1,257 @@ +//! `wasi:filesystem@0.2` binding surface for the shared WASI filesystem core. +//! Reads lower directly into caller memory; writes, metadata and path calls lower +//! borrowed slices and strings, so steady-state requests allocate nothing. +use super::super::wasi_fs::{Api, errno, error}; +use crate::{ + Error, ErrorKind, Result, + fs::{FileMetadata, FileOptions, FileTime, FileType, TimeChange}, +}; +use wasip2::{ + clocks::wall_clock::Datetime, + filesystem::types::{ + Descriptor, DescriptorFlags, DescriptorStat, DescriptorType, DirectoryEntryStream, + ErrorCode, NewTimestamp, OpenFlags, PathFlags, + }, + io::streams::OutputStream, +}; + +pub(in crate::backend) struct P2; + +fn errno_of(code: ErrorCode) -> i32 { + use ErrorCode as E; + match code { + E::Access => 2, + E::WouldBlock => 6, + E::Already => 7, + E::BadDescriptor => 8, + E::Busy => 10, + E::Deadlock => 16, + E::Quota => 19, + E::Exist => 20, + E::FileTooLarge => 22, + E::IllegalByteSequence => 25, + E::InProgress => 26, + E::Interrupted => 27, + E::Invalid => 28, + E::Io => 29, + E::IsDirectory => 31, + E::Loop => 32, + E::TooManyLinks => 34, + E::MessageSize => 35, + E::NameTooLong => 37, + E::NoDevice => 43, + E::NoEntry => 44, + E::NoLock => 46, + E::InsufficientMemory => 48, + E::InsufficientSpace => 51, + E::NotDirectory => 54, + E::NotEmpty => 55, + E::NotRecoverable => 56, + E::Unsupported => 58, + E::NoTty => 59, + E::NoSuchDevice => 60, + E::Overflow => 61, + E::NotPermitted => 63, + E::Pipe => 64, + E::ReadOnly => 69, + E::InvalidSeek => 70, + E::TextFileBusy => 74, + E::CrossDevice => 75, + } +} +fn fail(code: ErrorCode) -> Error { + error(errno_of(code)) +} +fn follow(yes: bool) -> PathFlags { + if yes { + PathFlags::SYMLINK_FOLLOW + } else { + PathFlags::empty() + } +} +fn kind(t: DescriptorType) -> FileType { + match t { + DescriptorType::RegularFile => FileType::File, + DescriptorType::Directory => FileType::Directory, + DescriptorType::SymbolicLink => FileType::Symlink, + DescriptorType::BlockDevice => FileType::BlockDevice, + DescriptorType::CharacterDevice => FileType::CharDevice, + DescriptorType::Fifo => FileType::Fifo, + DescriptorType::Socket => FileType::Socket, + DescriptorType::Unknown => FileType::Unknown, + } +} +fn time(t: Datetime) -> FileTime { + FileTime { + seconds: t.seconds as i64, + nanoseconds: t.nanoseconds, + } +} +fn metadata(s: DescriptorStat) -> FileMetadata { + FileMetadata { + kind: kind(s.type_), + size: s.size, + mode: None, + device: None, + inode: None, + links: Some(s.link_count), + uid: None, + gid: None, + rdev: None, + block_size: None, + blocks: None, + accessed: s.data_access_timestamp.map(time), + modified: s.data_modification_timestamp.map(time), + changed: s.status_change_timestamp.map(time), + created: None, + } +} +fn timestamp(change: TimeChange) -> Result { + Ok(match change { + TimeChange::Keep => NewTimestamp::NoChange, + TimeChange::Now => NewTimestamp::Now, + TimeChange::At(t) => NewTimestamp::Timestamp(Datetime { + seconds: u64::try_from(t.seconds).map_err(|_| error(errno::EINVAL))?, + nanoseconds: t.nanoseconds, + }), + }) +} + +impl Api for P2 { + type Descriptor = Descriptor; + type Appender = OutputStream; + type Entries = DirectoryEntryStream; + fn preopens() -> Vec<(Descriptor, String)> { + wasip2::filesystem::preopens::get_directories() + } + fn open_at( + dir: &Descriptor, + path: &str, + o: &FileOptions, + directory: bool, + ) -> Result { + let mut open = OpenFlags::empty(); + if o.create || o.exclusive { + open |= OpenFlags::CREATE; + } + if o.exclusive { + open |= OpenFlags::EXCLUSIVE; + } + if o.truncate { + open |= OpenFlags::TRUNCATE; + } + if directory { + open |= OpenFlags::DIRECTORY; + } + let mut flags = DescriptorFlags::empty(); + if o.read { + flags |= DescriptorFlags::READ; + } + if o.write || o.append { + flags |= DescriptorFlags::WRITE; + } + if o.sync { + flags |= DescriptorFlags::FILE_INTEGRITY_SYNC; + } + if o.data_sync { + flags |= DescriptorFlags::DATA_INTEGRITY_SYNC; + } + if flags.is_empty() { + return Err(Error::new(ErrorKind::InvalidInput)); + } + dir.open_at(follow(o.follow_symlinks), path, open, flags) + .map_err(fail) + } + fn read(file: &Descriptor, output: &mut [u8], offset: u64) -> Result<(usize, bool)> { + super::abi::file_read(file, output, offset).map_err(|code| { + // SAFETY: the host returned a valid error-code discriminant. + fail(unsafe { ErrorCode::_lift(code) }) + }) + } + fn write(file: &Descriptor, bytes: &[u8], offset: u64) -> Result { + file.write(bytes, offset).map(|n| n as usize).map_err(fail) + } + fn append( + file: &Descriptor, + appender: &mut Option, + bytes: &[u8], + ) -> Result { + if appender.is_none() { + *appender = Some(file.append_via_stream().map_err(fail)?); + } + let stream = appender.as_ref().expect("append stream"); + // blocking-write-and-flush accepts at most 4096 bytes; report a short write. + let n = bytes.len().min(4096); + stream + .blocking_write_and_flush(&bytes[..n]) + .map_err(|_| error(errno::EBADF))?; + Ok(n) + } + fn stat(file: &Descriptor) -> Result { + file.stat().map(metadata).map_err(fail) + } + fn stat_at(dir: &Descriptor, path: &str, yes: bool) -> Result { + dir.stat_at(follow(yes), path).map(metadata).map_err(fail) + } + fn sync(file: &Descriptor, data_only: bool) -> Result<()> { + if data_only { + file.sync_data() + } else { + file.sync() + } + .map_err(fail) + } + fn set_size(file: &Descriptor, size: u64) -> Result<()> { + file.set_size(size).map_err(fail) + } + fn set_times(file: &Descriptor, accessed: TimeChange, modified: TimeChange) -> Result<()> { + file.set_times(timestamp(accessed)?, timestamp(modified)?) + .map_err(fail) + } + fn set_times_at( + dir: &Descriptor, + path: &str, + yes: bool, + accessed: TimeChange, + modified: TimeChange, + ) -> Result<()> { + dir.set_times_at( + follow(yes), + path, + timestamp(accessed)?, + timestamp(modified)?, + ) + .map_err(fail) + } + fn entries(dir: &Descriptor) -> Result { + dir.read_directory().map_err(fail) + } + fn next_entry(entries: &mut DirectoryEntryStream) -> Result> { + entries + .read_directory_entry() + .map(|entry| entry.map(|e| (kind(e.type_), e.name))) + .map_err(fail) + } + fn create_directory_at(dir: &Descriptor, path: &str) -> Result<()> { + dir.create_directory_at(path).map_err(fail) + } + fn remove_directory_at(dir: &Descriptor, path: &str) -> Result<()> { + dir.remove_directory_at(path).map_err(fail) + } + fn unlink_file_at(dir: &Descriptor, path: &str) -> Result<()> { + dir.unlink_file_at(path).map_err(fail) + } + fn rename_at(dir: &Descriptor, path: &str, new_dir: &Descriptor, new_path: &str) -> Result<()> { + dir.rename_at(path, new_dir, new_path).map_err(fail) + } + fn link_at(dir: &Descriptor, path: &str, new_dir: &Descriptor, new_path: &str) -> Result<()> { + dir.link_at(PathFlags::empty(), path, new_dir, new_path) + .map_err(fail) + } + fn symlink_at(dir: &Descriptor, target: &str, path: &str) -> Result<()> { + dir.symlink_at(target, path).map_err(fail) + } + fn readlink_at(dir: &Descriptor, path: &str) -> Result { + dir.readlink_at(path).map_err(fail) + } +} diff --git a/crates/turnloop/src/backend/wasi_p3.rs b/crates/turnloop/src/backend/wasi_p3.rs index 1c8fd4c..bdb7f83 100644 --- a/crates/turnloop/src/backend/wasi_p3.rs +++ b/crates/turnloop/src/backend/wasi_p3.rs @@ -3,10 +3,11 @@ //! Experimental: host-yield boundedness is unproven. Allocation gates require //! release on the pinned p3 compiler; see docs/upstream/wasi-p3-wait.md. mod abi; +mod fs; mod return_storage; mod wait_set; use crate::{ - backend::{Backend, Event, Operation, Outcome, PollInfo, Request, Wake}, + backend::{Backend, Event, Filesystem, Operation, Outcome, PollInfo, Request, Wake}, *, }; use std::{ @@ -136,6 +137,7 @@ pub struct WasiP3 { wake: Arc, wait_set: WaitSet, returns: Rc, + files: super::wasi_fs::Files, } fn direction(op: &Operation) -> usize { usize::from(!matches!( @@ -342,12 +344,17 @@ impl WasiP3 { unsafe impl Backend for WasiP3 { type Wake = WasiWake; type Detached = Detached; + const FILESYSTEM: Filesystem = Filesystem::Backend; + fn fs(&mut self, op: OpId, handle: Option, request: FsRequest) -> Result<()> { + self.files.submit(op, handle, request) + } fn new(config: &Config, pool: BufferPool) -> Result { Ok(Self { resources: (0..config.max_handles).map(|_| None).collect(), ops: (0..config.max_operations).map(|_| None).collect(), ready: VecDeque::with_capacity(config.max_handles), cancelled: VecDeque::with_capacity(config.max_operations), + files: super::wasi_fs::Files::new(config, pool.clone()), pool, wake: Arc::new(WasiWake), wait_set: WaitSet::new(), @@ -461,6 +468,7 @@ unsafe impl Backend for WasiP3 { request.operation, Operation::ProcessExit | Operation::WatchSignal + | Operation::WatchFs | Operation::SendHandle(_) | Operation::RecvHandle ) { @@ -510,6 +518,9 @@ unsafe impl Backend for WasiP3 { Ok(()) } fn cancel(&mut self, op: OpId) -> Result<()> { + if self.files.cancel(op) { + return Ok(()); + } let p = self .ops .get(op.index()) @@ -533,7 +544,7 @@ unsafe impl Backend for WasiP3 { Ok(()) } fn has_work(&self) -> bool { - !self.ready.is_empty() || !self.cancelled.is_empty() + !self.ready.is_empty() || !self.cancelled.is_empty() || self.files.has_work() } fn poll( @@ -552,6 +563,7 @@ unsafe impl Backend for WasiP3 { result: Ok(Outcome::Cancelled), }); } + self.files.run(events); self.run_ready(events); if self.has_work() || !events.is_empty() || events.len() == events.capacity() { return Ok(PollInfo::default()); @@ -620,6 +632,7 @@ unsafe impl Backend for WasiP3 { )) } fn release(&mut self, h: Handle) { + self.files.release(h); if self.get(h).is_ok() { self.ready.retain(|&at| at != h); self.resources[h.index()] = None; @@ -835,6 +848,7 @@ fn execute( match &p.request.operation { Operation::ProcessExit | Operation::WatchSignal + | Operation::WatchFs | Operation::SendHandle(_) | Operation::RecvHandle => return Err(Error::new(ErrorKind::Unsupported)), Operation::Connect => { diff --git a/crates/turnloop/src/backend/wasi_p3/fs.rs b/crates/turnloop/src/backend/wasi_p3/fs.rs new file mode 100644 index 0000000..2f18047 --- /dev/null +++ b/crates/turnloop/src/backend/wasi_p3/fs.rs @@ -0,0 +1,613 @@ +//! `wasi:filesystem@0.3` binding surface for the shared WASI filesystem core. +//! +//! Experimental, like the rest of the 0.3 backend. Every import is lowered by hand +//! with its canonical layout (wasip3 0.8.0), and each raw call preserves the +//! caller's shadow-stack context (see `abi::preserving_stack`): generated bindings +//! trap on the pinned toolchain in debug builds once a call re-enters the guest. +//! A request blocks the agent on a private waitable set until its subtask, stream +//! or future completes. File bytes move directly between host streams and caller +//! memory; paths are borrowed. Only strings the host returns (directory entry and +//! link names, rare error text) are owned allocations. +use super::{ + abi::preserving_stack, + wait_set::{WaitSet, subtask_drop}, +}; +use crate::{ + Result, + backend::wasi_fs::{Api, error}, + fs::{FileMetadata, FileOptions, FileTime, FileType, TimeChange}, +}; +use std::{ + alloc::{Layout, dealloc}, + ptr, +}; +use wasip3::{ + filesystem::types::{Descriptor, DirectoryEntry, ErrorCode}, + wit_future::FuturePayload, + wit_stream::StreamPayload, +}; + +type Done = std::result::Result<(), ErrorCode>; +/// Canonical stream/future operation still pending. +const BLOCKED: u32 = u32::MAX; +const MAX_STREAM: usize = (1 << 28) - 1; +/// wasi-libc errno for each `error-code` discriminant; `other` (36) reads as EIO. +const ERRNO: [i32; 36] = [ + 2, 7, 8, 10, 16, 19, 20, 22, 25, 26, 27, 28, 29, 31, 32, 34, 35, 37, 43, 44, 46, 48, 51, 54, + 55, 56, 58, 59, 60, 61, 63, 64, 69, 70, 74, 75, +]; + +#[link(wasm_import_module = "wasi:filesystem/types@0.3.0")] +unsafe extern "C" { + #[link_name = "[async-lower][method]descriptor.open-at"] + fn raw_open_at(params: *mut u8, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.stat"] + fn raw_stat(handle: i32, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.stat-at"] + fn raw_stat_at(handle: i32, flags: i32, path: *const u8, len: usize, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.sync"] + fn raw_sync(handle: i32, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.sync-data"] + fn raw_sync_data(handle: i32, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.set-size"] + fn raw_set_size(handle: i32, size: i64, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.set-times"] + fn raw_set_times(params: *mut u8, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.set-times-at"] + fn raw_set_times_at(params: *mut u8, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.create-directory-at"] + fn raw_create_directory_at(handle: i32, path: *const u8, len: usize, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.remove-directory-at"] + fn raw_remove_directory_at(handle: i32, path: *const u8, len: usize, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.unlink-file-at"] + fn raw_unlink_file_at(handle: i32, path: *const u8, len: usize, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.readlink-at"] + fn raw_readlink_at(handle: i32, path: *const u8, len: usize, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.rename-at"] + fn raw_rename_at(params: *mut u8, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.link-at"] + fn raw_link_at(params: *mut u8, results: *mut u8) -> i32; + #[link_name = "[async-lower][method]descriptor.symlink-at"] + fn raw_symlink_at(params: *mut u8, results: *mut u8) -> i32; + #[link_name = "[method]descriptor.read-via-stream"] + fn raw_read_via_stream(handle: i32, offset: i64, results: *mut u8); + #[link_name = "[method]descriptor.write-via-stream"] + fn raw_write_via_stream(handle: i32, data: i32, offset: i64) -> i32; + #[link_name = "[method]descriptor.append-via-stream"] + fn raw_append_via_stream(handle: i32, data: i32) -> i32; + #[link_name = "[method]descriptor.read-directory"] + fn raw_read_directory(handle: i32, results: *mut u8); +} +#[link(wasm_import_module = "wasi:filesystem/preopens@0.3.0")] +unsafe extern "C" { + #[link_name = "get-directories"] + fn raw_get_directories(results: *mut u8); +} + +thread_local! { + // One private set: a request joins exactly one waitable, so every event is its own. + static WAIT: WaitSet = WaitSet::new(); +} +/// Block until the waitable's pending operation reports its packed result. +fn complete(handle: u32, code: u32) -> u32 { + if code != BLOCKED { + return code; + } + WAIT.with(|set| { + set.join(handle); + let code = loop { + let (_, waitable, code) = set.step(true); + if waitable == handle { + break code; + } + }; + set.remove(handle); + code + }) +} +/// Run an async-lowered import and block until it returns, then drop its subtask. +fn call(f: impl FnOnce() -> i32) { + let packed = preserving_stack(f) as u32; + let task = packed >> 4; + if task == 0 { + return; + } + if packed & 15 < 2 { + WAIT.with(|set| { + set.join(task); + while !matches!(set.step(true), (_, waitable, code) if waitable == task && code >= 2) {} + set.remove(task); + }); + } + // SAFETY: the subtask returned and left the private set. + unsafe { subtask_drop(task) }; +} + +fn byte(area: *const u8, offset: usize) -> u8 { + // SAFETY: callers pass an initialized canonical area covering `offset`. + unsafe { area.add(offset).read() } +} +fn word(area: *const u8, offset: usize) -> u32 { + // SAFETY: callers pass an initialized canonical area covering the word. + unsafe { area.add(offset).cast::().read_unaligned() } +} +fn wide(area: *const u8, offset: usize) -> u64 { + // SAFETY: callers pass an initialized canonical area covering the value. + unsafe { area.add(offset).cast::().read_unaligned() } +} +fn put(area: *mut u8, offset: usize, bytes: &[u8]) { + // SAFETY: callers pass writable parameter storage covering the value. + unsafe { ptr::copy_nonoverlapping(bytes.as_ptr(), area.add(offset), bytes.len()) } +} +fn put_str(area: *mut u8, offset: usize, text: &str) { + put(area, offset, &(text.as_ptr() as u32).to_le_bytes()); + put(area, offset + 4, &(text.len() as u32).to_le_bytes()); +} +/// Free a canonical string the host allocated through this guest's allocator. +fn free_string(pointer: u32, len: u32) { + if len != 0 { + // SAFETY: the host lowered this owned list through cabi_realloc with align 1. + unsafe { + dealloc( + pointer as *mut u8, + Layout::from_size_align_unchecked(len as usize, 1), + ) + }; + } +} +fn take_string(pointer: u32, len: u32) -> String { + if len == 0 { + return String::new(); + } + // SAFETY: an owned, host-validated UTF-8 list allocated by the Rust heap. + unsafe { String::from_raw_parts(pointer as *mut u8, len as usize, len as usize) } +} +/// Decode an error-code at `offset`, releasing `other` text. +fn error_at(area: *const u8, offset: usize) -> crate::Error { + let code = byte(area, offset) as usize; + if code >= ERRNO.len() { + if byte(area, offset + 4) == 1 { + free_string(word(area, offset + 8), word(area, offset + 12)); + } + return error(29); + } + error(ERRNO[code]) +} +fn unit(area: &[u32; 5]) -> Result<()> { + let area = area.as_ptr().cast::(); + if byte(area, 0) == 0 { + Ok(()) + } else { + Err(error_at(area, 4)) + } +} +fn kind(tag: u8) -> FileType { + match tag { + 0 => FileType::BlockDevice, + 1 => FileType::CharDevice, + 2 => FileType::Directory, + 3 => FileType::Fifo, + 4 => FileType::Symlink, + 5 => FileType::File, + 6 => FileType::Socket, + _ => FileType::Unknown, + } +} +fn stat_result(area: &[u64; 14]) -> Result { + let area = area.as_ptr().cast::(); + if byte(area, 0) != 0 { + return Err(error_at(area, 8)); + } + let tag = byte(area, 8); + if tag == 7 && byte(area, 12) == 1 { + free_string(word(area, 16), word(area, 20)); + } + let time = |at: usize| { + (byte(area, at) == 1).then(|| FileTime { + seconds: wide(area, at + 8) as i64, + nanoseconds: word(area, at + 16), + }) + }; + Ok(FileMetadata { + kind: kind(tag), + size: wide(area, 32), + mode: None, + device: None, + inode: None, + links: Some(wide(area, 24)), + uid: None, + gid: None, + rdev: None, + block_size: None, + blocks: None, + accessed: time(40), + modified: time(64), + changed: time(88), + created: None, + }) +} +fn timestamp(area: *mut u8, offset: usize, change: TimeChange) { + match change { + TimeChange::Keep => put(area, offset, &[0]), + TimeChange::Now => put(area, offset, &[1]), + TimeChange::At(t) => { + put(area, offset, &[2]); + put(area, offset + 8, &t.seconds.to_le_bytes()); + put(area, offset + 16, &t.nanoseconds.to_le_bytes()); + } + } +} +/// Wait for a byte stream's `future>`, then drop it. +fn finish(future: u32) -> Result<()> { + let mut area = [0u32; 5]; + let vtable = ::VTABLE; + // SAFETY: owned future handle and a pinned 20-byte result area. + let code = + preserving_stack(|| unsafe { (vtable.start_read)(future, area.as_mut_ptr().cast()) }); + complete(future, code); + // SAFETY: the value was delivered; the readable end is no longer needed. + preserving_stack(|| unsafe { (vtable.drop_readable)(future) }); + unit(&area) +} +fn abandon(future: u32) { + // SAFETY: owned, unread future handle. + preserving_stack(|| unsafe { (::VTABLE.drop_readable)(future) }); +} +/// Write all bytes into a new stream consumed by `start`, returning the count written. +fn write_stream(bytes: &[u8], start: impl FnOnce(i32) -> i32) -> Result { + let vtable = ::VTABLE; + // SAFETY: stream.new creates a fresh owned (readable, writable) pair. + let pair = preserving_stack(|| unsafe { (vtable.new)() }); + let (readable, writable) = (pair as u32, (pair >> 32) as u32); + let future = preserving_stack(|| start(readable as i32)) as u32; + let mut written = 0; + while written < bytes.len() { + let rest = &bytes[written..]; + // SAFETY: owned writable end and initialized bytes borrowed until completion. + let code = preserving_stack(|| unsafe { + (vtable.start_write)(writable, rest.as_ptr(), rest.len().min(MAX_STREAM)) + }); + let code = complete(writable, code); + written += (code >> 4) as usize; + if code & 15 != 0 { + break; + } + } + // SAFETY: closing the owned writable end tells the host the data is complete. + preserving_stack(|| unsafe { (vtable.drop_writable)(writable) }); + finish(future).map(|()| written) +} + +pub(in crate::backend) struct P3; +pub(in crate::backend) struct Entries { + stream: u32, + future: Option, +} +impl Drop for Entries { + fn drop(&mut self) { + // SAFETY: owned directory stream handle with no pending read. + preserving_stack(|| unsafe { + (::VTABLE.drop_readable)(self.stream) + }); + if let Some(future) = self.future.take() { + abandon(future); + } + } +} + +impl Api for P3 { + type Descriptor = Descriptor; + type Appender = (); + type Entries = Entries; + fn preopens() -> Vec<(Descriptor, String)> { + let mut area = [0u32; 2]; + // SAFETY: aligned result area for list>. + preserving_stack(|| unsafe { raw_get_directories(area.as_mut_ptr().cast()) }); + let (list, count) = (area[0] as *const u8, area[1] as usize); + let preopens = (0..count) + .map(|i| { + let at = i * 12; + // SAFETY: each element transfers one owned descriptor handle. + let descriptor = unsafe { Descriptor::from_handle(word(list, at)) }; + ( + descriptor, + take_string(word(list, at + 4), word(list, at + 8)), + ) + }) + .collect(); + if count != 0 { + // SAFETY: the host allocated the element array with this layout. + unsafe { + dealloc( + list.cast_mut(), + Layout::from_size_align_unchecked(count * 12, 4), + ) + }; + } + preopens + } + fn open_at( + dir: &Descriptor, + path: &str, + o: &FileOptions, + directory: bool, + ) -> Result { + let open = u8::from(o.create || o.exclusive) + | (u8::from(directory) << 1) + | (u8::from(o.exclusive) << 2) + | (u8::from(o.truncate) << 3); + let access = u8::from(o.read) + | (u8::from(o.write || o.append) << 1) + | (u8::from(o.sync) << 2) + | (u8::from(o.data_sync) << 3); + if access == 0 { + return Err(crate::Error::new(crate::ErrorKind::InvalidInput)); + } + let mut params = [0u32; 5]; + let p = params.as_mut_ptr().cast::(); + put(p, 0, &dir.handle().to_le_bytes()); + put(p, 4, &[u8::from(o.follow_symlinks)]); + put_str(p, 8, path); + put(p, 16, &[open, access]); + let mut results = [0u32; 5]; + // SAFETY: pinned parameters (borrowing `path`) and results outlive the subtask. + call(|| unsafe { raw_open_at(p, results.as_mut_ptr().cast()) }); + let r = results.as_ptr().cast::(); + if byte(r, 0) != 0 { + return Err(error_at(r, 4)); + } + // SAFETY: success transferred one owned descriptor handle. + Ok(unsafe { Descriptor::from_handle(word(r, 4)) }) + } + fn read(file: &Descriptor, output: &mut [u8], offset: u64) -> Result<(usize, bool)> { + let mut handles = [0u32; 2]; + // SAFETY: owned result handles are written to the pinned area. + preserving_stack(|| unsafe { + raw_read_via_stream( + file.handle() as i32, + offset as i64, + handles.as_mut_ptr().cast(), + ) + }); + let (stream, future) = (handles[0], handles[1]); + let vtable = ::VTABLE; + let (mut filled, mut eof) = (0, false); + while filled < output.len() { + let rest = &mut output[filled..]; + // SAFETY: owned readable end; the exclusive output region outlives completion. + let code = preserving_stack(|| unsafe { + (vtable.start_read)(stream, rest.as_mut_ptr(), rest.len().min(MAX_STREAM)) + }); + let code = complete(stream, code); + filled += (code >> 4) as usize; + if code & 15 != 0 { + eof = true; + break; + } + } + // SAFETY: owned readable end with no pending operation. + preserving_stack(|| unsafe { (vtable.drop_readable)(stream) }); + if eof { + finish(future)?; + } else { + abandon(future); + } + Ok((filled, eof)) + } + fn write(file: &Descriptor, bytes: &[u8], offset: u64) -> Result { + write_stream(bytes, |readable| { + // SAFETY: transfers the owned readable end; returns an owned future handle. + unsafe { raw_write_via_stream(file.handle() as i32, readable, offset as i64) } + }) + } + fn append(file: &Descriptor, _: &mut Option<()>, bytes: &[u8]) -> Result { + write_stream(bytes, |readable| { + // SAFETY: transfers the owned readable end; returns an owned future handle. + unsafe { raw_append_via_stream(file.handle() as i32, readable) } + }) + } + fn stat(file: &Descriptor) -> Result { + let mut results = [0u64; 14]; + // SAFETY: pinned 112-byte result area outlives the subtask. + call(|| unsafe { raw_stat(file.handle() as i32, results.as_mut_ptr().cast()) }); + stat_result(&results) + } + fn stat_at(dir: &Descriptor, path: &str, follow: bool) -> Result { + let mut results = [0u64; 14]; + // SAFETY: borrowed path and pinned result area outlive the subtask. + call(|| unsafe { + raw_stat_at( + dir.handle() as i32, + i32::from(follow), + path.as_ptr(), + path.len(), + results.as_mut_ptr().cast(), + ) + }); + stat_result(&results) + } + fn sync(file: &Descriptor, data_only: bool) -> Result<()> { + let mut results = [0u32; 5]; + // SAFETY: pinned result area outlives the subtask. + call(|| unsafe { + if data_only { + raw_sync_data(file.handle() as i32, results.as_mut_ptr().cast()) + } else { + raw_sync(file.handle() as i32, results.as_mut_ptr().cast()) + } + }); + unit(&results) + } + fn set_size(file: &Descriptor, size: u64) -> Result<()> { + let mut results = [0u32; 5]; + // SAFETY: pinned result area outlives the subtask. + call(|| unsafe { + raw_set_size( + file.handle() as i32, + size as i64, + results.as_mut_ptr().cast(), + ) + }); + unit(&results) + } + fn set_times(file: &Descriptor, accessed: TimeChange, modified: TimeChange) -> Result<()> { + let mut params = [0u64; 7]; + let p = params.as_mut_ptr().cast::(); + put(p, 0, &file.handle().to_le_bytes()); + timestamp(p, 8, accessed); + timestamp(p, 32, modified); + let mut results = [0u32; 5]; + // SAFETY: pinned parameters and results outlive the subtask. + call(|| unsafe { raw_set_times(p, results.as_mut_ptr().cast()) }); + unit(&results) + } + fn set_times_at( + dir: &Descriptor, + path: &str, + follow: bool, + accessed: TimeChange, + modified: TimeChange, + ) -> Result<()> { + let mut params = [0u64; 8]; + let p = params.as_mut_ptr().cast::(); + put(p, 0, &dir.handle().to_le_bytes()); + put(p, 4, &[u8::from(follow)]); + put_str(p, 8, path); + timestamp(p, 16, accessed); + timestamp(p, 40, modified); + let mut results = [0u32; 5]; + // SAFETY: pinned parameters (borrowing `path`) and results outlive the subtask. + call(|| unsafe { raw_set_times_at(p, results.as_mut_ptr().cast()) }); + unit(&results) + } + fn entries(dir: &Descriptor) -> Result { + let mut handles = [0u32; 2]; + // SAFETY: owned result handles are written to the pinned area. + preserving_stack(|| unsafe { + raw_read_directory(dir.handle() as i32, handles.as_mut_ptr().cast()) + }); + Ok(Entries { + stream: handles[0], + future: Some(handles[1]), + }) + } + fn next_entry(entries: &mut Entries) -> Result> { + let vtable = ::VTABLE; + loop { + let Some(future) = entries.future else { + return Ok(None); + }; + let mut element = [0u32; 6]; + // SAFETY: owned readable end and one pinned 24-byte element slot. + let code = preserving_stack(|| unsafe { + (vtable.start_read)(entries.stream, element.as_mut_ptr().cast(), 1) + }); + let code = complete(entries.stream, code); + if code >> 4 == 1 { + let e = element.as_ptr().cast::(); + let tag = byte(e, 0); + if tag == 7 && byte(e, 4) == 1 { + free_string(word(e, 8), word(e, 12)); + } + return Ok(Some((kind(tag), take_string(word(e, 16), word(e, 20))))); + } + if code & 15 != 0 { + entries.future = None; + finish(future)?; + return Ok(None); + } + } + } + fn create_directory_at(dir: &Descriptor, path: &str) -> Result<()> { + let mut results = [0u32; 5]; + // SAFETY: borrowed path and pinned result area outlive the subtask. + call(|| unsafe { + raw_create_directory_at( + dir.handle() as i32, + path.as_ptr(), + path.len(), + results.as_mut_ptr().cast(), + ) + }); + unit(&results) + } + fn remove_directory_at(dir: &Descriptor, path: &str) -> Result<()> { + let mut results = [0u32; 5]; + // SAFETY: borrowed path and pinned result area outlive the subtask. + call(|| unsafe { + raw_remove_directory_at( + dir.handle() as i32, + path.as_ptr(), + path.len(), + results.as_mut_ptr().cast(), + ) + }); + unit(&results) + } + fn unlink_file_at(dir: &Descriptor, path: &str) -> Result<()> { + let mut results = [0u32; 5]; + // SAFETY: borrowed path and pinned result area outlive the subtask. + call(|| unsafe { + raw_unlink_file_at( + dir.handle() as i32, + path.as_ptr(), + path.len(), + results.as_mut_ptr().cast(), + ) + }); + unit(&results) + } + fn rename_at(dir: &Descriptor, path: &str, new_dir: &Descriptor, new_path: &str) -> Result<()> { + let mut params = [0u32; 6]; + let p = params.as_mut_ptr().cast::(); + put(p, 0, &dir.handle().to_le_bytes()); + put_str(p, 4, path); + put(p, 12, &new_dir.handle().to_le_bytes()); + put_str(p, 16, new_path); + let mut results = [0u32; 5]; + // SAFETY: pinned parameters (borrowing both paths) and results outlive the subtask. + call(|| unsafe { raw_rename_at(p, results.as_mut_ptr().cast()) }); + unit(&results) + } + fn link_at(dir: &Descriptor, path: &str, new_dir: &Descriptor, new_path: &str) -> Result<()> { + let mut params = [0u32; 7]; + let p = params.as_mut_ptr().cast::(); + put(p, 0, &dir.handle().to_le_bytes()); + put(p, 4, &[0]); + put_str(p, 8, path); + put(p, 16, &new_dir.handle().to_le_bytes()); + put_str(p, 20, new_path); + let mut results = [0u32; 5]; + // SAFETY: pinned parameters (borrowing both paths) and results outlive the subtask. + call(|| unsafe { raw_link_at(p, results.as_mut_ptr().cast()) }); + unit(&results) + } + fn symlink_at(dir: &Descriptor, target: &str, path: &str) -> Result<()> { + let mut params = [0u32; 5]; + let p = params.as_mut_ptr().cast::(); + put(p, 0, &dir.handle().to_le_bytes()); + put_str(p, 4, target); + put_str(p, 12, path); + let mut results = [0u32; 5]; + // SAFETY: pinned parameters (borrowing both strings) and results outlive the subtask. + call(|| unsafe { raw_symlink_at(p, results.as_mut_ptr().cast()) }); + unit(&results) + } + fn readlink_at(dir: &Descriptor, path: &str) -> Result { + let mut results = [0u32; 5]; + // SAFETY: borrowed path and pinned result area outlive the subtask. + call(|| unsafe { + raw_readlink_at( + dir.handle() as i32, + path.as_ptr(), + path.len(), + results.as_mut_ptr().cast(), + ) + }); + let r = results.as_ptr().cast::(); + if byte(r, 0) != 0 { + return Err(error_at(r, 4)); + } + Ok(take_string(word(r, 4), word(r, 8))) + } +} diff --git a/crates/turnloop/src/backend/watch.rs b/crates/turnloop/src/backend/watch.rs new file mode 100644 index 0000000..e6ef98c --- /dev/null +++ b/crates/turnloop/src/backend/watch.rs @@ -0,0 +1,527 @@ +//! Filesystem watches for the epoll and kqueue backends, without turnloop threads. +//! +//! * Linux/Android: one inotify descriptor per loop, registered with epoll. +//! Watches of the same inode share a watch descriptor. +//! * kqueue (BSD, iOS, and files on macOS): EVFILT_VNODE on the loop's kqueue. +//! * macOS directories: FSEvents streams on a private serial dispatch queue (the +//! OS delivers on its own threads) that publish to this loop and notify it. +//! +//! Records are parsed as the OS reports them into bounded per-watch storage and +//! delivered as pooled batches. Recursion is available only through FSEvents. +use super::{ + poller::{Poller, Ready}, + unix::Detached, +}; +use crate::{ + backend::{Event, Operation, Outcome, Request}, + fs::watch::Ring, + *, +}; +use std::{collections::VecDeque, os::fd::AsRawFd}; +#[cfg(target_os = "macos")] +#[path = "fsevents.rs"] +mod fsevents; + +#[cfg(any(target_os = "linux", target_os = "android"))] +/// Reserved poller key of the loop's inotify descriptor; no handle key has an +/// index of 0xFFFF_FFFE. +const INOTIFY_KEY: u64 = u64::MAX - 1; + +enum Source { + #[cfg(any(target_os = "linux", target_os = "android"))] + Inotify { wd: i32 }, + #[cfg(not(any(target_os = "linux", target_os = "android")))] + Vnode(std::os::fd::OwnedFd), + #[cfg(target_os = "macos")] + FsEvents(fsevents::Stream), +} +struct Entry { + handle: Handle, + op: Option, + queued: bool, + /// Final path component, reported for events on the watched root itself. + name: Box<[u8]>, + /// Records of loop-thread sources (FSEvents keeps its own, shared with its queue). + ring: Ring, + source: Source, +} +impl Entry { + fn with_ring(&mut self, f: impl FnOnce(&mut Ring) -> R) -> R { + #[cfg(target_os = "macos")] + if let Source::FsEvents(stream) = &self.source { + return stream.with_ring(f); + } + f(&mut self.ring) + } +} +pub(super) struct Watches { + entries: Vec>, + /// Indices of occupied entries, so event dispatch never scans every handle slot. + active: Vec, + ops: Vec>, + ready: VecDeque, + cancelled: VecDeque, + pool: BufferPool, + #[cfg(any(target_os = "linux", target_os = "android"))] + inotify: Option, + #[cfg(target_os = "macos")] + fsevents: fsevents::Streams, +} +#[cfg(any(target_os = "linux", target_os = "android"))] +struct Inotify { + fd: std::os::fd::OwnedFd, + buffer: Box<[u8]>, + readable: bool, +} + +fn basename(path: &FsPath) -> Box<[u8]> { + use std::os::unix::ffi::OsStrExt; + let bytes = path.as_path().as_os_str().as_bytes(); + let trimmed = match bytes.iter().rposition(|&b| b != b'/') { + Some(end) => &bytes[..=end], + None => return bytes.into(), + }; + match trimmed.iter().rposition(|&b| b == b'/') { + Some(slash) => trimmed[slash + 1..].into(), + None => trimmed.into(), + } +} + +impl Watches { + pub fn new(config: &Config, pool: BufferPool) -> Self { + Self { + entries: (0..config.max_handles).map(|_| None).collect(), + active: Vec::with_capacity(config.max_handles), + ops: vec![None; config.max_operations], + ready: VecDeque::with_capacity(config.max_handles), + cancelled: VecDeque::with_capacity(config.max_operations), + pool, + #[cfg(any(target_os = "linux", target_os = "android"))] + inotify: None, + #[cfg(target_os = "macos")] + fsevents: fsevents::Streams::new(config.max_handles), + } + } + #[cfg_attr(not(target_os = "macos"), allow(unused_variables))] + pub fn set_notifier(&mut self, notifier: Notifier) { + #[cfg(target_os = "macos")] + self.fsevents.set_notifier(notifier); + } + fn get(&self, h: Handle) -> Option<&Entry> { + self.entries + .get(h.index()) + .and_then(Option::as_ref) + .filter(|e| e.handle == h) + } + pub fn contains(&self, h: Handle) -> bool { + self.get(h).is_some() + } + pub fn start( + &mut self, + h: Handle, + path: &FsPath, + options: WatchOptions, + poller: &mut P, + ) -> Result<()> { + if self.entries.get(h.index()).is_none_or(Option::is_some) { + return Err(Error::new(ErrorKind::InvalidInput)); + } + let source = self.source(h, path, options, poller)?; + #[cfg(target_os = "macos")] + let ring = if matches!(source, Source::FsEvents(_)) { + Ring::empty() + } else { + Ring::new() + }; + #[cfg(not(target_os = "macos"))] + let ring = Ring::new(); + self.entries[h.index()] = Some(Entry { + handle: h, + op: None, + queued: false, + name: basename(path), + ring, + source, + }); + self.active.push(h.index()); + Ok(()) + } + #[cfg(any(target_os = "linux", target_os = "android"))] + fn source( + &mut self, + _h: Handle, + path: &FsPath, + options: WatchOptions, + poller: &mut P, + ) -> Result { + use std::os::fd::FromRawFd; + if options.recursive { + return Err(Error::new(ErrorKind::Unsupported)); + } + if self.inotify.is_none() { + // SAFETY: integer flags; a successful call returns a new owned descriptor. + let fd = unsafe { libc::inotify_init1(libc::IN_NONBLOCK | libc::IN_CLOEXEC) }; + if fd < 0 { + return Err(super::poller::last_error()); + } + // SAFETY: inotify_init1 returned a new descriptor owned by nobody else. + let fd = unsafe { std::os::fd::OwnedFd::from_raw_fd(fd) }; + poller.register(fd.as_raw_fd(), INOTIFY_KEY)?; + self.inotify = Some(Inotify { + fd, + buffer: vec![0; 64 * 1024].into_boxed_slice(), + readable: false, + }); + } + let inotify = self.inotify.as_ref().expect("inotify"); + // As libuv: every event except attribute and content changes is a rename. + let mask = libc::IN_ATTRIB + | libc::IN_CREATE + | libc::IN_MODIFY + | libc::IN_DELETE + | libc::IN_DELETE_SELF + | libc::IN_MOVE_SELF + | libc::IN_MOVED_FROM + | libc::IN_MOVED_TO; + // SAFETY: live inotify descriptor and NUL-terminated prepared path. + let wd = unsafe { + libc::inotify_add_watch(inotify.fd.as_raw_fd(), path.native().as_ptr(), mask) + }; + if wd < 0 { + return Err(super::poller::last_error()); + } + Ok(Source::Inotify { wd }) + } + #[cfg(not(any(target_os = "linux", target_os = "android")))] + fn source( + &mut self, + h: Handle, + path: &FsPath, + options: WatchOptions, + poller: &mut P, + ) -> Result { + #[cfg(target_os = "macos")] + { + let mut stat = std::mem::MaybeUninit::::uninit(); + // SAFETY: NUL-terminated path and writable output, read only after success. + if unsafe { libc::stat(path.native().as_ptr(), stat.as_mut_ptr()) } < 0 { + return Err(super::poller::last_error()); + } + // SAFETY: successful stat initialized the structure. + let directory = + unsafe { stat.assume_init_ref() }.st_mode & libc::S_IFMT == libc::S_IFDIR; + // As libuv: FSEvents for directories, kqueue for files. + if directory { + return self + .fsevents + .start(h, path, options.recursive) + .map(Source::FsEvents); + } + } + if options.recursive { + return Err(Error::new(ErrorKind::Unsupported)); + } + vnode(h, path, poller).map(Source::Vnode) + } + pub fn submit(&mut self, request: &Request) -> Result<()> { + let h = request.handle; + if !matches!(request.operation, Operation::WatchFs) + || self.get(h).is_none_or(|e| e.op.is_some()) + || self.ops.get(request.op.index()).is_none_or(Option::is_some) + { + return Err(Error::new(ErrorKind::InvalidInput)); + } + let entry = self.entries[h.index()].as_mut().expect("watch entry"); + entry.op = Some(request.op); + self.ops[request.op.index()] = Some(h); + Ok(()) + } + pub fn cancel(&mut self, op: OpId) -> bool { + let Some(h) = self.ops.get(op.index()).copied().flatten() else { + return false; + }; + let Some(entry) = self.entries[h.index()] + .as_mut() + .filter(|e| e.handle == h && e.op == Some(op)) + else { + return false; + }; + entry.op = None; + entry.with_ring(Ring::clear); + self.ops[op.index()] = None; + self.cancelled.push_back(op); + true + } + /// Claim a poller readiness event belonging to a watch source. + pub fn ready(&mut self, e: Ready) -> bool { + #[cfg(any(target_os = "linux", target_os = "android"))] + if e.key == INOTIFY_KEY { + if let Some(inotify) = &mut self.inotify { + inotify.readable = true; + } + return true; + } + let Some(entry) = self + .entries + .get_mut(e.key as u32 as usize) + .and_then(Option::as_mut) + .filter(|entry| entry.handle.key() == e.key) + else { + return false; + }; + #[cfg(not(any(target_os = "linux", target_os = "android")))] + if matches!(entry.source, Source::Vnode(_)) && entry.op.is_some() { + let flags = e.vnode; + if flags & (libc::NOTE_WRITE | libc::NOTE_EXTEND | libc::NOTE_ATTRIB) != 0 { + entry.ring.push(WatchKind::Change, &entry.name); + } + if flags & (libc::NOTE_RENAME | libc::NOTE_DELETE | libc::NOTE_REVOKE) != 0 { + entry.ring.push(WatchKind::Rename, &entry.name); + } + if !entry.queued && !entry.ring.is_empty() { + entry.queued = true; + self.ready.push_back(entry.handle); + } + } + let _ = (e, entry); + true + } + pub fn has_work(&self) -> bool { + #[cfg(any(target_os = "linux", target_os = "android"))] + if self.inotify.as_ref().is_some_and(|i| i.readable) { + return true; + } + #[cfg(target_os = "macos")] + if self.fsevents.has_work() { + return true; + } + !self.cancelled.is_empty() || (!self.ready.is_empty() && self.pool.available()) + } + #[cfg(any(target_os = "linux", target_os = "android"))] + fn read_inotify(&mut self) { + let Some(inotify) = &mut self.inotify else { + return; + }; + // Bound one poll's work; remaining readiness stays cached for the next. + for _ in 0..16 { + if !inotify.readable { + break; + } + // SAFETY: live nonblocking descriptor and exclusively owned buffer. + let n = unsafe { + libc::read( + inotify.fd.as_raw_fd(), + inotify.buffer.as_mut_ptr().cast(), + inotify.buffer.len(), + ) + }; + if n < 0 { + let e = super::poller::last_error(); + if e.os != Some(libc::EINTR) { + // EAGAIN ends the edge; any other error is reported as loss. + inotify.readable = false; + if e.os != Some(libc::EAGAIN) { + for &i in &self.active { + self.entries[i].as_mut().expect("active watch").ring.lost(); + } + } + } + continue; + } + let bytes = &inotify.buffer[..n as usize]; + let mut at = 0; + let header = std::mem::size_of::(); + while at + header <= bytes.len() { + // SAFETY: the kernel wrote a complete event header at this offset. + let event: libc::inotify_event = + unsafe { std::ptr::read_unaligned(bytes[at..].as_ptr().cast()) }; + let name_end = (at + header + event.len as usize).min(bytes.len()); + let raw = &bytes[at + header..name_end]; + let name = &raw[..raw.iter().position(|&b| b == 0).unwrap_or(raw.len())]; + at = name_end; + for &i in &self.active { + let entry = self.entries[i].as_mut().expect("active watch"); + let Source::Inotify { wd } = &mut entry.source; + if event.mask & libc::IN_Q_OVERFLOW != 0 { + entry.ring.lost(); + } else if *wd != event.wd { + continue; + } else if event.mask & libc::IN_IGNORED != 0 { + // The kernel removed this watch; a reused number must not match. + *wd = -1; + continue; + } else if entry.op.is_some() { + let kind = if event.mask & !(libc::IN_ATTRIB | libc::IN_MODIFY) != 0 { + WatchKind::Rename + } else { + WatchKind::Change + }; + let name = if name.is_empty() { &entry.name } else { name }; + entry.ring.push(kind, name); + } + if entry.op.is_some() && !entry.queued && !entry.ring.is_empty() { + entry.queued = true; + self.ready.push_back(entry.handle); + } + } + } + } + } + pub fn poll(&mut self, events: &mut Vec>) { + #[cfg(any(target_os = "linux", target_os = "android"))] + self.read_inotify(); + #[cfg(target_os = "macos")] + self.fsevents.drain(|h| { + let Some(entry) = self + .entries + .get_mut(h.index()) + .and_then(Option::as_mut) + .filter(|e| e.handle == h) + else { + return; + }; + if let Source::FsEvents(stream) = &entry.source { + stream.rearm(); + } + if entry.op.is_some() && !entry.queued { + entry.queued = true; + self.ready.push_back(h); + } + }); + while events.len() < events.capacity() { + let Some(op) = self.cancelled.pop_front() else { + break; + }; + events.push(Event { + op, + terminal: true, + result: Ok(Outcome::Cancelled), + }); + } + for _ in 0..self.ready.len() { + if events.len() == events.capacity() { + break; + } + let Some(h) = self.ready.pop_front() else { + break; + }; + let Some(entry) = self.entries[h.index()].as_mut().filter(|e| e.handle == h) else { + continue; + }; + let Some(op) = entry.op else { + entry.queued = false; + continue; + }; + if entry.with_ring(|ring| ring.is_empty()) { + entry.queued = false; + continue; + } + let Some(mut lease) = self.pool.acquire() else { + // Pool exhaustion: keep the records; resume when a lease returns. + self.ready.push_front(h); + break; + }; + let (overflow, empty) = + entry.with_ring(|ring| (ring.take(&mut lease), ring.is_empty())); + if empty { + entry.queued = false; + } else { + self.ready.push_back(h); + } + events.push(Event { + op, + terminal: false, + result: Ok(Outcome::Watch { + events: lease, + overflow, + }), + }); + } + } + pub fn release(&mut self, h: Handle, poller: &mut P) { + let Some(entry) = self.entries.get_mut(h.index()).and_then(|slot| { + if slot.as_ref().is_some_and(|e| e.handle == h) { + slot.take() + } else { + None + } + }) else { + return; + }; + let _ = &poller; + if let Some(at) = self.active.iter().position(|&i| i == h.index()) { + self.active.swap_remove(at); + } + if entry.queued { + self.ready.retain(|&queued| queued != h); + } + match entry.source { + #[cfg(any(target_os = "linux", target_os = "android"))] + Source::Inotify { wd } => { + let shared = self.active.iter().any(|&i| { + matches!( + self.entries[i].as_ref().expect("active watch").source, + Source::Inotify { wd: other } if other == wd + ) + }); + if wd >= 0 + && !shared + && let Some(inotify) = &self.inotify + { + // SAFETY: live inotify descriptor and a watch owned by this loop. + unsafe { libc::inotify_rm_watch(inotify.fd.as_raw_fd(), wd) }; + } + } + #[cfg(not(any(target_os = "linux", target_os = "android")))] + Source::Vnode(fd) => { + // Closing the descriptor removes its knote from the kqueue. + drop(fd); + } + #[cfg(target_os = "macos")] + Source::FsEvents(stream) => self.fsevents.stop(h, stream), + } + } +} + +#[cfg(not(any(target_os = "linux", target_os = "android")))] +fn vnode(h: Handle, path: &FsPath, poller: &mut P) -> Result { + use std::os::fd::FromRawFd; + #[cfg(target_vendor = "apple")] + let flags = libc::O_EVTONLY | libc::O_CLOEXEC; + #[cfg(not(target_vendor = "apple"))] + let flags = libc::O_RDONLY | libc::O_CLOEXEC; + // SAFETY: NUL-terminated prepared path and integer flags. + let fd = unsafe { libc::open(path.native().as_ptr(), flags) }; + if fd < 0 { + return Err(super::poller::last_error()); + } + // SAFETY: open returned a new descriptor owned by nobody else. + let fd = unsafe { std::os::fd::OwnedFd::from_raw_fd(fd) }; + // SAFETY: kevent is plain C data; every field this filter reads is set below. + let mut change: libc::kevent = unsafe { std::mem::zeroed() }; + change.ident = fd.as_raw_fd() as _; + change.filter = libc::EVFILT_VNODE as _; + change.flags = (libc::EV_ADD | libc::EV_CLEAR) as _; + change.fflags = libc::NOTE_ATTRIB + | libc::NOTE_WRITE + | libc::NOTE_RENAME + | libc::NOTE_DELETE + | libc::NOTE_EXTEND + | libc::NOTE_REVOKE; + change.udata = h.key() as usize as _; + // SAFETY: live kqueue and descriptor, one initialized change, no output wait. + if unsafe { + libc::kevent( + poller.fd(), + &change, + 1, + std::ptr::null_mut(), + 0, + std::ptr::null(), + ) + } < 0 + { + return Err(super::poller::last_error()); + } + Ok(fd) +} diff --git a/crates/turnloop/src/backend/web.rs b/crates/turnloop/src/backend/web.rs index 23f4618..560b28a 100644 --- a/crates/turnloop/src/backend/web.rs +++ b/crates/turnloop/src/backend/web.rs @@ -240,6 +240,7 @@ unsafe impl Backend for Web { request.operation, Operation::ProcessExit | Operation::WatchSignal + | Operation::WatchFs | Operation::SendHandle(_) | Operation::RecvHandle ) { diff --git a/crates/turnloop/src/blocking.rs b/crates/turnloop/src/blocking.rs index 247a044..a995186 100644 --- a/crates/turnloop/src/blocking.rs +++ b/crates/turnloop/src/blocking.rs @@ -1,5 +1,7 @@ //! Lazily started process-wide bounded pool, with a completion port per loop. use crate::{BlockingResult, Error, ErrorKind, Notifier, OpId, Payload, Result, queue::Queue}; +#[cfg(not(target_arch = "wasm32"))] +use std::sync::atomic::AtomicUsize; use std::{ net::{SocketAddr, ToSocketAddrs}, sync::{ @@ -34,6 +36,8 @@ pub struct DnsRequest { } pub(crate) enum WorkOutput { ExternalWait(crate::WaitResult), + #[cfg(not(target_arch = "wasm32"))] + Fs(crate::fs::Reply), Blocking(Payload), Resolved(Vec), } @@ -87,7 +91,7 @@ impl WorkPort { let _ = self.notifier.notify(); } } -#[cfg(any(turnloop_backend = "kqueue", turnloop_backend = "epoll"))] +#[cfg(not(target_arch = "wasm32"))] pub(crate) trait ReusableWork: Send + Sync { fn run(&self); } @@ -108,13 +112,18 @@ mod native { } enum Task { Boxed(Job), - #[cfg(any(turnloop_backend = "kqueue", turnloop_backend = "epoll"))] Reusable(Arc), } struct State { jobs: Mutex>, + // Queue capacity promised to accepted operations that will push later. + // Modified only while holding `jobs`: queued + reserved <= queue_capacity. + reserved: AtomicUsize, ready: Condvar, stopping: AtomicBool, + /// Workers that finished thread startup; `start` returns only once all have. + started: Mutex, + started_changed: Condvar, } struct Pool { state: Arc, @@ -124,8 +133,11 @@ mod native { fn start(config: PoolConfig) -> Result { let state = Arc::new(State { jobs: Mutex::new(VecDeque::with_capacity(config.queue_capacity)), + reserved: AtomicUsize::new(0), ready: Condvar::new(), stopping: AtomicBool::new(false), + started: Mutex::new(0), + started_changed: Condvar::new(), }); let mut handles = Vec::with_capacity(config.threads); for i in 0..config.threads { @@ -133,6 +145,12 @@ mod native { let worker = thread::Builder::new() .name(format!("turnloop-blocking-{i}")) .spawn(move || { + // The runtime's per-thread setup (thread name, current-thread + // handle, TLS destructor lists; allocating on Windows) has run. + *s.started + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) += 1; + s.started_changed.notify_all(); loop { let job = { let mut jobs = s @@ -150,12 +168,8 @@ mod native { } jobs.pop_front().expect("nonempty job queue") }; - #[cfg(not(any(turnloop_backend = "kqueue", turnloop_backend = "epoll")))] - let Task::Boxed(job) = job; - #[cfg(any(turnloop_backend = "kqueue", turnloop_backend = "epoll"))] let job = match job { Task::Boxed(job) => job, - #[cfg(any(turnloop_backend = "kqueue", turnloop_backend = "epoll"))] Task::Reusable(work) => { work.run(); continue; @@ -182,6 +196,20 @@ mod native { } } } + // Return only after every worker finished starting. Thread startup runs + // asynchronously and allocates on some platforms; it must not overlap the + // first steady-state jobs of any loop. + let mut started = state + .started + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + while *started < config.threads { + started = state + .started_changed + .wait(started) + .unwrap_or_else(std::sync::PoisonError::into_inner); + } + drop(started); // Workers live for the process lifetime, shared by every submitting loop. Ok(Pool { state, config }) } @@ -207,7 +235,7 @@ mod native { .jobs .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); - if jobs.len() == pool.config.queue_capacity { + if jobs.len() + pool.state.reserved.load(Ordering::Relaxed) >= pool.config.queue_capacity { return Err(Error::new(ErrorKind::ResourceLimit)); } jobs.push_back(Task::Boxed(Job { @@ -220,8 +248,7 @@ mod native { pool.state.ready.notify_one(); Ok(()) } - #[cfg(any(turnloop_backend = "kqueue", turnloop_backend = "epoll"))] - pub(crate) fn reusable(config: PoolConfig, work: Arc) -> Result<()> { + fn pool(config: PoolConfig) -> Result<&'static Pool> { if config.threads == 0 || config.queue_capacity == 0 { return Err(Error::new(ErrorKind::InvalidInput)); } @@ -232,18 +259,61 @@ mod native { if pool.config != config { return Err(Error::new(ErrorKind::InvalidInput)); } - let mut jobs = pool - .state + Ok(pool) + } + fn lock(pool: &Pool) -> std::sync::MutexGuard<'_, VecDeque> { + pool.state .jobs .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - if jobs.len() == pool.config.queue_capacity { + .unwrap_or_else(std::sync::PoisonError::into_inner) + } + #[cfg(any(turnloop_backend = "kqueue", turnloop_backend = "epoll"))] + pub(crate) fn reusable(config: PoolConfig, work: Arc) -> Result<()> { + let pool = pool(config)?; + let mut jobs = lock(pool); + if jobs.len() + pool.state.reserved.load(Ordering::Relaxed) >= pool.config.queue_capacity { return Err(Error::new(ErrorKind::ResourceLimit)); } jobs.push_back(Task::Reusable(work)); pool.state.ready.notify_one(); Ok(()) } + /// Promise one queue slot to an accepted operation (ResourceLimit when full). + pub(crate) fn reserve(config: PoolConfig) -> Result<()> { + let pool = pool(config)?; + let jobs = lock(pool); + let reserved = pool.state.reserved.load(Ordering::Relaxed); + if jobs.len() + reserved >= pool.config.queue_capacity { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + pool.state.reserved.store(reserved + 1, Ordering::Relaxed); + Ok(()) + } + /// Return an unused reservation (the operation ended before it was queued). + pub(crate) fn unreserve() { + let pool = POOL + .get() + .and_then(|p| p.as_ref().ok()) + .expect("reserved pool"); + let _jobs = lock(pool); + let reserved = pool.state.reserved.load(Ordering::Relaxed); + pool.state.reserved.store(reserved - 1, Ordering::Relaxed); + } + /// Queue reusable work into its reservation. Never fails and never grows the + /// preallocated queue, because the reservation already counted this slot. + pub(crate) fn push_reserved(work: Arc) { + let pool = POOL + .get() + .and_then(|p| p.as_ref().ok()) + .expect("reserved pool"); + let mut jobs = lock(pool); + let reserved = pool.state.reserved.load(Ordering::Relaxed); + pool.state.reserved.store(reserved - 1, Ordering::Relaxed); + debug_assert!(jobs.len() < pool.config.queue_capacity); + jobs.push_back(Task::Reusable(work)); + drop(jobs); + pool.state.ready.notify_one(); + } } pub(crate) fn submit( config: PoolConfig, @@ -336,3 +406,5 @@ mod models { #[cfg(any(turnloop_backend = "kqueue", turnloop_backend = "epoll"))] pub(crate) use native::reusable; +#[cfg(not(target_arch = "wasm32"))] +pub(crate) use native::{push_reserved, reserve, unreserve}; diff --git a/crates/turnloop/src/buffer.rs b/crates/turnloop/src/buffer.rs index 2496f14..2395843 100644 --- a/crates/turnloop/src/buffer.rs +++ b/crates/turnloop/src/buffer.rs @@ -137,7 +137,9 @@ impl BufferPool { #[cfg(any( turnloop_backend = "kqueue", turnloop_backend = "epoll", - turnloop_backend = "iocp" + turnloop_backend = "iocp", + turnloop_backend = "wasi_p2", + all(turnloop_backend = "wasi_p3", feature = "wasi-p3-experimental") ))] pub(crate) fn available(&self) -> bool { !self.inner.borrow().is_empty() diff --git a/crates/turnloop/src/completion.rs b/crates/turnloop/src/completion.rs index 9460086..32a8ebd 100644 --- a/crates/turnloop/src/completion.rs +++ b/crates/turnloop/src/completion.rs @@ -32,6 +32,15 @@ pub struct Completion { #[derive(Debug)] /// Host-visible I/O, timer, process, service and lifecycle results. pub enum OpResult { + /// A typed filesystem result. + Fs(crate::FsResult), + /// Nonterminal batch of filesystem watch records; parse with `WatchEvents`. + Watch { + /// Initialized record bytes, in native event order. + events: BufLease, + /// Events were lost before this batch; rescan the watched scope. + overflow: bool, + }, /// Completion of a registered host wait condition. ExternalWait(crate::WaitResult), /// Child exit, after reaping; produced exactly once per accepted spawn. diff --git a/crates/turnloop/src/driver.rs b/crates/turnloop/src/driver.rs index ee9feac..0ee3c1c 100644 --- a/crates/turnloop/src/driver.rs +++ b/crates/turnloop/src/driver.rs @@ -1,5 +1,6 @@ use crate::{ - backend::{Backend, Event, Operation, Outcome, Request}, + backend::{Backend, Event, Filesystem, Operation, Outcome, Request}, + fs::FsOutput, table::Table, timer::DriverTimerQueue as TimerQueue, *, @@ -20,6 +21,8 @@ const POST_EVENTS: usize = 2; #[derive(Clone, Copy)] enum Kind { Socket, + /// A typed file or directory handle served by the native pool service. + File, Timer { op: Option, repeat: Option, @@ -27,6 +30,8 @@ enum Kind { } struct Resource { kind: Kind, + /// Reserved by an in-flight open; addressable only after `Opened`. + hidden: bool, referenced: bool, pending: usize, closing: Option, @@ -42,6 +47,8 @@ struct Op { timed_out: bool, stop: bool, external_wait: bool, + /// A typed filesystem request (pool service or backend, per `B::FILESYSTEM`). + fs: bool, /// Counted in `native_pending`: a socket-handle operation or a request the /// backend accepted natively (such as WASI DNS). DESIGN §10 rule 3 keys /// queued-turn discovery on these operations. @@ -71,6 +78,9 @@ pub struct Driver { poster: Poster, external: bool, work_port: std::sync::Arc, + #[cfg(not(target_arch = "wasm32"))] + files: crate::fs::Service, + metadata: crate::fs::MetadataPool, owner: u64, thread: ThreadId, handles: Table, @@ -105,10 +115,8 @@ impl Driver { .and_then(|n| n.checked_add(config.max_operations)) .and_then(|n| n.checked_add(config.max_handles)) .ok_or(Error::new(ErrorKind::InvalidInput))?; - let mut backend = B::new( - &config, - BufferPool::new(config.pooled_buffers, config.pooled_buffer_size), - )?; + let buffers = BufferPool::new(config.pooled_buffers, config.pooled_buffer_size); + let mut backend = B::new(&config, buffers.clone())?; let notifier = Notifier::new(backend.waker()); backend.set_notifier(notifier.clone()); let poster = Poster::new(config.post_capacity, notifier.clone()); @@ -130,6 +138,9 @@ impl Driver { notifier, poster, external: false, + #[cfg(not(target_arch = "wasm32"))] + files: crate::fs::Service::new(&config, work_port.clone(), buffers), + metadata: crate::fs::MetadataPool::new(config.max_operations), work_port, owner, thread: thread::current().id(), @@ -164,6 +175,7 @@ impl Driver { } self.handles .get(h.key) + .filter(|r| !r.hidden) .ok_or(Error::new(ErrorKind::NotFound)) } fn new_handle(&mut self, kind: Kind) -> Result { @@ -171,6 +183,7 @@ impl Driver { .handles .insert(Resource { kind, + hidden: false, referenced: true, pending: 0, closing: None, @@ -179,7 +192,7 @@ impl Driver { tail: None, }) .ok_or(Error::new(ErrorKind::ResourceLimit))?; - if matches!(kind, Kind::Socket) { + if matches!(kind, Kind::Socket | Kind::File) { self.refs += 1; } Ok(Handle { @@ -205,6 +218,7 @@ impl Driver { stop: false, job_cancel: None, external_wait: false, + fs: false, native, previous, next: None, @@ -360,7 +374,8 @@ impl Driver { /// Include or exclude a handle and its pending/queued operations from loop liveness. pub fn set_ref(&mut self, h: Handle, referenced: bool) -> Result<()> { let r = self.resource(h)?; - let weight = r.pending + usize::from(matches!(r.kind, Kind::Socket) || r.closing.is_some()); + let weight = r.pending + + usize::from(matches!(r.kind, Kind::Socket | Kind::File) || r.closing.is_some()); if r.referenced != referenced { if referenced { self.refs += weight; @@ -761,6 +776,9 @@ impl Driver { crate::external_wait::cancel(id); #[cfg(target_arch = "wasm32")] self.backend.deadline_changed(self.next_deadline()); + } else if op.fs && B::FILESYSTEM == Filesystem::Pool { + #[cfg(not(target_arch = "wasm32"))] + self.files.cancel(id); } else if let Some(cancel) = op.job_cancel { cancel.store(true, Ordering::Release); } else if self.backend.cancel(id).is_err() { @@ -786,7 +804,9 @@ impl Driver { if r.closing.is_some() { return Err(Error::new(ErrorKind::InvalidInput)); } - self.backend.prepare_close(h)?; + if !matches!(r.kind, Kind::File) { + self.backend.prepare_close(h)?; + } let r = self.handles.get_mut(h.key).expect("validated"); r.closing = Some(token); // Socket handles already carry a reference. Inactive timer handles do @@ -855,6 +875,148 @@ impl Driver { self.backend.deadline_changed(self.next_deadline()); Ok(op) } + /// Submit a typed filesystem request; see [`FsRequest`] for ordering and + /// ownership. Native targets run it on the shared blocking pool, where queue + /// exhaustion rejects the request with ResourceLimit before acceptance. Requests + /// naming a handle need a visible, open file handle that is not closing. + pub fn fs(&mut self, request: FsRequest, token: Token) -> Result { + if B::FILESYSTEM == Filesystem::Unsupported { + return Err(Error::new(ErrorKind::Unsupported)); + } + let kind = if B::FILESYSTEM == Filesystem::Pool { + Kind::File + } else { + Kind::Socket + }; + let target = request.handle(); + if let Some(h) = target { + let r = self.resource(h)?; + if r.closing.is_some() + || std::mem::discriminant(&r.kind) != std::mem::discriminant(&kind) + { + return Err(Error::new(ErrorKind::InvalidInput)); + } + } + let opened = if request.opens() { + let h = self.new_handle(kind)?; + self.handles.get_mut(h.key).expect("new handle").hidden = true; + Some(h) + } else { + None + }; + let handle = opened.or(target); + let op = match self.new_op(handle, token) { + Ok(op) => op, + Err(e) => { + if let Some(h) = opened { + self.discard_hidden(h, false); + } + return Err(e); + } + }; + self.ops.get_mut(op.key).expect("new request").fs = true; + let accepted = match B::FILESYSTEM { + #[cfg(not(target_arch = "wasm32"))] + Filesystem::Pool => self.files.submit(op, handle, request), + _ => self.backend.fs(op, handle, request), + }; + if let Err(e) = accepted { + self.retire(op); + self.outstanding -= 1; + if let Some(h) = opened { + self.discard_hidden(h, false); + } + return Err(e); + } + // DESIGN §10 rule 3: a request the backend executes needs its native step + // even while posts are queued, like a natively accepted lookup. Requests on + // backend handles are already counted; path requests are counted here. Pool + // requests are not native: their results arrive as queued work. + if B::FILESYSTEM == Filesystem::Backend { + let op = self.ops.get_mut(op.key).expect("accepted request"); + if !op.native { + op.native = true; + self.native_pending += 1; + } + } + Ok(op) + } + /// Whether a request waiting for a pooled lease can start now. + fn files_waiting(&self) -> bool { + #[cfg(not(target_arch = "wasm32"))] + { + self.files.has_work() + } + #[cfg(target_arch = "wasm32")] + { + false + } + } + /// Remove a handle that never became visible, releasing any native object. + fn discard_hidden(&mut self, h: Handle, native: bool) { + let r = self.handles.remove(h.key).expect("hidden handle"); + debug_assert!(r.hidden); + if r.referenced { + self.refs -= 1; + } + if native { + match r.kind { + #[cfg(not(target_arch = "wasm32"))] + Kind::File => self.files.release(h), + _ => self.backend.release(h), + } + } + } + /// Deliver a typed filesystem terminal result, publishing or discarding an + /// opened handle. A failed or cancelled open never exposes its handle. + fn finish_fs(&mut self, id: OpId, result: OpResult) { + let Some(h) = self.ops.get(id.key).and_then(|op| op.handle) else { + self.finish(id, result, true); + return; + }; + if !self.handles.get(h.key).is_some_and(|r| r.hidden) { + self.finish(id, result, true); + return; + } + if matches!(result, OpResult::Fs(FsResult::Opened(_))) { + self.handles.get_mut(h.key).expect("hidden handle").hidden = false; + self.finish(id, result, true); + return; + } + self.finish(id, result, true); + let q = self.queued.back_mut().expect("queued open result"); + debug_assert_eq!(q.completion.op, Some(id)); + q.completion.handle = None; + self.discard_hidden(h, true); + } + /// Watch a file or directory with the platform's native API: inotify (Linux), + /// FSEvents for directories and kqueue for files (macOS), kqueue (BSD/iOS), + /// ReadDirectoryChangesW (Windows). Nonterminal `OpResult::Watch` batches follow + /// until `fs_watch_stop` (Stopped, then Closed) or `close` (Cancelled, then Closed). + /// WASI and the web return Unsupported. Node `watchFile` is a host timer plus `Stat`. + pub fn fs_watch( + &mut self, + path: &FsPath, + options: WatchOptions, + token: Token, + ) -> Result { + let h = self.new_handle(Kind::Socket)?; + if let Err(e) = self + .backend + .fs_watch(h, path, options) + .and_then(|()| self.submit(h, Operation::WatchFs, token).map(|_| ())) + { + self.backend.release(h); + self.handles.remove(h.key); + self.refs -= 1; + return Err(e); + } + Ok(h) + } + /// Stop a filesystem watch, delivering Stopped before the final Closed. + pub fn fs_watch_stop(&mut self, h: Handle, token: Token) -> Result<()> { + self.signal_stop(h, token) + } /// Submit an owned Send closure to the bounded shared blocking pool. pub fn blocking BlockingResult + Send + 'static>( &mut self, @@ -921,6 +1083,7 @@ impl Driver { !self.queued.is_empty() || !self.poster.is_empty() || !self.work_port.is_empty() + || self.files_waiting() || self.backend.has_work(), )?; Ok(integration) @@ -949,8 +1112,11 @@ impl Driver { if let Some(h) = closed && let Some(r) = self.handles.remove(h.key) { - if matches!(r.kind, Kind::Socket) { - self.backend.release(h); + match r.kind { + Kind::Socket => self.backend.release(h), + #[cfg(not(target_arch = "wasm32"))] + Kind::File => self.files.release(h), + _ => {} } if r.referenced { self.refs -= 1; @@ -965,6 +1131,21 @@ impl Driver { let Some(op) = self.ops.get(e.op.key).cloned() else { return; }; + if op.fs && e.terminal { + let result = if op.cancel { + OpResult::Cancelled + } else { + match e.result { + Ok(Outcome::Fs { output, lease }) => { + fs_result(&self.metadata, op.handle, output, lease) + } + Err(e) => OpResult::Err(e), + Ok(_) => OpResult::Err(Error::new(ErrorKind::Other)), + } + }; + self.finish_fs(e.op, result); + return; + } let result = if op.cancel { if !e.terminal { return; @@ -1002,6 +1183,10 @@ impl Driver { Ok(Outcome::Wrote(n)) => OpResult::Wrote(n), Ok(Outcome::RecvFrom { n, from, lease }) => OpResult::RecvFrom { n, from, lease }, Ok(Outcome::Shutdown) => OpResult::Shutdown, + Ok(Outcome::Watch { events, overflow }) => OpResult::Watch { events, overflow }, + Ok(Outcome::Fs { output, lease }) => { + fs_result(&self.metadata, op.handle, output, lease) + } Ok(Outcome::Cancelled) => OpResult::Cancelled, } }; @@ -1012,6 +1197,9 @@ impl Driver { self.assert_owner(); self.backend.validate_timeout(timeout)?; out.clear(); + // Leases released by the host since the last turn can start waiting requests. + #[cfg(not(target_arch = "wasm32"))] + self.files.retry(); let notified = self.notifier.begin(); let start = self.backend.now(); self.expire_connects(start)?; @@ -1101,6 +1289,21 @@ impl Driver { let Some(op) = self.ops.get(work.op.key) else { continue; }; + #[cfg(not(target_arch = "wasm32"))] + if op.fs { + let (cancel, handle) = (op.cancel, op.handle); + let (lease, metadata) = self.files.complete(work.op); + let result = match work.result { + _ if cancel => OpResult::Cancelled, + Ok(crate::blocking::WorkOutput::Fs(reply)) => { + fs_result(&self.metadata, handle, reply.output(metadata), lease) + } + Ok(_) => OpResult::Err(Error::new(ErrorKind::Other)), + Err(e) => OpResult::Err(e), + }; + self.finish_fs(work.op, result); + continue; + } let result = if op.cancel { if op.stop { OpResult::Stopped @@ -1109,6 +1312,10 @@ impl Driver { } } else { match work.result { + #[cfg(not(target_arch = "wasm32"))] + Ok(crate::blocking::WorkOutput::Fs(_)) => { + OpResult::Err(Error::new(ErrorKind::Other)) + } Ok(crate::blocking::WorkOutput::ExternalWait(r)) => OpResult::ExternalWait(r), Ok(crate::blocking::WorkOutput::Blocking(p)) => OpResult::Blocking(p), Ok(crate::blocking::WorkOutput::Resolved(a)) => OpResult::Resolved(a), @@ -1137,12 +1344,16 @@ impl Driver { false, ); } + // Cancelled or failed reads may have returned leases during this turn. + #[cfg(not(target_arch = "wasm32"))] + self.files.retry(); self.drain(out); if self.external { self.notifier.external_park( !self.queued.is_empty() || !self.poster.is_empty() || !self.work_port.is_empty() + || self.files_waiting() || self.backend.has_work(), )?; } @@ -1157,6 +1368,38 @@ impl Driver { } } +/// Attach the lease and convert a worker or backend filesystem result. +fn fs_result( + metadata: &crate::fs::MetadataPool, + handle: Option, + output: FsOutput, + mut lease: Option, +) -> OpResult { + let mut fill = |n: usize| { + if let Some(lease) = &mut lease { + lease.set_len(n); + } + }; + OpResult::Fs(match output { + FsOutput::Opened => FsResult::Opened(handle.expect("open request handle")), + FsOutput::Read(n) => { + fill(n); + FsResult::Read { n, lease } + } + FsOutput::Wrote(n) => FsResult::Wrote(n), + FsOutput::Metadata(m) => FsResult::Metadata(metadata.lease(m)), + FsOutput::Directory { n, eof } => { + fill(n); + FsResult::Directory { n, lease, eof } + } + FsOutput::Bytes(n) => { + fill(n); + FsResult::Bytes { n, lease } + } + FsOutput::Done => FsResult::Done, + }) +} + impl Drop for Driver { fn drop(&mut self) { crate::external_wait::close(self.owner); @@ -1285,6 +1528,7 @@ mod clock_contract { cached_work: bool, lookup: Option, lookup_ready: bool, + file_request: Option, accept_connect: bool, pending: Option, cancellation: Option, @@ -1297,6 +1541,14 @@ mod clock_contract { unsafe impl Backend for Host { type Wake = NoWake; type Detached = (); + const FILESYSTEM: Filesystem = Filesystem::Backend; + fn fs(&mut self, op: OpId, handle: Option, request: FsRequest) -> Result<()> { + if handle.is_some() || !matches!(request, FsRequest::Stat { .. }) { + return Err(Error::new(ErrorKind::Unsupported)); + } + assert!(self.file_request.replace(op).is_none()); + Ok(()) + } fn new(_: &Config, _: BufferPool) -> Result { Ok(Self { now: Instant::now(), @@ -1306,6 +1558,7 @@ mod clock_contract { cached_work: false, lookup: None, lookup_ready: false, + file_request: None, accept_connect: false, pending: None, cancellation: None, @@ -1379,6 +1632,18 @@ mod clock_contract { ) -> Result { assert_eq!(timeout, Some(Duration::ZERO)); self.polls += 1; + // Like the WASI backends: accepted filesystem requests run in poll. + if let Some(op) = self.file_request.take() { + events.push(Event { + op, + terminal: true, + result: Ok(Outcome::Fs { + output: FsOutput::Done, + lease: None, + }), + }); + return Ok(PollInfo::default()); + } if self.lookup_ready && let Some(op) = self.lookup.take() { @@ -1478,6 +1743,61 @@ mod clock_contract { assert!(driver.backend.lookup.is_none()); } + #[test] + fn backend_path_request_is_a_pending_native_operation() { + let mut driver = Driver::::new(Config::default()).expect("host loop"); + let path = FsPath::new("/synthetic").expect("path"); + let mut out = Completions::with_capacity(1); + // Sustained posts: without the native flag, every turn has queued work and + // no native operation, so the backend (where the request runs) is skipped. + driver + .poster() + .post(Token(2), Payload::U64(0)) + .expect("post"); + let op = driver + .fs( + FsRequest::Stat { + path, + follow_symlinks: true, + }, + Token(1), + ) + .expect("backend accepts the path request"); + assert_eq!(driver.native_pending, 1, "path request counted as native"); + let (mut done, mut posts) = (0, 0); + for turn in 0..4 { + let polls = driver.backend.polls; + let info = driver.turn(Timeout::Now, &mut out).expect("turn"); + assert_eq!(info.os_waits, 0, "queued turns never block"); + assert!(info.os_waits + info.discovery_polls <= 1); + assert_eq!( + driver.backend.polls - polls, + usize::from(turn == 0), + "one native step while the request is pending, none after" + ); + for c in out.drain() { + match c.result { + OpResult::Fs(FsResult::Done) => { + assert_eq!(c.op, Some(op)); + done += 1; + } + OpResult::Posted(_) => posts += 1, + other => panic!("unexpected {other:?}"), + } + } + driver + .poster() + .post(Token(2), Payload::U64(turn)) + .expect("replenish"); + } + assert_eq!(done, 1, "the request completed despite queued posts"); + assert_eq!( + driver.native_pending, 0, + "terminal completion retires the flag" + ); + assert!(posts >= 3); + } + #[test] fn connection_deadline_waits_for_acknowledgement_and_retains_cancellation_errors() { let mut driver = Driver::::new(Config::default()).expect("host loop"); diff --git a/crates/turnloop/src/fs/mod.rs b/crates/turnloop/src/fs/mod.rs new file mode 100644 index 0000000..146d493 --- /dev/null +++ b/crates/turnloop/src/fs/mod.rs @@ -0,0 +1,781 @@ +//! Typed filesystem requests, native watches and their portable byte records. +//! +//! * **Native** (Linux, macOS/BSD, Windows): every request runs on the shared +//! bounded blocking pool (DESIGN D8). Operations on one file or directory handle +//! run in submission order; independent path requests may complete in any order. +//! Adopted descriptors (`Detached::from_fd`/`from_handle`, stdio) keep their stream +//! paths: reusable pool jobs on Unix, one synchronous worker per handle on Windows. +//! * **WASI 0.2/0.3**: the same requests map onto `wasi:filesystem`, resolved only +//! against preopened directories. There is no ambient path authority. +//! * **Web**: Unsupported. No OPFS host mapping is defined. +//! +//! Every accepted request completes exactly once with `OpResult::Fs`, `Err`, +//! or `Cancelled`. A request that has started on a pool thread runs to its end; +//! cancellation then reports `Cancelled` and discards its result. Buffers follow +//! the D3 contract: provided memory is accessed only until the terminal completion. +use crate::{BufLease, Handle, ReadBuf, Result, WriteBuf}; +use std::{path::Path, sync::Arc}; + +#[derive(Debug)] +struct PathInner { + path: std::path::PathBuf, + #[cfg(unix)] + native: std::ffi::CString, + #[cfg(windows)] + native: Box<[u16]>, + #[cfg(target_os = "wasi")] + text: Box, +} + +#[derive(Clone, Debug)] +/// A validated, prepared pathname. Preparing converts once; cloning never allocates. +/// +/// On WASI an absolute path resolves against the preopened directory with the +/// longest matching name, and a relative path against a preopen named `.`. +/// Paths outside every preopen fail with `NotFound`; the host enforces confinement. +pub struct FsPath(Arc); +impl FsPath { + /// Prepare a pathname. Embedded NULs (and non-UTF-8 names on WASI) are InvalidInput. + pub fn new(path: impl AsRef) -> Result { + let path = path.as_ref(); + #[cfg(unix)] + let native = { + use std::os::unix::ffi::OsStrExt; + std::ffi::CString::new(path.as_os_str().as_bytes()).map_err(|_| invalid())? + }; + #[cfg(windows)] + let native = { + use std::os::windows::ffi::OsStrExt; + let mut wide: Vec = path.as_os_str().encode_wide().collect(); + if wide.contains(&0) { + return Err(invalid()); + } + wide.push(0); + wide.into_boxed_slice() + }; + #[cfg(target_os = "wasi")] + let text = { + let text = path.to_str().ok_or_else(invalid)?; + if text.contains('\0') { + return Err(invalid()); + } + text.into() + }; + #[cfg(not(any(unix, windows, target_os = "wasi")))] + if path.to_str().is_none_or(|s| s.contains('\0')) { + return Err(invalid()); + } + Ok(Self(Arc::new(PathInner { + path: path.to_path_buf(), + #[cfg(any(unix, windows))] + native, + #[cfg(target_os = "wasi")] + text, + }))) + } + /// The original pathname, for host diagnostics and error messages. + pub fn as_path(&self) -> &Path { + &self.0.path + } + #[cfg(unix)] + pub(crate) fn native(&self) -> &std::ffi::CStr { + &self.0.native + } + #[cfg(windows)] + pub(crate) fn wide(&self) -> &[u16] { + &self.0.native + } + #[cfg(target_os = "wasi")] + #[allow(dead_code)] + pub(crate) fn text(&self) -> &str { + &self.0.text + } +} +fn invalid() -> crate::Error { + crate::Error::new(crate::ErrorKind::InvalidInput) +} + +#[derive(Clone, Copy, Debug)] +/// Open options covering Node's string and numeric open flags. +pub struct FileOptions { + /// Allow reads. + pub read: bool, + /// Allow writes. + pub write: bool, + /// Create the file when it is absent. + pub create: bool, + /// Fail with AlreadyExists when the file exists (implies create). + pub exclusive: bool, + /// Truncate an existing file at open. + pub truncate: bool, + /// Every write appends, including writes that name an offset (as on Linux). + pub append: bool, + /// Synchronous file-integrity writes (O_SYNC; FILE_FLAG_WRITE_THROUGH on Windows). + pub sync: bool, + /// Synchronous data-integrity writes (O_DSYNC; FILE_FLAG_WRITE_THROUGH on Windows). + pub data_sync: bool, + /// Follow a final symbolic link; false fails on a link (O_NOFOLLOW). + pub follow_symlinks: bool, + /// Creation permissions filtered by the umask. Windows maps a missing owner + /// write bit to the read-only attribute; WASI uses host defaults. + pub mode: u32, +} +impl Default for FileOptions { + fn default() -> Self { + Self { + read: true, + write: false, + create: false, + exclusive: false, + truncate: false, + append: false, + sync: false, + data_sync: false, + follow_symlinks: true, + mode: 0o666, + } + } +} + +#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)] +/// Native file type; also the record tag of directory entries. +pub enum FileType { + /// The native API did not report a type. + Unknown = 0, + /// Regular file. + File = 1, + /// Directory. + Directory = 2, + /// Symbolic link (or a Windows name-surrogate reparse point). + Symlink = 3, + /// Block device. + BlockDevice = 4, + /// Character device. + CharDevice = 5, + /// Named pipe. + Fifo = 6, + /// Local socket. + Socket = 7, +} +impl FileType { + fn from_tag(tag: u8) -> Self { + match tag { + 1 => Self::File, + 2 => Self::Directory, + 3 => Self::Symlink, + 4 => Self::BlockDevice, + 5 => Self::CharDevice, + 6 => Self::Fifo, + 7 => Self::Socket, + _ => Self::Unknown, + } + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd)] +/// Wall-clock timestamp relative to the Unix epoch, with nanoseconds retained. +pub struct FileTime { + /// Signed whole seconds from the Unix epoch. + pub seconds: i64, + /// Fractional nanoseconds, below one billion. + pub nanoseconds: u32, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +/// Metadata for Node `Stats` and `watchFile` snapshots. A field the platform does +/// not report is None; values are never invented. Windows `mode` follows libuv's +/// attribute mapping: type bits plus 0o666, or 0o444 when read-only. +pub struct FileMetadata { + /// File type. + pub kind: FileType, + /// Length in bytes. + pub size: u64, + /// Mode including type bits. + pub mode: Option, + /// Device (Windows: volume serial number). + pub device: Option, + /// Inode (Windows: file index). + pub inode: Option, + /// Hard-link count. + pub links: Option, + /// Owner user id. + pub uid: Option, + /// Owner group id. + pub gid: Option, + /// Device id of a special file. + pub rdev: Option, + /// Preferred I/O block size. + pub block_size: Option, + /// Allocated 512-byte blocks. + pub blocks: Option, + /// Last access. + pub accessed: Option, + /// Last content modification. + pub modified: Option, + /// Last status change. + pub changed: Option, + /// Creation (birth) time. + pub created: Option, +} + +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +/// `access(2)` checks. All false tests existence only. +pub struct AccessMode { + /// Readable by the caller. + pub read: bool, + /// Writable by the caller (Windows: not read-only, or a directory). + pub write: bool, + /// Executable or searchable by the caller. + pub execute: bool, +} + +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +/// Windows symbolic-link flavour; ignored elsewhere. +pub enum SymlinkKind { + /// File symbolic link. + #[default] + File, + /// Directory symbolic link. + Directory, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +/// Timestamp update for `SetTimes`. +pub enum TimeChange { + /// Leave this timestamp unchanged. + Keep, + /// Set it to the current time. + Now, + /// Set it to this time. + At(FileTime), +} + +#[derive(Clone, Debug)] +/// A metadata operation's subject: a path or an open handle. +pub enum FsTarget { + /// Address by path. + Path { + /// Prepared path. + path: FsPath, + /// Operate on the link target rather than a final symbolic link. + follow_symlinks: bool, + }, + /// Address an open file or directory handle. + File(Handle), +} + +#[derive(Debug)] +/// One typed request, submitted with `Loop::fs`. Operations on the same handle run +/// FIFO. A cancelled mutation that already started is not rolled back. +pub enum FsRequest { + /// Open a file. Completes with `FsResult::Opened(handle)`; on error or + /// cancellation no handle becomes visible. + Open { + /// Prepared path. + path: FsPath, + /// Access and creation options. + options: FileOptions, + }, + /// Open a directory stream for `ReadDir`. Completes with `Opened(handle)`. + OpenDir { + /// Prepared path. + path: FsPath, + }, + /// Close the native descriptor after earlier queued operations, reporting its + /// error. Later requests fail. Release the handle itself with `Loop::close`, + /// which otherwise closes a still-open descriptor on the loop thread. + Close { + /// Open file or directory. + file: Handle, + }, + /// Read once (Node `fs.read`). `offset: None` reads at and advances the + /// handle's cursor. Zero bytes with a nonempty buffer means end of file. + Read { + /// Open file. + file: Handle, + /// Provided memory or a pooled lease (pool exhaustion delays the start). + buffer: ReadBuf, + /// Positional read offset, or None for the cursor. + offset: Option, + }, + /// Write once (Node `fs.write`); the count may be short. Hosts loop for writeFile. + Write { + /// Open file. + file: Handle, + /// Stable input bytes. + buffer: WriteBuf, + /// Positional write offset, or None for the cursor. + offset: Option, + }, + /// Flush file data, and metadata unless `data_only`. + Sync { + /// Open file. + file: Handle, + /// `fdatasync` instead of `fsync`. + data_only: bool, + }, + /// Set the file length without moving its cursor. + Truncate { + /// Open file. + file: Handle, + /// New length in bytes. + size: u64, + }, + /// Metadata by path (`follow_symlinks: false` is `lstat`). + Stat { + /// Prepared path. + path: FsPath, + /// Follow a final symbolic link. + follow_symlinks: bool, + }, + /// Metadata of an open handle. + Fstat { + /// Open file or directory. + file: Handle, + }, + /// Read directory entry records ([`DirEntries`]); `.` and `..` are omitted. + /// A name never spans completions. An empty page with `eof` ends the stream. + ReadDir { + /// Directory opened with `OpenDir`. + dir: Handle, + /// Record output (ResourceLimit if the next name cannot fit an empty buffer). + buffer: ReadBuf, + }, + /// Create one directory; recursive creation belongs to the host. + Mkdir { + /// Prepared path. + path: FsPath, + /// Permissions filtered by the umask (ignored on Windows and WASI). + mode: u32, + }, + /// Remove an empty directory. + Rmdir { + /// Prepared path. + path: FsPath, + }, + /// Remove a file or symbolic link. + Unlink { + /// Prepared path. + path: FsPath, + }, + /// Rename, replacing an existing destination where the platform allows. + Rename { + /// Existing path. + from: FsPath, + /// New path. + to: FsPath, + }, + /// Create a hard link. + Link { + /// Existing file. + existing: FsPath, + /// New link path. + link: FsPath, + }, + /// Create a symbolic link whose content is `target`'s original text. + Symlink { + /// Link content; not resolved against preopens on WASI. + target: FsPath, + /// New link path. + link: FsPath, + /// Windows link flavour. + kind: SymlinkKind, + }, + /// Read a symbolic link's content into bytes (`FsResult::Bytes`). + ReadLink { + /// Prepared path. + path: FsPath, + /// Output (ResourceLimit if the content does not fit). + buffer: ReadBuf, + }, + /// Resolve an absolute canonical path into bytes (`FsResult::Bytes`). + RealPath { + /// Prepared path. + path: FsPath, + /// Output (ResourceLimit if the path does not fit). + buffer: ReadBuf, + }, + /// Check accessibility (Node `fs.access`). + Access { + /// Prepared path. + path: FsPath, + /// Checked permissions. + mode: AccessMode, + }, + /// Change permissions (Windows: the read-only attribute from the owner write bit). + Chmod { + /// Path or handle. + target: FsTarget, + /// Permission bits. + mode: u32, + }, + /// Change ownership; None leaves an id unchanged. + Chown { + /// Path or handle. + target: FsTarget, + /// New user id. + uid: Option, + /// New group id. + gid: Option, + }, + /// Change access and modification times. + SetTimes { + /// Path or handle. + target: FsTarget, + /// Access time update. + accessed: TimeChange, + /// Modification time update. + modified: TimeChange, + }, + /// Copy file contents (and permissions), failing on an existing destination if exclusive. + CopyFile { + /// Source file. + from: FsPath, + /// Destination file. + to: FsPath, + /// Fail with AlreadyExists when `to` exists. + exclusive: bool, + }, +} +impl FsRequest { + /// The handle whose FIFO this request joins, if any. + pub(crate) fn handle(&self) -> Option { + match self { + Self::Close { file } + | Self::Read { file, .. } + | Self::Write { file, .. } + | Self::Sync { file, .. } + | Self::Truncate { file, .. } + | Self::Fstat { file } + | Self::ReadDir { dir: file, .. } + | Self::Chmod { + target: FsTarget::File(file), + .. + } + | Self::Chown { + target: FsTarget::File(file), + .. + } + | Self::SetTimes { + target: FsTarget::File(file), + .. + } => Some(*file), + _ => None, + } + } + /// Whether the request creates a new handle. + pub(crate) fn opens(&self) -> bool { + matches!(self, Self::Open { .. } | Self::OpenDir { .. }) + } + /// The read buffer to fill, if the request produces bytes. + #[cfg_attr( + any(turnloop_backend = "web", turnloop_backend = "unsupported"), + allow(dead_code) + )] + pub(crate) fn read_buffer(&mut self) -> Option<&mut ReadBuf> { + match self { + Self::Read { buffer, .. } + | Self::ReadDir { buffer, .. } + | Self::ReadLink { buffer, .. } + | Self::RealPath { buffer, .. } => Some(buffer), + _ => None, + } + } +} + +#[derive(Debug)] +/// Successful typed results. Failures use `OpResult::Err`. +pub enum FsResult { + /// A new, visible file or directory handle; release it with `Loop::close`. + Opened(Handle), + /// Bytes read; zero with a nonempty buffer is end of file. + Read { + /// Bytes initialized in the provided buffer or lease. + n: usize, + /// The pooled lease, when `ReadBuf::Pooled` was requested. + lease: Option, + }, + /// Bytes written; may be short. + Wrote(usize), + /// File metadata, in reusable loop storage (keeps completions small). + Metadata(Metadata), + /// Directory entry records ([`DirEntries`]). + Directory { + /// Initialized record bytes. + n: usize, + /// The pooled lease, when requested. + lease: Option, + /// No further entries remain. + eof: bool, + }, + /// Link content or a canonical path (native bytes on Unix, WTF-8 on Windows, UTF-8 on WASI). + Bytes { + /// Initialized bytes. + n: usize, + /// The pooled lease, when requested. + lease: Option, + }, + /// The operation succeeded without a value. + Done, +} + +/// Metadata result held in reusable per-loop storage; dereferences to +/// [`FileMetadata`]. Dropping it returns the storage, so steady-state `stat` +/// requests allocate nothing once the loop has served one. Copy the value out +/// to keep it beyond the completion. +// The pool holds boxes so a lease can move one out and dereference it stably. +#[allow(clippy::vec_box)] +pub struct Metadata { + value: Option>, + pool: std::rc::Rc>>>, +} +impl std::ops::Deref for Metadata { + type Target = FileMetadata; + fn deref(&self) -> &FileMetadata { + self.value.as_ref().expect("live metadata") + } +} +impl std::fmt::Debug for Metadata { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + (**self).fmt(f) + } +} +impl Drop for Metadata { + fn drop(&mut self) { + let mut pool = self.pool.borrow_mut(); + if let Some(value) = self.value.take() + && pool.len() < pool.capacity() + { + pool.push(value); + } + } +} +/// Bounded reusable metadata storage, one per loop. +#[derive(Clone)] +#[allow(clippy::vec_box)] +pub(crate) struct MetadataPool(std::rc::Rc>>>); +impl MetadataPool { + pub fn new(capacity: usize) -> Self { + Self(std::rc::Rc::new(std::cell::RefCell::new( + Vec::with_capacity(capacity), + ))) + } + pub fn lease(&self, metadata: FileMetadata) -> Metadata { + let value = match self.0.borrow_mut().pop() { + Some(mut boxed) => { + *boxed = metadata; + boxed + } + None => Box::new(metadata), + }; + Metadata { + value: Some(value), + pool: self.0.clone(), + } + } +} + +/// Backend- or worker-side typed result; the core attaches leases and handles. +/// Metadata is inline: results travel through fixed, preallocated event storage, +/// and boxing would allocate per `stat`. The pool path keeps metadata in its slot. +#[derive(Debug)] +#[allow(clippy::large_enum_variant)] +pub enum FsOutput { + /// The request's new handle is bound. + Opened, + /// Bytes read. + Read(usize), + /// Bytes written. + Wrote(usize), + /// Metadata. + Metadata(FileMetadata), + /// Directory record bytes. + Directory { + /// Initialized record bytes. + n: usize, + /// No further entries remain. + eof: bool, + }, + /// Result bytes (link content or canonical path). + Bytes(usize), + /// Success without a value. + Done, +} + +/// Size of a record header: tag, reserved zero, little-endian u16 name length. +pub const RECORD_HEADER: usize = 4; +/// Append one record; false when it does not fit the remaining space. +#[cfg_attr(target_arch = "wasm32", allow(dead_code))] +pub(crate) fn put_record(output: &mut [u8], used: &mut usize, tag: u8, name: &[u8]) -> bool { + let Ok(len) = u16::try_from(name.len()) else { + return false; + }; + let end = *used + RECORD_HEADER + name.len(); + if end > output.len() { + return false; + } + output[*used] = tag; + output[*used + 1] = 0; + output[*used + 2..*used + 4].copy_from_slice(&len.to_le_bytes()); + output[*used + RECORD_HEADER..end].copy_from_slice(name); + *used = end; + true +} +#[derive(Clone, Debug)] +struct Records<'a>(&'a [u8]); +impl<'a> Iterator for Records<'a> { + type Item = (u8, &'a [u8]); + fn next(&mut self) -> Option { + if self.0.len() < RECORD_HEADER { + return None; + } + let len = u16::from_le_bytes([self.0[2], self.0[3]]) as usize; + let end = (RECORD_HEADER + len).min(self.0.len()); + let record = (self.0[0], &self.0[RECORD_HEADER..end]); + self.0 = &self.0[end..]; + Some(record) + } +} + +#[derive(Clone, Debug)] +/// Iterate `(type, name)` directory records from a `FsResult::Directory` page. +/// Names are native bytes on Unix, WTF-8 on Windows and UTF-8 on WASI. +pub struct DirEntries<'a>(Records<'a>); +impl<'a> DirEntries<'a> { + /// Parse the initialized bytes of one page. + pub fn new(bytes: &'a [u8]) -> Self { + Self(Records(bytes)) + } +} +impl<'a> Iterator for DirEntries<'a> { + type Item = (FileType, &'a [u8]); + fn next(&mut self) -> Option { + self.0 + .next() + .map(|(tag, name)| (FileType::from_tag(tag), name)) + } +} + +#[derive(Clone, Copy, Debug, Default)] +/// Filesystem watch options. +pub struct WatchOptions { + /// Include all descendants. Supported by FSEvents (macOS) and + /// ReadDirectoryChangesW (Windows); inotify and kqueue return Unsupported, + /// and hosts emulate recursion with one watch per directory, as Node does on Linux. + pub recursive: bool, +} + +#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)] +/// Node `fs.watch` event type. +pub enum WatchKind { + /// An entry appeared, disappeared or was renamed (or the watched root itself). + Rename = 1, + /// Contents or metadata changed. + Change = 2, +} + +#[derive(Clone, Debug)] +/// Iterate `(kind, name)` records of an `OpResult::Watch` batch, in native order. +/// Names are relative to the watched directory using native separators; an event +/// on the watched root (or a watched file) carries the root's final component. +pub struct WatchEvents<'a>(Records<'a>); +impl<'a> WatchEvents<'a> { + /// Parse the initialized bytes of one batch. + pub fn new(bytes: &'a [u8]) -> Self { + Self(Records(bytes)) + } +} +impl<'a> Iterator for WatchEvents<'a> { + type Item = (WatchKind, &'a [u8]); + fn next(&mut self) -> Option { + self.0.next().map(|(tag, name)| { + ( + if tag == WatchKind::Change as u8 { + WatchKind::Change + } else { + WatchKind::Rename + }, + name, + ) + }) + } +} + +#[cfg(not(target_arch = "wasm32"))] +mod service; +#[cfg(not(target_arch = "wasm32"))] +pub(crate) use service::{Reply, Service}; +#[cfg(not(target_arch = "wasm32"))] +pub(crate) mod watch; + +/// Encode UTF-16 as WTF-8 (lossless for unpaired surrogates), appending to `out`. +#[cfg(windows)] +pub(crate) fn wtf8(units: &[u16], out: &mut [u8], used: &mut usize) -> bool { + let start = *used; + for c in char::decode_utf16(units.iter().copied()) { + let code = match c { + Ok(c) => c as u32, + Err(e) => u32::from(e.unpaired_surrogate()), + }; + let mut bytes = [0u8; 4]; + let len = if code < 0x80 { + bytes[0] = code as u8; + 1 + } else if code < 0x800 { + bytes[0] = 0xc0 | (code >> 6) as u8; + bytes[1] = 0x80 | (code & 0x3f) as u8; + 2 + } else if code < 0x10000 { + bytes[0] = 0xe0 | (code >> 12) as u8; + bytes[1] = 0x80 | ((code >> 6) & 0x3f) as u8; + bytes[2] = 0x80 | (code & 0x3f) as u8; + 3 + } else { + bytes[0] = 0xf0 | (code >> 18) as u8; + bytes[1] = 0x80 | ((code >> 12) & 0x3f) as u8; + bytes[2] = 0x80 | ((code >> 6) & 0x3f) as u8; + bytes[3] = 0x80 | (code & 0x3f) as u8; + 4 + }; + if *used + len > out.len() { + *used = start; + return false; + } + out[*used..*used + len].copy_from_slice(&bytes[..len]); + *used += len; + } + true +} + +#[cfg(all(test, not(loom)))] +mod tests { + use super::*; + #[test] + fn records_round_trip_and_truncated_input_stops() { + let mut page = [0u8; 32]; + let mut used = 0; + assert!(put_record(&mut page, &mut used, 2, b"dir")); + assert!(put_record(&mut page, &mut used, 1, b"")); + assert!(!put_record(&mut page, &mut used, 1, &[b'x'; 30])); + let entries: Vec<_> = DirEntries::new(&page[..used]).collect(); + assert_eq!( + entries, + [ + (FileType::Directory, &b"dir"[..]), + (FileType::File, &b""[..]) + ] + ); + let events: Vec<_> = WatchEvents::new(&page[..used]).collect(); + assert_eq!(events[0].0, WatchKind::Change); + assert_eq!(events[1].0, WatchKind::Rename); + assert_eq!(DirEntries::new(&page[..3]).count(), 0); + } + #[cfg(windows)] + #[test] + fn wtf8_preserves_unpaired_surrogates() { + let mut out = [0u8; 16]; + let mut used = 0; + assert!(wtf8(&[0x61, 0xd800, 0x20ac], &mut out, &mut used)); + assert_eq!(&out[..used], b"a\xed\xa0\x80\xe2\x82\xac"); + } +} diff --git a/crates/turnloop/src/fs/service.rs b/crates/turnloop/src/fs/service.rs new file mode 100644 index 0000000..e8a8b45 --- /dev/null +++ b/crates/turnloop/src/fs/service.rs @@ -0,0 +1,408 @@ +//! Native typed filesystem requests on the shared bounded blocking pool. +//! +//! Storage is reserved per operation slot and per handle when the loop starts, so +//! steady-state requests reuse it. Each accepted request reserves one pool queue +//! slot, which makes starting a queued FIFO successor infallible and bounded. +//! Only the head of a handle's FIFO is ever on the pool; a request needing a +//! pooled lease waits (without spinning) until one is available. +use super::{FileMetadata, FsOutput, FsRequest}; +use crate::{ + BufLease, BufferPool, Config, Error, ErrorKind, Handle, IoBufMut, OpId, PoolConfig, ReadBuf, + Result, + blocking::{ReusableWork, WorkOutput, WorkPort, WorkResult}, +}; +use std::{ + collections::VecDeque, + sync::{ + Arc, Condvar, Mutex, MutexGuard, + atomic::{AtomicBool, Ordering}, + }, +}; +#[cfg(unix)] +#[path = "unix.rs"] +mod sys; +#[cfg(windows)] +#[path = "windows.rs"] +mod sys; + +pub(super) fn lock(m: &Mutex) -> MutexGuard<'_, T> { + m.lock().unwrap_or_else(std::sync::PoisonError::into_inner) +} + +/// Native object behind one file or directory handle. +#[derive(Default)] +pub(super) enum Object { + /// Not opened yet, or released. + #[default] + Empty, + /// Closed by an explicit Close request. + Closed, + File(sys::File), + Dir(sys::Dir), +} + +struct Job { + op: OpId, + request: FsRequest, + object: Option>>, +} +// SAFETY: a job carries prepared paths, owned bytes, handle state guarded by a +// mutex, and host buffers whose D3 contract keeps them valid and exclusively +// owned by this request until its result is published or the loop is dropped. +// `Service::drop` withdraws queued jobs and waits for running ones. +unsafe impl Send for Job {} + +struct Shared { + work: Arc, + running: Mutex, + idle: Condvar, +} +/// Compact worker result for the loop's completion queue. Metadata stays in the +/// operation slot, keeping every queued completion small. +pub(crate) enum Reply { + Opened, + Read(usize), + Wrote(usize), + Metadata, + Directory { n: usize, eof: bool }, + Bytes(usize), + Done, +} +impl Reply { + pub fn output(self, metadata: Option) -> FsOutput { + match self { + Self::Opened => FsOutput::Opened, + Self::Read(n) => FsOutput::Read(n), + Self::Wrote(n) => FsOutput::Wrote(n), + Self::Metadata => FsOutput::Metadata(metadata.expect("published metadata")), + Self::Directory { n, eof } => FsOutput::Directory { n, eof }, + Self::Bytes(n) => FsOutput::Bytes(n), + Self::Done => FsOutput::Done, + } + } +} +struct Slot { + job: Mutex>, + metadata: Mutex>, + cancel: AtomicBool, + shared: Arc, +} +impl ReusableWork for Slot { + fn run(&self) { + // A dropped loop withdraws queued jobs; the queue entry then does nothing. + let Some(job) = lock(&self.job).take() else { + return; + }; + let op = job.op; + let result = if self.cancel.load(Ordering::Acquire) { + drop(job); + Err(Error::new(ErrorKind::Cancelled)) + } else { + execute(job) + }; + let result = result.map(|output| match output { + FsOutput::Opened => Reply::Opened, + FsOutput::Read(n) => Reply::Read(n), + FsOutput::Wrote(n) => Reply::Wrote(n), + FsOutput::Metadata(metadata) => { + *lock(&self.metadata) = Some(metadata); + Reply::Metadata + } + FsOutput::Directory { n, eof } => Reply::Directory { n, eof }, + FsOutput::Bytes(n) => Reply::Bytes(n), + FsOutput::Done => Reply::Done, + }); + // The job and its buffers were consumed above: no access after publication. + self.shared.work.complete(WorkResult { + op, + result: result.map(WorkOutput::Fs), + }); + let mut running = lock(&self.shared.running); + *running -= 1; + self.shared.idle.notify_all(); + } +} + +#[derive(Clone, Copy, Eq, PartialEq)] +enum Stage { + /// Accepted behind the handle's FIFO head. + Queued, + /// FIFO head waiting for a pooled lease. + Waiting, + /// Handed to the pool. + Running, + /// Cancelled before starting; the terminal result is already published. + Withdrawn, +} +struct Active { + op: OpId, + handle: Option, + previous: Option, + next: Option, + stage: Stage, + request: Option, + lease: Option, +} + +pub(crate) struct Service { + slots: Vec>, + objects: Vec>>, + active: Vec>, + heads: Vec>, + tails: Vec>, + waiting: VecDeque, + shared: Arc, + pool: BufferPool, + config: PoolConfig, +} +impl Service { + pub fn new(config: &Config, work: Arc, pool: BufferPool) -> Self { + let shared = Arc::new(Shared { + work, + running: Mutex::new(0), + idle: Condvar::new(), + }); + // Darwin's std mutex allocates its storage on first lock: do it at setup. + drop(lock(&shared.running)); + let slots = (0..config.max_operations) + .map(|_| { + let slot = Arc::new(Slot { + job: Mutex::new(None), + metadata: Mutex::new(None), + cancel: AtomicBool::new(false), + shared: shared.clone(), + }); + drop(lock(&slot.job)); + drop(lock(&slot.metadata)); + slot + }) + .collect(); + let objects = (0..config.max_handles) + .map(|_| { + let object = Arc::new(Mutex::new(Object::Empty)); + drop(lock(&object)); + object + }) + .collect(); + Self { + slots, + objects, + active: (0..config.max_operations).map(|_| None).collect(), + heads: vec![None; config.max_handles], + tails: vec![None; config.max_handles], + waiting: VecDeque::with_capacity(config.max_operations), + shared, + pool, + config: config.blocking_pool, + } + } + /// Accept a request, or reject it without retaining or touching its buffers. + pub fn submit(&mut self, op: OpId, handle: Option, request: FsRequest) -> Result<()> { + crate::blocking::reserve(self.config)?; + let i = op.index(); + debug_assert!(self.active[i].is_none()); + let previous = handle.and_then(|h| self.tails[h.index()]); + self.active[i] = Some(Active { + op, + handle, + previous, + next: None, + stage: Stage::Queued, + request: Some(request), + lease: None, + }); + if let Some(h) = handle { + match previous { + Some(tail) => self.active[tail].as_mut().expect("FIFO tail").next = Some(i), + None => self.heads[h.index()] = Some(i), + } + self.tails[h.index()] = Some(i); + } + if previous.is_none() { + self.start(i); + } + Ok(()) + } + fn start(&mut self, i: usize) { + let active = self.active[i].as_mut().expect("startable request"); + debug_assert!(matches!(active.stage, Stage::Queued | Stage::Waiting)); + let request = active.request.as_mut().expect("unstarted request"); + if let Some(buffer) = request.read_buffer() + && matches!(buffer, ReadBuf::Pooled) + { + let Some(mut lease) = self.pool.acquire() else { + if active.stage != Stage::Waiting { + active.stage = Stage::Waiting; + self.waiting.push_back(i); + } + return; + }; + let bytes = lease.writable(); + // SAFETY: the lease stays in `Active` until the loop thread consumes the + // published result, so this region outlives the worker's access. + *buffer = ReadBuf::Provided(unsafe { + IoBufMut::from_raw_parts(bytes.as_mut_ptr(), bytes.len()) + }); + active.lease = Some(lease); + } + active.stage = Stage::Running; + let slot = &self.slots[i]; + slot.cancel.store(false, Ordering::Release); + *lock(&slot.job) = Some(Job { + op: active.op, + request: active.request.take().expect("unstarted request"), + object: active.handle.map(|h| self.objects[h.index()].clone()), + }); + *lock(&self.shared.running) += 1; + crate::blocking::push_reserved(slot.clone()); + } + /// Start requests that were waiting for a pooled lease. + pub fn retry(&mut self) { + while self.pool.available() { + let Some(i) = self.waiting.pop_front() else { + break; + }; + self.start(i); + } + } + /// Whether `retry` can make progress now. + pub fn has_work(&self) -> bool { + !self.waiting.is_empty() && self.pool.available() + } + fn unlink(&mut self, i: usize) { + let active = self.active[i].as_mut().expect("linked request"); + let (previous, next, handle) = (active.previous.take(), active.next.take(), active.handle); + let Some(h) = handle else { + return; + }; + match previous { + Some(p) => self.active[p].as_mut().expect("FIFO predecessor").next = next, + None => self.heads[h.index()] = next, + } + match next { + Some(n) => self.active[n].as_mut().expect("FIFO successor").previous = previous, + None => self.tails[h.index()] = previous, + } + } + /// Begin cancellation. Unstarted requests publish Cancelled at once; running + /// ones finish (or observe the flag first) and are reported as Cancelled. + pub fn cancel(&mut self, op: OpId) { + let i = op.index(); + let Some(active) = self.active[i].as_mut().filter(|a| a.op == op) else { + return; + }; + match active.stage { + Stage::Running => self.slots[i].cancel.store(true, Ordering::Release), + Stage::Withdrawn => {} + Stage::Queued | Stage::Waiting => { + let was_head = active.previous.is_none(); + if active.stage == Stage::Waiting { + self.waiting.retain(|&w| w != i); + } + active.stage = Stage::Withdrawn; + active.request = None; + let next = active.next; + self.unlink(i); + crate::blocking::unreserve(); + self.shared.work.complete(WorkResult { + op, + result: Err(Error::new(ErrorKind::Cancelled)), + }); + if was_head && let Some(next) = next { + self.start(next); + } + } + } + } + /// Consume the published terminal result: advance the FIFO and return the + /// lease and any metadata the worker stored in the slot. + pub fn complete(&mut self, op: OpId) -> (Option, Option) { + let i = op.index(); + let Some(active) = self.active[i].as_ref().filter(|a| a.op == op) else { + return (None, None); + }; + if active.stage == Stage::Running { + let next = active.next; + self.unlink(i); + if let Some(next) = next { + self.start(next); + } + } + let metadata = lock(&self.slots[i].metadata).take(); + (self.active[i].take().and_then(|a| a.lease), metadata) + } + /// Reset a released handle's state; a still-open descriptor closes here. + pub fn release(&mut self, h: Handle) { + debug_assert!(self.heads[h.index()].is_none()); + let object = std::mem::take(&mut *lock(&self.objects[h.index()])); + drop(object); + } +} +impl Drop for Service { + fn drop(&mut self) { + let mut withdrawn = 0; + for (i, active) in self.active.iter().enumerate() { + let Some(active) = active else { continue }; + match active.stage { + Stage::Running => { + self.slots[i].cancel.store(true, Ordering::Release); + if lock(&self.slots[i].job).take().is_some() { + withdrawn += 1; + } + } + Stage::Queued | Stage::Waiting => crate::blocking::unreserve(), + Stage::Withdrawn => {} + } + } + let mut running = lock(&self.shared.running); + *running -= withdrawn; + while *running != 0 { + running = self + .shared + .idle + .wait(running) + .unwrap_or_else(std::sync::PoisonError::into_inner); + } + } +} + +pub(super) fn output(buffer: &mut ReadBuf) -> &mut [u8] { + let ReadBuf::Provided(buffer) = buffer else { + unreachable!("pooled buffers are provided before a request starts") + }; + // SAFETY: an accepted request exclusively owns its output region until publication. + unsafe { std::slice::from_raw_parts_mut(buffer.as_mut_ptr(), buffer.len()) } +} +fn execute(job: Job) -> Result { + let Job { + request, object, .. + } = job; + let mut object = object.as_ref().map(|o| lock(o)); + match request { + FsRequest::Open { path, options } => { + let object = object.as_mut().expect("open handle state"); + **object = Object::File(sys::open(&path, &options)?); + Ok(FsOutput::Opened) + } + FsRequest::OpenDir { path } => { + let object = object.as_mut().expect("open handle state"); + **object = Object::Dir(sys::open_dir(&path)?); + Ok(FsOutput::Opened) + } + FsRequest::Close { .. } => { + let object = object.as_mut().expect("close handle state"); + match std::mem::replace(&mut **object, Object::Closed) { + Object::File(file) => sys::close(file).map(|()| FsOutput::Done), + Object::Dir(dir) => { + drop(dir); + Ok(FsOutput::Done) + } + previous => { + **object = previous; + Err(sys::bad_descriptor()) + } + } + } + request => sys::execute(request, object.as_deref_mut()), + } +} diff --git a/crates/turnloop/src/fs/unix.rs b/crates/turnloop/src/fs/unix.rs new file mode 100644 index 0000000..50d63ce --- /dev/null +++ b/crates/turnloop/src/fs/unix.rs @@ -0,0 +1,588 @@ +//! Unix filesystem system calls, executed only on blocking-pool threads. +use super::{Object, output}; +use crate::{ + Error, ErrorKind, Result, + fs::{ + AccessMode, FileMetadata, FileOptions, FileTime, FileType, FsOutput, FsPath, FsRequest, + FsTarget, TimeChange, put_record, + }, +}; +use std::{ + ffi::{CStr, c_int}, + os::fd::{AsRawFd, FromRawFd, IntoRawFd, OwnedFd, RawFd}, + ptr::NonNull, +}; + +pub(in crate::fs) struct File(OwnedFd); +pub(in crate::fs) struct Dir(NonNull); +// SAFETY: the directory stream is used by one pool thread at a time, under the +// handle's mutex, and closed exactly once by Drop. +unsafe impl Send for Dir {} +impl Drop for Dir { + fn drop(&mut self) { + // SAFETY: this owner holds the only reference to the open stream. + unsafe { libc::closedir(self.0.as_ptr()) }; + } +} + +fn last() -> Error { + std::io::Error::last_os_error().into() +} +fn os(code: c_int) -> Error { + std::io::Error::from_raw_os_error(code).into() +} +fn check(result: c_int) -> Result<()> { + if result < 0 { Err(last()) } else { Ok(()) } +} +/// Retry a call interrupted before it had any effect. +fn retry(mut call: impl FnMut() -> isize) -> Result { + loop { + let n = call(); + if n >= 0 { + return Ok(n as usize); + } + let e = last(); + if e.os != Some(libc::EINTR) { + return Err(e); + } + } +} +pub(in crate::fs) fn bad_descriptor() -> Error { + os(libc::EBADF) +} +fn offset(value: u64) -> Result { + value.try_into().map_err(|_| os(libc::EINVAL)) +} +fn file(object: Option<&mut Object>) -> Result { + match object { + Some(Object::File(f)) => Ok(f.0.as_raw_fd()), + Some(Object::Dir(d)) => { + // SAFETY: live stream owned by this handle. + Ok(unsafe { libc::dirfd(d.0.as_ptr()) }) + } + _ => Err(bad_descriptor()), + } +} + +pub(in crate::fs) fn open(path: &FsPath, o: &FileOptions) -> Result { + let writes = o.write || o.append; + let mut flags = match (o.read, writes) { + (true, false) => libc::O_RDONLY, + (false, true) => libc::O_WRONLY, + (true, true) => libc::O_RDWR, + (false, false) => return Err(Error::new(ErrorKind::InvalidInput)), + } | libc::O_CLOEXEC; + if o.create || o.exclusive { + flags |= libc::O_CREAT; + } + if o.exclusive { + flags |= libc::O_EXCL; + } + if o.truncate { + flags |= libc::O_TRUNC; + } + if o.append { + flags |= libc::O_APPEND; + } + if o.sync { + flags |= libc::O_SYNC; + } + if o.data_sync { + flags |= libc::O_DSYNC; + } + if !o.follow_symlinks { + flags |= libc::O_NOFOLLOW; + } + let fd = retry(|| { + // SAFETY: NUL-terminated prepared path; integer flags and creation mode. + unsafe { libc::open(path.native().as_ptr(), flags, o.mode as libc::c_uint) as isize } + })?; + // SAFETY: open returned a new descriptor owned by nobody else. + Ok(File(unsafe { OwnedFd::from_raw_fd(fd as RawFd) })) +} +pub(in crate::fs) fn open_dir(path: &FsPath) -> Result { + // SAFETY: NUL-terminated prepared path. + let stream = unsafe { libc::opendir(path.native().as_ptr()) }; + NonNull::new(stream).map(Dir).ok_or_else(last) +} +pub(in crate::fs) fn close(file: File) -> Result<()> { + // SAFETY: the owned descriptor is closed exactly once. EINTR is not retried: + // the descriptor is released either way on Linux and macOS. + check(unsafe { libc::close(file.0.into_raw_fd()) }) +} + +pub(in crate::fs) fn execute(request: FsRequest, object: Option<&mut Object>) -> Result { + match request { + FsRequest::Read { + mut buffer, + offset: at, + .. + } => { + let fd = file(object)?; + let at = at.map(offset).transpose()?; + let out = output(&mut buffer); + retry(|| { + // SAFETY: exclusive, live output region; descriptor owned by this handle. + unsafe { + match at { + Some(at) => libc::pread(fd, out.as_mut_ptr().cast(), out.len(), at), + None => libc::read(fd, out.as_mut_ptr().cast(), out.len()), + } + } + }) + .map(FsOutput::Read) + } + FsRequest::Write { + buffer, offset: at, .. + } => { + let fd = file(object)?; + let at = at.map(offset).transpose()?; + let bytes = buffer.as_slice(); + retry(|| { + // SAFETY: initialized input kept alive by the accepted request. + unsafe { + match at { + Some(at) => libc::pwrite(fd, bytes.as_ptr().cast(), bytes.len(), at), + None => libc::write(fd, bytes.as_ptr().cast(), bytes.len()), + } + } + }) + .map(FsOutput::Wrote) + } + FsRequest::Sync { data_only, .. } => { + let fd = file(object)?; + sync(fd, data_only)?; + Ok(FsOutput::Done) + } + FsRequest::Truncate { size, .. } => { + let fd = file(object)?; + let size = offset(size)?; + // SAFETY: owned descriptor and representable length. + retry(|| unsafe { libc::ftruncate(fd, size) as isize })?; + Ok(FsOutput::Done) + } + FsRequest::Stat { + path, + follow_symlinks, + } => stat_path(path.native(), follow_symlinks).map(FsOutput::Metadata), + FsRequest::Fstat { .. } => stat_fd(file(object)?).map(FsOutput::Metadata), + FsRequest::ReadDir { mut buffer, .. } => { + let Some(Object::Dir(dir)) = object else { + return Err(bad_descriptor()); + }; + read_dir(dir, output(&mut buffer)) + } + FsRequest::Mkdir { path, mode } => { + // SAFETY: prepared path and mode bits. + check(unsafe { libc::mkdir(path.native().as_ptr(), mode as libc::mode_t) }) + .map(|()| FsOutput::Done) + } + FsRequest::Rmdir { path } => { + // SAFETY: prepared path. + check(unsafe { libc::rmdir(path.native().as_ptr()) }).map(|()| FsOutput::Done) + } + FsRequest::Unlink { path } => { + // SAFETY: prepared path. + check(unsafe { libc::unlink(path.native().as_ptr()) }).map(|()| FsOutput::Done) + } + FsRequest::Rename { from, to } => { + // SAFETY: both prepared paths live through the call. + check(unsafe { libc::rename(from.native().as_ptr(), to.native().as_ptr()) }) + .map(|()| FsOutput::Done) + } + FsRequest::Link { existing, link } => { + // SAFETY: both prepared paths live through the call. + check(unsafe { libc::link(existing.native().as_ptr(), link.native().as_ptr()) }) + .map(|()| FsOutput::Done) + } + FsRequest::Symlink { target, link, .. } => { + // SAFETY: both prepared paths live through the call. + check(unsafe { libc::symlink(target.native().as_ptr(), link.native().as_ptr()) }) + .map(|()| FsOutput::Done) + } + FsRequest::ReadLink { path, mut buffer } => { + let out = output(&mut buffer); + // SAFETY: prepared path and exclusive output region. + let n = unsafe { + libc::readlink(path.native().as_ptr(), out.as_mut_ptr().cast(), out.len()) + }; + if n < 0 { + return Err(last()); + } + if n as usize == out.len() { + // The content may have been truncated to the buffer. + return Err(Error::new(ErrorKind::ResourceLimit)); + } + Ok(FsOutput::Bytes(n as usize)) + } + FsRequest::RealPath { path, mut buffer } => { + let mut resolved = [0 as libc::c_char; libc::PATH_MAX as usize + 1]; + // SAFETY: the resolved buffer holds PATH_MAX bytes plus a NUL, as required. + if unsafe { libc::realpath(path.native().as_ptr(), resolved.as_mut_ptr()) }.is_null() { + return Err(last()); + } + // SAFETY: successful realpath NUL-terminated the buffer. + let bytes = unsafe { CStr::from_ptr(resolved.as_ptr()) }.to_bytes(); + let out = output(&mut buffer); + if bytes.len() > out.len() { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + out[..bytes.len()].copy_from_slice(bytes); + Ok(FsOutput::Bytes(bytes.len())) + } + FsRequest::Access { path, mode } => { + let AccessMode { + read, + write, + execute, + } = mode; + let bits = if read { libc::R_OK } else { 0 } + | if write { libc::W_OK } else { 0 } + | if execute { libc::X_OK } else { 0 }; + let bits = if bits == 0 { libc::F_OK } else { bits }; + // SAFETY: prepared path and access bits. + check(unsafe { libc::access(path.native().as_ptr(), bits) }).map(|()| FsOutput::Done) + } + FsRequest::Chmod { target, mode } => { + let mode = mode as libc::mode_t; + // SAFETY: prepared path or owned descriptor, and mode bits. + check(unsafe { + match &target { + FsTarget::Path { + path, + follow_symlinks: true, + } => libc::chmod(path.native().as_ptr(), mode), + FsTarget::Path { path, .. } => libc::fchmodat( + libc::AT_FDCWD, + path.native().as_ptr(), + mode, + libc::AT_SYMLINK_NOFOLLOW, + ), + FsTarget::File(_) => libc::fchmod(file(object)?, mode), + } + }) + .map(|()| FsOutput::Done) + } + FsRequest::Chown { target, uid, gid } => { + let uid = uid.map_or(libc::uid_t::MAX, |v| v as libc::uid_t); + let gid = gid.map_or(libc::gid_t::MAX, |v| v as libc::gid_t); + // SAFETY: prepared path or owned descriptor, and scalar ids (-1 keeps). + check(unsafe { + match &target { + FsTarget::Path { + path, + follow_symlinks: true, + } => libc::chown(path.native().as_ptr(), uid, gid), + FsTarget::Path { path, .. } => libc::lchown(path.native().as_ptr(), uid, gid), + FsTarget::File(_) => libc::fchown(file(object)?, uid, gid), + } + }) + .map(|()| FsOutput::Done) + } + FsRequest::SetTimes { + target, + accessed, + modified, + } => { + let times = [timespec(accessed), timespec(modified)]; + // SAFETY: two initialized timespecs; prepared path or owned descriptor. + check(unsafe { + match &target { + FsTarget::Path { + path, + follow_symlinks, + } => libc::utimensat( + libc::AT_FDCWD, + path.native().as_ptr(), + times.as_ptr(), + if *follow_symlinks { + 0 + } else { + libc::AT_SYMLINK_NOFOLLOW + }, + ), + FsTarget::File(_) => libc::futimens(file(object)?, times.as_ptr()), + } + }) + .map(|()| FsOutput::Done) + } + FsRequest::CopyFile { + from, + to, + exclusive, + } => copy_file(&from, &to, exclusive).map(|()| FsOutput::Done), + FsRequest::Open { .. } | FsRequest::OpenDir { .. } | FsRequest::Close { .. } => { + unreachable!("handled by the service") + } + } +} + +fn sync(fd: RawFd, data_only: bool) -> Result<()> { + #[cfg(target_vendor = "apple")] + { + let _ = data_only; + // As libuv: fsync does not flush the drive cache on Apple platforms. + // SAFETY: owned descriptor, integer command. + if unsafe { libc::fcntl(fd, libc::F_FULLFSYNC) } == 0 { + return Ok(()); + } + // SAFETY: owned descriptor. + retry(|| unsafe { libc::fsync(fd) as isize }).map(drop) + } + #[cfg(not(target_vendor = "apple"))] + { + // SAFETY: owned descriptor. + retry(|| unsafe { + if data_only { + libc::fdatasync(fd) as isize + } else { + libc::fsync(fd) as isize + } + }) + .map(drop) + } +} + +fn timespec(change: TimeChange) -> libc::timespec { + match change { + TimeChange::Keep => libc::timespec { + tv_sec: 0, + tv_nsec: libc::UTIME_OMIT, + }, + TimeChange::Now => libc::timespec { + tv_sec: 0, + tv_nsec: libc::UTIME_NOW, + }, + TimeChange::At(t) => libc::timespec { + tv_sec: t.seconds as libc::time_t, + tv_nsec: t.nanoseconds as _, + }, + } +} +fn time(seconds: i64, nanoseconds: i64) -> Option { + Some(FileTime { + seconds, + nanoseconds: nanoseconds as u32, + }) +} +// mode_t is u16 on Apple platforms and u32 elsewhere. +#[allow(clippy::unnecessary_cast)] +fn kind(mode: u32) -> FileType { + match mode & libc::S_IFMT as u32 { + m if m == libc::S_IFREG as u32 => FileType::File, + m if m == libc::S_IFDIR as u32 => FileType::Directory, + m if m == libc::S_IFLNK as u32 => FileType::Symlink, + m if m == libc::S_IFBLK as u32 => FileType::BlockDevice, + m if m == libc::S_IFCHR as u32 => FileType::CharDevice, + m if m == libc::S_IFIFO as u32 => FileType::Fifo, + m if m == libc::S_IFSOCK as u32 => FileType::Socket, + _ => FileType::Unknown, + } +} +#[allow(clippy::unnecessary_cast)] +fn from_stat(s: &libc::stat) -> FileMetadata { + FileMetadata { + kind: kind(s.st_mode as u32), + size: s.st_size as u64, + mode: Some(s.st_mode as u32), + device: Some(s.st_dev as u64), + inode: Some(s.st_ino as u64), + links: Some(s.st_nlink as u64), + uid: Some(s.st_uid as u32), + gid: Some(s.st_gid as u32), + rdev: Some(s.st_rdev as u64), + block_size: Some(s.st_blksize as u64), + blocks: Some(s.st_blocks as u64), + accessed: time(s.st_atime as i64, s.st_atime_nsec as i64), + modified: time(s.st_mtime as i64, s.st_mtime_nsec as i64), + changed: time(s.st_ctime as i64, s.st_ctime_nsec as i64), + #[cfg(any(target_vendor = "apple", target_os = "freebsd"))] + created: time(s.st_birthtime as i64, s.st_birthtime_nsec as i64), + #[cfg(not(any(target_vendor = "apple", target_os = "freebsd")))] + created: None, + } +} +#[cfg(all(target_os = "linux", target_env = "gnu"))] +fn statx(fd: RawFd, path: &CStr, flags: c_int) -> Option> { + let mut s = std::mem::MaybeUninit::::uninit(); + // SAFETY: valid directory descriptor or AT_FDCWD, NUL-terminated path, writable output. + let r = unsafe { + libc::statx( + fd, + path.as_ptr(), + flags, + libc::STATX_BASIC_STATS | libc::STATX_BTIME, + s.as_mut_ptr(), + ) + }; + if r < 0 { + let e = last(); + // Old kernels and seccomp filters: use the stat family instead. + if matches!(e.os, Some(libc::ENOSYS | libc::EPERM)) { + return None; + } + return Some(Err(e)); + } + // SAFETY: successful statx initialized the structure. + let s = unsafe { s.assume_init() }; + let t = |t: libc::statx_timestamp| time(t.tv_sec, i64::from(t.tv_nsec)); + let mode = u32::from(s.stx_mode); + Some(Ok(FileMetadata { + kind: kind(mode), + size: s.stx_size, + mode: Some(mode), + device: Some(libc::makedev(s.stx_dev_major, s.stx_dev_minor)), + inode: Some(s.stx_ino), + links: Some(u64::from(s.stx_nlink)), + uid: Some(s.stx_uid), + gid: Some(s.stx_gid), + rdev: Some(libc::makedev(s.stx_rdev_major, s.stx_rdev_minor)), + block_size: Some(u64::from(s.stx_blksize)), + blocks: Some(s.stx_blocks), + accessed: t(s.stx_atime), + modified: t(s.stx_mtime), + changed: t(s.stx_ctime), + created: (s.stx_mask & libc::STATX_BTIME != 0) + .then(|| t(s.stx_btime)) + .flatten(), + })) +} +fn stat_path(path: &CStr, follow: bool) -> Result { + #[cfg(all(target_os = "linux", target_env = "gnu"))] + if let Some(result) = statx( + libc::AT_FDCWD, + path, + if follow { 0 } else { libc::AT_SYMLINK_NOFOLLOW }, + ) { + return result; + } + let mut s = std::mem::MaybeUninit::::uninit(); + // SAFETY: NUL-terminated path and writable output, read only after success. + check(unsafe { + if follow { + libc::stat(path.as_ptr(), s.as_mut_ptr()) + } else { + libc::lstat(path.as_ptr(), s.as_mut_ptr()) + } + })?; + // SAFETY: successful stat initialized the structure. + Ok(from_stat(unsafe { s.assume_init_ref() })) +} +fn stat_fd(fd: RawFd) -> Result { + #[cfg(all(target_os = "linux", target_env = "gnu"))] + if let Some(result) = statx(fd, c"", libc::AT_EMPTY_PATH) { + return result; + } + let mut s = std::mem::MaybeUninit::::uninit(); + // SAFETY: owned descriptor and writable output, read only after success. + check(unsafe { libc::fstat(fd, s.as_mut_ptr()) })?; + // SAFETY: successful fstat initialized the structure. + Ok(from_stat(unsafe { s.assume_init_ref() })) +} + +fn set_errno(value: c_int) { + // SAFETY: the thread-local errno location is valid for this thread. + unsafe { + #[cfg(any(target_os = "linux", target_os = "android"))] + { + *libc::__errno_location() = value; + } + #[cfg(any(target_vendor = "apple", target_os = "freebsd"))] + { + *libc::__error() = value; + } + #[cfg(not(any( + target_os = "linux", + target_os = "android", + target_vendor = "apple", + target_os = "freebsd" + )))] + { + *libc::__errno() = value; + } + } +} +fn read_dir(dir: &mut Dir, out: &mut [u8]) -> Result { + let stream = dir.0.as_ptr(); + let mut used = 0; + loop { + // SAFETY: live stream exclusively used by this request. + let position = unsafe { libc::telldir(stream) }; + set_errno(0); + // SAFETY: live stream; the entry is read before the next readdir call. + let entry = unsafe { libc::readdir(stream) }; + let Some(entry) = NonNull::new(entry) else { + let e = std::io::Error::last_os_error(); + if e.raw_os_error().is_some_and(|code| code != 0) { + return Err(e.into()); + } + return Ok(FsOutput::Directory { n: used, eof: true }); + }; + // SAFETY: readdir returned a live entry with a NUL-terminated name. + let (name, tag) = unsafe { + let entry = entry.as_ref(); + ( + CStr::from_ptr(entry.d_name.as_ptr()).to_bytes(), + entry.d_type, + ) + }; + if name == b"." || name == b".." { + continue; + } + let tag = match tag { + libc::DT_REG => FileType::File, + libc::DT_DIR => FileType::Directory, + libc::DT_LNK => FileType::Symlink, + libc::DT_BLK => FileType::BlockDevice, + libc::DT_CHR => FileType::CharDevice, + libc::DT_FIFO => FileType::Fifo, + libc::DT_SOCK => FileType::Socket, + _ => FileType::Unknown, + }; + if !put_record(out, &mut used, tag as u8, name) { + // Leave the entry for the next page. + // SAFETY: position came from telldir on this live stream. + unsafe { libc::seekdir(stream, position) }; + if used == 0 { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + return Ok(FsOutput::Directory { + n: used, + eof: false, + }); + } + } +} + +fn copy_file(from: &FsPath, to: &FsPath, exclusive: bool) -> Result<()> { + let source = open( + from, + &FileOptions { + read: true, + ..FileOptions::default() + }, + )?; + let metadata = stat_fd(source.0.as_raw_fd())?; + if metadata.kind == FileType::Directory { + return Err(os(libc::EISDIR)); + } + let permissions = metadata.mode.unwrap_or(0o644) & 0o7777; + let target = open( + to, + &FileOptions { + read: false, + write: true, + create: true, + exclusive, + truncate: !exclusive, + mode: permissions, + ..FileOptions::default() + }, + )?; + let mut reader = std::fs::File::from(source.0); + let mut writer = std::fs::File::from(target.0); + // std uses copy_file_range/sendfile on Linux and a stack buffer elsewhere. + std::io::copy(&mut reader, &mut writer).map_err(Error::from)?; + // SAFETY: owned destination descriptor and the source's permission bits. + check(unsafe { libc::fchmod(writer.as_raw_fd(), permissions as libc::mode_t) }) +} diff --git a/crates/turnloop/src/fs/watch.rs b/crates/turnloop/src/fs/watch.rs new file mode 100644 index 0000000..429254d --- /dev/null +++ b/crates/turnloop/src/fs/watch.rs @@ -0,0 +1,136 @@ +//! Bounded per-watch record storage shared by the native watch backends. +//! +//! Native events are parsed as soon as the OS reports them, so one slow host +//! never stalls another watch sharing a kernel queue. Records wait here until a +//! pooled lease is available. A record that does not fit is dropped and the next +//! batch reports `overflow`, which tells the host to rescan the watched scope. +use crate::{ + BufLease, + fs::{RECORD_HEADER, WatchKind, put_record}, +}; + +#[cfg_attr(not(any(unix, windows)), allow(dead_code))] +pub(crate) struct Ring { + bytes: Box<[u8]>, + len: usize, + overflow: bool, +} +#[cfg_attr(not(any(unix, windows)), allow(dead_code))] +impl Ring { + /// Records retained per watch, allocated once when the watch starts. + pub const CAPACITY: usize = 16 * 1024; + pub fn new() -> Self { + Self { + bytes: vec![0; Self::CAPACITY].into_boxed_slice(), + len: 0, + overflow: false, + } + } + /// Storage-free placeholder for watches whose records live elsewhere. + #[cfg_attr(not(target_os = "macos"), allow(dead_code))] + pub fn empty() -> Self { + Self { + bytes: Box::default(), + len: 0, + overflow: false, + } + } + pub fn push(&mut self, kind: WatchKind, name: &[u8]) { + if !put_record(&mut self.bytes, &mut self.len, kind as u8, name) { + self.overflow = true; + } + } + /// Append a record whose UTF-16 name is encoded as WTF-8 in place. + #[cfg(windows)] + pub fn push_wide(&mut self, kind: WatchKind, name: &[u16]) { + let start = self.len; + let mut end = start + RECORD_HEADER; + if end > self.bytes.len() + || !crate::fs::wtf8(name, &mut self.bytes, &mut end) + || end - start - RECORD_HEADER > usize::from(u16::MAX) + { + self.overflow = true; + return; + } + let length = (end - start - RECORD_HEADER) as u16; + self.bytes[start] = kind as u8; + self.bytes[start + 1] = 0; + self.bytes[start + 2..start + 4].copy_from_slice(&length.to_le_bytes()); + self.len = end; + } + pub fn lost(&mut self) { + self.overflow = true; + } + pub fn is_empty(&self) -> bool { + self.len == 0 && !self.overflow + } + pub fn clear(&mut self) { + self.len = 0; + self.overflow = false; + } + /// Move the leading whole records that fit into `lease`, returning (and + /// clearing) the overflow flag. A record larger than an empty lease is dropped + /// as lost, so a small pool buffer can never stall delivery. + pub fn take(&mut self, lease: &mut BufLease) -> bool { + let out = lease.writable(); + let (mut used, mut at) = (0, 0); + while at + RECORD_HEADER <= self.len { + let size = RECORD_HEADER + + usize::from(u16::from_le_bytes([self.bytes[at + 2], self.bytes[at + 3]])); + if used + size > out.len() { + if used != 0 { + break; + } + self.overflow = true; + } else { + out[used..used + size].copy_from_slice(&self.bytes[at..at + size]); + used += size; + } + at += size; + } + self.bytes.copy_within(at..self.len, 0); + self.len -= at; + lease.set_len(used); + std::mem::take(&mut self.overflow) + } +} + +#[cfg(all(test, not(loom)))] +mod tests { + use super::*; + use crate::{BufferPool, WatchEvents}; + #[test] + fn batches_preserve_order_and_report_loss_once() { + let pool = BufferPool::new(1, 12); + let mut ring = Ring::new(); + ring.push(WatchKind::Rename, b"a"); + ring.push(WatchKind::Change, b"bcdef"); + ring.push(WatchKind::Change, b"this name cannot fit a lease"); + ring.push(WatchKind::Rename, b"z"); + let mut lease = pool.acquire().expect("lease"); + assert!(!ring.take(&mut lease)); + assert_eq!( + WatchEvents::new(lease.as_slice()).collect::>(), + [(WatchKind::Rename, &b"a"[..])] + ); + drop(lease); + let mut lease = pool.acquire().expect("lease"); + assert!(!ring.take(&mut lease)); + assert_eq!(lease.as_slice(), b"\x02\x00\x05\x00bcdef"); + drop(lease); + let mut lease = pool.acquire().expect("lease"); + assert!(ring.take(&mut lease), "oversized record is reported lost"); + assert_eq!( + WatchEvents::new(lease.as_slice()).collect::>(), + [(WatchKind::Rename, &b"z"[..])] + ); + assert!(ring.is_empty()); + for _ in 0..Ring::CAPACITY { + ring.push(WatchKind::Change, b"x"); + } + assert!(!ring.is_empty()); + drop(lease); + let mut lease = pool.acquire().expect("lease"); + assert!(ring.take(&mut lease), "full storage reports loss"); + } +} diff --git a/crates/turnloop/src/fs/windows.rs b/crates/turnloop/src/fs/windows.rs new file mode 100644 index 0000000..7874b0c --- /dev/null +++ b/crates/turnloop/src/fs/windows.rs @@ -0,0 +1,707 @@ +//! Windows filesystem calls on synchronous handles, executed only on pool threads. +//! Positional I/O passes an OVERLAPPED offset to ReadFile/WriteFile; cursor I/O +//! keeps a per-handle cursor, serialized by the handle's FIFO. +use super::{Object, output}; +use crate::{ + Error, ErrorKind, Result, + fs::{ + AccessMode, FileMetadata, FileOptions, FileTime, FileType, FsOutput, FsPath, FsRequest, + FsTarget, RECORD_HEADER, SymlinkKind, TimeChange, wtf8, + }, +}; +use std::{ + mem::{size_of, zeroed}, + os::windows::io::{AsRawHandle, FromRawHandle, OwnedHandle}, + ptr, +}; +use windows_sys::Win32::{ + Foundation::*, + Storage::FileSystem::*, + System::IO::{OVERLAPPED, OVERLAPPED_0, OVERLAPPED_0_0}, +}; +use windows_sys::core::BOOL; + +pub(in crate::fs) struct File { + handle: OwnedHandle, + cursor: u64, + append: bool, +} +pub(in crate::fs) struct Dir { + handle: OwnedHandle, + pattern: Box<[u16]>, + search: HANDLE, + data: Box, + pending: bool, + done: bool, +} +// SAFETY: the search handle is used by one pool thread at a time under the +// handle's mutex and closed exactly once by Drop. +unsafe impl Send for Dir {} +impl Drop for Dir { + fn drop(&mut self) { + if self.search != INVALID_HANDLE_VALUE { + // SAFETY: live search handle owned by this directory stream. + unsafe { FindClose(self.search) }; + } + } +} + +fn last() -> Error { + std::io::Error::last_os_error().into() +} +fn os(code: u32) -> Error { + std::io::Error::from_raw_os_error(code as i32).into() +} +fn check(ok: BOOL) -> Result<()> { + if ok == 0 { Err(last()) } else { Ok(()) } +} +pub(in crate::fs) fn bad_descriptor() -> Error { + os(ERROR_INVALID_HANDLE) +} +fn create(path: &FsPath, access: u32, disposition: u32, flags: u32) -> Result { + // SAFETY: NUL-terminated prepared path, default security and no template. + let h = unsafe { + CreateFileW( + path.wide().as_ptr(), + access, + FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, + ptr::null(), + disposition, + flags, + ptr::null_mut(), + ) + }; + if h == INVALID_HANDLE_VALUE { + return Err(last()); + } + // SAFETY: CreateFileW returned a new handle owned by nobody else. + Ok(unsafe { OwnedHandle::from_raw_handle(h) }) +} +fn raw(object: Option<&mut Object>) -> Result { + match object { + Some(Object::File(f)) => Ok(f.handle.as_raw_handle()), + Some(Object::Dir(d)) => Ok(d.handle.as_raw_handle()), + _ => Err(bad_descriptor()), + } +} +fn nofollow(follow: bool) -> u32 { + if follow { + 0 + } else { + FILE_FLAG_OPEN_REPARSE_POINT + } +} + +pub(in crate::fs) fn open(path: &FsPath, o: &FileOptions) -> Result { + let mut access = match (o.read, o.write || o.append) { + (true, false) => FILE_GENERIC_READ, + (false, true) => FILE_GENERIC_WRITE, + (true, true) => FILE_GENERIC_READ | FILE_GENERIC_WRITE, + (false, false) => return Err(Error::new(ErrorKind::InvalidInput)), + }; + if o.append { + access = (access & !FILE_WRITE_DATA) | FILE_APPEND_DATA; + } + // The disposition table follows libuv's fs__open. + let disposition = match (o.create || o.exclusive, o.exclusive, o.truncate) { + (true, true, _) => CREATE_NEW, + (true, false, true) => CREATE_ALWAYS, + (true, false, false) => OPEN_ALWAYS, + (false, _, true) => TRUNCATE_EXISTING, + (false, _, false) => OPEN_EXISTING, + }; + let mut flags = + FILE_ATTRIBUTE_NORMAL | FILE_FLAG_BACKUP_SEMANTICS | nofollow(o.follow_symlinks); + if (o.create || o.exclusive) && o.mode & 0o200 == 0 { + flags |= FILE_ATTRIBUTE_READONLY; + } + if o.sync || o.data_sync { + flags |= FILE_FLAG_WRITE_THROUGH; + } + Ok(File { + handle: create(path, access, disposition, flags)?, + cursor: 0, + append: o.append, + }) +} +pub(in crate::fs) fn open_dir(path: &FsPath) -> Result { + let handle = create( + path, + FILE_LIST_DIRECTORY | FILE_READ_ATTRIBUTES, + OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS, + )?; + let wide = &path.wide()[..path.wide().len() - 1]; + let mut pattern = Vec::with_capacity(wide.len() + 3); + pattern.extend_from_slice(wide); + if !matches!(wide.last(), Some(&0x5c | &0x2f)) { + pattern.push(0x5c); + } + pattern.extend_from_slice(&[0x2a, 0]); + Ok(Dir { + handle, + pattern: pattern.into_boxed_slice(), + search: INVALID_HANDLE_VALUE, + data: Box::default(), + pending: false, + done: false, + }) +} +pub(in crate::fs) fn close(file: File) -> Result<()> { + let handle = std::os::windows::io::IntoRawHandle::into_raw_handle(file.handle); + // SAFETY: the owned handle is closed exactly once. + check(unsafe { CloseHandle(handle) }) +} + +fn overlapped(offset: u64) -> OVERLAPPED { + OVERLAPPED { + Internal: 0, + InternalHigh: 0, + Anonymous: OVERLAPPED_0 { + Anonymous: OVERLAPPED_0_0 { + Offset: offset as u32, + OffsetHigh: (offset >> 32) as u32, + }, + }, + hEvent: ptr::null_mut(), + } +} + +pub(in crate::fs) fn execute(request: FsRequest, object: Option<&mut Object>) -> Result { + match request { + FsRequest::Read { + mut buffer, offset, .. + } => { + let Some(Object::File(file)) = object else { + return Err(bad_descriptor()); + }; + let out = output(&mut buffer); + let at = offset.unwrap_or(file.cursor); + let mut ov = overlapped(at); + let mut n = 0u32; + let len = out.len().min(u32::MAX as usize) as u32; + // SAFETY: synchronous handle, exclusive output region and a live + // OVERLAPPED that only carries the offset for this call. + if unsafe { + ReadFile( + file.handle.as_raw_handle(), + out.as_mut_ptr(), + len, + &mut n, + &mut ov, + ) + } == 0 + { + let e = last(); + if e.os != Some(ERROR_HANDLE_EOF as i32) { + return Err(e); + } + n = 0; + } + if offset.is_none() { + file.cursor += u64::from(n); + } + Ok(FsOutput::Read(n as usize)) + } + FsRequest::Write { buffer, offset, .. } => { + let Some(Object::File(file)) = object else { + return Err(bad_descriptor()); + }; + let bytes = buffer.as_slice(); + // All-ones offsets append at the end of the file. + let at = if file.append { + u64::MAX + } else { + offset.unwrap_or(file.cursor) + }; + let mut ov = overlapped(at); + let mut n = 0u32; + let len = bytes.len().min(u32::MAX as usize) as u32; + // SAFETY: synchronous handle, initialized input kept alive by the request. + check(unsafe { + WriteFile( + file.handle.as_raw_handle(), + bytes.as_ptr(), + len, + &mut n, + &mut ov, + ) + })?; + if offset.is_none() && !file.append { + file.cursor += u64::from(n); + } + Ok(FsOutput::Wrote(n as usize)) + } + FsRequest::Sync { .. } => { + // SAFETY: live owned handle. + check(unsafe { FlushFileBuffers(raw(object)?) }).map(|()| FsOutput::Done) + } + FsRequest::Truncate { size, .. } => { + let info = FILE_END_OF_FILE_INFO { + EndOfFile: i64::try_from(size).map_err(|_| os(ERROR_INVALID_PARAMETER))?, + }; + // SAFETY: live handle and a correctly sized information record. + check(unsafe { + SetFileInformationByHandle( + raw(object)?, + FileEndOfFileInfo, + (&info as *const FILE_END_OF_FILE_INFO).cast(), + size_of::() as u32, + ) + }) + .map(|()| FsOutput::Done) + } + FsRequest::Stat { + path, + follow_symlinks, + } => { + let h = create( + &path, + FILE_READ_ATTRIBUTES, + OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS | nofollow(follow_symlinks), + )?; + metadata(h.as_raw_handle()).map(FsOutput::Metadata) + } + FsRequest::Fstat { .. } => metadata(raw(object)?).map(FsOutput::Metadata), + FsRequest::ReadDir { mut buffer, .. } => { + let Some(Object::Dir(dir)) = object else { + return Err(bad_descriptor()); + }; + read_dir(dir, output(&mut buffer)) + } + FsRequest::Mkdir { path, .. } => { + // SAFETY: prepared path, default security. + check(unsafe { CreateDirectoryW(path.wide().as_ptr(), ptr::null()) }) + .map(|()| FsOutput::Done) + } + FsRequest::Rmdir { path } => { + // SAFETY: prepared path. + check(unsafe { RemoveDirectoryW(path.wide().as_ptr()) }).map(|()| FsOutput::Done) + } + FsRequest::Unlink { path } => { + // SAFETY: prepared path. + check(unsafe { DeleteFileW(path.wide().as_ptr()) }).map(|()| FsOutput::Done) + } + FsRequest::Rename { from, to } => { + // SAFETY: both prepared paths live through the call. + check(unsafe { + MoveFileExW( + from.wide().as_ptr(), + to.wide().as_ptr(), + MOVEFILE_REPLACE_EXISTING, + ) + }) + .map(|()| FsOutput::Done) + } + FsRequest::Link { existing, link } => { + // SAFETY: both prepared paths live through the call. + check(unsafe { + CreateHardLinkW(link.wide().as_ptr(), existing.wide().as_ptr(), ptr::null()) + }) + .map(|()| FsOutput::Done) + } + FsRequest::Symlink { target, link, kind } => { + let base = if kind == SymlinkKind::Directory { + SYMBOLIC_LINK_FLAG_DIRECTORY + } else { + 0 + }; + let create = |flags| { + // SAFETY: both prepared paths live through the call. + unsafe { CreateSymbolicLinkW(link.wide().as_ptr(), target.wide().as_ptr(), flags) } + }; + // Unprivileged creation needs developer mode; older systems reject the flag. + if create(base | SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE) { + return Ok(FsOutput::Done); + } + let e = last(); + if e.os != Some(ERROR_INVALID_PARAMETER as i32) { + return Err(e); + } + if create(base) { + Ok(FsOutput::Done) + } else { + Err(last()) + } + } + FsRequest::ReadLink { path, mut buffer } => { + use std::os::windows::ffi::OsStrExt; + let content = std::fs::read_link(path.as_path()).map_err(Error::from)?; + let units: Vec = content.as_os_str().encode_wide().collect(); + let out = output(&mut buffer); + let mut used = 0; + if !wtf8(&units, out, &mut used) { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + Ok(FsOutput::Bytes(used)) + } + FsRequest::RealPath { path, mut buffer } => { + let h = create( + &path, + FILE_READ_ATTRIBUTES, + OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS, + )?; + let mut wide = [0u16; 32768]; + // SAFETY: live handle and writable output of the stated length. + let n = unsafe { + GetFinalPathNameByHandleW( + h.as_raw_handle(), + wide.as_mut_ptr(), + wide.len() as u32, + FILE_NAME_NORMALIZED | VOLUME_NAME_DOS, + ) + } as usize; + if n == 0 { + return Err(last()); + } + if n >= wide.len() { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + let full = &wide[..n]; + let out = output(&mut buffer); + let mut used = 0; + const VERBATIM: &[u16] = &[0x5c, 0x5c, 0x3f, 0x5c]; + const UNC: &[u16] = &[0x55, 0x4e, 0x43, 0x5c]; + let fits = if let Some(rest) = full.strip_prefix(VERBATIM) { + if let Some(share) = rest.strip_prefix(UNC) { + wtf8(&[0x5c, 0x5c], out, &mut used) && wtf8(share, out, &mut used) + } else { + wtf8(rest, out, &mut used) + } + } else { + wtf8(full, out, &mut used) + }; + if !fits { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + Ok(FsOutput::Bytes(used)) + } + FsRequest::Access { path, mode } => { + // SAFETY: prepared path. + let attributes = unsafe { GetFileAttributesW(path.wide().as_ptr()) }; + if attributes == INVALID_FILE_ATTRIBUTES { + return Err(last()); + } + let AccessMode { write, .. } = mode; + if write + && attributes & FILE_ATTRIBUTE_READONLY != 0 + && attributes & FILE_ATTRIBUTE_DIRECTORY == 0 + { + return Err(os(ERROR_ACCESS_DENIED)); + } + Ok(FsOutput::Done) + } + FsRequest::Chmod { target, mode } => { + let readonly = mode & 0o200 == 0; + let toggle = |attributes: u32| { + let next = if readonly { + attributes | FILE_ATTRIBUTE_READONLY + } else { + attributes & !FILE_ATTRIBUTE_READONLY + }; + if next == 0 { + FILE_ATTRIBUTE_NORMAL + } else { + next + } + }; + match target { + FsTarget::Path { path, .. } => { + // SAFETY: prepared path. + let attributes = unsafe { GetFileAttributesW(path.wide().as_ptr()) }; + if attributes == INVALID_FILE_ATTRIBUTES { + return Err(last()); + } + // SAFETY: prepared path and attribute bits. + check(unsafe { SetFileAttributesW(path.wide().as_ptr(), toggle(attributes)) })?; + } + FsTarget::File(_) => { + let h = raw(object)?; + // SAFETY: FILE_BASIC_INFO is plain integers. + let mut basic: FILE_BASIC_INFO = unsafe { zeroed() }; + // SAFETY: live handle and a correctly sized output record. + check(unsafe { + GetFileInformationByHandleEx( + h, + FileBasicInfo, + (&mut basic as *mut FILE_BASIC_INFO).cast(), + size_of::() as u32, + ) + })?; + let update = FILE_BASIC_INFO { + CreationTime: 0, + LastAccessTime: 0, + LastWriteTime: 0, + ChangeTime: 0, + FileAttributes: toggle(basic.FileAttributes), + }; + // SAFETY: live handle; zero times leave the timestamps unchanged. + check(unsafe { + SetFileInformationByHandle( + h, + FileBasicInfo, + (&update as *const FILE_BASIC_INFO).cast(), + size_of::() as u32, + ) + })?; + } + } + Ok(FsOutput::Done) + } + FsRequest::Chown { .. } => Err(Error::new(ErrorKind::Unsupported)), + FsRequest::SetTimes { + target, + accessed, + modified, + } => { + let opened; + let h = match &target { + FsTarget::Path { + path, + follow_symlinks, + } => { + opened = create( + path, + FILE_WRITE_ATTRIBUTES, + OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS | nofollow(*follow_symlinks), + )?; + opened.as_raw_handle() + } + FsTarget::File(_) => raw(object)?, + }; + let a = filetime(accessed); + let m = filetime(modified); + let pointer = |t: &Option| t.as_ref().map_or(ptr::null(), |t| t as *const _); + // SAFETY: live handle; null pointers leave a timestamp unchanged. + check(unsafe { SetFileTime(h, ptr::null(), pointer(&a), pointer(&m)) }) + .map(|()| FsOutput::Done) + } + FsRequest::CopyFile { + from, + to, + exclusive, + } => { + // SAFETY: both prepared paths live through the call. + check(unsafe { + CopyFileW( + from.wide().as_ptr(), + to.wide().as_ptr(), + BOOL::from(exclusive), + ) + }) + .map(|()| FsOutput::Done) + } + FsRequest::Open { .. } | FsRequest::OpenDir { .. } | FsRequest::Close { .. } => { + unreachable!("handled by the service") + } + } +} + +const EPOCH_DIFFERENCE: i64 = 11_644_473_600; +fn from_ticks(ticks: i64) -> Option { + (ticks != 0).then(|| FileTime { + seconds: ticks.div_euclid(10_000_000) - EPOCH_DIFFERENCE, + nanoseconds: (ticks.rem_euclid(10_000_000) * 100) as u32, + }) +} +fn ticks(t: FILETIME) -> i64 { + ((u64::from(t.dwHighDateTime) << 32) | u64::from(t.dwLowDateTime)) as i64 +} +fn filetime(change: TimeChange) -> Option { + let ticks = match change { + TimeChange::Keep => return None, + TimeChange::Now => { + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default(); + (now.as_secs() as i64 + EPOCH_DIFFERENCE) * 10_000_000 + + i64::from(now.subsec_nanos() / 100) + } + TimeChange::At(t) => { + (t.seconds + EPOCH_DIFFERENCE) * 10_000_000 + i64::from(t.nanoseconds / 100) + } + } as u64; + Some(FILETIME { + dwLowDateTime: ticks as u32, + dwHighDateTime: (ticks >> 32) as u32, + }) +} +/// Name-surrogate reparse points (symbolic links and junctions) read as links. +fn is_link(h: HANDLE, attributes: u32) -> bool { + if attributes & FILE_ATTRIBUTE_REPARSE_POINT == 0 { + return false; + } + // SAFETY: plain integer record. + let mut tag: FILE_ATTRIBUTE_TAG_INFO = unsafe { zeroed() }; + // SAFETY: live handle and a correctly sized output record. + unsafe { + GetFileInformationByHandleEx( + h, + FileAttributeTagInfo, + (&mut tag as *mut FILE_ATTRIBUTE_TAG_INFO).cast(), + size_of::() as u32, + ) != 0 + && tag.ReparseTag & 0x2000_0000 != 0 + } +} +fn metadata(h: HANDLE) -> Result { + // SAFETY: plain integer record. + let mut info: BY_HANDLE_FILE_INFORMATION = unsafe { zeroed() }; + // SAFETY: live handle and writable output record. + if unsafe { GetFileInformationByHandle(h, &mut info) } == 0 { + let e = last(); + // SAFETY: live handle. + let kind = match unsafe { GetFileType(h) } { + FILE_TYPE_CHAR => FileType::CharDevice, + FILE_TYPE_PIPE => FileType::Fifo, + _ => return Err(e), + }; + return Ok(FileMetadata { + kind, + size: 0, + mode: Some(if kind == FileType::CharDevice { + 0o020666 + } else { + 0o010666 + }), + device: None, + inode: None, + links: None, + uid: None, + gid: None, + rdev: None, + block_size: None, + blocks: None, + accessed: None, + modified: None, + changed: None, + created: None, + }); + } + // SAFETY: plain integer record. + let mut basic: FILE_BASIC_INFO = unsafe { zeroed() }; + // SAFETY: live handle and a correctly sized output record. + let basic = (unsafe { + GetFileInformationByHandleEx( + h, + FileBasicInfo, + (&mut basic as *mut FILE_BASIC_INFO).cast(), + size_of::() as u32, + ) + } != 0) + .then_some(basic); + let attributes = info.dwFileAttributes; + let kind = if is_link(h, attributes) { + FileType::Symlink + } else if attributes & FILE_ATTRIBUTE_DIRECTORY != 0 { + FileType::Directory + } else { + FileType::File + }; + let type_bits = match kind { + FileType::Symlink => 0o120000, + FileType::Directory => 0o040000, + _ => 0o100000, + }; + let permissions = if attributes & FILE_ATTRIBUTE_READONLY != 0 { + 0o444 + } else { + 0o666 + }; + Ok(FileMetadata { + kind, + size: (u64::from(info.nFileSizeHigh) << 32) | u64::from(info.nFileSizeLow), + mode: Some(type_bits | permissions), + device: Some(u64::from(info.dwVolumeSerialNumber)), + inode: Some((u64::from(info.nFileIndexHigh) << 32) | u64::from(info.nFileIndexLow)), + links: Some(u64::from(info.nNumberOfLinks)), + uid: None, + gid: None, + rdev: None, + block_size: None, + blocks: None, + accessed: from_ticks(ticks(info.ftLastAccessTime)), + modified: from_ticks(ticks(info.ftLastWriteTime)), + changed: basic.and_then(|b| from_ticks(b.ChangeTime)), + created: from_ticks(ticks(info.ftCreationTime)), + }) +} + +fn read_dir(dir: &mut Dir, out: &mut [u8]) -> Result { + let mut used = 0; + loop { + if dir.done { + return Ok(FsOutput::Directory { n: used, eof: true }); + } + if dir.search == INVALID_HANDLE_VALUE { + // SAFETY: NUL-terminated pattern and writable find record. + let search = unsafe { FindFirstFileW(dir.pattern.as_ptr(), &mut *dir.data) }; + if search == INVALID_HANDLE_VALUE { + let e = last(); + if e.os == Some(ERROR_FILE_NOT_FOUND as i32) { + dir.done = true; + continue; + } + return Err(e); + } + dir.search = search; + dir.pending = true; + } + if !dir.pending { + // SAFETY: live search handle and writable find record. + if unsafe { FindNextFileW(dir.search, &mut *dir.data) } == 0 { + let e = last(); + if e.os != Some(ERROR_NO_MORE_FILES as i32) { + return Err(e); + } + dir.done = true; + continue; + } + dir.pending = true; + } + let data = &*dir.data; + let length = data + .cFileName + .iter() + .position(|&c| c == 0) + .unwrap_or(data.cFileName.len()); + let name = &data.cFileName[..length]; + if name == [0x2e] || name == [0x2e, 0x2e] { + dir.pending = false; + continue; + } + let attributes = data.dwFileAttributes; + let tag = if attributes & FILE_ATTRIBUTE_REPARSE_POINT != 0 + && data.dwReserved0 & 0x2000_0000 != 0 + { + FileType::Symlink + } else if attributes & FILE_ATTRIBUTE_DIRECTORY != 0 { + FileType::Directory + } else { + FileType::File + }; + let start = used; + let mut end = used + RECORD_HEADER; + let fits = end <= out.len() + && wtf8(name, out, &mut end) + && end - start - RECORD_HEADER <= usize::from(u16::MAX); + if !fits { + if used == 0 { + return Err(Error::new(ErrorKind::ResourceLimit)); + } + return Ok(FsOutput::Directory { + n: used, + eof: false, + }); + } + out[start] = tag as u8; + out[start + 1] = 0; + let len = (end - start - RECORD_HEADER) as u16; + out[start + 2..start + 4].copy_from_slice(&len.to_le_bytes()); + used = end; + dir.pending = false; + } +} diff --git a/crates/turnloop/src/lib.rs b/crates/turnloop/src/lib.rs index e458f22..1e571e9 100644 --- a/crates/turnloop/src/lib.rs +++ b/crates/turnloop/src/lib.rs @@ -73,6 +73,13 @@ pub use backend::unix::Detached; mod blocking; pub use blocking::{DnsRequest, PoolConfig}; +mod fs; +pub use fs::{ + AccessMode, DirEntries, FileMetadata, FileOptions, FileTime, FileType, FsPath, FsRequest, + FsResult, FsTarget, Metadata, RECORD_HEADER, SymlinkKind, TimeChange, WatchEvents, WatchKind, + WatchOptions, +}; + mod time; pub use time::Instant; diff --git a/crates/turnloop/src/types.rs b/crates/turnloop/src/types.rs index ebf19ca..466e3d8 100644 --- a/crates/turnloop/src/types.rs +++ b/crates/turnloop/src/types.rs @@ -57,6 +57,16 @@ pub enum ErrorKind { BrokenPipe, /// A configured capacity or native resource limit was reached. ResourceLimit, + /// Permissions or a capability boundary denied a filesystem operation. + PermissionDenied, + /// The target already exists (for example an exclusive create). + AlreadyExists, + /// A path component that must be a directory is not one. + NotADirectory, + /// A file operation targeted a directory. + IsADirectory, + /// A directory to be removed or replaced is not empty. + DirectoryNotEmpty, /// An error without a more specific portable category. Other, } @@ -88,6 +98,11 @@ impl From for Error { I::ConnectionReset | I::ConnectionAborted => ErrorKind::ConnectionReset, I::BrokenPipe => ErrorKind::BrokenPipe, I::OutOfMemory => ErrorKind::ResourceLimit, + I::PermissionDenied => ErrorKind::PermissionDenied, + I::AlreadyExists => ErrorKind::AlreadyExists, + I::NotADirectory => ErrorKind::NotADirectory, + I::IsADirectory => ErrorKind::IsADirectory, + I::DirectoryNotEmpty => ErrorKind::DirectoryNotEmpty, _ => ErrorKind::Other, }; Self { diff --git a/docs/lanes/tl-i06.md b/docs/lanes/tl-i06.md new file mode 100644 index 0000000..783f944 --- /dev/null +++ b/docs/lanes/tl-i06.md @@ -0,0 +1,405 @@ +# tl-i06 — typed file operations and filesystem watches + +Status: implemented for every platform in scope. macOS runtime, WASI 0.2 and WASI +0.3 (Wasmtime 46) are verified here; Linux, Windows and FreeBSD compile cleanly +(strict cross-clippy) but their runtime is **UNRUN** and must come from CI. +Base: `main` 63f5eee (the Codex checkpoint was rebased onto it, then replaced). +Commits are on `lane/tl-i06`; nothing was pushed. No dependency, lockfile or soak +policy change. + +## 1. What the checkpoint had, and what was kept + +The Codex checkpoint (`2fc61e7`, originally `d96ba84`) was a useful sketch but was +not kept as code: + +- Watches ran one helper thread per watch on Linux, FreeBSD and Windows. DESIGN + §5a.4 moves fs-watch off dedicated threads, and each turn scanned every watch + slot (O(max_handles) per turn). Events carried no file names, which Node's + `fs.watch(eventType, filename)` needs. +- The pool path returned a post-acceptance `ResourceLimit` when the shared queue + was full while a FIFO successor started, used `expect` on a withdrawn job + (loop drop would panic a pool thread) and forbade non-regular files. +- Directory listing re-opened and skipped `cookie` entries per page (O(n²), and + wrong under concurrent mutation). There was no WASI implementation. +- It overwrote `LANE_REPORT.md` (the adapters-db report); that file is restored. + +Kept ideas: prepared `FsPath`, handle-scoped FIFOs, `Arc` reusable pool work, +the error-kind additions, and FSEvents on a dispatch queue. + +## 2. Request surface (API) + +Everything is in `turnloop::` (module `crates/turnloop/src/fs/mod.rs`). + +```rust +impl Loop { + // Native: shared blocking pool. WASI: wasi:filesystem preopens. Web: Unsupported. + pub fn fs(&mut self, request: FsRequest, token: Token) -> Result; + // inotify / FSEvents+kqueue / kqueue / ReadDirectoryChangesW. WASI, web: Unsupported. + pub fn fs_watch(&mut self, path: &FsPath, options: WatchOptions, token: Token) -> Result; + pub fn fs_watch_stop(&mut self, watch: Handle, token: Token) -> Result<()>; // Stopped, Closed +} +pub struct FsPath; // prepared once (CString / UTF-16 / UTF-8); clone = Arc bump +pub enum FsRequest { + Open { path, options: FileOptions }, OpenDir { path }, // -> Opened(Handle) + Close { file }, // native close with its error + Read { file, buffer: ReadBuf, offset: Option }, // None = handle cursor + Write { file, buffer: WriteBuf, offset: Option }, // one write, may be short + Sync { file, data_only }, Truncate { file, size }, + Stat { path, follow_symlinks }, Fstat { file }, // -> Metadata + ReadDir { dir, buffer: ReadBuf }, // -> Directory records + Mkdir { path, mode }, Rmdir { path }, Unlink { path }, Rename { from, to }, + Link { existing, link }, Symlink { target, link, kind }, ReadLink { path, buffer }, + RealPath { path, buffer }, // -> Bytes + Access { path, mode: AccessMode }, Chmod { target: FsTarget, mode }, + Chown { target, uid, gid }, SetTimes { target, accessed: TimeChange, modified }, + CopyFile { from, to, exclusive }, +} +pub enum FsResult { Opened(Handle), Read { n, lease }, Wrote(usize), Metadata(Metadata), + Directory { n, lease, eof }, Bytes { n, lease }, Done } +OpResult::Fs(FsResult) +OpResult::Watch { events: BufLease, overflow: bool } // nonterminal batches +``` + +- **FileOptions** covers Node's string and numeric flags: read, write, create, + exclusive, truncate, append, sync (O_SYNC), data_sync (O_DSYNC), + follow_symlinks (O_NOFOLLOW), mode (umask-filtered; Windows maps the owner write + bit to the read-only attribute as libuv does). +- **FileMetadata**: kind, size, mode, device, inode, links, uid, gid, rdev, + block_size, blocks, accessed/modified/changed/created with nanoseconds. A field + the platform does not report is `None`. Linux uses `statx` for birth time. + Windows mode follows libuv (type bits | 0o666, or 0o444 when read-only). + `Metadata` is a lease into per-loop storage (dereferences to `FileMetadata`) so + a 232-byte value never enters every `Completion`; steady-state `stat` is + allocation-free. +- **Records** (`RECORD_HEADER` = 4 bytes: tag, 0, u16 length): `DirEntries` + yields `(FileType, name)`, `WatchEvents` yields `(WatchKind::{Rename, Change}, + name)`. Names are native bytes on Unix, WTF-8 on Windows, UTF-8 on WASI. A page + never splits a name; a buffer too small for the next name is `ResourceLimit` + and the entry is kept for the next read. +- **Open handles** are reserved at submission but stay hidden until `Opened`; a + failed or cancelled open never exposes a handle (its reservation is released). + `Close` closes the native object and reports its error; `Loop::close` releases + the handle (closing a still-open descriptor on the loop thread). +- **Errors**: new `ErrorKind`s `PermissionDenied`, `AlreadyExists`, + `NotADirectory`, `IsADirectory`, `DirectoryNotEmpty`; `os` keeps errno / + `GetLastError`, and on WASI the wasi-libc errno for the `error-code`. +- **Watch semantics** follow libuv: every event but content/attribute change is + `Rename`; names are relative to the watched directory with native separators; + an event on the watched root (or a watched file) carries its final component. + `overflow: true` means events were lost (kernel overflow, or the bounded + 16 KiB per-watch store filled while leases were held); the host rescans. + FSEvents reports cumulative flags per path, so a change to a recently created + file can read as `Rename` on macOS — the same as libuv/Node there. + +### Perry mapping (consumer survey of `perry-runtime/src/fs`) + +Perry performs every fs operation synchronously on the JS thread today. The +surface covers what its promise/callback/FileHandle/stream paths call: +open/close/read/write (positional and cursor), readv/writev (as repeated +Read/Write), stat/lstat/fstat (all Stats fields incl. ns and bigint), opendir, +readdir (withFileTypes from record tags; `Unknown` falls back to lstat as Node +does), mkdir, rmdir, rename, unlink, fsync/fdatasync, truncate (Open+Truncate), +ftruncate, access/exists, chmod/fchmod/lchmod, chown family, utimes/futimes/ +lutimes, symlink/readlink/realpath/link, copyFile(COPYFILE_EXCL). Host +compositions: recursive mkdir/rm/readdir/cp, mkdtemp (random name + Mkdir retry +on `AlreadyExists`), writeFile/appendFile (Write loop), `fs.watchFile` +(host timer + `Stat`, no driver thread), recursive `fs.watch` on Linux (one watch +per directory, as Node 20+ does). Not in the surface: statfs and COPYFILE_FICLONE +(host blocking jobs remain available). + +## 3. Decisions + +- **D8 resolved (DESIGN.md amended):** typed requests use the one shared bounded + pool on Linux, macOS/BSD **and Windows**. Windows per-handle synchronous workers + remain only for adopted descriptors (stdio, `Detached::from_handle`), whose reads + can block indefinitely on pipes/consoles and must not occupy the shared pool. +- **Queue reservations:** each accepted request reserves one pool queue slot + (`blocking::reserve`), so a full queue rejects *before* acceptance and a FIFO + successor always starts (`push_reserved` never fails or grows the queue). + Boxed host jobs honour reservations too. Only a FIFO head is ever queued. +- **Watches are loop handles, no turnloop threads (§5a.4):** Linux: one inotify fd + per loop registered with epoll (shared watch descriptors for the same inode); + kqueue: `EVFILT_VNODE` on the loop's kqueue (files on macOS; everything on + BSD/iOS, carried through a new `Ready::vnode` field); macOS directories: + FSEvents streams (CoreServices loaded lazily with `dlopen`, no launch-time + framework dependency) on one private serial dispatch queue — the only OS-owned + threads, now listed as an exception in §5a.4; Windows: overlapped + `ReadDirectoryChangesW` associated with the loop's IOCP under a never-reused + key, re-armed per completion, storage retired until the kernel acknowledges. +- **WASI:** no threads, so requests run inside `poll`, at most the event budget per + turn, against preopens only (no ambient path authority). 0.2 lowers reads + directly into caller memory through the backend's scratch-arena + `cabi_realloc`. 0.3 lowers every import by hand and blocks on a private + waitable set, preserving the shadow-stack context around each raw call: the + generated bindings trap in debug on the pinned toolchain (reproduced with a bare + `wasip3::filesystem::preopens::get_directories()` call, outside turnloop). + Unsupported before acceptance on WASI: RealPath, Chmod, Chown, Access with + permission bits, and watches. +- **Web:** Unsupported; no OPFS host mapping was invented. +- **Pool vs backend selection** is a `Backend::FILESYSTEM` constant (`Pool`, + `Backend`, `Unsupported`), so the driver branches statically. + +## 4. Implementation map + +| Area | Files | +|---|---| +| Surface, records, metadata leases | `crates/turnloop/src/fs/mod.rs` | +| Native pool service (FIFO, reservations, withdrawal on drop) | `fs/service.rs` | +| Unix / Windows system calls | `fs/unix.rs`, `fs/windows.rs` | +| Bounded watch record store | `fs/watch.rs` | +| inotify + kqueue vnode + FSEvents | `backend/watch.rs`, `backend/fsevents.rs` | +| ReadDirectoryChangesW on IOCP | `backend/iocp/watch.rs` | +| WASI core (queues, preopen resolution) | `backend/wasi_fs.rs` | +| WASI 0.2 / 0.3 bindings | `backend/wasi_p2/fs.rs` (+ `abi::file_read`), `backend/wasi_p3/fs.rs` | +| Driver wiring | `driver.rs` (`fs`, `fs_watch`, hidden handles), `backend/mod.rs` contract | +| Contracts | `turnloop-contract/src/filesystem.rs`, `tests/filesystem.rs`, `tests/wasi.rs`, `tests/allocations.rs` | +| Wasmtime fixture preopen | `scripts/ci/wasmtime-runner.sh` (fresh `/turnloop-fs`, removed on exit) | + +## 5. What the tests prove + +Shared contracts (native and both WASI versions): + +- `bytes_metadata_namespace`: cursor and positional writes and reads (provided and + pooled) with bytes verified by `std::fs`, EOF, sync, fstat/stat kind and size + (inode, device, link count and type bits natively), truncate, nanosecond `SetTimes` round trip, `Close` then rejected + requests, hard link count, symlink + lstat + readlink, realpath (native), access, + exclusive copy + `AlreadyExists`, rename, `DirectoryNotEmpty`, directory pages at + 9/16/4096 bytes alternating provided/pooled buffers, full cleanup, `!alive()`. +- `errors`: `NotFound`, `AlreadyExists`, `DirectoryNotEmpty` and (not on Windows, + which reports a missing path) `NotADirectory`, each with its native code; a 3-byte page is `ResourceLimit` and the entry is delivered next. +- `fifo_cancel_close`: close before any turn: every request terminates exactly once + before `Closed`; an unstarted read never touches its buffer; file content matches + whether the head write won or was cancelled. +- `pooled_lease_wait`: with the only lease held, a pooled read waits across three + timer expiries with at most two turns each (no spin), then completes with bytes. +- WASI `capability_scope`: paths outside preopens are `NotFound`, unsupported + requests reject before acceptance, watches are `Unsupported`. + +Native only: + +- `pool_backpressure_rejects_before_acceptance`: with all pool threads held, exactly + `queue_capacity` requests are accepted, the next is `ResourceLimit`, host jobs are + bounded by the same reservations, every accepted request completes exactly once. +- `cancellation_withdraws_unstarted_and_reports_started`: a queued cancel completes + immediately without touching its buffer; the queued head reports `Cancelled`; the + FIFO successor then runs with correct bytes. +- `cancelled_open_and_loop_drop_never_touch_buffers`: a cancelled open exposes no + handle; dropping the loop withdraws its queued pool jobs, buffers mutated after + drop are never touched when the pool threads later dequeue the stale entries. +- `permission_denied_is_reported`: read-only file rejects writers and `Access(write)` + with `PermissionDenied` (requires an unprivileged runner). +- `watch_directory`: liveness barrier, content change (`Change`; FSEvents may + report `Rename` for a recently created file), rename then delete with the + old name before the new, no nested names, `Stopped` before `Closed`, nothing after. +- `watch_file`: file watch reports only its base name, `Change` then `Rename`; + close gives `Cancelled` then `Closed`. +- `watch_backpressure`: with the only lease held, 3000 events produce no batch and + no spin (idle turns without an OS wait ≤ 1 per timer), and loss is reported. +- `watch_recursive`: nested names with native separators on macOS/Windows, none in a + flat watch; `Unsupported` on inotify/kqueue. +- Allocation gates (`tests/allocations.rs`): 200 rounds of typed write, provided + read, pooled read, fstat, stat and truncate allocate **zero** times (loop thread; + all threads on Windows; release WASI 0.2 and 0.3), after proving the counter + observes an allocation; 100 watch batches allocate zero times natively. + +## 6. Verification ledger + +Every command, in order, with its result. `CARGO_BUILD_JOBS=4`. Clippy flags are +`-D warnings -D clippy::undocumented_unsafe_blocks` throughout. + +| # | Command | Result | +|---|---|---| +| 1 | Inspection: `cat`/`sed`/`grep`/`git show` of the brief, DESIGN.md, REMAINING_WORK.md I06, checkpoint diff and log tail, core sources; read-only survey of Perry `perry-runtime/src/fs` and `perry-stdlib` | PASS | +| 2 | `git fetch origin main && git rebase origin/main` (checkpoint onto 63f5eee) | PASS | +| 3 | `cargo check -p turnloop` (first build of the new surface) | FAIL (missing `fs/watch.rs`, non-exhaustive `Operation::WatchFs`), fixed, then PASS | +| 4 | `cargo clippy -p turnloop --all-targets -- $FLAGS` | FAIL (dead `Service::has_work`, unannotated `transmute`, large enum variants), fixed, then PASS | +| 5 | `cargo test -p turnloop-contract --test filesystem -- --test-threads=1` (macOS) | FAIL (directory page sort by type; an FSEvents stream left running after a failed test's loop drop, UB check on the dispatch thread), fixed: sort by name, stream teardown moved into `Stream::drop` | +| 6 | same, rerun | FAIL (`watch_directory`: FSEvents reported the append to a new file as Rename; `watch_backpressure`: 10 turns counted as spin although each had an OS wait), fixed: FSEvents cumulative-flag tolerance on macOS only, spin measured as turns with no OS wait and no output | +| 7 | same (after fixes) | PASS 11/11, repeated 3 times PASS | +| 8 | `cargo fmt --all && cargo fmt --all --check` | PASS | +| 9 | `cargo clippy --target x86_64-unknown-linux-gnu -p turnloop -p turnloop-contract --all-targets -- $FLAGS` | FAIL (`Ready::vnode` unread on epoll, `Ring::empty` unused, `unnecessary_cast` of mode bits), fixed, then PASS | +| 10 | `cargo clippy --target x86_64-pc-windows-msvc -p turnloop -p turnloop-contract --all-targets -- $FLAGS` | FAIL (`BOOL` import; dead watch store before IOCP watches existed), fixed, then PASS | +| 11 | clippy: macOS, Linux, Windows (`-p turnloop -p turnloop-contract --all-targets`) after IOCP watches | PASS ×3 | +| 12 | `cargo test -p turnloop-contract --test allocations -- --test-threads=1 steady_typed steady_watch` | FAIL (watch subject stalled: macOS reports content changes on close, not on write), fixed (reopen per round on Unix), then PASS | +| 13 | `cargo clippy --target wasm32-wasip2 -p turnloop -p turnloop-contract --all-targets -- $FLAGS` | FAIL (`BufferPool::available` cfg, `WatchFs` match arms), fixed; FAIL (`large_enum_variant` of `FsResult` on 32-bit), fixed by metadata leases; PASS | +| 14 | clippy macOS / wasm32-wasip2 / Linux / Windows after metadata leases | FAIL (`vec_box`), allowed with reason, then PASS ×4 | +| 15 | `CARGO_TARGET_WASM32_WASIP2_RUNNER=scripts/ci/wasmtime-runner.sh cargo +nightly-2026-08-20 test --target wasm32-wasip2 -p turnloop-contract --test wasi --all-features -- --test-threads=1 filesystem` | PASS 5/5 | +| 16 | same with `--release --test allocations ... steady_typed` | PASS (zero allocations) | +| 17 | `cargo +nightly-2026-09-07 clippy --target wasm32-wasip3 -p turnloop -p turnloop-contract --all-targets --features turnloop/wasi-p3-experimental,turnloop-contract/wasi-p3-experimental -- $FLAGS` (generated-binding version) | FAIL (unused import), fixed, then PASS | +| 18 | `cargo +nightly-2026-09-07 test --target wasm32-wasip3 -p turnloop-contract --test wasi --all-features -- --test-threads=1 filesystem` (generated-binding version) | FAIL: wasm trap (out-of-bounds) in `wasip3 get_directories` lifting, debug | +| 19 | same `--release` (generated-binding version) | PASS 5/5 (not accepted: debug must pass) | +| 20 | Probe test calling bare `wasip3::filesystem::preopens::get_directories()` in the p3 debug `wasi` binary (temporary `wasip3` dev-dependency, reverted) | FAIL (same trap): toolchain issue, not turnloop | +| 21 | p3 clippy (#17) after hand-lowering every import | PASS | +| 22 | p3 `--test wasi ... filesystem`, debug and `--release` | PASS 5/5 and 5/5 | +| 23 | p3 `--release --test allocations ... steady_typed` | PASS (zero allocations) | +| 24 | `cargo fmt --all --check` | PASS | +| 25 | `cargo test -p turnloop -p turnloop-contract -- --test-threads=1` | PASS (lib 15, contract lib 22, allocations 12, filesystem 11, native_surface 17, lifetimes 1, others 0 on macOS) | +| 26 | `cargo test --workspace -- --test-threads=1` (foreground) | UNRUN: killed by the tool's 10-minute limit while compiling | +| 27 | `cargo test --workspace -- --test-threads=1` (background) | PASS, 0 failed. Correction: the log also captured the still-running cargo from #26 (every doc-test group appears twice), so the "80 groups / 339 tests" first recorded here was double-counted; a clean single run is #47 | +| 28 | `cargo test -p turnloop-contract --test filesystem -- --test-threads=1` (with the loop-drop test) | PASS 12/12 | +| 29 | `cargo clippy --workspace --all-targets -- $FLAGS` | PASS | +| 30 | `cargo clippy --workspace --all-targets --all-features -- $FLAGS` | PASS | +| 31 | `cargo clippy --target x86_64-unknown-linux-gnu -p turnloop -p turnloop-contract -p turnloop-io --all-targets [--all-features] -- $FLAGS` | PASS ×2 | +| 32 | `cargo clippy --target x86_64-pc-windows-msvc -p turnloop -p turnloop-contract -p turnloop-io --all-targets [--all-features] -- $FLAGS` | PASS ×2 | +| 33 | `cargo clippy --target wasm32-wasip2 -p turnloop -p turnloop-contract -p turnloop-io --all-targets --all-features -- $FLAGS` | PASS | +| 34 | `cargo +nightly-2026-09-07 clippy --target wasm32-wasip3 -p turnloop -p turnloop-contract -p turnloop-io --all-targets --all-features -- $FLAGS` | PASS | +| 35 | `cargo clippy --target wasm32-unknown-unknown -p turnloop -p turnloop-contract --all-targets --all-features -- $FLAGS` | FAIL (`FsRequest::read_buffer` unused on web), fixed | +| 36 | web clippy (#35) default and `--all-features` | PASS ×2 | +| 37 | `cargo clippy --target wasm32-wasip1-threads -p turnloop --all-targets --all-features -- $FLAGS` (unsupported backend) | FAIL (same dead code), fixed, then PASS | +| 38 | `bash scripts/ci/no-tokio.sh` | PASS (all 9 policy targets, default and all features) | +| 39 | `python3 scripts/ci/soak.py` | PASS (251 locked versions; the existing rustls exception) | +| 40 | `python3 scripts/ci/run-tests.py wasi --target wasm32-wasip2` | PASS (core lib 8; `wasi` debug 27 and release 27; `allocations` release 10) | +| 41 | `python3 scripts/ci/run-tests.py wasi --target wasm32-wasip3` | FAIL: core lib 10, `wasi` debug 28 and `allocations` release 11 passed (all filesystem tests passed in debug and release); release `wasi` aborted in the pre-existing `timer_precision` contract: median lateness 2.31 ms > 2 ms bound, samples 1.07–6.44 ms, host load average 148–182 (shared machine). Not filesystem code: no fs request runs in that test. | +| 41a | `cargo +nightly-2026-09-07 test --locked --release --target wasm32-wasip3 -p turnloop-contract --test wasi --all-features -- --nocapture --test-threads=1 timer_precision` ×3 | PASS ×3 (median 1.18, 1.86, 1.17 ms) | +| 41b | `python3 scripts/ci/run-tests.py wasi --target wasm32-wasip3` (rerun) | PASS (core lib 10; `wasi` debug 28 and release 28, median 1.17 ms; `allocations` release 11) | +| 42 | `cargo test -p turnloop -p turnloop-contract --all-features -- --test-threads=1` | PASS: 88 tests (lib 15, contract lib 22, allocations 13, executor 6, filesystem 12, native_surface 17, lifetimes 1, doc 2) | +| 43 | Linux runtime (epoll/inotify), Windows runtime (IOCP/RDCW), FreeBSD | UNRUN (no host) | +| 44 | `git fetch origin` | PASS, but this clone's `origin` (the local `windlass` clone) still had `main` at 63f5eee; `git fetch gh main` fetched GitHub `main` f6fc128 containing #20 and #16 | +| 45 | `git merge --no-ff gh/main` | 2 conflicts (see §10), resolved keeping both sides; merge-only tree `cargo check -p turnloop -p turnloop-contract --all-targets` PASS; committed `75011b4` | +| 46 | Rule 3 fixes: `cargo test -p turnloop --lib -- --test-threads=1 backend_path_request natively_accepted queued_core` | PASS 3/3; sabotage (flag disabled): FAIL as expected | +| 46a | `cargo test -p turnloop-contract --test filesystem -- --test-threads=1` | FAIL: `watch_backpressure` (2 idle turns: I01 now reports a notification-driven zero-timeout poll as `discovery_polls`, not `os_waits`); spin redefined as a turn with no native call and no output, plus ≤ 64 turns per 20 ms window; then PASS 13/13, `watch_backpressure` ×3 PASS; sabotage (watch `has_work` ignoring lease availability): FAIL as expected | +| 46b | wasip2 `--test wasi ... filesystem` | first version of the starvation contract passed even with the flag disabled (an open handle's request made the backend native); split into a path-only phase: PASS 6/6, sabotage FAIL ("starved behind queued posts after 480525 turns") | +| 47 | Post-merge `/scratchpad/verify2.sh`: `cargo fmt --all --check` | PASS | +| 48 | clippy `--workspace --all-targets` default and `--all-features` | PASS ×2 | +| 49 | clippy `--target x86_64-unknown-linux-gnu -p turnloop -p turnloop-contract -p turnloop-io --all-targets` default and `--all-features` | PASS ×2 | +| 50 | clippy `--target x86_64-pc-windows-msvc` (same packages) default and `--all-features` | PASS ×2 | +| 51 | clippy `--target wasm32-wasip2` (same packages) `--all-features`; `+nightly-2026-09-07 --target wasm32-wasip3` `--all-features` | PASS ×2 | +| 52 | clippy `--target wasm32-unknown-unknown -p turnloop -p turnloop-contract --all-targets` default and `--all-features` | PASS ×2 | +| 53 | `cargo test --workspace -- --test-threads=1` | PASS: 64 result groups, 278 tests, 0 failed | +| 54 | `python3 scripts/ci/run-tests.py wasi --target wasm32-wasip2` | PASS (core lib 11; `wasi` debug 33, release 33 with median lateness 1.66 ms; `allocations` release 11) | +| 55 | `python3 scripts/ci/run-tests.py wasi --target wasm32-wasip3` | PASS (core lib 13; `wasi` debug 33, release 33 with median 1.11 ms; `allocations` release 12) | +| 56 | `bash scripts/ci/no-tokio.sh` | PASS (18 target/feature graphs) | +| 57 | `python3 scripts/ci/soak.py` | PASS (251 locked versions, 1 existing exception) | + +`$FLAGS` = `-D warnings -D clippy::undocumented_unsafe_blocks`. Wasmtime 46.0.0 was +linked into this clone's ignored `.tools/bin` from the `wasm2` clone (read only). +Clippy runs through #37 used the default `nightly-2026-08-20` toolchain except the +p3 target (`nightly-2026-09-07`). + +## 7. Commits on `lane/tl-i06` + +| Commit | Content | +|---|---| +| `2fc61e7` | Codex checkpoint, rebased onto `main` 63f5eee (superseded by the commits below) | +| `b19e786` | Typed surface, pool service with reservations, Unix/Windows calls, inotify/kqueue/FSEvents watches, native contracts; restores `LANE_REPORT.md` | +| `1417d44` | ReadDirectoryChangesW on IOCP; filesystem and watch allocation gates | +| `284349d` | WASI core and 0.2 bindings; metadata leases; Wasmtime runner preopen | +| `5d83e8c` | WASI 0.3 hand-lowered bindings | +| `84adc74` | DESIGN D8/§5a.4/§6/§7.6 clarification; loop-drop buffer test; IOCP watch terminal cleanup; web/unsupported dead-code allowance | +| `58e6bda` | this report | +| `75011b4` | merge of GitHub `main` f6fc128 (#20 I01, #16 Windows semantics) | +| `d0d35b3` | DESIGN §10 rule 3 applied to filesystem requests; regressions | +| `e796489` | report update for the merge | +| `0424920` | allocation gates record thread, size and alignment of counted allocations on Windows | +| `cc9fc43` | blocking pool starts fully before first use; Windows watch gate appends through fresh handles | +| `18445f5` | `fifo_cancel_close` accepts a started, cancelled write | +| (last) | report update for PR #22 Windows CI | + +## 8. Needs CI (runtime not available here) + +- **Linux (epoll):** `tests/filesystem.rs` (inotify paths, statx, fdatasync, + permission test as non-root), `tests/allocations.rs` fs/watch gates. +- **Windows (IOCP):** the same binaries: `fs/windows.rs` (CreateFileW dispositions, + OVERLAPPED offsets, FindFirstFileW pages, reparse tags, CopyFileW, symlink + privilege on the runner), `iocp/watch.rs` (RDCW completions, cancellation + acknowledgement, retired storage on release/drop), all-thread allocation gate. +- **FreeBSD (kqueue, best effort):** vnode watches and `st_birthtime` compile paths + were not cross-checked (no FreeBSD target installed). +- **WASI:** the required `run-tests.py wasi` jobs for both targets; this lane ran + them locally (see ledger) with the updated Wasmtime runner. + +## 9. Open items + +- WASI 0.3 remains experimental (feature-gated). Each request blocks the agent on a + private waitable set inside `poll`; this is an additional wait beyond the one + counted in `PollInfo`, like 0.2's synchronous host calls. Directory entry and link + names are owned strings from the host on both WASI versions (not gated). +- Linux recursive watches are left to the host (Node's own approach); a built-in + tree-walking option would need pool-backed scans. +- Vectored `readv`/`writev`, `statfs` and clone-copy are not in the surface. +- I01 is merged and applied (§10). + +## 10. Merge of `main` with I01 (#20) and Windows semantics (#16) + +Merged GitHub `main` f6fc128 with a merge commit (`75011b4`). This clone's `origin` +remote is the local `windlass` clone, whose `main` was still 63f5eee, so the merge +source was `gh/main`. + +Conflicts, both resolved by keeping both sides: + +- `crates/turnloop/src/driver.rs`, `struct Op` and its initializer in `new_op`: + this lane's `fs: bool` (typed filesystem request) and I01's `native: bool` + (counted in `native_pending`) are both kept. +- `crates/turnloop-contract/tests/allocations.rs`, end of file: this lane's + filesystem/watch gates and helpers, then #16's + `windows_sync_pipe_character_and_disk_workers_reuse_classification_without_allocating`. + +Auto-merged without conflict and reviewed: `backend/kqueue.rs` (the new +`Ready::vnode` field next to I01's `PollInfo::native`), `backend/epoll.rs`, +`backend/mod.rs`, `backend/unix.rs`, `backend/wasi_p2.rs`, `backend/wasi_p3.rs`, +`backend/web.rs`, `completion.rs`, `DESIGN.md`, and `backend/iocp/mod.rs`, where +#16 made sync workers a per-direction pair and treats `port::STOP` as a wake; the +watch keys stay below `STOP`, so watch completions route as before. + +Rule 3 applied (`d0d35b3`): + +- `Driver::fs`: once a `Filesystem::Backend` (WASI) backend accepts a request, the + operation is native (`native = true`, `native_pending += 1`), as I01 does for + lookups. Handle requests and opens were already native through their + `Kind::Socket` handle; path requests were not, so a producer posting before every + turn starved them (the sabotage run spun for 480,525 turns). Terminal completion + retires the flag through `retire`. +- Native pool requests are **not** native operations, matching I01's definition of + queued work (blocking-pool results): their results arrive through the work port, + so a queued turn with only pool requests makes no native call. + §10 rule 3 now names both cases. +- Regressions: `driver::clock_contract::backend_path_request_is_a_pending_native_operation` + (synthetic backend: one native step while pending, `os_waits == 0` on queued + turns, flag retired) and the shared contract `queued_posts_do_not_starve_requests` + (path phase, open, handle request under a post before every turn; asserts + `os_waits == 0`, `discovery_polls <= 1`, and zero discovery polls for pool + requests) run natively and on WASI 0.2/0.3. +- Counters: `filesystem::wait` asserts `os_waits + discovery_polls <= 1`; + `watch_backpressure` counts a spin only for a turn with neither a blocking wait + nor a discovery poll and no output, and bounds all turns per 20 ms window. + +## 11. PR #22 Windows CI: allocation gates + +CI run 34939104575 failed only two Windows allocation gates. Probes ran on a +throwaway `diag/tl-i06-windows-alloc` branch (runs 34940257417 and 34940923718, +deleted afterwards) on the unfixed code. + +- **`steady_watch_batches_allocate_nothing` stalled in round 0 (all modes).** The + gate appended through one long-lived handle. NTFS does not report that to + `ReadDirectoryChangesW`: the probe showed no event for a write on an open handle, + and `Change:subject` after `FlushFileBuffers`, after reopen+write+close and after + reopen+write+flush+close (identical in all three Windows modes). The watch + backend was correct; the workload never produced a notification. The gate now + appends, flushes and closes a fresh handle each round through `CreateFileW` with + a UTF-16 path prepared before measurement (std converts every Windows path on + the heap, which the gate would otherwise count). +- **`steady_typed_file_requests_allocate_nothing`: 2 allocations on other threads + (default mode only, intermittent).** The first submission started the process + pool, whose threads start asynchronously; on Windows std allocates inside the new + thread before user code (the thread name as UTF-16). The probes measured that a + named `turnloop-blocking-0` thread allocates exactly once on itself, `size 40 + align 2`, before its closure runs. With CPU hogs delaying worker startup, the gate + failed in all three modes, and every counted allocation was `size 40 align 2` on + a pool worker thread (e.g. `[thread 6584, 7004, 7012]` with workers + `[3060, 7004, 7012, 6584]`). Fix: each worker reports the end of thread startup + and the first pool submission returns only after every worker has started. The + gate is unchanged. +- **Found by the same probes:** `fifo_cancel_close` failed once on Windows (executor + mode): the head write was already running on a pool thread when close cancelled + it, so it wrote and reported `Cancelled` — correct per D8. The contract now + accepts complete or absent content for a cancelled write, never partial content. +- The Windows allocation gates now report the thread id, size and alignment of the + first counted allocations on any thread when they fail. + +Verification before pushing: `cargo fmt --all --check`; strict clippy native and +`--target x86_64-pc-windows-msvc` / `x86_64-unknown-linux-gnu` for `-p turnloop +-p turnloop-contract -p turnloop-io --all-targets`; `cargo test -p turnloop -p +turnloop-contract -- --test-threads=1` on macOS: PASS. CI run 34940860132 on +`18445f5`: success, every job, including all three Windows modes where both gates, +`fifo_cancel_close` and every watch contract passed (run 34940742526 on +`cc9fc43` was superseded by that push and cancelled by the PR concurrency group). diff --git a/scripts/ci/wasmtime-runner.sh b/scripts/ci/wasmtime-runner.sh index 59cc767..ba601e5 100755 --- a/scripts/ci/wasmtime-runner.sh +++ b/scripts/ci/wasmtime-runner.sh @@ -9,4 +9,8 @@ done < <(compgen -e) # Fixture certificates/dumps use absolute paths supplied by test-servers.py. root=$(cd -- "$script_dir/../.." && pwd) if [[ -d "$root/.tools" ]]; then fixture_args+=(--dir "$root/.tools"); fi -exec "$root/.tools/bin/wasmtime" run -S inherit-network=y -S allow-ip-name-lookup=y -W timeout=120s "${fixture_args[@]}" "$@" +# Filesystem contracts get a fresh, empty preopen; it is the only path authority. +fs_root=$(mktemp -d "${TMPDIR:-/tmp}/turnloop-wasi-fs.XXXXXX") +trap 'rm -rf -- "$fs_root"' EXIT +fixture_args+=(--dir "$fs_root::/turnloop-fs") +"$root/.tools/bin/wasmtime" run -S inherit-network=y -S allow-ip-name-lookup=y -W timeout=120s "${fixture_args[@]}" "$@"