diff --git a/PLAN.md b/PLAN.md index ed81642..3d794e2 100644 --- a/PLAN.md +++ b/PLAN.md @@ -180,7 +180,16 @@ Plan : ajouter Dolphin (KDE) et COSMIC Files à Nautilus, sans changer le daemon - [x] Plugins Dolphin C++ / KF6 (`integrations/dolphin/`) : `pdriveoverlay` (`KOverlayIconPlugin`, emblèmes, cache borné, appels D-Bus asynchrones) et `pdriveactions` (`KAbstractFileItemActionPlugin`, menus limités aux fichiers du drive). Textes du domaine gettext `pdrive` par `i18nd` ; `scripts/update-po.sh` extrait `i18nd`. - [x] Paquets : spec RPM (`__requires_exclude_from`), `debian/` (`dh_shlibdeps -X`, plus de `libnautilus-extension4` en `Depends`), PKGBUILD (`optdepends` `nautilus` et `dolphin`), `scripts/install-system.sh` (construit ce qui est détecté) et `uninstall.sh`, scripts de conteneur, job CI `dolphin-plugin` (Debian 13), README, `packaging/README.md`, pages de manuel. - Limites : pas d'emblèmes dans COSMIC Files (aucune API d'extension) et ses deux actions apparaissent sur tous les fichiers ; Nemo, Thunar et Caja non traités ; les paquets ne nettoient pas les fichiers de configuration de l'utilisateur à la désinstallation. -- [ ] À tester à la main : COSMIC (favori, actions, notification hors drive, interrupteur), session KDE réelle (emblèmes, menus, entrée dans Emplacements, pas de second volume dans « Périphériques », icônes `emblem-pdrive-*` avec le thème Breeze), changement d'environnement sans réinstaller, installation du paquet sur une machine GNOME seule (ni KF6 ni Qt6 tirés). +- [x] Vérifié le 13/11/2026 (RPM de test, Fedora 45, session COSMIC) : COSMIC Files (favori unique, actions « PDrive : … », notification hors drive, interrupteur qui retire puis remet les entrées) ; Dolphin 26.08 hors Plasma (emblèmes, actions limitées au drive, entrée « Proton Drive » dans Emplacements, pas de second volume dans « Périphériques ») ; paquets RPM (rpmlint 0 erreur), DEB (lintian sans remarque hors `-dbgsym`, `Depends` sans KF6, Qt6 ni `libnautilus-extension`) et Arch (namcap 0 erreur) ; `install-system.sh` puis `uninstall.sh` dans un conteneur Fedora 45. Le plugin d'emblèmes était d'abord ignoré par Dolphin, qui exige `Q_PLUGIN_METADATA` et non une `KPluginFactory`. +- [ ] Reste à tester : emblèmes avec le thème Breeze dans une vraie session Plasma, passage d'une session COSMIC à une session KDE sans réinstaller, `uninstall.sh` après `install.sh` (retrait des entrées des gestionnaires) et avec des plugins appartenant à un paquet, AppImage. + +### Performances avec COSMIC Files (13/11/2026) + +- [x] Type de montage `fuse.s3fs` (source toujours `pdrive`) : COSMIC Files ne traite comme distants que certains types de montage et lisait donc chaque fichier (détection du type, téléchargement complet) et chaque sous-dossier (comptage des éléments) en séquence. Nautilus, Dolphin, GIO, KIO et Solid ne traitent pas ce type autrement. À retirer si COSMIC Files détecte un jour ces montages autrement. +- [x] Tête de fichier gardée en cache portée de 16 à 32 Kio (`xdg-mime` lit jusqu'à 18 730 octets, GIO 16 Kio) ; les anciennes têtes de 16 Kio restent servies pour ce qu'elles couvrent, une lecture plus longue lance le téléchargement comme avant. +- [x] Un lecteur qui ferme le fichier sans avoir lu au-delà de la tête n'attend plus les 20 s : le téléchargement est annulé dès que la tête est écrite (le premier bloc chiffré, 4 Mio au plus, reste téléchargé). Les lectures réelles gardent le délai de 20 s. +- [x] Lecture anticipée du noyau limitée à la longueur de la tête (`max_readahead`) : avec la fenêtre par défaut, une lecture de 18 730 octets devenait une requête de 64 Kio, au-delà de la tête, donc un téléchargement complet. Raisonnement d'après le code du noyau, non mesuré sur la machine. +- [x] Préchargement des sous-dossiers : quand un dossier est listé, ses sous-dossiers non encore chargés sont listés en arrière-plan (un seul niveau, 4 requêtes simultanées au plus). Un seul listage par dossier, déclenché par l'utilisateur, jamais répété (la suite vient des événements Drive) ; une demande pendant un listage en cours l'attend. Les miniatures des dossiers préchargés ne sont récupérées qu'à l'ouverture du dossier, pour n'ajouter ni requête ni écriture dans le cache de miniatures. ## 3. Autres pistes (backlog) diff --git a/README.md b/README.md index 4796ad1..3df7c22 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,9 @@ flowchart LR - **`pdrived`** mounts the drive with FUSE in a hidden directory (`~/.local/share/pdrive/.mount`, ignored by GLib and gvfs so that the sidebar shows a single entry), and `~/ProtonDrive` is a symbolic link to it. It owns the content cache, the upload - journal, the thumbnails and the D-Bus service. + journal, the thumbnails and the D-Bus service. The file system type is `fuse.s3fs` (the source + stays `pdrive`): COSMIC Files only treats a fixed list of mount types as remote, and otherwise + reads every file to guess its type, which downloads it. - **D-Bus** (session bus, name `io.github.bpresles.PDrive`): interface `io.github.bpresles.PDrive1` (per-file state, offline pinning, account, quota, recent activity, mount / unmount) and the CloudProviders objects declared in @@ -277,6 +279,9 @@ Without a journal, the log is `$XDG_STATE_HOME/pdrive/pdrive.log` (by default - File permissions and a modification date alone are not kept (Proton Drive does not store them). - Files "Shared with me" and Photos are not handled. - `O_NOATIME` is refused on files that are not in the cache. +- Opening a folder also lists its sub-folders in the background (one level, a few requests at a + time), and the first 32 KiB of a file read by a file manager to guess its type are kept in the + cache: the rest of a file is only downloaded if something reads further. - Deleting a file moves it directly to the Proton trash, not to the Nautilus trash. - Rendering on KDE and GNOME, and the translated Nautilus and Dolphin menus, still need more real-world testing. - COSMIC Files shows no state emblems (no extension API), and its two PDrive actions appear on every diff --git a/crates/pdrived/src/content.rs b/crates/pdrived/src/content.rs index f911522..6115dfb 100644 --- a/crates/pdrived/src/content.rs +++ b/crates/pdrived/src/content.rs @@ -6,16 +6,17 @@ //! reads are served as soon as the requested range has been written. The SDK delivers the //! content in order, so a read waits until the download has reached the end of its range. //! -//! A download nobody reads any more is cancelled after a grace period, so that a program reading -//! only the beginning of a big file does not fetch all of it. What was received is kept as the -//! "head" of the file, which serves later reads of that beginning (e.g. content type sniffing) -//! without downloading again. +//! A download nobody reads any more is cancelled, so that a program reading only the beginning +//! of a big file does not fetch all of it: at once (as soon as the head is written) if the last +//! reader never went beyond the head, after a grace period otherwise. What was received is kept +//! as the "head" of the file, which serves later reads of that beginning (e.g. content type +//! sniffing) without downloading again. use std::collections::HashMap; use std::fs::File; use std::io::Write; use std::path::PathBuf; -use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; use std::time::Duration; @@ -25,8 +26,12 @@ use crate::activity::{ActivityLog, Kind, State}; use tokio::sync::watch; const IDLE_CANCEL_DELAY: Duration = Duration::from_secs(20); -/// Smallest beginning worth keeping: GIO sniffs at most 16 KiB. -pub const HEAD_LENGTH: u64 = 16 * 1024; +/// Smallest beginning worth keeping: GIO sniffs at most 16 KiB and `xdg-mime` (COSMIC Files) +/// 18 730 bytes, the longest magic number range of shared-mime-info. It is also the largest +/// read-ahead the kernel is allowed (see `fs.rs`), so that sniffing a file never goes beyond it. +/// Heads cached with a shorter length (older versions kept 16 KiB) are still served for the reads +/// they cover; longer reads download the content, which then replaces them. +pub const HEAD_LENGTH: u64 = 32 * 1024; #[derive(Clone, Copy, PartialEq)] pub enum Progress { @@ -39,6 +44,8 @@ pub enum Progress { pub struct Transfer { progress: watch::Receiver, readers: AtomicUsize, + /// End of the furthest range any reader asked for. + read_end: AtomicU64, cancelled: AtomicBool, cancellation: Option, runtime: tokio::runtime::Handle, @@ -46,12 +53,28 @@ pub struct Transfer { impl Transfer { /// Waits until `end` bytes are available (or the download ended). Returns false on failure. + /// Records the range as read. pub async fn wait_for(&self, end: u64) -> bool { + self.read_end.fetch_max(end, Ordering::SeqCst); + self.wait_written(end).await + } + + async fn wait_written(&self, end: u64) -> bool { let mut progress = self.progress.clone(); let result = progress.wait_for(|p| !matches!(p, Progress::Running(written) if *written < end)).await; matches!(result.as_deref(), Ok(Progress::Running(_) | Progress::Done)) } + /// Whether nobody reads this download and nobody read further than the head: the rest is + /// not wanted (type sniffing, thumbnails of what is already known…). + fn only_head_wanted(&self) -> bool { + self.readers.load(Ordering::SeqCst) == 0 && self.read_end.load(Ordering::SeqCst) <= HEAD_LENGTH + } + + fn running(&self) -> bool { + matches!(*self.progress.borrow(), Progress::Running(_)) + } + /// Cancels the download (once). Callable from any thread. fn cancel(self: &Arc) { if self.cancelled.swap(true, Ordering::SeqCst) { @@ -199,6 +222,7 @@ impl ContentCache { let transfer = Arc::new(Transfer { progress: receiver, readers: AtomicUsize::new(1), + read_end: AtomicU64::new(0), cancelled: AtomicBool::new(false), cancellation: Some(cancellation), runtime: self.runtime.clone(), @@ -302,13 +326,27 @@ impl ContentCache { if transfer.readers.fetch_sub(1, Ordering::SeqCst) != 1 { return; } - self.runtime.spawn(async move { - tokio::time::sleep(IDLE_CANCEL_DELAY).await; - let running = matches!(*transfer.progress.borrow(), Progress::Running(_)); - if running && transfer.readers.load(Ordering::SeqCst) == 0 { - transfer.cancel(); - } - }); + self.runtime.spawn(cancel_when_unwanted(transfer, IDLE_CANCEL_DELAY)); + } +} + +/// Cancels a download nobody reads any more: once its head is written if no reader went beyond +/// the head, otherwise after `idle_delay` (a reader may come back, e.g. a copy resuming). +async fn cancel_when_unwanted(transfer: Arc, idle_delay: Duration) { + if transfer.only_head_wanted() { + transfer.wait_written(HEAD_LENGTH).await; + if transfer.running() && transfer.only_head_wanted() { + transfer.cancel(); + return; + } + // Finished meanwhile, or someone else reads it now (its own release decides). + if !transfer.running() || transfer.readers.load(Ordering::SeqCst) > 0 { + return; + } + } + tokio::time::sleep(idle_delay).await; + if transfer.running() && transfer.readers.load(Ordering::SeqCst) == 0 { + transfer.cancel(); } } @@ -495,12 +533,108 @@ mod tests { Transfer { progress: receiver, readers: AtomicUsize::new(1), + read_end: AtomicU64::new(0), cancelled: AtomicBool::new(false), cancellation: None, runtime: tokio::runtime::Handle::current(), } } + #[test] + fn the_head_covers_what_sniffers_read() { + // xdg-mime reads up to 18 730 bytes, GIO 16 KiB. + const { assert!(HEAD_LENGTH >= 18_730 && HEAD_LENGTH >= 16 * 1024) }; + assert_eq!(HEAD_LENGTH, 32 * 1024); + } + + #[test] + fn keep_head_keeps_what_is_longer_than_the_head() { + let dir = tempfile::tempdir().unwrap(); + let (partial, head) = (dir.path().join("x.part"), dir.path().join("x.head")); + std::fs::write(&partial, vec![1u8; 4 * 1024 * 1024]).unwrap(); + keep_head(&partial, &head, 4 * 1024 * 1024); + assert_eq!(std::fs::metadata(&head).unwrap().len(), 4 * 1024 * 1024); + // An old 16 KiB head is now too short to be kept from a new download. + std::fs::write(&partial, vec![1u8; 16 * 1024]).unwrap(); + std::fs::remove_file(&head).unwrap(); + keep_head(&partial, &head, 16 * 1024); + assert!(!head.exists()); + } + + #[tokio::test] + async fn a_reader_stopping_inside_the_head_cancels_once_the_head_is_written() { + let (sender, receiver) = watch::channel(Progress::Running(0)); + let transfer = Arc::new(transfer(receiver)); + // Reads what a sniffer reads, then closes. + assert!(!transfer.only_head_wanted()); + transfer.readers.fetch_sub(1, Ordering::SeqCst); + assert!(transfer.only_head_wanted()); + let task = tokio::spawn(cancel_when_unwanted(transfer.clone(), Duration::from_secs(3600))); + tokio::task::yield_now().await; + sender.send_replace(Progress::Running(HEAD_LENGTH - 1)); + tokio::task::yield_now().await; + assert!(!task.is_finished() && !transfer.cancelled.load(Ordering::SeqCst)); + sender.send_replace(Progress::Running(HEAD_LENGTH)); + task.await.unwrap(); + assert!(transfer.cancelled.load(Ordering::SeqCst)); + } + + #[tokio::test] + async fn a_reader_going_beyond_the_head_keeps_the_grace_period() { + let (sender, receiver) = watch::channel(Progress::Running(0)); + let transfer = Arc::new(transfer(receiver)); + sender.send_replace(Progress::Running(HEAD_LENGTH + 1)); + assert!(transfer.wait_for(HEAD_LENGTH + 1).await); + transfer.readers.fetch_sub(1, Ordering::SeqCst); + assert!(!transfer.only_head_wanted()); + cancel_when_unwanted(transfer.clone(), Duration::from_millis(50)).await; + assert!(transfer.cancelled.load(Ordering::SeqCst)); + // Within the grace period, a new reader prevents the cancellation. + let (_sender, receiver) = watch::channel(Progress::Running(10)); + let transfer = Arc::new(self::transfer(receiver)); + transfer.read_end.store(HEAD_LENGTH + 1, Ordering::SeqCst); + transfer.readers.store(0, Ordering::SeqCst); + let task = tokio::spawn(cancel_when_unwanted(transfer.clone(), Duration::from_millis(100))); + transfer.readers.fetch_add(1, Ordering::SeqCst); + task.await.unwrap(); + assert!(!transfer.cancelled.load(Ordering::SeqCst)); + } + + #[tokio::test] + async fn a_second_reader_prevents_the_early_cancellation() { + let (sender, receiver) = watch::channel(Progress::Running(0)); + let transfer = Arc::new(transfer(receiver)); + transfer.readers.store(0, Ordering::SeqCst); + let task = tokio::spawn(cancel_when_unwanted(transfer.clone(), Duration::from_secs(3600))); + tokio::task::yield_now().await; + // Another handle joins the download while the head is being written. + transfer.readers.fetch_add(1, Ordering::SeqCst); + sender.send_replace(Progress::Running(HEAD_LENGTH)); + task.await.unwrap(); + assert!(!transfer.cancelled.load(Ordering::SeqCst)); + } + + #[tokio::test] + async fn a_finished_download_is_not_cancelled() { + let (sender, receiver) = watch::channel(Progress::Running(0)); + let transfer = Arc::new(transfer(receiver)); + transfer.readers.store(0, Ordering::SeqCst); + let task = tokio::spawn(cancel_when_unwanted(transfer.clone(), Duration::from_secs(3600))); + tokio::task::yield_now().await; + sender.send_replace(Progress::Done); + task.await.unwrap(); + assert!(!transfer.cancelled.load(Ordering::SeqCst)); + } + + #[tokio::test] + async fn wait_for_records_the_furthest_range() { + let (_sender, receiver) = watch::channel(Progress::Done); + let transfer = transfer(receiver); + transfer.wait_for(100).await; + transfer.wait_for(50).await; + assert_eq!(transfer.read_end.load(Ordering::SeqCst), 100); + } + #[tokio::test] async fn wait_for_returns_when_the_range_is_written() { let (sender, receiver) = watch::channel(Progress::Running(0)); diff --git a/crates/pdrived/src/drive.rs b/crates/pdrived/src/drive.rs index 1141a07..3e076d0 100644 --- a/crates/pdrived/src/drive.rs +++ b/crates/pdrived/src/drive.rs @@ -2,6 +2,7 @@ // Copyright (C) 2026 Bertrand Presles //! State shared by the FUSE callbacks and the event watcher. +use std::collections::{HashMap, HashSet}; use std::path::PathBuf; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex, OnceLock}; @@ -14,6 +15,7 @@ use pdsdk::proto::{ThumbnailType, file_thumbnail}; use crate::activity::{ActivityLog, Kind}; use crate::content::{ContentCache, Opened}; +use crate::listing::InFlight; use crate::pins::Pins; use crate::replay::{self, Replay}; use crate::thumbnails::ThumbnailCache; @@ -23,6 +25,9 @@ use crate::writes::Staging; /// Fetching the thumbnails of a folder listing gives up after this delay. const THUMBNAILS_TIMEOUT: Duration = Duration::from_secs(15); +/// Folder listings made ahead of time at once. +const PREFETCH_PARALLELISM: usize = 4; + /// A remote change waiting to be applied by the FUSE callback that the watcher triggers /// through the mount (so that the kernel emits inotify events for it). pub enum Change { @@ -79,6 +84,13 @@ pub struct Drive { /// Serializes replays through the mount (see `replay.rs`). pub replay_lock: tokio::sync::Mutex<()>, deferred: Mutex>, + /// Folder listings in flight. + listings: InFlight, + /// Sub-folders queued or being listed ahead of time. + prefetching: Mutex>, + prefetch_slots: tokio::sync::Semaphore, + /// Files with a Proton thumbnail in folders listed ahead of time, by folder. + deferred_thumbnails: Mutex>>, } impl Drive { @@ -117,6 +129,10 @@ impl Drive { pending: Mutex::new(None), replay_lock: tokio::sync::Mutex::new(()), deferred: Mutex::new(Vec::new()), + listings: InFlight::default(), + prefetching: Mutex::new(HashSet::new()), + prefetch_slots: tokio::sync::Semaphore::new(PREFETCH_PARALLELISM), + deferred_thumbnails: Mutex::new(HashMap::new()), } } @@ -189,8 +205,55 @@ impl Drive { Ok(nodes.into_iter().filter_map(NodeInfo::from_node).collect()) } - /// Makes sure the listing of a folder is loaded. + /// Makes sure the listing of a folder is loaded, and lists its sub-folders in the background + /// (see [`Drive::prefetch_subfolders`]). pub async fn load_children(self: &Arc, ino: u64) -> Result<(), Errno> { + self.load_folder(ino, true).await?; + // Thumbnails of a folder listed ahead of time are fetched when the user gets there. + let deferred = self.deferred_thumbnails.lock().unwrap_or_else(|e| e.into_inner()).remove(&ino); + if let Some(nodes) = deferred { + let drive = self.clone(); + tokio::spawn(async move { drive.ensure_thumbnails(nodes).await }); + } + self.prefetch_subfolders(ino); + Ok(()) + } + + /// Lists, in the background, the sub-folders of a folder that the user is looking at, so + /// that opening one of them (or a file manager counting its items) does not wait for the + /// network. One level only, at most [`PREFETCH_PARALLELISM`] requests at a time. + /// + /// This is not polling: each folder is listed at most once, because the user listed its + /// parent, and never listed again by this code; later changes come from the Drive events + /// (`watch.rs`). A folder whose listing fails is only retried when the user looks at its + /// parent again. Thumbnails are not fetched for these listings (see `load_folder`). + fn prefetch_subfolders(self: &Arc, ino: u64) { + let subfolders = self.tree().unloaded_subfolders(ino); + for subfolder in subfolders { + // Already queued by a previous listing of the parent. + if !self.prefetching.lock().unwrap_or_else(|e| e.into_inner()).insert(subfolder) { + continue; + } + let drive = self.clone(); + tokio::spawn(async move { + if let Ok(_slot) = drive.prefetch_slots.acquire().await { + // An error is logged by `load_folder`; the user's own request will retry. + let _ = drive.load_folder(subfolder, false).await; + } + drive.prefetching.lock().unwrap_or_else(|e| e.into_inner()).remove(&subfolder); + }); + } + } + + /// Loads the listing of a folder unless it is loaded. Requests for a folder already being + /// listed wait for that listing. Thumbnails of the files are only fetched for a listing the + /// user asked for: for the others (prefetch) they are kept for later, because fetching them + /// costs requests and writes into the thumbnail cache for folders nobody may open. + async fn load_folder(self: &Arc, ino: u64, wanted_by_user: bool) -> Result<(), Errno> { + self.listings.run(ino, self.fetch_folder(ino, wanted_by_user)).await + } + + async fn fetch_folder(self: &Arc, ino: u64, wanted_by_user: bool) -> Result<(), Errno> { let uid = { let tree = self.tree(); let entry = tree.get(ino).ok_or(Errno::ENOENT)?; @@ -207,15 +270,22 @@ impl Drive { let with_thumbnails: Vec = children.iter().filter(|n| matches!(n.kind, NodeKind::File { has_thumbnail: true, .. })).cloned().collect(); { let mut tree = self.tree(); - // Another request may have loaded it meanwhile, or the folder may be gone. + // The folder may be gone, or loaded meanwhile by a change event. if tree.get(ino).is_none() || tree.is_loaded(ino) { return Ok(()); } tree.set_children(ino, children); } - // In the background: the listing must not wait for thumbnails. - let drive = self.clone(); - tokio::spawn(async move { drive.ensure_thumbnails(with_thumbnails).await }); + if with_thumbnails.is_empty() { + return Ok(()); + } + if wanted_by_user { + // In the background: the listing must not wait for thumbnails. + let drive = self.clone(); + tokio::spawn(async move { drive.ensure_thumbnails(with_thumbnails).await }); + } else { + self.deferred_thumbnails.lock().unwrap_or_else(|e| e.into_inner()).insert(ino, with_thumbnails); + } Ok(()) } diff --git a/crates/pdrived/src/fs.rs b/crates/pdrived/src/fs.rs index cd2b338..bf8cd71 100644 --- a/crates/pdrived/src/fs.rs +++ b/crates/pdrived/src/fs.rs @@ -18,8 +18,8 @@ use std::sync::{Arc, Mutex}; use std::time::{Duration, SystemTime}; use fuser::{ - BsdFileFlags, Errno, FileAttr, FileHandle, FileType, Filesystem, FopenFlags, Generation, INodeNo, LockOwner, OpenAccMode, OpenFlags, RenameFlags, - ReplyAttr, ReplyCreate, ReplyData, ReplyDirectory, ReplyEmpty, ReplyEntry, ReplyOpen, ReplyWrite, Request, TimeOrNow, WriteFlags, + BsdFileFlags, Errno, FileAttr, FileHandle, FileType, Filesystem, FopenFlags, Generation, INodeNo, KernelConfig, LockOwner, OpenAccMode, OpenFlags, + RenameFlags, ReplyAttr, ReplyCreate, ReplyData, ReplyDirectory, ReplyEmpty, ReplyEntry, ReplyOpen, ReplyWrite, Request, TimeOrNow, WriteFlags, }; use crate::content::{Opened, Transfer}; @@ -29,6 +29,8 @@ use crate::tree::{Entry, EntryKind}; /// Attributes may be cached briefly; remote changes are pushed with explicit invalidations. const TTL: Duration = Duration::from_secs(1); const BLOCK_SIZE: u32 = 4096; +/// Largest read-ahead requested from the kernel (see `Filesystem::init`). +const MAX_READAHEAD: u32 = crate::content::HEAD_LENGTH as u32; #[derive(Clone, Copy)] struct Owner { @@ -189,6 +191,15 @@ fn caller_name(req: &Request) -> Option { } impl Filesystem for DriveFs { + fn init(&mut self, _req: &Request, config: &mut KernelConfig) -> std::io::Result<()> { + // The kernel reads ahead of the application: a read of 18 730 bytes (type sniffing) at + // the start of a file becomes a 64 KiB request with the default window, which goes + // beyond the cached head and would download the whole content. Never ask for more than + // the head. Too large a value is refused with the nearest valid one: ignore it. + let _ = config.set_max_readahead(MAX_READAHEAD); + Ok(()) + } + fn lookup(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEntry) { let Some(name) = name.to_str().map(str::to_owned) else { return reply.error(Errno::ENOENT); diff --git a/crates/pdrived/src/listing.rs b/crates/pdrived/src/listing.rs new file mode 100644 index 0000000..a32c002 --- /dev/null +++ b/crates/pdrived/src/listing.rs @@ -0,0 +1,106 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Bertrand Presles +//! Folder listings in flight: one request per folder, however many callers want it. + +use std::collections::HashMap; +use std::future::Future; +use std::sync::{Arc, Mutex}; + +/// Runs one operation at a time per key. A caller arriving while another one runs the same key +/// waits for it, then runs its own operation, which is expected to notice that the work is done +/// (e.g. the folder is loaded) and return at once. +#[derive(Default)] +pub struct InFlight { + locks: Mutex>>>, +} + +impl InFlight { + pub async fn run(&self, key: u64, operation: impl Future) -> T { + let lock = self.locks().entry(key).or_default().clone(); + let result = { + let _turn = lock.lock().await; + operation.await + }; + // Forget the key unless someone else holds or waits for it (the map and `lock` own it). + let mut locks = self.locks(); + if Arc::strong_count(&lock) == 2 { + locks.remove(&key); + } + result + } + + fn locks(&self) -> std::sync::MutexGuard<'_, HashMap>>> { + self.locks.lock().unwrap_or_else(|e| e.into_inner()) + } + + #[cfg(test)] + fn len(&self) -> usize { + self.locks().len() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::{AtomicUsize, Ordering}; + + /// What `Drive::load_folder` does: check, list once, mark as loaded. + async fn load(flights: Arc, loaded: Arc>, requests: Arc, key: u64) { + let work = async { + if *loaded.lock().unwrap() { + return; + } + requests.fetch_add(1, Ordering::SeqCst); + tokio::time::sleep(std::time::Duration::from_millis(30)).await; + *loaded.lock().unwrap() = true; + }; + flights.run(key, work).await; + } + + #[tokio::test] + async fn concurrent_callers_share_one_request() { + let flights = Arc::new(InFlight::default()); + let (loaded, requests) = (Arc::new(Mutex::new(false)), Arc::new(AtomicUsize::new(0))); + let tasks: Vec<_> = (0..5).map(|_| tokio::spawn(load(flights.clone(), loaded.clone(), requests.clone(), 1))).collect(); + for task in tasks { + task.await.unwrap(); + } + assert_eq!(requests.load(Ordering::SeqCst), 1); + assert!(*loaded.lock().unwrap()); + assert_eq!(flights.len(), 0); + } + + #[tokio::test] + async fn a_waiter_follows_a_failed_request_with_its_own() { + let flights = Arc::new(InFlight::default()); + let attempts = Arc::new(AtomicUsize::new(0)); + let attempt = |flights: Arc, attempts: Arc| async move { + flights + .run(1, async { + let n = attempts.fetch_add(1, Ordering::SeqCst); + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + if n == 0 { Err(()) } else { Ok(()) } + }) + .await + }; + let first = tokio::spawn(attempt(flights.clone(), attempts.clone())); + tokio::task::yield_now().await; + let second = tokio::spawn(attempt(flights.clone(), attempts.clone())); + assert!(first.await.unwrap().is_err()); + assert!(second.await.unwrap().is_ok()); + } + + #[tokio::test] + async fn different_folders_do_not_wait_for_each_other() { + let flights = Arc::new(InFlight::default()); + let requests = Arc::new(AtomicUsize::new(0)); + let started = std::time::Instant::now(); + let tasks: Vec<_> = (0..4).map(|key| tokio::spawn(load(flights.clone(), Arc::default(), requests.clone(), key))).collect(); + for task in tasks { + task.await.unwrap(); + } + assert_eq!(requests.load(Ordering::SeqCst), 4); + assert!(started.elapsed() < std::time::Duration::from_millis(110)); + assert_eq!(flights.len(), 0); + } +} diff --git a/crates/pdrived/src/main.rs b/crates/pdrived/src/main.rs index 010e3f6..226515c 100644 --- a/crates/pdrived/src/main.rs +++ b/crates/pdrived/src/main.rs @@ -21,6 +21,7 @@ mod content; mod dbus; mod drive; mod fs; +mod listing; mod mountpoint; mod pins; mod replay; @@ -185,8 +186,14 @@ fn mount(drive: &Arc) -> Result> { let target = &drive.mount_target; mountpoint::prepare_dir(target)?; let mut config = Config::default(); + // The file system type is "fuse.s3fs" (source still "pdrive"): COSMIC Files only treats a + // fixed list of mount types as remote, and without that it reads the content of every file + // (type sniffing) and lists every sub-folder (item counts) one after the other, which + // downloads files. "fuse.s3fs" is on that list and has no special handling elsewhere: GLib + // (is_remote_fs_type), KIO (any "fuse.*" not on /dev/ is already remote), Solid and libmount + // do not know it. Remove this if COSMIC Files detects these mounts differently one day. config.mount_options = - vec![MountOption::FSName("pdrive".into()), MountOption::Subtype("pdrive".into()), MountOption::NoAtime, MountOption::DefaultPermissions]; + vec![MountOption::FSName("pdrive".into()), MountOption::Subtype("s3fs".into()), MountOption::NoAtime, MountOption::DefaultPermissions]; let session = fuser::Session::new(DriveFs::new(drive.clone(), tokio::runtime::Handle::current()), target, &config) .with_context(|| format!("cannot mount on {}", target.display()))?; drive.set_session(Some(session.notifier())); diff --git a/crates/pdrived/src/tree.rs b/crates/pdrived/src/tree.rs index 6288fff..a59c297 100644 --- a/crates/pdrived/src/tree.rs +++ b/crates/pdrived/src/tree.rs @@ -140,6 +140,12 @@ impl Tree { } } + /// Sub-folders of a loaded folder whose own listing is not loaded yet. + pub fn unloaded_subfolders(&self, ino: u64) -> Vec { + let children = self.children(ino).unwrap_or_default(); + children.iter().copied().filter(|c| matches!(self.entries[c].kind, EntryKind::Folder { children: None })).collect() + } + pub fn is_loaded(&self, ino: u64) -> bool { self.children(ino).is_some() } @@ -400,6 +406,19 @@ mod tests { } } + #[test] + fn unloaded_subfolders_lists_the_folders_without_listing() { + let mut tree = tree_with(vec![folder("a", "a"), folder("b", "b"), file("f", "f", "r")]); + let a = tree.ino_of("a").unwrap(); + let b = tree.ino_of("b").unwrap(); + assert_eq!(tree.unloaded_subfolders(ROOT_INO), vec![a, b]); + tree.set_children(a, vec![]); + assert_eq!(tree.unloaded_subfolders(ROOT_INO), vec![b]); + // Not loaded itself, or not a folder: nothing. + assert!(tree.unloaded_subfolders(b).is_empty()); + assert!(tree.unloaded_subfolders(tree.ino_of("f").unwrap()).is_empty()); + } + fn tree_with(nodes: Vec) -> Tree { let mut tree = Tree::new("root".into()); tree.set_children(ROOT_INO, nodes);