Skip to content

Faster folder loading in COSMIC Files, no full download for content sniffing - #4

Merged
bpresles merged 4 commits into
masterfrom
cosmic-files-perf
Nov 13, 2026
Merged

bpresles merged 4 commits into
masterfrom
cosmic-files-perf

Conversation

@bpresles

@bpresles bpresles commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Summary

COSMIC Files (1.11) treats fuse.pdrive as a local file system. For every entry of a folder, it lists each sub-folder to count its items and reads the beginning of each file to guess its type (xdg-mime-rs, plain open, up to 18 730 bytes). Measured on a real folder with 72 sub-folders and 24 files: 35.6 s to count the sub-folders on the first visit, and 0.8 to 1.7 s per uncached file. Uncached files were downloaded completely: the 16 KiB cached head was smaller than the read, and a download without readers ran for 20 more seconds.

  • Mount type fuse.s3fs (FSName stays pdrive): COSMIC Files only treats listed types as remote, and fuse.s3fs is one of them. The type has no special handling in GLib (is_remote_fs_type), KIO (KMountPoint::isOnRemoteStorage already treats any fuse.* not from /dev/ as remote), Solid (fuse.sshfs, fuse.rclone, fuse.ceph only) or libmount (mnt_fstype_is_netfs), so Nautilus and Dolphin are unchanged. To be removed if COSMIC Files detects such mounts another way.
  • 32 KiB head instead of 16 KiB: covers GIO (16 KiB) and xdg-mime (18 730 bytes). A 16 KiB head already in the cache still serves the reads it covers.
  • Sniffing does not download the whole file: a transfer records the end of the largest range read; when the last reader closes after reading only within the head, the download is cancelled as soon as the head is written, instead of after 20 s. Real reads (copy, opening in an application) keep the current behaviour.
  • Kernel readahead capped at 32 KiB (set_max_readahead(HEAD_LENGTH)): with the 128 KiB default, a short read becomes a ~64 KiB FUSE request that exceeds the head. Serving only the available part of a readahead request was rejected: a short reply to a readahead request leaves zero-filled pages in the page cache.
  • Sub-folder prefetch: listing a folder lists its unloaded sub-folders in the background, one level, at most 4 requests at a time. Concurrent load_children calls for the same folder wait for the request in flight instead of sending another one. Each folder is listed once, then kept up to date by Drive events (no polling). Thumbnails of prefetched folders are fetched when the folder is first opened.

Rebased on master after #3 (only PLAN.md conflicted: both sections kept).

Test plan

  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace (pdrived: 101, including the new tests for the head, early cancellation, in-flight listing deduplication and unloaded_subfolders)
  • make -C extension
  • COSMIC Files on the real drive (test RPM with Dolphin and COSMIC Files integration #3 and Faster folder loading in COSMIC Files, no full download for content sniffing #4, Fedora): folders show much faster (checked by the user)
  • Replay of the old COSMIC Files scan on the 72 sub-folder folder: counting 35.6 s → 13.2 s on the first visit (prefetch), 0 s after. Files under 4 MiB are still fully fetched when sniffed (their first encrypted block is the whole file); the gain for larger files was not measured
  • findmnt: type fuse.s3fs, source pdrive; read_ahead_kb of the mount = 32
  • Nautilus unchanged (checked by the user)
  • Dolphin: "Proton Drive" place and emblems shown, no extra network device under Devices (checked by the user, test RPM with Dolphin and COSMIC Files integration #3)
  • Dolphin: thumbnails and item counts unchanged
  • Copy of a large file: same speed as before

🤖 Generated with Claude Code

@bpresles bpresles mentioned this pull request Nov 13, 2026
15 of 19 tasks
bpresles and others added 3 commits November 13, 2026 22:06
…remote

COSMIC Files only treats a fixed list of mount types as remote and read
every file and sub-folder of a "local" folder one after the other, which
downloaded the files. fuse.s3fs is on that list and has no special
handling in GLib, KIO, Solid or libmount. The source stays "pdrive".

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
COSMIC Files guesses the type of every file by reading up to 18 730
bytes, which went past the 16 KiB head and downloaded the whole file.

- HEAD_LENGTH is 32 KiB (older 16 KiB heads still serve the reads they
  cover).
- The furthest range read is tracked per transfer: when the last reader
  closes the file without going beyond the head, the download is
  cancelled as soon as the head is written instead of after 20 s.
- The kernel read-ahead is limited to the head length, so that a small
  read is not turned into a bigger request.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Opening a folder lists its not yet loaded sub-folders once, one level
deep, four requests at a time, so that item counts and navigation do not
wait for the network. Requests for a folder being listed wait for that
listing instead of starting another. Thumbnails of folders listed ahead
of time are fetched when the folder is opened. No polling: later changes
still come from the Drive events.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bpresles
bpresles merged commit 60c2165 into master Nov 13, 2026
4 checks passed
bpresles added a commit that referenced this pull request Nov 14, 2026
…niffing (#4)

* pdrived: mount as fuse.s3fs so that COSMIC Files treats the drive as remote

COSMIC Files only treats a fixed list of mount types as remote and read
every file and sub-folder of a "local" folder one after the other, which
downloaded the files. fuse.s3fs is on that list and has no special
handling in GLib, KIO, Solid or libmount. The source stays "pdrive".

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pdrived: keep a 32 KiB head and stop downloads that only sniff a file

COSMIC Files guesses the type of every file by reading up to 18 730
bytes, which went past the 16 KiB head and downloaded the whole file.

- HEAD_LENGTH is 32 KiB (older 16 KiB heads still serve the reads they
  cover).
- The furthest range read is tracked per transfer: when the last reader
  closes the file without going beyond the head, the download is
  cancelled as soon as the head is written instead of after 20 s.
- The kernel read-ahead is limited to the head length, so that a small
  read is not turned into a bigger request.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pdrived: list the sub-folders of a listed folder in the background

Opening a folder lists its not yet loaded sub-folders once, one level
deep, four requests at a time, so that item counts and navigation do not
wait for the network. Requests for a folder being listed wait for that
listing instead of starting another. Thumbnails of folders listed ahead
of time are fetched when the folder is opened. No polling: later changes
still come from the Drive events.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* PLAN.md: what was tested of the Dolphin and COSMIC Files integration

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
@bpresles
bpresles deleted the cosmic-files-perf branch November 14, 2026 10:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant