Skip to content

Improve Perf of MemoryFileSystem.EnumerateItems - #110

Merged
xoofx merged 1 commit into
xoofx:mainfrom
lahma:perf/memfs-enumerate
Oct 19, 2026
Merged

xoofx merged 1 commit into
xoofx:mainfrom
lahma:perf/memfs-enumerate

Conversation

@lahma

@lahma lahma commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

For every child of every visited directory, MemoryFileSystem.EnumerateItemsImpl built the child path with directoryPath / nodePair.Key. That converts the name to a UPath, which validates and normalizes it, then concatenates, and validates and normalizes the whole combined path again. For the children of /, the combined //name even takes the slow normalization path and allocates a second string.

It also assigned the three DateTime times of each node to the DateTimeOffset fields of FileSystemItem. The nodes store local times, so each assignment looks up the UTC offset of the local time zone at that time: three conversions per item.

Changes

  • Child paths are combined by concatenation (CombineChildPath), which gives exactly the same path as directoryPath / name: the name of a child node is always a single part of a validated path (no separator, not . or ..), and the directory path is absolute and normalized. The concatenation is therefore already normalized and is built with the internal new UPath(string, safe: true). A relative directory path (not expected) falls back to directoryPath / name.
  • Identical times are converted only once: a directory, a file not written or read since its creation, or a file not read since its last write. The times are compared on their raw 64-bit representation, not with DateTime ==: equality only compares ticks, while the conversion also depends on the Kind and on the daylight-saving flag of an ambiguous local time. So every enumerated CreationTime, LastWriteTime and LastAccessTime is the same value as before.

The async counterpart (MemoryFileSystemAsync.gen.cs) is regenerated with Zio.AsyncCodeGen.

EnumeratePathsImpl is deliberately left as it is. I also tried computing the child path there only when the child is returned or followed: that made a selective pattern 43% faster, but consistently made an * enumeration slower in my runs, so it isn't part of this PR.

Tests

  • TestMemoryFileSystemEnumeration.TestEnumeratedPathsAreNormalized: paths and items of a tree with children of /, nested children, and names with dots (.hidden, x..y, .z.) and spaces; each path equals UPath.Combine(parent, name) and is already normalized; also search patterns with a directory part.
  • TestMemoryFileSystemEnumeration.TestEnumeratedItemTimesMatchTheFileSystemTimes: each item time EqualsExact the DateTimeOffset of the corresponding Get*Time value, for a directory (all times equal), a written file, a read file (all times different), the same ticks with different Kinds, and explicitly set times.
  • TestAsyncFileSystem.TestMemoryFileSystemAsyncEnumeratesNormalizedPathsAndTimes: the same check for MemoryFileSystemAsync.

All tests pass on net10.0 and net472.

Benchmarks

A MemoryFileSystem of 50,000 files in 1,051 directories (/test/suite-XX/group-YY/test-case-ZZZ.js):

[Benchmark] public long EnumerateItemsAllDirectories() { long total = 0; foreach (var item in _fs.EnumerateItems("/", SearchOption.AllDirectories)) total += item.Length; return total; }
Benchmark main this PR
EnumerateItems("/", AllDirectories) 13.0-13.5 ms 8.9-9.0 ms (-31% to -35%)

The ranges come from four interleaved runs of each. Allocation is unchanged (5.08 MB). BenchmarkDotNet 0.15.8 default job, .NET 10, AMD Ryzen 9 5950X, pinned to one core.

🤖 Generated with Claude Code

EnumerateItemsImpl combined each child with `directoryPath / name`, which
validates and normalizes the name and then the whole combined path again, and
converted the creation, last write and last access times of every child to
DateTimeOffset, each conversion looking up the UTC offset of the local time
zone.

- Combine a child path by concatenation: the name of a child node is always a
  single part of a validated path and the directory path is absolute and
  normalized, so the result is already normalized and is the same path.
- Convert identical times only once. The times are compared on their full
  binary representation, as the conversion also depends on the Kind (and on
  the daylight saving flag of an ambiguous local time) which DateTime
  equality ignores, so the enumerated values are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@xoofx
xoofx merged commit b6fe6e5 into xoofx:main Oct 19, 2026
0 of 2 checks passed
@lahma
lahma deleted the perf/memfs-enumerate branch October 19, 2026 20:12
lahma added a commit to lahma/test262-harness-dotnet that referenced this pull request Oct 20, 2026
0.25.0 contains only our three upstream changes to MemoryFileSystem:
a lock-free fast path for node locks (xoofx/zio#109), faster
EnumerateItems (xoofx/zio#110) and the fix for a deadlock between a
rename inside a sub-directory and concurrent lookups (xoofx/zio#111).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants