easyp generate is very slow on Windows - dependency roots are re-walked for every target

Summary

easyp generate --all gets very slow on Windows as a projects dependency modules grow even when only a few files from them are generated

Almost all of the time goes into file-metadata syscalls (Stat, Lstat, EvalSymlinks) and not into compiling or running plugins. Linux makes the same calls, but they are much cheaper there, so the problem mostly shows up on Windows

Version: main @ 793353c / v1.0.0-nightly.20261014.1

Cause

Repeated walks - every generation target walks every dependencies full file tree up to three times via CheckSourceCollisions and two FileModules calls (internal/generation/module.go:30,45, descriptor_set.go:124).
Repeated checks per file - for every file visited, the walk repeats symlink resolution and os.Stat checks whose results do not change during a run (internal/modules/sources.go:95,100,171, internal/sourceview/view.go:289).

Possible fixes

Walk each dependency root once per target and share the result between CheckSourceCollisions and FileModules.
Move the per-file work that does not change such as resolving the walks root path, out of the per-file loop.
Cache results for installed dependency snapshots within a run. They are hash-verified and don't change during a run.

.spec/config/dependency.md says local source resolution stays uncached, so I'm not sure caching is acceptable here.

Question

Is caching installed snapshots OK? If not, how would you prefer to make this faster? I'm happy to open a PR once there is an agreed approach

Thanks for the detailed investigation! I've prepared fixes in PR #247: #247.

The changes avoid repeated dependency walks and metadata checks by reusing verified snapshot inventories within a command, while keeping mutable local sources uncached. So far, the performance measurements are from Linux, not Windows.

Could you test this PR on your Windows project and compare easyp generate --all before and after? I'd really appreciate your timings and any feedback on regressions.

Tested on Windows 11 with v1.0.0-nightly.20261014.1 vs #247 (f5f9e46), timings exclude dependency downloads

25-target project (25 deps, 2,199 files) is down from ~28.7s per target to ~3.7s per target.
Single-target project (1 dep, 481 files) is down from ~7.11s to ~3.40s

Output is identical apart from the protoc version header

Some more suggestions on how to make this even faster

install modules in parallel - on a cold cache, downloading dependencies one after another dominates total run time once a project has more than a few of them
skip retries when a repo doesn't exist - a missing repo will fail the same way every time, so retrying only adds wait time
compile builtin WASM plugins once per run - they are currently recompiled for every target. A shared wazero.NewCompilationCache() plus CompileModule and InstantiateModule would avoid that
run plugins concurrently, then write outputs in plugin order (so the output stays deterministic)
index the workspace's module declarations once per run. currently the workspace is walked again for every generate.modules entry

Really appreciate the responsiveness!

Navigation Menu

Repository navigation

easyp generate is very slow on Windows - dependency roots are re-walked for every target #245

Summary

Cause

Possible fixes

Question

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

easyp generate is very slow on Windows - dependency roots are re-walked for every target #245

Description

Summary

Cause

Possible fixes

Question

Activity

Yakwilik commented on Oct 26, 2026

DoseOfCode commented on Oct 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions