Skip to content

fix: avoid repeated dependency walks and snapshot construction - #247

Open
Yakwilik wants to merge 3 commits into
mainfrom
fix/dependency-snapshot-performance
Open

Yakwilik wants to merge 3 commits into
mainfrom
fix/dependency-snapshot-performance

Conversation

@Yakwilik

@Yakwilik Yakwilik commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Problem

Large dependencies make generation spend most of its time traversing installed files and repeating metadata/symlink checks for every target. Cold dependency resolution also reads wide destination directories repeatedly and discards a completed snapshot before installation builds it again.

This fixes #245 and the reported slow easyp get googleapis snapshot phase.

Changes

  • Reuse a verified installed snapshot and its regular-file inventory within one command, keyed by source, commit, content hash and root selection. Concurrent callers share verification and can cancel while waiting.
  • Use the inventory for dependency traversal, nested module boundaries and global import-collision checks. Build module ownership once per target and share it with managed options and descriptor attribution.
  • Feed indexed inputs directly into generation when the selected target is an installed dependency, including path/package-filtered generation. Preserve the existing directory traversal order.
  • Reuse resolved parent metadata only in explicitly immutable Git views. Mutable local sources and replacements continue to resolve filesystem state on each call.
  • Cache destination-directory listings during private snapshot construction, skip unnecessary alias traversal and the redundant final filtering walk, and atomically retain completed fetched snapshots. Installation reuses the hashed files; HEAD/tag object retention uses the existing local checkout.
  • Keep h1, the lock format and snapshot contents unchanged. A fresh cache handle verifies installed contents again; explicit VerifyCached always bypasses the in-memory result. Tidy independently rechecks contents before rewriting imports, including repository wrappers without the optional verifier.
  • Fix shared Git-object reads for absolute Windows paths and relative alternates, including offline snapshot reconstruction after deletion.
  • Add regression coverage and update the source/cache contract in .spec.

A newly published snapshot does not receive a persistent verification stamp from a hash computed before publication. Only the verifier with matching before/after fingerprints writes that stamp.

Measurements

Linux, Go 1.26.6, googleapis commit 886464f687986b8b38392d83957a5a2ead4dcf71 with 7,450 proto files. The Git source is local, excluding network time. Baseline is 62fe940. These are indicative single-run CLI measurements, not Windows timing claims.

Scenario Before After
Cold get, separate empty EasyP caches 31.60 s 16.67 s
Warm generate --all, 12 projects sharing googleapis 116.50 s 1.86 s

Each generation project imports google/type/date.proto, enables managed mode and exports descriptors with imports. All 12 output filenames and descriptor bytes are identical. Cold-get lockfiles are byte-identical.

Validation

  • GitHub Actions on the previous commit: Linux CI passed lint, tests, ARM cross-compilation, native example checks, developer task checks and release configuration validation. The temporary Windows regression job also passed and has since been removed from the PR.
  • Race-enabled snapshot/cache, source-index, immutable-walker, core and generation regressions pass.
  • All packages pass with -race -count=1. The initial full run's three-minute timeout was too short for builtin WASM compilation in internal/api; rerunning that package with -timeout=15m passed (201.6 s, 321 passing test/subtest actions). Final source-index/core/generation regressions were rerun after the ordering adjustment.
  • golangci-lint 2.14.0 run ./...: 0 issues.
  • git diff --check: clean.
  • The immutable-walker regression checks operation counts at depths 1, 8 and 24. The core inventory regression works without a physical source directory, proving that indexed generation does not fall back to filesystem traversal.
  • Full snapshot/cache package race tests and lint were rerun successfully after the Windows alternate-path correction.

Closes #245.

@Yakwilik
Yakwilik marked this pull request as ready for review October 26, 2026 01:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

easyp generate is very slow on Windows - dependency roots are re-walked for every target

1 participant