Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .spec/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,13 @@ mod vendor

<code>EnsureLockedSources</code> may install dependencies. <code>ReadManifest</code>, <code>ReadLock</code>, <code>LocalSources</code>, and <code>CachedSources</code> do not download or rewrite project files. <code>CachedSources</code> assumes installation/hash verification has already occurred.

Verified installed modules can supply an immutable regular-file inventory through the optional <code>SourceFiles</code> cache capability. <code>SourceRoots</code> uses it for dependency discovery and nested boundaries without repeated filesystem traversal. Local roots and replacements continue to resolve current filesystem state. Generation computes collision checks and file ownership together and shares the result with managed options and descriptor output.

Generation, policy commands, and module operations share root/collision checks. <code>WalkProtoFiles</code> owns dependency-source traversal, including exclusion of hidden directories, vendored sources and nested module boundaries. Generator discovery and policy ancestor traversal remain separate: their inclusion and inheritance rules differ.

Nested protobuf module roots are rebased by <code>module_config</code> from the nested manifest directory into the checkout directory. Import names remain relative to those roots; repository/module prefixes do not become part of an import.

The Git cache lives beneath the CLI-supplied EasyP directory in <code>v1/git</code>; <code>internal/adapters/gitmodules/object_cache.go</code> maintains reusable bare object repositories with OS locks. <code>Cache.Cached</code> reads installed metadata; <code>Install</code> supplies content verification before cached metadata is trusted.
The Git cache lives beneath the CLI-supplied EasyP directory in <code>v1/git</code>; <code>internal/adapters/gitmodules/object_cache.go</code> maintains reusable bare object repositories with OS locks. <code>Cache.Cached</code> reads installed metadata; <code>Install</code> supplies content verification before cached metadata is trusted. Successful verification and the inventory are reused within one cache handle, keyed by source, commit, content hash and selected roots. A new handle rechecks the snapshot; explicit <code>VerifyCached</code> bypasses the in-memory result. Revision resolution atomically retains completed snapshots so installation does not materialize them a second time.

Local replacements use an ephemeral main-module overlay graph and never alter the shared lock. Tidy validates the overlay; get/update may edit explicit manifest requirements. Vendor copies the effective local graph without touching the lock. Explicit frozen operation verifies existing native manifests and locked graphs; automatic unknown-import-to-module discovery is intentionally not implemented.

Expand Down
16 changes: 12 additions & 4 deletions .spec/config/dependency.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,13 @@ See [Go major version suffixes](https://go.dev/ref/mod#major-version-suffixes) a

The CLI resolves <code>EASYPPATH</code> once for a command that needs the cache (default <code>$HOME/.easyp</code>). <code>gitmodules.Cache</code> owns <code>&lt;EASYPPATH&gt;/v1/git</code>, source keys, temporary checkout names and installation paths. Installed snapshots live under its <code>modules/&lt;source-key&gt;/&lt;snapshot-key&gt;</code> layout, where the snapshot key hashes commit plus content hash; reusable bare object stores live under <code>objects/&lt;remote-key&gt;</code>, with OS locks for concurrent fetches. Source/commit bindings under <code>sources/</code> identify these stores for read-only policy access. Application callers request cached module metadata and its physical directory; they do not assemble cache paths.

<code>Cache.Install</code> verifies locked contents, including cache hits. A newly installed snapshot is checked with the full <code>h1:</code> directory hash. On Darwin/Linux, later unchanged hits use a sidecar verification stamp keyed by the expected lock hash plus an inode/ctime metadata fingerprint; any metadata change invalidates the stamp and forces the full directory hash again. Content changes therefore remain detectable even when size and mtime are restored. Platforms without a strong change token keep the full-hash path. Stamp hits first check the current snapshot selection policy. Read-only validation never writes verification stamps. The stamp is an optimization of the per-user local cache, not an additional source of dependency identity. Cached objects support shallow pinned-commit fetches with an advertised-history fallback; neither path substitutes HEAD for an unavailable pin. <code>Cache.Cached</code> reads installed metadata without downloading or rewriting module contents and requires prior verification. Git credentials and transport configuration remain the responsibility of system Git. Ordinary native source use does not require objects for unselected policies. Explicit <code>mod download</code> repairs missing policy object sources, preserving every reached pin in a shared repository; validation reports missing sources without fetching or writing.
<code>Cache.Install</code> verifies a snapshot on its first use by a cache handle, then reuses that successful verification and its file inventory within the command. The key includes source, commit, content hash and root selection. Installed snapshots are immutable for that lifetime; local replacements are never admitted to this cache. A new command creates a new handle and checks installed contents again. <code>Cache.VerifyCached</code> explicitly rechecks contents even on the same handle, without downloading or writing verification stamps; a failed check invalidates the remembered result.

A newly acquired snapshot is checked with the full <code>h1:</code> directory hash. On Darwin/Linux, an unchanged snapshot from a previous command can use a sidecar stamp keyed by the expected lock hash plus an inode/ctime metadata fingerprint; any metadata change forces the full directory hash again. Content changes remain detectable on the next command or explicit verification even when size and mtime are restored. Platforms without a strong change token keep the full-hash path on first use. Stamp hits first check the current snapshot selection policy. The stamp is an optimization of the per-user local cache, not an additional source of dependency identity. A completed, hashed snapshot produced during revision resolution is installed atomically and reused by the following install step; provisional root inspections cannot publish an incomplete snapshot.

The optional <code>Cache.SourceFiles</code> capability exposes a copy of the verified snapshot's regular-file inventory, including configuration markers needed for nested module boundaries. Dependency source discovery, global import-collision checks and module ownership use this inventory instead of walking the installed filesystem for each target. Generation shares one collision/ownership index across managed options and descriptor attribution. This does not limit collision detection to files reached by compilation.

Cached objects support shallow pinned-commit fetches with an advertised-history fallback; neither path substitutes HEAD for an unavailable pin. <code>Cache.Cached</code> reads installed metadata without downloading or rewriting module contents and requires prior verification. Git credentials and transport configuration remain the responsibility of system Git. Ordinary native source use does not require objects for unselected policies. Explicit <code>mod download</code> repairs missing policy object sources, preserving every reached pin in a shared repository; validation reports missing sources without fetching or writing.

<code>EASYP_PROXY</code>, when set to a host or <code>host:port</code>, replaces the host of Git remotes used to fetch dependencies. A module declared as <code>github.com/group/package</code> is cloned from <code>https://gitlab.com/group/package</code> when the variable is <code>gitlab.com</code>. Explicit <code>https</code> and <code>ssh</code> URLs keep their path and user; only the host changes, including a port supplied by the variable. Absolute local paths are unchanged. The published module identity, lock source, and content hash stay the original module path. An empty value disables substitution. <code>config.Config.LoadEnvironment</code> reads the variable into <code>Config.Proxy</code>; it is not an <code>easyp.yaml</code> field.

Expand All @@ -81,8 +87,10 @@ phases. Start/finish records include status and elapsed time; inventories includ
file counts and selected roots/filters. Slow phases report elapsed progress every
15 seconds at info level, including without <code>--debug</code>. These logs do
not change source selection or dependency identity. Indexed tree and blob
metadata are reused only within one immutable Git filesystem; mutable local
source resolution remains uncached.
metadata are reused within one immutable Git filesystem. Verified installed
snapshot inventories are reused within one command. Mutable local source
resolution remains uncached between calls. A walk resolves its local boundary
once; reusing descendant metadata is restricted to explicitly immutable views.

Deleting the old <code>easyp.lock</code> is not required for migration. Its absence
does not change an explicit <code>git_repo.url: repo@&lt;commit&gt;</code> pin.
Expand Down Expand Up @@ -163,7 +171,7 @@ Explicit <code>--frozen</code> requires <code>protobuf.mod</code> and <code>prot

All root replacement directives are forbidden, even when unused. Validation rejects them before opening replacement directories or accessing dependency caches/remotes. Dependency manifests still supply transitive requirements; dependency-local replacements remain ignored. The locked closure must satisfy direct and transitive requirements with no missing or stale entries. Malformed locks, incompatible versions/commits and unexpected graph entries fail instead of triggering resolution.

The frozen path does not call the version resolver, query HEAD or tags, or create/rewrite manifests and locks. Exact locked commits may be downloaded when absent from cache; installed sources are lock-hash verified on cold paths, while unchanged Darwin/Linux warm hits reuse the verified metadata stamp and fall back to a full <code>h1:</code> check after any detected filesystem change. This permits reproducible dependency selection with a cold cache and does not imply offline execution. Existing Git credentials, transport configuration, and plugin network requirements still apply.
The frozen path does not call the version resolver, query HEAD or tags, or create/rewrite manifests and locks. Exact locked commits may be downloaded when absent from cache. First use by the command verifies the snapshot against the lock, using the Darwin/Linux metadata stamp where available and falling back to a full <code>h1:</code> check after a detected filesystem change. Later targets reuse that verified immutable snapshot within the same command. This permits reproducible dependency selection with a cold cache and does not imply offline execution. Existing Git credentials, transport configuration, and plugin network requirements still apply.

- <code>mod download</code> validates and installs the locked graph; <code>mod vendor</code> validates it before replacing vendor output.
- <code>mod tidy</code>, <code>get</code>, <code>mod update</code>, and <code>init</code> refuse frozen operation. <code>migrate</code>, including preview and interactive mode, is also rejected.
Expand Down
6 changes: 5 additions & 1 deletion internal/adapters/gitmodules/bsr_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,11 @@ func TestFetchBSRDependencies(t *testing.T) {
entries, readErr := os.ReadDir(cache.root)
require.NoError(t, readErr)
for _, entry := range entries {
assert.Equal(t, "objects", entry.Name(), "temporary checkout survived Fetch")
allowed := []string{"objects"}
if err == nil {
allowed = append(allowed, "modules", "sources")
}
assert.Contains(t, allowed, entry.Name(), "temporary checkout survived Fetch")
}
if tt.wantErrIs != nil {
require.ErrorIs(t, err, tt.wantErrIs)
Expand Down
9 changes: 6 additions & 3 deletions internal/adapters/gitmodules/cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"fmt"
"log/slog"
"path/filepath"
"sync"

"github.com/easyp-tech/easyp/internal/adapters/bsr"
"github.com/easyp-tech/easyp/internal/adapters/gitcommand"
Expand All @@ -21,14 +22,16 @@ type Cache struct {
bsrResolver modules.BSRResolver
logger logger.Logger
proxy string
stateMu sync.Mutex
state *cacheRunState
}

// WithLogger returns an isolated cache handle with dependency progress logging.
// Historical-pin scopes preserve this logger through their existing shallow copy.
// Logger and historical-pin scopes share verified immutable snapshot state.
func (c *Cache) WithLogger(log logger.Logger) *Cache {
scoped := *c
scoped := c.scoped()
scoped.logger = log
return &scoped
return scoped
}

func (c *Cache) operationContext(ctx context.Context, source, version string) context.Context {
Expand Down
99 changes: 99 additions & 0 deletions internal/adapters/gitmodules/cache_run.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
package gitmodules

import (
"context"
"fmt"
"os"
"sync"

v1 "github.com/easyp-tech/easyp/internal/config/v1"
)

// A cache handle belongs to one command. Only its successfully installed,
// immutable snapshots share verification and file inventories within that run.
type cacheRunState struct {
mu sync.Mutex
snapshots map[string]*cachedSnapshot
}

type cachedSnapshot struct {
gate chan struct{}
verified bool
files []string
identity os.FileInfo
}

func (snapshot *cachedSnapshot) lock(ctx context.Context) error {
select {
case snapshot.gate <- struct{}{}:
if err := ctx.Err(); err != nil {
snapshot.unlock()
return err
}
return nil
case <-ctx.Done():
return ctx.Err()
}
}

func (snapshot *cachedSnapshot) unlock() { <-snapshot.gate }

// accept records a snapshot hashed before its atomic publication. The caller
// holds the per-snapshot gate; no second file-content pass is needed.
func (snapshot *cachedSnapshot) accept(installed string, files []string) error {
info, err := os.Lstat(installed)
if err != nil {
return fmt.Errorf("Lstat: %w", err)
}
// Persistent verification stamps require matching fingerprints before and
// after the content hash. Publication has only the already computed hash;
// a later command verifies the installed tree before writing a stamp.
snapshot.files, snapshot.identity, snapshot.verified = files, info, true
return nil
}

func (c *Cache) runState() *cacheRunState {
c.stateMu.Lock()
defer c.stateMu.Unlock()
if c.state == nil {
c.state = &cacheRunState{snapshots: make(map[string]*cachedSnapshot)}
}
return c.state
}

func (c *Cache) scoped() *Cache {
return &Cache{root: c.root, bsrResolver: c.bsrResolver, logger: c.logger, proxy: c.proxy, state: c.runState()}
}

func (c *Cache) snapshotState(installed string) *cachedSnapshot {
state := c.runState()
state.mu.Lock()
defer state.mu.Unlock()
if snapshot := state.snapshots[installed]; snapshot != nil {
return snapshot
}
snapshot := &cachedSnapshot{gate: make(chan struct{}, 1)}
state.snapshots[installed] = snapshot
return snapshot
}

// SourceFiles returns the sorted logical file inventory of a snapshot verified
// by this cache handle. Paths are relative to its installation directory and
// include metadata so consumers can preserve nested module ownership.
// The returned slice is independent of the command's immutable cached inventory.
// A nil slice means this handle has no verified inventory and callers must use
// their ordinary source traversal. A verified empty snapshot returns an empty,
// non-nil slice. This method never verifies or trusts a directory's existence.
func (c *Cache) SourceFiles(entry v1.LockedModule) ([]string, error) {
snapshot := c.snapshotState(v1ModuleCachePath(c.root, entry))
if err := snapshot.lock(context.Background()); err != nil {
return nil, fmt.Errorf("lock: %w", err)
}
defer snapshot.unlock()
if !snapshot.verified {
return nil, nil
}
files := make([]string, len(snapshot.files))
copy(files, snapshot.files)
return files, nil
}
Loading
Loading