Skip to content

Commit 4288a72

Browse files
committed
Keep scoped scans fast on large pnpm stores
The previous commit matched member links by expanding every crawled npm purl's store peer variants, re-listing the whole store once per purl. A path-scoped scan of a 3,000-entry pnpm store went from 0.24s to 35s (debug build). Links in each node_modules root are now matched by their target's own package.json name@version, in one pass however many purls are asked for. The same 3,000-entry scan takes 0.24s again, and a link to a peer variant the crawl skipped still counts. Assisted-by: Claude Code:claude-opus-5-5
1 parent 84b3690 commit 4288a72

2 files changed

Lines changed: 44 additions & 34 deletions

File tree

‎crates/socket-patch-cli/CLI_CONTRACT.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -210,7 +210,7 @@ For a **9.0 root lock**, the CLI ensures `pnpm-workspace.yaml` carries `trustLoc
210210
**Path-scoped scans (`scan [PATHS]...`, v5.0)**: what a PATH means depends on the mode.
211211

212212
* **Hosted and vendored mode (bare `scan` included) — project directories** (`run_project_dirs`). Each PATH is a directory, or a glob (`*?[`) matching directories, relative to `--cwd`; the set is sorted and deduplicated, and each directory is scanned on its own exactly as if it were `--cwd` (its own lockfiles, ledgers and `.socket/`). With more than one directory, each run is headed `== <dir> ==` on stdout (unless `--silent`), and the exit code is the worst of the runs. Usage errors (exit 2, stderr only, before any scan): a PATH that is not a directory (`` `X` is not a directory``), a glob matching no directory (`` `X` matches no directory``), an invalid glob, and `--json` with more than one directory (`--json takes one project directory (N given); run one scan per directory`), so stdout stays one document. Likewise `--vex` with more than one directory (`--vex takes one project directory (N given); run one scan per directory`): the one output path would be overwritten by each run.
213-
* **Agent mode (and a mode-less `--prune`/`--global` report) — installed-path globs** scoping DISCOVERY at the **purl level**: a package is in scope iff ANY of its crawled installed copies sits under a matching path (every copy, enumerated the way `rollback`'s path targets enumerate them: a pnpm workspace member's link into the root `.pnpm` store (or npm's linked `.store`), including an alias link such as `packages/a/node_modules/lp` for `"lp": "npm:left-pad@1.3.0"`, and a nested member copy count, not just the one copy the crawl records per purl), and a selected package is then handled with ALL its copies (scoping selects which packages are considered, never which copies). Glob semantics (shared with `rollback`'s path targets, `src/path_scope.rs`): Unix-shell globs with `require_literal_separator` — `*`/`?` never cross a `/`, `**` spans directories; a pattern matching any **ancestor** directory of the copy path also matches, so a bare `scan packages/foo` scopes the whole subtree without `/**`; relative patterns match against the copy path relativized to `--cwd`, absolute patterns against the absolute path (the ONLY way to reach paths outside the project tree, e.g. `--global` stores — a relative pattern never matches outside `--cwd`); leading `./` and trailing `/` are normalized away, matching is purely textual (no filesystem access or symlink resolution), case-sensitive except on Windows (whose filesystems are not); an unparseable or empty pattern is a usage error (exit 2). **The prune universe is never narrowed**: the path filter is applied strictly AFTER the `scanned_purls` capture (and after `--ecosystems`), so `scan PATHS --prune` prunes exactly what an unscoped `scan --prune` would — a scoped scan can never treat an out-of-scope package as uninstalled (the same fail-safe as the `--ecosystems` filter). Lockfile-only and vendor-ledger supplement records have no installed path and are EXCLUDED from a path-scoped scan, surfaced as one run-level `path_scope_excluded_supplements` warning carrying the count. A scope matching nothing is a normal empty scan — exit 0, zero packages, **no GC** (the zero-package early return fires before any GC). `PATHS` combine with `--mode agent`/`--sync`/`--prune`/`--global`. Every scan JSON envelope past the argument checks (success, zero-package, all-batches-failed) carries a `paths` key echoing the patterns verbatim (empty array when unscoped; a hosted/vendored per-directory run is unscoped, so it is `[]`). One-sentence duality rule: **a target that selects nothing is an error on `rollback` (exit 1) and an empty scan on an agent-mode `scan` (exit 0)**.
213+
* **Agent mode (and a mode-less `--prune`/`--global` report) — installed-path globs** scoping DISCOVERY at the **purl level**: a package is in scope iff ANY of its crawled installed copies sits under a matching path (every copy, enumerated the way `rollback`'s path targets enumerate them: any package link in a `node_modules` root, such as a pnpm workspace member's link into the root `.pnpm` store (or npm's linked `.store`), alias links included (`packages/a/node_modules/lp` for `"lp": "npm:left-pad@1.3.0"`, matched by its target's own `package.json` name@version), and a nested member copy count, not just the one copy the crawl records per purl), and a selected package is then handled with ALL its copies (scoping selects which packages are considered, never which copies). Glob semantics (shared with `rollback`'s path targets, `src/path_scope.rs`): Unix-shell globs with `require_literal_separator` — `*`/`?` never cross a `/`, `**` spans directories; a pattern matching any **ancestor** directory of the copy path also matches, so a bare `scan packages/foo` scopes the whole subtree without `/**`; relative patterns match against the copy path relativized to `--cwd`, absolute patterns against the absolute path (the ONLY way to reach paths outside the project tree, e.g. `--global` stores — a relative pattern never matches outside `--cwd`); leading `./` and trailing `/` are normalized away, matching is purely textual (no filesystem access or symlink resolution), case-sensitive except on Windows (whose filesystems are not); an unparseable or empty pattern is a usage error (exit 2). **The prune universe is never narrowed**: the path filter is applied strictly AFTER the `scanned_purls` capture (and after `--ecosystems`), so `scan PATHS --prune` prunes exactly what an unscoped `scan --prune` would — a scoped scan can never treat an out-of-scope package as uninstalled (the same fail-safe as the `--ecosystems` filter). Lockfile-only and vendor-ledger supplement records have no installed path and are EXCLUDED from a path-scoped scan, surfaced as one run-level `path_scope_excluded_supplements` warning carrying the count. A scope matching nothing is a normal empty scan — exit 0, zero packages, **no GC** (the zero-package early return fires before any GC). `PATHS` combine with `--mode agent`/`--sync`/`--prune`/`--global`. Every scan JSON envelope past the argument checks (success, zero-package, all-batches-failed) carries a `paths` key echoing the patterns verbatim (empty array when unscoped; a hosted/vendored per-directory run is unscoped, so it is `[]`). One-sentence duality rule: **a target that selects nothing is an error on `rollback` (exit 1) and an empty scan on an agent-mode `scan` (exit 0)**.
214214

215215
`scan --mode vendored` swaps the in-place apply for the vendor pipeline: discover → download the selected patch records **into memory** (no manifest write) → vendor every selected dependency via the same engine as the `vendor` command (under the same lock). Vendored mode is **manifest-free (v5.0)**: `.socket/manifest.json` is never written or read by a vendored run; each ledger entry carries `detached: true` plus an embedded copy of the patch record (`record`) as its verification source, and the run's footprint is `.socket/vendor/**` only. The vendor step's scope is what discovery selected — the former "whole manifest is vendored" re-vendor on an empty discovery is retired (`repair` verifies and redownloads committed vendored state; `scan --prune` reconciles ledger entries whose dependency left the lockfile). The vendor-ledger discovery supplement (the fresh-clone rule: a ledger entry with no installed copy stays discoverable because its committed artifact IS the dependency) holds only while the lockfile still resolves through that artifact: an entry the lockfile in-use probe (the one `--prune` reverts by) proves unwired, because the dependency was upgraded or removed, is NOT discovered and so is never re-vendored. A run without a non-hosted `--prune` reports it through the run-level `vendor_ledger_entry_unwired` warning; a `--prune` run reverts it in its GC and exits 0. That GC runs even when the crawl found no packages, as its vendored half alone (the manifest prune stays skipped there). A package the ledger holds at an older patch uuid is still **re-vendored automatically** when discovery selects the newer patch (its old uuid dir is removed — `vendor_stale_artifact_removed`) — unless the patch service has no prebuilt artifact for the newer patch yet (or at all): an npm-family package then keeps the older patch and is a `skipped` `vendor_prebuilt_pending` / `vendor_prebuilt_unavailable` event, not a failure (#954); same-uuid re-runs reuse the embedded record, skip the patch-view fetch, and are `already_vendored` skips. **Legacy manifest-mode entries**: when a vendored run vendors a purl that also has a `.socket/manifest.json` record (a project vendored by a pre-5.0 binary, or by standalone `vendor` from an agent-mode manifest), that manifest record is dropped in the same run — the ledger becomes the owner (migration write); an emptied manifest is left as `{"patches": {}}`, never deleted. The migration is reported through the run-level `warnings[]` (stderr in human mode), never as a run error: `vendor_manifest_record_migrated` (`N manifest records moved to the vendor ledger (vendored mode is manifest-free): <purls>`) or `vendor_manifest_migration_failed` (the manifest or the ledger could not be read or rewritten; the legacy records were left in place) — so a corrupt `.socket/manifest.json` no longer fails a vendored run (standalone `vendor`, the one manifest-driven writer, still fails closed on it). With `--prune`, GC runs **after** the vendor step (the step never reads the manifest, and running the sweep last lets it reclaim what the run itself orphaned — a migrated legacy record's blobs, a superseded uuid dir). Under `--json` (v5.0, MAJOR) the download phase and the vendor engine record into the ONE scan envelope (no nested `download` / `vendor` objects): a fetched patch is a `downloaded` event (the patch metadata in `details`, plus `details.oldUuid` when the ledger holds the purl at another uuid — the re-vendor the vendor step then performs — whose human `[fetch]` line reads `<purl> (replacing <short uuid>)`), a refused or failed one a `failed` event with its code, a patch the ledger already holds at the selected uuid no download event at all (reused, no fetch); then the vendor engine's events (`applied`, `rebuilt`, `skipped`, `failed`, `removed`, its uncounted advisories) follow with their warnings and sidecars. Every one of these events carries `details.mode: "vendored"`. Patch blobs are held in memory (see "Patch sources stay in memory" under the vendor contract). `--dry-run` previews per patch: `verified` (to vendor; `oldUuid` on a re-vendor), `skipped` / `already_vendored` (in sync), or `skipped` with the refusal code for an npm purl the wet run's Bun preflight (see the `get --mode vendored` bullet below) would refuse, or whose hosted pnpm pin the takeover would be refused on (see "Takeover reconciliation") — without network downloads or disk writes; the preview never flips status or exit (the human path — `scan` and `get` alike, through one shared printer — prints `[would-refuse] <purl> (<code>): <detail>` lines behind the `--silent` gate). Interactive mode prompts "Download and vendor N patches?" (singular for one).
216216

‎crates/socket-patch-cli/src/ecosystem_dispatch.rs‎

Lines changed: 43 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ use std::path::{Path, PathBuf};
99
use crate::args::GlobalArgs;
1010

1111
use socket_patch_core::crawlers::npm_crawler::{
12-
retain_live_store_copies, with_store_peer_variant_copies,
12+
build_npm_purl, parse_package_name, retain_live_store_copies, with_store_peer_variant_copies,
1313
};
1414
use socket_patch_core::crawlers::walk_pool;
1515
use socket_patch_core::crawlers::CargoCrawler;
@@ -650,8 +650,8 @@ pub async fn find_all_packages_for_rollback_reusing(
650650
/// The install paths that reach each purl, for `scan`'s and `rollback`'s
651651
/// path targets: [`find_all_packages_for_rollback_reusing`]'s physical
652652
/// copies, plus each npm package link directly in a `node_modules` root
653-
/// (scoped ones one level down) that resolves to one of them or to one
654-
/// of their store peer variants.
653+
/// (scoped ones one level down) whose target's `package.json` names one
654+
/// of the npm purls asked for.
655655
///
656656
/// The crawler records a plain-name link at its own path
657657
/// (`packages/a/node_modules/ms`), but never an alias link
@@ -660,23 +660,28 @@ pub async fn find_all_packages_for_rollback_reusing(
660660
/// once (#856). Path scoping asks which paths install the package
661661
/// instead, and pnpm's isolated linker or npm's `install-strategy=linked`
662662
/// reach a workspace member's ALIAS dependency only through such a link
663-
/// (#1266). The member's link can resolve to a peer variant the crawl
664-
/// skipped once it found the root's copy, so the variants count too.
665-
/// The result is for path matching only: it can name one physical copy
666-
/// more than once.
663+
/// (#1266). A link is matched by its target's own identity, not against
664+
/// the crawled copies, so a link to a store peer variant the crawl
665+
/// skipped still counts, and the cost is one pass over the root links
666+
/// however many purls are asked for. The result is for path matching
667+
/// only: it can name one physical copy more than once.
667668
pub async fn find_package_paths_for_path_scope(
668669
partitioned: &HashMap<Ecosystem, Vec<String>>,
669670
options: &CrawlerOptions,
670671
silent: bool,
671672
prior: Option<&NpmCrawlSnapshot>,
672673
) -> HashMap<String, Vec<PathBuf>> {
673-
let has_npm = partitioned
674-
.get(&Ecosystem::Npm)
675-
.is_some_and(|purls| !purls.is_empty());
674+
let mut wanted: HashMap<PurlKey, Vec<String>> = HashMap::new();
675+
for purl in partitioned.get(&Ecosystem::Npm).into_iter().flatten() {
676+
wanted
677+
.entry(PurlKey::new(purl))
678+
.or_default()
679+
.push(purl.clone());
680+
}
676681
let npm_roots = match prior.and_then(|p| p.roots_for(options)) {
677-
Some(roots) if has_npm => Some(roots.to_vec()),
678-
None if has_npm => NpmCrawler.get_node_modules_paths(options).await.ok(),
679-
_ => None,
682+
_ if wanted.is_empty() => None,
683+
Some(roots) => Some(roots.to_vec()),
684+
None => NpmCrawler.get_node_modules_paths(options).await.ok(),
680685
};
681686
let mut found = dispatch_find(
682687
partitioned,
@@ -689,22 +694,8 @@ pub async fn find_package_paths_for_path_scope(
689694
let Some(roots) = npm_roots else {
690695
return found;
691696
};
692-
let mut by_target: HashMap<PathBuf, Vec<String>> = HashMap::new();
693-
for (purl, paths) in &found {
694-
if Ecosystem::from_purl(purl) != Some(Ecosystem::Npm) {
695-
continue;
696-
}
697-
for path in with_store_peer_variant_copies(paths.clone()).await {
698-
if let Ok(target) = tokio::fs::canonicalize(&path).await {
699-
by_target.entry(target).or_default().push(purl.clone());
700-
}
701-
}
702-
}
703-
if by_target.is_empty() {
704-
return found;
705-
}
706697
// A failed enumeration only loses the extra links; the copies stand.
707-
let links = tokio::task::spawn_blocking(move || npm_links_to(&by_target, &roots))
698+
let links = tokio::task::spawn_blocking(move || npm_links_to(&wanted, &roots))
708699
.await
709700
.unwrap_or_default();
710701
for (purl, link) in links {
@@ -717,10 +708,10 @@ pub async fn find_package_paths_for_path_scope(
717708
}
718709

719710
/// Every package link directly in one of `roots` (scoped ones one level
720-
/// down) that resolves to a key of `by_target`, paired with each purl
721-
/// that key belongs to.
711+
/// down) whose target directory's `package.json` `name@version` is a key
712+
/// of `wanted`, paired with each purl spelling that key stands for.
722713
fn npm_links_to(
723-
by_target: &HashMap<PathBuf, Vec<String>>,
714+
wanted: &HashMap<PurlKey, Vec<String>>,
724715
roots: &[PathBuf],
725716
) -> Vec<(String, PathBuf)> {
726717
let entries = |dir: &Path| -> Vec<(String, PathBuf, std::fs::FileType)> {
@@ -735,6 +726,25 @@ fn npm_links_to(
735726
})
736727
.collect()
737728
};
729+
let identity = |link: &Path| -> Option<PurlKey> {
730+
let target = std::fs::canonicalize(link).ok()?;
731+
let text =
732+
socket_patch_core::utils::fs::read_regular_to_string_sync(&target.join("package.json"))
733+
.ok()?;
734+
let pkg: serde_json::Value =
735+
serde_json::from_str(socket_patch_core::formats::text::strip_bom(&text)).ok()?;
736+
let name = pkg.get("name")?.as_str()?;
737+
let version = pkg.get("version")?.as_str()?;
738+
if name.is_empty() || version.is_empty() {
739+
return None;
740+
}
741+
let (namespace, leaf) = parse_package_name(name);
742+
Some(PurlKey::new(&build_npm_purl(
743+
namespace.as_deref(),
744+
&leaf,
745+
version,
746+
)))
747+
};
738748
let mut out = Vec::new();
739749
for root in roots {
740750
let mut links = Vec::new();
@@ -751,10 +761,10 @@ fn npm_links_to(
751761
}
752762
}
753763
for link in links {
754-
let Ok(target) = std::fs::canonicalize(&link) else {
764+
let Some(key) = identity(&link) else {
755765
continue;
756766
};
757-
for purl in by_target.get(&target).into_iter().flatten() {
767+
for purl in wanted.get(&key).into_iter().flatten() {
758768
out.push((purl.clone(), link.clone()));
759769
}
760770
}

0 commit comments

Comments
 (0)