From a7a14cf0e509197b191ba0bff23d732e5064a0a6 Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Sun, 18 Oct 2026 16:57:19 +0200 Subject: [PATCH 01/10] =?UTF-8?q?docs(adr):=20specify=20phases=20A2?= =?UTF-8?q?=E2=80=93A8=20of=20the=20layered=20index?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- ...shared-base-index-and-worktree-overlays.md | 432 +++++++++++++++++- 1 file changed, 426 insertions(+), 6 deletions(-) diff --git a/docs/adr/0016-shared-base-index-and-worktree-overlays.md b/docs/adr/0016-shared-base-index-and-worktree-overlays.md index 55953d12..82a469e2 100644 --- a/docs/adr/0016-shared-base-index-and-worktree-overlays.md +++ b/docs/adr/0016-shared-base-index-and-worktree-overlays.md @@ -20,11 +20,13 @@ scan cost should be proportional to the worktree's diff, with query results iden Group worktrees by blake3 of the canonical git common dir and split the index into an immutable **base** and a per-worktree **overlay**: -- **Base** — an index of `origin/HEAD` (falling back to the main worktree HEAD) built from git objects - through the existing rev scan, so it never contains uncommitted edits. Generations advance by a CoW - clone plus `git diff-tree C_n..C_n+1` applied through the incremental writer when files have drifted by - 2% or more, or daily; at most three exist, each published read-only so the daemon keeps one handle and - one block cache per group. +- **Base** — an index of `origin/HEAD` (falling back to the main worktree HEAD) built from the committed + tree, so it never contains uncommitted edits. The tree is exported to a scratch directory and scanned + like a working tree, because the rev scan alone does not run the resolve pass. Generations advance by + a CoW clone plus `git diff-tree C_n..C_n+1` applied through the incremental writer when files have + drifted by 2% or more, or daily; at most three are live, each published once and never written again, + so the daemon keeps one handle and one block cache per group (fjall admits one process per directory, + so other processes reach the base through the daemon). - **Overlay** — `delta.msgpack` records each path as added, modified or deleted relative to the base manifest by content hash; `overlay.fjall` holds rows only for changed paths and re-stitched importers. Shadow sets `S_path` (delta paths), `S_refs` (`S_path` plus re-stitched importers) and `D_hash` (base @@ -44,6 +46,409 @@ and a per-worktree **overlay**: `documents`/`doc_links` are re-keyed to `(scope, content_hash)`, with queries over-fetching and filtering by live hashes. Documented deviation: top-k overlap with a per-worktree table stays ≥0.95. +## Specification + +Everything below is normative for A2–A8. Paths and symbols are as of 0.30.0 (`bd1f0a5b`). Where the code +contradicts the text of issue #87 or the summary above, the code wins and the correction is listed first. + +### Corrections to the issue text + +| Claim in #87 | What the code does | Consequence | +|---|---|---| +| "Rev scan builds the base" (`ScanSource::Rev`) | `scan_with_cancel` runs the resolve pass only for `ScanSource::WorkingTree` (`scanner.rs`, `if matches!(source, ScanSource::WorkingTree)`), and the resolver reads the filesystem (`intel/resolve_pass.rs` `std::fs::read`, `intel/resolver.rs` `root.join`, `canonicalize`). A rev scan therefore writes no `refs_by_*`, `import_targets` or `importers_by_probe` rows, and the rev branch of `process_file` writes no grep bloom either. | The base builder exports the committed tree to a private scratch directory and runs a normal working-tree scan on it (A4). A virtual git-tree filesystem for the resolver was rejected: it duplicates the resolver's I/O surface for a transient 1-3 GB export. | +| "Published read-only so the daemon keeps one handle" | Fjall's directory lock is exclusive even for a read-only open (`Store::open_read_only_no_index`, `mcp/index_route.rs` module doc). Read-only is a logical property, not something the OS enforces on a shared fjall. | One process per machine can hold a given base generation. The daemon is that holder; every other process reaches the base through `IndexRoute::Host` / `Daemon` or degrades to `InRam`, exactly as a second `serve` does today (A4). | +| "`code_chunks` is already content-addressed" | `chunk_id` is `:` (`chunk.rs`) and the sidecar blobs are content-addressed, but the Lance rows are keyed `(scope, path)` (`lance/mod.rs` `replace_code_chunks`, `delete_code_chunks`) and `scope` is the normalised `origin` URL (`git/remote.rs` `scope_key`), identical for every worktree of a clone. | Sharing a Lance table needs a new key for all three tables, not just `documents`/`doc_links` (A7). | +| A3 "shipped" (progress comment) | Only the ADR shipped. `src/index/view.rs` does not exist; `IndexDb` keyspaces are `pub(crate)` and read directly (`mcp/helpers_calls_scan.rs` `calls_by_path`, `mcp/helpers_calls.rs`, `search/bm25.rs` `code_bm25_postings`, `intel/stage_budget.rs`). | A3 is still a code phase and a prerequisite of A5. | +| `repo_group_key` hashes "the git common dir" | It hashes the canonical main worktree root, the parent of the common dir (`store_layout.rs` `repo_group_key`, `git/mod.rs` `main_worktree_root`). Equivalent for a normal clone; a bare clone or a moved main checkout re-keys the group. | Keep the function; record the resolved common dir in `group.json` so a re-key is diagnosable (A4). | +| BM25 `meta` is exact as "base + overlay - shadowed" | True, but `upsert_bm25_file` / `remove_bm25_file` delete every posting of the path's `chunk_id`s (`index/writer.rs` `stage_bm25_deletes_for`), and `chunk_id` embeds the content hash. Two paths with identical bytes share their postings, so removing or editing one of them deletes the postings the other still needs. | Pre-existing defect in the standalone index, not specific to layering, and it breaks the A5 equivalence gate on the "duplicate content" scenario. A5.0 fixes it first (below). | +| `main_root_of` / `repo_group_key` cover a workspace | A workspace rooted at a *subdirectory* of a repository also gets a group key (`Repo::discover` walks up), but a base built from the whole tree would not match a subdirectory scan. | Layering applies only when the canonical root equals the git workdir (the `store_seed::main_root_of` predicate); otherwise the workspace stays standalone (A4). | + +### Vocabulary + +- **group**: `repo_group_key(root)`; its directory is `repo_group_dir(root)` = `cache/repos//` (`store_layout.rs`). +- **wt_key**: `workspace_key(root)`; its directory is `workspace_cache_dir(root)` (unchanged, still holds `.lock`, + `workspace.json`, `status.json`, git caches, and the `staged` / `rev-*` views, which stay standalone). +- **W**: the paths a working-tree scan would index (`scanner_candidates::walk_candidates` after `Filters`). +- **T**: the keys of the base manifest (`Index.files` of the pinned generation), i.e. the filtered paths of `tree(C_base)`. +- **H_base**: the set of content hashes in the base manifest, with a per-hash path count. +- **Shadow sets**: `S_path`, `S_refs`, `D_hash` as defined in the Decision. + +### A2: re-key `grep_bloom` by content hash + +The plan in #87 stands; this fixes the details the plan left open. + +- **Key and row.** Key = the 64-character lowercase hex blake3 (`hashing::hex`), so two paths with identical + bytes share one row. Row = `version(1) | bits`; `ROW_VERSION` stays 1 because the partition-revision bump + (below) removes every old row. `build_row(bytes)`; `GrepEntry { rel, hash_hex, size_bytes, mtime_ns }` + (`index/grep_bloom.rs`, serde, because it crosses the daemon wire). +- **Live-file guard.** `grep_skip_verdicts(root, &[GrepEntry], needle)` skips a file only if the row for + `entry.hash_hex` rejects the needle *and* `stat(root/rel)` equals the **indexed** `(size_bytes, mtime_ns)` of + the entry, taken from `FileEntry` (`store.rs`) through `FileMeta` (`mcp/l1_cache.rs`, gains `mtime_ns`). A file + whose indexed mtime is 0 (git sources) never equals a live stat and stays a candidate. +- **Stamp refresh (not in the plan).** The stamp used to live in the row and `restamp_grep_bloom` refreshed it + when a file was touched with unchanged content. The stamp now lives in the `FileEntry`, and + `process_file` returned `Unchanged` without upserting one, so a touched file would have stayed a permanent + grep candidate. The unchanged-content branch of `scanner_file::process_file` therefore returns a + `FileResult` carrying a refreshed `FileEntry` (same hash, new `size_bytes` / `mtime`) for working-tree scans. +- **Writer.** `IndexWriter::upsert_grep_bloom(hash_hex, row)` overwrites an identical row; `remove_file` no longer + touches `grep_bloom`. `WorkerIndexBatch::{stage,restamp,backfill}_grep_bloom` key on the hash; backfill reads + the file only when the hash has no row and drops the bytes if they no longer hash to the indexed value. +- **Sweep.** `IndexDb::sweep_grep_bloom(&live)` deletes every row whose hash is not in `live`. `scanner::scan_with_cancel` + calls it right after the stale-entry purge with `live` = hashes of `store.index.files`, so a pass that was + cancelled or aborted (which returns before the purge) never sweeps. Edited and deleted contents leave their + row behind until that sweep, which is the only deletion path. +- **Schema.** `INDEX_PARTITION_REVISION` 6 to 7 (`index/mod.rs`). Cost: every user re-indexes once (a 17-33 min + scan on armis) because the bump wipes the whole fjall. A cheaper option exists (ROW_VERSION 2, no bump, the + sweep deletes the path-keyed rows) and has precedent in the keyspace's own doc comment (added "without a + schema bump"); the bump is kept because the plan asks for it and it removes any mixed-format window. +- **Wire.** `IndexReadQuery::GrepBloom { pattern, entries }` (`comms/index_read_proto.rs`). A client and daemon of + different builds fail to decode, and `IndexRoute::grep_skip` already degrades that to "sweep every file". +- **Layered note.** Rows are content-addressed and never shadowed: a layered read does `overlay.get(h)` then + `base.get(h)`; a stale base row for a dead hash is harmless because nothing asks for it. + +### A3: `IndexView` (standalone only) + +- `src/index/view.rs`: `enum Part` naming the 16 layered keyspaces (below), and + `trait IndexView: Send + Sync { get(Part, &[u8]); prefix(Part, &[u8]); range(Part, Bound, Bound); iter(Part); }` + returning `Box> + '_>`, plus the named accessors that exist today + on `IndexDb`: `references_to`, `definition_of`, `import_targets_of`, `importers_probing`, `is_probed`, + `import_probes_complete`, `symbols_by_name_lookup`, `symbols_index_is_empty`, `bm25_stats`, + `grep_bloom_row`, `grep_skip_verdicts`, and the two `NameDict` handles (`callee_names`, `trait_names`). +- `StandaloneView(&IndexDb)` delegates 1:1. `IndexRoute::Local` holds `Box`; `index_read_against`, + `query.rs`, `mcp/helpers_calls*.rs`, `mcp/helpers_impls.rs`, `search/{bm25,exact}.rs` and the resolve pass + (`intel/resolve_pass.rs`, which reads `import_targets`, `importers_by_probe`, `refs_*`) take `&dyn IndexView`. + Writes keep going through `IndexWriter` on the concrete `IndexDb`. +- The existing suite is the gate. Perf discipline applies: the dyn call is per iterator, not per row, but a + p95 regression above 3% on `references` / `callers` (benches in `benches/`) switches `IndexView` to an enum + with the two variants instead of a trait object. + +### A4: repo-group layout, base generations, migration + +**Config.** New `[index]` table in `config/v1.rs` (`IndexConfig`, `deny_unknown_fields`, in `ConfigV1` after +`cache`): `layered: bool`, default `false` until A8. Regenerate `schema/` with the existing +`config_schema` test. The env var `BASEMIND_INDEX_LAYERED=0|1` overrides the file (rollback lever, same pattern as +`BASEMIND_NO_SEED`). The mode is resolved once per `Store::open`; the daemon applies a changed value when it next +opens the workspace. Layered applies to a workspace only when all hold: `layered` is true, the canonical root is the +git workdir, `Repo::discover` succeeds, a base commit resolves, and no embedding lane is enabled until A7 has +shipped (`documents.embed`, `code_search.embed`: their Lance rows are still keyed by worktree). + +**Layout.** + +```text +cache/repos// + memory.fjall/ A0, untouched + group.json {format, main_root, common_dir, created_unix} + base.lock flock; serialises build / advance / retire / GC for the group + base/ + gen-/{index.fjall/, index.msgpack, base.json} published, never rewritten + .tmp--/ builder staging (incl. checkout/); swept after 1 h when unlocked + .retired-/ renamed out of selection, deleted once fjall can be locked + last-failure.json {error, unix}; a failed build is not retried for 10 min + overlays//{overlay.fjall/, delta.msgpack, overlay.json} + lance/ A7 +cache/workspaces// unchanged; `workspace.json` gains `repo_group` and `index_mode` +``` + +`WorkspaceMarker` (`store_layout.rs`) has no `deny_unknown_fields`, so the extra fields are invisible to older +binaries. `store_layout.rs` gains `base_dir`, `generation_dir`, `overlay_dir`, `BASE_LOCK_FILE` and friends; none of +`repos/` is ever listed by the budget eviction today and that changes in A6. + +**`base.json` (format 1, JSON, written last).** + +```json +{ "format": 1, "generation": 7, "commit": "", "tree": "", "source_ref": "origin/HEAD", + "built_from_generation": 6, "built_unix": 0, "basemind_version": "0.31.0", + "index_schema_ver": 0, "blob_schema_ver": 0, "min_extract_epoch": 1, + "scan_digest": "", "embed_policy": "", "files": 65123, "bytes": 0, "probes_complete": true } +``` + +A generation is *ready* iff `base.json` parses with `format == 1`. It is *compatible* iff `index_schema_ver == +INDEX_SCHEMA_VER`, `blob_schema_ver == extract::SCHEMA_VER`, `min_extract_epoch >= extract::EXTRACT_EPOCH` and +`scan_digest` equals the digest of the running config. `scan_digest` is the blake3 of the canonical JSON of the config +subtrees that change which rows a scan writes: `scan.{include,exclude,max_file_bytes}`, `languages`, `code_intel`, +`code_search.enabled`. `scan.extra_roots` is excluded: files outside the repository are always overlay-only. An +incompatible generation is ignored and collected, the same wipe-on-mismatch story as `INDEX_SCHEMA_VER` +(`schema-and-blob-compat` rule). + +**`index.msgpack` of a generation** is the existing `store::Index` (`files`, `doc_files` empty, `embed_policy`), +with every `FileEntry.mtime` normalised to 0. The base is code-tier only; the document tier stays in the overlay +until A7. + +**Base commit.** `refs/remotes/origin/HEAD` resolved with `Repo::resolve_rev`; if absent or unresolvable, `HEAD` of +the main worktree (`Repo::main_worktree_root`). basemind never fetches. + +**Builder (cold).** Holds `base.lock` (new `LockHolder::BaseBuild`; contention reports the holder from `.lock.meta`). + +1. Allocate `n = 1 + max(existing gen / retired numbers)`; create `.tmp-*`. +2. Export `tree(C)` to `.tmp-*/checkout/` via new `Repo::export_tree(rev, dest)` (regular files only; symlinks and + submodules are skipped, matching the scanner's filters). Peak disk 1-3 GB, deleted before publish. +3. Run `scanner::scan` with `ScanSource::WorkingTree` rooted at the checkout, `EmbedMode` off, into a private + `Store` whose view dir is `.tmp-*/`; this runs the wholesale resolve pass and sets `import_probe_ver`. +4. Normalise mtimes, rewrite `index.msgpack` atomically, flush every keyspace (the `rotate_memtable_and_wait` + loop in `IndexDb::open_inner`), close fjall, write `base.json` (tmp + rename inside `.tmp-*`), fsync the directory. +5. `rename(.tmp-*, gen-)` (atomic, same filesystem), then update `CURRENT` (tmp + rename, an advisory hint only). + +Readers never trust `CURRENT`; they list `base/gen-*` and read `base.json`. Blobs written by an aborted build stay +valid (content-addressed), so a restart re-runs mostly fjall writes. A disk-full or poisoned fjall (`IndexDb::is_poisoned`, +`fault.rs`, #79) aborts the build, removes `.tmp-*`, and writes `last-failure.json`. + +**Racing builders.** Two worktrees that both find no ready generation both try `base.lock` non-blocking. The loser +does not build: it re-checks `base/` every 2 s, answers queries from the `InRam` projection with the existing +partial-scan health notice naming the winner, and takes over (sweeping stale `.tmp-*`) if the flock is released +without a ready generation, which is what a crash of the winner looks like. At most one build per group is ever in +flight. + +**Bootstrap from an existing standalone index (the migration fast path).** If the group has no ready generation but +a workspace has a compatible standalone `views/working` (`store_seed::find_source` picks the source: main worktree +first, else the most files; its `.lock` must be acquirable by us, the same rule as `seed_working_view`), the builder +clones it (`store_seed::clone_dir`, CoW with a bounded plain copy) into `.tmp-*`, vets it (`vet_clone`), and corrects +it to `tree(C)` with the *advance* procedure below, treating every path where the source differs from `C` +(`Repo::status_against(C)` plus untracked plus basemind-only files, which are removed) as the diff. Cost is O(diff), +not a rescan, and the first base costs one CoW clone instead of 17-33 min. + +**Advance (new generation from an old one).** Under `base.lock`, for the newest ready generation `k` and target `m`: + +1. Quiesce `k`: if this process is the daemon it takes the write side of the group's `RwLock>`, + drains, and drops its handle; otherwise it tries to open `k`'s fjall and aborts the advance on `Locked`. A live + fjall is never copied (the rule in the `store_seed.rs` module doc). +2. Clone `gen-k` to `.tmp-*` and reopen the daemon handle immediately. +3. `Repo::diff_trees(C_k, C_m)` (new, gix tree diff; a rename is a delete plus an add) filtered through `Filters`. +4. Export only the added/modified blobs plus the unchanged files the resolve pass probes (A1's `probed_only` + expansion) to `.tmp-*/checkout/`, apply with `scan_paths_with_observer` through the normal writer, run the + incremental resolve pass, then steps 4-5 of the cold build. +If the filesystem has no CoW and free space is below twice the generation size, the advance is skipped and retried +later. + +**Selection.** An overlay pins `base_generation`. On open without a usable pin (first open, pin retired, +incompatible) choose among compatible ready generations the one with the smallest candidate count (`|K|` below, +computed once per generation, at most three git status calls), ties to the newest. A pin is kept while its +generation exists and is compatible. + +**Opening the base.** `IndexDb::open_code_only(gen_dir)`: like `open_tolerating_held_memory` with the memory +store always detached, because the base must not take the group's exclusive `memory.fjall` lock. The daemon's +`workspace_pool.rs` keeps one `Arc` per `(group, generation)`; every pooled workspace of the group shares +it and its block cache. A non-daemon process uses `open_index_with_retry`'s budget and then treats `Locked` like a +lost lock today: it routes through `IndexRoute::Daemon` if a daemon answers, else `InRam`. Open question 1. + +**Migration of a workspace (standalone to layered), one-way, under the workspace `.lock`.** + +1. Resolve or build a base (above). 2. Compute the overlay (A5, "Building the overlay") from the live tree. +2. Verify: for 200 random paths plus every path of `S_path`, compare the merged rows of each layered keyspace with + the standalone index (`IndexView` makes this one function); any difference aborts and keeps the standalone view. +3. Write `overlay.json` with `state = "ready"`, set `index_mode = "layered"` and `repo_group` in `workspace.json`, + rename `views/working` to `views/working.migrated`, remove it on the next successful open (so a crash between the + two leaves a restorable index). `store_seed::seed_working_view` is skipped for layered-eligible workspaces. + +### A5: overlay write path and the read merge + +**A5.0 (prerequisite, standalone fix).** BM25 postings are refcounted by content hash. `upsert_bm25_file` and +`remove_bm25_file` take a `still_referenced: &dyn Fn(&str /*hash*/) -> bool`, supplied from +`Store.index.files` hash counts, and skip the posting delete when another indexed path holds the hash. A regression +test (two paths, identical content, remove one, the other's postings survive) is red on `main` today. + +**`delta.msgpack` (format 1, MessagePack named, `rmp_serde`, same atomic tmp + rename as `Store::flush`).** + +```rust +struct Delta { + format: u16, // 1 + seq: u64, // monotonically increasing per commit + base_generation: u32, base_commit: String, + entries: BTreeMap, // S_path + restitched: BTreeSet, // S_refs minus S_path + doc_entries: BTreeMap, // the whole document tier until A7 + doc_tombstones: AHashMap, + embed_policy: String, +} +enum DeltaEntry { Added(FileEntry), Modified(FileEntry), Deleted } +``` + +`Added` is a path not in T; `Modified` is a path in T whose hash differs; `Deleted` is a path in T not in W. A path +whose live hash equals its base hash is never in the delta (a revert removes the entry and its overlay rows). A +version mismatch of `format` discards the overlay (it is a cache). Derived and never persisted, rebuilt at open in +O(delta): `S_path = keys(entries)`, `S_refs = S_path U restitched`, and +`D_hash = { base hash h of p in S_path } \ { h : base still has an unshadowed path with h } \ { hashes of Added/Modified }`, +computed from the per-hash path counts of H_base. + +**`overlay.json` (format 1).** `{ format, worktree_key, root, base_generation, base_commit, base_scan_digest, +index_schema_ver, head, state: "ready"|"dirty", delta_seq, changed_files, changed_bytes, updated_unix }`. + +**Crash safety.** A commit is: (1) set `state = "dirty"`; (2) one fjall batch containing the rows and `meta.delta_seq = +seq`; (3) write `delta.msgpack` with the same `seq` (the commit point, as `index.msgpack` is today); (4) `state = +"ready"`. On open, `dirty`, or `meta.delta_seq != delta.seq`, or a base pin that is missing or incompatible, discards +the overlay and rebuilds it. The overlay is a cache; blobs are global, so a rebuild costs O(diff), not O(repo). + +**Building the overlay (full pass).** `K`, the candidates, is `{ p in W : p not in T }` U `{ p in W : p in D }` U +`{ p in W : git neither tracks nor reports p }` (basemind-only files: gitignored but admitted by `scan.include`, and +`scan.extra_roots` files), where `D` is the set of tracked paths that differ from `tree(C_base)` in the index or the +worktree, from new `Repo::status_against(tree_sha)` (gix status with `head_tree`), whose untracked part is added to +`K`. Only `K` is read and hashed. Each `p in K` becomes `Added`, `Modified`, or nothing (equal hash); each `p in T \ W` is +`Deleted`. Then the rows of every `Added` / `Modified` path are written through the ordinary `IndexWriter` into the +overlay fjall, and the incremental resolve pass (A1) runs with reads from the merged view and writes to the overlay: +the importers it re-stitches form `restitched`. `Store.index.files` is the *effective* manifest (base overlaid with +the delta), so `store.lookup`, `MapCache::build`, the blob GC and every existing caller keep working; only +`Store::flush` differs and persists the delta, driven by a `dirty_paths` set, with a full diff only at open. The +stale purge in `scan_with_cancel` is replaced in layered mode by the `Deleted` computation above. + +**Watcher and `scan_paths`.** Events go through `scan_paths_with_observer` unchanged. For each path the post-state +is classified against T with the same four cases (revert, modify, add, delete) and the delta, overlay rows and +`restitched` are updated in one commit (the crash protocol above). The `.git` directory is already excluded; a +`HEAD` / index change is not a watcher event today, so a branch switch is picked up by the next full pass, which +the daemon schedules whenever `git rev-parse HEAD` of the worktree changed since `overlay.json.head`. + +**Concurrency.** The overlay writer is whoever holds the workspace `.lock` (`acquire_lock_as`), the same single +writer as today (the daemon when running). The base is read-only to everyone but the builder. A layered reader loads +`Arc` (the three sets) and opens its base and overlay iterators under a short read lock; the writer takes the +write lock only to commit the batch and swap the `Arc`, so a reader never sees a path in both layers or in +neither. Fjall iterators are snapshots, so the lock is not held while iterating. + +**Merge rule per partition (every keyspace opened by `IndexDb::open_inner`).** Merged order is plain byte order of the +raw key (both layers use the same encoders in `index/keys*.rs`); keys never collide across layers because a base row +is dropped exactly when the overlay may hold a row for the same shadow unit. + +| Keyspace | Key (shadow unit) | Overlay holds | Base row dropped when | Notes | +|---|---|---|---|---| +| `meta` | string keys | `delta_seq`, overlay-only `code_bm25_n`, `code_bm25_total_len` | never merged as rows | `schema_ver`: both must equal `INDEX_SCHEMA_VER`. `import_probe_ver`: effective value is the base's (a generation is published only with probes complete); the overlay never writes it. BM25 stats below. | +| `symbols_by_path` | `rel` + start | rows of `Added`/`Modified` | `rel in S_path` | | +| `symbols_by_name` | name + kind + `rel` + start | same | `rel in S_path` | read by `symbols_by_name_lookup`; ordered by name then kind then `rel`, so the merge interleaves layers inside one name | +| `calls_by_path` | `rel` + start | same | `rel in S_path` | | +| `calls_by_callee` | callee + `rel` + start | same | `rel in S_path` | backs `references` / `callers`; `callee_names` dictionary = base U overlay (a superset of names is already allowed by `name_dict.rs`) | +| `imports_by_path` | `rel` + module + start | same | `rel in S_path` | written, no reader today; rule kept so the equivalence test can compare it | +| `imports_by_module` | module + `rel` + start | same | `rel in S_path` | as above | +| `implementations_by_path` | `rel` + trait + type + start | same | `rel in S_path` | | +| `implementations_by_trait` | trait + type + `rel` + start | same | `rel in S_path` | `trait_names` dictionary = base U overlay | +| `refs_by_path` | use `rel` + start | edges whose use is in `S_refs` | `use in S_refs` | `definition_of` hits one layer per use path | +| `refs_by_def` | def `rel` + start + use `rel` + start | edges whose use is in `S_refs` (def anywhere) | `use in S_refs` | def-side changes are covered because every cross-file user of a changed def is an importer that probed it and is therefore in `S_refs` (A1's exactness, which the A5 gate must prove for barrels and tsconfig) | +| `import_targets` | importer + local_start (incl. reserved manifest row) | rows of importers in `S_refs` | `importer in S_refs` | | +| `importers_by_probe` | probe hash + importer | rows of importers in `S_refs` | `importer in S_refs` | | +| `code_bm25_postings` | term + `chunk_id` (`:`) | postings of hashes not in H_base | hash of `chunk_id` in `D_hash` | a hash still live through another path keeps its base postings and is not rewritten | +| `code_bm25_by_path` | `rel` + `chunk_id` | rows of `Added`/`Modified` | `rel in S_path` | forward map; also the source of the BM25 stats correction | +| `grep_bloom` | content hash | rows for hashes with no base row | never | union by key, overlay first (A2) | +| `embeddings` | reserved, never written | none | n/a | must stay empty in both layers; asserted by the gate | +| `memory_by_key`, `memory_archive`, `proposals` | not in base or overlay | none | n/a | live in `memory.fjall` (A0); base and overlay handles are opened detached | + +**Totals, caps and cursors.** The merged iterator yields exactly the `(key, value)` sequence a standalone index of +the same tree yields, filter applied before anything is counted. So `scan_calls_by_name` and `scan_impls_fjall` +(`scan_cap = max(limit * 8, 2000)`, `total_is_partial`, `has_more`) are unchanged, and a cursor is still the raw key +of the last hit as an exclusive lower bound: it is valid on both layers because each layer positions with `range`. +Cursors cross a rebase or generation change only if the merged sequence is identical, which is not guaranteed; a +cursor older than the last overlay commit is served best-effort (the existing behaviour when a standalone index is +rescanned between pages). BM25: `bm25_stats()` returns `N = N_base - |base by_path rows of S_path| + N_overlay` and +`total_len` likewise, with the subtraction computed from the shadowed base rows at open and on each `S_path` change; +`df(term)` is the count of merged postings rows. + +**Gate.** `tests/layered_equivalence.rs`: a proptest over three worktrees of one group with random edit scripts +(modify, add, delete, rename, duplicate content, barrel re-export, `tsconfig` paths change, Python package +`__init__`), each compared against a freshly built standalone index of the same tree for every `Part`, every named +accessor, every MCP mode that reads the index, with totals, `has_more` and cursor round trips. Behind +`[index] layered = false` until green on 1000 cases. + +### A6: generations, rebase, promotion, GC + +**Advance triggers** (evaluated by the daemon maintenance task at start, hourly, and after each overlay full pass; +by `basemind scan` when no daemon runs): `|diff_trees(C_cur, origin/HEAD)|` is at least 2% of `base.json.files`, or +the newest generation is older than 24 h and that diff is non-empty, or an overlay breaches the rebase threshold +while the newest generation is behind `origin/HEAD`. One advance at a time per group (`base.lock`). + +**Rebase and promotion** are evaluated after each overlay commit. Breach = `changed_files > 15%` of `base.json.files` +or `changed_bytes > 25%` of the base manifest's total `size_bytes`. + +- If a newer compatible generation exists whose candidate count is lower, **rebase**: build `overlays//next/` against + it from scratch (O(candidates); blobs are reused), then swap the directories with two renames while the old + overlay serves reads. A crash leaves a disposable `next/`. +- Otherwise, or if the rebased overlay still breaches 10% / 20%, **promote**: clone the pinned generation into + `views/working` (`store_seed` machinery, daemon-quiesced as in *Advance*), apply the overlay's delta through + `scan_paths` plus the incremental resolve pass, set `index_mode = "standalone"`, delete the overlay. A promoted + workspace is never re-layered automatically (no flapping); `BASEMIND_INDEX_LAYERED=1` plus `basemind cache reset + ` is the manual way back. Promotion that cannot quiesce the base is deferred and warned in `doctor`. + +**Liveness.** A generation is live iff it is the newest ready compatible one or an overlay in the group pins it. At +most three are live; when publishing a fourth, the overlays pinned to the oldest live generation are invalidated (their +overlay dir is deleted, they rebuild on next open). There is no eager rebase of idle worktrees. A non-live generation +is renamed to `.retired-` and removed when its fjall lock can be taken. An incompatible or half-published +generation (no `base.json`) is non-live. An overlay is live iff `workspaces//workspace.json` names an existing +root; orphans go with `reap_orphaned_workspaces_in`, which must also remove `repos//overlays/` +(group read from the marker), and an overlay older than its workspace's `index_mode` flip is removed. + +**Interaction with `store_gc*.rs`.** + +- *Blob liveness* (`collect_referenced_hashes`, `collect_referenced_hashes_global_in`, `LiveBlobs`): also union, per + group, the `index.msgpack` of every `base/gen-*` and `.tmp-*` whose flock is held, and every + `overlays/*/delta.msgpack` (hashes of `Added` / `Modified`; `doc_entries` into `docs`). A manifest that is + unreadable and not provably a foreign schema aborts the sweep (the existing safety rule); a foreign-schema + generation is deleted under `base.lock` instead of healed. The 6-hour `BLOB_GC_GRACE` covers builds in flight. +- *Budget* (`store_gc_budget::cache_footprint`, `eviction_candidates`, `rebuildable_workspace_paths`): the footprint + gains `repos/*/base` and `repos/*/overlays`; evicting a workspace also removes its overlay; a group's `base/` is a + candidate only after every workspace of the group is evicted or idle past the grace window, and never while any + of its overlays or `base.lock` is locked. `memory.fjall` stays untouched. +- *Protection* (`store_gc_protect.rs`): a workspace whose overlay pins generation `k` makes `k` part of its + protected footprint. +- *Locking*: destructive sweeps keep the machine-wide lock in `store_gc_live.rs`; per-group work takes `base.lock` + with a try-lock and skips a busy group. +- *Lance* rows: after A7, rows whose hash no live manifest references and that are older than the blob grace window. + +### A7: shared Lance tables + +- **Placement.** `repos//lance/-/`, one directory per `(embedding_model, dim)`, because + `LanceStore::open` wipes the whole directory on a `LanceMeta` mismatch (`lance/mod.rs`) and a worktree with a different + preset must not wipe its siblings; such a worktree stays standalone. `memory.lance` stays per workspace. A per-group + `lance.lock` flock serialises write transactions across CLI and daemon. +- **Schema.** `documents`, `doc_links` and `code_chunks` gain `content_hash` (Utf8). Row identity becomes + `(scope, content_hash, chunk_idx)`; `path` stops being part of identity. This is a column change, so it ships in a + `RELEASE_MINOR` bump (`MEMORY_SCHEMA_VER` rule), with wipe-on-mismatch as the migration: the vectors live in the + `.doc.msgpack` / `.chunk.msgpack` blobs, so the rebuild re-flushes rows from blobs with no model inference (as + `purge_unembedded_*` and the seed path already rely on). +- **Writes.** The scanner inserts rows for a hash only if none exist (`content_hash IN (batch)` probe), under + `lance.lock`. Rows are never deleted when a path changes; the A6 GC deletes unreferenced hashes. +- **Reads.** KNN returns the `k' = clamp(3k, k + 64, 4096)` nearest rows, drops rows whose hash is not in the + effective manifest's hash set, and expands each surviving hash to every live path holding it (the reverse map is + built at open), one hit per `(path, chunk)`. If fewer than `k` hits survive and `k'` was full, double `k'` up to + 4096 and stop. This also replaces taking the hit path from the blob (`mcp/helpers_code_search.rs` `hydrate_one`, which + reads `chunk.path` from the content-addressed sidecar and is wrong for duplicated content). Ties are ordered by + `(distance, path, chunk_idx)` in both modes. +- **Documented deviation.** ANN recall differs from a per-worktree table. Gate: top-10 overlap with a per-worktree + table at least 0.95 over 200 queries per tier on armis (`src/eval`); below that, raise `k'` or use exact search for the + shared table. +- **Doc tier.** Until this phase the overlay carries every document (`Delta.doc_entries`); after it, documents delta + like code. `scope` stays the normalised remote URL; it no longer separates worktrees. + +### A8: layered by default + +Preconditions: A5 gate green on 1000 cases, A6 GC merged, A7 shipped, measurement targets met. Then: default +`[index] layered = true`; delete `store_seed.rs`, `BASEMIND_NO_SEED`, `seed_working_view` and its callers in +`Store::open_with_holder`, and the lance-staleness sentinel `seed:rebuild-lance`; migration (A4) runs for every +standalone workspace on its next writer open. + +**Rollback.** Per workspace: `[index] layered = false` or `BASEMIND_INDEX_LAYERED=0` makes the next writer open +promote (A6 procedure) or, if the base is gone, cold-scan into `views/working`; a read-only open under `false` +with an overlay present ignores the overlay, reports "no index, run basemind scan" and writes nothing. Binary +downgrade: `repos//{base,overlays,lance}` are never listed by an older GC (it only knows `memory.fjall`) and +the extra `workspace.json` fields are ignored, so an older binary simply cold-scans a fresh standalone view; the +cost is one rescan and the stranded disk, reclaimable with `basemind cache clear`. No format here changes +`INDEX_SCHEMA_VER` or `extract::SCHEMA_VER`, so a downgrade never wipes a base. + +### Measurement plan + +On armis (89k files, 65k indexed), 1 worktree then 5, standalone vs layered, same machine, cold cache and warm: + +- `du -sh` of the base generation and of each overlay (target overlay < 50 MB with a typical 50-file diff); + `du` of `grep_bloom` before and after A2 (expect a drop proportional to duplicate content); +- time to first answer on a cold second worktree (target < 60 s vs 17-33 min) and the first-base build time; +- peak RSS of scan and of the daemon with 5 worktrees open (the base block cache is shared once); +- p50 / p95 latency for `references`, `callers`, `implementations`, `search_code` keyword and semantic, `code grep`; + a regression above 10% p95 on any read mode blocks A8; +- watcher latency when a widely imported file changes (the `restitched` set and the re-stitch time); +- grep bloom skip ratio and candidate counts on 20 fixed patterns before and after A2 (must not worsen); +- promotion / rebase frequency over a week of normal use. + +### Open questions for the owner + +1. **Base access without a daemon.** Fjall admits one process, so with `layered` on every CLI invocation in a group + contends for the base. The proposal above degrades like a lost lock today (`Daemon` route, else `InRam`). The + alternative is to make the daemon mandatory for layered workspaces (auto-spawn through `singleton_spawn.rs`, and + fall back to standalone if it cannot start). Which is acceptable for the A8 default? +2. **A2 partition-revision bump.** The bump forces one full re-index for every user. The no-bump variant (ROW_VERSION 2 plus the + sweep) avoids it. The ADR keeps the bump because #87 prescribes it; say if the cheaper variant is preferred. + ## Consequences Per-worktree disk and scan cost become proportional to the diff, and one block cache serves a whole group. @@ -51,7 +456,13 @@ The read path grows a merge layer that must preserve order, totals and cursors f correctness gate (a proptest over three worktrees with random edits, comparing every mode's results and cursors to a standalone full index) exists to catch a merge that does not. `store_seed.rs` is removed once layered mode is the default. Phases A0 (per-repo memory store) and A1 (probe index and O(affected) -re-stitch) shipped in 0.30.0; A2–A8 remain. +re-stitch) shipped in 0.30.0; A2–A8 remain, specified below. + +Costs accepted by this design: a layered workspace depends on the daemon for concurrent base access; the +first base of a group costs one cold build (or one CoW clone when a standalone index exists); migration is +one-way per workspace (rollback re-materialises a standalone index); the document and embedding tiers stay +per worktree until A7; and the overlay is treated as a disposable cache, so any doubt about its consistency +costs a rebuild proportional to the diff rather than a repair. ## Alternatives considered @@ -59,3 +470,12 @@ re-stitch) shipped in 0.30.0; A2–A8 remain. - **Deduplicate full copies.** Rejected: compaction breaks the sharing. - **One shared Fjall with path-prefixed keys.** Rejected: name scans multiply by the worktree count, and one lock covers the whole group. +- **A virtual git-tree filesystem for the resolver, to build the base without exporting.** Rejected: it + re-implements the resolver's filesystem surface (`intel/resolver.rs`, `resolve_pass.rs`) to save a transient + 1–3 GB export. +- **Eager rebase of idle worktrees when a generation is retired.** Rejected: it needs the worktree on disk + and its lock; invalidating the overlay and rebuilding it at next open is simpler and costs O(diff). +- **Repairing a suspect overlay in place.** Rejected: the overlay is a cache of the diff; wiping and + rebuilding it is cheaper to reason about than reconciling `delta.msgpack` against fjall. +- **Opening the base from every process.** Not possible: fjall's directory lock is exclusive, which is why + the daemon holds it (open question 1 covers the no-daemon case). From 762cec26efcf76fccebd3ad3761511876910798f Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Sun, 18 Oct 2026 17:27:39 +0200 Subject: [PATCH 02/10] feat(index): key grep_bloom rows by content hash Co-Authored-By: Claude Opus 5.5 --- src/comms/daemon_forward_handlers.rs | 4 +- src/comms/index_read_proto.rs | 5 +- src/index/grep_bloom.rs | 250 +++++++++++++++++++-------- src/index/mod.rs | 11 +- src/index/writer.rs | 20 ++- src/mcp/helpers_grep.rs | 67 +++++-- src/mcp/index_read.rs | 15 +- src/mcp/index_route.rs | 13 +- src/mcp/index_route_tests.rs | 43 +++-- src/mcp/l1_cache.rs | 5 + src/mcp/state.rs | 1 + src/scanner.rs | 18 ++ src/scanner_file.rs | 20 ++- src/scanner_index_batch.rs | 75 ++++---- tests/grep_bloom_scan.rs | 110 +++++++++--- 15 files changed, 469 insertions(+), 188 deletions(-) diff --git a/src/comms/daemon_forward_handlers.rs b/src/comms/daemon_forward_handlers.rs index 46e156d8..3035e601 100644 --- a/src/comms/daemon_forward_handlers.rs +++ b/src/comms/daemon_forward_handlers.rs @@ -220,12 +220,12 @@ pub(crate) fn index_read_via_pool( .ok_or_else(|| "the daemon holds this workspace without a fjall index".to_string())?; return crate::mcp::index_read::call_scan(&idx, name, *limit, cursor.as_deref()); } - if let IndexReadQuery::GrepBloom { pattern, paths } = query { + if let IndexReadQuery::GrepBloom { pattern, entries } = query { let (idx, store_root) = pool .with_workspace(root, |store| (store.index_db.clone(), store.root.clone())) .map_err(|error| error.to_string())?; let idx = idx.ok_or_else(|| "the daemon holds this workspace without a fjall index".to_string())?; - return crate::mcp::index_read::grep_skip(&idx, &store_root, pattern, paths); + return crate::mcp::index_read::grep_skip(&idx, &store_root, pattern, entries); } pool.with_workspace(root, |store| crate::mcp::index_read::index_read_against(store, query)) .map_err(|error| error.to_string())? diff --git a/src/comms/index_read_proto.rs b/src/comms/index_read_proto.rs index 8264dd8a..25c98b3e 100644 --- a/src/comms/index_read_proto.rs +++ b/src/comms/index_read_proto.rs @@ -50,8 +50,9 @@ pub enum IndexReadQuery { GrepBloom { /// The grep regex source. pattern: String, - /// Files to test, at most [`MAX_GREP_BLOOM_PATHS`]. - paths: Vec, + /// Files to test with the hash and stamp they were indexed under, at most + /// [`MAX_GREP_BLOOM_PATHS`]. + entries: Vec, }, /// The `implementations_by_trait` scan behind `implementations`. ImplScan { diff --git a/src/index/grep_bloom.rs b/src/index/grep_bloom.rs index 71cac5e6..18a62668 100644 --- a/src/index/grep_bloom.rs +++ b/src/index/grep_bloom.rs @@ -1,10 +1,11 @@ //! Per-file trigram bloom filters: the candidate PREFILTER behind `code grep` (ADR-0012). //! -//! Each indexed file gets one row in the `grep_bloom` keyspace (key = repo-relative path) holding a -//! small bloom filter over the file's byte trigrams plus the `(size, mtime_ns)` stamp the bloom was -//! built under. A query extracts the literals every match MUST contain, turns them into trigram -//! hashes and tests each row; a file the bloom rejects, whose stamp still matches the live file, -//! cannot contain a match and is never opened. A bloom can only say "definitely absent", so it +//! Each distinct file content gets one row in the `grep_bloom` keyspace (key = hex content hash, so +//! two paths with identical bytes share a row) holding a small bloom filter over the content's byte +//! trigrams. A query extracts the literals every match MUST contain, turns them into trigram hashes +//! and tests each row; a file whose indexed hash selects a rejecting bloom, and whose live +//! `(size, mtime_ns)` still equals the stamp it was indexed under, cannot contain a match and is +//! never opened. A bloom can only say "definitely absent", so it //! decides which files to READ, never what a match is: the real regex still runs on every survivor. //! //! Everything here errs towards "candidate": a missing, older-version or malformed row, a changed @@ -22,8 +23,8 @@ use crate::path::RelPath; /// Row layout revision (first byte). A row with any other value is treated as absent, so a future /// layout change needs no index wipe: old rows read as candidates until the file is next scanned. const ROW_VERSION: u8 = 1; -/// `version(1) | size(8) | mtime_ns(8)`, then the bit array. -const HEADER_BYTES: usize = 17; +/// `version(1)`, then the bit array. +const HEADER_BYTES: usize = 1; /// Bloom bytes per source byte is `1 / BYTES_DIVISOR` (ADR-0012 sizing: 1/8 keeps the candidate set /// within ~2x of the exact trigram answer on the armis corpus at 12.8% of corpus bytes; 1/16 was /// up to ~8x worse on selective needles). @@ -59,15 +60,12 @@ fn trigram(w: &[u8]) -> u32 { (u32::from(w[0]) << 16) | (u32::from(w[1]) << 8) | u32::from(w[2]) } -/// Build the stored row for a file whose bytes the caller already holds, stamped with the -/// `(size, mtime_ns)` it was read under. -pub fn build_row(bytes: &[u8], size: u64, mtime_ns: i64) -> Vec { +/// Build the stored row for content the caller already holds. +pub fn build_row(bytes: &[u8]) -> Vec { let bloom_bytes = (bytes.len() / BYTES_DIVISOR).clamp(MIN_BLOOM_BYTES, MAX_BLOOM_BYTES); let nbits = (bloom_bytes * 8) as u64; let mut row = vec![0u8; HEADER_BYTES + bloom_bytes]; row[0] = ROW_VERSION; - row[1..9].copy_from_slice(&size.to_le_bytes()); - row[9..17].copy_from_slice(&mtime_ns.to_le_bytes()); let bits = &mut row[HEADER_BYTES..]; for w in bytes.windows(3) { let mixed = mix(trigram(w)); @@ -79,31 +77,23 @@ pub fn build_row(bytes: &[u8], size: u64, mtime_ns: i64) -> Vec { row } -/// A decoded row: the stamp it was built under and its bit array. -struct Row<'a> { - size: u64, - mtime_ns: i64, - bits: &'a [u8], +fn parse_row(row: &[u8]) -> Option<&[u8]> { + (row.len() > HEADER_BYTES && row[0] == ROW_VERSION).then(|| &row[HEADER_BYTES..]) } -fn parse_row(row: &[u8]) -> Option> { - if row.len() <= HEADER_BYTES || row[0] != ROW_VERSION { - return None; - } - Some(Row { - size: u64::from_le_bytes(row[1..9].try_into().ok()?), - mtime_ns: i64::from_le_bytes(row[9..17].try_into().ok()?), - bits: &row[HEADER_BYTES..], - }) -} +pub use crate::scanner_file::mtime_nanos; -/// The `(size, mtime_ns)` stamp a stored row was built under, or `None` for an absent / foreign row. -pub fn row_stamp(row: &[u8]) -> Option<(u64, i64)> { - parse_row(row).map(|r| (r.size, r.mtime_ns)) +/// One file a grep may test: its path, the content hash it was indexed under, and the +/// `(size, mtime_ns)` stamp of that indexing. The stamp is what the live file is held to, so a file +/// edited since the scan is never skipped on the strength of its old content's bloom. +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct GrepEntry { + pub rel: RelPath, + pub hash_hex: String, + pub size_bytes: u64, + pub mtime_ns: i64, } -pub use crate::scanner_file::mtime_nanos; - /// What a regex requires of a file, as trigram hashes: every group must pass, and a group passes when /// ANY of its alternatives has ALL its trigrams in the bloom. #[derive(Debug)] @@ -166,45 +156,67 @@ impl Needle { } impl IndexDb { - /// Fetch the stored bloom row for `rel`, if any. - pub fn grep_bloom_row(&self, rel: &RelPath) -> Option { - self.grep_bloom.get(rel.as_bytes()).ok().flatten() + /// Fetch the stored bloom row for the content `hash_hex`, if any. + pub fn grep_bloom_row(&self, hash_hex: &str) -> Option { + self.grep_bloom.get(hash_hex.as_bytes()).ok().flatten() } - /// Delete `rel`'s row without touching the rest of its index entries: how a test models an index - /// built before the keyspace existed. + /// Delete the row for `hash_hex`: how a test models an index built before the keyspace existed. #[doc(hidden)] - pub fn drop_grep_bloom_row_for_test(&self, rel: &RelPath) { - self.grep_bloom.remove(rel.as_bytes()).expect("remove bloom row"); + pub fn drop_grep_bloom_row_for_test(&self, hash_hex: &str) { + self.grep_bloom.remove(hash_hex.as_bytes()).expect("remove bloom row"); + } + + /// Number of stored rows, one per distinct content. + #[doc(hidden)] + pub fn grep_bloom_row_count(&self) -> usize { + self.grep_bloom.iter().count() + } + + /// True when `hash_hex` has an intact row: nothing to (re)build. + pub fn grep_bloom_current(&self, hash_hex: &str) -> bool { + self.grep_bloom_row(hash_hex) + .is_some_and(|row| parse_row(&row).is_some()) } - /// True when `rel` has an intact row stamped `(size, mtime_ns)`: nothing to (re)build. - pub fn grep_bloom_current(&self, rel: &RelPath, size: u64, mtime_ns: i64) -> bool { - self.grep_bloom_row(rel) - .and_then(|row| row_stamp(&row)) - .is_some_and(|stamp| stamp == (size, mtime_ns)) + /// Delete every row whose hash is not in `live`, returning how many were dropped. Rows are shared + /// by every path holding the content, so only a hash no indexed file references may go. + pub fn sweep_grep_bloom(&self, live: &ahash::AHashSet<&str>) -> Result { + let mut writer = self.writer(); + let mut dropped = 0; + for guard in self.grep_bloom.iter() { + let (key, _) = guard.into_inner()?; + if std::str::from_utf8(&key).is_ok_and(|hash| live.contains(hash)) { + continue; + } + writer.remove_grep_bloom(&key); + dropped += 1; + } + writer.commit()?; + Ok(dropped) } - /// For each of `paths`, whether the file can be SKIPPED by a grep for `needle`: it has an intact - /// row, the row rejects the needle, and the live file still has the stamp the row was built - /// under. Rows are fetched and dropped one at a time, in parallel; nothing is held between files. - pub fn grep_skip_verdicts(&self, root: &Path, paths: &[&RelPath], needle: &Needle) -> Vec { - paths + /// For each of `entries`, whether the file can be SKIPPED by a grep for `needle`: its content has + /// an intact row, the row rejects the needle, and the live file still has the stamp the entry was + /// indexed under. Rows are fetched and dropped one at a time, in parallel; nothing is held + /// between files. + pub fn grep_skip_verdicts(&self, root: &Path, entries: &[GrepEntry], needle: &Needle) -> Vec { + entries .par_iter() - .map(|rel| { - let Some(row) = self.grep_bloom_row(rel) else { + .map(|entry| { + let Some(row) = self.grep_bloom_row(&entry.hash_hex) else { return false; }; - let Some(row) = parse_row(&row) else { + let Some(bits) = parse_row(&row) else { return false; }; - if needle.passes(row.bits) { + if needle.passes(bits) { return false; } - // The bloom describes the file as scanned; the sweep reads it as it is now. Only a - // file still carrying the scan-time stamp may be trusted to lack the literal. - std::fs::metadata(root.join(rel.to_path_buf())) - .is_ok_and(|m| m.len() == row.size && mtime_nanos(&m) == row.mtime_ns) + // The bloom describes the content as indexed; the sweep reads the file as it is now. + // Only a file still carrying its indexed stamp may be trusted to lack the literal. + std::fs::metadata(root.join(entry.rel.to_path_buf())) + .is_ok_and(|m| m.len() == entry.size_bytes && mtime_nanos(&m) == entry.mtime_ns) }) .collect() } @@ -215,13 +227,13 @@ mod tests { use super::*; fn row_for(text: &str) -> Vec { - build_row(text.as_bytes(), text.len() as u64, 7) + build_row(text.as_bytes()) } fn rejects(pattern: &str, text: &str) -> bool { let needle = Needle::compile(pattern).expect("usable needle"); let row = row_for(text); - !needle.passes(parse_row(&row).expect("row").bits) + !needle.passes(parse_row(&row).expect("row")) } #[test] @@ -249,14 +261,14 @@ mod tests { fn case_insensitive_literals_match_every_casing() { let needle = Needle::compile("(?i)needle").expect("expanded to case variants"); for text in ["xx NEEDLE xx", "xx needle xx", "xx NeEdLe xx"] { - assert!(needle.passes(parse_row(&row_for(text)).unwrap().bits), "{text}"); + assert!(needle.passes(parse_row(&row_for(text)).unwrap()), "{text}"); } } #[test] fn unicode_literals_are_matched_on_their_utf8_bytes() { let needle = Needle::compile("caf\u{e9}s").expect("literal"); - assert!(needle.passes(parse_row(&row_for("two caf\u{e9}s here")).unwrap().bits)); + assert!(needle.passes(parse_row(&row_for("two caf\u{e9}s here")).unwrap())); } #[test] @@ -269,18 +281,116 @@ mod tests { assert!(parse_row(&[]).is_none()); } - #[test] - fn the_stamp_round_trips() { - let row = build_row(b"abcdef", 6, -12); - assert_eq!(row_stamp(&row), Some((6, -12))); - } - #[test] fn row_size_tracks_the_file_within_its_bounds() { - assert_eq!(build_row(b"abc", 3, 0).len(), HEADER_BYTES + MIN_BLOOM_BYTES); + assert_eq!(build_row(b"abc").len(), HEADER_BYTES + MIN_BLOOM_BYTES); let big = vec![b'x'; 8 * 1024]; - assert_eq!(build_row(&big, 8192, 0).len(), HEADER_BYTES + 1024); + assert_eq!(build_row(&big).len(), HEADER_BYTES + 1024); let huge = vec![b'x'; 4 * 1024 * 1024]; - assert_eq!(build_row(&huge, 0, 0).len(), HEADER_BYTES + MAX_BLOOM_BYTES); + assert_eq!(build_row(&huge).len(), HEADER_BYTES + MAX_BLOOM_BYTES); + } + + fn open_db() -> (tempfile::TempDir, IndexDb) { + let dir = tempfile::tempdir().expect("tempdir"); + let db = IndexDb::open(dir.path()).expect("index"); + (dir, db) + } + + fn put(db: &IndexDb, hash_hex: &str, text: &str) { + let mut writer = db.writer(); + writer.upsert_grep_bloom(hash_hex, row_for(text)); + writer.commit().expect("commit"); + } + + #[test] + fn a_row_round_trips_under_its_content_hash() { + let (_dir, db) = open_db(); + put(&db, "aa11", "fn alpha() {}"); + assert_eq!( + db.grep_bloom_row("aa11").as_deref(), + Some(row_for("fn alpha() {}").as_slice()) + ); + assert!(db.grep_bloom_current("aa11")); + assert!(!db.grep_bloom_current("bb22")); + } + + #[test] + fn a_hash_that_prefixes_another_reads_only_its_own_row() { + let (_dir, db) = open_db(); + put(&db, "ab", "short key"); + put(&db, "abc", "longer key"); + assert_eq!( + db.grep_bloom_row("ab").as_deref(), + Some(row_for("short key").as_slice()) + ); + assert_eq!( + db.grep_bloom_row("abc").as_deref(), + Some(row_for("longer key").as_slice()) + ); + assert!(db.grep_bloom_row("a").is_none()); + } + + #[test] + fn the_sweep_drops_only_hashes_no_live_file_references() { + let (_dir, db) = open_db(); + for hash in ["h1", "h2", "h3"] { + put(&db, hash, hash); + } + let live: ahash::AHashSet<&str> = ["h1", "h3"].into_iter().collect(); + assert_eq!(db.sweep_grep_bloom(&live).expect("sweep"), 1); + assert!(db.grep_bloom_current("h1") && db.grep_bloom_current("h3")); + assert!(!db.grep_bloom_current("h2")); + assert_eq!( + db.sweep_grep_bloom(&live).expect("sweep"), + 0, + "a second sweep is a no-op" + ); + } + + #[test] + fn two_paths_with_the_same_content_share_one_row_and_both_skip() { + let (_dir, db) = open_db(); + let root = tempfile::tempdir().expect("root"); + let text = "fn unrelated() { println!(\"hello\"); }\n"; + let mut entries = Vec::new(); + for name in ["a.rs", "b.rs"] { + std::fs::write(root.path().join(name), text).expect("write"); + let meta = std::fs::metadata(root.path().join(name)).expect("stat"); + entries.push(GrepEntry { + rel: RelPath::from(name), + hash_hex: "same".to_owned(), + size_bytes: meta.len(), + mtime_ns: mtime_nanos(&meta), + }); + } + put(&db, "same", text); + put(&db, "same", text); + assert_eq!(db.grep_bloom_row_count(), 1); + let needle = Needle::compile("zzz_absent_literal").expect("needle"); + assert_eq!(db.grep_skip_verdicts(root.path(), &entries, &needle), vec![true, true]); + } + + #[test] + fn a_live_file_that_moved_off_its_indexed_stamp_is_a_candidate() { + let (_dir, db) = open_db(); + let root = tempfile::tempdir().expect("root"); + std::fs::write(root.path().join("a.rs"), "fn unrelated() {}\n").expect("write"); + let meta = std::fs::metadata(root.path().join("a.rs")).expect("stat"); + put(&db, "h", "fn unrelated() {}\n"); + let needle = Needle::compile("zzz_absent_literal").expect("needle"); + let entry = |size_bytes| GrepEntry { + rel: RelPath::from("a.rs"), + hash_hex: "h".to_owned(), + size_bytes, + mtime_ns: mtime_nanos(&meta), + }; + assert_eq!( + db.grep_skip_verdicts(root.path(), &[entry(meta.len())], &needle), + vec![true] + ); + assert_eq!( + db.grep_skip_verdicts(root.path(), &[entry(meta.len() + 1)], &needle), + vec![false] + ); } } diff --git a/src/index/mod.rs b/src/index/mod.rs index a1ce038d..fcd2e7f4 100644 --- a/src/index/mod.rs +++ b/src/index/mod.rs @@ -43,8 +43,9 @@ use thiserror::Error; /// into its meta keyspace at *create* time and recovers it on reopen, ignoring the create options /// for a keyspace that already exists, so an index built by an earlier build would keep the 64 MiB /// default forever unless the schema mismatch forces it to be recreated; `+6` the LZ4-at-every-level -/// block compression ([`index_keyspace_options`]), persisted at create time for the same reason. -const INDEX_PARTITION_REVISION: u32 = 6; +/// block compression ([`index_keyspace_options`]), persisted at create time for the same reason; +/// `+7` the `grep_bloom` re-key from path to content hash, whose old rows would read as foreign hashes. +const INDEX_PARTITION_REVISION: u32 = 7; /// Bumped whenever the on-disk key layout changes — the sum of `RELEASE_MINOR` and the /// [`INDEX_PARTITION_REVISION`] offset, monotonic across both. When `RELEASE_MINOR` next bumps, @@ -332,9 +333,9 @@ pub struct IndexDb { /// `code_bm25_by_path`: forward companion keyed by file → its chunks' `(chunk_id, doclen, /// terms)`, so a re-scan deletes the previous postings in O(prefix). Always created. pub(crate) code_bm25_by_path: Keyspace, - /// `grep_bloom`: file path -> per-file trigram bloom filter + the `(size, mtime_ns)` stamp it was - /// built under (see [`grep_bloom`]). A candidate PREFILTER for `code grep` only; a missing or - /// stale row means "candidate", which is why this partition was added without a schema bump and + /// `grep_bloom`: hex content hash -> trigram bloom filter of that content (see [`grep_bloom`]), + /// shared by every path holding the bytes. A candidate PREFILTER for `code grep` only; a missing + /// row means "candidate", which is why this partition was added without a schema bump and /// backfills on rescan. pub(crate) grep_bloom: Keyspace, #[allow(dead_code)] diff --git a/src/index/writer.rs b/src/index/writer.rs index ebfa019c..d46a2191 100644 --- a/src/index/writer.rs +++ b/src/index/writer.rs @@ -141,18 +141,22 @@ impl IndexWriter { /// Drop every index entry for `rel`. Used when a file is removed from the scan set. pub fn remove_file(&mut self, rel: &RelPath) -> Result<(), IndexError> { - self.stage_deletes_for(rel)?; + self.stage_deletes_for(rel) + } + + /// Store the grep bloom `row` (built by [`super::grep_bloom::build_row`] from the bytes the + /// scanner already holds) under the content hash `hash_hex`. Keyed by content alone, so the insert + /// overwrites an identical row and no delete is staged; the row outlives any one path and is + /// dropped by [`IndexDb::sweep_grep_bloom`]. Counted against the staged-byte budget like every + /// other entry. + pub fn upsert_grep_bloom(&mut self, hash_hex: &str, row: Vec) { let (db, batch, staged_bytes) = (&self.db, &mut self.batch, &mut self.staged_bytes); - stage_remove(batch, staged_bytes, &db.grep_bloom, rel.as_bytes().to_vec()); - Ok(()) + stage_insert(batch, staged_bytes, &db.grep_bloom, hash_hex.as_bytes().to_vec(), row); } - /// Replace `rel`'s grep bloom row with `row` (built by [`super::grep_bloom::build_row`] from the - /// bytes the scanner already holds). Keyed by path alone, so the insert overwrites the previous - /// row and no delete is staged. Counted against the staged-byte budget like every other entry. - pub fn upsert_grep_bloom(&mut self, rel: &RelPath, row: Vec) { + pub(super) fn remove_grep_bloom(&mut self, key: &[u8]) { let (db, batch, staged_bytes) = (&self.db, &mut self.batch, &mut self.staged_bytes); - stage_insert(batch, staged_bytes, &db.grep_bloom, rel.as_bytes().to_vec(), row); + stage_remove(batch, staged_bytes, &db.grep_bloom, key.to_vec()); } /// Replace the resolved-reference edges whose *use* is in `use_rel` with those derived from diff --git a/src/mcp/helpers_grep.rs b/src/mcp/helpers_grep.rs index 8f71129b..5f676fd2 100644 --- a/src/mcp/helpers_grep.rs +++ b/src/mcp/helpers_grep.rs @@ -16,6 +16,7 @@ use super::ServerState; use super::cursor::Cursor; use super::helpers::{SEARCH_LIMIT_DEFAULT, SEARCH_LIMIT_MAX}; use super::types::{GrepHit, GrepTruncation, WorkspaceGrepParams, WorkspaceGrepResponse}; +use crate::index::grep_bloom::GrepEntry; use crate::path::RelPath; /// Upper bound on the file content one `workspace_grep` call may read, summed from the sizes the @@ -97,12 +98,21 @@ pub(super) async fn run_workspace_grep( let root = state.shared.root.as_path(); let paths: Vec<&RelPath> = scanned.iter().map(|(path, _)| *path).collect(); + let entries: Vec = scanned + .iter() + .map(|(path, meta)| GrepEntry { + rel: (*path).clone(), + hash_hex: meta.hash_hex.to_string(), + size_bytes: meta.size_bytes, + mtime_ns: meta.mtime_ns, + }) + .collect(); // Files the per-file trigram blooms prove cannot match are never opened. They keep their slot in // `paths` (counted as zero), so window order, `file_idx` and every cursor are those of the full sweep. let skip = if bloom_prefilter_enabled() { super::index_route::IndexRoute::resolve(state) .await - .grep_skip(root, ¶ms.pattern, &paths) + .grep_skip(root, ¶ms.pattern, &entries) .await } else { None @@ -680,18 +690,33 @@ mod tests { // ---- trigram-bloom prefilter equivalence ------------------------------------------------- - /// Index the current bytes and stat of every readable file, as a scan would. - fn scan_blooms(db: &crate::index::IndexDb, dir: &Path, paths: &[RelPath]) { + /// Index the current bytes and stat of every readable file, as a scan would, returning one entry per + /// path (an unreadable file gets an entry no row answers to). + fn scan_blooms(db: &crate::index::IndexDb, dir: &Path, paths: &[RelPath]) -> Vec { let mut writer = db.writer(); + let mut entries = Vec::new(); for rel in paths { let abs = dir.join(rel.to_path_buf()); let (Ok(bytes), Ok(meta)) = (std::fs::read(&abs), std::fs::metadata(&abs)) else { + entries.push(GrepEntry { + rel: rel.clone(), + hash_hex: "unreadable".to_owned(), + size_bytes: 0, + mtime_ns: 0, + }); continue; }; - let row = crate::index::grep_bloom::build_row(&bytes, meta.len(), crate::scanner_file::mtime_nanos(&meta)); - writer.upsert_grep_bloom(rel, row); + let hash_hex = crate::hashing::hex(&crate::hashing::hash_bytes(&bytes)); + writer.upsert_grep_bloom(&hash_hex, crate::index::grep_bloom::build_row(&bytes)); + entries.push(GrepEntry { + rel: rel.clone(), + hash_hex, + size_bytes: meta.len(), + mtime_ns: crate::scanner_file::mtime_nanos(&meta), + }); } writer.commit().expect("commit blooms"); + entries } fn set_mtime_ahead(path: &Path, secs: u64) { @@ -715,7 +740,7 @@ mod tests { std::fs::write(dir.path().join("d2/unique.txt"), "alpha unique_token_qq beta\n").expect("write"); paths.push(RelPath::from("d2/unique.txt")); - scan_blooms(&db, dir.path(), &paths); + let mut entries = scan_blooms(&db, dir.path(), &paths); // After the scan: an edit that adds a token (size changes), a same-size edit with a bumped // mtime, a deletion, and a file the scan never saw. @@ -732,6 +757,12 @@ mod tests { std::fs::remove_file(dir.path().join("d1/f1.txt")).expect("delete"); std::fs::write(dir.path().join("d0/new.txt"), "brand new unique_token_qq needle\n").expect("new"); paths.push(RelPath::from("d0/new.txt")); + entries.push(GrepEntry { + rel: RelPath::from("d0/new.txt"), + hash_hex: "never-indexed".to_owned(), + size_bytes: 0, + mtime_ns: 0, + }); let refs: Vec<&RelPath> = paths.iter().collect(); let patterns = [ @@ -764,7 +795,7 @@ mod tests { let re = Regex::new(pattern).expect("regex"); let literal = (regex::escape(pattern) == pattern).then(|| Finder::new(pattern.as_bytes()).into_owned()); let skip = crate::index::grep_bloom::Needle::compile(pattern) - .map(|needle| db.grep_skip_verdicts(dir.path(), &refs, &needle)); + .map(|needle| db.grep_skip_verdicts(dir.path(), &entries, &needle)); skipped_somewhere |= skip.as_ref().is_some_and(|s| s.iter().any(|&x| x)); for skip_hits in [0u32, 2] { let filtered = count_all(dir.path(), &refs, skip.as_deref(), &re, literal.as_ref(), skip_hits); @@ -792,7 +823,7 @@ mod tests { // The stale and new files must stay candidates for the token they now hold. let needle = crate::index::grep_bloom::Needle::compile("unique_token_qq").expect("needle"); - let skip = db.grep_skip_verdicts(dir.path(), &refs, &needle); + let skip = db.grep_skip_verdicts(dir.path(), &entries, &needle); for (rel, skipped) in paths.iter().zip(&skip) { if ["d3/f3.txt", "d4/f4.txt", "d0/new.txt", "d2/unique.txt"].contains(&rel.to_string().as_str()) { assert!(!skipped, "{rel} holds the token and must not be skipped"); @@ -811,30 +842,40 @@ mod tests { let db = crate::index::IndexDb::open(idx_dir.path()).expect("index"); std::fs::write(dir.path().join("a.txt"), "nothing relevant here\n").expect("write"); let rel = RelPath::from("a.txt"); + let meta = std::fs::metadata(dir.path().join("a.txt")).expect("stat"); + let entry = GrepEntry { + rel: rel.clone(), + hash_hex: crate::hashing::hex(&crate::hashing::hash_bytes(b"nothing relevant here\n")), + size_bytes: meta.len(), + mtime_ns: crate::scanner_file::mtime_nanos(&meta), + }; let needle = crate::index::grep_bloom::Needle::compile("absent_literal").expect("needle"); assert_eq!( - db.grep_skip_verdicts(dir.path(), &[&rel], &needle), + db.grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), vec![false], "no row" ); let mut writer = db.writer(); - writer.upsert_grep_bloom(&rel, vec![99; 40]); + writer.upsert_grep_bloom(&entry.hash_hex, vec![99; 40]); writer.commit().expect("commit"); assert_eq!( - db.grep_skip_verdicts(dir.path(), &[&rel], &needle), + db.grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), vec![false], "foreign row" ); scan_blooms(&db, dir.path(), std::slice::from_ref(&rel)); assert_eq!( - db.grep_skip_verdicts(dir.path(), &[&rel], &needle), + db.grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), vec![true], "current row" ); let mut writer = db.writer(); writer.remove_file(&rel).expect("remove"); writer.commit().expect("commit"); - assert!(db.grep_bloom_row(&rel).is_none(), "removing the file removes its row"); + assert!( + db.grep_bloom_row(&entry.hash_hex).is_some(), + "the row is shared by content, so removing one path keeps it" + ); } #[test] diff --git a/src/mcp/index_read.rs b/src/mcp/index_read.rs index daab5a3c..94ef10d1 100644 --- a/src/mcp/index_read.rs +++ b/src/mcp/index_read.rs @@ -33,7 +33,7 @@ pub(crate) fn index_read_against(store: &Store, query: &IndexReadQuery) -> Resul match query { IndexReadQuery::CallScan { name, limit, cursor } => call_scan(idx, name, *limit, cursor.as_deref()), IndexReadQuery::CallsInFiles { paths } => calls_in_files(idx, paths), - IndexReadQuery::GrepBloom { pattern, paths } => grep_skip(idx, &store.root, pattern, paths), + IndexReadQuery::GrepBloom { pattern, entries } => grep_skip(idx, &store.root, pattern, entries), IndexReadQuery::ImplScan { trait_name, language, @@ -79,21 +79,22 @@ pub(crate) fn grep_skip( idx: &IndexDb, root: &std::path::Path, pattern: &str, - paths: &[crate::path::RelPath], + entries: &[crate::index::grep_bloom::GrepEntry], ) -> Result { use crate::comms::index_read_proto::MAX_GREP_BLOOM_PATHS; - if paths.len() > MAX_GREP_BLOOM_PATHS { + if entries.len() > MAX_GREP_BLOOM_PATHS { return Err(format!( "grep bloom names {} files; at most {MAX_GREP_BLOOM_PATHS} per request", - paths.len() + entries.len() )); } // No usable literal: nothing can be skipped, and saying so is the answer, not an error. let Some(needle) = crate::index::grep_bloom::Needle::compile(pattern) else { - return Ok(IndexReadResult::GrepSkip(vec![false; paths.len()])); + return Ok(IndexReadResult::GrepSkip(vec![false; entries.len()])); }; - let refs: Vec<&crate::path::RelPath> = paths.iter().collect(); - Ok(IndexReadResult::GrepSkip(idx.grep_skip_verdicts(root, &refs, &needle))) + Ok(IndexReadResult::GrepSkip( + idx.grep_skip_verdicts(root, entries, &needle), + )) } fn calls_in_files(idx: &IndexDb, paths: &[crate::path::RelPath]) -> Result { diff --git a/src/mcp/index_route.rs b/src/mcp/index_route.rs index 37a88ecb..6b8434aa 100644 --- a/src/mcp/index_route.rs +++ b/src/mcp/index_route.rs @@ -24,6 +24,7 @@ use super::MapCache; use super::helpers_calls_scan::{CallRef, CallScanPage, calls_in_file_fjall, scan_calls_by_name, scan_calls_in_ram}; use super::helpers_impls::{ImplScanPage, scan_impls_in_ram, scan_impls_session}; use crate::index::IndexDb; +use crate::index::grep_bloom::GrepEntry; use crate::path::RelPath; #[cfg(all(feature = "comms", any(unix, windows)))] @@ -112,28 +113,28 @@ impl IndexRoute { } } - /// Which of `paths` provably cannot match the grep `pattern` (`true` = skip), per the per-file + /// Which of `entries` provably cannot match the grep `pattern` (`true` = skip), per the per-file /// trigram blooms. `None` means "no prefilter available" (no reachable index, a forward that /// failed, an unusable reply): the caller then sweeps every file, which is always correct. pub(super) async fn grep_skip( &self, root: &std::path::Path, pattern: &str, - paths: &[&RelPath], + entries: &[GrepEntry], ) -> Option> { if let Self::Local(idx) = self { let needle = crate::index::grep_bloom::Needle::compile(pattern)?; - return Some(blocking(|| idx.grep_skip_verdicts(root, paths, &needle))); + return Some(blocking(|| idx.grep_skip_verdicts(root, entries, &needle))); } #[cfg(all(feature = "comms", any(unix, windows)))] { // Compiled here too so a pattern with no usable literal costs no round trip. crate::index::grep_bloom::Needle::compile(pattern)?; - let mut skip = Vec::with_capacity(paths.len()); - for chunk in paths.chunks(crate::comms::index_read_proto::MAX_GREP_BLOOM_PATHS) { + let mut skip = Vec::with_capacity(entries.len()); + for chunk in entries.chunks(crate::comms::index_read_proto::MAX_GREP_BLOOM_PATHS) { let query = IndexReadQuery::GrepBloom { pattern: pattern.to_owned(), - paths: chunk.iter().map(|&path| path.clone()).collect(), + entries: chunk.to_vec(), }; match self.forward(query).await { Some(Ok(IndexReadResult::GrepSkip(part))) if part.len() == chunk.len() => skip.extend(part), diff --git a/src/mcp/index_route_tests.rs b/src/mcp/index_route_tests.rs index e89b2b86..9b2bb6e6 100644 --- a/src/mcp/index_route_tests.rs +++ b/src/mcp/index_route_tests.rs @@ -18,6 +18,7 @@ use super::types_impls::FindImplementationsParams; use super::{HostBackend, MapCache}; use crate::comms::index_read_proto::{IndexReadQuery, IndexReadResult}; use crate::config::ConfigV1; +use crate::index::grep_bloom::GrepEntry; use crate::path::RelPath; use crate::scanner::{EmbedMode, ScanSource, scan}; use crate::store::{Store, VIEW_WORKING}; @@ -502,11 +503,27 @@ fn corpus_paths() -> Vec { paths } +fn corpus_entries(fx: &Fixture) -> Vec { + let store = fx.store.store.lock().expect("store"); + corpus_paths() + .into_iter() + .map(|rel| { + let entry = store.lookup(&rel).expect("indexed corpus file"); + GrepEntry { + rel, + hash_hex: entry.hash_hex.clone(), + size_bytes: entry.size_bytes, + mtime_ns: entry.mtime, + } + }) + .collect() +} + #[test] fn grep_bloom_verdicts_are_identical_on_the_local_and_forwarded_routes() { let fx = fixture(); let paths = corpus_paths(); - let refs: Vec<&RelPath> = paths.iter().collect(); + let entries = corpus_entries(&fx); for pattern in [ "driver_7\\(", "shared_helper_3", @@ -514,8 +531,8 @@ fn grep_bloom_verdicts_are_identical_on_the_local_and_forwarded_routes() { "Child\\(Tr1\\)", "zzz_not_in_corpus", ] { - let local = block_on(fx.local.grep_skip(&fx.root, pattern, &refs)).expect("local verdicts"); - let host = block_on(fx.host.grep_skip(&fx.root, pattern, &refs)).expect("forwarded verdicts"); + let local = block_on(fx.local.grep_skip(&fx.root, pattern, &entries)).expect("local verdicts"); + let host = block_on(fx.host.grep_skip(&fx.root, pattern, &entries)).expect("forwarded verdicts"); assert_eq!(local, host, "{pattern}: routes must agree"); assert_eq!(local.len(), paths.len()); let re = regex::Regex::new(pattern).expect("regex"); @@ -533,25 +550,29 @@ fn grep_bloom_verdicts_are_identical_on_the_local_and_forwarded_routes() { #[test] fn a_pattern_without_a_required_literal_or_a_failed_forward_means_no_prefilter() { let fx = fixture(); - let paths = corpus_paths(); - let refs: Vec<&RelPath> = paths.iter().collect(); - assert!(block_on(fx.local.grep_skip(&fx.root, r"\w+", &refs)).is_none()); - assert!(block_on(fx.host.grep_skip(&fx.root, ".*", &refs)).is_none()); - assert!(block_on(fx.broken_host.grep_skip(&fx.root, "driver_7", &refs)).is_none()); + let entries = corpus_entries(&fx); + assert!(block_on(fx.local.grep_skip(&fx.root, r"\w+", &entries)).is_none()); + assert!(block_on(fx.host.grep_skip(&fx.root, ".*", &entries)).is_none()); + assert!(block_on(fx.broken_host.grep_skip(&fx.root, "driver_7", &entries)).is_none()); } #[test] fn an_oversized_grep_bloom_batch_is_refused() { let fx = fixture(); - let paths: Vec = (0..=crate::comms::index_read_proto::MAX_GREP_BLOOM_PATHS) - .map(|i| RelPath::from(format!("pkg0/m{i}.rs").as_bytes())) + let entries: Vec = (0..=crate::comms::index_read_proto::MAX_GREP_BLOOM_PATHS) + .map(|i| GrepEntry { + rel: RelPath::from(format!("pkg0/m{i}.rs").as_bytes()), + hash_hex: String::new(), + size_bytes: 0, + mtime_ns: 0, + }) .collect(); let store = fx.store.store.lock().expect("store"); let error = super::index_read::index_read_against( &store, &IndexReadQuery::GrepBloom { pattern: "driver".into(), - paths, + entries, }, ) .expect_err("an unbounded wire batch must be rejected"); diff --git a/src/mcp/l1_cache.rs b/src/mcp/l1_cache.rs index 89df62d8..f9a9d10b 100644 --- a/src/mcp/l1_cache.rs +++ b/src/mcp/l1_cache.rs @@ -56,6 +56,9 @@ pub(crate) struct FileMeta { pub(crate) hash_hex: Box, pub(crate) language: Box, pub(crate) size_bytes: u64, + /// File mtime in nanoseconds as indexed (0 = unknown); with `size_bytes` the stamp grep holds the + /// live file to before trusting the content's bloom. + pub(crate) mtime_ns: i64, } /// Sorted view of every indexed file that has a readable L1 blob. @@ -154,6 +157,7 @@ impl From<&crate::store::FileEntry> for FileMeta { hash_hex: entry.hash_hex.as_str().into(), language: entry.language.as_str().into(), size_bytes: entry.size_bytes, + mtime_ns: entry.mtime, } } } @@ -164,6 +168,7 @@ impl Clone for FileMeta { hash_hex: self.hash_hex.clone(), language: self.language.clone(), size_bytes: self.size_bytes, + mtime_ns: self.mtime_ns, } } } diff --git a/src/mcp/state.rs b/src/mcp/state.rs index 1b9992c6..8a574761 100644 --- a/src/mcp/state.rs +++ b/src/mcp/state.rs @@ -367,6 +367,7 @@ impl MapCache { hash_hex: hash_hex.as_str().into(), language: map.language.as_str().into(), size_bytes: map.size_bytes, + mtime_ns: 0, }; l1.seed(&hash_hex, Arc::new(map)); metas.push((path, meta)); diff --git a/src/scanner.rs b/src/scanner.rs index 6776f88c..1fc2c361 100644 --- a/src/scanner.rs +++ b/src/scanner.rs @@ -573,6 +573,7 @@ pub fn scan_with_observer( observer.on_file(FileResult::bare(k.clone(), FileStatus::Removed)); report.stats.removed += 1; } + sweep_grep_bloom_rows(store); if let Some(change) = &policy { // Recorded only when the purge ran to completion (a panic leaves `applied` false too), so a @@ -645,6 +646,23 @@ pub fn scan_with_observer( Ok(report) } +/// Drop the grep bloom rows of every content no indexed file references any more. Rows are shared by +/// content, so a removed or edited file leaves its old row behind until this full-pass sweep. +fn sweep_grep_bloom_rows(store: &Store) { + let Some(idx) = store.index_db.as_ref() else { + return; + }; + let live: ahash::AHashSet<&str> = store + .index + .files + .values() + .map(|entry| entry.hash_hex.as_str()) + .collect(); + if let Err(error) = idx.sweep_grep_bloom(&live) { + tracing::warn!(%error, "grep bloom sweep failed; unreferenced rows stay until the next full scan"); + } +} + /// Move the pass's breadcrumb (and the per-phase memory tally) to `phase`, if one could be written at all. Folding the `Option` in /// here keeps every phase boundary a single line and keeps "no breadcrumb" from being a condition /// the scan branches on: evidence is never a precondition for the work. diff --git a/src/scanner_file.rs b/src/scanner_file.rs index 9f8b4c2e..9dbf76d3 100644 --- a/src/scanner_file.rs +++ b/src/scanner_file.rs @@ -262,7 +262,7 @@ fn process_file( { // Bytes were not read, so a missing grep bloom (an index that predates it) is backfilled // here rather than leaving the file a permanent grep candidate. - index_batch.backfill_grep_bloom(root, &RelPath::from(rel), meta.len(), mtime); + index_batch.backfill_grep_bloom(root, &RelPath::from(rel), &existing.hash_hex); return FileResult::bare(rel.to_string(), FileStatus::Unchanged); } } @@ -310,12 +310,20 @@ fn process_file( && sidecars_present && embed_state_satisfied(store, config, filters, rel, hash_hex_str, embed) { - // Content is unchanged but the stamp moved (touch, checkout): re-stamp the bloom from the - // bytes already in hand so grep keeps trusting it. + // Content is unchanged but the stamp moved (touch, checkout): refresh the entry's stamp, which + // grep holds the live file to, and make sure the content's bloom row exists. + let mut result = FileResult::bare(rel.to_string(), FileStatus::Unchanged); if matches!(source, ScanSource::WorkingTree) { - index_batch.restamp_grep_bloom(&RelPath::from(rel), &bytes, size_bytes, mtime); + index_batch.restamp_grep_bloom(hash_hex_str, &bytes); + if (existing.size_bytes, existing.mtime) != (size_bytes, mtime) { + result.upsert = Some(FileEntry { + size_bytes, + mtime, + ..existing.clone() + }); + } } - return FileResult::bare(rel.to_string(), FileStatus::Unchanged); + return result; } let want_l2 = filters.eager_l2 && store.index_db.is_some(); @@ -398,7 +406,7 @@ fn process_file( } // Grep reads the working tree, so only a working-tree scan describes what it will sweep. if matches!(source, ScanSource::WorkingTree) { - index_batch.stage_grep_bloom(&rel_path, &bytes, size_bytes, mtime); + index_batch.stage_grep_bloom(hash_hex_str, &bytes); } #[cfg(feature = "code-search")] diff --git a/src/scanner_index_batch.rs b/src/scanner_index_batch.rs index 2af44696..17ac82c2 100644 --- a/src/scanner_index_batch.rs +++ b/src/scanner_index_batch.rs @@ -206,44 +206,42 @@ impl<'a> WorkerIndexBatch<'a> { self.commit_if_full(); } - /// Stage the grep trigram bloom for a file whose bytes the scanner already holds, stamped with - /// the `(size, mtime_ns)` they were read under. No extra read: the bloom is built from the very - /// buffer the extractor parsed. Counts against the byte budget (rows are up to 1/8 of the file) - /// and may force a commit, exactly as [`Self::stage_bm25`] does. A `None` index is a no-op. - pub(crate) fn stage_grep_bloom(&mut self, rel: &RelPath, bytes: &[u8], size: u64, mtime_ns: i64) { + /// Stage the grep trigram bloom for the content `hash_hex`, whose bytes the scanner already + /// holds. No extra read: the bloom is built from the very buffer the extractor parsed. Counts + /// against the byte budget (rows are up to 1/8 of the file) and may force a commit, exactly as + /// [`Self::stage_bm25`] does. A `None` index is a no-op. + pub(crate) fn stage_grep_bloom(&mut self, hash_hex: &str, bytes: &[u8]) { let Some(index) = self.index else { return; }; - let row = crate::index::grep_bloom::build_row(bytes, size, mtime_ns); + let row = crate::index::grep_bloom::build_row(bytes); self.writer .get_or_insert_with(|| index.writer()) - .upsert_grep_bloom(rel, row); + .upsert_grep_bloom(hash_hex, row); self.commit_if_full(); } - /// [`Self::stage_grep_bloom`] unless the stored row already carries this exact stamp. For the + /// [`Self::stage_grep_bloom`] unless the content already has an intact row. For the /// content-unchanged-but-restamped path, which is re-entered on every scan until the store's own /// entry is refreshed and must not rewrite an identical row each time. - pub(crate) fn restamp_grep_bloom(&mut self, rel: &RelPath, bytes: &[u8], size: u64, mtime_ns: i64) { - if self - .index - .is_some_and(|index| index.grep_bloom_current(rel, size, mtime_ns)) - { + pub(crate) fn restamp_grep_bloom(&mut self, hash_hex: &str, bytes: &[u8]) { + if self.index.is_some_and(|index| index.grep_bloom_current(hash_hex)) { return; } - self.stage_grep_bloom(rel, bytes, size, mtime_ns); + self.stage_grep_bloom(hash_hex, bytes); } /// Backfill the bloom of a file the scan classed `Unchanged` (so its bytes were NOT read): an - /// index built before blooms existed, or one whose row went stale, gets its row on the next - /// rescan without a wipe. A current row costs one point lookup; only a missing/stale one costs - /// a read, once. A file that cannot be read is left rowless (a candidate, never skipped). - pub(crate) fn backfill_grep_bloom(&mut self, root: &std::path::Path, rel: &RelPath, size: u64, mtime_ns: i64) { + /// index built before blooms existed, or one whose row was swept, gets its row on the next + /// rescan without a wipe. A current row costs one point lookup; only a missing one costs a read, + /// once. A file that cannot be read, or whose bytes no longer hash to `hash_hex`, is left + /// rowless (a candidate, never skipped). + pub(crate) fn backfill_grep_bloom(&mut self, root: &std::path::Path, rel: &RelPath, hash_hex: &str) { use std::io::Read; let Some(index) = self.index else { return; }; - if index.grep_bloom_current(rel, size, mtime_ns) { + if index.grep_bloom_current(hash_hex) { return; } let Ok(file) = std::fs::File::open(root.join(rel.to_path_buf())) else { @@ -259,7 +257,10 @@ impl<'a> WorkerIndexBatch<'a> { if file.take(meta.len().saturating_add(1)).read_to_end(&mut bytes).is_err() { return; } - self.stage_grep_bloom(rel, &bytes, meta.len(), crate::scanner_file::mtime_nanos(&meta)); + if crate::hashing::hex(&crate::hashing::hash_bytes(&bytes)) != hash_hex { + return; + } + self.stage_grep_bloom(hash_hex, &bytes); } /// Report what the writer has staged since the last report to the ledger, and commit when @@ -727,31 +728,45 @@ mod tests { static LEDGER: StagedByteLedger = StagedByteLedger::new(u64::MAX); let (_index_dir, db) = fresh_index(); let dir = tempfile::tempdir().expect("tempdir"); - std::fs::write(dir.path().join("a.rs"), "fn needle_function() {}\n".repeat(40)).expect("write"); - let meta = std::fs::metadata(dir.path().join("a.rs")).expect("stat"); - let (size, mtime) = (meta.len(), crate::scanner_file::mtime_nanos(&meta)); + let body = "fn needle_function() {}\n".repeat(40); + std::fs::write(dir.path().join("a.rs"), &body).expect("write"); + let hash = crate::hashing::hex(&crate::hashing::hash_bytes(body.as_bytes())); let rel = RelPath::from("a.rs"); let mut batch = WorkerIndexBatch::with_budget(Some(&db), u64::MAX, &LEDGER); - batch.backfill_grep_bloom(dir.path(), &rel, size, mtime); + batch.backfill_grep_bloom(dir.path(), &rel, &hash); assert!( LEDGER.in_flight() > 0, "the staged bloom row must count against the byte ledger" ); batch.finish(); assert_eq!(LEDGER.in_flight(), 0); - assert!(db.grep_bloom_current(&rel, size, mtime), "backfill wrote the row"); + assert!(db.grep_bloom_current(&hash), "backfill wrote the row"); let mut again = WorkerIndexBatch::with_budget(Some(&db), u64::MAX, &LEDGER); - again.backfill_grep_bloom(dir.path(), &rel, size, mtime); - again.restamp_grep_bloom(&rel, b"ignored", size, mtime); + again.backfill_grep_bloom(dir.path(), &rel, &hash); + again.restamp_grep_bloom(&hash, b"ignored"); assert_eq!(LEDGER.in_flight(), 0, "a current row is neither rebuilt nor restaged"); again.finish(); + db.drop_grep_bloom_row_for_test(&hash); let mut moved = WorkerIndexBatch::with_budget(Some(&db), u64::MAX, &LEDGER); - moved.restamp_grep_bloom(&rel, b"fn needle_function() {}", size, mtime + 1); - assert!(LEDGER.in_flight() > 0, "a moved stamp re-stages the row"); + moved.restamp_grep_bloom(&hash, body.as_bytes()); + assert!(LEDGER.in_flight() > 0, "a missing row is re-staged"); moved.finish(); - assert!(db.grep_bloom_current(&rel, size, mtime + 1)); + assert!(db.grep_bloom_current(&hash)); + } + + /// The bytes on disk must hash to the indexed hash, or the row would describe other content. + #[test] + fn backfill_refuses_bytes_that_no_longer_match_the_indexed_hash() { + static LEDGER: StagedByteLedger = StagedByteLedger::new(u64::MAX); + let (_index_dir, db) = fresh_index(); + let dir = tempfile::tempdir().expect("tempdir"); + std::fs::write(dir.path().join("a.rs"), "fn edited() {}\n").expect("write"); + let mut batch = WorkerIndexBatch::with_budget(Some(&db), u64::MAX, &LEDGER); + batch.backfill_grep_bloom(dir.path(), &RelPath::from("a.rs"), "not-the-hash"); + batch.finish(); + assert_eq!(db.grep_bloom_row_count(), 0); } } diff --git a/tests/grep_bloom_scan.rs b/tests/grep_bloom_scan.rs index b9894c12..ab4f84dc 100644 --- a/tests/grep_bloom_scan.rs +++ b/tests/grep_bloom_scan.rs @@ -5,7 +5,6 @@ use std::path::Path; -use basemind::index::grep_bloom::row_stamp; use basemind::path::RelPath; use basemind::store::{Store, VIEW_WORKING}; @@ -33,9 +32,26 @@ fn scan(root: &Path) -> Store { }) } -fn stamp(store: &Store, rel: &str) -> Option<(u64, i64)> { - let row = store.index_db.as_ref()?.grep_bloom_row(&RelPath::from(rel))?; - row_stamp(&row) +fn hash_of(root: &Path, rel: &str) -> String { + basemind::hashing::hex(&basemind::hashing::hash_bytes( + &std::fs::read(root.join(rel)).expect("read"), + )) +} + +fn has_row(store: &Store, hash_hex: &str) -> bool { + store + .index_db + .as_ref() + .is_some_and(|idx| idx.grep_bloom_row(hash_hex).is_some()) +} + +fn row_count(store: &Store) -> usize { + store.index_db.as_ref().map_or(0, |idx| idx.grep_bloom_row_count()) +} + +fn indexed_stamp(store: &Store, rel: &str) -> (u64, i64) { + let entry = store.lookup(RelPath::from(rel)).expect("indexed file"); + (entry.size_bytes, entry.mtime) } fn live_stamp(root: &Path, rel: &str) -> (u64, i64) { @@ -44,37 +60,80 @@ fn live_stamp(root: &Path, rel: &str) -> (u64, i64) { } #[test] -fn scan_writes_refreshes_and_removes_bloom_rows() { +fn scan_writes_refreshes_and_sweeps_bloom_rows() { basemind::store::init_isolated_cache(); let dir = tempfile::tempdir().expect("tempdir"); let root = dir.path(); std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); std::fs::write(root.join("b.rs"), "pub fn beta_function_name() -> u32 { 2 }\n").unwrap(); + let (a_old, b_old) = (hash_of(root, "a.rs"), hash_of(root, "b.rs")); let store = scan(root); - assert_eq!( - stamp(&store, "a.rs"), - Some(live_stamp(root, "a.rs")), - "scan stamps the row" + assert!( + has_row(&store, &a_old) && has_row(&store, &b_old), + "scan writes a row per content" ); - assert_eq!(stamp(&store, "b.rs"), Some(live_stamp(root, "b.rs"))); + assert_eq!(row_count(&store), 2); drop(store); - // Edit: the row follows the new bytes. std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 100_000 }\n").unwrap(); + let a_new = hash_of(root, "a.rs"); let store = scan(root); - assert_eq!( - stamp(&store, "a.rs"), - Some(live_stamp(root, "a.rs")), - "edit refreshes the row" - ); + assert!(has_row(&store, &a_new), "edit writes the new content's row"); + assert!(!has_row(&store, &a_old), "the superseded content's row is swept"); drop(store); - // Delete: the row goes with the file. std::fs::remove_file(root.join("b.rs")).unwrap(); let store = scan(root); - assert_eq!(stamp(&store, "b.rs"), None, "a removed file leaves no row"); - assert!(stamp(&store, "a.rs").is_some()); + assert!(!has_row(&store, &b_old), "a removed file's row is swept"); + assert!(has_row(&store, &a_new)); + assert_eq!(row_count(&store), 1); +} + +#[test] +fn identical_files_share_one_row_which_outlives_either_path() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + let text = "pub fn shared_function_name() -> u32 { 1 }\n"; + std::fs::write(root.join("a.rs"), text).unwrap(); + std::fs::write(root.join("b.rs"), text).unwrap(); + let hash = hash_of(root, "a.rs"); + + let store = scan(root); + assert_eq!(row_count(&store), 1, "two paths, one content, one row"); + drop(store); + + std::fs::remove_file(root.join("b.rs")).unwrap(); + let store = scan(root); + assert!(has_row(&store, &hash), "a.rs still references the content"); + drop(store); + + std::fs::remove_file(root.join("a.rs")).unwrap(); + let store = scan(root); + assert_eq!(row_count(&store), 0, "no file references the content any more"); +} + +#[test] +fn a_touched_file_keeps_its_indexed_stamp_current() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); + let store = scan(root); + assert_eq!(indexed_stamp(&store, "a.rs"), live_stamp(root, "a.rs")); + drop(store); + + let file = std::fs::OpenOptions::new().write(true).open(root.join("a.rs")).unwrap(); + file.set_modified(std::time::SystemTime::now() + std::time::Duration::from_secs(60)) + .unwrap(); + drop(file); + let store = scan(root); + assert_eq!( + indexed_stamp(&store, "a.rs"), + live_stamp(root, "a.rs"), + "same content, moved stamp: the entry follows so grep keeps trusting the row" + ); } #[test] @@ -83,19 +142,14 @@ fn rescan_backfills_a_row_an_older_index_lacks() { let dir = tempfile::tempdir().expect("tempdir"); let root = dir.path(); std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); + let hash = hash_of(root, "a.rs"); let store = scan(root); - // Simulate an index built before the keyspace existed: the file is indexed, its bloom is gone. let idx = store.index_db.clone().expect("index"); - idx.drop_grep_bloom_row_for_test(&RelPath::from("a.rs")); - assert_eq!(stamp(&store, "a.rs"), None); + idx.drop_grep_bloom_row_for_test(&hash); + assert!(!has_row(&store, &hash)); drop(idx); drop(store); - // Nothing changed on disk, so the scan classes the file Unchanged -- and must still backfill. let store = scan(root); - assert_eq!( - stamp(&store, "a.rs"), - Some(live_stamp(root, "a.rs")), - "rescan backfills the row" - ); + assert!(has_row(&store, &hash), "rescan backfills the row"); } From 7c34d2981c2746fec52c1d3c3d0260db7ecc1d39 Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Mon, 19 Oct 2026 07:33:33 +0200 Subject: [PATCH 03/10] refactor(index): route index reads through an IndexView trait Co-Authored-By: Claude Opus 5.5 --- src/comms/daemon_forward_handlers.rs | 10 +- src/index/grep_bloom.rs | 46 +-- src/index/mod.rs | 39 +-- src/index/name_dict.rs | 11 +- src/index/view.rs | 445 +++++++++++++++++++++++++++ src/index/view_tests.rs | 104 +++++++ src/intel/resolve_pass.rs | 15 +- src/intel/stage_budget.rs | 7 +- src/mcp/codegraph.rs | 6 +- src/mcp/codegraph_tests.rs | 39 ++- src/mcp/helpers_archmap.rs | 14 +- src/mcp/helpers_calls.rs | 12 +- src/mcp/helpers_calls_scan.rs | 19 +- src/mcp/helpers_calls_scan_tests.rs | 56 ++-- src/mcp/helpers_code_search.rs | 6 +- src/mcp/helpers_community.rs | 4 +- src/mcp/helpers_graph.rs | 6 +- src/mcp/helpers_graphview.rs | 12 +- src/mcp/helpers_impls.rs | 16 +- src/mcp/helpers_traverse.rs | 10 +- src/mcp/index_read.rs | 11 +- src/mcp/index_route.rs | 12 +- src/mcp/index_route_tests.rs | 3 +- src/mcp/index_view_tests.rs | 161 ++++++++++ src/mcp/mod.rs | 2 + src/mcp/shared_state.rs | 8 +- src/mcp/tools_graph.rs | 4 +- src/query.rs | 5 +- src/scanner.rs | 7 +- src/scanner_index_batch.rs | 64 ++-- src/search/bm25.rs | 9 +- src/search/exact.rs | 4 +- src/store.rs | 9 +- tests/scan_smoke.rs | 2 +- 34 files changed, 982 insertions(+), 196 deletions(-) create mode 100644 src/index/view.rs create mode 100644 src/index/view_tests.rs create mode 100644 src/mcp/index_view_tests.rs diff --git a/src/comms/daemon_forward_handlers.rs b/src/comms/daemon_forward_handlers.rs index 3035e601..17f61bb6 100644 --- a/src/comms/daemon_forward_handlers.rs +++ b/src/comms/daemon_forward_handlers.rs @@ -218,14 +218,14 @@ pub(crate) fn index_read_via_pool( .with_workspace(root, |store| store.index_db.clone()) .map_err(|error| error.to_string())? .ok_or_else(|| "the daemon holds this workspace without a fjall index".to_string())?; - return crate::mcp::index_read::call_scan(&idx, name, *limit, cursor.as_deref()); + return crate::mcp::index_read::call_scan(&idx.view(), name, *limit, cursor.as_deref()); } if let IndexReadQuery::GrepBloom { pattern, entries } = query { let (idx, store_root) = pool .with_workspace(root, |store| (store.index_db.clone(), store.root.clone())) .map_err(|error| error.to_string())?; let idx = idx.ok_or_else(|| "the daemon holds this workspace without a fjall index".to_string())?; - return crate::mcp::index_read::grep_skip(&idx, &store_root, pattern, entries); + return crate::mcp::index_read::grep_skip(&idx.view(), &store_root, pattern, entries); } pool.with_workspace(root, |store| crate::mcp::index_read::index_read_against(store, query)) .map_err(|error| error.to_string())? @@ -270,19 +270,19 @@ pub(crate) fn code_search_lanes_against( use crate::comms::code_search_proto::CodeSearchLaneResult; let limit = (query.limit as usize).min(MAX_FORWARDED_LANE_LIMIT); - let Some(db) = store.index_db.as_ref() else { + let Some(db) = store.index_view() else { return Err( "the daemon holds this workspace without a fjall index, so neither the keyword nor the \ exact lane can be read" .to_string(), ); }; - let keyword = crate::search::bm25::bm25_search(db, &query.query, limit) + let keyword = crate::search::bm25::bm25_search(&db, &query.query, limit) .into_iter() .map(|hit| (hit.chunk_id, hit.score)) .collect(); let exact = if query.want_exact { - crate::search::exact::exact_lane_chunk_ids(store, db, &query.query, limit) + crate::search::exact::exact_lane_chunk_ids(store, &db, &query.query, limit) } else { Vec::new() }; diff --git a/src/index/grep_bloom.rs b/src/index/grep_bloom.rs index 18a62668..7920e074 100644 --- a/src/index/grep_bloom.rs +++ b/src/index/grep_bloom.rs @@ -201,27 +201,37 @@ impl IndexDb { /// indexed under. Rows are fetched and dropped one at a time, in parallel; nothing is held /// between files. pub fn grep_skip_verdicts(&self, root: &Path, entries: &[GrepEntry], needle: &Needle) -> Vec { - entries - .par_iter() - .map(|entry| { - let Some(row) = self.grep_bloom_row(&entry.hash_hex) else { - return false; - }; - let Some(bits) = parse_row(&row) else { - return false; - }; - if needle.passes(bits) { - return false; - } - // The bloom describes the content as indexed; the sweep reads the file as it is now. - // Only a file still carrying its indexed stamp may be trusted to lack the literal. - std::fs::metadata(root.join(entry.rel.to_path_buf())) - .is_ok_and(|m| m.len() == entry.size_bytes && mtime_nanos(&m) == entry.mtime_ns) - }) - .collect() + skip_verdicts(|hash| self.grep_bloom_row(hash), root, entries, needle) } } +/// [`IndexDb::grep_skip_verdicts`] over any row source, so a layered view shares the verdict logic. +pub(crate) fn skip_verdicts( + row_of: impl Fn(&str) -> Option + Sync, + root: &Path, + entries: &[GrepEntry], + needle: &Needle, +) -> Vec { + entries + .par_iter() + .map(|entry| { + let Some(row) = row_of(&entry.hash_hex) else { + return false; + }; + let Some(bits) = parse_row(&row) else { + return false; + }; + if needle.passes(bits) { + return false; + } + // The bloom describes the content as indexed; the sweep reads the file as it is now. + // Only a file still carrying its indexed stamp may be trusted to lack the literal. + std::fs::metadata(root.join(entry.rel.to_path_buf())) + .is_ok_and(|m| m.len() == entry.size_bytes && mtime_nanos(&m) == entry.mtime_ns) + }) + .collect() +} + #[cfg(test)] mod tests { use super::*; diff --git a/src/index/mod.rs b/src/index/mod.rs index fcd2e7f4..17220f2b 100644 --- a/src/index/mod.rs +++ b/src/index/mod.rs @@ -24,6 +24,9 @@ pub mod keys_governance; pub mod keys_probe; pub(crate) mod memory_store; pub(crate) mod name_dict; +pub mod view; +#[cfg(test)] +mod view_tests; pub mod writer; use std::path::{Path, PathBuf}; @@ -289,55 +292,55 @@ pub struct IndexDb { pub(crate) db: Database, /// Carries the `schema_ver` row read + stamped in [`IndexDb::open`]; also read and written /// by the BM25 corpus-stats recompute. - pub(crate) meta: Keyspace, - pub(crate) symbols_by_path: Keyspace, + pub(in crate::index) meta: Keyspace, + pub(in crate::index) symbols_by_path: Keyspace, /// Reserved fast-path partition: written on every upsert so that future name-based /// symbol search can skip the in-RAM linear scan. Not yet read by any MCP query path; /// kept to avoid a schema migration when the read path lands. - pub(crate) symbols_by_name: Keyspace, - pub(crate) calls_by_path: Keyspace, - pub(crate) calls_by_callee: Keyspace, + pub(in crate::index) symbols_by_name: Keyspace, + pub(in crate::index) calls_by_path: Keyspace, + pub(in crate::index) calls_by_callee: Keyspace, /// Distinct callee names of `calls_by_callee`; lets the substring name scan range-scan only the /// matching names instead of walking the partition. - pub(crate) callee_names: std::sync::Arc, + pub(in crate::index) callee_names: std::sync::Arc, /// Reserved fast-path partition: written on every upsert so that future /// `dependents`-by-module queries can use a prefix scan instead of iterating the /// full import set. Not yet read by any MCP query path; kept to avoid a schema /// migration when the read path lands. - pub(crate) imports_by_module: Keyspace, - pub(crate) imports_by_path: Keyspace, + pub(in crate::index) imports_by_module: Keyspace, + pub(in crate::index) imports_by_path: Keyspace, /// `implementations_by_trait`: prefix scans on trait name — backs `find_implementations`. - pub(crate) implementations_by_trait: Keyspace, + pub(in crate::index) implementations_by_trait: Keyspace, /// Distinct trait names of `implementations_by_trait`, as [`Self::callee_names`] is for callees. - pub(crate) trait_names: std::sync::Arc, + pub(in crate::index) trait_names: std::sync::Arc, /// `implementations_by_path`: companion to keep the per-file delete on upsert O(prefix). - pub(crate) implementations_by_path: Keyspace, + pub(in crate::index) implementations_by_path: Keyspace, /// `refs_by_def`: scope/import-resolved reference edges keyed by defining site — backs the /// resolved `find_references` / `find_callers`. Written by the scanner's resolve pass (B3). - pub(crate) refs_by_def: Keyspace, + pub(in crate::index) refs_by_def: Keyspace, /// `refs_by_path`: companion keyed by the use file — O(prefix) delete on re-resolve and the /// forward lookup behind `goto_definition`. Written by the scanner's resolve pass (B3). - pub(crate) refs_by_path: Keyspace, + pub(in crate::index) refs_by_path: Keyspace, /// `import_targets`: importer + import-binding offset -> the definition the stitch bound it to, /// plus one reserved probe-manifest row per importer (see [`keys_probe`]). - pub(crate) import_targets: Keyspace, + pub(in crate::index) import_targets: Keyspace, /// `importers_by_probe`: probed path hash + importer. Every path the resolver consulted for an /// importer's imports, so the incremental re-stitch finds exactly the importers a changed, /// added or deleted path can affect. - pub(crate) importers_by_probe: Keyspace, + pub(in crate::index) importers_by_probe: Keyspace, /// `code_bm25_postings`: term → chunks (with inlined tf + doclen) for the code-search BM25 /// keyword lane. Always created for DB stability; read + written only under the `code-search` /// feature (hence `dead_code`-allowed on a default build, mirroring `embeddings`). #[allow(dead_code)] - pub(crate) code_bm25_postings: Keyspace, + pub(in crate::index) code_bm25_postings: Keyspace, /// `code_bm25_by_path`: forward companion keyed by file → its chunks' `(chunk_id, doclen, /// terms)`, so a re-scan deletes the previous postings in O(prefix). Always created. - pub(crate) code_bm25_by_path: Keyspace, + pub(in crate::index) code_bm25_by_path: Keyspace, /// `grep_bloom`: hex content hash -> trigram bloom filter of that content (see [`grep_bloom`]), /// shared by every path holding the bytes. A candidate PREFILTER for `code grep` only; a missing /// row means "candidate", which is why this partition was added without a schema bump and /// backfills on rescan. - pub(crate) grep_bloom: Keyspace, + pub(in crate::index) grep_bloom: Keyspace, #[allow(dead_code)] pub(crate) embeddings: Keyspace, /// `memory_by_key`: scope + key → msgpack `MemoryRecord`. Lives in the per-repo-group diff --git a/src/index/name_dict.rs b/src/index/name_dict.rs index 17f49a61..2a8761e5 100644 --- a/src/index/name_dict.rs +++ b/src/index/name_dict.rs @@ -345,7 +345,7 @@ pub(crate) fn scan_prefixes( keyspace: &Keyspace, prefixes: Vec>, cursor_after: Option>, -) -> impl Iterator + '_ { +) -> impl Iterator + '_ { prefixes.into_iter().flat_map(move |prefix| { let lower = match &cursor_after { // The cursor lies beyond this name's whole range. @@ -361,6 +361,7 @@ pub(crate) fn scan_prefixes( .map(|lower| keyspace.range::, _>((lower, upper))) .into_iter() .flatten() + .map(super::view::kv) }) } @@ -373,9 +374,13 @@ pub(crate) fn name_ordered_keys<'a>( needle: &str, cursor_after: Option<&[u8]>, lower: Bound>, -) -> Box + 'a> { +) -> super::view::KvIter<'a> { match dict.matching_prefixes(keyspace, needle) { Some(prefixes) => Box::new(scan_prefixes(keyspace, prefixes, cursor_after.map(<[u8]>::to_vec))), - None => Box::new(keyspace.range::, _>((lower, Bound::Unbounded))), + None => Box::new( + keyspace + .range::, _>((lower, Bound::Unbounded)) + .map(super::view::kv), + ), } } diff --git a/src/index/view.rs b/src/index/view.rs new file mode 100644 index 00000000..e78a59bc --- /dev/null +++ b/src/index/view.rs @@ -0,0 +1,445 @@ +//! Read-only view over the fjall index. +//! +//! Every read of the index goes through [`IndexView`] so a later phase can answer from a shared base +//! index plus a per-worktree overlay without touching a call site (ADR-0016, phase A3). Writers keep +//! using [`IndexWriter`](super::writer::IndexWriter) on the concrete [`IndexDb`]. +//! +//! The trait has two layers. The generic layer (`get` / `prefix` / `range` / `iter`, addressed by +//! [`Part`]) is what a merging implementation has to provide; the named accessors are the lookups the +//! read path performs, each delegating 1:1 to the matching [`IndexDb`] method in [`StandaloneView`]. + +use std::borrow::Borrow; +use std::ops::Bound; +use std::path::Path; + +use fjall::{Keyspace, Slice}; + +use super::grep_bloom::{GrepEntry, Needle}; +use super::{IndexDb, IndexError, name_dict}; +use crate::extract::SymbolKind; +use crate::path::RelPath; + +/// One `(key, value)` row, or the error that interrupted the walk. +pub type KvResult = Result<(Slice, Slice), IndexError>; + +/// Boxed row iterator. One virtual call per row; the boxing itself happens once per scan. +pub type KvIter<'a> = Box + 'a>; + +/// The keyspaces a layered index merges. The memory, proposal and embedding keyspaces are not here: +/// they are per-repo-group or per-worktree state and never layered. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub enum Part { + Meta, + SymbolsByPath, + SymbolsByName, + CallsByPath, + CallsByCallee, + ImportsByModule, + ImportsByPath, + ImplementationsByTrait, + ImplementationsByPath, + RefsByDef, + RefsByPath, + ImportTargets, + ImportersByProbe, + CodeBm25Postings, + CodeBm25ByPath, + GrepBloom, +} + +impl Part { + /// Every layered keyspace, in declaration order. + pub const ALL: [Part; 16] = [ + Part::Meta, + Part::SymbolsByPath, + Part::SymbolsByName, + Part::CallsByPath, + Part::CallsByCallee, + Part::ImportsByModule, + Part::ImportsByPath, + Part::ImplementationsByTrait, + Part::ImplementationsByPath, + Part::RefsByDef, + Part::RefsByPath, + Part::ImportTargets, + Part::ImportersByProbe, + Part::CodeBm25Postings, + Part::CodeBm25ByPath, + Part::GrepBloom, + ]; + + /// The keyspace's on-disk name. + pub const fn name(self) -> &'static str { + match self { + Part::Meta => "meta", + Part::SymbolsByPath => "symbols_by_path", + Part::SymbolsByName => "symbols_by_name", + Part::CallsByPath => "calls_by_path", + Part::CallsByCallee => "calls_by_callee", + Part::ImportsByModule => "imports_by_module", + Part::ImportsByPath => "imports_by_path", + Part::ImplementationsByTrait => "implementations_by_trait", + Part::ImplementationsByPath => "implementations_by_path", + Part::RefsByDef => "refs_by_def", + Part::RefsByPath => "refs_by_path", + Part::ImportTargets => "import_targets", + Part::ImportersByProbe => "importers_by_probe", + Part::CodeBm25Postings => "code_bm25_postings", + Part::CodeBm25ByPath => "code_bm25_by_path", + Part::GrepBloom => "grep_bloom", + } + } +} + +impl IndexDb { + fn keyspace(&self, part: Part) -> &Keyspace { + match part { + Part::Meta => &self.meta, + Part::SymbolsByPath => &self.symbols_by_path, + Part::SymbolsByName => &self.symbols_by_name, + Part::CallsByPath => &self.calls_by_path, + Part::CallsByCallee => &self.calls_by_callee, + Part::ImportsByModule => &self.imports_by_module, + Part::ImportsByPath => &self.imports_by_path, + Part::ImplementationsByTrait => &self.implementations_by_trait, + Part::ImplementationsByPath => &self.implementations_by_path, + Part::RefsByDef => &self.refs_by_def, + Part::RefsByPath => &self.refs_by_path, + Part::ImportTargets => &self.import_targets, + Part::ImportersByProbe => &self.importers_by_probe, + Part::CodeBm25Postings => &self.code_bm25_postings, + Part::CodeBm25ByPath => &self.code_bm25_by_path, + Part::GrepBloom => &self.grep_bloom, + } + } + + /// Raw keyspace access for tests that assert on stored rows or drive the name dictionaries. + #[cfg(test)] + pub(crate) fn keyspace_for_test(&self, part: Part) -> &Keyspace { + self.keyspace(part) + } + + #[cfg(test)] + pub(crate) fn callee_dict(&self) -> &std::sync::Arc { + &self.callee_names + } + + #[cfg(test)] + pub(crate) fn trait_dict(&self) -> &std::sync::Arc { + &self.trait_names + } + + /// A borrowing [`IndexView`] over this database. + pub fn view(&self) -> StandaloneView<&IndexDb> { + StandaloneView(self) + } +} + +/// `Some(&dyn IndexView)` for an optional concrete view, for the call sites that take an optional one. +pub fn as_dyn(view: &Option) -> Option<&dyn IndexView> { + view.as_ref().map(|v| v as &dyn IndexView) +} + +pub(crate) fn kv(guard: fjall::Guard) -> KvResult { + guard.into_inner().map_err(IndexError::from) +} + +/// Read access to the index. See the module docs. +pub trait IndexView: Send + Sync { + /// Point lookup of `key` in `part`. + fn get(&self, part: Part, key: &[u8]) -> Result, IndexError>; + + /// Every row of `part` whose key starts with `prefix`, in key order. + fn prefix<'a>(&'a self, part: Part, prefix: &[u8]) -> KvIter<'a>; + + /// Rows of `part` between the bounds, in key order. + fn range<'a>(&'a self, part: Part, lower: Bound>, upper: Bound>) -> KvIter<'a>; + + /// Every row of `part`, in key order. + fn iter<'a>(&'a self, part: Part) -> KvIter<'a>; + + /// Keys of `calls_by_callee` after `lower`, in key order, whose callee name may contain `needle` + /// (callers still filter by name). Served from the callee-name dictionary when it is built. + fn callee_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a>; + + /// [`Self::callee_keys_matching`] for the trait names of `implementations_by_trait`. + fn trait_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a>; + + /// True when the index holds no per-file symbol entries. + fn symbols_index_is_empty(&self) -> bool; + + /// Resolved uses of the definition at `(def_path, def_start)`. + fn references_to(&self, def_path: &RelPath, def_start: u32) -> Vec<(RelPath, u32)>; + + /// The definition the use at `(use_path, use_start)` resolves to. + fn definition_of(&self, use_path: &RelPath, use_start: u32) -> Option<(RelPath, u32)>; + + /// The bindings of `importer`: `(local_start, def_path, def_start)`. + fn import_targets_of(&self, importer: &RelPath) -> Vec<(u32, RelPath, u32)>; + + /// Importers that probed a path `path` could satisfy. + fn importers_probing(&self, path: &str) -> Vec; + + /// Whether any importer probed a path `path` could satisfy. + fn is_probed(&self, path: &str) -> bool; + + /// Whether the probe keyspaces cover every importer. + fn import_probes_complete(&self) -> bool; + + /// Up to `cap` symbols named `name` (prefix match), as `(name, kind, path, start_byte)`. + fn symbols_by_name_lookup(&self, name: &str, cap: usize) -> Vec<(String, SymbolKind, RelPath, u32)>; + + /// BM25 corpus statistics `(doc_count, total_len)`; `None` before the first scan. + fn bm25_stats(&self) -> Option<(u64, u64)>; + + /// The stored trigram bloom row for the content `hash_hex`. + fn grep_bloom_row(&self, hash_hex: &str) -> Option { + self.get(Part::GrepBloom, hash_hex.as_bytes()).ok().flatten() + } + + /// For each of `entries`, whether a grep for `needle` can skip the file. + fn grep_skip_verdicts(&self, root: &Path, entries: &[GrepEntry], needle: &Needle) -> Vec { + super::grep_bloom::skip_verdicts(|hash| self.grep_bloom_row(hash), root, entries, needle) + } +} + +/// The unlayered view: one [`IndexDb`], read as is. `T` is `&IndexDb` to borrow it or `IndexDb` to +/// own a handle (clones share the same keyspaces). +pub struct StandaloneView(pub T); + +impl> StandaloneView { + fn db(&self) -> &IndexDb { + self.0.borrow() + } +} + +impl + Send + Sync> IndexView for StandaloneView { + fn get(&self, part: Part, key: &[u8]) -> Result, IndexError> { + Ok(self.db().keyspace(part).get(key)?) + } + + fn prefix<'a>(&'a self, part: Part, prefix: &[u8]) -> KvIter<'a> { + Box::new(self.db().keyspace(part).prefix(prefix).map(kv)) + } + + fn range<'a>(&'a self, part: Part, lower: Bound>, upper: Bound>) -> KvIter<'a> { + Box::new(self.db().keyspace(part).range::, _>((lower, upper)).map(kv)) + } + + fn iter<'a>(&'a self, part: Part) -> KvIter<'a> { + Box::new(self.db().keyspace(part).iter().map(kv)) + } + + fn callee_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a> { + let db = self.db(); + name_dict::name_ordered_keys(&db.calls_by_callee, &db.callee_names, needle, cursor_after, lower) + } + + fn trait_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a> { + let db = self.db(); + name_dict::name_ordered_keys( + &db.implementations_by_trait, + &db.trait_names, + needle, + cursor_after, + lower, + ) + } + + fn symbols_index_is_empty(&self) -> bool { + self.db().symbols_index_is_empty() + } + + fn references_to(&self, def_path: &RelPath, def_start: u32) -> Vec<(RelPath, u32)> { + self.db().references_to(def_path, def_start) + } + + fn definition_of(&self, use_path: &RelPath, use_start: u32) -> Option<(RelPath, u32)> { + self.db().definition_of(use_path, use_start) + } + + fn import_targets_of(&self, importer: &RelPath) -> Vec<(u32, RelPath, u32)> { + self.db().import_targets_of(importer) + } + + fn importers_probing(&self, path: &str) -> Vec { + self.db().importers_probing(path) + } + + fn is_probed(&self, path: &str) -> bool { + self.db().is_probed(path) + } + + fn import_probes_complete(&self) -> bool { + self.db().import_probes_complete() + } + + fn symbols_by_name_lookup(&self, name: &str, cap: usize) -> Vec<(String, SymbolKind, RelPath, u32)> { + self.db().symbols_by_name_lookup(name, cap) + } + + fn bm25_stats(&self) -> Option<(u64, u64)> { + self.db().bm25_stats() + } +} + +#[cfg(test)] +pub(crate) use recording::{ReadLog, RecordingView}; + +/// A view that records which reads reach it, so a test can prove a code path is routed through +/// [`IndexView`] and a later layered implementation has nothing to bypass. +#[cfg(test)] +mod recording { + use std::sync::{Arc, Mutex}; + + use super::*; + + /// Shared handle on the reads a [`RecordingView`] has served, as `"[:]"` strings. + #[derive(Clone, Default)] + pub(crate) struct ReadLog(Arc>>); + + impl ReadLog { + fn record(&self, op: &str, part: Option) { + let entry = part.map_or_else(|| op.to_owned(), |p| format!("{op}:{}", p.name())); + self.0.lock().expect("read log").push(entry); + } + + pub(crate) fn saw(&self, entry: &str) -> bool { + self.0.lock().expect("read log").iter().any(|e| e == entry) + } + + pub(crate) fn entries(&self) -> Vec { + self.0.lock().expect("read log").clone() + } + + pub(crate) fn clear(&self) { + self.0.lock().expect("read log").clear(); + } + } + + /// Delegates every read to a [`StandaloneView`] after logging it. + pub(crate) struct RecordingView { + inner: StandaloneView, + log: ReadLog, + } + + impl RecordingView { + pub(crate) fn new(db: IndexDb) -> (Self, ReadLog) { + let log = ReadLog::default(); + ( + Self { + inner: StandaloneView(db), + log: log.clone(), + }, + log, + ) + } + } + + impl IndexView for RecordingView { + fn get(&self, part: Part, key: &[u8]) -> Result, IndexError> { + self.log.record("get", Some(part)); + self.inner.get(part, key) + } + + fn prefix<'a>(&'a self, part: Part, prefix: &[u8]) -> KvIter<'a> { + self.log.record("prefix", Some(part)); + self.inner.prefix(part, prefix) + } + + fn range<'a>(&'a self, part: Part, lower: Bound>, upper: Bound>) -> KvIter<'a> { + self.log.record("range", Some(part)); + self.inner.range(part, lower, upper) + } + + fn iter<'a>(&'a self, part: Part) -> KvIter<'a> { + self.log.record("iter", Some(part)); + self.inner.iter(part) + } + + fn callee_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a> { + self.log.record("callee_keys_matching", None); + self.inner.callee_keys_matching(needle, cursor_after, lower) + } + + fn trait_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a> { + self.log.record("trait_keys_matching", None); + self.inner.trait_keys_matching(needle, cursor_after, lower) + } + + fn symbols_index_is_empty(&self) -> bool { + self.log.record("symbols_index_is_empty", None); + self.inner.symbols_index_is_empty() + } + + fn references_to(&self, def_path: &RelPath, def_start: u32) -> Vec<(RelPath, u32)> { + self.log.record("references_to", None); + self.inner.references_to(def_path, def_start) + } + + fn definition_of(&self, use_path: &RelPath, use_start: u32) -> Option<(RelPath, u32)> { + self.log.record("definition_of", None); + self.inner.definition_of(use_path, use_start) + } + + fn import_targets_of(&self, importer: &RelPath) -> Vec<(u32, RelPath, u32)> { + self.log.record("import_targets_of", None); + self.inner.import_targets_of(importer) + } + + fn importers_probing(&self, path: &str) -> Vec { + self.log.record("importers_probing", None); + self.inner.importers_probing(path) + } + + fn is_probed(&self, path: &str) -> bool { + self.log.record("is_probed", None); + self.inner.is_probed(path) + } + + fn import_probes_complete(&self) -> bool { + self.log.record("import_probes_complete", None); + self.inner.import_probes_complete() + } + + fn symbols_by_name_lookup(&self, name: &str, cap: usize) -> Vec<(String, SymbolKind, RelPath, u32)> { + self.log.record("symbols_by_name_lookup", None); + self.inner.symbols_by_name_lookup(name, cap) + } + + fn bm25_stats(&self) -> Option<(u64, u64)> { + self.log.record("bm25_stats", None); + self.inner.bm25_stats() + } + } +} diff --git a/src/index/view_tests.rs b/src/index/view_tests.rs new file mode 100644 index 00000000..fcea9fd5 --- /dev/null +++ b/src/index/view_tests.rs @@ -0,0 +1,104 @@ +//! The `IndexView` contract: the generic layer reads exactly what the keyspaces hold. The read paths +//! that go through it are covered in `mcp::index_view_tests`. + +use super::IndexDb; +use super::view::{IndexView, Part}; +use crate::config::ConfigV1; +use crate::scanner::{EmbedMode, ScanSource, scan}; +use crate::store::{Store, VIEW_WORKING}; + +fn scanned_store(root: &std::path::Path) -> Store { + for k in 0..6 { + std::fs::write( + root.join(format!("m{k}.rs")), + format!( + "pub trait Tr{k} {{ fn run(&self); }}\n\ + pub struct S{k};\n\ + impl Tr{k} for S{k} {{ fn run(&self) {{ shared_helper(); }} }}\n\ + pub fn driver_{k}() {{ shared_helper(); local_{k}(); }}\n" + ), + ) + .expect("write corpus file"); + } + let mut store = Store::open(root, VIEW_WORKING).expect("open store"); + scan( + root, + &mut store, + &ConfigV1::with_defaults(), + ScanSource::WorkingTree, + EmbedMode::Inline, + ) + .expect("scan"); + store +} + +#[test] +fn the_generic_layer_reads_what_each_keyspace_holds() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let db = store.index_db.as_ref().expect("index"); + let view = db.view(); + let mut non_empty = 0; + for part in Part::ALL { + let keyspace = db.keyspace_for_test(part); + let direct: Vec<_> = keyspace + .iter() + .map(|g| g.into_inner().expect("row")) + .map(|(k, v)| (k.to_vec(), v.to_vec())) + .collect(); + let rows = |iter: super::view::KvIter<'_>| -> Vec<(Vec, Vec)> { + iter.map(|r| r.expect("row")) + .map(|(k, v)| (k.to_vec(), v.to_vec())) + .collect() + }; + assert_eq!(rows(view.iter(part)), direct, "{} iter", part.name()); + assert_eq!(rows(view.prefix(part, b"")), direct, "{} empty prefix", part.name()); + assert_eq!( + rows(view.range(part, std::ops::Bound::Unbounded, std::ops::Bound::Unbounded)), + direct, + "{} unbounded range", + part.name() + ); + if let Some((key, value)) = direct.first() { + non_empty += 1; + assert_eq!( + view.get(part, key).expect("get").map(|v| v.to_vec()), + Some(value.clone()) + ); + assert_eq!(rows(view.prefix(part, &key[..1])).first(), direct.first()); + } + assert_eq!(view.get(part, &[0xFF; 3]).expect("get"), None); + } + assert!( + non_empty >= 6, + "the corpus must populate the layered keyspaces, got {non_empty}" + ); +} + +#[test] +fn parts_name_exactly_the_layered_keyspaces() { + let names: std::collections::BTreeSet<_> = Part::ALL.iter().map(|p| p.name()).collect(); + assert_eq!(names.len(), Part::ALL.len(), "part names are unique"); + for name in &names { + assert!( + super::KEYSPACE_MEMTABLE_BYTES.iter().any(|(k, _)| k == name), + "{name} must be a real keyspace" + ); + } + for unlayered in ["embeddings", "memory_by_key", "memory_archive", "proposals"] { + assert!( + !names.contains(unlayered), + "{unlayered} is per-group or per-worktree state" + ); + } +} + +#[test] +fn store_level_reads_use_the_store_view() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let view = store.index_view().expect("view"); + let db: &IndexDb = store.index_db.as_ref().expect("index"); + assert_eq!(view.symbols_index_is_empty(), db.symbols_index_is_empty()); + assert!(!view.symbols_index_is_empty()); +} diff --git a/src/intel/resolve_pass.rs b/src/intel/resolve_pass.rs index 5c2c9c11..3e7615a2 100644 --- a/src/intel/resolve_pass.rs +++ b/src/intel/resolve_pass.rs @@ -52,6 +52,7 @@ use crate::backpressure::FootprintGate; use crate::chunk_drive::{ChunkCut, drive_chunks_governed}; use crate::config::MaxFootprint; use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::intel::model::FileResolvedRefs; use crate::intel::stage_budget::BudgetedWriter; use crate::lang; @@ -154,8 +155,8 @@ pub(crate) struct PathDelta<'a> { /// The paths of `candidates` that some importer's resolution consulted — the only non-indexed /// paths whose change can matter. A cheap point lookup per path. -pub(crate) fn probed_only(index_db: &IndexDb, candidates: Vec) -> Vec { - candidates.into_iter().filter(|rel| index_db.is_probed(rel)).collect() +pub(crate) fn probed_only(view: &dyn IndexView, candidates: Vec) -> Vec { + candidates.into_iter().filter(|rel| view.is_probed(rel)).collect() } /// Incremental resolve pass for the watcher: only `delta.changed` files' intra facts are restaged, @@ -171,7 +172,7 @@ pub(crate) fn resolve_pass_incremental( return; }; #[cfg(any(feature = "code-intel-js", feature = "code-intel-stack"))] - if !index_db.import_probes_complete() { + if !index_db.view().import_probes_complete() { tracing::info!("resolve pass: import-probe index not built yet — running the wholesale pass once"); resolve_pass(root, store, precise, footprint); return; @@ -184,7 +185,7 @@ pub(crate) fn resolve_pass_incremental( resolve_in_chunks(root, store, index_db, &changed_snapshot, precise, footprint, |facts| { harvest_cross_file_facts(facts, &mut importers); }); - let affected = xfile_incremental::affected_importers(index_db, store, delta); + let affected = xfile_incremental::affected_importers(&index_db.view(), store, delta); let affected_snapshot = snapshot_of(store, affected.iter()); resolve_in_chunks(root, store, index_db, &affected_snapshot, precise, footprint, |facts| { harvest_cross_file_facts(facts, &mut importers); @@ -372,18 +373,18 @@ mod xfile_incremental { use ahash::AHashSet; use super::PathDelta; - use crate::index::IndexDb; + use crate::index::view::IndexView; use crate::intel::resolver::SpecifierResolver; use crate::store::Store; /// Indexed, resolver-capable importers that are NOT in `delta.changed` (those are re-stitched /// anyway) but consulted a touched path: a changed, added or deleted file, or a non-indexed /// config file. One prefix scan of `importers_by_probe` per touched path. - pub(super) fn affected_importers(index_db: &IndexDb, store: &Store, delta: &PathDelta<'_>) -> Vec { + pub(super) fn affected_importers(view: &dyn IndexView, store: &Store, delta: &PathDelta<'_>) -> Vec { let changed: AHashSet<&str> = delta.changed.iter().map(String::as_str).collect(); let mut affected: AHashSet = AHashSet::new(); for touched in delta.changed.iter().chain(delta.removed).chain(delta.other) { - for importer in index_db.importers_probing(touched) { + for importer in view.importers_probing(touched) { let Some(key) = importer.as_str() else { continue; }; diff --git a/src/intel/stage_budget.rs b/src/intel/stage_budget.rs index e60b720a..5a1c6f60 100644 --- a/src/intel/stage_budget.rs +++ b/src/intel/stage_budget.rs @@ -193,7 +193,10 @@ mod tests { "the item-count bound must not be what committed here" ); assert!( - db.refs_by_def.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::RefsByDef) + .iter() + .next() + .is_some(), "the byte budget must have forced a commit already" ); assert!(batch.finish()); @@ -264,7 +267,7 @@ mod tests { } let dump = |db: &IndexDb| -> Vec> { - db.refs_by_def + db.keyspace_for_test(crate::index::view::Part::RefsByDef) .iter() .map(|guard| (*guard.into_inner().expect("read entry").0).to_vec()) .collect() diff --git a/src/mcp/codegraph.rs b/src/mcp/codegraph.rs index d2be412f..a530d151 100644 --- a/src/mcp/codegraph.rs +++ b/src/mcp/codegraph.rs @@ -27,7 +27,7 @@ use rmcp::ErrorData as McpError; use super::MapCache; use super::helpers_calls::for_each_call_in_file; use super::helpers_graph::is_function_like; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::path::RelPath; /// The kind of relationship an edge represents. @@ -409,7 +409,7 @@ pub(crate) fn new_graph_memo(budget_bytes: u64) -> GraphMemo { /// deterministic graph). A poisoned lock is recovered — a lost memo entry only costs one rebuild. pub(crate) fn build_memoized( memo: &Mutex, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, opts: &BuildOpts, ) -> Result, McpError> { @@ -566,7 +566,7 @@ fn resolve_named_edge( /// `scan_cap`. Folding the lanes into one pass would change which files the cap cuts, so the passes /// stay separate and the outline cache absorbs the repeat reads. The build is memoized /// ([`build_memoized`]), so a repo pays this at most once per snapshot per lane set. -pub(crate) fn build(idx: Option<&IndexDb>, cache: &MapCache, opts: &BuildOpts) -> Result { +pub(crate) fn build(idx: Option<&dyn IndexView>, cache: &MapCache, opts: &BuildOpts) -> Result { let kinds = opts.kinds; // Compared byte-wise, not as `&str`: for UTF-8 paths a byte prefix and a `str` prefix are the // same test, and matching on bytes additionally keeps non-UTF-8 indexed paths in scope instead diff --git a/src/mcp/codegraph_tests.rs b/src/mcp/codegraph_tests.rs index 7bba5979..e9e80012 100644 --- a/src/mcp/codegraph_tests.rs +++ b/src/mcp/codegraph_tests.rs @@ -2,6 +2,7 @@ //! under the 1000-line module cap). Included via `#[path]` from `codegraph.rs`. use super::*; +use crate::index::view::as_dyn; use crate::store::Store; use tempfile::TempDir; @@ -59,7 +60,7 @@ fn provenance_fixture() -> (TempDir, Store, MapCache) { fn built(store: &Store, cache: &MapCache, kinds: EdgeKindSet) -> CodeGraph { build( - store.index_db.as_ref(), + as_dyn(&store.index_view()), cache, &BuildOpts { kinds, @@ -431,7 +432,7 @@ fn focus_scopes_the_build_to_files_under_the_path_prefix() { ..EdgeKindSet::none() }; let g = build( - store.index_db.as_ref(), + as_dyn(&store.index_view()), &cache, &BuildOpts { kinds, @@ -675,7 +676,8 @@ fn opts(kinds: EdgeKindSet) -> BuildOpts { #[test] fn memo_serves_the_same_build_for_one_snapshot() { let (_dir, store, cache) = provenance_fixture(); - let idx = store.index_db.as_ref(); + let view = store.index_view(); + let idx = as_dyn(&view); let memo = Mutex::new(new_graph_memo(0)); let first = build_memoized(&memo, idx, &cache, &opts(EdgeKindSet::all())).expect("first build"); @@ -701,7 +703,8 @@ fn memo_serves_the_same_build_for_one_snapshot() { #[test] fn graph_memo_evicts_to_its_byte_budget() { let (_dir, store, cache) = provenance_fixture(); - let idx = store.index_db.as_ref(); + let view = store.index_view(); + let idx = as_dyn(&view); let sized = Mutex::new(new_graph_memo(0)); let one = build_memoized(&sized, idx, &cache, &opts(EdgeKindSet::all())).expect("build"); @@ -732,7 +735,8 @@ fn graph_memo_evicts_to_its_byte_budget() { #[test] fn memo_keys_on_lanes_and_index_mode() { let (_dir, store, cache) = provenance_fixture(); - let idx = store.index_db.as_ref(); + let view = store.index_view(); + let idx = as_dyn(&view); let memo = Mutex::new(new_graph_memo(0)); let calls_only = EdgeKindSet { @@ -776,12 +780,29 @@ fn memo_isolates_snapshots_by_fingerprint() { ); let memo = Mutex::new(new_graph_memo(0)); - let ga = build_memoized(&memo, store_a.index_db.as_ref(), &cache_a, &opts(EdgeKindSet::all())).expect("a"); - let gb = build_memoized(&memo, store_b.index_db.as_ref(), &cache_b, &opts(EdgeKindSet::all())).expect("b"); + let ga = build_memoized( + &memo, + as_dyn(&store_a.index_view()), + &cache_a, + &opts(EdgeKindSet::all()), + ) + .expect("a"); + let gb = build_memoized( + &memo, + as_dyn(&store_b.index_view()), + &cache_b, + &opts(EdgeKindSet::all()), + ) + .expect("b"); assert!(!Arc::ptr_eq(&ga, &gb), "distinct fingerprints are distinct entries"); - let ga_again = - build_memoized(&memo, store_a.index_db.as_ref(), &cache_a, &opts(EdgeKindSet::all())).expect("a again"); + let ga_again = build_memoized( + &memo, + as_dyn(&store_a.index_view()), + &cache_a, + &opts(EdgeKindSet::all()), + ) + .expect("a again"); assert!( Arc::ptr_eq(&ga, &ga_again), "snapshot A still hits its own entry after B was inserted" diff --git a/src/mcp/helpers_archmap.rs b/src/mcp/helpers_archmap.rs index 26eed0f1..0297698c 100644 --- a/src/mcp/helpers_archmap.rs +++ b/src/mcp/helpers_archmap.rs @@ -31,7 +31,7 @@ use super::helpers_graph::is_function_like; use super::helpers_graphview::{DEFAULT_MAX_EXPORT_EDGES, MAX_MAX_EDGES}; use super::kneedle::knee_cutoff; use super::types_archmap::{ArchEdge, ArchNode, ArchitectureMapParams, ArchitectureMapResponse, CycleCluster}; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::path::RelPath; const PAGERANK_ITERS: usize = 20; @@ -192,7 +192,7 @@ impl RepoGraph { /// Build the graph from the L1 cache + call sites. `idx = Some` prefix-scans the /// Fjall `calls_by_path` keyspace per file; `idx = None` reads the in-RAM call index /// (read-only session). Bounded by `edge_scan_cap` total call sites. - pub(crate) fn build(idx: Option<&IndexDb>, cache: &MapCache, edge_scan_cap: usize) -> Result { + pub(crate) fn build(idx: Option<&dyn IndexView>, cache: &MapCache, edge_scan_cap: usize) -> Result { let files: Vec = cache.paths().cloned().collect(); let mut id_of: AHashMap = AHashMap::with_capacity(files.len()); for (i, p) in files.iter().enumerate() { @@ -274,7 +274,7 @@ impl RepoGraph { /// when the overlay is disabled or there's no git repo. pub(crate) fn run_architecture_map( shared: &super::shared_state::SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, churn: Option<&AHashMap>, params: ArchitectureMapParams, @@ -539,7 +539,7 @@ fn build_group_nodes( fn run_tier_grouped( shared: &super::shared_state::SharedReadStack, rg: &RepoGraph, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, churn: Option<&AHashMap>, focus: Option<&RelPath>, @@ -638,7 +638,7 @@ fn build_cycles( /// path builds no codegraph and pays nothing extra. fn grouped_lane_edges( shared: &super::shared_state::SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, rg: &RepoGraph, group_of: &[Option], @@ -819,7 +819,7 @@ fn collect_symbol_candidates( /// distinct-callee fan-out. Isolated because it is the only `?`-fallible (call-index) step here. fn build_symbol_call_edges( survivors: &[SymCand], - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, ) -> Result { let mut by_file: AHashMap> = AHashMap::new(); @@ -916,7 +916,7 @@ fn finalize_symbol_edges(edge_map: &AHashMap<(u32, u32), u32>, kept: usize, max_ fn run_tier_symbol( rg: &RepoGraph, cache: &MapCache, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, churn: Option<&AHashMap>, focus: Option<&RelPath>, params: &ArchitectureMapParams, diff --git a/src/mcp/helpers_calls.rs b/src/mcp/helpers_calls.rs index 1c04829c..8f5da5f5 100644 --- a/src/mcp/helpers_calls.rs +++ b/src/mcp/helpers_calls.rs @@ -14,7 +14,7 @@ pub(crate) use super::helpers_calls_scan::InRamCallIndex; use super::helpers_calls_scan::{CallRef, CallScanPage, budget_call_page, calls_in_file_fjall}; use super::index_route::IndexRoute; use super::types::ReferenceHit; -use crate::index::IndexDb; +use crate::index::view::{IndexView, Part}; use crate::path::RelPath; /// How `find_callers` fetches the cross-file resolved uses of a definition. The precise @@ -108,7 +108,7 @@ impl RefsSource<'_> { /// rather than one daemon round trip per file; `references` / `callers` / `implementations` are the /// point queries that forward instead. pub(super) fn for_each_call_in_file bool>( - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, path: &RelPath, mut f: F, @@ -133,12 +133,12 @@ pub(super) fn for_each_call_in_file bool>( /// `(line, column)` as 1-based / 0-based respectively. Falls back to `(0, 0)` when the /// row/col fields aren't populated (older L2 blobs predating the field's introduction). pub(super) fn resolve_call_line_col( - idx: &crate::index::IndexDb, + idx: &dyn crate::index::view::IndexView, rel: &crate::path::RelPath, start_byte: u32, ) -> (u32, u32) { let key = crate::index::keys::call_by_path(rel, start_byte); - let value = match idx.calls_by_path.get(key) { + let value = match idx.get(Part::CallsByPath, &key) { Ok(Some(v)) => v, _ => return (0, 0), }; @@ -519,9 +519,9 @@ mod tests { /// The route a writer session takes: its own open fjall index. fn local_route(store: &Store) -> super::IndexRoute { - super::IndexRoute::Local(Box::new( + super::IndexRoute::Local(Box::new(crate::index::view::StandaloneView( store.index_db.clone().expect("a writer store opens the fjall index"), - )) + ))) } /// Scan `root` and return the store plus its map cache. diff --git a/src/mcp/helpers_calls_scan.rs b/src/mcp/helpers_calls_scan.rs index 97bb101c..10b14731 100644 --- a/src/mcp/helpers_calls_scan.rs +++ b/src/mcp/helpers_calls_scan.rs @@ -11,6 +11,7 @@ use rmcp::ErrorData as McpError; use super::cursor::Cursor; use super::helpers_calls::resolve_call_line_col; use super::types::ReferenceHit; +use crate::index::view::Part; pub(crate) struct CallScanPage { pub total: u32, @@ -71,7 +72,7 @@ pub(super) fn budget_call_page(page: CallScanPage, max_tokens: Option) -> B /// the cursor (exclusive). The cursor returned in [`CallScanPage::next_cursor`] is the /// last key emitted on this page — pass it back on the next call to advance. pub(crate) fn scan_calls_by_name( - idx: &crate::index::IndexDb, + idx: &dyn crate::index::view::IndexView, name: &str, limit: usize, cursor_after: Option<&[u8]>, @@ -90,12 +91,8 @@ pub(crate) fn scan_calls_by_name( let scan_cap = limit.saturating_mul(8).max(2_000); let mut has_more = false; let mut matched: usize = 0; - for guard in - crate::index::name_dict::name_ordered_keys(&idx.calls_by_callee, &idx.callee_names, name, cursor_after, lower) - { - let (k, _) = guard - .into_inner() - .map_err(|e| McpError::internal_error(format!("index iter: {e}"), None))?; + for guard in idx.callee_keys_matching(name, cursor_after, lower) { + let (k, _) = guard.map_err(|e| McpError::internal_error(format!("index iter: {e}"), None))?; let Some((callee, rel, start)) = crate::index::keys::parse_call_by_callee(&k) else { continue; }; @@ -142,7 +139,7 @@ pub(crate) fn scan_calls_by_name( /// reader behind the writer session's per-file lookups and the daemon's forwarded /// `CallsInFiles`, so the two cannot drift. pub(crate) fn calls_in_file_fjall( - idx: &crate::index::IndexDb, + idx: &dyn crate::index::view::IndexView, path: &crate::path::RelPath, ) -> Result, McpError> { let prefix = crate::index::keys::calls_by_path_prefix(path); @@ -151,10 +148,8 @@ pub(crate) fn calls_in_file_fjall( None => Bound::Unbounded, }; let mut out = Vec::new(); - for guard in idx.calls_by_path.range::, _>((Bound::Included(prefix), upper)) { - let (_, v) = guard - .into_inner() - .map_err(|e| McpError::internal_error(format!("index iter: {e}"), None))?; + for guard in idx.range(Part::CallsByPath, Bound::Included(prefix), upper) { + let (_, v) = guard.map_err(|e| McpError::internal_error(format!("index iter: {e}"), None))?; let call: crate::extract::Call = match rmp_serde::from_slice(&v) { Ok(c) => c, Err(_) => continue, diff --git a/src/mcp/helpers_calls_scan_tests.rs b/src/mcp/helpers_calls_scan_tests.rs index 345fc0d2..bd89a9d0 100644 --- a/src/mcp/helpers_calls_scan_tests.rs +++ b/src/mcp/helpers_calls_scan_tests.rs @@ -143,7 +143,7 @@ fn write_file(db: &IndexDb, rng: &mut Rng, vocab: &[String], i: usize) { type CallPage = (u32, bool, Vec>, Vec<(String, u32, u32)>, bool); fn call_page(db: &IndexDb, needle: &str, limit: usize, cursor: Option<&[u8]>) -> CallPage { - let page = scan_calls_by_name(db, needle, limit, cursor).unwrap(); + let page = scan_calls_by_name(&db.view(), needle, limit, cursor).unwrap(); let hits = page.hits.iter().map(|h| (h.callee.clone(), h.line, h.column)).collect(); ( page.total, @@ -158,7 +158,7 @@ type ImplPage = (usize, bool, Vec>, bool); fn impl_page(db: &IndexDb, needle: &str, language: Option<&str>, limit: usize, cursor: Option<&[u8]>) -> ImplPage { let page = scan_impls_fjall( - db, + &db.view(), needle, language, limit, @@ -198,9 +198,9 @@ fn assert_equivalent(db: &IndexDb, label: &str) { for limit in [2usize, 50, 400] { let (mut want_cursor, mut got_cursor): (Option>, Option>) = (None, None); for step in 0..10 { - db.callee_names.set_disabled(true); + db.callee_dict().set_disabled(true); let want = call_page(db, needle, limit, want_cursor.as_deref()); - db.callee_names.set_disabled(false); + db.callee_dict().set_disabled(false); let got = call_page(db, needle, limit, got_cursor.as_deref()); assert_eq!(want, got, "{label}: calls needle={needle:?} limit={limit} step={step}"); if !want.4 { @@ -215,9 +215,9 @@ fn assert_equivalent(db: &IndexDb, label: &str) { for limit in [2usize, 100] { let mut cursor: Option> = None; for step in 0..10 { - db.trait_names.set_disabled(true); + db.trait_dict().set_disabled(true); let want = impl_page(db, needle, language, limit, cursor.as_deref()); - db.trait_names.set_disabled(false); + db.trait_dict().set_disabled(false); let got = impl_page(db, needle, language, limit, cursor.as_deref()); assert_eq!( want, got, @@ -235,10 +235,12 @@ fn assert_equivalent(db: &IndexDb, label: &str) { fn build_dicts(db: &IndexDb) { // Without this the small test vocabulary routes most needles to the walk and proves nothing. - db.callee_names.always_use_dictionary(); - db.trait_names.always_use_dictionary(); - db.callee_names.build_blocking(&db.calls_by_callee); - db.trait_names.build_blocking(&db.implementations_by_trait); + db.callee_dict().always_use_dictionary(); + db.trait_dict().always_use_dictionary(); + db.callee_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::CallsByCallee)); + db.trait_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::ImplementationsByTrait)); } #[test] @@ -252,13 +254,13 @@ fn dictionary_scans_match_the_full_walk() { } build_dicts(&db); assert!( - db.callee_names.len() > 100, + db.callee_dict().len() > 100, "dictionary should hold the distinct callees" ); assert_equivalent(&db, "after build"); // The corpus must exercise the cap, or the partial-total path is untested. - let capped = scan_calls_by_name(&db, "run", 1, None).unwrap(); + let capped = scan_calls_by_name(&db.view(), "run", 1, None).unwrap(); assert!(capped.total_is_partial, "hot name must hit scan_cap"); // Rescan: rewrite some files (names change, old keys are removed), drop others, add new files @@ -281,8 +283,10 @@ fn dictionary_scans_match_the_full_walk() { assert_equivalent(&db, "after new names"); // A fold merges the new names into the snapshot without changing any answer. - db.callee_names.build_blocking(&db.calls_by_callee); - db.trait_names.build_blocking(&db.implementations_by_trait); + db.callee_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::CallsByCallee)); + db.trait_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::ImplementationsByTrait)); assert_equivalent(&db, "after rebuild"); } @@ -298,7 +302,7 @@ fn names_recorded_before_the_build_completes_are_not_lost() { // Written while no snapshot exists (as during a background build): recorded, not yet folded. write_file(&db, &mut rng, &["late_arrival".to_string()], 99); build_dicts(&db); - let page = scan_calls_by_name(&db, "late_arrival", 10, None).unwrap(); + let page = scan_calls_by_name(&db.view(), "late_arrival", 10, None).unwrap(); assert!(page.total > 0); assert_equivalent(&db, "late arrival"); } @@ -312,9 +316,9 @@ fn unbuilt_dictionary_still_answers_via_the_walk() { for i in 0..30 { write_file(&db, &mut rng, &vocab, i); } - db.callee_names.set_disabled(true); + db.callee_dict().set_disabled(true); let want = call_page(&db, "run", 20, None); - db.callee_names.set_disabled(false); + db.callee_dict().set_disabled(false); // First query: kicks off the background build and is served by the walk. let first = call_page(&db, "run", 20, None); assert_eq!(want, first); @@ -333,19 +337,21 @@ fn time_name_scans() { }; let db = IndexDb::open(std::path::Path::new(&view)).unwrap(); let t = Instant::now(); - db.callee_names.build_blocking(&db.calls_by_callee); + db.callee_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::CallsByCallee)); eprintln!( "callee dictionary: {} names, {} KiB resident, built in {:?}", - db.callee_names.len(), - db.callee_names.snapshot_bytes() / 1024, + db.callee_dict().len(), + db.callee_dict().snapshot_bytes() / 1024, t.elapsed() ); let t = Instant::now(); - db.trait_names.build_blocking(&db.implementations_by_trait); + db.trait_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::ImplementationsByTrait)); eprintln!( "trait dictionary: {} names, {} KiB resident, built in {:?}", - db.trait_names.len(), - db.trait_names.snapshot_bytes() / 1024, + db.trait_dict().len(), + db.trait_dict().snapshot_bytes() / 1024, t.elapsed() ); let needles = [ @@ -364,11 +370,11 @@ fn time_name_scans() { let mut dict = Vec::new(); let mut first = None; for round in 0..5 { - db.callee_names.set_disabled(true); + db.callee_dict().set_disabled(true); let t = Instant::now(); let want = call_page(&db, needle, 50, None); walk.push(t.elapsed()); - db.callee_names.set_disabled(false); + db.callee_dict().set_disabled(false); let t = Instant::now(); let got = call_page(&db, needle, 50, None); dict.push(t.elapsed()); diff --git a/src/mcp/helpers_code_search.rs b/src/mcp/helpers_code_search.rs index 8ee8e6c2..8e5aa809 100644 --- a/src/mcp/helpers_code_search.rs +++ b/src/mcp/helpers_code_search.rs @@ -231,13 +231,13 @@ async fn fjall_lanes( ) -> Result { { let store = state.shared.store.read().await; - if let Some(db) = store.index_db.as_ref() { - let keyword = bm25_search(db, query, limit) + if let Some(db) = store.index_view() { + let keyword = bm25_search(&db, query, limit) .into_iter() .map(|hit| (hit.chunk_id, hit.score)) .collect(); let exact = if want_exact { - exact_lane_chunk_ids(&store, db, query, limit) + exact_lane_chunk_ids(&store, &db, query, limit) } else { Vec::new() }; diff --git a/src/mcp/helpers_community.rs b/src/mcp/helpers_community.rs index e1038cf9..0878f3f4 100644 --- a/src/mcp/helpers_community.rs +++ b/src/mcp/helpers_community.rs @@ -20,7 +20,7 @@ use super::traverse::Adjacency; use super::types::LifecycleNotice; use super::types_community::{CommunitiesParams, CommunitiesResponse, Community}; use super::types_traverse::GraphNode; -use crate::index::IndexDb; +use crate::index::view::IndexView; /// Sweep bound for both detection algorithms; they converge well inside this on real graphs. const COMMUNITY_MAX_ITERS: u32 = 50; @@ -62,7 +62,7 @@ pub(super) fn label_for(adj: &Adjacency, cache: &MapCache, ranked: &[u32]) -> St /// `communities` — cluster the code-graph into de-facto modules with deterministic labels. pub(super) fn run_communities( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, params: CommunitiesParams, notice: Option, diff --git a/src/mcp/helpers_graph.rs b/src/mcp/helpers_graph.rs index 3f9d5d55..531f76a1 100644 --- a/src/mcp/helpers_graph.rs +++ b/src/mcp/helpers_graph.rs @@ -148,12 +148,12 @@ pub(super) async fn run_graph(state: &super::ServerState, params: GraphParams) - let started = std::time::Instant::now(); state.await_cache_ready().await; let store = state.shared.store.read().await; - let idx = store.index_db.as_ref().cloned(); + let view = store.index_db.as_ref().cloned().map(crate::index::view::StandaloneView); let basemind_dir = store.basemind_dir.clone(); drop(store); let cache = state.shared.cache.load_full(); let shared = &state.shared; - let idx = idx.as_ref(); + let idx = crate::index::view::as_dyn(&view); if idx.is_none() { // The graph walks read every file's call sites, so a session with no Fjall index serves them // from the in-RAM projection. Build it now so the notice below reflects whether its byte @@ -420,7 +420,7 @@ pub(super) fn is_function_like(kind: SymbolKind) -> bool { /// Entry point — builds the shared calls graph, projects it, and runs the BFS. pub(super) fn run_call_graph( shared: &SharedReadStack, - idx: Option<&crate::index::IndexDb>, + idx: Option<&dyn crate::index::view::IndexView>, params: CallGraphParams, cache: &MapCache, notice: Option, diff --git a/src/mcp/helpers_graphview.rs b/src/mcp/helpers_graphview.rs index d462ad0d..8aedd32f 100644 --- a/src/mcp/helpers_graphview.rs +++ b/src/mcp/helpers_graphview.rs @@ -24,7 +24,7 @@ use super::types::LifecycleNotice; use super::types_graphview::{ DisplayParams, DisplayResponse, GraphExportParams, GraphExportResponse, UiParams, UiResponse, }; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::path::RelPath; /// Sweep bound for community detection when tagging nodes; converges well inside this on real graphs. @@ -83,7 +83,7 @@ fn select_nodes(partition: &community::Partition, n: usize, max_nodes: usize) -> #[allow(clippy::too_many_arguments)] pub(super) fn build_graph_view( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, kinds: EdgeKindSet, min_conf: f32, @@ -262,7 +262,7 @@ fn prune_exports(dir: &std::path::Path, budget: u64) { /// `graph_export` — render the code-graph into a chosen text format. pub(super) fn run_graph_export( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, basemind_dir: &std::path::Path, params: GraphExportParams, @@ -440,7 +440,7 @@ fn spawn_opener(program: &str, args: &[&std::ffi::OsStr]) -> OpenOutcome { /// runtime. The render/build portion stays synchronous inline, matching `run_graph_export`. pub(super) async fn run_display( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, basemind_dir: &std::path::Path, params: DisplayParams, @@ -546,7 +546,7 @@ pub(super) struct UiParts { #[allow(clippy::too_many_arguments)] pub(super) fn render_ui_parts( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, format: &str, edges: &str, @@ -611,7 +611,7 @@ const UI_NOT_SERVED_DETAIL: &str = "no basemind daemon is serving the interactiv /// is the launch — the launch outcome is not reported. pub(super) async fn run_ui( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, basemind_dir: &std::path::Path, params: UiParams, diff --git a/src/mcp/helpers_impls.rs b/src/mcp/helpers_impls.rs index 9efebde9..80bf7f2a 100644 --- a/src/mcp/helpers_impls.rs +++ b/src/mcp/helpers_impls.rs @@ -87,7 +87,7 @@ pub(super) async fn run_find_implementations( /// answers them from its in-RAM map while the daemon answers them from its store, and both must /// run this exact loop. pub(crate) fn scan_impls_fjall( - idx: &crate::index::IndexDb, + idx: &dyn crate::index::view::IndexView, trait_name: &str, language: Option<&str>, limit: usize, @@ -108,16 +108,8 @@ pub(crate) fn scan_impls_fjall( let mut has_more = false; let mut matched: usize = 0; - for guard in crate::index::name_dict::name_ordered_keys( - &idx.implementations_by_trait, - &idx.trait_names, - trait_name, - cursor_after, - lower, - ) { - let (k, _) = guard - .into_inner() - .map_err(|e| McpError::internal_error(format!("impl index iter: {e}"), None))?; + for guard in idx.trait_keys_matching(trait_name, cursor_after, lower) { + let (k, _) = guard.map_err(|e| McpError::internal_error(format!("impl index iter: {e}"), None))?; let Some((trait_name, impl_type, rel, start_byte)) = crate::index::keys::parse_impl_by_trait(&k) else { continue; @@ -167,7 +159,7 @@ pub(crate) fn scan_impls_fjall( /// [`scan_impls_fjall`] with the session's environment: language from the in-RAM file view and /// `(row, col)` from the cached L1 outline. pub(crate) fn scan_impls_session( - idx: &crate::index::IndexDb, + idx: &dyn crate::index::view::IndexView, cache: &super::MapCache, trait_name: &str, language: Option<&str>, diff --git a/src/mcp/helpers_traverse.rs b/src/mcp/helpers_traverse.rs index c69b7975..d4a98ce5 100644 --- a/src/mcp/helpers_traverse.rs +++ b/src/mcp/helpers_traverse.rs @@ -20,7 +20,7 @@ use super::types_traverse::{ GraphEdge, GraphNode, NeighborsParams, NeighborsResponse, PathParams, PathResponse, SubgraphParams, SubgraphResponse, }; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::path::RelPath; const DEFAULT_DEPTH: u32 = 2; @@ -234,7 +234,7 @@ fn kind_to_str_rationale(kind: crate::extract::RationaleKind) -> &'static str { /// adjacency plus whether the underlying call scan was truncated. fn build_adjacency( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, kinds: EdgeKindSet, ) -> Result<(Adjacency, bool), McpError> { @@ -289,7 +289,7 @@ fn remap_edges(edges: &[WalkEdge], index_of: &AHashMap) -> Vec, + idx: Option<&dyn IndexView>, cache: &MapCache, params: NeighborsParams, notice: Option, @@ -347,7 +347,7 @@ pub(super) fn run_neighbors( /// `path` — confidence-weighted shortest route between two symbols. pub(super) fn run_path( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, params: PathParams, notice: Option, @@ -397,7 +397,7 @@ pub(super) fn run_path( /// `subgraph` — the neighborhood around a symbol, cut to the central head. pub(super) fn run_subgraph( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, params: SubgraphParams, notice: Option, diff --git a/src/mcp/index_read.rs b/src/mcp/index_read.rs index 94ef10d1..c49bc909 100644 --- a/src/mcp/index_read.rs +++ b/src/mcp/index_read.rs @@ -19,7 +19,7 @@ use super::types_impls::ImplementationHit; use crate::comms::index_read_proto::{ IndexReadQuery, IndexReadResult, MAX_CALLS_IN_FILES, WireCall, WireCallHit, WireCallScan, WireImplHit, WireImplScan, }; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::store::Store; /// Why a forwarded scan could not run: the workspace the daemon holds has no fjall index, so the @@ -29,7 +29,8 @@ const NO_INDEX: &str = "the daemon holds this workspace without a fjall index, s /// Run a forwarded scan against an open workspace store. pub(crate) fn index_read_against(store: &Store, query: &IndexReadQuery) -> Result { - let idx = store.index_db.as_ref().ok_or_else(|| NO_INDEX.to_string())?; + let view = store.index_view().ok_or_else(|| NO_INDEX.to_string())?; + let idx: &dyn IndexView = &view; match query { IndexReadQuery::CallScan { name, limit, cursor } => call_scan(idx, name, *limit, cursor.as_deref()), IndexReadQuery::CallsInFiles { paths } => calls_in_files(idx, paths), @@ -64,7 +65,7 @@ pub(crate) fn index_read_against(store: &Store, query: &IndexReadQuery) -> Resul /// handle instead of the workspace store lock: this scan walks the whole `calls_by_callee` /// partition and would otherwise block a rescan for its duration. pub(crate) fn call_scan( - idx: &IndexDb, + idx: &dyn IndexView, name: &str, limit: u32, cursor: Option<&[u8]>, @@ -76,7 +77,7 @@ pub(crate) fn call_scan( /// The `GrepBloom` arm on its own, so the daemon can run it holding only a cloned [`IndexDb`] handle /// and the workspace root instead of the store lock: it fetches a row and `stat`s a file per path. pub(crate) fn grep_skip( - idx: &IndexDb, + idx: &dyn IndexView, root: &std::path::Path, pattern: &str, entries: &[crate::index::grep_bloom::GrepEntry], @@ -97,7 +98,7 @@ pub(crate) fn grep_skip( )) } -fn calls_in_files(idx: &IndexDb, paths: &[crate::path::RelPath]) -> Result { +fn calls_in_files(idx: &dyn IndexView, paths: &[crate::path::RelPath]) -> Result { if paths.len() > MAX_CALLS_IN_FILES { return Err(format!( "calls_in_files names {} files; at most {MAX_CALLS_IN_FILES} per request", diff --git a/src/mcp/index_route.rs b/src/mcp/index_route.rs index 6b8434aa..f6add724 100644 --- a/src/mcp/index_route.rs +++ b/src/mcp/index_route.rs @@ -23,8 +23,8 @@ use rmcp::ErrorData as McpError; use super::MapCache; use super::helpers_calls_scan::{CallRef, CallScanPage, calls_in_file_fjall, scan_calls_by_name, scan_calls_in_ram}; use super::helpers_impls::{ImplScanPage, scan_impls_in_ram, scan_impls_session}; -use crate::index::IndexDb; use crate::index::grep_bloom::GrepEntry; +use crate::index::view::{IndexView, StandaloneView}; use crate::path::RelPath; #[cfg(all(feature = "comms", any(unix, windows)))] @@ -37,7 +37,7 @@ use { /// How a session reaches the fjall index its reference reads need. pub(super) enum IndexRoute { /// This session's own open index. - Local(Box), + Local(Box), /// The daemon's workspace pool, in-process (daemon-hosted connection). #[cfg(all(feature = "comms", any(unix, windows)))] Host { @@ -58,7 +58,7 @@ impl IndexRoute { /// Decide the route for this session. Never fails: an unreachable daemon is the `InRam` route. pub(super) async fn resolve(state: &super::ServerState) -> Self { if let Some(idx) = state.shared.store.read().await.index_db.clone() { - return Self::Local(Box::new(idx)); + return Self::Local(Box::new(StandaloneView(idx))); } #[cfg(all(feature = "comms", any(unix, windows)))] { @@ -164,7 +164,7 @@ impl IndexRoute { cursor_after: Option<&[u8]>, ) -> Result { if let Self::Local(idx) = self { - return scan_calls_by_name(idx, name, limit, cursor_after); + return scan_calls_by_name(&**idx, name, limit, cursor_after); } #[cfg(all(feature = "comms", any(unix, windows)))] match self @@ -194,7 +194,7 @@ impl IndexRoute { paths: &[RelPath], ) -> Result>, McpError> { if let Self::Local(idx) = self { - return paths.iter().map(|path| calls_in_file_fjall(idx, path)).collect(); + return paths.iter().map(|path| calls_in_file_fjall(&**idx, path)).collect(); } #[cfg(all(feature = "comms", any(unix, windows)))] match self @@ -229,7 +229,7 @@ impl IndexRoute { cursor_after: Option<&[u8]>, ) -> Result { if let Self::Local(idx) = self { - return scan_impls_session(idx, cache, trait_name, language, limit, cursor_after); + return scan_impls_session(&**idx, cache, trait_name, language, limit, cursor_after); } #[cfg(all(feature = "comms", any(unix, windows)))] match self diff --git a/src/mcp/index_route_tests.rs b/src/mcp/index_route_tests.rs index 9b2bb6e6..d126f66d 100644 --- a/src/mcp/index_route_tests.rs +++ b/src/mcp/index_route_tests.rs @@ -19,6 +19,7 @@ use super::{HostBackend, MapCache}; use crate::comms::index_read_proto::{IndexReadQuery, IndexReadResult}; use crate::config::ConfigV1; use crate::index::grep_bloom::GrepEntry; +use crate::index::view::StandaloneView; use crate::path::RelPath; use crate::scanner::{EmbedMode, ScanSource, scan}; use crate::store::{Store, VIEW_WORKING}; @@ -164,7 +165,7 @@ fn fixture() -> Fixture { broken: true, }); Fixture { - local: IndexRoute::Local(Box::new(idx)), + local: IndexRoute::Local(Box::new(StandaloneView(idx))), host: IndexRoute::Host { host: Arc::clone(&host) as Arc, root: root.clone(), diff --git a/src/mcp/index_view_tests.rs b/src/mcp/index_view_tests.rs new file mode 100644 index 00000000..a489638c --- /dev/null +++ b/src/mcp/index_view_tests.rs @@ -0,0 +1,161 @@ +//! Proof that the read paths reach the fjall index only through [`IndexView`](crate::index::view): +//! each test hands the code under test a [`RecordingView`] and asserts on what was read. A layered +//! implementation can therefore replace the view without touching a call site. + +use super::helpers_calls_scan::{calls_in_file_fjall, scan_calls_by_name}; +use super::helpers_impls::scan_impls_fjall; +use super::index_route::IndexRoute; +use crate::config::ConfigV1; +use crate::index::grep_bloom::{GrepEntry, Needle}; +use crate::index::view::{IndexView, ReadLog, RecordingView}; +use crate::path::RelPath; +use crate::scanner::{EmbedMode, ScanSource, scan}; +use crate::store::{Store, VIEW_WORKING}; + +const FILES: usize = 6; + +fn scanned_store(root: &std::path::Path) -> Store { + for k in 0..FILES { + std::fs::write( + root.join(format!("m{k}.rs")), + format!( + "pub trait Tr{k} {{ fn run(&self); }}\n\ + pub struct S{k};\n\ + impl Tr{k} for S{k} {{ fn run(&self) {{ shared_helper(); }} }}\n\ + pub fn driver_{k}() {{ shared_helper(); local_{k}(); }}\n" + ), + ) + .expect("write corpus file"); + } + let mut store = Store::open(root, VIEW_WORKING).expect("open store"); + scan( + root, + &mut store, + &ConfigV1::with_defaults(), + ScanSource::WorkingTree, + EmbedMode::Inline, + ) + .expect("scan"); + store +} + +fn recording(store: &Store) -> (RecordingView, ReadLog) { + RecordingView::new(store.index_db.clone().expect("index")) +} + +fn grep_entries(store: &Store) -> Vec { + (0..FILES) + .map(|k| { + let rel = RelPath::from(format!("m{k}.rs").as_bytes()); + let entry = store.lookup(&rel).expect("indexed"); + GrepEntry { + rel, + hash_hex: entry.hash_hex.clone(), + size_bytes: entry.size_bytes, + mtime_ns: entry.mtime, + } + }) + .collect() +} + +#[test] +fn call_and_implementation_scans_read_through_the_view() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let db = store.index_db.clone().expect("index"); + let (view, log) = recording(&store); + + let direct = scan_calls_by_name(&db.view(), "shared_helper", 50, None).expect("direct scan"); + let routed = scan_calls_by_name(&view, "shared_helper", 50, None).expect("routed scan"); + assert_eq!(routed.total, 2 * FILES as u32, "two call sites per file"); + assert_eq!((routed.total, &routed.hit_keys), (direct.total, &direct.hit_keys)); + assert!(log.saw("callee_keys_matching"), "name scan: {:?}", log.entries()); + assert!(log.saw("get:calls_by_path"), "line/column resolve: {:?}", log.entries()); + + log.clear(); + let calls = calls_in_file_fjall(&view, &RelPath::from("m0.rs")).expect("calls"); + assert!(!calls.is_empty()); + assert!(log.saw("range:calls_by_path"), "per-file calls: {:?}", log.entries()); + + log.clear(); + let impls = scan_impls_fjall(&view, "Tr", None, 50, None, |_, _| true, |_, _| (0, 0)).expect("impls"); + assert_eq!(impls.total, FILES); + assert!(log.saw("trait_keys_matching"), "implementations: {:?}", log.entries()); +} + +#[test] +fn probe_and_grep_prefilter_reads_go_through_the_view() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let (view, log) = recording(&store); + + let probed = crate::intel::resolve_pass::probed_only(&view, vec!["nowhere.rs".to_owned()]); + assert!(probed.is_empty()); + assert!(log.saw("is_probed"), "watcher probe filter: {:?}", log.entries()); + + log.clear(); + let entries = grep_entries(&store); + let needle = Needle::compile("zzz_definitely_absent").expect("needle"); + let verdicts = view.grep_skip_verdicts(dir.path(), &entries, &needle); + assert_eq!(verdicts.len(), entries.len()); + assert!(log.saw("get:grep_bloom"), "grep prefilter: {:?}", log.entries()); +} + +#[cfg(feature = "code-search")] +#[test] +fn both_search_lanes_read_through_the_view() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let (view, log) = recording(&store); + + let exact = crate::search::exact::exact_lane_chunk_ids(&store, &view, "driver_1", 10); + assert!(!exact.is_empty(), "the exact lane resolves a defined function"); + assert!(log.saw("symbols_by_name_lookup"), "exact lane: {:?}", log.entries()); + + log.clear(); + let keyword = crate::search::bm25::bm25_search(&view, "driver", 10); + assert!(!keyword.is_empty(), "the keyword lane finds the chunk"); + assert!(log.saw("bm25_stats"), "keyword lane stats: {:?}", log.entries()); + assert!( + log.saw("prefix:code_bm25_postings"), + "keyword lane postings: {:?}", + log.entries() + ); +} + +#[tokio::test(flavor = "multi_thread")] +async fn the_local_route_answers_every_read_from_its_view() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let cache = super::MapCache::build(&store, 0); + let (view, log) = recording(&store); + let route = IndexRoute::Local(Box::new(view)); + + let page = route.scan_calls(&cache, "shared_helper", 50, None).await.expect("scan"); + assert_eq!(page.total, 2 * FILES as u32); + assert!( + log.saw("callee_keys_matching"), + "references/callers: {:?}", + log.entries() + ); + + log.clear(); + let paths = vec![RelPath::from("m1.rs")]; + let per_file = route.calls_in_files(&cache, &paths).await.expect("calls"); + assert!(!per_file[0].is_empty()); + assert!(log.saw("range:calls_by_path"), "resolved callers: {:?}", log.entries()); + + log.clear(); + let impls = route.scan_impls(&cache, "Tr", None, 50, None).await.expect("impls"); + assert_eq!(impls.total, FILES); + assert!(log.saw("trait_keys_matching"), "implementations: {:?}", log.entries()); + + log.clear(); + let entries = grep_entries(&store); + let verdicts = route + .grep_skip(dir.path(), "zzz_definitely_absent", &entries) + .await + .expect("verdicts"); + assert_eq!(verdicts.len(), entries.len()); + assert!(log.saw("get:grep_bloom"), "grep prefilter: {:?}", log.entries()); +} diff --git a/src/mcp/mod.rs b/src/mcp/mod.rs index f5e646d1..0a3cad0b 100644 --- a/src/mcp/mod.rs +++ b/src/mcp/mod.rs @@ -69,6 +69,8 @@ pub(crate) mod index_read; mod index_route; #[cfg(test)] mod index_route_tests; +#[cfg(test)] +mod index_view_tests; mod kneedle; mod l1_cache; mod lean; diff --git a/src/mcp/shared_state.rs b/src/mcp/shared_state.rs index c57b6830..5dc545c9 100644 --- a/src/mcp/shared_state.rs +++ b/src/mcp/shared_state.rs @@ -18,6 +18,7 @@ use lru::LruCache; use tokio::sync::{Notify, RwLock}; use super::{MapCache, OUTLINE_CACHE_CAP, OutlineCache, ServerOptions, telemetry}; +use crate::index::view::IndexView; use crate::store::Store; /// Heavy, workspace-level read state shared across every connection to one `(root, view)`. @@ -187,9 +188,8 @@ fn last_scan_degraded(store: &Store) -> bool { pub(crate) fn boot_plan(store: &Store, options: &ServerOptions) -> (bool, bool) { let view_is_working = store.view == crate::store::VIEW_WORKING; let fjall_index_empty = store - .index_db - .as_ref() - .map(|db| db.symbols_index_is_empty()) + .index_view() + .map(|view| view.symbols_index_is_empty()) .unwrap_or(false); let needs_initial_scan = (options.daemon_writer || !options.read_only) && view_is_working @@ -301,7 +301,7 @@ impl SharedReadStack { /// fingerprint, so it invalidates itself the moment the caller's `cache` differs. pub(crate) fn graph( &self, - idx: Option<&crate::index::IndexDb>, + idx: Option<&dyn crate::index::view::IndexView>, cache: &MapCache, opts: &super::codegraph::BuildOpts, ) -> Result, rmcp::ErrorData> { diff --git a/src/mcp/tools_graph.rs b/src/mcp/tools_graph.rs index 5530cebc..01226bd9 100644 --- a/src/mcp/tools_graph.rs +++ b/src/mcp/tools_graph.rs @@ -113,12 +113,12 @@ impl BasemindServer { ) -> Result<(String, &'static str), McpError> { self.state.await_cache_ready().await; let store = self.state.shared.store.read().await; - let idx = store.index_db.as_ref().cloned(); + let view = store.index_db.as_ref().cloned().map(crate::index::view::StandaloneView); drop(store); let cache = self.state.shared.cache.load_full(); let parts = render_ui_parts( &self.state.shared, - idx.as_ref(), + crate::index::view::as_dyn(&view), &cache, format, edges, diff --git a/src/query.rs b/src/query.rs index d9bc1c91..aad0ad5c 100644 --- a/src/query.rs +++ b/src/query.rs @@ -1,6 +1,7 @@ use thiserror::Error; use crate::extract::{FileMapL1, FileMapL2, Symbol, SymbolKind}; +use crate::index::view::IndexView; use crate::path::RelPath; use crate::store::{Store, StoreError}; @@ -129,7 +130,7 @@ pub fn dependents_of(store: &Store, module: &str) -> Result, QueryE /// and are therefore unavailable in the fallback (same caveat as `goto_definition`'s cross-file /// hop). pub fn resolved_references(store: &Store, def_path: &RelPath, def_start: u32) -> Vec<(RelPath, u32)> { - match store.index_db.as_ref() { + match store.index_view() { Some(index) => index.references_to(def_path, def_start), None => resolved_references_from_blob(store, def_path, def_start), } @@ -162,7 +163,7 @@ fn resolved_references_from_blob(store: &Store, def_path: &RelPath, def_start: u /// Fjall `refs_by_path` partition when the index is open, else the file's `.rref` blob (intra-file /// bindings only; cross-file targets need the index open). pub fn definition_of(store: &Store, use_path: &RelPath, use_start: u32) -> Option<(RelPath, u32)> { - match store.index_db.as_ref() { + match store.index_view() { Some(index) => index.definition_of(use_path, use_start), None => definition_of_from_blob(store, use_path, use_start), } diff --git a/src/scanner.rs b/src/scanner.rs index 1fc2c361..8a170cd2 100644 --- a/src/scanner.rs +++ b/src/scanner.rs @@ -783,10 +783,9 @@ pub fn scan_paths_with_observer( let nothing_removed = removed.is_empty() && doc_removed.is_empty(); #[cfg(not(feature = "documents"))] let nothing_removed = removed.is_empty(); - let other = store - .index_db - .as_ref() - .map_or_else(Vec::new, |db| crate::intel::resolve_pass::probed_only(db, other)); + let other = store.index_db.as_ref().map_or_else(Vec::new, |db| { + crate::intel::resolve_pass::probed_only(&db.view(), other) + }); if rels.is_empty() && nothing_removed && other.is_empty() { return Ok(ScanReport::default()); } diff --git a/src/scanner_index_batch.rs b/src/scanner_index_batch.rs index 17ac82c2..20e64061 100644 --- a/src/scanner_index_batch.rs +++ b/src/scanner_index_batch.rs @@ -372,16 +372,16 @@ mod tests { /// compared byte for byte. fn dump_all(db: &IndexDb) -> Vec, Vec)>> { vec![ - dump(&db.symbols_by_path), - dump(&db.symbols_by_name), - dump(&db.calls_by_path), - dump(&db.calls_by_callee), - dump(&db.imports_by_module), - dump(&db.imports_by_path), - dump(&db.implementations_by_trait), - dump(&db.implementations_by_path), - dump(&db.code_bm25_postings), - dump(&db.code_bm25_by_path), + dump(db.keyspace_for_test(crate::index::view::Part::SymbolsByPath)), + dump(db.keyspace_for_test(crate::index::view::Part::SymbolsByName)), + dump(db.keyspace_for_test(crate::index::view::Part::CallsByPath)), + dump(db.keyspace_for_test(crate::index::view::Part::CallsByCallee)), + dump(db.keyspace_for_test(crate::index::view::Part::ImportsByModule)), + dump(db.keyspace_for_test(crate::index::view::Part::ImportsByPath)), + dump(db.keyspace_for_test(crate::index::view::Part::ImplementationsByTrait)), + dump(db.keyspace_for_test(crate::index::view::Part::ImplementationsByPath)), + dump(db.keyspace_for_test(crate::index::view::Part::CodeBm25Postings)), + dump(db.keyspace_for_test(crate::index::view::Part::CodeBm25ByPath)), ] } @@ -403,7 +403,10 @@ mod tests { "the file-count bound must not be what committed here" ); assert!( - db.symbols_by_path.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_some(), "the byte budget must have forced a commit already" ); drop(batch); @@ -426,13 +429,19 @@ mod tests { assert!(batch.stage(&RelPath::from(format!("src/f{i}.rs")), &l1, None)); } assert!( - db.symbols_by_path.iter().next().is_none(), + db.keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_none(), "nothing may commit before the file counter reaches the bound" ); assert!(batch.stage(&RelPath::from("src/last.rs"), &l1, None)); assert!( - db.symbols_by_path.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_some(), "the {INDEX_COMMIT_BATCH}th file must force the commit" ); assert_eq!(batch.staged, 0, "the file counter resets at its own commit"); @@ -463,13 +472,22 @@ mod tests { let mut batch = WorkerIndexBatch::with_budget(Some(&db), INDEX_COMMIT_BATCH_BYTES, &SCAN_STAGED_BYTES); let mut files = 0usize; - while db.symbols_by_path.iter().next().is_none() && files < INDEX_COMMIT_BATCH { + while db + .keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_none() + && files < INDEX_COMMIT_BATCH + { assert!(batch.stage(&RelPath::from(format!("src/f{files}.rs")), &l1, None)); files += 1; } assert!( - db.symbols_by_path.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_some(), "staging {INDEX_COMMIT_BATCH} bulky files must have committed something" ); assert!( @@ -512,7 +530,11 @@ mod tests { let mut under_budget = WorkerIndexBatch::with_budget(Some(&control), 8 * 1024 * 1024, &LEDGER); under_budget.stage_bm25(&RelPath::from("src/small.rs"), &postings(1, 4)); assert!( - control.code_bm25_postings.iter().next().is_none(), + control + .keyspace_for_test(crate::index::view::Part::CodeBm25Postings) + .iter() + .next() + .is_none(), "staging well under the byte budget must leave the batch uncommitted" ); drop(under_budget); @@ -523,7 +545,10 @@ mod tests { assert_eq!(batch.staged, 0, "bm25 staging must not bump the file counter"); assert!( - db.code_bm25_postings.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::CodeBm25Postings) + .iter() + .next() + .is_some(), "bm25 staging alone must be able to force a commit" ); drop(batch); @@ -559,7 +584,10 @@ mod tests { "the file-count bound must not be what committed here" ); assert!( - db.symbols_by_path.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_some(), "the ceiling commit must have reached disk" ); worker.finish(); diff --git a/src/search/bm25.rs b/src/search/bm25.rs index 8453ab28..c12b4097 100644 --- a/src/search/bm25.rs +++ b/src/search/bm25.rs @@ -19,7 +19,8 @@ use std::cmp::Ordering; use ahash::{AHashMap, AHashSet}; use crate::chunk::CodeChunk; -use crate::index::{IndexDb, keys}; +use crate::index::keys; +use crate::index::view::{IndexView, Part}; /// Upper bound on a single token's byte length. Tokens longer than this (minified blobs, base64 /// spilled into a comment) are dropped — they are never real query terms and would bloat the index. @@ -134,7 +135,7 @@ pub fn bm25_term_score(tf: u32, doclen: u32, avgdl: f32, idf: f32) -> f32 { /// Score `query` against the BM25 keyword index in `db` and return the top `limit` chunk hits, /// highest score first (ties broken by `chunk_id` for a stable order). Returns empty when the index /// is empty, the query has no tokens, or no chunk matches. -pub fn bm25_search(db: &IndexDb, query: &str, limit: usize) -> Vec { +pub fn bm25_search(db: &dyn IndexView, query: &str, limit: usize) -> Vec { let terms = tokenize_query(query); if terms.is_empty() || limit == 0 { return Vec::new(); @@ -150,8 +151,8 @@ pub fn bm25_search(db: &IndexDb, query: &str, limit: usize) -> Vec { let mut scores: AHashMap = AHashMap::new(); for term in &terms { let mut postings: Vec<(String, u32, u32)> = Vec::new(); - for guard in db.code_bm25_postings.prefix(keys::code_bm25_postings_prefix(term)) { - let (k, v) = match guard.into_inner() { + for guard in db.prefix(Part::CodeBm25Postings, &keys::code_bm25_postings_prefix(term)) { + let (k, v) = match guard { Ok(kv) => kv, Err(error) => { tracing::warn!(%term, ?error, "bm25: posting read failed; skipping entry"); diff --git a/src/search/exact.rs b/src/search/exact.rs index 015a20fd..4af11c48 100644 --- a/src/search/exact.rs +++ b/src/search/exact.rs @@ -11,7 +11,7 @@ use ahash::{AHashMap, AHashSet}; use crate::chunk::{CodeChunk, CodeChunkBlob}; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::path::RelPath; use crate::store::Store; @@ -50,7 +50,7 @@ pub fn owning_chunk_id(chunks: &[CodeChunk], byte_offset: u32) -> Option<&str> { /// ranked exact-name matches ahead of longer prefix matches (`parseConfig` before `parseConfigFile`). /// Returns an empty vec for non-identifier queries or when nothing resolves. `cap` bounds the symbol /// scan. Reads chunk sidecars — blocking I/O, so call off the async reactor. -pub fn exact_lane_chunk_ids(store: &Store, db: &IndexDb, query: &str, cap: usize) -> Vec { +pub fn exact_lane_chunk_ids(store: &Store, db: &dyn IndexView, query: &str, cap: usize) -> Vec { if !is_identifier_query(query) { return Vec::new(); } diff --git a/src/store.rs b/src/store.rs index 454ada69..49652739 100644 --- a/src/store.rs +++ b/src/store.rs @@ -7,6 +7,7 @@ use serde::{Deserialize, Serialize}; use thiserror::Error; use crate::extract::SCHEMA_VER; +use crate::index::view::IndexView; use crate::index::{IndexDb, IndexError}; #[cfg(feature = "intelligence")] use crate::lance::LanceStore; @@ -305,6 +306,12 @@ pub struct Store { } impl Store { + /// A read view over the fjall index, or `None` when this session holds none. Every index read + /// goes through it (see [`crate::index::view`]). + pub fn index_view(&self) -> Option> { + self.index_db.as_ref().map(IndexDb::view) + } + /// Open the store for a specific view. View names are flat strings: `"working"`, /// `"staged"`, `"rev-"`. Each view has its own `index.msgpack` under /// `.basemind/views//`; blobs are shared in `.basemind/blobs/`. @@ -357,7 +364,7 @@ impl Store { }; let index_db = open_index_with_retry(root, &view_dir)?; let index_rebuild_needed = - !index.files.is_empty() && (index_db.wiped_on_open() || index_db.symbols_index_is_empty()); + !index.files.is_empty() && (index_db.wiped_on_open() || index_db.view().symbols_index_is_empty()); if index_rebuild_needed { index.files.clear(); } diff --git a/tests/scan_smoke.rs b/tests/scan_smoke.rs index f83dd682..fe48a62e 100644 --- a/tests/scan_smoke.rs +++ b/tests/scan_smoke.rs @@ -1206,7 +1206,7 @@ fn deferred_embed_mode_indexes_symbols_and_keyword_lane_but_writes_no_vectors() assert_eq!(hits.len(), 1, "deferred scan must still index symbols"); let db = store.index_db.as_ref().expect("index db present"); - let keyword = basemind::search::bm25::bm25_search(db, "parse", 10); + let keyword = basemind::search::bm25::bm25_search(&db.view(), "parse", 10); assert!(!keyword.is_empty(), "deferred scan must populate the BM25 keyword lane"); assert!( From 5a005be3445b9a9284f5af31c707ed1ccf936e38 Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Mon, 19 Oct 2026 14:19:43 +0200 Subject: [PATCH 04/10] fix(index): migrate grep_bloom rows by row version instead of a schema revision bump Co-Authored-By: Claude Opus 5.5 --- src/index/grep_bloom.rs | 17 +++--- src/index/mod.rs | 10 ++-- tests/grep_bloom_scan.rs | 108 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 123 insertions(+), 12 deletions(-) diff --git a/src/index/grep_bloom.rs b/src/index/grep_bloom.rs index 7920e074..64fcc344 100644 --- a/src/index/grep_bloom.rs +++ b/src/index/grep_bloom.rs @@ -20,9 +20,10 @@ use regex_syntax::hir::literal::{ExtractKind, Extractor}; use super::IndexDb; use crate::path::RelPath; -/// Row layout revision (first byte). A row with any other value is treated as absent, so a future -/// layout change needs no index wipe: old rows read as candidates until the file is next scanned. -const ROW_VERSION: u8 = 1; +/// Row layout revision (first byte). A row with any other value is treated as absent, so a layout +/// change needs no index wipe: old rows read as candidates until a scan sweeps and rebuilds them. +/// 1 was the path-keyed row carrying a `(size, mtime_ns)` stamp; 2 is the content-hash-keyed row. +const ROW_VERSION: u8 = 2; /// `version(1)`, then the bit array. const HEADER_BYTES: usize = 1; /// Bloom bytes per source byte is `1 / BYTES_DIVISOR` (ADR-0012 sizing: 1/8 keeps the candidate set @@ -179,14 +180,16 @@ impl IndexDb { .is_some_and(|row| parse_row(&row).is_some()) } - /// Delete every row whose hash is not in `live`, returning how many were dropped. Rows are shared - /// by every path holding the content, so only a hash no indexed file references may go. + /// Delete every row whose hash is not in `live` or whose layout is not [`ROW_VERSION`], returning + /// how many were dropped. Rows are shared by every path holding the content, so only a hash no + /// indexed file references may go. Path-keyed rows from before the content-hash re-key are never + /// in `live`, and older-version rows are unreadable, so both leave here. pub fn sweep_grep_bloom(&self, live: &ahash::AHashSet<&str>) -> Result { let mut writer = self.writer(); let mut dropped = 0; for guard in self.grep_bloom.iter() { - let (key, _) = guard.into_inner()?; - if std::str::from_utf8(&key).is_ok_and(|hash| live.contains(hash)) { + let (key, row) = guard.into_inner()?; + if parse_row(&row).is_some() && std::str::from_utf8(&key).is_ok_and(|hash| live.contains(hash)) { continue; } writer.remove_grep_bloom(&key); diff --git a/src/index/mod.rs b/src/index/mod.rs index 17220f2b..980441eb 100644 --- a/src/index/mod.rs +++ b/src/index/mod.rs @@ -46,9 +46,8 @@ use thiserror::Error; /// into its meta keyspace at *create* time and recovers it on reopen, ignoring the create options /// for a keyspace that already exists, so an index built by an earlier build would keep the 64 MiB /// default forever unless the schema mismatch forces it to be recreated; `+6` the LZ4-at-every-level -/// block compression ([`index_keyspace_options`]), persisted at create time for the same reason; -/// `+7` the `grep_bloom` re-key from path to content hash, whose old rows would read as foreign hashes. -const INDEX_PARTITION_REVISION: u32 = 7; +/// block compression ([`index_keyspace_options`]), persisted at create time for the same reason. +const INDEX_PARTITION_REVISION: u32 = 6; /// Bumped whenever the on-disk key layout changes — the sum of `RELEASE_MINOR` and the /// [`INDEX_PARTITION_REVISION`] offset, monotonic across both. When `RELEASE_MINOR` next bumps, @@ -338,8 +337,9 @@ pub struct IndexDb { pub(in crate::index) code_bm25_by_path: Keyspace, /// `grep_bloom`: hex content hash -> trigram bloom filter of that content (see [`grep_bloom`]), /// shared by every path holding the bytes. A candidate PREFILTER for `code grep` only; a missing - /// row means "candidate", which is why this partition was added without a schema bump and - /// backfills on rescan. + /// row (including an older-version or path-keyed one) means "candidate", which is why the layout + /// moved on by `ROW_VERSION` with no schema bump: the scan's sweep drops stale rows and + /// backfills the current ones. pub(in crate::index) grep_bloom: Keyspace, #[allow(dead_code)] pub(crate) embeddings: Keyspace, diff --git a/tests/grep_bloom_scan.rs b/tests/grep_bloom_scan.rs index ab4f84dc..857fbef1 100644 --- a/tests/grep_bloom_scan.rs +++ b/tests/grep_bloom_scan.rs @@ -153,3 +153,111 @@ fn rescan_backfills_a_row_an_older_index_lacks() { let store = scan(root); assert!(has_row(&store, &hash), "rescan backfills the row"); } + +/// A row exactly as the path-keyed layout (`ROW_VERSION` 1) wrote it: +/// `version(1) | size(8) | mtime_ns(8)`, then an all-zero bit array. All-zero bits reject every +/// needle, so a reader that trusted this row would wrongly skip the file. +fn v1_row() -> Vec { + let mut row = vec![0u8; 17 + 64]; + row[0] = 1; + row +} + +fn put_raw(store: &Store, key: &str, row: Vec) { + let idx = store.index_db.as_ref().expect("index"); + let mut writer = idx.writer(); + writer.upsert_grep_bloom(key, row); + writer.commit().expect("commit"); +} + +fn raw_row(store: &Store, key: &str) -> Option> { + store + .index_db + .as_ref() + .and_then(|idx| idx.grep_bloom_row(key)) + .map(|row| row.to_vec()) +} + +fn grep_entry(store: &Store, root: &Path, rel: &str) -> basemind::index::grep_bloom::GrepEntry { + let entry = store.lookup(RelPath::from(rel)).expect("indexed file"); + basemind::index::grep_bloom::GrepEntry { + rel: RelPath::from(rel), + hash_hex: hash_of(root, rel), + size_bytes: entry.size_bytes, + mtime_ns: entry.mtime, + } +} + +/// Replace the current row set with what an index built before the content-hash re-key holds: a +/// version-1 row under the path, and (as an A2 build wrote) a version-1 row under the hash. +fn downgrade_to_v1(store: &Store, rel: &str, hash: &str) { + store + .index_db + .as_ref() + .expect("index") + .drop_grep_bloom_row_for_test(hash); + put_raw(store, rel, v1_row()); + put_raw(store, hash, v1_row()); + assert_eq!(raw_row(store, rel).expect("path row")[0], 1, "fixture writes a v1 row"); + assert_eq!(raw_row(store, hash).expect("hash row")[0], 1, "fixture writes a v1 row"); +} + +#[test] +fn v1_rows_are_absent_on_read_so_grep_scans_every_file() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); + let hash = hash_of(root, "a.rs"); + let store = scan(root); + let entry = grep_entry(&store, root, "a.rs"); + + let absent = basemind::index::grep_bloom::Needle::compile("zzz_absent_literal").expect("needle"); + let present = basemind::index::grep_bloom::Needle::compile("alpha_function_name").expect("needle"); + let idx = store.index_db.clone().expect("index"); + let verdicts = |needle| idx.grep_skip_verdicts(root, std::slice::from_ref(&entry), needle); + assert_eq!( + verdicts(&absent), + vec![true], + "a current row skips a file lacking the literal" + ); + + downgrade_to_v1(&store, "a.rs", &hash); + assert_eq!( + verdicts(&absent), + vec![false], + "a v1 row (which rejects everything) must not skip" + ); + assert_eq!(verdicts(&present), vec![false]); +} + +#[test] +fn the_next_incremental_scan_sweeps_v1_rows_and_rebuilds_content_keyed_ones() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); + let hash = hash_of(root, "a.rs"); + let store = scan(root); + downgrade_to_v1(&store, "a.rs", &hash); + drop(store); + + let store = scan(root); + assert!(raw_row(&store, "a.rs").is_none(), "the path-keyed row is swept"); + let rebuilt = raw_row(&store, &hash).expect("rebuilt under the content hash"); + assert_eq!(rebuilt[0], 2, "the v1 row under the hash was replaced by a v2 row"); + assert_eq!(row_count(&store), 1); + let entry = grep_entry(&store, root, "a.rs"); + let absent = basemind::index::grep_bloom::Needle::compile("zzz_absent_literal").expect("needle"); + let idx = store.index_db.clone().expect("index"); + assert_eq!(idx.grep_skip_verdicts(root, &[entry], &absent), vec![true]); +} + +#[test] +fn migrating_grep_bloom_rows_does_not_change_the_index_schema_version() { + assert_eq!( + basemind::index::INDEX_SCHEMA_VER, + basemind::version::RELEASE_MINOR as u32 + 6, + "pre-A2 value: a schema bump would force every user to re-index" + ); +} From 234cdc81112fe29eaef8b1a0ee98e3f1e3229a5f Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Mon, 19 Oct 2026 14:19:50 +0200 Subject: [PATCH 05/10] docs(adr): address review of the layered index specification Co-Authored-By: Claude Opus 5.5 --- ...shared-base-index-and-worktree-overlays.md | 319 ++++++++++++++---- 1 file changed, 261 insertions(+), 58 deletions(-) diff --git a/docs/adr/0016-shared-base-index-and-worktree-overlays.md b/docs/adr/0016-shared-base-index-and-worktree-overlays.md index 82a469e2..85551386 100644 --- a/docs/adr/0016-shared-base-index-and-worktree-overlays.md +++ b/docs/adr/0016-shared-base-index-and-worktree-overlays.md @@ -24,9 +24,12 @@ and a per-worktree **overlay**: tree, so it never contains uncommitted edits. The tree is exported to a scratch directory and scanned like a working tree, because the rev scan alone does not run the resolve pass. Generations advance by a CoW clone plus `git diff-tree C_n..C_n+1` applied through the incremental writer when files have - drifted by 2% or more, or daily; at most three are live, each published once and never written again, - so the daemon keeps one handle and one block cache per group (fjall admits one process per directory, - so other processes reach the base through the daemon). + drifted by 2% or more, or daily; at most three are live, each published once and never logically + rewritten (a Fjall generation still replays its journal and compacts when opened, see *Lifecycle*). + The base is read through a `BaseReader`, so its storage is pluggable (*Base storage*). With a Fjall + base the daemon keeps one handle and one block cache per group (fjall admits one process per + directory, so other processes reach the base through the daemon); an immutable mmapped base can be + opened by every process. - **Overlay** — `delta.msgpack` records each path as added, modified or deleted relative to the base manifest by content hash; `overlay.fjall` holds rows only for changed paths and re-stitched importers. Shadow sets `S_path` (delta paths), `S_refs` (`S_path` plus re-stitched importers) and `D_hash` (base @@ -48,17 +51,17 @@ and a per-worktree **overlay**: ## Specification -Everything below is normative for A2–A8. Paths and symbols are as of 0.30.0 (`bd1f0a5b`). Where the code -contradicts the text of issue #87 or the summary above, the code wins and the correction is listed first. +Everything below is normative for A2–A8. Paths and symbols are as of 0.30.0 (`bd1f0a5b`), plus A2 (`efe41b76`) +and A3 (`26be3d56`) on this branch. Where the code contradicts the text of issue #87 or the summary above, the code wins and the correction is listed first. ### Corrections to the issue text | Claim in #87 | What the code does | Consequence | |---|---|---| | "Rev scan builds the base" (`ScanSource::Rev`) | `scan_with_cancel` runs the resolve pass only for `ScanSource::WorkingTree` (`scanner.rs`, `if matches!(source, ScanSource::WorkingTree)`), and the resolver reads the filesystem (`intel/resolve_pass.rs` `std::fs::read`, `intel/resolver.rs` `root.join`, `canonicalize`). A rev scan therefore writes no `refs_by_*`, `import_targets` or `importers_by_probe` rows, and the rev branch of `process_file` writes no grep bloom either. | The base builder exports the committed tree to a private scratch directory and runs a normal working-tree scan on it (A4). A virtual git-tree filesystem for the resolver was rejected: it duplicates the resolver's I/O surface for a transient 1-3 GB export. | -| "Published read-only so the daemon keeps one handle" | Fjall's directory lock is exclusive even for a read-only open (`Store::open_read_only_no_index`, `mcp/index_route.rs` module doc). Read-only is a logical property, not something the OS enforces on a shared fjall. | One process per machine can hold a given base generation. The daemon is that holder; every other process reaches the base through `IndexRoute::Host` / `Daemon` or degrades to `InRam`, exactly as a second `serve` does today (A4). | +| "Published read-only so the daemon keeps one handle" | Fjall's directory lock is exclusive even for a read-only open (`Store::open_read_only_no_index`, `mcp/index_route.rs` module doc). Read-only is a logical property, not something the OS enforces on a shared fjall. | For a Fjall base, one process per machine can hold a given generation. The daemon is that holder; every other process reaches the base through `IndexRoute::Host` / `Daemon` or degrades to `InRam`, exactly as a second `serve` does today (A4). Conditional on the base-storage prototype: an mmapped sorted-run base has no such lock (*Base storage*). | | "`code_chunks` is already content-addressed" | `chunk_id` is `:` (`chunk.rs`) and the sidecar blobs are content-addressed, but the Lance rows are keyed `(scope, path)` (`lance/mod.rs` `replace_code_chunks`, `delete_code_chunks`) and `scope` is the normalised `origin` URL (`git/remote.rs` `scope_key`), identical for every worktree of a clone. | Sharing a Lance table needs a new key for all three tables, not just `documents`/`doc_links` (A7). | -| A3 "shipped" (progress comment) | Only the ADR shipped. `src/index/view.rs` does not exist; `IndexDb` keyspaces are `pub(crate)` and read directly (`mcp/helpers_calls_scan.rs` `calls_by_path`, `mcp/helpers_calls.rs`, `search/bm25.rs` `code_bm25_postings`, `intel/stage_budget.rs`). | A3 is still a code phase and a prerequisite of A5. | +| A3 "shipped" (progress comment) | When the ADR was written only the ADR had shipped: `src/index/view.rs` did not exist and `IndexDb` keyspaces were read directly. A2 landed in `efe41b76` and A3 in `26be3d56`. | A3 is a prerequisite of A5 and is done, but `view.rs` still returns `fjall::Slice` and its named accessors are required methods (A3 follow-up below). | | `repo_group_key` hashes "the git common dir" | It hashes the canonical main worktree root, the parent of the common dir (`store_layout.rs` `repo_group_key`, `git/mod.rs` `main_worktree_root`). Equivalent for a normal clone; a bare clone or a moved main checkout re-keys the group. | Keep the function; record the resolved common dir in `group.json` so a re-key is diagnosable (A4). | | BM25 `meta` is exact as "base + overlay - shadowed" | True, but `upsert_bm25_file` / `remove_bm25_file` delete every posting of the path's `chunk_id`s (`index/writer.rs` `stage_bm25_deletes_for`), and `chunk_id` embeds the content hash. Two paths with identical bytes share their postings, so removing or editing one of them deletes the postings the other still needs. | Pre-existing defect in the standalone index, not specific to layering, and it breaks the A5 equivalence gate on the "duplicate content" scenario. A5.0 fixes it first (below). | | `main_root_of` / `repo_group_key` cover a workspace | A workspace rooted at a *subdirectory* of a repository also gets a group key (`Repo::discover` walks up), but a base built from the whole tree would not match a subdirectory scan. | Layering applies only when the canonical root equals the git workdir (the `store_seed::main_root_of` predicate); otherwise the workspace stays standalone (A4). | @@ -78,13 +81,16 @@ contradicts the text of issue #87 or the summary above, the code wins and the co The plan in #87 stands; this fixes the details the plan left open. - **Key and row.** Key = the 64-character lowercase hex blake3 (`hashing::hex`), so two paths with identical - bytes share one row. Row = `version(1) | bits`; `ROW_VERSION` stays 1 because the partition-revision bump - (below) removes every old row. `build_row(bytes)`; `GrepEntry { rel, hash_hex, size_bytes, mtime_ns }` + bytes share one row. Row = `version(1) | bits`; `ROW_VERSION` goes 1 to 2, so every old row (path-keyed, + 17-byte header) parses as absent and reads as a candidate. `build_row(bytes)`; `GrepEntry { rel, hash_hex, size_bytes, mtime_ns }` (`index/grep_bloom.rs`, serde, because it crosses the daemon wire). - **Live-file guard.** `grep_skip_verdicts(root, &[GrepEntry], needle)` skips a file only if the row for `entry.hash_hex` rejects the needle *and* `stat(root/rel)` equals the **indexed** `(size_bytes, mtime_ns)` of the entry, taken from `FileEntry` (`store.rs`) through `FileMeta` (`mcp/l1_cache.rs`, gains `mtime_ns`). A file - whose indexed mtime is 0 (git sources) never equals a live stat and stays a candidate. + whose indexed mtime is 0 (git sources) never equals a live stat and stays a candidate: `mtime_ns == 0` is never skipped. + A file whose mtime is within 2 s of the scan's stat time is untrusted for skipping (the same-tick edit hole: an edit + inside the filesystem's timestamp granularity can leave `(size, mtime)` unchanged), so it stays a candidate until a + later scan restamps it. - **Stamp refresh (not in the plan).** The stamp used to live in the row and `restamp_grep_bloom` refreshed it when a file was touched with unchanged content. The stamp now lives in the `FileEntry`, and `process_file` returned `Unchanged` without upserting one, so a touched file would have stayed a permanent @@ -96,11 +102,14 @@ The plan in #87 stands; this fixes the details the plan left open. - **Sweep.** `IndexDb::sweep_grep_bloom(&live)` deletes every row whose hash is not in `live`. `scanner::scan_with_cancel` calls it right after the stale-entry purge with `live` = hashes of `store.index.files`, so a pass that was cancelled or aborted (which returns before the purge) never sweeps. Edited and deleted contents leave their - row behind until that sweep, which is the only deletion path. -- **Schema.** `INDEX_PARTITION_REVISION` 6 to 7 (`index/mod.rs`). Cost: every user re-indexes once (a 17-33 min - scan on armis) because the bump wipes the whole fjall. A cheaper option exists (ROW_VERSION 2, no bump, the - sweep deletes the path-keyed rows) and has precedent in the keyspace's own doc comment (added "without a - schema bump"); the bump is kept because the plan asks for it and it removes any mixed-format window. + row behind until that sweep, which is the only deletion path for a full scan. The sweep is key-only (it never decodes a + row) and runs only for `WorkingTree` scans, because a rev scan writes no bloom rows and would sweep them all. The + watcher-driven `scan_paths` also deletes a content row when its hash leaves the manifest (no other path holds it), so + incremental edits do not wait for the next full pass. +- **Schema.** `INDEX_PARTITION_REVISION` stays 6 and `INDEX_SCHEMA_VER` is unchanged, so no user re-indexes. Old + rows are migrated by `ROW_VERSION` alone: a row of another version is absent on read (the file is a candidate and + is scanned), a one-time version sweep, guarded by a `meta` marker so it runs once per index, deletes path-keyed and + other-version rows, and an incremental scan rebuilds the content-keyed rows (backfill reads each unchanged file once). - **Wire.** `IndexReadQuery::GrepBloom { pattern, entries }` (`comms/index_read_proto.rs`). A client and daemon of different builds fail to decode, and `IndexRoute::grep_skip` already degrades that to "sweep every file". - **Layered note.** Rows are content-addressed and never shadowed: a layered read does `overlay.get(h)` then @@ -108,6 +117,8 @@ The plan in #87 stands; this fixes the details the plan left open. ### A3: `IndexView` (standalone only) +Shipped in `26be3d56`; the follow-ups below are required before A4 and are being applied on this branch. + - `src/index/view.rs`: `enum Part` naming the 16 layered keyspaces (below), and `trait IndexView: Send + Sync { get(Part, &[u8]); prefix(Part, &[u8]); range(Part, Bound, Bound); iter(Part); }` returning `Box> + '_>`, plus the named accessors that exist today @@ -118,6 +129,18 @@ The plan in #87 stands; this fixes the details the plan left open. `query.rs`, `mcp/helpers_calls*.rs`, `mcp/helpers_impls.rs`, `search/{bm25,exact}.rs` and the resolve pass (`intel/resolve_pass.rs`, which reads `import_targets`, `importers_by_probe`, `refs_*`) take `&dyn IndexView`. Writes keep going through `IndexWriter` on the concrete `IndexDb`. +- **Backend-neutral values (follow-up, required by A3 and A4).** `KvResult`, `get` and `grep_bloom_row` in + `src/index/view.rs` return `fjall::Slice`, and `kv` takes a `fjall::Guard`. A follow-up commit on this branch replaces + them with a backend-neutral value type (cheap to clone, derefs to `[u8]`), so that no `fjall` type appears in the + trait and a sorted-run `BaseReader` can implement it without copying into Fjall types. Only `StandaloneView` and the + Fjall base reader convert at their edge. +- **Named accessors are default methods (follow-up).** Today `references_to`, `definition_of`, `import_targets_of`, + `importers_probing`, `is_probed`, `import_probes_complete`, `symbols_by_name_lookup`, `bm25_stats` and + `symbols_index_is_empty` are required methods that `StandaloneView` forwards to `IndexDb`. They become default methods + written over the generic `get` / `prefix` / `range` layer (the `Meta` scalars through `get(Part::Meta, ..)`), so a + layered view inherits them and cannot drift from the standalone decoding. `StandaloneView` may override a default only + for speed and the equivalence gate proves the override equal. `callee_keys_matching` / `trait_keys_matching` stay + required: a layered view must merge them (A5, *NameDict*). - The existing suite is the gate. Perf discipline applies: the dyn call is per iterator, not per row, but a p95 regression above 3% on `references` / `callers` (benches in `benches/`) switches `IndexView` to an enum with the two variants instead of a trait object. @@ -127,10 +150,37 @@ The plan in #87 stands; this fixes the details the plan left open. **Config.** New `[index]` table in `config/v1.rs` (`IndexConfig`, `deny_unknown_fields`, in `ConfigV1` after `cache`): `layered: bool`, default `false` until A8. Regenerate `schema/` with the existing `config_schema` test. The env var `BASEMIND_INDEX_LAYERED=0|1` overrides the file (rollback lever, same pattern as -`BASEMIND_NO_SEED`). The mode is resolved once per `Store::open`; the daemon applies a changed value when it next +`BASEMIND_NO_SEED`). `max_group_bytes: u64` caps a group's `base/` plus `overlays/` (*Disk budget*). The mode is resolved once per `Store::open`; the daemon applies a changed value when it next opens the workspace. Layered applies to a workspace only when all hold: `layered` is true, the canonical root is the git workdir, `Repo::discover` succeeds, a base commit resolves, and no embedding lane is enabled until A7 has -shipped (`documents.embed`, `code_search.embed`: their Lance rows are still keyed by worktree). +shipped (`documents.embed`, `code_search.embed`: their Lance rows are still keyed by worktree). `documents.embed` +defaults to `true` (`config/documents.rs` `default_embed`), `code_search.embed` to `false`, so out of the box layering +is off until A7 and A4–A6 apply only to users who set `documents.embed = false`. The armis measurement for A4–A6 must +run with embeds off, and A8 (layered by default) depends on A7. + +**Base storage (pluggable; prototype-first).** The base is read through a `BaseReader` behind `IndexView`; the overlay +and every mutable store stay Fjall. `base.json` carries `storage = "fjall" | "sorted-run"` and a reader is chosen per +generation from it. Owner decision: the base is prototype-first. An immutable mmapped sorted-run base, exported from +a finished Fjall generation, is built and benchmarked before A4 is implemented, and A4 must not be locked to a Fjall +base. Everything below that is Fjall-specific is marked *(Fjall option)* and is conditional on the prototype outcome: +the `index.fjall/` layout, `IndexDb::open_code_only`, the quiesce and `RwLock>` in *Advance*, the +generation clone, the daemon-only access rationale, and open question 1 and the matching Consequences text about one +process per base. A mmapped base can be opened by every process, which would mostly dissolve open question 1. + +*A4-prototype gate.* Export one armis base generation to a sorted-run file set and measure, against the Fjall +generation, on the same machine, cold and warm: + +| Measure | Acceptance (to be fixed by the owner from the first run) | +|---|---| +| base size on disk | no larger than the Fjall generation | +| build time (export from Fjall, and cold build) | recorded; export under 10 min | +| `references` / `callers` p95 through the `BaseReader` | within the 3% / 10% budgets of A3 and the measurement plan | +| open cost (time to first answer, RSS) | below Fjall open incl. journal replay | +| multi-process open | N processes open one generation concurrently, no lock, each reads correctly | + +If sorted-run meets the gate it becomes the default storage and the *(Fjall option)* items are dropped from the +spec; otherwise Fjall stays and the text applies as written. The `IndexView` and merge contract below are written +against `BaseReader` and hold for both. **Layout.** @@ -140,7 +190,7 @@ cache/repos// group.json {format, main_root, common_dir, created_unix} base.lock flock; serialises build / advance / retire / GC for the group base/ - gen-/{index.fjall/, index.msgpack, base.json} published, never rewritten + gen-/{index.fjall/ | index.run/, index.msgpack, base.json} published (index.fjall/ is the Fjall option) .tmp--/ builder staging (incl. checkout/); swept after 1 h when unlocked .retired-/ renamed out of selection, deleted once fjall can be locked last-failure.json {error, unix}; a failed build is not retried for 10 min @@ -185,8 +235,14 @@ the main worktree (`Repo::main_worktree_root`). basemind never fetches. 3. Run `scanner::scan` with `ScanSource::WorkingTree` rooted at the checkout, `EmbedMode` off, into a private `Store` whose view dir is `.tmp-*/`; this runs the wholesale resolve pass and sets `import_probe_ver`. 4. Normalise mtimes, rewrite `index.msgpack` atomically, flush every keyspace (the `rotate_memtable_and_wait` - loop in `IndexDb::open_inner`), close fjall, write `base.json` (tmp + rename inside `.tmp-*`), fsync the directory. -5. `rename(.tmp-*, gen-)` (atomic, same filesystem), then update `CURRENT` (tmp + rename, an advisory hint only). + loop in `IndexDb::open_inner`) and close fjall *(Fjall option)*. Closing makes the SSTs and the journal durable; the + builder then fsyncs every file under `.tmp-*`, so nothing published depends on page-cache state. +5. Write `base.json` last: write `base.json.tmp`, fsync it, rename it to `base.json`, fsync `.tmp-*`. +6. `rename(.tmp-*, gen-)` (atomic, same filesystem), then fsync the parent `base/` directory so the rename is + durable, then update `CURRENT` (tmp + rename, an advisory hint only). Every fsync is `F_FULLFSYNC` on macOS + (plain `fsync` does not flush the drive cache there; `src/store.rs` uses `sync_all` only, so this is new). + A crash before the rename leaves a `.tmp-*`; after it, a complete generation. A generation without a parseable + `base.json` is never ready. Readers never trust `CURRENT`; they list `base/gen-*` and read `base.json`. Blobs written by an aborted build stay valid (content-addressed), so a restart re-runs mostly fjall writes. A disk-full or poisoned fjall (`IndexDb::is_poisoned`, @@ -208,43 +264,93 @@ not a rescan, and the first base costs one CoW clone instead of 17-33 min. **Advance (new generation from an old one).** Under `base.lock`, for the newest ready generation `k` and target `m`: -1. Quiesce `k`: if this process is the daemon it takes the write side of the group's `RwLock>`, - drains, and drops its handle; otherwise it tries to open `k`'s fjall and aborts the advance on `Locked`. A live - fjall is never copied (the rule in the `store_seed.rs` module doc). -2. Clone `gen-k` to `.tmp-*` and reopen the daemon handle immediately. +1. *(Fjall option)* Quiesce `k`: if this process is the daemon it takes the write side of the group's + `RwLock>`, drains, and drops its handle; otherwise it tries to open `k`'s fjall and aborts the advance + on `Locked`. A live fjall is never copied (the rule in the `store_seed.rs` module doc). Readers of the group stall + while the write side is held. The stall is bounded by a budget, `[index] advance_stall_ms` (default 10000), covering + the drain of in-flight queries, the CoW clone (O(files)) and the reopen of `k` (open cost, which includes + journal replay and the `rotate_memtable_and_wait` flush in `open_inner`, see *Lifecycle*). Exceeding it aborts the + advance, reopens `k`, and retries after a backoff; readers are never stalled past the budget. A sorted-run base needs + no quiesce: its files are immutable, so the clone is a hardlink or CoW copy and no reader is touched. +2. Clone `gen-k` to `.tmp-*` and, for Fjall, reopen the daemon handle immediately. 3. `Repo::diff_trees(C_k, C_m)` (new, gix tree diff; a rename is a delete plus an add) filtered through `Filters`. 4. Export only the added/modified blobs plus the unchanged files the resolve pass probes (A1's `probed_only` expansion) to `.tmp-*/checkout/`, apply with `scan_paths_with_observer` through the normal writer, run the incremental resolve pass, then steps 4-5 of the cold build. If the filesystem has no CoW and free space is below twice the generation size, the advance is skipped and retried -later. +later (*Disk budget*). + +**Lifecycle (what "published" means).** A published generation is never rewritten by basemind's logic, but a Fjall +generation is not read-only on disk: opening it replays the journal and compacts, and `IndexDb::open_inner` calls +`rotate_memtable_and_wait` on every keyspace (`src/index/mod.rs`; best-effort, a failure only logs). So every open of a +Fjall generation can write, a CoW clone of it un-shares as compaction rewrites pages, and *Advance* must quiesce the +daemon handle before cloning. The builder's flush loop leaves the journal empty, which bounds replay at the next open +to nothing for a freshly published generation. A sorted-run generation has none of this and is read-only in fact. **Selection.** An overlay pins `base_generation`. On open without a usable pin (first open, pin retired, incompatible) choose among compatible ready generations the one with the smallest candidate count (`|K|` below, computed once per generation, at most three git status calls), ties to the newest. A pin is kept while its generation exists and is compatible. -**Opening the base.** `IndexDb::open_code_only(gen_dir)`: like `open_tolerating_held_memory` with the memory +**Opening the base.** *(Fjall option)* `IndexDb::open_code_only(gen_dir)`: like `open_tolerating_held_memory` with the memory store always detached, because the base must not take the group's exclusive `memory.fjall` lock. The daemon's -`workspace_pool.rs` keeps one `Arc` per `(group, generation)`; every pooled workspace of the group shares +`src/comms/workspace_pool.rs` keeps one `Arc` per `(group, generation)`; every pooled workspace of the group shares it and its block cache. A non-daemon process uses `open_index_with_retry`'s budget and then treats `Locked` like a lost lock today: it routes through `IndexRoute::Daemon` if a daemon answers, else `InRam`. Open question 1. -**Migration of a workspace (standalone to layered), one-way, under the workspace `.lock`.** - -1. Resolve or build a base (above). 2. Compute the overlay (A5, "Building the overlay") from the live tree. -2. Verify: for 200 random paths plus every path of `S_path`, compare the merged rows of each layered keyspace with - the standalone index (`IndexView` makes this one function); any difference aborts and keeps the standalone view. -3. Write `overlay.json` with `state = "ready"`, set `index_mode = "layered"` and `repo_group` in `workspace.json`, - rename `views/working` to `views/working.migrated`, remove it on the next successful open (so a crash between the - two leaves a restorable index). `store_seed::seed_working_view` is skipped for layered-eligible workspaces. +**Migration of a workspace (standalone to layered), one-way, under the workspace `.lock`.** One workspace at a time per +machine (a machine-wide try-lock; the others stay standalone and retry on their next open). + +1. Check the free-space precondition (*Disk budget*); resolve or build a base (above). +2. Compute the overlay (A5, "Building the overlay") from the live tree. +3. Verify: for 200 random paths plus every path of `S_path`, compare the merged rows of each layered keyspace with + the standalone index (`IndexView` makes this one function); any difference aborts, removes the overlay, and keeps + the standalone view. Verify cannot see a defect the standalone index and its clone share, which is why the BM25 + layout stamp below is checked first. +4. Write `overlay.json` with `state = "ready"`, set `index_mode = "layered"` and `repo_group` in `workspace.json`, + rename `views/working` to `views/working.migrated`, and delete it right after verify and the commit above succeed + (a crash between the rename and the delete leaves a restorable index; the next open finishes the delete). The + transient cost is the old index kept until then, about +2.9 GB for the one workspace being migrated. + `store_seed::seed_working_view` is skipped for layered-eligible workspaces. + +**Damaged-index inheritance.** `vet_clone` (`store_seed.rs`) checks only `msgpack` and the Fjall `schema_ver`, so a +standalone index that already lost postings for duplicate content (the defect A5.0 fixes) would be cloned into the +base and verify would compare against the same defect. `meta` therefore gains a `bm25_layout` stamp (written by A5.0's +fixed writer). `vet_clone` requires the stamp to equal the current layout; a missing or older stamp rejects the +clone as a base source, and a standalone index with an old stamp does a one-time postings re-derive from the +content-addressed chunk blobs on its next scan (the `by_path` rows hold the term list without frequencies, so they +cannot rebuild the postings). Only then is it eligible as a clone source. + +**Disk budget.** A group holds up to 3 live generations, about 2.7 GB each, and compaction un-shares the CoW clones, +so the realistic footprint is the sum, not one generation plus deltas. `[index] max_group_bytes` (default 12 GiB, to be +measured) caps `base/` plus `overlays/`. Build, advance and migrate each check, under `base.lock`, that free space on +the volume is at least the generation size plus the cap on the transient (export plus clone) and that the group stays +under the cap, else they are skipped, a `last-failure.json`-style marker is written, and `doctor` says why. When disk is +tight the newest generation replaces the oldest instead of the group keeping 3 (the oldest goes through the *Liveness* +rules, so a pinned one is never deleted from under a reader). ### A5: overlay write path and the read merge -**A5.0 (prerequisite, standalone fix).** BM25 postings are refcounted by content hash. `upsert_bm25_file` and -`remove_bm25_file` take a `still_referenced: &dyn Fn(&str /*hash*/) -> bool`, supplied from -`Store.index.files` hash counts, and skip the posting delete when another indexed path holds the hash. A regression -test (two paths, identical content, remove one, the other's postings survive) is red on `main` today. +**A5.0 (prerequisite, standalone fix).** Postings are shared by content hash, so a path-driven delete must not +remove them while another path holds the hash. A refcount handed to `upsert_bm25_file` / `remove_bm25_file` from +`Store.index.files` is rejected: refcounts computed while parallel `WorkerIndexBatch` staging runs race the serial apply +pass (a count taken before a sibling path's batch lands is stale). Instead the writer's per-path delete of +`code_bm25_postings` is dropped (the `code_bm25_by_path` forward rows of the path are still deleted) and a +**hash-liveness sweep** next to `sweep_grep_bloom_rows` (`scanner.rs`) removes, after the stale-entry purge of a +`WorkingTree` scan, every posting whose `chunk_id` hash is not in the hash set of `store.index.files`. This is +symmetric with A2: both sweeps run from the post-pass state, are key-only for postings (the hash is parsed from the +key, `keys::parse_code_bm25_posting_chunk_id`) and tolerate leftovers until the next full pass. The watcher's +`scan_paths` deletes the postings of a hash that left the manifest, using the post-pass hash set. The same sweep is the +layered `D_hash` rule applied to a standalone index. + +`N` semantics, pinned: `N` and `total_len` come from `code_bm25_by_path`, which has one row per `(path, chunk_id)`, so +`N` counts path-chunks (two paths with identical content count twice). `df(term)` counts `code_bm25_postings` rows, +which are keyed by `chunk_id`, so it counts unique chunks (once per distinct content). The asymmetry is intended: it is +what a standalone index computes today and what the layered formula (below) reproduces. The equivalence test asserts it +explicitly with a duplicate-content scenario (`N` twice, `df` once) in both modes. + +Red tests on `main` today: two paths, identical content, remove one, the other's postings survive. `meta` gets a +`bm25_layout` stamp written by this fix; see *Damaged-index inheritance* (A4). **`delta.msgpack` (format 1, MessagePack named, `rmp_serde`, same atomic tmp + rename as `Store::flush`).** @@ -301,13 +407,50 @@ writer as today (the daemon when running). The base is read-only to everyone but write lock only to commit the batch and swap the `Arc`, so a reader never sees a path in both layers or in neither. Fjall iterators are snapshots, so the lock is not held while iterating. -**Merge rule per partition (every keyspace opened by `IndexDb::open_inner`).** Merged order is plain byte order of the -raw key (both layers use the same encoders in `index/keys*.rs`); keys never collide across layers because a base row -is dropped exactly when the overlay may hold a row for the same shadow unit. +**Merge contract.** Merged order is plain byte order of the raw key (both layers use the same encoders in +`index/keys*.rs`); keys never collide across layers because a base row is dropped exactly when the overlay may hold a +row for the same shadow unit. Tombstones, snapshots and scalars are specified here once. + +- **Tombstones are masks, not rows.** Overlay `get` internally distinguishes *tombstone* (the key's shadow unit is + masked, so the answer is the overlay's, and absence there is final) from *absent* (not masked, so fall through to the + base). The mask is the shadow set of the table below; at an equal key the overlay shadows the base in `prefix` / + `range` / `iter`. No tombstone rows are stored. +- **Inverted keyspaces use a per-path mask.** `calls_by_callee`, `symbols_by_name`, `refs_by_def`, + `implementations_by_trait`, `imports_by_module` and `importers_by_probe` hold the path inside the key, not as a + prefix, so a delete cannot be a range delete. Rejected: per-key tombstones derived by walking the `*_by_path` + forward keyspaces (it costs a base walk per deleted path and writes a row per key). Chosen: the shadowed-path set the + ADR already defines (`S_path`, and `S_refs` for the ref and probe keyspaces; a `Deleted` path is in `S_path` with no + overlay rows). Each base row of such a keyspace is dropped when the path parsed from its key by the keyspace's + `Part` parser (`keys::parse_symbol_by_name`, `parse_call_by_callee`, `parse_ref_by_def` use-path, + `parse_impl_by_trait`, `parse_import_by_module`, `parse_importer_by_probe`) is in the mask. Forward `*_by_path` + keyspaces use the cheaper prefix check. +- **NameDict.** `callee_keys_matching` and `trait_keys_matching` return a key-ordered (`len`, then bytes), deduplicated + union of the base and overlay dictionaries, one per layer (the dictionary is a superset filter, `name_dict.rs`, so a name that survives only in a + shadowed base row yields a harmless empty scan). The merged iterator dedups equal names across layers and honours + `cursor_after` and `lower` on the merged order. +- **Snapshot / pin handle.** A query takes one `ViewSnapshot { Arc, base reader pin, overlay iterator + snapshot }` and every `get` / `prefix` / `range` of that query goes through it, so the two layers and the mask are + read at one consistent point even if the writer commits or a rebase swaps the overlay mid-query. The pin keeps the + generation (and its overlay directory) alive until the snapshot drops (A6, *Liveness*). +- **Meta scalars.** `code_bm25_n`, `code_bm25_total_len` and the import-probe completeness flag are precomputed + scalars, so they need a decision rather than a row merge. BM25: effective value = base scalar + overlay delta, where + the delta is `N_overlay - |base by_path rows of S_path|` (and the same for `total_len`) with the shadowed term computed + from the base `code_bm25_by_path` rows of `S_path` at open and on each `S_path` change. This is the same formula as + the BM25 text under *Totals, caps and cursors*; the overlay's own `code_bm25_n` / `code_bm25_total_len` hold + `N_overlay` / `total_len_overlay` (its own rows only), not corpus totals. `recompute_bm25_stats` on an overlay + therefore sweeps only overlay rows. `import_probes_complete`: base value (a generation is published only complete) + AND the overlay's own flag (cleared while the overlay's resolve pass is incomplete). +- **Visited-row budget.** `scan_cap` caps hits, not rows examined (`mcp/helpers_calls_scan.rs`). A filtered scan whose + mask drops many base rows (a widely edited name) can examine far more rows than it yields, so the merged iterator + carries a visited-row budget (default `scan_cap * 64`); when it is hit the scan stops, returns what it has and sets the + existing partial flag (`total_is_partial`). The equivalence gate runs with the budget unbounded and a separate test + proves the partial flag. + +**Merge rule per partition (every keyspace opened by `IndexDb::open_inner`).** | Keyspace | Key (shadow unit) | Overlay holds | Base row dropped when | Notes | |---|---|---|---|---| -| `meta` | string keys | `delta_seq`, overlay-only `code_bm25_n`, `code_bm25_total_len` | never merged as rows | `schema_ver`: both must equal `INDEX_SCHEMA_VER`. `import_probe_ver`: effective value is the base's (a generation is published only with probes complete); the overlay never writes it. BM25 stats below. | +| `meta` | string keys | `delta_seq`, overlay-only `code_bm25_n`, `code_bm25_total_len` | never merged as rows | `schema_ver`: both must equal `INDEX_SCHEMA_VER`. `import_probe_ver`: base AND overlay flag (merge contract). `bm25_layout`: both must equal the current layout. BM25 stats: base plus overlay delta (merge contract). | | `symbols_by_path` | `rel` + start | rows of `Added`/`Modified` | `rel in S_path` | | | `symbols_by_name` | name + kind + `rel` + start | same | `rel in S_path` | read by `symbols_by_name_lookup`; ordered by name then kind then `rel`, so the merge interleaves layers inside one name | | `calls_by_path` | `rel` + start | same | `rel in S_path` | | @@ -333,13 +476,21 @@ of the last hit as an exclusive lower bound: it is valid on both layers because Cursors cross a rebase or generation change only if the merged sequence is identical, which is not guaranteed; a cursor older than the last overlay commit is served best-effort (the existing behaviour when a standalone index is rescanned between pages). BM25: `bm25_stats()` returns `N = N_base - |base by_path rows of S_path| + N_overlay` and -`total_len` likewise, with the subtraction computed from the shadowed base rows at open and on each `S_path` change; -`df(term)` is the count of merged postings rows. +`total_len` likewise (the formula of the *Meta scalars* decision, with the subtraction computed from the shadowed base +rows at open and on each `S_path` change); `df(term)` is the count of merged postings rows, so `N` is per path-chunk +and `df` per unique chunk (A5.0). + +**Manifest memory.** The effective manifest does not clone the base `FileEntry`s per worktree (65k entries times the +worktree count). The base manifest is shared as one `Arc` per `(group, generation)`, and each worktree holds a +delta map (`delta.entries` plus the derived sets); `lookup` consults the delta first and then the shared base. Code +that needs an owned `Store.index.files` map materialises it on demand, not at open. **Gate.** `tests/layered_equivalence.rs`: a proptest over three worktrees of one group with random edit scripts (modify, add, delete, rename, duplicate content, barrel re-export, `tsconfig` paths change, Python package `__init__`), each compared against a freshly built standalone index of the same tree for every `Part`, every named -accessor, every MCP mode that reads the index, with totals, `has_more` and cursor round trips. Behind +accessor, every MCP mode that reads the index, with totals, `has_more` and cursor round trips. The `callee_names` / `trait_names` dictionaries are exempt from +byte-equality (each layer's dictionary may hold stale names, and a layered one is a union); the gate compares the +query results they serve instead. Behind `[index] layered = false` until green on 1000 cases. ### A6: generations, rebase, promotion, GC @@ -363,8 +514,16 @@ or `changed_bytes > 25%` of the base manifest's total `size_bytes`. **Liveness.** A generation is live iff it is the newest ready compatible one or an overlay in the group pins it. At most three are live; when publishing a fourth, the overlays pinned to the oldest live generation are invalidated (their -overlay dir is deleted, they rebuild on next open). There is no eager rebase of idle worktrees. A non-live generation -is renamed to `.retired-` and removed when its fjall lock can be taken. An incompatible or half-published +overlay dir is deleted, they rebuild on next open), except an overlay pinned by an active workspace handle or open in +the daemon: that one is never deleted from under a reader. It is rebased onto a newer generation first (the swap of +*Rebase and promotion*, so the old overlay serves reads until the new one is ready), or, if the rebase cannot run, the +oldest generation and its overlay stay live past the limit until the handle drops (the pin is released when the last +`ViewSnapshot` and handle for it are gone), and the next maintenance pass retires them. Readers meanwhile keep reading +the pinned generation and overlay unchanged and see the answers of that older tree; `doctor` reports the overstay. +Under disk pressure (*Disk budget*) the overstay is bounded by `max_group_bytes`; past it the workspace is promoted +or its overlay is dropped and the open handle is reopened. There is no eager rebase of idle worktrees. A non-live generation +is renamed to `.retired-` and removed when its fjall lock can be taken *(Fjall option; a sorted-run generation is +removed once no snapshot pins it)*. An incompatible or half-published generation (no `base.json`) is non-live. An overlay is live iff `workspaces//workspace.json` names an existing root; orphans go with `reap_orphaned_workspaces_in`, which must also remove `repos//overlays/` (group read from the marker), and an overlay older than its workspace's `index_mode` flip is removed. @@ -413,7 +572,7 @@ root; orphans go with `reap_orphaned_workspaces_in`, which must also remove `rep ### A8: layered by default -Preconditions: A5 gate green on 1000 cases, A6 GC merged, A7 shipped, measurement targets met. Then: default +Preconditions: A5 gate green on 1000 cases, A6 GC merged, A7 shipped (A8 depends on A7, because A4 disables layering whenever an embedding lane is on and `documents.embed` defaults to `true`), measurement targets met. Then: default `[index] layered = true`; delete `store_seed.rs`, `BASEMIND_NO_SEED`, `seed_working_view` and its callers in `Store::open_with_holder`, and the lance-staleness sentinel `seed:rebuild-lance`; migration (A4) runs for every standalone workspace on its next writer open. @@ -426,9 +585,47 @@ the extra `workspace.json` fields are ignored, so an older binary simply cold-sc cost is one rescan and the stranded disk, reclaimable with `basemind cache clear`. No format here changes `INDEX_SCHEMA_VER` or `extract::SCHEMA_VER`, so a downgrade never wipes a base. +### Observability + +- `doctor`, `status.json` and the stats surface expose `index_mode` (`standalone` | `layered`) and, when layered was + not chosen, the reason: `embed_on`, `subdirectory_root`, `no_base_commit`, `config_off`, `base_building`, + `disk_budget`, `incompatible_base`. +- For a layered workspace: the pinned generation and its `base.json` summary, overlay size and `changed_files` / + `changed_bytes`, rebase and promote counts, base-build state (idle, building by whom, last failure), generations + overstaying the limit, and merge-read latency (p50 / p95 of the merged iterators per `Part`). +- `tracing` events on every state transition: base build start / publish / abort, advance start / quiesce / abort, + generation retire, overlay commit / discard, rebase, promote, migration start / verify-fail / commit. + +### Acceptance criteria + +Each phase is done when its line holds; the fault-injection list below is the shared gate for A4 and A6. + +- **A2.** The grep-bloom, version-sweep and watcher-delete tests pass; the 20-pattern skip ratio does not worsen. +- **A3.** The existing suite passes through `IndexView`; no `fjall` type appears in the trait; the named accessors are + default methods; `references` / `callers` p95 within the A3 budget. +- **A4.** Prototype gate evaluated and storage chosen; a cold build, a racing second builder, a bootstrap-by-clone and a + migration each produce a generation / overlay whose merged rows equal a standalone index (A5 verify function); + publish is crash-safe at every step; disk preconditions skip rather than fail; all with embeds off. +- **A5.** A5.0 fix and BM25 `N` / `df` equivalence pass; the equivalence gate is green on 1000 cases; overlay under 50 MB + for a 50-file diff; visited-row budget proven. +- **A6.** Advance, rebase and promote are exercised end to end; no overlay or generation pinned by a live handle is + deleted; GC leaves exactly the live set and protects in-flight `.tmp-*`; `max_group_bytes` is honoured; the + fault-injection list passes. +- **A7 / A8.** Per their sections and the measurement plan. + +Fault injection (each is a test that kills or fails the step and asserts the post-state and a clean re-run): + +- publish crash at each step: before `base.json.tmp`, after its fsync, after its rename, after `rename(.tmp-*, gen-n)`, + after the `base/` fsync, after `CURRENT`; +- racing builders (two processes, one winner, loser takes over after a winner crash); +- GC of `.tmp-*` and `.retired-*` (grace honoured, a locked or pinned one kept); +- quiesce and advance (stall budget exceeded, `Locked`, crash mid-clone); +- migration abort (crash between each step, restore from `views/working.migrated`) and migration-verify failure + (injected row difference keeps the standalone view and removes the overlay). + ### Measurement plan -On armis (89k files, 65k indexed), 1 worktree then 5, standalone vs layered, same machine, cold cache and warm: +On armis (89k files, 65k indexed), 1 worktree then 5, standalone vs layered, same machine, cold cache and warm, with embeds off (`documents.embed = false`) until A7 ships, because A4 does not layer otherwise: - `du -sh` of the base generation and of each overlay (target overlay < 50 MB with a typical 50-file diff); `du` of `grep_bloom` before and after A2 (expect a drop proportional to duplicate content); @@ -442,12 +639,15 @@ On armis (89k files, 65k indexed), 1 worktree then 5, standalone vs layered, sam ### Open questions for the owner -1. **Base access without a daemon.** Fjall admits one process, so with `layered` on every CLI invocation in a group +1. **Base access without a daemon** *(Fjall option; mostly dissolved if the A4 prototype adopts a mmapped sorted-run + base, which every process can open)*. Fjall admits one process, so with `layered` on every CLI invocation in a group contends for the base. The proposal above degrades like a lost lock today (`Daemon` route, else `InRam`). The - alternative is to make the daemon mandatory for layered workspaces (auto-spawn through `singleton_spawn.rs`, and + alternative is to make the daemon mandatory for layered workspaces (auto-spawn through `src/comms/singleton_spawn.rs`, and fall back to standalone if it cannot start). Which is acceptable for the A8 default? -2. **A2 partition-revision bump.** The bump forces one full re-index for every user. The no-bump variant (ROW_VERSION 2 plus the - sweep) avoids it. The ADR keeps the bump because #87 prescribes it; say if the cheaper variant is preferred. +2. **A2 migration.** Decided: `ROW_VERSION` 2 plus the sweep, no partition-revision bump. +3. **Daemon-less layered mode.** Decided: it degrades like a lost lock today (`Daemon` route, else `InRam`). +4. **Base storage.** Decided: prototype-first. A sorted-run base is built and benchmarked before A4 (*Base storage*); + Fjall stays for overlays and mutable stores; A4 is not locked to a Fjall base. ## Consequences @@ -456,9 +656,11 @@ The read path grows a merge layer that must preserve order, totals and cursors f correctness gate (a proptest over three worktrees with random edits, comparing every mode's results and cursors to a standalone full index) exists to catch a merge that does not. `store_seed.rs` is removed once layered mode is the default. Phases A0 (per-repo memory store) and A1 (probe index and O(affected) -re-stitch) shipped in 0.30.0; A2–A8 remain, specified below. +re-stitch) shipped in 0.30.0; A2 (`efe41b76`) and A3 (`26be3d56`) shipped on this branch; A4–A8 remain, specified +below. Until A7, layering reaches only users with embeds off (`documents.embed` defaults to `true`). -Costs accepted by this design: a layered workspace depends on the daemon for concurrent base access; the +Costs accepted by this design: with a Fjall base, a layered workspace depends on the daemon for concurrent base access +(a mmapped sorted-run base would remove that, pending the A4 prototype); the first base of a group costs one cold build (or one CoW clone when a standalone index exists); migration is one-way per workspace (rollback re-materialises a standalone index); the document and embedding tiers stay per worktree until A7; and the overlay is treated as a disposable cache, so any doubt about its consistency @@ -477,5 +679,6 @@ costs a rebuild proportional to the diff rather than a repair. and its lock; invalidating the overlay and rebuilding it at next open is simpler and costs O(diff). - **Repairing a suspect overlay in place.** Rejected: the overlay is a cache of the diff; wiping and rebuilding it is cheaper to reason about than reconciling `delta.msgpack` against fjall. -- **Opening the base from every process.** Not possible: fjall's directory lock is exclusive, which is why - the daemon holds it (open question 1 covers the no-daemon case). +- **Opening a Fjall base from every process.** Not possible: fjall's directory lock is exclusive, which is why + the daemon holds it for a Fjall base (open question 1 covers the no-daemon case). An immutable mmapped base is the + alternative under prototype (*Base storage*). From 2347fa5295f1c5a7611da7ba84cac5a08fc8860a Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Mon, 19 Oct 2026 16:14:09 +0200 Subject: [PATCH 06/10] fix(index): bound grep_bloom growth under the watcher and close the same-tick skip hole Co-Authored-By: Claude Opus 5.5 --- src/index/grep_bloom.rs | 235 ++++++------------------ src/index/grep_bloom_tests.rs | 333 ++++++++++++++++++++++++++++++++++ src/lib.rs | 1 + src/mcp/index_route_tests.rs | 57 +++++- src/scanner.rs | 39 +++- src/scanner_file.rs | 32 ++++ src/store.rs | 21 ++- src/store_hash_refs.rs | 71 ++++++++ tests/git_smoke.rs | 56 ++++++ tests/grep_bloom_scan.rs | 219 +++++++++++++++++++--- 10 files changed, 853 insertions(+), 211 deletions(-) create mode 100644 src/index/grep_bloom_tests.rs create mode 100644 src/store_hash_refs.rs diff --git a/src/index/grep_bloom.rs b/src/index/grep_bloom.rs index 64fcc344..9bce59c3 100644 --- a/src/index/grep_bloom.rs +++ b/src/index/grep_bloom.rs @@ -5,8 +5,8 @@ //! trigrams. A query extracts the literals every match MUST contain, turns them into trigram hashes //! and tests each row; a file whose indexed hash selects a rejecting bloom, and whose live //! `(size, mtime_ns)` still equals the stamp it was indexed under, cannot contain a match and is -//! never opened. A bloom can only say "definitely absent", so it -//! decides which files to READ, never what a match is: the real regex still runs on every survivor. +//! never opened. A bloom can only say "definitely absent", so it decides which files to READ, +//! never what a match is: the real regex still runs on every survivor. //! //! Everything here errs towards "candidate": a missing, older-version or malformed row, a changed //! stamp, a pattern with no usable literal, an unreadable file. Only an intact row that was built @@ -40,6 +40,11 @@ const HASHES: u64 = 2; /// A literal shorter than this has no trigram to test. const MIN_LITERAL_BYTES: usize = 3; +/// `meta` row recording the [`ROW_VERSION`] the keyspace was last version-swept under. Present and +/// equal means every row is current, so a sweep need not read values; set only after a sweep that +/// read them all succeeded. +const META_ROW_SWEPT_VER: &[u8] = b"grep_bloom_swept_ver"; + #[inline] fn mix(trigram: u32) -> u64 { let mut x = u64::from(trigram).wrapping_mul(0x9E37_79B9_7F4A_7C15); @@ -164,12 +169,14 @@ impl IndexDb { /// Delete the row for `hash_hex`: how a test models an index built before the keyspace existed. #[doc(hidden)] + #[cfg(any(test, feature = "test-support"))] pub fn drop_grep_bloom_row_for_test(&self, hash_hex: &str) { self.grep_bloom.remove(hash_hex.as_bytes()).expect("remove bloom row"); } /// Number of stored rows, one per distinct content. #[doc(hidden)] + #[cfg(any(test, feature = "test-support"))] pub fn grep_bloom_row_count(&self) -> usize { self.grep_bloom.iter().count() } @@ -180,25 +187,54 @@ impl IndexDb { .is_some_and(|row| parse_row(&row).is_some()) } - /// Delete every row whose hash is not in `live` or whose layout is not [`ROW_VERSION`], returning - /// how many were dropped. Rows are shared by every path holding the content, so only a hash no - /// indexed file references may go. Path-keyed rows from before the content-hash re-key are never - /// in `live`, and older-version rows are unreadable, so both leave here. + /// Delete every row whose hash is not in `live`, returning how many were dropped. Rows are shared + /// by every path holding the content, so only a hash no indexed file references may go. Path-keyed + /// rows from before the content-hash re-key are never in `live`, so they leave here too. + /// + /// Row values are read only until the keyspace has been swept once under the current + /// [`ROW_VERSION`] (recorded in `meta`): that pass also drops rows whose layout is older, which + /// are unreadable. Afterwards every row is current, so the sweep is key-only and a large index + /// is not read back in full on each scan. pub fn sweep_grep_bloom(&self, live: &ahash::AHashSet<&str>) -> Result { + let versioned = self + .meta + .get(META_ROW_SWEPT_VER)? + .is_some_and(|v| v.first() == Some(&ROW_VERSION)); let mut writer = self.writer(); let mut dropped = 0; for guard in self.grep_bloom.iter() { - let (key, row) = guard.into_inner()?; - if parse_row(&row).is_some() && std::str::from_utf8(&key).is_ok_and(|hash| live.contains(hash)) { + let (key, current) = if versioned { + (guard.key()?, true) + } else { + let (key, row) = guard.into_inner()?; + (key, parse_row(&row).is_some()) + }; + if current && std::str::from_utf8(&key).is_ok_and(|hash| live.contains(hash)) { continue; } writer.remove_grep_bloom(&key); dropped += 1; } writer.commit()?; + if !versioned { + self.meta.insert(META_ROW_SWEPT_VER, [ROW_VERSION])?; + } Ok(dropped) } + /// Delete the rows of `hashes`, none of which any indexed file references now. The incremental + /// counterpart of [`Self::sweep_grep_bloom`]: cost is the number of hashes, not the keyspace. + pub fn remove_grep_bloom_rows<'a>( + &self, + hashes: impl IntoIterator, + ) -> Result<(), super::IndexError> { + let mut writer = self.writer(); + for hash in hashes { + writer.remove_grep_bloom(hash.as_bytes()); + } + writer.commit() + } + /// For each of `entries`, whether the file can be SKIPPED by a grep for `needle`: its content has /// an intact row, the row rejects the needle, and the live file still has the stamp the entry was /// indexed under. Rows are fetched and dropped one at a time, in parallel; nothing is held @@ -228,182 +264,17 @@ pub(crate) fn skip_verdicts( return false; } // The bloom describes the content as indexed; the sweep reads the file as it is now. - // Only a file still carrying its indexed stamp may be trusted to lack the literal. - std::fs::metadata(root.join(entry.rel.to_path_buf())) - .is_ok_and(|m| m.len() == entry.size_bytes && mtime_nanos(&m) == entry.mtime_ns) + // Only a file still carrying its indexed stamp may be trusted to lack the literal. A + // stamp of 0 is "unknown" (a git source, or an mtime too close to the clock when it was + // taken, see `scanner_file::settled_mtime`), and an unavailable live mtime also reads 0, + // so it must never count as a match. + entry.mtime_ns != 0 + && std::fs::metadata(root.join(entry.rel.to_path_buf())) + .is_ok_and(|m| m.len() == entry.size_bytes && mtime_nanos(&m) == entry.mtime_ns) }) .collect() } #[cfg(test)] -mod tests { - use super::*; - - fn row_for(text: &str) -> Vec { - build_row(text.as_bytes()) - } - - fn rejects(pattern: &str, text: &str) -> bool { - let needle = Needle::compile(pattern).expect("usable needle"); - let row = row_for(text); - !needle.passes(parse_row(&row).expect("row")) - } - - #[test] - fn a_present_literal_is_never_rejected() { - let text = "fn main() { let post_processing = 1; }\n".repeat(50); - assert!(!rejects("post_processing", &text)); - assert!(!rejects(r"post_processing\s*=", &text)); - assert!(!rejects("(?i)POST_PROCESSING", &text)); - assert!(!rejects("zzz_absent|post_processing", &text)); - } - - #[test] - fn an_absent_literal_is_rejected() { - assert!(rejects("eq9fsw", "fn main() { println!(\"hello\"); }")); - } - - #[test] - fn patterns_without_a_required_trigram_sweep_everything() { - for pattern in [".*", r"\w+", "ab", "a|bcd", "(?i)k", "^", r"\d{3}"] { - assert!(Needle::compile(pattern).is_none(), "{pattern} must not be prefiltered"); - } - } - - #[test] - fn case_insensitive_literals_match_every_casing() { - let needle = Needle::compile("(?i)needle").expect("expanded to case variants"); - for text in ["xx NEEDLE xx", "xx needle xx", "xx NeEdLe xx"] { - assert!(needle.passes(parse_row(&row_for(text)).unwrap()), "{text}"); - } - } - - #[test] - fn unicode_literals_are_matched_on_their_utf8_bytes() { - let needle = Needle::compile("caf\u{e9}s").expect("literal"); - assert!(needle.passes(parse_row(&row_for("two caf\u{e9}s here")).unwrap())); - } - - #[test] - fn foreign_or_short_rows_do_not_parse() { - let mut row = row_for("hello world"); - assert!(parse_row(&row).is_some()); - row[0] = ROW_VERSION + 1; - assert!(parse_row(&row).is_none()); - assert!(parse_row(&[ROW_VERSION; HEADER_BYTES]).is_none()); - assert!(parse_row(&[]).is_none()); - } - - #[test] - fn row_size_tracks_the_file_within_its_bounds() { - assert_eq!(build_row(b"abc").len(), HEADER_BYTES + MIN_BLOOM_BYTES); - let big = vec![b'x'; 8 * 1024]; - assert_eq!(build_row(&big).len(), HEADER_BYTES + 1024); - let huge = vec![b'x'; 4 * 1024 * 1024]; - assert_eq!(build_row(&huge).len(), HEADER_BYTES + MAX_BLOOM_BYTES); - } - - fn open_db() -> (tempfile::TempDir, IndexDb) { - let dir = tempfile::tempdir().expect("tempdir"); - let db = IndexDb::open(dir.path()).expect("index"); - (dir, db) - } - - fn put(db: &IndexDb, hash_hex: &str, text: &str) { - let mut writer = db.writer(); - writer.upsert_grep_bloom(hash_hex, row_for(text)); - writer.commit().expect("commit"); - } - - #[test] - fn a_row_round_trips_under_its_content_hash() { - let (_dir, db) = open_db(); - put(&db, "aa11", "fn alpha() {}"); - assert_eq!( - db.grep_bloom_row("aa11").as_deref(), - Some(row_for("fn alpha() {}").as_slice()) - ); - assert!(db.grep_bloom_current("aa11")); - assert!(!db.grep_bloom_current("bb22")); - } - - #[test] - fn a_hash_that_prefixes_another_reads_only_its_own_row() { - let (_dir, db) = open_db(); - put(&db, "ab", "short key"); - put(&db, "abc", "longer key"); - assert_eq!( - db.grep_bloom_row("ab").as_deref(), - Some(row_for("short key").as_slice()) - ); - assert_eq!( - db.grep_bloom_row("abc").as_deref(), - Some(row_for("longer key").as_slice()) - ); - assert!(db.grep_bloom_row("a").is_none()); - } - - #[test] - fn the_sweep_drops_only_hashes_no_live_file_references() { - let (_dir, db) = open_db(); - for hash in ["h1", "h2", "h3"] { - put(&db, hash, hash); - } - let live: ahash::AHashSet<&str> = ["h1", "h3"].into_iter().collect(); - assert_eq!(db.sweep_grep_bloom(&live).expect("sweep"), 1); - assert!(db.grep_bloom_current("h1") && db.grep_bloom_current("h3")); - assert!(!db.grep_bloom_current("h2")); - assert_eq!( - db.sweep_grep_bloom(&live).expect("sweep"), - 0, - "a second sweep is a no-op" - ); - } - - #[test] - fn two_paths_with_the_same_content_share_one_row_and_both_skip() { - let (_dir, db) = open_db(); - let root = tempfile::tempdir().expect("root"); - let text = "fn unrelated() { println!(\"hello\"); }\n"; - let mut entries = Vec::new(); - for name in ["a.rs", "b.rs"] { - std::fs::write(root.path().join(name), text).expect("write"); - let meta = std::fs::metadata(root.path().join(name)).expect("stat"); - entries.push(GrepEntry { - rel: RelPath::from(name), - hash_hex: "same".to_owned(), - size_bytes: meta.len(), - mtime_ns: mtime_nanos(&meta), - }); - } - put(&db, "same", text); - put(&db, "same", text); - assert_eq!(db.grep_bloom_row_count(), 1); - let needle = Needle::compile("zzz_absent_literal").expect("needle"); - assert_eq!(db.grep_skip_verdicts(root.path(), &entries, &needle), vec![true, true]); - } - - #[test] - fn a_live_file_that_moved_off_its_indexed_stamp_is_a_candidate() { - let (_dir, db) = open_db(); - let root = tempfile::tempdir().expect("root"); - std::fs::write(root.path().join("a.rs"), "fn unrelated() {}\n").expect("write"); - let meta = std::fs::metadata(root.path().join("a.rs")).expect("stat"); - put(&db, "h", "fn unrelated() {}\n"); - let needle = Needle::compile("zzz_absent_literal").expect("needle"); - let entry = |size_bytes| GrepEntry { - rel: RelPath::from("a.rs"), - hash_hex: "h".to_owned(), - size_bytes, - mtime_ns: mtime_nanos(&meta), - }; - assert_eq!( - db.grep_skip_verdicts(root.path(), &[entry(meta.len())], &needle), - vec![true] - ); - assert_eq!( - db.grep_skip_verdicts(root.path(), &[entry(meta.len() + 1)], &needle), - vec![false] - ); - } -} +#[path = "grep_bloom_tests.rs"] +mod tests; diff --git a/src/index/grep_bloom_tests.rs b/src/index/grep_bloom_tests.rs new file mode 100644 index 00000000..1722671d --- /dev/null +++ b/src/index/grep_bloom_tests.rs @@ -0,0 +1,333 @@ +use super::*; + +fn row_for(text: &str) -> Vec { + build_row(text.as_bytes()) +} + +fn rejects(pattern: &str, text: &str) -> bool { + let needle = Needle::compile(pattern).expect("usable needle"); + let row = row_for(text); + !needle.passes(parse_row(&row).expect("row")) +} + +#[test] +fn a_present_literal_is_never_rejected() { + let text = "fn main() { let post_processing = 1; }\n".repeat(50); + assert!(!rejects("post_processing", &text)); + assert!(!rejects(r"post_processing\s*=", &text)); + assert!(!rejects("(?i)POST_PROCESSING", &text)); + assert!(!rejects("zzz_absent|post_processing", &text)); +} + +#[test] +fn an_absent_literal_is_rejected() { + assert!(rejects("eq9fsw", "fn main() { println!(\"hello\"); }")); +} + +#[test] +fn patterns_without_a_required_trigram_sweep_everything() { + for pattern in [".*", r"\w+", "ab", "a|bcd", "(?i)k", "^", r"\d{3}"] { + assert!(Needle::compile(pattern).is_none(), "{pattern} must not be prefiltered"); + } +} + +#[test] +fn case_insensitive_literals_match_every_casing() { + let needle = Needle::compile("(?i)needle").expect("expanded to case variants"); + for text in ["xx NEEDLE xx", "xx needle xx", "xx NeEdLe xx"] { + assert!(needle.passes(parse_row(&row_for(text)).unwrap()), "{text}"); + } +} + +#[test] +fn unicode_literals_are_matched_on_their_utf8_bytes() { + let needle = Needle::compile("caf\u{e9}s").expect("literal"); + assert!(needle.passes(parse_row(&row_for("two caf\u{e9}s here")).unwrap())); +} + +#[test] +fn foreign_or_short_rows_do_not_parse() { + let mut row = row_for("hello world"); + assert!(parse_row(&row).is_some()); + row[0] = ROW_VERSION + 1; + assert!(parse_row(&row).is_none()); + assert!(parse_row(&[ROW_VERSION; HEADER_BYTES]).is_none()); + assert!(parse_row(&[]).is_none()); +} + +#[test] +fn row_size_tracks_the_file_within_its_bounds() { + assert_eq!(build_row(b"abc").len(), HEADER_BYTES + MIN_BLOOM_BYTES); + let big = vec![b'x'; 8 * 1024]; + assert_eq!(build_row(&big).len(), HEADER_BYTES + 1024); + let huge = vec![b'x'; 4 * 1024 * 1024]; + assert_eq!(build_row(&huge).len(), HEADER_BYTES + MAX_BLOOM_BYTES); +} + +fn open_db() -> (tempfile::TempDir, IndexDb) { + let dir = tempfile::tempdir().expect("tempdir"); + let db = IndexDb::open(dir.path()).expect("index"); + (dir, db) +} + +fn put(db: &IndexDb, hash_hex: &str, text: &str) { + let mut writer = db.writer(); + writer.upsert_grep_bloom(hash_hex, row_for(text)); + writer.commit().expect("commit"); +} + +#[test] +fn a_row_round_trips_under_its_content_hash() { + let (_dir, db) = open_db(); + put(&db, "aa11", "fn alpha() {}"); + assert_eq!( + db.grep_bloom_row("aa11").as_deref(), + Some(row_for("fn alpha() {}").as_slice()) + ); + assert!(db.grep_bloom_current("aa11")); + assert!(!db.grep_bloom_current("bb22")); +} + +#[test] +fn a_hash_that_prefixes_another_reads_only_its_own_row() { + let (_dir, db) = open_db(); + put(&db, "ab", "short key"); + put(&db, "abc", "longer key"); + assert_eq!( + db.grep_bloom_row("ab").as_deref(), + Some(row_for("short key").as_slice()) + ); + assert_eq!( + db.grep_bloom_row("abc").as_deref(), + Some(row_for("longer key").as_slice()) + ); + assert!(db.grep_bloom_row("a").is_none()); +} + +#[test] +fn the_sweep_drops_only_hashes_no_live_file_references() { + let (_dir, db) = open_db(); + for hash in ["h1", "h2", "h3"] { + put(&db, hash, hash); + } + let live: ahash::AHashSet<&str> = ["h1", "h3"].into_iter().collect(); + assert_eq!(db.sweep_grep_bloom(&live).expect("sweep"), 1); + assert!(db.grep_bloom_current("h1") && db.grep_bloom_current("h3")); + assert!(!db.grep_bloom_current("h2")); + assert_eq!( + db.sweep_grep_bloom(&live).expect("sweep"), + 0, + "a second sweep is a no-op" + ); +} + +#[test] +fn two_paths_with_the_same_content_share_one_row_and_both_skip() { + let (_dir, db) = open_db(); + let root = tempfile::tempdir().expect("root"); + let text = "fn unrelated() { println!(\"hello\"); }\n"; + let mut entries = Vec::new(); + for name in ["a.rs", "b.rs"] { + std::fs::write(root.path().join(name), text).expect("write"); + let meta = std::fs::metadata(root.path().join(name)).expect("stat"); + entries.push(GrepEntry { + rel: RelPath::from(name), + hash_hex: "same".to_owned(), + size_bytes: meta.len(), + mtime_ns: mtime_nanos(&meta), + }); + } + put(&db, "same", text); + put(&db, "same", text); + assert_eq!(db.grep_bloom_row_count(), 1); + let needle = Needle::compile("zzz_absent_literal").expect("needle"); + assert_eq!(db.grep_skip_verdicts(root.path(), &entries, &needle), vec![true, true]); +} + +#[test] +fn a_live_file_that_moved_off_its_indexed_stamp_is_a_candidate() { + let (_dir, db) = open_db(); + let root = tempfile::tempdir().expect("root"); + std::fs::write(root.path().join("a.rs"), "fn unrelated() {}\n").expect("write"); + let meta = std::fs::metadata(root.path().join("a.rs")).expect("stat"); + put(&db, "h", "fn unrelated() {}\n"); + let needle = Needle::compile("zzz_absent_literal").expect("needle"); + let entry = |size_bytes| GrepEntry { + rel: RelPath::from("a.rs"), + hash_hex: "h".to_owned(), + size_bytes, + mtime_ns: mtime_nanos(&meta), + }; + assert_eq!( + db.grep_skip_verdicts(root.path(), &[entry(meta.len())], &needle), + vec![true] + ); + assert_eq!( + db.grep_skip_verdicts(root.path(), &[entry(meta.len() + 1)], &needle), + vec![false] + ); +} + +/// Plant a row under `hash` whose first byte is not the current layout, as an older build left it. +fn put_stale_layout(db: &IndexDb, hash: &str) { + let mut row = row_for(hash); + row[0] = ROW_VERSION - 1; + let mut writer = db.writer(); + writer.upsert_grep_bloom(hash, row); + writer.commit().expect("commit"); +} + +#[test] +fn the_first_sweep_drops_older_layout_rows_and_records_it_in_meta() { + let (_dir, db) = open_db(); + put(&db, "h1", "current"); + put_stale_layout(&db, "h2"); + let live: ahash::AHashSet<&str> = ["h1", "h2"].into_iter().collect(); + assert!(db.meta.get(META_ROW_SWEPT_VER).expect("meta").is_none()); + assert_eq!( + db.sweep_grep_bloom(&live).expect("sweep"), + 1, + "a live hash with an unreadable row goes" + ); + assert!(db.grep_bloom_current("h1") && db.grep_bloom_row("h2").is_none()); + assert_eq!( + db.meta.get(META_ROW_SWEPT_VER).expect("meta").as_deref(), + Some([ROW_VERSION].as_slice()) + ); +} + +#[test] +fn later_sweeps_read_keys_only_and_still_remove_dead_keys() { + let (_dir, db) = open_db(); + let live: ahash::AHashSet<&str> = ["h1", "h2"].into_iter().collect(); + db.sweep_grep_bloom(&live).expect("first sweep"); + + put(&db, "h1", "current"); + put_stale_layout(&db, "h2"); + put(&db, "dead", "no file holds this"); + assert_eq!(db.sweep_grep_bloom(&live).expect("steady sweep"), 1); + assert!( + db.grep_bloom_row("dead").is_none(), + "a dead key is removed without reading it" + ); + assert!( + db.grep_bloom_row("h2").is_some(), + "the value of a live key is not decoded after the version sweep ran" + ); +} + +#[test] +fn without_the_marker_the_version_sweep_reads_values_again() { + let (_dir, db) = open_db(); + let live: ahash::AHashSet<&str> = ["h1"].into_iter().collect(); + db.sweep_grep_bloom(&live).expect("first sweep"); + put_stale_layout(&db, "h1"); + assert_eq!( + db.sweep_grep_bloom(&live).expect("steady sweep"), + 0, + "marker present: values unread" + ); + db.meta.remove(META_ROW_SWEPT_VER).expect("forget the marker"); + assert_eq!(db.sweep_grep_bloom(&live).expect("version sweep"), 1); +} + +#[test] +fn remove_grep_bloom_rows_drops_only_the_named_hashes() { + let (_dir, db) = open_db(); + for hash in ["h1", "h2", "h3"] { + put(&db, hash, hash); + } + db.remove_grep_bloom_rows(["h1", "h3", "never-indexed"]) + .expect("remove"); + assert!(db.grep_bloom_current("h2")); + assert!(db.grep_bloom_row("h1").is_none() && db.grep_bloom_row("h3").is_none()); +} + +fn skip_for(db: &IndexDb, root: &Path, entry: GrepEntry) -> bool { + let needle = Needle::compile("zzz_absent_literal").expect("needle"); + db.grep_skip_verdicts(root, &[entry], &needle)[0] +} + +#[test] +fn an_unknown_indexed_stamp_is_never_trusted_even_when_the_live_stamp_is_also_unknown() { + let (_dir, db) = open_db(); + let root = tempfile::tempdir().expect("root"); + let path = root.path().join("a.rs"); + std::fs::write(&path, "fn unrelated() {}\n").expect("write"); + std::fs::File::options() + .write(true) + .open(&path) + .and_then(|f| f.set_modified(std::time::UNIX_EPOCH)) + .expect("set mtime to the epoch"); + let meta = std::fs::metadata(&path).expect("stat"); + assert_eq!(mtime_nanos(&meta), 0, "the live stamp reads as 0"); + put(&db, "h", "fn unrelated() {}\n"); + let entry = GrepEntry { + rel: RelPath::from("a.rs"), + hash_hex: "h".to_owned(), + size_bytes: meta.len(), + mtime_ns: 0, + }; + assert!(!skip_for(&db, root.path(), entry), "0 == 0 is not a match"); +} + +#[test] +fn settled_mtime_zeroes_a_stamp_within_two_seconds_of_the_clock_either_side() { + use crate::scanner_file::settled_mtime; + let now = 10_000_000_000_i64; + assert_eq!(settled_mtime(now, now), 0); + assert_eq!(settled_mtime(now - 1_999_999_999, now), 0); + assert_eq!( + settled_mtime(now + 1_999_999_999, now), + 0, + "a slightly future stamp is untrusted too" + ); + assert_eq!(settled_mtime(now - 2_000_000_000, now), now - 2_000_000_000); + assert_eq!(settled_mtime(now + 60_000_000_000, now), now + 60_000_000_000); + assert_eq!(settled_mtime(0, now), 0); +} + +/// Every text `pattern` matches must pass the needle; `usable` also pins that the pattern is +/// prefiltered at all, so the loop cannot pass by the needle quietly not existing. +fn never_false_skips(pattern: &str, usable: bool, texts: &[&str]) { + let re = regex::Regex::new(pattern).expect("regex"); + let needle = Needle::compile(pattern); + assert_eq!(needle.is_some(), usable, "{pattern}: usable"); + let Some(needle) = needle else { return }; + let mut matched = 0; + for text in texts.iter().filter(|text| re.is_match(text)) { + matched += 1; + assert!( + needle.passes(parse_row(&row_for(text)).expect("row")), + "{pattern} matches {text:?} but the bloom rejects it" + ); + } + assert!(matched > 0, "{pattern}: no sample matched, the check proved nothing"); +} + +#[test] +fn mixed_length_alternations_never_false_skip() { + for (pattern, usable, texts) in [ + ("a|bcd", false, &["a", "bcd", "xx a xx"][..]), + ("abc|d", false, &["d", "abc"][..]), + ("foo|ba", false, &["ba", "foo"][..]), + ("(abc|d)ef", true, &["abcef", "def", "xx def"][..]), + ("foobar|bazqux", true, &["foobar", "bazqux"][..]), + ("x(abc|de)y", true, &["xabcy", "xdey"][..]), + ] { + never_false_skips(pattern, usable, texts); + } +} + +#[test] +fn word_boundary_literals_never_false_skip() { + never_false_skips(r"\bfoo\b", true, &["foo", "a foo b", "(foo)"]); + never_false_skips(r"\bfoobar\w*", true, &["foobar", "x foobarbaz"]); +} + +#[test] +fn unicode_case_folding_never_false_skips() { + never_false_skips("(?i)kit", true, &["KIT", "kit", "\u{212A}it", "\u{212A}IT"]); + never_false_skips("(?i)strasse", true, &["STRASSE", "\u{17F}trasse", "Stra\u{17F}se"]); + never_false_skips("(?i)caf\u{e9}s", true, &["CAF\u{c9}S", "caf\u{e9}s"]); +} diff --git a/src/lib.rs b/src/lib.rs index fc8bf4e9..9c859fc8 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -67,6 +67,7 @@ pub mod store_gc_budget; pub(crate) mod store_gc_live; pub mod store_gc_protect; pub mod store_gc_workspace; +mod store_hash_refs; pub mod store_layout; mod store_lock; mod store_seed; diff --git a/src/mcp/index_route_tests.rs b/src/mcp/index_route_tests.rs index d126f66d..93ad1ebf 100644 --- a/src/mcp/index_route_tests.rs +++ b/src/mcp/index_route_tests.rs @@ -43,7 +43,9 @@ fn write_corpus(root: &Path) { h = k % 5, o = k % 7, ); - std::fs::write(dir.join(format!("m{k}.rs")), source).expect("rust file"); + let file = dir.join(format!("m{k}.rs")); + std::fs::write(&file, source).expect("rust file"); + age(&file); } std::fs::create_dir_all(root.join("py")).expect("py dir"); std::fs::write( @@ -51,6 +53,16 @@ fn write_corpus(root: &Path) { b"class Base:\n pass\n\n\nclass Child(Tr1):\n def go(self):\n get_arg()\n", ) .expect("python file"); + age(&root.join("py/a.py")); +} + +/// Move a file's mtime an hour back: a stamp too close to the clock is not trusted for skipping. +fn age(path: &Path) { + std::fs::File::options() + .write(true) + .open(path) + .and_then(|f| f.set_modified(std::time::SystemTime::now() - std::time::Duration::from_secs(3600))) + .expect("age file"); } /// Stands in for the daemon's workspace pool: answers `host_index_read` from a writer store, after @@ -59,6 +71,8 @@ struct StoreHost { store: Mutex, /// When set, every read fails, as a daemon that is restarting would. broken: bool, + /// When set, every read is answered with this instead, as a daemon on another build might. + forced_reply: Option, } impl HostBackend for StoreHost { @@ -76,6 +90,9 @@ impl HostBackend for StoreHost { if self.broken { return Err("daemon unavailable".into()); } + if let Some(reply) = &self.forced_reply { + return Ok(reply.clone()); + } let query: IndexReadQuery = round_trip(&query); let store = self.store.lock().expect("store lock"); let reply = super::index_read::index_read_against(&store, &query)?; @@ -158,11 +175,13 @@ fn fixture() -> Fixture { let host: Arc = Arc::new(StoreHost { store: Mutex::new(store), broken: false, + forced_reply: None, }); // A second store handle is not needed for the broken host: it never reads. let broken: Arc = Arc::new(StoreHost { store: Mutex::new(Store::open_read_only_no_index(&root, VIEW_WORKING).expect("store")), broken: true, + forced_reply: None, }); Fixture { local: IndexRoute::Local(Box::new(StandaloneView(idx))), @@ -557,6 +576,42 @@ fn a_pattern_without_a_required_literal_or_a_failed_forward_means_no_prefilter() assert!(block_on(fx.broken_host.grep_skip(&fx.root, "driver_7", &entries)).is_none()); } +/// A route whose daemon answers every read with `reply`. +fn host_replying(fx: &Fixture, reply: IndexReadResult) -> IndexRoute { + let host: Arc = Arc::new(StoreHost { + store: Mutex::new(Store::open_read_only_no_index(&fx.root, VIEW_WORKING).expect("store")), + broken: false, + forced_reply: Some(reply), + }); + IndexRoute::Host { + host, + root: fx.root.clone(), + } +} + +#[test] +fn a_mismatched_grep_bloom_reply_degrades_to_a_full_sweep() { + let fx = fixture(); + let entries = corpus_entries(&fx); + let replies = [ + ("wrong length", IndexReadResult::GrepSkip(vec![true; entries.len() - 1])), + ("wrong variant", IndexReadResult::CallsInFiles(Vec::new())), + ]; + for (what, reply) in replies { + let route = host_replying(&fx, reply); + assert!( + block_on(route.grep_skip(&fx.root, "driver_7", &entries)).is_none(), + "{what}: no verdicts, so every file is searched" + ); + } + let agreeing = host_replying(&fx, IndexReadResult::GrepSkip(vec![false; entries.len()])); + assert_eq!( + block_on(agreeing.grep_skip(&fx.root, "driver_7", &entries)), + Some(vec![false; entries.len()]), + "a well-formed reply is still used" + ); +} + #[test] fn an_oversized_grep_bloom_batch_is_refused() { let fx = fixture(); diff --git a/src/scanner.rs b/src/scanner.rs index 8a170cd2..37ca3694 100644 --- a/src/scanner.rs +++ b/src/scanner.rs @@ -573,7 +573,10 @@ pub fn scan_with_observer( observer.on_file(FileResult::bare(k.clone(), FileStatus::Removed)); report.stats.removed += 1; } - sweep_grep_bloom_rows(store); + // Only a working-tree pass writes bloom rows; a rev or staged pass would sweep them all away. + if matches!(source, ScanSource::WorkingTree) { + sweep_grep_bloom_rows(store); + } if let Some(change) = &policy { // Recorded only when the purge ran to completion (a panic leaves `applied` false too), so a @@ -647,7 +650,8 @@ pub fn scan_with_observer( } /// Drop the grep bloom rows of every content no indexed file references any more. Rows are shared by -/// content, so a removed or edited file leaves its old row behind until this full-pass sweep. +/// content, so a removed or edited file leaves its old row behind until this full-pass sweep (or, for +/// a watcher batch, [`drop_unheld_grep_bloom_rows`]). fn sweep_grep_bloom_rows(store: &Store) { let Some(idx) = store.index_db.as_ref() else { return; @@ -663,6 +667,27 @@ fn sweep_grep_bloom_rows(store: &Store) { } } +/// Drop the grep bloom rows of `hashes` that no indexed file holds any more. `hashes` are the +/// contents the pass's changed or removed paths held before it ran, so the cost follows the batch, +/// not the index; a hash another path still holds (or one the pass just wrote for a new path) stays. +/// Runs after the pass has applied its entries and committed its rows, so it sees the final manifest. +fn drop_unheld_grep_bloom_rows(store: &Store, hashes: &[String]) { + let Some(idx) = store.index_db.as_ref() else { + return; + }; + let unheld: ahash::AHashSet<&str> = hashes + .iter() + .map(String::as_str) + .filter(|hash| !store.hash_is_held(hash)) + .collect(); + if unheld.is_empty() { + return; + } + if let Err(error) = idx.remove_grep_bloom_rows(unheld) { + tracing::warn!(%error, "grep bloom row removal failed; unreferenced rows stay until the next full scan"); + } +} + /// Move the pass's breadcrumb (and the per-phase memory tally) to `phase`, if one could be written at all. Folding the `Option` in /// here keeps every phase boundary a single line and keeps "no breadcrumb" from being a condition /// the scan branches on: evidence is never a precondition for the work. @@ -792,6 +817,13 @@ pub fn scan_paths_with_observer( let scope = derive_scope(root, &source); let mut report = ScanReport::default(); + // The contents these paths hold now: once the pass has replaced or dropped them, a content no + // other path holds leaves its grep bloom row too (a long-lived watcher never runs the full sweep). + let mut retired: Vec = rels + .iter() + .chain(&removed) + .filter_map(|rel| store.lookup(rel).map(|entry| entry.hash_hex.clone())) + .collect(); let drive = ScanDrive { root, filters: filter.filters(), @@ -840,6 +872,7 @@ pub fn scan_paths_with_observer( } for rel in &removed { + retired.extend(store.lookup(rel).map(|entry| entry.hash_hex.clone())); store.remove(rel); if let Some(idx) = store.index_db.as_ref() { let mut w = idx.writer(); @@ -859,6 +892,8 @@ pub fn scan_paths_with_observer( report.stats.removed += 1; } + drop_unheld_grep_bloom_rows(store, &retired); + store.flush()?; store.write_status_sidecar(); diff --git a/src/scanner_file.rs b/src/scanner_file.rs index 9dbf76d3..02976198 100644 --- a/src/scanner_file.rs +++ b/src/scanner_file.rs @@ -298,6 +298,13 @@ fn process_file( return FileResult::bare(rel.to_string(), FileStatus::SkippedNonUtf8); } + // A stamp taken too close to the clock cannot vouch for the bytes just read (see `settled_mtime`). + let mtime = if matches!(source, ScanSource::WorkingTree) { + settled_mtime(mtime, now_nanos()) + } else { + mtime + }; + let hash = hashing::hash_bytes(&bytes); let hex_buf = hashing::hex_buf(&hash); let hash_hex_str = hashing::hex_str(&hex_buf); @@ -649,6 +656,31 @@ pub fn mtime_nanos(metadata: &std::fs::Metadata) -> i64 { .unwrap_or(0) } +/// How close to the clock a file's mtime may be before the filesystem's timestamp granularity can +/// no longer tell a later edit from the content just read. +const RACY_WINDOW_NANOS: u64 = 2_000_000_000; + +/// The wall clock as nanoseconds since the Unix epoch, saturating like [`mtime_nanos`]. +fn now_nanos() -> i64 { + SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .ok() + .map_or(0, |d| i64::try_from(d.as_nanos()).unwrap_or(i64::MAX)) +} + +/// The mtime to record for bytes read when the clock said `now_ns`: `mtime_ns` itself when it is far +/// enough from `now_ns`, else 0. An edit landing in the same timestamp tick as the read could keep +/// both size and mtime, so the grep bloom guard (which holds a live file to its indexed +/// `(size, mtime)`) must not trust such a stamp; 0 is the "unknown" stamp it never trusts. The next +/// scan re-reads the file, finds the same hash and restamps it once it has settled. +pub(crate) fn settled_mtime(mtime_ns: i64, now_ns: i64) -> i64 { + if mtime_ns.abs_diff(now_ns) < RACY_WINDOW_NANOS { + 0 + } else { + mtime_ns + } +} + fn read_working_tree( root: &Path, rel: &str, diff --git a/src/store.rs b/src/store.rs index 49652739..0328e2f3 100644 --- a/src/store.rs +++ b/src/store.rs @@ -12,6 +12,7 @@ use crate::index::{IndexDb, IndexError}; #[cfg(feature = "intelligence")] use crate::lance::LanceStore; use crate::path::RelPath; +use crate::store_hash_refs::HashRefs; /// Cache layout: where the global cache root is, how a worktree root maps to a workspace cache /// dir, and the `workspace.json` marker recording that mapping. Lives in its own module (like @@ -283,6 +284,10 @@ pub struct Store { pub view_dir: PathBuf, pub view: String, pub index: Index, + /// Which content hashes `index.files` holds, kept in step by [`Store::upsert`] / [`Store::remove`] + /// so "does any file still hold this content" is a lookup, not a pass over the manifest. Mutate + /// `index.files` through those two methods only. + hash_refs: HashRefs, /// Fjall-backed inverted index over symbols / calls / imports. Lives under /// `view_dir/index.fjall/`. Reads + writes from any caller go through `IndexDb`, /// which is cheap to clone (internally Arc'd). `None` in read-only mode when the @@ -376,6 +381,7 @@ impl Store { blobs_shared, view_dir, view: view.to_string(), + hash_refs: HashRefs::of(&index.files), index, index_db, #[cfg(feature = "intelligence")] @@ -470,6 +476,7 @@ impl Store { blobs_shared, view_dir, view: view.to_string(), + hash_refs: HashRefs::of(&index.files), index, index_db, #[cfg(feature = "intelligence")] @@ -507,11 +514,21 @@ impl Store { } pub fn upsert(&mut self, rel: impl Into, entry: FileEntry) { - self.index.files.insert(rel.into(), entry); + self.hash_refs.acquire(&entry.hash_hex); + if let Some(prior) = self.index.files.insert(rel.into(), entry) { + self.hash_refs.release(&prior.hash_hex); + } } pub fn remove(&mut self, rel: impl AsRef<[u8]>) { - self.index.files.remove(bstr::BStr::new(rel.as_ref())); + if let Some(prior) = self.index.files.remove(bstr::BStr::new(rel.as_ref())) { + self.hash_refs.release(&prior.hash_hex); + } + } + + /// True when some indexed file holds the content `hash_hex`. + pub fn hash_is_held(&self, hash_hex: &str) -> bool { + self.hash_refs.is_held(hash_hex) } /// Look a file up by its repository-relative path. Accepts any byte source — diff --git a/src/store_hash_refs.rs b/src/store_hash_refs.rs new file mode 100644 index 00000000..0ed82ba9 --- /dev/null +++ b/src/store_hash_refs.rs @@ -0,0 +1,71 @@ +//! Reference counts over the content hashes the file manifest holds. + +use ahash::AHashMap; + +use crate::path::RelPath; +use crate::store::FileEntry; + +/// How many manifest entries hold each content hash. Keys are 8-byte digests of the hex string, not +/// the 64-character strings, since there is one per distinct content. Two hashes sharing a digest +/// share a count, so a collision can only keep a hash "held" a little longer, never free a held one. +#[derive(Debug, Default)] +pub(crate) struct HashRefs(AHashMap); + +fn digest(hash_hex: &str) -> u64 { + // Fixed seeds: the digest only has to be stable within the process. + ahash::RandomState::with_seeds(1, 2, 3, 4).hash_one(hash_hex) +} + +impl HashRefs { + pub(crate) fn of(files: &AHashMap) -> Self { + let mut refs = Self::default(); + for entry in files.values() { + refs.acquire(&entry.hash_hex); + } + refs + } + + pub(crate) fn acquire(&mut self, hash_hex: &str) { + *self.0.entry(digest(hash_hex)).or_default() += 1; + } + + pub(crate) fn release(&mut self, hash_hex: &str) { + if let std::collections::hash_map::Entry::Occupied(mut held) = self.0.entry(digest(hash_hex)) { + if *held.get() > 1 { + *held.get_mut() -= 1; + } else { + held.remove(); + } + } + } + + pub(crate) fn is_held(&self, hash_hex: &str) -> bool { + self.0.contains_key(&digest(hash_hex)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_hash_stays_held_until_its_last_holder_releases_it() { + let mut refs = HashRefs::default(); + refs.acquire("h1"); + refs.acquire("h1"); + refs.acquire("h2"); + refs.release("h1"); + assert!(refs.is_held("h1") && refs.is_held("h2")); + refs.release("h1"); + assert!(!refs.is_held("h1") && refs.is_held("h2")); + } + + #[test] + fn releasing_an_unheld_hash_is_a_no_op() { + let mut refs = HashRefs::default(); + refs.release("never"); + refs.acquire("h1"); + refs.release("never"); + assert!(refs.is_held("h1")); + } +} diff --git a/tests/git_smoke.rs b/tests/git_smoke.rs index b37fb7ca..b52b07ad 100644 --- a/tests/git_smoke.rs +++ b/tests/git_smoke.rs @@ -377,3 +377,59 @@ fn list_worktrees_reports_main_and_linked_with_branches() { assert_eq!(d.branch, None, "detached HEAD has no branch"); assert_eq!(d.head_sha.as_deref(), Some(head.as_str())); } + +/// Only a working-tree pass writes grep bloom rows, so the full-pass sweep (which deletes every row no +/// manifest entry names) must not run on a git source: it would erase rows it cannot know are wanted. +#[test] +fn staged_and_rev_scans_do_not_sweep_grep_bloom_rows() { + let (dir, cfg) = init_repo(); + let root = dir.path(); + fs::write(root.join("a.rs"), b"pub fn clean_one() {}\n").unwrap(); + run(root, &["add", "a.rs"]); + run(root, &["commit", "-qm", "init"]); + let repo = Repo::discover(root).unwrap(); + let head_sha = repo.resolve_rev("HEAD").unwrap(); + let rev_view = format!("rev-{}", &head_sha[..7]); + + let plant = |store: &Store| { + let mut writer = store.index_db.as_ref().expect("index").writer(); + writer.upsert_grep_bloom("orphan", basemind::index::grep_bloom::build_row(b"held by no file")); + writer.commit().expect("commit"); + }; + let planted = |store: &Store| { + store + .index_db + .as_ref() + .expect("index") + .grep_bloom_row("orphan") + .is_some() + }; + + let mut staged = Store::open(root, VIEW_STAGED).unwrap(); + plant(&staged); + scan( + root, + &mut staged, + &cfg, + ScanSource::Staged(&repo), + basemind::scanner::EmbedMode::Inline, + ) + .unwrap(); + assert!(planted(&staged), "a staged scan must not sweep"); + drop(staged); + + let mut rev = Store::open(root, &rev_view).unwrap(); + plant(&rev); + scan( + root, + &mut rev, + &cfg, + ScanSource::Rev { + repo: &repo, + sha: head_sha, + }, + basemind::scanner::EmbedMode::Inline, + ) + .unwrap(); + assert!(planted(&rev), "a rev scan must not sweep"); +} diff --git a/tests/grep_bloom_scan.rs b/tests/grep_bloom_scan.rs index 857fbef1..b47859a5 100644 --- a/tests/grep_bloom_scan.rs +++ b/tests/grep_bloom_scan.rs @@ -4,34 +4,73 @@ //! asserts what grep is allowed to trust. use std::path::Path; +use std::time::{Duration, SystemTime}; use basemind::path::RelPath; +use basemind::scanner::{EmbedMode, ScanCancel, ScanSource}; use basemind::store::{Store, VIEW_WORKING}; -fn scan(root: &Path) -> Store { +fn config(root: &Path) -> basemind::config::Config { let mut cfg = basemind::config::default_for_root(root); cfg.documents.embed = false; cfg.code_search.embed = false; + cfg +} + +/// Run `work` on a fresh thread, as the scanner's own callers do. +fn off_main(work: impl FnOnce() -> T + Send) -> T { let _ = basemind::lang::ensure_grammars().expect("grammar bootstrap"); - std::thread::scope(|scope| { - scope - .spawn(|| { - let mut store = Store::open(root, VIEW_WORKING).expect("open store"); - basemind::scanner::scan( - root, - &mut store, - &cfg, - basemind::scanner::ScanSource::WorkingTree, - basemind::scanner::EmbedMode::Inline, - ) - .expect("scan"); - store - }) - .join() - .expect("scan thread") + std::thread::scope(|scope| scope.spawn(work).join().expect("scan thread")) +} + +fn scan_cancellable(root: &Path, cancel: &ScanCancel) -> Store { + let cfg = config(root); + off_main(|| { + let mut store = Store::open(root, VIEW_WORKING).expect("open store"); + basemind::scanner::scan_with_cancel( + root, + &mut store, + &cfg, + ScanSource::WorkingTree, + EmbedMode::Inline, + cancel, + ) + .expect("scan"); + store }) } +fn scan(root: &Path) -> Store { + scan_cancellable(root, &ScanCancel::new()) +} + +/// The watcher's path: rescan only `rels` into the open `store`. +fn scan_paths(root: &Path, store: &mut Store, rels: &[&str]) { + let cfg = config(root); + let paths: Vec<_> = rels.iter().map(|rel| root.join(rel)).collect(); + off_main(|| basemind::scanner::scan_paths(root, store, &cfg, &paths, EmbedMode::Inline).expect("scan_paths")); +} + +/// Move a file's mtime an hour back, past the window in which its stamp is too fresh to trust. +fn age(root: &Path, rel: &str) { + let file = std::fs::OpenOptions::new().write(true).open(root.join(rel)).unwrap(); + file.set_modified(SystemTime::now() - Duration::from_secs(3600)) + .unwrap(); +} + +fn write_aged(root: &Path, rel: &str, text: &str) { + std::fs::write(root.join(rel), text).unwrap(); + age(root, rel); +} + +fn orphan_row(store: &Store, key: &str) { + put_raw( + store, + key, + basemind::index::grep_bloom::build_row(b"nothing indexed holds this"), + ); +} + fn hash_of(root: &Path, rel: &str) -> String { basemind::hashing::hex(&basemind::hashing::hash_bytes( &std::fs::read(root.join(rel)).expect("read"), @@ -119,7 +158,7 @@ fn a_touched_file_keeps_its_indexed_stamp_current() { basemind::store::init_isolated_cache(); let dir = tempfile::tempdir().expect("tempdir"); let root = dir.path(); - std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); let store = scan(root); assert_eq!(indexed_stamp(&store, "a.rs"), live_stamp(root, "a.rs")); drop(store); @@ -207,7 +246,7 @@ fn v1_rows_are_absent_on_read_so_grep_scans_every_file() { basemind::store::init_isolated_cache(); let dir = tempfile::tempdir().expect("tempdir"); let root = dir.path(); - std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); let hash = hash_of(root, "a.rs"); let store = scan(root); let entry = grep_entry(&store, root, "a.rs"); @@ -236,7 +275,7 @@ fn the_next_incremental_scan_sweeps_v1_rows_and_rebuilds_content_keyed_ones() { basemind::store::init_isolated_cache(); let dir = tempfile::tempdir().expect("tempdir"); let root = dir.path(); - std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); let hash = hash_of(root, "a.rs"); let store = scan(root); downgrade_to_v1(&store, "a.rs", &hash); @@ -254,10 +293,142 @@ fn the_next_incremental_scan_sweeps_v1_rows_and_rebuilds_content_keyed_ones() { } #[test] -fn migrating_grep_bloom_rows_does_not_change_the_index_schema_version() { +fn migrating_grep_bloom_rows_does_not_wipe_the_index() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + let hash = hash_of(root, "a.rs"); + let store = scan(root); + downgrade_to_v1(&store, "a.rs", &hash); + drop(store); + + let store = scan(root); + assert!(!store.reset_stale_schema && !store.index_rebuild_needed); + assert!( + !store.index_db.as_ref().expect("index").wiped_on_open(), + "rows migrate by row version, so no user re-indexes for them" + ); + assert!(store.lookup(RelPath::from("a.rs")).is_some()); +} + +#[test] +fn a_cancelled_scan_does_not_sweep() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + let store = scan(root); + orphan_row(&store, "orphan"); + drop(store); + + let cancel = ScanCancel::new(); + cancel.cancel(); + let store = scan_cancellable(root, &cancel); + assert!( + has_row(&store, "orphan"), + "a partial pass cannot tell dead rows from unvisited ones" + ); + drop(store); + + let store = scan(root); + assert!(!has_row(&store, "orphan"), "the next complete pass sweeps it"); +} + +#[test] +fn scan_paths_deletes_the_superseded_row_on_edit_and_on_delete() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + write_aged(root, "b.rs", "pub fn beta_function_name() -> u32 { 2 }\n"); + let (a_old, b_old) = (hash_of(root, "a.rs"), hash_of(root, "b.rs")); + let mut store = scan(root); + + std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 100_000 }\n").unwrap(); + let a_new = hash_of(root, "a.rs"); + scan_paths(root, &mut store, &["a.rs"]); + assert!( + has_row(&store, &a_new) && !has_row(&store, &a_old), + "an edit moves the row" + ); + assert!(has_row(&store, &b_old), "an untouched file keeps its row"); + + std::fs::remove_file(root.join("b.rs")).unwrap(); + scan_paths(root, &mut store, &["b.rs"]); + assert!(!has_row(&store, &b_old), "a deleted file's row goes"); + assert_eq!(row_count(&store), 1, "no growth across incremental edits"); +} + +#[test] +fn scan_paths_keeps_a_row_while_another_path_still_holds_the_content() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + let shared = "pub fn shared_function_name() -> u32 { 1 }\n"; + write_aged(root, "a.rs", shared); + write_aged(root, "b.rs", shared); + let hash = hash_of(root, "a.rs"); + let mut store = scan(root); + + std::fs::write(root.join("a.rs"), "pub fn changed_function_name() -> u32 { 2 }\n").unwrap(); + scan_paths(root, &mut store, &["a.rs"]); + assert!(has_row(&store, &hash), "b.rs still holds the old content"); + + std::fs::remove_file(root.join("b.rs")).unwrap(); + scan_paths(root, &mut store, &["b.rs"]); + assert!(!has_row(&store, &hash), "the last holder is gone"); +} + +#[test] +fn scan_paths_keeps_a_row_a_path_in_the_same_batch_moved_onto() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + write_aged(root, "b.rs", "pub fn beta_function_name() -> u32 { 2 }\n"); + let (a_old, b_old) = (hash_of(root, "a.rs"), hash_of(root, "b.rs")); + let mut store = scan(root); + + std::fs::write(root.join("a.rs"), "pub fn beta_function_name() -> u32 { 2 }\n").unwrap(); + std::fs::write(root.join("b.rs"), "pub fn gamma_function_name() -> u32 { 3 }\n").unwrap(); + scan_paths(root, &mut store, &["a.rs", "b.rs"]); + assert!(has_row(&store, &b_old), "a.rs now holds b.rs's old content"); + assert!(!has_row(&store, &a_old)); + assert_eq!(row_count(&store), 2); +} + +#[test] +fn a_freshly_written_file_is_never_skipped_until_a_later_scan_restamps_it() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "warm.rs", "pub fn warm_up_function() -> u32 { 0 }\n"); + let mut store = scan(root); + let absent = basemind::index::grep_bloom::Needle::compile("zzz_absent_literal").expect("needle"); + let verdict = |store: &Store| { + let entry = grep_entry(store, root, "a.rs"); + let idx = store.index_db.clone().expect("index"); + idx.grep_skip_verdicts(root, &[entry], &absent)[0] + }; + + // The first scan paid for the grammar bootstrap; an incremental pass on the open store is + // quick, so the new file's mtime is well inside the window when it is read. + std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); + scan_paths(root, &mut store, &["a.rs"]); assert_eq!( - basemind::index::INDEX_SCHEMA_VER, - basemind::version::RELEASE_MINOR as u32 + 6, - "pre-A2 value: a schema bump would force every user to re-index" + indexed_stamp(&store, "a.rs").1, + 0, + "a stamp this close to the clock is not recorded" + ); + assert!(!verdict(&store), "an untrusted stamp keeps the file a candidate"); + + age(root, "a.rs"); + scan_paths(root, &mut store, &["a.rs"]); + assert_eq!( + indexed_stamp(&store, "a.rs"), + live_stamp(root, "a.rs"), + "restamped once settled" ); + assert!(verdict(&store), "a settled stamp is trusted again"); } From 938dc8b2dbbc75914a031e72435f81c9d0c2aac3 Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Mon, 19 Oct 2026 17:26:40 +0200 Subject: [PATCH 07/10] refactor(index): make IndexView backend-neutral and build its named reads on the generic layer Co-Authored-By: Claude Opus 5.5 --- src/comms/workspace_pool_tests.rs | 3 +- src/index/grep_bloom.rs | 23 +-- src/index/grep_bloom_tests.rs | 28 +-- src/index/mod.rs | 154 -------------- src/index/view.rs | 327 ++++++++++++++++++------------ src/index/view_tests.rs | 134 +++++++++--- src/index/writer_tests.rs | 14 +- src/mcp/helpers_calls_scan.rs | 8 +- src/mcp/helpers_grep.rs | 16 +- src/mcp/index_view_tests.rs | 73 +++++-- src/query.rs | 2 + src/scanner_code.rs | 5 +- tests/config_scan_rules.rs | 4 +- tests/git_smoke.rs | 2 + tests/grep_bloom_scan.rs | 14 +- 15 files changed, 425 insertions(+), 382 deletions(-) diff --git a/src/comms/workspace_pool_tests.rs b/src/comms/workspace_pool_tests.rs index 5b311ef9..88286ac4 100644 --- a/src/comms/workspace_pool_tests.rs +++ b/src/comms/workspace_pool_tests.rs @@ -6,6 +6,7 @@ use std::time::Duration; use super::*; +use crate::index::view::IndexView; /// History host for the fixtures, which care about the read stack and not about git history. /// @@ -932,7 +933,7 @@ fn a_workspace_opened_over_a_wiped_index_is_marked_and_rebuilt_by_the_next_resca assert!(!entry.rebuild_pending.load(std::sync::atomic::Ordering::Acquire)); let repopulated = pool .with_workspace(ws.path(), |s| { - !s.index_db.as_ref().expect("index").symbols_index_is_empty() && !s.index_rebuild_needed + !s.index_db.as_ref().expect("index").view().symbols_index_is_empty() && !s.index_rebuild_needed }) .expect("workspace"); assert!(repopulated, "the fjall index holds symbols again"); diff --git a/src/index/grep_bloom.rs b/src/index/grep_bloom.rs index 9bce59c3..4c8b1f1b 100644 --- a/src/index/grep_bloom.rs +++ b/src/index/grep_bloom.rs @@ -18,6 +18,7 @@ use rayon::prelude::*; use regex_syntax::hir::literal::{ExtractKind, Extractor}; use super::IndexDb; +use super::view::{IndexBytes, IndexView}; use crate::path::RelPath; /// Row layout revision (first byte). A row with any other value is treated as absent, so a layout @@ -162,11 +163,6 @@ impl Needle { } impl IndexDb { - /// Fetch the stored bloom row for the content `hash_hex`, if any. - pub fn grep_bloom_row(&self, hash_hex: &str) -> Option { - self.grep_bloom.get(hash_hex.as_bytes()).ok().flatten() - } - /// Delete the row for `hash_hex`: how a test models an index built before the keyspace existed. #[doc(hidden)] #[cfg(any(test, feature = "test-support"))] @@ -183,7 +179,8 @@ impl IndexDb { /// True when `hash_hex` has an intact row: nothing to (re)build. pub fn grep_bloom_current(&self, hash_hex: &str) -> bool { - self.grep_bloom_row(hash_hex) + self.view() + .grep_bloom_row(hash_hex) .is_some_and(|row| parse_row(&row).is_some()) } @@ -234,19 +231,11 @@ impl IndexDb { } writer.commit() } - - /// For each of `entries`, whether the file can be SKIPPED by a grep for `needle`: its content has - /// an intact row, the row rejects the needle, and the live file still has the stamp the entry was - /// indexed under. Rows are fetched and dropped one at a time, in parallel; nothing is held - /// between files. - pub fn grep_skip_verdicts(&self, root: &Path, entries: &[GrepEntry], needle: &Needle) -> Vec { - skip_verdicts(|hash| self.grep_bloom_row(hash), root, entries, needle) - } } -/// [`IndexDb::grep_skip_verdicts`] over any row source, so a layered view shares the verdict logic. -pub(crate) fn skip_verdicts( - row_of: impl Fn(&str) -> Option + Sync, +/// The verdict logic behind [`IndexView::grep_skip_verdicts`], over any row source. +pub(super) fn skip_verdicts( + row_of: impl Fn(&str) -> Option + Sync, root: &Path, entries: &[GrepEntry], needle: &Needle, diff --git a/src/index/grep_bloom_tests.rs b/src/index/grep_bloom_tests.rs index 1722671d..b5c31eb0 100644 --- a/src/index/grep_bloom_tests.rs +++ b/src/index/grep_bloom_tests.rs @@ -81,7 +81,7 @@ fn a_row_round_trips_under_its_content_hash() { let (_dir, db) = open_db(); put(&db, "aa11", "fn alpha() {}"); assert_eq!( - db.grep_bloom_row("aa11").as_deref(), + db.view().grep_bloom_row("aa11").as_deref(), Some(row_for("fn alpha() {}").as_slice()) ); assert!(db.grep_bloom_current("aa11")); @@ -94,14 +94,14 @@ fn a_hash_that_prefixes_another_reads_only_its_own_row() { put(&db, "ab", "short key"); put(&db, "abc", "longer key"); assert_eq!( - db.grep_bloom_row("ab").as_deref(), + db.view().grep_bloom_row("ab").as_deref(), Some(row_for("short key").as_slice()) ); assert_eq!( - db.grep_bloom_row("abc").as_deref(), + db.view().grep_bloom_row("abc").as_deref(), Some(row_for("longer key").as_slice()) ); - assert!(db.grep_bloom_row("a").is_none()); + assert!(db.view().grep_bloom_row("a").is_none()); } #[test] @@ -141,7 +141,10 @@ fn two_paths_with_the_same_content_share_one_row_and_both_skip() { put(&db, "same", text); assert_eq!(db.grep_bloom_row_count(), 1); let needle = Needle::compile("zzz_absent_literal").expect("needle"); - assert_eq!(db.grep_skip_verdicts(root.path(), &entries, &needle), vec![true, true]); + assert_eq!( + db.view().grep_skip_verdicts(root.path(), &entries, &needle), + vec![true, true] + ); } #[test] @@ -159,11 +162,12 @@ fn a_live_file_that_moved_off_its_indexed_stamp_is_a_candidate() { mtime_ns: mtime_nanos(&meta), }; assert_eq!( - db.grep_skip_verdicts(root.path(), &[entry(meta.len())], &needle), + db.view().grep_skip_verdicts(root.path(), &[entry(meta.len())], &needle), vec![true] ); assert_eq!( - db.grep_skip_verdicts(root.path(), &[entry(meta.len() + 1)], &needle), + db.view() + .grep_skip_verdicts(root.path(), &[entry(meta.len() + 1)], &needle), vec![false] ); } @@ -189,7 +193,7 @@ fn the_first_sweep_drops_older_layout_rows_and_records_it_in_meta() { 1, "a live hash with an unreadable row goes" ); - assert!(db.grep_bloom_current("h1") && db.grep_bloom_row("h2").is_none()); + assert!(db.grep_bloom_current("h1") && db.view().grep_bloom_row("h2").is_none()); assert_eq!( db.meta.get(META_ROW_SWEPT_VER).expect("meta").as_deref(), Some([ROW_VERSION].as_slice()) @@ -207,11 +211,11 @@ fn later_sweeps_read_keys_only_and_still_remove_dead_keys() { put(&db, "dead", "no file holds this"); assert_eq!(db.sweep_grep_bloom(&live).expect("steady sweep"), 1); assert!( - db.grep_bloom_row("dead").is_none(), + db.view().grep_bloom_row("dead").is_none(), "a dead key is removed without reading it" ); assert!( - db.grep_bloom_row("h2").is_some(), + db.view().grep_bloom_row("h2").is_some(), "the value of a live key is not decoded after the version sweep ran" ); } @@ -240,12 +244,12 @@ fn remove_grep_bloom_rows_drops_only_the_named_hashes() { db.remove_grep_bloom_rows(["h1", "h3", "never-indexed"]) .expect("remove"); assert!(db.grep_bloom_current("h2")); - assert!(db.grep_bloom_row("h1").is_none() && db.grep_bloom_row("h3").is_none()); + assert!(db.view().grep_bloom_row("h1").is_none() && db.view().grep_bloom_row("h3").is_none()); } fn skip_for(db: &IndexDb, root: &Path, entry: GrepEntry) -> bool { let needle = Needle::compile("zzz_absent_literal").expect("needle"); - db.grep_skip_verdicts(root, &[entry], &needle)[0] + db.view().grep_skip_verdicts(root, &[entry], &needle)[0] } #[test] diff --git a/src/index/mod.rs b/src/index/mod.rs index 980441eb..7dbec0de 100644 --- a/src/index/mod.rs +++ b/src/index/mod.rs @@ -648,44 +648,6 @@ impl IndexDb { writer::IndexWriter::new(self.clone()) } - /// True when the secondary index holds no per-file symbol entries. Cheap — peeks at - /// the first key of `symbols_by_path` rather than counting. - /// - /// Used by the MCP startup auto-scan to detect a present-but-empty Fjall index (e.g. a - /// `views//index.fjall/` that was wiped or removed out-of-band while the msgpack - /// `index.msgpack` survived). In that state the in-RAM map cache looks populated but the - /// Fjall-backed tools (`find_references` / `search_symbols`) would silently return nothing, - /// so a rescan is warranted even though the RAM cache is non-empty. - pub fn symbols_index_is_empty(&self) -> bool { - self.symbols_by_path.iter().next().is_none() - } - - /// Resolved references to the definition at `(def_path, def_start)` — the scope/import-resolved - /// backing for `find_references`. Returns each binding `(use_path, use_start)`; empty when the - /// definition has no resolved uses (or resolution never ran for its file). - pub fn references_to(&self, def_path: &crate::path::RelPath, def_start: u32) -> Vec<(crate::path::RelPath, u32)> { - let prefix = keys::refs_by_def_prefix(def_path, def_start); - let mut out = Vec::new(); - for guard in self.refs_by_def.prefix(prefix) { - if let Ok((k, _)) = guard.into_inner() - && let Some((_def_path, _def_start, use_path, use_start)) = keys::parse_ref_by_def(&k) - { - out.push((use_path, use_start)); - } - } - out - } - - /// Whether the probe keyspaces cover every importer (a wholesale resolve pass completed since - /// they were created). When false, the incremental pass must run the wholesale one first. - pub fn import_probes_complete(&self) -> bool { - self.meta - .get(META_IMPORT_PROBE_VER) - .ok() - .flatten() - .is_some_and(|v| v.first() == Some(&IMPORT_PROBE_VER)) - } - /// Mark the probe keyspaces complete (`true`) or in flux (`false`). The wholesale pass clears /// the stamp before it rewrites any importer and sets it only after the stitch committed. pub fn set_import_probes_complete(&self, complete: bool) -> Result<(), IndexError> { @@ -697,37 +659,6 @@ impl IndexDb { Ok(()) } - /// Every importer whose resolution consulted `path` (repo-relative, forward-slashed) or one of - /// its ancestor directories: a prefix scan of `importers_by_probe` per candidate, O(importers). - /// - /// The ancestors matter because the resolver does not descend into a directory it found - /// missing. An importer that probed `util` and found nothing never probed `util/index.ts`, so - /// adding `util/index.ts` must reach it through the directory. (A directory that existed when - /// the importer resolved is never recorded, so a path under an existing directory costs only - /// empty scans.) - pub fn importers_probing(&self, path: &str) -> Vec { - probe_candidates(path) - .flat_map(|candidate| { - let prefix = keys::importers_by_probe_prefix(&keys::probe_hash(candidate)); - self.importers_by_probe - .prefix(prefix) - .filter_map(|guard| guard.into_inner().ok()) - .filter_map(|(k, _)| keys::parse_importer_by_probe(&k)) - .collect::>() - }) - .collect() - } - - /// Whether any importer consulted `path` or an ancestor directory of it (see - /// [`Self::importers_probing`]). Cheap enough to gate a rescan on a non-indexed file - /// (`tsconfig.json`, `package.json`) the watcher saw change. - pub fn is_probed(&self, path: &str) -> bool { - probe_candidates(path).any(|candidate| { - let prefix = keys::importers_by_probe_prefix(&keys::probe_hash(candidate)); - self.importers_by_probe.prefix(prefix).next().is_some() - }) - } - /// Every row of the four resolution keyspaces (`refs_by_def`, `refs_by_path`, `import_targets`, /// `importers_by_probe`) as `(keyspace, key, value)`, in keyspace then key order. Lets an /// equivalence test compare an incremental rescan with a full one row for row. @@ -748,91 +679,6 @@ impl IndexDb { Ok(rows) } - /// The stitched `(local_start, def_path, def_start)` bindings of `importer`. - pub fn import_targets_of(&self, importer: &crate::path::RelPath) -> Vec<(u32, crate::path::RelPath, u32)> { - self.import_targets - .prefix(keys::import_targets_prefix(importer)) - .filter_map(|guard| guard.into_inner().ok()) - .filter_map(|(k, v)| keys::parse_import_target(&k, &v)) - .map(|(_, local_start, def_path, def_start)| (local_start, def_path, def_start)) - .collect() - } - - /// The definition the use at `(use_path, use_start)` binds to — backs `goto_definition`. - /// Cross-file definitions take precedence over the local import binding that an imported use - /// also resolves through. `None` when the position isn't a resolved reference. - pub fn definition_of( - &self, - use_path: &crate::path::RelPath, - use_start: u32, - ) -> Option<(crate::path::RelPath, u32)> { - let prefix = keys::refs_by_use_prefix(use_path, use_start); - let mut local = None; - for guard in self.refs_by_path.prefix(prefix) { - if let Ok((k, _)) = guard.into_inner() - && let Some((_use_path, _use_start, def_path, def_start)) = keys::parse_ref_by_path(&k) - { - if def_path != *use_path { - return Some((def_path, def_start)); - } - local = Some((def_path, def_start)); - } - } - local - } - - /// Symbols whose name starts with `name`, from the `symbols_by_name` keyspace — an index-backed - /// prefix scan (length-prefixed keys isolate `Foo` from `Foobar`). Returns `(name, kind, path, - /// start_byte)` for each match, capped at `cap` entries. Backs the code-search **exact lane**: - /// an identifier-shaped query resolves to the symbols that define it, which then map to their - /// owning chunks. The returned `name` lets the caller rank exact-name matches ahead of longer - /// prefix matches. `start_byte` is the L1 `Symbol.start_byte` (node start), which falls inside - /// the symbol's owning chunk span. - pub fn symbols_by_name_lookup( - &self, - name: &str, - cap: usize, - ) -> Vec<(String, crate::extract::SymbolKind, crate::path::RelPath, u32)> { - let prefix = keys::symbols_by_name_prefix(name); - let mut out = Vec::new(); - for guard in self.symbols_by_name.prefix(prefix) { - if out.len() >= cap { - break; - } - if let Ok((k, _)) = guard.into_inner() - && let Some((matched, kind, rel, start_byte)) = keys::parse_symbol_by_name(&k) - { - out.push((matched, kind, rel, start_byte)); - } - } - out - } - - /// Corpus-global BM25 stats for the code-search keyword lane: `(N, total_len)` where `N` is the - /// number of indexed chunks and `total_len` the sum of their token lengths (so `avgdl = - /// total_len / N`). Read from the `meta` keyspace at query time. `None` (or `N == 0`) means the - /// BM25 index is empty — no chunks were indexed, or [`recompute_bm25_stats`] never ran. - /// - /// [`recompute_bm25_stats`]: Self::recompute_bm25_stats - pub fn bm25_stats(&self) -> Option<(u64, u64)> { - let n = self - .meta - .get(META_BM25_DOC_COUNT) - .ok() - .flatten() - .and_then(|b| <[u8; 8]>::try_from(&b[..]).ok()) - .map(u64::from_be_bytes)?; - let total_len = self - .meta - .get(META_BM25_TOTAL_LEN) - .ok() - .flatten() - .and_then(|b| <[u8; 8]>::try_from(&b[..]).ok()) - .map(u64::from_be_bytes) - .unwrap_or(0); - Some((n, total_len)) - } - /// Recompute the corpus-global BM25 stats by sweeping the `code_bm25_by_path` forward keyspace /// once (one entry per chunk; only its 4-byte `doclen` prefix is decoded — the term list is not /// touched) and stamping `(N, total_len)` into `meta`. Runs single-threaded in the scanner's diff --git a/src/index/view.rs b/src/index/view.rs index e78a59bc..00b13644 100644 --- a/src/index/view.rs +++ b/src/index/view.rs @@ -6,21 +6,75 @@ //! //! The trait has two layers. The generic layer (`get` / `prefix` / `range` / `iter`, addressed by //! [`Part`]) is what a merging implementation has to provide; the named accessors are the lookups the -//! read path performs, each delegating 1:1 to the matching [`IndexDb`] method in [`StandaloneView`]. +//! read path performs, written once as default methods over that layer so a layered view inherits them +//! (ADR-0016 A5 "Merge contract"). No fjall type appears in a signature: values come back as +//! [`IndexBytes`], which a later mapped base can serve without a copy. use std::borrow::Borrow; -use std::ops::Bound; +use std::ops::{Bound, Deref}; use std::path::Path; -use fjall::{Keyspace, Slice}; +use fjall::Keyspace; use super::grep_bloom::{GrepEntry, Needle}; -use super::{IndexDb, IndexError, name_dict}; +use super::{ + IMPORT_PROBE_VER, IndexDb, IndexError, META_BM25_DOC_COUNT, META_BM25_TOTAL_LEN, META_IMPORT_PROBE_VER, keys, + name_dict, probe_candidates, +}; use crate::extract::SymbolKind; use crate::path::RelPath; +/// Bytes read from the index: a key or a value. +/// +/// A refcounted handle, not a copy: today it wraps the fjall block slice the read already produced, and +/// a mapped immutable base will add a variant borrowing the mapping, so a merging view hands either +/// out per row without allocating. The enum is private so no backend type is nameable from a signature. +#[derive(Clone)] +pub struct IndexBytes(Repr); + +#[derive(Clone)] +enum Repr { + Fjall(fjall::Slice), +} + +impl IndexBytes { + fn from_fjall(slice: fjall::Slice) -> Self { + Self(Repr::Fjall(slice)) + } +} + +impl Deref for IndexBytes { + type Target = [u8]; + + fn deref(&self) -> &[u8] { + match &self.0 { + Repr::Fjall(slice) => slice, + } + } +} + +impl AsRef<[u8]> for IndexBytes { + fn as_ref(&self) -> &[u8] { + self + } +} + +impl PartialEq for IndexBytes { + fn eq(&self, other: &Self) -> bool { + **self == **other + } +} + +impl Eq for IndexBytes {} + +impl std::fmt::Debug for IndexBytes { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + (**self).fmt(f) + } +} + /// One `(key, value)` row, or the error that interrupted the walk. -pub type KvResult = Result<(Slice, Slice), IndexError>; +pub type KvResult = Result<(IndexBytes, IndexBytes), IndexError>; /// Boxed row iterator. One virtual call per row; the boxing itself happens once per scan. pub type KvIter<'a> = Box + 'a>; @@ -48,8 +102,10 @@ pub enum Part { } impl Part { - /// Every layered keyspace, in declaration order. - pub const ALL: [Part; 16] = [ + /// Every layered keyspace, in declaration order. Checked against [`KEYSPACE_MEMTABLE_BYTES`] in + /// `view_tests`, so a keyspace added to either list alone fails there. + #[cfg(test)] + pub(super) const ALL: [Part; 16] = [ Part::Meta, Part::SymbolsByPath, Part::SymbolsByName, @@ -69,7 +125,8 @@ impl Part { ]; /// The keyspace's on-disk name. - pub const fn name(self) -> &'static str { + #[cfg(test)] + pub(super) const fn name(self) -> &'static str { match self { Part::Meta => "meta", Part::SymbolsByPath => "symbols_by_path", @@ -136,24 +193,25 @@ impl IndexDb { } /// `Some(&dyn IndexView)` for an optional concrete view, for the call sites that take an optional one. -pub fn as_dyn(view: &Option) -> Option<&dyn IndexView> { +pub(crate) fn as_dyn(view: &Option) -> Option<&dyn IndexView> { view.as_ref().map(|v| v as &dyn IndexView) } -pub(crate) fn kv(guard: fjall::Guard) -> KvResult { - guard.into_inner().map_err(IndexError::from) +pub(super) fn kv(guard: fjall::Guard) -> KvResult { + let (key, value) = guard.into_inner().map_err(IndexError::from)?; + Ok((IndexBytes::from_fjall(key), IndexBytes::from_fjall(value))) } /// Read access to the index. See the module docs. pub trait IndexView: Send + Sync { /// Point lookup of `key` in `part`. - fn get(&self, part: Part, key: &[u8]) -> Result, IndexError>; + fn get(&self, part: Part, key: &[u8]) -> Result, IndexError>; /// Every row of `part` whose key starts with `prefix`, in key order. fn prefix<'a>(&'a self, part: Part, prefix: &[u8]) -> KvIter<'a>; /// Rows of `part` between the bounds, in key order. - fn range<'a>(&'a self, part: Part, lower: Bound>, upper: Bound>) -> KvIter<'a>; + fn range<'a>(&'a self, part: Part, lower: Bound<&[u8]>, upper: Bound<&[u8]>) -> KvIter<'a>; /// Every row of `part`, in key order. fn iter<'a>(&'a self, part: Part) -> KvIter<'a>; @@ -175,39 +233,143 @@ pub trait IndexView: Send + Sync { lower: Bound>, ) -> KvIter<'a>; - /// True when the index holds no per-file symbol entries. - fn symbols_index_is_empty(&self) -> bool; + /// True when the index holds no per-file symbol entries. Cheap: peeks at the first key of + /// `symbols_by_path` rather than counting. + /// + /// Used by the MCP startup auto-scan to detect a present-but-empty index (e.g. a + /// `views//index.fjall/` wiped out-of-band while the msgpack `index.msgpack` survived). The + /// in-RAM map cache then looks populated while the index-backed tools would silently return + /// nothing, so a rescan is warranted. + fn symbols_index_is_empty(&self) -> bool { + self.iter(Part::SymbolsByPath).next().is_none() + } - /// Resolved uses of the definition at `(def_path, def_start)`. - fn references_to(&self, def_path: &RelPath, def_start: u32) -> Vec<(RelPath, u32)>; + /// Resolved uses of the definition at `(def_path, def_start)`: the scope/import-resolved backing + /// for `find_references`, each as `(use_path, use_start)`. Empty when the definition has no + /// resolved uses (or resolution never ran for its file). + fn references_to(&self, def_path: &RelPath, def_start: u32) -> Vec<(RelPath, u32)> { + let prefix = keys::refs_by_def_prefix(def_path, def_start); + self.prefix(Part::RefsByDef, &prefix) + .filter_map(|row| row.ok()) + .filter_map(|(k, _)| keys::parse_ref_by_def(&k)) + .map(|(_, _, use_path, use_start)| (use_path, use_start)) + .collect() + } - /// The definition the use at `(use_path, use_start)` resolves to. - fn definition_of(&self, use_path: &RelPath, use_start: u32) -> Option<(RelPath, u32)>; + /// The definition the use at `(use_path, use_start)` resolves to; backs `goto_definition`. + /// Cross-file definitions take precedence over the local import binding that an imported use also + /// resolves through. `None` when the position isn't a resolved reference. + fn definition_of(&self, use_path: &RelPath, use_start: u32) -> Option<(RelPath, u32)> { + let prefix = keys::refs_by_use_prefix(use_path, use_start); + let mut local = None; + for row in self.prefix(Part::RefsByPath, &prefix) { + if let Ok((k, _)) = row + && let Some((_use_path, _use_start, def_path, def_start)) = keys::parse_ref_by_path(&k) + { + if def_path != *use_path { + return Some((def_path, def_start)); + } + local = Some((def_path, def_start)); + } + } + local + } - /// The bindings of `importer`: `(local_start, def_path, def_start)`. - fn import_targets_of(&self, importer: &RelPath) -> Vec<(u32, RelPath, u32)>; + /// The stitched bindings of `importer`: `(local_start, def_path, def_start)`. + fn import_targets_of(&self, importer: &RelPath) -> Vec<(u32, RelPath, u32)> { + self.prefix(Part::ImportTargets, &keys::import_targets_prefix(importer)) + .filter_map(|row| row.ok()) + .filter_map(|(k, v)| keys::parse_import_target(&k, &v)) + .map(|(_, local_start, def_path, def_start)| (local_start, def_path, def_start)) + .collect() + } - /// Importers that probed a path `path` could satisfy. - fn importers_probing(&self, path: &str) -> Vec; + /// Every importer whose resolution consulted `path` (repo-relative, forward-slashed) or one of its + /// ancestor directories: a prefix scan of `importers_by_probe` per candidate, O(importers). + /// + /// The ancestors matter because the resolver does not descend into a directory it found missing. + /// An importer that probed `util` and found nothing never probed `util/index.ts`, so adding + /// `util/index.ts` must reach it through the directory. (A directory that existed when the + /// importer resolved is never recorded, so a path under an existing directory costs only empty + /// scans.) + fn importers_probing(&self, path: &str) -> Vec { + probe_candidates(path) + .flat_map(|candidate| { + let prefix = keys::importers_by_probe_prefix(&keys::probe_hash(candidate)); + self.prefix(Part::ImportersByProbe, &prefix) + .filter_map(|row| row.ok()) + .filter_map(|(k, _)| keys::parse_importer_by_probe(&k)) + .collect::>() + }) + .collect() + } - /// Whether any importer probed a path `path` could satisfy. - fn is_probed(&self, path: &str) -> bool; + /// Whether any importer consulted `path` or an ancestor directory of it (see + /// [`Self::importers_probing`]). Cheap enough to gate a rescan on a non-indexed file + /// (`tsconfig.json`, `package.json`) the watcher saw change. + fn is_probed(&self, path: &str) -> bool { + probe_candidates(path).any(|candidate| { + let prefix = keys::importers_by_probe_prefix(&keys::probe_hash(candidate)); + self.prefix(Part::ImportersByProbe, &prefix).next().is_some() + }) + } - /// Whether the probe keyspaces cover every importer. - fn import_probes_complete(&self) -> bool; + /// Whether the probe keyspaces cover every importer (a wholesale resolve pass completed since they + /// were created). When false, the incremental pass must run the wholesale one first. + fn import_probes_complete(&self) -> bool { + self.get(Part::Meta, META_IMPORT_PROBE_VER) + .ok() + .flatten() + .is_some_and(|v| v.first() == Some(&IMPORT_PROBE_VER)) + } - /// Up to `cap` symbols named `name` (prefix match), as `(name, kind, path, start_byte)`. - fn symbols_by_name_lookup(&self, name: &str, cap: usize) -> Vec<(String, SymbolKind, RelPath, u32)>; + /// Symbols whose name starts with `name` (length-prefixed keys isolate `Foo` from `Foobar`), as + /// `(name, kind, path, start_byte)`, capped at `cap`. Backs the code-search exact lane: an + /// identifier-shaped query resolves to the symbols that define it, which then map to their owning + /// chunks. The returned `name` lets the caller rank exact-name matches ahead of longer prefix + /// matches. `start_byte` is the L1 `Symbol.start_byte` (node start), which falls inside the + /// symbol's owning chunk span. + fn symbols_by_name_lookup(&self, name: &str, cap: usize) -> Vec<(String, SymbolKind, RelPath, u32)> { + let prefix = keys::symbols_by_name_prefix(name); + let mut out = Vec::new(); + for row in self.prefix(Part::SymbolsByName, &prefix) { + if out.len() >= cap { + break; + } + if let Ok((k, _)) = row + && let Some(symbol) = keys::parse_symbol_by_name(&k) + { + out.push(symbol); + } + } + out + } - /// BM25 corpus statistics `(doc_count, total_len)`; `None` before the first scan. - fn bm25_stats(&self) -> Option<(u64, u64)>; + /// Corpus-global BM25 stats for the code-search keyword lane: `(N, total_len)` where `N` is the + /// number of indexed chunks and `total_len` the sum of their token lengths (so `avgdl = total_len + /// / N`). `None` (or `N == 0`) means the BM25 index is empty: no chunks were indexed, or + /// [`IndexDb::recompute_bm25_stats`] never ran. + fn bm25_stats(&self) -> Option<(u64, u64)> { + let stat = |key: &[u8]| { + self.get(Part::Meta, key) + .ok() + .flatten() + .and_then(|b| <[u8; 8]>::try_from(&b[..]).ok()) + .map(u64::from_be_bytes) + }; + let n = stat(META_BM25_DOC_COUNT)?; + Some((n, stat(META_BM25_TOTAL_LEN).unwrap_or(0))) + } /// The stored trigram bloom row for the content `hash_hex`. - fn grep_bloom_row(&self, hash_hex: &str) -> Option { + fn grep_bloom_row(&self, hash_hex: &str) -> Option { self.get(Part::GrepBloom, hash_hex.as_bytes()).ok().flatten() } - /// For each of `entries`, whether a grep for `needle` can skip the file. + /// For each of `entries`, whether the file can be SKIPPED by a grep for `needle`: its content has + /// an intact row, the row rejects the needle, and the live file still has the stamp the entry was + /// indexed under. Rows are fetched and dropped one at a time, in parallel; nothing is held between + /// files. fn grep_skip_verdicts(&self, root: &Path, entries: &[GrepEntry], needle: &Needle) -> Vec { super::grep_bloom::skip_verdicts(|hash| self.grep_bloom_row(hash), root, entries, needle) } @@ -215,7 +377,7 @@ pub trait IndexView: Send + Sync { /// The unlayered view: one [`IndexDb`], read as is. `T` is `&IndexDb` to borrow it or `IndexDb` to /// own a handle (clones share the same keyspaces). -pub struct StandaloneView(pub T); +pub struct StandaloneView(pub(crate) T); impl> StandaloneView { fn db(&self) -> &IndexDb { @@ -224,16 +386,16 @@ impl> StandaloneView { } impl + Send + Sync> IndexView for StandaloneView { - fn get(&self, part: Part, key: &[u8]) -> Result, IndexError> { - Ok(self.db().keyspace(part).get(key)?) + fn get(&self, part: Part, key: &[u8]) -> Result, IndexError> { + Ok(self.db().keyspace(part).get(key)?.map(IndexBytes::from_fjall)) } fn prefix<'a>(&'a self, part: Part, prefix: &[u8]) -> KvIter<'a> { Box::new(self.db().keyspace(part).prefix(prefix).map(kv)) } - fn range<'a>(&'a self, part: Part, lower: Bound>, upper: Bound>) -> KvIter<'a> { - Box::new(self.db().keyspace(part).range::, _>((lower, upper)).map(kv)) + fn range<'a>(&'a self, part: Part, lower: Bound<&[u8]>, upper: Bound<&[u8]>) -> KvIter<'a> { + Box::new(self.db().keyspace(part).range::<&[u8], _>((lower, upper)).map(kv)) } fn iter<'a>(&'a self, part: Part) -> KvIter<'a> { @@ -265,42 +427,6 @@ impl + Send + Sync> IndexView for StandaloneView { lower, ) } - - fn symbols_index_is_empty(&self) -> bool { - self.db().symbols_index_is_empty() - } - - fn references_to(&self, def_path: &RelPath, def_start: u32) -> Vec<(RelPath, u32)> { - self.db().references_to(def_path, def_start) - } - - fn definition_of(&self, use_path: &RelPath, use_start: u32) -> Option<(RelPath, u32)> { - self.db().definition_of(use_path, use_start) - } - - fn import_targets_of(&self, importer: &RelPath) -> Vec<(u32, RelPath, u32)> { - self.db().import_targets_of(importer) - } - - fn importers_probing(&self, path: &str) -> Vec { - self.db().importers_probing(path) - } - - fn is_probed(&self, path: &str) -> bool { - self.db().is_probed(path) - } - - fn import_probes_complete(&self) -> bool { - self.db().import_probes_complete() - } - - fn symbols_by_name_lookup(&self, name: &str, cap: usize) -> Vec<(String, SymbolKind, RelPath, u32)> { - self.db().symbols_by_name_lookup(name, cap) - } - - fn bm25_stats(&self) -> Option<(u64, u64)> { - self.db().bm25_stats() - } } #[cfg(test)] @@ -324,10 +450,6 @@ mod recording { self.0.lock().expect("read log").push(entry); } - pub(crate) fn saw(&self, entry: &str) -> bool { - self.0.lock().expect("read log").iter().any(|e| e == entry) - } - pub(crate) fn entries(&self) -> Vec { self.0.lock().expect("read log").clone() } @@ -357,7 +479,7 @@ mod recording { } impl IndexView for RecordingView { - fn get(&self, part: Part, key: &[u8]) -> Result, IndexError> { + fn get(&self, part: Part, key: &[u8]) -> Result, IndexError> { self.log.record("get", Some(part)); self.inner.get(part, key) } @@ -367,7 +489,7 @@ mod recording { self.inner.prefix(part, prefix) } - fn range<'a>(&'a self, part: Part, lower: Bound>, upper: Bound>) -> KvIter<'a> { + fn range<'a>(&'a self, part: Part, lower: Bound<&[u8]>, upper: Bound<&[u8]>) -> KvIter<'a> { self.log.record("range", Some(part)); self.inner.range(part, lower, upper) } @@ -396,50 +518,5 @@ mod recording { self.log.record("trait_keys_matching", None); self.inner.trait_keys_matching(needle, cursor_after, lower) } - - fn symbols_index_is_empty(&self) -> bool { - self.log.record("symbols_index_is_empty", None); - self.inner.symbols_index_is_empty() - } - - fn references_to(&self, def_path: &RelPath, def_start: u32) -> Vec<(RelPath, u32)> { - self.log.record("references_to", None); - self.inner.references_to(def_path, def_start) - } - - fn definition_of(&self, use_path: &RelPath, use_start: u32) -> Option<(RelPath, u32)> { - self.log.record("definition_of", None); - self.inner.definition_of(use_path, use_start) - } - - fn import_targets_of(&self, importer: &RelPath) -> Vec<(u32, RelPath, u32)> { - self.log.record("import_targets_of", None); - self.inner.import_targets_of(importer) - } - - fn importers_probing(&self, path: &str) -> Vec { - self.log.record("importers_probing", None); - self.inner.importers_probing(path) - } - - fn is_probed(&self, path: &str) -> bool { - self.log.record("is_probed", None); - self.inner.is_probed(path) - } - - fn import_probes_complete(&self) -> bool { - self.log.record("import_probes_complete", None); - self.inner.import_probes_complete() - } - - fn symbols_by_name_lookup(&self, name: &str, cap: usize) -> Vec<(String, SymbolKind, RelPath, u32)> { - self.log.record("symbols_by_name_lookup", None); - self.inner.symbols_by_name_lookup(name, cap) - } - - fn bm25_stats(&self) -> Option<(u64, u64)> { - self.log.record("bm25_stats", None); - self.inner.bm25_stats() - } } } diff --git a/src/index/view_tests.rs b/src/index/view_tests.rs index fcea9fd5..8871d046 100644 --- a/src/index/view_tests.rs +++ b/src/index/view_tests.rs @@ -1,8 +1,10 @@ //! The `IndexView` contract: the generic layer reads exactly what the keyspaces hold. The read paths //! that go through it are covered in `mcp::index_view_tests`. +use std::ops::Bound; + use super::IndexDb; -use super::view::{IndexView, Part}; +use super::view::{IndexView, KvIter, Part}; use crate::config::ConfigV1; use crate::scanner::{EmbedMode, ScanSource, scan}; use crate::store::{Store, VIEW_WORKING}; @@ -32,6 +34,27 @@ fn scanned_store(root: &std::path::Path) -> Store { store } +type Rows = Vec<(Vec, Vec)>; +type Case<'a> = (&'a str, Bound<&'a [u8]>, Bound<&'a [u8]>); + +fn rows(iter: KvIter<'_>) -> Rows { + iter.map(|r| r.expect("row")) + .map(|(k, v)| (k.to_vec(), v.to_vec())) + .collect() +} + +fn direct_rows(db: &IndexDb, part: Part) -> Rows { + db.keyspace_for_test(part) + .iter() + .map(|g| g.into_inner().expect("row")) + .map(|(k, v)| (k.to_vec(), v.to_vec())) + .collect() +} + +fn only(direct: &Rows, keep: impl Fn(&[u8]) -> bool) -> Rows { + direct.iter().filter(|(k, _)| keep(k)).cloned().collect() +} + #[test] fn the_generic_layer_reads_what_each_keyspace_holds() { let dir = tempfile::tempdir().expect("tempdir"); @@ -39,66 +62,119 @@ fn the_generic_layer_reads_what_each_keyspace_holds() { let db = store.index_db.as_ref().expect("index"); let view = db.view(); let mut non_empty = 0; + let mut bounded = 0; for part in Part::ALL { - let keyspace = db.keyspace_for_test(part); - let direct: Vec<_> = keyspace - .iter() - .map(|g| g.into_inner().expect("row")) - .map(|(k, v)| (k.to_vec(), v.to_vec())) - .collect(); - let rows = |iter: super::view::KvIter<'_>| -> Vec<(Vec, Vec)> { - iter.map(|r| r.expect("row")) - .map(|(k, v)| (k.to_vec(), v.to_vec())) - .collect() - }; - assert_eq!(rows(view.iter(part)), direct, "{} iter", part.name()); - assert_eq!(rows(view.prefix(part, b"")), direct, "{} empty prefix", part.name()); + let direct = direct_rows(db, part); + let name = part.name(); + assert_eq!(rows(view.iter(part)), direct, "{name} iter"); + assert_eq!(rows(view.prefix(part, b"")), direct, "{name} empty prefix"); assert_eq!( - rows(view.range(part, std::ops::Bound::Unbounded, std::ops::Bound::Unbounded)), + rows(view.range(part, Bound::Unbounded, Bound::Unbounded)), direct, - "{} unbounded range", - part.name() + "{name} unbounded range" + ); + assert_eq!(view.get(part, &[0xFF; 3]).expect("get"), None, "{name} miss"); + let Some((mid, value)) = direct.get(direct.len() / 2) else { + continue; + }; + non_empty += 1; + assert_eq!( + view.get(part, mid).expect("get").map(|v| v.to_vec()), + Some(value.clone()), + "{name} get" ); - if let Some((key, value)) = direct.first() { - non_empty += 1; + for len in [1, mid.len() / 2, mid.len()] { + let prefix = &mid[..len.max(1)]; assert_eq!( - view.get(part, key).expect("get").map(|v| v.to_vec()), - Some(value.clone()) + rows(view.prefix(part, prefix)), + only(&direct, |k| k.starts_with(prefix)), + "{name} prefix of {len} bytes of a mid-keyspace key" ); - assert_eq!(rows(view.prefix(part, &key[..1])).first(), direct.first()); } - assert_eq!(view.get(part, &[0xFF; 3]).expect("get"), None); + let last = &direct[direct.len() - 1].0; + let cases: [Case<'_>; 6] = [ + ("[mid, ..)", Bound::Included(mid), Bound::Unbounded), + ("(mid, ..)", Bound::Excluded(mid), Bound::Unbounded), + ("(.., mid]", Bound::Unbounded, Bound::Included(mid)), + ("(.., mid)", Bound::Unbounded, Bound::Excluded(mid)), + ("[mid, last]", Bound::Included(mid), Bound::Included(last)), + ("(mid, last)", Bound::Excluded(mid), Bound::Excluded(last)), + ]; + for (label, lower, upper) in cases { + let in_lower = |k: &[u8]| match lower { + Bound::Included(b) => k >= b, + Bound::Excluded(b) => k > b, + Bound::Unbounded => true, + }; + let in_upper = |k: &[u8]| match upper { + Bound::Included(b) => k <= b, + Bound::Excluded(b) => k < b, + Bound::Unbounded => true, + }; + let expected = only(&direct, |k| in_lower(k) && in_upper(k)); + assert_eq!(rows(view.range(part, lower, upper)), expected, "{name} range {label}"); + if direct.len() > 2 { + bounded += 1; + } + } } assert!( non_empty >= 6, "the corpus must populate the layered keyspaces, got {non_empty}" ); + assert!( + bounded >= 6 * 6, + "bounded ranges must be exercised on populated keyspaces" + ); } #[test] fn parts_name_exactly_the_layered_keyspaces() { let names: std::collections::BTreeSet<_> = Part::ALL.iter().map(|p| p.name()).collect(); assert_eq!(names.len(), Part::ALL.len(), "part names are unique"); + let unlayered = ["embeddings", "memory_by_key", "memory_archive", "proposals"]; for name in &names { assert!( super::KEYSPACE_MEMTABLE_BYTES.iter().any(|(k, _)| k == name), "{name} must be a real keyspace" ); + assert!(!unlayered.contains(name), "{name} is per-group or per-worktree state"); } - for unlayered in ["embeddings", "memory_by_key", "memory_archive", "proposals"] { + for (keyspace, _) in super::KEYSPACE_MEMTABLE_BYTES { assert!( - !names.contains(unlayered), - "{unlayered} is per-group or per-worktree state" + names.contains(keyspace) || unlayered.contains(keyspace), + "keyspace {keyspace} is neither a layered Part nor listed as unlayered" ); } } #[test] -fn store_level_reads_use_the_store_view() { +fn named_reads_follow_the_data_they_derive_from() { + let empty_dir = tempfile::tempdir().expect("tempdir"); + let empty = IndexDb::open(empty_dir.path()).expect("open"); + assert!(empty.view().symbols_index_is_empty()); + assert_eq!(empty.view().bm25_stats(), None); + assert!(!empty.view().import_probes_complete()); + let dir = tempfile::tempdir().expect("tempdir"); let store = scanned_store(dir.path()); let view = store.index_view().expect("view"); - let db: &IndexDb = store.index_db.as_ref().expect("index"); - assert_eq!(view.symbols_index_is_empty(), db.symbols_index_is_empty()); assert!(!view.symbols_index_is_empty()); + let defined = view.symbols_by_name_lookup("driver_1", 10); + assert_eq!(defined.len(), 1, "one definition of driver_1: {defined:?}"); + assert_eq!( + view.symbols_by_name_lookup("driver_1", 0), + vec![], + "cap 0 reads nothing" + ); + let all_runs = view.symbols_by_name_lookup("run", 100); + assert!( + all_runs.len() > 3, + "every trait and impl `run` is indexed: {all_runs:?}" + ); + assert_eq!( + view.symbols_by_name_lookup("run", 3), + all_runs[..3], + "the cap truncates in key order" + ); } diff --git a/src/index/writer_tests.rs b/src/index/writer_tests.rs index 2c4fd1d9..4023583f 100644 --- a/src/index/writer_tests.rs +++ b/src/index/writer_tests.rs @@ -1,5 +1,6 @@ use super::*; use crate::extract::{Call, FileMapL2, Import, SymbolKind}; +use crate::index::view::IndexView; use tempfile::TempDir; fn fresh_db() -> (TempDir, IndexDb) { @@ -25,7 +26,7 @@ fn open_reports_a_wipe_only_when_the_stamped_schema_version_differs() { let wiped = IndexDb::open_with_memory(dir.path(), &memory).expect("reopen"); assert!(wiped.wiped_on_open(), "a mismatched revision wipes the index"); - assert!(wiped.symbols_index_is_empty()); + assert!(wiped.view().symbols_index_is_empty()); drop(wiped); let again = IndexDb::open_with_memory(dir.path(), &memory).expect("third open"); @@ -485,7 +486,7 @@ fn bm25_dual_partition_consistency() { assert_eq!(db.code_bm25_by_path.iter().count(), 1); db.recompute_bm25_stats().unwrap(); - assert_eq!(db.bm25_stats(), Some((1, 1))); + assert_eq!(db.view().bm25_stats(), Some((1, 1))); let mut w = db.writer(); w.remove_bm25_file(&rel).unwrap(); @@ -550,9 +551,12 @@ fn import_probe_rows_roundtrip_and_leave_with_the_importer() { #[test] fn import_probes_are_incomplete_until_stamped() { let (_d, db) = fresh_db(); - assert!(!db.import_probes_complete(), "a fresh index has no probe rows to trust"); + assert!( + !db.view().import_probes_complete(), + "a fresh index has no probe rows to trust" + ); db.set_import_probes_complete(true).unwrap(); - assert!(db.import_probes_complete()); + assert!(db.view().import_probes_complete()); db.set_import_probes_complete(false).unwrap(); - assert!(!db.import_probes_complete()); + assert!(!db.view().import_probes_complete()); } diff --git a/src/mcp/helpers_calls_scan.rs b/src/mcp/helpers_calls_scan.rs index 10b14731..18e0cff0 100644 --- a/src/mcp/helpers_calls_scan.rs +++ b/src/mcp/helpers_calls_scan.rs @@ -143,12 +143,10 @@ pub(crate) fn calls_in_file_fjall( path: &crate::path::RelPath, ) -> Result, McpError> { let prefix = crate::index::keys::calls_by_path_prefix(path); - let upper: Bound> = match super::cursor::prefix_upper_bound(&prefix) { - Some(b) => Bound::Excluded(b), - None => Bound::Unbounded, - }; + let upper = super::cursor::prefix_upper_bound(&prefix); + let upper = upper.as_deref().map_or(Bound::Unbounded, Bound::Excluded); let mut out = Vec::new(); - for guard in idx.range(Part::CallsByPath, Bound::Included(prefix), upper) { + for guard in idx.range(Part::CallsByPath, Bound::Included(&prefix), upper) { let (_, v) = guard.map_err(|e| McpError::internal_error(format!("index iter: {e}"), None))?; let call: crate::extract::Call = match rmp_serde::from_slice(&v) { Ok(c) => c, diff --git a/src/mcp/helpers_grep.rs b/src/mcp/helpers_grep.rs index 5f676fd2..ee27014e 100644 --- a/src/mcp/helpers_grep.rs +++ b/src/mcp/helpers_grep.rs @@ -429,6 +429,7 @@ fn extract_line(source: &str, line_starts: &[usize], line_idx: usize) -> String #[cfg(test)] mod tests { use super::*; + use crate::index::view::IndexView; #[test] fn packs_and_unpacks_a_file_and_hit_ordinal() { @@ -795,7 +796,7 @@ mod tests { let re = Regex::new(pattern).expect("regex"); let literal = (regex::escape(pattern) == pattern).then(|| Finder::new(pattern.as_bytes()).into_owned()); let skip = crate::index::grep_bloom::Needle::compile(pattern) - .map(|needle| db.grep_skip_verdicts(dir.path(), &entries, &needle)); + .map(|needle| db.view().grep_skip_verdicts(dir.path(), &entries, &needle)); skipped_somewhere |= skip.as_ref().is_some_and(|s| s.iter().any(|&x| x)); for skip_hits in [0u32, 2] { let filtered = count_all(dir.path(), &refs, skip.as_deref(), &re, literal.as_ref(), skip_hits); @@ -823,7 +824,7 @@ mod tests { // The stale and new files must stay candidates for the token they now hold. let needle = crate::index::grep_bloom::Needle::compile("unique_token_qq").expect("needle"); - let skip = db.grep_skip_verdicts(dir.path(), &entries, &needle); + let skip = db.view().grep_skip_verdicts(dir.path(), &entries, &needle); for (rel, skipped) in paths.iter().zip(&skip) { if ["d3/f3.txt", "d4/f4.txt", "d0/new.txt", "d2/unique.txt"].contains(&rel.to_string().as_str()) { assert!(!skipped, "{rel} holds the token and must not be skipped"); @@ -851,7 +852,8 @@ mod tests { }; let needle = crate::index::grep_bloom::Needle::compile("absent_literal").expect("needle"); assert_eq!( - db.grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), + db.view() + .grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), vec![false], "no row" ); @@ -859,13 +861,15 @@ mod tests { writer.upsert_grep_bloom(&entry.hash_hex, vec![99; 40]); writer.commit().expect("commit"); assert_eq!( - db.grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), + db.view() + .grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), vec![false], "foreign row" ); scan_blooms(&db, dir.path(), std::slice::from_ref(&rel)); assert_eq!( - db.grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), + db.view() + .grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), vec![true], "current row" ); @@ -873,7 +877,7 @@ mod tests { writer.remove_file(&rel).expect("remove"); writer.commit().expect("commit"); assert!( - db.grep_bloom_row(&entry.hash_hex).is_some(), + db.view().grep_bloom_row(&entry.hash_hex).is_some(), "the row is shared by content, so removing one path keeps it" ); } diff --git a/src/mcp/index_view_tests.rs b/src/mcp/index_view_tests.rs index a489638c..249d41dc 100644 --- a/src/mcp/index_view_tests.rs +++ b/src/mcp/index_view_tests.rs @@ -43,6 +43,25 @@ fn recording(store: &Store) -> (RecordingView, ReadLog) { RecordingView::new(store.index_db.clone().expect("index")) } +/// One `callee_keys_matching` page plus the `calls_by_path` point read that resolves each of the +/// `2 * FILES` call sites to a line and column. +fn name_scan_reads() -> Vec { + let mut reads = vec!["callee_keys_matching".to_owned()]; + reads.extend(std::iter::repeat_n("get:calls_by_path".to_owned(), 2 * FILES)); + reads +} + +/// One bloom-row point read per entry; the prefilter fetches them in parallel, so only the multiset +/// is stable. +fn bloom_reads() -> Vec { + vec!["get:grep_bloom".to_owned(); FILES] +} + +fn sorted(mut reads: Vec) -> Vec { + reads.sort(); + reads +} + fn grep_entries(store: &Store) -> Vec { (0..FILES) .map(|k| { @@ -69,18 +88,17 @@ fn call_and_implementation_scans_read_through_the_view() { let routed = scan_calls_by_name(&view, "shared_helper", 50, None).expect("routed scan"); assert_eq!(routed.total, 2 * FILES as u32, "two call sites per file"); assert_eq!((routed.total, &routed.hit_keys), (direct.total, &direct.hit_keys)); - assert!(log.saw("callee_keys_matching"), "name scan: {:?}", log.entries()); - assert!(log.saw("get:calls_by_path"), "line/column resolve: {:?}", log.entries()); + assert_eq!(log.entries(), name_scan_reads(), "name scan"); log.clear(); let calls = calls_in_file_fjall(&view, &RelPath::from("m0.rs")).expect("calls"); assert!(!calls.is_empty()); - assert!(log.saw("range:calls_by_path"), "per-file calls: {:?}", log.entries()); + assert_eq!(log.entries(), ["range:calls_by_path"], "per-file calls"); log.clear(); let impls = scan_impls_fjall(&view, "Tr", None, 50, None, |_, _| true, |_, _| (0, 0)).expect("impls"); assert_eq!(impls.total, FILES); - assert!(log.saw("trait_keys_matching"), "implementations: {:?}", log.entries()); + assert_eq!(log.entries(), ["trait_keys_matching"], "implementations"); } #[test] @@ -91,14 +109,34 @@ fn probe_and_grep_prefilter_reads_go_through_the_view() { let probed = crate::intel::resolve_pass::probed_only(&view, vec!["nowhere.rs".to_owned()]); assert!(probed.is_empty()); - assert!(log.saw("is_probed"), "watcher probe filter: {:?}", log.entries()); + assert_eq!( + log.entries(), + ["prefix:importers_by_probe"], + "a path with no ancestor directory costs one probe-prefix scan" + ); + + log.clear(); + assert!(view.importers_probing("a/b/c.ts").is_empty()); + assert_eq!( + log.entries(), + ["prefix:importers_by_probe"; 3], + "the path, then each ancestor directory" + ); + + log.clear(); + assert!(!view.import_probes_complete()); + assert_eq!(log.entries(), ["get:meta"], "probe stamp"); + + log.clear(); + assert!(!view.symbols_index_is_empty()); + assert_eq!(log.entries(), ["iter:symbols_by_path"], "emptiness peeks the first key"); log.clear(); let entries = grep_entries(&store); let needle = Needle::compile("zzz_definitely_absent").expect("needle"); let verdicts = view.grep_skip_verdicts(dir.path(), &entries, &needle); assert_eq!(verdicts.len(), entries.len()); - assert!(log.saw("get:grep_bloom"), "grep prefilter: {:?}", log.entries()); + assert_eq!(sorted(log.entries()), bloom_reads(), "grep prefilter"); } #[cfg(feature = "code-search")] @@ -110,16 +148,15 @@ fn both_search_lanes_read_through_the_view() { let exact = crate::search::exact::exact_lane_chunk_ids(&store, &view, "driver_1", 10); assert!(!exact.is_empty(), "the exact lane resolves a defined function"); - assert!(log.saw("symbols_by_name_lookup"), "exact lane: {:?}", log.entries()); + assert_eq!(log.entries(), ["prefix:symbols_by_name"], "exact lane"); log.clear(); let keyword = crate::search::bm25::bm25_search(&view, "driver", 10); assert!(!keyword.is_empty(), "the keyword lane finds the chunk"); - assert!(log.saw("bm25_stats"), "keyword lane stats: {:?}", log.entries()); - assert!( - log.saw("prefix:code_bm25_postings"), - "keyword lane postings: {:?}", - log.entries() + assert_eq!( + log.entries(), + ["get:meta", "get:meta", "prefix:code_bm25_postings"], + "keyword lane: both corpus stats, then one posting scan for the one query term" ); } @@ -133,22 +170,18 @@ async fn the_local_route_answers_every_read_from_its_view() { let page = route.scan_calls(&cache, "shared_helper", 50, None).await.expect("scan"); assert_eq!(page.total, 2 * FILES as u32); - assert!( - log.saw("callee_keys_matching"), - "references/callers: {:?}", - log.entries() - ); + assert_eq!(log.entries(), name_scan_reads(), "references/callers"); log.clear(); let paths = vec![RelPath::from("m1.rs")]; let per_file = route.calls_in_files(&cache, &paths).await.expect("calls"); assert!(!per_file[0].is_empty()); - assert!(log.saw("range:calls_by_path"), "resolved callers: {:?}", log.entries()); + assert_eq!(log.entries(), ["range:calls_by_path"], "resolved callers"); log.clear(); let impls = route.scan_impls(&cache, "Tr", None, 50, None).await.expect("impls"); assert_eq!(impls.total, FILES); - assert!(log.saw("trait_keys_matching"), "implementations: {:?}", log.entries()); + assert_eq!(log.entries(), ["trait_keys_matching"], "implementations"); log.clear(); let entries = grep_entries(&store); @@ -157,5 +190,5 @@ async fn the_local_route_answers_every_read_from_its_view() { .await .expect("verdicts"); assert_eq!(verdicts.len(), entries.len()); - assert!(log.saw("get:grep_bloom"), "grep prefilter: {:?}", log.entries()); + assert_eq!(sorted(log.entries()), bloom_reads(), "grep prefilter"); } diff --git a/src/query.rs b/src/query.rs index aad0ad5c..195d5d58 100644 --- a/src/query.rs +++ b/src/query.rs @@ -184,6 +184,7 @@ fn definition_of_from_blob(store: &Store, use_path: &RelPath, use_start: u32) -> mod tests { use super::*; use crate::config::ConfigV1; + use crate::index::view::IndexView; use crate::scanner::{ScanSource, scan}; use crate::store::{Store, VIEW_WORKING}; @@ -228,6 +229,7 @@ mod tests { .index_db .as_ref() .expect("index open") + .view() .references_to(&path, def_start); assert!(!via_blob.is_empty(), "blob fallback must find the intra uses"); assert_eq!( diff --git a/src/scanner_code.rs b/src/scanner_code.rs index e1c8e670..56f98b36 100644 --- a/src/scanner_code.rs +++ b/src/scanner_code.rs @@ -543,6 +543,7 @@ pub(crate) fn flush_code_batches( mod tests { use super::*; use crate::chunk::CodeChunk; + use crate::index::view::IndexView; fn chunk(chunk_id: &str, searchable_text: &str) -> CodeChunk { CodeChunk { @@ -790,7 +791,7 @@ mod tests { .unwrap(); writer.commit().unwrap(); db.recompute_bm25_stats().unwrap(); - assert_eq!(db.bm25_stats().map(|(n, _)| n), Some(1)); + assert_eq!(db.view().bm25_stats().map(|(n, _)| n), Some(1)); store.upsert( "src/a.rs", crate::store::FileEntry { @@ -808,7 +809,7 @@ mod tests { purge_unembedded_code(&mut store, &config, &filters, "repo:s"); let db = store.index_db.as_ref().unwrap(); assert_eq!( - db.bm25_stats().map(|(n, _)| n).unwrap_or(0), + db.view().bm25_stats().map(|(n, _)| n).unwrap_or(0), 0, "keyword postings are gone" ); diff --git a/tests/config_scan_rules.rs b/tests/config_scan_rules.rs index f074ea0b..ec6d63fe 100644 --- a/tests/config_scan_rules.rs +++ b/tests/config_scan_rules.rs @@ -256,13 +256,15 @@ fn embed_policy_is_recorded_and_tracks_config_changes() { #[cfg(feature = "code-search")] mod code_chunks { + use basemind::index::view::IndexView; + use super::*; fn bm25_docs(store: &Store) -> u64 { store .index_db .as_ref() - .and_then(|db| db.bm25_stats()) + .and_then(|db| db.view().bm25_stats()) .map_or(0, |(n, _)| n) } diff --git a/tests/git_smoke.rs b/tests/git_smoke.rs index b52b07ad..df71a5d8 100644 --- a/tests/git_smoke.rs +++ b/tests/git_smoke.rs @@ -12,6 +12,7 @@ use std::process::Command; use basemind::config::ConfigV1; use basemind::git::Repo; +use basemind::index::view::IndexView; use basemind::scanner::{ScanSource, scan}; use basemind::store::{Store, VIEW_STAGED, VIEW_WORKING}; use tempfile::TempDir; @@ -401,6 +402,7 @@ fn staged_and_rev_scans_do_not_sweep_grep_bloom_rows() { .index_db .as_ref() .expect("index") + .view() .grep_bloom_row("orphan") .is_some() }; diff --git a/tests/grep_bloom_scan.rs b/tests/grep_bloom_scan.rs index b47859a5..f7771754 100644 --- a/tests/grep_bloom_scan.rs +++ b/tests/grep_bloom_scan.rs @@ -6,6 +6,7 @@ use std::path::Path; use std::time::{Duration, SystemTime}; +use basemind::index::view::IndexView; use basemind::path::RelPath; use basemind::scanner::{EmbedMode, ScanCancel, ScanSource}; use basemind::store::{Store, VIEW_WORKING}; @@ -81,7 +82,7 @@ fn has_row(store: &Store, hash_hex: &str) -> bool { store .index_db .as_ref() - .is_some_and(|idx| idx.grep_bloom_row(hash_hex).is_some()) + .is_some_and(|idx| idx.view().grep_bloom_row(hash_hex).is_some()) } fn row_count(store: &Store) -> usize { @@ -213,7 +214,7 @@ fn raw_row(store: &Store, key: &str) -> Option> { store .index_db .as_ref() - .and_then(|idx| idx.grep_bloom_row(key)) + .and_then(|idx| idx.view().grep_bloom_row(key)) .map(|row| row.to_vec()) } @@ -254,7 +255,10 @@ fn v1_rows_are_absent_on_read_so_grep_scans_every_file() { let absent = basemind::index::grep_bloom::Needle::compile("zzz_absent_literal").expect("needle"); let present = basemind::index::grep_bloom::Needle::compile("alpha_function_name").expect("needle"); let idx = store.index_db.clone().expect("index"); - let verdicts = |needle| idx.grep_skip_verdicts(root, std::slice::from_ref(&entry), needle); + let verdicts = |needle| { + idx.view() + .grep_skip_verdicts(root, std::slice::from_ref(&entry), needle) + }; assert_eq!( verdicts(&absent), vec![true], @@ -289,7 +293,7 @@ fn the_next_incremental_scan_sweeps_v1_rows_and_rebuilds_content_keyed_ones() { let entry = grep_entry(&store, root, "a.rs"); let absent = basemind::index::grep_bloom::Needle::compile("zzz_absent_literal").expect("needle"); let idx = store.index_db.clone().expect("index"); - assert_eq!(idx.grep_skip_verdicts(root, &[entry], &absent), vec![true]); + assert_eq!(idx.view().grep_skip_verdicts(root, &[entry], &absent), vec![true]); } #[test] @@ -409,7 +413,7 @@ fn a_freshly_written_file_is_never_skipped_until_a_later_scan_restamps_it() { let verdict = |store: &Store| { let entry = grep_entry(store, root, "a.rs"); let idx = store.index_db.clone().expect("index"); - idx.grep_skip_verdicts(root, &[entry], &absent)[0] + idx.view().grep_skip_verdicts(root, &[entry], &absent)[0] }; // The first scan paid for the grammar bootstrap; an incremental pass on the open store is From 27e410a5aaee73ef69c3efd7b0d34a37d94f5a8f Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Mon, 19 Oct 2026 17:31:29 +0200 Subject: [PATCH 08/10] docs: point the layered-index plan at the ADR-0016 specification where they diverge Co-Authored-By: Claude Opus 5.5 --- docs/layered-index-design.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/layered-index-design.md b/docs/layered-index-design.md index 2e5aa478..edb7edc1 100644 --- a/docs/layered-index-design.md +++ b/docs/layered-index-design.md @@ -1,6 +1,6 @@ # Layered index: design and implementation plan -- **Status:** Design accepted, not implemented +- **Status:** A2 and A3 implemented; A4–A8 not started - **Date:** 2026-10-18 - **Decision record:** [ADR-0016](adr/0016-shared-base-index-and-worktree-overlays.md) - **Tracking issue:** #87 @@ -11,6 +11,15 @@ this file records *how* — the on-disk layout, the data model, the exact read-m the write path, the correctness gate, and the file-by-file phase breakdown. Symbol names and paths are as of 0.30.0. +> **Superseded in part.** The ADR-0016 *Specification* section is authoritative wherever it and this +> file disagree. It corrects this plan against the code and records the owner decisions taken since: +> A2 migrates by `ROW_VERSION` 2 with **no** `INDEX_PARTITION_REVISION` bump; `IndexView` returns +> boxed iterators over a backend-neutral `IndexBytes`, not visitors over `Vec`; the base is read +> through a pluggable `BaseReader` (Fjall or an immutable sorted-run format, chosen by a prototype +> before A4); the base is built by scanning an exported tree as a working tree (a `Rev` scan has no +> resolve pass); and `code_chunks` is keyed `(scope, path)`, so A7 must re-key it too. The sections +> below that these touch carry a pointer to the ADR. + ## 1. Problem Each git worktree is its own workspace: the store key is blake3 of the canonical worktree root @@ -189,6 +198,9 @@ it holds. This keeps `upsert_file`'s read-before-write and per-file commit seman ## 8. `IndexView` trait (A3) +> Implemented differently; see ADR-0016 §A3 and `src/index/view.rs`. The sketch below is the +> original proposal, kept for the rationale. + The read path today takes a concrete `&IndexDb` (or a cloned handle) and calls Fjall directly. A3 introduces a trait so the layered implementation can slot in with **no behaviour change**: @@ -251,7 +263,7 @@ Each phase is its own commit, lands behind `[index] layered = false`, and keeps the live-file guard uses the **indexed** stamp from `FileEntry` instead of the row's own. - `src/index/writer.rs`: `upsert_grep_bloom(hash, row)`; `remove_file` stops deleting the now-shared row. - `src/index/mod.rs`: `sweep_grep_bloom(&live_hashes)` (drop rows no live file references), run in the - full-scan purge; bump `INDEX_PARTITION_REVISION` 6 → 7. + full-scan purge. Migration is by `ROW_VERSION` 2 with no partition-revision bump (ADR-0016 §A2). - `src/scanner_index_batch.rs`: stage by hash; `restamp_grep_bloom` / `backfill_grep_bloom` key on hash. - `src/mcp/l1_cache.rs`: add `mtime_ns` to `FileMeta`. - `src/mcp/helpers_grep.rs`, `src/mcp/index_route.rs`, `src/mcp/index_read.rs`, @@ -268,7 +280,8 @@ Each phase is its own commit, lands behind `[index] layered = false`, and keeps - `src/store_layout.rs`: `repos//base/gen-/` and `repos//overlays//` path helpers + `base.json` / `overlay.json` sidecars. -- Base builder: `ScanSource::Rev` over `origin/HEAD` into `gen-0/`, published read-only. +- Base builder: an exported tree of `origin/HEAD` scanned as a working tree into `gen-0/`, published + read-only, in the storage format the A4 prototype selects (ADR-0016 §A4). - Migration: a workspace whose only index is the legacy standalone becomes base + overlay on first layered open; `store_seed.rs` stays for now. @@ -288,7 +301,7 @@ Each phase is its own commit, lands behind `[index] layered = false`, and keeps ### A7 — shared Lance tables - `documents` / `doc_links` re-keyed to `(scope, content_hash)`; queries over-fetch and filter by live - hashes. `code_chunks` is already content-addressed. + hashes. `code_chunks` is keyed `(scope, path)` and needs the same re-key (ADR-0016 §A7). - Documented deviation: top-k overlap with a per-worktree table stays ≥0.95. ### A8 — layered by default From b2c87c15dbe4f6e79aa9cdb69b24b9c818457a18 Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Mon, 19 Oct 2026 21:27:20 +0200 Subject: [PATCH 09/10] docs(adr): size the layered index defaults from measured armis numbers Co-Authored-By: Claude Opus 5.5 --- ...shared-base-index-and-worktree-overlays.md | 24 ++++++++++++------- docs/layered-index-design.md | 2 +- 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/docs/adr/0016-shared-base-index-and-worktree-overlays.md b/docs/adr/0016-shared-base-index-and-worktree-overlays.md index 85551386..04dadb80 100644 --- a/docs/adr/0016-shared-base-index-and-worktree-overlays.md +++ b/docs/adr/0016-shared-base-index-and-worktree-overlays.md @@ -150,7 +150,7 @@ Shipped in `26be3d56`; the follow-ups below are required before A4 and are being **Config.** New `[index]` table in `config/v1.rs` (`IndexConfig`, `deny_unknown_fields`, in `ConfigV1` after `cache`): `layered: bool`, default `false` until A8. Regenerate `schema/` with the existing `config_schema` test. The env var `BASEMIND_INDEX_LAYERED=0|1` overrides the file (rollback lever, same pattern as -`BASEMIND_NO_SEED`). `max_group_bytes: u64` caps a group's `base/` plus `overlays/` (*Disk budget*). The mode is resolved once per `Store::open`; the daemon applies a changed value when it next +`BASEMIND_NO_SEED`). `max_group_bytes: Option` caps a group's `base/` plus `overlays/` (*Disk budget*); unset means *auto*. The mode is resolved once per `Store::open`; the daemon applies a changed value when it next opens the workspace. Layered applies to a workspace only when all hold: `layered` is true, the canonical root is the git workdir, `Repo::discover` succeeds, a base commit resolves, and no embedding lane is enabled until A7 has shipped (`documents.embed`, `code_search.embed`: their Lance rows are still keyed by worktree). `documents.embed` @@ -267,10 +267,13 @@ not a rescan, and the first base costs one CoW clone instead of 17-33 min. 1. *(Fjall option)* Quiesce `k`: if this process is the daemon it takes the write side of the group's `RwLock>`, drains, and drops its handle; otherwise it tries to open `k`'s fjall and aborts the advance on `Locked`. A live fjall is never copied (the rule in the `store_seed.rs` module doc). Readers of the group stall - while the write side is held. The stall is bounded by a budget, `[index] advance_stall_ms` (default 10000), covering + while the write side is held. The stall is bounded by a budget, `[index] advance_stall_ms` (default 2000), covering the drain of in-flight queries, the CoW clone (O(files)) and the reopen of `k` (open cost, which includes journal replay and the `rotate_memtable_and_wait` flush in `open_inner`, see *Lifecycle*). Exceeding it aborts the - advance, reopens `k`, and retries after a backoff; readers are never stalled past the budget. A sorted-run base needs + advance, reopens `k`, and retries after a backoff; readers are never stalled past the budget. The default is sized + from armis: a CoW clone of its 1.7 GB, 102-file index takes 0.36 s on APFS, so 2 s covers drain, clone and reopen + with headroom while staying under an interactive tool-call timeout; an advance is background work, so an abort + costs only a retry. A sorted-run base needs no quiesce: its files are immutable, so the clone is a hardlink or CoW copy and no reader is touched. 2. Clone `gen-k` to `.tmp-*` and, for Fjall, reopen the daemon handle immediately. 3. `Repo::diff_trees(C_k, C_m)` (new, gix tree diff; a rename is a delete plus an add) filtered through `Filters`. @@ -321,12 +324,16 @@ clone as a base source, and a standalone index with an old stamp does a one-time content-addressed chunk blobs on its next scan (the `by_path` rows hold the term list without frequencies, so they cannot rebuild the postings). Only then is it eligible as a clone source. -**Disk budget.** A group holds up to 3 live generations, about 2.7 GB each, and compaction un-shares the CoW clones, -so the realistic footprint is the sum, not one generation plus deltas. `[index] max_group_bytes` (default 12 GiB, to be -measured) caps `base/` plus `overlays/`. Build, advance and migrate each check, under `base.lock`, that free space on +**Disk budget.** A group holds up to 3 live generations (1.7 GB each on armis today), and compaction un-shares the CoW clones, +so the realistic footprint is the sum, not one generation plus deltas. `[index] max_group_bytes` caps `base/` plus +`overlays/`. Its default, *auto*, is 4x the newest ready generation's size: two live generations, the transient export +plus clone of a build, and the overlays (each holds changed paths only). A fixed byte default would be wrong at both +ends: too small for a repo whose single generation exceeds it, and far above need for a small repo. On armis (1.7 GB +generation) auto is about 7 GB, against 33 x 1.7 GB of standalone indexes today. Retention is therefore 2 generations +by default (newest plus the one pinned overlays are rebasing off), 3 only when the cap allows. Build, advance and migrate each check, under `base.lock`, that free space on the volume is at least the generation size plus the cap on the transient (export plus clone) and that the group stays under the cap, else they are skipped, a `last-failure.json`-style marker is written, and `doctor` says why. When disk is -tight the newest generation replaces the oldest instead of the group keeping 3 (the oldest goes through the *Liveness* +tight the newest generation replaces the oldest instead of the group keeping 2 (the oldest goes through the *Liveness* rules, so a pinned one is never deleted from under a reader). ### A5: overlay write path and the read merge @@ -442,7 +449,8 @@ row for the same shadow unit. Tombstones, snapshots and scalars are specified he AND the overlay's own flag (cleared while the overlay's resolve pass is incomplete). - **Visited-row budget.** `scan_cap` caps hits, not rows examined (`mcp/helpers_calls_scan.rs`). A filtered scan whose mask drops many base rows (a widely edited name) can examine far more rows than it yields, so the merged iterator - carries a visited-row budget (default `scan_cap * 64`); when it is hit the scan stops, returns what it has and sets the + carries a visited-row budget (default `scan_cap * 64`, at least 128k rows given the `scan_cap` floor of 2000, about + 0.1 s of fjall prefix iteration, so a heavily shadowed name degrades to partial rather than stalling); when it is hit the scan stops, returns what it has and sets the existing partial flag (`total_is_partial`). The equivalence gate runs with the budget unbounded and a separate test proves the partial flag. diff --git a/docs/layered-index-design.md b/docs/layered-index-design.md index edb7edc1..9023b47a 100644 --- a/docs/layered-index-design.md +++ b/docs/layered-index-design.md @@ -295,7 +295,7 @@ Each phase is its own commit, lands behind `[index] layered = false`, and keeps ### A6 — generations, rebase, promotion, GC -- Generation advance (CoW clone + `diff-tree`), publication-by-rename, three-generation retention, +- Generation advance (CoW clone + `diff-tree`), publication-by-rename, two-generation retention (three when `max_group_bytes` allows; ADR-0016 *Disk budget*), overlay GC wired into `store_gc`, orphan reap. ### A7 — shared Lance tables From 9928a10db4109ebedf4fb2402a72dd9dff5920a3 Mon Sep 17 00:00:00 2001 From: Na'aman Hirschfeld Date: Mon, 19 Oct 2026 21:27:20 +0200 Subject: [PATCH 10/10] test(daemon): run the memory forwarding step only when the memory feature is built Co-Authored-By: Claude Opus 5.5 --- tests/cli_daemon.rs | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/tests/cli_daemon.rs b/tests/cli_daemon.rs index 61b2f216..70ba2afe 100644 --- a/tests/cli_daemon.rs +++ b/tests/cli_daemon.rs @@ -217,11 +217,14 @@ fn live_daemon_serves_rescan_memory_and_blocks_destructive_commands() { assert_eq!(code(&rescan), 0, "{}", text(&rescan)); assert!(text(&rescan).contains("via daemon"), "{}", text(&rescan)); - let put = env.run(&["memory", "put", "k1", "forwarded value"]); - assert_eq!(code(&put), 0, "memory put must forward, not fail: {}", text(&put)); - let get = env.run(&["memory", "get", "k1"]); - assert_eq!(code(&get), 0, "{}", text(&get)); - assert!(text(&get).contains("forwarded value"), "{}", text(&get)); + // `memory` mode exists only in a binary built with the `memory` feature. + if cfg!(feature = "memory") { + let put = env.run(&["memory", "put", "k1", "forwarded value"]); + assert_eq!(code(&put), 0, "memory put must forward, not fail: {}", text(&put)); + let get = env.run(&["memory", "get", "k1"]); + assert_eq!(code(&get), 0, "{}", text(&get)); + assert!(text(&get).contains("forwarded value"), "{}", text(&get)); + } let callers = env.run(&["--json", "code", "callers", "a.rs", "alpha"]); assert_eq!(code(&callers), 0, "{}", text(&callers));