diff --git a/docs/adr/0016-shared-base-index-and-worktree-overlays.md b/docs/adr/0016-shared-base-index-and-worktree-overlays.md index 55953d12..04dadb80 100644 --- a/docs/adr/0016-shared-base-index-and-worktree-overlays.md +++ b/docs/adr/0016-shared-base-index-and-worktree-overlays.md @@ -20,11 +20,16 @@ scan cost should be proportional to the worktree's diff, with query results iden Group worktrees by blake3 of the canonical git common dir and split the index into an immutable **base** and a per-worktree **overlay**: -- **Base** — an index of `origin/HEAD` (falling back to the main worktree HEAD) built from git objects - through the existing rev scan, so it never contains uncommitted edits. Generations advance by a CoW - clone plus `git diff-tree C_n..C_n+1` applied through the incremental writer when files have drifted by - 2% or more, or daily; at most three exist, each published read-only so the daemon keeps one handle and - one block cache per group. +- **Base** — an index of `origin/HEAD` (falling back to the main worktree HEAD) built from the committed + tree, so it never contains uncommitted edits. The tree is exported to a scratch directory and scanned + like a working tree, because the rev scan alone does not run the resolve pass. Generations advance by + a CoW clone plus `git diff-tree C_n..C_n+1` applied through the incremental writer when files have + drifted by 2% or more, or daily; at most three are live, each published once and never logically + rewritten (a Fjall generation still replays its journal and compacts when opened, see *Lifecycle*). + The base is read through a `BaseReader`, so its storage is pluggable (*Base storage*). With a Fjall + base the daemon keeps one handle and one block cache per group (fjall admits one process per + directory, so other processes reach the base through the daemon); an immutable mmapped base can be + opened by every process. - **Overlay** — `delta.msgpack` records each path as added, modified or deleted relative to the base manifest by content hash; `overlay.fjall` holds rows only for changed paths and re-stitched importers. Shadow sets `S_path` (delta paths), `S_refs` (`S_path` plus re-stitched importers) and `D_hash` (base @@ -44,6 +49,614 @@ and a per-worktree **overlay**: `documents`/`doc_links` are re-keyed to `(scope, content_hash)`, with queries over-fetching and filtering by live hashes. Documented deviation: top-k overlap with a per-worktree table stays ≥0.95. +## Specification + +Everything below is normative for A2–A8. Paths and symbols are as of 0.30.0 (`bd1f0a5b`), plus A2 (`efe41b76`) +and A3 (`26be3d56`) on this branch. Where the code contradicts the text of issue #87 or the summary above, the code wins and the correction is listed first. + +### Corrections to the issue text + +| Claim in #87 | What the code does | Consequence | +|---|---|---| +| "Rev scan builds the base" (`ScanSource::Rev`) | `scan_with_cancel` runs the resolve pass only for `ScanSource::WorkingTree` (`scanner.rs`, `if matches!(source, ScanSource::WorkingTree)`), and the resolver reads the filesystem (`intel/resolve_pass.rs` `std::fs::read`, `intel/resolver.rs` `root.join`, `canonicalize`). A rev scan therefore writes no `refs_by_*`, `import_targets` or `importers_by_probe` rows, and the rev branch of `process_file` writes no grep bloom either. | The base builder exports the committed tree to a private scratch directory and runs a normal working-tree scan on it (A4). A virtual git-tree filesystem for the resolver was rejected: it duplicates the resolver's I/O surface for a transient 1-3 GB export. | +| "Published read-only so the daemon keeps one handle" | Fjall's directory lock is exclusive even for a read-only open (`Store::open_read_only_no_index`, `mcp/index_route.rs` module doc). Read-only is a logical property, not something the OS enforces on a shared fjall. | For a Fjall base, one process per machine can hold a given generation. The daemon is that holder; every other process reaches the base through `IndexRoute::Host` / `Daemon` or degrades to `InRam`, exactly as a second `serve` does today (A4). Conditional on the base-storage prototype: an mmapped sorted-run base has no such lock (*Base storage*). | +| "`code_chunks` is already content-addressed" | `chunk_id` is `:` (`chunk.rs`) and the sidecar blobs are content-addressed, but the Lance rows are keyed `(scope, path)` (`lance/mod.rs` `replace_code_chunks`, `delete_code_chunks`) and `scope` is the normalised `origin` URL (`git/remote.rs` `scope_key`), identical for every worktree of a clone. | Sharing a Lance table needs a new key for all three tables, not just `documents`/`doc_links` (A7). | +| A3 "shipped" (progress comment) | When the ADR was written only the ADR had shipped: `src/index/view.rs` did not exist and `IndexDb` keyspaces were read directly. A2 landed in `efe41b76` and A3 in `26be3d56`. | A3 is a prerequisite of A5 and is done, but `view.rs` still returns `fjall::Slice` and its named accessors are required methods (A3 follow-up below). | +| `repo_group_key` hashes "the git common dir" | It hashes the canonical main worktree root, the parent of the common dir (`store_layout.rs` `repo_group_key`, `git/mod.rs` `main_worktree_root`). Equivalent for a normal clone; a bare clone or a moved main checkout re-keys the group. | Keep the function; record the resolved common dir in `group.json` so a re-key is diagnosable (A4). | +| BM25 `meta` is exact as "base + overlay - shadowed" | True, but `upsert_bm25_file` / `remove_bm25_file` delete every posting of the path's `chunk_id`s (`index/writer.rs` `stage_bm25_deletes_for`), and `chunk_id` embeds the content hash. Two paths with identical bytes share their postings, so removing or editing one of them deletes the postings the other still needs. | Pre-existing defect in the standalone index, not specific to layering, and it breaks the A5 equivalence gate on the "duplicate content" scenario. A5.0 fixes it first (below). | +| `main_root_of` / `repo_group_key` cover a workspace | A workspace rooted at a *subdirectory* of a repository also gets a group key (`Repo::discover` walks up), but a base built from the whole tree would not match a subdirectory scan. | Layering applies only when the canonical root equals the git workdir (the `store_seed::main_root_of` predicate); otherwise the workspace stays standalone (A4). | + +### Vocabulary + +- **group**: `repo_group_key(root)`; its directory is `repo_group_dir(root)` = `cache/repos//` (`store_layout.rs`). +- **wt_key**: `workspace_key(root)`; its directory is `workspace_cache_dir(root)` (unchanged, still holds `.lock`, + `workspace.json`, `status.json`, git caches, and the `staged` / `rev-*` views, which stay standalone). +- **W**: the paths a working-tree scan would index (`scanner_candidates::walk_candidates` after `Filters`). +- **T**: the keys of the base manifest (`Index.files` of the pinned generation), i.e. the filtered paths of `tree(C_base)`. +- **H_base**: the set of content hashes in the base manifest, with a per-hash path count. +- **Shadow sets**: `S_path`, `S_refs`, `D_hash` as defined in the Decision. + +### A2: re-key `grep_bloom` by content hash + +The plan in #87 stands; this fixes the details the plan left open. + +- **Key and row.** Key = the 64-character lowercase hex blake3 (`hashing::hex`), so two paths with identical + bytes share one row. Row = `version(1) | bits`; `ROW_VERSION` goes 1 to 2, so every old row (path-keyed, + 17-byte header) parses as absent and reads as a candidate. `build_row(bytes)`; `GrepEntry { rel, hash_hex, size_bytes, mtime_ns }` + (`index/grep_bloom.rs`, serde, because it crosses the daemon wire). +- **Live-file guard.** `grep_skip_verdicts(root, &[GrepEntry], needle)` skips a file only if the row for + `entry.hash_hex` rejects the needle *and* `stat(root/rel)` equals the **indexed** `(size_bytes, mtime_ns)` of + the entry, taken from `FileEntry` (`store.rs`) through `FileMeta` (`mcp/l1_cache.rs`, gains `mtime_ns`). A file + whose indexed mtime is 0 (git sources) never equals a live stat and stays a candidate: `mtime_ns == 0` is never skipped. + A file whose mtime is within 2 s of the scan's stat time is untrusted for skipping (the same-tick edit hole: an edit + inside the filesystem's timestamp granularity can leave `(size, mtime)` unchanged), so it stays a candidate until a + later scan restamps it. +- **Stamp refresh (not in the plan).** The stamp used to live in the row and `restamp_grep_bloom` refreshed it + when a file was touched with unchanged content. The stamp now lives in the `FileEntry`, and + `process_file` returned `Unchanged` without upserting one, so a touched file would have stayed a permanent + grep candidate. The unchanged-content branch of `scanner_file::process_file` therefore returns a + `FileResult` carrying a refreshed `FileEntry` (same hash, new `size_bytes` / `mtime`) for working-tree scans. +- **Writer.** `IndexWriter::upsert_grep_bloom(hash_hex, row)` overwrites an identical row; `remove_file` no longer + touches `grep_bloom`. `WorkerIndexBatch::{stage,restamp,backfill}_grep_bloom` key on the hash; backfill reads + the file only when the hash has no row and drops the bytes if they no longer hash to the indexed value. +- **Sweep.** `IndexDb::sweep_grep_bloom(&live)` deletes every row whose hash is not in `live`. `scanner::scan_with_cancel` + calls it right after the stale-entry purge with `live` = hashes of `store.index.files`, so a pass that was + cancelled or aborted (which returns before the purge) never sweeps. Edited and deleted contents leave their + row behind until that sweep, which is the only deletion path for a full scan. The sweep is key-only (it never decodes a + row) and runs only for `WorkingTree` scans, because a rev scan writes no bloom rows and would sweep them all. The + watcher-driven `scan_paths` also deletes a content row when its hash leaves the manifest (no other path holds it), so + incremental edits do not wait for the next full pass. +- **Schema.** `INDEX_PARTITION_REVISION` stays 6 and `INDEX_SCHEMA_VER` is unchanged, so no user re-indexes. Old + rows are migrated by `ROW_VERSION` alone: a row of another version is absent on read (the file is a candidate and + is scanned), a one-time version sweep, guarded by a `meta` marker so it runs once per index, deletes path-keyed and + other-version rows, and an incremental scan rebuilds the content-keyed rows (backfill reads each unchanged file once). +- **Wire.** `IndexReadQuery::GrepBloom { pattern, entries }` (`comms/index_read_proto.rs`). A client and daemon of + different builds fail to decode, and `IndexRoute::grep_skip` already degrades that to "sweep every file". +- **Layered note.** Rows are content-addressed and never shadowed: a layered read does `overlay.get(h)` then + `base.get(h)`; a stale base row for a dead hash is harmless because nothing asks for it. + +### A3: `IndexView` (standalone only) + +Shipped in `26be3d56`; the follow-ups below are required before A4 and are being applied on this branch. + +- `src/index/view.rs`: `enum Part` naming the 16 layered keyspaces (below), and + `trait IndexView: Send + Sync { get(Part, &[u8]); prefix(Part, &[u8]); range(Part, Bound, Bound); iter(Part); }` + returning `Box> + '_>`, plus the named accessors that exist today + on `IndexDb`: `references_to`, `definition_of`, `import_targets_of`, `importers_probing`, `is_probed`, + `import_probes_complete`, `symbols_by_name_lookup`, `symbols_index_is_empty`, `bm25_stats`, + `grep_bloom_row`, `grep_skip_verdicts`, and the two `NameDict` handles (`callee_names`, `trait_names`). +- `StandaloneView(&IndexDb)` delegates 1:1. `IndexRoute::Local` holds `Box`; `index_read_against`, + `query.rs`, `mcp/helpers_calls*.rs`, `mcp/helpers_impls.rs`, `search/{bm25,exact}.rs` and the resolve pass + (`intel/resolve_pass.rs`, which reads `import_targets`, `importers_by_probe`, `refs_*`) take `&dyn IndexView`. + Writes keep going through `IndexWriter` on the concrete `IndexDb`. +- **Backend-neutral values (follow-up, required by A3 and A4).** `KvResult`, `get` and `grep_bloom_row` in + `src/index/view.rs` return `fjall::Slice`, and `kv` takes a `fjall::Guard`. A follow-up commit on this branch replaces + them with a backend-neutral value type (cheap to clone, derefs to `[u8]`), so that no `fjall` type appears in the + trait and a sorted-run `BaseReader` can implement it without copying into Fjall types. Only `StandaloneView` and the + Fjall base reader convert at their edge. +- **Named accessors are default methods (follow-up).** Today `references_to`, `definition_of`, `import_targets_of`, + `importers_probing`, `is_probed`, `import_probes_complete`, `symbols_by_name_lookup`, `bm25_stats` and + `symbols_index_is_empty` are required methods that `StandaloneView` forwards to `IndexDb`. They become default methods + written over the generic `get` / `prefix` / `range` layer (the `Meta` scalars through `get(Part::Meta, ..)`), so a + layered view inherits them and cannot drift from the standalone decoding. `StandaloneView` may override a default only + for speed and the equivalence gate proves the override equal. `callee_keys_matching` / `trait_keys_matching` stay + required: a layered view must merge them (A5, *NameDict*). +- The existing suite is the gate. Perf discipline applies: the dyn call is per iterator, not per row, but a + p95 regression above 3% on `references` / `callers` (benches in `benches/`) switches `IndexView` to an enum + with the two variants instead of a trait object. + +### A4: repo-group layout, base generations, migration + +**Config.** New `[index]` table in `config/v1.rs` (`IndexConfig`, `deny_unknown_fields`, in `ConfigV1` after +`cache`): `layered: bool`, default `false` until A8. Regenerate `schema/` with the existing +`config_schema` test. The env var `BASEMIND_INDEX_LAYERED=0|1` overrides the file (rollback lever, same pattern as +`BASEMIND_NO_SEED`). `max_group_bytes: Option` caps a group's `base/` plus `overlays/` (*Disk budget*); unset means *auto*. The mode is resolved once per `Store::open`; the daemon applies a changed value when it next +opens the workspace. Layered applies to a workspace only when all hold: `layered` is true, the canonical root is the +git workdir, `Repo::discover` succeeds, a base commit resolves, and no embedding lane is enabled until A7 has +shipped (`documents.embed`, `code_search.embed`: their Lance rows are still keyed by worktree). `documents.embed` +defaults to `true` (`config/documents.rs` `default_embed`), `code_search.embed` to `false`, so out of the box layering +is off until A7 and A4–A6 apply only to users who set `documents.embed = false`. The armis measurement for A4–A6 must +run with embeds off, and A8 (layered by default) depends on A7. + +**Base storage (pluggable; prototype-first).** The base is read through a `BaseReader` behind `IndexView`; the overlay +and every mutable store stay Fjall. `base.json` carries `storage = "fjall" | "sorted-run"` and a reader is chosen per +generation from it. Owner decision: the base is prototype-first. An immutable mmapped sorted-run base, exported from +a finished Fjall generation, is built and benchmarked before A4 is implemented, and A4 must not be locked to a Fjall +base. Everything below that is Fjall-specific is marked *(Fjall option)* and is conditional on the prototype outcome: +the `index.fjall/` layout, `IndexDb::open_code_only`, the quiesce and `RwLock>` in *Advance*, the +generation clone, the daemon-only access rationale, and open question 1 and the matching Consequences text about one +process per base. A mmapped base can be opened by every process, which would mostly dissolve open question 1. + +*A4-prototype gate.* Export one armis base generation to a sorted-run file set and measure, against the Fjall +generation, on the same machine, cold and warm: + +| Measure | Acceptance (to be fixed by the owner from the first run) | +|---|---| +| base size on disk | no larger than the Fjall generation | +| build time (export from Fjall, and cold build) | recorded; export under 10 min | +| `references` / `callers` p95 through the `BaseReader` | within the 3% / 10% budgets of A3 and the measurement plan | +| open cost (time to first answer, RSS) | below Fjall open incl. journal replay | +| multi-process open | N processes open one generation concurrently, no lock, each reads correctly | + +If sorted-run meets the gate it becomes the default storage and the *(Fjall option)* items are dropped from the +spec; otherwise Fjall stays and the text applies as written. The `IndexView` and merge contract below are written +against `BaseReader` and hold for both. + +**Layout.** + +```text +cache/repos// + memory.fjall/ A0, untouched + group.json {format, main_root, common_dir, created_unix} + base.lock flock; serialises build / advance / retire / GC for the group + base/ + gen-/{index.fjall/ | index.run/, index.msgpack, base.json} published (index.fjall/ is the Fjall option) + .tmp--/ builder staging (incl. checkout/); swept after 1 h when unlocked + .retired-/ renamed out of selection, deleted once fjall can be locked + last-failure.json {error, unix}; a failed build is not retried for 10 min + overlays//{overlay.fjall/, delta.msgpack, overlay.json} + lance/ A7 +cache/workspaces// unchanged; `workspace.json` gains `repo_group` and `index_mode` +``` + +`WorkspaceMarker` (`store_layout.rs`) has no `deny_unknown_fields`, so the extra fields are invisible to older +binaries. `store_layout.rs` gains `base_dir`, `generation_dir`, `overlay_dir`, `BASE_LOCK_FILE` and friends; none of +`repos/` is ever listed by the budget eviction today and that changes in A6. + +**`base.json` (format 1, JSON, written last).** + +```json +{ "format": 1, "generation": 7, "commit": "", "tree": "", "source_ref": "origin/HEAD", + "built_from_generation": 6, "built_unix": 0, "basemind_version": "0.31.0", + "index_schema_ver": 0, "blob_schema_ver": 0, "min_extract_epoch": 1, + "scan_digest": "", "embed_policy": "", "files": 65123, "bytes": 0, "probes_complete": true } +``` + +A generation is *ready* iff `base.json` parses with `format == 1`. It is *compatible* iff `index_schema_ver == +INDEX_SCHEMA_VER`, `blob_schema_ver == extract::SCHEMA_VER`, `min_extract_epoch >= extract::EXTRACT_EPOCH` and +`scan_digest` equals the digest of the running config. `scan_digest` is the blake3 of the canonical JSON of the config +subtrees that change which rows a scan writes: `scan.{include,exclude,max_file_bytes}`, `languages`, `code_intel`, +`code_search.enabled`. `scan.extra_roots` is excluded: files outside the repository are always overlay-only. An +incompatible generation is ignored and collected, the same wipe-on-mismatch story as `INDEX_SCHEMA_VER` +(`schema-and-blob-compat` rule). + +**`index.msgpack` of a generation** is the existing `store::Index` (`files`, `doc_files` empty, `embed_policy`), +with every `FileEntry.mtime` normalised to 0. The base is code-tier only; the document tier stays in the overlay +until A7. + +**Base commit.** `refs/remotes/origin/HEAD` resolved with `Repo::resolve_rev`; if absent or unresolvable, `HEAD` of +the main worktree (`Repo::main_worktree_root`). basemind never fetches. + +**Builder (cold).** Holds `base.lock` (new `LockHolder::BaseBuild`; contention reports the holder from `.lock.meta`). + +1. Allocate `n = 1 + max(existing gen / retired numbers)`; create `.tmp-*`. +2. Export `tree(C)` to `.tmp-*/checkout/` via new `Repo::export_tree(rev, dest)` (regular files only; symlinks and + submodules are skipped, matching the scanner's filters). Peak disk 1-3 GB, deleted before publish. +3. Run `scanner::scan` with `ScanSource::WorkingTree` rooted at the checkout, `EmbedMode` off, into a private + `Store` whose view dir is `.tmp-*/`; this runs the wholesale resolve pass and sets `import_probe_ver`. +4. Normalise mtimes, rewrite `index.msgpack` atomically, flush every keyspace (the `rotate_memtable_and_wait` + loop in `IndexDb::open_inner`) and close fjall *(Fjall option)*. Closing makes the SSTs and the journal durable; the + builder then fsyncs every file under `.tmp-*`, so nothing published depends on page-cache state. +5. Write `base.json` last: write `base.json.tmp`, fsync it, rename it to `base.json`, fsync `.tmp-*`. +6. `rename(.tmp-*, gen-)` (atomic, same filesystem), then fsync the parent `base/` directory so the rename is + durable, then update `CURRENT` (tmp + rename, an advisory hint only). Every fsync is `F_FULLFSYNC` on macOS + (plain `fsync` does not flush the drive cache there; `src/store.rs` uses `sync_all` only, so this is new). + A crash before the rename leaves a `.tmp-*`; after it, a complete generation. A generation without a parseable + `base.json` is never ready. + +Readers never trust `CURRENT`; they list `base/gen-*` and read `base.json`. Blobs written by an aborted build stay +valid (content-addressed), so a restart re-runs mostly fjall writes. A disk-full or poisoned fjall (`IndexDb::is_poisoned`, +`fault.rs`, #79) aborts the build, removes `.tmp-*`, and writes `last-failure.json`. + +**Racing builders.** Two worktrees that both find no ready generation both try `base.lock` non-blocking. The loser +does not build: it re-checks `base/` every 2 s, answers queries from the `InRam` projection with the existing +partial-scan health notice naming the winner, and takes over (sweeping stale `.tmp-*`) if the flock is released +without a ready generation, which is what a crash of the winner looks like. At most one build per group is ever in +flight. + +**Bootstrap from an existing standalone index (the migration fast path).** If the group has no ready generation but +a workspace has a compatible standalone `views/working` (`store_seed::find_source` picks the source: main worktree +first, else the most files; its `.lock` must be acquirable by us, the same rule as `seed_working_view`), the builder +clones it (`store_seed::clone_dir`, CoW with a bounded plain copy) into `.tmp-*`, vets it (`vet_clone`), and corrects +it to `tree(C)` with the *advance* procedure below, treating every path where the source differs from `C` +(`Repo::status_against(C)` plus untracked plus basemind-only files, which are removed) as the diff. Cost is O(diff), +not a rescan, and the first base costs one CoW clone instead of 17-33 min. + +**Advance (new generation from an old one).** Under `base.lock`, for the newest ready generation `k` and target `m`: + +1. *(Fjall option)* Quiesce `k`: if this process is the daemon it takes the write side of the group's + `RwLock>`, drains, and drops its handle; otherwise it tries to open `k`'s fjall and aborts the advance + on `Locked`. A live fjall is never copied (the rule in the `store_seed.rs` module doc). Readers of the group stall + while the write side is held. The stall is bounded by a budget, `[index] advance_stall_ms` (default 2000), covering + the drain of in-flight queries, the CoW clone (O(files)) and the reopen of `k` (open cost, which includes + journal replay and the `rotate_memtable_and_wait` flush in `open_inner`, see *Lifecycle*). Exceeding it aborts the + advance, reopens `k`, and retries after a backoff; readers are never stalled past the budget. The default is sized + from armis: a CoW clone of its 1.7 GB, 102-file index takes 0.36 s on APFS, so 2 s covers drain, clone and reopen + with headroom while staying under an interactive tool-call timeout; an advance is background work, so an abort + costs only a retry. A sorted-run base needs + no quiesce: its files are immutable, so the clone is a hardlink or CoW copy and no reader is touched. +2. Clone `gen-k` to `.tmp-*` and, for Fjall, reopen the daemon handle immediately. +3. `Repo::diff_trees(C_k, C_m)` (new, gix tree diff; a rename is a delete plus an add) filtered through `Filters`. +4. Export only the added/modified blobs plus the unchanged files the resolve pass probes (A1's `probed_only` + expansion) to `.tmp-*/checkout/`, apply with `scan_paths_with_observer` through the normal writer, run the + incremental resolve pass, then steps 4-5 of the cold build. +If the filesystem has no CoW and free space is below twice the generation size, the advance is skipped and retried +later (*Disk budget*). + +**Lifecycle (what "published" means).** A published generation is never rewritten by basemind's logic, but a Fjall +generation is not read-only on disk: opening it replays the journal and compacts, and `IndexDb::open_inner` calls +`rotate_memtable_and_wait` on every keyspace (`src/index/mod.rs`; best-effort, a failure only logs). So every open of a +Fjall generation can write, a CoW clone of it un-shares as compaction rewrites pages, and *Advance* must quiesce the +daemon handle before cloning. The builder's flush loop leaves the journal empty, which bounds replay at the next open +to nothing for a freshly published generation. A sorted-run generation has none of this and is read-only in fact. + +**Selection.** An overlay pins `base_generation`. On open without a usable pin (first open, pin retired, +incompatible) choose among compatible ready generations the one with the smallest candidate count (`|K|` below, +computed once per generation, at most three git status calls), ties to the newest. A pin is kept while its +generation exists and is compatible. + +**Opening the base.** *(Fjall option)* `IndexDb::open_code_only(gen_dir)`: like `open_tolerating_held_memory` with the memory +store always detached, because the base must not take the group's exclusive `memory.fjall` lock. The daemon's +`src/comms/workspace_pool.rs` keeps one `Arc` per `(group, generation)`; every pooled workspace of the group shares +it and its block cache. A non-daemon process uses `open_index_with_retry`'s budget and then treats `Locked` like a +lost lock today: it routes through `IndexRoute::Daemon` if a daemon answers, else `InRam`. Open question 1. + +**Migration of a workspace (standalone to layered), one-way, under the workspace `.lock`.** One workspace at a time per +machine (a machine-wide try-lock; the others stay standalone and retry on their next open). + +1. Check the free-space precondition (*Disk budget*); resolve or build a base (above). +2. Compute the overlay (A5, "Building the overlay") from the live tree. +3. Verify: for 200 random paths plus every path of `S_path`, compare the merged rows of each layered keyspace with + the standalone index (`IndexView` makes this one function); any difference aborts, removes the overlay, and keeps + the standalone view. Verify cannot see a defect the standalone index and its clone share, which is why the BM25 + layout stamp below is checked first. +4. Write `overlay.json` with `state = "ready"`, set `index_mode = "layered"` and `repo_group` in `workspace.json`, + rename `views/working` to `views/working.migrated`, and delete it right after verify and the commit above succeed + (a crash between the rename and the delete leaves a restorable index; the next open finishes the delete). The + transient cost is the old index kept until then, about +2.9 GB for the one workspace being migrated. + `store_seed::seed_working_view` is skipped for layered-eligible workspaces. + +**Damaged-index inheritance.** `vet_clone` (`store_seed.rs`) checks only `msgpack` and the Fjall `schema_ver`, so a +standalone index that already lost postings for duplicate content (the defect A5.0 fixes) would be cloned into the +base and verify would compare against the same defect. `meta` therefore gains a `bm25_layout` stamp (written by A5.0's +fixed writer). `vet_clone` requires the stamp to equal the current layout; a missing or older stamp rejects the +clone as a base source, and a standalone index with an old stamp does a one-time postings re-derive from the +content-addressed chunk blobs on its next scan (the `by_path` rows hold the term list without frequencies, so they +cannot rebuild the postings). Only then is it eligible as a clone source. + +**Disk budget.** A group holds up to 3 live generations (1.7 GB each on armis today), and compaction un-shares the CoW clones, +so the realistic footprint is the sum, not one generation plus deltas. `[index] max_group_bytes` caps `base/` plus +`overlays/`. Its default, *auto*, is 4x the newest ready generation's size: two live generations, the transient export +plus clone of a build, and the overlays (each holds changed paths only). A fixed byte default would be wrong at both +ends: too small for a repo whose single generation exceeds it, and far above need for a small repo. On armis (1.7 GB +generation) auto is about 7 GB, against 33 x 1.7 GB of standalone indexes today. Retention is therefore 2 generations +by default (newest plus the one pinned overlays are rebasing off), 3 only when the cap allows. Build, advance and migrate each check, under `base.lock`, that free space on +the volume is at least the generation size plus the cap on the transient (export plus clone) and that the group stays +under the cap, else they are skipped, a `last-failure.json`-style marker is written, and `doctor` says why. When disk is +tight the newest generation replaces the oldest instead of the group keeping 2 (the oldest goes through the *Liveness* +rules, so a pinned one is never deleted from under a reader). + +### A5: overlay write path and the read merge + +**A5.0 (prerequisite, standalone fix).** Postings are shared by content hash, so a path-driven delete must not +remove them while another path holds the hash. A refcount handed to `upsert_bm25_file` / `remove_bm25_file` from +`Store.index.files` is rejected: refcounts computed while parallel `WorkerIndexBatch` staging runs race the serial apply +pass (a count taken before a sibling path's batch lands is stale). Instead the writer's per-path delete of +`code_bm25_postings` is dropped (the `code_bm25_by_path` forward rows of the path are still deleted) and a +**hash-liveness sweep** next to `sweep_grep_bloom_rows` (`scanner.rs`) removes, after the stale-entry purge of a +`WorkingTree` scan, every posting whose `chunk_id` hash is not in the hash set of `store.index.files`. This is +symmetric with A2: both sweeps run from the post-pass state, are key-only for postings (the hash is parsed from the +key, `keys::parse_code_bm25_posting_chunk_id`) and tolerate leftovers until the next full pass. The watcher's +`scan_paths` deletes the postings of a hash that left the manifest, using the post-pass hash set. The same sweep is the +layered `D_hash` rule applied to a standalone index. + +`N` semantics, pinned: `N` and `total_len` come from `code_bm25_by_path`, which has one row per `(path, chunk_id)`, so +`N` counts path-chunks (two paths with identical content count twice). `df(term)` counts `code_bm25_postings` rows, +which are keyed by `chunk_id`, so it counts unique chunks (once per distinct content). The asymmetry is intended: it is +what a standalone index computes today and what the layered formula (below) reproduces. The equivalence test asserts it +explicitly with a duplicate-content scenario (`N` twice, `df` once) in both modes. + +Red tests on `main` today: two paths, identical content, remove one, the other's postings survive. `meta` gets a +`bm25_layout` stamp written by this fix; see *Damaged-index inheritance* (A4). + +**`delta.msgpack` (format 1, MessagePack named, `rmp_serde`, same atomic tmp + rename as `Store::flush`).** + +```rust +struct Delta { + format: u16, // 1 + seq: u64, // monotonically increasing per commit + base_generation: u32, base_commit: String, + entries: BTreeMap, // S_path + restitched: BTreeSet, // S_refs minus S_path + doc_entries: BTreeMap, // the whole document tier until A7 + doc_tombstones: AHashMap, + embed_policy: String, +} +enum DeltaEntry { Added(FileEntry), Modified(FileEntry), Deleted } +``` + +`Added` is a path not in T; `Modified` is a path in T whose hash differs; `Deleted` is a path in T not in W. A path +whose live hash equals its base hash is never in the delta (a revert removes the entry and its overlay rows). A +version mismatch of `format` discards the overlay (it is a cache). Derived and never persisted, rebuilt at open in +O(delta): `S_path = keys(entries)`, `S_refs = S_path U restitched`, and +`D_hash = { base hash h of p in S_path } \ { h : base still has an unshadowed path with h } \ { hashes of Added/Modified }`, +computed from the per-hash path counts of H_base. + +**`overlay.json` (format 1).** `{ format, worktree_key, root, base_generation, base_commit, base_scan_digest, +index_schema_ver, head, state: "ready"|"dirty", delta_seq, changed_files, changed_bytes, updated_unix }`. + +**Crash safety.** A commit is: (1) set `state = "dirty"`; (2) one fjall batch containing the rows and `meta.delta_seq = +seq`; (3) write `delta.msgpack` with the same `seq` (the commit point, as `index.msgpack` is today); (4) `state = +"ready"`. On open, `dirty`, or `meta.delta_seq != delta.seq`, or a base pin that is missing or incompatible, discards +the overlay and rebuilds it. The overlay is a cache; blobs are global, so a rebuild costs O(diff), not O(repo). + +**Building the overlay (full pass).** `K`, the candidates, is `{ p in W : p not in T }` U `{ p in W : p in D }` U +`{ p in W : git neither tracks nor reports p }` (basemind-only files: gitignored but admitted by `scan.include`, and +`scan.extra_roots` files), where `D` is the set of tracked paths that differ from `tree(C_base)` in the index or the +worktree, from new `Repo::status_against(tree_sha)` (gix status with `head_tree`), whose untracked part is added to +`K`. Only `K` is read and hashed. Each `p in K` becomes `Added`, `Modified`, or nothing (equal hash); each `p in T \ W` is +`Deleted`. Then the rows of every `Added` / `Modified` path are written through the ordinary `IndexWriter` into the +overlay fjall, and the incremental resolve pass (A1) runs with reads from the merged view and writes to the overlay: +the importers it re-stitches form `restitched`. `Store.index.files` is the *effective* manifest (base overlaid with +the delta), so `store.lookup`, `MapCache::build`, the blob GC and every existing caller keep working; only +`Store::flush` differs and persists the delta, driven by a `dirty_paths` set, with a full diff only at open. The +stale purge in `scan_with_cancel` is replaced in layered mode by the `Deleted` computation above. + +**Watcher and `scan_paths`.** Events go through `scan_paths_with_observer` unchanged. For each path the post-state +is classified against T with the same four cases (revert, modify, add, delete) and the delta, overlay rows and +`restitched` are updated in one commit (the crash protocol above). The `.git` directory is already excluded; a +`HEAD` / index change is not a watcher event today, so a branch switch is picked up by the next full pass, which +the daemon schedules whenever `git rev-parse HEAD` of the worktree changed since `overlay.json.head`. + +**Concurrency.** The overlay writer is whoever holds the workspace `.lock` (`acquire_lock_as`), the same single +writer as today (the daemon when running). The base is read-only to everyone but the builder. A layered reader loads +`Arc` (the three sets) and opens its base and overlay iterators under a short read lock; the writer takes the +write lock only to commit the batch and swap the `Arc`, so a reader never sees a path in both layers or in +neither. Fjall iterators are snapshots, so the lock is not held while iterating. + +**Merge contract.** Merged order is plain byte order of the raw key (both layers use the same encoders in +`index/keys*.rs`); keys never collide across layers because a base row is dropped exactly when the overlay may hold a +row for the same shadow unit. Tombstones, snapshots and scalars are specified here once. + +- **Tombstones are masks, not rows.** Overlay `get` internally distinguishes *tombstone* (the key's shadow unit is + masked, so the answer is the overlay's, and absence there is final) from *absent* (not masked, so fall through to the + base). The mask is the shadow set of the table below; at an equal key the overlay shadows the base in `prefix` / + `range` / `iter`. No tombstone rows are stored. +- **Inverted keyspaces use a per-path mask.** `calls_by_callee`, `symbols_by_name`, `refs_by_def`, + `implementations_by_trait`, `imports_by_module` and `importers_by_probe` hold the path inside the key, not as a + prefix, so a delete cannot be a range delete. Rejected: per-key tombstones derived by walking the `*_by_path` + forward keyspaces (it costs a base walk per deleted path and writes a row per key). Chosen: the shadowed-path set the + ADR already defines (`S_path`, and `S_refs` for the ref and probe keyspaces; a `Deleted` path is in `S_path` with no + overlay rows). Each base row of such a keyspace is dropped when the path parsed from its key by the keyspace's + `Part` parser (`keys::parse_symbol_by_name`, `parse_call_by_callee`, `parse_ref_by_def` use-path, + `parse_impl_by_trait`, `parse_import_by_module`, `parse_importer_by_probe`) is in the mask. Forward `*_by_path` + keyspaces use the cheaper prefix check. +- **NameDict.** `callee_keys_matching` and `trait_keys_matching` return a key-ordered (`len`, then bytes), deduplicated + union of the base and overlay dictionaries, one per layer (the dictionary is a superset filter, `name_dict.rs`, so a name that survives only in a + shadowed base row yields a harmless empty scan). The merged iterator dedups equal names across layers and honours + `cursor_after` and `lower` on the merged order. +- **Snapshot / pin handle.** A query takes one `ViewSnapshot { Arc, base reader pin, overlay iterator + snapshot }` and every `get` / `prefix` / `range` of that query goes through it, so the two layers and the mask are + read at one consistent point even if the writer commits or a rebase swaps the overlay mid-query. The pin keeps the + generation (and its overlay directory) alive until the snapshot drops (A6, *Liveness*). +- **Meta scalars.** `code_bm25_n`, `code_bm25_total_len` and the import-probe completeness flag are precomputed + scalars, so they need a decision rather than a row merge. BM25: effective value = base scalar + overlay delta, where + the delta is `N_overlay - |base by_path rows of S_path|` (and the same for `total_len`) with the shadowed term computed + from the base `code_bm25_by_path` rows of `S_path` at open and on each `S_path` change. This is the same formula as + the BM25 text under *Totals, caps and cursors*; the overlay's own `code_bm25_n` / `code_bm25_total_len` hold + `N_overlay` / `total_len_overlay` (its own rows only), not corpus totals. `recompute_bm25_stats` on an overlay + therefore sweeps only overlay rows. `import_probes_complete`: base value (a generation is published only complete) + AND the overlay's own flag (cleared while the overlay's resolve pass is incomplete). +- **Visited-row budget.** `scan_cap` caps hits, not rows examined (`mcp/helpers_calls_scan.rs`). A filtered scan whose + mask drops many base rows (a widely edited name) can examine far more rows than it yields, so the merged iterator + carries a visited-row budget (default `scan_cap * 64`, at least 128k rows given the `scan_cap` floor of 2000, about + 0.1 s of fjall prefix iteration, so a heavily shadowed name degrades to partial rather than stalling); when it is hit the scan stops, returns what it has and sets the + existing partial flag (`total_is_partial`). The equivalence gate runs with the budget unbounded and a separate test + proves the partial flag. + +**Merge rule per partition (every keyspace opened by `IndexDb::open_inner`).** + +| Keyspace | Key (shadow unit) | Overlay holds | Base row dropped when | Notes | +|---|---|---|---|---| +| `meta` | string keys | `delta_seq`, overlay-only `code_bm25_n`, `code_bm25_total_len` | never merged as rows | `schema_ver`: both must equal `INDEX_SCHEMA_VER`. `import_probe_ver`: base AND overlay flag (merge contract). `bm25_layout`: both must equal the current layout. BM25 stats: base plus overlay delta (merge contract). | +| `symbols_by_path` | `rel` + start | rows of `Added`/`Modified` | `rel in S_path` | | +| `symbols_by_name` | name + kind + `rel` + start | same | `rel in S_path` | read by `symbols_by_name_lookup`; ordered by name then kind then `rel`, so the merge interleaves layers inside one name | +| `calls_by_path` | `rel` + start | same | `rel in S_path` | | +| `calls_by_callee` | callee + `rel` + start | same | `rel in S_path` | backs `references` / `callers`; `callee_names` dictionary = base U overlay (a superset of names is already allowed by `name_dict.rs`) | +| `imports_by_path` | `rel` + module + start | same | `rel in S_path` | written, no reader today; rule kept so the equivalence test can compare it | +| `imports_by_module` | module + `rel` + start | same | `rel in S_path` | as above | +| `implementations_by_path` | `rel` + trait + type + start | same | `rel in S_path` | | +| `implementations_by_trait` | trait + type + `rel` + start | same | `rel in S_path` | `trait_names` dictionary = base U overlay | +| `refs_by_path` | use `rel` + start | edges whose use is in `S_refs` | `use in S_refs` | `definition_of` hits one layer per use path | +| `refs_by_def` | def `rel` + start + use `rel` + start | edges whose use is in `S_refs` (def anywhere) | `use in S_refs` | def-side changes are covered because every cross-file user of a changed def is an importer that probed it and is therefore in `S_refs` (A1's exactness, which the A5 gate must prove for barrels and tsconfig) | +| `import_targets` | importer + local_start (incl. reserved manifest row) | rows of importers in `S_refs` | `importer in S_refs` | | +| `importers_by_probe` | probe hash + importer | rows of importers in `S_refs` | `importer in S_refs` | | +| `code_bm25_postings` | term + `chunk_id` (`:`) | postings of hashes not in H_base | hash of `chunk_id` in `D_hash` | a hash still live through another path keeps its base postings and is not rewritten | +| `code_bm25_by_path` | `rel` + `chunk_id` | rows of `Added`/`Modified` | `rel in S_path` | forward map; also the source of the BM25 stats correction | +| `grep_bloom` | content hash | rows for hashes with no base row | never | union by key, overlay first (A2) | +| `embeddings` | reserved, never written | none | n/a | must stay empty in both layers; asserted by the gate | +| `memory_by_key`, `memory_archive`, `proposals` | not in base or overlay | none | n/a | live in `memory.fjall` (A0); base and overlay handles are opened detached | + +**Totals, caps and cursors.** The merged iterator yields exactly the `(key, value)` sequence a standalone index of +the same tree yields, filter applied before anything is counted. So `scan_calls_by_name` and `scan_impls_fjall` +(`scan_cap = max(limit * 8, 2000)`, `total_is_partial`, `has_more`) are unchanged, and a cursor is still the raw key +of the last hit as an exclusive lower bound: it is valid on both layers because each layer positions with `range`. +Cursors cross a rebase or generation change only if the merged sequence is identical, which is not guaranteed; a +cursor older than the last overlay commit is served best-effort (the existing behaviour when a standalone index is +rescanned between pages). BM25: `bm25_stats()` returns `N = N_base - |base by_path rows of S_path| + N_overlay` and +`total_len` likewise (the formula of the *Meta scalars* decision, with the subtraction computed from the shadowed base +rows at open and on each `S_path` change); `df(term)` is the count of merged postings rows, so `N` is per path-chunk +and `df` per unique chunk (A5.0). + +**Manifest memory.** The effective manifest does not clone the base `FileEntry`s per worktree (65k entries times the +worktree count). The base manifest is shared as one `Arc` per `(group, generation)`, and each worktree holds a +delta map (`delta.entries` plus the derived sets); `lookup` consults the delta first and then the shared base. Code +that needs an owned `Store.index.files` map materialises it on demand, not at open. + +**Gate.** `tests/layered_equivalence.rs`: a proptest over three worktrees of one group with random edit scripts +(modify, add, delete, rename, duplicate content, barrel re-export, `tsconfig` paths change, Python package +`__init__`), each compared against a freshly built standalone index of the same tree for every `Part`, every named +accessor, every MCP mode that reads the index, with totals, `has_more` and cursor round trips. The `callee_names` / `trait_names` dictionaries are exempt from +byte-equality (each layer's dictionary may hold stale names, and a layered one is a union); the gate compares the +query results they serve instead. Behind +`[index] layered = false` until green on 1000 cases. + +### A6: generations, rebase, promotion, GC + +**Advance triggers** (evaluated by the daemon maintenance task at start, hourly, and after each overlay full pass; +by `basemind scan` when no daemon runs): `|diff_trees(C_cur, origin/HEAD)|` is at least 2% of `base.json.files`, or +the newest generation is older than 24 h and that diff is non-empty, or an overlay breaches the rebase threshold +while the newest generation is behind `origin/HEAD`. One advance at a time per group (`base.lock`). + +**Rebase and promotion** are evaluated after each overlay commit. Breach = `changed_files > 15%` of `base.json.files` +or `changed_bytes > 25%` of the base manifest's total `size_bytes`. + +- If a newer compatible generation exists whose candidate count is lower, **rebase**: build `overlays//next/` against + it from scratch (O(candidates); blobs are reused), then swap the directories with two renames while the old + overlay serves reads. A crash leaves a disposable `next/`. +- Otherwise, or if the rebased overlay still breaches 10% / 20%, **promote**: clone the pinned generation into + `views/working` (`store_seed` machinery, daemon-quiesced as in *Advance*), apply the overlay's delta through + `scan_paths` plus the incremental resolve pass, set `index_mode = "standalone"`, delete the overlay. A promoted + workspace is never re-layered automatically (no flapping); `BASEMIND_INDEX_LAYERED=1` plus `basemind cache reset + ` is the manual way back. Promotion that cannot quiesce the base is deferred and warned in `doctor`. + +**Liveness.** A generation is live iff it is the newest ready compatible one or an overlay in the group pins it. At +most three are live; when publishing a fourth, the overlays pinned to the oldest live generation are invalidated (their +overlay dir is deleted, they rebuild on next open), except an overlay pinned by an active workspace handle or open in +the daemon: that one is never deleted from under a reader. It is rebased onto a newer generation first (the swap of +*Rebase and promotion*, so the old overlay serves reads until the new one is ready), or, if the rebase cannot run, the +oldest generation and its overlay stay live past the limit until the handle drops (the pin is released when the last +`ViewSnapshot` and handle for it are gone), and the next maintenance pass retires them. Readers meanwhile keep reading +the pinned generation and overlay unchanged and see the answers of that older tree; `doctor` reports the overstay. +Under disk pressure (*Disk budget*) the overstay is bounded by `max_group_bytes`; past it the workspace is promoted +or its overlay is dropped and the open handle is reopened. There is no eager rebase of idle worktrees. A non-live generation +is renamed to `.retired-` and removed when its fjall lock can be taken *(Fjall option; a sorted-run generation is +removed once no snapshot pins it)*. An incompatible or half-published +generation (no `base.json`) is non-live. An overlay is live iff `workspaces//workspace.json` names an existing +root; orphans go with `reap_orphaned_workspaces_in`, which must also remove `repos//overlays/` +(group read from the marker), and an overlay older than its workspace's `index_mode` flip is removed. + +**Interaction with `store_gc*.rs`.** + +- *Blob liveness* (`collect_referenced_hashes`, `collect_referenced_hashes_global_in`, `LiveBlobs`): also union, per + group, the `index.msgpack` of every `base/gen-*` and `.tmp-*` whose flock is held, and every + `overlays/*/delta.msgpack` (hashes of `Added` / `Modified`; `doc_entries` into `docs`). A manifest that is + unreadable and not provably a foreign schema aborts the sweep (the existing safety rule); a foreign-schema + generation is deleted under `base.lock` instead of healed. The 6-hour `BLOB_GC_GRACE` covers builds in flight. +- *Budget* (`store_gc_budget::cache_footprint`, `eviction_candidates`, `rebuildable_workspace_paths`): the footprint + gains `repos/*/base` and `repos/*/overlays`; evicting a workspace also removes its overlay; a group's `base/` is a + candidate only after every workspace of the group is evicted or idle past the grace window, and never while any + of its overlays or `base.lock` is locked. `memory.fjall` stays untouched. +- *Protection* (`store_gc_protect.rs`): a workspace whose overlay pins generation `k` makes `k` part of its + protected footprint. +- *Locking*: destructive sweeps keep the machine-wide lock in `store_gc_live.rs`; per-group work takes `base.lock` + with a try-lock and skips a busy group. +- *Lance* rows: after A7, rows whose hash no live manifest references and that are older than the blob grace window. + +### A7: shared Lance tables + +- **Placement.** `repos//lance/-/`, one directory per `(embedding_model, dim)`, because + `LanceStore::open` wipes the whole directory on a `LanceMeta` mismatch (`lance/mod.rs`) and a worktree with a different + preset must not wipe its siblings; such a worktree stays standalone. `memory.lance` stays per workspace. A per-group + `lance.lock` flock serialises write transactions across CLI and daemon. +- **Schema.** `documents`, `doc_links` and `code_chunks` gain `content_hash` (Utf8). Row identity becomes + `(scope, content_hash, chunk_idx)`; `path` stops being part of identity. This is a column change, so it ships in a + `RELEASE_MINOR` bump (`MEMORY_SCHEMA_VER` rule), with wipe-on-mismatch as the migration: the vectors live in the + `.doc.msgpack` / `.chunk.msgpack` blobs, so the rebuild re-flushes rows from blobs with no model inference (as + `purge_unembedded_*` and the seed path already rely on). +- **Writes.** The scanner inserts rows for a hash only if none exist (`content_hash IN (batch)` probe), under + `lance.lock`. Rows are never deleted when a path changes; the A6 GC deletes unreferenced hashes. +- **Reads.** KNN returns the `k' = clamp(3k, k + 64, 4096)` nearest rows, drops rows whose hash is not in the + effective manifest's hash set, and expands each surviving hash to every live path holding it (the reverse map is + built at open), one hit per `(path, chunk)`. If fewer than `k` hits survive and `k'` was full, double `k'` up to + 4096 and stop. This also replaces taking the hit path from the blob (`mcp/helpers_code_search.rs` `hydrate_one`, which + reads `chunk.path` from the content-addressed sidecar and is wrong for duplicated content). Ties are ordered by + `(distance, path, chunk_idx)` in both modes. +- **Documented deviation.** ANN recall differs from a per-worktree table. Gate: top-10 overlap with a per-worktree + table at least 0.95 over 200 queries per tier on armis (`src/eval`); below that, raise `k'` or use exact search for the + shared table. +- **Doc tier.** Until this phase the overlay carries every document (`Delta.doc_entries`); after it, documents delta + like code. `scope` stays the normalised remote URL; it no longer separates worktrees. + +### A8: layered by default + +Preconditions: A5 gate green on 1000 cases, A6 GC merged, A7 shipped (A8 depends on A7, because A4 disables layering whenever an embedding lane is on and `documents.embed` defaults to `true`), measurement targets met. Then: default +`[index] layered = true`; delete `store_seed.rs`, `BASEMIND_NO_SEED`, `seed_working_view` and its callers in +`Store::open_with_holder`, and the lance-staleness sentinel `seed:rebuild-lance`; migration (A4) runs for every +standalone workspace on its next writer open. + +**Rollback.** Per workspace: `[index] layered = false` or `BASEMIND_INDEX_LAYERED=0` makes the next writer open +promote (A6 procedure) or, if the base is gone, cold-scan into `views/working`; a read-only open under `false` +with an overlay present ignores the overlay, reports "no index, run basemind scan" and writes nothing. Binary +downgrade: `repos//{base,overlays,lance}` are never listed by an older GC (it only knows `memory.fjall`) and +the extra `workspace.json` fields are ignored, so an older binary simply cold-scans a fresh standalone view; the +cost is one rescan and the stranded disk, reclaimable with `basemind cache clear`. No format here changes +`INDEX_SCHEMA_VER` or `extract::SCHEMA_VER`, so a downgrade never wipes a base. + +### Observability + +- `doctor`, `status.json` and the stats surface expose `index_mode` (`standalone` | `layered`) and, when layered was + not chosen, the reason: `embed_on`, `subdirectory_root`, `no_base_commit`, `config_off`, `base_building`, + `disk_budget`, `incompatible_base`. +- For a layered workspace: the pinned generation and its `base.json` summary, overlay size and `changed_files` / + `changed_bytes`, rebase and promote counts, base-build state (idle, building by whom, last failure), generations + overstaying the limit, and merge-read latency (p50 / p95 of the merged iterators per `Part`). +- `tracing` events on every state transition: base build start / publish / abort, advance start / quiesce / abort, + generation retire, overlay commit / discard, rebase, promote, migration start / verify-fail / commit. + +### Acceptance criteria + +Each phase is done when its line holds; the fault-injection list below is the shared gate for A4 and A6. + +- **A2.** The grep-bloom, version-sweep and watcher-delete tests pass; the 20-pattern skip ratio does not worsen. +- **A3.** The existing suite passes through `IndexView`; no `fjall` type appears in the trait; the named accessors are + default methods; `references` / `callers` p95 within the A3 budget. +- **A4.** Prototype gate evaluated and storage chosen; a cold build, a racing second builder, a bootstrap-by-clone and a + migration each produce a generation / overlay whose merged rows equal a standalone index (A5 verify function); + publish is crash-safe at every step; disk preconditions skip rather than fail; all with embeds off. +- **A5.** A5.0 fix and BM25 `N` / `df` equivalence pass; the equivalence gate is green on 1000 cases; overlay under 50 MB + for a 50-file diff; visited-row budget proven. +- **A6.** Advance, rebase and promote are exercised end to end; no overlay or generation pinned by a live handle is + deleted; GC leaves exactly the live set and protects in-flight `.tmp-*`; `max_group_bytes` is honoured; the + fault-injection list passes. +- **A7 / A8.** Per their sections and the measurement plan. + +Fault injection (each is a test that kills or fails the step and asserts the post-state and a clean re-run): + +- publish crash at each step: before `base.json.tmp`, after its fsync, after its rename, after `rename(.tmp-*, gen-n)`, + after the `base/` fsync, after `CURRENT`; +- racing builders (two processes, one winner, loser takes over after a winner crash); +- GC of `.tmp-*` and `.retired-*` (grace honoured, a locked or pinned one kept); +- quiesce and advance (stall budget exceeded, `Locked`, crash mid-clone); +- migration abort (crash between each step, restore from `views/working.migrated`) and migration-verify failure + (injected row difference keeps the standalone view and removes the overlay). + +### Measurement plan + +On armis (89k files, 65k indexed), 1 worktree then 5, standalone vs layered, same machine, cold cache and warm, with embeds off (`documents.embed = false`) until A7 ships, because A4 does not layer otherwise: + +- `du -sh` of the base generation and of each overlay (target overlay < 50 MB with a typical 50-file diff); + `du` of `grep_bloom` before and after A2 (expect a drop proportional to duplicate content); +- time to first answer on a cold second worktree (target < 60 s vs 17-33 min) and the first-base build time; +- peak RSS of scan and of the daemon with 5 worktrees open (the base block cache is shared once); +- p50 / p95 latency for `references`, `callers`, `implementations`, `search_code` keyword and semantic, `code grep`; + a regression above 10% p95 on any read mode blocks A8; +- watcher latency when a widely imported file changes (the `restitched` set and the re-stitch time); +- grep bloom skip ratio and candidate counts on 20 fixed patterns before and after A2 (must not worsen); +- promotion / rebase frequency over a week of normal use. + +### Open questions for the owner + +1. **Base access without a daemon** *(Fjall option; mostly dissolved if the A4 prototype adopts a mmapped sorted-run + base, which every process can open)*. Fjall admits one process, so with `layered` on every CLI invocation in a group + contends for the base. The proposal above degrades like a lost lock today (`Daemon` route, else `InRam`). The + alternative is to make the daemon mandatory for layered workspaces (auto-spawn through `src/comms/singleton_spawn.rs`, and + fall back to standalone if it cannot start). Which is acceptable for the A8 default? +2. **A2 migration.** Decided: `ROW_VERSION` 2 plus the sweep, no partition-revision bump. +3. **Daemon-less layered mode.** Decided: it degrades like a lost lock today (`Daemon` route, else `InRam`). +4. **Base storage.** Decided: prototype-first. A sorted-run base is built and benchmarked before A4 (*Base storage*); + Fjall stays for overlays and mutable stores; A4 is not locked to a Fjall base. + ## Consequences Per-worktree disk and scan cost become proportional to the diff, and one block cache serves a whole group. @@ -51,7 +664,15 @@ The read path grows a merge layer that must preserve order, totals and cursors f correctness gate (a proptest over three worktrees with random edits, comparing every mode's results and cursors to a standalone full index) exists to catch a merge that does not. `store_seed.rs` is removed once layered mode is the default. Phases A0 (per-repo memory store) and A1 (probe index and O(affected) -re-stitch) shipped in 0.30.0; A2–A8 remain. +re-stitch) shipped in 0.30.0; A2 (`efe41b76`) and A3 (`26be3d56`) shipped on this branch; A4–A8 remain, specified +below. Until A7, layering reaches only users with embeds off (`documents.embed` defaults to `true`). + +Costs accepted by this design: with a Fjall base, a layered workspace depends on the daemon for concurrent base access +(a mmapped sorted-run base would remove that, pending the A4 prototype); the +first base of a group costs one cold build (or one CoW clone when a standalone index exists); migration is +one-way per workspace (rollback re-materialises a standalone index); the document and embedding tiers stay +per worktree until A7; and the overlay is treated as a disposable cache, so any doubt about its consistency +costs a rebuild proportional to the diff rather than a repair. ## Alternatives considered @@ -59,3 +680,13 @@ re-stitch) shipped in 0.30.0; A2–A8 remain. - **Deduplicate full copies.** Rejected: compaction breaks the sharing. - **One shared Fjall with path-prefixed keys.** Rejected: name scans multiply by the worktree count, and one lock covers the whole group. +- **A virtual git-tree filesystem for the resolver, to build the base without exporting.** Rejected: it + re-implements the resolver's filesystem surface (`intel/resolver.rs`, `resolve_pass.rs`) to save a transient + 1–3 GB export. +- **Eager rebase of idle worktrees when a generation is retired.** Rejected: it needs the worktree on disk + and its lock; invalidating the overlay and rebuilding it at next open is simpler and costs O(diff). +- **Repairing a suspect overlay in place.** Rejected: the overlay is a cache of the diff; wiping and + rebuilding it is cheaper to reason about than reconciling `delta.msgpack` against fjall. +- **Opening a Fjall base from every process.** Not possible: fjall's directory lock is exclusive, which is why + the daemon holds it for a Fjall base (open question 1 covers the no-daemon case). An immutable mmapped base is the + alternative under prototype (*Base storage*). diff --git a/docs/layered-index-design.md b/docs/layered-index-design.md index 2e5aa478..9023b47a 100644 --- a/docs/layered-index-design.md +++ b/docs/layered-index-design.md @@ -1,6 +1,6 @@ # Layered index: design and implementation plan -- **Status:** Design accepted, not implemented +- **Status:** A2 and A3 implemented; A4–A8 not started - **Date:** 2026-10-18 - **Decision record:** [ADR-0016](adr/0016-shared-base-index-and-worktree-overlays.md) - **Tracking issue:** #87 @@ -11,6 +11,15 @@ this file records *how* — the on-disk layout, the data model, the exact read-m the write path, the correctness gate, and the file-by-file phase breakdown. Symbol names and paths are as of 0.30.0. +> **Superseded in part.** The ADR-0016 *Specification* section is authoritative wherever it and this +> file disagree. It corrects this plan against the code and records the owner decisions taken since: +> A2 migrates by `ROW_VERSION` 2 with **no** `INDEX_PARTITION_REVISION` bump; `IndexView` returns +> boxed iterators over a backend-neutral `IndexBytes`, not visitors over `Vec`; the base is read +> through a pluggable `BaseReader` (Fjall or an immutable sorted-run format, chosen by a prototype +> before A4); the base is built by scanning an exported tree as a working tree (a `Rev` scan has no +> resolve pass); and `code_chunks` is keyed `(scope, path)`, so A7 must re-key it too. The sections +> below that these touch carry a pointer to the ADR. + ## 1. Problem Each git worktree is its own workspace: the store key is blake3 of the canonical worktree root @@ -189,6 +198,9 @@ it holds. This keeps `upsert_file`'s read-before-write and per-file commit seman ## 8. `IndexView` trait (A3) +> Implemented differently; see ADR-0016 §A3 and `src/index/view.rs`. The sketch below is the +> original proposal, kept for the rationale. + The read path today takes a concrete `&IndexDb` (or a cloned handle) and calls Fjall directly. A3 introduces a trait so the layered implementation can slot in with **no behaviour change**: @@ -251,7 +263,7 @@ Each phase is its own commit, lands behind `[index] layered = false`, and keeps the live-file guard uses the **indexed** stamp from `FileEntry` instead of the row's own. - `src/index/writer.rs`: `upsert_grep_bloom(hash, row)`; `remove_file` stops deleting the now-shared row. - `src/index/mod.rs`: `sweep_grep_bloom(&live_hashes)` (drop rows no live file references), run in the - full-scan purge; bump `INDEX_PARTITION_REVISION` 6 → 7. + full-scan purge. Migration is by `ROW_VERSION` 2 with no partition-revision bump (ADR-0016 §A2). - `src/scanner_index_batch.rs`: stage by hash; `restamp_grep_bloom` / `backfill_grep_bloom` key on hash. - `src/mcp/l1_cache.rs`: add `mtime_ns` to `FileMeta`. - `src/mcp/helpers_grep.rs`, `src/mcp/index_route.rs`, `src/mcp/index_read.rs`, @@ -268,7 +280,8 @@ Each phase is its own commit, lands behind `[index] layered = false`, and keeps - `src/store_layout.rs`: `repos//base/gen-/` and `repos//overlays//` path helpers + `base.json` / `overlay.json` sidecars. -- Base builder: `ScanSource::Rev` over `origin/HEAD` into `gen-0/`, published read-only. +- Base builder: an exported tree of `origin/HEAD` scanned as a working tree into `gen-0/`, published + read-only, in the storage format the A4 prototype selects (ADR-0016 §A4). - Migration: a workspace whose only index is the legacy standalone becomes base + overlay on first layered open; `store_seed.rs` stays for now. @@ -282,13 +295,13 @@ Each phase is its own commit, lands behind `[index] layered = false`, and keeps ### A6 — generations, rebase, promotion, GC -- Generation advance (CoW clone + `diff-tree`), publication-by-rename, three-generation retention, +- Generation advance (CoW clone + `diff-tree`), publication-by-rename, two-generation retention (three when `max_group_bytes` allows; ADR-0016 *Disk budget*), overlay GC wired into `store_gc`, orphan reap. ### A7 — shared Lance tables - `documents` / `doc_links` re-keyed to `(scope, content_hash)`; queries over-fetch and filter by live - hashes. `code_chunks` is already content-addressed. + hashes. `code_chunks` is keyed `(scope, path)` and needs the same re-key (ADR-0016 §A7). - Documented deviation: top-k overlap with a per-worktree table stays ≥0.95. ### A8 — layered by default diff --git a/src/comms/daemon_forward_handlers.rs b/src/comms/daemon_forward_handlers.rs index 46e156d8..17f61bb6 100644 --- a/src/comms/daemon_forward_handlers.rs +++ b/src/comms/daemon_forward_handlers.rs @@ -218,14 +218,14 @@ pub(crate) fn index_read_via_pool( .with_workspace(root, |store| store.index_db.clone()) .map_err(|error| error.to_string())? .ok_or_else(|| "the daemon holds this workspace without a fjall index".to_string())?; - return crate::mcp::index_read::call_scan(&idx, name, *limit, cursor.as_deref()); + return crate::mcp::index_read::call_scan(&idx.view(), name, *limit, cursor.as_deref()); } - if let IndexReadQuery::GrepBloom { pattern, paths } = query { + if let IndexReadQuery::GrepBloom { pattern, entries } = query { let (idx, store_root) = pool .with_workspace(root, |store| (store.index_db.clone(), store.root.clone())) .map_err(|error| error.to_string())?; let idx = idx.ok_or_else(|| "the daemon holds this workspace without a fjall index".to_string())?; - return crate::mcp::index_read::grep_skip(&idx, &store_root, pattern, paths); + return crate::mcp::index_read::grep_skip(&idx.view(), &store_root, pattern, entries); } pool.with_workspace(root, |store| crate::mcp::index_read::index_read_against(store, query)) .map_err(|error| error.to_string())? @@ -270,19 +270,19 @@ pub(crate) fn code_search_lanes_against( use crate::comms::code_search_proto::CodeSearchLaneResult; let limit = (query.limit as usize).min(MAX_FORWARDED_LANE_LIMIT); - let Some(db) = store.index_db.as_ref() else { + let Some(db) = store.index_view() else { return Err( "the daemon holds this workspace without a fjall index, so neither the keyword nor the \ exact lane can be read" .to_string(), ); }; - let keyword = crate::search::bm25::bm25_search(db, &query.query, limit) + let keyword = crate::search::bm25::bm25_search(&db, &query.query, limit) .into_iter() .map(|hit| (hit.chunk_id, hit.score)) .collect(); let exact = if query.want_exact { - crate::search::exact::exact_lane_chunk_ids(store, db, &query.query, limit) + crate::search::exact::exact_lane_chunk_ids(store, &db, &query.query, limit) } else { Vec::new() }; diff --git a/src/comms/index_read_proto.rs b/src/comms/index_read_proto.rs index 8264dd8a..25c98b3e 100644 --- a/src/comms/index_read_proto.rs +++ b/src/comms/index_read_proto.rs @@ -50,8 +50,9 @@ pub enum IndexReadQuery { GrepBloom { /// The grep regex source. pattern: String, - /// Files to test, at most [`MAX_GREP_BLOOM_PATHS`]. - paths: Vec, + /// Files to test with the hash and stamp they were indexed under, at most + /// [`MAX_GREP_BLOOM_PATHS`]. + entries: Vec, }, /// The `implementations_by_trait` scan behind `implementations`. ImplScan { diff --git a/src/comms/workspace_pool_tests.rs b/src/comms/workspace_pool_tests.rs index 5b311ef9..88286ac4 100644 --- a/src/comms/workspace_pool_tests.rs +++ b/src/comms/workspace_pool_tests.rs @@ -6,6 +6,7 @@ use std::time::Duration; use super::*; +use crate::index::view::IndexView; /// History host for the fixtures, which care about the read stack and not about git history. /// @@ -932,7 +933,7 @@ fn a_workspace_opened_over_a_wiped_index_is_marked_and_rebuilt_by_the_next_resca assert!(!entry.rebuild_pending.load(std::sync::atomic::Ordering::Acquire)); let repopulated = pool .with_workspace(ws.path(), |s| { - !s.index_db.as_ref().expect("index").symbols_index_is_empty() && !s.index_rebuild_needed + !s.index_db.as_ref().expect("index").view().symbols_index_is_empty() && !s.index_rebuild_needed }) .expect("workspace"); assert!(repopulated, "the fjall index holds symbols again"); diff --git a/src/index/grep_bloom.rs b/src/index/grep_bloom.rs index 71cac5e6..4c8b1f1b 100644 --- a/src/index/grep_bloom.rs +++ b/src/index/grep_bloom.rs @@ -1,11 +1,12 @@ //! Per-file trigram bloom filters: the candidate PREFILTER behind `code grep` (ADR-0012). //! -//! Each indexed file gets one row in the `grep_bloom` keyspace (key = repo-relative path) holding a -//! small bloom filter over the file's byte trigrams plus the `(size, mtime_ns)` stamp the bloom was -//! built under. A query extracts the literals every match MUST contain, turns them into trigram -//! hashes and tests each row; a file the bloom rejects, whose stamp still matches the live file, -//! cannot contain a match and is never opened. A bloom can only say "definitely absent", so it -//! decides which files to READ, never what a match is: the real regex still runs on every survivor. +//! Each distinct file content gets one row in the `grep_bloom` keyspace (key = hex content hash, so +//! two paths with identical bytes share a row) holding a small bloom filter over the content's byte +//! trigrams. A query extracts the literals every match MUST contain, turns them into trigram hashes +//! and tests each row; a file whose indexed hash selects a rejecting bloom, and whose live +//! `(size, mtime_ns)` still equals the stamp it was indexed under, cannot contain a match and is +//! never opened. A bloom can only say "definitely absent", so it decides which files to READ, +//! never what a match is: the real regex still runs on every survivor. //! //! Everything here errs towards "candidate": a missing, older-version or malformed row, a changed //! stamp, a pattern with no usable literal, an unreadable file. Only an intact row that was built @@ -17,13 +18,15 @@ use rayon::prelude::*; use regex_syntax::hir::literal::{ExtractKind, Extractor}; use super::IndexDb; +use super::view::{IndexBytes, IndexView}; use crate::path::RelPath; -/// Row layout revision (first byte). A row with any other value is treated as absent, so a future -/// layout change needs no index wipe: old rows read as candidates until the file is next scanned. -const ROW_VERSION: u8 = 1; -/// `version(1) | size(8) | mtime_ns(8)`, then the bit array. -const HEADER_BYTES: usize = 17; +/// Row layout revision (first byte). A row with any other value is treated as absent, so a layout +/// change needs no index wipe: old rows read as candidates until a scan sweeps and rebuilds them. +/// 1 was the path-keyed row carrying a `(size, mtime_ns)` stamp; 2 is the content-hash-keyed row. +const ROW_VERSION: u8 = 2; +/// `version(1)`, then the bit array. +const HEADER_BYTES: usize = 1; /// Bloom bytes per source byte is `1 / BYTES_DIVISOR` (ADR-0012 sizing: 1/8 keeps the candidate set /// within ~2x of the exact trigram answer on the armis corpus at 12.8% of corpus bytes; 1/16 was /// up to ~8x worse on selective needles). @@ -38,6 +41,11 @@ const HASHES: u64 = 2; /// A literal shorter than this has no trigram to test. const MIN_LITERAL_BYTES: usize = 3; +/// `meta` row recording the [`ROW_VERSION`] the keyspace was last version-swept under. Present and +/// equal means every row is current, so a sweep need not read values; set only after a sweep that +/// read them all succeeded. +const META_ROW_SWEPT_VER: &[u8] = b"grep_bloom_swept_ver"; + #[inline] fn mix(trigram: u32) -> u64 { let mut x = u64::from(trigram).wrapping_mul(0x9E37_79B9_7F4A_7C15); @@ -59,15 +67,12 @@ fn trigram(w: &[u8]) -> u32 { (u32::from(w[0]) << 16) | (u32::from(w[1]) << 8) | u32::from(w[2]) } -/// Build the stored row for a file whose bytes the caller already holds, stamped with the -/// `(size, mtime_ns)` it was read under. -pub fn build_row(bytes: &[u8], size: u64, mtime_ns: i64) -> Vec { +/// Build the stored row for content the caller already holds. +pub fn build_row(bytes: &[u8]) -> Vec { let bloom_bytes = (bytes.len() / BYTES_DIVISOR).clamp(MIN_BLOOM_BYTES, MAX_BLOOM_BYTES); let nbits = (bloom_bytes * 8) as u64; let mut row = vec![0u8; HEADER_BYTES + bloom_bytes]; row[0] = ROW_VERSION; - row[1..9].copy_from_slice(&size.to_le_bytes()); - row[9..17].copy_from_slice(&mtime_ns.to_le_bytes()); let bits = &mut row[HEADER_BYTES..]; for w in bytes.windows(3) { let mixed = mix(trigram(w)); @@ -79,31 +84,23 @@ pub fn build_row(bytes: &[u8], size: u64, mtime_ns: i64) -> Vec { row } -/// A decoded row: the stamp it was built under and its bit array. -struct Row<'a> { - size: u64, - mtime_ns: i64, - bits: &'a [u8], +fn parse_row(row: &[u8]) -> Option<&[u8]> { + (row.len() > HEADER_BYTES && row[0] == ROW_VERSION).then(|| &row[HEADER_BYTES..]) } -fn parse_row(row: &[u8]) -> Option> { - if row.len() <= HEADER_BYTES || row[0] != ROW_VERSION { - return None; - } - Some(Row { - size: u64::from_le_bytes(row[1..9].try_into().ok()?), - mtime_ns: i64::from_le_bytes(row[9..17].try_into().ok()?), - bits: &row[HEADER_BYTES..], - }) -} +pub use crate::scanner_file::mtime_nanos; -/// The `(size, mtime_ns)` stamp a stored row was built under, or `None` for an absent / foreign row. -pub fn row_stamp(row: &[u8]) -> Option<(u64, i64)> { - parse_row(row).map(|r| (r.size, r.mtime_ns)) +/// One file a grep may test: its path, the content hash it was indexed under, and the +/// `(size, mtime_ns)` stamp of that indexing. The stamp is what the live file is held to, so a file +/// edited since the scan is never skipped on the strength of its old content's bloom. +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct GrepEntry { + pub rel: RelPath, + pub hash_hex: String, + pub size_bytes: u64, + pub mtime_ns: i64, } -pub use crate::scanner_file::mtime_nanos; - /// What a regex requires of a file, as trigram hashes: every group must pass, and a group passes when /// ANY of its alternatives has ALL its trigrams in the bloom. #[derive(Debug)] @@ -166,121 +163,107 @@ impl Needle { } impl IndexDb { - /// Fetch the stored bloom row for `rel`, if any. - pub fn grep_bloom_row(&self, rel: &RelPath) -> Option { - self.grep_bloom.get(rel.as_bytes()).ok().flatten() - } - - /// Delete `rel`'s row without touching the rest of its index entries: how a test models an index - /// built before the keyspace existed. + /// Delete the row for `hash_hex`: how a test models an index built before the keyspace existed. #[doc(hidden)] - pub fn drop_grep_bloom_row_for_test(&self, rel: &RelPath) { - self.grep_bloom.remove(rel.as_bytes()).expect("remove bloom row"); - } - - /// True when `rel` has an intact row stamped `(size, mtime_ns)`: nothing to (re)build. - pub fn grep_bloom_current(&self, rel: &RelPath, size: u64, mtime_ns: i64) -> bool { - self.grep_bloom_row(rel) - .and_then(|row| row_stamp(&row)) - .is_some_and(|stamp| stamp == (size, mtime_ns)) - } - - /// For each of `paths`, whether the file can be SKIPPED by a grep for `needle`: it has an intact - /// row, the row rejects the needle, and the live file still has the stamp the row was built - /// under. Rows are fetched and dropped one at a time, in parallel; nothing is held between files. - pub fn grep_skip_verdicts(&self, root: &Path, paths: &[&RelPath], needle: &Needle) -> Vec { - paths - .par_iter() - .map(|rel| { - let Some(row) = self.grep_bloom_row(rel) else { - return false; - }; - let Some(row) = parse_row(&row) else { - return false; - }; - if needle.passes(row.bits) { - return false; - } - // The bloom describes the file as scanned; the sweep reads it as it is now. Only a - // file still carrying the scan-time stamp may be trusted to lack the literal. - std::fs::metadata(root.join(rel.to_path_buf())) - .is_ok_and(|m| m.len() == row.size && mtime_nanos(&m) == row.mtime_ns) - }) - .collect() - } -} - -#[cfg(test)] -mod tests { - use super::*; - - fn row_for(text: &str) -> Vec { - build_row(text.as_bytes(), text.len() as u64, 7) + #[cfg(any(test, feature = "test-support"))] + pub fn drop_grep_bloom_row_for_test(&self, hash_hex: &str) { + self.grep_bloom.remove(hash_hex.as_bytes()).expect("remove bloom row"); } - fn rejects(pattern: &str, text: &str) -> bool { - let needle = Needle::compile(pattern).expect("usable needle"); - let row = row_for(text); - !needle.passes(parse_row(&row).expect("row").bits) - } - - #[test] - fn a_present_literal_is_never_rejected() { - let text = "fn main() { let post_processing = 1; }\n".repeat(50); - assert!(!rejects("post_processing", &text)); - assert!(!rejects(r"post_processing\s*=", &text)); - assert!(!rejects("(?i)POST_PROCESSING", &text)); - assert!(!rejects("zzz_absent|post_processing", &text)); + /// Number of stored rows, one per distinct content. + #[doc(hidden)] + #[cfg(any(test, feature = "test-support"))] + pub fn grep_bloom_row_count(&self) -> usize { + self.grep_bloom.iter().count() } - #[test] - fn an_absent_literal_is_rejected() { - assert!(rejects("eq9fsw", "fn main() { println!(\"hello\"); }")); + /// True when `hash_hex` has an intact row: nothing to (re)build. + pub fn grep_bloom_current(&self, hash_hex: &str) -> bool { + self.view() + .grep_bloom_row(hash_hex) + .is_some_and(|row| parse_row(&row).is_some()) } - #[test] - fn patterns_without_a_required_trigram_sweep_everything() { - for pattern in [".*", r"\w+", "ab", "a|bcd", "(?i)k", "^", r"\d{3}"] { - assert!(Needle::compile(pattern).is_none(), "{pattern} must not be prefiltered"); + /// Delete every row whose hash is not in `live`, returning how many were dropped. Rows are shared + /// by every path holding the content, so only a hash no indexed file references may go. Path-keyed + /// rows from before the content-hash re-key are never in `live`, so they leave here too. + /// + /// Row values are read only until the keyspace has been swept once under the current + /// [`ROW_VERSION`] (recorded in `meta`): that pass also drops rows whose layout is older, which + /// are unreadable. Afterwards every row is current, so the sweep is key-only and a large index + /// is not read back in full on each scan. + pub fn sweep_grep_bloom(&self, live: &ahash::AHashSet<&str>) -> Result { + let versioned = self + .meta + .get(META_ROW_SWEPT_VER)? + .is_some_and(|v| v.first() == Some(&ROW_VERSION)); + let mut writer = self.writer(); + let mut dropped = 0; + for guard in self.grep_bloom.iter() { + let (key, current) = if versioned { + (guard.key()?, true) + } else { + let (key, row) = guard.into_inner()?; + (key, parse_row(&row).is_some()) + }; + if current && std::str::from_utf8(&key).is_ok_and(|hash| live.contains(hash)) { + continue; + } + writer.remove_grep_bloom(&key); + dropped += 1; } - } - - #[test] - fn case_insensitive_literals_match_every_casing() { - let needle = Needle::compile("(?i)needle").expect("expanded to case variants"); - for text in ["xx NEEDLE xx", "xx needle xx", "xx NeEdLe xx"] { - assert!(needle.passes(parse_row(&row_for(text)).unwrap().bits), "{text}"); + writer.commit()?; + if !versioned { + self.meta.insert(META_ROW_SWEPT_VER, [ROW_VERSION])?; } + Ok(dropped) } - #[test] - fn unicode_literals_are_matched_on_their_utf8_bytes() { - let needle = Needle::compile("caf\u{e9}s").expect("literal"); - assert!(needle.passes(parse_row(&row_for("two caf\u{e9}s here")).unwrap().bits)); - } - - #[test] - fn foreign_or_short_rows_do_not_parse() { - let mut row = row_for("hello world"); - assert!(parse_row(&row).is_some()); - row[0] = ROW_VERSION + 1; - assert!(parse_row(&row).is_none()); - assert!(parse_row(&[ROW_VERSION; HEADER_BYTES]).is_none()); - assert!(parse_row(&[]).is_none()); - } - - #[test] - fn the_stamp_round_trips() { - let row = build_row(b"abcdef", 6, -12); - assert_eq!(row_stamp(&row), Some((6, -12))); + /// Delete the rows of `hashes`, none of which any indexed file references now. The incremental + /// counterpart of [`Self::sweep_grep_bloom`]: cost is the number of hashes, not the keyspace. + pub fn remove_grep_bloom_rows<'a>( + &self, + hashes: impl IntoIterator, + ) -> Result<(), super::IndexError> { + let mut writer = self.writer(); + for hash in hashes { + writer.remove_grep_bloom(hash.as_bytes()); + } + writer.commit() } +} - #[test] - fn row_size_tracks_the_file_within_its_bounds() { - assert_eq!(build_row(b"abc", 3, 0).len(), HEADER_BYTES + MIN_BLOOM_BYTES); - let big = vec![b'x'; 8 * 1024]; - assert_eq!(build_row(&big, 8192, 0).len(), HEADER_BYTES + 1024); - let huge = vec![b'x'; 4 * 1024 * 1024]; - assert_eq!(build_row(&huge, 0, 0).len(), HEADER_BYTES + MAX_BLOOM_BYTES); - } +/// The verdict logic behind [`IndexView::grep_skip_verdicts`], over any row source. +pub(super) fn skip_verdicts( + row_of: impl Fn(&str) -> Option + Sync, + root: &Path, + entries: &[GrepEntry], + needle: &Needle, +) -> Vec { + entries + .par_iter() + .map(|entry| { + let Some(row) = row_of(&entry.hash_hex) else { + return false; + }; + let Some(bits) = parse_row(&row) else { + return false; + }; + if needle.passes(bits) { + return false; + } + // The bloom describes the content as indexed; the sweep reads the file as it is now. + // Only a file still carrying its indexed stamp may be trusted to lack the literal. A + // stamp of 0 is "unknown" (a git source, or an mtime too close to the clock when it was + // taken, see `scanner_file::settled_mtime`), and an unavailable live mtime also reads 0, + // so it must never count as a match. + entry.mtime_ns != 0 + && std::fs::metadata(root.join(entry.rel.to_path_buf())) + .is_ok_and(|m| m.len() == entry.size_bytes && mtime_nanos(&m) == entry.mtime_ns) + }) + .collect() } + +#[cfg(test)] +#[path = "grep_bloom_tests.rs"] +mod tests; diff --git a/src/index/grep_bloom_tests.rs b/src/index/grep_bloom_tests.rs new file mode 100644 index 00000000..b5c31eb0 --- /dev/null +++ b/src/index/grep_bloom_tests.rs @@ -0,0 +1,337 @@ +use super::*; + +fn row_for(text: &str) -> Vec { + build_row(text.as_bytes()) +} + +fn rejects(pattern: &str, text: &str) -> bool { + let needle = Needle::compile(pattern).expect("usable needle"); + let row = row_for(text); + !needle.passes(parse_row(&row).expect("row")) +} + +#[test] +fn a_present_literal_is_never_rejected() { + let text = "fn main() { let post_processing = 1; }\n".repeat(50); + assert!(!rejects("post_processing", &text)); + assert!(!rejects(r"post_processing\s*=", &text)); + assert!(!rejects("(?i)POST_PROCESSING", &text)); + assert!(!rejects("zzz_absent|post_processing", &text)); +} + +#[test] +fn an_absent_literal_is_rejected() { + assert!(rejects("eq9fsw", "fn main() { println!(\"hello\"); }")); +} + +#[test] +fn patterns_without_a_required_trigram_sweep_everything() { + for pattern in [".*", r"\w+", "ab", "a|bcd", "(?i)k", "^", r"\d{3}"] { + assert!(Needle::compile(pattern).is_none(), "{pattern} must not be prefiltered"); + } +} + +#[test] +fn case_insensitive_literals_match_every_casing() { + let needle = Needle::compile("(?i)needle").expect("expanded to case variants"); + for text in ["xx NEEDLE xx", "xx needle xx", "xx NeEdLe xx"] { + assert!(needle.passes(parse_row(&row_for(text)).unwrap()), "{text}"); + } +} + +#[test] +fn unicode_literals_are_matched_on_their_utf8_bytes() { + let needle = Needle::compile("caf\u{e9}s").expect("literal"); + assert!(needle.passes(parse_row(&row_for("two caf\u{e9}s here")).unwrap())); +} + +#[test] +fn foreign_or_short_rows_do_not_parse() { + let mut row = row_for("hello world"); + assert!(parse_row(&row).is_some()); + row[0] = ROW_VERSION + 1; + assert!(parse_row(&row).is_none()); + assert!(parse_row(&[ROW_VERSION; HEADER_BYTES]).is_none()); + assert!(parse_row(&[]).is_none()); +} + +#[test] +fn row_size_tracks_the_file_within_its_bounds() { + assert_eq!(build_row(b"abc").len(), HEADER_BYTES + MIN_BLOOM_BYTES); + let big = vec![b'x'; 8 * 1024]; + assert_eq!(build_row(&big).len(), HEADER_BYTES + 1024); + let huge = vec![b'x'; 4 * 1024 * 1024]; + assert_eq!(build_row(&huge).len(), HEADER_BYTES + MAX_BLOOM_BYTES); +} + +fn open_db() -> (tempfile::TempDir, IndexDb) { + let dir = tempfile::tempdir().expect("tempdir"); + let db = IndexDb::open(dir.path()).expect("index"); + (dir, db) +} + +fn put(db: &IndexDb, hash_hex: &str, text: &str) { + let mut writer = db.writer(); + writer.upsert_grep_bloom(hash_hex, row_for(text)); + writer.commit().expect("commit"); +} + +#[test] +fn a_row_round_trips_under_its_content_hash() { + let (_dir, db) = open_db(); + put(&db, "aa11", "fn alpha() {}"); + assert_eq!( + db.view().grep_bloom_row("aa11").as_deref(), + Some(row_for("fn alpha() {}").as_slice()) + ); + assert!(db.grep_bloom_current("aa11")); + assert!(!db.grep_bloom_current("bb22")); +} + +#[test] +fn a_hash_that_prefixes_another_reads_only_its_own_row() { + let (_dir, db) = open_db(); + put(&db, "ab", "short key"); + put(&db, "abc", "longer key"); + assert_eq!( + db.view().grep_bloom_row("ab").as_deref(), + Some(row_for("short key").as_slice()) + ); + assert_eq!( + db.view().grep_bloom_row("abc").as_deref(), + Some(row_for("longer key").as_slice()) + ); + assert!(db.view().grep_bloom_row("a").is_none()); +} + +#[test] +fn the_sweep_drops_only_hashes_no_live_file_references() { + let (_dir, db) = open_db(); + for hash in ["h1", "h2", "h3"] { + put(&db, hash, hash); + } + let live: ahash::AHashSet<&str> = ["h1", "h3"].into_iter().collect(); + assert_eq!(db.sweep_grep_bloom(&live).expect("sweep"), 1); + assert!(db.grep_bloom_current("h1") && db.grep_bloom_current("h3")); + assert!(!db.grep_bloom_current("h2")); + assert_eq!( + db.sweep_grep_bloom(&live).expect("sweep"), + 0, + "a second sweep is a no-op" + ); +} + +#[test] +fn two_paths_with_the_same_content_share_one_row_and_both_skip() { + let (_dir, db) = open_db(); + let root = tempfile::tempdir().expect("root"); + let text = "fn unrelated() { println!(\"hello\"); }\n"; + let mut entries = Vec::new(); + for name in ["a.rs", "b.rs"] { + std::fs::write(root.path().join(name), text).expect("write"); + let meta = std::fs::metadata(root.path().join(name)).expect("stat"); + entries.push(GrepEntry { + rel: RelPath::from(name), + hash_hex: "same".to_owned(), + size_bytes: meta.len(), + mtime_ns: mtime_nanos(&meta), + }); + } + put(&db, "same", text); + put(&db, "same", text); + assert_eq!(db.grep_bloom_row_count(), 1); + let needle = Needle::compile("zzz_absent_literal").expect("needle"); + assert_eq!( + db.view().grep_skip_verdicts(root.path(), &entries, &needle), + vec![true, true] + ); +} + +#[test] +fn a_live_file_that_moved_off_its_indexed_stamp_is_a_candidate() { + let (_dir, db) = open_db(); + let root = tempfile::tempdir().expect("root"); + std::fs::write(root.path().join("a.rs"), "fn unrelated() {}\n").expect("write"); + let meta = std::fs::metadata(root.path().join("a.rs")).expect("stat"); + put(&db, "h", "fn unrelated() {}\n"); + let needle = Needle::compile("zzz_absent_literal").expect("needle"); + let entry = |size_bytes| GrepEntry { + rel: RelPath::from("a.rs"), + hash_hex: "h".to_owned(), + size_bytes, + mtime_ns: mtime_nanos(&meta), + }; + assert_eq!( + db.view().grep_skip_verdicts(root.path(), &[entry(meta.len())], &needle), + vec![true] + ); + assert_eq!( + db.view() + .grep_skip_verdicts(root.path(), &[entry(meta.len() + 1)], &needle), + vec![false] + ); +} + +/// Plant a row under `hash` whose first byte is not the current layout, as an older build left it. +fn put_stale_layout(db: &IndexDb, hash: &str) { + let mut row = row_for(hash); + row[0] = ROW_VERSION - 1; + let mut writer = db.writer(); + writer.upsert_grep_bloom(hash, row); + writer.commit().expect("commit"); +} + +#[test] +fn the_first_sweep_drops_older_layout_rows_and_records_it_in_meta() { + let (_dir, db) = open_db(); + put(&db, "h1", "current"); + put_stale_layout(&db, "h2"); + let live: ahash::AHashSet<&str> = ["h1", "h2"].into_iter().collect(); + assert!(db.meta.get(META_ROW_SWEPT_VER).expect("meta").is_none()); + assert_eq!( + db.sweep_grep_bloom(&live).expect("sweep"), + 1, + "a live hash with an unreadable row goes" + ); + assert!(db.grep_bloom_current("h1") && db.view().grep_bloom_row("h2").is_none()); + assert_eq!( + db.meta.get(META_ROW_SWEPT_VER).expect("meta").as_deref(), + Some([ROW_VERSION].as_slice()) + ); +} + +#[test] +fn later_sweeps_read_keys_only_and_still_remove_dead_keys() { + let (_dir, db) = open_db(); + let live: ahash::AHashSet<&str> = ["h1", "h2"].into_iter().collect(); + db.sweep_grep_bloom(&live).expect("first sweep"); + + put(&db, "h1", "current"); + put_stale_layout(&db, "h2"); + put(&db, "dead", "no file holds this"); + assert_eq!(db.sweep_grep_bloom(&live).expect("steady sweep"), 1); + assert!( + db.view().grep_bloom_row("dead").is_none(), + "a dead key is removed without reading it" + ); + assert!( + db.view().grep_bloom_row("h2").is_some(), + "the value of a live key is not decoded after the version sweep ran" + ); +} + +#[test] +fn without_the_marker_the_version_sweep_reads_values_again() { + let (_dir, db) = open_db(); + let live: ahash::AHashSet<&str> = ["h1"].into_iter().collect(); + db.sweep_grep_bloom(&live).expect("first sweep"); + put_stale_layout(&db, "h1"); + assert_eq!( + db.sweep_grep_bloom(&live).expect("steady sweep"), + 0, + "marker present: values unread" + ); + db.meta.remove(META_ROW_SWEPT_VER).expect("forget the marker"); + assert_eq!(db.sweep_grep_bloom(&live).expect("version sweep"), 1); +} + +#[test] +fn remove_grep_bloom_rows_drops_only_the_named_hashes() { + let (_dir, db) = open_db(); + for hash in ["h1", "h2", "h3"] { + put(&db, hash, hash); + } + db.remove_grep_bloom_rows(["h1", "h3", "never-indexed"]) + .expect("remove"); + assert!(db.grep_bloom_current("h2")); + assert!(db.view().grep_bloom_row("h1").is_none() && db.view().grep_bloom_row("h3").is_none()); +} + +fn skip_for(db: &IndexDb, root: &Path, entry: GrepEntry) -> bool { + let needle = Needle::compile("zzz_absent_literal").expect("needle"); + db.view().grep_skip_verdicts(root, &[entry], &needle)[0] +} + +#[test] +fn an_unknown_indexed_stamp_is_never_trusted_even_when_the_live_stamp_is_also_unknown() { + let (_dir, db) = open_db(); + let root = tempfile::tempdir().expect("root"); + let path = root.path().join("a.rs"); + std::fs::write(&path, "fn unrelated() {}\n").expect("write"); + std::fs::File::options() + .write(true) + .open(&path) + .and_then(|f| f.set_modified(std::time::UNIX_EPOCH)) + .expect("set mtime to the epoch"); + let meta = std::fs::metadata(&path).expect("stat"); + assert_eq!(mtime_nanos(&meta), 0, "the live stamp reads as 0"); + put(&db, "h", "fn unrelated() {}\n"); + let entry = GrepEntry { + rel: RelPath::from("a.rs"), + hash_hex: "h".to_owned(), + size_bytes: meta.len(), + mtime_ns: 0, + }; + assert!(!skip_for(&db, root.path(), entry), "0 == 0 is not a match"); +} + +#[test] +fn settled_mtime_zeroes_a_stamp_within_two_seconds_of_the_clock_either_side() { + use crate::scanner_file::settled_mtime; + let now = 10_000_000_000_i64; + assert_eq!(settled_mtime(now, now), 0); + assert_eq!(settled_mtime(now - 1_999_999_999, now), 0); + assert_eq!( + settled_mtime(now + 1_999_999_999, now), + 0, + "a slightly future stamp is untrusted too" + ); + assert_eq!(settled_mtime(now - 2_000_000_000, now), now - 2_000_000_000); + assert_eq!(settled_mtime(now + 60_000_000_000, now), now + 60_000_000_000); + assert_eq!(settled_mtime(0, now), 0); +} + +/// Every text `pattern` matches must pass the needle; `usable` also pins that the pattern is +/// prefiltered at all, so the loop cannot pass by the needle quietly not existing. +fn never_false_skips(pattern: &str, usable: bool, texts: &[&str]) { + let re = regex::Regex::new(pattern).expect("regex"); + let needle = Needle::compile(pattern); + assert_eq!(needle.is_some(), usable, "{pattern}: usable"); + let Some(needle) = needle else { return }; + let mut matched = 0; + for text in texts.iter().filter(|text| re.is_match(text)) { + matched += 1; + assert!( + needle.passes(parse_row(&row_for(text)).expect("row")), + "{pattern} matches {text:?} but the bloom rejects it" + ); + } + assert!(matched > 0, "{pattern}: no sample matched, the check proved nothing"); +} + +#[test] +fn mixed_length_alternations_never_false_skip() { + for (pattern, usable, texts) in [ + ("a|bcd", false, &["a", "bcd", "xx a xx"][..]), + ("abc|d", false, &["d", "abc"][..]), + ("foo|ba", false, &["ba", "foo"][..]), + ("(abc|d)ef", true, &["abcef", "def", "xx def"][..]), + ("foobar|bazqux", true, &["foobar", "bazqux"][..]), + ("x(abc|de)y", true, &["xabcy", "xdey"][..]), + ] { + never_false_skips(pattern, usable, texts); + } +} + +#[test] +fn word_boundary_literals_never_false_skip() { + never_false_skips(r"\bfoo\b", true, &["foo", "a foo b", "(foo)"]); + never_false_skips(r"\bfoobar\w*", true, &["foobar", "x foobarbaz"]); +} + +#[test] +fn unicode_case_folding_never_false_skips() { + never_false_skips("(?i)kit", true, &["KIT", "kit", "\u{212A}it", "\u{212A}IT"]); + never_false_skips("(?i)strasse", true, &["STRASSE", "\u{17F}trasse", "Stra\u{17F}se"]); + never_false_skips("(?i)caf\u{e9}s", true, &["CAF\u{c9}S", "caf\u{e9}s"]); +} diff --git a/src/index/mod.rs b/src/index/mod.rs index a1ce038d..7dbec0de 100644 --- a/src/index/mod.rs +++ b/src/index/mod.rs @@ -24,6 +24,9 @@ pub mod keys_governance; pub mod keys_probe; pub(crate) mod memory_store; pub(crate) mod name_dict; +pub mod view; +#[cfg(test)] +mod view_tests; pub mod writer; use std::path::{Path, PathBuf}; @@ -288,55 +291,56 @@ pub struct IndexDb { pub(crate) db: Database, /// Carries the `schema_ver` row read + stamped in [`IndexDb::open`]; also read and written /// by the BM25 corpus-stats recompute. - pub(crate) meta: Keyspace, - pub(crate) symbols_by_path: Keyspace, + pub(in crate::index) meta: Keyspace, + pub(in crate::index) symbols_by_path: Keyspace, /// Reserved fast-path partition: written on every upsert so that future name-based /// symbol search can skip the in-RAM linear scan. Not yet read by any MCP query path; /// kept to avoid a schema migration when the read path lands. - pub(crate) symbols_by_name: Keyspace, - pub(crate) calls_by_path: Keyspace, - pub(crate) calls_by_callee: Keyspace, + pub(in crate::index) symbols_by_name: Keyspace, + pub(in crate::index) calls_by_path: Keyspace, + pub(in crate::index) calls_by_callee: Keyspace, /// Distinct callee names of `calls_by_callee`; lets the substring name scan range-scan only the /// matching names instead of walking the partition. - pub(crate) callee_names: std::sync::Arc, + pub(in crate::index) callee_names: std::sync::Arc, /// Reserved fast-path partition: written on every upsert so that future /// `dependents`-by-module queries can use a prefix scan instead of iterating the /// full import set. Not yet read by any MCP query path; kept to avoid a schema /// migration when the read path lands. - pub(crate) imports_by_module: Keyspace, - pub(crate) imports_by_path: Keyspace, + pub(in crate::index) imports_by_module: Keyspace, + pub(in crate::index) imports_by_path: Keyspace, /// `implementations_by_trait`: prefix scans on trait name — backs `find_implementations`. - pub(crate) implementations_by_trait: Keyspace, + pub(in crate::index) implementations_by_trait: Keyspace, /// Distinct trait names of `implementations_by_trait`, as [`Self::callee_names`] is for callees. - pub(crate) trait_names: std::sync::Arc, + pub(in crate::index) trait_names: std::sync::Arc, /// `implementations_by_path`: companion to keep the per-file delete on upsert O(prefix). - pub(crate) implementations_by_path: Keyspace, + pub(in crate::index) implementations_by_path: Keyspace, /// `refs_by_def`: scope/import-resolved reference edges keyed by defining site — backs the /// resolved `find_references` / `find_callers`. Written by the scanner's resolve pass (B3). - pub(crate) refs_by_def: Keyspace, + pub(in crate::index) refs_by_def: Keyspace, /// `refs_by_path`: companion keyed by the use file — O(prefix) delete on re-resolve and the /// forward lookup behind `goto_definition`. Written by the scanner's resolve pass (B3). - pub(crate) refs_by_path: Keyspace, + pub(in crate::index) refs_by_path: Keyspace, /// `import_targets`: importer + import-binding offset -> the definition the stitch bound it to, /// plus one reserved probe-manifest row per importer (see [`keys_probe`]). - pub(crate) import_targets: Keyspace, + pub(in crate::index) import_targets: Keyspace, /// `importers_by_probe`: probed path hash + importer. Every path the resolver consulted for an /// importer's imports, so the incremental re-stitch finds exactly the importers a changed, /// added or deleted path can affect. - pub(crate) importers_by_probe: Keyspace, + pub(in crate::index) importers_by_probe: Keyspace, /// `code_bm25_postings`: term → chunks (with inlined tf + doclen) for the code-search BM25 /// keyword lane. Always created for DB stability; read + written only under the `code-search` /// feature (hence `dead_code`-allowed on a default build, mirroring `embeddings`). #[allow(dead_code)] - pub(crate) code_bm25_postings: Keyspace, + pub(in crate::index) code_bm25_postings: Keyspace, /// `code_bm25_by_path`: forward companion keyed by file → its chunks' `(chunk_id, doclen, /// terms)`, so a re-scan deletes the previous postings in O(prefix). Always created. - pub(crate) code_bm25_by_path: Keyspace, - /// `grep_bloom`: file path -> per-file trigram bloom filter + the `(size, mtime_ns)` stamp it was - /// built under (see [`grep_bloom`]). A candidate PREFILTER for `code grep` only; a missing or - /// stale row means "candidate", which is why this partition was added without a schema bump and - /// backfills on rescan. - pub(crate) grep_bloom: Keyspace, + pub(in crate::index) code_bm25_by_path: Keyspace, + /// `grep_bloom`: hex content hash -> trigram bloom filter of that content (see [`grep_bloom`]), + /// shared by every path holding the bytes. A candidate PREFILTER for `code grep` only; a missing + /// row (including an older-version or path-keyed one) means "candidate", which is why the layout + /// moved on by `ROW_VERSION` with no schema bump: the scan's sweep drops stale rows and + /// backfills the current ones. + pub(in crate::index) grep_bloom: Keyspace, #[allow(dead_code)] pub(crate) embeddings: Keyspace, /// `memory_by_key`: scope + key → msgpack `MemoryRecord`. Lives in the per-repo-group @@ -644,44 +648,6 @@ impl IndexDb { writer::IndexWriter::new(self.clone()) } - /// True when the secondary index holds no per-file symbol entries. Cheap — peeks at - /// the first key of `symbols_by_path` rather than counting. - /// - /// Used by the MCP startup auto-scan to detect a present-but-empty Fjall index (e.g. a - /// `views//index.fjall/` that was wiped or removed out-of-band while the msgpack - /// `index.msgpack` survived). In that state the in-RAM map cache looks populated but the - /// Fjall-backed tools (`find_references` / `search_symbols`) would silently return nothing, - /// so a rescan is warranted even though the RAM cache is non-empty. - pub fn symbols_index_is_empty(&self) -> bool { - self.symbols_by_path.iter().next().is_none() - } - - /// Resolved references to the definition at `(def_path, def_start)` — the scope/import-resolved - /// backing for `find_references`. Returns each binding `(use_path, use_start)`; empty when the - /// definition has no resolved uses (or resolution never ran for its file). - pub fn references_to(&self, def_path: &crate::path::RelPath, def_start: u32) -> Vec<(crate::path::RelPath, u32)> { - let prefix = keys::refs_by_def_prefix(def_path, def_start); - let mut out = Vec::new(); - for guard in self.refs_by_def.prefix(prefix) { - if let Ok((k, _)) = guard.into_inner() - && let Some((_def_path, _def_start, use_path, use_start)) = keys::parse_ref_by_def(&k) - { - out.push((use_path, use_start)); - } - } - out - } - - /// Whether the probe keyspaces cover every importer (a wholesale resolve pass completed since - /// they were created). When false, the incremental pass must run the wholesale one first. - pub fn import_probes_complete(&self) -> bool { - self.meta - .get(META_IMPORT_PROBE_VER) - .ok() - .flatten() - .is_some_and(|v| v.first() == Some(&IMPORT_PROBE_VER)) - } - /// Mark the probe keyspaces complete (`true`) or in flux (`false`). The wholesale pass clears /// the stamp before it rewrites any importer and sets it only after the stitch committed. pub fn set_import_probes_complete(&self, complete: bool) -> Result<(), IndexError> { @@ -693,37 +659,6 @@ impl IndexDb { Ok(()) } - /// Every importer whose resolution consulted `path` (repo-relative, forward-slashed) or one of - /// its ancestor directories: a prefix scan of `importers_by_probe` per candidate, O(importers). - /// - /// The ancestors matter because the resolver does not descend into a directory it found - /// missing. An importer that probed `util` and found nothing never probed `util/index.ts`, so - /// adding `util/index.ts` must reach it through the directory. (A directory that existed when - /// the importer resolved is never recorded, so a path under an existing directory costs only - /// empty scans.) - pub fn importers_probing(&self, path: &str) -> Vec { - probe_candidates(path) - .flat_map(|candidate| { - let prefix = keys::importers_by_probe_prefix(&keys::probe_hash(candidate)); - self.importers_by_probe - .prefix(prefix) - .filter_map(|guard| guard.into_inner().ok()) - .filter_map(|(k, _)| keys::parse_importer_by_probe(&k)) - .collect::>() - }) - .collect() - } - - /// Whether any importer consulted `path` or an ancestor directory of it (see - /// [`Self::importers_probing`]). Cheap enough to gate a rescan on a non-indexed file - /// (`tsconfig.json`, `package.json`) the watcher saw change. - pub fn is_probed(&self, path: &str) -> bool { - probe_candidates(path).any(|candidate| { - let prefix = keys::importers_by_probe_prefix(&keys::probe_hash(candidate)); - self.importers_by_probe.prefix(prefix).next().is_some() - }) - } - /// Every row of the four resolution keyspaces (`refs_by_def`, `refs_by_path`, `import_targets`, /// `importers_by_probe`) as `(keyspace, key, value)`, in keyspace then key order. Lets an /// equivalence test compare an incremental rescan with a full one row for row. @@ -744,91 +679,6 @@ impl IndexDb { Ok(rows) } - /// The stitched `(local_start, def_path, def_start)` bindings of `importer`. - pub fn import_targets_of(&self, importer: &crate::path::RelPath) -> Vec<(u32, crate::path::RelPath, u32)> { - self.import_targets - .prefix(keys::import_targets_prefix(importer)) - .filter_map(|guard| guard.into_inner().ok()) - .filter_map(|(k, v)| keys::parse_import_target(&k, &v)) - .map(|(_, local_start, def_path, def_start)| (local_start, def_path, def_start)) - .collect() - } - - /// The definition the use at `(use_path, use_start)` binds to — backs `goto_definition`. - /// Cross-file definitions take precedence over the local import binding that an imported use - /// also resolves through. `None` when the position isn't a resolved reference. - pub fn definition_of( - &self, - use_path: &crate::path::RelPath, - use_start: u32, - ) -> Option<(crate::path::RelPath, u32)> { - let prefix = keys::refs_by_use_prefix(use_path, use_start); - let mut local = None; - for guard in self.refs_by_path.prefix(prefix) { - if let Ok((k, _)) = guard.into_inner() - && let Some((_use_path, _use_start, def_path, def_start)) = keys::parse_ref_by_path(&k) - { - if def_path != *use_path { - return Some((def_path, def_start)); - } - local = Some((def_path, def_start)); - } - } - local - } - - /// Symbols whose name starts with `name`, from the `symbols_by_name` keyspace — an index-backed - /// prefix scan (length-prefixed keys isolate `Foo` from `Foobar`). Returns `(name, kind, path, - /// start_byte)` for each match, capped at `cap` entries. Backs the code-search **exact lane**: - /// an identifier-shaped query resolves to the symbols that define it, which then map to their - /// owning chunks. The returned `name` lets the caller rank exact-name matches ahead of longer - /// prefix matches. `start_byte` is the L1 `Symbol.start_byte` (node start), which falls inside - /// the symbol's owning chunk span. - pub fn symbols_by_name_lookup( - &self, - name: &str, - cap: usize, - ) -> Vec<(String, crate::extract::SymbolKind, crate::path::RelPath, u32)> { - let prefix = keys::symbols_by_name_prefix(name); - let mut out = Vec::new(); - for guard in self.symbols_by_name.prefix(prefix) { - if out.len() >= cap { - break; - } - if let Ok((k, _)) = guard.into_inner() - && let Some((matched, kind, rel, start_byte)) = keys::parse_symbol_by_name(&k) - { - out.push((matched, kind, rel, start_byte)); - } - } - out - } - - /// Corpus-global BM25 stats for the code-search keyword lane: `(N, total_len)` where `N` is the - /// number of indexed chunks and `total_len` the sum of their token lengths (so `avgdl = - /// total_len / N`). Read from the `meta` keyspace at query time. `None` (or `N == 0`) means the - /// BM25 index is empty — no chunks were indexed, or [`recompute_bm25_stats`] never ran. - /// - /// [`recompute_bm25_stats`]: Self::recompute_bm25_stats - pub fn bm25_stats(&self) -> Option<(u64, u64)> { - let n = self - .meta - .get(META_BM25_DOC_COUNT) - .ok() - .flatten() - .and_then(|b| <[u8; 8]>::try_from(&b[..]).ok()) - .map(u64::from_be_bytes)?; - let total_len = self - .meta - .get(META_BM25_TOTAL_LEN) - .ok() - .flatten() - .and_then(|b| <[u8; 8]>::try_from(&b[..]).ok()) - .map(u64::from_be_bytes) - .unwrap_or(0); - Some((n, total_len)) - } - /// Recompute the corpus-global BM25 stats by sweeping the `code_bm25_by_path` forward keyspace /// once (one entry per chunk; only its 4-byte `doclen` prefix is decoded — the term list is not /// touched) and stamping `(N, total_len)` into `meta`. Runs single-threaded in the scanner's diff --git a/src/index/name_dict.rs b/src/index/name_dict.rs index 17f49a61..2a8761e5 100644 --- a/src/index/name_dict.rs +++ b/src/index/name_dict.rs @@ -345,7 +345,7 @@ pub(crate) fn scan_prefixes( keyspace: &Keyspace, prefixes: Vec>, cursor_after: Option>, -) -> impl Iterator + '_ { +) -> impl Iterator + '_ { prefixes.into_iter().flat_map(move |prefix| { let lower = match &cursor_after { // The cursor lies beyond this name's whole range. @@ -361,6 +361,7 @@ pub(crate) fn scan_prefixes( .map(|lower| keyspace.range::, _>((lower, upper))) .into_iter() .flatten() + .map(super::view::kv) }) } @@ -373,9 +374,13 @@ pub(crate) fn name_ordered_keys<'a>( needle: &str, cursor_after: Option<&[u8]>, lower: Bound>, -) -> Box + 'a> { +) -> super::view::KvIter<'a> { match dict.matching_prefixes(keyspace, needle) { Some(prefixes) => Box::new(scan_prefixes(keyspace, prefixes, cursor_after.map(<[u8]>::to_vec))), - None => Box::new(keyspace.range::, _>((lower, Bound::Unbounded))), + None => Box::new( + keyspace + .range::, _>((lower, Bound::Unbounded)) + .map(super::view::kv), + ), } } diff --git a/src/index/view.rs b/src/index/view.rs new file mode 100644 index 00000000..00b13644 --- /dev/null +++ b/src/index/view.rs @@ -0,0 +1,522 @@ +//! Read-only view over the fjall index. +//! +//! Every read of the index goes through [`IndexView`] so a later phase can answer from a shared base +//! index plus a per-worktree overlay without touching a call site (ADR-0016, phase A3). Writers keep +//! using [`IndexWriter`](super::writer::IndexWriter) on the concrete [`IndexDb`]. +//! +//! The trait has two layers. The generic layer (`get` / `prefix` / `range` / `iter`, addressed by +//! [`Part`]) is what a merging implementation has to provide; the named accessors are the lookups the +//! read path performs, written once as default methods over that layer so a layered view inherits them +//! (ADR-0016 A5 "Merge contract"). No fjall type appears in a signature: values come back as +//! [`IndexBytes`], which a later mapped base can serve without a copy. + +use std::borrow::Borrow; +use std::ops::{Bound, Deref}; +use std::path::Path; + +use fjall::Keyspace; + +use super::grep_bloom::{GrepEntry, Needle}; +use super::{ + IMPORT_PROBE_VER, IndexDb, IndexError, META_BM25_DOC_COUNT, META_BM25_TOTAL_LEN, META_IMPORT_PROBE_VER, keys, + name_dict, probe_candidates, +}; +use crate::extract::SymbolKind; +use crate::path::RelPath; + +/// Bytes read from the index: a key or a value. +/// +/// A refcounted handle, not a copy: today it wraps the fjall block slice the read already produced, and +/// a mapped immutable base will add a variant borrowing the mapping, so a merging view hands either +/// out per row without allocating. The enum is private so no backend type is nameable from a signature. +#[derive(Clone)] +pub struct IndexBytes(Repr); + +#[derive(Clone)] +enum Repr { + Fjall(fjall::Slice), +} + +impl IndexBytes { + fn from_fjall(slice: fjall::Slice) -> Self { + Self(Repr::Fjall(slice)) + } +} + +impl Deref for IndexBytes { + type Target = [u8]; + + fn deref(&self) -> &[u8] { + match &self.0 { + Repr::Fjall(slice) => slice, + } + } +} + +impl AsRef<[u8]> for IndexBytes { + fn as_ref(&self) -> &[u8] { + self + } +} + +impl PartialEq for IndexBytes { + fn eq(&self, other: &Self) -> bool { + **self == **other + } +} + +impl Eq for IndexBytes {} + +impl std::fmt::Debug for IndexBytes { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + (**self).fmt(f) + } +} + +/// One `(key, value)` row, or the error that interrupted the walk. +pub type KvResult = Result<(IndexBytes, IndexBytes), IndexError>; + +/// Boxed row iterator. One virtual call per row; the boxing itself happens once per scan. +pub type KvIter<'a> = Box + 'a>; + +/// The keyspaces a layered index merges. The memory, proposal and embedding keyspaces are not here: +/// they are per-repo-group or per-worktree state and never layered. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub enum Part { + Meta, + SymbolsByPath, + SymbolsByName, + CallsByPath, + CallsByCallee, + ImportsByModule, + ImportsByPath, + ImplementationsByTrait, + ImplementationsByPath, + RefsByDef, + RefsByPath, + ImportTargets, + ImportersByProbe, + CodeBm25Postings, + CodeBm25ByPath, + GrepBloom, +} + +impl Part { + /// Every layered keyspace, in declaration order. Checked against [`KEYSPACE_MEMTABLE_BYTES`] in + /// `view_tests`, so a keyspace added to either list alone fails there. + #[cfg(test)] + pub(super) const ALL: [Part; 16] = [ + Part::Meta, + Part::SymbolsByPath, + Part::SymbolsByName, + Part::CallsByPath, + Part::CallsByCallee, + Part::ImportsByModule, + Part::ImportsByPath, + Part::ImplementationsByTrait, + Part::ImplementationsByPath, + Part::RefsByDef, + Part::RefsByPath, + Part::ImportTargets, + Part::ImportersByProbe, + Part::CodeBm25Postings, + Part::CodeBm25ByPath, + Part::GrepBloom, + ]; + + /// The keyspace's on-disk name. + #[cfg(test)] + pub(super) const fn name(self) -> &'static str { + match self { + Part::Meta => "meta", + Part::SymbolsByPath => "symbols_by_path", + Part::SymbolsByName => "symbols_by_name", + Part::CallsByPath => "calls_by_path", + Part::CallsByCallee => "calls_by_callee", + Part::ImportsByModule => "imports_by_module", + Part::ImportsByPath => "imports_by_path", + Part::ImplementationsByTrait => "implementations_by_trait", + Part::ImplementationsByPath => "implementations_by_path", + Part::RefsByDef => "refs_by_def", + Part::RefsByPath => "refs_by_path", + Part::ImportTargets => "import_targets", + Part::ImportersByProbe => "importers_by_probe", + Part::CodeBm25Postings => "code_bm25_postings", + Part::CodeBm25ByPath => "code_bm25_by_path", + Part::GrepBloom => "grep_bloom", + } + } +} + +impl IndexDb { + fn keyspace(&self, part: Part) -> &Keyspace { + match part { + Part::Meta => &self.meta, + Part::SymbolsByPath => &self.symbols_by_path, + Part::SymbolsByName => &self.symbols_by_name, + Part::CallsByPath => &self.calls_by_path, + Part::CallsByCallee => &self.calls_by_callee, + Part::ImportsByModule => &self.imports_by_module, + Part::ImportsByPath => &self.imports_by_path, + Part::ImplementationsByTrait => &self.implementations_by_trait, + Part::ImplementationsByPath => &self.implementations_by_path, + Part::RefsByDef => &self.refs_by_def, + Part::RefsByPath => &self.refs_by_path, + Part::ImportTargets => &self.import_targets, + Part::ImportersByProbe => &self.importers_by_probe, + Part::CodeBm25Postings => &self.code_bm25_postings, + Part::CodeBm25ByPath => &self.code_bm25_by_path, + Part::GrepBloom => &self.grep_bloom, + } + } + + /// Raw keyspace access for tests that assert on stored rows or drive the name dictionaries. + #[cfg(test)] + pub(crate) fn keyspace_for_test(&self, part: Part) -> &Keyspace { + self.keyspace(part) + } + + #[cfg(test)] + pub(crate) fn callee_dict(&self) -> &std::sync::Arc { + &self.callee_names + } + + #[cfg(test)] + pub(crate) fn trait_dict(&self) -> &std::sync::Arc { + &self.trait_names + } + + /// A borrowing [`IndexView`] over this database. + pub fn view(&self) -> StandaloneView<&IndexDb> { + StandaloneView(self) + } +} + +/// `Some(&dyn IndexView)` for an optional concrete view, for the call sites that take an optional one. +pub(crate) fn as_dyn(view: &Option) -> Option<&dyn IndexView> { + view.as_ref().map(|v| v as &dyn IndexView) +} + +pub(super) fn kv(guard: fjall::Guard) -> KvResult { + let (key, value) = guard.into_inner().map_err(IndexError::from)?; + Ok((IndexBytes::from_fjall(key), IndexBytes::from_fjall(value))) +} + +/// Read access to the index. See the module docs. +pub trait IndexView: Send + Sync { + /// Point lookup of `key` in `part`. + fn get(&self, part: Part, key: &[u8]) -> Result, IndexError>; + + /// Every row of `part` whose key starts with `prefix`, in key order. + fn prefix<'a>(&'a self, part: Part, prefix: &[u8]) -> KvIter<'a>; + + /// Rows of `part` between the bounds, in key order. + fn range<'a>(&'a self, part: Part, lower: Bound<&[u8]>, upper: Bound<&[u8]>) -> KvIter<'a>; + + /// Every row of `part`, in key order. + fn iter<'a>(&'a self, part: Part) -> KvIter<'a>; + + /// Keys of `calls_by_callee` after `lower`, in key order, whose callee name may contain `needle` + /// (callers still filter by name). Served from the callee-name dictionary when it is built. + fn callee_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a>; + + /// [`Self::callee_keys_matching`] for the trait names of `implementations_by_trait`. + fn trait_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a>; + + /// True when the index holds no per-file symbol entries. Cheap: peeks at the first key of + /// `symbols_by_path` rather than counting. + /// + /// Used by the MCP startup auto-scan to detect a present-but-empty index (e.g. a + /// `views//index.fjall/` wiped out-of-band while the msgpack `index.msgpack` survived). The + /// in-RAM map cache then looks populated while the index-backed tools would silently return + /// nothing, so a rescan is warranted. + fn symbols_index_is_empty(&self) -> bool { + self.iter(Part::SymbolsByPath).next().is_none() + } + + /// Resolved uses of the definition at `(def_path, def_start)`: the scope/import-resolved backing + /// for `find_references`, each as `(use_path, use_start)`. Empty when the definition has no + /// resolved uses (or resolution never ran for its file). + fn references_to(&self, def_path: &RelPath, def_start: u32) -> Vec<(RelPath, u32)> { + let prefix = keys::refs_by_def_prefix(def_path, def_start); + self.prefix(Part::RefsByDef, &prefix) + .filter_map(|row| row.ok()) + .filter_map(|(k, _)| keys::parse_ref_by_def(&k)) + .map(|(_, _, use_path, use_start)| (use_path, use_start)) + .collect() + } + + /// The definition the use at `(use_path, use_start)` resolves to; backs `goto_definition`. + /// Cross-file definitions take precedence over the local import binding that an imported use also + /// resolves through. `None` when the position isn't a resolved reference. + fn definition_of(&self, use_path: &RelPath, use_start: u32) -> Option<(RelPath, u32)> { + let prefix = keys::refs_by_use_prefix(use_path, use_start); + let mut local = None; + for row in self.prefix(Part::RefsByPath, &prefix) { + if let Ok((k, _)) = row + && let Some((_use_path, _use_start, def_path, def_start)) = keys::parse_ref_by_path(&k) + { + if def_path != *use_path { + return Some((def_path, def_start)); + } + local = Some((def_path, def_start)); + } + } + local + } + + /// The stitched bindings of `importer`: `(local_start, def_path, def_start)`. + fn import_targets_of(&self, importer: &RelPath) -> Vec<(u32, RelPath, u32)> { + self.prefix(Part::ImportTargets, &keys::import_targets_prefix(importer)) + .filter_map(|row| row.ok()) + .filter_map(|(k, v)| keys::parse_import_target(&k, &v)) + .map(|(_, local_start, def_path, def_start)| (local_start, def_path, def_start)) + .collect() + } + + /// Every importer whose resolution consulted `path` (repo-relative, forward-slashed) or one of its + /// ancestor directories: a prefix scan of `importers_by_probe` per candidate, O(importers). + /// + /// The ancestors matter because the resolver does not descend into a directory it found missing. + /// An importer that probed `util` and found nothing never probed `util/index.ts`, so adding + /// `util/index.ts` must reach it through the directory. (A directory that existed when the + /// importer resolved is never recorded, so a path under an existing directory costs only empty + /// scans.) + fn importers_probing(&self, path: &str) -> Vec { + probe_candidates(path) + .flat_map(|candidate| { + let prefix = keys::importers_by_probe_prefix(&keys::probe_hash(candidate)); + self.prefix(Part::ImportersByProbe, &prefix) + .filter_map(|row| row.ok()) + .filter_map(|(k, _)| keys::parse_importer_by_probe(&k)) + .collect::>() + }) + .collect() + } + + /// Whether any importer consulted `path` or an ancestor directory of it (see + /// [`Self::importers_probing`]). Cheap enough to gate a rescan on a non-indexed file + /// (`tsconfig.json`, `package.json`) the watcher saw change. + fn is_probed(&self, path: &str) -> bool { + probe_candidates(path).any(|candidate| { + let prefix = keys::importers_by_probe_prefix(&keys::probe_hash(candidate)); + self.prefix(Part::ImportersByProbe, &prefix).next().is_some() + }) + } + + /// Whether the probe keyspaces cover every importer (a wholesale resolve pass completed since they + /// were created). When false, the incremental pass must run the wholesale one first. + fn import_probes_complete(&self) -> bool { + self.get(Part::Meta, META_IMPORT_PROBE_VER) + .ok() + .flatten() + .is_some_and(|v| v.first() == Some(&IMPORT_PROBE_VER)) + } + + /// Symbols whose name starts with `name` (length-prefixed keys isolate `Foo` from `Foobar`), as + /// `(name, kind, path, start_byte)`, capped at `cap`. Backs the code-search exact lane: an + /// identifier-shaped query resolves to the symbols that define it, which then map to their owning + /// chunks. The returned `name` lets the caller rank exact-name matches ahead of longer prefix + /// matches. `start_byte` is the L1 `Symbol.start_byte` (node start), which falls inside the + /// symbol's owning chunk span. + fn symbols_by_name_lookup(&self, name: &str, cap: usize) -> Vec<(String, SymbolKind, RelPath, u32)> { + let prefix = keys::symbols_by_name_prefix(name); + let mut out = Vec::new(); + for row in self.prefix(Part::SymbolsByName, &prefix) { + if out.len() >= cap { + break; + } + if let Ok((k, _)) = row + && let Some(symbol) = keys::parse_symbol_by_name(&k) + { + out.push(symbol); + } + } + out + } + + /// Corpus-global BM25 stats for the code-search keyword lane: `(N, total_len)` where `N` is the + /// number of indexed chunks and `total_len` the sum of their token lengths (so `avgdl = total_len + /// / N`). `None` (or `N == 0`) means the BM25 index is empty: no chunks were indexed, or + /// [`IndexDb::recompute_bm25_stats`] never ran. + fn bm25_stats(&self) -> Option<(u64, u64)> { + let stat = |key: &[u8]| { + self.get(Part::Meta, key) + .ok() + .flatten() + .and_then(|b| <[u8; 8]>::try_from(&b[..]).ok()) + .map(u64::from_be_bytes) + }; + let n = stat(META_BM25_DOC_COUNT)?; + Some((n, stat(META_BM25_TOTAL_LEN).unwrap_or(0))) + } + + /// The stored trigram bloom row for the content `hash_hex`. + fn grep_bloom_row(&self, hash_hex: &str) -> Option { + self.get(Part::GrepBloom, hash_hex.as_bytes()).ok().flatten() + } + + /// For each of `entries`, whether the file can be SKIPPED by a grep for `needle`: its content has + /// an intact row, the row rejects the needle, and the live file still has the stamp the entry was + /// indexed under. Rows are fetched and dropped one at a time, in parallel; nothing is held between + /// files. + fn grep_skip_verdicts(&self, root: &Path, entries: &[GrepEntry], needle: &Needle) -> Vec { + super::grep_bloom::skip_verdicts(|hash| self.grep_bloom_row(hash), root, entries, needle) + } +} + +/// The unlayered view: one [`IndexDb`], read as is. `T` is `&IndexDb` to borrow it or `IndexDb` to +/// own a handle (clones share the same keyspaces). +pub struct StandaloneView(pub(crate) T); + +impl> StandaloneView { + fn db(&self) -> &IndexDb { + self.0.borrow() + } +} + +impl + Send + Sync> IndexView for StandaloneView { + fn get(&self, part: Part, key: &[u8]) -> Result, IndexError> { + Ok(self.db().keyspace(part).get(key)?.map(IndexBytes::from_fjall)) + } + + fn prefix<'a>(&'a self, part: Part, prefix: &[u8]) -> KvIter<'a> { + Box::new(self.db().keyspace(part).prefix(prefix).map(kv)) + } + + fn range<'a>(&'a self, part: Part, lower: Bound<&[u8]>, upper: Bound<&[u8]>) -> KvIter<'a> { + Box::new(self.db().keyspace(part).range::<&[u8], _>((lower, upper)).map(kv)) + } + + fn iter<'a>(&'a self, part: Part) -> KvIter<'a> { + Box::new(self.db().keyspace(part).iter().map(kv)) + } + + fn callee_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a> { + let db = self.db(); + name_dict::name_ordered_keys(&db.calls_by_callee, &db.callee_names, needle, cursor_after, lower) + } + + fn trait_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a> { + let db = self.db(); + name_dict::name_ordered_keys( + &db.implementations_by_trait, + &db.trait_names, + needle, + cursor_after, + lower, + ) + } +} + +#[cfg(test)] +pub(crate) use recording::{ReadLog, RecordingView}; + +/// A view that records which reads reach it, so a test can prove a code path is routed through +/// [`IndexView`] and a later layered implementation has nothing to bypass. +#[cfg(test)] +mod recording { + use std::sync::{Arc, Mutex}; + + use super::*; + + /// Shared handle on the reads a [`RecordingView`] has served, as `"[:]"` strings. + #[derive(Clone, Default)] + pub(crate) struct ReadLog(Arc>>); + + impl ReadLog { + fn record(&self, op: &str, part: Option) { + let entry = part.map_or_else(|| op.to_owned(), |p| format!("{op}:{}", p.name())); + self.0.lock().expect("read log").push(entry); + } + + pub(crate) fn entries(&self) -> Vec { + self.0.lock().expect("read log").clone() + } + + pub(crate) fn clear(&self) { + self.0.lock().expect("read log").clear(); + } + } + + /// Delegates every read to a [`StandaloneView`] after logging it. + pub(crate) struct RecordingView { + inner: StandaloneView, + log: ReadLog, + } + + impl RecordingView { + pub(crate) fn new(db: IndexDb) -> (Self, ReadLog) { + let log = ReadLog::default(); + ( + Self { + inner: StandaloneView(db), + log: log.clone(), + }, + log, + ) + } + } + + impl IndexView for RecordingView { + fn get(&self, part: Part, key: &[u8]) -> Result, IndexError> { + self.log.record("get", Some(part)); + self.inner.get(part, key) + } + + fn prefix<'a>(&'a self, part: Part, prefix: &[u8]) -> KvIter<'a> { + self.log.record("prefix", Some(part)); + self.inner.prefix(part, prefix) + } + + fn range<'a>(&'a self, part: Part, lower: Bound<&[u8]>, upper: Bound<&[u8]>) -> KvIter<'a> { + self.log.record("range", Some(part)); + self.inner.range(part, lower, upper) + } + + fn iter<'a>(&'a self, part: Part) -> KvIter<'a> { + self.log.record("iter", Some(part)); + self.inner.iter(part) + } + + fn callee_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a> { + self.log.record("callee_keys_matching", None); + self.inner.callee_keys_matching(needle, cursor_after, lower) + } + + fn trait_keys_matching<'a>( + &'a self, + needle: &str, + cursor_after: Option<&[u8]>, + lower: Bound>, + ) -> KvIter<'a> { + self.log.record("trait_keys_matching", None); + self.inner.trait_keys_matching(needle, cursor_after, lower) + } + } +} diff --git a/src/index/view_tests.rs b/src/index/view_tests.rs new file mode 100644 index 00000000..8871d046 --- /dev/null +++ b/src/index/view_tests.rs @@ -0,0 +1,180 @@ +//! The `IndexView` contract: the generic layer reads exactly what the keyspaces hold. The read paths +//! that go through it are covered in `mcp::index_view_tests`. + +use std::ops::Bound; + +use super::IndexDb; +use super::view::{IndexView, KvIter, Part}; +use crate::config::ConfigV1; +use crate::scanner::{EmbedMode, ScanSource, scan}; +use crate::store::{Store, VIEW_WORKING}; + +fn scanned_store(root: &std::path::Path) -> Store { + for k in 0..6 { + std::fs::write( + root.join(format!("m{k}.rs")), + format!( + "pub trait Tr{k} {{ fn run(&self); }}\n\ + pub struct S{k};\n\ + impl Tr{k} for S{k} {{ fn run(&self) {{ shared_helper(); }} }}\n\ + pub fn driver_{k}() {{ shared_helper(); local_{k}(); }}\n" + ), + ) + .expect("write corpus file"); + } + let mut store = Store::open(root, VIEW_WORKING).expect("open store"); + scan( + root, + &mut store, + &ConfigV1::with_defaults(), + ScanSource::WorkingTree, + EmbedMode::Inline, + ) + .expect("scan"); + store +} + +type Rows = Vec<(Vec, Vec)>; +type Case<'a> = (&'a str, Bound<&'a [u8]>, Bound<&'a [u8]>); + +fn rows(iter: KvIter<'_>) -> Rows { + iter.map(|r| r.expect("row")) + .map(|(k, v)| (k.to_vec(), v.to_vec())) + .collect() +} + +fn direct_rows(db: &IndexDb, part: Part) -> Rows { + db.keyspace_for_test(part) + .iter() + .map(|g| g.into_inner().expect("row")) + .map(|(k, v)| (k.to_vec(), v.to_vec())) + .collect() +} + +fn only(direct: &Rows, keep: impl Fn(&[u8]) -> bool) -> Rows { + direct.iter().filter(|(k, _)| keep(k)).cloned().collect() +} + +#[test] +fn the_generic_layer_reads_what_each_keyspace_holds() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let db = store.index_db.as_ref().expect("index"); + let view = db.view(); + let mut non_empty = 0; + let mut bounded = 0; + for part in Part::ALL { + let direct = direct_rows(db, part); + let name = part.name(); + assert_eq!(rows(view.iter(part)), direct, "{name} iter"); + assert_eq!(rows(view.prefix(part, b"")), direct, "{name} empty prefix"); + assert_eq!( + rows(view.range(part, Bound::Unbounded, Bound::Unbounded)), + direct, + "{name} unbounded range" + ); + assert_eq!(view.get(part, &[0xFF; 3]).expect("get"), None, "{name} miss"); + let Some((mid, value)) = direct.get(direct.len() / 2) else { + continue; + }; + non_empty += 1; + assert_eq!( + view.get(part, mid).expect("get").map(|v| v.to_vec()), + Some(value.clone()), + "{name} get" + ); + for len in [1, mid.len() / 2, mid.len()] { + let prefix = &mid[..len.max(1)]; + assert_eq!( + rows(view.prefix(part, prefix)), + only(&direct, |k| k.starts_with(prefix)), + "{name} prefix of {len} bytes of a mid-keyspace key" + ); + } + let last = &direct[direct.len() - 1].0; + let cases: [Case<'_>; 6] = [ + ("[mid, ..)", Bound::Included(mid), Bound::Unbounded), + ("(mid, ..)", Bound::Excluded(mid), Bound::Unbounded), + ("(.., mid]", Bound::Unbounded, Bound::Included(mid)), + ("(.., mid)", Bound::Unbounded, Bound::Excluded(mid)), + ("[mid, last]", Bound::Included(mid), Bound::Included(last)), + ("(mid, last)", Bound::Excluded(mid), Bound::Excluded(last)), + ]; + for (label, lower, upper) in cases { + let in_lower = |k: &[u8]| match lower { + Bound::Included(b) => k >= b, + Bound::Excluded(b) => k > b, + Bound::Unbounded => true, + }; + let in_upper = |k: &[u8]| match upper { + Bound::Included(b) => k <= b, + Bound::Excluded(b) => k < b, + Bound::Unbounded => true, + }; + let expected = only(&direct, |k| in_lower(k) && in_upper(k)); + assert_eq!(rows(view.range(part, lower, upper)), expected, "{name} range {label}"); + if direct.len() > 2 { + bounded += 1; + } + } + } + assert!( + non_empty >= 6, + "the corpus must populate the layered keyspaces, got {non_empty}" + ); + assert!( + bounded >= 6 * 6, + "bounded ranges must be exercised on populated keyspaces" + ); +} + +#[test] +fn parts_name_exactly_the_layered_keyspaces() { + let names: std::collections::BTreeSet<_> = Part::ALL.iter().map(|p| p.name()).collect(); + assert_eq!(names.len(), Part::ALL.len(), "part names are unique"); + let unlayered = ["embeddings", "memory_by_key", "memory_archive", "proposals"]; + for name in &names { + assert!( + super::KEYSPACE_MEMTABLE_BYTES.iter().any(|(k, _)| k == name), + "{name} must be a real keyspace" + ); + assert!(!unlayered.contains(name), "{name} is per-group or per-worktree state"); + } + for (keyspace, _) in super::KEYSPACE_MEMTABLE_BYTES { + assert!( + names.contains(keyspace) || unlayered.contains(keyspace), + "keyspace {keyspace} is neither a layered Part nor listed as unlayered" + ); + } +} + +#[test] +fn named_reads_follow_the_data_they_derive_from() { + let empty_dir = tempfile::tempdir().expect("tempdir"); + let empty = IndexDb::open(empty_dir.path()).expect("open"); + assert!(empty.view().symbols_index_is_empty()); + assert_eq!(empty.view().bm25_stats(), None); + assert!(!empty.view().import_probes_complete()); + + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let view = store.index_view().expect("view"); + assert!(!view.symbols_index_is_empty()); + let defined = view.symbols_by_name_lookup("driver_1", 10); + assert_eq!(defined.len(), 1, "one definition of driver_1: {defined:?}"); + assert_eq!( + view.symbols_by_name_lookup("driver_1", 0), + vec![], + "cap 0 reads nothing" + ); + let all_runs = view.symbols_by_name_lookup("run", 100); + assert!( + all_runs.len() > 3, + "every trait and impl `run` is indexed: {all_runs:?}" + ); + assert_eq!( + view.symbols_by_name_lookup("run", 3), + all_runs[..3], + "the cap truncates in key order" + ); +} diff --git a/src/index/writer.rs b/src/index/writer.rs index ebfa019c..d46a2191 100644 --- a/src/index/writer.rs +++ b/src/index/writer.rs @@ -141,18 +141,22 @@ impl IndexWriter { /// Drop every index entry for `rel`. Used when a file is removed from the scan set. pub fn remove_file(&mut self, rel: &RelPath) -> Result<(), IndexError> { - self.stage_deletes_for(rel)?; + self.stage_deletes_for(rel) + } + + /// Store the grep bloom `row` (built by [`super::grep_bloom::build_row`] from the bytes the + /// scanner already holds) under the content hash `hash_hex`. Keyed by content alone, so the insert + /// overwrites an identical row and no delete is staged; the row outlives any one path and is + /// dropped by [`IndexDb::sweep_grep_bloom`]. Counted against the staged-byte budget like every + /// other entry. + pub fn upsert_grep_bloom(&mut self, hash_hex: &str, row: Vec) { let (db, batch, staged_bytes) = (&self.db, &mut self.batch, &mut self.staged_bytes); - stage_remove(batch, staged_bytes, &db.grep_bloom, rel.as_bytes().to_vec()); - Ok(()) + stage_insert(batch, staged_bytes, &db.grep_bloom, hash_hex.as_bytes().to_vec(), row); } - /// Replace `rel`'s grep bloom row with `row` (built by [`super::grep_bloom::build_row`] from the - /// bytes the scanner already holds). Keyed by path alone, so the insert overwrites the previous - /// row and no delete is staged. Counted against the staged-byte budget like every other entry. - pub fn upsert_grep_bloom(&mut self, rel: &RelPath, row: Vec) { + pub(super) fn remove_grep_bloom(&mut self, key: &[u8]) { let (db, batch, staged_bytes) = (&self.db, &mut self.batch, &mut self.staged_bytes); - stage_insert(batch, staged_bytes, &db.grep_bloom, rel.as_bytes().to_vec(), row); + stage_remove(batch, staged_bytes, &db.grep_bloom, key.to_vec()); } /// Replace the resolved-reference edges whose *use* is in `use_rel` with those derived from diff --git a/src/index/writer_tests.rs b/src/index/writer_tests.rs index 2c4fd1d9..4023583f 100644 --- a/src/index/writer_tests.rs +++ b/src/index/writer_tests.rs @@ -1,5 +1,6 @@ use super::*; use crate::extract::{Call, FileMapL2, Import, SymbolKind}; +use crate::index::view::IndexView; use tempfile::TempDir; fn fresh_db() -> (TempDir, IndexDb) { @@ -25,7 +26,7 @@ fn open_reports_a_wipe_only_when_the_stamped_schema_version_differs() { let wiped = IndexDb::open_with_memory(dir.path(), &memory).expect("reopen"); assert!(wiped.wiped_on_open(), "a mismatched revision wipes the index"); - assert!(wiped.symbols_index_is_empty()); + assert!(wiped.view().symbols_index_is_empty()); drop(wiped); let again = IndexDb::open_with_memory(dir.path(), &memory).expect("third open"); @@ -485,7 +486,7 @@ fn bm25_dual_partition_consistency() { assert_eq!(db.code_bm25_by_path.iter().count(), 1); db.recompute_bm25_stats().unwrap(); - assert_eq!(db.bm25_stats(), Some((1, 1))); + assert_eq!(db.view().bm25_stats(), Some((1, 1))); let mut w = db.writer(); w.remove_bm25_file(&rel).unwrap(); @@ -550,9 +551,12 @@ fn import_probe_rows_roundtrip_and_leave_with_the_importer() { #[test] fn import_probes_are_incomplete_until_stamped() { let (_d, db) = fresh_db(); - assert!(!db.import_probes_complete(), "a fresh index has no probe rows to trust"); + assert!( + !db.view().import_probes_complete(), + "a fresh index has no probe rows to trust" + ); db.set_import_probes_complete(true).unwrap(); - assert!(db.import_probes_complete()); + assert!(db.view().import_probes_complete()); db.set_import_probes_complete(false).unwrap(); - assert!(!db.import_probes_complete()); + assert!(!db.view().import_probes_complete()); } diff --git a/src/intel/resolve_pass.rs b/src/intel/resolve_pass.rs index 5c2c9c11..3e7615a2 100644 --- a/src/intel/resolve_pass.rs +++ b/src/intel/resolve_pass.rs @@ -52,6 +52,7 @@ use crate::backpressure::FootprintGate; use crate::chunk_drive::{ChunkCut, drive_chunks_governed}; use crate::config::MaxFootprint; use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::intel::model::FileResolvedRefs; use crate::intel::stage_budget::BudgetedWriter; use crate::lang; @@ -154,8 +155,8 @@ pub(crate) struct PathDelta<'a> { /// The paths of `candidates` that some importer's resolution consulted — the only non-indexed /// paths whose change can matter. A cheap point lookup per path. -pub(crate) fn probed_only(index_db: &IndexDb, candidates: Vec) -> Vec { - candidates.into_iter().filter(|rel| index_db.is_probed(rel)).collect() +pub(crate) fn probed_only(view: &dyn IndexView, candidates: Vec) -> Vec { + candidates.into_iter().filter(|rel| view.is_probed(rel)).collect() } /// Incremental resolve pass for the watcher: only `delta.changed` files' intra facts are restaged, @@ -171,7 +172,7 @@ pub(crate) fn resolve_pass_incremental( return; }; #[cfg(any(feature = "code-intel-js", feature = "code-intel-stack"))] - if !index_db.import_probes_complete() { + if !index_db.view().import_probes_complete() { tracing::info!("resolve pass: import-probe index not built yet — running the wholesale pass once"); resolve_pass(root, store, precise, footprint); return; @@ -184,7 +185,7 @@ pub(crate) fn resolve_pass_incremental( resolve_in_chunks(root, store, index_db, &changed_snapshot, precise, footprint, |facts| { harvest_cross_file_facts(facts, &mut importers); }); - let affected = xfile_incremental::affected_importers(index_db, store, delta); + let affected = xfile_incremental::affected_importers(&index_db.view(), store, delta); let affected_snapshot = snapshot_of(store, affected.iter()); resolve_in_chunks(root, store, index_db, &affected_snapshot, precise, footprint, |facts| { harvest_cross_file_facts(facts, &mut importers); @@ -372,18 +373,18 @@ mod xfile_incremental { use ahash::AHashSet; use super::PathDelta; - use crate::index::IndexDb; + use crate::index::view::IndexView; use crate::intel::resolver::SpecifierResolver; use crate::store::Store; /// Indexed, resolver-capable importers that are NOT in `delta.changed` (those are re-stitched /// anyway) but consulted a touched path: a changed, added or deleted file, or a non-indexed /// config file. One prefix scan of `importers_by_probe` per touched path. - pub(super) fn affected_importers(index_db: &IndexDb, store: &Store, delta: &PathDelta<'_>) -> Vec { + pub(super) fn affected_importers(view: &dyn IndexView, store: &Store, delta: &PathDelta<'_>) -> Vec { let changed: AHashSet<&str> = delta.changed.iter().map(String::as_str).collect(); let mut affected: AHashSet = AHashSet::new(); for touched in delta.changed.iter().chain(delta.removed).chain(delta.other) { - for importer in index_db.importers_probing(touched) { + for importer in view.importers_probing(touched) { let Some(key) = importer.as_str() else { continue; }; diff --git a/src/intel/stage_budget.rs b/src/intel/stage_budget.rs index e60b720a..5a1c6f60 100644 --- a/src/intel/stage_budget.rs +++ b/src/intel/stage_budget.rs @@ -193,7 +193,10 @@ mod tests { "the item-count bound must not be what committed here" ); assert!( - db.refs_by_def.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::RefsByDef) + .iter() + .next() + .is_some(), "the byte budget must have forced a commit already" ); assert!(batch.finish()); @@ -264,7 +267,7 @@ mod tests { } let dump = |db: &IndexDb| -> Vec> { - db.refs_by_def + db.keyspace_for_test(crate::index::view::Part::RefsByDef) .iter() .map(|guard| (*guard.into_inner().expect("read entry").0).to_vec()) .collect() diff --git a/src/lib.rs b/src/lib.rs index fc8bf4e9..9c859fc8 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -67,6 +67,7 @@ pub mod store_gc_budget; pub(crate) mod store_gc_live; pub mod store_gc_protect; pub mod store_gc_workspace; +mod store_hash_refs; pub mod store_layout; mod store_lock; mod store_seed; diff --git a/src/mcp/codegraph.rs b/src/mcp/codegraph.rs index d2be412f..a530d151 100644 --- a/src/mcp/codegraph.rs +++ b/src/mcp/codegraph.rs @@ -27,7 +27,7 @@ use rmcp::ErrorData as McpError; use super::MapCache; use super::helpers_calls::for_each_call_in_file; use super::helpers_graph::is_function_like; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::path::RelPath; /// The kind of relationship an edge represents. @@ -409,7 +409,7 @@ pub(crate) fn new_graph_memo(budget_bytes: u64) -> GraphMemo { /// deterministic graph). A poisoned lock is recovered — a lost memo entry only costs one rebuild. pub(crate) fn build_memoized( memo: &Mutex, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, opts: &BuildOpts, ) -> Result, McpError> { @@ -566,7 +566,7 @@ fn resolve_named_edge( /// `scan_cap`. Folding the lanes into one pass would change which files the cap cuts, so the passes /// stay separate and the outline cache absorbs the repeat reads. The build is memoized /// ([`build_memoized`]), so a repo pays this at most once per snapshot per lane set. -pub(crate) fn build(idx: Option<&IndexDb>, cache: &MapCache, opts: &BuildOpts) -> Result { +pub(crate) fn build(idx: Option<&dyn IndexView>, cache: &MapCache, opts: &BuildOpts) -> Result { let kinds = opts.kinds; // Compared byte-wise, not as `&str`: for UTF-8 paths a byte prefix and a `str` prefix are the // same test, and matching on bytes additionally keeps non-UTF-8 indexed paths in scope instead diff --git a/src/mcp/codegraph_tests.rs b/src/mcp/codegraph_tests.rs index 7bba5979..e9e80012 100644 --- a/src/mcp/codegraph_tests.rs +++ b/src/mcp/codegraph_tests.rs @@ -2,6 +2,7 @@ //! under the 1000-line module cap). Included via `#[path]` from `codegraph.rs`. use super::*; +use crate::index::view::as_dyn; use crate::store::Store; use tempfile::TempDir; @@ -59,7 +60,7 @@ fn provenance_fixture() -> (TempDir, Store, MapCache) { fn built(store: &Store, cache: &MapCache, kinds: EdgeKindSet) -> CodeGraph { build( - store.index_db.as_ref(), + as_dyn(&store.index_view()), cache, &BuildOpts { kinds, @@ -431,7 +432,7 @@ fn focus_scopes_the_build_to_files_under_the_path_prefix() { ..EdgeKindSet::none() }; let g = build( - store.index_db.as_ref(), + as_dyn(&store.index_view()), &cache, &BuildOpts { kinds, @@ -675,7 +676,8 @@ fn opts(kinds: EdgeKindSet) -> BuildOpts { #[test] fn memo_serves_the_same_build_for_one_snapshot() { let (_dir, store, cache) = provenance_fixture(); - let idx = store.index_db.as_ref(); + let view = store.index_view(); + let idx = as_dyn(&view); let memo = Mutex::new(new_graph_memo(0)); let first = build_memoized(&memo, idx, &cache, &opts(EdgeKindSet::all())).expect("first build"); @@ -701,7 +703,8 @@ fn memo_serves_the_same_build_for_one_snapshot() { #[test] fn graph_memo_evicts_to_its_byte_budget() { let (_dir, store, cache) = provenance_fixture(); - let idx = store.index_db.as_ref(); + let view = store.index_view(); + let idx = as_dyn(&view); let sized = Mutex::new(new_graph_memo(0)); let one = build_memoized(&sized, idx, &cache, &opts(EdgeKindSet::all())).expect("build"); @@ -732,7 +735,8 @@ fn graph_memo_evicts_to_its_byte_budget() { #[test] fn memo_keys_on_lanes_and_index_mode() { let (_dir, store, cache) = provenance_fixture(); - let idx = store.index_db.as_ref(); + let view = store.index_view(); + let idx = as_dyn(&view); let memo = Mutex::new(new_graph_memo(0)); let calls_only = EdgeKindSet { @@ -776,12 +780,29 @@ fn memo_isolates_snapshots_by_fingerprint() { ); let memo = Mutex::new(new_graph_memo(0)); - let ga = build_memoized(&memo, store_a.index_db.as_ref(), &cache_a, &opts(EdgeKindSet::all())).expect("a"); - let gb = build_memoized(&memo, store_b.index_db.as_ref(), &cache_b, &opts(EdgeKindSet::all())).expect("b"); + let ga = build_memoized( + &memo, + as_dyn(&store_a.index_view()), + &cache_a, + &opts(EdgeKindSet::all()), + ) + .expect("a"); + let gb = build_memoized( + &memo, + as_dyn(&store_b.index_view()), + &cache_b, + &opts(EdgeKindSet::all()), + ) + .expect("b"); assert!(!Arc::ptr_eq(&ga, &gb), "distinct fingerprints are distinct entries"); - let ga_again = - build_memoized(&memo, store_a.index_db.as_ref(), &cache_a, &opts(EdgeKindSet::all())).expect("a again"); + let ga_again = build_memoized( + &memo, + as_dyn(&store_a.index_view()), + &cache_a, + &opts(EdgeKindSet::all()), + ) + .expect("a again"); assert!( Arc::ptr_eq(&ga, &ga_again), "snapshot A still hits its own entry after B was inserted" diff --git a/src/mcp/helpers_archmap.rs b/src/mcp/helpers_archmap.rs index 26eed0f1..0297698c 100644 --- a/src/mcp/helpers_archmap.rs +++ b/src/mcp/helpers_archmap.rs @@ -31,7 +31,7 @@ use super::helpers_graph::is_function_like; use super::helpers_graphview::{DEFAULT_MAX_EXPORT_EDGES, MAX_MAX_EDGES}; use super::kneedle::knee_cutoff; use super::types_archmap::{ArchEdge, ArchNode, ArchitectureMapParams, ArchitectureMapResponse, CycleCluster}; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::path::RelPath; const PAGERANK_ITERS: usize = 20; @@ -192,7 +192,7 @@ impl RepoGraph { /// Build the graph from the L1 cache + call sites. `idx = Some` prefix-scans the /// Fjall `calls_by_path` keyspace per file; `idx = None` reads the in-RAM call index /// (read-only session). Bounded by `edge_scan_cap` total call sites. - pub(crate) fn build(idx: Option<&IndexDb>, cache: &MapCache, edge_scan_cap: usize) -> Result { + pub(crate) fn build(idx: Option<&dyn IndexView>, cache: &MapCache, edge_scan_cap: usize) -> Result { let files: Vec = cache.paths().cloned().collect(); let mut id_of: AHashMap = AHashMap::with_capacity(files.len()); for (i, p) in files.iter().enumerate() { @@ -274,7 +274,7 @@ impl RepoGraph { /// when the overlay is disabled or there's no git repo. pub(crate) fn run_architecture_map( shared: &super::shared_state::SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, churn: Option<&AHashMap>, params: ArchitectureMapParams, @@ -539,7 +539,7 @@ fn build_group_nodes( fn run_tier_grouped( shared: &super::shared_state::SharedReadStack, rg: &RepoGraph, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, churn: Option<&AHashMap>, focus: Option<&RelPath>, @@ -638,7 +638,7 @@ fn build_cycles( /// path builds no codegraph and pays nothing extra. fn grouped_lane_edges( shared: &super::shared_state::SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, rg: &RepoGraph, group_of: &[Option], @@ -819,7 +819,7 @@ fn collect_symbol_candidates( /// distinct-callee fan-out. Isolated because it is the only `?`-fallible (call-index) step here. fn build_symbol_call_edges( survivors: &[SymCand], - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, ) -> Result { let mut by_file: AHashMap> = AHashMap::new(); @@ -916,7 +916,7 @@ fn finalize_symbol_edges(edge_map: &AHashMap<(u32, u32), u32>, kept: usize, max_ fn run_tier_symbol( rg: &RepoGraph, cache: &MapCache, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, churn: Option<&AHashMap>, focus: Option<&RelPath>, params: &ArchitectureMapParams, diff --git a/src/mcp/helpers_calls.rs b/src/mcp/helpers_calls.rs index 1c04829c..8f5da5f5 100644 --- a/src/mcp/helpers_calls.rs +++ b/src/mcp/helpers_calls.rs @@ -14,7 +14,7 @@ pub(crate) use super::helpers_calls_scan::InRamCallIndex; use super::helpers_calls_scan::{CallRef, CallScanPage, budget_call_page, calls_in_file_fjall}; use super::index_route::IndexRoute; use super::types::ReferenceHit; -use crate::index::IndexDb; +use crate::index::view::{IndexView, Part}; use crate::path::RelPath; /// How `find_callers` fetches the cross-file resolved uses of a definition. The precise @@ -108,7 +108,7 @@ impl RefsSource<'_> { /// rather than one daemon round trip per file; `references` / `callers` / `implementations` are the /// point queries that forward instead. pub(super) fn for_each_call_in_file bool>( - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, path: &RelPath, mut f: F, @@ -133,12 +133,12 @@ pub(super) fn for_each_call_in_file bool>( /// `(line, column)` as 1-based / 0-based respectively. Falls back to `(0, 0)` when the /// row/col fields aren't populated (older L2 blobs predating the field's introduction). pub(super) fn resolve_call_line_col( - idx: &crate::index::IndexDb, + idx: &dyn crate::index::view::IndexView, rel: &crate::path::RelPath, start_byte: u32, ) -> (u32, u32) { let key = crate::index::keys::call_by_path(rel, start_byte); - let value = match idx.calls_by_path.get(key) { + let value = match idx.get(Part::CallsByPath, &key) { Ok(Some(v)) => v, _ => return (0, 0), }; @@ -519,9 +519,9 @@ mod tests { /// The route a writer session takes: its own open fjall index. fn local_route(store: &Store) -> super::IndexRoute { - super::IndexRoute::Local(Box::new( + super::IndexRoute::Local(Box::new(crate::index::view::StandaloneView( store.index_db.clone().expect("a writer store opens the fjall index"), - )) + ))) } /// Scan `root` and return the store plus its map cache. diff --git a/src/mcp/helpers_calls_scan.rs b/src/mcp/helpers_calls_scan.rs index 97bb101c..18e0cff0 100644 --- a/src/mcp/helpers_calls_scan.rs +++ b/src/mcp/helpers_calls_scan.rs @@ -11,6 +11,7 @@ use rmcp::ErrorData as McpError; use super::cursor::Cursor; use super::helpers_calls::resolve_call_line_col; use super::types::ReferenceHit; +use crate::index::view::Part; pub(crate) struct CallScanPage { pub total: u32, @@ -71,7 +72,7 @@ pub(super) fn budget_call_page(page: CallScanPage, max_tokens: Option) -> B /// the cursor (exclusive). The cursor returned in [`CallScanPage::next_cursor`] is the /// last key emitted on this page — pass it back on the next call to advance. pub(crate) fn scan_calls_by_name( - idx: &crate::index::IndexDb, + idx: &dyn crate::index::view::IndexView, name: &str, limit: usize, cursor_after: Option<&[u8]>, @@ -90,12 +91,8 @@ pub(crate) fn scan_calls_by_name( let scan_cap = limit.saturating_mul(8).max(2_000); let mut has_more = false; let mut matched: usize = 0; - for guard in - crate::index::name_dict::name_ordered_keys(&idx.calls_by_callee, &idx.callee_names, name, cursor_after, lower) - { - let (k, _) = guard - .into_inner() - .map_err(|e| McpError::internal_error(format!("index iter: {e}"), None))?; + for guard in idx.callee_keys_matching(name, cursor_after, lower) { + let (k, _) = guard.map_err(|e| McpError::internal_error(format!("index iter: {e}"), None))?; let Some((callee, rel, start)) = crate::index::keys::parse_call_by_callee(&k) else { continue; }; @@ -142,19 +139,15 @@ pub(crate) fn scan_calls_by_name( /// reader behind the writer session's per-file lookups and the daemon's forwarded /// `CallsInFiles`, so the two cannot drift. pub(crate) fn calls_in_file_fjall( - idx: &crate::index::IndexDb, + idx: &dyn crate::index::view::IndexView, path: &crate::path::RelPath, ) -> Result, McpError> { let prefix = crate::index::keys::calls_by_path_prefix(path); - let upper: Bound> = match super::cursor::prefix_upper_bound(&prefix) { - Some(b) => Bound::Excluded(b), - None => Bound::Unbounded, - }; + let upper = super::cursor::prefix_upper_bound(&prefix); + let upper = upper.as_deref().map_or(Bound::Unbounded, Bound::Excluded); let mut out = Vec::new(); - for guard in idx.calls_by_path.range::, _>((Bound::Included(prefix), upper)) { - let (_, v) = guard - .into_inner() - .map_err(|e| McpError::internal_error(format!("index iter: {e}"), None))?; + for guard in idx.range(Part::CallsByPath, Bound::Included(&prefix), upper) { + let (_, v) = guard.map_err(|e| McpError::internal_error(format!("index iter: {e}"), None))?; let call: crate::extract::Call = match rmp_serde::from_slice(&v) { Ok(c) => c, Err(_) => continue, diff --git a/src/mcp/helpers_calls_scan_tests.rs b/src/mcp/helpers_calls_scan_tests.rs index 345fc0d2..bd89a9d0 100644 --- a/src/mcp/helpers_calls_scan_tests.rs +++ b/src/mcp/helpers_calls_scan_tests.rs @@ -143,7 +143,7 @@ fn write_file(db: &IndexDb, rng: &mut Rng, vocab: &[String], i: usize) { type CallPage = (u32, bool, Vec>, Vec<(String, u32, u32)>, bool); fn call_page(db: &IndexDb, needle: &str, limit: usize, cursor: Option<&[u8]>) -> CallPage { - let page = scan_calls_by_name(db, needle, limit, cursor).unwrap(); + let page = scan_calls_by_name(&db.view(), needle, limit, cursor).unwrap(); let hits = page.hits.iter().map(|h| (h.callee.clone(), h.line, h.column)).collect(); ( page.total, @@ -158,7 +158,7 @@ type ImplPage = (usize, bool, Vec>, bool); fn impl_page(db: &IndexDb, needle: &str, language: Option<&str>, limit: usize, cursor: Option<&[u8]>) -> ImplPage { let page = scan_impls_fjall( - db, + &db.view(), needle, language, limit, @@ -198,9 +198,9 @@ fn assert_equivalent(db: &IndexDb, label: &str) { for limit in [2usize, 50, 400] { let (mut want_cursor, mut got_cursor): (Option>, Option>) = (None, None); for step in 0..10 { - db.callee_names.set_disabled(true); + db.callee_dict().set_disabled(true); let want = call_page(db, needle, limit, want_cursor.as_deref()); - db.callee_names.set_disabled(false); + db.callee_dict().set_disabled(false); let got = call_page(db, needle, limit, got_cursor.as_deref()); assert_eq!(want, got, "{label}: calls needle={needle:?} limit={limit} step={step}"); if !want.4 { @@ -215,9 +215,9 @@ fn assert_equivalent(db: &IndexDb, label: &str) { for limit in [2usize, 100] { let mut cursor: Option> = None; for step in 0..10 { - db.trait_names.set_disabled(true); + db.trait_dict().set_disabled(true); let want = impl_page(db, needle, language, limit, cursor.as_deref()); - db.trait_names.set_disabled(false); + db.trait_dict().set_disabled(false); let got = impl_page(db, needle, language, limit, cursor.as_deref()); assert_eq!( want, got, @@ -235,10 +235,12 @@ fn assert_equivalent(db: &IndexDb, label: &str) { fn build_dicts(db: &IndexDb) { // Without this the small test vocabulary routes most needles to the walk and proves nothing. - db.callee_names.always_use_dictionary(); - db.trait_names.always_use_dictionary(); - db.callee_names.build_blocking(&db.calls_by_callee); - db.trait_names.build_blocking(&db.implementations_by_trait); + db.callee_dict().always_use_dictionary(); + db.trait_dict().always_use_dictionary(); + db.callee_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::CallsByCallee)); + db.trait_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::ImplementationsByTrait)); } #[test] @@ -252,13 +254,13 @@ fn dictionary_scans_match_the_full_walk() { } build_dicts(&db); assert!( - db.callee_names.len() > 100, + db.callee_dict().len() > 100, "dictionary should hold the distinct callees" ); assert_equivalent(&db, "after build"); // The corpus must exercise the cap, or the partial-total path is untested. - let capped = scan_calls_by_name(&db, "run", 1, None).unwrap(); + let capped = scan_calls_by_name(&db.view(), "run", 1, None).unwrap(); assert!(capped.total_is_partial, "hot name must hit scan_cap"); // Rescan: rewrite some files (names change, old keys are removed), drop others, add new files @@ -281,8 +283,10 @@ fn dictionary_scans_match_the_full_walk() { assert_equivalent(&db, "after new names"); // A fold merges the new names into the snapshot without changing any answer. - db.callee_names.build_blocking(&db.calls_by_callee); - db.trait_names.build_blocking(&db.implementations_by_trait); + db.callee_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::CallsByCallee)); + db.trait_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::ImplementationsByTrait)); assert_equivalent(&db, "after rebuild"); } @@ -298,7 +302,7 @@ fn names_recorded_before_the_build_completes_are_not_lost() { // Written while no snapshot exists (as during a background build): recorded, not yet folded. write_file(&db, &mut rng, &["late_arrival".to_string()], 99); build_dicts(&db); - let page = scan_calls_by_name(&db, "late_arrival", 10, None).unwrap(); + let page = scan_calls_by_name(&db.view(), "late_arrival", 10, None).unwrap(); assert!(page.total > 0); assert_equivalent(&db, "late arrival"); } @@ -312,9 +316,9 @@ fn unbuilt_dictionary_still_answers_via_the_walk() { for i in 0..30 { write_file(&db, &mut rng, &vocab, i); } - db.callee_names.set_disabled(true); + db.callee_dict().set_disabled(true); let want = call_page(&db, "run", 20, None); - db.callee_names.set_disabled(false); + db.callee_dict().set_disabled(false); // First query: kicks off the background build and is served by the walk. let first = call_page(&db, "run", 20, None); assert_eq!(want, first); @@ -333,19 +337,21 @@ fn time_name_scans() { }; let db = IndexDb::open(std::path::Path::new(&view)).unwrap(); let t = Instant::now(); - db.callee_names.build_blocking(&db.calls_by_callee); + db.callee_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::CallsByCallee)); eprintln!( "callee dictionary: {} names, {} KiB resident, built in {:?}", - db.callee_names.len(), - db.callee_names.snapshot_bytes() / 1024, + db.callee_dict().len(), + db.callee_dict().snapshot_bytes() / 1024, t.elapsed() ); let t = Instant::now(); - db.trait_names.build_blocking(&db.implementations_by_trait); + db.trait_dict() + .build_blocking(db.keyspace_for_test(crate::index::view::Part::ImplementationsByTrait)); eprintln!( "trait dictionary: {} names, {} KiB resident, built in {:?}", - db.trait_names.len(), - db.trait_names.snapshot_bytes() / 1024, + db.trait_dict().len(), + db.trait_dict().snapshot_bytes() / 1024, t.elapsed() ); let needles = [ @@ -364,11 +370,11 @@ fn time_name_scans() { let mut dict = Vec::new(); let mut first = None; for round in 0..5 { - db.callee_names.set_disabled(true); + db.callee_dict().set_disabled(true); let t = Instant::now(); let want = call_page(&db, needle, 50, None); walk.push(t.elapsed()); - db.callee_names.set_disabled(false); + db.callee_dict().set_disabled(false); let t = Instant::now(); let got = call_page(&db, needle, 50, None); dict.push(t.elapsed()); diff --git a/src/mcp/helpers_code_search.rs b/src/mcp/helpers_code_search.rs index 8ee8e6c2..8e5aa809 100644 --- a/src/mcp/helpers_code_search.rs +++ b/src/mcp/helpers_code_search.rs @@ -231,13 +231,13 @@ async fn fjall_lanes( ) -> Result { { let store = state.shared.store.read().await; - if let Some(db) = store.index_db.as_ref() { - let keyword = bm25_search(db, query, limit) + if let Some(db) = store.index_view() { + let keyword = bm25_search(&db, query, limit) .into_iter() .map(|hit| (hit.chunk_id, hit.score)) .collect(); let exact = if want_exact { - exact_lane_chunk_ids(&store, db, query, limit) + exact_lane_chunk_ids(&store, &db, query, limit) } else { Vec::new() }; diff --git a/src/mcp/helpers_community.rs b/src/mcp/helpers_community.rs index e1038cf9..0878f3f4 100644 --- a/src/mcp/helpers_community.rs +++ b/src/mcp/helpers_community.rs @@ -20,7 +20,7 @@ use super::traverse::Adjacency; use super::types::LifecycleNotice; use super::types_community::{CommunitiesParams, CommunitiesResponse, Community}; use super::types_traverse::GraphNode; -use crate::index::IndexDb; +use crate::index::view::IndexView; /// Sweep bound for both detection algorithms; they converge well inside this on real graphs. const COMMUNITY_MAX_ITERS: u32 = 50; @@ -62,7 +62,7 @@ pub(super) fn label_for(adj: &Adjacency, cache: &MapCache, ranked: &[u32]) -> St /// `communities` — cluster the code-graph into de-facto modules with deterministic labels. pub(super) fn run_communities( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, params: CommunitiesParams, notice: Option, diff --git a/src/mcp/helpers_graph.rs b/src/mcp/helpers_graph.rs index 3f9d5d55..531f76a1 100644 --- a/src/mcp/helpers_graph.rs +++ b/src/mcp/helpers_graph.rs @@ -148,12 +148,12 @@ pub(super) async fn run_graph(state: &super::ServerState, params: GraphParams) - let started = std::time::Instant::now(); state.await_cache_ready().await; let store = state.shared.store.read().await; - let idx = store.index_db.as_ref().cloned(); + let view = store.index_db.as_ref().cloned().map(crate::index::view::StandaloneView); let basemind_dir = store.basemind_dir.clone(); drop(store); let cache = state.shared.cache.load_full(); let shared = &state.shared; - let idx = idx.as_ref(); + let idx = crate::index::view::as_dyn(&view); if idx.is_none() { // The graph walks read every file's call sites, so a session with no Fjall index serves them // from the in-RAM projection. Build it now so the notice below reflects whether its byte @@ -420,7 +420,7 @@ pub(super) fn is_function_like(kind: SymbolKind) -> bool { /// Entry point — builds the shared calls graph, projects it, and runs the BFS. pub(super) fn run_call_graph( shared: &SharedReadStack, - idx: Option<&crate::index::IndexDb>, + idx: Option<&dyn crate::index::view::IndexView>, params: CallGraphParams, cache: &MapCache, notice: Option, diff --git a/src/mcp/helpers_graphview.rs b/src/mcp/helpers_graphview.rs index d462ad0d..8aedd32f 100644 --- a/src/mcp/helpers_graphview.rs +++ b/src/mcp/helpers_graphview.rs @@ -24,7 +24,7 @@ use super::types::LifecycleNotice; use super::types_graphview::{ DisplayParams, DisplayResponse, GraphExportParams, GraphExportResponse, UiParams, UiResponse, }; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::path::RelPath; /// Sweep bound for community detection when tagging nodes; converges well inside this on real graphs. @@ -83,7 +83,7 @@ fn select_nodes(partition: &community::Partition, n: usize, max_nodes: usize) -> #[allow(clippy::too_many_arguments)] pub(super) fn build_graph_view( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, kinds: EdgeKindSet, min_conf: f32, @@ -262,7 +262,7 @@ fn prune_exports(dir: &std::path::Path, budget: u64) { /// `graph_export` — render the code-graph into a chosen text format. pub(super) fn run_graph_export( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, basemind_dir: &std::path::Path, params: GraphExportParams, @@ -440,7 +440,7 @@ fn spawn_opener(program: &str, args: &[&std::ffi::OsStr]) -> OpenOutcome { /// runtime. The render/build portion stays synchronous inline, matching `run_graph_export`. pub(super) async fn run_display( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, basemind_dir: &std::path::Path, params: DisplayParams, @@ -546,7 +546,7 @@ pub(super) struct UiParts { #[allow(clippy::too_many_arguments)] pub(super) fn render_ui_parts( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, format: &str, edges: &str, @@ -611,7 +611,7 @@ const UI_NOT_SERVED_DETAIL: &str = "no basemind daemon is serving the interactiv /// is the launch — the launch outcome is not reported. pub(super) async fn run_ui( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, basemind_dir: &std::path::Path, params: UiParams, diff --git a/src/mcp/helpers_grep.rs b/src/mcp/helpers_grep.rs index 8f71129b..ee27014e 100644 --- a/src/mcp/helpers_grep.rs +++ b/src/mcp/helpers_grep.rs @@ -16,6 +16,7 @@ use super::ServerState; use super::cursor::Cursor; use super::helpers::{SEARCH_LIMIT_DEFAULT, SEARCH_LIMIT_MAX}; use super::types::{GrepHit, GrepTruncation, WorkspaceGrepParams, WorkspaceGrepResponse}; +use crate::index::grep_bloom::GrepEntry; use crate::path::RelPath; /// Upper bound on the file content one `workspace_grep` call may read, summed from the sizes the @@ -97,12 +98,21 @@ pub(super) async fn run_workspace_grep( let root = state.shared.root.as_path(); let paths: Vec<&RelPath> = scanned.iter().map(|(path, _)| *path).collect(); + let entries: Vec = scanned + .iter() + .map(|(path, meta)| GrepEntry { + rel: (*path).clone(), + hash_hex: meta.hash_hex.to_string(), + size_bytes: meta.size_bytes, + mtime_ns: meta.mtime_ns, + }) + .collect(); // Files the per-file trigram blooms prove cannot match are never opened. They keep their slot in // `paths` (counted as zero), so window order, `file_idx` and every cursor are those of the full sweep. let skip = if bloom_prefilter_enabled() { super::index_route::IndexRoute::resolve(state) .await - .grep_skip(root, ¶ms.pattern, &paths) + .grep_skip(root, ¶ms.pattern, &entries) .await } else { None @@ -419,6 +429,7 @@ fn extract_line(source: &str, line_starts: &[usize], line_idx: usize) -> String #[cfg(test)] mod tests { use super::*; + use crate::index::view::IndexView; #[test] fn packs_and_unpacks_a_file_and_hit_ordinal() { @@ -680,18 +691,33 @@ mod tests { // ---- trigram-bloom prefilter equivalence ------------------------------------------------- - /// Index the current bytes and stat of every readable file, as a scan would. - fn scan_blooms(db: &crate::index::IndexDb, dir: &Path, paths: &[RelPath]) { + /// Index the current bytes and stat of every readable file, as a scan would, returning one entry per + /// path (an unreadable file gets an entry no row answers to). + fn scan_blooms(db: &crate::index::IndexDb, dir: &Path, paths: &[RelPath]) -> Vec { let mut writer = db.writer(); + let mut entries = Vec::new(); for rel in paths { let abs = dir.join(rel.to_path_buf()); let (Ok(bytes), Ok(meta)) = (std::fs::read(&abs), std::fs::metadata(&abs)) else { + entries.push(GrepEntry { + rel: rel.clone(), + hash_hex: "unreadable".to_owned(), + size_bytes: 0, + mtime_ns: 0, + }); continue; }; - let row = crate::index::grep_bloom::build_row(&bytes, meta.len(), crate::scanner_file::mtime_nanos(&meta)); - writer.upsert_grep_bloom(rel, row); + let hash_hex = crate::hashing::hex(&crate::hashing::hash_bytes(&bytes)); + writer.upsert_grep_bloom(&hash_hex, crate::index::grep_bloom::build_row(&bytes)); + entries.push(GrepEntry { + rel: rel.clone(), + hash_hex, + size_bytes: meta.len(), + mtime_ns: crate::scanner_file::mtime_nanos(&meta), + }); } writer.commit().expect("commit blooms"); + entries } fn set_mtime_ahead(path: &Path, secs: u64) { @@ -715,7 +741,7 @@ mod tests { std::fs::write(dir.path().join("d2/unique.txt"), "alpha unique_token_qq beta\n").expect("write"); paths.push(RelPath::from("d2/unique.txt")); - scan_blooms(&db, dir.path(), &paths); + let mut entries = scan_blooms(&db, dir.path(), &paths); // After the scan: an edit that adds a token (size changes), a same-size edit with a bumped // mtime, a deletion, and a file the scan never saw. @@ -732,6 +758,12 @@ mod tests { std::fs::remove_file(dir.path().join("d1/f1.txt")).expect("delete"); std::fs::write(dir.path().join("d0/new.txt"), "brand new unique_token_qq needle\n").expect("new"); paths.push(RelPath::from("d0/new.txt")); + entries.push(GrepEntry { + rel: RelPath::from("d0/new.txt"), + hash_hex: "never-indexed".to_owned(), + size_bytes: 0, + mtime_ns: 0, + }); let refs: Vec<&RelPath> = paths.iter().collect(); let patterns = [ @@ -764,7 +796,7 @@ mod tests { let re = Regex::new(pattern).expect("regex"); let literal = (regex::escape(pattern) == pattern).then(|| Finder::new(pattern.as_bytes()).into_owned()); let skip = crate::index::grep_bloom::Needle::compile(pattern) - .map(|needle| db.grep_skip_verdicts(dir.path(), &refs, &needle)); + .map(|needle| db.view().grep_skip_verdicts(dir.path(), &entries, &needle)); skipped_somewhere |= skip.as_ref().is_some_and(|s| s.iter().any(|&x| x)); for skip_hits in [0u32, 2] { let filtered = count_all(dir.path(), &refs, skip.as_deref(), &re, literal.as_ref(), skip_hits); @@ -792,7 +824,7 @@ mod tests { // The stale and new files must stay candidates for the token they now hold. let needle = crate::index::grep_bloom::Needle::compile("unique_token_qq").expect("needle"); - let skip = db.grep_skip_verdicts(dir.path(), &refs, &needle); + let skip = db.view().grep_skip_verdicts(dir.path(), &entries, &needle); for (rel, skipped) in paths.iter().zip(&skip) { if ["d3/f3.txt", "d4/f4.txt", "d0/new.txt", "d2/unique.txt"].contains(&rel.to_string().as_str()) { assert!(!skipped, "{rel} holds the token and must not be skipped"); @@ -811,30 +843,43 @@ mod tests { let db = crate::index::IndexDb::open(idx_dir.path()).expect("index"); std::fs::write(dir.path().join("a.txt"), "nothing relevant here\n").expect("write"); let rel = RelPath::from("a.txt"); + let meta = std::fs::metadata(dir.path().join("a.txt")).expect("stat"); + let entry = GrepEntry { + rel: rel.clone(), + hash_hex: crate::hashing::hex(&crate::hashing::hash_bytes(b"nothing relevant here\n")), + size_bytes: meta.len(), + mtime_ns: crate::scanner_file::mtime_nanos(&meta), + }; let needle = crate::index::grep_bloom::Needle::compile("absent_literal").expect("needle"); assert_eq!( - db.grep_skip_verdicts(dir.path(), &[&rel], &needle), + db.view() + .grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), vec![false], "no row" ); let mut writer = db.writer(); - writer.upsert_grep_bloom(&rel, vec![99; 40]); + writer.upsert_grep_bloom(&entry.hash_hex, vec![99; 40]); writer.commit().expect("commit"); assert_eq!( - db.grep_skip_verdicts(dir.path(), &[&rel], &needle), + db.view() + .grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), vec![false], "foreign row" ); scan_blooms(&db, dir.path(), std::slice::from_ref(&rel)); assert_eq!( - db.grep_skip_verdicts(dir.path(), &[&rel], &needle), + db.view() + .grep_skip_verdicts(dir.path(), std::slice::from_ref(&entry), &needle), vec![true], "current row" ); let mut writer = db.writer(); writer.remove_file(&rel).expect("remove"); writer.commit().expect("commit"); - assert!(db.grep_bloom_row(&rel).is_none(), "removing the file removes its row"); + assert!( + db.view().grep_bloom_row(&entry.hash_hex).is_some(), + "the row is shared by content, so removing one path keeps it" + ); } #[test] diff --git a/src/mcp/helpers_impls.rs b/src/mcp/helpers_impls.rs index 9efebde9..80bf7f2a 100644 --- a/src/mcp/helpers_impls.rs +++ b/src/mcp/helpers_impls.rs @@ -87,7 +87,7 @@ pub(super) async fn run_find_implementations( /// answers them from its in-RAM map while the daemon answers them from its store, and both must /// run this exact loop. pub(crate) fn scan_impls_fjall( - idx: &crate::index::IndexDb, + idx: &dyn crate::index::view::IndexView, trait_name: &str, language: Option<&str>, limit: usize, @@ -108,16 +108,8 @@ pub(crate) fn scan_impls_fjall( let mut has_more = false; let mut matched: usize = 0; - for guard in crate::index::name_dict::name_ordered_keys( - &idx.implementations_by_trait, - &idx.trait_names, - trait_name, - cursor_after, - lower, - ) { - let (k, _) = guard - .into_inner() - .map_err(|e| McpError::internal_error(format!("impl index iter: {e}"), None))?; + for guard in idx.trait_keys_matching(trait_name, cursor_after, lower) { + let (k, _) = guard.map_err(|e| McpError::internal_error(format!("impl index iter: {e}"), None))?; let Some((trait_name, impl_type, rel, start_byte)) = crate::index::keys::parse_impl_by_trait(&k) else { continue; @@ -167,7 +159,7 @@ pub(crate) fn scan_impls_fjall( /// [`scan_impls_fjall`] with the session's environment: language from the in-RAM file view and /// `(row, col)` from the cached L1 outline. pub(crate) fn scan_impls_session( - idx: &crate::index::IndexDb, + idx: &dyn crate::index::view::IndexView, cache: &super::MapCache, trait_name: &str, language: Option<&str>, diff --git a/src/mcp/helpers_traverse.rs b/src/mcp/helpers_traverse.rs index c69b7975..d4a98ce5 100644 --- a/src/mcp/helpers_traverse.rs +++ b/src/mcp/helpers_traverse.rs @@ -20,7 +20,7 @@ use super::types_traverse::{ GraphEdge, GraphNode, NeighborsParams, NeighborsResponse, PathParams, PathResponse, SubgraphParams, SubgraphResponse, }; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::path::RelPath; const DEFAULT_DEPTH: u32 = 2; @@ -234,7 +234,7 @@ fn kind_to_str_rationale(kind: crate::extract::RationaleKind) -> &'static str { /// adjacency plus whether the underlying call scan was truncated. fn build_adjacency( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, kinds: EdgeKindSet, ) -> Result<(Adjacency, bool), McpError> { @@ -289,7 +289,7 @@ fn remap_edges(edges: &[WalkEdge], index_of: &AHashMap) -> Vec, + idx: Option<&dyn IndexView>, cache: &MapCache, params: NeighborsParams, notice: Option, @@ -347,7 +347,7 @@ pub(super) fn run_neighbors( /// `path` — confidence-weighted shortest route between two symbols. pub(super) fn run_path( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, params: PathParams, notice: Option, @@ -397,7 +397,7 @@ pub(super) fn run_path( /// `subgraph` — the neighborhood around a symbol, cut to the central head. pub(super) fn run_subgraph( shared: &SharedReadStack, - idx: Option<&IndexDb>, + idx: Option<&dyn IndexView>, cache: &MapCache, params: SubgraphParams, notice: Option, diff --git a/src/mcp/index_read.rs b/src/mcp/index_read.rs index daab5a3c..c49bc909 100644 --- a/src/mcp/index_read.rs +++ b/src/mcp/index_read.rs @@ -19,7 +19,7 @@ use super::types_impls::ImplementationHit; use crate::comms::index_read_proto::{ IndexReadQuery, IndexReadResult, MAX_CALLS_IN_FILES, WireCall, WireCallHit, WireCallScan, WireImplHit, WireImplScan, }; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::store::Store; /// Why a forwarded scan could not run: the workspace the daemon holds has no fjall index, so the @@ -29,11 +29,12 @@ const NO_INDEX: &str = "the daemon holds this workspace without a fjall index, s /// Run a forwarded scan against an open workspace store. pub(crate) fn index_read_against(store: &Store, query: &IndexReadQuery) -> Result { - let idx = store.index_db.as_ref().ok_or_else(|| NO_INDEX.to_string())?; + let view = store.index_view().ok_or_else(|| NO_INDEX.to_string())?; + let idx: &dyn IndexView = &view; match query { IndexReadQuery::CallScan { name, limit, cursor } => call_scan(idx, name, *limit, cursor.as_deref()), IndexReadQuery::CallsInFiles { paths } => calls_in_files(idx, paths), - IndexReadQuery::GrepBloom { pattern, paths } => grep_skip(idx, &store.root, pattern, paths), + IndexReadQuery::GrepBloom { pattern, entries } => grep_skip(idx, &store.root, pattern, entries), IndexReadQuery::ImplScan { trait_name, language, @@ -64,7 +65,7 @@ pub(crate) fn index_read_against(store: &Store, query: &IndexReadQuery) -> Resul /// handle instead of the workspace store lock: this scan walks the whole `calls_by_callee` /// partition and would otherwise block a rescan for its duration. pub(crate) fn call_scan( - idx: &IndexDb, + idx: &dyn IndexView, name: &str, limit: u32, cursor: Option<&[u8]>, @@ -76,27 +77,28 @@ pub(crate) fn call_scan( /// The `GrepBloom` arm on its own, so the daemon can run it holding only a cloned [`IndexDb`] handle /// and the workspace root instead of the store lock: it fetches a row and `stat`s a file per path. pub(crate) fn grep_skip( - idx: &IndexDb, + idx: &dyn IndexView, root: &std::path::Path, pattern: &str, - paths: &[crate::path::RelPath], + entries: &[crate::index::grep_bloom::GrepEntry], ) -> Result { use crate::comms::index_read_proto::MAX_GREP_BLOOM_PATHS; - if paths.len() > MAX_GREP_BLOOM_PATHS { + if entries.len() > MAX_GREP_BLOOM_PATHS { return Err(format!( "grep bloom names {} files; at most {MAX_GREP_BLOOM_PATHS} per request", - paths.len() + entries.len() )); } // No usable literal: nothing can be skipped, and saying so is the answer, not an error. let Some(needle) = crate::index::grep_bloom::Needle::compile(pattern) else { - return Ok(IndexReadResult::GrepSkip(vec![false; paths.len()])); + return Ok(IndexReadResult::GrepSkip(vec![false; entries.len()])); }; - let refs: Vec<&crate::path::RelPath> = paths.iter().collect(); - Ok(IndexReadResult::GrepSkip(idx.grep_skip_verdicts(root, &refs, &needle))) + Ok(IndexReadResult::GrepSkip( + idx.grep_skip_verdicts(root, entries, &needle), + )) } -fn calls_in_files(idx: &IndexDb, paths: &[crate::path::RelPath]) -> Result { +fn calls_in_files(idx: &dyn IndexView, paths: &[crate::path::RelPath]) -> Result { if paths.len() > MAX_CALLS_IN_FILES { return Err(format!( "calls_in_files names {} files; at most {MAX_CALLS_IN_FILES} per request", diff --git a/src/mcp/index_route.rs b/src/mcp/index_route.rs index 37a88ecb..f6add724 100644 --- a/src/mcp/index_route.rs +++ b/src/mcp/index_route.rs @@ -23,7 +23,8 @@ use rmcp::ErrorData as McpError; use super::MapCache; use super::helpers_calls_scan::{CallRef, CallScanPage, calls_in_file_fjall, scan_calls_by_name, scan_calls_in_ram}; use super::helpers_impls::{ImplScanPage, scan_impls_in_ram, scan_impls_session}; -use crate::index::IndexDb; +use crate::index::grep_bloom::GrepEntry; +use crate::index::view::{IndexView, StandaloneView}; use crate::path::RelPath; #[cfg(all(feature = "comms", any(unix, windows)))] @@ -36,7 +37,7 @@ use { /// How a session reaches the fjall index its reference reads need. pub(super) enum IndexRoute { /// This session's own open index. - Local(Box), + Local(Box), /// The daemon's workspace pool, in-process (daemon-hosted connection). #[cfg(all(feature = "comms", any(unix, windows)))] Host { @@ -57,7 +58,7 @@ impl IndexRoute { /// Decide the route for this session. Never fails: an unreachable daemon is the `InRam` route. pub(super) async fn resolve(state: &super::ServerState) -> Self { if let Some(idx) = state.shared.store.read().await.index_db.clone() { - return Self::Local(Box::new(idx)); + return Self::Local(Box::new(StandaloneView(idx))); } #[cfg(all(feature = "comms", any(unix, windows)))] { @@ -112,28 +113,28 @@ impl IndexRoute { } } - /// Which of `paths` provably cannot match the grep `pattern` (`true` = skip), per the per-file + /// Which of `entries` provably cannot match the grep `pattern` (`true` = skip), per the per-file /// trigram blooms. `None` means "no prefilter available" (no reachable index, a forward that /// failed, an unusable reply): the caller then sweeps every file, which is always correct. pub(super) async fn grep_skip( &self, root: &std::path::Path, pattern: &str, - paths: &[&RelPath], + entries: &[GrepEntry], ) -> Option> { if let Self::Local(idx) = self { let needle = crate::index::grep_bloom::Needle::compile(pattern)?; - return Some(blocking(|| idx.grep_skip_verdicts(root, paths, &needle))); + return Some(blocking(|| idx.grep_skip_verdicts(root, entries, &needle))); } #[cfg(all(feature = "comms", any(unix, windows)))] { // Compiled here too so a pattern with no usable literal costs no round trip. crate::index::grep_bloom::Needle::compile(pattern)?; - let mut skip = Vec::with_capacity(paths.len()); - for chunk in paths.chunks(crate::comms::index_read_proto::MAX_GREP_BLOOM_PATHS) { + let mut skip = Vec::with_capacity(entries.len()); + for chunk in entries.chunks(crate::comms::index_read_proto::MAX_GREP_BLOOM_PATHS) { let query = IndexReadQuery::GrepBloom { pattern: pattern.to_owned(), - paths: chunk.iter().map(|&path| path.clone()).collect(), + entries: chunk.to_vec(), }; match self.forward(query).await { Some(Ok(IndexReadResult::GrepSkip(part))) if part.len() == chunk.len() => skip.extend(part), @@ -163,7 +164,7 @@ impl IndexRoute { cursor_after: Option<&[u8]>, ) -> Result { if let Self::Local(idx) = self { - return scan_calls_by_name(idx, name, limit, cursor_after); + return scan_calls_by_name(&**idx, name, limit, cursor_after); } #[cfg(all(feature = "comms", any(unix, windows)))] match self @@ -193,7 +194,7 @@ impl IndexRoute { paths: &[RelPath], ) -> Result>, McpError> { if let Self::Local(idx) = self { - return paths.iter().map(|path| calls_in_file_fjall(idx, path)).collect(); + return paths.iter().map(|path| calls_in_file_fjall(&**idx, path)).collect(); } #[cfg(all(feature = "comms", any(unix, windows)))] match self @@ -228,7 +229,7 @@ impl IndexRoute { cursor_after: Option<&[u8]>, ) -> Result { if let Self::Local(idx) = self { - return scan_impls_session(idx, cache, trait_name, language, limit, cursor_after); + return scan_impls_session(&**idx, cache, trait_name, language, limit, cursor_after); } #[cfg(all(feature = "comms", any(unix, windows)))] match self diff --git a/src/mcp/index_route_tests.rs b/src/mcp/index_route_tests.rs index e89b2b86..93ad1ebf 100644 --- a/src/mcp/index_route_tests.rs +++ b/src/mcp/index_route_tests.rs @@ -18,6 +18,8 @@ use super::types_impls::FindImplementationsParams; use super::{HostBackend, MapCache}; use crate::comms::index_read_proto::{IndexReadQuery, IndexReadResult}; use crate::config::ConfigV1; +use crate::index::grep_bloom::GrepEntry; +use crate::index::view::StandaloneView; use crate::path::RelPath; use crate::scanner::{EmbedMode, ScanSource, scan}; use crate::store::{Store, VIEW_WORKING}; @@ -41,7 +43,9 @@ fn write_corpus(root: &Path) { h = k % 5, o = k % 7, ); - std::fs::write(dir.join(format!("m{k}.rs")), source).expect("rust file"); + let file = dir.join(format!("m{k}.rs")); + std::fs::write(&file, source).expect("rust file"); + age(&file); } std::fs::create_dir_all(root.join("py")).expect("py dir"); std::fs::write( @@ -49,6 +53,16 @@ fn write_corpus(root: &Path) { b"class Base:\n pass\n\n\nclass Child(Tr1):\n def go(self):\n get_arg()\n", ) .expect("python file"); + age(&root.join("py/a.py")); +} + +/// Move a file's mtime an hour back: a stamp too close to the clock is not trusted for skipping. +fn age(path: &Path) { + std::fs::File::options() + .write(true) + .open(path) + .and_then(|f| f.set_modified(std::time::SystemTime::now() - std::time::Duration::from_secs(3600))) + .expect("age file"); } /// Stands in for the daemon's workspace pool: answers `host_index_read` from a writer store, after @@ -57,6 +71,8 @@ struct StoreHost { store: Mutex, /// When set, every read fails, as a daemon that is restarting would. broken: bool, + /// When set, every read is answered with this instead, as a daemon on another build might. + forced_reply: Option, } impl HostBackend for StoreHost { @@ -74,6 +90,9 @@ impl HostBackend for StoreHost { if self.broken { return Err("daemon unavailable".into()); } + if let Some(reply) = &self.forced_reply { + return Ok(reply.clone()); + } let query: IndexReadQuery = round_trip(&query); let store = self.store.lock().expect("store lock"); let reply = super::index_read::index_read_against(&store, &query)?; @@ -156,14 +175,16 @@ fn fixture() -> Fixture { let host: Arc = Arc::new(StoreHost { store: Mutex::new(store), broken: false, + forced_reply: None, }); // A second store handle is not needed for the broken host: it never reads. let broken: Arc = Arc::new(StoreHost { store: Mutex::new(Store::open_read_only_no_index(&root, VIEW_WORKING).expect("store")), broken: true, + forced_reply: None, }); Fixture { - local: IndexRoute::Local(Box::new(idx)), + local: IndexRoute::Local(Box::new(StandaloneView(idx))), host: IndexRoute::Host { host: Arc::clone(&host) as Arc, root: root.clone(), @@ -502,11 +523,27 @@ fn corpus_paths() -> Vec { paths } +fn corpus_entries(fx: &Fixture) -> Vec { + let store = fx.store.store.lock().expect("store"); + corpus_paths() + .into_iter() + .map(|rel| { + let entry = store.lookup(&rel).expect("indexed corpus file"); + GrepEntry { + rel, + hash_hex: entry.hash_hex.clone(), + size_bytes: entry.size_bytes, + mtime_ns: entry.mtime, + } + }) + .collect() +} + #[test] fn grep_bloom_verdicts_are_identical_on_the_local_and_forwarded_routes() { let fx = fixture(); let paths = corpus_paths(); - let refs: Vec<&RelPath> = paths.iter().collect(); + let entries = corpus_entries(&fx); for pattern in [ "driver_7\\(", "shared_helper_3", @@ -514,8 +551,8 @@ fn grep_bloom_verdicts_are_identical_on_the_local_and_forwarded_routes() { "Child\\(Tr1\\)", "zzz_not_in_corpus", ] { - let local = block_on(fx.local.grep_skip(&fx.root, pattern, &refs)).expect("local verdicts"); - let host = block_on(fx.host.grep_skip(&fx.root, pattern, &refs)).expect("forwarded verdicts"); + let local = block_on(fx.local.grep_skip(&fx.root, pattern, &entries)).expect("local verdicts"); + let host = block_on(fx.host.grep_skip(&fx.root, pattern, &entries)).expect("forwarded verdicts"); assert_eq!(local, host, "{pattern}: routes must agree"); assert_eq!(local.len(), paths.len()); let re = regex::Regex::new(pattern).expect("regex"); @@ -533,25 +570,65 @@ fn grep_bloom_verdicts_are_identical_on_the_local_and_forwarded_routes() { #[test] fn a_pattern_without_a_required_literal_or_a_failed_forward_means_no_prefilter() { let fx = fixture(); - let paths = corpus_paths(); - let refs: Vec<&RelPath> = paths.iter().collect(); - assert!(block_on(fx.local.grep_skip(&fx.root, r"\w+", &refs)).is_none()); - assert!(block_on(fx.host.grep_skip(&fx.root, ".*", &refs)).is_none()); - assert!(block_on(fx.broken_host.grep_skip(&fx.root, "driver_7", &refs)).is_none()); + let entries = corpus_entries(&fx); + assert!(block_on(fx.local.grep_skip(&fx.root, r"\w+", &entries)).is_none()); + assert!(block_on(fx.host.grep_skip(&fx.root, ".*", &entries)).is_none()); + assert!(block_on(fx.broken_host.grep_skip(&fx.root, "driver_7", &entries)).is_none()); +} + +/// A route whose daemon answers every read with `reply`. +fn host_replying(fx: &Fixture, reply: IndexReadResult) -> IndexRoute { + let host: Arc = Arc::new(StoreHost { + store: Mutex::new(Store::open_read_only_no_index(&fx.root, VIEW_WORKING).expect("store")), + broken: false, + forced_reply: Some(reply), + }); + IndexRoute::Host { + host, + root: fx.root.clone(), + } +} + +#[test] +fn a_mismatched_grep_bloom_reply_degrades_to_a_full_sweep() { + let fx = fixture(); + let entries = corpus_entries(&fx); + let replies = [ + ("wrong length", IndexReadResult::GrepSkip(vec![true; entries.len() - 1])), + ("wrong variant", IndexReadResult::CallsInFiles(Vec::new())), + ]; + for (what, reply) in replies { + let route = host_replying(&fx, reply); + assert!( + block_on(route.grep_skip(&fx.root, "driver_7", &entries)).is_none(), + "{what}: no verdicts, so every file is searched" + ); + } + let agreeing = host_replying(&fx, IndexReadResult::GrepSkip(vec![false; entries.len()])); + assert_eq!( + block_on(agreeing.grep_skip(&fx.root, "driver_7", &entries)), + Some(vec![false; entries.len()]), + "a well-formed reply is still used" + ); } #[test] fn an_oversized_grep_bloom_batch_is_refused() { let fx = fixture(); - let paths: Vec = (0..=crate::comms::index_read_proto::MAX_GREP_BLOOM_PATHS) - .map(|i| RelPath::from(format!("pkg0/m{i}.rs").as_bytes())) + let entries: Vec = (0..=crate::comms::index_read_proto::MAX_GREP_BLOOM_PATHS) + .map(|i| GrepEntry { + rel: RelPath::from(format!("pkg0/m{i}.rs").as_bytes()), + hash_hex: String::new(), + size_bytes: 0, + mtime_ns: 0, + }) .collect(); let store = fx.store.store.lock().expect("store"); let error = super::index_read::index_read_against( &store, &IndexReadQuery::GrepBloom { pattern: "driver".into(), - paths, + entries, }, ) .expect_err("an unbounded wire batch must be rejected"); diff --git a/src/mcp/index_view_tests.rs b/src/mcp/index_view_tests.rs new file mode 100644 index 00000000..249d41dc --- /dev/null +++ b/src/mcp/index_view_tests.rs @@ -0,0 +1,194 @@ +//! Proof that the read paths reach the fjall index only through [`IndexView`](crate::index::view): +//! each test hands the code under test a [`RecordingView`] and asserts on what was read. A layered +//! implementation can therefore replace the view without touching a call site. + +use super::helpers_calls_scan::{calls_in_file_fjall, scan_calls_by_name}; +use super::helpers_impls::scan_impls_fjall; +use super::index_route::IndexRoute; +use crate::config::ConfigV1; +use crate::index::grep_bloom::{GrepEntry, Needle}; +use crate::index::view::{IndexView, ReadLog, RecordingView}; +use crate::path::RelPath; +use crate::scanner::{EmbedMode, ScanSource, scan}; +use crate::store::{Store, VIEW_WORKING}; + +const FILES: usize = 6; + +fn scanned_store(root: &std::path::Path) -> Store { + for k in 0..FILES { + std::fs::write( + root.join(format!("m{k}.rs")), + format!( + "pub trait Tr{k} {{ fn run(&self); }}\n\ + pub struct S{k};\n\ + impl Tr{k} for S{k} {{ fn run(&self) {{ shared_helper(); }} }}\n\ + pub fn driver_{k}() {{ shared_helper(); local_{k}(); }}\n" + ), + ) + .expect("write corpus file"); + } + let mut store = Store::open(root, VIEW_WORKING).expect("open store"); + scan( + root, + &mut store, + &ConfigV1::with_defaults(), + ScanSource::WorkingTree, + EmbedMode::Inline, + ) + .expect("scan"); + store +} + +fn recording(store: &Store) -> (RecordingView, ReadLog) { + RecordingView::new(store.index_db.clone().expect("index")) +} + +/// One `callee_keys_matching` page plus the `calls_by_path` point read that resolves each of the +/// `2 * FILES` call sites to a line and column. +fn name_scan_reads() -> Vec { + let mut reads = vec!["callee_keys_matching".to_owned()]; + reads.extend(std::iter::repeat_n("get:calls_by_path".to_owned(), 2 * FILES)); + reads +} + +/// One bloom-row point read per entry; the prefilter fetches them in parallel, so only the multiset +/// is stable. +fn bloom_reads() -> Vec { + vec!["get:grep_bloom".to_owned(); FILES] +} + +fn sorted(mut reads: Vec) -> Vec { + reads.sort(); + reads +} + +fn grep_entries(store: &Store) -> Vec { + (0..FILES) + .map(|k| { + let rel = RelPath::from(format!("m{k}.rs").as_bytes()); + let entry = store.lookup(&rel).expect("indexed"); + GrepEntry { + rel, + hash_hex: entry.hash_hex.clone(), + size_bytes: entry.size_bytes, + mtime_ns: entry.mtime, + } + }) + .collect() +} + +#[test] +fn call_and_implementation_scans_read_through_the_view() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let db = store.index_db.clone().expect("index"); + let (view, log) = recording(&store); + + let direct = scan_calls_by_name(&db.view(), "shared_helper", 50, None).expect("direct scan"); + let routed = scan_calls_by_name(&view, "shared_helper", 50, None).expect("routed scan"); + assert_eq!(routed.total, 2 * FILES as u32, "two call sites per file"); + assert_eq!((routed.total, &routed.hit_keys), (direct.total, &direct.hit_keys)); + assert_eq!(log.entries(), name_scan_reads(), "name scan"); + + log.clear(); + let calls = calls_in_file_fjall(&view, &RelPath::from("m0.rs")).expect("calls"); + assert!(!calls.is_empty()); + assert_eq!(log.entries(), ["range:calls_by_path"], "per-file calls"); + + log.clear(); + let impls = scan_impls_fjall(&view, "Tr", None, 50, None, |_, _| true, |_, _| (0, 0)).expect("impls"); + assert_eq!(impls.total, FILES); + assert_eq!(log.entries(), ["trait_keys_matching"], "implementations"); +} + +#[test] +fn probe_and_grep_prefilter_reads_go_through_the_view() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let (view, log) = recording(&store); + + let probed = crate::intel::resolve_pass::probed_only(&view, vec!["nowhere.rs".to_owned()]); + assert!(probed.is_empty()); + assert_eq!( + log.entries(), + ["prefix:importers_by_probe"], + "a path with no ancestor directory costs one probe-prefix scan" + ); + + log.clear(); + assert!(view.importers_probing("a/b/c.ts").is_empty()); + assert_eq!( + log.entries(), + ["prefix:importers_by_probe"; 3], + "the path, then each ancestor directory" + ); + + log.clear(); + assert!(!view.import_probes_complete()); + assert_eq!(log.entries(), ["get:meta"], "probe stamp"); + + log.clear(); + assert!(!view.symbols_index_is_empty()); + assert_eq!(log.entries(), ["iter:symbols_by_path"], "emptiness peeks the first key"); + + log.clear(); + let entries = grep_entries(&store); + let needle = Needle::compile("zzz_definitely_absent").expect("needle"); + let verdicts = view.grep_skip_verdicts(dir.path(), &entries, &needle); + assert_eq!(verdicts.len(), entries.len()); + assert_eq!(sorted(log.entries()), bloom_reads(), "grep prefilter"); +} + +#[cfg(feature = "code-search")] +#[test] +fn both_search_lanes_read_through_the_view() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let (view, log) = recording(&store); + + let exact = crate::search::exact::exact_lane_chunk_ids(&store, &view, "driver_1", 10); + assert!(!exact.is_empty(), "the exact lane resolves a defined function"); + assert_eq!(log.entries(), ["prefix:symbols_by_name"], "exact lane"); + + log.clear(); + let keyword = crate::search::bm25::bm25_search(&view, "driver", 10); + assert!(!keyword.is_empty(), "the keyword lane finds the chunk"); + assert_eq!( + log.entries(), + ["get:meta", "get:meta", "prefix:code_bm25_postings"], + "keyword lane: both corpus stats, then one posting scan for the one query term" + ); +} + +#[tokio::test(flavor = "multi_thread")] +async fn the_local_route_answers_every_read_from_its_view() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = scanned_store(dir.path()); + let cache = super::MapCache::build(&store, 0); + let (view, log) = recording(&store); + let route = IndexRoute::Local(Box::new(view)); + + let page = route.scan_calls(&cache, "shared_helper", 50, None).await.expect("scan"); + assert_eq!(page.total, 2 * FILES as u32); + assert_eq!(log.entries(), name_scan_reads(), "references/callers"); + + log.clear(); + let paths = vec![RelPath::from("m1.rs")]; + let per_file = route.calls_in_files(&cache, &paths).await.expect("calls"); + assert!(!per_file[0].is_empty()); + assert_eq!(log.entries(), ["range:calls_by_path"], "resolved callers"); + + log.clear(); + let impls = route.scan_impls(&cache, "Tr", None, 50, None).await.expect("impls"); + assert_eq!(impls.total, FILES); + assert_eq!(log.entries(), ["trait_keys_matching"], "implementations"); + + log.clear(); + let entries = grep_entries(&store); + let verdicts = route + .grep_skip(dir.path(), "zzz_definitely_absent", &entries) + .await + .expect("verdicts"); + assert_eq!(verdicts.len(), entries.len()); + assert_eq!(sorted(log.entries()), bloom_reads(), "grep prefilter"); +} diff --git a/src/mcp/l1_cache.rs b/src/mcp/l1_cache.rs index 89df62d8..f9a9d10b 100644 --- a/src/mcp/l1_cache.rs +++ b/src/mcp/l1_cache.rs @@ -56,6 +56,9 @@ pub(crate) struct FileMeta { pub(crate) hash_hex: Box, pub(crate) language: Box, pub(crate) size_bytes: u64, + /// File mtime in nanoseconds as indexed (0 = unknown); with `size_bytes` the stamp grep holds the + /// live file to before trusting the content's bloom. + pub(crate) mtime_ns: i64, } /// Sorted view of every indexed file that has a readable L1 blob. @@ -154,6 +157,7 @@ impl From<&crate::store::FileEntry> for FileMeta { hash_hex: entry.hash_hex.as_str().into(), language: entry.language.as_str().into(), size_bytes: entry.size_bytes, + mtime_ns: entry.mtime, } } } @@ -164,6 +168,7 @@ impl Clone for FileMeta { hash_hex: self.hash_hex.clone(), language: self.language.clone(), size_bytes: self.size_bytes, + mtime_ns: self.mtime_ns, } } } diff --git a/src/mcp/mod.rs b/src/mcp/mod.rs index f5e646d1..0a3cad0b 100644 --- a/src/mcp/mod.rs +++ b/src/mcp/mod.rs @@ -69,6 +69,8 @@ pub(crate) mod index_read; mod index_route; #[cfg(test)] mod index_route_tests; +#[cfg(test)] +mod index_view_tests; mod kneedle; mod l1_cache; mod lean; diff --git a/src/mcp/shared_state.rs b/src/mcp/shared_state.rs index c57b6830..5dc545c9 100644 --- a/src/mcp/shared_state.rs +++ b/src/mcp/shared_state.rs @@ -18,6 +18,7 @@ use lru::LruCache; use tokio::sync::{Notify, RwLock}; use super::{MapCache, OUTLINE_CACHE_CAP, OutlineCache, ServerOptions, telemetry}; +use crate::index::view::IndexView; use crate::store::Store; /// Heavy, workspace-level read state shared across every connection to one `(root, view)`. @@ -187,9 +188,8 @@ fn last_scan_degraded(store: &Store) -> bool { pub(crate) fn boot_plan(store: &Store, options: &ServerOptions) -> (bool, bool) { let view_is_working = store.view == crate::store::VIEW_WORKING; let fjall_index_empty = store - .index_db - .as_ref() - .map(|db| db.symbols_index_is_empty()) + .index_view() + .map(|view| view.symbols_index_is_empty()) .unwrap_or(false); let needs_initial_scan = (options.daemon_writer || !options.read_only) && view_is_working @@ -301,7 +301,7 @@ impl SharedReadStack { /// fingerprint, so it invalidates itself the moment the caller's `cache` differs. pub(crate) fn graph( &self, - idx: Option<&crate::index::IndexDb>, + idx: Option<&dyn crate::index::view::IndexView>, cache: &MapCache, opts: &super::codegraph::BuildOpts, ) -> Result, rmcp::ErrorData> { diff --git a/src/mcp/state.rs b/src/mcp/state.rs index 1b9992c6..8a574761 100644 --- a/src/mcp/state.rs +++ b/src/mcp/state.rs @@ -367,6 +367,7 @@ impl MapCache { hash_hex: hash_hex.as_str().into(), language: map.language.as_str().into(), size_bytes: map.size_bytes, + mtime_ns: 0, }; l1.seed(&hash_hex, Arc::new(map)); metas.push((path, meta)); diff --git a/src/mcp/tools_graph.rs b/src/mcp/tools_graph.rs index 5530cebc..01226bd9 100644 --- a/src/mcp/tools_graph.rs +++ b/src/mcp/tools_graph.rs @@ -113,12 +113,12 @@ impl BasemindServer { ) -> Result<(String, &'static str), McpError> { self.state.await_cache_ready().await; let store = self.state.shared.store.read().await; - let idx = store.index_db.as_ref().cloned(); + let view = store.index_db.as_ref().cloned().map(crate::index::view::StandaloneView); drop(store); let cache = self.state.shared.cache.load_full(); let parts = render_ui_parts( &self.state.shared, - idx.as_ref(), + crate::index::view::as_dyn(&view), &cache, format, edges, diff --git a/src/query.rs b/src/query.rs index d9bc1c91..195d5d58 100644 --- a/src/query.rs +++ b/src/query.rs @@ -1,6 +1,7 @@ use thiserror::Error; use crate::extract::{FileMapL1, FileMapL2, Symbol, SymbolKind}; +use crate::index::view::IndexView; use crate::path::RelPath; use crate::store::{Store, StoreError}; @@ -129,7 +130,7 @@ pub fn dependents_of(store: &Store, module: &str) -> Result, QueryE /// and are therefore unavailable in the fallback (same caveat as `goto_definition`'s cross-file /// hop). pub fn resolved_references(store: &Store, def_path: &RelPath, def_start: u32) -> Vec<(RelPath, u32)> { - match store.index_db.as_ref() { + match store.index_view() { Some(index) => index.references_to(def_path, def_start), None => resolved_references_from_blob(store, def_path, def_start), } @@ -162,7 +163,7 @@ fn resolved_references_from_blob(store: &Store, def_path: &RelPath, def_start: u /// Fjall `refs_by_path` partition when the index is open, else the file's `.rref` blob (intra-file /// bindings only; cross-file targets need the index open). pub fn definition_of(store: &Store, use_path: &RelPath, use_start: u32) -> Option<(RelPath, u32)> { - match store.index_db.as_ref() { + match store.index_view() { Some(index) => index.definition_of(use_path, use_start), None => definition_of_from_blob(store, use_path, use_start), } @@ -183,6 +184,7 @@ fn definition_of_from_blob(store: &Store, use_path: &RelPath, use_start: u32) -> mod tests { use super::*; use crate::config::ConfigV1; + use crate::index::view::IndexView; use crate::scanner::{ScanSource, scan}; use crate::store::{Store, VIEW_WORKING}; @@ -227,6 +229,7 @@ mod tests { .index_db .as_ref() .expect("index open") + .view() .references_to(&path, def_start); assert!(!via_blob.is_empty(), "blob fallback must find the intra uses"); assert_eq!( diff --git a/src/scanner.rs b/src/scanner.rs index 6776f88c..37ca3694 100644 --- a/src/scanner.rs +++ b/src/scanner.rs @@ -573,6 +573,10 @@ pub fn scan_with_observer( observer.on_file(FileResult::bare(k.clone(), FileStatus::Removed)); report.stats.removed += 1; } + // Only a working-tree pass writes bloom rows; a rev or staged pass would sweep them all away. + if matches!(source, ScanSource::WorkingTree) { + sweep_grep_bloom_rows(store); + } if let Some(change) = &policy { // Recorded only when the purge ran to completion (a panic leaves `applied` false too), so a @@ -645,6 +649,45 @@ pub fn scan_with_observer( Ok(report) } +/// Drop the grep bloom rows of every content no indexed file references any more. Rows are shared by +/// content, so a removed or edited file leaves its old row behind until this full-pass sweep (or, for +/// a watcher batch, [`drop_unheld_grep_bloom_rows`]). +fn sweep_grep_bloom_rows(store: &Store) { + let Some(idx) = store.index_db.as_ref() else { + return; + }; + let live: ahash::AHashSet<&str> = store + .index + .files + .values() + .map(|entry| entry.hash_hex.as_str()) + .collect(); + if let Err(error) = idx.sweep_grep_bloom(&live) { + tracing::warn!(%error, "grep bloom sweep failed; unreferenced rows stay until the next full scan"); + } +} + +/// Drop the grep bloom rows of `hashes` that no indexed file holds any more. `hashes` are the +/// contents the pass's changed or removed paths held before it ran, so the cost follows the batch, +/// not the index; a hash another path still holds (or one the pass just wrote for a new path) stays. +/// Runs after the pass has applied its entries and committed its rows, so it sees the final manifest. +fn drop_unheld_grep_bloom_rows(store: &Store, hashes: &[String]) { + let Some(idx) = store.index_db.as_ref() else { + return; + }; + let unheld: ahash::AHashSet<&str> = hashes + .iter() + .map(String::as_str) + .filter(|hash| !store.hash_is_held(hash)) + .collect(); + if unheld.is_empty() { + return; + } + if let Err(error) = idx.remove_grep_bloom_rows(unheld) { + tracing::warn!(%error, "grep bloom row removal failed; unreferenced rows stay until the next full scan"); + } +} + /// Move the pass's breadcrumb (and the per-phase memory tally) to `phase`, if one could be written at all. Folding the `Option` in /// here keeps every phase boundary a single line and keeps "no breadcrumb" from being a condition /// the scan branches on: evidence is never a precondition for the work. @@ -765,16 +808,22 @@ pub fn scan_paths_with_observer( let nothing_removed = removed.is_empty() && doc_removed.is_empty(); #[cfg(not(feature = "documents"))] let nothing_removed = removed.is_empty(); - let other = store - .index_db - .as_ref() - .map_or_else(Vec::new, |db| crate::intel::resolve_pass::probed_only(db, other)); + let other = store.index_db.as_ref().map_or_else(Vec::new, |db| { + crate::intel::resolve_pass::probed_only(&db.view(), other) + }); if rels.is_empty() && nothing_removed && other.is_empty() { return Ok(ScanReport::default()); } let scope = derive_scope(root, &source); let mut report = ScanReport::default(); + // The contents these paths hold now: once the pass has replaced or dropped them, a content no + // other path holds leaves its grep bloom row too (a long-lived watcher never runs the full sweep). + let mut retired: Vec = rels + .iter() + .chain(&removed) + .filter_map(|rel| store.lookup(rel).map(|entry| entry.hash_hex.clone())) + .collect(); let drive = ScanDrive { root, filters: filter.filters(), @@ -823,6 +872,7 @@ pub fn scan_paths_with_observer( } for rel in &removed { + retired.extend(store.lookup(rel).map(|entry| entry.hash_hex.clone())); store.remove(rel); if let Some(idx) = store.index_db.as_ref() { let mut w = idx.writer(); @@ -842,6 +892,8 @@ pub fn scan_paths_with_observer( report.stats.removed += 1; } + drop_unheld_grep_bloom_rows(store, &retired); + store.flush()?; store.write_status_sidecar(); diff --git a/src/scanner_code.rs b/src/scanner_code.rs index e1c8e670..56f98b36 100644 --- a/src/scanner_code.rs +++ b/src/scanner_code.rs @@ -543,6 +543,7 @@ pub(crate) fn flush_code_batches( mod tests { use super::*; use crate::chunk::CodeChunk; + use crate::index::view::IndexView; fn chunk(chunk_id: &str, searchable_text: &str) -> CodeChunk { CodeChunk { @@ -790,7 +791,7 @@ mod tests { .unwrap(); writer.commit().unwrap(); db.recompute_bm25_stats().unwrap(); - assert_eq!(db.bm25_stats().map(|(n, _)| n), Some(1)); + assert_eq!(db.view().bm25_stats().map(|(n, _)| n), Some(1)); store.upsert( "src/a.rs", crate::store::FileEntry { @@ -808,7 +809,7 @@ mod tests { purge_unembedded_code(&mut store, &config, &filters, "repo:s"); let db = store.index_db.as_ref().unwrap(); assert_eq!( - db.bm25_stats().map(|(n, _)| n).unwrap_or(0), + db.view().bm25_stats().map(|(n, _)| n).unwrap_or(0), 0, "keyword postings are gone" ); diff --git a/src/scanner_file.rs b/src/scanner_file.rs index 9f8b4c2e..02976198 100644 --- a/src/scanner_file.rs +++ b/src/scanner_file.rs @@ -262,7 +262,7 @@ fn process_file( { // Bytes were not read, so a missing grep bloom (an index that predates it) is backfilled // here rather than leaving the file a permanent grep candidate. - index_batch.backfill_grep_bloom(root, &RelPath::from(rel), meta.len(), mtime); + index_batch.backfill_grep_bloom(root, &RelPath::from(rel), &existing.hash_hex); return FileResult::bare(rel.to_string(), FileStatus::Unchanged); } } @@ -298,6 +298,13 @@ fn process_file( return FileResult::bare(rel.to_string(), FileStatus::SkippedNonUtf8); } + // A stamp taken too close to the clock cannot vouch for the bytes just read (see `settled_mtime`). + let mtime = if matches!(source, ScanSource::WorkingTree) { + settled_mtime(mtime, now_nanos()) + } else { + mtime + }; + let hash = hashing::hash_bytes(&bytes); let hex_buf = hashing::hex_buf(&hash); let hash_hex_str = hashing::hex_str(&hex_buf); @@ -310,12 +317,20 @@ fn process_file( && sidecars_present && embed_state_satisfied(store, config, filters, rel, hash_hex_str, embed) { - // Content is unchanged but the stamp moved (touch, checkout): re-stamp the bloom from the - // bytes already in hand so grep keeps trusting it. + // Content is unchanged but the stamp moved (touch, checkout): refresh the entry's stamp, which + // grep holds the live file to, and make sure the content's bloom row exists. + let mut result = FileResult::bare(rel.to_string(), FileStatus::Unchanged); if matches!(source, ScanSource::WorkingTree) { - index_batch.restamp_grep_bloom(&RelPath::from(rel), &bytes, size_bytes, mtime); + index_batch.restamp_grep_bloom(hash_hex_str, &bytes); + if (existing.size_bytes, existing.mtime) != (size_bytes, mtime) { + result.upsert = Some(FileEntry { + size_bytes, + mtime, + ..existing.clone() + }); + } } - return FileResult::bare(rel.to_string(), FileStatus::Unchanged); + return result; } let want_l2 = filters.eager_l2 && store.index_db.is_some(); @@ -398,7 +413,7 @@ fn process_file( } // Grep reads the working tree, so only a working-tree scan describes what it will sweep. if matches!(source, ScanSource::WorkingTree) { - index_batch.stage_grep_bloom(&rel_path, &bytes, size_bytes, mtime); + index_batch.stage_grep_bloom(hash_hex_str, &bytes); } #[cfg(feature = "code-search")] @@ -641,6 +656,31 @@ pub fn mtime_nanos(metadata: &std::fs::Metadata) -> i64 { .unwrap_or(0) } +/// How close to the clock a file's mtime may be before the filesystem's timestamp granularity can +/// no longer tell a later edit from the content just read. +const RACY_WINDOW_NANOS: u64 = 2_000_000_000; + +/// The wall clock as nanoseconds since the Unix epoch, saturating like [`mtime_nanos`]. +fn now_nanos() -> i64 { + SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .ok() + .map_or(0, |d| i64::try_from(d.as_nanos()).unwrap_or(i64::MAX)) +} + +/// The mtime to record for bytes read when the clock said `now_ns`: `mtime_ns` itself when it is far +/// enough from `now_ns`, else 0. An edit landing in the same timestamp tick as the read could keep +/// both size and mtime, so the grep bloom guard (which holds a live file to its indexed +/// `(size, mtime)`) must not trust such a stamp; 0 is the "unknown" stamp it never trusts. The next +/// scan re-reads the file, finds the same hash and restamps it once it has settled. +pub(crate) fn settled_mtime(mtime_ns: i64, now_ns: i64) -> i64 { + if mtime_ns.abs_diff(now_ns) < RACY_WINDOW_NANOS { + 0 + } else { + mtime_ns + } +} + fn read_working_tree( root: &Path, rel: &str, diff --git a/src/scanner_index_batch.rs b/src/scanner_index_batch.rs index 2af44696..20e64061 100644 --- a/src/scanner_index_batch.rs +++ b/src/scanner_index_batch.rs @@ -206,44 +206,42 @@ impl<'a> WorkerIndexBatch<'a> { self.commit_if_full(); } - /// Stage the grep trigram bloom for a file whose bytes the scanner already holds, stamped with - /// the `(size, mtime_ns)` they were read under. No extra read: the bloom is built from the very - /// buffer the extractor parsed. Counts against the byte budget (rows are up to 1/8 of the file) - /// and may force a commit, exactly as [`Self::stage_bm25`] does. A `None` index is a no-op. - pub(crate) fn stage_grep_bloom(&mut self, rel: &RelPath, bytes: &[u8], size: u64, mtime_ns: i64) { + /// Stage the grep trigram bloom for the content `hash_hex`, whose bytes the scanner already + /// holds. No extra read: the bloom is built from the very buffer the extractor parsed. Counts + /// against the byte budget (rows are up to 1/8 of the file) and may force a commit, exactly as + /// [`Self::stage_bm25`] does. A `None` index is a no-op. + pub(crate) fn stage_grep_bloom(&mut self, hash_hex: &str, bytes: &[u8]) { let Some(index) = self.index else { return; }; - let row = crate::index::grep_bloom::build_row(bytes, size, mtime_ns); + let row = crate::index::grep_bloom::build_row(bytes); self.writer .get_or_insert_with(|| index.writer()) - .upsert_grep_bloom(rel, row); + .upsert_grep_bloom(hash_hex, row); self.commit_if_full(); } - /// [`Self::stage_grep_bloom`] unless the stored row already carries this exact stamp. For the + /// [`Self::stage_grep_bloom`] unless the content already has an intact row. For the /// content-unchanged-but-restamped path, which is re-entered on every scan until the store's own /// entry is refreshed and must not rewrite an identical row each time. - pub(crate) fn restamp_grep_bloom(&mut self, rel: &RelPath, bytes: &[u8], size: u64, mtime_ns: i64) { - if self - .index - .is_some_and(|index| index.grep_bloom_current(rel, size, mtime_ns)) - { + pub(crate) fn restamp_grep_bloom(&mut self, hash_hex: &str, bytes: &[u8]) { + if self.index.is_some_and(|index| index.grep_bloom_current(hash_hex)) { return; } - self.stage_grep_bloom(rel, bytes, size, mtime_ns); + self.stage_grep_bloom(hash_hex, bytes); } /// Backfill the bloom of a file the scan classed `Unchanged` (so its bytes were NOT read): an - /// index built before blooms existed, or one whose row went stale, gets its row on the next - /// rescan without a wipe. A current row costs one point lookup; only a missing/stale one costs - /// a read, once. A file that cannot be read is left rowless (a candidate, never skipped). - pub(crate) fn backfill_grep_bloom(&mut self, root: &std::path::Path, rel: &RelPath, size: u64, mtime_ns: i64) { + /// index built before blooms existed, or one whose row was swept, gets its row on the next + /// rescan without a wipe. A current row costs one point lookup; only a missing one costs a read, + /// once. A file that cannot be read, or whose bytes no longer hash to `hash_hex`, is left + /// rowless (a candidate, never skipped). + pub(crate) fn backfill_grep_bloom(&mut self, root: &std::path::Path, rel: &RelPath, hash_hex: &str) { use std::io::Read; let Some(index) = self.index else { return; }; - if index.grep_bloom_current(rel, size, mtime_ns) { + if index.grep_bloom_current(hash_hex) { return; } let Ok(file) = std::fs::File::open(root.join(rel.to_path_buf())) else { @@ -259,7 +257,10 @@ impl<'a> WorkerIndexBatch<'a> { if file.take(meta.len().saturating_add(1)).read_to_end(&mut bytes).is_err() { return; } - self.stage_grep_bloom(rel, &bytes, meta.len(), crate::scanner_file::mtime_nanos(&meta)); + if crate::hashing::hex(&crate::hashing::hash_bytes(&bytes)) != hash_hex { + return; + } + self.stage_grep_bloom(hash_hex, &bytes); } /// Report what the writer has staged since the last report to the ledger, and commit when @@ -371,16 +372,16 @@ mod tests { /// compared byte for byte. fn dump_all(db: &IndexDb) -> Vec, Vec)>> { vec![ - dump(&db.symbols_by_path), - dump(&db.symbols_by_name), - dump(&db.calls_by_path), - dump(&db.calls_by_callee), - dump(&db.imports_by_module), - dump(&db.imports_by_path), - dump(&db.implementations_by_trait), - dump(&db.implementations_by_path), - dump(&db.code_bm25_postings), - dump(&db.code_bm25_by_path), + dump(db.keyspace_for_test(crate::index::view::Part::SymbolsByPath)), + dump(db.keyspace_for_test(crate::index::view::Part::SymbolsByName)), + dump(db.keyspace_for_test(crate::index::view::Part::CallsByPath)), + dump(db.keyspace_for_test(crate::index::view::Part::CallsByCallee)), + dump(db.keyspace_for_test(crate::index::view::Part::ImportsByModule)), + dump(db.keyspace_for_test(crate::index::view::Part::ImportsByPath)), + dump(db.keyspace_for_test(crate::index::view::Part::ImplementationsByTrait)), + dump(db.keyspace_for_test(crate::index::view::Part::ImplementationsByPath)), + dump(db.keyspace_for_test(crate::index::view::Part::CodeBm25Postings)), + dump(db.keyspace_for_test(crate::index::view::Part::CodeBm25ByPath)), ] } @@ -402,7 +403,10 @@ mod tests { "the file-count bound must not be what committed here" ); assert!( - db.symbols_by_path.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_some(), "the byte budget must have forced a commit already" ); drop(batch); @@ -425,13 +429,19 @@ mod tests { assert!(batch.stage(&RelPath::from(format!("src/f{i}.rs")), &l1, None)); } assert!( - db.symbols_by_path.iter().next().is_none(), + db.keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_none(), "nothing may commit before the file counter reaches the bound" ); assert!(batch.stage(&RelPath::from("src/last.rs"), &l1, None)); assert!( - db.symbols_by_path.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_some(), "the {INDEX_COMMIT_BATCH}th file must force the commit" ); assert_eq!(batch.staged, 0, "the file counter resets at its own commit"); @@ -462,13 +472,22 @@ mod tests { let mut batch = WorkerIndexBatch::with_budget(Some(&db), INDEX_COMMIT_BATCH_BYTES, &SCAN_STAGED_BYTES); let mut files = 0usize; - while db.symbols_by_path.iter().next().is_none() && files < INDEX_COMMIT_BATCH { + while db + .keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_none() + && files < INDEX_COMMIT_BATCH + { assert!(batch.stage(&RelPath::from(format!("src/f{files}.rs")), &l1, None)); files += 1; } assert!( - db.symbols_by_path.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_some(), "staging {INDEX_COMMIT_BATCH} bulky files must have committed something" ); assert!( @@ -511,7 +530,11 @@ mod tests { let mut under_budget = WorkerIndexBatch::with_budget(Some(&control), 8 * 1024 * 1024, &LEDGER); under_budget.stage_bm25(&RelPath::from("src/small.rs"), &postings(1, 4)); assert!( - control.code_bm25_postings.iter().next().is_none(), + control + .keyspace_for_test(crate::index::view::Part::CodeBm25Postings) + .iter() + .next() + .is_none(), "staging well under the byte budget must leave the batch uncommitted" ); drop(under_budget); @@ -522,7 +545,10 @@ mod tests { assert_eq!(batch.staged, 0, "bm25 staging must not bump the file counter"); assert!( - db.code_bm25_postings.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::CodeBm25Postings) + .iter() + .next() + .is_some(), "bm25 staging alone must be able to force a commit" ); drop(batch); @@ -558,7 +584,10 @@ mod tests { "the file-count bound must not be what committed here" ); assert!( - db.symbols_by_path.iter().next().is_some(), + db.keyspace_for_test(crate::index::view::Part::SymbolsByPath) + .iter() + .next() + .is_some(), "the ceiling commit must have reached disk" ); worker.finish(); @@ -727,31 +756,45 @@ mod tests { static LEDGER: StagedByteLedger = StagedByteLedger::new(u64::MAX); let (_index_dir, db) = fresh_index(); let dir = tempfile::tempdir().expect("tempdir"); - std::fs::write(dir.path().join("a.rs"), "fn needle_function() {}\n".repeat(40)).expect("write"); - let meta = std::fs::metadata(dir.path().join("a.rs")).expect("stat"); - let (size, mtime) = (meta.len(), crate::scanner_file::mtime_nanos(&meta)); + let body = "fn needle_function() {}\n".repeat(40); + std::fs::write(dir.path().join("a.rs"), &body).expect("write"); + let hash = crate::hashing::hex(&crate::hashing::hash_bytes(body.as_bytes())); let rel = RelPath::from("a.rs"); let mut batch = WorkerIndexBatch::with_budget(Some(&db), u64::MAX, &LEDGER); - batch.backfill_grep_bloom(dir.path(), &rel, size, mtime); + batch.backfill_grep_bloom(dir.path(), &rel, &hash); assert!( LEDGER.in_flight() > 0, "the staged bloom row must count against the byte ledger" ); batch.finish(); assert_eq!(LEDGER.in_flight(), 0); - assert!(db.grep_bloom_current(&rel, size, mtime), "backfill wrote the row"); + assert!(db.grep_bloom_current(&hash), "backfill wrote the row"); let mut again = WorkerIndexBatch::with_budget(Some(&db), u64::MAX, &LEDGER); - again.backfill_grep_bloom(dir.path(), &rel, size, mtime); - again.restamp_grep_bloom(&rel, b"ignored", size, mtime); + again.backfill_grep_bloom(dir.path(), &rel, &hash); + again.restamp_grep_bloom(&hash, b"ignored"); assert_eq!(LEDGER.in_flight(), 0, "a current row is neither rebuilt nor restaged"); again.finish(); + db.drop_grep_bloom_row_for_test(&hash); let mut moved = WorkerIndexBatch::with_budget(Some(&db), u64::MAX, &LEDGER); - moved.restamp_grep_bloom(&rel, b"fn needle_function() {}", size, mtime + 1); - assert!(LEDGER.in_flight() > 0, "a moved stamp re-stages the row"); + moved.restamp_grep_bloom(&hash, body.as_bytes()); + assert!(LEDGER.in_flight() > 0, "a missing row is re-staged"); moved.finish(); - assert!(db.grep_bloom_current(&rel, size, mtime + 1)); + assert!(db.grep_bloom_current(&hash)); + } + + /// The bytes on disk must hash to the indexed hash, or the row would describe other content. + #[test] + fn backfill_refuses_bytes_that_no_longer_match_the_indexed_hash() { + static LEDGER: StagedByteLedger = StagedByteLedger::new(u64::MAX); + let (_index_dir, db) = fresh_index(); + let dir = tempfile::tempdir().expect("tempdir"); + std::fs::write(dir.path().join("a.rs"), "fn edited() {}\n").expect("write"); + let mut batch = WorkerIndexBatch::with_budget(Some(&db), u64::MAX, &LEDGER); + batch.backfill_grep_bloom(dir.path(), &RelPath::from("a.rs"), "not-the-hash"); + batch.finish(); + assert_eq!(db.grep_bloom_row_count(), 0); } } diff --git a/src/search/bm25.rs b/src/search/bm25.rs index 8453ab28..c12b4097 100644 --- a/src/search/bm25.rs +++ b/src/search/bm25.rs @@ -19,7 +19,8 @@ use std::cmp::Ordering; use ahash::{AHashMap, AHashSet}; use crate::chunk::CodeChunk; -use crate::index::{IndexDb, keys}; +use crate::index::keys; +use crate::index::view::{IndexView, Part}; /// Upper bound on a single token's byte length. Tokens longer than this (minified blobs, base64 /// spilled into a comment) are dropped — they are never real query terms and would bloat the index. @@ -134,7 +135,7 @@ pub fn bm25_term_score(tf: u32, doclen: u32, avgdl: f32, idf: f32) -> f32 { /// Score `query` against the BM25 keyword index in `db` and return the top `limit` chunk hits, /// highest score first (ties broken by `chunk_id` for a stable order). Returns empty when the index /// is empty, the query has no tokens, or no chunk matches. -pub fn bm25_search(db: &IndexDb, query: &str, limit: usize) -> Vec { +pub fn bm25_search(db: &dyn IndexView, query: &str, limit: usize) -> Vec { let terms = tokenize_query(query); if terms.is_empty() || limit == 0 { return Vec::new(); @@ -150,8 +151,8 @@ pub fn bm25_search(db: &IndexDb, query: &str, limit: usize) -> Vec { let mut scores: AHashMap = AHashMap::new(); for term in &terms { let mut postings: Vec<(String, u32, u32)> = Vec::new(); - for guard in db.code_bm25_postings.prefix(keys::code_bm25_postings_prefix(term)) { - let (k, v) = match guard.into_inner() { + for guard in db.prefix(Part::CodeBm25Postings, &keys::code_bm25_postings_prefix(term)) { + let (k, v) = match guard { Ok(kv) => kv, Err(error) => { tracing::warn!(%term, ?error, "bm25: posting read failed; skipping entry"); diff --git a/src/search/exact.rs b/src/search/exact.rs index 015a20fd..4af11c48 100644 --- a/src/search/exact.rs +++ b/src/search/exact.rs @@ -11,7 +11,7 @@ use ahash::{AHashMap, AHashSet}; use crate::chunk::{CodeChunk, CodeChunkBlob}; -use crate::index::IndexDb; +use crate::index::view::IndexView; use crate::path::RelPath; use crate::store::Store; @@ -50,7 +50,7 @@ pub fn owning_chunk_id(chunks: &[CodeChunk], byte_offset: u32) -> Option<&str> { /// ranked exact-name matches ahead of longer prefix matches (`parseConfig` before `parseConfigFile`). /// Returns an empty vec for non-identifier queries or when nothing resolves. `cap` bounds the symbol /// scan. Reads chunk sidecars — blocking I/O, so call off the async reactor. -pub fn exact_lane_chunk_ids(store: &Store, db: &IndexDb, query: &str, cap: usize) -> Vec { +pub fn exact_lane_chunk_ids(store: &Store, db: &dyn IndexView, query: &str, cap: usize) -> Vec { if !is_identifier_query(query) { return Vec::new(); } diff --git a/src/store.rs b/src/store.rs index 454ada69..0328e2f3 100644 --- a/src/store.rs +++ b/src/store.rs @@ -7,10 +7,12 @@ use serde::{Deserialize, Serialize}; use thiserror::Error; use crate::extract::SCHEMA_VER; +use crate::index::view::IndexView; use crate::index::{IndexDb, IndexError}; #[cfg(feature = "intelligence")] use crate::lance::LanceStore; use crate::path::RelPath; +use crate::store_hash_refs::HashRefs; /// Cache layout: where the global cache root is, how a worktree root maps to a workspace cache /// dir, and the `workspace.json` marker recording that mapping. Lives in its own module (like @@ -282,6 +284,10 @@ pub struct Store { pub view_dir: PathBuf, pub view: String, pub index: Index, + /// Which content hashes `index.files` holds, kept in step by [`Store::upsert`] / [`Store::remove`] + /// so "does any file still hold this content" is a lookup, not a pass over the manifest. Mutate + /// `index.files` through those two methods only. + hash_refs: HashRefs, /// Fjall-backed inverted index over symbols / calls / imports. Lives under /// `view_dir/index.fjall/`. Reads + writes from any caller go through `IndexDb`, /// which is cheap to clone (internally Arc'd). `None` in read-only mode when the @@ -305,6 +311,12 @@ pub struct Store { } impl Store { + /// A read view over the fjall index, or `None` when this session holds none. Every index read + /// goes through it (see [`crate::index::view`]). + pub fn index_view(&self) -> Option> { + self.index_db.as_ref().map(IndexDb::view) + } + /// Open the store for a specific view. View names are flat strings: `"working"`, /// `"staged"`, `"rev-"`. Each view has its own `index.msgpack` under /// `.basemind/views//`; blobs are shared in `.basemind/blobs/`. @@ -357,7 +369,7 @@ impl Store { }; let index_db = open_index_with_retry(root, &view_dir)?; let index_rebuild_needed = - !index.files.is_empty() && (index_db.wiped_on_open() || index_db.symbols_index_is_empty()); + !index.files.is_empty() && (index_db.wiped_on_open() || index_db.view().symbols_index_is_empty()); if index_rebuild_needed { index.files.clear(); } @@ -369,6 +381,7 @@ impl Store { blobs_shared, view_dir, view: view.to_string(), + hash_refs: HashRefs::of(&index.files), index, index_db, #[cfg(feature = "intelligence")] @@ -463,6 +476,7 @@ impl Store { blobs_shared, view_dir, view: view.to_string(), + hash_refs: HashRefs::of(&index.files), index, index_db, #[cfg(feature = "intelligence")] @@ -500,11 +514,21 @@ impl Store { } pub fn upsert(&mut self, rel: impl Into, entry: FileEntry) { - self.index.files.insert(rel.into(), entry); + self.hash_refs.acquire(&entry.hash_hex); + if let Some(prior) = self.index.files.insert(rel.into(), entry) { + self.hash_refs.release(&prior.hash_hex); + } } pub fn remove(&mut self, rel: impl AsRef<[u8]>) { - self.index.files.remove(bstr::BStr::new(rel.as_ref())); + if let Some(prior) = self.index.files.remove(bstr::BStr::new(rel.as_ref())) { + self.hash_refs.release(&prior.hash_hex); + } + } + + /// True when some indexed file holds the content `hash_hex`. + pub fn hash_is_held(&self, hash_hex: &str) -> bool { + self.hash_refs.is_held(hash_hex) } /// Look a file up by its repository-relative path. Accepts any byte source — diff --git a/src/store_hash_refs.rs b/src/store_hash_refs.rs new file mode 100644 index 00000000..0ed82ba9 --- /dev/null +++ b/src/store_hash_refs.rs @@ -0,0 +1,71 @@ +//! Reference counts over the content hashes the file manifest holds. + +use ahash::AHashMap; + +use crate::path::RelPath; +use crate::store::FileEntry; + +/// How many manifest entries hold each content hash. Keys are 8-byte digests of the hex string, not +/// the 64-character strings, since there is one per distinct content. Two hashes sharing a digest +/// share a count, so a collision can only keep a hash "held" a little longer, never free a held one. +#[derive(Debug, Default)] +pub(crate) struct HashRefs(AHashMap); + +fn digest(hash_hex: &str) -> u64 { + // Fixed seeds: the digest only has to be stable within the process. + ahash::RandomState::with_seeds(1, 2, 3, 4).hash_one(hash_hex) +} + +impl HashRefs { + pub(crate) fn of(files: &AHashMap) -> Self { + let mut refs = Self::default(); + for entry in files.values() { + refs.acquire(&entry.hash_hex); + } + refs + } + + pub(crate) fn acquire(&mut self, hash_hex: &str) { + *self.0.entry(digest(hash_hex)).or_default() += 1; + } + + pub(crate) fn release(&mut self, hash_hex: &str) { + if let std::collections::hash_map::Entry::Occupied(mut held) = self.0.entry(digest(hash_hex)) { + if *held.get() > 1 { + *held.get_mut() -= 1; + } else { + held.remove(); + } + } + } + + pub(crate) fn is_held(&self, hash_hex: &str) -> bool { + self.0.contains_key(&digest(hash_hex)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_hash_stays_held_until_its_last_holder_releases_it() { + let mut refs = HashRefs::default(); + refs.acquire("h1"); + refs.acquire("h1"); + refs.acquire("h2"); + refs.release("h1"); + assert!(refs.is_held("h1") && refs.is_held("h2")); + refs.release("h1"); + assert!(!refs.is_held("h1") && refs.is_held("h2")); + } + + #[test] + fn releasing_an_unheld_hash_is_a_no_op() { + let mut refs = HashRefs::default(); + refs.release("never"); + refs.acquire("h1"); + refs.release("never"); + assert!(refs.is_held("h1")); + } +} diff --git a/tests/cli_daemon.rs b/tests/cli_daemon.rs index 61b2f216..70ba2afe 100644 --- a/tests/cli_daemon.rs +++ b/tests/cli_daemon.rs @@ -217,11 +217,14 @@ fn live_daemon_serves_rescan_memory_and_blocks_destructive_commands() { assert_eq!(code(&rescan), 0, "{}", text(&rescan)); assert!(text(&rescan).contains("via daemon"), "{}", text(&rescan)); - let put = env.run(&["memory", "put", "k1", "forwarded value"]); - assert_eq!(code(&put), 0, "memory put must forward, not fail: {}", text(&put)); - let get = env.run(&["memory", "get", "k1"]); - assert_eq!(code(&get), 0, "{}", text(&get)); - assert!(text(&get).contains("forwarded value"), "{}", text(&get)); + // `memory` mode exists only in a binary built with the `memory` feature. + if cfg!(feature = "memory") { + let put = env.run(&["memory", "put", "k1", "forwarded value"]); + assert_eq!(code(&put), 0, "memory put must forward, not fail: {}", text(&put)); + let get = env.run(&["memory", "get", "k1"]); + assert_eq!(code(&get), 0, "{}", text(&get)); + assert!(text(&get).contains("forwarded value"), "{}", text(&get)); + } let callers = env.run(&["--json", "code", "callers", "a.rs", "alpha"]); assert_eq!(code(&callers), 0, "{}", text(&callers)); diff --git a/tests/config_scan_rules.rs b/tests/config_scan_rules.rs index f074ea0b..ec6d63fe 100644 --- a/tests/config_scan_rules.rs +++ b/tests/config_scan_rules.rs @@ -256,13 +256,15 @@ fn embed_policy_is_recorded_and_tracks_config_changes() { #[cfg(feature = "code-search")] mod code_chunks { + use basemind::index::view::IndexView; + use super::*; fn bm25_docs(store: &Store) -> u64 { store .index_db .as_ref() - .and_then(|db| db.bm25_stats()) + .and_then(|db| db.view().bm25_stats()) .map_or(0, |(n, _)| n) } diff --git a/tests/git_smoke.rs b/tests/git_smoke.rs index b37fb7ca..df71a5d8 100644 --- a/tests/git_smoke.rs +++ b/tests/git_smoke.rs @@ -12,6 +12,7 @@ use std::process::Command; use basemind::config::ConfigV1; use basemind::git::Repo; +use basemind::index::view::IndexView; use basemind::scanner::{ScanSource, scan}; use basemind::store::{Store, VIEW_STAGED, VIEW_WORKING}; use tempfile::TempDir; @@ -377,3 +378,60 @@ fn list_worktrees_reports_main_and_linked_with_branches() { assert_eq!(d.branch, None, "detached HEAD has no branch"); assert_eq!(d.head_sha.as_deref(), Some(head.as_str())); } + +/// Only a working-tree pass writes grep bloom rows, so the full-pass sweep (which deletes every row no +/// manifest entry names) must not run on a git source: it would erase rows it cannot know are wanted. +#[test] +fn staged_and_rev_scans_do_not_sweep_grep_bloom_rows() { + let (dir, cfg) = init_repo(); + let root = dir.path(); + fs::write(root.join("a.rs"), b"pub fn clean_one() {}\n").unwrap(); + run(root, &["add", "a.rs"]); + run(root, &["commit", "-qm", "init"]); + let repo = Repo::discover(root).unwrap(); + let head_sha = repo.resolve_rev("HEAD").unwrap(); + let rev_view = format!("rev-{}", &head_sha[..7]); + + let plant = |store: &Store| { + let mut writer = store.index_db.as_ref().expect("index").writer(); + writer.upsert_grep_bloom("orphan", basemind::index::grep_bloom::build_row(b"held by no file")); + writer.commit().expect("commit"); + }; + let planted = |store: &Store| { + store + .index_db + .as_ref() + .expect("index") + .view() + .grep_bloom_row("orphan") + .is_some() + }; + + let mut staged = Store::open(root, VIEW_STAGED).unwrap(); + plant(&staged); + scan( + root, + &mut staged, + &cfg, + ScanSource::Staged(&repo), + basemind::scanner::EmbedMode::Inline, + ) + .unwrap(); + assert!(planted(&staged), "a staged scan must not sweep"); + drop(staged); + + let mut rev = Store::open(root, &rev_view).unwrap(); + plant(&rev); + scan( + root, + &mut rev, + &cfg, + ScanSource::Rev { + repo: &repo, + sha: head_sha, + }, + basemind::scanner::EmbedMode::Inline, + ) + .unwrap(); + assert!(planted(&rev), "a rev scan must not sweep"); +} diff --git a/tests/grep_bloom_scan.rs b/tests/grep_bloom_scan.rs index b9894c12..f7771754 100644 --- a/tests/grep_bloom_scan.rs +++ b/tests/grep_bloom_scan.rs @@ -4,38 +4,94 @@ //! asserts what grep is allowed to trust. use std::path::Path; +use std::time::{Duration, SystemTime}; -use basemind::index::grep_bloom::row_stamp; +use basemind::index::view::IndexView; use basemind::path::RelPath; +use basemind::scanner::{EmbedMode, ScanCancel, ScanSource}; use basemind::store::{Store, VIEW_WORKING}; -fn scan(root: &Path) -> Store { +fn config(root: &Path) -> basemind::config::Config { let mut cfg = basemind::config::default_for_root(root); cfg.documents.embed = false; cfg.code_search.embed = false; + cfg +} + +/// Run `work` on a fresh thread, as the scanner's own callers do. +fn off_main(work: impl FnOnce() -> T + Send) -> T { let _ = basemind::lang::ensure_grammars().expect("grammar bootstrap"); - std::thread::scope(|scope| { - scope - .spawn(|| { - let mut store = Store::open(root, VIEW_WORKING).expect("open store"); - basemind::scanner::scan( - root, - &mut store, - &cfg, - basemind::scanner::ScanSource::WorkingTree, - basemind::scanner::EmbedMode::Inline, - ) - .expect("scan"); - store - }) - .join() - .expect("scan thread") + std::thread::scope(|scope| scope.spawn(work).join().expect("scan thread")) +} + +fn scan_cancellable(root: &Path, cancel: &ScanCancel) -> Store { + let cfg = config(root); + off_main(|| { + let mut store = Store::open(root, VIEW_WORKING).expect("open store"); + basemind::scanner::scan_with_cancel( + root, + &mut store, + &cfg, + ScanSource::WorkingTree, + EmbedMode::Inline, + cancel, + ) + .expect("scan"); + store }) } -fn stamp(store: &Store, rel: &str) -> Option<(u64, i64)> { - let row = store.index_db.as_ref()?.grep_bloom_row(&RelPath::from(rel))?; - row_stamp(&row) +fn scan(root: &Path) -> Store { + scan_cancellable(root, &ScanCancel::new()) +} + +/// The watcher's path: rescan only `rels` into the open `store`. +fn scan_paths(root: &Path, store: &mut Store, rels: &[&str]) { + let cfg = config(root); + let paths: Vec<_> = rels.iter().map(|rel| root.join(rel)).collect(); + off_main(|| basemind::scanner::scan_paths(root, store, &cfg, &paths, EmbedMode::Inline).expect("scan_paths")); +} + +/// Move a file's mtime an hour back, past the window in which its stamp is too fresh to trust. +fn age(root: &Path, rel: &str) { + let file = std::fs::OpenOptions::new().write(true).open(root.join(rel)).unwrap(); + file.set_modified(SystemTime::now() - Duration::from_secs(3600)) + .unwrap(); +} + +fn write_aged(root: &Path, rel: &str, text: &str) { + std::fs::write(root.join(rel), text).unwrap(); + age(root, rel); +} + +fn orphan_row(store: &Store, key: &str) { + put_raw( + store, + key, + basemind::index::grep_bloom::build_row(b"nothing indexed holds this"), + ); +} + +fn hash_of(root: &Path, rel: &str) -> String { + basemind::hashing::hex(&basemind::hashing::hash_bytes( + &std::fs::read(root.join(rel)).expect("read"), + )) +} + +fn has_row(store: &Store, hash_hex: &str) -> bool { + store + .index_db + .as_ref() + .is_some_and(|idx| idx.view().grep_bloom_row(hash_hex).is_some()) +} + +fn row_count(store: &Store) -> usize { + store.index_db.as_ref().map_or(0, |idx| idx.grep_bloom_row_count()) +} + +fn indexed_stamp(store: &Store, rel: &str) -> (u64, i64) { + let entry = store.lookup(RelPath::from(rel)).expect("indexed file"); + (entry.size_bytes, entry.mtime) } fn live_stamp(root: &Path, rel: &str) -> (u64, i64) { @@ -44,37 +100,80 @@ fn live_stamp(root: &Path, rel: &str) -> (u64, i64) { } #[test] -fn scan_writes_refreshes_and_removes_bloom_rows() { +fn scan_writes_refreshes_and_sweeps_bloom_rows() { basemind::store::init_isolated_cache(); let dir = tempfile::tempdir().expect("tempdir"); let root = dir.path(); std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); std::fs::write(root.join("b.rs"), "pub fn beta_function_name() -> u32 { 2 }\n").unwrap(); + let (a_old, b_old) = (hash_of(root, "a.rs"), hash_of(root, "b.rs")); let store = scan(root); - assert_eq!( - stamp(&store, "a.rs"), - Some(live_stamp(root, "a.rs")), - "scan stamps the row" + assert!( + has_row(&store, &a_old) && has_row(&store, &b_old), + "scan writes a row per content" ); - assert_eq!(stamp(&store, "b.rs"), Some(live_stamp(root, "b.rs"))); + assert_eq!(row_count(&store), 2); drop(store); - // Edit: the row follows the new bytes. std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 100_000 }\n").unwrap(); + let a_new = hash_of(root, "a.rs"); let store = scan(root); - assert_eq!( - stamp(&store, "a.rs"), - Some(live_stamp(root, "a.rs")), - "edit refreshes the row" - ); + assert!(has_row(&store, &a_new), "edit writes the new content's row"); + assert!(!has_row(&store, &a_old), "the superseded content's row is swept"); drop(store); - // Delete: the row goes with the file. std::fs::remove_file(root.join("b.rs")).unwrap(); let store = scan(root); - assert_eq!(stamp(&store, "b.rs"), None, "a removed file leaves no row"); - assert!(stamp(&store, "a.rs").is_some()); + assert!(!has_row(&store, &b_old), "a removed file's row is swept"); + assert!(has_row(&store, &a_new)); + assert_eq!(row_count(&store), 1); +} + +#[test] +fn identical_files_share_one_row_which_outlives_either_path() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + let text = "pub fn shared_function_name() -> u32 { 1 }\n"; + std::fs::write(root.join("a.rs"), text).unwrap(); + std::fs::write(root.join("b.rs"), text).unwrap(); + let hash = hash_of(root, "a.rs"); + + let store = scan(root); + assert_eq!(row_count(&store), 1, "two paths, one content, one row"); + drop(store); + + std::fs::remove_file(root.join("b.rs")).unwrap(); + let store = scan(root); + assert!(has_row(&store, &hash), "a.rs still references the content"); + drop(store); + + std::fs::remove_file(root.join("a.rs")).unwrap(); + let store = scan(root); + assert_eq!(row_count(&store), 0, "no file references the content any more"); +} + +#[test] +fn a_touched_file_keeps_its_indexed_stamp_current() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + let store = scan(root); + assert_eq!(indexed_stamp(&store, "a.rs"), live_stamp(root, "a.rs")); + drop(store); + + let file = std::fs::OpenOptions::new().write(true).open(root.join("a.rs")).unwrap(); + file.set_modified(std::time::SystemTime::now() + std::time::Duration::from_secs(60)) + .unwrap(); + drop(file); + let store = scan(root); + assert_eq!( + indexed_stamp(&store, "a.rs"), + live_stamp(root, "a.rs"), + "same content, moved stamp: the entry follows so grep keeps trusting the row" + ); } #[test] @@ -83,19 +182,257 @@ fn rescan_backfills_a_row_an_older_index_lacks() { let dir = tempfile::tempdir().expect("tempdir"); let root = dir.path(); std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); + let hash = hash_of(root, "a.rs"); let store = scan(root); - // Simulate an index built before the keyspace existed: the file is indexed, its bloom is gone. let idx = store.index_db.clone().expect("index"); - idx.drop_grep_bloom_row_for_test(&RelPath::from("a.rs")); - assert_eq!(stamp(&store, "a.rs"), None); + idx.drop_grep_bloom_row_for_test(&hash); + assert!(!has_row(&store, &hash)); drop(idx); drop(store); - // Nothing changed on disk, so the scan classes the file Unchanged -- and must still backfill. let store = scan(root); + assert!(has_row(&store, &hash), "rescan backfills the row"); +} + +/// A row exactly as the path-keyed layout (`ROW_VERSION` 1) wrote it: +/// `version(1) | size(8) | mtime_ns(8)`, then an all-zero bit array. All-zero bits reject every +/// needle, so a reader that trusted this row would wrongly skip the file. +fn v1_row() -> Vec { + let mut row = vec![0u8; 17 + 64]; + row[0] = 1; + row +} + +fn put_raw(store: &Store, key: &str, row: Vec) { + let idx = store.index_db.as_ref().expect("index"); + let mut writer = idx.writer(); + writer.upsert_grep_bloom(key, row); + writer.commit().expect("commit"); +} + +fn raw_row(store: &Store, key: &str) -> Option> { + store + .index_db + .as_ref() + .and_then(|idx| idx.view().grep_bloom_row(key)) + .map(|row| row.to_vec()) +} + +fn grep_entry(store: &Store, root: &Path, rel: &str) -> basemind::index::grep_bloom::GrepEntry { + let entry = store.lookup(RelPath::from(rel)).expect("indexed file"); + basemind::index::grep_bloom::GrepEntry { + rel: RelPath::from(rel), + hash_hex: hash_of(root, rel), + size_bytes: entry.size_bytes, + mtime_ns: entry.mtime, + } +} + +/// Replace the current row set with what an index built before the content-hash re-key holds: a +/// version-1 row under the path, and (as an A2 build wrote) a version-1 row under the hash. +fn downgrade_to_v1(store: &Store, rel: &str, hash: &str) { + store + .index_db + .as_ref() + .expect("index") + .drop_grep_bloom_row_for_test(hash); + put_raw(store, rel, v1_row()); + put_raw(store, hash, v1_row()); + assert_eq!(raw_row(store, rel).expect("path row")[0], 1, "fixture writes a v1 row"); + assert_eq!(raw_row(store, hash).expect("hash row")[0], 1, "fixture writes a v1 row"); +} + +#[test] +fn v1_rows_are_absent_on_read_so_grep_scans_every_file() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + let hash = hash_of(root, "a.rs"); + let store = scan(root); + let entry = grep_entry(&store, root, "a.rs"); + + let absent = basemind::index::grep_bloom::Needle::compile("zzz_absent_literal").expect("needle"); + let present = basemind::index::grep_bloom::Needle::compile("alpha_function_name").expect("needle"); + let idx = store.index_db.clone().expect("index"); + let verdicts = |needle| { + idx.view() + .grep_skip_verdicts(root, std::slice::from_ref(&entry), needle) + }; + assert_eq!( + verdicts(&absent), + vec![true], + "a current row skips a file lacking the literal" + ); + + downgrade_to_v1(&store, "a.rs", &hash); + assert_eq!( + verdicts(&absent), + vec![false], + "a v1 row (which rejects everything) must not skip" + ); + assert_eq!(verdicts(&present), vec![false]); +} + +#[test] +fn the_next_incremental_scan_sweeps_v1_rows_and_rebuilds_content_keyed_ones() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + let hash = hash_of(root, "a.rs"); + let store = scan(root); + downgrade_to_v1(&store, "a.rs", &hash); + drop(store); + + let store = scan(root); + assert!(raw_row(&store, "a.rs").is_none(), "the path-keyed row is swept"); + let rebuilt = raw_row(&store, &hash).expect("rebuilt under the content hash"); + assert_eq!(rebuilt[0], 2, "the v1 row under the hash was replaced by a v2 row"); + assert_eq!(row_count(&store), 1); + let entry = grep_entry(&store, root, "a.rs"); + let absent = basemind::index::grep_bloom::Needle::compile("zzz_absent_literal").expect("needle"); + let idx = store.index_db.clone().expect("index"); + assert_eq!(idx.view().grep_skip_verdicts(root, &[entry], &absent), vec![true]); +} + +#[test] +fn migrating_grep_bloom_rows_does_not_wipe_the_index() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + let hash = hash_of(root, "a.rs"); + let store = scan(root); + downgrade_to_v1(&store, "a.rs", &hash); + drop(store); + + let store = scan(root); + assert!(!store.reset_stale_schema && !store.index_rebuild_needed); + assert!( + !store.index_db.as_ref().expect("index").wiped_on_open(), + "rows migrate by row version, so no user re-indexes for them" + ); + assert!(store.lookup(RelPath::from("a.rs")).is_some()); +} + +#[test] +fn a_cancelled_scan_does_not_sweep() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + let store = scan(root); + orphan_row(&store, "orphan"); + drop(store); + + let cancel = ScanCancel::new(); + cancel.cancel(); + let store = scan_cancellable(root, &cancel); + assert!( + has_row(&store, "orphan"), + "a partial pass cannot tell dead rows from unvisited ones" + ); + drop(store); + + let store = scan(root); + assert!(!has_row(&store, "orphan"), "the next complete pass sweeps it"); +} + +#[test] +fn scan_paths_deletes_the_superseded_row_on_edit_and_on_delete() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + write_aged(root, "b.rs", "pub fn beta_function_name() -> u32 { 2 }\n"); + let (a_old, b_old) = (hash_of(root, "a.rs"), hash_of(root, "b.rs")); + let mut store = scan(root); + + std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 100_000 }\n").unwrap(); + let a_new = hash_of(root, "a.rs"); + scan_paths(root, &mut store, &["a.rs"]); + assert!( + has_row(&store, &a_new) && !has_row(&store, &a_old), + "an edit moves the row" + ); + assert!(has_row(&store, &b_old), "an untouched file keeps its row"); + + std::fs::remove_file(root.join("b.rs")).unwrap(); + scan_paths(root, &mut store, &["b.rs"]); + assert!(!has_row(&store, &b_old), "a deleted file's row goes"); + assert_eq!(row_count(&store), 1, "no growth across incremental edits"); +} + +#[test] +fn scan_paths_keeps_a_row_while_another_path_still_holds_the_content() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + let shared = "pub fn shared_function_name() -> u32 { 1 }\n"; + write_aged(root, "a.rs", shared); + write_aged(root, "b.rs", shared); + let hash = hash_of(root, "a.rs"); + let mut store = scan(root); + + std::fs::write(root.join("a.rs"), "pub fn changed_function_name() -> u32 { 2 }\n").unwrap(); + scan_paths(root, &mut store, &["a.rs"]); + assert!(has_row(&store, &hash), "b.rs still holds the old content"); + + std::fs::remove_file(root.join("b.rs")).unwrap(); + scan_paths(root, &mut store, &["b.rs"]); + assert!(!has_row(&store, &hash), "the last holder is gone"); +} + +#[test] +fn scan_paths_keeps_a_row_a_path_in_the_same_batch_moved_onto() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "a.rs", "pub fn alpha_function_name() -> u32 { 1 }\n"); + write_aged(root, "b.rs", "pub fn beta_function_name() -> u32 { 2 }\n"); + let (a_old, b_old) = (hash_of(root, "a.rs"), hash_of(root, "b.rs")); + let mut store = scan(root); + + std::fs::write(root.join("a.rs"), "pub fn beta_function_name() -> u32 { 2 }\n").unwrap(); + std::fs::write(root.join("b.rs"), "pub fn gamma_function_name() -> u32 { 3 }\n").unwrap(); + scan_paths(root, &mut store, &["a.rs", "b.rs"]); + assert!(has_row(&store, &b_old), "a.rs now holds b.rs's old content"); + assert!(!has_row(&store, &a_old)); + assert_eq!(row_count(&store), 2); +} + +#[test] +fn a_freshly_written_file_is_never_skipped_until_a_later_scan_restamps_it() { + basemind::store::init_isolated_cache(); + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + write_aged(root, "warm.rs", "pub fn warm_up_function() -> u32 { 0 }\n"); + let mut store = scan(root); + let absent = basemind::index::grep_bloom::Needle::compile("zzz_absent_literal").expect("needle"); + let verdict = |store: &Store| { + let entry = grep_entry(store, root, "a.rs"); + let idx = store.index_db.clone().expect("index"); + idx.view().grep_skip_verdicts(root, &[entry], &absent)[0] + }; + + // The first scan paid for the grammar bootstrap; an incremental pass on the open store is + // quick, so the new file's mtime is well inside the window when it is read. + std::fs::write(root.join("a.rs"), "pub fn alpha_function_name() -> u32 { 1 }\n").unwrap(); + scan_paths(root, &mut store, &["a.rs"]); + assert_eq!( + indexed_stamp(&store, "a.rs").1, + 0, + "a stamp this close to the clock is not recorded" + ); + assert!(!verdict(&store), "an untrusted stamp keeps the file a candidate"); + + age(root, "a.rs"); + scan_paths(root, &mut store, &["a.rs"]); assert_eq!( - stamp(&store, "a.rs"), - Some(live_stamp(root, "a.rs")), - "rescan backfills the row" + indexed_stamp(&store, "a.rs"), + live_stamp(root, "a.rs"), + "restamped once settled" ); + assert!(verdict(&store), "a settled stamp is trusted again"); } diff --git a/tests/scan_smoke.rs b/tests/scan_smoke.rs index f83dd682..fe48a62e 100644 --- a/tests/scan_smoke.rs +++ b/tests/scan_smoke.rs @@ -1206,7 +1206,7 @@ fn deferred_embed_mode_indexes_symbols_and_keyword_lane_but_writes_no_vectors() assert_eq!(hits.len(), 1, "deferred scan must still index symbols"); let db = store.index_db.as_ref().expect("index db present"); - let keyword = basemind::search::bm25::bm25_search(db, "parse", 10); + let keyword = basemind::search::bm25::bm25_search(&db.view(), "parse", 10); assert!(!keyword.is_empty(), "deferred scan must populate the BM25 keyword lane"); assert!(