feat(index): layered index A2–A3 — content-keyed grep_bloom, IndexView trait, and the A2–A8 spec

docs/layered-index-design.md

@@ -1,6 +1,6 @@
 # Layered index: design and implementation plan
 
-- **Status:** Design accepted, not implemented
+- **Status:** A2 and A3 implemented; A4–A8 not started
 - **Date:** 2026-10-18
 - **Decision record:** [ADR-0016](adr/0016-shared-base-index-and-worktree-overlays.md)
 - **Tracking issue:** #87
@@ -11,6 +11,15 @@ this file records *how* — the on-disk layout, the data model, the exact read-m
 the write path, the correctness gate, and the file-by-file phase breakdown. Symbol names and paths are
 as of 0.30.0.
 
+> **Superseded in part.** The ADR-0016 *Specification* section is authoritative wherever it and this
+> file disagree. It corrects this plan against the code and records the owner decisions taken since:
+> A2 migrates by `ROW_VERSION` 2 with **no** `INDEX_PARTITION_REVISION` bump; `IndexView` returns
+> boxed iterators over a backend-neutral `IndexBytes`, not visitors over `Vec<u8>`; the base is read
+> through a pluggable `BaseReader` (Fjall or an immutable sorted-run format, chosen by a prototype
+> before A4); the base is built by scanning an exported tree as a working tree (a `Rev` scan has no
+> resolve pass); and `code_chunks` is keyed `(scope, path)`, so A7 must re-key it too. The sections
+> below that these touch carry a pointer to the ADR.
+
 ## 1. Problem
 
 Each git worktree is its own workspace: the store key is blake3 of the canonical worktree root
@@ -189,6 +198,9 @@ it holds. This keeps `upsert_file`'s read-before-write and per-file commit seman
 
 ## 8. `IndexView` trait (A3)
 
+> Implemented differently; see ADR-0016 §A3 and `src/index/view.rs`. The sketch below is the
+> original proposal, kept for the rationale.
+
 The read path today takes a concrete `&IndexDb` (or a cloned handle) and calls Fjall directly. A3
 introduces a trait so the layered implementation can slot in with **no behaviour change**:
 
@@ -251,7 +263,7 @@ Each phase is its own commit, lands behind `[index] layered = false`, and keeps
   the live-file guard uses the **indexed** stamp from `FileEntry` instead of the row's own.
 - `src/index/writer.rs`: `upsert_grep_bloom(hash, row)`; `remove_file` stops deleting the now-shared row.
 - `src/index/mod.rs`: `sweep_grep_bloom(&live_hashes)` (drop rows no live file references), run in the
-  full-scan purge; bump `INDEX_PARTITION_REVISION` 6 → 7.
+  full-scan purge. Migration is by `ROW_VERSION` 2 with no partition-revision bump (ADR-0016 §A2).
 - `src/scanner_index_batch.rs`: stage by hash; `restamp_grep_bloom` / `backfill_grep_bloom` key on hash.
 - `src/mcp/l1_cache.rs`: add `mtime_ns` to `FileMeta`.
 - `src/mcp/helpers_grep.rs`, `src/mcp/index_route.rs`, `src/mcp/index_read.rs`,
@@ -268,7 +280,8 @@ Each phase is its own commit, lands behind `[index] layered = false`, and keeps
 
 - `src/store_layout.rs`: `repos/<group>/base/gen-<n>/` and `repos/<group>/overlays/<ws_key>/` path
   helpers + `base.json` / `overlay.json` sidecars.
-- Base builder: `ScanSource::Rev` over `origin/HEAD` into `gen-0/`, published read-only.
+- Base builder: an exported tree of `origin/HEAD` scanned as a working tree into `gen-0/`, published
+  read-only, in the storage format the A4 prototype selects (ADR-0016 §A4).
 - Migration: a workspace whose only index is the legacy standalone becomes base + overlay on first
   layered open; `store_seed.rs` stays for now.
 
@@ -282,13 +295,13 @@ Each phase is its own commit, lands behind `[index] layered = false`, and keeps
 
 ### A6 — generations, rebase, promotion, GC
 
-- Generation advance (CoW clone + `diff-tree`), publication-by-rename, three-generation retention,
+- Generation advance (CoW clone + `diff-tree`), publication-by-rename, two-generation retention (three when `max_group_bytes` allows; ADR-0016 *Disk budget*),
   overlay GC wired into `store_gc`, orphan reap.
 
 ### A7 — shared Lance tables
 
 - `documents` / `doc_links` re-keyed to `(scope, content_hash)`; queries over-fetch and filter by live
-  hashes. `code_chunks` is already content-addressed.
+  hashes. `code_chunks` is keyed `(scope, path)` and needs the same re-key (ADR-0016 §A7).
 - Documented deviation: top-k overlap with a per-worktree table stays ≥0.95.
 
 ### A8 — layered by default

src/comms/daemon_forward_handlers.rs

@@ -218,14 +218,14 @@ pub(crate) fn index_read_via_pool(
             .with_workspace(root, |store| store.index_db.clone())
             .map_err(|error| error.to_string())?
             .ok_or_else(|| "the daemon holds this workspace without a fjall index".to_string())?;
-        return crate::mcp::index_read::call_scan(&idx, name, *limit, cursor.as_deref());
+        return crate::mcp::index_read::call_scan(&idx.view(), name, *limit, cursor.as_deref());
     }
-    if let IndexReadQuery::GrepBloom { pattern, paths } = query {
+    if let IndexReadQuery::GrepBloom { pattern, entries } = query {
         let (idx, store_root) = pool
             .with_workspace(root, |store| (store.index_db.clone(), store.root.clone()))
             .map_err(|error| error.to_string())?;
         let idx = idx.ok_or_else(|| "the daemon holds this workspace without a fjall index".to_string())?;
-        return crate::mcp::index_read::grep_skip(&idx, &store_root, pattern, paths);
+        return crate::mcp::index_read::grep_skip(&idx.view(), &store_root, pattern, entries);
     }
     pool.with_workspace(root, |store| crate::mcp::index_read::index_read_against(store, query))
         .map_err(|error| error.to_string())?
@@ -270,19 +270,19 @@ pub(crate) fn code_search_lanes_against(
     use crate::comms::code_search_proto::CodeSearchLaneResult;
 
     let limit = (query.limit as usize).min(MAX_FORWARDED_LANE_LIMIT);
-    let Some(db) = store.index_db.as_ref() else {
+    let Some(db) = store.index_view() else {
         return Err(
             "the daemon holds this workspace without a fjall index, so neither the keyword nor the \
              exact lane can be read"
                 .to_string(),
         );
     };
-    let keyword = crate::search::bm25::bm25_search(db, &query.query, limit)
+    let keyword = crate::search::bm25::bm25_search(&db, &query.query, limit)
         .into_iter()
         .map(|hit| (hit.chunk_id, hit.score))
         .collect();
     let exact = if query.want_exact {
-        crate::search::exact::exact_lane_chunk_ids(store, db, &query.query, limit)
+        crate::search::exact::exact_lane_chunk_ids(store, &db, &query.query, limit)
     } else {
         Vec::new()
     };

src/comms/index_read_proto.rs

@@ -50,8 +50,9 @@ pub enum IndexReadQuery {
     GrepBloom {
         /// The grep regex source.
         pattern: String,
-        /// Files to test, at most [`MAX_GREP_BLOOM_PATHS`].
-        paths: Vec<RelPath>,
+        /// Files to test with the hash and stamp they were indexed under, at most
+        /// [`MAX_GREP_BLOOM_PATHS`].
+        entries: Vec<crate::index::grep_bloom::GrepEntry>,
     },
     /// The `implementations_by_trait` scan behind `implementations`.
     ImplScan {

src/comms/workspace_pool_tests.rs

@@ -6,6 +6,7 @@
 use std::time::Duration;
 
 use super::*;
+use crate::index::view::IndexView;
 
 /// History host for the fixtures, which care about the read stack and not about git history.
 ///
@@ -932,7 +933,7 @@ fn a_workspace_opened_over_a_wiped_index_is_marked_and_rebuilt_by_the_next_resca
     assert!(!entry.rebuild_pending.load(std::sync::atomic::Ordering::Acquire));
     let repopulated = pool
         .with_workspace(ws.path(), |s| {
-            !s.index_db.as_ref().expect("index").symbols_index_is_empty() && !s.index_rebuild_needed
+            !s.index_db.as_ref().expect("index").view().symbols_index_is_empty() && !s.index_rebuild_needed
         })
         .expect("workspace");
     assert!(repopulated, "the fjall index holds symbols again");