chore(experiments): lab-only D-RST-3 borrowed pattern-CSR view (deferred, evidence only)

experiments/relstore-visit/.gitignore

@@ -0,0 +1,2 @@
+target/
+Cargo.lock

experiments/relstore-visit/Cargo.toml

@@ -0,0 +1,27 @@
+[package]
+name = "petgraph-relstore-visit"
+version = "0.0.1"
+edition = "2024"
+rust-version = "1.99"
+publish = false
+description = "D-RST-3: a borrowed, version-pinned pattern-CSR view that petgraph's unmodified BFS/DFS traverse"
+
+# Isolated on purpose, like experiments/lance-fold-hop: building the petgraph
+# workspace never pulls in lance-graph. The `quack` feature is the only way the
+# substrate enters, and it is off by default.
+[workspace]
+
+[features]
+default = []
+# The F2b-shaped falsifier: a Quack compiled one-hop over the SAME prepared
+# buffers. Pinned to lance-graph main at the D-RMAP-1 merge (fcd1f60e).
+quack = ["dep:lance-graph-quack", "dep:lance-graph-mask-risc"]
+
+[dependencies]
+petgraph = { path = "../../crates/petgraph", default-features = false, features = ["std"] }
+fixedbitset = { version = "0.5.7", default-features = false }
+lance-graph-quack = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "fcd1f60ea863da56a18032baf3989b9aae0e8c02", optional = true }
+lance-graph-mask-risc = { git = "https://github.com/AdaWorldAPI/lance-graph", rev = "fcd1f60ea863da56a18032baf3989b9aae0e8c02", optional = true }
+
+[[example]]
+name = "relstore_cost"

experiments/relstore-visit/README.md

@@ -0,0 +1,113 @@
+# relstore-visit (D-RST-3)
+
+A borrowed, version-pinned pattern-CSR view that petgraph's own `Bfs` and
+`Dfs` traverse unchanged. This is D-RST-3 of the lance-graph plan
+`.claude/plans/relstore-quack-petgraph-traversal-v1.md` (#1622): the view only,
+before the relationship store (D-RST-1/2) exists.
+
+It is isolated like `experiments/lance-fold-hop` (its own `[workspace]`), and
+it does not use the quarantined `OutgoingHop`. The default build does not
+depend on lance-graph. The `quack` feature adds `lance-graph-quack` and
+`lance-graph-mask-risc` at lance-graph main `fcd1f60e`.
+
+```sh
+cargo test --release                  # 9 tests
+cargo test --release --features quack # + the shared-layout falsifier
+cargo run --release --features quack --example relstore_cost
+```
+
+## What is here
+
+| item | role |
+|---|---|
+| `RelationStore::prepare` | Stands in for the store's layout preparation. Builds `u32` offsets, `u32` neighbour slots, `u64` fact ids and the `u32` source lane, ordered by `(src, fact id)`. Counted by `prepares()`. |
+| `prepare_from_blasgraph_parts` | The same layout from BlasGraph `CsrStorage`'s index arrays (`nrows`, `ncols`, `row_ptrs`, `col_indices`), in its vocabulary. There is no values parameter, so the 2 KiB `BitVec`s cannot be passed in. |
+| `PreparedRelation::view(version)` | A `PatternCsr<'_>`: three borrowed slices and a version. Refused (`Stale`) for any other version. |
+| `PatternCsr` | `GraphBase`, `GraphRef`, `IntoNeighbors`, `Visitable` (`FixedBitSet`), `NodeIndexable`, `NodeCount`, `NodeCompactIndexable`, `Data`, `IntoEdgeReferences`, `IntoEdges`. `NodeId = u32`, `EdgeId = u64` = the fact id. |
+| `FactEdge::row()` | The physical row, reported apart from the fact id. |
+| `quack_hop::one_hop` (feature `quack`) | `src == s` lowered by Quack, executed by mask-risc over the prepared `src` lane; neighbours read from the prepared `dst` lane at the kept rows. |
+
+Refusals, never silent drops: `OutOfDomain { fact, src, dst, n_nodes }`,
+`DuplicateFact`, `Stale { prepared, asked }`, `TooLarge`, `MalformedCsr`.
+
+## Tests (`tests/visit.rs`)
+
+The oracle is a `DiGraph` built from the facts, not from the layout. Each
+source's edges are inserted in reverse `(src, fact id)` order, because
+`DiGraph` yields neighbours in reverse insertion order; the view and the
+oracle must then visit in exactly the same order.
+
+- BFS and DFS orders and out-edge sets equal the oracle from every start:
+  a hand fixture (parallel edges, a self-loop, an isolated node, a one-way
+  edge) and three random relations (checked to contain parallels and loops).
+- Empty relations, including zero nodes.
+- An out-of-domain endpoint is refused with its values (both endpoints), and
+  the last in-domain node is accepted. Refused prepares are not counted.
+- A stale relation or revision is refused.
+- Duplicate fact ids are refused.
+- Reversed arrival order rebuilds the identical layout; after compaction
+  (every third fact kept, new revision) every fact keeps its endpoints while
+  most rows move.
+- The view's buffer pointers equal the prepared buffers; every neighbour
+  slice lies inside the prepared `dst` buffer; building a view and walking
+  every neighbour and out-edge list allocate nothing (thread-local counting
+  allocator).
+- BlasGraph index arrays give the same layout as `prepare`; malformed parts
+  are refused.
+- `quack`: for every node, the Quack hop equals the view's neighbours, reads
+  the same `dst` buffer at the same version, and the store prepared once.
+
+Disable runs (each guard broken alone, its test red, then restored):
+
+| guard broken | test that failed |
+|---|---|
+| destination domain check removed | `an_out_of_domain_endpoint_is_refused_never_dropped` |
+| stale-version check removed | `a_view_is_refused_for_any_other_version` |
+| duplicate-id check removed | `duplicate_fact_ids_are_refused` |
+| rows ordered by `src` only (arrival order within a source) | `fact_ids_survive_reordering_and_compaction_rows_do_not` |
+| view holds a copy of `dst` | `the_view_borrows_and_allocates_nothing` |
+| edge id = physical row | `fact_ids_survive_reordering_and_compaction_rows_do_not` |
+| Quack hop reads a copy of `dst` | `petgraph_and_quack_read_one_prepared_layout` |
+
+## Measurements (`examples/relstore_cost.rs`)
+
+Release, one machine, medians of 7; ranges over three runs. Random relation,
+`m` facts over `n` nodes, parallels and loops included. Every BFS is checked
+to visit the same node set in all three arms.
+
+| stage | n 1k, m 10k | n 100k, m 1M |
+|---|---|---|
+| prepare (sort, offsets, 3 lanes) | 0.66–0.72 ms (6 allocs) | 308–396 ms (6 allocs) |
+| view construction | 0.035 µs, 0 allocs | 0.035 µs, 0 allocs |
+| neighbour walk, all nodes: view / Csr / DiGraph | 1.9 / 1.7 / 3.1 ns per edge | 2.0 / 1.8–1.9 / 5.5–7.2 ns per edge |
+| BFS from 0: view / Csr / DiGraph | 29–47 / 32–48 / 48–64 µs | 6.8–12.9 / 7.2–14.8 / 42–51 ms |
+| BFS allocations (view) | 10 | 17 |
+| layout bytes | 16.4 B per edge | 16.4 B per edge |
+| 64 one-hops: Quack / view | 134–137 µs / 0.3 µs | 14.0–18.2 ms / 0.3 µs |
+
+What they say:
+
+- The borrowed view costs the same as petgraph's `Csr` to walk and to
+  traverse, and allocates only petgraph's traversal state. The first run
+  showed the view 2.7× faster than `Csr` on BFS; repeat runs put them within
+  noise of each other, so that number was noise.
+- Preparation is the cost: 300–400 ns per edge at 1M, about 50× the `Csr`
+  arm's build (the `Csr` arm is given pre-sorted, deduplicated pairs, so
+  this is not like for like). It is an indirect sort plus a second sort for
+  the duplicate-id check. This is measured, not optimized; per the plan, an
+  optimization waits for a workload that prepares often.
+- The Quack hop is correct and reads the same buffers, but it scans the
+  whole `src` lane per query (O(E), ~3.5 ns per row) and lowers a program
+  per query (3 allocations). The view answers in O(degree). The lane is
+  sorted by `src`, so a range read would close the gap, but Quack cannot
+  assume that without a sealed ordering witness (D-DIAMOND `Cmp::Range` via
+  an ordered-lane witness). That is an admission question for D-RST-1/2.
+
+## Limits
+
+- BlasGraph is not a dependency (lance-graph core pulls lance, datafusion and
+  lancedb). Its vocabulary is reused by signature; the parts constructor is a
+  preparation (`usize` to `u32`), not a borrow.
+- `RelationStore` and `RelationVersion` are stand-ins. The real version comes
+  from the store's seal (`LayoutSeal.built_from`), which does not exist yet.
+- Out-edges only. `IntoNeighborsDirected` needs a second (reverse) layout.

experiments/relstore-visit/clippy.toml

@@ -0,0 +1,2 @@
+# This experiment is rust-version 1.99 (see Cargo.toml), not the fork crates' 1.85.
+msrv = "1.99"

experiments/relstore-visit/examples/relstore_cost.rs

@@ -0,0 +1,248 @@
+//! Costs of the borrowed pattern-CSR view, each stage measured on its own.
+//!
+//! Run: `cargo run --release --example relstore_cost` (add `--features quack`
+//! for the Quack one-hop arm). One machine, warm caches, medians of REPS runs.
+//! Every traversal is checked against the petgraph `DiGraph` arm before it is
+//! timed.
+
+use std::alloc::{GlobalAlloc, Layout, System};
+use std::hint::black_box;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use std::time::{Duration, Instant};
+
+use petgraph::csr::Csr;
+use petgraph::graph::{DiGraph, NodeIndex};
+use petgraph::visit::{Bfs, IntoNeighbors};
+use petgraph_relstore_visit::{Fact, RelationStore, RelationVersion};
+
+struct Counting;
+static ALLOCS: AtomicUsize = AtomicUsize::new(0);
+
+unsafe impl GlobalAlloc for Counting {
+    unsafe fn alloc(&self, l: Layout) -> *mut u8 {
+        ALLOCS.fetch_add(1, Ordering::Relaxed);
+        unsafe { System.alloc(l) }
+    }
+    unsafe fn dealloc(&self, p: *mut u8, l: Layout) {
+        unsafe { System.dealloc(p, l) }
+    }
+}
+
+#[global_allocator]
+static GLOBAL: Counting = Counting;
+
+const REPS: usize = 7;
+const V1: RelationVersion = RelationVersion {
+    relation: 1,
+    revision: 1,
+};
+
+fn median(mut v: Vec<Duration>) -> Duration {
+    v.sort();
+    v[v.len() / 2]
+}
+
+/// Median time and the allocation count of the LAST run.
+fn time<T>(mut f: impl FnMut() -> T) -> (Duration, usize, T) {
+    let mut ts = Vec::with_capacity(REPS);
+    let mut last = None;
+    let mut allocs = 0;
+    for _ in 0..REPS {
+        let a = ALLOCS.load(Ordering::Relaxed);
+        let t = Instant::now();
+        let v = black_box(f());
+        ts.push(t.elapsed());
+        allocs = ALLOCS.load(Ordering::Relaxed) - a;
+        last = Some(v);
+    }
+    (median(ts), allocs, last.unwrap())
+}
+
+fn us(d: Duration) -> f64 {
+    d.as_secs_f64() * 1e6
+}
+
+fn facts(n: u32, m: usize, seed: u64) -> Vec<Fact> {
+    let mut s = seed;
+    let mut next = || {
+        s = s.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1);
+        (s >> 33) as u32
+    };
+    (0..m)
+        .map(|i| Fact {
+            id: (i as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15),
+            src: next() % n,
+            dst: next() % n,
+        })
+        .collect()
+}
+
+fn section(n: u32, m: usize) {
+    let fs = facts(n, m, 42);
+    println!("\n## n {n}, m {m}");
+
+    let mut store = RelationStore::new();
+    let (prep, prep_allocs, rel) = time(|| store.prepare(V1, n, &fs).unwrap());
+    // Everything below reads this layout; none of it may prepare another.
+    let prepared = store.prepares();
+    let (view_t, view_allocs, v) = time(|| rel.view(V1).unwrap());
+
+    // Arms built from the same facts.
+    let (graph_t, _, g) = time(|| {
+        let mut g = DiGraph::<(), u64>::with_capacity(n as usize, fs.len());
+        for _ in 0..n {
+            g.add_node(());
+        }
+        let mut sorted = fs.clone();
+        sorted.sort_by_key(|f| (f.src, std::cmp::Reverse(f.id)));
+        for f in sorted {
+            g.add_edge(
+                NodeIndex::new(f.src as usize),
+                NodeIndex::new(f.dst as usize),
+                f.id,
+            );
+        }
+        g
+    });
+    let mut pairs: Vec<(u32, u32)> = fs.iter().map(|f| (f.src, f.dst)).collect();
+    pairs.sort_unstable();
+    pairs.dedup();
+    let (csr_t, _, csr) =
+        time(|| Csr::<(), (), petgraph::Directed, u32>::from_sorted_edges(&pairs).unwrap());
+
+    // Correctness before timing: BFS orders agree with the DiGraph arm.
+    for s in [0, n / 2, n - 1] {
+        let mut a = Bfs::new(v, s);
+        let mut b = Bfs::new(&g, NodeIndex::new(s as usize));
+        loop {
+            let (x, y) = (a.next(v), b.next(&g));
+            assert_eq!(x, y.map(|y| y.index() as u32));
+            if x.is_none() {
+                break;
+            }
+        }
+    }
+
+    let walk = |f: &dyn Fn(u32) -> u64| (0..n).map(f).fold(0u64, |a, b| a.wrapping_add(b));
+    let (walk_view, walk_view_allocs, s1) =
+        time(|| walk(&|u| v.neighbors(u).map(u64::from).sum::<u64>()));
+    let (walk_graph, _, s2) = time(|| {
+        walk(&|u| {
+            g.neighbors(NodeIndex::new(u as usize))
+                .map(|x| x.index() as u64)
+                .sum::<u64>()
+        })
+    });
+    let (walk_csr, _, _) = time(|| {
+        walk(&|u| {
+            csr.neighbors_slice(u)
+                .iter()
+                .map(|&x| u64::from(x))
+                .sum::<u64>()
+        })
+    });
+    assert_eq!(s1, s2);
+
+    let bfs_all = |start: u32| {
+        let mut b = Bfs::new(v, start);
+        let mut k = 0u32;
+        while let Some(x) = b.next(v) {
+            k = k.wrapping_add(x);
+        }
+        k
+    };
+    let (bfs_view, bfs_view_allocs, k_view) = time(|| bfs_all(0));
+    let (bfs_graph, _, k_graph) = time(|| {
+        let mut b = Bfs::new(&g, NodeIndex::new(0));
+        let mut k = 0u32;
+        while let Some(x) = b.next(&g) {
+            k = k.wrapping_add(x.index() as u32);
+        }
+        k
+    });
+    let (bfs_csr, _, k_csr) = time(|| {
+        let mut b = Bfs::new(&csr, 0u32);
+        let mut k = 0u32;
+        while let Some(x) = b.next(&csr) {
+            k = k.wrapping_add(x);
+        }
+        k
+    });
+
+    // The three BFS arms visit the same node set (same wrapping sum).
+    assert_eq!((k_view, k_view), (k_graph, k_csr));
+    let per_edge = |d: Duration| d.as_secs_f64() * 1e9 / m as f64;
+    println!("stage, us, allocs, ns/edge");
+    println!(
+        "prepare (sort + offsets + 3 lanes), {:.1}, {prep_allocs}, {:.2}",
+        us(prep),
+        per_edge(prep)
+    );
+    println!("view construction, {:.3}, {view_allocs}, -", us(view_t));
+    println!(
+        "build DiGraph arm, {:.1}, -, {:.2}",
+        us(graph_t),
+        per_edge(graph_t)
+    );
+    println!(
+        "build Csr arm ({} unique pairs of {m}), {:.1}, -, {:.2}",
+        pairs.len(),
+        us(csr_t),
+        per_edge(csr_t)
+    );
+    println!(
+        "neighbour walk, all nodes: view {:.1} ({walk_view_allocs} allocs) | DiGraph {:.1} | Csr {:.1}  [{:.2} / {:.2} / {:.2} ns/edge]",
+        us(walk_view),
+        us(walk_graph),
+        us(walk_csr),
+        per_edge(walk_view),
+        per_edge(walk_graph),
+        per_edge(walk_csr),
+    );
+    println!(
+        "BFS from 0: view {:.1} ({bfs_view_allocs} allocs) | DiGraph {:.1} | Csr {:.1}",
+        us(bfs_view),
+        us(bfs_graph),
+        us(bfs_csr)
+    );
+    println!(
+        "layout {} B for {m} edges ({:.1} B/edge)",
+        rel.bytes(),
+        rel.bytes() as f64 / m as f64
+    );
+
+    #[cfg(feature = "quack")]
+    {
+        use petgraph_relstore_visit::quack_hop::one_hop;
+        let sources: Vec<u32> = (0..64).map(|i| (i * 7919) % n).collect();
+        let mut out = Vec::new();
+        let (quack, quack_allocs, _) = time(|| {
+            let mut k = 0usize;
+            for &s in &sources {
+                out.clear();
+                one_hop(&rel, V1, s, &mut out).unwrap();
+                k += out.len();
+            }
+            k
+        });
+        let (view_hop, _, _) = time(|| {
+            sources
+                .iter()
+                .map(|&s| v.neighbors(s).count())
+                .sum::<usize>()
+        });
+        println!(
+            "64 one-hop queries: Quack (lower + full src scan + dst read) {:.1} us ({quack_allocs} allocs) | view neighbours {:.2} us",
+            us(quack),
+            us(view_hop)
+        );
+    }
+    assert_eq!(store.prepares(), prepared, "a consumer prepared a second layout");
+}
+
+fn main() {
+    println!("relstore-visit cost | release, medians of {REPS}");
+    for (n, m) in [(1_000, 10_000), (100_000, 1_000_000)] {
+        section(n, m);
+    }
+}