|
| 1 | +# graphrag + doc-retrieval on the V3 SoA — integration plan v1 |
| 2 | + |
| 3 | +> **Status:** DESIGN — evaluation + wave plan (no bytes land here). Aligned to |
| 4 | +> **#708 / D-TRI-6** (`E-RUNG-ASCENT-WIRED-1`, `E-MASLOW-PYRAMID-OF-COGNITION-1`) |
| 5 | +> and the `triangle-tenants-gestalt-separation-v1` Maslow pyramid. Any OGAR |
| 6 | +> classid touch is **mint-gated** (batched, baton-audited). Probe-first: the |
| 7 | +> **graph-load-bearing baseline** is the FIRST deliverable, before any Leiden/PPR |
| 8 | +> code. |
| 9 | +> **Grounding:** three Opus reader-agents (v3 waves+census · CausalEdge64/witness |
| 10 | +> substrate · OGAR↔lance-graph boundary) + `.claude/knowledge/graphrag-rs-inventory.md` |
| 11 | +> (`E-V3-GRAPHRAG-INV-1`, the pre-existing automataIA/graphrag-rs audit) + the |
| 12 | +> automataIA repo survey (graphrag-rs ★519, graph-librarian-rs, wasm-typst-studio-rs, |
| 13 | +> lodviz-rs). Citations are as-of `efd21d2` (#707) — verify before coding. |
| 14 | +
|
| 15 | +## §0. The reframe — direction correction (READ FIRST) |
| 16 | + |
| 17 | +The task said "create crates **inside lance-graph feeding into** ogar-doc." Run |
| 18 | +through `assembler-vs-storage-substrate.md` + `compilation-vs-runtime-substrate.md`, |
| 19 | +that **inverts the dependency** and is WRONG-SHELF. The corrected direction: |
| 20 | + |
| 21 | +- **OGAR = assembler.** It OWNS document ingestion, the `document 0x080B` mint, |
| 22 | + and persistence — already built as `ogar-from-docv1` → `ogar-doc-ir` → |
| 23 | + `ogar-doc` (doc-W4), all in `/home/user/OGAR/crates/`. A lance-graph |
| 24 | + doc-transcode crate would **duplicate `ogar-from-docv1`** and invert the dep. |
| 25 | +- **lance-graph = spine.** It provides **contract types** (`ClassView`, facet / |
| 26 | + envelope, masks) that `ogar-doc` builds against — OGAR consumes |
| 27 | + `lance-graph-contract`, **never the reverse** — **plus query capability** over |
| 28 | + the *calcified* substrate. |
| 29 | +- **Lance = calcification.** "There is no ingest API." The SoA/Lance write is |
| 30 | + calcification by the batch writer on-behalf of a mailbox owner. |
| 31 | +- **The door-knocking test forbids** any build step or typed-value constructor |
| 32 | + asking the running substrate for permission. `doc.v1 → ogar_doc_ir` is a plain |
| 33 | + Rust fn, unit-testable with nothing running. |
| 34 | + |
| 35 | +**Therefore the honest shape of "feeds both ogar-doc and graphrag":** a single |
| 36 | +**query/view surface** over the calcified document-graph, consumed by two |
| 37 | +readers — `crates/graphrag` (retrieval) and OGAR `ogar-doc`'s |
| 38 | +`reconstruct_document` ("re-issue with updated knowledge" / "documents in this |
| 39 | +community"). `ogar-doc` calls it **via a `lance-graph-contract` trait** |
| 40 | +(`DocGraphQuery`), so OGAR still depends only on the contract. graphrag provides |
| 41 | +the impl. **Neither ingests; both read.** |
| 42 | + |
| 43 | +## §1. Evaluation — what to build, what to reject |
| 44 | + |
| 45 | +| Candidate crate | Verdict | Why | |
| 46 | +|---|---|---| |
| 47 | +| lance-graph doc-transcode / perceptual-IR / `document`-mint crate | **REJECT** | Duplicates OGAR `ogar-from-docv1`/`ogar-doc-ir`/`ogar-doc`; inverts the dep; WRONG-SHELF. Doc ingestion is assembler-side. | |
| 48 | +| Monolithic `graphrag` (extraction→graph→retrieval from scratch, à la graphrag-rs) | **REJECT** | Extraction, SPO edges, fact store, vector/CAM-PQ retrieval already EXIST (§2). Violates *wire-don't-invent*. graphrag-rs = REUSE-AS-REFERENCE only (`graphrag-rs-inventory.md`); its `LanceDBStore` is a stub, its `InferenceEngine` is the no-singleton anti-exhibit. | |
| 49 | +| **`crates/graphrag`** — thin retrieval **orchestrator**, query-side only | **BUILD** | Binds existing legs onto the #708 rung-ascent loop; BUILDS only the 3 genuine gaps (§2). Reads calcified substrate; never ingests. | |
| 50 | +| **`DocGraphQuery` trait** in `lance-graph-contract` + a `DocGraphView` read projection | **BUILD (thin)** | The shared surface both graphrag and `ogar-doc` consume. Keeps OGAR on the contract, not the impl. | |
| 51 | +| New lance-graph OGAR-seam crate | **REJECT** | `lance-graph-ogar` already IS the seam (re-exports ogar-vocab Class/codebook + `ClassView` impl). Extend it if needed. | |
| 52 | + |
| 53 | +**Net: two BUILD targets** — `crates/graphrag` (the orchestrator + the 3 gaps) and |
| 54 | +a thin `DocGraphQuery` contract surface. Everything else is wiring. |
| 55 | + |
| 56 | +## §2. Substrate reuse — the census (build only the gaps) |
| 57 | + |
| 58 | +REUSE (do **not** rebuild — the substrate provides these): |
| 59 | + |
| 60 | +- **SPO edge = `causal_edge::CausalEdge64`** (8 B, `#[repr(transparent)]`). v2 |
| 61 | + layout (default `causal-edge-v2-layout`): S/P/O palette idx (bits 0-23), NARS |
| 62 | + **frequency**/**confidence** u8 (24-39), Pearl-2³ causal mask (40-42), |
| 63 | + direction triad (43-45), inference mantissa i4 (46-49), plasticity (50-52), |
| 64 | + **W-slot witness handle** (53-58), truth-band lens (59-60), spare (61-63). |
| 65 | + **Disambiguation hazard:** two `CausalEdge64` exist — use `causal_edge` (SPO |
| 66 | + palette), NOT `thinking_engine::layered` (8×u8 strength vector). |
| 67 | +- **NARS truth = native `(freq u8, conf u8)`** in the edge (bits 24-39); |
| 68 | + `expectation()`, `evidence_weight()` on `CausalEdge64`. Don't add a |
| 69 | + confidence/weight field. |
| 70 | +- **Provenance/witness.** `EpisodicWitness64` is **not a symbol** — it names a |
| 71 | + *queued* SoA column (`soa_view.rs` `episodic_witness()` deferred). Today use |
| 72 | + `EpisodicEdges64` (4×16-bit MRU episodic edges) + `WitnessTable<64>` / |
| 73 | + `WitnessEntry{mailbox_ref, spo_fact_ref}` resolving the CE64 v2 W-slot. Don't |
| 74 | + add a provenance field; **materializing the column is a candidate probe**, not |
| 75 | + an assumption. |
| 76 | +- **Fact store = AriGraph `TripletGraph` + SPO-G quad.** `Triplet` + G-slot |
| 77 | + (`ContextTag::{Observation, Intervention}`) = the quad; `CounterfactualSpoG` |
| 78 | + from `TripletGraph::intervene_on` (Pearl rung-2). Promote a CE64 to a graph |
| 79 | + edge via `spo_bridge::promote_to_spo` past the truth gate. Cold, in |
| 80 | + lance-graph core — **not** a SoA column. |
| 81 | +- **Read surface = `MailboxSoaView`** (zero-copy; `edges:[CausalEdge64;N]` is the |
| 82 | + EdgeColumn). Don't re-encode to Arrow. Node adjacency = the 16-byte `EdgeBlock` |
| 83 | + (coarse, key-side); identity/class = `NodeGuid` / `class_id` u16 (resolves via |
| 84 | + `lance-graph-ontology`). Three edge encodings — EdgeBlock (adjacency) / CE64 |
| 85 | + (causal weight) / EpisodicEdges64+WitnessTable (provenance) — **never conflate**. |
| 86 | +- **Extraction (upstream proposer) = `lance-graph-arm-discovery`** (Aerial+ |
| 87 | + transcode, integer codebook-distance oracle → `{s,p,o,f,c}` ndjson NARS-truth |
| 88 | + SPO candidates) + `nsm/parser.rs` PoS-FSM→SPO. No-LLM, workspace-native. |
| 89 | +- **Vector retrieval = CAM-PQ** (`cam_pq/{ivf,storage,udf}`; codec ICC 0.9999, |
| 90 | + `cam-pq-production-wiring-v1`). *(The exact vector-search entry symbol moved |
| 91 | + since the agent read — verify the current entry point.)* |
| 92 | + |
| 93 | +BUILD (the genuine gaps — each gated on §6 P0): |
| 94 | + |
| 95 | +1. **Hierarchical Leiden community detection** — in-tree is example-only |
| 96 | + (`jc/examples/splat_louvain_modularity.rs`, `splat_lpa_label_propagation.rs`); |
| 97 | + graphrag-rs `leiden.rs` is single-level, REUSE-AS-REFERENCE. Build hierarchical |
| 98 | + over the CE64/SPO-G graph. |
| 99 | +2. **HippoRAG-PPR** (personalized PageRank with reset distribution) — plain |
| 100 | + `hdr_pagerank` (`blasgraph/ops.rs`) + `ScentCsr::spmv` exist; the reset/seed |
| 101 | + distribution + dual-signal (passage-weight) is the gap. Reference |
| 102 | + graphrag-rs `hipporag_ppr.rs`. |
| 103 | +3. **BM25 keyword arm** — ABSENT; small pure function. Reference graphrag-rs |
| 104 | + `KeywordExtractor`. |
| 105 | + |
| 106 | +## §3. The #708 alignment — retrieval IS rung ascent |
| 107 | + |
| 108 | +**Do not build a bespoke retrieval-escalation ladder.** graphrag retrieval |
| 109 | +dispatches through the **`RungElevator`** wired by #708 (D-TRI-6). Per #708: |
| 110 | +`on_gate()` advances one rung per dispatch; the current rung selects the cycle's |
| 111 | +cascade breadth via `rung_widened_layer_mask` — a UNION over the **8-bit |
| 112 | +predicate-plane mask (CAUSES..BECOMES)**, **identity at base (zero regression), |
| 113 | +superset-monotone above**. `RungLevel` is canonical in `lance-graph-contract` |
| 114 | +(post-#708 dedup; thinking-engine re-exports). **BLOCK ascends, FLOW relaxes to |
| 115 | +base.** |
| 116 | + |
| 117 | +Map retrieval onto the Maslow pyramid (`triangle-tenants-gestalt-separation-v1` |
| 118 | +§3a; the rung-content ladder in `persona-vs-rung-ladder.md`): |
| 119 | + |
| 120 | +| Rung | Pyramid level | Retrieval action | Predicate-plane mask | |
| 121 | +|---|---|---|---| |
| 122 | +| 0–1 (base, FLOW) | observation / gestalt | CAM-PQ vector + BM25 surface lookup | identity | |
| 123 | +| 2 | SPO 2³ | SPO-G edge hop over `CausalEdge64` | widen (Pearl rung-1 planes) | |
| 124 | +| 3 | CE64 NARS candidates | HippoRAG-PPR + community-scoped expansion, NARS-truth-weighted | wider CAUSES..BECOMES union | |
| 125 | +| 3–4 | NARS candidate design / revision | community summaries (Rig oracle → compiled template, W3) | full | |
| 126 | +| 4 (apex) | counterfactual | Pearl rung-2 intervention (`ContextTag` G-slot) | — | |
| 127 | + |
| 128 | +**The graph is load-bearing precisely because BLOCK ascends the elevator** |
| 129 | +(retrieval surprise / low NARS confidence / contradiction widens the traversal). |
| 130 | +graphrag never re-decides the level — it reads the driver's `RungLevel` and |
| 131 | +supplies the wider graph walk. This is the anti-decorative-graph guarantee (vs |
| 132 | +the graph-librarian-rs anti-pattern where the graph is built but never traversed). |
| 133 | + |
| 134 | +## §4. Crate topology (recommended) |
| 135 | + |
| 136 | +``` |
| 137 | +crates/graphrag/ (NEW, lance-graph, W5 query consumer) |
| 138 | + src/ |
| 139 | + lib.rs |
| 140 | + retrieve.rs # the rung-ascent orchestrator (consumes RungElevator; the load-bearing walk) |
| 141 | + community.rs # BUILD: hierarchical Leiden over CE64/SPO-G |
| 142 | + ppr.rs # BUILD: HippoRAG reset-distribution PPR (atop hdr_pagerank/spmv) |
| 143 | + keyword.rs # BUILD: BM25 |
| 144 | + summarize.rs # community summaries via Rig oracle → compiled template (W3 dep) |
| 145 | + deps: lance-graph-contract, causal-edge, cognitive-shader-driver (RungElevator), |
| 146 | + lance-graph (arigraph query surface), bgz-tensor/cam_pq (vector), ndarray. |
| 147 | + NO ingest; NO OGAR dep; reads calcified substrate. |
| 148 | +
|
| 149 | +lance-graph-contract/ (EXTEND) |
| 150 | + src/doc_graph.rs # DocGraphQuery trait + DocGraphView read projection (the shared surface) |
| 151 | +
|
| 152 | +lance-graph-ogar/ (EXISTING seam — extend only if the trait needs OGAR ClassView glue) |
| 153 | +``` |
| 154 | + |
| 155 | +`ogar-doc` (OGAR) consumes `DocGraphQuery` **via the contract** — that is the |
| 156 | +correct "feeds ogar-doc": ogar-doc is a **caller** of graphrag's query surface, |
| 157 | +never fed data by a lance-graph crate. |
| 158 | + |
| 159 | +## §5. Wave sequencing (aligned to v3 W0–W6 + D-TRI) |
| 160 | + |
| 161 | +Neither crate is a new W-wave. Sequenced **after the W1 keystone** |
| 162 | +(`mailbox_owner()` shipped #631; batch-writer W1b in-PR): |
| 163 | + |
| 164 | +- **G0 — baseline probe (FIRST, no code): P-GRAPH-LOADBEARING** (§6). Gate on |
| 165 | + truth-architect / measurement-before-synthesis. |
| 166 | +- **D-GR-1** — `DocGraphQuery` trait + `DocGraphView` read projection in the |
| 167 | + contract (both consumers build against it). Zero SoA writes. |
| 168 | +- **D-GR-2** — `crates/graphrag` scaffold + `retrieve.rs` binding **existing** |
| 169 | + CAM-PQ + SPO-G hops onto the canonical `RungLevel` / `RungElevator` (#708). |
| 170 | + Mirrors the #708 settlement probe (BLOCK ascends → wider walk; FLOW at base). |
| 171 | +- **D-GR-3** — BUILD the 3 gaps (`community.rs` Leiden, `ppr.rs` HippoRAG, |
| 172 | + `keyword.rs` BM25), **each individually gated** on G0 showing it beats |
| 173 | + vector-only. |
| 174 | +- **D-GR-4** — `summarize.rs` community summaries via the Rig oracle → compiled |
| 175 | + template (depends on W3 `template-runtime`). |
| 176 | +- **D-GR-5** — wire `ogar-doc` `reconstruct_document` + "similar/related |
| 177 | + documents" to `DocGraphQuery` (cross-repo; baton-audited classid handshake). |
| 178 | + |
| 179 | +Depends-on: W1b (any persisted result is born-stamped), W2 (retrieval cycles = |
| 180 | +kanban lanes if a cycle persists), W3 (summaries = compiled templates). Composes |
| 181 | +**atop** `oxigraph-arigraph-cognitive-shader-soa-merge-v1` (that plan builds the |
| 182 | +unified SoA context; graphrag is the retrieval layer over it — §8). |
| 183 | + |
| 184 | +## §6. Falsifiers / probes (probe-first) |
| 185 | + |
| 186 | +- **P-GRAPH-LOADBEARING (G0, the gate).** On a document corpus, measure retrieval |
| 187 | + quality **with vs without** graph traversal (vector-only vs vector+SPO-G+PPR). |
| 188 | + KILL condition: if the graph does not beat vector-only on multi-hop/global |
| 189 | + questions, do **not** build Leiden/PPR — the graph would be decorative |
| 190 | + (graph-librarian-rs anti-pattern). Mirrors the 1BRC addressing-tax probe. |
| 191 | +- **P-RUNG-RETRIEVAL.** Hard/contradictory query → BLOCK → elevator ascends → |
| 192 | + wider CAUSES..BECOMES mask → higher recall; easy query → FLOW → stays at base |
| 193 | + (identity mask, cheap). Mirrors #708's D-TRI-6 settlement probe. |
| 194 | +- **P-PPR-MULTIHOP.** HippoRAG-PPR beats vector-only on 2+-hop questions. |
| 195 | +- **P-COMMUNITY-GLOBAL.** Leiden community summaries beat flat chunk-concat on |
| 196 | + global/thematic questions (the LightRAG dual-level claim). |
| 197 | +- **jc battery** (ICC / Spearman / Cronbach) before any new lane-reading backs a |
| 198 | + claim (v3 standing gate). |
| 199 | + |
| 200 | +## §7. Alignment hazards (and how this plan avoids each) |
| 201 | + |
| 202 | +1. **write-on-behalf / born-stamped (W1 keystone).** graphrag is read-mostly; the |
| 203 | + only writes (persisted community membership / summaries) route the batch-writer |
| 204 | + `cast(on_behalf = envelope.mailbox_owner(), …)` — never write-as-self (the smb |
| 205 | + `LanceConnector::upsert` ORPHAN-WRITE is the cautionary tale). v3-mailbox-warden |
| 206 | + gate. |
| 207 | +2. **No-singleton.** No global `Arc<RwLock<KnowledgeGraph>>`. Community/PPR operate |
| 208 | + over `MailboxSoA` rows + AriGraph; results are ephemeral or calcified |
| 209 | + per-mailbox. graphrag-rs `InferenceEngine` (graph-as-config-struct) is the |
| 210 | + explicit anti-exhibit. |
| 211 | +3. **16-byte facet never-widen + classid canon-high.** Reuse existing task |
| 212 | + classids (`0x080B`/`0x0807`/`0x0808`/`0x080A`, hi-u16 = concept, canon-high). A |
| 213 | + persisted community-membership lane = a **new `ValueTenant` discriminant in the |
| 214 | + 328 B headroom** (additive-at-end, `ENVELOPE_LAYOUT_VERSION` unchanged), |
| 215 | + field-isolation-matrix gated by v3-envelope-auditor. `u8:u8` never widened to |
| 216 | + u16/u24. |
| 217 | +4. **Probe-first / wire-don't-invent.** COMPONENT-MAP + `graphrag-rs-inventory.md` |
| 218 | + ARE the precomputed "does it exist" search; §2 leans on them. G0 gates code. |
| 219 | +5. **CausalEdge64 duplication + cross-repo baton.** Always qualify |
| 220 | + `causal_edge::CausalEdge64`. Any OGAR classid for the D-GR-5 seam is a |
| 221 | + **batched** mint + sync-fuse, baton-handoff-auditor gated. |
| 222 | + |
| 223 | +## §8. Relationship to existing plans |
| 224 | + |
| 225 | +- **`graphrag-rs-inventory.md`** (`E-V3-GRAPHRAG-INV-1`) — the component-by-component |
| 226 | + automataIA/graphrag-rs audit. **Mandatory pre-read**; this plan does not re-fork. |
| 227 | +- **`oxigraph-arigraph-cognitive-shader-soa-merge-v1`** — builds the unified |
| 228 | + SoA context (Oxigraph RDF + AriGraph episodes + SPO). graphrag is the |
| 229 | + **retrieval layer atop it**; they compose, not compete. |
| 230 | +- **`entropy-ladder-spo-rung-v1`** (R1 shipped) — SPO/NARS rung decomposition; |
| 231 | + the rung mapping in §3 must reconcile with its entropy-level = fact-position. |
| 232 | +- **`temporal-markov-and-style-classes-v1`** (ACTIVE) — governs the retrieval |
| 233 | + substrate (episodic axis = Lance versions); graphrag rides `temporal.rs`. |
| 234 | +- **`triangle-tenants-gestalt-separation-v1`** (D-TRI-1..6) + **#708** — the |
| 235 | + Maslow pyramid graphrag retrieval ascends (§3). |
| 236 | +- **`normalized-entity-holy-grail-v1`**, **`episodic-risc-spine-v1`**, |
| 237 | + **`cam-pq-production-wiring-v1`**, **`wikidata-lazy-spine-hydration-v1`** — |
| 238 | + adjacent; entity-resolution + episodic-addressing + codec + KG-hydration inputs. |
| 239 | + |
| 240 | +## §9. Board hygiene · D-ids · gate |
| 241 | + |
| 242 | +- D-ids: **D-GR-1..5** (this plan) — STATUS_BOARD rows on land. |
| 243 | +- New plan → INTEGRATION_PLANS.md prepend (this commit); EPIPHANIES entry only |
| 244 | + when a finding lands (probe result), not for the plan itself. |
| 245 | +- **Gate:** G0 (P-GRAPH-LOADBEARING) is the first deliverable — **no Leiden/PPR |
| 246 | + code before it is green.** D-GR-5 (OGAR seam) is mint-gated + baton-audited. |
| 247 | +- **Open questions:** (O1) does `ogar-doc` `reconstruct_document` genuinely need a |
| 248 | + runtime graph query, or is the "updated knowledge" re-issue a batch re-persist? |
| 249 | + (settles whether D-GR-5 is a live query seam or a re-run). (O2) materialize the |
| 250 | + `EpisodicWitness64` column now (helps PPR recency) or stay on |
| 251 | + `EpisodicEdges64`+`WitnessTable`? (probe). (O3) reconcile the §3 rung mapping |
| 252 | + with `entropy-ladder-spo-rung-v1`'s entropy levels — same ladder or two? |
0 commit comments