Skip to content

Commit bd7fa7f

Browse files
committed
docs(plan): graphrag + doc-retrieval on the V3 SoA — integration plan v1
Evaluation + wave plan for feeding a new query-side crates/graphrag and OGAR's ogar-doc from the existing SoA + CausalEdge64 + (queued) EpisodicWitness64 substrate, aligned to #708 / D-TRI-6 (the rung-ascent loop) and the v3 waves. Key rulings the plan encodes: - Direction reframe (assembler-vs-storage boundary): OGAR owns doc ingestion + the document 0x080B mint (ogar-from-docv1 -> ogar-doc-ir -> ogar-doc, built); lance-graph provides contract types OGAR consumes + query. graphrag is query-side ONLY, reads the calcified substrate, never ingests. A lance-graph doc-transcode crate is REJECTED (duplicates ogar-from-docv1, inverts the dep). - REUSE the substrate, BUILD only 3 gaps: hierarchical Leiden, HippoRAG-PPR reset-distribution, BM25. Extraction/SPO/fact-store/vector-CAM-PQ already exist; causal_edge::CausalEdge64 is the SPO edge (not the thinking-engine variant); EpisodicWitness64 is a queued column (today WitnessTable + EpisodicEdges64). graphrag-rs = REUSE-AS-REFERENCE (E-V3-GRAPHRAG-INV-1). - Retrieval = rung ascent (#708): dispatches through RungElevator + canonical RungLevel; BLOCK ascends and widens the CAUSES..BECOMES predicate-plane mask, FLOW relaxes to base. The graph is load-bearing because surprise ascends the elevator. Maps onto the triangle-tenants Maslow pyramid (D-TRI). - Probe-first: G0 P-GRAPH-LOADBEARING (graph vs vector-only) gates all Leiden/PPR code. Composes atop oxigraph-arigraph-cognitive-shader-soa-merge-v1. No bytes land; D-GR-5 (OGAR seam) mint-gated + baton-audited. Board hygiene: INTEGRATION_PLANS prepend in the same commit. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent efd21d2 commit bd7fa7f

2 files changed

Lines changed: 275 additions & 0 deletions

File tree

‎.claude/board/INTEGRATION_PLANS.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,26 @@
1+
## 2026-07-17 — graphrag-doc-retrieval-soa-integration v1 — DESIGN (probe-gated)
2+
3+
**Plan:** `.claude/plans/graphrag-doc-retrieval-soa-integration-v1.md`
4+
**Scope:** evaluation + wave plan for a query-side `crates/graphrag` (retrieval
5+
orchestrator) + a `DocGraphQuery` contract surface, both reading the *calcified*
6+
document-graph substrate. Reframes the task's "lance-graph feeds ogar-doc": OGAR
7+
owns doc ingestion + the `document 0x080B` mint (`ogar-from-docv1` →
8+
`ogar-doc-ir` → `ogar-doc`, all built); lance-graph provides the contract types
9+
OGAR consumes + query capability. graphrag is query-side ONLY — never ingests
10+
(door-knocking-compiler test; assembler-vs-storage boundary). REUSE:
11+
`causal_edge::CausalEdge64` (SPO edge), native NARS `(freq,conf)`,
12+
`WitnessTable`/`EpisodicEdges64` (provenance; `EpisodicWitness64` = a *queued*
13+
column), AriGraph `TripletGraph`+`spo_bridge` (fact store), `MailboxSoaView`,
14+
CAM-PQ, `lance-graph-arm-discovery` (extraction proposer). BUILD only the 3
15+
gaps: hierarchical Leiden, HippoRAG-PPR reset-distribution, BM25. **Retrieval =
16+
rung ascent** (#708 / D-TRI-6 `RungElevator` + canonical `RungLevel` +
17+
predicate-plane widen; BLOCK ascends, FLOW at base) up the §3a Maslow pyramid.
18+
Probe-first: **G0 P-GRAPH-LOADBEARING** (graph beats vector-only?) gates all
19+
Leiden/PPR code. graphrag-rs (automataIA) = REUSE-AS-REFERENCE
20+
(`E-V3-GRAPHRAG-INV-1`). Composes atop
21+
`oxigraph-arigraph-cognitive-shader-soa-merge-v1`. D-GR-1..5; D-GR-5 (OGAR seam)
22+
mint-gated + baton-audited. No bytes land.
23+
124
## 2026-07-17 — triangle-tenants-gestalt-separation v1 — DESIGN (mint-gated)
225

326
**Plan:** `.claude/plans/triangle-tenants-gestalt-separation-v1.md`
Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
# graphrag + doc-retrieval on the V3 SoA — integration plan v1
2+
3+
> **Status:** DESIGN — evaluation + wave plan (no bytes land here). Aligned to
4+
> **#708 / D-TRI-6** (`E-RUNG-ASCENT-WIRED-1`, `E-MASLOW-PYRAMID-OF-COGNITION-1`)
5+
> and the `triangle-tenants-gestalt-separation-v1` Maslow pyramid. Any OGAR
6+
> classid touch is **mint-gated** (batched, baton-audited). Probe-first: the
7+
> **graph-load-bearing baseline** is the FIRST deliverable, before any Leiden/PPR
8+
> code.
9+
> **Grounding:** three Opus reader-agents (v3 waves+census · CausalEdge64/witness
10+
> substrate · OGAR↔lance-graph boundary) + `.claude/knowledge/graphrag-rs-inventory.md`
11+
> (`E-V3-GRAPHRAG-INV-1`, the pre-existing automataIA/graphrag-rs audit) + the
12+
> automataIA repo survey (graphrag-rs ★519, graph-librarian-rs, wasm-typst-studio-rs,
13+
> lodviz-rs). Citations are as-of `efd21d2` (#707) — verify before coding.
14+
15+
## §0. The reframe — direction correction (READ FIRST)
16+
17+
The task said "create crates **inside lance-graph feeding into** ogar-doc." Run
18+
through `assembler-vs-storage-substrate.md` + `compilation-vs-runtime-substrate.md`,
19+
that **inverts the dependency** and is WRONG-SHELF. The corrected direction:
20+
21+
- **OGAR = assembler.** It OWNS document ingestion, the `document 0x080B` mint,
22+
and persistence — already built as `ogar-from-docv1` → `ogar-doc-ir` →
23+
`ogar-doc` (doc-W4), all in `/home/user/OGAR/crates/`. A lance-graph
24+
doc-transcode crate would **duplicate `ogar-from-docv1`** and invert the dep.
25+
- **lance-graph = spine.** It provides **contract types** (`ClassView`, facet /
26+
envelope, masks) that `ogar-doc` builds against — OGAR consumes
27+
`lance-graph-contract`, **never the reverse** — **plus query capability** over
28+
the *calcified* substrate.
29+
- **Lance = calcification.** "There is no ingest API." The SoA/Lance write is
30+
calcification by the batch writer on-behalf of a mailbox owner.
31+
- **The door-knocking test forbids** any build step or typed-value constructor
32+
asking the running substrate for permission. `doc.v1 → ogar_doc_ir` is a plain
33+
Rust fn, unit-testable with nothing running.
34+
35+
**Therefore the honest shape of "feeds both ogar-doc and graphrag":** a single
36+
**query/view surface** over the calcified document-graph, consumed by two
37+
readers — `crates/graphrag` (retrieval) and OGAR `ogar-doc`'s
38+
`reconstruct_document` ("re-issue with updated knowledge" / "documents in this
39+
community"). `ogar-doc` calls it **via a `lance-graph-contract` trait**
40+
(`DocGraphQuery`), so OGAR still depends only on the contract. graphrag provides
41+
the impl. **Neither ingests; both read.**
42+
43+
## §1. Evaluation — what to build, what to reject
44+
45+
| Candidate crate | Verdict | Why |
46+
|---|---|---|
47+
| lance-graph doc-transcode / perceptual-IR / `document`-mint crate | **REJECT** | Duplicates OGAR `ogar-from-docv1`/`ogar-doc-ir`/`ogar-doc`; inverts the dep; WRONG-SHELF. Doc ingestion is assembler-side. |
48+
| Monolithic `graphrag` (extraction→graph→retrieval from scratch, à la graphrag-rs) | **REJECT** | Extraction, SPO edges, fact store, vector/CAM-PQ retrieval already EXIST (§2). Violates *wire-don't-invent*. graphrag-rs = REUSE-AS-REFERENCE only (`graphrag-rs-inventory.md`); its `LanceDBStore` is a stub, its `InferenceEngine` is the no-singleton anti-exhibit. |
49+
| **`crates/graphrag`** — thin retrieval **orchestrator**, query-side only | **BUILD** | Binds existing legs onto the #708 rung-ascent loop; BUILDS only the 3 genuine gaps (§2). Reads calcified substrate; never ingests. |
50+
| **`DocGraphQuery` trait** in `lance-graph-contract` + a `DocGraphView` read projection | **BUILD (thin)** | The shared surface both graphrag and `ogar-doc` consume. Keeps OGAR on the contract, not the impl. |
51+
| New lance-graph OGAR-seam crate | **REJECT** | `lance-graph-ogar` already IS the seam (re-exports ogar-vocab Class/codebook + `ClassView` impl). Extend it if needed. |
52+
53+
**Net: two BUILD targets** — `crates/graphrag` (the orchestrator + the 3 gaps) and
54+
a thin `DocGraphQuery` contract surface. Everything else is wiring.
55+
56+
## §2. Substrate reuse — the census (build only the gaps)
57+
58+
REUSE (do **not** rebuild — the substrate provides these):
59+
60+
- **SPO edge = `causal_edge::CausalEdge64`** (8 B, `#[repr(transparent)]`). v2
61+
layout (default `causal-edge-v2-layout`): S/P/O palette idx (bits 0-23), NARS
62+
**frequency**/**confidence** u8 (24-39), Pearl-2³ causal mask (40-42),
63+
direction triad (43-45), inference mantissa i4 (46-49), plasticity (50-52),
64+
**W-slot witness handle** (53-58), truth-band lens (59-60), spare (61-63).
65+
**Disambiguation hazard:** two `CausalEdge64` exist — use `causal_edge` (SPO
66+
palette), NOT `thinking_engine::layered` (8×u8 strength vector).
67+
- **NARS truth = native `(freq u8, conf u8)`** in the edge (bits 24-39);
68+
`expectation()`, `evidence_weight()` on `CausalEdge64`. Don't add a
69+
confidence/weight field.
70+
- **Provenance/witness.** `EpisodicWitness64` is **not a symbol** — it names a
71+
*queued* SoA column (`soa_view.rs` `episodic_witness()` deferred). Today use
72+
`EpisodicEdges64` (4×16-bit MRU episodic edges) + `WitnessTable<64>` /
73+
`WitnessEntry{mailbox_ref, spo_fact_ref}` resolving the CE64 v2 W-slot. Don't
74+
add a provenance field; **materializing the column is a candidate probe**, not
75+
an assumption.
76+
- **Fact store = AriGraph `TripletGraph` + SPO-G quad.** `Triplet` + G-slot
77+
(`ContextTag::{Observation, Intervention}`) = the quad; `CounterfactualSpoG`
78+
from `TripletGraph::intervene_on` (Pearl rung-2). Promote a CE64 to a graph
79+
edge via `spo_bridge::promote_to_spo` past the truth gate. Cold, in
80+
lance-graph core — **not** a SoA column.
81+
- **Read surface = `MailboxSoaView`** (zero-copy; `edges:[CausalEdge64;N]` is the
82+
EdgeColumn). Don't re-encode to Arrow. Node adjacency = the 16-byte `EdgeBlock`
83+
(coarse, key-side); identity/class = `NodeGuid` / `class_id` u16 (resolves via
84+
`lance-graph-ontology`). Three edge encodings — EdgeBlock (adjacency) / CE64
85+
(causal weight) / EpisodicEdges64+WitnessTable (provenance) — **never conflate**.
86+
- **Extraction (upstream proposer) = `lance-graph-arm-discovery`** (Aerial+
87+
transcode, integer codebook-distance oracle → `{s,p,o,f,c}` ndjson NARS-truth
88+
SPO candidates) + `nsm/parser.rs` PoS-FSM→SPO. No-LLM, workspace-native.
89+
- **Vector retrieval = CAM-PQ** (`cam_pq/{ivf,storage,udf}`; codec ICC 0.9999,
90+
`cam-pq-production-wiring-v1`). *(The exact vector-search entry symbol moved
91+
since the agent read — verify the current entry point.)*
92+
93+
BUILD (the genuine gaps — each gated on §6 P0):
94+
95+
1. **Hierarchical Leiden community detection** — in-tree is example-only
96+
(`jc/examples/splat_louvain_modularity.rs`, `splat_lpa_label_propagation.rs`);
97+
graphrag-rs `leiden.rs` is single-level, REUSE-AS-REFERENCE. Build hierarchical
98+
over the CE64/SPO-G graph.
99+
2. **HippoRAG-PPR** (personalized PageRank with reset distribution) — plain
100+
`hdr_pagerank` (`blasgraph/ops.rs`) + `ScentCsr::spmv` exist; the reset/seed
101+
distribution + dual-signal (passage-weight) is the gap. Reference
102+
graphrag-rs `hipporag_ppr.rs`.
103+
3. **BM25 keyword arm** — ABSENT; small pure function. Reference graphrag-rs
104+
`KeywordExtractor`.
105+
106+
## §3. The #708 alignment — retrieval IS rung ascent
107+
108+
**Do not build a bespoke retrieval-escalation ladder.** graphrag retrieval
109+
dispatches through the **`RungElevator`** wired by #708 (D-TRI-6). Per #708:
110+
`on_gate()` advances one rung per dispatch; the current rung selects the cycle's
111+
cascade breadth via `rung_widened_layer_mask` — a UNION over the **8-bit
112+
predicate-plane mask (CAUSES..BECOMES)**, **identity at base (zero regression),
113+
superset-monotone above**. `RungLevel` is canonical in `lance-graph-contract`
114+
(post-#708 dedup; thinking-engine re-exports). **BLOCK ascends, FLOW relaxes to
115+
base.**
116+
117+
Map retrieval onto the Maslow pyramid (`triangle-tenants-gestalt-separation-v1`
118+
§3a; the rung-content ladder in `persona-vs-rung-ladder.md`):
119+
120+
| Rung | Pyramid level | Retrieval action | Predicate-plane mask |
121+
|---|---|---|---|
122+
| 0–1 (base, FLOW) | observation / gestalt | CAM-PQ vector + BM25 surface lookup | identity |
123+
| 2 | SPO 2³ | SPO-G edge hop over `CausalEdge64` | widen (Pearl rung-1 planes) |
124+
| 3 | CE64 NARS candidates | HippoRAG-PPR + community-scoped expansion, NARS-truth-weighted | wider CAUSES..BECOMES union |
125+
| 3–4 | NARS candidate design / revision | community summaries (Rig oracle → compiled template, W3) | full |
126+
| 4 (apex) | counterfactual | Pearl rung-2 intervention (`ContextTag` G-slot) | — |
127+
128+
**The graph is load-bearing precisely because BLOCK ascends the elevator**
129+
(retrieval surprise / low NARS confidence / contradiction widens the traversal).
130+
graphrag never re-decides the level — it reads the driver's `RungLevel` and
131+
supplies the wider graph walk. This is the anti-decorative-graph guarantee (vs
132+
the graph-librarian-rs anti-pattern where the graph is built but never traversed).
133+
134+
## §4. Crate topology (recommended)
135+
136+
```
137+
crates/graphrag/ (NEW, lance-graph, W5 query consumer)
138+
src/
139+
lib.rs
140+
retrieve.rs # the rung-ascent orchestrator (consumes RungElevator; the load-bearing walk)
141+
community.rs # BUILD: hierarchical Leiden over CE64/SPO-G
142+
ppr.rs # BUILD: HippoRAG reset-distribution PPR (atop hdr_pagerank/spmv)
143+
keyword.rs # BUILD: BM25
144+
summarize.rs # community summaries via Rig oracle → compiled template (W3 dep)
145+
deps: lance-graph-contract, causal-edge, cognitive-shader-driver (RungElevator),
146+
lance-graph (arigraph query surface), bgz-tensor/cam_pq (vector), ndarray.
147+
NO ingest; NO OGAR dep; reads calcified substrate.
148+
149+
lance-graph-contract/ (EXTEND)
150+
src/doc_graph.rs # DocGraphQuery trait + DocGraphView read projection (the shared surface)
151+
152+
lance-graph-ogar/ (EXISTING seam — extend only if the trait needs OGAR ClassView glue)
153+
```
154+
155+
`ogar-doc` (OGAR) consumes `DocGraphQuery` **via the contract** — that is the
156+
correct "feeds ogar-doc": ogar-doc is a **caller** of graphrag's query surface,
157+
never fed data by a lance-graph crate.
158+
159+
## §5. Wave sequencing (aligned to v3 W0–W6 + D-TRI)
160+
161+
Neither crate is a new W-wave. Sequenced **after the W1 keystone**
162+
(`mailbox_owner()` shipped #631; batch-writer W1b in-PR):
163+
164+
- **G0 — baseline probe (FIRST, no code): P-GRAPH-LOADBEARING** (§6). Gate on
165+
truth-architect / measurement-before-synthesis.
166+
- **D-GR-1** — `DocGraphQuery` trait + `DocGraphView` read projection in the
167+
contract (both consumers build against it). Zero SoA writes.
168+
- **D-GR-2** — `crates/graphrag` scaffold + `retrieve.rs` binding **existing**
169+
CAM-PQ + SPO-G hops onto the canonical `RungLevel` / `RungElevator` (#708).
170+
Mirrors the #708 settlement probe (BLOCK ascends → wider walk; FLOW at base).
171+
- **D-GR-3** — BUILD the 3 gaps (`community.rs` Leiden, `ppr.rs` HippoRAG,
172+
`keyword.rs` BM25), **each individually gated** on G0 showing it beats
173+
vector-only.
174+
- **D-GR-4** — `summarize.rs` community summaries via the Rig oracle → compiled
175+
template (depends on W3 `template-runtime`).
176+
- **D-GR-5** — wire `ogar-doc` `reconstruct_document` + "similar/related
177+
documents" to `DocGraphQuery` (cross-repo; baton-audited classid handshake).
178+
179+
Depends-on: W1b (any persisted result is born-stamped), W2 (retrieval cycles =
180+
kanban lanes if a cycle persists), W3 (summaries = compiled templates). Composes
181+
**atop** `oxigraph-arigraph-cognitive-shader-soa-merge-v1` (that plan builds the
182+
unified SoA context; graphrag is the retrieval layer over it — §8).
183+
184+
## §6. Falsifiers / probes (probe-first)
185+
186+
- **P-GRAPH-LOADBEARING (G0, the gate).** On a document corpus, measure retrieval
187+
quality **with vs without** graph traversal (vector-only vs vector+SPO-G+PPR).
188+
KILL condition: if the graph does not beat vector-only on multi-hop/global
189+
questions, do **not** build Leiden/PPR — the graph would be decorative
190+
(graph-librarian-rs anti-pattern). Mirrors the 1BRC addressing-tax probe.
191+
- **P-RUNG-RETRIEVAL.** Hard/contradictory query → BLOCK → elevator ascends →
192+
wider CAUSES..BECOMES mask → higher recall; easy query → FLOW → stays at base
193+
(identity mask, cheap). Mirrors #708's D-TRI-6 settlement probe.
194+
- **P-PPR-MULTIHOP.** HippoRAG-PPR beats vector-only on 2+-hop questions.
195+
- **P-COMMUNITY-GLOBAL.** Leiden community summaries beat flat chunk-concat on
196+
global/thematic questions (the LightRAG dual-level claim).
197+
- **jc battery** (ICC / Spearman / Cronbach) before any new lane-reading backs a
198+
claim (v3 standing gate).
199+
200+
## §7. Alignment hazards (and how this plan avoids each)
201+
202+
1. **write-on-behalf / born-stamped (W1 keystone).** graphrag is read-mostly; the
203+
only writes (persisted community membership / summaries) route the batch-writer
204+
`cast(on_behalf = envelope.mailbox_owner(), …)` — never write-as-self (the smb
205+
`LanceConnector::upsert` ORPHAN-WRITE is the cautionary tale). v3-mailbox-warden
206+
gate.
207+
2. **No-singleton.** No global `Arc<RwLock<KnowledgeGraph>>`. Community/PPR operate
208+
over `MailboxSoA` rows + AriGraph; results are ephemeral or calcified
209+
per-mailbox. graphrag-rs `InferenceEngine` (graph-as-config-struct) is the
210+
explicit anti-exhibit.
211+
3. **16-byte facet never-widen + classid canon-high.** Reuse existing task
212+
classids (`0x080B`/`0x0807`/`0x0808`/`0x080A`, hi-u16 = concept, canon-high). A
213+
persisted community-membership lane = a **new `ValueTenant` discriminant in the
214+
328 B headroom** (additive-at-end, `ENVELOPE_LAYOUT_VERSION` unchanged),
215+
field-isolation-matrix gated by v3-envelope-auditor. `u8:u8` never widened to
216+
u16/u24.
217+
4. **Probe-first / wire-don't-invent.** COMPONENT-MAP + `graphrag-rs-inventory.md`
218+
ARE the precomputed "does it exist" search; §2 leans on them. G0 gates code.
219+
5. **CausalEdge64 duplication + cross-repo baton.** Always qualify
220+
`causal_edge::CausalEdge64`. Any OGAR classid for the D-GR-5 seam is a
221+
**batched** mint + sync-fuse, baton-handoff-auditor gated.
222+
223+
## §8. Relationship to existing plans
224+
225+
- **`graphrag-rs-inventory.md`** (`E-V3-GRAPHRAG-INV-1`) — the component-by-component
226+
automataIA/graphrag-rs audit. **Mandatory pre-read**; this plan does not re-fork.
227+
- **`oxigraph-arigraph-cognitive-shader-soa-merge-v1`** — builds the unified
228+
SoA context (Oxigraph RDF + AriGraph episodes + SPO). graphrag is the
229+
**retrieval layer atop it**; they compose, not compete.
230+
- **`entropy-ladder-spo-rung-v1`** (R1 shipped) — SPO/NARS rung decomposition;
231+
the rung mapping in §3 must reconcile with its entropy-level = fact-position.
232+
- **`temporal-markov-and-style-classes-v1`** (ACTIVE) — governs the retrieval
233+
substrate (episodic axis = Lance versions); graphrag rides `temporal.rs`.
234+
- **`triangle-tenants-gestalt-separation-v1`** (D-TRI-1..6) + **#708** — the
235+
Maslow pyramid graphrag retrieval ascends (§3).
236+
- **`normalized-entity-holy-grail-v1`**, **`episodic-risc-spine-v1`**,
237+
**`cam-pq-production-wiring-v1`**, **`wikidata-lazy-spine-hydration-v1`** —
238+
adjacent; entity-resolution + episodic-addressing + codec + KG-hydration inputs.
239+
240+
## §9. Board hygiene · D-ids · gate
241+
242+
- D-ids: **D-GR-1..5** (this plan) — STATUS_BOARD rows on land.
243+
- New plan → INTEGRATION_PLANS.md prepend (this commit); EPIPHANIES entry only
244+
when a finding lands (probe result), not for the plan itself.
245+
- **Gate:** G0 (P-GRAPH-LOADBEARING) is the first deliverable — **no Leiden/PPR
246+
code before it is green.** D-GR-5 (OGAR seam) is mint-gated + baton-audited.
247+
- **Open questions:** (O1) does `ogar-doc` `reconstruct_document` genuinely need a
248+
runtime graph query, or is the "updated knowledge" re-issue a batch re-persist?
249+
(settles whether D-GR-5 is a live query seam or a re-run). (O2) materialize the
250+
`EpisodicWitness64` column now (helps PPR recency) or stay on
251+
`EpisodicEdges64`+`WitnessTable`? (probe). (O3) reconcile the §3 rung mapping
252+
with `entropy-ladder-spo-rung-v1`'s entropy levels — same ladder or two?

0 commit comments

Comments
 (0)