diff --git a/.arxiv/2026-10_four_russians_m4ri_reverse_lookup_excerpts.md b/.arxiv/2026-10_four_russians_m4ri_reverse_lookup_excerpts.md new file mode 100644 index 000000000..3db73f669 --- /dev/null +++ b/.arxiv/2026-10_four_russians_m4ri_reverse_lookup_excerpts.md @@ -0,0 +1,31 @@ +# Four Russians / M4RI and reverse lookup: lookup tables versus row copying + +**Sources:** [M4RI repository](https://github.com/malb/m4ri) and Martin R. Albrecht, Gregory Bard, Clément Pernet, *Efficient Dense Gaussian Elimination over the Field with Two Elements* ([PDF](https://martinralbrecht.wordpress.com/wp-content/uploads/2011/11/ple.pdf)); plus Y. Lou and Y. Xu, *An Improved Method of the Four Russians for Inversion using Reverse Lookup Table*, IEEE ISIT 2026, [DOI:10.1109/ISIT62367.2026.11653817](https://doi.org/10.1109/ISIT62367.2026.11653817). +**Research status (2026-10-21):** Abstract/readme-grounded extracts. IEEE work is **not an arXiv article**. No implementation or timing evidence for Lance-graph. + +## Source-grounded excerpts + +The classic Four Russians idea precomputes combinations for a **block** of pivot rows so a group of elementary operations may be applied through one indexed combination. In M4RI, those combinations use **GF(2) row XOR**. Blocking limits repeated scanning at the cost of a lookup table. Original M4RI gives an O(n^3/log n)-style reduction for GF(2) elimination. + +Lou–Xu (2026) describe a **reverse lookup table** recording a *row index* for a given row value, rather than a cached *row vector*, and report retaining asymptotic time O(n^3/log n) with auxiliary storage O(n log n) instead of O(n^2). Treat those as the authors' algorithmic claims: no independent local verification and no inference that a QR256 truth matrix needs only 256 B. + +M4RI (library) is GPLv2+; keep it as an **external oracle/process** or follow a clean-room procedure rather than copying source into a differently licensed production crate. + +## Hypothetical transfer to QR256 + +A valuable **index-not-payload** pattern matches the user's zero-copy SoA aim: keep row combinations, codebook references or replay offsets, not duplicated payload vectors. But there are two distinct domains: + +1. **GF(2) M4RI:** XOR cancellations, rank, inversion and parity constraints. +2. **Boolean-semiring blocking:** OR of intersections, existential relational composition, bounded reachability. + +Substituting XOR for OR is forbidden. The second case needs its own small-table scheme or independent bit-matrix oracle; never call that M4RI correctness. + +## Falsifiers / test path F + +- **F1, GF(2) reference:** Compare scalar XOR Gaussian, blocked Four Russians and reverse-index experiment at dimensions 16,64,128,256. Require equal rank and row-equivalent echelon results, not byte-identical pivot order. Measure table construction, bytes, cache misses, XOR count and overall time. +- **F2, Boolean lookup:** For random/sparse/dense QR256 source×target tiles, compare blocked Boolean composition with scalar triple loop and `tile_gemm::BitMat::compose`. Deliberately include duplicate two-hop paths; OR must remain 1 while XOR may cancel to 0. +- **F3, 4-bit vs 8-bit block:** Measure lookup size and build/execute crossover including 1,8,64,1,000,000 subsequent folds. Preregister termination if LUT creation or cache misses outweigh saved Boolean operations. +- **F4, reverse index / zero-copy:** Hold stable borrowed Arrow lanes and compare a stored row vector with an index into the original row. Verify exact value equality, source generation invalidation and no hidden `to_vec` or payload copy. Count index arrays and metadata separately. +- **Silence twin:** Matrix with no profitable reuse chooses ordinary bit-masks or simple scalar path, rather than building a speculative LUT. + +**Note:** A 256×256 Boolean matrix is 65,536 bits = 8 KiB, while 256 u8 selectors are only 256 B. The latter can *address* the former but cannot encode arbitrary 65,536 independent truth bits. diff --git a/.arxiv/2026-10_qr256_gaussian_egglog_rotation_falsifier_paths.md b/.arxiv/2026-10_qr256_gaussian_egglog_rotation_falsifier_paths.md new file mode 100644 index 000000000..7362688c6 --- /dev/null +++ b/.arxiv/2026-10_qr256_gaussian_egglog_rotation_falsifier_paths.md @@ -0,0 +1,80 @@ +# QR256 / Inception: pre-registered hypothetical test paths + +**Date:** 2026-10-21. **Status:** RESEARCH-ONLY TEST DESIGN; not executed. **Owners to review:** relalg, contract, R2IL, mask-risc, Quack, mork-pathmap-probe, crepe-plan-probe, Thought Git. +**Sources:** companion notes in this .arxiv shelf: [2201.06508](2201.06508_gaussian_greedy_reversible_circuit_excerpt.md), [2610.17934](2610.17934_chordal_codes_elimination_order_excerpt.md), [2610.08701](2610.08701_gaussian_partial_pivot_growth_warning_excerpt.md), [2609.28901](2609.28901_quantum_gaussian_reversible_proof_excerpt.md), [2304.04332](2304.04332_egglog_equality_saturation_datalog_excerpt.md), [Four Russians](2026-10_four_russians_m4ri_reverse_lookup_excerpts.md). + +## 0. Representation and authorization contract + +- The **8 KiB 256×256 packed Boolean matrix** is the hot, resident execution state. Its axes must be *typed* (e.g. method×entity membership or source×target adjacency). One does **not** represent a general binary relation by a method×entity unary plane alone. +- A `u8` method palette supplies up to 256 **method selectors**, not 256 matrix-valued truths. A pair of u8 axes forms a 16-bit local coordinate, but its bits require storage or a separately verified virtual backing. +- The **cold sidecar** records global↔local mapping, source generations, borrowed SoA row ordinals, BEFORE/AFTER snapshots or hashes plus reconstructible snapshots, dirty deltas, typed rewrite-law IDs, proof references, and authorized writeback positions. It is not an inner-loop operand. +- During PREPARE, values may be read from the **same** source table and resolved by an attested frozen codebook; counted/witnessed same-table seeks are permitted under [ADMISSION B / #1640](https://github.com/AdaWorldAPI/lance-graph/pull/1640). **Cross-table foreign-value-directed position reads remain forbidden**, including preparation. A preallocated map does not automatically confer admission. Future exceptions demand an explicit new decision. +- Existing zero-copy index map is lab-only [#1659](https://github.com/AdaWorldAPI/lance-graph/pull/1659), co-addressed multi-side join [#1658](https://github.com/AdaWorldAPI/lance-graph/pull/1658), 256-rule CSE [#1653](https://github.com/AdaWorldAPI/lance-graph/pull/1653), Boolean tile [D-RMAP-2](../.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md). Do not substitute GraphBLAS, C/MKL, or a new runtime for already-probed native kernels. + +**Cost accounting:** cold bind/build/index; source bytes touched; hot Boolean folds; cache misses; output tuple materialization; dirty update; proof DAG bytes and checker time; writeback/commit separately. The motivating 20 ms preparation + 2 s operations is a user hypothesis, **not a measurement**. For 1,000,000 operations, amortized preparation is 20 ns/operation if bind is 20 ms; for 4 billion operations it is 5 ps/operation. + +## 1. Paths / hypotheses / independent oracles + +| ID | Hypothesis / path | Oracle | Can-fire (required) | Can-stay-silent / kill | +|---|---|---|---|---| +| P1 | **Same-ordinal AD/Entra/EXO:** witnessed shared identity means A[i],B[i] borrow exact source rows, without joins | independent immutable GUID→row map | distinct fields across 3 sources bind correctly and reused | same row number but different external ID MUST refuse; separate deletion/reorder/stale generation | +| P2 | **Many-to-many group membership:** bound CSR/offset index plus Boolean QR register answers EXISTS without tuple enumeration | scalar multiset join and transitive memberships fixture | preserved duplicates for COUNT/BAG via ordinal sidecar | set truth does not imply multiplicity; group cycle must terminate | +| P3 | **Cross-ontology:** Mouse→Cheese→Milk→Cow and Cheese→Käserei use explicitly typed identity bridges | scalar graph path with source lineage | no repeated global resolution after bind | mismatched ontology homonym and out-of-focus escape MUST refuse/continue; no value-steered foreign read | +| P4 | **Four Russians GF(2):** block-pivot XOR LUT or reverse-index lookup amortizes linear elimination | independent GF(2) Gaussian reference | equal rank, row space and reconstructible pivot schedule | all-zero/singular inputs and tiny matrices must not claim inversion or speculative speedup | +| P5 | **Boolean tile:** bitwise OR/AND relational composition beats repeated raw joins after reuse | scalar Boolean matrix multiply and existing BitMat | 1,8,64,256 focus runs and densities 0.1%,1%,10%,50%,100%; exact tuple/mask equality | two distinct paths to same target: OR=1, XOR=0; bag answer differs from set answer | +| P6 | **Gaussian substitution / E-graph:** a=2b,b=2c,z=2a → z=8c; 256 templates × 4096 frames = 1,048,576 applications | exact symbolic substitution evaluator, Datafrog for typed relational subset | shared proof DAG and lower source-read count; exact equivalence under declared laws | f64 reassociation, u8 saturation, contradictory cycles, nonsharing source demand MUST NOT produce false exactness/compression | +| P7 | **Chordal ordering:** PEO on proven chordal primal graphs reduces intermediate fill | independent PEO/triangulation checker and GF(2) row operations | identical results, reduced peak fill, count cost | 4-cycle without chord; cyclic join that survives semijoins; arbitrary ontology not assumed chordal | +| P8 | **GEPP robustness:** f64 elimination correctness requires conditioning/growth checks | high-precision linear solver | well-conditioned solved LGS agrees | fixed absolute 1e-12 cutoff fails scale invariance; failure cannot contaminate exact QR semiring | +| P9 | **Reversible proof sidecar:** elementary GF(2) row steps invert; lossy Boolean OR needs extra BEFORE evidence | independent replay checker over raw 8 KiB state | before→rule schedule→after, inverse when legally reversible | sidecar holding only final OR matrix cannot reconstruct originals; stale source or no write acknowledgement cannot mark committed | +| P10 | **Dirty algebra:** `CandidateDirty`, `ActualDirty = BEFORE XOR AFTER`, `CommittedDirty` distinct | full recompute from clean snapshot | changed input updates only affected legal dependencies | multiple supports + single deletion must NOT delete surviving reachability; no-op delta remains silent | +| P11 | **Rotation as indexed view:** 3×3 CCW 90° through coordinate arithmetic; 256×256 QR without physical copy | explicit scalar permutation and four-rotation inverse | rotated view matches oracle; 4×90° returns original bytes | rotation of source×target directed relation may change semantics; demand axis/type witness | +| P12 | **Amortization:** reused frame over 1,000,000 folds outperforms rebind each time | identical answers and independent timing harness | break-even observed with cold binding separated from repeated folds | if cold index, staging or proof grows with every fold, reject speedup claim | + +## 2. Rotation (user's exact fixture) + +Input: +```text +1 2 3 +4 5 6 +7 8 9 +``` +Requested CCW 90° rotation: +```text +3 6 9 +2 5 8 +1 4 7 +``` +For N×N matrix in row-major coordinates (zero-based): `new[r,c] = old[c,N-1-r]`. Equivalently an old cell `(r,c)` moves to `(N-1-c,r)`. Check: old(0,2)=3 → new(0,0); old(2,0)=7 → new(2,2)? **No:** under this formula old(2,0)→new(2,2)=7, while the actual fixture has new(2,0)=1 and new(2,2)=7. This explicitly guards against mixing lookup and movement directions. + +For a **logical/view rotation**, do not move bytes. Bind `QRView {base, axis_types, transform: RotateCCW90, generation}`; emit transformed address arithmetic at access time. Composing two quarter-turns yields 180°; four returns identity. Only physicalize if the layout economics warrant it. Track dirty positions through the *same bijection*. A lazy view is zero payload copies, not zero address arithmetic. + +**Semantic hazard:** a binary directed adjacency matrix has both axes drawn from the same entity domain, but rotation is transpose plus reversal, **not generally a graph isomorphism relabeling**. A true consistent node relabeling must apply the **same permutation to both endpoints**, `R'[P(i),P(j)] = R[i,j]`; this is not a 90° image rotation. For method×entity, swapping axes changes their types, so a rotated view is only admissible as a physical/presentation transform with explicit swapped axis metadata and inverse mapping. Never interpret rotated bit positions as new facts. + +## 3. Schedule / proof shape (illustrative, not ABI) + +```text +BIND frame_id, source_seals[], codebook_digest, palette_version +EXEC method8, domain_kind, axes, QR256, operands, focus_version +RULE proof_node_id, law_id, child_refs[], guard_ids[] +DELTA actual_dirty256x256, before_root, after_root +REPLAY inverse_coordinate_map, row_ordinal_refs, deterministic_schedule +COMMIT target_seal, authorization, store_ack // separate from a proof of computation +``` + +A proof DAG compresses repeated **structure**; arbitrary independent output facts still require proportional data/entropy. Hashes prove identity *relative to retained data*, not mathematical validity. Verify each rewrite and independent execution oracle; the checker must never trust the producer's claimed `success`. + +## 4. Experiment stages and gates + +**S0** Documentation review: keep Gaussian REAL_F64, GF2_XOR and BOOL_OR_AND as separate contracts. Inventory the already-existing masked tile and semijoin paths; no duplicated engine. + +**S1** Low-risk independent fixtures: 3×3 rotation, 16/64/256 exact Boolean bit rows, scalar closure/compose, two-path XOR-vs-OR counterexample, stale witness refusals. Run these before benchmarks. + +**S2** Cold optimization research: egglog outside production, bounded fuel, rewrite/equality proof reconstruction on the exact subset, normalized DAG size and signature/provenance check. + +**S3** Optional GF(2) and chordal probes: clean-room method of Four Russians, reverse-index LUT, PEO order, numerical growth adversaries. GF(2) oracle remains external if GPL constraints apply. + +**S4** Million-fold amortization: 256×4096 shared-rule test plus nonsharing control; binding versus execution versus trace-check overhead, source bytes and RSS. Reject without wall-time parity and all refusal guards. + +**S5** Writeback only after contract ruling: do not reinterpret a preparatory gather as admitted; no writes based on hypothetical experiments. + +**Success =** correct typed operations; reproducible and verifiable BEFORE→AFTER; mutation/stale/refusal oracles; measured cost reduction net of preparation and trace. **Failure =** wrong truth bit, lost multiplicity/escape, illegal foreign address, nondeterministic witness, unverifiable derivation, silent coordinate semantic change. diff --git a/.arxiv/2201.06508_gaussian_greedy_reversible_circuit_excerpt.md b/.arxiv/2201.06508_gaussian_greedy_reversible_circuit_excerpt.md new file mode 100644 index 000000000..33a99e613 --- /dev/null +++ b/.arxiv/2201.06508_gaussian_greedy_reversible_circuit_excerpt.md @@ -0,0 +1,25 @@ +# Gaussian elimination versus greedy synthesis: exact reversible linear transforms + +**Source:** Timothée Goubault de Brugière et al., *Gaussian Elimination versus Greedy Methods for the Synthesis of Linear Reversible Circuits*, [arXiv:2201.06508v1](https://arxiv.org/abs/2201.06508), [HTML](https://arxiv.org/html/2201.06508v1), abstract, §1, conclusion. Submitted 2022-01-17, journal reference 2021. +**Research status (2026-10-21):** Primary-source excerpt, research only. Paraphrases, not copied passages. **No Lance-graph benchmark or runtime admission.** + +## Source-grounded excerpt + +The paper synthesizes *linear reversible circuits*, representable by invertible matrices over GF(2). It compares Gaussian elimination coupled with tuned LU/greedy selection against purely greedy circuit synthesis. The claimed output improvements are CNOT **count** and **circuit depth**, not CPU speed for relational joins. Reported preference: an optimized Gaussian/greedy method on random operators above 150 wires; purely greedy methods are near-optimal in their small-domain trials below 30 wires. Both ranges are from this paper's circuits, not transferable performance thresholds. + +Elementary row additions over GF(2) are XORs. Reversibility matters because a sequence of elementary operations can be recorded or inverted. The representation is not the Boolean OR/AND semiring used for existential Datalog joins. + +## Hypothetical transfer to lance-graph + +Use **operation selection and schedule extraction** as an idea: given a GF(2)-typed 256-column matrix, produce a compact palette-coded sequence of row-XOR, swap, and pivot choices. Compare to naive row-by-row elimination, an M4RI-style blocked baseline and a scalar reference. Treat the schedule as a proof witness, not a record of subjective reasoning. + +For relalg existential joins, use separate AND/OR rules; an XOR elimination schedule is **not** interchangeable with relational existence. + +## Falsifiable tests + +- **G1, GF(2) parity:** For random and singular 8/64/256-column inputs, apply the extracted elementary-operation schedule and independently verify `E · A = R` under GF(2). If E is not preserved, replay the logged elementary steps; do not claim inversion for singular input. +- **G2, no false semiring parity:** Select a two-path graph with two derivations of the same target. XOR yields 0, OR yields 1. Any attempt to reuse the XOR schedule as a Datalog EXISTS kernel must refuse or fail this negative oracle. +- **G3, cost model:** Compare CNOT-oriented schedule length with actual scalar/SIMD wall time, cold plan cost, replay bytes and cache traffic. A shorter circuit is *not* automatically faster on the CPU. +- **Silence twin:** A no-op row addition or a permuted list of independent operations must not emit a committed QR dirty bit. + +**Existing related work:** [D-RMAP-2 tile plan](../.claude/plans/d-rmap-2-context-tile-algebra-probe-v1.md), [D-RELALG-UNIFIED](../.claude/plans/d-relalg-unified-v1.md). diff --git a/.arxiv/2304.04332_egglog_equality_saturation_datalog_excerpt.md b/.arxiv/2304.04332_egglog_equality_saturation_datalog_excerpt.md new file mode 100644 index 000000000..ae53d40c5 --- /dev/null +++ b/.arxiv/2304.04332_egglog_equality_saturation_datalog_excerpt.md @@ -0,0 +1,30 @@ +# egglog: share equivalent expressions, then extract a checked relational program + +**Source:** Yihong Zhang et al., *Better Together: Unifying Datalog and Equality Saturation*, [arXiv:2304.04332v4](https://arxiv.org/abs/2304.04332), [HTML](https://arxiv.org/html/2304.04332v4); revised 2023-05-15. §§1–5 and appendices on equation solving/matrices. +**Implementation source:** [egraphs-good/egglog](https://github.com/egraphs-good/egglog), [3.0 proof documentation](https://egraphs-good.github.io/egglog/docs/egglog/proof/index.html). **Research status:** primary-source and API reading, not library integration. + +## Source-grounded excerpt + +egglog combines Datalog's incremental / semi-naive rule processing with equality saturation's **congruence closure**, term rewriting and extraction. Extensional relational facts, functional dependencies and equality classes must be modeled distinctly. The paper demonstrates pointer-analysis and floating-point rewrite applications. It does not assert that every Datalog join or side-effecting program admits an equality-preserving Boolean rewrite. + +Current egglog 3.0 has a read-only **proof reconstruction** API. Its `ProofStore` is a hash-consed arena sharing subproofs via `ProofId`; proofs justify *equality propositions between terms*. That alone is not a proof that a physical Lance write succeeded. + +## Hypothetical transfer + +Cold, version-pinned optimizer outside the hot mask-RISC path: +`relalg/R2IL terms -> sorted/canonicalized predicates -> e-class saturation -> extract minimal witnessed DAG -> palette8 opcodes -> QR256 folds`. + +Metric should combine bindings, source reads, mask materializations, dirty propagation and replay cost. A shorter AST can perform worse if extraction destroys co-addressing or induces repeated preparation. + +Small motivating equality: `a = 2*b; b = 2*c; z = 2*a`, hence `z = 8*c` under exact integer arithmetic. Under saturating u8 arithmetic or f64 reassociation this needs a separate law or a refusal. Cyclic equations require conflict/termination handling; avoid unbounded saturation. + +## Falsifiers / test path E + +- **E1, substitution:** Prove `z == 8*c` with exact integer constants and capture a replayable DAG. Remove a premise; the proof must fail or be incomplete. +- **E2, million requested applications:** 256 templates × 4,096 focus contexts = 1,048,576 applications. Measure input program records, shared e-nodes/e-classes, extracted program op count, certificate bytes, planner time, memory, validation time, and end-to-end execution. Compare against an intentionally nonsharing workload; no universal small-certificate promise. +- **E3, typed laws:** Existential `OR` versus parity `XOR` and set versus bag COUNT must not unify. Duplicate derivations give a mandatory counterexample. +- **E4, proof/witness:** Check extracted expression equivalence and separately validate BEFORE/AFTER by independent QR scalar oracle; stale coordinate/palette generation refuses. +- **E5, termination:** Recursive cyclic term rules obey fuel/node limits; no infinite expansion; compare bound-limited saturation to ordinary relalg normalization. +- **Silence twin:** An algebraically redundant rewrite changes neither result nor committed dirty bit. + +**Existing seam:** [D-RELALG-UNIFIED](../.claude/plans/d-relalg-unified-v1.md), [#1653](https://github.com/AdaWorldAPI/lance-graph/pull/1653), [#1644](https://github.com/AdaWorldAPI/lance-graph/pull/1644). **No new production engine authorized.** diff --git a/.arxiv/2609.28901_quantum_gaussian_reversible_proof_excerpt.md b/.arxiv/2609.28901_quantum_gaussian_reversible_proof_excerpt.md new file mode 100644 index 000000000..1df2d454f --- /dev/null +++ b/.arxiv/2609.28901_quantum_gaussian_reversible_proof_excerpt.md @@ -0,0 +1,26 @@ +# Quantum Gaussian circuit: the reversible-history analogy + +**Source:** Hochang Lee, Kyung Chul Jeong, Panjin Kim, *A Quantum Circuit for Gaussian Elimination*, [arXiv:2609.28901v1](https://arxiv.org/abs/2609.28901), [HTML](https://arxiv.org/html/2609.28901v1); 2026-09-24. Abstract, §§1, 2, 4. +**Research status (2026-10-21):** Primary-source excerpt. **Quantum speedup of Lance-graph is not claimed.** + +## Source-grounded excerpt + +The authors give a **reversible quantum circuit** for Gaussian elimination over arbitrary finite fields (generalizing prior GF(2) work). Their pseudo-row-echelon representation preserves sufficient information to make the normally many-to-one elimination transformation reversible. They analyze Toffoli gate count/depth and garbage/supplemental space. One reported design uses no extra **persistent garbage** beyond input storage; this does not mean there are no temporary qubits, classical pointers, or cheap quantum hardware for QR256. + +The paper explicitly distinguishes a reversible state transform from a purely lossy row echelon reduction, since an irreversible transform cannot be inverted from its output alone. + +## Hypothetical transfer + +For **classical**, finite, deterministic QR transitions, borrow the principle: save a minimal reversible operation schedule or enough before/delta evidence to **verify** the transition and optionally undo it, while keeping read-only SoA buffers untouched. Writeback remains a separately acknowledged transaction. Mathematical equivalence of expressions alone does not certify an external database commit. + +The user's proposed 8 KiB QR truth state is *classical*. No qubit, entanglement, quantum oracle, or performance claim follows from using a reversible trace encoding. + +## Falsifiers / test path Q + +- **Q1, reverse schedule:** GF(2) elementary XOR and row-swap steps replay forward and backward to the exact original bit matrix. Record pivot choices and versioned initial state; compare full 8 KiB. +- **Q2, lossy step:** Pure `OR`, projection and existential folding are many-to-one. A log that contains only final QR plus an operation name must **fail** reversal. Add BEFORE+delta or another adequate witness and verify rollback. +- **Q3, resource accounting:** Compare full snapshots, XOR dirty deltas, hash-consed proof DAG, and operation schedule. Report bytes, cold verification time, inverse replay time and refusal rate. +- **Q4, writeback separation:** An unchanged source generation permits verified computation; changing source generation before commit must refuse or revalidate. Simulate failure between prepare, sidecar approval and write acknowledgement. +- **Silence twin:** No-op transition emits no actual dirty bits and never logs a successful data-store commit. + +**Verdict:** Research transfer is *replay/information preservation*, not a quantum execution backend. diff --git a/.arxiv/2610.08701_gaussian_partial_pivot_growth_warning_excerpt.md b/.arxiv/2610.08701_gaussian_partial_pivot_growth_warning_excerpt.md new file mode 100644 index 000000000..e17ce1b05 --- /dev/null +++ b/.arxiv/2610.08701_gaussian_partial_pivot_growth_warning_excerpt.md @@ -0,0 +1,25 @@ +# Large growth happens: numerical Gaussian elimination is not Boolean fold algebra + +**Source:** Daniel A. Spielman and Xifan Yu, *Large Growth Happens: Gaussian Elimination with Partial Pivoting on Random Matrices*, [arXiv:2610.08701v1](https://arxiv.org/abs/2610.08701), [HTML](https://arxiv.org/html/2610.08701v1); 2026-10-06. Abstract, §§1, 2.1–2.2. +**Research status (2026-10-21):** Primary source read; numerical-stability negative-control source, not a claimed QR speedup. + +## Source-grounded excerpt + +The paper establishes a nontrivial **lower bound** on the probability of large element growth during Gaussian elimination with partial pivoting (GEPP), for random Gaussian matrices. It rules out certain previously conjectured fast-decaying tails. It works with *real-valued* numerical matrices and floating-point stability analysis, not matrices over GF(2) and not existential Boolean relations. + +A growth factor compares the largest value in the upper triangular factor with the largest value in the input. Even if factorization gives a formally correct decomposition, floating-point results can lose accuracy when intermediate magnitudes grow. + +## Hypothetical relevance + +Explicitly type every proposed elimination `AlgebraKind`: REAL_F64, GF2_XOR, BOOL_OR_AND, COUNT_BAG, or other. Never let an e-graph or 8-bit method palette silently exchange associativity/rewrites between floating-point and exact finite/Boolean domains. + +In the user's illustrative `solve_lgs` implementation, partial pivoting is sensible, but `pivot.abs() < 1e-12` is an absolute cutoff and scale-dependent. Validate all input rows, reject nonfinite entries and test residual/error on badly scaled and near-singular cases. + +## Falsifiers / test path N + +- **N1, real systems:** Wilkinson-like and other adverse-growth matrices, near-singular, badly scaled, exact and ill-conditioned controls. Check residual norm, relative solution error, growth factor and pivot cutoff sensitivity against a high-precision reference. +- **N2, metamorphic scaling:** Multiply both A and b by a small/large nonzero scalar. Mathematical solution is unchanged; a fixed absolute pivot cutoff can yield different refusal status. This is an expected negative finding to fix, not a numerical success. +- **N3, rewrite legality:** For f64, reassociation may change results; the cold extractor either attaches a proven tolerance bound or refuses exact-proof labeling. Contrast with GF(2) XOR and Boolean OR/AND, which are exact in their respective domains. +- **Silence twin:** An unchanged, version-pinned QR256 Boolean mask must not incur floating-point pivot work. + +**Verdict:** Strong **guardrail** for any f64 path; paper makes no claim about graph-fold throughput. diff --git a/.arxiv/2610.17934_chordal_codes_elimination_order_excerpt.md b/.arxiv/2610.17934_chordal_codes_elimination_order_excerpt.md new file mode 100644 index 000000000..27e3b6ec5 --- /dev/null +++ b/.arxiv/2610.17934_chordal_codes_elimination_order_excerpt.md @@ -0,0 +1,26 @@ +# Chordal codes: eliminate in an order that avoids fill-in + +**Source:** Yulin Chen, Laigang Guo, Tong Li, Mingyang Zhu, *Chordal Codes: Code Construction and Efficient Gaussian-Elimination Decoding*, [arXiv:2610.17934v1](https://arxiv.org/abs/2610.17934), [HTML](https://arxiv.org/html/2610.17934v1); submitted 2026-10-13. §§II-A, III, IV. +**Research status (2026-10-21):** Primary-source excerpt; proposed applications below are **hypotheses**, not demonstrated by the paper for QR256. + +## Source-grounded excerpt + +Maximum-likelihood decoding over an erasure channel becomes a finite-field linear system involving a column-submatrix of the parity-check matrix. Gaussian elimination may introduce **fill-in**, extra nonzeros that cost memory and work. The paper defines chordal codes whose **primal graph** is chordal and analyzes a **perfect elimination ordering** (PEO). Erasure-induced column submatrices inherit the relevant chordal primal-graph property. The advantage concerns that *structured family*, not an assertion that any arbitrary graph has a zero-fill order. + +Here the primal graph has one vertex per matrix **column**, with an edge whenever two columns co-occur as nonzero entries of some matrix row. It is not automatically the directed relationship graph of an ontology. + +## Hypothetical transfer to lance-graph + +The promising operation is **plan ordering before elimination**, not a new math backend. For a set of relational variables and shared subexpressions, build a variable-interaction graph; if a proven legal decomposition exists, explore a low-fill elimination order. Preserve the precise operation algebra (GF(2), Boolean OR/AND, or bag counts) and its equivalence obligations. Plan-level fill-in means new intermediate dependencies/materialized planes; it is an **analogy** until measured. + +A QR256 state has only 8 KiB of packed truth bits, but an unfortunate elimination order could still create large intermediate products, explanation traces, or off-focus edges. + +## Falsifiers / test path C + +- **C1:** Construct chordal and nonchordal **primal** graphs from typed linear equations. Compute a PEO with an independent recognition oracle; compare peak fill-in, rank, correctness and row-XOR counts against natural order, reversed order and greedy min-fill. A nonchordal input must never be labeled no-fill. +- **C2:** For acyclic star/chain relalg bodies, compare elimination orders on **identical** set semantics, including a 256-method/256-slot focus. Measure intermediate planes and DAG nodes, not just number of rewrites. +- **C3:** A cyclic triangle where semijoins do not filter anything must remain a counterexample: identical pairwise domain projections do not guarantee the whole join is nonempty (cf. [#1587](https://github.com/AdaWorldAPI/lance-graph/pull/1587)). +- **C4:** Hubs, power-law degree, reordered row ordinals and external focus escapes. If a PEO-based claim misses an escape or increases total time on a sparse case, stop promotion. +- **Silence twin:** A coordinate permutation that only changes presentation must not create new facts or writeback deltas. + +**Verdict:** Research as a *cold optimizer heuristic*; no claim that chordal erasure-code complexity transfers wholesale to Datalog. diff --git a/.claude/plans/qr256-powershell-ordinal-polyfill-bridge-v1.md b/.claude/plans/qr256-powershell-ordinal-polyfill-bridge-v1.md new file mode 100644 index 000000000..a3d138acd --- /dev/null +++ b/.claude/plans/qr256-powershell-ordinal-polyfill-bridge-v1.md @@ -0,0 +1,127 @@ +# D-QR-POLY: PowerShell positional template → ordered borrowed QR256 → algebra → replay + +**Date:** 2026-10-21. **Status:** DESIGN / CROSS-PR MAPPING, research only. No production implementation or admission. +**Physical source:** [PR #1659](https://github.com/AdaWorldAPI/lance-graph/pull/1659) (open lab, branch `ccr-0455e606-wmtsor`, head `cd499c4e` at inspection). +**Logical research:** [PR #1660](https://github.com/AdaWorldAPI/lance-graph/pull/1660) (this docs-only draft). +**Coordination:** [Linear EXO-50](https://linear.app/exo-red/issue/EXO-50/agent-message-bus-gpt-xgrok-ccsession-id). +**Prior rulings:** [#1640](https://github.com/AdaWorldAPI/lance-graph/pull/1640) ADMISSION B, [D-RELALG](d-relalg-unified-v1.md), [D-RMAP-1A](d-rmap-separate-tables-thought-palettes-v1.md), [D-RMAP-2](d-rmap-2-context-tile-algebra-probe-v1.md), [#1657](https://github.com/AdaWorldAPI/lance-graph/pull/1657) P0 folding gate. + +## 1. Correct the history: #1559 and #1560 are not the PowerShell polyfill + +- [#1559](https://github.com/AdaWorldAPI/lance-graph/pull/1559) is OGAR authorized document resolution and RBAC field/row masking. +- [#1560](https://github.com/AdaWorldAPI/lance-graph/pull/1560) is Pearl counterfactual verdict, Rubicon pin and CE64 ledger correction. +- Neither introduced the PowerShell-shaped positional wrapper. The exact `"{0}{1}" -f $laneA[$i],$laneB[$i]` analogy is explicitly documented in the code-level notes of **#1659**. EXO-50's 2026-10-21 tuple/format discussion likewise clarifies that `"{0}..{15}" -f $lane.split` is a *16-position carving instruction*, not a new `QualiaI4_16D` type or a tuple decoder. +- [#1504](https://github.com/AdaWorldAPI/lance-graph/pull/1504) previously demonstrated a separate class-qualified method declaration → Quack/R2IL algebra polyfill. It is prior art for declarative lowering, **not proof** that a reusable position-template wrapper currently exists. + +**New missing seam:** a tiny declarative **position-template compiler** between a front-end's named operands and a #1659 `JoinDomain/Side/Run` or same-ordinal row cursor. It needs no string construction, row object materialization, extra interpreter or new graph engine. + +## 2. Literal PowerShell model, formalized + +The motivating notation is: + +```powershell +"{0}{1}" -f $laneG1[$i], $laneG2[$i] +"{0}..{15}" -f $lane.split +``` + +Interpret the *placeholders* as compile-time slot selectors, and `$i` as a bound ordinal under one witnessed address domain. **Do not** implement PowerShell formatting at runtime. String formatting is only the metaphor. + +**Same-ordinal fast path:** `Template([Operand(laneA, i), Operand(laneB, i)])`, where equality of row identity across lanes has been explicitly attested by the same sealed order, source/target identity mapping and generation. Identical numeric offsets in unrelated tables do **not** witness identity. + +**Co-addressed multi-side path (#1659):** for local join slot `s`, each side supplies its *own* `off[s]..off[s+1]` and `pos[]`, and its original lane is read as `lane_side[pos_side[k]]`. This preserves source row positions without allocating copies of the source payload. Duplicate side entries remain available to bag terminals. Domain identity and offsets must be versioned. + +**16-byte carving template:** a template with placeholders 0..15 selects byte positions of one contiguous borrowed value/lane with a declared 16-byte layout. It does not reinterpret the borrowed bytes as a new type or allocate 16 row objects. Bounds and field/scope authorization apply before reading. + +**Important:** `"{0}{1}"` is not a collision-free key encoding for arbitrary values (e.g., `1,23` vs `12,3`). If both are `u8` and a packed key is necessary, `(u16::from(a) << 8) | u16::from(b)` is injective; otherwise use typed pairs or length framing. The intended hot path should avoid forming a key at all. + +## 3. Precise ownership: address/prefetch sidecar vs QR state vs replay sidecar + +The word **sidecar** describes two different objects and must never hide their cost or lifetime: + +| Piece | Resident / cold | Owns | Does NOT own | +|---|---|---|---| +| `BorrowedSoA` | Original source | Immutable Arrow/Lance payload buffers, their true row ordinals, borrowed/refcounted lifetimes | New tuple payload or graph matrix copy | +| `OrderedAddressView` (working name; proposed) | **Prepared, read-side hot metadata** | `JoinDomain`, `present`, `off`, `pos`, borrowed `ScalarBuffer` owners, order/codebook/version/scope witnesses; ordinal iteration and optional one-slot-ahead prefetch | Source payload clones or truth about a foreign table | +| `QR256` | **Hot execution** | Exactly 256×256 bits = 8 KiB for one fully materialized Boolean matrix, plus independently accounted scratch/dirty planes | Payloads, global IDs, bag multiplicity, implicit ontology semantics | +| `ProofReplaySidecar` | **Cold trace/writeback** | BEFORE/AFTER reference or reconstructible deltas, mapping/version witness IDs, template/rewrite IDs, proof DAG, dirty masks, authorized writeback ordinals, commit receipts | Runtime tuple values, permission to bypass auth, automatic completion of external write | + +The materialized QR **package** is `OrderedAddressView + QR256`: a borrowed pointer/index/prefetch substrate plus an explicit bit-packed relational projection. It is not 8 KiB in total; only the dense Boolean *plane* is 8 KiB. The view adds `present/off/pos`, indexes, witness metadata and buffer handles. In Rust prefer bounds-checked slices and `ScalarBuffer`/Arc ownership over dangling raw pointers. + +**Two QR axis contracts (never conflate):** +1. `Method8 × LocalSlot8`: unary method/entity membership or operation schedule, indexed by a pair of `u8` coordinates (a `u16` key can select a bit). +2. `SourceSlot8 × TargetSlot8` **per predicate**: binary graph adjacency suitable for Boolean `mxv/mxm`, closure and reachability. A bank of 256 such predicates occupies up to 2 MiB of dense truth planes, not one 8 KiB plane. + +Pseudocode (an explanatory declaration, **not** a committed API): + +```text +template "{0}{1}" -> ordered borrow operands (side0, side1) +resolve template -> witnessed per-side ordinal ranges / lane positions +prepare QR[map, focus] -> OR-aggregate relationship presence into 256² bits +BlasGraph-style OR/AND -> QR resident mask-risc / existing ndarray kernels +emit demand Count/Exists -> masks/offsets, zero tuple allocations +emit demand Rows -> borrowed ordinal lens; only materialize if explicitly requested +record proof -> cold BEFORE, rule DAG, actual dirty XOR, AFTER +authorized writeback -> separate acknowledged storage transaction +``` + +## 4. BlasGraph reimagined as QR, not replaced by another graph engine + +Existing `crates/lance-graph/src/graph/blasgraph/{matrix,semiring}.rs` implements GraphBLAS-inspired operations using **owned CSR/CSC** and `BitVec` values. It is *not* the 8 KiB per-predicate borrowed QR tile, and its transpose/extract may allocate. Do not claim zero-copy conversion from BlasGraph. + +Re-use its **operator semantics** and independent oracle values: Boolean `lor_land`, masked `mxv/mxm`, `ewise`, reductions and transposition. The hot candidate stays the **already existing** QR/bit-tile/mask-risc/ndarray code described in D-RMAP-2; do not implement a new GraphBLAS engine, import C, or duplicate kernels. + +`QR_A ∘ QR_B` (two-hop existence) means `C[i,j] = OR_k(A[i,k] AND B[k,j])`, which differs from parity `XOR`. Set- and bag-count semantics must remain explicitly separate. A **logical rotation/transpose view** changes how row/column coordinates are read; treating 90° image rotation as graph edge relabeling is invalid without a separate proof. + +## 5. Binding and prefetch economics + +Actual #1659 evidence: `Side { generation, present, off, pos, lane }`; `Run` dereferences original values using `lane[pos[k]]`. Dense domain uses `rank+popcount` and sparse uses sorted IDs. `Run::prefetch` issues `ndarray::simd::prefetch_read_t0` for a subsequent run's first lane element. These are source-code facts, not proof that memory latency is eliminated. + +**Failed #1659 gate:** on unpruned 50k-row fixtures, preparing `pos` by counting sort of unordered rows costs outside the µs budget; random count/scatter writes dominate. Proposed *but not decided*: attest already-sealed key order and derive run offsets without scatter. See [EXO-114](https://linear.app/exo-red/issue/EXO-114) for order classification and [EXO-46](https://linear.app/exo-red/issue/EXO-46) for the global→focus admission decision. A key ordered for one role may not be ordered for both sides, requiring separate declared indexes or a permutation. **Do not resort borrowed source rows.** + +Amortized cost: +`T(K) = T_attest + T_index + T_QR_projection + K·T_mask_fold + T_terminal + T_replay + T_writeback`. +Report indexing/warmup, cache misses and output materialization independently. The 20 ms bind + 4 billion logical operations / 2 s motivating example is a hypothetical budget, **not a benchmark**. Per-fold nanosecond/µs gates and whole-batch elapsed time must be labeled distinctly. If a supposedly hot per-fold path measures in ms, treat it as a failed gate per #1657, not a success. + +## 6. Steps, source ownership and issue links + +| Phase | Deliverable | Reuse / owner | Admission | +|---|---|---|---| +| **S0: source census** | Inspect actual #1659 head and #1660 docs; identify literal `"{0}{1}"` selector without duplicating tuple decoders | #1659 `join.rs`, D-RMAP-1A, EXO-50 coordination | Docs/analysis | +| **S1: typed wrapper** | Minimal declarative projection shape `Template(selector[])` over borrowed lane+ordinal, checked matching frame/generation/auth; 16-lane position fixture | #1659 owns physical `Run`; [EXO-107](https://linear.app/exo-red/issue/EXO-107) lens gap | Lab only | +| **S2: order/inversion** | Run-offset fast path from preattested sorted key vs existing `pos[]` scatter; independent order witness including same-row identity | [EXO-114](https://linear.app/exo-red/issue/EXO-114), #1659 failed gate | Write-path ruling required | +| **S3: QR projection** | Materialize one typed 8 KiB Boolean source×target tile from borrowed ordinal views; store optional dirty mask; no row copies | D-RMAP-2, #1660 P5 | Lab only, obey #1640 | +| **S4: BlasGraph parity** | Match Boolean `AND/OR`, composition, masked `mxv/mxm`, COUNT/EXISTS without tuples | existing BlasGraph/bit-tile; [EXO-48](https://linear.app/exo-red/issue/EXO-48) | No new backend | +| **S5: rotations/views** | 3×3 `123/456/789 → 369/258/147`, 256×256 permutations and dirty maps | #1660 P11, typed axes | Refuse semantic edge changes | +| **S6: proof compression** | Palette-u8 rewrite IDs, egglog equivalence DAG, Gaussian/GF(2) typed separation, BEFORE/AFTER/dirty checker | #1660 P4/P6/P7/P9/P10 | Cold only | +| **S7: warm million-fold** | 256×4096 sharing case vs no-sharing; count source bytes and proof bytes as well as elapsed time | #1653, #1657, #1660 P12 | No production merge | +| **S8: writeback** | Separate commit receipt/transaction, inverses only when algebraically legal, generation-aware replay | Thought Git EXO-55/58/98 | Later gate | + +Other overlaps: [EXO-79](https://linear.app/exo-red/issue/EXO-79) D-MPS physical join; [EXO-86](https://linear.app/exo-red/issue/EXO-86) contract/relalg; [EXO-102](https://linear.app/exo-red/issue/EXO-102) D-RU; [EXO-104](https://linear.app/exo-red/issue/EXO-104) tuple violation; [EXO-115](https://linear.app/exo-red/issue/EXO-115) Quack subset→contract; [EXO-50](https://linear.app/exo-red/issue/EXO-50) session bus. Do not claim or alter these owners without handoff. + +## 7. Research probes / independent falsifiers + +| Probe | Positive expectation | Mandatory negative / stop gate | +|---|---|---| +| **QR-P01: template** | `"{0}{1}"` references two borrowed lanes at the witnessed ordinal; 16 positions projected without `String`, `Vec`, or `to_vec` | Change one lane's row identity but retain equal numeric index: **refuse** | +| **QR-P02: sparse/dense domain** | `rank+popcount` and ordered-sparse path yield identical stable slots | `pos` out of bounds, duplicate rekey, stale source seal or unseen key: **refuse** | +| **QR-P03: ordered fast path** | Already sealed key order yields offsets without copied payload or re-sort; benchmark 1/8/64 reuse | Unsorted or only lexicographically LE-image sorted key: **refuse** | +| **QR-P04: prefetch** | Compare no-prefetch, 1-slot and N-slot lookahead using *identical* borrowed accesses; report cache misses and ns | No win or worse memory traffic => no promotion; hint is not a gather exception | +| **QR-P05: QR composition** | 256×256 Boolean `OR_k(AND)` matches scalar oracle + existing bit-tile / BlasGraph oracle | Two distinct paths to same target: XOR parity 0 != Boolean existence 1 | +| **QR-P06: set/bag/terminal** | `EXISTS` and `COUNT` are mask/offset folds; `Rows` is an on-demand borrowed lens preserving original ordinals | Duplicate memberships, cross-product and external-focus escapes must not be silently dropped or heap materialized | +| **QR-P07: view rotation** | 3×3 given fixture, 4×90° identity, 256×256 transpose/flip/CCW view and same-bijection dirty propagation | Method×entity axes swapped without metadata; directed graph semantics altered: **refuse** | +| **QR-P08: mathematical compression** | `a=2b; b=2c; z=2a => z=8c`; 256 templates × 4096 frames; proof DAG and optimizer cost measured | Saturating u8, float regrouping, conflicting cycles, independent expressions show no false equivalence/small-certificate guarantee | +| **QR-P09: replay/writeback** | Version-pinned BEFORE + exact dirty + AFTER reconstructs independent scalar state; actual changes traced to source ordinals | Only AFTER exists for lossy OR, ACL revoked or DB ack absent: **never claim reversible/committed** | +| **QR-P10: security/admission** | Same-table witnessed seeks, frozen-codebook lookups where admitted; cross-source co-address only after independently witnessed identity | Value from table A drives position in table B at any phase: **refuse under ADMISSION B** | + +Independent reference should be *structurally different* from the candidate: scalar bit enumeration, independent GUID mapping, an external GF(2) solver or existing BlasGraph where truly equivalent. Reusing the same mask-risc code as both oracle and candidate does not certify correctness. + +## 8. Decision requests before any implementation + +1. **Owner assignment:** #1659's lab binder remains with its current session; #1660 remains docs/research. Who owns the tiny named-operand→position-template wrapper/lens? Coordinate through EXO-50 before code. +2. **Ordered write-path:** Is a sealed key-order witness and a dedicated key-sort at write time permissible, and who owns that index? One shared row order rarely satisfies every possible key. +3. **QR physical boundary:** Which specific typed axis scheme is approved for the first tile, source×target adjacency vs method×slot membership? No generic untyped 256×256 claim. +4. **Commit surface:** Does replay sidecar remain read-only until Thought Git writer receives separate authorization and transaction acknowledgment? Recommended yes. +5. **Admissibility:** No cross-table positional foreign gather or new interpreter is implicitly permitted by the phrase *PowerShell polyfill*. + +**Success:** small declarative lens compiles to #1659 borrowed ordinal access, QR truth folding reuses existing mask-risc and BlasGraph semantics, no heap tuples, stale-order/scope guards work, cold proof replay is independently checkable, and all performance claims include binding + sidecar + output costs. +**Failure:** source payload copied, separate graph engine, illegitimate gather, false row identity, XOR-for-OR, silently dropped multiplicity/provenance, stale commit witness or unexplained ms hot-fold path.